Information

[2023/02/03] Notification of recalculation of final evaluation score

From 9:30 on Monday, February 6th, we will replace the benchmark function for the final evaluation and recalculate the final evaluation score of the files you have submitted so far.No action is required from you for this support, but please note that you will not be able to post new files while the score of your own submitted file is being recalculated.In addition, there is no change in the provisional evaluation score.

[2023/02/02] Apology and notification to all participants

We will extend the qualifying period of this contest and replace the benchmark function for the final evaluation.In addition, the qualifying period after the extension will be until 2/9 24:00.The contest should have ended at 24:00 on February 2nd, but due to a mistake in setting the deadline, it ended at 0:00 on February 2nd. It is a response to what happened.At around 9:35 on February 2nd, we returned the contest to its original state and resumed the contest. In order to maintain fairness, we decided to take the above measures.As soon as the replacement of the benchmark function for the final evaluation is completed, all past submission files except for error submissions will be recalculated sequentially and the final evaluation value will be updated.

We apologize for any inconvenience this may cause.There is no problem in submitting new files during the extended qualifying period.

[2022/12/23] We have started accepting submissions.
[2022/12/23] Deleted the item "Assessment of the size and execution speed" in the evaluation.
[2022/12/19]  Updated schedule.

Objective

While early social implementation of artificial intelligence technology is expected, technology that accelerates the introduction of artificial intelligence technology is crucial. When introducing artificial intelligence technology there is an issue in that it takes a tremendous amount of time to tune establishment and studying of artificial intelligence models. In order to solve this problem, NEDO has a project called “Development of integration technology that becomes the core of the next-generation artificial intelligence and robots/ Research and development that expands the areas of application of artificial intelligence technology”: As a common fundamental technology that promotes early social implementation of artificial intelligence technology, the project develops technology that shortens the time to tune establishment and studying of the artificial intelligence models as well as an artificial intelligence acceleration modules using the technology. 

Specifically, technologies such as exploration for Hyperparameter Optimization (HPO hereafter) and Neutral Architecture Search (NAS hereafter) become important as a tuning task in practical settings. To solve these issues we have developed an HPO Library “aiaccel” which optimizes hyperparameters in a short time for the research and development theme “Establishment of cyber physical value chain and AI introduction acceleration technology.” This Library is published as open-source software which can accommodate a newly developed HPO module. The Library can also utilize the AI Bridging Cloud Infrastructure (ABCI hereafter) which is owned by the National Institute of Advanced Industrial Science and Technology (AIST) as the development environment.

In order to further promote this project it is essential to induce technological development in a competitive environment which is joined by a wide range of participants including universities, corporations and overseas researchers.  Therefore, we decided to hold an AI introduction acceleration module competition which will induce the creation of AI introduction acceleration modules that display excellent individual performance and versatility.  

For the inaugural year we are holding a competition that assesses the performance of HPO modules to optimize hyperparameters, such as learning rate. Please take this wonderful opportunity to try using the ABCI, a world’s largest computing infrastructure for artificial intelligence processing.   


Overview

The competition consists of the first and final rounds. Ten qualifying teams which passed the first round can proceed to the final match. 


First-Round
Final Round
Period
December 16, 2022 to February 2, 2023.
Scheduled to start at the middle of February 2023.
TaskOptimization of HPO module
Optimization of HPO module (Deep Learning)
Requirement
Developing a module that works with aiaccel.
Developing a module that works with aiaccel.
Eligibility
Must be able to participate in the final round.

Must add and publish the open-source license to the source code if awarded with a prize. 

Before the start of the competition personal identification and documents required to use the ABCI must be submitted(※).

Test Environment
Implemented with a computational resource owned by the individual participants.
The optimization is tested using the ABCI’s computational resource through SIGNATE by posting the HPO module.
Assessment Environment
Assessed with SIGNATE by posting the HPO module.
Assessed with SIGNATE by posting the HPO module.
Assessment Method
Optimization accuracy is assessed using multiple benchmark functions.
Optimization accuracy of hyperparameters of deep learning is assessed. 
Leaderboard Update Frequency
Real time
Once a week.
PrizeTop ten teams with high accuracy: Eligibility to enter the final round.

Quantitative evaluation: ¥700,000 × 5 teams

Qualitative evaluation: ¥250,000 × 2 teams


Tasks for the First Round 

The participants will develop an HPO module that complies with the IF specifications* regulated by the HPO Library “aiaccel”. The performance will be assessed using multiple benchmarks functions. 

*See Data tab for details.


Optimization image




※Note:Terms of agreement in document required to use the ABCI in the final round. 

  1. You must meet the following requirements:
    2. You must not be in violation of the Foreign Exchange and Foreign Trade Act (No. 228 of 1949), government and ministerial ordinances and notifications related to security export control based on the Act, security export control regulation held by the National Institute of Advanced Industrial Science and Technology, and other regulations held by the National Institute of Advanced Industrial Science and Technology based on the said regulation, and you must fall under any of the following:
    • A Japanese citizen who resides in Japan.
    • A Japanese citizen who works for a diplomatic mission abroad.
    • An alien who works for an office in Japan.
    • An alien with over six months having passed since entering Japan.
  2. Being able to provide the following personal information on all team members:
    • First and last name.
    • The name of the institution or corporation the participant belongs to or its nationality.
    • The name of the department of the above institution/corporation.
    • Email address.
    • Length of stay in Japan.
    • Organization abroad the participant belongs to.
    • A copy of personal identification with photo (personal identification with photo means a document that we can determine that the description written on the document exists and the corporation the user belongs to guarantees the user’s identification to the third party), or a personal identification without photo issued by the corporation the user belongs to and a copy of the driver’s license with photo. 
  3. The participant must not fall under any of the following: 
    • In an employment contract with foreign corporation and government. 
    • Obtaining or promised to obtain a profit of more than 25% of the annual income from the foreign government.
    • Receiving specific instructions or requests from a foreign government regarding behavior while in Japan. (Further details will be published on the website as soon as determined.)

Final Round

Ten teams who passed the first round advance to the final round. In the final round the optimization of hyperparameters in deep learning will be assessed. We are considering awarding a certain amount of ABCI points to the individuals who have participated in the final round. 

Posting Method

The participants shall post the source code of HPO module they developed. Once posted, the source code will be automatically executed and HPO module will be assessed. Please see the attached data “readme.md” for how to create the file to post.

Note:  It will take time for the assessment results to be reflected on the leaderboard as it requires execution of the source code. 
Note: If the entire process, including loading the module, takes longer than 5 hours, it will be considered an error and will be disqualified. 

Accuracy Assessment

In the first round, the superiority of optimization accuracy will be assessed using multiple benchmark functions which are often used in BBO (Black-Box Optimization) academic essays. 

  • The superiority of optimization accuracy will be assessed with approximately five multidimensional benchmark functions, including schwefel function.  
  • Hyperparameters that are tuned in each function are five-dimensional and only real numbers and integers are allowed. Categorical variables are not used. 
  • The number of assessments is expected to be approximately 100 attempts in serial processing.
  • For the accuracy released from each benchmark function the scale will be normalized, and the average value will be used. 

The decision flow of participants advancing to the final round

  1. The assessment from the start throughout the end of the first round (temporary assessment) will be assessed using a part of benchmark functions that are intended for assessment. The post-competition assessment (final assessment) will be assessed using different benchmark functions. The leaderboard will automatically switch to the final assessment in the first round when the competition is ended, which will determine the rankings of participants based on it. Therefore, the rankings may significantly change between during and after the competition.
    Note: Up to two submission files can be selected, and the file with the highest final evaluation score will be automatically adopted.
  2. If there are teams which have received the same score, the participants who entered the competition earlier will be considered higher ranked. 
  3. The top 10 ranked teams in the final assessment will advance to the final round and they will be notified by our office.
  4. The participants who are advancing to the final round must submit the following:
    • Letter of intent to participate in the final round.
    • Personal ID and other documents required to use the ABCI.
    • Explanatory materials for the source code (Please specify preprocessed and learning areas. 
    • Execution environment (OS version, software and analysis method used)
      Note: If existing or published HPO modules, please submit source code implementing the existing HPO module before improvement and document stating improved points. 
  5. During the acceptance/inspection period the potential prize-winning candidates and their submitted models will become disqualified if they fall under any of the following:
    • Not corresponding to our office when contacted or demanded within a designated period.
    • No intention to participate in the final round. 
    • Not meeting the participation eligibility or rules.
    • Absence of personal ID or other documents required to use the ABCI.
    • Not meeting the requirements to use the ABCI.
    • When our office determines something is inappropriate.
  6. The candidates to advance to the final round will be determined among the individuals who are confirmed by the office. 

Note: Medallions and points will be awarded after the final round has been finished.
Note: If there is a disqualified participant/s among the individuals who have advanced to the final round, the runner-up will not be automatically moved up and the remaining candidates may be able to advance to the final round.
Note: In a situation where a participant has become disqualified or found to have engaged in misconduct, medallions and points may not be awarded based on the SIGNATE’s own discretion. 

Assessment of accuracy in the final round

In the final round the superiority of optimization accuracy through hyperparameter optimization in the actual deep learning as described below is expected to be assessed.  

  • For deep learning models, hyperparameter optimization will be executed for approximately two network models, including WideResNet, and the superiority of recognition accuracy of the final model will be assessed.  
  • The data set to be used are approximately three types, including CIFAR10 or CIFAR100.
  • The number of hyperparameters for tuning is approximately five including the learning rate. Only real numbers and integers are allowed, and categorical variables are not used.
  • The number of assessments is expected to be 100 attempts in serial processing.

Prerequisites for participation

  • Please understand the objectives set forth for the competition and establish a system with practicality in mind.
  • Participation by cheating or ignoring the rules will not be permitted.
  • Slandering others and offensive behaviors regarding public order and morals is forbidden.
  • In a case of a serious offense determined by the office, please note that the eligibility to the competition prizes and membership eligibility may be forfeited. 

System Usage

  • One account per person.
  • The number of team members is up to five individuals and the team can be created until December 31, 2022. (Click here for how to create the team.)
    Note: If the team advances to the final round, the team must have the same members for the final round. 

Information Handling Precautions

  • (During the competition) Sharing the codes with anybody else outside the team is forbidden. However, they can be shared on the forum which can be viewed by all participants.
  • (After the competition) Please see the “Information Disclosure Policy.”

Implementation Method

  • Please develop a module that works and complies with IF specifications set forth by “aiaccel.”
  • You can also improve and develop existing HPO modules that have already been published. However, in that case the accuracy must be better than the existing HPO module to win a prize. Please ensure compliance with and confirm the following if using an existing module. 
    • Follow the license designated by the existing HPO module.
    • Specify the source (data location and links to essays) of the existing HPO module and how you improved which part of the module clearly in the source code explanatory document.
    • Being able to add an open-source license to the improved HPO module and publish it. 

Disclosure policy

As a general rule, in accordance with Article 4, Paragraph 1 of the terms of participation, diclosing any contents such as insights and deliverables transmitted through the information or data provided by our company in relation to this competition is not permitted, however, only after the completion of this competition and for non-commercial purposes, it will be possible to disclose the contents within the score of the table below
Model *1
Public
Analysis results *2
Public
Public : Posting to social media sites, blogs and source repositories, and citing to papers
Restricted : Using in a limited range from research, education to seminars, where many unspecified people cannot access
*1 Execution unit source code and learned models
*2 The insights obtained using the information and data provided, or the solutions including scripts and processed data such as summary statistics

▼ Special Contract on this Contest
In accordance with Article III “Prizes and Rights” of the special regulations established for this contest as described below, participants will either choose to make their own submission data, source code and other relevant items be licensed under an Open Source Initiative approved license (www.opensource.org) or not. This license imposes no restrictions on commercial use for licensed models, source code, and any results attained using said source code. Incidentally, the participants who will not choose this license will not be able to prize winner candidate, even though they can participate in the contest. In regard to posting and/or other publication on any social media site, blog or similar by the participant of information relating to any algorithm developed by said participant or any other materials produced for or related to this contest (including materials used for final submissions and/or final evaluations/inspections), no restrictions apply to posting and/or publication as long as the participant clearly explains that the posted/published items relate to this contest, and absolutely posts the positing and/or publication information (including the link URL to the pages) on the contest forum in order to let all the participants know it.


Terms of Participation in SIGNATE Competition

In order to participate in the Competition, you are required to agree to these Terms, in addition to the Terms of Use of SIGNATE.JP Site (hereinafter referred to as the “Terms of Use”). You should participate in the Competition after reading carefully and agreeing to these Terms. These Terms, the matters that are displayed as “additional matters” that you have agreed to when participating in a Competition, the Terms of Use and other terms and conditions that you have agreed to (hereinafter collectively referred to as “these Terms, etc.”) shall all be binding on the Participant.


Article 1 Definitions

1. For the purpose of these Terms, the following terms shall be defined as follows:

(1) “Site” means the website “SIGNATE (https://signate.jp)” on which the Competitions are posted.
(2) “Competition” means any competition on AI development or data analysis on the Site as held by the Host.
(3) “Host” is the host(s) of the Competition. The Host may be SIGNATE, Inc. (hereinafter referred to as the “Company”) or the Company’s client companies, affiliated companies, schools or organizations, etc. (hereinafter referred to as the “Client(s)”).
(4) “Participant(s)” means the member(s) (which mean “member(s)” defined in the Terms of Use, and the same shall apply hereinafter) who participate in a Competition.
(5) “Submissions” means, collectively, the analysis and prediction results, prediction models and reports, etc. as submitted in the Competition.
(6) “Final Submissions” means the Submissions submitted by a Participant that the Participant has specified as a final submission on the prescribed page in the Site by the time of completion of a Competition.
(7) “Winner Candidate” means the Participant who has received a notice from the Company that he/she is nominated as a winner candidate.
(8) “Submissions for Final Judgment” means the Submissions and other items designated by the Company as submitted by a Winner Candidate pursuant to the instructions of the Company.
(9) “Final Judgment” means the acceptance inspection and judgment, including reproducibility verification, by the Company for the Final Submissions and Submissions for Final Judgment of a Winner Candidate.
(10) “Winner” means the Winner Candidate who is informed by the Company that he/she has won a prize.
(11) “OSS” means software licensed based on OSS License Terms.
(12) “OSS License Terms” means any of the following license terms
(1) Either the GNU General Public License or the GNU Lesser General Public License published by the Free Software Foundation, Inc.;
(2) License terms listed at www.opensource.org/licenses/ or derivatives thereof;
(3) License terms that regard the software as “free software” or “open source software”; and
(4) License terms or agreements similar to the license terms listed in each of the three preceding items which request the user to disclose, distribute or license to a third party, or not to exercise, etc., the licensed software, its derivatives and the intellectual property rights associated therewith in whole or in part.

Article 2 Competition

1. A member who desires to participate in a Competition shall be required to agree to these Terms, etc. and to satisfy the conditions for participation as specified in each such Competition. Any person who is not a member shall not participate in any Competition.
2. Participants shall participate in each Competition in the manner as advised by the Company and shall be obligated to comply with the rules as prescribed in each Competition.
3. Participants may submit the Submissions for the assignment of a Competition during the period of such Competition and submit as many proposals on the method of solving the problem as specified by the Company to the Host by the end of the period of the said Competition.
4. Participants may submit Submissions in the form specified in the Competition and specify those Submissions as a Final Submission on the prescribed page in the Site by the end time specified by the said Competition.
5. Participants’ Final Submissions shall be evaluated in accordance with the evaluation method specified in the Competition and the final ranking shall be determined based on such evaluation.
6. Participants may, as a general rule, check their own evaluation results and the evaluation results of each of the other Participants for Submissions that may be evaluated quantitatively on the Site.
7. Participants shall be solely liable for their own Submissions, including the legality and non-infringement of the Submission.
8. Participants shall not submit any Submissions that have no direct relationship to each Competition.
9. Unless otherwise provided for, Participants may not directly communicate to, consult with, make a request to, solicit or take any other actions with the Host in respect of the matters related to a Competition during the period of the said Competition.
10. Participants shall direct any questions or concerns regarding any Competition to the Company or the third party designated by the Company in accordance with the procedures prescribed by the Company as posted on the Site.
11. Participants shall produce Submissions in compliance with the OSS License Terms related to the OSS when using or incorporating OSS in a Submission. However, Participants shall not use or incorporate OSS for which commercial use is prohibited in a Submission.
12. The Host shall not be obligated to pay any remuneration or other consideration for any act of the Participants in a Competition under any pretext. And the Company shall not be obligated to pay any remuneration or other consideration other than those prescribed in the following Article for any act of the Participants in a Competition.

Article 3 Reward and Vesting of Rights

1. Unless otherwise provided for, any Participant shall satisfy the requirements set forth in the following items in order to be entitled to receive a reward in any Competition that offers a reward:

(1) To be a winner;
(2) To agree to transfer to the Host and the relevant transferee of rights in such Competition all transferable rights, such as copyrights (including the rights as prescribed in Article 27 and Article 28 of the Copyright Act, and the same shall apply hereinafter), rights to obtain patents and know-how, etc. in and to all analysis and prediction results, prediction models, reports, etc., written explanations on algorithms, source code and reproduction method, etc. (although not limited to these), and the Submissions contained in the Final Submissions and Submissions for Final Judgment (hereinafter referred to as the “Rights”) along with a guarantee that the Participant has the authority to transfer such Rights;
(3) To guarantee that any relevant transferee of rights may use the Rights contained in the Final Submissions and Submissions for Final Judgment for its own business and other purpose without any restriction and to agree to their exclusive use of such Rights;
(4) To agree not to exercise moral rights to the Rights against the relevant transferee of rights;
(5) To enter into an agreement for the transfer of the Rights with the relevant eligible transferee of rights, including the guarantee of and agreement to the matters in the preceding three (3) items and other reasonable provisions;
(6) To have the personal identity of such Participant verified by the Company; and
(7) Not to breach any provision of these Terms, etc.

2. Any Winner Candidate shall, after having received a notice from the Company that he/she is nominated as a winner candidate, submit the Submissions for Final Judgment on or before the designated date and communicate the matters requiring confirmation or response in relation to the Final Submissions and the Submissions for Final Judgment to the Company on or before the designated date, in accordance with the instructions of the Company. The Company shall carry out the final judgment based on such matters requiring confirmation or response. If the Company receives no confirmation or response satisfactory to the Company on or before the designated date, the Company may exclude such Winner Candidate from the subject of the final judgment and the Winner Candidate shall not raise any objection thereto.
3. If the Company considers that the Final Submissions or Submissions for Final Judgment need to be amended or modified, or there occur any additional matters requiring confirmation, in the course of the final judgment, any Winner Candidate shall take action or make response in relation to the matters that require amendment or modification, or the matters requiring confirmation, on or before the designated date in accordance with the instructions of the Company. If the Company receives no action or response satisfactory to the Company on or before the designated date, the Company may exclude such Winner Candidate from the final judgment and the Winner Candidate shall not raise any objection thereto.
4. The Company shall determine the Winner through the final judgment and inform the Winner to that effect.

Article 4 Confidentiality

1. Any Participant shall treat any information and data that they receive from the Company in relation to Competitions as well as knowledge and products, etc. obtained using such information and data (including Participant Submissions; hereinafter referred to as the “Company-Provided Information”) as confidential information and shall not disclose the same to any third party and use the same for any purpose other than for such Competition and purpose specified by the Company separately; provided, however, that the confidential information shall not include any information that falls under any of the following items:

(1) Information that is known to the public at the time it was received;
(2) Information that is already possessed by the Participant at the time it was received (only in the case where such Participant may demonstrate such fact by reasonable means);
(3) Information that becomes known to the public without the fault of the Participant after it was received;
(4) Information that is independently developed by the Participant without reference to any information received; or
(5) Information that is rightfully received from any third party having a right to disclose such information without the obligations of confidentiality (only in the case where such Participant may demonstrate such fact by reasonable means).

2. Any Winner shall handle his/her Final Submissions and Submissions for Final Judgment in the same manner as Company-Provided Information after receiving notification that they are a winner. And Participant Submissions other than the Winner’s Final Submissions and Submissions for Final Judgment shall not be included as Company-Provided Information after the Company has determined the Winner notwithstanding the provisions of the preceding paragraph.
3. Any Participant may publish any algorithms that they have developed as well as any other materials they have created in connection with their participation in a Competition (including Final Submissions and Submissions for Final Judgment) after that Competition in accordance with the Competition Information Disclosure Policy (hereinafter, "Information Disclosure Policy") posted on the Site. However, Participants shall observe the following matters when publishing information and may not publish any materials prohibited by the Information Disclosure Policy:

(1) Clarify that the materials were created in connection with participation in the Competition;
(2) Give credit in accordance with the notation method prescribed in the Information Disclosure Policy when the materials to be published contain part of a dataset; and
(3) Clarify the location accessible to all Participants (including but not limited to the Competition forum and other locations designated by the Company) where the materials are published (including links, etc.) if publishing materials outside the Competition forum.

4. Any Participant shall delete or return to the Company the Company-Provided Information (excluding the information prescribed in paragraph 3) immediately after the completion of a Competition.
5. If there is any separate arrangement in relation to the confidential information in a Competition, the provisions of such arrangement shall prevail over the provisions of these Terms.
6. If any dispute occurs between a third party and the Host or the Company due to the breach by any Participant of the provisions of this Article and such other party makes any claim against the Company or the Host, such Participant shall compensate for any damage, loss, expenses (including, but not limited to, attorneys’ fees), lost profits and lost revenues, etc. incurred by the Company and the Host.
7. The provisions of this Article shall survive the termination of the relevant Competition or the Participant’s completion of the procedures for withdrawal from all services provided by the Company, with respect to the Company-Provided Information and the Winner’s Final Submissions and Submissions for Final Judgment for a period of five (5) years thereafter.

Article 5 Prohibited Acts of Participants

1. The Company shall prohibit Participants from engaging in any of the following acts in any Competition:

(1) An act of cracking, cheating, spoofing and other misconduct;
(2) An act of redistributing data files included in the dataset;
(3) An act of directly communicating to, consulting with, making a request to, soliciting or responding to solicitation or other activities to other Participants or the Host (other than the Company) without the involvement of the Company for the purpose of furthering the Participant’s own interests;
(4) Any profitmaking activities using the Competition (including but not limited to solicitation or scouting activities, and use for a third party in educational business, etc.) without the prior approval of the Company in writing or any other manner specified by the Company;
(5) An Act of transferring, offering as collateral or otherwise disposing of the Participants’ standing, or the rights or obligations, as a Participant in any Competition (except with the prior written consent of the Company);
(6) Acts that infringe upon the intellectual property rights, trade secrets or any other rights of third parties; and
(7) Any other act in breach of these Terms, etc.

2. If the Company deems that a Participant has engaged in, or may engage in, any of the prohibited acts as prescribed in the preceding paragraph, the Company may, without prior notice to the Participant, disqualify the Participant from the Competition in which the Participant participates, temporarily suspend the Participant from using some or all of the services provided by the Company, withdraw the Participant’s membership, claim damages from the Participant or take any other measures deemed necessary by the Company.

Article 6 Modification of Terms

1. The Company may modify, add or delete any provisions of these Terms from time to time without the approval of Participants.

December 16, 2022