Notice
January 8, 2021 Revised the rule because there was some ambiguity about what method of labeling is allowed for evaluation image data.
January 14, 2021 Added the rule for the format of submissions in the Inference Speed Division.

January 19, 2021 The Runtime system has been modified so that it is no longer possible to load evaluation images when the ScoringService.get_model method is executed. If an error occurs as a result of executing the source code again after the modification, the submitted file may not be displayed on the leaderboard.
January 26, 2021 Fixed a bug in the Inference Speed Division where the submission file with the fastest inference processing speed would be prioritized in the leaderboard even if it did not meet the accuracy threshold.
January 26, 2021 Rerun some submission files that may not have been using the GPU correctly and updated the evaluation.
April 26, 2021 The video of the award ceremony is now available on YouTube.




Purpose
Advanced Defense Technology Center (ADTeC) of Acquisition, Technology & Logistics Agency (ATLA) has been conducting a research on autonomous driving in off-road conditions (unpaved and rough terrain). Unlike self-driving car on public roads, the digital 3D maps and infrastructure such as V2X (Vehicle to Infrastructures, Vehicle to Vehicle, etc.) are not available for off-road vehicles. Therefore, for off-road autonomous driving, it is important to recognize the movable ground plane and the obstacles with high accuracy and speed.
In this competition, we will create an algorithm to segment the image data of actual off-road driving, etc., and hold a contest to determine the segmentation accuracy and inference speed.

The submitted prediction models in this contest will not be directly introduced into the equipment. The prediction models submitted by the prizewinners will be used for comparison with the prediction models developed by ADTeC, after training the full data sets other than those of this contest. The know-how and other information in the submissions could be used in future research and development by the Ministry of Defense or ATLA.

Division
 There are two divisions: "Segmentation Accuracy" and "Inference Speed". You can participate in either one or both of these divisions.

DivisionSegmentation AccuracyInference Speed
SubjectCreate an algorithm to divide the region corresponding to an object at the pixel level from an image obtained by driving in an off-road environment.
Target of recognition
road, dirt road, other obstacle
Train DataImages obtained from actual driving in an off-road environment (5014 images)
Region images (3007 images) divided and labeled (19 categories) at the pixel level corresponding to the object
Region images (2007 images) divided and labeled (2 categories) at the pixel level corresponding to the object
Test DataImages obtained from actual driving in an off-road environment (640 images)
Images obtained from actual driving in an off-road environment (333 images)
*However, the data is not disclosed
Submit filePrediction result fileSource code of the model itself and the inference part
Evaluation MethodQuantitative evaluation by IoUQuantitative evaluation by inference speed
However, the model size and IoU must meet the threshold
Prize1st: 300,000yen
2nd: 200,000yen
3rd: 100,000yen		1st: 300,000yen
2nd: 200,000yen
3rd: 100,000yen


Subject
For images obtained by mainly off-road driving, you will be asked to partition the images into the multiple segment at the pixel level, such as the three categories of "road", "dirt road" and "other obstacle".
* You can download the python source code to convert the color-coded image files (png) into a submission file (json) at the data page.


Decision for winner
1. In the Segmentation Accuracy division, evaluations up to the last day of the contest (tentative evaluations) are made on a portion of the evaluation dataset, and evaluations after the end of the contest (final evaluations) are made on the rest of the evaluation dataset.
The leaderboard will automatically switch to the final rating at the end of the contest and the final rankings will be determined based on that rating. Therefore, the rankings may change significantly between the end of the contest and during the contest.
* We use different evaluation data for the Inference Speed division than for the Segmentation Accuracy division. However, since the same data set is used for the tentative and final evaluations, the tentative and final evaluation values do not change.
2. In the event of a tie, the participant who applied with the earlier date and time will be given the higher score.
3. The winner of the final rankings will be selected as a candidate for the prize and will be contacted by the office. 
4. Under the condition that no financial burden and no infringement of rights for the office are ensured, the candidates are required to submit the following.
・ The source code of the model (*In the case of the Inference Speed division, the part of the model that excludes the reasoning part submitted at the time of application, such as training and pre-processing)
・ Trained model (*For Segmentation Accuracy division only)
・ Description of the source code (clearly state the pre-processing, training, and inference parts of the code. If data other than the provided data or trained models are used, the source (location of the data, link to the paper, etc.) should also be clearly stated)
・ Implementing environment (OS version, software used and analysis method)
・ Interpretation of the data, improvement, suggestions from modeling, etc.
5. During the repeatability verification, candidates and their submitted models that fall under any of the following conditions will be disqualified from winning a prize.
・ Failure to respond to procedural communications and requests from the office within the specified time frame.
・ Not meeting the conditions and rules of participation.
・ The program does not work.
・ The score output from the trained model does not match the final evaluation score. (* For Segmentation Accuracy division only)
・ Unpredictable for new images.
・ When the office judges it to be unfair.
6. Winners will be selected from those whose reproducibility has been confirmed.

Use of the System
・One account is required per participant. It is not allowed for one person to use multiple accounts or share one account with multiple people.
・If you wish to participate in a team, please read this page and create your own team. (Creation deadline: 1/15/2021, up to 5 people including the team leader)
・Teams are managed independently by each department and must be created by each.
・The maximum number of submissions per day is 5.

Handling of Information
・The sharing of data or source code related to the predictions of this contest with participants who are not in the same team is prohibited.
・However, it is allowed only if the data source code is made available to all participants in the contest's forum.

Use of Data
・You can modify the image data for training (including manual labeling and label rewriting).
・You can re-training the model after adding pseudo-labels to the image data for evaluation, but manual labeling to the image data for evaluation is prohibited.
・The use of APIs (Google Cloud Vision API, etc.) that retrieve from the meta information such as labels from the input images is prohibited.
・You may use open-source (source is clear, anyone can access it, and it is not commercial) learned models and libraries (tensorflow, chainer, keras, etc.) and external data. If you use them, please indicate the source (e.g., links to papers).

Format of Submission (Only applicable to the Inference Speed Division)
・All inference processing, including loading or pre-processing for input data and post-processing for prediction results, should be written in the ScoringService.predict method.
・The ScoringService.predict method should return the prediction results for the image file stored in the path received as argument “input”.
・It is prohibited to perform inference processing in the ScoringService.get_model method.
・It is prohibited to perform inference processing on an image other than the one corresponding to the path received as argument input in ScoringService.predict.

Contestant Attitude
・We ask that all contestants strive to achieve practical, useful approaches with top priority placed on achieving corporate goals, helping to solve social problems, sharing research results with the public, and other overarching goals.

In both divisions, the evaluation function "IoU" is used to evaluate the segmentation accuracy.
The evaluation value takes a value between 0 and 1, with the higher the accuracy, the greater the value.

1) Compute the average of the overlap (IoU) between the predicted and correct (pixel level) regions for each category for an image.
2) Calculate the average of all the images to be evaluated (1)

For the inference speed division, the model size and inference time are evaluated according to the following flow.

　1. If the size of the submitted model exceeds the threshold of 1GB, it will not be considered for evaluation.

　2. Submitted models are run in SIGNATE's unified GPU-powered environment to measure inference time and accuracy.

　3. If it takes more than 60 minutes to complete the process, including from loading the model to outputting the prediction results, an error will occur and it is not eligible for evaluation.

　4. The threshold of segmentation accuracy is set to IoU=0.75, and if the threshold is not cleared, it will not be eligible for evaluation. (The segmentation accuracy and inference speed are displayed, but no ranking is given, and the leaderboard will show them as "-".)

　5. The ranking is determined based on the inference time for the models that cleared the threshold size and segmentation accuracy. (The unit of the inference time displayed on the leaderboard is seconds/image.)


Disclosure policy

As a general rule, in accordance with Article 4, Paragraph 1 of the terms of participation, diclosing any contents such as insights and deliverables transmitted through the information or data provided by our company in relation to this competition is not permitted, however, only after the completion of this competition and for non-commercial purposes, it will be possible to disclose the contents within the score of the table below
Model *1
Public
Analysis results *2
Public
Public : Posting to social media sites, blogs and source repositories, and citing to papers
Restricted : Using in a limited range from research, education to seminars, where many unspecified people cannot access
*1 Execution unit source code and learned models
*2 The insights obtained using the information and data provided, or the solutions including scripts and processed data such as summary statistics

Q. There was an obvious error in the annotation of the training data, how should I handle it?
A. You can modify the data for learning according to the rules, so please delete or modify the data. Even though the data is checked throughout, since it is difficult to detect and correct 100% of the errors, we ask that you analyze the current results as "correct". We appreciate your understanding.


Q. I can't find the submit button or the data download page. How can I participate in this competition?
A. As stated on the overview page, there are two categories in this competition. Please use the following dedicated pages to submit your work or download your data.



Q. Is there a problem in the rules if I use pseudo-labeled data to retrain the model?
A. There is no problem in terms of the rules of this competition. It is allowed to use models that have been re-trained with pseudo-labels on both training and test data.

Q. The rule says that if you use a trained model, please indicate the source, but where should I indicate it?
A. Please specify in the submission when you receive the prize candidate notification.

In order to participate in this Competitions, you must acknowledge and agree to the Competition Participation Terms, hereafter refereed as “these Terms”, in addition to the Terms of Use of SIGNATE.JP Site (hereinafter referred to as the "Terms of Use"). Before participating in this Competition, you must read carefully, and acknowledge and agree to these Terms. If you agree, these Terms, and other terms and conditions that you have agreed to shall be binding on the relevant parties as integral documents.

Article １. Glossary of terms

1. For the purpose of these Terms, the following terms shall be defined as follows:

(1) "Site" means the website "SIGNATE (https://signate.jp)" on which the Competitions are posted.
(2) "Host" is the host of this Competition. Advanced Defense Technology Center of Acquisition, Technology & Logistics Agency. is designated as Host of this Competition.
This Competition is organized by the SIGNATE, Inc. (hereafter referred to as the ”Company"). The Company is an independent contractor of the Host, and organizing this Competition, on behalf of the Host.
(3) "This Competition" means the competition on the accuracy and the speed of visual image segmentation of off-road images on the Site as held by the Host.
(4) "Participant(s)" means the member(s) who participate in this Competition.
(5) "Submissions" means, collectively, prediction results, prediction models and reports, etc. as submitted in this Competition.
(6) "Final Submissions" means the Submissions that are created by a Participant(s) on the prescribed page by the designated deadline in the Site.
(7) "Potential Winners" means the Participants who have received a winning notice from the Company that they are nominated as winners.
(8) "Submissions for Final Review" means the prediction model , learning data and reports, etc. as submitted by Potential Winners pursuant to the instructions of the Company.
(9) "Final Review" means the assessment, and verification of the Final Submissions, including validations and verification, by the Company, and determination of Final Winners.
(10) "Winners" means the Potential Winners who are informed by the Company that they have won a prize.

2. Unless otherwise defined in these Terms, the terms used in these Terms have the same meaning as the Terms of Use.

Article ２. Competition Eligibility

1. A member who desires to participate in this Competition, shall be required to agree to these Terms and to satisfy the conditions for participation as specified in this Competition. Any person who is not a member shall not participate in this Competition.
2. A personnel of Ministry of Defense of Japan, or ATLA, can participate in this Competition, provided that he/she is a member, but he/she are not eligible to receive the monetary rewards.
3. Participants shall participate in this Competition in the manner as described in the Site and shall be obligated to comply with the rules as prescribed in this Competition.
4. Participants may submit the Submissions for the assignment of this Competition during the period of this Competition and submit a proposal on the method of solving the problem by the end of the period of this Competition.
5. Participants may submit the Final Submissions specified in this Competition by the designated deadline by this Competition.
6. The Final Submissions as submitted shall be assessed by the evaluation metrics as specified in this Competition and the final rank order shall be determined based on such evaluation.
7. Any Participant may check the tentative ranking results, based on the quantitative metrics, of all Participants on the Site.
8. Participants shall be liable or otherwise responsible for their own Submissions, including their legality.
9. Participants shall not submit any Submissions that have no direct relationship to this Competition.
10. Unless otherwise provided for, Participants shall not directly communicate to, consult with, make a request to, solicit or take any other actions with the Host, in respect of the matters related to this Competition during the period of this Competition.
11. Any Participant who has uncertainty or questions about this Competition, shall make sure to contact the Company or its designee through the procedures prescribed by the Site.
12. The Host shall not be obligated to pay any remuneration or other consideration for any act of the Participants as prescribed in paragraphs hereof. The Company shall not be obligated to pay any remuneration or other consideration, other than those prescribed in the following Article, for any act of the Participants as prescribed in paragraphs hereof.

Article ３. Reward and Transfer of Rights

1. Unless otherwise provided for, any Participant shall satisfy the following requirements in order to be entitled to receive a reward in this Competition:

(1) To be winners;
(2) To agree to transfer all transferable rights, such as copyrights, patents and know-how, etc. and the prediction results, prediction model, algorithm, source code and documentations for the model verification and validation, etc., and the Submissions contained in the Final Submissions and Submissions for Final Review (including the rights as prescribed in Article 27 and Article 28 of the Copyright Act of Japan and the rights to obtain patents; hereinafter referred to as the "Rights") to the Host of this Competition;
(3) To agree that any relevant transferee of rights exclusively has the right to use the know-how contained in the Final Submissions and Submissions for Final Review for its own business and other purpose without any restriction;
(4) To agree not to exercise moral rights to the Rights against the relevant transferee of rights;
(5) To have the personal identity of such Participant verified by the Company; and
(6) Not to violate any provision of these Terms and the Terms of Use.

2. Any Potential Winners shall, after having received a notice from the Company that they are nominated as potential winners, submit the Submissions for Final Review on or before the designated date and communicate the matters requiring confirmation or response in relation to the Final Submissions and the Submissions for Final Review to the Company on or before the designated date, in accordance with the instructions of the Company. The Company shall carry out the final review based on such matters requiring confirmation or response. If the Company receives no confirmation or response satisfactory to the Company on or before the designated date, the Company may exclude such Potential Winners from the subject of the final review.
3. If the Company considers that the Final Submissions or Submissions for Final Review need to be amended or modified, or there occur any additional matters requiring confirmation, in the course of the final review, any Potential Winners shall take action or make response in relation to the matters that require amendment, etc. or the detailed information on the matters requiring confirmation, on or before the designated date in accordance with the instructions of the Company. If the Company receives no action or response satisfactory to the Company on or before the designated date, the Company may exclude such Potential Winners from the final review.
4. The Company shall determine the Winners through the final review and inform the Winners to that effect.

Article ４. Confidentiality Obligation

1. Participants shall treat any information, data, or such contents as insights and deliverables transmitted through the service where they receive from the Company in relation to this Competition (hereinafter referred to as the "Company-Provided Information") as confidential information, and shall not disclose the same to any third party and use the same for any purpose other than for this Competition and purpose specified by the Company separately; provided, however, that the confidential information shall not include any information that falls under any of the following items:

(1) Information that is known to the public at the time of the disclosure;
(2) Information that is already possessed by the Participant at the time of the disclosure (only in the case where such Participant may demonstrate such fact by reasonable means);
(3) Information that becomes known to the public without the fault of the Participant after the disclosure;
(4) Information that is independently developed by the Participant without reference to any information as disclosed (except for those Submissions of the person eligible for a prize which are evaluated); or
(5) Information that is rightfully disclosed by any third party having a right to do so without the obligations of confidentiality (only in the case where such Participant may demonstrate such fact by reasonable means).
2. Any Winners shall handle their Final Submissions and Submissions for Final Review in the same manner as prescribed in the paragraph 1 hereof.
3. After the completion of this Competition, Participants may published and/or disclose their algorithms, and their documents related this Competition, including the Final Submissions and Submissions for Final Review, provided they shall specify the source that these documents are created by the participation in this Competition. However he/she shall provide the link to the publication and/or the web page in the official forum of this Competition, so that the all Participants can access these sources. The published/disclosed documents containing the image data, he/she shall specify the notice of copyright below,

ATLA, MOD, Japan

4. Except described in the paragraph 3, Participants shall delete or return the Company-Provided Information after the completion of this Competition.
5. If any dispute occurs between the Host or the Company and other third party due to the violation the provisions of this Article by any Participant and such other party makes any claim against the Host or the Company, such Participant shall compensate for any damage, loss, expenses (including, but not limited to, attorneys’ fees), lost profits and lost revenues, etc. incurred by the Host and the Company.
6. The provisions of this Article shall survive the termination of this Competition or the Participant’s completion of the procedures for withdrawal from the service of the Company, with respect to the Company-Provided Information and the Winner’s Final Submissions and Submissions for Final Review for a period of five (5) years thereafter.

Article ５. Prohibited Acts of Participants

1. The Company shall prohibit Participants from engaging in any of the following acts in any Competition:

(1) An act of misconduct, such as cracking, cheating, and spoofing the others;
(2) An act of re-distribution of the image data or annotated data of the dataset of this Competition;
(3) An act of directly communicating to, consulting with, making a request to, soliciting or responding to solicitation or other activities to other Participants or the Host without the involvement of the Company;
(4) Any profitmaking activities using this Competition (including solicitation or scouting activities, and use for a third party in educational business, etc.) without the prior approval of the Company in writing or any other manner specified by the Company;
(5) Transfer, offering as collateral or other disposition of the status as a Participant or the rights or obligations as a Participant (except with the prior written consent of the Company); and
(6) Any other act of violation of the Terms of Use.
2. If the Company deems that a Participant engages in any of the prohibited acts as prescribed in the preceding paragraph, the Company may, without prior notice to the Participant, disqualify the Participant from this Competition in which the Participant participates, temporarily suspend the Participant from using the service of the Company, withdraw the Participant’s membership, claim damages from the Participant or take any other measures deemed necessary by the Company.

Article ６. Change, Discontinuation or Termination of Provision of Services under these Terms

1. The Company may change or temporarily suspend the services provided by the Company under these Terms without prior notice to the members.
2. Upon one (1) month prior notice to the members, the Company may suspend for a long period of time or terminate the services provided by the Company under these Terms.
3. The Company shall not be liable for any results or damage arising from the measures taken by the Company under this Article.

Article ７. Modification of these Terms

1. The Company may modify, add or delete any provisions of these Terms from time to time without the approval of the members.