January 19, 2021 The Runtime system has been modified so that it is no longer possible to load evaluation images when the ScoringService.get_model method is executed. If an error occurs as a result of executing the source code again after the modification, the submitted file may not be displayed on the leaderboard.
January 26, 2021 Fixed a bug in the Inference Speed Division where the submission file with the fastest inference processing speed would be prioritized in the leaderboard even if it did not meet the accuracy threshold.
January 26, 2021 Rerun some submission files that may not have been using the GPU correctly and updated the evaluation.
April 26, 2021 The video of the award ceremony is now available on YouTube.

Segmentation Accuracy data & submit page
Inference Speed data & submit page

Purpose
Advanced Defense Technology Center (ADTeC) of Acquisition, Technology & Logistics Agency (ATLA) has been conducting a research on autonomous driving in off-road conditions (unpaved and rough terrain). Unlike self-driving car on public roads, the digital 3D maps and infrastructure such as V2X (Vehicle to Infrastructures, Vehicle to Vehicle, etc.) are not available for off-road vehicles. Therefore, for off-road autonomous driving, it is important to recognize the movable ground plane and the obstacles with high accuracy and speed.
In this competition, we will create an algorithm to segment the image data of actual off-road driving, etc., and hold a contest to determine the segmentation accuracy and inference speed.

The submitted prediction models in this contest will not be directly introduced into the equipment. The prediction models submitted by the prizewinners will be used for comparison with the prediction models developed by ADTeC, after training the full data sets other than those of this contest. The know-how and other information in the submissions could be used in future research and development by the Ministry of Defense or ATLA.

Division
There are two divisions: "Segmentation Accuracy" and "Inference Speed". You can participate in either one or both of these divisions.

Subject
For images obtained by mainly off-road driving, you will be asked to partition the images into the multiple segment at the pixel level, such as the three categories of "road", "dirt road" and "other obstacle".
* You can download the python source code to convert the color-coded image files (png) into a submission file (json) at the data page.

Decision for winner
1. In the Segmentation Accuracy division, evaluations up to the last day of the contest (tentative evaluations) are made on a portion of the evaluation dataset, and evaluations after the end of the contest (final evaluations) are made on the rest of the evaluation dataset.
The leaderboard will automatically switch to the final rating at the end of the contest and the final rankings will be determined based on that rating. Therefore, the rankings may change significantly between the end of the contest and during the contest.
* We use different evaluation data for the Inference Speed division than for the Segmentation Accuracy division. However, since the same data set is used for the tentative and final evaluations, the tentative and final evaluation values do not change.
2. In the event of a tie, the participant who applied with the earlier date and time will be given the higher score.
3. The winner of the final rankings will be selected as a candidate for the prize and will be contacted by the office. 
4. Under the condition that no financial burden and no infringement of rights for the office are ensured, the candidates are required to submit the following.
・ The source code of the model (*In the case of the Inference Speed division, the part of the model that excludes the reasoning part submitted at the time of application, such as training and pre-processing)
・ Trained model (*For Segmentation Accuracy division only)
・ Description of the source code (clearly state the pre-processing, training, and inference parts of the code. If data other than the provided data or trained models are used, the source (location of the data, link to the paper, etc.) should also be clearly stated)
・ Implementing environment (OS version, software used and analysis method)
・ Interpretation of the data, improvement, suggestions from modeling, etc.
5. During the repeatability verification, candidates and their submitted models that fall under any of the following conditions will be disqualified from winning a prize.
・ Failure to respond to procedural communications and requests from the office within the specified time frame.
・ Not meeting the conditions and rules of participation.
・ The program does not work.
・ The score output from the trained model does not match the final evaluation score. (* For Segmentation Accuracy division only)
・ Unpredictable for new images.
・ When the office judges it to be unfair.
6. Winners will be selected from those whose reproducibility has been confirmed.

Use of the System
・One account is required per participant. It is not allowed for one person to use multiple accounts or share one account with multiple people.
・If you wish to participate in a team, please read this page and create your own team. (Creation deadline: 1/15/2021, up to 5 people including the team leader)
・Teams are managed independently by each department and must be created by each.
・The maximum number of submissions per day is 5.

Handling of Information
・The sharing of data or source code related to the predictions of this contest with participants who are not in the same team is prohibited.
・However, it is allowed only if the data source code is made available to all participants in the contest's forum.

Use of Data
・You can modify the image data for training (including manual labeling and label rewriting).
・You can re-training the model after adding pseudo-labels to the image data for evaluation, but manual labeling to the image data for evaluation is prohibited.
・The use of APIs (Google Cloud Vision API, etc.) that retrieve from the meta information such as labels from the input images is prohibited.
・You may use open-source (source is clear, anyone can access it, and it is not commercial) learned models and libraries (tensorflow, chainer, keras, etc.) and external data. If you use them, please indicate the source (e.g., links to papers).

Format of Submission (Only applicable to the Inference Speed Division)
・All inference processing, including loading or pre-processing for input data and post-processing for prediction results, should be written in the ScoringService.predict method.
・The ScoringService.predict method should return the prediction results for the image file stored in the path received as argument “input”.
・It is prohibited to perform inference processing in the ScoringService.get_model method.
・It is prohibited to perform inference processing on an image other than the one corresponding to the path received as argument input in ScoringService.predict.

Contestant Attitude
・We ask that all contestants strive to achieve practical, useful approaches with top priority placed on achieving corporate goals, helping to solve social problems, sharing research results with the public, and other overarching goals.

In both divisions, the evaluation function "IoU" is used to evaluate the segmentation accuracy.
The evaluation value takes a value between 0 and 1, with the higher the accuracy, the greater the value.

1) Compute the average of the overlap (IoU) between the predicted and correct (pixel level) regions for each category for an image.
2) Calculate the average of all the images to be evaluated (1)

For the inference speed division, the model size and inference time are evaluated according to the following flow.

　1. If the size of the submitted model exceeds the threshold of 1GB, it will not be considered for evaluation.

　2. Submitted models are run in SIGNATE's unified GPU-powered environment to measure inference time and accuracy.

　3. If it takes more than 60 minutes to complete the process, including from loading the model to outputting the prediction results, an error will occur and it is not eligible for evaluation.

　4. The threshold of segmentation accuracy is set to IoU=0.75, and if the threshold is not cleared, it will not be eligible for evaluation. (The segmentation accuracy and inference speed are displayed, but no ranking is given, and the leaderboard will show them as "-".)

　5. The ranking is determined based on the inference time for the models that cleared the threshold size and segmentation accuracy. (The unit of the inference time displayed on the leaderboard is seconds/image.)

Q. There was an obvious error in the annotation of the training data, how should I handle it?
A. You can modify the data for learning according to the rules, so please delete or modify the data. Even though the data is checked throughout, since it is difficult to detect and correct 100% of the errors, we ask that you analyze the current results as "correct". We appreciate your understanding.

Q. I can't find the submit button or the data download page. How can I participate in this competition?
A. As stated on the overview page, there are two categories in this competition. Please use the following dedicated pages to submit your work or download your data.

Q. Is there a problem in the rules if I use pseudo-labeled data to retrain the model?
A. There is no problem in terms of the rules of this competition. It is allowed to use models that have been re-trained with pseudo-labels on both training and test data.

Q. The rule says that if you use a trained model, please indicate the source, but where should I indicate it?
A. Please specify in the submission when you receive the prize candidate notification.

In order to participate in this Competitions, you must acknowledge and agree to the Competition Participation Terms, hereafter refereed as “these Terms”, in addition to the Terms of Use of SIGNATE.JP Site (hereinafter referred to as the "Terms of Use"). Before participating in this Competition, you must read carefully, and acknowledge and agree to these Terms. If you agree, these Terms, and other terms and conditions that you have agreed to shall be binding on the relevant parties as integral documents.

Article １. Glossary of terms

Article ２. Competition Eligibility

Article ３. Reward and Transfer of Rights

Article ４. Confidentiality Obligation

Article ５. Prohibited Acts of Participants

Article ６. Change, Discontinuation or Termination of Provision of Services under these Terms

Article ７. Modification of these Terms