Notice

1/29 We have changed the evaluation function "MAP@IoU=0.8" to "MAP@IoU=0.6".

Background

The Japanese government has put forth its "Space Industry Vision 2030" in hopes of expanding applicability beyond the space equipment industry to other relevant industries as well, with the aim of fast-paced doubling of the overall space industry's size by 2030. These endeavors have given rise to Japan's first satellite data platform, Tellus, which was developed for industrial applications. The platform is designed to facilitate easy use of satellite data—which, traditionally, has been largely inaccessible to most—by private-sector companies, universities, research institutions and other organizations, and even individuals. Tellus supports the creation of a wide variety of different business types in multiple fields that make use of more free and open space-based data.

The Tellus Satellite Challenge is a data analysis contest, designed to achieve understandable visual representations of satellite data usage examples, discover exceptional human resources in the area of analysis, promote greater awareness and understanding regarding satellite data types and formats, and achieve other such ends with the goal of promoting widespread utilization of the Tellus platform. Followed by the 1st Tellus Challenge under the theme of detection of landslides from synthetic-aperture radar (SAR) images, we are happy to present you a challenge of ship detection and classification from optical satellite images based on sea surface.

As an island nation surrounded by water, Japan is often required to monitor ship activities on the sea area comprehensively to ensure the safety and security related not only to industrial activities such as water transportation and fishery but also to national defence.
In this competition, you are required to create a model that detects ships on sea surface, identifies if they are moving, and classifies their ship type based on the satellite imagery data with the characteristics of high-resolution and wide coverage.
Based on the insights and findings obtained through the challenge, it is expected to be able to solve more complex problems such as classifying wide varieties of ship types and maritime flags and satisfy various social needs.

This competition uses data obtained by ASNARO-1 - one of the ASNARO (Advanced Satellite with New system Architecture for Observation) satellites equipped with a high-resolution optical imaging sensor that can deliver imagery at a ground resolution of under 0.5m, which is funded by METI Government of Japan and under development by NEC.

Task Description

In this competition, you are required to locate ships in images and classify them into 3 categories {ship_moving, ship_not_moving, barge}.

Areas: Odaiba / Kawasaki / Yokosuka / Osaka / Kobe / Fukuoka
Number of train images: 20
Number of test images: 21
Resolution: 50cm per 1px
Class labels: ship_moving / ship_not_moving / barge

※ Please note the train set includes several unknown labeled annotations, which you don't have to deal with.

The position of the object is assigned in bounding box = (x1, y1, x2, y2). One or more bounding boxes are defined for each image by establishing the upper-left corner of the image as the origin (coordinates 0,0) and specifying four factors: its upper-left coordinates (x1, y1), and the lower-right coordinates (x2, y2) of the bounding box(es). In addition, the target to be identified is targeted only on the water (not including onshore), and the whole of the ship or barge is displayed.

Evaluation Function
・The evaluation function MAP @ IoU = 0.6 will be used.
・An evaluation value between 0 and 1 will be assigned. The higher the accuracy, the higher the value will be.

Details
1.For test images, if the IoU for the overlap of ground truth and prediction areas is 0.6 or greater, area detection is considered to have been successful.
2.For test images, average precision (AP) will calculated in each prediction category for objects in each detection area.
3.For each test image, mean average precision (MAP) calculation will be carried out for all eligible prediction categories based on step 2 results.
4.MAP, as calculated in step 3, will be calculated for all test images.

Scoring and Ranking of Contestants

1. Provisional contestant scorings will be determined based on evaluation of select evaluation-use data sets and carried out up until the contest's final day. Final scorings will be determined after the contest has ended, using the remaining evaluation-use data sets. (The leaderboard display will automatically change from running scores to final evaluation scores once the contest has ended, and final contestant rankings will be shown at this time. There may be large differences between in-contest running results and post-contest final rankings.)

2. Regarding evaluation-based rankings, the contestant who applied first for participation in the contest will be ranked higher in the case of an identical (tied) score.

3. The following are required to carry out contestant ranking (for winners only):
    ・Prediction model source code
    ・Learning-complete model
    ・Procedure description document enabling reproduction of the prediction process (must clearly explain preprocessing, learning, and prediction processes)
    ・Computer/equipment operating environment information, including OS version, software utilized and analysis methods
    ・Random number seed (if Random Forest or similar was used to generate random numbers)
    ・Degree of prediction model contribution for all explanatory variables (when using a method that enables calculation of degree of prediction)
    ・Insights gained through data interpretation, manipulation and modeling

4. During the reproducibility evaluation period, the contestant will be disqualified from award eligibility in any of the following cases:
    ・The contestant does not respond within the specified timeframe to contact from, or a request/demand from, the contest administrative office regarding contest-related procedures
    ・The contestant fails to adhere to participant requirements, rules, etc.
    ・The model's prediction results cannot be reproduced
    ・The contestant's approach cannot be used for prediction with new images

Contestant Attitude

・We ask that all contestants strive to achieve practical, useful approaches with top priority placed on achieving corporate goals, helping to solve social problems, sharing research results with the public, and other overarching goals.

System Usage

・Each person is limited to one account only. However, team-based participation is allowed as long as the one-account rule is followed. When participating as a team rather than as an individual, the following procedures are required:
       1. Registration with SIGNATE for all team members
       2. Submission, via contact form, of the team representative's e-mail address as well as the e-mail addresses of all other team members

Handling of Contest-related Information

・Sharing of prediction-related data and source codes with other contestants is prohibited.

Data Usage

・Submission of manually labeled result data is prohibited.
・Rewriting of learning data labels and then using said data for model learning is prohibited. However, automated data augmentation via learning-use image data processing for use as learning data is allowed.
・Model learning using any data aside from that distributed to contestants is prohibited.
・Use of Google Cloud Vision API and other such APIs is prohibited. However, use of open-source learning-complete models, libraries and similar is allowed (source code publicly open and available to anyone, and not designed for commercial use; e.g., Tensorflow, Chainer, Keras, etc.). When using a learning-complete model, clearly cite the source using a hyperlink to relevant documentation or similar. 

Implementation Method

・Learning tools for models are limited to open-source, free tools (Python, R, etc.).
・Divide source code in three parts as shown below (preprocessing, learning and prediction), and carry out implementation for each.
        1. Preprocess
            Module that reads supplied data, carries out preprocessing, and outputs each file in a format that can be input into the model. Divide into files for learning and files for evaluation ("get_train_data" and "get_test_data" or similar), and define a function for preprocessing. 
        2. Learning
            Module that reads files created in step 1 to enable model learning. Define a function that can output the learning-complete model, feature quantity importance level and cross validation assessment results.
        3. Prediction
            Module that reads the test data created in step 1 and model created in step 2 and outputs prediction results in file format.

Contest sponsor:

Japanese Ministry of Economy, Trade and Industry (METI)

Support:

SAKURA Internet Inc.

【Additional Notes】
Use of ASNARO-1 data:
When using ASNARO-1 data, contestants must understand and adhere to the following.
(1) Do not use ASNARO-1 data provided by us outside of the competition.
(2) We do not guarantee the quality of ASNARO-1 data provided in the competition and aren’t liable for any errors or omissions in that data or content or for any damages or loss you might suffer in connection with it.
【Matters to be updated】
(The provisions of paragraph 1 of Article 3 [Reward and Vesting of Rights] of the Terms below shall be amended as follows [including the surviving provisions].):
1. Unless otherwise provided for, any Participant shall satisfy the following requirements in order to be entitled to receive a reward in any Competition that offers a reward:
(1) To be a winner;
(2) To grant to the Host and its supporting companies (hereinafter referred to as the “Licensees”) a non-exclusive license without restriction as to term and territory and free of charge to use all rights necessary for modifying the Final Submissions and Submissions for Final Judgment for commercial or non-commercial purposes;
(3) To grant to the Licensees a non-exclusive license without restriction as to term and territory and free of charge to use all rights necessary for publishing and use, the Final Submissions and Submissions for Final Judgment and the Final Submissions and Submissions for Final Judgment as modified on the satellite data platform “Tellus” to be started to be provided in future (hereinafter referred to as “Tellus”) for commercial or non-commercial purposes, and for granting a sublicense to any contractor for the operation of Tellus and user of Tellus (together with the rights as prescribed in the preceding item, hereinafter collectively referred to as the “Rights”);
(4) To agree not to exercise moral rights to the Rights against the Licensees and their sub-licensees;
(5) To enter into agreements for the grant of license of the Rights with the Licensees including the consent to the preceding three (3) items and any other reasonable provisions;
(6) If the Host or any supporting company desires to receive an exclusive license or transfer of the rights contained in the Final Submissions and Submissions for Final Judgment, to accept the consultation about the agreement on such license or transfer;
(7) To have the personal identity of such Participant verified by the Company; and
(8) Not to breach any provision of these Terms and the Terms of Use.

(The provisions of Article 4 [Confidentiality] of the Terms below shall be amended as follows [including the surviving provisions].):
1. Any Participant shall treat any information and data that they receive from the Company in relation to each Competition (hereinafter referred to as the “Company-Provided Information”) as confidential information and shall not disclose the same to any third party and use the same for any purpose other than for such Competition and purpose specified by the Company separately; provided, however, that the confidential information shall not include any information that falls under any of the following items:
(1) Information that is known to the public at the time of the disclosure;
(2) Information that is already possessed by the Participant at the time of the disclosure (only in the case where such Participant may demonstrate such fact by reasonable means);
(3) Information that becomes known to the public without the fault of the Participant after the disclosure;
(4) Information that is independently developed by the Participant without reference to any information as disclosed; or
(5) Information that is rightfully disclosed by any third party having a right to do so without the obligations of confidentiality (only in the case where such Participant may demonstrate such fact by reasonable means).
2. Any Participant shall delete or return to the Company the Company-Provided Information immediately after the completion of each Competition.
3. If there is any separate arrangement in relation to the confidential information in each Competition, the provisions of such arrangement shall prevail over the provisions of these Terms.
4. If any dispute occurs between the Host or other third party and the Company due to the breach by any Participant of the provisions of this Article and such other party makes any claim against the Company, such Participant shall compensate for any damage, loss, expenses (including, but not limited to, attorneys’ fees), lost profits and lost revenues, etc. incurred by the Company.
5. The provisions of this Article shall survive the termination of the relevant Competition or the Participant’s completion of the procedures for withdrawal from the service of the Company, with respect to the Company-Provided Information and the Winner’s Final Submissions and Submissions for Final Judgment for a period of five (5) years thereafter.