Evaluation Criteria

Qualification Competition

The grading criteria is divided into two parts. One for the object detection score and the other for semantic segmentation score. Each account for 50％ points in the qualification competition. The evaluation metric for these two parts is listed below.

The evaluation metric for object detection is based on MSCOCO object detection rule.

• The mean Average Precision (mAP) is used to evaluate the result.
• Intersection over union (IoU) threshold is set at 0.5.

The resulting average precision (AP) of each class will be calculated and the mean AP (mAP) over all classes is evaluated as the key metric.

The evaluation metric for semantic segmentation is based on the Cityscapes [3] Pixel-Level Semantic Labeling Task.

• The standard Jaccard Index (commonly known as the PASCAL VOC intersection-over-union metric IoU = TP ⁄ (TP+FP+FN) is used to evaluate the semantic segmentation results.
• TP, FP, and FN are the numbers of true positive, false positive, and false negative pixels, respectively.
• Pixels labeled as void do not contribute to the score.

The IoU compares the prediction region with the ground truth region for a class and quantifies this based on the area of overlap between both regions. The IoU is calculated for each semantic class in an image and the mean of all class IoU scores makes up the mean Intersection over Union (mIoU) score.

The total score for qualification competition is listed below.

• Accuracy (mIoU)–50％
  • The team with the highest accuracy will get the full score (50％) and the team with the lowest one will get zero. The rest teams will get scores directly proportional to the mIoU difference.
• Accuracy(mAP)–50％
  • The team with the highest accuracy will get the full score (50％) and the team with the lowest one will get zero. The rest teams will get scores directly proportional to the mAP difference.
    

Final Competition

• Mandatory Criteria
• Accuracy of final submission cannot be 5％ lower (include) than their submitted model of qualification.
• The summation of Preprocessing & Postprocessing time of final submission cannot be 50％ slower (include) than the inference time of the main model. (Evaluated on the host machine)

• [Host] Accuracy (mIoU)–25％
  The team with the highest accuracy will get the full score (25％) and the team with the lowest one will get zero. The rest teams will get scores directly proportional to the mIoU difference.
• [Host] Accuracy (mAP)–25％
  The team with the highest accuracy will get the full score (25％) and the team with the lowest one will get zero. The rest teams will get scores directly proportional to the mAP difference.
• [Host] Model Computational Complexity(GOPs/frame)–12.5％
  The team with the smallest GOP number per frame will get the full score (12.5％) and the team with the largest one will get zero. The rest teams will get scores directly proportional to the GOP value difference.
• [Host] Model size (number of parameters * bit width used in storing the parameters)–12.5％
  The team with the smallest model will get the full score (12.5％) and the team with the largest one will get zero. The rest teams will get scores directly proportional to the model size difference.
• [Device] Power consumption (average current computation on MediaTek Dimensity Series)–12.5％
  
• Measured by android battery fuel gauge on MediaTek’s Dimensity Series Platform [2]. The “BATTERY_PROPERTY_CURRENT_AVERAGE” mode is used in the evaluation.
• The team with a single model (w/o Preprocessing & Postprocessing) to complete the semantic segmentation task with the lowest power consumption will get the full score (12.5％) and the team with the largest one will get zero. The rest teams will get scores directly proportional to the average current computation difference.
• [Device] Speed on MediaTek Dimensity 9200 Series Platform–12.5％
  The team with a single model (w/o Preprocessing & Postprocessing) to complete the semantic segmentation task in the shortest time will get the full score (12.5％) and the team that takes the longest time will get zero score. The rest teams will get scores directly proportional to the execution time difference.
  The evaluation procedure will be toward the overall process from reading the private testing dataset in final to completing submission.csv file, including parsing image list, loading images, and any other overhead to conduct the semantic segmentation through the testing dataset.

Final Competition

The finalists have to hand in a package that includes SavedModel (should be compatible w/ freeze_graph.py@tensorflow_v2.0.0~v2.8.0) and inference script. We will deploy tensorflow model to MediaTek’s Dimensity Series platform and grade the final score by running the model. 

A technical report is required to reveal the model structure, complexity, and execution efficiency, etc.

Submission File

Upload the zip file naming submission.zip containing the following files:

1. Tensorflow Inference Package
   • An official Docker Image will be released by organizer
   • Tensorflow version are restricted to the following version.
     • Tensorflow v2.0.0~v2.8.0
     • Other versions or other frameworks are not allowed.
   • The following files must exist in the submitted inference package
     • Tensorflow SaveModel
       (refer to TF saved_model.md@tensorflow_v2.0.0~v2.8.0 for more detail)
     • run_model.py [image_list.txt] [path of output results]
       It runs your model to detect objects in test images listed in the image_list.txt and creates submission folder in the [path of output results]. The submission folder format is identical to the qualification competition.
     • Source code of your model
       The directory structure of source code shall be illustrated in README.txt
       
2. techreport.pdf
   • The technical report that describes the model structure, complexity, execution efficiency, implementation techniques, and experiment results, etc.

Coordinator Contacts