Evaluation Criteria

The evaluation metrics are based on Salient360! [1] evaluation metrics:

For evaluating the accuracy of predicted saliency maps, we consider the following metrics [2, 3]:

• Area Under Curve (AUC) proposed by Judd et al. (AUC-Judd) [4]
• Normalized Scanpath Saliency (NSS)
• Linear Correlation Coefficient (CC)
• Similarity of Histogram Intersection (SIM)
• Kullback-Leibler Divergence (KLD)

• Qualification Competition

  • We use five metrics to evaluate the prediction results. For each metric, the team with the best prediction will get the full points (20%) and the team with the worst one will get zero. The rest teams will get points directly proportional to the ranking. We will rank the 5 metrics individually, and then the score calculated based on the 5 ranking lists will be used to determine the position in the leaderboard.
    
    $$ Score=\sum_{i} (n-R_{i})\cdot \frac{20}{n-1} $$
    $$ R_{i}: \text {ranking in each metric} $$
    $$ n: \text {number of teams} $$
    
    

  • Besides, during the qualification competition period, each team has to submit a team composition document, including team name, leader, team members, affiliation, and contact information, etc.

• Final Competition
  
  
  • Mandatory Criteria
    The summation of Preprocessing & Postprocessing time of final submission cannot be 50% slower (include) than the inference time of the main model. (evaluated on the host machine)
  • [Host] Estimation accuracy – 40% (8% for each accuracy metric)
    We use five metrics to evaluate the prediction results. For each metric, the team with the best prediction will get the full points (8%) and the team with the worst one will get zero. The rest teams will get points directly proportional to the ranking. We will rank the 5 metrics individually, and then the score calculated based on the 5 ranking lists will be used to determine the position in the leaderboard.
    $$ Score=\sum_{i} (n-R_{i})\cdot \frac{20}{n-1} $$
    $$ R_{i}: \text {ranking in each metric} $$
    $$ n: \text {number of teams} $$
    
    
  • [Host] Model size (number of parameters * bit width used in storing the parameters) – 15%
    The team with the smallest model will get the full points (15%) and the team with the largest one will get zero. The rest teams will get points directly proportional to the model size ranking.
  • [Host] Model Computational Complexity (GOPs/frame) – 15%
    The team with the smallest GOP number per frame will get the full points (15%) and the team with the largest one will get zero. The rest teams will get points directly proportional to the GOP ranking.
  • [Device] Speed on MediaTek Dimensity Series Platform – 30%
    The team with a single model (w/o Preprocessing & Postprocessing) to complete the detection task in the shortest time will get the full points (30%) and the team that takes the longest time will get zero. The rest teams will get points directly proportional to the execution time ranking.
    
    The evaluation procedure will be toward the overall process from reading the private testing dataset to completing the output file.

Reference

[1] Salient360! Challenge: https://salient360.ls2n.fr/

[2] Bylinskii, Zoya, et al. "What do different evaluation metrics tell us about saliency models?." IEEE transactions on pattern analysis and machine intelligence 41.3 (2018): 740-757.

[3] Gutiérrez, Jesús, et al. "Toolbox and dataset for the development of saliency and scanpath models for omnidirectional/360 still images." Signal Processing: Image Communication 69 (2018): 35-42.

[4] Judd, Tilke, et al. "Learning to predict where humans look." 2009 IEEE 12th international conference on computer vision. IEEE, 2009.

Committee

Min-Chun Hu, National Tsing Hua University

Tse-Yu Pan, Natonal Tawian University of Science and Technology

Herman Prawiro, National Tsing Hua University

CM Cheng, MediaTek

Hsien-kai Kuo, MediaTek

Contacts

Email: vae.challenge@gmail.com