129

participants

Topic provider

Pervasive Artificial Intelligence Research (PAIR) Labs, National Yang Ming Chiao Tung University (NYCU), Taiwan

The Pervasive AI Research (PAIR) Labs, a group of national research labs funded by the National Science and Technology Council, Taiwan, is commissioned to achieve academic excellence, nurture local AI talents, build international linkage, and develop pragmatic approaches in the areas of applied AI technologies toward service, product, workflow, and supply chain innovation and optimization. PAIR is now a top-tier research lab located at the Guangfu campus of National Yang Ming Chiao Tung University and constituted of 10 distinguished research teams to conduct research in various applied AI areas.

Web Site: pairlabs.ai

LITEON 

LITEON Technology is a world-leading provider of opto-semiconductor, power supply management and key electronic products with global manufacturing facilities. In recent years, with its active deployment in the fields of cloud computing, automotive electronics, 5G, AIoT and optoelectronics, coupled with expansion of new business for smart life, LITEON continues to use its professionalism, rich industrial experience, flexible supply chain management with quick response and diverse worldwide operational centers, has become the best partner of global customers for creating value, innovation, and application of smart technology.

Web Site: https://www.liteon.com/en-us


Intelligent Vision System (iVS) Lab, National Yang Ming Chiao Tung University (NYCU), Taiwan 

The Intelligent Vision System (iVS) Lab at National Yang Ming Chiao Tung University is directed by Professor Jiun-In Guo. We are tackling practical open problems in autonomous driving research, which focuses on intelligent vision processing systems, applications, and SoC exploiting deep learning technology. 

Web Site: http://ivs.ee.nctu.edu.tw/ivs/


2023/12/21 Award Announcement

PAIR-LITEON Competition award has been announced. Here is the link: https://buckets.aidea-web.tw/ACM_MM_ASIA2023_Final_Announcement.pdf

2023/09/20 Rank Announcement

Dear participants: PAIR-LITEON Competition ranking has been announced. Here is the link: http://buckets.aidea-web.tw/ACM_MM_ASIA2023_Final.pdf

2023/08/03 PAIR-LITEON Competition Notification

Dear participants, Thank you for your participation in this competition. The qualification for advancing to the finals in the preliminary round is a minimum of 0.2 mAP. Thank you.

2023/07/28 Competition Notification

Dear participants, Thank you for taking part in our competition. With only 7 days left until the competition ends, if you have not yet uploaded your scores, kindly remember to do so. We will announce the minimum accuracy required to enter the finals on August 2nd. Thank you.

Introduction

Object detection in the computer vision area has been extensively studied and making tremendous progress in recent years using deep learning methods. However, due to the heavy computation required in most deep learning-based algorithms, it is hard to run these models on embedded systems, which have limited computing capabilities. In addition, the existing open datasets for object detection applied in ADAS applications with the 3-D AVM scene usually include pedestrian, vehicles, cyclists, and motorcycle riders in western countries, which is not quite similar to the crowded Asian countries like Taiwan with lots of motorcycle riders speeding on city roads, such that the object detection models training by using the existing open datasets cannot be applied in detecting moving objects in Asian countries like Taiwan.


In this competition, we encourage the participants to design object detection models that can be applied in Asian traffic with lots of fast speeding motorcycles running on city roads along with vehicles and pedestrians. The developed models not only fit for embedded systems but also achieve high accuracy at the same time.

This competition is divided into two stages: qualification and final competition.

  • Qualification competition: all participants submit their answers online. A score is calculated. We will announce the threshold at the beginning of the preliminary round. If participants achieve accuracy above the threshold during the qualification round, they will qualify to participate in the final round.
  • Final competition: the final score will be evaluated on MemryX CIM computing platform [4] for the final score.

The goal is to design a lightweight deep learning model suitable for constrained embedded system design to deal with traffic in Asian countries like Taiwan. We focus on detection accuracy, model size, computational complexity, performance optimization and the deployment on MemryX CIM SDK platform.

MemryX [4] uses a proprietary, configurable native dataflow architecture, along with at-memory computing that sets the bar for Edge AI processing. The system architecture fundamentally eliminates the data movement bottleneck, while supporting future generations (new hardware, new processes/chemistries and new AI models) — all with the same software.

Given the test image dataset, participants are asked to detect objects belonging to the following four classes {pedestrian, vehicle, scooter, bicycle} in each image, including class, bounding box, and confidence.


Reference

[1] COCO API: https://github.com/cocodataset/cocoapi

[2] Average Precision (AP): https://en.wikipedia.org/wiki/Evaluation_measures_(information_retrieval)#Average_precisio n

[3] Intersection over union (IoU): https://en.wikipedia.org/wiki/Jaccard_index

[4]  MemryX SDK: https://memryx.com/technology/

Prize

All the award winner candidates will be invited to submit contest papers to 2023 ACM Multimedia Asia Workshop for review. According to the paper review results, scoring results of the final competition, as well as the completeness of the submitted documents, the final award winners will be decided and announced. In addition, all the award recipients should attend the ACM Multimedia Asia 2023 Grand Challenge PAIR-LITEON Competition Special Session to present their work, otherwise, the award will be canceled.

According to the above-mentioned evaluation, we select three teams for regular awards, and some teams for the special awards. All the awards might be absent according to the conclusion of the review committee. 

Regular Award

  1. Champion:            $USD 1500
  2. 1st Runner-up:      $USD 1000
  3. 3rd-place:              $USD 700

Special Award

  1. Best pedestrian detection award – a award for the highest AP of pedestrian recognition in the final competition: $USD 300;
  2. Best bike detection award – award for the highest AP of bicycle recognition in the final competition: $USD 300;
  3. Best motorbike detection award – award for the highest AP of scooter recognition in the final competition: $USD 300;

Activity time

Schedule for Competition(UTC+8):

DateCompetition Schedule
2023/03/01~2023/04/30             Competition dataset Preparation
2023/04/01~2023/06/14Beta website
2023/06/15Website on-line
2023/06/15Qualification Competition Start
2023/07/01Release Public Testing Dataset for Qualification
2023/08/04Qualification Competition End
2023/08/05Final Competition Start
2023/08/05Release Private Testing Example Data for Final
2023/09/4Final Competition End
2023/09/17Rank Announcement
2023/09/17Release Private Testing Data for Final
2023/12/06                
Award for Special Session

Deadline for Submission:

DateImportant Dates for Participants
2023/08/4 12:00 PM UTCQualification Competition Submission
2023/09/4 12:00 PM UTCFinal Competition Submission
2023/10/5 12:00 PM UTCPaper Submission


Evaluation Criteria

Qualification Competition

The grading rule is based on MSCOCO object detection rule.

  • The mean Average Precision (mAP) is used to evaluate the result.
  • Intersection over union (IoU) threshold is set at 0.5-0.95.

The resulting average precision (AP) of each class will be calculated and the mean AP (mAP) over all classes is evaluated as the key metric.


Final Competition

  • Mandatory Criteria
    • Accuracy of final submission cannot be 5% lower (include) than their submitted model of qualification.
    • The summation of Preprocessing & Postprocessing time of final submission cannot be 50% slower (include) than the inference time of the main model. (evaluated on the host machine)
  • [Host] Accuracy (mIoU)–50%
    The team with the highest accuracy will receive a perfect score of full score (50%) and the teams that fall below one standard deviation of the accuracy will receive a score of zero. The remaining teams will be awarded scores proportionate to the difference in mAP (mean average precision).
  • [Host] Model size (number of parameters * bit width used in storing the parameters) –20%
    The team with the smallest model will get the full score (20%) and the teams that exceed one standard deviation of the model size will receive a score of zero.  The rest teams will get scores directly proportional to the model size difference.
  • [Host] Model Computational Complexity(GOPs/frame)–15%
    The team with the smallest GOP number per frame will get the full score (15%) and the teams that exceed one standard deviation of the model computational complexity will receive a score of zero.  The rest teams will get scores directly proportional to the GOP value difference.
  • [Device] Speed on MemryX CIM SDK platform–15%
    The team with a single model (w/o Preprocessing & Postprocessing) to complete the detection task in the shortest time will get the full score (15%) and the teams that exceed one standard deviation of the speed will receive a score of zero.  The rest teams will get scores directly proportional to the execution time difference.
  • The evaluation procedure will be toward the overall process from reading the private testing dataset in final to completing submission.csv file, including parsing image list, loading images, and any other overhead to conduct the detection through the testing dataset. 


Coordinator Contacts

Po-Chi Hu, pochihu@nycu.edu.tw
Darren JS Chen, Darren.JS.Chen@liteon.com
Jiun-In Guo, jiguo@nycu.edu.tw
Chia-Chi Tsai, cctsai@gs.ncku.edu.tw

Rules

  • Team mergers are not allowed in this competition.
  • Each team can consist of a maximum of 4 team members.
  • The task is open to the public. Individuals, institutions of higher education, research institutes, enterprises, or other organizations can all sign up for the task.
  • A leaderboard will be set up and make available publicly.
  • Multiple submissions are allowed before the deadline and the last one will be used to enter the final qualification consideration.
  • The upload date/time will be used as the tiebreaker.
  • Privately sharing code or data outside of teams is not permitted. It's okay to share code if made available to all participants on the forums.
  • Personnel of IVSLAB team are not allowed to participate in the task.
  • A common honor code should be observed. Any violation will be disqualified.