YOLO based object detection in Duckietown at night and day

YOLO-based Robust Object Detection in Duckietown

Posted on | by Duckietown Admin

YOLO-based Robust Object Detection in Duckietown

Project Resources
  • Objective: Implementing Advanced and Robust Object Detection for Duckietown.
  • Approach: Enhancing object detection robustness in Duckietown by utilizing a YOLO-based neural network trained on augmented datasets, focusing on safety and performance across varied lighting conditions.
  • Authors: Maximilian Stölzle and Stefan Lionar
Final Result
Code
Authors

Why Robust Object Detection?

Object detection is the ability of a robot to identify a feature in its surroundings that might influence its actions. For example, if an object is laid on the road it might represent an obstacle, i.e., a region of space that the Duckiebot cannot occupy. Robust object detection becomes particularly important when operating in dynamic environmental conditions.

Obstacles can be of various, shape or color and they can be detected through different sensing modalities, for example, through vision or lidar scanning. 

In this project, students use a purely vision-based approach for obstacle detection. Using vision is very tricky because small nuisances such as in-class variations (think of many different type of duckies) or environmental lighting conditions will dramatically affect the outcome. 

Robust object detection refers to the ability of a system to detect objects in a broad spectrum of operating conditions, and to do so reliably. 

Detecting object in Duckietown is therefore important to avoid static and moving obstacles, detect traffic signs and otherwise guarantee safe driving. 

Model Performance Under Normal and Low Lighting Conditions

Robust Object Detection: the challenges

Some of the key challenges associated with vision-based object detection are the following:

Robustness across variable lighting conditions: Ensuring accurate object detection under diverse lighting is complex due to changes in object appearance (check out why in our computer vision classes). The model must handle different lighting scenarios effectively.

Balancing robustness and performance: There’s a trade-off between robustness to lighting variations and achieving high accuracy in standard operating conditions. Prioritizing one may affect the other.

Integration and real-time performance: Integrating the trained neural network (NN) model into the Duckiebot’s system is required for real-time operation, avoid lags associated with transport of images across networks. The model’s complexity therefore must align with the computational resources available. This project was executed on DB19 model Duckiebots, equipped with Raspberry Pi 3B+ and a Coral board.

Data quality and generalization: Ensuring the model generalizes well despite potential biases in the training dataset and transfer learning challenges is crucial. Proper dataset curation and validation are essential.

Project Highlights

Here is the output of their work. Check out the github repository for more details!

Figure 1 shows the appearance of the Duckiebot with Google Coral USB Accelerator to enable real-time and online inference for the Robust Object Detection.
Figure 1. Duckiebot with Google Coral USB Accelerator.
Inference results sample showing annotated objects detected by the fully safety-weighted model in Duckietown.
Figure 2. Sample of inference results of our fully safety-weighted model.
Proposed locational weights for classification loss shown on an overlayed sample image (a) and a heatmap (b).
Figure 3. Proposed locational weights for classification loss.
Schematic of the object detection node showing the workflow between ROS tasks (Python2) and inference tasks (Python3) using separate scripts and temporary files for data exchange.
Figure 4. Schematic of object detection node.
Sample inference on validation set showing predictions by Extra Augmentation, Fully Safety-weighted, and Vanilla models.
Figure 5. Sample inference on validation set.
Sample inferences on new images under normal and low lights, showing detection performance of Vanilla, Extra Augmentation, and Fully Safety-weighted models.
Figure 6. Sample inferences on new images taken under normal and low lights.
Sample inferences under normal and highly illuminated backgrounds, comparing the performance of Vanilla, Extra Augmentation, and Fully Safety-weighted models.
Figure 7. Sample inferences on new images taken in under normally and highly illuminated backgrounds.

Robust Obstacle Detection: Results

Robust Object Detection: Authors

Maximilian Stölzle

Maximilian Stölzle is a former Duckietown student of class Autonomous Mobility on Demand at ETH Zurich, and currently works at MIT as a Visiting Researcher.

Stefan Lionar

Stefan Lionar is a former Duckietown student of class Autonomous Mobility on Demand at ETH Zurich, currently an Industrial PhD student at Sea AI Lab (SAIL), Singapore.

Learn more

Duckietown is a modular, customizable and state-of-the-art platform for creating and disseminating robotics and AI learning experiences.

It is designed to teach, learn, and do research: from exploring the fundamentals of computer science and automation to pushing the boundaries of knowledge.

Get Started
More Projects
Get Robots