Category: Post categories

Kevin Smith, STEM managing director at MassRobotics, Boston, MA, USA

Nurturing future engineers and leaders with Kevin Smith

Posted on | by Federico Tani

Nurturing future engineers and leaders with Kevin Smith

STEM instructor Kevin Smith shares his experience at MassRobotics: providing young learners with core skills to become future engineers and sector leaders.

Boston, MA, USA, April 22nd 2024: STEM Program Manager Kevin Smith shares his work at MassRobotics, a robotics startup incubator in Boston, MA, providing learning experiences to teach technological skills and inspire future engineers and industry leaders.

Quick links

Teaching robotics to nurture future engineers and leaders

We talked with Kevin Smith from MassRobotics to learn more about his teaching activities and the programs he is involved in, such as the Jumpstart fellowship program and the Summer Duckiedrone Academy.

Good morning Kevin! May I ask you to start by introducing yourself?

Hi! My name is Kevin Smith. I have the pleasure of leading the STEM program here at MassRobotics. Our program encompasses creating STEM learning experiences ranging from two hours to six months long. The objective is to help students grow by leveraging our environment of 85-plus robotics startups. This ecosystem provides us with the opportunity to understand where technology is going in the next five to ten years, and we make sure that all of the learning experiences are ingrained with technical expertise.

MassRobotics future engineer 2023
Very nice. We know you use Duckietown for some of your teaching activities, when did you first run into it?

I discovered Duckietown for the first time at the Drone Academy that we were hosting here at MassRobotics, sponsored by Amazon Robotics.

We got a chance to build drones for the first time and really bring some high-level experiences to the students. It’s been a couple of years since I’ve been a part of the Drone Academy and we recently wrapped up the most recent edition where we got to kick off and try some DD24, the newest version of the Duckietown drones and we were so excited about it.

Massrobotics and Duckiedrones future engineers and leaders
How do you use drones to inspire your students?

During the Drone Academy, we had the opportunity to teach students many dynamic skills within one week, building up a lot of core technical skills, but also using the experience as a point to have conversations with students to be able to open their minds to what happens beyond their High School environment.

So, for us, Duckiedrones are a huge technical resource to be able to teach technical skills, raise the level of rigor within the environment that we’re cultivating, and also help students think about what is beyond this phase that they’re in their life so they can become engineers in the future. 

Two future engineers and Duckiedrone DD24
Could you tell us more about the Drone Academy?

The MassRobotics Drone Academy is a one-week summer camp targeted to high-school learners where we leverage our STEM space and partnerships with Amazon, Duckietown, and Brown University to create dynamic learning experiences.

Brown University worked on and cultivated some of the first drones which Duckietown has taken to extreme lengths to make sure that the product is top-tier quality. The camp is offered directly to students at no cost, ensuring that anyone who chooses to participate in these growth experiences can do so.

That is great, you make me want to join the next edition! Are there other programs at MassRobotics you would like to share with us?

At MassRobotics, we host a plethora of different STEM experiences. One of our premiere programs is the Jumpstart program where from January to May young ladies come every Saturday, or at least three out of four Saturdays every month, and work for about six to eight hours with industry experts on CAD [Computer Assisted Design] and CNC [Computer Numerical Control] machines. Essentially they learn engineering skills within a couple of months, and after that they are paid a thousand dollars for their commitment.

But I think the biggest part is the fact that they get to experience real-world internships, and actually some of our interns from the Jumsptart program had the honor of interning at Duckietown too, where they were able to assist in the development and student experience with these drones we’re speaking of!

 

Jumpstart program participants
It sounds awesome! Who is the target of the Jumsptart program?

Jumpstart is a program targeting young ladies in high school, we think juniors are the ideal age, because once they finish their January through May sessions and gain all the technical skills and soft skills, they will be holding up to walk right into an internship which is very difficult to line up for high school students as you can imagine.

It’ll also give them a lot to write on their applications going to college in terms of experience and exposure but also the skills that those colleges are looking for.

We’re trying to cultivate the next generation of leaders, while fostering the creation equitable environments to make sure that these young ladies who step into the internship realm feel comfortable and competent

We're trying to cultivate the next generation of leaders, while fostering the creation of equitable environments to make sure that these young ladies who step into the internship realm feel comfortable and competent.

Kevin Smith
Would you recommend Duckietown to colleagues and students?

Yes, I think Duckietown is a very cool and innovative platform that allows students to explore.  It’s a platform to learn so many technical skills, but also to really start up a lot of conversations, and to digest what’s actually going on behind the scenes, how are these different components engaging and interacting with each other to create autonomous robots.

From the actual Duckietown [the self-driving cars component of Duckietown] I had the pleasure to see and work with, to the Duckiedrones, I believe they bring a lot for the students to explore and engage with.

Two boys assembling a DD24 Duckiedrone

Learn more about Duckietown

Duckietown enables state-of-the-art robotics and AI learning experiences.

It is designed to help teach, learn, and do research: from exploring the fundamentals of computer science and automation to pushing the boundaries of human knowledge.

Get started with Duckietown!
Read other Duckietown stories

Tell us your story

Are you an instructor, learner, researcher or professional with a Duckietown story to tell?

Reach out to us!

Goto-1: Autonomous Navigation using Dijkstra

Goto-1: Planning with Dijkstra

Posted on | by Duckietown Admin

Goto-1: Planning with Dijkstra

Project Resources
  • Objective: Having a Duckiebot drive autonomously from a starting position to any compliant ending position in Duckietown., and stopping there.
  • Approach: Implementing Dijkstra's algorithm for route planning and achieving global localization leveraging the Duckietown appearance specifications and indefinite navigation framework.
  • Authors: Johannes Boghaert
Final Result
Code
Authors

Why planning with Dijkstra?

Planning is one of the three main components, or “blocks”, in a traditional robotics architecture for autonomy: “to see, to plan, to act” (perception, planning, and control). 

The function of the planning “block” is to provide the autonomous decision-making part of the robots’ mind, i.e., the controller, with a reference path to follow.

In the context of Duckietown, planning in applied at different hierarchical levels, from lane following to city navigation. 

This project aimed to build upon the vision-based lane following pipeline, introducing a deterministic planning algorithm to allow one Duckiebot to go to from any location (or tile) on a compliant Duckietown map to a specific target tile (hence the name: Goto-1).

Dijkstra algorithm is a graph-based methodology to determine, in a computationally efficient manner, the shortest path between two nodes in the graph.

Write and test your own Dijkstra algorithm
Goto-1: Autonomous Navigation using Dijkstra
Navigation State Estimation

Autonomous Navigation: the challenges

The new planning capailities of Duckiebots enable autonomous navigation building on pre-existing functionalities, such as “lane following”, “intersection detection and identification”, and “intersection navigation” (we are operating in a scenario with only one agent on the map, so coordination and obstacle avoidance are not central to this progect).

Lane following in Duckietown is mainly vision-based, and as such suffers from the typical challenges of vision in robotics: motion blur, occlusions, sensitivity to environmental lighting conditions and “slow” sampling.

Intersection detection in Duckietown relies on the identification of the red lines on the road layer. Identification of the type of intersection, and relative location of the Duckiebot with respect to it, is instead achieved through the detection and interpretation of fiducial markers, appropriately specified and located on the map. In the case of Duckietown, April Tags (ATs) are used. Each AT, in addition to providing the necessary information regarding the type of intersection (3- or 4-way) and the position of the Duckiebot with respect to the intersection, is mapped to a unique ID in the Duckietown traffic sign database. 

These traffic signs IDs can be used to unamiguosly define the graph of the city roads. Based on this, and leveraging the lane following pipeline state estimator, it is possible to estimate the location (with tile accuracy) of the Duckiebot with respect to a global map reference frame, hence providing the agent sufficient information to know when to stop.

After stopping at an intersection, detecting and identifying it, Duckiebots are ready to choose which direction to go next. This is where the Dijkstra planning algorithm comes into play. After the planner communicates the desired turn to take, the Duckiebot drives through the intersection, before switchng back to lane following behavior after completing the crossing. In Duckietown, we refer to the combined operation of these states as “indefinite navigation”. 

Switching between different “states” of the robot mind (lane following, intersection detection and identification, intersection navigation, and then back to lane following) requires the careful design and implementation of a “finite state machine” which, triggered by specific events, allows for the Duckiebot to transition between these states. 

Integrating a new package within the existing indefinite navigation framework can cause inconsistencies and undefined behaviors, including unreliable AT detection, lane following difficulties, and inconsistent intersection navigation.

Performance evaluation of the GOTO-1 project involved testing three implementations with ten trials each, revealing variability in success rates.

Project Highlights

Here is the output of their work. Check out the GitHub repository for more details!

Diagram showing a Duckietown layout with three key sub-problems highlighted: Localization/Planning (A), Navigation, and Arrival/Stopping (B). The starting position of the Duckiebot is marked as 'A,' and the final destination is marked as 'B.'
Figure 1. Visualization of the three key sub-problems in the GOTO-1 project within a Duckietown configuration.
Flowchart of the GOTO-1 pipeline. Yellow blocks represent components from the existing indefinite navigation framework, orange blocks represent custom GOTO-1 modules, and unmarked inputs are either provided or self-determined.
Figure 2. Overview of the GOTO-1 pipeline, showing the integration of custom blocks and existing framework components.
Three images showing the transformation of a Duckietown map into two input matrices used for path planning: a graphical map definition, a cost matrix, and an input matrix representing allowable turn commands.
Figure 3. Transformation of the Duckietown map configuration into input matrices for path planning.
Image showing the cropping of a mask and summing pixel values to detect midline stripes.
Figure 4. Processing image data to track midline stripes by cropping and summing pixel values.
Charts evaluating the efficiency of the GOTO-1 system and the accuracy of apriltag detections.
Figure 5. Assessment of GOTO-1's efficiency and the accuracy of apriltag (AT) detections.
Terminal window showing the initialization of GOTO-1 with a detailed listing of input and tuning parameters. Subsequent localization and path planning actions are logged after lane following is started.
Figure 6. Initialization of GOTO-1 from the terminal with all parameters listed, followed by localization, path planning, and lane following actions logged.
Terminal window displaying logs of navigation, state estimation for the last mile problem, and the shutdown sequence of the Duckiebot upon reaching its destination.
Figure 7. Logging of navigation, state estimation for the last mile problem, and shutdown of the Duckiebot upon arrival.

Autonomous Navigation: Results

Autonomous Navigation: Authors

Johannes Boghaert

Johannes Boghaert is a former Duckietown student of class Autonomous Mobility on Demand at ETH Zurich, and currently serves as the CEO of Superlab Suisse, Switzerland.

Merlin Hosner

Merlin Hosner is a former Duckietown student and teaching assistant of the Autonomous Mobility on Demand class at ETH Zurich, and currently works as Process Development Engineer at Climeworks, Switzerland. Merlin was a mentor on this project.

Gioele Zardini

Gioele Zardini is a former Duckietown student and teaching assistant of the Autonomous Mobility on Demand class at ETH Zurich, and currently is an Assistant Professor at MIT. Merlin was a mentor on this project.

Learn more

Duckietown is a modular, customizable and state-of-the-art platform for creating and disseminating robotics and AI learning experiences.

It is designed to teach, learn, and do research: from exploring the fundamentals of computer science and automation to pushing the boundaries of knowledge.

Get Started
More Projects
Get Robots
Enhancing Visual Domain Randomization with Real Images for Sim-to-Real Transfer

Enhancing Visual Domain Randomization for Sim2Real Transfer

Posted on | by Duckietown Admin
General Information
  • Enhancing Visual Domain Randomization with Real Images for Sim-to-Real Transfer
  • András Béres and Bálint Gyires-Tóth
  • Budapest University of Technology and Economics, Hungary
  • Béres, András and Gyires-Tóth, Bálint (2023) Enhancing Visual Domain Randomization with Real Images for Sim-to-Real Transfer. INFOCOMMUNICATIONS JOURNAL, 15 (1). pp. 15-25. ISSN 2061-2079
Paper
Institution
Authors

Enhancing Visual Domain Randomization with Real Images for Sim-to-Real Transfer

Image showing the high level overview of the proposed method in the research Enhancing Visual Domain Randomization with Real Images for Sim-to-Real Transfer

One of the classical objections made to machine learning approaches to embeddded autonomy (i.e., to create agents that are deployed on real, physical, robots) is that training requires data, data requires experiement, and experiment are “expensive” (time, money, etc.). 

The natural counter argument to this is to use simulation to create the training data, because simulations are much less expensive than real world experiment; they can be ran continuously, with accellerated time, don’t require supervision, nobody gets tired, etc. 

But, as the experienced roboticist knows, “simulations are doomed to succeed”. This phrase encapsulates the notion that simulations do not contain the same wealth if information as the real world, because they are programmed to be what the programmer wants them to be useful for – they do not capture the complexity of the real world. Eventually things will “work” in simulation, but does that mean they will “work” in the real-world, too?

As Carl Sagan once said: “If you wish to make an applie pie from scratch, you must first reinvent the universe”. 

Domain randomization is an approach to mitigate the limitations of simulations. Instead of training an agent on one set of parameters defining the simulation, many simulations are instead ran, with different values of this parameters. E.g., in the context of a driving simulator like Duckietown, one set of parameters could make the sky purple instead of blue, or the lane markings have slightly different geometric properties, etc. The idea behind this approach is that the agent will be trained on a distribution of datasets that are all slightly different, hopefully making the agent more robust to real world nuisances once deployed in a physical body. 

In this paper,  the authors investigate specifically visual domain randomization. 

Learn about RL, navigation, and other robot autonomy topics at the link below!

Duckietown Educational Resources

Abstract

In order to train reinforcement learning algorithms, a significant amount of experience is required, so it is common practice to train them in simulation, even when they are intended to be applied in the real world. To improve robustness, camerabased agents can be trained using visual domain randomization, which involves changing the visual characteristics of the simulator between training episodes in order to improve their resilience to visual changes in their environment.

In this work, we propose a method, which includes realworld images alongside visual domain randomization in the reinforcement learning training procedure to further enhance the performance after sim-to-real transfer. We train variational autoencoders using both real and simulated frames, and the representations produced by the encoders are then used to train reinforcement learning agents.

The proposed method is evaluated against a variety of baselines, including direct and indirect visual domain randomization, end-to-end reinforcement learning, and supervised and unsupervised state representation learning.

By controlling a differential drive vehicle using only camera images, the method is tested in the Duckietown self-driving car environment. We demonstrate through our experimental results that our method improves learnt representation effectiveness and robustness by achieving the best performance of all tested methods.

Highlights - Enhancing Visual Domain Randomization with Real Images for Sim-to-Real Transfer

Here is a visual tour of the work of the authors. For more details, check out the full paper.

Example frames demonstrating visual domain randomization in the Duckietown Gym self-driving simulation environment, featuring varied visual characteristics to enhance training robustness.
Figure 1. Example frames with visual domain randomization from the Duckietown Gym self-driving simulated environment.
Diagram illustrating the proposed method that combines real images with visually randomized simulated images in unsupervised state representation learning, followed by training a control agent with reinforcement learning using a pretrained encoder network.
Figure 2. High level overview of the proposed method.
A block diagram showing the performance of direct versus invariance regularized domain randomization and supervised versus self-supervised state representation learning methods.
Figure 3. Benchmarked Baseline Methods for Domain Randomization and State Representation Learning.
Diagram showing the preprocessing pipeline.
Figure 4. An illustration of the preprocessing pipeline.
Images from the offline dataset displaying three distinct renderings of the same scene, showcasing varied visual perspectives.
Figure 5. Samples from the Offline Dataset with Three Different Renderings for Each Scene.
Table of evaluation results in the simulator without visual domain randomization, highlighting the proposed method in bold at the bottom row and underlining completion ratios above 70%.
Table 1. Evaluation Results in the Simulator Without Visual Domain Randomization.
Table showing evaluation results in the simulator with visual domain randomization, underlining completion ratios above 70% and highlighting the proposed method in bold at the bottom row.
Table 2. Evaluation Results in the Simulator with Visual Domain Randomization.
Table displaying evaluation results from real-world testing, underlining survival times of 20 or more and featuring the proposed method in bold at the bottom row.
Table 3. Evaluation Results in Real-World Testing.

Conclusion - Enhancing Visual Domain Randomization with Real Images for Sim-to-Real Transfer

Here are the conclusions from the authors of this paper:

“In this work we proposed a novel method for learning effective image representations for reinforcement learning, whose core idea is to train a variational autoencoder using visually randomized images from the simulator, but include images from the real world as well, as if it was just another visually different version of the simulator.

We evaluated the method in the Duckietown self-driving environment on the lane-following task, and our experimental results showed that the image representations of our proposed method improved the performance of the tested reinforcement learning agents both in simulation and reality. This demonstrates the effectiveness and robustness of the representations learned by the proposed method. We benchmarked our method against a wide range of baselines, and the proposed method performed among the best in all cases.

Our experiments showed that using some type of visual domain randomization is necessary for a successful simto- real transfer. Variational autoencoder-based representations tended to outperform supervised representations, and both outperformed representations learned during end-to-end reinforcement learning. Also, for visual domain randomization, when using no real images, invariance regularization-based methods seemed to outperform direct methods. Based on our results, we conclude that including real images in simulation-based reinforcement learning trainings is able to enhance the real world performance of the agent – when using the two-stage approach, proposed in this paper.”

Project Authors

András Béres

András Béres is currently working as a Junior Deep Learning Engineer at Continental, Hungary.

Bálint Gyires-Tóth

Bálint Gyires-Tóth is an associate professor at
Budapest University of Technology and Economics, Hungary.

Learn more

Duckietown is a platform for creating and disseminating robotics and AI learning experiences.

It is modular, customizable and state-of-the-art, and designed to teach, learn, and do research. From exploring the fundamentals of computer science and automation to pushing the boundaries of knowledge, Duckietown evolves with the skills of the user.

Learn about Duckietown
Get Started
Read more Papers
YOLO based object detection in Duckietown at night and day

YOLO-based Robust Object Detection in Duckietown

Posted on | by Duckietown Admin

YOLO-based Robust Object Detection in Duckietown

Project Resources
  • Objective: Implementing Advanced and Robust Object Detection for Duckietown.
  • Approach: Enhancing object detection robustness in Duckietown by utilizing a YOLO-based neural network trained on augmented datasets, focusing on safety and performance across varied lighting conditions.
  • Authors: Maximilian Stölzle and Stefan Lionar
Final Result
Code
Authors

Why Robust Object Detection?

Object detection is the ability of a robot to identify a feature in its surroundings that might influence its actions. For example, if an object is laid on the road it might represent an obstacle, i.e., a region of space that the Duckiebot cannot occupy. Robust object detection becomes particularly important when operating in dynamic environmental conditions.

Obstacles can be of various, shape or color and they can be detected through different sensing modalities, for example, through vision or lidar scanning. 

In this project, students use a purely vision-based approach for obstacle detection. Using vision is very tricky because small nuisances such as in-class variations (think of many different type of duckies) or environmental lighting conditions will dramatically affect the outcome. 

Robust object detection refers to the ability of a system to detect objects in a broad spectrum of operating conditions, and to do so reliably. 

Detecting object in Duckietown is therefore important to avoid static and moving obstacles, detect traffic signs and otherwise guarantee safe driving. 

Model Performance Under Normal and Low Lighting Conditions

Robust Object Detection: the challenges

Some of the key challenges associated with vision-based object detection are the following:

Robustness across variable lighting conditions: Ensuring accurate object detection under diverse lighting is complex due to changes in object appearance (check out why in our computer vision classes). The model must handle different lighting scenarios effectively.

Balancing robustness and performance: There’s a trade-off between robustness to lighting variations and achieving high accuracy in standard operating conditions. Prioritizing one may affect the other.

Integration and real-time performance: Integrating the trained neural network (NN) model into the Duckiebot’s system is required for real-time operation, avoid lags associated with transport of images across networks. The model’s complexity therefore must align with the computational resources available. This project was executed on DB19 model Duckiebots, equipped with Raspberry Pi 3B+ and a Coral board.

Data quality and generalization: Ensuring the model generalizes well despite potential biases in the training dataset and transfer learning challenges is crucial. Proper dataset curation and validation are essential.

Project Highlights

Here is the output of their work. Check out the github repository for more details!

Figure 1 shows the appearance of the Duckiebot with Google Coral USB Accelerator to enable real-time and online inference for the Robust Object Detection.
Figure 1. Duckiebot with Google Coral USB Accelerator.
Inference results sample showing annotated objects detected by the fully safety-weighted model in Duckietown.
Figure 2. Sample of inference results of our fully safety-weighted model.
Proposed locational weights for classification loss shown on an overlayed sample image (a) and a heatmap (b).
Figure 3. Proposed locational weights for classification loss.
Schematic of the object detection node showing the workflow between ROS tasks (Python2) and inference tasks (Python3) using separate scripts and temporary files for data exchange.
Figure 4. Schematic of object detection node.
Sample inference on validation set showing predictions by Extra Augmentation, Fully Safety-weighted, and Vanilla models.
Figure 5. Sample inference on validation set.
Sample inferences on new images under normal and low lights, showing detection performance of Vanilla, Extra Augmentation, and Fully Safety-weighted models.
Figure 6. Sample inferences on new images taken under normal and low lights.
Sample inferences under normal and highly illuminated backgrounds, comparing the performance of Vanilla, Extra Augmentation, and Fully Safety-weighted models.
Figure 7. Sample inferences on new images taken in under normally and highly illuminated backgrounds.

Robust Obstacle Detection: Results

Robust Object Detection: Authors

Maximilian Stölzle

Maximilian Stölzle is a former Duckietown student of class Autonomous Mobility on Demand at ETH Zurich, and currently works at MIT as a Visiting Researcher.

Stefan Lionar

Stefan Lionar is a former Duckietown student of class Autonomous Mobility on Demand at ETH Zurich, currently an Industrial PhD student at Sea AI Lab (SAIL), Singapore.

Learn more

Duckietown is a modular, customizable and state-of-the-art platform for creating and disseminating robotics and AI learning experiences.

It is designed to teach, learn, and do research: from exploring the fundamentals of computer science and automation to pushing the boundaries of knowledge.

Get Started
More Projects
Get Robots
Duckietown parnters with Massrobotics in Boston for the Duckiedrone summer 2024 academy

The 6th Annual Massrobotics Duckiedrone Academy

Posted on | by Jacopo Tani

Boston, MA, USA – Massrobotics, July 2024: instructors and learners gather at MassRobotics in Boston to learn about drone autonomy.

Quick links

The 6th Annual Drone Academy at MassRobotics

High school learners gathered at MassRobotics in Boston to learn about drone autonomy using the latest Duckiedrones, model DD24. 

With the support of Brown University and Amazon Robotics, learners deep-dived for a week in the science and technology of autonomous flight.

Starting from a box of parts, the Duckietown DD24 drone and accompanying pedagogical materials enable a rich set of learning experiences for newcomers to autonomy, as well as for seasoned veterans. 

Learners had the opportunity to practice soldering, electrical connections testing, software initialization for development and operations, actuator setup, sensor calibrations, low-level controller tuning, manual flight, and autonomous hovering. 

This summer academy followed a similar experience at Howard University, Washington DC, that took place in June 2024.

The new Duckiedrone (DD24)
Duckiedrone summer camp 2024 at massrobotics

The Duckiedrone is a DIY, Raspberry Pi-based drone designed to introduce learners to autonomous flight.

Get the Duckiedrone
Duckiedrone DD24

Learn more about Duckietown

The Duckietown platform enables state-of-the-art robotics and AI learning experiences.

It is designed to help teach, learn, and do research: from exploring the fundamentals of computer science and automation to pushing the boundaries of human knowledge.

Get Started with Duckietown!
Request a quote for your class
Leveraging Reward Consistency for Interpretable Feature Discovery in Reinforcement Learning

Reward Consistency for Interpretable Feature Discovery in RL

Posted on | by Duckietown Admin
General Information
  • Leveraging Reward Consistency for Interpretable Feature Discovery in Reinforcement Learning
  • Qisen Yang, Huanqian Wang, Mukun Tong, Wenjie Shi, Gao Huang, and Shiji Song
  • Tsinghua University, China
  • Q. Yang, H. Wang, M. Tong, W. Shi, G. Huang and S. Song, "Leveraging Reward Consistency for Interpretable Feature Discovery in Reinforcement Learning," in IEEE Transactions on Systems, Man, and Cybernetics: Systems, vol. 54, no. 2, pp. 1014-1025, Feb. 2024, doi: 10.1109/TSMC.2023.3312411.
Paper
University
Authors

Leveraging Reward Consistency for Interpretable Feature Discovery in Reinforcement Learning

Interpretable feature discovery RL

What is interpretable feature discovery in reinforcement learning?

To understand this, let’s introduce a few important topics:

Reinforcement Learning (RL): A machine learning approach where an agent gains the ability to make decisions by engaging with an environment to accomplish a specific objective. Interpretable Feature Discovery in RL is an approach that aims to make the decision-making process of RL agents more understandable to humans.

The need for interpretability: In real-world applications, especially in safety-critical domains like self-driving cars, it is crucial to understand why an RL agent makes a certain decision. Interpretability helps:

  • Build trust in the system
  • Debug and improve the model
  • Ensure compliance with regulations and ethical standards
  • Understand fault if accidents arise

Feature discovery: Feature discovery in this context refers to identifying the key artifacts (features) of the environment that the RL agent is focusing on while making decisions. For example, in a self-driving car simulation, relevant features might include the position of other cars, road signs, or lane markings.

Learn about RL, navigation, and other robot autonomy topics at the link below!

Duckietown Educational Resources

Abstract

The black-box nature of deep reinforcement learning (RL) hinders them from real-world applications. Therefore, interpreting and explaining RL agents have been active research topics in recent years. Existing methods for post-hoc explanations usually adopt the action matching principle to enable an easy understanding of vision-based RL agents. In this article, it is argued that the commonly used action matching principle is more like an explanation of deep neural networks (DNNs) than the interpretation of RL agents. 

It may lead to irrelevant or misplaced feature attribution when different DNNs’ outputs lead to the same rewards or different rewards result from the same outputs. Therefore, we propose to consider rewards, the essential objective of RL agents, as the essential objective of interpreting RL agents as well. To ensure reward consistency during interpretable feature discovery, a novel framework (RL interpreting RL, denoted as RL-in-RL) is proposed to solve the gradient disconnection from actions to rewards. 

We verify and evaluate our method on the Atari 2600 games as well as Duckietown, a challenging self-driving car simulator environment. The results show that our method manages to keep reward (or return) consistency and achieves high-quality feature attribution. Further, a series of analytical experiments validate our assumption of the action matching principle’s limitations.

Highlights - Leveraging Reward Consistency for Interpretable Feature Discovery in Reinforcement Learning

Here is a visual tour of the work of the authors. For more details, check out the full paper.

Figure showing the limitations of action matching: In the driving task illustrated in (a), two actions—turning left by 0.12 as shown in (a.1) and turning left by 0.15 as shown in (a.2)—represent the same behavior of "avoiding collision" and receive the same reward. In the scenario depicted in (b), the same actions—shown in (b.1) and (b.2)—result in different rewards depending on the task, but action matching methods would provide the same explanation for both.
Figure 1. Examples of action matching’s limitations.
Diagram of the reward-oriented interpretation method: a DNN with an encoder and decoder learns the mask and attentive state. The pretrained policy takes actions from both primitive and attentive states, with the environment providing corresponding rewards. Reward matching is hindered by disconnected gradient backward in supervised learning.
Figure 2. Architecture of Reward-Oriented Interpretation Method for Reinforcement Learning.
Diagram of interaction process: (a) shows how the pretrained policy (πpre) interacts with the environment during pretraining. (b) depicts the interpretation task where reward matching is treated as an RL problem, with the RL-in-RL policy (π̃) corresponding to the gray box in Fig. 2. The reward (r̃) is provided by both the environment (E) and the pretrained policy (πpre).
Figure 3. Interaction Process in Pretraining and Interpretation Tasks.
𝖠𝗅𝗀𝗈𝗋𝗂𝗍𝗁𝗆 𝟣 𝗈𝗎𝗍𝗅𝗂𝗇𝖾𝗌 𝗍𝗁𝖾 𝖱𝖫-𝗂𝗇-𝖱𝖫 𝗆𝗈𝖽𝖾𝗅, 𝗐𝗁𝖾𝗋𝖾 𝖺 𝗉𝗋𝖾𝗍𝗋𝖺𝗂𝗇𝖾𝖽 𝗉𝗈𝗅𝗂𝖼𝗒 𝜋 𝗉𝗋𝖾 π 𝗉𝗋𝖾 ​ 𝗂𝗇𝗍𝖾𝗋𝖺𝖼𝗍𝗌 𝗐𝗂𝗍𝗁 𝗍𝗁𝖾 𝖾𝗇𝗏𝗂𝗋𝗈𝗇𝗆𝖾𝗇𝗍 𝗍𝗈 𝖻𝖾 𝗂𝗇𝗍𝖾𝗋𝗉𝗋𝖾𝗍𝖾𝖽. 𝖳𝗁𝖾 𝗋𝖾𝗐𝖺𝗋𝖽 𝖿𝗎𝗇𝖼𝗍𝗂𝗈𝗇 𝑅 ( 𝑠 𝑡 , 𝑎 𝑡 ) 𝖱(𝗌 𝗍 ​ ,𝖺 𝗍 ​ ) 𝗂𝗌 𝗉𝗋𝗈𝗏𝗂𝖽𝖾𝖽 𝖻𝗒 𝗍𝗁𝖾 𝖾𝗇𝗏𝗂𝗋𝗈𝗇𝗆𝖾𝗇𝗍 𝐸 𝖤. 𝖳𝗁𝖾 𝖺𝗅𝗀𝗈𝗋𝗂𝗍𝗁𝗆 𝗂𝗇𝗏𝗈𝗅𝗏𝖾𝗌 𝗂𝗇𝗂𝗍𝗂𝖺𝗅𝗂𝗓𝗂𝗇𝗀 𝖺𝗇𝖽 𝗎𝗉𝖽𝖺𝗍𝗂𝗇𝗀 𝗌𝗉𝖾𝖼𝗂𝖿𝗂𝖼 𝖿𝗎𝗇𝖼𝗍𝗂𝗈𝗇𝗌 𝖺𝗇𝖽 𝗉𝖺𝗋𝖺𝗆𝖾𝗍𝖾𝗋𝗌, 𝗂𝗍𝖾𝗋𝖺𝗍𝗂𝗇𝗀 𝗈𝗏𝖾𝗋 𝖾𝗉𝗈𝖼𝗁𝗌 𝗎𝗇𝗍𝗂𝗅 𝖼𝗈𝗇𝗏𝖾𝗋𝗀𝖾𝗇𝖼𝖾. 𝖣𝗎𝗋𝗂𝗇𝗀 𝖾𝖺𝖼𝗁 𝖾𝗉𝗈𝖼𝗁, 𝗍𝗁𝖾 𝗌𝗍𝖺𝗍𝖾 𝗂𝗌 𝗂𝗇𝗂𝗍𝗂𝖺𝗅𝗂𝗓𝖾𝖽, 𝖺𝖼𝗍𝗂𝗈𝗇𝗌 𝖺𝗇𝖽 𝗋𝖾𝗐𝖺𝗋𝖽𝗌 𝖺𝗋𝖾 𝖼𝗈𝗆𝗉𝗎𝗍𝖾𝖽, 𝖺𝗇𝖽 𝗍𝗋𝖺𝗃𝖾𝖼𝗍𝗈𝗋𝗂𝖾𝗌 𝖺𝗋𝖾 𝗌𝖺𝗏𝖾𝖽. 𝖳𝗁𝖾 𝗎𝗉𝖽𝖺𝗍𝖾 𝗌𝗍𝖾𝗉 𝖾𝗆𝗉𝗅𝗈𝗒𝗌 𝗍𝗁𝖾 𝖯𝗋𝗈𝗑𝗂𝗆𝖺𝗅 𝖯𝗈𝗅𝗂𝖼𝗒 𝖮𝗉𝗍𝗂𝗆𝗂𝗓𝖺𝗍𝗂𝗈𝗇 (𝖯𝖯𝖮) 𝖺𝗅𝗀𝗈𝗋𝗂𝗍𝗁𝗆 𝗍𝗈 𝗆𝖺𝗑𝗂𝗆𝗂𝗓𝖾 𝖺 𝗌𝗉𝖾𝖼𝗂𝖿𝗂𝖾𝖽 𝗈𝖻𝗃𝖾𝖼𝗍𝗂𝗏𝖾.
Algorithm 1. RL-in-RL Model.
The attentive state is an overlaid combination of the state and the attentive heatmap of feature attribution. The feature importance ranges from 0 to 1 as the heatmap color changes from blue to red.
Figure 4. Performance of the proposed RL-in-RL model in the Duckietown environment.
Attentive features of RL-in-RL^K with observation length K = 10
Figure 5. Attentive features of RL-in-RL^K with observation length K = 10.
Comparison of interpretation methods on Atari 2600 games. The RL-in-RL model and the supervision-based method are shown with overlaid heatmaps. Perturbation-based and gradient-based methods highlight attention areas on the saliency-overlaid state. Subfigures include (a) Pong, (b) Ms. Pac-Man, and (c) Space Invaders.
Figure 6. Comparisons among different interpretation methods on Atari 2600.
Comparisons between the best-performing action matching method and our reward-oriented RL-in-RL on the Duckietown environment.
Figure 7. Comparisons between the best-performing action matching method and our reward-oriented RL-in-RL on the Duckietown environment.
(a) shows the percent episode return of the pretrained policy under various lane patterns, illustrating the impact of different lines (Fleft_w, Fy, Fright_w) on policy performance. (b) depicts the percent action divergence of the interpretation model under these lane patterns, comparing action consistency with the pretrained policy's actions and showing the effect of the lines on action consistency.
Figure 8. Quantitative analyses averaged across 100 random seeds.
Interpretation results from the action matching method, when the policy is pretrained without the middle yellow line.
Figure 9. Interpretation results from the action matching method, when the policy is pretrained without the middle yellow line.
Attention pattern of RL−in−RLᵃ.
Figure 10. Attention pattern of RL−in−RLᵃ.
Architecture of the encoder–decoder network in RL-in-RL
Figure 11. Architecture of the encoder–decoder network in RL-in-RL.
Additional results of the RL-in-RL model on Atari 2600 games. Attentive states are visualized as in Figure 4. The games displayed are: (a) Enduro, (b) Assault, and (c) Breakout.
Figure 12. More results of the RL-in-RL model on Atari2600.
𝖳𝗐𝗈 𝖺𝖻𝗅𝖺𝗍𝗂𝗈𝗇 𝗌𝗍𝗎𝖽𝗂𝖾𝗌 𝗈𝗇 𝗍𝗁𝖾 𝗁𝗒𝗉𝖾𝗋𝗉𝖺𝗋𝖺𝗆𝖾𝗍𝖾𝗋𝗌 𝗈𝖿 𝗍𝗁𝖾 𝖱𝖫-𝗂𝗇-𝖱𝖫 𝗆𝗈𝖽𝖾𝗅. (𝖺) 𝖲𝗁𝗈𝗐𝗌 𝗍𝗁𝖾 𝖾𝖿𝖿𝖾𝖼𝗍 𝗈𝖿 𝗌𝖾𝗍𝗍𝗂𝗇𝗀 𝛼 = 𝟢.𝟣 α=𝟢.𝟣, 𝗐𝗁𝖾𝗋𝖾 𝛼 α 𝗂𝗌 𝗍𝗁𝖾 𝗐𝖾𝗂𝗀𝗁𝗍 𝗈𝖿 𝗍𝗁𝖾 𝖺𝗎𝗑𝗂𝗅𝗂𝖺𝗋𝗒 𝗍𝖺𝗌𝗄 𝗅𝗈𝗌𝗌 𝖺𝗌 𝖽𝖾𝖿𝗂𝗇𝖾𝖽 𝗂𝗇 𝖤𝗊𝗎𝖺𝗍𝗂𝗈𝗇 (𝟪). (𝖻) 𝖲𝗁𝗈𝗐𝗌 𝗍𝗁𝖾 𝖾𝖿𝖿𝖾𝖼𝗍 𝗈𝖿 𝗌𝖾𝗍𝗍𝗂𝗇𝗀 𝛽 = 𝟢.𝟣 β=𝟢.𝟣, 𝗐𝗁𝖾𝗋𝖾 𝛽 β 𝗂𝗌 𝗎𝗌𝖾𝖽 𝗍𝗈 𝗌𝗉𝖾𝖾𝖽 𝗎𝗉 𝗍𝗁𝖾 𝗍𝗋𝖺𝗂𝗇𝗂𝗇𝗀 𝗉𝗋𝗈𝖼𝖾𝗌𝗌 𝖺𝗌 𝖽𝖾𝗌𝖼𝗋𝗂𝖻𝖾𝖽 𝗂𝗇 𝖠𝗉𝗉𝖾𝗇𝖽𝗂𝗑 𝖠.
Figure 13. Two ablation studies on the hyperparameters.
Comparative results of the RL-in-RL method and the action matching method’s ablation studies on the hyperparameter α.
Figure 14. Comparative results of the RL-in-RL method and the action matching method’s ablation studies on the hyperparameter α.
Visulization of attention shift in the zigzag turn. Three rows correspond to original states, attentive states, and masked states, respectively.
Figure 15. Visulization of attention shift in the zigzag turn.

Conclusion

Here are the conclusions from the authors of this paper:

“In this article, we discussed the limitations of the commonly used assumption, the action matching principle, in RL interpretation methods. It is suggested that action matching cannot truly interpret the agent since it differs from the reward-oriented goal of RL. Hence, the proposed method first leverages reward consistency during feature attribution and models the interpretation problem as a new RL problem, denoted as RL-in-RL. 

Moreover, it provides an adjustable observation length for one-step reward or multistep reward (or return) consistency, depending on the requirements of behavior analyses. Extensive experiments validate the proposed model and support our concerns that action matching would lead to redundant and noncausal attention during interpretation since it is dedicated to exactly identical actions and thus results in a sort of “overfitting.”

 Nevertheless, although RL-in-RL shows superior interpretability and dispenses with redundant attention, further exploration of interpreting RL tasks with explicit causality is left for future work.”

Project Authors

Qisen Yang

Qisen Yang is an Artificial Intelligence PhD Student at Tsinghua University, China.

Huanqian Wang

Huanqian Wang is currently pursuing the B.E. degree in control science and engineering with the Department of Automation, Tsinghua University, Beijing, China.

Mukun Tong

Mukun Tong is currently pursuing the B.E. degree in control science and engineering with the Department of Automation, Tsinghua University,
Beijing, China.

Wenjie Shi

Wenjie Shi received his Ph.D. degree in control science and engineering from the Department of Automation, Institute of Industrial Intelligence and System, Tsinghua University, Beijing, China, in 2022.

Gao Huang

Guang-Bin Huang is in the School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore.

Shiji Song

Shiji Song is currently a Professor with the Department of Automation, Tsinghua University, Beijing, China.

Learn more

Duckietown is a platform for creating and disseminating robotics and AI learning experiences.

It is modular, customizable and state-of-the-art, and designed to teach, learn, and do research. From exploring the fundamentals of computer science and automation to pushing the boundaries of knowledge, Duckietown evolves with the skills of the user.

Learn about Duckietown
Get Started
Read more Papers
Dynamic Obstacle Avoidance

Implementing vision based dynamic obstacle avoidance

Posted on | by Duckietown Admin

Implementing vision based dynamic obstacle avoidance

Project Resources
  • Objective: The objective of this project was to implement dynamic obstacle avoidance for Duckiebots.
  • Approach: Enhancing lane-following algorithms to detect and navigate around static and moving obstacles.
  • Authors: Nikolaj Witting, Fidel Esquivel Estay, Johannes Lienhart, and Paula Wulkop
Final Result
Code
Authors

Why dynamic obstacle avoidance?

Dynamic obstacle avoidance is the process of detecting a region of space that is not navigable (an obstacle), planning a path around it, and executing that plan.

When the obstacle moves, the plan needs to account for the future positions of the object as well, making the process significantly more complicated than passing a static obstacle. 

With this aim, the authors of this project designed and implemented a robust passing algorithm for Duckiebots in Duckietown.

The approach adopted was to develop a new LED-based detection system, modify the typical Duckietown lane following pipeline for planning around the obstacles, and deploying a new controller to execute manoeuvres. 

Learn computer vision with Duckietown

Dynamic obstacle avoidance:
the challenges

Some of the key challenges associated with this project are the following:

Detection Accuracy: The Duckiebot and Duckies detection systems occasionally produce false positives. Light sources from other Duckiebots or shiny objects can interfere with the LED detection, while yellow line segments can be mistaken for Duckies. Improving the reliability of detection under varying lighting conditions is essential.

Lane Following Stability: The Duckiebots sometimes become unstable while overtaking, especially when driving in the left lane. The lane-following system struggles with large lane pose angles or rapid changes in lane position, which can cause the Duckiebot to veer off the road. Enhancing the lane-following algorithm to maintain stability during lane changes is critical.

Velocity Estimation: Estimating the speed of moving Duckiebots accurately is challenging. The current position data obtained from LED detection fluctuates too much to provide a reliable velocity measurement. Developing a more robust method for estimating the velocity of other Duckiebots is needed to ensure safe and efficient overtaking.

Variable Speed Control: Implementing variable speed control during overtaking is problematic due to instability in the lane-following pipeline when speeds are dynamically adjusted. Adjusting speed based on the detected obstacle’s speed without losing lane stability is difficult, necessitating improvements in the lane control model to handle speed changes effectively.

Project Highlights

Here is the output of their work. Check out the github repository for more details!

For Dynamic Obstacle Avoidance, as shown on the left a slowly moving or static Duckiebot is overtaken, while on the right the rear Duckiebot is staying behind the obstacle to avoid oncoming traffic.
Fig. 1. Illustration of the mission.
The cropped input image seen on top, with the thresholded binary image below.
Fig. 2. Cropped Input and Thresholded Binary Image Comparison.
The similarity between the red and white LEDs as percieved by the Duckiebot is illustrated here. On the left are the red LED and on the right is the white LED. In the bottom half, the center color of each LED is drawn through itself and the opposite LED to show the similarity between them.
Fig. 3. The similarity between the red and white LEDs as percieved by the Duckiebot.
Yellow mask of an image with Duckie on lane.
Fig. 4. Yellow mask of an image with Duckie on lane.
Duckiebot driving towards Duckie
Fig. 5. Duckiebot driving towards Duckie.
This image is a flowchart for the demonstration of the decision logic for this project.
Fig. 6. Decision logic for this project.
Lane offset response to linearly changed d_offset parameter. The step response of the d_offset for a single overtaking action was tested.
Fig. 7. Lane offset response to linearly changed d_offset parameter.
This image shows the Overtaking maneuver of the Duckiebot.
Fig. 8. Overtaking maneuver.

Dynamic Obstacle Avoidance: Results

Dynamic Obstacle Avoidance: Authors

Nikolaj Witting

Nikolaj Witting is a former Duckietown student of class Autonomous Mobility on Demand at ETH Zurich, and currently works at Trackman as an Algorithm Developer.

Fidel Esquivel Estay

Fidel Esquivel Estay is a former Duckietown student of class Autonomous Mobility on Demand at ETH Zurich, currently serving as the Co-Founder at UpCircle.

Johannes Lienhart

Johannes Lienhart is a former Duckietown student of class Autonomous Mobility on Demand at ETH Zurich, currently serving as the CTO at Tethys Robotics.

Paula Wulkop

Paula Wulkop is a former Duckietown student of class Autonomous Mobility on Demand at ETH Zurich, where she is currently pursuing her Ph.D.

Learn more

Duckietown is a modular, customizable and state-of-the-art platform for creating and disseminating robotics and AI learning experiences.

It is designed to teach, learn, and do research: from exploring the fundamentals of computer science and automation to pushing the boundaries of knowledge.

Get Started
More Projects
Get Robots
Graph autonomous bots history

Towards Autonomous Driving with Small-Scale Cars: A Survey of Recent Development

Posted on | by Duckietown Admin
General Information
  • Towards Autonomous Driving with Small-Scale Cars: A Survey of Recent Development
  • Dianzhao Li, Paul Auerbach, Ostap Okhrin
  • Technische Universität Dresden, Germany
arXiv
University
Authors

Towards Autonomous Driving with Small-Scale Cars: A Survey of Recent Development

Towards Autonomous Driving with Small-Scale Cars: A Survey of Recent Development

“Towards Autonomous Driving with Small-Scale Cars: A Survey of Recent Development by Dianzhao Li, Paul Auerbach, and Ostap Okhrin is a review that highlights the rapid development of the industry and the important contributions of small-scale car platforms to robot autonomy research.

This survey is a valuable resource for anyone looking to get their bearings in the landscape of autonomous driving research.

We are glad see Duckietown – not only included on the list – but identified as one of the platforms that started a marked increase in the trend of yearly published papers. 

The mission of Duckietown, since we started at as a class at MIT, is to democratize access to the science and technology of robot autonomy. Part of how we intended to achieve this mission was to streamline the way autonomous behaviors for non-trivial robots were developed, tested and deployed in the real world. 

From 2018-2021 we ran several editions of the AI Driving Olympics (AI-DO): an international competition to benchmark the state of the art of embodied AI for safety-critical applications. It was a great experience – not only because it led to the development of the Challenges infrastructure, the Autolab infrastructure, and many agent baselines that catalyze further developments that are now available to the broader community, but even because it was the first time physical robots were brought the world’s leading scientific conference in Machine Learning (NeurIPS: the Neural Information Processing Systems conference – known as NIPS the first time AI-DO was launched). 

All this infrastructure development and testing might have been instrumental in making R&D in autonomous mobile robotics more efficient. Practitioners in the field know-how doing R&D is particularly difficult because final outcomes are the result of the tuple (robot) x (environment) x (task) – so not standardizing everything other than the specific feature under development (i.e., not following the ceteris paribus principle) often leads to apples and pair comparisons, i.e., bad science, which hampers the overall progress of the field.

We are happy to see Duckietown recognized as a contributor to facilitating the making of good science in the field. We beleive that even better and more science will come in the next years, as the students being educated with the Duckietown system start their professional journeys in academia or the workforce.

We are excited to see what the future of robot autonomy will look like, and we will continue doing our best by providing tools, workflows, and comprehensive resources to facilitate the professional development of the next generations of scientists, engineers, and practicioners in the field!

To learn more about Duckietown teaching resources follow the link below.

Starting around 2016, with the introduction of Duckietown, BARC, and Autorally, there was a significant increase in research papers.

Dianzhao Li, Paul Auerbach, Ostap Okhrin​
Duckietown Educational Resources

Abstract

We report the abstract of the authors’ work:

“While engaging with the unfolding revolution in autonomous driving, a challenge presents itself, how can we effectively raise awareness within society about this transformative trend? While full-scale autonomous driving vehicles often come with a hefty price tag, the emergence of small-scale car platforms offers a compelling alternative. 

These platforms not only serve as valuable educational tools for the broader public and young generations but also function as robust research platforms, contributing significantly to the ongoing advancements in autonomous driving technology. 

This survey outlines various small-scale car platforms, categorizing them and detailing the research advancements accomplished through their usage. The conclusion provides proposals for promising future directions in the field.”

Towards Autonomous Driving with Small-Scale Cars: A Survey of Recent Development

Here is a visual tour of the work. For more details, check out the full paper.

Based on published studies each year on Google Scholar with the search terms: robot car, the research on small-scale cars has seen substantial growth in the number of papers over the years. In the early 2000s, projects like s-bot, e-puck, and TurtleBot emerged. Starting around 2016, with the introduction of Duckietown, BARC, and Autorally, there was a significant increase in research papers. This trend continued with the development of projects like DeepRacer, Donkeycar, and F1TENTH. More computationally advanced small-scale cars have been introduced in recent years, such as ART/ATK and XTENTH-CAR.
Fig. 1-A. Illustration of the development and current states of small-scale car platforms, each depicted platform image sourced from its respective paper or website.
amples of small-scale car platforms, categorized into educational platforms and research platforms, including multiple vehicle setups such as ORAC and UDSSC
Fig 1-B. Samples of small-scale car platforms, categorized into educational platforms and research platforms, including multiple vehicle setups such as ORAC and UDSSC
Comparison of two pipelines for the autonomous driving system. An end-to-end system maps raw sensor inputs directly into control commands, whereas a modular system includes multiple subsystems to process the sensor inputs sequentially and output control commands.
Fig. 2. Comparison of two pipelines for the autonomous driving system.
Research in the literature concerning small-scale cars is classified into various benchmark tasks in this survey, ranging from localization and mapping, path planning and following, lane keeping to racing, and cooperative driving.whereas a modular system includes multiple subsystems to process the sensor inputs sequentially and output control commands.whereas a modular system includes multiple subsystems to process the sensor inputs sequentially and output control commands.
Fig. 3. Research in the literature concerning small-scale cars is classified into various benchmark tasks in this survey, ranging from localization and mapping, path planning and following, lane keeping to racing, and cooperative driving.whereas a modular system includes multiple subsystems to process the sensor inputs sequentially and output control commands.
Overall perception and control pipeline. The perception module receives onboard sensor data and produces a predicted occupancy map using a U-Net style generative neural network. The control algorithm receives the robot state, predicted occupancy map, and goal point and generates collision-free trajectories with RRT
Fig. 4. Overall perception and control pipeline. The perception module receives onboard sensor data and produces a predicted occupancy map using a U-Net style generative neural network. The control algorithm receives the robot state, predicted occupancy map, and goal point and generates collision-free trajectories with RRT
The probabilistic dynamics model is trained using a multistep loss that considers how uncertainty propagates. The trajectory tracking controller is used to predict a distribution of closed-loop trajectories. The divergence constrained optimizer is used to find a closed- loop trajectory with low divergence for MuSHR cars
Fig. 5. The probabilistic dynamics model is trained using a multistep loss that considers how uncertainty propagates. The trajectory tracking controller is used to predict a distribution of closed-loop trajectories. The divergence constrained optimizer is used to find a closed- loop trajectory with low divergence for MuSHR cars
Architecture of the proposed adaptive trajectory tracking control system
Fig. 6. Architecture of the proposed adaptive trajectory tracking control system
Different outputs of the multi-step image processing pipeline start from the image input from the camera, proceed to detect road markings according to the color with HSV thresholding, detect marking edges with a canny filter, and finalize with detection marking used to estimate the lateral displacement and angle offset.
Fig. 7. Image Processing Pipeline for Road Marking Detection and Lateral Displacement Estimation
Image patches are encoded using a ViT and predict a class label for each patch. Then, the coarse segmentation output is used for a potential-field based controller.
Fig. 8. Image patches are encoded using a ViT and predict a class label for each patch. Then, the coarse segmentation output is used for a potential-field based controller.
Framework proposes the perception module leverages camera images to produce impact attributes regarding the environment, then the DRL control module utilizes the information to control the agent perform car following behavior.
Fig. 9. Framework for Car Following Behavior Using Perception and DRL Control Modules
The cost map of the track in front of the vehicle is predicted with CNN.
Fig. 10. The cost map of the track in front of the vehicle is predicted with CNN.
End-to-end IL system used for an Autorally racing car.
Fig. 11. End-to-end IL system used for an Autorally racing car.
Overall schematics of the multi-vehicle, mixed reality RL approach. Both virtual and real DeepRacer vehicles exist within the simulation that manages the physics of the virtual cars and emulates collisions in mixed reality.
Fig. 12. Overall schematics of the multi-vehicle, mixed reality RL approach. Both virtual and real DeepRacer vehicles exist within the simulation that manages the physics of the virtual cars and emulates collisions in mixed reality.
Diagram of trajectory planner with C-MOBIL and C-IDM. The ego vehicle plans a trajectory and velocity profile based on its state and the actual state of the neighboring vehicles
Fig. 13. Diagram of trajectory planner with C-MOBIL and C-IDM. The ego vehicle plans a trajectory and velocity profile based on its state and the actual state of the neighboring vehicles

Summary and conclusion

Here is what the authors learned from this survey:

“In this paper, we offer an overview of the current state-of-the- art developments in small-scale autonomous cars. Through a detailed exploration of both past and ongoing research in this domain, we illuminate the promising trajectory for the advancement of autonomous driving technology with small-scale cars. We initially enumerate the presently predominant small-scale car platforms widely employed in academic and educational domains and present the configuration specifics of each platform. Similar to their full-size counterparts, the deployment of hyper-realistic simulation environments is imperative for training, validating, and testing autonomous systems before real-world implementation. To this end, we show the commonly employed universal simulators and platform-specific simulators.

Furthermore, we provide a detailed summary and categorization of tasks accomplished by small-scale cars, encompassing localization and mapping, path planning and following, lane-keeping, car following, overtaking, racing, obstacle avoidance, and more. Within each benchmarked task, we classify the literature into distinct categories: end-toend systems versus modular systems and traditional methods 20 versus ML-based methods. This classification facilitates a nuanced understanding of the diverse approaches adopted in the field. The collective achievements of small-scale cars are thus showcased through this systematic categorization. Since this paper aims to provide a holistic review and guide, we also outline the commonly utilized in various well-known platforms. This information serves as a valuable resource, enabling readers to leverage our survey as a guide for constructing their own platforms or making informed decisions when considering commercial options within the community.

We additionally present future trends concerning small-scale car platforms, focusing on different primary aspects. Firstly, enhancing accessibility across a broad spectrum of enthusiasts: from elementary students and colleagues to researchers, demands the implementation of a comprehensive learning pipeline with diverse entry levels for the platform. Next, to complete the whole ecosystem of the platform, a powerful car body, varying weather conditions, and communications issues should be addressed in a smart city setup. These trends are anticipated to shape the trajectory of the field, contributing significantly to advancements in real-world autonomous driving research.
While we have aimed to achieve maximum comprehensiveness, the expansive nature of this topic makes it challenging to encompass all noteworthy works. Nonetheless, by illustrating the current state of small-scale cars, we hope to offer a distinctive perspective to the community, which would generate more discussions and ideas leading to a brighter future of autonomous driving with small-scale cars.”

Project Authors

Dianzhao Li

Dianzhao Li is a research assistant at the Technische Universität Dresden, Dresden, Germany.

Paul Auerbach

Paul Auerbach is with Barkhausen Institut gGmbH, Dresden, Germany

Ostap Okhrin Technische Universität Dresden portrait

Ostap Okhrin is Chair of Statistics and Econometrics at the Institute of Economics and Transport, School of Transportation, Technische Universitat Dresden in Germany.

Learn more

Duckietown is a platform for creating and disseminating robotics and AI learning experiences.

It is modular, customizable and state-of-the-art, and designed to teach, learn, and do research. From exploring the fundamentals of computer science and automation to pushing the boundaries of knowledge, Duckietown evolves with the skills of the user.

Learn about Duckietown
Get Started
Read more Papers

 

End-to-end Deep RL (DRL) systems: in autonomous driving environments that rely on visual input for vehicle control face potential security risks, including:

  • State Adversarial Perturbations: Subtle alterations to visual input that mislead the DRL agent, causing incorrect decision-making.
  • Reward Tampering: Manipulation of the reward signal to misguide the learning process, leading the agent to adopt unsafe or inefficient policies.

These vulnerabilities can compromise the safety and reliability of self-driving vehicles.

Marek Dlugosz+AGH logo

Teaching robot autonomy at AGH Krakow with Prof. Długosz

Posted on | by Federico Tani

Teaching robot autonomy at AGH Krakow with Prof. Długosz

Prof. Marek Dlugosz tells us what he likes about teaching robot autonomy with Duckietown in his AI and Autonomous Vehicles laboratory in Krakow, Poland.

Krakow, Poland, April 29th 2024
Professor Marek Długosz from the Akademia Górniczo-Hutnicza (AGH) – University of Science and Technology in Krakow, shares his experience teaching robot autonomy with Duckietown.

Quick links

Increasing efficiency and saving time at the robot autonomy lab in AGH Krakow

We interviewed Prof. Marek Długosz form the AGH University of Science and Technology in Krakow, to learn more about his teaching activities and laboratory.

Good morning and thank you for taking the time to be with us! Could you introduce yourself?

My name is Marek Długosz and I am an assistant professor at the AGH University of Science and Technology in Krakow. My work is focused on robotic automatic functions theory, while last year I focused more on robot autonomy and autonomous vehicles. 

DB-21 AVG Dlugosz robot autonomy lab
Thank you. When have you encountered Duckietown for the first time?

I first learned about the Duckietown project while searching the internet for information, then I started to read more about it, and it looked very interesting for a few reasons. First of all, there’s not only a theoretical part, but there’s also the hardware part of this project which are the Duckiebots.

Then, there’s also a lot of lessons, examples and exercises for students to learn more about robotics. That’s one reason why I found it interesting. A second reason this project looked interesting is that it’s not only about one robot, but a fleet of robots, and also there is the speed regulation aspect. One of the main focus of my research is how to manage the speed of autonomous robots or autonomous cars, and I think that Duckietown can be perfect to check my ideas and algorithms rapidly and with ease. It is much easier to run five robots in my laboratory than to run four or five cars in reality!

DB-21 in smartcity, AGH robot autonomy lab
How do you use Duckietown in your activities?

My colleagues and I use Duckiebots during classes and lessons to better explain topics related to theoretical aspects of programming in robotics. What I find very attractive is the possibility for my students to practice. They can actually program a Duckiebot and implement algorithms, such as lane following, adaptive cruise control etc. This is really something that my students and colleagues like very much. Duckietown also has fantastic software, it’s very well organized, thanks to Docker, so there is no risk that some student makes a mistake and breaks one of my robots! Students prepare the Docker container, and once the exercise is complete, they delete this container.

We also use Duckietown in our research as I said earlier, to verify and check our algorithms, and how to manage fleets of robots. One of the special aspects of Duckietown are its smart cities, with its crossroads, signs, traffic lights, etc. For us, this aspect is very interesting. 

One of the latest projects done by my students was a system to localize Duckiebots on a plane using four or more video cameras, and we are about to publish the results of this project. 

Smart city Duckietown at AGH robot autonomy lab
Very interesting thank you. What is the age of the students you are teaching right now, and would you say the students are satisfied with Duckietown?

The age bracket is students between 20 and 23 years old. I’ll just say that very often, at the end of the lessons, students decide to stay and do more exercises, they find it incredibly interesting that Duckietown is not only a simulation, as we know in a simulation you can do anything, but when you start doing things practically, taking the hardware in your hands, and programming these robots, it’s a very, very different thing. 

Very often I have to stay after lessons and try to do more together with my students, to the point I must tell them to come back the day after because it’s getting late.

Also, in addition to regular lessons, in our university there is a Student Scientist Association, the members of which can go to special classes after the end of regular ones, to perform additional experiments and exercises. 

Duckietown is not only a simulation, in a simulation you can do anything, but when you start doing things practically, taking the hardware in your hands, and programming these robots, it’s a very, very different thing. 

Marek Długosz
Smartcity AVG robot autonomy lab
Did you encounter any challenges, problems or difficulties while using Duckietown?

At the beginning, we had some problems with the assembly of the Duckiebot. I even wrote a simple article about this, but we made some improvements, which I  described in this article, and now the Duckiebots work perfectly.

DB-21 AGH Krakow mod
Do you have anything else to add about your projects and experiences?

I hope that I can motivate enough students to participate in the AI-DO competition. I’d like it if some of my students participated in that kind of challenge. I would also like to let my students understand how well organized Duckietown software is, I think there’s an absolutely perfect architecture that helps minimize the risk of errors and mistakes and there’s perfect functionality between containers. We have a lot of students and these robots need to work for everyone, so minimizing the possibility of problems is excellent.

I think that Duckietown can be perfect to check my ideas and algorithms rapidly and with ease. It is much easier to run five robots in my laboratory than to run four or five cars in reality!

Marek Długosz
Would you recommend Duckietown to colleagues and students?

Yes of course, anytime I have the occasion I recommend this project. Ours is the Artificial Intelligence and Autonomous Vehicles laboratory, and it has rapidly become one of the most attractive ones in the University. Very often students show up just to see what we are doing, and every time I show Duckietown’s smart cities, and how robots drive around and stop at traffic lights, crossroads etc. It’s always a lot of fun.

DB-21 top view

Learn more about Duckietown

Duckietown enables state-of-the-art robotics and AI learning experiences.

It is designed to help teach, learn, and do research: from exploring the fundamentals of computer science and automation to pushing the boundaries of human knowledge.

Get started with Duckietown!
Read other Duckietown stories

Tell us your story

Are you an instructor, learner, researcher or professional with a Duckietown story to tell?

Reach out to us!

Ackermann steering Duckiebots and rocket

Development of an Ackermann steering autonomous vehicle

Posted on | by Duckietown Admin

Development of an Ackermann steering autonomous vehicle

Ackermann steering Duckiebots and rocket
Project Resources
  • Objective: The objective of this project was to create an Ackermann steering based self driving car in Duckietown.
  • Approach: Increasing the complexity (and realism) of the hardware chassis, from differential drive to Ackermann steering.
  • Authors: Merlin Hosner and Raphael Fröhlich
Final Result
Thesis
Code
Authors

Why Ackermann steering?

Ackermann steering is a configuration of wheels on a vehicle charachterized by four wheels, two in the back that are powered by a DC motor, and two in the front that steer though commands received by a servo motor. In contrast, differential drive robots have two wheels that are independently powered by two DC-motors, with a passive omnidrectional third wheel that acts as support. 

The dynamics (i.e., the “kind of movement”) of differential drive robots is quite different from real world automobiles, which, e.g., cannot turn on the spot. Ackerman steering achieves more realistic vehicle dynamics at cost: increased hardware complexity and mathematical modeling. But neither of these challenges have stopped talented Duckietown student from designing and implementing an Ackermann steering Duckiebot!

 

(Duckietown trivia: careful Duckietown observers will have noticed that the Duckiebot models historically have been called DB18, DB19, DB21, etc. – every wondered which would have been the DB20?) 

Ackermann steering in Duckietown: the challenges

Ackermann steering introduces more complex mathematical modeling, with respect to differential drive robots, in order to predict future movement hence elaborate pose estimates on the fly. The kinematic modeling of the front steering apparatus is non trivial, and the radius of curvature Ackermann steering robots showcase is very different from differential drive robots.

Differential drive robots are capable of turning on the spot (applying equal and opposite commands to the two wheels), while anyone who has ever tried parallel parking a real car, knows that this is not possible. 

How complex will it be for Ackermann steering robots to navigate Duckietown is the real challenge of this fun project.

The authors start from basic design elements through CAD, iterate through various bills of materials, make prototypes, and program them leveraging the Duckietown software infrastructure to achieve autonomous behaviors in Duckietown. 

Project Highlights

Here is the output of their work. Check out the documents for more details!

Configuration of a differential drive
Configuration of a differential drive
Configuration of a bicycle model
Configuration of a bicycle model
Extension to four wheels
Extension to four wheels
Trace of a four bar rigid link configuration
Trace of a four bar rigid link configuration
Geometric approximation of the Ackermann criteria
Geometric approximation of the Ackermann criteria
Block diagram of a back-calculation anti-windup logic
Block diagram of a back-calculation anti-windup logic
DBv2 placed in a lane with the control inputs in red and the control outputs in white. The steering angle γ is calculated from the angle ω. The linear velocity is represented by v
DBv2 placed in a lane with the control inputs in red and the control outputs in white. The steering angle γ is calculated from the angle ω. The linear velocity is represented by v
Our geometry for implementing the Ackermann criteria
Our geometry for implementing the Ackermann criteria
Ackermann criteria met on the steering of the DBv2
Ackermann criteria met on the steering of the DBv2
Ackermann criteria met on the steering of the DBv2
Ackermann criteria met on the steering of the DBv2
Ackermann criteria met on the steering of the DBv2
Ackermann criteria met on the steering of the DBv2

Ackermann steering: Results

(Turn on the sound for best experience!)

The autonomous behaviors of the Ackermann steering Duckiebot, a.k.a. DB20 or DBv2, shown above are the work of Timothy Scott, a former Duckietown student. 

Ackermann steering Duckiebot: Authors

Merlin Hosner, Duckietown student Ackermann steering project

Merlin Hosner is a former Duckietown student in the Institute for Dynamic Systems and Controls (IDSC) of ETH Zurich (D-MAVT), and currently works at Climeworks as a Process Development Engineer.

Raphael Frohlich Duckietown student Ackermann steering project

Rafael Fröhlich is a former Duckietown student in the Institute for Dynamic Systems and Controls (IDSC) of ETH Zurich (D-MAVT), where he is currently a Research Assistant.

Learn more

Duckietown is a modular, customizable and state-of-the-art platform for creating and disseminating robotics and AI learning experiences.

It is designed to teach, learn, and do research: from exploring the fundamentals of computer science and automation to pushing the boundaries of knowledge.

Get Started
More Projects
Get Robots