Ackermann steering Duckiebots and rocket

Development of an Ackermann steering autonomous vehicle

Posted on June 12, 2024 | by Duckietown Admin

Development of an Ackermann steering autonomous vehicle

Project Resources

Why Ackermann steering?

Ackermann steering is a configuration of wheels on a vehicle charachterized by four wheels, two in the back that are powered by a DC motor, and two in the front that steer though commands received by a servo motor. In contrast, differential drive robots have two wheels that are independently powered by two DC-motors, with a passive omnidrectional third wheel that acts as support.

The dynamics (i.e., the “kind of movement”) of differential drive robots is quite different from real world automobiles, which, e.g., cannot turn on the spot. Ackerman steering achieves more realistic vehicle dynamics at cost: increased hardware complexity and mathematical modeling. But neither of these challenges have stopped talented Duckietown student from designing and implementing an Ackermann steering Duckiebot!

(Duckietown trivia: careful Duckietown observers will have noticed that the Duckiebot models historically have been called DB18, DB19, DB21, etc. – every wondered which would have been the DB20?)

Ackermann steering in Duckietown: the challenges

Ackermann steering introduces more complex mathematical modeling, with respect to differential drive robots, in order to predict future movement hence elaborate pose estimates on the fly. The kinematic modeling of the front steering apparatus is non trivial, and the radius of curvature Ackermann steering robots showcase is very different from differential drive robots.

Differential drive robots are capable of turning on the spot (applying equal and opposite commands to the two wheels), while anyone who has ever tried parallel parking a real car, knows that this is not possible.

How complex will it be for Ackermann steering robots to navigate Duckietown is the real challenge of this fun project.

The authors start from basic design elements through CAD, iterate through various bills of materials, make prototypes, and program them leveraging the Duckietown software infrastructure to achieve autonomous behaviors in Duckietown.

Project Highlights

Here is the output of their work. Check out the documents for more details!

Trace of a four bar rigid link configuration

Geometric approximation of the Ackermann criteria

Block diagram of a back-calculation anti-windup logic

DBv2 placed in a lane with the control inputs in red and the control outputs in white. The steering angle γ is calculated from the angle ω. The linear velocity is represented by v

Our geometry for implementing the Ackermann criteria

Ackermann criteria met on the steering of the DBv2

Ackermann steering: Results

(Turn on the sound for best experience!)

The autonomous behaviors of the Ackermann steering Duckiebot, a.k.a. DB20 or DBv2, shown above are the work of Timothy Scott, a former Duckietown student.

Ackermann steering Duckiebot: Authors

Merlin Hosner is a former Duckietown student in the Institute for Dynamic Systems and Controls (IDSC) of ETH Zurich (D-MAVT), and currently works at Climeworks as a Process Development Engineer.

Rafael Fröhlich is a former Duckietown student in the Institute for Dynamic Systems and Controls (IDSC) of ETH Zurich (D-MAVT), where he is currently a Research Assistant.

Learn more

Duckietown is a modular, customizable and state-of-the-art platform for creating and disseminating robotics and AI learning experiences.

It is designed to teach, learn, and do research: from exploring the fundamentals of computer science and automation to pushing the boundaries of knowledge.

Vision-based Reinforcement Learning for Lane-Tracking Control

Posted on June 8, 2024 | by Duckietown Admin

General Information

Vision-based Reinforcement Learning for Lane-Tracking Control
András Kalapos, Csaba Gór, Róbert Moni, István Harmati
Budapest University of Technology and Economics
Kalapos, A., Gór, C., Moni, R. and Harmati, I., 2021. Vision-based reinforcement learning for lane-tracking control. Acta IMEKO, 10(3), pp.7-14.

Vision-based reinforcement learning for lane-tracking control

What is Vision-based Reinforcement Learning? A few important topics:

Reinforcement Learning: a machine learning paradigm where an agent learns to make decisions by interacting with an environment to achieve a goal. In this context, reinforcement learning is used to teach a vehicle how to drive within Duckietown lanes by providing rewards or penalties based on its actions.

Vision-based Control: The control of the vehicle is based on visual inputs, specifically images captured by a forward-facing camera. These images are processed by a neural network to determine appropriate steering actions, allowing the vehicle to track lanes and avoid collisions.

Simulation-to-Reality (sim2real) Transfer Learning: The trained policy, which learns to control the vehicle in a simulated environment, is transferred to real-world scenarios. The effectiveness of the trained model in real-world driving situations is evaluated, demonstrating the ability to generalize learning from simulation to reality.

Domain Randomization: This technique involves introducing variations or randomizations into the simulation environment during training. By exposing the agent to a wide range of simulated scenarios with different lighting conditions, road surfaces, and other environmental factors, domain randomization helps improve the model’s ability to generalize to unseen real-world conditions.

Learn about RL, navigation and other robot autonomy topics at the link below!

Abstract

The present study focused on vision-based end-to-end reinforcement learning in relation to vehicle control problems such as lane following and collision avoidance. The controller policy presented in this paper is able to control a small-scale robot to follow the right-hand lane of a real two-lane road, although its training has only been carried out in a simulation.

This model, realised by a simple, convolutional network, relies on images of a forward-facing monocular camera and generates continuous actions that directly control the vehicle. To train this policy, proximal policy optimization was used, and to achieve the generalisation capability required for real performance, domain randomisation was used. A thorough analysis of the trained policy was conducted by measuring multiple performance metrics and comparing these to baselines that rely on other methods.

To assess the quality of the simulation-to-reality transfer learning process and the performance of the controller in the real world, simple metrics were measured on a real track and compared with results from a matching simulation. Further analysis was carried out by visualising salient object maps.

Highlights - Vision-based reinforcement learning for lane-tracking control

Here is a visual tour of the work of the authors. For more details, check out the full paper.

Fig 1. Illustration of the policy architecture with the notations used.

Fig 2. Explanation of the proposed orientation reward

Fig 3. Examples of domain randomised observations

Fig 4. .a) Test track used for simulated reinforcement learning and baseline evaluations; b) and c) real and simulated test track used for the evaluation of the simulation-to-reality transfer.

Fig 5. Learning curves for the reinforcement learning agent with different action representations and reward functions.

Sequence of robot positions in a collision avoidance experiment with a policy trained using the modified orientation reward. After 𝑡=6 s, the controlled robot follows the vehicle in front at a short but safe distance until the end of the episode (approximate distance is calculated as the distance between the centre points of the robots minus the length of a robot). — Fig 6. Sequence of Robot Positions in Collision Avoidance Experiment with Modified Orientation Reward Trained Policy

Fig 7. Salient objects highlighted on observations in different domains and tasks. Blue regions represent high activations throughout the network.

Conclusion

Here are the conclusions from the authors of this paper:

“This work presented a solution to the problem of complex, vision-based lane following in the Duckietown environment using reinforcement learning to train an end-to-end steering policy capable of simulation-to-real transfer learning. It was found that the training is sensitive to problem formulation, such as the representation of actions.

This study has demonstrated that by using domain randomisation, a moderately detailed and accurate simulation is sufficient for training end-to-end lane-following agents that operate in a real environment. The performance of these agents was evaluated by comparing some basic metrics to match real and simulated scenarios.

Agents were also successfully trained to perform collision avoidance in addition to lane following. Finally, salient object visualisation was used to give an illustrative explanation of the inner workings of the policies in both the real and simulated domains.”.

Project Authors

András Kalapos is a Machine Learning PhD Student at Budapest University of Technology and Economics, Hungary.

Csaba Gór is a Machine Learning Engineer at Turbine, in Hungary.

Róbert Moni is a Senior Machine Learning Engineer at Continental.

István Harmati is an Associate Professor at Budapest University of Technology and Economics.

Learn more

Duckietown is a platform for creating and disseminating robotics and AI learning experiences.

It is modular, customizable and state-of-the-art, and designed to teach, learn, and do research. From exploring the fundamentals of computer science and automation to pushing the boundaries of knowledge, Duckietown evolves with the skills of the user.

End-to-end Deep RL (DRL) systems: in autonomous driving environments that rely on visual input for vehicle control face potential security risks, including:

State Adversarial Perturbations: Subtle alterations to visual input that mislead the DRL agent, causing incorrect decision-making.
Reward Tampering: Manipulation of the reward signal to misguide the learning process, leading the agent to adopt unsafe or inefficient policies.

These vulnerabilities can compromise the safety and reliability of self-driving vehicles.

Deep Reinforcement Learning for Autonomous Navigation on Duckietown Platform: Evaluation of Adversarial Robustness

Evaluating Adversarial Robustness in Duckietown Navigation

Posted on June 4, 2024 | by Duckietown Admin

General Information

Deep Reinforcement Learning for Autonomous Navigation on Duckietown Platform: Evaluation of Adversarial Robustness
Abdullah Hosseini, Saeid Houti, Junaid Qadir
Qatar University
A. Hosseini, S. Houti and J. Qadir, "Deep Reinforcement Learning for Autonomous Navigation on Duckietown Platform: Evaluation of Adversarial Robustness," 2023 International Symposium on Networks, Computers and Communications (ISNCC), Doha, Qatar, 2023, pp. 1-6, doi: 10.1109/ISNCC58260.2023.10323905.

Deep RL for Autonomous Navigation on Duckietown Platform: Evaluation of Adversarial Robustness

What is adversarial robustness in navigation tasks all about? A few important topics:

Reinforcement Learning (RL) is a type of machine learning where agents learn to make decisions by receiving rewards or penalties based on their actions in an environment. This is great because it removed the need for curated training datasets.

Deep Reinforcement Learning (DRL) enhances RL by using deep neural networks to process complex inputs and make decisions. Deep networks are neural networks with multiple layers.

Adversarial Robustness refers to a system’s ability to resist and maintain performance despite deliberate attacks or input perturbations.

Navigation is the task of finding feasible paths between points in the environment like Google Maps or similar systems provide us in everyday life.

Learn about RL, navigation and other robot autonomy topics at the link below.

Abstract

Self-driving cars have gained widespread attention in recent years due to their potential to revolutionize the transportation industry. However, their success critically depends on the ability of reinforcement learning (RL) algorithms to navigate complex environments safely. In this paper, we investigate the potential security risks associated with end-to-end deep RL (DRL) systems in autonomous driving environments that rely on visual input for vehicle control, using the open-source Duckietown platform for robotics and self-driving vehicles.

We demonstrate that current DRL algorithms are inherently susceptible to attacks by designing a general state adversarial perturbation and a reward tampering approach. Our strategy involves evaluating how attacks can manipulate the agent’s decision-making process and using this understanding to create a corrupted environment that can lead the agent towards low-performing policies. We introduce our state perturbation method, accompanied by empirical analysis and extensive evaluation, and then demonstrate a targeted attack using reward tampering that leads the agent to catastrophic situations.

Our experiments show that our attacks are effective in poisoning the learning of the agent when using the gradient-based Proximal Policy Optimization algorithm within the Duckietown environment. The results of this study are of interest to researchers and practitioners working in the field of autonomous driving, DRL, and computer security, and they can help inform the development of safer and more reliable autonomous driving systems.

Highlights - Evaluation of Adversarial Robustness Results

Here is a visual tour of the work of the authors. For more details, check out the paper link.

Fig. 1 - An illustration of the MDP framework, a fundamental concept in RL, where an agent interacts with an environment in a sequential manner to learn an optimal policy.

Fig. 2 - High-level architecture for training a DRL agent.

Average reward obtained by the agent for various reward functions in Duckietown — Fig. 3 - Average reward obtained by the agent for various reward functions.

Average reward of the agent under two different reward functions. Increasing the value of ϵ resulted in a decrease in the agent’s distance traveled and position-orientation reward as it completely deviated from the roadable area and faced towards the wrong direction — Fig. 4 - Average reward of the agent under two different reward functions.

Adversarial Navigation Robustness - Sequence of robot positions with DRL agent trained under adversarial and non-adversarial settings in a lane following experiment. The UAPFGSM method, making the agent move in circular movements with minimal perturbations, while adversarial reward tampering forces it to move in the opposite direction of the road. — Fig. 5 - Sequence of robot positions with DRL agent trained under adversarial and non-adversarial settings in a lane following experiment. The UAP- FGSM method, making the agent move in circular movements with minimal perturbations, while adversarial reward tampering forces it to move in the opposite direction of the road.

Fig. 6 - Saliency map provides insight into this aspect of the model’s behavior, particularly in the context of adversarial attacks. Due to the universal perturbation of the UAP-FGSM attack, the agent’s attention to nearby yellow- dashed lines is reduced.

Conclusion

Here are the conclusions from the authors of this paper:

“The focus of our study was to address adversarial attacks on deep reinforcement learning (DRL) agents, specifically examining state adversarial attacks and reward-tampering attacks.

We developed a parametric framework for state adversarial attacks and a non-parametric framework for reward tampering attacks, which enabled us to create effective attacks. We found that the performance of a DRL agent declined rapidly after the attack, and the deviation from the road was worse than that of standard DRL.

We used salient maps to provide a clear explanation of the policies’ internal operations in both the adversarial and non-adversarial aspects. Our research provides insight into the potential vulnerabilities of DRL agents and highlights the need for more robust and secure agents to mitigate the risk of adversarial attacks.

Moving forward, future work will focus on incorporating real-world analysis to test the performance of the DuckieBot under both adversarial and non-adversarial settings”.

Project Authors

Abdullah Hosseini is a Research and Development Specialist at Weill Cornell Medicine in Qatar.

Saeid Houti is a Software Developer at Ministry of Education and Higher Education Qatar.

Junaid Qadir is a Professor of Computer Engineering at Qatar University.

Learn more

Duckietown is a platform for creating and disseminating robotics and AI learning experiences.

End-to-end Deep RL (DRL) systems: in autonomous driving environments that rely on visual input for vehicle control face potential security risks, including:

State Adversarial Perturbations: Subtle alterations to visual input that mislead the DRL agent, causing incorrect decision-making.
Reward Tampering: Manipulation of the reward signal to misguide the learning process, leading the agent to adopt unsafe or inefficient policies.

These vulnerabilities can compromise the safety and reliability of self-driving vehicles.

Introducing Autonomous Parking in Duckietown Cities

Posted on May 27, 2024 | by Duckietown Admin

Introducing Autonomous Parking in Duckietown Cities

Project Resources

Why Autonomous Parking?

Parking is notoriously a hard task to master for many humans. Hence, students of the Autonomous Mobility on Demand course at ETH Zurich wanted to determine to what degree this applied to autonomous parking with Duckiebots.

The goal of the Autonomous Parking project was to design, implement, and test a complete autonomous parking solution compliant with the Duckietown ecosystem.

Duckiebots should be able to enter and exit a parking area, identify viable parking lots, actually park and exit their parking spot safely, and avoid collision with other Duckiebots during the entire process.

The vision is to integrate autonomous charging solutions into the parking area, so Duckiebots can charge themselves when needed.

Autonomous parking in Duckietown: the challenges

Leveraging the Duckietown lane following vision baseline provided a basic infrastructure to build upon.

Some technical challenges specific to this projects were:

Backward Lane Following: Duckiebots must drive backward to exit the parking lots but only have cameras on the front. It is required to adjust the Duckiebot’s control system for stable backward driving, by changing the pose estimation process and re-tuning the PID controller.

Dynamic Color Adaptation: the new parking lot design introduced additional appearance specifications to the Duckietown city setup, such as blue lines identifying parking areas. Modifying the Duckiebots’ native lane detector to recognize blue lines in addition to yellow, red, and white, allows for additional flexibility in lane following based on specified colors.

Time Slot Coordination: Managing the availability of parking spaces is crucial to minimize the probability of collisions between Duckiebots. This project tackled this challenge by implementing a time-slot system to manage parking exits to prevent collisions, using red LEDs for signaling to other Duckiebots.

Project Highlights

Here is a visual tour of the work of the authors.

Check out the documents for more details!

Autonomous parking lot with Duckiebots in Duckietown — Resulting parking lot area configuration within Duckietown. The parking lot is compliant with Duckietown appearance specifications, and features one entry/exit intersection. The parking lot closed-loop and modular configuration With the goal of enabling Duckiebots to autonomously enter, park, and exit parking spaces while avoiding collisions. This design exemplifies the project's focus on creating a functional and adaptable parking solution.

Figure 2 illustrates a conceptual design of a parking area within the Duckietown environment. Comprising various standardized tiles, this depiction showcases a modular layout suitable for accommodating Duckiebots. The design includes six straight tiles, four curved left tiles, one three-way center tile, and two empty tiles. These components are arranged to form a configurable parking space that can be adapted to fit different spatial constraints. Notably, the configuration incorporates a T-intersection positioned such that Duckiebots are required to execute specific maneuvers upon entry and exit, ensuring adherence to the predetermined functionality of the parking area. — Design of a parking area within the Duckietown environment. Comprising various standardized tiles, this example showcases a modular layout suitable for accommodating Duckiebots. The design includes six straight tiles, four curved left tiles, one three-way center tile, and two empty tiles. Notably, the configuration incorporates a T-intersection positioned such that Duckiebots are required to execute specific maneuvers upon entry and exit, ensuring adherence to the predetermined functionality of the parking area.

Autonomous parking in Duckietown, parking lot specifications — Parking spaces are delimited by a blue line with specified geometry. This configuration is functional to allow the Duckietown lane filter to determine the pose of the robot and facilitate precise entrance and exit from the parking space.

Project Parking Results

(Turn on the sound for best experience!)

Project Authors

Trevor Phillips is a former Duckietown student, now a Machine Learning SWE at Apple in Switzerland.

Vincenzo Polizzi is a former Duckietown student, now a Ph. D. student at the University of Toronto, Canada.

Linus Lingg is a former Duckietown student, now the Co-Founder and CTO of bottleplus in Switzerland.

Learn more

Duckietown is a modular, customizable and state-of-the-art platform for creating and disseminating robotics and AI learning experiences.

It is designed to teach, learn, and do research: from exploring the fundamentals of computer science and automation to pushing the boundaries of knowledge.

Dino Claro: a Duckietown journey from project to thesis

Posted on May 24, 2024 | by Duckietown Admin

Dino Claro's Duckietown journey: from project to graduate thesis

Dino Claro, a mechanical and mechatronics engineering graduate from the University of Cape Town, shares his Duckietown journey: with challenges and results.

Cape Town, February 13th, 2024: Dino Claro, Graduate Mechanical and Mechatronics Engineer at the University of Cape Town, shares his experience with Duckietown and the project he developed using Duckiebots for his masters thesis.

Quick links

Duckinator: an Odometry pose-estimation for the Duckiebot robotic car platform

Hello and welcome Mr. Dino Claro! Could you introduce yourself?

My name is Dino Claro and I’m a Graduate Mechanical and Mechatronics Engineer at the University of Cape Town.

Thanks for accepting to share your experience with us. When did you first run into Duckietown?

During vacation work at the University of Cape Town (UCT) Mechatronic Systems Group, I was given the open-ended task of estimating the pose of a robot car. The goal of the vacation work was to solve a problem independently but also free from the
stresses of receiving a mark or grade. There was no expectation for novel work. In fact, the vacation work was only two weeks, and the expected solution would have been straightforward, probably odometry-based. Thus, Duckinator was born.

That's when you decided to use Duckietown?

With two platforms available, a basic Arduino 4WD kit and the Duckiebot, I could simply not resist the Duckies’ pull. The idea of using a Linux-based platform geared toward AI was extremely exciting.

At the end of the two-week vacation work, I was still ploughing through Duckietown documentation, the EdX: Self-Driving Cars with Duckietown MOOC, and ROS tutorials. My pose estimation solution seemed very far down the road. At that point, I should have realised that the DB (besides its cute exterior) is nuanced, to say the least.

Could you describe us your project?

The early phase of my project was extremely rudimentary. I had only had a couple of weeks during the vacation work to play with the DB [Duckiebot]. I planned to continue with the EdX MOOC [Self-Driving Cars with Duckietown, 2023 edition] while researching Docker and ROS on the side for the first couple of weeks and then begin development. A pitfall with this technique was completing a section of the MOOC or some other tutorial and believing I could implement it myself. My initial thinking was that if the MOOC could be completed in 10 weeks or so and given that I have already a couple of weeks’ headstart due to vacation work, I should be able to implement my standalone autonomous solution for the DB in the 12-week frame.

Spoiler alert, Duckinator did not rival Tesla. I made the realisation about 4 weeks into the project. At that stage, I was in the Object Detection activity of the MOOC. With the world in a frenzy over AI and ML, I was itching to dip my hands in some of this mysterious ML stuff.

Dr. Pretorius obliged, and my plan from this point was to implement my own standalone Duckietown-compliant Docker image for the YOLOv5. Charged with the excitement of the new project direction, I began researching ML, computer vision algorithms and YOLO itself. Implementing the YOLOv5 model was relatively smooth sailing and I loved learning computer vision. In all honesty, my YOLOv5 model was just organising the Object Detection MOOC into a standalone Docker image as the MOOC hides the Docker image from the student. I obtained the training data using the MOOC helper files and then trained the YOLOv5 model using a very similar Google Colab script as provided by the MOOC.

I slightly extended the YOLOv5 model from the MOOC by training the model to detect DBs, which proved to be sort of successful. As I only had one Duckiebot, I tested the model by parking Duckinator in front of a mirror or putting it in front of my laptop showing photos of other DBs. Due to this shabby testing, I left this extension out of my write-up. This was all completed after week 7.

With the world in a frenzy over AI and ML, I was itching to dip my hands in some of this mysterious ML stuff.

Dino Claro

Did you meet your objectives?

Completing the Object Detection model effectively meant that my revised project brief had been met but as I still had some time, I needed to extend the model in some way.

Duckinator had eyes but I wanted to make it move … autonomously. I had the idea of creating a safety controller where the distance of objects from the duckiebot could be inferred using the predicted bound box and perspective geometry.

My theory went as follows: knowing the real-world size of all the objects the DB could detect and comparing this to the dimensions of the bounding box provided by the YOLO model, it would be possible to infer the depth of the object and this depth could then be used to base autonomous controller commands. This led me to research autonomous vehicle safety architectures/controllers and modern depth estimation algorithms. I soon realised that much more advanced autonomous architectures existed. For instance, modern autonomous vehicles fuse camera feeds, object detection models, kinematic models and various other sensors to generate vector or depth maps. The creation of these depth maps is extremely complex and a field of intense research.

What where the challenges you encountered during your project?

After coding up my algorithm for the projective projection algorithm, I obtained unexpected results. Negative in most cases. Describing my algorithm in more clarity to Dr. Pretorius, he made it clear that a simple projection perspective would not work in this case.

I was projecting everything from the camera image to the ground plane but of course, the duckies and any other objects do not exist solely on the ground plane. This being week 10 of the project, I had simply run out of time and had to scrap the projective perspective and had no time to implement any of the more complex algorithms out there.

I was devastated at the fact that Duckinator was not going to move.

Upon some reflection though, the YOLOv5 model was working quite well, and I had all this research about autonomous architectures and depth estimation. One of the autonomous architectures I researched was Braitenberg vehicles acting as, possibly, the simplest autonomous architecture. A basic Braitenberg controller was simple enough to implement and would mean once again Duckinator could move.

Bounding boxes were populated onto a black image and then divided into left and right region maps. These maps were then element-wise multiplied with a weight matrix to provide a scalar value which can be used for wheel commands. Using the ‘fear’ Braitenberg vehicle the DB would then steer away from any detected objects. Another realisation was that my project was experimental with one of the main goals being for it to act as a stepping stone for future projects.

At this stage (two weeks before my project was due) I was satisfied with a newly engineered aim: Evaluating the viability of the Duckietown platform at the undergraduate level by implementing an ML object detection model. The key outputs being the YOLOv5 model (Duckie Detector) as well as possible future projects and trajectories for students.

The real learning occurs when getting your hands dirty, experimenting and troubleshooting.

Dino Claro

What are your final considerations?

Reflecting on my journey from a complete beginner to a slightly more competent beginner, here’s my advice for those on a similar journey:

Begin with a blank Duckietown-compliant Docker image and dive into coding a demo, whether it’s based on my solution or another. Ultimately, the goal is to first understand the code and then attempt to recreate it without directly copying.

While documentation and EdX activities are useful in providing broad overviews and points of contact for debugging, relying solely on them may create a deceptive sense of competency.

The real learning occurs when getting your hands dirty, experimenting and troubleshooting.

Thank you very much for taking the time, we appreciated your story very much! Is there anything else you would like to add?

Yes, I would say that embracing the hands-on experience is key to understanding the platform and being immersed in the infectious ethos surrounding Duckietown.

On that note, I would like to express my gratitude to Dr. Pretorius for granting me the freedom to experiment and the opportunity to work with the Duckiebot. I eagerly await future projects and the growth of the Duckietown community

Learn more about Duckietown

Duckietown enables state-of-the-art robotics and AI learning experiences.

It is designed to help teach, learn, and do research: from exploring the fundamentals of computer science and automation to pushing the boundaries of human knowledge.

Tell us your story

Are you an instructor, learner, researcher or professional with a Duckietown story to tell?

Reach out to us!

Massachusettes Institute of Technology: first BCI hackathon

Posted on May 20, 2024 | by Duckietown Admin

Cambridge, MA, USA – McGovern Institute, 24-25 February 2024: Over 100 participants took part to the first MIT BCI hackathon, competing in teams to control Duckiebots using brain computer interfaces.

Quick links

Controlling Duckiebots using brain computer interfaces

Over 100 participants gathered at the Massachusetts Institute of Technology for the first BCI hackathon organized by Dr. Federico Claudi. The participants tried to control a Duckiebot using only brain computer interfaces, and competed in a series of tasks.

BCI is the field of research that studies how to measure, amplify, filter and utilize electrical signals from the brain to interact with external devices.

What made this hackathon distinctive was the hands-on challenge, where participants were tasked with controlling a physical robot. This not only tested participants’ technical skills but also showcased their ability to tackle real-world problems through innovative BCI applications.

The task teams competed on was having Duckiebots (DB21-J4) navigate a road loop as fast as possible while avoiding Duckies. Here is an example:

The hardware used in this competition was an X.on EEG headset, and Duckiebots for control. Also, the winning team’s solution will be soon made available as a reproducible Learning Experience with Duckietown – stay tuned!

The Duckiebot is a DIY, Raspberry Pi-based robot powered by Nvidia and designed for introducing learners to autonomous technologies.

If you would like to contribute in developing accessible BCI LXs with Duckietown, and support the dissemination of BCI research, e.g. reach out to us at [email protected].

Learn more about Duckietown

The Duckietown platform enables state-of-the-art robotics and AI learning experiences.

It is designed to help teach, learn, and do research: from exploring the fundamentals of computer science and automation to pushing the boundaries of human knowledge.

Safe Reinforcement Learning (RL) Thesis Project

Posted on May 17, 2024 | by Duckietown Admin

Safe Reinforcement Learning (Safe-RL) in Duckietown

Project Resources

Safe-Reinforcement Learning (Safe-RL): Project Description

Safe-RL Duckietown Project – In his thesis titled “Safe-RL-Duckietown“, Jan Steinmüller used safe reinforcement learning to train Duckiebots to follow a lane while keeping said robots safe during training.

Safe Reinforcement Learning involves learning policies that maximize expected returns while ensuring reasonable system performance and adhering to safety constraints throughout both the learning and deployment phases. Reinforcement learning is a machine learning paradigm where agents learn to make decisions by maximizing cumulative rewards through interaction with an environment, without the necessity for training data or models.

The final result was a trained agent capable of following lanes while avoiding unsafe positions.

This is an open source project, and can be reproduced and improved upon through the Duckietown platform.

Safe Reinforcement Learning: Project Highlights

Here is a visual tour of the work of the author.

Check out the documents for more details.

Implementation of the safe reinforcement learning (Safe-RL) Duckietown project — Implementation of the Safe-RL Duckietown project

safe reinforcement learning in Duckietown project: Safety Layer Description — Safety Layer Description

Process diagram of action selection and safety layer in the safe reinforcement learning project using Duckietown — Process diagram of action selection and safety layer

Results of the safe reinforcement learning (Safe-RL) Duckietown Project — Results of the Safe-RL Duckietown Project

Safe Reinforcement Learning: Results and Conclusions

Based on the results, it can be concluded that there is no disadvantage to using a safety layer when doing reinforcement learning since execution time is very similar. Moreover, the dramatically improved safety of the vehicle is helpful for the robot’s training as fewer actions with lower or even negative rewards will be executed. Because of this, reinforcement learning agents with safety layers learn faster and reduce the number of unsafe actions that are being executed.

Unfortunately, manual observation and intervention by the user were still necessary, however, the frequency was clearly reduced which further improved learning as the robots in testing did not know if an outside intervention was done which could result in an action being rewarded incorrectly.

It was also concluded that this project did not reach perfect safety with the implementation. Therefore a fully autonomous reinforcement learning training without any human intervention has not yet been achieved. A lot of improvement factors have been found that can further improve the safety and recovery rate. Additionally, some major problems which are not direct results of the reinforcement learning or safety layer have been identified.

These problems could be attempted to be fixed in different ways like improving the open source implementations of lane filter nodes or adding more sensors or cameras to the robot in order to extend the input data to the agent. Another area that was untouched during the research of this project was other vehicles inside the current lane. The safety layer could potentially be extended to also include safety features that should keep the robot safe from hitting other vehicles.

Read the full report here.

Project Author

Jan Steinmüller is a computer science student working in the computer networks and information security research group at Hochschule Bremen in Germany.

Dr. Amr Alanwar is an Assistant Professor at the Technical University of Munich (TUM).

Learn more

Duckietown is a modular, customizable and state-of-the-art platform for creating and disseminating robotics and AI learning experiences.

It is designed to teach, learn, and do research: from exploring the fundamentals of computer science and automation to pushing the boundaries of knowledge.

Visual deep reinforcement learning paper snippet

Vision-Based DRL Autonomous Driving Agent with Sim2Real Transfer

Posted on May 6, 2024 | by Duckietown Admin

General Information

Vision-based DRL Autonomous Driving Agent with Sim2Real Transfer
Dianzhao Li and Ostap Okhrin
Universität Dresden, Dresden
D. Li and O. Okhrin, "Vision-Based DRL Autonomous Driving Agent with Sim2Real Transfer," 2023 IEEE 26th International Conference on Intelligent Transportation Systems (ITSC), Bilbao, Spain, 2023, pp. 866-873, doi: 10.1109/ITSC57777.2023.10422677.

Vision-Based DRL Autonomous Driving Agent with Sim2Real Transfer

One way to obtain quick and cheap training data is to use simulation instead of real-world experiments. The question remains if the learnings of a simulation-trained agent apply to the real world. Sim2Real transfer is the field of research that studies this problem.

The challenge is particularly meaningful when using vision as the primary sensing capability for robots. Vision-based deep reinforcement learning (DRL) refers to a technique where ML agents, typically modeled as multi-layered neural networks, learn to “make decisions” directly from visual input.

The essence of RL is training robotic agents based on policies that reward desirable outcomes. This family of techniques typically leads to increased adaptability to operational scenarios.

To learn about RL and its place in the larger context of robot autonomy, check out the resources below.

Abstract

To achieve fully autonomous driving, vehicles must be capable of continuously performing various driving tasks, including lane keeping and car following, both of which are fundamental and well-studied driving ones. However, previous studies have mainly focused on individual tasks, and car following tasks have typically relied on complete leader-follower information to attain optimal performance.

To address this limitation, we propose a vision-based deep reinforcement learning (DRL) agent that can simultaneously perform lane-keeping and car-following maneuvers.

To evaluate the performance of our DRL agent, we compare it with a baseline controller and use various performance metrics for quantitative analysis. Furthermore, we conduct a real-world evaluation to demonstrate the Sim2Real transfer capability of the trained DRL agent.

To the best of our knowledge, our vision-based car following and lane-keeping agent with Sim2Real transfer capability is the first of its kind.

Highlights - Sim2Real transfer results

Here is a visual tour of the work of the authors. For all the details, check out the paper link.

Fig. 1. The proposed DRL framework for vision-based multi-task autonomous driving agents. The perception module leverages camera images to produce impact attributes regarding the environment, then the DRL control module utilizes the information to control the agent with enhanced generalization.

Duckiebot DB21 — Fig. 2. Robot car [Duckiebot] used during the real-world evaluation. (a) Side view of the robot car which equipped with front-view camera and Jetson Nano 2GB. (b) Back view of the car with a pattern of circles.

Fig. 3. An example velocity trajectory for the leading vehicle generated by the Ornstein-Uhlenbeck process for the training process.

Fig. 4. Training results of PPO agent with ten independent seeds over one million steps.

Fig. 5. Example trajectories of DRL agent (red) and baseline controller (green) following random leader trajectory (orange) during the evaluation.

Fig. 6. Distribution of TTC and time headway for DRL agent (red) and baseline controller (green).

Fig. 7. Example trajectories of DRL agent (red) and baseline controller (green) following a self defined external leader trajectory (orange) during the evaluation.

Conclusion

This study proposes a vision-based DRL agent that can simultaneously perform lane-keeping and car-following tasks.

The overall system is divided into two modules: the perception module and the control module. The perception module extracts task-relevant attributes of the surroundings, while the control module is a DRL agent that takes these attributes as input. To evaluate the performance of the DRL agent, we compare it with a baseline algorithm in both simulation and real-world environments.

In the simulation, we compare the car following and lane-keeping capabilities of the DRL agent and baseline controller using various performance metrics. In the real-world environment, we demonstrate that the DRL agent can follow the leading vehicle while maintaining lane-keeping ability.

In future work, we plan to enhance our DRL agent by incorporating a comfort factor to address unstable driving behavior. Additionally, we aim to deploy more advanced algorithms for improved generalization.

Project Authors

Dianzhao Li is a research assistant at the Technische Universität Dresden, Dresden, Germany.

Ostap Okhrin is Chair of Statistics and Econometrics at the Institute of Economics and Transport, School of Transportation, Technische Universitat Dresden in Germany.

Learn more

Duckietown is a platform for creating and disseminating robotics and AI learning experiences.

Anatidaephilia: centralized city-based SLAM (cSLAM)

Posted on April 26, 2024 | by Duckietown Admin

Anatidaephilia: centralized city-based SLAM (cSLAM)

Project Resources

cSLAM Project Description

Project cSLAM – Simultaneous Localization and Mapping (SLAM) is a successful approach for robots to estimate their position and orientation in the world they operate in, while at the same time creating a representation of their surroundings.

This project, centralized SLAM (or cSLAM), enables a Duckiebot to localize itself, while the watchtowers and Duckiebots work together to build a map of the city. The task is achieved by using the camera of the Duckiebot, together with watchtowers located along the path, to detect AprilTags attached to the tiles, the traffic signs, and the Duckiebot itself.

P. S. Anatidaephilia, is Latin for loving, and being addicted to, the idea that somewhere, somehow, a duck is watching you.

Project Highlights

Here is a visual tour of the work of the authors.

Check out the documents for more details!

Duckietown cSLAM algorithm physical architecture for localizing Duckiebots — cSLAM algorithm physical architecture for localizing Duckiebots. Watchtowers are traffic lights without lights, and are used to transform Duckietown in Autolabs.

Duckietown cSLAM logical architecture for merging sensor measurements from robots and the city.

Duckietown cSLAM graph optimizer representation — AprilTag detections from Watchtowers and Duckiebots are harmonized by solving a minimization problem.

RViz visualization of Autolab virtual twin in Duckietown cSLAM project — RViz reconstruction of experimental localization outcomes.

cSLAM Autolab watchtowers network diagnostic

cSLAM Project Results

(Turn on the sound for best experience!)

This work developed into a paper, check the article here.

Project Authors

Rohit Suri is a former Duckietown student, now a Roboticist at Venti Technologies in Singapore.

Aleksandar Petrov is a former Duckietown student, now a Ph. D. student at the University of Oxford.

Amaury Camus is a former Duckietown student, now a Lead Robotics engineer at Hydromea SA in Switzerland.

Francesco Milano is a former Duckietown student, now a Ph. D. student at ETH Zurich in Switzerland.

Benson Kuan is a former Duckietown student, now a Senior Robotics Research Engineer at DSO National Laboratories in Singapore.

Learn more

Duckietown is a modular, customizable and state-of-the-art platform for creating and disseminating robotics and AI learning experiences.

It is designed to teach, learn, and do research: from exploring the fundamentals of computer science and automation to pushing the boundaries of knowledge.

Learning Skills to Navigate without a Master: A Sequential Multi-Policy Reinforcement Learning Algorithm

Posted on April 4, 2024 | by Duckietown Admin

General Information

Learning Skills to Navigate without a Master: A Sequential Multi-Policy Reinforcement Learning Algorithm
Ambedkar Dukkipati, Rajarshi Banerjee, Ranga Shaarad Ayyagari, Dhaval Parmar Udaybhai
Indian Institute of Science
A. Dukkipati, R. Banerjee, R. S. Ayyagari and D. P. Udaybhai, "Learning Skills to Navigate without a Master: A Sequential Multi-Policy Reinforcement Learning Algorithm," 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Kyoto, Japan, 2022, pp. 2483-2489, doi: 10.1109/IROS47612.2022.9981607.

Learning Skills to Navigate without a Master: A Sequential Multi-Policy Reinforcement Learning Algorithm

Reinforcement learning (RL) is a rising star approach for developing autonomous robot agents. The essence of RL is training agents based on policies that reward desirable outcomes, which leads to increased adaptability to operational scenarios. Through iterations, robots refine their decision-making, optimizing actions based on rewards and penalties. This method provides robots with the flexibility to handle unpredictable situations, enhancing their efficiency and effectiveness in real-world tasks. To learn about RL with Duckietown, check out the resources below.

Abstract

Solving complex problems using reinforcement learning necessitates breaking down the problem into manageable tasks, and learning policies to solve these tasks. These policies, in turn, have to be controlled by a master policy that takes high-level decisions. Hence learning policies involves hierarchical decision structures. However, training such methods in practice may lead to poor generalization, with either sub-policies executing actions for too few time steps or devolving into a single policy altogether. In our work, we introduce an alternative approach to learn such skills sequentially without using an overarching hierarchical policy. We propose this method in the context of environments where a major component of the objective of a learning agent is to prolong the episode for as long as possible. We refer to our proposed method as Sequential Soft Option Critic. We demonstrate the utility of our approach on navigation and goal-based tasks in a flexible simulated 3D navigation environment that we have developed. We also show that our method outperforms prior methods such as Soft Actor-Critic and Soft Option Critic on various environments, including the Atari River Raid environment and the Gym-Duckietown self-driving car simulator.

Highlights

Here is a visual tour of the work of the authors.

For all the details, check out the paper link!

(a) An illustration of policies learned in our approach. (b) A representation of the state space partitioned by the termination functions of the options. Each oval corresponds to the set of states classified as non-termination states by the corresponding nested termination function. — Fig. 1. (a) An illustration of policies learned in our approach. Different policies learn different skills out of necessity to traverse the environment and not terminate the episode. The trajectory spawned by the sequence of policies that correctly learns to avoid terminating the episode is shown in green. The policy π1 is used when the agent is in the middle of the corridor. When the end of the corridor is reached, π2 is selected to take a turn and avoid a collision. (b) A representation of the state space partitioned by the termination functions of the options. Each oval corresponds to the set of states classified as non-termination states by the corresponding nested termination function.

Learning Skills to Navigate without a Master: A Sequential Multi-Policy Reinforcement Learning Algorithm image — Fig. 2. (a) & (b) The input to the neural network in our 3D navigation environment. These images obtained from the simulation are scaled (and also transformed to grayscale in the case of (a)) and appended with K previous images, and sent to the agent as input. (c) A view of the Duckietown environment.

Autonomous robot Navigation performance results obtained on various environments: SAC, SOC, SAC with HER — Fig. 3. The results obtained on various environments.

Outputs of the three learned policies when fed the corresponding input shown on the left. πω1 : orange, πω2 : blue, πω3 : green. Each policy outputs a Gaussian distribution, the active policy is the one filled with color. A positive value for the output action corresponds to turning right, and a negative value indicates a left turn. — Fig. 4. Outputs of the three learned policies when fed the corresponding input shown on the left. πω_1: orange, πω_2 : blue, πω_3 : green. Each policy outputs a Gaussian distribution, the active policy is the one filled with color. A positive value for the output action corresponds to turning right, and a negative value indicates a left turn.

Conclusion

In this paper, the authors proposed an algorithm called “Sequential Soft Option Critic” that allows adding new skills dynamically without the need for a higher-level master policy. This can be applicable to environments where a primary component of the objective is to prolong the episode. We show that this algorithm can be used to effectively incorporate diverse skills into an overall skill set, and it outperforms prior methods in several environments.

Learn more

Duckietown is a platform for creating and disseminating robotics and AI learning experiences.

Duckietown is modular, customizable and state-of-the-art. It is designed to teach, learn, and do research: from exploring the fundamentals of computer science and automation to pushing the boundaries of knowledge.