Pure Pursuit Lane Following with Obstacle Avoidance

Posted on June 16, 2025 | by Duckietown Admin

Pure Pursuit Lane Following with Obstacle Avoidance

Project Resources

Objective: Develop a lane following and obstacle avoidance system using the pure pursuit control for Duckiebots. (DB18)
Approach: Use an adaptive pure pursuit controller with dynamic speed tuning, integrated with vision-based deep-learning based object detection for obstacle-aware navigation.
Authors: Soroush Saryazdi, Dhaivat Bhatt, Robotics and Embodied AI Lab (REAL), Université de Montréal and is also affiliated with MILA.

Project highlights

Pure Pursuit Lane Following with Obstacle Avoidance - the objectives

Pure pursuit is a geometric path tracking algorithm used in autonomous vehicle control systems. It calculates the curvature of the road ahead by determining a target point on the trajectory and computing the required angular velocity to reach that point based on the vehicle’s kinematics.

Unlike proportional integral derivative (PID) control, which adjusts control outputs based on continuous error correction, pure pursuit uses a lookahead point to guide the vehicle along a trajectory, enabling stable convergence to the path without oscillations. This method avoids direct dependency on derivative or integral feedback, reducing complexity in environments with sparse or noisy error signals.

This project aims to implement a pure pursuit-based lane following system integrated with obstacle avoidance for autonomous Duckiebot navigation. The goal is to enable real-time tracking of lane centerlines while maintaining safety through detection and response to dynamic obstacles such as other Duckiebots or cones.

The pipeline includes a modified ground projection system, an adaptive pure pursuit controller for path tracking, and both image processing and deep learning-based object detection modules for obstacle recognition and avoidance.

Faster RCNN model architecture for pure pursuit and obstacle avoidance in Duckietown object detection — Faster RCNN Architecture with Feature Pyramid Network

Object detection results using Faster RCNN for pure pursuit and obstacle avoidance in Duckietown with bounding boxes for duckiebots and traffic cones — Faster RCNN Detection Output with Bounding Boxes

Duckietown object detection for pure pursuit and obstacle avoidance showing detected duckiebots and cones using deep learning — Detection Results for Obstacle Avoidance in Duckietown

The challenges and approach

The primary challenges in this project include robust target point estimation under variable lighting and environmental conditions, real-time object detection with limited computational resources, and smooth trajectory control in the presence of dynamic obstacles.

The approach involves modular integration of perception, planning, and control subsystems.

For perception, the system uses both classical image processing methods and a trained deep learning model for object detection, enabling redundancy and simulation compatibility.

For planning and control, the pure pursuit controller dynamically adjusts speed and steering based on the estimated target point and obstacle proximity. Target point estimation is achieved through ground projection, a transformation that maps image coordinates to real-world planar coordinates using a calibrated camera model. Real-time parameter tuning and feedback mechanisms are included to handle variations in frame rate and sensor noise.

Obstacle positions are also ground-projected and used to trigger stop conditions within a defined safety zone, ensuring collision avoidance through reactive control.

Looking for similar projects?

Check out the following works on path planning with Duckietown:

Pure Pursuit Lane Following with Obstacle Avoidance: Authors

Soroush Saryazdi is currently leading the Neural Networks team at Matic, supervised by Navneet Dalal.

Dhaivat Bhatt is currently working as a Machine learning research engineer at Samsung AI centre, Toronto.

Learn more

Duckietown is a modular, customizable, and state-of-the-art platform for creating and disseminating robotics and AI learning experiences.

Duckietown is designed to teach, learn, and do research: from exploring the fundamentals of computer science and automation to pushing the boundaries of knowledge.

These spotlight projects are shared to exemplify Duckietown’s value for hands-on learning in robotics and AI, enabling students to apply theoretical concepts to practical challenges in autonomous robotics, boosting competence and job prospects.

Deep Reinforcement Learning for Autonomous Lane Following

Posted on February 2, 2025 | by Duckietown Admin

Deep Reinforcement Learning for Autonomous Lane Following

Project Resources

Objective: Develop a deep reinforcement learning model for autonomous lane following with sim-to-real transfer.
Approach: Train a reinforcement learning agent in simulation using an autoencoder-based feature extraction pipeline and deploy it on a real Duckiebot with domain adaptation techniques.
Authors: Mickyas Tamiru Asfaw, David Bertoin, Valentin Guillet

Project highlights

Here is a visual tour of the author’s work on implementing deep reinforcement learning for autonomous lane following in Duckietown.

Basic autoencoder architecture used in deep reinforcement learning for feature extraction and dimensionality reduction. — Figure 2. Autoencoder Architecture for Deep Reinforcement Learning.

Image preprocessing and reconstruction using an autoencoder for deep reinforcement learning in autonomous lane following. — Figure 3. Image Preprocessing and Autoencoder Reconstruction.

Deep reinforcement learning for autonomous lane following in Duckietown: objective and importance

Would it not be great if we could train an end-to-end neural network in simulation, plug it in the physical robot and have it drive safely on the road?

Inspired by this idea, Mickyas worked to implement deep reinforcement learning (DRL) for autonomous lane following in Duckietown, training the agent using sim-to-real transfer.

The project focuses on training DRL agents, including Deep Deterministic Policy Gradient (DDPG), Twin Delayed DDPG (TD3), and Soft Actor-Critic (SAC), to learn steering control using high-dimensional camera inputs. It integrates an autoencoder to compress image observations into a latent space, improving computational efficiency.

The hope is for the trained DRL model to generalize from simulation to real-world deployment on a Duckiebot. This involves addressing domain adaptation, camera input variations, and real-time inference constraints, amongst other implementation challenges.

Autonomous lane following is a fundamental component of self-driving systems, requiring continuous adaptation to environmental changes, especially whn using vision as main sensing modality. This project identifies limitations in existing DRL algorithms when applied to real-world robotics, and explores modifications in reward functions, policy updates, and feature extraction methods analyzing the results through real world experimentation.

The method and challenges in implementing deep reinforcement learning in Duckietown

The method involves training a DRL agent in a simulated Duckietown environment (Gym Duckietown Simulator) using an autoencoder for feature extraction.

The encoder compresses image data into a latent space, reducing input dimensions for policy learning. The agent receives sequential encoded frames as observations and optimizes steering actions based on reward-driven updates. The trained model is then transferred to a real Duckiebot using a ROS-based communication framework.

Challenges for pulling this off include accounting for discrepancies between simulated and real-world camera inputs, which affect performance and generalization. Differences in lighting, surface textures, and image normalization require domain adaptation techniques.

Moreover, computational limitations on the Duckiebot prevent direct onboard execution, requiring a distributed processing setup.

Reward shaping influences learning stability, and improper design of the reward function leads to policy exploitation or suboptimal behavior. Debugging DRL models is complex due to interdependencies between network architecture, exploration strategies, and training dynamics.

The project addresses these challenges by refining preprocessing, incorporating domain randomization, and modifying policy structures.

Deep reinforcement learning for autonomous lane following: full report

Deep reinforcement learning for autonomous lane following in Duckietown: Authors

Mickyas Tamiru Asfaw is currently working as an AI Robotics and Innovation Engineer at the CESI lineact laboratory, France.

David Bertoin is currently working as a ML Applied Scientist at Photoroom, France.

Valentin Guillet is currently working as a Research engineer at IRT Saint Exupéry, France.

Learn more

Duckietown is a modular, customizable, and state-of-the-art platform for creating and disseminating robotics and AI learning experiences.

Duckietown is designed to teach, learn, and do research: from exploring the fundamentals of computer science and automation to pushing the boundaries of knowledge.

Reinforcement Learning for the Control of Autonomous Robots

Posted on November 6, 2024 | by Duckietown Admin

Reinforcement Learning for the Control of Autonomous Robots

Project Resources

Objective: Develop and evaluate reinforcement learning (RL) techniques for safe and autonomous navigation in any Duckietown
Approach: Develop, train and test RL algorithms including Deep Q-Networks (DQN), Deep Deterministic Policy Gradient (DDPG), and Proximal Policy Optimization (PPO), for autonomous lane-keeping and obstacle detection on a DB21 Duckiebot.
Authors: Bruno Fournier, Sébastien Biner

RL on Duckiebots - Project highlights

Here is a visual tour of the authors’ work on implementing reinforcement learning in Duckietown.

Diagram illustrating the basic principle of reinforcement learning applied to autonomous driving, showing an agent interacting with the environment, making decisions based on rewards and feedback. — Figure 1. Principle of Reinforcement Learning in Autonomous Driving.

Diagram showing the application of reinforcement learning in the Duckietown environment, with a Duckiebot navigating simulated roadways based on RL feedback. — Figure 2. Reinforcement Learning in the Duckietown Environment.

Comparison diagram showing the differences between Q-learning and Deep Q-Networks (DQN). — Figure 3. Q-learning vs. Deep Q-Networks (DQN).

Diagram depicting the learning process with the Deep Q-Network (DQN) model, showing how actions are taken based on state inputs and updated using Q-value estimations. — Figure 4. Learning Process with the Deep Q-Network (DQN) Model.

Diagram illustrating the architecture of the Deep Deterministic Policy Gradient (DDPG) algorithm, highlighting the actor and critic networks, experience replay, and target networks. — Figure 5. Architecture of the Deep Deterministic Policy Gradient (DDPG) Algorithm.

Image of a simulation environment in Duckietown, displaying a virtual map with roads, intersections, and a Duckie. — Figure 6. Simulation Environment in Duckietown.

Diagram of the Duckiebot test track with modular square elements, including straight lines, right-angle turns, and intersections, illuminated by two Walimex Pro LED lamps. — Figure 7. Modular Test Track for Duckiebot Driving Tests.

Diagram illustrating the Duckiebot’s reward factors: the distance from the center of the lane (laned) and the angle relative to the lane’s centerline (laneθ), used in the DQN reward function. — Figure 8. Reward Factors for DQN in Duckiebot Navigation.

Side-by-side images showing line detection in Duckietown before and after HSV parameter correction, illustrating improved clarity and accuracy of detected lines. — Figure 9. Line Detection Improvement with HSV Parameter Correction.

Diagram illustrating the structure of the PA2 DQN model, showing the pre-processing of RGB images before they are fed into the neural network for reinforcement learning. — Figure 10. DQN Model Structure.

Aerial view of the "loop_empty" training map used for DQN model training, featuring straight sections and both left and right turns. — Figure 11. Training Map for DQN.

Illustration highlighting the differences between simulation and reality in the context of Duckietown, including variations in color tones, camera angles, and environmental objects. — Figure 12. Differences Between Simulation and Reality.

Graph showing the average reward and average episode length for the DQN model in PA2 over multiple training episodes. — Figure 13. DQN (PA2): Average Reward and Average Episode Length.

Graph showing episode-based rewards during the first phase of DDPG training. — Figure 14. DDPG Training 1: Episode-Based Rewards.

Graph displaying episode-based rewards during the second phase of DDPG training. — Figure 15. DDPG Training 2: Episode-Based Rewards.

Graph showing the average rewards achieved during the training process. — Figure 16. Average Rewards During Training.

Graph showing the average distance traveled during each episode throughout the training. — Figure 17. Average Distance Traveled During Episodes.

Graph showing the evolution of the agent's speed throughout the training process. — Figure 18. Evolution of the Agent's Speed.

Graph showing the average reward and average episode length during Trial 1 of training. — Figure 19. Trial 1: Average Reward and Average Episode Length.

Graph showing the average reward and average episode length during Trial 2 of training. — Figure 20. Trial 2: Average Reward and Average Episode Length.

Graph showing the average reward and average episode length during Trial 3 of training. — Figure 21. Trial 3: Average Reward and Average Episode Length.

Graph showing the average reward and average episode length during Trial 4 of training. — Figure 22. Trial 4: Average Reward and Average Episode Length.

Visualization of the agent's trajectory on the evaluation track during testing. — Figure 23. Agent Trajectory on the Evaluation Track.

Graph showing the average reward and average episode length during Trial 5 of training. — Figure 24. Trial 5: Average Reward and Average Episode Length.

Graph showing the average reward and average episode length during Trial 6 of training. — Figure 25. Trial 6: Average Reward and Average Episode Length.

Visualization of the robot's trajectory as it negotiates a bend on the track. — Figure 26. Trajectory Taken by the Robot to Negotiate a Bend.

Graph showing the evolution of the safety factor throughout the training process. — Figure 27. Evolution of the Safety Factor.

Why reinforcement learning for the control of Duckiebots in Duckietown?

This thesis explores the use of reinforcement learning (RL) techniques to enable autonomous navigation in the Duckietown. Reinforcement learning is a type of machine learning where an agent learns to make decisions by performing actions in an environment and receiving feedback through rewards or penalties. The goal is to maximize long-term rewards.

This work focuses on implementing and comparing various RL algorithms—specifically Deep Q-Network (DQN), Deep Deterministic Policy Gradient (DDPG), and Proximal Policy Optimization (PPO) – to analyze performance in autonomous navigation. RL enables agents to learn behaviors by interacting with their environment and adapting to dynamic conditions. The PPO model was found demonstrating smooth driving using grayscale images for enhanced computational efficiency.

Another feature of this project is the integration of YOLO v5, an object detection model, which allowed the Duckiebot to recognize and stop for obstacles, improving its safety capabilities. This integration of perception and RL enabled the Duckiebot not only to follow lanes but also to navigate autonomously, making ‘real-time’ adjustments based on its surroundings.

By transferring trained models from simulation to physical Duckiebots (Sim2Real), the thesis evaluates the feasibility of applying these models to real-world autonomous driving scenarios. This work showcases how reinforcement learning and object detection can be combined to advance the development of safe, autonomous navigation systems, providing insights that could eventually be adapted for full-scale vehicles.

Reinforcement learning for the control of Duckiebots in Duckietown - the challenges

Implementing reinforcement learning, in this project faced a number of challeneges summarized below –

Transfer from Simulation to Reality (Sim2Real): Models trained in simulations often encountered difficulties when applied to real-world Duckiebots, requiring adjustments for accurate and stable performance.
Computational Constraints: Limited processing power on the Duckiebots made it challenging to run complex RL models and object detection algorithms simultaneously.
Stability and Safety of Learning Models: Guaranteeing that the Duckiebot’s actions were safe and did not lead to erratic behaviors or collisions required fine-tuning and extensive testing of the RL algorithms.
Obstacle Detection and Avoidance: Integrating YOLO v5 for obstacle detection posed challenges in ensuring smooth integration with RL, as both systems needed to work harmoniously for obstacle avoidance.

These challenges were addressed through algorithm optimization, iterative model testing, and adjustments to the hyperparameters.

Reinforcement learning for the control of Duckiebots in Duckietown: Results

Reinforcement learning for the control of Duckiebots in Duckietown: Authors

Bruno Fournier is currently pursuing Master of Science in Engineering, Data Science at the HES-SO Haute école spécialisée de Suisse occidentale, Switzerland.

Sébastien Biner is currently pursuing Bachelor of Science in Automotive and Vehicle Technology at the Berner Fachhochschule BFH, Switzerland.

Learn more

Duckietown is a modular, customizable and state-of-the-art platform for creating and disseminating robotics and AI learning experiences.

It is designed to teach, learn, and do research: from exploring the fundamentals of computer science and automation to pushing the boundaries of knowledge.

Safe Reinforcement Learning (RL) Thesis Project

Posted on May 17, 2024 | by Duckietown Admin

Safe Reinforcement Learning (Safe-RL) in Duckietown

Project Resources

Safe-Reinforcement Learning (Safe-RL): Project Description

Safe-RL Duckietown Project – In his thesis titled “Safe-RL-Duckietown“, Jan Steinmüller used safe reinforcement learning to train Duckiebots to follow a lane while keeping said robots safe during training.

Safe Reinforcement Learning involves learning policies that maximize expected returns while ensuring reasonable system performance and adhering to safety constraints throughout both the learning and deployment phases. Reinforcement learning is a machine learning paradigm where agents learn to make decisions by maximizing cumulative rewards through interaction with an environment, without the necessity for training data or models.

The final result was a trained agent capable of following lanes while avoiding unsafe positions.

This is an open source project, and can be reproduced and improved upon through the Duckietown platform.

Safe Reinforcement Learning: Project Highlights

Here is a visual tour of the work of the author.

Check out the documents for more details.

Implementation of the safe reinforcement learning (Safe-RL) Duckietown project — Implementation of the Safe-RL Duckietown project

safe reinforcement learning in Duckietown project: Safety Layer Description — Safety Layer Description

Process diagram of action selection and safety layer in the safe reinforcement learning project using Duckietown — Process diagram of action selection and safety layer

Results of the safe reinforcement learning (Safe-RL) Duckietown Project — Results of the Safe-RL Duckietown Project

Safe Reinforcement Learning: Results and Conclusions

Based on the results, it can be concluded that there is no disadvantage to using a safety layer when doing reinforcement learning since execution time is very similar. Moreover, the dramatically improved safety of the vehicle is helpful for the robot’s training as fewer actions with lower or even negative rewards will be executed. Because of this, reinforcement learning agents with safety layers learn faster and reduce the number of unsafe actions that are being executed.

Unfortunately, manual observation and intervention by the user were still necessary, however, the frequency was clearly reduced which further improved learning as the robots in testing did not know if an outside intervention was done which could result in an action being rewarded incorrectly.

It was also concluded that this project did not reach perfect safety with the implementation. Therefore a fully autonomous reinforcement learning training without any human intervention has not yet been achieved. A lot of improvement factors have been found that can further improve the safety and recovery rate. Additionally, some major problems which are not direct results of the reinforcement learning or safety layer have been identified.

These problems could be attempted to be fixed in different ways like improving the open source implementations of lane filter nodes or adding more sensors or cameras to the robot in order to extend the input data to the agent. Another area that was untouched during the research of this project was other vehicles inside the current lane. The safety layer could potentially be extended to also include safety features that should keep the robot safe from hitting other vehicles.

Read the full report here.

Project Author

Jan Steinmüller is a computer science student working in the computer networks and information security research group at Hochschule Bremen in Germany.

Dr. Amr Alanwar is an Assistant Professor at the Technical University of Munich (TUM).

Learn more

Duckietown is a modular, customizable and state-of-the-art platform for creating and disseminating robotics and AI learning experiences.

It is designed to teach, learn, and do research: from exploring the fundamentals of computer science and automation to pushing the boundaries of knowledge.