General Information

Title: Simulation to Real Domain Adaptation for Lane Segmentation
Authors: Márton Tim, Márton Szemenyei, Róbert Moni
Institution: Budapest University of Technology and Economics, Budapest, Hungary
Citation: M. Tim, M. Szemenyei and R. Moni, "Simulation to Real Domain Adaptation for Lane Segmentation," 2020 23rd International Symposium on Measurement and Control in Robotics (ISMCR), Budapest, Hungary, 2020, pp. 1-6, doi: 10.1109/ISMCR51255.2020.9263406.

Sim2Real Lane Segmentation via Domain Adaptation

This embodied AI work investigates Sim2Real transfer: the process of applying ML agents trained in simulation to real-world environments, for semantic lane segmentation in mobile robotics using domain adaptation techniques.

The study addresses the distributional shift between synthetic (simulated) and real-world data using unsupervised and semi-supervised learning approaches that minimize the need for manual annotation by learning from unlabeled data or limited labeled samples.

A convolutional neural network (CNN) with an encoder-decoder architecture is trained on labeled synthetic data generated in the Duckietown Gym and adapted to unlabeled real-world images captured in the physical Duckietown setup.

The method integrates:

Feature-level and pixel-level adaptation, aligning internal representations and input appearance between domains to ensure consistent segmentation.
Adversarial training, where a discriminator encourages the CNN to learn domain-invariant features.
Cycle-consistent generative adversarial networks (CycleGANs), which perform image-to-image translation to make synthetic images visually similar to real ones while preserving semantic structure.
Evaluation using mean Intersection over Union (mIoU) and pixel accuracy, both standard metrics for assessing segmentation quality.

The results demonstrate that domain adaptation enables effective Sim2Real transfer for lane detection in Duckietown with minimal supervision advancing the deployment of robust, label-efficient perception systems in embedded robotics and autonomous navigation.

Highlights - Sim2Real lane segmentation via domain adaptation

Here is a visual tour of the implementation of lane segmentation via domain adaptation by the authors. For all the details, check out the full paper.

Abstract

Here is the abstract of the work, directly in the words of the authors:

As the cost of labelling and collecting real world data remains an issue for companies, simulator training and transfer learning slowly evolved to be the foundation of many state-of the-art projects. In this paper these methods are applied in the Duckietown setup where self-driving agents can be developed and tested.

Our aim was to train a selected artificial neural network for right lane segmentation on simulator generated stream of images as a comparison baseline, then use domain adaptation to be more precise and stable in the real environment. We have tested and compared four knowledge transfer methods that included domain transformation using CycleGAN and semi-supervised domain adaptation via Minimax Entropy.

As the latter was previously untested in semantic segmentation according to our best knowledge, we have contributed to showing it is indeed possible and produces promising results. Finally we have shown that it could also create a model that fulfills our performance requirements of stability and accuracy.We show that the selected methods are equally eligible for the simulation to real transfer learning problem, and that the simplest method delivers the best performance.

Conclusion - Sim2Real lane segmentation via domain adaptation

Here is the conclusion according to the authors of this paper:

Our goal was to create a stable and accurate right lane segmentation network by means of simulator data and domain adaptation techniques. We have tested and compared four knowledge transfer methods that included domain transformation using CycleGAN and semi-supervised domain adaptation via Minimax Entropy. We have shown that in the given scenario simulator-trained models have relatively good performance on real images, though their stability is a key weakness.

Our findings demonstrate that domain transformation using CycleGAN has limited applicability in segmentation tasks due to its distorting effect on road geometry, however the similarity between training and testing domains did result in increased stability.

Unfortunately, histogram matching failed in our case to improve on the baseline solution, producing similar results to CycleGAN.

We have observed that one of the simplest domain adaptation methods, source and target combined domain training helped to produce the best performing model according to numerical evaluation.

We implemented and demonstrated how semi-supervised domain adaptation via Minimax Entropy, a complex, entropybased adversarial method is applicable for segmentation tasks.

In the end, all the existing results were compared and evaluated with the conclusion that source and target combined domain training produced the best results of all investigated methods tied with SSDA via Minimax Entropy. Thereby, the usability of the latter method in segmentation tasks has also been proven.

Did this work spark your curiosity?

Check out the following works on vehicle autonomy with Duckietown:

Project Authors

Márton Tim is currently working as a deep learning engineer at Continental, Hungary.

Márton Szemenyei is an Associate Professor at Budapest University of Technology and Economics, Hungary.

Robert Moni is currently working as a Senior Machine Learning Engineer at Continental, Hungary.

Learn more

Duckietown is a platform for creating and disseminating robotics and AI learning experiences.

It is modular, customizable and state-of-the-art, and designed to teach, learn, and do research. From exploring the fundamentals of computer science and automation to pushing the boundaries of knowledge, Duckietown evolves with the skills of the user.