General Information

Visual localization using multi-camera multi-robot system

Visual robot localization is a crucial problem in robotics: how to estimate the agents’ position using vision.

A common approach to solving it is through Simultaneous Localization and Mapping (SLAM) algorithms, using onboard sensors to map and estimate robot positions.

This work introduces a new algorithm for robot localization using AprilTag fiducial markers. It works on a rectangular map with four corner tags, requiring minimal configuration and offering flexibility in camera positions.

Unlike prior methods, this algorithm automatically stitches images from cameras, regardless of angle, and converts them into a top-down view for robot localization.

The approach promises flexibility, making adapting to dynamic camera setups easier without reconfiguration.

This solution offers automated robot localization with minimal setup, leveraging computer vision and AprilTags for more efficient mapping. The only constraint is the rectangular shape of the map and properly oriented corner markers, making it an ideal fit for scalable, adaptive robot environments.

Learn about robot autonomy, including perception, localization, and SLAM, starting from the link below!

Abstract

In the author’s words:

The article presents a general framework for detecting the boundaries of, stitching, adjusting perspective and finally localizing robot positions and azimuth angles for any rectangular map designated with AprilTag markers in the corners and possibly in the interior area.

At the same time, the focus of the researchers was to minimize the configuration required for the algorithm to operate – here limited to just the orientation and data of markers, dimensions of the map, markers and robots.

The location of cameras can be freely changed without the need to reconfigure anything or restart the program. This work has been tested on and turned out to be especially helpful for working with the Duckietown project.

Highlights - Visual localization using multi-camera multi-robot system

Here is a visual tour of the work of the authors. For more details, check out the full paper.

A miniature Duckietown setup with small robot vehicles, roads, and AprilTag markers used for testing visual localization and autonomous navigation. — Figure 1. Duckietown Test Environment.

A diagram showing common features (corner points) between two images captured by different cameras, used for stitching in the homography matrix algorithm. — Figure 2. Common Features Between Two Cameras.

A diagram illustrating the stitching step masks, where overlapping areas of images are blended with equal weights, and disjoint areas are added with full opacity. — Figure 3. Stitching Step Masks for Image Blending.

A stitched image with magenta points marking the map's corner boundaries, white cross marks for the corner markers' top-left corners, and red/blue crosses for inner markers. — Figure 4. Stitched Image with Map Boundaries and Marker Corners.

A diagram showing corner reprojection to a top-down perspective with colorful lines connecting matching points, orange centroids of detections, and white labels for corner codes. — Figure 5. Corner Reprojection to Top-Down Perspective.

A map showing robot positions marked by green crosses, with azimuth angles relative to north and coordinates represented as percentages of the map's dimensions. — Figure 6. Robot Position and Azimuth in Map Coordinates.

A stitched image showing the extrapolation of one missing corner, where the algorithm estimates its position to create a complete view of the map. — Figure 7. Extrapolation of Missing Corner in Stitched View.

A reprojection view showing the extrapolation of one missing corner, with the algorithm estimating the corner's position for a complete top-down perspective. — Figure 8. Extrapolation of Missing Corner in Reprojection View.

The final result of missing corner extrapolation, showing a complete map with the estimated corner integrated smoothly into the overall environment. — Figure 9. Final Result of Missing Corner Extrapolation.

A view showing common features in an image with no shared corners and three total corners, illustrating feature detection without overlapping corner points. — Figure 10. Common Features View with Zero Common Corners.

A stitched image where no corners are shared but three total corners are present, showing how the algorithm combines images despite the absence of common corners. — Figure 11. Stitched View with Zero Common Corners.

A reprojection view showing a scenario with no shared corners and three total corners, where the algorithm forms a complete top-down perspective despite the lack of common corners. — Figure 12. Reprojection View with Zero Common Corners.

The result view showing a complete image with zero common corners and three total corners, illustrating the successful integration of data without overlapping features. — Figure 13. Result View with Zero Common Corners.

A common features view showing zero common corners and three total corners, highlighting one common feature detected by the algorithm despite the lack of overlapping corners. — Figure 14. Common Features View with One Overlapping Feature.

A stitched image showing zero common corners and three total corners, highlighting one common feature detected, demonstrating the algorithm's ability to combine images effectively. — Figure 15. Stitched View with One Common Feature.

A reprojection view showing zero common corners and three total corners, emphasizing one common feature detected by the algorithm, illustrating effective data integration. — Figure 16. Reprojection View with One Common Feature.

The result view showing zero common corners and three total corners, highlighting one common feature detected by the algorithm, demonstrating successful data integration. — Figure 17. Result View with One Common Feature.

Conclusion - Visual localization using multi-camera multi-robot system

Here are the conclusions from the authors of this paper:

“The primary contribution and aim of this work is to provide a universal framework for stitching views of the same map from multiple cameras that can be freely moved and laid out around the map, with minimal required configuration.

The requirements for placement of codes are also loose: only the orientation with respect to the map frame is constrained and configuration of corner codes is required, as well as the lower limit of visible common markers on two images to be processed is 1, with no need for any corner markers to be present in both images at the same time.

The algorithms efficiency, however, depends on the quality of the homography matrices used in it, which implies that the more detections and corner detections, the better the result. It happens that the stitched / extrapolated coordinates may be off ’ground truth’ in some cases, or even stitching might fail, resulting in malformed output.

The authors provided experiments on two cameras, yet the algorithm may be run sequentially with images from more cameras. The algorithm may be improved in the future by applying more sophisticated methods of aggregating values of multiple detections of a given robot, such as a weighted combination of the position based on the quality of each detection.”

Project Authors

Artur Morys – Magiera is a PhD candidate at AGH University of Krakow, Poland.

Marek Długosz is a graduate and faculty member of the Faculty of Electrical Engineering, Automatics, Computer Science and Biomedical Engineering at the AGH University of Science and Technology in Krakow, Poland.

Paweł Skruch is a Professor of the AGH University of Science and Technology, Poland.

Learn more

Duckietown is a platform for creating and disseminating robotics and AI learning experiences.

It is modular, customizable and state-of-the-art, and designed to teach, learn, and do research. From exploring the fundamentals of computer science and automation to pushing the boundaries of knowledge, Duckietown evolves with the skills of the user.