From Visual Distance Estimation to Multi-Robot Collision Avoidance

Collective Intelligence
from a Synthetic and Biological Perspective

Pia Bideau
INRIA Grenoble, France
(formerly Excellence Cluster - Science of Intelligence, Technical University Berlin, Germany)

David Bierbach
Excellence Cluster - Science of Intelligence, Humboldt University Berlin, Germany

Wolfgang Hönig
Technical University Berlin, Germany

two robots two robots

Assignment

Summary

Distance estimation is an essential aspect of self-orientation, allowing agents to move in a natural environment. In particular when animals move in teams (e.g. fish schools, flocking birds), they seem to localize themself relative to their neighbors - efficient and accurate enough such that quite astonishing behaviors arise. Throughout this assignment students will develop a synthetic approach simulating simple collective behavior, which will be deployed on a lego-robot platform. The assignment covers four major parts:

  • Distance estimation. By discovering useful properties of perspective projection students are guided to develop an approach to estimate distance to other (possibly moving) agents. A classical object detector (YOLOv5) is extended to estimate distance to neighboring robots given bounding box detections.
  • Collision Avoidance. Practical insights into Buffered Voronoi Cells (BVC), a modern approach for distributed collision avoidance is provided. Together with single-robot planning techniques, BVC is used to plan smooth trajectories for many mobile robots - in simulation and on lego robots.
  • Synthesizing collective behavior. The results of the previous two assignment parts are combined to create synthetic behavior in the real world and in a multi-robot setting.
  • Analyzing natural collective behavior. With help of a robot that mimics appearance and behavior of conspecifics, students embody hypotheses regarding agent perception (e.g. distance estimation) and interactions and probe those in real world settings.

The project material is designed for graduate students with working knowledge of Python. For the teacher we provide lecture slides of the first, second and fourth assignment part; we provide a code framework that includes instructions for students that guide them to finish the implementation with a focus on perspective projection and path planning; and sample solutions for the teacher.

Topics Perspective projection, training/validation and testing in machine learning, buffered voronoi cells, path planning, motion planning, and multi-agent/multi-robot coordination.
Audience Graduate students who took an “introduction to artificial intelligence” or more advanced artificial intelligence classes.
Difficulty Easy to moderate difficulty. Distance estimation and path planning can be both finished within 2 hours. The deployment of the combined version of path planning based on distance estimation on the lego-robot platform will take additional 2 hours (without the hardware setup). This assignment can be made significantly harder (and longer) if less code is provided.
Strengths The project includes teacher slides providing an introduction to important concepts of distance estimation, single robot operation, and multi-robot collision avoidance. We provide instructions and code to deploy and test developed code on a lego robot platform. Lego robots provide an easy accessible way towards developing a working system in real world with overseable amount of efford.
Weaknesses The project requires working knowledge of Python 3. Students only have to fill in gaps in the provided source code, which might prevent them from understanding and building connections between individual parts. The fourth part of this assignment "Analyzing natural collective behavior" requires an experimental setup with animals. While observing natural behavior might be a valuable part in this context, its realization might not be feasible in many cases. We provide video material together with teacher instructions which could be used instead.
Variants
  • Variant A - full project:
    • Distance estimation via vision based on prerecorded data.
    • Path planning based on sensed positions of neighboring agents in simulation.
    • Integration of distance estimation and path planning on many mobile robots (lego robots).
    • Biometric robots interacting with live animals.
  • Variant B - synthetic part (testing on lego robot platform):
    • Distance estimation via vision based on prerecorded data.
    • Path planning based on sensed positions of neighboring agents in simulation.
    • Integration of distance estimation and path planning on many mobile robots (lego robots).
  • Variant C - synthetic part (offline and in simulation):
    • Distance estimation via vision based on prerecorded data.
    • Path planning based on sensed positions of neighboring agents in simulation.
  • Others:
    • The project can be made significantly harder by providing less code and letting students build their implementation from scratch.
    • The project can also be used for demonstration purposes, by providing the full code and let students change hyperparameters or problem settings.
Dependencies Requirements (assignment variant A, B and C):
  • Computer with Python 3 (Mac/Windows/Linux tested)
Optional Requirements for building two Lego robots (assignment variant A and B):
or Lego Mindstroms comprises sufficient bricks and motors for two robots. This kit is depreciated (end 2022), however all required bricks are still available in the Lego education spike prime set. The extension kit will be sufficient as the Lego specific control box (the hub) is not used in this project. A comprehensive comparison of the two kits can be found here. Expected costs for building two robots are about 600$.

Teaching Material


Overview
An overview over the full assignment and the individual goals to be achieved for each sub-assignment. Relevant references and further readings about content covered in this assignment are linked.


Slides
Lecture slides for the teacher for a helpful and motivating lecture prior to the project. Each assignment part comes with an indroductury lecture.


Code Archive
Commented Python code with missing blocks to be used with the task assignments. Documented code is provided for the synthetic part of the project:

  • Distance estimation.
  • Collision Avoidance.
  • Integration of distance estimation and collision avoidance on many mobile robots (lego robots).


Data
This data is required for the first part of the assignment - distance estimation via vision. It contains images from the ego-centric perspective of a lego robot. The images are annotated with bounding box detections and distances of all neighboring robots that are in the field of view. The data was recorded in a motion capture system.

  • Data to train an object detector for lego robot bounding box detection.
  • Data for training, validation and testing learned distances to neighboring lego robots.
  • Pretrained model (YOLOv5 small/nano) for lego robot detection.


Video Material (Analyzing natural collective behavior)
This Material contains:

  • a video recording of a biomimetric robotic fish that interacts with a live fish,
  • tracks of both the live fish and the robot,
  • assignment instructions and
  • reference literature.
This material can be used as a replacement of the live animal experiments, in case those are not feasible.


Teacher material
If you are a teacher, contact Pia Bideau to obtain the Code archive with the working assignment solution.