|Summary||In this assignment students use Image Based Visual Servoing to program a mobile robot to park in front of solid-color objects, like orange cones. The control of the robot is completely based on information extracted from a standard monocular camera. The assignment is structured in three parts: getting realtime images from the robot, blob detection, and closed loop porportional control. Students leverage open source software tools like OpenCV and ROS to implement this reactive planner.|
|Topics||robotics, linear feedback control systems, and computer vision|
|Audience||This assignment has been tailored from advanced highschool students to junior and senior level undergraduate students in Engineering and Computer Science.|
|Difficulty||The difficulty of the assignment is highly scalable based on the environment the robot encounters. For example, the complexity of the blob detection for orange cones in plain environments is much easier to implement than harder detections like a picture of a cat in low lighting environments. Additionally, the porportional feedback controller can be upgraded to include integral and derivative terms.|
|Variants||As mentioned in the difficulty section, this assignment can scale for different challenge levels based on the robot's environment. The task can also be modified for line following or added as a low level primitive in a higher level planning task.|
|Blob Detection and Visual Servoing||The collegiate course had one lab for blob detection and visual servoing. The instructions are at a much higher level for the students.|
|OpenCV Cone Detecting Tutorial|| This is a website tutorial we provided on detecting cones. After our students completed the visual servoing lab they had a more challenging problem of swerving through cones. We provided this tutorial to make sure all students could implement visual servoing and other OpenCV algorithms in python and C++.
Code for the tutorial
|Image passthrough Assignment||In this first assignment student create a python program called echo.py . This simple program receives an image from the robot's camera and publishes it to a rostopic. Students learn how to use OpenCV bridge to transition into the next assignment.|
|Blob Detector Assignment||The second assignment goes over a simple blob detection algorithm that segments an image based on a desired color, then finds the largest segmented part of the image to return as the blob. Students are encouraged to review other object detection algorithms from OpenCV.|
|Visual Servo Assignment||In this last assignment students create a proportional controller to try to center the blob in the middle of the image detected by the robot.|
We provided these solution and bag files after students had time to work on the assignment:
The visual servoing lab assignment has been offered in 2 courses, with another expected this Spring semester. We intend to update this site with any changes made to the curriculum. The following are courses we taught this lab exercise in:
MIT Course 6.141J/16.405J: Robotics: Science and Systems
The Robotics: Science and Systems course is a technical elective that teaches robotics to undergraduate students throughout MIT. The course features laboratory exercises for mechanical design, control systems implementation, as well as software development for planning and perception.
We taught this course with the visual servoing lab assignment in semester:
MIT Beaver Works Summer Institute 2016
The RACECAR class was offered for high school students. In the summer of 2016, the course was offered to 46 high-school students coming from across the United States. In a 4-week residential program, the students learned the foundations of robotics in theory lectures, practiced their skills in hands-on laboratory exercises. The class also included lectures on teamwork and collaboration, as well as seminars from established researchers in the field and experienced entrepreneurs. Students demonstrated their learning in a final course challenge. They designed and implemented software for fully autonomous mini race cars.
We are excited to offer the materials for our course open source at Github