Getting Set with OpenCV

The assignment page: Getting Set with OpenCV

Summary	This assignment asks students to build a program that plays the game of Set. Set is a game of visual perception and reasoning: in it players look through a group of face-up cards that contain a variety of shapes. Each of those shapes has four attributes: color, texture, form, and number. The game's goal is to find a trio of cards among which these attributes have all identical or all distinct values. As an example, the figure below shows nine of the 81 Set cards: Within this image the central card, the "east" card, and the "northeast" card form a set. We can verify this through the four attributes: The color of each of those three cards is different: blue, green, and red, respectively. The texture of each of those three cards is the same: they are all filled shapes. The form of each of those three cards is different: oval, squiggle, and diamond, respectively. The number of each of those three cards is different: one, three, and two shapes, respectively. In contrast, the three cards in the leftmost column do not form a set. Although the numbers and colors of shapes are all different and the forms are all the same (diamond), the fill-texture is neither all different nor all the same. Two of those cards have a solid fill and one is outline-only. This assignment challenges students to write a Set-playing program. Broader goals    The goals of this assignment are twofold: first, it stands alone as an introduction to image-processing. As we use it, however, it is more important as a motivating introduction to the OpenCV library, the largest and most ubiquitous software foundation for real-time vision processing. Many courses and research projects build atop that library. Yet for all of its strengths, OpenCV continues to be a challenge to newcomers. This assignment offers students a relatively smooth and task-directed path through which undergraduates can get Set with OpenCV before applying it in more advanced investigations.
Topics	This assignment provides an introduction to computer vision. In particular, it reinforces color segmentation, morphology and binary image processing, and it helps students get comfortable with real-time pixel processing and OpenCV's interface to it. On the other hand, this assignment does not reinforce game-search. In fact, one of the realizations that students often articulate is that humans and computers have very different difficulties with the game. Humans identify the cards effortlessly and struggle to find a set of three that meet the constraints. For a program (or, at least, the programmer) the difficult part is determining the shapes. Identifying the triples of cards that share or differ in all attributes turns out to be much more straightforward to specify computationally.
Audience	The intended audience for this assignment is undergraduates with experience in C or C++. In particular, no AI background is needed beyond the self-contained lecture for introducing and explaining the assignment.
Difficulty	This assignment is considered challenging, but doable, in general. Its difficulty can be varied by providing less - or more - scaffolding for the OpenCV library of image processing routines provided as starter code. Students in our AI Robotics course are given two weeks to complete both this introductory assignment and to integrate vision into a physical system they are working on.
Strengths	Set is a game many students like. They often enjoy building a game-player whose fundamental capability (vision) is very different than most or all of the AI agents/game players they know. In addition, this assignment is designed as a lead-in to further work, perhaps open-ended, using image processing. It provides motivation for some of the learning curve required to use the ubiquitous, but still idiosyncratic, OpenCV library.
Weaknesses	As noted, this assignment is best used as a stepping stone for student proejcts involving real-time vision. If an instructor wanted to use it as a self-contained introduction to image processing, the messiness of OpenCV - whose researchware origins still show - should be abstracted away. This could be done using OpenCV's Python interface, for example. However, because we do use this assignment as a gateway to undergraduate-accessible real-time vision, we have not yet created the scaffolding needed for an easily-deployed, self-contained Python version of this project.
Dependencies	[Concepts] The image processing needed is elementary enough that 1/2 lecture suffices, though we add context to fill out 1 75-minute lecture. [Languages] OpenCV offers C and C++ interfaces. Python support is theoretically available, but we have not evaluated it. [Operating Systems] We run identical code under Windows and Mac OS C. Linux support is theoretically available (and likely works identically, too), but we have not tried it.
Variants	This assignment's primary purpose lies in the variants that students can build on top of it. In the past, our students have built a wide variety of robotic vision projects atop their Set experience: real-time segmentations of campus paths, using OpenCV's watershed algorithm, that guided an autonomous outdoor vehicle a neural-network-based estimation of range-from-texture built atop OpenCV's capturing capabilities routines for systems that play physical games such as tag or the NES's Duck Hunt
Links to resources and materials	A link to a copy of the Getting Set with OpenCV assignment page. A link to the lecture slides we use to motivate the assignment. A link to the AI robotics course in which the assignment is a part. A link to our Getting Set with OpenCV wiki page with hints on installing, setting up, and using that library. A link to our starting files. In this zip archive both a Visual Studio 2008 "solution" and an XCode project file share the same source code. It runs identically under Windows and Mac OS X.