Computer Vision Pipelines

The Problem

In 2023–24, a team prop was placed on one of three spike marks at random, and the robot had to find it to choose its autonomous path. Teams designed their own props, but they had to be red or blue, which are everywhere in a competition gym: shirts, bumpers, tape, other robots. The lighting also changes from venue to venue. A simple color mask would find a lot of things that weren’t the prop.

What I Built

I solved part of the problem before writing any code, by designing the prop to be easy to pick out: a solid-colored cube. A cube looks square from the robot’s camera, and most red or blue clutter doesn’t.

The pipeline converts each frame to HSV and masks it for the prop’s color. It then rescales the masked pixels so their average saturation is always the same before applying a second, stricter threshold, which keeps the mask steady when the lighting changes. Every contour gets a bounding box, any box that isn’t close to square is thrown out, and the largest box left is the prop. The region it sits in picks the path.

Extending to Game Pieces

Once the robot could find the prop, I wanted it to score more in autonomous, which meant finding game pieces too. A second pipeline masks for the piece’s color and takes the largest contour. The camera’s field of view gives the angle to it, and the piece’s known width gives the distance:

x is the piece’s center in the image, and the focal length, in pixels, comes from a calibration program that measures a piece at a known distance.

Scoring also meant lining up on the backdrop, which has AprilTags on it. The robot uses the tag’s pose to set three speeds at once, each proportional to how far off it is: