OpenCV: Computer Vision Building Blocks

OpenCV is the toolbox that surrounds computer vision models: reading images and video, resizing, colour conversion, filtering, geometric transforms, contour finding, tracking, camera calibration, and drawing results. It predates the deep learning era and remains in nearly every vision system in production, because the parts of a pipeline it handles never went away.

Where it sits in a real pipeline

A deployed vision system is mostly not the model. Frames come from a camera or a file, get decoded, resized, colour-converted, and normalised; the model runs on a tensor; then raw outputs are filtered, mapped back to original image coordinates, and drawn or encoded. OpenCV owns everything except the middle box.

import cv2

capture = cv2.VideoCapture(0)
while True:
    ok, frame = capture.read()          # BGR, uint8, shape (H, W, 3)
    if not ok:
        break

    rgb = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
    resized = cv2.resize(rgb, (640, 640), interpolation=cv2.INTER_AREA)
    detections = model(resized)          # the only learned step

    for box, label in detections:
        x1, y1, x2, y2 = scale_to_original(box, frame.shape)
        cv2.rectangle(frame, (x1, y1), (x2, y2), (0, 255, 0), 2)
    cv2.imshow("out", frame)

Two details in that loop matter more than they look. read() returns a success flag that must be checked — cameras drop frames and files end, and ignoring it gives you a None that fails several lines later with a confusing error. And INTER_AREA is the correct interpolation for downscaling; the default bilinear introduces aliasing that measurably degrades detection on small objects.

Classical or learned

The dividing line is whether the thing you are looking for can be described geometrically or photometrically, or only recognised by example.

Classical wins when the target is defined: a fiducial marker, a barcode, a known template, motion between frames, a colour under controlled lighting, a straight edge, a document to deskew. These run in microseconds, need no training data, no GPU, and behave identically every time. Perspective correction on a scanned form is four points and a matrix, not a model.

Learned methods are needed when the category is only definable by example — recognising a species, a defect, a person, or an object under varying lighting and pose. Any attempt to hand-tune thresholds for these produces a brittle result that works in the room where it was calibrated.

Most systems use both, and the classical part usually makes the learned part cheaper: detect motion first and only run the model on frames where something changed, and you cut inference cost by an order of magnitude on a static camera.

The bugs everyone hits

Four OpenCV conventions produce wrong results without raising errors.

BGR, not RGB. imread and VideoCapture give you blue-green-red order. Every deep learning model and every plotting library expects RGB. Feeding BGR to a model does not crash — accuracy just drops, and colour-sensitive tasks fail outright. Convert at the boundary.

Two coordinate orders. image.shape is (height, width, channels). cv2.resize takes (width, height). Points are (x, y), arrays index [y, x]. A square image hides this bug; a non-square one surfaces it immediately.

Slices are views. roi = image[100:200, 100:200] shares memory with the original, so writing to it modifies the source. Use .copy() when you need independence.

uint8 arithmetic wraps. Adding to a uint8 image overflows past 255 back to zero, producing bright regions that turn black. cv2.add saturates correctly; the NumPy + operator does not.

The operations worth knowing by name

A small set of functions covers most of what a pipeline needs, and knowing they exist prevents a lot of reinvention.

For geometry: warpAffine and warpPerspective apply a transform matrix, which is how you deskew a document, correct a camera angle, or align two images. getPerspectiveTransform builds the matrix from four corresponding points, and that pair alone solves the entire "photograph of a form, want it flat" problem.

For isolating things: threshold and adaptiveThreshold turn greyscale into binary, with the adaptive version handling uneven lighting that fixed thresholds cannot. findContours then gives you the outlines of connected regions, with area, bounding box, and centroid available for each. Morphological operations — erode, dilate, and the morphologyEx combinations — clean up the speckle and gaps that thresholding leaves behind.

For motion: background subtraction identifies what changed against an accumulated model of the static scene, and optical flow estimates where pixels moved between frames. Both are cheap enough to run on every frame as a gate in front of an expensive detector.

For colour: converting to HSV separates hue from brightness, which makes colour-based selection far more robust to lighting change than working in RGB. This is the difference between a colour filter that works in one room and one that works generally.

Real-time on a budget

Frame rate problems are usually solvable without touching the model. Downscale before processing, since detection rarely needs full resolution. Run the detector every nth frame and use a cheap tracker in between. Skip frames rather than queueing them, because a backlog turns into growing latency. Convert colour once per frame rather than per operation, and avoid per-pixel Python loops entirely — a vectorised NumPy expression or a built-in OpenCV call is typically a hundred times faster than iterating.

Common mistakes

Forgetting the BGR conversion. No error, degraded accuracy.

Mixing (w, h) and (h, w). Invisible on square images, wrong everywhere else.

Hand-tuning thresholds for a task that needs learning. Works in the calibration room and nowhere else.

Ignoring the read() success flag. Turns a dropped frame into a confusing crash further down.