Optical Flow

When you watch a video, you effortlessly see which parts of the scene are moving and in what direction. Optical flow gives a computer a version of that ability: it estimates the apparent motion of pixels between consecutive video frames.

A map of apparent motion
Optical flow is represented as a vector field. For each tracked pixel or image region, a small arrow describes horizontal and vertical displacement: its direction shows where the appearance moved, and its length shows how far it moved between frames. It measures motion in the image plane, not necessarily the true 3D motion of an object. A car driving toward a camera, for example, can appear to expand outward in every direction even if it is travelling straight ahead.

How it is estimated
Classical methods begin with the brightness constancy assumption: a point keeps roughly the same visual intensity as it moves from one frame to the next. They search for a displacement that best aligns nearby image patches. Common approaches include:

Lucas–Kanade optical flow, which estimates motion for selected feature points using a small local neighborhood.
Horn–Schunck, which produces dense flow across the image while encouraging neighboring pixels to move smoothly.
Farnebäck flow, a widely used dense method available as cv.calcOpticalFlowFarneback in OpenCV.

A challenge called the aperture problem means an edge alone reveals only motion across the edge, not motion along it. Texture, corners, and larger neighborhoods resolve this ambiguity.

Why vision systems use it
Optical flow supports video tracking, action recognition, camera-motion estimation, and autonomous-vehicle perception. It can distinguish a stationary background from moving pedestrians, help stabilize shaky footage, or flag an item moving incorrectly on a production line. In medical video, it can describe tissue motion; in sports footage, it can help track players or a ball. Without motion estimates, two nearly identical video frames provide little direct evidence about what is changing.

Optical flow is a dense vector field that estimates the apparent pixel motion between consecutive video frames, giving each pixel a direction and speed. It captures motion caused by moving objects, camera movement, or both. Optical flow is essential for motion tracking, video stabilization, action recognition, frame interpolation, and estimating scene dynamics from video.

Imagine watching leaves drift across a pond. Even without measuring them, your eyes can tell which way each leaf is moving and roughly how fast. Optical flow gives a computer a similar sense of motion in video.

It describes the apparent movement of pixels—the tiny colored dots that make up an image—from one video frame to the next. A car moving right creates flow pointing right; a person walking forward may create flow that spreads outward as they get closer.

This helps AI notice motion, track objects, estimate where a camera is moving, and understand actions such as waving, running, or falling. It is especially useful when motion itself matters, not just what appears in a single picture.