DETR (Detection Transformer)

Imagine looking at a busy street scene and being asked to point out every car, pedestrian, bicycle, and traffic light—without checking thousands of tiny candidate boxes first. DETR, short for Detection Transformer, approaches object detection this way: it treats detection as finding a set of objects directly, rather than as filtering a large collection of proposed regions.

How DETR finds objects
DETR begins with a convolutional backbone, such as ResNet, which turns an image into a compact grid of visual features. A transformer encoder lets positions in that grid exchange information, so a pixel region containing a wheel can be interpreted in the context of the whole vehicle. A transformer decoder then uses a fixed number of learned object queries: think of them as slots asking, “Is there an object here, and if so, what is it?” Each query produces:

• a class label, including a special no object class;
• a bounding box, represented by its center position, width, and height.

Learning without duplicate boxes
Traditional detectors use anchors or dense candidate boxes, then rely on non-maximum suppression (NMS) to remove duplicates. DETR avoids both. During training, it uses Hungarian matching, an assignment algorithm that pairs each real object with one predicted query. A prediction is judged by its class accuracy and box quality, including overlap measures such as generalized IoU. Because one ground-truth object is matched to only one query, the model learns to make one clean prediction per object.

Why it matters in practice
This design makes the detection pipeline simpler and more globally aware: a detector can use the entire scene to distinguish, for example, two overlapping people in a video frame or separate instruments in a medical image. The original DETR trained slowly and struggled with small objects because it attended broadly across the image. Deformable DETR improved this by attending to a small set of useful feature locations at multiple scales. DETR’s set-based prediction idea has also influenced modern models for panoptic and instance segmentation.