What is the difference between semantic and instance segmentation?

Semantic segmentation labels every pixel with a class such as road or person but does not separate individuals, while instance segmentation additionally distinguishes each separate object, so two people get different labels.

Why is segmentation considered hard?

What counts as one region depends on the task and is often ambiguous, and natural images have texture, shading, and occlusion that blur the boundaries between objects, so there is rarely a single correct answer.

Image Segmentation

Image segmentation partitions an image into regions that correspond to objects or meaningful parts, assigning each pixel to a segment.

PaperMindでテーマを探す近日公開Find papers & topics

Tools & resources

スライドをダウンロード

Learn & explore

動画近日公開

Definition

Image segmentation is the assignment of each pixel to one of a set of regions or classes so that pixels within a region are coherent and regions correspond to distinct scene structures.

Scope

This topic covers thresholding and region-growing, clustering of pixels in color and feature space, graph-based methods such as normalized cuts, energy-minimization and active-contour formulations, and modern semantic and instance segmentation that label every pixel with an object class.

Core questions

How are pixels grouped into coherent regions?
What criterion distinguishes a good segmentation from a poor one?
How can global region structure be optimized rather than decided pixel by pixel?
How is segmentation extended to label pixels with object categories?

Key concepts

Thresholding and region growing
Clustering in feature space
Normalized cuts
Energy minimization and graph cuts
Active contours
Semantic and instance segmentation

Key theories

Graph-based segmentation: Treating pixels as graph nodes with edges weighted by similarity, the normalized cut criterion partitions the image to maximize within-region coherence relative to between-region dissimilarity, yielding globally balanced segmentations.
Energy minimization for labeling: Segmentation can be cast as minimizing an energy that rewards assigning pixels to fitting regions while penalizing label changes between similar neighbors, solvable efficiently by graph cuts and related optimization.

Clinical relevance

Segmentation is central to medical image analysis such as delineating organs and tumors, to scene understanding in autonomous driving, to remote-sensing land-cover mapping, and to image editing and object extraction.

History

Early segmentation relied on thresholding and region merging; graph-based methods such as normalized cuts in 2000 and energy-minimization approaches advanced the field, and deep convolutional networks later achieved dense semantic and instance segmentation.

Key figures

Jianbo Shi
Jitendra Malik

Seminal works

shi2000
szeliski2022

Frequently asked questions

What is the difference between semantic and instance segmentation?: Semantic segmentation labels every pixel with a class such as road or person but does not separate individuals, while instance segmentation additionally distinguishes each separate object, so two people get different labels.
Why is segmentation considered hard?: What counts as one region depends on the task and is often ambiguous, and natural images have texture, shading, and occlusion that blur the boundaries between objects, so there is rarely a single correct answer.