What Is Semantic Segmentation and How Does It Work?

Key takeaways

• Semantic segmentation is one of three subcategories of image segmentation, and it identifies, classifies, and labels each pixel within a digital image.

• Semantic segmentation models work by creating color-coded maps of images passed through a neural network architecture, helping computer systems distinguish objects in an image and understand their relationships.

• Semantic segmentation in computer vision is used in various industries and fields, including photography, social media, medical imaging, agriculture, and automotive.

To label data for semantic segmentation, assign a class label, outline objects with a bounding box, and group pixels using a segmentation mask. Continue reading to learn what semantic segmentation is, how it works, why it matters, how to do it yourself, and more about semantic segmentation models and datasets.

If you’re ready to strengthen your skills in this field, enroll in the Image Processing for Engineering and Science Specialization from MathWorks. In as little as four weeks, you can learn about algorithms, spatial analysis, computer vision, anomaly detection, and more.

What is semantic segmentation in computer vision?

Semantic segmentation is a computer vision technique that identifies, classifies, and labels each pixel within a digital image. Pixels are labeled according to the semantic features they share, such as color or placement. Semantic segmentation helps computer systems distinguish between objects in an image and understand their relationships. It’s one of three subcategories of image segmentation, alongside instance segmentation and panoptic segmentation.

If you’ve ever used a filter on Instagram or TikTok, you’ve employed semantic segmentation in the palm of your hand. Although this computer vision technique goes far beyond digital makeup and mustaches, you can find it hard at work in hospitals, farms, and even in Teslas.

Instance segmentation vs. semantic segmentation

While semantic segmentation simply annotates, or assigns a class to, each pixel in an image, instance segmentation goes further by creating a pixel-by-pixel boundary around each object, which provides the precise shape and area of each object. Instance segmentation provides greater detail so that you can distinguish “things” (countable objects), such as cars or people, and “stuff” (classes of uncountable objects) that vary in shape, such as roads or mountains. Semantic segmentation, on the other hand, groups closely adjacent and overlapping objects into a single label.

Semantic segmentation vs. panoptic segmentation

The panoptic segmentation process categorizes each pixel in an image using both “things” and “stuff,” whereas semantic segmentation focuses primarily on “stuff” and instance segmentation on “things.” Panoptic segmentation is a hybrid computer vision technique that combines semantic and instance segmentation, providing a unified, holistic view of a scene. It identifies and labels every pixel in an image with both a class label (stuff) and an instance identification (things).

Read more: Unlocking the Potential of Computer Vision Models

How does semantic image segmentation work?

Semantic segmentation models work by processing input images using a neural network architecture that groups related pixels into segmentation masks. During this process, each pixel is color-coded to represent a different class label. These defined spatial features help computers identify boundaries between objects and distinguish the foreground from the background, enabling them to produce a segmentation mask for each distinct portion of the image. Together, the segmentation masks result in a segmentation map that represents the original image.

Semantic segmentation models build on the benefits of applying a convolutional neural network (CNN) to image classification. CNNs enabled models to extract individual features and deduce which objects they represent. Before the introduction of CNNs, image classification models were often trained to recognize objects in images using labeled example photos, relying on raw pixel data that is prone to uncorrectable fluctuations due to variations in camera focus, lighting, and angle.

How to label images for semantic segmentation: The process

You can label images for semantic segmentation by assigning a class label, outlining with a bounding box, and grouping pixels using a segmentation mask.

1. Classification: Pixels in an image are assigned a class label representing particular objects.

2. Localization: Objects are outlined with a bounding box. A bounding box is a line drawn around the perimeter of an object.

3. Segmentation: In the localized image, pixels are grouped using a segmentation mask. A segmentation mask reduces noise by separating one portion of an image from the rest. One way to visualize segmentation masking is to imagine sliding a piece of black construction paper with a hole cut out over an image to isolate specific portions.

Semantic segmentation use cases

Semantic segmentation has many uses across industries, including agriculture, medical imaging, photography, social media, and automation. Because it can process information with pixel-level precision, semantic segmentation can help intelligent systems better understand spatial relationships and make important judgments, distinguishing it from other computer vision tasks. Consider some of the real-world applications of semantic segmentation:

• Photography and social media filters: All commonly used camera effects and filters on social media applications like Instagram and TikTok rely on semantic segmentation. It identifies the placement of eyes to apply sunglasses and allows cameras to switch between landscape and portrait formats.

• Medical imaging analyses: AI segmentation models trained on medical imagery can perform automated analysis to measure and detect anomalies at the pixel level. By highlighting and mapping anatomical features, segmentation enhances visualization, enabling more precise identification of tumors and other irregularities.

• Agriculture: Farmers employ AI and semantic segmentation to automate maintenance and manage the health of their crops. Computer vision technology helps farmers quickly detect at-risk portions of their fields to eradicate pests or contain infections.

• Self-driving cars: Autonomous vehicles rely heavily on semantic segmentation to identify obstacles, analyze road conditions, and map surroundings.

Does Tesla use semantic segmentation?

Yes, Tesla’s camera networks analyze images to perform semantic segmentation, object detection, and depth estimation. Ultimately, this helps the company train deep neural networks for problems related to self-driving cars, including perception and control.

How to do semantic segmentation

You can use many different tools and models, including open-source data sets and pretrained semantic segmentation models, to perform semantic segmentation. The technique you choose will depend on your specific application and the characteristics of the images you want to segment. Organizations such as MathWorks provide step-by-step guidance if you want to try to perform semantic segmentation on a test image using either a pretrained semantic segmentation model or a model you trained yourself. If you’re looking for a broader learning experience, you might consider the MathWorks Computer Vision Engineer Professional Certificate, which offers skill development in various areas of computer vision.

Semantic segmentation data sets

Data sets for semantic segmentation are typically large and complex. The more diverse the labels in the data set, the better the model can learn. The following are examples of a few commonly used segmentation data sets:

• Common Objects in Context (COCO): COCO is a large-scale data set used for captioning, key-point detection, object detection, and segmentation. It includes over 120,000 images with a wide variety of annotations, all refined by community feedback.

• Cityscapes Data Set: The central focus of the Cityscapes Data Set is the semantic understanding of city and street scenes. It includes 30 different classes, 25,000 annotated images, dense semantic segmentation, and instance segmentation for people and vehicles.

• ScanNet: ScanNet is an RGB-D video data set with 3D data. It comprises 2.5 million annotated reconstructions of indoor scenes from more than 1,500 scans with semantic segmentation annotations and surface reconstructions.

Semantic segmentation models

Today’s semantic segmentation models typically rely on deep learning algorithms with robust architectures. These methods, which often use CNNs, excel at feature extraction and offer superior performance and adaptability. The list below includes a few popular segmentation models:

• Pyramid Scene Parsing Network (PSPNet): PSPNet uses a pyramid parsing module to discern multi-level features, providing a more comprehensive context of an image. It’s capable of processing global and local information.

• Fully Convolutional Network (FCN): FCNs, which use an encoder-decoder structure, build on traditional CNN architecture to extract more information about an image. They offer streamlined training, support for input images of any size, and enable faster inference.

• U-net: U-net is another CNN-based model that uses an encoder-decoder architecture. It uses skip and shortcut connections to reduce information loss, making it an accurate option, especially with limited data.

• DeepLab: DeepLab models build on the original FCN and use atrous convolution, yielding clearer, more accurate results. If you have a complex or tricky segmentation project, you can use DeepLab v3+, which provides precise boundary delineation and captures multi-scale features.

Keep up with trends and job opportunities in AI

Subscribe to our weekly LinkedIn newsletter, Career Chat, for industry updates, tips, and trends. Then, explore free artificial intelligence resources to optimize your professional growth:

• Watch on YouTube: Machine Learning in Real Life: From Spotify to Healthcare

• Hear from an expert: The AI Advantage: 9 Questions with UC Davis AI Instructor Sadie St. Lawrence

• Learn the terminology: Artificial Intelligence Glossary: Learn AI Vocabulary

With Coursera Plus, you can learn and earn credentials at your own pace from over 350 leading companies and universities. With a monthly or annual subscription, you’ll gain access to over 10,000 programs. Just check the course page to confirm your selection is included.