Skip to content

Semantic segmentation and edge extractionđź”—

More Info

Learn about deep learning-based semantic segmentation and many other topics in interactive online courses at our MVTec Academy.

With semantic segmentation, we assign each pixel of the input image to a class using a deep learning network.

Figure 1. Possible example for semantic segmentation: Every pixel of the input image is assigned to a class, but neither the three different instances of the class “apple” nor the two different instances of the class “orange” are distinguished objects.

The result of semantic segmentation is an output image, in which the pixel value signifies the assigned class of the corresponding pixel in the input image. Thus, in HALCON the output image is of the same size as the input image.

For general DL networks, the deeper feature maps, representing more complex features, are usually smaller than the input image (see About the network and the training process on the Deep learning overview page).
To obtain an output of the same size as the input, HALCON uses segmentation networks with two components: an encoder and a decoder. The encoder determines features of the input image as done, for example, for deep-learning-based classification. As this information is “encoded” in a compressed format, the decoder is needed to reconstruct the information to the desired outcome, which, in this case, is the assignment of each pixel to a class.

Note

As pixels are classified, overlapping instances of the same class are not distinguished as distinct.

Edge extraction is a special case of semantic segmentation, where the model is trained to distinguish two classes: “edge” and “background”.

Figure 2. Possible example for edge extraction: A special case of semantic segmentation, where every pixel of the input image is assigned to one of the two classes: “edge” and “background”.

More Info

About datađź”—

We distinguish between data used for training and evaluation and data used for inference. While the latter consists of bare images, the former consist of images with their information and ground truth annotations. You provide the information defining, for each pixel, which class it belongs to (via the segmentation_image; see below for further explanation).

As a basic concept, the model handles data over dictionaries, meaning it receives the input data over a dictionary DLSample and returns the dictionaries DLResult and DLTrainResult. For more information about data handling, see HALCON DL model.

Data for training and evaluation

The training data is used to train a network for your specific task. The dataset consists of images and corresponding information. They have to be provided in a way the model can process them. For more information about the image requirements, see Images below.
The information about the images and their ground truth annotations is provided via the dictionary DLDataset. Additionally, a segmentation_image exists for each sample, which defines the class for every pixel.

  • Classes

    The different classes are the sets or categories differentiated by the network. They are defined in the dictionary DLDataset and are passed to the model via the operator set_dl_model_param.

    In semantic segmentation, two special cases are important: the class “background” and classes declared as “ignore”:

    “background” class
    The networks treats the background class like any other class. It is not necessary to have a background class either. However, if you have different classes in your dataset you are not interested in although they have to be learned by the network, you can set them as “background”. As a result, the class background will be more diverse. For more information, see the procedure preprocess_dl_samples.
    “ignore” classes
    You can declare one or multiple classes as “ignore”. Pixels assigned to an “ignore” class are ignored by the loss as well as for all measures and evaluations. For more information about the loss, see About the network and the training process on the Deep learning overview page.
    The network does not classify any pixel into a class declared as “ignore”. Further, the pixels labeled to belong to such a class will be classified by the network like every other pixel into a non-“ignore” class.
    In the example given in Figure 3 below, this means the network will classify also the pixels of the class “border”, but it will not classify any pixel into the class “border”. You can declare a class as “ignore” using the parameter 'ignore_class_ids' of set_dl_model_param.

    In edge extraction scenarios, only two classes are distinguished: “edge” and “background”. The “edge” class is labeled just like any other class. Thus, only one class is labeled and this class is called “edge”.

  • DLDataset

    Keys in DLDataset applicable only to semantic segmentation concern the segmentation_image (see the entry below). Via the keys segmentation_dir and segmentation_file_name, you provide the information how they are named and where they are saved.

  • segmentation_image

    For the network to learn what members of different classes look like, you specify, for each pixel of every image in the training dataset, to which class it belongs. This is done by storing the class encoded as pixel value in the corresponding segmentation_image for every pixel of the input image. These annotations are the ground truth annotations.

     

    Figure 3. Schema of segmentation_image. For visibility, gray values are used to represent numbers.
    Input image
    Corresponding segmentation_image providing the class annotations – 0: background (white), 1: orange, 2: lemon, 3: apple, 4: border (black) as a separate class to be declared as “ignore”

You also want enough training data to split it into three subsets: for training, for validation, and for testing the network. These subsets are preferably independent and identically distributed (see also About data splitting). For the splitting, you can use the procedure split_dl_data_set.

Images

Regardless of the application, the model poses requirements on the images, such as the dimensions, the gray value range, and the type. The specific values depend on the network itself and can be queried with get_dl_model_param. For more information about specific values of different models, see read_dl_model.

To fulfill these requirements, you may have to preprocess your images. Standard preprocessing is implemented in the procedure preprocess_dl_dataset and, for a sample, in preprocess_dl_samples. In case of custom preprocessing, these procedures offer guidance on the implementation.

Model output

The network output depends on the task:

  • Training

    As output, the operator train_dl_model_batch will return a dictionary DLTrainResult with the current value of the total loss as well as values for all other losses included in your model.

  • Inference and evaluation

    As output, the network will return a dictionary DLResult for every sample. For semantic segmentation, this dictionary will include, for each input image, the handles of the two following images:

    segmentation_image
    An image where each pixel has a value corresponding to the class its corresponding pixel has been assigned to (see the illustration below).
    segmentation_confidence
    An image, where each pixel has the confidence value out of the classification of the according pixel in the input image (see Figure 4 below).
     

    Figure 4. Different data images. For visibility, gray values are used to represent numbers.
    segmentation_image: Also pixels of classes declared as “ignore” (see Figure 3) are classified
    segmentation_confidence

Model parameters and hyperparametersđź”—

Apart from the general DL hyperparameters explained on the Deep Learning page, there are further hyperparameters relevant for semantic segmentation: the “class weights” (see below).

For a semantic segmentation model, the model parameters as well as the hyperparameters (with the exception of “class weights”) are set using set_dl_model_param. The model parameters are explained in more detail in get_dl_model_param.

Due to large memory usage, typically only small batch sizes are possible for a training. As a consequence, training is rather slow, and we advise using a higher momentum than, for example, for classification. The HDevelop example segment_pill_defects_deep_learning_2_train.hdev provides good initial parameter values for the training of a segmentation network in HALCON.

Class weights

With the hyperparameter “class weights,” you can assign each class the weight its pixels receive during training. Giving the unique classes a different weight, it is possible to force the network to learn the classes with different importance. This is useful when a class dominates the images.
For example, in defect detection, defects occupy only a small fraction of an image. In such a case, a network classifying every pixel as background (thus, “not defect”) would achieve good loss results in general. Assigning different weights to the distinct classes helps to re-balance the distribution. In short, you can focus the loss to train especially on those pixels you determine to be important.

The network obtains these weights via weight_image, an image created for every training sample. In weight_image, every pixel value corresponds to the weight the corresponding pixel of the input image receives during training.
You can create these images with the help of the following two procedures:

  • calculate_dl_segmentation_class_weights helps you to create the class weights. The procedure uses the concept of inverse class frequency weights.

  • gen_dl_segmentation_weight_images uses the class weights and generates the weight_image.

This step has to be done before the training. Usually, it is done during the preprocessing and therefore is part of the procedure preprocess_dl_dataset.

Note

  • The “class weights” hyperparameter is referred to as class_weights or ClassWeights within procedures.
  • When assigning a specific part of the image a weight of 0.0, these pixels do not contribute to the loss. For more information about the loss, see About the network and the training process.

Figure 5 below shows what such an image with different weights looks like.

 

Figure 5. Schema for weight_image. For visibility, gray values are used to represent numbers.
segmentation_image defining the classes for every pixel within the image – 0: background (white), 1: orange, 2: lemon, 3: apple, 4: border (black), declared as “ignore”.
Corresponding weight_image providing class weights – background: 1.0, orange: 30.0, lemon: 75.0. Pixels of classes declared as “ignore” (here: class border) will be ignored and get the weight 0.0.

Evaluation measures for the data from semantic segmentationđź”—

For semantic segmentation, the following evaluation measures are supported in HALCON.

Note

  • To be able to compute such a measure for an image, the related ground truth information is needed.
  • All the measure values explained below for a single image (for example, mean_iou) can also be calculated for an arbitrary number of images. For this, imagine a single large image formed by the ensemble of output images, for which the measure is computed.
  • All pixels of a class declared as “ignore” are ignored for the computation of the measures.
pixel_accuracy

The pixel accuracy is the ratio of all pixels that have been predicted with the correct class label to the total number of pixels.

 

Figure 6. Visual example of the pixel_accuracy:
segmentation_image defining the ground truth class for each pixel. Pixels of a class declared as “ignore” are drawn in black.
Output image – pixels of classes declared as “ignore” are classified.
Pixel accuracy as the ratio of the total orange areas. Pixels labeled as part of a class declared as “ignore” are ignored.

class_pixel_accuracy

The per-class pixel accuracy considers only pixels of a single class. It is defined as the ratio of correctly predicted pixels to the total number of pixels labeled with this class.

If a class does not occur, it receives a class_pixel_accuracy value of -1 and does not contribute to the average value mean_accuracy.

mean_accuracy
The mean accuracy is defined as the averaged per-class pixel accuracy (class_pixel_accuracy) of all occurring classes.
class_iou

The per-class intersection over union (IoU) gives, for a specific class, the ratio of correctly predicted pixels to the union of annotated and predicted pixels. Visually, this is the ratio of the intersection to the union of the areas; see Figure 7 below.

If a class does not occur, it receives a class_iou value of -1 and does not contribute to the mean_iou.

 

Figure 7. Visual example of the per-class IoU (class_iou), here: class “apple” only.
segmentation_image defining the ground truth class for each pixel. Pixels of a class declared as “ignore” are drawn in black.
Output image, also the pixels of classes declared as “ignore” are classified.
IoU as the ratio of the intersection to the union of the areas of pixels denoted as apple. Pixels labeled as part of a class declared as “ignore” are ignored.

mean_iou
The mean IoU is defined as the averaged per-class intersection over union (class_iou) of all occurring classes. Every occurring class has the same impact on this measure, independent of the number of pixels they contain.
frequency_weighted_iou
As for the mean IoU, the per-class IoU is calculated first. However, the contribution of each occurring class to this measure is weighted by the ratio of pixels belonging to that class. Classes with many pixels can dominate this measure.
pixel_confusion_matrix
For more information about confusion matrices, see Confusion matrix on the Deep learning overview page. It applies to semantic segmentation, as the instances are single pixels.