Semantic segmentation and edge extractionđź”—
More Info
Learn about deep learning-based semantic segmentation and many other topics in interactive online courses at our MVTec Academy.
With semantic segmentation, we assign each pixel of the input image to a class using a deep learning network.
Figure 1. Possible example for semantic segmentation: Every pixel of the input image is assigned to a class, but neither the three different instances of the class “apple” nor the two different instances of the class “orange” are distinguished objects.
The result of semantic segmentation is an output image, in which the pixel value signifies the assigned class of the corresponding pixel in the input image. Thus, in HALCON the output image is of the same size as the input image.
For general DL networks, the deeper feature maps, representing more complex features, are usually smaller than the input image (see About the network and the training process on the Deep learning overview page).
To obtain an output of the same size as the input, HALCON uses segmentation networks with two components: an encoder and a decoder.
The encoder determines features of the input image as done, for example, for deep-learning-based classification.
As this information is “encoded” in a compressed format, the decoder is needed to reconstruct the information to the desired outcome, which, in this case, is the assignment of each pixel to a class.
Note
As pixels are classified, overlapping instances of the same class are not distinguished as distinct.
Edge extraction is a special case of semantic segmentation, where the model is trained to distinguish two classes: “edge” and “background”.
Figure 2. Possible example for edge extraction: A special case of semantic segmentation, where every pixel of the input image is assigned to one of the two classes: “edge” and “background”.
More Info
- In HALCON, semantic segmentation with deep learning is implemented within the more general deep learning model. For more information about the latter, see Deep learning > HALCON DL model.
- For the specific system requirements to apply deep learning, see Requirements for deep learning and deep-learning-based methods.
About datađź”—
We distinguish between data used for training and evaluation and data used for inference.
While the latter consists of bare images, the former consist of images with their information and ground truth annotations.
You provide the information defining, for each pixel, which class it belongs to (via the segmentation_image; see below for further explanation).
As a basic concept, the model handles data over dictionaries, meaning it receives the input data over a dictionary DLSample and returns the dictionaries DLResult and DLTrainResult.
For more information about data handling, see HALCON DL model.
- Data for training and evaluation
-
The training data is used to train a network for your specific task. The dataset consists of images and corresponding information. They have to be provided in a way the model can process them. For more information about the image requirements, see Images below.
The information about the images and their ground truth annotations is provided via the dictionaryDLDataset. Additionally, asegmentation_imageexists for each sample, which defines the class for every pixel.-
Classes
The different classes are the sets or categories differentiated by the network. They are defined in the dictionary
DLDatasetand are passed to the model via the operatorset_dl_model_param.In semantic segmentation, two special cases are important: the class “background” and classes declared as “ignore”:
- “background” class
- The networks treats the background class like any other class.
It is not necessary to have a background class either.
However, if you have different classes in your dataset you are not interested in although they have to be learned by the network, you can set them as “background”.
As a result, the class background will be more diverse.
For more information, see the procedure
preprocess_dl_samples. - “ignore” classes
- You can declare one or multiple classes as “ignore”. Pixels assigned to an “ignore” class are ignored by the loss as well as for all measures and evaluations. For more information about the loss, see About the network and the training process on the Deep learning overview page.
- The network does not classify any pixel into a class declared as “ignore”.
Further, the pixels labeled to belong to such a class will be classified by the network like every other pixel into a non-“ignore” class.
In the example given in Figure 3 below, this means the network will classify also the pixels of the class “border”, but it will not classify any pixel into the class “border”. You can declare a class as “ignore” using the parameter'ignore_class_ids'ofset_dl_model_param.
In edge extraction scenarios, only two classes are distinguished: “edge” and “background”. The “edge” class is labeled just like any other class. Thus, only one class is labeled and this class is called “edge”.
-
DLDatasetKeys in
DLDatasetapplicable only to semantic segmentation concern thesegmentation_image(see the entry below). Via the keyssegmentation_dirandsegmentation_file_name, you provide the information how they are named and where they are saved. -
segmentation_imageFor the network to learn what members of different classes look like, you specify, for each pixel of every image in the training dataset, to which class it belongs. This is done by storing the class encoded as pixel value in the corresponding
segmentation_imagefor every pixel of the input image. These annotations are the ground truth annotations.Figure 3. Schema of
segmentation_image. For visibility, gray values are used to represent numbers.
Input image
Correspondingsegmentation_imageproviding the class annotations – 0: background (white), 1: orange, 2: lemon, 3: apple, 4: border (black) as a separate class to be declared as “ignore”
You also want enough training data to split it into three subsets: for training, for validation, and for testing the network. These subsets are preferably independent and identically distributed (see also About data splitting). For the splitting, you can use the procedure
split_dl_data_set. -
- Images
-
Regardless of the application, the model poses requirements on the images, such as the dimensions, the gray value range, and the type. The specific values depend on the network itself and can be queried with
get_dl_model_param. For more information about specific values of different models, seeread_dl_model.To fulfill these requirements, you may have to preprocess your images. Standard preprocessing is implemented in the procedure
preprocess_dl_datasetand, for a sample, inpreprocess_dl_samples. In case of custom preprocessing, these procedures offer guidance on the implementation. - Model output
-
The network output depends on the task:
-
Training
As output, the operator
train_dl_model_batchwill return a dictionaryDLTrainResultwith the current value of the total loss as well as values for all other losses included in your model. -
Inference and evaluation
As output, the network will return a dictionary
DLResultfor every sample. For semantic segmentation, this dictionary will include, for each input image, the handles of the two following images:segmentation_image- An image where each pixel has a value corresponding to the class its corresponding pixel has been assigned to (see the illustration below).
segmentation_confidence- An image, where each pixel has the confidence value out of the classification of the according pixel in the input image (see Figure 4 below).
Figure 4. Different data images. For visibility, gray values are used to represent numbers.
segmentation_image: Also pixels of classes declared as “ignore” (see Figure 3) are classified
segmentation_confidence
-
Model parameters and hyperparametersđź”—
Apart from the general DL hyperparameters explained on the Deep Learning page, there are further hyperparameters relevant for semantic segmentation: the “class weights” (see below).
For a semantic segmentation model, the model parameters as well as the hyperparameters (with the exception of “class weights”) are set using
set_dl_model_param.
The model parameters are explained in more detail in get_dl_model_param.
Due to large memory usage, typically only small batch sizes are possible for a training.
As a consequence, training is rather slow, and we advise using a higher momentum than, for example, for classification.
The HDevelop example segment_pill_defects_deep_learning_2_train.hdev provides good initial parameter values for the training of a segmentation network in HALCON.
Class weights
With the hyperparameter “class weights,” you can assign each class the weight its pixels receive during training.
Giving the unique classes a different weight, it is possible to force the network to learn the classes with different importance.
This is useful when a class dominates the images.
For example, in defect detection, defects occupy only a small fraction of an image.
In such a case, a network classifying every pixel as background (thus, “not defect”) would achieve good loss results in general.
Assigning different weights to the distinct classes helps to re-balance the distribution.
In short, you can focus the loss to train especially on those pixels you determine to be important.
The network obtains these weights via weight_image, an image created for every training sample.
In weight_image, every pixel value corresponds to the weight the corresponding pixel of the input image receives during training.
You can create these images with the help of the following two procedures:
-
calculate_dl_segmentation_class_weightshelps you to create the class weights. The procedure uses the concept of inverse class frequency weights. -
gen_dl_segmentation_weight_imagesuses the class weights and generates theweight_image.
This step has to be done before the training.
Usually, it is done during the preprocessing and therefore is part of the procedure preprocess_dl_dataset.
Note
- The “class weights” hyperparameter is referred to as
class_weightsorClassWeightswithin procedures. - When assigning a specific part of the image a weight of 0.0, these pixels do not contribute to the loss. For more information about the loss, see About the network and the training process.
Figure 5 below shows what such an image with different weights looks like.
Figure 5. Schema for weight_image.
For visibility, gray values are used to represent numbers.
segmentation_image defining the classes for every pixel within the image – 0: background (white), 1: orange, 2: lemon, 3: apple, 4: border (black), declared as “ignore”.
Corresponding weight_image providing class weights – background: 1.0, orange: 30.0, lemon: 75.0. Pixels of classes
declared as “ignore” (here: class border) will be ignored and get the weight 0.0.
Evaluation measures for the data from semantic segmentationđź”—
For semantic segmentation, the following evaluation measures are supported in HALCON.
Note
- To be able to compute such a measure for an image, the related ground truth information is needed.
- All the measure values explained below for a single image (for example,
mean_iou) can also be calculated for an arbitrary number of images. For this, imagine a single large image formed by the ensemble of output images, for which the measure is computed. - All pixels of a class declared as “ignore” are ignored for the computation of the measures.
pixel_accuracy-
The pixel accuracy is the ratio of all pixels that have been predicted with the correct class label to the total number of pixels.
Figure 6. Visual example of the
pixel_accuracy:
segmentation_imagedefining the ground truth class for each pixel. Pixels of a class declared as “ignore” are drawn in black.
Output image – pixels of classes declared as “ignore” are classified.
Pixel accuracy as the ratio of the total orange areas. Pixels labeled as part of a class declared as “ignore” are ignored. class_pixel_accuracy-
The per-class pixel accuracy considers only pixels of a single class. It is defined as the ratio of correctly predicted pixels to the total number of pixels labeled with this class.
If a class does not occur, it receives a
class_pixel_accuracyvalue of-1and does not contribute to the average valuemean_accuracy. mean_accuracy- The mean accuracy is defined as the averaged per-class pixel accuracy (
class_pixel_accuracy) of all occurring classes. class_iou-
The per-class intersection over union (IoU) gives, for a specific class, the ratio of correctly predicted pixels to the union of annotated and predicted pixels. Visually, this is the ratio of the intersection to the union of the areas; see Figure 7 below.
If a class does not occur, it receives a
class_iouvalue of-1and does not contribute to themean_iou.Figure 7. Visual example of the per-class IoU (
class_iou), here: class “apple” only.
segmentation_imagedefining the ground truth class for each pixel. Pixels of a class declared as “ignore” are drawn in black.
Output image, also the pixels of classes declared as “ignore” are classified.
IoU as the ratio of the intersection to the union of the areas of pixels denoted as apple. Pixels labeled as part of a class declared as “ignore” are ignored. mean_iou- The mean IoU is defined as the averaged per-class intersection over union (
class_iou) of all occurring classes. Every occurring class has the same impact on this measure, independent of the number of pixels they contain. frequency_weighted_iou- As for the mean IoU, the per-class IoU is calculated first. However, the contribution of each occurring class to this measure is weighted by the ratio of pixels belonging to that class. Classes with many pixels can dominate this measure.
pixel_confusion_matrix- For more information about confusion matrices, see Confusion matrix on the Deep learning overview page. It applies to semantic segmentation, as the instances are single pixels.