Operator Reference

DL Classification

The workflow and operators described in this chapter are obsolete and only provided for reasons backward compatibility. The operators will be removed with HALCON 25.05. New applications should use the workflow and operators described in Deep Learning / Classification.

This chapter explains how to use classification based on deep learning, both for the training and inference phases.

General Workflow

This section describes the main steps and the workflow for classification with Deep Learning using the obsolete workflow.

Prepare the Network and the Data

This part is about how to prepare and preprocess your data.

First, a pretrained network has to be read using the operator
- read_dl_classifierread_dl_classifierReadDlClassifierReadDlClassifierread_dl_classifier.
This operator is used as well when you want to read your own trained networks, after you saved them with write_dl_classifierwrite_dl_classifierWriteDlClassifierWriteDlClassifierwrite_dl_classifier.
To read the data for your deep learning classification training the procedure
- read_dl_classifier_data_set
is available. Using this procedure you can get a list of image file paths and their respective labels (the ground truth labels) as well as a list of the unique classes, to which at least one of the listed images belongs.
The network will impose several requirements on the images, as the image dimensions and the gray value range. The default values are listed in read_dl_classifierread_dl_classifierReadDlClassifierReadDlClassifierread_dl_classifier. These are the values with which the networks have been pretrained. The network architectures allow different image dimensions, which can be set with set_dl_classifier_paramset_dl_classifier_paramSetDlClassifierParamSetDlClassifierParamset_dl_classifier_param, but depending on the network a change may make a retraining necessary. The actually set values can be retrieved with
- get_dl_classifier_paramget_dl_classifier_paramGetDlClassifierParamGetDlClassifierParamget_dl_classifier_param.
The procedure preprocess_dl_classifier_images provides great guidance on how to implement such a preprocessing stage. We recommend to preprocess and store all images used for the training before starting the classifier training, since this speeds up the training significantly.
Next, we recommend to split the dataset into three distinct datasets which are used for training, validation, and testing, see the section “Data” in the chapter Deep Learning. This can be achieved using the procedure
- split_dl_classifier_data_set.
You need to specify the 'classes'"classes""classes""classes""classes" (determined before by use of read_dl_classifier_data_set) you want to differentiate with your classifier. For this, the operator
- set_dl_classifier_paramset_dl_classifier_paramSetDlClassifierParamSetDlClassifierParamset_dl_classifier_param
is available.

This operator can also be used to set hyperparameters, which are important for training, e.g., 'batch_size'"batch_size""batch_size""batch_size""batch_size" and 'learning_rate'"learning_rate""learning_rate""learning_rate""learning_rate". For a more detailed explanation, see the chapter Deep Learning and the documentation of set_dl_classifier_paramset_dl_classifier_paramSetDlClassifierParamSetDlClassifierParamset_dl_classifier_param.

Train the Network and Evaluate the Training Progress

Once your network is set up and your data prepared it is time to train the classifier for your specific task.

Set the hyperparameters used for training with the operator
- set_dl_classifier_paramset_dl_classifier_paramSetDlClassifierParamSetDlClassifierParamset_dl_classifier_param.
For an overview of possible hyperparameters, see the documentation of set_dl_classifier_paramset_dl_classifier_paramSetDlClassifierParamSetDlClassifierParamset_dl_classifier_param. Additional explanations can be found in the chapter Deep Learning.
To train the classifier the operator
- train_dl_classifier_batchtrain_dl_classifier_batchTrainDlClassifierBatchTrainDlClassifierBatchtrain_dl_classifier_batch
is available. The intermediate training results are stored in the output handle.

As the name of train_dl_classifier_batchtrain_dl_classifier_batchTrainDlClassifierBatchTrainDlClassifierBatchtrain_dl_classifier_batch indicates, this operator processes a batch of data (images and ground truth labels) at once. We iterate through our training data in order to train the classifier successively with train_dl_classifier_batchtrain_dl_classifier_batchTrainDlClassifierBatchTrainDlClassifierBatchtrain_dl_classifier_batch. You can repeat this process multiple times and iterate over so many training epochs until you are satisfied with the training result.
To know how well the classifier learns the new task, the procedure
- plot_dl_classifier_training_progress
is provided. With it you can plot the classification errors during training. To compute the input necessary for the visualization, the procedures
- select_percentage_dl_classifier_data,
- apply_dl_classifier_batchwise, and
- evaluate_dl_classifier
are available. With them you can reduce the number of images used for this classification validation, apply the classifier on the selected data and compute, for example, the top-1 error.

Apply and Evaluate the Final Classifier

Your classifier is trained for your task and ready to be applied. But before deploying in the real world you should evaluate how well the classifier performs on basis of your test data.

To apply the classifier on a set containing an arbitrary number of images, use the operator
- apply_dl_classifierapply_dl_classifierApplyDlClassifierApplyDlClassifierapply_dl_classifier.
The runtime of this operator depends on the number of batches needed for the given image set.

The results are returned in a handle.

To retrieve the predicted classes and confidences, use the operator
- get_dl_classifier_resultget_dl_classifier_resultGetDlClassifierResultGetDlClassifierResultget_dl_classifier_result.
Now it is time to evaluate these results. The performance of the classifier can be evaluated as during the training with evaluate_dl_classifier.

To visualize and analyze the classifier quality, the confusion matrix is a useful tool (see below for an explanation). For this, you can use the procedures
- gen_confusion_matrix
- gen_interactive_confusion_matrix.
The interactive procedure gives you the possibility to select examples of a specific category, but it does not work with exported code.

Additionally, after applying the classifier on a set of data, you can use the procedure
- get_dl_classifier_image_results
to display and return images according to certain criteria, e.g., wrongly classified ones. Then, you might want to use this input for the procedure
- dev_display_dl_classifier_heatmap,
to display a heatmap of the input image, with which you can analyze which regions of the image are relevant for the classification result.

Inference Phase

When your classifier is trained and you are satisfied with its performance, you can use it to classify new images. For this, you simply preprocess your images according to the network requirements (i.e., the same way as you did for your dataset used for training the classifier) and apply the classifier using

apply_dl_classifierapply_dl_classifierApplyDlClassifierApplyDlClassifierapply_dl_classifier.

Data for Classification

We distinguish between data used for training and data for inference. Latter one consists of bare images. But for the former one you already know to which class the images belong and provide this information over the corresponding labels.

The training data is used to train a classifier for your specific task. With the aid of this data the classifier can learn which classes are to be distinguished and how their representatives look like. In classification, the image is classified as a whole. Therefore, the training data consists of images and their ground truth labels, thus the class you say this image belongs to. Note that the images should be as representative as possible for your task. There are different possible ways, how to store and retrieve the ground truth labels. The procedure read_dl_classifier_data_set supports the following sources of the ground truth label for an image:

The last directory name containing the image
The file name.

For training a classifier, we use a technique called transfer learning (see the chapter Deep Learning). For this, you need less resources, but still a suitable set of data which is generally in the order of hundreds to thousands per class. While in general the network should be more reliable when trained on a larger dataset, the amount of data needed for training also depends on the complexity of the task. You also want enough training data to split it into three subsets, which are preferably independent and identically distributed, see the section “Data” in the chapter Deep Learning.

Regardless of the application, the network poses requirements on the images regarding the image dimensions, the gray value range, and the type. The specific values depend on the network itself and can be queried with get_dl_classifier_paramget_dl_classifier_paramGetDlClassifierParamGetDlClassifierParamget_dl_classifier_param. You can find guidance on how to implement such a preprocessing stage by the procedure preprocess_dl_classifier_images.

List of Operators

apply_dl_classifierApplyDlClassifierapply_dl_classifierApplyDlClassifierapply_dl_classifier: Infer the class affiliations for a set of images using a deep-learning-based classifier.

clear_dl_classifierClearDlClassifierclear_dl_classifierClearDlClassifierclear_dl_classifier: Clear a deep-learning-based classifier.

clear_dl_classifier_resultClearDlClassifierResultclear_dl_classifier_resultClearDlClassifierResultclear_dl_classifier_result: Clear a handle containing the results of the deep-learning-based classification.

clear_dl_classifier_train_resultClearDlClassifierTrainResultclear_dl_classifier_train_resultClearDlClassifierTrainResultclear_dl_classifier_train_result: Clear the handle of a deep-learning-based classifier training result.

deserialize_dl_classifierDeserializeDlClassifierdeserialize_dl_classifierDeserializeDlClassifierdeserialize_dl_classifier: Deserialize a deep-learning-based classifier.

get_dl_classifier_paramGetDlClassifierParamget_dl_classifier_paramGetDlClassifierParamget_dl_classifier_param: Return the parameters of a deep-learning-based classifier.

get_dl_classifier_resultGetDlClassifierResultget_dl_classifier_resultGetDlClassifierResultget_dl_classifier_result: Retrieve classification results inferred by a deep-learning-based classifier.

get_dl_classifier_train_resultGetDlClassifierTrainResultget_dl_classifier_train_resultGetDlClassifierTrainResultget_dl_classifier_train_result: Return the results for the single training step of a deep-learning-based classifier.

read_dl_classifierReadDlClassifierread_dl_classifierReadDlClassifierread_dl_classifier: Read a deep-learning-based classifier from a file.

serialize_dl_classifierSerializeDlClassifierserialize_dl_classifierSerializeDlClassifierserialize_dl_classifier: Serialize a deep-learning-based classifier.

set_dl_classifier_paramSetDlClassifierParamset_dl_classifier_paramSetDlClassifierParamset_dl_classifier_param: Set the parameters of a deep-learning-based classifier.

train_dl_classifier_batchTrainDlClassifierBatchtrain_dl_classifier_batchTrainDlClassifierBatchtrain_dl_classifier_batch: Perform a training step of a deep-learning-based classifier on a batch of images.

write_dl_classifierWriteDlClassifierwrite_dl_classifierWriteDlClassifierwrite_dl_classifier: Write a deep-learning-based classifier in a file.

Operators