Skip to content

Advanced object detection🔗

With advanced object detection, the goal is to find different objects within an image and assign them to a class. Multiple objects may appear in the same image and may partially overlap while still being detected as distinct objects. This is illustrated in Figure 1.

Figure 1. Possible example for object detection: Within the input image, three instances are found and assigned to a class.

Unlike image classification, which assigns a single label to an entire image, advanced object detection performs both object localization and classification within a single network.
It is based on an efficient detection architecture, which improves robustness and performance compared to previous approaches. In particular, the detection of objects of different sizes is improved and robustness against varying image conditions is increased.

The model predicts bounding boxes indicating the position of potential objects in the image. As output, the model returns the following information for each detected object:

  • Axis-aligned bounding box (instance type rectangle1)
  • Class assignment
  • Confidence value

The confidence value denotes a model-dependent score that reflects the relative certainty of the network that the predicted bounding box corresponds to an object of the assigned class.

Restriction

Currently, advanced object detection supports only axis-aligned bounding boxes (rectangle1). Therefore, all bounding boxes are axis-aligned rectangles.

More Info

About data🔗

We distinguish between data used for training and evaluation and data used for inference. Training and evaluation data consist of images together with annotations describing the object, whereas inference data consist of images only.

As a basic concept, the model handles data over dictionaries, meaning it receives the input data over a dictionary DLSample and returns results in dictionaries such as DLResult . For more information about data handling, see HALCON DL model.

Data for training and evaluation

The dataset consists of images and corresponding annotations. For each object, the class label and its location within the image must be provided.

Each object requires the following information:

  • For rectangle1:

    • Coordinates of the upper left corner (bbox_row1, bbox_col1)
    • Coordinates of the lower right corner (bbox_row2, bbox_col2)
  • For rectangle2:

    • Coordinates of the center point (bbox_row, bbox_col)
    • Orientation angle of the bounding box (bbox_phi)
    • Half edge lengths (bbox_length1, bbox_length2)
  • Corresponding class label

These parameters define the bounding box and are consistent with gen_rectangle1 and gen_rectangle2, respectively.

The dataset is organized in a dictionary DLDataset, which stores the images together with their annotations and additional information required for training and evaluation.

The example detect_pills_deep_learning_1.hdev illustrates how to prepare and structure such a dataset for axis-aligned bounding boxes (rectangle1). For oriented bounding boxes (rectangle2), see dl_advanced_detection_with_orientation_workflow.hdev.

Images

Regardless of the application, the model poses requirements on the images, such as the dimensions, the gray value range, and the type. The specific values depend on the network itself and can be queried with get_dl_model_param.

The required preprocessing and data augmentation steps are defined using a transform pipeline. The transformations are applied to the images at runtime before they are processed by the model and are not stored in the dataset.

Data for inference
For inference, only the images are required. The same transform pipeline as defined for the model should be applied to ensure consistent preprocessing.

Model parameters and hyperparameters🔗

Apart from the general DL hyperparameters explained on the Deep Learning page, there are further hyperparameters relevant for object detection:

  • 'bbox_heads_weight'
  • 'class_heads_weight'

These hyperparameters influence the weighting of the respective loss components during training.

For an advanced object detection model, several model parameters influence the predictions and, consequently, the evaluation results:

  • 'max_num_detections'
  • 'max_overlap'
  • 'max_overlap_class_agnostic'
  • 'min_confidence'

In advanced object detection, the model may predict multiple overlapping bounding boxes for the same object. To reduce such duplicate detections, non-maximum suppression (NMS) is applied.

The suppression behavior can be controlled using the parameters 'max_overlap' and 'max_overlap_class_agnostic'. Suggestions for the parameters 'max_overlap' and 'max_overlap_class_agnostic' can be determined from the dataset with the procedure determine_dl_model_detection_param.

The parameter 'max_overlap' defines the maximum allowed overlap between bounding boxes of the same class. If the overlap exceeds this threshold, only the bounding box with the highest confidence is kept.

The parameter 'max_overlap_class_agnostic' extends this suppression to bounding boxes of different classes.

The parameter 'min_confidence' defines the minimum confidence threshold for a predicted bounding box to be considered a valid detection. It is possible to set a value for each class separately, see set_dl_model_param for details.

These parameters influence the final detection results and, consequently, the evaluation.

The parameters can be set when creating the model or afterwards using the operator set_dl_model_param. For more information, see the operator get_dl_model_param.

Evaluation measures🔗

Note

To be able to compute such a measure for an image, the related ground truth information is needed.

Mean average precision (mAP) and average precision (AP) of a class for an IoU threshold (ap_iou_classname)

The AP value is an average of maximum precision at different recall values. In simple terms, it indicates whether the predicted objects for this class are generally correct detections. In this context, we pay more attention to the predictions with high confidence values. The higher the value, the better.

We want both the correct top-1 classification and localization to count a prediction as a hit. The measure telling us the correctness of the localization is the intersection over union (IoU): An instance is considered correctly localized if its IoU is greater than the required threshold. Hence, the AP value depends on the class and on the IoU threshold.
For more information about the IoU, see Understanding the IoU below.

You can obtain the specific AP values: the averages over the classes, the averages over the thresholds, and the averages over both the classes and the thresholds. The latter is the mean average precision (mAP), a measure of how well images are classified.

True positives, false positives, false negatives

The concept of true positive, false positives, and false negatives is explained in Deep Learning. It applies for object detection with the exception that there are different kinds of false positives, for example:

  • An instance was classified wrongly.
  • An instance was found where there is only background.
  • An instance was localized badly, meaning the IoU between the instance and its ground truth is lower than the evaluation IoU threshold.
  • There is a duplicate, thus at least two instances overlap mainly with the same ground truth bounding box, but they overlap not more than 'max_overlap' with each other, so none of them was suppressed.

Important

To be able to access these values, enable detailed evaluation by setting the parameter detailed_evaluation to 'true' in the evaluate_dl_model procedure.

Understanding the IoU🔗

The intersection over union (IoU) is a measure for the accuracy of an object detection. For a proposed bounding box, it compares the ratio between area of intersection and the area of overlap with the ground truth bounding box. The figure below shows an example.

 

Figure 2. Visual example of the IoU, illustrated for instance type 'rectangle1'
Input image with ground truth bounding box (orange) and predicted bounding box (light blue)
IoU as the ratio between the area intersection and the area overlap