Skip to content

Workflow for instance segmentation and object detection🔗

We assume that your dataset is already labeled; see also About data.

Tip

Take a look at the HDevelop example series detect_pills_deep_learning for an application. The example dl_instance_segmentation_workflow.hdev shows the complete workflow for an instance segmentation application.

  1. Create the model and preprocess the dataset.

    1. Create a model using the operator

      ⤷ The operator returns a handle 'DLModelHandle'.

      You have to specify at least the backbone and the number of classes to be distinguished. For instance segmentation, set the parameter 'instance_segmentation' to create an according model.

      Further parameters can be set via the dictionary DLModelDetectionParam. Their values should be well chosen for the specific task to help reduce memory consumption and runtime. See the operator documentation for more information.

      Anchor parameters suiting your dataset can be estimated using the procedure

      • determine_dl_model_detection_param.

      In case the dataset is not representing all orientations the network will face during training (for example, due to augmentation), the suggested values need to be adapted accordingly.

      Note

      After creating the model, its underlying network architecture is fixed to the specified input values.

      Alternatively, to read in a model you have already saved with write_dl_model, you can also use

    2. The information about what is found on each image of your training dataset needs to be read in. This can be done by reading the data out of one of the following:

      • DLDataset dictionary using read_dict

      • File in the COCO data format using the procedure read_dl_dataset_from_coco, which creates a dictionary DLDataset

      The dictionary DLDataset serves as a database storing all necessary information about your data. For more information about the data and the way it is transferred, see About data below and the page HALCON DL model.

    3. Split the dataset represented by the dictionary DLDataset. This can be done using the procedure

      • split_dl_dataset.

      ⤷ The resulting split is saved via the key split in each sample entry of DLDataset.

    4. The network imposes requirements on the images, for example, the image width and height. You can retrieve every single value using the operator

      Alternatively, you can retrieve all necessary parameters using the procedure

      • create_dl_preprocess_param_from_model.

      Important

      For classes declared by 'class_ids_no_orientation', the bounding boxes need to be treated specially during preprocessing. As a consequence, these classes should be set at this point latest.

      Now you can preprocess your dataset. For this, you can use the procedure

      • preprocess_dl_dataset.

      This procedure also offers guidance on how to implement a customized preprocessing procedure. We recommend preprocessing and storing all images used for the training before starting the training, as this speeds up the training significantly.

      To visualize the preprocessed data, you can use the procedure

      • dev_display_dl_data.

    These steps are also shown in the HDevelop example detect_pills_deep_learning_1_prepare.hdev.

  2. Train the model.

    1. Set the training parameters and store them in the dictionary TrainParam. These parameters include:

    2. Train the model. This can be done using the procedure

      • train_dl_model.

      The procedure expects the following:

      • Model handle DLModelHandle
      • Dictionary with the data information DLDataset
      • Dictionary with the training parameters TrainParam
      • Information how many epochs the training shall run

      ⤷ During the training, the total loss minimizes.

    These steps are also shown in the HDevelop example detect_pills_deep_learning_2_train.hdev.

  3. Evaluate the trained model.

    1. Set the model parameters that may influence the evaluation.

    2. The evaluation can conveniently be done using the procedure

      • evaluate_dl_model.

      This procedure expects a dictionary GenParamEval with the evaluation parameters.
      To get the data necessary for visualization, set the parameter detailed_evaluation to 'true'.

    3. You can visualize your evaluation results using the procedure

      • dev_display_detection_detailed_evaluation.

    These steps are also shown in the HDevelop example detect_pills_deep_learning_3_evaluate.hdev.

  4. Run inference on new images.

    1. Request the requirements the network imposes on the images using the operator

      or the procedure

      • create_dl_preprocess_param_from_model.
    2. Set the model parameter described in Model parameters and hyperparameters using the operator

      The 'batch_size' can be generally set independently from the number of images to be inferred. For more information about how to set this parameter for greater efficiency, see apply_dl_model.

    3. Generate a data dictionary DLSample for each image. This can be done using the procedure

      • gen_dl_samples_from_images.
    4. Every image has to be preprocessed as done for the training. For this, you can use the procedure

      • preprocess_dl_samples.
    5. Apply the model using the operator

    6. Retrieve the results from the dictionary 'DLResultBatch'.

    These steps are also shown in the HDevelop example detect_pills_deep_learning_4_infer.hdev.