Skip to content

Workflow for Deep 3D Matching🔗

About the general inference workflow🔗

This paragraph describes how to determine a 3D pose using the Deep 3D Matching method. An application scenario can be seen in the HDevelop example deep_3d_matching_workflow.hdev.

  1. Read the trained Deep 3D Matching model by using

  2. Optimize the deep learning network for the use with AI² interfaces:

    1. Extract the detection network from the deep 3d matching model using

    2. Optimize the parameter for inference with

    3. Set the optimized detection network using

    4. Repeat these steps for the 3D pose estimation network.

    5. Save the optimized model using

      Important

      The optimization of the model has significant impact on the runtime if it is done with every inference run. Hence, writing the optimized model saves time in the inference.

  3. Set the camera parameters using

  4. Apply the model using the operator

  5. Visualize the resulting 3D poses.

Creating synthetic training data🔗

The creation of realistic synthetic datasets for training Deep 3D Matching models in HALCON involves an integrated workflow with the Scene Engine. This enables realistic object placement, variable object and environmental properties, flexible camera perspectives, and photorealistic rendering.

A physics engine simulates realistic object placements by dropping them into the scene.

The Asset Manager defines material properties like texture, reflection, and transparency to create realistic surfaces. Various backgrounds and lighting scenarios add variability.

Strategic camera positioning enables capturing images from different angles and distances to simulate real observation conditions.

Photorealistic rendering accurately depicts light, shadows, and reflections, providing high-quality training data for Deep 3D Matching models.

The datasets generated in this way provide the extensive and diverse data needed for effectively training Deep 3D Matching Models in HALCON.

An application scenario using the Scene Engine can be seen in the HDevelop example deep_3d_matching_data_generation.hdev.

To create synthetic training data:

  1. Create the dataset.

    1. Read the CAD object model using

    2. Create a dictionary to collect the generated data using the procedure

      • create_dataset_deep_3d_matching.
    3. Save the dictionary using

  2. Set up the Scene Engine environment.

    1. Start Scene Engine using the operator

    2. Get the default parameters for the Scene Engine environment using the procedure

      • create_scene_engine_run_params.
    3. Set the parameters for material, surface finish, and color using the procedure

      • set_scene_engine_run_param.
    4. Set the camera setup using the procedure

      • set_scene_engine_run_param.
  3. Generate the data.

    1. Start the rendering process using the operator

    2. Get the ground truth data using the procedure

      • get_data_generation_gt.
    3. Save the dictionary with the generated data using

Training the model🔗

This section describes the training of the Deep 3D Matching model using synthetic data. For an application scenario, see also the HDevelop example deep_3d_matching_training_workflow.hdev

To train the model:

  1. Create the Deep 3D Matching model.

    1. Read the rendered dataset using

    2. Retrieve the CAD object model from the read dataset using the key 'orig_3d_model'.

    3. Create the Deep 3D Matching model containing both of the two model components

      • dl_model_detection and

      • dl_model_pose_estimation

      using the operator create_deep_matching_3d.

  2. Prepare for training.

    Before preprocessing, the dataset and the model need to be adapted, so that they can be used later on for training.

    1. Before training the components of the Deep 3D Matching model separately, the components need to be extracted from the model. This can be done using

    2. For the pose estimation component, the dataset needs to be converted into a format that can be processed by the model using

      • convert_dl_dataset_detection_to_pose_estimation.

      This creates a dictionary DLDataset, serving as a database that stores all necessary information about your data. For more information about datasets, see Deep learning > HALCON DL model.

  3. Preprocess the data.

    These steps need to be done separately for the detection and the pose estimation component of the model. See Data below for details on what data is required at each stage of the Deep 3D Matching workflow.

    1. Split the dataset represented by the dictionary DLDataset. This can be done using

      • split_dl_dataset.
    2. Specify preprocessing parameters, such as image size, and store them in a dictionary DLPreprocessParam, for which you can use

      • create_dl_preprocess_param_from_model.
    3. Preprocess your dataset. For this, you can use the procedure

      • preprocess_dl_dataset.
  4. Train the model.

    1. Set training parameters and store them in the dictionary TrainParam using

      • create_dl_train_param.
    2. Train the model. This can be done using

      • train_dl_model.

      The procedure expects:

      • the model handle DLModelHandle,

      • the dictionary DLDataset containing data information,

      • the dictionary TrainParam containing training parameters.

    After a successful training of both the detection network and the pose estimation network, the combined Deep 3D Matching model can be used for inference. See About the general inference workflow above.