Workflow for Deep 3D Matching🔗
About the general inference workflow🔗
This paragraph describes how to determine a 3D pose using the Deep 3D Matching method.
An application scenario can be seen in the HDevelop example deep_3d_matching_workflow.hdev.
-
Read the trained Deep 3D Matching model by using
-
Optimize the deep learning network for the use with AI² interfaces:
-
Extract the detection network from the deep 3d matching model using
-
Optimize the parameter for inference with
-
Set the optimized detection network using
-
Repeat these steps for the 3D pose estimation network.
-
Save the optimized model using
Important
The optimization of the model has significant impact on the runtime if it is done with every inference run. Hence, writing the optimized model saves time in the inference.
-
-
Set the camera parameters using
-
Apply the model using the operator
-
Visualize the resulting 3D poses.
Creating synthetic training data🔗
The creation of realistic synthetic datasets for training Deep 3D Matching models in HALCON involves an integrated workflow with the Scene Engine. This enables realistic object placement, variable object and environmental properties, flexible camera perspectives, and photorealistic rendering.
A physics engine simulates realistic object placements by dropping them into the scene.
The Asset Manager defines material properties like texture, reflection, and transparency to create realistic surfaces. Various backgrounds and lighting scenarios add variability.
Strategic camera positioning enables capturing images from different angles and distances to simulate real observation conditions.
Photorealistic rendering accurately depicts light, shadows, and reflections, providing high-quality training data for Deep 3D Matching models.
The datasets generated in this way provide the extensive and diverse data needed for effectively training Deep 3D Matching Models in HALCON.
An application scenario using the Scene Engine can be seen in the HDevelop example deep_3d_matching_data_generation.hdev.
To create synthetic training data:
-
Create the dataset.
-
Read the CAD object model using
-
Create a dictionary to collect the generated data using the procedure
create_dataset_deep_3d_matching.
-
Save the dictionary using
-
-
Set up the Scene Engine environment.
-
Start Scene Engine using the operator
-
Get the default parameters for the Scene Engine environment using the procedure
create_scene_engine_run_params.
-
Set the parameters for material, surface finish, and color using the procedure
set_scene_engine_run_param.
-
Set the camera setup using the procedure
set_scene_engine_run_param.
-
-
Generate the data.
-
Start the rendering process using the operator
-
Get the ground truth data using the procedure
get_data_generation_gt.
-
Save the dictionary with the generated data using
-
Training the model🔗
This section describes the training of the Deep 3D Matching model using synthetic data.
For an application scenario, see also the HDevelop example deep_3d_matching_training_workflow.hdev
To train the model:
-
Create the Deep 3D Matching model.
-
Read the rendered dataset using
-
Retrieve the CAD object model from the read dataset using the key
'orig_3d_model'. -
Create the Deep 3D Matching model containing both of the two model components
-
dl_model_detectionand -
dl_model_pose_estimation
using the operator
create_deep_matching_3d. -
-
-
Prepare for training.
Before preprocessing, the dataset and the model need to be adapted, so that they can be used later on for training.
-
Before training the components of the Deep 3D Matching model separately, the components need to be extracted from the model. This can be done using
-
For the pose estimation component, the dataset needs to be converted into a format that can be processed by the model using
convert_dl_dataset_detection_to_pose_estimation.
This creates a dictionary
DLDataset, serving as a database that stores all necessary information about your data. For more information about datasets, see Deep learning > HALCON DL model.
-
-
Preprocess the data.
These steps need to be done separately for the detection and the pose estimation component of the model. See Data below for details on what data is required at each stage of the Deep 3D Matching workflow.
-
Split the dataset represented by the dictionary
DLDataset. This can be done usingsplit_dl_dataset.
-
Specify preprocessing parameters, such as image size, and store them in a dictionary
DLPreprocessParam, for which you can usecreate_dl_preprocess_param_from_model.
-
Preprocess your dataset. For this, you can use the procedure
preprocess_dl_dataset.
-
-
Train the model.
-
Set training parameters and store them in the dictionary
TrainParamusingcreate_dl_train_param.
-
Train the model. This can be done using
train_dl_model.
The procedure expects:
-
the model handle
DLModelHandle, -
the dictionary
DLDatasetcontaining data information, -
the dictionary
TrainParamcontaining training parameters.
After a successful training of both the detection network and the pose estimation network, the combined Deep 3D Matching model can be used for inference. See About the general inference workflow above.
-