Skip to content

Workflow🔗

The example program %HALCONEXAMPLES%\hdevelop\3D-Matching\Deformable-Surface-Based\find_deformable_surface_model.hdev shows how to use the deformable surface-based 3D matching to find even deformed silicone baking molds.

Figure 1: Deformable silicone baking mold to be found in a 3D scene.

  1. Provide the 3D object model needed for the creation of the deformable surface model

    In contrast to shape-based 3D matching, for deformable surface-based matching, the 3D object model does not need to be available as CAD model but can also be provided by a 3D sensor or be derived by a 3D reconstruction.

    In the example program, the 3D object model is provided as CAD model (see Figure 2). This kind of data source was mainly chosen for visualization reasons. With this, it is easier to distinguish the deformed model from the search scene.

    Figure 2: 3D object model of the silicone baking mold.

    Note that for deformable surface-based matching information about the coordinates of the 3D points and their normals is needed. Thus, if a 3D object model is obtained from a CAD model or from multi-view stereo, the normals or a triangular or polygon mesh should be contained in the 3D object model.

  2. Create the deformable surface model with create_deformable_surface_model

    The operator create_deformable_surface_model creates a deformable surface model by sampling the 3D object model with a certain distance. The sampling distance can be adjusted with the parameter RelSamplingDistance.

    Note that a smaller value leads to a slower but more robust matching, whereas a larger value speeds up the matching but at the same time decreases the robustness.

    create_deformable_surface_model (ObjectModel3DReference, 0.03, 'stiffness', 0.85, DeformableSurfaceModel)

    If the 3D object model does not contain normals, they are determined by create_deformable_surface_model automatically.

    Note

    In this case, the orientation of the normals is ambiguous. Internally, they are automatically oriented such that all normals have a positive z component.

    The correct match will only be found if the normals of the model and of the search object point in the same direction.

    If the orientation of the search object may vary considerably, some effort should be made to orient the normals of the search object consistently with those of the model. For example, it might be helpful to check whether all normals point away from the center of gravity of the object and to flip those normals that do not.

    Of course, this correction of the normals must be performed for both the model and the search object.

    Another possibility is to create two models with the generic parameter 'model_invert_normals' set to 'true' and 'false', respectively.

    Both models can then be used for matching and the match with the higher score should be selected.

  3. Optionally define reference points with add_deformable_surface_model_reference_point

    It is possible to define a set of reference points with add_deformable_surface_model_reference_point.

    add_deformable_surface_model_reference_point (DeformableSurfaceModel, ReferencePointX, ReferencePointY, ReferencePointZ, ReferencePointIndex)

    Reference points are 3D points that can be placed at any position, i.e., it is not necessary that they lie on the object’s surface.

    After a possibly deformed object has been found with find_deformable_surface_model, the respective 3D position of the reference points in the deformed scene can be determined with get_deformable_surface_matching_result.

    find_deformable_surface_model (DeformableSurfaceModel, ObjectModel3DSearchSceneForDef, 0.03, 0, [], [], Score, DeformableSurfaceMatchingResult)
    get_deformable_surface_matching_result (DeformableSurfaceMatchingResult, 'reference_point_x', 'all', ReferencePointXDeformed)
    get_deformable_surface_matching_result (DeformableSurfaceMatchingResult, 'reference_point_y', 'all', ReferencePointYDeformed)
    get_deformable_surface_matching_result (DeformableSurfaceMatchingResult, 'reference_point_z', 'all', ReferencePointZDeformed)

    Reference points are, for example, helpful to determine the grasping points for the deformed 3D object.

  4. Optionally extend the range of deformations that the deformable surface model is able to cope with

    The supported range of deformations can be extended if information about the expected deformations is added to the deformable surface model.

    First, provide 3D scenes that contain deformed instances of the 3D object, then, detect those instances with find_deformable_surface_model, and finally, add the found deformed objects to the deformable surface model with add_deformable_surface_model_sample to extend the range of deformations that the deformable surface model is able to cope with.

    To make the matching in the 3D scenes used for the model extension faster and more robust, in the example program, the background is eliminated from the reconstructed 3D scene by eliminating all points that are close to the (known) background plane.

    gen_plane_object_model_3d (PlanePose, [], [], ObjectModel3DPlane)
    distance_object_model_3d (ObjectModel3D, ObjectModel3DPlane, [], 0, [], [])
    get_object_model_3d_params (ObjectModel3D, '&distance', Distances)
    select_points_object_model_3d (ObjectModel3D, '&distance', MinDistanceToPlane, max(Distances), ObjectModel3DThresholded)

    In the resulting 3D scene, the deformed object is detected with find_deformable_surface_model and the found deformed object is added to the deformable surface model with add_deformable_surface_model_sample:

    find_deformable_surface_model (DeformableSurfaceModel, ObjectModel3DSearchSceneForDef, 0.03, 0, [], [], Score, DeformableSurfaceMatchingResult)
    get_deformable_surface_matching_result (DeformableSurfaceMatchingResult, 'deformed_sampled_model', 0, ObjectModel3DDeformedSampled)
    add_deformable_surface_model_sample (DeformableSurfaceModel, ObjectModel3DDeformedSampled)
  5. Provide a 3D scene that represents the search data

    The 3D scene in which the object of interest is searched for must be provided as a 3D object model (see Figure 3 ), similar to the 3D object model that was used for the creation of the deformable surface model or for the extension of the supported deformation range.

    Figure 3: 3D scene that contains a deformed baking mold.

    Typically, the 3D object models used for the search are acquired by 3D sensors or reconstructed from stereo image data.

  6. Use the deformable surface model to search for the object in the search data

    With the deformable surface model that was created with create_deformable_surface_model or read from file with read_deformable_surface_model, the deformed object can be searched for in the search data using the operator find_deformable_surface_model.

    find_deformable_surface_model (DeformableSurfaceModel, ObjectModel3DSearchSceneForDef, 0.03, 0, [], [], Score, DeformableSurfaceMatchingResult)

    Several parameters can be set to control the search process. For detailed information, we recommend to read the description of the operator find_deformable_surface_model in the Reference Manual.

    If a model instance was found, the operator returns a score that describes the quality of the match and a handle that contains the results, like the deformed position of the reference points.

    After the matching, the result can be visualized. For this, the deformed reference model can be determined with get_deformable_surface_matching_result:

    get_deformable_surface_matching_result (DeformableSurfaceMatchingResult, 'deformed_model', 0, ObjectModel3DDeformed)

    The position of the reference points transformed to the deformed 3D object that was found can be determined with get_deformable_surface_matching_result. It can, for example, be used to grasp the deformed object.

    get_deformable_surface_matching_result (DeformableSurfaceMatchingResult, 'reference_point_x', 'all', ReferencePointXDeformed)
    get_deformable_surface_matching_result (DeformableSurfaceMatchingResult, 'reference_point_y', 'all', ReferencePointYDeformed)
    get_deformable_surface_matching_result (DeformableSurfaceMatchingResult, 'reference_point_z', 'all', ReferencePointZDeformed)

    In the example, the deformed 3D object model is visualized together with the reference points on top of the search scene (see Figure 1).

    Figure 4: The matching result shows the deformed object and reference points.

    Note that, especially, if the object does not have a distinct 3D shape, for example, because most of the points have a similar height, it might be necessary to manually eliminate the background and other objects from the 3D scene to ensure a proper matching result.

    Otherwise, the object might be found in the background because a relatively flat object fits perfectly to a relatively flat background, especially if deformations of the object are allowed.