3D vision🔗
-
All about depth from focus
-
All about grid rectification
-
All about mosaicking
-
All about robot vision
-
All about sheet of light
-
All about stereo systems
With HALCON you can perform 3D vision in various ways. The main applications comprise the 3D position recognition and the 3D inspection, which both consist of several different approaches with different characteristics, so that for a wide range of 3D vision tasks a proper solution can be provided.
More Info
Learn about 3D vision and many other topics in interactive online courses at our MVTec Academy.
-
What basic knowledge do you need for 3D vision?🔗
-
Typically, you have to calibrate your camera(s) before applying a 3D vision task. Especially, if you want to achieve accurate results, the camera calibration is essential, because it is of no use to extract edges with an accuracy of 1/40 pixel if the lens distortion of the uncalibrated camera accounts for a couple of pixels. This also applies if you use cameras with telecentric lenses.
Using a camera calibration, you can transform image processing results into arbitrary 3D coordinate systems and thus derive metrical information from images, regardless of the position and orientation of the camera with respect to the object. In other words, you can perform inspection tasks in 3D coordinates in specified object planes, which can be oriented arbitrarily with respect to the camera. This is, for example, useful if the camera cannot be mounted such that it looks perpendicular to the object surface.
-
How can you obtain an object’s 3D position and orientation?🔗
-
The position and orientation of 3D objects with respect to a given 3D coordinate system, which is needed, for example, for pick-and-place applications (3D alignment), can be determined by one of these methods:
-
The pose estimation of a known 3D object from corresponding points (known as mono 3d) is a rather general approach that includes a camera calibration and the extraction of at least three significant points for which the 3D object coordinates are known.
-
HALCON’s 3D matching locates known 3D objects based on a 3D model of the object. In particular, it automatically searches objects that correspond to a 3D model in the search data and determines their 3D poses. The model must be provided, for example, as a Computer Aided Design (CAD) model. Available approaches are the shape-based 3D matching that searches the model in 2D images and the surface-based 3D matching that searches the model in a 3D scene, that is, in a set of 3D points that is available as 3D object model, which can be obtained by a 3D reconstruction approach like stereo or sheet of light. Note that the surface-based matching is also known as “volume matching”, although it only relies on points on the object’s surface.
-
HALCON’s 3D primitives fitting fits a primitive 3D shape like a cylinder, sphere, or plane into a 3D scene, that is, into a set of 3D points that is available as a 3D object model, which can be obtained by a 3D reconstruction approach like stereo or sheet of light followed by a 3D segmentation.
-
The calibrated perspective matching locates perspectively distorted planar objects in images based on a 2D model. In particular, it automatically searches objects that correspond to a 2D model in the search images and determines their 3D poses. The model typically is obtained from a representative model image. Available approaches are the calibrated perspective deformable matching that describes the model by its contours and the calibrated descriptor-based matching that describes the model by a set of distinctive points that are called “interest points”.
-
The circle pose estimation and rectangle pose estimation use the perspective distortions of circles and rectangles to determine the pose of planar objects that contain circles and/or rectangles in a rather convenient way.
-
-
How can you inspect a 3D object?🔗
-
The inspection of 3D objects can be applied by different means. If the inspection in a specified plane is sufficient, you can use a camera calibration together with a 2D inspection (see mono 3d).
If the surface of the 3D object is needed and/or the inspection can not be reduced to a single specified plane, you can use a 3D reconstruction together with a 3D inspection. That is, you use the point, surface, or height information returned for a 3D object by a 3D reconstruction and inspect the object, for example, by comparing it to a reference point, surface, or height.
3D inspection
| Setup | Single Camera (specified plane) | Multiple cameras | Single camera (multiple images, additional hardware) | 3D sensor |
|---|---|---|---|---|
| uncalibrated | Photometric stereo | |||
| calibrated | Stereo vision | Depth from focus
Sheet of light Structured light |
time-of-flight |
| Setup | Single Camera (specified plane) | Multiple cameras | Single camera (multiple images, additional hardware) | 3D sensor |
|---|---|---|---|---|
| uncalibrated | ||||
| calibrated | Camera calibration | Stereo vision |
3D position recognition
| 3D Object | Planar object part (perspective view) | |
|---|---|---|
| Arbitrary objects | 3D mono Shape-based 3D matching |
Surface-based 3D matching |
| Objects with primitive shapes | 3D primitives fitting | 2D primitives fitting |
How can you reconstruct 3D objects?🔗
To determine points on the surface of arbitrary objects, the following approaches are available:
-
HALCON’s stereo vision functionality allows to determine the 3D coordinates of any point on the object surface based on two (binocular stereo) or more (multi-view stereo) images that are acquired suitably from different points of view (typically by separate cameras).
Using multi-view stereo, you can reconstruct a 3D object in full 3D, in particular, you can reconstruct it from different sides. -
A laser triangulation with sheet of light allows to get a height profile of the object.
Note that besides a single camera, additional hardware, in particular a laser line projector and a unit that moves the object relative to the camera and the laser, is needed. -
With depth from focus (DFF) a height profile can be obtained using images that are acquired by a single telecentric camera but at different focus positions.
In order to vary the focus position additional hardware like a translation stage or linear piezo stage is required.
Note that depending on the direction in which the focus position is modified, the result corresponds either to a height image or to a distance image.
A height image contains the distances between a specific object or measure plane and the object points, whereas the distance image typically contains the distances between the camera and the object points.
Both can be called also depth image or “Z image”. -
With photometric stereo a height image can be obtained using images that are acquired by a single telecentric camera but with at least three different telecentric illumination sources for which the spatial relations to the camera must be known.
Note that the height image reflects only relative heights, that is, with photometric stereo no calibrated 3D reconstruction is possible. -
With structured light a 3D object model is derived from projected patterns on a diffuse object surface.
The setup consists of calibrated camera and projector. -
Besides the 3D reconstruction approaches provided by HALCON, you can obtain 3D information also by specific 3D sensors like time of flight (TOF) cameras.
These cameras typically are calibrated and return X, Y, and Z images.
| 3D reconstruction approach | Hardware requirement | Object size | Possible results |
|---|---|---|---|
| Multi-view stereo | multiple cameras, calibration object |
approx. > 10 cm | 3D object model or X, Y, Z coordinates |
| Binocular stereo | two cameras, calibration object |
approx. > 10 cm | X, Y, Z coordinates, disparity image, or Z image (approach-specific) |
| Sheet of light | camera, laser line projector, unit to move object, calibration object |
object must fit onto moving unit | 3D object model, X, Y, Z images, or disparity image (approach-specific) |
| Depth from focus | telecentric camera, hardware to variate the focus position |
approx. > 10 cm | Z image |
| Photometric stereo | telecentric camera, at least three telecentric illumination sources |
restricted by field of view of telecentric camera | Z image |
| Structured light | camera, projector |
Stereo vision | few mm² to several m² |
| 3D sensors | special camera like calibrated TOF | Stereo vision | approx. 30 cm - 5 m |
How can you extend 3D vision to robot vision?🔗
A typical application area for 3D vision is robot vision, that is, using the results of machine vision to command a robot. In such applications you must perform an additional calibration: the so-called hand-eye calibration, which determines the relation between camera and robot coordinates. Again, this calibration must be performed only once (offline). Its results allow you to quickly transform machine vision results from camera into robot coordinates.
What tasks may be needed additionally?🔗
If the object that you want to inspect is too large to be covered by one image with the desired resolution, multiple images, each covering only a part of the object, can be combined into one larger mosaic image. This can be done either based on a calibrated camera setup with very high precision or highly automated for arbitrary and even varying image configurations.
If an image shows distortions that are different to the common perspective distortions or lens distortions, caused, for example, by a non-flat object surface, the so-called grid rectification can be applied to rectify the image.