About the used 3D camera model๐
In general, camera calibration means the exact determination of the parameters that model the (optical) projection of any 3D world point \(\mvVectorVar[w]{p}\) into a (sub-)pixel \((r,c)\) in the image. This is important if the original 3D pose of an object must be computed from the image (for example, for measuring industrial parts). The appropriate projection model depends on the camera type used in your setup.
Available camera models๐
For the modeling of this projection process, which is determined by the used combination of camera, lens, and frame grabber, HALCON provides the following 3D camera models:
- Area scan pinhole camera
- The combination of an area scan camera with a lens that effects a perspective projection on the object side of the lens and that may show radial and decentering distortions.
-
The lens may be a tilt lens, i.e., the optical axis of the lens may be tilted with respect to the cameraโs sensor (this is sometimes called a Scheimpflug lens). As hypercentric lenses also perform a perspective projection, cameras with hypercentric lenses are pinhole cameras.
The models for regular (i.e., non-tilt) pinhole and image-side telecentric lenses are identical. In contrast, the models for pinhole and image-side telecentric tilt lenses differ substantially, as described below.
- Area scan telecentric camera
- The combination of an area scan camera with a lens that is telecentric on the object-side of the lens, i.e., that effects a parallel projection on the object-side of the lens, and that may show radial and decentering distortions.
- The lens may be a tilt lens. The models for regular (i.e., non-tilt) bilateral and object-side telecentric lenses are identical. In contrast, the models for bilateral and object-side telecentric tilt lenses differ substantially, as described below.
- Line scan pinhole camera
- The combination of a line scan camera with a lens that effects a perspective projection and that may show radial distortions.
- Tilt lenses are currently not supported for line scan cameras.
- Line scan telecentric camera
- The combination of a line scan camera with a lens that effects a telecentric projection and that may show radial distortions.
- Tilt lenses are currently not supported for line scan cameras.
Transforming world coordinates into pixel coordinates๐
To transform a 3D point \(\mvVectorVar[w]{p} = (\mvVectorVar[w]{x},\mvVectorVar[w]{y},\mvVectorVar[w]{z})^T\), which is given in world coordinates, into a 2D point \(\mvVectorVar[i]{q}=(\mvVectorVar{r},\mvVectorVar{c})^T\), which is given in pixel coordinates, a chain of transformations is needed:
where
- \(\mvVectorVar[w]{p}\): 3D world point
- \(\mvVectorVar[c]{p}\): transformed into camera coordinate system
- \(\mvVectorVar[c]{q}\): projected into image plane (2D point, still in metric coordinates)
- \(\mvVectorVar[c]{{\tilde{q}}}\): lens distortion applied
- \(\mvVectorVar[t]{q}\): If a tilted lens is used, the point \(\mvVectorVar[c]{{\tilde{q}}}\) is projected on the point \(\mvVectorVar[t]{q}\) in the tilted image plane. In this case, the distorted point \(\mvVectorVar[c]{{\tilde{q}}}\) only lies on a virtual image plane of a system without tilt.
- \(\mvVectorVar[i]{q}\): pixel coordinates
The following sections describe these steps in more detail for area scan cameras and subsequently for line scan cameras.
-
Transformation step 1: \(\mvVectorVar[w]{p} \to \mvVectorVar[c]{p}\)
The point \(\mvVectorVar[w]{p}\) is transformed from world into camera coordinates (points as homogeneous vectors, compare
affine_trans_point_3d) by:\[\begin{eqnarray*} \mvVectorTwoD{\mvVectorVar[c]{p}}{1} \: = \: \mvHomVectorThreeD{x^{c}}{y^{c}}{z^{c}} \: = \: \mvHomMatrixThreeDRotTrans{\mvRotationMatrixVar{}{}} {\mvTranslationVectorVar{}{}} \cdot \mvVectorTwoD{\mvVectorVar[w]{p}}{1} \end{eqnarray*}\]with \(\mvRotationMatrixVar{}{}\) and \(\mvTranslationVectorVar{}{}\) being the rotation and translation matrices For more information, see 3D transformations and poses.
-
Transformation step 2: \(\mvVectorVar[c]{p} \to \mvVectorVar[c]{q}\)
If the underlying camera model is an area scan pinhole camera, the projection of \(\mvVectorVar[c]{p} =(\mvVectorVar[c]{x},\mvVectorVar[c]{y},\mvVectorVar[c]{z})^T\) into the image plane is described by the following equation:
\[\begin{eqnarray*} \mvVectorVar[c]{q} = {u \choose v} = \frac{f}{\mvVectorVar[c]{z}}{\mvVectorVar[c]{x} \choose \mvVectorVar[c]{y}} \end{eqnarray*}\]where \(f =\)
Focus. For cameras with hypercentric lenses, the following equation holds instead:\[\begin{eqnarray*} \mvVectorVar[c]{q} = {u \choose v} = \frac{-f}{\mvVectorVar[c]{z}}{\mvVectorVar[c]{x} \choose \mvVectorVar[c]{y}} \end{eqnarray*}\]If an area scan telecentric camera is used, the corresponding equation is:
\[\begin{eqnarray*} \mvVectorVar[c]{q} = {u \choose v} = m {\mvVectorVar[c]{x} \choose \mvVectorVar[c]{y}} \end{eqnarray*}\]where \(m =\)
Magnification. -
Transformation step 3: \(\mvVectorVar[c]{q} \to \mvVectorVar[c]{{\tilde{q}}}\)
For all types of cameras, the lens distortions can be modeled either by the division model or by the polynomial model. The division model uses one parameter
Kappato model the radial distortions.The following equations transform the distorted image plane coordinates into undistorted image plane coordinates if the division model is used:
\[\begin{eqnarray*} \left( \begin{array}{c} u \\ v \end{array} \right) = \frac{1}{1+\kappa (\tilde{u}^{2}+\tilde{v}^{2})} \left( \begin{array}{c} \tilde{u} \\ \tilde{v} \end{array} \right) \end{eqnarray*}\]These equations can be inverted analytically, which leads to the following equations transforming undistorted coordinates into distorted coordinates:
\[\begin{eqnarray*} \mvVectorVar[c]{{\tilde{q}}} = \left( \begin{array}{c} \tilde{u} \\ \tilde{v} \end{array} \right) = \frac{2}{1+\sqrt{1-4 \kappa (u^2+v^2)}} \left( \begin{array}{c} u \\ v \end{array} \right) \end{eqnarray*}\]The polynomial model uses three parameters (\(K_{1}, K_{2}, K_{3}\)) to model the radial distortions and two parameters (\(P_{1}, P_{2}\)) to model the decentering distortions.
The following equations transform the distorted image plane coordinates into undistorted image plane coordinates if the polynomial model is used:
\[\begin{eqnarray*} \left( \begin{array}{c} u \\ v \end{array} \right) = \left( \begin{array}{c} \tilde{u} + \tilde{u} \, (K_{1} r^2 + K_{2} r^4 + K_{3} r^6) + P_{1} (r^2 + 2 \tilde{u}^2) + 2 P_{2} \tilde{u} \tilde{v} \\ \tilde{v} + \tilde{v} \, (K_{1} r^2 + K_{2} r^4 + K_{3} r^6) + 2 P_{1} \tilde{u} \tilde{v} + P_{2} (r^2 + 2 \tilde{v}^2) \end{array} \right) \end{eqnarray*}\]with \(r = \sqrt{\tilde{u}^2 + \tilde{v}^2}\)
These equations cannot be inverted analytically. Therefore, distorted image plane coordinates must be calculated from undistorted image plane coordinates numerically.
-
Additional transformation step for tilt lenses: \(\mvVectorVar[c]{{\tilde{q}}} \to \mvVectorVar[t]{q}\)
If the camera lens is a tilt lens, the tilt of the lens with respect to the image plane is described by the rotation angle rot and the tilt angle tilt.
In this step, you have to further distinguish between different types of tilt lenses as described below. For an overview of different types of tilt lenses, see Tilt lenses and the Scheimpflug principle.
For projective tilt lenses and object-side telecentric tilt lenses (which perform a perspective projection on the image side of the lens), the projection of \(\mvVectorVar[c]{{\tilde{q}}} = (\tilde{u},\tilde{v})^T\) into the point \(\mvVectorVar[t]{q} = (\hat{u},\hat{v})^T\), which lies in the tilted image plane, is described by a projective 2D transformation, i.e., by the homogeneous \(3 \times 3\) matrix \(\mvHomMatrixVar{}{}\) (see
projective_trans_point_2d):\[\begin{eqnarray*} \left( \begin{array}{c} \mvVectorVar[t]{q} \\ {{q}^{t}_{w}} \end{array} \right) = \mvHomMatrixVar{}{} \cdot \left( \begin{array}{c} \mvVectorVar[c]{{\tilde{q}}} \\ {1} \end{array} \right) \end{eqnarray*}\]where \({{q}^{t}_{w}}\) is the additional coordinate from the projective transformation of a homogeneous point.
\[\begin{eqnarray*} \mvHomMatrixVar{}{}= \left( \begin{array}{ccc} h_{11} & h_{12} & h_{13} \\ h_{21} & h_{22} & h_{23} \\ h_{31} & h_{32} & h_{33} \end{array} \right) = \left( \begin{array}{ccc} q_{11} q_{33} - q_{13} q_{31} & q_{21} q_{33} - q_{23} q_{31} & 0 \\ q_{12} q_{33} - q_{13} q_{32} & q_{22} q_{33} - q_{23} q_{32} & 0 \\ q_{13}/d & q_{23}/d & q_{33} \end{array} \right) \end{eqnarray*}\]where \(d =\)
ImagePlaneDistand\[\begin{eqnarray*} \mvVectorVar{Q} &=& \left( \begin{array}{ccc} q_{11} & q_{12} & q_{13} \\ q_{21} & q_{22} & q_{23} \\ q_{31} & q_{32} & q_{33} \end{array} \right) \\ &=& \left( \begin{array}{ccc} (\cos\rho)^2 (1-\cos\tau) + \cos\tau & \cos\rho \sin\rho (1-\cos\tau) & \sin\rho \sin\tau \\ \cos\rho \sin\rho (1-\cos\tau) & (\sin\rho)^2 (1-\cos\tau) + \cos\tau & -\cos\rho \sin\tau \\ -\sin\rho \sin\tau & \cos\rho \sin\tau & \cos\tau \end{array} \right) \end{eqnarray*}\]with \(\rho =\) rot and \(\tau =\) tilt.
For image-side telecentric tilt lenses and bilateral telecentric tilt lenses (which perform a parallel projection on the image side of the lens), the projection onto the tilted image plane is described by a linear 2D transformation, i.e., by a \(2 \times 2\) matrix:
\[\begin{eqnarray*} \mvVectorVar{H} = \left( \begin{array}{cc} h_{11} & h_{12} \\ h_{21} & h_{22} \end{array} \right) = \frac{1}{q_{11} q_{22} - q_{12} q_{21}} \left( \begin{array}{cc} q_{22} & -q_{12} \\ -q_{21} & q_{11} \end{array} \right) \end{eqnarray*}\]where \(\mvVectorVar{Q}\) is defined as above for projective lenses.
-
Transformation step 4: \(\mvVectorVar[t]{q} \to \mvVectorVar[i]{q}\) / \(\mvVectorVar[c]{{\tilde{q}}} \to \mvVectorVar[i]{q}\)
Finally, the point \(\mvVectorVar[c]{{\tilde{q}}} = (\tilde{u},\tilde{v})^T\) (or \(\mvVectorVar[t]{q}\) for tilt lenses) is transformed from the image plane coordinate system into the image coordinate system (the pixel coordinate system):
\[\begin{eqnarray*} \mvVectorVar[i]{q} = {r \choose c} = \left( \frac{\displaystyle \tilde{v}}{\displaystyle S_{y}}+C_{y} \atop \frac{\displaystyle \tilde{u}}{\displaystyle S_{x}}+C_{x} \right) \end{eqnarray*}\]
For line scan cameras, also the relative motion between the camera and the object must be modeled. In HALCON, the following assumptions for this motion are made:
-
The camera moves with constant velocity along a straight line.
-
The orientation of the camera is constant.
-
The motion is equal for all images.
The motion is described by the motion vector \(V= (V_{x},V_{y},V_{z})^{T}\), which must be given in [meter/row] in the camera coordinate system. The motion vector describes the motion of the camera, assuming a fixed object. In fact, this is equivalent to the assumption of a fixed camera with the object traveling along \(-V\).
The camera coordinate system of line scan cameras is defined as follows:
- The origin of the coordinate system is the center of projection (for pinhole cameras) or the center of distortion (for telecentric cameras), respectively.
- The z-axis is identical to the optical axis and directed such that the visible points have positive z coordinates.
- The y-axis is perpendicular to the sensor line and to the z-axis. It is directed such that the motion vector has a positive y-component.
- The x-axis is perpendicular to the y- and z-axis, such that the x-, y-, and z-axis form a right-handed coordinate system.
As the camera moves over the object during the image acquisition, also the camera coordinate system moves relatively to the object, i.e., each image line has been imaged from a different position.
This means there would be an individual pose for each image line.
To make things easier, in HALCON all transformations from world coordinates into camera coordinates and vice versa are based on the pose of the first image line only.
The motion \(V\) is taken into account during the projection of the point \(\mvVectorVar[c]{p}\) into the image.
Consequently, only the pose of the first image line is computed by the operator find_calib_object (and stored by calibrate_cameras in the calibration results).
For line scan cameras, the transformation from world to camera coordinates (\(\mvVectorVar[w]{p} \to \mvVectorVar[c]{p}\)) works in the same way. Therefore, you can also apply transformation step 1 as described for area scan cameras above.
For line scan pinhole cameras, the projection of the point \(\mvVectorVar[c]{p}\), which is given in the camera coordinate system, into (sub-)pixel coordinates \((r,c)\) in the image is modeled as follows:
Assuming \(\mvVectorVar[c]{p} = \mvVectorThreeD{x}{y}{z}\), the following set of equations must be solved for \(m\), \(\tilde{u}\), and \(t\):
were \(u(\tilde{u},\tilde{v})\) and \(v(\tilde{u},\tilde{v})\) are the undistortion functions that are described above for area scan cameras and \(p_{v} = -S_{y} \cdot C_{y}\).
For line scan telecentric cameras, the following set of equations must be solved for \(\tilde{u}\) and \(t\):
with \(u(\tilde{u},\tilde{v})\), \(v(\tilde{u},\tilde{v})\) and \(p_{v}\) as defined above. Note that neither \(z\) nor \(V_{z}\) influence the projection for telecentric cameras.
The above formulas already include the compensation for image distortions.
Finally, the point is transformed into the image coordinate system, i.e., the pixel coordinate system: