stationary_camera_self_calibrationStationaryCameraSelfCalibrationStationaryCameraSelfCalibrationstationary_camera_self_calibrationT_stationary_camera_self_calibration🔗

Short description🔗

stationary_camera_self_calibrationStationaryCameraSelfCalibrationStationaryCameraSelfCalibrationstationary_camera_self_calibrationT_stationary_camera_self_calibration — Perform a self-calibration of a stationary projective camera.

Signature🔗

stationary_camera_self_calibration( integer NumImages, extent.x ImageWidth, extent.y ImageHeight, integer ReferenceImage, integer MappingSource, integer MappingDest, hom_mat2d HomMatrices2D, point.y Rows1, point.x Cols1, point.y Rows2, point.x Cols2, integer NumCorrespondences, string EstimationMethod, string CameraModel, string FixedCameraParams, out hom_mat2d CameraMatrices, out real Kappa, out hom_mat2d RotationMatrices, out point3d.x X, out point3d.y Y, out point3d.z Z, out real Error )

Description🔗

stationary_camera_self_calibrationStationaryCameraSelfCalibration performs a self-calibration of a stationary projective camera. Here, stationary means that the camera may only rotate around the optical center and may zoom. Hence, the optical center may not move. Projective means that the camera model is a pinhole camera that can be described by a projective 3D-2D transformation. In particular, radial distortions can only be modeled for cameras with constant parameters. If the lens exhibits significant radial distortions they should be removed, at least approximately, with change_radial_distortion_imageChangeRadialDistortionImage.

The camera model being used can be described as follows:

\[\begin{eqnarray*} \textbf{x} = \texttt{P} \textbf{X} \enspace . \end{eqnarray*}\]

Here, x is a homogeneous 2D vector, X a homogeneous 3D vector, and P a homogeneous 3x4 projection matrix. The projection matrix P can be decomposed as follows:

\[\begin{eqnarray*} \texttt{P} = \texttt{K}(\texttt{R}|\textbf{t}) \enspace . \end{eqnarray*}\]

Here, R is a 3x3 rotation matrix and t is an inhomogeneous 3D vector. These two entities describe the position (pose) of the camera in 3D space. This convention is analogous to the convention used in camera_calibrationCameraCalibration, i.e., for R=I and t=0 the x axis points to the right, the y axis downwards, and the z axis points forward. K is the calibration matrix of the camera (the camera matrix) which can be described as follows:

\[\begin{eqnarray*} \texttt{K} = \left(\begin{array}{ccc} af & sf & u \\ 0 & f & v \\ 0 & 0 & 1 \end{array}\right) \enspace . \end{eqnarray*}\]

Here, \(f\) is the focal length of the camera in pixels, \(a\) the aspect ratio of the pixels, \(s\) is a factor that models the skew of the image axes, and \((u,v)\) is the principal point of the camera in pixels. In this convention, the x axis corresponds to the column axis and the y axis to the row axis.

Since the camera is stationary, it can be assumed that t=0. With this convention, it is easy to see that the fourth coordinate of the homogeneous 3D vector X has no influence on the position of the projected 3D point. Consequently, the fourth coordinate can be set to 0, and it can be seen that X can be regarded as a point at infinity, and hence represents a direction in 3D. With this convention, the fourth coordinate of X can be omitted, and hence X can be regarded as inhomogeneous 3D vector which can only be determined up to scale since it represents a direction. With this, the above projection equation can be written as follows:

\[\begin{eqnarray*} \textbf{x} = \texttt{K} \texttt{R} \textbf{X} \enspace .\end{eqnarray*}\]

If two images of the same point are taken with a stationary camera, the following equations hold:

\[\begin{eqnarray*} \textbf{x}_{1} & = & \texttt{K}_{1} \texttt{R}_{1} \textbf{X} \\ \textbf{x}_{2} & = & \texttt{K}_{2} \texttt{R}_{2} \textbf{X} \end{eqnarray*}\]

and consequently

\[\begin{eqnarray*} \textbf{x}_{2} = \texttt{K}_{2} \texttt{R}_{2} \texttt{R}_{1}^{-1} \texttt{K}_{1}^{-1} \textbf{x}_{1} = \texttt{K}_{2} \texttt{R}_{12} \texttt{K}_{1}^{-1} \textbf{x}_{1} = \texttt{H}_{12} \textbf{x}_{1} \enspace . \end{eqnarray*}\]

If the camera parameters do not change when taking the two images, \(K_{1}=K_{2}\) holds. Because of the above, the two images of the same 3D point are related by a projective 2D transformation. This transformation can be determined with proj_match_points_ransacProjMatchPointsRansac. It needs to be taken into account that the order of the coordinates of the projective 2D transformations in HALCON is the opposite of the above convention. Furthermore, it needs to be taken into account that proj_match_points_ransacProjMatchPointsRansac uses a coordinate system in which the origin of a pixel lies in the upper left corner of the pixel, whereas stationary_camera_self_calibrationStationaryCameraSelfCalibration uses a coordinate system that corresponds to the definition used in camera_calibrationCameraCalibration, in which the origin of a pixel lies in the center of the pixel. For projective 2D transformations that are determined with proj_match_points_ransacProjMatchPointsRansac the rows and columns must be exchanged and a translation of \((0.5,0.5)\) must be applied. Hence, instead of \(\texttt{H}_{12} = \texttt{K}_{2} \texttt{R}_{12} \texttt{K}_{1}^{-1}\) the following equations hold in HALCON:

\[\begin{eqnarray*} \texttt{H}_{12} = \left(\begin{array}{ccc} 0 & 1 & 0.5 \\ 1 & 0 & 0.5 \\ 0 & 0 & 1 \end{array}\right) \texttt{K}_{2} \texttt{R}_{12} \texttt{K}_{1}^{-1} \left(\begin{array}{ccc} 0 & 1 & -0.5 \\ 1 & 0 & -0.5 \\ 0 & 0 & 1 \end{array}\right) \end{eqnarray*}\]

and

\[\begin{eqnarray*} \texttt{K}_{2} \texttt{R}_{12} \texttt{K}_{1}^{-1} = \left(\begin{array}{ccc} 0 & 1 & -0.5 \\ 1 & 0 & -0.5 \\ 0 & 0 & 1 \end{array}\right) \texttt{H}_{12} \left(\begin{array}{ccc} 0 & 1 & 0.5 \\ 1 & 0 & 0.5 \\ 0 & 0 & 1 \end{array}\right) \enspace . \end{eqnarray*}\]

From the above equation, constraints on the camera parameters can be derived in two ways. First, the rotation can be eliminated from the above equation, leading to equations that relate the camera matrices with the projective 2D transformation between the two images. Let \(\texttt{H}_{ij}\) be the projective transformation from image \(i\) to image \(j\). Then,

\[\begin{eqnarray*} \texttt{K}_{j} \texttt{K}_{j}^{\top} & = & \texttt{H}_{ij} \texttt{K}_{i} \texttt{K}_{i}^{\top} \texttt{H}_{ij}^{\top} \\ \texttt{K}_{j}^{-\top} \texttt{K}_{j}^{-1} & = & \texttt{H}_{ij}^{-\top} \texttt{K}_{i}^{-\top} \texttt{K}_{i}^{-1} \texttt{H}_{ij}^{-1} \end{eqnarray*}\]

From the second equation, linear constraints on the camera parameters can be derived. This method is used for EstimationMethodestimationMethodestimation_method \(=\) 'linear'"linear". Here, all source images \(i\) given by MappingSourcemappingSourcemapping_source and all destination images \(j\) given by MappingDestmappingDestmapping_dest are used to compute constraints on the camera parameters. After the camera parameters have been determined from these constraints, the rotation of the camera in the respective images can be determined based on the equation \(\texttt{R}_{ij} = \texttt{K}_{j}^{-1} \texttt{H}_{ij} \texttt{K}_{i}\) and by constructing a chain of transformations from the reference image ReferenceImagereferenceImagereference_image to the respective image. From the first equation above, a nonlinear method to determine the camera parameters can be derived by minimizing the following error:

\[\begin{eqnarray*} E = \sum_{(i,j) \in \{(s,d)\}} {\left\| \texttt{K}_{j} \texttt{K}_{j}^{\top} - \texttt{H}_{ij} \texttt{K}_{i} \texttt{K}_{i}^{\top} \texttt{H}_{ij}^{\top} \right\|}_{F}^{2} \end{eqnarray*}\]

Here, analogously to the linear method, \(\{(s,d)\}\) is the set of overlapping images specified by MappingSourcemappingSourcemapping_source and MappingDestmappingDestmapping_dest. This method is used for EstimationMethodestimationMethodestimation_method \(=\) 'nonlinear'"nonlinear". To start the minimization, the camera parameters are initialized with the results of the linear method. These two methods are very fast and return acceptable results if the projective 2D transformations \(\texttt{H}_{ij}\) are sufficiently accurate. For this, it is essential that the images do not have radial distortions. It can also be seen that in the above two methods the camera parameters are determined independently from the rotation parameters, and consequently the possible constraints are not fully exploited. In particular, it can be seen that it is not enforced that the projections of the same 3D point lie close to each other in all images. Therefore, stationary_camera_self_calibrationStationaryCameraSelfCalibration offers a complete bundle adjustment as a third method (EstimationMethodestimationMethodestimation_method \(=\) 'gold_standard'"gold_standard"). Here, the camera parameters and rotations as well as the directions in 3D corresponding to the image points (denoted by the vectors X above), are determined in a single optimization by minimizing the following error:

\[\begin{eqnarray*} E = \sum_{i=1}^{n} \Biggl( \sum_{j=1}^{m} {\left\| \textbf{x}_{ij} - \texttt{K}_{i} \texttt{R}_{i} \textbf{X}_{j} \right\|}^{2} + \frac{1}{\sigma^2} (u_{i}^2 + v_{i}^2) \Biggr) \end{eqnarray*}\]

In this equation, only the terms for which the reconstructed direction \(\textbf{X}_{j}\) is visible in image \(i\) are taken into account. The starting values for the parameters in the bundle adjustment are derived from the results of the nonlinear method. Because of the high complexity of the minimization the bundle adjustment requires a significantly longer execution time than the two simpler methods. Nevertheless, because the bundle adjustment results in significantly better results, it should be preferred.

In each of the three methods the camera parameters that should be computed can be specified. The remaining parameters are set to a constant value. Which parameters should be computed is determined with the parameter CameraModelcameraModelcamera_model which contains a tuple of values. CameraModelcameraModelcamera_model must always contain the value 'focus'"focus" that specifies that the focal length \(f\) is computed. If CameraModelcameraModelcamera_model contains the value 'principal_point'"principal_point" the principal point \((u,v)\) of the camera is computed. If not, the principal point is set to \((\textrm{ImageWidth}/2,\textrm{ImageHeight}/2)\). If CameraModelcameraModelcamera_model contains the value 'aspect'"aspect" the aspect ratio \(a\) of the pixels is determined, otherwise it is set to 1. If CameraModelcameraModelcamera_model contains the value 'skew'"skew" the skew of the image axes is determined, otherwise it is set to 0. Only the following combinations of the parameters are allowed: 'focus'"focus", ['focus', 'principal_point']["focus", "principal_point"], ['focus', 'aspect']["focus", "aspect"], ['focus', 'principal_point', 'aspect']["focus", "principal_point", "aspect"], and ['focus', 'principal_point', 'aspect', 'skew'].

Additionally, it is possible to determine the parameter Kappakappakappa, which models radial lens distortions, if EstimationMethodestimationMethodestimation_method \(=\) 'gold_standard'"gold_standard" has been selected. In this case, 'kappa'"kappa" can also be included in the parameter CameraModelcameraModelcamera_model. Kappakappakappa corresponds to the radial distortion parameter \(\kappa\) of the division model for lens distortions (see camera_calibrationCameraCalibration).

When using EstimationMethodestimationMethodestimation_method \(=\) 'gold_standard'"gold_standard" to determine the principal point, it is possible to penalize estimations far away from the image center. This can be done by adding a sigma to the parameter 'principal_point:0.5'"principal_point:0.5". If no sigma is given the penalty term in the above equation for calculating the error is omitted.

The parameter FixedCameraParamsfixedCameraParamsfixed_camera_params determines whether the camera parameters can change in each image or whether they should be assumed constant for all images. To calibrate a camera so that it can later be used for measuring with the calibrated camera, only FixedCameraParamsfixedCameraParamsfixed_camera_params \(=\) 'true'"true" is useful. The mode FixedCameraParamsfixedCameraParamsfixed_camera_params \(=\) 'false'"false" is mainly useful to compute spherical mosaics with gen_spherical_mosaicGenSphericalMosaic if the camera zoomed or if the focus changed significantly when the mosaic images were taken. If a mosaic with constant camera parameters should be computed, of course FixedCameraParamsfixedCameraParamsfixed_camera_params \(=\) 'true'"true" should be used. It should be noted that for FixedCameraParamsfixedCameraParamsfixed_camera_params \(=\) 'false'"false" the camera calibration problem is determined very badly, especially for long focal lengths. In these cases, often only the focal length can be determined. Therefore, it may be necessary to use CameraModelcameraModelcamera_model \(=\) 'focus'"focus" or to constrain the position of the principal point by using a small Sigma for the penalty term for the principal point.

The number of images that are used for the calibration is passed in NumImagesnumImagesnum_images. Based on the number of images, several constraints for the camera model must be observed. If only two images are used, even under the assumption of constant parameters not all camera parameters can be determined. In this case, the skew of the image axes should be set to 0 by not adding 'skew'"skew" to CameraModelcameraModelcamera_model. If FixedCameraParamsfixedCameraParamsfixed_camera_params \(=\) 'false'"false" is used, the full set of camera parameters can never be determined, no matter how many images are used. In this case, the skew should be set to 0 as well. Furthermore, it should be noted that the aspect ratio can only be determined accurately if at least one image is rotated around the optical axis (the z axis of the camera coordinate system) with respect to the other images. If this is not the case the computation of the aspect ratio should be suppressed by not adding 'aspect'"aspect" to CameraModelcameraModelcamera_model.

As described above, to calibrate the camera it is necessary that the projective transformation for each overlapping image pair is determined with proj_match_points_ransacProjMatchPointsRansac. For example, for a 2x2 block of images in the following layout

1	2
3	4

the following projective transformations should be determined, assuming that all images overlap each other: 1->2, 1->3, 1->4, 2->3, 2->4 and 3->4. The indices of the images that determine the respective transformation are given by MappingSourcemappingSourcemapping_source and MappingDestmappingDestmapping_dest. The indices are start at 1. Consequently, in the above example MappingSourcemappingSourcemapping_source \(=\) [1, 1, 1, 2, 2, 3][1, 1, 1, 2, 2, 3] and MappingDestmappingDestmapping_dest \(=\) [2, 3, 4, 3, 4, 4][2, 3, 4, 3, 4, 4] must be used. The number of images in the mosaic is given by NumImagesnumImagesnum_images. It is used to check whether each image can be reached by a chain of transformations. The index of the reference image is given by ReferenceImagereferenceImagereference_image. On output, this image has the identity matrix as its transformation matrix.

The 3x3 projective transformation matrices that correspond to the image pairs are passed in HomMatrices2DhomMatrices2Dhom_matrices_2d. Additionally, the coordinates of the matched point pairs in the image pairs must be passed in Rows1rows1rows_1, Cols1cols1cols_1, Rows2rows2rows_2, and Cols2cols2cols_2. They can be determined from the output of proj_match_points_ransacProjMatchPointsRansac with tuple_selectTupleSelect or with the HDevelop function subset. To enable stationary_camera_self_calibrationStationaryCameraSelfCalibration to determine which point pair belongs to which image pair, NumCorrespondencesnumCorrespondencesnum_correspondences must contain the number of found point matches for each image pair.

The computed camera matrices \(K_{i}\) are returned in CameraMatricescameraMatricescamera_matrices as 3x3 matrices. For FixedCameraParamsfixedCameraParamsfixed_camera_params \(=\) 'false'"false", NumImagesnumImagesnum_images matrices are returned. Since for FixedCameraParamsfixedCameraParamsfixed_camera_params \(=\) 'true'"true" all camera matrices are identical, a single camera matrix is returned in this case. The computed rotations \(R_{i}\) are returned in RotationMatricesrotationMatricesrotation_matrices as 3x3 matrices. RotationMatricesrotationMatricesrotation_matrices always contains NumImagesnumImagesnum_images matrices.

If EstimationMethodestimationMethodestimation_method \(=\) 'gold_standard'"gold_standard" is used, (Xxx, Yyy, Zzz) contains the reconstructed directions \(\textbf{X}_{j}\). In addition, Errorerrorerror contains the average projection error of the reconstructed directions. This can be used to check whether the optimization has converged to useful values.

If the computed camera parameters are used to project 3D points or 3D directions into the image \(i\) the respective camera matrix should be multiplied with the corresponding rotation matrix (with hom_mat2d_composeHomMat2dCompose).

Execution information🔗

Execution information

Multithreading type: reentrant (runs in parallel with non-exclusive operators).
Multithreading scope: global (may be called from any thread).
Processed without parallelization.

Parameters🔗

NumImagesnumImagesnum_images (input_control) integer → (integer)HTuple (Hlong)HTuple (int / long)intHtuple (Hlong)

Number of different images that are used for the calibration.

Restriction: NumImages >= 2

ImageWidthimageWidthimage_width (input_control) extent.x → (integer)HTuple (Hlong)HTuple (int / long)intHtuple (Hlong)

Width of the images from which the points were extracted.

Restriction: ImageWidth > 0

ImageHeightimageHeightimage_height (input_control) extent.y → (integer)HTuple (Hlong)HTuple (int / long)intHtuple (Hlong)

Height of the images from which the points were extracted.

Restriction: ImageHeight > 0

ReferenceImagereferenceImagereference_image (input_control) integer → (integer)HTuple (Hlong)HTuple (int / long)intHtuple (Hlong)

Index of the reference image.

MappingSourcemappingSourcemapping_source (input_control) integer-array → (integer)HTuple (Hlong)HTuple (int / long)Sequence[int]Htuple (Hlong)

Indices of the source images of the transformations.

MappingDestmappingDestmapping_dest (input_control) integer-array → (integer)HTuple (Hlong)HTuple (int / long)Sequence[int]Htuple (Hlong)

Indices of the target images of the transformations.

HomMatrices2DhomMatrices2Dhom_matrices_2d (input_control) hom_mat2d-array → (real)HTuple (double)HHomMat2D, HTuple (double)Sequence[float]Htuple (double)

Array of 3x3 projective transformation matrices.

Rows1rows1rows_1 (input_control) point.y-array → (real / integer)HTuple (double / Hlong)HTuple (double / int / long)Sequence[Union[float, int]]Htuple (double / Hlong)

Row coordinates of corresponding points in the respective source images.

Cols1cols1cols_1 (input_control) point.x-array → (real / integer)HTuple (double / Hlong)HTuple (double / int / long)Sequence[Union[float, int]]Htuple (double / Hlong)

Column coordinates of corresponding points in the respective source images.

Rows2rows2rows_2 (input_control) point.y-array → (real / integer)HTuple (double / Hlong)HTuple (double / int / long)Sequence[Union[float, int]]Htuple (double / Hlong)

Row coordinates of corresponding points in the respective destination images.

Cols2cols2cols_2 (input_control) point.x-array → (real / integer)HTuple (double / Hlong)HTuple (double / int / long)Sequence[Union[float, int]]Htuple (double / Hlong)

Column coordinates of corresponding points in the respective destination images.

NumCorrespondencesnumCorrespondencesnum_correspondences (input_control) integer-array → (integer)HTuple (Hlong)HTuple (int / long)Sequence[int]Htuple (Hlong)

Number of point correspondences in the respective image pair.

EstimationMethodestimationMethodestimation_method (input_control) string → (string)HTuple (HString)HTuple (string)strHtuple (char*)

Estimation algorithm for the calibration.

Default: 'gold_standard'"gold_standard"
List of values: 'gold_standard', 'linear', 'nonlinear'"gold_standard", "linear", "nonlinear"

CameraModelcameraModelcamera_model (input_control) string-array → (string)HTuple (HString)HTuple (string)Sequence[str]Htuple (char*)

Camera model to be used.

Default: ['focus', 'principal_point']["focus", "principal_point"]
List of values: 'aspect', 'focus', 'kappa', 'principal_point', 'skew'

FixedCameraParamsfixedCameraParamsfixed_camera_params (input_control) string → (string)HTuple (HString)HTuple (string)strHtuple (char*)

Are the camera parameters identical for all images?

Default: 'true'"true"
List of values: 'false', 'true'"false", "true"

CameraMatricescameraMatricescamera_matrices (output_control) hom_mat2d-array → (real)HTuple (double)HHomMat2D, HTuple (double)Sequence[float]Htuple (double)

(Array of) 3x3 projective camera matrices that determine the internal camera parameters.

Kappakappakappa (output_control) real(-array) → (real)HTuple (double)HTuple (double)Sequence[float]Htuple (double)

Radial distortion of the camera.

RotationMatricesrotationMatricesrotation_matrices (output_control) hom_mat2d-array → (real)HTuple (double)HHomMat2D, HTuple (double)Sequence[float]Htuple (double)

Array of 3x3 transformation matrices that determine rotation of the camera in the respective image.

Xxx (output_control) point3d.x-array → (real)HTuple (double)HTuple (double)Sequence[float]Htuple (double)

X-Component of the direction vector of each point if EstimationMethodestimationMethodestimation_method \(=\) 'gold_standard'"gold_standard" is used.

Yyy (output_control) point3d.y-array → (real)HTuple (double)HTuple (double)Sequence[float]Htuple (double)

Y-Component of the direction vector of each point if EstimationMethodestimationMethodestimation_method \(=\) 'gold_standard'"gold_standard" is used.

Zzz (output_control) point3d.z-array → (real)HTuple (double)HTuple (double)Sequence[float]Htuple (double)

Z-Component of the direction vector of each point if EstimationMethodestimationMethodestimation_method \(=\) 'gold_standard'"gold_standard" is used.

Errorerrorerror (output_control) real(-array) → (real)HTuple (double)HTuple (double)Sequence[float]Htuple (double)

Average error per reconstructed point if EstimationMethodestimationMethodestimation_method \(=\) 'gold_standard'"gold_standard" is used.

Example🔗

(HDevelop)


* Assume that Images contains four images in the layout given in the
* above description.  Then the following example performs the camera
* self-calibration using these four images.
From := [1,1,1,2,2,3]
To := [2,3,4,3,4,4]
HomMatrices2D := []
Rows1 := []
Cols1 := []
Rows2 := []
Cols2 := []
NumMatches := []
for J := 0 to |From|-1 by 1
    select_obj (Images, ImageF, From[J])
    select_obj (Images, ImageT, To[J])
    points_foerstner (ImageF, 1, 2, 3, 100, 0.1, 'gauss', 'true', \
                      RowsF, ColsF, _, _, _, _, _, _, _, _)
    points_foerstner (ImageT, 1, 2, 3, 100, 0.1, 'gauss', 'true', \
                      RowsT, ColsT, _, _, _, _, _, _, _, _)
    proj_match_points_ransac (ImageF, ImageT, RowsF, ColsF, RowsT, ColsT, \
                              'ncc', 10, 0, 0, 480, 640, 0, 0.5, \
                              'gold_standard', 2, 42, HomMat2D, \
                              Points1, Points2)
    HomMatrices2D := [HomMatrices2D,HomMat2D]
    Rows1 := [Rows1,subset(RowsF,Points1)]
    Cols1 := [Cols1,subset(ColsF,Points1)]
    Rows2 := [Rows2,subset(RowsT,Points2)]
    Cols2 := [Cols2,subset(ColsT,Points2)]
    NumMatches := [NumMatches,|Points1|]
endfor
stationary_camera_self_calibration (4, 640, 480, 1, From, To, \
                                    HomMatrices2D, Rows1, Cols1, \
                                    Rows2, Cols2, NumMatches, \
                                    'gold_standard', \
                                    ['focus','principal_point'], \
                                    'true', CameraMatrix, Kappa, \
                                    RotationMatrices, X, Y, Z, Error)

Result🔗

If the parameters are valid, the operator stationary_camera_self_calibrationStationaryCameraSelfCalibration returns the value 2 (H_MSG_TRUE). If necessary an exception is raised.

Combinations with other operators🔗

Combinations

Possible predecessors

proj_match_points_ransacProjMatchPointsRansac, proj_match_points_ransac_guidedProjMatchPointsRansacGuided

Possible successors

gen_spherical_mosaicGenSphericalMosaic

References🔗

Lourdes Agapito, E. Hayman, I. Reid: ``Self-Calibration of Rotating and Zooming Cameras’‘; International Journal of Computer Vision; vol. 45, no. 2; pp. 107–127; 2001.

Module🔗

Calibration