^{1}

^{2}

^{1}

^{2}

^{3}

^{1}

^{2}

^{1}

^{2}

^{3}

The Scheimpflug camera offers a wide range of applications in the field of typical close-range photogrammetry, particle image velocity, and digital image correlation due to the fact that the depth-of-view of Scheimpflug camera can be greatly extended according to the Scheimpflug condition. Yet, the conventional calibration methods are not applicable in this case because the assumptions used by classical calibration methodologies are not valid anymore for cameras undergoing Scheimpflug condition. Therefore, various methods have been investigated to solve the problem over the last few years. However, no comprehensive review exists that provides an insight into recent calibration methods of Scheimpflug cameras. This paper presents a survey of recent calibration methods of Scheimpflug cameras with perspective lens, including the general nonparametric imaging model, and analyzes in detail the advantages and drawbacks of the mainstream calibration models with respect to each other. Real data experiments including calibrations, reconstructions, and measurements are performed to assess the performance of the models. The results reveal that the accuracies of the RMM, PLVM, PCIM, and GNIM are basically equal, while the accuracy of GNIM is slightly lower compared with the other three parametric models. Moreover, the experimental results reveal that the parameters of the tangential distortion are likely coupled with the tilt angle of the sensor in Scheimpflug calibration models. The work of this paper lays the foundation of further research of Scheimpflug cameras.

The Scheimpflug camera, adopting the Scheimpflug condition by tilting the lens with respect to the sensor [

The Scheimpflug principle, traditionally credited to Theodor Scheimpflug in 1902, states that the object plane (the plane that is in focus), the thin lens’s plane, and the image plane must all meet in a single line, Scheimpflug line. The principle is applicable to both thin prism and thick prism models, which only needs to be modified accordingly [

Scheimpflug camera calibration is a necessary preliminary step to ensure its further high-quality measurement. The conventional calibration methods are not valid in this case because the assumptions used by classical calibration methodologies are not valid anymore for cameras undergoing Scheimpflug condition [

Therefore, more and more researchers have carried out related researches [

Generally, the parametric calibration methods of Scheimpflug camera fall into two categories according to the dimension of the tilted angle, the literature [

Furthermore, the parametric calibration methods with two-dimensional angles are supposed to be divided into two main categories: (1) modified pinhole imaging model (MPIM), as the name implies, is developed on the basis of the conventional pinhole imaging model and taking the imaging characteristics of Scheimpflug camera into account, see literature [

In general, the mainstream parametric camera calibration algorithms can be attributed to the nonlinear parameter optimization problem and the appropriate initialization is the key to the fast convergence and global minimum. The initialization acquisition methods can be classified into the three categories: (1) taking the nominal parameters of the camera as the initial values of the optimization algorithm [

While perspective projection serves as the dominant imaging model in nowadays’ computer vision, conventional camera calibration techniques are taylor made for specific camera model which may not suffice for an unknown imaging system (a black box). Thus, the general imaging models accommodating a wide range of devices have been proposed [

Aiming to offer a comprehensive review that provides an insight into recent calibration methods of Scheimpflug cameras with perspective lens, this paper presents a survey of recent calibration methods of Scheimpflug cameras and analyzes in detail the advantages and drawbacks of the mainstream calibration models with respect to each other. Besides, the general nonparametric imaging model is novelly introduced to solve the problem. Furthermore, Real data experiments are performed to validate the performance of different calibration models which therefore lays the foundation of further research of the Scheimpflug cameras.

Modified pinhole imaging model [

According to the different ways of describing the tilt effect of the lens, the MPIM can be simply divided into three categories: (1) extended distortion model (EDM), which extends the pinhole imaging model by considering the inclination of the sensor as an additional distortion [

Extended distortion model considers the inclination of the sensor as an additional distortion and develops a Scheimpflug camera calibration model based on the classical pinhole imaging model [_{C}Y_{C}Z_{C}_{C}_{0} and _{S}_{S}_{C}_{0}. The 3D world point _{W}_{W}_{W}_{W}_{0} at point _{i}_{S}_{S}

Schematic of imaging system applied with Scheimpflug condition.

The section views of the camera geometry when the image plane, respectively, tilts _{0} while point _{S}_{x}_{y}

Projection view of Scheimpflug camera lens [

According to the derivation of the literature [_{α}_{α}_{β}_{β}

Then, on the basis of conventional distortion model [_{n},_{n})^{T}_{1},_{2}) are the radial distortion coefficients. And it deserves noting that the tangential distortion coefficients are ignored here, as in many of the Scheimpflug camera calibration methods [

The tangential distortion mainly consists of decentering distortion and thin prism distortion. The thin prism distortion usually results from the defects of lens design and manufacture and the slight tilt of sensor plane [

In conclusion, the calibration method based on EDM is logically explicit, and the model is simple and easy to operate which can completely inherit the existing calibration process of the standard camera [

Moreover, the model illustrated here requires that the length of the lens should be provided in advanced [

Rotation matrix model models the lens-sensor configuration by an explicit rotation matrix about the optic axis and includes it as a part of intrinsic calibration parameter set [

As illustrated in Figure _{W}X_{W}Y_{W}Z_{W}_{L}X_{L}Y_{L}Z_{L}_{S}X_{S}Y_{S}Z_{S}_{S}_{u}_{u}_{u}_{d}_{d}_{d}_{W}_{W}

Schematic of imaging system applied with Scheimpflug condition [

Nevertheless, as shown in Figure _{S}_{u}_{d}

As illustrated in Figure _{W}_{S}_{N}_{ij}_{N},y_{N}^{T}_{S},y_{S}^{T}

Schematic of imaging system applied with Scheimpflug condition [

The world point _{W}_{N}_{W}

The gRAC with the rotation matrix can be established by combining formulas (

As far as the paper is concerned, the calibration method based on RMM is flexible, accurate, and robust, which is also easy to operate and initialize. Inevitably, there are still some drawbacks of the method. The analytical solution, as proposed in literature [

Point-line vector model shows that the transformation between the tilted image plane and ideal image plane is established on the basis of the intersections of the light ray and two image planes (ideal image plane and real tilted image plane) (see literature [

As illustrated in Figure _{C}Y_{C}Z_{C}

Schematic of imaging system applied with Scheimpflug condition.

The symbol _{W}(X_{W}_{W}_{W})_{S}_{P}

And the symbol ^{T} denotes the transpose of the vector _{Px}_{Py}_{Sx}_{Sy}_{S}_{S}

Without a loss of generality, assuming that the calibration target plane is on _{1} and _{2} are the first two columns of the rotation matrix

To summarise, the calibration method based on PLVM has the advantage of a robust and simple model, as well as convenient calibration process. However, the application of the model in literature [

The imaging models as described above assume that the angle of the chief ray in object space and the angle of the chief ray in image space are identical, which is incorrect. And this difference employs a significant influence on the Scheimpflug camera, which in general can be ignored in pinhole imaging model. So Kumar and Ahuja [

Figure

The geometry of standard thick lens [

The ray geometry of the real Scheimpflug camera model [

First, the object points are projected to the ideal image plane lying at a distance

Consider the definition of coordinate frames in literature [_{u}y_{u}z_{u}_{t}y_{t}z_{t}_{u}_{S}y_{S}z_{S}_{t}y_{t}z_{t}_{u}y_{u}z_{u}_{S}y_{S}z_{S}_{u}y_{u}z_{u}_{S}y_{S}z_{S}_{S}_{u}^{−1}_{S}_{u}_{S}y_{S}z_{S}_{t}y_{t}z_{t}_{ij}_{θ}_{θ}_{u}y_{u}z_{u}_{t}y_{t}z_{t}

The projection model of the Scheimpflug camera [

Furthermore, the projection from the world coordinate frame to the _{u}y_{u}z_{u}_{t}y_{t}z_{t}

In comparison with the aforementioned three calibration models, PCIM proposed in literature [

Great progress has been made during last decade in exploiting the general nonparametric imaging model for camera calibration [

As proposed in literature [_{i}

Schematic of general nonparametric imaging model.

We consider here a Scheimpflug camera with perspective lens, to validate the performance of GNIM in the Scheimpflug camera calibration. As shown in Figure _{K}_{K}

Furthermore, the optical center of the camera is defined as _{1}, _{2}, _{3}, 1). And without the loss of generality, the first calibration checkerboard is adopted as the global coordinate reference frame.

As the collinearity constraint proposed in literature [_{ij}^{jk}_{i}

On the whole, as has been noted, the general nonparametric imaging model is rather flexible and accommodates a wider range of imaging devices than the specific parametric imaging model. As in the case of Scheimpflug camera calibration, no matter whether the lens is tilted or not, the Scheimpflug camera can be regarded as the same with the ordinary camera in the general nonparametric imaging model. And the concise GNIM avoids the problem that the parameters in the conventional camera calibration methods easily converge to the local optimums.

Nonetheless, the nonparametric nature of GNIM results in the obscure physical meaning of the model. Moreover, the major drawback in generic calibration model, as presented in literature [

Two Scheimpflug imaging systems are calibrated to demonstrate the performance of different calibration models. One imaging system is a DSRL camera (Nikon D300s, 4288 × 2848 pixels, pixel size of 5.5

As for the first imaging system, two sets of experiments are carried out to quantify the performance of calibration models. One set is to adjust the tilt-shift lens to the angle approximate 0°, in the case of which the Scheimpflug camera model reduces to standard camera model. The other set rotates the tilt-shift lens to the angle approximate 7°. In view of the fact that both Scheimpflug imaging systems employed here just simply rotate around one single axis, the tilt angle rotating around the other axis is assumed to be 0°.

Tables _{1},_{2}) are considered, while calibration model (2) includes both the radial and tangential distortion coefficients (_{1},_{2},_{3},_{4}). As illustrated in the following tables, (

Calibration results of the first imaging system with tilt angle of 0° approximately.

Calibration model | Pinhole (1) | Pinhole (2) | RMM (1) | RMM (2) | PLVM (1) | PLVM (2) | PCIM (1) | PCIM (2) |
---|---|---|---|---|---|---|---|---|

— | — | 0.1928 | 0.1214 | 0.2149 | 0.7449 | 0.3175 | 0.2436 | |

— | — | −0.5407 | 2.7449 | 0.7623 | −3.1642 | 0.7844 | 0.6752 | |

1984.8581 | 2044.4909 | 2045.3706 | 2041.9526 | 2045.6208 | 2077.7127 | 2040.3889 | 2044.8924 | |

1499.5190 | 1461.1779 | 1461.8725 | 1446.3428 | 1462.4681 | 1447.5127 | 1458.2369 | 1462.3325 | |

48.0293 | 48.0073 | 48.0056 | 47.8958 | 47.9954 | 48.1980 | 48.1031 | 47.9975 | |

47.9819 | 47.9663 | 48.0024 | 47.9726 | 47.9954 | 48.1980 | 48.1031 | 47.9975 | |

_{1} |
−0.0777 | −0.0785 | −0.0837 | −0.0793 | −0.0852 | −0.0806 | −0.0798 | −0.0802 |

_{2} |
0.1294 | 0.1255 | 0.2126 | 0.1560 | 0.2329 | 0.0922 | 0.2330 | 0.1121 |

_{3} |
— | −0.0018 | — | 0.0022 | — | 0.0015 | — | 0.0019 |

_{4} |
— | 0.0025 | — | 0.0036 | — | 0.0028 | — | 0.0027 |

RMSE (pixel) | 0.1896 | 0.1620 | 0.1623 | 0.1614 | 0.1630 | 0.1608 | 0.1622 | 0.1619 |

Calibration results of the first imaging system with tilt angle of 7° approximately.

Calibration model | Pinhole (1) | Pinhole (2) | RMM (1) | RMM (2) | PLVM (1) | PLVM (2) | PCIM (1) | PCIM (2) |
---|---|---|---|---|---|---|---|---|

— | — | 6.4126 | 5.5814 | 6.3463 | 5.1486 | 6.2329 | 5.4688 | |

— | — | 0.9920 | 1.7603 | 0.8197 | 2.6659 | 1.2703 | 2.2456 | |

1986.0064 | 2099.2407 | 2055.3751 | 2098.9543 | 2058.8085 | 2121.2302 | 2049.6208 | 2140.7542 | |

1883.7801 | 1935.0271 | 1978.2631 | 1869.6842 | 1984.6398 | 1784.6398 | 1974.4682 | 1776.5569 | |

48.2265 | 48.2806 | 48.2024 | 48.5231 | 48.3472 | 48.4253 | 48.1954 | 48.1151 | |

48.4051 | 48.4798 | 48.1955 | 48.5018 | 48.3472 | 48.4253 | 48.1954 | 48.1151 | |

_{1} |
−0.0854 | −0.0824 | −0.0755 | −0.0836 | −0.0694 | −0.0663 | −0.0542 | −0.0945 |

_{2} |
0.2769 | 0.2271 | 0.0795 | 0.0725 | 0.0252 | −0.0085 | −0.0167 | 0.4497 |

_{3} |
— | 0.0022 | — | −0.0022 | — | 0.0107 | — | −0.9368 |

_{4} |
— | 0.0041 | — | 0.0098 | — | 0.0019 | — | 0.0023 |

RMSE (pixel) | 0.2021 | 0.1928 | 0.1701 | 0.1697 | 0.1693 | 0.1690 | 0.1685 | 0.1683 |

Calibration results of the second imaging system with tilt angle of 10° approximately.

Calibration model | Pinhole (1) | Pinhole (2) | RMM (1) | RMM (2) | PLVM (1) | PLVM (2) | PCIM (1) | PCIM (2) |
---|---|---|---|---|---|---|---|---|

— | — | 9.2544 | 3.0078 | 9.1981 | 2.9987 | 9.0974 | 4.6926 | |

— | — | −0.9193 | 0.0248 | 1.0680 | −0.0337 | 268.2745 | 261.3949 | |

1011.6078 | 1015.8023 | 1017.8545 | 1010.8408 | 1014.2669 | 1011.9323 | 1014.2182 | 1009.2238 | |

950.4627 | 373.6203 | 945.0634 | 439.4258 | 921.0598 | 684.9546 | 907.9720 | 424.2683 | |

25.2562 | 25.8743 | 26.3688 | 26.1073 | 26.4134 | 26.1090 | 26.4266 | 26.3146 | |

29.1527 | 27.1610 | 26.0862 | 26.1204 | 26.4134 | 26.1090 | 26.4266 | 26.3146 | |

_{1} |
−0.0643 | −0.0023 | −0.0695 | 0.0010 | −0.0748 | 0.0003 | −0.0454 | 0.0011 |

_{2} |
0.8235 | −1.2101 | 0.6203 | −1.4850 | 0.8104 | −1.4587 | 0.6705 | −1.4361 |

_{3} |
— | 0.0046 | — | 0.0044 | — | 0.0044 | — | 0.0005 |

_{4} |
— | 0.0005 | — | 0.0004 | — | 0.0002 | — | 0.0089 |

RMSE (pixel) | 0.1556 | 0.1548 | 0.1076 | 0.1071 | 0.1078 | 0.1076 | 0.1080 | 0.1078 |

From the calibration results of conventional pinhole model in the above three tables, it can be seen that pinhole model works well with the tilt angle of approximate 0°, yet when the tilt angle increases several degrees (such as 7° and 10°), the reprojection errors increase significantly whereas the aforementioned models almost remain constant. This is also consistent with the conclusion made in literature [

In terms of tilt angles in above tables, the calibration results of RMM (1), PLVM (1), and PCIM (1) are quite close to corresponding tilt angles measured manually in the adapter, in spite of minor deviations resulting from measurement noise. Furthermore, comparing the calibration results of models (1) and models (2), respectively, it is obvious to find that the formers give better results and smaller angle errors than the latters. More comprehensive distortion model in RMM (2), PLVM (2), and PCIM (2) does not necessarily result in better calibration results, and together with the previous analysis, it can be concluded that the parameters of tangential distortion are likely coupled with the tilt angles of the image plane in RMM, PLVM, and PCIM.

It is difficult to obtain high accuracy ground truth that serves as absolute reference for the Scheimpflug camera’s calibration results. Hence, this paper borrows the idea from the literature [

Suppose _{i}_{i}_{i}_{A}_{1},_{2}} and extrinsic parameter set _{B}^{k}^{k}^{i+ 1}^{i}^{i}

And the sub-Jacobian matrix

Assuming that _{m}_{m}^{2}^{2} is the standard deviation of

Thus, we can calculate the covariance matrix of the parameter set

Therefore, the uncertainties of the second imaging system’s calibration parameters using four kinds of models (1) can be obtained by the formula (

The uncertainties of the intrinsic parameters of the second imaging system using models (1).

Calibration model | ||||||
---|---|---|---|---|---|---|

Pinhole | — | — | 2.4277 | 78.7451 | 0.0995 | 0.4525 |

RMM | 0.0781 | 0.0126 | 0.7514 | 2.1001 | 0.0308 | 0.0649 |

PLVM | 0.0776 | 0.0133 | 0.7766 | 2.1731 | 0.0462 | 0.0462 |

PCIM | 0.0811 | 0.0135 | 0.7840 | 2.1417 | 0.0505 | 0.0505 |

As shown in Table

Moreover, the experiments examine the performance of different calibration models with respect to the number of the planes utilized to recover the camera parameters. To facilitate comparison, assuming that the tilt angles around the two axes are 10° and 0°, respectively, the results of tilt angles in different models are transformed into the deviations with respect to the reference values. Figure

Intrinsic parameters results versus the number of the images for different calibration models.

From Figures

As the Figures

Figures

In addition, as depicted in Figure

RMSE versus the number of the images for different calibration models.

On account of the fact that the GNIM does not have explicit intrinsic parameters used for comparison and the limit of pages, we choose the calibration results of any two checkerboards’ motion parameters, as illustrated in Table

Calibration results of motion parameters among calibration checkerboards using different calibration models.

Calibration model | (_{1}^{1}, _{2}^{1}, _{3}^{1}) |
(_{1}^{1}, _{2}^{1}, _{3}^{1}) |
(_{1}^{2}, _{2}^{2}, _{3}^{2}) |
(_{1}^{2}, _{2}^{2}, _{3}^{2}) |
---|---|---|---|---|

RMM (1) | (1.5067, 1.5080, −0.8159) | (−54.7739, −45.2021, 442.8741) | (1.9628, 0.7965, −0.3941) | (−66.5666, −26.0286, 400.8027) |

PLVM (1) | (1.5392, 1.4766, −0.8186) | (−53.7829, −48.3303, 441.7981) | (1.9208, 0.7914, −0.4100) | (−67.8423, −25.1659, 399.2397) |

PCIM (1) | (1.4928, 1.4434, −0.9046) | (−52.5684, −44.1141, 441.0225) | (1.8824, 0.7641, −0.4607) | (−65.8903, −25.5468, 397.2328) |

GNIM | (1.5123, 1.4682, −0.8481) | (−53.3728, −46.9573, 441.6816) | (1.9031, 0.7725, −0.4432) | (−66.7914, −25.6247, 398.4627) |

As shown in Table

Furthermore, the 3D coordinates and structure of the checkerboards are reconstructed with the help of calibrated parameters and obtained image points. As far as the experimental results indicate, the reconstruction results of the checkerboard points using the four different Scheimpflug calibration models are basically in accord with each other, as illustrated in Figure

The reconstructed results and error distribution of the 3D checkerboard points. (a) The reconstructed 3D checkerboard points and the fitted 3D plane. (b) The error distribution of reconstructed points with respect to the fitted plane.

The RMSE of reconstructed points with respect to the fitted plane.

Calibration model | RMM | PLVM | PCIM | GNIM |
---|---|---|---|---|

RMSE (mm) | 0.0300 | 0.0287 | 0.0295 | 0.0311 |

According to the reconstruction results in Figure

As shown in Figure

To further verify the accuracy, another two sets of experiments are carried out with the help of two-axis electric rotary table (SLT-2MA) and three-axis electric translation table (ZG14TA) as shown in Figures

Two-axis electric rotary table (SLT-2MA).

Three-axis electric translation table.

In the first experiment, the two-axis electric rotary table is controlled to a rotation of no less than 5° around the two axes for each rotation. As for the second experiment, we control the three-axis electric translation table to a translation of more than 10 mm for each movement, and both the operations are repeated ten times.

Meanwhile, we estimate the pose change of the cooperated mark via the first Scheimpflug imaging system with different imaging models. In this way, the spatial relationship between the coordinate frames of the checkerboard and the two-axis electric rotary table or three-axis electric translation table can be converted into the hand-eye calibration problem [

The pose estimation errors of the two sets of experiments.

Experiments | Calibration model | RMM | PLVM | PCIM | GNIM |
---|---|---|---|---|---|

1st | RMSE of rotation (°) | 0.0387 | 0.0409 | 0.0400 | 0.0417 |

RMSE of translation (mm) | 1.4659 | 1.5109 | 1.4905 | 1.5305 | |

2nd | RMSE of rotation (°) | 0.0412 | 0.0428 | 0.0424 | 0.0432 |

RMSE of translation (mm) | 1.5562 | 1.6042 | 1.5732 | 1.6655 |

It can be seen in Table

This paper presents a comprehensive survey of recent calibration methods of the Scheimpflug camera with perspective lens. The general nonparametric imaging model is novelly employed to deal with the problem as well. All the calibration models are briefly recalled and compared in detail with respect to each other, with some highlights on their respective advantages and drawbacks. Real data experiments including calibrations, reconstructions, and measurements are performed to validate the performance of the calibration models.

As the experimental results indicated, compared with the classic pinhole imaging model, the models undergoing Scheimpflug condition are the better description of Scheimpflug camera imaging, especially when the tilt angle is greater than 7°. Moreover, although the imaging models and the parameter forms are various, the accuracies of the four calibration models (RMM, PLVM, PCIM, and GNIM) are basically equal, while the accuracy of GNIM is slightly lower compared with the other three parametric models in view of the errors in reconstruction and pose estimation. Given that the PLVM and PCIM require the pixel aspect ratio to be known in advance, the calibration model of RMM is rather flexible. Besides, the experimental results reveal that the parameters of tangential distortion are likely coupled with the tilt angles of the image plane in the calibration models of RMM, PLVM, and PCIM. In the actual calibration task, the appropriate calibration model is supposed to be chosen according to the specific implementation condition.

The authors declare that there is no conflict of interests regarding the publication of this paper.

The research was supported by the National Natural Science Foundation of China (no. 51509251) and National Key Scientific Instrument and Equipment Development Project of China (no. 2013YQ140517).