In order to make the general user take vision tasks more flexibly and easily, this paper proposes a new solution for the problem of camera calibration from correspondences between model lines and their noisy image lines in multiple images. In the proposed method the common planar items in hand with the standard size and structure are utilized as the calibration objects. The proposed method consists of a closedform solution based on homography optimization, followed by a nonlinear refinement based on the maximum likelihood approach. To automatically recover the camera parameters linearly, we present a robust homography optimization method based on the edge model by redesigning the classic 3D tracking approach. In the nonlinear refinement procedure, the uncertainty of the image line segment is encoded in the error model, taking the finite nature of the observations into account. By developing the new error model between the model line and image line segment, the problem of the camera calibration is expressed in the probabilistic formulation. Simulation data is used to compare this method with the widely used planar pattern based method. Actual image sequences are also utilized to demonstrate the effectiveness and flexibility of the proposed method.
Camera calibration has always been an important issue in the field of computer vision, since it is a necessary step to extract metric information from 2D images. The goal of the camera calibration is to recover the mapping between the 3D space and the image plane, which can be separated into two sets of transformations. The first transformation is mapping of the 3D points in the scene to the 3D coordinates in the camera frame, which is described by the extrinsic parameters of the camera model. The second one involves mapping of the 3D points in the camera frame to the 2D coordinates in the image plane. This mapping is described by the intrinsic parameters which models the geometry and optical features of the camera. In general case, these two transformations can be expressed by the ideal pinhole camera model.
Up to now, much work for camera calibration has been done to accommodate various applications. Those approaches can be roughly grouped into two categories according to whether requiring a calibration object. This first type of camera calibration methods is named as metric calibration, which resolves the camera model with the help of metric information of a reference object. Camera calibration is performed by observing a calibration object whose geometry dimension is known with very high precision. The calibration object can be 3D object with several planes orthogonal to each other [
Our approach exploits the line/edge features of the handy objects to calibrate both the internal and external parameters of the camera, since they provide a large degree of stability to illumination and viewpoint changes and offer some resilience to hash imaging conditions such as noise and blur. A first challenge of the solution proposed in this paper is to automatically estimate the homography and establish the correspondences between model and image features. In this sense, we redesigned the model based tracking method [
The remainder of the paper is organized as follows. Section
The proposed algorithm is summarized in this section.
As can be seen in Figure
1D search from the model line to the image line.
The relationship between a model point
Suppose
Assuming a Gaussian distribution for
The conditional density of
Therefore, with the assumption that the observation errors for different sample points are statistically independent, a maximum likelihood estimation of the homography is
It is clear that proposed approach can obtain the maximum likelihood estimation of the homography by minimizing the sum of the square of normal distances.
The derivation of the interaction matrix for the proposed approach is based on the distance between the projection of sample point
Assume that we have a current estimation of the homography
The motion in the image is related to the twist in model space by computing the partial derivative of the normal distance with respect to
Then the corresponding Jacobian matrices can be obtained by
The error vector
The optimization problem for (
Then, the solution of (
Finally, the new homography can be computed according to (
With a series of homography matrices (more than three orientations), the camera parameters can be solved linearly by method [
In this paper, the camera calibration problem can be formulated in terms of a conditional density function that measures the probability of the image observations predicted from the camera parameters given the actual image observations. This section describes how to construct this conditional density function.
Consider the case where there are
Then, the maximum likelihood estimation of the camera parameters
By taking the negative logarithm, the problem of maximize a product is converted into a minimization of a sum, which is given as follows:
The intrinsic parameters of the camera
Throughout this paper, the perspective projection model is utilized. The relationship between a 3D world
As shown in Figure
Perspective projection of 3D line and point.
When noise is present in the measuring data, we denote
Let
Let the noises for the endpoints along the vertical direction be
It is clear that these two noises are negatively correlative. Since the observation noises conform to Gaussian random variables, the joint density for the random variables
Supposing that the length of the line segment is
When the number
Substituting (
From (
The measurement noise for the localization of the 2D line segments can be decomposed into two components: noise perpendicular to the line and noise along the length of the line. The first noise is modelled as a Gaussian random variable related to orientation error and the noise model has been derived in the last section, whilst the second one is assumed to conform to any distribution (not necessarily Gaussian) related to line fragmentation.
As can be seen in Figure
Relation between projection of the 3D line segment and its noisy observation.
It is assumed that the two random vectors
In the literature [
Therefore, with the assumption that the observation errors for different line segments are statistically independent, (
If the image line segment is fitted by LST and the intervals of the edge points are fixed for all of the image line segments, then we have
In this section, we will describe how to employ the nonlinear technique to solve the problem of camera calibration defined in the previous section. In the initial case, the camera parameters can be provided by the method which is similar to [
The distance from the point of the image line segment to the projection of the model line is given by
Assume that we have a current estimation of the rotation
The transformation from the reference frame to the camera frame can be rewritten as
Let
Then, the partial derivative of the error function
The partial derivative of the error function
The error vector
The optimization problem for (
If the incremental motion vector has been calculated, the new camera parameters can be computed as follows:
The proposed algorithm has been tested on simulated data generated by the computer and real image data captured from our smartphone. The closedform solution is yielded by the approach [
The simulated perspective camera is supposed to be 2 m from the plane object. The resolution of the virtual camera is
In this experiment, three planes with
Errors versus the noise level of the image points.
From Figure
In addition, we vary the number of sample points that are utilized to fit the line segment to validate the performance of the 4line based method with
Errors versus the number of sample points for 4line based method.
In this experiment, we investigate the performance of the proposed method with respect to the number of the images of the model planes. In the first three images, we use the same orientation and position of the model plane as those used in the last subsection. For the following images, the rotation axes are randomly chosen in a uniform sphere with the rotation angle fixed to 30° and the positions are randomly selected around
Errors versus the number of the images.
This experiment examines the performance of the proposed method with respect to the number of the lines utilized to recover the camera parameters. For our method, more than 4 lines should be employed. We vary the number of lines from 4 to 25. Three images of the model plane are also used with the same orientation and position as last subsection. 100 independent trials are conducted with the noise level fixed to 0.5 pixels for each number of the lines. The results are shown in Figure
Errors versus the number of the lines.
This subsection investigates the influence of the orientation of the model plane with respect to the image plane. In the experiment, three images are used with two of them similar to the last two planes in Section
Errors versus the orientations of the plane.
For the experiment with real data, the proposed algorithm is tested on several image sequences captured from the camera of the smartphone.
In the experiment, three image sequences are captured from the smartphone with a resolution of
Homography tracking for the chessboard plane. Projected model lines using the homography of the proposed method are drawn in blue. The interior corners extracted in the images are drawn as color circles with the color lines between them denoting the order. The first row shows that the corners can be extracted successfully when the angle between the model plane and the image plane is small. However, when the angle is increasing, the corner detection will be less precise and even fail, as can be seen in the second row. For the whole sequence, the proposed method can provide good match with the image edges.
Frame 220
Frame 480
Frame 640
Frame 973
Frame 1073
Frame 1113
Frame 1153
Frame 1213
Frame 1433
Frame 1533
In the last two image sequences, the covers of two books are chosen as the model planes, respectively. To validate the performance of the proposed homography tracking method, the books are put in the clutter environment with the smartphone undergoing large rotation and translation. Both of the last two sequences contain around 2000 images. Figure
Homography tracking for the book covers in clutter environment.
Planar object 1
Planar object 2
In this subsection, we applied our calibration technique and the corners based method to the four images sampled from the video captured by our smartphone (shown in Figure
Calibration results with real data of 4 images.
Methods/pixels 




Reprojection error 

Corners based method  1147.23  1146.68  475.39  258.04  0.41 
Lines based method  1150.76  1151.49  474.61  262.60  0.42 
4line based method  1141.14  1139.99  480.25  253.07  0.67 
Four images of model plane for camera calibration.
In order to further investigate the stability of the proposed method, we vary the number of lines from 4 to 23. The results are shown in Figure
Results versus the number of the model lines.
In this subsection, we applied the proposed method on two image sequences. In the first image sequence, the card with the size of 54.0 mm × 85.6 mm is utilized as the model object. The A4 paper with the size of 210 mm × 297 mm is chosen as the model object for the second image sequence. In the experiment, a series of images are sampled from the videos to calibrate the camera intrinsic parameters and then the camera pose is optimized for each image frame. After that, the structure from motion developed by the methods [
Results of imagebased modelling. The green lines are the projections of the four edges of the calibration items by using the recovered camera parameters. The green crosses are the projections of the model points. We can see that the proposed method provides good matches of the projections of the model lines with the observed positions in the images.
Reconstruction results of Luffy
Reconstruction results of Hulk
In practice, the corners detection often suffers from a failure, when the angle between the model and image plane is large or when some of the corners are invisible or corrupted by the image noise and blur. However, the edge detection is more stable in such case. Moreover, in the simulated experiments, since the line segment is fitted by the corners lying on it, the proposed method provides almost the same performance with the corners based method. In our homography tracking framework, much more image edge points corresponding to the sample model points are utilized, and therefore the line segment can be fitted with higher accuracy. In addition, the proposed method is more flexible and suitable for the general user of the smartphone who wants to take the vision task, since it only uses the common and handy planar object rather than the prepared planar pattern.
In this paper, we have investigated the possibility of camera calibration using common and handy planar objects undertaking general motion for the smartphone vision tasks. A linear algorithm supported by the edge model based homography tracking is proposed, followed by a nonlinear technique to refine the results. Both the computer simulated and real images have been utilized to validate the proposed algorithm. The experimental results exhibited that the proposed algorithm is valid and robust and provides more flexible performance than the widely used planar pattern based method.
In addition, for the general user who will do vision task, the prepared planar calibration may be not always in hand. However, the common items in our daily life almost have the standard size and planar structure. By exploiting the edge information, we proposed an easier and practical camera calibration method. Moreover, in the proposed method, the uncertainty of the image line segment was encoded in the error model, which takes the finite nature of the observations into account. The problem of camera calibration using lines was formalized in the probabilistic framework and solved by the maximum likelihood approach.
The authors declare that there is no conflict of interests regarding the publication of this paper.
The research was supported by the National Basic Research Program of China (2013CB733100) and National Natural Science Foundation of China Grant (no. 11332012).