Developing matching algorithms from stereo image pairs to obtain correct disparity maps for 3D reconstruction has been the focus of intensive research. A constant computational complexity algorithm to calculate dissimilarity aggregation in assessing disparity based on separable successive weighted summation (SWS) among horizontal and vertical directions was proposed but still not satisfactory. This paper presents a novel method which enables decoupled dissimilarity measure in the aggregation, further improving the accuracy and robustness of stereo correspondence. The aggregated cost is also used to refine disparities based on a local curve-fitting procedure. According to our experimental results on Middlebury benchmark evaluation, the proposed approach has comparable performance when compared with the selected state-of-the-art algorithms and has the lowest mismatch rate. Besides, the refinement procedure is shown to be capable of preserving object boundaries and depth discontinuities while smoothing out disparity maps.
Stereo vision is the technique of constructing a 3D description of the scene from stereo image pairs, which is important in many computer vision tasks such as inspection [
For passive stereo vision systems, stereo matching algorithms are crucial for correct and accurate depth estimation, which find for each pixel in one image the corresponding pixel in the other image. A 2D picture of displacements between corresponding pixels of a stereo image pair is named as a disparity map [
Reference [
Cost aggregation can be local [
Local methods tend to be sensitive to noise; however, and its correctness at regions with sparse texture or near depth discontinuities relies on proper selection of window size. To overcome this problem, [
Recent years have seen adaptive support weight approaches [
To simplify computation, [
An effective local stereo matching algorithm is introduced in [
In this paper, we present an improved stereo matching algorithm. Similar to [
The main contribution of this paper is to afford a decoupled aggregation algorithm to access the stereo matching cost under the framework of SWS. The algorithm is simply yet efficient as well as robust. In addition, the resultant disparity map is in a discrete space, which is unsuitable for image-based rendering. We propose a subpixel refinement technique that employs inferior candidate disparities, rather than spatial neighbors, to smooth out discrete values in the disparity map. By this arrangement for curve-fitting, even regions near discontinuous depth can be correctly refined. Moreover, this technique increases the resolution of a stereo algorithm with marginal additional computation.
Our stereo matching algorithm consists of three main stages. First, initial cost values are calculated based on dissimilarity measure between pixels in the reference and target images, and the costs are aggregated using the proposed method. Second, we perform initial disparity estimation by the use of a winner-takes-all minimum search based on the aggregated costs. Third, we check differences between the disparity values of corresponding pixel pairs for the existence of obscured regions and patch them by the smallest disparity values of nearby regions. Finally, the disparity map is refined by a proposed curve-fitting procedure.
Assuming that the image pair is rectified and horizontally aligned, two dissimilarity measures between the pixels on the reference image and the target image are used in this work.
The truncated absolute difference cost,
Besides, the truncated absolute gradient difference cost is defined as
Aggregation of primary costs determines correctness and accuracy of disparity estimation [
Once the
In (
The values of
The aggregation is a two-dimensional convolution. To reduce the computational complexity, each convolution is further decomposed into four one-dimensional convolutions [
Let us take the absolute difference part of
We have that
Note that, in (
In the vertical direction, if we define the top-to-bottom and bottom-to-top weighted sums,
In calculating (
Hence, the first part of the aggregated cost,
With similar procedure, we have that
These terms can all be written in the following recursive form:
In the last section, the matching cost is aggregated through weighted summation over the entire image, nd the disparity which provides minimum cost is assigned to the corresponding pixel. That is, the assigned disparity for a pixel in the reference image,
The initial disparity map normally contains obscured outlier regions. The disparities at these regions are significantly different when they are found from different reference images. If
Once the obscured regions are found, occlusion handling [
The disparity map obtained by the method proposed in the last section is discrete, since the disparity search space is an integer set. We propose a smoothing technique for the disparity refinement in this subsection.
Considering that the initial disparity has the smallest aggregation cost in the potential solution space, we may interpolate for refined value by fitting data sets to upward curves. Besides, rather than directly using the neighboring disparity for refinement, we use both the costs and disparity in the curve-fitting.
Assuming that
Firstly, the disparity-cost sets around the pixel,
As the minimum value of the curve is located at
Secondly, an upward parabola equation
The averaged value,
In contrast to the approach of [
The aggregated cost function of [
A performance comparison between the proposed method and the algorithm of [
A comparison of disparity estimation performance between [
In Figure
Comparison of percentage of miss-matches among the proposed algorithm and [
Image set | Algorithm | |
---|---|---|
Proposed | InfoPerm [ | |
Tsukuba | 3.53% | 5.99% |
Venus | 0.34% | 1.32% |
Teddy | 6.09% | 6.20% |
Cones | 3.60% | 4.58% |
In addition to InfoPerm [
Figure
Top-to-bottom:
The complete comparison of the mismatch rates between these algorithms is summarized in Table
Comparison of mismatch percentage among the proposed algorithm and several representative algorithms.
Algorithms | Tsukuba | Venus | Teddy | Cones | Average percent of mismatch | ||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|
nonocc. | all | disc. | nonocc. | all | disc. | nonocc. | all | disc. | nonocc. | all | disc. | ||
SNCC [ |
11.3 | 12.3 | 27.5 | 2.35 | 3.23 | 15.4 | 10.6 | 15.2 | 28.6 | 4.71 | 11.1 | 13.2 | 13.0 |
HistAggr2 [ |
15.2 | 15.7 | 16.4 | 6.44 | 7.00 | 12.5 | 11.5 | 18.2 | 24.4 | 7.90 | 13.7 | 14.2 | 13.6 |
RTCensus [ |
12.9 | 14.1 | 28.1 | 3.67 | 4.63 | 17.8 | 11.4 | 18.6 | 27.7 | 5.54 | 11.8 | 15.9 | 14.4 |
InfoPerm [ |
25.7 | 26.2 | 21.2 | 8.64 | 9.34 | 15.0 | 15.0 | 22.1 | 29.2 | 7.68 | 15.1 | 15.1 | 17.5 |
AdaptWeight [ |
18.1 | 18.8 | 18.6 | 7.77 | 8.40 | 15.8 | 17.6 | 23.9 | 34.0 | 14.0 | 19.7 | 20.6 | 18.1 |
FeatureGC [ |
8.22 | 8.86 | 13.3 | 4.58 | 4.73 | 10.1 | 14.7 | 17.0 | 32.5 | 11.5 | 18.0 | 21.0 | 13.7 |
ObjectStereo [ |
16.4 | 16.8 | 16.1 | 2.56 | 2.67 | 7.69 | 19.6 | 22.7 | 30.3 | 16.3 | 20.7 | 19.7 | 16.0 |
AdaptingBP [ |
19.1 | 19.3 | 17.4 | 4.84 | 5.08 | 7.84 | 12.8 | 16.7 | 26.3 | 7.02 | 13.2 | 14.0 | 13.6 |
Proposed | 9.12 | 10.2 | 15.0 | 1.04 | 1.72 | 8.35 | 11.2 | 17.0 | 26.8 | 6.36 | 12.6 | 15.1 | 11.2 |
nonocc.: the pixels in the nonoccluded region; disc.: the visible pixels near the discontinuous regions.
It is also interesting to note that the local stereo matching algorithms, such as RTCensus [
Stereo matching algorithms are crucial for correct and accurate depth estimation in passive stereo vision systems. A stereo matching algorithm processes rectified stereo image pairs to generate the disparity map, which is used to calculate the depth image (z-map), and hence the 3D point cloud in camera coordinates. For practical applications, the algorithms should require less computational resources and provide precise disparity maps.
In this paper, we proposed an efficient stereo matching algorithm and a refinement strategy for the disparity maps. The algorithm effectively aggregates cost values in terms of bilateral filtering by only four passes along regions, which is able to provide a decoupled dissimilarity measure aggregation while preserving computational efficiency. Besides, the refinement strategy is a simple application of the aggregated costs that use both the costs and disparity in the curve-fitting, rather than directly using the neighboring disparity for refinement.
Experimental results using the Middlebury stereo test bench [
The authors declare that there is no conflict of interests regarding the publication of this paper.
This paper was sponsored by Chang Gung Memorial Hospital, Chang Gung University, and the National Science Council, Taiwan, under Contract nos. CMRPD1C0021, CMRPD2C0051, NSC 100-2221-E-182-008, NSC 101-2221-E-182-006, and NSC 102-2221-E-182-073 and the National Science Foundation of China, under Contract no. 61271326.