Adaptive Fringe Projection for 3D ShapeMeasurement with Large Reflectivity Variations by Using Image Fusion and Predicted Search

(ere is always a great challenge for the structured light technique that it is difficult to deal with the surface with large reflectivity variations or specular reflection. (is paper proposes a flexible and adaptive digital fringe projection method based on image fusion and interpolated prediction search algorithm. (e multiple mask images are fused to obtain the required saturation threshold, and the interpolated prediction search algorithm is used to calculate the optimal projection gray-level intensity. (en, the projection intensity is reduced to achieve coordinate matching in the unsaturated condition, and the adaptive digital fringes with the optimal projection intensity are subsequently projected for phase calculation by using the heterodyne multifrequency phase-shifted method. (e experiments demonstrate that the proposed method is effective for measuring the high-reflective surface and unwrapping the phase in the local overexposure region completely. Compared with the traditional structured light measurement methods, our method can decrease the number of projected and captured images with higher modulation and better contrast. In addition, the measurement process only needs two prior steps and avoids hardware complexity, which is more convenient to apply to the industry.


Introduction
Structured light technique has been widely used in academic research and industrial fields because of the advantages in their full-field inspection, noncontact operation, low cost, and high precision [1,2]. However, the method still has some limitations, for example, the measured surface with optical characteristics should have enough diffuse reflection and no large-area specular reflection. In fact, when the coded fringe images are projected to measure the surface with large reflectivity variations, the high-reflective region is too bright to result in image saturation, which will easily lead to large deviation in the three-dimensional (3D) measurement [3,4]. erefore, it is an intractable challenge to measure the object with high-reflective surface accurately.
Many methods have been developed for the 3D shape measurement of the high-reflective surface [5][6][7][8][9][10][11][12][13][14][15][16][17][18][19][20][21][22][23][24]. Among them, an adaptive digital fringe projection (ADFP) method based on antiprojection theory was proposed, and it can adaptively adjust the projection intensity in gray level to make the camera capture the modulated fringes with optimal intensity [17][18][19][20][21][22]. For example, Babaie et al. [19] proposed a method to adjust the projection intensity by calculating the overall corresponding relationships between the projector and the camera, but this method used the global matching relationships to generate the fringe, which inaccurately adjusted the projection intensity. Waddington and Kofman [20] presented a method to be used under the variable ambient light condition that leaded to image saturation by an adapted maximum input gray level (MIGL). However, since the MIGL of the projector was only uniform to avoid image saturation, this method decreased the SNR for surface regions with low reflectivity. Li and Kofman [21] proposed an adaptive fringe projection method through the binary images to extract and compensate the overexposed region. Nevertheless, the phase information of the edge pixels in the overexposed region was easily deficient, and the calculation results were not accurate enough. Holographic matrix was used to calculate the camera coordinate and projector-coordinate mapping function, but it was not suitable for measuring stepped objects [22]. In addition, Li et al. [23] presented an algorithm to calculate the optimal projection gray level. Chen et al. [24] proposed an adaptive fringe projection technique to measure the 3D morphology of color objects. However, this method was sensitive to the ambient light and mostly used in dark conditions.
In this paper, we propose an adaptive digital fringe projection method based on image fusion and interpolated prediction search algorithm to achieve the 3D measurement of the high-reflective surface with large reflectivity variations. According to the reflectivity characteristics of the measured surface, the captured images with valid uniform gray level are fused with the mask images, and the interpolated prediction search algorithm is used to calculate the optimal projection intensity at each pixel. erefore, our method can adaptively adjust gray level intensity, avoid saturation in the captured images, and maintain higher intensity modulation. Compared with traditional optical 3D measurement methods, our method has optimal fringe contrast, which can achieve complete reconstruction in the overexposed region and effectively solve the problem of 3D shape measurement with high-reflective surface. Moreover, it will avoid additional hardware complexity and projector nonlinear gamma compensation. e rest of the paper is arranged as follows. Section 2 demonstrates the measurement principles of the proposed method, and Section 3 illustrates the adaptive digital fringe projection method; then, the measurement system setup and method procedure are described in detail, and we present the experimental results to illustrate the feasibility and reliability of our method in Section 4. e conclusions are presented in Section 5.

Basic
ree-Dimensional Measurement Principle. Figure 1 shows the coordinate system of the phase-shifted fringe projection system, where (O W ; X W , Y W , Z W ) is the world coordinate system. (O P ; X P , Y P , Z P ) and (U P , V P ) are the projector coordinate system and its pixel coordinate system, respectively; similarly, (O C ; X C , Y C , Z C ) and (U C , V C ) denote the camera coordinate system and its pixel coordinate system, respectively. Two arbitrary points M c � (u c , v c ) and M p � (u p , v p ) are taken as a pair, which means that they correspond to the same point M W � (X W , Y W , Z W ) and have the same phase. e correspondences of the paired points can be established by projecting sinusoidal fringe images. e multifrequency heterodyne method is used to unwrap the phase and measure the complex surface since this method is a pixel-by-pixel measurement. e measurement system sequentially projects four kinds of fringe images with different frequency phase shifts, and the phase can be calculated as ϕ 1 , ϕ 2 , and ϕ 3 by means of the four-step phaseshifted method. en, superposition phases ϕ 12 and ϕ 23 are, respectively, obtained and unwrapped phase ϕ 123 will be ultimately calculated from superposition phases ϕ 12 and ϕ 23 by using the heterodyne algorithm. e schematic diagram is shown in Figure 2.

Adaptive Digital Fringe Projection
Principle. In this case, "adaptive" refers to the 3D shape measurement for the surface with large reflectivity variations as well as the influence of ambient light and complex surface interreflections, and it can calculate the optimal projection intensity at the pixel level [25]. e schematic diagram is shown in Figure 3. Our method will generate adaptive digital fringe images based on image fusion and interpolated prediction search algorithm, which are projected onto the measured object so that the captured sinusoidal fringe images are unsaturated, and highprecision measurement results can be obtained by projecting the unsaturated sinusoidal fringes with higher modulation to calculate the phase. e processing flow is further decomposed into six steps, as shown in Figure 4.
(i) Step 1: projecting uniform gray-level image sequences: a series of image sequences in uniform gray level are projected onto the measured surface, and the captured image sequences and the mask image sequences are obtained after acquisition (ii) Step 2: obtaining valid gray-level images and mask images: the image sequences with valid uniform gray level are extracted by the mask image sequences to recalculate the gray level at each pixel and composite the final images (iii) Step 3: determining the saturation threshold: the saturation threshold is set by calculating the maximum pixel gray level in the composite image (iv) Step 4: calculating the optimal projection gray level: the optimal projection gray level is determined through the required saturation threshold and the interpolated prediction search algorithm (v) Step 5: camera-projector coordinate matching: the horizontal and vertical fringe sequences are projected onto the measured surface, and the absolute phase is calculated by the modulated fringe image under the low projection intensity to carry out camera-projector coordinate matching (vi) Step 6: 3D reconstruction: the adaptive digital fringe sequences are projected onto the measured surface, and the unwrapped phase and the 3D reconstruction are achieved by using the phase-shifted method based on heterodyne multifrequency

e Optimal Projection Gray-Level Calculation.
Since the projection intensity images have different brightness, the optimal projection gray level is obtained only if the captured fringe images are unsaturated. In this paper, the saturation threshold is obtained by the image fusion method, and the interpolated prediction search algorithm is used to obtain the optimal projection gray level. Our method can be described as follows: (i) Step 1: project image sequences S i � 255 − K × (i − 1) onto the measured surface, i � 1, 2, . . ., N, and K is the step size. e image sequences have uniform gray level and then are captured by the camera. eir corresponding gray levels are e projection intensity is initially set to a maximum gray level, which is 255. All of the pixels will be in the unsaturated state as the projection intensity, and the constant step K is reduced. (ii) Step 2: apply the threshold segmentation method.
One kind of pixel is considered to be zero if its gray level is greater than T, and the other is maintained as Mathematically, the image sequences are expressed as Among them, the segmentation threshold T is set to 250 to reserve some gray-level space since noise may lead to the saturation of the captured images. (iii) Step 3: design the inverse binary threshold and obtain the valid mask image sequences M i (u c , v c ), i � 1, 2, . . ., N. ey are a binary matrix that will be used for subsequent image fusion, i.e., the mask image sequences are expressed as where the largest threshold I maxval (u c , v c ) is usually set to 1 in which the inverse binary threshold can be realized. e pixel value is defined as the intensity that results from extracting the maximum unsaturated corresponding pixel from all the intensities of captured images I k (u c , v c ).   International Journal of Optics Furthermore, the mask image algorithm has been improved by using the method of transition compensation in the saturation region. e compensation value is gradually reduced, and the reconstruction error in the saturation region boundary is also reduced by adding transition compensation region around the local saturation region. Supposing that I n−max and I s−max are the maximum projection gray level of the saturated region and the adjacent region, respectively. e minimum surrounding ellipse algorithm is added to the transition region. If the length of the overcompensated region is L pixels, the long and short-half axes in the projection plane coordinate system are a and b, and the center coordinates of the ellipse are (C u , C v ); then, the transition overcompensation algorithm is given as follows: In formula (3), the floor represents downward rounding, s is the step size, and Δ is the transition compensation factor, which is used to compensate the incompleteness in the discrete data in the transition region, usually taking Δ as 0.1-0.2.
Step 5: calculate the optimal projection gray level. Due to the disadvantages and complexity of the threshold segmentation algorithm and information acquisition of the pixel value in the overexposed region, the maximum gray level in the composited images is not the optimal gray level of the final adaptive fringe projection images. erefore, the gray-level response curve of the camera-projector system needs to be computed as shown in the following equation: where I cam (u c , v c ) is the gray level of the image captured by the camera, I pro (u p , v p ) represents the projection gray level, p 1 is the modulation factor among the projector, camera, and objects for projection intensity, p 2 represents factors under specific measurement conditions, such as ambient light, and I n is the noise intensity. Formula (5) is a linear function, and the definition domain and the range are both finite and ordered sets, so we put forward the fast lookup algorithm based on interpolated prediction [26] to calculate formula (5). e Lagrange interpolation polynomial L (x) is used to dynamically predict the middle value in the sequence of numbers. Mathematically, the formula is expressed as where Y max represents the maximum gray level in the composited image in Step 4, which equals I cam (u c , v c ). L and H are the width and height of the composited image, which are dependent on the image resolution and initially set as , and then the value of the middle element (mid) is calculated by the value that needs to be looked up, which is the core idea of the interpolation prediction lookup algorithm. Compared with the target element Y max and the value of the middle element in the sequence of numbers, the optimal projection gray level x optimal which is equal to I pro (u p , v p ) can be obtained by dichotomous search and continuous iteration.

Phase Error Analysis.
Two kinds of phase errors are mainly considered in the phase-shifted fringe. One is caused by additive random noise from the system, and the other is caused by the measured object with high reflectivity.
As is known to all that when the gray level is in the normal range, the factors, such as projection intensity, surface reflectivity, ambient light, and noise, can affect the gray level captured by the camera. In fact, all other factors are considered to remain constant except that the noise is a random error, which will affect the final unwrapped phase quality. e relationships between the measurement graylevel random noise I n and the final phase error induced by the random noise can be expressed as the following equation [27]: where N is the number of phase-shifted steps, I m is the modulation intensity, φ(u c , v c ) is the phase, and I n is the random noise of the camera. It can be concluded from equation (8) that both increasing the number of phaseshifted steps and modulation intensity can reduce the phase error under the condition that the image random noise changes very little. Furthermore, we analyze the phase error because the gray-level image is saturated. For the camera with depth in D b bits, the maximum gray level of its captured image is limited to I max � 2 D b . If the captured gray-level intensity exceeds the response range of the camera, the gray-level image will not correctly represent the actual gray-level intensity. As shown in equation (9), the gray-level intensity beyond the camera response range is calculated as the highest gray level of the camera: where I p (u c , v c ) is the gray level of the camera image at the pixel point (u c , v c ) and I c (u c , v c ) is the captured light intensity. If the captured image is saturated, the error of the gray-level image at the saturated pixel point is shown in the following formula: In equation (10), ΔI p (u c , v c ) is the gray-level image error due to the image saturation truncation. Since the error caused by the saturation is much larger than that caused by the noise, the phase error Δφ(u c , v c ) caused by the saturation ΔI p (u c , v c ) is shown in the following equation: International Journal of Optics It can be concluded from equation (11) that the phase error caused by the saturation decreased when the number of phase-shifted steps N and the modulation intensity I m are increased.
From the above analysis, we can draw the conclusion that the phase error caused by saturation is derived from the error value where the gray level is truncated. e phase error at low gray level is derived from the noise, which is much less than the phase error caused by saturation. If the phase can be calculated and the captured images are unsaturated, it can be used to achieve high-precision camera-projector coordinate matching.
3.3. Camera-Projector Coordinate Matching. As mentioned above, the optimal projection intensity only presents the magnitude of the adapted intensity. However, its position in the projector pixel coordinate system is not addressed. erefore, the absolute phase resolution is the key process to map the camera pixel coordinate system to the projector pixel coordinate system.
In our case, a coordinate matching method for the projector-camera system is proposed to reduce the overall projection intensity in the unsaturated state. e method of calculating the optimal gray level can find the optimal projection intensity in Section 3.1 so that the whole intensity of the image is unsaturated. e projector projects the adaptive fringe image sequences onto the measured surface, and the deformed fringe images contain the 3D shape information of the measured object. e phase-shifted method is applied to calculate and unwrap the vertical phase φ v (u c , v c ) and the horizontal phase φ h (u c , v c ). ese calculated phases contain the projection intensity image coordinates (u p , v p ) corresponding to the camera pixel coordinates (u c , v c ). As shown in equations (12) and (13), the phase in the horizontal direction and vertical direction directly corresponds to the projector: where φ v (u c , v c ) and φ h (u c , v c ) are calculated by the phaseshifted method based on heterodyne multifrequency, N v and N h are the periods of the adaptive transverse and longitudinal fringe, and W and H are the width and height of the projected fringe images, respectively. e matching process is shown in Figure 5. After calculating the absolute phase of the orthogonal sinusoidal fringes, the projector and the camera coordinate systems have unique horizontal and vertical phase values at each pixel in the projection area. In this way, the absolute phase value is obtained precisely. According to the phase equality relationships, the pixel-to-pixel correspondences between the projector and the camera are established. Subsequently, the subpixel-level mapping relationships between the projector and the camera can be constructed by traversing the whole projector coordinate system. After that, the phase is calculated from the modulation fringe images, and the matching results are obtained through calculating the phase of the corresponding projection coordinates. Finally, 3D coordinates of the measured surface with high-reflective surface are achieved by the relevant calibration parameters. e mapping accuracy relates to the accuracy of corner extraction and the quality of the absolute phase, rather than affected by the calibration accuracy of camera parameters. e standard extraction functions are used to extract corner coordinates, and we have compensated the absolute phase error and considered the false calculation of the points at the edge of fringes in order to ensure the high quality of the absolute phase. ese methods can ensure the one-to-one correspondence between the camera and the projector pixel coordinates.
Its matching accuracy is shown in Table 1. It can be seen that the overall matching error between the camera and the projector is less than one pixel, and the matching accuracy meets our experimental needs. Moreover, the radial distortion and tangential distortion of the camera are corrected in the calibration, so the internal and external parameters of the camera in this paper have enough accuracy, and the accuracy error is less than 0.02 pixel/mm.
Since both the contour pixel extraction and the pixel gray level of a certain point are complex in the actual optical measuring system, several pixel points around the contour will affect the process. According to the proposed method in Section 3.1, the required low projection intensity x optimal can be obtained, which is satisfied within the dynamic range of the camera, i.e., the highest gray level of the captured image will be unsaturated. en, the adaptive fringe images I i (u, v), i � 1, 2, . . ., N, are generated and obtained by the following equation: where the phase value of the solution φ(u, v) is solved by equation (15) and δ i � i × 2π/N is the phase shift. In equation (14), the values are expressed as the average intensity I ′ (u, v) and the modulation intensity I ″ (u, v) of the fringe, respectively, which can be computed by equations (16) and (17).
e adaptive fringe images can be generated and calculated after mapping the coordinate system. In this case, our proposed method can precisely adjust the pixelwise 6 International Journal of Optics projection intensity, avoid image saturation, and maintain higher intensity modulation for the high-reflective surface.

Experiments and Results
To verify the feasibility and utility of the proposed method, the adaptive digital fringe 3D shape measurement system is set up, as shown in Figure 6. e system was composed of the computer, camera, and digital projector. e digital projector is DLP4500, whose resolution is 1440 pixels × 912 pixels, and the high-speed projection can be realized. e camera is a point gray digital camera, the maximum resolution is 2048 pixels × 1536 pixels, and its maximum frame rate can reach 121 fps. To minimize the influence of ambient light and mutual reflection during the process, the camera was adjusted to have a small aperture, the exposure time was fixed, and the gain was set to 0 dB. en, the exposure time of the camera was preferably set as integer times of 1/fp, where fp was the refresh frequency of the digital projector. Generally, fp was set to 60 Hz. We synchronized the camera and projector by triggering signals, which could ensure the perfect exposure time to cover the correct exposure time of the projection intensity images.
Firstly, a series of uniform gray-level images S i , i � 1, 2, . . ., 9, ranging from 30 to 255 with the constant step size of 30 in the gray level, were projected, and the corresponding images I k (u c , v c ), k � 1, 2,..., 9, were orderly captured. en, the image sequences with valid uniform gray level and the corresponding mask image sequences could be calculated by equations (1) and (2). Because the image noise was unavoidable, the threshold value I k (u c , v c ) of the pixel (u c , v c ) was taken as 248. After that, the mask image M i (u c , v c ) and the valid uniform gray-level image P i (u c , v c ) were fused into image H(u c , v c ) according to equation (4). e process is shown in Figure 7. e gray level of the composited image was taken as the set saturation threshold by the interpolated prediction search algorithm. Combining equations (6) and (7), the intermediate value was predicted dynamically. en, the optimal projection gray level could be obtained quickly by using the dichotomous method, and the value was calculated as 135 in our experiment. After that, the average intensity I′ (u, v) and the modulation intensity I″ (u, v) of the fringe were computed as 0.5, respectively. en, the coordinate mapping relationships were achieved at low gray level to generate the adaptive fringe. After many experiments, the results demonstrate that it was flexible and effective to match the coordinate of the camera-projector in the saturated area when the captured image was unsaturated. e comparison with the fringe projection effects is shown in Figures 8(a) and

Modulation stripe Projection fringe
Absolute phase Fringe pattern Figure 5: Schematic diagram of coordinate matching between the projector and the camera. Table 1: Reprojection error of camera pixels and projection pixels by our proposed method for Figure 5. . From the local details selected in the red frame, it could be seen that our proposed method could make the fringe modulation unsaturated.
As the gray histogram could effectively reflect the frequency of the certain gray level in the image, the gray histogram was commonly used to calculate the distribution of overexposed pixels. In order to highlight the local details of the image, we only extracted the distribution histogram of pixel gray level in the small red frame selection part of Figures 8(a) and 8(b), and contrast histograms are shown in Figures 9(a) and 9(b). It showed that the number of pixels with gray level above 255 was reduced greatly through using the proposed method. erefore, the optimal projection intensity from 128 to 150 had been verified, and it also illuminated that the measured surface could avoid image saturation and maintain higher SNR by using our method.
In our experiment, the phase-shifted method based on heterodyne multifrequency was used to carry out the unwrapped phase. ree groups of fringes were projected,  each of four sinusoidal fringes with different phase shifts. We set the fringe frequencies to λ 1 � 1/70, λ 2 � 1/64, and λ 3 � 1/ 59, and the corresponding wrapped phase values are ϕ 1 , ϕ 2 , and ϕ 3 , respectively. According to the heterodyne principle, the phases of ϕ 12 and ϕ 23 were obtained by superimposing λ 12 and λ 23 . en, the phase with frequencies λ 12 and λ 23 was superimposed to obtain the unwrapped phase ϕ 123 with only one periodic phase in the whole field [28]. e process is shown in Figure 10.
We analyzed the experimental results in extracting 0 to 945 lines from the final phase unwrapping diagram. As could be seen from Figure 11, our method could successfully unwrap the phase in the area where the measured surface is saturated.
Furthermore, the 3D reconstruction was generated by applying the phase-height mapping relationships. e traditional 3D measurement method based on fringe projection and the proposed method were used to measure the same metal workpiece with high-reflective surface. e high-reflective objects with different materials and reflection coefficients were used to realize 3D reconstruction, and the reflection coefficient range was from 0.40 to 0.90 in our experiment.
e reflection coefficient of the first metal workpiece was from 0.40 to 0.50 [29]. We compared the experimental results shown in Figures 12(a) and 12(b); it could draw the conclusion that there was an obvious point cloud-missing phenomenon of the measured surface in the 3D reconstruction map by using the traditional phase-shifted method, and the 3D reconstruction of the measured surface could be completely reconstructed by using our proposed method. e depth-compared result of the 3D point is shown in Figure 13. e measurement output comprised the 3D point cloud, so the quality of the point cloud was the most important indicator to evaluate the performance of the measurement system. Similarly, we measured the workpiece and merged the point clouds after alignment by using the cloud-based triangular-mesh reconstruction algorithm. Furthermore, we compared and analyzed the quality of the model data and the original point-cloud data. We set the deviation value, one side of the reference plane was negative, and the other side was a positive value. e maximum and minimum distance from the point to the plane were calculated by the least square fitting plane. e average error and the standard deviation are calculated as shown in Tables 2 and 3, which were used to evaluate quantitatively the validity of the proposed method.
From the quantitative analysis on Tables 2 and 3, we could demonstrate that the average error and standard deviation of the proposed method in the absolute and the positive directions were smaller than those obtained by the traditional method. e average error of the absolute direction was decreased by 71%.
We have measured a piece of machined aluminum, and its reflection coefficient was from 0.60 to 0.75 through the traditional ADFP method [21] and our proposed method, respectively. e comparison result of reconstruction effect maps is shown in Figures 14(a) and 14(b). e depthcompared result of the 3D point is shown in Figure 15.
From the quantitative analysis of Tables 4 and 5, the average error and standard deviation of the proposed method in the absolute direction and the forward direction were less than the measured values obtained by the conventional method. e average error of the absolute direction was reduced by 84.1%. e forward average error was decreased by 83.7%. e standard deviation of the absolute direction was reduced by 71.6%, and the forward standard deviation was reduced by 69.4%. e data further verified that our proposed method was flexible and adaptive for the 3D measurement of the high-reflective surface.
In addition, the experimental object was a metal processing object with the reflection coefficient of 0.75-0.90. e raw measured object is shown in Figure 16(a). e comparison result of reconstruction effect maps is shown in Figures 16(b) and 16(c). Heterodyne value between group 1 and group 2 Heterodyne value between group 2 and group 3 Final phase unwrapping From the quantitative analysis of Tables 6 and 7, the average error and standard deviation of the proposed method in the absolute direction and the forward direction were both less than the measured values obtained by the conventional method.
All in all, it could be seen from the above experiments that the method proposed in this paper had a wide range of applications and could effectively solve the problem of 3D shape measurement with large reflectivity variations.

Conclusions
An adaptive digital fringe projection method based on image fusion and interpolated prediction search algorithm is proposed to solve the point cloud-missing problem of 3D shape measurement with a high range of reflectivity. For overexposed pixels on high-reflective surfaces, appropriate gray-level intensity of the composite image is computed as the optimal projection intensity to avoid image saturation. Simultaneously, for dark pixels with low surface reflectivity, the high gray-level intensity is selected as the optimal projection intensity, which maintains higher SNR. e experiments show that the proposed method achieved high measurement accuracy for the high-reflective surface. e average error in the absolute direction is reduced by 84.1%, and the forward standard deviation is reduced by 69.4%. Our proposed method only needs two prior steps for measuring the high-reflective surface, without projecting and capturing a large number of fringe images at multiple intensities or too many exposure times; thereby, it avoids additional hardware complexity and makes the whole measurement easier to carry out and less laborious. However, the proposed method cannot be used in dynamic measurements; the optimal projection intensity will be predicted adaptively and achieve high measurement accuracy in the future.

Data Availability
e data used to support the findings of this study are included within the article.

Conflicts of Interest
e authors declare no conflicts of interest.