A Review of Machine Vision-Based Structural Health Monitoring: Methodologies and Applications

In the past two decades, a significant number of innovative sensing and monitoring systems based on the machine vision-based technology have been exploited in the field of structural health monitoring (SHM). This technology has some inherent distinctive advantages such as noncontact, nondestructive, long distance, high precision, immunity to electromagnetic interference, and large-range and multiple-target monitoring. A lot of machine vision-based structural dynamic measurement and structural state inspectionmethods have been proposed. Real-world applications are also carried out tomeasure the structural physical parameters such as the displacement, strain/stress, rotation, vibration, crack, and spalling. The purpose of this review article is devoted to presenting a summary of the basic theories and practical applications of the machine vision-based technology employed in structural monitoring as well as its systematic error sources and integration with other modern sensing techniques.


Introduction
The technology of structural health monitoring (SHM) emerged with an essential goal of safeguarding the operational safety of engineering structures, through deploying various types of sensors, monitoring diversified physical quantities, assessing structural condition and performance, and instructing routine inspection and maintenance [1][2][3][4][5][6].With regard to a large-scale SHM system, the innovative sensing technologies from a variety of fields, such as mechanics, electricity, electromagnetism, optics, thermology, and chemistry, make great contribution in accurately acquiring the huge amount of original data reflecting the real environmental and structural conditions.In the past three decades, worldwide researchers have devoted a considerable number of efforts in the development of novel sensing technologies for application in the SHM research field and achieved tremendous progresses.
In the past years, review works referring to the machine vision-based structural monitoring and condition assessment were carried out by some researchers.Wu and Casciati [24] gave a brief introduction of the vision-based positioning system for structural monitoring.Jiang et al. [25] reviewed the development and application of close-range photogrammetry in deformation and geometry measurement of bridges.Koch et al. [26] presented a comprehensive synthesis of the state of the art in the concrete and asphalt structure defect detection and condition assessment based on the computer vision technology.However, an all-round summary of the visionbased structural monitoring and condition assessment is still desirable.This paper aims to provide a comprehensive review of machine vision-based monitoring of civil engineering infrastructure focusing on the relevant methodologies and practical applications.

Machine Vision Methods
A vision-based measurement system generally consists of the image acquisition device (digital camera, lens, and image grabber), the computer, and an image processing software platform.In these mentioned components, the image processing software platform acts as the critical part which will be integrated with the specific computational algorithms to obtain the mechanical parameters in structural monitoring.

Image Processing Algorithms.
As illustrated in Figure 1, the images including the predefined targets are captured by the digital camera.With the digital image processing and pattern matching algorithm [27][28][29][30], the targets are tracked and the structural displacements at the target positions on the structure can be obtained.In this occasion, the horizontal and vertical displacements, called two-dimensional (2D) displacements, can be obtained with one digital camera by use of the appropriate image processing method, such as digital image correlation [31], mean-shift tracking algorithm [32], CamShift tracking algorithm [33], and Lucas-Kanade method [34].
As illustrated in Figure 2, when two cameras are used to capture the targets simultaneously and the geometrical relationship between the two cameras (camera  and camera b) is confirmed, the displacements of the targets in three coordinate directions, called three-dimensional (3D) displacements, can be derived by reconstructing the actual spatial displacements with the position changes on the two confirmed camera coordinates (the coordinates of Plane  and Plane b) and the relationship between the two cameras [35].
For the image processing algorithms applied in structural monitoring, a considerable amount of research has been carried out over the last decades.Wang et al. [36] presented a method to get the displacement results from the interferometric images by use of the phase-shifted image matching algorithm.Pieraccini et al. [37] obtained the structural displacement of real-scale buildings from the images captured by a microwave interferometer.Guo and Zhu [34] proposed a modified inverse compositional algorithm to reduce the computing time of the Lucas-Kanade template tracking algorithm and improved the efficiency of computer vision method further in the remote measurement of dynamic displacement.Fukuda et al. [38] developed a robust object search algorithm enabling accurate displacement measurement through tracking existing features on the structure.
Busca et al. [39] used two types of cameras to acquire multiple targets fixed on a railway bridge during the passage of a train and obtained the displacement responses by three vision methods, that is, digital image correlation method, edge detection method, and pattern matching method.Lee et al. [40] proposed a pose-graph optimized displacement estimation method for reducing the estimation errors of a visually served paired structured light system.Nayyerloo et al. [41] developed a vision system to monitor the seismic response of structures with a line scan camera.Chan et al. [42] proposed a CCD camera-based method to measure the vertical displacement of bridges.Santos et al. [43] performed the vision system calibration in structural displacement measurement of long-deck suspension bridges.

Systematic Errors Assessment and Reduction.
Different kinds of errors will occur in the application of visionbased measurement system.It is critical to find the influence factors, assess the systematic errors, and develop appropriate algorithms to reduce the errors.Lava et al. [44,45] estimated the errors of system in the digital image correlation procedure for large plastic deformation monitoring and investigated the influence of different causes, including the subset shape function interpolation order, adopted correlation coefficient, and subset size.Schreier et al. [31,46] analyzed the errors caused by the use of undermatched shape function and grayvalue interpolation in structural displacement measurement.Bornert et al. [47] studied the displacement errors assessment from synthetic speckle images and identified various error regimes.Yoneyama et al. [48] evaluated the effects of lens distortion on the displacement measurement and proposed a correction method when using the digital image correlation.Yu and Pan [49] investigated the errors due to the overmatched subset shape function via the numerical tests with deformation and the images from real experiments with high strain gradients.Baldi and Bertolino [50] conducted an experimental study to describe the errors caused by interpolation options.Zhou et al. [51] presented a method about the adaptive image subset offset to decrease the system errors in the incremental image correlation.Crammond et al. [52] investigated the relationship among the size and density of speckles and the measurement error within a pattern and identified that the physical properties had a significant impact on the precision of the displacement measurement.Yaofeng and Pang [53] investigated the effect of subset size on the accuracy of deformation measurement when using a digital image correlation algorithm.Lava et al. [54] estimated the errors produced when the camera alignment was nonperpendicular to planar sheet metal specimen's surface in a numerical experiment.Lecompte et al. [55] stated that the size of the speckle and the used pixel subset notably affected the error magnitude of the displacement measurement.
Santos et al. [56] proposed a vision system calibration approach to obtain an initial estimation of the object shape and camera parameters to reduce measurement errors.Ribeiro et al. [57] presented a video-based system for the dynamic displacement measurement of railway bridges and investigated several influence factors which will affect the measurement precision of the video-based system.Ma et al. [58] studied the error of strain measurement in digital image correlation method caused by self-heating of digital cameras.Haddadi and Belhabib [59] investigated the strain measurement errors due to digital image correlation technique with rigid-body motion.Fazzini et al. [60] estimated the errors due to digital image correlation in displacement measurement based on the generation of composite image models of genuine speckle patterns.Wu et al. [61] mounted a vision system to monitor the 2D plane dynamic response of a reduced scale frame fixed on a shaking table and discussed the physical meanings of the camera parameters, the balance between the system resolution and its field-of-view, and the upper limitation of maker density which would restrict the systematic error and measurement resolution.
The risk of the measurement uncertainty in the application of vision-based techniques to vibrating target measurements is very likely increased due to the motion blur generated by the camera-target motion.The motion blur will lead to significant systematic errors and incomplete measurement data because the target seeking process may not give exact detection.In recent years, research efforts have been devoted to the development of deblurring and denoising algorithms and blur image analysis methods [62][63][64].Wang et al. [65] proposed a method for vibration measurement based on the blurred images with the aid of the relationship between the geometric moments of the unblurred and blurred motion.Peng et al. [66] developed an image restoration method for the improvement of the quality of dynamic particle images for the purpose of solving the motion-blurred problem in an online particle imaging system for wear debris analysis.Becker [67] conducted a study of motion blur evaluation by use of different basic approaches and instruments with a variety of parameter variations.Wu et al. [68] presented a row by row degradation model of the images and developed a restoration approach to compensate the space-variant degradation.Ishida et al. [69] proposed a method to improve the recognition accuracy of camera-captured characters without restoring images.

Two-Dimensional (2D) Structural Displacement Monitoring.
With the image sequence, pattern matching algorithm, edge detection algorithm, and other image processing technologies, the structural displacement of predefined targets can be obtained.This can be used to obtain the dynamic displacement of several selected points on a certain structure for the purpose of structural monitoring.Feng et al. [70] proposed a vision system for noncontact structural displacement measurement in real time with the aid of an advanced template matching algorithm.Henke et al. [71] measured the deformation of building structures by use of digital image processing technique regarding the LED as the vision target.Park et al. [72] proposed a displacement measurement method based on machine vision technology to monitor the displacement of high-rise building structures by use of the partitioning approach and the verification experiments were conducted on a flexible steel column.Jáuregui et al. [73] employed the close-range terrestrial digital photogrammetry to measure the vertical deflection of bridges.Yoneyama et al. [74] used the digital image correlation to monitor the deflection of a new-built steel girder bridge during load tests.Kohut et al. [75] validated the feasibility and precision of a vision method for the measurement of steel bridge displacement.Dworakowski et al. [76] used the bridge deflection curve obtained by the in-plane displacement measurement based on the vision method to analyze the damage of the cantilever beam structures.
Lee et al. [77] employed the digital image processing techniques to obtain the real-time displacement of bridges and assess the bridge load carrying capacity.Ho et al. [78] developed a synchronous vision system for the real-time multipoint displacement measurement of civil infrastructure.Yang et al. [79] proposed an image-based method to measure the structural displacement, plane strain field, and cracks on the surface of the specimens under seismic loads.Fu and Moosa [80] proposed an optical method for displacement measurement with a high-resolution CCD camera.Olaszek [81] presented a computer vision method for realtime measuring the structural displacement and dynamic characteristics of bridges.Lee and Shinozuka [82] proposed a vision-based system to measure the dynamic displacement of bridges in real time with the aid of digital image processing techniques.Wahbeh et al. [83] developed a vision-based method to measure the absolute displacements in real time at selected locations of infrastructure.
The great advantage of the vision-based structural displacement measurement method is that the measurement targets can be multiple as soon as they are in the captured images.Choi et al. [84] introduced a dynamic displacement vision system which could perform the multimeasurement positions using a handset digital camcorder and the region of interest (ROI) was proposed to improve the measurement efficiency.Jurjo et al. [85] measured the large displacement at several points of the membrane simultaneously and estimated the strain and stress from the measured displacement.Lin et al. [86] presented a videogrammetry system to monitor the dynamic behavior of membrane roof structures.Lee and Shinozuka [87] developed a real-time vision-based system for structural displacement measurement of bridges by use of digital image processing techniques.

Three-Dimensional (3D) Structural Displacement Monitoring.
As mentioned in Section 3.1, the 2D (or in-plane) structural displacement can be obtained with the image sequence captured by only one camera.Combined with two or more digital cameras, two different image sequences from two shooting angles are captured, and the three-dimensional (in-plane and out-plane) structural displacement of selected points on a certain structures can be realized with vision reconstruction techniques.Grano and Zinno [88] designed a computer vision system for displacement monitoring during destructive tests.Park et al. [35] presented an approach to monitor the three-dimensional structural displacements with the aid of a high speed motion-capture system which has the advantages of high accuracy and high sampling rate.Jeon et al. [89] proposed a vision system with an artificial marker to monitor six-degree-of-freedom (6-DOF) structural displacements.Park et al. [90] used a motion-capture system to obtain the 3D displacement response of structures in wind tunnel experiments and obtained the dynamic properties of the test structure, including the natural frequency, mode shape, and damping ratio.Jeon et al. [91] developed vision system to measure the 6-DOF structural displacement based on the paired visual servoing method.Leifer et al. [92] performed three-dimensional acceleration measurement by a videogrammetry system through tracking the motion of targets on a modal shaker.Synnergren and Sjödahl [93] developed a photography system with stereoscopic digital speckles for 3D displacement field measurements and a camera calibration algorithm was used to evaluate the effect of lens distortion.Viéville and Lingrand [94] developed a visual motion perception module to estimate 3D displacements without calibration.
Hu et al. [95] proposed a 4-camera video system for 3D motion measurement of deformable objects.Ji and Chang [96] presented a marker-free stereovision method to monitor responses of the line-like structures in both spatial and temporal domains.Chang and Ji [97] proposed a videogrammetric method based on the principle of digital photogrammetry in close-range and computer vision technology to measure the 3D structural vibration response in lab and proposed a two-step calibration process to overcome the lens distortion problem. Lee et al. [98] developed a vision-based displacement measurement system with digital image processing techniques for real-time structural health monitoring of civil structures.Chang and Xiao [99] presented a single-camera approach for simultaneously measuring the 3D motion (including the translation and rotation) of a target attached on civil structures.Greenbaum et al. [100] developed a vision method to measure rigid-body motion including (translation and rotation) experimental displacement in three-dimension rocking motion.

Structural Strain and Stress
Monitoring.The vision-based methods were applied to acquire the structural strain and stress by use of the structural displacement obtained by the vision system and the relationship between the structural displacement and the strain and stress derived in the field of material mechanics.Carmo et al. [101] developed a method for assessment of steel strains on reinforced concrete members using solely surface measurement (crack width and spacing) with the aid of photogrammetry and image processing.Patterson et al. [102] described a material for the purpose of reference and calibration of the optical system for strain monitoring and designed a standardized test material.Winkler et al. [103] employed the digital image correlation method to measure the local deformations in steel monostrands.De Pauw et al. [104] used digital image correlation method to monitor the fatigue parameters of the coupon scale fretting tests.Gales et al. [105] introduced a digital image correlation method to measure the deformation and strain of the prestressing steel during high-temperature tests.Wang and Cuitio [106] applied the digital image correlation technique to capture the deformation patterns of polymeric foams.
McGinnis et al. [107] applied the 3D digital image correlation method for determination of in situ stresses of concrete structures.Obaidat and Attom [108] used two CCD cameras to obtain the strain in soil specimens of two soil tests.Maekawa et al. [109] proposed a noncontact measurement method based on the optical displacement sensors using LEDs to measure the vibration stress and used the acquired stress to evaluate the vibration fatigue failure of small-bore piping systems.Carroll et al. [110] used the digital image correlation method to measure the strain of structures during the fatigue crack initiation and growth and evaluate the condition of cracks.Moilanen et al. [111] proposed an imagebased method to monitor the planar strain and stress distribution in heterogeneous and soft materials.

Vibration Monitoring and Dynamic Characteristics Identification.
The structural displacement can be acquired with a high speed camera at a high sample rate which can satisfy the need of structural vibration monitoring and dynamic characteristics identification such as the natural frequency, modal damping ratio, and modal shape.Chen et al. [112] proposed a digital photogrammetry method for measurement of the ambient vibration response and identification of the mode shape ratio of stay cables with multiple camcorders.Oh et al. [113] presented a vision-based system for estimation of the dynamic characteristics of the structure by using displacement time histories for a motion-capture system.Jurjo et al. [114] proposed a structural displacement measurement method based on digital image processing techniques to conduct the dynamic analysis of slender structures.Fukuda et al. [115] developed a vision-based system to monitor the dynamic response of large-scale civil infrastructure which was more cost-effective.
Kim [116] proposed a multitemplate matching algorithm to obtain the modal parameters of a cable from blurred motion images.Park et al. [90] applied a motion-capture system to monitor the 3D displacement response of a structure in wind tunnel experiments and to identify the dynamic properties of the test structure.Caetano et al. [117] developed a vision-based system to monitor the vibration of slender structures.Jeon et al. [118] presented a method to conduct modal tests using a camera image which could measure the vibration of many points at the same time.Chung et al. [119] applied image processing technique to acquire nonlinear characteristic parameters of mechanical and structural systems.Ji and Chang [120] presented a nontarget imagebased method to measure small cable dynamic responses using an optical flow method.Kohut and Kurowski [121] developed a vision-based method to realize the 3D measurement of structural vibration displacements and modal characteristics by use of operational modal analysis algorithms.
The vibration-based structural damage detection methods have obtained great advances in the area of structural health monitoring.The vision-based dynamic monitoring methods can be applied in the procedure of the vibrationbased structural damage detection to give a stable signal input.Poudel et al. [122] obtained the structural dynamic displacement time series using high-resolution subpixel edge identification based image processing method and developed the mode shape difference function to detect structural damage.Patsias and Staszewski [123] developed a new damage detection method with the aid of wavelets and modal shape data which was measured optically.Li et al. [124] developed a digital image processing method to measure the rivulet vibration of an inclined cable in wind tunnel tests and evaluate the rivulet vibration characteristics.

Crack Inspection and Characterization.
With the aid of advanced image processing technology, the structural surface features can be analyzed from the images such as cracks and spalling on the steel and concrete structures.Yeum and Dyke [125] proposed a vision-based visual inspection technique through the automatic process and analysis of a large number of captured images for detection of the cracks near bolts on the bridges.Liu et al. [126] proposed a method for automated surface crack monitoring and assessment of concrete structures based on adaptive digital image processing.Halfawy and Hengmeechai [127] embedded the visionbased defection recognition system into the closed circuit television (CCTV) system mounted in the sewer to inspect its defections automatically.Adhikari et al. [128] presented an approach of automated condition assessment of concrete bridges based on digital image analyses.
German et al. [129] developed a column damage index for quantitative assessment of the visible damage (cracks and spalling) on the RC structural members enhanced by machine vision techniques to realize rapid building inspection after earthquake.Ho et al. [130] developed an imagebased system with three cameras attached to a cable climbing robot to detect surface damage of cables using image processing and pattern recognition techniques.Valenc ¸a et al. [131] presented a visual method which recognized the concrete health monitoring automatically including measuring the displacement and stain, detecting damages, identifying cracks, and restoration.
Gul et al. [132] proposed an image-based monitoring method for the detection of open gears of movable bridges in lubrication level to assess the condition and make maintenance decision.Sakagami [133] presented a remote nondestructive evaluation technique using infrared thermography to detect fatigue cracks and assess the structural integrity.Liu et al. [134] combined the two-dimensional image processing technology and three-dimensional reconstruction method to assess the crack characteristics of concrete structures automatically to solve the hindrance in the practical implementation of traditional two-dimensional method.Wu et al. [135] developed a crack defragmentation technique based on image processing techniques to improve the crack-detection accuracy in the road assessment task.

Integration Technology.
Through integrating with other sensing approaches, the machine vision technology was broadened to be used in more specific application categories.Vaghefi et al. [136] developed a combined nondestructive imaging technology on a bridge deck to yield both surface and subsurface indicators of condition.Stabile et al. [137] used a suite of microwave radar interferometer and a thermal camera to monitor dynamic displacement of bridges.Catbas et al. [138] carried out the bridge load rating with the aid of traditional sensors and traffic video data.Zaurin and Catbas [139] integrated the video images and sensor data from structural monitoring system to evaluate the safety condition of bridges.Luo et al. [140] developed a vision inspection system combining machine vision, laser interferometer, and coordinate measuring machine.Waldbjørn et al. [141] obtained the feedback signals (strain and displacement) by the fiber Bragg grating and digital image correlation when conducting the test control.Mazzoleni and Zappa [142] presented a vision-based method to estimate the vertical dynamic loading caused by human motions on structures.Hack and Leroy [143] used the cameras orthogonally aligned to monitor the mandrel position by measuring the rigid-body displacement based on a multivariate least-squares algorithm.Integrated with the monocular laser triangle measuring method, Gao et al. [144] used the image processing method to get the 3D displacement of superconducting tokamak magnets.

Conclusions
This paper provides a review of the research and progress in the area of structural monitoring of civil infrastructure by use of the machine vision-based technology.Based on a comprehensive review of the machine vision-based methods, technologies, applications, and systematic error assessment, the following concluding remarks are made: (i) due to their unique merits for structural monitoring, the machine visionbased technology has been widely used to measure the 2D and 3D structural displacement, strain/stress, dynamic response, crack, and spalling, (ii) the machine vision technology can be applied to conduct the structural dynamic parameter identification and damage diagnosis in combination with the existing vibration-based analysis methods, and (iii) the vision-based methods are potential to provide more valuable information for the visual inspection and structural monitoring through integrating with other sensing techniques.
Although the great advances and achievements on the vision-based SHM have been made, some critical limitations and challenges of the developed techniques are still existent; for example, (i) most of the current research is conducted through the laboratory tests with scale physical models, and the research outcomes may not be realized in the field continuous monitoring because of the complicated site conditions, (ii) the quality of the images captured by the vision device will be significantly affected by the surrounding environment conditions, such as the light variation, edge shelter by rain, snow and fog, ground vibration, and (iii) as an interdisciplinary and cutting-edge technology, a big challenge is still encountered from the establishment of a scientific and effective cooperative mechanism between the researchers from civil engineering and optoelectronic engineering.