Fast Force Estimation of Cable Structures Using Smartphone-Captured Video and Template Matching Algorithm

,


Introduction
Te cable component is an important force transmission structure of cable bridges, as the cable force afects the service life of the bridge.Terefore, it is necessary to accurately estimate the cable force [1][2][3].At present, the measurement of the dynamic characteristics of cable, namely, the vibration and natural frequencies, is a paramount step for vibration-based cable force estimation [4].Yu et al. [5] studied a wireless monitoring system for the bridge cable force based on the vibration frequency method and conducted experimental verifcation through laboratory and feld tests.Te results of feld experiments showed that the relative error between the wireless monitoring system and the reference wired system was less than 0.5%.Rainieri and Fabbrocino [6] developed an original algorithm for automatic output-only modal parameter estimation and found that it was robust and could provide the accurate estimation of modal parameters including damping ratio.Terefore, this algorithm has been used to develop vibration-based tensile load estimation systems for cables and tie rods.Zarbaf et al. [7] evaluated the stay cables of the new Ironton Russell Bridge using vibration-based force estimation and compared the results with the lift-of test results.Tey found that the diference between the estimated force and the directly measured force (lift test) was negligible.
Obviously, to estimate the cable force, the frst step is to obtain the frequency of the cable [8][9][10][11].Nowadays, this can be performed by installing an acceleration sensor on the cable [12].For example, Morgenthal et al. [13] proposed methods to facilitate such force recognition using a highly mobile measuring device connected to a battery-operated microcontroller and an acceleration sensor based on a modern MEMS integrated into a smartphone.Te results showed that the sensor resolution and sampling rate directly infuenced the accuracy of measurement and are the distinguishing criteria between competing systems.Xue and Shen [14] presented a short-time sparse time-domain algorithm combined with a simplifed half-wave method.Tis method uses only one acceleration sensor, which makes it simple, economical, convenient, and fast.Its reliability and accuracy were verifed by two experiments and one engineering application.However, this approach requires moving the sensor from one place to another to test all the cables of a large bridge, which limits its measurement effciency [15].In the traditional method, the accelerometer needs to be installed on the cable to obtain the cable vibration signal, which acts as the basis for estimating the cable force.However, this method has two shortcomings.On the one hand, the accelerometer needs to be installed at a certain height on the cable.Generally, this is done at a distance of more than 3 m from the bridge deck.Obviously, this necessitates a climbing operation plan that can cause trafc stop and even result in personal harm.Besides, accelerometers are expensive and have poor durability.In addition, it is not possible to install a large number of accelerometers on every bridge, as it will inevitably increase costs.
At present, computer vision has been widely used in structural health monitoring [16][17][18][19][20].In recent years, the rapid development of computer vision methods has provided a new solution for the more efective estimation of cable force [21].Kim et al. [22] constructed a vision-based monitoring system that uses image processing techniques to estimate the tension of stay cables during trafc use.Du et al. [23] used two measurement methods based on digital image technology, digital image processing, and digital image correlation to identify cable forces through the camera capture of single and multipoint images.To verify the reliability of this method, the results were compared with those obtained using an accelerometer, and the relative deviation between the two methods and the accelerometer was less than 5%.Chu et al. [24] proposed a vision-based contactless cable force monitoring system for cable frequency identifcation and tension estimation, and its applicability was verifed through experiments.Zhang et al. [25] presented a deep learning-based complex background segmentation method to estimate the cable forces of urban bridges from UAV-collected videos.Te efectiveness and robustness of this method were successfully verifed by a feld test of a city cable-stayed bridge.
Using a computer vision method, the vibration deformation curve of bridge cable can be obtained directly; however, because the vibration deformation curve contains a lot of environmental noise, an important step remains to evaluate the cable force, that is, data processing [26].At present, many methods are used for this purpose, such as the wavelet decomposition (WD) method, empirical mode decomposition (EMD) method, and ensemble EMD (EEMD) [27].For example, Swami and Jain [28] proposed an efective image denoising method that adaptively combines the features of wavelets, wave atoms, and curves.Te denoised image could perfectly present the smooth area and efectively preserve the texture and edge in the image.Zhang et al. [29] used the modifed EMD algorithm to denoise the fault signals prior to the maximum correlation kurtosis deconvolution.Li and Hu [30] derived a sensor fault detection method based on enhanced principal component analysis (PCA) using EEMD denoising.After EEMD preprocessing, the denoised data were smoother than the original data.Fang et al. [31] proposed an improved program based on empirical wavelet transform to eliminate the noise from Global Navigation Satellite System (GNSS) data and to identify the modal parameters of bridge structures.Te above methods, however, have limitations; the accuracy of the WD method depends on the wavelet function, and the accuracy of EMD and EEMD methods depends on the number of decomposition layers, which can lead to multimodal properties of the decomposed intrinsic modal function [32].To address these issues, Yang et al. [33] proposed a complete ensemble empirical mode decomposition with an adaptive noise joint wavelet packet threshold processing method for processing ultrasonic nondestructive testing defect signals.Compared with the EEMD algorithm, the signal-to-noise ratio was increased by 48.03%, and the root mean square error and total number of iterations were reduced by 38.77% and 33.34%, respectively.
Based on the above fndings, we establish the EMDWD method for data decomposition to combine the advantages of the WD and EMD methods.Tis framework achieves the fast estimation of the cable force of long-span bridges based on smartphone-captured video and a template matching algorithm.Specifcally, we can obtain a large number of cable vibration videos (3∼5 minutes) from smartphones at the bridge sidewalk, then analyze these data and estimate the cable force.Tis monitoring plan will obviously not cause any trafc stops, since smartphone video recordings can be made on the sidewalk or vacant land.In addition, no climbing operations are included, thus protecting personal safety.Firstly, the EMDWD model is constructed based on the WD and EMD methods to extract the vibration signal of bridge cable.Ten, the deformation curve of the cable is obtained based on the vibration identifcation model of the bridge cable by the template matching algorithm.Finally, the frequency of bridge cable is calculated based on FFT using smartphone-captured video, followed by the estimation of cable force.

Framework for Cable Force Identification
2.1.Template Matching Algorithm.Computer vision (CV) methods use digital cameras, digital sensors, smartphones, or other sensing vision devices to replace the analog vision function of the human eye [34].Compared with traditional monitoring methods, this approach can more accurately and quantitatively identify the displacement of the structure, as shown in Figure 1.As can be seen from Figure 1, the smartphone records the global coordinates of the target point on the structure to be measured in the form of image coordinates.Changes in the image coordinates of the target 2 Structural Control and Health Monitoring point can be recorded based on fxed smartphone coordinates when the target point moves, so as to monitor the displacement and deformation of the target point [35,36].Meanwhile, a single camera based on the CV method cannot fully monitor the spatial displacement of structures, such as out-plane displacement.Terefore, to monitor the spatial displacement of the structure, we need at least two cameras in this method.
In order to accurately obtain the displacement and deformation curve of the structure under external force load, it is necessary to continuously monitor the structure to be tested [37,38].At this time, the initial deformation of the structure must be determined, that is, the initial template.With the passage of time, the structure to be tested will also vibrate and deform.Taking the initial template as the origin, the displacement curve of the structure to be tested in this period of time can be obtained [39].At present, the gray template matching algorithm is a commonly used template matching method, which includes the average absolute diference algorithm, absolute error sum algorithm, error square sum algorithm, and square error square sum algorithm, among others [40].In recent years, template matching algorithms have been widely applied in the feld of structural health monitoring and have achieved remarkable results [11,41,42].
Te average absolute diference algorithm calculates the sum of the absolute values of gray diference between the subimage and the template image and then averages it, as shown in equation (1).Te average absolute diference is clearly smaller; hence the two graphs are rather similar [43].
where M × N denotes the size of the target graph; (i, j) denotes the pixel of the point to be measured; P is the target graph; and T is the template graph.Te absolute error sum algorithm and the average absolute diference algorithm are based on the same idea, but their similarity calculation formula is diferent, that is, ( Te error square sum algorithm is similar to the above algorithm, that is, ( Te square error square sum algorithm is similar to the above algorithm, that is,

Calculation of Cable Force.
If the infuence of cable bending stifness, cable sag efect, and cable damper are ignored, the relationship between cable force and its natural vibration frequency can be obtained using the string vibration theory, so as to calculate the force.When the constraints at both ends of the cable structure are hinged constraints, the measurement of cable force by the frequency method can be expressed as follows [44]: where f n denotes the nth order cable frequency (Hz); P denotes the force of cable (N); E denotes the elastic modulus of cable; � m denotes the mass per unit length of cable (kg/m); I denotes the moment of inertia of the cable; and L is the height of cable.
Commonly, the cable stifness can be found in the design maps.However, the physical properties of a bridge cable can change with time, such as due to rust and corrosion on the bridge cable.Terefore, the real stifness of the bridge cable will be lower than the design value and cannot be accurately obtained.To tackle this problem, natural frequencies of diferent orders are adopted to estimate the cable force, from which the specifc value of cable stifness information is no longer required.Te cable force estimation formula without known stifness is Te cable frequency can be calculated by the Fourier transform method (FFT), which assumes that any function can be expressed in countless series functions or integral forms, that is, the time-domain signal is converted into the frequency domain signal, and can be expressed by [45]

Structural Control and Health Monitoring
where f (t) denotes the time-domain signal; F (ω) denotes the frequency domain signal; F − 1 denotes the inverse operation of the function; and f denotes the frequency.

EMDWD Method for SHM Data Reconstruction
3.1.Empirical Mode Decomposition Method.Te WD method decomposes the data step by step through the wavelet function; thereby, the decomposition accuracy depends on the wavelet function.In contrast, the empirical mode decomposition (EMD) method does not need to set a function to decompose the data [46][47][48].Firstly, it obtains decomposition by interpolating a certain interval data, that is, by selecting the maximum and minimum values of local data and calculating their average value.Ten, it subtracts the average value from the original data to get new data.Finally, the original data is gradually decomposed into the intrinsic mode function (IMF) and residual, that is, [49].
where P(t) represents the SHM data; y i (t) denotes the ith IMF; and l n (t) is the residual.

Wavelet Decomposition Method.
Monitoring sequence data usually have waveform characteristics, and real-time sequence data have repeatability characteristics [50].Terefore, the sequence data can be represented by "infnite" waveform functions.Te wavelet decomposition (WD) method decomposes the monitoring data step by step in the way of high frequency and low frequency, that is, a time series data are decomposed by wavelet function [51]: where t denotes the time; a represents the scale factor; b represents the translation factor; j represents the scale coefcient; and k is the translation coefcient.By setting the threshold T for the above wavelet coefcients, the high-frequency data in the monitoring data can be fltered out; that is, the noise information contained in the wavelet function that is greater than the threshold is completely eliminated.Te current approaches for the selection of threshold include the SUREShrink threshold estimation method, BayesShrink threshold estimation method, VisuShrink threshold estimation method, and Minmax threshold estimation method.For the SUREShrink threshold estimation method, the wavelet coefcients are calculated for the threshold T based on the unbiased likelihood estimation criterion, such as [52].
For the BayesShrink threshold estimation method, the wavelet function is modeled based on statistical characteristics, and then the threshold T is obtained according to Bayesian estimation, such as [53].
Te VisuShrink threshold estimation method is a global threshold method.In other words, a unifed threshold T is adopted for all coefcients in the wavelet function [54]: Te Minmax threshold estimation method calculates the wavelet function to obtain the threshold T based on the maximum minimization principle [55]: where N represents the number of wavelet function layers; σ n represents the standard deviation of noise; σ w represents the standard deviation of the initial wavelet function; and I denotes the judgment function.

Wavelet-Based EMD.
Te decomposition efciency of the EMD method depends on the number of decomposition layers, which may cause the data confusion of decomposed eigenmodes.Terefore, this paper combines the WD and EMD methods to construct the EMDWD method.Unlike traditional EMD that extracts the signal trend by averaging the upper and lower envelopes intersecting the local maxima and minima of the signal, the wavelet-based empirical mode decomposition directly extracts the signal trend by applying the multilevel wavelet decomposition of consecutive approximations within the sifting process [56].In specifc, frstly, the EMD method is used to decompose the SHM data to obtain the IMFs and residual.Ten, the IMFs and residual are processed by the WD method to obtain new IMFs and residual.Finally, the SHM data are reconstructed based on the new IMFs and residual, which can be seen in Figure 2.  4(a)-7(a), respectively.Te acceleration signal was further converted in combination with the FFT method to calculate the frst-order and second-order frequencies of the four cables, as shown in Figures 4(b)-7(b), respectively.Te frst peak in the frequency spectra represents the frequency of the accelerometer, which is a sensor with zero fundamental frequency.

Field Test and Cable Force Identification of a Long-Span Bridge
Next, the smartphone was fxed in the direction perpendicular to the bridge cable.Finally, the pixel coordinate change of the target point was calculated according to the video data and combined with the template matching algorithm.Te frequency of the cable was calculated based on the FFT, and then the cable force was estimated.Te test arrangement is shown in Figure 8.As can be seen from the fgure, the layout orientation of the smartphone is perpendicular to the vibration direction of the cable, which is convenient for capturing the motion law of the cable.Te smartphone was placed at a horizontal distance of 2 m from the cable and a perpendicular distance of less than 1 m from the red tape.Te diameter of the cable was 0.077 m.
Generally, we used the accelerometer to measure the vibration frequency of the cable over a period of few seconds, because the sampling frequency of the accelerometer is higher than that of cable vibration.Meanwhile, we needed more time to capture videos of cable vibrations when using smartphones.In addition, more video data is required to efectively eliminate the impact of environmental and mobile phone vibrations.To compensate for these, the video frame rate was low.Te red tape is obviously diferent from the color of background and environment; hence, it was taken as the target point in the video.We adopted Mate30 smartphone with TAS-AL00 model as the vision equipment, in which the video resolution is 16 : 9, 1080 p, and the video frame rate is 30 fps, that is, a one-second video is composed of 30 images, and the screen is 2340 × 1080.Te size of the template is 400 * 300 pixels.Te monitoring video image and template image are shown in Figure 9. Te raw video will be directly computed without undergoing any further processing.We arranged the acceleration sensor along the direction of the cable.During the monitoring process, we tried to keep the posture of the smartphone parallel to the cable as much as possible to ensure that the smartphone image coordinates are parallel to the direction of the accelerometer.
According to the video data recorded on the bridge deck, a gray template matching model based on the square error square sum algorithm was established to obtain the vibration curve of the cable.It is important to note that smartphones are vulnerable to environmental loads during shooting, resulting in mobile phone vibration, which will lead to the original vibration (S y (t)) measured by the video data, including the real vibration of cable (S t (t)), environmental vibration (S h (t)), and smartphone vibration (S s (t)), which can be expressed as Smartphone vibration is always present and fxed.According to the literature, the vibration frequency of smartphones is very low.Environmental vibration is related to the surrounding wind feld, vehicles, pedestrians, etc., which leads to its signifcant uncertainty.To address the above challenges, frstly, we obtained the measured acceleration data of the standard cable and converted it into deformation data.Ten, we used smartphones to obtain deformation data of the cable under multisource loads.Finally, we determined the sum of smartphone vibration data and environmental vibration data by subtracting the measured data based on smartphone deformation data.Both types of vibration vary over time and are not constant values.

Structural Control and Health Monitoring
However, considering that we only have four cables to capture in our recording and the relatively short time required to obtain their vibration video data; we assumed that the environmental vibration would not change within 5 minutes.We immediately conducted visual monitoring of other cable vibrations using smartphones to obtain deformation data of the cables under multisource loads.Also, we obtained the sum of smartphone vibration data and environmental vibration data.As a result, we could calculate the actual vibration data of the cable.It should be noted that the vibration deformation data of the same vibration captured from diferent shooting angles exhibit certain diferences.Tis represents intuitive time-domain information.However, the frequency information extracted in this paper   Structural Control and Health Monitoring is obtained through data processing and is minimally affected by variations in the captured videos.By doing this, we can ensure the same results of the same vibration data in diferent locations and at diferent resolutions, so this technology can be used in the video images shot by diferent mobile phones.
For example, we took C1 cable as the standard cable, that is, subtracting the video data (S y (t)) from the measured data (S t (t)) to obtain the basic ambient noise value at frst.Ten, the noise reduction data were decomposed and reconstructed according to the EMDWD method.Finally, the frequency of signal data were calculated by the FFT method, and the fowchart is shown in Figure 10.
Te SHM data processing and calculation results are shown in Figures 11-14.At frst, we obtained the frequency value of C1 according to the accelerometer signal of C1.Ten, we generated a set of smartphone signals based on C1 frequency values, which we assume were the real vibration signals of C1 cable (S t (t)).Finally, we used the measured smartphone signal to subtract real vibration signals, and thus we can acquired the environmental vibration (S h (t)) and mobile phone vibration signal data (S s (t)).To accomplish this task, we used a smartphone to take a video of four cables vibrating simultaneously.Terefore, we assumed that the measurement noise of each cable during this period was the same.
Furthermore, we conducted control experiments based on Cable C2 signals to analyze the impact of EMD, WD, WDEMD, and EMDWD methods on signal data processing, which can be seen in Figure 12.We used traditional EMD and WD methods to process the Cable C2 data.Tis entails using the WD method to denoise data at frst, then using the EMD method to decompose the signal, to fnally reconstruct the signal.As can be seen Figure 12, when we used the WD  Structural Control and Health Monitoring method, there was signal interference between 0 and 2 Hz, i.e., data noise, which makes it difcult to accurately determine the frst-order frequency of the cable.On the contrary, the EMD method can efectively denoise signals from 0 to 2 Hz, so that we can obtain smoother curves.However, the EMD method cannot efectively capture higher-order frequencies, i.e., the third-order frequency of the cable, while the WD method can mine more efective information in the signal.Terefore, it was necessary to combine the two above methods to fully refect the efective information of data.Te EMDWD method could efectively denoise signals of 0∼2 Hz compared to the WDEMD, which demonstrates its efcacy in combining the advantages of EMD and WD methods.In other words, the EMDWD method used in this article is more efective.
Next, we used the EMDWD method to calculate the frequency of Cable C3 and Cable C4.It can be seen that the EMDWD method could efectively capture the cable frequency and provide data support for subsequent cable force evaluation.
For Cable C1 and C2, the mass per unit length was about 44.68 kg/m and the length was about 39.9 m; for Cable C3, the mass per unit length was about 59.25 kg/m and the length was about 16.6 m; for Cable C4, the mass per unit length was about 44.75 kg/m and the length was about 14.0 m.Te diameter of all cables was about 0.077 m.We brought these cable parameters into equation ( 12) and compared the estimated cable force, as shown in Figure 15.It can be seen from the fgure that the calculated frequency and force of cable based on computer vision is in good agreement with the measured frequency and force.Table 1 presents the discrepancies in cable force as determined through visual measurements versus actual measurements.It can be observed that the cable force measured visually exhibits relatively minor deviations when compared to the actual values, indicating a certain degree of reliability.

Conclusions
Smartphones have a higher cost-efectiveness than accelerometers, which includes price and lifespan.In addition, most people have a smartphone, which allows for a wider range of applications.Although the measurement accuracy of accelerometers is better than smartphones, the latter are a more economical alternative if the aim is not to monitor displacement but to perform the estimation of cable force.Terefore, this paper presents a framework for estimating the cable force of long-span bridges based on smartphonecaptured video and deep learning.Firstly, the EMDWD model was constructed based on the WD and EMD methods, which could efectively improve the accuracy of data processing.Ten, the vibration identifcation model of bridge cable based on a template matching algorithm was established, and the deformation curve of the cable was obtained.Finally, based on smartphone-captured videos, the frequency of bridge suspender was calculated and the cable   In this paper, in addition to the infuence of environmental vibration on video data, for the template matching algorithm, video picture background noise will also afect the test data, so how to accurately extract the goal of the cable is an important problem.In the following research, we will consider using multiple video images to estimate the spatial displacement of the cable and semantic segmentation to more efectively eliminate the background noise.Since the template matching requires the manual selection of the template for the tracking calculation, which is limited by the selection of the template, the quality of the template can afect the estimation results of the cable force.Terefore, we will consider a variety of algorithms in the future, based on the computer automatically extract cable features for deformation calculation.

Figure 1 :
Figure 1: Computer vision method with smartphone.

Figure 8 :
Figure 8: Visual monitoring of bridge cable.

Figure 13 :Figure 14 :Figure 15 :
Figure 13: Frequency of Cable C3 by CV using smartphone.(a) Data processing of Cable C3 by EMDWD.(b) Calculation frequency of Cable C3.

Figure 15 :
Figure 15: Comparison of results between acceleration sensor and smartphone.(a) Comparison of frst-order frequency.(b) Comparison of second-order frequency.(c) Comparison of cable force.
Structural Control and Health Monitoring a length (main bridge) of 3 × 210 m and a total length (including the approach bridge) of 1,855 m.Firstly, we installed the low-frequency acceleration sensor (sampling frequency of 1000 Hz) at about 3∼4 m height and fxed it with red tape.Te installation position of the acceleration sensor is shown in Figure3.Te DH5906W accelerometer has a maximum sampling rate of 1000 Hz, its vibration acceleration is 1 × g, its error is less than 10%, its weight is 0.26 kg, and its size is 94 mm × 56 mm × 26 mm.Tis type of accelerometer can be applied with the following restrictions: (1) the stifness of the cable should not be too large and (2) the slenderness ratio of the cable cannot be less than 10.Tis test took the longest Cable C1 in the left span cable system as the standard reference, and tested the longest Cable C2, Cable C3, and shortest Cable C4 in the middle span boom system of Jiubao Bridge.Te acceleration signals of the four cables (C1, C2, C3, and C4) of the measured bridge are shown in Figures

Table 1 :
Error of cable force.Several conclusions can be drawn from the results: (1) By setting obvious mark points and based on the template matching algorithm, the vibration curve of bridge cable can be obtained efectively.(2) Te proposed the EMDWD method can decompose the measured displacement into cable vibration and environmental vibration.By reconstructing the noise reduction data, the cable vibration signal can be efectively extracted.(3) Te efectiveness and robustness of the proposed framework were verifed by the feld test of the long-span bridge.Te results indicate that the cable forces estimated by using the proposed method are almost the same as the measured values.