Detection of the Bolt Loosening Angle through Semantic Key Point Extraction Detection by Using an Hourglass Network

,


Introduction
Bolt connections are key in steel structures, and loose bolts compromise the safety of a structure [1,2].Terefore, the efective and reliable real-time monitoring of bolted connections can improve the safety of steel structures and prevent structural damage [3,4].
Currently, manual visual inspection is the most commonly used method for examining bolt connections in steel structures, but it is labor-intensive and prone to error.Graybeal et al. [5] found that inspectors accurately identifed loose bolts in structural components at an accuracy rate of less than 50%.
Te acoustic-elastic method involves measuring the stress acting on a solid by using the ultrasonic wave velocity, which changes with this stress [8].Piezoelectric ceramic patches are pasted to both ends of a bolt to generate and collect ultrasonic waves, and the bolt looseness is judged by analyzing the changes in the acoustic wave characteristics.Diferent measurement methods can be selected according to the characteristics of the stress feld to be measured.In the electromechanical impedance method, the mechanical impedance of a structure is used.First, a piezoelectric patch is bonded to a bolted joint and excited by a high-frequency voltage.Subsequently, impedance signatures, which refect the local dynamics of the joint, are obtained and used to assess the degree of bolt looseness.Tis method is highly sensitive to changes in stress.However, the aforementioned two methods rely on cables and precision instruments and require manual intervention to extract damage parameters, which are required for damage detection.Under complex environmental conditions (such as drastic changes in temperature and humidity), the performance of a sensor on a bolt might deteriorate, which negatively afects the monitoring results [13]; therefore, the sensor-based bolt looseness detection method has not been widely applied.
Many researchers have combined machine vision technology with deep learning methods to detect bolt loosening, where loose bolts can be identifed by classifying them into loose and tight states (screw out the length or whether the drawing line coincides) or the specifc angle at which the bolt is loose.Park et al. [14] proposed a pioneering vision-based approach for estimating bolt rotations and signals of looseness in bolted connections.Teir method involves capturing digital images of the target connection and then based on the Hough transform [15] and Canny edge detection [16] to calculate the loosening angle.
In using the estimated loose and tight state methods, Cha et al. [17] used the Viola-Jones algorithm to detect bolts in images and estimate their screw out lengths and classifed bolt tightness by using a support vector machine.Zhang et al. [18] proposed a bolt loosening evaluation method based on autonomous deep learning.In this method, a faster region-based convolutional neural network (R-CNN) is used to identify and classify the heights of bolts and screws in real time.Yuan et al. [19] proposed an automatic method based on a mask R-CNN for detecting bolt looseness.In this method, bolt looseness is identifed through a change in screw height.Te trained model can detect bolt looseness from images with diferent backgrounds.Gong et al. [20] proposed a method based on deep learning and geometric imaging theory for quantitatively calculating the exposed length of bolts.Te average measurement error obtained with this method was 0.61 mm.Yang et al. [21] combined the traditional manual torque method commonly used in engineering with the YOLO v3 [22] and YOLO v4 [23] to reduce the cost of manual inspection and the rate of missed inspection.Deng et al. [24] proposed a novel vision-based method to conduct the loosening detection of three threaded joints in a T-junction pipe; a new generative adversarial network-based segmentation module was constructed to accurately segment marked bars, and the average detection accuracy was 94.7%.Deng et al. [25] proposed an automated method for detecting the loosening angle of mark bolted joints by integrating computer vision and geometric imaging theory.In lab-scale and real-scale environments, the average relative detection error was 3.5%.Te aforementioned method is suitable for monitoring high bolt loosening damage but not low bolt loosening damage; the traditional manual torque method needs to draw lines in advance, which is not convenient for actual monitoring.
In using calculated loosening angle methods, Park et al. used an R-CNN [26] and a faster R-CNN [27] to identify bolts in images and adopted the Hough transform to determine the rotation angles of bolt edges.However, their method cannot detect end-to-end bolt loosening damage and can only identify loosening angles from 0 °to 60 °.Zhao et al. [28] used a single-shot multibox network to identify bolts in images.Tey located the "bolt" (the real bolt in the picture) and "num" (a number mark on the bolt) in an image by using a regression box and then extracted the central coordinates for bolt angle calculation.Yu et al. [29] used three methods to mark bolts and adopted a single-shot multibox detector to determine the bolt loosening angle, which increased the bolt identifcation accuracy.However, in the approach of Yu et al., the center position deviation of the identifed rectangular frame was high, which resulted in a high error in the identifcation of the bolt loosening angle.Tis method is efective only when the bolt rotation angle is large, and pasting marks are unsuitable for use in extensive bolt detection.Terefore, the current method based on the bolt loosening angle has insufcient accuracy.In this method, each bolt must be examined separately to obtain its loosening angle, which is inconvenient when investigating numerous bolts.
Stacked hourglass networks [30] have achieved high performance in applications such as human posture estimation [31], vehicle positioning and recognition [32], construction robots [33], and construction site safety maintenance [34].Tese networks can be used to fnd the contours of a detected object accurately and rapidly.In the present study, a stacked hourglass network was used for automatically and rapidly determining the bolt loosening angle.Tis method can overcome the limitation of only using rectangular box regression positioning to calculate the bolt loosening angle.
Te practicality of the method is one of important issues; in real applications, the structure is painted with the same color.When the contrast between the object of interest and the background is low, it may be difcult to distinguish the key points from the surrounding areas.Tis can lead to errors in the detection process and reduce the accuracy of the method.However, there are some techniques that can be used to improve the contrast between the object of interest and the background.In order to better simulate the actual construction site environment, Nguyen et al. [35] used the bolted connection model coated with gray anticorrosion paint to simulate a real steel bridge.Pham et al. [36] used solidworks to create a bolt model sprayed with gray paint.Another approach to addressing the issue of low contrast is to use a diferent type of key point detector that is less reliant on the contrast between the object of interest and the background.For example, some key point detectors rely on the shape of the object rather than the contrast and may be more suitable for applications where the contrast is low.
Te proposed method has been specifcally designed and optimized for hexagon bolts.In general, the feasibility of the proposed method for other types of bolts would depend on the geometry and surface features of the bolt head, which determine the quality and robustness of the image processing and analysis algorithms.For instance, if the bolt head has similar geometry and surface features to a hexagon bolt, our method could potentially be adapted with minor modifcations.On the other hand, if the bolt head has a signifcantly diferent geometry or surface features, it may only require a new geometric analysis, but the overall 2 Structural Control and Health Monitoring process remains the same.For example, for round bolts, we can paste symbols on the bolt heads as key points.Te rest of this paper is organized as follows.Section 2 introduces the proposed method for determining the bolt loosening angle.Section 3 describes the adopted experimental setup, data set, and data acquisition system.Section 4 presents the experimental results of this study.Finally, Section 5 details the conclusions and limitations of this study and provides recommendations for future research.

Methodology
Te method proposed in this paper for automatically determining the bolt loosening angle involves three steps: (a) data preparation, (b) model training, and (c) model performance evaluation.Data preparation involves image collection, the defnition of key points, manual annotation, data augmentation, and automatic annotation.Model training mainly involves using a deep learning module and an optimization function to calculate the bolt loosening angle.Finally, model performance evaluation involves using the normalized error (NE), the percentage of correct key points (PCK), and other evaluation indicators to evaluate the model performance to obtain reference standards for model use.Te overall workfow of the proposed method is displayed in Figure 1.Te frst red box introduces the general steps of data preparation, key point extraction, and loosening angle calculation.Te second red box is the specifc structure of the hourglass network; "res" is the abbreviation of residual module.Te third red box is the calculation method of bolt looseness and model performance evaluation; θ is the bolt loosening angle.Tis method is described in detail in the following text.
2.1.Data Preparation.Captured using the camera of an iPhone12 (12-MP vision sensor, f/2.4 aperture, and focal length of 26 mm), 100 images of a single bolt with a resolution of 4000 × 3000 pixels were collected in a laboratory environment under diferent shooting distances, viewing angles, and illumination levels.To augment the data set, 25 of these images were rotated every 5 °by using the image, rotate function [37] of the Python Imaging Library to obtain 1800 augmented images.Te augmented images had the same shape as the original images but diferent rotation angles.Moreover, the rotation angles were known, and the problem of inaccurate rotation angle measurement was thus avoided.A total of 70%, 10%, and 20% of the augmented data set were used for training, verifcation, and testing, respectively.To balance both computational efciency and accuracy, images from all the data sets were cut into a size of 512 × 512 pixels.Te object in the images (i.e., bolts) was annotated using Labelme3.16.7 [38].Each bolt had the mark of a letter in the English alphabet; thus, these marks could be used as reference points, and each corner was numbered to make it regular.Te corner to the left of the letter on the bolt was annotated as P 1 , and the other corner points were sequentially annotated as P 2 , P 3 , P 4 , P 5 , and P 6 in the clockwise direction.JavaScript Object Notation (JSON) [39] (a lightweight data exchange format) fles were generated for model training.
For the automatic annotating augmented images, the corner coordinates can be obtained by the rotating angle according to the coordinate points marked in the original picture, taking the center of the image as the center of the circle and the distance between the coordinate point and the center of the circle as the radius.Te coordinates in the JSON fle of an original image were transformed into the coordinates of the rotated images by using the process described in the following text.
As displayed in Figure 2, (x 0 , y 0 ) are the center coordinates of each image, (x i , y i ) are the corner coordinates of each bolt, and the distance l i between each coordinate point and the center point of the original image is calculated.Subsequently, the angle q i between each coordinate point and the center point is determined by comparing with the xaxis.Te q i value obtained using equation ( 2) only ranges from 0 °to 90 °; therefore, the real angle between the aforementioned points (q i ′ ) should be determined by identifying the quadrant in which q i lies.
In image processing applications in Python, the positive direction of the y-axis points downward; correspondingly, the following inequalities are obtained: ( Finally, rotation is performed as follows to obtain the rotated horizontal and vertical coordinates:

Hourglass
Network.An hourglass network has a symmetric structure and contains a residual module.Te residual module can extract high-level features through the convolution operation, and it can retain the original information by using a skip route function.Te aforementioned module changes the depth of data without changing their size.Terefore, this module can be considered an advanced convolutional layer.Te structure of the residual module is displayed in the dotted box at the bottom of Figure 3. Te structure of a fourth-order hourglass module is displayed in the dotted box at the top of Figure 3. Tis module performs four downsampling and four upsampling processes, and it allows the network to extract feature information at diferent scales.Te right and left blocks Structural Control and Health Monitoring (C1-C4 and C4b-C1b, respectively) of the hourglass module are mirror images of each other.A similar set of blocks (C4a-C1a) also exists at the top of this module.Each block is combined with the block to its right through the plus sign.Because the network is symmetrical, its appearance is similar to that of an hourglass.After the feature layers are superimposed, the output feature map C1b not only retains the information of all layers but also has the same size as the input map.Heatmaps representing the probabilities of key points are generated through 1 × 1 convolution.Te output results of the frst hourglass module are then summed and input to the second hourglass module.Finally, the    Structural Control and Health Monitoring coordinates with the highest fnal prediction probability for each key point are obtained using the softargmax function.
To detect key points for bolts rapidly and accurately, a detector based on stacked hourglass networks was established in this study.In a deep learning network, the extracted features are gradually abstracted from the deep layers.For example, in a deep learning network designed for detecting bolts, the frst layer extracts low-level features, such as contours, the second layer extracts some higher-level semantic features on the basis of the frst layer, and the third layer extracts the most relevant features for the detection task, such as the complete bolt contour.In a convolutional neural network, a layer only retains the feature map of the layer immediately preceding it, which results in the loss of some information.In corner estimation, the highest recognition accuracy might be achieved for diferent corners in diferent feature maps.For example, P 1 points might be easily identifed in the feature map of the second layer, whereas P 4 points might be easier to identify than P 1 points in the feature map of the fourth layer.Terefore, an hourglass network can perform improved key point detection if it contains multiple feature graphs.

Objective Function Optimization.
In network training, an hourglass network usually adopts a loss function based on the l 2 norm that is commonly used in regression problems (i.e., the mean square error loss [40]).Such a loss function is expressed as follows: where y i represents the target true value, y p i represents the corresponding value predicted by the network, and n is the number of target key points.
Te squaring operation of the mean square error loss results in the model producing high losses at noise points and causes noise points to have high weights.When the model is optimized to reduce the losses at noise points, the overall model performance deteriorates.Terefore, in this study, the smooth l 1 loss function [40] was used to optimize the original loss function, reduce the weight assigned by the network to the noise point, and enhance the network's generalization ability.Te smooth l 1 loss function is expressed as follows: where y i represents the target true value, y p i represents the corresponding value predicted by the network, n is the number of target key points, λ is the loss adjustment coefcient (which is usually 0.5), and δ is the critical point between the square loss and the absolute loss.

Calculation of the Bolt Loosening Angle.
Te process used in this study for calculating the bolt loosening angle is illustrated in Figure 4. Te six corners of a bolt are detected using an hourglass network, and the center coordinates of the bolt are then obtained from the coordinates of the six corners.Because the hourglass network can use the relationship between key points for positioning, the letter on the bolt surface can be used as a mark to locate the bolt.Te diference in these angles is the bolt rotation angle.

Structural Control and Health Monitoring
Te corner coordinates of each bolt are (x i , y i ).On the basis of these coordinates, the center coordinates (x 0 , y 0 ) of each bolt can be calculated as follows: Te distance between each corner and the x-axis is calculated as follows: Te angle formed by an unrotated corner, the x-axis, and the center point is calculated using the following equation: Similarly, the angle θ j formed by a rotated corner, the xaxis, and the center point is obtained by using the relevant rotated coordinates in equation (9).
Te angle between the line and x-axis is calculated in Python, and the obtained angle is [− 180 °, 180 °], which is positive in quadrants 1 and 2 and negative in quadrants 3 and 4. Te default bolt rotation is counterclockwise; thus, when θ i and θ j are combined, the bolt loosening angle θ can be determined as follows: To minimize the error in the rotation angle of a corner, the rotation angles of the six corners of a bolt are averaged to obtain fnal loosening angle.

Model Evaluation.
Four indices, including the NE and PCK, were used to evaluate the performance of the developed hourglass model.Te NE and PCK are typical evaluation indicators for the solutions of most key point problems, such as facial key point estimation [41] and human posture estimation based on key points [31].When the NE and PCK were calculated, a distance normalization parameter had to be used because the area occupied by a bolt difered between images.In this study, the normalization parameter was set as the average distance between a bolt corner and the center point.

Normalized Error.
Te NE is the average normalized distance between the predicted key point and the ground truth [42].Te normalized distance is obtained by dividing the predicted Euclidean distance and ground truth value by a distance normalization parameter.A smaller NE indicates better model performance.Te NE is calculated using the following equation: where M is the total number of test images, K is the total number of key points, d m,k is the Euclidean distance between the predicted position and the ground truth of the k th key point in the m th image, and s m is the normalization parameter in the m th image.

Percentage of Correct Key Points.
Te PCK refers to the percentage of key points that are correctly predicted in the entire data set and is an indicator of the accuracy of a pose estimation model [43].According to the defnition of the human pose estimation problem, if the normalized distance between a predicted position and the ground truth position is within a given threshold α, where α ∈ [0, 1], the candidate key point is assumed to be correctly located.For human posture estimation, 0.05, 1, and 0.2 are usually used as the upper limits of α.Te key points of bolts are easier to locate than are those of human postures; therefore, the upper limit used to calculate the PCK curve of each model was set as α � 0.025 in the present study for model performance to be 6 Structural Control and Health Monitoring accurately evaluated.Te PCK is calculated using the following equation: where α is the threshold for calculating and drawing the PCK curve and the defnitions of the other parameters are the same as those in equation (11).[44] was selected as the network optimizer for the frst 1000 epochs, after which Adam [45] was used as the network optimizer.Tis approach was adopted because RMSprop has high convergence speed but low convergence accuracy, whereas Adam has low convergence speed but high convergence accuracy; thus, the aforementioned method increased the training speed and reduced the training time.Early-stopping technology [46] was used to monitor the loss on the validation set, so as to stop training when the model starts to overft and improve the generalization ability of the model.Te criteria for Early-stopping technique can be as follows: the loss on the train set does not decrease after multiple iterations of the model.2.Te loss on the validation set starts to rise.As shown in Figure 5, the fnal number of training iterations was 9,800, and the model loss decreased with an increase in the number of iterations until the model loss became stable.Table 1 presents all the training parameters.At this time, the model was stable, and its recognition accuracy was high.Finally, the trained model was applied to the test data set to evaluate its performance for new images.Te NE value was used to evaluate the model accuracy (Table 2).As presented in Table 2, automatic labeling considerably improved the accuracy of the hourglass model, with the NE values of all key points being reduced by 9.28% on average.Te NE values of the model trained through manual and automatic annotation difered between key points.For both annotation methods, the lowest NE was achieved for P 1 possibly because this point was closest to the letter mark on the bolt.Te NE values for P 1 under manual and automatic annotation were 7.98 × 10 − 3 and 7.70 × 10 − 3 , respectively.Te highest NE was achieved for P 3 under both annotation methods.Te NE values for this point under manual and automatic annotation were 10.94 × 10 − 3 and 10.22 × 10 − 3 , respectively.Te performance improvement (i.e., percentage reduction in NE) resulting from automatic annotation compared with manual annotation was diferent for each key point.Specifcally, the largest NE reductions when automatic annotation was used were observed for P 6 (18.1%) and P 4 (13.04%),whereas the smallest NE reduction was observed for P 5 (3.03%).

Training and Key Point Assessment
Te PCK curves of the hourglass model were calculated using diferent threshold values (α ∈ [0, 0.025]) under manual annotation and automatic annotation (Figure 6).In general, the PCK increased with the threshold.For each key point, the PCK increased considerably with the threshold value when α ∈ [0, 0.015], and the magnitude of increase decreased when α exceeded 0.015.Te average PCK value of diferent key points plateaued at 98.1% and 99.3% under manual annotation and automatic annotation, respectively.In general, under the two annotation methods, the PCK  Structural Control and Health Monitoring values for P 1 and P 4 were higher than the average PCK value, whereas the PCK values for P 3 and P 6 were lower than the average PCK value; thus, the network recognized the key points located closer to the letters on the bolt more accurately.Te PCK value of the adopted hourglass model was higher under automatic annotation than under manual annotation for the diferent key points.Tus, the automatic annotation method proposed in this paper can improve the identifcation accuracy of key points, which can enhance the detection accuracy of the bolt loosening angle.

Calculation of Bolt Loosening Angle
After undergoing training and testing, the developed stack hourglass network determined the corner coordinates of bolts in the data set images.Tese coordinates were then used to obtain the rotation angle, and the degree of bolt loosening and the bolt loosening damage was determined by detecting the change in the rotation angle.

Recognition Accuracy for Diferent Rotation Angles.
Table 3 presents the absolute values and errors of the bolt loosening angles obtained using the proposed method and the method of Zhao [10] under the same rotation angle.Zhao used a Single-Shot MultiBox Detector network to locate rectangular frames for a bolt and num; however, the center position deviation of the recognized rectangular frames was large, and only these two positioning points existed, which resulted in a large error for the bolt loosening angle.Te average error of the bolt loosening angle obtained using Zhao's method is 5% for angles of up to 360 °.By contrast, other bolt looseness detection methods can only recognize bolt loosening angles of up to 60 °.Table 3 indicates that under the same rotation angle, the proposed method provides a more accurate bolt loosening angle than does the method of Zhao.Te result was primarily because the hourglass network used in the proposed method achieves high-accuracy bolt looseness detection by overlaying feature layers and incorporating six key points in the last feature map.

Recognition Accuracy for Small Rotation Angles.
To determine the minimum rotation angle that is recognizable with the proposed method, an original image captured at a shooting distance of 30 cm under mild light conditions was rotated every 0.5 °between 0 °and 30 °by using Python to obtain 60 rotated images.
Figure 7 depicts the identifcation of bolt images with rotation angles of 0.5 °, 1 °, 2 °, and 4 °.Te results indicated that the average angular diference was approximately 0.4 °.For a rotation angle of 0.5 °and 1 °, the error rate reached 36% and 39%, the error rate was too large to use as minimum rotation angle.For a rotation angle of 2 °, the error rate reached 19%.Although this error rate was still large, the absolute angular diference was only approximately 0.38 °, and the average error rate was approximately 3%.Te absolute and percentage errors in the bolt loosening angle are presented in Table 4. Te error rate decreased with an increase in the rotation angle.On the basis of the obtained results, the minimum bolt loosening angle that can be identifed with the proposed method was determined to be 2 °.

Recognition Accuracy for Large Rotation Angles.
To determine the maximum rotation angle that is recognizable with the proposed method, an original image captured at a shooting distance of 30 cm under mild light conditions was rotated every 5 °between 0 °and 360 °by using Python to obtain 72 rotated images.Te developed model was then tested on these images.Te rotation angles of the bolt were calculated using the proposed method and compared with the actual rotation angles.Figure 8 displays the recognition results for rotation angles of 90 °, 180 °, 270 °, and 350 °.Te obtained results indicate that the proposed method can achieve 360 °detection of the bolt loosening angle.

Recognition Accuracy under Diferent Shooting
Distances.Te variation in the recognition accuracy of the proposed method with shooting distance was examined.Each image in the data set was rotated every 10 °from 0 °to 360 °in Python, yielding 36 images for every original image.Te recognition results obtained with the proposed method under diferent shooting distances are displayed in Figure 8. Te absolute and percentage errors in the bolt loosening angles obtained with the proposed method under diferent shooting distances are presented in Table 5, in which the numbers in parentheses indicate the rotation angle.Te results indicated that the detection accuracy of the proposed method was afected by the shooting distance.When the shooting distance was 20 cm, the average error in the bolt loosening angle was 0.27 °(average error rate of 0.29%); however, when the shooting distance was 50 cm, the average error in this angle increased to 1.07 °(average error rate of 1.10%).Figure 9 indicates that as the shooting distance increased, the absolute and percentage errors increased considerably; this was because the number of pixels occupied by the bolt and the image defnition was lower at higher shooting distances.Moreover, the letters on the bolt became blurred, which was inconducive for positioning and recognition by the hourglass network.Terefore, as the shooting distance increased, the detection accuracy for the bolt loosening angle decreased.Te recommended shooting distance for the proposed method is approximately 30 cm.

Figure 1 :
Figure 1: Overall workfow of the proposed method for automatically determining bolt loosening angle.(a) Data preparation.(b) Model training.(c) Model performance evaluation.

Figure 2 :
Figure 2: Automatic annotation process.(a) Calculate the angle of each coordinate point.(b) Rotate by set angle.

Figure 3 :
Figure 3: Key point detection for bolts by using a stacked hourglass network.

Figure 4 :
Figure 4: Process for calculating bolt loosening angle.(a) Determine the coordinates of each corner point and center point.(b) Calculate the angle between each corner point and the center point.(c) Calculate the angle between each corner point and the center point after loosening.(d) Calculate loose angle.

3. 3 .
Model Evaluation.Te performance of the hourglass network in identifying the key points of bolts under diferent annotation methods was examined through experiments.Te detection time and training time were the same in the experiments.Te detection time for each graph was 71.55 ms, and the training time was 5.32 h.

Figure 6 :
Figure 6: PCK curves of the stacked hourglass network for each key point of a bolt.(a) Manual label.(b) Auto label.
Training was conducted using the PyTorch framework.For training, the network was fed 512 × 512pixel, three-channel color images, and the batch size was 144 and initial learning rate was 1 × 10 − 4 .Te learning rate decreased by 10% for every 2000 steps in the training process.A total of 70%, 10%, and 20% of the data set (1313, 187, and 375 images, respectively) were used for training, validation, and testing, respectively.Root Mean Square Propagation (RMSprop)

Table 2 :
NEs obtained under manual and automatic labeling.

Table 3 :
[10]lute values and errors of the bolt loosening angles obtained using the proposed method and the method of Zhao[10]under the same rotation angle.

Table 4 :
Absolute and percentage errors in the bolt loosening angle when using the proposed method.