Fall prevention is an important issue particularly for the elderly. This paper proposes a camera-based line-laser obstacle detection system to prevent falls in the indoor environment. When obstacles are detected, the system will emit alarm messages to catch the attention of the user. Because the elderly spend a lot of their time at home, the proposed line-laser obstacle detection system is designed mainly for indoor applications. Our obstacle detection system casts a laser line, which passes through a horizontal plane and has a specific height to the ground. A camera, whose optical axis has a specific inclined angle to the plane, will observe the laser pattern to obtain the potential obstacles. Based on this configuration, the distance between the obstacles and the system can be further determined by a perspective transformation called homography. After conducting the experiments, critical parameters of the algorithms can be determined, and the detected obstacles can be classified into different levels of danger, causing the system to send different alarm messages.
When the elderly fall, it can harm their bodies and has serious negative mental impacts on them. In the United States, about 30% of adults above 65 years of age suffer a fall annually [
There are many risk factors that can cause the elderly to fall, and they can be roughly classified into three categories: intrinsic factors, extrinsic factors, and exposure to risk [
With respect to exposure to risks, Graafmans et al. suggest that the most inactive and the most active people have the highest risk of falls [
Researchers have worked for a long time to develop various kinds of fall prevention and fall detection systems [
Line-laser obstacle detection system sends alarm messages to catch the attention of the elderly when they encounter dangerous obstacles.
In order to detect the environmental hazards that may cause falls, related obstacle detection methods and technologies were deeply investigated. Traditionally, obstacle detection has been vital for many mobile robot applications [
Engineers have developed methods and various kinds of systems for obstacle detection based on the abovementioned range sensors. Manduchi et al. implemented a color stereo camera and a single axis radar for cross-country autonomous systems [
With respect to laser range finders, Fu et al. designed an integrated triangulation laser scanner for obstacle detection installed on miniature mobile robots [
Stereo vision is, however, computationally expensive. On the contrary, a camera-based line-laser technique that needs much less computation amounts is proposed to detect obstacles in this paper which is a part of Yang’s thesis [
In our system configuration, a line-laser is mounted on the side of the shoes and an RGB camera is fixed but tilted down on the top side of the shoes [
In our prototype, a Logitech C310 webcam that operates at 29 frames per second under a 640 × 480 resolution and a 405 nm wavelength laser are used. Both of them are mounted rigidly to have consistent calibration parameters. Moreover, to suppress the noise interference, a blue glass paper is used as a band-pass filter to resist the unnecessary light from the environment, as shown in Figure
(a) System configuration of line-laser obstacle detection system integrated on shoes and (b) prototype of the proposed system.
Our algorithm flowchart is shown in Figure
Software framework of line-laser obstacle detection system.
Once the obstacle detection event is triggered, a series of line-laser pattern extraction steps will be executed. First, the median filter is applied in order to suppress noises. In addition, an intensity threshold is used to identify the pixels where the laser light is. Then, the max value of each column of the image is extracted and considered as a potential laser pattern. After that, a segmentation method is applied to classify every laser pixel into several clusters that may denote the obstacles. Some clusters that do not represent obstacles are removed, and the clusters that denote the same obstacles together are merged. Subsequently, the real depths and widths of the obstacles are obtained by homography transformation. Finally, the system will send alarm messages according to the dangerous levels of the obstacles classified by their widths and depths. Each step in the software framework is illustrated in detail as follows.
In order to detect the obstacles, the obstacle detection event is triggered when a user steps on the ground. Besides, to detect higher obstacles, the event is triggered when users raise their feet at the max height during a gait cycle as well. In both cases, their feet are roughly horizontal to the ground. Therefore, the system is suitable to detect the obstacles in front of the users at these moments. Besides, this strategy will save computation consumption at other times during a gait cycle.
The difference between the two frames that are captured by the camera is utilized to identify when the users either step on the ground or raise their feet at the max height. This concept was first proposed by Fitzpatrick and Kemp [
In a gait, there are two phases: stance phase and swing phase. Usually, the SAD value between frames during a stance phase should be very small. Similarly, the SAD value will be small when the feet are at the max height. Once the users start to move their feet forward, the SAD value will become very large in contrast to the stance phase. In our experiment, the relatively small SAD value happens at the middle and at the end of every swing phase as shown in Figure
System trigger time determined by the SAD value.
Once the obstacle detection event is triggered, a median filter is then applied to suppress the bad influences coming from the environmental noise [
After applying the median filter, an intensity threshold is set to separate the line-laser pattern from the background. At this step, any pixel with an intensity below the threshold is recognized as background or noise. On the other hand, if the intensity of a pixel exceeds the intensity threshold, it will be considered as a candidate pixel on the obstacles.
Since our system is mainly designed for indoor application, the intensity distribution of different obstacles that are detected by the system is needed to be investigated. Therefore, twelve common construction materials are surveyed, as arranged in Figure
Common construction materials used indoors.
Reflection intensity distribution of common construction materials.
Subsequently, a line-laser pattern is extracted by the following steps. The pixel having the max intensity value in each column is collected to be the line-laser pattern pixel. However, to increase the performance, the image is initially rotated 90° for processing and then rotated back. In addition, because of the need to reduce the impact of noise, the average of the pixel intensities within a window is searched for each column instead, as indicated in Figure
(a) Max average value of pixels in a window is searched to be the extracted line-laser pixel. (b) Line-laser pattern extraction result. The clusters in the red circle, which denote the same obstacle, break into parts, and some small clusters from the noise exist in the image.
After the extraction of line-laser pattern, the extracted data are stored and processed again for obstacle clustering. Because of unstable line-laser intensities from the camera and disturbances from the environmental noise, the extracted line-laser pattern may suffer negative influence. In Figure
The goal of our system is to recognize how far and how large the potential obstacles are. Therefore, a segmentation algorithm is needed after executing the line-laser pattern extraction procedure. This algorithm classifies each pixel on the line-laser pattern into several clusters that are likely denoting the obstacles.
We assume all pixels of an obstacle will form only a continuous segment. In condition, the distance deviation of the current cluster in the image domain will not exceed a specific threshold
Search for line-laser pattern pixel by pixel for segmentation. If the
A suitable threshold
In order to eliminate the small clusters from the environmental noise, the width of the segments, which come from the noise of images, are counted. The statistical result indicates that the widths of 95% clusters of noises are below six pixels. Therefore, clusters with widths less than or equal to six pixels are recognized as noises. Furthermore, clusters with less than six pixels will have a 0.45 cm of width in the middle of our working region. If the horizontal distance between two neighboring clusters is less than the width of a foot, people may still kick on the gap between two obstacles and then trip over. Therefore, on the condition that the
(a) Result of line-laser pattern clustering. Some residual noise needs to be further eliminated. Besides, clusters that belong to the same obstacle should be merged. (b) The result after executing noise illumination and merging.
After executing the line-laser pattern clustering, the physical distance is needed to be determined by homography transformation. The calibration for homography transformation in our paper utilizes a checkerboard. The transformation denotes the relationship between the image coordinate and the real world coordinate of the checker grid, as shown in Figure
Calibration setup of our prototype.
A coordinate
By rewriting (
Four cross corners in real world coordinates and their corresponding points in the image coordinates are utilized to determine the homography transformation between two coordinates.
As long as the width and the depth of each obstacle can be calculated, the obstacle alarm level of each obstacle can be classified. Since the size and distance may affect the dangerous level, we use a coverage angle
Obstacle alarm level of each obstacle can be classified by the angle
The system distinguishes the obstacles into four alarm levels. Alarm level 1 means the obstacle is the most dangerous, causing the system to send the most urgent and loud alarm messages. On the other hand, when alarm level of the obstacles becomes 2 or 3, the alarm message will become weaker. Last, if the alarm level of the obstacles is 4, the system will not react and send any alarm messages.
In order to determine a proper SAD threshold for triggering the obstacle detection event, the camera continues recording SAD values during a gait cycle. A person wore the shoes and then walked around for a period of time. The collected data are shown in Figure
SAD values of several gaits. The threshold is determined below the average of SAD values in one gait.
Apart from the moment a user steps on the ground, the obstacle detection event should be triggered when people raise their foot at max height to detect higher obstacles. However, a SAD threshold to determine the moment in swing phase is difficult to be identified because of the unsteady characteristics of different steps. Besides, it may vary in the gait behavior. Therefore, a reasonable definition for the moment when the elders’ foot at max height highly depends on the collection of data. In our experiment, the moment is roughly at one third of a gait cycle, which has a relative small SAD value within the 3rd–7th frames during the swing phase. In practice, these five frames are temporarily stored, and the smallest SAD value is then selected as the moment for triggering obstacle detection event.
In obstacle detection event, we use a deviation threshold
In our system, a line-laser casts light on the obstacles and one camera observes the line patterns. Therefore, the cast line becomes broader and stronger at a smaller distance due to the perspective effect. However, after analyzing the line-laser pattern at each distance, the deviation threshold
On the other hand, the deviation of pixels of the same cluster varies and depends on the reflectivity of obstacles. Therefore, an experiment is designed to find out a suitable deviation threshold
Furthermore, an experiment is carried out to validate the precision of the distance estimation by homography. The line-laser obstacle detection system is mounted on a linear slider rigidly, and then the linear slider is used to adjust the distance between our system and a flat wall. The distance computation result by homography mapping will be compared to the standard distance measurement by the slider. From the validation experiment, the measurement error at 80 cm, the middle of the working distance, is less than 0.3 cm. At the nearest working distance, say 50 cm, the measured error is 1.7 cm, as shown in Table
Distance estimation.
Ground truth distance (cm) | Estimated distance (cm) | Error (cm) | Error (%) |
---|---|---|---|
50 | 51.7 | 1.7 | 3.4 |
60 | 61.1 | 1.1 | 1.8 |
70 | 70.4 | 0.4 | 0.5 |
80 | 79.8 | −0.2 | −0.3 |
90 | 88.9 | 1.1 | −1.2 |
100 | 99.5 | −0.5 | −0.5 |
The line-laser obstacle detection system classifies the obstacle alarm levels according to their angles. We define four alarm levels. When the obstacle coverage angle
The system performance is tested by detecting real obstacles, which may occur regularly indoors, as shown in Figure
(a) System detects a cable on the ground. The red clusters represent alarm level 1, the green clusters represent alarm level 2, and the blue clusters represent alarm level 3. The black clusters denote alarm level 4. (b) System detects scatterings on the ground. (c) The system detects a corner of a table.
With regard to the limitations of our method, the camera may suffer from the drawback of its low dynamic range. Besides the height of detectable, the obstacles must be larger than the plane level of the line-laser. Nevertheless, the feature of fall prevention is carried out by strategically sending an alarm message to the users.
In this paper, a camera-based line-laser obstacle detection system is proposed for designing fall prevention shoes of users. The system simply consists of an RGB camera, a filter, and a line-laser, so it is suitable to be installed on customer wearable devices, and the overall costs of the products are acceptable compared to shoes. We successfully verified the algorithms, including SAD threshold to trigger the obstacle detection event, line-laser pattern segmentation, homography transformation, and obstacle dangerous level classification. Finally, a prototype for the prevention of falls of the elderly was carried out.
The authors declare that they have no conflicts of interest.
This work was supported in part by the National Science Council of Taiwan under Grant no. NSC-100-2628-E-002-025-MY3.