Real-Time Chest Compression Quality Measurements by Smartphone Camera.

Out-of-hospital cardiac arrest (OHCA) is recognized as a global mortality challenge, and digital strategies could contribute to increase the chance of survival. In this paper, we investigate if cardiopulmonary resuscitation (CPR) quality measurement using smartphone video analysis in real-time is feasible for a range of conditions. With the use of a web-connected smartphone application which utilizes the smartphone camera, we detect inactivity and chest compressions and measure chest compression rate with real-time feedback to both the caller who performs chest compressions and over the web to the dispatcher who coaches the caller on chest compressions. The application estimates compression rate with 0.5 s update interval, time to first stable compression rate (TFSCR), active compression time (TC), hands-off time (TWC), average compression rate (ACR), and total number of compressions (NC). Four experiments were performed to test the accuracy of the calculated chest compression rate under different conditions, and a fifth experiment was done to test the accuracy of the CPR summary parameters TFSCR, TC, TWC, ACR, and NC. Average compression rate detection error was 2.7 compressions per minute (±5.0 cpm), the calculated chest compression rate was within ±10 cpm in 98% (±5.5) of the time, and the average error of the summary CPR parameters was 4.5% (±3.6). The results show that real-time chest compression quality measurement by smartphone camera in simulated cardiac arrest is feasible under the conditions tested.


Introduction
With a yearly number of out-of-hospital cardiac arrest (OHCA) incidents around 370,000-740,000 in Europe alone, and a low average survival rate of 7.6 % [1], OHCA is recognized as a major mortality challenge [2]. e time from collapse to care is crucial and there is a high focus on low response times of emergency medical services (EMS) [3]. A majority of EMS treated OHCAs are witnessed [4], and quality cardiopulmonary resuscitation (CPR), until EMS arrives, can have positive effects on survival [5][6][7]. e witness is often in close relation with the patient and could experience the situation as extremely stressful [8]. Studies have shown that telephone-assisted CPR (T-CPR) has a positive effect by getting more callers to start CPR and coaching callers to provide quality CPR [9][10][11].
Furthermore, CPR feedback has been shown to improve CPR quality [12][13][14][15]. Combining T-CPR with CPR feedback may improve CPR quality and survival from OHCA.
In the recent statement from the America Heart Association (AHA), the use of digital strategies to improve healthcare in general and to document its effect is encouraged [16,17]. Devices providing the bystander with CPR quality measurement by utilizing an accelerometer to measure CPR metrics are available [18][19][20]. A challenge with these devices is to get the users to carry it with them at all times. Smartwatches has a built-in accelerometer, and has been suggested as a tool for measuring CPR metric [21][22][23]. However, a very small percentage of the population wears a smartwatch at all times. e smartphone, on the contrary, is a digital device most people carry with them. In recent years, smartphone applications have been developed for CPR quality measurement and to support learning [24,25] and to help communicate the location of an emergency [26]. In addition, there are publications describing the use of the accelerometer in smartphones to measure CPR metrics [25,[27][28][29][30]. Smartphone solutions utilizing the accelerometer require the smartphone to be held on the patient's chest or strapped to the bystander's arm while performing CPR. ese solutions may be more suited for training than for actual emergencies since buttons causing phone connection interruptions with the emergency unit can accidentally be pressed when performing the compressions.
Our research group has earlier presented an application, QCPR cam-app 1.0, utilizing the smartphone camera to estimate the chest compression rate and provide feedback to both the bystander and the dispatcher while the phone is placed flat on the ground [31]. Besides from a small offline study by Frisch et al. [32] we have found no other published work or products that utilize the smartphone camera when measuring compression rate. QCPR cam-app 1.0 demonstrated accuracy issues when challenged with bystanders having long loose hair and in cases of people moving around the emergency scene. In this paper, we present test results of QCPR cam-app 2.0, improved to handle this, but also to provide more information by calculating a CPR summary report after CPR has ended. ese parameters can be used to evaluate each session and to generate data that can be used for dispatcher-caller quality improvement and research.

Materials and Methods
e application, QCPR cam-app 2.0, captures CPR movements utilizing the smartphone camera while the smartphone is placed flat on the ground next to the patient. From the detected motions, the algorithm estimates the chest compression rate and hands-off time and provides: (1) realtime objective feedback to the bystander, (2) real-time objective feedback to the dispatcher during the emergency call, and (3) a CPR summary report.

Illustration of Bystander and Dispatcher
Use. An illustration of the application in use can be seen in Figure 1(a), with screenshots in Figure 1(b). By clicking the emergency button, the application activates speaker mode, establishes telephone connection with the dispatcher and sends GPS location and real-time compression data to a web server available for the dispatcher. e bystander then places the smartphone at the opposite side of the patient, see Figure 1(a). e preview frames from the front camera are shown to the bystander, allowing him to position himself and to keep track of the ongoing activity in the field of view of the camera (Figure 1(b)). A speedometer is displayed next to the preview frame allowing the bystander to keep track of the applied compression rate.
A live sequence example of the proposed web server solution monitored by the dispatcher is shown in Figure 1(c). A 20 seconds sliding window providing the development and history of the compression rate in real-time is shown, where different colors are used to make the interpretation easier. Green dots correspond to compression rates in the desired range of 100-120 cpm and yellow outside. Above the graph, a circular color indicator provides information about the certainty of the reported compression rates. If the detections are carried out in low noise, the indicator is green, but if high noise conditions are present, i.e., some cases of long loose hair and from large disturbances, the indicator shifts to yellow. e bystander's GPS location is provided to the dispatcher, as seen in Figure 1(c).

Technical
Description. QCPR cam-app 2.0 was designed to handle the disturbance issues observed in QCPR cam-app 1.0, [31] and the technical description of the improvements are presented in more detail in the appendix. In short; All the estimations are performed on the smartphone, and the main steps in detection of compression rate are illustrated in Figure 2. In step 1, difference frames, g(i, j), are generated by thresholding the differences between subsequent input frames, f(i, j), from the camera. A dynamic region of interest (ROI) is established from the largest connected moving object and is updated each half second by checking the activity in the blocks around the ROI boundary. By using a dynamic ROI, we allow others to move around in the emergency scene without disturbing the detections. In step 2, we generate a signal, d(l), from the activity in the ROI and for each half second, timestep n, a short time Fourier transform (STFT) is performed on the three last seconds of d(l). A sliding Hanning window is applied to d(l) prior to the STFT. In step 3, the power spectrum density, D n (ω), found from the STFT is studied and a decision three is used to separate compression rates from noise. e decision three recognizes a system in the D n (ω) for cases of bystanders with long loose hair, thus solve the detection issues observed in QCPR cam-app 1.0 [31] for these cases. If a CR(n) is detected, it further undergoes some postprocessing steps, indicated in step 4, Figure 2. ese steps filter out and suppress noise by performing smoothing and removing short detection pauses caused by compression stops or disturbances. In step 5, the detected and filtered compression rate, CR f (n) (cpm), is displayed on the smartphone and sent to the web server and displayed to the dispatcher, providing the real-time feedback to both bystander and dispatcher.

CPR Summary Report.
After completion of a caller session, a set of CPR summary parameters are calculated by QCPR cam-app 2.0. e parameters, which are both shown on the smartphone screen for the bystander and saved on the web server for the dispatcher, are as follows:

Data Material and Evaluation
Measures. All experiments were performed on a Resusci Anne QCPR manikin. e QCPR cam-app 2.0 algorithm was implemented in Android Studio and the experiments were performed with a Sony Xperia Z5 Compact (Sony, Japan). A reference signal for the compression rate was provided by an optical encoder embedded in the Resusci Anne QCPR. A three-second long sliding window frequency analysis was performed on the signal each half second, providing the reference data, CR true (n), with the same sample rate as the compression rate detection, CR f (n), from the app.
To evaluate the results, different measurements were used: Average error (E), Performance (P), Relative error parameter (RE par ) and Bland Altman plots used to visualize Video frames Difference frames Difference signal Power spectral density 40-160 cpm Output comp. rate Figure 2: Simplified block scheme of the proposed system for chest compression rate measurement. Image frames from the smartphone front camera is used as input, and output is the detected compression rate, CR f (n).
the agreement between data provided by QCPR cam-app 2.0 and the reference data provided by Resusci Anne QCPR manikin. E is given in compressions per minute (cpm) and is the average error of the sequence, defined as where N is the number of samples of the sequence. For sequences containing discontinuity in the reference data, i.e., 30 : 2 session, we allowed errors in a ±1 s interval around the automatically detected discontinuities. is reduced the influence of insignificant delays on the error measure. P is defined as the percentage of time where |CR f (n) − CR true (n)| < Δ. According to guidelines [33][34][35] the acceptable compression rate is between 100 and 120 cpm, thus Δ � 10 (cpm) was chosen as an acceptance criterion. RE par measures the performance of the CPR summary parameters listed in Section 2.3. RE par is given in percentage and defined as where Par D is a CPR summary parameter estimated by the app and Par R the corresponding CPR parameter found from the reference signal. If the test contained more than one sequence, the results are presented with mean and standard deviations, i.e., μE(σE), μP(σP), and μRE par (σRE par ), found over the result values of the sequences. Desired detection results provide a low Average error, E, a low Relative error parameter, RE par , and a high Performance, P.

Experiments.
e performance of the QCPR cam-app 2.0 was tested in various conditions that could occur in real emergencies. e experiments were divided into five different tests-Smartphone position test, Outdoor test, Disturbance test, Random movement test, and CPR summary report test. Altogether, this sums up to approximately 162 minutes of CPR. Specifications for the subtests included in each test are listed in Table 1.
e Smartphone position test included seven test persons-two with short hair (SH), two with medium length loose hair (MLLH) i.e., chin/shoulder length, and three with long loose hair (LLH) i.e., chest length. Each of the test persons performed 8 subtests carried out indoor. e result for subtest RateP1, Table 1, was presented in Meinich-Bache et al. [36] to verify that QCPR cam-app 2.0 is able to estimate correct compression rate for test objects with various hair lengths and for different compression rates, which were an issue in QCPR cam-app 1.0 [31]. e subtest D1R110P1 included a person that walks around and behind the bystander during CPR, leaning over the patient, waving his arms, and thus causing disturbances.
ese results were also presented in Meinich-Bache et al. [36] to verify improvements of QCPR cam-app 2.0 over QCPR cam-app 1.0 where sometimes disturbances could take over the dynamic ROI [31]. e results of subtests RateP1 and D1R110P1 are repeated here for the reader to experience all the various tests that QCPR cam-app 2.0 has been exposed to.
Various other conditions were also tested in the Smartphone position test. ree camera positions were included: next to shoulder (Pos.1), 20 cm away from shoulder (Pos. 2), and next to head (Pos. 3). e camera positions are shown in Figure 3. 30 : 2 sessions were carried out for camera positions Pos 1, subtest 30 : 2P1, and Pos 3, subtest 30:2P3. Pos.3 was included to see if the algorithm provides false detection when the bystander is still visible in the image frame when performing rescue breaths. Since the bystander is not visible in the image frame while performing rescue breaths when the camera is positioned in Pos.2, this position is not relevant for the 30:2 sessions and therefore not included. Pos.2 is used to measure the algorithm's ability to detect when only a small part of the bystander is visible in the image frame and used in subtests R100P2 and R150P2. e algorithm was also tested in low lighting conditions, 7 lux, in subtest LightP1.
e Outdoor test included three test persons, one with each hair length; SH, MLLH, and LLH. e detections were carried out in cloudy (C) and sunny (S) weather, both with and without noisy background (B) i.e., trees.
e purpose of the Disturbance test was two-fold: (1) to measure the algorithm's ability to detect compression rate when there is a large disturbance present i.e., another moving person, and to (2) quantify the disturbance size relative to the bystander performing the compressions when the algorithm fails to detect due to too much noise. A second Sony Xperia Z5 Compact (Sony, Japan) phone was used to capture video recordings of the test, and the video is studied offline to perform the quantification. e bystander carried out continuous compressions during the sequence. e disturbing person moved around the patient, waving arms in different frequencies, standing behind and over the bystander while waving arms, stepping over patient etc.
In the Random movement test, no CPR was performed on the manikin and the purpose of the test was to measure the algorithm's resilience to false detections. e random movement included checking breathing and pulse of patient, turning patient, unzipping jacket, walking around, waving for help etc. ree test persons were included.
e CPR summary report test is an evaluation of the session summary parameters. e test included five different test persons with different hair lengths and the following test protocols: (i) e bystander sits next to patient with the smartphone in his hands. He/she presses the emergency call button and places the smartphone flat on the ground. For approximately 20 seconds, the bystander checks for patient's pulse and respiration before starting performing chest compressions. (ii) Next, four intervals of 120-second continuous compressions and 20-second pauses while checking for respiration are followed. (iii) e total sequence time is approximately 580 s, which is a typical response time for medical assistance [37][38][39][40].
e CPR summary parameters evaluated are the parameters explained in Section 2.3: TFSCR, TC, TWC, ACR, and NC.

Results
e error measurement results of all five tests are summarized in Table 2. e average compression rate detection error, E, was 2.7 compressions per minute (±5.0 cpm), the performance, P, accepted detections in 98% (±5.5) of the time and the relative error of the CPR summary parameters, RE par , were 4.5% (±3.6). In subtest R150P2 from the Smartphone position test, the results reveal some weaknesses when only a small part of the bystander is visible to the camera, the compression rate is as high as 150 cpm and the person performing compression has MLLH or LLH. In the two sequences with poor results, P of 56.2% and 80.2%, the bystander is only present in 4.6% and 6.9% of the image frame and an example from the largest one is shown in Figure 4   Journal of Healthcare Engineering e Bland Altman plot in Figure 5 shows the agreement between reference data and detection data for Smartphone position test, Outdoor test, and the Disturbance test. Each analysis in all the test sequences are here included.
e subtests with poorer results, 30:2P1, R150P2, and 30:2P3, is marked with the colors red, yellow, and purple, respectively. e total number of samples in the plot is 11718, and the number of samples with larger   Journal of Healthcare Engineering deviation than ±10 cpm compared to reference data is 180 (1.53%). In Figure 6 the Bland Altman plots show the agreement between the summary parameters calculated from the detection data and the summary parameters calculated from the reference data in the CPR summary report test.

Discussion
e results presented in this paper show that the camera in a smartphone can be used to measure chest compression rates and hands-off times under various conditions with good accuracy. Our proposed method allows for real-time feedback to both the bystander and to a dispatcher in real emergencies, which could improve CPR quality.

Challenges.
Although the algorithm works well with only a small part of the bystander being visible under low noise situations, we discovered reduced accuracy in two of the sequences where the bystander had long loose hair, compressed with a very high rate and were visible only in a small part of the image frame. In these sequences, the loose hair is sometimes almost the only thing visible in the image frame and QCPR cam-app 2 interprets this as compression in the rate the visible hair is bouncing in. ese two cases explain the yellow samples, in R150P2, that caused disagreement in the Bland Altman plot, Figure 5. To avoid these false detections the bystander should position the smartphone such that most of the head and shoulders are captured in the image frame.
We also experienced that repetitive random movements during compression pauses could cause the algorithm to detect a false stable-low compression rate causing QCPR cam-app 2 to calculate a longer TC and a shorter TWC. It could be observed that during compression pauses people often bend towards and away from the patient in a sometime very repetitive movement, and on a few occasion when the bystander had long loose hair, these movements caused the algorithm to interpret the movements as a stable but very low compression rate lasting a minimum of 5 seconds. ese false stable-low compression rate detections did not occur in the Random Movement Test when the test persons were asked to perform all kinds of different tasks that could be carried out before compression starts. Deactivating the dynamic rate range could solve this problem, but a consequence of this would be that compressions rates below 70 cpm would not be detected. e samples that show disagreement between the detections data and the reference data in Figure 5 for subtest 30 : 2P1 (red) and 30 : 2P3 (purple) occurs in the transitions between compression and compression pauses when performing 30:2 and do not significantly affect the visual presentation of the detected signal that is shown to the dispatcher.

Further Work.
e proposed system allows the bystander to have both hands free with compression feedback on the smartphone screen visible next to the patient which is different from accelerometer-based smartphone solutions that require the smartphone to be held on the patient's chest or strapped to the bystander's arm [24,25,[27][28][29][30]. is advantage could make the proposed solution suited for real emergencies where the phone is also used as a life line to the emergency unit. Studies comparing the proposed solution with the accelerometer-based solutions in simulated emergencies should be considered.
Testing of QCPR cam-app 2.0 in simulated real emergencies must be carried out in order to conclude if this method could be suited for real emergencies. In addition, studies with the aim of documenting the usability of the application, safety of the method, and effectiveness on the CPR quality also need to be carried out as suggested by Rumsfeld et al. [16]. If QCPR cam-app 2.0 shows a welldocumented positive effect on the CPR quality, it may be subject to appropriate medical device regulations and made available for clinical use [41,42]. e detected and stored compression rate signal and the CPR summary report provide further opportunity for evaluation, debriefing and quality improvement of the dispatcher-caller interaction. e stored data and the visual dispatcher feedback system can be used to provide continuing education in T-CPR for dispatchers, as AHA recommends in T-CPR guidelines [43]. In addition, these measurements can provide the EMS arriving at the scene with detailed information about the treatment the patient has received. A feature which records audio and video will be considered integrated in QCPR cam-app 2. A possible solution could be to let the recordings be automatically uploaded to a cloud storage when available bandwidth  would allow it. Still images, video, and audio could be made available for the dispatcher and allow for a better understanding of the emergency situations. Audio recordings may also be analyzed with respect to chest compression rate and inactivity to further improve measurement accuracy since most dispatcher protocols include prompting and counting loud while compressing on the chest. e collected data could also be utilized in a machine learning framework providing potential decision support in future systems.
We are currently investigating camera-based methods for measurement of compression depths [44]. In future work, we will try to develop a robust depth algorithm that could be implemented together with the proposed method. An implementation of depth measurement would make this solution a complete CPR quality measurement and feedback device. Although the proposed solution's main idea is to assist laypersons in real emergencies, we have also developed a training version of the solution called TCPR Link, available on App Store and Google Play [45,46] in selected countries. As AHA has announced, CPR feedback devices will also be required to use in all AHA CPR courses by February, 2019 [47].
Studies have also shown that both laypersons and professionals could benefit from objective feedback during CPR. In a study presented by Abella et al. [48], the CPRcertified rescuers performed chest compression rates <80 cpm in 36.9% of the CPR segments included in the study and rates of 100 ± 10 cpm in only 31.4% of the segments, clearly suggesting that CPR-certified rescuers could also benefit from the proposed solution.

Conclusion
Real-time chest compression quality measurement by smartphone camera is feasible for a range of bystanders, compression rates, camera positions, and noise conditions. is technology may be used to measure and improve the quality of telephone CPR and minimize hands-off times.

Appendix
Method is appendix provides a pseudocode description of method for measurement of chest compression rate. More details can be found in [31,36]. e application is called TCPR link and is available on App Store [43] and Google Play [44].
Let the input, f l (i, j), represent video frames, l, where (i, j) corresponds to row index i and column index j. Output is the filtered compression rate measurement, CR f (n), for each 0.5 sec analysis interval, n.
Input: f l (i, j), Output: CR f (n) while receiving image frames do Activity measurement: (1) Generating difference frame: for All pixels in frame do if |f l (i, j) − f l−1 (i, j)| < � ε then g l (i, j)⟵0 else g l (i, j)⟵f l (i, j) − f l−1 (i, j) end if end for (2) Dividing g l (i, j) into non-overlapping blocks and finding the sum of change in region block, R k over the received frames, L, in the last half second: Marks the blocks with S L R k (n) > than the average block activity, S L R (n), with an indicator function, Attributes found from D n (w): (1) Amplitude of the first significant peak, a p1 (n), (2) Amplitude of the second significant peak, a p2 (n), (3) Frequency of the first significant peak, f p1 (n), (4) Frequency of the second significant peak, f p2 (n) and (5) Mean amplitude hight of PSD, a PSD (n). CR(n)⟵decisionTree(a p1 (n), a p2 (n), f p1 (n), f p2 (n), a PSD (n), Post processing CR f (n) � CR(n) (10) Short spike/drop removal: if (11) Smoothing mean filter: forj � 1 : 3 do K � argmax J |CR f (n) − CR f (n − j)| < T mf , ∀j ≤ J end for CR f (n) � K k�0 a k CR f (n − k) where a k is the filter coefficients, K k�0 a k � 1 and a j � a i ∀ i, j.

Conflicts of Interest
Tonje Søraas Birkenes and Helge Myklebust are employees of Laerdal Medical.