Evolution of an Early Illness Warning System to Monitor Frail Elders in Independent Living

ABSTRACT This paper describes the evolution of an early illness warning system used by an interdisciplinary team composed of clinicians and engineers in an independent living facility. The early illness warning system consists of algorithms which analyze resident activity patterns obtained from sensors embedded in residents’ apartments. The engineers designed an automated reasoning system to generate clinically relevant alerts which are sent to clinicians when significant changes occur in the sensor data, for example declining activity levels. During January 2010 through July 2010, clinicians and engineers conducted weekly iterative review cycles of the early illness warning system to discuss concerns about the functionality of the warning system, to recommend solutions for the concerns, and to evaluate the implementation of the solutions. A total of 45 concerns were reviewed during this period. Iterative reviews resulted in greater efficiencies and satisfaction for clinician users who were monitoring elder activity patterns.


INTRODUCTION
One goal for the safe and usable design of clinical information technology is to maximize safety and minimize error.To achieve this goal, end-users, developers, and domain experts must work together as a team throughout the development process.During the development process, the team should base design decisions on user characteristics, understanding problems encountered by users, and on human information processing abilities as users interact with the clinical technology to make clinical decisions or to document care provided.Positive outcomes from this process include fewer errors involving patients, healthcare personnel, and other users; decreased training costs; a better fit with the way clinicians work and think; reduced time spent redesigning systems; and greater user satisfaction.
This paper describes the evolution of an early illness warning system being designed by members of the Center for Eldercare and Rehabilitation Technology (also called the Eldertech team) at the University of Missouri in the Midwestern US.The early illness warning system is being developed and used in an independent living facility, TigerPlace.The early illness warning system incorporates algorithms that issue automated alerts to clinicians, warning them of potential declines in activity levels and acute health events experienced by residents living in TigerPlace.

Human Factors Theories
During the lifecycle of a clinical information system, human factors theories are used by developers to guide research focused on safe and usable design of clinical information technology.Within these theories, three domains for human factors are identified, including the machine, the operator or the end user of the machine, and the environment [1,2].Human factors experts study human capabilities and limitations, and apply that knowledge to the design of safe, effective, and comfortable products, processes, and systems for the human beings involved [3].Important frameworks have been proposed for classifying software defects and user interaction problems such as layout of an interface, language included in labels, and visual cues [4].Outcomes that are useful for human factors experts to use during evaluation assist in determining the effectiveness, efficiency, and user satisfaction during users' interaction with computer interfaces [5] (See Figure 1).These outcomes have been used recently to identify important design issues in clinical technology [6].For example, in Human Computer Interaction effectiveness research, exploring the usefulness and safety of clinical 338 Evolution of an Early Illness Warning System to Monitor Frail Elders in Independent Living Human factors subsystems and outcome measures technologies, interactions with newly developed clinical software, inflexibility of computer interfaces, and poor navigation caused users to get lost in software applications.In efficiency studies, learnability, defined as the capability of a software product to enable users' to learn how to use it, and productivity were improved by increasing the screen density or amount of information on each computer screen.Finally, satisfaction, which is user perceived effectiveness or perceived efficiency, was greatly improved when graphics were included in the interface and were found to be very important for better navigation across applications.The evolution of our early illness warning system is described in this paper using human factor subsystems and outcomes as a framework to illustrate the development of such system being implemented in an independent living facility.

Sensor Technology for Early Indicators of Acute Events in Elderly Populations
Aging populations are increasing all over the world [7].With this dramatic shift, new methods are needed to monitor health and functional activity of older adults, while maximizing a clinician's time and effort to monitor them effectively.Sensor networks for eldercare are being investigated around the world as a means to facilitate better monitoring of aged people.For example, in Italy, wearable fall sensors are being used to monitor gait acceleration, sway and staggering [8].Researchers in Finland are experimenting with automated controlled light sources in smart homes [9].Furthermore, Honeywell has developed a passive monitoring system called a Life Style Assistant, which monitors mobility and medication compliance of elders [10].The uniqueness of our monitoring system is that it has been incorporated in a real life independent living setting where the research is being conducted.It is not a demonstration project; the project is longitudinally spanning over 5 years, and we're incorporating sensor information into electronic health records to facilitate better decision making by clinicians [11].
Clinicians have anecdotally reported for many years that functional changes (e.g., confusion, decreased activity, falling) are often an early indicator of an impending acute illness or an exacerbation of a chronic illness.More recently, these observations are being confirmed and research is focused on identifying ways to recognize at-risk older adults [12].Early illness recognition and treatment is key to improving health status with more rapid recovery after an acute illness or exacerbation of a chronic illness, and preventing morbidity and mortality in older adults [13][14][15].
While traditional assessments by health care providers can reveal signs of impending acute events, historically, this has involved a face-to-face assessment, either with the older adult physically present with the health care provider or, in some cases, connected by expensive telemedicine equipment.New ways of detecting subtle changes are needed; ways that are unobtrusive, yet accurately assess changes from an individual's normal activity pattern; ways that do not require face-to-face assessment of the individual by a health care provider; and ways that use what people normally do in the course of their daily activities of living to detect changes in normal behavior and to alert health care providers that further assessment is needed.We believe that through the use of a sensor technology system, we can accomplish the goal of earlier detection and intervention at the onset of acute health events.Using and evaluating the impact of such a sensor system is especially critical for the rapidly expanding population of older adults who are at risk of no longer being able to live at home following acute health events that result in hospitalizations or nursing home placement [16,17].

Current Integrated Passive Sensor Network
A variety of passive sensors are available for detecting motion, location, and functional activity.In our current work, we have installed a network of passive sensors in apartments of 34 older adults since 2005.Since fall 2005, 108 residents have been asked to participate; resulting in a recruitment rate of 31.5%.Written informed consent was obtained through the University's Institutional Review Board from every resident who had sensors installed in his/her apartment.
Our sample includes mostly women (n = 27), who are Caucasian; resident ages range from 64 to 97 years (mean: 86 years, Standard Deviation: 6.92 years).There is 1 married couple, all remaining residents are single.Sensors have been installed for an average of 1.8 years among all participating residents (range .4 to 5.05 years, Standard Deviation 1.23 years).Average resident length of stay is just over 3 years.Approximately 80% of the residents have at least one chronic disease, but many more have more than one including diabetes, heart disease, and arthritis [18,19].We have established the protocol for capturing and interpreting the data from these sensors which measure activity and physiological parameters (see Figure 2) [11].
In an apartment where one couple lives, we have an algorithm that generates a confidence of visitors (based on motion sensors firing in different locations in the apartment); with this algorithm, we tend to get higher than normal activities when multiple people are in the apartment.These sensors provide information about functional activity, operationalized as where a resident spends time in his/her apartment, physiological data including measures of restlessness and vital signs, and 340 Evolution of an Early Illness Warning System to Monitor Frail Elders in Independent Living when certain activities in the apartment occur.For example, functional activity informs users that a resident spends some time during the day in the kitchen and uses the stove, that a resident is frequently up and out of bed during the night using the bathroom, or while in bed, a resident experiences some periods of restlessness rather than a quiet, restful sleep.The network of motion, bed, and stove sensors illustrated in Figure 2 is being used by the Eldertech team to develop the early warning system.(The video sensor network has not yet been installed for continuous use in apartments; research and deployment plans are in progress.) The integrated sensor network uses passive infrared (PIR) motion sensors with X10 technology.Motion sensors are installed in targeted locations (i.e., bathroom, bed and bedroom) to detect activities (See figure 3).A bed sensor detects presence in bed as well as restlessness, breathing and pulse while sleeping (see figure 4).A data logger collects the motion and bed sensor data, time-date stamps each sensor firing, and generates a file which is regularly transmitted via a de-identified binary stream to a central server.Sensor data events are recorded into an activity database for residents who have the information system installed in their apartments.Embedded within this sensor network is an activity analysis module that is used to build a clinical decision support system that initiates alerts to health care providers about certain potentially important changes in sensor events that occur in the sensor data.Every data element in the network of sensor data can be visualized on a computer screen through a web-based interface that can be used to determine baseline activity patterns, to follow adverse health events that occur over time, and to establish criteria for when an alert should be initiated.Costs for the sensor system have been estimated just under $600.00 for each resident's apartment, excluding the ongoing technical support, installation costs and large data storage required for maintaining ongoing data.Figure 3 provides an example of the types of motion activities which are monitored on a daily basis by each sensor system installed and functioning.Figure 4 provides an example of the web-based interface showing changes in a TigerPlace resident's respiratory rate (increasing slow rate of breathing at night while in bed) that triggered an alert for potential change in health status.

Figure 3.
Histograms indicating the types of motion activities being monitored in each apartment with sensors.
This research was approved by the University of Missouri Institutional Review Board for protection of human subjects.Clinicians (4 expert gerontological nurses with one also having expertise in information science, and a family physician) were assigned to view sensor data on the secure web-based interface for 4-5 participating residents; this provided overlap of clinicians so that each resident's data were reviewed by more than one clinician.An iterative process of weekly team meetings of the clinicians with computer engineering research team members were used to inform the algorithmic development process for alerts that would eventually guide clinicians to potential detection of early illnesses.Computer engineering (CE) research team members adjusted algorithms and data displays, while the health care team members provided ongoing feedback so that adjustments functioned in the expected way for end-users (clinicians including nurses and physicians).We utilized the human factors subsystems (Machine, Operator, and Environment) and outcome measures (Effectiveness, Efficiency, and Satisfaction) to demonstrate the types of issues raised during our collaborative work, while developing the early illness warning system.
Data decision trails, alert conditions and logs of alerts received were maintained by both clinician and CE team members.These trails provided a source of documentation for the system developed, as well as a data source for evaluation.The plan was for clinicians to first retrospectively review the sensor data around known health events (emergency room visits, hospitalizations, and falls).This process took about four months and informed initial alert algorithm design.Then, the clinicians conducted prospective reviews of sensor data for each assigned resident for two months while initial alerts were developed by CE team members.Finally, clinicians received prospective alerts by email with a convenient link to the sensor interface so that they could readily connect to the sensor interface within the time frame of the alert condition.A link to evaluation 342 Evolution of an Early Illness Warning System to Monitor Frail Elders in Independent Living questions about the specific alert was also provided that enabled the clinician to respond to how effective alerts were at communicating significant changes in behavior.
Clinicians could indicate whether or not the alert had clinical value, were too sensitive to change, and which kinds of changes were viewed as most sensitive.For example, increases in bathroom activity at night might be viewed more important than increases in living room activity during the day (see Figure 3).Clinicians also noted valuable information when alerts were not as useful, suggesting that these alerts needed to be filtered out of the system.An example is when alerts for decreased activity occur during a period when a resident is out of the apartment for personal reasons, which was usually noted during weekly visits by the registered nurse coordinator at TigerPlace.Prototype alerts were developed by CE team members.Prototype alerts were designed using a simple algorithm based on finding outlying samples on a normal distribution, where the outliers indicate alerts.Samples consisted of daily periodic counts of sensor events as well as time in a location and density of motion (sensor events per unit time).Sample periods were initially set from Midnight to 23:59 for 24 hour alerts and 10 pm to 6 am for night time alerts.The normal distribution consisted of samples from a sliding window of x days immediately prior to the day of the alert; identifying the parameter x required experimentation and discussion by all team members.This algorithm was run on a set of sensors placed in each resident's apartment.Several time periods which contained known resident emergency room visits, hospitalizations, and falls were put through the algorithm and the results were compiled and shown to clinicians to get their feedback on how sensitive the algorithm should be to outliers.After analyzing and discussing the results, initial parameters were set to use a sliding window of two weeks prior to the alert being checked and to label outliers as sample points four standard deviations outside the mean.Alert's were sent to clinicians on a daily basis in the form of emails which contained links that allowed them to go to the web interface and see the sensor data pertinent to the alerts being generated, as well as to leave their clinical analysis on each alert.The clinical analysis on each alert is particularly important to get a general idea of which alerts are more clinically relevant in order to customize the alerts for the future, as well as to design new more sophisticated alerts that fuse several sensor outputs together, for instance, in systems of fuzzy rules [20].

RESULTS
Retrospective review of the sensor data around known health events (emergency room visits, hospitalizations, and falls) revealed key information that guided the initial development decisions for the early warning system.Patterns detected in this review revealed that in many cases, changes in some of the sensor data could be seen within a two week window of time preceding the events.For example, in Figure 4, there appears to be an undulating pattern of increased slow breathing followed by decreased slow breathing, while simultaneously the normal breathing pattern remains relatively constant.This change could indicate an impending respiratory condition.Based on our algorithm, an alert was issued for this condition, which could be used to notify a clinician of an impending problem.Prospective studies are currently underway to evaluate the relationships between when alerts are issued and resident outcomes at the Journal of Healthcare Engineering • Vol. 2 • No. 3 • 2011 Independent Living time of the alert.Another example is illustrated in Figure 5.A resident came home following open heart surgery and appeared to have sustained episodes of increased restlessness while in bed.Knowing the history, this resident could have been having postoperative pain uncontrolled by pain medication.Later on, the resident decided to move to his family member's home for more help, and upon returning to his own home appeared to have returned to baseline levels of restlessness, which could indicate the comfort levels had also returned to baseline.
These types of discoveries guided the time frame to be used for displaying the sensor data, a time-saving strategy for clinicians.Prior to these discoveries, clinicians had to navigate several displays showing months or weeks of data.This navigation with multiple time frames resulted in several minutes of time spent with every data display for each assigned resident; thus clinicians spent 10-17 minutes per resident per week reviewing displays.The clinician started with an initial display window automated to show the data from the likely time frame that was most clinically useful, then the clinician could drill down to tighter perspectives or zoom out to larger blocks of time as their interpretation progresses.
With prospective review of sensor data completed by clinical staff twice each week for their assigned residents, more suggestions were revealed in the weekly team meetings.CE research team members continuously revised the web interface for the clinicians, iteratively improving the clinical usefulness and speed of interpretation of potentially clinically relevant sensor information.These suggestions are displayed in Table 1, which illustrates concerns raised within each human factor domain (Machine, Operator, Environment) by each discipline (clinicians and engineers) and potential solutions recommended during each iterative review cycle at weekly meetings.Table 1 Figure 5.
Bed restlessness sensor data with adverse event.for each discipline participating in the weekly collaboration during the developmental phase of the early illness warning system from January 2010 to July 2010.A total of 45 concerns were discussed during this period.With the changes implemented over this period, the clinician time for reviewing displays has been reduced to 4-12 minutes per resident per week.This resulted in total savings of between 5 and 6 minutes per resident per week.We anticipate more reductions in time spent by clinicians as the interface is used next year in a prospective study in progress.During the prospective study, clinicians are comparing clinical outcomes of residents with care coordination enhanced by sensor networks and early illness recognition with other residents who have chosen not to have sensors installed in their apartments.Next, we will discuss concerns and solutions for singular and combined disciplines, which have resulted in greater efficiency and satisfaction for our collaboration.

Clinicians
Seven of the concerns about the early illness warning system were solely raised by clinicians.The majority of these concerns (4) were related to functions the operator could perform.For example, clinicians wanted to extend the use of the early warning system as a diagnostic system to identify trends, which could be useful as a clinical decision support tool for prompting clinicians when resident activity levels change.
Other issues raised in the Operator domain regarded the workload of the clinicians.For instance, when there was an increase in the responsibility of the clinicians to monitor more residents, which generated more alerts per clinician, clinicians became concerned about the uncertainty of the number of alerts they would be receiving each day.To address workload issues, clinicians recommended to improve the machine interface and reduce the number of tasks needing to be performed by creating defaults on the sensor interface, which set the default ending date range for sensor data review to 14 days prior to each alert.This change meant that there would be fewer data search tasks to be performed and would save valuable clinician time while searching and assessing volumes of residents' data.Additionally, workloads (resident assignments) were redistributed for each clinician based on diagnosis and medical history provided by the TigerPlace nurse care coordinator in an effort to spread the surveillance of higher acuity residents, which could generate more alerts, among clinicians.Clinicians also verbalized concerns for the evidence base used to support the features in the algorithm to create the warning mechanism.Clinicians wanted to add greater functionality to the automated system by including additional means to assess resident function, such as incorporating Actigraphs to measure sleep latency and efficiency, which would support decision making about restlessness measures in the early illness warning system.Clinicians also pointed out that some sensors seemed to make intuitive sense to measure resident function, while others may seem less significant.Increased bathroom visits, especially those occurring at night, contributed greatly to the clinicians' assessment of function.This was found to be so important that clinicians desired to have specific alerts created for nighttime activity levels in the bathroom and other areas of the resident apartment.Furthermore, clinicians were not satisfied with the ability to view these data via histograms; they wanted more information in the form of motion density maps, which provide more information per view (denser screen output) than a histogram map. Figure 6 shows a life style pattern displayed as a motion density map for 6 weeks of data [20,21].The horizontal axis is the day of the month; the vertical axis is time of day from midnight to midnight; the usual sleeping hours are mostly on the top of each motion density map.The black segments are time out of the apartment (such as for meals, activities, or community events).The more colorful areas are time of activity throughout the apartment.The color bar on the right shows increasing density colors from top to bottom; gray and yellow show lower densities (less movement), and green and blue show higher densities (more movement).

Engineers
Twenty-one of the singular concerns were related to the engineers' role in developing components of the early illness warning system.The majority of these concerns were issues related to developing the machine components of the system.Issues addressed included: (a) creating standardized outputs of visual scales and color of graphics to enable more efficient evaluation of sensor information across individual sensors and residents; (b) programming of the alerts to be sent to the clinicians, which included determining the most effective method for sending alert messages, content of alert messages, and incorporating recommended standard deviation parameters into alert configurations; (c) maintaining the functionality of the alert system; (d) positioning additional chair sensors when needed and exploring the use of added chair padding on sensitivity of sensors to maintain resident comfort and satisfaction levels; (e) configuring new sensor systems when residents were entered into the research.Other concerns by participating residents and addressed by engineers included location of hardware in the apartment.For example, it was important for equipment to seamlessly fit into the living areas [22].These issues increased resident satisfaction with the system.
There were eight concerns from the Environmental human factors subsystem raised during the iterative review and involved IT support from engineers.Issues addressed involved setting up access rights for residents, family members, and clinical staff, maintaining privileges for clinicians and researchers who were approved by the 358 Evolution of an Early Illness Warning System to Monitor Frail Elders in Independent Living Institutional Review Board to review the data, and providing instructions for accessing the protected websites where the sensor data could be viewed.Finally, the engineers were responsible for maintaining the integrity of the server and equipment in accordance with institutional policies and requirements, while maintaining and monitoring ongoing functionality and providing IT support for overall system efficiency and effectiveness.

Clinicians & Engineers
Seventeen of the concerns addressed during the iterative review cycle were shared between clinicians and engineers.The majority ( 14) of these were attributed to the Machine domain of the human factors subsystem.These included substantial discussions about how algorithms were developed.Developers wanted to use evidence based guidelines to augment bed restlessness algorithms; however there is a lack of these guidelines to inform the team how to implement in this type of early warning system.Clinicians had difficulty evaluating the data because of the lack of understanding of underlying features and data elements used to set up displays, such as motion density maps, which have not been used in clinical care.Clinicians needed extensive explanation about conversion steps for tables in the underlying algorithm, so that the interface design elements were more intuitive, which resulted in greater efficiency and effectiveness of clinical evaluations.Finally, a great deal of discussion occurred about the relevancy of alerts which were generated for review.For instance, clinicians did not want to be alerted for positive changes like increasing normal heart rates.These alerts were confusing and clinicians were not sure what to do with the information.

DISCUSSION
Human factors experts attempt to understand barriers and facilitators so people can properly use technology.These experts also provide design improvements for user interactions with technology in specific environments.Evaluating effectiveness, efficiency, and satisfaction with healthcare technologies requires multidisciplinary collaboration between developers (engineers) and healthcare providers (clinicians).
Collaborative efforts include (a) sharing expectancies for how the system will work for the end users, (b) exploring the use of the technology in actual, live settings to evaluate if expectations are being met for end users, and (c) iteratively reviewing solutions to concerns which are raised about inequalities between user's expectancies and proper use of technology.In the ongoing sensor study, the successes of our system and health outcomes for our residents depend on these collaborative efforts and are discussed below.

Sharing Expectancies
In human factors, perceptual and cognitive abilities of end users are critical to the safe design of healthcare technology.Human perception begins with the ability of a user to detect, identify, and recognize meaning of sensory input, while cognition requires greater cognitive effort for memorization, information processing, use of rules or strategies, hypothesizing, problem solving, learning and judgment [23].The collaborative nature of the Eldertech team provided a mechanism to share common and not so common perceptions and cognitive issues between engineers and clinicians as the early illness warning system was developed and evaluated simultaneously.Perceptions of the interface were enhanced as engineers and clinicians began to understand each discipline's language, which was necessary to create meaningful data architectures supporting the system.For example, engineers are not familiar with medical language like clinicians are, and using medical words on the interface like bradycardia and tachycardia were not as intuitive to them.In order to customize algorithms to meet the needs of clinicians, these common healthcare terms needed to be understood by all members.Fortunately, members of the current team have been working together for 2-5 years, and have already learned to communicate across their disciplines.New teams may find that an effective sharing process would take much longer to establish.
Cognitive abilities are also crucial to clinical technology design which can be effectively used by healthcare clinicians.The development of our early warning system required the use of different features (e.g., motion density, time in bed, restlessness, pulse rate, and bed breathing rate).Algorithms were developed to extract the features from the sensor data and to identify outliers that might correspond to changes in a resident's activity patterns as detected by the sensors.The use of such algorithms in healthcare technology is not a competency required for most practicing clinicians to perform their jobs.However, novel information systems employ automated reasoning systems to provide enhanced functionality to support better predictive and diagnostic support in technology used by clinicians for clinical decision making.The mutual benefit of our collaborative efforts was that engineers who designed sensor system architectures provided invaluable insights into how the selection of features can be used to discriminate changes in resident activity patterns, while clinicians were able to provide engineers an evidence base in which features could be selected based upon expert clinical experience and use of relevant clinical guidelines when available.

Evaluating Expectancies
Designing information systems that clash with human expectancies can lead to greater error when users interact with the technology and can lead to unintended consequences which may be more harmful to patients, result in poorer outcomes of care, and reduce effectiveness of the healthcare providers care delivery.As with any new technology, there is an element of not knowing what will happen.This is certainly true with our early illness warning system because these are novel technologies which have not been used in any clinical settings before.Uncertainty of not knowing what will happen can be managed well with collaborative efforts which focus on matching human expectancies with the conceptual models that are used to design technology.For instance, engineers who designed our system were very familiar with how certain attributes of the clinical alerts, such as frequency, would be affected by including more features into the algorithm, changing the size of standard deviations by 2, 3, or 4 times the daily average, and increasing the number of sliding window days for viewing data when an alert was initiated (see Table 1).However, clinicians were more reluctant to make these types of changes because they knew that one change could result in an overwhelming number of alerts being issued, increase the output needing to be evaluated for each interaction, and could drastically affect their workload.

360
Evolution of an Early Illness Warning System to Monitor Frail Elders in Independent Living Clinicians as a whole entered into the profession of healthcare with a decree to help their patients' live healthier lives; unfortunately, when technologies do not support this effort by requiring greater vigilance to monitor computer output, such as, addressing increased alert activity, clinicians will oftentimes neglect the computer in order to support the well being of the patient.The Eldertech team members engaged in collaborative efforts between engineers and clinicians to critique potential effects of algorithm changes on clinician workload, to test the functionality of the system including monitoring the frequency of warnings being produced by the algorithms, and to examine the usefulness of different parts of the technologies (e.g., sensors) for evaluating resident activity patterns.These ad hoc meetings provided a mechanism for engineers to explain underlying algorithms used to create the decision support system, to evaluate the output, and to listen to clinicians' concerns about the data.Furthermore, the clinicians were able to discuss their design concerns in an open forum where they could visualize and inspect iteratively how their concerns were being addressed, which positively impacted overall system performance measures.

Reviewing Expectancies Iteratively
Performance shaping factors, which enhance human computer interactions, include the amount of training a user receives in order to understand how interfaces function, amount and frequency of tasks to be performed for job functions, or environmental noise which can interfere with output.The capability of the sensor system to perform as expected by clinicians and engineers depended on collaborative input about system performance in the form of iterative reviews by both disciplines.
Ongoing iterative reviews provided enhanced situational awareness for how the sensor interface was functioning and how alert mechanisms were configured to issue warnings with ongoing weekly updates being performed by IT support personnel.Iterative reviews were important to enable clinicians to determine the relevancy of the sensor outputs to information received in the form of clinical alerts to changes in functional activity, which is a form of situational awareness.For clinicians, enhanced situational awareness is very important in dynamic environments where situations are always changing.This is especially true in an independent living facility like TigerPlace because most of the residents are older and frail, have complex healthcare problems with multiple comorbidities, and are at greater risk for declining functional levels.In environments where clinicians are overwhelmed with data, clinician workload is too high, and the environment has much noise (e.g., sensors or alerts with little relevance to activity pattern), unintended consequences such as inadequate supervision of residents, greater fatigue and burnout can occur.The Eldertech team used iterative review cycles where clinicians and engineers consulted with each other about the current state of functionality and use of information systems.This provided a means to address performance shaping factors that enhanced efficiency and user satisfaction with the early illness warning system.

CONCLUSION
Human factors evaluation has an important place in the development and implementation of clinical information systems.The evaluation must be conducted in an iterative fashion where concerns are attended to by all stakeholders who have interactions with the technology including clinicians who use the system and engineers who develop the system.Total participation by all users enhances the performance shaping factors that will ensure better outcomes in the form of a system which is more effective at completing the job it was designed to do, users who can use the system more efficiently, and a higher degree of user satisfaction.

Figure 2 .
Figure 2. Integrated sensor network.(Y-axis indicates the number of sensor firings over 24 hours in the apartment for each sensor type.)

Figure 4 .
Figure 4.The bed breathing rate interface for an actual TigerPlace resident showing an alarming condition.Each histogram represents one day.An email alert was sent to clinicians on 7-14-10 when the rate of breathing decreased after being normal for the previous week.(Y-axis indicates the total number of sensor firings for 24 hours for bed breathing rates.)

Table 1 . Iterative review and development of the early illness warning system. (Continued) Human Disciplines Concerns about factors Clinicians & early warning Human factors subsystems Engineers Engineers Clinicians system Solutions outcomes reveals
the singular (Clinicians or Engineers) and combined concerns (Clinicians & Engineers)