Qualitative Assessment of the Feasibility, Usability, and Acceptability of a Mobile Client Data App for Community-Based Maternal, Neonatal, and Child Care in Rural Ghana

Mobile phone applications may enhance the delivery of critical health services and the accuracy of health service data. Yet, the opinions and experiences of frontline health workers on using mobile apps to track pregnant and recently delivered women are underreported. This evaluation qualitatively assessed the feasibility, usability, and acceptability of a mobile Client Data App for maternal, neonatal, and child client data management by community health nurses (CHNs) in rural Ghana. The mobile app enabled CHNs to enter, summarize, and query client data. It also sent visit reminders for clients and provided a mechanism to report level of care to district officers. Fourteen interviews and two focus groups with CHNs, midwives, and district health officers were conducted, coded, and thematically analyzed. Results indicated that the app was easily integrated into care, improved CHN productivity, and was acceptable due to its capacity to facilitate client follow-up, data reporting, and decision-making. However, the feasibility and usability of the app were hindered by high client volumes, staff shortages, and software and device challenges. Successful integration of mobile client data apps for frontline health workers in rural and resource-poor settings requires real-time monitoring, program investments, and targeted changes in human resources.


Introduction
An estimated 66% of maternal deaths and 50% of deaths in children under the age of 5 occurred in sub-Saharan Africa in 2015 [1,2]. Although Ghana's health indicators largely surpass many African nations, the country has faced significant challenges in improving maternal and child survival. Despite nearly halving the number of maternal deaths since 1990, approximately 319 women die each year out of 100,000 live births from pregnancy-and birth-related complications-a figure nearly twice that of the Millennium Development Goal [1,3]. Among children under five, while overall deaths have declined by 51% since 1990, neonatal deaths occurring within 28 days after birth have largely remained stagnant, changing only from 30 to 29 deaths per 1,000 live births [2,4].
A vast majority of maternal, neonatal, and child deaths are preventable [1,[5][6][7][8]. However, limited access and utilization of skilled antenatal, delivery, and postpartum care services hinder the timely and safe delivery of life-saving interventions [6][7][8][9]. In addition, reliable and comprehensive data on the implementation of maternal, neonatal, and child health services are needed to monitor and ensure that lifesaving services are provided across the care continuum [10][11][12]. However, the inability of traditional health information systems to track new mothers and babies once they leave health facilities as well as inadequate communication between health workers at different levels of the health system has led to slower progress towards maternal, neonatal, and child health goals [13][14][15].
Global penetration of mobile phones has the potential to optimize delivery of high quality maternal, neonatal, and child health information and access to services [16][17][18][19][20][21]. In particular, mobile apps, or software applications developed specifically for use on small, wireless computing devices such as smartphones or tablets, have been shown to improve the quality of care provided to pregnant and recently delivered women through electronic decision support [17,18,22], health worker planning [23,24], and data collection and reporting [19,[25][26][27]. The reported ease of use and adaptability of mobile apps (or applications) for diverse populations also makes them an attractive, low-cost platform for developing health education and clinical care strategies [28][29][30]. Nonetheless, to increase efficacy and sustainability of mobile health applications, previous studies have underscored the importance of ensuring high user acceptance and usability of the technological innovation [31][32][33]. User-centered design and robust needs assessments are recognized as best practices for development technology [34]. Equipping frontline health workers with mobile phone applications may enhance the delivery of maternal, neonatal, and child health services as well as the accuracy of data capture and recording [16,26,35,36]. Yet, little research exists on health worker perceptions of mobile applications intended to improve maternal, neonatal, and child data management, despite user views being critical to product uptake and implementation [37][38][39]. In addition, the opinions and experiences of frontline health workers in sub-Saharan Africa on using mobile applications to track pregnant women and mother-infant pairs are underreported. Such information is critical for improving the long-term impact and sustainability of mobile health (mHealth) strategies for clinical and community health settings.
Evaluation Objective. This evaluation aimed to qualitatively examine the feasibility, usability, and acceptability of the Client Data Application (or Client Data App) that was part of the Mobile Technology for Community Health (MOTECH) program to support frontline health workers in the delivery of maternal, neonatal, and child care in rural Ghana.
The MOTECH program was initiated in 2009 as a partnership between the Ghana Health Service, Grameen Foundation, and Columbia University Mailman School of Public Health to leverage mobile technology to increase quantity and quality of maternal and infant care in rural areas and ultimately improve health outcomes. The Client Data App enabled community health nurses (CHNs) to use mobile phones to digitize care provided to women and children and thereby track pregnant women and mother-infant pairs needing care (Figure 1). CHNs then received reminders and alerts about clients' upcoming and missed care. Parallel care reminders and alerts, in addition to other actionable health information, were sent to pregnant women and mothers with infants less than 12 months of age as part of a mutually supportive "Mobile Midwife" program within MOTECH [40]. A separate evaluation, not discussed in this manuscript, is planned to examine health workers' views on MOTECH's Mobile Midwife service.  [4,40]. This region additionally has significant human resource shortages. In the Central Region of Ghana, there are over 25,000 individuals to one doctor compared to the nationwide ratio of nearly 12,000 individuals to one doctor [41,42]. The region also suffers from high vacancy rates of nurse-midwives trained to manage basic and emergency obstetric care [43][44][45]. Under these circumstances, mobile technologies that expand the reach of health workers are critical to overcoming barriers to care. Although telecommunication connectivity in the Central Region can be unreliable and mobile phones are not ubiquitous, phone ownership and access are high among health workers [46].
Maternal, neonatal, and child health services within Ghana's health system are primarily delivered via public health centers and community health posts. The health centers are staffed primarily by skilled health professionals (such as nurses and midwives) who offer comprehensive preventive and basic curative services, including minor surgeries and uncomplicated deliveries. In contrast, as part of Ghana's Community-based Health Planning and Services (CHPS) Initiative, community health posts are staffed by lowerskilled CHNs who provide health education, outreach and counseling, and basic curative services to clients via home visits and facility-based care [47]. Community health posts (referred to locally as CHPS facilities) are typically staffed by 2 to 3 CHNs who are required to have completed a twoyear postsecondary certificate program in obstetrics as well as general and community health nursing. MOTECH Ghana was originally designed for use in community health posts and later extended to the health centers, hospitals, and other private health facilities. Within the two evaluation districts, MOTECH was implemented in a total of 46 facilities, including 35 community health posts and 11 health centers.

Description of the Client Data App. The mobile Client
Data App used in this evaluation was delivered by low-cost GSM mobile Nokia 1680 and Nokia Asha 200 feature phones provided by Ghana Health Service, which helped CHNs and other users to digitize and track care delivered to motherinfant pairs in their area. The system's architecture was based on field-tested open-source software, including OpenXData for mobile data collection and OpenMRS for electronic medical records [48]. The client data system used a Java 2 Platform Micro Edition (J2ME) application to capture and store client data. All clients were assigned a unique MOTECH identification (ID) number upon registration to protect confidentiality and enable tracking across multiple facilities. During client encounters, CHNs first recorded care provided using five "simplified paper registers," which were developed by MOTECH to condense more than a dozen registers and streamline data collection. CHNs later entered data into digital forms on their mobile phones (Figure 2(a)). General packet radio service (GPRS) data channels were used to transfer these data from the phone to a central clinical data system that was stored on the MOTECH server ( Figure 2(b)). The data app system then crosschecked uploaded clinical information on timing and type of care given with national guidelines to estimate specific due dates for routine care. As a result, health workers received a weekly list via short message service (SMS) of pregnant clients and mother-infant pairs in their catchment area who were either due for or defaulted on care. CHNs were also able to query client data, enabling them to retrieve lists of defaulters or women due to deliver in the upcoming week, and to search for details about individual clients (Figure 2(c)). In addition, the Client Data App generated preselected monthly health reports that were required for national reporting, if client data were at least 85% complete and accurate three consecutive months. Previously, monthly health reports were numerous, redundant, and compiled by hand. Therefore, the Client Data App was intended to improve accuracy and processing speed.
The MOTECH developers designed the Client Data App for low-skilled health providers in rural and resourcepoor settings. To account for anticipated power and mobile network breaks, the Java-enabled Nokia handsets allowed for mobile forms to be completed and stored offline for uploading at a later time. Phones had dual subscriber identity module (SIM) capacity and were equipped with SIM cards from two different mobile operators in case of network or congestion problems [46]. Field testing during the prototyping stage served to align the app's features with user needs, including the simplification of data entry using check boxes, radio buttons, lists, and number fields. CHNs received in-person training as well as a detailed training manual. They could also refer to a MOTECH call center for technical assistance. Monthly prepaid airtime units were provided to all users to upload information. The Client Data App's interface was available for use in English. Phones were password-protected, and user authentication schemes were built into Java forms to maintain confidentiality of client data [46].

Measurement.
Qualitative in-depth interviews and focus groups were used to examine health worker perceptions on the Client Data App's feasibility, usability, and acceptability. For purposes of this evaluation, we defined the assessment areas as follows: Feasibility was defined as whether implementation of the Client Data App was easily and conveniently done, accounting for advantages and disadvantages to integrating the application into routine workflow. Usability was defined as whether the Client Data App could be used by CHNs to adequately record, track, and summarize data, including whether it functioned in a way that enhanced productivity or led to unproductive tasks due to errors. Acceptability was defined as whether CHNs and other stakeholders found the Client Data App likeable, including its interface and navigation features. These definitions were derived from similar prior research that qualitatively assessed user experiences for mHealth applications [28,[49][50][51][52][53][54][55].

Data Collection.
Data were collected at three levels of the health system: community health posts (referred to locally as CHPS facilities), health centers, and district health directorates. Semistructured interview guides were used at all levels. In-depth interviews with CHNs and midwives asked them to describe perceived benefits and drawbacks of the Client Data App, as well as their experience using it during clinic and community outreach activities. CHNs and midwives were also asked to assess advantages and disadvantages of using the Client Data App for tasks such as recording care, tracking clients, and verifying data with automatic health reports. Skilled nurses working at the health centers did not use the Client Data App and therefore were not recruited for data collection. Interviews with district health directors and district health information officers explored how the Client Data App affected the quality of data provided by CHNs and their ability to use and supervise submission of monthly health reports. Questions also examined views regarding local and scaled-up implementation. Other district health directorate staff, including disease control officers and public health officers, who did not engage with the Client Data App were not interviewed. Focus groups with CHNs were included to further investigate findings from the individual interviews and to obtain CHNs' recommendations for International Journal of Telemedicine and Applications 5 modifying the MOTECH data management system. Focus groups with midwives are not conducted given the limited number of midwives available at participating sites.
All interviews and focus groups were conducted in English by a local Ghanaian and a US researcher, both with experience conducting qualitative research. The Ghanaian researcher occasionally translated local terminology in Akan used by participants. The interviews were conducted at community health posts and health centers and ranged from 40 to 120 minutes. Focus groups were conducted in a centrally located and neutral space. The discussions and lasted approximately 90 minutes. Each interview and focus group was digitally recorded and transcribed verbatim.

Sample Selection.
One health center and three community health posts were randomly selected from each of the two participating districts. Purposive sampling was then used to identify CHNs, midwives, district health information officers, and district health directors with a minimum of six months experience using MOTECH's Client Data App. The target interview sample size was 14 individuals, representing one CHN and one midwife per health center, one CHN each from three community health posts in each district, and the district health director and district health information officer in each district. The target focus group sample size was two groups each with 7 to 8 CHNs. Given resources available, this sample size was expected to enable the evaluation to reach saturation in which no new findings emerged [56].

Analysis.
A qualitative thematic analysis was conducted by two public health graduate students. Interview and focus group transcripts were manually coded using a priori topical codes according to the evaluation's three assessment areas: feasibility, usability, and acceptability. Emergent subcodes were then developed based on patterns within each concept and which were relevant to the literature. We then followed an iterative process of developing a codebook, identifying salient themes, and integrating core findings [57]. When new themes were identified throughout this process, transcripts were reanalyzed to find evidence that verified or modified those themes. Later-stage interviews and focus groups were used to validate responses among member participants [58]. We also confirmed findings based on feedback from MOTECH implementation partners during various stages of data collection and analysis [59].

Ethics
Approval. The evaluation was approved by the institutional review board at the Johns Hopkins Bloomberg School of Public Health in Baltimore, Maryland, USA. As part of the MOTECH implementation plan, the Ghana Health Service in Accra, Ghana, also approved the evaluation's activities. All participants provided oral informed consent prior to data collection. This manuscript complies with the mHealth evidence reporting and assessment (mERA) checklist on reporting of health interventions using mobile technologies [60].

Results
Several findings emerged from our analyses, revealing both the achievements and challenges of MOTECH's implementation of the Client Data App. The emergent themes within each of the three assessment areas (feasibility, usability, and acceptability) are illustrated below with exemplary quotations. Key findings are summarized in Figure 3.

Participant Characteristics.
A total of 14 individual interviews were conducted, representing eight CHNs and two midwives as first-line users of the Client Data App, as well as two district health information officers and two district health directors (Table 1). Two focus groups were additionally carried out, consisting of a total of 15 CHNs. The majority of participants were female and within the ages of 26 to 30 years.

Feasibility.
We defined feasibility as whether implementation of the Client Data App was easily and conveniently achieved, accounting for advantages and disadvantages to integrating the application into routine care.

Easily Integrated into Routine
Care. Participants generally reported that the Client Data App was easily integrated into their workflow and data capture. CHNs were able to incorporate registration of eligible individuals, including issuing their MOTECH IDs, into their client visits. CHNs also noted being able to readily incorporate information provided by MOTECH alerts into their weekly routines. Based on the ease of integration, users were able to envision the Client Data App becoming a permanent part of their work.

Hardware and Uploading Requirements Posed Some
Inconveniences. However, limited mobile network connectivity in more rural areas of the district posed significant challenges to uploading client data. Consequently, CHNs had to preoccupy themselves with locating places with adequate connectivity in order to upload the mobile forms. This presented an added inconvenience, particularly for those CHNs  working in remote areas. For some CHNs, accommodating network connectivity challenges also required them to extend their workday.
We also have problems with network connectivity. 3.4. Acceptability. Acceptability was defined as whether CHNs and other stakeholders found the Client Data App likeable, including its interface and navigation features.

Praised for Providing Novel Job Aids. The Client Data
App was acceptable to users given that it introduced new tools to assist health workers in performing their assigned tasks. In general, CHNs were satisfied with MOTECH's alert system. In particular, CHNs and those at community health posts found that the automated monthly reports eased their workload for data reporting and motivated them to capture and use data.

Discussion
Our data indicate that MOTECH's Client Data App was a feasible, usable, and acceptable tool to aid health workers in collecting and tracking data to improve maternal, neonatal, and child health services. Health workers agreed that the Client Data App simplified individual client data collection, was easily integrated into their workflow, and enhanced their capacity to deliver follow-up services across the care continuum. These findings align with other studies that have demonstrated the benefits of equipping health workers with data management technologies to improve the continuity of care in rural populations [27,61]. At the same time, our results highlight several challenges that would need to be addressed to optimize the utility of a client data management system using mobile devices in resource-poor settings. First, our findings suggest that the successful integration of mobile applications into service delivery may necessitate targeted changes in human resources available at certain health facilities [62]. The greatest obstacle to implementation of the Client Data App was the combined effect of high client volumes, staff shortages, and poor network connectivity. CHNs tended to integrate mobile data entry into client care by waiting until the end of the day or week to upload data to the MOTECH server. Yet, the number of clients seen by health workers at the health centers and the long delays in uploading data precluded real-time submission. Thus, while data uploads were manageable for CHNs working at the community health posts, they were generally not manageable at health centers. Similar issues emerged within the context of a telemedicine program in Ghana's Amansi-West district, in which providers assumed a greater workload without a reduction in other tasks or increased personnel [63]. Our participants suggested training less skilled data staff or community health volunteers to assist with data entry, particularly for older providers who were sometimes less proficient with mobile apps [64]. In the context of future mHealth deployments, it will be important for implementing partners to work closely with the national health system and other government agencies to assess health workers' capacity to absorb additional mobile-based responsibilities and to explore possibilities for hiring and maintaining data staff.
Second, given the current cellular network in rural Ghana, it may have been possible to minimize network challenges through a more advanced monitoring system. Our results revealed that the lack of network reliability compromised the efficiency and usability of the Client Data App, notwithstanding the technological systems put in place to address these complications. Other studies have shown that poor network coverage and signal strength are often a major limitation to implementation of mobile technologies [23,27,31,65,66]. While the MOTECH team took careful measures to mitigate connectivity challenges, such as instituting a customer support call center and using high-storage devices with offline and multinetwork functionality, users still grew frustrated with the efforts required to upload data. The longs lags between data collection and submission, which CHNs reported as sometimes taking up to two weeks, limited the capacity of MOTECH to reliably track all client needs. A mechanism that allowed health workers to directly access clinical data to fix errors from incomplete uploads, rather than relying on a customer support center, may have enabled more immediate troubleshooting [40]. In addition, it is critical for implementers to carefully select what minimal clinical data are required for decision-making, rather than digitizing all paper registers or entry fields in order to maximize entry quality and speed.
Third, our findings shed light on the interconnectedness of the evaluation's assessment areas. Acceptance of the Client Data App was closely linked to its perceived usefulness to generate automated reports. When no problems were encountered during uploading, CHNs felt that the automated reports were a time saver. The MOTECH team had anticipated this would be the primary incentive for CHNs to engage with the Client Data App [46]. However, several health center-based CHNs were less motivated to enter data when the reports were not generated. These findings align with other research studies that have found technical issues such as screen freezes and delayed uploads impede efficient use and limit user uptake of mobile health applications [27,55,67]. A lower threshold of data completeness and accuracy for receiving automated reports may have improved usability. Similarly, engaging dedicated data staff to manage the Client Data App may have enhanced its usefulness and acceptance by increasing the number of facilities with minimum completeness standards and automated reports. Other strategies such as rewards and recognition may also have encouraged user uptake in the short term, yet their sustainability is questionable.
Studies conducted with frontline health workers prior to the integration of new technologies into their workflow have revealed high levels of acceptance and willingness to learn, despite the lack of experience with such tools [64,68,69]. In many cases, such interest is related to the appeal of innovative technologies as representative of modern medicine [64,67]. Thus, efforts should be taken to capitalize on and preserve existing positive attitudes during the implementation of mobile health applications by ensuring the usability of new devices. Specifically, incurring higher upfront costs for more advanced mobile handsets may improve data accuracy and user adoption as compared to inexpensive, less user-friendly devices. MOTECH selected the Nokia 1680 model, in part, due to its lower cost (40 USD) [46] for use in Awutu Senya. Yet, as our participants concluded, the small-sized keypad was more problematic than anticipated. In later deployments of MOTECH in Gomoa West as well as the other regions not explored in this evaluation, the phone was upgraded to a more expensive, user-friendly Nokia Asha 200 [40]. The evolution of mobile phones used by MOTECH highlights the value of continually assessing user experiences and reevaluating the trade-offs inherent in each decision. As the participants noted, using larger tablets or laptops in the future may further improve client data management at facility and community levels.
Finally, greater global focus has been placed on ensuring the interoperability of mobile and digital health innovations [38]. The proliferation of multiple disconnected mHealth systems in many countries has led to calls to examine how systems operating in the same country can exchange information across multiple platforms. Systems like Open-HIE (https://ohie.org/) and similar "shared health record" alliances are being explored as a mechanism to allow smaller mHealth products to share connectivity to a central backbone of core information. In our evaluation, participants identified a lack of interoperability with the national DHIMS-2 as an important barrier to maximizing utility. Although often more expensive and time-consuming during development, the future benefits of building mobile applications to global health information standards quickly materialize with increased scale, through interoperability with other facility and national aggregate systems.
Limitations and Strengths. This evaluation was limited by the following factors. First, self-reported data is subject to social desirability bias, and thus participants may have exaggerated their positive reactions to the Client Data App. However, this bias may have been minimized by the fact that the interviewers were external consultants and not affiliated with MOTECH. Second, the transferability of the findings to other MOTECH districts may be limited by the unique situation of the Awutu Senya site. For example, the district health director from Awutu Senya was highly dedicated to and involved in project implementation, which may not be true in other settings. Nevertheless, by engaging with diverse user groups, this evaluation provided rich insights into the Client Data App's implementation challenges from multiple perspectives. The use of two types of qualitative methods further strengthened the credibility of the findings, as laterstage interviews and the focus groups were able to provide feedback on early analyses.

Conclusion
Mobile phones hold great promise for overcoming health disparities among rural populations by bridging the gap between access to client health information and service provision. MOTECH's Client Data App is a promising tool to aid health workers in collecting and tracking data across the health care continuum. Results of this evaluation may be used to guide future research on mHealth innovations to address challenges related to infrastructure, human resources, and technology before and during program deployment. Qualitative assessments of user perceptions should remain a priority in efforts to optimize the use of mobile data applications to alleviate barriers to maternal, neonatal, and child care in Ghana and beyond.