Software bugs discovered by end-users are inevitable consequences of a vendor’s lack of testing. While they frequently result in costly system failures, one way to detect and prevent them is to engage the customer in acceptance testing during the release process. Yet, there is a considerable lack of empirical studies examining release management from end-users’ perspective. To address this gap, we propose and empirically test a release framework that positions the customer release manager in the center of the release process. Using a participatory action research strategy, a twenty-seven-month study was conducted to evaluate and improve the effectiveness of the framework through seven major and 39 minor releases.
This paper extends prior research on customer-driven release models [
A considerable portion of software bugs contributing to the largest system failures are discovered by customers in their production environments [
Common service level agreements between the vendor and the users have provisions for resolution of software bugs; however, these bugs are more likely to require several fixes. Reference [
A proven cost effective method to reduce software bugs and improve service delivery is through the adoption of release management framework (Table
Release Activity and Corresponding Articles.
Release Management Activity | Article and Description |
---|---|
Release Policy | [ |
Release Planning | [ |
Design and Develop Software | Variety of methods; handled by software or IT service vendor. Also see release acceptance [ |
Build and Configure Release | Variety of methods; handled by software vendor. |
Fit-for-Purpose Testing | [ |
Release Acceptance | [ |
Roll-out Planning | [ |
Communication, Preparation and Training | [ |
Distribution and Installation | [ |
Empirical studies examining the release management process from an end-users’ perspective are lacking. To address this gap, this paper presents a case study of the release management process from the customer’s point of view. The goal of the study is to develop a method for better integrating end-user perspectives in the release process. The main research question is how can organizations improve their release management process while minimizing the amount of software bugs in the production systems? Based on the study’s results, a framework is proposed to accomplish this goal and to improve the overall release management process, minimizing the occurrence of software bugs. The framework is unbounded by industry specific models and instead combines the traditional ITIL release management methodology with the project management methodology developed by the Project Management Institute (PMI). To our knowledge, this is the first empirical study to propose specific details of the customer release manager (CRM) role in each of the release management processes.
The balance of this paper will first review relevant literature in the area of release management, with a particular focus on each release component identified in the ITIL framework and articles addressing challenges in each activity. The research methods employed will then be detailed, followed by a review of the results. The proposed framework for incorporating the customer’s perspective in the release management is presented with a description of the CRM’s role for each process. The paper concludes with an overview of the study, limitations, and recommendations for future research.
Release management, a process associated with the delivery of high quality software to users [
The ITIL release management involves determining, acquiring, releasing, and deploying changes (such as bug fixes and enhancements) in an IT environment. The following table outlines the key release management activities according to the ITIL framework [
Release policy, the first activity of the release management process, represents a formal agreement on the strategic approach and includes (1) infrastructure used in the release; (2) acceptable schedule; (3) definition of major versus minor releases; (4) deliverables for each release; (5) roll-out and back-out plans; (6) documentation of releases; and (7) roles, responsibilities, escalation steps, contacts for vendors, and end-users [
This activity is concerned with the designation of resources, the roles and responsibilities of these resources, agreement on policies and procedures to be used during the release, decision on deliverables, and features [
This activity involves the process of designing the software based on requirements and developing the software included in the release. End-users engage with the software developers in writing the acceptance test cases as defined in the release acceptance activity [
This activity includes the compilation of modules stored in the software library to create the derived objects from the source objects. Reference [
This activity includes functional, operational, performance, and integration testing of the release. Lack of resources (including testers and test environments) impedes testing, resulting in releases with software bugs [
This activity includes testing software by the end-users and obtaining approval for the release to proceed. During this activity, the release package is deployed to the customer’s test environments with the coordination of the customer’s technical team. Typically, testing is performed by end-users, and the release acceptance is based on specific conditions which the released package must satisfy [
This activity involves the creation of a time table, resources, responsibilities, and events during the distribution and installation of the release. Reference [
This activity, involving notifications to release stakeholders (including software development, release, and end-user teams), roll-out meetings, and training sessions, heavily involves the end-users in preparation of the release deployment. Clearly defined procedures are necessary to keep end-users informed of the upcoming release changes, and trainings must be organized before the release delivery [
This activity concerns the deployment of the final changes into the live environments. Since only 9% of companies use a proper release automation tool [
The release manager role is not defined in the ITIL framework, yet it plays a key figure in the software release management process. The release manager is expected to possess all the qualifications of experienced project managers such as judgment, community building, attention to detail, and communications and management skills. While 60% of a release manager’s time is spent on release planning, 30% of the remaining time is spent on replanning activities due to changes in requirements and stakeholder priorities [
The ITIL framework, as implemented, does not fully address the issues that could cause software failure, because that framework is essentially focused on release management from the vendor’s perspective. Our position is that expanding the ITIL framework to include the customer perspective in the form of a customer release manager (CRM) function would reduce those errors.
Based on the literature review, a customer-driven release management framework is developed that combines the ITIL’s best practices framework for releases with the PMI process group methodology for project delivery. The framework considers each ITIL release management area as an unique subproject with incorporated PMI’s structured process groups and respective tasks. Figure
Release activity and project process group integration.
This new framework aims to address the common release issues associated with communication, coordination of vendor, and customer involvement in the release management activities.
Releases can be considered to be projects, as they are temporary endeavors with predefined start and finish dates [
The following section describes the role of the CRM in the framework’s release and process group activities.
During the release policy activity, the CRM should formulate and document optimal acceptable release criteria for the customer based on discussions with internal stakeholders. These criteria should be included in the contract and negotiated with the software vendor at the time of software procurement. The criteria should include triggers for each release (e.g., change requests, entry and exit conditions for each activity, staff skills, service and operations level agreements, UAT criteria, release design option such as phased approach, or push or pull of new software [
The release planning activity should also occur on the customer’s end. It should include a comprehensive planning effort on behalf of the release manager who should address the scope, time frame, possible costs, resources which will test the release, quality control, risks, communications strategy, and procurement of required hardware and software. The planning effort should be coordinated with the vendor’s release manager to confirm that planning estimates coincide with the vendor’s resource availability and that all stakeholders’ expectations are managed.
During the design and developing software activity, the CRM should coordinate the development of user acceptance test cases between the end-users and the vendor’s software team. This proactive effort will ensure that the vendor discovers issues during the fit-for-purpose testing activity and will not delay the release acceptance activity.
During the build and configuration activity, the CRM should communicate with the vendor release manager to address any issues associated with versioning of the release, security, and access rights that could jeopardize the release process.
During the fit-for-purpose testing activity, the CRM should organize a preliminary end-user testing with the vendor release manager into the vendor’s test environment. Reference [
During the release acceptance activity, the CRM should ensure that the new build is deployed to the customer’s test environment and should communicate with the UAT team to initiate the testing process. He or she will also report any bugs discovered to the vendor release manager and ensure that fixes are provided by the vendor and retested by the UAT team. During this activity, the CRM would adjust the release plan based on release progress, document, and enforce any changes that are reported.
During the roll-out planning activity, the CRM should prepare a detailed roll-out checklist that outlines key tasks, start and end times, key resources and their responsibilities, system backups, and back-out procedures should the installation fail. Testing of the release in production should also be planned at this time. The checklist should be coordinated with the vendor release manager to include vendor resources.
During the coordination, communication, and training activity, a decision on go or no go for the release should be made by all internal and external stakeholders. The CRM should direct end-users during the training sessions and host several trial runs of the upcoming installation with the roll-out team.
Finally, during the distribution and installation activity, the CRM and the vendor release manager should coordinate the delivery of the release. The CRM should monitor the installation, the final production testing of the release by key end-users, and make a decision if the release should be stopped based on the reported production bugs.
The proposed release management framework provides a flexible and innovative methodology that could be utilized by many individuals, departments, and vendors for the effective release of software. Moreover, a definition of the CRM role within each release process group from the proposed framework is provided where the CRM is appointed to coordinate the release process on the customer’s end. The CRM should be skilled in the PMI’s project management methodology and possess sufficient technical skills to be able to execute certain engineering and database operations (such as rolling out application installers and performing database backups and restorations for roll-back purposes). The CRM should get involved in the formulation of the customer release team and work with the vendor release manager to facilitate the planning, delivery, testing, and installation of the final software packages. He or she would accomplish this by applying the project management process groups through the release management activities.
From the very inception of each release, the CRM will embark on a thorough planning effort that integrates careful consideration of the enterprise’s environmental factors influencing the release’s success (e.g., organizational culture, end-user testing availability, IT hardware and software resource administration, and established organizational communication channels). Table
Proposed Customer Release Manager Activities and Process Group Mappings.
Release Activity | Project Management Process | ||||
---|---|---|---|---|---|
Initiating | Planning | Executing | Monitoring and Controlling | Closing | |
Release Policy | Determine optimal policy for the organization with appropriate stakeholders | Develop the release policy and add to procurement documentation | Negotiate policy with vendor at software procurement time. Enforce changes | Ensure policy is followed by vendor and document changes if necessary | Document and archive lessons learned |
|
|||||
Release Planning | Integrate lessons learned from prior releases. Review prioritized request for changes (RFC). Identify impacted end-users and address end-user enhancements needs | Plan release’s scope (RFC, enhancements), schedule, potential costs, UAT (use cases, quality control testing), resource roles, communication strategy, risks, hardware/software procurement needs, roll-out strategy, training, distribution, and installation strategy. Re-plan based on changes | Coordinate plans with the vendor’s release manager. Communicate plans with release stakeholders. Procure hardware or software. Enforce changes | Document changes based on vendor’s release manager’s feedback. Track and document any changes | Document and archive lessons learned |
|
|||||
Design and Develop the Software | Review the release scope, schedule, end-user resources plans, and integrate lessons learned on UAT cases from prior releases | Plan required use cases for the RFCs and enhancements with end-users and vendor release manager. Re-plan if there are new changes | Direct and manage the documentation effort of end-users. Coordinate progress and distribution of use cases with vendor’s release manager. Enforce changes | Verify all use cases have been completed and delivered. Track and document any changes | Document and archive lessons learned |
|
|||||
Build and Configure the Release | Integrate lessons learned on build and configuration from prior releases | Re-plan if there are changes | Address potential issues associated with the compilation and configuration with vendor release manager (version, access permissions, and security). Enforce changes | Track and document any changes | Document and archive lessons learned |
|
|||||
Fit-for-Purpose Testing | Integrate lessons learned on fit-for-purpose testing from prior releases | Plan the testing of a pilot. Re-plan if there are changes | Direct and manage the pilot testing. Enforce changes | Document and communicate issues and potential changes with end-users and vendor release manager | Document and archive lessons learned |
|
|||||
Release Acceptance | Integrate lessons learned on release acceptance from prior releases. Review release plans | Re-plan if there are changes | Direct and mange the deployment of release into test environment, the acceptance testing and communications of issues with vendor release manager, and the deployment of fixes to issues. Adjust release plans based on progress. Enforce changes | Document and track reported issues, changes, and report on progress with stakeholders | Document and archive lessons learned |
|
|||||
Roll-out Planning | Integrate lessons learned on roll-out from prior releases. Identify stakeholders involved in the roll-out (e.g. IT, key end-users, vendor resources) | Prepare distribution and installation time table checklist including back out procedures, production testing, each roll-out activity, and each participant’s role and responsibility. Re-plan if there are changes | Coordinate roll-out plan with vendor release manager. Enforce changes | Track and document any changes | Document and archive lessons learned |
|
|||||
Communication, Preparation, and Training | Integrate lessons learned from prior releases. Identify stakeholders who will perform the training | Review release plan for training, levels of support after the release installation. Re-plan if there are changes | Document any procedures, and communicate with end-users and vendor release manager to address any improvements. Direct and manage training. Make final decision on go/no go for release with vendor release manager and stakeholders. Manage trial runs for the installation. Enforce changes | Track and document any changes | Document and archive lessons learned |
|
|||||
Distribution and Installation | Integrate lessons learned on distribution and installation from prior releases | Re-plan if there are changes | Direct and manage the roll-out. Ensure backups are completed. Direct production testing and make decision of go-live versus stopping the release. Enforce changes | Track progress, document issues, and communicate with stakeholders | Document and archive lessons learned |
During the initiating process group, the CRM will determine release policy, define the initial scope of the release (including RFCs), schedule, costs, and identify both external stakeholders (e.g., vendor release manager and vendor resources) and internal stakeholders (organization’s end-users, managers, and IT resources). Authorization will be obtained to proceed with the release and release charter will be completed. This process group feeds from lessons learned in prior releases and requires a different review of archived documentation by CRM.
During the planning process, the overall release plan will be defined along with comprehensive plans for the tasks and end-user involvement during each of the remaining release processes (the design and development of the software, build and configuration of the release, fit-for-purpose testing, release acceptance, roll-out, communication, preparation, training, distribution, and installation of the release).
During the execution process, the release manager will coordinate, direct, and manage the work done by the end-user release team as well as the internal and external stakeholders (including vendor). He or she will also enforce any changes discovered during the monitoring and controlling process.
During the monitoring and controlling process, the release manager will track the progress of the release, monitor, and use corrective actions to realign the release back in line with the plan and document any changes. This process involves frequent communication with end-users to determine and inform users of pending changes.
During the closing process, the release manager will ensure that all deliverables have been released and lessons learned have been documented and archived.
To test the framework, a participatory action research (PAR) strategy was adopted [
Reference [
During the diagnosing stage, the researcher and participants identify an organizational problem and settle on a framework that will be used to address it. Next, the organization designates a team and the intervention as part of the planning stage. Research occurs during the action taking phase, followed by data collection, analysis, and summarization during the evaluating phase. In the final phase, specific lessons are documented for improvement during the following cycles. According to [
In this study, the testing of the framework followed the same PAR strategy and stages (Figure
Adopted participatory action research strategy.
The unit of analysis for this study was a two-year period of participation by one of the authors following the five phases of action research. The author, a certified PMI project management professional, was assigned as the leading project manager for the implementation of an ERP system at a large county in the southern quadrant of the United States between February 2010 and June 2012. The ERP system, a non-US developed system, was procured in 2007 with the goal of replacing an unsupported and rapidly aging system, which lacked vendor support and was entirely maintained by the county’s IT department. The goal of the new system was to improve the department’s business operations by delivering customized business workflows and streamlined data access for over 200 employees with fully dedicated support provided by the software vendor. The new system also included a new public web portal to handle citizen inquiries as well as an integrated voice response (IVR) system for permits and inspections status via phone and fax requests. It was heavily customized to fit the county’s business processes. Results for the success of the framework were determined by measuring the number of releases and release types, reported and resolved software bugs, number of system crashes, average release acceptance durations, and average installation durations prior to and after intervention of each cycle. The data was evaluated using tables, graphs, and charts.
The project implementation was divided into three phases. Phase 1, completed in 2008, included the delivery of the code enforcement module. Phase 2 included the planning and engineering modules and was delivered in 2009. Schedule delays and lack of acceptance testing during these phases resulted in the accumulation of a number of software bugs. The release management process was performed ad hoc with the vendor delivering the final module resulting in the customer accepting it into production without rigorous testing. The final phase, which included the permitting and inspections module, the public web portal, and the IVR system, was completed in 2011. This study covered the twenty-seven-month time frame required to implement the final phase and resolve accumulated software bugs from prior phases via the use of the ITIL release management process.
At the time of the study, the system had accumulated 45 major unresolved software bugs from the prior phases. Some of these errors caused six system crashes with significant user down time. Furthermore, critical functionality was missing on already delivered components. As a result, the project was halted until the vendor resolved the system performance issues and delivered the missing functionality. The author and the project team performed an analysis of the existing release management processes based on the ITIL framework and identified the following critical problem areas that were addressed in two action research cycles: (1) no existing release acceptance criteria; (2) inefficient customer roll-out planning process; (3) inefficient distribution and installation; (4) no existing release planning process in place; and (5) no existing release policy in place to address the process of release management.
Between March 2010 and June 2012, one of the researchers, the project team, and the software vendor worked to resolve 361 software bugs via seven major and 39 minor releases. Figure
Resolved bugs and corresponding releases (red = major releases; blue = minor release).
The following section identifies the action research cycles, identified problems, and the application of the framework to resolve these problems.
Release details.
Release date | Release type | Release version | Bugs discovered |
---|---|---|---|
3/23/2010 | Major | 3.3.3 | 17 |
6/8/2011 | Minor | 3.8.4.11 | 68 |
7/18/2011 | Minor | 3.8.4.12 | 77 |
8/20/2011 | Major | 3.8.8 | 62 |
8/29/2011 | Minor | 3.8.8 | 51 |
9/20/2011 | Minor | 3.8.9 | 68 |
1/6/2012 | Major | 3.8.11 | 26 |
uncover any patterns and recommend process refinements; propose improvements to the database backup logs and procedures; run performance checks on the application server to check for memory, CPU, and disk capacity issues with recommended configuration refinements; review primary processes with customer’s representative from each department to help identify bottlenecks in processes; and review application configuration to identify administrative errors and provide written documentation of all findings and recommendations.
who is involved from each team; what tasks are delegated to each participant; what are the contact phone numbers and email addresses for each person; what is the conference bridge number which members can use to join and remain on for the duration of the installation; what is the start and end time for each task on the release; who and what testing will be performed in the production system; and what is the roll-out procedure plan in case the installation fails?
Distribute the time table and host a preroll-out meeting with all stakeholders involved with the release to address questions and provide clarifications. (Integrating) Identify stakeholders involved in the roll-out (e.g. IT, key end-users, and vendor resources). Track and communicate changes and document lessons learned.
names of servers to be updated, dates, times, and responsible party for release notifications to the entire user pool; disabling of the production system; server snapshots; primary and secondary database backups; database restores; installations of the new release by vendor; UAT testing in production; desktop client roll-out to all users’ PC workstations; and certification of release completion.
During the preroll-out meeting, the CRM informed the stakeholders of their role in the release, including the fact that he would be calling out the completion of each task during the installation and letting each party know when to start their respective tasks. Next, the CRM called out each participant and asked them to read out their task. He solicited participants’ feedback about the task and addressed any inconsistencies or misunderstandings. At the end of the preroll-out meeting, each stakeholder was made aware of the expectations of his or her role, including start and end times and contact information for all participants and the CRM. The UAT team was also informed of the test cases to be executed in the production environment after the completion of installation.
Production system uptime.
Actual system downtime (minutes per month).
This appears to be the first study that proposes an integrated release management framework with a predefined CRM role. The paper presents many opportunities for future research on the release management process. For example, in release management, it would be valuable to examine the impact on software quality via use cases written by the customer team and used by the vendor software development team during the software coding process. This process was not investigated in this case but can prove valuable in order to understand not only how use cases impact software development but also how the interaction between the customer UAT team and the vendor’s software development team can work together better to produce higher quality software.
This research has begun the investigation of how the ITIL framework can be applied from the customer’s perspective within an organization. By viewing the impact of the release management process through the lens of the proposed integrated release management framework, it illustrates how release management can be accomplished electively through establishing coordination methods by the CRM with the vendor release manager and how problems are resolved through the centralization of the activities. Future research can examine the use of social media systems to capture lessons learned during releases [
Little research exists on the application of project management concepts for release management from the customer’s perspective. Frameworks like ITIL, COBIT, and ISO 20,000 examine software release from a single perspective, most commonly the vendor’s. Further research through a multicase study approach on the use of the proposed integrated release framework and the results achieved with it could answer questions on its advantages to other systems across a broad range of industries. This research will hopefully set the stage for investigating these and other important questions and for more effective and efficient release management processes.
The ITIL framework holds great promise for leveraging a structured method of delivery and acceptance of release; however, the lessons from this case study show that if the framework is only applied from the vendor’s perspective, the customer can incur a number of disadvantages. As a result, the proposed framework addresses the gaps that exist in literature to address the release management process from the customer’s perspective. Whether or not these lessons can be duplicated in other scenarios is an open question, as each organization offers unique challenges.
The authors declare that there is no conflict of interests regarding the publication of this paper.