The Impact of Standardized Order Sets on Quality and Financial Outcomes

The Impact of Standardized Order Sets on Quality and Financial Outcomes
David J. Ballard, MD, MSPH, PhD, Gerald Ogola, MS, MPH, Neil S. Fleming, PhD, Dave Heck, MD, Julie Gunderson, RN, BSN, MM, Raaj Mehta, Roger Khetan, MD, and Jeffrey D. Kerr, MD Institute for Health Care Research and Improvement, Baylor Health Care System, Dallas, TX (Dr. Ballard, Mr. Ogola, Dr. Fleming, Dr. Heck, Mr. Mehta); Baylor University Medical Center, Dallas, TX (Dr. Khetan); Office of Clinical Transformation, Baylor Health Care System, Dallas, TX (Dr. Kerr)
Address correspondence to: David J Ballard, MD, MSPH, PhD, Institute for Health Care Research and Improvement, 8080 North Central Expressway, Suite 500, Dallas, TX 75206; telephone: 214-265-3670; fax: 214-265-3640; e-mail: dj.ballard@https://www.wendangku.net/doc/b513788094.html,. advances-ballard_12
Objective: The objective of this project was to valuate impact of a standardize d order set on quality and financial performanc e.

Methods: We conducted an observatio nal study to examine order set use by hospital, discharge month, severity of illness and risk of mortality for pneumonia patients between March 2006 and September 2007. We also assessed impact on in-hospital mortality and 30-day readmissio n rates

using four measures: (1) Cox proportion al hazards regression, (2) Joint Commissio n Core Measures compliance using logistic regression, (3) length of stay, and (4) financial indicators using robust regression methods for highly skewed data. Results: A total of 3,301 patients met the inclusion

criteria. Over 19 months, order set use increased by 55 percent. Order set use significantl y improved in-hospital mortality [hazard ratio (95 percent confidence interval (CI): 0.66 (0.45; 0.97) or 0.67 (0.46; 0.98); and Core Measures compliance (relative risk, 95 percent CI: 1.24 (1.04; 1.48) or

1.22 (1.02; 1.45)] following covariate or propensity score risk adjustment. Conclusio n: Evidence-b ased pneumonia order sets can reduce inpatient mortality and increase delivery of important care processes.
Introduction
Baylor Health Care System (BHCS), an integrated health care delivery system located in North Texas, is engaged in a multiyear process and organizational redesign project that includes the implementation of an electronic health record (EHR) system supporting computerized physician order entry (CPOE) and point-of-care decision support. This process is intended to increase the overall standardization, quality, and efficiency of care.

As an intermediate step—partly to achieve some of the quality of care benefits associated with the standardization and streamlining of care offered by CPOE, and partly to familiarize physicians with the use of standardized orders—BHCS is developing system-wide standardized order sets to be made available through the physician intranet portal at all BHCS locations. Ultimately, these order sets will serve as the core library of order sets supporting the CPOE system.
Since its introduction in 2001, the intranet physician portal has provided secure access to patient health information from any location via the BHCS Network. Using the portal to disseminate order sets simplifies the process of applying updates universally in a timely manner and eliminates the need to provide printed copies at all physical locations. Additionally, this system introduces an intermediate level of computer use, which is intended to ease the transition from handwritten orders to CPOE. The Medical University of South Carolina pursued a similar strategy and reported success, both in attaining some CPOE-related benefits before implementing a full CPOE application and in achieving some of the cultural changes necessary for the successful implementation of CPOE.1
Previous research suggests that implementation of standardized order sets, templates, or protocols can improve compliance with recommended processes of care—such as early administration of aspirin, prescription of angiotensin converting enzyme inhibitors, and use of β-blockers for acute myocardial patients,2, 3, 4, 5, 6—and improve patient outcomes.3 The impact of such tools on resource use appears more variable, depending in part on the clinical area or type of care targeted. For instance, introduction of standardized order sets, care protocols, or critical pathways has been found to reduce overall length of stay, postoperative length of stay, and total charges for multiple surgical procedures, including total knee arthroplasty,7 appendectomy,8 total laryngectomy,9 cholecystectomy,10 carotid endarterectomy,11 gastrectomy,12 inguinal hernia repair,13 and colon surgery.14 In contrast, interventions to standardize treatment of conditions requiring inpatient medical rather than surgical management—such as pneumonia,15, 16, 17 congestive heart failure,13 and conservative management of acute appendicitis8—have had variable effects on length of stay and costs.

BHCS is in the process of developing and implementing more than 50 standardized order sets in a variety of clinical areas. The first of these—the adult pneumonia order set—was made available system-wide through the physician intranet portal in 2006. We investigated the effect of this order set on in-hospital mortality, compliance with evidence-based recommendations for pneumonia care, length of stay, cost of care, and fiscal operating margin.
Methods
Study Setting
BHCS is a not-for-profit, multihospital system in Dallas-Fort Worth, TX, that incorporates 20 owned, leased, affiliated, and short-stay hospitals with an annual total of more than 103,000 admissions. Only the eight acute care hospitals, where most patients with community-acquired pneumonia are treated, were included in this study.
BHCS is engaged in a multiyear process and organizational redesign project that is supported by the implementation of health information technology. The long-term goals for this project include:

Creating a culture that fosters interdisciplinary collaboration. Eliminating unnecessary variability in patient care. Developing and deploying the best evidence-based operational and clinical models.

Providing clinical decision support at the point of care. Providing caregivers with the opportunity to spend more time with patients.
Significantly improving quality and reducing errors.
The first phase of this redesign project has been paper-based for the most part, predominantly involving the establishment of monitoring and feedback systems to track performance on quality indicators that facilitate the design and implementation of

targeted quality improvement initiatives. These indicators include clinical preventive services delivery in the ambulatory care setting and Joint Commission Core Measures in the hospital setting. Introduced in 2004, the “Accelerating Best Care at Baylor” (ABC Baylor) class was designed to teach physicians, hospital administrators, nurse managers, and others the skills needed to actively lead quality improvement efforts and to facilitate process redesign.
The second phase of this multiyear project, currently ongoing, involves the standardization of care and the practice of evidence-based medicine through the development and implementation of standardized order sets and protocols. Although these tools are essentially paper-based, increasing technologic support (e.g., order set deployment via the intranet physician portal) is being introduced.
The third phase will involve the implementation of EHRs and CPOE, which will integrate the process redesign and order sets introduced during earlier phases.
Development of Order Sets
The identification of the most necessary order sets has been based on Diagnosis Related Group (DRG) data (particularly patient volumes), the Institute of Medicine’s 20 Priority Areas,18 BHCS performance on the Joint Commission Core Measures,19 and information from individual service lines (e.g., vascular, oncology, and radiology) about areas in which they feel the use of standardized order sets would have the greatest potential to improve quality of care.
The available evidence is reviewed, and a “straw model” is developed once a condition or procedure is identified as a target for a standardized order set. Appropriate leaders, physician champions (i.e., clinicians with dedicated BHCS-funded time for promoting quality improvement initiatives within BHCS), and other care providers and staff are identified and recruited for the development team. Sources used to identify the available evidence have included the National Library of Medicine, the Baylor Health Science Library, the Cochrane Database of Systematic Reviews, EMBASE, the University of Toronto Center for Evidence-Based Medicine, the Agency for Healthcare Research and

Quality (AHRQ) Evidence-Based Practice Center program, the AHRQ National Guideline Clearinghouse (NGC), UpToDate, and Zynx.
Additionally, through physician town hall meetings, departmental meetings, and direct contacts, all BHCS physicians have the opportunity to contribute to the content and format of the order set. Based on the information thus gathered and on their knowledge of local practices, a subspecialty team develops a working draft of the order set and pilots it within their own practices/departments. At minimum, this team includes a representative from each BHCS hospital, a pharmacist, a nursing representative, and a relevant BHCS physician champion. Following revisions to address any issues identified through the pilot testing, the order set is reviewed by the BHCS Pharmacy and Therapeutics Committee and the Patient Safety Committee. Following their approval, it is reviewed by the Physician Design Team, which includes physician champions, a BHCS pharmacist liaison, the BHCS Partnership Council leader, the Physician Team leader, and ad hocphysician leaders as needed. The Physician Design Team has final control over all order set content.
Finally, each order set is reviewed by the Quality and Fiscal Impact Committee and then sent to the Best Care Committee, a system-wide entity made up largely of hospital presidents, chief nursing officers, health care improvement directors, and physicians with specific quality improvement leadership roles. The order set is then deployed via the portal. Education on using the order sets has been provided to relevant care providers through “academic detailing”20 by physician champions.
Each order set is reviewed and updated annually by subspecialty teams, physician champions, and the Physician Design Team. Changes are reviewed by the BHCS Pharmacy and Therapeutics and Patient Safety Committees. In addition, new evidence from research and local experience is monitored, facilitating ad hoc review and revision of the order set. This ensures that the standardized order sets are consistent with the practice of high quality, evidence-based medicine.

Development and Deployment of the Adult Pneumonia Order Set
Beginning in early 2005, the BHCS Adult Pneumonia Order Set was developed by a systemwide multidisciplinary team including pharmacists, nurses, respiratory therapists, care coordinators, health information management staff, and physicians specializing in infectious diseases, pulmonology, internal medicine, and family practice. Since this was the first effort at system-wide standardization of care processes, the development of the Adult Pneumonia Order Set highlighted the need for much of the supporting structure for such efforts, including a good internal communications process and system-level groups in which stakeholders are brought together with their counterparts from other hospitals (e.g., the Pharmacy and Therapeutics Committee, which was formed in response to this need). It has been intertwined with substantial organizational learning and development of the necessary infrastructure, especially the creation and tasking of teams and committees that play key roles in the order set development process.
From November 2005 to February 2006, the Adult Pneumonia Order Set was piloted by the providers involved in its development at several of the BHCS acute care hospitals. Although no widespread effort was made to inform other care providers about the order set or to encourage its use during the pilot stage, the order set was available to all providers through the BHCS intranet.
Subsequent order sets have not been made generally accessible during the pilot stage because the appearance of the pneumonia order set on the intranet with no preceding education or information about its use created some confusion. Based on pilot experience, minor changes were made to the Adult Pneumonia Order Set prior to its system-wide deployment in order to increase its effectiveness and user-friendliness. These included the addition of passive decision support reminders related to the use of the analgesic Darvocet, the addition of a default care coordination consult, and a formatting change to eliminate confusion involving the separation of antibiotic groupings.

In March 2006, the Adult Pneumonia Order Set was deployed system-wide via the physician portal. At this time, “order set use” was made a required field in the integrated outcomes, resource, and case management system used for pneumonia patients at all BHCS hospitals (MIDAS+), facilitating the tracking of order set use. Strategies to increase awareness and encourage use of the order set included:
1. A high-profile awareness campaign, which was presented to the Best Care Committee and made available to frontline care providers through the BHCS intranet. 2. Just-in-time training provided to nursing units at some BHCS acute care hospitals. 3. Incorporation of the order set into the Baylor University Medical Center order entry system. 4. Academic detailing by physician champions.
Anecdotally, this last strategy was perceived as the most effective in raising awareness of and knowledge about the order set.
The Adult Pneumonia Order Set was the first standardized tool BHCS made a concerted effort to implement system-wide. For this reason, there was no preexisting method or infrastructure for widespread deployment. To increase standardization and improve quality of care, such tools and strategies are under development for the deployment of future order sets and other system-wide initiatives.
Patients for Evaluation of Pneumonia Order Set
All adults (>18 years) discharged from one of the eight BHCS acute care hospitals between March 1, 2006 and September 30, 2007, who had been admitted with a working diagnosis of pneumonia and who met the Joint Commission definition of pneumonia21 (based on ICD-9-CM diagnosis codes) were eligible for this study. Patients were excluded if “for comfort measures only” was recorded in their admitting physician orders or note, consultation notes, emergency department record, history and physical, physician orders, or progress notes.

Outcome Measures
The primary outcome measure was a difference in performance for clinical quality and financial indicators between pneumonia patients who were treated with or without the BHCS standardized Adult Pneumonia Order Set. Clinical quality indicators included inpatient mortality, readmission within 30 days, and compliance with the Joint Commission Core Measures for pneumonia, as indicated by the pneumonia composite compliance index. The core compliance index was based on eight of the national quality measures for pneumonia:
1. PN-1 oxygenation assessment. 2. PN-2 pneumococcal vaccination. 3. PN-3b blood culture before first antibiotic. 4. PN-4 adult smoking cessation advice/counseling. 5. PN-5b initial antibiotic received within 4 hours of hospital arrival. 6. PN-6a initial antibiotic selection for community acquired pneumonia (CAP) in immunocompetent ICU patients. 7. PN-6b initial antibiotic selection for CAP in immunocompetent non-ICU patients. 8. PN-7 influenza vaccination.
The core compliance index was calculated as the proportion of pneumonia patients eligible for the above measures who receive all the measures for which they are eligible.21 Financial indicators included length of stay, direct cost of care, expected payment (based on payer type), and contribution margin (calculated as expected payment less direct cost of care).
Data Collection
Data on order set use (“BHCS order set,” “personal order set,” or “no order set”), age, sex, race/ethnicity, admitting BHCS hospital, All Patient Refined Diagnosis Group (APR DRG) Severity of Illness (SOI) and Risk of Mortality (ROM), and delivery of the Joint

Commission Core Measures for Pneumonia were collected from MIDAS for each patient. “Personal order sets” were those developed by individual physicians, physician groups, or hospitals that had not undergone the full development and review process described above for the BHCS order sets. Length of stay, inpatient mortality (including time from admission to death), readmission within 30 days (including time from discharge to readmission), direct cost of care, expected payment, contribution margin, and diagnosis codes used to calculate Greenfield comorbidity scores were determined from administrative data.
Statistical Analysis
To ensure the statistical assumption of independent observations was met, the analysis considered only first hospital admission for pneumonia for patients with multiple admissions during the study period. Due to the continuous decline seen in personal order set use over the study period, the analysis focused on comparing BHCS order set use vs. no order set use.
Univariate analyses were conducted to examine the association between order set use and patient characteristics/outcomes of interest. Chi-square tests and Fisher’s exact tests were used to assess the association of order set use with categorical characteristics/outcomes (sex, race, facility, mortality, core measure compliance, and readmission within 30 days). For ordinally scaled measurements (APR DRG risk of mortality, severity of illness, Greenfield comorbidity score, and month of discharge), trend tests were also performed. Two-sample t-tests were used for mean comparisons of continuous outcomes or characteristics that did not violate the assumption of normality. Robust estimation and regression approaches were used for continuous outcomes that were highly skewed.22, 23
Multicollinearity of all covariates to be included in the adjusted analysis was assessed prior to performing multivariable analysis. No evidence of multicollinearity was observed, and the adjusted analysis was conducted following two approaches: covariate adjusted and propensity score adjusted. In the covariate-adjusted analysis, all covariates of

interest (age, sex, race, physician specialty [hospitalist vs. other], Greenfield comorbidity score, APR DRG risk of mortality/severity, payer type, admission source, hospital, and discharge month) were included in the regression model, and the adjusted effect of order set use was estimated.
The propensity score approach involved the creation of propensity scores to determine the conditional probability of a patient being treated with an order set given the set of the patient’s characteristics (age, sex, race, physician specialty [hospitalist vs. other], Greenfield comorbidity score, APR DRG risk of mortality/severity, payer type, admission source, hospital, and discharge month). Regression analysis with order set use and propensity score as covariates was then performed to determine the adjusted effect of the order set. APR DRG Risk of Mortality was used in the models for safety and effectiveness indicators, while APR DRG Severity of Illness was used for efficiency and fiscal indicators.
The effects of order set use on in-hospital mortality and on readmission within 30 days were assessed using Cox proportional hazard regression. Time to death during the hospital stay was considered for the in-hospital mortality model, while time to readmission from discharge date was considered for 30-day readmission. The adjusted effect of order set use on core measure compliance was modeled using logistic regression. However, since the outcome measure of interest (core measure compliance) was frequent in the study population (>70 percent), the resulting odds ratio overestimates the risk ratio.24 We therefore applied a simple approximation24 to obtain a better estimate of the true adjusted relative risk. Length of stay and cost data were modeled using robust regression methods. Analyses were conducted using SAS 9.1 (SAS Institute, Cary NC) and S-Plus 7.0 (Insightful Corp, Seattle, WA).
Results

Figure 1
Identification and exclusion of patients included in (more...) Click on image to enlarge
Figure 1

Identification and exclusion of patients included in the evaluation of the Baylor Health Care System Adult Pneumonia Order Set
Between March 1, 2006 and September 30, 2007, 4,032 adult patients admitted with a working diagnosis of pneumonia who met the Joint Commission definition of pneumonia and were not admitted for comfort care were discharged from the eight BHCS acute care hospitals (Figure
1
). Average
age among the patients meeting study inclusion criteria was 67 ± 17 years. About half (55 percent) were female, 75 percent white, 18 percent black, and 6 percent other.
Significant variation in order set use was observed by age (P = 0.01) but not by sex or race. When variation in order set use by APR DRG classes and Greenfield Comorbidity Score was examined using mean score statistics that take into account the ordinal nature of these categorizations, there was a significant association between order set use and APR DRG severity of illness (P <0.01) and APR DRG risk of mortality (P <0.01). Sicker patients were less likely to receive the order set. No significant difference was seen using the Greenfield comorbidity score (P = 0.42).

Table 1
Order set use by month for first pneumonia admissions (more...)
Table 1
Order set use by month for first pneumonia admissions of adult patients to Baylor Health Care System acute care hospitals: March 1, 2006 – September 30, 2007
Discharge Month March 2006 April 2006 May 2006 June 2006 July 2006 August 2006 September 2006
Total (N) 219 159 153 107 126 110 115
BHCS Order Set N (%) 59 (27) 48 (30) 63 (41) 51 (48) 50 (40) 51 (46) 59 (51)
No Order Set N (%) 112 (51) 73 (46) 73 (48) 47 (44) 65 (52) 57 (52) 49 (43)
Personal Order Set N (%) 48 (22) 38 (24) 17 (11) 9 (8) 11 (9) 2 (2) 7 (6)

Discharge Month
Total (N)
BHCS Order Set N (%) 82 (51) 97 (55)
No Order Set N (%) 68 (43) 74 (42)
Personal Order Set N (%) 10 (6) 5 (3)
October 2006 160 November 2006 December 2006 January 2007 252 February 2007 March 2007 April 2007 May 2007 June 2007 July 2007 August 2007 September 2007 Total 3301 261 209 196 158 172 143 124 233 228 176
135 (59)
84 (37)
9 (4)
146 (58) 174 (75)
100 (40) 55 (24)
6 (2) 4 (2)
212 (81) 159 (76) 159 (81) 133 (84) 141 (82) 113 (79) 102 (82)
46 (18) 49 (23) 35 (18) 24 (15) 31 (18) 30 (21) 22 (18)
3 (1) 1 (0) 2 (1) 1 (1) 0 (0) 0 (0) 0 (0)
2034 (62)
1094 (33)
173 (5)
Order set use by month is shown in Table 1. For first pneumonia admissions, BHCS order set use increased from 27 percent in March 2006 to 82 percent in September 2007 (P

<0.01); no order set and personal order set use declined (from 51 percent to 18 percent, and from 22 percent to 0 percent, respectively). Despite system-wide promotion of the Adult Pneumonia Order Set, dramatic variation in use was seen among hospitals (P <0.01), with use ranging from 43 percent of first admission pneumonia patients at one hospital to 91 percent at another.
Table 2
Unadjusted results comparing quality of care and financial (more...)
Table 2
Unadjusted results comparing quality of care and financial indicators for first pneumonia admissions to Baylor Health Care System acute care hospitals that used vs. did not use the order set: March 1, 2006 – September 30, 2007
Order set Safety and effectiveness indicators In-hospital mortality Pneumonia core measure compliance 2376 (76.0) 1585 (77.9) 791 (72.3) <0.01a All (N = 3128) N (%) 138(4.4) BHCS (N = None (N = 2034) N (%) 67 (3.3) 1094) N (%) 71 (6.5) P-value <0.01a

Order set Safety and effectiveness indicators Readmission within 30 days Efficiency and fiscal indicators Length of stay (days) Direct cost ($) 5418 (4488) Expected payment ($) Contribution to margin ($) 7131 (4483) 1797 (3879) 5092 (3918) 6642 (3794) 1592 (3616) 6022 (5432) 8105 (6004) 2229 (4453) 0.02b 0.01b 0.06b Mean (±SD)c 5.3 (3.7) Mean (±SD)c 5.2 (3.6) Mean (±SD)c 5.8 (4.4) 0.11b All (N = 3128) N (%) 349 (11.2) BHCS (N = None (N = 2034) N (%) 217 (10.7) 1094) N (%) 132 (12.1) P-value 0.24a
Based on Chi-square test
Based on robust ANOVA test
Robust mean ± standard deviation
Table 2 shows the unadjusted results for the effect of order set use on quality of care and fiscal indicators. In-hospital mortality was significantly lower among patients for whom the order set was used (P <0.01), as were expected payment (P <0.01) and contribution