An organization must configure its systems so data elements are collected exactly the same way over time. This approach ensures accurate and credible data for Quality Improvement ( QI) improvement and avoids a team’s wasted efforts on manual activities and reconfiguring its systems.
A successful approach to reliable data collection includes proven tools, techniques, processes, and frameworks, and often involves automating parts of the data collection process, if feasible. At minimum, a QI team should develop a well-documented plan with detailed steps for collecting each data element. Successful QI teams recommend that a detailed data collection plan is in place prior to actually collecting the data, or to develop the plan while the baseline is calculated. An effective data collection plan includes the following details for each measure:
- Name of the measure
- Denominator detail with inclusions and exclusions
- Data source for the denominator and include any specific queries to be run or report parameters that must be entered
- Numerator detail with inclusions and exclusions
- Data source for the numerator and include specific queries to be run, manual steps, orspecific sampling parameters
- If different individuals are assigned, identify who collects each data element and calculates the measure
- Include a calendar of measure-performance reporting, e.g., calculating Breast Cancer Screening measure on the second Tuesday of each month is appropriate if the Breast Cancer Screening QI Team reviews performance data on the second Friday of each month
In the figure above, a team decides to measure CD4 q counts in patients with HIV at the end of four months, while aiming for a standard of care, and again at the end of six months to compare others using the measure recommended by the HIV/AIDS Bureau (HAB). The four-month CD4 q measure was simplest to calculate from the electronic health record (EHR), while the six- month CD4 q measure was easily assessed using the organization’s CAREWare (CW) program. The measures were linked to the Quality Improvement Committee (QIC) calendar. The next sections discuss these important components in more detail.
What Data to Collect
During the next stage of the process, a QI team develops a collection strategy, methods, and tools before collecting the data. Most standardized measures, including the HRSA clinical quality measures, specify what data elements are needed to calculate the measure, as shown in Table 1. A QI team collects data elements for the denominator and numerator, then calculates the measure in the same way the baseline was calculated. A team also needs to incorporate any inclusions and exclusions for the numerator or denominator.
Denominator with Exclusions and Inclusions
In the BCS example, a QI team first determines a methodology to find all women aged 42 to 69 years who will be impacted by the QI project. If a team targets improvements on just one provider or practice team, it includes that provider’s or practice team’s specific patients. Targeting a smaller subset of patients within an entire practice is known as a population of focus (POF). Alternatively, a QI team may decide to focus on improving breast cancer screening for its entire population.
An organization’s QI team has several options for gathering information, but two common approaches are through a practice management system or an electronic health record (EHR). If a QI team focuses on all patients, the denominator is the number of female patients in the practice aged 42 to 69 years. If a team focuses on just one provider’s patients, the denominator is the number of female patients aged 42 to 69 years who are part of the provider’s patient panel. If a random sampling methodology is used, the denominator is the number of patients that reflect an adequate sample size.
Note: Random sampling is a method in which all members of a group (population or universe) have an equal and independent chance of being selected. Since improvement is the goal, not measurement, random sampling methodology allows an organization to reduce the QI team’s burden while focusing on performance improvement of the selected measure.
Some measures specify inclusions, which are specific parameters that must be present for inclusion in the denominator. For example, there are measures for diabetes mellitus that specify the type of diabetes diagnosis (gestational, type 1 or type 2) for inclusion in the denominator. For the BCS measure, there are no denominator inclusions.
Before finalizing the denominator, it is important to check for exclusions, which are not present in all measures. Exclusions are parameters that guide a team to remove specific patients from the denominator for well-defined reasons. For the BCS measure, there is the following exclusion criterion for the denominator:
Denominator Exclusion: Women who had a bilateral mastectomy wherein the administrative data does not indicate a mammogram was performed; the bilateral mastectomy must have occurred by December 31 of the measurement year.
For the BCS measure, a team needs to exclude women with bilateral mastectomies as defined in this denominator exclusion. These patients are found using CPT codes if using an EHR. Details on the specific codes are found in the Breast Cancer Screening module. A team using paper records often relies on clinicians to identify patients who had a bilateral mastectomy as a therapeutic or prophylactic intervention. Another technique used to evaluate the outliers for the measure is to review the medical records of patients who do not meet the numerator criteria to ensure they should be included.
Numerator with Exclusions and Inclusions
When considering the numerator, it is important to start with patients included in the denominator. In the clinical setting, for example, the denominator represents all patients eligible for or requiring certain care. The numerator represents those eligible patients who actually received care. It is important to ensure that the data collected matches the specifications of the measure. In the BCS measure, the numerator represents all women in the age range without bilateral mastectomy who had a mammogram within the last two years. While there are no exclusions for the numerator of the BCS measure, there is the following inclusion:
Numerator Inclusions: Documentation in the medical record must note the date the test was performed and the test results (or a copy of a mammogram result), or the record notates the date and results of a test ordered by another provider.
This inclusion specifies that ordering the mammogram is insufficient: the results of the mammography must be evident to include the patient as having received the appropriate care.
If a team uses an EHR, the system can generate a report indicating if a mammogram report from a certain date is present in the record. An EHR’s capability also allows reports that include the patient’s name, date of the mammogram, and the results. It is important to ensure that only patients with documented mammogram results are included in the numerator; however, looking for computerized physician order entries (CPOE) for mammography is an inaccurate approach for determining this measure’s numerator.
Even with an EHR, a team may discover that data capture for the numerator of the BCS measure is a more manual process. For example, the records of the patients within the age range are gathered via the EHR, but their mammography reports are reviewed manually to ensure test results are recorded—a process similar to a paper-based practice.
When confronted with a manual process, a team may choose to use a sampling methodology to decrease its burden. Sampling is a process where the performance measure is calculated on a subset of the total; it is a valid method for performance measurement when done appropriately. It is imperative that the sample is chosen randomly to accurately reflect the population. A team with a smaller patient volume, however, may choose to evaluate all patients in the denominator.
Randomized sampling is the most recommended method for performance improvement, because it ensures every patient has an equal chance of inclusion in the sample without bias. Some HRSA programs require specific parameters for using sampling, and an organization should check the Program Guidance before choosing it.
Methodology for Obtaining a Random Sample
Measurement is intended to speed improvement; however, if a QI team stalls in the in measurement process because it perceives more data is needed, changes are unnecessarily deferred. It is important to remember that improvement is the goal—not measurement. When a team gathers sufficient data to make a reasonable judgment, it should move to the next step. Instead of measuring the entire process (e.g., for an entire month, include all patients waiting in the clinic), it is more efficient to measure a sample (e.g., sample every sixth patient for one week, or sample the next eight patients) to help a team understand how a system is performing. There are a number of approaches for obtaining a random sample; however, an organization should review its program requirements before using a random sampling methodology.
Note: This post is an adaptation of the “Managing Data for Performance Improvement” document, by the U.S. Department of Health and Human services. We focus on the collection of data elements needed to obtain the HRSA Clinical Quality Measures.