Performance improvement is a key part of every trauma program. But according to David Kashmer, MD, MBA, many trauma teams fail to make effective use of their most important PI resource — data.

“Using data incorrectly in performance improvement can actually decrease quality,” said Dr. Kashmer, a trauma and acute care surgeon and healthcare quality expert. Poorly handled data, he noted, can mask quality problems and in some cases undermine teamwork.

Dr. Kashmer is a Lean Six Sigma Master Black Belt with extensive experience using data-driven quality strategies. He recently analyzed five mistakes that teams make when using data in trauma PI. He also explained strategies for making data collection easier, collecting more reliable data, interpreting PI data accurately, and monitoring data to make sure performance improvements “stick” over the long haul.

1. Focusing on isolated cases

In most trauma programs, performance improvement focuses on failures within individual cases. “The problem is that reacting to single-case outbreaks actually introduces more variation into a system,” Dr. Kashmer said. “When we make changes based on a single terrible case or one angry email, we are essentially ‘tampering’ with the system. That increases the risk of a Type 1 error — making a change because we think something is wrong when it really isn’t.”

Focusing on failures can also open the PI process to political manipulation. “In healthcare, the squeaky wheel commonly gets the grease,” Dr. Kashmer said. “So when your PI program is built solely on case review, you make it vulnerable to whatever powerful person or group squeaks the loudest or most often. Not only that, you also miss opportunities to repair important issues that no one has squeaked about.”

“All this does not mean single cases are useless,” Dr. Kashmer clarifies, “only that we should be sure to consider cases in the context of our whole system.” To do that, trauma PI teams need to use broad data samples to correctly identify the factors that produce bad outcomes.

“In complex systems such as trauma care, most failures are conspiracies of six or seven underlying factors,” Dr. Kashmer said. “For example, typically it’s not just that the surgeon didn’t decide fast enough to go to the OR. There are likely OR prep issues, ED transfer issues, EMR issues and other factors that create friction within the system. Many factors blend together to give the output we experience as a delay in getting a patient to the operating room. But it’s difficult for us to see the complexity of a whole system and how the probabilities of different parts blend together. If we aren’t careful, we latch onto overly simple answers that focus on just one or two elements.”

The key to avoiding this mistake is to use tools that help identify the full range of factors that could be contributing to poor performance. One of the most effective tools is the Ishikawa diagram, also known as a “fishbone diagram.”

“An Ishikawa diagram is a good tool to use for analyzing failures in complex systems, especially when there is disagreement about the root cause of a problem or when there is not a lot of data on what is happening,” Dr. Kashmer said.

To create a basic Ishikawa diagram, draw a horizontal line. Write “Effect” at the right end of the line and note the problem. Next, draw six lines extending outward from the main line. Label these lines with “the five M’s and one P”: Materials, Machine, Method, Mother Nature, Management and People.

Working as a team, fill in the chart with all the factors that could be contributing to the identified problem. For example, if a team was brainstorming possible causes for delayed operative care in a patient with a small bowel injury, their Ishikawa diagram might list some of the following potential factors:

• Materials:
  • The patient was morbidly obese and unable to fit into the CT scanner. (“Since CTs may help find bowel injuries earlier, failure to perform CT may delay recognition of these injuries,” Dr. Kashmer noted.)
  • Morbid obesity may have limited the ability of the abdominal exam to demonstrate small bowel injury.
  • (Note: In healthcare, “materials” are commonly the patient factors, because patients are what the system works on. “So morbid obesity may be listed as an important patient factor — it doesn’t make it impossible to find a small bowel injury promptly, but it may make it more difficult.”)
• Machine:
  • Lab tests, including amylase and lactate, were not drawn in a timely fashion owing to EMR issues.
  • The CT was busy with another patient (not actually an issue for the case under consideration, but included by the team for the sake of identifying all the potential factors that could lead to OR delays).
• Method:
  • It requires five calls to contact the OR team.
  • It takes the team 30 minutes to drive in from home.
• Mother nature:
  • Poor weather hampered team response.
  • The trauma team got busy with other activations, delaying reexamination of the patient.
• Management/Measurement:
  • OR management requires the trauma surgeon to make several calls and justify use of an operating room.
• People:
  • Someone failed to answer the phone.
  • The surgical team missed an obvious sign that the patient required the OR.

“This scenario highlights how many different factors, some controllable and some not controllable, line up to make it more likely that this patient with this injury will take longer to get to the OR,” Dr. Kashmer said.

Once you have filled out your fishbone, label the different factors as either “controllable” or “noise” that cannot be controlled. “Collect data on the factors involved, and then perform multiple regression analysis to identify which controllable and non-controllable factors are truly correlated with the effect,” he said. “Sometimes you discover interesting things — for instance, that factors you cannot directly control are ones that correlate with the outcome. More importantly, sometimes you learn that factors you can control impact the probability of a defect. Focus on those.”

“Remember, each category likely has multiple factors, and the factors you identify with the Ishikawa tool are still just opinion,” he said. “You still must collect data and analyze it properly if you want to go by more than just your opinion. But this process helps ensure your team looks at all the potential causes of a PI problem, not just the ‘people’ issues that so often attract the most attention.”

2. Relying on hospital databases

Many trauma PI initiatives use clinical and administrative databases maintained by the hospital. “The problem is that hospital databases sometimes don’t represent the process as well as data collected directly from the system,” Dr. Kashmer said. “These databases tend to filter out information about problems.”

Dr. Kashmer believes it often makes sense to use trauma registry data in PI. However, he cautions trauma leaders about the reliability of this data. A recent study suggests the accuracy rate of trauma registry data is quite low.

“Remember, data that makes it into registries and warehouses in all fields is often cleaned up somehow,” he said. “Many times, staff record numbers as estimates or when they remember to do so. It’s typically not intentional, but in acute trauma situations what makes it on the paper may not be perfect. More importantly, your particular issue may need a piece of data that isn’t commonly recorded. Or you may need data that has a different operational definition than the same data field in the registry.”

“I know most of us use registry data for trauma PI,” Dr. Kashmer said. “It’s what we have and it’s much better than what we commonly see in other services. However, for addressing a PI issue I highly suggest collecting some data right from the process. It’s amazing how different things look when you have prospective data that didn’t come from a data warehouse or a registry.”

Prospective data collection allows teams to focus on very specific and useful endpoints. “If we want to find out how long it takes to get fresh frozen plasma to the bedside, we want to have a member of the trauma team marking times for 100 consecutive doses,” Dr. Kashmer said. “Collecting data right from the process lets you make sure you are measuring useful endpoints in a way that is statistically rigorous.”

One other advantage of using prospective data is it allows you to structure data collection to ensure anonymity. This helps prevent the finger-pointing that is so detrimental to performance improvement.

“Some people are worried that data will uncover some problem with them personally,” Dr. Kashmer said. “When I work with a trauma team, we don’t include names in the data collection and the data are not traceable to individual providers.”

3. Making data collection painful

Some trauma PI teams fail by trying to collect too much data. “You don’t need a million data points,” Dr. Kashmer said. “Usually four or five well-chosen endpoints are all you need to adequately characterize performance.”

Dr. Kashmer recommends a systematic approach to planning data collection. First, create a SIPOC (Suppliers-Input-Process-Output-Customers) diagram of the PI project. Start in the middle with process, and define the five or six high-level steps that characterize the process you are working on. In healthcare the input is generally the patient, and the output is the patient having received some defined intervention.

Second, decide on the endpoints you need to track. “In medicine, we often want to get data on 40 different endpoints,” Dr. Kashmer said. “That’s really not necessary. Usually you only need about five endpoints to completely characterize a system.” Using your SIPOC diagram, select one or two measures from each of the input, process and output phases.

For example, say you are often unable to get an EMS report for severely injured patients admitted to your trauma center. You want to get data on this issue to (a) determine if there really is a problem and (b) understand how to fix it. Data collection in this scenario might be as simple as tracking a handful of endpoints:

• Input measures: To track inputs, you create a measure of EMS report completeness. For example, you decide on five key elements you require in an EMS report and score each report on a scale of 1 to 5.
• Process measures: You decide to create a measure called time until EMS report. The clock starts when EMS arrives on scene and stops when someone at the trauma center is notified about the patient, either prehospital via phone/radio or by report when the patient arrives in the trauma bay. (The project’s goal may be to minimize the time from arrival on scene to trauma team notification.)
• Output measures: Some of the value of the prehospital EMS report is that it allows preparation for patient arrival, appropriate level of trauma activation, and advance warning for part of the trauma team so it can arrive on time. Therefore, you may choose to measure undertriage rate, provider arrival time, or a tailored endpoint you create that has value at your center.

“Part of the art of selecting endpoints is deciding early on whether you will collect continuous or discrete data,” Dr. Kashmer said. “From the point of view of simplicity, each approach has advantages and disadvantages.”

Discrete data are “categorical” — for example, yes/no, ground/helicopter or live/die. Continuous data are data that can be divided into smaller and smaller units and still be meaningful — for instance, time or length.

“The main advantage of discrete data is that it can be collected very easily and rapidly,” Dr. Kashmer said. “The downside is that discrete data will often require a large sample size to ensure statistical validity.” In contrast, the main advantage of continuous data is that it generally requires a smaller sample size to demonstrate something meaningful.

Whichever endpoints you choose, calculate sample size ahead of time. (For an explanation of how to calculate sample size for both continuous and discrete data, read How You Measure the Surgical Checklist Determines What You Find.)

“While it is true that many PI projects do not collect enough data, others seem to get stuck in data collection mode indefinitely,” Dr. Kashmer said. “Establishing a statistically valid sample size will help you make sure your team avoids unnecessary work.”

4. Fumbling the analysis

Correct analysis makes the difference between useful data and muddled information. According to Dr. Kashmer, trauma program staff often need training in data analysis techniques. At a minimum, it is important to understand methods for analyzing and interpreting continuous and discrete data and performing multiple and logistic regressions.

“One key skill is understanding how to interpret histograms,” Dr. Kashmer said. “You need to be able to say to the team, ‘This is how our data distribution curve looks, here is our upper specification limit and here are our defects.’”

For example, a state may require surgeons to be present in the trauma bay within 10 minutes of patient arrival. “In this case, the upper specification limit is defined by the regulatory authority,” Dr. Kashmer said. “Any data points above that limit are your defects. From that, you can calculate your defect rate.”

Data distribution curves can help trauma teams visualize their performance and better understand their PI opportunities. However, one common pitfall is misinterpreting the concept of defects and defect rates.

“Most service industries work at the rate of one defect in every 1,000 opportunities, and that is also what we usually see when we begin working on a healthcare process,” Dr. Kashmer said. “Everyone feels good because 999 times out of a thousand, things are going great. But in trauma, that one defect can be a real problem. That same level of performance, for example, would mean that one plane would crash each day at a busy airport like O’Hare. So that typical level of performance doesn’t work in healthcare.”

According to Dr. Kashmer, trauma teams should aim for a much lower defect rate. For example, a rate of 0.0004% — or one defect for every 250,000 opportunities — is achievable. “In my experience, that can be achieved by an excellent PI process. It’s tough to get there, but it is doable. In healthcare, we all aim for zero defects, of course, and some quality improvement systems do achieve incredibly low rates like one defect per every million opportunities.”

5. Monitoring the wrong control data

“The control phase of a PI project lets you know when to intervene — when to trim the weeds again,” Dr. Kashmer said. However, most trauma teams use the wrong control charts.

“Many people decide to use an I-MR chart — or ‘individuals and moving range’ chart — to monitor their performance,” Dr. Kashmer said. When used correctly, an I-MR chart can detect variations in performance that indicate a process is out of control.

“An I-MR can let you know when a process is broken, often before a true error occurs,” he said. “The problem, which most people don’t realize, is that I-MR charts are only effective if your data are normally distributed.”

The mathematics describing normal versus non-normal data distribution can be complex. For the sake of simplicity, one can say that normally distributed data will produce a bell-shaped curve.

“If your data do not describe the normal curve that we all love, many of us feel an I-MR chart won’t be able to tell you much,” Dr. Kashmer said. “You will first need to perform a data transformation, such as the Box-Cox transformation, which can force data into a normal distribution by raising it to a power.”

Another common pitfall is monitoring control data before a process has been optimized. “The only thing a control chart can tell you is whether your system is performing as expected,” Dr. Kashmer said.

“Say that your state gives you two hours to get patients to the OR,” he said. “If your normal data distribution centers at two hours, your control chart will tell you that everything is within the control limits and, essentially, that everything is fine. Unless you get an event that is more than three standard deviations away, the control chart will not raise any alarms.”

“I remember a trauma center that thought it was doing just fine because its control chart for time until surgeon arrival in the trauma bay looked great,” Dr. Kashmer said. “No points were out of control. But guess what: The system was just performing at its routine level of variation.”

“Even though the control chart looked fine, surgeons were frequently unable to arrive in the ED until well beyond 15 minutes,” he said. “The routine level of variation was totally unacceptable, and the trauma center had been misled by the control chart.”

Stronger teams

A rigorous approach to data is challenging, but the results can be energizing. “When a trauma team begins to look at data in a rigorous way that is non-pejorative and keeps them focused on the system instead of picking on individuals, things improve,” Dr. Kashmer said.

“At first, teams often find they’re not in a good place,” he said. “But when a team gets an accurate look at its performance through good data, people start asking what changes they can make to improve their performance as a team and a system. Next thing you know, that data- and team-driven approach allows for constant improvement and success.”

About the Expert

David Kashmer, MD, MBA, is a trauma and acute care surgeon and healthcare quality expert. He is a Lean Six Sigma Master Black Belt and was recently named to the Malcolm Baldrige Board of Examiners.

Dr. Kashmer writes frequently about data-driven healthcare quality improvement at TheSurgicalLab.com and TheHealthcareQualityBlog.com. His writing has also been published by TheHill.com, Insights.TheSurgicalLab.com and OR Manager. He is the author of the recent Amazon bestseller, Volume To Value: Proven Methods for Achieving High Quality in Healthcare.