Calculating inter-rater reliability (IRR) is an essential part of ensuring trauma data quality. It is also a key component of registrar education. However, many trauma program managers and registry leaders do not use this tool as effectively as possible.
IRR measures the level of agreement between separate chart abstractions. After a trauma registrar has abstracted a chart, it is re-abstracted by a trauma program manager, PI coordinator or other team member. The level of agreement between the two raters is expressed as a percentage. This IRR percentage can be used to assess registrar performance and to identify team members who need additional education.
Typically, however, program leaders focus on IRR at the chart level. For example, the data validation process might yield an IRR of 96% for the chart of Patient A. The problem is that this approach provides a high-level view of registrar accuracy, but it does not guide specific improvements. While the registrar’s overall accuracy for a chart may be 96%, he or she might be consistently making abstraction errors on several important data points.
Our recommendation: Track IRR by individual data point and assess accuracy gaps at the data-point level. This strategy allows trauma program leaders to get a much more accurate measure of each registrar’s performance. It also enables a more focused approach to registry team education.
How to use the data-point IRR strategy
First, identify the data points that are most important to your trauma program. These might be factors that affect:
- probability of survival (for example, AIS coding, initial vitals or GCS)
- risk adjustment (such as pre-existing conditions or prehospital management)
- or benchmarking (like ICU days or vent days)
Second, during your data validation process, calculate and track IRR individually for these select data points. This is not a significant expansion of scope. You will still be re-abstracting 5% to 10% of charts, as recommended by the ACS Committee on Trauma. You will simply be calculating IRR for each key data point and tracking performance at that level.
Third, use the results to identify educational needs. We recommend considering any data-point IRR under 95% as an opportunity to improve. Education efforts can use specific cases to illustrate mistakes and clarify data definitions. In our experience, focusing on specific data points increases learner retention and increases registrar confidence.
Use IRR results to update your internal trauma data dictionary
It will often make sense to update your hospital’s internal trauma registry data dictionary to reflect the findings of your IRR process.
For example, the process might reveal a consistent discrepancy for “ED admit time.” Guided by staff IRR rates on this data point, you might discover that registrars are capturing this data from different locations within the electronic medical record (EMR).
Registrar A is recording the ED admit time from the ED Encounter Summary in the EMR, while Registrar B is getting that data from the ED Patient Care Timeline. Sometimes these EMR values are the same, but they are often completely different.
Best practice is to make sure all registrars record data from the same location, which should also be the best location. With this goal in mind, you might update your hospital’s internal trauma registry data dictionary with the following hierarchy:
1. ED Patient Care Timeline
If there is no clear time in this location, then…
2. ED Encounter Summary
If there is no clear time in this location, then…
3. Pre-Hospital Run Sheet
Data-point IRR can help you identify similar source issues and make sure your internal data dictionary specifies a clear hierarchy. This will help ensure data consistency and increase your confidence in your trauma data.
Monitor and track IRR on specific points
Make sure to collect enough IRR data before making assessments about registrar performance or changing registry processes. For an accurate view of total abstraction quality, get data validation results on a minimum of 15 to 20 charts.
Here’s why: If you perform data validation on five charts and the same data point is incorrect on two of them, the registrar’s IRR on that point will be 60%. However, the same two mistakes across 15 charts will yield an IRR of 86.67%. This will still be an area that needs improvement, but it will not be as alarming as 60% accuracy.
For the purpose of ongoing monitoring, trauma programs should retain all documentation used in data validation. Track IRR rates for each data point and pay attention to trends.
Make sure to establish a standard for dating registry records. Tracking by date — either the admit date or the chart closure date — will help program leaders spot trends and make comparisons.
One final recommendation: When an individual data point achieves and maintains a 99% IRR threshold, consider removing that point from your data validation process.
Benefits of this approach
Tracking IRR for individual data points gives a better overall view of registrar performance and data accuracy. As the foundation of an educational strategy, it can lead to steady increases in overall IRR. This will increase data reliability and build your confidence in TQIP benchmarking reports. In addition, this strategy can allow trauma programs to use their registrar education budget more effectively.
Do you want to strengthen and streamline your data validation processes? Pomphrey Consulting uses proprietary software from Buck Data Validation Systems (BDVS) to automatically report data-point IRR. This system streamlines key aspects of data-point IRR, giving trauma program leaders powerful tools for improving data quality. BDVS can be customized to your program’s needs and internal data validation processes. To find out more, explore Data Management Services from Pomphrey Consulting.