Abstract
Data in trauma registries need to be reliable when used for evaluation of injury management, trauma protocols and hospital statistics. The aim of this audit was to analyse the reliability of the data in the Trauma Centre West Netherlands (TCWN) region.
Routinely registered trauma patients from all nine hospitals in the TCWN region were re-registered by a registrar for analysis.
Nine hospitals in the TCWN region in the Netherlands.
A randomly selected representative trauma population sample of 350 patients and a sample of 100 polytrauma patients were re-registered and used for analysis.
Re-registration of trauma patients in the Trauma Registry.
The inter-rater agreement on Injury Severity Score (ISS), number of Abbreviated Injury Scale (AIS) codes, identical codes and survival status were analysed using Kappa's coefficient and intraclass correlation coefficients.
The inter-rater agreement on ISS and number of AIS codes were, respectively, almost perfect (ICC = 0.81) and substantial (ICC = 0.76) in the trauma population sample, and substantial (ICC = 0.70) and fair (ICC = 0.33) in the polytrauma sample. For patients with serious injuries (AIS ≥ 2) in the population sample, the inter-rater agreement on ISS (ICC = 0.87) and number of AIS codes (ICC = 0.84) were almost perfect.
These results confirm that the Dutch regional registry system works well and may serve as a reliable basis for prospective analysis of national and international trauma care. Particular attention should be paid to the coding of polytrauma patients as discrepancies are more likely to occur in this group.
Introduction
Research on the quality of trauma registries is scarce. Accurate and consistent registration of trauma cases is essential for evaluating injury management and trauma protocols, and for research [1–4]. Correct registration must be maintained since an unreliable registry can lead to misleading hospital statistics at both regional and national levels [5, 6]. However, maintaining a high level of accuracy and consistency within the registry poses a challenge when many people are involved in the registration process.
Discrepancies in trauma registration have been reported due to various determinants such as misinterpretation of the physician's notes by the registrar and a possible lack of the registrar's basic medical knowledge [1, 2, 5, 7–9]. Although registrars may be experienced and trained, trauma surgeons are more capable of accurately interpreting clinical data due to their training and clinical experience [2].
The Dutch National Trauma Registry (NTR) includes data supplied by all 11 trauma regions in the Netherlands. The NTR registry is based on the Major Trauma Outcome Study (MTOS) with additional pre-hospital data [10].
In the registry, traumatic injuries are coded according to the Abbreviated Injury Scale (AIS) system, enabling physicians and researchers to compare injuries by severity. In 1998 updated 1990 Revision of the AIS included up to 1300 injury descriptions for both blunt and penetrating injuries [11]. The AIS system is used for trauma registration worldwide and allows for (inter)national comparisons of hospital performance in handling trauma cases.
The aim of this audit was to evaluate the reliability of the registration of general trauma cases and specifically polytrauma patients admitted to hospitals in trauma region West in the Netherlands. The outcomes of this research will help to further improve the registration quality and therefore quality of care for trauma patients.
Methods
Trauma registry
The trauma region of the Trauma Centre West Netherlands (TCWN) includes nine hospitals: three Level 1 trauma centres, three Level 2 and three Level 3 trauma centres [12]. According to regional guidelines, all severely injured patients in the region are primarily treated in one of the three Level 1 hospitals.
Each hospital has its own registrar who registers trauma patients in the NTR. New trauma registrars are trained in coding the medical chart data according to the AIS system in the online NTR database. During registration, registrars are encouraged to consult senior trauma surgeons on cases that they find unclear.
According to the inclusion criteria for the NTR, all trauma patients admitted to hospital after evaluation in the Emergency Department (ED) within 48 h after trauma are documented in the trauma registry. Trauma patients who died in the ED and those transferred from another hospital within 48 h after trauma are also documented in the registry. Not included are patients who are electively admitted to hospital for operative treatment after trauma, admitted for reasons other than their traumatic injury or discharged after being evaluated in the ED.
All injuries of included patients are coded according to the AIS classification system [11]. The last digit of each AIS code reflects the injury severity ranging from minor (Severity 1) to major (Severity 6). Injuries with AIS severity codes of 3 or higher are considered serious injuries. The AIS severity codes are used to calculate the overall Injury Severity Score (ISS). This is defined as the sum of squares of the highest AIS severity code of the three most severely injured AIS body regions. Patients with ISS ≥ 16 are considered to be polytrauma patients.
Audit
Patient selection
From the 4100 patients entered into the regional trauma registry database in the first 6 months of 2013, 2 samples were randomly drawn without replacement.
A representative sample of the entire registered trauma population (‘population sample’, n = 350). From each hospital, a proportionate sample was drawn to reflect the actual distribution of patients within the region.
A representative sample of polytrauma patients (ISS ≥ 16) admitted to the three Level 1 hospitals (‘polytrauma sample’, n = 100).
For each sample, a minimum of 100 patients was considered adequate to obtain reliability estimates with sufficient precision. Based on a minimally acceptable level of reliability (ICC 0.61 = substantial agreement) and using the formula derived by Walter et al., we calculated that we could demonstrate an ICC of at least 0.75 in 100 observations made by two independent observers [13]. In both samples, patients who did not meet the inclusion criteria of the trauma registry, as stated in the ‘Methods’, were excluded and replaced by other randomly selected patients.
Study procedure
The MTOS required data and pre-hospital data were registered [10], including patient characteristics (sex, age, discharge status), injury type, cause of injury, mode of transportation to the hospital and vital parameters in the pre-hospital and ED setting. The AIS codes registered were determined according to the injuries described in the patients’ charts by the physicians involved in the treatment in the ED and during hospitalization. Radiology reports and operation records were also used to find and check injuries sustained.
For this audit, the initial registration of the selected patients was compared with a second, blinded re-registration performed by the Audit registrar. The Audit registrar was familiar with the coding system and considered highly experienced having worked for 3 years as a registrar prior to January 2013.
The re-registrations by the Audit registrar were entered into a separate database with the same structure as the actual trauma registry database.
Investigated parameters
The parameters investigated in this audit were as follows: the presence of polytrauma (in the population sample), ISS, the total number of registered AIS codes per patient and the number of patients with identical AIS codes, and in-hospital mortality.
The severity of injuries may influence the reliability of the registration. Registrars may choose to omit superficial injuries as they have little effect on the overall ISS. Also, different codes can be given for the same superficial injuries. To analyse whether superficial injuries affect the reliability of the registration, patients from the population sample with only superficial or minor injuries (AIS 1 and 2) were removed. The reliability of the ISS and number of AIS codes registered were then analysed again.
Statistical analysis
The population sample and polytrauma sample were analysed separately. The data of the re-registrations were compared with the data of the original registrations.
Cohen's kappa coefficient (κ) was used to determine the inter-rater agreement on the registration of patient survival and the presence of polytrauma in the population sample. The intraclass correlation coefficient (ICC) was used to assess the inter-rater agreement on the ISS and the number of AIS codes registered. The degree of agreement was interpreted according to the classification by Landis and Koch: indicating values 0.01–0.20 as slight agreement, 0.21–0.40 as fair agreement, 0.41–0.60 as moderate agreement, 0.61–0.80 as substantial agreement and above 0.80 as almost perfect agreement [14].
The difference in numbers of AIS codes registered by the registrars was calculated per case by subtracting the number of codes given by the Audit registrar from the number of codes initially documented. The difference in ISS was calculated in the same way. Systematic differences for the number of AIS codes and ISS between registrars were evaluated using the paired t-test.
The analysis was performed using SPSS version 20.
Results
In the audit 35 of the 350 patients selected for the population sample and 1 patient in the polytrauma sample did not meet the inclusion criteria of the Trauma Registry. Of one other patient, the medical records could not be retrieved. These patients were excluded from the audit and replaced by 37 other randomly selected patients from the original database. Table 1 presents the details for exclusion. The baseline characteristics of the population sample and polytrauma sample are presented in Table 2.
Reasons for exclusion from the audit
| Reason for exclusion | No. of patients |
|---|---|
| >48 h between trauma and presentation to an emergency department | 16 |
| Admittance to a non-surgical department for non-trauma related reasons | 9 |
| No clear trauma mechanism | 2 |
| No traumatic injury | 6 |
| Not admitted to hospital | 3 |
| No patient record found | 1 |
| Total | 37 |
| Reason for exclusion | No. of patients |
|---|---|
| >48 h between trauma and presentation to an emergency department | 16 |
| Admittance to a non-surgical department for non-trauma related reasons | 9 |
| No clear trauma mechanism | 2 |
| No traumatic injury | 6 |
| Not admitted to hospital | 3 |
| No patient record found | 1 |
| Total | 37 |
Reasons for exclusion from the audit
| Reason for exclusion | No. of patients |
|---|---|
| >48 h between trauma and presentation to an emergency department | 16 |
| Admittance to a non-surgical department for non-trauma related reasons | 9 |
| No clear trauma mechanism | 2 |
| No traumatic injury | 6 |
| Not admitted to hospital | 3 |
| No patient record found | 1 |
| Total | 37 |
| Reason for exclusion | No. of patients |
|---|---|
| >48 h between trauma and presentation to an emergency department | 16 |
| Admittance to a non-surgical department for non-trauma related reasons | 9 |
| No clear trauma mechanism | 2 |
| No traumatic injury | 6 |
| Not admitted to hospital | 3 |
| No patient record found | 1 |
| Total | 37 |
Baseline characteristics of study samples
| Population sample (n = 350) | Polytrauma sample (n = 100) | |
|---|---|---|
| Age (years), median (IQR)a | 62 (30–82) | 61 (43–76) |
| Male gender, n (%) | 164 (47) | 64 (64) |
| Blunt trauma mechanism, n (%) | 337 (96) | 98 (98) |
| Length of hospital stay (days), median (IQR)a | 3 (2–7) | 7 (3–12) |
| Admission to ICU, n (%) | 19 (5) | 46 (46) |
| Length of ICU stay (days), median (IQR)a | 2 (1–3) | 2 (1–3) |
| Population sample (n = 350) | Polytrauma sample (n = 100) | |
|---|---|---|
| Age (years), median (IQR)a | 62 (30–82) | 61 (43–76) |
| Male gender, n (%) | 164 (47) | 64 (64) |
| Blunt trauma mechanism, n (%) | 337 (96) | 98 (98) |
| Length of hospital stay (days), median (IQR)a | 3 (2–7) | 7 (3–12) |
| Admission to ICU, n (%) | 19 (5) | 46 (46) |
| Length of ICU stay (days), median (IQR)a | 2 (1–3) | 2 (1–3) |
aIQR = interquartile range.
Baseline characteristics of study samples
| Population sample (n = 350) | Polytrauma sample (n = 100) | |
|---|---|---|
| Age (years), median (IQR)a | 62 (30–82) | 61 (43–76) |
| Male gender, n (%) | 164 (47) | 64 (64) |
| Blunt trauma mechanism, n (%) | 337 (96) | 98 (98) |
| Length of hospital stay (days), median (IQR)a | 3 (2–7) | 7 (3–12) |
| Admission to ICU, n (%) | 19 (5) | 46 (46) |
| Length of ICU stay (days), median (IQR)a | 2 (1–3) | 2 (1–3) |
| Population sample (n = 350) | Polytrauma sample (n = 100) | |
|---|---|---|
| Age (years), median (IQR)a | 62 (30–82) | 61 (43–76) |
| Male gender, n (%) | 164 (47) | 64 (64) |
| Blunt trauma mechanism, n (%) | 337 (96) | 98 (98) |
| Length of hospital stay (days), median (IQR)a | 3 (2–7) | 7 (3–12) |
| Admission to ICU, n (%) | 19 (5) | 46 (46) |
| Length of ICU stay (days), median (IQR)a | 2 (1–3) | 2 (1–3) |
aIQR = interquartile range.
The number of patients analysed per hospital varied between 3 and 55 reflecting the actual distribution of patients in the regional hospitals. In the population sample (n = 350), the Initial registrar registered 18 patients as polytrauma patients. The Audit registrar registered 16 polytrauma patients in the same sample (Table 3). The agreement on the presence of polytrauma between the registrars was substantial (κ = 0.75, 95% confidence interval [CI] 0.67–0.83).
Agreement on the presence of polytrauma between the Initial and Audit registrars in the population sample (n = 350), κ = 0.75
| Polytrauma according to audit registrar | ||||
|---|---|---|---|---|
| No | Yes | Total | ||
| Polytrauma according to Initial registrar | No | 338 | 4 | 342 |
| Yes | 6 | 12 | 18 | |
| Total | 344 | 16 | 350 | |
| Polytrauma according to audit registrar | ||||
|---|---|---|---|---|
| No | Yes | Total | ||
| Polytrauma according to Initial registrar | No | 338 | 4 | 342 |
| Yes | 6 | 12 | 18 | |
| Total | 344 | 16 | 350 | |
Agreement on the presence of polytrauma between the Initial and Audit registrars in the population sample (n = 350), κ = 0.75
| Polytrauma according to audit registrar | ||||
|---|---|---|---|---|
| No | Yes | Total | ||
| Polytrauma according to Initial registrar | No | 338 | 4 | 342 |
| Yes | 6 | 12 | 18 | |
| Total | 344 | 16 | 350 | |
| Polytrauma according to audit registrar | ||||
|---|---|---|---|---|
| No | Yes | Total | ||
| Polytrauma according to Initial registrar | No | 338 | 4 | 342 |
| Yes | 6 | 12 | 18 | |
| Total | 344 | 16 | 350 | |
Injury severity score
Scatterplots of the ISS according to the initial registrar (x-axis) and the audit registrar (y-axis) in the population sample (A) and polytrauma sample (B).
The inter-rater agreement on ISS in the polytrauma sample was fair (ICC = 0.33, 95% CI 0.14–0.50; Fig. 1B). In 35 cases (35%), the ISS score given by both registrars was identical. The difference in ISS between the registrars ranged from −14 to +59 points. The ISS values of the Initial registrar were, on average, 3.7 points (95% CI 2.0–5.4) higher than those assigned by the Audit registrar (P < 0.001).
In one case, a difference of 59 points was seen between the registrars. The Initial registrar registered a cervical cord laceration, a severity score of 6. Injuries with a severity of 6 receive an ISS of 75 indicating a non-survivable injury. The Audit registrar registered a cervical cord contusion, injury severity of 4. Excluding this one patient from the analysis led to a moderate inter-rater agreement on ISS (ICC = 0.47, 95% CI 0.26–0.64) and a range of differences from −14 to 24.
Number of AIS codes
Distribution of the difference in number of registered AIS codes between the registrars (Initial registrar–Audit registrar) in the population sample (A) and the polytrauma sample (B).
Patients with severe injuries
After exclusion of patients with only minor injuries (AIS = 1 or 2), 146 cases remained in the population sample. Compared with the complete population sample (ICC 0.81), the inter-rater agreement on ISS increased to almost perfect (ICC = 0.87, 95% CI 0.83–0.91). In 110 (75%) of the cases without only minor injuries, there was no difference in the number of codes registered. Compared with the complete population sample (ICC 0.76), the inter-rater agreement on the number of AIS codes increased to almost perfect (ICC = 0.84, 95% CI 0.78–0.88).
All cases within the polytrauma sample had AIS codes of 3 and higher only, as registered by both registrars. Of the 100 cases, 12 had identical injuries registered by both registrars.
Survival status
The registrars documented the same survival status in 349 of the 350 patients in the population sample (99.7%). The reliability of the survival status was almost perfect (κ = 0.91). Agreement on survival status in the polytrauma sample was perfect (κ = 1).
Discussion
The aim of this study was to analyse the reliability of the data in the trauma registry of the TCWN, paying special attention to the polytrauma patients (ISS ≥ 16). The agreement on ISS between registrars was almost perfect in the population sample, but only fair in the sample of polytrauma patients. Agreement on the presence of polytrauma in the population sample was substantial.
Documented ISS values were less reliable in the polytrauma sample than in the general population sample (ICC = 0.33 vs. ICC = 0.81). Even after the removal of one extreme outlier with a 59-point difference in ISS between the registrars, the inter-rater agreement on ISS in the polytrauma sample was only moderate (ICC = 0.47). This difference in reliability may be due to there being fewer injuries coded per patient in the population sample. On average, there were 1.3 codes registered per patient in the population sample compared with an average of 4 codes per patient in the polytrauma sample. This difference leads to a higher chance of inconsistent registration. Also, the complexity of injuries sustained by polytrauma patients may lead to discrepancies in registration between registrars. Misinterpretation of injuries noted in patient charts by the registrar can also lead to differences in the registration of injuries and their severity.
Another source for discrepancies could be the unawareness of all injuries sustained after trauma. During hospitalization, other injuries that were first overlooked in the ED may become apparent, or the severity of an injury may alter due to further diagnostics.
Overlooking or not including superficial injuries in the trauma registry has only a slight effect on the inter-rater agreement on ISS. It is, however, important to include minor injuries in the trauma registry so that a complete picture of the trauma patient is portrayed.
Comparing the ISS between the registrars showed that in 71 cases (20.2%) in the population sample, there was a difference in ISS of 4 or more points. This corresponds with an injury severity code of 2 or more. The addition or omission of such injuries can lead to differences in the polytrauma status of a patient between registrars.
Another Dutch study by Olthof et al. analysed similar parameters in the trauma registry in the Amsterdam region using the same selection criteria [5]. This enables the comparison of registration reliability in two regions in the Netherlands. The inter-rater agreement on registration of ISS was similar in both studies (ICC = 0.81 and 0.84, absolute agreement 58% and 63%). The inter-rater agreement on the number of AIS codes registered was substantial in both studies, with an absolute agreement of 62% and 63%. Also, the inter-rater reliability of the survival status agreement was almost perfect in both studies (Cohen's κ = 0.91 vs. κ = 0.82, absolute agreement 99% for both). From these similar results in two trauma region studies, it can be concluded that the current methods of data registration in the Dutch NTR work well.
Strengths and limitations of this study
Unlike other studies on this topic, we only used one registrar to re-register the cases. This was done to eliminate additional inter-rater variation that could occur within the re-registration process [3, 4]. It was believed that this registrar would perform the re-registration to the high standard needed for this research. However, we do realize that using only one registrar for re-registration may have introduced some bias.
Upon hospitalization, patients’ injuries can initially be misdiagnosed by one of the attending doctors. To prevent that incorrect diagnoses are recorded in the trauma registry, injuries reported in the medical charts are compared with radiology conclusions, operation reports and/or changes in diagnoses made by other physicians. Also in the registration process, a trauma surgeon is consulted to give an expert opinion when a diagnosis is unclear.
Recommendations for improving the quality of trauma registries
Based on the results of our study, recommendations can be given for organizing trauma registries in such a way that they are accurate and consistent. These recommendations are of special interest for countries where no trauma registry is in place or in the early stages of development.
In order to eliminate coding bias and allow for international comparison, similar or the same coding systems should be used [15]. Ideally, the physicians and trauma surgeons involved in the care of trauma patients should be the ones to select the patients for registration and register their injuries, as they are more likely to accurately register the injuries [5]. However, this increase in workload for the trauma surgeons probably renders this a non-feasible option. The second-best option is that the coding of trauma patients’ injuries in the registry is supervised by a trauma surgeon. In our trauma region, the additional confirmation of injuries, specifically for the group of polytrauma patients, by a trauma surgeon, was introduced after this audit. To obtain a more accurate registration in the future, registrars could be educated further and receive training at regular intervals to perform to a higher standard. Research has shown that tests on knowledge and accurate registration of injuries through training programmes can improve the registration [16]. This could be implemented in the future in order to monitor registration quality and identify points for improvement. Together with an increased collaboration with the trauma surgeons, this should enhance the standard of registration.
Finally, trauma registries may also be used as part of programmes for monitoring and improving patient safety and quality of trauma care. For this purpose, data on quality indicators should be routinely collected in a consistent and reliable manner in the registry [15, 17, 18].
Conclusion
In conclusion, this study shows that the way in which the trauma registry in the Netherlands is organized leads to a high standard of registration. This conclusion is strengthened by similar results from a previous study of another regional Dutch registry. Recommendations for further improvement are that registrars should strictly adhere to the inclusion and exclusion criteria and that trauma surgeons should be involved in the registration, especially of polytrauma patients, as discrepancies are more likely to occur in this group. Registrars should be educated and receive regular training in order to maintain a high standard of trauma registry.
