RO-ILS: Radiation Oncology Incident Learning System® releases regular reports and education to the radiation oncology community based on safety events and errors submitted by participating U.S. practices. Aggregate data trends held steady over the past eight years, with expected fluctuations associated with data element changes (e.g., when a data element became required, new answer options were added). That was, until now. The RO-ILS Q4 2021 Aggregate Data Report indicates a striking new trend in 2021 related to the workflow step in which events occurred.
Between 2017 and 2020, when data element 208. Occurred Workflow was required, event occurrence during treatment delivery (including imaging) has traditionally comprised an average of 26% (Figure 1). However, for 2021 that number was 41%. This represents a 58% increase over historical norms. Fewer events were seen to occur in treatment planning and pre-treatment quality assurance (QA) and review, whereas other domains remained stable relative to historical norms.
Figure 1:
Unsurprisingly, the downstream metric for where the event was discovered (207. Discovered_Workflow) similarly shifted to the right, with 29% of events discovered at on-treatment QA compared to a historical average of 11% (Figure 2).
Why are we seeing this change in RO-ILS data? Did more events actually occur at the linear accelerator (linac) and if so, why? Did fewer errors happen during treatment planning and why might that be the case? Or are there other factors that impact the data?
It is important to understand there are two basic interpretations of this finding: that this is real, or that this is artifactual. Perhaps there really is a shift in where in the workflow errors occurred in 2021 (e.g., more events happened at treatment delivery). Conversely, it could be artifactual — that the events occurring at treatment delivery are stagnant in number, but that we are seeing a relative percentage increase in these events due to selective non-reporting of more upstream events, a risk of any voluntary incident reporting program. Although more analysis is needed, we have some initial thoughts as members of the Radiation Oncology Healthcare Advisory Council (RO-HAC) who review and analyze RO-ILS data.
First, let's consider interpretation 1: More events are happening at treatment delivery, fewer events at other workflow steps. This real trend could either be a positive or negative indicator of safety.
Let’s be optimistic and consider that this is a real trend and is actually a good thing. Perhaps advancement in technology may explain this phenomenon. For example, with the rise of new equipment like surface guided radiation therapy (SGRT), practices may be better able to detect errors occurring during treatment that were otherwise opaque before. SGRT can be very helpful in detecting the improper application of shifts, which has been demonstrated in the RO-ILS data to be a major pathway of significant errors. Additionally, there has been a move toward increased automation of treatment planning processes, which may result in a reduction in errors occurring earlier in the process of care. If enough practices acquired or implemented SGRT and/or knowledge-based planning tools between 2020 and 2021, then this could result in a shift in where in the process events are occurring.
How about the impact of the pandemic? That has certainly brought about a dramatic shift in a short period of time. COVID-19 affected the ability of certain staff members to work from home initially, which then brought on a wave of employees seeking work from home or hybrid arrangements as part of routine practice. As treatment planning can be done remotely, this may have altered the work environment for dosimetrists more than for other members of the radiation oncology team. Are at-home work environments for dosimetrists conducive to higher safety standards, leading to fewer errors in treatment planning?
Importantly, incident learning systems, including RO-ILS, encourage reporting. More events can indicate heightened awareness within that process of care, a strong culture of safety and the desire to address issues, no matter how trivial. Technology advancements such as adaptive radiation therapy are increasing the complexity of treatment delivery. With the adoption of new systems, an increase of events is expected as practices work to perfect processes. More events within a given process could result in an increased detection of errors that otherwise would have been hidden or unknown.
Now, let us consider that this trend is real, but that it’s actually a bad thing. Multiple waves of COVID-19 in 2021 left many practices with significant numbers of staff out at one time for various reasons (e.g., personal, family illness, quarantine). This lack could have resulted in temporary deviations in the staffing policies of two radiation therapists per linac or fewer on-site physician or physics staff available to draw upon for guidance with a challenging setup. COVID-19 had a heightened impact on radiation therapists, as they were required to continuously support treatment delivery on site, while other professional roles may have been able to limit their public and workplace exposures for at least some portion of time. The compounded stress may have impacted performance, team relations and the work environment. This could result in COVID-19’s influence on safety being greater in the treatment delivery space than in the treatment planning space.
Now, let's consider interpretation 2: The number of events happening at treatment delivery is stable, but there are other confounding factors that skew the data.
In the United States, incident learning is voluntary, a crux of the legal protections afforded by the Patient Safety and Quality Improvement Act of 2005 and through participation in patient safety organizations (PSO). Estimates are that voluntary reporting only accounts for approximately 10% of the events that actually occur. Therefore, we must be careful when analyzing the data to ensure we appreciate this limitation. If PSO protections were not present, we would see even fewer of the events that occur, so it is essential that the anonymous, confidential work of the PSO remains supported.
One can imagine that the radiation oncology workforce is tired. With so many constraints on their time, in the second year of the pandemic, incident learning may have become a lower priority. RO-ILS recommends reporting any error that passes through the first checkpoint, regardless of whether it reaches the patient or is caught further downstream at a later safety check. However, we know not all practices are able to do this and that not all staff work in a culture of safety that supports this.
Although the number of events reported to RO-ILS has been fairly stable, the incident learning system has been functioning in an environment with staffing shortages, disrupted processes and individuals dealing with the mental task load of the pandemic. It is plausible that events occurring during treatment planning and on-treatment QA, which can be caught by normal QA, are deemed less of a priority to report than events happening at the linac. Therefore, with an underreporting of treatment planning and on-treatment QA occurring events, the relative percentage of events occurring at the linac would appear to rise.
In addition to practice-specific reporting thresholds, it is possible that changes in reporting patterns among RO-ILS enrolled practices have impacted the overall national trends. The extent to which some practices are reporting to the PSO cannot be ruled out as a cause of this trend.
What do you think? Which interpretation is more plausible to you? Importantly, are you seeing the same trends in your local practice’s incident learning data? As an individual practice, with knowledge of your practice-specific circumstances (e.g., staff, technology advancements, culture) the meaning of incident learning trends can be better postulated.
With support from the PSO, RO-HAC will dig deeper into the RO-ILS data to try to better understand possible contributing factors to this national trend. In the meantime, why do you think we are seeing this shift in the national data? Please comment below with your observations and thoughts.
