By Susan M. Dallabrida, Ph.D. VP, Clinical Science & Consulting Services, ERT
Scientific rationale for optimizing electronic clinical outcome assessments in clinical trials
Capturing clinical trial assessments electronically provides higher quality and more reliable and accurate datasets than capturing this data on paper. Recognition of such improvement in data quality has resulted in an increased adoption of electronic clinical outcome assessments (eCOA).
The benefits of capturing high quality clinical trial data via eCOA has been widely recognized for more than a decade. Researchers have documented significant improvements in patient protocol compliance and data quality, a reduction of missing data and data ‘noise,’ and most importantly, increased study power with fewer patients.(1)
However, some study teams who are new to eCOA occasionally raise questions about whether to use paper forms as a back-up in case of electronic data collection failure. As a result, some study teams request that provisions for a paper back-up system be in place.
In this article, evidence will be presented illustrating that paper back-up provisions are unnecessary and do not provide the desired benefits. Paper back-ups produce little to no value and can call into question the overall quality of the clinical trial data being captured. In cases where a back-up solution is essential, alternative electronic approaches are recommended.
Electronic Device Failure? Not Likely.
Electronic device failure or device loss occurs very infrequently – historically, less than one percent of the time (2). And, technology advancements are expected to further reduce this occurrence (3). Most provisioned devices are theft-adverse, meaning they are electronically “locked down” and have no other function beyond the collection of trial data. Thus, the probability of needing any type of back-up solution is very low.
In the event that an electronic device fails, most providers have multiple back-up electronic systems in place. They also have recommendations for minimizing any statistical impact, including:
In most cases, the speed by which any device failure can be remedied with a back-up device provides an effective strategy for maintaining the consistent electronic capture of high-quality data in clinical trials.
Instances of missing data that cannot be readily addressed by a replacement electronic back-up device solution comprise a small fraction of the less than 1 percent of the total number of these cases of device failure or loss. In the case that every data point is deemed to be critical to determinations of efficacy and/or patient safety, such as primary endpoint in a Phase III study for a PRO-based labeling claim in a trial with a small patient sample, the recommended back-up solution beyond replacement devices is to have a web-based or interactive-voice response (IVR)-based electronic system in place. Using an alternate electronic data capture method will maintain high data quality should technology failures occur. Additionally, missing data can be recovered through imputation analysis. Clinical trial protocols commonly contain a statistical plan for handling missing data and what imputation analysis will be used to cover these rare potential occasions.
The quality of data collected on paper is markedly inferior to electronic data capture. Capturing clinical trial data on paper is not recommended by regulators and should be considered undesirable. Given the very limited instances in which device failure or loss occurs, the speed at which such events can be resolved by one or more of the above referenced electronic solutions, and the significantly reduced likelihood of the data collected on paper being accurate, the value of including paper as a back-up provision for any study is very limited.
The standard for the quality of data collected that is suitable for regulatory submissions have been defined by the FDA to include the following five criteria: attributable, legible, contemporaneous, original and accurate (abbreviated by the acronym ALCOA), (4). Data collected electronically maps to these criteria, whereas data collected via paper does not.
Electronic data capture additionally provides a modality that fulfills several other criteria outlined in the FDA PRO Guidance (5), including prescribed questionnaire order, required responses to items, limiting missing data, and verifiable time and date stamps. One of the most important aspects addressed via eCOA is real-time data collection that follows clinical trial design. The FDA PRO Guidance is strong in asserting that data must be collected in a contemporaneous manner:
“If a patient diary or some other form of unsupervised data entry is used, we plan to review the clinical trial protocol to determine what steps are taken to ensure that patients make entries according to the clinical trial design and not, for example, just before a clinic visit when their reports will be collected.”
At a glance, paper back-ups may seem like an easy and cost-effective solution, but this notion is incorrect. In addition to being ineffective, paper back-ups represent a costly and time-consuming approach for little to no actualized gain ― and the potential for negative impact on trial outcomes. Further, additional overhead is required for a paper back-up provision in a protocol:
In addition to these overhead costs, there is a likely tendency for paper forms to be repeatedly used for more than a paper back-up once it is introduced. Once paper forms enter a study ― even if for a legitimate instance ― there is a recognized temptation for site personnel to subsequently use paper instead of an electronic device. There is no way to disengage paper once it enters a study; sufficient amounts of paper can cloud any potential actual signal.
The quest to avoid missing data should focus on the evidence at hand, which is that eCOA devices fail, are damaged, or are lost less than 1 percent of the time. Having a back-up device strategy in place reduces this to a fraction of that one percent, so an electronic back-up system is recommended. Adding a paper back-up provision does not ensure that any fraction will be fully captured; at most half of the 1 percent are viable. Paper back-up provisions represent a strategy of diminishing returns and should be considered unnecessary.