CEO SUMMARY: Researchers with the COVID-19 Testing Project used a multidisciplinary effort to analyze and compare the performance of 12 serological tests. One finding is that, 16 to 20 days or more after a confirmed infection with a molecular test, many of the 12 serological tests analyzed were 80% positive. Using donated blood samples, the researchers assessed the volume and number of antibodies the SARS-CoV-2 produced in infected patients.
FROM MID-MARCH UNTIL MAY 4, manufacturers launched more than 200 serological tests to identify the antibodies patients produce in response to infection from the new coronavirus.
The performance of these tests was mostly unknown because the federal Food and Drug Administration (FDA)—under the rules it issued on March 16—allowed almost all of these 200 tests on the market without the usual review. Without performance data except from the manufacturers themselves, clinical laboratories followed the caveat emptor principal of buyer beware when choosing among these tests for clinical use.
The FDA reviewed the performance of only a select few of the tests that applied for the FDA’s emergency use authorizations (EUAs) before May 4, when the FDA issued new guidance for serological tests for SARSCoV-2, the virus that causes the COVID-19 illness.
Seeking to fill the gap in performance review, a team of researchers analyzed the performance of 12 of the serology tests in a significant effort to do a comparative review to benefit clinical laboratories, public health officials, patients, and clinicians.
The researchers provided a significant contribution, because the FDA’s March 16 policy for serological tests provided what the agency called “regulatory flexibility for developers offering such tests without FDA review and without an EUA.” Test manufacturers were required to simply notify the agency that they had validated their tests, and they needed to provide disclaimers about the limitations of the tests.
Additionally, the federal agency added that, “The FDA does not review the validation, or accuracy, of data for these tests unless an EUA is submitted.”
Emergency Use Authorization
As of May 7, the FDA listed the performance data for only 12 serological assays that had received EUAs and yet, according to published reports, the agency’s website listed 200 or more serological assays that were approved for sale in the United States.
The regulations on March 16 were intended to make it faster and easier for test manufacturers to bring tests to market to track the SARS-CoV-2 pandemic. As a result of requiring little in terms of supporting performance data, test manufacturers flooded the market with tests, some of which performed poorly, produced inaccurate results, or were fraudulent.
In response, on May 4 the FDA issued new rules requiring all companies seeking to sell serological tests in the United States to apply for EUAs. According to a review of FDA data on May 8, 116 were no longer authorized and 12 had received EUAs. The FDA site made no mention of what became of the other antibody tests, suggesting that the manufacturers may have withdrawn them from the market. (See “FDA Replaces March 16 Serology COVID-19 Test Rules” in this issue.)
Researchers Step Up
In the seven weeks between the FDA’s March 16 decision and the May 4 revised policy on antibody tests, there was no accepted, industrywide standard for a high-quality COVID-19 serological assay for labs and pathologists to use for comparison. Therefore, the team of researchers from two universities and two bioengineering companies in California decided to develop a way to compare the performance of 12 serological tests on offer.
In April, the researchers published their findings online. The lack of performance data had created a pressing need for an independent body to produce a consistent method to evaluate the performance of serological tests for the SARS-CoV-2 infection, said Patrick D. Hsu, PhD, an Assistant Professor and Faculty Fellow at the University of California, Berkeley. A bioengineer and geneticist who normally works in genome editing, Hsu is one of the researchers leading the multidisciplinary COVID-19 Testing Project (https://covidtestingproject.org).
The researchers involved in the project represent UC Berkeley, the University of California San Francisco, the Chan Zuckerberg Biohub, and the Innovative Genomics Institute.
Given the urgent need for comparative data on the assays, the researchers performed head-to-head comparisons of commercially-available lateral flow assays (also known as rapid serology tests) and enzyme-linked immunosorbent assays (ELISAs), the researchers wrote on the web.
“We do not have a gold standard for evaluating serological tests, and that’s something we look for in modern medicine,” Hsu commented in an interview with The Dark Report. “Therefore,” he continued, “we conducted this study by taking in case samples from PCR-positive patients that have been seen in San Francisco hospitals.”
Hospital Patients’ Specimens
Upon admission, those patients were given the reverse transcription-polymerase chain reaction (RT-PCR) test. Over the course of their hospital stay, blood from these patients was drawn multiple times for follow-up testing and the researchers used those samples in their analysis.
The results of the research could inform healthcare providers and public health and government officials conducting serological testing for the new coronavirus, Hsu explained. “To power our analysis adequately, we used different time intervals and then placed all the samples into five-day bins,” Hsu explained. “This is a question of biology, because we have patients who were seroconverted when their antibodies were expected to rise.
“We know from past research that the IgM antibody is generally thought to rise first, whereas IgG antibodies can take more time while the immune system revs up,” noted Hsu. “But without making any assumptions about what type of antibodies would come first and which type of tests might be more sensitive, we wanted to compare all of the assays head-to-head in a very systematic fashion.
“To do that for each of the time-point intervals for every specimen, we tested the performance of each one against the lateral flow assays (LFAs) and against two ELISAs.
LFA versus ELISA Bake-Off
“For each specimen, we wanted to evaluate how the tests performed, collect that data, and then compare the results,” he said. “That way we could see what comes from this bake-off.
“Generally what we found is that, at 16 to 20 days, or at more than 20 days, many of these tests were over 80% positive,” Hsu explained. “What we don’t know, however, is whether that 80% positive rate should be 100% positive. All we know is that these samples were PCR positive before we ran our analysis.
“We don’t necessarily know when antibodies will rise and if the antibodies should all have developed to show a rate of 100% positive after three weeks,” observed Hsu. “We certainly expect samples from certain patients not to have seroconverted because some might be immunosuppressed, for example. When we reviewed the patient data, we decided that assumption might be correct.
“Also, we had a relatively small sample size of patients who were still hospitalized after 20 days,” he recalled. “That makes sense intuitively, because after that many days, these patients would likely have been released because they were from an ambulatory population.”
Adding Assays to the Study
During the interview, Hsu said he and his colleagues were considering expanding the sample set beyond the initial 12 COVID-19 serological assays analyzed. At press time, it was not known if the researchers would continue after the FDA issued new rules on May 4. After the initial interview, Hsu did not respond to questions from The Dark Report.
“We released our findings as a preprint so that we could report our preliminary results in advance of a formal peer review,” he said during the interview on May 2. “We did that because we saw the urgency of the situation and the number of tests that were fraudulent but on the market.” If they expand the sample set, the researchers may want to add data from later time points.
Although antibody tests have an important role to play during the pandemic, treating physicians may want to confirm a positive antibody test with another antibody test, he suggested.
“You could confirm results with a different type of serology test that might look at a different antibody isotype or a different type of antigen,” Hsu explained. “You could look at IgM verses IgG versus IgA. Or you could look for reactivity against proteins in the virus. That type of confirmatory testing could help improve significantly the specificity of a result by looking across multiple different antigens or antibody isotypes.”
From the pre-print manuscript of their findings, the researchers have posted the results of the performance analysis of the 12 assays on the web, and they are interested in adding to the data as they evaluate more tests, Hsu said.
Evaluating More Tests
Some lab companies that make COVID-19 serological tests were interested in adding their tests to the dataset, he said. “There has been an outpouring of interest from many manufacturers or distributors that want their tests evaluated,” Hsu commented. “We’re trying to figure out the best way to prioritize those requests because there are many more tests that we could evaluate, but we would need more blood samples from infected patients.
“As researchers in this new field, we also want to understand the correspondence between venipuncture serum—which is how our study was conducted— with blood from fingerstick sample collection,” commented Hsu. “Plus, we want to increase the number of samples that we have in the later time points from symptom onset.
“Our goal is not to exhaustively evaluate every test—in part because that might not be possible,” he added. “And, this work is not what most of us do normally. A lot of the scientists working on this project are repurposing themselves to add to the response among scientists to address this new coronavirus. We want to highlight this issue and perhaps inspire others to scale up this work to do more test-performance evaluations.
“At the least, we hope that some regulatory bodies will review our data,” concluded Hsu. “Those would be very good outcomes for our work.”
At a time when large numbers of COVID-19 serological tests are coming into the market, but without the comparable range of data provided with diagnostic assays that are reviewed by the FDA and cleared for use, the work of the COVID-19 Testing Project is a valuable resource for clinical laboratories.
In Manuscript for Peer Review, Researchers Describe Evaluation of COVID-19 Serological Tests
IN A MANUSCRIPT PREPARED LAST MONTH FOR PUBLICATION IN A PEER-REVIEWED JOURNAL, researchers from the COVID-19 Testing Project described the steps they followed to evaluate 12 serological assays.
The researchers collected 130 samples of blood from 80 individuals who had molecular tests that confirmed infections from the SARS-CoV-2 virus. They also had 108 blood samples from another group of patients and those specimens were pre-COVID-19 samples collected from American Red Cross blood donors.
The analysis shows that the 12 tests performed relatively well in identifying IgM, IgG, or both antibodies. “For each test, we quantified detection of IgM and/or IgG antibodies by time period from onset of symptoms and assessed specificity and cross-reactivity,” the researchers wrote in the pre-publication manuscript. For their analysis, the researchers tested the patients’ specimens over five time periods: one-to-five days, six-to-10 days, 11-to-15 days, 16-to-20 days, and more than 20 days.
“This study also seeks to provide feedback to manufacturers about areas of success and necessary improvement,” the researchers explained. That feedback would be useful given that the extent and time to development of antibodies are not fully understood and may vary among all patients, even those who have RT-PCR confirmed cases, they added.
Rather than report the sensitivity of each assay, the researchers compared the percent positivity rate by time interval, in part because the percent positivity rate rose over time after the onset of symptoms. One important finding was that high rates of positive results were not reached until at least two weeks after the onset of clinical symptoms. Therefore, “diagnosis at time of symptom onset thus remains dependent on viral detection methods.”
Another important result is that for patients with more severe levels of illness, the 12 assays showed a trend to higher positive rates within time intervals. But the researchers urged caution when interpreting this finding because they had limited data from ambulatory patients.
“The majority of samples evaluated after 20 days post-symptom onset had detectable anti-SARS-CoV-2 antibodies, suggesting good to excellent sensitivity for all evaluated tests in hospitalized patients three or more weeks into their disease course,” they wrote.
More research is needed on ambulatory or asymptomatic patients to guide the appropriate use of serological testing and it’s important to note that researchers do not know the extent to which positive serology-test results reflect a protective immune response, they wrote.
To ensure consistent and meaningful results, Hsu and colleagues recognized that patients tend to respond differently when infected with the virus. Many are asymptomatic initially and may not produce antibodies for many days or several weeks, he said.
Research Highlights about COVID-19 Serology Tests
FOR MOST OF THE TESTS EVALUATED IN THE COVID-19 TESTING PROJECT, the research showed specificity greater than 95% for most of the tests and specificity of more than 99% for two lateral flow assays (LFAs) and for one of the two enzyme-linked immunosorbent assays (ELISAs), the researchers reported.
There were two ELISAs (Epitope and an in-house developed test) and 10 immunochromatographic lateral flow assays:
- Deep Blue
- Sure Check
Researchers provided data on each assay showing the percentage of specificity for each antibody (IgM, IgG, or both) and the percentage of positive results for each antibody at each of the five time intervals (one-to-five days, six-to-10 days, 11-to-15 days, 16-to-20 days, and more than 20 days.
Contact Patrick Hsu, MD, at firstname.lastname@example.org.