California Research Team Analyzes Performance of 12 Serological Tests

Consistent method used to evaluate different serological tests

This is an excerpt of a 2.241-word article in the May 11, 2020 issue of THE DARK REPORT (TDR). The full article is available to members of The Dark Intelligence Group.

CEO SUMMARY: Researchers with the COVID-19 Testing Project used a multidisciplinary effort to analyze and compare the performance of 12 serological tests, and shared their results with the Dark Report. One finding is that, 16 to 20 days or more after a confirmed infection with a molecular test, many of the 12 serological tests analyzed were 80% positive. Using donated blood samples, the researchers assessed the volume and number of antibodies the SARS-CoV-2 produced in infected patients.

FROM MID-MARCH UNTIL MAY 4, manufacturers launched more than 200 serological tests to identify the antibodies patients produce in response to infection from the new coronavirus.

The performance of these tests was mostly¬†unknown because the federal Food and¬†Drug Administration (FDA)‚ÄĒunder the¬†rules it issued on March 16‚ÄĒallowed almost¬†all of these 200 tests on the market without¬†the usual review. Without performance data¬†except from the manufacturers themselves,¬†clinical laboratories followed the caveat emptor¬†principal of buyer beware when choosing¬†among these tests for clinical use.

The FDA reviewed the performance of only a select few of the tests that applied for the FDA’s emergency use authorizations (EUAs) before May 4, when the FDA issued new guidance for serological tests for SARSCoV-2, the virus that causes the COVID-19 illness.

Seeking to fill the gap in performance review, a team of researchers analyzed the performance of 12 of the serological tests in a significant effort to do a comparative review to benefit clinical laboratories, public health officials, patients, and clinicians.

The researchers provided a significant¬†contribution, because the FDA‚Äôs March 16¬†policy for serological tests provided what¬†the agency called ‚Äúregulatory flexibility¬†for developers offering such tests without¬†FDA review and without an EUA.‚ÄĚ Test¬†manufacturers were required to simply¬†notify the agency that they had validated¬†their tests, and they needed to provide disclaimers¬†about the limitations of the tests.

Additionally, the federal agency added that,¬†‚ÄúThe FDA does not review the validation,¬†or accuracy, of data for these tests unless an¬†EUA is submitted.‚ÄĚ

Emergency Use Authorization

As of May 7, the FDA listed the performance data for only 12 serological assays that had received EUAs and yet, according to published reports, the agency’s website listed 200 or more serological assays that were approved for sale in the United States.

The regulations on March 16 were intended to make it faster and easier for test manufacturers to bring tests to market to track the SARS-CoV-2 pandemic. As a result of requiring little in terms of supporting performance data, test manufacturers flooded the market with tests, some of which performed poorly, produced inaccurate results, or were fraudulent.

In response, on May 4 the FDA issued¬†new rules requiring all companies seeking¬†to sell serological tests in the United States¬†to apply for EUAs. According to a review¬†of FDA data on May 8, 116 were no longer¬†authorized and 12 had received EUAs. The¬†FDA site made no mention of what became¬†of the other antibody tests, suggesting that¬†the manufacturers may have withdrawn them¬†from the market. (See ‚ÄúFDA Replaces March¬†16 Serology COVID-19 Test Rules‚ÄĚ in this issue.)

Researchers Step Up

In the seven weeks between the FDA’s March 16 decision and the May 4 revised policy on antibody tests, there was no accepted, industrywide standard for a high-quality COVID-19 serological test for labs and pathologists to use for comparison. Therefore, the team of researchers from two universities and two bioengineering companies in California decided to develop a way to compare the performance of 12 serological tests on offer.

In April, the researchers published their findings online. The lack of performance data had created a pressing need for an independent body to produce a consistent method to evaluate the performance of serological tests for the SARS-CoV-2 infection, said Patrick D. Hsu, PhD, an Assistant Professor and Faculty Fellow at the University of California, Berkeley. A bioengineer and geneticist who normally works in genome editing, Hsu is one of the researchers leading the multidisciplinary COVID-19 Testing Project (

The researchers involved in the project represent UC Berkeley, the University of California San Francisco, the Chan Zuckerberg Biohub, and the Innovative Genomics Institute.

Given the urgent need for comparative data on the assays, the researchers performed head-to-head comparisons of commercially-available lateral flow assays (also known as rapid serology tests) and enzyme-linked immunosorbent assays (ELISAs), the researchers wrote on the web.

‚ÄúWe do not have a gold standard for¬†evaluating serological tests, and that‚Äôs¬†something we look for in modern medicine,‚Ä̬†Hsu commented in an interview¬†with The Dark Report. ‚ÄúTherefore,‚Ä̬†he continued, ‚Äúwe conducted this study¬†by taking in case samples from PCR-positive¬†patients that have been seen in¬†San Francisco hospitals.‚ÄĚ

Hospital Patients’ Specimens

Upon admission, those patients were given the reverse transcription-polymerase chain reaction (RT-PCR) test. Over the course of their hospital stay, blood from these patients was drawn multiple times for follow-up testing and the researchers used those samples in their analysis.

The results of the research could¬†inform healthcare providers and public¬†health and government officials conducting¬†serological testing for the new coronavirus,¬†Hsu explained.¬†‚ÄúTo power our analysis adequately,¬†we used different time intervals and then¬†placed all the samples into five-day bins,‚Ä̬†Hsu explained. ‚ÄúThis is a question of biology,¬†because we have patients who were¬†seroconverted when their antibodies were¬†expected to rise.

‚ÄúWe know from past research that¬†the IgM antibody is generally thought to¬†rise first, whereas IgG antibodies can take¬†more time while the immune system revs¬†up,‚ÄĚ noted Hsu. ‚ÄúBut without making any¬†assumptions about what type of antibodies¬†would come first and which type of¬†tests might be more sensitive, we wanted¬†to compare all of the assays head-to-head¬†in a very systematic fashion.

“To do that for each of the time-point intervals for every specimen, we tested the performance of each one against the lateral flow assays (LFAs) and against two ELISAs.

LFA versus ELISA Bake-Off

‚ÄúFor each specimen, we wanted to evaluate¬†how the tests performed, collect that¬†data, and then compare the results,‚ÄĚ he¬†said. ‚ÄúThat way we could see what comes¬†from this bake-off.

‚ÄúGenerally what we found is that, at¬†16 to 20 days, or at more than 20 days,¬†many of these tests were over 80% positive,‚Ä̬†Hsu explained. ‚ÄúWhat we don‚Äôt¬†know, however, is whether that 80% positive¬†rate should be 100% positive. All¬†we know is that these samples were PCR¬†positive before we ran our analysis.

‚ÄúWe don‚Äôt necessarily know when¬†antibodies will rise and if the antibodies¬†should all have developed to show a¬†rate of 100% positive after three weeks,‚Ä̬†observed Hsu. ‚ÄúWe certainly expect samples¬†from certain patients not to have¬†seroconverted because some might be¬†immunosuppressed, for example. When¬†we reviewed the patient data, we decided¬†that assumption might be correct.

‚ÄúAlso, we had a relatively small sample¬†size of patients who were still hospitalized¬†after 20 days,‚ÄĚ he recalled. ‚ÄúThat¬†makes sense intuitively, because after that¬†many days, these patients would likely¬†have been released because they were¬†from an ambulatory population.‚ÄĚ