CEO SUMMARY: When it announced its new antibody test for the novel coronavirus SARS-CoV-2 on May 3, Roche Holdings explained that its sensitivity level after 14 days was 100%, which is an important point to understand about the test. When assessing Roche’s claims, two experts agreed that the test performs exceptionally well. Abbott Laboratories also reported high performance levels for its COVID-19 antibody test in its April 27 press release announcing the FDA’s EUA for the test.
CLINICAL LABORATORY DIRECTORS AND PATHOLOGISTS may have wondered about the validity of the claims Roche Holdings AG made on Sunday, May 3, about its new serological test to identify the presence of antibodies for SARS-CoV-2.
When Roche announced that the Food and Drug Administration (FDA) granted the Swiss biotech giant an emergency use authorization (EUA) for its Elecsys Anti- SARS-CoV-2 antibody test, the company stated that the test has a specificity greater than 99.8% and sensitivity of 100% 14 days after confirmation with polymerase chain reaction (PCR). This sensitivity level after 14 days is an important caveat.
Sensitivity and Specificity
That same day, The Wall Street Journal reported, “Roche says its serological test has proven 100% accurate at detecting COVID-19 antibodies in the blood, and 99.8% accurate at ruling out the presence of those antibodies. In other words, only two in every 1,000 samples lacking the antibodies would produce a ‘false positive’ result.”
When assessing the company’s claim, two experts in clinical lab testing pushed back slightly on the news coverage, but overall they agreed that Roche’s serological test performs exceptionally well.
“What you are seeing is a newspaper description of sensitivity and specificity,” explained Roger D. Klein, MD, JD, a former adviser to the FDA and a Faculty Fellow at the Center for Law, Science and Innovation at the Sandra Day O’Connor School of Law. “That said, the Roche assay looks like a good test. It is an electrochemiluminescence assay that uses what’s called a sandwich principle.
“It employs a recombinant version of the virus’ nucleocapsid protein as a detecting antigen,” he added. “Two differentially- labeled antigens bind to the antibody, where one allows for isolation and the other for detection.
“To validate the assay, Roche used about 5,476 specimens, which, by itself, is a large number,” he added. “It specifically looked at cross reactivity for a number of viruses, such as HIV, hepatitis B, hepatitis C, including 40 specimens from individuals with commonly-circulating coronaviruses, and 40 specimens from people with colds for a total of 130 specimens in this phase, and found no cross reactivity.
Varying Time Points
“Roche also assessed sensitivity by testing a total 204 PCR-positive specimens from 69 infected people at varying time points after the PCR diagnosis,” Klein explained. “The 100% sensitivity was based on 29 specimens taken at 14 or more days after the PCR test. Therefore, it can claim 100% sensitivity—on a reasonable, but relatively small number of specimens.
“We should note, however, that the sensitivity drops off considerably to 88.1% at seven to 13 days after the diagnostic PCR test, and to only 65.5% at zero to six days after the diagnostic test,” he added. “The point is that it takes time for an infected person to generate antibodies, and that’s why Roche’s COVID-19 test is most accurate at least two weeks after PCR diagnosis,” commented Klein. “The test is not sensitive early in the course of infection, presumably because infected individuals have not yet developed antibodies.”
This finding in Roche’s analysis is similar to the findings that other researchers have uncovered in their work analyzing serological tests.
Looking at specificity, Klein noted excellent results in this area as well. “Specificity—which is of great concern because it reflects false positive results—is very high based on tests of 5,476 presumably negative specimens,” he explained. “These specimens were of different types and appeared to include the cross-reactivity specimens above, such as 40 coronavirus and 40 cold specimens. The largest numbers of specimens were from ‘routine diagnostic’ and blood donor patients.
“There were only 10 false positive results in these presumably negative specimens—seven in the routine diagnostic group and three among the blood donors, yielding 99.8% specificity,” he added.
“It’s important to note that the positive predictive value (PPV) is the number of true positives divided by true positives plus false positives,” Klein explained. “PPV depends on the prevalence of antibodies in the population, which means that this test will perform much better for identifying antibody positive people in New York, for example, than it will perform in central Montana.
“If 25% of people in New York City are positive, then among every 5,000 people tested there would be 1,250 true positives and 10 false positives, yielding a PPV of 99.2%,” he calculated. “That percentage comes from dividing 1,250 by 1,260. This means almost all individuals in New York City who test positive would have antibodies to the virus.
Prevalence of the Infection
“But then consider what would happen in central Montana if 0.1% are positive,” he said. “In this situation, because the prevalence of the infection is so low, the PPV would also be much lower. With 5,000 tests there would be only five true positives, but there would still also be 10 false positives.
“Dividing the five true positives by all 15 positive results yields only 33% of the positive results actually having antibodies,” he added. “In that case, two thirds of the tests would be false positives!
“Now consider what it means to have a negative predictive value (NPV) based on the sensitivity of 100% at 14 days or more,” Klein said. “The NPV would be very high wherever the test is used, which means that if the result is negative, that person probably doesn’t have antibodies.”
A second expert offers a slightly different analysis and cautioned that, as with many tests, it’s important to look closely at the validation data. Michael A. Noble, MD, a professor in the Department of Pathology and Laboratory Medicine at the University of British Columbia, in Vancouver said, “The devil is in the details.”
At UBC, Noble is also the Chair of the Clinical Microbiology Proficiency Testing Program and Chair of the Program Officefor Laboratory Quality Management.
Unique SARS-CoV-2 Protein
“It sounds like what the company is saying is that it isolated a protein that’s unique to SARS-CoV-2, but which is not found on SARS or MERS or the other Alpha or Beta coronaviruses generally,” he explained. “I would take Roche’s word (with caution). It has done a broad and extensive studyof all the variations found within those general human coronaviruses. That said, I am usually very skeptical of laboratory work done in a hurry.
“For any lab, sensitivity testing is challenging,” emphasized Noble. “Therefore, I suspect that within the roughly 6,000 patient samples, more than 5,500 were symptomatic and were of a variety of ages,” he explained. “Also, it could be that the company’s study had 500 mild or asymptomatic patients.
“If we assume that all of the 5,500 should be positive, and all the 500 should be negative, then we can do some arithmetic,” he said. “Roche says it has 100% sensitivity, which means that the 5,500 all have the antibody. Then, if only one of the 500 did have antibodies, that would give them a specificity of 99.8%.
Size of Asymptomatic Group
“But, if there are not 500 people without symptoms of COVID-19, but 5,000, and instead of picking up only the expected 10, the test picks up 25 positives, then the specificity goes down to 99.5%,” he added.
“That reflects the fact that the bigger the size of the asymptomatic group gets—included the ‘worried well,’ the more the specificity of the test will drop.”
From there, Noble recounted what happened during the spread of novel infectious pathogens in the past. “In these earlier outbreaks, both the United States and Canada, we limited the number of people being tested without symptoms to as close to zero as possible,” he said. “I think that may be done specifically to make the specificity of the reverse transcription PCR test results look good, at least relatively speaking.
“That’s what was done in the early days of HIV to prevent people from being falsely reported as positive,” he noted. “I suspect that when the United States opens up COVID-19 testing to everyone, we will find that the number of asymptomatic people will be five to 10 times the number of symptomatic people. At that point, all sorts of confusion will reign.”
Abbott’s Antibody Test Also Performs Well
When the federal Food and Drug Administration approved Abbott Laboratory’s serological test for the novel coronavirus SARS-CoV-2, it reported high numbers for sensitivity and specificity. The scores were not quite as high as those of the Roche test, but were impressive, nonetheless.
Both tests have emergency use authorizations (EUAs) from the FDA and the CE Mark. The FDA granted the Abbott test—called Architect SARSCoV-2 IgG—an EUA on April 27 for the enzyme-linked immunosorbent assay (ELISA). The agency said the high-throughput assay’s ability to detect IgG antibodies had a sensitivity level of 100% (88/88), and a specificity of 99.6% (1,066/1,070).
Late in April, Abbott was scaling up its manufacturing for antibody testing. At that time, the company said it expected to ship about one million tests to U.S. customers that week and ship four million antibody tests during April. Roche said that in May it would provide “high double-digit millions of tests” in Europe and in the United States and ship more tests thereafter.
Contact Roger Klein, MD, JD, at email@example.com; Michael Noble, MD, at firstname.lastname@example.org or 604-827-1337.