CEO SUMMARY: When this hospital lab adopted molecular screening tests for Chlamydia trachomatis and Neisseria gonorrhoaea, physicians soon noticed a change in the rate of false positives. In researching the performance of the molecular assays compared to cultures, pathologists at this laboratory gained a better appreciation of how the introduction of a new molecular assay may disturb clinicians’ practice patterns.
MANY HOSPITAL LABORATORIES, have yet to establish a molecular diagnostics testing program. The technology is still complex and the economics remain uncertain for many types of molecular assays.
Even pioneering hospital laboratories are learning new and unexpected lessons from their molecular testing programs. That’s certainly the case at the laboratory of Resurrection Medical Center in Chicago, Illinois.
“Each new molecular test we add to our menu introduces new variables and often creates clinical consequences we had not anticipated at the time we decided to set up and offer that molecular test,” observed Michael L. Mihalov, M.D., Medical Director of the laboratory at Resurrection. “This has been consistently true during my ten years of experience in molecular pathology.
“A significant lesson learned here about molecular diagnostics is that all assays are not equal,” he explained. “By that, I mean that the impact of a specific assay with a known sensitivity and specificity can affect clinicians in problematic ways, relative to other test methodologies.
“As an example, we are dealing with a problem of false positive results from the molecular tests we offer to clinicians for Chlamydia trachomatis and Neisseria gonorrhoeae screening,” continued Mihalov. “It was about three years ago that we set up and began offering molecular tests for these two diseases.
“Over time, certain physicians began to question some of our positive results. As we worked through these cases with the physicians, we began to appreciate how the differences in performance between culture and the molecular assay was responsible for these outcomes,” Mihalov stated.
“In the past, culturing for Chlamydia and Gonorrhea was the gold standard,” he noted. “Although sensitivity was not fantastic, there are essentially no false positives, so specificity approached 100%. In contrast, molecular tests for these two diseases have a much better sensitivity. But when manufacturers unnecessarily try to push the limits of sensitivity even higher, the result is false positives and lower specificity. On the surface the sensitivity may sound great—around 97-98%.
“But that small difference in specificity between the two diagnostic test technologies was enough to affect the clinical practice of physicians ordering these tests,” said Mihalov. “Our molecular assay, at 97% specificity, means that, for every 200 patients screened with these tests, on average six would get false positive results. Since our prevalence is 0.5%, the result is we got only one true positive in every 200 cases—that’s a pretty poor positive predictive value.”
Mihalov explained that because most physicians have had years of experience with the culture method for detection—and its near-100% sensitivity—to suddenly begin seeing a regular number of false positive results (from the molecular assays) was most unwelcome. As they contacted the laboratory and worked through each case with him, they were not sympathetic to his explanation that, in dropping the culture methodology for a molecular-based screening test, the laboratory gained speed to result but lost a recognizable degree of accuracy on the false positive side of the equation.
“One consequence of this development is that physicians become more defensive in their practice, particularly when it comes to screening tests,” said Mihalov. “I am aware of one ob-gyn group practice in Chicago that stopped screening for Chlamydia and Gonorrhea. They had a very low incidence of these diseases among their patient population and one reason for stopping screening was to avoid having to deal with false positive results.”
Mihalov’s experience with false positives caused his laboratory team to study the problem. Their findings are instructive to laboratory directors and pathologists currently evaluating molecular diagnostic assays for their laboratory. It will not be a surprise to learn that the problem is rooted in the relationship between sensitivity and specificity. The unexpected twist is how the objectives in developing a molecular test for HIV screening affected companies that developed molecular screening assays for other infectious diseases, including Chlamydia trachomatis and Neisseria gonorrhoeae.
“Our molecular assays, at 97% sensitivity, mean that, for every 200 patients screened with these tests, on average six would get false positive results.”
“When we first established our molecular testing program for these two diseases, we were running the specimen once,” recalled Mihalov. “It didn’t take long for our lab to get physician calls notifying us of false positive results. At that time, we determined that in the Resurrection system, our prevalence ranged from 0.5% to 5%.
“We next called the manufacturer and obtained detailed data on their internal studies of the assay,” he stated. “Their data showed that the specificity of the assay was around 97%. Actually though, to reach and maintain that level, we have to run each positive or equivocal specimen in triplicate.”
“However, even at 97% specificity, we have not overcome the false negative situation with the clinicians,” Mihalov said. “That’s because they are accustomed to the near-100% specificity of cultures when screening for these two diseases. Certainly manufacturers struggle with where to set sensitivity and specificity in every diagnostic assay, but every laboratory loses credibility with physicians when it issues false positive results. As an example, our clinicians are not thrilled when they learn that the positive Gonorrhea tests we reported on 60 year-old women in long-term, monogamous relationships were actually false positives.”
HIV Screening Objectives
Having studied the specificity problem in using molecular technologies to screen for Chlamydia and Gonorrhea, Mihalov believes IVD manufacturers designed these tests with a mindset shaped by HIV screening. “If you go back to the earliest days of HIV screening, the objective of the healthcare system then was to identify as many HIV-positive individuals as possible with the screening test,” he said.
“The bell-shaped curve was intentionally shifted to detect low positives,” stated Mihalov. “After all, public health authorities considered the undiagnosed HIV-positive patient to be the greatest threat. So generating a relatively high level of false positive HIV screening results was the trade-off. Clinicians using such tests understood this trade-off, so the rates of false positives were acceptable.
“But, as you move from diseases like HIV and HCV to Chlamydia and Gonorrhea, heightened sensitivity at the cost of lower specificity in screening tests will affect clinicians and patients in ways that may be counter-productive,” he observed. “Another example is pre-natal cystic fibrosis screening where false positives have serious physical and emotional implications for a patient.
“My point here is to alert others to an obvious fact that can be over-looked when making decisions to replace an existing screening test methodology with a molecular test. The molecular technology will create different outcomes—for both the laboratory and referring physicians,” declared Mihalov. “Thus, the important question we now ask in our laboratory is ‘What do we really want to achieve when we adopt a new molecular test over an existing methodology, like culture in the case of Chlamydia and Gonorrhea?’”
Mihalov had other observations. He believes that technologies like PCR have certain limitations. Because these assays use biological products like polymerases, they have an inherent degree of variability that can be troublesome in infectious disease testing. For these and other reasons, he predicts that demand will be strong for alternative technologies to PCR, such as signal amplification methodologies.
Molecular Versus Culture
However, there was one question Mihalov was unprepared to answer. When asked by THE DARK REPORT how his laboratory’s current cost of molecular screening for Chlamydia Trachomatis and Neisseria gonorrhoeae compared to the culture method—particularly since his lab was doing triplicate molecular testing to achieve the 97% specificity—Mihalov wasn’t certain.
“With our attention focused on other issues, we’ve not audited our costs since implementation of these molecular tests. We should evaluate the clinical-effectiveness and cost-effectiveness of molecular versus culture methods for Chlamydia and Gonorrhea,” he answered candidly. “When I have that information, I’ll share the findings with you!”