CEO SUMMARY: During the past five years, several companies entered the lab marketplace with claims that their new Pap test technologies are improvements over conventional Pap smear methods. Armed with investment capital from Wall Street, these companies launched aggressive sales and marketing campaigns to clinical labs, pathologists, payers, physicians, and women. The diversity of advertising claims has, at a minimum, caused much confusion among laboratorians about the true cost/benefit effectiveness of various new Pap test technologies. Our guest writer attempts to sort through the various issues.
PART ONE OF A SERIES
BY JOSEPH PLANDOWSKI
EDITOR’S NOTE: Despite FDA approval of several competing new technologies for Pap testing, there continues to be a lack of consensus within the pathology profession and the clinical laboratory community about the benefits and cost effectiveness of these products.
Joseph Plandowski tracks and studies new Pap test technology. He was a director for an emerging cytology products company that offered an enhanced cytology microscopy work station system to cytology laboratories. In this first of a multi-part series on the market acceptance of new Pap smear technologies, Mr. Plandowski presents seldom-publicized aspects of the way data from studies is used to support the aggressive marketing of these new Pap test technologies. In future installments, he will look at the economic impact to labs when they switch from conventional Pap smear methods to these new Pap test technologies.
IN THE BATTLE TO INTRODUCE new technology to conventional Pap smear testing procedures, there is no shortage of sales and marketing efforts.
In particular, marketing campaigns to introduce liquid-based preparation (LBP) products have been intense. These include innumerable trade shows, journal advertising, television commercials, press releases, brochures, and self-interest groups hyping liquid based preparation kits for Pap testing.
Two companies currently have FDA clearance to market LBP Pap tests. Cytyc Corporation received clearance for its ThinPrep® product in May 1996. Almost three years later, in June 1999, TriPath Imaging, Inc. (formerly AutoCyte, Inc.), received clearance for its PREP® product. These products are also known as thin-layer preparations (TLP).
In the race to build market share for LBP Pap test kits, Cytyc currently enjoys a huge market share lead over TriPath. I believe this is due primarily to Cytyc’s marketing prowess, rather than its earlier clearance by the FDA. Interestingly, Cytyc and TriPath are embroiled in several lawsuits against each other, including a Cytyc-initiated lawsuit claiming cryopreservative patent infringement by TriPath. If it can’t be defended, the result may have a severe adverse effect on Cytyc. Don’t be surprised if this becomes a tortoise-and-hare scenario.
Background To Pap Issues
After several years of marketing efforts to women, physicians, and laboratory management, there is plenty of confusion about the true costs and clinical benefits of new Pap test technologies. That’s because these companies selectively emphasize the most positive aspects of trials and studies involving their products. This means that other clinical data from the same study which may indicate questionable, even negative results, are not brought to the attention of potential customers, users, and laboratory managers.
Thus, it is not without justification that a number of very knowledgeable and experienced cytopathologists have complained that the full and true story about the clinical efficacy of new Pap smear testing technologies has yet to be brought into public debate. They point out that women, physicians, laboratory managers, payers, and even Wall Street investors are only getting selected parts of the story from the collective group of emerging cytology companies.
It’s an undisputed fact that conventional Pap smear methodology is complex. It requires a fairly sophisticated understanding of this process to best understand why Pap smear testing is an effective screening method for cervical cancer.
This sophisticated understanding certainly does not exist among women in the lay public. And even physicians who regularly offer their patients Pap smear tests, may lack the up-to-date clinical sophistication to correctly evaluate the results from a particular study of a specific new Pap smear technology.
It is the complexity of the conventional Pap smear testing process, then, that makes it easier for emerging cytology technology companies to advertise and position their products in ways that can generate misperceptions among physicians, payers and patients.
For example, Cytyc promotes its thin-layer preparation Pap smear test as a “replacement” of the conventional Pap test. As a result, journalists and commentators, untrained in clinical cytology, prepare newspaper and television features which refer to the LBP Pap test as a technological break-through that has replaced an old technique, the conventional Pap smear.
Bethesda System Terms Complex for Lay Women
There are infinite variables which affect the results of any laboratory’s particular Pap smear screening program.
That is why current methods used in Pap smear screening represent a complex process. Comparing the results of independent clinical studies across a number of screening sites is challenging even to the experts.
The Bethesda system of Pap smear classification is in wide use. For lay women, understanding the ramifications of its most common categories can be problematical.
- ASCUS (atypical squamous cell of undetermined significance): These cellular changes exceed those which can be a benign process, but fall short of an intraepithelial lesion or cancer. Many of these will revert to normal. 10% to 15% will progress.
- LSIL (low-grade squamous intraepithelial lesion): This is the HPV cat- egory. Low-grade SIL (HPV/mild dysplasia/CIN1). About 25% will have a high grade SIL at colposcopy. Lesion has a 15% chance of progression to in situ cancer (CIS).
- LGSIL (high grade squamous intraepithelial lesion): High-grade SIL (moderate and severe dysplasia/CIN 2, 3, and CIS). Lesion has a 50% chance of progression to CIS.
- Squamous cell carcinoma: probable invasive cancer, requires histology
LBP is Not A Replacement
In reality, LBP is an additive step in the conventional Pap smear process, not a replacement of the process. Moreover, thin-layer preparation itself is not a new technology. For almost 20 years, thin-layer preparation has been used for many types of specimens. The “newness” is in its application to Pap testing.
To put the LBP Pap test into perspective, it is important to briefly review conventional Pap smear methodology. That process begins in a physician’s office where cervical cells are collected with a cell collection device and smeared onto a microscope slide. The physician immediately sprays the cells with a fixative to prevent air-drying of the cells, then sends the slide to the lab.
When the laboratory receives the Pap smear slide, it stains the cells and protects them by adding a plastic or glass coverslip over them. The slide is now ready for screening by a cytotechnologist. All slides containing abnormal cells are referred to a pathologist for final diagnosis.
When thin-layer Pap products are used, the process follows a similar procedure. A physician still collects the cervical cells. But instead of smearing the cells onto a microscope slide, the physician inserts the cell collection device with collected cells into a vial containing a proprietary solution that preserves the cells until they reach the laboratory.
At the laboratory, the vials are processed in proprietary devices (instruments) which separate the cells from the preservative solution. Then the cells are deposited in a 13 mm diameter (TriPath) or a 20 mm diameter (Cytyc) thin-layer circle on a microscope slide. The cells are stained and protected by a glass or plastic coverslip. Now the slides are ready for screening by a cytotechnologist. If an abnormality is found, the slide is reviewed by a pathologist.
It’s important to recognize that, once a slide is prepared by either the conventional or thin-layer method, the remainder of the Pap smear process is virtually identical. The sole difference is that, in the LBP Pap test, collected cells are put directly into a proprietary preservative solution rather than cells being smeared onto a microscope slide (as in the conventional manner).
LPB Is Not A Replacement
Thus, when representations are made that the LBP Pap test is a replacement to conventional Pap testing, it should be noted that only the step between cell collection and deposition onto a microscope slide is different. Otherwise, the entire process is the same in either case. That is why LBP does not replace the conventional Pap test process.
Probably the most controversial part of the debate about the clinical efficacy of liquid-based Pap preparations centers around “sensitivity” (the ability of a diagnostic test to accurately identify a positive specimen), and “specificity” (the ability to not identify a negative specimen as being positive). It is acknowledged that conventional Pap smear screening methodology has relatively low rates of sensitivity and specificity. Yet, with regular screening cycles, this is sufficient to make conventional Pap smears a highly-effective diagnostic screening test.
Thus, it is against this background that the claims of various new Pap technology companies must be evaluated. After reviewing the clinical data supplied by Cytyc’s ThinPrep and TriPath’s PREP, the FDA issued clearances that allow both companies to at least claim equivalence with conventional Pap smears.
Additional FDA Clearance
But Cytyc also received an additional clearance from the FDA in November 1996. This permitted it to expand its claims for ThinPrep to include improved effectiveness in detecting LSIL and more severe lesions versus a conventional Pap smear (improved sensitivity), based on results from a subset of its clinical trials sites dubbed “screening centers.”
In contrast to labs serving a high risk population, these Pap smear screening centers had a relatively low prevalence of disease, thus increasing the likelihood of false positive diagnosis. As most pathologists know, many screening tests will show higher sensitivity if the specificity is lower, i.e., a higher number of false positives are generated. In this particular clinical study, when Mark Sherman, M.D. adjudicated the slides, he confirmed a large number of false positives from these three screening centers.
The most recent papers published during 1999 corroborate Cytyc’s claims for increased sensitivity. However, other results in those papers present disturbing findings that are not noted in any of Cytyc’s press releases.
One of the papers, authored by Martha L. Hutchinson, M.D., appeared in the April 25, 1999 issue of Cancer (Cancer Cytopathology), a prestigious and well-respected journal. This is a particularly good study to review, because histology was done on the non-negative Pap results. In the paper, Dr. Hutchinson reports that compared to the final diagnosis, ThinPrep detected 93% of the cases with HSIL and 100% of the cases with carcinoma. This compares with the conventional Pap smear which detected 78% and 91%, respectively.
Other Important Finding
Cytyc issued a press release on April 28, 1999 which trumpets these results. However, entirely absent in Cytyc’s press release is Dr. Hutchinson’s other important finding. That is, improvement in detection rates came with “a concurrent significant increase in colposcopy referrals.”
Specifically, 1,095 of the 8,636 patients (13%), screened by ThinPrep underwent a colposcopy. Of the 1,095 patients, only 565 (52%) were judged to have an actual abnormal Pap test after colposcopy.
In contrast, conventional Pap smears done on these same women resulted in 579 (7%) receiving a colposcopy. Of the 579 patients, 451 (78%) were judged to have an abnormal Pap test. Colposcopy with biopsy is an uncomfortable, if not painful procedure. It is also expensive. In any event, the supposed increase in disease detection by the ThinPrep product can be attributed to overdiagnosis (false positives).
Another finding that Cytyc did not mention in its April 28, 1999 press release is that the ThinPrep diagnoses agreed with final case diagnoses in 7,379 of the 8,636 patients (85%) compared to 7,669 of the 8,636 patients (89%) who had a conventional Pap smear.
The conventional Pap smear concurred with the final diagnoses in 290 more cases than ThinPrep! In my opinion, this particular clinical result certainly does not support an argument that Cytyc’s thin-layer preparation is an improvement over conventional Pap smear preparation methods.
Dr. Hutchinson’s study was sup- ported by the U.S. National Cancer Institute. Her paper is interesting reading even for the layman. For example, ThinPrep’s 93% HSIL detection rate was determined from its ability to detect 117 of the 126 cases with HGIL as a final diagnosis.
And, ThinPrep’s 100% carcinoma detection rate was determined from its ability to detect all 11 of the cases with a final diagnosis of carcinoma. This compares with the conventional Pap smear, which detected 98 of the 126 HGIL cases and 10 of the 11 carcinoma cases. All of these cases are from a population study of 8,636 patients.
Issue Of Overcall
But both of these results do not reflect the issue of overcall—of sensitivity versus specificity. If the cytotechs in this study were to define every slide as abnormal, sensitivity would be 100% (because no actual abnormals were missed), but the specificity would be zero (because every normal slide was judged to be abnormal).
Dr. Hutchinson summarizes her paper “suggesting that the ThinPrep method is at least as good as conventional cytology [my italics] in detecting SIL and carcinoma.” Dr. Hutchinson is at Women & Infants Hospital, Brown University, Providence, Rhode Island.
Two pathologists from the Quest Diagnostics Incorporated laboratory in Boston did a study of ThinPrep and published their findings in the September 1999 issue of Archives of Pathology and Laboratory Medicine. Cytyc quickly issued a press release on September 15, 1999. It heralded ThinPrep’s 103% increase in the detection of HSIL and a 73% increase in the detection of LSIL compared to the conventional Pap test.
Omitted in Cytyc’s press release is any mention of a 26% decrease in detection of carcinoma reported in this study. The press release also failed to disclose that, in this study, ThinPrep generated a 205% increase in the number of cases diagnosed as “unsatisfactory for evaluation.”
In this same study, the authors also presented data for a subset of physician accounts that completely switched their patients to the use of ThinPrep. Here the increase in detection is even more dramatic. It rises to 129% for HSIL and 74% for LSIL. However, all cervical cancers in this subset of patients were missed, not an insignificant finding. Furthermore, the percent of cases diagnosed as “unsatisfactory for evaluation” increased to 300%.
Pathologists and laboratorians will recognize the problems caused by a diagnosis of “unsatisfactory for evaluation.” Standard practice for any patient with a Pap test diagnosis of “unsatisfactory for evaluation” is to repeat the test. Both the physician and the laboratory generally perform this follow-up Pap test at no cost to the patient.
Thus, any new Pap test technology which generates an increased number of “unsatisfactory for evaluation” diagnoses triggers a cascade of additional (and unreimbursed) costs to the physician and the laboratory. In addition, the patient is inconvenienced and often irate at her physician.
Issues of Sensitivity and Specificity Greatly Influence Outcomes of Studies
Why is there so much disagreement and debate about the interpretation of studies involving liquid-based preparation (LBP) systems for Pap test screening?
It is because the analysis and interpretation of sensitivity and specificity for diagnostic screening tests is like a Gordian Knot- convoluted, complex, and unclear.
To calculate sensitivity, one must know how much true disease actually exists in the test population—this is the denominator. In most studies of
LBP Pap test technologies, the denom- inator is unknown. Thus, physicians conducting studies do not calculate “sensitivity” as such, but rather “improved detection.” Moreover, they typically do not prove that the additional disease detection is real (actual positive) versus false positives.
That is because, if the denominator (true disease in a studied population) is unknown and the study does not exclude false positives, any answer is possible and all answers are probably wrong.
Finally, as laboratories well know, a higher rate of “unsatisfactory” Pap test diagnoses frequently causes the referring physician to switch the account to a competing laboratory. Since OB/GYN accounts are among the most profitable of physician specialties for laboratories, loss of this business is financially painful.
In recent years, several marketing and advertising campaigns by the various vendors of new Pap test technologies created problems for OB/GYNs. Many women, after reading advertisements and media stories on why new Pap test technology is “better,” began to doubt the effectiveness of the conventional Pap smear. Physicians were asked by these women why their office didn’t offer these “new” and “better” Pap tests.
Dealing with concerned, fearful female patients proved to be a significant problem. That is why the American College of Obstetricians and Gynecologists (ACOG) declared its position on recently-introduced Pap test technologies. In a news release dated July 31, 1998, ACOG stated “Despite the recent FDA approvals of new Pap test screening techniques (ThinPrep, AutoPap®, PAPNET®), these technologies do not represent the current standard of care in cervical cancer screening.”
ACOG went further to explain that it “issued the document partly in response to aggressive direct-to-consumer advertising that led many women to feel they are at greater risk for undiagnosed cervical cancer if they don’t use the latest technology.”
Exactly one year later, July 31, 1999, ACOG issued another news release stating “Invest health resources in widespread Pap screening, not new technologies.” ACOG argues that “new cervical cancer screening technologies are not likely to help women most in need of cervical cancer testing and could even widen the economic gap between women who get Pap smears and those who don’t.”
The U.S. Department of Health and Human Services (HHS) issued a lengthy and detailed report in April 1999 entitled Evaluation of Cervical Cytology. HHS compared the new technologies for cervical cytological screening with conventional Pap testing in terms of diagnostic accuracy, costs, effectiveness, and cost-effectiveness in adult women. HHS findings were revealing. It stated that “the imprecision in estimates of effectiveness and the cost of the new [Pap test] technologies makes drawing firm conclusions about their relative cost-effectiveness problematic.”
Clearly, the specific studies cited here show that the clinical effectiveness of thin-layer based preparation for Pap testing has yet to demonstrate clear superiority over conventional Pap smear preparation. In fact, there is credible evidence that LBP Pap smears actually add costs to the healthcare system without contributing clinically and economically worthwhile benefits.
Additional Clinical Studies
Additional studies now under way may help resolve these issues. In the meantime, there are several other areas of concern that involve LBP Pap tests. Some of these are:
1) A study published in The American Journal of Clinical Pathology in February 1994 reported that, with conventional Pap smear collection and preparation, as much as 80% of the collected cells are not transferred to the slide, but discarded with the collection device. The discarded cell collection device may contain abnormal cells. However, most conventional Pap test slides contain an average of 150,000 to 300,000 cells.
While the LBP collection process saves all the collected cells in the preservative fluid, only about 50,000 cells are deposited on the slide. The rest remain in the preservative fluid. If there were 300 abnormal cells on the cell collection device containing 500,000 cells, LBP technology does not ensure that all those 300 abnormal cells would be deposited on the slide. Statistically, only 50 abnormal cells would be put on the slide. The other 450 abnormal cells remain in the preservative fluid which is eventually discarded. With a low prevalence of abnormal cells, there is also the possibility that none of them end up on a slide prepared by thin-layer methods.
2)When Cytyc supplied data to the FDA under its original application, the data from ThinPrep’s six clinical trial sites left some unanswered questions. The six sites were equally split between hospital laboratories and screening centers, and encompassed 6,747 patients. The results were far from equal. The hospital laboratories demonstrated only a 6% improvement in disease detection using ThinPrep versus conventional Pap tests while the screening centers demonstrated a 65% improvement. When averaged, the overall results demonstrated an 18% improvement. But this 18% improvement does not include the FDA-required adjudication of results from the different sites to determine the actual diagnostic truth. In fact, that adjudication, when done by Mark Sherman, M.D., showed only a 5% increase in sensitivity, which was not statistically significant.
This site disparity was of concern to some in the industry and resulted in letters to the FDA questioning its clearance of ThinPrep. TriPath’s submission of data to the FDA was supported by studies on 8,983 patients from eight sites. Its PREP liquid-based Pap test demonstrated an improvement of 17% in screening sensitivity relative to the conventional PAP smear.
Observation & Conclusions
I find it surprising that, after several years of working with these new Pap test technologies, two things have failed to occur.
First, despite an ever-increasing volume of day-to-day clinical experience with these new Pap test technologies, no clear consensus exists among pathologists and laboratory executives about whether these products really do make a positive contribution.
That, in itself, may be an important observation. Healthcare technology that is robust and effective usually gains widespread acceptance with a minimum of opposition and criticism. Certainly the current ongoing debate by laboratorians about new LBP Pap test technologies indicates that the products themselves have failed to demonstrate a clearcut superiority that would engender unanimous support.
No Compelling Argument
Second,no manufacturer of the several new LBP Pap test technologies seems to have funded a sizable and comprehensive clinical study that, by its size and design, would make a compelling argument that its technology is unquestionably more cost-effective than conventional Pap smear methodology.
Pap smears contribute to women’s health, and that is a politically-correct goal which attracts an abundance of grant money from foundations, charitable trusts, and the government. It would certainly seem reasonable that these entities would fund a sizable study for any new Pap test technology that would improve women’s health and add value to the healthcare system.
For laboratory managers and pathologists currently evaluating new LBP Pap test technologies, I would suggest that the lack of lab industry consensus, combined with the lack of any compelling, irrefutable cost-effectiveness, means that most new LBP Pap test technologies may not yet be “ready for prime time.”
Rather, these new products may have great application in specific laboratory organizations which serve unique populations around the United States. But for most labs, there remain more questions than answers about the true cost/benefit performance of various new Pap test technologies.