CEO SUMMARY: KLAS Research published its Top 20 Best in KLAS Awards in December and ranked Psyche WindoPath the top system in two categories. KLAS, which rates as many as 56 categories of healthcare software and professional services products in its annual report, does not provide much detail in how it conducts surveys of users, nor the number of users of each product who were surveyed. KLAS does say that it uses stringent methodology to ensure all research is accurate, honest, and impartial.
RELEASED LAST MONTH, the newest rankings of healthcare software and professional services products KLAS Research KLAS provided THE DARK REPORT with an extract of the full report and asked that the details of the ratings for the anatomic pathology systems covered developed annually placed WindoPath by Psyche Systems Corporation as the best-rated anatomic pathology information system.
In its report titled “Top 20 Best in KLAS Awards: Software & Professional Services,” KLAS provided ratings for a wide range of healthcare software categories. Laboratory information systems (LIS) are rated in several of these categories.
In the category of “Anatomic Pathology,” KLAS listed six anatomic pathology information systems. WindoPath by Psyche Systems was rated as first and Sunquest Information Systems’ CoPath was rated as second.
KLAS, a company founded in 1996 and based in Orem, Utah, builds its ratings from surveys of healthcare professionals. (See sidebar on page 17 for KLAS’ explanation of the methodology it uses.) In this regard, the company states its ratings are based on feedback from users of the various information products that KLAS rates.
KLAS provided THE DARK REPORT with an extract of the full report and asked that the details of the ratings for the anatomic pathology systems covered in the report not be published. KLAS generates money by selling its reports to healthcare providers and vendors. It also provides advisory services, performs custom research, and provides on-site consulting services.
Rating AP Info Systems
Across the anatomic pathology profession, it is believed that the two most widely used anatomic pathology information systems are Cerner Corporation’s CoPath/Millennium Anatomic Pathology and Elekta’s PowerPath Anatomic Pathology. Given the market shares held respectively by these two products, it was interesting that the KLAS ratings placed neither system in the top two places.
However, both systems were included in the list of six anatomic pathology information systems rated and ranked by KLAS. The other two systems included in the survey were Meditech’s C/S Anatomical Pathology and SCC Soft Computer’s SoftPath.
In general, few in the anatomic pathology profession have much knowledge about KLAS and its rating service. Typically, when pathologists consider upgrading or purchasing a new anatomic pathology information system, they tend to rely on RFPs, interviews with vendors and reference checks with existing users. They may also consider the market share held by different path information products as they make a decision about which anatomic pathology information system to purchase.
Steps in the Buying Process
As part of the buying process, it is also helpful to know how many new customers each anatomic pathology vendor acquired during recent years. But that information is not easily available to most pathology group practices.
KLAS is a company which wants to provide more detailed information to healthcare providers preparing to upgrade or purchase software and information systems. In its most recent “Top 20 Best in KLAS Awards: Software & Professional Services” report, KLAS provides rankings in 41 categories of “software solutions” and 15 categories of “professional services.”
Within the laboratory segment, KLAS ranks laboratory information systems in two primary categories, with eight subcategories. There are three sub-categories for anatomic pathology.
Data Used for Rankings
It is not easy to understand the precise data inputs used by KLAS to develop its rankings. For example, was the number one ranking for Psyche’s WindoPath based on surveys conducted with only pathology labs that acquired a new or upgraded pathology system during the 12 months prior to the survey’s release last month?
If so, that would place a different interpretation on the survey’s findings than if a statistically significant number of existing, long-time customers for each of the six pathology systems were included in the survey.
On the following page, one of the pathology profession’s leading experts on clinical laboratory and pathology informatics discusses the challenges involved in evaluating and ranking the capabilities and performance of different LIS products and anatomic pathology information systems.
Finally, any client or regular reader of THE DARK REPORT with experience using a rating service like KLAS is invited to contact us and share their experience.
How KLAS Develops Its Software Rankings
TO BUILD ITS RANKINGS of different healthcare software products, KLAS relies on interviews and direct feedback from provider organizations. In its report, “Top 20 Best in KLAS Awards: Software & Professional Services,” it described the data collection process as follows:
KLAS utilizes a three-step process to collect candid performance data. First, KLAS collects a series of direct product evaluations completed by healthcare provider organizations.
Second, KLAS performs in-depth, confidential interviews with the IT executives and department directors completing the questionnaire to gather valuable insight into specific strengths, weaknesses and future expectations for the product.
Third, the gathered data is subjected immediately to an internal audit to verify completeness and accuracy, and to make sure the anonymity of the provider organization is maintained. During the audit, each data set is reviewed by a KLAS executive and at least two other people.
Lab Informatics Expert Explains Why Ranking Laboratory Info Software Can Be Problematic
THIRD-PARTY SERVICES TO RANK HEALTHCARE SOFTWARE PRODUCTS face many challenges and typically produce assessments that are not likely to be helpful to the typical buyer of laboratory information system (LIS) software.
That’s the assertion of Bruce A. Friedman, M.D., Active Emeritus Professor of Pathology at the University of Michigan Medical School and President, Pathology Education Consortium. For almost three decades, Friedman has been among the leading experts in laboratory informatics. His well-read blog at www.labsoftnews.com regularly tackles issues involving laboratory software and new information technologies.
“The reason I say the rankings are meaningless is because laboratory information systems are unbelievably complex systems,” explained Friedman. “You have to evaluate gradations by price, functionality, the target market, and a number of other factors, including size and complexity of the labs it supports.
Months To Rate An LIS
Further, a thorough and proper evaluation of a laboratory information system takes weeks and months,” he said. “This process is familiar to anyone who has had the responsibility of evaluating LIS products when his or her laboratory prepares to either upgrade or purchase a new LIS.
“The first challenge for a software rating system is that few laboratories are identical in their test menu, specimen volume, instrumentation, staffing, and use of information technology,” noted Friedman. “It is why clinical laboratories and anatomic pathology groups use a request for proposal (RFP) process that generally requires several months or more to conduct.
“Such an RFP for a clinical lab or pathology system will ask the vendor to address up to 150 specific items,” he stated. “Take the function of lab ordering. The RFP will ask each vendor questions such as: Can your software do A, B, or C, and can it accomplish D or E? The vendor will respond with a ‘yes,’ ‘no,’ or ‘in development’ relative to each of these functions.
“Then the lab crunches these numbers to come up with some kind of rated average as to how the capabilities of each vendor’s software matches the specifications of the laboratory,” he explained. “Not only is the laboratory truly evaluating each vendor’s system with regard to its own operations, but, it is also evaluating the vendor itself and its financial stability.
“As part of this evaluation process, the lab wants to know the extent to which other people recognize these vendors as being reputable and their system reliable,” he added. “It is a complex process but it is transparent. When requested, labs will often send you a copy of their RFP.
“Plus, the responses of the vendors derived from the RFP can be added as an appendix to the contract so that the vendors are then held to their responses legally,” he explained. “Frankly, that kind of a process is the only one I would trust.”
Friedman noted that a ranking service also has the challenge of evaluating a new product versus long-established LIS products. “Epic’s LIS product is called Beaker. It is in use in maybe 10 hospitals and is in its earliest stages of development,” he noted.
“A best-of-breed LIS is one that has been tested in the market for a number of years, can go head-to-head with any LIS and perform in a superior fashion,” he explained. “Now, should a new LIS like Beaker, with a handful of sites, be listed as a top LIS comparable with a system from a Sunquest, an SCC or a Cerner, all of which have been on the market for at least 10 years and all of which have 200 to 500 installations, depending on how you count them?”