Receiver-operating characteristic curves and likelihood ratios: improvements over traditional methods for the evaluation and application of veterinary clinical pathology tests

Receiver-operating characteristic (ROC) curves provide a cutoff-independent method for the evaluation of continuous or ordinal tests used in clinical pathology laboratories. The area under the curve is a useful overall measure of test accuracy and can be used to compare different tests (or different equipment) used by the same tester, as well as the accuracy of different diagnosticians that use the same test material. To date, ROC analysis has not been widely used in veterinary clinical pathology studies, although it should be considered a useful complement to estimates of sensitivity and specificity in test evaluation studies. In addition, calculation of likelihood ratios can potentially improve the clinical utility of such studies because likelihood ratios provide an indication of how the post-test probability changes as a function of the magnitude of the test results. For ordinal test results, likelihood ratios can be calculated on a category-specific basis from the empirical data or by using the slope of the line joining adjacent category limits on the ROC curve. For continuous test results, data need to be categorized into intervals for estimation of likelihood ratios, or they can be calculated as the slope (tangent) to the ROC curve at a unique test value. We use ROC analysis and calculate likelihood ratios to evaluate the performance of tests reported in 2 articles previously published in this journal.     

Using and interpreting diagnostic tests.

Diagnostic tests are invaluable to the practice of veterinary medicine. Using them correctly and interpreting the results appropriately depend on having a good understanding of the basic principles outlined in this article. Topics covered include sensitivity and specificity, agreement among tests, using multiple tests, and other issues related to the use and interpretation of diagnostic tests. The most important principle is recognition that the interpretation of test results varies across populations and requires an estimate of the prevalence of the infection (or disease) in the population being studied.     

Rational use of diagnostic and screening tests

Veterinarians have a vast and ever-expanding array of diagnostic tests available to them. However, this abundance can be an embarrassment of riches that confounds diagnosis and undermines patient care if we do not make critical and informed decisions about the selection and interpretation of the tests we employ. Effective use of diagnostic tests requires a deliberate and informed approach. We must consider the strengths and weaknesses of the tests themselves and the clinical context, and we must be wary of the many biases that skew our use and interpretation of diagnostic tests. Understanding sensitivity and specificity, likelihood, prevalence and predictive value, the basic principles of Bayesian reasoning, and the cognitive biases that drive inappropriate testing are all critical to ensuring our use of imaging and laboratory testing improves patient outcomes.     

A comparison of diagnostic tests for hairy shaker disease

Abstract

Extract

From 1982 to 1988, the virus culture interference assay was used by the Animal Health Laboratory Network as a diagnostic test for hairy shaker (HS) disease (known as border disease in the United Kingdom). However, this assay is slow and relatively expensive, and in some studies of the prevalence of HS disease it gave inexplicably high rates of 28–33% ⁽¹⁾ Orr, MB and Montgomery, RH. 1983. Hairy shaker update in Otago and Southland. Surveillance, 10(4): 13–13.  [Google Scholar] ⁽²⁾ Smith, KR. 1986. Hairy shaker disease: Estimate of the true prevalence of this disease in Otago and Southland. Surveillance, 13(2): 8–9.  [Google Scholar] ⁽³⁾ Orr, MB. 1988. Sheep abortions - Invermay 1988. Surveillance, 16(3): 24–25.  [Google Scholar]) compared with other studies in which the prevalence rates were 1–8% ⁽⁴⁾ Horner, G. 1983. Hairy shaker disease: The North Island situation. Surveillance, 10(3): 21–22.  [Google Scholar] (G. Horner, pers. comm.) The present project was designed to compare the virus culture interference test with an immunoperoxidase (IPX) method and an antigen capture ELISA for the detection of HS virus.     

Comparison of diagnostic tests for the detection of equine infectious anemia antibody

Two diagnostic tests are approved for detecting antibody to equine infectious anemia virus: the agar-gel immunodiffusion (AGID) test and the competitive enzyme-linked immunosorbent assay (ELISA). A total of 420 sera from National Veterinary Services Laboratories check sets were tested with the AGID and competitive ELISA. A 100% correlation was obtained. The AGID and competitive ELISA were further used to test difficult samples with low levels of equine infectious anemia antibody (weak positives). A third test (Western blot) was also used with these weak positive samples to resolve any discordant results.     

Comparison of the sensitivity of different diagnostic tests for pancreatitis in cats

The objective of this study was to compare the sensitivity of different diagnostic tests for pancreatitis in cats. Twenty-one cats with confirmed pancreatitis were evaluated at the Small Animal Clinic of the School of Veterinary Medicine in Hannover, Germany, between September 1997 and January 1999. Clinical signs of affected cats were nonspecific, with 95% of the cats showing anorexia and 86% lethargy. Also, hematologic and biochemical abnormalities of affected cats were nonspecific. Serum feline trypsin-like immunoreactivity (fTLI) in these 21 cats with pancreatitis was 127.5 +/- 109.5 microg/L (mean +/- SD; range, 24-500 microg/L). Fourteen of the 21 cats with pancreatitis had complicating conditions. Their serum fTLI was 153.9 +/- 124.3 microg/L (mean +/- SD; range, 29 500 microg/L). In this study, abdominal ultrasound showed a sensitivity for pancreatitis of 24%, and abdominal computed tomography had a sensitivity of 20%. Serum fTLI had a sensitivity between 86% when a cut-off value of 49 microg/L was used (upper limit of the control range) and 33% when a cut-off value of 100 microg/L was used. We conclude that in this group of cats with pancreatitis, measurement of serum fTLI was the most sensitive diagnostic test of those evaluated. Abdominal ultrasound, however, may be a valuable diagnostic tool in some cats with pancreatitis.     

Evaluation of cellular and serological diagnostic tests for the detection of Mycobacterium bovis-infected goats

We have evaluated the comparative intradermal skin (CID) test, the interferon gamma (IFN-γ) assay, two ELISAs with bovine PPD antigen, the standard and the anamnestic using sera obtained, respectively, at the time of the tuberculin injection and 15 days later, for the diagnosis of caprine tuberculosis (TB). The sensitivity and specificity results were high for the CID test (83.7%, 100%), the IFN-γ assay (83.7%, 96%) and the anamnestic ELISA (88.6%, 95.8%). In contrast, they were comparatively low for the standard ELISA (54.9%, 88%). However, test results with the standard ELISA were positive in a group of goats with cavitating TB (100%). A combination of the CID test and the IFN-γ assay offered the highest sensitivity, 95.8%, and also high specificity, 96%. In spite of this, the evidence that the serological tests were most sensitive for the detection of goats with severe lesions (100% positivity) suggested that a combination of CID test and anamnestic ELISA may be most useful as part of an eradication campaign against caprine TB.     

Evaluation of diagnostic tests for Johne's disease in young cattle

OBJECTIVE: To investigate the development of immune responses in calves experimentally and naturally infected with Mycobacterium paratuberculosis and to evaluate the potential for diagnostic tests to detect infected calves. DESIGN: Sequential testing of four treatment groups of calves over a 2 year period. PROCEDURE: Twenty-nine calves were allocated to four groups. Group D calves were orally dosed with M paratuberculosis, group N calves naturally exposed to M paratuberculosis, group V calves vaccinated for M paratuberculosis, and group C were control calves (not infected or vaccinated). Blood and faecal specimens were collected from each calf at monthly intervals to 18 months of age and then every 2 months until they were slaughtered between the ages of 21 and 29 months. Specimens were tested using absorbed EIA, IFN-gamma EIA and faecal culture. The infection status of the calves was confirmed by extensive histopathological examination and tissue culture. RESULTS: M paratuberculosis infection was confirmed in 10 calves, comprising six of eight orally dosed calves, three of five naturally exposed calves and one of nine vaccinated calves. The six artificially infected calves and one naturally infected calf were detected shedding M paratuberculosis in their faeces. Results with positive absorbed EIA were obtained from one artificially infected calf, one naturally infected calf and three vaccinated calves. All calves including controls had positive results on at least one occasion using the IFN-gamma EIA. In addition, seven calves had positive bovine tuberculosis results using the IFN-gamma EIA, even though bovine tuberculosis has been eradicated from Australia. CONCLUSION: Detection of M paratuberculosis infection in young cattle continues to be difficult using current tests.     

Review of diagnostic plaque reduction neutralization tests for flavivirus infection

Flavivirus infections (including Japanese encephalitis, West Nile encephalitis and dengue fever/severe dengue) present a worldwide public health problem. Recent climate change may affect the geographical distribution of the arthropod vectors for these viruses and so the risk of flavivirus epidemics may increase. Many methods have been developed for the serological diagnosis of flavivirus infections, such as haemagglutination inhibition assay, enzyme-linked immunosorbent assay, and immunofluorescence in staining. However, the specificity of these assays varies.The plaque reduction neutralizing test (PRNT) using live viruses is currently the ‘gold standard’ for the differential serodiagnosis of flaviviruses. The specificity of results obtained with PRNT is better than that for other protocols and many laboratories apply the PRNT protocol to the differential serodiagnosis of flaviviruses. Here, recent refinements to the PRNT protocols with genetically modified recombinant viruses or reporter-harbouring virus-like particles are reviewed. Further, the problems associated with the differential serodiagnosis of flaviviruses using PRNT are discussed.     

Influence of diagnostic tests for estimating prevalence during surveillance programs

The influence of diagnostic tests on the estimation of the prevalence and the calculation of sample sizes with respect to different sampling schemes are presented in this paper. These sampling schemes are used for the implementation of surveillance programs. Assuming "perfect tests" (i.e. sensitivity = specificity = 100%) the calculated sample sizes, e.g. for an IBR/IPV-surveillance or a paratuberculosis survey, are half of the sample sizes considering sensitivity and specificity of the diagnostic tests. However the probability not to identify infected livestocks may be 4 times higher neglecting the test characteristics and assuming perfect tests.     

Comparison and interpretation of diagnostic tests for feline immunodeficiency virus infection.

Feline sera were submitted to the Cornell Feline Health Center (n = 497) or to the New York State Diagnostic Laboratory (n = 1,565) for feline immunodeficiency virus (FIV) testing. Some sera (n = 166) were submitted for confirmation of previous FIV-positive results; 151 of these sera had been tested at the referring veterinary practice or laboratory, using an in-house ELISA. Excluding the samples submitted for confirmation, a total of 173 samples (9.1%) were FIV-positive; 11.6% of the clinically ill or high-risk cats and 0.49% of the healthy, low risk cats were positive for FIV antibody. A commercially available ELISA for detection of antibody to FIV was evaluated in relation to the immunofluorescent antibody (IFA) test and the immunoblot assay. The ELISA was interpreted according to the manufacturer's instructions, with the ratio of sample optical density to positive control optical density (S/P) determining a positive or negative result. The ELISA results based on the S/P interpretation were compared with a kinetics-based (KELA) interpretation of the ELISA. The KELA values were reported as positive, negative, or equivocal. Using the immunoblot as the standard, ELISA (S/P interpretation) had sensitivity of 0.93 and specificity of 0.98, whereas the IFA test had sensitivity of 0.95 and specificity of 0.98. However, the sensitivity and specificity of the ELISA (S/P interpretation) were markedly reduced for sample results falling in the KELA equivocal range, indicating that equivocal results were valid interpretations for some sera. A high number (22.5%) of the samples submitted for confirmation of a positive result from use of the in-house ELISA were determined to be negative for FIV antibody.(ABSTRACT TRUNCATED AT 250 WORDS)     

Epidemiologic issues in the validation of veterinary diagnostic tests

In this review, we critically discuss the objectives, methods and limitations of different approaches for the validation of diagnostic tests. We show (based on published data and our own experiences) that estimates for the diagnostic sensitivity and specificity may vary among populations and/or subpopulations of animals, conditional on the distribution of influential covariates. Additional variability in those parameter estimates may be attributable to the sampling strategy. The uncertainty about diagnostic parameters is of concern for the decision-maker in the context of clinical diagnosis or quantitative risk assessment as well as for the epidemiologist who uses test data for prevalence estimation or risk-factor studies. Examples for the calculation of diagnostic parameters are presented together with bias-avoidance strategies. We suggest guidelines for an epidemiologic approach to test validation of veterinary diagnostic tests.     

Estimation of receiver-operating characteristic curves to determine accuracy of a competitive enzyme-linked immunosorbent assay for the serodiagnosis of Brucella infection in domestic water buffalo (Bubalus bubalis) and cattle

OBJECTIVE: To estimate receiver-operating characteristic (ROC) curves for a competitive ELISA (c-ELISA) that is used in serodiagnosis of brucellosis in water buffalo and cattle, to determine the most appropriate positive cutoff value for the c-ELISA in confirmation of infection, and to evaluate species differences in c-ELISA function. SAMPLE POPULATION: Sera from 4 herds of cattle (n = 391) and 4 herds of water buffalo (381). PROCEDURE: Serum samples were evaluated for Brucella-specific antibodies by use of a c-ELISA. On the basis of previous serologic test results, iterative simulation modeling was used to classify animals as positive or negative for Brucella infection without the use of a gold standard. Accuracy of c-ELISA for diagnosis of infection was compared between cattle and water buffalo by comparison of areas under ROC curves. RESULTS: A positive cutoff value of 30% inhibition for c-ELISA yielded sensitivity and specificity estimates, respectively, of 83.9 and 92.6% for cattle and 91.4 and 95.4% for water buffalo. A positive cutoff value of 35% inhibition yielded sensitivity and specificity estimates, respectively, of 83.9 and 96.2% for cattle and 88.0 and 974% for water buffalo. Areas under ROC curves were 0.94 and 0.98 for cattle and water buffalo, respectively. CONCLUSIONS AND CLINICAL RELEVANCE: ROC curves can be estimated by use of iterative simulation methods to determine optimal cutoff values for diagnostic tests with quantitative outcomes. A cutoff value of 35% inhibition for the c-ELISA was found to be most appropriate for confirmation of Brucella infection in cattle and water buffalo.