Frequentist and Bayesian approaches to prevalence estimation using examples from Johne's disease

Although frequentist approaches to prevalence estimation are simple to apply, there are circumstances where it is difficult to satisfy assumptions of asymptotic normality and nonsensical point estimates (greater than 1 or less than 0) may result. This is particularly true when sample sizes are small, test prevalences are low and imperfect sensitivity and specificity of diagnostic tests need to be incorporated into calculations of true prevalence. Bayesian approaches offer several advantages including direct computation of range-respecting interval estimates (e.g. intervals between 0 and 1 for prevalence) without the requirement of transformations or large-sample approximations, direct probabilistic interpretation, and the flexibility to model in a straightforward manner the probability of zero prevalence. In this review, we present frequentist and Bayesian methods for animal- and herd-level true prevalence estimation based on individual and pooled samples. We provide statistical methods for detecting differences between population prevalence and frequentist methods for sample size and power calculations. All examples are motivated using Mycobacterium avium subspecies paratuberculosis infection and we provide WinBUGS code for all examples of Bayesian estimation.     

Bayesian estimation of diagnostic accuracy of a new bead-based antibody detection test to reveal Toxoplasma gondii infections in pig populations

The success of a Toxoplasma gondii surveillance program in European pig production systems depends partly on the quality of the test to detect infection in the population. The test accuracy of a recently developed serological bead-based assay (BBA) was investigated earlier using sera from experimentally infected animals. In this study, the accuracy of the BBA was determined by the use of sera from animals from two field subpopulations. As no T. gondii infection information of these animals was available, test accuracy was determined through a Bayesian approach allowing for conditional dependency between BBA and an ELISA test. The priors for prevalence were based on available information from literature, whereas for specificity vague non-informative priors were used. Priors for sensitivity were based either on available information or specified as non-informative. Posterior estimates for BBA sensitivity and specificity were (mode) 0.855 (Bayesian 95% credibility interval (bCI) 0.702–0.960) and 0.913 (bCI 0.893–0.931), respectively. Comparing the results of BBA and ELISA, sensitivity was higher for the BBA while specificity was higher for ELISA. Alternative priors for the sensitivity affected posterior estimates for sensitivity of both BBA and ELISA, but not for specificity. Because the difference in prevalence between the two subpopulations is small, and the number of infected animals is small as well, the precision of the posterior estimates for sensitivity may be less accurate in comparison to the estimates for specificity. The estimated value for specificity of BBA is at least optimally defined for testing pigs from conventional and organic Dutch farms.     

A Bayesian approach for the evaluation of six diagnostic assays and the estimation of Cryptosporidium prevalence in dairy calves

The prevalence of Cryptosporidium in calves and the test properties of six diagnostic assays (microscopy (ME), an immunofluorescence assay (IFA), two ELISA and two PCR assays) were estimated using Bayesian analysis. In a first Bayesian approach, the test results of the four conventional techniques were used: ME, IFA and two ELISA. This four-test approach estimated that the calf prevalence was 17% (95% Probability Interval (PI): 0.1-0.28) and that the specificity estimates of the IFA and ELISA were high compared to ME. A six-test Bayesian model was developed using the test results of the 4 conventional assays and 2 PCR assays, resulting in a higher calf prevalence estimate (58% with a 95% PI: 0.5-0.66) and in a different test evaluation: the sensitivity estimates of the conventional techniques decreased in the six-test approach, due to the inclusion of two PCR assays with a higher sensitivity compared to the conventional techniques. The specificity estimates of these conventional assays were comparable in the four-test and six-test approach. These results both illustrate the potential and the pitfalls of a Bayesian analysis in estimating prevalence and test characteristics, since posterior estimates are variables depending both on the data at hand and prior information included in the analysis. The need for sensitive diagnostic assays in epidemiological studies is demonstrated, especially for the identification of subclinically infected animals since the PCR assays identify these animals with reduced oocyst excretion, which the conventional techniques fail to identify.     

On the interpretation of test sensitivity in the two-test two-population problem: Assumptions matter

Bayesian analyses of diagnostic test accuracy often require the assumption of constant test accuracy among populations to ensure model identifiability. In a prior study (Toft, N., Jørgensen, E., Højsgaard, S., 2005. Diagnosing diagnostic tests: evaluating the assumptions underlying the estimation of sensitivity and specificity in the absence of a gold standard. Prev. Vet. Med. 68, 19–33), the sensitivity estimate from a two-test two-population model was shown to be weighted toward the population with the higher prevalence of infection. In the present study, we provided analytical formulae that give insight into the effect of assuming constant sensitivity when this assumption was false. To further investigate the effect of failure of the assumption of constant sensitivity, we also simulated several data sets under the assumption that the first test's sensitivity varied in the two populations. Bayesian conditional independence models that presumed constant sensitivities were implemented in WinBUGS and posterior estimates (mean and 95% probability intervals) were evaluated based on the known true values of the parameters. Findings from the Bayesian analyses of several scenarios indicated that the posterior mean was a good estimate of the weighted mean of the sensitivities in the two populations, when one test was perfectly specific. When neither test was perfectly specific, the Bayesian posterior mean for test 1 sensitivity was either greater than the larger of the two true sensitivities, or smaller than both, and estimates of prevalence and the second test's specificity were incorrect. The implication is that estimates of some parameters will be biased if test sensitivities are not constant across populations. Without a perfectly specific test, and if the assumption of constant sensitivity fails, the only solution we are aware of would involve incorporating prior information on at least two parameters.     

Rational use of diagnostic and screening tests

Veterinarians have a vast and ever-expanding array of diagnostic tests available to them. However, this abundance can be an embarrassment of riches that confounds diagnosis and undermines patient care if we do not make critical and informed decisions about the selection and interpretation of the tests we employ. Effective use of diagnostic tests requires a deliberate and informed approach. We must consider the strengths and weaknesses of the tests themselves and the clinical context, and we must be wary of the many biases that skew our use and interpretation of diagnostic tests. Understanding sensitivity and specificity, likelihood, prevalence and predictive value, the basic principles of Bayesian reasoning, and the cognitive biases that drive inappropriate testing are all critical to ensuring our use of imaging and laboratory testing improves patient outcomes.     

Effect of diagnostic testing error on intracluster correlation coefficient estimation

Estimation of the intracluster correlation coefficient (ICC) for infectious animal diseases may be of interest for survey planning and for calculating variance inflation factors for estimators of prevalence. Typically, diagnostic tests with imperfect sensitivity and specificity are used in surveys. In such studies, where animals from multiple herds are tested, the ICC often is estimated using apparent (test-based) rather than true prevalence data. Through Monte Carlo simulation, we examined the effect of substituting diagnostic test outcomes for true infection status on an ANOVA estimator of ICC, which was designed for use with true infection status data. We considered effects of diagnostic test sensitivity and specificity on the estimated ICC when the true ICC value and infection status of the sampled individuals were known. The ANOVA estimator underestimated the true ICC when the diagnostic test was imperfect. We also demonstrated, under the beta-binomial model, that the ICC based on apparent infection status for individuals is < or = ICC based on true infection status. In addition, we propose a Bayesian model for estimating the ICC that incorporates imperfect sensitivity and specificity and illustrate the Bayesian model using a simulation study and one example; a seroprevalence survey of ovine progressive pneumonia in U.S. sheep flocks.     

A comparison of methods to estimate the prevalence of ovine Johne's infection from pooled faecal samples

OBJECTIVE: To compare estimates of ovine Johne's infection prevalence produced by several alternate methods based on pooled faecal culture (PFC) results with prevalence estimates based on individual faecal culture (IFC). PROCEDURE: Seven methods for estimating prevalence of infection based on PFC results were incorporated in a computer program, including methods for imperfect test sensitivity and specificity, for variable pool size and a Bayesian method that incorporates prior knowledge about test performance and prevalence. These methods were then used to analyse PFC data at one observation 30 months post-vaccination in a field trial of a killed vaccine for the control of OJD, undertaken on three farms in New South Wales. RESULTS: Prevalence estimates, for three methods that assume a perfect test, were close to the IFC estimate, whereas for three other methods that assume an imperfect test, the estimated prevalence was generally higher than the IFC estimate. In comparison, the Bayesian approach produced more variable estimates that were substantially higher than the IFC estimate when an inappropriately high prior estimate of prevalence was used. CONCLUSION: Despite the limitations of each method, two methods provided accurate and reasonable estimates of the prevalence assessed by IFC in all instances, and are appropriate for the analysis of data from this vaccine trial. One of these methods also has the advantage of allowing for variable pool size. However, further research is needed to develop a method that will simultaneously account for variation in pool size and in test sensitivity and specificity.     

Estimation of sensitivity and specificity of two diagnostics tests for bovine immunodeficiency virus using Bayesian techniques

The validation of assays for bovine immunodeficiency virus (BIV) in cattle is hampered by the absence of a gold standard. Two tests that often are used to detect BIV are the indirect fluorescent-antibody assay (IFA) and the nested-set polymerase chain-reaction assay (PCR). IFA detects an antibody response whereas PCR detects the provirus in white blood cells.Using Bayesian techniques performed simultaneously on animals from two different dairy herds, we estimated the performance of the IFA and PCR assays and infection prevalence. Bayesian techniques also were used to derive posterior distributions of sensitivities, specificities, and prevalences. The Bayesian estimates were IFA sensitivity=60%, IFA specificity=88%, PCR sensitivity=80%, PCR specificity=86%, Herd A prevalence=20%, and Herd B prevalence=71%. Although PCR was the more sensitive assay, substantial misclassification of infection would be expected in epidemiological studies of BIV regardless of which assay was used.     

An update on FIV and FeLV test performance using a Bayesian statistical approach

BACKGROUND: Screening tests for feline retroviruses are thought to have high sensitivity and specificity, although previous studies that evaluated these tests have limitations. Novel statistical approaches have been developed that allow the estimation of sensitivity and specificity in situations where the true state of the disease in individual animals cannot be assured. OBJECTIVE: The purpose of this study was to evaluate the sensitivity and specificity of a variety of retrovirus tests, including some screening tests, in a population of cats potentially infected with either feline leukemia virus (FeLV) and/or feline immunodeficiency virus (FIV) by using a Bayesian statistical approach. METHODS: Four hundred and ninety blood samples from cats being evaluated for FIV infection were tested by 2 rapid immunomigration tests (Witness single [WS], Witness combi [WC]) and a plate-based ELISA (Petcheck) for FIV antibody, and by a newly designed real-time polymerase chain reaction (PCR) assay for FIV provirus. Four hundred and ninety-five blood samples from cats being evaluated for FeLV infection were tested by 2 rapid immunomigration tests (WS, WC) and a plate-based ELISA (Petcheck) for FeLV antigen, and by a FeLV virus isolation technique. Results were then analyzed by using a Bayesian statistical method. RESULTS: For FIV tests, median sensitivity estimates were 0.98 for WS, 0.97 for WC, 0.98 for ELISA, and 0.92 for PCR. Median specificity estimates were 0.96 for WS, 0.96 for WC, 0.93 for ELISA, and 0.99 for PCR. For FeLV tests, median sensitivity estimates were 0.97 for WS, 0.97 for WC, 0.98 for ELISA, and 0.91 for virus isolation. Median specificity estimates were 0.96 for WS, 0.96 for WC, 0.98 for ELISA, and 0.99 for virus isolation. CONCLUSIONS: The use of Bayesian statistical methods overcomes a variety of methodologic problems associated with diagnostic test evaluations, including the lack of a definitive reference test. The sensitivity and the specificity of all 6 evaluated screening tests was high: however, specificity estimates were slightly lower than those reported by most recent studies.     

Bayesian mixture models for within-herd prevalence estimates of bovine paratuberculosis based on a continuous ELISA response

Diagnostic inference by use of assays such as ELISA is usually done by dichotomizing the optical density (OD)-values based on a predetermined cut-off. For paratuberculosis, a slowly developing infection in cattle and other ruminants, it is known that laboratory factors as well as animal specific covariates influence the OD-value, but while laboratory factors are adjusted for, the animal specific covariates are seldom utilized when establishing cut-offs. Furthermore, when dichotomizing an OD-value, information is lost. Considering the poor diagnostic performance of ELISAs for diagnosis of paratuberculosis, a framework for utilizing the continuous OD-values as well as known coavariates could be useful in addition to the traditional approaches, e.g. for estimating within-herd prevalences.

The objective of this study was to develop a Bayesian mixture model with two components describing the continuous OD response of infected and non-infected cows, while adjusting for known covariates. Based on this model, four different within-herd prevalence indicators were considered: the mean prevalence in the herd; the age adjusted prevalence of the herd for better between-herd comparisons; the rank of the age adjusted prevalence to better compare across time; and a threshold-based prevalence to describe differences between herds. For comparison, the within-herd prevalence and associated rank using a traditional dichotomization approach based on a single cut-off for an OD corrected for laboratory variation was estimated in a Bayesian model with priors for sensitivity and specificity.

The models were applied to the OD-values of a milk ELISA using samples from all lactating cows in 100 Danish dairy herds in three sampling rounds 13 months apart. The results of the comparison showed that including covariates in the mixture model reduced the uncertainty of the prevalence estimates compared to the cut-off based estimates. This allowed a more informative ranking of the herds where low ranking and high ranking herds were easier to identify.     

A practical approach to calculate sample size for herd prevalence surveys

When designing a herd-level prevalence study that will use an imperfect diagnostic test, it is necessary to consider the test sensitivity and specificity. A new approach was developed to take into account the imperfections of the test. We present an adapted formula that, when combined with an existing piece of software, allows improved planning. Bovine paratuberculosis is included as an example infection because it originally stimulated the work. Examples are provided of the trade-off between the benefit (low number of herds) and the disadvantage (large number of animals per herd and exclusion of small herds) that are associated with achieving high herd-level sensitivity and specificity. We demonstrate the bias in the estimate of prevalence and the underestimate of the confidence range that would arise if we did not account for test sensitivity and specificity.     

Comparison of serologic tests for detection of Brucella infections in cattle and water buffalo (Bubalus bubalis)

OBJECTIVE: To estimate sensitivity and specificity of 4 commonly used brucellosis screening tests in cattle and domestic water buffalo of Trinidad, and to compare test parameter estimates between cattle and water buffalo. ANIMALS: 391 cattle and 381 water buffalo. PROCEDURE: 4 Brucella-infected herds (2 cattle and 2 water buffalo) and 4 herds (2 of each species) considered to be brucellosis-free were selected. A minimum of 100 animals, or all animals > 1 year of age, were tested from each herd. Serum samples were evaluated for Brucella-specific antibodies by use of standard plate agglutination test (SPAT), card test (CT), buffered plate agglutination test (BPAT), and standard tube agglutination test (STAT). A Bayesian approach was used to estimate sensitivity and specificity of diagnostic tests without the use of a gold standard, assuming conditional independence of tests. RESULTS: Sensitivity and specificity estimates in cattle, respectively, were SPAT, 66.7 and 98.9; CT, 72.7 and 99.6; BPAT, 88.1 and 98.1; and STAT, 80.2 and 99.3. Corresponding test estimates in water buffalo, respectively, were SPAT, 51.4 and 99.3; CT, 90.4 and 99.4; BPAT, 96.3 and 90.7; and STAT, 75.0 and 98.8. Sensitivity of the CT and specificity of the BPAT were different between cattle and water buffalo with at least 95% probability. CONCLUSIONS AND CLINICAL RELEVANCE: Brucellosis serologic test performance varied by species tested, but BPAT had the highest sensitivity for screening cattle and water buffalo. Sensitivity and specificity of more than 2 screening tests can be estimated simultaneously without a gold standard by use of Bayesian techniques.     

Strategies for two-stage sampling designs for estimating herd-level prevalence

We propose a herd-level sample-size formula based on a common adjustment for prevalence estimates when diagnostic tests are imperfect. The formula depends on estimates of herd-level sensitivity and specificity. With Monte Carlo simulations, we explored the effects of different intracluster correlations on herd-level sensitivity and specificity. At low prevalence (e.g. 1% of animals infected), herd-level sensitivity increased with increasing intracluster correlation and many herds were classified as positive based only on false-positive test results. Herd-level sensitivity was less affected at higher prevalence (e.g. 20% of animals infected). A real-life example was developed for estimating ovine progressive pneumonia prevalence in sheep. The approach allows researchers to balance the number of herds and the total number of animals sampled by manipulating herd-level test characteristics (such as the number of animals sampled within a herd).     

Bayes and diagnostic testing

Interpretation of the result of a diagnostic test depends not only on the actual test result(s) but also on information external to this result, namely the test's sensitivity and specificity. This external information (also called prior information) must be combined with the data to yield the so-called updated, posterior estimates of the true prevalence and the test characteristics. The Bayesian approach offers a natural, intuitive framework in which to carry out this estimation process. The influence of the prior information on the final result may not be ignored. Guidance for the choice of prior information not in conflict with the data can be obtained from a set of statistics and indices (DIC, p(D), Bayes-p).     

Bayesian estimation of true between-herd and within-herd prevalence of Salmonella in Danish veal calves

Specialised veal producers that purchase and raise calves from several dairy herds are potentially at high risk of delivering Salmonella-infected animals to slaughter. However, the true prevalence of Salmonella infected veal producing herds and the prevalence of infected calves delivered to slaughter from infected herds are unknown in Denmark. Due to uncertainties about test sensitivity and specificity, these prevalences are not straightforward to assess. The objective of this study was to estimate the within-herd- and between-herd prevalence of Salmonella in veal calves delivered for slaughter to abattoirs in Denmark. Furthermore, it was investigated to which extent the estimates differed between a setup using both serological tests and faecal culture, compared to just serological tests, and whether the applied sampling scheme in the national surveillance programme in Denmark was sufficient to establish high posterior estimates of freedom from infection in individual herds. We used Bayesian analysis to avoid bias as a result of fixed test validity estimates. Serological test results from 753 animals and faecal culture from 1233 animals from 68 randomly selected Danish veal producing herds that delivered more than 100 calves to slaughter per year were used to estimate the prevalences and estimates of freedom from Salmonella. Serological test results of 7726 animals from 185 herds were used to compare the difference in prevalence estimates between serology alone vs. faecal culture combined with serology. We estimated that 34-57% of specialised veal producing herds were infected with Salmonella. Within the infected herds, 21-49% of the animals were infected. Few herds obtained high posterior estimates for the probability of freedom from infection given the collected data, with only six of 68 herds obtaining posterior probability of being infected less than 10%. Furthermore, this study indicated that serology is sufficiently sensitive and specific to be used for estimating the prevalence of Salmonella-infected specialised veal producing herds.     

Evaluation of three commercial enzyme-linked immunosorbent assays for the detection of antibodies against Salmonella spp. in meat juice from finishing pigs in Spain

The control of animal salmonellosis is considered as a major objective in Europe and indirect ELISAs will be important tools for the implementation of control programs for this infection in pigs. We analyse the results yielded by three commercial ELISAs (Herdcheck Swine Salmonella, SALMOTYPE Pig Screen, and PrioCHECK Salmonella) on meat juice samples from a population of slaughter pigs of Aragon, NW Spain, to assess their efficacy using traditional and latent-class approaches. Overall, the Herdcheck Swine Salmonella detected more Salmonella-infected pigs than the other two tests, but its relative sensitivity was low (65.9%). A similar result was observed when only serotypes detectable by this test were considered (69.1%). When a Bayesian approach was used the Herdcheck Swine Salmonella showed also the highest overall accuracy (sensitivity = 88% and specificity = 74%). Our results suggest that a relatively small proportion of the observed prevalence in herds would be explained by using these ELISAs. Also, this study points out that when different ELISA tests are used within the same herd, results may differ substantially. Thus, caution is advised if it is decided to use these assays for herd health classification in Spanish Salmonella control programs.     

Validation of the indirect fluorescent antibody and the complement fixation tests for the diagnosis of Theileria equi

The indirect fluorescent antibody (IFA) test for Theileria equi was evaluated to assess test's suitability for the serological diagnosis of equine piroplasmosis, to provide performance parameters for the purpose of test validation, and to compare it with the complement fixation (CF) test. Using a protocol that included Evan's blue, the specificity of the IFA test was estimated at 99.0% for T. equi by the classical method of analysis, and 96.6% by the Bayesian method. The use of Evan's blue in the test protocol increased test specificity and contributed to an excellent test agreement between two collaborating laboratories (kappa = 0.96). Using Bayesian analysis, the sensitivity estimate for the IFA test was 89.2%. The CF test sensitivity and specificity estimates for T. equi were 63.1 and 96.4%, respectively, as determined by Bayesian analysis. The IFA test was more sensitive than the CF test but the specificity estimates were similar.     

Epidemiologic issues in the validation of veterinary diagnostic tests

In this review, we critically discuss the objectives, methods and limitations of different approaches for the validation of diagnostic tests. We show (based on published data and our own experiences) that estimates for the diagnostic sensitivity and specificity may vary among populations and/or subpopulations of animals, conditional on the distribution of influential covariates. Additional variability in those parameter estimates may be attributable to the sampling strategy. The uncertainty about diagnostic parameters is of concern for the decision-maker in the context of clinical diagnosis or quantitative risk assessment as well as for the epidemiologist who uses test data for prevalence estimation or risk-factor studies. Examples for the calculation of diagnostic parameters are presented together with bias-avoidance strategies. We suggest guidelines for an epidemiologic approach to test validation of veterinary diagnostic tests.