Method comparison in the clinical laboratory

Studies comparing a new method with an established method, to assess whether the new measurements are comparable with existing ones, are frequently conducted in clinical pathology laboratories. Assessment usually involves statistical analysis of paired results from the 2 methods to objectively investigate sources of analytical error (total, random, and systematic). In this review article, the types of errors that can be assessed in performing this task are described, and a general protocol for comparison of quantitative methods is recommended. The typical protocol has 9 steps: 1) state the purpose of the experiment, 2) establish a theoretical basis for the method comparison experiment, 3) become familiar with the new method, 4) obtain estimates of random error for both methods, 5) estimate the number of samples to be included in the method comparison experiment, 6) define acceptable difference between the 2 methods, 7) measure the patient samples, 8) analyze the data, and 9) judge acceptability. The protocol includes the essential investigations and decisions needed to objectively assess the overall analytical performance of a new method compared to a reference or established method. The choice of statistical methods and recommendations of decision criteria within the stages are discussed. Use of the protocol for decision-making is exemplified by the comparison of 2 methods for measuring alanine aminotransferase activity in serum from dogs. Finally, a protocol for comparing simpler semiquantitative methods with established methods that measure on a continuous scale is suggested.     

Enzyme-linked immunosorbent assay, single radial immunodiffusion, and indirect methods for the detection of failure of transfer of passive immunity in dairy calves

BACKGROUND: Confirmatory tests for failure of transfer of passive immunity (FTPI) in dairy calves require direct measurements of the serum immunoglobulin G concentration. Enzyme-linked immunosorbent assay (ELISA) has advantages over single radial immunodiffusion (SRID) in terms of cost and time. OBJECTIVES: To evaluate the agreement between ELISA and SRID, and to compare the diagnostic performance of ELISA with indirect methods, in the detection of FTPI in calves. ANIMALS: One hundred and fifteen dairy calves (aged 0-10 days) from 23 calf-rearing facilities. METHODS: Prospective, observational study. The agreement between SRID and ELISA was determined by the Bland-Altman method. Fixed bias (SRID - ELISA) was calculated. For comparison of the diagnostic performance of ELISA with indirect methods, sensitivity, specificity, and area under the curve (AUC) of receiver operating characteristic (ROC) curves were calculated at cut-off values of 500 and 1,000 mg/dL. RESULTS: The agreement between SRID and ELISA was 94%. Fixed bias (SRID - ELISA) was 140 +/- 364 mg/dL. The AUC and sensitivity of ELISA at the cut-off value of 1,000 mg/dL were higher than those of indirect methods (P<.004). The specificity of ELISA at the cut-off value of 1,000 mg/dL was not higher than that of indirect methods, except for serum total protein concentration assay. CONCLUSION AND CLINICAL IMPORTANCE: ELISA exhibited good diagnostic performance and good agreement with SRID. ELISA is an adequate method for both screening and confirmatory tests for FTPI in dairy calves at the cut-off value of 500 mg/dL.     

Critically appraising studies reporting assessing diagnostic tests.

Studies that report the sensitivity and specificity of diagnostic tests are susceptible to flaws that can introduce bias and lead to incorrect estimates. This article uses the quality assessment of diagnostic accuracy studies checklist to describe how to appraise a study reporting diagnostic test comparisons critically. The article also contains a glossary of terms that are useful in discussions about diagnostic tests.     

Computed tomographic arthrography is a useful adjunct to survey computed tomography and arthroscopic evaluation of the canine shoulder joint

The aim of this retrospective, methods comparison study was to assess the diagnostic utility of computed tomographic arthrography in the assessment of various intraarticular shoulder pathologies in dogs in comparison with survey computed tomography (CT), using arthroscopic examination as the reference standard. Computed tomography, computed tomographic arthrography, and arthroscopic findings of 46 scapulohumeral joints of dogs with forelimb lameness were reviewed retrospectively. Predefined sites were assessed for the presence or absence of disease. If a lesion was present, a prespecified pathology was designated. Computed tomographic arthrography was found to be a safe technique which provided a superior diagnostic efficacy relative to survey CT for the assessment of the biceps tendon and biceps tendon sheath (sensitivity 71%, specificity 75%, positive likelihood ratio 2.9, negative likelihood ratio 0.38) and humeral head cartilage (sensitivity 65%, specificity 97%, positive likelihood ratio 19, negative likelihood ratio 0.37). Computed tomography and computed tomographic arthrography provided additional diagnostic information to arthroscopy in regard to osteophytosis, subchondral defects, and joint mice. Computed tomographic arthrography alone was of limited diagnostic value for assessment of the medial and lateral glenohumeral ligaments (sensitivity 13% and 0%, specificity 1% and 78%, positive likelihood ratios unmeasurable and 0, negative likelihood ratios 0.88 and 1.29, respectively) and the subscapularis tendon (sensitivity 14%, specificity 98%, positive likelihood ratio 5.7, negative likelihood ratio 0.88). Computed tomographic arthrography is therefore a useful adjunct to survey CT and arthroscopic evaluation of the canine shoulder joint, however, is not a replacement for these techniques.     

ASVCP guidelines: allowable total error guidelines for biochemistry

As all laboratory equipment ages and contains components that may degrade with time, initial and periodically scheduled performance assessment is required to verify accurate and precise results over the life of the instrument. As veterinary patients may present to general practitioners and then to referral hospitals (both of which may each perform in‐clinic laboratory analyses using different instruments), and given that general practitioners may send samples to reference laboratories, there is a need for comparability of results across instruments and methods. Allowable total error (TE_a) is a simple comparative quality concept used to define acceptable analytical performance. These guidelines are recommendations for determination and interpretation of TE_a for commonly measured biochemical analytes in cats, dogs, and horses for equipment commonly used in veterinary diagnostic medicine. TE_a values recommended herein are aimed at all veterinary settings, both private in‐clinic laboratories using point‐of‐care analyzers and larger reference laboratories using more complex equipment. They represent the largest TEa possible without generating laboratory variation that would impact clinical decision making. TE_a can be used for (1) assessment of an individual instrument's analytical performance, which is of benefit if one uses this information during instrument selection or assessment of in‐clinic instrument performance, (2) Quality Control validation, and (3) as a measure of agreement or comparability of results from different laboratories (eg, between the in‐clinic analyzer and the reference laboratory). These guidelines define a straightforward approach to assessment of instrument analytical performance.     

Ultrasonographic intestinal muscularis thickening in dogs with histologically confirmed inflammatory bowel disease: 13 cases (2010–2021)

Ultrasonographic intestinal muscularis thickening has not been described as an imaging feature of canine inflammatory bowel disease. In this retrospective case series, patients were identified by searching sonographic reports for “muscularis” and/or “muscular layer.” Patients were included if small intestinal muscularis thickening was reported, and sonographic images and histopathological samples of the small intestine were available for review. Cases with small intestines nodules, masses, or complete loss of wall layering were excluded. Sonographic images were retrospectively evaluated for jejunal muscularis layer thickness, and ratios of intestinal layer measurements were performed. Histological samples were retrospectively reviewed. Thirteen dogs met inclusion criteria: all dogs had sonographic intestinal muscularis thickening relative to the submucosa (>1.0, range of 1.3–2.5), and most dogs had muscular layer thickness above normal published ranges (11/13; all 13/13 above the weight-specific mean). More than half of the patients had overall normal wall thickness (11/13) and several had normal mucosal echogenicity (6/13). Therefore, in some dogs, the only sonographic abnormality in the small intestine was muscularis thickening. No dogs had lymphadenomegaly. Endoscopic partial-thickness (n = 11, duodenum and/or ileum) or surgical full-thickness (n = 2) samples confirmed inflammatory bowel disease. Direct comparison between jejunum sonographic characteristics and histology features was limited due to both partial thickness biopsies and lack of direct comparison between anatomical locations of ultrasonographic assessment and biopsy site. However, no cases that met the inclusion criteria had normal small intestinal histology. Comparable to cats, dogs with ultrasonographic intestinal muscularis thickening may have inflammatory bowel disease, and further workup for enteropathy is indicated.     

Diagnostic imaging – evaluating image quality using visual grading characteristic (VGC) analysis

Radiologists are regularly faced with the task of comparing image quality obtained using different imaging systems or settings. Visual grading techniques can be used to evaluate the quality of images by grading the clarity of reproduction of anatomical or pathological structures. The methods, which include “visual grading analysis (VGA)” and the “image criteria (IC) study”, are characterised by their attractive simplicity and reliability. Non-parametric rank-invariant statistical methods are suitable techniques for statistical analysis of VGA-data. Båth and Månsson (2007) introduced such a method and termed it “visual grading characteristics (VGC) analysis”. This paper gives an overview of the principle together with an example of its use in veterinary radiology. The aim of this review article is to encourage veterinary researchers to apply this method which has proven valuable in the human field. Basically, the method can also be applied for the analysis of other categories of images (e.g. histological sections, cytological smears) in cases where the task is to evaluate features subjectively on the basis of a score, allowing some degree of freedom of decision. Furthermore, the aim of the investigation is not necessarily restricted to quality aspects. Other questions such as the effects of treatment options on the appearance of certain structures can be compared as well.     

Relationships between young stallions’ temperament and their behavioral reactions during standardized veterinary examinations

Horse handling and veterinary examination can induce hazardous stress reactions. Such reactions occur especially in young and less-trained horses, particularly stallions, and make their handling a risk for breeders, grooms, and medical staff. Moreover, these stressful situations will affect the animal’s health and welfare. Because stress reactivity is thought to be partly determined by genetic factors, scientists, veterinarians, and breeders are likely to be interested in adding temperament assessments to stallion selection schemes, as it is already done in some countries. This study assesses young stallions’ temperament and its comparison with their stress reactions during a standardized veterinary examination for studbook admission. The assessment consists of a general examination, a lameness examination including flexion tests, an endoscopy of the upper airway, and a standardized radiological examination. During the years 2008 and 2009, 93 stallions were evaluated. Stallions were observed from the moment they were unloaded from the trailer at the clinic until the end of veterinary examinations. In addition to the behavioral observations made by the experimenter, each staff member in charge of the examination filled in a short questionnaire about the horse’s temperament and the “easiness of manipulation” for the performed examinations. Breeders were asked to complete a longer questionnaire about their horse’s temperament. The assessments of “aggressiveness,” “sociability,” and “learning level” temperament traits were the most consistent, as shown by the significant Spearman correlations between judges’ assessments. Undesirable behaviors during veterinary examinations leading to handling difficulties were associated with a low “easiness of manipulation” score assessed by the clinical staff. These low “easiness of manipulation” scores were positively correlated to temperament traits such as “anxiousness” and “aggressiveness” and negatively correlated to others such as “sociability” or “learning level.” Temperament assessment and behavioral observations can therefore be used to anticipate behaviors that make a horse difficult to handle during veterinary examinations. Thus, it may be important to include temperament assessment as a feature in the selection of breeding stallions—as already practiced for some breeds in some countries. Such evaluations may promote the welfare of horses and ease of handling as well as safety for the handler.     

Issues and pitfalls in method comparison studies

Method comparison studies are needed for validation of new methods of measurements, for example, non‐invasive blood pressure measurements against standard reference methods. After a brief introduction into method comparison studies, this paper is organized in three sections. The first section deals with the widely, though not always appropriately, used classical Bland‐Altman plot with the limits of agreement and its extensions with non‐constant bias and multiple observations. The second section comments on other statistical approaches including correlation coefficients, linear regressions and sensitivities and specificities which are sometimes seen in method comparison studies. The third section proposes the usage of linear mixed effects models as a flexible way to deal with questions associated with method comparison studies.     

Development of a real-time polymerase chain reaction assay for rapid diagnosis of neuropathogenic strains of equine herpesvirus-1.

This communication reports the development and performance assessment of a rapid diagnostic test for identifying horses actively infected with the neurovirulent pathotype of equine herpesvirus-1 (EHV-1). The test is a real-time polymerase chain reaction (PCR)-based assay that uses EHV-1 pathotype-specific TaqMan(R) reporter probes for discrimination between neuropathogenic and non-neuropathogenic strains of EHV-1 in equine blood or nasal swabs. The diagnostic performance of the new technique was evaluated by testing specimens collected from 234 horses involved in recent outbreaks of EHV-1 myeloencephalopathy at three separate thoroughbred racetracks and one large riding/boarding stable. Side-by-side comparison of the EHV-1 pathotyping results yielded by the new single-step, PCR-based allelic discrimination technique (24-hour turn-around-time) with those generated by a multi-step, conventional nested PCR followed by nucleotide sequencing of the amplified DNA (4-day turn-around-time) revealed complete agreement between the 2 test methods. The ability to rapidly identify horses infected with neuropathogenic strains of EHV-1 using a single-step, PCR-based method has significant implications for future diagnostic evaluation of suspect animals.     

Using the correct regression analysis technique in method comparison studies

Regression analysis frequently is used to evaluate a new clinical laboratory method. Results from the new method are compared to results from an existing more established method. If measurement error exists in the established method, then least squares may not be an appropriate statistical method to use for the regression analysis. An errors variable regression analysis model was used to evaluate data from five method comparison studies. Results were compared to least squares analyses performed on the same data. When significant measurement error existed in the "reference" method, the errors variable analysis provided a less biased estimate of the regression statistics, which differed markedly from the least squares results. Inappropriate use of least squares in method comparison studies can lead to erroneous conclusions about the relationship of a new analytical method to an existing method.     

Development of a bead-based multiplex PCR assay for the simultaneous detection of multiple Mycoplasma species

We describe the development and analytical validation of a 7-plex polymerase chain reaction assay coupled to a bead-based liquid suspension array for detection of multiple ruminant Mycoplasma spp. The assay employs a combination of newly designed and previously validated primer-probe sets that target genetic loci specific for Mycoplasma bovis, Mycoplasma mycoides cluster, Mycoplasma mycoides subsp. mycoides SC (MmmSC) and Mycoplasma capricolum subspecies capripneumoniae (Mccp). Analytical sensitivity for the targeted Mycoplasma species ranged from 10 fg to 1 pg of purified gDNA extracted from broth cultures (approximately 8-800 MmmSC genome equivalents). In silico comparison of primers and probes, and analytical assessment with a range of near-neighbor Mycoplasma species and multiple bacterial respiratory pathogens demonstrated 100% analytical specificity of the assay. To assess assay performance and diagnostic specificity, 192 bovine respiratory samples were analyzed by incorporating a high throughput DNA extraction platform. The assay correctly classified all samples as negative for MmmSC or Mccp. All 33 field samples confirmed as positive for M. bovis by sequencing the uvrC gene were positive in the assay. The results from this study indicate that the bead-based liquid suspension array will provide a reliable, analytically sensitive and specific platform to simultaneously interrogate ruminant respiratory samples for multiple Mycoplasma species, including M. mycoides cluster organisms that are exotic to the United States. Sequential addition of primer-probe sets to the assay did not significantly impact analytical sensitivity of individual primer-probe combinations, suggesting that expanding the assay to include more Mycoplasma species will not compromise overall performance.     

Evaluation of diagnostic tests in the absence of a gold standard using an Anaplasma marginale field data set

The evaluation of newly developed diagnostic tests (tests) commonly involves the comparison of the test outcomes (pos/neg.) of a sample of animals to those of a reference test (gold standard) in order to derive sensitivity and specificity estimates. Often, however, new tests have to be evaluated against an imperfect reference test since a true gold standard test is either too expensive or too costly to apply. This results in bias in the test characteristic estimates. To solve this problem, latent class and Bayesian models can be used to estimate sensitivity and specificity when evaluating a diagnostic test in the absence of a gold standard. They require at least two imperfect reference tests applied to all individuals in the study. In our approach we used a two-test two-population scenario. Both the gold standard and these modelling approaches rely on various assumptions. When violated, biased results will be obtained. The analysis of field data from an Anaplasma marginale outbreak in cattle in Switzerland with four diagnostic procedures (detection of the agent, serology, PCR and hematocrit measurements) was used as a practical example to demonstrate and critically discuss the approaches taken. In this relatively small data set (n = 275) the estimates for the test characteristics obtained by the different methods were quite similar. Overall, the bias in the point estimates depended mainly on the chosen estimation approach. All tests showed a non-negligible correlation mainly in the test sensitivities. This emphasizes the importance of taking into account test dependence even if it seems not biologically plausible at first thought.     

Estimation of sensitivity and specificity of diagnostic tests and disease prevalence when the true disease state is unknown

The performance of a new diagnostic test is frequently evaluated by comparison to a perfect reference test (i.e. a gold standard). In many instances, however, a reference test is less than perfect. In this paper, we review methods for estimation of the accuracy of a diagnostic test when an imperfect reference test with known classification errors is available. Furthermore, we focus our presentation on available methods of estimation of test characteristics when the sensitivity and specificity of both tests are unknown. We present some of the available statistical methods for estimation of the accuracy of diagnostic tests when a reference test does not exist (including maximum likelihood estimation and Bayesian inference). We illustrate the application of the described methods using data from an evaluation of a nested polymerase chain reaction and microscopic examination of kidney imprints for detection of Nucleospora salmonis in rainbow trout.     

Development of a multiplex assay for the detection of antibodies to Borrelia burgdorferi in horses and its validation using Bayesian and conventional statistical methods

Lyme disease is a zoonotic, vector-borne disease and occurs in mammals including horses. The disease is induced by infection with spirochetes of the Borrelia burgdorferi sensu lato group. Infection of mammalian hosts requires transmission of spirochetes by infected ticks during tick bites. Lyme disease diagnosis is based on clinical signs, possible exposure to infected ticks, and antibody testing which is traditionally performed by ELISA and Western blotting (WB). This report describes the development and validation of a new fluorescent bead-based multiplex assay for the detection of antibodies to B. burgdorferi outer surface protein A (OspA), OspC and OspF antigens in horse serum. Testing of 562 equine sera was performed blindly and in parallel by using WB and the new multiplex assay. Because a true gold standard is missing for Lyme antibody testing, we performed and compared different statistical approaches to validate the new Lyme multiplex assay. One approach was to use WB results as a 'relative gold standard' in ROC-curve and likelihood-ratio analyses of the new test. Cut-off values and interpretation ranges of the multiplex assay were established by the analysis. The second statistical approach used a Bayesian model for the calculation of diagnostic sensitivities and specificities of the multiplex assay. The Bayesian analysis takes into consideration that no true gold standard exists for detecting antibodies to B. burgdorferi and estimated sensitivities and specificities of both tests that were compared. Therefore, the Bayesian analysis also resulted in an evaluation of diagnostic sensitivity and specificity of WB. Overall, the new assay was characterized by low background values and a wide dynamic quantification range for the detection of antibodies to OspA, OspC and OspF antigens of B. burgdorferi. The diagnostic sensitivity and specificity for the OspA bead-based assay were calculated as 49% and 85%, respectively, and by a standard ROC curve analysis only because the Bayesian model could not be run on this parameter. The Bayesian-derived diagnostic sensitivities of the OspC and OspF assays were 80% and 86%, respectively. For comparison, the Bayesian-derived estimates for WB resulted in sensitivities of 72% for OspC and 80% for OspF. The Bayesian diagnostic specificities of the multiplex assay were 79% and 69% for OspC and OspF, respectively. WB analysis had specificities of 92% for OspC and 77% for OspF. Although the analysis of a new assay in the absence of a true gold standard remains challenging, the approach used here can help to address this problem when new technologies and traditionally used test standards differ significantly in their analytical sensitivities, which consequently causes problems in the calculation of diagnostic sensitivity and sensitivity values for the new assay. In summary, the new multiplex assay for the detection of antibodies to B. burgdorferi OspA, OspC and OspF antigens in horse serum has improved analytical and diagnostic sensitivities compared to WB analysis. Multiplex analysis is a valuable quantitative tool that simultaneously detects antibodies indicative for natural infection with and/or vaccination against the Lyme pathogen.     

Effect of gamma-irradiation on serum samples on the diagnostic performance of ELISA methods for the detection of trypanosomal antibodies

The study investigated the effect of gamma-irradiation on bovine serum samples on the ability of enzyme-linked immunosorbent assay (ELISA) methods to detect trypanosomal antibodies. The serum samples were analysed using two standardised indirect ELISA systems. Higher measurement values were observed for most gamma-irradiated antibody positive and negative test samples. Using cut-off points, determined from the analysis of a non-irradiated trypanosomal antibody-negative population, the gamma-irradiated sera data showed that there was an increased risk of misclassifying samples as false positive or cross-reactive due to increased analytical sensitivity and decreased analytical specificity. The intraplate precision and agreement between tested and expected values of measurements were not altered throughout. The impact on the assays' diagnostic performance was estimated by analysing diagnostic sensitivity, diagnostic specificity and related parameters. The data demonstrated that although there was a bias of higher measurement values after gamma-irradiation, this could be compensated after readjustment of the cut-off points to obtain best separation of antibody-positive and -negative samples. Thus, for each assay, no significant difference of the diagnostic proficiency was found before and after gamma-irradiation. The practical implications are discussed of a serum sterilisation procedure using (60)Co gamma-rays for routine sample testing, assay validation and trypanosomosis monitoring and tsetse-fly control and eradication programmes.     

Validation of molecular-diagnostic techniques in the parasitological laboratory

Diagnostic laboratories today often operate according to standard quality management procedures such as ISO/IEC 17025. This requires that only validated methods are used. Validation procedures help to document that a particular protocol used by the accredited laboratory has a guaranteed performance in that particular laboratory. Several study designs exist for validation procedures. Computer programmes are available to help with the statistical analysis of validation results. The agreement beyond chance of results obtained in the protocol that is to be validated can be compared to those achieved in an already established test (agreement). For a method that is used under routine conditions or for epidemiological studies, it is necessary to assess the diagnostic sensitivity and diagnostic specificity of the technique. These parameters can be estimated by comparing the method that needs to be validated with an existing reliable method ('gold standard'). This is done by testing a standard set of well-documented samples using both techniques in parallel. Approaches using Bayes' theorem are used to perform gold standard-free validations. Many PCR-based methods are characterised by an excellent analytical sensitivity and are thus good candidates for diagnostic tools of the required diagnostic sensitivity. However, the high level of analytical sensitivity can also make molecular techniques susceptible to cross-contamination and carry-over problems leading to false-positive results. Moreover, the presence of inhibitors can cause false-negative results. After an initial validation, test performance needs to be continuously monitored, e.g. by using combined Shewhart-CUSUM control routines, and test results compared to those obtained by other laboratories (proficiency testing).     

Methods for measuring genetic progress in dairy traits

This article presents the results of Mr. Na's thesis (1972). A method to estimate genetic progress which Everett et al. (1967) applied to a simulated population of cows is tried on two real populations of cows. Data collected in 1963–1967 for the daughters and in 1948–1959 for the dams are used. The computed estimates are of little value because of the bias caused by the selection of sires. The reason why selection of bulls affects the estimation of genetic gain, which is not obvious, is described.The method seems to be appropriate for large quantities of data where adjustment for the influence of selection among sires is possible. In order to correct for age of dams, dams' performance, and selection of sires it is necessary to record the identification number of the dam of each cow and the number of daughters and their mean performance of each bull.Another method of estimation which requires a second “progeny test” for the old bulls is discussed. A fairly high number of bulls with two progeny tests would be necessary if a precise estimate of genetic progress is desired.     

Statistical Evaluation of Test Accuracy Studies for Toxoplasma gondii in Food Animal Intermediate Hosts

The availability of accurate diagnostic tests is essential for the detection and control of Toxoplasma gondii infections in both definitive and intermediate hosts. Sensitivity, specificity and the area under the receiver‐operating characteristic (ROC) curve are commonly used measures of test accuracy for infectious diseases such as toxoplasmosis. These test performance characteristics are important considerations when selecting from among a group of tests for a specific testing purpose. In this study, we reviewed statistical approaches to evaluation of tests for toxoplasmosis with and without a gold‐standard (reference) test, including use of ROC analysis and likelihood ratios which retain the diagnostic information inherent in a quantitative test result. We use previously published data from a comparison of the accuracy of serological tests for swine toxoplasmosis to demonstrate suggested methods of data analysis. We make recommendations for statistical analysis and reporting of test evaluation studies for T. gondii in food animals based on our own experiences and those of others.     

川西北牧区家庭牧场优化模式研究示范阶段总结

本文叙述了“九五”重大农业攻关项目“川西北牧区家庭牧场优化模式研究示范”课题的实施进展情况,已研究提出了适合四川省牧区推行的家庭牧场优化模式,即“五推、四化、三配套”,简称“五·四·三”模式。此项目采用示范与试验相结合,取得了初步成效。使高寒牧区实现了“一步跨千年”的历史变革,为牧区畜牧业实现跨世纪持续发展奠定了良好基础。