Differentiating between analytical and diagnostic performance evaluation with a focus on the method comparison study and identification of bias

Prior to introduction of a new method to the diagnostic laboratory, analytical performance must be validated to ensure operation within the manufacturer's specifications and/or within predetermined quality requirements. In addition, the new method may require diagnostic performance assessment to ensure it differentiates between diseased and nondiseased individuals as intended. These 2 phases of assessment, while complementary, are not equivalent and require a different set of experiments, statistical analyses, and interpretation. Studies of analytical performance typically include a method comparison experiment, the purpose of which is to identify bias (inaccuracy) of the “test” (or “index”) method (new method) relative to a “comparative method” (established method). Analysis of method comparison data is facilitated by commercial software programs that present the statistical significance of identified bias; however, the clinical relevance of any bias also should be considered. Studies of diagnostic performance should not be pursued until analytical performance is fully characterized and may not be required for well‐established tests or for those for which results are nonspecific (ie, not referable to a specific disease or condition). Diagnostic performance assessment may include assessment of sensitivity, specificity, predictive values, odds ratios, and/or likelihood ratios. The purpose of this review is to clarify differences between the assessment of analytical and diagnostic performance, and to explore the method comparison study and bias assessment from a perspective not addressed in prior veterinary articles.     

Using the correct regression analysis technique in method comparison studies

Regression analysis frequently is used to evaluate a new clinical laboratory method. Results from the new method are compared to results from an existing more established method. If measurement error exists in the established method, then least squares may not be an appropriate statistical method to use for the regression analysis. An errors variable regression analysis model was used to evaluate data from five method comparison studies. Results were compared to least squares analyses performed on the same data. When significant measurement error existed in the "reference" method, the errors variable analysis provided a less biased estimate of the regression statistics, which differed markedly from the least squares results. Inappropriate use of least squares in method comparison studies can lead to erroneous conclusions about the relationship of a new analytical method to an existing method.     

Validation of molecular-diagnostic techniques in the parasitological laboratory

Diagnostic laboratories today often operate according to standard quality management procedures such as ISO/IEC 17025. This requires that only validated methods are used. Validation procedures help to document that a particular protocol used by the accredited laboratory has a guaranteed performance in that particular laboratory. Several study designs exist for validation procedures. Computer programmes are available to help with the statistical analysis of validation results. The agreement beyond chance of results obtained in the protocol that is to be validated can be compared to those achieved in an already established test (agreement). For a method that is used under routine conditions or for epidemiological studies, it is necessary to assess the diagnostic sensitivity and diagnostic specificity of the technique. These parameters can be estimated by comparing the method that needs to be validated with an existing reliable method ('gold standard'). This is done by testing a standard set of well-documented samples using both techniques in parallel. Approaches using Bayes' theorem are used to perform gold standard-free validations. Many PCR-based methods are characterised by an excellent analytical sensitivity and are thus good candidates for diagnostic tools of the required diagnostic sensitivity. However, the high level of analytical sensitivity can also make molecular techniques susceptible to cross-contamination and carry-over problems leading to false-positive results. Moreover, the presence of inhibitors can cause false-negative results. After an initial validation, test performance needs to be continuously monitored, e.g. by using combined Shewhart-CUSUM control routines, and test results compared to those obtained by other laboratories (proficiency testing).     

Clinical application of biological variation data to facilitate interpretation of canine and feline laboratory results

Interpretation of laboratory results is based on comparison of the patient's own results against established decision thresholds or reference intervals in the context of the clinical presentation and history. Blood measurand analysis has pre‐analytical, analytical and physiological sources of variation, which may complicate interpretation of results. Biological variation describes the physiological random fluctuation of blood measurands around a homeostatic set point, which varies within and between individuals. This article reviews the practical applications of biological variation in the everyday clinical setting. Examples are offered to highlight how biological variation can be used to: (1) assess the usefulness of subject‐based reference intervals, (2) determine measurand homeostatic set points, (3) interpret single or serial results for diagnosis of disease and (4) evaluate changes in serial results during monitoring.     

基于人工消化液与密闭消化器的酶法测定豆粕鸭代谢能值的研究

本试验旨在通过探讨基于人工消化液与密闭消化器的酶法条件下酶法与排空强饲法的测试精度及测定豆粕鸭代谢能值的差异,为鸭饲料养分生物学效价酶学评定方法的设计提供参考.酶法与排空强饲法测试精度的比较研究采用完全随机设计,其中酶法每个消化阶段设5个重复,每个重复1个样品,排空强饲法设4个重复测定,每个重复3只试验鸭,比较两方法间重复测定的极差、相对标准差偏差.酶法与排空强饲法测定豆粕鸭代谢能值的比较研究中,采用配对设计,对25个豆粕样品的酶水解物能值与排空强饲代谢能作系统比较.结果表明,鸭排空强饲法重复测定的日粮干物质消化率极差为2.89％,相对偏差为1.57％,代谢能值极差为0.36 MJ· kg-1,相对标准偏差为1.13％.酶法重复测定的日粮干物质消化率极差和相对标准偏差分别为1.57％和1.13％,其测试的精度可以达到鸭排空强饲法类似的测试精度.25个豆粕样品的鸭排空强饲法真代谢能(TME)平均值、酶水解物能值平均值分别为13.33和13.55 MJ·kg-1 DM,两平均值间无显著性差异(P＞0.05).在考虑到允许误差的前提下,本试验所用酶法准确估测豆粕鸭代谢能值的概率为68％(n=25,差值为0.85 MJ· kg-1以内).     

Internal quality control of a turbidimetric immunoassay for canine serum C-reactive protein based on pooled patient samples

BACKGROUND: Optimized internal quality control (IQC) procedures are important to ensure that only results without medically important errors are used for medical decision-making and to ensure that unnecessary rejection of valid analytical runs is avoided. Additionally, estimates of the analytical performance can be derived from IQC data. In the absence of available species-specific standards of a compound, the use of alternative control materials based on patient samples is a possibility, although investigations on the suitability of this approach are needed. OBJECTIVES: The objective of the study was to plan and implement a simple IQC procedure with control material based on pooled canine serum samples for a turbidimetric immunoassay (TIA) for the determination of human C-reactive protein (CRP) that recently was validated for the determination of canine serum CRP, and to assess the clinical analytical performance of the assay. METHODS: Proposed guidelines for the planning and implementation of IQC procedures were followed by using 2 control materials. Quality requirements of the assay were defined objectively by means of available data on biological variation, and goals for IQC performance were defined according to recommendations (probability of error detection [P(ed)] >.90 and of false rejection [P(fr)] <.05). Analytical performance was evaluated by means of medical decision charts. RESULTS: The control rule of 1(2.5s) (ie, rejection of the analytical run if at least 1 of 2 control materials deviates from the mean by more than 2.5 SD) fulfilled the criteria of predicted IQC performance (P(ed) =.94-1.00, P(fr) =.03). The IQC method was successfully implemented over a 14-week period. The observed coefficient of variation in the period of monitoring was 3.8% (low) and 2.9% (high), which equals excellent analytical performance. CONCLUSIONS: It was possible to plan and implement a simple IQC procedure for the CRP-TIA with control materials based on canine serum samples that fulfilled the criteria of high error detection and low false rejection of valid analytical runs. The assay showed excellent long-term analytical performance over a 14-week period.     

Use of posterior predictive assessments to evaluate model fit in multilevel logistic regression

Assessing the fit of a model is an important final step in any statistical analysis, but this is not straightforward when complex discrete response models are used. Cross validation and posterior predictions have been suggested as methods to aid model criticism. In this paper a comparison is made between four methods of model predictive assessment in the context of a three level logistic regression model for clinical mastitis in dairy cattle; cross validation, a prediction using the full posterior predictive distribution and two “mixed” predictive methods that incorporate higher level random effects simulated from the underlying model distribution. Cross validation is considered a gold standard method but is computationally intensive and thus a comparison is made between posterior predictive assessments and cross validation. The analyses revealed that mixed prediction methods produced results close to cross validation whilst the full posterior predictive assessment gave predictions that were over-optimistic (closer to the observed disease rates) compared with cross validation. A mixed prediction method that simulated random effects from both higher levels was best at identifying the outlying level two (farm-year) units of interest. It is concluded that this mixed prediction method, simulating random effects from both higher levels, is straightforward and may be of value in model criticism of multilevel logistic regression, a technique commonly used for animal health data with a hierarchical structure.     

Evaluation of five portable blood glucose meters for use in dogs

OBJECTIVE: To evaluate clinical and analytical accuracy of 5 portable blood glucose meters (PBGM) used to measure blood glucose concentrations in dogs and to determine potential sources of error. DESIGN: Prospective study. ANIMALS: 221 dogs. PROCEDURE: Venous blood samples were obtained, and results of the 5 PBGM were compared with results of a hexokinase reference method. Agreement among methods was determined by use of error grid analysis and statistical methods. RESULTS: Accuracy of the PBGM varied with glucose concentration of the sample. The largest differences between results of the PBGM and results of the reference method were obtained with samples with high glucose concentrations; 4 PBGM tended to underestimate and 1 PBGM tended to overestimate the true glucose concentration. Absolute differences between results of the PBGM and results of the reference method were small for samples with low glucose concentrations and samples with concentrations in the reference range. None of the PBGM yielded measurements that would result in clinically unacceptable errors. Within-run and between-day precision was good for all PBGM, and results were not affected by use of EDTA or heparin to anticoagulate blood. Readings of the PBGM were significantly higher for blood samples with low Hct than for samples with normal Hct. For 3 PBGM, samples < 3 microliters resulted in inaccurate measurements. CONCLUSIONS AND CLINICAL RELEVANCE: Results suggest that currently available PBGM are sufficiently accurate for use in clinical practice to determine blood glucose concentrations in dogs.     

Analysis of video-recorded images to determine linear and angular dimensions in the growing horse.

Studies of growth and conformation require statistical methods that are not applicable to subjective conformation standards used by breeders and trainers. A new system was developed to provide an objective approach for both science and industry, based on analysis of video images to measure aspects of conformation that were represented by angles or lengths. A studio crush was developed in which video images of horses of different sizes were taken after bone protuberances, located by palpation, were marked with white paper stickers. Screen pixel coordinates of calibration marks, bone markers and points on horse outlines were digitised from captured images and corrected for aspect ratio and 'fish-eye' lens effects. Calculations from the corrected coordinates produced linear dimensions and angular dimensions useful for comparison of horses for conformation and experimental purposes. The precision achieved by the method in determining linear and angular dimensions was examined through systematically determining variance for isolated steps of the procedure. Angles of the front limbs viewed from in front were determined with a standard deviation of 2-5 degrees and effects of viewing angle were detectable statistically. The height of the rump and wither were determined with precision closely related to the limitations encountered in locating a point on a screen, which was greater for markers applied to the skin than for points at the edge of the image. Parameters determined from markers applied to the skin were, however, more variable (because their relation to bone position was affected by movement), but still provided a means by which a number of aspects of size and conformation can be determined objectively for many horses during growth. Sufficient precision was achieved to detect statistically relatively small effects on calculated parameters of camera height position.     

穴位注射治疗鸭大肠杆菌病的初步研究

为改善家禽胴体品质和减少抗生素使用数量,根据中兽医辨证论治理论和整体的观点,采用穴位注射给药的方法对人工感染大肠杆菌病鸭作治疗保护试验,结果按氟甲砜霉素注射液推荐剂量的40％-50％给药仍有较好的治疗效果,与按100％推荐剂量胸肌注射给药的治疗组比较,差异不显著(P>0.05),而与按50％推荐剂量胸肌注射给药的治疗组比较差异显著(P<0.05),穴位注射给药有较好的疗效。试验结果显示了减少抗生素使用量和改善家禽胴体品质的治疗新方法的可行性。