Describing the within laboratory and between laboratory agreement of a serum ELISA in a national laboratory network

Results from laboratory assays for detection of animal disease are often assessed for repeatability (agreement within laboratory) and reproducibility (agreement between laboratories). This work aimed to understand the strengths and limitations of available methods for describing these quantities. Five major veterinary laboratories in Australia volunteered to participate in a designed evaluation based on repeat testing of twenty bovine sera. Sampling was stratified so that ten of the sera were negative to the virus neutralisation test (VNT) for antibody to bovine herpes virus 1 (BHV-1) and the remaining ten sera were VNT positive. Each serum was divided into 50 replicates and each laboratory assayed one replicate of each serum on a weekly basis using a commercial ELISA for BHV-1. Laboratories were blinded to the identity of sera. The data on sample to positive control ratio (S/P) for these 1000 individual assays were collated, sources of variance analysed using a random effects model, and reliability coefficients (ρ) obtained from the variance estimates as quantitative measures of within and between laboratory agreement. Coefficient of variation (CV) was calculated for combinations of sera and laboratory. CV was found to be higher for sera with the lowest mean S/P values (VNT -ve sera). For VNT -ve sera, agreement of S/P within laboratory was low to moderate (ρ: 0.01-0.27) and the agreement between all labs was low (ρ=0.02). Reliability coefficients for VNT +ve sera were very high for agreement within laboratories (ρ: 0.63-0.92) and moderate for agreement between laboratories (ρ=0.52). As well, simulation demonstrated that sero-prevalence has a dramatic affect on the reliability coefficient if sampling were to be irrespective of VNT status. We conclude that there are some limitations with the available approaches for assessing agreement within and between laboratories. Although reliability coefficients have some drawbacks they are an attractive way of reducing reliance on subjective assessment of agreement.