Intra- and inter-observer agreement of a protocol for clinical examination of dairy cows

In 2004,five veterinarians examined 283 dairy cows from four Danish dairy herds and independently assigned scores for the clinical signs lameness, hock lesions and other cutaneous lesions to each cow using a clinical protocol. We evaluated the inter-observer and the intra-observer agreement using prevalence-adjusted, bias-adjusted kappa (PABAK). We chose two different cut-offs between the ordinal scores for classifying cows as healthy or diseased for each of the clinical signs, and we compared the ability of the observers to discriminate between healthy and diseased cows for the two cut-offs. Without any formal training of the observers we found PABAK in the range 0.36-0.88 and we concluded that the choice of cut-off had an effect on the observed agreement among the observers.     

Inter-observer reliability testing of pig welfare outcome measures proposed for inclusion within farm assurance schemes

Consistency of assessment is essential to the farm assurance process. This study evaluated the inter-observer reliability of 31 farm assurance assessors, six veterinarians and four researchers for five pig welfare outcome measures proposed for inclusion into the UK pig farm assurance schemes. These were (1) tail lesions, (2) body lesions, (3) lameness, (4) pigs requiring hospitalisation and (5) oral behaviour. The following inter-observer reliability testing methods against a gold standard Trainer were used: a comparison of farm prevalence and the numbers of affected pigs in each pen identified by observers, Cohen's kappa (κ), Kendall's W, proportional agreement, sensitivity, and specificity. All measures achieved potentially high levels of inter-observer reliability and it was concluded that none should be excluded from farm assurance at this stage. However, across all the measures, 45% of observers did not record an overall farm prevalence 'close' to that of the gold standard Trainer. With the level of training and testing that took place in this study there would be a danger of significant bias occurring in a national assessment scheme. The data collected enabled some comparison of the methods used to assess inter-observer reliability. It is suggested that when the aim is to achieve agreement between observers on the overall farm prevalence the inter-observer reliability testing should focus on the closeness of the overall farm prevalence recorded by observers, but that other types of analysis may be helpful during training.     

Low intensity heart murmurs in boxer dogs: inter-observer variation and effects of stress testing

Inter-observer variation in the detection and grading of low intensity heart murmurs in boxer dogs was investigated. Six veterinarians with different levels of experience examined 27 boxers by cardiac auscultation. The dogs were auscultated before and after exercise, and the results were compared with phonocardiographic and echocardiographic examinations performed at rest and during two different stress tests. A subvalvular aortic ridge was identified in six dogs on two-dimensional echocardiography. Using dogs with low intensity murmurs or dogs free of heart murmurs, inter-observer agreement was positively correlated to the level of experience at rest (weighted kappa [kappa] 0.14 to 0.75), while the agreement was poor after exercise (weighted kappa 0.01 to 0.36). The presence of a subvalvular aortic ridge was associated with higher aortic flow velocities (P<0.002) and higher auscultatory murmur grading (P<0.001). There was an increase in murmur duration during one kind of stress test (P<0.001) and in aortic flow velocity during the other (P=0.001).     

Sound recording and digital phonocardiography of cardiac murmurs in dogs by using a sensor-based electronic stethoscope

The goals of this study were to present a technique of digitalised sound recordings and phonocardiograms (dPCGs), and to analyse its diagnostic capabilities. Heart sounds of 20 dogs were auscultated in vivo (on-line) and recorded with dPCGs by two authors using a Welch Allyn Meditron Stethoscope System. Sound recordings were auscultated off-line and blindly by four different observers having various auscultatory experiences, then listened to while viewing dPCGs. The results were compared to echocardiographic diagnoses. There was a significant agreement (p < 0.001) between on-line and off-line auscultatory findings regarding the four observers, ranging from 45% to 75% (weighted kappa values: 0.72 to 0.87). The best agreement was achieved by Observer 1 having the highest experience. Significant differences (p < 0.05) were found between Observer 1 and Observer 4 (with the lowest experience) in judging the quality of the murmurs during the off-line and blind auscultation. However, there were only minimal differences (95% to 100% agreements) in dPCG analyses among the four observers regarding intensity and quality of the murmurs while simultaneously listening to and viewing the dPCGs. Significant correlations were found between the traditional '0 to 6 scale' and a new '0 to 3 scale' murmur intensity gradings by all observers (correlation coefficients 0.640 to 0.908; p < 0.01 to p < 0.001). Analysis of dPCGs might be a valuable, additional tool helping with the diagnosis of canine cardiac murmurs, especially for those with less cardiological experience.     

Reliability of indicators of sheep welfare assessed by a group observation method

Assessments of animal-based outcomes form the core of routine veterinary clinical examinations and are being increasingly used as indicators of animal welfare. A method of group observation that did not require gathering and handling of individual sheep, was used to assess eight animal-based indicators of sheep welfare: demeanour, skin irritation, wool loss, excessive panting, coughing, lameness, and cleanliness of the ventral abdominal and 'breech' (perineum/gluteal/caudal hindlimb) areas. The inter-observer reliability of two or three observers who independently assessed these indicators was tested on 2406 adult sheep and growing lambs across 36 farms and the intra-observer reliability of an experienced, veterinary assessor--the 'test standard observer'--was assessed on 88 adult sheep during four on-farm assessments. Observer reliability was evaluated using Cronbach's alpha (α) comparison of the recorded proportion (%) of sheep affected by each welfare condition and by binomial logistic regression modelling. High levels of inter-observer reliability were identified for the assessment of group lameness (α 0.76-1.00) and cleanliness of the breech area (α 0.97-1.00). Excellent intra-observer reliability was determined for lameness (α 0.99), cleanliness of the breech area (α 0.97), demeanour (α 1.00) and wool loss (α 1.00). In addition, proportion data and logistic regression models identified few between-observer differences. The results suggest that welfare outcomes based on observations of the behaviour and physical appearance of individual animals within a group may offer a reliable and feasible measurement tool for the on-farm assessment of sheep welfare.     

The effects of feeding and fasting on gastrointestinal sounds in adult horses

The effect of changes in feed intake on auscultatable gastrointestinal sounds has not been systematically studied. Disagreement also is present in the literature about variation in sounds according to the quadrant of auscultation. Gastrointestinal sounds were recorded over the center of the left dorsal, left ventral, right ventral, and right dorsal quadrants and over the middle of the right abdominal flank. During 24 hours (n = 4) or 48 hours (n = 5) of fasting, there was a reduction in the intensity of gastrointestinal sounds as assessed by analysis of sound recordings. There was also a reduction in the number of mixing-like and propulsive-like sounds heard by 2 blinded observers. After refeeding, there was a marked increase in sound. Sound intensity varied among abdominal quadrants, but blinded observers did not notice significant differences in the number of mixing-like sounds. The left dorsal quadrant was quieter than others during fasting and refeeding. The right ventral quadrant appeared to be least affected by fasting, and sounds were louder over the right ventral and right middle quadrants than over the others. The blinded observers' perceptions of sound correlated poorly with one another and with objective measures of sound intensity. This experiment demonstrates the effectiveness of computerized analysis of abdominal sound in detecting a reduction in the intensity of gastrointestinal sounds during fasting and their return during refeeding. The left dorsal quadrant was quieter than other quadrants, likely because of its position over the small colon. There was considerable observer variation in the number of intestinal sounds heard.     

Standardization of canine meningioma grading: Validation of new guidelines for reproducible histopathologic criteria

Canine meningiomas are currently graded using the human grading system. Recently published guidelines have adapted the human grading system for use in dogs. The goal of this study was to validate the new guidelines for canine meningiomas. To evaluate the inter-observer agreement, 5 veterinary surgical pathologists graded 158 canine meningiomas following the human grading system alone or with the new guidelines. The inter-observer agreement for histologic grade and each of the grading criteria (mitotic grade, invasion, spontaneous necrosis, macronucleoli, small cells, hypercellularity, pattern loss and anaplasia) was evaluated using the Fleiss kappa index. The diagnostic accuracy (sensitivity and specificity) was assessed by comparing the diagnoses obtained with the 2 grading systems with a consensus grade (considered the reference classification). The consensus histologic grade was obtained by agreement between 4 experienced veterinary neuropathologists following the guidelines. Compared with the human grading alone, the canine-specific guidelines increased the inter-observer agreement for: histologic grade (κ = 0.52); invasion (κ = 0.67); necrosis (κ = 0.62); small cells (κ = 0.36); pattern loss (κ = 0.49) and anaplasia (κ = 0.55). Mitotic grade agreement remained substantial (κ = 0.63). The guidelines improved the sensitivity in identifying grade 1 (95.6%) and the specificity in identifying grade 2 (96.2%) meningiomas. In conclusion, the new grading guidelines for canine meningiomas are associated with an overall improvement in the inter-observer agreement and higher diagnostic accuracy in diagnosing grade 1 and grade 2 meningiomas.     

Observations of variable inter-observer agreement for clinical evaluation of faecal consistency in grow-finishing pigs

Inter-observer agreement for assessment of faecal consistency in pigs was evaluated using a scoring system with 3 categories. In a pilot study, 3 observers performed an examination of faecal samples post-collection. The samples were obtained from pigs (12-13 weeks old) in 4 herds with a history of diarrhoea associated with Lawsonia intracellularis, Brachyspira spp. and/or Porcine Circovirus Type 2. Observer 1 examined all the faecal samples from the 4 herds. Observer 2 only examined the faecal samples from herds 1 and 2. Observer 3 only examined the faecal samples from herds 3 and 4. We observed a substantial agreement in faecal consistency scores between Observers 1 and 3 (kappa=0.64, 95% CI: 0.51-0.78). In contrast, only a fair agreement was observed between Observers 1 and 2 (kappa=0.24, 95% CI: 0.14-0.34). The variations in inter-observer agreement detected in the current study suggest that misclassification error can be a problem in studies assessing faecal consistency. Solutions may include developing a standardized system for scoring the consistency of pig faeces, calibration when more than one observer is involved in clinical studies and using a more objective measure of faecal consistency.     

An indirect ELISA for detection of Encephalitozoon cuniculi infection in farmed blue foxes (Alopex lagopus)

Infection with the intracellular microsporidium Encephalitozoon cuniculi can cause serious disease, encephalitozoonosis, in the blue fox (Alopex lagopus). The disease diagnosis is based on clinical signs and pathological findings, and detection of E. cuniculi or circulating antibodies directed against the parasite. Indirect immunofluorescence (IFAT) and carbon immunoassay (CIA) are the most commonly used serological methods for diagnosis in this species. In the present study, an indirect ELISA (enzyme linked immunosorbent assay) was established and evaluated against IFAT by testing of 205 field samples from blue foxes. There was high agreement between the results of the ELISA and CIA (kappa=0.99), and the ELISA and IFAT (kappa=0.958). There was no significant statistical difference between the tests (p>0.05). It was concluded that the ELISA could be used to identify seropositive farmed blue foxes. The advantage of the ELISA lies in the potential of screening large numbers of animals with the goal of eradicating E. cuniculi infection in the farms.     

Reliability of an injury scoring system for horses

Background

The risk of injuries is of major concern when keeping horses in groups and there is a need for a system to record external injuries in a standardised and simple way. The objective of this study, therefore, was to develop and validate a system for injury recording in horses and to test its reliability and feasibility under field conditions.

Methods

Injuries were classified into five categories according to severity. The scoring system was tested for intra- and inter-observer agreement as well as agreement with a ''golden standard'' (diagnosis established by a veterinarian). The scoring was done by 43 agricultural students who classified 40 photographs presented to them twice in a random order, 10 days apart. Attribute agreement analysis was performed using Kendall''s coefficient of concordance (Kendall''s W), Kendall''s correlation coefficient (Kendall''s τ) and Fleiss'' kappa. The system was also tested on a sample of 100 horses kept in groups where injury location was recorded as well.

Results

Intra-observer agreement showed Kendall''s W ranging from 0.94 to 0.99 and 86% of observers had kappa values above 0.66 (substantial agreement). Inter-observer agreement had an overall Kendall''s W of 0.91 and the mean kappa value was 0.59 (moderate). Agreement for all observers versus the ''golden standard'' had Kendall''s τ of 0.88 and the mean kappa value was 0.66 (substantial). The system was easy to use for trained persons under field conditions. Injuries of the more serious categories were not found in the field trial.

Conclusion

The proposed injury scoring system is easy to learn and use also for people without a veterinary education, it shows high reliability, and it is clinically useful. The injury scoring system could be a valuable tool in future clinical and epidemiological studies.     

Multiattribute evaluation of two simple tests for the detection of Cryptosporidium parvum in calf faeces

There is a need for simple and inexpensive diagnostic and screening tests for the detection of Cryptosporidium parvum infection in calves. A sucrose wet mount test and a lateral immunochromatography test were evaluated for epidemiological sensitivity and specificity, cost per test, simplicity, test time and ease of batching. Polymerase chain reaction-restriction fragment length polymorphism (PCR-RFLP) of the Cryptosporidium oocyst wall protein (COWP) gene locus, with gel electrophoresis, was used as a gold standard. Cohen's kappa statistic of agreement (kappa) between the Ontario Veterinary College (OVC) sucrose wet mount test and COWP PCR-RFLP was 0.82, and the sensitivity and specificity of the OVC sucrose wet mount test were 88.6% and 93.8%, respectively. The sensitivity and specificity of the lateral immunochromatography test were 78.3% and 93.3%, respectively, and agreement between this test and PCR-RFLP was good (kappa=0.73). There was substantial agreement between the OVC sucrose wet mount test and the lateral immunochromatography test (kappa=0.84). Both tests were inexpensive and easy to use; however, the lateral immunochromatography test was faster and simpler to perform than the sucrose wet mount test, and was generally more user-friendly. These tests provide practitioners and researchers with cheap, quick and accurate methods of detecting C. parvum infection in young calves.     

Observer variation in the histological diagnosis of plasmacytoid leukemia (marine anemia).

The reliability of past histological criteria for the diagnosis of plasmacytoid leukemia was investigated by studying observer variation in the diagnosis of the disease. Participants blindly evaluated a series of histological samples and classified them as positive, negative or questionable cases of the disease. Intra- and interpathologist agreement were used to assess reliability and were determined by calculating the observed agreement in diagnosis and the kappa statistic. There was poor overall agreement for the classification of sample cases. Questionable cases were most frequently misclassified but marked disagreement also existed for positive and negative samples. Access to historical information prior to histological examination influenced agreement. The results show that previously described histological criteria alone were insufficient for establishing the diagnosis of plasmacytoid leukemia.     

The inter- and intra-observer reliability of a locomotion scoring scale for sheep

A seven point locomotion scoring scale, ranging from 0=normal locomotion to 6=unable to stand or move, has been developed. To test the between and within observer reliability of the scale, 65 movie clips of sheep with normal and varying degrees of abnormal locomotion were made. Three observers familiar with sheep locomotion were trained to read the movie clips. Thirty clips were randomly selected and used to test between and within observer agreement. There was high inter-(intra-class correlation coefficient [ICC]=0.93, weighted kappa [kappa(w)]=0.93) and intra-(ICC=0.90, kappa(w)=0.91) observer reliability, with no evidence of observer bias. The main between score differences were for scores 0 (normal) and 1 (uneven posture and shortened stride but no head movement). The results indicate that the locomotion scoring scale using groups of defined observations for each point on the scale was reliable and may be a useful research tool to identify and monitor locomotion in individual sheep when used by trained observers.     

Intra- and inter-observer reliability of combined segmental measurement techniques for predicting immediate post-deployment intraluminal tracheal stent length in dogs

This study evaluated segmental measurement techniques for predicting immediate post-deployment intraluminal tracheal stent length in dogs with naturally occurring tracheal collapse. Radiographs of 12 client-owned dogs that underwent intraluminal tracheal stent placement were retrospectively reviewed. Tracheal lengths were divided into 1, 2, 3, or 4 equal segments. Stent lengths were predicted using the widest dorsoventral height of each segment, with and without the addition of 10%, and an accompanying foreshortening chart. Techniques were compared for intra- and inter-observer reliability, and post-deployment stent length predictability. There was good to high intra- and inter-observer reliability for all segmental measurements; median intra-class correlation coefficients were 0.98 and 0.92, respectively. Measuring 2 segments without the addition of 10% to the widths was significantly more accurate in predicting immediate post-deployment stent length in terms of percent (P = 0.03) and absolute difference (P = 0.02). Segmental measuring techniques are repeatable amongst observers and may help guide stent selection.     

Agreement between T2 and haste sequences in the evaluation of thoracolumbar intervertebral disc disease in dogs

The purpose of this study was to compare half-Fourier-acquisition single-shot turbo spin-echo (HASTE) and T2-weighted (T2-W) sequences in dogs with thoracolumbar disc extrusion. MRI studies in 60 dogs (767 individual intervertebral disc spaces) were evaluated. Agreement between T2-W and HASTE sequences was assessed for two criteria: presence of an extradural lesion and treatment recommendation. There was moderate agreement between T2-W and HASTE sequences as to presence of an extradural lesion (kappa = 0.575). HASTE was in agreement in 96.1% of the sites where no extradural lesion was identified on T2-W images, but only in 58.1% of the sites where extradural lesions were identified on T2-W images. There was also moderate agreement between T2-W and HASTE sequences as to treatment recommendations (kappa = 0.476). HASTE was in agreement in 98.4% of the sites where a lesion was considered nonsurgical on T2 but only 82.1% of sites a lesion was considered surgical on T2. In 1.0% of sites considered not surgical and in 9.8% of sites considered equivocal based on T2-W images, a surgical lesion was identified on HASTE. Acquisition of a HASTE sequence in addition to conventional sequences may be beneficial in determining the severity of spinal cord compression in some cases when evaluating the canine spine.     

Reliability of manual measurements of corneal thickness obtained from healthy canine eyes using spectral-domain optical coherence tomography (SD-OCT)

The purpose of this study was to manually measure corneal thickness in canine eyes using a spectral-domain optical coherence tomography (SD-OCT) device and to assess intra- and inter-observer reliability of this technique. Twenty healthy dogs with a mean age of 4.7 y were examined. A 6-mm corneal pachymetry protocol was carried out by 1 operator using 1 SD-OCT device in both eyes of each animal. Measurements were obtained manually and in duplicate by 2 independent investigators (> 24 h apart), using the built-in caliper function. Measurements included epithelial thickness (ET), non-epithelial thickness (NET), and central corneal thickness (CCT). The overall mean ET, NET, and CCT for all eyes examined were 72.3 ± 4.6 μm, 538.9 ± 42.5 μm, and 611.2 ± 40.3 μm, respectively. There was no significant difference in ET, NET, or CCT based on the eye examined [oculus dexter (OD) versus oculus sinister (OS)], age, or gender of the animal. There was no significant difference in replicate measurements of ET, NET, or CCT done by the same operator, although a small but significant difference was noted between operators for ET measurements only. The mean difference in ET between operators was 0.6 μm (P = 0.03). The coefficient of variation ranged from 0.5% to 9.27% and intraclass correlation coefficient ranged from 0.35 to 0.97. Based on these results, manual measurements of corneal thickness in canine eyes using a portable SD-OCT device provided ET, NET, and CCT measurements with clinically acceptable intra- and inter-observer reliability.     

Comparison of blood polymerase chain reaction and enzyme-linked immunosorbent assay for detection of Mycobacterium avium subsp. paratuberculosis infection in cattle and sheep.

A study was carried out to compare the performance of enzyme-linked immunosorbent assay (ELISA) and blood polymerase chain reaction (PCR) for diagnosis of paratuberculosis in cattle and sheep. For cattle, a set of 278 samples from 1 paratuberculosis-affected Friesian farm was used; it included 80 ELISA-positive samples and 198 ELISA-negative samples from an age-matched group. Ninety-four samples were from heifers and 184 were from 2-5-year-old cows. The overall analysis showed a clear association (Fisher exact test [FET] P = 0.0049) but a weak negative agreement (45.3%, kappa = -0.1665 +/- 0.0994) between the 2 tests. It reflected a moderate agreement among heifers (87.7%, kappa = 0.4471 +/- 0.2435) and a moderate disagreement among cows (62.7%, kappa = -0.3670 +/- 0.1057). For sheep, 496 blood samples from 53 Latxa dairy flocks were used; 180 of the blood samples were from dam/offspring pairs. The overall association between the 2 tests on ovine samples was strong (FET, P = 0.0005), whereas the agreement was low (kappa = 0.1622 +/- 0.1188). There was slightly better agreement for ewes (kappa = 0.2135 +/- 0.1992) than for lambs (kappa = 0.1193 +/- 0.1301). There was also a highly unlikely proportion of dam/offspring positive results (FET, P < 0.0001, kappa = 0.6269 +/- 0.1854). Four of 6 lambs that were necropsied 1 year after testing had paratuberculosis microscopic lesions in the ileocecal valve (3 lambs) or a PCR-positive result (4 lambs). These results suggest that blood PCR testing might be a potentially useful new approach in paratuberculosis diagnosis, especially in young animals.     

Validation of a subjective counting method for a horn flies (Haematobia irritans irritans) (Diptera: Muscidae) population in a cattle herd

A trained observer direct count method to measure horn fly population was evaluated to determine the reliability (inter-observer agreement) and its validity when compared with a "gold standard" method (video film). All the counts were performed with the animals restrained in a chute in a single herd. A direct count of horn flies by each of two observers on opposite sides of the animal was made. In addition a videotape recording of the counting surface on each animal was made. Horn flies were counted on 80 cows in the morning twice monthly from 22 October 1999 to 24 March 2000. The correlation between observer counts was high as was the correlation between observer counts and counts made from a videotape recording. Direct counts by trained observers were highly reliable and had good validity. However, videotape recording can be useful when the horn fly population is high, because it is faster in the field than the direct count method. The results of this study suggest that the direct count method can be used with confidence under field conditions.