Resources.There are many excellent resources available that provide more insight into the process of reliability, or test sets to use directly in training. We've gathered a few that we use here.
|
Checklist
We've consolidated some of the website's information into a companion checklist that we believe best captures the required components of the reliability process and recommended components that should be part of best practice (what we strive to do). This checklist can be accessed here.
Sitemap
If you're interested in using this website as an assignment for a class or working group or lab, we've created a sitemap listing each unique page and link to serve as a guide. We have used this internally to ensure our groups have progressed through each page. You can download the sitemap tracking sheet here.
Additional readings
- Textbooks:
- Observing Animal Behaviour: Design and Analysis of Quantitative Data, Dawkins 2007
- Epidemiology: Beyond the Basics (4th edition), Szklo and Nieto 2019
- Measuring Behaviour (4th edition), Bateson and Martin 2021
- Peer-reviewed journal articles:
- Observational study of behavior: Sampling methods (1974)
- Intraclass correlations: Uses in assessing rater reliability (1979)
- Bias, prevalence, and kappa (1993)
- Measuring agreement in method comparison studies (1999)
- Repeatability for Gaussian and non-Gaussian data: a practical guide for biologists (2010)
- A guideline of selecting and reporting intraclass correlation coefficients for reliability research (2016)
- Assessing observer variability: A user's guide (2017)
- Common pitfalls in statistical analysis: Measures of agreement (2017)
- Testing the reliability of inter-rater reliability (2020)
- Evaluation of inter-observer reliability of animal welfare indicators: Which is the best index to use? (2021)
Additional links to peer-reviewed journal articles can be found on relevant Metrics pages.
Example reliability tests
The Tucker Lab Heat Stress in Cattle site has tests that automatically generate kappa scores for the following metrics:
The PAACO Dairy Welfare Auditor Training website has several tests that automatically generate kappa scores for the following metrics in dairy cattle:
The Cow-Calf Health and Handling Assessment website has several tests that automatically generate kappa scores for the following metrics in cow-calf ranches:
Health measures:Handling measures:
- Respiration rate (cattle)
- Panting (cattle)
The PAACO Dairy Welfare Auditor Training website has several tests that automatically generate kappa scores for the following metrics in dairy cattle:
- Hygiene
- Body condition score
- Broken tail
- Hock injuries
- Knee injuries
- Locomotion
- Other injuries
- Lying surface
The Cow-Calf Health and Handling Assessment website has several tests that automatically generate kappa scores for the following metrics in cow-calf ranches:
Health measures:Handling measures:
Looking for more?
Looking for something that isn't listed here? Feel free to contact Dr. Blair Downey. We're still working on adding new resources to this page, but have many SOPs, test subsets, troubleshooting examples, and code beyond what is listed that we would be happy to share.
