Team Makeup. |
Who is involved in data collection?
The level of detail used in reliability training often depends on 2 key factors: team size and experience level.
1. Team size
|
If only one person (you!) is collecting data, reliability training can be fairly straightforward. Having a team size of 1 does not negate the necessity of conducting formal analyses. Comparing against a gold standard, if it exists, or a trusted colleague, can be beneficial in evaluating your definitions, methodology, and experimental repeatability. Comparing against yourself over time (intra-rater reliability) is also important to capture and describe any drift that may occur.
|
|
If a large team is collecting data, care is needed to ensure everyone is scoring in the same manner. Consistent training across all individuals is important, and comparisons should be made, at a minimum, between each observer and the expert (inter-rater reliability).
|
Wait - who is the expert?
We use the term "expert" when referring to the comparisons we make in reliability training. Many people can be considered an expert, but this depends on their familiarity with the specific outcomes of interest:
Importantly, a beginner can also become the expert. This is common when a new graduate student begins a project. You may be a novice when developing your proposal and definitions, but after repeated practice with your outcomes, you may become the expert for future comparisons.
- Your PI (principal investigator) or advisor
- A lab colleague
- A trusted colleague outside of your group
- You!
Importantly, a beginner can also become the expert. This is common when a new graduate student begins a project. You may be a novice when developing your proposal and definitions, but after repeated practice with your outcomes, you may become the expert for future comparisons.
2. Experience level
If your team is made up of beginners, the training should be scaffolded, or designed to increase complexity over time. This sets observers up to succeed during the formal training and data collection.
If the observer(s) are highly skilled with the outcome of interest, they may not need scaffolded training, and may be able to jump straight into a formal reliability test without first conducting simpler practice tests.
If the team includes a mix, we expose all individuals to the same type of training. For example, if the beginners need an orientation to many example videos and a short practice test before taking a formal reliability test, we also ensure any experts follow that same 3-step process.
If the observer(s) are highly skilled with the outcome of interest, they may not need scaffolded training, and may be able to jump straight into a formal reliability test without first conducting simpler practice tests.
If the team includes a mix, we expose all individuals to the same type of training. For example, if the beginners need an orientation to many example videos and a short practice test before taking a formal reliability test, we also ensure any experts follow that same 3-step process.
