Training. |
How do you train those involved?
Your training process may differ depending on what your team makeup looks like. We tend to approach training in slightly different ways if you are training yourself or others.
Training yourself
Training yourself is a crucial component in the reliability process, whether you are the sole person responsible for data collection or the lead of a team. In both cases, training yourself against a gold standard, if it exists, or another expert is a necessary first step. Once you achieve success here, you will then complete intra-rater reliability by completing a full training set at least twice and comparing your results. Once you achieve success here, you can then move on to training others.
In developing trainings, you'll need to consider a few key questions about your goals and potential troubleshooting:
You'll also need to choose the subsets of exercises, photos, and/or videos that you use in your training.
In developing trainings, you'll need to consider a few key questions about your goals and potential troubleshooting:
- Have you explicitly defined the steps that need to be performed by the trainee (you)?
- Have you identified what "success" in the training will look like?
- How will you limit memorization, since you need to score the same training set twice to compare against yourself? Common ways of doing this may include spacing out when you score each set (e.g. at least one week in between) and re-ordering the videos or photos within the training set between your first and second scores (often done via randomization).
- Is the training designed to evaluate if the trainee can identify both successes and problems, or occurrence and non-occurrence of behaviors? You want to ensure that you are not just confirming trainees can identify success (e.g. yes, we are weighing this animal correctly) but also problems (e.g. no, this animal is not being weighed correctly because only 3/4 legs are on the scale)
You'll also need to choose the subsets of exercises, photos, and/or videos that you use in your training.
Training others
Scaffolding your training, or slowly increasing complexity to best set trainees up for success, is very helpful when training a team, especially when they have different experience levels. We often start by familiarizing trainees to our definitions and outcomes of interest, then move on to a more formal training stage.
In addition to the considerations identified when training yourself, you may now also need to consider:
You'll also need to choose the subsets of exercises, photos, and/or videos that you use in your training. These test subsets should be different in the initial orientation training and the formal training that you use for reliability comparisons.
- What are the most likely mistakes or problems that trainees may have, and how will you address those (i.e. via re-training or more orientation to the outcome of interest)? These challenges may be evident from your own intra-reliability training, or training against an expert. We also find it helpful to train only 1 or 2 people first, then train the entire team once those individuals achieve success. This can provide more context for the kinds of challenges your team may experience.
- If multiple rounds of training are involved, which scores will you use to formally evaluate success - only the final training or an average across all trainings?
- If multiple rounds of re-training are involved, how will you prevent memorization? As in training yourself, this may include spacing trainings out, re-ordering photos or videos in a training set, and/or incorporating new training videos or live trainings that the trainee has not experienced yet. In the latter case, you will also need to score those new trainings.
You'll also need to choose the subsets of exercises, photos, and/or videos that you use in your training. These test subsets should be different in the initial orientation training and the formal training that you use for reliability comparisons.
