Training. |
How do you train those involved?
We often consider reliability training to have 2 components: an orientation and a more formal stage.
Orientation
We start the training process by familiarizing ourselves and our trainees to the outcomes of interest. At this stage, our goals are to ensure our team knows the outcomes that they’ll be trained on, instill trainee confidence in the technique or definitions, and cultivate the ability to successfully complete or score under ideal conditions. Orientation should improve the likelihood of success during the next, more formal, training stage.
As part of orientation, we often begin by reviewing the written SOP that details our task or technique, or the written definitions and showing a few clear-cut photo or video examples of each. At this stage, we do not expect trainees to understand gray areas related to the outcome(s), or recognize every way outcome(s) may be expressed.
Depending on the complexity of the task and the skill level of the trainees, the next step in orientation may involve short tests for practice. For example, if the ultimate goal is to score 10 behaviors simultaneously, we will orient trainees with a practice test made up of short video clips, where all they need to do is identify if a single behavior did or did not occur in each. While this is not how they would score during true data collection (10 behaviors instead of only 1), this approach prepares them for success in the formal training stage. It can build confidence and skill.
As part of orientation, we often begin by reviewing the written SOP that details our task or technique, or the written definitions and showing a few clear-cut photo or video examples of each. At this stage, we do not expect trainees to understand gray areas related to the outcome(s), or recognize every way outcome(s) may be expressed.
Depending on the complexity of the task and the skill level of the trainees, the next step in orientation may involve short tests for practice. For example, if the ultimate goal is to score 10 behaviors simultaneously, we will orient trainees with a practice test made up of short video clips, where all they need to do is identify if a single behavior did or did not occur in each. While this is not how they would score during true data collection (10 behaviors instead of only 1), this approach prepares them for success in the formal training stage. It can build confidence and skill.
Next steps
Following orientation, we move to formal evaluation of the trainee's command of the outcomes. This is the level at which reliability metrics are calculated. When preparing for this, we undergo a few steps.
-
Test subsets
-
Assessment
<
>
Before you begin formal training, you need to choose the subsets of exercises, photos, or videos for this process. There are specific features to consider as you collect these subsets to ensure your trainees are exposed to a relevant and robust set of examples and conditions. Click below to learn more.
Once you have started collecting your test subsets, it’s time to consider what your evaluation process will be. This is influenced by who is on your team: teams of just one (you!) or many. Click below to learn about key considerations while you develop your assessment.
