Assessment.
Assessment, or formal training, can be asynchronous or synchronous. In asynchronous training, you will develop video and photo sets (your test subsets) that all trainees score. Their results from this are then compared to an expert's scores of the same material to generate reliability metrics. In synchronous training, you will undergo live sessions where trainees collect data at the same time as the expert, and the data from these sessions are compared to generate reliability metrics.
The training process may differ depending on what your team makeup looks like. We tend to approach training in slightly different ways if you are training yourself or others. These are important considerations whether your training is asynchronous or synchronous.
The training process may differ depending on what your team makeup looks like. We tend to approach training in slightly different ways if you are training yourself or others. These are important considerations whether your training is asynchronous or synchronous.
Assessing yourself
Training yourself is a crucial component in the reliability process, whether you are the sole person responsible for data collection or the lead of a team. In both cases, training yourself against a gold standard, if it exists, or another expert is a necessary first step. Once you achieve success here, you will then complete intra-rater reliability by completing a full training set at least twice and comparing your results. Once you achieve success here, you can then move on to training others, if you are not the sole person involved in data collection.
In developing trainings, we find it helpful to consider a few key questions about the goals:
In developing trainings, we find it helpful to consider a few key questions about the goals:
- Have you explicitly defined the steps that need to be performed by the trainee (you)?
- Have you identified what "success" in the training will look like?
- How will you limit memorization, since you need to score the same training set twice to compare against yourself? Common ways of doing this may include spacing out when you score each set (e.g. at least one week in between) and re-ordering the videos or photos within the training set between your first and second scores.
- Is the training designed to evaluate if the trainee can identify both successes and problems, or occurrence and non-occurrence of behaviors? You want to ensure that you are not just confirming trainees can identify success (e.g. yes, we are weighing this animal correctly) but also problems (e.g. no, this animal is not being weighed correctly because only 3/4 legs are on the scale).
Assessing others
In addition to the considerations identified when training yourself, you may now also need to consider:
- What are the most likely mistakes or problems that trainees may have, and how will you address those (i.e. via re-training or more orientation to the outcome of interest)? These challenges may be evident from your own intra-reliability training, or training against an expert. We also find it helpful to train only 1 or 2 people first, then train the entire team once those individuals achieve success. This can provide more context for the kinds of challenges your team may experience.
- If multiple rounds of training are involved, which scores will you use to formally evaluate success - only the final training or an average across all trainings?
- If multiple rounds of re-training are involved, how will you prevent memorization? As in training yourself, this may include spacing trainings out, re-ordering photos or videos in a training set, or incorporating new training videos or live trainings that the trainee has not experienced yet. In the latter case, you will also need to score those new trainings.
