Training Criteria.​

Across all types of data collection, there are a few common considerations that inform how we design our training methods and criteria.

1. Philosophy: Set up for success

• Scaffold the introduction to the technique to provide context and rationale. Naïve observers will need more scaffolding than experienced ones
• All observers will need to begin with being shown the behaviors or technique and oriented to the definition. A specific subset of training materials may be required for this
• Use bespoke trainings when inconsistencies are identified
• Caution is needed to prevent memorization

2. Training criteria

• Have you explicitly defined the steps that need to be performed by the trainee?
• Have you identified what "success" in the training will look like? 
• Have you considered what the common mistakes or problems may be, and how to address those (i.e. via re-training or more orientation to what is being asked)?
• If multiple rounds of training are involved, which scores will you use to evaluate success (i.e. only the final training or an average across all trainings)?
• Is the training designed to evaluate if the trainee can identify both successes and problems, or occurrence and non-occurrence of behaviors?

3. Features of test subsets
​Choosing the exercises, photos, and/or videos used in training tests can require some troubleshooting to ensure these materials help in achieving your goals of developing reliable trainees. While test features vary based on the type of data being collected, there are some common considerations:

• Variation. We aim to develop tests that capture the range of what trainees will encounter when working independently. Some consistency test metrics, like kappa scores, have specific guidance about the variability that should be included.
• Matching the level of measurement. Test subsets should match the level of measurement that will be reported in the manuscript, within reason. For example, training observers on 10 minutes worth of video may not accurately describe their reliability if data is ultimately scored across 24-h periods. Similarly, training individuals to identify behavior using instantaneous sampling in training may not be appropriate if data is ultimately scored continuously. However, this depends on practicality and the goal of the observer consistency exercise. 
• Match the modality. When possible, match the modality of the test subset to the methodology used during data collection. For example, if behavior will be scored in person, training observers only on video may not represent their ability to correctly score the same behaviors in person. Time, efficiency, and efficacy would all be good reasons to violate this recommendation.
• Time. It should be possible for trainees to complete the testing in a reasonable period of time.
• Prevent memorization. When possible, presentation order within tests should be reordered or randomized. If re-training, caution is needed to prevent memorization.