Timeline.​​

Reliability training should be addressed at multiple stages throughout a research project:

Before data collection

Test observer consistency. Individuals should only advance to data collection once they are confirmed to be reliable using a combination of visual observation and metrics. 

During data collection

Test observer consistency to evaluate whether there was drift from the beginning of the experiment. If individuals are inconsistent, the approach to data collection may need revision.

If the data were collected in a format that can be revisited (i.e. video, photos), individuals with poor reliability should be re-trained before collecting or scoring additional data. Once they are confirmed to be reliable again, they should re-score their previous data. 

If the data were collected in a format that can not be revisited (e.g. live observations or body weight measurements at specific time points), data may need to be excluded, and poor reliability should be discussed in the manuscript. If affected data are not excluded, caution should be used in interpreting them. Individuals should be re-trained, and data collection should not continue until they are confirmed to be reliable again.

In some cases, revising your approach to data collection, such as simplifying your outcomes of interest, or changing your continuous behavioral outcomes to categorical outcomes, may improve reliability and prevent the need for re-scoring data.

After data collection

Test observer consistency to evaluate whether there was drift from the beginning of the experiment. If individuals are inconsistent, the approach to data collection may need revision.

Poor reliability after the experiment ends should be approached in the same way as poor reliability during the experiment. If data were collected in a format that can not be revisited, caution should be used in interpreting the data, and this should be discussed in the manuscript. Consider whether you can change your data collection strategy to improve reliability instead.

Many researchers, ourselves included, tend to focus on reliability at the start of a project. However, evaluating reliability before, during, and after data collection is best practice. Teams may drift over time, especially if a project is conducted over a long period of time. 

This approach, of revisiting reliability throughout the stages of a project, is laid out in the checklist.

<< Diving deeper

< Rare outcomes

Troubleshooting >