Percent agreement. |
|
This describes the percentage of times that individuals agree on a set of paired, categorical observations. The percent agreement is calculated as number of agreements divided by the total number of observations, multiplied by 100. Higher values, closer to 100, represent better agreement.
Percent agreement does not account for chance agreement between observers, and is thus likely to overstate how much they agree. This can be exacerbated if the observations represent categorical data with >2 categories, or if the observations are not balanced, with some categories overrepresented or missing altogether. This descriptive analysis should not be used in isolation because it is possible to have high percent agreement and still have other, systematic forms of disagreement. However, it may be as a starting point and to complement a formal statistical test (e.g. concordance).
Percent agreement does not account for chance agreement between observers, and is thus likely to overstate how much they agree. This can be exacerbated if the observations represent categorical data with >2 categories, or if the observations are not balanced, with some categories overrepresented or missing altogether. This descriptive analysis should not be used in isolation because it is possible to have high percent agreement and still have other, systematic forms of disagreement. However, it may be as a starting point and to complement a formal statistical test (e.g. concordance).
Example
Categorical training test results |
In this example, Observer 1 (Expert) and Observer 2 (Trainee) scored the same 10 videos for nesting behavior. Presence was scored as a 1 while absence was a 0. By looking at both sets of scores, we can then create a contingency table, below, that summarizes cases where both observers agree and disagree.
|
Contingency table |
We can see that the expert said the behavior happened 4 times (score 1) and didn't happen 6 times (score 0). Of the 6 times it did not happen, the trainee correctly identified it as not occurring 3 times, but incorrectly identified nesting behavior as occurring 3 times. To calculate percent agreement, we divide the number of agreements (7) by the total number of observations (10) and multiply by 100. In this case, the percent agreement is 70%.
|
