Very often, conformity studies are an indirect attempt to validate a new assessment system or instrument. In other words, in the absence of a definitive criterion variable or a “gold standard”, the accuracy of a scale or instrument is assessed by comparing its results when used by different evaluators. Perhaps this is where we would like to use methods that tackle the problem of real concern – to what extent do the assessments reflect the true characteristic we want to measure? If the observed conformity is due only to chance, that is, if the evaluations are completely independent, then each diagonal element is a product of both marginals. Here`s the coverage of quantity and the instructive attribution of disagreements, while Kappa hides the information. In addition, Kappa introduces some challenges in calculation and interpretation, as Kappa is a ratio. It is possible that the kappa ratio returns an indefinite value due to zero in the denominator. Moreover, a report does not betray either its counter or its denominator. It is more informative for researchers to point out disagreements in two components, quantity and allocation. These two components describe the relationship between the categories more clearly than a single summary statistic. If forecast accuracy is the goal, researchers can more easily think about how to improve forecasting by using two components of quantity and allocation instead of a Kappa report. [2] A joint review of the PA and the NA focuses on the potential concern that the PO could be subject to accidental inflation or distortion in the event of extreme policy interest rates. Such inflation, if it exists at all, would concern only the most frequent category. This is because, although the percentage of concordance is the same, the percentage of concordance that would occur “by chance” is significantly higher in the first case (0.54 versus 0.46).

Cohens Kappa is a unique synthesis index that describes the strength of the Inter-Rater agreement. Note that Cohen`s kappa only measures the concordance between two evaluators. For a similar degree of compliance (Fleiss` kappa) used if there are more than two evaluators, see Fleiss (1971). The Fleiss Kappa is however a multi-miss generalization of Scott`s Pi statistics, not Cohens Kappa…