Giovagnoli, Alessandra, Marzialetti, Johnny and Wynn, Henry P. 
ORCID: 0000-0002-6448-1080
(2008)
A new approach to inter-rater agreement through stochastic orderings: the discrete case.
Metrika, 67 (3).
pp. 349-370.
ISSN 0026-1335
Abstract
We wish to study inter-rater agreement comparing groups of observers who express their ratings on a discrete or ordinal scale. The starting point is that of defining what we mean by “agreement”. Given d observers, let the scores they assign to a given statistical unit be expressed as a d-vector in the real space. We define a deterministic ordering among these vectors, which expresses the degree of the raters’ agreement. The overall scoring of the raters on the sample space will be a d-dimensional random vector. We then define an associated partial ordering among the random vectors of the ratings, illustrate a number of its properties, and look at order-preserving functions (agreement measures). In this paper we also show how to test the hypothesis of greater agreement against the unrestricted hypothesis, and the hypothesis of equal agreement against the hypothesis that an agreement ordering holds. The test is applied to real data on two medical observers rating clinical guidelines.
| Item Type: | Article |
|---|---|
| Official URL: | http://link.springer.com/journal/184 |
| Additional Information: | © 2008 Springer-Verlag |
| Divisions: | Centre for Analysis of Time Series |
| Subjects: | Q Science > QA Mathematics |
| Date Deposited: | 26 Feb 2014 11:23 |
| Last Modified: | 13 Sep 2024 22:33 |
| URI: | http://eprints.lse.ac.uk/id/eprint/55871 |
Actions (login required)
 |
View Item |
