Library Header Image
LSE Research Online LSE Library Services

A new approach to inter-rater agreement through stochastic orderings: the discrete case

Giovagnoli, Alessandra, Marzialetti, Johnny and Wynn, Henry P. ORCID: 0000-0002-6448-1080 (2008) A new approach to inter-rater agreement through stochastic orderings: the discrete case. Metrika, 67 (3). pp. 349-370. ISSN 0026-1335

Full text not available from this repository.

Identification Number: 10.1007/s00184-007-0137-4


We wish to study inter-rater agreement comparing groups of observers who express their ratings on a discrete or ordinal scale. The starting point is that of defining what we mean by “agreement”. Given d observers, let the scores they assign to a given statistical unit be expressed as a d-vector in the real space. We define a deterministic ordering among these vectors, which expresses the degree of the raters’ agreement. The overall scoring of the raters on the sample space will be a d-dimensional random vector. We then define an associated partial ordering among the random vectors of the ratings, illustrate a number of its properties, and look at order-preserving functions (agreement measures). In this paper we also show how to test the hypothesis of greater agreement against the unrestricted hypothesis, and the hypothesis of equal agreement against the hypothesis that an agreement ordering holds. The test is applied to real data on two medical observers rating clinical guidelines.

Item Type: Article
Official URL:
Additional Information: © 2008 Springer-Verlag
Divisions: Centre for Analysis of Time Series
Subjects: Q Science > QA Mathematics
Date Deposited: 26 Feb 2014 11:23
Last Modified: 16 May 2024 00:50

Actions (login required)

View Item View Item