22 Dec 2021

22 Dec 2021

Review status: this preprint is currently under review for the journal CP.

Quantifying and reducing researcher subjectivity in the generation of climate indices from documentary sources

George Adamson1, David Nash2,3, and Stefan Grab3 George Adamson et al.
  • 1Department of Geography, King’s College London, London, United Kingdom
  • 2School of Applied Sciences, University of Brighton, Brighton, United Kingdom
  • 3School of Geography, Archaeology and Environmental Studies, University of the Witwatersrand, Johannesburg, South Africa

Abstract. The generation of index-based series of meteorological phenomena, derived from narrative descriptions of weather and climate in historical documentary sources, is a common method to reconstruct past climatic variability. This study is the first to explicitly examine the degree of inter-rater variability in producing such series, a potential source of bias in index-based analyses. Two teams of raters were asked to produce a five-category annual rainfall index series for the same dataset, consisting of transcribed narrative descriptions of meteorological variability for 11 ‘rain-years’ in nineteenth-century Lesotho, originally collected by Nash and Grab (2010). One group of raters (n = 71) comprised of students studying for postgraduate qualifications in climatology or a related discipline; the second group (n = 6) consisted of professional meteorologists and historical climatologists working in southern Africa. Inter-rater reliability was high for both groups, at r = 0.99 for the student raters and r = 0.94 for the professional raters, although ratings provided by the student group disproportionately averaged to the central value (0: normal/seasonal rains) where variability was high. Back-calculation of intraclass correlation using the Spearman-Brown prediction formula showed that a target reliability of 0.9 could be obtained with as few as eight student raters, and four professional raters. This number reduced to two when examining a subset of the professional group (n = 4) who had previously published historical climatology papers on southern Africa. We therefore conclude that variability between researchers should be considered minimal where index-based climate reconstructions are generated by trained historical climatologists working in groups of two or more.

George Adamson et al.

Status: open (until 22 Feb 2022)

Comment types: AC – author | RC – referee | CC – community | EC – editor | CEC – chief editor | : Report abuse

George Adamson et al.

George Adamson et al.


Total article views: 326 (including HTML, PDF, and XML)
HTML PDF XML Total BibTeX EndNote
271 52 3 326 2 2
  • HTML: 271
  • PDF: 52
  • XML: 3
  • Total: 326
  • BibTeX: 2
  • EndNote: 2
Views and downloads (calculated since 22 Dec 2021)
Cumulative views and downloads (calculated since 22 Dec 2021)

Viewed (geographical distribution)

Total article views: 307 (including HTML, PDF, and XML) Thereof 307 with geography defined and 0 with unknown origin.
Country # Views %
  • 1


Latest update: 21 Jan 2022
Short summary
Descriptions of climate held in archives are a valuable source of past climate variability, but there is a large potential for error in assigning quantitative indices (e.g. −2: v. dry to +2: v. wet) to descriptive data. This is the first study to examine this uncertainty. We gave the same dataset to 71 postgraduate students and six professional scientists, findings that error can be minimised by taking an average of indices developed by 8 postgraduates, and only two professional climatologists.