The method for model-proxy comparison introduced here is based on a probabilistic approach. If f1(x) and f2(x) are two probability (density) functions of a certain variable x, one can measure the degree of similariy between these two distributions by calculating the overlapping coefficient OVL e.g., OVL=∫-∞∞min⁡(f1(x),f2(x))dx. Hence, OVL yields the degree of overlapping between f1 and f2, or in other words, the degree of similarity between the two distributions. Here f1 and f2 represent the probability distributions from climate models (hereafter referred to with subscript m: fm) and proxies (subscript p: fp), and x some climate variable. Since OVL is a probabilistic measure it is expressed in %, and thus varies between 0% (no agreement) and 100% (complete agreement). For simplicity, we use OVL=50% as a threshold for satisfactory agreement. Note that OVL refers to agreement and not to any statistical probability. These are, however, somewhat related: if the SD of the proxy data is greater than that of the model, OVL is similar to the probability that the true value in the proxy is captured by the model ± two SDs (Fig. S1 in the Supplement).

A major advantage of using OVL over other probabilistic measures is its simplicity; Eq. () can be applied to any model and proxy data once fm and fp are known. Prior to calculating OVL, however, one must ensure that fm and fp are probability distributions of a variable, which is represented at the same point in time and space. Hence, OVL should essentially be used for point-by-point comparisons only. The drawback of the OVL method is that it does not provide any information on the (dis)agreement expressed in data units (e.g. in ∘C or mm yr-1). Therefore OVL should preferably be used together with some measure of the distance between the model and proxy, e.g. the difference between the mean estimates.

The main advantage of the overlapping (OVL) method is that the proxy uncertainty and model variability are accounted for as they alter the SDs of the corresponding probability distributions. A necessary requirement for calculating OVL (Eq. ) is that one can estimate the probability distributions of both the model and the proxy data. For this, it is necessary to have sufficient data to calculate data-specific statistical quantities such as the mean (x‾) and the SD (σ). Second, it is important to fit appropriate probability distributions to the data.

In the following we discuss probability distributions derived from climate models and proxy data. We restrict our discussion to examples from our previous work, which includes vegetation proxies based on the co-existence approach of the mean annual temperature (MAT) and precipitation (MAP) , as well as mammal proxies of MAP . Note, however, that the methodology provided here is also applicatable to other quantitative proxy sources.

In climate models, temperature and precipitation variability usually follow a Gaussian (Normal) and a gamma distribution, respectively. For both distributions, only x‾m and σm are required to calculate the corresponding probability functions. For simplicity, we define σm as the “inter-variability” of x; for example, when x‾m is the annual mean, σm represents the inter-annual variability. Since the climate variability recorded in the proxy is often associated with lower frequency, it is likely that inter-annual variations in the model underestimate the variability of the proxy data. In proxy records it is however often difficult to separate between uncertainty from variability. Consequently, assuming inter-annual variability is sufficient to understand the general behaviour of OVL.

The use of a gamma instead of a Gaussian distribution for precipitation is motivated by the fact that the probability frequency is zero for negative values, which is one of the main features of the gamma distribution. Furthermore, the gamma distribution can potentially change the shape depending on the characteristics of the data. If x‾<σ, the gamma distribution is exponentional with decreasing probabilities toward higher values e.g., Fig. 1 in. If, on the other hand x‾>σ, the gamma distribution is zero at x=0 and has a skewed “Gaussian-like” shape Fig. 1 in. The advantage of using the gamma distribution to represent precipitation frequency has been noted in several studies, e.g., , and .

Vegetation proxies based on the co-existence approach yield intervals with homogeneous probability rather than a most likely estimate and a standard error . Hence, this proxy is represented by a distribution with equal probability for all values within the co-existence range [xminxmax]: f(x)=1/(xmax-xmin),xmin≤x≤xmax0,elsewhere. Using mammal proxies to estimate past MAP yields an estimate of the mean along with a standard error of 388 mm . To avoid negative precipitation values, mammal data is best represented by a gamma distribution.

To better understand the behavior of OVL, Fig.  shows some illustrative examples. Figure a shows an example of the homogeneous probability distribution that we use for the vegetation proxy (see Eq. ). Thus, when comparing vegetation proxies with model data, one must deal with two different probability distributions. As a consequence, the maximum OVL value cannot be as high as 100%, but will depend on the properties of the corresponding distributions. Figure a depicts the maximum possible agreement between the homogeneous distribution and a Gaussian distribution (≈82%); compared with a gamma distribution instead, the maximum agreement is very similar to the Gaussian case (OVL≈81%; not shown).

In the next two panels (Fig. b and c) we consider OVL of two gamma distributions. The “model” distribution is assumed to be the same in both cases, but the “proxy” distribution differs significantly; in the first example (b), the proxy has the same variability as the model, but its mean value is shifted so that the mean value of the model is clearly situated outside the error bars of the proxy. In the second example (c), the proxy and model have the same mean, but the proxy error is 4× the model variability (i.e. σp=4σm). Despite having the same mean value in (c), our results suggest that the model-proxy agreement is worse than in (b) (OVL=40% vs. OVL=45%).

The poor agreement in Fig. c is associated with the large uncertainty of the proxy. To reveal a better understanding on how proxy uncertainty ranges influence OVL, we derive a simple analytical relationship between OVL and (proxy) uncertainty based on two hypothetical probability distributions with triangular geometry (see Appendix A and Fig. S2). If both of these triangular distributions have the same mean (Δx‾=0 in Fig. S2), the overlapping coefficient is simply given by (see Appendix A for derivation): OVL=21+q, where q is a non-dimensional parameter of the length ratio (i.e. SD ratio) of the wider triangle to the narrower. Thus, assuming that the proxy uncertainty is greater than the model variability, q is simply given by q≡σp/σm. If we apply the simple analytical solution in Eq. () to the example in Fig. c we obtain OVL=40% (q=4⇒OVL=2/(1+4)=40%), thus exactly the same as in the case with two gamma distributions in Fig. c. In fact, despite its simplicity, the analytical solution shows a very good agreement with other conventional probability distributions for the whole q spectrum (Fig. a). As a result, if the mean values from the model and proxy are similar, OVL>50% requires that q<3, i.e. that the proxy uncertainty is less than 3× the model variability. In other words, if q>3 the model and proxy are most probably not in agreement even if their means are similar. Thus, proxies with such large uncertainty should preferably not be used for a quantitative evaluation of the model performance.

Somewhat surprisingly, however, is that large uncertainty proxies in some cases have a greater agreement with the model if their means are further apart. This is illustrated in Fig. b, which shows OVL as a function of q with the proxy mean located at x‾p=x‾m+2σm. In particular for the exponentional shape of the gamma distribution, there is a range of q values between ∼2 and ∼6.5 for which OVL is greater than 50%. Hence, for proxies that are best represented by an exponential gamma distribution, e.g. low precipitation estimates with a large uncertainty, the agreement with the model becomes better if the model mean is significantly lower than the proxy mean. In this special case, similar mean estimates are only desirable for a good agreement if the proxy uncertainty is within the same order of magnitude as the model variability.

In this section, we apply the OVL approach to real climate model and proxy data. For this purpose we use data from a previously published paleoclimate modelling study . conducted a fully-coupled atmosphere–ocean simulation of the late Miocene (7–11 millions of years ago) using ECHAM5/MPI-OM, where ECHAM5 is the atmosphere model e.g. and MPI-OM the ocean model e.g.. In , ECHAM5/MPI-OM was integrated to equilibrium (2500 years) with late Miocene boundary conditions. The late Miocene climatology was constructed from the last 10 years of the 2500-year simulation. These results were then compared to late Miocene estimates of MAT and MAP from vegetation fossils (co-existence approach) as well as MAP from mammal hypsodonty. In total, the proxy dataset consists of 472 fossil site records with 69 (60) MAT (MAP) estimates derived from vegetation and 343 MAP estaimates from mammals, respectively. The proxy dataset is the same as in except for a few additional vegetation fossils from Siberia . Therefore the reader is referred to for more details on the proxy records and model results. In this study we discuss only the methodology of the model-proxy comparison.

To evaluate the model-proxy agreement used a method that was first introduced by . This method considers the minimum distance between the climate ranges of the model and proxy. If this distance is zero, i.e. the error bars of the proxy overlap with the model variability, the model and proxy are considered to be in agreement. If there is no overlap, the model-proxy (dis)agreement is quantified by the minimum distance between the climate intervals. Hereafter we refer to this method as the “minimum distance method”. For the vegetation proxy these intervals are constructed as a result of the co-existence approach, which yields a maximum and a minimum estimate. For the mammal proxy, the intervals are constructed by taking the mean MAP estimate ± the standard error of 388 mm. If the mean estimate is less than 388 mm, the lower bound of the interval is set to zero. The climate intervals in the model were constructed by taking the maximum and minimum values of MAT and MAP for each gridpoint from the last 10 yr of the model simulation.

Using the minimum distance method, the modelled MAT agrees fairly well with the MAT from the vegetation proxy (Fig. a). The best agreement is achieved over the central parts of Europe and Asia. North (South) of these regions the model is generally too cold (warm). In terms of MAP, the model-proxy agreement seems to be even better (Fig. b), especially when comparing with the mammal data. With respect to the vegetation proxy, the model has a good agreement over most of Asia, but produces too dry conditions over most of Europe. In essence, the good agreement between the model and the mammal proxy in Fig. b results from the fact that the modelled MAP over most localities is within the interval given by the MAP estimate from mammals ± the standard error (388 mm).

Figure c and d show the corresponding values of OVL for the MAT and MAP cases, respectively. The grey markers depict the cases for which OVL>50%. According to the OVL calculations the MAT agreement over central Europe remains fairly high, whereas the agreement over central Asia vanishes almost completely as compared with the minimum distance method. In terms of MAP, the difference is even more pronounced; for most of the localities that exhibited a model-proxy agreement with the minimum distance method, OVL is substantially lower than 50%.

To reveal some insight into the low OVL, Fig. e and f display the corresponding values of q (=σp/σm) for all localities. Above we found that similar model and proxy means can yield OVL>50% only if q<3. For most vegetation proxies, q is less than 3 (Fig. e and circles in f). Hence, for these records the relatively poor OVL agreement is caused by different mean values between the model and the proxy. The large uncertainty of the mammal data, however, yields mostly high q values, especially over southern Europe as well as western and central Asia (triangles in Fig. f). Thus, the poor OVL agreement in these regions can be explained by large uncertainties in the proxy. Further, the somewhat better agreement over southeast Asia coincides with much lower q values in that region. Note that there are a couple of localities that exhibit OVL>50% despite high q values (e.g. on the Arabic peninsula). These localities are predominately associated with low precipitation rates, i.e. an exponential shape of the gamma distribution. Hence, for these localities the situation discussed in Fig. b applies: despite a high q value, OVL is large because the model mean is significantly smaller than the mean estimate from the proxy (dashed line in Fig. b).