Since the first general circulation climate models were introduced in the 1960s , an ever-increasing amount of effort has been spent developing and improving these models to produce simulations that are increasingly more realistic and feature more of the processes and interactions present in the real world. The progress and understanding of the processes governing the Earth's climate as a result has been impressive. However, even after decades of research, the range of predictions of some key characteristics of the Earth's future climate coming from these models are actually increasing rather than narrowing with time, one particular characteristic being the amount of warming due to doubling of CO2 at equilibrium, known as equilibrium climate sensitivity (ECS, ). Even though the latest state-of-the-art climate models in the Coupled Model Intercomparison Project 6 (CMIP6, ; CMIP6_database, 2021) have a larger range of ECS values ([1.84; 5.68 K]) than previous CMIP model ensembles , the latest IPCC estimates have actually narrowed. For decades the IPCC “likely” range for ECS was between 1.5 and 4.5 K. In the latest report this was reduced to between 2.5 and 4 K with a best estimate of 3 K.

There have been numerous attempts to constrain ECS using the historical warming record, paleoclimate data and climate model experiments. Researchers have also used the emergent constraint technique to constrain ECS (see, for example, , and the many references listed in ). The basic idea of emergent constraints is to identify an observable of the climate x that varies significantly across a climate model ensemble and that exhibits a statistically significant relationship f(x) with another variable y describing an aspect of the climate model's future state. The relationship y=f(x)+ε, is referred to as an “emergent relationship” where ε is a relatively small departure from f. Since x is observable, it can be measured in the real world. As such, f may then place a useful constraint on y, provided that the measurement uncertainty in x is small compared to the range of simulated values. This constraint is “emergent” because the emergent relationship f cannot be diagnosed from a single climate model. It becomes apparent only when the full ensemble is analysed.

There are pitfalls with the emergent constraint approach that must be guarded against particularly when the emergent relationships are not founded on well-understood physical processes. For example, data-mining outputs from climate models could lead to spurious correlations and less than robust constraints on future changes . Care is also needed drawing statistical inferences from ensembles of small numbers of models. The problem is compounded if models within the ensemble share common components giving a smaller effective ensemble size . Observations used to guide model development also may lead to dependencies and common structural inaccuracies .

One way of guarding against spurious correlations between x and y is to use analytical solutions of simplified models of the full-complexity climate models to predict the emergent relationship f; f can then be tested against the results from the complex models. This approach was used in (CHW18) where the analytical solution of the one-box or Hasselmann model provided an emergent relationship between the statistics of historical global annual mean temperature variability (x) and ECS (y; see Sect. 2 for further details). This emergent relationship was tested and found to hold in the CMIP5 (; CMIP5_database, 2021) models, although this was not without some debate regarding the applicability of the theory (; see Sect.  for a discussion of these points). However, since these works were published, the newer CMIP6 ensemble has become available. showed that many of the emergent constraints found in CMIP5 were much weaker in the newer ensemble, including that of CHW18.

As the constraint in CHW18 was based on a relationship derived from reasonable physical principles, it is of interest to find out why it got weaker in CMIP6. Some possible reasons are as follows.

The simple theory is not applicable to climate models and the real world. However, simple models (particularly two-box models) are regularly used to reproduce the annual global mean temperature response of climate models, and they do it well (see ).

The central interest of this paper is to test the assumptions that go into the derivation of the emergent relationship in CHW18. These assumptions are outlined in Sect.  and then tested in the CMIP5 and CMIP6 model ensembles with the aim of understanding why the emergent relationship in CHW18 is weaker for the CMIP6 model ensemble. Of course all assumptions will be ultimately wrong if perfect agreement is expected (the often used quote “all models are wrong” applies). However, “some models are useful”, and we look for agreement “for all practical purposes (FAPP)”, a term coined by John Bell . We will largely not be interested in the final step of obtaining the emergent constraint that results from combining the emergent relationship with observations for reasons we will outline later in the paper.

The structure of the rest of the paper is as follows. In Sect.  we review the methodology of CHW18 and how it is used in this study. In Sect.  we explicitly list, discuss and test the assumptions in CHW18 and show which assumption fails for the CMIP6 model ensemble. In Sect.  we show how to recover a robust emergent relationship in both CMIP5 and CMIP6 ensembles by including the forcing parameter in the predictor x. In Sect.  we perform a rigorous test of the emergent relationship theory by numerical simulation and show it does a reasonable job (FAPP) reproducing the results seen in each of the ensembles of the full-complexity CMIP climate models. We discuss and conclude in Sect. .

The response of the global mean surface air temperature anomaly T(t) with time t to forcing Q(t) is assumed to be well modelled by the one-box or Hasselmann model (, hereafter H76) in CHW18. Forcing in this model comes from random, short-timescale weather noise and other external sources such as solar radiation and changes in greenhouse gas concentrations. Air temperature sensitivity to forcing is parameterised by λ, a term that lumps all the effects of the Earth system's feedbacks together. The single box has heat capacity C. In this model, T(t) evolves according to 1CdTdt=Q(t)-λT(t). Solving this model results in a linear relation between ECS and a metric of temperature variability Ψ, which is a form of fluctuation–dissipation theorem . Explicitly 2ECS=2Q2×CO2σQΨ. Here Q2×CO2 is the radiative forcing resulting from doubling the atmospheric CO2 concentration and σQ is the standard deviation of a zero mean white-noise process designed to model the fast (sub-annual), chaotic weather forcing on the slower Earth system components. Ψ can be measured from temperature observations and is defined as 3Ψ=σT-log⁡α1T, where σT is the standard deviation and α1T is the autocorrelation at 1-year lag of annual global mean temperature. Details of this derivation can be found in CHW18 and .

CHW18 calculated the pair of values (Ψi, ECSi) for each of the n=16 CMIP5 climate models labelled by i∈{1,2,…,n} performing a simulation of the historical period 1880–2016. Plotting the n pairs confirmed the theoretically expected Ψ vs. ECS linear “emergent relationship” with good correlation (r=0.77, r in this paper denotes Pearson's correlation coefficient). Combining this resulting emergent relationship with Ψ from observational records of the same period gave an emergent constraint on ECS of 2.8±0.6 °C (plus minus values are 66 % confidence intervals).

Although there were more CMIP5 models available than the n=16 used in CHW18, the choice of one model per modelling centre was made to avoid biasing the emergent constraint towards similar models. Where multiple models were available from the same centre, the model with the lowest root-mean-square error to the observational temperature record was chosen. and repeated the analysis of CHW18 including these additional models and thus had a larger CMIP5 ensemble (larger n). They found the emergent relationship got slightly weaker, although it was still “highly significant” in the language of .

In this paper we use the CHW18 methodology (further detailed in the original paper) and apply it to CMIP5 and CMIP6 models with the following differences. Here we look at the historical period 1880–2005 for both CMIP5 and CMIP6 ensembles following rather than 1880–2016 as in CHW18. This is because the standard CMIP5 historical experiment ends in 2005. Increasing the time period to present day by concatenating with one of the CMIP rcp or ssp future projection experiments slightly increases the strength of the correlation in the emergent relationship. We also use a different ensemble of 15 CMIP5 models corresponding with those analysed in . also lists FGOALS-s2; however, we leave this model out as it does not have a historical simulation with which to calculate Ψ. We use the ensemble as their published parameter values are used in Sect.  to run simulations of box models. These simulations are used to compare the theory with the full-complexity CMIP5 models. To make a fair comparison limits us to the same set. For the CMIP6 ensemble we use all models that have the necessary simulations for our analysis (piControl, historical and abrupt-4xCO2), a set of n=33 models. For both CMIP ensembles we use one run for each model, preferably the one labelled r1i1p1 (CMIP5) or r1i1p1f1 (CMIP6) where it exists. We look at the results for different runs of the same model in Sect. , although we find no qualitative changes to the findings with the r1i1p1 (CMIP5) or r1i1p1f1 choices. A list of models used and their parameter values is given in Appendix .

The (Ψ, ECS) emergent relationships for CMIP5 and CMIP6 ensembles are shown in Fig. . The CMIP5 ensemble shows good correlation between Ψ and ECS, r(Ψ,ECS)=0.66; however, for CMIP6 this is weaker, r(Ψ,ECS)=0.31, confirming the results of CHW18 (although with slightly different historical period and set of CMIP5 models) and (CMIP6).

use the following definitions for significance based on p value. An emergent relationship is called “highly significant” if p<0.02, “barely significant” if 0.02≤p<0.05, “almost significant” if 0.05≤p<0.1 and “far from significant” if p≥0.1. We adopt their definitions in this paper. We find the (Ψ, ECS) emergent relationship highly significant for CMIP5 and almost significant for CMIP6.

The following assumptions are made in the CHW18 methodology to obtain the emergent relationship between Ψ and ECS.

A1. The T(t) response to Q(t) is modelled well by H76 for timescales greater than 1 year and less than the detrending window length (55 years in CHW18).

A2. H76 is solved with a random, white-noise forcing Q(t) of zero mean and standard deviation σQ. This is designed to parameterise internally generated variability (from weather, for example, ). It is assumed that the response from all other sources of forcing in the historical period such as (but not limited to) greenhouse gases (GHGs), solar irradiance and volcanoes can be removed via detrending to a good approximation so that Eq. () applies to this period in both observations and CMIP models.

A3. The forcing parameters, Q2×CO2 and σQ in Eq. (), are uncorrelated to ECS and their variation is small relative to the variation in Ψ. This requirement makes Ψ a good predictor of ECS.

There are further assumptions concerning the quantification of sources of uncertainty (structural, observational, etc.) in deriving the emergent constraint in CHW18. These are considered in more detail in and . However, as we look only at the emergent relationship here, these will not discussed further.

We have confirmed that Ψ is a good predictor of ECS for CMIP5 models, albeit not for CMIP6 models. In Fig. , when σQ is included in the x-axis predictor variable, a good emergent relationship is recovered for both CMIP ensembles, and both have highly significant p values of p<0.001. One can also include Q2×CO2 (although it is uncorrelated to ECS in both CMIP ensembles) in the predictor, i.e. ECS ∝Q2×CO2σQΨ, to get a similarly skilful emergent relationship (figure not shown). We restrict this to Ψ/σQ as minimal degrees of freedom are preferred.

Where does the skill in predicting ECS using Ψ/σQ come from? In CMIP5, it came from Ψ (an observable). There is no skill in σQ (it is uncorrelated with ECS). In CMIP6 the converse is roughly correct. There is limited correlation with ECS from Ψ but good correlation from σQ, which, to our knowledge, is unfortunately not directly observable.

Theoretically, these findings should hold equally well in the piControl run, although the emergent relationships should have slightly weaker correlation for reasons outlined in Sect.  and . Again, we find this is roughly true (see Fig. ). For the piControl experiments we analyse the longest common period simulated in the CMIP5 and CMIP6 ensembles, which is 200 years.

H76 in Eq. () predicts ECS =kQ2×CO2σQΨ with a constant of proportionality k=2. We investigate the empirical value of k for the full-complexity models next. As this relation should hold for all models running any experiment (provided you can remove the forced signal), we have plotted ECS against Q2×CO2σQΨ for an ensemble composed of all CMIP5 and CMIP6 models running both piControl and historical experiments (a total of 96 data points) to determine the empirical k (Fig. a). While the proportionality holds with a high correlation value and significance (r=0.74, p<0.001), the empirical constant is k∼22 rather than the 2 predicted by H76, i.e. the theoretical prediction of ECS from H76 is lower than the full-complexity models suggest.

showed the two-box and diffusion models also shared the linear ECS–Ψ proportionality of H76 FAPP, albeit with slightly different variables and constants. We have therefore also compared the more realistic two-box model theoretical predictions of k (Eq. 23 in ) to the full-complexity models (Fig. b). Using the two-box values in Tables  and  brings the empirically determined k∼1.32 closer to the two-box theoretical value (k=2) with similar high correlation and significance (r=0.76, p<0.001). The two-box model adds a second, longer timescale to H76, mimicking the full-complexity models more closely; however, the theoretical k is still slightly low. The lower prediction of k than the empirical results suggest this could be due to the full-complexity models having other timescales that the conceptual models do not. Although the conceptual models predict a linear ECS-Ψ proportionality also seen in the full-complexity CMIP models, they do not predict the constant of proportionality well. This is why the empirically determined k should be used to obtain an emergent constraint as in CHW18.

For both CMIP ensembles we have used one run for each model, preferably the one labelled or r1i1p1 (CMIP5) or r1i1p1f1 (CMIP6) where it exists; however, we could have equally chosen any r*i*p* (CMIP5) or r*i*p*f* (CMIP6) for each model provided multiple runs of the same model exist. In this section we show the results for r1i1p1 or r1i1p1f1 are representative of a typical random run choice.

For models with multiple runs, we have drawn at random one run (r*i*p* or r*i*p*f* for CMIP5 and CMIP6 respectively) for each model and repeated the analysis in the previous sections multiple times. For the historical runs, in both CMIP5 and CMIP6 ensembles, many models do repeated runs, sometimes multiple times. For example, the CMIP6 model CanESM5 has the most runs, performing the historical experiment 50 times. There are 1.6×1012 and 5.4×1020 unique permutations for the same set of CMIP5 and CMIP6 models, respectively, performing the historical experiment. These numbers are clearly too large to search exhaustively. We have therefore drawn 1000 unique permutations for the historical experiment and repeated the analysis in this paper, i.e. calculated the Pearson correlation, r, for every one of these 1000 permutations. The results are shown in the upper half of Tables  (CMIP5) and  (CMIP6). We find that the results reported for r1i1p1 and r1i1p1f1 where they exist are fairly typical of a randomly chosen set of runs; i.e. they fall within 1 standard deviation of the mean value in the CMIP5 ensemble. The CMIP6 historical experiment r(Ψ,ECS) with r1i1p1f1 is slightly higher than would be expected (mean value for a randomly chosen permutation is r=0.18±0.11, far from significant, compared to r=0.31, almost significant). We have also listed the outermost values (min and max) found in the 1000 random run choice distribution for completeness. As one would expect with a large enough sample, there is a chance of finding non-representative correlations; i.e. for the CMIP5 historical ensemble there is a small possibility that you might find an r(Ψ,ECS)=0.14 (far from significant) or even r(Ψ,ECS)=0.82 (highly significant) in a random pick of runs. These are, however, not typical results.

For the piControl runs, there are 15 and 24 unique permutations for the same set of CMIP5 and CMIP6 models, respectively. The relatively low number of unique permutations is due to the low number of repeated runs for the piControl experiment. We have run the same analysis in this paper; i.e. we calculated the Pearson correlation, r, for every one of these permutations. The results are shown in the lower half of Tables  (CMIP5) and  (CMIP6). Because of the low number of unique permutations, the range of results is much narrower than the historical experiment. Again, we find that the results reported for r1i1p1 and r1i1p1f1 where they exist are fairly typical of a randomly chosen set of runs; i.e. they fall within 1 standard deviation of the mean value in the CMIP5 and CMIP6 ensembles with the exception of CMIP5 r(Ψ,ECS)=0.64 (highly significant) for r1i1p1 compared to r=0.61±0.02 (highly significant) for a randomly chosen permutation. This is actually the highest value of r (max) found in that experiment.

In Sect.  we found that by including the forcing parameter σQ in the predictor for ECS an emergent relationship could be recovered for both CMIP5 and CMIP6 ensembles. These relationships are present in both the historical and piControl experiments, giving confidence in the underlying theoretical basis FAPP.

In this section we make a more demanding test of the theoretical basis by asking if theory alone can simulate the full-complexity CMIP model ensemble r(Ψ,ECS) and r(Ψ/σQ,ECS) results. To do this, we create a H76 model emulator of each of the i∈{1,2,…,n} full-complexity CMIP5 and CMIP6 models used in the preceding figures. With the emulator H76 models we can build emulator H76 CMIP5 and CMIP6 ensembles and run analogous historical and piControl experiments with them. This will allow us to compare the results of the pure theory used in CHW18 with that of the full-complexity CMIP model ensembles. We also fit the more complete two-box model in addition for comparison (see Sect. ).

The aim of this paper was to understand why the strong emergent relationship from CHW18 found in the CMIP5 model ensemble weakened in the newer CMIP6 ensemble. This emergent relationship was based on reasonable, albeit simple, physical principles, and thus it is interesting (and important) to understand the differences between the theory and full-complexity models. A number of assumptions (Sect. ) were made in deriving the theoretical emergent relationship between the predictor Ψ, a metric based on annual global mean temperature variability and ECS, the predictand in CHW18. We have shown the “no correlation between forcing and ECS” assumption no longer holds for the CMIP6 ensemble. In particular, the parameter σQ describing random forcing from internally generated variability, is correlated to ECS in CMIP6, and when this parameter is incorporated into the predictand, a good emergent relationship is recovered for both CMIP ensembles.

Assumption A3 stated that the forcing parameters, Q2×CO2 and σQ, could be treated as constants across a model ensemble. While this is a fair assumption for Q2×CO2 for both CMIP ensembles and σQ in the CMIP5 ensemble, we have shown that σQ is correlated to ECS in the CMIP6 ensemble. We have also shown that when the predictor of ECS is changed to Ψ/σQ, good emergent relationships are recovered in both CMIP ensembles for both piControl and historical experiments. We also showed that pure theory could reproduce the full-complexity CMIP model results using H76 and two-box CMIP emulator ensembles. Although the proportionality between ECS and the predictor Q2×CO2σQΨ has a high correlation and significance, simple pure theory underestimates the constant of proportionality. Aside from this, these results give us confidence the theoretical basis of CHW18 still applies to CMIP6 models as it did for CMIP5 FAPP. Testing the theoretical basis was the underlying aim of our study.

However, several questions remain. Can we estimate σQ from observations and therefore get an emergent constraint on ECS from the CMIP6 ensemble? Why is σQ correlated to ECS in CMIP6 and not CMIP5? σQ is a parameter designed to reproduce the observed global annual mean temperature variability, σT, in the non-chaotic H76 and two-box models. In the full-complexity models and the real world, this parameter attempts to capture chaotic internal variability and sub-annual (fast) feedbacks. It is fitted in this study using σT (an observable) as well as the unobservable two-box parameters. The reliance on these unobservable two-box parameters makes it appear that getting an estimate of σQ in the real world and thus an emergent constraint may be tough. However, there may be observable proxies for σQ that we have not yet found.

An obvious place to start looking for a proxy for σQ is in basic theory. The simplest example that one can imagine is 8Q(t)=N(t)+λT(t), where N(t) is the net top-of-the-atmosphere radiative flux. However this still requires knowledge of λ. Even given knowledge of λ it is well known that N is poorly correlated to T where most of the change in N and T is driven by internal variability (although this relation works very well for large forced trends; e.g. the Gregory method works well applied to large stepped increases in CO2). There are several models and methods that get much better correlations between N and T when most of the changes are driven internally by taking into account the spatial distributions (the so-called pattern effect; see ). We leave this to a future study.

The question of why σQ is correlated to ECS in CMIP6 and not CMIP5 is also left unanswered. However, one can speculate why this may be the case. As previously mentioned σQ is a fitting parameter that is designed to capture the effect of chaotic internal variability and sub-annual (fast) feedbacks on global mean temperature variability. showed that the increased range of ECS in the CMIP6 models could be explained by the increased range in cloud feedbacks (see also ). As σQ is fitted to annual temperature time series, some of this fast (sub-annual) cloud feedback effect could be included in σQ, correlating it to ECS. We leave concrete answers to a future study.

We have understood what assumption in the theoretical emergent relationship for CHW18 was responsible for the weakened correlation in CMIP6, namely that σQ is correlated with ECS. When accounted for, good emergent relationships are recovered. Although the information that the simple theory holds FAPP is useful and that σQ is correlated in CMIP6 with ECS is interesting, it is disappointingly not useful in constraining ECS due to the unobservable nature (we think) of σQ. In this sense, the method in CHW18 does not produce a useful emergent constraint on CMIP6 because the extra degree of freedom in σQ needs to be incorporated.

tested 11 emergent constraints found in CMIP5, and nearly all of these got weaker in CMIP6. We do not know whether they failed for similar reasons. Indeed, many of them do not have a simple theoretical model as a basis for their emergent relationship, which means that assumption testing, the approach we follow in this paper, would be difficult to do. This is why we argue that emergent constraints should be based on a testable, falsifiable theoretical model. This aids understanding and lifts emergent constraint research from looking for strong correlations between variables to a more scientific approach of testing hypotheses of how the Earth system works. However, looking at all these other emergent constraints and identifying why they got weaker in CMIP6 would be very beneficial to understanding and useful to the community.

Emergent constraints based on theory with minimal degrees of freedom are most likely to be the most robust and useful. Constraints such as on snow albedo feedback where the predictor (seasonal cycle snow albedo feedback) and predictand (climate change snow albedo feedback) are the essentially the same variable have been shown to be robust through three CMIP generations . Other constraints of this type that are likely to be more robust are the transient climate response constraints of and where near-term historical warming is the predictor of future longer-term warming.

Even if emergent relationships based on sound theoretical principles do fail there is still information to be gleaned on understanding why. Today, there is even more of an opportunity for the top-down insights of specific conceptual models to meet and complement the comprehensive, bottom-up approach from state-of-the-art climate models; there are many more high-quality observations; the global warming signal has also become clearer over time; and there is also a large archive of past and present climate model simulations.