Emergent constraints on equilibrium climate sensitivity in CMIP5: do they hold for CMIP6?

An important metric for temperature projections is the equilibrium climate sensitivity (ECS), which is defined as the global mean surface air temperature change caused by a doubling of the atmospheric CO₂ concentration. The range for ECS assessed by the Intergovernmental Panel on Climate Change (IPCC) Fifth Assessment Report is between 1.5 and 4.5 K and has not decreased over the last decades. Among other methods, emergent constraints are potentially promising approaches to reduce the range of ECS by combining observations and output from Earth System Models (ESMs). In this study, we systematically analyze 11 published emergent constraints on ECS that have mostly been derived from models participating in the Coupled Model Intercomparison Project Phase 5 (CMIP5) project. These emergent constraints are – except for one that is based on temperature variability – all directly or indirectly based on cloud processes, which are the major source of spread in ECS among current models. The focus of the study is on testing if these emergent constraints hold for ESMs participating in the new Phase 6 (CMIP6). Since none of the emergent constraints considered here have been derived using the CMIP6 ensemble, CMIP6 can be used for cross-checking of the emergent constraints on a new model ensemble. The application of the emergent constraints to CMIP6 data shows a decrease in skill and statistical significance of the emergent relationship for nearly all constraints, with this decrease being large in many cases. Consequently, the size of the constrained ECS ranges (66 % confidence intervals) widens by 51 % on average in CMIP6 compared to CMIP5. This is likely because of changes in the representation of cloud processes from CMIP5 to CMIP6, but may in some cases also be due to spurious statistical relationships or a too small number of models in the ensemble that the emergent constraint was originally derived from. The emergently- constrained best estimates of ECS also increased from CMIP5 to CMIP6 by 12 % on average. This can be at least partly explained by the increased number of high-ECS (above 4.5 K) models in CMIP6 without a corresponding change in the constraint predictors, suggesting the emergence of new feedback processes rather than changes in strength of those previously dominant. Our results support previous studies concluding that emergent constraints should be based on an independently verifiable physical mechanism, and that process-based emergent constraints on ECS should rather be thought of as constraints for the process or feedback they are actually targeting.

1 Introduction

A bulk measure of the sensitivity of the climate system to carbon dioxide in the atmosphere (CO₂) is commonly expressed by the equilibrium climate sensitivity (ECS), an idealized metric defined as the mean global surface air temperature change that results from a doubling of the atmospheric CO₂ concentration over pre-industrial levels once the climate system reached equilibrium. In 1979, the Charney report determined an ECS range of 1.5 to 4.5 K for the Earth system (Charney et al., 1979). This range had not changed substantially by the time of the Intergovernmental Panel on Climate Change (IPCC) Fifth Assessment Report (AR5) (Collins et al., 2013) and is close to the range of the Earth system models participating in the Coupled Model Intercomparison Project Phase 5 (CMIP5, Taylor et al., 2012).

This large range of model climate sensitivity values can be largely attributed to differences in cloud feedbacks (Boucher et al., 2013). In particular, model differences in the change in shortwave reflection of low-level clouds changes in response to climate change dominate the uncertainties in the global warming projections, particularly in the tropics but also at midlatitudes (Brient and Schneider, 2016; Vial et al., 2013). Over the years, various lines of evidence have been exploited to constrain the range of ECS, including paleoclimate data and analysis of the current observed warming trend (Knutti et al., 2017a). A new assessment using this evidence has narrowed the 66 % range (17 %–83 %) to 2.6–3.9 K (Sherwood et al., 2020), but in the meantime CMIP6 models are displaying a wider range of ECSs (see below).

Figure 1 Assessed ECS ranges (blue bars) from the Charney report (Charney et al., 1979) and the different Assessment Reports (ARs) of the Intergovernmental Panel on Climate Change (IPCC). The numbers correspond to individual CMIP5 and CMIP6 models; see Tables A1 and A2 for details. Adapted and updated from Meehl et al. (2020).

The use of emergent constraints is another promising approach to reduce the uncertainty in ECS (Eyring et al., 2019). Originally applied to the hydrological cycle and the snow–albedo feedback (Allen and Ingram, 2002; Hall and Qu, 2006), emergent constraints offer the possibility to constrain future projections of Earth system model (ESM) ensembles with observations. Their theoretical basis is an emergent relationship between an observable quantity in the past or present-day climate and a quantity related to the future climate (such as ECS). Typically, the observable quantity is related to a climate feedback, allowing the emergent relationship to be physically motivated by some key processes driving this feedback. Such a physical mechanism is a crucial prerequisite for the plausibility of an emergent constraint: due to large number of possible observables and small number of models, spurious emergent relationships are possible just by chance, which was shown by statistical tests (Caldwell et al., 2014). Caldwell et al. (2018) evaluated the credibility of several published emergent constraints on ECS. Using a feedback decomposition analysis, they assessed whether the published emergent relationship could be explained by the proposed mechanism. Out of 19 emergent constraints on ECS, only 4 of them were considered credible, while the rest of them were either considered not plausible or could not be tested using this approach. In addition, Caldwell et al. (2018) performed out-of-sample tests on five emergent constraints originally trained on older CMIP versions, by applying them to the CMIP5 ensemble. They found that out only one of the five passed this test.

Table 1 Overview of the 11 emergent constraints on the ECS used in this study. Observations marked with an asterisk (^*) are identical to the ones used in original publication.

Table 2 List of CMIP5 models alongside the index used in the figures of this study and a reference.

In this paper, we follow up on the work of Caldwell et al. (2018) by analyzing 11 published emergent constraints on ECS, which are summarized in Table 1, and assessing whether they still hold for the new CMIP6 model ensemble (Eyring et al., 2016). We first calculate these emergent constraints for the most recent ensemble used to derive all but one of them – CMIP5 (Taylor et al., 2012) – and then test whether they still hold in the CMIP6 ensemble. The one exception is the emergent constraint of Volodin (2008), which was derived on CMIP3 data. While the model range of ECS in CMIP5 is between 2.1 and 4.7 K, the CMIP6 model range is considerably larger, 1.8 to 5.6 K (Meehl et al., 2020); see Fig. 1. Possible reasons for this increased ECS range are changes in the extratropical cloud parameterizations and microphysics in the CMIP6 models (Zelinka et al., 2020). However, despite including more detailed cloud physical processes, further analyses suggest that the high-sensitivity models might overestimate the future warming trend (Tokarska et al., 2020). The large ECS range in CMIP6 emphasizes the need for reliable methods to constrain the uncertainty range of future climate projections with observations. The CMIP6 ensemble can be used for an independent testing of the constraints on previously unknown data. If the proposed underlying physical mechanisms are robust, i.e., targeting a key feedback mechanism controlling most of the observed CMIP6 spread, the emergent constraints would be expected to hold when applied to CMIP6 data. In this analysis we thus test the robustness of the constraints to new models and models with advances in model design over time.

Section 2 provides an overview of the data and methods used. Section 3 gives a discussion of the 11 emergent constraints on ECS and their results when derived from the CMIP5 or CMIP6 ensemble, including an analysis of their statistical significance. The paper ends with a discussion and summary in Sects. 4 and 5.

2 Data and methods

2.1 Equilibrium climate sensitivity (ECS)

In this study we use the output from climate models participating in CMIP5 and CMIP6, shown in Tables 2 and 3, respectively. Traditionally, ECS is defined as the global mean surface air temperature change after an instantaneous doubling of the atmospheric CO₂ concentration once the climate system reaches radiative equilibrium. Since running a fully coupled ESM into equilibrium is computationally expensive (this would require thousands of model years; see Rugenstein et al., 2020), ECS is typically approximated by a so-called “effective climate sensitivity”, which is derived from the first 150 years that follow an instantaneous quadrupling of the atmospheric CO₂ concentration (4×CO₂ run). Since the ESMs are not in radiative equilibrium during these 150 years, a regression of the top-of-atmosphere net downward radiation N versus the global mean surface air temperature change ΔT extrapolated to N=0 gives an estimate of the equilibrium warming (Gregory et al., 2004). In this paper, we use the term “ECS” to denote this effective climate sensitivity derived from the Gregory regression method. In this calculation, the linear fit of a corresponding pre-industrial control run is subtracted from the 4×CO₂ run to account for energy leakage and remove any model drift that is present in the control climate without adding noise (Andrews et al., 2012).

Table 3 As in Table 2 but for CMIP6 models included in this study.

Even though it is widely used in literature, this Gregory regression method is known to be only an approximation of the true climate sensitivity. As shown by Sherwood et al. (2020), the effective climate sensitivity is 6 % lower than the best estimate of the true equilibrium warming obtained from integrating the climate models until a new equilibrium is reached. This number is a model-based estimate of the compensation between changing feedbacks with time and the differences introduced by considering a 4×CO₂ run instead of a 2×CO₂ run. However, only a few ESMs provide simulations long enough to assess the true climate sensitivity. The CMIP endorsed LongRunMIP (Rugenstein et al., 2019) could be a promising way to estimate the true climate sensitivity that can then be used to reevaluate emergent constraints and their proposed underlying physical mechanisms.

2.2 Calculation of emergent constraints on ECS

An overview of the 11 emergent constraints analyzed in this study including the variables required for their calculations is given in Table 1 and the following section. We chose these particular emergent constraints since these were already implemented in the ESMValTool (see Sect. 2.4) at the time of writing this study, which greatly facilitated this analysis. For all emergent constraints, we use the historical simulations of CMIP5 and CMIP6 in order to ensure maximum agreement with the observational data. If necessary, the historical simulation of CMIP5 is extended after its final year 2005 with data from the RCP8.5 scenario (Riahi et al., 2011). Note that we only use data through 2014, during which time all RCP scenarios behave similarly and the choice of the scenario is not expected to affect results considerably. Such an extension is not needed for CMIP6 models as their historical simulations cover a longer time period until 2014.

To evaluate the resulting constrained probability distribution of ECS, we use the following nomenclature: let x_m be the x axis variable (i.e., the observable, constraining variable) of climate model m and y_m its corresponding target variable (ECS in our case). Following Cox et al. (2018), we use ordinary least-squares regression to fit the linear model

$$\begin{array}{}\text{(1)}& {\widehat{y}}_m\left(x_m\right)=a+b{x}_m,\end{array}$$

where ${\widehat{y}}_m$ is the predicted target variable for predictor x_m, a the intercept of the linear regression line and b the slope of the linear regression line. Fitting the regression model includes minimizing the standard error of the estimate

$$\begin{array}{}\text{(2)}& s^{\mathrm{2}}={\displaystyle \frac{\mathrm{1}}{M-\mathrm{2}}}\sum _{m=\mathrm{1}}^M{\left(y_m-{\widehat{y}}_m\right)}^{\mathrm{2}}.\end{array}$$

where M is the total number of climate models.

In the standard emergent constraint approach, the constrained target variable y (here ECS) is given by the regression $\widehat{y}\left(x\right)$ evaluated at an observed or observationally based (in case of using reanalysis data) value x that has not been used to fit the regression line. In that case, the corresponding uncertainty in the prediction of $\widehat{y}$ is given by the standard prediction error

$$\begin{array}{}\text{(3)}& {\mathit{\sigma }}_{\widehat{y}}^{\mathrm{2}}\left(x\right)=s^{\mathrm{2}}\left(\mathrm{1}+{\displaystyle \frac{\mathrm{1}}{M}}+{\displaystyle \frac{{\left(x-\stackrel{\mathrm{‾}}{x}\right)}^{\mathrm{2}}}{\sum _{m=\mathrm{1}}^M{\left(x_m-\stackrel{\mathrm{‾}}{x}\right)}^{\mathrm{2}}}}\right).\end{array}$$

Here, $\stackrel{\mathrm{‾}}{x}$ is the arithmetic mean of x over all models. Assuming Gaussian errors, this equation defines the conditional probability density function (PDF) for predicting a value of y given x, i.e., the posterior distribution:

$$\begin{array}{}\text{(4)}& P\left(y\mathrm{|}x\right)={\displaystyle \frac{\mathrm{1}}{\sqrt{\mathrm{2}\mathit{\pi }{\mathit{\sigma }}_{\widehat{y}}^{\mathrm{2}}\left(x\right)}}}\exp \left(-{\displaystyle \frac{{\left(y-\widehat{y}\left(x\right)\right)}^{\mathrm{2}}}{\mathrm{2}{\mathit{\sigma }}_{\widehat{y}}^{\mathrm{2}}\left(x\right)}}\right).\end{array}$$

This distribution can be interpreted as the posterior distribution in the regression model based on climate model output but constrained by matching the observable x. However, the observation of x (referred to as x₀) is not error free and has uncertainties associated with it. Assuming again an unbiased Gaussian, the resulting observational probability density for observing x₀ given the true value x is given by

$$\begin{array}{}\text{(5)}& P\left(x_{\mathrm{0}}\mathrm{|}x\right)={\displaystyle \frac{\mathrm{1}}{\sqrt{\mathrm{2}\mathit{\pi }{\mathit{\sigma }}_x^{\mathrm{2}}}}}\exp \left(-{\displaystyle \frac{{\left(x_{\mathrm{0}}-x\right)}^{\mathrm{2}}}{\mathrm{2}{\mathit{\sigma }}_x^{\mathrm{2}}}}\right),\end{array}$$

where ${\mathit{\sigma }}_x^{\mathrm{2}}$ is the variance of the observation about the true value. Assuming a uniform prior P(x)∝1 with no cut-offs (i.e., the cut-offs are positive and negative infinity, which forms an “improper” prior) and using Bayes' theorem implies P(x|x₀)=P(x₀|x). In a final step, numerical integration is used to calculate the marginal probability density for the constrained prediction of the target variable y dependent on the observation x₀:

$$\begin{array}{}\text{(6)}& P\left(y\mathrm{|}{x}_{\mathrm{0}}\right)=\underset{-\mathrm{\infty }}{\overset{+\mathrm{\infty }}{\int }}P\left(y\mathrm{|}x\right)P\left(x\mathrm{|}{x}_{\mathrm{0}}\right)\mathrm{d}x.\end{array}$$

By assuming a uniform prior in x, we also assume a uniform prior in y (the true ECS), since x and y are linearly related (see Eq. 1); in other words, an ECS near 8 K would be deemed just as probable as one near 4 K if both are equally consistent with the observational best estimate x₀. We do this for simplicity. The PDFs would shift somewhat lower with a broad prior on processes instead (see Sherwood et al., 2020), but we are concerned here with how outcomes compare using CMIP5 versus CMIP6 data, rather than the exact ranges obtained. Such comparisons are not sensitive to the prior.

As typically done in other studies proposing a single emergent constraint on ECS, we do not explicitly take model interdependency into account when applying the linear regression model (see Eq. 1) to the model ensemble data. We simply assume that the individual data points (i.e., climate models) are independent. As some modeling groups provide output from multiple ESMs and some ESMs from different modeling groups share components and code, this is clearly not the case. Duplicated code in multiple models is expected to lead to an overestimation of the sample size of a model ensemble and may result in spurious correlations (Sanderson et al., 2015). Possible approaches could be to stop treating all models equally by either applying a model weighting based on a model's interdependence with the other models or by simply reducing the ensemble size considering models only that are above a given (yet to be defined) interdependence score. Promising approaches to quantify the model interdependency that could be followed include, for example, the studies of Sanderson et al. (2015, 2017) and Knutti et al. (2017b).

Another limitation of our approach is the statistical model itself. Similar to many other emergent constraint studies, we use an ordinary least-squares linear regression model for each emergent constraint. However, in some cases this might not be appropriate, e.g., when we expect nonlinear behavior or when physical constraints can be used to derive further constraints for the regression model like a zero intercept (Annan et al., 2020; Jimenez-de-la-Cuesta and Mauritsen, 2019; Renoult et al., 2020).

We further note that observational uncertainties can also play a role, as using different observational datasets for a given variable as a proxy for observational uncertainty might lead to different emergent constraints. As this study uses only one combination of observational datasets to calculate the emergent constraints, as was the case in the original published emergent constraint studies, the error estimations given by our analysis are expected to underestimate the true error. This could be investigated by systematic tests using different observational datasets and/or combinations thereof as a proxy for observational uncertainty. Where available, additional observational uncertainty estimates could be used to give better estimates of the constrained range of ECS. A major challenge associated with this is, however, to determine how observational uncertainties propagate to the spatial scales and timescales represented by the models because of the not typically well-known correlation of observational errors in space and time (e.g., Bellprat et al., 2017).

2.3 Statistical significance of emergent relationships

To evaluate the statistical significance of the different emergent relationships, we use a two-sided t test to determine how likely the correlation found between the predictors and ECS would be to appear by chance. The null hypothesis for this test is that the predictor and ECS are not linearly correlated, i.e., the true underlying Pearson correlation coefficient of the population is zero. In that case, the variable

$$\begin{array}{}\text{(7)}& t={\displaystyle \frac{r\sqrt{M-\mathrm{2}}}{\sqrt{\mathrm{1}-r^{\mathrm{2}}}}},\end{array}$$

has a Student's t distribution with M−2 degrees of freedom; r is the Pearson correlation coefficient evaluated on the sample (x_m, y_m). In this study, we indicate the statistical significance with the p value, which describes the probability of obtaining an absolute sample Pearson correlation coefficient greater than |r| if the null hypothesis is true, i.e., the predictor and ECS are not linearly correlated. Smaller p values indicate higher significance. Although threshold values such as p<0.05 are often used to declare “significance,” here we focus mainly on how p values are affected by the change from CMIP5 to CMIP6, noting that they may be biased low anyway by the assumptions discussed in Sect. 2.2.

2.4 ESMValTool

All figures in this paper are produced with the Earth System Model Evaluation Tool (ESMValTool) version 2 (Eyring et al., 2020; Lauer et al., 2020; Righi et al., 2020). The ESMValTool is an open-source community diagnostics and performance metrics tool for the evaluation of Earth system models (https://www.esmvaltool.org/, last access: 18 December 2020). An ESMValTool recipe (configuration file defining input data, preprocessing steps and diagnostics to be applied) is available that can be used to reproduce all figures in this paper. This also allows redoing the analysis presented in this study once new model simulations from CMIP6 or other model ensembles become available.

3 Comparison of emergent constraints on ECS for CMIP5 and CMIP6

In this section we describe and discuss the 11 emergent constraints on ECS summarized in Table 1 using CMIP5 and CMIP6 data (Sect. 3.1 to 3.11) and provide a best estimate for ECS and statistical significance of the 11 emergent constraints in Sect. 3.12. While most of these emergent constraints have been derived using data from the CMIP5 and/or CMIP3 ensembles, to our knowledge none of them has been evaluated on the CMIP6 ensemble so far. The results for the individual emergent constraints described in the following are shown in Figs. 2 to 5. The left columns in these figures show the emergent relationships, including the uncertainty of the linear regressions (blue and orange shaded areas; see Eq. 3) and the uncertainty in the observations (gray shaded area; see Eq. 5). The right columns show the probability distributions of ECS in the original model ensemble (histogram) and the constrained distribution given by the emergent constraints (blue and orange line; see Eq., 6). Table 4 shows the corresponding 66 % confidence intervals (i.e., 17 %–83 % intervals) of ECS derived from the probability distributions given by Eq. (6) and the p values used to assess the significance of the emergent relationships.

Figure 2 Emergent constraints BRI, COX, and LIP applied to the CMIP5 ensemble (blue) and CMIP6 ensemble (orange). (a–c) Emergent relationships (solid blue and orange lines) for the CMIP models (numbers for individual models are specified in Tables A1 and A2). The shaded areas around the regression lines correspond to the standard prediction errors (Eq. 3), which defines the error in the regression model itself. The vertical dashed black line corresponds to the observational reference (see Table 1 for details on the individual observational datasets used) with its uncertainty range given as standard error (gray shaded area). The horizontal dashed lines show the best estimates of the constrained ECS for CMIP5 (blue) and CMIP6 (orange). The colored dots mark the CMIP5 (blue) and CMIP6 (orange) multi-model means. The p value in the legend corresponds to the hypothesis test introduced in Sect. 2.3 and describes the probability to obtain an absolute correlation coefficient |r| or higher under the null hypothesis that the true underlying correlation coefficient between the predictor and ECS is zero. (d–f) Probability densities for the constrained ECS following Eq. (6) (solid lines) and the unconstrained model ensembles (histograms). Note that for each individual emergent constraint, a different subset of climate models is used due to the availability of data (see Tables A1 and A2 for details). Thus, these histograms may differ for the different constraints.

Figure 3 As in Fig. 2 but for the constraints SHD, SHL, and SHS.

Figure 4 As in Fig. 2 but for the constraints SU, TIH, and TII.

Figure 5 As in Fig. 2 but for the constraints VOL and ZHA.

Table 4 Overview of the constrained ECS ranges and p values for all 11 analyzed emergent constraints. If not further specified, the uncertainty ranges correspond to the 66 % confidence intervals (17 %–83 %). For CMIP5 and CMIP6, these are evaluated from the probability distribution given by Eq. (6) (see also the right columns of Figs. 2 to 5). Note that even though CMIP5 models were used for some constraints in the original publications, the constrained ranges in column 2 and column 3 might differ due to the use of different models (in this paper, we use output from all CMIP models that is publicly available; see the data availability section for details). The p values describing the significance of the emergent relationships are defined as the probability to obtain an absolute correlation coefficient |r| or higher under the null hypothesis that the true underlying correlation coefficient between the predictor and ECS is zero. Smaller p values point to higher significance and vice versa (for details see Sect. 2.3).

3.1 Response of shortwave cloud reflectivity to changes in sea surface temperature (BRI)

In this emergent constraint proposed by Brient and Schneider (2016), ECS is correlated with the tropical low-level cloud (TLC) albedo, i.e., using the covariance of clouds with changes in sea surface temperatures (SSTs). Differences in the TLC albedo account for more than half of the variance of the ECS in the CMIP5 ensemble. Following Brient and Schneider (2016), TLC regions are defined as grid points that are in the driest quartile of 500 hPa relative humidity of all grid cells over the ocean between 30^∘ S and 30^∘ N. The albedo of the TLC is obtained by calculating the ratio of TOA shortwave cloud radiative forcing and solar insolation averaged over the TLC region. The regression coefficients of deseasonalized variations of TLC shortwave albedo and SST (in % K⁻¹) are then used as an emergent constraint for ECS. Here, we use observational data from HadISST for SST (Rayner et al., 2003), ERA-Interim for 500 hPa relative humidity (Dee et al., 2011) and CERES-EBAF (Loeb et al., 2018) for the TOA radiative fluxes over the time period 2001–2005. In the original publication, Brient and Schneider (2016) use similar observation-based datasets with the exception of SST, where they take data from the Extended Reconstructed Sea Surface Temperature (Smith and Reynolds, 2003) as reference instead. Our analysis yields a 66 % confidence range for ECS of 3.72 K ± 0.59 K for CMIP5 (R²=0.38) and 4.32 K ± 1.07 K for CMIP6, with much lower R²=0.12. The original publication stated a best estimate of 4.0 K, with a very low likelihood of values below 2.3 K (90 % confidence). The statistical significance of the emergent relationship dropped from p=0.0005 for CMIP5 to p=0.0355 for CMIP6.

3.2 Temperature variability (COX)

The emergent constraint on ECS proposed by Cox et al. (2018) uses a temperature variability metric Ψ that is based on the interannual variation of global mean temperature calculated from its variance (in time) and 1 year lag autocorrelation. In contrast to the majority of emergent constraints that focus on cloud-related processes, this constraint is based on the fluctuation–dissipation theorem, which relates the long-term response of the climate system to an external forcing (ECS) and short-term variations of the climate system (climate variability). This arguably places the constraint on a more solid theoretical foundation, although several questions were raised on the robustness of the results to choices made in the analysis (Brown et al., 2018; Po-Chedley et al., 2018; Rypdal et al., 2018). For example, Annan et al. (2020) showed that the assumed linear relationship between Ψ and ECS does not hold when adding a deep ocean to the model. As observational data, here we use the HadCRUT4 dataset (Morice et al., 2012) over the time period 1880–2014. Under the COX constraint we thus assess a 66 % ECS range of 3.03 K ± 0.73 K for CMIP5 (R²=0.31) and 3.71 K ± 1.09 K for CMIP6 (R²=0.01). Cox et al. (2018) derived a 66 % range of 2.8 K ± 0.6 K from a different subset of CMIP5 models but the same observations. When moving from CMIP5 to CMIP6, the significance of the emergent relation drops massively from p=0.0032 to p=0.5415, respectively.

3.3 Southern Hemisphere Hadley cell extent (LIP)

The results of Lipat et al. (2017) show that the multi-year average extent of the Hadley cell correlates with ECS in CMIP5 models. The Hadley cell edge is defined as the latitude of the first two grid cells from the Equator going south where the zonal average 500 hPa mass stream function calculated from December–January–February means of the meridional wind field changes sign from negative to positive. Lipat et al. (2017) explain this correlation by tying it to the observed correlation of the interannual variability in midlatitude clouds and their radiative effects with the poleward extent of the Hadley cell. For the calculation of the emergent constraint, we use reanalysis data from ERA-Interim (Dee et al., 2011) for the meridional wind speed over the time period 1980–2005. Our application of this emergent constraint gives ECS 66 % ranges of 2.97 K ± 0.75 K for CMIP5 (R²=0.18) and 3.75 K ± 1.11 K for CMIP6 (R²<0.01). The original publication does not specify an ECS range. For CMIP6, the emergent constraint shows a much lower statistical significance (p=0.6791) than for CMIP5 (p=0.0228).

3.4 Large-scale lower-tropospheric mixing (SHD)

Sherwood et al. (2014) proposed that the degree of mixing in the lower troposphere determines the response of boundary-layer clouds and humidity to climate warming, as the associated moisture transport would increase rapidly in a warmer atmosphere due to the Clausius–Clapeyron relationship. The large-scale component D of this mixing is defined as the ratio of shallow to deep overturning. D is calculated from the vertical velocities averaged over two height regions: 850 and 700 hPa for shallow overturning and 600, 500, and 400 hPa for deep overturning. Both quantities are averaged over parts of the tropical ocean region away from the regions of highest SST and strongest mid-level ascent, specifically the region 30^∘ S–30^∘ N, 160^∘ W–30^∘ E, wherever air is ascending at low levels. As observationally based data, we use vertical velocities from ERA-Interim (Dee et al., 2011) over the time period 1989–1998 similar to the original publication. We derive ECS 66 % confidence ranges of 3.65 K ± 0.64 K for CMIP5 (R²=0.28) and 3.77 K ± 1.06 K for CMIP6 (R²=0.03). Sherwood et al. (2014) do not give a best estimate for ECS based on the large-scale component of mixing D or its small-scale counterpart S (Sect. 3.5) but instead for the sum of D+S only (see Sect. 3.6). The regression shows a much lower significance for CMIP6 (p=0.2805) than for CMIP5 (p=0.0037).

3.5 Small-scale lower-tropospheric mixing (SHS)

The small-scale mixing S (Sherwood et al., 2014) is calculated from the differences in relative humidity and temperature between 700 and 850 hPa. The differences are averaged over all grid cells within the upper quartile of the annual mean 500 hPa ascent rate (within ascending regions) in the tropics. The tropics are defined as region between 30^∘ S and 30^∘ N. In the Cloud Feedback Model Intercomparison Project models (CFMIP, Webb et al., 2017), for which convective tendencies were available, upward moisture transport by parameterized convection was shown to increase more rapidly with warming for higher values of S. We use reanalysis data from ERA-Interim (Dee et al., 2011) for temperature and relative humidity to calculate the observationally based constraint (1989–1998). Our analysis shows a 66 % range of ECS of 3.07 K ± 0.73 K for CMIP5 (R²=0.13) and 3.48 K ± 1.07 K for CMIP6 (R²=0.12). The correlation of S and ECS shows a slightly higher significance in the CMIP6 ensemble (p=0.0396) than in the CMIP5 ensemble (p=0.0647). The SHS constraint is one of the two analyzed emergent constraints (ZHA being the other exception) that shows a higher statistical significance for the CMIP6 than for the CMIP5 ensemble.

3.6 Lower tropospheric mixing index (SHL)

The lower tropospheric mixing index (LTMI) formulated by Sherwood et al. (2014) is defined as the sum of the small-scale mixing S (see Sect. 3.5) and the large-scale mixing D (see Sect.  3.4), which are supposed to capture complementary components of the total mixing phenomenon. Sherwood et al. (2014) argue that the increase in dehydration depends on initial mixing linking it to cloud feedbacks and thus also to ECS. For this constraint, we derive an ECS 66 % confidence range of 3.42 K ± 0.65 K for CMIP5 (R²=0.41) and 3.67 K ± 1.06 K for CMIP6 (R²=0.16). Sherwood et al. (2014) give a best estimate of about 4 K with a lower limit of 3 K. Similar to both other constraints by Sherwood et al. (2014), SHD and SHS, the statistical significance of the SHL emergent relation decreased in CMIP6 (p=0.0138) compared to CMIP5 (p=0.0002).

3.7 Error in vertical profile of relative humidity (SU)

Another emergent constraint on ECS that targets uncertainties in cloud feedbacks was proposed by Su et al. (2014). They show that changes in the Hadley circulation are physically connected to changes in tropical clouds and thus ECS. Consequently, the inter-model spread in the change of the Hadley circulation in an ensemble of climate models is well correlated with the corresponding changes in the TOA cloud radiative effect. Moreover, Su et al. (2014) found a correlation between a model's ECS and its ability to represent the present-day Hadley circulation. The latter is calculated from the tropical (45^∘ S–40^∘ N) zonal-mean vertical profiles of relative humidity from the surface to 100 hPa. These profiles are then used to define the x axis of the SU constraint by calculating a performance metric based on the slope of the linear regression between a climate model's relative humidity profile and the corresponding observational reference. Similarly to the original publication, we use humidity observations from AIRS (Aumann et al., 2003) for pressure levels greater than 300 hPa and MLS-Aura data (Beer, 2006) for pressure levels of less than 300 hPa. Our analysis yields a constrained 66 % range of ECS of 3.30 K ± 0.88 K for CMIP5 (R²=0.08) and 3.77 K ± 1.35 K for CMIP6 (R²=0.05). The original publication gives a best estimate of 4 K with a lower limit of 3 K. Figure 4 shows that in addition to the low R² values, the emergent relationship shows different slopes for CMIP5 and CMIP6. For the CMIP5, the expected positive correlation is found, while for CMIP6, a negative correlation is found. This suggests that the constraint is not working (any more) when applied to the CMIP6 data. Consequently, the SU constraint shows a weaker statistical significance in the CMIP6 ensemble (p=0.1935) than for the CMIP5 ensemble (p=0.1676). The SU constraint is related to an emergent constraint on ECS proposed by Fasullo and Trenberth (2012), who correlated May–August zonal-mean relative humidity against ECS. In contrast to Su et al. (2014), they did not use the entire tropics, but identified two distinct regions with largest correlation.

3.8 Tropical mid-tropospheric humidity asymmetry index (TIH)

Tian (2015) found a link between mid-tropospheric humidity over the tropical Pacific and simulated moisture, precipitation, clouds, large-scale circulation, and thus ECS in CMIP3 and CMIP5 models. The study explains this link with the similarity of mid-tropospheric humidity and precipitation patterns as both are related to the ITCZ. The proposed tropical mid-tropospheric humidity asymmetry index to constrain ECS is defined as relative bias (in %) in simulated annual mean 500 hPa specific humidity averaged over the Southern Hemisphere (SH) tropical Pacific (30^∘ S–0^∘ N, 120^∘ E–80^∘ W) minus the bias averaged over the Northern Hemisphere (NH) tropical Pacific (20^∘–0^∘ N, 120^∘ E–80^∘ W) when compared with observations. Similar to the SU constraint, the index proposed by Tian (2015) seems to be related to the emergent constraint by Fasullo and Trenberth (2012), who found correlations between relative humidity of the middle and upper troposphere and ECS. Here, we use humidity observations from AIRS (Aumann et al., 2003) over the time period 2003–2005 as the reference dataset. We assess a 66 % ECS range of 3.88 K ± 0.75 K for CMIP5 (R²=0.24) and 4.15 K ± 1.10 K for CMIP6 (R²=0.06). Tian (2015) specifies a best estimate of 4.0 K. The significance of the emergent relationship dropped massively from p=0.0089 in CMIP5 to p=0.1348 in CMIP6.

3.9 Southern ITCZ index (TII)

In addition to the humidity index, Tian (2015) proposed an emergent constraint on ECS based on the southern ITCZ index (Bellucci et al., 2010; Hirota et al., 2011). This index is defined as the climatological annual mean precipitation bias averaged over the southeastern Pacific (30^∘ S–0^∘ N, 150–100^∘ W). The southern ITCZ index is calculated in mm d⁻¹ and dominated by the so-called double ITCZ, a common problem in many CMIP5 climate models. Tian (2015) found a link between double-ITCZ bias and simulated moisture, precipitation, clouds, and large-scale circulation in CMIP3 and CMIP5 models. He argues that this could explain the link found between the double-ITCZ bias and ECS. As reference data, we use observed precipitation data for the years 1986–2005 from GPCP (Adler et al., 2003). We calculate an ECS 66 % confidence range of 3.87 K ± 0.67 K for CMIP5 (R²=0.33) and 3.84 K ± 1.09 K for CMIP6 (R²<0.01). Tian (2015) specifies a best estimate of 4.0 K. The emergent relationship shows a much lower statistical significance in CMIP6 (p=0.8236) than in CMIP5 (p=0.0013).

3.10 Difference between tropical and midlatitude cloud fraction (VOL)

The study by Volodin (2008) aims at constraining ECS based on the geographical distribution of clouds in climate models. Since this early emergent constraint was originally trained on CMIP3 models, both CMIP5 and CMIP6 are out-of-sample tests for it. Volodin (2008) shows that high ECS models tend to simulate a higher total cloud cover over the southern midlatitudes and a lower total cloud cover over the tropics (relative to the multi-model mean). This can be used to establish an emergent relationship between the ECS and the difference in tropical total cloud cover (28^∘ S–28^∘ N) and the southern midlatitude total cloud cover (56–36^∘ S). Analogous to the original study, we use the ISCCP-D2 data (Rossow and Schiffer, 1991) as observational reference. For the VOL constraint, we calculate a constrained 66 % range of ECS of 3.74 K ± 0.64 K for CMIP5 (R²=0.38) and 4.21 K ± 1.04 K for CMIP6 (R²=0.18), whereas the original publication gives a range of 3.6 K ± 0.4 K (standard deviation) for a climate model ensemble of CMIP3 models. The emergent constraint by Volodin (2008) shows a lower significance in the CMIP6 ensemble (p=0.0056) than in the CMIP5 ensemble (p=0.0004).

Figure 6 Emergent relationship ZHA (Zhai et al., 2015) for different subsets of CMIP5 models. Blue circles show the 15 CMIP5 models used in the original publication (except for CESM1-CAM5): the solid blue line and blue shaded area show the emergent relationships evaluated on these models including the uncertainty range. In our study, we added 11 more CMIP5 models (red circles). The corresponding emergent relationship that considers all available CMIP5 models is shown in red colors. This relationship shows a considerably lower coefficient of determination (R²) and higher p value than the relationship using the original subset of CMIP5 models. The vertical dashed line and shaded area correspond to the observational reference, and the horizontal dashed lines show the corresponding ECS constraints using this observation.

3.11 Response of seasonal marine boundary layer cloud fraction to SST changes (ZHA)

Zhai et al. (2015) focus on the variations of marine boundary layer clouds (MBLCs), which largely contribute to the shortwave cloud feedback and thus to the uncertainty in modeled ECS. Their central quantity is the response of the MBLC fraction to changes in the sea surface temperature (SST) in subtropical oceanic subsidence regions for both hemispheres (20–40^∘). On short (seasonal) and long (centennial under a forcing) timescales, this quantity is well correlated with ECS among an ensemble of CMIP3 and CMIP5 models. Together with observations of cloud fraction from CloudSat/CALIPSO (Mace et al., 2009), SST from AMSRE SST (AMSR-E, 2011), and vertical velocity from ERA-Interim (Dee et al., 2011), the seasonal response of MBLC fraction to changes in SST forms an emergent constraint on ECS. We assess a 66 % ECS range of 3.35 K ± 0.74 K for CMIP5 (R²=0.05) and 3.79 K ± 0.67 K for CMIP6 (R²=0.62). In their original publication, Zhai et al. (2015) found an ECS range of 3.90 K ± 0.45 K (standard deviation) for a combination of CMIP3 and CMIP5 models. In terms of statistical significance, the results of the ZHA constraints are somewhat surprising: although CMIP5 data (in combination with CMIP3 data) were successfully used in their original publication, our approach finds that the statistical significance of the emergent relationship is much higher in the unseen CMIP6 ensemble (p<0.0001) than in the previously available CMIP5 ensemble (p=0.2567). The ZHA constraint is the only emergent constraint analyzed here that shows this extreme behavior (only one other constraint, SHS, shows a slightly higher significance in CMIP6; all other constraints show lower significances in CMIP6). The reason for the erratic skill in CMIP5 is the set of climate models used. For our analysis, we use 11 additional CMIP5 models that were not used in the original publication (i.e., ACCESS1-0, ACCESS1-3, bcc-csm1-1, bcc-csm1-1-m, CCSM4, GFDL-ESM2G, GFDL-ESM2M, IPSL-CM5A-MR, IPSL-CM5B-LR, MPI-ESM-MR and MPI-ESM-P). Due to a lack of publicly available data, the model CESM1-CAM5 that is used in the original publication is not included in our analysis. The effect of choosing different subsets of CMIP5 models on the emergent relationship is illustrated in Fig. 6. Using the original CMIP5 models from the original publication gives a considerably higher correlation (R²=0.38) than using all available CMIP5 models (R²=0.05). This result shows a strong dependency of this emergent constraint on the subset of climate models used. Nonetheless, the performance on CMIP6 models is, surprisingly, the best of all the constraints, and much better than on either subset of CMIP5 models.

3.12 Constrained ECS ranges and statistical significance of the 11 emergent constraints

In most cases, the emergent relationships (left columns of Figs. 2 to 5) show the same sign of the slope (as expected from the theory) for CMIP5 and CMIP6, with the SU constraint being the only exception. However, the coefficient of determination (R²) is lower for CMIP6 compared to CMIP5 for all one constraint: ZHA. The probability distributions of the constrained ECS that we obtain (right columns of Figs. 2 to 5) give similar results: except for the ZHA constraint, the constraint on the CMIP6 ensemble is weaker, i.e., the constrained PDFs derived from the CMIP6 ensemble are broader than their respective CMIP5 counterparts. As shown in Table 4, for CMIP5, the range of the best (maximum likelihood) estimates for ECS is 2.97 to 3.88 K, while the corresponding CMIP6 best estimates are higher for almost every tested emergent constraint (TII being the only exception), resulting in a range of best estimates of 3.48 to 4.32 K. Using the arithmetic mean of all analyzed emergent constraints, this results in a mean increase of the ECS best estimate of 12 % in CMIP6 compared to CMIP5. Similarly, the size of the 66 % ECS ranges (17 %–83 % confidence) shows values of 1.16 to 1.75 K in CMIP5 and 1.32 to 2.70 K in CMIP6, resulting in an increase of 51 % averaged over all emergent constraints. In summary, the R² of the emergent relationships and the constrained range of ECS each depend strongly on the climate model ensemble used, even though a physical explanation is given for each emergent constraint that is thought to be valid for every climate model ensemble. The same behavior is found for the statistical significance of the emergent relationships using the null hypothesis that there is no correlation between the predictor and ECS (see Sect. 2.3). Except for the ZHA and SHS constraints, every emergent relationship investigated shows a lower statistical significance (i.e., higher p value) in the CMIP6 ensemble than in the CMIP5 ensemble. If a conventional significance test of p<0.05 was applied, 8 of the 11 constraints would pass this test on CMIP5 model data but only 5 (BRI, SHL, SHS, VOL, and ZHA) would pass this test on CMIP6. This is still much better than would be expected purely by chance. Hence, there is still skill in at least a few of the constraints, but it is much lower than was suggested in nearly all of the initial studies.

4 Discussion

As shown in the previous sections, most emergent relationships show smaller coefficients of determination when evaluated on the new CMIP6 ensemble compared to the CMIP5 ensemble. In this section, we discuss possible reasons for, and implications of, these differences. As reported by Caldwell et al. (2014), the large amount of data provided by modern ESMs can generate spurious correlations of variables between past climate and ECS just by chance, especially when only a small number of climate models is considered. This would cause the performance of the emergent constraint to be reduced on out-of-sample data (like the new CMIP6 ensemble), since the emergent relationship appeared just by chance and not because of a physically based mechanism.

A further reason for the weaker emergent relationships in CMIP6 may be the increased complexity of the participating ESMs. Each emergent constraint approach is based on the assumption that a single observable process or physical aspect in the current climate dominates the uncertainty in ECS. Some emergent constraints such as ZHA and BRI relate changes in cloud properties (ZHA: low-level cloud fraction; BRI: cloud reflectivity) on seasonal or interannual timescales driven by changes in SST to ECS. This means that it has to be implicitly assumed that the observable changes in these properties on seasonal or interannual timescales are related to those occurring as a result of climate forcing in a way successfully captured by the ESMs. While this assumption seems to make sense, we do not know whether the ESMs cover all relevant processes of the real Earth system. For example, it may be possible that there exist processes that are unimportant in the ESMs (and hence are not captured by the emergent constraints) but are actually important in reality. This lack of relevant processes may lead to an overconfident constraint. Thus, the more complex ESMs of the CMIP6 ensemble are more likely to capture relevant processes of the true climate, which leads to weaker emergent relationships. On the other hand, emergent constraints on the less complex CMIP3 and CMIP5 ensemble may be overconfident.

For CMIP6 models, Zelinka et al. (2020) showed that cloud feedbacks and thus ECS in high-sensitivity models are to some extent associated with changes in clouds over the Southern Ocean, while in CMIP3 and CMIP5 the uncertainty in cloud feedbacks is dominated by clouds in the subtropical subsidence regions. One might speculate that a possible reason for this might be an improved simulation of clouds over the Southern Ocean in some models (Bodas-Salcedo et al., 2019; Gettelman et al., 2019a), as shown for some pre-CMIP6 model versions evaluated by Lauer et al. (2018). The findings of Zelinka et al. (2020) can also at least partly explain the larger inter-model spread in climate sensitivity due to a greater diversity of cloud feedbacks, which also results in a weaker emergent constraint compared with CMIP5 models, as most of them constrained low-cloud feedbacks. They found that on average, the shortwave low cloud feedback is larger in CMIP6 than in CMIP5, which they primarily relate to changes in the representation of clouds. As a possible explanation, Zelinka et al. (2020) give an increase in mean-state supercooled liquid water (i.e., increase in the cloud water liquid fraction) in mixed-phase clouds resulting in less pronounced increases in low-level cloud cover and water content with warmer SSTs particularly in midlatitudes.

Our findings suggest that the process-oriented emergent constraints (i.e., all of the emergent constraints investigated here except COX) are only successful in constraining ECS as long as the uncertainty in ECS is dominated by the same process or feedback. In the CMIP5 ensemble, cloud feedback is the main contributor to the spread in ECS with low-level clouds in tropical subsidence regions dominating the spread in cloud feedback (e.g., Ceppi et al., 2017). If any other process or feedback is biased (or missing) in the ensemble as a whole, then these process-oriented emergent constraints will be biased in their estimates of ECS. The appearance of diverse new feedback processes in CMIP6 could explain the reduced skill when applied to CMIP6 data, and a tendency for these to be positive would explain the upward shift in the model ECS distribution that is not captured by the CMIP5-trained constraints. Process-oriented emergent constraints are therefore perhaps best thought of as constraints on the processes that they target, rather than constraints on ECS.

Emergent constraints that use global temperature change as a way of constraining ECS could in principle overcome this problem. If one feedback is biased in an ensemble the constraint might still work as both global temperature change and ECS might similarly reflect the sum of all feedbacks. Emergent constraints of this kind include the tropical temperature during the Last Glacial Maximum (Hargreaves et al., 2012), tropical temperature anomalies during the mid-Pliocene Warm Period (Hargreaves and Annan, 2016), and post-1970s warming (Jimenez-de-la-Cuesta and Mauritsen, 2019). This seems to be supported by the findings of Tokarska et al. (2020), who tested an emergent constraint for the transient climate response based on recent global warming trends on the CMIP5 and CMIP6 ensembles with similar results for both model ensembles. Nevertheless, these temperature-based estimates are sensitive to assumptions about forcings and unforced decadal temperature variations, which could also be incorrect, as could model-predicted relationships between feedbacks on short and long timescales that are implicit in most such measures.

5 Summary

This paper assesses 11 different emergent constraints on ECS, of which most are directly or indirectly related to cloud feedbacks, by applying them to results from ESMs contributing to CMIP5 and CMIP6. Of particular interest are the results from CMIP6, since all analyzed emergent constraints were published prior to the availability of CMIP6 data. In summary, the best estimate of ECS averaged over all emergent constraints increases by 12 % when moving from relationships trained on CMIP5 to those trained on CMIP6. Some increase is predicted by every constraint we analyzed and can be at least partly explained by the increased multi-model mean ECS of CMIP6, which was not accompanied by systematic changes in the constraint variables that could explain this increase, leading to regression fits with higher intercept values at observed constraint values. This is also illustrated by the CMIP5 and CMIP6 multi-model means in the left columns of Figs. 2 to 5 (colored dots), in which the connecting line between the CMIP5 and CMIP6 multi-model mean is not parallel to the CMIP5 emergent relationships for all emergent constraints.

Our results also show that, except for ZHA, all emergent relationships are weaker (in terms of the coefficient of determination R²) in CMIP6 compared to CMIP5, which means that the corresponding emergent relationships are able to explain less of the ECS variation simulated by the newer CMIP6 models than by those of CMIP5. This is also demonstrated by the statistical significance, which is lower in CMIP6 than in CMIP5 for all emergent constraint except for ZHA and SHS. As described in Sect. 2.3, our test for statistical significance uses the null hypothesis that there is no correlation between the predictor variable of the emergent constraint and ECS. Further evidence of the decreased performance of the emergent constraints in CMIP6 is given by the size of the constrained ECS ranges, which widens by 51 % in CMIP6 compared to CMIP5 on average. Moreover, our more detailed analysis of the ZHA constraint (see Fig. 6) showed that this emergent constraint is very sensitive to outliers and the subset of the climate model ensemble used to fit the emergent relationship. Such a behavior might not be unique to the ZHA constraint but could apply to other emergent constraints as well. This in turn suggests that the number of climate models commonly used for emergent constraints might be too low, leading to non-robust relationships.

Our analysis makes a number of simplifying assumptions common to other studies, such as model independence, discussed in Sect. 2.1 and 2.2. These assumptions affect the significance of emergent relationships and the PDFs of ECS based on a constraint. However, they do not affect our main conclusions here, which concern the change in performance on CMIP6 relative to CMIP5 and the implications for robustness and future use of emergent constraints.

ECS is the product of the complex interactions of the many components and feedbacks. Thus, constraining ECS with a single physical process might overly simplify this problem. Such single process-oriented emergent constraints therefore do not seem to be helpful in constraining ECS but should probably rather be thought of as constraints for the process or feedback they are actually targeting (if that can be clearly identified). With increasing computational resources available to climate science, more and more detailed process interactions can be taken into account in a modern ESM. In contrast, the predecessor versions CMIP3 and CMIP5 were less complex with simpler atmospheric process representations, so constraining uncertainties of a single dominant process may have allowed for an apparently more successful constraint of ECS than would be achieved in more complex models. As a conclusion, we argue that to constrain ECS in a more robust way, it might be beneficial to apply multivariate approaches that are able to consider multiple (different) relevant physical processes and feedbacks at once and thus are able to get a broader picture of the complex reality. A possible approach for this is given by Bretherton and Caldwell (2020), who combine the information from multiple emergent constraints on ECS using a multivariate Gaussian and multiple linear regression. For the CMIP3 and CMIP5 ensembles, they find an increased best estimate relative to the unweighted ensemble mean similar to the participating individual emergent constraints, but with lower uncertainty range. Moreover, new machine learning techniques are a promising avenue forward for such multivariate approaches and for constraining uncertainties in multi-model projections (Schlund et al., 2020) with the aim of further improving climate modeling and analysis (Reichstein et al., 2019).

Appendix A

Table A1 All participating CMIP5 models including their ECS values (in K) and x axis values for the different emergent constraints. Details on all constraints (including the units) are given in Table 1. The leftmost column corresponds to the index used in all plots.

Table A2 As in Table A1 but for the CMIP6 models.