For many decades, climate models have been an important tool for gaining a better understanding of the climate system, and for making inferences about future climate and the consequences of climate change. Over the years many different models have become available which is generally beneficial to get a more accurate picture of reality. There are, for instance, different possibilities on how to parameterize processes that cannot be resolved in the models, which leads to uncertainties in the simulated data that can be represented by using an ensemble of models. This type of uncertainty is referred to as model- or structural uncertainty, and it is the primary uncertainty addressed by a multi-model ensemble. In the most straightforward way, each model in an ensemble is considered equally important and thus receives equal weight. This is therefore known as the “model democracy” approach, where simple multi-model means (MMMs) are computed and the ensemble spread is used to quantify uncertainty. While this has long been the standard approach, it comes along with several problems that can be addressed by weighting the models in the ensemble.

A general challenge is that we must work with “ensembles of opportunity”: the available models do not represent a statistically perfect ensemble in the sense of a set of unbiased and independent random samples e.g., see. They were not constructed to cover the entire range of plausible outcomes, nor is this space evenly sampled as the models can be dependent on one another in complex and often intransparent ways. Due to such inter-model dependencies, they may, for instance, have shared biases, which essentially means that some models in the ensemble do not provide as much new information as others e.g., see. Furthermore, the models may differ in how plausible they are in light of the observational data. Therefore, there are two main aspects that weights assigned to models in an ensemble should account for: model performance and model dependence.

Various methods have been proposed in the literature to compute model weights that reflect one or both aspects. A method that has received much attention is ClimWIP from , which was developed from a long line of previous work e.g.,. It combines weights that are calculated separately for independence, defined in terms of model output similarities, and performance. Both types of weights are computed based on the assumption of a Gaussian error model and by taking one or more diagnostic variables into account. The diagnostic variables used for computing performance weights may be different from those used to calculate independence weights. Meanwhile, the Reliability Ensemble Averaging method (REA) is a similar method introduced by which also considers model performance and independence. Another, conceptually different approach is what is often referred to as Bayesian Model Averaging (BMA). While the other methods mentioned above compute model weights based on the performance and/or independence of the individual models in the ensemble, BMA-weights are based on the performance of the weighted ensemble itself. Therefore, the BMA-weights are selected so that the weighted average model simulates the observed data as closely as possible e,g.. It should be noted that this definition of BMA differs from what BMA refers to in the context of statistical models where it was first applied, but since the application of BMA on forecasting surface temperatures by , it has become a common way to refer to the method used in at least some climate science studies today.

Although the methods differ in the details, they can mainly be distinguished by whether the ensemble is considered as a whole, which we will refer to as “Ensemble Performance Weighting”, or as individual members, which we will refer to as “Individual Performance Weighting”. All approaches to computing model weights have in common that they need to provide answers to the following two questions: (i) how to compute model performance and (ii) how to account for potential dependencies between models. Concerning model performance, , for instance, use a method, embedded in information-theoretic concepts, that computes performance weights relative to the inverse of the deviations between model output and data. Similarly, use inverse-distance power weighting (1AEir, r>0, AE_i: absolute error of model i) to compute performance weights. Others define performance based on the assumption of independent and identically distributed Gaussian errors e.g. . Note that all such individual-performance weighting approaches where the weights are a function of the sum of squared errors yield the same ranking order, only the magnitude of the weights differs (see Appendix  for examples).

Meanwhile, independence is often defined in terms of model similarities e.g. see. The more similar the output of two models is, the more dependent they are considered to be. This is reasonable in that two dependent models will have shared biases which is reflected in similar model output for both. Another option is to use meta information about the climate models that could provide further insights about co-dependencies between the models, e.g., information about the modeling group, specific model components or model resolution e.g. see. However, contrary to the model output, this information is not always available, making model similarities a pragmatic choice to define model independence. Furthermore, even if meta information about the models were available, there would not be a unique straightforward definition of independence. As pointed out by , considering full distributions over weight vectors rather than a single weight per model implicitly provides information about inter-model dependencies. The idea is that this is due to the correlation between the posterior weights of different models. This is an important claim that is fundamental to the distinction between “individual” performance weighting and “ensemble” performance weighting.

Here we aim to clearly define these two approaches within the common language of probabilities in Bayesian terms, so that we can easily compare and apply both methods in a consistent way. We use simple, idealized examples to test the methods in a systematic way, with a particular focus on the question of how inter-model dependencies are accounted for in a model weighting exercise. We also explore how the results obtained from applying this approach are affected by different methodological choices about the prior distribution over weights, which can be important and should be a prominent aspect of the ensemble performance weighting approach.

The paper is structured as follows. In Sect.  we lay out the differences between the two approaches and define the terminology we use. In Sects.  and , we discuss choices for the prior and likelihood distributions and consider the influence of using a set of different diagnostic variables, including a simple example. In Sect.  we consider the issue of model dependence through simple examples, while in Sect. , we consider the particular realistic case of using Equilibrium Climate Sensitivity (ECS) as a performance diagnostic. In Sect. , we discuss the overall implications of our work. Section  concludes the paper and provides an outlook on future work.

In the following we will provide definitions for the two general approaches for computing model weights, the individual and the performance weighting approach, and specify the terminology that we use throughout the paper. This is summarized in Box . Note that generally, lowercase letters refer to model output/observations for a specific climate variable, while uppercase letters refer to collections, e.g., of different variables or models. An exception are uppercase calligraphic letters, Mi and Y, which we use to refer to a model i or an observational dataset as a category.

In the individual weighting approach the prior is defined over models (as categories), while in the ensemble weighting approach the prior is defined over weight vectors. In the absence of any available information about the models before considering the data, like information about model dependencies, for example, the most neutral choice for the prior is a distribution that treats each model, or in the ensemble weighting approach, each weight vector, as equally likely. In the individual performance weighting approach, this corresponds to an equal probability for each model, or pi=1N where N is the number models. Mathematically this is described by a categorical distribution: 6Mi∼Categoricalp=1N,…,1N. If available, prior knowledge can be taken into account by adjusting the parameter vector p accordingly, subject to the constraint ∑i=1Npi=1. Analogously, in the ensemble performance weighting approach, we draw weight vectors from a Dirichlet distribution with parameter vector α where we set each entry αi to the same constant value c: 7w∼Dirichlet(α=[c,…,c])withc>0. The Dirichlet distribution is defined over N-dimensional vectors that sum up to 1. When all αi are identical, the vector α reduces to a single scalar α, also called the concentration parameter. Figure a shows the distribution of each component wi of weight vectors drawn from a Dirichlet distribution over 3-dimensional vectors (corresponding to an ensemble of three models) and how it changes with varying α. The expected value for wi, i.e. for model i, is given by αiα0 where α0 = ∑i=1Nαi. Therefore, when all parameters αi are set to the same value c, the expected value for each wi is always equal to 1N since E[wi]=αiα0=cN⋅c=1N. This is the case in the two bottom rows of Fig. a, where α=1 and α=1N respectively. It is, therefore, a reasonable choice to set all αi to a single value when we do not have any prior knowledge about the models' weights. Yet, the sampled vectors differ depending on the absolute value of α: when α=1, the prior mass is evenly spread and all wi are close to the expected value 1N whereas when α<1, the weights tend to be either large or small (U-shaped distribution). In the asymptotic case where α→0, the weight vectors approach one-hot vectors, i.e. nearly all probability mass is assigned to a single model, while all others receive almost no weight. The upper three rows of Fig. a further show how the distribution of each wi changes when an exemplary basis parameter vector of [0.1, 0.4, 0.5] is multiplied by 0.1, 10 and 100 respectively. The larger the parameter values αi are, the tighter the distributions scatter around the respective mean value. Therefore, the Dirichlet distribution allows us to easily incorporate potential prior knowledge about the value of a model's weight: the more certain we are a priori, the larger we set the value of the corresponding parameter αi. There are alternative distributions for vectors that sum up to 1 such as the Logit Normal distribution. These were, however, not explored in this work.

Especially in cases where we only have few or not very informative data, the choice of the prior needs careful consideration as, in that case, the posterior will be more influenced by the prior. Figure b shows how the choice of the prior distribution over the weight vectors influences the data that is considered in the first place by the weighted average models. Here, we generated arbitrary data for 38 models (grey stars) by drawing from a Normal Distribution with μ=3, σ=1 (black curve) to represent an ensemble of opportunity. The three colored densities show the distribution of the weighted averages under Dirichlet priors (over w) with varying concentration parameter α. In all three cases, the expected value for each wi equals 1N, reflecting that we do not have any prior knowledge about the models' weights. When α=1 (green density), all weight vectors that we consider through the prior distribution assign approximately equal probability to all wi, so that we are naturally restricted to weighted averages that sit tightly around the multi-model mean. When α is extremely small purple density where α=1500, the considered “weighted” averages effectively correspond to the output of individual models as the prior distribution generates vectors that approach one-hot vectors (where αj=1, αi=0∀i≠j). Setting α=1N (orange density) turns out to be a reasonable choice in that the weight vectors drawn from this prior distribution allow us to cover a large part of the spread of the model ensemble. This is due to the variance of the Dirichlet distribution, with scalar concentration parameter α, defined as var(wi)=λ⋅1N⋅α+1 where λ=N-1N2, i.e. λ does not depend on α. The variance of the prior weights will therefore be very small if α does not depend on the number of models (assuming a moderate number of models, in Fig. b, we use N=38), even if it is set to a small value like 1. And since a small variance in the prior weights yields a highly homogeneous set of weight vectors that are considered in the first place, the data that is a priori considered by the weighted average models is restricted accordingly.

In summary, in the individual performance weighting approach, a uniform prior over the models will likely be the best choice in most cases, unless one has prior knowledge that justifies to prefer certain models over others. In the ensemble weighting approach, the prior should be considered more carefully as there are several choices with quite different implications even if we assume no prior knowledge, reflected by an expected value for each wi of 1N. In this case, α=1N seems to be a good choice.

The choice of how to evaluate model performance lies at the heart of every method to compute weights for models in an ensemble. For data on a spatial grid, an example for a performance metric f is the inverse mean squared error between model output and observed data: 8fmiv,yv=1∑sws⋅miv(s)-yv(s)2 where s iterates over spatial locations and ws refers to weights (such as area weighting) normalized to sum to 1.

In the individual performance weighting approach, the weight for model i is then proportional to the performance of model i, based on performance metric f, and assuming a uniform prior over models here for simplicity. 9wi∝fmiv,yv This approach was, for instance, used by . To formulate the weighting in a Bayesian framework, we need to define a proper generative likelihood function instead of relying on a heuristic performance metric. The likelihood relates model output to the observed data in a probabilistic way and allows us to incorporate uncertainties that are not accounted for when using a heuristic performance metric. A possible choice for the likelihood is to assume independent and identically distributed (i.i.d.) Gaussian errors so that 10yv=miv+ϵ,ϵ∼N0,σ211yv∼Nmiv,σ2 where mi refers to the output of any model Mi, be it from a weighted-average model or an individual model. Assuming i.i.d. errors is often unrealistic, but is commonly used as a convenient simplification. The performance weights as computed with ClimWIP are an example of assumed Gaussian errors using individual performance weighting, while , for instance, use Gaussian errors with ensemble performance weighting. Using Gaussian errors requires choosing a value for the variance, σ2. In the individual performance weighting approach, σ2 acts as a hyperparameter that regulates the strength of the relative weighting between models: the smaller σ2, the fewer models are considered to simulate the observed data well. Thus, the number of models that are assigned weights substantially greater than 0 decreases with decreasing σ2. For ClimWIP, tune a parameter (σD) that essentially represents the variance σ2, with the objective to obtain a weighting that is neither too weak (all equal) nor too aggressive (all weight on few models only). As an objective measure of where this value should be, follow and run perfect model tests with different values for σD. They then set σD to the smallest value such that more than 80 %, (90 % in ) of the perfect models lie in the 10 %–90 % (5 %–95 % in ) range predicted by weighting all but the respective “perfect” model. Note, that the ranking of which models are best remains the same irrespective of σ2; only the magnitudes of the weights change. In ClimWIP, the performance weight of a model i is proportional to the likelihood of the observed data under a Gaussian distribution: 12wi∝Ly∣μ=miv,σ2=σD22⋅PMi. In general, it is also possible to sample σ2, which would make the tuning procedure like it was done by unnecessary. However, with ClimWIP, or the individual performance weighting approach in general, this would simply result in a nearly equal weighting since σ2 would be tuned to increase the likelihood of the observed data under every model. Contrary to that, in the ensemble weighting approach, this problem does not arise as the posterior over σ2 represents our uncertainty across the weighted-average models, so that there is no direct relation between the individual model weights and σ2. A common prior distribution for the unknown variance of a normal distribution is, for instance, the inverse-gamma distribution e.g., seeChapter 2.6, which we also use in the examples in the following sections where we apply the ensemble performance weighting approach. The joint posterior distribution over weight vectors and variance σ2 is then given by: 13Pw,σ2∣M∝Ly∣μ=m*v,σ2⋅P(w)⋅Pσ2 where m*v is the output for variable v of the weighted average model using weight vector w. Setting σ2 to a fixed value in the ensemble weighting approach is also an option. , for instance, use the following log likelihood function: -12∑s(Y(s)-Mi(s))2, where s iterates over all grid cells

refer to this function (their Eq. 4) as the likelihood, but it seems that the log likelihood is intended.

We would like to use three schematic (toy) examples to illustrate how the possible lack of independence between ensemble members can influence the weights obtained when using individual or ensemble performance weighting. In each case, we use an ensemble of two base models, M1 and M2, whose output is systematically constructed based on the 1980–2014 mean near-surface air temperature anomalies with respect to the global mean, obtained from ERA5 .

In the first case, we assume that the first model (M1) perfectly simulates the Eastern Hemisphere, while in the Western Hemisphere, a random error sampled from N(0,2) is added to the field. For the second model (M2), the same random error is added in the Eastern Hemisphere, while the Western hemisphere is perfectly simulated. In this way, we get two models with the exact same mean squared error (MSE) with respect to the observed data and we know that the weighting that would minimize the total combined error would be w1=w2=0.5. The data for this case are shown in Fig. .

For better visualization, Figs. – exclude the polar regions, but all data was used in the computations.

In the second case, we set the output of the two models everywhere to the observed data and independently add the noise to both models (again sampled from N(0,2)). Therefore, in this example, the mean squared errors will be similar, but are very unlikely to cancel out at every grid point. The constructed data for this case are shown in Fig. .

In the third example, we do not construct data, but use the output for the same variable and diagnostic from two real models, namely from AWI-CM-1-1-MR and CESM2-WACCM. These data are shown in Fig. .

In each of the three toy examples described above, we build ensembles of different sizes by adding copies of model M2. We know that, by construction, the weight of model M1 (included just once) should always have the same value, irrespective of the number of copies of model M2 included in the ensemble (and conversely, the sum of weights attributed to the included copies of model M2 should be equal to the constant value 1-w1). In the first toy example, both models have the exact same performance (same MSE) so that the weight for M1 should remain at 12, while the individual copies of model M2 should each receive a weight of 12⋅1n2 where n2 is the number of included copies for M2. The same pattern should apply to the other two toy examples, but with the weight for M1 remaining at whatever value it receives when model M2 is included in the ensemble only once. In the second toy example, where we use random errors, this should still be close to 12 whereas in the third example with real model data it depends on how much better or worse M1 is compared to M2. In the case that the model weights deviate from these expectations, it implies that the known (strong) dependence between the copies of model M2 is not properly accounted for.

Figure  shows the resulting weights for these toy examples, using individual or ensemble performance weighting. In the individual performance weighting approach, we evaluate the models by their MSEs (identical to using the sum of squared errors). In the ensemble performance weighting approach, we use a Gaussian likelihood function with the variance jointly sampled with the weight vectors; the shown weights are the mean posterior weights. For the prior over weight vectors, we use a Dirichlet distribution with αi=1N∀i where N is the respective total number of models in the considered ensemble.

In the individual weighting approach, the weights for model M1 decrease as N increases, essentially following the relationship w1=1N. In this case the weight given to any individual copy of model M2 follows the same relationship. This means that the total weight assigned to M2 increases to N-1N. Conversely, in the ensemble weighting approach, neither the weight for model M1 nor the summed weight across all copies of model M2 changes with increasing N. Thus, only the ensemble weighting approach is able to account for the dependencies among the models in the ensemble that we introduced by adding copies of model M2 to the ensemble. When performance is measured for the entire ensemble, adding copies of one model does not have an effect on the mean weights since no new information is added. In the individual weighting approach, a newly added copy, however, is seen to add information to the ensemble by definition as every model is considered separately. Implicitly accounting for model dependence is a strong advantage of the ensemble weighting approach, making the additional computation of independence weights unnecessary. Nonetheless, in the individual weighting approach, it is possible to combine performance and separately determined independence weights so that the weight of dependent models is reduced e.g. see tests from.

To show why the ensemble weighting approach implicitly accounts for model dependency, Fig.  shows the posterior weights for our third toy example, using real data of two CMIP6 models in an ensemble with three members, including M1 once and M2 twice. We can see that there is a strong correlation between the weights of the two copies of model M2 (Fig. a). If the weight of the first copy is high, then the weight of the second copy is necessarily low. This effect was also shown for a simpler case by . Contrary to that, the weight of (each copy of) model M2 does not depend on the weight of model M1 except for the constraint that the weights of all models have to sum up to 1. This is also reflected in the variance of the posterior weights (Fig. b): while the variance for the weights of the two copies of model M2 is large, the variance for the independent model (M1) is quite small.

Furthermore, remember that any single weight vector sampled from the posterior distribution does not reflect each model's performance like it does in the individual performance weighting approach. For example, when w1=0.8, w2.1=0.2 and w2.2=0.0 where w2.1 and w2.2 refer to the weights for the two copies of model M2, this does not mean that the first copy of model M2 shows a better performance than the second copy of the same model with respect to the observed data. Nonetheless, the average weight vector does reflect the relative performance of each model within the given ensemble. In this example, the average weight of each copy of M2 is the same, reflecting that each copy of this model shows equivalent performance (or contributes the same information to the ensemble mean). This difference between sampled weight vectors and the mean weight vector highlights how the conceptualization of the problem is fundamentally different than when individual performance weighting is used.

ECS (Equilibrium Climate Sensitivity) as well as TCR (Transient Climate Response) are promising metrics to compute model weights as they are strongly related to future climate change and most of the uncertainty in future climate projections is attributable to the uncertainty in ECS . Here, we will focus on ECS and show how to use it as performance metric in a fully Bayesian way. ECS has been used previously to compute model weights, for example by who also used an ensemble weighting approach, as we do here, yet without spelling it out completely in a Bayesian framework.

We aim to infer the posterior probability of weight vectors w given model output M (here referring to the models' ECS values): 14P(w∣M)∝P(M*∣w)⋅P(w). We use all CMIP6 models for which we have ECS values, as well as near-surface-air temperatures for the historical time period from 1850–1900 and under the shared socioeconomic pathway scenario 5-8.5 (SSP5-8.5) available (N=38). As likelihood function, i.e. to evaluate the “performance” of a weighted average model, that we also refer to as M* model, we use the distribution of likely ECS values that derived based on various lines of evidence. For the prior distribution over weight vectors w we use a Dirichlet distribution. As we showed in Section , the choice of the parameterization of the Dirichlet prior distribution is important since it influences the range of the considered data (here ECS values) of the resulting M* models. Ideally, we want the prior weights to span the entire range between 0 and 1 so that a large part of the individual models' ECS values is covered by the prior weighted average models. We achieve this by setting the concentration parameter α to 1N. Figure  shows the distribution of the ECS values for the weighted average models (M*), based on the prior (pink density) and posterior weights (green density), using Dirichletα=1N (left) and, for comparison, using Dirichlet(α=1) (right). Only when the prior is sufficiently broad (Fig. a), does the mean ECS value of the M* models, computed based on the posterior weights, shift towards the likely range of the target distribution at around 3 °C, below the multi-model mean. When α=1 (Fig. b), the range of a priori considered ECS values tightly sits around the multi-model mean as, with this parameterization, on average each model receives equal weight.

We use the computed posterior over weight vectors and apply it to future projections of the increase in global mean near-surface air temperature. As shown in Fig. , the expected increase in global mean temperatures is lower in the weighted (blue) than in the unweighted (orange) ensemble which we would hope to see, given the known tendency of CMIP6 models to overestimate future warming e.g. . The uncertainty range (inner 90 % quantile) is substantially reduced in the weighted ensemble (posterior weights, blue shaded region) as compared to the unweighted ensemble spread (orange shaded region). A reduced uncertainty range is expected by the mere fact that we consider weighted averages. For comparison, the purple shaded region in Fig. , shows the 0.05 and 0.95 quantiles for the prior weighted ensemble, i.e. using the prior weights (Dirichletα=1N). The 90 % inner quantile range of the unweighted ensemble is in between 1.26 and 1.56 times larger than the 90 % inner quantile range of the prior weighted ensemble (mean 1.36). This shows how much of the reduced uncertainty of the posterior weighted ensemble is just due to the choice of the prior distribution, and how much can be attributed to the data. In any case, it is important to note that the weighting is not necessarily better the smaller the resulting uncertainty range is. If, in the ensemble weighting approach, the prior strongly restricts the considered weighted averages (as in Fig. b), this will influence the uncertainty range of the weighted ensemble for the target variable in that it will be unreasonably small. Therefore, in the ensemble performance weighting approach, prior predictive checks become indispensable for a reasonable interpretation of the weighted ensemble. Figure  in the appendix (Sect. ) shows the drastically reduced uncertainty range for the weighted projections when using a Dirichlet prior with α=1, i.e. corresponding to the data in Fig. b.

How skillful a weighting approach generally is compared to using an unweighted ensemble should furthermore always be assessed through tests . If the computed weights are based on historical data, perfect model tests are an option, where one model is removed from the ensemble, and its output is considered as “observations”. It can then be checked whether the pseudo-observations lie in the predicted range for future timesteps. This procedure is then repeatedly done, each time using a different model as “perfect model” e.g., see. Another option is classical calibration-validation where a subset of the observed data is left out for computing the weights. These data can then be used to test whether the weighted ensemble predicts the out-of-sample data better than the unweighted ensemble. Using ECS as a basis for assessing model performance, as we do here, is different in that it involves an independently derived likelihood function. In this case, one option is to check whether the ECS-based weighting is valuable for weighting the model output for the historical period. As expected due to the relatively small influence of ECS on the warming in the historical period and after consideration of the results in Fig. , the ECS-based weighting is not particularly valuable to weigh the historical period. The summed RMSEs between the observations and the predicted mean values across all years between 1950 and 2014, is nearly identical for the unweighted and the weighted M* model using the mean posterior weight vector. With the weighted ensemble, the RMSE is only marginally smaller (37.68 vs. 37.83 or on average per timestep 0.5796 vs. 0.5820). However, long-term warming is much more strongly governed by ECS than near-term warming, so that the influence of ECS-based weights, in terms of the divergence between weighted and unweighted projections, is larger for projected future warming.

With this paper we aimed to find a common language to categorise different ways to compute model weights, describe the different choices that have to be made together with their implications, and generally highlight their advantages and disadvantages. Despite the focus of the paper on an application in the climate domain, the discussed approaches may be relevant for ensemble weighting problems across domains as the problems faced by combining any ensemble of individual models are generally similar e.g., see. We differentiate between two general approaches: ensemble and individual performance weighting. While in the former weights are computed based on the performance of a linear combination of all models, in the latter, weights are computed directly from the performance of each individual model.

As we showed by means of toy examples, a practical advantage of the ensemble performance weighting approach is that it implicitly accounts for the issue of inter-model dependencies since each weight represents the relative contribution of the corresponding model to the weighted average instead of representing its performance as such. Put differently, the mean weight from the ensemble performance approach depends not only on how closely the model simulates the observed data, but also on the other models in the ensemble: If there is a model mj in the ensemble that provides the same information as a model mi, the average weight for both models will be reduced accordingly. Similar to defining model independence in terms of model output similarities, the “independence” part in the resulting mean weight vector, obtained from applying the ensemble performance weighting approach, does not directly depend on the observational data. While some consider it an advantage of the independence weights to not depend on observational data e.g., others argue that the distance to observational data should have an influence on the independence weights e.g.. In any case, what can be noted is that neither the individual performance weighting nor the ensemble performance weighting approach takes into account whether two models represent the relevant physical processes for the target output variable in the same way or differently. Even if two models have similar output, they perhaps still should be considered independent (i.e. not be downweighted) if their outputs result from different representations of the underlying processes. Similarly, two models that generate different outputs, but share similar representations, should be considered less independent (i.e. be downweighted) than they are based on their different outputs. Note that in terms of the influence on the uncertainties, the worst case yet is when models with similar outputs are not downweighted and the similar outputs result from similar representations of the underlying process as in this case, the result would be an underestimation of the uncertainties.

We further considered two additional aspects under the ensemble performance weighting approach, namely the use of multiple diagnostics to measure performance as well as the influence of the choice of the prior distribution over weights. When using multiple diagnostics, we showed that applying them jointly so that the weights are tuned to maximize the overall likelihood, i.e. encompassing all diagnostics, yields better results in terms of the summed RMSE between the weighted average model (using the mean posterior weight vector) and the observed data than applying the diagnostics sequentially and then taking the average of the computed weight vectors. The choice of which diagnostics are most appropriate for measuring performance is a big question on its own e.g., see that we have not addressed here. Since, in the case of climate projections, we are assessing the performance of physical models, it makes sense to analyze variables that are related to the basic dynamic processes relevant for the target . In any case, it should always be made clear that there is no universal answer and that any probabilistic projections are conditional on the metrics and diagnostic climate variables that were used. Furthermore, when multiple diagnostics are considered, these should be as independent of one another as possible since otherwise we would run into the same problem that we have with co-dependent models in the individual performance weighting approach: good models (diagnostics) would be erroneously overestimated while poor models (diagnostics) would be erroneously underestimated.

Our example of using ECS as a performance metric demonstrates the importance, in the ensemble performance weighting approach, of choosing an appropriate prior distribution for the weight vectors. For any set of weight vectors, the resulting set of the weighted averages (of the diagnostic variable) will have a reduced spread as compared to the unweighted ensemble. How much it is reduced, however, strongly depends on the nature of the set of applied weight vectors. Both, Dirichlet(1) and Dirichlet(1/N), are natural choices for the prior distribution as in both cases the expected value of the weight for each model equals 1/N. With Dirichlet(1/N) large model weights are similarly likely as smaller model weights whereas with Dirichlet(1), on average, the individual model weights tend to be smaller. Therefore, the Dirichlet(1) prior is much more constrained towards values that lie near the expected value than is the Dirichlet(1/N) prior. This, in turn, means that much more data would be necessary, using Dirichlet(1), to end up with a posterior distribution that assigns most probability mass on values in the tails of the distribution. Thus, we generally recommend to use a Dirichlet prior with shape parameter 1/N and to further make prior predictive checks that ensure that the data that is a priori generated by the weighted average models is realistic and largely covers the range of data from the ensemble members. Moreover, for a reasonable interpretation of the projected likely range of the weighted ensemble, using the expected weight of the posterior distribution obtained from applying the ensemble performance weighting approach, it is necessary to verify how much of the reduced range is effectively attributable to the data and not a mere effect of the averaging itself. Therefore, a comparison between the weighted projections based on the prior weights and based on the posterior weights is a minimal requirement to validate the weighting (see Fig. ).

In fact, the interpretation of the projected uncertainty range, and the interpretation of the uncertainties in general, deserve some further discussion. The predicted uncertainty range for our target variable (e.g., future projections of the increase of global mean temperature in Fig. ) based on the computed posterior weight vectors in part represents the uncertainty in the computed weights themselves. This is a kind of uncertainty that cannot be included by its nature in the individual performance weighting approach. However, accounting for it does not automatically imply that structural differences between the models are well treated. For large ensembles, this is likely not so problematic, as the ensemble members can be expected to be diverse enough to represent all plausible model trajectories. However, care should be taken when ensembles of opportunity are used, to ensure the uncertainty estimate is not overly narrow. In addition, to get a more representative estimate of the uncertainty around future projections, it would be important to include other kinds of uncertainties that we have not considered here and are often ignored. This includes the uncertainty of the individual models themselves (which can be estimated based on multiple model runs) as well as the uncertainty of the observations that are used for the performance evaluation. The Bayesian formulation of weighting that we proposed here conveniently allows us to make such additions.

Other ensemble weighting methods exist that may inherently characterize the uncertainty of the ensemble better. For example, mixture-density approaches may offer an alternative that preserves cross-model variance. The adaptation of the classical Bayesian model averaging (BMA) approach from , for instance, computes mixture densities that combine the probability densities of the individual models (that need to be defined, e.g. via multiple model runs if available), yielding larger uncertainty ranges. They applied this approach to weather forecasts where there is abundant training data and the focus is on short timescales. The need to define a probability density for individual models might be relaxed if a scoring rule comparing cumulative distributions e.g. is used instead of a likelihood, borrowing from the generalized-Bayes idea . A thorough comparison of these and related approaches like Bayesian stacking see is beyond the scope of this paper, but represents a valuable direction for future work.

A limitation of the ensemble performance weighting approach is that it is restricted to linear combinations of individual models. However, in some cases it might be more appropriate to combine models in a non-linear way. Some models, for instance, perform much better in particular regions of the globe than in others, suggesting that the weights may be modeled to differ across grid points, or regions e.g.,. As the choice of the diagnostics, whether or not it is helpful to compute weights dependent on a region or even on a grid-point level also depends on the objective. If the objective is to compute the likely range of global mean temperatures in the future, it seems more plausible to stick with one weight per model whereas weights on the level of grid cells may be useful when considering likely ranges of temperatures within a certain region only.

A key focus of this work has been to compare the different assumptions and implications of the individual and the ensemble performance weighting approach. Overall, past work indicate that ensemble performance weighting provides benefits over individual performance weighting. As just mentioned, there may be cases in which ensemble performance weighting is not appropriate. However, there are many cases in the literature where the final goal is to obtain good performance of the combined ensemble, i.e. we are interested in knowing the weighted-mean prediction. In those cases, ensemble performance weighting is more straightforwardly connected to the goal and provides an estimate of the uncertainty in the weights which is not accounted for in the individual performance weighting approach. If, on the other hand, the goal is simply to rank models or model versions against each other, then the individual performance weighting approach is straightforward in its application.

With this paper we aimed to find a common terminology to categorise different approaches to compute model weights that facilitates a comparison between them, and makes their underlying assumptions transparent. For that we distinguish between two general methods that we call the individual performance weighting and the ensemble performance weighting approach. The former comprises all methods that compute a single weight vector where each model is assigned a probability relative to its individual performance while methods belonging to the latter compute a distribution over weight vectors tuned to optimize the performance of the weighted ensemble as a whole. The ensemble performance weighting approach, for instance, has the advantage to implicitly account for model inter-dependencies. However, which method is more appropriate will be different from case to case depending on the target application of the model weights. Here we applied the ensemble performance weighting approach to an ensemble of climate models, but the approach is not constrained to the domain of the Earth Sciences. We have shown that if the ensemble performance weighting approach is applied to compute weighted projections of global mean temperatures, there are two minimal requirements which should ideally both be reported besides the resulting weighted posterior projections: the weighted prior projections, i.e. the projections based on the prior weights, and prior predictive checks that show that the a priori considered weighted diagnostic data reasonably represents the ensemble and does not constrain it too much. Overall, our analyses showed the importance of explicitly acknowledging the influence of our choices, starting with the general approach to compute model weights.