An investigation of weighting schemes  suitable for incorporating large ensembles  into multi-model ensembles

Merrifield, Anna Louise; Brunner, Lukas; Lorenz, Ruth; Medhaug, Iselin; Knutti, Reto

doi:https://doi.org/10.5194/esd-11-807-2020

Articles | Volume 11, issue 3

https://doi.org/10.5194/esd-11-807-2020

© Author(s) 2020. This work is distributed under
the Creative Commons Attribution 4.0 License.

Special issue:

Large Ensemble Climate Model Simulations: Exploring Natural...

https://doi.org/10.5194/esd-11-807-2020

© Author(s) 2020. This work is distributed under
the Creative Commons Attribution 4.0 License.

Articles | Volume 11, issue 3

Research article

| Highlight paper

|

16 Sep 2020

Research article | Highlight paper |

| 16 Sep 2020

An investigation of weighting schemes suitable for incorporating large ensembles into multi-model ensembles

Anna Louise Merrifield, Lukas Brunner, Ruth Lorenz, Iselin Medhaug, and Reto Knutti

Download

Final revised paper (published on 16 Sep 2020)
Preprint (discussion started on 21 Nov 2019)

Interactive discussion

Status: closed

AC: Author comment | RC: Referee comment | SC: Short comment | EC: Editor comment

- Printer-friendly version

- Supplement

RC1: 'Review of "A weighting scheme to incorporate large ensembles in multi-model ensemble projections" by Merrifield et al.', Anonymous Referee #1, 10 Jan 2020
- AC1: 'Response to “Review of "A weighting scheme to incorporate large ensembles in multi-model ensemble projections" by Merrifield et al.”', Anna Merrifield, 29 Feb 2020
RC2: 'Review of "A weighting scheme to incorporate large ensembles in multi-model ensemble projections" by Merrifield et al.', Anonymous Referee #2, 14 Jan 2020
- AC2: 'Response to “Review of "A weighting scheme to incorporate large ensembles in multi-model ensemble projections" by Merrifield et al.”', Anna Merrifield, 29 Feb 2020

Peer-review completion

AR: Author's response | RR: Referee report | ED: Editor decision

ED: Reconsider after major revisions (01 Apr 2020) by Sebastian Milinski

AR by Anna Wenzel on behalf of the Authors (19 May 2020) Author's response

ED: Referee Nomination & Report Request started (05 Jun 2020) by Sebastian Milinski

RR by Anonymous Referee #1 (17 Jun 2020)

Suggestions for revision or reasons for rejection

I really appreciate the responses to my comments and the major modifications made to the paper. I think the paper is more interesting now, with a very thorough investigation of issues associated with weighting, that goes beyond the simple incorporation of SMILES in ensemble projections (by the way, I'm not sure that the title is really optimal now). The paper will be suitable for publication after minor revisions.

The line numbers refer to the version with highlighted changes in the author response.

L4 "SMILES represent internal variability…" The formulation seems strange to me.

L124 You could say a word on these differences of physics (they are quite limited actually)

Figure 1. What is shown for CMIP5? Statistics on all the members, with equal weighting, without taking into account that some models provide several IC members? It should be noted in the legend as it is not an approach that one would normally used, with maybe a reference to the discussion in section 3.1.
Spread: please say in the legend how the spread is calculated in practice. There are many ways to define a spread.

L218. For wiIII. I don't understand the 1/N2 factor. It should be 1/N. Is this just a mistake in the written equation (the title of the subsection is 1/N scaling after all) or are the analyses impacted?

L236. Does each model get a unique performance weight as in section 3.3 or are different performance weights given to each IC member of a model? It is not clear.

Line 378-379. It is not totally obvious to me: if a model is almost the truth, is almost perfectly realistic, and all the other models are wrong, why would it be problematic for this model to be over-represented? How do we decide in practice that a model is over-represented or not?

Line 386-387. I'm not sure I understand that reasoning. The spread could be the same just coincidentally.

L389. "In light of the clear necessity": Not that clear to me.

L567: "A weighting scheme, such as the one assessed here, is thus ideal for providing justifiable estimates of uncertainty…" Which weighting scheme? Several weighting schemes are discussed, and for me it is not obvious which one is the best among the last three.
This paper actually shows that different weighting schemes that make sense can lead to different results in terms of spread and even response. It is therefore still very difficult to provide justifiable "estimates of uncertainty". And as discussed in Appendix B some ad-hoc choices have to be made: values of sigma d and s etc. I really appreciate the different tests used by the authors to define these parameters, they did a very good job, but some subjectivity remains.

L625. I'm not sure to understand exactly how it is done. How the prediction intervals corresponding to each truth are calculated in practice?
For each truth, do you calculate the weighted mean and weighted standard deviation (for each value of sigma), which you use to compute the 10-90% prediction interval supposing a Gaussian distribution, and then check whether the truth is within this interval?

I suppose that the basic "equal weighting" approach pass the test, right? Therefore, how do you decide that your weighting scheme is better than the standard democracy? You say that you choose the smallest value of sigma that passes the test, but why the smallest value of sigma should be used? Why is it preferable?

L657: "… and set sigma s at two standard deviations below the SMILE Sij mean value". I understand the logic, but why this specific choice of two standard deviation below Sij mean value?

Hide

ED: Publish subject to minor revisions (review by editor) (03 Jul 2020) by Sebastian Milinski

AR by Anna Merrifield on behalf of the Authors (13 Jul 2020) Author's response Manuscript

ED: Publish as is (03 Aug 2020) by Sebastian Milinski

Download

The requested paper has a corresponding corrigendum published. Please read the corrigendum first before downloading the article.

Article (6333 KB)
Full-text XML

Short summary

Justifiable uncertainty estimates of future change in northern European winter and Mediterranean summer temperature can be obtained by weighting a multi-model ensemble comprised of projections from different climate models and multiple projections from the same climate model. Weights reduce the influence of model biases and handle dependence by identifying a projection's model of origin from historical characteristics; contributions from the same model are scaled by the number of members.