Development and prospects of the regional  MiKlip decadal prediction system over Europe:  predictive skill, added value of regionalization,  and ensemble size dependency

Reyers, Mark; Feldmann, Hendrik; Mieruch, Sebastian; Pinto, Joaquim G.; Uhlig, Marianne; Ahrens, Bodo; Früh, Barbara; Modali, Kameswarrao; Laube, Natalie; Moemken, Julia; Müller, Wolfgang; Schädler, Gerd; Kottmeier, Christoph

doi:https://doi.org/10.5194/esd-10-171-2019

Articles | Volume 10, issue 1

https://doi.org/10.5194/esd-10-171-2019

© Author(s) 2019. This work is distributed under
the Creative Commons Attribution 4.0 License.

https://doi.org/10.5194/esd-10-171-2019

© Author(s) 2019. This work is distributed under
the Creative Commons Attribution 4.0 License.

Articles | Volume 10, issue 1

Research article

|

20 Mar 2019

Research article |

| 20 Mar 2019

Development and prospects of the regional MiKlip decadal prediction system over Europe: predictive skill, added value of regionalization, and ensemble size dependency

Mark Reyers, Hendrik Feldmann, Sebastian Mieruch, Joaquim G. Pinto, Marianne Uhlig, Bodo Ahrens, Barbara Früh, Kameswarrao Modali, Natalie Laube, Julia Moemken, Wolfgang Müller, Gerd Schädler, and Christoph Kottmeier

Download

Final revised paper (published on 20 Mar 2019)
Preprint (discussion started on 04 Sep 2017)

Interactive discussion

Status: closed

AC: Author comment | RC: Referee comment | SC: Short comment | EC: Editor comment

- Printer-friendly version

- Supplement

RC1: 'Review', Anonymous Referee #1, 10 Oct 2017
- AC1: 'Reply to the interactive comment Reviewer1', Mark Reyers, 16 Jan 2018
RC2: 'Review of "Development and prospects of the regional MiKlip decadal prediction system over Europe: Predictive skill, added value of regionalization and ensemble size dependency"', Anonymous Referee #2, 19 Dec 2017
- AC2: 'Reply to the interactive comment Reviewer2', Mark Reyers, 16 Jan 2018

Peer-review completion

AR: Author's response | RR: Referee report | ED: Editor decision

ED: Reconsider after major revisions (01 Feb 2018) by Andrey Gritsun

AR by Mark Reyers on behalf of the Authors (03 May 2018) Author's response Manuscript

ED: Referee Nomination & Report Request started (18 May 2018) by Andrey Gritsun

RR by Anonymous Referee #1 (03 Jun 2018)

RR by Anonymous Referee #3 (03 Sep 2018)

Suggestions for revision or reasons for rejection

The manuscript “Development and prospects of the regional MiKlip decadal prediction system over Europe: Predictive skill, added value of regionalization and ensemble size dependency” by Mark Reyers et al. provides an insight into regional decadal climate prediction activities that have been undertaken within the MiKlip initiative.

The manuscript targets at answering four central research questions:
a) Is there a potential for skillful regional decadal predictions in Europe?
b) Does the regional downscaling provide an added value for decadal predictions?
c) Does the regional decadal predictive skill depend on the ensemble size?
d) How does the sample size affect the skill estimates?

I was not involved in the first round of the review process. I read the manuscript first without having a look at the response letter before to remain unbiased in this respect. I appreciate the effort the authors spent so far to develop the manuscript. However, even though a regional decadal prediction system is something of great interest for the climate science community, I have to suggest rejecting the manuscript given its current shape and scientific focus.
This for a number of general reasons:

1. The authors state that MiKlip is the first effort worldwide establishing a decadal prediction system for the regional scale (I guess they mean the included dynamic downscaling here.). To my knowledge this statement is absolutely correct but this is for a reason. The dynamical downscaling of the decadal predictions is associated with large computational costs. The authors have to prove that these costs are outperformed by benefits in prediction skill for certain variables or phenomena. The manuscript currently fails completely to deliver this evidence:

1.1. I do not understand why the authors chose perennial means of temperature, precipitation, and wind as the basis for their validation of a regional decadal prediction system. The variability of such multi-year means of primary meteorological parameters will be mainly determined by low-frequency variability of large-scale circulation patterns (e.g. NAO). There is no need to employ an RCM to model this, this is done (more or less successfully) by the driving GCM. The only benefit provided by the RCM for these variables' means is (hopefully) some correction of the conditional bias (improved variance) due to a better resolution of orography and land-use. However, such (conditional) bias-correction could be done much cheaper by means of statistical downscaling. I would like to read an argumentation why the authors expect regional downscaling to be valuable in this respect.

1.2 Additionally, I have problems interpreting/understanding the "proof" of added value (i.e. skill) the downscaling yields compared to the GCM predictions. I understand that the authors present Fig. 4 & 5 targeting this question. The problem with Fig. 4 is that the key message (RCM prediction better than GCM prediction) is based on qualitative evaluation. I agree that it seems from Fig. 4 that skill scores are shifted up and right in the scatter diagrams when comparing the RCM predictions to the GCM predictions. But this is just a result from eye-balling the plots. There is no direct quantification of the gain (and its significance). In principle it would be possible to show differences between 4a&b, 4c&d, and 4e&f, respectively. However, I do not really suggest to go for this solution (calculating the differences of these scatter diagrams). I would rather propose to present skill maps (as done for Fig 2 & 3) again, that is a map plot showing skill of RCM-predictions with the GCM-predictions as reference forecast.
Fig.5 is problematic, too. First, I do not understand what the authors indicate by the background color. They say it is the "absolute skill" (page 18 line 32). Is this supposed to mean that the background color indicates whether there is skill over the climatological forecast, while the dots mark skill over the GCM-predictions? If so, that is fine but should be explicitely written in the manuscript (and the figure caption). And I do not understand the difference between ACC and delta_ACC in this diagram. An additional problem of this plot (as with most of the analyses) is that it does not take the uncertainty into account (see my dedicated issue below). When do the authors consider skill scores to be equal??? Strictly speaking, every prediction should be considered equal that is not proven to be significantly better than the reference forecast. My assumption is, that this would lead to a totally yellow table diagram in Fig. 5.
My general doubt regarding the benefit from dynamical downscaling is further supported by Fig. 6. Even though compiled for another research question (the ensemble size-dependent bias), this figure shows impressively (for the example of the IP-region) that the downscaling hardly yields any benefit. If the authors would quantify the uncertainties, I am pretty sure that all RCM-curves would be situated within the confidence intervals of the respective GCM curves.
All this sums up to my point of view that the manuscript fails to provide evidence of (significantly) added value achieved by downscaling for the analyzed variables. From my point of view, the great potential of employing a non-hydrostatic RCM for decadal predictions could be in the representation of extremes, especially for precipitation. However, frequencies of extremes or something similar are not addressed by the manuscript.

2. A point already issued by one of the last round's reviewers (and explicitely mentioned in the editor's decision) is the low number (5) of initializations. The authors present figure 7 in the revised manuscript and state that these plots prove a general compliance of the results derived from five initializations with those based on 41 initializations. This is true for temperature where skill basically exists for whole Europe due to extrenally forced global warming. However, for precipitation and wind there are some important differences between results based on only 5 and 41 initializations, respectively. I resist from naming them explicitely, the key message here is that the uncertainty associated with all skill scores calculated for the manuscript is huge. And of course the skill estimates based on 41 initializations in Fig. 7 still feature substantial uncertainty, they are not the ultimate "true" skill. This is the essential problem of the manuscript: uncertainty is never quantified. Without such quantification of uncertainty/significance, the results cannot be taken scientifically serious.
Hence, the manuscript is not able to achieve its central aim that is providing evidence or at least significant supportive indication of a potential benefit from dynamical downscaling for decadal climate prediction.

3. The scientific questions c) and d), both targeting the ensemble-size-dependent bias of (regional) prediction skill are not worth tackling in a way done by this manuscript. From my point of view, the manuscript does not add anything scientifically new to the literature in this respect. Of course, it is absolutely important to account for this skill-bias when comparing predictions made by ensembles of differing size and it might be helpful to indicate this bias even for analyses of ensembles with same size in order to quantify potential skill of (hypothically) larger ensembles. But there is no doubt about the general question if there is a bias: there is! The authors should have a look into original papers addressing this issue, such as Murphy (1990), Ferro (2007), and Sienz et al. (2016; there might by older papers than Sienz et al., tackling the bias of MSE/RMSE but I currently remember only this rather recent one). They will learn that the skill bias for all metrics used is essentially related to the variance of the target parameter and the actual bias is a result from the additional noise component due to an inaccurate estimate of the signal due to the limited ensemble size. The theoretical behaviour of the bias depending on ensemble-size is thus also known and can be estimated with existing approaches given by these papers. By the way, this disproves the author's statement that only the CRPSS could be corrected, which is why they refrained from using a de-biased version of the CRPSS in order to maintain comparability to the other metrics (page 20 lines 26-30).

Murphy, J.M. (1990): Assessment of the practical utility of extended range ensemble forecasts. Q.J.R. Meteorol. Soc., 116: 89-125. doi:10.1002/qj.49711649105
Ferro, C.A.T. (2007): Comparing Probabilistic forecasting systems with the brier score. Wea. Forecasting, 22: 1076-1088. doi:10.1175/WAF1034.1
Sienz, F., Müller, W.A., Pohlmann, H (2016): Ensemble size impact on the decadal predictive skill assessment. Meteorol. Z., 25(6): 645-655. doi:10.1127/metz/2016/0670

4. The authors present their analyses based on three metrics, that is the MSESS, the ACC, and the CRPSS. However, there is hardly any information on why the authors chose these three metrics: What is the additional information that is provided by ACC and CRPSS compared to using the MSESS only? What do we learn from differences between the various metrcis presented in Fig. 5 (e.g. What does it mean that CRPSS for the FR-region wrt wind is positive for the RCM-prediction compared to the GCM-prediction but negative for all other skill metrics?). There are a few hints in Sec. 3.2 when introducing the metrics, however, some of them are rather misleading. This is especially true for the CRPSS which does not only contain information about the "reliability" of forecasts but also about "resolution" (see page 15 line 16). Additionally, it does contain much more information than only checking whether the ensemble spread is an adequate representation of forecast uncertainty (compare page 20 lines 25-26). Last but not least, the authors do not consider the relationship between the different metrics. This is most relevent for the MSE(SS) and the ACC. Following Murphy (1988) the MSE for a forecast f measured against the obervation o can be decomposed into:

MSE(f,o)=(<f> - <o>)^2 + s_f^2 + s_o^2 - 2 s_f s_o R_{fo}

where <f> and <o> are the means of f and o, respectively, s_f^2 and s_o^2 are the sample variance of f and o, respectively, and R_{fo} is the sample correlation of f and o. As soon as anomalies are considered, as done in the study by Reyers et al., the first term becomes 0 for all instances. A positive MSESS (meaning a lower MSE than the reference forecast) hence can be achieved by a smaller forecast variance (this is where the ensemble size is relevant, by the way => larger ensembles yield smaller noise variance) or an increase in correlation. The latter is additionally calculated for the present study, although - as just described - closely linked to the MSE(SS). This might make sense if drawing the right conclusions, especially for cases where ACC and MSESS develop differently. However, currently it seems that the study just somehow "quantifies skill" (whatever that means) without any attempt to assess what the reasons are why the RCM is performing better/worse than the GCM predictions (and still failing because not quantifying the uncertainty). I suggest that the authors present less skill metrics (maximum of two) but clearly think about about which to choose and really spend some effort to disentangle the different meanings provided by these. One natural choice would be to choose MSESS and CRPSS which comprises a maximum of information. However, it might be better to stay with deterministic scores only. This would be the MSE(SS) as primary verification metric with the ACC as complementary metric in order to assess whether it is really the signal that is predicted better/worse or whether it is only a change in forecast variance (which might be the case when comparing RCM and GCM predictions).

Murphy, A.H. (1988): Skill Scores Based on the Mean Square Error and Their Relationships to the Correlation Coefficient. Mon. Wea. Rev., 116, 2417-2425, doi:10.1175/1520-0493(1988)116<2417:SSBOTM>2.0.CO;2

Hide

ED: Reconsider after major revisions (19 Sep 2018) by Andrey Gritsun

AR by Mark Reyers on behalf of the Authors (30 Oct 2018)

ED: Referee Nomination & Report Request started (22 Nov 2018) by Andrey Gritsun

RR by Anonymous Referee #3 (28 Jan 2019)

Suggestions for revision or reasons for rejection

Dear Mark Reyers et al!

I appreciate your efforts in revising the manuscript "Development and prospects of the regional MiKlip decadal prediction system over Europe: Predictive skill, added value of regionalization and ensemble size dependency", addressing most of the concerns risen by me and Reviewer 1 appropriately. However, I get the feeling that some of your responses and revisions were compiled a little bit hastily. This is why I still consider a number of revisions necessary. I am very confident that you will be able to address these appropriately, this is why I suggest that your manuscript may be accepted subject to minor revisions. Please see my detailed comments below.

General comments:
-----------------
1. I suggest changing the order of results shown in section 4.1 (Fig. 2 & 3). It seems more logical to me - and hence easier to follow for the reader - to present the skill analyses starting with a reference forecast that would be expected to be rather easy to outperform (climatological forecast) before using reference forecasts that are a priori known to contain skill (the uninitialzed simulations). The results in section 4.2 then continue this logical order by using even more sophisticated reference forecasts, the initialized global predictions.

2. I have the feeling that you did not fully get the point of my comment 3 of last round's review, that is the issue of ensemble size and its impact on skill. I agree that this issue has to be considered seperately from the issue of few initializations (rather use this term than "sample size", from my understanding the "sample size" comprises both, the number of initializations times the ensemble size).
Let's focus on the ensemble size issue here. I still insist on my point of view that this problem of an ensemble-size dependent bias is known (and solutions or let's say workarounds). This is a mathematical issue of the various skill metrics as they are and has nothing to do with the origin of the actual data (synthetic, weather or climate prediction, global or regional). In the end the bias itself and its behaviour with a growing number of ensemble members depends only on the signal-to-noise-ratio (which crucially depends on the variance of the targeted variable). Anyway, as I wrote in my previous review: I appreciate your effort of tackling this issue (many studies do not, and probably aren't even aware of this problem). However, your results do not add anything new to the scientific literature regarding the existence of this bias. That is why, I oppose against your research question 3 (Does [...] skill depend on ensemble size?).
To resolve this issue, I suggest the following:
Please reformulate your research question 3 to something like "How does ensemble size impact regional decadal prediction skill?" It's a small change but this "how" makes a difference from my point of view. And please, carefully revise your text in a sense that it does not sound anymore as if the basic question (dependence yes or no) is something that has never been addressed before. Maybe you can write something like "the ensemble-size dependent skill bias has never been demonstarted based on regional decadal climate predictions before".

3. Regarding your research question 4: As shortly mentioned above, please refrain from naming this issue as a matter of sample size. The sample size in the end is determined by the ensemble size and the number of initializations. Please make clear that you address the issue of few initializations here!

Specific comments:
------------------
1. Page 2, line 15: I suggest including a reference to Eade et al. (2012) related to the prediction of extremes.

2. Page 2, lines 31-33: Please split in two sentences. First one to end after "...techniques". And I suggest to replace "outstanding" with "exceptionally".

3. Page 3, line 7: "aspread" should be "spread".

4. Page 3, line 12: Replace "for the decadal predictability" by "regarding skill". The reference to the decadal timespan comes later in the very same sentence. And "predictability" actually is to be differentiated from "prediction skill" but this is another discussion and unfortunately done inaccurate by many colleagues.

5. Page 3, lines 21-22: Please reformulate reserach questions 3 and 4 according to the suggestions made in my general comments.

6. Page 3, line 32: As already mentioned by another reviewer during the discussion stage of this manuscript (and answered correctly by you), ocean temperature and salinity are NOT taken from NCEP/NOAA reanalysis. Please pay attention to this issue and describe the initialization procedure correctly.

7. Page 4, lines 7-9: I suggest removing the sentence regarding the downscaling from this paragraph. It's almost exactly repeated in the following paragraph (lines 15-17) which is specifically dedicated to the downscaling.

8. Just as a matter of curiosity: Did you test whether it makes a difference for your skill estimates if you take (interpolated) winds from ERA-reanalyses directly instead of using the ERA-driven CCLM-simulation as a reference?

9. Page 4, lines 27-28: It is not correct that "historical" simulations are forced ONLY by aerosol and greenhouse gas concentrations. There is a number of other external forcings that are prescribed. Please rephrase accordingly.

10. Page 4, lines 31-32: What is the interpolation method you used? Did you test for alternative interpolation methods and the related impact on skill? It's quite a range of resolutions you are using here. Given that you interpolate from much coarser but also from (slightly) higher resolution (featuring a rotated grid) to the 0.25deg-grid, I guess bi-linear interpolation for all datasets would be the most appropriate solution. In any case, please indicate your the method chosen by you.

11. Page 5, lines 1-2: How were the anomalies calculated for the hindcasts, that is to mean, how did you calculate a climatology from the hindcasts? This question may sound stupid, but the devil is in the detail and it might even be, that your hindcasts feature drifts (even though it is anomaly initialization). I think the latest recommendation in this respect is to define a baseline period that is covered by the same number of initializations for all lead times and then calculate a climatology for every single lead time separately. This is not possible for you given that you have initializations only five-yearly. I don't ask for a change in your approach here (whatever you did) but please describe precisely how you did it.

12. Page 5, line 4: I suggest replacing "mainly" by "partly". You do spatial averaging over the regions only for the results presented in tables 1-3. However, the number of plots where you did not perform spatial averaging is much higher.

13.Page 5, line 21: Remove the reference to Goddard et al (2013) here. They were not the ones defining the MSESS.

14. Page 6, line 1: Two issues regarding the MSESS-formula:
14.1. Please replace the vertical bars (indicating the calculation of an absolute value) by brackets. Essentially it doesn't make a difference here but, you should follow the derivation presented by Murphy (1988).
14.2. More generally; i wonder if it is useful to present a formula for the MSESS that holds only for the climatology as reference forecast. You present results, too, that are based on other reference forecasts, so it may be better to stay with the basic definition of MSE and MSESS as provided on page 5 already. If you want to present some decomposition, I suggest to stay with the more general one, I wrote down in my previous review. And given that these derivations are provided by other papers already, I don't see the need to indicate some derivation of the conditional bias in your paper. From my point of view, you could directly provide the formula for the conditional bias as it is (see below). But please spend a few sentences, describing explicitely that the MSESS (partly) depends on the correlation but also on the condictional bias. This would be an important step, helping the reader to establish links between your MSESS- and ACC-results. This is what I asked for in my previous review.

15. Page 6, line 4: Now the vertical bars are definitely wrong. Your tables contain negative values for the CB, too. So, it should be brackets (or nothing).

16. Page 6, line 14: The correlation (ACC) is not only independent from the mean bias but also from the variance of the specific target variables.

17. Page 7, line 1-3: Please avoid using "bias" in this context. This might be misleading. Maybe just replace by "deviation", or something similar.

18. Page 7, line 19: Delete "of the hindcasts and decadal predictions".

19. Page 7, line 20: Replace "more reliable" by something like "better" or similar. Reliability has a specific meaning in forecast verification that is not meant here.

20. Page 8, lines 30-33: Two issues regarding your thersholds for coloring MSESS- and CB-values in the tabel:
20.1. Your thersholds in the text partly don't match the thresholds mentioned in the caption. Please check carefully!
20.2. The choice of these thresholds seems totally arbitrary here. Is there some though behind it? If so, please explain. At least for the MSESS it would have made much more sense to me to also use a certain significance threshold as justification for coloring here.

21. Page 9, line 1-3: Why do you suddenly use a t-test for assessing statistical significance? It would have been possible to use the same bootstrapping approach for CB_AV, too.

22. Page 10, line 4-5: As mentioned above, it is no open question if there is a dependency. This would be one of the instances where I would ask you for a reformulation in the sense of "demonstrating skill bias dependency" (see my general comment 2 above).

23. Page 10, lines 8 and follwing, as well as Fig. 5: I think it might be quite misleading to demonstrate this bias dependency compared to the unitialzed simulations. They suffer from a bias, too, and currently it is not clear from the manuscript whether you reduce the number of unitialised simulations for the reference forecast, too, or use their full ensemble in every instance. I strongly suggest to demontrate the skill bias comparing to the climatology as reference forecast. This one is unbiased!

24. Page 11, lines 3-7: I think you definitely should remind the reader here once again that your box-whisker plots do not contain the full uncertainty of the skill score estimates!

25. Page 11, lines 11-12: This is also included in the CMIP6-DCPP requirements/recommendations, sp please include a reference to Boer et al. (2016) here, too.

26. Page 11, line 14: Please rephrase the research question and your following text in the sense of my general comment 3, so make it mor clear that you are addressing the issue of few initializations here.

27. Page 11, line 23: Replace "starting years" by "initializations only".

28. Page 11, line 24: Replace "starting years" by "initializations".

29. Page 12, line 5: Replace "predictability in" by "prediction skill of".

30. Page 12, line 27-28: Rephrase. Maybe something like "Based on MPI_b1 data [...] we could show that results derived from only those five initializations used in our study qualitatively agree with results based a full set of annual initializations".

31. Page 12, line 32: Replace "predictability" by "prediction skill".

32. Page 12, line 19: Be cautious with such statements regarding the relevance of drifts (and subsequent corrections) in case of anomaly initializations. Some studies show that these feature drifts, too. Reformulate to something like "the general expectation is that drift correction is less important for prediction systems employing anomaly initialization".

33. Page 12, line 22: Replace "starting dates" by "initializations".

34. Page 14, line 1: Replace "predictability" by "prediction skill".

35. Page 14, line 3: Maybe it's worth mentioning that this huge amount of 1000 RCM model years may also be a valuable set for other studies, not necessarily related to decadal prediction. I mean, it's a huge dataset in comparably high resolution representing the European climate of the recent past...

Hide

ED: Publish subject to minor revisions (review by editor) (04 Feb 2019) by Andrey Gritsun

AR by Mark Reyers on behalf of the Authors (13 Feb 2019) Author's response Manuscript

ED: Publish subject to technical corrections (01 Mar 2019) by Andrey Gritsun

AR by Mark Reyers on behalf of the Authors (07 Mar 2019) Author's response Manuscript

Short summary

In this study, the regional MiKlip decadal prediction system is evaluated. This system has been established to deliver highly resolved forecasts for the timescale of 1 to 10 years for Europe. Evidence of the general potential for regional decadal predictability for the variables temperature, precipitation, and wind speed is provided, but the performance of the prediction system depends on region, variable, and system generation.