Uncertainty-informed selection of CMIP6 Earth System Model subsets for use in multisectoral and impact models

Snyder, Abigail; Prime, Noah; Tebaldi, Claudia; Dorheim, Kalyn

doi:https://doi.org/10.5194/esd-2023-41

Preprints

https://doi.org/10.5194/esd-2023-41

Preprints

05 Jan 2024

| 05 Jan 2024

Status: a revised version of this preprint was accepted for the journal ESD and is expected to appear here in due course.

Uncertainty-informed selection of CMIP6 Earth System Model subsets for use in multisectoral and impact models

Abigail Snyder, Noah Prime, Claudia Tebaldi, and Kalyn Dorheim

Abstract. Earth System Models (ESMs) are heavily used to provide inputs to impact and multisectoral dynamic models. Therefore, representing the full range of model uncertainty, scenario uncertainty, and interannual variability that ensembles of ESMs capture, is critical to the exploration of the future co-evolution of the integrated human-Earth system. The pre-eminent source of these ensembles has been the Coupled Model Intercomparison Project (CMIP). With more modeling centers participating in each new CMIP phase, the size of the ESM archive is rapidly increasing, which can be intractable for impact modelers to effectively utilize due to computational constraints and the challenges of analyzing large datasets. In this work, we present a method to select a subset of the latest phase, CMIP6, models for use as inputs to a sectoral impact or multisectoral models, while still representing the range of model uncertainty, scenario uncertainty, and interannual variability of the full CMIP6 ESM results. This method is intended to help human-relevant impact and multisectoral modelers select climate information from the CMIP archive efficiently. This is particularly critical for large ensemble experiments of multisectoral dynamic models that may be varying additional features beyond climate inputs in a factorial design, thus putting constraints on the number of climate simulations that can be used. We focus on temperature and precipitation outputs of ESMs, as these are two of the most used variables among impact models and many other key input variables for impacts are at least correlated with one or both of temperature and precipitation (e.g. relative humidity). Besides preserving the multi-model ensemble variance characteristics, we prioritize selecting ESMs in the subset that preserve the very likely distribution of equilibrium climate sensitivity values as assessed by the latest IPCC report. This approach could be applied to other output variables of ESMs and, when combined with emulators, offers a flexible framework for designing more efficient experiments on human-relevant climate impacts. It can also provide greater insight into the properties of existing ESMs and the method may be informative for future experiment planning across ESMs.

Received: 01 Dec 2023 – Discussion started: 05 Jan 2024

Publisher's note: Copernicus Publications remains neutral with regard to jurisdictional claims made in the text, published maps, institutional affiliations, or any other geographical representation in this preprint. The responsibility to include appropriate place names lies with the authors.

Download & links

Abigail Snyder, Noah Prime, Claudia Tebaldi, and Kalyn Dorheim

Status: closed

RC1:
'Comment on esd-2023-41', Anonymous Referee #1, 09 Jan 2024

The study posits a strategy for selecting 5 CMIP6 GCMs that are suitable globally for impact model applications based on temperature and precipitation characteristics and the IPCC likely ECS range. The results would benefit from context, both in terms of how the study compares to previous model subselection exercises (why select the same set of models for all regions?) and in a deeper dive into the origins of the IPCC likely ECS range (where it comes from, what constraint assumptions are being made). Additionally, the methodology is hard to follow in the appendix and is worth moving to the main text. The primary issue I have, though, is that taking the "model uncertainty, scenario uncertainty, and interannual variability of the full CMIP6 ESM results" is inappropriate in an ensemble of opportunity like CMIP6 without a careful audit of model dependence.
Recommended Literature:
Abramowitz, G., Herger, N., Gutmann, E., Hammerling, D., Knutti, R., Leduc, M., Lorenz, R., Pincus, R., and Schmidt, G. A.: ESD Reviews: Model dependence in multi-model climate ensembles: weighting, sub-selection and out-of-sample testing, Earth Syst. Dynam., 10, 91–105, https://doi.org/10.5194/esd-10-91-2019, 2019.
Brands, S.: A circulation-based performance atlas of the CMIP5 and 6 models for regional climate studies in the Northern Hemisphere mid-to-high latitudes, Geosci. Model Dev., 15, 1375–1411, https://doi.org/10.5194/gmd-15-1375-2022, 2022.
Merrifield, A. L., Brunner, L., Lorenz, R., Humphrey, V., and Knutti, R.: Climate model Selection by Independence, Performance, and Spread (ClimSIPS v1.0.1) for regional applications, Geosci. Model Dev., 16, 4715–4747, https://doi.org/10.5194/gmd-16-4715-2023, 2023.
Specific Comments:
L60-62: "In a world unburdened by time and computing constraints, an impact model would take as input every projected data set available to have a full understanding of possible outcomes." - An ensemble of every projected data set in CMIP6 does not confer the full understanding of possible outcomes. It would include 50 initial condition ensemble members of certain ESMs and one ensemble member for others. Is the first 50x more likely to be true? Beyond the unequal voting power of the large ensembles in CMIP6, the ensemble contains a number of "hidden dependencies": models with different names but near-identical code. For uncertainty to mean "our full understanding of possible outcomes", model dependence must be handled properly.
L81-82: While this is an interesting objective, the size of the initial condition ensembles submitted to an exercise like CMIP is a function of computational resources and goodwill (they are submitting "free" data for others) on the side of the modeling centers. It is beneficial to many researchers that CMIP is inclusive and does not "pick favorites" thus encouraging participation.
Table 1: Of the 22 models you are using, 6 are connected, either by legacy or because they use a version, to NCAR's Community Atmosphere Model (CAM) development cycle. As this leaves the potential for CAM to have more influence on your uncertainty benchmark, the choice must be discussed. Additional similar models, such as ACCESS-CM2 / UKESM1-0-LL and MPI-ESM1-2- HR / MPI-ESM1-2- LR, present could be creating a “rather heterogeneous, clustered distribution, with families of closely related models lying close together but with significant voids in-between model clusters” (description of CMIP5 from Sanderson, B. M., Knutti, R., and Caldwell, P.: A representative democracy to reduce interdependency in a multimodel ensemble, J. Climate, 28, 5171–5194, https://doi.org/10.1175/JCLI-D-14-00362.1, 2015.). Could multiple similar models elevate outliers (e.g., MIROC) in your metric?
L166: Please justify the citation of Scafetta, 2022. See https://agupubs.onlinelibrary.wiley.com/doi/10.1029/2022GL102530
In step 3 of Table 2, how are you computing an ensemble average for the models that only provide a single run?
L224: Omit "over time?"
L315: "Models who more closely match the trend of observational data (W5E5v2.0 (Lange et al., 2021)) over the historic period will have their observations hold more weight. " Why? Trends are highly sensitive to internal variability, which is inherently random in temporal phase, i.e. no reason a model and observation should have the same sequence of it. A match in trend between observations and a model over a particular time period often occurs by chance and is not indicative model performance.
Deser, C., Phillips, A., Alexander, M. A., and Smoliak, B. V.: Projecting North American climate over the next 50 years: Uncertainty due to internal variability, J. Climate, 27, 2271–2296, https://doi.org/10.1175/JCLI-D-13-00451.1, 2014.

Citation: https://doi.org/10.5194/esd-2023-41-RC1
- AC2: 'Reply to all reviewer comments', Abigail Snyder, 16 Apr 2024
  
  The comment was uploaded in the form of a supplement: https://esd.copernicus.org/preprints/esd-2023-41/esd-2023-41-AC2-supplement.pdf
  
  Citation: https://doi.org/10.5194/esd-2023-41-AC2
- AC1: 'Reply to all reviewer comments', Abigail Snyder, 16 Apr 2024
  
  The comment was uploaded in the form of a supplement: https://esd.copernicus.org/preprints/esd-2023-41/esd-2023-41-AC1-supplement.pdf
  
  Citation: https://doi.org/10.5194/esd-2023-41-AC1
RC2:
'Comment on esd-2023-41', Anonymous Referee #2, 11 Jan 2024

Review of “Uncertainty-informed selection of CMIP6 Earth System Model subsets for use in multisectoral and impact models” by Snyder et al.
The presented study develops a model selection procedure that aims to preserve the distribution of total uncertainty as defined by Hawkins and Sutton. The motivation and need for some kind sub-selection of global climate models for impact models is clear and well-motivated by the authors. Their method is fairly straight-forward and is shown to work for the metric the authors use as validation.
I think this approach could be valuable contribution to be used as an objective selection criterion if the authors manage to manage to better describe the caveats and limits of this approach and better showcase the effect of the sub-selection. In its current state I can not recommend the manuscript for publication based on the major comments outlined in the following.

Major comments
I am very critical about the idea of optimizing a sub-selection method only for the distribution of the three sources of uncertainty. I note that the authors also manually select by ECS but this seems to be not really part of the method and is presented more as an afterthought.
In any case, such an approach opens up the risk of selecting highly dependent models (as seems in fact to be the case in the presented example with both ACCESS versions being part of the suggested sub-ensemble) as well as objectively bad models. Both are situations which should be avoided I would argue.
In fact, in a recent study Merrifield et al. (2023; 10.5194/gmd-16-4715-2023) set out to solve a similar problem but also consider model performance and independence. Their results should at least be compared to the results presented in this manuscript.
I have to admit I do not fully understand the role of the scenario uncertainty this manuscript. In an idealized case the scenario uncertainty as defined by HS is only dependent on the scenarios used. Why is it considered here, what is it relation to model selection? If it is sensitive to the selected (ECS preserving) model ensemble this is only a sign that the HS method is not properly able to isolate forced response, model uncertainty, and internal variability (as discussed in Lehner et al. 2020) is it not?
With respect to the verification metric, I am not sure if optimizing for the relative distribution of uncertainty is really ideal. At least the authors should consider also looking at the absolute position of their selected models in temperature-precipitation change space. I could imagine a situation where the relative distribution of uncertainties in the subset is very similar to the full ensemble without the subset being representative when considering, e.g., absolute changes.

Minor comments

line 33: “Scenario simulations from CMIP (most recently through ScenarioMIP, (O’Neill et al. 2016)”
closing braked missing

36: “Using such multi-model ensembles captures the process and structural uncertainties represented by sampling across ESMs, scenario uncertainty,”
Minor point but a multi model is not necessary to represent scenario uncertainty?
Similar statement also in the abstract: “In this work, we present a method to select a subset of the latest phase, CMIP6, models for use as inputs to a sectoral impact or multisectoral models, while still representing the range of model uncertainty, scenario uncertainty, and interannual variability of the full CMIP6 ESM results”

45: “For Earth system modelers, variability across ESMs’ projections of future climate variables can be significant (Hawkins and Sutton 2009; Hawkins and Sutton 2011; Lehner et al. 2020) and so the participation of multiple modeling centers running multiple scenarios is critical to understanding the future of the Earth system.”
I agree with this statement but “variability across ESMs” is a bit vague. Of the three sources of variability considered in the cited studies only one really needs a multi-model ensemble. I am also not sure if these are really the right studies to cite here as they mainly look at relative contributions of individual sources of uncertainty.

57: “However the total burden across modeling centers to sample across ESMs and scenarios still remains high, even with this potential efficiency.” I am not sure I understand the first part of this sentence?

72: “these competing priorities” what are the ‘competing priorities’ here?

Tab 1: I am a bit confused about the use of the word ESM in this manuscript. This table clearly also contains models versions with are not ESMs but GCMs? (ACCESS-CM2 for example)

108: “and the interannual standard deviation”
It is unclear to me how this is done or in which time-period.

114: At this point I am also wondering how initial-condition members are considered exactly? Is seems that the indices would be sensible to the number of ensemble members?

Fig 1: this figure seems unproportionally big for the amount of information it conveys.

Fig 3: I am not quite sure what information this figure is supposed to convey to the reader? In particular, since it is hardly discussed in the text. For example, is it somehow relevant that MPI or MRI (hard to distinguish) seem to behave quite different in PC2?

165: “Core Writing Team & (eds.),” strange citation
How many subsets fulfill this criterion?

Fig 4: Again I am not sure what I learn from this plot.

199: “To manage the inspection of three time series” Is ‘three time-series’ referring to Hawkins and Suttons model, forced, and internal components? This is not clear to me here. If yesI would note that in HS internal variability is not a timeseries but constant over time if memory serves.

223: “For temperature, we see that interannual variability is often performing well, with increasingly better performance over time? over time.” Typo

Citation: https://doi.org/10.5194/esd-2023-41-RC2
- AC1: 'Reply to all reviewer comments', Abigail Snyder, 16 Apr 2024
  
  The comment was uploaded in the form of a supplement: https://esd.copernicus.org/preprints/esd-2023-41/esd-2023-41-AC1-supplement.pdf
  
  Citation: https://doi.org/10.5194/esd-2023-41-AC1
RC3:
'Comment on esd-2023-41', Anonymous Referee #3, 13 Feb 2024

Review of esd-2023-41
Title: Uncertainty-informed selection of CMIP6 Earth System Model subsets for use in multisectoral and impact models
Authors: Abigail Snyder, Noah Prime, Claudia Tebaldi, Kalyn Dorheim
Overall Recommendation: Minor Revisions
This study, submitted to Earth System Dynamics, highlights a new approach to select subsets of ESMs (and potentially GCMs) for use in multi-sectoral and impacts modeling. The approach is novel and attempts to capture multiple sources of uncertainty in the subset and does have potential utility for multiple applications beyond what was mentioned in the manuscript. However, I do think the manuscript could be improved by added justification in a few places and addressing some particular aspects from prior literature that is relevant. In particular, the attempt at subset selection itself is not a new effort and the authors should place this manuscript in the context of prior attempts at subset selection. My general comments to improve the manuscript are below, followed by specific comments. I look forward to seeing the revised manuscript in print.

General Comments

There are a number of current issues in literature related to the selection of ESMs (or even just a set of climate projections) for use in applications that were not discussed here but are quite relevant.

First and foremost, the challenge of ensemble subset selection is not new. It would be worthwhile given the focus on using ESMs in impacts models to place this article in context with existing literature. For example, Parding et al (2020) produced the GCMeval tool, which is designed as an interactive tool for evaluation and selection of climate model ensembles for use in multiple applications. The methods in the GCMeval tool are relatively simplistic. In the context of this manuscript, I am left to wonder how different this approach is to the approach of Parding et al (2020) or other previous research. It would strengthen the manuscript to briefly discuss the differences between this approach and others in prior literature. Ideally, there should be some analytic comparisons between these approaches, but I believe a brief discussion of the other literature on this topic would suffice for this manuscript. In addition, the authors are addressing the “practitioner’s dilemma” (Barsugli et al. 2013). Though it was not mentioned by name, it may also be worthwhile to discuss how this approach builds on the literature associated with this well-known challenge.

Second, the approach in this manuscript may well retain “hot-models”, particularly as one goes to large subset sizes. The literature regarding if one should use hot-models is mixed, ranging from omitting hot models entirely (Hausfather et al. 2022), to down-weighting hot models based on ECS (Massoud et al. 2023), or simply keeping hot models as they may not have serious impacts on impacts modeling (Asenjan et al. 2023). It would be worthwhile for the authors to address how their method handles the “hot models” problem specifically, when their approach may indeed retain these models in their subsets.

Third, the focus is on impacts modeling with ESMs, but this is not the only use for ESM output. Depending on the needs of a stakeholder, one may not need additional modeling, but rather require complex variables derived from ESM output. A prime example of this is the use of spring phenological indices to determine projected changes in first leaf, first bloom, or the likelihood of a false spring (Gerst et al. 2020; Allstadt et al; 2015; Peterson and Abatzoglou, 2014). The approach in this manuscript could well be used in situations that don’t require additional modeling per se, but do require derivation from ESM or downscaled ESM output. Such literature should be briefly mentioned with the authors comment.

Fourth, the Hawkins and Sutton (2009) approach does have some valid criticisms. I suggest looking at the recent work of Lafferty and Sriver (2023), which also uses Hawkins and Sutton (2009) and the work of Wootten et al. (2017). The Lafferty and Sriver (2023) article addresses the critiques around the Hawkins and Sutton (2009) approach.

Finally, while the selection of ESMs is important, most impacts modelers do not use the ESMs directly, but use the downscaled ESM output (whether dynamically or statistically downscaled). The authors mention this briefly in passing, but it is important to acknowledge this in the conclusions also. Downscaling is itself another source of uncertainty, so it is a question of if this approach could also be applied with ESMs downscaled with multiple approaches.

Literature mentioned above:
Allstadt, A. J., S. J. Vavrus, P. J. Heglund, A. M. Pidgeon, W. E. Thogmartin, and V. C. Radeloff, 2015: Spring plant phenology and false springs in the conterminous US during the 21st century. Environmental Research Letters, 10, https://doi.org/10.1088/1748-9326/10/10/104008
Asenjan, M.R., F. Brissette, J.-L. Martel, and R. Arsenault, 2023: Understanding the influence of “hot” models in climate impact studies: a hydrological perspective. Hydrology and Earth System Sciences, 27, 4355-4367, DOI: 10.5194/hess-27-4355-2023
Barsugli, J., and Coauthors, 2013: The Practitioner’s Dilemma: How to Assess the Credibility of Downscaled Climate Projections. Eos Transactions, 94, 424–425, https://doi.org/10.1002/2013EO460005.
Gerst, K. L., T. M. Crimmins, E. E. Posthumus, A. H. Rosemartin, and M. D. Schwartz, 2020: How well do the spring indices predict phenological activity across plant species? Int J Biometeorol, 64, 889–901, https://doi.org/10.1007/s00484-020-01879-z.
Hausfather, Z., K. Marvel, G. A. Schmidt, J. W. Nielsen-Gammon, and M. Zelinka, 2022: Climate simulations: recognize the ‘hot model’ problem. Nature, 605, 26–29, https://doi.org/10.1038/d41586-022-01192-2.
Lafferty, D.C. and R.L. Sriver, 2023: Downscaling and bias-correction contribute considerable uncertainty to local climate projections in CMIP6. Nature Partner Journal – Climate and Atmospheric Science, 6, doi: https://doi.org/10.1038/s41612-023-00486-0
Parding, K. M., and Coauthors, 2020: GCMeval – An interactive tool for evaluation and selection of climate model ensembles. Climate Services, 18, 100167, https://doi.org/10.1016/j.cliser.2020.100167.
Peterson, A. G., and J. T. Abatzoglou, 2014: Observed changes in false springs over the contiguous United States. Geophysical Research Letters, 41, 3307–3314, https://doi.org/10.1002/2014GL061184.Received.
Wootten, A., A. Terando, B. J. Reich, R. P. Boyles, and F. Semazzi, 2017: Characterizing Sources of Uncertainty from Global Climate Models and Downscaling Techniques. Journal of Applied Meteorology and Climatology, 56, 3245–3262, https://doi.org/10.1175/JAMC-D-17-0087.1.

Minor Comments:

Line 84, first mention of Table 2: Table 2 defines the process for selection to aid the reader, yet it is positioned in the text far from the section. It’s also mentioned before Table 1 and Table 1 appears sooner in the text. I suggest reordering Table 2 and Table 1 and placing the renamed Table 2 earlier in the text to help the reader.

Line 105: “For each scenario and region in each ESM,…” – Am I correct that the averages calculated are across all the initializations of each ESM? Or is it the average across all models?

Lines 127-128: “Based on this figure,…explaining 71.8% of variance.” – Why only 5 eigenvectors? Why not more?

Line 224: “…better performance over time? over time.” – This is a typo.

Lines 220-240: It seems the discussion focused on temperature plots only, but I didn’t see any comparison of temperature vs. precipitation results.

Citation: https://doi.org/10.5194/esd-2023-41-RC3
- AC1: 'Reply to all reviewer comments', Abigail Snyder, 16 Apr 2024
  
  The comment was uploaded in the form of a supplement: https://esd.copernicus.org/preprints/esd-2023-41/esd-2023-41-AC1-supplement.pdf
  
  Citation: https://doi.org/10.5194/esd-2023-41-AC1

Status: closed

RC1:
'Comment on esd-2023-41', Anonymous Referee #1, 09 Jan 2024

The study posits a strategy for selecting 5 CMIP6 GCMs that are suitable globally for impact model applications based on temperature and precipitation characteristics and the IPCC likely ECS range. The results would benefit from context, both in terms of how the study compares to previous model subselection exercises (why select the same set of models for all regions?) and in a deeper dive into the origins of the IPCC likely ECS range (where it comes from, what constraint assumptions are being made). Additionally, the methodology is hard to follow in the appendix and is worth moving to the main text. The primary issue I have, though, is that taking the "model uncertainty, scenario uncertainty, and interannual variability of the full CMIP6 ESM results" is inappropriate in an ensemble of opportunity like CMIP6 without a careful audit of model dependence.
Recommended Literature:
Abramowitz, G., Herger, N., Gutmann, E., Hammerling, D., Knutti, R., Leduc, M., Lorenz, R., Pincus, R., and Schmidt, G. A.: ESD Reviews: Model dependence in multi-model climate ensembles: weighting, sub-selection and out-of-sample testing, Earth Syst. Dynam., 10, 91–105, https://doi.org/10.5194/esd-10-91-2019, 2019.
Brands, S.: A circulation-based performance atlas of the CMIP5 and 6 models for regional climate studies in the Northern Hemisphere mid-to-high latitudes, Geosci. Model Dev., 15, 1375–1411, https://doi.org/10.5194/gmd-15-1375-2022, 2022.
Merrifield, A. L., Brunner, L., Lorenz, R., Humphrey, V., and Knutti, R.: Climate model Selection by Independence, Performance, and Spread (ClimSIPS v1.0.1) for regional applications, Geosci. Model Dev., 16, 4715–4747, https://doi.org/10.5194/gmd-16-4715-2023, 2023.
Specific Comments:
L60-62: "In a world unburdened by time and computing constraints, an impact model would take as input every projected data set available to have a full understanding of possible outcomes." - An ensemble of every projected data set in CMIP6 does not confer the full understanding of possible outcomes. It would include 50 initial condition ensemble members of certain ESMs and one ensemble member for others. Is the first 50x more likely to be true? Beyond the unequal voting power of the large ensembles in CMIP6, the ensemble contains a number of "hidden dependencies": models with different names but near-identical code. For uncertainty to mean "our full understanding of possible outcomes", model dependence must be handled properly.
L81-82: While this is an interesting objective, the size of the initial condition ensembles submitted to an exercise like CMIP is a function of computational resources and goodwill (they are submitting "free" data for others) on the side of the modeling centers. It is beneficial to many researchers that CMIP is inclusive and does not "pick favorites" thus encouraging participation.
Table 1: Of the 22 models you are using, 6 are connected, either by legacy or because they use a version, to NCAR's Community Atmosphere Model (CAM) development cycle. As this leaves the potential for CAM to have more influence on your uncertainty benchmark, the choice must be discussed. Additional similar models, such as ACCESS-CM2 / UKESM1-0-LL and MPI-ESM1-2- HR / MPI-ESM1-2- LR, present could be creating a “rather heterogeneous, clustered distribution, with families of closely related models lying close together but with significant voids in-between model clusters” (description of CMIP5 from Sanderson, B. M., Knutti, R., and Caldwell, P.: A representative democracy to reduce interdependency in a multimodel ensemble, J. Climate, 28, 5171–5194, https://doi.org/10.1175/JCLI-D-14-00362.1, 2015.). Could multiple similar models elevate outliers (e.g., MIROC) in your metric?
L166: Please justify the citation of Scafetta, 2022. See https://agupubs.onlinelibrary.wiley.com/doi/10.1029/2022GL102530
In step 3 of Table 2, how are you computing an ensemble average for the models that only provide a single run?
L224: Omit "over time?"
L315: "Models who more closely match the trend of observational data (W5E5v2.0 (Lange et al., 2021)) over the historic period will have their observations hold more weight. " Why? Trends are highly sensitive to internal variability, which is inherently random in temporal phase, i.e. no reason a model and observation should have the same sequence of it. A match in trend between observations and a model over a particular time period often occurs by chance and is not indicative model performance.
Deser, C., Phillips, A., Alexander, M. A., and Smoliak, B. V.: Projecting North American climate over the next 50 years: Uncertainty due to internal variability, J. Climate, 27, 2271–2296, https://doi.org/10.1175/JCLI-D-13-00451.1, 2014.

Citation: https://doi.org/10.5194/esd-2023-41-RC1
- AC2: 'Reply to all reviewer comments', Abigail Snyder, 16 Apr 2024
  
  The comment was uploaded in the form of a supplement: https://esd.copernicus.org/preprints/esd-2023-41/esd-2023-41-AC2-supplement.pdf
  
  Citation: https://doi.org/10.5194/esd-2023-41-AC2
- AC1: 'Reply to all reviewer comments', Abigail Snyder, 16 Apr 2024
  
  The comment was uploaded in the form of a supplement: https://esd.copernicus.org/preprints/esd-2023-41/esd-2023-41-AC1-supplement.pdf
  
  Citation: https://doi.org/10.5194/esd-2023-41-AC1
RC2:
'Comment on esd-2023-41', Anonymous Referee #2, 11 Jan 2024

Review of “Uncertainty-informed selection of CMIP6 Earth System Model subsets for use in multisectoral and impact models” by Snyder et al.
The presented study develops a model selection procedure that aims to preserve the distribution of total uncertainty as defined by Hawkins and Sutton. The motivation and need for some kind sub-selection of global climate models for impact models is clear and well-motivated by the authors. Their method is fairly straight-forward and is shown to work for the metric the authors use as validation.
I think this approach could be valuable contribution to be used as an objective selection criterion if the authors manage to manage to better describe the caveats and limits of this approach and better showcase the effect of the sub-selection. In its current state I can not recommend the manuscript for publication based on the major comments outlined in the following.

Major comments
I am very critical about the idea of optimizing a sub-selection method only for the distribution of the three sources of uncertainty. I note that the authors also manually select by ECS but this seems to be not really part of the method and is presented more as an afterthought.
In any case, such an approach opens up the risk of selecting highly dependent models (as seems in fact to be the case in the presented example with both ACCESS versions being part of the suggested sub-ensemble) as well as objectively bad models. Both are situations which should be avoided I would argue.
In fact, in a recent study Merrifield et al. (2023; 10.5194/gmd-16-4715-2023) set out to solve a similar problem but also consider model performance and independence. Their results should at least be compared to the results presented in this manuscript.
I have to admit I do not fully understand the role of the scenario uncertainty this manuscript. In an idealized case the scenario uncertainty as defined by HS is only dependent on the scenarios used. Why is it considered here, what is it relation to model selection? If it is sensitive to the selected (ECS preserving) model ensemble this is only a sign that the HS method is not properly able to isolate forced response, model uncertainty, and internal variability (as discussed in Lehner et al. 2020) is it not?
With respect to the verification metric, I am not sure if optimizing for the relative distribution of uncertainty is really ideal. At least the authors should consider also looking at the absolute position of their selected models in temperature-precipitation change space. I could imagine a situation where the relative distribution of uncertainties in the subset is very similar to the full ensemble without the subset being representative when considering, e.g., absolute changes.

Minor comments

line 33: “Scenario simulations from CMIP (most recently through ScenarioMIP, (O’Neill et al. 2016)”
closing braked missing

36: “Using such multi-model ensembles captures the process and structural uncertainties represented by sampling across ESMs, scenario uncertainty,”
Minor point but a multi model is not necessary to represent scenario uncertainty?
Similar statement also in the abstract: “In this work, we present a method to select a subset of the latest phase, CMIP6, models for use as inputs to a sectoral impact or multisectoral models, while still representing the range of model uncertainty, scenario uncertainty, and interannual variability of the full CMIP6 ESM results”

45: “For Earth system modelers, variability across ESMs’ projections of future climate variables can be significant (Hawkins and Sutton 2009; Hawkins and Sutton 2011; Lehner et al. 2020) and so the participation of multiple modeling centers running multiple scenarios is critical to understanding the future of the Earth system.”
I agree with this statement but “variability across ESMs” is a bit vague. Of the three sources of variability considered in the cited studies only one really needs a multi-model ensemble. I am also not sure if these are really the right studies to cite here as they mainly look at relative contributions of individual sources of uncertainty.

57: “However the total burden across modeling centers to sample across ESMs and scenarios still remains high, even with this potential efficiency.” I am not sure I understand the first part of this sentence?

72: “these competing priorities” what are the ‘competing priorities’ here?

Tab 1: I am a bit confused about the use of the word ESM in this manuscript. This table clearly also contains models versions with are not ESMs but GCMs? (ACCESS-CM2 for example)

108: “and the interannual standard deviation”
It is unclear to me how this is done or in which time-period.

114: At this point I am also wondering how initial-condition members are considered exactly? Is seems that the indices would be sensible to the number of ensemble members?

Fig 1: this figure seems unproportionally big for the amount of information it conveys.

Fig 3: I am not quite sure what information this figure is supposed to convey to the reader? In particular, since it is hardly discussed in the text. For example, is it somehow relevant that MPI or MRI (hard to distinguish) seem to behave quite different in PC2?

165: “Core Writing Team & (eds.),” strange citation
How many subsets fulfill this criterion?

Fig 4: Again I am not sure what I learn from this plot.

199: “To manage the inspection of three time series” Is ‘three time-series’ referring to Hawkins and Suttons model, forced, and internal components? This is not clear to me here. If yesI would note that in HS internal variability is not a timeseries but constant over time if memory serves.

223: “For temperature, we see that interannual variability is often performing well, with increasingly better performance over time? over time.” Typo

Citation: https://doi.org/10.5194/esd-2023-41-RC2
- AC1: 'Reply to all reviewer comments', Abigail Snyder, 16 Apr 2024
  
  The comment was uploaded in the form of a supplement: https://esd.copernicus.org/preprints/esd-2023-41/esd-2023-41-AC1-supplement.pdf
  
  Citation: https://doi.org/10.5194/esd-2023-41-AC1
RC3:
'Comment on esd-2023-41', Anonymous Referee #3, 13 Feb 2024

Review of esd-2023-41
Title: Uncertainty-informed selection of CMIP6 Earth System Model subsets for use in multisectoral and impact models
Authors: Abigail Snyder, Noah Prime, Claudia Tebaldi, Kalyn Dorheim
Overall Recommendation: Minor Revisions
This study, submitted to Earth System Dynamics, highlights a new approach to select subsets of ESMs (and potentially GCMs) for use in multi-sectoral and impacts modeling. The approach is novel and attempts to capture multiple sources of uncertainty in the subset and does have potential utility for multiple applications beyond what was mentioned in the manuscript. However, I do think the manuscript could be improved by added justification in a few places and addressing some particular aspects from prior literature that is relevant. In particular, the attempt at subset selection itself is not a new effort and the authors should place this manuscript in the context of prior attempts at subset selection. My general comments to improve the manuscript are below, followed by specific comments. I look forward to seeing the revised manuscript in print.

General Comments

There are a number of current issues in literature related to the selection of ESMs (or even just a set of climate projections) for use in applications that were not discussed here but are quite relevant.

First and foremost, the challenge of ensemble subset selection is not new. It would be worthwhile given the focus on using ESMs in impacts models to place this article in context with existing literature. For example, Parding et al (2020) produced the GCMeval tool, which is designed as an interactive tool for evaluation and selection of climate model ensembles for use in multiple applications. The methods in the GCMeval tool are relatively simplistic. In the context of this manuscript, I am left to wonder how different this approach is to the approach of Parding et al (2020) or other previous research. It would strengthen the manuscript to briefly discuss the differences between this approach and others in prior literature. Ideally, there should be some analytic comparisons between these approaches, but I believe a brief discussion of the other literature on this topic would suffice for this manuscript. In addition, the authors are addressing the “practitioner’s dilemma” (Barsugli et al. 2013). Though it was not mentioned by name, it may also be worthwhile to discuss how this approach builds on the literature associated with this well-known challenge.

Second, the approach in this manuscript may well retain “hot-models”, particularly as one goes to large subset sizes. The literature regarding if one should use hot-models is mixed, ranging from omitting hot models entirely (Hausfather et al. 2022), to down-weighting hot models based on ECS (Massoud et al. 2023), or simply keeping hot models as they may not have serious impacts on impacts modeling (Asenjan et al. 2023). It would be worthwhile for the authors to address how their method handles the “hot models” problem specifically, when their approach may indeed retain these models in their subsets.

Third, the focus is on impacts modeling with ESMs, but this is not the only use for ESM output. Depending on the needs of a stakeholder, one may not need additional modeling, but rather require complex variables derived from ESM output. A prime example of this is the use of spring phenological indices to determine projected changes in first leaf, first bloom, or the likelihood of a false spring (Gerst et al. 2020; Allstadt et al; 2015; Peterson and Abatzoglou, 2014). The approach in this manuscript could well be used in situations that don’t require additional modeling per se, but do require derivation from ESM or downscaled ESM output. Such literature should be briefly mentioned with the authors comment.

Fourth, the Hawkins and Sutton (2009) approach does have some valid criticisms. I suggest looking at the recent work of Lafferty and Sriver (2023), which also uses Hawkins and Sutton (2009) and the work of Wootten et al. (2017). The Lafferty and Sriver (2023) article addresses the critiques around the Hawkins and Sutton (2009) approach.

Finally, while the selection of ESMs is important, most impacts modelers do not use the ESMs directly, but use the downscaled ESM output (whether dynamically or statistically downscaled). The authors mention this briefly in passing, but it is important to acknowledge this in the conclusions also. Downscaling is itself another source of uncertainty, so it is a question of if this approach could also be applied with ESMs downscaled with multiple approaches.

Literature mentioned above:
Allstadt, A. J., S. J. Vavrus, P. J. Heglund, A. M. Pidgeon, W. E. Thogmartin, and V. C. Radeloff, 2015: Spring plant phenology and false springs in the conterminous US during the 21st century. Environmental Research Letters, 10, https://doi.org/10.1088/1748-9326/10/10/104008
Asenjan, M.R., F. Brissette, J.-L. Martel, and R. Arsenault, 2023: Understanding the influence of “hot” models in climate impact studies: a hydrological perspective. Hydrology and Earth System Sciences, 27, 4355-4367, DOI: 10.5194/hess-27-4355-2023
Barsugli, J., and Coauthors, 2013: The Practitioner’s Dilemma: How to Assess the Credibility of Downscaled Climate Projections. Eos Transactions, 94, 424–425, https://doi.org/10.1002/2013EO460005.
Gerst, K. L., T. M. Crimmins, E. E. Posthumus, A. H. Rosemartin, and M. D. Schwartz, 2020: How well do the spring indices predict phenological activity across plant species? Int J Biometeorol, 64, 889–901, https://doi.org/10.1007/s00484-020-01879-z.
Hausfather, Z., K. Marvel, G. A. Schmidt, J. W. Nielsen-Gammon, and M. Zelinka, 2022: Climate simulations: recognize the ‘hot model’ problem. Nature, 605, 26–29, https://doi.org/10.1038/d41586-022-01192-2.
Lafferty, D.C. and R.L. Sriver, 2023: Downscaling and bias-correction contribute considerable uncertainty to local climate projections in CMIP6. Nature Partner Journal – Climate and Atmospheric Science, 6, doi: https://doi.org/10.1038/s41612-023-00486-0
Parding, K. M., and Coauthors, 2020: GCMeval – An interactive tool for evaluation and selection of climate model ensembles. Climate Services, 18, 100167, https://doi.org/10.1016/j.cliser.2020.100167.
Peterson, A. G., and J. T. Abatzoglou, 2014: Observed changes in false springs over the contiguous United States. Geophysical Research Letters, 41, 3307–3314, https://doi.org/10.1002/2014GL061184.Received.
Wootten, A., A. Terando, B. J. Reich, R. P. Boyles, and F. Semazzi, 2017: Characterizing Sources of Uncertainty from Global Climate Models and Downscaling Techniques. Journal of Applied Meteorology and Climatology, 56, 3245–3262, https://doi.org/10.1175/JAMC-D-17-0087.1.

Minor Comments:

Line 84, first mention of Table 2: Table 2 defines the process for selection to aid the reader, yet it is positioned in the text far from the section. It’s also mentioned before Table 1 and Table 1 appears sooner in the text. I suggest reordering Table 2 and Table 1 and placing the renamed Table 2 earlier in the text to help the reader.

Line 105: “For each scenario and region in each ESM,…” – Am I correct that the averages calculated are across all the initializations of each ESM? Or is it the average across all models?

Lines 127-128: “Based on this figure,…explaining 71.8% of variance.” – Why only 5 eigenvectors? Why not more?

Line 224: “…better performance over time? over time.” – This is a typo.

Lines 220-240: It seems the discussion focused on temperature plots only, but I didn’t see any comparison of temperature vs. precipitation results.

Citation: https://doi.org/10.5194/esd-2023-41-RC3
- AC1: 'Reply to all reviewer comments', Abigail Snyder, 16 Apr 2024
  
  The comment was uploaded in the form of a supplement: https://esd.copernicus.org/preprints/esd-2023-41/esd-2023-41-AC1-supplement.pdf
  
  Citation: https://doi.org/10.5194/esd-2023-41-AC1

Abigail Snyder, Noah Prime, Claudia Tebaldi, and Kalyn Dorheim

Data sets

code and data for Uncertainty-informed selection of CMIP6 Earth System Model subsets for use in multisectoral and impact models Abigail Snyder and Noah Prime https://zenodo.org/records/10256736

Abigail Snyder, Noah Prime, Claudia Tebaldi, and Kalyn Dorheim

Viewed

Total article views: 615 (including HTML, PDF, and XML)

HTML	PDF	XML	Total	BibTeX	EndNote
455	120	40	615	33	31

HTML: 455
PDF: 120
XML: 40
Total: 615
BibTeX: 33
EndNote: 31

Views and downloads (calculated since 05 Jan 2024)

Month	HTML	PDF	XML	Total
Jan 2024	147	46	4	197
Feb 2024	108	19	10	137
Mar 2024	53	9	3	65
Apr 2024	35	13	10	58
May 2024	25	10	5	40
Jun 2024	21	11	2	34
Jul 2024	19	4	5	28
Aug 2024	22	2	1	25
Sep 2024	21	5	0	26
Oct 2024	4	1	0	5

Cumulative views and downloads (calculated since 05 Jan 2024)

Month	HTML	PDF	XML	Total
Jan 2024	147	46	4	197
Feb 2024	108	19	10	137
Mar 2024	53	9	3	65
Apr 2024	35	13	10	58
May 2024	25	10	5	40
Jun 2024	21	11	2	34
Jul 2024	19	4	5	28
Aug 2024	22	2	1	25
Sep 2024	21	5	0	26
Oct 2024	4	1	0	5

Viewed (geographical distribution)

Total article views: 606 (including HTML, PDF, and XML) Thereof 606 with geography defined and 0 with unknown origin.

Country	#	Views	%

Latest update: 04 Oct 2024

Chief editor

Earth System Models (ESMs) are used as inputs in impact models to estimate future climate risks. Hence, accurately representing the entire spectrum of uncertainty in ESMs is vital for comprehending the future co-evolution of the coupled human-natural system. Numerous ESMs are part of the CMIP6 suite and it is impossible to incorporate all of them in impact modeling. This study will help in selecting a suitable subset of ESMs that will reduce computing costs while preserving the range of uncertainty.

Short summary

From running Earth system models (ESMs) to using their outputs to identify impacts, modeling the integrated human-Earth system is expensive. This work presents a method to identify a smaller subset of ESMs from the full set that preserves the uncertainty characteristics of the full set. This results in a smaller number of runs that an impact modeler can use to assess how uncertainty propagates from the Earth to the human system while still capturing the range of outcomes provided by ESMs.


Total:	0
HTML:	0
PDF:	0
XML:	0