Review of Massoud et al. entitled “Seasonal forecasting skill for the High Mountain Asia region in the Goddard Earth Observing System”

 

General comments:

This manuscript evaluates the seasonal forecast skill for hydrometeorology over the High Mountain Asia (HMA) region from the Goddard Earth Observing System (GEOS). As the author suggests, “S2S forecasting for HMA is in its infancy”. Their results show that the GEOS-S2S system’s ability to forecast HMA hydrometeorology on the seasonal timescale is limited. The authors raise some issues of the GEOS for seasonal hydrometeorological forecasts. These results help to improve the ability of the seasonal forecast model in the future. Therefore, the scientific questions are of interest. Their introduction provides context and objectives for their work, which catches the reader's interest. The data and methods are described in detail and are reasonable. The results of the evaluation and discussion are well written. In general, this paper is well prepared and fits the scope of ESD.

Specific comments:

1. The credibility of verification data is a great challenge. Even though the authors use multiple data, I still think the current results are quite uncertain due to credibility of verification data.

2. The S2S (subseasonal to seasonal) prediction project database (http://www.s2sprediction.net/) provides reforecasts by many operational forecast systems. Can the forecasting skills of the GEOS be compared with models that participate in the S2S prediction project?

3. It appears that the area of focus includes some low-elevation areas within the range of Figure 1 (e.g., parts of India). Are the authors calculating some statistics (e.g., Figure 2, Table 2) for the entire area of Figure 1? Should low-altitude areas be masked out?

4. Section 3 describes the results in detail. However, there seems to be a lack of an in-depth scientific explanation. For example, what are sources and effects of the forecast errors.

5. I noticed that the skills of GEOS vs. MERRA-2 and observation are quite different (Figure 2a vs. 2b). The ubRMSEs in Figures 5d and 6d show the issue. Which result should I believe? Why are there such obvious differences in skill when using different verification data (especially SM, TWS)? How do the authors interpret such differences of results?

6. High anomaly correlation or low ubRMSE indicates better forecasting skills. Both the anomaly correlation and ubRMSE represent the correspondence between forecasts and observations. It looks like it is acceptable to use just one metrics. Why use both anomaly correlation and low ubRMSE?

7. Section 3.2 and Figure 4: It appears that the annual cycles have large uncertainties, mainly hydrological variables. The anomalies are derived by removing the annual cycle. This might greatly affect the credibility of the results. How does the author address this issue? There should be an explanation.

8. The reviewer did not get the point of Figure 3. This figure depicts the difference in skill between variables and between forecast lead times. Different variables have different predictability. Forecast skill decreases with forecast lead time as a matter of course. What is the purpose of comparing their relative skills?

Minor comments:

Line 23 and 25: "ranges" shoud be "range".

Line 118: “…five mountain ranges, including the Himalayas, Inner Tibetan Plateau, Karakoram, and Hindu Kush.” Shoud be “four”?

 Summary

This study evaluates the subseasonal predictions from the NASA GEOS5-S2S hindcasts for 1981-2016 over the High Mountain Asia (HMA) domain with a focus on a set of hydrometeorological variables including 2-m air temperature, precipitation, snow cover fraction, snow water equivalent, soil moisture, and total water storage.  The evaluation was done against two reanalyses and other independent datasets, and the evaluation focuses on monthly time scale with lead times up to 3 months. Unbiased root mean square error and anomaly correlation are the major metrics used in this evaluation.

Overall, the study provides useful information about the predictive skill of the NASA GEOS5-S2S hindcast over the HMA region.  The manuscript is well written and easy to understand, and the quality of the visualizations is generally good. However, the study falls short in several important aspect regarding forecast verification at subseasonal to seasonal time scales.  The value and the contribution of this study to our understanding about predictability of the climate system at S2S time scale is very limited. I believe at the minimum a major revision is needed. I list my major concerns and some specific comments below. 

Major issues

- For prediction beyond the typical weather scale (i.e, 1-2 weeks), probabilistic forecast is more appropriate and useful than deterministic forecast given the chaotic nature of the climate system, which is why S2S forecast with numerical models needs to produce ensemble predictions.  In this study, only unbiased root mean square error (ubRMSE) and the anomaly correlation (ACC) of the ensemble mean were used, which is useful but only shows very limited aspects of the forecast quality.  There are many verification metrics that can be used for ensemble predictions such as those listed at https://www.cawcr.gov.au/projects/verification/#Methods_for_probabilistic_forecasts. I'd highly recommend that a few more meaningful metrics are included in this study. 

- My biggest concern regarding the analysis is its over-simplified approach to deal with the spatial heterogeneity within the study domain. The study domain is quite large; more importantly, it is very heterogeneous with distinct climates and land surface characteristics including elevation, land cover type, etc.  As shown in Figures 7-12, temperature, precipitation and other hydrometeorological variables and model's skill in predicting these quantities can vary drastically across the domain. Spatially averaging them across high mountain ranges, the Tibet Plateau, Taklamakan desert, and the Indian subcontinent does not make much sense, and evaluating the spatially averaged quantities is not very meaningful and insightful. It is not clear what these spatial averages physically mean and how verification at such a level can help us to understand the model deficiency in a meaningful way. Although Figure 7-12 highlight the spatial heteorogeneity, the evaluation is only limited to the ensemble mean, spread, and ubRMSE. I'd suggest that the authors divide the domain into multiple smaller regions that are more homogeneous or multiple watersheds where the spatial averages are more meaningful, and conduct the forecast verification of these regional quantities using multiple metrics (probabilistic and deterministic).

Minor issues

- line 13:  "where water resource needs change depending on ..."  although this sentence is correct, it could be a little confusing as either "needs" or "change" can be interpreted as the verb, thus resulting in different meanings.

- line 13: how is intensity of the hydrological cycle defined?  It was not mentioned in the study.

- line 30: "a range of factors",  the predictability itself is also an important factor.

- line 34: remove the comma before "where"

- line 35-36:  This sentence reads a little awkward, please consider rephrase.

- line 40: Part of the study domain is heavily populated, but the majority of HMA do not have much population, such as Tibet Plateau and dessert. 

- line 43: The term "water tower" of the Earth have been used for many years among researchers in Asia, so some earlier literature needs to be cited here to be more appropriate.

- line 144: This is only over the real-time forecast period, isn't it? Please clarify that these 6 additional members are not available in the hindcast period and thus not used in the evaluation.

- line 147: "a long period for forecast validation"  "validation" and "verification" are different terms although they are related.  One can verify if a forecast is correct or wrong, but you cannot validate a forecast when the forecast is wrong. So it would be more appropriate to say "forecast verification" or "forecast evaluation" here.

- line 233: remove "in our evaluation" as it is redundant with "in this study" at the beginning of the sentence.

- line 234-235: Does this mean the dataset is heterogeneous in space and time? If that is the case, how does this affect the evaluation?  Please explain.

- line 282: It would be useful to give the equation for R_anom too.  Does this includes both space and time dimensions? 

- line 291-292: Is the ensemble spread also lead-time dependent? 

- Section 3.2: Since the evaluation metrics are based on anomalies, what purpose does this section serve in the paper? 

- Figure 4 and others:  Since the gridded model forecast is spatially averaged over the large domain with different masks for different variables, it would be useful to show the masks for these variables in Figure 1 so that readers know how the spatial average is calculated.

- Section 3.3: This section is about the absolute error.  Because of the seasonality discussed in the previous section, it is not surprising that errors are generally larger during the season when the absolute value of variable is also large.  So it will be necessary and more informative to discuss the relative errors beyond the absolute error.

- Figure 6: For each panel,the y-axis should be set to the same range as that in the corresponding panel in Figure 5.

- Line 489-490: The results in this study do not seem to back up this statement.

- Line 518-520: This statement is speculative.  It would be more appropriate to provide justifications.

- Line 529: How is 4% cold bias calculated? Using different units such as Kelvin, Celsius will certainly result in different percentage change?  So a statement like this does not make much sense. 

- Line 603-604: This statement assumes that the ensemble spread of the forecast is informative.  The assumption may or may not be true. Linking a smaller forecast spread with higher skill is unjustified and questionable.

- Line 633-634: It is not clear how this study achieve this as it does not provide much insights that can guide model improvements.

This paper examines the seasonal prediction skill of the NASA’s Goddard Earth Observing System (GEOS) S2S prediction system over High Mountain Asia for a series of hydrological variables. A set of observational data and the MERRA-2 and ERA-5 re-analyses are used as benchmarks.

Monthly means of reforecasts over the period 1981 to 2016 are analysed.

The paper provides a clear and well written description of the system and presents a clear analysis the forecasts of these hydrological variables. It provides a solid detailed reference to further assess more specific aspects of the performance of the GEOS S2S prediction system over HMA. The Figures are clear but, while many detailed descriptions are provided, I find the paper a little thin on new science and on understanding potential sources of predictability. At the very least, some more discussion points should be included.

MAIN COMMENTS

1) The skills over this large, heterogenous region are small, especially when measured against independent data. I wonder if such small skills are relevant at all. The prediction system may have higher skill in some variables, in more limited domains or at certain times of year.

Low skill is not unexpected given the large area, the varied topography (from lowlands to high mountains) and land cover, and the different regional climates of this large region. For example, only a small Southeastern part of the Tibetan Plateau (and hence, only parts of the HMA) is influenced by the Indian summer monsoon (ISM).

It is well-known that the skills depend on the verification data, and they would be higher, in most cases, when verified MERRA-2 given the parent model. It would be of interest to verify some variables against several datasets, or better, against merged datasets that take into account uncertainties in the various observations.  I realise that such merged dataset might not exist over this region, but the point could be mentioned in the Discussion.

2)Concerning actual societal needs, the reliability of such forecasts is a question of utmost importance that needs to be addressed in a probabilistic context.  Is it possible to quantify the reliability of the forecasts with the current system using standard metrics? At least, the outlook could be mentioned in the Discussion.

3)I wonder about the relationship between surface temperature and the snowpack. Is there is a strong coupling between the two in the forecasts during some months? This could provide a source of skill.

4) Improved prediction of the circulation could lead to improved skill. The authors mention the importance of the ISM. I believe that wintertime precipitation over the northern part of HMA is brought by the so-called westerly disturbances.  The authors could mention in the Discussion, whether the dynamics and the associated with precipitation is well represented in the forecast.

5)There has been a significant effort in recent years to assess the impact of land initialisation (esp. snow, soil moisture) in S2S and seasonal forecasts and some studies are relevant for the HMA region  yet there is little mention of that relevant literature.

Koster, R. D., Mahanama, S. P. P., Yamada, T. J., Balsamo, G., Berg, A. A., Boisserie, M., et al. (2011). GLACE2: The second phase of the global land atmosphere coupling experiment: Soil moisture contribution to subseasonal forecast skill. Journal of Hydrometeorology,12(5), 805–822.

Senan, R., Orsolini, Y.J., Weisheimer, A. et al. Impact of springtime Himalayan–Tibetan Plateau snowpack on the onset of the Indian summer monsoon in coupled seasonal forecasts. Clim Dyn 47, 2709–2725 (2016). https://doi.org/10.1007/s00382-016-2993-y

MINOR COMMENTS

• The words Seasonal forecasts and S2S forecasts seem to be used loosely throughout the paper. The seasonal forecasts are 9-month long but only the first 3 months are analysed. Some operational centers have different set-ups for Seasonal and S2S prediction systems. The authors could double check that S2S is used as it is meant.
• It was not clear to me whether total precipitation is liquid precipitation or if it contains also solid precipitation.
• The information on ensemble size should be presented more clearly (Abstract, or Table)

Wording

 L58: the foothills of the Himalayas perhaps better than the foot of the Himalayas (?)

 L560: precipitation is used twice in same sentence.