All datasets considered in this study are pre-processed in an analogous manner to enable a direct comparison. First, all data are bilinear interpolated to the grid used for the E-OBS data (0.25∘ × 0.25∘ resolution). At each grid point, monthly anomaly time series are computed by subtracting the long-term means for the period 1961–2010 from the interpolated raw datasets. Finally, annual values are derived and multi-annual means for lead years 1–5 are built for further evaluation.

Following the suggestion of Goddard et al. (2013), the skill analysis is partly performed for spatial means. Spatial averaging of the anomaly time series is performed for eight PRUDENCE regions over Europe (see Fig. 1; Christensen and Christensen, 2007). Note that we only used grid points over land surfaces for the spatial means as E-OBS data are not available over the oceans. Additionally, we calculated the predictive skill on the basis of all individual grid points for specific analysis.

The following metrics are used to evaluate the performance of the global and regional hindcast ensembles and to address the four key questions: the mean squared error skill score (MSESS) and the anomaly correlation coefficient (ACC), which are both measures for the forecast accuracy. The skill metrics are applied to the pre-processed time series described in Sect. 3.1 and are computed for multi-annual means for lead time years 1–5 after initialization. Recent studies analysing the MiKlip decadal prediction system demonstrated that the MiKlip ensemble performs best for the first years after initialization for a wide range of variables, while the skill diminishes for longer forecast periods. For example, Müller et al. (2012) found highest skill scores for years 1–4 and 2–5 for annual mean surface temperature for both the North Atlantic region and global means. The same is true for annual wind speed and wind energy potentials over central Europe, for which skilful predictions are mainly restricted to the first years after initialization (years 1–4), while negative skill scores are found for longer lead time periods (Moemken et al., 2016). Kruschke et al. (2014) provided evidence that the prediction skill for winter cyclones over the North Atlantic region is best for years 2–5 and reduced for longer time periods. Following the recommendation by Goddard et al. (2013), we focus, in the following, on the lead time years 1–5 after initialization.

The deterministic MSESS (Murphy, 1988) is defined as MSESS(HRO)=1-MSEhindMSEref with MSEhind=1N∑i=1NHi‾-Oi2andMSEref=1N∑i=1NRi‾-Oi2, where i=1, …, N is the time index, MSEhind is the mean squared error (MSE) between the ensemble mean of the dynamical downscaled hindcasts (Hi) and the verification data (Oi), and MSEref is the mean squared error of a reference dataset (Ri). In this study, the uninitialized historical simulations, the climatology, or the global initialized hindcasts are used as reference. A MSESS could be between minus infinity and one, with positive MSESS meaning that the hindcasts are closer to the verification dataset than the reference, indicating that the initialization and downscaling lead to higher accuracy in predicting observed values.

Following Murphy (1988) and given that anomalies are used (as in this study), the MSESS with the climatology as reference (i.e. Ri≡O‾) can be decomposed as follows: MSESS(H,O‾,O)=rH,O2-rH,O-SHSO2, where rH,O is the correlation between the hindcasts (Hi) and the verification data (Oi), and SH and SO are the sample variances of the simulation ensembles and the observations, respectively. The second term is the conditional bias. CB=rH,O-SHSO. In the case that another reference is applied, the decomposition is as follows: MSESS(H,R,O)=rH,O2-rH,O-SHSO2-rR,O2+rR,O-SRSO21-rR,O2+rR,O-SRSO2 or MSESS(H,R,O)=MSESS(H,O‾,O)-MSESS(R,O‾,O)1-MSESS(R,O‾,O), where MSESS(H, R, O) denotes the skill score of a hindcast H to a reference R given the observations O. MSESS(H, O‾, O) and MSESS(R, O‾, O) are the respective skill scores against the climatology.

Hence, MSESS depends on the conditional bias, as well as on the correlation. It is smaller than the correlation in the case that there is a conditional bias, for which the optimal value is 0. CB depends on the balance between correlation and the ratio of the standard deviation between the ensemble and the observation. The improvement or added value of CB is calculated according to Kadow et al. (2015) as CBAV=rref,obs-SrefSO-rhind,obs-ShindSO. The correlation or ACC (anomaly correlation coefficient; e.g. Wilks, 2011) is computed as the Pearson correlation between the ensemble mean of the hindcasts at a certain location i and the corresponding observations (Obs): ACCi=1N∑tHtOtSHSO, where t=1, …, N is the time index. The ACC quantifies the accuracy of the predictions only in terms of the temporal course, while it is independent from the variance of the target variable and from the mean bias. To compare the performance of the hindcasts and of the uninitialized historical runs, we compute the difference of the ACC of the hindcasts minus the ACC of the historical runs for several issues (hereafter delta_ACC).

The significance of the skill scores is determined by using a bootstrapping approach at the 95 % level (Kadow et al., 2015) and a t test of the respective distributions.

In this section, we address the first key question and analyse the general potential for skilful regional decadal predictions over Europe. Figure 2 shows MSESS plots for temperature, precipitation, and surface wind speed in CCLM_b0 and CCLM_b1 with the climatology as reference. Not surprisingly, the MSESS is positive for most of Europe for temperature. It is significant for most regions in western Europe in CCLM_b0 and large parts of southern Europe in CCLM_b1 (Fig. 2a and b). This is due to the strong positive trend in the observed temperature, which is predicted by the hindcasts but not captured by the climatology. Deviations between both ensembles are larger for precipitation (Fig. 2c and d), where the MSESS fields are distinctly patchier when compared to temperature (Fig. 2a and b). While positive and significant skill scores are found over large parts of western Europe in CCLM_b1 (Fig. 2d), MSESS values are mostly negative over this region in CCLM_b0 (Fig. 2c). For wind speed (Fig. 2e and f), a positive MSESS is found only for northern Europe and CCLM_b0, where skill scores are often significant. As at the same time negative skill scores are found for other European regions in both ensembles, the climatology is generally closer to the observations than the hindcasts. In this respect, we have analysed the respective spatial mean wind speed time series for the Iberian Peninsula (Prudence region 2) and CCLM_b0. The wind speed shows a slight negative trend in CCLM_b0, while the trend is slightly positive for the observational dataset (not shown). At the same time, the decadal variability for wind speed is quite small over this region in all datasets (it ranges from 0.02 to -0.02 in CCLM_b0 and E-OBS). Hence, the deviation of the climatology from the observations, and thus its MSE, is generally small in this region, resulting in a negative MSESS when using the climatology as reference (see also Eq. 3 for MSESS in Sect. 3.2).

A different picture is revealed when using the uninitialized historical runs as reference dataset for the MSESS computation (Fig. 3). For temperature (Fig. 3a and b), positive skill scores are found in both ensembles over Scandinavia and for southeastern Europe, and at some grid points this prediction skill is significant. A stripe of negative values occurs over the British Isles and central Europe. The analysis of the time series for Mid-Europe (spatial mean over Prudence region 4) reveals that this negative skill mainly results from a strong temperature increase from dec1960 to dec1970 in the observations, while CCLM_b0 and CCLM_b1 depict a decrease in temperature (not shown), which in fact was observed in southern Europe for instance. As a consequence, the temperature in the hindcasts has larger deviations than the uninitialized simulations compared to the observations during the first half of the considered period, but agree well to the observations from dec1980 onwards. The largest deviations between CCLM_b0 and CCLM_b1 are found for Iberia, parts of southern France, and Italy, where the MSESS is positive for CCLM_b1 but neutral to negative for CCLM_b0.

Again, deviations in the MSESS fields between both ensembles are larger for precipitation (Fig. 3c and d), reflecting the local character of rainfall. Both ensembles show positive and partly significant MSESS values for regions in Scandinavia and eastern Europe, and to a lesser extent for Iberia and the British Isles (Fig. 3c and d). In CCLM_b1, predictive skill is also identified over western central Europe. Thus for CCLM_b1 positive skill is found for larger areas indicating an added value of the improved initialization procedure in baseline1 compared to baseline0.

Regarding wind speed, the predictive skill in CCLM_b0 (Fig. 3e) shows high and significant MSESS values over Scandinavia, Iberia, southern Italy, and along the coasts of the North Sea and Baltic Sea, while negative values are found, for example, over parts of France, southern Germany, and the Alpine region. In CCLM_b1, the MSESS depicts positive values over most of western and central Europe, while negative values are now identified along the eastern coast of the Baltic Sea (Fig. 3f). Overall the predictive skill of CCLM_b0 is slightly higher and affects a larger area, indicating that the changes in the initialization method do not improve the results for wind speed.

We conclude that in terms of the MSESS accuracy there generally is a potential for skilful decadal predictions over Europe in the regional MiKlip ensembles. However, the skill pattern depends on the region and the variable. For individual regions, the initialization leads to an added value for accurate (retrospective) forecasts several years ahead, while for some regions the uninitialized historical runs deliver better predictions. Also the discrepancies between the two hindcast generations (CCLM_b0 and CCLM_b1) are rather heterogeneous. While for temperature we only found a slight shift in the pattern due to the different initialization methods, discrepancies can be large for precipitation and wind speed depending on the region.

Recent studies document that the application of regional climate models may improve climate simulations, in particular over complex terrain (Berg et al., 2013; Feldmann et al., 2013; Hackenbruch et al., 2016). This is mainly due to a more realistic representation of the topography (e.g. mountain ranges or coastlines) in the regional climate models (RCMs) compared to global-scale general circulation models (GCMs). In this section, we analyse whether the downscaling with a regional climate model also leads to an added value for decadal predictions over Europe. With this aim, we use MPI_b0 and MPI_b1 as reference datasets for the MSESS shown in Fig. 4 (see also Sect. 3.2).

Generally, significant improvements in the prediction skill by dynamical downscaling are restricted to limited geographical areas, and they strongly depend on the variable. For temperature a significant added value of downscaling is found over Scandinavia and southeast Europe in CCLM_b0, and over southeast Europe and the British Isles in CCLM_b1 (Fig. 4a and b). Therefore, the regionalization typically provides an improvement in regions where the global hindcasts show mostly medium to lower skill. In some regions with already high skill in the MPI-ESM hindcasts there is no improvement by the downscaling. This includes, for example, an area from the British Isles over France to Germany in baseline0 and regions along the western Mediterranean coast in baseline1. However, the global model outperforms the regional model over large parts of northwest Europe in the baseline0 ensemble (see also Table 1).

Again, rather patchy MSESS fields are obtained for precipitation. Nevertheless, there are several regions with significantly improved prediction skills in the CCLM ensembles compared to MPI_b0 and MPI_b1 (Fig. 4c and d). For example, both CCLM_b0 and CCLM_b1 reveal significant positive MSESS over eastern Germany and over parts of Scandinavia.

The added value of downscaling is most pronounced for wind in CCLM_b0 (Fig. 4e). Significant improvements of the MSESS are detected for southeast Europe, Italy, Scandinavia, and for coastal areas of Iberia, France, and England. Areas with an added value of downscaling are also existent in CCLM_b1 (Fig. 4f), but visibly reduced compared to CCLM_b0.

In Tables 1–3 we summarize the analysis of skill (cf. Fig. 2) and added value (cf. Fig. 4) of the regional hindcasts compared to the climatology for the three variables as spatial means over the PRUDENCE regions (cf. Fig. 1). The tables display the MSESS as well as its components correlation (ACC) and conditional bias (CB), according to the Murphy decomposition (cf. Sect. 3.2). The formatting of the cells in the left half of the tables display distinctly positive (bold numbers), negative (italic numbers), or slightly positive skill (no formatting) derived from the results shown in Fig. 2 for the MSESS. A common threshold for all three variables is chosen in a way that above this level a skill score is regarded as significant by the bootstrapping procedure. The thresholds for the formatting are above (bold) or below (italic) ±0.3 for the MSESS, and ±0.4 for ACC. Since the optimal value for the conditional bias is zero, a low CB is depicted for the absolute value, |CB|, being below 0.2 (bold), and a high CB for |CB| being larger than 0.3 (italic). The formatting for the added value in the right half of the tables corresponds to the sign of the skill difference (dynamical downscaled minus global MPI; bold: distinctly positive; italic: distinctly negative) in a region. The threshold is 5 % (±0.05) and chosen in an analogous way to the left half of the tables.

For temperature (Table 1), the MSESS and ACC are high in all regions except Scandinavia (SC). The correlation is above 0.8 in CCLM_b0 for the northwestern part of Europe – namely the British Isles (BI), France (FR), Mid-Europe (ME), and the Alps (AL). For CCLM_b1 the highest correlation is found more in the southern part of the region, namely in France (FR), the Iberian Peninsula (IP), the Alps (AL), and the Mediterranean (MD). The CB is low for most areas, except France (FR) and Scandinavia (SC) in CCLM_b0. The MSESS is high due to the high ACC and the low CB. MSESS and ACC are higher and CB lower for b1 than for b0 over the whole of Europe (EU). A significant added value of the regionalization of baseline0 occurs for Scandinavia (SC) and Eastern Europe (EA) as well as for Europe (EU). In these regions CB is significantly lower for CCLM compared to the MPI-ESM hindcasts. CCLM_b1 shows an added value for the regions British Isles (BI), the Mediterranean (MD) and Eastern Europe (EA) as well as over the entire domain (EU). The main cause is again a reduced conditional bias.

The MSESS for precipitation (Table 2) is much lower than for temperature. It is mostly negative for CCLM_b0 and only slightly positive for some regions in CCLM_b1. For CCLM_b0 a significant positive correlation is only found for the Iberian Peninsula (IP). The CB is generally negative (except for IP), which is inherited from the global hindcasts, since for MPI_b0 CB is even lower than for CCLM_b0. Over the whole domain the added value is low and not significant. For CCLM_b1, a positive ACC is uncovered for France (FR), Mid-Europe (ME) and Scandinavia (SC). The conditional bias is much lower than for CCLM_b0 and only slightly negative in most areas. An added value in relation to MPI_b1 occurs in those regions where ACC of CCLM_b1 is positive. There is also an added value for the whole domain (EU), but it is only significant for ACC.

The highest ACC for the 10 m wind speed is revealed for northern Europe (BI and SC) as well as in the Mediterranean (MD) for CCLM_b0, and in the southern regions for CCLM_b1 (Table 3). These are all regions with low to moderate conditional bias. CB is strongly negative in regions with a low skill. Interestingly, the added value for the MSESS of CCLM_b0 is significant in all regions, and for most regions a significant improvement is also visible with respect to ACC and CB. For CCLM_b1, on the other hand, significant improvements of MSESS and ACC compared to MPI_b1 are only found for a few regions.

We conclude that regional downscaling may indeed provide an added value for decadal predictions over Europe, both for individual grid points as well as for spatial means. However, this added value is not systematic but depends on variable and region.

Past studies suggest that the ensemble size of a prediction system has an impact on the forecast skill of a model (Richardson, 2001; Ferro et al., 2008). Generally, there is a consensus that the prediction skill for both seasonal and decadal predictions is enhanced when the number of ensemble members is increased. Kadow et al. (2015) analysed the global MiKlip baseline1 generation and concluded that the forecast accuracy for surface temperature for lead years 1 and 2–9 is improved for nearly the whole globe when the ensemble size is increased from 3 to 10 members. This is in line with the findings of Sienz et al. (2016) who examined the prediction skill for North Atlantic sea surface temperatures in the same hindcast ensemble. Also for seasonal predictions of the North Atlantic Oscillation a forecast system profits from increasing size (e.g. Scaife et al., 2014). However, the ensemble-size-dependent skill bias has never been demonstrated based on regional decadal climate predictions before. With this aim, we analyse the impact of the ensemble size on the predictive skill for the eight PRUDENCE regions in Europe in both the regional and the global MiKlip ensembles. In the following, results are only shown for the Iberian Peninsula (IP), as the findings are similar for the other PRUDENCE regions. Figure 5 exhibits the dependency of MSESS and delta_ACC when compared to the historical simulations for lead years 1–5 (y axis) on the ensemble size (x axis) for all three variables spatially averaged over IP. For each ensemble size n (n varying between 2 and 10), the solid coloured lines depict the averaged skill scores for all permutations of n-member ensemble combinations for each of the four individual hindcast ensembles (MPI_b0, MPI_b1, CCLM_b0, and CCLM_b1). Note that for the reference dataset always the full ensemble of 10 members of the uninitialized historical runs is used, independently of the ensemble size of the initialized hindcasts.

As expected, enhanced predictive skill can be observed when the number of members is increased stepwise for both the global and the regional hindcast ensembles. MSESS shows a rather logarithmic relationship with increasing n, depicting the highest skill scores for the 10 member ensembles for all three variables (Fig. 5a–c). On the other hand, the lowest skill scores are always found for the 2-member ensembles. This ensemble size dependency of MSESS is systematic and is detected in both hindcast generations for all variables over all eight PRUDENCE regions (not shown), regardless of whether the skill scores are negative or positive. In some cases, the ensemble size increase even leads to a shift from negative MSESS values to positive values in one or more of the ensembles (e.g. Fig. 5a and c). In contrast, no systematic conclusion can be stated for the delta_ACC, as the ensemble size dependency of the predictive skill depends on the variable and the considered MiKlip ensemble (Fig. 5d–f). Nevertheless, there are also examples for delta_ACC where the ensemble size dependency is similar to that of MSESS, e.g. for temperature (Fig. 5d). These results suggest that a regional decadal prediction system generally benefits from larger ensemble sizes, either in terms of more skilful decadal forecasts or at least of a reduction of the bias or the uncertainty, depending on the variable and the hindcast generation. Note that for most variables and skill scores the hindcast generation is more important for the skill than the resolution. In addition, most diagrams indicate an added value of downscaling. For temperature and wind speed, both generations of CCLM surpass their MPI counterparts for both skill scores, indicating a systematic added value of downscaling. This is particularly visible for wind in the b0 ensemble, where the prediction skill of CCLM is distinctly better than for MPI-ESM-LR (Fig. 5c and f). This is mainly due to higher skill scores over orographic structured terrains of IP in CLM_b0 compared to MPI_b0 (cf. Fig. 4).

For ensembles with less than 10 members, the skill scores of all possible n-member ensemble combinations are averaged. For selected ensembles, box and whisker plots of these n-member combinations are shown for delta_ACC in Fig. 5d–f. Given that we are doing permutations without replacement, the spread between the individual n-member ensembles declines with an increasing number of members n, and this decline should therefore not be over-interpreted. Nevertheless, the spread is quite large not only for small ensemble sizes but also for ensembles with n>5. For instance, delta_ACC for wind in CCLM_b0 (MPI_b0) varies between 0 and +1.6 (-0.1 and +1.1) for the 2-member ensembles (Fig. 5f). Even for the 7-member ensemble, results can differ quite strongly depending on the selection of the ensemble members. Similar results are found for temperature and precipitation. These findings clearly demonstrate the necessity of using large ensembles to reduce uncertainties. Furthermore, only for high numbers of ensemble members (eight or more), the delta_ACC curve for CCLM_b0 is above the range of uncertainty in MPI_b0 in the case of precipitation (Fig. 5e) and wind (Fig. 5f). This indicates that the prediction skill may only be significantly improved when the whole ensemble is dynamically downscaled. The same applies for the improvement from baseline0 to baseline1 in the case of temperature (Fig. 5d).

In summary, our findings confirm what is already verified in recent studies for global decadal predictions: the predictive skill of a regional decadal prediction system is generally improved when the size of the hindcast ensembles increases. This is valid for all variables, regions, and hindcast ensembles considered in this study. The skill scores converge towards a certain value in most cases for MSESS in all hindcasts (see Fig. 5a–c). The increments in added value by increasing the number of ensemble members decrease for more than 5 members. Nevertheless, it is recommended to use 10 members or more for the skill assessment of decadal predictions on the regional scale, as is also endorsed in the Decadal Climate Prediction Project (DCPP) contribution to CMIP6 (Boer et al., 2016).

A lesson learned from the CMIP5 decadal experiments is that more starting years and thus a larger number of initializations is beneficial to establish robust skill estimates (Boer et al., 2016). This has been reflected in the progress from the first global MiKlip hindcast generation baseline0 to the second generation baseline1. Whereas baseline0 provides 10 ensemble members every fifth year (compliant with the CMIP5 experimental protocol), baseline1 provides this ensemble size for each starting year of the hindcast period. To assess the impact of using only five initializations (as used elsewhere in the paper) on the robustness of our main conclusions we performed a sensitivity analysis with the global baseline1 ensemble, for which all starting years are available. For this, we compared the sample with ten-yearly starting dates with the full yearly initialized MPI-ESM-LR baseline1 ensemble over the same period from 1960 to 2000.

Figure 6 presents a comparison between the ACC scores for the sample with five initializations (Fig. 6a, c, e) and the sample with all 41 initializations (Fig. 6b, d, f). For all three variables the score maps show in general comparable spatial distributions. The skill maps for the sample with all initializations usually depict a smoother spatial distribution with less extreme skill values and larger areas with significant skill scores. The regional averages over most of the PRUDENCE regions are comparable. However, in some regions larger differences can occur: for temperature over Ireland and Scotland, for precipitation over parts of France and eastern Europe, and for wind from northeastern Spain towards the Alps. Similar results are found for MSESS (not shown) for which not only the number of initializations of MPI_b1 is increased but also the number of uninitialized historical runs.

It is obvious that more initializations increase the robustness of the skill assessment, especially with respect to quantitative estimates and significance of the results. Therefore, this work supports the recommendations made for CMIP6 by Boer et al. (2016) to generate hindcast ensembles with yearly starting dates. Nevertheless, using only five initializations already represents the general features and to some extent the significance of the regional distribution. Therefore, the analysis of the sample with all initializations confirms the qualitative findings from Sect. 4.1. The results regarding the added value and the ensemble size dependence are less affected by the number of initializations. Given the above findings, we conclude that the results obtained here for a limited number of initializations qualitatively comparable to those which would be obtained for samples with distinctly more initializations.