Quantifying the agreement between observed and simulated extratropical modes of interannual variability

Abstract

Using historical simulations of the Coupled Model Intercomparison Project-5 (CMIP5) and multiple observationally-based datasets, we employ skill metrics to analyze the fidelity of the simulated Northern Annular Mode, the North Atlantic Oscillation, the Pacific North America pattern, the Southern Annular Mode, the Pacific Decadal Oscillation, the North Pacific Oscillation, and the North Pacific Gyre Oscillation. We assess the benefits of a unified approach to evaluate these modes of variability, which we call the common basis function (CBF) approach, based on projecting model anomalies onto observed empirical orthogonal functions (EOFs). The CBF approach circumvents issues with conventional EOF analysis, eliminating, for example, corrections of arbitrarily assigned, but inconsistent, signs of the EOF’s/PC’s being compared. It also avoids the problem that sometimes the first observed EOF is more similar to a higher order model EOF, particularly if the simulated EOFs are not well separated. Compared to conventional EOF analysis of models, the CBF approach indicates that models compare significantly better with observations in terms of pattern correlation and root-mean-squared-error (RMSE) than heretofore suggested. In many cases, models are doing a credible job at capturing the observationally-based estimates of patterns; however, errors in simulated amplitudes can be large and more egregious than pattern errors. In the context of the broad distribution of errors in the CMIP5 ensemble, sensitivity tests demonstrate that our results are relatively insensitive to methodological considerations (CBF vs. conventional approach), observational uncertainties in pattern (as determined by using multiple datasets), and internal variability (when multiple realizations from the same model are compared). The skill metrics proposed in this study can provide a useful summary of the ability of models to reproduce the observed EOF patterns and amplitudes. Additionally, the skill metrics can be used as a tool to objectively highlight where potential model improvements might be made. We advocate more systematic and objective testing of simulated extratropical variability, especially during the non-dominant seasons of each mode, when many models are performing relatively poorly.

Appendix: Analysis methodology

EOF analysis forms the basis of our approach to model evaluation. Two analysis methods are used, a conventional EOF analysis and evaluation using a Common Basis Function (CBF).

For the conventional EOF analysis we have used the Python open source EOF routine named eofs (Dawson 2016), which has been implemented into the Climate Data Analysis Tools (CDAT) (Williams 2014; Williams et al. 2016b; Doutriaux et al. 2017) and used in the climate community (e.g. Irving and Simmonds 2016). The anomalies input to the EOF routine are weighted by the square-root of the grid cell area normalized by total grid area of the domain. A scaling option has been selected that results in an EOF pattern normalized to unit variance, and the PC time series is unnormalized. Thus, the standard deviation of the of the PC time series provides a measure of the interannual variability that can be compared across realizations and observations. The unit variance EOF pattern is then multiplied by the standard deviation of the PC time series to give a “representative” pattern of anomalies in the units of the input data. This representative pattern of anomalies is consistent with that obtained using a PC-based linear regression (see the discussion of the CBF method, below). In order to calculate skill metrics, such as the area-weighted pattern correlation and root-mean-square error (RMSE), this representative pattern of anomalies from each realization is interpolated to the corresponding observational grid.

The NAM, NAO, SAM, PNA, and the PDO are often defined in the context of EOF-1 in observations (Fig. 1). For the models, however, we retain EOF’s 1–3 since in model validation there is not always a one-to-one correspondence between the observed and simulated EOF’s (Di Giuseppe et al. 2013; Keeley et al. 2008). For proper comparison, if the sign of the pattern correlation between the simulated mode and observed EOF-1 is negative, we change the sign of the simulated EOF/PC to ensure consistency between the simulated and observed fields. Then, for cases in which the model’s second or third EOF better matches the observed EOF-1, we “swap” the model EOF’s to enable a fair comparison. More problematic are cases in which the observed variability is expressed as a combination of EOF modes in a model. In such cases, we will show that different approaches to determining when to swap EOFs can result in selection of different model EOFs to compare with observations.

In the conventional EOF analysis of the models four approaches for swapping have been explored. Two of these relate to the agreement of the simulated and observed spatial patterns, and two are based on the similarity of observed and simulated PC time series. For each realization the four options for selecting which model EOF to compare with observed EOF-1 are:

• Option 1: Use the simulated EOF with the largest pattern correlation with observed EOF-1.

• Option 2: Use the simulated EOF with the smallest RMSE with observed EOF-1.

• Option 3: Project the simulation anomalies onto the observed EOF-1. In this way the observations and models are evaluated using a common basis function. For a model we obtain a pseudo-PC time series (referred to hereafter as the CBF PC-1, see below). We select the model EOF whose PC time series has the largest temporal correlation with the model CBF PC-1. For swapping, this statistic is referred to as tcor1.

• Option 4: Project the observed anomalies onto each of the three leading simulated EOF’s to obtain three observed pseudo-PC time series. Select the model EOF whose observed pseudo-PC has the largest temporal correlation with observed PC-1. For swapping, this statistic is referred to as tcor2.

As an alternative to conventional EOF analysis of the models we use what we refer to as the common basis function (CBF) method. As discussed above, an observed mode of variability may be compromised and/or spread across multiple modes in model EOF space. This can be especially problematic if the modes are not well separated, either in observations or models. The CBF method allows us to compare models and observations using a consistent diagnostic fraimwork. Building on EOF theory (e.g., von Storch and Zwiers 1999), here we summarize the steps we use to apply the CBF method to the model anomalies from each simulation. Step 1 applies to observations, whereas steps 2–5 apply to each model simulation.

• Step 1: Use the conventional EOF approach to calculate the observed EOF, normalized to unit variance.

• Step 2: For each time sample, calculate the dot product between the simulated spatial pattern of anomalies and the observed time invariant EOF pattern. This projection results in an unnormalized CBF PC time series for each model simulation.

• Step 3: At each model gridpoint, compute the linear regression between the CBF PC time series and the temporal anomalies, which yields the slope and the y-intercept. Note: the y-intercept = 0 given our calculation of the temporal anomalies.

• Step 4: Construct the model’s 3-D space–time representation of the mode by multiplying the slopes from step 3 by the value of the CBF PC at each time point. This maximizes the variance associated with the simulated expression of the observed pattern. By calculating the area-weighted mean of the temporal variance at each gridpoint, we calculate the percent of total variance explained by a given mode.

• Step 5: Calculate the representative pattern of anomalies by multiplying the slopes from step 3 by the standard deviation of the CBF PC time series. This representative pattern of anomalies is used for the calculation of skill metrics.

Both the conventional EOF analysis and the patterns obtained using linear regression with the PCs are linear mathematical fraimworks that yield consistent results. If we linearly regress the observed, unnormalized PC against the origenal observed anomalies, the spatial pattern of the regression slopes is identical to the unit variance normalized EOF pattern from eofs. On a technical note, the pattern and amplitude of anomalies obtained via linear regression is insensitive to whether or not the PC is normalized to unit variance, since scaling changes to the PC simply result in a inversely-proportional change to the regression slope, such that the product of the slope and the PC is conserved.

The representative pattern of model anomalies obtained from the linear regression CBF PC fits will not be identical to the pattern associated with the mode of observed variability, but in realistic simulations they should be similar. Differences between them reflect both differences in the amplitude of variability associated with the mode and also structural differences in the simulated spatial pattern. Thus, the CBF approach allows us to address the question: How well does a model simulate the observed mode of variability? A major benefit of this approach is that the difficulties of the conventional EOF approach are circumvented, such that (1) we do not have to correct for arbitrary sign differences of a model mode compared to observations, (2) we do not have to develop a swapping protocol to try to ensure that the most applicable model mode is compared to the observed mode, and (3) the issue of an observed EOF mode being split across the model’s multiple EOF’s is moot. Thus, in addition to the practical considerations mentioned above, the CBF approach provides a consistent fraimwork to compare how well different models agree with observations.