Quantifying the quality of coastal morphological predictions

This thesis investigates the behaviour of the often used point-wise skill score, the MSESSini a.k.a. BSS, and develops new error metrics that, as opposed to point-wise metrics, take the spatial structure of morphological patterns into account. The MSESSini measures the relative accuracy of a morphological prediction over a prediction of zero morphological change, using the mean-squared error (MSE) as the accuracy measure. The main findings about the MSESSini are: 1) a generic ranking, based on values for MSESSini, has limited validity, since the zero change reference model fails to make model performance comparable across different prediction situations; 2) the combination of larger, persistent and smaller, intermittent scales of cumulative change may lead to an increase of skill with time, without the prediction on either of these scales becoming more skilful with time; 3) in the presence of inevitable location errors, the MSESSini favours predictions that underestimate the variance of cumulative bed changes and 4) existing methods to correct for measurement error are inconsistent in either their skill formulation or their suggested classification scheme. In order to overcome the inherent limitations of point-wise metrics, three novel diagnostic tools for the spatial validation of 2D morphological predictions are developed. First, a field deformation or warping method deforms the predictions towards the observations, minimizing the squared point-wise error. Error measures are formulated based on both the smooth displacement field between predictions and observations and the residual point-wise error field after the deformation. In contrast with the RMSE, the method captures the visual closeness of morphological patterns. Second, an optimal transport method defines the distance between predicted and observed morphological fields in terms of an optimal sediment transport field. The optimal corrective transport field moves the misplaced sediment from the predicted to the observed morphology at the lowest quadratic transportation cost. The root-mean-squared value of the optimal transport field, the RMSTE, is proposed as a new error metric. As opposed to the field deformation method, the optimal transport method is mass-conserving, parameter-free and symmetric. The RMSTE, unlike the RMSE, is able to discriminate between predictions that differ in the misplacement distance of predicted morphological features. It also avoids the consistent reward of the underestimation of morphological variability that the RMSE is prone to. Third, a scale-selective validation approach allows any metric to selectively address multiple spatial scales. It employs a smoothing filter in such a way that, in addition to the domain-averaged statistics, localized validation statistics and maps of prediction quality are obtained per scale (geographic extent or areal size of focus). The employed skill score weights how well the morphological structure and variability are simulated, while avoiding to reward the underestimation of variability. To fully describe prediction quality multiple metrics are required with a weighting determined by the goal of the simulation. Point-wise metrics should be supplemented with an error decomposition, as to avoid undesired underestimation of variability. Further, a set of performance metrics must include a metric, e.g. the RMSTE, that accounts for the spatial structure of the observed and predicted morphological fields.