New methods for the analysis and evaluation of agro-environmental simulation models

This research line deals with the development of methods and metrics for the evaluation of simulation models and with the development of fuzzy-based, multi-metric evaluation methods specific for model type and/or application context.

The new metrics proposed target:
- model robustness (Ir) (Confalonieri et al., 2010a);
- model balance (Confalonieri, 2010);
- model plasticity (Confalonieri et al., 2012).

The robustness indicator (from 0 to +inf; optimum = 0) is based on the relationship between variability of errors and of explored conditions. In practice, the standard deviation of the Nash and Sutcliffe modelling efficiency is related with the standard deviation of a normalized, synthetic agrometeorological indicator (SAM; (rain-ET0)/(rain+ET0); Confalonieri et al., 2010a) calculated on the same datasets used to derive modelling efficiencies. The idea behind the robustness indicator is that positive values should be assigned to models presenting a stability in uncertainty while changing the conditions of application, since this feature decreases the risk of incoherent behaviours when models are used to extrapolate information about a system previously unexplored.

Model balance and plasticity are quantified using sensitivity analysis techniques.
The model plasticity indicator (L) quantifies the aptitude of a model to change the sensitivity to its parameters (quantified using the Sobol' method) while changing the conditions of application. Concordance among parameters relevance (total order effect) estimated using the top-down concordance coefficient (TDCC) under different conditions allows quantifying changes in the way models react to different environments. The concordance among the different SA results is then related to the variability of a normalized agrometeorological indicator (SAM; (rain-ET0)/(rain+ET0); Confalonieri et al., 2010a) used to characterize the explored conditions. L is calculated as TDCC • exp(sSAM - 1) (with sSAM being the standard deviation of SAM), and ranges from 0 to about 1.51, with highest plasticity at 0.

Model balance is based on the homogeneity among sensitivity indices of the model parameters. After a variance-based sensitivity analysis method (e.g., Sobol') is applied, the Grubbs test for outliers detection is performed on the sensitivity metrics to evaluate whether the relevance of a parameter (or of a few of them) is significantly higher than the others. Unbalanced models (largely depending from one or few parameters) are considered as exposing users to risks during the calibration.

These and other metrics for, e.g., accuracy and complexity, were used to develop fyzzy-based, multi-metric evaluation procedures specific for models estimating crop growth (Confalonieri et al., 2009), soil hydrology (considering in an integrated way multi-layer dynamics; Confalonieri et al., 2010b), hourly air relative humidity (Bregaglio et al., 2010), leaf wetness (Bregaglio et al., 2011).

An approach for quantifying the relevance of weather variables on the outputs of agrometeorological models was also developed (Confalonieri, 2012), based on the combined use of a weather generator and a standard sensitivity analysis method (Morris in the specific case). The method represents a solution to the problem of applying standard sensitivity analysis methods to relate weather inputs (array of daily values for each variable, with variables non independent) and outputs representing the state of variables in a specific moment during the simulation.

References:

Bregaglio, S., Donatelli, M., Confalonieri, R., Acutis, M., Orlandini, S., 2010. An integrated evaluation of thirteen modelling solutions for the generation of hourly values of air relative humidity. Theoretical and Applied Climatology, 102, 429-438.

Bregaglio, S., Donatelli, M., Confalonieri, R., Acutis, M., Orlandini, S., 2011. Multi metric evaluation of leaf wetness models for large-area application of plant disease models. Agricultural and Forest Meteorology, 151, 1163-1172.

Confalonieri, R., 2010. Monte Carlo based sensitivity analysis of two crop simulators and considerations on model balance. European Journal of Agronomy, 33, 89-93.

Confalonieri, R., 2012. Combining a weather generator and a standard sensitivity analysis method to quantify the relevance of weather variables on agrometeorological models outputs. Theoretical and Applied Climatology, 108, 19-30.

Confalonieri, R., Acutis, M., Bellocchi, G., Donatelli, M., 2009. Multi-metric evaluation of the models WARM, CropSyst, and WOFOST for rice. Ecological Modelling, 220, 1395-1410.

Confalonieri, R., Bregaglio, S., Acutis, M., 2010a. A proposal of an indicator for quantifying model robustness based on the relationship between variability of errors and of explored conditions. Ecological Modelling, 221, 960-964.

Confalonieri, R., Bregaglio, S., Bocchi, S., Acutis, M., 2010b. An integrated procedure to evaluate hydrological models. Hydrological Processes, 24, 2762-2770.

Confalonieri, R., Bregaglio, S., Acutis, M., 2012. Quantifying plasticity in simulation models. Ecological Modelling, 225, 159-166.