Validation of data on temperature and relative humidity; deviation between calculated values and measured data with MR (n = 92160 hours, n.s. = not significant)
The mission of the Central Institution for Decision Support Systems (DSS) in Crop Protection (German acronym ZEPP) is to develop, collect and examine existing forecasting and simulation models for important agricultural and horticultural pests and diseases and to adapt these models for practical use. More than 40 weather-based forecasting models for pests and diseases have been successfully developed within the last years. The occurrence of diseases/pests and periods of high-intensity attacks can be calculated with high accuracy. The forecast models are based on different concepts. These range from simple temperature sum models to complex population matrices with integrated rate based algorithms to calculate growth, reproduction and distribution of noxious organisms.
DSS are employed for the
estimation of disease/pest risk
estimation of the necessity for pesticide treatments
forecast of the optimal timing for field assessments
forecast of the optimal timing for pesticide treatments
recommendation of appropriate pesticides.
Results of DSS are distributed to the farmers via warning services, using different transmission media (bulletins, letters, faxes and telephone answering machines) and via the internet platform www.isip.de (Information System for Integrated Plant Production) (Röhrig & Sander, 2004). The predictions are suitable for integrated as well as organic farming.
In the following chapter the three basic parts that lead to the creation of a comprehensive and modern DSS for forecasting and warning in integrated crop protection will be analysed. Meteorological data are needed as well as assessed field data as input for decision support systems. With these input data the decision support systems calculate an output result, e.g. the date of the first appearance of a pest. In the first part of this chapter a software is presented which was developed to administer the data of weather stations and to make it available for prognosis models. In the second part the results of a study how to increase the accuracy of simulation models by using Geographic Information Systems (GIS) are presented. The influence of elevation and geographical location on temperature and relative humidity are interpolated using GIS methods, whereas precipitation data was obtained from radar measurements. These meteorological data were then used as input for the simulation models. The output of the models is presented as spatial risk maps in which areas of maximum risk of a disease are displayed. It is expected that by using GIS methods the acceptance of model outputs will be increased by the farmers. Finally model validation is one of the essential requirements of the model development process to guarantee that models are accepted and used to support decision making. Validation ensures that the model meets its intended requirements in terms of the employed methods and the obtained results. The ultimate aim of model validation is to make the model useful ensuring that the model addresses the right problem, provides accurate information about the system which is being modelled and makes it actually be used. Methods for the validation and evaluation of forecasting models are described in the third part of this chapter.
2. Management of meteorological data
The meteorological data in Germany are provided on the one hand by the German meteorological service, on the other hand some federal states in Germany built up their own meteorological networks. At the moment data of 148 stations of the German meteorological service and 417 stations owned by the federal states are available. In sum these are data of 565 stations which can be used to run decision support systems. The federal states use the locally installed program AgmedaWin (Agrometeorological database for Windows) (Keil & Kleinhenz, 2007 ) to administer the data of their own weather stations. With AgmedaWin they can import, check, evaluate and store the meteorological data and export them to the internet system www.isip.de in which the data are imported and forecast models are run. AgmedaWin is used so far in 8 states in Germany since 2005. Also the data of the German meteorological service are transferred to www.isip.de.
2.1. Import of meteorological data
Meteorological data can be imported in AgmedaWin as long as the file containing the meteorological data is an ASCII file. The format of the ASCII files of different stations can vary, depending on the type and the manufacturer and depending on the sensor equipment of the weather station.
With AgmedaWin it is possible to import data having different formats. For this reason an “import wizard” was developed. With the import wizard it is possible to describe exactly the format of almost every ASCII import file by defining “import profiles”. The advantage of this solution is that no changes in the program are necessary when a station with a new data format is added. In this case only a new import profile is defined and assigned to the weather station.
How the format of an ASCII file can be described with an import profile is shown in fig. 1 (a-c).
When defining a new import profile it first has to get a profile name. Then some general configurations can be made by for example entering the number of caption lines and the interval of measurements used in the import file. In AgmedaWin only hourly values are stored. So if the interval of measurements is smaller than 60 minutes the values are automatically aggregated to hourly values before they are imported. Next it is possible to define a valid line. In the example in fig. 1 only values of lines which have a “;” at position 10 will be imported. Also it has to be defined if the sensor values in the import file are arranged in fixed columns or if they are separated by a delimiter (fig. 1.a).
The positions and the lengths of the date and time items as year, month, day, hour and minute have to be entered. It also has to be selected whether in the import file the hours are marked from 00:00-23:00 or from 01:00-24:00 (fig. 1.b).
Also the sensors stored in the import file and the positions and lengths of the sensor values must be entered. In the example in fig. 1.c the value of relative humidity would be expected in column 9 with a length of 5. The format of null values can be entered. Also simple calculations can be done with the values and conditions can be checked before the values are imported (fig. 1.c). After an import profile is defined it is saved and then assigned to a weather station. Only if an import profile is assigned to a weather station data for this station can be imported.
2.2. Gap filling
Decision support systems need complete meteorological data as input. Therefore it is very important to make sure that gaps in the data are filled. In AgmedaWin this can be done in different ways.
Gaps can be filled manually by the weather administrator by simply entering or changing the values.
A gap can also be filled by copying the data of a selected station into the gap.
Small gaps which are not bigger than three hours can be filled automatically by executing a linear interpolation.
Big gaps which are bigger than three hours and not bigger than 15 days can be filled with the data of the best adjacent station. The best adjacent station is found by calculating the correlation coefficient. The data of the station with the best correlation coefficient are then merged into the gap by vertical moving and horizontal rotating so that the tangential points at the beginning and the end fit exactly. This is done by graphical support.
2.3. Plausibility checking
Plausibility checking is a very important feature in AgmedaWin because if the meteorological data provided for the prognosis models are wrong the models will give wrong results.
In AgmedaWin two kinds of plausibility checking are possible.
The first kind is the internal plausibility checking. With this method data of a single station can be checked. Several checking algorithms can be defined and adjusted in AgmedaWin.
Examples for checking algorithms:
checking of lower and upper limits
several algorithms for checking the dynamic of the data (e.g.: 12 equal values of air temperature in series would be marked as implausible)
comparing a value with its previous and following value
When values are found to be implausible by the checking routines they are marked with the plausibility sign “*”. The weather administrator has to decide what to do with the marked values. Either he would check if the sensor still works correctly or he could define that a value is plausible although the plausibility checking had marked it as implausible by changing the plausibility sign manually.
The second kind of plausibility checking is the external plausibility checking. With this method the data of adjacent stations can be compared. At first groups of stations in a subregion with similar climatical conditions are defined. Then the deviations of the daily mean values are calculated and shown in a cross table. Deviations which exceed a defined limit are marked. The external plausibility checking can be a help to detect defect sensors.
2.4. Representation of meteorological data
In AgmedaWin meteorological data can be represented in different ways.
The data of all stations can be shown on a map after selecting the sensor, the aggregation (hourly, daily or monthly values) and the date and time.
Data of all sensors of one station can be represented in table form.
Also the values of selected stations and sensors can be represented in a diagram. The user has several possibilities to adjust and configure the representation by selecting or unselecting sensors, by highlighting single sensors or changing the colours. Also it is possible to zoom and scroll in the diagram and to print and save it (fig. 2).
2.5. Evaluation of meteorological data
The following evaluations are available in AgmedaWin:
Sum analysis: Output of the sum of all values of a chosen sensor and station depending on a specified lower and upper limit.
Limit analysis: Showing all hours fulfilling a defined condition.
Evaluations of wind speed and wind direction: Showing a percentaged frequency distribution of wind speed and wind direction values and representing a wind rose of all wind direction values of a specified period.
Long-time mean values: Calculation and output of long-time mean values per month for specified stations and sensors.
Climatic water balance: The climatic water balance is calculated from the evapo -transpiration (Penman, 1948) and the rain fall. The result is represented in a graph or can be stored as a file (fig. 3).
2.6. Export interfaces
In AgmedaWin an ASCII / Excel export interface is implemented. The user has several possibilities to configure the format of the generated output files. He has to enter the period, the export format (ASCII or Excel) and the aggregation interval and he has to select the weather stations and sensors. He can also choose whether he wanted the plausibility signs and the minimum and maximum values of the selected aggregation interval to be exported.
The other very important export interface is the export to www.isip.de which consists of two steps. In the first step a XML file is generated containing the information about the data to be exported. This file is zipped and in the second step transferred to www.isip.de via FTP (File Transfer Protocol).
The ISIP export can be done either manually or automatically. If it is done automatically only data which have been changed since the last transfer or which are new are exported to www.isip.de. With this method it is guaranteed that in www.isip.de always the same meteorological data are available as in the local AgmedaWin databases.
3. Use of geographic information systems in crop protection warning service
In the previous chapter the management of meteorological data originating from individual weather stations and their importance for simulation models was highlighted. However, in some agricultural areas, the distance between weather stations (MS) exceeds 60 km. Forecast models did not give satisfactory results for fields separated by such large distances to MSs ( Zeuner, 2007 ).
With the help of Geographic Information Systems (GIS) a plot-specific classification of temperature and relative humidity has been developed using complex statistical interpolation methods described by Zeuner (2007 ). The method, however, cannot be applied to the parameter precipitation. Especially in the case of frequent spatially and temporally limited rainfall (so-called convective rainfall event), the interpolation for precipitation does not give plausible results (Zeuner & Kleinhenz, 2007, Zeuner & Kleinhenz, 2008, Zeuner & Kleinhenz, 2009). Precipitation data with a high spatial resolution may be obtained from radar measurements.
Using these spatial input parameters for the currently available disease forecast models should lead to accurate forecasting for areas in-between two or more distant MSs. With the use of GIS, daily spatial risk maps for diseases and pests can be created in which the spatial and the temporal process of first appearance and regional development are documented (Fig.4). These risk maps may lead to improved control and a reduction in fungicide use.
In order to store the results of interpolation, a grid was laid out over Germany. At present, the Governmental Crop Protection Services (GCPS) use about 570 MSs to represent an agricultural area of aprox. 200.000 km2, or an average of one MS per 350 km2. With the new GIS method, grid cells have a size of 1 km2 and, after interpolation, are represented by virtual weather stations (Liebig & Mummenthey, 2002)
3.2. Spatial data of temperature and relative humidity
For the interpolation of temperature and relative humidity the multiple regression method was chosen because it gave the best results by the shortest calculation time of all tested interpolation methods. The first calculations with the four interpolation methods (Inverse Distance Weighted, Spline, Kriging and Multiple Regression) showed that deterministic interpolation methods were not suitable. The general purpose of multiple regressions (the term was first used by Pearson, 1908) is to learn more about the relationship between several independant or predictor variables and a dependant or criterion variable. MR is an interpolation method that allows simultaneous testing and modelling of multiple independant variables (Cohen, et al., 2003). Parameters that have an influence on temperature and relative humidity, e.g. elevation, slope, aspect, can therefore be tested simultaneously. MR uses matrix multiplication and only variables with a defined minimum influence that will be included into the model. The result of MR is a formula (x = const + A1*const1 + A2*const2+ A3*const3+…+ Ax*const) which allows a calculation of a parameter set for each grid cell from which independent variables are known ( Zeuner, 2007 ).
|year||temperature [°C]||relative humidity [%]|
To validate the results of the interpolation, 13 MSs were ignored in the interpolation process. After interpolation, the deviation between calculated values and measured data of these stations was compared. The study was conducted from January to August in the years 2003 to 2006. For all stations, MR gave results with highest accuracy (Tab. 1). In all cases, the coefficient of determination (CoD) ranged between 96 and 99% for temperature and 92 and 96% for relative humidity, respectively. For the 13 MSs, the mean deviation for temperature was less than 0.1°C and for relative humidity less than 0.6% as calculated with MR. The absolute maximum and minimum for temperature was less than 4.7°C and for relative humidity less than 32.6%. The data also were tested for significance between calculated and measured data using a t-test. The test indicated that for all stations the differences between the calculated and measured values were random. The MR method gave plausible results, so it was chosen to interpolate the meteorological data to be used as input for the forecasting models.
3.3. Spatial precipitation data
16 radar stations are run by the German meteorological service to record precipitation all over Germany. These stations do not measure the amount of precipitation at ground level but the signal reflected from the rain drops in the atmosphere. These measurements at first only allowed calculation of an unspecific ‘precipitation intensity’, a shortcoming. With the system RADOLAN intensity is now calibrated online with data from a comprehensive network of ombrometers, using complex mathematic algorithms. As a result the amount of precipitation can be provided in a spatial resolution of 1 km² (Bartels, 2006). These calibrated amounts of precipitation based on radar measured rainfall intensities are referred to as “radar data” in the following. The validation of precipitation data took place in intensely used agricultural areas, joining the radar grid with stations of the meteorological network. In this way, it was possible to relate each station to a grid cell.
The radar derived precipitation at the station’s grid cell and the actually measured data formed the basis for the statistical verification. Since rain events differ throughout the year, two representative months (May and August 2007) were selected to analyse uniform rainfalls in spring as well as convective rainfall events in summer. This resulted in a validation dataset of 1488 hours for each MS. Depending on the region, the number of MSs ranged from 9 to 29. In addition, the influence of the distance between radar station and MSs was analysed.
Furthermore, a leaf wetness simulation model used by ZEPP (Racca, 2001, unpublished) was run on data from both methods of precipitation measurement and the results were compared.
The parameters for the amount of precipitation, number of hours with precipitation and calculated leaf wetness showed high correlations between radar values and measured data. The maximum of the hourly deviation of the amount of precipitation was 0.06 mm. In hours with rainfall the deviation was slightly higher (0.36 mm). No correlation could be detected for the distance between radar stations and MSs. For hourly rainfall pattern, a correlation of 91.4% between stations and validation areas was measured. The best correlations were obtained for the leaf wetness model for which values > 99.9% were achieved.
The results clearly show that the use of radar data as an input parameter in disease forecast models is valid. By adding data of temperature and relative humidity with high spatial resolution, an optimal basis for plot-specific forecasts has been established. Moreover, this system allows the exact detection of local convective rainfall events, which at the moment often remain undetected using individual weather stations. Significant improvements of the spatial forecasting by plant disease simulation models can be expected from the use of radar data.
3.4. Introducing spatial risk maps into practice (www.isip.de)
ISIP, the Information System for Integrated Plant production www.isip.de, is a Germany-wide online decision support system. It has been initiated in 2001 by the German Crop Protection Services as a common portal, thus achieving synergies by pooling existing information. Target groups are farmers as well as advisors.
Since information transfer is the primary task of extension services, the system is intended to make this work more efficient by using modern information technology. Therefore a bi-directional data flow between the services and the farmers was developed. By combining general with specific data, recommendations can be refined from regional to individual. The information is primarily distributed via HTML pages, thus a browser is necessary to use the system (Röhrig & Sander, 2004).
In 2010 a new way of presenting results of prognosis models for plant pests and diseases has been implemented. Using interpolated meteorological data in a high spatial resolution as input paramters, so-called ‘risk maps’ are drawn (fig. 5). These maps have several advantages compared to results representing point information based on single weather stations:
A risk maps more suitable to identify hot spots and eases the interpretation of the model’s results.
The user does not have to choose a specific weather station, which might even not be valid for his plant production site.
The maps are produced conform to the OGC standards, thus can be used in other systems.
In addition to the GIS functionalities of zooming and panning, it is possible to scroll through the maps of the last ten days. This gives an excellent overview of the temporal development of the pest or disease risk.
The system is supplemented by a spatial three-day weather forecast 3 day offered by the German Meteorological Service. It is expected that this further supports the decision and management processes of the farmer.
4. Model validation and evaluation in crop protection warning service
Whereas in the previous two chapters the management, validation and preparation of meteorological data which are essential for the forecasting models were discussed the next chapter will address the complex issue concerning the validity of the simulation models used in pest integrated control.
Considering the general mathematical simulation of real events or specific models for plant disease epidemiology, there are several definitions of model validation (Kranz & Royle, 1978, Racca, et al., 2010b, Reynolds, et al., 1981, Sargent, 1998, Schlesinger, 1979, Teng, 1985, Welch, et al., 1981). The validation process can be summarised as the comparison between the virtual (simulated) and the real (actual) system. According to (Balci & Sargent, 1984) the model validation of a generic model will be separated in two methods: the subjective validation techniques and the statistical techniques. In other words, the model validation strictly depends on the modelled system, the model output and the availability of observed field data for the validation.
According to the classification due to ZEPP (Racca, et al., 2010b) the simulation models can be summarised in:
Pests and disease models:
Type 1: Models predicting first appearance of a disease or a pest.
Type 2: Complex simulation models predicting the epidemiological development or population dynamics like the disease severity, the disease incidence or pest abundance.
Type 3: Models predicting a target event like the overriding of an action threshold or periods with high risk for an epidemiological development of the disease or of pest population dynamic.
Ontogenesis models are able to simulate the crop-growing. These models can be used in combination with the diseases or corresponding pest models.
For each kind of these models a subjective and/or a statistical validation is possible.
The evaluation of the model performance depends on the aim and the type of model and is strictly correlated with the validation results. Generally, based on the ZEPP experience, a model has a good performance when during the validation phase a quote of correct prognoses up to 75% is overridden and/or the underestimations are less than 15%.
4.1. Validation of type 1 models
Type 1 models forecast the first appearance of a disease or a pest at the beginning of the growing season. They are mainly used by advisors to determine the beginning of the regional monitoring activities and by farmers to do the initial checks on their fields. The results are generally given on a regional level considering the region as the area surrounding a weather station. Examples of these models currently used by the GCPS are CERCBET1, ERYBET1, UROBET1 and RAMUBET1 used to predict the appearance of Cercospora leaf spot (Cercospora beticola) (Roßberg, et al., 2000), powdery mildew (Erysiphe betae), rust (Uromyces betae) and ramularia (Ramularia beticola) (Racca, et al., 2010a) on sugar beet, SIMCOL1 for the lupin anthracnose (Colletotrichum lupini) (Racca & Tschöpe, 2010 ), SIMPHYT1 and SIMBLIGHT1 for the potato leaf blight (Phytopthora infestans) (Kleinhenz, et al., 2007), SIMPEROTA1 for blue mold disease of tobacco (Peronospora tabacina) ( Racca, et al., 2007 ). An example for pests is the SIMLEP1-Start model to forecast the appearance of the hibernating adults of Colorado potato beetle (Leptinotarsa decemlineata) (Jörg, et al., 2007).
All type 1 models can be validated by using statistical and/or subjective validation methods. Some of these methods will be described below with a few examples.
The subjective validation of CERCBET1 took place retrospectively with monitoring - data of the years 1995 to 2008 in all German sugar beet growing areas.
The validation was made by comparing
the date of the first appearance forecasted by the model and observed in the field;
the forecasted and observed date when 50% of the fields in one region are infected.. This date represents the distribution of the disease in several fields in one region, the 50th percentiles. At this point the probability to detect a Cercospora infection in a field is very high and the disease had been established in this region. For the validation the available data are grouped in “regions” near a representative weather station. To detect the distribution of the infected field, only regions with surveys in more than four sugar beet fields were taken into consideration.
In any case the forecasting was considered:
correct - when the difference between the forecasted and the observed date was in a range of one week (± 7 days);
early/late - when the difference between the forecasted and the observed date was bigger than one week (> ± 7 days).
The subject of the validation method in this case is to consider a period of ± 7 days correct for these kind of model results. We consider that the data for the validation derive from regional monitoring arranged by the GCPS. This monitoring is done weekly. So, one week of delay or one week of earlier forecast is acceptable for this model.
The result of the validation is summarised in Table 2.
|first appearance||50 % infected fields|
Concerning the first appearance, in about 64% of the forecasts the model was able to predict the disease appearance correct, in 32% the date was too early and in 4% of the cases the forecasted date of disease appearance was too late. The model shows more accuracy in the prediction of the 50% infected field date. Analyzing the results of the validation, the trend of the model to anticipate the occurrence of disease can be identified (Tab.2). This trend can also be explained by analyzing the same data which was used for the validation. Sometimes sample size t used in the surveys is not appropriate for detecting a rare event like the appearance of first necrotic spots (Roßberg, et al., 2000). It can also be difficult to recognize the first symptoms at the leaves (the first spots of Cercospora could be taken for Alternaria sp., Phoma sp. or bacterial spots).
For an appropriate statistical validation (Racca, et al., 2010b, Rossi, et al., 1997a, Teng, 1981) simulation and field data were regarded as two independant random samples to compare the distribution. It is possible to apply some parametric tests like the t-test (comparison of mean values) and the F-test (comparison of standard deviation), but also a non-parametric method like the Kolmogorov-Smirnov test (computing the maximum distance between the cumulative distributions of two samples) (Tab.3)
|year||n||first appearance||50% infected fields|
|t-test||F-test||Kol.Smirn. test||t-test||F-test||Kol.Smirn. test|
Considering the statistical analysis of model results, it must be concluded that the presence of significant differences between the distributions does not show a good correlation according to the data. However, analyzing the subjective validation method we conclude that the model is satisfactory if we consider the principal aim of the model, determine the date of beginning of the monitoring system. An early forecast can also be accepted (in only 12% of the total cases the forecast was more than 3 weeks before the observed first appearance).
A similar validation was done for the SIMBLIGHT1 model. The model predicts the risk of a potato late blight outbreak and recommends the date for the first treatment. A subjective validation was done and the model results were considered valid when the predicted date of the first treatment was earlier than the date of the field observation. More than 700 observations during the period 1994-2005 were recorded for the validation. For most years the proportion of correct forecasts reached more than 90% and a statistical validation was not done (Kleinhenz, et al., 2007).
For models that provide results in a binary response it is possible to apply a different statistical method of evaluation and validation. As one example in the model ERYBET1 the dates of onset of the disease (powdery mildew on sugar beet) were classified into two groups: early onset (before July 31) and late onset (after July 31). The two groups are discriminated by a binary logistic regression model using the winter weather as input parameters ( Racca, et al., 2010a).
Binary logistic regressions results in a value between 0 and 1. Discrimination of individual cases in two classes is done by definition of a “cut-off” threshold. Depending on the “cut-off” the following classification of the cases is defined:
true negative: cases correctly classified in early onset (model specificity)
false negatives: the values of early onset were classified in late onset (underestimation)
true positive: cases correctly classified in late onset (model sensitivity)
false positives: the values of late onset were classified in early onset (overestimation)
The cut-off value is simply chosen mathematically or graphically so that the rate of false positives and false negatives is minimized (Hadjicostas, 2006). The model validation is done using a ROC (Receiver Operating Characteristic) curve (Madden, et al., 2008).
In a ROC curve the true positive rate (Sensitivity) is plotted in function of the false positive rate (1-Specificity) for different cut-off points of a parameter. Each point on the ROC curve represents a sensitivity/specificity pair corresponding to a particular decision threshold. The area under the ROC curve is a measure of how well a parameter can distinguish between two diagnostic groups (Fig. 6).
Accuracy is measured by the area under the ROC curve (AUC). An area of 1 represents a perfect test, an area of 0.5 represents a worthless test. A rough guide for classifying the accuracy of a diagnostic test is the traditional academic point system: 0.9-1 = excellent; 0.8-0.9 = good, 0.7-0.8 = fair, 0.6-0.7 = poor, 0.5-0.6 = fail.
Since the AUC of the ERYBET1 model is equal to 0.73 it is statistically acceptable. Furthermore, with a cut-off equal to 0.74 (estimated to obtain for the model a weighted combination of specificity and sensitivity) the values shown in Table 5 are obtained for the classification of the onsets. The percentage of correct classification is about 65% of all cases. The underestimations were less than 35% and so the model is considered a good estimator of the early or late onset of disease dependant only on winter weather conditions.
|disease onset classification||early||late||sum||correct (%)|
4.2. Validation of type 2 models
Type 2 models are examples for classical simulation models. Generally, they are very complex. The aim is to predict the epidemic development (expressed as disease severity and/or disease incidence) for the diseases or to predict the various development stages of insects. These models are used, in the examination phase, as a basis for identifying the parameters and variables for the construction of type 1 and 3 models. Examples of these models are CERCODEP which is able to simulate the epidemics of Cercospora leaf spot on sugar beet (Rossi, et al., 1994), RUSTDEP for leaf rust (Puccinia recondita) on winter wheat (Rossi, et al., 1997b), SEPTRI2 for leaf blotch (Septoria triciti) on winter wheat (Erven, et al., 2008, Erven, et al., 2010), SIMPHYT2 for the potato leaf blight (Roßberg, et al., 2001) and SIMLEP2 to forecast the phenological development of the Colorado potato beetle (Roßberg, et al., 1999).
The following three models are used as examples for validation methods: PUCREC for leaf rust in winter rye (Puccinia recondita), PUCTRI for leaf rust in winter wheat (Puccinia triticina) and SEPTRI for leaf blotch (Septoria triciti) in winter wheat.
The first two models simulate the development of the epidemics of rust on different leaf layers expressed as disease incidence (Racca, et al., 2008, Räder, et al., 2007). Models were validated with both subjective and statistical methods of field data, collected from 2002 to 2005. In total, 51 data sets for PUCREC and 37 for PUCTRI were available to investigate the predictive ability of the models.
The subjective validation simply consists of the comparison of the simulated disease incidence with data, recorded in the field (fig. 7). The simulation is decided to be correct when the simulated disease incidence ranges between the confidence interval of the recorded disease incidence. Overestimation is given when the simulated value overrides the highest level of the confidence interval in opposition to underestimation when the simulated value is under the lowest level of the confidence interval. Validation was done for both models and for each leaf layer F (flag leaf) to F-3 (Tab.5).
|leaf layer||PUCREC -winter rye||PUCTRI -winter wheat|
According to the subjective validation the field data and the model results coincided well. In most of the cases (from 71 to 86%), the disease incidence progress was correctly simulated (Tab. 5). In a few cases PUCREC underestimated (2 - 8%) or overestimated (10 – 20%) the epidemic progress of P. recondita. For winter wheat a considerable share of overestimations occurred (18 – 29%). This means that epidemics simulated by PUCTRI started earlier and progressed faster than observed in the field. For winter wheat no underestimations could be observed.
The statistical validation was done with two parametric (regression analysis, hypothesis test) and one non-parametric test (Kolmogorov-Smirnov).
The simulated disease incidence (dependant variable) is simply linear correlated with the recorded data (independant variable). The “null hypothesis” demonstrates that “a” (intercept of the regression line) is equal to 0 and “b” (slope of regression) is equal to 1 (tested using the Student t-test) (Tab.6).
|Leaflayer||PUCREC -winter rye||PUCTRI -winter wheat|
|Regression parameters||Kolm.-Smirn.||Regression parameters||Kolm.-Smirn.|
The statistical validation gave very satisfactory results with both (parametric and non-parametric) methods. The high number of non-significant cases of the regression parameters and the Kolmogorov-Smirnov test mean that the model is considered a statistically accurate simulator of the field data (Teng, 1981).
SEPTRI is a model able to simulate the disease epidemics of Septoria tritici on winter wheat. In particular the model calculates the infection probability, the infection rate, the lesion growth and the sporulation for wheat cultivars for three susceptibility levels: high susceptible, mean susceptible and low susceptible. The simulation results are the values of disease severity per leaf layer.
|leaf layer||cultivar susceptibility|
According to the subjective validation, the model results are satisfactory. In most of the cases (from 50 to 81.3%), excluding the leaf layer F-3, the disease severity progress was simulated correctly (Tab. 7). For leaf layer F-3 a considerable share of underestimation occurred (from 43.8 to 75%). This means that epidemics on this leaf layer were poorly simulated. Otherwise for use of the model in praxis the leaf layer F-3 has less importance because the action threshold for the treatment scheduling is generally based on leaf layer F-2.
|leaf layer||SEPTRI – all cultivar susceptibilities|
The statistical validation of the SEPTRI model gives satisfactory results for the leaf layer F, F-1 and F-2 with both (parametric and non-parametric) methods.
4.3. Validation of type 3 models
Type 3 models are derived from models of type 2. The output is various. The simulation of the development of the disease is often done to forecast the overriding of the action threshold and to give a recommendation for a fungicide or insecticide spraying. Models of type 3 are sometimes combined with routines for the calculation of the effectiveness of pesticides. They may include agricultural parameters like type of rotation, fertilization, irrigation and cultivar susceptibility which interact with the epidemics of the specific disease. The type 3 models can be used on regional and field-specific levels. Some examples of the most successful type 3 models are CERCBET3 (Racca & Jörg, 2007), ERYBET3, UROBET3 and RAMUBET3 (Racca, et al., 2010a) for Cercospora leaf spot, powdery mildew, rust and ramularia leaf spot respectively on sugar beet, SIMPHYT3 for potato leaf blight (Gutsche, 1999), PUCREC3 and PUCTRI3 for cereal leaf rusts (Räder, et al., 2007) and SIMLEP3 for Colorado potato beetle (Jörg, et al., 2007).
Some validation methods for type 3 models can be described using sugar beet leaf disease models as examples. CERCBET3, ERYBET3, UROBET3 and RAMUBET3 simulate the progress of sugar beet leaf diseases, expressed as disease incidence development, and forecast the overriding of an action threshold, suggested for the treatments during the sugar beet growing season (Racca & Jörg, 2007, Racca, et al., 2004). For most sugar beet growing areas in Germany the Regional Plant Protection Offices use an action threshold for the fungicide treatment to control sugar beet leaf diseases based both on time and disease incidence (DI). Particularly the Plant Protection Services suggest the following strategy: action threshold 5% DI until the end of July, 15% DI before the 15th of August and 45% DI later than 15th August (Jörg & Krauthausen, 1996). For a subjective validation the weekly assessment data of the disease incidence were confronted with simulated data. The difference between the assessed and simulated date exceeding the relevant action threshold (5, 15 and 45% of disease incidence) was classified as follows:
too early: the forecast exceeds the action threshold more than 7 days earlier than the assessed date;
correct: the forecast differs not more than ± 7 days from the assessed date;
too late: the forecast exceeds the action threshold more than 7 days later than the assessed date.
The results of this subjective validation is summarised in Tab. 9.
|model||n||too early||correct||too late|
|CERCBET3||71||10.9 %||84.2 %||4.9 %|
|ERYBET3||555||28.4 %||63.8 %||2.9 %|
|UROBET3||652||16.5 %||77.2 %||6.3 %|
|RAMUBET3||241||9.6%||86.0 %||2.5 %|
All sugar beet type 3 models gave very satisfactory results. The action threshold was forecasted correctly in about 84% of the cases for cercospora leaf spot, in about 69% for powdery mildew, in about 77% of the cases for rust and in about 86% of the cases for ramularia leaf spot.
For the CERCBET3 model the same pool of data was useful for an appropriate statistical validation. For all three action thresholds the simulated date of threshold exceeding was linear regressed with the real date (Tab.10).
|action threshold 5%||5.43||0.86||*||*||0.72||0.82|
|action threshold 15%||20.76||0.31||*||*||0.31||0.45|
|action threshold 45%||32.64||0.24||*||*||0.19||0.36|
The statistic analysis shows different results compared to the subjective validation. There is only a strong statistical correlation for the action threshold 5% (r2 = 0.72). The other action thresholds are poor correlated (r2=0.31 and 0.19). For all regressions the intercept and the slope are significant at p<0.05.
One step further, using the concordance correlation coefficient ρc (Lin, 1989) can be overcome the statistical validation of avoiding problems with misinterpreted results in the regression analysis (failed t-test for a and b). The values of ρc range from 1 perfect agreement to -1, total lack of agreement. Validation of the action threshold 5% shows a high value of ρc. There is a good agreement between the simulated and the real observations.
In practical use of the model, a correct timing of the overriding of the first threshold (action threshold 5%) is very important. In some cases a treatment in this early phase is able to decelerate the epidemic of the disease and a second or third spraying can be avoided. In this case the combination of subjective and statistical validation is very satisfactory. Another example for the validation of a type 3 model can be illustrated with the model SIMLEP3. The model simulates the development of Leptinotarsa decemlineata from the beginning of egg laying to the occurrence of the old larvae at a field specific scale (Jörg, et al., 2007). SIMLEP3 was validated in Germany and in several European countries. The method is only subjective, comparing the forecasting dates of the maximum abundance of egg cluster and young larvae with field observations. The model output was considered correct when the forecast was within an interval of one week compared to the observed date (Tab. 11).
|country||maximum abundance of|
|egg clusters||young larvae|
|% correct||% too early/late||n||% correct||% too early/late||n|
In general, SIMLEP3 results were very satisfying. The first occurrence of young larvae in most of the cases was predicted correctly. Nevertheless, differences between forecasting and observed date ranging from 18 days too early up to 10 days too late were registered. Good results were also obtained for the prediction of maximum egg cluster occurrence. Throughout Germany, Poland, Austria and Italy the mean share of correct forecasts given from SIMLEP3 (both egg clusters and young larvae) amounted to about 92%. In Austria, the share of correct predictions was the lowest (approx. 70%) and in Germany the share of correct predictions exceeded 90%. Maximum occurrences of young larvae predictions were correct in about 93% of the cases on the European scale. Again, optimum results were obtained in Italy and Poland. In Austria and Germany, the share of correct forecasts exceeded 85%. Subjective validation efforts showed that SIMLEP3 is able to give correct forecasts for the most important development stages of L. decemlineata with respect to control efforts. The validation also demonstrates the possibility to expand the use of the model throughout Europe.
4.4. Validation of ontogenesis models
Ontogenesis models are ontogenetic models which simulate the development of crops expressed as BBCH growth stages (Hack, et al., 1992) over time. SIMONTO-models are based on the modelling approaches of CERES-Wheat (USA) (Gabrielle, et al., 1998) and ONTO-models (Germany) (Wernecke & Claus, 1996). The ontogenetic progress in SIMONTO is reflected by a developmental rate which is a function of temperature and photoperiod. Parameters for the different models (winter oilseed rape and winter cereals) were estimated by employing the Monte-Carlo-method (Falke, et al., 2006, Roßberg, et al., 2005). More than 13800 single observations of BBCH growth stages for winter cereals from 2003 to 2008 were available for the model validation. In the first step of the statistical validation, the observed BBCH growth stages are simply linear regressed with the model. A high r2 (0.88) suggests a good correlation between the data. Both regression parameters, a and b, are significant but the concordance correlation coefficient of 0.92 demonstrates a good agreement between the data. The model is apparently good to simulate the reality. Unfortunately, the BBCH growth stages are not strictly arithmetical dependant. Some stages could appear very early in the season and stay constant for a long time. The simple arithmetic difference between two BBCH growth stages can be minimal but the difference in days between the two stages themselves can be very big. For example, the arithmetical difference between BBCH 21 (beginning of the tillering) and 22 (2 tiller detectable on the plants) is only = 1 but sometimes BBCH 21 is recorded on the fields in autumn and simulated by the model in spring. This can mean a difference of up to 5 months. Again the model should also be validated with a subjective method. For SIMONTO a scoring model approach is used (Roßberg, et al., 2005). The difference in days between the simulation and the observation is classified with a subjective flaw dot table:
model more than 7 days too early or more than 7 days too late: flaw dot 7;
model too early or too late (4 to 7 days): flaw dot 3;
model too early or too late (1 to 3 days): flaw dot 1;
no difference between simulation and observation: flaw dot 0;
The sum of the flaw dots could be classified in a flaw dot coefficient with values varying from 0 (perfect model) to 7 (simulation extremely early or extremely late). Concerning the results in Tab. 12 the flaw dot coefficient is comprised in a range from 1.27 to 3.10. Values are classified when the simulation results arrive with a delay or an advance of up to 3 days. Values above 3 indicate a higher difference of days between simulation and reality.
Since most of the flaw dot coefficients are lower than 3 we can conclude that the model accurately simulates the reality with a gap of acceptable days. In this case, the subjective validation is essential because the statistical validation could lead to misleading results.
|BBCH||n||case with||flaw dot||flaw dot|
|flaw dot 7||flaw dot 3||flaw dot 1||flaw dot 0|
Another type 4 model validated in 2010 (Tschöpe & Racca, 2010) was SIMONTO-lupin, a model which simulates the crop stages of the lupin (Lupinus angustifolius).
For the subjective validation the simulated progress of ontogenesis was compared visually with the observed BBCH growth stages in the field. The classification of the cases in correct, overestimated and underestimated was the same used for the type 2 model validation. In total 215 data sets were analysed by this validation method. 88.4% of the cases were simulated correctly, 9.8% were underestimated and 1.9% were overestimated.
For the statistical validation the model output (dependant variable) was compared to the field data (independent variable) with the help of a regression analysis. The average of the coefficient of determination was 0.984 and the slope b was in all cases not significant.
|BBCH stage||n||too early||correct||too late|
|61||229||0 %||86.0 %||14.0 %|
|69||22.7 %||75.5 %||1.7 %|
Like the BBCH growth stages of the cereals also the BBCH stages of the lupin are not strictly arithmetically dependant. Also in this case another validation criterion was needed. Again the deviation in days between the ontogenesis in field and the simulated ontogenesis was compared. If the deviation between the both dates was ±7 days the prognosis was rated as “correct”. Otherwise the prognosis was too early or too late (Tab. 13).
In total 229 data sets were analysed by this validation method for the BBCH-stages: Start of flowering (BBCH 61) and end of flowering (BBCH 69). BBCH 61 achieved a hit rate of 86.0% correct forecasts, 14.0% of the dates were simulated too late. Concerning the BBCH 69 it was simulated correctly in 75.5% of all cases, 22.7% of the simulations were too early and in 1.7% of the cases the simulated BBCH was too late.
Decision support systems in plant protection need plausible and complete meteorological data as main input. Meteorological data on the one hand are provided by the German meteorological service. On the other hand several states in Germany built up their own meteorological networks. These states use the software AgmedaWin for import, management, presentation, evaluation and export of the measured data. Core of the program is a flexible import module which facilitates the import of files with different formats from all types of weather stations by describing the structure of the files with import profiles. Several algorithms are integrated in AgmedaWin to ensure plausibility and completeness of the data. The program also includes a module to compare data of corresponding stations. With an XML-based export interface the data are transferred from AgmedaWin to the internet system www.isip.de where all data are stored and used as input for the decision support systems. Furthermore the unprocessed meteorological data can be evaluated in www.isip.de or downloaded as files in different formats by external users.
The plausibility and completeness of meteorological data as main input for the models is the most important pre-condition to get correct prognosis results. However by using meteorological data of weather stations a good prognosis is only reached in the scope of a weather station. That is the reason why the ZEPP developed a new technology based on Geographic Information Systems (GIS). With the help of GIS it is possible to obtain results with higher accuracy for disease and pest simulation models. The influence of geographical factors on temperature and relative humidity were interpolated with GIS methods getting meteorological data for every km2 in Germany. The parameter precipitation was taken by radar measured precipitation data and the results of all measured meteorological data were used as input for the simulation models. The output of these models is presented as spatial risk maps in which areas of maximum risk of the disease outbreak, infection pressure or pest appearances are displayed. The modern presentation methods of GIS lead to an easy interpretation and will furthermore promote the use of the system by farmers.
Finally the validation of a simulation model is a critical point in the development of the model itself. Unfortunately, there is no set of specific tests or decision-making algorithms which can determine the best method to validate a model. The subjective methods are certainly more intuitive and provide easy answers with easy interpretations. In this case, the decision for the method depends on the experience of the one who validates the model. It is important to know, for example, what weight has to be indicated to the over- and especially to the underestimation of the results of the model. Careful attention must be payed to the quality of data which is available for the validation. They should certainly be adequate in number and represent the real agronomical system. Unfortunately there is no “manual for the validation of a model” but a combination of statistical and subjective methods gives good judgement of benefits of a decision support system which helps to plan crop protection activities like field assessments or pesticide use.