Complexity in hydroecological modelling: A …...sure of statistical uncertainty is more realistic, making an information theory approach more reflective of the complexity in real‐world
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Received: 22 December 2017 Revised: 27 June 2018 Accepted: 27 June 2018
DOI: 10.1002/rra.3328
R E S E A R CH AR T I C L E
Complexity in hydroecological modelling: A comparison ofstepwise selection and information theory
pling data were provided by the Environment Agency for six sites
(Figure 1; EA, 2016); the sampling methodology follows the Environ-
ment Agency's standard semi‐quantitative protocol (see Murray‐Bligh
(1999). Seventy‐two macro‐invertebrate samples, collected in the
FIGURE 2 Flow duration curve (top) andaverage spring LIFE scores (bottom) during thestudy period
spring season (April–June, 1993–2012), were used to determine LIFE
scores at the species level; see Figure 2 for the average spring LIFE
scores during the study period. The ecological data were paired with
the antecedent seasonal hydrologic indices.
2.2 | Modelling scenarios
The multiple linear regression modelling approaches are applied to two
scenarios. In scenario A, the 10 (interannual) hydrologic indices
described previously are considered. Scenario B incorporates ecologi-
cal lag in response, a reflection of the inherent complexity of the
hydroecological relationship. Following Visser et al. (2017), 30 hydro-
logic indices result from the interannual indices being time‐offset up
to 2 years (t‐2).
2.3 | Stepwise regression
Two methods of stepwise selection are applied, backwards and bidi-
rectional. Being unidirectional, backwards represents greater econ-
omy, performing fewer steps to select the smallest model. The
algorithms are specified to remove variables which are not significant
(alpha threshold = 0.05) and hence presumed unimportant to the
hydroecological relationship. Bidirectional stepwise selection is
applied using the function step, from the base statistical package
stats, whereas the backwards algorithm is applied using the
ols_step_backward function from olsrr, a package for the devel-
opment of ordinary least squares regression models (Hebbali, 2017).
These methods yielded the same models, therefore no further differ-
entiation is made.
4 VISSER ET AL.
2.4 | Information theory
The information theory approach provides a quantitative measure of
support for candidate models. Subsequently, inference is made from
multiple models through model averaging. The candidate models are
evaluated with respect to the three steps detailed below; for further
information, see Burnham and Anderson (2002).
Step 1. Loss of information from model f
Kullback–Leibler measures the amount of information lost when
model g is used to approximate reality, f . The model with the least
information loss (greatest supporting evidence of the candidates) is
considered the best approximation of reality.
The information loss, I( f , g), is determined through computation of
an information criterion. The Akaike Information Criterion (AIC) repre-
sents the standard estimate (Burnham & Anderson, 2002). In
hydroecological modelling, the sample size is often small relative to
the number of variables; here, a second order bias correction, AICc,
is used (Burnham & Anderson, 2002).
Step 2. Evidence in support of model gi
The value of AICc is dependent on the scale of the data; the goal
is to achieve the smallest loss of information. This difference is
rescaled and ranked relative to the minimum value of AICc:
Δi ¼ AICci−AICcmin for i ¼ 1;2;…;R: (1)
This provides a measure of evidence, from which the likelihood that
model gi is the best approximating model can be determined. This is
known as the Akaike weight, w, ranging from 1 to 0, for the most
and least likely models, respectively:
wi ¼exp −
12Δi
� �
∑Rr¼1 exp −
12Δr
� �: (2)
Step 3. Multimodel inference
The best approximating model is inferred from a weighted combi-
nation of all the candidates. Parameter averages, bθ, are the sum of the
Akaike weights for each model containing the predictor, bθ:bθ ¼ ∑R
i¼1wibθi: (3)
Parameter averages are ranked, such that the highest value represents
the most important in the model.
2.5 | Package glmulti
There are two options for the application of information theory in R:
MuMIn (Bartoń, 2018) and glmulti (Calcagno, 2013). The application
of the former centres around “dredging” (data mining) to determine
the model subset (e.g., see Grueber, Nakagawa, Laws, and Jamieson
(2011)). The package glmulti offers apposite functionality (see
below) and has been developed and applied in a relevant discipline
(see). In glmulti, information theory is applied to subsets of models
selected by a genetic algorithm (GA) from which the multimodel aver-
age is derived using the function coef. A GA is a type of optimization
that mimics biological evolution. The GA incorporates an immigration
operator, allowing reconsideration of removed variables. Immigration
increases the level of randomisation and hence the likelihood of model
convergence on the global optima (the best models from the available
data) rather than some local optima (Calcagno & de Mazancourt,
2010). Inference from a consensus of five replicate GA runs has been
shown by Calcagno and de Mazancourt (2010) to greatly improve
convergence.
2.6 | Analysis
For each scenario/approach, the best approximating model is derived.
The comparative assessment looks at model structure, modelling error
and statistical uncertainty.
The analysis of the model structures begins with a review of the
selected indices and summary statistics (adjusted R‐squared and P
values). Being evidence‐centric, these statistics are at odds with the
underlying philosophies of information theory (revisited in the discus-
sion). Instead, importance, the relative weight of evidence in support
of each index in the model (Step 3), is considered.
Model error assesses how well the given model simulates the
data, here, the observed data. Analysis centres on relative error,
defined as the measure of error difference divided by observed value.
These errors are presented as an observed‐simulated plot. The distri-
bution and magnitude of modelling errors is further considered
through probability density functions.
Uncertainty is introduced throughout the modelling process. In
this paper, the focus is on statistical uncertainty defined by Warmink
et al. (2010, p.1520) as a measure of “the difference between a simu-
lated value and an observation” and “the possible variation around the
simulated and observed values,” quantified as 1:96·ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffivariance
p, where
1.96 represents the 95% confidence level. Simply put the model with
the least uncertainty, and hence, the most support should be the best
representation of reality. In practice, statistical uncertainty dictates the
usefulness of the model. Inaccurate appreciation of this uncertainty,
however, prevents meaningful interpretation of the results, leading
to less than optimal decision‐making (Warmink et al., 2010).
The type of statistical uncertainty quoted is dependent on the
modelling approach. For the stepwise approach, parameter (condi-
tional) uncertainty, a measure of the parameter variance in the
selected model, is provided. However, model selection represents a
further source of statistical uncertainty (Anderson, 2007); when a
model is derived from a single data set, there is a chance that other
replicate data sets, of the same size and from the same process, would
lead to the selection of different models. As a multimodel average,
information theory provides a measure for this additional uncertainty,
referred to herein as structural uncertainty.
A Monte Carlo approach (MC) is used to explore model parameter
space (uncertainty at the 95% confidence interval represents the
upper/lower bounds). Traditional MC methods suffer from clumping
of points; this occurs because the points “know” (Caflisch, 1998)
VISSER ET AL. 5
nothing about each other. To reduce the number of simulations
required, a Quasi‐MC method (Sobol‐sequence) is applied, where ele-
ments are correlated and more uniformly well‐distributed; 200 simula-
tions appeared sufficient. The relative error distributions (based on the
observed data) are again plotted. An extract of these plots, at the 5/
50/95% densities illustrates the error distribution across the
simulations.
3 | RESULTS
3.1 | Scenario A
3.1.1 | Model structure
The structure of the best approximating models is detailed in Table 1
and Figure 3 (facet 1). The information theory multimodel average,
features five hydrologic indices, with a focus on low flows in summer
and winter (Qs90 and Qw90). The stepwise selected model is similar,
except here the Qs90 index is not present, with this model favouring
less extreme low flows (Qs75).
Summary statistics for the two best approximating models are
detailed in Table 1, with both achieving similar adjusted R‐squared
values. The P value, the principal selection characteristic in stepwise
TABLE 1 Model structures and summary statistics
Scenario Approach Model
A Stepwise selection LIFE = − 3.50Qs50 + 5.45Qs75 −
A Information theory LIFE = − 1.54Qs50 + 1.46Qs75 +
B Stepwise selection LIFE ¼ −4:67Qts50þ 6:71Qt
s75−2
B Information theory LIFE ¼ −0:88Qts50þ 2:57Qt
s90−1
FIGURE 3 Model structure and estimates ofparameter coefficients. Text overlays indicatethe estimate, and for information theory,parameter importance (square brackets). Forscenario B, each facet indicates a time‐offset[Colour figure can be viewed atwileyonlinelibrary.com]
approaches, is distinctly lower in the information theory model. The
second and third best‐performing stepwise selection models saw the
removal of the Qs25 index, and then Qs10 in the final step. These
models have a similar fit to the selected model, with adjusted
R‐squared values of 0.52 and 0.55, respectively. For the information
theory model, an estimate of the relative weight of evidence in
support of each index (Figure 3, facet 1) suggests that the winter
hydrologic indices are the most meaningful.
3.1.2 | Model error
Modelling errors are presented in Figure 4. Overall, there appear to be
minimal differences between the two approaches, with the stepwise
selected model featuring marginally less error. In Figure 4a, it can be
seen that the models perform slightly worse at the extremes, with the
stepwise model achieving a slightly better fit overall. This is
further evidenced in Figure 4b, where errors can be seen to concentrate
on the left. Finally, the fitted distributions in Figure 4c feature consider-
able overlap, further emphasizing the similarities in model performance.
3.1.3 | Model uncertainty
The statistical uncertainty, relative to the parameter estimate, is
summarized in Figure 5 (facet 1); the stepwise selected model
FIGURE 4 Scenario A modelling error. Observed simulated LIFE scores (left); probability density functions (fitted to a normal distribution) ofrelative error (top right); absolute relative error cumulative density functions (bottom right) [Colour figure can be viewed at wileyonlinelibrary.com]
FIGURE 5 Uncertainty (95% confidence
interval) relative to parameter estimates. Forscenario B, each facet indicates a time‐offset[Colour figure can be viewed atwileyonlinelibrary.com]
6 VISSER ET AL.
displays the least uncertainty. Differences are most notable in
hydrological summer, suggesting greater confidence in the winter
indices; this is in agreement with the information theory importance
statistic.
Further inference regarding the implications of statistical uncer-
tainty is made through the consideration of MC simulations
(Figure 6). The cumulative density function (fitted to a normal distribu-
tion; Figure 6a,b) for each simulation provides an overview of the
errors. This is further clarified in Figure 6c), where the errors at cumu-
lative densities of 5/50/95% indicate the distribution of error across
the simulations. For 5% of the data, the majority of the simulations
feature 2.5% absolute error or less; this represents approximately
9% (stepwise) and 16% (information theory) of the simulations.
At 50/95, the errors are similarly spread; the majority of stepwise
models have approximately 20% error, whereas for information
FIGURE 6 Scenario A, distribution of modelling errors following MC simulation. (a,b) Cumulative density function of the absolute relative errorper simulation (fitted to a normal distribution); (c) distribution of the absolute relative error for 5/50/95% of the data [Colour figure can be viewedat wileyonlinelibrary.com]
VISSER ET AL. 7
3.2 | Scenario B
3.2.1 | Model structure
Here, the differences between model structures are greater than Sce-
nario A (Table 1 and Figure 3, facets 2–4). The stepwise selected
model incorporates two nonlagged and two lagged indices. The two
nonlagged parameters represent summer median and moderate low
flows (Qs50 and Qs75). The large coefficients of these two parameters
suggests a preference for mid range flows which are not too low or
high; in this, the scenario B model is broadly consistent with scenario
A. However, the model takes no account of winter flows. In contrast,
the information theory model structures (and measures of parameter
importance) for both scenarios are similar, with the only difference
being the inclusion of lagged winter high flow (t‐2). Physically, this
could represent the time delay of the groundwater recharge. There
is no acknowledgement of this phenomenon in the stepwise selected
model, whether subject to lag or otherwise. In this scenario, the sum-
mary statistics (Table 1, rows 3 and 4) associated with the stepwise
model remain relatively static. However, the adjusted R‐squared for
the information theory model is 14% greater than the stepwise model.
Overall, the information theory model indicates a preference for
variability in flow magnitude, possibly a reflection of the seasonal
nature of the flow regime. Winter flows stand out as the most impor-
tant facet of the flow regime. In contrast, the stepwise selected model
suggests a preference for more uniform flows (that are not too low);
unusually, winter flows are considered unimportant.
3.2.2 | Model error
The errors associated with each model are detailed in Figure 7. At first
glance, Figure 7a suggests that the models perform equally well for
lower LIFE scores, whereas for higher values the information theory
model provides marginally better estimates. This is reinforced in
Figure 7b, where the relative errors are centred around 0% and −4%
for the information theory and stepwise models, respectively. The
stepwise model also has a tendency to overestimate.
The extent of these differences is evident in Figure 7c. For the
information theory model, 56% of the estimated data points have 5%
or less absolute relative error, in fact, almost 50% of the data has less
than 2.5%. This is in direct contrast to the stepwise selected model,
where only 15% of the data has less than 2.5% absolute relative error;
this increases to approximately 48% at 5%. Themodels do not converge
until approximately 9.25% absolute relative error, that is, the largest
errors for both models are comparable.
3.2.3 | Model uncertainty
Relative to scenario A, these is an increase in the range of statistical
uncertainty (Figure 5, facets 2–4), particularly for the information
FIGURE 7 Scenario B modelling error. Observed simulated LIFE scores (left); probability density functions (fitted to a normal distribution) ofrelative error (top right); absolute relative error cumulative density functions (bottom right) [Colour figure can be viewed at wileyonlinelibrary.com]
FIGURE 8 Scenario B, distribution of modelling errors following MC simulation. (a,b) Cumulative density function of the absolute relative errorper simulation (fitted to a normal distribution); (c) distribution of the absolute relative error for 5/50/95% of the data [Colour figure can be viewedat wileyonlinelibrary.com]
Anderson, D. R. (2007). Model based inference in the life sciences. New York:Springer.
Arthington, A. H. (2012). Chapter 9. Introduction to Environmental FlowMethods. In: Environmental Flows: Saving Rivers in the Third Millennium.California: University of California Press.
Bartoń, K. (2018). Package 'MuMIn' Version 1.40.4. Retrieved fromhttps://cran. r‐project.org/web/packages/MuMIn/
Bradley, D. C., Streetly, M. J., Cadman, D., Dunscombe, M., Farren, E., &Banham, A. (2017). A hydroecological model to assess the relativeeffects of groundwater abstraction and fine sediment pressures onriverine macro‐invertebrates. River Research and Applications., 33,1630–1641. https://doi.org/10.1002/rra.3191
Burnham, K. P., & Anderson, D. (2002). Model selection and multi‐modelinference: A pratical information‐theoretic approach. New York: Springer.
Burnham, K. P., Anderson, D. R., & Huyvaert, K. P. (2011). AIC model selec-tion and multimodel inference in behavioral ecology: Somebackground, observations, and comparisons. Behavioral Ecology andSociobiology, 65, 23–35. https://doi.org/10.1007/s00265‐010‐1029‐6
Burnham, K. P., & Anderson, D. R. (2014). P‐values are only an index toevidence: 20th‐ vs. 21st‐century statistical science. Ecology, 95(3),627–630.
Caflisch, R. E. (1998). Monte Carlo and quasi‐Monte Carlo methods. ActaNumer, 7, 1–49. https://doi.org/10.1017/S0962492900002804
Calcagno, V. (2013). glmulti: Model selection and multimodel inferencemade easy. Version 1.0.7. Retrieved from https://cran.r‐project.org/package=glmulti
Calcagno, V., & de Mazancourt, C. (2010). glmulti: An R package for easyautomated model selection with (generalized) linear models. Journal ofStatistical Software, 34, 1–29.
Chadd, R. P., England, J. A., Constable, D., Dunbar, M. J., Extence, C. A.,Leeming, D. J., … Wood, P. J. (2017). An index to track the ecologicaleffects of drought development and recovery on riverine invertebratecommunities. Ecological Indicators, 82, 344–356. https://doi.org/10.1016/j.ecolind.2017.06.058
Clarke, R., & Dunbar, M. (2005). Producing generalised LIFE response curves.Bristol: Environment Agency.
Dahlgren, J. P. (2010). Alternative regression methods are not consideredin Murtaugh (2009) or by ecologists in general. Ecology Letters, 13,E7–E9. https://doi.org/10.1111/j.1461‐0248.2010.01460.x
EA. (2016). River Nar macroinvertebrate monitoring data. (Available uponrequest from the EA.).
Exley, K. (2006). River Itchen macro‐invertebrate community relationship toriver flow changes. Winchester: Environment Agency.
Extence, C. A., Balbi, D. M., & Chadd, R. P. (1999). River flow indexing usingBritish benthic macroinvertebrates: A framework for settinghydroecological objectives. Regulated Rivers: Research & Management,15, 545–574. https://doi.org/10.1002/(sici)1099‐1646(199911/12)15:6<545::aid‐rrr561>3.0.co;2‐w
Greenwood, M. J., & Booker, D. J. (2015). The influence of antecedentfloods on aquatic invertebrate diversity, abundance and communitycomposition. Ecohydrology, 8, 188–203. https://doi.org/10.1002/eco.1499
Grueber, C. E., Nakagawa, S., Laws, R. J., & Jamieson, I. G. (2011).Multimodel inference in ecology and evolution: Challenges and solu-tions. Journal of Evolutionary Biology, 24, 699–711. https://doi.org/10.1111/j.1420‐9101.2010.02210.x
Hebbali, A. (2017). olsrr: Tools for Teaching and Learning OLS Regression.Version 0.3.0. Available: https://CRAN.R‐project.org/package=olsrr
Hegyi, G., & Garamszegi, L. Z. (2011). Using information theory as a substi-tute for stepwise regression in ecology and behavior. BehavioralEcology and Sociobiology, 65, 69–76. https://doi.org/10.1007/s00265‐010‐1036‐7
Hurvich, C. M., & Tsai, C.‐L. (1990). The impact of model selection on infer-ence in linear regression. The American Statistician, 44, 214–217.https://doi.org/10.2307/2685338
Knight, R. R., Brian Gregory, M., & Wales, A. K. (2008). Relating streamflowcharacteristics to specialized insectivores in the Tennessee RiverValley: A regional approach. Ecohydrology, 1, 394–407.
Lake, P. S. (2013). Resistance, resilience and restoration. Ecological Manage-ment & Restoration, 14, 20–24. https://doi.org/10.1111/emr.12016
Lytle, D. A., & Poff, N. L. (2004). Adaptation to natural flow regimes. Trendsin Ecology & Evolution, 19, 94–100. https://doi.org/10.1016/j.tree.2003.10.002
Monk, W. A., Wood, P. J., Hannah, D. M., & Wilson, D. A. (2007). Selectionof river flow indices for the assessment of hydroecological change.River Research and Applications, 23, 113–122. https://doi.org/10.1002/rra.964
Murray‐Bligh, J.A. (1999). Quality management systems for environmentalmonitoring: Biological techniques, BT001. Procedure for collecting andanalysing macro‐invertebrate samples. Version 2.0. Retrieved fromBristol:
Parasiewicz, P., Rogers, J. N., Vezza, P., Gortazar, J., Seager, T., Pegg, M., …Comoglio, C. (2013). Applications of the MesoHABSIM SimulationModel. In I. Maddock, A. Harby, P. Kemp, & P. Wood (Eds.),Ecohydraulics: An integrated approach (pp. 109–124). John Wiley &Sons, Ltd.
Poff, N. L., Allan, J. D., Bain, M. B., Karr, J. R., Prestegaard, K. L., Richter, B.D., … Stromberg, J. C. (1997). The natural flow regime. Bioscience, 47,769–784. https://doi.org/10.2307/1313099
Poff, N. L., & Zimmerman, J. K. H. (2010). Ecological responses to alteredflow regimes: A literature review to inform the science and manage-ment of environmental flows. Freshwater Biology, 55, 194–205.https://doi.org/10.1111/j.1365‐2427.2009.02272.x
R Core Team. (2017). R: A language and environment for statistical com-puting. Retrieved from https://www.r‐project.org/
Sear, D.A., Newson, M., Old, J.C., & Hill, C. (2005). Geomorphologicalappraisal of the River Nar Site of Special Scientific Interest. (N684):English Nature.
Stephens, P. A., Buskirk, S. W., Hayward, G. D., & MartÍNez Del Rio, C.(2005). Information theory and hypothesis testing: A call for pluralism.Journal of Applied Ecology, 42, 4–12. https://doi.org/10.1111/j.1365‐2664.2005.01002.x
Steyerberg, E. W., Eijkemans, M. J., & Habbema, J. D. (1999). Stepwiseselection in small data sets: A simulation study of bias in logistic regres-sion analysis. Journal of Clinical Epidemiology, 52, 935–942.
Surridge, B. W. J., Bizzi, S., & Castelletti, A. (2014). A framework for cou-pling explanation and prediction in hydroecological modelling.Environmental Modelling & Software, 61, 274–286. https://doi.org/10.1016/j.envsoft.2014.02.012
Visser, A., Beevers, L., & Patidar, S. (2017). Macro‐invertebrate communityresponse to multi‐annual hydrological indicators. River Research andApplications, 33, 707–717. https://doi.org/10.1002/rra.3125
Warmink, J. J., Janssen, J. A. E. B., Booij, M. J., & Krol, M. S. (2010). Iden-tification and classification of uncertainties in the application ofenvironmental models. Environmental Modelling & Software, 25,1518–1527. https://doi.org/10.1016/j.envsoft.2010.04.011
Wasserstein, R. L., & Lazar, N. A. (2016). The ASA's statement on p‐values:Context, process, and purpose. The American Statistician, 70, 129–133.https://doi.org/10.1080/00031305.2016.1154108
Whittingham, M. J., Stephens, P. A., Bradbury, R. B., & Freckleton, R. P.(2006). Why do we still use stepwise modelling in ecology and behav-iour? The Journal of Animal Ecology, 75, 1182–1189. https://doi.org/10.1111/j.1365‐2656.2006.01141.x
Wood, P. J., & Armitage, P. D. (2004). The response of the macroinverte-brate community to low‐flow variability and supra‐seasonal droughtwithin a groundwater dominated stream. Archiv für Hydrobiologie,161, 1–20. https://doi.org/10.1127/0003‐9136/2004/0161‐0001
Wood, P. J., Hannah, D. M., Agnew, M. D., & Petts, G. E. (2001). Scales ofhydroecological variability within a groundwater‐dominated stream.Regulated Rivers: Research & Management, 17, 347–367. https://doi.org/10.1002/rrr.658
Worrall, T. P., Dunbar, M. J., Extence, C. A., Laizé, C. L. R., Monk, W. A., &Wood, P. J. (2014). The identification of hydrological indices for thecharacterization of macroinvertebrate community response to flowregime variability. Hydrological Sciences Journal, 59, 645–658. https://doi.org/10.1080/02626667.2013.825722
How to cite this article: Visser AG, Beevers L, Patidar S. Com-
plexity in hydroecological modelling: A comparison of stepwise
selection and information theory. River Res Applic. 2018;1–12.