A two-step-ahead recurrent neural network for stream-flow forecasting

HYDROLOGICAL PROCESSESHydrol. Process. 18, 81–92 (2004)Published online 7 November 2003 in Wiley InterScience (www.interscience.wiley.com). DOI: 10.1002/hyp.1313

A two-step-ahead recurrent neural network forstream-flow forecasting

Li-Chiu Chang, Fi-John Chang* and Yen-Ming ChiangDepartment of Bioenvironmental Systems Engineering, National Taiwan University, Taipei, Taiwan, ROC

Abstract:

In many engineering problems, such as flood warning systems, accurate multistep-ahead prediction is criticallyimportant. The main purpose of this study was to derive an algorithm for two-step-ahead forecasting based on areal-time recurrent learning (RTRL) neural network that has been demonstrated as best suited for real-time applicationin various problems. To evaluate the properties of the developed two-step-ahead RTRL algorithm, we first comparedits predictive ability with least-square estimated autoregressive moving average with exogenous inputs (ARMAX)models on several synthetic time-series. Our results demonstrate that the developed two-step-ahead RTRL network hasefficient ability to learn and has comparable accuracy for time-series prediction as the refitted ARMAX models. Wethen investigated the two-step-ahead RTRL network by using the rainfall–runoff data of the Da-Chia River in Taiwan.The results show that the developed algorithm can be successfully applied with high accuracy for two-step-aheadreal-time stream-flow forecasting. Copyright 2003 John Wiley & Sons, Ltd.

KEY WORDS recurrent neural networks; stream flow; rainfall–runoff modelling; multistep ahead

INTRODUCTION

Stream-flow forecasting is crucial for water management and flood warning. Long-term forecasts are generallyused for distributing water for irrigation and mitigating drought. Short-term forecasts, with lead times ofhours and days, are necessary for flood warning and real-time reservoir operation. Taiwan has a subtropicalclimate where typhoons, usually coupled with heavy rainfall, hit the island around four times a year, causingdownstream flooding within a few hours. A short-term stream-flow forecast is, hence, particularly importantfor Taiwan. Owing to the geophysical conditions, most reservoirs in Taiwan are relatively small comparedwith the amount of water falling on the watershed in a typhoon. Although a typhoon can cause flooding, italso can be the most important water source. Consequently, failure to operate a reservoir might cause seriousdamage to the dam itself and/or increase downstream flooding. On the other hand, if a typhoon does not bringenough water while the reservoir is emptied for flood control purposes, it can cause a serious water deficit forthe coming drought season. Clearly, successful water management requires accurate stream-flow forecasting.

Various stream-flow forecasting models have been proposed, which broadly can be classified into physicallybased processes and stochastically based approaches. The physically based process may not be feasible,especially in our case, because:

1. the stream flow is an end product of a number of complex rainfall–runoff processes in a watershed, whichvary both in time and space;

2. the data that are available to assist in definition of control variables for the processes such as rainfallintensity, evaporation, infiltration rate, runoff-coefficients, etc., are limited in both the spatial and tempo-ral dimensions.

* Correspondence to: Fi-John Chang, Department of Bioenvironmental Systems Engineering and Hydrotech Research Institute, NationalTaiwan University, Taipei, Taiwan, 10617 ROC. E-mail: [email protected]

Received 8 April 2002Copyright 2003 John Wiley & Sons, Ltd. Accepted 4 February 2003

82 L-C. CHANG, F-J. CHANG AND Y-M. CHIANG

Traditionally, the stochastically based approach for stream-flow forecast has been tackled by using a lineartechnique, such as autoregression (AR), autoregressive moving average with exogenous inputs (ARMAX)(Salas et al., 1985) and also non-linear regression (Chang and Hwang, 1999). In the past several years,the use of artificial neural networks (ANNs) in hydrological modelling has seen rapid growth, such asrainfall estimation in forecasting (Hsu et al., 1999; Luk et al., 2001), real-time reservoir operation (Changand Chang, 2001) and stream-flow forecasting (Atiya et al., 1999; Sajikumar et al., 1999; Govindaraju andRao, 2000; Chang and Chen, 2001; Chang et al., 2001). Most of these applied neural networks have used amultilayer feed-forward network, namely multilayer perceptron (MLP) (Maier and Dandy, 2000). The MLPis a static and memory-less network that is powerful and effective for non-linear static mapping, however, itcannot effectively track, temporal processing (Ham and Kostanic, 2001). Hydrological time-series, especiallythe runoff in a storm event, involve a significant temporal process, consequently, the MLP networks havedifficulties in modelling these processes.

Recently, dynamic recurrent neural networks (RNNs) have been attracting much attention from the scientificcommunity (Campolucci et al., 1999; Zhou and Xu, 2001), because they are very effective for temporalprocessing. Recurrent neural networks (RNNs) are basically dynamic systems where the states evolveaccording to certain non-linear state equations. Because of their dynamic nature, RNNs have been appliedsuccessfully to a wide variety of problems, such as system identification (Zhang and Wang, 2001), financialprediction (Saad et al., 1998) and stream-flow forecasting (Coulibaly et al., 2001). The main purpose of thisstudy was to enhance our last study (Chang et al., 2002), which used an RNN for one-step-ahead stream-flowforecasting and devised a two-step-ahead RNN algorithm for stream-flow forecasting.

THE MULTISTEP-AHEAD PROBLEM

Most of the forecasting models consider one-step-ahead forecasts. In many engineering problems, suchas reservoir operation and flood warning systems, it is very beneficial and desirable that the modelsprovide accurate multistep-ahead prediction. Multistep-ahead prediction, however, is very difficult to achievesuccessfully, because there is a lack of real data in the future to adjust the model’s performance. The model,in general, can only necessitate the recursive use of one-step-ahead predictions for many time-steps into thefuture. Consequently, a small one-step-ahead prediction error at the beginning can accumulate and propagateto the future, resulting in poor prediction accuracy.

Only a few earlier studies have successfully used neural networks for multistep prediction. Based on back-propagation through time, Su et al. (1992) proposed long-term predictions of chemical processes. Schenker andAgarwal (1995) reported long-range prediction for poorly known systems through training distinct networks.A long-range prediction strategy was proposed by Prasad et al. (1998). Several neural networks for multistepprediction in time—series have recently proposed empirical models for multistep ahead prediction (Parloset al., 2000).

In this study, our goal was to derive an algorithm of two-step-ahead forecasting based on the real-timerecurrent learning (RTRL) neural network that was presented by Williams and Zipser (1989) and had beenwidely used in various problems (Meert, 1998; Tresp et al., 1999; Hirasawa, 2000; Benson and Carrasco,2001; Li, 2002). The original RTRL algorithm derives its name from the fact that adjustments are madeto the synaptic weights of a fully connected recurrent network in real time and it is best suited for realtime applications (Haykin, 1999; Mandic and Chambers, 2000). Figure 1 shows the architecture of such arecurrent network.

TWO-STEP-AHEAD REAL-TIME RECURRENT LEARNING (RTRL) ALGORITHM

The original RTRL algorithm is used for one-step-ahead forecasting. Based on the original RTRL architecture,we derive a two-step-ahead RTRL algorithm and its architecture is shown in Figure 2. There are M external

Copyright 2003 John Wiley & Sons, Ltd. Hydrol. Process. 18, 81–92 (2004)

RECURRENT NEURAL NETWORK FOR STREAM-FLOW FORECASTING 83

Figure 1. The architecture of a one-step real-time recurrent learning network

Figure 2. The architecture of a two-step real-time recurrent learning network

inputs and K outputs. Let x(t) denote the M ð 1 input vector to the network at discrete time t, y(t C 1)denote the corresponding N ð 1 vector one step later at time t C 1 in the processing layer, and z(t C 1) andz(t C 2) denote the corresponding K ð 1 output vector for the original (one-step-ahead) and the two-step-ahead RTRL, respectively. It is easy to find that the only difference between these two architectures is thetime in the output layer. In the original RTRL algorithm, the processing layer to the output layer performs asimultaneous mapping process, i.e. from y(t C 1) to z(t C 1). In the two-step-ahead RTRL algorithm, however,the processing layer to the output performs a one-step-ahead mapping process, i.e. from y(t C 1) to z(t C 2).The detailed algorithms of two-step-ahead RTRL are shown as follows.

The input x(t) and one-step delayed output vector in the processing layer y(t) are concatenated to formthe �M C N� ð 1 vector m(t), whose i th element is denoted by mi(t). Let A denote the set of indices i forwhich xi(t) is an external input, and B denote the set of indices i for which yi(t) is the output of a unit inthe network. We thus have

mi�t� D{

xi�t� if i 2 Ayi�t� if i 2 B

�1�

The network is fully interconnected. There are M ð N forward connections and N ð N feedback connections.Let W denote the N ð �M C N� recurrent weight matrix of the network.

Copyright 2003 John Wiley & Sons, Ltd. Hydrol. Process. 18, 81–92 (2004)

84 L-C. CHANG, F-J. CHANG AND Y-M. CHIANG

The processing and output layers are also fully connected. Let V denote the N ð K weight matrix. W $ wji

and V $ vkj are the matrix form.The net activity of neuron j at time t C 1, for j 2 B, is computed by

netj�t C 1� D∑

i2A[B

wji�t�mi�t� �2�

The output of neuron j in the processing layer is given by passing netj�t C 1� through the non-linearityf (.), yielding

yj�t C 1� D f�netj�t C 1�� �3�

The net output of neuron k in the output layer at time t C 2 is computed by

netk�t C 2� D∑

vkj�t C 1�yj�t C 1� �4�

zk�t C 2� D f�netk�t C 2�� �5�

Let dk�t C 2� denote the target value of neuron k at time t C 2. Then we define a time-varying K ð 1 errorvector e(t C 2) whose k th element is

ek�t C 2� D dk�t C 2� � zk�t C 2� �6�

Define the instantaneous overall network error at time t C 2 as

E�t C 2� D 1

2

K∑kD1

e2k �t C 2� �7�

The cost function is obtained by summing E�t C 2� over all time T

Etotal DT∑

tD1

E�t C 2� �8�

To minimize the cost function, the gradient descent method is applied to adjust the weights (V and W)along the negative of rEtotal. Because the total error is the sum of the errors at the individual time-steps,one way to compute this gradient is by accumulating the value of rE for each time-step along the trajectory.Then, the total Vkj is the sum of vkj for each time-step within the range from the start step t0 to the endstep t1.

Vkj Dt1∑

tDt0C1

vkj�t C 1� �9�

The weight change for any particular weight vkj can thus be written as

vkj�t C 1� D ��1∂E�t C 2�

∂vkj�t C 1��10�

where �1 is the learning-rate parameter.Now

∂E�t C 2�

∂vkj�t C 1�D �ek�t C 2�f0�netk�t C 2��yj�t C 1� �11�

The same method is also implemented for weight wmn where

wmn�t� D ��2∂E�t C 2�

∂wmn�t��12�

Copyright 2003 John Wiley & Sons, Ltd. Hydrol. Process. 18, 81–92 (2004)

RECURRENT NEURAL NETWORK FOR STREAM-FLOW FORECASTING 85

and �2 is the learning-rate parameter. The partial derivative ∂E�t C 2�/∂wmn�t� of equation (12) can be obtainedby the chain rule for differentiation as follows

∂E�t C 2�

∂wmn�t�D

[K∑

kD1

�ek�t C 2�f0�netk�t C 2��vkj�t C 1�

]∂yj�t C 1�

∂wmn�t��13�

) ∂yj�t C 1�

∂wmn�t�D f0�netj�t C 1��

∂netj�t C 1�

∂wmn�t��14�

) ∂netj�t C 1�

∂wmn�t�D

∑i2A[B

∂�wji�t�mi�t��

∂wmn�t��15�

) ∂netj�t C 1�

∂wmn�t�D

∑i2A[B

[wji�t�

∂mi�t�

∂wmn�t�C ∂wji�t�

∂wmn�t�mi�t�

]�16�

The derivative ∂wji�t�/∂wmn�t� is equal to 1 only when j D m and i D n; otherwise, it is zero. We maytherefore rewrite Equation (16) as

∂netj�t C 1�

∂wmn�t�D

∑i2A[B

wji�t�∂mi�t�

∂wmn�t�C υmjmn�t� �17�

where υmj is the Kronecker delta with value 1 if and only if j D m; otherwise zero.From the definition of mi�t� in Equation (1), we also note that

∂mi�t�

∂wmn�t�D

{0 if i 2 A

∂yi�t�∂wmn�t� if i 2 B �18�

We assume that the initial state of the network at time t D 0 has no functional dependence on the synapticweights, so that

∂yj�0�

∂wmn�0�D 0 �19�

∂yj�t C 1�

∂wmn�t�D f0netj�t C 1��

[∑i2B

wji�t�∂yi�t�

∂wmn�t�C υmjmn�t�

]�20�

Let ∂yj�t C 1�/∂wmn�t� ³ ∂yj�t C 1�/∂wmn�t C 1� and define a dynamic system described by a triple-

indexed set of variables f�jmng, where �j

mn�t� D ∂yj�t�∂wmn�t� for all j 2 B, m 2 B and n 2 A [ B.

For each time-step t and all appropriate m, n and j, the dynamics of the system are governed by

�jmn�t C 1� D f0�netj�t C 1��

[∑i2B

wji�t��imn�t� C υmjmn�t�

]�21�

with initial condition �jmn�0� D 0.

Then the weight changes can be computed from Equations (12)–(21) as

wmn�t� D �2

[∑ek�t C 2�f0�netk�t C 2��vkj�t C 1�

]�j

mn�t C 1� �22�

vkj�t C 1� D �1ek�t C 2�f0�netk�t C 2��yj�t C 1� �23�

Copyright 2003 John Wiley & Sons, Ltd. Hydrol. Process. 18, 81–92 (2004)

86 L-C. CHANG, F-J. CHANG AND Y-M. CHIANG

TESTED AND VERIFIED BY USING SYNTHETIC TIME-SERIES

The autoregressive moving average with exogenous inputs (ARMAX) model, which was presented by Box andJenkins (1976), has been used frequently for hydrological time-series generation and forecasting (Salas et al.,1985). To show the robustness and usefulness of RTRL, we compared the predictive ability of least-squareestimated ARMAX predictors and two-step RTRL on three different time-series of synthetic data.

The three time-series models are

AR(1): �xt � 0Ð4� D 0Ð75�xt�1 � 0Ð4� C et with et ¾i.i.d N�0, 1Ð0� �24�

AR(2): �xt � 0Ð4� D 0Ð6�xt�1 � 0Ð4� C 0Ð2�xt�2 � 0Ð4� C et with et ¾i.i.d N�0, 1Ð0� �25�

ARMAX(2,1;3): x�t� D 0Ð6x�t � 1� C 0Ð2x�t � 2� C 0Ð82u1�t � 2� � 0Ð15u1�t � 3� C 0Ð25u1�t � 4�

C 0Ð78u2�t � 2� C 0Ð2u2�t � 3� C 0Ð16u2�t � 4� C 0Ð85u3�t � 2�

� 0Ð35u3�t � 3� C 0Ð02u3�t � 4� C 0Ð92u4�t � 2� C 0Ð1u4�t � 3� �26�

C 0Ð35u4�t � 4� C e�t� � 0Ð7e�t � 1� with et ¾i.i.d N�0, 1Ð0�

The et vectors are chosen from an independent and identically normal distribution (i.i.d) with mean zero andvariance one, and the input vectors u are generated from a random gaussian signal (type D ‘rgs’ in MATLAB,the Maths Works, Inc). Each series is generated by MATLAB. After the first 50 generated data are skipped,the initial 100 data are used to construct the RTRL network and to re-estimate the optimal parameters ofthe ARMAX models. For simplicity, without model identification, we use the given types of the ARMAXmodels, and only their parameters are estimated. The least-square estimator in MATLAB is used to estimatethe ARMAX models’ parameters.

The two-step-ahead prediction is performed based on (i) the constructed RTRL networks and (ii) the fittedARMAX models on each case, respectively. As the prediction focuses on optimum prediction by the minimummean squared error, the criteria of mean square error (MSE), mean absolute error (MAE) and the goodness-of-fit with respect to the benchmark (Gbench) (Nash and Sutcliffe, 1970; Seibert, 2001) are used for the purposeof comparison. These are defined as

MSE D

n∑iD1

� OQi � Qi�2

n�27�

MAE D

n∑iD1

j OQi � Qij

n�28�

Gbench D 1 �

n∑iD1

�Qi � OQi�2

n∑iD1

�Qi � QQi�2

�29�

where Qi is the observed value in i step, OQi is the forecasted value in i step, and QQi is the previous observedvalue in i step, which is QQi D Qi�2 in our case.

To learn the models’ performance at different stages, the models’ predictions (two-step ahead) in (i) trainingsets (100 data) and (ii) testing sets (200 data) at three different periods are presented, respectively. The resultsare summarized in Tables I–III. The results of RTRL in the training set indicate that after about 30 stepson-line input training, RTRL can grasp the major trends in all cases. All of those indicate RTRL has efficient

Copyright 2003 John Wiley & Sons, Ltd. Hydrol. Process. 18, 81–92 (2004)

RECURRENT NEURAL NETWORK FOR STREAM-FLOW FORECASTING 87

ability for learning and high accuracy. The results in the testing sets indicate that (i) the RTRL networks havecomparable MAE, MSE and Gbench values to those of refitted AR (1) and AR (2) models, and (ii) the RTRLnetworks have better results than ARMAX models. Moreover, the developed RTRL networks could performthe two-step ahead forecasting more consistently than the refitted ARMAX models.

APPLICATION

The above methodology is then applied to the upstream part of the Da-Chia River for predicting real-timestream flow. The Da-Chia River is located in central Taiwan with a total catchment size of 1236 km2. Being

Table I. The performance of fitted autoregression (AR(1)) and two-step real-time recurrent learning (RTRL) models atdifferent stages

Time step AR(1) Two-step RTRL

MAE MSE Gbench MAE MSE Gbench

Training 1–30 1Ð049 1Ð680 0Ð103 1Ð297 2Ð831 �0Ð5781–100 0Ð951 1Ð452 0Ð155 1Ð107 1Ð920 �0Ð040

Testing 101–130 1Ð002 1Ð612 0Ð094 0Ð879 1Ð326 0Ð246151–180 1Ð137 1Ð837 0Ð241 1Ð077 1Ð737 0Ð281271–300 0Ð754 0Ð965 0Ð254 0Ð872 1Ð379 0Ð214101–300 0Ð980 1Ð530 0Ð199 0Ð984 1Ð607 0Ð186

Table II. The performance of fitted autoregression (AR(2)) and two-step real-time recurrent learning (RTRL) models atdifferent stages

Time step AR(2) Two-step RTRL

MAE MSE Gbench MAE MSE Gbench

Training 1–30 0Ð726 0Ð932 0Ð405 1Ð046 2Ð753 �0Ð7051–100 0Ð878 1Ð206 0Ð211 0Ð987 1Ð794 �0Ð057

Testing 101–130 1Ð033 1Ð687 �0Ð003 1Ð016 1Ð652 0Ð018151–180 0Ð879 1Ð064 0Ð102 0Ð850 0Ð988 0Ð166271–300 0Ð894 1Ð014 0Ð198 0Ð790 1Ð012 0Ð199101–300 0Ð939 1Ð292 0Ð134 0Ð914 1Ð294 0Ð133

Table III. The performance of fitted ARMAX(2,1;3) and real-time recurrent learning (RTRL) models at different stages

Time step ARMAX(2,1;3) Two-step RTRL

MAE MSE Gbench MAE MSE Gbench

Training 1–30 0Ð955 1Ð328 0Ð658 1Ð375 3Ð327 0Ð1451–100 1Ð283 2Ð488 0Ð336 1Ð269 2Ð750 0Ð422

Testing 101–130 1Ð870 4Ð628 �0Ð321 1Ð196 2Ð084 0Ð405151–180 1Ð185 2Ð094 0Ð344 1Ð056 1Ð658 0Ð481271–300 1Ð338 2Ð554 0Ð402 1Ð349 2Ð685 0Ð372101–300 1Ð317 2Ð501 0Ð308 1Ð165 2Ð041 0Ð436

Copyright 2003 John Wiley & Sons, Ltd. Hydrol. Process. 18, 81–92 (2004)

88 L-C. CHANG, F-J. CHANG AND Y-M. CHIANG

Figure 3. Location of gauge stations in the upstream part of the Da-Chia River

the steepest channel in Taiwan, the Da-Chia River has a length of 140 km and an average channel slopeof 1/39. A series of hydraulic structures were constructed for power generation and water supply for theTaichung metropolitan area. Locations of the basin (area D 514 km2) and the gauge stations used are shownin Figure 3. A station for the stream-flow data is denoted by a triangle and the precipitation stations bycircles. Son-Mou gauge station was established to measure the inflow of the De-Chi Reservoir, the upmostand pivotal reservoir in the Da-Chia River. Accurate stream-flow forecasting is extremely important for theoperation of the De-Chi Reservoir. The streamflow (cms) and precipitation (mm/h) data used here are gatheredfrom Taiwan Power Company.

There are four rainfall gauges above the Son-Mou stream-flow gauge. To construct two-step-ahead stream-flow forecasting, we investigated the developed RTRL and ARMAX models’ performance based on currentrainfall at four gauges and stream flow. The performances of these two methods are presented based on thecriteria of MAE, relative mean absolute error (RMAE) and Gbench. The relative mean absolute error (RMAE)is defined as

RMAE D MAE

Q�30�

The hourly data of 1995 are used to construct the two-step-ahead ARMAX and the RTRL network. TheARMAX(2,1;3) model is

OS�t C 2� D a1 OS�t C 1� C a2S�t� C2∑

iD0

b1iR1�t � i� C2∑

iD0

b2iR2�t � i� C2∑

iD0

b3iR3�t � i�

C2∑

iD0

b4iR4�t � i� C c1e�t C 1� �31�

Copyright 2003 John Wiley & Sons, Ltd. Hydrol. Process. 18, 81–92 (2004)

RECURRENT NEURAL NETWORK FOR STREAM-FLOW FORECASTING 89

Figure 4. The architecture of the real-time recurrent learning network for two-step-ahead stream-flow (forecasting of the Da-Chia River)

1200observationtwo-step RTRLARMAX1000

800

cms

600

400

200

00 1000 2000 3000 4000 5000 6000

hr7000 8000 9000

Figure 5. Two-step-ahead stream-flow forecasting by the ARMAX and the two-step real-time recurrent learning (RTRL) models in thevicinity of annual peak flow in 1996

where S(t) is streamflow of the Son-Mou stream-flow gauge at time t ; OS�t C 2�, and OS�t C 1� are estimatedstream flow at time t C 2 and t C 1 by the model; R1�t � i�, R2�t � i�, R3�t � i� and R4�t � i� are precipitationat four rainfall gauges at time t � i, respectively; e�t C 1� is the model error at time t C 1; and a1, bij and c1

are the model’s parameters to be estimated. The model and its parameters can be obtained from MATLAB.The input layer of RTRL is established by using the current four rain-gauge and stream-flow data, and the

processing layer is constructed to have five nodes. The architecture of the network is illustrated in Figure 4.

Copyright 2003 John Wiley & Sons, Ltd. Hydrol. Process. 18, 81–92 (2004)

90 L-C. CHANG, F-J. CHANG AND Y-M. CHIANG

Table IV. Table of model results for different years for the Da-Chia River: Qo, annual mean of observed stream flow (cms);Qp, observed annual peak flow (cms); OQp, estimated annual peak flow (cms)

Year Qo Qp ARMAX(2,1;3) Two-step RTRL

MAE RMAE OQp Gbench MAE RMAE OQp Gbench

1995 19Ð01 105 0Ð440 0Ð023 101 0Ð209 0Ð388 0Ð020 96 0Ð2661996 24Ð04 1090 0Ð886 0Ð037 987 0Ð036 0Ð676 0Ð028 1089 0Ð3761997 24Ð04 798 0Ð925 0Ð038 667 0Ð009 0Ð757 0Ð031 762 0Ð1921998 32Ð24 713 1Ð090 0Ð034 677 0Ð037 0Ð918 0Ð028 691 0Ð3381999 21Ð55 203 0Ð738 0Ð034 194 0Ð138 0Ð686 0Ð032 194 0Ð2172000 33Ð92 1150 1Ð658 0Ð049 859 0Ð017 1Ð548 0Ð046 923 0Ð221

00 1000 2000 3000

hr4000 5000 6000 7000 8000 9000

100

200

300

400

500

600

cms

700

800

observationtwo-step RTRLARMAX

Figure 6. Two-step-ahead stream-flow forecasting by the ARMAX and the two-step real-time recurrent learning (RTRL) models in thevicinity of annual peak flow in 1998

The learning rate �1 is 40Ð0 and �2 is 80Ð0. We found that after 30 steps of on-line input training, RTRL canappropriately predict the 2-h ahead stream flow; that is, the network tends to be stable in forecasting ability.The structures of two-step-ahead ARMAX and RTRL are then applied to six different years without anyfurther modifications. Summarized results of both methods are presented in Table IV. The results show thatRTRL has smaller MAE and RMAE values and much higher Gbench values than ARMAX in all cases. TheARMAX can only produce small Gbench values (closed to zero) for all the test years (1996 to 2000), whereasthe RTRL produces high Gbench values (higher than 0 Ð 25) for most of the years. Apparently, RTRL has muchbetter performance than ARMAX. Figures 5–7 show the observed and forecast stream flow by RTRL andARMAX in 1996, 1998 and 2000, respectively. In these figures, the observed and forecast histograms of thepeak flow are circled and enlarged, so that the two models’ performance are easily distinguished. The resultsdemonstrate that the RTRL can well forecast two-step-ahead stream-flow values, whereas the ARMAX has

Copyright 2003 John Wiley & Sons, Ltd. Hydrol. Process. 18, 81–92 (2004)

RECURRENT NEURAL NETWORK FOR STREAM-FLOW FORECASTING 91

00 1000 2000 3000 4000 5000 6000

hr7000 8000 9000

200

400

600

800

1000

cms

1200observation

two-step RTRL

ARMAX

1400

Figure 7. Two-step-ahead stream-flow forecasting by the ARMAX and the two-step real-time recurrent learning (RTRL) models in thevicinity of annual peak flow in 2000

a significant time-shift problem and can not well forecast two-step-ahead stream flow. Although the devisedRTRL has a slight time-shift problem, the forecast, in general, is quite good and adequate.

CONCLUSIONS

Most of the presented neural networks for hydrosystem modelling are classified into static neural networks,which can only simulate the short-term memory structures within processes; consequently the extraordinarytime variate characteristics of hydrological time-series, especially in our case, could not be modelled well. Thereal-time recurrent learning (RTRL) networks, which have a representation of dynamic internal feedback loopsto store information for later use and to enhance the efficiency of learning, provide an alternative approachof the ANNs for stream forecasting and demonstrate its applicability and high accuracy in our previous studyfor one-step-ahead forecasting (Chang et al., 2002).

Many engineering problems, such as flood warning systems, require that the models can provide multistep-ahead prediction. In this study, we derived a two-step-ahead RTRL algorithm and demonstrated its effectivelearning ability and accurate forecasting ability by using a synthetic time-series model and Da-Chia Riverdata. For the purpose of comparison, two-step-ahead forecasting by using fitted ARMAX models was alsoperformed. In the case of three synthetic ARMAX time-series, the results indicate that the developed two-step-ahead RTRL algorithm has comparable performance to the least-square refitted ARIMA models fortwo-step-ahead prediction in all cases. In the case of building a rainfall–runoff model for the Da-Chia River,the developed RTRL network has much better performance on two-step-ahead stream flow than that of the fittedARMAX model. This study demonstrates that the developed two-step RTRL network has high practicabilityand accurate forecasting for two-step-ahead real-time stream-flow forecasting.

Copyright 2003 John Wiley & Sons, Ltd. Hydrol. Process. 18, 81–92 (2004)

92 L-C. CHANG, F-J. CHANG AND Y-M. CHIANG

ACKNOWLEDGEMENTS

This paper is based on partial work supported by National Science Council, R.O.C. (Grant No. NSC90-2811-B-002-037). In addition, the authors are sincerely grateful to the reviewers for their valuable comments andsuggestions.

REFERENCES

Atiya AF, El-Shoura SM, Shaheen SI, El-Sherif MS. 1999. A comparison between neural-network forecasting techniques—case study: riverflow forecasting. IEEE Transactions on Neural Networks 10(2): 402–409.

Benson M, Carrasco RA. 2001. Linear and non-linear space diversity combining algorithms over fading channels in TDMA systems.International Journal of Communication Systems 14(6): 619–632.

Box GEP, Jenkins GM. 1976. Time Series Analysis Forecasting and Control , 2nd edn. Holden-Day: San Francisco.Campolucci P, Uncini A, Piazza F, Rao BD. 1999. On-line learning algorithms for locally recurrent neural networks. IEEE Transactions on

Neural Networks 10(2): 253–271.Chang LC, Chang FJ. 2001. Intelligent control of modelling of real time reservoir operation. Hydrological Processes 15(9): 1621–1634.Chang FJ, Chang LC, Huang HL. 2002. Real-time recurrent learning neural network for stream-flow forecasting. Hydrological Processes

16(13): 2577–2588.Chang FJ, Chen YC. 2001. A counterpropagation fuzzy-neural network modeling approach to real-time streamflow prediction. Journal of

Hydrology 245: 153–164.Chang FJ, Hu HF, Chen YC. 2001. Counterpropagation fuzzy-neural network for streamflow reconstructing. Hydrological Processes 15(2):

219–232.Chang FJ, Hwang YY. 1999. A self-organization algorithm for real-time flood forecast. Hydrological Processes 13(2): 123–138.Coulibaly P, Anctil F, Bobee B. 2001. Multivariate reservoir inflow forecasting using temporal neural networks. Journal of Hydrological

Engineering 6(5): 367–376.Govindaraju RS, Rao AR (eds). 2000. Artificial Neural Networks in Hydrology . Kluwer: Rotterdam.Ham FM, Kostanic I. 2001. Principles of Neurocomputing for Science and Engineering . McGraw-Hill: Boston Burr Ridge, IL.Haykin S. 1999. Neural Networks: a Comprehensive Foundation, 2nd edn. Prentice Hall: Upper Saddle River, NJ.Hirasawa K, Murata J, Hu J, Jin C. 2000. Universal learning network and its application to robust control. IEEE Transactions on Systems

Man and Cybernetics: Part B Cybernetics 30(3): 419–430.Hsu K-L, Gupta HV, Gao XG, Sorooshian S. 1999. Estimation of physical variables from multichannel remotely sensed imagery using a

neural network: application to rainfall estimation. Water Resources Research 35(5): 1605–1618.Li CG, He SB, Liao XF, Yu JB. 2002. Using recurrent neural network for adaptive predistortion linearization of RF amplifiers. International

Journal of RF and Microwave Computer-aided Engineering 12(1): 125–130.Luk KC, Ball JE, Sharma A. 2001. An application of artificial neural networks for rainfall forecasting. Mathematical and Computer Modelling

33: 683–693.Maier HR, Dandy GC. 2000. Neural networks for prediction and forecasting of water resources variables: review of modelling issues and

applications. Environmental Modelling and Software 15: 101–124.Mandic DP, Chambers JA. 2000. A normalised real time recurrent learning algorithm. Signal Processing 80: 1909–1916.Meert K. 1998. A real-time recurrent learning network structure for data reconciliation. Artificial Intelligence in Engineering 12(3): 213–218.Nash JE, Sutcliffe JV. 1970. River flow forecasting through conceptual models, part I—a discussion of principles. Journal of Hydrology

10: 282–290.Parlos AG, Rais OT, Atiya AF. 2000. Multi-step-ahead prediction using dynamic recurrent neural networks. Neural Networks 13: 765–786.Prasad G, Swidenbank E, Hogg BW. 1998. A neural net model-based mutivariable long-range predictive control strategy applied in thermal

power plant control. IEEE Transactions on Energy Conversion 13: 176–182.Saad EW, Prokhorov DV, Wunsch DC. 1998. Comparative study of stock trend prediction using time delay, recurrent and probabilistic

neural networks. IEEE Transactions on Neural Networks 9(6): 1456–1470.Sajikumar N, Thandaveswara BS. 1999. A non-linear rainfall–runoff model using an artificial neural network. Journal of Hydrology 216:

32–55.Salas JD, Delleur JW, Yevjevich V, Lane WL. 1985. Applied Modeling of Hydrologic Time Series . Water Resources Publications: Littleton,

Colorado.Schenker B, Agarwal M. 1995. Long-range prediction for poorly-known systems. International Journal of Control 62: 227–238.Seibert J. 2001. On the need for benchmarks in hydrological modeling. Hydrological Processes 15(6): 1063–1064.Su HT, McAvoy TJ, Werbos P. 1992. Long-term prediction of chemical processes using recurrent neural networks: A parallel training

approach. Industrial Applications of Chemical Engineering Research 31: 1338–1352.Tresp V, Briegel T, Moody J. 1999. Neural-network models for the blood glucose metabolism of a diabetic. IEEE Transactions on Neural

Network 10(5): 1204–1213.Williams R, Zipser D. 1989. A learning algorithm for continually running fully recurrent neural network. Neural Computation 1: 270–280.Zhang Y, Wang J. 2001. Recurrent neural networks for nonlinear output regulation. Automatica 37: 1161–1173.Zhou SM, Xu LD. 2001. A new type of recurrent fuzzy neural network for modeling dynamic systems. Knowledge-based Systems 14:

243–251.

Copyright 2003 John Wiley & Sons, Ltd. Hydrol. Process. 18, 81–92 (2004)