CSIRO Land and Water Science Report 37/07 · Science Report 37/07. Deriving moisture availability from time series ... (Print format) ISBN 978 0 643 09506 9 (Web format) CSIRO Land

Tim R. McVicar, Tom G. Van Niel, Lingtao Li, Edward A. King and Randall J. Donohue

Deriving moisture availability from time series remote sensing for ecohydrological applications: Development of a prototype near real-time operational system.

CSIRO Land and Water Science Report 37/07

For further information:

CSIRO Land and WaterTim McVicarPhone: (02) 6246 5741Email: [email protected]: www.clw.csiro.au

Deriving m

oisture availability from tim

e series remote sensing C

SIRO Land and W

ater Science Report 37/07

Deriving moisture availability from time series remote sensing for ecohydrological applications:

Development of a prototype near real-time operational system.

Tim R. McVicar, Tom G. Van Niel, Lingtao Li,

Edward A. King and Randall J. Donohue


June 2007

Copyright and Disclaimer © 2007 CSIRO To the extent permitted by law, all rights are reserved and no part of this publication covered by copyright may be reproduced or copied in any form or by any means except with the written permission of CSIRO. Important Disclaimer: CSIRO advises that the information contained in this publication comprises general statements based on scientific research. The reader is advised and needs to be aware that such information may be incomplete or unable to be used in any specific situation. No reliance or actions must therefore be made on that information without seeking prior expert professional, scientific and technical advice. To the extent permitted by law, CSIRO (including its employees and consultants) excludes all liability to any person for any consequences, including but not limited to all losses, damages, costs, expenses and any other compensation, arising directly or indirectly from using this publication (in part or in whole) and any information or material contained in it. Authors: Tim R. McVicar 1, Tom G. Van Niel 2, Lingtao Li 3, Edward A. King 4 and Randall J. Donohue5 1 CSIRO Land and Water and eWater CRC, GPO Box 1666, Canberra, 2601, ACT, Australia Tel.: +61-2-6246-5741, e-mail: [email protected]

2 CSIRO Land and Water and eWater CRC, Private Bag No. 5, Wembley, 6913,WA, Australia Tel.: +61-8-9333-6705, e-mail: [email protected] 3 CSIRO Land and Water and eWater CRC, GPO Box 1666, Canberra, 2601, ACT, Australia Tel.: +61-2-6246-5809, e-mail: [email protected] 4 CSIRO Marine and Atmospheric Research, GPO Box 3023, Canberra, 2601, ACT, Australia Tel.: +61-2-6246-5894, e-mail: [email protected] 5 CSIRO Land and Water and eWater CRC, GPO Box 1666, Canberra, 2601, ACT, Australia Tel.: +61-2-6246-5803, e-mail: [email protected] Cover: Time series of monthly Normalised Difference Temperature Index (NDTI) composites from April to September 1995 are shown starting from the top-left and ending at the bottom-right. When NDTI is low (shown in red) there is little moisture available for actual evapotranspiration (i.e., the environment is relatively dry) and when NDTI is high (shown in blue) moisture is freely available so actual evapotranspiration is occurring near its potential (i.e., the environment is relatively wet). Black is the ocean. For bibliographic purposes, this report may be cited as: McVicar T.R., Van Niel T.G., Li, L.T., King, E.A. and Donohue, R.J. (2007) Deriving moisture availability from time series remote sensing for ecohydrological applications: Development of a prototype near real-time operational system. CSIRO Land and Water Science Report 37/07, Canberra, Australia, 144 pp. A PDF version is available at: http://www.clw.csiro.au/publications/science/2007/sr37-07.pdf ISSN 1834-6618 ISBN 978 0 643 09505 2 (Print format) ISBN 978 0 643 09506 9 (Web format)

CSIRO Land and Water Page i

Acknowledgements This report is part of the National Heritage Trust (NHT) funded Australian Water Availability

Project (AWAP) and eWater CRC project D1. AWAP is a collaborative project between the

Bureau of Rural Sciences (BRS), Bureau of Meteorology (BoM) and CSIRO (both Land and

Water (CLW), and Marine and Atmospheric Research (CMAR)). Thanks to Drs John Sims

and David Barratt for their continued support for developing observational outputs from

optical (thermal and reflective) remote sensing suitable for use in ecohydrological

applications. Drs David Jones and William Wang (both from National Climate Centre, BoM),

answered many questions and unwaveringly supported the ideal that gridded daily

meteorological surfaces must be matched with daily error surfaces, thereby allowing error

propagation to be spatio-temporally explicit.

Dr Rob Vertessy (ex-Chief of CLW and now Chief Hydrologist BoM) advocated and strongly

supported the development of the WRON (Water Resources Observation Network)

computing facility in CLW, and without this resource the massive task of processing the

nearly 9,000 afternoon AVHRR overpasses (occupying 750 Gb of disk space) from April

1992 to current would have been insurmountable. The entire information system, including

specific time-of-day meteorological surfaces, requires approximately 2 Tb of disk space. Mr

Garry Swan, Brendan Spetch and Mick Hartcher (CLW IT specialists) provided wonderful

high-end IT support.

CSIRO Land and Water Page ii

Executive Summary The Normalised Difference Temperature Index (NDTI) is a measure of moisture availability

(ma), the ratio of actual evapotranspiration (ETa) to potential evapotranspiration (ETp). NDTI

is derived by linking meteorological point data and optical (both thermal and reflective)

remotely sensed data in a way that optimally utilises process understanding of ma and the

inherent strengths of these different data types. NDTI is generated using a ‘calculate-then-

interpolate’ approach to best use the strengths of the two primary datasets - meteorological

data are temporally dense (and spatially sparse), whereas remotely sensed data are spatially

dense (and temporally sparse). The spatial interpolation is performed using covariates that

are heavily governed by the spatially dense remotely sensed data, which allow for better

modelling of ma in-between meteorological stations (e.g., over water bodies, which would

otherwise not be well represented). Importantly, the NDTI is readily calculated across large regions; it does not require interpolated gridded precipitation or irrigation data as inputs, and therefore partly provides an independent means to monitor water application compliance. Given that ma ≅ NDTI, and that ma = ETa / ETp, then multiplying

the resultant NDTI surfaces by daily ETp surfaces provides a means to measure daily ETa

with units of mm day-1.

Fortnightly NDTI surfaces from April 1992 to 2006 for the Murray-Darling Basin, and adjacent

coastal catchments, with a spatial resolution of 1 km have been generated using almost

9,000 AVHRR (Advanced Very High Resolution Radiometer) images. To rapidly process

such a large number of files (750 Gb disk space), and to routinely process current and future

remotely sensed data from any number of possible sensors, a prototype operational system

has been developed. The system has a modular design, requires little operator input, and

automatic error checking methodologies have been implemented. The core of the NDTI is

the inversion of a 2-source resistance energy balance model (REBM) which is run at the

specific date-time of each remotely sensed overpass.

Ecohydrological assessment of interactions between the NDTI and the commonly used

Normalised Difference Vegetation Index (NDVI) for three hydrologically relevant land-uses (a

reservoir, an irrigation area, and an agriculture-forest interface) show, as expected, that the

NDTI responds more quickly to changes in water availability, than the NDVI does. This has

implications for adequately modelling soil evaporation, and evaporation from water bodies,

that NDVI-based models of ETa can not adequately account for.

CSIRO Land and Water Page iii

The prototype operational system is now ready to be applied to all Australia for the last 21

years, and to routinely process current observations. Following this, further research would

be required to assess how the improved spatial metrics from remote sensing can be used to

assess risks to ecohydrological stability and agricultural production by detailed analysis of

moisture availability (NDTI) and moisture utilization (NDVI), spatio-temporal variability, and

trends for all Australia over the last 15 years. These products, and resulting analysis, would

be suitable for distribution via on-line web-services as they could be routinely produced and

would be widely relevant to a number of water resource and environmental managers and

policy-makers. While the remotely sensed data currently used are AVHRR, any sensor with

both thermal and reflective capacity – such as MODIS (MODerate Resolution Imaging

Spectroradiometer), ETM+ (Enhanced Thematic Mapper Plus), AATSR (Advanced Along-

Track Scanning Radiometer) and ASTER (Advanced Spaceborne Thermal Emission and

Reflection Radiometer) – could be used.

CSIRO Land and Water Page iv

Table of Contents Copyright and Disclaimer ii Acknowledgements i Executive Summary ii 1 Introduction 1 2 Physical Models of Moisture Availability 3 3 Examples of NDTI / NDVI Characterisation of Hydrological Land-Uses 7

3.1 Lake Victoria 8 3.2 Coleambally Irrigation Area 12 3.3 Forest – dryland agriculture interface at Tumut 16

4 Developing a Prototype Operational NDTI Generation System 20 4.1 Data collection 22 4.2 Quality assess SILO Point Patch Database annually 23 4.3 Quality assess BoM wind data and convert from wind run to wind speed 25 4.4 Convert fPAR data to percent vegetation cover 26 4.5 Generate annual .DAT files and .SS files 27 4.6 Run PRERAD to generate .PSF files 28 4.7 Generate and interpolate daily atmospheric transmittance 29 4.8 Calculate Rs_in at the date-time of image acquisition 30 4.9 Use .ALL files in RADWAT to create .COV files 31 4.10 Interpolate Ta and ea at the date-time of image acquisition 33 4.11 Generate Rn and Ts-Ta at the date-time of image acquisition 34 4.12 Run RADWAT to create files containing overpass information 36 4.13 Generate overpass NDTI surfaces 39 4.14 Generate fortnightly and monthly NDTI composites 39

5 Monthly Gap-filled NDTI Composite Images from 1992 to 2006 42 6 Conclusions 59 7 References 61 8 Appendix A: Monthly NDTI / NDVI Characterisation 66

8.1 Lake Victoria 67 8.2 Coleambally Irrigation Area 69 8.3 Forest - dryland agriculture interface at Tumut 71

9 Appendix B: Fortnightly Gap-filled NDTI Composite Images from 1992 to 2006 73 10 Appendix C: IDL code 90

10.1 IDL code to quality assess SILO Point Patch Database annually 90 10.2 IDL code to quality assess BoM wind data and convert from wind run to wind speed 92 10.3 IDL code to convert fPAR data to percent vegetation cover 96 10.4 IDL code to generate annual .DAT files and .SS files 98

CSIRO Land and Water Page v

10.5 IDL code to run PRERAD to generate .PSF files 101 10.6 IDL code to generate and interpolate daily atmospheric transmittance 102 10.7 IDL code to calculate Rs_in at the date-time of image acquisition 106 10.8 IDL code to use .ALL files in RADWAT to create .COV files 106 10.9 IDL code to interpolate Ta and ea at the date-time of image acquisition 113 10.10 IDL code to generate Rn and Ts-Ta at the date-time of image acquisition 116 10.11 IDL code to run RADWAT to create files containing overpass information 119 10.12 IDL code to generate overpass NDTI surfaces 134 10.13 IDL code to generate initial fortnightly and monthly NDTI composites 138 10.14 IDL code to gap-fill fortnightly and monthly NDTI composites 142

CSIRO Land and Water Page vi

CSIRO Land and Water Page 1

1 Introduction In recent years reduced availability of water has resulted in several large cities in Australia

having water restrictions, crop production has been below-average (detectable in the

Nation’s current account deficit), and widespread severe wind erosion has resulted in large

dust storms being experienced. Hence there is heightened demand for accurate, timely and

consistent appraisals of the Nation’s water resources. To meet this demand, a partnership

project, called the Australian Water Availability Project (AWAP) has been established with

funding provided by the National Heritage Trust. AWAP partners are the Bureau of Rural

Sciences (BRS), the Bureau of Meteorology (BoM) and CSIRO (both Land and Water (CLW)

and Marine and Atmospheric Research (CMAR)). The objective of the project is: “To

develop an operational prototype of a new and integrated approach to monitoring and

predicting soil moisture and other components of the water balance, for spatial scales

ranging from 1km to all of Australia, and for time scales ranging from fortnightly to decades.”

Currently there is no nationally consistent source of information on water balance fluxes and

stores that utilises the inherent strengths of meteorological network and remote sensing data.

Meteorological data are temporally dense (and spatially sparse), whereas remotely sensed

data are spatially dense (and temporally sparse). In AWAP, both the direct interpretation of

moisture availability and prediction of select key components of the water balance (e.g., soil

moisture) will utilise both meteorological and regional time series remotely sensed

databases. In this report we document the development of prototype system that directly

interprets a key component of the regional water balance, moisture availability, from optical

(meaning both thermal and reflective) remotely sensed data. This is achieved by generating

a 15 year time-series of Normalised Difference Temperature Index (NDTI) images from the

AVHRR (Advanced Very High Resolution Radiometer) sensor. Any sensor with both thermal

and reflective capacity – such as MODIS (MODerate Resolution Imaging Spectroradiometer),

ETM+ (Enhanced Thematic Mapper Plus), AATSR (Advanced Along-Track Scanning

Radiometer) and ASTER (Advanced Spaceborne Thermal Emission and Reflection

Radiometer) could be used; here AVHRR is used to provide a long-term context of changes

to a key component of Austrlia’s water balance. Importantly, the NDTI can also be readily

calculated across large regions. It does not require interpolated gridded precipitation or

irrigation data as inputs, and therefore provides a means to detect ‘spurious and

unaccounted’ water application thus enabling compliance to be monitored.

This research integrates two data types with very different spatial and temporal scales.

Remotely sensed data are spatially dense, with a specific spatial resolution; they are


recorded over large areas in a matter of seconds at a specific time for specific wavelengths.

This means for the extent of the image, remotely sensed data are a “census” at a particular

spatial scale recorded at a specific time. Depending on the amount of cloud cover and the

satellite repeat characteristics, optical remotely sensed data may only be available for an

area of interest once every week or even just once every month. On the other hand

meteorological data are recorded sparsely with the points often separated by tens to

hundreds of kilometres. The variables measured at these points represent a certain area.

However, the exact area being represented by a given point measurement is unknown as the

spatial autocorrelation is unknown. Thus remotely sensed data is spatially dense but

temporally sparse while meteorological data are spatially sparse but temporally dense.

Within the AWAP framework the idea is to best utilise these characteristics.

This report is organised as follows: Section 2 provides an overview of the methods that are

currently available (and how these methods may be refined) to produce spatial estimates of

moisture availability. Section 3 provides three examples of time series of NDTI and NDVI

(Normalised Difference Vegetation Index) imagery for three hydrologically relevant land-

uses: (a) Lake Victoria; (b) Coleambally Irrigation Area (CIA); and (c) the forest-dryland

agriculture interface in the vicinity of Tumut, New South Wales (NSW). Section 4 provides

an overview of the design of the current information system used to generate the NDTI in

near real time. Current monthly NDTI composites from April 1992 to December 2006 are

shown in Section 5. Conclusions, some specific issues related to making the current

information system operational and a review of future scientific directions to better utilise time

series remotely sensed based ecohydrological indices are presented in Section 6. The

information system developed here is suitable to generate all-Australian 1 km resolution daily

time-step moisture availability products from 1986 onwards.


2 Physical Models of Moisture Availability Regional remotely sensed data can be used in an information system to monitor moisture

availability - ma, which is defined as the ratio of actual evapotranspiration (ETa) to potential

evapotranspiration (ETp). That is, ma = ETa / ETp. The remotely sensed variables required

to calculate ma are surface temperature (Ts), fraction vegetation cover (VegCov), and albedo

(α ). When these variables are combined with meteorological data (specifically air

temperature (Ta), net radiation (Rn), wind speed (u) and some measure of vapor pressure

(ea)) there exists the opportunity to regionally estimate ma.

Much international and Australian research has illustrated the worth of using optical remote

sensing (both thermal and reflective) to derive regional information about the key

components of the water balance. For detailed reviews see (Kustas and Norman 1996;

McVicar and Jupp 1998). In much previous research (Kustas and Humes 1997; Kustas and

Jackson 1999) spatial estimates of key components of the water balance have been

determined in small catchments for short experimental field campaigns. While this has lead

to vast improvements in process understanding, the data requirements to perform this style

of analysis are not routinely available for operational management of large regions for long

periods. To overcome this problem the NDTI was developed (McVicar and Jupp 2002) which

utilises the complimentary inherent data densities of meteorological (temporally dense and

spatially sparse) and remotely sensed data (temporally sparse and spatially dense).

The NDTI is defined as:

NDTI T TT T

s=−−

∞

∞

( )( )0

(1)

The two bounding temperatures, T0 and T∞, are derived by inverting a dual-source

Resistance Energy Balance Model (REBM), this is achieved by setting the vegetation

resistance (rsv) and ground resistance (rsg) to pre-defined values (Jupp et al. 1998). T0 is a

REBM inverted surface temperature when rsv = rsg = 0 s m-1, that is when it is assumed ETa =

ETp. T∞ is a REBM inverted surface temperature when the vegetation and ground (or soil)

resistances are set to infinity, mathematically that is rsv = rsg = ∞ s m-1, that is when it is

assumed ETa = 0 W m-2. Ts is the daytime remotely sensed measurement of surface

temperature; it can be derived from any space, or airborne thermal sensor. If the REBM and

the meteorological data (Ta, Rn, u and ea) estimated at the time of remotely sensed data

acquisition are well defined, a time series of Ts should fall within the envelope defined by the

limits T0 and T∞. Figure 1 shows this relationship for 3 years (July 1999 to June 2002) of


AVHRR Ts data recorded at Cobar. At the time of change over from an ‘older’ NOAA satellite

and its ‘newer’ operational replacement there can be up to 3 hours differences in the local

observation time due to orbital decay in the NOAA series of satellites carrying the AVHRR

sensor (Price 1991). Figure 2 shows that the local time of image acquisition at the end of the

operational life of NOAA14 was approximately 3 hours later than data acquired early in the

operational life of NOAA16, i.e., the local observation time was approximately 1700 hours at

the end of the life of NOAA14, and early in the life of NOAA16 it was about 1400 hours.

Figure 1. Time series of AVHRR observed temperature, Ts, and the REBM-inverted temperatures, T0 and T∞ (denoted Tinf on the figure), at Cobar, central NSW from 1 July 1999 to 30 June 2002. The change over from NOAA14 to NOAA16 is shown by the vertical grey line. The large change in temperatures due to local changes in time of data acquisition for the two satellites is clearly seen when comparing the summer of 2000 with the summer of 2001 (and when comparing the winters of the same two years). Any anomalous data (i.e., Ts < T0 or Ts > T∞,) has been removed, and a box-car average with a width of 15 (i.e., 7 values on either side) was run from 1 Jan 1999 to 31 Dec 2002 to ensure edges of the time series were not adversely impacted by the box-car averaging.

Figure 2. A time series plot showing the local time of image acquisition of afternoon AVHRR images for the Murray-Darling Basin, south-east Australia from January 1992 to December 2006. The vertical grey lines show change over dates between NOAA satellites, which are labelled on the plot. Only AVHRR data with more than 5% valid land pixels in south-east Australia are shown.


To make the most use of the spatial and temporal qualities of remote sensing and

meteorological datasets, the NDTI was calculated using a REBM at the BoM stations that

record maximum and minimum air temperature (Tx and Tn, respectively) and precipitation (P)

every day which are temporally interpolated to the image overpass times (McVicar and Jupp

1999). The NDTI values calculated at all of these BoM points were then spatially

interpolated using ANUSPLIN (Hutchinson 2004) to develop NDTI surfaces. A quint-variate

partial thin plate spline (PTPS), incorporating a bi-variate thin plate spline (TPS) function of

longitude and latitude with constant linear dependencies on Ts-Ta, VegCov and Rn, was used

to generate the NDTI surface (McVicar and Jupp 2002).

Scientifically the six main processes in the current system to generate spatial NDTI are:

1.) the specific time-of-day meteorological data was approximated by temporally interpolating

from daily extreme (maximum and minimum air temperatures) and daily integral

(precipitation) data (McVicar and Jupp 1999);

2.) remote sensing processing algorithms were utilised to convert the raw data into the

variables needed, specifically Ts, VegCov and α (McVicar and Jupp 2002);

3.) estimates of Ta and ea were generated using all meteorological stations, these were used

in the generation of the covariates Ts-Ta, and Rn (with VegCov being directly observed from

the remote sensing – as described in step 2 above);

4.) the NDTI was calculated by inverting the REBM (Jupp et al. 1998) where the

meteorological data and cloud-free AVHRR intersect. That is, the process based REBM was

ONLY run at the meteorological stations for the specific days and time-of-day that cloud free

remote sensing was recorded;

5.) for each afternoon overpass outputs from the inversion of the REBM were then spatially

interpolated between the meteorological stations using a quint-variate partial thin plate spline

(PTPS), incorporating a bi-variate thin plate spline (TPS) function of longitude and latitude

with constant linear dependencies on Ts-Ta, VegCov and Rn, using ANUSPLIN (Hutchinson

2004); and

6.) the resultant single overpass NDTI images were then temporally composited to provide

monthly or fortnightly assessments of ma at 1 km resolution.

McVicar and Jupp (2002) illustrated how the NDTI is generated from the point BoM

observation stations and linked with remotely sensed spatial modelling to provide single

afternoon NDTI images. To illustrate the potential of the NDTI for the derivation of

ecohydrological metrics an initial dataset was developed from 1986 to 1994 (McVicar and

Van Niel 2005). This initial dataset comprised 97 near monthly AVHRR images, the size of


the AVHRR data on disk was 0.75Gb, and with all intermediate files the amount of disk

storage required was 11.5 Gb (i.e., a 15-fold increase of hard disk was required). The

afternoon overpasses in the current database from April 1992 to current included 8,889

AVHRR files (many days have two images that acquire data for part of south-east Australia)

which require approximately 750 Gb to store these data. If there was the same 15-fold

increase in data volume, this would mean that the processing would generate 10,950 Gb (or

10.7 Tb) of data – obviously this was unfeasible and the number of intermediate files had to

be reduced, by, wherever possible, concatenating processing steps. This meant that the

processing chain to produce the initial (or research) NDTI database required a major

overhaul; our primary aim was to develop a prototype operational system by judicious and

effective use of scripting, file naming and sub-folder structures to enable the 8,889 images to

be processed operationally. In the next section, case studies demonstrating the uses of

NDTI time series for different hydrologically-relevant land-uses are discussed.


3 Examples of NDTI / NDVI Characterisation of Hydrological Land-Uses

To illustrate the utility of the NDTI images, and comparing it to the widely used NDVI, we

show these data for three areas with spatial and temporal variability of moisture availability

and vegetation vigour. This analysis is performed for fortnightly composites for 2001 and

highlights typical ecohydrological information present in NDTI, NDVI imagery and their

interactions. The three areas illustrated are: (1) Lake Victoria; (2) Coleambally Irrigation

Area; (3) forest-dryland agriculture interface in the vicinity of Tumut (see Figure 3). We first

show the NDTI and NDVI contexts of the feature of interest, then for 80 km transects located

across each area (Figure 3b-d) )the NDTI and NDVI values are extracted and shown, and

finally for individual pixels along the transects time series plots are shown.

(b)

(c)

(a)

(d) Figure 3. The south-eastern Australia study site and the location of the three transects are shown in (a). The transects for Lake Victoria (b), the Coleambally Irrigation Area (CIA) (c), and the forest-dryland agriculture interface near Tumut, NSW (d) are shown over the NDTI composite for the first fortnight in January 2001.

For the fortnightly images the layout of the 24 fortnights is shown in Figure 4.


Jan_1 Jan_2 Feb_1 Feb_2

Mar_1 Mar_2 Apr_1 Apr_2

May_1 May_2 Jun_1 Jun_2

Jul_1 Jul_2 Aug_1 Aug_2

Sep_1 Sep_2 Oct_1 Oct_2

Nov_1 Nov_2 Dec_1 Dec_2

Figure 4. Schematic showing the layout of the progression of fortnightly composite images, where each month is given by the first 3 letters of that month, and 1 and 2 mean the first fortnight (days 1 through 15) and second fortnight (days 16 through to the end of the month) respectively.

3.1 Lake Victoria Lake Victoria can store 677 Gl and it provides a buffering capacity to ensure that reliable

daily flow of 3 Gl to 7 Gl per day (depending on season) can be provided to South Australia

(Murray-Darling Basin Commission 2006). It is located in the south-west of NSW, its location

being further west than all major tributaries flowing into the Murray River (including the

Darling River), and most of the NSW and Victorian irrigation water off-takes along the Murray

River. By locating this large storage further downstream of connections which can both add

and remove water from the system is the essence of the buffering capacity (Murray-Darling

Basin Commission 2006).

Figure 5a shows the fortnightly NDTI for 2001 in the vicinity of Lake Victoria. As expected

the water body has a persistently high NDTI throughout, and is easily visible in the summer

months where there is high contrast with the surrounding Mallee shrub. Over winter (from


May to August, inclusive) due to widespread precipitation in the area, the actual water body

becomes more difficult to identify. Figure 5b is the NDVI imagery for the equivalent area and

sampling characteristics. As expected the water body has a continued low NDVI. Models

using NDVI-driven estimates of Leaf Area Index (LAI) or fraction of Photosynthecially

Available Radiation (fPAR) to characterise actual evaporation would be severely limited in

this environment, as they would likely estimate that ETa was also low. However, methods

using the differential between the surface and air temperatures (e.g., NDTI) are not adversely

impacted in these situations. Accurate characterisation of ETa from large water bodies such

as Lake Victoria is important for advanced water accounting systems that are currently being

built for Australia.

0 . 0 0 . 5 1 . 0 - 0 .0 5 0 .4 7 1 .0 0

(a) NDTI ≅ moisture availability (b) NDVI ≅ vegetation vigour Figure 5. Fortnightly NDTI images (a) and fortnightly NDVI images (b) around the Lake Victoria transect for 2001. NDTI measures moisture availability (high = wet) and NDVI reflects vegetation vigour (high = lush).


Both NDTI and NDVI data were extracted for a 80 km long transect running through the

centre of the context images (and the Lake Victoria waterbody provided in Figure 3). Figure

6 represents the time series for 2001; the x-axis represents the distance along this transect

and the y-axis is time at a fortnightly time step. Figure 6a shows that the NDTI is constantly

high in the Lake, and the increase in NDTI during the winter months (starting in the first

fortnight of May) of the area surrounding the lake is clearly seen. The 4 to 6 week lagged

response of the NDVI to greater moisture availability is seen in Figure 6b. In Figure 6c to e

standard time series plots are shown for individual pixels. The lag between the relative

increase of NDTI in winter and the subsequent increase in NDVI is clearly illustrated in

Figure 6e, where the NDTI is decreasing in September when the maximum NDVI is

observed. The monthly NDTI and NDVI composites are shown in Appendix A.


(a) NDTI ≅ moisture availability

(b) NDVI ≅ vegetation vigour

(c) (d) (e)

Figure 6. Fortnightly time series plot from 2001 for an 80 km transect located across Lake Victoria of (a) NDTI and (b) NDVI. Note these do not have the same colour stretch. The black line in (a) and (b) represent the profile of the first fortnightly compositing period in each case. Per-pixel time series are shown for pixels located at (c) 15 km, (d) 40 km, and (e) 65 km, respectively, with the black line being the NDTI and the grey line the NDVI.


3.2 Coleambally Irrigation Area

The Coleambally Irrigation Area (CIA) is a rice-based irrigation area, with other crops also

grown in summer including corn, soybeans and sorghum (Van Niel and McVicar 2004b; Van

Niel et al. 2003). There are a number of drains west of the irrigation area used to remove

excess water applied to the region. Rice is flood irrigated, with the other row crops having

irrigated water applied to small depressions (called furrows) separating the soil that the rows

of crops are planted along. In winter, cereals (wheat, oats, barely) and grazing pastures are

the primary agricultural activities (Van Niel and McVicar 2003) – the irrigation demand in

winter is lower as landholders attempt to utilise residual soil moisture after successfully

growing summer crops. A review of the application of remote sensing technology to assist

management within an irrigation system is provided by Van Niel and McVicar (2004a); the

following example compliments that work, by highlighting how large area remotely sensed

data (i.e., AVHRR or MODIS) can be used to monitor irrigation system performance noting

that this can be performed for all irrigation systems located in the Murray-Darling Basin, and

here the CIA is used an example.

Figure 7a shows the fortnightly NDTI images from 2001 for the vicinity of the CIA. The extent

of the irrigation area is clearly seen in the first fortnight of January. The increase of moisture

availability associated with the outfall drain to the west of the CIA is seen in the second

fortnight of January and the first fortnight in February, while the dryland agriculture east of

CIA provides strong constant low moisture availability right up to the border of the CIA.

Patchy precipitation is received in the first fortnight of February, and in the last fortnights of

April and November. Widespread heavy precipitation is received in June and July, resulting

in these months having high NDTI values for all parts of the image. The dryland agricultural

land surrounding the CIA dries during spring (September and October), with the winter crops

actively growing in this period. Flood irrigation of rice paddocks commences in mid October

with most paddocks being filled by the end of November. The vegetation vigour, shown by

the NDVI (Figure 7b), lags the application of irrigation water within the CIA, and the

senescence and harvesting of the summer crops in March to June is seen (Figure 7b).

Following this, the gradual greening due to the winter crops is seen, with these being

harvested in late November (Figure 7b). West of the irrigation area the green-up occurring in

February and March (Figure 7b) clearly follows the increase in moisture availability that starts

in the last fortnight of January (Figure 7a). To the north and east of the irrigation area the

increase in vegetation vigour of the dryland cropping and grazing during September and

October (Figure 7b), following the wide spread winter rains is also clearly seen.


0 . 0 0 . 5 1 . 0 0 . 0 0 . 5 1 . 0 (a) NDTI ≅ moisture availability (b) NDVI ≅ vegetation vigour

Figure 7. Fortnightly NDTI images (a) and fortnightly NDVI images (b) around the CIA transect for 2001. NDTI measures moisture availability (high = wet) and NDVI reflects vegetation vigour (high = lush).

The 80 km long transect (see Figure 3) extracted through the centre of the context images

(also running through the centre of the irrigation area) for NDTI and NDVI, see Figure 7a and

b, respectively further highlights the patterns of the three distinctive regions in both the NDTI

(Figure 8a) and NDVI (Figure 8b) images. The three regions are: (1) west of CIA where the

impact of the CIA drain is seen (0 km to 30 km along the transect); (2) the CIA proper (30 km

to 58 km); and (3) dryland agriculture east of the CIA (58 km to 80 km). Figure 8a and b

represents the time series for 2001; the x-axis represents the distance along this transect

and the y-axis is time at a fortnightly time step. While the same gross patterns are revealed

in the context images (which will not be discussed to avoid repetition) some finer scale


features are worthy of discussion. For example, west of the CIA the NDTI increases in the

second fortnight of January, yet it takes another fortnight for the NDVI to respond. During

this period direct evaporation from the soil surface would have been high before the

vegetation responded. Additionally, in the last fortnight of November there was a large

supply of water due to landholder management of flood irrigation of the rice paddocks, while

at the same time the NDVI (as expected from a water surface) was low. In this early part of

the growing season there would have been considerable evaporation from the water surface

of the rice paddocks, which would not be detected in reflective based estimates of ETa.

Figure 8c to e are standard time series plots for individual pixels. The accurate estimation of

ETa from intensive irrigation areas (such as CIA) is an important feature of accurate water

accounting systems that are currently being developed for Australia. The lag between the

relative increase of NDTI and subsequent increases of NDVI are clearly seen. Monthly NDTI

and NDVI composites are shown in Appendix A.




(c) (d) (e)

Figure 8. Fortnightly time series plot from 2001 for an 80 km transect located across Coleambally Irrigation Area of (a) NDTI and (b) NDVI. The black line in (a) and (b) represent the profile of the first fortnightly compositing period in each case. Per-pixel time series are shown for pixels located at (c) 15 km, (d) 40 km, and (e) 65 km, respectively, with the black line being the NDTI and the grey line the NDVI.


3.3 Forest – dryland agriculture interface at Tumut To illustrate the utility of the NDTI/NDVI time series imagery for monitoring changes due to

land-use, not directly related to a hydrological land-use (as the two previous examples of a

lake and an irrigation area show), a region with a forest - dryland agriculture interface was

selected in the vicinity of Tumut, south-eastern NSW (Figure 3). Both the time series of

context images and transect will simply be named ‘Tumut’ to represent this forest - dryland

agriculture land-use interface in the following.

The Tumut NDTI fortnightly composite context images (Figure 9a) illustrate that the moisture

availability is relative lower in the agricultural area in the north-west of the image in summer

(January and December) than the adjacent forested areas located in the south-east of the

context image. This spatial pattern is confirmed in the NDVI Tumut context images (Figure

9b). Throughout winter there was widespread precipitation resulting in subsequent increases

of both the NDTI and NDVI (Figure 9a and b, respectively). As expected, the NDTI of the

agricultural lands decreases more rapidly than that of the adjacent forests in spring

(September and October) and this decrease in moisture availability is associated with

increasing moisture utilisation illustrated by the NDVI of the dryland agriculture being slightly

higher than the NDVI of the adjacent forests in late September through to the end of October,

see Figure 9a and b, respectively.


0 . 0 0 . 5 1 . 0 0 .0 0 0 .4 3 0 .8 5

(a) NDTI ≅ moisture availability (b) NDVI ≅ vegetation vigour Figure 9. Fortnightly NDTI images (a) and fortnightly NDVI images (b) around the forest - dryland agriculture land-uses transect near Tumut for 2001. NDTI measures moisture availability (high = wet) and NDVI reflects vegetation vigour (high = lush).

The 80 km long transect (see Figure 3) running through the centre of the context images

(see Figure 9a and b for NDTI and NDVI, respectively) further highlights the patterns of the

two distinctive land-uses and is illustrated in both the NDTI (Figure 10a) and NDVI (Figure

10b). The two land-uses are: (1) dryland agriculture in the western portion of the image (0

km to 38 km along the transect); and (2) forests (from 38 km to 80 km). The two land-uses

are clearly distinguishable. In the west the dryland agriculture shows a relative lower NDTI in

summer, with moderate values recorded during autumn, with the relative maxima occurring

in spring. The NDVI follows a similar cycle, with about a 6 week to 8 week lag. Figure 10c


shows this lag, with the local maxima of the NDTI occurring in July followed by the maxima of

the NDVI occurring in September. The time series plots of both forested pixels (Figure 10d

and e) show that the NDTI is constantly above 0.6, with more variability and slightly lower

values being observed close to the forest-agriculture interface (Figure 10d) compared to the

forested pixel that is surrounded by forest on all sides (Figure 10e). This finding is in

agreement with the well known feature of enhanced ETa (and hence lower moisture

availability) at the edges of large stands of trees – usually associated with plantations or

fragments of forests following clearing – the so-called ‘edge effect’ (Camargo and Kapos

1995; Giambelluca et al. 2003; Kapos 1989). In both plots the NDVI values were relatively

constant. Monthly NDTI and NDVI composites for this transect are shown in Appendix A.




(c) (d) (e)

Figure 10. Fortnightly time series plot from 2001 for an 80 km transect located across a forest – dryland agriculture land-uses in the vicinity of Tumut (a) NDTI and (b) NDVI. The black line in (a) and (b) represent the profile of the first fortnightly compositing period in each case. Per-pixel time series are shown for pixels located at (c) 15 km, (d) 40 km, and (e) 65 km, respectively, with the black line being the NDTI and the grey line the NDVI.


4 Developing a Prototype Operational NDTI Generation System

Operationally, there are 14 main steps involved in generating spatial NDTI surfaces, they

are:

1. Data Collection 2. Quality assess SILO Point Patch Database Annually 3. Quality assess BoM Wind Data and convert from wind run to wind speed 4. Convert fPAR data to Percent Vegetation Cover 5. Generate annual .DAT files and .SS files 6. Run PRERAD to Generate .PSF files 7. Generate and Interpolate Daily Atmospheric Transmittance 8. Calculate Rs_in at the date-time of image acquisition 9. Use .ALL files in RADWAT to create .COV files 10. Interpolate Ta and ea at the date-time of image acquisition 11. Generate Rn and Ts-Ta at the date-time of image acquisition 12. Run RADWAT to create files containing overpass information 13. Generate Overpass NDTI Surfaces 14. Generate Fortnightly and Monthly NDTI Composites

Each of these processing steps is described in detail below, and a flow chart is included

wherever applicable for easy and quick assimilation of the processing steps. Definition of

terms and acronyms for the terminology used primarily in this section, and in other parts of

the text, is provided in Table 1.

Table 1. Definition of terms used in this and other sections of the report.

Term Definition

SILO PPD SILO Patched Point Dataset (where missing gaps in meteorological

stations have been in-filled by using output from spatial interpolation at

the date-geographic intersection).

fPAR fraction of Photosynthetically Active Radiation absorbed by vegetation

.DAT This type of file contains the quality assessed SILO PPD daily data and

Bureau of Meteorology average daily wind-speed data (if recorded) for

each isolated meteorological station. It is used as input to PRERAD.

.SS This type of file contains information about the study site, here the

locations are never different from the .DAT file. It is used as input to

PRERAD.


.PSF This type of file contains daily extreme and integral meteorological data.

It primarily differs from the .DAT file by having incoming solar radiation

(Rs_in) calculated as a total and fraction diffuse component using methods

outlined in McVicar and Jupp (1999). It is output from PREAD and input

to RADWAT.

.OVP This type of file contains data specific to one geographic location (BoM

station or flux tower) for the time series of overpasses at that location. It

includes date(s) and time(s), and in this study also includes Ts and %

Veg Cover obtained from AVHRR remotely sensed images. If data from

a flux tower were being used in the REBM then Ta, relative humidity,

Rs_in, wind speed, and vegetation height could also be provided. Multiple

times can be provided each day if such data exist (as would be the case

for a flux tower) with a separate .OVP file needed for each geographic

location.

.ALL This type of file only contains all the date-time data specific to each

overpass regardless of cloud cover or satellite coverage; this means that

the temporal interpolation of Ta and ea is conducted for all meteorological

stations for all dates and is independent of remotely sensed data. It is

output from RADWAT.

.COV This type of file only contains geographic information (longitude, latitude

and elevation) and specific time-of-day meteorological data (temporally

interpolated in RADWAT) suitable for spatial interpolation by ANUSPLIN.

It is the output of reformatting the .ALL file (one station, many times) to

produce the .COV file (all stations, one time).

PRERAD This is a FORTRAN executable that is run for each meteorological

station. It performs the required daily time-step calculations (primarily

modelling the radiation environment) before running the REBM.

RADWAT This is a FORTRAN executable that is run for each meteorological

station. It performs the required REBM calculations at the specific dates-

times that optical (thermal and reflective) remotely sensed imagery are

available to force the model.

.NIN This is the Ndti INput file. It contains all the data to drive ANUSPLIN

(specifically SPLINA) to fit a quint-variate partial thin plate spline model,

incorporating a bi-variate thin plate spline function of longitude and

latitude with constant linear dependencies on Ts - Ta, VegCov and Rn.

.NHW This is the Ndti Half Way (or intermediate) file, which contains overpass


output from RADWAT. It is combined with data from overpass surfaces

to create .NIN files.

4.1 Data collection NDTI can be generated from any remotely sensed data where: (1) both thermal and

reflective channels are acquired; and (2) the date and specific time-of-day of data acquisition

are known. The first point means that sensors such as AVHRR, MODIS, TM, ETM+, AASTR

and ASTER are suitable remotely sensed data sources. The second point is valid for single-

overpass imagery. Some compositing algorithms, such as the MODIS land surface

temperature product, is the average surface temperature of all cloud-free observations in a

given compositing period. The averaging of several days means it is unwise to utilise the

resultant MODIS product as input to a process-based REBM. Of course, single afternoon

MODIS images, where the date and specific time-of-day of data acquisition are known are

eminently suitable as input to a process-based REBM.

The current prototype operational system uses AVHRR data managed by CSIRO Marine and

Atmospheric Research (King 1998; King 2000; King 2003). Single afternoon overpasses are

used in the NDTI processing, where the date and time-of-day that the image are acquired are

known. Table 2 shows the number of AVHRR images available for use between 1992 and

2006. We only process AVHRR overpasses if valid (i.e., not cloud affected or outside the

swath) AVHRR data intersects with at least 25 valid BoM stations.

Table 2. Number of AVHRR images from 1992 through 2006 and number of SILO PPD meteorological stations that pass our assessment of > 90% recorded data.

Year Number of images Number of valid meteorological stations

1992 492 234 1993 653 237 1994 472 228 1995 581 234 1996 630 233 1997 645 223 1998 647 226 1999 654 213 2000 646 204 2001 650 215 2002 654 215 2003 653 203 2004 359 200 2005 586 193 2006 567 193 Total 8889


The files from CMAR have the specific naming convention which underpins the scripting of

the entire system. An example is, ah16_20010101_035315_23 – where the field ah

represents AVHRR, 16 is the number of the NOAA series of satellites, next the GMT date is

provided as YYYYMMDD. In our example this is the 1st Jan 2001, then the average GMT

time of non-water and non-cloud affected pixels (i.e., the valid land pixels) is calculated and

provided as HHmmSS, and finally the percentage of the land surface with valid pixels (i.e.,

non-cloud affected and not past the edge of the acquisition swath) are calculated. In the text

above YYYY means year, MM month, DD day, HH hour, mm minute and SS second. The

files are stored in sub-folders arranged by year and month of image acquisition.

Other datasets used in this processing include:

• SILO Point Patch Database (PPD) data: daily, from 1980 through 2006; • Bureau of Meteorology (BoM) windrun data: daily, from 1980 through 2006; • Vegetation coverage percentage images: monthly, from 1992 through 2006

generated from AVHRR data; and • Australian 9 arc-second DEM.

4.2 Quality assess SILO Point Patch Database annually SILO PPD data are infilled from interpolated data wherever actual BoM meteorological

measurements are missing. As we put a higher certainty on measured data than

interpolated data, we performed an analysis to determine how many meteorological stations

had substantially measured data as opposed to substantially interpolated data. This quality

assessment of the SILO PPD data was performed on an annual basis, which allowed for a

user defined threshold for choosing which stations were acceptable for each year. For

example, with a threshold of at least 90% measured data (i.e., < 10% interpolated) for the

first three variables (maximum temperature (Tx), minimum temperature (Tn), and precipitation

(P)), the overall number of stations that would be used if we consider that the station has to

pass the threshold over the entire time between 1992 and 2006 is 144 (magenta line, Figure

11). However, as many more stations pass the threshold for specific years, but not for the

entire time extent, when considered on a yearly basis, many more stations can be used (blue

line, Figure 11).


Figure 11. Number of stations that pass the 90% rule applied to each year in turn (black line) and if 90% rule was applied to all years would result in only 144 stations being used (grey line).

The procedure for SILO PPD quality control is depicted in

Figure 12 below. The SILO PPD text files for each station are input into the IDL program

called checkMetYearly.pro (see Appendix 10.1), which runs the IDL program

stationQualityYearly.pro (see Appendix 10.1 for details). The quality control text report is

output that allows the user to test various threshold values. For our research we used a 90%

threshold on all of Tx, Tn, and P.

SILO PPD data(*.TXT)

QC Report(*.TXT)

checkMetYearly.pro

stationQualityYearly.pro

Figure 12. Flowchart showing the major steps involved in assessing the quality of the SILO PPD annually.


4.3 Quality assess BoM wind data and convert from wind run to wind speed

SILO PPD does not come with any wind speed data. Instead of simply assuming a wind

speed of 2 ms-1, another option is to obtain the daily wind run data (with units of km per day)

measured below 10 feet (or 3 m) above the land-surface from the BoM and convert it to wind

speed using the general rule that 1/3 of the wind occurs at night and 2/3 occurs during the

day (Smith et al. 1991). Therefore, we need to multiply 2/3 by wind run and multiply by 1000

to place the units in m, and divide by daylength (N) in seconds (i.e., multiply N*60*60). The

BoM data were shifted by a day forward to match the SILO PPD data (who already do this

shift for users). That is, BoM wind run data are recorded at 0900 local time. We assumed

that most of this wind run would be attributed to the first 15 hours associated with the

afternoon of the day prior to the recording.

The key IDL function called in the process is calcDayLength, which calculated the daylength

based on the FAO-56 (Allen et al. 1998) calculation from the latitude and day of year (see

Figure 13 and Appendix 10.2). The wind run data were nulled (-1) if the BoM wind run

equalled zero because this only happens when the original wind run data were blank and

were made to be zero because it was read into a floating point variable. Wind run was also

nulled (-1) if ((qualityFlag ne 'W') AND (qualityFlag ne 'S') AND (qualityFlag ne 'I')) AND

((accDays eq 1) OR (accDays eq '')).

WindRun_to_WindSpeed.pro

readBoMWindAscii.pro

calcdaylength.pro

BoM Wind Speed(*.TXT)

BoM Wind Run(*.TXT)

Figure 13. Flowchart showing conversion of BoM daily wind run data (units km per day) into average daily day-time wind speed (units m s-1).


4.4 Convert fPAR data to percent vegetation cover Fractional cover (FC) describes the proportion of ground area covered by foliage and is

generally equivalent to Percentage Foliage Cover. We used measures of fPAR (the fraction

of Photosynthetically Active Radiation absorbed by vegetation) to make estimates of cover.

The fPAR data were derived from 0.01° resolution monthly AVHRR red and near-infrared

reflectance data corrected for atmospheric and sensor calibration effects, following the

method of Donohue et al. (2007). This dataset covers the entire Australian land surface.

fPAR is linearly related to FC (Asrar et al. 1984; Carlson and Ripley 1997; Lu et al. 2003)

and, by making the assumption that when fPAR is 0, FC = 0.0 and that when fPAR is 0.95

(the maximum possible value), FC = 1.00, FC can be estimated as FC = fPAR / 0.95. This

was performed using the using IDL code fapr2Cover.pro (see Figure 14 and Appendix 10.3).

We then subset each monthly vegetation coverage image between 1992 through 2006 to our

south eastern Australia study site. This was done using IDL code subset_f_Coverage.pro

(see Figure 14 and Appendix 10.3). Null data were also changed from -999 to -9999 in the

resultant subset images to comply with our convention. When FC images were produced,

the land mask also excluded lakes from land, we filled these nulls with the mean monthly

data according to a new land mask which masked out ocean only.


SE AustraliaVegetation

Coverage Images

Vegetation Coverage Images

fPAR2cover.pro

fPAR Images

subset_f_coverage.pro

Null fillled SE Australia

Vegetation Coverage Images

fill_nulls_veg_cov_DOIT.pro fill_nulls.pro

Figure 14. Flowchart showing general steps to produce monthly FC images for south east Australia.

4.5 Generate annual .DAT files and .SS files Files with the suffix .DAT and .SS are primary inputs into PRERAD, the .DAT file contains

basic site header information and daily meteorological data that is extreme air temperatures

(Tx, Tn), integral precipitation (P) and average wind speed. The daily meteorological data

must be consecutive with each and every day being present and both the date YYYYMMDD

and day of year fields are required, as these are used in internal checking by PRERAD to

ensure that the user has not inadvertently made an error with the date / day of year

calculations. The .SS file contains information that may be different from the location of the


.DAT file – this logic was to allow for spatial interpolation internal to the MTCLIM package

(Hungerford et al. 1989) – however in our processing the location of the .DAT file and .SS file

were always exactly the same as all spatial interpolation conducted here was performed

using ANUSPLIN (Hutchinson 2004). We generated DAT and SS files using the procedure

described below in Figure 15 (also see Appendix 10.4 for the relevant IDL code). The main

program called make_dat_ss.pro takes in a dictionary file (station file), silo PPD data, and

BoM wind speed data (if it exists) and gleans relevant information from the station file to

generate the appropriate *.DAT files. In this step the *.SS files for each year are also

generated. It puts the data in separate directories by year and also checks for BoM wind

speed data for each station and merges the wind speed with the SILO PPD data.

SILO PPD data(*.TXT)

*.DAT(Base Met)

make_dat_ss.pro

readBoMWindAscii.pro

readSiloMet.pro

Met Station Dictionary

(.TXT)

*.SS(Study Site)

BoM Wind Speed(*.TXT)

Figure 15. Flowchart capturing primary steps to generate the *.DAT and *.SS files each year from the SILO PPD, dictionary of meteorological stations and BoM wind speed database.

4.6 Run PRERAD to generate .PSF files PRERAD is a FORTRAN executable program that performs daily time-step calculations,

primarily of the radiation environment, as documented in McVicar and Jupp (1999). These

calculations produce total daily solar radiation, fraction of diffuse radiation, potential

evapotranspiration and daily atmospheric transmittance (τ ). For each run, PRERAD takes

one *.DAT file and one relevant *.SS file as input data, and then outputs one *.PSF file,

Figure 16. The IDL code Parameter4PRERAD.pro was used to produce yearly parameter

files for PRERAD (Appendix 10.5). We then ran PRERAD for each year at the DOS

command line using each PRERAD parameter file. For example, for year 2001, we would

type “PRERAD_b < 2001_prerad.par” at the DOS prompt. All the years .PSF files were

generated like this via DOS batch files.


Parameter4Prerad.pro

.PSF (Daily

Summary File)

PRERAD.EXE

.DAT (Station

Summary File)

.SS (Station

Summary File)

Figure 16. Flowchart PRERAD processing to produce the .PSF daily summary files.

4.7 Generate and interpolate daily atmospheric transmittance The next step involved bi-variate thin plate spline interpolation of daily atmospheric

transmittance (τ ) using ANUSPLIN Version 4.3 (Hutchinson 2004); an example of this

output is provided in McVicar and Jupp (2002). The daily atmospheric transmittance values

from the .PSF files, generated from PRERAD, are saved as one file per station (i.e., many

days per file (for each geographic location where there is a BoM station)). For input to

ANUSPLIN, we needed to convert these to one file per day (i.e., many stations per file (for

each day in the time series)). This was performed using the IDL code SortPSFByDay.pro

(see Figure 17 and Appendix 10.6). We named the sorted daily atmospheric transmittance

files as “YYYYMMDD.TRN” to help specifically identify them from the .PSF files – these

where YYYY, MM and DD, which are year, month and day, respectively. The .TRN files

were then interpolated using ANUSPLIN (meaning both SPLINA and LAPGRD) to produce

daily atmospheric transmittance ASCII surfaces for future use in the incoming shortwave

radiation (Rs_in) calculation performed with SOL_RAD.


.TRN (Daily

Transmittance File)

SortPsfByDay.PRO

.PSF (Saily

Summary File)

SPLINA

SPLINA Daily .CMD file

.SUR file(+ other

SPLINA files)

LAPGRD

LAPGRD Daily .CMD

file

Daily At_Trans ASCII surface

Create_SPLINA_cmd_Files.pro

Create_SPLINA_Batch_Runs.pro

Create_LAPGRD_cmd_Files.pro

Create_LAPGRD_Batch_Runs.pro

=

=

Figure 17. Flowchart showing the generation of the daily atmospheric transmittance.

4.8 Calculate Rs_in at the date-time of image acquisition Modelling of the incoming shortwave radiation environment was performed in an image

processing package using a program called SOL_RAD; it can be thought of as a specific

time-of-day version of SRAD (Moore et al. 1993; Wilson and Gallant 2000). The main

difference is that SRAD uses the Ångström–Prescott equation (Prescott 1940), which is

based on the ratio of bright sunshine hours (n) to total sunshine hours (N) as its measure of

atmospheric transmittance, whereas in SRAD the daily atmospheric transmittance (τ ) is

derived from south-east Australia optimized (McVicar and Jupp 1999) versions of the Bristow

and Campbell (1984) coefficients. The required topographic information (including grids of

elevation above sea level, slope and aspect generated from an ANUDEM generated 9 arc

second DEM) were appended to the daily atmospheric transmittance ASCII surfaces created

in the previous step (Section 4.7) to successfully model incoming shortwave radiation. This

was performed by executing a series of DOS batch files. RS_IN.BAT was used as the slave


or engine file and CALL_RS_IN_FOR_COV.BAT was the master or driver file that was run

for every day of each year and called RS_IN.BAT many times passing the few parameters

needed. The date and time of the specific overpass are some of the few parameters needed

and these were obtained from the AVHRR file names. Figure 18 shows the flowchart of Rs_in

modelling and Appendix 10.7 shows the IDL code used for this processing step.

Rs_in.bat

BRINGIN

.IMG

IMAGEMAN

.IMG

Bringin .PAR

Imageman.PAR

SOL_RAD SOL_RAD.PAR

(AVHRR image names are to specify the time, dd, mm and yy for Sol_Rad)

Rs_in_Batchrun_WRON.pro

= AVHRR Names

3 Channel image(elevation, slope and aspect)

Rs_in.IMG

Figure 18. Flowchart of processing to modelling incoming shortwave radiation.

4.9 Use .ALL files in RADWAT to create .COV files To assist with modelling other radiation components, specifically longwave incoming, we

used RADWAT to produce specific time-of-day meteorological interpolation input for use in

SPLINA. In this way we used all valid met stations that passed the 90% quality threshold

discussed in section 4.2 to ultimately interpolate vapor pressure (ea) and air temperature (Ta)

at the time of the image acquisition. The input files for use in this current processing step are

referred to as .ALL files as they are used for processing all stations for all times. In section

4.10 we describe a very similar process using remote sensing-derived variables (Rn and Ts -

Ta) in which interpolation cannot be run over all stations for all times because some areas are

either cloud affected or outside the AVHRR swath. By separating these two steps, we

ensure that ea and Ta are interpolated using as many input BoM stations as possible.

Otherwise, if the steps were combined into one, then these variables would be interpolated

with fewer input points than were truly available.


The IDL program called Int_Creation.pro (see Appendix 10.8) reads in the AVHRR image

names and creates input files (what we call .INT files) for RADWAT. The .INT files are in the

format required by RADWAT, which as stated above, outputs an .All file for each station (see

Figure 19 below). The program SortAllByImageAcq.pro (see Appendix 10.8) does the

conversion from Relative Humidity (%) to vapor pressure (kPa) and vapor pressure deficit

(kPa) and also converts the .ALL files from having many image acquisition records for each

station in a separate text file to having many station records for each image acquisition in

separate text files. That is, in stead of having individual .ALL files sorted by station, we

converted them to being sorted by image acquisition, and thus they were directly useable for

spatial interpolation of each separate image acquisition.

We used the Tetens (1930) approximation of saturated vapor pressure outlined in Allen et al.

(1998) to convert from specific time-of-day relative humidity to specific time-of-day vapor

pressure (ea). This was implemented as:

Rel. humidity . ( )100a s ae e T=

(2)

where ea is the atmospheric water vapor pressure (kPa); Rel. humidity is the specific time-of-day relative humidity (%); and es(Ta) is the saturation vapor pressure (kPa) at the specific time-of-day air temperature (°C).

The saturation vapor pressure (es) at air temperature (Ta) is calculated as:

( ) 17.27 TT 0.6108 expT 237.3se ×⎡ ⎤= ⎢ ⎥+⎣ ⎦

(3)

where es(T) is the saturation vapor pressure (kPa) at air temperature T (°C); and T is the air temperature (°C).

As we are not working with daily or monthly extremes there was no need to calculate the

saturation vapor pressure at each of these extremes and average the result to account for

non-linear underlying air temperature (°C) / saturation vapor pressure (kPa) relationship

(Murray 1967). Vapor pressure deficient (VPD) is simply calculated as es – ea.

To maximize subsequent processing flexibility the 8 variables recorded in the resulting .COV

files are:

1. BoM Station number 2. Longitude (decimal degrees) 3. Latitude (decimal degrees) 4. DEM elevation (m) 5. Ta (°C) 6. Relative Himidity (%) 7. ea (kPa)


8. es - ea (or VPD with units kPa)

.INT (All Stations)

RADWAT.PSF (Daily

Summary)

.ALL (All Stations,

All Times)

SortAllByImageAcq.PRO

.COV (Tair, ea, VPD)

=

AVHRR Images

Create_Parameter_For_Radwat_All.Pro

Int_Creation_WRON.pro

U:\AWAP\Data\RADCALC\ALL\run_radwat.bat

Figure 19. Flowchart of processing to create .COV files for all time at all stations (as input to spatially interpolate the meteorological covariates required to generate the NDTI surfaces).

4.10 Interpolate Ta and ea at the date-time of image acquisition Once the .COV files were generated, the ea, Ta and VPD data contained in each .COV file

were then interpolated for all of south eastern Australia using ANUSPLIN. This is depicted

in Figure 20, below, where the output surfaces are labeled with a .GRD extension (see also

Appendix 10.9 for the relevant IDL code). Following McVicar et al. (2007) a tri-variate partial

thin plate spline (TPS) incorporating a bi-variate TPS function of longitude and latitude and a

constant linear dependence on elevation was used to interpolate these meteorological

surfaces. In all cases the order of the spline was 2 (i.e., m = 2); for more details see McVicar

et al. (2005).


SPLINA

Tair.GRD ea.GRD

.SUR,.LOG

& others

LAPGRD

VPD.GRD

.COV (Tair, ea, VPD)


Create_SPLINA_cmd_Files_Ta_Ea_VPD.pro

Create_SPLINA_Batch_Runs_Ta_Ea_VPD.pro

LAPGRD CMD file

Create_LAPGRD_cmd_Files_Ta_Ea_VPD.pro

Create_LAPGRD_Batch_Runs_Ta_Ea_VPD.pro

=

=

Figure 20. Flowchart showing the processing to develop specific time-of-day meteorological surfaces that are used as to develop covariates for NDTI spatial interpolation.

4.11 Generate Rn and Ts-Ta at the date-time of image acquisition Net radiation (Rn) is the balance of the incoming and outgoing shortwave radiation

components defined by:

= − + −n s_in s_out l_in l_outR R R R R (4)

where: Rn (W m–2) is allwave net radiation at the surface; Rs_in (W m–2) is the downward shortwave radiation (0.15–4 μm) from the sun and atmosphere;

Rs_out (W m–2) is the upward shortwave radiation reflected by the surface; Rl_in (W m–2) is the longwave radiation (>4 μm) emitted from the atmosphere toward the

surface; and Rl_out (W m–2) is the longwave radiation emitted from the surface into the atmosphere.

Rs_in (W m–2) is derived from SOL_RAD (see Section 4.8); with Rs_out (W m–2) being derived

from Rs_in times the surface albedo (α ). Saunders (1990) estimated α as 0.5 R1 + 0.5 R2

where R1 and R2 are the per-pixel fractional reflectance of AVHRR channel 1 and channel 2,

respectively. As AVHRR reflective channels are used to determine α , it is important that

calibrations are applied to the AVHRR reflective channels. For the AVHRR data we used,

sensor degradation (i.e., responsivity and space count corrections) were applied to NOAA11

and NOAA14 as specified in CALWATCH (Mitchell 1999). To cast this in CALWATCH terms

this means that responsivity (S1 and S2) and space count (C01 and C02) corrections have

been applied to both channels 1 and 2 for NOAA11 and NOAA14 as per CALWATCH


(Mitchell 1999). For NOAA16 and NOAA18 NESDIS pre-launch calibrations were used. To

cast this in CALWATCH terms this means that responsivity (S1 and S2) and space count

(C01 and C02) corrections were applied to both channels 1 and 2 for NOAA16 and NOAA18

as per NESDIS pre-launch calibrations. Since this processing has been performed it has

been shown that the time dependent changes in NOAA16 responsivity are stable with time

(Vermote and Saleous 2006), and NOAA18 has not been acquired for a long enough period

for its degradation to be characterised. Rl_in (W m–2) is derived from:

4_l in a aR Tε σ= (5)

where: εa is the effective atmospheric emissivity; σ is the Stefan-Boltzmann constant (5.67 x 10-8 W m-2 K-4); and Ta is air temperature at reference height (K). Prata’s (1996) model was used to determine εa as :

1/ 21 (1 )exp{ (1.2 3.0 ) }aε ξ ξ⎡ ⎤= − + − +⎣ ⎦ (6)

Where a

46.5T

aeξ⎛ ⎞

= ⎜ ⎟⎝ ⎠

.

where ea is the vapour pressure (hPa or, equivalently, millibars; note that kPa must be multiplied by 10 to convert them to hPa); and Ta is air temperature (K) at reference height.

Rl_out (W m–2) was derived from:

4_l out s sR Tε σ= (7)

where εs is the surface emissivity; and Ts is the surface temperature (K) derived from the radiance measured by thermal AVHRR channels 4

and 5 that are converted to brightness temperatures, denoted T4 and T5 respectively, using Plank’s law. These are then used in a split window algorithm Ts = T5 + 3 (T4 - T5) + 0.5. The split window coefficients were determined by simulating absorption for AVHRR channels 4 and 5 bandwidths using Australian mid-latitude atmospheric data (Maher and Lee 1977) and the LOWTRAN6 package (Kneizys et al. 1983). The same split window algorithm is used to generate the .OVP file.

With εs calculated as

ε ε εs v v v gf f= + −( )1 (8)

where: εv is the vegetation emissivity (0.98); εg is the ground (or soil) emissivity (0.96); and fv is the fraction of vegetation. The constant values for εv and εg were derived from (Wan and Dozier 1996), with fv being

derived from reflective remotely sensed data, using the dataset produced by Donohue et al.

(2007). The relatively small component of outgoing longwave radiation by reflected sky

radiation was ignored (Jackson et al. 1985, p. 156).


IDL code Calc_Rn_Surface_WRON.pro (see Figure 21 and Appendix 10.10) outputs AVHRR

image specific (i.e., specific for the date and time of image acquisition) surfaces of Rn and Ts

– Ta by reading relevant AVHRR images, Vegetation coverage, ea, Ta, and Rs_in surfaces.

The Rs_out, Rl_in, Rl_out, α and Ts are calculated ‘on-the-fly’, reducing the requirement to store

intermediate files on disk – therefore greatly reducing the need for hard disk space.

Calc_Rl_in

Tair.GRDea.GRD Albedo.GRDTs.GRD frac_Veg.GRD Rs_in.GRD

AVHRR Chns 4 and 5.GRD


Calc_Rl_out Calc_Rs_out(alb*Rs_in)


Rl_in.GRD Rl_out.GRD Rs_out.GRD

Calc_Rn

Rn.GRD

Split Window Algorithm

Saunders (1990)

Algorithm

At_Tr, DEM (z,slp,asp), ovp time

SOL_RAD

Calc_Rn_Surface_WRON.pro

Ts-Ta.GRD

Ts - Ta

Figure 21. Flowchart showing how Rn and Ts-Ta are generated.

4.12 Run RADWAT to create files containing overpass information RADWAT is a FORTRAN executable program that performs specific time-of-day resistance

energy balance modelling (Jupp 1990; Jupp et al. 1998). Here remotely sensed based Ts

data (derived from AVHRR; any sensor with thermal capacity could be utilised) was used as

input with the REBM being run in a dual-source mode. Using dual-source mode, means that

temperatures of both the ground and vegetation are considered (being modeled as a fraction

of vegetation cover), and to provide the extra constraint needed to solve the dual-source

REBM here a ‘minimum power’ constraint (i.e., where the sum of the square of the latent and


sensible heat fluxes from the ground and vegetation surfaces is minimized thereby

numerically simulating a conservative biophysical system) was solved. RADWAT allows

users to explore many aspects of the energy balance and options are available to run the

model in single-source mode (using both a Penman-Monteith and full case model) or by

providing the extras constraint in the dual-source that the aerodynamic temperature is

equivalent to the vegetation temperature. This constraint is not suitable to the vast areas of

low vegetation coverage in Australia. RADWAT also allows replacing forcing data from Ts to

either flux tower measurements of the sensible heat flux (H) or measurements of moisture

availability (the ratio of ETa / ETp). These options are primarily accessed where such

detailed flux tower data exist to support detailed processing understanding. As is common

for large area operational remote sensing-based systems (like that developed here)

meteorological data is not usually available for the specific time-of-day that the remotely

sensed data was acquired. To calculate specific time-of-day meteorological data RADWAT

implements the algorithms for temporal interpolation for air temperature, vapor pressure,

incoming shortwave radiation and wind speed reported in McVicar and Jupp (1999). From the AVHRR data the .OVP (or OVerPass) files are created for input into RADWAT

(along with the daily summary processed site files .PSF generated in Section 4.6; see also

Appendix 10.5). The specific time-of-day thermal remotely sensed driven output of RADWAT

are .RAD files, are saved as one file per station (i.e., many days per file (for each geographic

location where there is a BoM station)); see Figure 22. For input to ANUSPLIN, we needed

to convert these to one file per day (i.e., many stations per file (for each day in the time

series)). This is achieved by the IDL program called SortRadByImgAcq.PRO (Appendix

10.11) which produces the intermediate output file .NHW (or NDTI “Half Way” files).

Following McVicar and Jupp (2002), to generate specific time-of-day NDTI surfaces

ANUSPLIN was run using a quint-variate partial thin plate spline model, incorporating a bi-

variate thin plate spline function of longitude and latitude with constant linear dependencies

on Ts - Ta, VegCov and Rn. These data were extracted from the specific time-of-day surfaces

of Ts - Ta and Rn (generated in Section 4.11) and monthly surfaces of FC (generated in

Section 4.4) using the IDL program NHW2NIN.PRO (Appendix 10.11). In a previous

analysis of the sensitivity of the NDTI to its various forcing factors it was concluded that the

NDTI was relatively insensitive to perturbations in FC (McVicar and Jupp 2002), and as FC

changes more slowly than moisture availability, use of monthly time-step for FC data was

warranted.

The .NHW stands for NDTI “Half Way” file and .NIN stands for NDTI INput file (to SPLINA).

The data contained in each file are listed below. There are 5 columns (in order) in the .NHW

files being: 1:BoM station number; 2:longitude (dd); 3:latitude (dd); 4:DEM (m); and 5:NDTI


(ratio). In the .NIN files 8 variables are stored. They are (in order): 1: BoM station number;

2:longitude (dd); 3:latitude (dd); 4:DEM (m); 5: Rn (W m-2); 6: Ts - Ta (°C) 7: FC (%); and 8:

NDTI (ratio).

.OVP (cloud free)

RAD_WAT.PSF(Daily

Summary)

.RAD (Cloud Free,

Specific Times)

SortRadByImageAcq.PRO

.NIN (NDTI)

Rn.GRD

.NHW (NDTI)

NHW2NIN.PROTs-Ta.GRD

Veg_Cov.GRD

Create_Parameter_For_Radwat_RAD.pro

Check_Cols.pro

(for NDTI interpolation)

*_Checked.txt

(from PRERAD)

AVHRR Images

AVHRR_Cloud_Check.pro

Met Station Dictionary

OVP_Creation_WRON.pro)

Figure 22. Flowchart showing how the .OVP, .RAD and .NIN files are created.


4.13 Generate overpass NDTI surfaces NDTI surfaces for each relevant AVHRR image overpass time are interpolated using

ANUSPLIN Version 4.3 (Hutchinson 2004), specifically the SPLINA and LAPGRD programs

that are part of the ANUSPLIN package. Following McVicar and Jupp (2002), ANUSPLIN

was run with a quint-variate partial thin plate spline model, incorporating a bi-variate thin

plate spline function of longitude and latitude with constant linear dependencies on Ts - Ta,

VegCov and Rn. The .NIN files (Section 4.12) contain all variables needed by SPLINA

including geographic position, NDTI value (from RADWAT), as well as the specific time-of-

day values of Ts - Ta and Rn (generated in Section 4.11), and monthly values of FC

(generated in Section 4.4) generated at the BoM stations where the REBM was run. These

data are required as input into LAPGRD in order to generate the overpass NDTI surfaces.

Figure 23 schematically shows the steps needed to produce the NDTI surfaces, while

Appendix 10.12 shows the IDL code used.

SPLINA

Rn.GRD

NDTI.GRD

.SUR,.LOG

& others

LAPGRDTs-Ta.GRD

.NIN

Veg_Cov.GRD


Create_SPLINA_cmd_Files_NDTI.pro

LAPGRD CMD file

Create_LAPGRD_cmd_Files_NDTI.pro

Create_hdrs.PRO

Create_SPLINA_Batch_Runs_NDTI.pro

=

Create_LAPGRD_Batch_Runs_NDTI.pro

=

Figure 23. Flowchart of running ANUSPLIN to generate the overpass NDTI surfaces.

4.14 Generate fortnightly and monthly NDTI composites The individual specific time-of-day NDTI surfaces generated in section 4.13 above were

combined into both monthly (Section 5) and fortnightly (Section 9) composite images. The

IDL code used to perform the initial compositing is called Compostie_images.pro and is

shown in Appendix 10.13. To generate a single composite image, every specific time-of-day


NDTI image within a specified compositing period was read in to memory and stacked into a

single large multi-band image. The mean value through the stack was then determined for

each x, y location. Instances of missing data due to cloud cover or out-of-range values were

identified and not used in determining the mean value for a given x, y location. NDTI values

were considered out-of-range if they were < 0 or > 1. The single-band mean composite of

the multi-band stack was saved to disk. This process was repeated for every compositing

period between April 1992, when our data record starts, and December 2006, when our data

record ends. To help understanding of the compositing procedure, an example from a

hypothetical 4-day compositing period is given in Figure 24 below.

Day 1 Day 2 Day 3 Day 4 Result

0.55 0.50 0.60 0.55 0.65 0.60 0.60 -9999 0.60 0.55

0.90 0.20 1.00 0.10 1.10 -0.15 1.05 0.15 0.95 0.15 Figure 24. Schematic of NDTI compositing procedure for a theoretical 4-day composite. The numbers in the first four 2 x 2 arrays (left to right) represent single overpass NDTI values at the same locations for four days. Null values (depicted by a -9999, here) and out-of-range values (<0 or >1) are coloured gray and are excluded from the resultant mean composite calculation. The result of the mean NDTI compositing procedure for the 4 day observation period is shown in the final 2 x 2 array: the top-left pixel of the result is the mean of the 4 input top-left pixels, the top-right pixel is the mean of the 3 valid input top-right pixels (Day 4 is excluded due to a null value), the bottom-left pixel is the mean of the two valid bottom-left pixels (Day 3 and Day 4 are excluded due to high out-of-range values), and the bottom-right pixel is the mean of the three valid bottom-right pixels (Day 3 is excluded due to a low out-of-range value). Please note, the compositing periods actually used for this report were fortnightly (nominally 15 days) and monthly (determined by number of days in each month). The minimum compositing period for our work was a fortnight, which was strictly defined as

two composites per month. The first fortnight always consisted of the first 15 days of the

month and the second fortnight consisted of the rest of the days in that month. This means

that the first fortnightly composite was always 15 days, while the second composite was 13

days for non leap year Februaries, 14 days for leap year Februaries, 15 days for 30-day

months, or 16 days for 31-day months. We also generated monthly composites, which were

defined by the number of days in the month, so these ranged between 28 and 31 days. In

most cases, the compositing algorithm accessed multiple specific time-of-day NDTI images

for each day. For example, when generating the 24 fortnightly composites for 2006, 518

specific time-of-day NDTI images were used, which was an average of 21.58 images per

compositing period or 1.42 images per day for 2006.

Any gaps in these initial monthly or fortnightly composites due to clouds or out-of-range

values were then filled based on the overall mean value for the specific month (Figure 26) or


the overall mean value for the specific fortnight (Figure 50). The gap-filled pixles were

identified and saved in a separate image for each composite period. This means that the

original non gap-filled composite images can be recovered so other (more sophisticated)

methods can be assessed in the future. The gap-filled composites are displayed in Section 5

for monthly timesteps and in Section 9 for fortnightly timesteps. The IDL code used to

perform the gap filling of the initial composites is called

Fill_nulls_ndti_composites_hierarchy_2tier_only.pro and is shown in Appendix 10.14.


5 Monthly Gap-filled NDTI Composite Images from 1992 to 2006

The following figures are the gap-filled monthly NDTI composites from 1992 to 2006,

inclusive. The layout of the 12 months is schematically shown below (Figure 25). Each

month is given by the first 3 letters of that month, with the gap-filling following the method

outlined in Section 4.14 used. Gap-filling means that if a cell were blank (this means no

AVHRR data were available, or there are currently no AVHRR data available to produce a

within-range NDTI image) it was filled by the monthly average of entire time series (shown in

Figure 26). For example, in 1994 Table 2 shows there were less images available compared

to most other years, this was due to no AVHRR being acquired late in 1994 (Figure 2).

Additionally, the AVHRR data that were acquired in the winter of 1994 were observed very

late in the day (when Rn is positive and low, or negative, as the sun is close to setting)

resulting in very small Ts – Ta differences, and while RADWAT is numerically stable enough

to solve such challenging input data, physically unreasonable NDTI values occurred during

that period. First we show the monthly average NDTI over the entire time series (Figure 26),

and following this monthly gap-filled composites from 1992 to 2006, see Figure 27 to Figure

41 respectively, are provided.

Jan Feb Mar

Apr May Jun

Jul Aug Sep

Oct Nov Dec

Figure 25. Schematic showing the layout of the progression of monthly composite images, where each month is given by the first 3 letters of that month.


0 . 0 0 . 5 1 . 0 Figure 26. Overall average monthly NDTI composite images from 1992 to 2006, inclusive. When NDTI = 0.0 there is no moisture available for actual evapotranspiration (i.e., ETa = 0 mm day-1 or the environment is dry) and when NDTI = 1.0 moisture is freely available so actual evapotranspiration is occurring at its potential (i.e., ETa = ETp or the environment is wet). Black is the ocean.


0 . 0 0 . 5 1 . 0 Figure 27. Monthly NDTI composite images for 1992 (see caption of Figure 26 for more detail).


0 . 0 0 . 5 1 . 0

Figure 28. Monthly NDTI composite images for 1993 (see caption of Figure 26 for more detail).


0 . 0 0 . 5 1 . 0



0 . 0 0 . 5 1 . 0



0 . 0 0 . 5 1 . 0



0 . 0 0 . 5 1 . 0



0 . 0 0 . 5 1 . 0



0 . 0 0 . 5 1 . 0



0 . 0 0 . 5 1 . 0



0 . 0 0 . 5 1 . 0



0 . 0 0 . 5 1 . 0



0 . 0 0 . 5 1 . 0



0 . 0 0 . 5 1 . 0



0 . 0 0 . 5 1 . 0



0 . 0 0 . 5 1 . 0



6 Conclusions The NDTI - a direct measurement of moisture availability (ma; the ratio of ETa to ETp) - has

been developed for almost 9,000 afternoon single-overpass AVHRR images from April 1992

to 2006. The NDTI is calculated by inverting a 2-source resistance energy balance model

which is run at the specific date-time of each overpass. Fortnightly and monthly gap-filled

composite NDTI images have been developed from these single-overpasses. The resultant

NDTI images clearly illustrate on a temporal and spatial basis a signal which is driven by

climatic variability.

Ecohydrological assessment of interactions between the NDTI and the commonly used NDVI

for three hydrologically relevant study sites (Lake Victoria, Coleambally Irrigation Area, and a

dryland agriculture-forest interface near Tumut) show, as expected, that the NDTI responds

more quickly to changes in water availability than the NDVI does. This has implications for

adequately modelling soil evaporation that NDVI-based models of ETa can not adequately

account for. This observation (i.e., a more rapid response of the NDTI than the NDVI to a

key component of the water balance) is made at both fortnightly and monthly time-steps.

NDTI surfaces are derived by linking meteorological point data and optical (both thermal and

reflective) remotely sensed data in a way that optimally utilises process understanding of ma

and the inherent strengths of these different data types. NDTI is generated using a

‘calculate-then-interpolate’ approach to best use the strengths of the two primary datasets -

meteorological data are temporally dense (and spatially sparse), whereas remotely sensed

data are spatially dense (and temporally sparse). The spatial interpolation is performed

using covariates that are heavily governed by the spatially dense remotely sensed data,

which allow for better modelling of ma in-between meteorological stations (e.g., over water

bodies, which would otherwise not be well represented). Importantly, the NDTI is readily calculated across large regions; it does not require interpolated gridded precipitation or irrigation data as inputs, and therefore provides a means to detect ‘spurious and unaccounted’ water application thus enabling compliance to be monitored. Given that

ma ≅ NDTI, and that ma = ETa / ETp, then multiplying the resultant NDTI surfaces by daily

ETp surfaces provides a means to measure daily ETa with units of mm day-1.

To rapidly process the almost 9,000 AVHRR images, and to routinely process current and

future remotely sensed data from any number of possible sensors, a prototype operational

system has been developed. The design of the information system to generate the NDTI is

modular, requires little operator input, and automatic error checking methodologies have


been implemented. The completed prototype operational system could be extended to all

Australia. Further research quantifying the spatio-temporal variability and trends between

moisture availability (NDTI) and moisture utilization (NDVI) are warranted to assess risks to:

(1) landscape ecohydrological stability and functioning; and (2) agricultural production.

Resulting products and analysis would be suitable for distribution via on-line web-services as

they could be routinely produced and would be widely relevant to a number of water resource

and environmental managers and policy-makers. An example of one of the style of products

available from such time series analysis are the anomaly images (McVicar and Van Niel

2005) created by placing current conditions into historical context (McVicar et al. 2003).

While the remotely sensed data currently used to generate the NDTI are AVHRR, any sensor

with both thermal and reflective capacity – such as MODIS (MODerate Resolution Imaging

Spectroradiometer), ETM+ (Enhanced Thematic Mapper Plus), AATSR (Advanced Along-

Track Scanning Radiometer) and ASTER (Advanced Spaceborne Thermal Emission and

Reflection Radiometer) – could be used. This means that larger regional-scale studies can

be performed using 1 km resolution data (from the AVHRR, MODIS and AATSR

instruments), with smaller farm-to-paddock scale monitoring undertaken using 60 m

resolution ETM+ data and 90 m resolution ASTER data.


7 References

Allen RG, Pereira LS, Raes D, Smith M (1998) 'Crop evapotranspiration - Guidelines for computing crop water requirements.' FAO Irrigation and Drainage Paper 56, Rome, Italy http://www.fao.org/docrep/X0490E/X0490E00.htm.

Asrar G, Fuchs M, Kanemasu ET, Hatfield JL (1984) Estimating absorbed photosynthetic radiation and leaf-area index from spectral reflectance in wheat. Agronomy Journal 76, 300-306.

Bristow KL, Campbell GS (1984) On the Relationship Between Incoming Solar Radiation and Daily Maximum and Minimum Temperature. Agricultural and Forest Meteorology 31, 159-166.

Camargo JLC, Kapos V (1995) Complex edge effects on soil moisture and microclimate in central Amazonian forest Journal of Tropical Ecology 11, 205-221.

Carlson TN, Ripley DA (1997) On the relation between NDVI, fractional vegetation cover, and leaf area index. Remote Sensing of Environment (China) 62, 241-252.

Donohue RJ, Roderick ML, McVicar TR (2007) 'Correcting long-term AVHRR reflectance data using the vegetation cover triangle.' CSIRO Land and Water, Canberra http://www.clw.csiro.au/publications/science/2007/sr26-07.pdf.

Giambelluca TW, Ziegler AD, Nullet MA, Truong DM, Tran LT (2003) Transpiration in a small tropical forest patch. Agricultural and Forest Meteorology 117, 1-22.

Hungerford RD, Nemani RR, Running SW, Coughlin JC (1989) 'MTCLIM: A mountain microclimate simulation model.' USDA Forest Service, INT-414

Hutchinson MF (2004) ANUSPLIN Version 4.3 User Guide, http://cres.anu.edu.au/outputs/software.php. In. (The Australian National University, Centre for Resource and Environmental Studies: Canberra)

Jackson RD, Pinter Jr. PJ, Reginato RJ (1985) Net Radiation Calculated from Remote Multispectral and Ground Station Meteorological Data. Agricultural and Forest Meteorology 35, 153-164.

Jupp DLB (1990) Constrained two-layer models for estimating evapotranspiration. In 'Proceedings of the 11th Asian Conference on Remote Sensing'. Guangzhou, China, November, 1990 pp. B.4.1-B.4.6


Jupp DLB, Tian G, McVicar TR, Qin Y, Fuqin L (1998) 'Soil Moisture and Drought Monitoring Using Remote Sensing I: Theoretical Background and Methods.' CSIRO Earth Observation Centre, Canberra http://www.eoc.csiro.au/pubrep/scirpt/jstc1.pdf.

Kapos V (1989) Effects of isolation on the water status of forest patches in the Brazilian Amazon Journal of Tropical Ecology 5, 173-185.

King EA (1998) AVHRR Activities at the EOC in Canberra. In 'Proceedings of the Land AVHRR Workshop, 9th Australasian Remote Sensing and Photogrammetry Conference'. Sydney. (Ed. TR McVicar) pp. 59-65. (CSIRO Land and Water)

King EA (2000) Stitching the Australian 1 km archive. In 'Proceedings of the Land EnvSat Workshop, 10th Australasian Remote Sensing and Photogrammetry Conference'. (Ed. TR McVicar) pp. 41-49. (CSIRO Land and Water: Adelaide) http://www.clw.csiro.au/publications/technical2000/tr36-00.pdf.

King EA (2003) 'The Australian AVHRR Data Set at CSIRO/EOC: Origins, Processes, Holdings and Prospects.' CSIRO Earth Observation Centre Report 2003/04 (and CSIRO Atmospheric Research Report 2003/32), CSIRO Australia http://www.eoc.csiro.au/tech_reps/2003/tr2003_04.pdf.

Kneizys FX, Shettle EP, Abreu LW, Anderson GP, Chetwynd JH, Gallery WO, Selby JE, Clough SA (1983) 'Atmospheric transmittance/radiance: computer code LOWTRAN 6. Technical Report AFGL-TR-83-0187.' Optical Physics Division, U.S. Air Force Geophysics Laboratory, Hanscom Air Force Base, Massachusetts

Kustas WP, Humes KS (1997) Spatially distributed sensible heat flux over a semiarid watershed. Part II: Use of a variable resistance approach with radiometric surface temperatures. Journal of Applied Meteorology 36, 293-301.

Kustas WP, Jackson TJ (1999) The impact on area-averaged heat fluxes using remotely sensed data at different resolutions: A case study using Washita ‘92 Data. Water Resources Research 35, 1539-1550.

Kustas WP, Norman JM (1996) Use of remote sensing for evapotranspiration monitoring over land surfaces. Hydrological Sciences Journal des Sciences Hydrologiques 41, 495-516.

Lu H, Raupach MR, McVicar TR, Barrett DJ (2003) Decomposition of Vegetation Cover into Woody and Herbaceous Components Using AVHRR NDVI Time Series. Remote Sensing of Environment 86, 1-18.

Maher JV, Lee DM (1977) 'Upper Air Statistics Australia. Surface to 5 mb 1957 to 1975.' (Department of Science, Bureau of Meteorology: Canberra)


McVicar TR, Briggs PR, King EA, Raupach MR (2003) 'A Review of Predictive Modelling from a Natural Resource Management Perspective: The Role of Remote Sensing of the Terrestrial Environment.' CSIRO Land and Water Client Report to the Bureau of Rural Sciences (also available as CSIRO Earth Observation Centre Report 2003/03 and CSIRO Atmospheric Research Report 2003/31), Canberra, Australia http://www.clw.csiro.au/publications/consultancy/2003/Review_Of_Remote_Sensing.pdf.

McVicar TR, Jupp DLB (1998) The current and potential operational uses of remote sensing to aid decisions on Drought Exceptional Circumstances in Australia: A Review. Agricultural Systems 57, 399- 468.

McVicar TR, Jupp DLB (1999) Estimating one-time-of-day meteorological data from standard daily data as inputs to thermal remote sensing based energy balance models. Agricultural and Forest Meteorology 96, 219-238.

McVicar TR, Jupp DLB (2002) Using covariates to spatially interpolate moisture availability in the Murray-Darling Basin: a novel use of remotely sensed data. Remote Sensing of Environment 79, 199-212.

McVicar TR, Li LT, Van Niel TG, Hutchinson MF, Mu XM, Liu ZH (2005) 'Spatially Distributing 21 Years of Monthly Hydrometeorological Data in China: Spatio-Temporal Analysis of FAO-56 Crop Reference Evapotranspiration and Pan Evaporation in the Context of Climate Change.' CSIRO Land and Water Technical Report, 8/05, Canberra, Australia http://www.clw.csiro.au/publications/technical2005/tr8-05.pdf.

McVicar TR, Van Niel TG (2005) 'Deriving moisture availability from time series remote sensing for drought assessment. Report to the Reference Group of the Australian Water Availability Project (AWAP).' CSIRO Land and Water Client Report, Canberra, Australia http://www.clw.csiro.au/publications/consultancy/2005/deriving_moisture_availability_AWAP.pdf.

McVicar TR, Van Niel TG, Li L, Hutchinson MF, Mu X, Liu Z (2007) Spatially distributing monthly reference evapotranspiration and pan evaporation considering topographic influences. Journal of Hydrology 338, 196-220.

Mitchell RM (1999) 'Calibration status of the NOAA AVHRR solar reflectance channels: CalWatch Revision 1.' CSIRO Atmospheric Research Technical Paper No. 42, Melbourne http://www.eoc.csiro.au/cats/calwatch/.

Moore ID, Norton TW, Williams JE (1993) Modelling Environmental Heterogeneity in Forested Landscapes. Journal of Hydrology 150, 717-747.

Murray-Darling Basin Commission (2006) Lake Victoria. In. (http://www.mdbc.gov.au/rmw/river_murray_system/lake_victoria)


Murray FW (1967) On the computation of saturation vapour pressure. Journal of Applied Meteorology 6, 203-204.

Prata AJ (1996) A new long-wave formula for estimating downward clear-sky radiation at the surface. Quarterly Journal of the Royal Meteorological Society 122, 1127-1151.

Prescott JA (1940) Evaporation from water surface in relation to solar radiation. Transactions of the Royal Society of South Australia 64, 114-125.

Price JC (1991) Timing of NOAA afternoon passes. International Journal of Remote Sensing 12, 193-198.

Saunders RW (1990) The determination of broad band surface albedo from AVHRR visible and near-infrared radiances. International Journal of Remote Sensing 11, 49-67.

Smith M, Allen RG, Monteith JL, Perrier A, Pereira LS, Segeren A (1991) 'Expert consultation on procedures for revision of FAO guidelines for prediction of crop water requirements.' FAO Land and Water Development Division, Rome, Italy

Tetens O (1930) Über einige meteorologische Begriffe. Zeitschrift Geophysic 6, 297-309.

Van Niel TG, McVicar TR (2003) A simple method to improve field-level rice identification: Toward operational monitoring with satellite remote sensing. Australian Journal of Experimental Agriculture 43, 379-387.

Van Niel TG, McVicar TR (2004a) Current and potential uses of optical remote sensing in rice-based irrigation systems: a review. Australian Journal of Agricultural Research 55, 155-185.

Van Niel TG, McVicar TR (2004b) Determining temporal windows of crop discrimination with remote sensing: a case study in south-eastern Australia. Computers and Electronics in Agriculture 45, 91-108.

Van Niel TG, McVicar TR, Fang HL, Liang S (2003) Calculating environmental moisture for per-field discrimination of rice crops. International Journal of Remote Sensing 24, 885-890.

Vermote EF, Saleous NZ (2006) Calibration of NOAA16 AVHRR over a desert site using MODIS data. Remote Sensing of Environment 105, 214-220.

Wan Z, Dozier J (1996) A Generalized Split-Window Algorithm for Retrieving Land-Surface Temperature from Space. IEEE Transactions on Geoscience and Remote Sensing 34, 892-905.


Wilson JP, Gallant JC (2000) Secondary Topographic Attributes. In 'Terrain Analysis'. (Eds JP Wilson and JC Gallant) pp. 87-131. (John Wiley & Sons, Inc.: New York)


8 Appendix A: Monthly NDTI / NDVI Characterisation

For completeness, this appendix shows the same analysis for the same three areas used in

Section 3, expect data are provided at a monthly composite. That is NDTI and NDVI gep-

filled composite images are shown, the time-series from the transect running through the

centre of the composite image are shown, and pixel level time-series plots are shown. The

progression of the monthly images is shown in Figure 42. Results from this section highlight

the temporal detail provided at the fortnightly level (see Section 3) compared to the monthly

level provided here; where more generalised patterns (that may be more suited to some

management applications) are provided.

Jan Feb Mar

Apr May Jun

Jul Aug Sep

Oct Nov Dec

Figure 42. Schematic showing the layout of the progression of monthly composite images, where each month is given by the first 3 letters of that month.


8.1 Lake Victoria

0 . 0 0 . 5 1 . 0 (a) NDTI ≅ moisture availability

- 0 .0 5 0 .4 7 1 .0 0

(b) NDVI ≅ vegetation vigour Figure 43. Monthly NDTI images (a) and monthly NDVI images (b) around the Lake Victoria transect for 2001.




(c) (d) (e)

Figure 44. Monthly time series plot from 2001 for an 80 km transect located across Lake Victoria of (a) NDTI and (b) NDVI. The black line in (a) and (b) represent the profile of the first fortnightly compositing period in each case. Per-pixel time series are shown for pixels located at (c) 15 km, (d) 40 km, and (e) 65 km, respectively, with the black line being the NDTI and the grey line the NDVI.


8.2 Coleambally Irrigation Area


0 . 0 0 . 5 1 . 0 (b) NDVI ≅ vegetation vigour

Figure 45. Monthly NDTI images (a) and monthly NDVI images (b) around the CIA transect for 2001.




(c) (d) (e)

Figure 46. Monthly time series plot from 2001 for an 80 km transect located across Coleambally Irrigation Area of (a) NDTI and (b) NDVI. The black line in (a) and (b) represent the profile of the first fortnightly compositing period in each case. Per-pixel time series are shown for pixels located at (c) 15 km, (d) 40 km, and (e) 65 km, respectively, with the black line being the NDTI and the grey line the NDVI.


8.3 Forest - dryland agriculture interface at Tumut


0 .0 0 0 .4 3 0 .8 5

(b) NDVI ≅ vegetation vigour Figure 47. Monthly NDTI images (a) and monthly NDVI images (b) around the forest-dryland agriculture transect near Tumut for 2001.




(c) (d) (e)

Figure 48. Monthly time series plot from 2001 for an 80 km transect located across a forest-dryland agriculture interface near Tumut of (a) NDTI and (b) NDVI. The black line in (a) and (b) represent the profile of the first fortnightly compositing period in each case. Per-pixel time series are shown for pixels located at (c) 15 km, (d) 40 km, and (e) 65 km, respectively, with the black line being the NDTI and the grey line the NDVI.


9 Appendix B: Fortnightly Gap-filled NDTI Composite Images from 1992 to 2006

The following figures are the fortnightly NDTI gap-filled composites from 1992 to 2006,

inclusive. The layout of the 24 fortnights is schematically shown below (Figure 49). Each

month is given by the first 3 letters of that month, and 1 and 2 mean the first fortnight (days 1

through 15) and second fortnight (days 16 through to the end of the month) respectively.

First we show the fortnightly average NDTI over the entire time series (Figure 50), following

this the fortnightly composites from 1992 to 2006, see Figure 51 to Figure 65 respectively,

are shown.

Jan_1 Jan_2 Feb_1 Feb_2

Mar_1 Mar_2 Apr_1 Apr_2

May_1 May_2 Jun_1 Jun_2

Jul_1 Jul_2 Aug_1 Aug_2

Sep_1 Sep_2 Oct_1 Oct_2

Nov_1 Nov_2 Dec_1 Dec_2

Figure 49. Schematic showing the layout of the progression of fortnightly composite images, where each month is given by the first 3 letters of that month, and 1 and 2 mean the first fortnight (days 1 through 15) and second fortnight (days 16 through to the end of the month) respectively.


0 .0 0 .5 1 .0 Figure 50. Overall average fortnightly NDTI composite images for 1992 to 2006, inclusive. When NDTI = 0.0 there is no moisture available for actual evapotranspiration (i.e., ETa = 0 mm day-1 or the environment is dry) and when NDTI = 1.0 moisture is freely available so actual evapotranspiration is occurring at its potential (i.e., ETa = ETp or the environment is wet). Black is the ocean.


0 . 0 0 . 5 1 . 0 Figure 51. Fortnightly NDTI composite images for 1992 (see caption of Figure 50 for more detail).


0 . 0 0 . 5 1 . 0

Figure 52. Fortnightly NDTI composite images for 1993 (see caption of Figure 50 for more detail).


0 . 0 0 . 5 1 . 0



0 . 0 0 . 5 1 . 0



0 . 0 0 . 5 1 . 0



0 . 0 0 . 5 1 . 0



0 . 0 0 . 5 1 . 0



0 . 0 0 . 5 1 . 0



0 . 0 0 . 5 1 . 0



0 . 0 0 . 5 1 . 0



0 . 0 0 . 5 1 . 0



0 . 0 0 . 5 1 . 0



0 . 0 0 . 5 1 . 0



0 . 0 0 . 5 1 . 0



0 . 0 0 . 5 1 . 0



10 Appendix C: IDL code This appendix contains all the IDL code referred to in Section 4 and represents the bulk of

the important programming that was performed for this research. The sub-heading titles

below relate to the relevant sub-sections in Section 4 in which specific IDL programs were

referred to.

10.1 IDL code to quality assess SILO Point Patch Database annually Pro checkMetYearly ;Tom Van Niel ;13-Apr-2006 ;generates report of data quality for all SILO PPD text files in the ;SiloPPDdirPath SiloPPDdirPath='W:\Data\MET\SILO_PPD\' SiloPPDTemplate='U:\idl_code\AWAP\templates\siloPPDTemplate.sav' outTextFile='W:\metCheckReportYearly.txt' yrs=27 CD, SiloPPDdirPath staList = FILE_SEARCH('*.txt') outArray=fltarr((yrs)+1,n_elements(staList)) For i=0,n_elements(staList)-1 Do Begin sta=staList(i) data=readSiloMet(SiloPPDTemplate, sta) sta=STRSPLIT( sta, '.', /EXTRACT) outArray[0,i]=float(sta(0)) For j=0, yrs-1 Do Begin lowVal=19800000+(j*ulong(10000)) highVal=lowVal+10000 yearIndex=where((data.(0) ge lowVal) and (data.(0) lt highVal),numDays) ;convert structure to an array before passing to staion qual function yearData=fltarr(8,numDays) For k=0,7 Do Begin yearData[k,*]=data.(k)[yearIndex] EndFor staQual=stationQualityYearly(yearData) outArray[j+1,i]=staQual EndFor print, sta EndFor OPENW, 1, outTextFile PRINTF, 1, outArray, format="(I8, "+strtrim(string(yrs),2)+"F8.2)" CLOSE, 1 End


Function stationQualityYearly, data ;Tom Van Niel ;13-Apr-2006 ;performs data quality check on point patch SILO data ;returns 7 element fltarray of % orig met sta. data (non interp, etc) ;1st element is AND between Tmx,Tmn,P, the remaining 6 elements ; are Tmx,Tmn,P,Evap,Rdn,and VP ;e.g., ; staQual=stationQuality(data) flagList=[3,5,7] totNum=float(n_elements(data[0,*])) ;make tmp array to perform bitwise and b=bytarr(totNum) For m=0,2 Do Begin ;find location of non-interp met stations for the given var index=where(data[flagList(m),*] eq 0,count) ;put these in an array to perform bitwise and a=bytarr(totNum) if (count ne 0) then a[index]=1 dataQual=(count/totNum)*100 ;for the first 3 vars (Tx,Tn,P) do an "and" to find their intersection CASE m OF 0: b=a 1: b and= a 2: b and= a ENDCASE EndFor ;insert the AND of the first three vars into the first element of outarray tnNum=total(b) dataQual=(tnNum/totNum)*100. return, dataQual End


10.2 IDL code to quality assess BoM wind data and convert from wind run to wind speed

Pro WindRun_to_WindSpeed ;Tom Van Niel ;01-Mar-2007 ;converts BOM wind run data to windspeed that will eventually be appended ;to SILO PPD data and saved to *.dat format for use in NDTI calculations ; ;this program outputs a single textfile for each station of BOM windrun ;data. The output file contains: Year, Month, Day, DOY, Windspeed ; ;These separate windspeed text files need to be appended to a copy of the ;SILO PPD data BEFORE conversion to *.dat format ; ;This program also outputs any years that have nDays <> 365 or 366 so ;they can be checked to make sure the data are all consecutive (no gaps) ; ;the following is output to the screen: ;STA, startyear, endyear, yearofinterest, nDaysinYearofInterest, ;startDayinYearofInterest, endDayinYearofInterest ; ;From this info, we can check that the yearofinterest (i.e., the year ;that might not have consecutive data)is in fact either the start or end ;year of record and that the nDays in theyearofinterest equals the ;endDay - startDay + 1 ; ; If the yearofinterest is either the start or end ;year and the nDays match endDay - startDay + 1, then everything is OK. ; ;set null data value ND=-2147483648 ;set ouput directory outname='W:\Data\MET\Wind_Data_BOM\Calculated_Windspeed_from_WindRun\' ;read in station file ENVI_READ_COLS, 'W:\spreadsheets_and_texts\stations_nostrings.txt', sta_array BIGstalist=sta_array(0,*) latlist=sta_array(3,*) ;read in and manipulate BOM wind run data BOMwindDirPath='W:\DATA\MET\wind_data_BOM\' BOMwindAsciiTemplateFN='U:\idl_code\AWAP\BOMwindAsciiTemplate.sav' ;outTextFile='U:\AWAP\spreadsheets_and_texts\BOM_wind_daylength.txt' CD, BOMwindDirPath staList = FILE_SEARCH('*DATA*.txt') summaryArray=lonarr(8,n_elements(staList)) For i=0,n_elements(staList)-1 Do Begin sta=staList(i) ;read in BOM wind run ascii file ;data = READ_ASCII(sta, DATA_START=1, DELIMITER=',') data=readBOMWindAscii(BOMwindAsciiTemplateFN, sta) windrun=data.(15) ;calc DOY for first record year=data.(2) month=data.(3)


day=data.(4) qualityFlag=data.(16) accDays=data.(17) ;make sure to remove last record, which is a bogus line nrec=n_elements(data.(0))-1 year=year[0:nrec-1] month=month[0:nrec-1] day=day[0:nrec-1] qualityFlag=qualityFlag[0:nrec-1] accDays=accDays[0:nrec-1] DOY=day ;put doy of first day of each month in a list to use in DOY calculation doylist=[0,31,59,90,120,151,181,212,243,273,304,334] For j=1,12 do begin index=where(month eq j, count) if (count ne 0) then doy[index]=doy[index]+doylist[j-1] Endfor ;adjust doy if it is a leap year ; leap year if it is divisible by 4 but not by 100 ; If a year is divisible by 4 and by 100, it is not a leap ; year unless it is also divisible by 400 leapindex=where(((year mod 4 eq 0) AND (year mod 100 eq 0) AND (year mod 400 eq 0)) OR $ ((year mod 4 eq 0) AND (year mod 100 ne 0)) AND (month ge 3),leapcount) if (leapcount ne 0) then doy[leapindex]=doy[leapindex]+1 sta=STRSPLIT( sta, '_', /EXTRACT) stanum=long(sta(2)) sta=STRTRIM(STRING(stanum),2) ;now find the station number in the overall list and get its latitude staindex=where(bigstalist eq stanum, stacount) lat=latlist[staindex] ;we have the lat and the doylist, so call the daylength function N=calcDayLength(lat(0), doy) ;as the BOM wind run is predominantly measured on the previous day from ;which it was recorded, need to shift the wind run data to be associated ;with doy-1 e.g., windrun data is measured from 9am Monday to 9am Tuesday ;and then recorded at 9am Tuesday. This means that Tuesday's reading is ;mostly associated with Monday ; ;to fix this, truncate first record of windrun data and last record of ;doy and daylength ; ;when adding back to textfile, start at record 1, not 0 windrun=windrun[1:nrec-2];truncate first record for day shift and last record as it is just a blank line doy=doy[0:nrec-3];truncate last two lines - the last blank line and the 2nd last to match the day shift day=day[0:nrec-3] month=month[0:nrec-3] year=year[0:nrec-3] qualityFlag=qualityFlag[0:nrec-3] accDays=accDays[0:nrec-3] N=N[0:nrec-3]


;1/3 windrun occurs at night, 2/3 occurs during day, ; so mult 2/3 by windrun and mult by 1000 to place units in m ; and divide by daylength in seconds (have to mult N*60*60) windspeed=(((2./3)*windrun)*1000.)/(N*60.*60) ;define first mask for windspeed - if windrun = 0, then windspeed = -1 ;(null) index=where((windspeed eq 0) OR (windrun eq ND), zerocount) ;null windspeed based on the first mask if (zerocount ne 0) then windspeed[index]=-1 ;define 2nd mask for windspeeds based on the qualityFlag and num days ;used to accum. the windrun ; and use the mask to null out any innappropriate values index=where((qualityFlag ne 'W') AND (qualityFlag ne 'S') AND (qualityFlag ne 'I') AND ((accDays eq 1) OR (accDays eq '')),passed, complement=nulls, ncomplement=failed) if (failed ne 0) then windspeed[nulls]=-1 ;now determine how many of the valid records pass and fail the quality ;and accumulation criteria qualFail=0 & accFail=0 if (Passed ne 0) then begin qualindex=where((qualityFlag eq 'W') OR (qualityFlag eq 'S') OR (qualityFlag eq 'I'), qualFail) accindex=where((accDays ne 1) AND (accDays ne ''), accFail) nindex=where(qualityflag eq 'N', ncount) endif summaryArray[*,i]=[stanum,nrec-2, passed, failed, qualFail,accfail,ncount,zerocount] ;split up the windspeed into yearly chunks and check whether there are ;any years with data that does not eq 365 or 366 ; if there are, then print them to the screen firstyear=year[0] lastyear=year[n_elements(year)-1] numyears=lastyear-firstyear+1 yearArray=fltarr(numyears,366) ;print, stanum, firstyear, lastyear For j=0,numyears-1 Do begin index=where(year eq firstyear+j,count) firstindex=index[0] lastindex=index[count-1] firstday=doy[firstindex] lastday=doy[lastindex] if ((count ne 366) AND (count ne 365)) then print, stanum, firstyear, lastyear, firstyear+j, count, firstday, lastday Endfor ;output station file to output directory outArray=[[year],[month],[day],[doy],[windspeed]] outArray=Transpose(outArray) OPENW, 1, outname+sta+'.txt' PRINTF, 1, outArray, format="(4I6, F12.4)" CLOSE, 1 EndFor ;now print out summary of missing and masked values print, 'Station',' nRecs',' validWindData',' Passed',' Failed',' FailedQual',' FailedAccum' print, summaryarray End function calcDayLength, lat, doy ;calculate daylength based on FAO-56 crop ET pub:


; FAO book (ISBN 92-5-104219-5) Crop evapotranspiration - ; Guidelines for computing crop water requirements - ; FAO Irrigation and drainage paper 56 (http://www.fao.org/docrep/X0490E/x0490e00.htm). ;input must be a single latitude value and an array of day of year ;returns same size array of daylength in decimal hours ;e.g., ;ENVI> lat=37.5 ;ENVI> doy=indgen(365)+1 ;ENVI> N=calcDayLength(lat, doy) ;ENVI> print, N[0:5] ; 9.46857 9.47912 9.49052 9.50276 9.51582 9.52971 ;-------------------------------------------------------------------------- ; USING EQ's 24, 25, & 34: ; Daylight hours, N = (24/pi)*Ws ; Ws = sunset hour angle in radians = arccos [-tan (latitude) tan (sol_dec_angle ) ; sol_dec_angle = 0.409*sin(2pi/365*DOY-1.39) ;initialise variables radians=!pi/180. ;calculate solar declination angle solar_dec=0.409*sin(2.*!pi/365*doy-1.39) ;calculate sunset hour angle in radians Ws=acos(-1.*tan(radians*lat)*tan(solar_dec)) ;calculate daylength N=(24/!pi)*Ws ;return daylength array return, N End function readBOMWindAscii, templateFN, dataFN ;reads in met ascii files (e.g., BOM wind run) ; into pre-defined structure ;e.g., for BOM wind run: ; data=readBOMWindAscii('U:\AWAP\idl_code\BOMwindAsciiTemplate.sav', 'U:\AWAP\Data\MET\Wind_Data_BOM\DC02D_Data_090103_999999991387493.txt') ; ;restore the pre-defined ascii template for the met data ; after it is restored, it is a var called by this name restore, templateFN ;read in the data using the structure data = READ_ASCII(dataFN, TEMPLATE=BOMwindAsciiTemplate) return, data End


10.3 IDL code to convert fPAR data to percent vegetation cover PRO fpar2cover, mth, res, dset ; Purpose: convert fPAR to coverage ; mth variable specifying the month - integer value ; dset array containing the returned dataset ; the Equation from Randal is c = 100 * F / 0.95 nd = -999. ; month 135 - 287 for mth = 288, 311 do begin openr,unit,string('W:\DATA\RS\corrected_fpar\fpar01_',mth,'.rst',format='(A,I0,A)'),/get_lun temp = assoc(unit,bytarr(4501,3501),0) dset = temp[0] & free_lun,unit nulCell = where (dset eq 0) ;null is 0 outC = 100 * (dset/100.0 - 1) /0.95 ;fPAR was scaled to 100 - 195. outC[nulCell] = nd ;put back null value OPENW, cIMG, 'W:\DATA\RS\corrected_fpar\cvrg01_'+strtrim(string(mth),2)+'.flt', /get_lun WRITEU, cIMG, outC close, cIMG free_lun, cIMG endfor END pro subset_f_coverage, imageName ;20 April 2007 ;Purpose: subset SEA part of the f_coverage surfaces from whole Australia ; surfaces ;Input: forest coverage surfaces which are converted from Randall's ; fpar surfaces ;Output: forest coverage surfaces just for south east australia shortFile = file_basename(imageName) randall_mth = fix(strmid(shortFile, 6,3, /REVERSE_OFFSET)) year = 1981 + randall_mth/12 month = randall_mth mod 12 + 1 year = strtrim(string(year),2);convert to string if month lt 10 then month = '0' + strtrim(string(month),2) else month = strtrim(string(month),2);convert to string ND = -9999. ;get the right vegetation coverage image and subset for SEA outPath='W:\Data\RS\corrected_fpar\converted_coverage\SE_Australia\ outName = year + month + '_' + strtrim(string(randall_mth),2) + '.flt' samples = 1601 lines = 1701 f_image = fltarr(samples,lines) if (file_test(imageName) eq 0) then print, 'f image not fount' else begin ;open coverage file f_samples = 4501 f_lines = 3501 in_cvrg = fltarr(f_samples,f_lines) subset_start = [2800, 1300];see above 2 lines OPENR, surface_in, imageName, /GET_LUN readu, surface_in, in_cvrg free_lun, surface_in ;get the right subset for South East Aus f_image = in_cvrg[subset_start[0]:subset_start[0]+samples -1, subset_start[1]: subset_start[1]+lines - 1] endelse ;change null from -999. to -9999. w=where (f_image eq -999., count)


if (count gt 0) then begin f_image(w) = -9999. end ;Write out Rn image OPENW, surface_out, outPath + outName, /get_lun WRITEU, surface_out, f_image CLOSE, surface_out free_lun, surface_out End f_path = 'W:\Data\RS\corrected_fpar\converted_coverage\' f_List = file_search(f_path +'*.flt', count=FileNumber) If (FileNumber eq 0) then print, 'no f_image found' else begin ;For j=0, FileNumber-1 Do Begin For j=FileNumber-25, FileNumber-1 Do Begin subset_f_coverage, f_List[j] Endfor endelse end function fill_nulls, inImage, fillImage, ND ;TVN/29-Apr-2007/ ;finds nulls in an input image and fills it with values from another image ; ;assumes inImage and fillImage are same size and type and that ;the fillimage has no nulls outImage=inImage valid=where(inImage ne ND, vCount, complement=nulls, ncomplement=ncount) if (ncount ne 0) then outImage[nulls]=fillImage[nulls] return, outImage END pro fill_nulls_veg_cov_DOIT ;TVN/29-Apr-2007/ ;fills all nulls in each of the monthly veg cover data with monthly mean ;data ;initialise variables samples=1601 lines=1701 basepath='W:\Data\RS\corrected_fpar\converted_coverage\SE_Australia\' outpath=basepath+'\filled_nulls\' fillFileName=basepath+'monthly_mean.flt' ext='flt' ND=-9999. month_num_array=['01','02','03','04','05','06','07','08','09','10','11','12'] z=n_elements(month_num_array) outarray=fltarr(samples,lines) ;Read in fill data fImage = fltarr(samples,lines,z) ;Open input files OPENR, unit, fillFileName, /GET_LUN readu, unit, fImage free_lun, unit For month=0, z-1 DO BEGIN monthstring = month_num_array[month]


fileList=FILE_SEARCH(basepath,'*'+monthstring+'_'+'*.'+ext) numImages=n_elements(fileList) mnthFillImage=fImage[*,*,month] For i=0, numImages-1 Do Begin ;Now read image data into a variable inVAR = fltarr(samples,lines) ;Open input files inVARName = fileList[i] OPENR, unit, inVARName, /GET_LUN readu, unit, inVAR free_lun, unit ;call the function that fills the in image outArray=fill_nulls(inVar, mnthFillImage, ND) ;write out stacked image OPENW, unit, outpath+file_basename(fileList[i]) WRITEU, unit, outarray CLOSE, unit free_lun, unit print, outpath+file_basename(fileList[i]) Endfor EndFor END

10.4 IDL code to generate annual .DAT files and .SS files pro make_dat_ss ;TVN/02-Mar-2007 ;Replaces fortran code called "dic2hdr" ;Generates silo *.dat file and SS file with correct headers ; takes in a dictionary file (station file) and silo PPD data ; gleans relevant info from station file to generate the correct ; header needed for processing, and merges with SILO PPD data to ; make the appropriate *.dat file. Also generates SS files. ; Puts data in separate directories by year ;initialise variables threshold=90 startyear=2006 path='W:\' SiloPPDdirPath=path+'Data\MET\SILO_PPD\' datbasepath=Path+'Data\RADCALC\DATS\' ssbasepath=Path+'Data\RADCALC\SS\' siloTemplateFN='U:\idl_code\AWAP\templates\siloPPDTemplate.sav' staTemplateFN='U:\idl_code\AWAP\templates\staTemplate.sav' staDataFN=path+'spreadsheets_and_texts\stations.txt' ;metCheckyearlyFN=path+'spreadsheets_and_texts\metCheckReportYearly.txt' metCheckyearlyFN=path+'metCheckReportYearly_2006only.txt' BOMwindAsciiTemplateFN='U:\idl_code\AWAP\BOMwindAsciiTemplate.sav' BOMwinddirPath=Path+'Data\MET\Wind_Data_BOM\Calculated_Windspeed_from_WindRun\' ;get sta list from BOM Wind directory CD, BOMwindDirPath BOMstaList = FILE_SEARCH('*.txt') If (n_elements(BOMstalist) eq 0) then print, kdjfkdjf


;read in text files ;read in station text file using structure restore, staTemplateFN staData = READ_ASCII(staDataFN, TEMPLATE=staAsciiTemplate) ;read in metcheckyearly text file, which is a summary of quality of each station for a given year envi_read_cols, metCheckYearlyFN, metCheckYearly totStaList=metCheckYearly[0,*] ;For each year in metCheckYearly, define the list of valid stations based on the input threshold ;start at 1 to skip station identifier column For j=1, n_elements(metCheckYearly[*,0])-1 Do Begin ;get index of valid stations staIndex=where(metCheckYearly[j,*] GE threshold, nValid) staList=totStaList[staIndex] ;go through each valid station in stalist and extract relevant info For i=0, n_elements(staList)-1 Do Begin sta=STRTRIM(STRING(long(staList(i))),2)+'.txt' datpath=datbasepath+STRTRIM(STRING(startyear+j-1),2)+'\' sspath=ssbasepath+STRTRIM(STRING(startyear+j-1),2)+'\' data=readSiloMet(SiloTemplateFN, SiloPPDdirPath+sta) sta=STRSPLIT( sta, '.', /EXTRACT) ;define index for the year of interest yrIndex=where((data.(0) gt ((startyear+j-2)*(10000.)+1220)) AND $

(data.(0) lt ((startyear+j)*(10000.)+0111)),nDays) first=STRTRIM(STRING(data.(0)[yrIndex[0]]),2) first=STRMID(first, 0, 4)+' '+STRMID(first, 4, 2)+' '+STRMID(first, 6,$

2) ;find the selected station in the dictionary and get relevant info staIndex=where(staData.(0) eq long(sta(0))) staName = staData.(2)[staIndex] lat = staData.(5)[staIndex] latdeg = fix(lat) latmin = round((abs(lat) - abs(latdeg))*60.) lon = staData.(6)[staIndex] londeg = fix(lon) lonmin = round((abs(lon) - abs(londeg))*60.) elev = round(staData.(9)[staIndex]) CASE staData.(8)[staIndex] OF 'ACT': timeZone = 10 'NSW': timeZone = 10 'QLD': timeZone = 10 'TAS': timeZone = 10 'VIC': timeZone = 10 'NT': timeZone = 9.5 'SA': timeZone = 9.5 'WA': timeZone = 8 ENDCASE ; outarray=fltarr(5,nDays) outarray[0,*]=data.(1)[yrIndex] outarray[1,*]=data.(2)[yrIndex] outarray[2,*]=data.(4)[yrIndex] outarray[3,*]=data.(6)[yrIndex] outarray[4,*]=-1 ;now check to see if there is windspeed data avail for this station from ;BOM ; if so, then replace relevant part of windspeed staIndex=where(BOMstalist eq sta[0]+'.'+sta[1], staIndexCount)


if (staIndexCount ne 0) then begin ;read in BOM wind SPEED ascii file for the given station ENVI_READ_COLS, BOMwindDirPath+sta[0]+'.'+sta[1], BOMdata ymd=BOMdata(0,*)*10000.+BOMdata(1,*)*100.+BOMdata(2,*) ;define index for the year of interest BOMyrIndex=where((ymd ge ((startyear+j-2)*(10000.)+1222)) AND (ymd le$

((startyear+j)*(10000.)+0110)),BOMnDays) if (nDays ne 0) then Begin ;get the wind data BOMwind=BOMdata[4,*] BOMwind=BOMwind[BOMyrIndex] ;find the starting point referenced through the doy doyStart=BOMdata[3,BOMyrIndex[0]] doyEnd=BOMdata[3,BOMyrIndex[BOMnDays-1]] oafirst=where(outarray[0,*] eq doyStart, oafnum) oalast=where(outarray[0,*] eq doyEnd, oalnum) ;overwrite the -1's in the wind data with the BOM values if (oafnum gt 1) then oafirst=min(oafirst) if (oalnum gt 1) then oalast=max(oalast) outarray[4,oafirst:oalast]=BOMwind endif endif ;output *.DAT file outfile = datpath+STRTRIM(STRING(sta(0)),2)+'.dat' OPENW, 1, outfile PRINTF, 1, STRTRIM(STRING(nDays),2), $ format="(A"+STRTRIM(fix(alog10(nDays)+1),2)+")" PRINTF, 1, staName, 'BOM ID',sta(0), $ format="(A"+STRTRIM(n_elements(aName),2)+", A7, A6)" PRINTF, 1, 'Date of first data (Year Month Day) PRINTF, 1, first, format="(A10)" PRINTF, 1, 'Latdeg Latmin Longdeg Longmin Time Zone PRINTF, 1, latdeg, latmin, londeg, lonmin, timeZone, format="(2I3,I4,I3,$

F5.1)" PRINTF, 1, 'Altitude (m) Slope (deg) Aspect (deg) NE & NW Horizons (deg) PRINTF, 1, elev,0.0,0.0,0.0,0.0, $ format="(I"+STRTRIM(fix(alog10(elev)+1),2)+".1, 4f4.1) PRINTF, 1, 'Base Station Z reference (Ht of screen (m))' PRINTF, 1, 2, format="(I1)" PRINTF, 1, 'Bristow & Campbell Atc(0), B & C Coefficients' PRINTF, 1, '0.807 0.175 0.849' PRINTF, 1, 'Daily Met Data (DOY Tmax (deg C) Tmin (deg C) Precip (mm)$

Wind (ms-1))' PRINTF, 1, outarray, format="(I3,4f8.2)" CLOSE, 1 ;Output the *.SS file outfile = sspath+STRTRIM(STRING(sta(0)),2)+'.ss' OPENW, 1, outfile PRINTF, 1, staName, 'Actual Pnt BOM ID',sta(0), $ format="(A"+STRTRIM(n_elements(aName),2)+", A18, A6)" PRINTF, 1, latdeg, latmin, londeg, lonmin, timeZone, format="(2I3,I4,I3,$

F5.1)" PRINTF, 1, elev,0.0,0.0,0.0,0.0, $ format="(I"+STRTRIM(fix(alog10(elev)+1),2)+".1, 4f4.1) PRINTF, 1, '1.0 0.15'


CLOSE, 1 EndFor EndFor End function readSiloMet, templateFN, dataFN ;reads in silo Met station text file into pre-defined structure ;returns data SILO met data array ;e.g., ; data=readSiloMet('G:\AWAP\idl_code\siloPPDTemplate.sav',$ ; 'G:\AWAP\SILO_PPD\11001.txt') ; ;restore the pre-defined ascii template for the SILO met data ; this template was named siloAsciiTemplate and when restored ; is a var called by this name restore, templateFN ;read in the data using the structure data = READ_ASCII(dataFN, TEMPLATE=siloAsciiTemplate) return, data End function readBOMWindAscii, templateFN, dataFN ;reads in met ascii files (e.g., BOM wind run) ; into pre-defined structure ;e.g., for BOM wind run: ; data=readBOMWindAscii('U:\AWAP\idl_code\BOMwindAsciiTemplate.sav',$ ; 'U:\AWAP\Data\MET\Wind_Data_BOM\DC02D_Data_090103_999999991387493.txt') ; ;restore the pre-defined ascii template for the met data ; after it is restored, it is a var called by this name restore, templateFN ;read in the data using the structure data = READ_ASCII(dataFN, TEMPLATE=BOMwindAsciiTemplate) return, data End

10.5 IDL code to run PRERAD to generate .PSF files ;This pro create parameter files for prerad_b.exe, with one parameter file per year ;Started: 2007.03.09 ;By: Li Lingtao Pro parameter4Prerad rootDir = 'W:\DATA\RADCALC\' for year = 2006, 2006 do begin outFileDir = rootDir + 'psf\' + strtrim(string(year),2) ;check if the destination directory exists, create one if not outDirOK = file_search(outFileDir,COUNT=N_FOUND) IF N_FOUND EQ 0 THEN SPAWN, 'mkdir ' + outFileDir outFile = outFileDir + '\'+ strtrim(string(year),2)+'_prerad.par' openw, fOut, outFile, /get_lun; datFiles=file_search(rootDir+'dats\'+strtrim(string(year),2)+'\',$

'*.dat', count=datFileNum) datFileNameOnly = strarr(datFileNum) for i=0, datFileNum-1 do begin datFileNameOnly[i]=(strsplit(file_basename(datFiles[i]),'.',$

/extract))[0]


endfor ssFiles=file_search(rootDir+'ss\'+strtrim(string(year),2)+'\',$ '*.ss', count=ssFileNum) ssFileNameOnly = strarr(ssFileNum) for i=0, ssFileNum-1 do begin ssFileNameOnly[i]=(strsplit(file_basename(ssFiles[i]),'.',$

/extract))[0] endfor for i=0, datFileNum-1 do begin ssFile = where (ssFileNameOnly eq datFileNameOnly[i], count) if count GT 0 then begin ;check if the output .psf exists, delete it if yes,

;otherwise this par file won't work outFileName = rootDir + $ 'psf\'+strtrim(string(year),2)+'\'+ $ datFileNameOnly[i]+'.psf' outFileOK = file_search(outFileName,COUNT=N_FOUND) IF N_FOUND EQ 1 THEN FILE_DELETE, outFileName;SPAWN, $

'del ' + outFileName printf, fOut,"b" printf, fOut, datFiles[i] printf, fOut, "s" printf, fOut, ssFiles[ssFile] printf, fOut, "d" printf, fOut, outFileName endif endfor printf, fOut,"q" close, fOut free_lun, fOut endfor end

10.6 IDL code to generate and interpolate daily atmospheric transmittance

pro SortPSFByDay ;TVN/02-Apr-2007 ;Takes in psfs and sorts all valid stations into separate ;files for each image acquisition ; ;initialise variables startyear=2006 endyear=2006 path='W:\Data\RadCalc\psf\' outpath='W:\Data\RADCALC\Spatial_Int_Daily\transmittance\' staFile='W:\spreadsheets_and_texts\stations_nostrings.txt' ;read in station file to get lat longs ;read in station file ENVI_READ_COLS, staFile, sta_array BIGstalist=sta_array(0,*) BIGlatlist=sta_array(3,*) BIGlonlist=sta_array(4,*)


For i=startyear, endyear Do Begin ;get sta list from BOM Wind directory CD, path+strtrim(string(i),2) psfList = FILE_SEARCH('*.psf') If (n_elements(psflist) eq 0) then print, kdjfkdjf ;Read in each psf file and concat them together For j=0, n_elements(psflist)-1 Do Begin ;read in psf data into a variable called matrix data = READ_ASCII(path+strtrim(string(i),2)+'\'+psflist[j],$

DATA_START=14, DELIMITER=',') matrix=data.(0) dateArray=lonarr(n_elements(matrix[0,*])) ;However, since read_ascii is a hunk of garbage, it messes up the date ;field. Therefore, use this code to manually read overwrite matrix. ;Still use read_ascii above in order to automatically define the size ;of the matrix aline = '' startrec=14 counter=0 OPENR, unit, path+strtrim(string(i),2)+'\'+psflist[j], /GET_LUN while not eof (unit) do begin readf, unit, aline aRec=strSplit(aline, ',', /Extract) if (counter ge startRec) then datearray[counter-startrec]=aRec[1] counter++ endwhile free_lun, unit ;set up necesarry lists stalist=lonarr(n_elements(matrix[0,*])) lonList=fltarr(n_elements(matrix[0,*])) latList=fltarr(n_elements(matrix[0,*])) ;determine the station, lat, and lon for this file and populate the ;relevant lists ;station list a=strsplit(psflist[j],'.',/EXTRACT) a=long(a[0]) stalist[*]=a ;lon and lat lists aindex=where(BIGstaList eq a) lonlist[*]=BIGlonlist[aindex] latlist[*]=BIGlatList[aindex] ;insert station, lon, and lat lists into the BIG rad array matrix=[[matrix],transpose(stalist), transpose(lonlist),$ transpose(latlist)] ;concat each separate array together into a big array if (j eq 0) then Begin bigArray=matrix ImgID=transpose(datearray) Endif else Begin bigArray=[[bigArray],[matrix]] ImgID=[[ImgID],[transpose(dateArray)]] endelse Endfor ;Sort the big array based on image acquisition


; get list of image acquisitions from the big array ;make a unique identifier for each overpass by using date (*100) plus ;time (to one decimal place) ;imgID needs to be rearranged first imgAcqList=imgID[UNIQ(imgID, SORT(imgID))] ;Loop through each unique image acquistion and get any valid data For k=0, n_elements(imgAcqList)-1 Do Begin index=where(imgID eq imgAcqList[k], count) ;write out all data for this day ;insert filter to the line below if needed (e.g., zen angle < 45 ;degrees) outarray=fltarr(4,count) outarray[0,*]=bigarray[11,index]; station number outarray[1,*]=bigarray[12,index]; longitude outarray[2,*]=bigarray[13,index]; latitude outarray[3,*]=round(bigarray[10,index]*100.); Trdir (trans % * 100) ;name outarray by image acq date and include all stations OPENW, 1, outpath+strtrim(string(i),2)+$

'\'+strtrim(string(long(imgAcqList[k])),2)+'.trn' ;PRINTF, 1, 'Station ', 'Lon ', 'Lat ', 'Trdir_(%*100)' PRINTF, 1, outArray, format="(I10, 2F10.4, I10)" CLOSE, 1 print, outpath+strtrim(string(i),2)+'\'+$

strtrim(string(long(imgAcqList[k])),2)+'.trn' Endfor Endfor End

pro Create_SPLINA_cmd_Files ;TVN/LLT Apr-2007 ;LLT Modified on 30 April 2007 ;create cmd files for each surface to be splined ; assumes a yearly directory structure ; requires a template *.cmd file as input at the base directory specified below ;initialise variables startyear=2006 endyear=2006 path='W:\DATA\RADCALC\Spatial_Int_Daily\transmittance\' outpath='W:\DATA\RADCALC\Spatial_Int_Daily\transmittance\SPLIN_cmd_files\' surfacepath='W:\DATA\RADCALC_SURFACES\At_Trans_0m' ext='trn' magicRecord=[19,20,21,22,23] template=outpath+'template_splna.cmd' ;make sure template exists Result = FILE_TEST(template) if (result ne 1) then print, dkfjkdj For i=startyear, endyear Do Begin ;put all files into a list CD, path+strtrim(string(i),2) List = FILE_SEARCH('*.'+ext) If (n_elements(List) eq 0) then print, kdjfkdjf For j=0, n_elements(List)-1 Do Begin filename=strsplit(list[j],'.', /Extract)


mmdd=strmid(filename[0],4,4) ;read in the template file and modify the "magic" record where the ;filename is kept counter=0 aline='' sarray=strarr(1) OPENR, unit, template, /GET_LUN while not eof (unit) do begin readf, unit, aline if counter eq 14 then begin strput, aline, strtrim(string(i),2)+'\'+filename[0],48 endif else begin tmp = where (magicRecord eq counter+1, count) if count gt 0 then strput, aline, $ strtrim(string(i),2)+'\'+filename[0],64 endelse if (counter eq 0) then begin strput, aline, filename[0],0 $

sarray[0]=aline endif else sarray=[[sarray],[aline]] counter++ endwhile free_lun, unit ;output the new modified cmd file using input basename OPENW, 1, outpath+strtrim(string(i),2)+'\'+filename[0]+'_SPLINA.cmd' PRINTF, 1, sarray CLOSE, 1 print, outpath+strtrim(string(i),2)+'\'+filename[0]+'_SPLINA.cmd' Endfor Endfor End pro Create_Splina_Batch_Runs ;TVN/05-Apr-2007 ;create batch file to run all splina commands ;assumes a yearly directory structure ;initialise variables startyear=2006 endyear=2006 path='W:\DATA\RADCALC\Spatial_Int_Daily\transmittance\SPLIN_cmd_files\' outpath='W:\DATA\RADCALC\Spatial_Int_Daily\transmittance\SPLIN_cmd_files\' ext='cmd' aline1='splina' aline2='_SPLINA.bat' For i=startyear, endyear Do Begin ;put all files into a list CD, path+strtrim(string(i),2) List = FILE_SEARCH('*.'+ext) nRecs=n_elements(list) If (n_elements(List) eq 0) then print, kdjfkdjf sarray=strarr(nRecs) For j=0, nRecs-1 Do Begin filename=strsplit(list[j],'.', /Extract) sarray[j]=aline1+' < '+list[j]+' > '+filename[0]+'.log' Endfor


;output the new modified cmd file using input basename OPENW, 1, outpath+strtrim(string(i),2)+'\'$

+strtrim(string(i),2)+aline2 PRINTF, 1, sarray CLOSE, 1 print, outpath+strtrim(string(i),2)+'\'+strtrim(string(i),2)+aline2 Endfor End

10.7 IDL code to calculate Rs_in at the date-time of image acquisition pro Rs_in_BatchRun_WRON ;based on Tom's createsplinbatruns.pro ;create batch file to run rs_in.bat (bringin, combine and sol_rad) for all .cov file times. ; startyear=2006 endyear=2006 ;rootPath='W:\Data\RS\AVHRR_Daily\' rootPath='W:\Data\RS\AVHRR_Daily\' outPath = 'W:\At_Trans\' ext='.hdr' TimeZone = 10 ;monthStr = ['01\','02\','03\','04\','05\','06\','07\','08\','09\','10\','11\','12\'] For i=startyear, endyear Do Begin avhrrPath = rootPath + strtrim(string(i),2);+ '\' + monthStr[m] fileList = file_search(avhrrPath, '*.hdr', count=FileNumber); If (FileNumber eq 0) then print, 'no avhrr image files' else begin cmdArray=strarr(FileNumber) For j=0, FileNumber-1 Do Begin filename=strsplit(fileList[j],'.', /Extract) theFile = file_basename(fileList[j]) dateTime = strmid(theFile, [5, 9, 11, 14], [4,2,2,4]) time_local = fix(dateTime[3])+TimeZone * 100 if time_local lt 1000 then time_local = strmid ('000' +$ strtrim(string(time_local),2), 3,4, /REVERSE_OFFSET ) else$ time_local = strtrim(string(time_local),2) cmdArray[j]='call rs_in '+ time_local+' '+dateTime[2]+'$ '+dateTime[1]+' '+dateTime[0] Endfor ;output the batch file OPENW, 1, outPath+strtrim(string(i),2)+ '\'$ +'call_rs_in_for_cov_'+strtrim(string(i),2)+'_new.bat' PRINTF, 1, cmdArray, format = '(a0)' CLOSE, 1 endelse Endfor End

10.8 IDL code to use .ALL files in RADWAT to create .COV files ;Purpose: Create INT file ;Input: 1 ;Output: Many txt files (one station one file, containing tempral AVHRR dada, see lines below for the ;format) ;Phoenix Overpass Data ;17 2 1 2 1 1.013 ;D 67 19830308 800 283.5 89.5 0.40 1.20 108.0 283.9 88.3 ;D 67 19830308 830 285.6 89.5 0.40 2.00 247.1 285.3 83.7


;Steps: ;1. get all checked AVHRR data first and sort by date - outside this function ;2. read in one checked AVHRR file and write out many (in the INT format shown above) ;3. read the second checked AVHRR file and write out many (append to files created in above step) ;History: 13 March 2007 add function to extract coverage data ;Modified to WRON server a)trunk the seconds in file names (previous version uses rounding) ; b)change file locations ; c)read f imgae from SEA dir drectly Function get_maxDay, year, month m = year mod 4 if (m eq 0 && month eq 2) then begin k = 29 endif else begin case month of '02': k = 28 '04': k = 30 '06': k = 30 '09': k = 30 '11': k = 30 else: k = 31 endcase endelse return, K end Pro WriteStationFile, outputDir, avhrrFileName, staCode ;read in one record - avhrrValue - from Cloud check data, ;use avhrr image name as a key to write out one station file ;avhrrValue array columns timeZone_h = 10 timeZone_m = 0 imgNameStr = strsplit(file_basename (avhrrFileName), '_', /extract) theYear=fix(strmid(imgNameStr[1], 0,4)) theMonth=fix(strmid(imgNameStr[1], 4,2)) theDay=fix(strmid(imgNameStr[1], 6,2)) hh=fix(strmid(imgNameStr[2], 0,2));hour mm=fix(strmid(imgNameStr[2], 2,2));minute hh = hh + timeZone_h ;GMT time to local time mm = mm + timeZone_m if mm GE 60 then begin mm = mm - 60 hh = hh + 1 endif if hh GE 24 then begin maxDay = get_maxDay(theYear, theMonth) hh = hh - 24 theDay = theDay + 1 if theDay GT maxDay then begin theDay = theDay - maxDay theMonth = theMonth + 1 endif if theMonth GT 12 then begin theMonth = theMonth - 12 theYear = theYear + 1 endif


endif DoY = JULDAY(theMonth, theDay,theYear) - JULDAY(1,0,theYear);day of the year ;CONVERT DATE/TIME TO STRING theYear = strtrim(string(theYear),2) if theMonth lt 10 then theMonth = '0' + strtrim(string(theMonth),2) else $

theMonth = strtrim(string(theMonth),2) if theDay lt 10 then theDay = '0' + strtrim(string(theDay),2) else$

theDay = strtrim(string(theDay),2) if hh lt 10 then hh = '0' + strtrim(string(hh),2) else hh = strtrim(string(hh),2) if mm lt 10 then mm = '0' + strtrim(string(mm),2) else mm = strtrim(string(mm),2) outFile = outputDir + strtrim(string(staCode, format = '(i)'),2)+'.INT' if (file_test(outFile) eq 1) then begin ; if the file already exists, append the record openw, /append, fout, outFile, /get_lun endif else begin openw, fout, outFile, /get_lun; otherwise, create one. printf, fout, string(staCode, format = '(i)') +' Overpass Data' ;format= '()';need to modify printf, fout, '17 2 1 2 0 0.411554';format= '()';need to modify endelse ;calculate data needed in INT file SurfT = '-1' Cover = '-1' Height = '-1';use -1 for RadCalc if data missed Wind = '-1' SolarIr = '-1' AirT = '-1' Humidity = '-1' ;need to modify AVHRR value printf, fout, 'D ', DoY, theYear+theMonth+theDay,hh+mm,SurfT, Cover,$ Height,Wind,SolarIr,AirT,Humidity, format = '(a1,1x,i3,1x,a8,1x,a4,a7,1x,a6,5a3)' close, fout free_lun, fout end Pro ManyPts2ManyDates, rootDir_INT, avhrrName, year ;this procedure reads in station info file and find out the valid ones for a specific year theYearCol = year - 1980 ;AVHRR Year = 2004 threshold_staCheck = 90; threshold of valid stations checkedMetSta = 'W:\spreadsheets_and_texts\metCheckReportYearly.txt';checked stations nCols = 0 ; the number of columns in the input file nline1 = 0 ; number of lines in checked met station file aline = ' ' openr, f_MetSta1, checkedMetSta, /get_lun readf, f_MetSta1, aline nCols = n_elements(strsplit(aline, STRING(9B) )) while not eof(f_MetSta1) do begin nline1++ readf, f_MetSta1, aline endwhile point_lun, f_MetSta1, 0; put pointer back arrMetSta1 = lonarr(nCols, nline1);one more column for the station code ;readf, f_MetSta1, arrMetSta1; this is data


envi_read_cols, checkedMetSta, arrMetSta1 tmp = where (arrMetSta1[theYearCol+1,*] GE threshold_staCheck, nValid);if the valus is 1, then the station (line) is OK for the year(column), arrValidSta =fltarr(nValid) arrValidSta = (arrMetSta1[0,*])[tmp] ;first col is station code close, f_MetSta1 free_lun, f_MetSta1 nameStr = strsplit(file_basename (avhrrName), '_', /extract);ah16 20010102 052360 37 Checked.txt for i = 0, n_elements(arrValidSta)-1 do begin WriteStationFile, rootDir_INT, avhrrName, arrValidSta[i] endfor end Pro Modify_INT, inFile ;this procedure modifies the INT file created above (pro WriteStationFile) ;it reads in an INT file, counts the number of records, then wtites out a modified INT with correct line 2. aline = '' nLines = 0 openr, tmpFile, inFile, /get_lun while not eof (tmpFile) do begin readf, tmpFile, aline nLines++ ;number of lines of the INT file endwhile POINT_LUN, tmpFile, 0 tmpArray = strarr(1,nLines) readf, tmpFile, tmpArray ;read in as a one dimensional string array close, tmpFile free_lun, tmpFile ;change first few charactors of line2 pos = strpos(tmpArray[0,1], ' ');end of first number in the line part2= strmid(tmpArray[0,1],pos);get the rest of the line newNum = strtrim(string(nlines - 2),2);number of station records = total lines - two header lines tmpArray[0,1] = newNum + part2 ; modify line 2 openw, tmpFile, inFile, /get_lun printf, tmpFile, tmpArray; write out the new INT file close, tmpFile free_lun, tmpFile end ;Starting point for the function. Get all outputs from AVHRR_Cloud_Check.pro ;rootDir = 'G:\AWAP\RADCALC_test_for_LLT_only\' for year = 2006,2006 do begin ; rootDir_INT = 'W:\DATA\RADCALC\INT\2001\' ;rootDir = 'W:\DATA\RS\AVHRR_Daily\2001\' rootDir_INT = 'W:\DATA\RADCALC\INT\'+strtrim(string(year), 2)+'\' rootDir = 'W:\Data\RS\AVHRR_Daily\'+strtrim(string(year), 2)+'\' files=file_search(rootDir, '*.hdr', count=FileNumber) if FileNumber GT 0 then begin for i=0, FileNumber-1 do begin ManyPts2ManyDates, rootDir_INT, files[i], year


endfor endif ;if the output INT files need to be modified (say for number of Records), do it here. files2=file_search(rootDir_INT, '*.INT', count=INTFileNumber) if FileNumber GT 0 then begin for i=0, INTFileNumber-1 do begin Modify_INT, files2[i] endfor endif endfor end ;This pro create parameter files for radwat.exe, with one parameter file per year ;Started: 2007.03.09 ;By: Li Lingtao ;The difference between this code and Create_Parameter_For_Radwat.pro is: ;this one needs .int and .psf to generate .all file Pro Create_Parameter_For_Radwat_All rootDir = 'W:\DATA\RADCALC\' for year = 2006, 2006 do begin outFileDir = rootDir + 'all\' + strtrim(string(year),2) ;check if the destination directory exists, create one if not outDirOK = file_search(outFileDir,COUNT=N_FOUND) IF N_FOUND EQ 0 THEN SPAWN, 'mkdir ' + outFileDir outFile = outFileDir + '\'+ strtrim(string(year),2)+'_radwat.par' openw, fOut, outFile, /get_lun; psfFiles=file_search(rootDir+'psf\'+strtrim(string(year),2)+'\', '*.psf', count=psfFileNum) psfFileNameOnly = strarr(psfFileNum) for i=0, psfFileNum-1 do begin psfFileNameOnly[i]=(strsplit(file_basename(psfFiles[i]),'.', /extract))[0] endfor intFiles=file_search(rootDir+'int\'+strtrim(string(year),2)+'\', '*.int', count=intFileNum) intFileNameOnly = strarr(intFileNum) for i=0, intFileNum-1 do begin intFileNameOnly[i]=(strsplit(file_basename(intFiles[i]),'.', /extract))[0] endfor for i=0, intFileNum-1 do begin psfFile = where (psfFileNameOnly eq intFileNameOnly[i], count) if count GT 0 then begin ;check if the output .all exists, delete it if yes, otherwise this par file won't work outFileName = rootDir + 'all\'+strtrim(string(year),2)+'\'+ $ intFileNameOnly[i]+'.all' outFileOK = file_search(outFileName,COUNT=N_FOUND) IF N_FOUND EQ 1 THEN FILE_DELETE, outFileName;SPAWN, 'del ' +$

outFileName printf, fOut,"s" printf, fOut, psfFiles[psfFile] printf, fOut, "r" printf, fOut, intFiles[i] printf, fOut, outFileName endif


endfor printf, fOut,"q" close, fOut free_lun, fOut endfor end pro SortAllByImageAcq ;TVN/17-Apr-2007 ;Takes in output from RadCalc and sorts all stations into separate ;files for each image acquisition. ; ;Outputs sorted data by image acquisition date for input to spline (spline in, ;called "cov") ; ;initialise variables startyear=2006 endyear=2006 path='W:\Data\RADCALC\ALL\' outpath='W:\Data\RADCALC\Spatial_Int_Overpass\' staFile='W:\spreadsheets_and_texts\stations_nostrings_DEM.txt' ND = -9999. ;read in station file to get lat longs ;read in station file ENVI_READ_COLS, staFile, sta_array BIGstalist=sta_array(0,*) BIGlatlist=sta_array(3,*) BIGlonlist=sta_array(4,*) BIGelevlist=sta_array(5,*) BIGDEMlist=sta_array(10,*) For i=startyear, endyear Do Begin ;get sta list from BOM Wind directory CD, path+strtrim(string(i),2) radList = FILE_SEARCH('*.all') If (n_elements(radlist) eq 0) then print, kdjfkdjf ;Read in each rad file (i.e., one station at a time) and concat them together For j=0, n_elements(radlist)-1 Do Begin envi_read_cols, path+strtrim(string(i),2)+'\'+radlist[j], matrix ;set up necesarry lists stalist=lonarr(n_elements(matrix[0,*])) lonList=fltarr(n_elements(matrix[0,*])) latList=fltarr(n_elements(matrix[0,*])) demList=fltarr(n_elements(matrix[0,*])) elevList=fltarr(n_elements(matrix[0,*])) ;determine the station, elev, lat, and lon for this file and populate the relevant lists ;station list a=strsplit(radlist[j],'.',/EXTRACT) a=long(a[0]) stalist[*]=a ;lon and lat lists aindex=where(BIGstaList eq a) lonlist[*]=BIGlonlist[aindex] latlist[*]=BIGlatList[aindex] demlist[*]=BIGDEMList[aindex]


elevlist[*]=BIGelevList[aindex] ;check if DEM eq -999 (null), if so, replace with station elev from dictionary nullDEM=where(demlist eq -999, nullDEMcount) if (nullDEMcount ne 0) then Begin demlist[nullDEM]=elevlist[nullDEM] nullELEV=where(elevlist[nullDEM] eq -1, nullelevcount) ;if elev from dictionary is null (-1) then stop program - this needs ;to be fixed. Bad idea to make station elev nulls eq -1 as this is a valid ;elevation - need to get original elevs again, then we can do something ;different here. if (nullelevcount ne 0) then print, dkfjdkj endif ;insert station, lon, and lat and DEM lists into the BIG rad array matrix=[[matrix],transpose(stalist), transpose(lonlist), transpose(latlist), transpose(demlist)] ;concat each separate array together into a big array if (j eq 0) then bigArray=matrix else bigArray=[[bigArray],[matrix]] Endfor ;Add Ea and VPD columns to the big array based on LLT's code ;which uses Tair and Rh ; E0 =0.6108 * exp (17.27 * bigarray[8,*] / (bigarray[8,*] + 237.3)) Ea = bigarray[9,*]/100.0 * E0 VPD = E0 - Ea nd_Tair = where (bigarray[8,*] eq ND, count_Ta) nd_Rh = where (bigarray[9,*] eq ND, count_Rh) if count_Ta NE 0 then begin Ea [nd_Tair] = ND VPD [nd_Tair] = ND endif if count_Rh NE 0 then begin Ea [nd_Rh] = ND VPD [nd_Rh] = ND endif ;insert Ea and VPD into the BIG rad array bigarray=[[bigarray],Ea,VPD] ;Sort the big "ALL" array based on image acquisition ; get list of image acquisitions from the big array ;make a unique identifier for each overpass by using date (*10000) plus time (hhmm) imgID=(bigArray[2,*]*10000.)+bigArray[5,*] imgAcqList=imgID[UNIQ(imgID, SORT(imgID))] ;Loop through each unique image acquistion and get associated data For k=0, n_elements(imgAcqList)-1 Do Begin ;split up the uniq image acq ids into date and time str=STRTRIM(STRING(ulong64(imgAcqlist[k])),2) imgDate=STRMID(str,0,8) imgTime=STRMID(str,8,4) ;define the items in the big array that have this date and time index=where((bigArray[2,*] eq long(imgDate)) and (bigArray[5,*] eq fix(imgTime)), count)


;write out specified data for this image acquisition date outarray=fltarr(8,count) outarray[0,*]=bigarray[33,index]; station number outarray[1,*]=bigarray[34,index]; longitude outarray[2,*]=bigarray[35,index]; latitude outarray[3,*]=bigarray[36,index]; DEM outarray[4,*]=bigarray[8,index]; Ta (Deg C) outarray[5,*]=bigarray[9,index]; Rh (%) outarray[6,*]=bigarray[37,index]; Ea (kPa) outarray[7,*]=bigarray[38,index]; VPD (kPa) OPENW, 1, outpath+strtrim(string(i),2)+'\'+strtrim(string(long(imgDate)),2)+'_'+imgTime+'.cov' PRINTF, 1, outArray, format="(I10, 7F10.4)" CLOSE, 1 Endfor Endfor End

10.9 IDL code to interpolate Ta and ea at the date-time of image acquisition

pro Create_SPLINA_cmd_Files_Ta_Ea_VPD ;TVN/LLT Apr-2007 ;create cmd files for each surface to be splined for overpass variables (Ta, Ea and VPD) ;assumes a yearly directory structure ;requires a template *.cmd file as input at the base directory specified below ;initialise variables startyear=2006 endyear=2006 path='W:\DATA\RADCALC\Spatial_Int_Overpass\' outpath='W:\DATA\RADCALC\Spatial_Int_Overpass\SPLIN_cmd_files\' ext='cov'; magicRecord=[20,21,22,23,24] template=outpath+'template_Ta_Ea_VPD_splna.cmd' ;make sure template exists Result = FILE_TEST(template) if (result ne 1) then print, dkfjkdj For i=startyear, endyear Do Begin ;put all files into a list CD, path+strtrim(string(i),2) List = FILE_SEARCH('*.'+ext, count=n) If (n eq 0) then print, kdjfkdjf For j=0, n_elements(List)-1 Do Begin filename=strsplit(list[j],'.', /Extract) ;read in the template file and modify the "magic" record where the filename is kept counter=0 aline='' sarray=strarr(1) OPENR, unit, template, /GET_LUN while not eof (unit) do begin readf, unit, aline if counter eq 15 then begin strput, aline, strtrim(string(i),2)+'\'+filename[0],37


endif else begin tmp = where (magicRecord eq counter+1, count) if count gt 0 then strput, aline, strtrim(string(i),2)+'\'+filename[0],53 endelse if (counter eq 0) then begin strput, aline, filename[0],0 sarray[0]=aline endif else sarray=[[sarray],[aline]] counter++ endwhile free_lun, unit ;output the new modified cmd file using input basename OPENW, 1, outpath+strtrim(string(i),2)+'\'+filename[0]+'_Ta_Ea_VPD_SPLINA.cmd' PRINTF, 1, sarray CLOSE, 1 print, outpath+strtrim(string(i),2)+'\'+filename[0]+'_Ta_Ea_VPD_SPLINA.cmd' Endfor Endfor End pro Create_SPLINA_Batch_Runs_Ta_Ea_VPD ;TVN/05-Apr-2007 ;create batch file to run all splina commands for overpass variables (Ta, Ea and VPD) ;assumes a yearly directory structure ;initialise variables startyear=2006 endyear=2006 path='W:\DATA\RADCALC\Spatial_Int_Overpass\SPLIN_cmd_files\' ext='cmd' aline1='splina' aline2='_SPLINA.bat' For i=startyear, endyear Do Begin ;put all files into a list CD, path+strtrim(string(i),2) List = FILE_SEARCH('*.'+ext) nRecs=n_elements(list) If (n_elements(List) eq 0) then print, kdjfkdjf sarray=strarr(nRecs) For j=0, nRecs-1 Do Begin filename=strsplit(list[j],'.', /Extract) sarray[j]=aline1+' < '+list[j]+' > '+filename[0]+'.log' Endfor ;output the batch file OPENW, 1, path+strtrim(string(i),2)+'\'+strtrim(string(i),2)+aline2 PRINTF, 1, sarray CLOSE, 1 print, path+strtrim(string(i),2)+'\'+strtrim(string(i),2)+aline2 Endfor End pro Create_Lapgrd_cmd_Files_Ta_Ea_VPD ;TVN/LLT Apr-2007


;LLT modified 30 April 2007 ;create cmd files for each surface to be lapgrd-ed for overpass variables (Ta, Ea and VPD) ;assumes a yearly directory structure ;requires a template *.cmd file as input at the base directory specified below ;initialise variables startyear=2006 endyear=2006 path = 'W:\DATA\RADCALC\Spatial_Int_Overpass\' outpath='W:\DATA\RADCALC\Spatial_Int_Overpass\LAPGRD_cmd_files\' ext='cov' template=outpath+'template_Ta_Ea_VPD_lapgrd.cmd' ;make sure template exists Result = FILE_TEST(template) if (result ne 1) then print, dkfjkdj For i=startyear, endyear Do Begin ;put all files into a list CD, path+strtrim(string(i),2) List = FILE_SEARCH('*.'+ext, count = n) If (n eq 0) then print, kdjfkdjf For j=0, n_elements(List)-1 Do Begin filename=strsplit(list[j],'.', /Extract) mmdd=strmid(filename[0],4,4) ;read in the template file and modify the "magic" record where the filename is kept counter=0 aline='' sarray=strarr(1) OPENR, unit, template, /GET_LUN while not eof (unit) do begin readf, unit, aline if counter eq 14 then begin strput, aline, strtrim(string(i),2)+'\Ta\'+filename[0],28 endif else if counter eq 15 then begin strput, aline, strtrim(string(i),2)+'\Ea\'+filename[0],28 endif else if counter eq 16 then begin strput, aline, strtrim(string(i),2)+'\VPD\'+filename[0],28 endif else begin if counter eq 0 then strput, aline, strtrim(string(i),2)+'\'+filename[0],53 endelse if (counter eq 0) then sarray[0]=aline else sarray=[[sarray],[aline]] counter++ endwhile free_lun, unit ;output the new modified cmd file using input basename OPENW, 1, outpath+strtrim(string(i),2)+'\'+filename[0]+'_Ta_Ea_VPD_LAPGRD.cmd' PRINTF, 1, sarray CLOSE, 1 print, outpath+strtrim(string(i),2)+'\'+filename[0]+'_Ta_Ea_VPD_LAPGRD.cmd' Endfor Endfor End pro Create_LAPGRD_Batch_Runs_Ta_Ea_VPD ;TVN/05-Apr-2007


;Modified LLT 30 April 2007 ;create batch file to run all LAPGRD commands for Overpass variables (Ta, Ea and VPD) ;assumes a yearly directory structure ;initialise variables startyear=2006 endyear=2006 path='W:\DATA\RADCALC\Spatial_Int_Overpass\LAPGRD_cmd_files\' ext='cmd' aline1='LAPGRD' aline2='_LAPGRD.bat' For i=startyear, endyear Do Begin ;put all files into a list CD, path+strtrim(string(i),2) List = FILE_SEARCH('*.'+ext) nRecs=n_elements(list) If (n_elements(List) eq 0) then print, kdjfkdjf sarray=strarr(nRecs) For j=0, nRecs-1 Do Begin filename=strsplit(list[j],'.', /Extract) sarray[j]=aline1+' < '+list[j]+' > '+filename[0]+'.log' Endfor ;output the batch file OPENW, 1,path+strtrim(string(i),2)+'\'+strtrim(string(i),2)+aline2 PRINTF, 1, sarray CLOSE, 1 Endfor End

10.10 IDL code to generate Rn and Ts-Ta at the date-time of image acquisition

pro Calc_Rn_Surface_WRON, yearStr, imageName ;19 April 2007 ;Purpose: Calculate Rn and Tdiff (Tdif = Ts - Ta) ;Input: Rs_in surfaces, and Rs_out, Rl_in, Rl_out on-fly ;Output: Rn surface, and Tdif surface ;Equation: Rn = Rs_in - Rs_out + Rl_in - Rl_out ;27 April 2007 Modified to read subset f images ;4 may 2007, edit for WRON timeZone = 10 dateTime = strmid(imageName, [0, 2, 4], [2,2,4]);mm, dd, time: 12231530.IMG utm_Time = fix(dateTime[2]) - timeZone * 100 if utm_Time lt 1000 then utm_Time = strmid ('000' + strtrim(string(utm_Time),2), 3,4,$

/REVERSE_OFFSET ) else utm_Time = strtrim(string(utm_Time),2) ; put here if utm_Time is minus. as for 2001 there are all positives ;initialise variables ND = -9999. avhrr_Path='W:\Data\RS\AVHRR_Daily\' + yearStr + '\' + dateTime[0] + $ '\';ah16_20010101_035300_23


avhrr_filename_start = yearStr+ dateTime[0]+dateTime[1]+'_'+utm_Time avhrr_file = file_search(avhrr_Path,'*'+avhrr_filename_start+'*', count = n ) if (n GE 1) then avhrr_filename = avhrr_file[0] else begin ;in case time is different to avhrr image time as we rounded it in Rs_in creation rem = utm_Time mod 100 if (rem GT 0) then utm_Time = utm_Time -1 else utm_Time = utm_Time - 41 if utm_Time lt 1000 then utm_Time = strmid ('000' + strtrim(string(utm_Time),2), 3,4,$

/REVERSE_OFFSET ) else utm_Time = strtrim(string(utm_Time),2) avhrr_filename_start = yearStr+ dateTime[0]+dateTime[1]+'_'+utm_Time avhrr_file = file_search('*'+avhrr_Path,avhrr_filename_start+'*', count = n ) avhrr_filename = avhrr_file[0] endelse Ea_Path='W:\Data\RCALC_SURF\Overpass\' + yearStr + '\Ea\' Ea_filename = yearStr+ $ dateTime[0]+dateTime[1]+'_'+dateTime[2]+'_Ea_lapgrd.grd';20010101_1353_Ea_lapgrd.grd Ea_filename = Ea_Path + Ea_filename Ta_Path='W:\Data\RCALC_SURF\Overpass\' + yearStr + '\Ta\' Ta_filename = yearStr+ $ dateTime[0]+dateTime[1]+'_'+dateTime[2]+'_Ta_lapgrd.grd';20010101_1353_Ta_lapgrd.grd Ta_filename = Ta_Path + Ta_filename outPathRn='W:\Data\RCALC_SURF\Overpass\' + yearStr + '\Rnet\' outPathTdiff='W:\Data\RCALC_SURF\Overpass\' + yearStr + '\Tdiff\' samples = 1601 lines = 1701 Rs_in_image = fltarr(samples,lines) Ea_image = fltarr(samples,lines) Ta_image = fltarr(samples,lines) f_image = fltarr(samples,lines) ;get Rs_in from the surfaces created by microBRIAN Rn_in_path = 'W:\Rs_in\' Rs_in_image = read_binary(Rn_in_path+yearStr+'\'+imageName,DATA_START = 512, $

DATA_TYPE = 2,DATA_DIMS = [1601, 1701]); Rs_in_image OPENR, surface_in, Ea_filename, /GET_LUN readu, surface_in, Ea_image free_lun, surface_in Ea_image = Ea_image * 10 ;convert to hPa from kPa OPENR, surface_in, Ta_filename, /GET_LUN readu, surface_in, Ta_image free_lun, surface_in Ta_image = Ta_image + 273.15 ; convert to K from C ;use SEA f image directly ;get the right vegetation coverage image and subset for SEA f_path = 'W:\DATA\RS\corrected_fpar\converted_coverage\SE_Australia\filled_nulls\' month_in = strtrim((fix(yearStr) - 1981) * 12 + fix(dateTime[0]) -1, 2) ;get Randall's month f_filename = yearStr+ dateTime[0]+'_'+month_in+'.flt' f_filename = f_path + f_filename OPENR, surface_in, f_filename, /GET_LUN readu, surface_in, f_image free_lun, surface_in ;calculate Rs_out. avhrr_image = fltarr(samples,lines,8)


OPENR, surface_in, avhrr_filename, /GET_LUN readu, surface_in, avhrr_image free_lun, surface_in ; Note that the Chn 1 and 2 data are provided as a reflectance % from EAK ; To properly invoke the scaling for the albedo, this must be a fraction, hence divided by 100 albedo = ((avhrr_image[*,*,0] + avhrr_image[*,*,1] ) * 0.5)/100 Rs_out_image = albedo * Rs_in_image ;calculate Rl_in from Ea and Ta surfaces created by ANUSPLIN ;Eq come from Tim's note ;Rl_in = EPSILONa * SIGMA * Ta ^ 4 ;EPSILONa = [1-(1+XI)exp{-(1.2+3.0*XI)^0.5} ] ;XI = 46.5 * (ea / Ta) XI =46.5 * (Ea_image / Ta_image);Ea in hPa, Ta in K EPSILONa = (1- (1+XI) * exp(-(1.2+3.0*XI)^0.5)) SIGMA = 5.67 * 10 ^ (-8.0) Rl_in_image = EPSILONa * SIGMA * (Ta_image) ^ 4 ;calculate Rl_out from Ts and vegetation coverage ;Eq come from Tim's note ;Rl_out = EPSILONs * SIGMA * Ts ^ 4 ;EPSILONs = fv*EPSILONv + (1-fv) * EPSILONg EPSILONv = 0.98 EPSILONg = 0.96 ; Note AVHRR Chn 4 is channel 3 in the data EAK processed; when counting from 0 this = 2nd array Ts_image = avhrr_image[*,*,3] + 3*(avhrr_image[*,*,2] - avhrr_image[*,*,3]) + 0.5 ;t5 + 3*(t4-t5) + 0.5 t_dif = Ts_image - Ta_image ; Note that Randall's veg cover image is provided as a %, it needs to be a fraction hence is ; divided by 100 in the following EPSILONs = f_image/100. * EPSILONv + (1 - f_image/100.) * EPSILONg Rl_out_image = EPSILONs * SIGMA * Ts_image ^ 4 Rn_image = Rs_in_image - Rs_out_image + Rl_in_image - Rl_out_image w=where (Ea_image eq ND, count) if (count gt 0) then begin Rn_image(w) = ND t_dif(w) = ND end w=where (f_image eq ND, count) if (count gt 0) then begin Rn_image(w) = ND t_dif(w) = ND end w=where (Rs_in_image eq -32767, count) if (count gt 0) then begin Rn_image(w) = ND t_dif(w) = ND end


w=where (avhrr_image[*,*,1] eq -999, count) if count gt 0 then begin Rn_image(w) = ND t_dif(w) = ND end ;Write out Rn image outRn_filename = yearStr + dateTime[0]+dateTime[1]+'_'+dateTime[2]+'_Rn.flt' OPENW, surface_out, outPathRn + outRn_filename, /get_lun WRITEU, surface_out, Rn_image CLOSE, surface_out free_lun, surface_out t_dif_filename = yearStr + dateTime[0]+dateTime[1]+'_'+dateTime[2]+'_tdif.flt' OPENW, surface_out, outPathTdiff + t_dif_filename, /get_lun WRITEU, surface_out, t_dif CLOSE, surface_out free_lun, surface_out End Rs_in_path = 'W:\Rs_in\' startYear = 2006 EndYear = 2006 for i = startYear, endYear do begin yearString = strtrim(string(i), 2) Rs_in_List = file_search(Rs_in_path+yearString+'\'+'*.IMG', count=FileNumber) If (FileNumber eq 0) then print, 'no Rs_in images found' else begin For j=0, FileNumber-1 Do Begin theFile = file_basename(Rs_in_List[j]) Calc_Rn_Surface_WRON, yearString, theFile Endfor endelse endfor end

10.11 IDL code to run RADWAT to create files containing overpass information

;IDL 6 function ;Purpose: Cloud checking: read point values of met stations from AVHRR surfaces ;input files: AVHRR surfaces + met station location file 'U:\AWAP\spreadsheets_and_texts\stations.txt' ; and 'U:\AWAP\spreadsheets_and_texts\metCheckReportYearly.txt' ;output file: a text file showing the AVHRR channel values and cloud condition of the stations ;Version History: 8 Dec 2006, based on Extract_pt_from_Surface.pro for ACIAR CSHC data Function AVHRR_Cloud_Check, rootDir, Surface_fileName ;read in two met station files: one contains lat/long of all stations(stations.csv), one contains met data ;quality for a specific year(metCheckReportYearly.csv) ;open checked met station file to get valid station codes theYearCol = 21 ;AVHRR Year = 2001 threshold_staCheck = 90; threshold of valid stations threshold_Zenith = 50; threshold of valid stations checkedMetSta = 'U:\AWAP\spreadsheets_and_texts\metCheckReportYearly.txt';checked stations nCols = 0 ; the number of columns in the input file nline1 = 0 ; number of lines in checked met station file nline2 = 0; number of lines in big met station file


aline = ' ' openr, f_MetSta1, checkedMetSta, /get_lun readf, f_MetSta1, aline nCols = n_elements(strsplit(aline, STRING(9B) )) while not eof(f_MetSta1) do begin nline1++ readf, f_MetSta1, aline endwhile point_lun, f_MetSta1, 0; put pointer back arrMetSta1 = lonarr(nCols, nline1);one more column for the station code readf, f_MetSta1, arrMetSta1; this is data ;if the valus is 1, then the station (line) is OK for the year(column), tmp = where (arrMetSta1[theYearCol+1,*] GE threshold_staCheck, nValid) n_OutFileCols = 11 ;staCode,TimeZone, Long, Lat,X, Y, c1, c2, c3, c4, cloudFlag staCode =0 timeZone=1 Longi =2 Lat=3 X1=4 Y1=5 c1=6 c2=7 c3=8 c4=9 cloudFlag=10 arrValidSta =fltarr(n_OutFileCols,nValid) arrValidSta[staCode,*] = (arrMetSta1[0,*])[tmp] ;first col is station code close, f_MetSta1 free_lun, f_MetSta1 ;open the met station file with lat/long to get lat/longs of the valid stations xyMetSta = 'U:\AWAP\spreadsheets_and_texts\stations.txt' openr, f_MetSta2, xyMetSta, /get_lun readf, f_MetSta2, aline; this is header line nCols = n_elements(strsplit(aline, STRING(9B) )) while not eof(f_MetSta2) do begin readf, f_MetSta2, aline nline2++ endwhile point_lun, f_MetSta2, 0; put pointer back arrMetSta2 = strarr(nCols, nline2); to put all info of lat/long station file readf, f_MetSta2, aline; this is header line for i = 0, nline2-1 do begin;this is data readf, f_MetSta2, aline arrMetSta2[*,i] = strsplit(aline, STRING(9B), /extract) endfor ;thisState = 'ACT' tZone = 10 for i = 0, nValid-1 do begin thisSta = where (float(strtrim((arrMetSta2[0,*]),2)) eq arrValidSta[staCode,i]) thisState = strtrim(arrMetSta2[8,thisSta],2);state


case strupcase(thisState) of 'NT': tZone = 9.5 'SA': tZone = 9.5 'WA': tZone = 8 else: tZone = 10 endcase arrValidSta[timeZone,i] = tZone arrValidSta[Longi,i] = float(strtrim(arrMetSta2[6,thisSta],2));longi arrValidSta[Lat,i] = float(strtrim(arrMetSta2[5,thisSta],2));lat endfor close, f_MetSta2 free_lun, f_MetSta2 ;up to now, we have an array contains valid met stations and their lat/longs ;get image info from image file name. this may be used late. fname_split = strsplit(Surface_fileName, '_', /extract);ah16_20010129_054602_16 imgDate = fname_split[1] imgTime = fname_split[2] imgCloudflag = fname_split[3] ;get info from ENVI header headerFile = rootDir+Surface_fileName+'.hdr' OPENR, fHeader, headerFile, /GET_LUN paraArray = fltarr(6) while not eof (fHeader) do begin Readf, fHeader, aline for i=0, 6 do begin if (STRMID(aline, 0, 5) eq 'sampl') then begin tmp = strsplit(aline, /extract, ' ') paraArray[0] = float(strtrim(tmp[2],2)) endif else if(STRMID(aline, 0, 5) eq 'lines') then begin tmp = strsplit(aline, /extract, ' ') paraArray[1] = float(strtrim(tmp[2],2)) endif else if(STRMID(aline, 0, 5) eq 'bands') then begin tmp = strsplit(aline, /extract, ' ') paraArray[2] = float(strtrim(tmp[2],2)) endif else if(STRMID(aline, 0, 5) eq 'map i') then begin tmp = strsplit(aline, /extract, ' ') paraArray[3] = float(strtrim(tmp[7],2)) ;longi at starting point paraArray[4] = float(strtrim(tmp[8],2)) ;lat at starting point paraArray[5] = float(strtrim(tmp[9],2)) ;pixel size endif endfor endwhile close, fHeader free_lun, fHeader ;header files from EAK untrustful, so just use fixed numbers samples = 1601 lines = 1701 bands = 8 topLeftLong = 137.99500 topLeftLat = -22.99500 pixelsize = 0.01 bottomRightLong = topLeftLong + pixelsize * samples


bottomRightLat = topLeftLat - pixelsize * lines ;The AVHRR image has 8 channels according to EAK, their names are ;normalisedReflectance_avhrr_1 ;normalisedReflectance_avhrr_2 ;brightnessTemperature_avhrr_4 ;brightnessTemperature_avhrr_5 ;satelliteAzimuthGrid ;satelliteZenithGrid ;sunAzimuthGrid ;sunZenithGrid ;read in surface file in_Surface = fltarr(samples,lines,bands) OPENR, f_Surface, rootDir+Surface_fileName, /GET_LUN readu, f_Surface, in_Surface for i = 141 ,nValid -1 do begin dn_avhrr_1 = -999 x=float(arrValidSta[Longi,i]) ;longi y=float(arrValidSta[Lat,i]) ;lat x_img = floor((x-topLeftLong)/pixelsize)+1;check this equation with Tom y_img = floor((topLeftLat-y)/pixelsize)+1;check this equation with Tom

if ( x GE (topLeftLong) && x LE bottomRightLong && (y LE topLeftLat) && $ (y GE bottomRightLat)) then begin

dn_avhrr_1 = in_Surface[x_img-1,y_img-1, 0]; channel 1. dn_avhrr_2 = in_Surface[x_img-1,y_img-1, 1]; channel 2. dn_avhrr_4 = in_Surface[x_img-1,y_img-1, 2]; channel 3. dn_avhrr_5 = in_Surface[x_img-1,y_img-1, 3]; channel 4. dn_satelliteZenith = in_Surface[x_img-1,y_img-1, 5]; channel 6. arrValidSta[X1,i] = x_img ;x coor on image, not lat/long arrValidSta[Y1,i] = y_img ;y coor on image, not lat/long arrValidSta[c1,i] = dn_avhrr_1 arrValidSta[c2,i] = dn_avhrr_2 arrValidSta[c3,i] = dn_avhrr_4 arrValidSta[c4,i] = dn_avhrr_5 endif ;if dn_avhrr_1 EQ -999 then arrValidSta[cloudFlag,i] = 0 else arrValidSta[cloudFlag,i]= 1;cloud is 0 ;if take into account Zenith angle, use following code instead th eline above if dn_avhrr_1 EQ -999. then arrValidSta[cloudFlag,i] = 0 else begin

if (dn_satelliteZenith GT threshold_Zenith) || (dn_satelliteZenith eq -999.) then $ arrValidSta[cloudFlag,i]= 1 else begin

arrValidSta[cloudFlag,i]= 2 endelse endelse endfor ;the array of valid met is now ready, this is for one pixel. close, f_Surface free_lun, f_Surface OPENW, f_out, rootDir+Surface_fileName+'_Checked.txt', /GET_LUN printf, f_out, 'StaCode','Zone','Long','Lat','X_IMG','Y_IMG','avhrr_1','avhrr_2',$

'avhrr_4','avhrr_5','Cloud_Zenith', format='(a7,a5,2a12,2a6,4a12,a14)' printf, f_out, arrValidSta, format = '(i7,f5.1,2f12.6, 2i6,4f12.6,i10)' close, f_out free_lun, f_out cfc = where (arrValidSta[cloudFlag,*] GE 1, count) ;cloud free cells aCloudRatio = fltarr(5)


aCloudRatio[0] = nValid ; number of valid station aCloudRatio[1] = count ; number of cloud free station aCloudRatio[2] = count*100.00/nValid ; percentage of cloud free station ;cloud free cells with Zenith angle smaller than threshold zenith_OK = where (arrValidSta[cloudFlag,*] eq 2, count2) aCloudRatio[3] = count2 ; number of cloud free station aCloudRatio[4] = count2*100.00/nValid ; percentage of cloud free station return, aCloudRatio; this ratio is for the whole AVHRR image end monthStr = ['01\','02\','03\','04\','05\','06\','07\','08\','09\','10\','11\','12\'] for j = 0, 0 do begin rootDir = 'U:\AWAP\Data\RS\AVHRR_Daily\2001\' rootDir = rootDir + monthStr[j] files = file_search(rootDir, '*.hdr', count=FileNumber); if (FileNumber GT 0) then begin outRationArray = strarr(6, FileNumber); surppose each imag has a hdr file for i=0, FileNumber-1 do begin ;fName=file_basename(files[i]) fName=strsplit(files[i], /extract, '.') fName_base = file_basename(fName) ;thisIMGRation = AVHRR_Cloud_Check(rootDir, fName_base[0]) thisIMGRation = AVHRR_Cloud_Check(rootDir, 'ah16_20010102_052334_37') outRationArray[0, i] = fName_base[0] outRationArray[1, i] = string(fix(thisIMGRation[0])) outRationArray[2, i] = string(fix(thisIMGRation[1])) outRationArray[3, i] = string(thisIMGRation[2], format = '(f6.2)') outRationArray[4, i] = string(fix(thisIMGRation[3])) outRationArray[5, i] = string(thisIMGRation[4], format = '(f6.2)') endfor

OPENW, f_out, rootDir +'AVHRR_img_summary.txt', /GET_LUN printf, f_out, 'Image name', 'Valid_Sta', 'CloudFree_Sta', 'Cloudfree(%)','Zen<=50', $

'Zen_Cloud_OK(%)', format='(a23, a10, a14, a13, a17, a16)' printf, f_out, outRationArray, format='(a23, a10, a14, a13, a17, a16)' close, f_out free_lun, f_out

endif endfor end ;Purpose: Create OVP file ;Input: outputs from AVHRR_Cloud_Check.pro ;(many txt files each contains many points' AVHRR image data, see lines below for the format). ;StaCode Long Lat X_IMG Y_IMG avhrr_1 avhrr_2 avhrr_4 avhrr_5 1=Cloudfree ; 15594 134.686493 -23.463499 0 0 0.000000 0.000000 0.000000 0.000000 0 ; 17031 138.063705 -29.648100 7 665 29.307428 28.258816 314.406067 311.286957 1 ; ;Output: Many txt files (one station one file, containing tempral AVHRR dada, ;see lines below for the format) ;Phoenix Overpass Data ;17 2 1 2 1 1.013 ;D 67 19830308 800 283.5 89.5 0.40 1.20 108.0 283.9 88.3 ;D 67 19830308 830 285.6 89.5 0.40 2.00 247.1 285.3 83.7


;Steps: ;1. get all checked AVHRR data first and sort by date - outside this function ;2. read in one checked AVHRR file and write out many (in the OVP format shown above) ;3. read the second checked AVHRR file and write out many (append to files created in above step) ; ;History: 13 March 2007 add function to extract coverage data ;Modified to WRON server a)trunk the seconds in file names (previous version uses rounding) ; b)change file locations ; c)read f imgae from SEA dir drectly Function Get_Coverage_Value,year,month,x_array, y_array ;from Randall's month to year_month: year = 1981 + a/12, month = a mod 12 + 1 ;from year_month to Randall's month year_int=fix(year) month_int=fix(month) out_cvrg_array = fltarr(n_elements(x_array)) out_cvrg_array[*] = -9999. samples = 1601 lines = 1701 f_image = fltarr(samples,lines) f_path = 'W:\DATA\RS\corrected_fpar\converted_coverage\SE_Australia\filled_nulls\' month_in = strtrim(string((year_int - 1981) * 12 + month_int -1), 2) ;Randall's month f_filename = year + month + '_'+month_in+'.flt' f_filename = f_path + f_filename OPENR, surface_in, f_filename, /GET_LUN readu, surface_in, f_image free_lun, surface_in if (file_test(f_filename) eq 0) then out_cvrg_array[*] = -9999. else begin ;open coverage file for i = 0 ,n_elements(x_array) -1 do begin if (x_array[i] GT 0 and y_array[i] GT 0) then $

out_cvrg_array[i] = f_image[x_array[i]-1,y_array[i]-1] else out_cvrg_array[i] = -9999. endfor endelse return, out_cvrg_array end Function get_maxDay, year, month m = year mod 4 if (m eq 0 && month eq 2) then begin k = 29 endif else begin case month of '02': k = 28 '04': k = 30 '06': k = 30 '09': k = 30 '11': k = 30 else: k = 31 endcase endelse return, K end Pro WriteStationFile, outputDir, avhrrFileName, avhrrValue, coverage ;read in one record - avhrrValue - from Cloud check data, ;use avhrr image name as a key to write out one station file


;avhrrValue array columns staCode = 0 tZone = 1 Longi = 2 Lat = 3 X_IMG = 4 Y_IMG = 5 avhrr_1 = 6 avhrr_2 = 7 avhrr_4 = 8 avhrr_5 = 9 Cloudfree = 10 timeZone_h = 10 timeZone_m = 0 if (avhrrValue[Cloudfree] GT 0) then begin;the station is not covered by cloud ;ah16 20010102 052360 37 Checked.txt imgNameStr = strsplit(file_basename (avhrrFileName), '_', /extract) theYear=fix(strmid(imgNameStr[1], 0,4)) theMonth=fix(strmid(imgNameStr[1], 4,2)) theDay=fix(strmid(imgNameStr[1], 6,2)) hh=fix(strmid(imgNameStr[2], 0,2));hour mm=fix(strmid(imgNameStr[2], 2,2));minute hh = hh + timeZone_h ;GMT time to local time mm = mm + timeZone_m if mm GE 60 then begin mm = mm - 60 hh = hh + 1 endif if hh GE 24 then begin maxDay = get_maxDay(theYear, theMonth) hh = hh - 24 theDay = theDay + 1 if theDay GT maxDay then begin theDay = theDay - maxDay theMonth = theMonth + 1 endif if theMonth GT 12 then begin theMonth = theMonth - 12 theYear = theYear + 1 endif endif DoY = JULDAY(theMonth, theDay,theYear) - JULDAY(1,0,theYear);day of the year ;CONVERT DATE/TIME TO STRING theYear = strtrim(string(theYear),2) if theMonth lt 10 then theMonth = '0' + strtrim(string(theMonth),2) else $

theMonth = strtrim(string(theMonth),2) if theDay lt 10 then theDay = '0' + strtrim(string(theDay),2) else $

theDay = strtrim(string(theDay),2) if hh lt 10 then hh = '0' + strtrim(string(hh),2) else hh = strtrim(string(hh),2) if mm lt 10 then mm = '0' + strtrim(string(mm),2) else mm = strtrim(string(mm),2)


outFile = outputDir + strtrim(string(avhrrValue[staCode], format = '(i)'),2)+'.ovp' if (file_test(outFile) eq 1) then begin ; if the file already exists, append the record openw, /append, fout, outFile, /get_lun endif else begin openw, fout, outFile, /get_lun; otherwise, create one.

;format= '()';need to modify printf, fout, string(avhrrValue[staCode], format = '(i)') +' Overpass Data' printf, fout, '17 2 1 2 0 0.411554';format= '()';need to modify endelse ;calculate data needed in OVP file SurfT = avhrrValue[avhrr_5] + 3*(avhrrValue[avhrr_4] - avhrrValue[avhrr_5]) + 0.5; if coverage eq -999. then Cover = strtrim(string(-1),2) else $

Cover = string(coverage, format = '(f6.2)'); if coverage eq -9999. then Cover = strtrim(string(-1),2) else begin if coverage eq 0. then coverage = 0.01 Cover = string(coverage, format = '(f6.2)') endelse Height = -1;use -1 for RadCalc if data missed Wind = -1 SolarIr = -1 AirT = -1 Humidity = -1 printf, fout, 'D ', DoY, theYear+theMonth+theDay,hh+mm,SurfT, Cover,$ Height,Wind,SolarIr,AirT,Humidity, $

format = '(a1,1x,i3,1x,a8,1x,a4,f7.2,1x,a6,5i3)';need to modify AVHRR value close, fout

free_lun, fout endif end Pro ManyPts2ManyDates, rootDir_ovp, avhrrName ;this procedure reads in a text file (one image, many stations) ;then send one record - a station - to Pro WriteStationFile to write out if file_test(avhrrName) eq 0 then begin print, 'file ' + avhrrName+ ' not exist, task cancelled' endif header='' nLines=0 openr, f_In_IMG, avhrrName, /get_lun readf, f_In_IMG, header POINT_LUN, -f_In_IMG, pos nCols = n_elements(strsplit(header, ' '));Need to check if nCols is stll correct later while not eof (f_In_IMG) do begin readf, f_In_IMG, header nLines++ ;number of lines of the OVP file endwhile POINT_LUN, f_In_IMG, pos aIMG = fltarr(nCols,nLines) readf, f_In_IMG, aIMG close, f_In_IMG free_lun, f_In_IMG


nameStr = strsplit(file_basename (avhrrName), '_', /extract);ah16 20010102 052360 37 Checked.txt theYear=strmid(nameStr[1], 0,4) theMonth=strmid(nameStr[1], 4,2) col_x = 4;envi col number of the image col_y = 5;envi line number of the image cvrg_arr = Get_Coverage_Value(theYear,theMonth,aIMG[col_x,*], aIMG[col_y,*] ) ;get % coverage here can save running time than station by station for i = 0, nLines-1 do begin

;line by line (i.e., station by station) creating output by calling WriteStationFile WriteStationFile, rootDir_ovp, avhrrName, aIMG[*,i], cvrg_arr[i] endfor end Pro Modify_OVP, inFile ;this procedure modifies the OVP file created above (pro WriteStationFile) ;it reads in an OVP file, counts the number of records, then wtites out a modified ;OVP with correct line 2. aline = '' nLines = 0 openr, tmpFile, inFile, /get_lun while not eof (tmpFile) do begin readf, tmpFile, aline nLines++ ;number of lines of the OVP file endwhile POINT_LUN, tmpFile, 0 tmpArray = strarr(1,nLines) readf, tmpFile, tmpArray ;read in as a one dimensional string array close, tmpFile free_lun, tmpFile ;change first few charactors of line2 pos = strpos(tmpArray[0,1], ' ');end of first number in the line part2= strmid(tmpArray[0,1],pos);get the rest of the line newNum = strtrim(string(nlines - 2),2);number of station records = total lines - two header lines tmpArray[0,1] = newNum + part2 ; modify line 2 openw, tmpFile, inFile, /get_lun printf, tmpFile, tmpArray; write out the new OVP file close, tmpFile free_lun, tmpFile end ;Starting point for the function. Get all outputs from AVHRR_Cloud_Check.pro for year = 2006, 2006 do begin

rootDir_OVP = 'W:\DATA\RADCALC\OVP\'+strtrim(string(year), 2)+'\' rootDir = 'W:\Data\RS\AVHRR_Daily\'+strtrim(string(year), 2)+'\'

files=file_search(rootDir, '*_Checked.txt', count=FileNumber) if FileNumber GT 0 then begin for i=0, FileNumber-1 do begin ManyPts2ManyDates, rootDir_ovp, files[i] endfor endif

;if the output ovp files need to be modified (say for number of Records), do it here. files2=file_search(rootDir_OVP, '*.ovp', count=OVPFileNumber)


if FileNumber GT 0 then begin for i=0, OVPFileNumber-1 do begin Modify_OVP, files2[i] endfor endif endfor end

;This pro create parameter files for radwat.exe, with one parameter file per year ;Started: 2007.03.09 ;By: Li Lingtao ;The difference between this code and Create_Parameter_For_Radwat_All.pro is: ;this one needs .ovp and .psf to generate .rad file, the other needs .int/psf to generate .all file Pro Create_Parameter_For_Radwat_RAD rootDir = 'W:\DATA\RADCALC\' for year = 2006, 2006 do begin outFileDir = rootDir + 'rad\' + strtrim(string(year),2) ;check if the destination directory exists, create one if not outDirOK = file_search(outFileDir,COUNT=N_FOUND) IF N_FOUND EQ 0 THEN SPAWN, 'mkdir ' + outFileDir outFile = outFileDir + '\'+ strtrim(string(year),2)+'_radwat_rad.par' openw, fOut, outFile, /get_lun; psfFiles=file_search(rootDir+'psf\'+strtrim(string(year),2)+'\', '*.psf', count=psfFileNum) psfFileNameOnly = strarr(psfFileNum) for i=0, psfFileNum-1 do begin psfFileNameOnly[i]=(strsplit(file_basename(psfFiles[i]),'.', /extract))[0] endfor ovpFiles=file_search(rootDir+'ovp\'+strtrim(string(year),2)+'\', '*.ovp', count=ovpFileNum) ovpFileNameOnly = strarr(ovpFileNum) for i=0, ovpFileNum-1 do begin ovpFileNameOnly[i]=(strsplit(file_basename(ovpFiles[i]),'.', /extract))[0] endfor for i=0, ovpFileNum-1 do begin psfFile = where (psfFileNameOnly eq ovpFileNameOnly[i], count) if count GT 0 then begin ;check if the output .rad exists, delete it if yes, otherwise this par file won't work outFileName = rootDir + 'rad\'+strtrim(string(year),2)+'\'+ $ ovpFileNameOnly[i]+'.rad' outFileOK = file_search(outFileName,COUNT=N_FOUND) IF N_FOUND EQ 1 THEN FILE_DELETE, outFileName;SPAWN, 'del ' +$

outFileName printf, fOut,"s" printf, fOut, psfFiles[psfFile] printf, fOut, "r" printf, fOut, ovpFiles[i] printf, fOut, outFileName endif endfor printf, fOut,"q"


close, fOut free_lun, fOut endfor end

pro Check_Cols,startyear,endyear,path,ext,expectedCols,n_headerlines ;/LLT/TVN ;this is to check if each line in a file contains right number of records ;18 April 2007 ;initialise variables Count=0 For i=startyear, endyear Do Begin ;put all files into a list List = FILE_SEARCH(path+strtrim(string(i),2),'*.'+ext) If (n_elements(List) eq 0) then print, kdjfkdjf For j=0, n_elements(List)-1 Do Begin fileName=List[j]

cols = 0 openr, fin, fileName, /get_lun aline='' lineNum = n_HeaderLines for k = 0, n_HeaderLines -1 do begin ;skip headers readf, fin, aline endfor; k while not eof(fin) do begin lineNum++ readf, fin, aline cols=n_elements(strsplit(aline, /extract, ',')) if (cols ne expectedCols) then Begin print, ' '+fileName + ' has ' + STRTRIM(string(cols),2) +' columns in line '+ $ STRTRIM(string(lineNum),2)+', expected$

'+STRTRIM(STRING(expectedCols),2) Count++ endif endwhile close, fin free_lun, fin EndFor; j Endfor; i Print, ' Total number of '+STRTRIM(STRING(COUNT),2)+' files had the wrong number of columns' End pro SortRadByImageAcq ;TVN/02-Apr-2007 ;Takes in output from RadCalc (or RADWAT?) and sorts all valid stations into separate ;files for each image acquisition. ;Outputs sorted data by image acquisition date for input to spline (spline in, ;called "sin") ; ;TVN/17-Apr-2007/ adjust original code so it does not output Tair nor Rh, uses the ;25 station filter ; ;TVN/20-Apr-2007/ added a filter to select records that have unreasonably high or ;low ET or NDTI values. The original *.sin files are the same and are still output ;but now we also output two additional files per image acquisition. These are


;*.ehw and *.nhw (i.e., et half way, and ndti half way) where these new file ;have the unrealistic ET or NDTI records removed. The initial rules for removing ;records are: ; For NDTI: if (NDTI LT -0.25) OR (NDTI GT 1.25) ; For ET: if (ET LT 0) OR (NDTI GT 1.25) ; ;initialise variables startyear=2006 endyear=2006 path='W:\Data\RADCALC\RAD\' outpath='W:\Data\RADCALC\Spatial_Int_Overpass\' staFile='W:\spreadsheets_and_texts\stations_nostrings_DEM.txt' staThresh=25 ;read in station file to get lat longs ;read in station file ENVI_READ_COLS, staFile, sta_array BIGstalist=sta_array(0,*) BIGlatlist=sta_array(3,*) BIGlonlist=sta_array(4,*) BIGelevlist=sta_array(5,*) BIGDEMlist=sta_array(10,*) For i=startyear, endyear Do Begin ;get sta list from BOM Wind directory CD, path+strtrim(string(i),2) radList = FILE_SEARCH('*.rad') If (n_elements(radlist) eq 0) then print, kdjfkdjf ;Read in each rad file (i.e., one station at a time) and concat them together For j=0, n_elements(radlist)-1 Do Begin envi_read_cols, path+strtrim(string(i),2)+'\'+radlist[j], matrix ;set up necesarry lists stalist=lonarr(n_elements(matrix[0,*])) lonList=fltarr(n_elements(matrix[0,*])) latList=fltarr(n_elements(matrix[0,*])) demList=fltarr(n_elements(matrix[0,*])) elevList=fltarr(n_elements(matrix[0,*])) ;determine the station, elev, lat, and lon for this file and populate the relevant lists ;station list a=strsplit(radlist[j],'.',/EXTRACT) a=long(a[0]) stalist[*]=a ;lon and lat lists aindex=where(BIGstaList eq a) lonlist[*]=BIGlonlist[aindex] latlist[*]=BIGlatList[aindex] demlist[*]=BIGDEMList[aindex] elevlist[*]=BIGelevList[aindex] ;check if DEM eq -999 (null), if so, replace with station elev from dictionary nullDEM=where(demlist eq -999, nullDEMcount) if (nullDEMcount ne 0) then Begin demlist[nullDEM]=elevlist[nullDEM] nullELEV=where(elevlist[nullDEM] eq -1, nullelevcount) ;if elev from dictionary is null (-1) then stop program - this needs ;to be fixed. Bad idea to make station elev nulls eq -1 as this is a valid ;elevation - need to get original elevs again, then we can do something


;different here. if (nullelevcount ne 0) then print, dkfjdkj endif ;insert station, lon, and lat and DEM lists into the BIG rad array matrix=[[matrix],transpose(stalist), transpose(lonlist), transpose(latlist), transpose(demlist)] ;concat each separate array together into a big array if (j eq 0) then bigArray=matrix else bigArray=[[bigArray],[matrix]] Endfor ;j ;Sort the big "ALL" array based on image acquisition ; get list of image acquisitions from the big array ;make a unique identifier for each overpass by using date (*10000) plus time (hhmm) imgID=(bigArray[2,*]*10000.)+bigArray[5,*] imgAcqList=imgID[UNIQ(imgID, SORT(imgID))] ;Loop through each unique image acquistion and get any valid data For k=0, n_elements(imgAcqList)-1 Do Begin ;split up the uniq image acq ids into date and time str=STRTRIM(STRING(ulong64(imgAcqlist[k])),2) imgDate=STRMID(str,0,8) imgTime=STRMID(str,8,4) ;define the items in the big array that have this date and time index=where((bigArray[2,*] eq long(imgDate)) and (bigArray[5,*] eq fix(imgTime)), count) ;write out specified data for this image acquisition date only if it passes ;the filter for number of stations. this filter is the intersection between ;met stations and valid image data (e.g., cloud free) ; If (count GE staThresh) then Begin; 25 records (stations) are needed to interpolate outarray=fltarr(6,count) outarray[0,*]=bigarray[33,index]; station number outarray[1,*]=bigarray[34,index]; longitude outarray[2,*]=bigarray[35,index]; latitude outarray[3,*]=bigarray[36,index]; DEM outarray[4,*]=bigarray[14,index]; ET Day (mm/day) outarray[5,*]=bigarray[20,index]; NDTI (ratio) ;output the ehw file eIndex=where((outarray[4,*] GE 0) AND (outarray[5,*] LE 1.25), eCount) ;only print out the ehw file if there are more than 25 stations (or whatever we change

;staThresh to be) AND ET Day is GE 0 AND NDTI is LE 1.25 ; ;NOTE, this will result in some ehw files that contain less than 25 points ;if (eCount ge staThresh) then begin if (eCount ge 1) then begin OPENW, 1, outpath+strtrim(string(i),2)+'\'+imgDate+'_'+imgTime+'.ehw' PRINTF, 1, [[outarray[0,eindex]],outarray[1,eindex],$

outarray[2,eindex],outarray[3,eindex],outarray[4,eindex]],$ format="(I10, 4F12.6)"

CLOSE, 1 endif ;only print out the nhw file if there are more than 25 stations (or whatever we

;change staThresh to be) AND NDTI is between -0.25 and 1.25 nIndex=where((outarray[5,*] GE -0.25) AND (outarray[5,*] LE 1.25), nCount) ;if there are any records, then print out the nhw file ;if (nCount ge staThresh) then begin


if (nCount ge 1) then begin OPENW, 1, outpath+strtrim(string(i),2)+'\'+imgDate+'_'+imgTime+'.nhw' PRINTF, 1, [[outarray[0,nindex]],outarray[1,nindex],outarray[2,nindex],$

outarray[3,nindex],outarray[5,nindex]], format="(I10, 4F12.6)" CLOSE, 1 endif ;put summary values of nhw and ehw into an array if (k eq 0) then enhwMatrix=[imgDate,imgTime,eCount,nCount,Count] else $ enhwMatrix=[[enhwMatrix],[imgDate,imgTime,eCount,nCount,Count]] Endfor ;k ;output summary text file of enh and nhw counts OPENW, 1, outpath+strtrim(string(i-1),2)+'ehw_nhw_sinCounts.txt' PRINTF, 1, ' YYYMMDD',' HHMM',' ehw',' nhw',' sin' PRINTF, 1, enhwMatrix, format="(I10, 4I6)" CLOSE, 1 Endfor ;i End ;IDL 6 function ;Purpose:create ,nin file for interpolation of NDTI ;Method: Extract point Rn, Ta and f data for stations and add to .sin ;input file: Rn, Ta and f surfaces, .sin files (point location) ;output file: one txt file containing .nhw + Rn, Tdiff and f value ;Version History: 20 April 2007 ;based on Extract_pts_from surfaces Pro nhw2nin_WRON, nhwFile fName = file_basename(nhwFile);20011231_1404.nhw fName_Tokens = strmid(fName, [0,4,6,9], [4,2,2,4]);yy,mm,dd,hhmm samples = 1601 lines = 1701 pixelsize = 0.01 leftLongi = 137.995 topLat = -22.995 rightLongi = 154.005 bottomLat = -40.005 ;Read in nhw file ;the 6 columns in nhwArray are: ;0:station number, ;1:longitude (dd) ;2:latitude (dd) ;read in .nhw file envi_read_cols, nhwFile, nhwArray n_Cols = n_elements(nhwArray[*,0]) n_Lines = n_elements(nhwArray[0,*]) ;read in Rn surfaces Rn_Dir = 'W:\Data\RCALC_SURF\Overpass\' + fName_Tokens[0] + '\Rnet\' Rn_fName = $ fName_Tokens[0]+fName_Tokens[1]+fName_Tokens[2]+'_'+fName_Tokens[3]+'_Rn.flt Rn_fName = Rn_Dir + Rn_fName ;read in Tdiff surfaces Tdiff_Dir = 'W:\Data\RCALC_SURF\Overpass\' + fName_Tokens[0] + '\Tdiff\' Tdiff_fName = $ fName_Tokens[0]+fName_Tokens[1]+fName_Tokens[2]+'_'+fName_Tokens[3]+'_tdif.flt Tdiff_fName = Tdiff_Dir + Tdiff_fName


;read in f surfaces f_Dir = 'W:\DATA\RS\corrected_fpar\converted_coverage\SE_Australia\filled_nulls' f_fName = file_Search(f_Dir, fName_Tokens[0]+fName_Tokens[1]+'*.flt', count=n) if n GT 1 then f_fName = f_fName[0] ;grab point data from surfaces Rn_surface = fltarr(samples,lines) Tdiff_surface = fltarr(samples,lines) f_surface = fltarr(samples,lines) thisRn = fltarr(n_Lines) thisTdiff = fltarr(n_Lines) thisf = fltarr(n_Lines) OPENR, fSur, Rn_fName, /GET_LUN readu, fSur, Rn_surface free_lun, fSur OPENR, fSur, Tdiff_fName, /GET_LUN readu, fSur, Tdiff_surface free_lun, fSur OPENR, fSur, f_fName, /GET_LUN readu, fSur, f_surface free_lun, fSur for i = 0, n_Lines-1 do begin if ( nhwArray[1,i] GE leftLongi && nhwArray[1,i] LE rightLongi && $ (nhwArray[2,i] GE bottomLat) && (nhwArray[2,i] LE topLat)) then begin thisRn[i] = Rn_surface[floor((nhwArray[1,i]-leftLongi)/pixelsize), $

floor((topLat-nhwArray[2,i])/pixelsize)] thisTdiff[i] = Tdiff_surface[floor((nhwArray[1,i]-leftLongi)/pixelsize), $ floor((topLat-nhwArray[2,i])/pixelsize)] thisf[i] = f_surface[floor((nhwArray[1,i]-leftLongi)/pixelsize), $

floor((topLat-nhwArray[2,i])/pixelsize)] endif endfor ;add 3 more columns to .nhw file - note that the NDTI (or independent var for ANUSPLIN MUST be last column in the date file) ninArray =fltarr(n_Cols+3, n_Lines) ninArray[0:n_Cols-2,*] = nhwArray[0:n_Cols-2, *] ninArray[n_Cols-1,*] = thisRn ninArray[n_Cols,*] = thisTdiff ninArray[n_Cols+1,*] = thisf ninArray[n_Cols+2,*] = nhwArray[n_Cols-1,*] ;if Rn, Tdiff or f is null, delete the line ND = -9999. invalid_Rn = where (ninArray[4,*] eq ND) invalid_Tdiff = where (ninArray[5,*] eq ND) invalid_f = where (ninArray[6,*] eq ND) ;if Rn and Tdiff out of SPLINA ranges, get rid of them as well. ttt = [-1, invalid_Rn,invalid_Tdiff,invalid_f] invalid_all = ttt [uniq(ttt, sort(ttt))] if n_elements(invalid_all) LE 1 then begin ;output .nin file openw, fout, (strsplit(nhwFile, '.', /extract))[0] + ".nin", /get_lun


printf, fout, ninArray, format= '(i10, 7F11.4)' free_lun, fout endif else begin ; if ((n_Lines - n_elements(invalid_all)+1) ge staValid) then begin ninArray_out = fltarr(n_Cols + 3, n_Lines - n_elements(invalid_all) + 1) q = 0 for p = 0, n_Lines - 1 do begin ;big array with null if (where (invalid_all eq p) eq -1) then begin ninArray_out[*,q] = ninArray[*,p] q ++ endif endfor ;output .nin file openw, fout, (strsplit(nhwFile, '.', /extract))[0] + ".nin", /get_lun printf, fout, ninArray_out, format= '(i10, 7F11.4)' free_lun, fout ; endif endelse end for i = 2006,2006 do begin nhwFileDir = 'W:\DATA\RADCALC\Spatial_Int_Overpass\' + strtrim(string(i),2)+'\' nhw_List = file_search(nhwFileDir +'*.nhw', count=FileNumber) If (FileNumber eq 0) then print, 'no .nhw file found' else begin For j=0, FileNumber-1 Do Begin nhw2nin_WRON, nhw_List[j] Endfor endelse endfor end

10.12 IDL code to generate overpass NDTI surfaces pro Create_Splina_cmd_Files_NDTI ;TVN/LLT Apr-2007 ;create cmd files for each surface to be splined ; assumes a yearly directory structure ; requires a template *.cmd file as input at the base directory specified below ;initialise variables startyear=2006 endyear=2006 path='W:\DATA\RADCALC\Spatial_Int_Overpass\' outpath='W:\DATA\RADCALC\Spatial_Int_Overpass\SPLIN_cmd_files\' ;utpath='W:\Data\RADCALC\Spatial_Int_Overpass\SPLIN_cmd_files\' ext='nin' template=outpath+'template_NDTI_splna.cmd' ;magicRecord=15 magicRecord=[22,23,24,25,26] ;make sure template exists Result = FILE_TEST(template) if (result ne 1) then print, dkfjkdj For i=startyear, endyear Do Begin ;put all files into a list CD, path+strtrim(string(i),2) List = FILE_SEARCH('*.'+ext, count=n)


If (n eq 0) then print, kdjfkdjf For j=0, n_elements(List)-1 Do Begin filename=strsplit(list[j],'.', /Extract) ;read in the template file and modify the "magic" record where the filename is kept counter=0 aline='' sarray=strarr(1) OPENR, unit, template, /GET_LUN while not eof (unit) do begin readf, unit, aline if counter eq 17 then begin strput, aline, strtrim(string(i),2)+'\'+filename[0],37 endif else begin tmp = where (magicRecord eq counter+1, count) if count gt 0 then strput, aline, strtrim(string(i),2)+'\'+filename[0],53 endelse if (counter eq 0) then begin strput, aline, filename[0],0 sarray[0]=aline endif else sarray=[[sarray],[aline]] counter++ endwhile free_lun, unit ;output the new modified cmd file using input basename OPENW, 1, outpath+strtrim(string(i),2)+'\'+filename[0]+'_NDTI_SPLINA.cmd' PRINTF, 1, sarray CLOSE, 1 print, outpath+strtrim(string(i),2)+'\'+filename[0]+'_NDTI_SPLINA.cmd' Endfor Endfor End pro Create_SPLINA_Batch_Runs_NDTI ;TVN/05-Apr-2007 ;create batch file to run all splina commands ; assumes a yearly directory structure ;initialise variables startyear=2006 endyear=2006 path='W:\DATA\RADCALC\Spatial_Int_Overpass\SPLIN_cmd_files\' ext='NDTI_SPLINA.cmd' aline1='SPLINA' aline2='_NDTI_SPLINA.bat' For i=startyear, endyear Do Begin ;put all files into a list CD, path+strtrim(string(i),2) List = FILE_SEARCH('*'+ext) nRecs=n_elements(list) If (n_elements(List) eq 0) then print, kdjfkdjf sarray=strarr(nRecs)


For j=0, nRecs-1 Do Begin filename=strsplit(list[j],'.', /Extract) sarray[j]=aline1+' < '+list[j]+' > '+filename[0]+'.log' Endfor ;output the batch file OPENW, 1, path+strtrim(string(i),2)+'\'+strtrim(string(i),2)+aline2 PRINTF, 1, sarray CLOSE, 1 print, path+strtrim(string(i),2)+'\'+strtrim(string(i),2)+aline2 Endfor End pro Create_Lapgrd_cmd_Files_NDTI ;TVN/LLT Apr-2007 ;Modified by LLT ;create cmd files for each surface to be splined ; assumes a yearly directory structure ; requires a template *.cmd file as input at the base directory specified below ;initialise variables startyear=2001 endyear=2001 ;path = 'W:\DATA\RADCALC\Spatial_Int_Overpass\' path = 'W:\DATA\RADCALC\Spatial_Int_Overpass\SPLIN_cmd_files\' outpath='W:\DATA\RADCALC\Spatial_Int_Overpass\LAPGRD_cmd_files\' ext='_NDTI_SPLINA.sur' template=outpath+'template_NDTI_lapgrd.cmd' ;make sure template exists Result = FILE_TEST(template) if (result ne 1) then print, 'Trouble finding the template' For i=startyear, endyear Do Begin ;put all files into a list CD, path+strtrim(string(i),2) List = FILE_SEARCH('*'+ext, count = n) If (n eq 0) then print, kdjfkdjf For j=0, n_elements(List)-1 Do Begin filename=strsplit(list[j],'.', /Extract) ;get right f_image yyyymm = strmid(filename[0], 0,6) yyyymmdd = strmid(filename[0], 0,8) time = strmid(filename[0], 9,4) f_image = $ FILE_SEARCH('W:\DATA\RS\corrected_fpar\converted_coverage\SE_Australia\filled_nulls\',$

yyyymm+'*.flt') ;read in the template file and modify the "magic" record where the filename is kept counter=0 aline='' sarray=strarr(1) OPENR, unit, template, /GET_LUN while not eof (unit) do begin readf, unit, aline


if counter eq 11 then begin strput, aline, strtrim(string(i),2)+'\Rnet\'+yyyymmdd+'_'+time,28 endif else if counter eq 13 then begin strput, aline, strtrim(string(i),2)+'\Tdiff\'+yyyymmdd+'_'+time,28 endif else if counter eq 18 then begin strput, aline, strtrim(string(i),2)+'\NDTI\'+yyyymmdd+'_'+time,28 endif else if counter eq 15 then begin aline = f_image endif else begin if counter eq 0 then strput, aline, strtrim(string(i),2)+'\'+filename[0],58 endelse if (counter eq 0) then sarray[0]=aline else sarray=[[sarray],[aline]] counter++ endwhile free_lun, unit ;output the new modified cmd file using input basename OPENW, 1, outpath+strtrim(string(i),2)+'\'+strmid(filename[0],0,13)+'_NDTI_LAPGRD.cmd' PRINTF, 1, sarray CLOSE, 1 print, outpath+strtrim(string(i),2)+'\'+filename[0]+'_NDTI_LAPGRD.cmd' Endfor Endfor End pro Create_LAPGRD_Batch_Runs_NDTI ;TVN/05-Apr-2007 ;Modified LLT 30 April 2007 ;create batch file to run all LAPGRD commands for Overpass variables (Ta, Ea and VPD) ;assumes a yearly directory structure ;initialise variables startyear=2001 endyear=2001 path='W:\DATA\RADCALC\Spatial_Int_Overpass\LAPGRD_cmd_files\' ext='NDTI_LAPGRD_Tdiff_Interp.cmd' aline1='W:\Programs\Anuspl43\bin\LAPGRD' aline2='_NDTI_LAPGRD_Tdiff_Interp.bat' For i=startyear, endyear Do Begin ;put all files into a list CD, path+strtrim(string(i),2) List = FILE_SEARCH('*'+ext) nRecs=n_elements(list) If (n_elements(List) eq 0) then print, kdjfkdjf sarray=strarr(nRecs) For j=0, nRecs-1 Do Begin filename=strsplit(list[j],'.', /Extract) sarray[j]=aline1+' < '+list[j]+' > '+filename[0]+'.log' Endfor ;output the batch file OPENW, 1,path+strtrim(string(i),2)+'\'+strtrim(string(i),2)+aline2 PRINTF, 1, sarray CLOSE, 1 Endfor End


pro create_HDRs, imgName, hdrTemplate nline = 0 aline = ' ' openr, f_hdr, hdrTemplate, /get_lun while not eof(f_hdr) do begin nline++ readf, f_hdr, aline endwhile hdrArray=strarr(1,nline) point_lun, f_hdr, 0; put pointer back readf, f_hdr, hdrArray close, f_hdr free_lun, f_hdr outFile = (strsplit(imgName, '.', /extract))[0] + '.hdr' openw, tmpFile, outFile, /get_lun printf, tmpFile, hdrArray; write out file close, tmpFile free_lun, tmpFile end img_Path = 'W:\Data\RCALC_SURF\Overpass\' startYear = 2006 EndYear = 2006 arcHeader = 'W:\Data\RCALC_SURF\ArcHeader_Template.hdr' enviHeader ='W:\Data\RCALC_SURF\EnviHeader_Template.hdr' for i = startYear, endYear do begin yearString = strtrim(string(i), 2) img_List = file_search(img_Path+yearString+'\Tdiff\','*.flt', count=FileNumber) If (FileNumber eq 0) then print, 'no images found' else begin For j=0, FileNumber-1 Do Begin theFile = file_basename(img_List[j]) create_HDRs, img_List[j], arcHeader Endfor endelse endfor end

10.13 IDL code to generate initial fortnightly and monthly NDTI composites

Pro Composite_images ;TVN/18-Apr-2007/ ; ;initialise variables samples=1601 lines=1701 ND=-9999.;-999.;-9999. startyear=2001 endyear=2001 origPath='W:\Data\RCALC_SURF\Overpass\' outPath='W:\Data\RCALC_SURF\Overpass\composites\FN_Ta_2001\' yearFlag=1 ext='grd'


count = 0 minVal=0;-0.25;-1.;0. maxVal=1;1.25;1. compositeType='mean';'max';'mean' productType='NDTI';’NDVI’ monthStrArray=['01','02','03','04','05','06','07','08','09','10','11','12'] ;numDays=31; numDays = compositing period, e.g., 10 daily, 14 daily, etc...will be specified by user ;set up day values to start the search on for 10 daily compositing; ;this array will change for other compositing periods, and will eventually be ;provided by the user when the program is called... dayStrArray=['01','16'] ;fortnightly numDaysArray=[15,16]; fortnightly ;dayStrArray=['01']; monthly ;numDaysArray=[31]; monthly For year=startyear, endyear Do Begin yearStr=STRTRIM(STRING(year),2) path=origPath+yearStr+'\'+productType+'\' For month=0, n_elements(monthStrArray)-1 Do Begin monthStr=monthStrArray[month] For interval=0,n_elements(dayStrArray)-1 Do Begin dayStr=dayStrArray[interval] numDays=numDaysArray[interval] ;define list of all files within the specific compositing period List=returnNDTIfileList(path, yearStr, monthStr, dayStr, numDays, ext) ;make sure the file search found something if (list[0] ne '-1') then begin ;put all the images in the list into a stack stack=returnFloatStack(List, samples, lines, 0) ;Get mean, min, or max of the stack as the first go of compositing outArray=fltarr(samples,lines) For s=0, samples-1 Do Begin For l=0, lines-1 Do Begin stackIndex=where((stack[s,l,*] NE ND) AND (stack[s,l,*] GE minVal) AND (stack[s,l,*] LE maxVal), stackCount) if (stackCount NE 0) then begin if (compositeType eq 'mean') then outarray[s,l]=mean(stack[s,l,stackindex]) if (compositeType eq 'max') then outarray[s,l]=max(stack[s,l,stackindex]) if (compositeType eq 'min') then outarray[s,l]=min(stack[s,l,stackindex]) endif else outarray[s,l]=ND Endfor;l Endfor;s ;write out composite images ;if the composit period is less than monthly, then print out one way: if (n_elements(numDaysArray) GT 1) then Begin if (yearFlag eq 1) then $ outName=outpath+yearStr+'\'+yearStr+MonthStr+'_'+STRTRIM(STRING(INTERVAL),2)+$

'_'+compositeType+'COMPOSITE.flt' else $ $ outName=outpath+yearStr+MonthStr+'_'+STRTRIM(STRING(INTERVAL),2)+'_'+$ compositeType+'COMPOSITE.flt'

OPENW, unit, outName, /GET_LUN WRITEU, unit, outarray CLOSE, unit free_lun, unit print, outName


endif else Begin if (yearFlag eq 1) then $ outName=outpath+yearStr+'\'+yearStr+MonthStr+'_'+compositeType+'COMPOSITE.flt' else $ outName=outpath+yearStr+MonthStr+'_'+compositeType+'COMPOSITE.flt' OPENW, unit, outName, /GET_LUN WRITEU, unit, outarray CLOSE, unit free_lun, unit print, outName endelse endif EndFor;interval EndFor;month EndFor;year END function returnFloatStack, List, samples, lines, band ;TVN/24-Apr-2007/ ;returns a floating point stack of the specified band for the list of image names ; ;where band=the idl index of the band number that should be stacked -assumes bsq ; ;example: return a stack of all the NDTI image in the second half of Jan 2001 ; List=returnNDTIfileList('U:\AWAP\Data\RCALC_SURF\Overpass\', '2001', '01', '16', 16, 'grd') ; stack=returnFloatStack(List, 1601, 1701, 0) ;Stack each image in the list numImages=n_elements(List) print, numImages stack = fltarr(samples,lines,numImages) For m=0, numImages-1 Do Begin ;read image data into a variable ;Open input files if (band ne 0) then invar=fltarr(samples,lines,band+1) else invar=fltarr(samples,lines) inVARName = List[m] OPENR, unit, inVARName, /GET_LUN readu, unit, invar close, unit free_lun, unit if (band ne 0) then stack[*,*,m]=invar[*,*,band] else stack[*,*,m]=invar Endfor;m return, stack End function returnNDTIfileList, p, yearStr, monthStr, dayStr, numDays, ext ;TVN/24-Apr-2007/ ;returns a list of all available NDTI images in a given path ;for a given year, month, start day, and number of days ; ;example: return a list of all the NDTI images for the second half of Jan 2001 ; ;List=returnNDTIfileList('U:\AWAP\Data\RCALC_SURF\Overpass\NDTI\', '2001', '01', '16', 16, 'grd') ; ;ENVI> print, list ;U:\AWAP\Data\RCALC_SURF\Overpass\2001\NDTI\2001011614403_NDTI_lapgrd.grd


; U:\AWAP\Data\RCALC_SURF\Overpass\2001\NDTI\2001011714303_NDTI_lapgrd.grd ;U:\AWAP\Data\RCALC_SURF\Overpass\2001\NDTI\2001011814193_NDTI_lapgrd.grd ;U:\AWAP\Data\RCALC_SURF\Overpass\2001\NDTI\2001011815593_NDTI_lapgrd.grd ;U:\AWAP\Data\RCALC_SURF\Overpass\2001\NDTI\2001011914093_NDTI_lapgrd.grd ;U:\AWAP\Data\RCALC_SURF\Overpass\2001\NDTI\2001011915493_NDTI_lapgrd.grd ;U:\AWAP\Data\RCALC_SURF\Overpass\2001\NDTI\2001012013593_NDTI_lapgrd.grd ;U:\AWAP\Data\RCALC_SURF\Overpass\2001\NDTI\2001012113483_NDTI_lapgrd.grd ;U:\AWAP\Data\RCALC_SURF\Overpass\2001\NDTI\2001012213383_NDTI_lapgrd.grd ;U:\AWAP\Data\RCALC_SURF\Overpass\2001\NDTI\2001012313273_NDTI_lapgrd.grd ;U:\AWAP\Data\RCALC_SURF\Overpass\2001\NDTI\2001012413173_NDTI_lapgrd.grd ;U:\AWAP\Data\RCALC_SURF\Overpass\2001\NDTI\2001012414583_NDTI_lapgrd.grd ;U:\AWAP\Data\RCALC_SURF\Overpass\2001\NDTI\2001012514483_NDTI_lapgrd.grd ;U:\AWAP\Data\RCALC_SURF\Overpass\2001\NDTI\2001012614383_NDTI_lapgrd.grd ;U:\AWAP\Data\RCALC_SURF\Overpass\2001\NDTI\2001012714273_NDTI_lapgrd.grd ;U:\AWAP\Data\RCALC_SURF\Overpass\2001\NDTI\2001012814173_NDTI_lapgrd.grd ;U:\AWAP\Data\RCALC_SURF\Overpass\2001\NDTI\2001012815573_NDTI_lapgrd.grd ;U:\AWAP\Data\RCALC_SURF\Overpass\2001\NDTI\2001012914063_NDTI_lapgrd.grd ;U:\AWAP\Data\RCALC_SURF\Overpass\2001\NDTI\2001012915473_NDTI_lapgrd.grd ;U:\AWAP\Data\RCALC_SURF\Overpass\2001\NDTI\2001013013563_NDTI_lapgrd.grd ;U:\AWAP\Data\RCALC_SURF\Overpass\2001\NDTI\2001013015373_NDTI_lapgrd.grd ;U:\AWAP\Data\RCALC_SURF\Overpass\2001\NDTI\2001013113463_NDTI_lapgrd.grd ;U:\AWAP\Data\RCALC_SURF\Overpass\2001\NDTI\2001013115263_NDTI_lapgrd.grd ;initialise variables tmpCount=0 fnStart = '*'+yearStr+monthStr numArray=indgen(numDays)+fix(dayStr) searchStrings=strarr(numDays) ;insert '0's in front of the number strings if necessary For i=0,numDays-1 Do begin str=STRTRIM(STRING(numarray[i]),2) numDigits=STRLEN(str) ;if numDigits eq 1, then insert a zero in front of it if (numDigits eq 1) then searchStrings[i]=fnStart+'0'+str+'*.'+ext else searchStrings[i]=fnStart+str+'*.'+ext End;i ;search based on the possible filenames for the specified compositing period ;put all files into a list For j=0, n_elements(searchStrings)-1 Do begin ;for recursive searching use two search variables tmp=FILE_SEARCH(p,searchStrings[j]) ;for non-recursive searching, cd into the path and use one search variable ;CD, p ;tmp=FILE_SEARCH(searchStrings[j]) If (tmp[0] ne '') then begin if (tmpCount eq 0) then List=tmp else List=[LIST,tmp] tmpCount++ Endif EndFor;j ;if no files were found in the search, set the list to -1 if (n_elements(list) eq 0) then list=['-1'] return, list End


10.14 IDL code to gap-fill fortnightly and monthly NDTI composites pro fill_nulls_ndti_composites_hierarchy_2tier_only ;TVN/12-July-2007/ ;fills all nulls in each of the Monthly or FN'ly composites ; ;initialise variables stYear=1990 enYear=2006 samples=1601 lines=1701 ext='.flt' interval='MTH';'FN' basePath='W:\Data\RCALC_SURF\Overpass\composites\'+interval+'\';0 to 1 mean composites GMname=basePath+'grandMean_'+interval+ext;Grand mean composites outpath='W:\Data\RCALC_SURF\Overpass\composites\'+interval+'_2tier_filled_nulls_z1\' ND=-9999. month_num_array=['01','02','03','04','05','06','07','08','09','10','11','12'] if (interval eq 'FN') then Begin numIntervals=2 z=24 endif else Begin numIntervals=1 z=12 endelse array=fltarr(samples,lines); zero to one FN mean composite GMstack=fltarr(samples,lines,z) ;Read in grand mean data just once. OPENR, unit, GMName, /GET_LUN readu, unit, GMstack free_lun, unit ;define the background nulls backgrndNulls=where(GMstack[*,*,0] eq ND, BNcount) ;go through each (FN for each) Month for each Year For year=styear, enYear Do Begin yearStr=STRTRIM(STRING(year),2) Count=0 For month=0, 11 DO BEGIN monthStr=month_num_array[month] For i=0,numIntervals-1 Do BEGIN intervalString=STRTRIM(STRING(i),2) p=basePath+yearStr+'\' if (interval eq 'FN') then searchStr=yearStr+monthStr+'_'+intervalString+'_*'+ext else searchStr=yearStr+monthStr+'*'+ext FileList=FILE_SEARCH(p,searchStr,count=FileCount) if (FileCount gt 1)then Begin print, 'File count for '+searchStr+', should have been 1 or less. Skipping this one..' break endif


;Now read in the zero-to-one FN image and the all-data-truncated image into variables if (FileCount ne 0) then Begin fileName = FileList[0] OPENR, unit, fileName, /GET_LUN readu, unit, array free_lun, unit endif else array[*,*]=ND; if there was no file for this FN (or MTH) then use a completely nulled array ;get the correct band of the GM stack GMarray=GMstack[*,*,count] outarray=Array msk=bytarr(samples,lines) ;Fill nulls in all data composite with GM composite valid=where(array ne ND, vCount, complement=nulls, ncomplement=ncount) if (ncount ne 0) then begin outArray[nulls]=GMarray[nulls] msk[nulls]=3 endif if (vcount ne 0) then msk[valid]=2 ;make sure to set the ocean nulls to 0 in the mskarray msk[backgrndNulls]=0 ;write out version 1 of the filled image - using 2 tiers and it's associated mask if (fileCount eq 0) then begin if (interval eq 'FN') then outName=yearStr+monthStr+'_'+intervalString+'_'+'meanCOMPOSITE_filled'+ext else outName=yearStr+monthStr+'_'+'meanCOMPOSITE_filled'+ext endif else begin tmp=strsplit(FileList[0],'.',/extract) outName=file_basename(tmp[0])+'_filled.flt' endelse OPENW, unit, outpath+yearStr+'\'+outName, /GET_LUN WRITEU, unit, outarray CLOSE, unit free_lun, unit print, outpath+yearStr+'\'+outName if (fileCount eq 0) then begin if (interval eq 'FN') then outName=yearStr+monthStr+'_'+intervalString+'_'+'meanCOMPOSITE_filled_LUT.byt' else outName=yearStr+monthStr+'_'+'meanCOMPOSITE_filled_LUT.byt' endif else begin tmp=strsplit(FileList[0],'.',/extract) outName=file_basename(tmp[0])+'_filled_LUT.byt' endelse OPENW, unit, outpath+yearStr+'\'+outName, /GET_LUN WRITEU, unit, msk CLOSE, unit free_lun, unit print, outpath+yearStr+'\'+outName


Count++ EndFor EndFor EndFor END

Tim R. McVicar, Tom G. Van Niel, Lingtao Li, Edward A. King and Randall J. Donohue

Deriving moisture availability from time series remote sensing for ecohydrological applications: Development of a prototype near real-time operational system.


For further information:

CSIRO Land and WaterTim McVicarPhone: (02) 6246 5741Email: [email protected]: www.clw.csiro.au

Deriving m

oisture availability from tim

e series remote sensing C

SIRO Land and W

ater Science Report 37/07

CSIRO Land and Water Science Report 37/07 · Science Report 37/07. Deriving moisture availability from time series ... (Print format) ISBN 978 0 643 09506 9 (Web format) CSIRO Land

Documents