Scientific Investigations; Support from Research Data Archives for Computing in Atmospheric Sciences 2001 29 October, 2001 Steven Worley National Center.

Scientific Investigations; Support from Research Data

Archivesfor

Computing in Atmospheric Sciences 2001

29 October, 2001Steven Worley

National Center for Atmospheric ResearchScientific Computing Division

Key Steps of Scientific Investigations

• Formulate the questions and review the state of understanding

• Search and discover data• Access data• Analyzes data• Community sharing and archive • Document new understandings

Search and Discover Data

• How? Web based Information Server• Salient Features

– 2.5K + html pages (metadata)– All datasets are described (500+)– Location of all data files in MSS– Higher level information

• Catalogs• Project specific descriptions

Always current dataset descriptions

Features

• Organization Navigation

• Archive Navigation

• Pull down menus

• Search

• Project Links

Dataset Page

• Title and Brief description

• Systematic Navigation

• Metadata highlights

• Period of Record

• Usage

• Variables

• Related Sites (NOAA)

• Contact Person

• Related Datasets

Brief Archive History and Specifications

• Started in middle 1960’s, (35 years)

• Managed by nine people

• 211K data files

• 17 TB in a MSS

• 530 datasets – all sizes

Global Observations

P.O.R # Yrs Incep.

Date

Comments

Rawinsondes 1946-

on

55 1967 Upper Air

Pibals 1942-

on

59 1973 Upper Air, wind

Aircraft 1947-

on

52 1973 USAF and

Commer.

Sat. cloud wind

drift

1967-

on

34 1973 GOES and GTS

Satellite

Soundings

1969-

92

25 1973 TOVS +

irradiance

Surface Synoptic 1948-

on

53 1975 some much older

Ocean Surface 1794-

on

203 1981 COADS

Usages:

• Input for global atmospheric reanalysis

• Basic long term climate assessment and case studies

Operational and Composite Analyses

U.S. Analyses for the N. H. (Early Operational outputs and composites) P.O.R. Comments

Daily SLP Analysis 1889-on Composite of data sources, 2 x daily later period

Selected Early Analyses 1946,1950 - on 700mb, 500mb, 300mb NMC Oper. Analysis 1962-on Z &T @ 10mb – sfc. (11 lev)

Global Operational Analyses NCEP/NMC 1976-on Many levels and variables ECMWF 1980-on Many levels and variables

Special Analyses Australian 1972-1992 Discontinued FNOC (U.S. Navy) 1973-1993 Discontinued

• Daily SLP is a small but very popular dataset, e.g. NAO evaluations

• Two main operational centers provide the best current analyses

ECMWF Global Operational Analyses Data Product Period of

Record Temporal Res.

Spatial Res. (dg)

Update Cycle

# Levs.

# Vars.

Major Variables

Upper Air 1985- 06/ 2001

6 hr ~1.125 6 mn 21 8 z,t,wind,rh

Surface 1985- 06/ 2001

6 hr ~1.125 6 mn 1 47 p,t,wind,soil.t, soil.moist.

Supplemental 1985- 06/ 2001

6 hr ~1.125 6 mn 16 rad.,stress,heat.flux, clouds

Extension 1991- 06/ 2001

6 hr ~1.125 6 mn 18 precip,heat.flux

Sf c/ Up.Air Low Resolution

1985- 06/ 2001

12 hr 2.5 1 mn 21+ 14 sf c.t,sf c. p,z,t,wind,rh

Sf c/ Up.Air †

Low Resolution

1985- 06/ 2001

1 mn 2.5 ~1 mn 21+ 14 sf c.t,sf c. p,z,t,wind,rh

† Computed by the SCD/ DSS

Key Aspects• Medium size archive – 170 Gigabytes• multi-(product, temporal res., spatial res.) - complex

Concerns;

• Restricted distribution• U.S. non-profits and UCAR members only• Need online authentication and authorization for easy access

NCEP Operational Analyses Data Product Period of

Record Temporal Res.

Spatial Res. (dg)

Update Cycle

# Levs.

# Vars.

Major Variables

Final Analysis Global 2.5

1976- 08/ 2001

6 hr 2.5 1 mn 11+ 15 z,t,wind,rh, sf c.t, sf c.p

Final Analysis Global 1.0

09/ 1999 - today

6 hr 1.0 Daily (FTP)

26+ 71 z,t,wind,rh,vorticity sf c.t,sf c.p

ETA-3D N. America

05/ 1995- 07/ 2001

6 hr 40 (km) 1 mn 26+ 5 z,t,wnd,sh, precip(f orecast)

ETA-Surface N. America

05/ 1995- 07/ 2001

6 hr 40 (km) 1 mn 12 wind,sf c.p,sf c.t, soil.t,soil.p

LFM (1971-1995) and NGM (1984-cont), N. America, 190km and 6 hr resolution, are available but ETA is considered a superior replacement.

Highlights

• Frequent updates to FNL, 1º, daily via FTP

• High resolution N. America product, ETA at 40km

• No distribution restrictions or cost

Reanalyses

P.O.R # Yrs Incep. Date

NCEP/NCAR Reanalysis

I

1948-06/2001 53 1994

ECMWF ERA-15 1979-1993 15 1994

NCEP Reanalysis II 1979-06/2001 22 1998

Notes:

• ERA-15 is finished, ERA-40 is running now

• NCEP II, primarily experimental run

NCEP/NCAR Global Atmospheric Reanalysis Data Product Period of

Record Temporal Res.

Spatial Res. (dg)

Update Cycle

# Levs.

# Vars.

Major Variables

Analysis on Pressure Sf c.

1948- 6/ 2001

6 hr 2.5 1-2 mn 17 7 u,v,z,t,rh

Analysis on Sigma Sf c.

1948- 6/ 2001

6 hr 192x94 Gaussian

1-2 mn 28 6 u,v,t,sph,rel.vort,

Analysis on Theta Sf c.

1948- 6/ 2001

6 hr 2.5 1-2 mn 11 10 N**2, ab.vort,u,v, t,rh,pot.vort

Surf ace Flux Fields

1948- 6/ 2001

6 hr 2.5 1-2 mn 12 Clouds, rad.flx, soil.moist,heat.flx precip

Monthly Mean Anal. P. Sf c.

1948- 2000

1 mn 2.5 1-2 mn 17+ 36 u,v,z,t,rh

CD-ROMS 1953- 1999

12 hr, 1 day, 1mn

2.5 3-6 12 u,v,z,t,rh,heat.flx, rad,flx,precip

model qc’ed observations are returned f orecasts, once every 5 days a f orecast fi elds, 6 hr, available out to 8 days

Outstanding Features• Three different coordinate surfaces• Very long analysis, 2+ Terabytes size• Unrestricted distribution• CD-ROMS are very popular

Countries Receiving Reanalysis CDROMs

Highlights• Over 8900 CDROMs 1997-09/2001

• Recipients; U.S. 46%, Japan 11%, (Canada, UK) 4%, (Germany, India) 3%, (Australia, S.Korea, Spain, Mexico, Norway, Russia, France) 2%

Reanalysis Users for 2001 (4th qtr estimated)

209 From the MSS [157 Jan.-Sep.] 47 On CDROM [35] 48 Custom data orders on FTP or Tape [36] 540 From the online server [406]

844 Total Served

0

50

100

150

200

250

Un

iqu

e U

sers

1995 1996 1997 1998 1999 2000 2001

Years

NCEP/NCAR Renalysis from the MSS

Estimate

Other Users

Univ. Users

NCAR Users

Reanalysis Data Distributed for 2001 (4th qtr estimated)

• 9616 GB from the MSS [7230 GB Jan.-Sep.]

• 808 GB On CD-ROM [935, @650Mb/CDROM]• 1383 GB Custom orders, FTP and tape [1040]• 88 GB From the online server [66 GB]

11895 GB, 11.9 TB Total

0

1000

2000

3000

4000

5000

6000

7000

8000

9000

10000

Dat

a A

mo

un

t (G

B)

1995 1996 1997 1998 1999 2000 2001

Years

NCEP/NCAR Reanalysis from the MSS

Estimate(GB)

Other (GB)

Univ. (GB)

NCAR (GB)

GCIP Model Data Center Collection

High resolution atmospheric models focused on energy and hydrology cycles.

GCIP: GEWEX Continental-Scale International Project / GEWEX : Global Energy and Water Cycle Exper.

• Critical data for N. American mesoscale studies• Complete archive is about 1 Terabyte

Eta –NCEP 3 hr 40 km25 lvs

5/1995 – 7/2001

MAPS – FSL NOAA

3 hr 40 km5 lvs

8/1996 - 7/2001

GEM – Canadian

6 hr 41 km28 lvs

4/1997 – 6/2001

Ocean Model Data

MICOM; Miami Isopynic Coordinate Ocean Model, 1/12th degree 70N to 28 S, 16-20 layers

COADSClim. Forcing

6 yrs 305 Gigabytes

ECMWFClim. Forcing

2 yrs 164 Gigabytes

ECMWF Daily Forcing

5 yrs 415 Gigabytes(1979-1983)

University of Miami

6-yr Mean T at 5 meters

Dataset Sizes and Scales

• Today – ~ 800 Unique users– ~ 12 Terabytes data transferred– 2 Terabyte dataset size– Example: NCEP/NCAR Reanalysis

• Near Future Excludes TB-PB Level 0 and 1 satellite and the super

scale experimental models– Numbers of Users, ~ same– Data transferred, 5x to 10x more ?– Dataset size, 2-20 TB– Examples:

• Ocean and Atmosphere models • ECMWF Reanalysis (ERA40)

Access to Data

Methods• NCAR computers

– From the local MSS

• Web data server • Custom data packages

– by request (FTP, tape, CDROM)

Users • World class programmer• Research Scientist• Graduate Students• Undergraduate Students

Data Access in the future

• Do we continue doing what we are doing?

“Absolutely”Why? It Works– Over 1000 users annually

• Very diverse skills

– The archive is a heterogeneous collection• Many formats (ASCII, Binary, GrIB, BUFR, netCDF, HDF)• Many sizes (1 MB to 2 TB)

– Capable of serving large and small projects

Maintain a variety of flexible methods

Data Access in the future

• Keys to handling future larger collections– Plan to create useful data products

• Condensed datasets from high resolution output• Group most popular variables products together

– Serve many, e.g. CDROMS and WWW

– Continue to develop emerging online data systems

• User driven subset selection with graphics and data download options

• Server-side elementary analysis– Multi-dataset comparisons– Statistical summaries and basic meteorological calculations

– Our development is the “Community Data Portal”

Data Analysis

• Tools– NCAR Command Language (NCL) software

• Features in brief– I/O for many ‘standard’ data formats– Easy adaptations to read any format– 100’s meteorological functions– “Publication quality” graphics

– The CDP is capable of analysis• NCL is one of several middleware packages

Community Sharing

• Support for the scientist– A place to distribute new data results

• Possibly with authentication and authorization control

• E.g. model outputs

– Spin off benefit• New data resources for the archive• Many users can then use new product

NCEP Operational Analyses blended with QSCAT Satellite data

Wind Stress Curl, 01/24/2000 1800 UTC

a) NCEP Operational ONLY

b) NCEP + QSCAT swaths

c) OI blend of NCEP + QSCAT

Blending by Colorado Research Associates

We archive all three products.

a b

c

Key Steps of Scientific Investigations

• Formulate the questions and review the state of understanding

• Search and discover data• Access data• Analyzes data• Community sharing and archive • Document new understandings