Scientific Investigations; Support from Research Data Archives for Computing in Atmospheric Sciences 2001 29 October, 2001 Steven Worley National Center.
Post on 25-Dec-2015
214 Views
Preview:
Transcript
Scientific Investigations; Support from Research Data
Archivesfor
Computing in Atmospheric Sciences 2001
29 October, 2001Steven Worley
National Center for Atmospheric ResearchScientific Computing Division
Key Steps of Scientific Investigations
• Formulate the questions and review the state of understanding
• Search and discover data• Access data• Analyzes data• Community sharing and archive • Document new understandings
Search and Discover Data
• How? Web based Information Server• Salient Features
– 2.5K + html pages (metadata)– All datasets are described (500+)– Location of all data files in MSS– Higher level information
• Catalogs• Project specific descriptions
Always current dataset descriptions
Dataset Page
• Title and Brief description
• Systematic Navigation
• Metadata highlights
• Period of Record
• Usage
• Variables
• Related Sites (NOAA)
• Contact Person
• Related Datasets
Brief Archive History and Specifications
• Started in middle 1960’s, (35 years)
• Managed by nine people
• 211K data files
• 17 TB in a MSS
• 530 datasets – all sizes
Global Observations
P.O.R # Yrs Incep.
Date
Comments
Rawinsondes 1946-
on
55 1967 Upper Air
Pibals 1942-
on
59 1973 Upper Air, wind
Aircraft 1947-
on
52 1973 USAF and
Commer.
Sat. cloud wind
drift
1967-
on
34 1973 GOES and GTS
Satellite
Soundings
1969-
92
25 1973 TOVS +
irradiance
Surface Synoptic 1948-
on
53 1975 some much older
Ocean Surface 1794-
on
203 1981 COADS
Usages:
• Input for global atmospheric reanalysis
• Basic long term climate assessment and case studies
Operational and Composite Analyses
U.S. Analyses for the N. H. (Early Operational outputs and composites) P.O.R. Comments
Daily SLP Analysis 1889-on Composite of data sources, 2 x daily later period
Selected Early Analyses 1946,1950 - on 700mb, 500mb, 300mb NMC Oper. Analysis 1962-on Z &T @ 10mb – sfc. (11 lev)
Global Operational Analyses NCEP/NMC 1976-on Many levels and variables ECMWF 1980-on Many levels and variables
Special Analyses Australian 1972-1992 Discontinued FNOC (U.S. Navy) 1973-1993 Discontinued
• Daily SLP is a small but very popular dataset, e.g. NAO evaluations
• Two main operational centers provide the best current analyses
ECMWF Global Operational Analyses Data Product Period of
Record Temporal Res.
Spatial Res. (dg)
Update Cycle
# Levs.
# Vars.
Major Variables
Upper Air 1985- 06/ 2001
6 hr ~1.125 6 mn 21 8 z,t,wind,rh
Surface 1985- 06/ 2001
6 hr ~1.125 6 mn 1 47 p,t,wind,soil.t, soil.moist.
Supplemental 1985- 06/ 2001
6 hr ~1.125 6 mn 16 rad.,stress,heat.flux, clouds
Extension 1991- 06/ 2001
6 hr ~1.125 6 mn 18 precip,heat.flux
Sf c/ Up.Air Low Resolution
1985- 06/ 2001
12 hr 2.5 1 mn 21+ 14 sf c.t,sf c. p,z,t,wind,rh
Sf c/ Up.Air †
Low Resolution
1985- 06/ 2001
1 mn 2.5 ~1 mn 21+ 14 sf c.t,sf c. p,z,t,wind,rh
† Computed by the SCD/ DSS
Key Aspects• Medium size archive – 170 Gigabytes• multi-(product, temporal res., spatial res.) - complex
Concerns;
• Restricted distribution• U.S. non-profits and UCAR members only• Need online authentication and authorization for easy access
NCEP Operational Analyses Data Product Period of
Record Temporal Res.
Spatial Res. (dg)
Update Cycle
# Levs.
# Vars.
Major Variables
Final Analysis Global 2.5
1976- 08/ 2001
6 hr 2.5 1 mn 11+ 15 z,t,wind,rh, sf c.t, sf c.p
Final Analysis Global 1.0
09/ 1999 - today
6 hr 1.0 Daily (FTP)
26+ 71 z,t,wind,rh,vorticity sf c.t,sf c.p
ETA-3D N. America
05/ 1995- 07/ 2001
6 hr 40 (km) 1 mn 26+ 5 z,t,wnd,sh, precip(f orecast)
ETA-Surface N. America
05/ 1995- 07/ 2001
6 hr 40 (km) 1 mn 12 wind,sf c.p,sf c.t, soil.t,soil.p
LFM (1971-1995) and NGM (1984-cont), N. America, 190km and 6 hr resolution, are available but ETA is considered a superior replacement.
Highlights
• Frequent updates to FNL, 1º, daily via FTP
• High resolution N. America product, ETA at 40km
• No distribution restrictions or cost
Reanalyses
P.O.R # Yrs Incep. Date
NCEP/NCAR Reanalysis
I
1948-06/2001 53 1994
ECMWF ERA-15 1979-1993 15 1994
NCEP Reanalysis II 1979-06/2001 22 1998
Notes:
• ERA-15 is finished, ERA-40 is running now
• NCEP II, primarily experimental run
NCEP/NCAR Global Atmospheric Reanalysis Data Product Period of
Record Temporal Res.
Spatial Res. (dg)
Update Cycle
# Levs.
# Vars.
Major Variables
Analysis on Pressure Sf c.
1948- 6/ 2001
6 hr 2.5 1-2 mn 17 7 u,v,z,t,rh
Analysis on Sigma Sf c.
1948- 6/ 2001
6 hr 192x94 Gaussian
1-2 mn 28 6 u,v,t,sph,rel.vort,
Analysis on Theta Sf c.
1948- 6/ 2001
6 hr 2.5 1-2 mn 11 10 N**2, ab.vort,u,v, t,rh,pot.vort
Surf ace Flux Fields
1948- 6/ 2001
6 hr 2.5 1-2 mn 12 Clouds, rad.flx, soil.moist,heat.flx precip
Monthly Mean Anal. P. Sf c.
1948- 2000
1 mn 2.5 1-2 mn 17+ 36 u,v,z,t,rh
CD-ROMS 1953- 1999
12 hr, 1 day, 1mn
2.5 3-6 12 u,v,z,t,rh,heat.flx, rad,flx,precip
model qc’ed observations are returned f orecasts, once every 5 days a f orecast fi elds, 6 hr, available out to 8 days
Outstanding Features• Three different coordinate surfaces• Very long analysis, 2+ Terabytes size• Unrestricted distribution• CD-ROMS are very popular
Countries Receiving Reanalysis CDROMs
Highlights• Over 8900 CDROMs 1997-09/2001
• Recipients; U.S. 46%, Japan 11%, (Canada, UK) 4%, (Germany, India) 3%, (Australia, S.Korea, Spain, Mexico, Norway, Russia, France) 2%
Reanalysis Users for 2001 (4th qtr estimated)
209 From the MSS [157 Jan.-Sep.] 47 On CDROM [35] 48 Custom data orders on FTP or Tape [36] 540 From the online server [406]
844 Total Served
0
50
100
150
200
250
Un
iqu
e U
sers
1995 1996 1997 1998 1999 2000 2001
Years
NCEP/NCAR Renalysis from the MSS
Estimate
Other Users
Univ. Users
NCAR Users
Reanalysis Data Distributed for 2001 (4th qtr estimated)
• 9616 GB from the MSS [7230 GB Jan.-Sep.]
• 808 GB On CD-ROM [935, @650Mb/CDROM]• 1383 GB Custom orders, FTP and tape [1040]• 88 GB From the online server [66 GB]
11895 GB, 11.9 TB Total
0
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
Dat
a A
mo
un
t (G
B)
1995 1996 1997 1998 1999 2000 2001
Years
NCEP/NCAR Reanalysis from the MSS
Estimate(GB)
Other (GB)
Univ. (GB)
NCAR (GB)
GCIP Model Data Center Collection
High resolution atmospheric models focused on energy and hydrology cycles.
GCIP: GEWEX Continental-Scale International Project / GEWEX : Global Energy and Water Cycle Exper.
• Critical data for N. American mesoscale studies• Complete archive is about 1 Terabyte
Eta –NCEP 3 hr 40 km25 lvs
5/1995 – 7/2001
MAPS – FSL NOAA
3 hr 40 km5 lvs
8/1996 - 7/2001
GEM – Canadian
6 hr 41 km28 lvs
4/1997 – 6/2001
Ocean Model Data
MICOM; Miami Isopynic Coordinate Ocean Model, 1/12th degree 70N to 28 S, 16-20 layers
COADSClim. Forcing
6 yrs 305 Gigabytes
ECMWFClim. Forcing
2 yrs 164 Gigabytes
ECMWF Daily Forcing
5 yrs 415 Gigabytes(1979-1983)
University of Miami
6-yr Mean T at 5 meters
Dataset Sizes and Scales
• Today – ~ 800 Unique users– ~ 12 Terabytes data transferred– 2 Terabyte dataset size– Example: NCEP/NCAR Reanalysis
• Near Future Excludes TB-PB Level 0 and 1 satellite and the super
scale experimental models– Numbers of Users, ~ same– Data transferred, 5x to 10x more ?– Dataset size, 2-20 TB– Examples:
• Ocean and Atmosphere models • ECMWF Reanalysis (ERA40)
Access to Data
Methods• NCAR computers
– From the local MSS
• Web data server • Custom data packages
– by request (FTP, tape, CDROM)
Users • World class programmer• Research Scientist• Graduate Students• Undergraduate Students
Data Access in the future
• Do we continue doing what we are doing?
“Absolutely”Why? It Works– Over 1000 users annually
• Very diverse skills
– The archive is a heterogeneous collection• Many formats (ASCII, Binary, GrIB, BUFR, netCDF, HDF)• Many sizes (1 MB to 2 TB)
– Capable of serving large and small projects
Maintain a variety of flexible methods
Data Access in the future
• Keys to handling future larger collections– Plan to create useful data products
• Condensed datasets from high resolution output• Group most popular variables products together
– Serve many, e.g. CDROMS and WWW
– Continue to develop emerging online data systems
• User driven subset selection with graphics and data download options
• Server-side elementary analysis– Multi-dataset comparisons– Statistical summaries and basic meteorological calculations
– Our development is the “Community Data Portal”
Data Analysis
• Tools– NCAR Command Language (NCL) software
• Features in brief– I/O for many ‘standard’ data formats– Easy adaptations to read any format– 100’s meteorological functions– “Publication quality” graphics
– The CDP is capable of analysis• NCL is one of several middleware packages
Community Sharing
• Support for the scientist– A place to distribute new data results
• Possibly with authentication and authorization control
• E.g. model outputs
– Spin off benefit• New data resources for the archive• Many users can then use new product
NCEP Operational Analyses blended with QSCAT Satellite data
Wind Stress Curl, 01/24/2000 1800 UTC
a) NCEP Operational ONLY
b) NCEP + QSCAT swaths
c) OI blend of NCEP + QSCAT
Blending by Colorado Research Associates
We archive all three products.
a b
c
top related