1 July 16, 2007 LeRoy Spayd Chief, Operations and Requirements Division Office of Climate, Water, and Weather Services NOAA’s National Weather Service Unidata Policy Committee Unidata Policy Committee NOAA/NWS Update NOAA/NWS Update
1
July 16, 2007LeRoy Spayd
Chief, Operations and Requirements DivisionOffice of Climate, Water, and Weather Services
NOAA’s National Weather Service
Unidata Policy CommitteeUnidata Policy CommitteeNOAA/NWS UpdateNOAA/NWS Update
2
Outline
• NCEP Model plans
• CONDUIT
• MADIS – surface data
• NOAAPort plans
• AWIPS II plans
• NOAA Satellite plans
3
Trade-Offs in Operational NWP
• Where to apply increased computer power• Physics – complexity, frequency of calls …
• Dynamics – semi-Lagrangian, finite volume …
• Resolution – seems more is always better
• Data assimilation – 3+DVar, 4DVar, ETKF …
• Update frequency – every 6, 3 or 1 hourly …
• Size of ensemble – is bigger always better …
• Reforecasting +/or Reanalysis
• Many of these decisions have been dictated by The Jigsaw Puzzle, e.g. no room for 4DVar.
4
The Jigsaw PuzzleNCEP Production Suite
Weather, Ocean & Climate Forecast SystemsVersion 3.1 October 20, 2004
0102030405060708090
100
0:00 0:30 1:00 1:30 2:00 2:30 3:00 3:30 4:00 4:30 5:00 5:30 6:00
6 Hour Cycle
Perc
ent U
sed
RR/RTMAFireWXWAVESHUR/HRWGFSfcstGFSanalGFSensNAMfcstNAManalSREFAir QualityGlblOceanMonthlySeasonal
RUC GFS Anl Hur/HRW
GFS FcstNAM Fcst
NAM Anl Waves
SREF GENS
5
NCEP Production SuiteWeather, Ocean & Climate Forecast Systems
Version 3.0 April 9, 2004
0
20
40
60
80
100
0:00 0:30 1:00 1:30 2:00 2:30 3:00 3:30 4:00 4:30 5:00 5:30 6:00
6 Hour Cycle: Four Times/Day
Perc
ent U
sed
RUCFIREWXWAVESHUR/HRWGFSfcstGFSanalGFSensETAfcstETAanalSREFAir QualityOCEANMonthlySeasonal
Reforecast
NA
Manal
CFS
SREF NAM
GFS
WAV
HUR
Next Generation PrototypePhase 1 - 2009
3-hourly GDAS (2)1-hourly RDAS (6)
GENS/NAEFS
RTOFSAQ
GFS A
nal
NCEP Production SuiteWeather, Ocean, Land & Climate Forecast Systems
Added• 1-Hourly RDAS• 3-Hourly GDAS• Reanalysis/
Reforecast
Computing factor: 3
6
NCEP Production SuiteWeather, Ocean & Climate Forecast Systems
Version 3.0 April 9, 2004
0
20
40
60
80
100
0:00 0:30 1:00 1:30 2:00 2:30 3:00 3:30 4:00 4:30 5:00 5:30 6:00
6 Hour Cycle: Four Times/Day
Perc
ent U
sed
RUCFIREWXWAVESHUR/HRWGFSfcstGFSanalGFSensETAfcstETAanalSREFAir QualityOCEANMonthlySeasonal
Reforecast
GFS A
nal
NA
MA
nal
CFS & MFS
GFS
WAV
HUR
GENS/NAEFS
Next Generation PrototypePhase 2 - 2011
GDAS
SREF
RDAS
RTOFSHydro / NIDIS AQ
NAM
NCEP Production SuiteWeather, Ocean, Land & Climate Forecast Systems
AQ
Added• Hydro/NIDIS
productsMoved
• GFS ½ h earlierExpanded
• Hurricane & waveproductsIncorporated
• Multi-domainrapid updating
Computing factor: 9
7
NCEP Production SuiteWeather, Ocean & Climate Forecast Systems
Version 3.0 April 9, 2004
0
20
40
60
80
100
0:00 0:30 1:00 1:30 2:00 2:30 3:00 3:30 4:00 4:30 5:00 5:30 6:00
6 Hour Cycle: Four Times/Day
Perc
ent U
sed
RUCFIREWXWAVESHUR/HRWGFSfcstGFSanalGFSensETAfcstETAanalSREFAir QualityOCEANMonthlySeasonal
Reforecast
GFS A
nal
NA
MA
nal
CFS & MFS
GFS
WAV
HUR
GENS/NAEFS
Next Generation PrototypePhase 3 - 2013
GDAS
SREF
RDAS
RTOFS
NAM
NCEP Production SuiteWeather, Ocean, Land & Climate Forecast Systems
AQHydro / NIDIS/FFAQ
Computing factor: 27
Added• Flash flood
productsMoved
• SREF concurrentto NAMExpanded
• Reforecast capability
8
CFSMFS
NCEP Production SuiteWeather, Ocean & Climate Forecast Systems
Version 3.0 April 9, 2004
0
20
40
60
80
100
0:00 0:30 1:00 1:30 2:00 2:30 3:00 3:30 4:00 4:30 5:00 5:30 6:00
6 Hour Cycle: Four Times/Day
Perc
ent U
sed
RUCFIREWXWAVESHUR/HRWGFSfcstGFSanalGFSensETAfcstETAanalSREFAir QualityOCEANMonthlySeasonal
WAV
CFS & MFS
GENS/NAEFSGFS
Next Generation PrototypePhase 4 - 2015
Regional
Rap Refresh
GlobalHUR
SREF
Reforecast
Hydro / NIDIS/FF
Hydro
NAM
GDAS
RDAS
RTOFS RTOFSAQ
NCEP Production SuiteWeather, Ocean, Land & Climate Forecast Systems
AQ
Computing factor: 81
Added• Hourly GDAS
Moved• GFS concurrent
to NAM & SREFExpanded
• Hurricanecapability (hires)
9
CFS & MFS
NCEP Production SuiteWeather, Ocean & Climate Forecast Systems
Version 3.0 April 9, 2004
0
20
40
60
80
100
0:00 0:30 1:00 1:30 2:00 2:30 3:00 3:30 4:00 4:30 5:00 5:30 6:00
6 Hour Cycle: Four Times/Day
Perc
ent U
sed
RUCFIREWXWAVESHUR/HRWGFSfcstGFSanalGFSensETAfcstETAanalSREFAir QualityOCEANMonthlySeasonal
CFSMFS
WAVGFSRegional
Rap Refresh
GlobalSREFReforecast
Hydro
NAM
GDAS
RDAS
RTOFS
RTO
FS
CFS & MFSAQ Hydro / NIDIS/FF AQ
GENS/NAEFS
>100% of 2015 computing
Next Generation PrototypeFinal – 2017+
NCEP Production SuiteWeather, Ocean, Land & Climate Forecast Systems
GLOBAL NGATS
HU
R
Computing factor: > 240
ECOSYSTEMS
SPACE WEATHER
HENS
10
WRF Lessons Learned
• The burden of community modeling involvement by any institution is substantial but especially so for an operational one
• Nearly all NMM enhancements must be done or finished by NCEP (DTC may off-load some of this … eventually)
• Community directions and rate of evolution are not always in sync with operational needs
• Operational needs are not high on the priority list of community developers (unless operations can provide a big part of the $upport)
• Community needs are often driven by foreign partners and/or the funding sources
11
The Downsides of WRF & Why NCEP is Moving to ESMF
• WRF: Must take / use all of it• WRF: Not most efficient• WRF: Highly complex, deeply layered code, not well suited for nearly
continuous development & enhancement of NCEP Operational environment• ESMF: Can choose only the parts you want and NCEP is choosing to take
just a minimum• ESMF: Vast majority of code is being written by NCEP with simplicity, clarity
and efficiency as guides
12
Community-basedESMF Development
• Strategy and roles: • Focus on single component instead of entire model system• Collaborative, not competitive• NCEP/EMC
– Maintains primary components for each part of Production Suite and for each application
– Supports ESMF applications that are used in operations– In collaboration with community
– Integrates new ESMF-based components into operations– Performs final testing and preparation of upgrades of supported
components in operations• Collaborators
– Provide – Component upgrades to be tested in operational setting– Institutional support for their contributed components– Diversity and expertise complementary to operations
– Work through DTC, JCSDA, CTB, etc.
13
Preparing for the Future
• Observations (number and availability)• Advanced Polar and Geostationary sounders (~100 X greater)
– NPOESS (delivered <60 minutes globally) – 2012-2015 (or later)– METOP (1-4) – 2007 – NPP (delivered 90-120 minutes globally) – 2009– GOES-R – 2013 (or later)
• Next-generation Doppler radar• Advanced post-processing techniques for multi-model ensemble (e.g. NAEFS
project)• Bias correction• 2nd moment correction• CPC “consolidation” to quantify “value-added”
• Advanced dissemination strategies• E.g. NOMADS (“Fat server/Thin Client” technology)
• Next-Generation Air Traffic-control System (NGATS)• Geographically consistent solutions• Global to terminal scales• At least hourly updating globally
14
Preparing for the Future
• Three principals for moving forward1. Maturing, ensemble-based, probabilistic systems offer the most
potential benefits across wide spectrum of forecast services
2. Ensemble compositiona. Managed component diversity
b. Components must be institutionally supported (operational or major research institution)
3. Product deliverya. Time is critical (perishable product)
b. Information availability must be maximized
15
CONDUIT
• In-Situ, Radar and Model Data Requirements will be increasing
• Need a backup capability given increasing real-time use by Partners
• More observations from mesonets and MADIS
• Higher resolution global and regional, climate, water, weather, air quality models
• Increasing reliance on ensembles (NAEFS and SREF)
• Potential IDD backup to NOAAPORT data stream
• Need increasing dialogue with NOAA and community on what CONDUIT could provide
16
NOAAPORT
• Currently 10 Megabits/sec of bandwidth
• We peak out at about 75% of this
• Plans to add bandwidth linked to Satellite programs in Budget process
• +10 MB/sec in 2010 to support NPOESS/NPP
• +20 MB/sec (40MB/sec total) in 2014 to support GOES R
17
MADIS – An Overview
• MADIS – Meteorological Assimilation Data Ingest System• Developed by FSL in 2001
• Data management system / architecture that is flexible, expandable and interoperable
• Provides government and non-government mesonet, upper-air, and coastal data and QC to NOAA and the enterprise
• Data are informed by metadata
• Transitioning MADIS to NWS operations will provide 24x7 maintenance support with offsite system backup• Leverages NOAA’s extensive data management infrastructure and investment
18
MADIS Data Sources and Outputs
SFC-LANDSFC-MARINE
U/A-IN SITU
U/A-REMOTESENSING
SATELLITE
GRIDSMETADATA
MADISCollection,QC, and
Distribution
INFORMATIONBASES(QCed
DATASETS)
QUALITYCONTROL
INFO & Meta Data(Data QC
Flags)
19
MADIS Observations(as of June 13, 2007)
• Current MADIS Sites
• Current Networks > 150• Largest Networks
> Community CollaborativeRain, Hail, and Snow Network(CoCoRaHS)
> AWS Convergence Tech. Inc.> Citizen Weather Observing
Program (CWOP)> Interagency Fire Center’s
Remote Automated WeatherSystem (RAWS)
Meteorological Mesonet = 21,147Hydrological Mesonet = 6,978
Modernized COOP = 199UrbaNet = 886
Total = 29,210
20
MADIS – NWS Transition (con’t)
• Initial Operating Capability – FY09, Q1-Q2• Provides “current” functionality for real-time MADIS abilities, about 20K surface
and upper air obs baselined
• Final Operating Capability – end FY10 / early FY11• Expanded NOAA and select non-NOAA datasets, offsite backup
21
What is AWIPS Evolution?
• AWIPS Evolution• A long-term project which delivers a modern, robust software infrastructure that provides the foundation
for future system level enhancements• AWIPS II
• Implements a modern Services Oriented Architecture (SOA) infrastructure• First output of AWIPS Evolution and provides the foundation for all subsequent improvements
• AWIPS Evolution System Improvements• Integration of “orphan” systems (e.g., Weather Event Simulator)• Migration of N-AWIPS into the SOA to create a seamless weather enterprise that supports all levels of
NWS operations from National Centers to WSOs• Data Delivery Enhancements
– “Smart push-smart pull” data access– Katrina satellite WAN back up
• Integrated visual collaboration– Graphical collaboration at all levels of the weather enterprise extending to trusted external partners
• Visualization Enhancements• Information Generation Enhancements
– Re-architecture of the generation of all NWS products and services
22
AWIPS EvolutionOutcomes
• Short-term (1-3 years)• Shorten transition of research to operations
• Improve software O&M and technology refresh– Fewer DRs and TTs
– Focus on hardening and productionizing for life cycle support
• Minimize adverse impacts on operations from software and hardware upgrades
• Long-term (3-10 years)• Increase integration of AWIPS and National Center AWIPS
• Improve performance and functionality of AWIPS
• Improve collaboration at all levels of NWS operations
• Increase access to all environmental data for decision making
23
AWIPS IIRe-Architecture Approach
• Perform “black-box” conversion• Preserve existing functionality, look and feel on top of new infrastructure
• Thorough field validation and acceptance before deployment• No loss of functionality
• Deployed system current with deployed AWIPS capability (i.e., OB9)• Use open source projects - No proprietary code
• JAVA and open source projects enable AWIPS II to be platform and OS independent– No plans to move from Linux
• Objective is to make AWIPS II available for collaborative development• OS, Platform independence allows non-Linux based research to be easily
integrated into AWIPS II
24
AWIPS II Features
• AWIPS Development Environment (ADE)• Used by all AWIPS developers (National, Regional, & Local)• Developers concentrate on new capabilities, not re-implementing existing ones (i.e. screen I/O,
communications protocols, data access routines, logging routines, or other previously developed capabilities)
• Software can be developed on a variety of platforms • Robust infrastructure for improved software O&M
• Use of plug-ins: visualization extensions; new data types and transforms• System level, remediation, core services reduce system complexity• Improved support for local requirements (e.g., local apps, scripts, plug-ins)
• Common AWIPS Visualization Environment (CAVE)• Provides a common development and execution environment for AWIPS GUIs (e.g. D2D, NMAP, GFE,
etc.)• Ability to pan/zoom large data sets (Raster & Vector) with flexibility over data rendering• GIS tools• Thin Client (Web Browser) enabled
• Dynamic Load balancing• Processing dynamically allocated among available CPUs
25
AWIPS IIWhat gets us excited so far…
• Dynamic load balancing• Failover handled automatically• Enables consideration of tailored hardware configurations
• Mathematically intensive calculations handed off to the graphics card• Significant performance improvements
• Progressive disclosure of all data• Imagery via quad tree tiling, grids and observations
• Integrated thin client• Allows baseline solution to be extended to CWSUs, WSOs, and IMETs
• Integrated drawing and graphical collaboration• Tools built into the infrastructure, implemented in 2011
• Built in GIS via geotools library• Scripting level access to practically all system level services and functions• LESS CODE
• Potential order of magnitude reduction in amount of software with increase in functionality (current software has over 4,000,000 lines of code)
26
AWIPS IIWhat does it mean to you?
• Transition (Mid 2009 - mid 2010)• Limited changes during transition
• Only minor updates to products and services
• AWIPS II – 2010• More robust infrastructure
• Faster software installations – less downtime while delivering new software
• Ability to develop and implement new applications more quickly
27
AWIPS EvolutionWhat does it mean to you?
• AWIPS II – 2011• Thin client support
– Integrates CWSUs, WSOs and Incident Meteorologists• NAWIPS migrated to SOA
– One infrastructure for meteorological applications spanning operations from National Centers to WSOs
• Improved satellite back up for terrestrial network– Improves continuity of operations during Katrina-like events
• Smart push-smart pull data delivery– Improved access to broader sets of data than is currently delivered
over the SBN• Integrated graphical collaboration
– Improved coordination at all levels of NWS weather enterprise
28
AWIPS EvolutionWhat does it mean to you?
• AWIPS II – 2012-2014• Extend graphical collaboration
– NOAA offices
– Trusted external partners, e.g., DHS and Emergency Managers
• Smart push-smart pull data delivery– Extend data services to other NWS services for product delivery
• Re-architect generation of products and services– More responsive to customer requests, e.g. CAP
– Streamline process so developers and meteorologists focus on content vice format
29
The ChallengeSatellite Systems/Global Measurements
Aqua
Terra SORCE
SeaWiFS
Aura
TRMM
Meteor/SAGE
GRACE
ICESat
Cloudsat
Jason
CALIPSO
GIFTSTOPEX
Landsat
NOAA/POES
GOES-R
WindSAT
NPP
COSMIC/GPS
SSMIS
NPOESS
MSG
30
Satellite is operational beyond design life
On-orbit GOES storage
Continuity of GOES Operational Satellite Program
2009 20102004 2005 2006 2007 2008 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020
GOES 10 South America Support
GOES 11
CY
GOES EastGOES 12
GOES O
Operational
GOES R
GOES West
GOES 13 On-orbit Spare
GOES S
GOES P
31
ABI Improvements
1/5 Disc1/5 Disc
GOESGOES--I/PI/P5 Minute Coverage5 Minute Coverage
GOESGOES--RR
Full DiscFull Disc
ABI covers the earth approximately five times faster than the current Imager.
32
ABI: Improved Resolution . . .Corresponding Simulated GOES Imager Spectral Bands: Simulated “ABI” Spectral Bands:
. . . over a wider spectrum
33
Space Weather Instruments
Simulated SXI (Solar X-ray Imager) images: GOES R will produce multi-band "color" images at the same rate as GOES N/P produces single band images. (Images from NGDC website
GOES-R Space Weather Instruments– Space Environmental In Situ Suite (SEISS):
provides charged particle population measurements including proton, electron, and heavy ion fluxescontribute to the global observations used in NOAA's Space Weather Operations to continuously specify and forecast
conditions in the space environment– Solar Imaging Suite (SIS)
• Solar X-ray flux magnitude; solar EUV flux from 5 to 129 nm; coronal holes locations; solar flares; coronal mass ejections
– MagnetometerGOES-R Improvements
– Solar X-ray image dynamic range, resolution, and sensitivity – EUV measurements for improved modeling of ionosphere and thermosphere– Medium energy radiation environment responsible for spacecraft charging
Solar flares travel towards Earth at about 600,000 to 2 million MPH
Solar flares travel towards Earth at about 600,000 to 2 million MPH
34
Geostationary Lightning Mapper (GLM)
• Detects total strikes: in cloud, cloud to cloud, and cloud to ground
– Compliments today’s land based systems that only measures cloud to ground (about 15% of the total lightning)
•Increased coverage over oceans and lands– Currently no ocean coverage, and limited land coverage in dead zones
Figure from NASA.
35
NOAA Planned Missions - Polar
55
2009 20102004 2005 2006 2007 2008 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020
NOAA N-Prime
NPPNPOESS C1
PM Orbit
NOAA 18
METOP-AMETOP-B
METOP-C
NOAA 17
Mid-AM Orbit
NPOESS C2
Early-AM Orbit
DMSP 13DMSP 17
DMSP 19DMSP 20
36
Summary
• NOAA’s data production from NWP, radars, satellites, surface obswill continue to grow at a phenomenal pace
• The Nation needs a robust Unidata to fully exploit this technology investment and ensure all members of the community can work together to advance the science and improve weather, water and climate services to the public