1 Dr. Frederica Darema Senior Science and Technology Advisor Director, Next Generation Software Program NSF Dynamic Data Driven Application Systems (DDDAS) A new paradigm for applications/simulations and measurement methodology … and how it would impact CyberInfrastructure!
71
Embed
1 Dr. Frederica Darema Senior Science and Technology Advisor Director, Next Generation Software Program NSF Dynamic Data Driven Application Systems (DDDAS)
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
1
Dr. Frederica DaremaSenior Science and Technology Advisor
Director, Next Generation Software Program
NSF
Dynamic Data Driven Application Systems(DDDAS)
A new paradigm for applications/simulations
andmeasurement methodology
… and how it would impact CyberInfrastructure!
2
Measurements ExperimentField-Data
User
Theory
(First P
rincip
les)
Simula
tions
(Math
.Modeli
ng
Phenomenol
ogy)Experiment
MeasurementsField-Data
(on-line/archival)User
Theory
(First P
rincip
les)
Simula
tions
(Math
.Modelin
g
Phenomenolo
gy
Observ
ation M
odeling
Design)
OLD
(serialized and static)
NEW PARADIGM
(Dynamic Data-Driven Simulation Systems)
Challenges:Application Simulations DevelopmentAlgorithms Computing Systems Support
Dynam
ic
Feed
back
& C
ontro
l
Loop
What is DDDAS(Symbiotic Measurement&Simulation Systems)
3
Examples of Applications benefiting from the new paradigm• Engineering (Design and Control)
– aircraft design, oil exploration, semiconductor mfg, structural eng– computing systems hardware and software design
(performance engineering)
• Crisis Management and Environmental Systems– transportation systems (planning, accident response)– weather, hurricanes/tornadoes, floods, fire propagation
MSTAR (DARPA)(Moving and Stationary Target Acquisition and Recognition)
Predict Extract
Focus ofAttention
SAR Image &Collateral Data- DTED, DFAD- Site Models- EOSAT imagery
ROAD
TREES
GRASS
H2O
Regions of Interest (ROI)
SegmentedTerrain Map
Indexing
...Search Tree
...Index Database(created off-line)
Search
Target & Scene Model Database(created off line)
Match
LocalScene Map
GRASS
TREES
TR
EE
S
ROI Hypothesis
x
y BMP-2
Shadow (?)
Match Results
Score = 0.75
Form Associations Refine Pose & Score
x1,y1, x2,y2,
Analyze Mismatch
TreeClutter
GroundClutter
ShadowObscuration ?
Feature-to-Model Traceback
Task Predict Task Extract
LocalScene Map
GRASS
TREES
TR
EE
S
ROI Hypothesis
x
y BMP-
2
Target& ClutterDatabase
StatisticalModel
CAD
Semantic Tree
ClutterDatabase
11
Application
Integration
Interoperability
The e-Business / (CIM, CIE)
Enterprise Messaging
ManufacturingProduct DBsInventory Shipping
Order Processing Customer ServiceSales Management
Process Coordination Management & Monitoring
Data
Integration
Interoperability
DistributorChannel
Businessto
Business
Webe-commerce
Businessto
Customer
Mobile WorkersKnowledge WorkersBusiness Communications
12
Compare with Classical (Old) Supply Chain
PartsSupplier
Manufacturing Distribution RetailCustomer
Customer
Manufacturing Distribution RetailCustomer
Customer
Manufacturing Distribution RetailCustomer
Customer
PartsSupplier
Transportation Supplier
13
Some Technology Challenges in Enabling DDDAS
• Application development– interfaces of applications with measurement systems– dynamically select appropriate application components– ability to switch to different algorithms/components
depending on streamed data
• Algorithms – tolerant to perturbations of dynamic input data– handling data uncertainties
• Systems supporting such dynamic environments– dynamic execution support on heterogeneous
environments– Extended Spectrum of platforms: assemblies of Sensor
Networks and Computational Grid platforms– GRID Computing, and Beyond!!!
14
What is Grid Computing?
coordinated problem solving on dynamic and heterogeneous resource
assemblies
QuickTime™ and a decompressor
are needed to see this picture.
QuickTime™ and a decompressor
are needed to see this picture.
IMAGING INSTRUMENTS
COMPUTATIONALRESOURCES
LARGE-SCALE DATABASES
DATA ACQUISITION ,ANALYSIS
ADVANCEDVISUALIZATION
Example: “Telescience Grid”, Courtesy of Ellisman & Berman /UCSD&NPACI
15
DynamicallyLink
&Execute
The NGS Program developsTechnology for integrated feedback & control Runtime Compiling System (RCS) and Dynamic Application
CompositionApplication
Model
Application Program
ApplicationIntermediate
Representation
CompilerFront-End
CompilerBack-End Performance
Measuremetns&
Models
DistributedProgramming
Model
ApplicationComponents
&Frameworks
Dynamic AnalysisSituation
LaunchApplication (s)
Distributed Platform
Ada
ptab
leco
mpu
ting
Syst
ems
Infr
astr
uctu
re
Distributed Computing Resources
MPP NOW
SAR
tac-com
database
firecntl
firecntl
alg accelerator
database
SP
….
16
Some more Challenges on Applications Development
Issues• Handling Data Streams in addition to Data Sets• Handling different data structures – semantic
information• Interfaces to Measurement Systems
- Interactive Visualization and Steering • Standards for data exchange• Combining Local and Global Knowledge• Model Interactions• Application control of measurement systems• Dynamic Application Composition and Runtime
support(Examples from ITR supported efforts)
17
Important Point:
DDDAS is not just DATA ASSIMILATION!!!
• Data Assimilation compares/corrects specific calculated points with experiments, rather than dynamically as need
• Data Assimilation does not include the notion of the simulation/application controlling the measurement process
Rather…Data Assimilation techniques can be used in
• Application Composition Frameworks and…. • Interoperability extended to include measurements• Data Models and Data Management
– Extend the notion of Data Exchange Standards (Applications and Measurements)
19
Additional Considerations/Requirements on Hardware and Software Systems
• Extended Spectrum of platforms – Assemblies of Computational Grid and Sensor
Networks platforms
• Systems Architectures including Measurement Systems
• Programming Environments • Application, System, and Resource Management• Models of the Computational Infrastructure • Security and Fault Tolerance• DDDAS will accentuate and create the need for
advances in such areas
20
Perform
ance
Engineering
Dynamic
Compilers
&
Application
Composition
Dynamic Data-Driven
Applicatio
n Systems
--
Symbiotic
Measurement&Simulatio
n
Systems
TowardsEnabling DDDAS
NGS Program
Today’s Grid
Environments:
“Users
shouldn’t
Have to be Heroes
to Achieve Grid
Program Perform
ance”
and... beca
use heroism
is not e
nough
21
Impact to CyberInfrastructure
• The CyberInfrastructure that will result when thinks of the present paradigm of (disjoint) simulations and measurements will be different than the CyberInfrastructure needed to support DDDAS
• For example, bandwidth requirements, resource allocation and other middleware and systems software policies, prioritization, security, fault tolerance, recovery, QoS, etc…, will be different when one needs to guarantee data streaming to an executing simulation or control of measurement process
22
Why Now is the Time for DDDAS
• Technological progress has prompted advances in some of the challenges– Computing speeds advances (uni- and multi-
– Systems Software– Applications Advances (parallel & grid computing)
– Algorithms advances (parallel &grid computing, numeric and non-numeric techniques: dynamic meshing, data assimilation)
• Examples of efforts in: – Systems Software– Applications– Algorithms
23
Agency Efforts• NSF
– NGS: The Next Generation Software Program (1998- )• develops systems software supporting dynamic resource execution
– Scalable Enterprise Systems Program (1999, 2000-2003)• geared towards “commercial” applications (Chaturvedi example)
– ITR: Information Technology Research (NSF-wide, FY00-04)• has been used as an opportunity to support DDDAS related efforts• In FY00 1 NGS/DDDAS proposal received; deemed best, funded• In FY01, 46 ~DDDAS pre-proposals received; many meritorious;
24 proposals received; 8 were awarded• In FY02, 31 ~DDDAS proposals received; 8(10) awards• In FY02, so far: received 35 (“Small” ITR) proposals ~DDDAS;
more expected in the “Medium ITR” category -
– Gearing towards a DDDAS program• expect participation from other NSF Directorates
• Looking for participation from other agencies!
24
“~DDDAS” proposals awarded in FY00 ITR Competition
• Pingali, Adaptive Software for Field-Driven Simulations
25
“~DDDAS” proposals awarded in FY01 ITR Competition
• Biegler – Real-Time Optimization for Data Assimilation and Control of Large Scale Dynamic Simulations
• Car – Novel Scalable Simulation Techniques for Chemistry, Materials Science and Biology
• Knight – Data Driven design Optimization in Engineering Using Concurrent Integrated Experiment and Simulation
• Lonsdale – The Low Frequency Array (LOFAR) – A Digital Radio Telescope
• McLaughlin – An Ensemble Approach for Data Assimilation in the Earth Sciences
• Patrikalakis – Poseidon – Rapid Real-Time Interdisciplinary Ocean Forecasting: Adaptive Sampling and Adaptive Modeling in a Distributed Environment
• Pierrehumbert- Flexible Environments for Grand-Challenge Climate Simulation
• Wheeler- Data Intense Challenge: The Instrumented Oil Field of the Future
26
“~DDDAS” proposals awarded in FY02 ITR Competition
• Carmichael – Development of a general Computational Framework for the Optimal Integration of Atmospheric Chemical Transport Models and Measurements Using Adjoints
• Douglas-Ewing-Johnson – Predictive Contaminant Tracking Using Dynamic Data Driven Application Simulation (DDDAS) Techniques
• Evans – A Framework for Environment-Aware Massively Distributed Computing
• Farhat – A Data Driven Environment for Multi-physics Applications
• Guibas – Representations and Algorithms for Deformable Objects
• Karniadakis – Generalized Polynomial Chaos: Parallel Algorithms for Modeling and Propagating Uncertainty in Physical and Biological Systems
• Oden – Computational Infrastructure for Reliable Computer Simulations
• Trafalis – A Real Time Mining of Integrated Weather Data
Computing portals which will enable reservoir simulation and geophysical calculations to interact dynamically with the data and with each other and which will provide a variety of visual and quantitative tools. Test data provided by oil and service companies
THE INSTRUMENTED OILFIELD
III. IT Technologies:
Data management, data visualization, parallel computing, and decision-making tools such as new wave propagation and multiphase, multi- component flow and transport computational portals, reservoir production:
35
Data Management and Manipulation
Visualization
Field Measurements
Simulation Models
Reservoir MonitoringField Implementation
Data Analysis
Production ForecastingWell Management
ReservoirPerformance
Data Collections from Simulations and Field Measurements
Economic Modeling and Well Management
Multiple Realizations
36
ITR Project
A Data Intense Challenge:
The Instrumented Oilfield of the Future
II. Industrial Support (Data):
i. British Petroleum (BP)
ii. Chevron
iii. International Business Machines (IBM)
iv. Landmark
v. Shell
vi. Schlumberger
37
Dynamic Contrast ImagingDCE-MRI (Osteosarcoma)
38
Dynamic Contrast Enhanced Imaging
• Dynamic image quantification techniques– Use combination of static and dynamic image
information to determine anatomic microstructure and to characterize physiological behavior
– Fit pharmacokinetic models (reaction-convection-diffusion equations)
– Collaboration with Michael Knopp, MD
39
• Dynamic image registration– Correct for patient tissue motion during
study– Register anatomic structures between
studies and over time
• Normalization– Images acquired with different patterns
spatio-temporal resolutions– Images acquired using different imaging
modalities (e.g. MR, CT, PET)
Dynamic Contrast Enhanced Imaging
40
Clinical Studies using Dynamic Contrast Imaging
• 1000s of dynamic images per research study
• Iterative investigation of image quantification, image registration and image normalization techniques
• Assess techniques’ ability to correctly characterize anatomy and pathophysiology
• “Ground truth” assessed by– Biopsy results– Changes in tumor structure and activity over
time with treatment
41
1370
1421
1438
prior to therapy
after 2 cycles
after 4 cycles
1370
14211421
1438
Knopp M, OSU Radiology / dkfz
42
Software Support
• Component Framework for Combined Task/Data Parallelism – Use defines sequence of pipelined components
-- “filter group” – User directive tells preprocessor/runtime
system to generate and instantiate copies of filters
– Many filter groups can be simultaneously active– Integration proceeding with Globus/Network
Weather Service
43
Virtual Microscope
44
Adaptive Software Project
•Cornell University
–CS department (Keshav Pingali)
–Civil and Environmental Engineering (Tony Ingraffea)
•Mississippi State University
•University of Alabama, Birmingham
–Mechanical and Aerospace (Bharat Soni)
•College of William and Mary
•Ohio State University
•Clark-Atlanta University
45
SCOPE of ASP
• Implement a system for multi-physics multi-scale adaptive CSE simulations– computational fracture mechanics– chemically-reacting flow simulation
• Understand principles of implementing adaptive software systems
Cracks: They’re Everywhere!
46
ASP Test Problem
47
Problem description
• Regenerative cooling nozzle from NASA– Simplified geometry
• Chemically-reacting flow in interior of pipe• Nozzle is cooled by fluid-flow in eight smaller
channels at periphery of pipe• Problem:
– simulate flows– determine crack growth– couple the multi-physics models – When successful add the ability to inject
monitoring measurements
48
Understanding fracture
• Wide range of length and time scales• Macro-scale (1in- )
– components used in engineering practice• Meso-scale (1-1000 microns)
– poly-crystals• Micro-scale (1-1000 Angstroms)
– collections of atoms
10-3 10-6 10-9 m
49
Chemically-reacting flows
• MSU/UAB expertise in chemically-reacting flows
• LOCI: system for automatic synthesis of multi-disciplinary simulations
of physical, biological, acoustical phenomena• Extend state-of-the-art via feedback from
acoustics to physical&biological oceanography
• Application in fisheries management, but also in oil-slick containment
53
Interdisciplinary Ocean Science
54
Development ofa General Computational Framework
for the Optimal Integration of Atmospheric Chemical Transport Models and
Measurements Using Adjoints
Greg Carmichael (Dept. of Chem. Eng., U. Iowa)Adrian Sandu (Dept. of Comp. Sci., Mich. Inst.
Tech.)John Seinfeld (Dept. Chem. Eng., Cal. Tech.)Tad Anderson (Dept. Atmos. Sci., U. Washington)Peter Hess (Atmos. Chem., NCAR)Dacian Daescu (Inst. of Appl. Math., U. Minn.)
55
Application: The Design of Better Observation Strategies to Improve Chemical Forecasting Capabilities.
Example flight path of the NCAR C-130 flown to intercept a dust storm in East Asia that was forecasted using chemical models as part of the NSF Ace-Asia (Aerosol
Characterization Experiment in Asia) Field ExperimentWill help to Better Determine Where and When to Fly and How to More
Shown are measured CO along the aircraft flight path, the brown isosurface represents modeled dust (100 ug/m3), and the blue isosurface is CO (150 ppb) shaded by the fraction due to biomass burning
(green is more than 50%).
56
Project Goal:
To develop
general computational tools, and associated software,
for assimilation of atmospheric chemical and optical measurements into chemical transport models (CTMs).
These tools are to be developed so that users need not be experts in adjoint modeling and optimization theory.
57
Approach: •Develop novel and efficient algorithms for 4D-data assimilation in CTMs; Develop general software support tools to facilitate the construction of discrete adjoints to be used in any CTM; •Apply these techniques to important applications including: (a) analysis of emission control strategies for Los Angeles; (b) the integration of measurements and models to produce a consistent/optimal analysis data set for the AceAsia intensive field experiment; (c) the inverse analysis to produce a better estimate of emissions; and (d) the design of observation strategies to improve chemical forecasting capabilities.
58
Data Assimilation for Chemical Models
Solid lines represent current capabilities Dotted lines represent new analysis capabilities
Future: enable DDDAS capabilities
59
General Software Tools Framework to Facilitate the Close Integration of Measurements and Models
The framework will provide tools for: 1) construction of the adjoint model; 2) handling large datasets; 3) checkpointing support; 4) optimization; 5) analysis
of results; 6) remote access to data and computational resources.
properties• Sensitivity/failure analysis• Gaussian and non-Gaussian processes• Polynomial Chaos vs. Monte Carlo• Stochastic spectral/hp element methods
Irreducible versus epistemic uncertainty
“…Because I had worked in the closest possible ways with
physicists and engineers, I knew that our data can never be precise…”
Norbert Wiener
Slides Courtesy of Karniadakis/Brown
61
Partially Correlated non-Uniform Random Inflow
Vorticity: Regions of Uncertainty
•Pressure
•Deterministic
•Stochastic
62
Non-uniform Gaussian Random BC
Umean along centerline Vmean along centerline
• Exponential correlationb/xx2
2121e)x,x(C
1.0• Stochastic input:
• 2D K-L expansion
• 4th-order Hermite-Chaos expansion
• 15-term expansion
63
Non-uniform Exponential Random BC
Umean along centerline Vmean along centerline
• Exponential correlationb/xx2
2121e)x,x(C
1.0• Stochastic input:
• 2D K-L expansion
• 4th-order Laguerre-Chaos expansion
• 15-term expansion
64
Research Opportunities in UncertaintyResearch Opportunities in Uncertainty
UUncertainty analysis is a fertile and much needed area for inter-disciplinary researchEEstimates of uncertainties in model inputs are desperately needed
Uncertainty Uncertainty Ignorance Ignorance
65
What about Industry &DDDAS• Industry has history of
– forging new research and technology directions and – adapting and productizing technology which has demonstrated promise
• Need to strengthen the joint academe/industry research collaborations; joint projects / early stages
• Technology transfer– establish path for tech transfer from academic research to industry– joint projects, students, sabbaticals (academe <----> industry)
• Initiatives from the Federal Agencies / PITAC• Cross-agency co-ordination • Effort analogous to VLSI, Networking, and
Parallel and Scalable computing• Industry is interested in DDDAS
66
i
e
nt
gratio
n
Research and Technology Roadmap (emphasis on multidisciplinary research)