GriPhyN/LIGO prototype 10/2001 Ewa Deelman, ISI Realizing LIGO Virtual Data Caltech: Roy Williams, Albert Lazzarini, Phil Ehrens, Kent Blackburn ISI: Laura Pearlman, Leila Meshkat, Gaurang Mehta, Carl Kesselman, Ewa Deelman UWM: Scott Koranda, Bruce Allen
Realizing LIGO Virtual Data. Caltech: Roy Williams, Albert Lazzarini, Phil Ehrens, Kent Blackburn ISI: Laura Pearlman, Leila Meshkat, Gaurang Mehta, Carl Kesselman, Ewa Deelman UWM: Scott Koranda, Bruce Allen. Outline. - PowerPoint PPT Presentation
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
GriPhyN/LIGO prototype 10/2001 Ewa Deelman, ISI
RealizingLIGO Virtual Data
Caltech: Roy Williams, Albert Lazzarini, Phil Ehrens, Kent Blackburn
ISI: Laura Pearlman, Leila Meshkat, Gaurang Mehta, Carl Kesselman, Ewa Deelman
UWM: Scott Koranda, Bruce Allen
GriPhyN/LIGO prototype 10/2001 Ewa Deelman, ISI
Outline• LIGO experiment and LDAS (LIGO Data
Analysis System)• General and LIGO specific Virtual Data
line removal,bandwidth reduction,regression analysis
wrapperAPIparallel event
detection analysis
metaDataAPIinsertion/querying of statistical and
relational results intoLIGO Database
Tables
UserAPIsUserAPIs
Job/ID
Tcl Tcl
fram
es
controlMonitorAPImonitor all LDAS APIs
jobs and Logfiles,start/stop LDAS APIscontrol and monitorparallel processes
mpiAPIstart/monitorwrapperAPI,
load balance multiplejobs
me
ssag
es
eventMonitorAPIevent summary
analysis,detection confidence,
post-detectionprocessing
messages
lightWeightAPIread & writeLIGO_LWdocuments
ilwd
Anonymous FTP, Web Server, E-Mail
LIGO_LW Fileslight-weight
(XML based)
E-MailJob Status,
location of results
LIG
OLW
Asst Mgr Asst Mgr Job/ID
LDASComponentsused by MPIMock DataChallenge
TclCommands
ilwd
emai
l
LIG
OLW
frameAPIdown select channels
&concatenate series
dataConditionAPI
statistical summary,
line removal,
bandwidth reduction,
regression analysis
managerAPI
Asst Mgr Asst Mgr
Anonymous FTP, Web Server, E-Mail
LIGO_LW Fileslight-weight
(XML based)
LIG
OL
W
LIG
OL
W
E-MailJob Status,
location of results
GriPhyN/LIGO prototype 10/2001 Ewa Deelman, ISI
Virtual Data Scenario
• (LIGO) “Conduct a pulsar search on the data collected from Oct 16 2000 to Jan 1 2001”
GriPhyNLIGO DataSpecification
LIGO DataProduct
XML XML
GriPhyN/LIGO prototype 10/2001 Ewa Deelman, ISI
The “GriPhyN box” functionality
• For each requested data value, need to– Understand the request
– Determine if it is instantiated; if so, where; if not, how to compute it
– Plan data movements and computations required to obtain all results
– Execute this plan
– Monitor progress
– Make requested value available
GriPhyN/LIGO prototype 10/2001 Ewa Deelman, ISI
LIGO’s virtual data(in prototype)
• Virtual Data Products– Full frame for a specific time interval– Individual channel for a specific time interval– Decimated channel for a specific time interval
• Understand an XML-specified request• Acquire user’s proxy credentials• Consult replica catalog for available data• Construct a plan to produce data not available• Execute the plan• Return requested data in Frame or XML format
GriPhyN/LIGOLIGO SpecificDataSpecification
LIGO DataProduct
XML XML
GriPhyN/LIGO prototype 10/2001 Ewa Deelman, ISI
HTTPfrontend
MyProxyserver
ReplicaCatalog
ExecutorCondorG/DAGMan
Planner MonitoringMDS
TransformationCatalog
GridFTP GRAM/LDAS
LDAS
GridCVS
Logs
Storage Resource
GridFTP
ComputeResource
GRAM
xml
Cgi interface
G-DAG (DAGMan)
GriPhyN/LIGO prototype 10/2001 Ewa Deelman, ISI
GriPhyN/LIGO prototype 10/2001 Ewa Deelman, ISI
Request Interpreter
<?xml version="1.0"?> <!DOCTYPE LIGO-LW SYSTEM "LIGO-LW.dtd"> <LIGO_LW Name="banana" Type="VirtualDataRequest">
– Integrate with RC– Integrate with MyProxy– Better job monitoring
Longer term• Support a richer set of Virtual Data products• Implement and incorporate the Transformation Catalog• Investigate Derived Data Catalog (abstract DAGs)• Investigate more complex planning techniques
– Query estimation• Execution
– Error handling and recovery– Alternative strategies
GriPhyN/LIGO prototype 10/2001 Ewa Deelman, ISI
Prototype to Tool• Enable a close collaboration between LIGO participants
and other gravitational wave communities such as Virgo and GEO
• Replica Catalogs to provide information about data existence (Register new selected data in the catalog to make it available)
• Including Materialized Data (no need to recompute)• Provide access to data analysis software and systems
(Transformation Catalog)• Use the prototype as the basis for a data exchange and
replication system (security)• Provide access to world-wide computing resources