Contact: Junwei Cao ([email protected]) SC2005, Seattle, WA, November 12-18, 2005 The authors gratefully acknowledge the support of the United States National Science Foundation for the construction and operation of the LIGO Laboratory and the Particle Physics and Astronomy Research Council of the United Kingdom, the Max-Planck-Society and the State of Niedersachsen/Germany for support of the construction and operation of the GEO600 detector. The authors also gratefully acknowledge the support of the research by these agencies and by the Australian Research Council, the Natural Sciences and Engineering Research Council of Canada, the Council of Scientific and Industrial Research of India, the Department of Science and Technology of India, the Spanish Ministerio de Educacion y Ciencia, the John Simon Guggenheim Foundation, the Leverhulme Trust, the David and Lucile Packard Foundation, the Research Corporation, and the Alfred P. Sloan Foundation. This work is directly supported under National Science Foundation award #0326281. Grid Enabled LIGO Data Monitoring Junwei Cao 1 , Erik Katsavounidis 1 , and John Zweizig 2 1 LIGO Laboratory - Massachusetts Institute of Technology, Cambridge, MA 02139, USA 2 LIGO Laboratory - California Institute of Technology, Pasadena, CA 91125, USA LIGO: Laser Interferometer Gravitational-wave Observatory The LSC Data Grid (LDG) AEI/G olm C ardiff B irm ingham • AEI/G olm AEI/G olm C ardiff B irm ingham • C ardiff C ardiff B irm ingham • Users are interfaced with a LIG O friendly language. [job] id = test m onitor= rm on args = -optopt input= opt [data] observatory = @ H@ L type = @ RDS_R_L3@ RDS_R_L3 start= 751658000 end = 751676993 ldm.sub [jcao@ldaspc1 ~]$ cd ldm [jcao@ldaspc1 ldm]$ source setup.sh [jcao@ldaspc1 ldm]$ cd ../rmon [jcao@ldaspc1 rmon]$ ldm_agent [jcao@ldaspc1 rmon]$ ldm_submit ldm.sub Job test has been submitted. [jcao@ldaspc1 rmon]$ more ldm_test_condor.out Processing multi list file: ldm_test_CIT_multilist.txt Number of lists added: 2 Total data streams: 2 …… startgps=751658000 stride=16 r-statistic=-0.00251782 …… grid-enabled run of rmon DMT offline monitor using LDM universe = globus globusscheduler = ldas-grid.ligo.caltech.edu/jobm anager-condor log = ldm _test_condor.log output= ldm _test_condor.out error= ldm _test_condor.err should_transfer_files = YES w hen_to_transfer_output= O N _EXIT transfer_input_files = ldm _test_C IT_m ultilist.txt,ldm _test_C IT_filelist1.txt, ldm _test_C IT_filelist2.txt,/hom e/jcao/rm on/opt argum ents = -inlists ldm _test_C IT_m ultilist.txt-optopt environm ent= LD _LIB R A R Y_PA TH =/dso-test/jcao/dol/lib executable = /hom e/jcao/rmon/rm on Q ueue automatically generated Condor submission file Users do not bother with technical details of LS C data grid services. Data are located and file lists are generated automatically Users are interfaced with a LIG O friendly language. [job] id = test m onitor= rm on args = -optopt input= opt [data] observatory = @ H@ L type = @ RDS_R_L3@ RDS_R_L3 start= 751658000 end = 751676993 ldm.sub [jcao@ldaspc1 ~]$ cd ldm [jcao@ldaspc1 ldm]$ source setup.sh [jcao@ldaspc1 ldm]$ cd ../rmon [jcao@ldaspc1 rmon]$ ldm_agent [jcao@ldaspc1 rmon]$ ldm_submit ldm.sub Job test has been submitted. [jcao@ldaspc1 rmon]$ more ldm_test_condor.out Processing multi list file: ldm_test_CIT_multilist.txt Number of lists added: 2 Total data streams: 2 …… startgps=751658000 stride=16 r-statistic=-0.00251782 …… grid-enabled run of rmon DMT offline monitor using LDM universe = globus globusscheduler = ldas-grid.ligo.caltech.edu/jobm anager-condor log = ldm _test_condor.log output= ldm _test_condor.out error= ldm _test_condor.err should_transfer_files = YES w hen_to_transfer_output= O N _EXIT transfer_input_files = ldm _test_C IT_m ultilist.txt,ldm _test_C IT_filelist1.txt, ldm _test_C IT_filelist2.txt,/hom e/jcao/rm on/opt argum ents = -inlists ldm _test_C IT_m ultilist.txt-optopt environm ent= LD _LIB R A R Y_PA TH =/dso-test/jcao/dol/lib executable = /hom e/jcao/rmon/rm on Q ueue automatically generated Condor submission file Users do not bother with technical details of LS C data grid services. Data are located and file lists are generated automatically /data/node10/fram e/S3/L3/LHO /H -RD S_R_L3-751658016-16.gw f /data/node11/fram e/S3/L3/LHO /H -RD S_R_L3-751658032-16.gw f /data/node12/fram e/S3/L3/LHO /H -RD S_R_L3-751658048-16.gw f /data/node13/fram e/S3/L3/LHO /H -RD S_R_L3-751658064-16.gw f /data/node14/fram e/S3/L3/LHO /H -RD S_R_L3-751658080-16.gw f /data/node15/fram e/S3/L3/LHO /H -RD S_R_L3-751658096-16.gw f /data/node16/fram e/S3/L3/LHO /H -RD S_R_L3-751658112-16.gw f /data/node10/fram e/S3/L3/LLO /L-R D S _R _L3-751658016-16.gw f /data/node11/fram e/S3/L3/LLO /L-R D S _R _L3-751658032-16.gw f /data/node12/fram e/S3/L3/LLO /L-R D S _R _L3-751658048-16.gw f /data/node13/fram e/S3/L3/LLO /L-R D S _R _L3-751658064-16.gw f /data/node14/fram e/S3/L3/LLO /L-R D S _R _L3-751658080-16.gw f /data/node15/fram e/S3/L3/LLO /L-R D S _R _L3-751658096-16.gw f /data/node16/fram e/S3/L3/LLO /L-R D S _R _L3-751658112-16.gw f rm on filelist1.txt filelist2.txt multilist.txt stride 16.0 channel_1 H 1:LSC -A S_Q channel_2 L1:LS C -A S_Q opt filelist1.txt filelist2.txt [jcao@ldaspc1 rmon]$ export LD_LIBRARY_PATH=/opt/lscsoft/dol/lib [jcao@ldaspc1 rmon]$ ./rmon -opt opt -inlists multilist.txt Processing multi list file: multilist.txt Number of lists added: 2 Total data streams: 2 Processing frame list file: /home/jcao/rmon/filelist1.txt Number of files added: 1188 Total frame files: 1188 Processing frame list file: /home/jcao/rmon/filelist2.txt Number of files added: 1188 Total frame files: 1188 channel[1]=H1:LSC-AS_Q channel[2]=L1:LSC-AS_Q startgps=751658000 stride=16 r-statistic=-0.00251782 startgps=751658016 stride=16 r-statistic=-0.0122699 startgps=751658032 stride=16 r-statistic=0.0168868 …… standalone run of rmon DMT offline monitor /data/node10/fram e/S3/L3/LHO /H -RD S_R_L3-751658016-16.gw f /data/node11/fram e/S3/L3/LHO /H -RD S_R_L3-751658032-16.gw f /data/node12/fram e/S3/L3/LHO /H -RD S_R_L3-751658048-16.gw f /data/node13/fram e/S3/L3/LHO /H -RD S_R_L3-751658064-16.gw f /data/node14/fram e/S3/L3/LHO /H -RD S_R_L3-751658080-16.gw f /data/node15/fram e/S3/L3/LHO /H -RD S_R_L3-751658096-16.gw f /data/node16/fram e/S3/L3/LHO /H -RD S_R_L3-751658112-16.gw f /data/node10/fram e/S3/L3/LLO /L-R D S _R _L3-751658016-16.gw f /data/node11/fram e/S3/L3/LLO /L-R D S _R _L3-751658032-16.gw f /data/node12/fram e/S3/L3/LLO /L-R D S _R _L3-751658048-16.gw f /data/node13/fram e/S3/L3/LLO /L-R D S _R _L3-751658064-16.gw f /data/node14/fram e/S3/L3/LLO /L-R D S _R _L3-751658080-16.gw f /data/node15/fram e/S3/L3/LLO /L-R D S _R _L3-751658096-16.gw f /data/node16/fram e/S3/L3/LLO /L-R D S _R _L3-751658112-16.gw f rm on filelist1.txt filelist2.txt multilist.txt stride 16.0 channel_1 H 1:LSC -A S_Q channel_2 L1:LS C -A S_Q opt filelist1.txt filelist2.txt [jcao@ldaspc1 rmon]$ export LD_LIBRARY_PATH=/opt/lscsoft/dol/lib [jcao@ldaspc1 rmon]$ ./rmon -opt opt -inlists multilist.txt Processing multi list file: multilist.txt Number of lists added: 2 Total data streams: 2 Processing frame list file: /home/jcao/rmon/filelist1.txt Number of files added: 1188 Total frame files: 1188 Processing frame list file: /home/jcao/rmon/filelist2.txt Number of files added: 1188 Total frame files: 1188 channel[1]=H1:LSC-AS_Q channel[2]=L1:LSC-AS_Q startgps=751658000 stride=16 r-statistic=-0.00251782 startgps=751658016 stride=16 r-statistic=-0.0122699 startgps=751658032 stride=16 r-statistic=0.0168868 …… standalone run of rmon DMT offline monitor Applications Infrastructures End users & applications Application enabling Middleware / S ervices Operating S ystems and … FC3 GCC A utotools P ython M ySQ L Job scheduling /C ondor C ondor-G D ata transfer/G ridFTP W orklfow m anagem ent/C ondorD AG m an G rid security /G lobus GSI R esource m anagem ent/G lobus G R A M R esource location service /G lobus C atalog service /G lobus Inform ation service /G lobus M DS VDS LSC certU tils LS C S ecurity management LSC dataFind LSC segFind LDR LS C Data management The LIG O D ata M onitoring (LD M )Environm ent Glue O nasys LS C J ob management LDAS DMT LALApps LSC CA VOMS The LS C Data Grid Client/S erver E nvironment Version 3.5 (using V DT 1.3.5) Matlab Applications Infrastructures End users & applications Application enabling Middleware / S ervices Operating S ystems and … FC3 GCC A utotools P ython M ySQ L Job scheduling /C ondor C ondor-G D ata transfer/G ridFTP W orklfow m anagem ent/C ondorD AG m an G rid security /G lobus GSI R esource m anagem ent/G lobus G R A M R esource location service /G lobus C atalog service /G lobus Inform ation service /G lobus M DS VDS LSC certU tils LS C S ecurity management LSC dataFind LSC segFind LDR LS C Data management The LIG O D ata M onitoring (LD M )Environm ent Glue O nasys LS C J ob management LDAS DMT LALApps LSC CA VOMS The LS C Data Grid Client/S erver E nvironment Version 3.5 (using V DT 1.3.5) Matlab QUEUED SCHEDULED LO C ATIN G LOCATED RUNNING FIN ISH ED ldm _agent ldm_locate_script ldm_exec_script ldm_subm it RELEASED ldm _rm ldm_q REJECTED LSC dataFind condor_subm it LDM Other tools client server LSC dataFind Server G lobus Job M anager ldm _agent C ondor [A G EN T] RESO URCES = @ MIT@ CIT@ LHO@ LLO SITES = /hom e/jcao/ldm /etc/LD M _SITES EXEC = /hom e/jcao/ldm/bin/ldm_exec_script LO C A TE = /hom e/jcao/ldm /bin/ldm _locate_script PID = /hom e/jcao/ldm /var/ldm .pid LO G = /hom e/jcao/ldm /var/ldm .log LDG = /hom e/jcao/ldg-3.0/ [M IT] lscdatafindserver= ldas-gridm on.m it.edu globusscheduler = ldas-grid.m it.edu/jobmanager-condor environm ent= LD _LIB R AR Y_P A TH =/dso-test/hom e/jcao/dol/lib [C IT] lscdatafindserver= ldas-gridm on.ligo.caltech.edu globusscheduler = ldas-grid.ligo.caltech.edu/jobm anager-condor environm ent= LD _LIB R AR Y_P A TH =/dso-test/jcao/dol/lib [LHO ] lscdatafindserver= ldas-gridm on.ligo-w a.caltech.edu globusscheduler = ldas-grid.ligo-w a.caltech.edu/jobmanager-condor environm ent= LD _LIB R AR Y_P A TH =/dso-test/jcao/dol/lib [LLO ] lscdatafindserver= ldas-gridm on.ligo-la.caltech.edu globusscheduler = ldas-grid.ligo-la.caltech.edu/jobm anager-condor environm ent= LD _LIB R AR Y_P A TH =/data2/jcao/dol/lib LDM_CONFIG LDM_SITES M odules developed ordeployed M odules designed and underdeveloped condor_m aster QUEUED SCHEDULED LO C ATIN G LOCATED RUNNING FIN ISH ED ldm _agent ldm_locate_script ldm_exec_script ldm_subm it RELEASED ldm _rm ldm_q REJECTED LSC dataFind condor_subm it LDM Other tools LDM Other tools client server client server LSC dataFind Server G lobus Job M anager ldm _agent C ondor [A G EN T] RESO URCES = @ MIT@ CIT@ LHO@ LLO SITES = /hom e/jcao/ldm /etc/LD M _SITES EXEC = /hom e/jcao/ldm/bin/ldm_exec_script LO C A TE = /hom e/jcao/ldm /bin/ldm _locate_script PID = /hom e/jcao/ldm /var/ldm .pid LO G = /hom e/jcao/ldm /var/ldm .log LDG = /hom e/jcao/ldg-3.0/ [M IT] lscdatafindserver= ldas-gridm on.m it.edu globusscheduler = ldas-grid.m it.edu/jobmanager-condor environm ent= LD _LIB R AR Y_P A TH =/dso-test/hom e/jcao/dol/lib [C IT] lscdatafindserver= ldas-gridm on.ligo.caltech.edu globusscheduler = ldas-grid.ligo.caltech.edu/jobm anager-condor environm ent= LD _LIB R AR Y_P A TH =/dso-test/jcao/dol/lib [LHO ] lscdatafindserver= ldas-gridm on.ligo-w a.caltech.edu globusscheduler = ldas-grid.ligo-w a.caltech.edu/jobmanager-condor environm ent= LD _LIB R AR Y_P A TH =/dso-test/jcao/dol/lib [LLO ] lscdatafindserver= ldas-gridm on.ligo-la.caltech.edu globusscheduler = ldas-grid.ligo-la.caltech.edu/jobm anager-condor environm ent= LD _LIB R AR Y_P A TH =/data2/jcao/dol/lib LDM_CONFIG LDM_SITES M odules developed ordeployed M odules designed and underdeveloped condor_m aster • The LIGO project aims to make the first direct detection of gravitational waves as predicted by Einstein’s General Theory of Relativity. • LIGO manages and operates two national observatories for gravitational waves, located at Hanford, WA and Livingston, LA. • The LIGO Scientific Collaboration (LSC) consists of nearly 400 scientists from about 40 institutions worldwide. • In the LSC, a grid computing infrastructure is utilized for LIGO data analysis and monitoring with thousands of CPUs and terabytes of data storage capabilities distributed over 10 sites in the USA and Europe. • The LSC Data Grid (LDG) client/server environment is built on top of the Virtual Data Toolkit (VDT). The LIGO Data Monitoring (LDM) Environment • LDM interfaces users with a LIGO friendly language instead of the more CS flavor Condor language. • Technical details of grid computing are hidden from end users (mostly gravitational wave physicists in the LSC). • The experience gained in this work will be applied for future deployment of the Open Science Grid (OSG) infrastructure for LIGO applications. The Data Monitoring Toolkit (DMT) • DMT defines tools and environments necessary to support continuous data monitoring of LIGO interferometers. • While DMT is essentially a production-level software package being used in LIGO control rooms, researchers find the enrichment of DMT libraries make it an ideal environment for offline LIGO data analysis. For More Information … http://www.ligo.caltech.edu http://www.ligo.mit.edu/lsc-itr2003 http://www.ligo.org http://www.lsc-group.phys.uwm. edu/lscdatagrid http://www.ligo.mit.edu/ldg http://www.griphyn.org http://www.ivdgl.org http://www.opensciencegrid.org For the LIGO Scientific Collaboration (LSC) LIGO Document No. G050573-00-E