MyProxy, GridFTP, GRAM & so much more XSEDE Gateways

MyProxy,  GridFTP,  GRAM  &  so  much  more  

XSEDE  Gateways  Tutorial  Matt  McKenzie,  PhD    

https://www.xsede.org/gateways-­‐overview  

¡ What  type  of  Gateway?  § Who  is  your  audience?  ▪  Desktop  application  ▪  Large  portal/website  interface    

¡  Structured  staging  plans    §  Short  &  long  term  goals  §  Precise  list  of  requirements  for  your  gateway  §  Choose  technologies  based  on  resources  and  time  

¡  Bring  your  development  team  with  you  ¡ Make  friends  at  your  resource  site(s)  

¡  For  scalability  Community  Accounts  (CA)  are  used  instead  of  unique  XSEDE  accounts  §  A  single  community  user  account  is  created  ▪  Gateway  uses  the  CA  to  launch  jobs  ▪  SAML  attribute  (citizenship  info)  is  sent  along  CA  request  

§  Gateway  user  of  the  CA  typically  has  privileges  to  only  run  a  limited  set  of  commands  ▪  Community  Shell  –  commsh       cat  commsh.mmcken6.conf  

#  Generic  commands  DirectAccess  cc  **  DirectAccess  aprun  **  DirectAccess  /bin/cat  *    

¡  A  service  for  securing  private  keys  §  Keys  stored  encrypted  with  password  §  Keys  never  leave  the  MyProxy  server    

¡   A  service  for  retrieving  proxy  credentials  §  Delegate  these  credentials  to  access  other  resources  

 ¡  A  commonly  used  service  for  gateway/portal  security    

¡  myproxy-­‐logon  §  Retrieve  stored  credential  §  Requires  passphrase  that  was  used  when  it  was  stored  §  First  command  you  should  always  execute    

¡  myproxy-­‐info  §  Get  stored  credential  information    

¡  myproxy-­‐init  §  Generate  a  proxy  from  credentials  on  local  machine  &  store  it  

on  the  proxy  server  ¡  myproxy-­‐destroy  

§  Remove  credential  

¡  Useful  flags  ¡  -­‐s  server_name  ¡  -­‐l  user_name  

¡  GridFTP:  Globus  provides  this  service  §  Implements  FTP  protocol  using  GSI  authentication  

¡  globus-­‐url-­‐copy  §  Client  tool  that  performs  file  management  tasks  

¡  globus-­‐url-­‐copy  Source_URL  Destination_URL  §  URL  looks  like:  §  gsiftp://<hostname:port>/path  ▪  globus-­‐url-­‐copy  gsiftp://gridftp.kraken.nics.xsede.org/PATH/TO/FILE  gsiftp://gridftp.blacklight.psc.teragrid.org/PATH/TO/FILE  

§  The  file://  protocol  is  deprecated,  please  do  not  use  

http://www.loni.org

LONI HPC Enablement Workshop – LaTech University, October 23, 2008

!"#$%&'(

Basic Transfer One control channel, several

parallel data channels

Third-party

Transfer Control channels to each server, several parallel

data channels between servers Striped Transfer

Control channels to each server on one node, several parallel data channels

between servers and data channels spread across nodes

myproxy.teragrid.org  

Myproxy-­‐logon  Globus-­‐job-­‐run  My  machine  

Gatekeeper   LRM  

jobmanager  jobmanager  

jobmanager  

Condor  pool  

PBS:  Compute  nodes  

Fork:  Local  host  

1 2 3                                                                  

globusrun          where_to/who_to_talk_to                          what_to_do  globusrun          url_to_resource/jobmanager-­‐X      RSL  file  

¡  Globusrun  –  Execute  &  manage  jobs  via  GRAM  ¡  Useful  options  

§  -­‐p  ▪  Verify  syntax  of  the  RSL  specification  

§  -­‐a  ▪   Authenticate  only  

§  -­‐y  or  –refresh-­‐proxy  ▪  Attempt  to  delegate  a  new      X.509  proxy  to  the  job  manager  managing  a  

jobID  §  -­‐status      §  -­‐f  RSL_FILENAME  §  -­‐r  Resource  Contact  ▪  Host:Port/Service  

§  -­‐b  ▪  Submit  job  and  exit  GRAM  service  

¡  XML  language  ¡  Attribute  &  value  pairings  ¡  GRAM  attributes:  executable,  arguments,  count,  directory,  maxtime,  jobtype,  project,…  §  A  different  way  to  describe  a  PBS  script  

¡  LRM  interprets  the  RSL  attributes  to  manage  the  GRAM  request  

 &  (project=TG-­‐STA110014S)      (jobtype=mpi)      (directory=/lustre/scratch/mmcken6/apoa1/)      (count=24)      (executable=/lustre/scratch/mmcken6/namd2)      (arguments=apoa1.namd)  

#!/bin/bash  #PBS  –l  size=24  #PBS  –A  TG-­‐STA110014S  cd  /lustre/scratch/mmcken6/apoa1/  aprun  –n  24  /lustre/scratch/mmcken6/namd2  apoa1.namd    

:~>globusrun  -­‐o  -­‐r  grid.nics.utk.edu:2119/jobmanager-­‐pbs  '&(directory=/lustre/scratch/$USER)(executable=/lustre/scratch/$USER/helloworld)(jobtype=mpi)(count=12)(project=my_allocation)'  

Gatekeeper  

jobmanager  jobmanager  

jobmanager  

Condor  pool  

PBS:  Compute  nodes  

Fork:  Local  host  

¡  jobmanager-­‐pbs  '&(directory=/lustre/scratch/user)(executable=/lustre/scratch/$USER/helloworld)(jobtype=mpi)(count=12)(project=my_allocation)'  

¡  Jobmanager-­‐pbs  autogenerates  a  PBS  script  ¡  Jobtype=mpi    

§  Places  “aprun  –n”  or  “mpirun  –np”  §  Site/machine  dependent  

¡  Jobtype=single  §  This  is  solitary,  non-­‐parallel  job  §  Most  flexible  option  ▪  One  can  set  aprun/mpirun/other  exe  and  its  various  arguments  ▪  '&(executable=aprun)(arguments="-­‐n"  "24"  "helloworld")  (directory=/lustre/scratch/user)(jobtype=single)(count=24)(maxtime=10)  (project=my_allocation)'  

¡  Jobtype=mpi  §  Generates  a  PBS  for  MPI  jobs  

¡  Jobtype=multiple  §  Parallel  applications  that  do  not  depend  on  MPI  §  Check  your  site  friend  on  how  this  is  implemented  ▪  @  NICS  =  many  under  1  submission  ▪  @  SDSC  =  many  submissions  

~>  more  sub_namd.rsl  &  (project=TG-­‐STA110014S)      (jobtype=mpi)      (directory=/lustre/scratch/mmcken6/apoa1/)      (count=24)      (executable=/sw/xt/namd/2.8/cnl3.1_gnu4.6.1/bin/namd2)      (arguments=apoa1.namd)    ~>  globusrun  -­‐o  -­‐r  grid.nics.utk.edu:2119/jobmanager-­‐pbs  -­‐f  sub_namd.rsl  GRAM  Job  failed  because  the  authorization  system  denied  the  request  –    not  authorized  to  run  the  specified  executable  (error  code  165)    

Solution  Email  [email protected]  or  talk  to  your  XSEDE  site  friend  Could  you  please  add  “/sw/xt/namd/2.8/cnl3.1_gnu4.6.1/bin/namd2”    to  my  commsh?  

~>globusrun  -­‐o  -­‐r  grid.nics.utk.edu:2119/jobmanager-­‐pbs  '&(directory=/lustre/scratch/$USER)  (executable=/lustre/scratch/$USER/helloworld)(jobtype=mpi)(count=12)(project=my_allocation)'    Hello  world  from  process  8  of  12  Hello  world  from  process  10  of  12  Hello  world  from  process  7  of  12  Hello  world  from  process  3  of  12  Hello  world  from  process  11  of  12  Hello  world  from  process  4  of  12  Hello  world  from  process  5  of  12  Hello  world  from  process  1  of  12  Hello  world  from  process  0  of  12  Hello  world  from  process  9  of  12  Hello  world  from  process  2  of  12  Hello  world  from  process  6  of  12    Application  5579880  resources:  utime  0,  stime  0  

¡  Good  overview  of  the  parts  of  GRAM5  §  http://www.globus.org/toolkit/docs/5.0/5.0.0/execution/gram5/user/  

¡  Good  manual  for  RSL  files  §  http://www.globus.org/toolkit/docs/2.4/gram/rsl_spec1.html/#Simple%20RSL%20Examples  

¡ Many,  many  examples  and  error  codes  §  http://www.nics.tennessee.edu/computing-­‐resources/grid_services  

Seen  ¡  Scientific  workflows  on  XSEDE  ¡  ParamChem  Gateway  ¡  Xbaya,  graphical  workflow  composer  ¡  Airavata  overview  &  tutorial  

Coming    ¡  Airavata  software  toolkit  

§  Does  more  than  build  scientific  workflows