1 TeraGrid’07, June 4-8, 2007, Madison, WI Building Blocks for a Simple TeraGrid Science Gateway Yan Liu and Shaowen Wang Yan Liu and Shaowen Wang Grid Research & educatiOn group @ Grid Research & educatiOn group @ ioWa (GROW) ioWa (GROW) The University of Iowa The University of Iowa June 4, 2007 June 4, 2007 -- A GISolve Approach -- A GISolve Approach
53
Embed
TeraGrid’07, June 4-8, 2007, Madison, WI 1 Building Blocks for a Simple TeraGrid Science Gateway Yan Liu and Shaowen Wang Grid Research & educatiOn group.
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
11 TeraGrid’07, June 4-8, 2007, Madison, WI
Building Blocks for a Simple TeraGrid Science Gateway
Yan Liu and Shaowen WangYan Liu and Shaowen Wang
Grid Research & educatiOn group @ ioWa Grid Research & educatiOn group @ ioWa (GROW)(GROW)
Build user-friendly science gateway Build user-friendly science gateway interfaces for scientists to perform DMS interfaces for scientists to perform DMS analysis transparently on TeraGridanalysis transparently on TeraGrid
MySQL database accountsMySQL database accounts– Same user names as TeraGrid training accountsSame user names as TeraGrid training accounts– Password same as your portal account passwordPassword same as your portal account password
Tutorial server UNIX accountsTutorial server UNIX accounts– Same user names as TeraGrid training accountsSame user names as TeraGrid training accounts– Password dispatched with tutorial materialsPassword dispatched with tutorial materials– BASH shell environmentBASH shell environment
Tutorial packageTutorial package– In $HOME/SimpleGridIn $HOME/SimpleGrid
1212 TeraGrid’07, June 4-8, 2007, Madison, WI
Exercise 1: Setup Exercise 1: Setup Development Development EnvironmentEnvironment
Login to one of the three tutorial Login to one of the three tutorial serversservers– SSH login: username@${tutorial_server}SSH login: username@${tutorial_server}
Environment setupEnvironment setup
cd ~# add simplegrid-env.sh in your .bashrcvi .simplegrid-env.sh
1313 TeraGrid’07, June 4-8, 2007, Madison, WI
OutlineOutline
Part I: Basic TeraGrid services and Part I: Basic TeraGrid services and tools tools
Part II: SimpleGrid API Part II: SimpleGrid API development for Grid-enabled development for Grid-enabled analysisanalysis
Part III: SimpleGrid portal Part III: SimpleGrid portal developmentdevelopment
1414 TeraGrid’07, June 4-8, 2007, Madison, WI
Part I: Part I: Basic TeraGrid Basic TeraGrid Services and ToolsServices and Tools
Questions to answerQuestions to answer– Where are your home directories on all 3 Where are your home directories on all 3
TeraGrid sites?TeraGrid sites?– What is your grid certificate identity?– How do you create a grid proxy with a
specified duration using MyProxy?– Are you able to transfer a sample dataset to
a specified TeraGrid site?– How to submit a job and return immediately
SSH loginSSH login Setup your TeraGrid account environmentSetup your TeraGrid account environment
## content of your .soft setting@teragrid-basic# TeraGrid wide Globus 4 and Grid software [email protected]# Platform recommended development software suite@teragrid-dev
Deploy DMS binary package to all three sites Deploy DMS binary package to all three sites
[tomcat@grow-dev1 ~]$ myproxy-logon -l gisolve -t 100 -s myproxy.teragrid.orgEnter MyProxy pass phrase:A credential has been received for user gisolve in /tmp/x509up_u502.
[tomcat@grow-dev1 ~]$ grid-proxy-infosubject : /C=US/O=National Center for Supercomputing Applications/CN=Gisolve Community Userissuer : /C=US/O=National Center for Supercomputing Applications/CN=Certification Authorityidentity : /C=US/O=National Center for Supercomputing Applications/CN=Gisolve Community Usertype : end entity credentialstrength : 1024 bitspath : /tmp/x509up_u502timeleft : 99:59:54 (4.1 days)
http://teragrid.org/userinfo/access/
1717 TeraGrid’07, June 4-8, 2007, Madison, WI
Transfer a Sample Transfer a Sample DatasetDataset
Get a grid proxyGet a grid proxy Transfer a dataset to specified TeraGrid Transfer a dataset to specified TeraGrid
sitessites Submit Grid jobs to run DMS binary Submit Grid jobs to run DMS binary
executables against the transferred executables against the transferred datasetdataset
Collect results backCollect results back Visualize resultsVisualize results
– Will be shown laterWill be shown later
2222 TeraGrid’07, June 4-8, 2007, Madison, WI
Part II: SimpleGrid API Development Part II: SimpleGrid API Development for Grid-enabled Analysisfor Grid-enabled Analysis
How to do Part I in a programming way?How to do Part I in a programming way? PurposePurpose
– Automate the access to TeraGrid resources as a Grid-Automate the access to TeraGrid resources as a Grid-enabled applicationenabled application
PackagePackage– Location: SimpleGrid/simplegrid/src/Location: SimpleGrid/simplegrid/src/– SimpleGrid API for DMS analysisSimpleGrid API for DMS analysis
RSL generationRSL generation– Application specificApplication specific
GT2 job submissionGT2 job submission– Use batch mode in which a call Use batch mode in which a call
returns immediately with a job handlereturns immediately with a job handle
2626 TeraGrid’07, June 4-8, 2007, Madison, WI
GRAM Job Submission: GRAM Job Submission: MPIMPI
MPI on TeraGridMPI on TeraGrid– User “softenv | grep mpi” to see different User “softenv | grep mpi” to see different
MPI settingsMPI settings MPI job submission through GlobusMPI job submission through Globus
– Specify “count” element in RSLSpecify “count” element in RSL ““count” is the number of processes to spawncount” is the number of processes to spawn
– Specify “host_count” element in RSLSpecify “host_count” element in RSL ““host_count” is the number of CPUs for MPI host_count” is the number of CPUs for MPI
executionexecution– ““count = host_count” means each CPU runs count = host_count” means each CPU runs
one MPI processone MPI process– Specify “jobType=mpi” in RSLSpecify “jobType=mpi” in RSL
Test SimpleGrid APITest SimpleGrid API– Source simplegrid-env.shSource simplegrid-env.sh– SimpleGrid/bin/runTest2.shSimpleGrid/bin/runTest2.sh
3030 TeraGrid’07, June 4-8, 2007, Madison, WI
Part III: SimpleGrid Part III: SimpleGrid Portal DevelopmentPortal Development
Science gateways to TeraGridScience gateways to TeraGrid– Provide an online access points to TeraGridProvide an online access points to TeraGrid– Aggregate domain science application-level capabilitiesAggregate domain science application-level capabilities– Hide the complexity of using TeraGridHide the complexity of using TeraGrid– Portal is commonly used to meet such needsPortal is commonly used to meet such needs
FocusFocus– Provide transparent access to TeraGrid for domain Provide transparent access to TeraGrid for domain
scientistsscientists– Build application-oriented workflow inside portal for easy Build application-oriented workflow inside portal for easy
access by usersaccess by users– Technical details for portlet developmentTechnical details for portlet development
GridSphereGridSphere PortletAPIPortletAPI GridSphere-based portlet developmentGridSphere-based portlet development
3131 TeraGrid’07, June 4-8, 2007, Madison, WI
GridSphereGridSphere
FeaturesFeatures– http://www.gridsphere.orghttp://www.gridsphere.org– Open sourceOpen source– PortletAPI (JSR-168) compliantPortletAPI (JSR-168) compliant– Object persistence support through HibernateObject persistence support through Hibernate
GridSphere 3.0.5GridSphere 3.0.5– Download: Download: https://svn.gridsphere.org/https://svn.gridsphere.org/– Requirements: java, ant, tomcat, mysqlRequirements: java, ant, tomcat, mysql– Build: ant installBuild: ant install
Be sure to change hibernate.properties, build.propertiesBe sure to change hibernate.properties, build.properties Copy mysql jdbc driver to Copy mysql jdbc driver to
${CATALINE_HOME}/common/lib/${CATALINE_HOME}/common/lib/– Development using EclipseDevelopment using Eclipse
Instructions will be added in our online documentationInstructions will be added in our online documentation
Location: SimpleGrid/simplegridLocation: SimpleGrid/simplegrid Two portletsTwo portlets
– UserPortletUserPortlet Display user informationDisplay user information Initialize or renew grid proxyInitialize or renew grid proxy
– Automatic renewal after initializationAutomatic renewal after initialization
– DMSPortletDMSPortlet DMS workflow controlDMS workflow control Build user interfaces for DMS analysisBuild user interfaces for DMS analysis
Each tutorial user will build and deploy their Each tutorial user will build and deploy their version of SimpleGrid portlets (project name is version of SimpleGrid portlets (project name is simplegrid_${USER})simplegrid_${USER})
3434 TeraGrid’07, June 4-8, 2007, Madison, WI
Introduction to Introduction to SimpleGrid SimpleGrid DevelopmentDevelopment
lib/lib/– Libraries for Globus and JFreeChartLibraries for Globus and JFreeChart
webapp/jsp/webapp/jsp/– JSP pages for user interface renderingJSP pages for user interface rendering
webapp/WEB-INF/persistence/hibernate.propertieswebapp/WEB-INF/persistence/hibernate.properties– Configuration for backend database accessConfiguration for backend database access
Defines unique portlet id, the portlet class, etc.Defines unique portlet id, the portlet class, etc. webapp/simplegrid.propertieswebapp/simplegrid.properties
– SimpleGrid specific configuration about Grid resourcesSimpleGrid specific configuration about Grid resources– Loaded at the first time the UserPortlet is visitedLoaded at the first time the UserPortlet is visited
build.xmlbuild.xml– Ant build file to compile, deploy, and install SimpleGrid portlets to GridSphereAnt build file to compile, deploy, and install SimpleGrid portlets to GridSphere– Need to change if you have your own libraries or packages developedNeed to change if you have your own libraries or packages developed
build.propertiesbuild.properties– Define project name and location of GridSphere where SimpleGrid finds Define project name and location of GridSphere where SimpleGrid finds
dependent GridSphere java classes and PortletAPI (javax.portlet.*) definitionsdependent GridSphere java classes and PortletAPI (javax.portlet.*) definitions
PortletAPI (JSR-168 specification)PortletAPI (JSR-168 specification)– Portlet specificationPortlet specification– What are not defined?What are not defined?
Object persistence: store portlet status in permanent storageObject persistence: store portlet status in permanent storage Portlet rendering methodPortlet rendering method
– JSP, Velocity, GridSphere Visual UI (JSP taglib), Java Server FaceJSP, Velocity, GridSphere Visual UI (JSP taglib), Java Server Face GridSphere: JSR-168 compliant portlet containerGridSphere: JSR-168 compliant portlet container
– Implemented javax.portlet.*Implemented javax.portlet.*– Hibernate as object persistence technologiesHibernate as object persistence technologies– JPS-based Visual UI renderingJPS-based Visual UI rendering– What are not standardized? What are not standardized?
ActionPortlet based on Visual UIActionPortlet based on Visual UI Portlet service framework (from WebSphere)Portlet service framework (from WebSphere)
– Other open source portlet containersOther open source portlet containers Apache Jetspeed/Pluto, SakaiApache Jetspeed/Pluto, Sakai
Deploy your SimpleGrid project Deploy your SimpleGrid project (simplegrid_${USER}) to gridsphere (simplegrid_${USER}) to gridsphere containercontainer– Use “ant install” for a fresh installUse “ant install” for a fresh install– Use “ant deploy” for update (it does not clear Use “ant deploy” for update (it does not clear
database content)database content)– Look at Look at
$CATALINA_HOME/webapps/simplegrid_$$CATALINA_HOME/webapps/simplegrid_${USER}/WEB-INF/lib to see project jars {USER}/WEB-INF/lib to see project jars generated by deploymentgenerated by deployment
How does Gridsphere know your project?How does Gridsphere know your project?
4444 TeraGrid’07, June 4-8, 2007, Madison, WI
GridSphere-based GridSphere-based Portlet DevelopmentPortlet Development
Visual UI taglib for renderingVisual UI taglib for rendering
Portlet classPortlet class– Extend ActionPortletExtend ActionPortlet– All beans are constructed from ActionFormEvent or All beans are constructed from ActionFormEvent or
RenderFromEventRenderFromEvent– Get PortletRequest and PortletResponse from FormEventGet PortletRequest and PortletResponse from FormEvent– Use setNextState() to point to JSP page in rendering methodsUse setNextState() to point to JSP page in rendering methods– Use setNextState() to point to rendering method in action Use setNextState() to point to rendering method in action
UserPortlet::show()UserPortlet::show()– Default rendering method which displays user Default rendering method which displays user
information and grid proxy configuration in JSPinformation and grid proxy configuration in JSP– In JSP, Bean is used to render the values of user In JSP, Bean is used to render the values of user
forms forms Java code is embedded to directly access Java code is embedded to directly access
RenderRequest for grid proxy informationRenderRequest for grid proxy information
UserPortlet::configSimpleCred()UserPortlet::configSimpleCred()– Action method to retrieve grid proxy either Action method to retrieve grid proxy either
from MyProxy server or locally from a valid from MyProxy server or locally from a valid proxy fileproxy file
Action classAction class– Get parameters: ActionRequest.getParameter()Get parameters: ActionRequest.getParameter()– Rendering: VelocityContext.put(name, object)Rendering: VelocityContext.put(name, object)– Action methods: called based on their names through Java Action methods: called based on their names through Java
DMS Analysis DMS Analysis Workflow in PortalWorkflow in Portal
Workflow control is done by keeping Workflow control is done by keeping state information in PortletSession as state information in PortletSession as attributesattributes
Portlet communicationPortlet communication– Need to coordinate two portletsNeed to coordinate two portlets
Learn how SimpleGrid remembers Learn how SimpleGrid remembers current status of each jobcurrent status of each job
4949 TeraGrid’07, June 4-8, 2007, Madison, WI
DMSPortlet InterfaceDMSPortlet Interface
5050 TeraGrid’07, June 4-8, 2007, Madison, WI
Exercise 5Exercise 5
Run DMS portlet and get visualization Run DMS portlet and get visualization resultsresults– Result page plots results after the analysisResult page plots results after the analysis– Do you see your job is running on TeraGrid?Do you see your job is running on TeraGrid?
Refresh job statusRefresh job status Verify it by using “qstat | grep gisolve” to see the Verify it by using “qstat | grep gisolve” to see the
status of your jobstatus of your job
– Thread in SimpleGrid portletsThread in SimpleGrid portlets File transfer and result visualization access the same File transfer and result visualization access the same
portal server and are time-consumingportal server and are time-consuming Threads are used to respond to user requests quicklyThreads are used to respond to user requests quickly
5151 TeraGrid’07, June 4-8, 2007, Madison, WI
Visualization on Visualization on TeraGridTeraGrid
SimpleGrid uses JFreeChart for SimpleGrid uses JFreeChart for simple local visualizationsimple local visualization
We are evaluating the integration We are evaluating the integration between GISolve and ParaView on between GISolve and ParaView on TeraGrid to support advanced TeraGrid to support advanced visualizationvisualization
http://tg-portal.uc.teragrid.org/
5252 TeraGrid’07, June 4-8, 2007, Madison, WI
SummarySummary
SimpleGrid provides basic components to SimpleGrid provides basic components to develop science and engineering gatewaysdevelop science and engineering gateways– This tutorial includes code, links, and examplesThis tutorial includes code, links, and examples– Based on a real-world GIS applicationBased on a real-world GIS application
SimpleGrid is also used in GISolve, SimpleGrid is also used in GISolve, TeraGrid GIScience gateway TeraGrid GIScience gateway
Gateway server setup instructions and Gateway server setup instructions and additional materials will be provided in additional materials will be provided in GISolve/TeraGrid online documentationGISolve/TeraGrid online documentation
5353 TeraGrid’07, June 4-8, 2007, Madison, WI
Q & AQ & A
Questions and comments?Questions and comments? Thanks!Thanks!