CSinParallel Workshop OnRamp : An Interactive Learning Portal for Parallel Computing Environments Samantha Foley [email protected]http://cs.uwlax.edu/~ssfoley 1 Josh Hursey [email protected]http://cs.uwlax.edu/~jjhursey/ Project supported by SIGCSE Special Projects Grant (May 2015), Blue Waters Student Internship program (2015), and the UW-L Computer Science Department https://github.com/ssfoley/onramp
12
Embed
CSinParallel Workshop OnRamp: An Interactive Learning ...€¦ · • May 2015 • SIGCSE Grant started (project development started) • Sept. 2015 (alpha release) • PCEs: SLURM,
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
• High-Level Goals• Encourage students to explore parallel and distributed computing
concepts without the overhead of PCE system software peculiarities. • Help students transition to using the native PCE, eventually. • Bring together existing educational hardware & curriculum modules
efforts into a flexible, portable architecture.
• Basic Architecture
4 https://github.com/ssfoley/onramp
OnRamp Server
LittleFeOnRamp PCE
VM ClusterOnRamp PCE
XSEDEOnRamp PCE
ClusterOnRamp PCE
Portal
OnRamp to Parallel Computing Project�
• Web Portal (client facing side of the OnRamp Server)• Users do not need accounts on each PCE (use a shared UNIX account). • Users can be grouped into one or more Workspaces. • Workspaces are assigned PCE & Module combinations.
• Users view Module instructions, launch jobs, view results, transfer files. • Automatically generated & validated custom forms for each Module.
• OnRamp Server• Broker between the users of the OnRamp Portal and the PCEs. • Enforce policy, authenticate users, and manage security. • Cache files between Users and PCEs.
• OnRamp PCE• Custom drivers for each type of software environment (SLURM, PBS, LSF). • Manage modules, and user files associated with jobs. • Launch and monitor jobs on the system.
6
OnRamp Server
LittleFeOnRamp PCE
VM ClusterOnRamp PCE
XSEDEOnRamp PCE
ClusterOnRamp PCE
Portal
Developing ‘turn-‐key’ scripts to setup virtual
clusters.
https://github.com/ssfoley/onramp
OnRamp to Parallel Computing Project�
• Curriculum Modules• Complete freedom to structure the module as you like. • Write a few python scripts to ‘hook’ into the OnRamp architecture. • Configuration files allow you to specify custom, module-specific parameters and
validation requirements for users running the code. • Custom documentation/instructions • Can be tailored to the PCE environment
7
OnRamp Server
LittleFeOnRamp PCE
VM ClusterOnRamp PCE
XSEDEOnRamp PCE
ClusterOnRamp PCE
Curriculum modules
A
Z
B
Z
A
B
R
Z
Portal
https://github.com/ssfoley/onramp
OnRamp to Parallel Computing Project�Curriculum Module Life cycle�
• Admin deploys Module on PCE using Portal • Module can be deployed from git, svn, tar.gz, file system • OnRamp PCE unpacks Module and runs onramp_deploy.py• Script can compile the code, generate documentation, …
• OnRamp PCE sends back Module generated docs and configuration files.
• Once deployed: • Admin can ‘test’ the module by trying to run it. • Admin can assign it to the users in one or more Workspaces.
8
LittleFeOnRamp PCE
OnRamp Server
Portal
A
A
https://github.com/ssfoley/onramp
OnRamp to Parallel Computing Project�Curriculum Module Life cycle�
• Users configure & run Modules • View Module and PCE documentation and instructions • Custom parameters defined in onramp_uioptions.spec• On job submission: (user defined parameters are available) • OnRamp PCE runs onramp_preprocess.py• OnRamp PCE sets up batch script and submits to queue • Batch script will run onramp_run.py• On job completion • Batch script will run onramp_postprocess.py
9
LittleFeOnRamp PCE
OnRamp Server
Portal A
https://github.com/ssfoley/onramp
OnRamp to Parallel Computing Project�Curriculum Module Life cycle�
• After job completion, users can: • View result output (stdout/stderr) • Compare results to other runs • Access files from the run and module • Resubmit the job after adjusting some parameters • (Future) Visualize, graph results • (Future) Share results and/or collaborate
10
LittleFeOnRamp PCE
OnRamp Server
Portal A
https://github.com/ssfoley/onramp
OnRamp to Parallel Computing Project�Roadmap�
11 https://github.com/ssfoley/onramp
• May 2015• SIGCSE Grant started (project development started)
• Sept. 2015 (alpha release) • PCEs: SLURM, LittleFe (PBS) • Server: Workspaces/Users, job launch/monitoring, basic admin. capabilities • Modules: MPI Hello World & Ring, MPI/OpenMP Area Under the Curve
(Shodor)
• Jan. 2016 (beta release) • PCEs: Rocks Cluster, Virtual Machine (scripted) • Server: Module customized submissions, full admin. Capabilities • Modules: HPL, Monte Carlo (CSinParallel), Parameter search exemplar
• March 2016 (v1.0 release) • PCEs: Cloud provisioned PCEs (scripted) • Server: Demonstrate Coaching up to Level 2
OnRamp to Parallel Computing Project�
The OnRamp Project, provides a web-based portal that coaches users through interactive tutorials that teach them about the
software ecosystem and parallel computing while allowing them to launch & explore parallel applications from day one.