Http://interactive-grid.eu [email protected] .
Dec 18, 2015
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
e http://interactive-
grid.eu
Interactive gridaccess for
Matlab
Marcus HardtSCC @ FZK
e http://interactive-
grid.eu
Idea: Computer power <=> Electrical power
From Electrical power grid => computational grid
• Across organisational domains / countries• Transparent access to
● Computing● Data● Network
• Large scale installations
Grid Computing
e http://interactive-
grid.eu
Grid middleware
Middleware := Softwarelayer between
application and operating system
gLite: one grid middleware●Development driven by CERN●Tools for data+computing of new accelerator●10 PB/year * 20 years, random access● Job based:
• Job = Complete application + description• Send job to remote compute center• Get result back after job is finished
e http://interactive-
grid.eu
Using a lightbulb in the job based grid world Describe the lightbulb
Voltage, Watts, AmountLighting_time, ...
e http://interactive-
grid.eu
Describe the lightbulbVoltage, Watts, AmountLighting_time, ...
Submit request for electricity to broker
Using a lightbulb in the job based grid world
e http://interactive-
grid.eu
Describe the lightbulbVoltage, Watts, AmountLighting_time, ...
Submit request for electricity to broker
=> Powerplant chosen for you
Using a lightbulb in the job based grid world
e http://interactive-
grid.eu
Describe the lightbulbVoltage, Watts, AmountLighting_time, ...
Submit request for electricity to broker
=> Powerplant chosen for you=> Send lightbulb to powerplant
Using a lightbulb in the job based grid world
e http://interactive-
grid.eu
Describe the lightbulbVoltage, Watts, AmountLighting_time, ...
Submit request for electricity to broker
=> Powerplant chosen for you=> Send lightbulb to powerplant=> Wait for electricity
Using a lightbulb in the job based grid world
e http://interactive-
grid.eu
Describe the lightbulbVoltage, Watts, AmountLighting_time, ...
Submit request for electricity to broker
=> Powerplant chosen for you=> Send lightbulb to powerplant=> Wait for electricity=> Lightbulb glows
Using a lightbulb in the job based grid world
e http://interactive-
grid.eu
Describe the lightbulbVoltage, Watts, AmountLighting_time, ...
Submit request for electricity to broker
=> Powerplant chosen for you=> Send lightbulb to powerplant=> Wait for electricity=> Lightbulb glows
Results come back
Using a lightbulb in the job based grid world
e http://interactive-
grid.eu
Describe the lightbulbVoltage, Watts, AmountLighting_time, ...
Submit request for electricity to broker
=> Powerplant chosen for you=> Send lightbulb to powerplant=> Wait for electricity=> Lightbulb glows
Results come back ●About 20% of the bulbs broken
Using a lightbulb in the job based grid world
e http://interactive-
grid.eu
Yes!We submit a cable-job !
Is interactivity a solution?
e http://interactive-
grid.eu
The interactive channel
A “cable” connects user with resource
e http://interactive-
grid.eu
Client interface for Java, C, Fortran, Matlab, Octave, ...
Easy to use:y=problem(x) <=> y=gs_call('problem',
x)●Transport input parameters to remote side●Execute “problem”●Transport result back
GridSolve interface
=> Reduce complexity of the grid to one function call
e http://interactive-
grid.eu
Life-Demo
Two concurrent runs of ”backpropagation” algorithm
Local part:●Matlab computes ”backpropagation” on my laptop
Grid part:●Matlab computes ”backpropagation” using the grid ●We use
• GridSolve interfaces (used in backpropagation_parallel)
• Migrating Desktop (MD): grid Integration environment • Grid resource allocation (via jobs from MD)
e http://interactive-
grid.eu
We can...... use grid resources from Matlab... compute more pixels... in shorter time... develop algorithms faster
Download a movie of the life demonstration:●http://marcus.hardt-it.de/grid4matlab
Life-Demo
e http://interactive-
grid.eu
Summary
Current status●Grid useable within Applications (like Matlab)
• Interactively• Without much grid specific knowledge
Work in progress● Improve minor itches with GridSolve●Simplify grid allocation
Future work●Software deployment●Data management● Inter process communication (MPI)
e http://interactive-
grid.eu
What's missing?
Seamless●Don't compile standalone application
Interactive●No overhead (< 10 s)●No manual data movement
From Matlab●Run Matlab-functions remotely
Goal:●Seamless● Interactive●Grid access●From
matlab
Example:Large Excel Table
●Excel must run locally
●Computation in the grid
e http://interactive-
grid.eu
USCT Algorithm
Characteristicts:● Input: 20 GB (full set)●Computing time depends
• on output size / resolution• amount of input data
Matlab●Strategic development platform (95% sourcecode)
Goals for grid access:●Seamless● Interactive●from Matlab
35MB 20GB 20GB DataVoxels
1 Hour 1.5 Months 150 Years Time40962 1282x100 40962x3410
e http://interactive-
grid.eu
GridSolve (GS)/gLite integration
Send GS-servers to gLite clusters●Package GridSolve + My software●Send packages into gLite jobs● Install packages on WorkerNodes (WN)
Create GS-service hosts (GS-agent) Ensure network connectivity
●GS-client, GS-agent, GS-proxy, GS-server
All this happens behind the scenes!
e http://interactive-
grid.eu
How to do it?
1. Make Matlab run on gLite
2. Integrate GridSolve with gLite
=> Grid in Matlab using Gridsolve & RPCGIMGER
e http://interactive-
grid.eu
USCT reconstruction := “Black Box”
Computation takes long (days, weeks, years)
e http://interactive-
grid.eu
Initial approach to parallel execution:●Partitioning of data●Many parallel jobs
Using gLite
e http://interactive-
grid.eu
A lot of work is left to the user
Using gLite in practise
e http://interactive-
grid.eu
Using gLite
Goal:●Seamless● Interactive●Grid access●From
matlab
e http://interactive-
grid.eu
What's missing?
Seamless●Don't compile standalone application
Interactive●No overhead (< 10 s)●No manual data movement
From Matlab●Run Matlab-functions remotely
Goal:●Seamless● Interactive●Grid access●From
matlab
Related Documents
