Issues in (Financial) High Performance Computing John Darlington Director Imperial College Internet Centre Fast Financial Algorithms and Computing 4th July 2007
Jan 03, 2016
Issues in (Financial) High Performance Computing
John DarlingtonDirector
Imperial College Internet Centre
Fast Financial Algorithms and Computing4th July 2007
Issues in (Financial) High Performance Computing
John DarlingtonDirector
Imperial College Internet Centre
Fast Financial Algorithms and Computing4th July 2007
• Building (rebuilding) complex applications
• Component-based software
• Mapping to complex (dynamically changing) machines
• Intelligent implementation and resource selection
• Resource procurement and maintenance
• On-demand utility computing
• Capacity planning
• Stochastic optimisation
• Building (rebuilding) complex applications
• Component-based software
• Mapping to complex (dynamically changing) machines
• Intelligent implementation and resource selection
• Resource procurement and maintenance
• On-demand utility computing
• Capacity planning
• Stochastic optimisation
• Abstraction Levels for Components
= Roles= Sources of Information ~ captured as meta-data
• Developer (Algorithm Designer)• Produces Implementations
• Scientist (Analyst)• Designs Useful Components
• User (Front Desk Trader or Business Team)• Composes and Executes Application
• Abstraction Levels for Components
= Roles= Sources of Information ~ captured as meta-data
• Developer (Algorithm Designer)• Produces Implementations
• Scientist (Analyst)• Designs Useful Components
• User (Front Desk Trader or Business Team)• Composes and Executes Application
Component Abstraction LevelsComponent Abstraction Levels
Separation of Concerns: Meaning, Behaviour & Implementation
Separation of Concerns: Meaning, Behaviour & Implementation
LinearSolver
Matrix
VectorVector
Matrix
VectorJacobi
Matrix
Vector VectorLU
Vector
Push Model Pull Model
Parallel LU Sequential LU
End-User: Application StructureEnd-User: Application Structure
Composes application Connections between ports Ports are typed
Composes application Connections between ports Ports are typed
Mesh Generator
Finite Element Model
Time Integrator
TemporalBoundaries
SpatialBoundaries
NumericalSolverTime Dependent PDE
Scientist: Component Meaning Scientist: Component Meaning
Defines Component Abstraction Ports
Meaning of methods Data Flow
Ontology of component Semantic Constraints
e.g. conditional numerical stability
Defines Component Abstraction Ports
Meaning of methods Data Flow
Ontology of component Semantic Constraints
e.g. conditional numerical stability 2
2min
2
crit dhc
tt
Implementer: Performance CharacteristicsImplementer: Performance Characteristics
Implements component Information to support decision making:
Communication Properties Performance Model (More Later) Platform Constraints Required Libraries (e.g. BLAS, LAPACK, MPI)
Information to enable execution: Configurations Execution Environment
Implements component Information to support decision making:
Communication Properties Performance Model (More Later) Platform Constraints Required Libraries (e.g. BLAS, LAPACK, MPI)
Information to enable execution: Configurations Execution Environment
Deployment of ComponentsDeployment of Components
ComponentDesignTools
Scientist Developer
ImplementationAnnotating
Tools
Repository
ApplicationDesignTools
End User
Application DescriptionDocument
Code CodeCode
ApplicationMapper
Code
Run-TimeRepresentation
Grid
Container
Access ResourceInformation
APO
Application Proxy Object
RTR
Code
Information FlowInformation Flow
Run TimeRepresentation
ApplicationMapper
ExecutionPlan
ComponentMeta-Data
ApplicationDescription
ImplementationMeta-Data
CompositePerformance
Model
UserRequirements
ResourceInformation
Information Capture & UtilisationInformation Capture & Utilisation
UserRequirements
Software & HardwareCapabilities
SoftwareComponentMeta-data
ResourceMeta-data
ApplicationStructure
SelectionPolicy
UserRights
Application asInteractive Service
Meaning
Implementation
Behaviour HardwareCharacteristics
UsagePolicy
ResourceState
CompositeApplication
Workflow Model
PerformanceModel
ExecutionPlan
The Utility MarketThe Utility Market
Amazon Elastic Compute Cloud (Amazon EC2) - Limited Beta
Amazon Elastic Compute Cloud (Amazon EC2) is a web service that provides resizable compute capacity in the cloud. It is designed to make web-scale computing easier for developers
Infr
astr
uctu
re P
rovi
ders
Infr
astr
uctu
re P
rovi
ders
Util
ity P
rovi
ders
Util
ity P
rovi
ders
Ser
vice
Hos
ters
Ser
vice
Hos
ters
Mobile Phone / PDA
Laptop
Interactive TV
Mobile Phone / PDA
Laptop
Interactive TV
Data and Services
myLifemyLife
myLifemyLife
Anatomy of the New InternetAnatomy of the New Internet
Network Computer Internet Power
Broker
The ServiceEconomy
Capacity Planning: Stochastic Optimisation (Patel, Afzal)
Capacity Planning: Stochastic Optimisation (Patel, Afzal)
deployable software
Queuing services
Require reservation
GRIDDISCOVERY
SCHEDULER
BROKER
Workflow
QoS Document
End-User
Packaged Code Services
Execution Services
Running Software Services
Payment Service
Monitoring Service
Performance Repository
Plug various scheduling algorithms within the scheduler of broker
• 2-stage stochastic programming approach• Schedule each and every task of workflows individually
• Global workload allocation• Schedule a collection of workflow jobs
• Combined• Use both techniques to perform further optimisation
• Other traditional algorithms• Uses CPLEX 9.0 to perform deterministic/stochastic optimisation• Automatically generate constraint equations and optimisation programs
QoS-constrained workflows • Simple and complex workflows, heterogenous workload• Deadline requirements (complete in time t from now)• Cost requirements (should cost only as per the budget specified)• Reliability requirements (should execute with certain reliability)• Arrive with general inter-arrival time distribution
Plug various scheduling algorithms within the scheduler of broker
• 2-stage stochastic programming approach• Schedule each and every task of workflows individually
• Global workload allocation• Schedule a collection of workflow jobs
• Combined• Use both techniques to perform further optimisation
• Other traditional algorithms• Uses CPLEX 9.0 to perform deterministic/stochastic optimisation• Automatically generate constraint equations and optimisation programs
QoS-constrained workflows • Simple and complex workflows, heterogenous workload• Deadline requirements (complete in time t from now)• Cost requirements (should cost only as per the budget specified)• Reliability requirements (should execute with certain reliability)• Arrive with general inter-arrival time distribution