Software and Hardware Requirements for Next-Generation Data Analytics John Feo Center for Adaptive Supercomputing Software Pacific Northwest National Laboratory October, 2010
Dec 28, 2015
Software and Hardware Requirements for Next-Generation Data Analytics
John Feo
Center for Adaptive Supercomputing SoftwarePacific Northwest National Laboratory
October, 2010
Graphs are everywhere in science
Astrophysics Problem: Outlier detection. Challenges: massive datasets, temporal variations.Graph problems: clustering, matching.
BioinformaticsProblem: Identifying drug target proteins.Challenges: Data heterogeneity, quality.Graph problems: centrality, clustering.
Social InformaticsProblem: Discover emergent communities, model spread of information.Challenges: new analytics routines, uncertainty in data.Graph problems: clustering, shortest paths, flows.
… and in commerce
Sample queries: Allegiance switching: identify entities that switch communities.Community structure: identify the genesis and dissipation of communitiesPhase change: identify significant change in the network structureThought leaders: identify influential individuals that drive events
Graph features:Topology: Interaction graph is low-diameter and has no good separatorsIrregularity: Communities are not uniform in sizeOverlap: individuals are members of one or more communities
1000x growth
in 3 years!
has more than 300 million active users
Small-world and scale-free
Low diameter (small-world):work explodes
difficult to partition/load-balance
high % of nodes are visited quickly
“Six degrees of separation”
Scale-free (power-law):difficult to partition/load-balance
work concentrates in a few nodes
4
0.25
0.50
1.00
Blockk-way
Number of partitions
Ra
tio
of
ed
ge
s c
ut
RMAT graph with a million vertices
Grids, Erdős–Rényi, and Scale-Free GraphsUSA Roadmap
Erdős–Rényi
Scale-Free
Communication trace from execution of ½-approx weighted matching
(data distributed using Metis)
5
Challenges
Problem sizeTon of bytes, not ton of flops
Little data locality
Have only parallelism to tolerate latencies
Low computation to communication ratioSingle word access
Threads limited by loads and stores
Synchronization points are simple elementsNode, edge, record
Work tends to be dynamic and imbalancedLet any processor execute any thread
System requirements
Global shared memoryNo simple data partitions
Local storage for thread private data
Network support for single word accessesTransfer multiple words when locality exists
Multi-threaded processorsHide latency with parallelism
Single cycle context switching
Multiple outstanding loads and stores per thread
Full-and-empty bitsEfficient synchronization
Wait in memory
Message driven operationsDynamic work queues
Hardware support for thread migration
Cray XMT
Center for Adaptive Supercomputer Software
Driving development of next-generation multithreaded architectures and methods for
irregular problems
Driving development of next-generation multithreaded architectures and methods for
irregular problems
DATA
Scientific Simulations
Sensor Networks
Internet
Databases
Data Analytics
Knowledge Discovery
Trend Analysis
Science
Policy
Commerce
Sponsored by DODSponsored by DOD
Partners
Analytic methods and applications
Community thought leaders
Blog Analysis
Community Activities
FaceBook - 300 M users
Connect-the-dots
Bus
Hayashi
Zaire
Train
Anthrax
MoneyEndo
National Security
People, Places, & Actions
Semantic Web
Anomaly detection
Security
N-x contingency analysis
SmartGrid
Chapel for hybrid systems
Next generation multithreaded architectures
Communication software for hybrid systems
Performance analysis and toolsCompiler and runtime system
SmartGrid Sensor Networks
Mesh generation
N-x contingency analysisSemantic Databases
Bayesian networks Social networks
Arc
hite
ctur
eR
untim
eS
yste
mLa
ngu
ages
Met
hods
App
lica
tions
Research focus areas
MapReduce
Clustering
Computer SecurityBioInformatics
PathsShortest path
Betweenness
Min/max flow
StructuresSpanning trees
Connected components
Graph isomorphism
GroupsMatching/Coloring
Partitioning
Equivalence
Methods for data analytics
Influential FactorsDegree distribution
Normal
Scale-free
Planar or non-planar
Static or dynamic
Weighted or unweightedWeight distribution
Typed or untyped edges
Load imbalanceNon-planar
Concurrent insertsand deletions
Difficult to partition
Systems for large-scale analytics
Cray XMT
Graph resides in
XMT memory
Graph resides in
XMT memory
RDBSruns on cluster
RDBSruns on cluster
Netezza TwinFin
vap
wspd_va
tbsky 31
sky ir temp
precip-tbrg
percent_opaque
radar7
radar13
radar19
vap
wspd_va
tbsky 31
sky ir temp
precip-tbrg
percent_opaque
radar7
radar13
radar19
vap
wspd_va
tbsky 31
sky ir temp
precip-tbrg
percent_opaque
radar7
radar13
radar19
Replicate per time step
Add dependencies across time steps (not shown)
Dynamic Bayesian Network Model for Atmospheric Sensor Network Validation
Convert dynamic Bayesian network to junction tree for inferencing
Each node in the junction tree is a clique or super node containing several nodes from original Bayesian network
Junction Tree based “Evidence Propagation” is an efficient method of propagating the effect of any variable’s state to every other variable in the BN
vap
wspd_va
tbsky 31
sky ir
temp
precip-
tbrg
percent_opaque
radar7
radar13
radar19
vap
wspd_va
tbsky 31
sky ir
temp
precip-
tbrg
percent_opaque
radar7
radar13
radar19
vap
wspd_va
tbsky 31
sky ir
temp
precip-
tbrg
percent_opaque
radar7
radar13
radar19
DBN to Junction Tree Conversion
Evidence Propagation is highly irregular
Compute per node is unbalancedDegree per node is irregularData moves up and down
Loop parallelism intra-nodeTask parallelism inter-node (recursion, futures)Data flow schedulingData synchronization
SMALL SYSTEMS HAVE 100S OF MILLIONS OF NODES
Atmospheric Sensor Network Validation Framework
Semantic analysis
Understanding the relationships among data
Data intensive science
National security
Commerce
Data and relationships best expressed as triples and graphs
<John owns Dog>
18
PNNL, SNL, Cray
Patient Blue bumps
Pink rash High fever
John Yes _ Yes
Alice _ Yes _
Mary _ _ Yes
18
Blue bumps
JohnAlice
Mary
has symptom
Pink rash
has symptom
High Feverhas symptom
has symptom
Mayo Clinic’s patient database has 650K columns
XMT’s potential for semantic analysis
Machine Programming Model Performance (inferences per sec)
Author
X86, 32 nodes, 128 cores
MPI ~ 600 K inf/sec Weaver and Hendler (ISWC 2009)
X86, 64 nodes, 256 cores
Hadoop ~550K – 800K Urbani et al(ESWC 2010)
256 Treadstorm processors
C++ ~2.2M w/ read time ~13M w/o read time
RDFS closure
Inferring new relationships and attributes
Rule based
Original Diagram from Urbani et al. "Scalable Distributed Reasoning using MapReduce" ISWC 2009
JOB 3: Delete Duplicates
JOB 0: Transitive Closure
<John studied under Jim Browne> +
<Jim Browne teaches at UT Austin>
<John attended UT Austin>
865 million triples
Summary
The new HPC is irregular and sparse
Bad news: we need new architectures
Good news: there are commercial and consumer applications
Shared memory is necessary, but not sufficient
Need processors that can fill the memory system with requests
Need memory systems that support millions of simultaneous requests
Need fine-grain hardware synchronization in memory