HEC FSIO Workshop August 9, 2011 Communications and Protocols Active Networks and Active Object Storage John A. Chandy Department of Electrical and Computer.

HEC FSIO Workshop

August 9, 2011

Communications and ProtocolsCommunications and Protocols

Active Networks and Active Object StorageActive Networks and Active Object Storage

John A. ChandyJohn A. ChandyDepartment of Electrical and Computer EngineeringDepartment of Electrical and Computer Engineering

Janardhan Singaraju, Ajith Thamarakuzhi, Cengiz Karakoyunlu, Orko Momin, Mike Runde, Janardhan Singaraju, Ajith Thamarakuzhi, Cengiz Karakoyunlu, Orko Momin, Mike Runde, Paul WortmanPaul Wortman

Active Storage NetworksActive Storage Networks

• Active DisksActive Disks

– Intelligence at the disk can distribute computation to parallel Intelligence at the disk can distribute computation to parallel disksdisks

– Process data in streamsProcess data in streams

– Disks only have local view of dataDisks only have local view of data

• Active Storage NetworkActive Storage Network

– Network has a global view of dataNetwork has a global view of data

– Distributed caching of file system metadata and dataDistributed caching of file system metadata and data

– Redundancy optimizationsRedundancy optimizations

Active Storage Networks

HEC FSIO Workshop

August 9, 2011

Active storage networksActive storage networks

1 /3 Gbps

1/3 Gbps

1/3 Gbps1 Gbps

• An ASN is comprised of a smart switch along with intelligence An ASN is comprised of a smart switch along with intelligence embedded in the I/O network. embedded in the I/O network.

• Network Switches have global view of the data and can Network Switches have global view of the data and can perform in-stream data reduction and transformation perform in-stream data reduction and transformation operations.operations.

• ASN can enhance storage node performance as well as ASN can enhance storage node performance as well as improve the computational performance of the parallel I/O improve the computational performance of the parallel I/O systems.systems.

Active Storage Networks

HEC FSIO Workshop

August 9, 2011

Network switch topologyNetwork switch topology

• 2-dilated flattened butterfly2-dilated flattened butterfly

Active Storage Networks

HEC FSIO Workshop

August 9, 2011

1

1

2 5

9

2

3

4

3

5

6

4

7

8

6

7

8

10

11

12

13

14

15

16

Hardware ImplementationHardware Implementation

• NetFPGA board from StanfordNetFPGA board from Stanford

• 4 GigE connects4 GigE connects

• 2 SATA connectors for node to node communication2 SATA connectors for node to node communication

• PCI bus for node to node communicationPCI bus for node to node communication

Active Storage Networks

HEC FSIO Workshop

August 9, 2011

Active Storage NetworksActive Storage Networks

• Application operationsApplication operations

– Reduction operations - min/max, k-means Reduction operations - min/max, k-means clustering, searchclustering, search

– Transformational operations – streaming, sort, Transformational operations – streaming, sort,

• File System OperationsFile System Operations

– LockingLocking

– Redundancy optimizationsRedundancy optimizations

Active Storage Networks

HEC FSIO Workshop

August 9, 2011

Parallelization techniquesParallelization techniques

• Functional units are re-used on reaching the Functional units are re-used on reaching the reconfigurable hardware area limits.reconfigurable hardware area limits.

• Data level parallelism by distributing the Data level parallelism by distributing the data to several functional units in several data to several functional units in several switch elements.switch elements.

• Functional level parallelism by distributing Functional level parallelism by distributing functions to several elements. functions to several elements.

Active Storage Networks

HEC FSIO Workshop

August 9, 2011

KK-means clustering-means clustering

Active Storage Networks

HEC FSIO Workshop

August 9, 2011

Runtime per iterationRuntime per iteration

Active Storage Networks

HEC FSIO Workshop

August 9, 2011

Data searchData search

Active Storage Networks

HEC FSIO Workshop

August 9, 2011

Kmin/KmaxKmin/Kmax

Active Storage Networks

HEC FSIO Workshop

August 9, 2011

Data sortData sort

Active Storage Networks

HEC FSIO Workshop

August 9, 2011

Redundancy optimizationsRedundancy optimizations

Active Storage Networks

HEC FSIO Workshop

August 9, 2011

• RAIDed files, parity calculated in switchRAIDed files, parity calculated in switch

File lockingFile locking

• Lock table in switchLock table in switch

Active Storage Networks

HEC FSIO Workshop

August 9, 2011

Active Storage Networks

HEC FSIO WorkshopAugust 9, 2011

Active Storage NetworksActive Storage Networks

• Lessons LearnedLessons Learned

– Hardware design is hardHardware design is hard

– HW Libraries can helpHW Libraries can help

– ASNs make most sense for reductionsASNs make most sense for reductions

– Storage systems optimizations show promiseStorage systems optimizations show promise

• What needs to get doneWhat needs to get done

– Better HW designBetter HW design

– Application and FS hooksApplication and FS hooks

– When to do ASN and when to do SW?When to do ASN and when to do SW?

Active Storage Networks

HEC FSIO WorkshopAugust 9, 2011

Active Object StorageActive Object Storage

• Active DisksActive Disks

– Intelligence at the disk can distribute computation to parallel disksIntelligence at the disk can distribute computation to parallel disks

• Active Object Storage for Parallel File SystemsActive Object Storage for Parallel File Systems

– Active Disks for OSDsActive Disks for OSDs

– Use Active Storage to improve parallel file system performanceUse Active Storage to improve parallel file system performance

– Use Active Storage to improve parallel file system reliabilityUse Active Storage to improve parallel file system reliability

– Application aware storage and autonomic storage using active OSDs.Application aware storage and autonomic storage using active OSDs.

Active DisksActive Disks

• Can we use OSDs to make Active Disks a reality?Can we use OSDs to make Active Disks a reality?

– Application-aware storageApplication-aware storage

• Object attributes can give hints to the diskObject attributes can give hints to the disk

• Application specificApplication specific

– Parallel File SystemsParallel File Systems

• Felix et al. added a filtering layer to Lustre to provide active Felix et al. added a filtering layer to Lustre to provide active processingprocessing

– T10 OSD?T10 OSD?

Active Storage Networks

HEC FSIO Workshop

August 9, 2011

Active Storage Networks

HEC FSIO WorkshopAugust 9, 2011

Active Disks using OSDActive Disks using OSD

• Previous ImplementationPrevious Implementation

– Based on disc-osdBased on disc-osd

– Object-oriented (Java)Object-oriented (Java)• Attach object types to storage objectsAttach object types to storage objects

• Define methods for object typesDefine methods for object types

• New ImplementationNew Implementation

– Based on osc-osd (supported by Panasas)Based on osc-osd (supported by Panasas)

– RPC - Call functions on OSD remotelyRPC - Call functions on OSD remotely

– Execute Engines – C, Java, Python, etc.Execute Engines – C, Java, Python, etc.

Active Storage Networks

HEC FSIO WorkshopAugust 9, 2011

Active Disks using OSDActive Disks using OSD

• How do you move code from client to target within How do you move code from client to target within OSD framework?OSD framework?

– Create an object with the codeCreate an object with the code

– Each function object has a special attribute that defines Each function object has a special attribute that defines the type of associated execute enginethe type of associated execute engine

– OSD can support multiple execute enginesOSD can support multiple execute engines

Active Storage Networks

HEC FSIO WorkshopAugust 9, 2011

Active Disks using OSDActive Disks using OSD

• How do you execute the method remotely within the OSD How do you execute the method remotely within the OSD framework?framework?

– New EXECUTE FUNCTION command so that we can New EXECUTE FUNCTION command so that we can invoke a functioninvoke a function

– We use the CDB continuation to specify the parametersWe use the CDB continuation to specify the parameters

– Results (if any) returned directly or written to a new objectResults (if any) returned directly or written to a new object

From T10/08-185r5 changes to OSD-2From T10/08-185r5 changes to OSD-2

Active Storage Networks

HEC FSIO WorkshopAugust 9, 2011

Active Disks using OSDActive Disks using OSD

Client

OSD

iSCSIOSD Target C Engine

Java Engine

Function Object

.so, .class

Active Storage Networks

HEC FSIO WorkshopAugust 9, 2011

Active Disks using OSDActive Disks using OSD

• Status:Status:

– C and Java engines completeC and Java engines complete

– Python engine soonPython engine soon

– OrangeFS support for OSDsOrangeFS support for OSDs

SummarySummary

• Active storage networksActive storage networks

– Improves performance of computation kernelsImproves performance of computation kernels

– Useful in parallel file system optimizationsUseful in parallel file system optimizations

• Active storage for improved file system Active storage for improved file system performanceperformance

• Acknowledgements: NSF CCF-0621448, CCF-Acknowledgements: NSF CCF-0621448, CCF-093787093787

Active Storage Networks

HEC FSIO Workshop

August 9, 2011

Communication and ProtocolsCommunication and Protocols

Active Storage Networks

HEC FSIO Workshop

August 9, 2011

• Coherence schemesCoherence schemes

• Scalable abstractions for scientific dataScalable abstractions for scientific data

• Scalable replication, relocation, failure detection, and fault toleranceScalable replication, relocation, failure detection, and fault tolerance

• Topology aware storage layoutTopology aware storage layout

• Wide area storage access protocolsWide area storage access protocols

• Cloud storage?Cloud storage?

• Inter-stack communication?Inter-stack communication?

• Memory hierarchy?Memory hierarchy?