SUNDR: Secure Untrusted Data Repository Jinyuan LiN.Y.U. Maxwell KrohnM.I.T. David MazièresN.Y.U. Dennis ShashaN.Y.U. Slides modified for CS739 OSDI 2004.

SUNDR:Secure Untrusted Data

Repository

Jinyuan LiN.Y.U.Maxwell KrohnM.I.T.David MazièresN.Y.U.Dennis ShashaN.Y.U.Slides modified for CS739

OSDI 2004

NYU Secure Computer Systems Group

2

Motivation

File system integrity is critical sourceforge.net: 90,000+ projects, including kernel projects


3

Goal

Prevent undetected tampering with your files!


4

Current approaches

Trust system administrator to do a good job Keep up with latest security patches

Restrict accesses as much as possible

…


5

Not always reliable


6

SUNDR’s approach SUNDR is a network f/s designed for running on untrusted, or even compromised server Place trust in users authorized to modify particular files, not in server or admins maintaining server Malicious user who gains control can’t convince clients to accept altered contents because lacks write permission

SUNDR properties: Unauthorized operations will be immediately detected By whom??

If server drops operations, can be caught eventually

What doesn’t SUNDR protect against?


7

Talk Outline

Motivation Design A strawman file system SUNDR design

Implementation Evaluation


8

Setting: New Slide

Untrusted Network

Untrusted Network

SUNDR server

SUNDRclient

SUNDRclient

userA

userB

Separates administration of servers from administration of file systemSingle server hosts multiple file systems

Create file system: Create Superuser with public/private key pair; Server knows only public key

Users: Public/private key pair When move between clients, remember private key and last operation


9

Ideal File system semanticsFile system calls can be mapped to fetch/modify operations

Fetch – client downloads new data Modify – client makes new change visible to others In their implementation, when does client call fetch and modify?

“Fetch-modify” consistency: A fetch reflects exactly the modifications that happen before it

Impossible without online trusted parties

Goal: Get as close to possible to “fetch-modify” consistency without online trusted parties


10

Strawman File System

AModify f2 sig3

AModify f1 sig1

BFetch f4 sig2

AModify f2 sig3

BFetch f2 sig4

AModify f1 sig1

BFetch f4 sig2

AModify f1 sig1

BFetch f4 sig2

AModify f1 sig1

BFetch f4 sig2

sig1

sig2

AModify f2 sig3

sig3

File server

A: echo “A was here” >> /share/aaa

B: cat /share/aaa

userA

userB


11

An ordering relation

AModify f1 sig1

BFetch f4 sig2

AModify f2 sig3

BFetch f2 sig4

AModify f1 sig1

BFetch f4 sig2

AModify f2 sig3 We define ≤ relation:

LogA ≤ LogB iff LogA is prefix of LogB

A’s latest log:

B’s latest log:

LogA

LogB


12

Detecting attacks by the server

AModify f2 sig3

AModify f1 sig1

BFetch f4 sig2

BFetch f2 sig3

A: echo “A was here” >> /share/aaa AModify f1 sig1

BFetch f4 sig2

AModify f1 sig1

BFetch f4 sig2

AModify f1 sig1

BFetch f4 sig2

AModify f2 sig3

A

BB: cat /share/aaa(stale result!)

File server


13

AModify f1 sig1

BFetch f4 sig2

BFetch f2 sig3b

AModify f1 sig1

BFetch f4 sig2

AModify f2 sig3a

A’s log and B’s log canno longer be ordered:LogA ≤ LogB, LogB ≤ LogA

A’s latest log:

B’s latest log:

Detecting attacks by the server

LogA

LogB

sig1

sig2

sig3a


14

Properties of Strawman FS What problems exist with strawman approach?

High overhead (bw to send history), no concurrency

What can’t a bad server do? Bad server can’t make up operations users

didn’t perform Can’t corrupt data

What can a bad server do? Bad server can conceal users’ operations

from each other, however, will be detected if users check with each other Call this property “fork consistency”

Can refuse to serve data


15

Fork Consistency: A tale of two worlds

File Server

A’s view B’s view

…

…


16

Implications of fork consistency

Closest possible consistency to “fetch-modify” without online trusted parties Can be leveraged with online trusted parties to detect violations of “fetch-modify” consistency What are different ways this detection can be done?

users periodically gossip to check violations How do you find out about other users?

or deploy a trusted online “timestamp” box Periodically writes to one globally readable file Possible?

ABTATA ABTBTB


17

Talk Outline

Motivation Design Strawman FS SUNDR approach



18

SUNDR architecture

block server stores blocks retrievable by content-hashconsistency server orders all events

Untrusted Network

Untrusted Network

consistency server

block server

SUNDR server-side

SUNDRclient

SUNDRclient

userA

userB


19

SUNDR data structures (I) Strawman FS problem: High bandwidth and storage consumption Need to reconstruct the whole file system

Don’t ship entire history: instead ship digest of fs stateEach file is writable by one user or group Partition inodes by allowed writer Hash each partition down to a 20-byte digest (I-handle)

Fast way to check all files covered by hash are correct


20

Hash tree (1): File handle Each file is hashed into a 20-byte value using a hash tree

Blocks are stored and indexed by their content-hash on the block server

data1

Metadata

H(data1)

H(data2)

H(iblk1)

data2

data3

data4H(data3)

H(data4)

iblk1

20-byte File HandleI-hash

i-node

What is this?


21

Hash tree (2): FS digest

Hash all files writable by each user/group to a 20-byte digest

From this digest, client can retrieve and verify any block of any file (SFSRO, CFS, Pond …)

2 20-byte

3 20-byte

4 20-byte

i-table i-node 2

i-node 3

i-node 4

20-byte digest

i-num

I-handle


22

Hash tree (3): Directories (New Slide)

What do directory blocks contain? Maps name to <user/group owner, I-number>

Not I-hash --> Must use I-table

Why not point to I-node directly? Would have to modify directories (recursively) when file data changes

If change data block, what structures change? Contents of one I-node, contents of I-table, I-handle (not directory)


23

Hash tree (4): Groups(New Slide)

Why have groups? Allow multiple users to write same file Need digest guaranteed to represent file

What does group I-table do? Maps I-num to (user,I-num)

Updates group table to point to last writer

Why not point to I-node directly? Performance: Fewer changes to group I-handle when same user modifies file


24

SUNDR FS 2 20-byte3 20-byte4 20-byte

digest

2 20-byte3 20-byte4 20-byte

digest

Superuser:

UserA:

SUNDR State

How to fetch “/share/aaa”?

/:Dir entry: (share, Superuser, 3)

Lookup “/”

/share:Dir entry: (aaa, UserA, 4)

Lookup “/share”

Fetch “/share/aaa”

digestUserB: …

234

digestGroupG:


25

SUNDR data structure (II) Want server to order users’ fetch/modify operations Goal: Expose server’s failure to order operations properly

Sign version vector along with digest Version vectors will expose ordering failures


26

Version structure

Each user has its own version structure (VST)Consistency server keeps latest VSTs of all usersClients fetch all other users’ VSTs (VSL) from server before each operation and cache themWe order VSTA ≤ VSTB iff all the version numbers in VSTA are less than or equal in VSTB

VSTA

Signature A

ADigest A

A - 1B - 1G - 1

VSTB

≤ Signature B

BDigest B

A - 1B - 2G - 2

Version vector

I-handle


27

Version structure (New Slide)

Get lockGet VSL from serverUpdate your VST

Compute new handles of anything that changes (self and groups) Update version vector to match last updates according to VSL Increment version # appropriately

Perform consistency check Check old VST in VSL Ensure total global ordering for VSL

Sign VST and send back to server

VSTA

Signature A

ADigest A

A - 1B - 1G - 1

VSTB

≤ Signature B

BDigest B

A - 1B - 2G - 2

Version vector

I-handle


28

Updating VST: An exampleConsistency Server

B

AA-0

B-0

A: echo “A was here” >> /share/aaa

B: cat /share/aaa

DigA

A

A-1

B-1DigA

A

A-1

B-1DigA

A

A-1

B-1DigA

A

A-0

B-1DigB

B

A-1

B-2DigB

B

A-1

B-2DigB

B

A-0

B-1DigB

B

VSTA ≤ VSTB


29

Detecting attacksConsistency Server

BA: echo “A was here” >> /share/aaa

B: cat /share/aaa (stale!)

A

A-1

B-1DigA

A

A-0

B-0DigA

A

A-0

B-1DigA

B

A-1

B-1DigA

A A-0

B-1DigB

B

A-0

B-2DigB

B

A-0

B-2DigB

B

A’s latest VST and B’s can no longer be ordered:VSTA ≤ VSTB, VSTB ≤ VSTA

≤A-0

B-0DigA

A


30

Supporting concurrent operations

Two clients may issue operations concurrently Problem if two clients increment version numbers concurrently

Cannot order VSTs!

How does second client know what vector to sign? If operations don’t conflict, can just include first user’s forthcoming version number in VST

But how do you know if concurrent operations conflict?

Solution: Pre-declare operations in signed updates Server returns latest VSTs and all pending updates, thereby ordering them before current operation

User computes new VST including pending updates User signs and commits new VST


31

Concurrent update of VSTs

Consistency ServerA

B

update: B-2

update: A-1

A: echo “A was here” >>/share/aaa

B: cat /share/bbb

A-0

B-0DigA

A

A-1

A-0

B-1DigB

B

A-0

B-1DigB

B

A-1 B-2

A-0

B-0DigA

A

VSTA ≤ VSTB

A-1

B-1DigA

A

A-1 A-1

B-1DigA

A

A-1

A-1

B-2DigB

B

A-1B-2A-1

B-2DigB

B

A-1B-2


32

Talk Outline

Motivation Design Straw-man FS SUNDR approach



33

SUNDR Implementation

SUNDR

client daemon

Kernel

User

Client Machine Domain

xfs.ko redirector

VFS

FS operations

consistency server

block server

SUNDR server-side setup


34

Block server implementation

bstore

Index System

Permanent Log

Temporary Log

Stripe 1

Stripe 3

Stripe 2

SCSI

EIDE

(cf. Venti)


35

Evaluation Running on FreeBSD 4.9 PentiumIV 3G, 3G RAM, LAN Two configurations:

SUNDR : write updates to disk synchronously SUNDR/NVRAM : simulates effects of NVRAM

Block server benchmark STORE: 18.4MB/s (peak), 11.9MB/s (sustained) FETCH: 1.2MB/s (random), 25.5MB/s (sequential)

Esign cryptographic overhead Sign: 155us Verify: 100us


36

LFS small file benchmark

0

2

4

6

8

10

12

Create (1K) Read (1K) Unlink

NFSv2 NFSv3 SUNDR SUNDR/ NVRAM

Seconds


37

LFS multiple clients CREATE

0

5

10

15

20

25

30

35

1 2 3NFSv2 NFSv3SUNDR SUNDR/ NVRAM

Seconds

Number of users


38

Emacs installation performance

0

1020

304050

6070

Untar Config Gmake Install Remove

NFSv3 SUNDR SUNDR/ NVRAM

Seconds


39

Emacs multiple clients untar

010

2030

40

5060

7080

1 2 3

NFSv3 SUNDR SUND/ NVRAM

Seconds

Number of users


40

CVS Experiments (New Slide)

Phase SUNDR SUNDR NVRAM

NSF3 SSH

Import 13.0 10.0 4.9 7.0

Checkout

13.5 11.5 11.6 18.2

Commit 38.9 32.8 15.7 11.5

Update 19.1 15.9 13.3 11.5

Total 84.5 70.2 45.5 48.2


41

Conclusion SUNDR provides file system integrity with untrusted servers Users detect unauthorized operations immediately Users can detect consistency violations eventually How does this compare to using replication given untrusted (Byzantine) components?

Yes, SUNDR is a practical file system Performance is close to NFS; Agree?

Useful techniques to remember Merkle hash trees Version vectors

SUNDR: Secure Untrusted Data Repository Jinyuan LiN.Y.U. Maxwell KrohnM.I.T. David MazièresN.Y.U. Dennis ShashaN.Y.U. Slides modified for CS739 OSDI 2004.

Documents

log b slide

f2 sig3 b fetch f2 sig4

shareaaa b

f2 sig3 file server

log b iff log

possible slide

multiple file systems

reliable slide