FAUST: Fail-Aware Untrusted Storage

FAUST: Fail-Aware Untrusted Storage

Christian Cachin

IBM Zurich

Idit Keidar

Technion

Alex Shraer, Technion, Israel

Joint work with:

Agenda

• Motivation

• Defining a Fail-Aware Untrusted Service

• FAUST: Fail-Aware Untrusted STorage

• Conclusions

Once upon a time…

DSN paperAlice Bob

Jlduiy baios kjshh jcouhc lj ljxc jcjjzxkcj hyah dg dgf gh hnbbv dg dgf gh hnbbv hjy v fg djjiaj djsjd ,lmcgg nhycgygb cnjik d ji khcgtu jnn mckpkk,c jjkpackm kj ihgdiash hasdg gdsihasd jdsihjhd dfg dgtrsd hjy v fg djjiaj djsjd ,lmcgg nhycgygb cnjik d mjikhcgtujnn mckpkk,c jjkpackm kj ihgdiashhasdg hays v fg djjiaj djsjd ,lmcgg nhycgygb cnjik d mjikhcg tujnn mckpkk,c jjkpackm kj ihgdiashhasdg gdsihasd jdsihjhd dfg dgtrsd hjy v fg djjiaj djsjd ,lmcgg nhycgygb cnjik d mjikhcgtujnn mckpkk,c jjkpackm kj ihgdiashhasdg dsihasd jdsihjhd dfg dgtrsd hjy v fg djjiaj djsjd ,lmcgg nhycgygb cnjik d mjikhcgtujnn mckpkk,c jjkp ackm kj ihgdiashhasdg hjy v fg djjiaj djsjd ,lm cgg nhycgygb cnjik d mjikhcgtujnn mckpkk,c packm kj ihgdiashhasdg gdsihasd jddfg dgtrsd hjy v fg djjiaj djsjd ,lmcgg nhycgygb

DSN paperAlice Bob


strong consistency (linearizability/atomicity)strong liveness (wait-freedom)

AliceBob

Don't be Evil (DobeE) Inc.DobeE OnlineDocs

sometimes (email)

But what if…Back in High school…

AliceBob

DSN paperAlice Bob


DSN paperAlice Bob


What would we like to achieve?• No bogus data

– Still, inconsistencies are possible

• Alice and Bob to detect inconsistencies – Rollback to consistent state

• When server is correct (the common case) clients should have strong consistency and liveness for their operations– Linearizability and wait-freedom

Agenda

• Motivation



• Conclusions

Eventual Consistency• Client’s operations can complete even though they

are only “weakly” consistent

• Client is notified when its operations become “strongly” consistent– May invoke other operations without waiting for notifications

• Eventual consistency is widely accepted and used– Starting with Bayou (SOSP 95)– Today in commercial systems, e.g., Amazon’s Dynamo (SOSP 07)

Fail-Awareness [Fetzer & Cristian 96]

• Provide notifications when the service does not guarantee its specified semantics– means that the service shouldn’t be used

• Allows clients to take appropriate recovery actions

Fail-Aware Untrusted Service• When the server is correct – strong consistency and

strong liveness

• Execution is always “weakly” consistent– at least causal consistency

• Stability notifications (next slide)

• Accurate failure notifications

• Detection completeness: All operations eventually become stable unless failure is detected

Example with Correct Server

• Alice & Bob are from Europe, Carlos is from America

• Its daytime in Europe, and Carlos is asleep

• Alice’s “stability cut”:

• Stability w.r.t all clients = perfect consistency

http://www.animationfactory.com/af_machines_factory_variant_page_compressor_gears_revolving.html

Agenda

• Motivation



• Conclusions

Fork consistency [Mazières&Shasha, PODC 02]

Server may present different views to clients “Fork” their views of the history

In such case, their views are forked ever after (never again see each others new updates) or else, server is exposed as faulty

Fork-Consistency – an Example

read(X1) → null

read(X1) → nullwrite(X1,u)

write(X1,u)C1

read(X3) → nullC1

C2

C3

write(X3,w)

start (X1= X2 = X3 = null)

read(X1) → u

read(X3) → nullread(X1) → u

write(X3, w) read(X3) → null

C1C3

C2 is forked from C1


read(X3) → null


C2

What about the common case?

C2 must receive signed context of the write Must wait until the write commits What if the write crashes? Formal proof in [Cachin, Shelat, Shraer 07]

read(X1) → null

write(X1,u)C1C1

C2

read(X1) → u

C2 is forked from C1 A Join – not allowed with fork

consistency

My context is: start(I am the first

operation)

My context is: start(I am the first

operation)

Something is wrong!

REPLY: I scheduled op1, op2, … before youvalue, signed context of corresponding write

COMMIT op (signed context)

SUBMIT write operation

Client Server

SUBMIT read operation

REPLY: I scheduled op1, op2, … before you

We need a weaker consistency notion

C2

Two weaker conditions have been defined recently: Fork-* [Li & Mazieres, NSDI 2007] Fork-sequential-consistency [Oprea & Reiter, DISC 2006]

We show: both require blocking, in executions with 1 correct server

Weak fork

At-most one join: If server forks the views of two clients, they may see at most one more common operation Same as in Fork-*

Trust server about last operations by other clients in context No checks needed Weaker than Fork-*

Causally consistent views Stronger than Fork-*

No blocking needed - allows wait-free protocols.

In fact, suffices to achieve all properties of FAUST

C2

read(X1) → null

write(X1,u)C1C1

C2

read(X1) → u

C2 read(X1) → null

write(X1,u)C1C1

C2

read(X1) → u

write(X1,v)

read(X1) → v

REPLY: I scheduled op1, op2, … before youvalue, signed context of last committed op of writer

COMMIT op (signed context)Client Server

SUBMIT read operation

Server claims that C1 committed no operations

Server can hide context of second write, but must provide context of first write

C2 detects the error

USTOR: The untrusted storage protocol

Some properties of USTOR• Linearizable executions with correct server

– A correct server schedules & processes the ops in order of receipt

• Wait-free with correct server– A read does not need to wait till some write COMMITs

– COMMIT info can arrive with SUBMIT of next operation

• All executions are causally consistent– From properties of Weak-fork

• Detects server failure, if server returns inconsistent context information

• What’s missing for FAUST?– Accurate notifications to clients: operation stability, server failure

– Completeness: for every op, eventually one of the following happens:• op indicated as stable with respect to other clients• server failure detected

From USTOR to FAUST

Application

FAUST Protocol

USTOR Protocol(client side)

Client-Server Channel Client-to-Client Comm.

PROBECONTEXTFAILURE

stabilitynotifications

server failurenotificationsread / write

read / write

SUBMIT COMMITREPLY

FAUST Layer at client Ci

• Maintain the latest known context: contexti

• Periodically (when no user ops) read data stored by others– Find out their latest known context, update contexti

– Ci always makes operations – others will see that contexti changes

• If no new info from Ck for a long time:

– send a direct PROBE message (e.g., email) to Ck

– Ck responds with CONTEXT message containing contextk

• If contextk is inconsistent with contexti

– send a FAILURE message to all– report failure to application

• Otherwise, if contextk includes j-th operation of Ci

– j-th operation reported stable at Ci w.r.t. Ck

Conclusion• Need for semantics and client-side tools for

services provided by “clouds”– Clients should be able to detect if semantics are violated

– ACM SIGACT News Column 34, Distributed Computing in the Clouds;

http://www.ee.technion.ac.il/~idish/sigactNews

• Our suggestion: Fail-Aware Untrusted Service– FAUST provides such semantics for simple storage

• When a read/write operation completes causal consistency (at least) is guaranteed

• Eventually, more information is obtained. Then:– Indicate to client that his operation preserves stronger semantics

OR: report server failure

• Common case: when server is correct, each operation completes independently of others + perfect consistency

FAUST: Fail-Aware Untrusted Storage

Documents