Understanding the Robustness of SSDs under Power … · Understanding the Robustness of SSDs under Power Fault ... - 4 have power-loss protection ... •Host System - Debian 6.0 w

Understanding the Robustness of SSDs under Power Fault

Joseph Tucek

Mark Lillibridge

HP Labs

Mai Zheng

Feng Qin

The Ohio State University

2

Solid-State Drives (SSDs)

- a “truly revolutionary and disruptive” technology

• Great performance

• Low power consumption

3

Solid-State Drives (SSDs)

- a “truly revolutionary and disruptive” technology

• Great performance

• Low power consumption

• *** behavior in adverse conditions ?

4

Power Faults

- a threat never gone

5

Power Faults

- a threat never gone

Jul. 2012:“POWER OUTAGE Hits London Data Center ...”

Jul. 2012:“... human error was responsible for a data center POWER OUTAGE ...”

Jun. 2012:“Amazon Data Center LOSES POWER During Storm …”

Aug, 2011:“Colocation provider Colo4 experienced a POWER OUTAGE …”

May 2010:“Car Crash Triggers Amazon POWER OUTAGE …”

Nov. 2010:“About 3,000 servers at Montreal web host iWeb experienced an OUTAGE …”

Jan. 2013:“ A POWER OUTAGE at a key New Jersey data center ...”

6

Power Faults

- a threat never gone

Jul. 2012:“POWER OUTAGE Hits London Data Center ...”

Jul. 2012:“... human error was responsible for a data center POWER OUTAGE ...”

Jun. 2012:“Amazon Data Center LOSES POWER During Storm …”

Aug, 2011:“Colocation provider Colo4 experienced a POWER OUTAGE …”

May 2010:“Car Crash Triggers Amazon POWER OUTAGE …”

Nov. 2010:“About 3,000 servers at Montreal web host iWeb experienced an OUTAGE …”

Jan. 2013:“ A POWER OUTAGE at a key New Jersey data center ...”

Potential Failures

8

Simple Failures

record

after power fault before power fault

• Bit Corruption

• Metadata

Corruption

• Dead Device

mess all data

9

• Shorn Writes

Simple Failures

• Flying Writes

1 2 disk block #

old

new

1 2

after power fault before power fault

new old

10

Serializable state

Unserializable state

block #

Write Completion Time

0 A1 2 A3 4

0 A2 2 A3 4

A3

0 1 2 3 4

Complex Failure: Unserializable Writes

A1

A2

thread A

Testing Framework

12

Switcher

Design

Workers Workers

Checker Workers

Scheduler

❶

write records ❸

read & check ❷

power off/on

Control Circuit

13

What to Write?

❶

write records ❸

read & check

0000

0000

0000 • all 0’s ?

14

What to Write?

❶

write records ❸

read & check

4239

0817

3625 • all 0’s ?

• random numbers?

15

fixed-sized header

checksum

timestamp

block#

thread_id

op_cnt

seed

… …

Special Record Format

- allows detecting all 6 types of failures

16

fixed-sized header

checksum

timestamp

block#

thread_id

op_cnt

seed

… …

Special Record Format

- allows detecting all 6 types of failures

Bit corruption & Shorn writes

17

fixed-sized header

checksum

timestamp

block#

thread_id

op_cnt

seed

… …

Special Record Format

- allows detecting all 6 types of failures

Bit corruption & Shorn writes

Flying writes

18

fixed-sized header

checksum

timestamp

block#

thread_id

op_cnt

seed

… …

Special Record Format

- allows detecting all 6 types of failures

Bit corruption & Shorn writes

Flying writes

Unserializable writes

19

fixed-sized header

checksum

timestamp

block#

thread_id

op_cnt

seed

… …

Special Record Format

- allows detecting all 6 types of failures

Bit corruption & Shorn writes

Flying writes

Unserializable writes

20

fixed-sized header

checksum

timestamp

block#

thread_id

op_cnt

seed

… …

Special Record Format

- allows detecting all 6 types of failures

Bit corruption & Shorn writes

Flying writes

Unserializable writes

regenerating records

21

fixed-sized header

checksum

timestamp

block#

thread_id

op_cnt

seed

… …

Special Record Format

- allows detecting all 6 types of failures

Bit corruption & Shorn writes

Flying writes

Unserializable writes

regenerating records

Metadata corruption &

Dead device

22

Special Record Format

- allows detecting all 6 types of failures

all 0’s?

random numbers?

duplicates of header

extendable padding fixed-sized header

checksum

timestamp

block#

thread_id

op_cnt

seed

… …

23

Advanced FTL: Compression

24

Randomization of Record Content

- avoid interference of compression

regular format

random mask

random format

25

Deriving Completion-time Partial Order

- a key step of unserializable writes detection

Time

A1’

A2

A1

B1

B2

B1’

thread A thread B

A1’

: generating time of

1st record of A

A1

: completion time of

writing 1st record of A

26

Deriving Completion-time Partial Order

- a key step of unserializable writes detection

Time

A1’

A2

A1

B1

B2

B1’

thread A thread B

Intra-thread:

A1 -> A1’ -> A2

27

Deriving Completion-time Partial Order

- a key step of unserializable writes detection

Time

A1’

A2

A1

B1

B2

B1’

thread A thread B

Intra-thread:

A1 -> A1’ -> A2

B1 -> B1’ -> B2

28

Deriving Completion-time Partial Order

- a key step of unserializable writes detection

Time

A1’

A2

A1

B1

B2

B1’

thread A thread B

Intra-thread:

A1 -> A1’ -> A2

B1 -> B1’ -> B2

Inter-thread:

A1 -> B1 -> A2 -> B2

29

Deriving Completion-time Partial Order

- a key step of unserializable writes detection

Time

A1’

A2

A1

B1

B2

B1’

thread A thread B

Intra-thread:

A1 -> A1’ -> A2

B1 -> B1’ -> B2

Inter-thread:

A1 -> B1 -> A2 -> B2

A1’ -> B1’ or

B1’ -> A1’

Conservatively report no errors

30

Deriving Completion-time Partial Order

- a key step of unserializable writes detection

Time

A1’

A2

A1

B1

B2

B1’

thread A thread B

Intra-thread:

A1 -> A1’ -> A2

B1 -> B1’ -> B2

31

Deriving Completion-time Partial Order

- a key step of unserializable writes detection

Time

A1’

A2

A1

B1

B2

B1’

thread A thread B

Intra-thread:

A1 -> A1’ -> A2

B1 -> B1’ -> B2

Inter-thread:

A2 -> B1

32

Deriving Completion-time Partial Order

- a key step of unserializable writes detection

Time

A1’

A2

A1

B1

B2

B1’

More details in our paper

& Golab et al. PODC’11

thread A thread B

Intra-thread:

A1 -> A1’ -> A2

B1 -> B1’ -> B2

Inter-thread:

A2 -> B1

A1’ -> B1’

33

Power Fault Injection

SSD

❷

power off/on

34

Power Fault Injection

SSD

Host

❷

power off/on

Results

36

Experimental Environment

• Block Devices

- 15 SSDs and 2 hard drives

- SLC & MLC

- Manufactured in 2009 – 2012

- 4 have power-loss protection

- Low-end to high-end ($0.63/GB - $6.50/GB)

• Host System

- Debian 6.0 w/ 2.6.32 kernel

- LSI Logic SAS controller

- no filesystem on devices

- Synchronized & Direct I/O (O_SYNC | O_DIRECT)

37

Summary of Observations

Failures # of SSDs

Bit Corruption 3

Metadata Corruption 1

Dead Device 1

Shorn Writes 3

Flying Writes 0

Unserializable Writes 8

None 2

• 13 of 15 SSDs exhibit failure(s)

• 2 perfect SSDs

• 5 of 6 failures observed

38

Shorn Writes: Subpage Programming

new

new

512 bytes

pattern #2

pattern #1

4K bytes

39

Serialization Errors: Avg. Numbers Per Fault

increasing price ($/GB)

0.125 0.25

0.5 1 2 4 8

16 32 64

128 256 512

1024

15 7 6 10 11 12 8 9 4 13 2 5 14

Avg

. # o

f se

rial

izat

ion

err

ors

p

er fa

ult

SSD ID

0

40

Serialization Errors: Avg. Numbers Per Fault

increasing price ($/GB)

0.125 0.25

0.5 1 2 4 8

16 32 64

128 256 512

1024

15 7 6 10 11 12 8 9 4 13 2 5 14

Avg

. # o

f se

rial

izat

ion

err

ors

p

er fa

ult

SSD ID

0

SLC

41

Serialization Errors: Patterns Over Time

1

10

100

1000

1 10 20 30 40 50 60 70 80 90 100

testing cycle #

SSD#2 SSD#4 SSD#7 SSD#8

# o

f se

rial

izat

ion

err

ors

42

• 1 SSD

• 8 injected power faults

• lost 31% (72 GB) data

Metadata Corruption

Dead Device

• 1 SSD

• 136 injected power faults

• can no long be detected by host

43

Conclusion

• An effective methodology to expose bugs in block devices under power fault

• Important implications to storage design

• e.g. write ahead logging V.S. unserializable writes

Thank you!

44

Pristine Version of Our Paper can be Found at:

http://www.cse.ohio-state.edu/~zhengm/