Zi nginx conf_2015

TLS Cross-host Session Resumption via ngx_luaforward secrecy, best practices and more24/09/2015

Zi Lin [email protected] @lziest

#nginx #nginxconf

About Me• System Engineer at CloudFlare Inc.

• Touch everything about TLS• Nginx/OpenSSL• CFSSL• Internal system security infrastructure

2

Agenda

#nginx #nginxconf3

1

2

3

4

TLS Handshake 101Brief introduction on TLS handshake and session resumption

Cross-host session resumptionWhy it poses an engineering problem

Session resumption by ngx_luadesigns for both session id and session ticketSolve session resumption at CloudFlareHow we achieve session resumptionswith performance and forward secrecy

CC BY 2.0 image by Brenda Clarke

#nginx #nginxconf

TLS Handshake

4

source: https://youtu.be/KvxCv_yrcCY

#nginx #nginxconf

TLS HandshakeRSA

5

ClientHello

ServerHello

ClientKeyExchange

Server RandomClient Random

Encryptedpremaster secret

Premaster secret

= + +Session Key

Finished

RSA Private Key

#nginx #nginxconf

TLS HandshakeDiffie-Hellman

6

ClientHello

ServerHello

ServerKeyExchange

ClientKeyExchange

Server RandomClient Random

6

= + +Session Key

+=+=

Server DHClient DH

Finished

#nginx #nginxconf

TLS Handshake is expensive

7

#nginx #nginxconf

TLS HandshakePerformance

8

RSA Decryption ECDH

ops/sec, NIST p-curve

0

150

300

450

600

2048 40960

650

1300

1950

2600

256 384 521

ops/sec

#nginx #nginxconf9

TLS HandshakeLatency

Network Round-trip Latency

0

40ms

80ms

120ms

160ms

Local Coast to Coast

#nginx #nginxconf

TLS Session Resumption

ClientHello

ServerHello

Hi Server,Remember me?

Yup, let’s reuse the

cipher

Session Key

#nginx #nginxconf

Resumed by Session ID

11

Hi Server,Remember me?

session #2

ClientHello

ServerHello

Yup, let’s reuse the

cipher

lookup session idSession Key

#nginx #nginxconf

Resumed by Session ID

12

Hi Server,Remember me? session #100K

ClientHello

ServerHello

Yup, let’s reuse the

cipher

lookup session id

10K session/sec * 3600 sec * 100 B/session = 3.6GB

Session Key

#nginx #nginxconf

Let Clients Do Storage:Session Ticket

13

Finished

Session ticket

Keep this!

lookup ticket key=

#nginx #nginxconf

Resumed by Session Ticket

14

Hi Server,Remember this?

ClientHello

ServerHello

Yup, let’s reuse the

cipher

lookup ticket key

=Session Key

Session Ticket

#nginx #nginxconf15

Much better

#nginx #nginxconf

Nginx supports both mechanisms!But what if there are >1 servers?

16

#nginx #nginxconf

If each server maintains its own states

17

TLS Handshake

Session Resumption Fail

0

0.25

0.5

0.75

1

1 2 5 10 100

Success rate

The More, The Less Merrier

#nginx #nginxconf18

#nginx #nginxconf

CloudFlare must support both session resumption mechanisms across hosts

19

Have to, because some IEs and most Safarisdon’t support session ticket yet.

https://www.howsmyssl.com

#nginx #nginxconf

CloudFlare must support both session resumption mechanisms across hosts

20

SSL sessions %

0 15 30 45 60

ID Support Ticket Support

#nginx #nginxconf

Cross-host Session Resumption by Id

21

ClientHello

ServerHellolookup

session id

store sessionby id

#nginx #nginxconf

Cross-host Session Ticket Resumption

22

ClientHello

ServerHellolookup

ticket key

=

#nginx #nginxconf

Forward Secrecy

23

Hi Server,Remember this

know ticket key,knows all

hours laterHi Server,

Remember this

#nginx #nginxconf

Shared ticket key rotation

24

updateticket key

timestamp key

T_0

T_1

T_2

T_3

key schedule

#nginx #nginxconf

Resumed by session ticket

25

Yup, reuse the cipher

Hi Server,Remember this

btw, use this next time

=re_encrypt( )

ticket keyrotated

#nginx #nginxconf

Forward Secrecy

26

Hi Server,Remember this

???

hours laterHi Server,

Remember this

#nginx #nginxconf

Cross-host Session Resumption

27

ClientHello

ServerHellolookup

ticket key

=

time keyT_0T_1T_2T_3

updateticket key

#nginx #nginxconf

Requirements for Cross-HostSession Resumption

• Shared key store: memcached interface• Performance - Non-blocking I/O• Performance - multi-tiered caching• Security - Session encryption; Ticket key encryption;• Usability - Easy-to-maintain Configuration

28

#nginx #nginxconf

A new nginx conf module?

29

Shared Memcachedsession store

#nginx #nginxconf30

Sample solution: Augmented nginx confSet "ssl_session_cache memcached:<name>[:<host>[:<port>]]"

• Blocking I/O :(• Need to add session encryption :(• No tests. :(

#nginx #nginxconf

• Shared session store: memcached interface• Performance - Non-blocking I/O• Performance - multi-tiered caching• Security - Session encryption• Usability - Script as Configuration• Have Tests!

31

Let’s do it in ngx_lua a.k.a. Openresty

#nginx #nginxconf32

ngx_lua in a nutshellWrite C callbacks in Lua

with almost no performance hit

#nginx #nginxconf33

Follow the path of ssl-cert-by-lua @agentzh

#nginx #nginxconf

OpenSSL TLS server-side state machines

34

• OpenSSL state machine needs to have a state for non-blocking session I/O

WaitForReceive

WaitForSend

ProcessClientMessage WaitForSession

WaitForCertCallback**Need Nginx patch

#nginx #nginxconf

Non-blocking session I/Owith Nginx/OpenSSL

35

OpenSSL TLS Handshake

State Machine

• Event Handler needs to know the handshake is ongoing with session I/O

#nginx #nginxconf

Minimal changes in Nginx/OpenSSL

• OpenSSL patch, ported from BoringSSL

• Nginx patch (on top of ssl-cert-by-lua patch)

36

#nginx #nginxconf

https://github.com/openresty/lua-nginx-module

37

branch ssl-session-by-lua on top of ssl-cert-by-lua

Development in ngx_lua

Still under internal review/testing

#nginx #nginxconf

Lua scripting for Session I/O

38

cache.lua: 250 loc, 7.5kbmemc.lua: 298 loc, 8kbshdict.lua: 81 loc, 1.8kb

#nginx #nginxconf

Cross-host Session Ticket Resumption

39

ClientHello

ServerHellolookup

ticket key

=

#nginx #nginxconf

Ticket key rotation• Rotate the ticket encryption key once a while• Nginx embeds a key array as OpenSSL ex_data

40

[key_0, key_1, key_2, … key_n]

encryption decryption

[new-key, key_0, key_1, key_2, … key_n]

#nginx #nginxconf41

Ticket key rotation• Lock-free key synchronization

[key_0, key_{n}, key_{n-1}, … key_1, key_0]

[key_0, new-key, key_{n-1}, …, key_1, key_0]

[new-key, new-key, key_{n-1}, …, key_1, key_0]

[new-key, key_{n-1}, new-key, …, key_1, key_0]

[new-key, key_{n-1}, key_{n-2}, …, key_0, new-key]

Key update loop

#nginx #nginxconf

Let’s modify Nginx?• Shared ticket key store: memcached interface• Non-blocking I/O• Implementation of a timer• Ticket key encryption• Configuration

42

#nginx #nginxconf

Let’s do it in ngx_lua

43

• Shared ticket key store: memcached interface• Non-blocking I/O• Implementation of a timer• Ticket key encryption• Configuration

#nginx #nginxconf44

There is no need to modify Nginx/OpenSSL!

Only scripting in nginx.conf

Let’s do it in ngx_lua

#nginx #nginxconf45

periodicallypoll

periodicallypoll

HKG

SIN

Master

timestamp key

T_0T_1T_2T_3

timestamp key

T_0T_1T_2T_3

timestamp key

T_0T_1T_2T_3

Going further at CloudFlare

#nginx #nginxconf

A Sneak Peek intoSession resumption in TLSv1.3

46

Finished

Session ticket

Keep this!

lookup ticket key= Session ID

#nginx #nginxconf

Cross-host Session Resumption

47

ClientHello

ServerHellolookup

ticket key

=

time keyT_0T_1T_2T_3

updateticket key

id

id

#nginx #nginxconf

Team• Yichun Zhang @agentzh• Nick Sullivan @grittygrease• Shuxin Yang• Jiale Zhi @_calio• Guanlan Dai

48

#nginx #nginxconf

Recap• Cross-host TLS session resumption by id• New ngx_lua directives:

ssl_session_store_by_lua ssl_session_fetch_by_lua

• Small patches in Nginx/OpenSSL• TLSv1.3 compatible

• Cross-host TLS session ticket resumption with forward secrecy • Scripting in init_worker_by_lua and init_by_lua• No need to touch Nginx or OpenSSL• TLSv1.3 compatible

49

#nginx #nginxconf

Caching is necessary• Can’t spend 100ms for each session retrieval

• Shared memory cache - workers• Lua cache - single worker

• Worse case is pretty rare

50