HTTPS @Scale

HTTPS @ScaleArvind Mani, LinkedIn

Bio

● Head, Data & Infrastructure Security @ LinkedIn

● Prior - Yahoo, PayPal, Symantec, McAfee

What is HTTPS?

● HTTPS is HTTP over a channel secured by TLS or SSL.

● TLS (predecessor SSL) are crypto protocols designed to authenticate peer in client to server communication and ensure confidentiality and integrity of data.

LinkedIn HTTPS Timeline

Mar 2012 Opt-in HTTPS June 2013 team assembled (netops, security, CDNOps, traffic) to make site default HTTPS. Dec 2013 Launch default HTTPS in NLJan - Jun 2014 Rollout default HTTPS to US, EU, most of APACDec 2014 - Default HTTPS except CN and ZA

HTTPS traffic

Challenges

Mixed ContentSite SpeedScaling TLS Session UpgradeSecurity

What is it?On a https page, resource origin is insecure.

What are the consequences?MITMMixed Active Content Blocking - script, frames

Mixed Content

Measuring Mixed Content

Content-Security-Policy-Report-Only Headerdefault-src https:; img-src https: data:; report-uri https://www.linkedin.com/lite/contentsecuritySample{"csp-report":{"blocked-uri":"http://www.lixanalytics.com/foo","document-uri":"https://www.linkedin.com/profile?trk=ww","original-policy":"default-src https:; img-src https: data:; report-uri https://www.linkedin.com/csp","referrer":"","violated-directive":"default-src https:"}}

CSP Gotchas

● Set on just HTTPS response● Sample ● Exclude report requests from any page-view

rate-limiting ● unsafe-eval and unsafe-inline● May not report on NPAPI Plugins● Must Scale● No cookies even if report-uri is same domain

Fixing Mixed Content

Ads 3rd, 4th, 5th party pixels must all be https, even in iframe!Source Code Scanner fix hardcoded external links in dust templatesDynamic Scanner fix logical bugs, links constructed by backend applications

Challenges

Mixed ContentSite SpeedScaling TLS Session UpgradeSecurity

HTTP vs HTTPS

It all boils down to TLS Handshake latency!

TLS False Start

● Full handshake is 2 roundtrip

● Client validates “Finished” from server before sending Application Data

● With Client side False Start, client sends “Application Data” before validating Finished => just 1 roundtrip

● Is this secure?

TLS Session Ticket

● Pre-Master Secret

● What is in a ticket?

● Fallback is full handshake

● Is this secure?● Tickets and

PFS

Network Roundtrip

● Terminate TLS closer to user ● Where to build Points-of-Presence (PoP)? See LinkedIn presentation at Velocity 2015

● Static content - CDN map

Online Certificate Status Protocol

● Claims of 30% slow down: o DNSo TCPo OCSP request/response for chain

However:● OCSP response cached by browser (even days), ocsp

response served off CDN● Chrome disables OCSP checks by default

OCSP Stapling

Server caches OCSP response, sends in Certificate Status RequestPros● Captive Portal● Preserves browsing privacy● Supported by nginx, traffic server, Apache, etcCons● OCSP stapling (solution) not supported by mobile ● Stapling increases TLS handshake message size.

Cert Chain Size, initCwnd, SAN

Scaling TLS

TLS Termination hardware vs softwareCDN TLS capacity multi CDN strategyTLS Tuneup TLS record size, byte overhead, AES-NI

Challenges

Mixed ContentSite SpeedScaling TLS Session UpgradeSecurity

“No login” HTTP -> HTTPS Upgrade

http? ajax? bot?rampsegment

yes no no yes

● Set “migration underway” hint cookie with TTL 10 minutes (avoid repeatedly upgrading bot that refuses to be upgraded)

● 301 to HTTPS● Kill old authentication cookie+session/mint new authentication cookie+session ● Set new authentication cookie secure => new cookie was never on wire in plaintext!● Set TLS hint cookie that is NOT secure ● Implement reverse for rollback

Challenges

Mixed ContentSite SpeedScaling TLS Session UpgradeSecurity

SSL Stripping

How it Works?Attacker MITM HTTP requests. 301 or rejecting HTTP requests does not help.MitigationHost Strict Transport Security (HSTS) and preload

HSTS

Strict-Transport-Security: max-age=31536000; includeSubDomains;

preload

Window of vulnerability! Rollback with max-age=0Preload● Submit site https://hstspreload.appspot.com/ to enter

into hardcoded list ● Domain should not serve HTTP traffic● includeSubDomains (www), preload flag, max age >=

18 weeksHandle preload with care - hard to rollback

Perfect Forward Secrecy

● ClientKeyExchange - with RSA “pre-master” secret encrypted with server public key

● DH vs DHE vs ECDHE● Forward Secrecy with Session Resumption

TLS Configuration

● Disable weak/export ciphers ● Disable SSL 3.0, TLS 1.0??● Enable PFS ciphersuites● Attacks on SSL/TLS - insecure

renegotiation, BEAST, CRIME, FREAK, POODLE, logjam, heartbleed (OpenSSL implementation)

Pinning*What problem does Pinning solve?Rogue CA, some MITM, reduces trust requirements

What do you Pin? server certificate, server public key, public key of cert in chainPublic-Key-Pins-Report-Only: max-age=2592000; pin-sha256="E9CZ9INDbd+2eRQozYqqbQ2yXLVKB9+xcprMF+44U1g="; pin-sha256="LPJNul+wow4m6DsqxbninhsWHlwfp0JecwQzYpOLmCQ="; report-uri="https://other.example.net/pkp-report"

Pinning Made Easier

● Fiddler, Corporate MITM supported*● Reports on Pin failures● Backup Pin required● Testingchrome://net-internals/#hsts

EdgeNetwork Security*

● HSM● Secure Edge <-> Datacenter communication

References

● https://www.imperialviolet.org/2010/06/25/overclocking-ssl.html

● https://github.com/linkedin/adfullssl● http://www.w3.org/TR/CSP/ (note Level 2)● http://velocityconf.com/devops-web-

performance-2015/public/schedule/detail/42026

● https://blog.mozilla.org/tanvi/2013/04/10/mixed-content-blocking-enabled-in-firefox-23/

References

● https://www.igvita.com/2013/10/24/optimizing-tls-record-size-and-buffering-latency/

● https://hstspreload.appspot.com/● http://vincent.bernat.im/en/blog/2011-ssl-

perfect-forward-secrecy.html

We’re Hiring

[email protected]://www.linkedin.com/in/arvindmaniProjects● Inter-service Authentication & Access Control● Web Application Security Frameworks & Tools● Anti-scraping/DoS infrastructure● Crypto libraries & services