Your Tests Are Not Your Specs

Tests are not specsHow to write actual specifications with TLA+

Hillel Wayne@hillelogram

[email protected]

Rules• Interrupt me if I’m speaking too fast• You don’t need to understand the code samples• Ask questions if you have them• I may punt them until later

Espark LearningOr core, same thing

MDM

School

MDM

School

Business logic

TDD• Write a failing test• Fix the failing test• Refactor

Describe EmailSender context “when called” it “should send an email” it “should not send two emails”

Describe EmailSender context “when called” context “and the email API is lagging out” it “should send an email” it “should not send two emails”

Describe EmailSender context “when called” context “and the email API is lagging out” context “and there are three separate email services” context “and our AP database has a partition” it “should send an email” it “should not send two emails”

TESTS ARE NOT SPECS

Programming Languages

• I/O• Implementations• Concrete• Fast• Designed to run

Specification Languages

• No I/O• Definitions• Abstract• “Comprehensive”• Designed to check

Maximum of Setdef maximum(numbers) max = numbers.first numbers.each do |n| if n > max max = n end maxend

Maximum(numbers) == CHOOSE max \in numbers: \A n \in numbers: max >= n

Alice BobMoney

Variables alice_account = 4, bob_account = 0;

Process transfer \in {1}Variables to_transfer \in 1..5Begin Withdraw: alice_account := alice_account – to_transfer; Deposit: bob_account := bob_account + to_transfer;End process;

NoOverDrafts == alice_account >= 0

1 Withdraw Deposit

2 Withdraw Deposit

3 Withdraw Deposit

4 Withdraw Deposit

5 Withdraw Broken

Variables alice_account = 4, bob_account = 0;

Process transfer \in {1}Variables to_transfer \in 1..5Begin Start: if to_transfer <= alice_account then Withdraw: alice_account := alice_account – to_transfer; Deposit: bob_account := bob_account + to_transfer; end if;End process;

Variables alice_account = 4, bob_account = 0;

Process transfer \in {1, 2}Variables to_transfer \in 1..5Begin Start: if to_transfer <= alice_account then Withdraw: alice_account := alice_account – to_transfer; Deposit: bob_account := bob_account + to_transfer; end if;End process;

CheckWithdraw

CheckWithdraw

Deposit

WithdrawWithdrawCheck

Variables alice_account = 4, bob_account = 0;

Process transfer \in {1}Variables to_transfer \in 1..5Begin Start: if to_transfer <= alice_account then Withdraw: alice_account := alice_account – to_transfer; Deposit: bob_account := bob_account + to_transfer; end if;End process;

<>[](alice_account + bob_account = 4)

Check Withdraw Crash

Use case

• For performance reasons, we should install the app on more than one device per API call.• For stability reasons, we could never show the MDM it was incorrect

about something. So if we tell it to install an app it thinks it installed, the system wigs out.

EXTENDS Integers, TLC, SequencesCONSTANTS Devices

(* --algorithm BatchInstallvariables AppScope \in [Devices -> {0, 1}]; Installs \in [Devices -> BOOLEAN]; batch_pool = {}; lock = FALSE;

define PoolNotEmpty == batch_pool # {}end define

procedure ChangeAppScope()variables cache;begin GetLock: await ~lock; lock := TRUE; Cache: cache := batch_pool; Filter: cache := cache \intersect {d \in Devices: AppScope[d] = 0}; Add: AppScope := [d \in Devices |-> IF d \in cache THEN AppScope[d] + 1 ELSE AppScope[d] ]; Clean:

batch_pool := batch_pool \ cache; lock := FALSE; return;end procedure

fair process SyncDevice \in Devicesbegin Sync: if Installs[self] then batch_pool := batch_pool \union {self}; end if; if PoolNotEmpty then either call ChangeAppScope(); or skip; end either; end if;end process;

fair process TimeLoop = 0begin Start: while TRUE do await PoolNotEmpty; call ChangeAppScope(); end while;end processend algorithm;

Filter: cache := cache \intersect {d \in Devices: AppScope[d] = 0};Clean: batch_pool := batch_pool \ cache;

GetLock: await ~lock; lock := TRUE;

TLA+ SUCKS

Slow

Unfriendly• F[a] vs F(a) vs F a• [A -> B] vs [A |-> B]• = vs == vs := vs /= vs #• Whitespace is sometimes significant• …And More!

SPECS ARE NOT TESTS

Conclusions• Tests check implementation of code• Specs check design of system• Formal specification helps find complicated bugs• TLA+ will save your butt

Resources• http://lamport.azurewebsites.net/tla/tla.html• http://lamport.azurewebsites.net/tla/book.html• https://learntla.com (me!)

Questions?

Hillel Wayne

[email protected]

Twitter: @hillelogram

Site: www.learntla.com