True End-to-End Testing in Scala

TRUEEND-TO-END

TESTING SCALA

Orr Sella

IN

Scalapeño 2014

AGENDAWHAT ARE THEY WHY WE NEED THEM HOW TO WRITE THEM

END-TO-END TESTS

TEST HARNESS DRIVERS FAKES …

AGENDAWHAT ARE THEY WHY WE NEED THEM HOW TO WRITE THEM

END-TO-END TESTS

TEST HARNESS DRIVERS FAKES …CONCRETE EXAMPLE

github.com/orrsella/scala-e2e-testing

http://github.com/orrsella/scala-e2e-testing














BIG SUBJECT, WE CAN’T COVER EVERYTHING

Disclaimer

WhySCALA & END-TO-END TESTS

1 THE APPS Typical Scala applications are ripe for end-to-end tests !

THE LANGUAGE & ECOSYSTEM Build tools, testing libraries and language features are a great fit

2

SOME TERMINOLOGYFirst,

The Bible: GOOSINTRO TO TDD !

TESTING TECHNIQUES !

LOTS OF CODE & EXAMPLES

LEVELS OFTHETESTING KNOWN TO MAN

1 UNIT Do our objects do the right thing? !

INTEGRATION Does our code work against code we can’t change? !

END-TO-END Does the whole system work?

23

(not really)

RIGHT BALANCEWHAT ISTHE

OF THE THREE?

INVERSETHEICE CREAMCONE(or pyramid)

NUMBER OF TESTS


END-TO-END

INTEGRATION

UNIT

NUMBER OF TESTS


END-TO-END

INTEGRATION

UNIT

MANUAL TESTING (if you must)

NUMBER OF TESTS

END-TO-ENDWHAT

ARETESTS?

END-TO-END TESTSInteract with the system from the outside

Black Box

MOST IMPORTANTLY:

Exercising both the system and the process by which it’s built and deployed

Include interaction with external environment

AUTOMATICAN IDEALBUILDWOULD

1 Compile and unit-test the code

Integrate and package the system

Perform a production-like deployment to a realistic environment

Exercise the system through its external access points

234

AUTOMATICAN IDEALBUILDWOULD

1 Compile and unit-test the code

Integrate and package the system

Perform a production-like deployment to a realistic environment

Exercise the system through its external access points

234

THIS SOUNDS LIKE A LOT OF WORK

Whoa,

THIS SOUNDS LIKE A LOT OF WORK

Whoa,

IT IS.

IS IT WORTH IT?But,

IS IT WORTH IT?But,

YES. (I hope to convince you)

END-TO-ENDWHY

ARETESTS SO IMPORTANT?

REASONSHERE ARE3MAIN

(many more...)

FLUSH OUTTHE UNKNOWNS

1

Requires asking (and answering) many awkward questions

Expose uncertainty early, including technical and organizational risks

Forces to understand how a system fits into the world

Identify all the integration (read: potential failure) points

FLUSH OUTTHE UNKNOWNS

1

Requires asking (and answering) many awkward questions

Expose uncertainty early, including technical and organizational risks

Forces to understand how a system fits into the world

Identify all the integration (read: potential failure) points

EXAMPLE IN 3 SLIDES

FEEDBACK &CONFIDENCE

2

Feedback that would otherwise only show in staging/production

Safety net of system-wide regression

Assurance that our chosen technology stack (“plumbing”) works as expected

Confidence to make “risky” system-wide changes (e.g: swap datastore, application server)

FORCEDAUTOMATION

3

Everything must be automated so it can be tested

Crucial for deployment which is error-prone

Automatic Build-Deploy-Test cycle means we can ship frequently

TIME FOR A CONCRETEEXAMPLE:

TIME FOR A CONCRETEEXAMPLE:Memento

TIME FOR A CONCRETEEXAMPLE:Memento

github.com/orrsella/scala-e2e-testing


















NOTE-TAKING WEB SERVICE

Memento?


Memento?

STORE, RETRIEVE, SEARCH, TRANSLATE


Memento?

STORE, RETRIEVE, SEARCH, TRANSLATE“MICROSERVICE”


Memento?

STORE, RETRIEVE, SEARCH, TRANSLATE“MICROSERVICE”REST API

LOAD BALANCER HAPROXY

REVERSE PROXY NGINX

APP SERVER FINATRA

DATASTORE ELASTICSEARCH

TRANSLATION SERVICE YANDEX


REVERSE PROXY NGINX

APP SERVER FINATRA


TRANSLATION SERVICE YANDEXSCALA!..


REVERSE PROXY NGINX

APP SERVER FINATRA


TRANSLATION SERVICE YANDEXSCALA!..

PRETTY STRAIGHTFORWARD

WHAT CAN POSSIBLY GO WRONG & BE MISCONFIGURED?

Ok. So.

i.e. integration (failure) points


PORT 80 /ETC/HAPROXY/HAPROXY.CFG

LIST OF REVERSE PROXIES:PORTS

/ETC/DEFAULT/HAPROXY

REVERSE PROXY NGINX

APP SERVER FINATRA

PROXY PORT 7770

/ETC/NGINX/NGINX.CONF

INIT.D SCRIPT

/ETC/NGINX/SITES-AVAILABLE

JAVA

APPLICATION.CONF LOGBACK.XML

DEPLOY NEW VERSION



PORT 9300MAPPING

URLAPI KEY

/ETC/ELSATICSEARCH.YML

JAVA

(just a partial list…)

SO A LOT CAN GO WRONG

Ok,

(read: will)

SO A LOT CAN GO WRONG

Ok,

(read: will)

WE WANT OUR END-TO-END TESTS

TO EXERCISE ALL OF THIS!

HOW DO WE TEST ALL OF THISEND-TO-END?

HOW DO WE TEST ALL OF THISEND-TO-END?here is one approach…


THERE ARE OTHERS


THERE ARE OTHERS

BUT THIS IS THE BEST, OBVIOUSLY

REQUIRED FORSTEPSEND-TO-END TESTS

1

BUILD & PACKAGE Package the app for production deployment

2

3

0VIRTUAL MACHINE Fire-up a local VM instance with production-like env

CONFIGURATION & DEPLOYMENT Run production config and deployment code, having our entire system on one box

RUN TESTS Execute end-to-end tests against the VM

ORCHESTRATEHOW DO

WEALL OF THIS?

THE TEST HARNESS

Enter:

EXTENSION OF THE BUILD Runs all the necessary setup/teardown actions and the tests themselves !

STANDARD/SIMPLE/SCALA BUILD TOOL Build and package our app (sbt-native-packager) Fire-up the virtual machine (Vagrant) Run configuration and deployment scripts (Ansible)

(aka sbt)

VAGRANTMeet

PORTABLE DEV ENVIRONMENTS Lightweight, reproducible and programmable !

VIRTUALIZATION Wrapper around VirtualBox, VMWare, more !

SIMPLE & QUICK Vagrantfile => `$ vagrant up`

2 CONFIGURATION & DEPLOYMENT Run production config and deployment code, having our entire system on one box

STEPS REQUIRED FORTHEEND-TO-END TESTS

BUILD & PACKAGE Package the app for production-like deployment

3

0


1 VIRTUAL MACHINE Fire-up a local VM instance with production-like env

Test Harness:MANAGING VAGRANT

SBT

// project/Vagrant.scala!object Vagrant {! private lazy val vagrant = settingKey[Vagrant]("vagrant")! lazy val settings = Seq( test in EndToEndTest <<= (test in EndToEndTest).dependsOn(publishLocal), testOptions in EndToEndTest += Tests.Setup(() => vagrant.value.setup()), testOptions in EndToEndTest += Tests.Cleanup(() => vagrant.value.cleanup()) )}


SBT

// project/Vagrant.scala!object Vagrant {! private lazy val vagrant = settingKey[Vagrant]("vagrant")! lazy val settings = Seq( test in EndToEndTest <<= (test in EndToEndTest).dependsOn(publishLocal), testOptions in EndToEndTest += Tests.Setup(() => vagrant.value.setup()), testOptions in EndToEndTest += Tests.Cleanup(() => vagrant.value.cleanup()) )}

TEARDOWN HOOK

SETUP HOOK


SBTclass Vagrant(vagrantFile: File) {! // cli method wrappers private def up() = Process("vagrant" :: "up" :: Nil, dir)! private def provision() = Process("vagrant" :: "provision" :: Nil, dir)!! def setup(): Unit = { prevStatus = status() prevStatus match { case Running => provision() case Saved => up(); provision() case NotCreated => up() case Unknown => up() } }! def cleanup(): Unit = if (prevStatus != Running) suspend()}





Tip: VAGRANT STATE “JUGGLING”, SAVES A LOT

OF TIME!


1


3

0VIRTUAL MACHINE Fire-up a local VM instance with production-like env



DEPLOYMENTTHESCRIPTS

CONFIGURATION MANAGEMENT Ensure that our system is configured properly (OS, settings, packages, file system, etc.) !

DEPLOY THE APP Upgrade to the latest version just compiled and packaged, as would be done in production !

APPLICATION CONFIG Configure the application itself the same way it would in production (but with test values)

(Provision in Vagrant parlance)

ANSIBLE BUT ANY CM TOOL WILL DO

Specifically,

(Chef, Puppet, Shell scripts)

Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|! config.vm.box = "ubuntu/trusty64" config.vm.network "forwarded_port", guest: 7770, host: 7769 ...! config.vm.provision "ansible" do |ansible| ansible.playbook = "ansible/site.yml" ansible.inventory_path = "ansible/inventories/vagrant" end!end

Provision:VAGRANTFILE





VAGRANT INVOKES THE PROVISIONER



VIRTUAL MACHINE Fire-up a local VM instance with production-like env


3 RUN TESTS Execute end-to-end tests against the VM

2

10



VIRTUAL MACHINE Fire-up a local VM instance with production-like env



2

10

CHECK OUT CODE EXAMPLE



1

BUILD & PACKAGE Package the app for production-like deployment0VIRTUAL MACHINE Fire-up a local VM instance with production-like env


WRITING THEFinally,

TESTS THEMSELVES

GUIDELINESSOMEE2E TESTSFOR effective

TEST SPARINGLY End-to-End tests are slow; test few “sunny-day” scenarios !

ABSTRACTION Tests validate features and business logic, not implementation details !

AVOID COUPLING Tests should exercise the system from the outside without reaching for its guts

Memento:NOTES CONTROLLERTEST

class NotesControllerEndToEndTest extends Specification with NotesControllerDriver with ResponseMatchers {! "Notes controller" should { "add a note and then successfully get it" in {! val addResp = anAddNoteRequest.withText("Hello world!").execute() addResp must beOk addResp.noteId must not beEmpty! val getResp = aGetNoteRequest.withId(addResp.noteId).execute() getResp must beOk getResp.text must_== "Hello world!" } }}



USING SPECS2





TESTING THE SYSTEM BY

USING ITSELF



TEST IS DECOUPLED FROM THE CONTROLLER API ITSELF BY USING DRIVERS

Interact with the SUT directly instead of the test

Coupled to the API/protocol

Run same test with different drivers

Drivers:ABSTRACTING THE SUT


trait NotesControllerDriver {! def anAddNoteRequest = AddNoteRequest()! case class AddNoteRequest(text: String = "Lorem ipsum") extends Request[AddNoteResponse] {! val method = HttpMethod.POST val path = "/notes" val params: Map[String, String] = Map() val headers: Map[String, String] = Map() val body: Option[String] = Some("{ \"text\": \"" + text + "\" }")! def withText(text: String) = copy(text = text) } ...}



ONLY THE DRIVER INTIMATELY KNOWS THE CONTROLLER










class AddNoteResponse(response: Response)

extends BaseResponse(response)

with JsonResponse {! lazy val noteId = (json \ "noteId").

extract[String]

}







extract[String]

}

ONLY THE DRIVER KNOWS THE

RESPONSE IS JSON







extract[String]

}

ONLY THE DRIVER KNOWS THE

RESPONSE IS JSONOR THE FIELD

NAME

(back to the test…)





Matchers:ABSTRACTING THE SUT

The other side of the Driver, i.e: the response

Help decoupling from the protocol, make the test be about features

Make test more readable





trait ResponseMatchers extends Matchers {

private implicit def intToIntMatcher(t: Int): Matcher[Int] = beEqualTo(t)

! def beOk = haveStatus(200) def beBadRequest = haveStatus(400)

def beNotFound = haveStatus(404)

! def haveStatus(status: Matcher[Int])

: Matcher[Response] =

((_: Response).status) ^^ status

}




REVERSE PROXY NGINX

APP SERVER FINATRA




REVERSE PROXY NGINX

APP SERVER FINATRA



TESTINGEXTERNAL SERVICES

As always,LIFE IS ABOUT COMPROMISES AND SETTING BOUNDARIES

As always,LIFE IS ABOUT COMPROMISES AND SETTING BOUNDARIES

=> USE FAKES

FAKES FOR

Avoid making external network calls in tests

Create Fakes based on available documentation

Fakes should have minimal implementation to only support testing

Test the Fake against the real service in a contract test that’s manually run

USEEXTERNAL SERVICES

NOT EASILY AVAILABLE(Sometimes called Simplicators)

class NotesControllerEndToEndTest extends Specification with NotesControllerDriver with ResponseMatchers {! private val port = 9921 private val server = new FakeYandexTranslateServer(port)! step { server.start() }! "Notes controller" should { "add a note and then get it translated" in { ... val translated = aTranslateRequest.withId(noteId).withLang("es").execute() translated must beOk translated.text must_== "Buenos días" } }! step { server.stop() }}

class NotesControllerEndToEndTest extends Specification with NotesControllerDriver with ResponseMatchers {! private val port = 9921 private val server = new FakeYandexTranslateServer(port)! step { server.start() }! "Notes controller" should { "add a note and then get it translated" in { ... val translated = aTranslateRequest.withId(noteId).withLang("es").execute() translated must beOk translated.text must_== "Buenos días" } }! step { server.stop() }}

class FakeYandexTranslateServer(port: Int) extends SimpleHttpServer(port) {

! private val json = """{ |"code": 200, |"lang": "en-es", |"text": ["Buenos días"]

|} """.stripMargin! override protected def onSimpleReque

st(request: HttpRequest): String = json

}


REVERSE PROXY NGINX

APP SERVER FINATRA




REVERSE PROXY NGINX

APP SERVER FINATRA



WHAT YOU SHOULDDO NEXT

CONTINUE FROM HERE

How to

CHECK OUT THE EXAMPLE github.com/orrsella/scala-e2e-testing !

END-TO-END TEST NEW PROJECTS It is easier to get started with e2e tests on a new project, integrating into an existing one is harder !

READ Growing Object-Oriented Software Guided by Tests


CONTINUE FROM HERE

How to

CHECK OUT THE EXAMPLE github.com/orrsella/scala-e2e-testing !

END-TO-END TEST NEW PROJECTS It is easier to get started with e2e tests on a new project, integrating into an existing one is harder !

READ Growing Object-Oriented Software Guided by Tests


THANK YOU

True End-to-End Testing in Scala

Software

system work

risky systemwidechanges

system fitsinto

notetakingweb servicestor

n c r e t e e x

entire system

endtoend teststo e x

searc h