AUTOMATE THYSELF Luis Majano @lmajano @ortussolutions
AUTOMATE THYSELF
Luis Majano @lmajano @ortussolutions
WHO AM I?
• Luis Majano• Computer Engineer• El Salvador • CEO of Ortus Solutions• Creator of all things Box• Sandals & Beaches -> ESRI -> Ortus
Story of Why People don’t testTesting Tools
Executing TestsAutomation
Continuous Integration
NOT ABOUT:BDD
Basics of Testing
Where to start?
Test Harness
What to test?What not to test?
WHY PEOPLE DON’T TEST
COMFORT
WHY PEOPLE DON’T TEST
New MethodologyNew Learned
Behavior
BIGGEST LIES IN SOFTWARE
Don’t worry, we will create the tests and
refactor it later!
Testing will double my estimates on time
HOW DO YOU KNOW YOUR FEATURE IS DONE?
Manual testing is only a momentary satisfaction!
HOW IT WORKS?
Bugs cost money
Cost To Fix Time detected Requirements Design Building Testing Post-Release
Time Introduced
Requirements 1x 3x 5-10x 10x 10-100x
Design --- 1x 10x 15x 25-100x
Building -- -- 1x 10x 10-25x
^ Kaner, Cem; James Bach, Bret Pettichord (2001). Lessons Learned in Software Testing: A Context-Driven Approach. Wiley. p. 4. ISBN 0-471-08112-4.
• Just do it!
• You will get dirty
• It can hurt (a little)
• Learned behavior
NO MORE EXCUSES
It will accelerate your development!
Continuous IntegrationContinuous Delivery
Continuous Improvement
AUTOMATION
AGENDA• Proven event-driven ColdFusion Framework
• Follows OO principles and best practices
• Conventions Based
• Non-intrusive framework you code in ColdFusion
• More than MVC => ColdBox Platform
• Software Toolkit
• Remote Framework
• NOT A MAGIC FRAMEWORK, YOU STILL NEED TO CODE!!
cbVue applicationinstall coldbox-samples/cbVue
TestBox Exposed
API Docs
Core
Test Browser
Test Harness
Test Runner
Samples
install testbox —saveDev
install testbox@be —saveDev
Test Harness
Automated test results!
xUnit/BDD Test Bundles
Harness bootstrap
HTML Runner
ANT Runner
Test Bundle CFCs
• Inherits from testbox.system.BaseSpec
RUN TESTS
• Execute bundles via the URL• http://localhost/tests/bundle.cfc
• SOAP Runner• HTTP/REST Runner• ANT Runner• NodeJS Runner• CommandBox Runner• Custom Runner
HOW IT WORKS?TESTBOX REPORTERS
•ANTJunit : Variant of JUnit for Jenkins•Codexwiki : Mediawiki syntax•Console : Server console•Doc : Semantic HTML•Dot : Awesome Dots•JSON•JUnit•Raw : CFML Struct•Simple : A basic HTML reporter•Text : Back to the 80’s, Text•MinText : Great for CommandBox Runners•XML •Tap : A test anything protocol reporter•Min : Minimalistic Report
Developer Automation
HOW IT WORKS?• Execute bundles via the URL
• http://localhost/tests/bundle.cfc?method=runRemote• SOAP Runner• HTTP/REST Runner• ANT Runner• NodeJS Runner• CommandBox Runner• Custom Runner
Bundle CFC Runner
HTML Runner
CommandBox Runner
Update box.json with runner location
testbox run
{ "runner": "http://127.0.0.1:57538/tests/runner.cfm", "directory": "/tests/specs", "recurse": true}
NodeJS Test Runnernpm install -g testbox-runner
create .testbox-runnerrc
1
2
Run it : testbox-runner2
HOW IT WORKS?• Execute bundles via the URL
• http://localhost/tests/bundle.cfc?method=runRemote• SOAP Runner• HTTP/REST Runner• ANT Runner• NodeJS Runner• CommandBox Runner• Custom Runner
ANT TESTING
HOW IT WORKS?DEVELOPER
AUTOMATION
1.CommandBox Watchers2.Grunt Watchers3.Grunt Watchers + testbox-runner4.Grunt Watchers + CommandBox5.ColdBox Elixir
HOW IT WORKS?
Code->Tests->Commit->Pull->Run->Results->Refactor->Repeat->
CONTINUOUS INTEGRATION
HOW IT WORKS?
Code->Tests->Commit->Pull->Run->Results->Refactor->Repeat->
CI SERVERS
• Luis Majano
• Computer Engineer
Jenkins + CommandBoxwget http://mirrors.jenkins-ci.org/war-stable/latest/jenkins.war
box server start warpath=jenkins.war
1
2
Jenkins Features
•FREE - Open Source Java application•Not so easy to configure and use•CI = Jenkins Jobs•Thousands of plugins and extensions• Jobs can be scheduled, parallelized, executed over different machines
• Jenkins Slaves•Powerful but convoluted
Travis CI - Sign Up
https://travis-ci.org/
Sign Up
1
2
Travis Features
•FREE for Open Source Projects•Runs distributed VM’s and Container Support•Triggers Build Script via git repository commits (.travis.yml)•Multiple language support•Many integrations and extensions•Many notification types•No ability to schedule/manual builds•Great for open source projects!
Accounts
Activate For Repositories
1
2
Create .travis.yml3
Commit to repository4
Travis CI - Getting Started
CI Script runs on each commit, pull request, branches
TRAVIS + ANTlanguage: javabranches: only: - developmentsudo: requireddist: trusty
before_install:- sudo apt-key adv --keyserver keys.gnupg.net --recv 6DA70622- sudo echo "deb http://downloads.ortussolutions.com/debs/noarch /" | sudo tee -a /etc/apt/sources.list.d/commandbox.list
install:- sudo apt-get update && sudo apt-get --assume-yes install commandbox- box install- box server start
script: - ant -f tests/build.xml
TRAVIS + COMMANDBOXlanguage: java
sudo: requireddist: trusty
before_install:- sudo apt-key adv --keyserver keys.gnupg.net --recv 6DA70622- sudo echo "deb http://downloads.ortussolutions.com/debs/noarch /" | sudo tee -a /etc/apt/sources.list.d/commandbox.list
install:- sudo apt-get update && sudo apt-get --assume-yes install commandbox- cd tests- box install- box server start
script:- box testbox run
TRAVIS + MULTIPLE ENGINESlanguage: java
sudo: required
env: matrix: - [email protected] - ENGINE=lucee@5 - ENGINE=adobe@2016 - ENGINE=adobe@11 - ENGINE=adobe@10
before_install:- sudo apt-key adv --keyserver keys.gnupg.net --recv 6DA70622- sudo echo "deb http://downloads.ortussolutions.com/debs/noarch /" | sudo tee -a /etc/apt/sources.list.d/commandbox.list
install:- sudo apt-get update && sudo apt-get --assume-yes install commandbox- box install
before_script:- box server start cfengine=$ENGINE port=8500
script:- box testbox run
Gitlab
https://gitlab.com/users/sign_in
Sign Up
1
2
or Self-Hosted3
Gitlab Features•All in one tool: CI, repository, docker registry, issues/milestones, etc.• Same as Travis in concept - CI Runner•Docker based•CI Runners can be decoupled to a docker swarm• Idea of CI pipelines
•Pipelines composed of jobs executed in stages• Jobs can have dependencies, artifacts, services, etc•Schedule Pipelines (Cron)• Jobs can track environments
•Great stats + charting
.GITLAB-CI.YMLimage: ortussolutions/commandbox:alpine
stages: - build - test - deploy
build_app: stage: build script: # Install dependencies - box install production=true
run_tests: stage: test only: - master # when: manual, always, on_success, on_failure script: - box server start - box testbox run
Resources• TestBox : ortussolutions.com/products/testbox
• CommandBox: ortussolutions.com/products/commandbox
• Slack: boxteam.herokuapp.com
• CFML Slack: #box-products
• travis-ci.org
• jenkins.io
• gitlab.com
• @lmajano
• @ortussolutions
Q & A