RubyKaigi 2015-DIY-Testing

DIY (Do-it-yourself)

Testing

Ruby Engineer on the drivers team.

Based in Berlin.Maintain the gems:

bson mongo mongoid bson_ext mongo_kerberos

@EmStolfo

MongoDB DriversC C++ C# Java Node.js Perl PHP Python Ruby Scala

MongoDB Drivers

Interface to the server.

10 official drivers available.

Many started as OS projects.

History

No specifications.Divergent codebases

and features.

MongoDB DriversReality

Server “interface” was not consistent.

MongoDB DriversReality

Organizations can experience 1 interface.

Specifications guide design and provide documentation.

Rewrites with collective knowledge.

MongoDB DriversRevamp

Server “interface” is a single product.

MongoDB DriversRevamp

Specifications

Equally valuable to the drivers team as to the community.

Server Discovery and Monitoring Spec

Logic required to make an application using MongoDB highly available.

How to validate compliance?

Requirements for validating compliance• Unit tests. • Integration tests. • Tests define as “data”. • Language-agnostic. • Reproducible scenarios.

DIY Testing

Our custom tests: Unit / Integration Build your own test framework.

define

Format - YAML Tests

Scenarios

Our approach:“DIY” testing

benefitsFormat: YAML

• Describes data. • Can translate to actions. • Most languages can use a YAML parsing

library. • Driver authors write a reusable harness. • Changes in the spec can be

communicated via additional YAML tests or changes to existing ones.

Test requirements• Unit tests. • Integration tests. • Defined as “data”. • Language-agnostic. • Reproducible scenarios.

Unit test - YAMLsingle/direct_connection_standalone.yml

Unit test - Test objectspec/support/server_discovery_and_monitoring.rb

Unit test - specspec/mongo/server_discovery_and_monitoring_spec.rb

Unit test - YAMLrs/discover_primary.yml

Test requirements• Unit tests. • Integration tests. • Defined as “data”. • Language-agnostic. • Reproducible scenarios.

Mongo Orchestration

Mongo Orchestration

Define clusters using JSON.Manipulate clusters via RESTful API.

Mongo OrchestrationHTTP server providing REST interface to manage

multiple MongoDB processes on the same machine.

Implemented in python.

What about Automation!?

• Is intended for testing. • Starts processes on one machine. • Allows fault injection. • Does not have a UI. • Has no protection against downtime. • Has one agent, so no resiliency. • Does not handle operation tasks.

Mongo Orchestration..!

Create different cluster topologies.

Single

Mongo Orchestration

Replica Set

Sharded Cluster

Setup

Single serverconfig options

Replica Setconfig options

Sharded Clusterconfig options

Define different installations in a config file

mongo-orchestration.config

Set up a Replica Sethttp://bit.ly/mo-usage

Benefits of MO• Reproducible test scenarios. • Abstracts differing configuration

options across MongoDB versions. • Uniform interface regardless of OS. • Ability to define different locations

of MongoDB installations for multi-version testing.

Multi-version testingMulti-topology testing

SSLAuthentication

Used internally

Drivers integration tests

No need for cluster managerstest/tools/mongo_config.rb(cluster manager in 1.x ruby driver)

Jenkins testing

Integration tests

Integration testsTest driver behavior. Input: cluster state

Output: driver behavior

Mongo Orchestration

Test harness (Ruby driver example)

PR #584 [WIP] Integration tests using Mongo Orchestration

YAML integration testrs/connection/primary-not-available.yml (I)

YAML integration testrs/connection/primary-not-available.yml (II)

Harness structure

1. Resource (MO cluster) 2. Specification (parsed YAML) 3. Test (can be run)

PR #584 [WIP] Integration tests using Mongo Orchestration

Resourcespec/support/mongo_orchestration/resource.rb

Specificationspec/support/mongo_orchestration/spec.rb

Testspec/support/mongo_orchestration/operation/client_operation.rb

Restarted primary integration testrs/connection/primary-restarted.yml (I)

Restarted primary integration testrs/connection/primary-restarted.yml (II)

Restarted primary integration testrs/connection/primary-restarted.yml (III)

How to build your own testing framework

Ask yourself1. Unit or integration tests? 2. Need to be language-agnostic? 3. Directly related to specs or

documentation?

Test framework entities

Resource Wrapper around

the external component.

Test framework entities

Specification The parsed YAML.

Test framework entities

State manipulator Processor of phases to get

to a certain state.

Test framework entities

Tests Can be executed.

Result is ‘pass’ or ‘fail’.

Make the test runner as generic as possible.

Install Mongo Orchestration

Resources

http://bit.ly/ruby-kaigi-diy-testing@EmStolfo