Data Strategies in Testing

www.beaufortfairmont.com 984.244.2313© 2016 Beaufort Fairmont, LLC All rights reserved

Why do we care about the data used by the system under test?

http://www.beaufortfairmont.com


Data StrategiesPAUL MERRILL

@dpaulmerrill



AgendaDefine Terms Identify Constraints Walk Through Examples



Define Terms



Define TermsData Strategy - the combination of code, procedure and infrastructure that affect how tests interact with data to stimulate the System Under Test (SUT).



Define TermsSigned Data - data which has been modified so that it is easily recognizable.

For example, an account with a user name like “TestUser” or an email address like

“[email protected]”


mailto:[email protected]


Define TermsConstraint - a characteristic that applies a force in a way that is limiting to an automated test or its environment.



Define TermsConstraints factor down to:

Policy Cost Test Expectations Infrastructure People



Identify Constraints



Examples of ConstraintsShared Environment

– Cost – Policy – Infrastructure

Downtime – Policy – Cost – People

Technical Skills – Cost – People



Examples of ConstraintsData Source Growth

– Cost – Policy – Infrastructure

Repeatable Tests – Policy – Test Expectations – People – Infrastructure

Data Variance – Policy



Example Data Strategies



Strategies Have 2 Parts:Creational Strategy - how and when data is created. Cleanup Strategy - the method by which a data source is returned to a benign state.



Elementary ApproachCreation - Tests assume specific data in SUT - no creational strategy Clean up - No clean-up strategy, no management of data



Elementary Approach

SUT

Execution 1

Execution 2

Execution 3

Data Source



Elementary Approach

Pros:Simple No implementation (quick)

Cons:Tests are not repeatable No isolation of data Does not work on different systems



Refresh Data Source ApproachCreation - Tests assume specific data in SUT Cleanup - Data managed by refreshing the data source prior to test execution



Refresh Data Source Approach

SUT

Execution 1

Data Source

Refresh

Execution 2

Refresh

Execution 3

Refresh



Refresh Data Source ApproachPros:

Clean test bed (data-wise) per test Uniform, one-size-fits-all process No variance of data Less time/cost to implement Total control of data

Cons:Timely Requires Downtime Blocks shared environments No variance of data Stale data over time Doesn’t work on different systems



Selfish Data GenerationCreation - Tests create the data they need in SUT Clean up - No management of data after generation May be random (e.g. via Faker) Create with batch process or in test Create via SUT or directly in Data Source Create In-test or pre-test



Selfish Data Generation

SUT

Execution 1

Execution 2

Execution 3

Data Source

Generate Data

Generate Data

Generate Data



Selfish Data GenerationPros:

Unique data per test Does not block shared environments No downtime No stale data Tests can run anywhere Tests become repeatable

Cons:Unchecked growth of data source Requires understanding of data model Tests need to know about data



Data Generation & Batch CleanupCreation - Data generated for each test Cleanup - Data cleaned up after test run Clean up may happen in data source or via SUT



Data Generation & Batch Cleanup

SUT

Execution 1

Execution 2

Execution 3Data

Source

Generate Data

Generate Data

Generate Data

Batch Clean up



Data Generation & Batch CleanupPros:

Unique data per test Does not block shared environments Minimal downtime No stale data Growth of data source kept in check Tests run anywhere Tests are repeatable

Cons:Requires deep understanding of data model Must know what to clean Significant scaffolding must be created



Data Generation & Immediate Cleanup

Creation - tests generate their own data Clean up - for each test case immediately after execution Uses a TearDown method supplied by the framework



Data Generation & Immediate Cleanup

SUT

Execution 1

Execution 2

Execution 3

Data Source

Generate Data

Generate Data

Generate Data

Clean up

Clean up

Clean up



Data Generation & Immediate CleanupPros:

Tests are repeatable Unique data per test Does not block shared environments Minimal downtime No stale data Growth of data source kept in check No data left over from other tests Tests run anywhere

Cons:Requires deep understanding of data model Must know what to clean Significant scaffolding must be created



Creational StrategiesElementary Refreshed Data

SourceData Generation

Shared EnvironmentLow Downtime

Repeatable TestsData Variance

Minimal Data GrowthSignificant Technical Skills



Cleanup StrategiesElementary (none)

Refresh Data Source

Batch Immediate (Signed, Marked or Persisted)

Shared EnvironmentLow DowntimeRepeatable TestsStrong Technical Skills



How do we know what data to clean up?



Signed DataRecord Name: “test_21323” Field Name: “Record Name” Format: “test_<random5digs>” Persist in: – a data source – code



Marked DataAn “isTestData” field or Data that is otherwise known to be test data because of its format or type, for example: – A “test user” created it – Linked to known test denotation



Internally Persisted DataKeep a list in memory: – userIds:

• 12312 • 4322 • 23423

– emailIds: • 12312 • 423423 • 234



Externally Persisted DataKeep a record in a data source: – userIds:

• 12312 • 4322 • 23423

– emailIds: • 12312 • 423423 • 234



In ReviewDefined Terms Identified Constraints Walked Through Examples



Thank you!PAUL MERRILL

@dpaulmerrill


Data Strategies in Testing

Software