Chasing Quality In Cloud Computing Testing Different Levels Of Quality Requirements Kees Blokland [email protected] Polteq Testing Services BV, The Netherlands Download recent version from www.polteq.com
Jun 24, 2015
logo van Flair17-12-2010Polteq logo_RGB.png
Chasing Quality In Cloud Computing
Testing Different Levels Of Quality Requirements
Kees Blokland
[email protected] Polteq Testing Services BV, The Netherlands
Download recent version from www.polteq.com
logo van Flair17-12-2010Polteq logo_RGB.png
2
Going to the cloud…
Test
managemen
t
applications
CRMERP
healthfinance
(test) environments
storage
logo van Flair17-12-2010Polteq logo_RGB.png
3
bandwidth
Going to the cloud…
Test
managemen
t
applications
CRMERP
healthfinance
(test) environments
storage
virtualizationENABLERS
SOAstandard software
internet
logo van Flair17-12-2010Polteq logo_RGB.png
4
Deployment models– private cloud– community cloud– public cloud– hybrid cloud
Service Models
Cloud Computing according to NIST
Essential characteristics
On-demand service
Broad network access
Resource pooling
Rapid elasticity
Measured service
Software as a Service
Platform as a Service
Infrastructure as a Service
US: National Institute of Standards and Technology http://www.nist.gov
logo van Flair17-12-2010Polteq logo_RGB.png
5
Cloud Computing: risks and requirements
Essential characteristics On-demand serviceBroad network accessResource poolingRapid elasticityMeasured service
Deployment models– private cloud– community cloud– public cloud– hybrid cloud
Service Models
SaaS – Software as a Service
PaaS – Platform as a Service
IaaS – Infrastructure as a Service
SaaSPaaSIaaS
Security?
Performance? Legislation?
Privacy?
Vendor lock-in?
Elasticity?
Testability?
Multi platform?
User experience?
Migration? Continuity?
Integration?
logo van Flair17-12-2010Polteq logo_RGB.png
6
From risk to test
Risk groups Test groups
PerformanceSecurityContinuityFunctionalityMaintainabilityLegislation and regulationsSuppliers…
PerformanceSecurityContinuityMigrationFunctionalityMaintainability Legislation End-to-endSelectionImplementationOperation…
logo van Flair17-12-2010Polteq logo_RGB.png
7
Risk Groups – so far
PerformanceSecurityContinuityFunctionalityMaintainabilityLegislation and regulationsSuppliers…
PerformanceSecurityContinuityMigrationFunctionalityMaintainability Legislation End-to-endSelectionImplementationOperation…
logo van Flair17-12-2010Polteq logo_RGB.png
8
Risk group: performance
• Response times too long– insufficient concurrent users– at (un)expected peaks
• Scalability, elasticity not working• Latency too high• Bandwidth, throughput too low• Up/download speed insufficient
! Other customers! Over-book, subscription model! Slow internet connection
On-demand service
Broad network access
Resource pooling
Rapid
elasticity
Measured service
logo van Flair17-12-2010Polteq logo_RGB.png
9
Risk group: security
• Unauthorized access– administrators cloud service supplier– authorization/authentication inadequate– cyber crime, hackers, authorities– into cloud equipment building– ‘somewhere’ on the connection
• Data integrity– erased, not erased– unusable (loss of decryption key)
! Insecure internet connection! Insufficient data separation in equipment! Bring Your Own, insecure behavior users
On-demand service
Broad network access
Resource pooling
logo van Flair17-12-2010Polteq logo_RGB.png
10
Risk group: continuity
• Cloud service unavailable– % availability is not achieved– supplier bankrupt or a conflict– internet connection lost
• Fall back plan does not work
! Internet connection malfunction! Other suppliers disturb the service! Supplier redundancy failure! Business instability supplier
On-demand service
Broad network access
Resource pooling
Rapid
elasticity
Measured service
logo van Flair17-12-2010Polteq logo_RGB.png
11
• No fit on the business process• Low score on user friendliness• Not accessible everywhere• Not all mobile devices are supported• The equipment/configuration is not well performed• Customization is not well built• Integration with other systems fails
! Limitations in the Cloud Service! Bring Your Own Device, New Ways of Working! The evil Internet
Risk group: functionality
On-demand service
Broad network access
logo van Flair17-12-2010Polteq logo_RGB.png
12
Risk group: maintainability
• Cloud service not testable• Manuals are inadequate because of changes• An end-to-end test is not possible• Unclear who is to solve problems• Cloud service not adaptable to new requirements
! Cloud service changes unannounced! Cloud service not configurable! No test environment for cloud service! No helpdesk
On-demand service
How to keep it up and running?
logo van Flair17-12-2010Polteq logo_RGB.png
13
Risk group: legislation and regulations
• Violating EU data protection directive– location, security data– ownership, agreements with data processors
• Violating EU data retention directive• Bankruptcy of supplier inhibits keeping obligations• No grip on what happens to data
– warrant in other country
! Where are my data?! Conflicting or unclear legislation! Role of (unreliable) authorities
US: Patriot Act
On-demand service
Broad network access
Rapid
elasticity
logo van Flair17-12-2010Polteq logo_RGB.png
14
Risk group: supplier
• Bankruptcy, conflict• At the mercy of the supplier
– (pay-per-use) conditions change– cloud service changes
• Quality not stable, unreliable• Difficult to switch
– to another supplier– back
! Vendor lock-in, powerful supplier! No insight in quality SW development! Developments (technology, growth, take-overs, …)
There is no fit any more
Supplier unmasked
logo van Flair17-12-2010Polteq logo_RGB.png
15
Test Groups – so far
PerformanceSecurityContinuityFunctionalityMaintainabilityLegislation and regulationsSuppliers…
PerformanceSecurityContinuityMigrationFunctionalityMaintainability Legislation End-to-endSelectionImplementationOperation…
logo van Flair17-12-2010Polteq logo_RGB.png
16
Test Groups – so far
PerformanceSecurityContinuityFunctionalityMaintainabilityLegislation and regulationsSuppliers…
PerformanceSecurityContinuityMigrationFunctionalityMaintainability Legislation End-to-endSelectionImplementationOperation…End-to-end testing
Operational profilesTesting of PackagesBVA-STT-DCoT-DCT
Load & Stress
logo van Flair17-12-2010Polteq logo_RGB.png
17
Test group: performance
• What are the acceptance criteria?• Load testing• Stress testing
– not always allowed– what happens at the boundaries of the “bundle”
• Endurance test, volume test– restricted possibilities: fair use policy– monitors
• Elasticity, pay-per-use– LOAD+PCT+BVA
logo van Flair17-12-2010Polteq logo_RGB.png
18
Test group: performance
• Test cases based on load profiles
• Load profiles based on operational profiles
• Test environment = production environment
• Testing in real time– under operating conditions– with the “cloud shop open”
logo van Flair17-12-2010Polteq logo_RGB.png
19
Testing Elasticity
100
usage
time
Load profile – ‘UP’
99
100
101
Boundary values ‘UP’ Load profile – ‘DOWN’
Boundary values ‘UP’test case 1: usage=99, paid for 100test case 2: usage=100, paid for 100test case 3: usage=101, paid for 200
Boundary values ‘DOWN’test case 1: usage=101, paid for 200test case 2: usage=100, paid for 100test case 3: usage=99, paid for 100
200
max=100max=100
wantextension?
max=200200 billed
max=100100 billed
no
yes
Process Cycle Test
logo van Flair17-12-2010Polteq logo_RGB.png
20
Test group: security
• Make inventory of security measures– Internet connection
– Cloud service
– Client
http/ssl vpnwifi/wap data encryption
login identity management autorisation profile
access to building logs
IDaaS
weak passwords
authorisation
pincode mobiles
door closed patch routine
patch routine
social engineering
firewall
firewall
Security measuresAuthorisationAuthenticationTechnical facilities Security updatesBehaviour of peopleLogging
logo van Flair17-12-2010Polteq logo_RGB.png
21
Test group: security
• Testing and assessing– Assessing end-to-end security architecture
– Functional tests
– Tests by specialists
authorisation authentication encryption logs
encryption technique authentication technique
technical infrastructure
physical security
data separation
audit trailspatch update routine
hackers test audit
Specialists n
eeded
Specialists n
eeded
logo van Flair17-12-2010Polteq logo_RGB.png
22
Test group: continuity
• Testing of redundancy, fall back• Off line• Continuous end-to-end regression test
• Measuring the availability– 99.99….9%– critical moments– MTBF, MTTR
• What-if scenarios– disaster recovery– internet unavailable– …
Fail over testing with State Transition Test
logo van Flair17-12-2010Polteq logo_RGB.png
23
Test group: migration
• Where goes the data?• To/from/between cloud services
• Data repair: testing data• Testing the data conversion tool
• Data conversion– checklist – performance– security
! cloud service is
tested
! detailed planning
! sufficient tim
e
! technical knowledge
CHECKLIST MIGRATIONminimal disruptionno data lossconversion successfullyno hanging transactionsno loss due to bad data…
logo van Flair17-12-2010Polteq logo_RGB.png
24
• Testing SaaS = testing of standard software package• Testing:
– fit between cloud service and business process– configuration of the cloud service– integration of cloud service with other systems– multi client platforms– the end-to-end business process
• What is the test basis?– the old system– process descriptions, use cases– (functional) operational profiles
Test group: functionality
ClassificationTrees
ProcessModels
logo van Flair17-12-2010Polteq logo_RGB.png
25
Test group: maintainability
• Test environments– Public: none, stubs & mocks– Private: to be negotiated
• Manuals– Public: instructions for use– Private: custom manuals, also for maintenance
• Change procedure– Public: announcements supplier– Private: to be negotiated
• Helpdesk– Incident handling
logo van Flair17-12-2010Polteq logo_RGB.png
26
Test group: legislation and regulation
• Storage and processing of data– examples…
• Influence of the authorities– examples…
• How is the test manager supposed to deal with it?– ensure that it is taken into account– ensure that lawyers are involved– bridge between ICT and lawyer
what is the ris
k of non-compliancy?
example: who does n
ot use production data for testin
g?
logo van Flair17-12-2010Polteq logo_RGB.png
27
Broad role of the Test Manager
Implementation testing, testing, testing
logo van Flair17-12-2010Polteq logo_RGB.png
28
PerformanceSecurityContinuityMigrationFunctionalityMaintainability Legislation End-to-endSelectionImplementationOperation…
Implementation: what to test?
Risk groups
Test groupsPerformanceSecurityContinuityFunctionalityMaintainabilityLegislation and regulationsSuppliers…
Cloud Service selected!
logo van Flair17-12-2010Polteq logo_RGB.png
29
Broad role of the Test Manager
Selection
Implementation
risks, criteria, advice, contract
testing, testing, testing
logo van Flair17-12-2010Polteq logo_RGB.png
30
Selection: the risks
PublicSaaSPublicSaaS
Intention: introducing Cloud Computing
Cloud Risks
logo van Flair17-12-2010Polteq logo_RGB.png
31
Selection: criteria
Intention: introducing Cloud Computing
Selection criteria
Cost reductionBusiness processPerformanceScalabilityNew ways of workingContinuityMigrationSecurityIntegration…
logo van Flair17-12-2010Polteq logo_RGB.png
32
Broad role of the Test Manager
Selection
Implementation
Operation
risks, criteria, advice, contract
testing, testing, testing
end-to-end regression test, evaluation
logo van Flair17-12-2010Polteq logo_RGB.png
33
Operation: everything is moving
OperationOperation
changes in other systems
internet changes
changes in clients
changes in
business
process
changes in cloud service
changes cloud supplier
Release Calendar?Change Process?
Continuous End-to-end Test
growth
logo van Flair17-12-2010Polteq logo_RGB.png
34
Operation, role of the test manager
• Make inventory of cloud continuity risks– everything is moving!
• Periodic end-to-end testing– is it still working?
…- end-to-end-to-end-to-end-end-to-end-to-end-to-end-to-…
logo van Flair17-12-2010Polteq logo_RGB.png
35
Cloud & perspective of testing
From Risk To Test
Everything is moving
Broad Role Test Manager
End to End and the rest
logo van Flair17-12-2010Polteq logo_RGB.png
Questions?
logo van Flair17-12-2010Polteq logo_RGB.png
Thank you!