Designing Tests for Smart TVs Ceren Şahin Gebizli Vestel Electronics, R&D Test Architect, M.Sc. ISTQB CTAL-TA Nordic Testing Days 2017 7-9 June 2017, Tallinn, Estonia
Designing Tests for Smart TVs
Ceren Şahin Gebizli
Vestel Electronics, R&D
Test Architect, M.Sc.
ISTQB CTAL-TA
Nordic Testing Days 20177-9 June 2017, Tallinn, Estonia
Challenges
• Short time-to-market
• Limited resources
• Large code baseo 5M LOC in total
• Large modelso Thousands of states and transitions
• Importance of User
Perception
2Image ref: https://www.vestel.com.tr/
Traditional Testing
3
• Checklists based on experiences
• Lack of coverage information
• Lack of inputs for test automation
• Long time for preparing and executing
tests.
• Some critical faults are missing
Model-based Testing
Test Model
AB
C
DE
MBT Tool
A,E,B,A,EA,E,B,C,DC,D,E,B,AC,A,E,B,A
...
A,E,B,A,EA,E,B,C,DC,D,E,B,AC,A,E,B,A
...
Test Cases
• Focus on features that are mostly used
• Focus on scenarios that are mostly error-prone
• Focus on scenarios that reveal different failures
Effective test case generation;
4
System Models used for MBT
Hierarchical Markov chains defined with the MaTeLo tool (http://www.all4tec.net)
transitionprobabilities
states that can comprise sub-
models
finishstate
start state
5
Longevity Tests
6
• Detecting the mostly used
modules of TVs by getting
data from real users
• Designing a usage model
• Execution of each
generated test case on
test automation system
• Executing test cases
simulates 1 year usage of
product in 23 days.• 1 test case ~1.5 h
Image ref: https://www.vestel.com.tr/
Usage Profile
7Image ref: https://www.vestel.com.tr/
Selected Modules with
Probabilities
8
Software Module # of Visits Calculated Proability
Portal 1900 0.146
Youtube 2250 0.173
HBBTV 500 0.038
Media Browser (Video) 1750 0.134
Media Browser (Audio) 400 0.03
Media Browser (Picture) 100 0.007
PVR Ready 1000 0.076
Channel List 1750 0.134
EPG 2000 0.153
Teletext 1250 0.096
HDMI-SCART 100 0.007
Test Model
9
Test Automation
10
• Embedding test scripts to the corresponding
transitions.
• Hundreds of test cases and test scripts are
generated automatically by Model Based Test
tool.
• Test scripts are executed automatically on our
VesTA automation system.
Test Automation
11
Test Automation
12
Test Automation
13
Results and Future WorkTraditional methods:• Test cases preparation : 2d
• Test scripting of these cases : 3d
• Number of test cases: 300
• Test execution time : 15h (%100 automatic)
• Faults: 4
No idea about simulating the yearly usage
Model-based Testing:• Test design and scripting in parallel: 1d
• Number of test cases: 79
• Test execution time : 1.5h for each test case (%100 automatic)
• Faults: 10
Detection of new critical faults by simulating the 1 year usage of end-users in 23 days.
14
Conclusions
• Consumer electronics domain
• Context of an industrial case study for MBT of a
Smart TV system
• User profiles give an idea about most important
modules, features and test scenarios.
• Generated automatic tests can be run for different
software versions as regression test.
15
Thanks! Questions welcome..
16