QualiSystems Proprietary & Confidential 1 TestShell Framework Introduction 5/30/22 Standards of Excellence Richard Hsu
May 22, 2015
QualiSystems Proprietary & Confidential1
TestShell FrameworkIntroduction
Wednesday, April 12, 2023
Standards of E
xcellence
Richard Hsu
9001:2000 90003:2004
Global Software CompanySanta Clara, USALondon, UKMunich, GermanyTel-Aviv, Israel APAC
• Singapore , APAC HQ • Shanghai, China • Taipei, Taiwan• Seoul, Korea
• APAC Channels, Australia, Japan , India
Founded 2004
Flag product – TestShellTM Test-Automation Framework
“Customer First” Culture
QualiSystems – Company Overview
QualiSystems Proprietary & Confidential2
What we do - TestShell Framework
QualiSystems Proprietary & Confidential3
End-to-End framework for Test Automation that supports the entire life cycle of a test for components, devices, or systems
TestShell automates and simplifies the process of testing, allows you to focus on pure testing, and thus substantially:
Improve Product Quality
Shorten Time to Market
Reduce Expenditures
Lengthy test processes that can take up to several daysThe wide array of benchmark applications/Scripts include Hardware and SoftwareThe variety of vendors and models requiring diverse specificationsThe need to control and manage many distributed test stations Time-consuming test result collation and analysisCompound aggregation of extensive amounts of test data
Testing Challenges without Automation
QualiSystems Proprietary & Confidential4
Time Lost
Quality Impacted
Expense Overhead
Resources Unexploited
Significantly reduce Time-To-Market Full regression cycle 64 hours → 24 hours at USI (環隆電氣 )Single test cycle 15 days → 1.5 day at Cameo (友勁科技 )
Significantly reduce Operational ExpendituresResources involved in test creation down by 50% at Deutsche TelekomSingle test cycle effort 2 Days → 5 hours at Telstra
Managing and Improving Quality ControlDaily Build and regression at Ceragon NetworksCustomer service centers at CrowSaving X10 on RMA at Syneron
Proven ROI within the first year of use
QualiSystems Proprietary & Confidential5
QualiSystems Proprietary & Confidential6
Our Customers – across the value chain
End-to-End Test Automation Framework
7 QualiSystems Proprietary & Confidential
Write Tests & Regressions
Create reusable functions
Control Test Assets
Coordinate Test Execution
Launch Test Stations
Run Tests & Regressions
Collect Logs & Results
Monitor Real-time
Aggregate Results
Customized Reports
Customized Dashboards
Create
ExecuteSchedule
Analyze
CentralManagement
Tests & Regressions
Assets
Equipment
Test Results
TestShell Platform Details
QualiSystems Proprietary & Confidential8
End-to-End Test Automation Framework
9 QualiSystems Proprietary & Confidential
Create
ExecuteSchedule
Analyze
CentralManagement
• Messages, Instructions• Alerts, Remarks• Watchers, Graphs• etc.
Dialogs
• Variables• Cases, Loops, Delays• Parallelization, events• etc.
Logic Modules
• Traffic generators• Network analyzers• Spectrum, scope • Generators, etc.
T&M Modules
• Symbian• Windows Mobile• iPhone, Blackberry• Android
Mobile Modules
• Mail & SMS, File Engine• GUI Automation, Attributes• Requirements Management,
QTP ,LR , Etc..
Utilities
• Telnet, SSH, TCP, UDP• SNMP (& Traps), FTP• Traffic Generators, Switches• etc.
Networking
• Web GUI• Web Services• etc.
Web Modules
• Exe, DLLs, CLI, VIs• Script-Engines• TCL, Perl, Python, VB, Java• MatLab, OCR
Scripts & Software
TestShell SystemInterfaces to Software and hardware compnents (Any many more..)
QualiSystems Proprietary & Confidential11
Testing in the All IP World - Example Scenario
Typical test flow for validating :
Allocate all testing resources
Configure the Device Under Test
Configure Traffic Generator(s)
Applying traffic to the DUT and
sniffing /getting statistics
Analyzing
Typical tests:Traffic blasting, protocol emulation, load & stress,
conformance , acceptance, interoperability, etc
Typical validations:
End-to-End: Path Recovery, Clock Recovery,
Delay, Latency, Throughput, Loss Ratio,
Information-leakage, routing/switching performance, etc
Start
Setup Topology
Run Traffic
Analyze Results
End
Configure Traffic Generator (ex. IXIA
, SPIRENT)
Configure DUT
QualiSystems Proprietary & Confidential12
Testing in the All IP World - Example Scenario
Multiple Network Elements from different vendors such as:
Ethernet SwitchOSSWireless devices (Access Points)WiMAX/4G (STB)Multi Service RouterHigh Speed RouterBBRARDSLAMGatewayModem
Start
Setup Topology
Run Traffic
Analyze Results
Store & Report Data
End
Configure Traffic
Configure DUT
QualiSystems Proprietary & Confidential13
Testing in the All IP World - Example Scenario
Various network protocols and interfaces
Start
Setup Topology
Run Traffic
Analyze Results
Store & Report Data
End
Configure Traffic
Configure DUT
TelnetSSH
SFTP
TCP
Serial
UDP
SNMP
SNMP Traps
GPIB
HTTP
VISA
SOAP
SMTP
FTP
TFTP
QualiSystems Proprietary & Confidential14
Testing in the All IP World - Example Scenario
Multiple test equipment typesfrom different vendors
Traffic GeneratorsNetwork EmulatorsNetwork SniffersNetwork Analyzers Baseband EmulatorsLogic analyzersSignal analyzersOscilloscopesSignal generators
Start
Setup Topology
Run Traffic
Analyze Results
Store & Report Data
End
Configure Traffic
Configure DUT
QualiSystems Proprietary & Confidential15
Testing in the All IP World - Example Scenario
DEMO
TestShell Common Scenario for Network
QualiSystems Proprietary & Confidential16
DEMO 1: Router Testing via Traffic Generator- Configure DUT by Telnet
Check aliveCommand line
Configure Router/Switch
TelnetRun TrafficDriver
Analyze ResultsDriver
Traffic Sender
Traffic Receiver
DataData
Rou
ter
DEMO 1: Router Testing via Traffic Generator (cont.)Code Free Test Creation as easy as Visio
Studio FlowVisio Flow
18 QualiSystems Proprietary & Confidential
DEMO 2: AP Router Testing via Traffic Generator- Configure DUT by Website(GUI)
Check aliveCommand line
Configure RouterWebsite(GUI)
Run TrafficDriver
Analyze ResultsDriver
Traffic Sender
Traffic Receiver
DataData
Special features for Wireless Device:1. Use different configuration files to complete
diverse testing, ex. different mode(802.11n,b,g), different channel(1~11) or different encryption type(WEP, TKIP).
2. Use excel file to maintain the path of configuration files and import data of excel file by “File Engine” on Testshell.
3. TestShell’s GUI Automator can provide “Inspect” and “Valuate” feature to check states of web screen.
4. TestShell can get snapshoot of testing result and export to report automatically.
DEMO 2: AP Router Testing via Traffic Generator (cont.)
- Configure DUT by Website(GUI)
QualiSystems Proprietary & Confidential20
Drag & Drop Test Creator – non programmers can join the automationReusable Building BlocksRecord and Replay ActionsIntegration with both Traffic Generators and T&MSupported interfaces:Network Protocols (Telnet, SSH, Serial, TCP, UDP, SNMP, etc)GUI – Web, Java, Win appLogic and DialogIntegration with external files (configuration and results)Resource Asset Management & DeploymentDocumentationScriptsExecutablesDlls
Tests Creation - Benefits
End-to-End Test Automation Framework
22 QualiSystems Proprietary & Confidential
Create
ExecuteSchedule
Analyze
CentralManagement
QualiSystems Proprietary & Confidential23
Scheduler Snapshot
Tests scheduling – overnight & weekend
Resource management & Equipment utilization
Automatic scheduling according to time, station and priority
Test scheduling across single or multiple stations
Advanced "Outlook-like" interface
Virtual station locking
Fully customizable execution rules
Automatic real-time execution conflict handling
Tests Scheduling – Scheduler
End-to-End Test Automation Framework
25 QualiSystems Proprietary & Confidential
Create
ExecuteSchedule
Analyze
CentralManagement
Execution planningSelect Test or RegressionSelect Unit Under Test and TopologyRecord Admin informationSerial, Version, Batch etc… (Dynamic)
RunComplete RunStep by StepAutomatic troubleshooting and action executionReal-time data tracking
Standardize data collectionData transfer guarantee
Tests Execution – Runner
End-to-End Test Automation Framework
27 QualiSystems Proprietary & Confidential
Create
ExecuteSchedule
Analyze
CentralManagement
Melbourne
3,3 V
12 mA
51 kb/s81,9101,3
MHz3E-07128 ms
21°C69%11,9 mA
Test Analysis - InSightCentral data collection
5,3Canbara3,2 V
57 kb/sSydney
21,6°C19 W102,1
MHz
Multi-Site Repository
Test Lab 1 Test Lab nTest Lab 2
Data from all test labs and stations is collected and stored in a central server
Data is then ready for comprehensive analysis, using TestShell InSight
www
Test Analysis - InSightOnline customizable display
The analayzed data is presented via online Dasboards:• Providing real time analysis• Reviewed any time anywhere
www
The dashboards are completely customizable according to user preferences, supporting multiple analysis & display options, including OLAP
End-to-End Test Automation Framework
30 QualiSystems Proprietary & Confidential
Create
ExecuteSchedule
Analyze
CentralManagement
Test Management modules - Snapshot
QualiSystems Proprietary & Confidential31 QualiSystems Proprietary & Confidential
Central repository across the organization
Tests and Regression management
DUT management
Fully scalable and open frameworkTests version control Permission and user control
TestShell Components
QualiSystems Proprietary & Confidential32
TestShell Applications
QualiSystems Proprietary & Confidential33
TestShell Studio
TestShell Runner
TestShell InSight
TestShell Server
Create
Execute
Analyze
ManageTestShell SchedulerTestShell Controller
QualiSystems Proprietary & Confidential34
Thank you
To Schedule a Live Demonstration go to: www.qualisystems.com/demo