Transcript
Telefónica Digital
TELEFÓNICA DIGITAL
Barcelona, March 26th 2014
Architecture DesignWeb Performance OptimizationWeb Performance Optimization
No instruments
Users Review
Late or none Performance Testing
No Real User Monitoring
Reactive Performance Tuning
No tools, no performance dashboard, performance is for sysadmins and operatorsNo tools, no performance dashboard, performance is for sysadmins and operators
Releases are costly and it may take several months of workReleases are costly and it may take several months of work
Manual testing of each release after code freezeManual testing of each release after code freeze
Non functional Requirements are most likely ignoredNon functional Requirements are most likely ignored
In production there is no monitoring of the traffic and how it affects the businessIn production there is no monitoring of the traffic and how it affects the business
Users feedback is usually negative and there is no interaction with developers and designersUsers feedback is usually negative and there is no interaction with developers and designers
Application’s performance affects directly to market’s performanceApplication’s performance affects directly to market’s performance
• Continuous Integration– Functional testing– Automation– Monitoring
Continuous Integration for functional testing is working already in nightly buildsContinuous Integration for functional testing is working already in nightly builds
Automation reduces time to market for the applicationsAutomation reduces time to market for the applications
Monitoring the real user behaviour and not just healthcheck of serversMonitoring the real user behaviour and not just healthcheck of servers
Error and risks managementError and risks management
Tuning and bugfixingTuning and bugfixing
Listening to user feedbackListening to user feedback
The FutureThe Future
• Continuous Performance Integration– Performance tests integrated in Jenkins– Automation of the trend reports– Real User Monitoring Real time feedback
Telefónica DigitalProduct Development & Innovation
AGILE PERFORMANCE
20
Pruebas de RendimientoPruebas de RendimientoProactive Performance testing for each release. Load tests will discover the flaws and bottlenecks, the application or the systemProactive Performance testing for each release. Load tests will discover the flaws and bottlenecks, the application or the systemmay have in production environmentmay have in production environment
TestingTestingTestingTesting
Conocer el escenario productivo para tomar una buena decisión sobre cómo orientar las pruebas de rendimiento es Conocer el escenario productivo para tomar una buena decisión sobre cómo orientar las pruebas de rendimiento es La hipótesis de partida con más éxito.La hipótesis de partida con más éxito.Conocer el escenario productivo para tomar una buena decisión sobre cómo orientar las pruebas de rendimiento es Conocer el escenario productivo para tomar una buena decisión sobre cómo orientar las pruebas de rendimiento es La hipótesis de partida con más éxito.La hipótesis de partida con más éxito.Conocer el escenario productivo para tomar una buena decisión sobre cómo orientar las pruebas de rendimiento es Conocer el escenario productivo para tomar una buena decisión sobre cómo orientar las pruebas de rendimiento es La hipótesis de partida con más éxito.La hipótesis de partida con más éxito.Conocer el escenario productivo para tomar una buena decisión sobre cómo orientar las pruebas de rendimiento es Conocer el escenario productivo para tomar una buena decisión sobre cómo orientar las pruebas de rendimiento es La hipótesis de partida con más éxito.La hipótesis de partida con más éxito.Conocer el escenario productivo para tomar una buena decisión sobre cómo orientar las pruebas de rendimiento es Conocer el escenario productivo para tomar una buena decisión sobre cómo orientar las pruebas de rendimiento es La hipótesis de partida con más éxito.La hipótesis de partida con más éxito.Conocer el escenario productivo para tomar una buena decisión sobre cómo orientar las pruebas de rendimiento es Conocer el escenario productivo para tomar una buena decisión sobre cómo orientar las pruebas de rendimiento es La hipótesis de partida con más éxito.La hipótesis de partida con más éxito.Conocer el escenario productivo para tomar una buena decisión sobre cómo orientar las pruebas de rendimiento es Conocer el escenario productivo para tomar una buena decisión sobre cómo orientar las pruebas de rendimiento es La hipótesis de partida con más éxito.La hipótesis de partida con más éxito.Conocer el escenario productivo para tomar una buena decisión sobre cómo orientar las pruebas de rendimiento es Conocer el escenario productivo para tomar una buena decisión sobre cómo orientar las pruebas de rendimiento es La hipótesis de partida con más éxito.La hipótesis de partida con más éxito.Conocer el escenario productivo para tomar una buena decisión sobre cómo orientar las pruebas de rendimiento es Conocer el escenario productivo para tomar una buena decisión sobre cómo orientar las pruebas de rendimiento es La hipótesis de partida con más éxito.La hipótesis de partida con más éxito.High availability of the application and the system is the goal of a ready for service status. The application and the systems must be stable , High availability of the application and the system is the goal of a ready for service status. The application and the systems must be stable , efficient and dimensioned according to the usage.efficient and dimensioned according to the usage.
AvailabilityAvailabilityAvailabilityAvailability
Not only the response time is important, an intelligent use of the resources is needed to grow in the future. Efficiency, understood as Not only the response time is important, an intelligent use of the resources is needed to grow in the future. Efficiency, understood as capacity to dispose of the system resources to achive an objective, in our case response time and uptimecapacity to dispose of the system resources to achive an objective, in our case response time and uptime
VelocityVelocityVelocityVelocity
Being able to grow depending on the necessities of the market, users and new technologies is one of the questions to which a performance Being able to grow depending on the necessities of the market, users and new technologies is one of the questions to which a performance engineerfor will have to answerengineerfor will have to answer
ScalabilityScalabilityScalabilityScalability
A performance test is easy. It is easy to design non realistic scenarios. It is easy to collect irrelevant data. Even with a good scenario and A performance test is easy. It is easy to design non realistic scenarios. It is easy to collect irrelevant data. Even with a good scenario and Appropiate data, it is easy to use and incorrect statistic method to analysis the results. Appropiate data, it is easy to use and incorrect statistic method to analysis the results. - Alberto Savoia- Alberto Savoia
WorkLoadsWorkLoadsWorkLoadsWorkLoads
One of the most important parts of a good performance test design is to have an appropiate load test environment, as similar as possible to One of the most important parts of a good performance test design is to have an appropiate load test environment, as similar as possible to Production at all levels, networking, systems and application architecture.Production at all levels, networking, systems and application architecture.
PreProductionPreProductionPreProductionPreProduction
Otro título
EscenariosEscenariosEscenariosEscenariosEscenariosEscenariosEscenariosEscenariosEscenariosEscenariosEscenariosEscenariosEscenariosEscenariosEscenariosEscenariosEscenariosEscenariosEscenariosEscenarios
To know the production environment is key to take good decisions about how to design a performance test plan. Designing a plan according To know the production environment is key to take good decisions about how to design a performance test plan. Designing a plan according To real traffic and usage of the platform is key in creating validation criteriaTo real traffic and usage of the platform is key in creating validation criteria
MonitoringMonitoringMonitoringMonitoring
DevelopersDevelopers, , DBA's,DBA's, QA's, DevOps, product owners ... All the team is part of performanceQA's, DevOps, product owners ... All the team is part of performance
Performance TeamsPerformance TeamsPerformance TeamsPerformance Teams
Otro título
There are many tools available in the market for load testing and monitoring. An effort in evaluating these tools will benefit at long term the There are many tools available in the market for load testing and monitoring. An effort in evaluating these tools will benefit at long term the Execution of the tests. However, the most important part is how the reports are generated and who is going to interpret them. Execution of the tests. However, the most important part is how the reports are generated and who is going to interpret them.
ToolsToolsToolsTools
Otro título
Mas puntos
Not only unique users or session times are important. How the users work with the application and the psicology of the them are key to Not only unique users or session times are important. How the users work with the application and the psicology of the them are key to Understand the results and how it affects to business. Understand the results and how it affects to business.
Real User MonitoringReal User MonitoringReal User MonitoringReal User Monitoring
Keep it simple, use cache wisely, invest in testing and monitoring, create a culture of performance in all the organizationKeep it simple, use cache wisely, invest in testing and monitoring, create a culture of performance in all the organization
Best PracticesBest PracticesBest PracticesBest Practices
Users vs request
• Yeah right … but how many users?
SCENARIOS: Identify the scenarios that are most commonly executed or most resource-intensive
METRICS: Only well-selected metrics that are analyzed correctly and contextually provide information of value.
WORKLOAD MODEL: User Session Duration in average. It is important to define the load levels that will translate into concurrent usage, overslapping users, or user sessions per second.
THINK TIMES: User thinktimes Pause between pages during a User Session depending on the User Type*
User Types: Identify the User … new, revisiting or both.
Performance Acceptance Criteria: Response time, System load, Throughput ...
Tuning
Techonology develops at high speed. To bring out the best of our product, business and techonology need to evolve by the hand. Investing in Techonology develops at high speed. To bring out the best of our product, business and techonology need to evolve by the hand. Investing in Performance research is crucial to keep up with other internet competitors. Performance research is crucial to keep up with other internet competitors.
InnovationInnovationInnovationInnovation
Telefónica DigitalProduct Development & Innovation
Establish the Funnel
41
Telefónica DigitalProduct Development & Innovation
Determine the session navigation
42
Telefónica DigitalProduct Development & Innovation
Establish the Session time
43
Telefónica DigitalProduct Development & Innovation
Test Strategies
Load Test: Stepping Increasing
Goal: Determine the maximum throughput attainable without errors and loss of responsiveness
Model: threadsMetrics: Throughput
44
Telefónica DigitalProduct Development & Innovation
Test Strategies
Load Test: Variance/Uniform
Goal: determine the maximum throughput for a fixed number of threads in a fixed time span
Model: Threads/throughputMetrics: Max Number of threads, Max throughput
45
Telefónica DigitalProduct Development & Innovation
Test Strategies
Load Test: Soak/Avalanche
Goal: determine the limits and bottlenecks of the system during a peak load.
Model: Threads/throughputMetrics: Time to Manage the Peak of Requests
46
Telefónica DigitalProduct Development & Innovation
Test Strategies
Load Test: Spike
Goal: determine the limits and bottlenecks of the system during a sudden peak load under load circumstances
Model: throughputMetrics: Time to Manage the Peak of Requests
47
Telefónica DigitalProduct Development & Innovation
Test Strategies
Load Test: Endurance/Robustness
Goal: determine if the system looses performance during a long period of stationary load
Model: ThroughputMetrics: Optimized Throughput
48
Telefónica DigitalProduct Development & Innovation
Understand the Project Vision and Context
Project Vision
Project Context
Understand the system
Understand the Project Environment
Understand the Performance Build Schedule
49
Telefónica DigitalProduct Development & Innovation
Improved way of working
Improve performance unit testing by pairing performance testers with developers.
Assess and configure new hardware by pairing performance testers with administrators.
Evaluate algorithm efficiencyMonitor resource usage trends, and usage of profilers in dev/qa
environmentsMeasure response times in production and trends in dev/qa
environmentsCollect data for scalability and capacity planning.
50
Telefónica DigitalProduct Development & Innovation
Configure the Test Environment
Set up isolated networking environment
Procure hardware as similar as possible to production or at least keeping ration amongst all elements
Coordinate bank of IP’s for IP spoofing
Monitoring tools and operating systems like production
Load generation tools or develop your own
51
Telefónica DigitalProduct Development & Innovation
Identify and Coordinate Tasks
Work item execution method
Specifically what data will be collected
Specifically how that data will be collected
Who will assist, how, and when
Sequence of work items by priority
52
Telefónica DigitalProduct Development & Innovation
Execute Task(s)
Keys to Conducting a Performance-Testing Task
• Analyze results immediately and revise the plan accordingly.
• Work closely with the team or sub-team that is most relevant to the task.
• Communicate frequently and openly across the team.
• Record results and significant findings.
• Record other data needed to repeat the test later.
• Revisit performance-testing priorities after no more than two days.
53
Telefónica DigitalProduct Development & Innovation
Analyze Results and Report
pause periodically to consolidate results
conduct trend analysis
create stakeholder reports,
pair with developers, architects, and administrators to analyze results
54
But … what are you actually doing day by day?But … what are you actually doing day by day?
Telefónica DigitalProduct Development & Innovation
Case of STUDY
56
• HTML5 trends using Yslow and Firebug with Selenium
• API testing using JMeter
• BackEnd services using LoadUI reusing functional tests from soapUI
• Streaming Content
Branches comparison
top related