Adopting an Experimentation Philosophy - Optimizelypages.optimizely.com/rs/361-GER-922/images/OPT_Adopting... · Source: Gartner Predicts a Customer Experience Battlefield, February

Adopting an Experimentation PhilosophyClosing the Gap to a World-Class Optimization PracticeHudson Arnold - Senior Strategy Consultant at Optimizely

Table of ContentsIntroduction:

Chapter 1: Historic Transformation Creates Unprecedented Urgency for Investment in Customer Experience

Chapter 2: Digital at the Forefront: Continual Optimization Drives Huge Value, but Magnifies the ‘Alignment Gap’ for Many Organizations

Chapter 3: Empower Teams by Formalizing a Methodology

Chapter 4: Tool Effectively for Each Phase

Chapter 5: Foster Creativity with an Intelligence Report

Chapter 6: Incentivize Professional Growth

Chapter 7: Drive Focus, Drive Execution: Benchmark Operational Metrics

Chapter 8: Recommended Next Steps

3

4

6

9

12

16

19

20

22

ADOPTING AN EXPERIMENTATION PHILOSOPHY 3

IntroductionDigital experiences are increasingly key battleground for competition, yet optimization (the use of experimentation, and web personalization to be most effective) as a practice is often siloed, under-resourced, and as a result underperforming.

How can companies maximize their investments and take full advantage of the resources they already have to achieve better performance?

• Skills and Experience: Optimization requires a uniquecombination of analytical and creative skills, which in combination are relatively scarce. Companies are left with difficult choices tomeet the need

• Organizational Adoption: Optimization is often siloed intodepartments and skill sets that can only address ‘part of thepuzzle’, and not formally measured with operational metrics

• Technology Usage: No matter the stack, any organization canmaximize productivity and quality by better training and smartlyinvesting in best-of-breed solutions along their journey

Read on to learn how to:

• Remove bottlenecks and unify siloed & understaffed teamsby formalizing an optimization methodology, empoweringindividuals and groups to take ownership to execute it

• Holistically address the tooling requirements that each stepof the optimization methodology present, including research,program management, experience delivery and analysis

• Foster latent creativity in your team by instituting a regularcadence of research (an intelligence reporting process) toensure ongoing improvements with an optimization program

• Institute upskilling programs across relevant disciplines thatincentivize employees to learn new tools & skills, contributeideas in contests, participate in user groups and conferences,and get certified on technology and processes

• Ensure focus by benchmarking your team’s output withoperational metrics unique to an optimization program

ADOPTING AN EXPERIMENTATION PHILOSOPHY

CHAPTER 1

4

Historic Transformation Creates Unprecedented Urgency for Investment in Customer ExperienceHistoric transformations continue to increase the quality of interactions between consumers and businesses. In many industries, barriers to entry have been dramatically reduced due to innovations in bringing products to market (globalized and agile manufacturing, easier access to capital, disruption to previously value-added white collar service markets), with product quality & cost differentials increasingly difficult to maintain in the face of massive disruption from companies like Amazon. Consumers’ instantaneous and always-on access to socially verifiable information, and reduced ‘switching costs’ in the digital shopping experience are only increasing the efficiency of transactions both informational (product information, price, perceived value) and practical (purchasing, support, shipping).

This dramatic increase in efficiency & competition is creating unprecedented value for consumers but transfers the key basis of competition in many industries away from pure product/price differentiation and toward a more holistic and experiential relationship, one in which the product or service plays a key but limited role.

CHAPTER 1

ADOPTING AN EXPERIMENTATION PHILOSOPHY

CHAPTER 1

5

“The connected economy continues its path of creative destruction, so much so that it is changing the nature of competition itself. Nearly all companies surveyed by Gartner (89%) believe that customer experience will be their primary basis for competition... For example, consumer product companies that have relied on developing new features, improved customer service and product innovation to drive growth now see a future where competitive advantage will be based on the customer experience.

In many industries, hypercompetition has eroded traditional product and service advantages, making customer experience the new competitive battlefield, says Jake Sorofman, research director at Gartner. With an overabundance of alternatives and ubiquitous access to pricing and product information, consumers have little reason to remain loyal to a particular brand.”

Source: Gartner Predicts a Customer Experience Battlefield, February 18, 2015

Customer experience, which includes the holistic experiences of a company’s brand and direct response marketing, brick & mortar support, product, loyalty programs, etc., is far from being strictly the purview of luxury services. Today, customer experience is being heavily invested in by companies with historically poor consumer ratings, such as Comcast, which in 2015 announced a “multi-year plan to reinvent the customer experience and to create a culture focused on exceeding customers’ expectations, at all levels of the company”. This plan not only addresses long-held consumer difficulties such as frustrating in-person appointments, but proactively attempts to position Comcast competitively with emergent cable and OTT competitors like Google and Apple TV by investing in in-store experiences, an on-demand service mobile app, and the core tv-watching experience.

Radical, multi-year business transformations like the one Comcast is undergoing may be an optimal immediate step to reverse historical customer dissatisfaction for companies with comparable inertia and unit economics. However, organizations with less massive infrastructure, smaller consumer switching costs, fewer consumer touchpoints and channels, or simply a better ‘customer experience status quo’ are quietly investing and succeeding with a more evolutionary approach to optimizing their customer experiences.


CHAPTER 2

Digital at the Forefront: Continual Optimization Drives Huge Value, but Magnifies the ‘Alignment Gap’ for Many Organizations Due to the ubiquity of devices and internet access, relatively low costs of iteration and measurement compared to physical infrastructure, and the channel serving as the focal point of disruptive new entrants in fields as diverse as insurance, investing, and fashion, continuous optimization is most effective in digital touchpoints. Compared to the heavy-lifting required to experiment in physical retail locations, online experiences can be completely revamped at a fraction of the time and expense.

Across the web and in native applications, experimentation, personalization, data modeling, and other techniques are driving relentless improvement in advertising, marketing, and experience delivery. Some of these channels, particularly customer acquisition and direct response advertising, are already relatively mature industries, in which established best practices tend to deliver diminishing returns for marginal investment. Other channels, including website and app optimization, are still rapidly evolving and creating disproportionate ROI for companies just starting out.

Some businesses, particularly digital-first companies (such as Priceline Group, Airbnb, Atlassian and Netflix) are setting the standard for how a process of continual optimization drives growth. At these companies, their core website and application experiences are not only a channel, but serve as the product itself. Their focus on and investment in optimizing their experiences provides valuable lessons for every other company working to most effectively compete on the basis of customer experiences.

CHAPTER 2


CHAPTER 2

The vast majority of companies fall somewhere between these extremes, but most exemplify more criteria of the second group than the first.

Mind The Alignment Gap

Mature Optimization-Led Organizations

• Continual optimization is deeply embedded throughoutthe digital customer experience

• Experimentation is an iterative, integrated process, fullyingrained into each channel’s workflow.

• Teams work independently on their specific optimizationroadmaps, while a central group manages data collectionand analysis, empowers individuals with resources andprocess, and optimizes workflows based on operationalmetrics.

• A combination of in-house experimentation frameworksand dedicated third-party solutions conduct experiments,deploy personalized experiences, and surface insights.

• The strategy and metrics for improving the performanceof digital experiences are directly linked to companysuccess and are widely reported both internally andexternally.

• Experimentation is resourced with either full or part-time contributions from the needed skill sets across theorganization.

• Collective optimization-specific expertise creates a virtuous cycle of learnings and best practices.

Early Stage Optimization Organizations

• The website or application serves as a marketing or saleschannel rather than core product, and may only recentlybe receiving attention of top leadership as a strategicasset and key differentiator.

• Resources required for continual optimization, likedevelopment, design, program management, or analytics,are often divided across functional silos with differentgoals & operating metrics, or may not even be availableat all.

• Individuals face interdisciplinary challenges of design,planning, production, and analysis with limitedcollaborative experience.

• Stakeholder groups with different goals for optimizingthe channel, like marketing, product, or merchandising,may have only limited access or ability to make evensuperficial changes.

• Optimization goals and processes are owned by a singleindividual without executive sponsorship, or may not beformalized at all.

• Digital strategy and maturity may have only limitedattribution of digital engagement to bottom-linerevenue, and consumer intentions and digital behaviormay be poorly understood, preventing strategicalignment and clarity.

VS


CHAPTER 2

The gap between these two sets of attributes represents the key opportunity to create and sustain a competitive advantage in experimentation.

This gap has numerous root causes, from business model, organizational structure, technology investments, and available skill sets, and may appear vast. Yet, there are measures that any organization can take to efficiently leapfrog these hurdles to the front of their competitive group.

The following steps are relevant for companies looking to maximize their existing people and technology resources, as well as for those considering new investments in technology, people, or agency services to drive their experimentation strategy.

“Most CI professionals that Forrester interviewed for this report estimated that 30% or fewer digital customer interactions are used in some form of online testing”, while “only 61% of firms have had an active online testing program for two or more years.†”

Source: Forrester, 2016


CHAPTER 3

Empower Teams by Formalizing a Methodology The foundational step towards maximizing the value of optimization is to map out the inputs, outputs, team involvement, and milestones that the work requires. This mapping will make it easy for various teams to understand the commitments they will need to make, and serves as an operational checklist that will streamline workflow and align individuals and departments with each other.

This methodology is universal in its core structure, but allows flexibility and can be adapted to organizations of various sizes and structures. Some organizations may discover that the entire workflow or certain steps are best entirely left to an expert agency, while others may find that one person needs to manage the entire process. Organizations still new to the process may find it useful in determining future costs, despite not having a program in place.

Read on to learn how to hold your team accountable to a world-class methodology

CHAPTER 3

Preparation Set your program up for long-term success Ideation

Generate ideas for testing or

Planning Design individual tests and campaigns

Analysis Analyze and make decisions about your results

Execution Check that your test or campaign is set up correctly and launch

An Experimentation Methodology


CHAPTER 3

PreparationWhen you build an optimization strategy for your company, you’ll first align your program’s testing efforts to your company goals. Without a strategic guide, you run the risk of building unfocused or inefficient tests that consume time and resources without producing results. How do you ensure that your tests will make a real difference?

• Connect the optimization program to key company metrics• Create a charter to guide the program• Identify skills and competencies needed for the optimization team• Technology implementation

IdeationIdeation, the process of generating powerful hypotheses to drive experiments and campaigns, is one of the most important steps in experimentation. It’s also one of the hardest. Without a structured framework supporting the creative stage of experimentation, it can be difficult to consistently generate impactful ideas.

• Create a business intelligence report to drive ideation• Create an idea submission form to build an optimization culture• Ideate for your optimization program

PlanningHow do you decide what to optimize and when? To figure out which experiments and campaigns to run first and which to place into your backlog, use a prioritization framework to evaluate your ideas.

• Outline concrete steps for each test or campaign while ensuringa scalable, consistent process

• Prioritize optimization ideas and build a roadmap• Create individual experiment plan

ExecutionHow do you avoid the typical pitfalls of creating and launching experiments? A rigorous QA process ensures that your experiment looks and works the way you want before you show it to visitors.

• Once you’ve finished building your experiment, take a little extra time to verify your setup so you can trust your results

• Six steps to build an experiment• Create a pre-launch checklist: QA• Launch and monitor experiment

AnalysisYour experiment results–whether winning, losing, or inconclusive–are an incredibly valuable resource. The data on your Results page helps you learn about your visitors, make data-driven business decisions, and feed the iterative cycle of your experimentation program.

• Interpret your results• Take action based on your results• Share your results with stakeholders• Work with a developer to implement changes in production


CHAPTER 3

When defining this methodology, the organization will need to determine:

• The goals of the optimization program and its alignment tocompany-wide success

• What the sequence of steps to execute optimization work areand why they’re important

• The inputs and outputs of each step, and associated keydeliverables & milestones

• Which team or individual owns each step and the methodologyas a whole

• The necessary tracking and governance to ensure the rightsteps are being taken at the right time

The benefits of a solid methodology are real for any company without one in place, as well as for many companies with a sub-optimal or unformalized one. By mapping and aligning the inputs, outputs, and milestones of the optimization methodology, the company will:

• Socialize the organization’s commitment to competing oncustomer experience and demonstrating the business impact ofoptimization at the most senior levels and across departments

• Acknowledge strengths and weaknesses: highlight the work iscurrently being done well, areas for improvement, and potentialfuture expansion, as well as what is not feasible

• Empower teams and hold them accountable to their respectiveroles in the process

• Set realistic shared expectations about resourcing

• Create a common understanding of what types of experimentsand campaigns are attainable given the available resources

• Structurally enable better collaboration and smarter planning

To learn more about this methodology, visit the Optimizely Knowledge Base series Optimization Methodology: Running your Optimization Program


CHAPTER 4

Tool Effectively for Each PhaseA key point to consider is that, for most organizations, there may already be an existing workflow, document, or tool that facilitates an analogous activity within the organization. Often, this tool may be useful and adaptable to an optimization methodology (this might be a team charter written in a word document, or a workflow being managed in JIRA), while in other cases the tool may need to be re-evaluated once the requirements for it are better understood.

Teams that choose the right tools for each optimization stage reduce anxiety and confusion and execute with more agility and purpose.

Evaluate the inputs and outputs of your methodology with the question list that follows to better determine how to tool for each phase.

CHAPTER 4


CHAPTER 4

Preparation

• Do we have a process in place for marketing technology implementation? What documentation is available?

• Do other teams, individuals, or even the company already have a mission statement or document for a related discipline, like analyticsor digital marketing? Can we adopt it for this program? Should optimization be added to those documents instead of created new?

• How will the team talk to each other on an ongoing basis? Would email aliases, a Slack channel, Asana project, group in JIRA orConfluence, or another method be best?

• Is there documentation already outlining digital strategy and experience measurement? How can that language and format be usedor revised to include optimization?

• Of the desired skill sets at the company, what level of dedicated time is likely or possible? Should alternative solutions be consideredfor tasks that will have limited resourcing?

• If you’re starting from scratch, ask your optimization vendors how they support and empower their customers, what their product visionis, and get a sense of how their positioning aligns to your goals and methodology. What aspects of your optimization strategy, be it acertain kind of use case or analytical mentality, would be helped or hindered?

Ideation

• What existing data sources are consulted for digital strategy and optimization? Is the team constrained by their existing tools? If so, canthey make a cost/benefit business case for new intelligence tools or 3rd party research?

• Are ideas from across departments currently used? If not, would a submission form or email alias help?


CHAPTER 4

Planning

• For smaller, independent teams, would a spreadsheet suffice? Why or why not?

• How would a lighter-weight software solution like Asana or Trello function comparably?

• For larger groups, would development of custom solution using APIs be preferable to a purpose-built optimization management tool,task management solution or collaboration tool like JIRA? How do you already use project management software?

Execution

• Given the human and process investment in the optimization program, how will you quantify the impact of improvements to customerexperiences in terms of the benefit to the organization?

• Can you quantify the value of certain features, in the use cases they’ll facilitate, or of a tool’s ease of use, in hours saved and productivity gained? The time and difficulty of learning and socializing the software?

• What about the opportunity cost of data that’s difficult to reconcile with your existing data, or that’s prone to statistical misinterpretation?

• If you’re considering migrating to a different platform, try quantifying the switching costs against potential time savings, extensibility,etc. What impact would switching have on your optimization program’s operational metrics?

Analysis

• Does your reporting need to resolve into a data warehouse, or is it ingested and combined manually by analysts producing reportsfrom different systems?

• How does your optimization platform integrate with the systems you consider your source of truth?

• Does your team use custom processes in a modeling tool, spreadsheet, or SQL database, to perform statistical analysis? Is it moreimportant to get raw data in a usable format or to get robust statistical reporting? Do you need to choose?

• How will your web analytics reporting need to be configured to facilitate your optimization work?

• How does your team normally present performance reports to one another? Is the current format working well or not? How so?


CHAPTER 5

Foster Creativity with an Intelligence ReportResearch suggests that one of the most pressing bottlenecks in an optimization methodology is the ability to consistently generate and execute a volume of quality optimization hypotheses. Generating hypotheses is one of the most cost-effective bottlenecks to remove, because unlike a lack of development resources, absence of the proper tools, or cultural reluctance to commit to optimization, teams and individuals have relatively direct control of idea generation for experimentation.

One root cause of this issue is that the real output of optimization is, very often, what traditionally has been the purview of creative specialists; graphic designers, UX specialists, copywriters, or creative directors. Experimentation is, in many ways, a discipline with a ‘creative’ output.

Yet, the vast majority of optimization programs are led by individuals or groups with ‘non-creative’ backgrounds, like performance marketers, analysts, project managers, merchandisers, etc.

While design and UI are very often involved, the teams that drive optimization are often business units or engineers: marketing, ecommerce, product, engineering, etc.

Which teams are INVOLVED in optimization at your company?

Source: Optimization Benchmark Report

Marketing

52% 32%

53% 14%

40% 12%

36% 12%

35% 9%

31% 8%

30% 5%

23% 3%

20% 3%

3% 3%

Product

Sales

Design

Engineering

Customer Service or Support

UI or Front-End Development

E-commerce

Executive Staff

Other

Marketing

Product

Executive Staff

E-commerce

Engineering

Other

UI or Front-End Development

Design

Customer Service or Support

Sales

What team LEADS optimization at your company?

CHAPTER 5


CHAPTER 5

The good news is that companies can foster latent creativity by being more proactive about collecting insights:

“Creativity is just connecting things. When you ask creative people how they did something, they feel a little guilty because they didn’t really do it, they just saw something. It seemed obvious to them after a while. That’s because they were able to connect experiences they’ve had and synthesize new things.”

Steve Jobs

Optimizely research surfaces 3 key opportunities that companies trying to feed stronger ideation can address Source: Optimization Benchmark Report

• Companies’ optimization programs over-rely on quantitative,behavioral data, and are likely not optimally effective at using it

• Optimization programs use too few data sources to sourcetheir ideas

• Optimization programs can apply more progressive, rigorousprocesses to ensure that the right data is used in the right wayat the right time to improve customer experiences

By creating an intelligence report that synthesizes readily available direct data (information that your company tracks about itself, such as web analytics, user personas, survey data, etc.) with indirect data (outward-looking intelligence such as competitive reviews, industry trends in UX, thought leadership from agencies, vendors, and events), your team can efficiently assemble the same information a dedicated creative professional would use in their work.

While conducting just one intelligence report will create significant advantages for optimization programs, the highest potential lies in optimizing the execution and usage of the report itself. Here are some simple tips for creating an effective intelligence report.

These factors lead to a challenging situation for many companies: a relatively technical group of analytically-minded professionals trying to drive better business outcomes in a discipline that requires creative outputs.


CHAPTER 5

An intelligence report will be most effective when:

• It’s conducted on a regular cadence. Whether you conduct quarterly, monthly, or bi-weekly planning, these stages in which your team’screativity and decision-making will be most critical should be fedwith rich, analyzed data.

• It portrays insights, focusing only on the data that has a story to tell.An intelligence review should be full of takeaways, summaries, andinsights. It is not a performance report of KPIs and raw data, but atool for empowering creativity.

• The report addresses a wide variety of direct and indirect datasources, facilitating insights.

• It collects and draws connections between the disparate datasources, but is presented as a single deliverable. Everythingmeaningful should be at your fingertips.

Finally, while the effort of a truly comprehensive intelligence report may seem daunting, there are many different degrees of detail that can be undertaken, depending on your organization’s size, resources, and maturity as a digital optimization team.

Here are some simple steps to take action on this idea in your organization:

• Identify the different data sources you have access to currently,additional ones you can explore with no additional capitalinvestment

• Mark when your next critical planning stage will happen

• Outline what your first version of an intelligence report shouldinclude

• Set a goal to have the report ready and socialized by the beginning of that planning stage

Learn more about the data sources and reports proven to effective for ideation in the Knowledge Base series on Ideation, and the Big Book of Experimentation for 35 real world, successful experimental concepts.

Intelligence Report Cadence

Q1 Q2 Q3 Q4

Intelligence Report Created

Quarterly Planning


CHAPTER 6

Incentivize Professional GrowthAlignment around a solid methodology, access to the right tools, and a dedication to research will have a dramatically positive impact on almost every organization’s experimentation. Yet, the competitive environment is incredibly dynamic. Technology constantly evolves, continually changing what’s possible in optimization, while consumer expectations grow and shift.

In order to decrease the risk of future team turnover and competitive disruption over the long term, organizations can make cost-effective investments in the skills of the people managing and executing the optimization program.

Upskilling programs don’t need to be the domain of expensive trainers or university courses. Companies like Optimizely and Google make significant investment in their certification and education programs, generally free with a nominal fee for certification exams, and many ‘soft skills’ or industry trends can be maintained with simple effort.

Some simple, cost-effective steps you can take ensure your team is sharp and evolving:

• Set goals for all of your team members to achieve the technologyand industry certifications most relevant to their role.

• Subsidize the cost of the exams themselves, publicly track adoption, and reward completions with simple gifts, which can be as small asa free dinner or as large as a plane ticket.

• Encourage and recognize participation with conferences and usergroups, as well as participation in forums and open-source codeprojects.

• Assign a rotating team member to make a monthly ‘new andnotable’ report (or addendum to regularly scheduled intelligencereports), noting and sharing industry insights. This report shouldinclude recommendations and be circulated broadly,

• Regular ‘lunch n’ learns’ on advancements within and betweenteams can reduce internal confusion, while core team membersshould assume a responsibility to regularly assess and discussnew products and features in their toolkit.

• Invite candid feedback from customers, team members, andpartners on the state of your experiences. Regularly socialize thisfeedback and empower individual contributors to interpret it.

CHAPTER 6


CHAPTER 7

Drive Focus, Drive Execution: Benchmark Operational MetricsEven for experienced, well-resourced teams, dynamics unique to optimization present consistent challenges. Most organizations meet these challenges in an unsystematic way, relying on the individuals closest to the work to address them as best they can. In this report, we’ll address the challenges of experiment velocity, experiment quality, program efficiency, and program agility.

For leaders trying to maintain operational excellence, measuring & benchmarking the team’s performance in meeting those challenges ensures that focus is kept on execution. This executional focus allows executives and program managers to recognize and address potential issues earlier, while serving as a reliable leading indicator for overall program performance and so serving as excellent quarterly goals.

Each of these leading indicators have a qualitative connection to the optimization process and a quantifiable variable. By tracking, reporting, and socializing these variables in a dashboard or similar snapshot communication method, the team will be able to gauge progress and make course corrections to strategy, process, and resourcing.

CHAPTER 7


CHAPTER 7

Operational Metrics: Benchmark Your Optimization Program

Velocity Efficiency Quality Agility

Description The volume or quantity of tests being ran, the reach of personalization campaigns

The rate at which tests move through the production cycle, from a scoping through approval and launch

The average likelihood that a given experiment will produce business impact (conversion increase or learning generated)

The degree that the testing program acts on results and iterates

Metrics Throughput:# of tests per property per month/week

% of site traffic reached by personalization campaign

Drag:% hours spent redeveloping experiments due to QA or troubleshooting issues compared to ‘productive’ time

Length in hours/day of production cycle

Impact Rate:% of experiments generating meaningful result (with a statistically significant winner or loser)

Net improvement generated by personalization campaigns

Iteration:The % of experiments put into production and iterated upon

# of days from result to action on average

Why it’s Important A large % of experiments don’twin, personalization is limited by reach; improving throughput will ultimately generate more success due to the law of large numbers.

Execution is unfortunately usually noticed and measured in negative terms; it can be a bottleneck to achieving velocity, quality, and agility.

The fastest and most efficient experiment production will still be limited by the breadth and depth of experimental tactics being employed.

Personalization net impact should be benchmarked, so that Reach grows over time.

Experimentation is a means to a more fluid and iterative production cycle, which should be fed both by up-front ‘waterfall’ planning as well as agile adaptation

Risks Bottlenecked often in ideation & planning phases of methodology, and held back by production inefficiency.

A proper methodology with checks and balances for adherence must be in place, and inadequate access to the right skill sets, or lack of training can reduce efficiency.

The primary blockers are effective hypotheses, experiment design, and technical roadblocks.

Lack of a playbook for results analysis, alignment between leadership and operations on success criteria can reduce agility.

Whether in manually-updated spreadsheets or reports derived from systems like JIRA, establish a reporting framework that captures the throughput, drag, impact rate, and iteration your program is able to generate, and create a habit of optimizing the program toward this data.


CHAPTER 8

Recommended Next StepsFor organizations pursuing continual optimization, the steps prescribed in this paper are well-proven to maximize the value of limited resources. Importantly, they can mostly be undertaken with zero incremental capital investment, relying instead on shifting organizational behavior in order to adopt an experimentation philosophy.

For the executive or intrapreneur trying to build momentum and improve ROI from their optimization program, adopting these behaviors should be a top priority, yet they do require coordination between teams and executive sponsorship.

Online testing lacks executive visibility. Only 32% of respondents to Forrester’s Q3 2015 Global Online Testing Platform Customer Online Survey were executive level or higher. The rest of the respondents were departmental managers or individual contributors. Executive budget holders still do not appreciate the potential value that online testing can deliver to the business. Nicola Arnese, VP of eCommerce and digital at Alitalia, informed Forrester that convincing executives of the importance of online testing technologies is the biggest obstacle to growing and maturing their online testing program.

Source: Forrester Continuous Optimization Report

CHAPTER 8


CHAPTER 8

Leverage these resources, analyst reports, and other resources like case studies to make a compelling case for the value that optimization can deliver to executives.

By understanding the potential value that successful firms are able to derive from a practice of continual optimization along with the structural challenges (the ‘alignment gap’) that hinder many other organizations from realizing the value of the practice, you will be able to take steps and make recommendations to close that gap. Leaders will be better equipped to align their teams, ‘manage up’ with goal-setting and quantification, and avoid the potential of losing market share in the rapidly evolving competitive landscape of digital experiences.

For more context on why the issues and recommended steps here are fundamental, read the full Forrester Continuous Optimization report for clarifying research and insight into the state of the industry. Optimizely’s case studies and e-books contain tightly presented evidence and guidance from professionals in the space as well.

Resources:

The Optimizely Methodology

Forrester Continuous Optimization Report

The State of Optimization Benchmark Report

Gartner on Customer Experience

Optimizely Case Studies

Optimizely eBooks

Optimizely, our community, and our network of technology and solutions partners are devoted to helping our colleagues, customers, and clients address these issues Talk to us!

The Optimizely Community

Optimizely Partners List

Optimizely Sales

San Francisco, CA (HQ) 631 Howard Street, Suite 100 San Francisco, CA 94105 1-800-252-9480

New York, NY 162 5th Ave, 2nd Floor New York, NY 10010 1-800-252-9480

Netherlands (EMEA HQ) Nes 76 1012 KE Amsterdam +31 (0)20 26 100 60

United Kingdom Barratt House, 3rd Floor 341-349 Oxford StreetLondon, W1C 2JE+44 (0)203 6956190

Germany GmbH Spichernstrasse 6 50672 Cologne +49 (0)221 828 297 24

About OptimizelyOptimizely provides the world’s leading experimentation platform, Optimizely X–enabling organizations to experiment with pricing, recommendations, and product features across every channel to improve customer experiences and drive revenue.

Optimizely meets the diverse needs of thousands of customers worldwide looking to deliver connected experiences to their audiences across channels. To date, those customers have created and delivered more than 30 billion optimized visitor experiences.

Optimizely powers experiences for the world’s leading brands

Adopting an Experimentation Philosophy - Optimizelypages.optimizely.com/rs/361-GER-922/images/OPT_Adopting... · Source: Gartner Predicts a Customer Experience Battlefield, February

Documents