Build 2016 - P502 - App Flighting and Beta Testing in the Windows Store

#Build2016

App flighting and beta testingin the Windows StoreJonathan Garrigues & Beth Anne KatzProgram Managers

App Flighting Goals

The Windows Store allows developers to do controlled testing in production……to confirm the quality of updates before deploying them to all customers.

…to minimize the impact of bugs that only reveal themselves at scale.

…to learn about the effectiveness of new experiences from a limited set of customers, before these experiences are ready to ship to everyone.

New tools for testing in productionAvailable nowPackage flightingDeliver different packages to different people who already own the app. Manage groups of known customersSpecify people by Microsoft account email addresses.

Coming soonTargeted app availabilityRestrict who can see and acquire your app in the Store.

Gradual rolloutControl how quickly package updates are delivered.

Let users opt-in to app beta groupsInvite customers to opt-in to receive your beta quality experiences.

Everyone else gets the non-flighted packages

Package flightsDifferent packages delivered only to group members

New concept of customer groups

Known customers specified by Microsoft account email address

Customers who have opted-in to an app beta group

Customer groups

Who can see / acquire the app?

Everyone can see / Everyone can acquire

Only those with direct link can see / Anyone with direct link can acquire

Targeted app availabilityOnly group members with a direct link can see / acquire the app

Only those with direct link can see / Only those w/ promo code can acquire

Which packages do they get?

Package flightingSample scenarioYou already have an app in the Windows Store, with several million customers.

You want to validate that your latest bug fix works end to end with a limited set of known people who already have the app installed.

Once you have confidence in the fix, you’ll make the update available to everyone.

Flight packages to multiple groups simultaneously

Package flights contain only binary packages.

Only group members get these updates.

Your app or game has to be published before you can create a package flight.

Non-flighted submissionPackages v1.0

Store listings

App distribution settings (available to everyone)

Group of 100 co-workers

Package flight CPackages v1.1

Group of 15 testers

Group of 1500 enthusiasts

Package flight BPackages v1.2

Package flight A

Packages v1.3

Demo

Creating and publishing a package flight

Supported client versionsPackage flighting works onWindows 10 Desktop Version 10586+ (November)Windows 10 Mobile Version 10586.63+ (February)

For earlier clientsUsers not eligible to receive package flights continue to get updates from the non-flighted submission.

Package flighting customer experienceUpdates to flights happen automaticallyUnless your app tells them, customers don’t know that an update is from a package flight.Make sure you maintain the quality level in your flights your customers expect.Clients check for updates automatically once per day.Customers expecting flights (such as testers) can manually check for updates in the Store to get a new flight immediately.

Get consent before flighting “beta” quality packagesIf you are offering customers a way to sign up to receive a “V.Next beta” experience by providing their Microsoft account email addresses, make sure they understand the quality level of the experience you will be sending them.

Customer feedback channelsWindows Feedback AppYour app can invite your customers to provide feedback using the Windows Feedback App.Customers can to see and up-vote feedback for the same version they are running.Learn more in the video App Feedback: Connect with your customers

Ratings and reviewsRatings and reviews for packages in a flight are hidden from other customers.All ratings and reviews are visible in Dev Center.If a package becomes available to everyone, ratings and reviews for that package will be visible to everyone.

Adjust audience based on analyticsPer-version telemetryYou can make decisions about increasing the audience for specific packages based on real world performance data.

Each package in your flights has a specific version.

Dev Center health report allows you to filter crashes, hangs, etc. by packageversion.

Demo

Moving packages between flights

Coming soon

In the Store, gradual rollout in progress

View gradual package rollout telemetry

Coming soonGradual rolloutControl the speed of any package update by percentage of the target audience.No need to identify specific users.Halt the rollout if you find an issue.Independently and simultaneously rollout flights and non-flights .

Coming soonTargeted app availabilityRestrict who can see and acquire your app or game in the Store.Only people who are group members can see and acquire the app or game, using a direct link.Controlled as part of your non-flighted submission.

Colleagues

Enthusiasts

Group of 100 Co-workers

Test Team

Coming soonLet users opt-in to app beta groupsDescribe how you will use the app beta group and invite your customers to join.

Customers can directly opt-in / opt-out.

Enables V.First beta scenarios when used with targeted app availability (app not public yet)

Enables V.Next beta scenarios when used with package flighting (app is public, beta is for next version).

SummaryTry out in Dev Center Today: Package flightingManage groups of known customers

Features coming soon:Targeted app availabilityGradual rolloutLet users opt-in to app beta groups

Thank you.