Hearing voices AUGMENTED REALITY AUDIO CAN CHANGE THE WAY PEOPLE EXPERIENCE THE WORLD, BOTH AT HOME AND IN TRAVEL Tim Haynes Principal Storyteller Sabre Labs
Hearing voicesAUGMENTED REALITY AUDIO CAN CHANGE THE WAY PEOPLE EXPERIENCE THE WORLD, BOTH AT HOME AND IN TRAVEL
Tim HaynesPrincipal StorytellerSabre Labs
What is augmented reality (AR) audio?
• A layer of digital audio information that enriches the physical world
• Usually associated with playback of location-based (geotagged) digital audio information
• Includes the hardware and software necessary to play layers of geotagged content in a meaningful way
• Can also include digital augmentation of aural senses to affect a user’s perception of an environment
Partial graveyard of failed AR audio platforms
Causes for the graveyardDiscoverability — Lack of standard name or buzzword• 2011, “dynamic spatial audio” & “ambient spatial audio”
were buzzwords for the next big thing with smartphones• Also: location-based audio, geotagged audio, locative
audio, augmented reality audio, location-aware audio…Failure of the “why” — Startups failed to find a broad and compelling use case for AR audio, focusing on nichesLack of partnership — Most AR audio apps are startups and not integrated into, partnered with or promoted by an existing, successful platform and user base
Continuing challenge: poor mobile interfacesContent playback• Many apps (both past and present) were never
optimized for mobile, even in-app• If it’s hard to find or to play content, an app will failContent creation• Most apps have very poor interfaces for adding
location-specific content, especially mobile creation• The best creation interfaces have tended to be focused
on art and documentary uses in niche locations
Usability premise:Audio makes the real world more interesting
This is Massachusetts Ave. in Cambridge, MA.On the next slide, try different clips of music and see how it changes the way the scene feels.I recommend trying Yann Tiersen, Amelie Soundtrack, “La valse d’Amelie, version original,” followed by some classic jazz and then something hip hop or alt-country.
How is AR audio being used?
• Art — theater and conceptual art installations• Documentary— sharing and cataloguing stories• Gaming — AR and MR possibilities are endless• Navigation — associating sounds with places• Notifications — directions, alerts, messages, etc.• Social — sharing experiences, both real and virtual• Tours — cities, historical sites, museums
Placing sounds in space as interactive artArt
2010 – Audio Graffitiproject, Connecticut
2010 – Scapes project*, Decordova Sculpture Park, Lincoln, MA*created w/Roundware
Blended audio tours of art spacesArt
Voices: FAMSF* (2015) An “immersive soundscape” with multiple voices and music centered on 21 sculptures; users can personalize with a mix of curators, art experts and fellow patrons*created w/Roundware
Ghosts of poets in Harvard YardArt
re~verse (2015)commissioned for Harvard Litfest, “consists of a continuous layer of location-sensitive music punctuated with recorded voices of poets who have visited Harvard in the past and given readings of their work.”
*created w/Roundware
High Street StoriesChristchurch, NZ, aftermath of 2011 earthquakes
Docu
men
tary
Tributaries*Tyneside, Northern England, WWI recordings
Docu
men
tary
*created w/Roundware
Listening to the voices I forgot where I was and was taken somewhere else.”
—KERRY BOSSONSPLAY STORE REVIEW OF TRIBUTARIES
“
Docu
men
tary
Location-based gaming is on the rise
Gam
ing
Geocaching Zombies, Run! Pokemon GO
Incorporating AR audio into gaming
Gam
ing
• Several ambitious / experimental smartphone games over the past few years have used AR audio as their main interface:• Inception and Dimensions produced location-based sound and music
and use the phone’s microphone to generate a unique soundscape• Papa Sangre allowed players to navigate a virtual horror soundscape,
attempting to gather lost souls and escape enemies• Meltdown never made it out of prototype, but was an Alien-esque
game in which you hunted aliens on foot using only your ears
• None of these AR audio games are still around (AFAIK) • Immersion in AR games can benefit tremendously from
location-based audio
Using AR audio to navigate
Nav
igat
ion
• Turn-by-turn directions are a common feature of virtually all GPS-based mapping apps, but that’s the most basic use
• Just as the sounds of a city help tell you where you are, an AR audio layer of sound effects and music can convey additional spatial or location information
• Art installations are examples—sound effects and music are tied to places in a exhibition and volume and directionality are directly affected by your location
MIT Media Labs, Loco-Radio project
Nav
igat
ion
Music for navigation and discovery
Using AR audio to give people important context
Not
ifica
tions
• At it’s most basic, having earbuds in can provide private notification of messages or alerts
• More importantly, location-aware alerts can inform people of:• Points of interest• Transportation options• Potential hazards
• On the commerce side—location alerts can market promotions, sales, and/or experience availability
Discoverable content for the visually impaired
Not
ifica
tions
Ariadne GPS (2012)Featured as part of Apple Keynote 2012; create personalized maps with alerts and access audio information through touch about maps in any area. Allows users to understand an area they are unable to see with their eyes
Social elements of AR audio overlap other uses
Soci
al
• Startups have attempted to use location-based audio with chat, friend proximity and other social services
• Art and documentary installations connect visitors with other participants over time
• Gaming uses have the potential to be deeply social• Synchronized tour groups are inherently social as
shared experiences
People want to feel an authentic connection with... what makes a place different from all the other places they could visit in the world.”
—ANDREW MASONDETOUR FOUNDER AND CEO
“Tour
s
Audio tours are increasingly important to travelTour
s
• Plethora of apps for “traditional” audio tours: Izi, Pocket Guide, Rick Steves, UniGuide, etc…
• High adoption by Millennial and Z generations• Detour is the current AR audio darling, one-upping
traditional tour approaches and incorporating limited use of AR audio elements
+
DetourNPR-style tours of San Francisco +9 other cities
• Current tours are professionally produced, taking 3-6 months each
• Participants need to follow tour directions
• Music and speech run separately to account for different movement rates
• Groups can sync tours to share in the experience
• “Descript” turns editing voice to word processing
Tour
s
Audio tours can be cinematic experiences that get you closer to the feeling of walking in another person’s shoes than any other medium can.”
—ANDREW MASONDETOUR FOUNDER AND CEO
“Tour
s
Touring doesn’t have to be proscribedTour
s
• I experimented with various platforms for AR audio content:• Audio Walk — Assign audio files to geographic locations; all
features currently appear broken• Echoes — Limited content (mostly European), significant login
issues and lack of support for content creation• GeoTourist — Basic tours, decent discoverability; open-to-public,
Web-based content creation tool (no mobile creation)• HearUsHere — Handful of tours, poor discoverability of nearby
content, and content creation done only by company• Roundware – really excellent open-source, but it would take a
team effort to create a viable experience• SonicMaps — Terrible interface; lots of broken links; counter-
intuitive content creation
What are the advantages of AR audio?
• Constant context• Adds information without monopolizing attention• Fully adaptive to individual pace and schedule• No fiddling with a device – you interface by moving• Aural illusion of directionality and distance• The brain is really good at filtering sounds, allowing
audio AR to target different layers of your attention
How does AR audio apply to travel?
AR audio opportunities for travel: Art• Theater performances could overlap w/games, potentially
allowing users to take on roles asynchronous w/other users or create new kinds of improve performances
• Art installations enrich content and add a participatory element to art cultural experiences – either standalone AR audio or integrated w/other art – potentially driving additional traffic to specific locations
AR audio opportunities for travel: Documentary• Needs a shared social platform for documentary content for
viability• If not a standardized creation app, at least a site or app that
aggregates, curates and maps niche documentary apps to increase discoverability
• Documentary AR audio allows people to experience locations from new and diverse perspectives, enriching the value of traveling—and spending additional time—in a particular place
AR audio opportunities for travel: Gaming• Huge opportunities for AR audio gaming experiences, both
standalone and with AR visuals (more common), as well as integrated into deeper merged reality experiences
• Location-based digital souvenirs are likely to be a huge part of future travel, largely driven by younger generations; if not the impetus for travel, definitely part of the anticipated travel experience
• When AR audio and visuals combine, “gamification” of everyday life is likely to become more common; lessons learned now by experimenting will help shape future uses
AR audio opportunities for travel: Navigation• AR audio is increasingly essential to travel, both for direction
and for discoverability• Layering a location-based soundtrack could be a powerful
wayfinding method for large buildings or city areas, tying locations or areas to certain music or soundscapes
• Layered with gaming elements, audio navigation can potentially drive people to visit new places
• In the near future, may start to include “superpowers” like near-real time translation of voices and music
AR audio opportunities for travel: Notifications• Being able to layer directions with other sounds makes travel
more seamless (vs. current method of most audio apps/streams demanding exclusivity (e.g. Google Maps stops your music to give you directions)
• Audio alerts can help maintain hands-free, eyeballs-free travel through both indoor and outdoor spaces
• Listening to messages (and advertisements) can be layered as part of background audio, helping with multitasking while moving
AR audio opportunities for travel: Social• To date, efforts to make AR audio social have largely failed, in part
due to the challenge of most AR audio apps doing too little to be worth using; none have been better than existing social options
• Airbnb Tours may be the app that creates critical mass to encourage AR audio experiences, and the leisure nature of much of Airbnb’s business may help give AR audio an inherent social utility of experiencing AR audio in the real world together
• A simpler route that may emerge is for a photo-first platform like Instagram to make it easy to integrate audio clips with geotaggedimages, changing how people catalogue both everyday and travel experiences
AR audio opportunities for travel: Tours• AR audio elements need to be integrated into existing city-
discoverability platforms, from mainstream services like TripAdvisor & Rough Guides to startups like Dojo and Peek
• Tour needs to take guidance from both Detour and from prototypes like Loco-radio, taking users and usage possibilities into account in soundscape design
• When Detour opens its platform and Descript audio editing, it may radically accelerate tour content development (could take off the same way podcasts have)
• Integration of AR audio content into Google Maps (or similar) would be a game changer, normalizing ad hoc tour design
Recommendations to experiment w/ AR audio
Considerations for implementation• Rate of user’s movement (walking, biking, driving, etc.)• Sensory immersion level (e.g. walking=high, driving=low)• Scaling for distance and directionality of sound• Foreground vs. background layers (speech vs. music or noise)• Overlapping layers of sound• Appropriate duration of audio elements (affected by rate of
movement and density of audio elements)
Reco
mm
enda
tions
GeotouristTop option for basic, location-based tour creation
Reco
mm
enda
tions
RoundwareTop development platform for sophisticated AR audio• Roundtable is an open-source platform being used for many of the
most creative examples of AR audio • Ideal for creating and experimenting with “niche” experiences• Allows real-time, geotagged collection of audio from smartphones• Filters and plays back collective, continuous audio streams• Creates seamless, non-linear, location-sensitive layer of audio
mixed on the fly based on user input• Uses Python for server, Swift for iOS app, Java for Android app• “Soon” to be expanded to include video, photos and text
Reco
mm
enda
tions
Possible inflection points for AR audio taking off• Airbnb promoting Detour, then Detour making both creation
and distribution tools open to the public• Apple, Google or Microsoft making AR audio an integral part
of their existing map software• Shift to wireless headphones may be a minor, but notable step
towards always-on AR audio• Visual AR (Hololens, etc.) becoming widespread, and bringing
AR audio along with it (several years out)
Reco
mm
enda
tions
Thank you for reading.I HOPE THIS HAS BEEN HELPFUL.IF YOU HAVE QUESTIONS OR WISH TO DISCUSS TRAVEL IMPLICATIONS OF AUGMENTED REALITY OR OF OTHER TECHS, PLEASE REACH [email protected]