Polly
Doctoral Thesis Proposal
Viral Entertainment as a Vehicle for Disseminating Core
Development ServicesAgha Ali Raza
Thesis Committee:Roni Rosenfeld, LTI, CMU (Chair)Jaime
Carbonell, LTI, CMUBhiksha Raj, LTI, CMUUmar Saif, ITU,
PakistanBill Thies, MSRI, IndiaLanguage Technologies
Institute1ContributorsCMUChristina MiloJehanzeb SherwaniGuy
AlsterYibin LinHaohan WangRita SinghChristos FaloutsosDanai
KoutraJay YoonLUMS/ITU (Polly-Lahore)Umar SaifMansoor PervaizSamia
RazaqFarhan Ul HaqZain TariqBabajob.com (Polly-Babajob)Sean
BlagsvedtMaya ChandrasekaranArchna Bhatia
MSRI (Polly-MSRI)Bill ThiesIndrani MedhiSpandana Gella
IIT Delhi (Polly-JMV)Zahir KoradiaAaditeshwar Seth
IntroViral Ent.ConduitTest-bedReproducible2ICT4DInformation and
Communication Technologies for Development
More and better information and communication furthers the
development of a society.
Can lead to better management of available resources, improved
monitoring & reporting of corruption and more connectivity
among people
To achieve impact at a massive scale:Robust solutions to reach
the masses using available means with minimum resource
expendituresIntroViral Ent.ConduitTest-bedReproducible3Can
facilitate development where information and connectivity are the
missing components
How do you reach low-literate masses?TV, radio are
non-interactivePCs are not feasibleSmart phones are not always
feasibleText is problematic for the non-literate
Speech over simple phones is a viable way to reachlow-literate
masses
IntroViral Ent.ConduitTest-bedReproducible4PCs are not feasible
for low-literate massesNon-affluent cannot afford themUnskilled
cannot operate themStable electricity, Internet etc. cannot be
relied uponLow cost phones may be a feasible solution
Low-cost phones often only support voice & SMSVideo not
feasibleGraphical interactions not always possible
Text is problematic for low literate usersSpeech may be the only
viable interaction paradigm
Long Term VisionSpeech-based information access for low-literate
people in developing countries
Examples of information services:speech-based Craig's
Listspeech-based citizen journalismspeech-based message
boards/blogshealth, agriculture, jobs, education,IntroViral
Ent.ConduitTest-bedReproducible5We want to facilitate development
among low-literate usersWe want to do that by providing them access
to information and communication technologiesAnd we believe that it
can be effectively done using speech based services over simple
phones
We envision information services like Speech based access
to:health informationagricultural informationJob search
And communication services like, speech-based:Market
placeMessage boards and blogsCitizen journalismProblem 1: User
Training & MotivationSpeech Interfacesusually require user
training e.g. Healthline (Sherwani et al 2009), Avaaj Otalo (Patel
et al 2010)
Explicit training is not a scalable solution
Solution: Incentivize people to train themselvesWhere there's a
will there's a way [Smyth et al 2010]
Entertainment turns UI Barriers into mere speed bumps
IntroViral Ent.ConduitTest-bedReproducible6Even simple IVR
systems are often a challenge for the low-literate
6Problem 2: Mass DisseminationHow to advertise / communicate to
masses who dont read?
Solution: Viral Spread
IntroViral Ent.ConduitTest-bedReproducible77GoalsDevelop Viral
Entertainment as a vehicle for disseminating Development related
telephone based services
Research Objectives
Find the right kind of EntertainmentUse it to Introduce and
Popularize speech interfaces Use entertainment as a Viral Conduit
for delivering Core Development ServicesSetup an Experimental
Test-bed for testing speech interface choicesIntroViral
Ent.ConduitTest-bedReproducible8Can we find a:simple-to-understand,
non-controversial, quick-to-engage-and-spread form of entertainment
that is suitable for low-literate telephone users formerly
inexperienced with automated dialog systems?
Could it become Viral?
Research QuestionsIntroViral Ent.ConduitTest-bedReproducible9A
telephone-based, voice-based application which allowed users to
listen to songs recorded by others, as well as to record their own
songs and to forward them to friends
Our First Attempt: Songline
User Feedback (Pakistan)About the idea:Songs and music are
considered controversialPrivacy concerns
About the Interface:Entering phone numbers is tediousBusy tone
is confusingDetailed call tree and numerous options are
confusingIntroViral Ent.ConduitTest-bedReproducible1010Polly
Polly is a telephone-based, voice-based application which allows
users to make a short recording of their voice, modify it and send
the modified version to friends.
FeaturesNon-controversial entertainmentPeer-to-peerSimple &
Easy-to-understand (few options)
IntroViral Ent.ConduitTest-bedReproducible11112011-Pilot
LaunchSeeded with the 32 users on March 31, 2011Remained up for 22
days Amassed 2,032 users who took part in 10,629 interactions
IntroViral Ent.ConduitTest-bedReproducible12122011-Pilot
LaunchSeeded with the 32 users on March 31, 2011Remained up for 22
days Amassed 2,032 users who took part in 10,629 interactions
Reasons for Shut down:Line capacity (single phone line)
saturated within two weeksLine remained busy during peak hours
resulting in frustrated users The international call charges
($0.126/minute) were becoming a significant financial burden for
us
People kept calling Polly 40 days after its shut down when we
finally stopped monitoring the calls.
IntroViral Ent.ConduitTest-bedReproducible1313Can we use
Entertainment as a delivery vehicle for core development
services?
Research QuestionsIntroViral
Ent.ConduitTest-bedReproducible14
2012 Large Scale DeploymentDeployed locally in LahoreCall
charges went down to $0.023/minuteIncreased capacity to 30 phone
linesIncreased number of voice manipulations to sixAdded Job
Audio-Browser
IntroViral Ent.ConduitTest-bedReproducible1515First Information
Service: Job Audio-BrowserScan Pakistani newspapers for jobs for
low-skilled workers
Record these adsInvite Pollys users to audio-browse them
Ref: paperpk.comIntroViral
Ent.ConduitTest-bedReproducible1616IntroViral
Ent.ConduitTest-bedReproducible
17172012 Large Scale DeploymentLaunched on May 09, 2012:Cold
seeded with 5 most-frequent users of the pilot
After 141 days (mid-September 2012):495,000 calls85,000
usersSpreading to 1,000 new people daily
27,000 people used the job search servicelistened 279,000 times
to job ads and forwarded them 22,000 times to their friends.
IntroViral Ent.ConduitTest-bedReproducible18
2012 Large Scale Deployment
IntroViral Ent.ConduitTest-bedReproducible192012 Large Scale
Deployment
IntroViral Ent.ConduitTest-bedReproducible20After a YearPolly
had accumulated:636,000 calls165,000 users
34,000 people used the job search servicelistened 386,199 times
to 728 job ads and 19,000 users forwarded them 34,000 times to
their friends.
IntroViral Ent.ConduitTest-bedReproducible21Can we use the
virally spreading entertainment service as an Experimental test-bed
for performing randomized controlled trials and demographic
studies?Research QuestionIntroViral
Ent.ConduitTest-bedReproducible22
User Retention
A few people continue long term
Enthusiasm is Lost Quickly
IntroViral Ent.ConduitTest-bedReproducible2323
Randomized Controlled TrialsSo far we had been paying all
airtime charges using missed call mechanismGoal: Reduce our airtime
charges while maintaining system spread among the poorQuestion: How
will various quotas affect user behavior?
Introduced a caller-paid lineCalls picked up, caller pays
airtimeImpose various quotas on the number of toll-free calls per
day (for each user). When quota exceeded, direct caller to
caller-paid lineIntroViral Ent.ConduitTest-bedReproducible24
Caller-paid Line
Significantly less activity as compared to the toll-free
lineIntroViral Ent.ConduitTest-bedReproducible25
RCT: Effect of Daily Quota of 7 Calls
No appreciable difference after a weekIntroViral
Ent.ConduitTest-bedReproducible26User Demographics
Determined by listening to a sample of recordings:
Used mostly by Punjabi speaking men
IntroViral Ent.ConduitTest-bedReproducible27undetermined (young
children, old people, too much background noise, etc.).5388
recordings27User DemographicsEstimated from 207 survey calls
Low SES(No Education)Socio-Economic StatusPrimarily used by
low-educated, low-income peopleIntroViral
Ent.ConduitTest-bedReproducible2828Geographical Spread
- Seeded in Lahore and Okara
- Reached all parts of Pakistan.
- And also a handful of calls from:IndiaBelgiumOmanSaudi
ArabiaUAE
IntroViral Ent.ConduitTest-bedReproducible29Analysis of User
BehaviorDoes more experience using Polly lead to:
an improvement in users interaction skills?Use of more advanced
features,Fewer mistakes.
2. any change in usage preferences?
(Analyses are based on: 50,414 users, 292,951 calls, 934,742
menu interactions)
IntroViral Ent.ConduitTest-bedReproducible30
Barge-in Behavior in Main Menu
Clear increase in the use of barge-in with experienceIntroViral
Ent.ConduitTest-bedReproducible31
Number of times user previously encountered main menuFailed
Forwarding attempts
Clear decrease with experienceIntroViral
Ent.ConduitTest-bedReproducible32
Early Differences in Call ComplexityCan be used to predict user
retention!
IntroViral Ent.ConduitTest-bedReproducible33
33Reproducibility: Can our setup and results be reproduced in a
different country?
Research QuestionIntroViral
Ent.ConduitTest-bedReproducible34
Babajob.com is one of Indias largest informal and entry level
job portal that connects registered job seekers and employers using
their website (primarily), voice services as well as SMS(Another
benefit: A limitation of our previous study was our inability to
track employment outcomes)
Goal: Advertise Babajob to low-literate job seekers through
Polly and connect them to Babajobs voice services
The Babajob Collaboration(Sean Blagsvedt, Maya
Chandrasekaran)IntroViral Ent.ConduitTest-bedReproducible35
A local phone number in Bangalore to receive the missed
callsPolly calls back from the USSupport for Kannada (Archna) and
Hindi (Rita) voice promptsCall transfer to Babajob IVR system from
Pollys main menu.We added a feature: directory of previously-called
friendsSetupIntroViral Ent.ConduitTest-bedReproducible36
Automated calls to 84 active Babajob users in the Driver
category followed by an SMSStaff members at Babajob volunteered to
pass Pollys number on to 5 of their friends
ResultsVery little activity that declined within a few weeksVery
few users; mostly listening to job ads (not much playing or
forwarding)We suspect these users to be Job Data brokersSeeding
Attempts (via Babajob)IntroViral
Ent.ConduitTest-bedReproducible37
On Jan 9, 2014 a set of undergraduate students were asked to
spread word about Polly. They posted on their university Facebook
pages and message boardsIn the last week of Jan, 2014 Spandana gave
Pollys demo to a security guard in her building and asked him to
spread the word
ResultsOver the last 106 days:5,051 successful calls 2,793
call-back requests (from 441 users)2,258 message delivery calls (to
699 users)
Seeding Attempts (via MSRI)(Bill Thies, Indrani Medhi, Spandana
Gella)IntroViral Ent.ConduitTest-bedReproducible38
MSRI Seeding: Viral, but Not ExponentialIntroViral
Ent.ConduitTest-bedReproducible39
Neither exponential growing nor decaying!Steady stream of new
usersBeing used primarily by low-literate users in West Bengal and
Delhi
Still OngoingWhy isnt Polly Spreading Exponentially in
India?IntroViral Ent.ConduitTest-bedReproducible40Back-Channel
SpreadObservation: Polly is often introduced to new users not via a
Polly messageIndication of this: some users place their first call
without ever receiving a Polly message (Word of Mouth users)47.5%
of call-initiators in Polly-LHR pilot15% of call-initiators in
Polly-LHR full deployment18% of call-initiators in Polly-BLR
In person introduction? +demonstration?Explained in a phone
conversation?
IntroViral Ent.ConduitTest-bedReproducible41
Why is Spread Non-Exponential? (Hypotheses)Introduced as a free
voice messaging systemDifferent perception perpetuated by
back-channel spread?Introduced to college undergraduatesDifferent
behavior in this SES?Indians have bad past experience with IVR
systems?SPAM, scamsComplex billing mechanisms due to telecom
circles (=area codes)?Linguistic complexity?Dead-Skunk hypothesis:
something subtle is broken in the user experienceCultural
differences: Users expressed less interest in voice mods
IntroViral Ent.ConduitTest-bedReproducible42
JMV-collaboration (Zahir Koradia, Aaditeshwar Seth)Jharkhand
Mobile Vaani is a citizen radio-over-phone platform that makes
recorded radio content accessible through phone callsGoal:
Cross-spread: Polly and JMV
Benefits:JMV operates in the Hindi-BeltThey already have a
trusting user-baseMore flexibility of seedingThey have a vested
interest in this venture
Questions: Can Polly grow exponentially in India?Will Polly-JMV
cross-spread?
Remaining Work43
Launch Polly-JMV (May 2014)Complete analysis of Polly-BLR data
and remaining experiments (May, June 2014)Analyze data from
Polly-JMV (June, July 2014)Write it up! (July 2014)Thesis Defence
(end of July, 2014)
Thank you!Questions?
Remaining Work44
LimitationsI am not on-groundI do not understand the
culture/speak the language
Either some inherent cultural/sociological difference that we do
not appreciateWrong system image (a voice SMS system) due to
incorrect seeding? (Back-channel hypothesis).Peoples' past
experience with IVR systems has left a bad taste? In Pakistan it
was an untouched population.Phobia of pressing keys lest thy may
lose money?Fear of calling/picking up calls from another telecom
circle?More entertainment options? More free messaging options? Is
the entertainment void situation among low-SES Indians similar to
their Pakistani counterparts?Busy people, less time.
Hypotheses regarding Non-Virality45
Some system bug that we have not been able to isolateAre we
getting all the missed call requests from all the regions?Is the
SMS okay. (No!)Is the caller ID okay? (it was not. Just fixed.)Are
there pauses? Delays? buggy IVR? (None that we are aware of)Is the
language okay? (we believe so)
Hypotheses regarding Non-Virality46Microphone vs telephone based
prompt recordingsTranslation of prompts. Formal vs
informal.Outgoing caller-id problem.Quick detection of bugs. The
silent voice prompt.Problems in seeding.On ground presence and
understanding the local culture, norms, language.BLR Lessons
LearnedCaveat: Confounding User TypesThis analysis lumps together
long-term and short-term users.Hence, it confounds true learning by
any one user with differences between the different user
types.Long-term users may be more adept at using IVR systems to
start with. Changes in User Preferences:(now controlling for user
type)We define 3 user sets and explore changes in their usage
patterns as a function of their experience (number of prior Polly
calls):
Short-term users: 2,701 users who interacted with Polly exactly
5 times.Intermediate-term users: 1,862 users who interacted with
Polly exactly 10 or 11 times.Long-term users: 1,523 users who
interacted with Polly 30+ times.Menu Choices vs.
ExperienceShort-term Users (made exactly 5 calls each)
Change Y axis label to Avg. # responses per call, per
user.Remove spurious X axis tick marks (0.5, 1.5, etc.)
50Menu Choices vs. ExperienceIntermediate-term Users (made
exactly 10 or 11 calls)
Compared to short-term users; activity starts at a higher level
and climbs higher!Menu Choices vs. ExperienceLong-term Users (made
30+ calls)
Compared to intermediate-term users; activity starts at a higher
level and climbs higher!Common Trends among all user sets- The
Tendency to:1. Press 2 (forward) starts off at a high value and
stabilizesBack ChannelSystems popular image as a messaging system
2. Press 3 (next effect) increases with experienceSystem
ExplorationHunting for an effect of choice 3. Press 0 (re-record)
and 1 (repeat) decreases with experienceAdapting to 15 seconds
recording limitPlaying with the system, alone or with friends
- Invalid button presses are rare
ContributionsMulti-platform support (Voxeo Tropo,
FreeSwitch)
Ability to do quick remote pilot launch followed by local
deployment
Much more analysis to be doneDetailed logs, audio
recordings,...
Future WorkCross sell Polly-Jharkhand Mobile Vaani
Re-launch Polly in Pakistan to serve as a clearinghouse for jobs
and skill-training opportunities
Make Polly available over VBTS (Village Base Station)
Polly-Pilot: Main Use
Number of recordingsIntroGoalsPollyAnalysisPlansRemaining
percentage goes to unclear recordings Histograms of numbers
56Polly was used forAmong a sample of 63 users (reached through
survey calls):34 used it for fun; 17 for utility; and 12 for
bothFun:making prank calls to friendshello-hi/random messages,
Poetrybrowsing job ads as a pastime! Utility: More serious users
defined Polly as a voice messaging system that they use to:send
occasion (holiday, birthday) greetings, to request a call-back, to
know a friends whereabouts,to browse and apply for jobs.
Four blind users defined Polly as an alternative to text
messaging and praised it profusely.
Demographics and what Polly was used
forCountEntertainment*Utility*All recordings39974%56%By
GenderMale35973%58%Female2673%38%Unclear14By
LanguageUrdu6456%53%Punjabi19468%42%English4100%0%Pushto
(est.)10093%77%other/mixed37* Categories are neither comprehensive
nor mutually exclusive.Demographics and Usage patterns of
caller-paid callersCountEntertainment*Utility*All
recordings538825%75%By
GenderMale471326%79%Female59023%81%Unclear93By
LanguageUrdu113513%89%Punjabi348029%76%English2341%59%Pushto
(est.)70330%76%other/mixed45* Categories are neither comprehensive
nor mutually exclusive.Demographics and Usage patterns of toll-free
callers