7/31/2019 Talk - Australasian Association Philosophy, 2012 - University of Wollongong
1/50
Neither Touch nor Vision:Sensory Substitution as Artificial
Synaesthesia?
Mirko [email protected]
Department of Cognitive Science
ARC Centre of Excellence in
Cognition and its Disorders
mailto:[email protected]:[email protected]7/31/2019 Talk - Australasian Association Philosophy, 2012 - University of Wollongong
2/50
Sensory substitution devices (SSDs) thusprovide through an unusual sensory
modality (the substitutingmodality) access
to items of the world that are generallyexperienced through another sensory
channel(the substitutedmodality)
The term sensory substitutionrefers to the use of a sensory
modality to supply environmentalinformation normally gathered by
another sense
So, the Idea
7/31/2019 Talk - Australasian Association Philosophy, 2012 - University of Wollongong
3/50
The principles of sensorysubstitution have been
formulated by Bach-y-Rita,
who conducted experimentswith the potential of the skin as
a medium for transmittingpictorial material
His endeavours have led to theproduction of two categories of
systems. Visual-to-tactile substitutiondevices that convert images into
tactile stimuli, and visual-to-auditorysubstitution systems that transform
images into sounds
7/31/2019 Talk - Australasian Association Philosophy, 2012 - University of Wollongong
4/50
To give you concreteexamples of the functioning ofthese systems, I next focus ontwo of the devices (the Brain-
Port and the vOICe), andbriefly discuss them asillustrations of the two
categories abovementioned
7/31/2019 Talk - Australasian Association Philosophy, 2012 - University of Wollongong
5/50
The Brain Port
7/31/2019 Talk - Australasian Association Philosophy, 2012 - University of Wollongong
6/50
The vOICe
7/31/2019 Talk - Australasian Association Philosophy, 2012 - University of Wollongong
7/50
What type ofperceptual
experiencethe
practiced user ofa sensory
substitutiondevice can be
said to undergo?
SSDs raise many interesting philosophicalquestions
7/31/2019 Talk - Australasian Association Philosophy, 2012 - University of Wollongong
8/50
Three possible solutions
SSD perception remains inthe substituting modality (so itis either auditory/tactile)
SSD perception switches
from the substituting to thesubstituted modality (so it
become visual with training)
SSD perception is neitherof them but something
different and partially new
7/31/2019 Talk - Australasian Association Philosophy, 2012 - University of Wollongong
9/50
Spatial Encoding & Cognitive Inference
Prinz and Block accept thatthe SSD perceiver has
experiences with spatialsignificance but they deny
that this spatial significanceis visual in character
They concede that SSDs encode spatial contents but argue
that this only enables the visually impaired to use somefeatures of the proximal stimulus to make cognitive inferenceson the basis of dedicated neural pathways
7/31/2019 Talk - Australasian Association Philosophy, 2012 - University of Wollongong
10/50
To corroborate this claim,Block appeals to a study in
which Bach-y-Rita and Kercel
(2003) report the experienceof a TDUuser.
The subject, they write, onlydescribes visual means of
analysis (e.g. parallax) butnot visual phenomenology
and even during taskperformance with the sensory
systemshe can perceivepurely tactile sensationswhen asked to concentrate
on these sensations
7/31/2019 Talk - Australasian Association Philosophy, 2012 - University of Wollongong
11/50
Prinz concedes that once thesubjects have mastered the
device, they can use the
apparatus to avoid obstacles,locate items in the world or adjust
their behavioural dispositions
Driving a car allowsus to experience the
surface of the asphalt;
yet the qualities of thisexperience, cannot besaid to be visual but
rather remain
essentially tactileHe nevertheless remains scepticalabout the possibilities of treatingthese responses as distinctively
visual because there are conditionsunder which we can use touch to
sense objects that are not incontact with our bodies
7/31/2019 Talk - Australasian Association Philosophy, 2012 - University of Wollongong
12/50
In essence, Block and Prinzthink that SSD perception istactile/auditory (B) because
it is not visual (A)
For Block and Prinz, spatialencoding of contents and distal
cognitive inferences are
therefore the best things animpaired user can get out of the
coupling with these devices
I entirely disagree with themon (B) but I am quitesympathetic with (A)
7/31/2019 Talk - Australasian Association Philosophy, 2012 - University of Wollongong
13/50
Does the pars construens of the B&Punderstanding persuade us?
Why dont Prinz andBlock offer us an
account of how wemake the inference
from tactile sensationsto distal phenomena?That is, why dont they
give us an explanationof what the inferenceis by which we get it?
It seems that Prinz andBlock at most establishthat distal attribution does
not entail a change inmode of perception tovision. It doesnt followthat SSD perception
remains in the substitutingmodality. It could, for
instance, result in a newform of perceiving and
experiencing
7/31/2019 Talk - Australasian Association Philosophy, 2012 - University of Wollongong
14/50
Prinz and Block thought thatfocusing on some TDU reportswould suffice to make generalclaims about SSD perception,
and that TDU considerationswould also apply to other visual-to-tactile substitution devices andto visual-to-auditory substitution
systems. But this is amethodological mistake!
Methodological issues in theirargument (1)
7/31/2019 Talk - Australasian Association Philosophy, 2012 - University of Wollongong
15/50
if Block and Prinz had analysed other types of systems ordiscriminated more accurately among the testimonies;
they would have probably found out that training directlyimpacts on the capacity to interpret the stimuli and bears
on the ability to perceive with the device.
Methodological issues in theirargument (2)
In other words, if they had look at a widerrange of cases and devices would have
noticed the incompleteness of their positiveproposal
7/31/2019 Talk - Australasian Association Philosophy, 2012 - University of Wollongong
16/50
Is it rather Vision?
Hurley and No haveargued that aftersubstantial trainingand adaptation thephenomenology of
the perceptionobtained through the
coupling with anSSD switches from
tactile/auditory tovisual
7/31/2019 Talk - Australasian Association Philosophy, 2012 - University of Wollongong
17/50
Similarities betweenNatural Vision and SSD
Perception
Same SensorimotorContingencies
Appeal toPhenomenological
Reports
The Arguments for sayingit is Vision
7/31/2019 Talk - Australasian Association Philosophy, 2012 - University of Wollongong
18/50
Seeing and Seeing with an SSD - Analogies
SSD users attest to theemergence of visual effects
(such as parallax, looming, andzooming) or of visual illusions
(the waterfall illusion) afterwearing the device
Both natural vision and
SSD perception aregoverned and directedby analogous laws of
occlusion
7/31/2019 Talk - Australasian Association Philosophy, 2012 - University of Wollongong
19/50
The Nature of SSD perception
The more the user is trained with thedevice, the more she masters it. Themore the subject masters the device,
the more invariants her acquired
perception shares with vision. Themore invariants the acquired
perception shares with natural visionthe more it resembles it
Depends on the amount and quality of
sensorimotor contingencies that theacquired perception shares with naturalvision and on the intensity and duration
of the coupling with the device
7/31/2019 Talk - Australasian Association Philosophy, 2012 - University of Wollongong
20/50
Very soon after I hadlearned how to scan, the
sensations no longer felt as
if they were on my back,and I became less and less
aware that vibrating pinswere making contact with
my skin[Guarniero (1974),p. 104].
Training is necessary to discover theresemblances with vision!
7/31/2019 Talk - Australasian Association Philosophy, 2012 - University of Wollongong
21/50
Does the Hurley and No understandingreally hold up? (Probably not)
Striem-Amit and colleagues (2012)reported that 5 out of 9 practisedusers of the vOICe exceeded theWorld Health Organization (WHO)
blindness acuity threshold (20/400)when using the device
There is still a substantial gap (380/400) between the
perception that these systems offer and the perceptualexperiences that one enjoys through natural vision!
7/31/2019 Talk - Australasian Association Philosophy, 2012 - University of Wollongong
22/50
7/31/2019 Talk - Australasian Association Philosophy, 2012 - University of Wollongong
23/50
SSDs do not provide their userswith colours, neither endow
them with the capacity ofsmooth tracking objects, or theability to fully perceive depth
Sure there can still be visionwithout colours, but it seems
extremely difficult to experiencecharacteristically visual
sensations without the possibilityof discriminating depth properly orwithout the ability to smooth track
even slow moving objects
More Problems !
7/31/2019 Talk - Australasian Association Philosophy, 2012 - University of Wollongong
24/50
For these reasons, SSDs seem tobe a long way from the goal of
achieving a real sense of vision
Using tactile or auditory stimulation, it
is possible only to provide a few
aspects of normal visual impressions,like the quality of being out there in theworld, and of conveying informationabout spatial layout and object form.
But the image-like quality of vision still
seems far away [ORegan (2011),p.142-143]
7/31/2019 Talk - Australasian Association Philosophy, 2012 - University of Wollongong
25/50
...gathered a group of six
normal sighted subjectsblindfolded and trainedthem with the vOICe for
fifteen hours
The participantsunderwent a series ofexperiments involving
localization, recognition,
and discrimination tasks
While performing these
experiments, the subjects weresolicited to report their qualitativeapprehension of the use of the
device both for localization and for
recognition/discrimination tasks
In particular, the participantswere asked the following
question: to which sensorymodality would you compareyour experience
Auvrayet al 2007
7/31/2019 Talk - Australasian Association Philosophy, 2012 - University of Wollongong
26/50
7/31/2019 Talk - Australasian Association Philosophy, 2012 - University of Wollongong
27/50
While subjects could develop visual-likeabilities in localisation tasks, the very
same people had difficulties inexperiencing full-blown visual awareness
in recognition and discrimination tasks
As a consequence, the conveyedqualitative experience was not
automatically associated to either audition
or vision but its task-like nature was verymuch emphasized
Interpreting the Results
7/31/2019 Talk - Australasian Association Philosophy, 2012 - University of Wollongong
28/50
An option that these findingsleave open is to interpret the
perception with the device asfloating between different sensory
modalities in accordance to thetask given and the tool used
So from these results one couldinfer that SSD perception, in
expert users at least, is differentboth from hearing/touch and
vision but neverthelesspossesses both of these
components
7/31/2019 Talk - Australasian Association Philosophy, 2012 - University of Wollongong
29/50
SSDs provide a novel
mode of experience to itsuser and the best way tocapture this novelty is tothink about the use of an
SSD as being somethingcloser to the developmentof a new set of automatic
recognition abilities,emerging from other
sensory modalities andother pre-existing
capacities
Beyond Vision?
7/31/2019 Talk - Australasian Association Philosophy, 2012 - University of Wollongong
30/50
Something new: but what exactly?
If the phenomenology of SSD
perception doesnt stay in onemodality but exploits the pre-existing senses to give thevisually impaired a different
kind of perception, couldnt wejust speculate that SSDperception, in giving the
visually impaired somethingnew, blends vision with
hearing or touch?
7/31/2019 Talk - Australasian Association Philosophy, 2012 - University of Wollongong
31/50
Artificial Synaesthesia?
Synaesthesia is acondition in whichstimulation of one
sensory or
cognitive pathwayleads to automaticand unintentionaloccurrences in a
second sensory orcognitive pathway
7/31/2019 Talk - Australasian Association Philosophy, 2012 - University of Wollongong
32/50
Synaesthesia isoften described asa merging of thesenses, a cross-modal union of
different sensorymodalities or of
different featureswithin the same
sensory modality
7/31/2019 Talk - Australasian Association Philosophy, 2012 - University of Wollongong
33/50
Different categories ofsynaesthetic
experience
Higher VS LowerSynaesthetes
[Ramachandran andHubbard (2001) ]
Projectors VS Associators
[Dixon et al. 2004]
The 5 definitional aspects of synaesthesia
7/31/2019 Talk - Australasian Association Philosophy, 2012 - University of Wollongong
34/50
The 5 definitional aspects of synaesthesia[Cytowic and Eagleman 2009]
Involuntary ExtremelyMemorable
Ladenwith Affect**
Consistent***
SpatiallyExtended*
Th N i B hi d
7/31/2019 Talk - Australasian Association Philosophy, 2012 - University of Wollongong
35/50
The Neuroscience BehindSynaesthesia
To explain the neurocognitive mechanisms thatcharacterise this phenomenon Cohen Kadosh, Walsh,
and Henik and Grossenbacher & Lovelace (2001), havesuggested that synaesthesia is due to disinhibition orunmasking of signals between or within brain areas
Th N i b hi d SSD (1/2)
7/31/2019 Talk - Australasian Association Philosophy, 2012 - University of Wollongong
36/50
The Neuroscience behind SSDs (1/2)
Amedi et al. (2007) show that the
the lateral-occipital-temporalcortex (LOtv) is activated when
using the vOICe
Ptito and Kupers (2005)
demonstrate that patients tonguescan act as portals to convey
somatosensory information to thevisual cortex
SSDs work by exploiting thecross-modal plasticity of our
brain!
7/31/2019 Talk - Australasian Association Philosophy, 2012 - University of Wollongong
37/50
The Neuroscience behind SSDs (2/2)
Kupers and colleagues(2011) have favouredan account of cross-
modal plasticity in SSDusers that involvesdisinhibition of existingpathways over a viewthat prescribes cortical
reorganisation
7/31/2019 Talk - Australasian Association Philosophy, 2012 - University of Wollongong
38/50
If cross-modalplasticity in SSD users
is explained in termsof disinhibition and thisform of disinhibition or
unmasking alsocharacterises the
phenomenon ofsynaesthesia, couldnt
we propose that aform of artificially
induced synaesthesiacan occur in SSD
perception?
7/31/2019 Talk - Australasian Association Philosophy, 2012 - University of Wollongong
39/50
A Report:
Monochrome artificially induced synaesthesia in certain
frequencies of sound...The thing I experience is not incolor, is in my mind's eye, and can be very distracting.The shapes are consistent and can be reproduced bythe same sound. It is almost as if you had a computer
with two monitors running simultaneously differentpictures, one was a very grey blurred version of the realworld, and the other was a pure grey background with abig semi-circular light grey arc on it, and sometimes you
switched your attention between both [Ward & Meijer(2010),p.497-498].
7/31/2019 Talk - Australasian Association Philosophy, 2012 - University of Wollongong
40/50
PF:Now it has developed into what I perceive as colorExperimenter:Really? Before you had said that it is notcolored?PF:Yes, thats true.But before my brain wasntseeing the finer detail. Over time my brain seems to have
developed, and pulled out everything it can from thesoundscape and then used my memory to color everything
Another Report:The emergence of Colors
.Experimenter:How long ago was it when youstarted having the colors?PF:Gradual, gradual
but it is strongest now. Within the past year, yearand a half, after my depth perception[Ward &
Meijer (2010)]
7/31/2019 Talk - Australasian Association Philosophy, 2012 - University of Wollongong
41/50
Stability/Consistency over time: a hallmarkof Synaestetic Experience !
PF also reports hersynaesthetic experiences
to the soundscapesproduced by the vOICe to
be stable over time
....I thought my mood, time,
environment factors, etc.,would change the images,
color, motion effects. It doesnot[quoted in Ward &
Meijer (2010)]
7/31/2019 Talk - Australasian Association Philosophy, 2012 - University of Wollongong
42/50
What about the other 4 definitional aspectsof synaesthesia?
Quite often many objects
spatial properties (suchas shape, width and
location) can be
experienced by the blindperceivers through
hearing or touch and thenspatialized when the
users are asked to reportabout their position
7/31/2019 Talk - Australasian Association Philosophy, 2012 - University of Wollongong
43/50
Is SSD perception automatic?
After extensivepractise, the device
gets increasinglytransparent, its
boundariesprogressively fade
away, and theperception experienced
through the coupling
with it becomesinvoluntary
7/31/2019 Talk - Australasian Association Philosophy, 2012 - University of Wollongong
44/50
Is SSD perception highly memorable?
It is memorable because itis extremely learnable. Viatraining the vOICe userslearn how to interpret the
stimulation coming from the
device and their braingradually internalise thevOICe rules for mapping,redeploying them even
when the subjects are notwearing the device
7/31/2019 Talk - Australasian Association Philosophy, 2012 - University of Wollongong
45/50
SSD perception isnt however
(normally) laden with affect. Intruth not all forms of
synaesthesia are laden withaffect and so this particulardefinitional aspect doesnt
seem to be necessary for the
emergence of synaestheticexperiences
Nor does the fact that SSDperception isnt laden with
affect preclude thecorroboration of my argument
What have we learned?
7/31/2019 Talk - Australasian Association Philosophy, 2012 - University of Wollongong
46/50
SSD perceptioncould resemblea novel (cross-modal) form of
perception,
namely a kindof artificially
inducedsynaesthesia
What have we learned?
First, the experienced users of an SSDseem to undergo a form of perceiving that
fulfil many of the key properties thatcharacterise standard synaesthetic
experiences
Third, there is a strong analogy, at the level of
neuro-cognitive mechanisms of activation (theidea of disinhibition), between developmentalforms of synaesthesia and SSD perception
Second, both SSD perception andsynaesthesia are cross-modal
7/31/2019 Talk - Australasian Association Philosophy, 2012 - University of Wollongong
47/50
However, the level ofgenerality and the
credibility of the reportsthat provide empirical
grounds for the analogyI just presented is at
stake
What kinds of generalisationcan we make based on two
single reports and whatcredibility do these reports
can achieve?
Sadly, no hard evidence has been produced so far to confirmthe idea that SSD perception is a form of synaesthesia andunfortunately we just don't have those many proficient blind
SSDs users yet to test this hypothesis empirically and makea statistical argument on a larger scale
7/31/2019 Talk - Australasian Association Philosophy, 2012 - University of Wollongong
48/50
In this paper, Inevertheless hope to
have demonstratedthat SSD perception in
experienced user isneither tactile nor
visual (in astraightforward way)
and that it rathercounts as something
different (cross-modal),
and quite possibly new;a kind of artificially
induced synaesthesia?
7/31/2019 Talk - Australasian Association Philosophy, 2012 - University of Wollongong
49/50
7/31/2019 Talk - Australasian Association Philosophy, 2012 - University of Wollongong
50/50
Special Thanks to:
John Sutton
Julian Kiverstein
Andy Clark Malika Auvray
All of You!