Top Banner

of 50

Talk - Australasian Association Philosophy, 2012 - University of Wollongong

Apr 05, 2018

Download

Documents

Mirko Farina
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
  • 7/31/2019 Talk - Australasian Association Philosophy, 2012 - University of Wollongong

    1/50

    Neither Touch nor Vision:Sensory Substitution as Artificial

    Synaesthesia?

    Mirko [email protected]

    Department of Cognitive Science

    ARC Centre of Excellence in

    Cognition and its Disorders

    mailto:[email protected]:[email protected]
  • 7/31/2019 Talk - Australasian Association Philosophy, 2012 - University of Wollongong

    2/50

    Sensory substitution devices (SSDs) thusprovide through an unusual sensory

    modality (the substitutingmodality) access

    to items of the world that are generallyexperienced through another sensory

    channel(the substitutedmodality)

    The term sensory substitutionrefers to the use of a sensory

    modality to supply environmentalinformation normally gathered by

    another sense

    So, the Idea

  • 7/31/2019 Talk - Australasian Association Philosophy, 2012 - University of Wollongong

    3/50

    The principles of sensorysubstitution have been

    formulated by Bach-y-Rita,

    who conducted experimentswith the potential of the skin as

    a medium for transmittingpictorial material

    His endeavours have led to theproduction of two categories of

    systems. Visual-to-tactile substitutiondevices that convert images into

    tactile stimuli, and visual-to-auditorysubstitution systems that transform

    images into sounds

  • 7/31/2019 Talk - Australasian Association Philosophy, 2012 - University of Wollongong

    4/50

    To give you concreteexamples of the functioning ofthese systems, I next focus ontwo of the devices (the Brain-

    Port and the vOICe), andbriefly discuss them asillustrations of the two

    categories abovementioned

  • 7/31/2019 Talk - Australasian Association Philosophy, 2012 - University of Wollongong

    5/50

    The Brain Port

  • 7/31/2019 Talk - Australasian Association Philosophy, 2012 - University of Wollongong

    6/50

    The vOICe

  • 7/31/2019 Talk - Australasian Association Philosophy, 2012 - University of Wollongong

    7/50

    What type ofperceptual

    experiencethe

    practiced user ofa sensory

    substitutiondevice can be

    said to undergo?

    SSDs raise many interesting philosophicalquestions

  • 7/31/2019 Talk - Australasian Association Philosophy, 2012 - University of Wollongong

    8/50

    Three possible solutions

    SSD perception remains inthe substituting modality (so itis either auditory/tactile)

    SSD perception switches

    from the substituting to thesubstituted modality (so it

    become visual with training)

    SSD perception is neitherof them but something

    different and partially new

  • 7/31/2019 Talk - Australasian Association Philosophy, 2012 - University of Wollongong

    9/50

    Spatial Encoding & Cognitive Inference

    Prinz and Block accept thatthe SSD perceiver has

    experiences with spatialsignificance but they deny

    that this spatial significanceis visual in character

    They concede that SSDs encode spatial contents but argue

    that this only enables the visually impaired to use somefeatures of the proximal stimulus to make cognitive inferenceson the basis of dedicated neural pathways

  • 7/31/2019 Talk - Australasian Association Philosophy, 2012 - University of Wollongong

    10/50

    To corroborate this claim,Block appeals to a study in

    which Bach-y-Rita and Kercel

    (2003) report the experienceof a TDUuser.

    The subject, they write, onlydescribes visual means of

    analysis (e.g. parallax) butnot visual phenomenology

    and even during taskperformance with the sensory

    systemshe can perceivepurely tactile sensationswhen asked to concentrate

    on these sensations

  • 7/31/2019 Talk - Australasian Association Philosophy, 2012 - University of Wollongong

    11/50

    Prinz concedes that once thesubjects have mastered the

    device, they can use the

    apparatus to avoid obstacles,locate items in the world or adjust

    their behavioural dispositions

    Driving a car allowsus to experience the

    surface of the asphalt;

    yet the qualities of thisexperience, cannot besaid to be visual but

    rather remain

    essentially tactileHe nevertheless remains scepticalabout the possibilities of treatingthese responses as distinctively

    visual because there are conditionsunder which we can use touch to

    sense objects that are not incontact with our bodies

  • 7/31/2019 Talk - Australasian Association Philosophy, 2012 - University of Wollongong

    12/50

    In essence, Block and Prinzthink that SSD perception istactile/auditory (B) because

    it is not visual (A)

    For Block and Prinz, spatialencoding of contents and distal

    cognitive inferences are

    therefore the best things animpaired user can get out of the

    coupling with these devices

    I entirely disagree with themon (B) but I am quitesympathetic with (A)

  • 7/31/2019 Talk - Australasian Association Philosophy, 2012 - University of Wollongong

    13/50

    Does the pars construens of the B&Punderstanding persuade us?

    Why dont Prinz andBlock offer us an

    account of how wemake the inference

    from tactile sensationsto distal phenomena?That is, why dont they

    give us an explanationof what the inferenceis by which we get it?

    It seems that Prinz andBlock at most establishthat distal attribution does

    not entail a change inmode of perception tovision. It doesnt followthat SSD perception

    remains in the substitutingmodality. It could, for

    instance, result in a newform of perceiving and

    experiencing

  • 7/31/2019 Talk - Australasian Association Philosophy, 2012 - University of Wollongong

    14/50

    Prinz and Block thought thatfocusing on some TDU reportswould suffice to make generalclaims about SSD perception,

    and that TDU considerationswould also apply to other visual-to-tactile substitution devices andto visual-to-auditory substitution

    systems. But this is amethodological mistake!

    Methodological issues in theirargument (1)

  • 7/31/2019 Talk - Australasian Association Philosophy, 2012 - University of Wollongong

    15/50

    if Block and Prinz had analysed other types of systems ordiscriminated more accurately among the testimonies;

    they would have probably found out that training directlyimpacts on the capacity to interpret the stimuli and bears

    on the ability to perceive with the device.

    Methodological issues in theirargument (2)

    In other words, if they had look at a widerrange of cases and devices would have

    noticed the incompleteness of their positiveproposal

  • 7/31/2019 Talk - Australasian Association Philosophy, 2012 - University of Wollongong

    16/50

    Is it rather Vision?

    Hurley and No haveargued that aftersubstantial trainingand adaptation thephenomenology of

    the perceptionobtained through the

    coupling with anSSD switches from

    tactile/auditory tovisual

  • 7/31/2019 Talk - Australasian Association Philosophy, 2012 - University of Wollongong

    17/50

    Similarities betweenNatural Vision and SSD

    Perception

    Same SensorimotorContingencies

    Appeal toPhenomenological

    Reports

    The Arguments for sayingit is Vision

  • 7/31/2019 Talk - Australasian Association Philosophy, 2012 - University of Wollongong

    18/50

    Seeing and Seeing with an SSD - Analogies

    SSD users attest to theemergence of visual effects

    (such as parallax, looming, andzooming) or of visual illusions

    (the waterfall illusion) afterwearing the device

    Both natural vision and

    SSD perception aregoverned and directedby analogous laws of

    occlusion

  • 7/31/2019 Talk - Australasian Association Philosophy, 2012 - University of Wollongong

    19/50

    The Nature of SSD perception

    The more the user is trained with thedevice, the more she masters it. Themore the subject masters the device,

    the more invariants her acquired

    perception shares with vision. Themore invariants the acquired

    perception shares with natural visionthe more it resembles it

    Depends on the amount and quality of

    sensorimotor contingencies that theacquired perception shares with naturalvision and on the intensity and duration

    of the coupling with the device

  • 7/31/2019 Talk - Australasian Association Philosophy, 2012 - University of Wollongong

    20/50

    Very soon after I hadlearned how to scan, the

    sensations no longer felt as

    if they were on my back,and I became less and less

    aware that vibrating pinswere making contact with

    my skin[Guarniero (1974),p. 104].

    Training is necessary to discover theresemblances with vision!

  • 7/31/2019 Talk - Australasian Association Philosophy, 2012 - University of Wollongong

    21/50

    Does the Hurley and No understandingreally hold up? (Probably not)

    Striem-Amit and colleagues (2012)reported that 5 out of 9 practisedusers of the vOICe exceeded theWorld Health Organization (WHO)

    blindness acuity threshold (20/400)when using the device

    There is still a substantial gap (380/400) between the

    perception that these systems offer and the perceptualexperiences that one enjoys through natural vision!

  • 7/31/2019 Talk - Australasian Association Philosophy, 2012 - University of Wollongong

    22/50

  • 7/31/2019 Talk - Australasian Association Philosophy, 2012 - University of Wollongong

    23/50

    SSDs do not provide their userswith colours, neither endow

    them with the capacity ofsmooth tracking objects, or theability to fully perceive depth

    Sure there can still be visionwithout colours, but it seems

    extremely difficult to experiencecharacteristically visual

    sensations without the possibilityof discriminating depth properly orwithout the ability to smooth track

    even slow moving objects

    More Problems !

  • 7/31/2019 Talk - Australasian Association Philosophy, 2012 - University of Wollongong

    24/50

    For these reasons, SSDs seem tobe a long way from the goal of

    achieving a real sense of vision

    Using tactile or auditory stimulation, it

    is possible only to provide a few

    aspects of normal visual impressions,like the quality of being out there in theworld, and of conveying informationabout spatial layout and object form.

    But the image-like quality of vision still

    seems far away [ORegan (2011),p.142-143]

  • 7/31/2019 Talk - Australasian Association Philosophy, 2012 - University of Wollongong

    25/50

    ...gathered a group of six

    normal sighted subjectsblindfolded and trainedthem with the vOICe for

    fifteen hours

    The participantsunderwent a series ofexperiments involving

    localization, recognition,

    and discrimination tasks

    While performing these

    experiments, the subjects weresolicited to report their qualitativeapprehension of the use of the

    device both for localization and for

    recognition/discrimination tasks

    In particular, the participantswere asked the following

    question: to which sensorymodality would you compareyour experience

    Auvrayet al 2007

  • 7/31/2019 Talk - Australasian Association Philosophy, 2012 - University of Wollongong

    26/50

  • 7/31/2019 Talk - Australasian Association Philosophy, 2012 - University of Wollongong

    27/50

    While subjects could develop visual-likeabilities in localisation tasks, the very

    same people had difficulties inexperiencing full-blown visual awareness

    in recognition and discrimination tasks

    As a consequence, the conveyedqualitative experience was not

    automatically associated to either audition

    or vision but its task-like nature was verymuch emphasized

    Interpreting the Results

  • 7/31/2019 Talk - Australasian Association Philosophy, 2012 - University of Wollongong

    28/50

    An option that these findingsleave open is to interpret the

    perception with the device asfloating between different sensory

    modalities in accordance to thetask given and the tool used

    So from these results one couldinfer that SSD perception, in

    expert users at least, is differentboth from hearing/touch and

    vision but neverthelesspossesses both of these

    components

  • 7/31/2019 Talk - Australasian Association Philosophy, 2012 - University of Wollongong

    29/50

    SSDs provide a novel

    mode of experience to itsuser and the best way tocapture this novelty is tothink about the use of an

    SSD as being somethingcloser to the developmentof a new set of automatic

    recognition abilities,emerging from other

    sensory modalities andother pre-existing

    capacities

    Beyond Vision?

  • 7/31/2019 Talk - Australasian Association Philosophy, 2012 - University of Wollongong

    30/50

    Something new: but what exactly?

    If the phenomenology of SSD

    perception doesnt stay in onemodality but exploits the pre-existing senses to give thevisually impaired a different

    kind of perception, couldnt wejust speculate that SSDperception, in giving the

    visually impaired somethingnew, blends vision with

    hearing or touch?

  • 7/31/2019 Talk - Australasian Association Philosophy, 2012 - University of Wollongong

    31/50

    Artificial Synaesthesia?

    Synaesthesia is acondition in whichstimulation of one

    sensory or

    cognitive pathwayleads to automaticand unintentionaloccurrences in a

    second sensory orcognitive pathway

  • 7/31/2019 Talk - Australasian Association Philosophy, 2012 - University of Wollongong

    32/50

    Synaesthesia isoften described asa merging of thesenses, a cross-modal union of

    different sensorymodalities or of

    different featureswithin the same

    sensory modality

  • 7/31/2019 Talk - Australasian Association Philosophy, 2012 - University of Wollongong

    33/50

    Different categories ofsynaesthetic

    experience

    Higher VS LowerSynaesthetes

    [Ramachandran andHubbard (2001) ]

    Projectors VS Associators

    [Dixon et al. 2004]

    The 5 definitional aspects of synaesthesia

  • 7/31/2019 Talk - Australasian Association Philosophy, 2012 - University of Wollongong

    34/50

    The 5 definitional aspects of synaesthesia[Cytowic and Eagleman 2009]

    Involuntary ExtremelyMemorable

    Ladenwith Affect**

    Consistent***

    SpatiallyExtended*

    Th N i B hi d

  • 7/31/2019 Talk - Australasian Association Philosophy, 2012 - University of Wollongong

    35/50

    The Neuroscience BehindSynaesthesia

    To explain the neurocognitive mechanisms thatcharacterise this phenomenon Cohen Kadosh, Walsh,

    and Henik and Grossenbacher & Lovelace (2001), havesuggested that synaesthesia is due to disinhibition orunmasking of signals between or within brain areas

    Th N i b hi d SSD (1/2)

  • 7/31/2019 Talk - Australasian Association Philosophy, 2012 - University of Wollongong

    36/50

    The Neuroscience behind SSDs (1/2)

    Amedi et al. (2007) show that the

    the lateral-occipital-temporalcortex (LOtv) is activated when

    using the vOICe

    Ptito and Kupers (2005)

    demonstrate that patients tonguescan act as portals to convey

    somatosensory information to thevisual cortex

    SSDs work by exploiting thecross-modal plasticity of our

    brain!

  • 7/31/2019 Talk - Australasian Association Philosophy, 2012 - University of Wollongong

    37/50

    The Neuroscience behind SSDs (2/2)

    Kupers and colleagues(2011) have favouredan account of cross-

    modal plasticity in SSDusers that involvesdisinhibition of existingpathways over a viewthat prescribes cortical

    reorganisation

  • 7/31/2019 Talk - Australasian Association Philosophy, 2012 - University of Wollongong

    38/50

    If cross-modalplasticity in SSD users

    is explained in termsof disinhibition and thisform of disinhibition or

    unmasking alsocharacterises the

    phenomenon ofsynaesthesia, couldnt

    we propose that aform of artificially

    induced synaesthesiacan occur in SSD

    perception?

  • 7/31/2019 Talk - Australasian Association Philosophy, 2012 - University of Wollongong

    39/50

    A Report:

    Monochrome artificially induced synaesthesia in certain

    frequencies of sound...The thing I experience is not incolor, is in my mind's eye, and can be very distracting.The shapes are consistent and can be reproduced bythe same sound. It is almost as if you had a computer

    with two monitors running simultaneously differentpictures, one was a very grey blurred version of the realworld, and the other was a pure grey background with abig semi-circular light grey arc on it, and sometimes you

    switched your attention between both [Ward & Meijer(2010),p.497-498].

  • 7/31/2019 Talk - Australasian Association Philosophy, 2012 - University of Wollongong

    40/50

    PF:Now it has developed into what I perceive as colorExperimenter:Really? Before you had said that it is notcolored?PF:Yes, thats true.But before my brain wasntseeing the finer detail. Over time my brain seems to have

    developed, and pulled out everything it can from thesoundscape and then used my memory to color everything

    Another Report:The emergence of Colors

    .Experimenter:How long ago was it when youstarted having the colors?PF:Gradual, gradual

    but it is strongest now. Within the past year, yearand a half, after my depth perception[Ward &

    Meijer (2010)]

  • 7/31/2019 Talk - Australasian Association Philosophy, 2012 - University of Wollongong

    41/50

    Stability/Consistency over time: a hallmarkof Synaestetic Experience !

    PF also reports hersynaesthetic experiences

    to the soundscapesproduced by the vOICe to

    be stable over time

    ....I thought my mood, time,

    environment factors, etc.,would change the images,

    color, motion effects. It doesnot[quoted in Ward &

    Meijer (2010)]

  • 7/31/2019 Talk - Australasian Association Philosophy, 2012 - University of Wollongong

    42/50

    What about the other 4 definitional aspectsof synaesthesia?

    Quite often many objects

    spatial properties (suchas shape, width and

    location) can be

    experienced by the blindperceivers through

    hearing or touch and thenspatialized when the

    users are asked to reportabout their position

  • 7/31/2019 Talk - Australasian Association Philosophy, 2012 - University of Wollongong

    43/50

    Is SSD perception automatic?

    After extensivepractise, the device

    gets increasinglytransparent, its

    boundariesprogressively fade

    away, and theperception experienced

    through the coupling

    with it becomesinvoluntary

  • 7/31/2019 Talk - Australasian Association Philosophy, 2012 - University of Wollongong

    44/50

    Is SSD perception highly memorable?

    It is memorable because itis extremely learnable. Viatraining the vOICe userslearn how to interpret the

    stimulation coming from the

    device and their braingradually internalise thevOICe rules for mapping,redeploying them even

    when the subjects are notwearing the device

  • 7/31/2019 Talk - Australasian Association Philosophy, 2012 - University of Wollongong

    45/50

    SSD perception isnt however

    (normally) laden with affect. Intruth not all forms of

    synaesthesia are laden withaffect and so this particulardefinitional aspect doesnt

    seem to be necessary for the

    emergence of synaestheticexperiences

    Nor does the fact that SSDperception isnt laden with

    affect preclude thecorroboration of my argument

    What have we learned?

  • 7/31/2019 Talk - Australasian Association Philosophy, 2012 - University of Wollongong

    46/50

    SSD perceptioncould resemblea novel (cross-modal) form of

    perception,

    namely a kindof artificially

    inducedsynaesthesia

    What have we learned?

    First, the experienced users of an SSDseem to undergo a form of perceiving that

    fulfil many of the key properties thatcharacterise standard synaesthetic

    experiences

    Third, there is a strong analogy, at the level of

    neuro-cognitive mechanisms of activation (theidea of disinhibition), between developmentalforms of synaesthesia and SSD perception

    Second, both SSD perception andsynaesthesia are cross-modal

  • 7/31/2019 Talk - Australasian Association Philosophy, 2012 - University of Wollongong

    47/50

    However, the level ofgenerality and the

    credibility of the reportsthat provide empirical

    grounds for the analogyI just presented is at

    stake

    What kinds of generalisationcan we make based on two

    single reports and whatcredibility do these reports

    can achieve?

    Sadly, no hard evidence has been produced so far to confirmthe idea that SSD perception is a form of synaesthesia andunfortunately we just don't have those many proficient blind

    SSDs users yet to test this hypothesis empirically and makea statistical argument on a larger scale

  • 7/31/2019 Talk - Australasian Association Philosophy, 2012 - University of Wollongong

    48/50

    In this paper, Inevertheless hope to

    have demonstratedthat SSD perception in

    experienced user isneither tactile nor

    visual (in astraightforward way)

    and that it rathercounts as something

    different (cross-modal),

    and quite possibly new;a kind of artificially

    induced synaesthesia?

  • 7/31/2019 Talk - Australasian Association Philosophy, 2012 - University of Wollongong

    49/50

  • 7/31/2019 Talk - Australasian Association Philosophy, 2012 - University of Wollongong

    50/50

    Special Thanks to:

    John Sutton

    Julian Kiverstein

    Andy Clark Malika Auvray

    All of You!