Page 1
Iconicity, Frequency, and Sign Iconicity, Frequency, and Sign length effects in length effects in
TSL lexical processingTSL lexical processingYi-Hsuan ChiuYi-Hsuan Chiu
Cognitive Neuropsychology Laboratory, National YanCognitive Neuropsychology Laboratory, National Yang-Ming University, Taipei, Taiwang-Ming University, Taipei, Taiwan
Institute of Neuroscience, School of Life Science, NatioInstitute of Neuroscience, School of Life Science, National Yang-Ming University, Taipei, Taiwannal Yang-Ming University, Taipei, Taiwan
Page 2
An animal or a tool?
DOG KEY
Page 3
Iconicity of sign languageIconicity of sign languageThe components ofThe components of
LocationLocationHandshapeHandshapeMovementMovement
constrain the meaning of the objects!constrain the meaning of the objects!
Language ---------- Language ---------- ThoughtThought
Page 4
Iconicity of sign languageIconicity of sign languageThe components ofThe components of
LocationLocationHandshapeHandshapeMovementMovement
constrain the meaning of the objects!constrain the meaning of the objects!
Language ---------- Language ---------- ThoughtThought
iconiciiconicityty
Page 5
Iconicity of sign languageIconicity of sign languageThe components ofThe components of
LocationLocationHandshapeHandshapeMovementMovement
constrain the meaning of the objects!constrain the meaning of the objects!
iconiciiconicityty
Language --- Language --- Thought Thought
Page 6
The usage of Sign languages in The usage of Sign languages in TaiwanTaiwan
Sign language in TaiwanSign language in TaiwanNature sign language (Taiwan Sign Language)Nature sign language (Taiwan Sign Language)
ArbitraryArbitraryIconicityIconicity
Vision-basedVision-based Manipulation-basedManipulation-based
{the spatial information (location, handshape, mo{the spatial information (location, handshape, movement) is meaningful}vement) is meaningful}
Chinese Character Sign LanguageChinese Character Sign Language
Page 7
Iconicity in Sign LanguageIconicity in Sign Language
Page 8
Visuion-basedVisuion-based Manipulation-Manipulation-basedbased
handshapehandshape All possibleAll possibleshape-for-shape iconicityshape-for-shape iconicityGINGER GINGER Path-for-shape iconicityPath-for-shape iconicityVACEVACE
Ease articulatorEase articulator
shape-for-shape iconicity shape-for-shape iconicity KITCHEN-KNIFE KITCHEN-KNIFE GLASSGLASS
locationlocation body-for-body iconicity body-for-body iconicity ELEPHANTELEPHANTNeutral: VACENeutral: VACE Neutral: KEY, GLASSNeutral: KEY, GLASS
movementmovement atemporal motion atemporal motion
Fictive motionFictive motion
Path-for-path iconicityPath-for-path iconicityFactive motionFactive motion
Page 9
Sign lexical accessSign lexical access Task: Picture signing, picture delay signing, Task: Picture signing, picture delay signing,
picture naming, picture delay naming picture naming, picture delay naming Subjects: Subjects: deaf signerdeaf signer, , interpreterinterpreter (hear signer), and (hear signer), and
normalnormal (hear non-singer) (hear non-singer) Independent variablesIndependent variables
Iconicity: manipulation, visionIconicity: manipulation, vision Frequency: high, lowFrequency: high, low Sign length: short, longSign length: short, long
Dependent variable: reaction timeDependent variable: reaction time
+ &$Release button to sign/ name out
RT
Page 10
Picture signing / picture namingPicture signing / picture naming
+ &$Release button to sign/ name out
RT
Picture delay signing / picture delay namingPicture delay signing / picture delay naming
+ &$
Release button to sign/ name out
RT
*
1s1s
Page 11
Behavior studyBehavior study
Page 12
Picture naming/ signingPicture naming/ signing Language modalityLanguage modality NamingNaming SigningSigning
Visual processing/Visual processing/ConceptConcept
0-150ms 0-150ms Lexical accessLexical access
150-275ms 150-275ms (lemma) (lemma) 275-400ms 275-400ms (Phonological encoding)(Phonological encoding)
Phonetic/ArticulationPhonetic/Articulation
400-600ms 400-600ms
Page 13
Picture naming/ signingPicture naming/ signing
NamingNaming SigningSigning
ConceptConcept Iconicity: no difference (all noun)Iconicity: no difference (all noun)
Lexical accessLexical access Sign length: Rt (Short < Long)Sign length: Rt (Short < Long)
Frequency: Rt (High < Low)Frequency: Rt (High < Low)
ArticulationArticulation
RT
Page 14
Picture delay naming/ signingPicture delay naming/ signing
Delay NamingDelay Naming Delay SigningDelay Signing
ConceptConcept Iconicity: no difference (all noun)Iconicity: no difference (all noun)
Lexical accessLexical access Sign length: Rt (ShortSign length: Rt (Short = = Long)Long)
Frequency: Rt (High Frequency: Rt (High == Low) Low)
ArticulationArticulation
RT
Language modality
Page 15
SubjectsSubjects
Deaf Deaf InterpreterInterpreter NormalNormal
SigningSigning
NamingNaming ??
Hearing Hearing abilityability
Dif.: Hearing loss Dif.: Language experience
However, what is the interpreter’s mother tongue?
Page 16
Deaf signing (n=24)Deaf signing (n=24)
All subjects learned TSL before their All subjects learned TSL before their puberty, and took TSL as their daily puberty, and took TSL as their daily language.language.
M=14, F=10M=14, F=10Mean age= 37.04 years-old (19-57ys)Mean age= 37.04 years-old (19-57ys)
Page 17
Deaf and normal (N=24)Deaf and normal (N=24)Deaf signing and normal naming
300
800
1300
Signing and naming
Rea
ctio
n tim
e
1數列 657.4459492 448.3453684 932.1413069 397.8814922
Signing De. Signing Naming De. Naming
D/SD/S N/NN/N pp **
SubjectSubject(deaf/ normal)(deaf/ normal)
552.90552.90 665.01665.01 0.00000.0000 ****
TaskTask(signing/delay signing)(signing/delay signing)
794.79794.79 423.11423.11 0.00000.0000 ****
InteractionInteraction 0.00000.0000 ****
Page 18
Deaf and normal (N=24)Deaf and normal (N=24)
pp **
InteractionInteraction 0.00000.0000 ****
sign-de. signsign-de. sign 657.45657.45 448.35448.35 0.00000.0000 ****
name- de. namename- de. name 932.14932.14 397.88397.88 0.00000.0000 ****
sign-namesign-name 657.45657.45 932.14932.14 0.00000.0000 ****
de. sign- de. namede. sign- de. name 448.35448.35 397.88397.88 0.11200.1120
Deaf signing and normal naming
300
500
700
900
1100
Signing Naming
Signing or naming
Rea
ctio
n tim
e
Picture
Picture delay**
**
**
IconicityIconicity
Articulation
Page 19
N=24N=24 DeafDeaf InterpreterInterpreter NormalNormal
Signing/Signing/
GesturingGesturingIconicityIconicity ** ****
FrequencyFrequency **** **** ****
Sign lengthSign length **** **** ****
Delay Delay signingsigning
IconicityIconicity
FrequencyFrequency
Sign lengthSign length
NamingNaming IconicityIconicity ****
FrequencyFrequency **** ****
Sign lengthSign length **** ****
Delay Delay namingnaming
IconicityIconicity
FrequencyFrequency
Sign lengthSign length **
Page 20
ERP studyERP study
Page 21
Picture naming/ signingPicture naming/ signing
Iconicity: (verb < noun)~ Iconicity: (verb < noun)~ (manipulation < vision) N400(manipulation < vision) N400
Frequency: 225-250ms (High < Low)Frequency: 225-250ms (High < Low)
350-450ms (High < Low) 350-450ms (High < Low) N400N400
Sign length: 60-120ms (Short < Long) Sign length: 60-120ms (Short < Long)
Page 22
Time windowTime windowWindow (ms)Window (ms) IconicityIconicity FrequencyFrequency Sign lengthSign length
60-9060-90
90-12090-120
60-12060-120
125-175125-175
125-225125-225
150-200150-200
200-300200-300
250-350250-350
285-335285-335
350-400350-400
450-550450-550
550-600550-600
Page 23
Location (channel) Location (channel)
Brain areasBrain areas ChannelsChannels Ico.Ico. Fre.Fre. Leng.Leng.
Frontal cortexFrontal cortex F1, FCz, F2F1, FCz, F2 M>VM>V
M<VM<V
H<LH<L
Inferior frontal (44)/ Inferior frontal (44)/ premotor cortex (6)premotor cortex (6)
FC5, FC6FC5, FC6 M>VM>V
Motor cortexMotor cortex C3, C4C3, C4 M>VM>V
Occipital cortexOccipital cortex O1, Oz, O2O1, Oz, O2 M<VM<V S<LS<L
Page 24
Deaf SigningDeaf Signing (N=8) (N=8) andand Normal Normal naming naming (N=12)(N=12)
O2O1
OZ
T8
FCZ
T7
PO5
POZ
PO6
F6F5 FPZ
Subject: EEG file: name.avg Recorded : 12:55:36 24-May-2004Rate - 500 Hz, HPF - 0.05 Hz, LPF - 100 Hz, Notch - off
NeuroscanSCAN 4.3Printed : 16:47:45 28-May-2004
Page 25
NounsNouns and and verbsverbs
Federmeier et al. 2000Federmeier et al. 2000
NounVerb
N400: N>V
Page 26
Deaf signing (Iconicity: Deaf signing (Iconicity: MM//VV)) (N=8) (N=8)
ms
-100.0 150.0 400.0 650.0 900.0uV 0.0
-1.0
-2.0
-3.0
-4.0
1.0
2.0
3.0
4.0
*Msign_0.avgVsign_0.avg
Electrode: F1
Subject: EEG file: Msign_0.avg Recorded : 15:54:01 30-Apr-2004Rate - 500 Hz, HPF - 0.05 Hz, LPF - 100 Hz, Notch - off
NeuroscanSCAN 4.3Printed : 18:30:13 12-May-2004
ms
-100.0 150.0 400.0 650.0 900.0uV 0.0
-1.0
-2.0
-3.0
-4.0
1.0
2.0
3.0
4.0
*Msign_0.avgVsign_0.avg
Electrode: F2
Subject: EEG file: Msign_0.avg Recorded : 15:54:01 30-Apr-2004Rate - 500 Hz, HPF - 0.05 Hz, LPF - 100 Hz, Notch - off
NeuroscanSCAN 4.3Printed : 18:30:36 12-May-2004
ms
-100.0 150.0 400.0 650.0 900.0uV 0.0
-1.0
-2.0
-3.0
-4.0
1.0
2.0
3.0
4.0
*Msign_0.avgVsign_0.avg
Electrode: FZ
Subject: EEG file: Msign_0.avg Recorded : 15:54:01 30-Apr-2004Rate - 500 Hz, HPF - 0.05 Hz, LPF - 100 Hz, Notch - off
NeuroscanSCAN 4.3Printed : 18:29:16 12-May-2004
ms
-100.0 150.0 400.0 650.0 900.0uV 0.0
-1.0
-2.0
-3.0
-4.0
1.0
2.0
3.0
4.0
*Msign_0.avgVsign_0.avg
Electrode: PO3
Subject: EEG file: Msign_0.avg Recorded : 15:54:01 30-Apr-2004Rate - 500 Hz, HPF - 0.05 Hz, LPF - 100 Hz, Notch - off
NeuroscanSCAN 4.3Printed : 18:32:40 12-May-2004
ms
-100.0 150.0 400.0 650.0 900.0uV 0.0
-1.0
-2.0
-3.0
-4.0
1.0
2.0
3.0
4.0
*Msign_0.avgVsign_0.avg
Electrode: POZ
Subject: EEG file: Msign_0.avg Recorded : 15:54:01 30-Apr-2004Rate - 500 Hz, HPF - 0.05 Hz, LPF - 100 Hz, Notch - off
NeuroscanSCAN 4.3Printed : 18:32:23 12-May-2004
ms
-100.0 150.0 400.0 650.0 900.0uV 0.0
-1.0
-2.0
-3.0
-4.0
1.0
2.0
3.0
4.0
*Msign_0.avgVsign_0.avg
Electrode: PO4
Subject: EEG file: Msign_0.avg Recorded : 15:54:01 30-Apr-2004Rate - 500 Hz, HPF - 0.05 Hz, LPF - 100 Hz, Notch - off
NeuroscanSCAN 4.3Printed : 18:33:00 12-May-2004
ms
-100.0 150.0 400.0 650.0 900.0uV 0.0
-1.0
-2.0
-3.0
-4.0
1.0
2.0
3.0
4.0
*Msign_0.avgVsign_0.avg
Electrode: CP6
Subject: EEG file: Msign_0.avg Recorded : 15:54:01 30-Apr-2004Rate - 500 Hz, HPF - 0.05 Hz, LPF - 100 Hz, Notch - off
NeuroscanSCAN 4.3Printed : 18:34:11 12-May-2004
Page 27
Frequency (Frequency (HH<<LL)) 225-250225-250: : HH<<LL 350-450350-450: : HH<<LL
Muente et al. Muente et al. 20012001
Page 28
Deaf signing (Fre.: Deaf signing (Fre.: HH//LL)) (N=8) (N=8)
ms
-100.0 150.0 400.0 650.0 900.0
uV0.0
-1.0
-2.0
-3.0
-4.0
-5.0
1.0
2.0
3.0
4.0
*Hfresign_0.avgLfresign_0.avg
Electrode: F1
Subject: EEG file: Hfresign_0.avg Recorded : 08:49:15 01-May-2004Rate - 500 Hz, HPF - 0.05 Hz, LPF - 100 Hz, Notch - off
NeuroscanSCAN 4.3Printed : 15:36:09 12-May-2004
ms
-100.0 150.0 400.0 650.0 900.0
uV0.0
-1.0
-2.0
-3.0
-4.0
-5.0
1.0
2.0
3.0
4.0
*Hfresign_0.avgLfresign_0.avg
Electrode: F8
Subject: EEG file: Hfresign_0.avg Recorded : 08:49:15 01-May-2004Rate - 500 Hz, HPF - 0.05 Hz, LPF - 100 Hz, Notch - off
NeuroscanSCAN 4.3Printed : 15:39:37 12-May-2004
ms
-100.0 150.0 400.0 650.0 900.0
uV0.0
-1.0
-2.0
-3.0
-4.0
-5.0
1.0
2.0
3.0
4.0
*Hfresign_0.avgLfresign_0.avg
Electrode: C4
Subject: EEG file: Hfresign_0.avg Recorded : 08:49:15 01-May-2004Rate - 500 Hz, HPF - 0.05 Hz, LPF - 100 Hz, Notch - off
NeuroscanSCAN 4.3Printed : 15:42:06 12-May-2004
ms
-100.0 150.0 400.0 650.0 900.0
uV0.0
-1.0
-2.0
-3.0
-4.0
-5.0
1.0
2.0
3.0
4.0
*Hfresign_0.avgLfresign_0.avg
Electrode: C6
Subject: EEG file: Hfresign_0.avg Recorded : 08:49:15 01-May-2004Rate - 500 Hz, HPF - 0.05 Hz, LPF - 100 Hz, Notch - off
NeuroscanSCAN 4.3Printed : 15:42:29 12-May-2004
ms
-100.0 150.0 400.0 650.0 900.0
uV0.0
-1.0
-2.0
-3.0
-4.0
-5.0
1.0
2.0
3.0
4.0
*Hfresign_0.avgLfresign_0.avg
Electrode: F3
Subject: EEG file: Hfresign_0.avg Recorded : 08:49:15 01-May-2004Rate - 500 Hz, HPF - 0.05 Hz, LPF - 100 Hz, Notch - off
NeuroscanSCAN 4.3Printed : 15:37:14 12-May-2004
225-250ms225-250ms:H<L
Page 29
Sign length (Sign length (SS< < LL))
60-120ms60-120ms: : SS<<LL
Assadollahi et al. 2003 MEG
Short wordLong word
Page 30
ms
-100.0 150.0 400.0 650.0 900.0uV 0.0
-1.0
-2.0
-3.0
-4.0
1.0
2.0
3.0
4.0
*Slensign_0.avgLlensign_0.avg
Electrode: OZ
Subject: EEG file: Slensign_0.avg Recorded : 08:49:15 01-May-2004Rate - 500 Hz, HPF - 0.05 Hz, LPF - 100 Hz, Notch - off
NeuroscanSCAN 4.3Printed : 18:03:19 12-May-2004
ms
-100.0 150.0 400.0 650.0 900.0uV 0.0
-1.0
-2.0
-3.0
-4.0
1.0
2.0
3.0
4.0
*Slensign_0.avgLlensign_0.avg
Electrode: O1
Subject: EEG file: Slensign_0.avg Recorded : 08:49:15 01-May-2004Rate - 500 Hz, HPF - 0.05 Hz, LPF - 100 Hz, Notch - off
NeuroscanSCAN 4.3Printed : 18:03:49 12-May-2004
ms
-100.0 150.0 400.0 650.0 900.0uV 0.0
-1.0
-2.0
-3.0
-4.0
1.0
2.0
3.0
4.0
*Slensign_0.avgLlensign_0.avg
Electrode: PO5
Subject: EEG file: Slensign_0.avg Recorded : 08:49:15 01-May-2004Rate - 500 Hz, HPF - 0.05 Hz, LPF - 100 Hz, Notch - off
NeuroscanSCAN 4.3Printed : 18:04:35 12-May-2004
ms
-100.0 150.0 400.0 650.0 900.0uV 0.0
-1.0
-2.0
-3.0
-4.0
1.0
2.0
3.0
4.0
*Slensign_0.avgLlensign_0.avg
Electrode: PO6
Subject: EEG file: Slensign_0.avg Recorded : 08:49:15 01-May-2004Rate - 500 Hz, HPF - 0.05 Hz, LPF - 100 Hz, Notch - off
NeuroscanSCAN 4.3Printed : 18:06:06 12-May-2004
Deaf signing (Leng.: Deaf signing (Leng.: SS//LL)) (N=8) (N=8)60-120ms60-120ms:S<L
Page 31
ERP data summaryERP data summary
IconicityIconicity FrequencyFrequency Sign lengthSign length
N400N400
N>V N>V classclass
V>M V>M ico.ico.
225-250225-250
H<LH<L
60-12060-120
S<LS<L
Deaf signingDeaf signing ** ** **
Normal namingNormal naming **
Interpreter signingInterpreter signing ** S>L ?S>L ?
Interpreter namingInterpreter naming **
Page 32
SummarySummaryFrequency, sign length effects in TSL lexiconFrequency, sign length effects in TSL lexicon~~
Chinese lexicon (spoken language lexicon)Chinese lexicon (spoken language lexicon)
Sign language Iconicity ~ Sign language Iconicity ~ semanticsemanticRT: signing< naming, delay signing~ delay namingRT: signing< naming, delay signing~ delay namingERP: 100-200, 200-300ms , signing< namingERP: 100-200, 200-300ms , signing< namingNormal gesture: RT: M<VNormal gesture: RT: M<V
Iconicity: Iconicity: ERP: Manipulation/Vision~ Verbs/NounsERP: Manipulation/Vision~ Verbs/Nouns