-
ExamplesConditional independence
General independence
Graphs and Conditional Independence
Steffen Lauritzen, University of Oxford
Graphical Models, Lecture 1, Michaelmas Term 2011
October 10, 2011
Steffen Lauritzen, University of Oxford Graphs and Conditional
Independence
-
ExamplesConditional independence
General independence
A directed graphical model
Directed graphical model (Bayesian network) showing
relationsbetween risk factors, diseases, and symptoms.
Steffen Lauritzen, University of Oxford Graphs and Conditional
Independence
-
ExamplesConditional independence
General independence
A pedigree
Graphical model for a pedigree from study of Werner’s
syndrome.Each node is itself a graphical model.
Steffen Lauritzen, University of Oxford Graphs and Conditional
Independence
-
ExamplesConditional independence
General independence
A large pedigree
p1379p1380 p1381 p1382 p1383p1384p1385p1386p1387
p1129p1128 p1133p1134p1135 p1136p1137
p1536p1535
p1131p1408p1407p1406p1405p1404
p1126
p1396 p1395p1394 p1393p1392 p1391p1390p1389p1388
p1127
p1426 p1425p1424p1423
p1130
p1565p1564p1530
p1132
p1073
p1457p1456p1455 p1454
p1078
p1349
p1079
p1226p1225 p1224p1223 p1222p1221p1220
p1076
p1242 p1241p1240p1239 p1238p1237
p1561
p1236
p1575
p1235
p1499
p1234
p1480p1511 p1534
p1233
p1219
p1074
p1376p1375 p1374 p1373p1372 p1371p1370 p1369p1368
p1077
p1232 p1231p1230 p1229p1228p1227
p1072
p1438p1437p1436 p1435 p1434 p1439
p1080
p1362p1361 p1360
p1190
p1449p1450p1451 p1452p1453
p1187
p1427 p1428p1429 p1430p1431 p1432p1433
p1184 p1189p1188p1186p1185
p1075
p927p928p929
p331
p544p543p542
p328
p1585 p657p659p660
p1556p1555 p1557
p767
p1514
p1568
p765
p875p876p877
p766
p661p662
p745
p842
p1367 p1366p1365 p1364p1363
p746p747
p658
p506
p744p742p743
p512
p712p709p711
p784 p783p782 p781p780p779p778
p704
p864p843
p706
p856p855
p710
p869 p868p870
p708
p1465p1443p1442
p707
p1422
p705
p511
p760p701p702
p1576
p773p879
p772 p774p775 p776p777
p1572 p1571
p840
p703
p507
p716
p509p510p508
p505
p1170
p1573
p1167
p1169p1168 p1166p1165p1163p1162 p1161
p1554p1553
p1164
p964
p1463p1464
p963
p965
p644p645
p504
p571 p572
p1488 p1487p1486
p1173
p1569
p1174 p1171p1175p1176p1177 p1172
p1099
p1183
p1096
p1110 p1109p1108 p1107
p1547p1533
p1296 p1295p1297 p1298p1299 p1300p1301p1302 p1303
p1105
p1350 p1351p1352
p1102p1106p1104 p1103
p1097 p1098p1100p1101
p930 p931
p1591
p1603 p1602p1601 p1600p1599 p1598p1597p1596p1595 p1594 p1604
p1590 p1589
p1586
p1592
p1587
p932p933
p503
p327
p541
p1460p1377
p1581p1580
p1461 p1378
p1081p1082
p1044p1045
p540p937 p621p622
p750 p749
p625
p624 p759
p1093 p1091p1090p1089
p1322p1321p1320p1319 p1318 p1317p1316p1315 p1314p1313p1323
p1092
p623p626
p1083
p934 p935p936 p606
p538
p576
p539
p329p330
p70p51
p34
p1070
p1459 p1094
p1462
p1095
p1071
p1119p1118
p1479
p1117 p1116p1115p1113 p1112
p1359p1358p1357p1356p1355 p1354p1353
p1114
p1532p1542
p1120
p1032
p1159p1158 p1157
p1475p1503p1502 p1501p1500
p1155
p1515 p1543p1485
p1154
p1513p1469
p1153
p1545p1546
p1156p1152 p1151p1160
p1033p1034
p901
p959 p958 p957p956p955
p1069
p1584
p1068
p1468
p1067
p1328
p1063p1066
p1440 p1441
p1065p1064 p1061p1060
p1348 p1347 p1346p1345 p1344 p1343p1342 p1341p1340 p1339
p1062
p954
p1562
p1247
p1484
p1245
p1560 p1559
p1246 p1248p1249p1250 p1251p1252p1253 p1254
p1087p1084 p1085
p1255p1256p1257 p1258 p1259p1260 p1261 p1262 p1263p1264
p1086 p1088
p949 p953 p952p951
p1327 p1324p1325p1326
p1001p1002 p999
p1286
p1507 p1531
p1287p1288p1289 p1290p1291p1292p1293p1294
p998 p997
p1312
p1548p1549
p1304
p1582
p1306p1311 p1310p1309p1308 p1307 p1305
p1000
p1409p1410p1411 p1412 p1413p1414p1415 p1416 p1467
p1148
p1150p1403 p1400 p1401p1402
p1149
p950
p902
p1040 p1031p1041
p903p904p906
p1039p1038 p1037p1036p1035
p905
p263
p198 p26
p440 p441
p647 p646
p442
p231
p228
p446
p730
p513
p1211p1212
p1570p1574
p1210 p1213 p1214 p1215p1216p1217
p519
p518 p517 p516p515p514
p445
p600
p599
p748
p601
p602
p444
p1042
p1147
p1397 p1398p1399
p1141p1142
p1498p1497 p1496p1466
p1143
p1506 p1524p1523p1522
p1144
p1541p1505p1504 p1478p1474
p1145
p1567p1544
p1146
p1043
p892
p443p447
p227
p269
p960
p1218
p961p962
p268p632
p751
p1420p1419 p1418 p1417p1421
p752
p633
p634
p264
p620
p1593
p757
p756
p796 p795p794
p753
p854p848
p754
p755
p619p618
p267 p266
p582
p265
p226
p966p968
p1284p1283 p1282p1281p1280p1279 p1278p1277p1276p1275 p1274
p1273p1285
p969
p970 p971
p972
p973
p1209 p1208p1207 p1206p1205p1204p1203
p1566
p1202
p1470p1471p1472
p1200p1198
p1540 p1539p1538p1537
p1199
p1516p1517 p1518p1519
p1201
p967
p888
p229
p230
p5p6
p907 p908p909p910p911 p912
p560 p562
p570p569
p561p563
p106 p340
p181
p439
p438
p1550 p1551 p1552
p736
p731
p732 p846 p803p802 p801 p800 p799p798p797
p733p734p735p737
p536p594
p847
p652
p651
p650
p648 p649
p810 p809p808p807p806 p805p804
p653
p593
p1003
p764p763p762
p861p862
p761
p729
p595 p596p597p598
p437
p881 p880
p436
p526
p434
p1028
p1448p1447 p1446p1445p1444
p1029p1030
p476
p700p699 p698p697
p849
p696
p818p819p820 p821p822p823 p824 p825
p695
p865p872p878
p694 p693p689 p690 p692
p811p812 p813p814 p815p816p817
p691
p683 p1022p1024 p1025
p1528p1527p1526p1525 p1529
p1179
p1583 p1563
p1180p1181
p1178
p1023
p1243 p1244
p1473
p1140 p1138
p1489p1476
p1139
p1026p1027
p919p921 p920p887 p886
p435
p284
p581p580p579p578
p341 p342
p49 p183p182
p73 p346
p52 p185
p283
p259
p564
p260
p395
p399
p677
p758
p676
p397
p720 p722p721p719 p718 p717
p398
p396
p549
p257
p258
p394
p845
p672p667 p668 p671
p669 p670p866p857
p673 p674
p675
p391
p389p1111
p390p393p392
p261
p262
p27p28
p252
p474
p640 p643
p826p827p828 p829p830 p831 p832p839
p642p641 p639
p256
p410
p400
p412
p401
p414p413 p411
p402p403p404 p405 p406p407p408 p409
p387p386 p385
p489p490
p383 p384
p1059p1057 p1058
p388
p251 p254 p255
p863
p841p792 p791p790p789p788p787 p786p785
p738
p853p852 p851 p850
p739
p873 p874p867
p740
p502
p1579
p1056p1050
p1048 p1049 p1051
p1495
p1052p1053p1054
p1578 p1577
p1055
p501
p678
p793
p682
p681
p680p679
p500
p498p499
p253
p241
p475
p242
p943 p974p975p976p977 p978p979
p890
p1182p980p981p982 p983p984 p985
p889 p473
p1007p1008 p1009 p1010 p1011 p1012
p915p917 p918
p1021
p916
p461
p1013
p891
p243
p15 p16
p535
p306
p532
p713
p556p714
p715
p531p530p529p528p527
p304
p728p726
p1477p1490
p1193
p1491p1492 p1493p1494
p1192
p1191 p741p1194
p727
p555
p305
p36 p37
p148
p525
p3
p431
p885p884
p429
p883p882
p430
p638
p1046 p1047
p637 p635p636
p428
p1020 p1018p1016
p1512
p1458 p1197
p1017
p1019
p432
p433
p274p275
p84p85
p497
p858p844
p836
p496
p838
p688 p687 p686 p685p684
p491
p723
p494p495 p493 p492
p272
p419p416 p418p417
p420
p273
p8
p1014 p1015
p893
p995 p994
p1483p1482 p1481
p1196p1195
p996
p894
p895 p896
p232 p449
p603 p605
p604
p448
p382
p768 p769p770p771
p381
p380
p379
p378 p630p629p628 p627
p376
p617 p612p614
p1329 p1330p1331p1332 p1333 p1334p1335p1336p1337 p1338
p613
p871
p616
p615
p611
p377
p233
p450
p2
p270p271
p427
p860 p859
p837
p725 p724
p533 p534
p426
p554p553
p415
p278
p41
p577 p458p456 p454p455p453 p460p459 p457
p336p337
p42p56
p551p550 p552
p50p314
p315
p98p99
p900 p899 p897p898
p573
p942
p574 p575
p566
p941
p944
p939p940
p565p567 p568
p296p297
p92 p93
p343
p352p351 p350
p344
p104 p105
p76
p246 p349 p472p471p470p469
p468
p466
p609
p610
p607p608
p467 p465
p244
p425 p421
p423 p424 p422
p245p247
p17 p18
p1
p520
p656 p655
p664
p663
p665
p835p834 p833
p666
p654
p521
p631p922p923
p546p548 p547 p545 p926 p925p924
p522
p279
p79 p80
p299
p39p40
p477p478
p479
p480
p287
p312
p353
p311
p332 p333p523 p524
p310
p288
p89 p90
p43
p187 p55 p67
p174
p69
p1004
p1005
p1006
p913
p1125
p1124 p1123p1122
p1121
p914
p145 p463
p464
p462
p236 p234p235
p237
p61 p62
p11 p7
p83
p298
p33p178
p206
p250
p22p249p248
p19 p20
p286
p176 p285
p483 p482p481
p291 p307
p308
p309
p29 p97
p150
p159
p127
p177
p25
p301p300
p94p95
p316
p317 p318
p100
p488p487 p486p485
p484
p335p334
p292 p293p294p295
p68 p91
p31
p320p345 p451
p452
p10p282
p59 p60
p96
p125
p363
p141 p155p24
p23
p146
p120p110 p132p130
p4p12
p47
p240
p339p290 p289
p238p239
p13p14
p180p38p321
p48 p175
p144
p211 p375
p374
p369 p588
p1520p1521
p586
p1508p1509 p1510
p584
p1558
p945
p585
p368
p151 p152
p195
p372
p592
p591p589
p590
p373
p170p171
p168p364
p142
p172 p202
p131p163
p169
p118
p107 p122p119
p989
p1272p1271 p1270p1269 p1268p1267 p1266 p1265
p986 p987
p990
p991p992p993
p1614p1613 p1612p1611 p1610p1609p1608p1607p1606 p1605
p988
p938
p143 p537
p46p280 p281
p87 p88
p276
p277
p86 p72
p203
p116 p123p109
p138
p164
p200
p225
p192
p947 p1588
p946p948
p139p587
p193 p365
p367
p366
p153p154
p194
p189
p147
p124p219
p191
p126p128 p583
p362
p190p137
p188
p165 p157
p173
p114 p160
p149 p207 p112 p224
p319
p35
p9
p121
p204 p201
p216 p166
p133
p208
p220p217 p371
p370
p134 p135
p108
p117
p162
p111
p221
p115
p158
p205
p167
p215p213
p196
p197
p223
p214p209
p179
p113p212 p222p129 p210p161
p184
p103
p156
p218
p557p558 p559
p313
p44 p199
p53
p140
p82
p303 p302
p21 p30
p45 p136
p322p323p324
p63 p64
p347
p325p326
p101 p102
p74p338
p65p66
p348
p75p186
p359p360p361
p354p355 p356p357p358
p77p78
p81
Family relationship of 1641 members of Greenland
Eskimopopulation.
Steffen Lauritzen, University of Oxford Graphs and Conditional
Independence
-
ExamplesConditional independence
General independence
Basic ideas and pitfallsFormal definitionFundamental
properties
Independence
We recall that two random variables X and Y are independent
if
P(X ∈ A |Y = y) = P(X ∈ A)
or, equivalently, if
P{(X ∈ A) ∩ (Y ∈ B)} = P(X ∈ A)P(Y ∈ B).
For discrete variables this is equivalent to
pij = pi+p+j
where pij = P(X = i ,Y = j) and pi+ =∑
j pij etc., whereas forcontinuous variables the requirement is a
factorization of the jointdensity:
fXY (x , y) = fX (x)fY (y).
When X and Y are independent we write X ⊥⊥Y .Steffen Lauritzen,
University of Oxford Graphs and Conditional Independence
-
ExamplesConditional independence
General independence
Basic ideas and pitfallsFormal definitionFundamental
properties
Admissions to Berkeley by department
Here are three variables A: Admitted?, S : Sex, and
D:Department.
Department Sex Whether admittedYes No
I Male 512 313Female 89 19
II Male 353 207Female 17 8
III Male 120 205Female 202 391
IV Male 138 279Female 131 244
V Male 53 138Female 94 299
VI Male 22 351Female 24 317
When dealing with complex systems of many random variables,
wemust have a concept which is more sophisticated, but
equallyfundamental: that of conditional independence.
Steffen Lauritzen, University of Oxford Graphs and Conditional
Independence
-
ExamplesConditional independence
General independence
Basic ideas and pitfallsFormal definitionFundamental
properties
For three variables it is of interest to see whether
independenceholds for fixed value of one of them, e.g. is the
admissionindependent of sex for every department separately?
We denote this as A⊥⊥ S |D and display it graphically asu u uA D
S
Algebraically, this corresponds to the relations
pijk = pi+|k p+j |k p++k =pi+k p+jk
p++k.
Steffen Lauritzen, University of Oxford Graphs and Conditional
Independence
-
ExamplesConditional independence
General independence
Basic ideas and pitfallsFormal definitionFundamental
properties
Marginal and conditional independence
Note that there the two conditions
A⊥⊥ S , A⊥⊥ S |D
are very different and will typically not both hold unless we
eitherhave A⊥⊥ (D,S) or (A,D)⊥⊥ S , i.e. if one of the variables
arecompletely independent of both of the others.
This fact is a simple form of what is known as
Yule–Simpsonparadox.It can be much worse than this: A positive
conditional associationcan turn into a negative marginal
association and vice-versa.
Steffen Lauritzen, University of Oxford Graphs and Conditional
Independence
-
ExamplesConditional independence
General independence
Basic ideas and pitfallsFormal definitionFundamental
properties
Admissions revisited
Admissions to Berkeley
Sex Whether admittedYes No
Male 1198 1493Female 557 1278
Note this marginal table shows much lower admission rates
forfemales.Considering the departments separately, there is only a
differencefor department I, and it is the other way around...
Steffen Lauritzen, University of Oxford Graphs and Conditional
Independence
-
ExamplesConditional independence
General independence
Basic ideas and pitfallsFormal definitionFundamental
properties
Admissions to Berkeley by department
Department Sex Whether admittedYes No
I Male 512 313Female 89 19
II Male 353 207Female 17 8
III Male 120 205Female 202 391
IV Male 138 279Female 131 244
V Male 53 138Female 94 299
VI Male 22 351Female 24 317
Apart from Department I, it holds that A⊥⊥ S |D. In DepartmentI,
a higher proportion of females are admitted!
Steffen Lauritzen, University of Oxford Graphs and Conditional
Independence
-
ExamplesConditional independence
General independence
Basic ideas and pitfallsFormal definitionFundamental
properties
Florida murderers
Sentences in 4863 murder cases in Florida over the six
years1973-78
SentenceMurderer Death Other
Black 59 2547White 72 2185
The table shows a greater proportion of white murderers
receivingdeath sentence than black (3.2% vs. 2.3%), although
thedifference is not big, the picture seems clear.
Steffen Lauritzen, University of Oxford Graphs and Conditional
Independence
-
ExamplesConditional independence
General independence
Basic ideas and pitfallsFormal definitionFundamental
properties
Controlling for colour of victim
SentenceVictim Murderer Death Other
Black Black 11 2309White 0 111
White Black 48 238White 72 2074
Now the table for given colour of victim shows a very
differentpicture. In particular, note that 111 white murderers
killed blackvictims and none were sentenced to death.
Steffen Lauritzen, University of Oxford Graphs and Conditional
Independence
-
ExamplesConditional independence
General independence
Basic ideas and pitfallsFormal definitionFundamental
properties
Formal definition
Random variables X and Y are conditionally independent given
therandom variable Z if
L(X |Y ,Z ) = L(X |Z ).
We then write X ⊥⊥Y |Z (or X ⊥⊥P Y |Z )Intuitively:Knowing Z
renders Y irrelevant for predicting X .Factorisation of
densities:
X ⊥⊥Y |Z ⇐⇒ f (x , y , z)f (z) = f (x , z)f (y , z)⇐⇒ ∃a, b : f
(x , y , z) = a(x , z)b(y , z).
Steffen Lauritzen, University of Oxford Graphs and Conditional
Independence
-
ExamplesConditional independence
General independence
Basic ideas and pitfallsFormal definitionFundamental
properties
Undirected graphical models
3 6
1 5 7
2 4
u uu u u
u u@@@
���
@@@
@@@
@@@
���
���
For several variables, complex systems of conditional
independencecan for example be described by undirected graphs.Then
a set of variables A is conditionally independent of set B,given
the values of a set of variables C if C separates A from B.
For example in picture above
1⊥⊥{4, 7} | {2, 3}, {1, 2}⊥⊥ 7 | {4, 5, 6}.
Steffen Lauritzen, University of Oxford Graphs and Conditional
Independence
-
ExamplesConditional independence
General independence
Basic ideas and pitfallsFormal definitionFundamental
properties
Directed graphical models
Directed graphs are also natural models for
conditionalindpendence:
3 6
1 5 7
2 4
u uu u u
u u
-
@@@R
����
@@@R
-
@@@R
@@@R
����
����
-
Any node is conditional independent of its non-descendants,
givenits immediate parents. So, for example, in the above picture
wehave
5⊥⊥{1, 4} | {2, 3}, 6⊥⊥{1, 2, 4} | {3, 5}.
Steffen Lauritzen, University of Oxford Graphs and Conditional
Independence
-
ExamplesConditional independence
General independence
Basic ideas and pitfallsFormal definitionFundamental
properties
For random variables X , Y , Z , and W it holds
(C1) If X ⊥⊥Y |Z then Y ⊥⊥X |Z ;(C2) If X ⊥⊥Y |Z and U = g(Y ),
then X ⊥⊥U |Z ;(C3) If X ⊥⊥Y |Z and U = g(Y ), then X ⊥⊥Y | (Z
,U);(C4) If X ⊥⊥Y |Z and X ⊥⊥W | (Y ,Z ), then
X ⊥⊥ (Y ,W ) |Z ;If density w.r.t. product measure f (x , y , z
,w) > 0 also
(C5) If X ⊥⊥Y | (Z ,W ) and X ⊥⊥Z | (Y ,W ) thenX ⊥⊥ (Y ,Z ) |W
.
Steffen Lauritzen, University of Oxford Graphs and Conditional
Independence
-
ExamplesConditional independence
General independence
Basic ideas and pitfallsFormal definitionFundamental
properties
Proof of (C5)
We have
X ⊥⊥Y | (Z ,W ) ⇒ f (x , y , z ,w) = a(x , z ,w)b(y , z
,w).Similarly
X ⊥⊥Z | (Y ,W ) ⇒ f (x , y , z ,w) = g(x , y ,w)h(y , z ,w).If f
(x , y , z ,w) > 0 for all (x , y , z ,w) it thus follows
that
g(x , y ,w) = a(x , z ,w)b(y , z ,w)/h(y , z ,w).
The left-hand side does not depend on z so let z = z0 be
fixed.Then we have
g(x , y ,w) = ã(x ,w)b̃(y ,w).
Insert this into the second expression for f to get
f (x , y , z ,w) = ã(x ,w)b̃(y ,w)h(y , z ,w) = a∗(x ,w)b∗(y ,
z ,w)
which shows X ⊥⊥ (Y ,Z ) |W .Steffen Lauritzen, University of
Oxford Graphs and Conditional Independence
-
ExamplesConditional independence
General independence
Graphoids and independence modelsExamples
Conditional independence can be seen as encoding
abstractirrelevance. With the interpretation: Knowing C , A is
irrelevant forlearning B, (C1)–(C4) translate into:
(I1) If, knowing C , learning A is irrelevant for learning
B,then B is irrelevant for learning A;
(I2) If, knowing C , learning A is irrelevant for learning
B,then A is irrelevant for learning any part D of B;
(I3) If, knowing C , learning A is irrelevant for learning B,it
remains irrelevant having learnt any part D of B;
(I4) If, knowing C , learning A is irrelevant for learning Band,
having also learnt A, D remains irrelevant forlearning B, then both
of A and D are irrelevant forlearning B.
The property analogous to (C5) is slightly more subtle and
notgenerally obvious. Also the symmetry (C1) is a special property
ofprobabilistic conditional independence, rather than of
generalirrelevance.
Steffen Lauritzen, University of Oxford Graphs and Conditional
Independence
-
ExamplesConditional independence
General independence
Graphoids and independence modelsExamples
An independence model ⊥σ is a ternary relation over subsets of
afinite set V . It is semi-graphoid if for all subsets A, B, C ,
D:
(S1) if A⊥σ B |C then B ⊥σ A |C (symmetry);(S2) if A⊥σ (B ∪ D)
|C then A⊥σ B |C and A⊥σ D |C
(decomposition);
(S3) if A⊥σ (B ∪ D) |C then A⊥σ B | (C ∪ D) (weakunion);
(S4) if A⊥σ B |C and A⊥σ D | (B ∪ C ), thenA⊥σ (B ∪ D) |C
(contraction).
It is a graphoid if (S1)–(S4) holds and
(S5) if A⊥σ B | (C ∪ D) and A⊥σ C | (B ∪ D) thenA⊥σ (B ∪ C ) |D
(intersection).
It is compositional if also
(S6) if A⊥σ B |C and A⊥σ D |C then A⊥σ (B ∪ D)
|C(composition).
Steffen Lauritzen, University of Oxford Graphs and Conditional
Independence
-
ExamplesConditional independence
General independence
Graphoids and independence modelsExamples
Separation in undirected graphs
Let G = (V ,E ) be finite and simple undirected graph
(noself-loops, no multiple edges).
For subsets A,B,S of V , let A⊥G B | S denote that S separates
Afrom B in G, i.e. that all paths from A to B intersect S .Fact:
The relation ⊥G on subsets of V is a compositionalgraphoid.
This fact is the reason for choosing the name ‘graphoid’ for
suchindependence model.
Steffen Lauritzen, University of Oxford Graphs and Conditional
Independence
-
ExamplesConditional independence
General independence
Graphoids and independence modelsExamples
Systems of random variables
For a system V of labeled random variables Xv , v ∈ V , we use
theshorthand
A⊥⊥B |C ⇐⇒ XA⊥⊥XB |XC ,
where XA = (Xv , v ∈ A) denotes the variables with labels in
A.The properties (C1)–(C4) imply that ⊥⊥ satisfies thesemi-graphoid
axioms for such a system, and the
An independence model of this kind is said to be
probabilistic.
Probabilistic independence models are often not
compositional.
Steffen Lauritzen, University of Oxford Graphs and Conditional
Independence
ExamplesConditional independenceBasic ideas and pitfallsFormal
definitionFundamental properties
General independenceGraphoids and independence
modelsExamples