Incorporating Prior Information in Causal Discovery Rodney O'Donnell, Jahangir Alam, Bin Han, Kevin Korb and Ann Nicholson
Jan 15, 2016
Incorporating Prior Information in Causal Discovery
Rodney O'Donnell, Jahangir Alam,
Bin Han, Kevin Korb and Ann Nicholson
Outline• Methods for learning causal models
– Data mining, Elicitation, Hybrid approach
• Algorithms for learning causal models– Constraint based– Metric based (including our CaMML)
• Incorporating priors into CaMML– 5 different types of priors
• Experimental Design
• Experimental Results
Learning Causal Bayesian Networks
Elicitation Data mining
Requires domain knowledge Requires large dataset
Expensive and time-consuming
Sometimes, the algorithms are “stupid”
(no prior knowledge →no common sense)
Partial knowledge may be insufficient
Data only tells part of the story
A hybrid approach
• Combine the domain knowledge and the facts learned from data
• Minimize the expert’s effort in domain knowledge elicitation
Elicitation Data Mining
Causal BN
• Enhance the efficiency of the learning process– Reduce / bias the search space
Objectives
• Generate different prior specification methods
• Comparatively study the influences of priors on the BN structural learning
• Future: apply the methods to the Heart Disease modeling project
Causal learning algorithms
• Constraint based– Pearl & Verma’s algorithm, PC
• Metric based– MML, MDL, BIC, BDe, K2,
K2+MWST,GES,CaMML
• Priors on structure– Optional vs. Required – Hard vs. Soft
Priors on structure
Required Optional Hard Soft
K2 (BNT) yes yes
K2+MWST
(BNT)yes yes
GES
(Tetrad)yes yes
PC (Tetrad ) yes yes
CaMML yes yes yes
CaMML• MML metric based• MML vs. MDL
– MML can be derived from Bayes’ Theorem (Wallace)– MDL is a non-Bayesian method
• Search: MCMC sampling through TOM space– TOM = DAG + total ordering – TOM is finer than DAG
A
B C
Two TOMs: ABC, ACB
A B C
One Tom: ABC
Priors in CaMML: arcs
Experts may provide priors on pairwise relations:1. Directed arcs:
– e.g. {A→B 0.7} (soft)– e.g. {A→D 1.0} (hard)
2. Undirected arcs– E.g. {A─C 0.6} (soft)
3. {A→B 0.7; B→A 0.8; A─C 0.6}– Represented by 2 adjacency matrices
0.7
0.8
A B C
A
B
C
Directed arcs
0.6
0.6
A B C
A
B
C
Undirected arcs
Priors in CaMML: arcs (continued)
• MML cost for each pair
AB: log(0.7) + log(1-0.8)
AC: log(1-0.6)
BC: log( default arc prior)
expert specified network
A0.7
0.8
0.6
B C
One candidate network
A
B C
Priors in CaMML: Tiers
• Expert can provide prior on an additional pairwise relation
• Tier: Temporal ordering of variables
E.g., Tier {A>>C 0.6;B>>C 0.8}
IMML(h)=log(0.6)+log(1-0.8)
A
C
B
One possible TOM
Priors in CaMML: edPrior
• Expert specifies single network, plus a confidence– e.g. EdConf=0.7
• Prior is based on edit distance from this network
A
B
Expert specified network
C
IMML(h)=-2*(log0.7-log(1-0.7))
One candidate network :ED=2
A
B C
Priors in CaMML: KTPrior
• Again, expert specifies single network, plus a confidence– e.g. KTConf = 0.7
• Prior is based on Kendall-Tau Edit distance from this network – KTEditDist = KT + undirected ED
A B C
Expert specified dagTOM: ABC
A
B C
A candidate TOM: ACB
IMML(h)=-3*(log0.7-log(1-0.7))
• B-C order in expert TOM disagrees with candidate TOM • KTEditDist = KT(1) + Undirected ED (2) = 3
Experiment 1: Design• Prior
– weak, strong– correct, incorrect
• Size of dataset– 100,1000,10k and 100k– For each size we randomly generate 30 datasets
• Algorithms: – CaMML– K2 (BNT)– K2+MWST (BNT)– GES (TETRAD)– PC (TETRAD)
• Models: AsiaNet, “Model6”(An artificial model)
Models: AsiaNet and “Model6”
Experimental Design
Priors
Algorithms
Sample Size
Experiment Design: Evaluation
• ED: Difference between Structures
• KL: Difference between distributions
Model6 (1000 samples)
Model6 (10k samples)
AsiaNet (1000 Samples)
Experiment 1: Results
• With default priors: CaMML is comparable to or outperforms other algorithms
• With full tiers: – There is no statistically significant differences
between CaMML and K2– GES is slightly behind, PC performs poorly.
• CaMML is the only method allowing soft priors: – with the prior 0.7, CaMML is comparable to other
algorithms with full tiers– With stronger prior, CaMML performs better
• CaMML performs significantly better with expert’s priors than with uniform priors
Expertiment 2:Is CaMML well calibrated?
• Biased prior– Expert’s confidence may not be consistent
with the expert’s skill
e.g, expert 0.99 sure but wrong about a connection
– Biased hard prior– Soft prior and data will eventually overcome
the bad prior
Is CaMML well calibrated?
• Question: Does CaMML reward well calibrated experts?
• Experimental design– Objective measure: How good is a proposed
structure?: • ED: 0-14
– Subjective measure: Expert’s confidence• 0.5 to 0.9999
– How good is the learned structure?• KL distance
Effect of expert skill and confidence on quality of learned model
Better ← Expert Skill → Worse
Overconfidence penalized
Justified confidence rewarded
Unconfident expert
Experiment 2: Results
• CaMML improves the elicited structure and approaches the true structure
• CaMML improves when the expert confidence matches with the expert skill
Conclusions
• CaMML is comparable to other algorithms when given equivalent prior knowledge
• CaMML can incorporate more flexible prior knowledge
• CaMML’s results improve when expert is skillful or well calibrated
Thanks