[Kim+ ICML2012] Dirichlet Process with Mixed Random Measures : A Nonparametric Topic Model for Labeled Data
Post on 12-Nov-2014
19310 Views
Preview:
DESCRIPTION
Transcript
[Kim+ ICML2012] Dirichlet Process
with Mixed Random Measures : A
Nonparametric Topic Model for
Labeled Data
2012/07/28
Nakatani Shuyo @ Cybozu Labs, Inc
twitter : @shuyo
LDA(Latent Dirichlet Allocation)
[Blei+ 03]
• Unsupervised Topic Model
– Each word has an unobserved topic
• Parametric
– The topic size K is given in advance
via Wikipedia
Labeled LDA [Ramage+ 09]
• Supervised Topic Model
– Each document has an observed label
• Parametric
via [Ramage+ 09]
Generative Process for L-LDA
• 𝜷𝑘~Dir 𝜼
• Λ𝑘𝑑~Bernoulli Φ𝑘
• 𝜽 𝑑 ~Dir 𝜶 𝑑
– where 𝜶 𝑑 = 𝛼𝑘 𝑘 Λ𝑘𝑑=1
• 𝑧𝑖𝑑~Multi 𝜽 𝑑
• 𝑤𝑖𝑑~Multi 𝜷
𝑧𝑖𝑑
via [Ramage+ 09]
restricted to labeled parameters
topics corresponding to observed labels
Pros/Cons of L-LDA
• Pros
– Easy to implement
• Cons
– It is necessary to specify label-topic
correspondence manually
• Its performance depends on the corresponds
※) My implementation is here : https://github.com/shuyo/iir/blob/master/lda/llda.py
via [Ramage+ 09]
DP-MRM [Kim+ 12]
– Dirichlet Process with Mixed Random Measures
• Supervised Topic Model
• Nonparametric
– K is not the topic size, but the label size
𝑥𝑗𝑖 𝐻 𝐺𝑗 𝜃𝑗𝑖 𝐺0𝑘
𝛽 𝛾𝑘
𝑟𝑗 𝜆j
𝜂 𝐾
𝐷
𝑁𝑗
𝛼
Generative Process for DP-MRM
• 𝐻 = Dir 𝛽
• 𝐺0𝑘~DP 𝛾𝑘 , 𝐻
• 𝜆𝑗~Dir 𝒓𝑗𝜂 where 𝒓𝑗 = 𝐼𝑘∈label 𝑗
• 𝐺𝑗~DP 𝛼, 𝜆𝑗𝑘𝐺0𝑘
𝑘∈label 𝑗
• 𝜃𝑗𝑖~𝐺𝑗 , 𝑥𝑗𝑖~𝐹 𝜃𝑗𝑖 = Multi 𝜃𝑗𝑖
Each label has a random measure as topic space
mixed random measures
𝑥𝑗𝑖 𝐻 𝐺𝑗 𝜃𝑗𝑖 𝐺0𝑘
𝛽 𝛾𝑘
𝑟𝑗 𝜆j
𝜂 𝐾
𝐷
𝑁𝑗
𝛼
Stick Breaking Process
• 𝑣𝑙𝑘~Beta 1, 𝛾𝑘 , 𝜋𝑙
𝑘 = 𝑣𝑙𝑘 1 − 𝑣𝑑
𝑘𝑙−1𝑑=0
• 𝜙𝑙𝑘~𝐻, 𝐺0
𝑘 = 𝜋𝑙𝑘𝛿
𝜙𝑙𝑘
∞𝑙=0
• 𝜆𝑗~Dir 𝒓𝑗𝜂 , 𝑤𝑗𝑡~Beta 1, 𝛼 , 𝜋𝑗𝑡 = 𝑤𝑗𝑡 1 − 𝑤𝑗𝑑𝑡−1𝑑=0
• 𝑘𝑗𝑡~Multi 𝜆𝑗 , 𝜓𝑗𝑡~𝐺0𝑘𝑗𝑡 , 𝐺𝑗 = 𝜋𝑗𝑡𝛿𝜓𝑗𝑡
∞𝑡=0
Chinese Restaurant Franchise
• 𝑡𝑗𝑖 : table index of 𝑖-th term in 𝑗-th document
• 𝑘𝑗𝑡, 𝑙𝑗𝑡: dish indexes on 𝑡-th table of 𝑗-th
document
This layer consists on only a single DP G0
on normal HDP
Inference (1)
• Sampling 𝑡
Inference (2)
• Sampling 𝑘 and 𝑙
Experiments
via [Kim+ 12]
• DP-MRM gives label-topic probabilistic
corresponding automatically.
• L-LDA can also predict single labeled document to
assign a common second label to any documents.
via [Kim+ 12]
References
• [Kim+ ICML2012] Dirichlet Process with Mixed
Random Measures : A Nonparametric Topic
Model for Labeled Data
• [Ramage+ EMNLP2009] Labeled LDA : A
supervised topic model for credit attribution in
multi-labeled corpora
• [Blei+ 2003] Latent Dirichlet Allocation
top related