Graphical Multiagent Models Quang Duong Computer Science and Engineering Chair: Michael P. Wellman 1
Dec 15, 2015
2
Example: Election In The City Of AA
Political discussion
Vote
May, political analyst
• Phone surveys• Demographic information• Party registration• …
3
Modeling Objectives
Construct a model that takes into account people (agent) interactions (graph edges) in:
– Representing joint probability of all vote outcomes*– Computing marginal and conditional probabilities
Vote Republican or
Democrat?
4
Modeling Objectives (cont.)
Generate predictions:– Individual actions, dynamic behavior induced by individual
decisions– Detailed or aggregate
5
More Applications Of Modeling Multiagent Behavior
Financial Institutions
Computer Network/ Internet
Social Network
6
Challenges: Uncertaintyfrom the system modeler’s perspective
1a. Agent choiceVote for personal favorite or conform with
others?
1b. CorrelationWill the historic district of AA unanimously pick
one candidate to support?
1c. InterdependenceMay does not know all friendship relations in AA
7
Challenges: Complexity
2a. Representation and inferenceNumber of all action configurations (all vote
outcomes) is exponential in the number of agents (people).
2b. Historical informationPeople may change their minds about whom to
vote for after discussions.
8
Existing Approaches That This Work Builds On
Game-theory Approach:• Assume game structure/perfect rationality
Statistical Modeling Approach:• Aggregate statistical measures/ make simplifying assumptions
9
Approach Outline
Graphical Multiagent Models (GMMs) are probabilistic graphical models designed to
• Facilitate expressions of different knowledge sources about agent reasoning
• Capture correlated behaviors
while
• Exploiting dependence structure
uncertainty
complexity
10
Roadmap
(Ch. 3) GMM(static)
(Ch. 4) History-
Dependent GMM
(Ch. 6) Application: Information
Diffusion
(Ch. 2) Background
(Ch. 5) Learning
Dependence Graph Structure
(Ch. 2) Background
11
Multiagent Systems
• n agents {1,…,i,…,n}
• Agent i chooses action ai, joint action (action configuration) of the system: a = (a1,…, an)
• In dynamic settings: – time period t, time horizon T.– history Ht of history horizon h, Ht = (at-h,…,at-1)
12
Game Theory
Each player (agent i) chooses a strategy (action ai).
Strategy profile (joint action a) of all players.
Payoff function: ui(ai,a-i)
Player i‘s regret εi(a): maximum gain if player i chooses strategy ai
’, instead of strategy ai, given than everyone else fixes their strategies.
a* is a Nash equilibrium (NE) if for every player i, regret εi(a) = 0.
13
Graphical Representations of Multiagent Systems
1. Graphical Game Models [Kearns et al. ‘01]
An agent’s payoff depends on strategy chosen by itself and its neighbors Ji
Payoff/utility: ui(ai,aJi)
Similar approaches: Multiagent influence diagrams (MAIDs) [Koller & Milch
’03]Networks of Influence Diagrams [Gal & Pfeffer ’08]Action-graph games [Jiang et al ‘11].
14
Graphical Representations (cont.)
2. Probabilistic graphical models
Markov random field (static) [Kindermann & Laurie ’80, KinKoller & Friedman ‘09]
Dynamic Bayesian Networks [Kanazawa & Dean ’89, Ghahramani ’98]
15
Probabilistic Graphical Models
This Work
demonstrate and examine
the benefits of applying probabilistic graphical models to the problem of modeling multiagent behavior
in scenarios with different sets of assumptions and information available to the system modeler.
Building on
Game Models
incorporating
16
Roadmap
(Ch. 3) GMM(static)
(Ch. 4) History-
Dependent GMM
(Ch. 5) Learning
Dependence Graph Structure
(Ch. 6) Application: Information
Diffusion
(Ch. 2) Background
1. Overview2. Examples3. Knowledge Combination4. Empirical Study
17
Graphical Multiagent Models (GMMs)
[Duong, Wellman & Singh ‘08]
• Nodes: agents. Edges: dependencies among agent actions
• Dependence neighborhood Ni
113
6
2 7
5
4
18
GMMs
Pr(a) ∝ Πi πi(aNi)
Joint probabilitydistribution ofsystem’s actions
potential of neighborhood’s joint actions
Factor joint probability distribution into neighborhood potentials.
(Markov random field for graphical games [Daskalakis & Papadimitriou ’06])
19
Example GMMs
• Markov Random Field for computing pure strategy Nash equilibrium
• Markov Random Field for computing correlated equilibrium
• Information diffusion GMMs [Ch. 6]
• Regret GMMs [Ch. 3]
20
Examples: Regret potential
Assume a graphical gameRegret ε(aNi)
πi(aNi) = exp(-λ εi(aNi))
Illustration:
Assume: prefers Republican to Democrat (fixing others’ choices)
Near zero λ: picks randomlyLarger λ: more likely to pick Republican
21
Flexibility: Knowledge Combination• Assume known graph structures, given GMMs G1 and G2 that
represent 2 different knowledge sources
Final GMMfinalG
GMM 2GMM 1 Knowledge Combination
Regret GMMreG
Heuristic Rule-based
GMMhG
1. Direct update2. Opinion pool3. Mixing data
23
Empirical Study
• Combining knowledge sources in one GMM improves predictions
• Combined models fail to improve on input models when input does not capture any underlying behavior
ratio > 1: combined model performs better than input model
Mixing data GMM vs. regret GMM
Mixing data GMM vs. heuristic GMM
26
Summary Of Contributions (Ch. 3)
(I.A) GMMs accommodate expressions of different knowledge sources
(I.B) This flexibility allows the combination of models for improved predictions
27
Roadmap
(Ch. 3) GMM(static)
(Ch. 4) History-
Dependent GMM
(Ch. 6) Application: Information
Diffusion
(Ch. 2) Background
1. Consensus Dynamics2. Description3. Joint vs. individual behavior4. Empirical study
(Ch. 5) Learning
Dependence Graph Structure
28
Example: Consensus Dynamics
[Kearns et al. ’09] abstracted version of the AA mayor election example
2
3
4
5
1
6
Agent Blue consensus
Red consensus
neither
1 1.0 0.5 0
2 0.5 1.0 0
Observation graphAgent 1’s perspective
Examine the ability to make collective decisions with limited communication and observation
Modeling Multiagent Behavior In Consensus Dynamics Scenario
time
Time series action data + observation graph
1. Predict detailed actions 2. Predict aggregate measures
or
31
History-Dependent Graphical Multiagent Models (hGMMs)
[Duong, Wellman, Singh & Vorobeychik ’10]
We condition actions on abstracted history Ht
Note: dependence graphs can be different from observation graphs.
1 1 1
t-1 t t+1
32
hGMMs
(Undirected) within-time edges: dependencies between agent actions in the same time period, and define dependence neighborhood Ni for each agent i.
A GMM at every time t
1 1 1
t-1 t t+1
33
hGMMs
(Directed) across-time edges: dependencies of agent i’s action on some abstraction of prior actions by agents in i’s conditioning set Γi
Example: frequency function.
1 1 1
t-1 t+1
34
hGMMs
Pr(at | H) ∝ Πi πi(atNi | Ht
Γi)
Joint probabilitydistribution ofsystem’s actions at time t
potential of neighborhood’s joint actions at t
history of the conditioning set
35
Challenge: Dependence
• Conditional independence
• Dependence induced by history abstraction/summarization (*)
1
2
1
2
1
2
t-2 t-1 t
1
2
1
2
1
2
t-2 t-1 t
36
Individual vs. Joint Behavior Models
Given complete history, autonomous agents’ behaviors are conditionally independent Individual behavior models:
πi(ati | Ht
Γi,complete)
Joint behavior models allow specifying any action dependence within one’s within-time neighborhood, given some (abstracted) history
πi(atNi | Ht
Γi,abstracted)
37
Empirical Study: Summary
Evaluation: compares joint behavior and individual behavior models by likelihood of testing data (time-series votes)
* Observation graph defines both dependence neighborhoods N and conditioning sets Γ
1. Joint behavior outperform individual behavior models for shorter history lengths, which induce more action dependence.
2. Approximation does not deteriorate performance
38
Summary Of Contributions (Ch. 4)
(II.A) hGMMs support inference about system dynamics
(II.B) hGMMs allow the specification of action dependence emerging from history abstraction
39
Roadmap
(Ch. 3) GMM(static)
(Ch. 4) History-
Dependent GMM
(Ch. 6) Application: Information
Diffusion
(Ch. 2) Background
1. Learning Graphical Game Models
(Ch. 5) Learning
Dependence Graph structure
2. Learning hGMMs
42
Learning History-Dependent Graphical Multiagent Models
ObjectiveGiven action data + observation graph, build a model that predicts:
– Detailed actions in next period– Aggregate measures of actions in the more distant future
Challenge: Learn dependence graph– (Within-time) Dependence graph ≠ observation graph – Complexity of the dependence graph
43
Consensus Dynamics Joint Behavior Model
Extended Joint Behavior hGMM (eJCM)
πi(aNi | HtΓi) = ri(aNi) f(ai , Ht
Γi)γ Ι(ai , Ht
i)β
1. ri(aNi) = reward for action ai, discounted by the number of dissenting neighbors in Ni
2. frequency of ai chosen previously by agents in the conditioning set Γi 3. inertia proportional to how long i has maintained its most recent action
1 2 3
44
Consensus Dynamics Individual Behavior Models
1. Extended Individual Behavior hGMM (eICM): similar to eJCM but assumes that Ni contains i only
πi(ai | HtΓi) = Pr(ai | Ht
Γi) ∝ ri(ai) f(ai , HtΓi)
γ Ι(ai , Hti)β
2. Proportional Response Model (PRM): only incorporates the most recent time period [Kearns et al., ‘09]:Pr(ai | Ht
Γi) ∝ ri(ai) f(ai , HtΓi)
3. Sticky Proportional Response Model (sPRM)
45
Learning hGMMS
Input:• <action observations (time series)>
• observation graph
Search space:1.Model parameters γ, β
2.Within-time edges
Output:
hGMM
Objective: likelihood of dataConstraint: max node degree
46
Greedy Learning
Initialize the graph with no edges
Repeat:Add edges that generate the biggest increase (>0) in the training data’s likelihood
Until no edge can be added without violating the maximum node degree constraint
47
Empirical Study: Learning from human-subject data
Use asynchronous human-subject dataVary the following environment parameters: • Discretization intervals, delta (0.5 and 1.5 seconds)• History lengths, h• Graph structures/payoff functions: coER_2, coPA_2, & power22
(strongly connected minority)
Goal: evaluate eJCM, eICM, PRM, and sPRM using 2 metrics• Negative likelihood of agents’ actions• Convergence rates/outcomes
48
Predicting Dynamic Behavior
eJCMs and eICMs outperform the existing PRMs/sPRMs
eJCMs predict actions in the next time period noticeably more accurately than PRMs and sPRMs, and (statistically significantly) more accurate than eICMs
49
Predicting Consensus Outcomes
eJCMs have comparable prediction performance with other models in 2 settings: coER_2 and coPA_2.In power22, eJCM predict consensus probability and colors much more accurately.
50
Graph Analysis
In learned graphs, intra edges >> inter edges.
In power22, a large majority of edges are intra red identify the presence of a strongly connected red minority
51
Summary Of Contributions (Ch. 5.2)
(II.B) [revisit] This study highlights the importance of joint behavior modeling
(III.C) It is feasible to learn both dependence graph structure and model parameters
(III.D) Learned dependence graphs can be substantially different from observation graphs
52
Modeling Multiagent Systems: Step By Step
Given as input
Learn from data
Intuition, background information
Approximation
Dependence graph structure
Potential function
GMMhGMM
Observation graph structure
53
Roadmap
(Ch. 3) GMM(static)
(Ch. 4) History-
Dependent GMM
(Ch. 6) Application: Information
Diffusion
(Ch. 2) Background
(Ch. 5) Learning
Dependence Graph structure
1. Definition2. Joint behavior modeling 3. Learning missing edges4. Experiments
54
Networks with Unobserved Links
• Links facilitate how information diffuses from one node to another
• Real-world nodes have links unobserved by third parties
Truenetwork G*
ObservedNetwork G
55
Problem
Given: a network (with missing links) and snapshots of the network states over time.
Objective: model information diffusions on this network
[Duong, Wellman & Singh ‘11]
1. Network G2. Diffusion traces (on G*)
56
Approach 1: Structure Learning
Recover missing edges
• Learn network G’• Learn parameters of an individual behavior model built on G’• Learning algorithms: NetInf [Gomez-Rodriguez et al. ’10] and
MaxInf
57
Approach 2: Potential Learning
Construct an hGMM on G without recovering missing links
• hGMMs allow capturing state correlations between neighbors who appear disconnected in the input network
• Theoretical evidence [6.3.2]• Empirical illustrations: hGMMs outperform individual behavior
models on learned graph– random graph with sufficient training data– preferential attachment graph (varying amounts of data)
58
Summary of Contributions (Ch. 6)
(II.C) Joint behavior hGMM, can capture state dependence caused by missing edges
59
Conclusions
1. The machinery of probabilistic graphical models helps to improve modeling in multiagent systems by: • allowing the representation and combination of different
knowledge sources of agent reasoning • relaxing assumptions about action dependence (which may be a
result of history abstraction or missing edges)
2. One can learn from action data both: (i) model parameters, and (ii) dependence graph structure, which can be different from interaction/observation graph structure
60
Conclusions (cont.)
3. The GMM framework contributes to the integration of:
• strategic behavior modeling techniques from AI and economics• probabilistic models from statistics that can efficiently extract
behavior patterns from massive amount of data
for the goal of understanding fast-changing and complex multiagent systems.
61
Summary
• Graphical multiagent models: flexibility to represent different knowledge sources and combine them [UAI ’08]
• History-dependent GMM: capture dependence in dynamic settings [AAMAS ’10, AAMAS ’12]
• Learning graphical game models [AAAI ’09]• Learning hGMM dependence graph, distinguishing
observation/interactions graphs and probabilistic dependence graphs [AAMAS ‘12]
• Modeling information diffusion in networks with unobserved links [SocialCom ‘11]
62
Acknowledgments• Advisor: Professor Michael P. Wellman• Committee members: Prof. Satinder Singh Baveja, Prof. Edmund H.
Durfee, and Asst. Prof. Long Nguyen• Research collaborators: Yevgeniy Vorobeychik (Sandia Labs), Michael
Kearns (U Penn), Gregory Frazier (Apogee Research), David Pennock and others (Yahoo/Microsoft Research)
• Undergraduate advisor: David Parkes.
• Family• Friends• CSE staff