Visualizing Dynamics: from t-SNE to SEMI-MDPs › icmlviz2016 › assets › papers › 10.pdfusing a manual inspection of a t-Distributed Stochastic Neighbor Embedding (t-SNE) map

Visualizing Dynamics: from t-SNE to SEMI-MDPs

Nir Ben Zrihem* [email protected] Zahavy* [email protected] Mannor [email protected]

Electrical Engineering Department, The Technion - Israel Institute of Technology, Haifa 32000, Israel

AbstractDeep Reinforcement Learning (DRL) is a trend-ing field of research, showing great promisein many challenging problems such as playingAtari, solving Go and controlling robots. WhileDRL agents perform well in practice we arestill missing the tools to analayze their perfor-mance and visualize the temporal abstractionsthat they learn. In this paper, we present a novelmethod that automatically discovers an internalSemi Markov Decision Process (SMDP) modelin the Deep Q Network’s (DQN) learned rep-resentation. We suggest a novel visualizationmethod that represents the SMDP model by a di-rected graph and visualize it above a t-SNE map.We show how can we interpret the agent’s policyand give evidence for the hierarchical state ag-gregation that DQNs are learning automatically.Our algorithm is fully automatic, does not requireany domain specific knowledge and is evaluatedby a novel likelihood based evaluation criteria.

1. IntroductionDQN is an off-policy learning algorithm that uses a Convo-lutional Neural Network (CNN) (Krizhevsky et al., 2012)to represent the action-value function and showed supe-rior performance on a wide range of problems (Mnih et al.,2015). The success of DQN, and that of Deep Neural Net-work (DNN) in general, is explained by its ability to learngood representations of the data automatically. Unfortu-nately, its high representation power is also making it com-plex to train and hampers its wide use.

Visualization can play an essential role in understandingDNNs. Current methods mainly focus on understandingthe spatial structure of the data. For example, Zeiler &Fergus (2014) search for training examples that cause high

* These authors have contributed equally

neural activation at specific neurons, Erhan et al. (2009)created training examples that maximizes the neural activ-ity of a specific neuron and Yosinski et al. (2014) inter-preted each layer as a group. However, none of these meth-ods analyzed the temporal structure of the data.

Good temporal representation of the data can speed upthe prerformence of Reinforcement Learning (RL) algo-rithms (Dietterich, 2000; Dean & Lin, 1995; Parr, 1998;Hauskrecht et al., 1998), and indeed there is a growinginterest in developing hierarchical DRL algorithms. Forexample, Tessler et al. (2016) pre-trained skill networksusing DQNs and developed a Hierarchical DRL Network(H-DRLN). Their architecture learned to control betweenoptions operating at different temporal scales and demon-strated superior performance over the vanilla DQN in solv-ing tasks at Minecraft. Kulkarni et al. (2016) took a dif-ferent approach, they manually pre-defined sub-goals fora given task and developed a hierarchical DQN (h-DQN)that is operating at different time scales. This architecturemanaged to learn how to solve both the sub-goals and theoriginal task and outperformed the Vanilla DQN in the thechallenging ATARI game ’Montezuma’s Revenge’. Boththese methods used prior knowledge about the hierarchy ofa task in order to solve it. However it is still unclear how toautomatically discover the hierarchy in a specific domaina-priori.

Interpretability of DQN policies is an urging issue that hasmany important applications. For example, it may help todistil a cumbersome model into a simple one (Rusu et al.,2015) and will increase the human confidence in the per-formance of DRL agents. By understanding what the agenthas learned we can also decide where to grant it controland where to take over. Finally, we can improve learningalgorithms by finding their weaknesses.

The internal model principle (Francis & Wonham, 1975)states that every good solution to a control problem mustbe a model of the problem it solves (”Every good key mustbe a model of the lock it opens”). This line of thoughthas an interesting application to control theory and biology(Yi et al., 2000). It suggests that to do the best job of reg-


ulating some system, a control apparatus should include amodel of that system. Sontag (2003) formulated these ideasmathematically for linear and non linear control systems,claiming that if a system Σ is solving a control task underreasonable technical assumptions, then Σ must necessarilycontain a subsystem which is capable of predicting the dy-namics of the system. In this work we follow the same lineof thought. We claim that DQNs are learning an underly-ing Semi Markov Decision Process (SMDP) of a problem,without implicitly being asked.

Zahavy et al. (2016) showed that by using hand-crafted fea-tures, they can interpret the policies learned by DQN agentsusing a manual inspection of a t-Distributed StochasticNeighbor Embedding (t-SNE) map (Van der Maaten &Hinton, 2008). They also revealed that DQNs are automati-cally learning temporal representations such as hierarchicalstate aggregation and temporal abstractions. On the otherhand, they use a manual reasoning of a t-SNE map, a te-dious process that requires careful inspection as well as anexperienced eye.

However, we suggest a method that is fully automatic. In-stead of manually designing features, we use clustering al-gorithms to reveal the underlying structure of the t-SNEmap. But instead of naively applying classical methods,we designed novel time-aware clustering algorithms thattake into account the temporal structure of the data. Us-ing this approach we are able to automatically reveal theunderlying dynamics and rediscover the temporal abstrac-tions showed in (Zahavy et al., 2016). Moreover, we showthat our method reveals an underlying SMDP model andconfront this hypothesis qualitatively, by designing a novelvisualization tool, and quantitatively, by developing likeli-hood criteria which we later test empirically.

The result is an SMDP model that gives a simple expla-nation on how the agent solves the task - by decomposingit automatically into a set of sub-problems and learning aspecific skill at each. Thus, we claim that we have foundan internal model in DQN’s representation, which can beused for automatic sub-goal detection in future work.

2. BackgroundWe briefly review the standard reinforcement learningframework of discrete-time, finite Markov decision pro-cesses (MDPs). In this framework, the goal of an RLagent is to maximize its expected return by learning a pol-icy π : S → ∆A which is a mapping from states s ∈ Sto a probability distribution over some action space A. Attime t the agent observes a state st ∈ S, selects an actionat ∈ A, and receives a reward rt. Following the agentsaction choice, it transitions to the next state st+1 ∈ S.We consider infinite horizon problems where the cumula-

tive return at time t is given by Rt =∑∞t′=t γ

t′−trt, andγ ∈ [0, 1] is the discount factor. The action-value functionQπ(s, a) = E[Rt|st = s, at = a, π] represents the ex-pected return after observing state s, taking action a afterwhich following policy π. The optimal action-value func-tion obeys a fundamental recursion known as the optimalBellman equation,

Q∗(st, at) = E[rt + γmax

a′Q∗(st+1, a

′)]

Deep Q Networks: The DQN algorithm (Mnih et al.,2015) approximates the optimal Q function using a CNN,by optimizing the network weights such that the expectedTD error of the optimal Bellman equation is minimized:

Est,at,rt,st+1‖Qθ (st, at)− yt‖22 (1)

where

yt =

rt if st+1 is terminal

rt + γmaxa’Qθtarget

(st+1, a

′)

otherwise .

Notice that this is an offline learning algorithm, meaningthat the tuples {st,at, rt, st+1, γ} are collected from theagents experience and are stored in the Experience Re-play (ER) (Lin, 1993). The ER is a buffer that stores theagents experiences at each time-step t, for the purpose ofultimately training the DQN parameters to minimize theloss function. When we apply mini-batch training updates,we sample tuples of experience at random from the pool ofstored samples in the ER. The DQN maintains two sepa-rate Q-networks. The current Q-network with parametersθ, and the target Q-network with parameters θtarget. Theparameters θtarget are set to θ every fixed number of iter-ations. In order to capture the game dynamics, the DQNrepresents the state by a sequence of image frames.

Skills, Options, Macro-actions (Sutton et al., 1999), aretemporally extended control structure, denoted by σ. Askill is defined by a triple σ =< I, π, β > where I isthe set of states where the skill can be initiated, π is theintra-skill policy, which determines how the skill behaveswhen encountering states, and β is the set of terminationprobabilities determining when a skill will stop executing.β is typically either a function of state s or time t. AnyMDP with a fixed set of skills is a Semi-Markov DecisionProcess (SMDP). Planning with skills can be performedby predicting for each state in the skill initiation set I , thestate in which the skill will terminate and the total rewardreceived along the way. More formally, an SMDP can bedefined by a five-tuple < S,Σ, P,R, γ > where S is a set


of states, Σ is a set of skills, and P is the transition proba-bility kernel.

Rσs = E[rσs ] = E[rt+1 +γrt+2 + · · ·+γk−1rt+k|st = s, σ](2)

represents the expected discounted sum of rewards receivedduring the execution of a skill σ initialized from a states, and γ ∈ [0, 1] is the discount factor. The Skill Pol-icy µ : S → ∆Σ is a mapping from states to a proba-bility distribution over skills Σ. The action-value functionQ : S × Σ → R represents the long-term value of tak-ing a skill σ ∈ Σ from a state s ∈ S and thereafter al-ways selecting skills according to policy µ and is definedby Q(s, σ) = E[

∑∞t=0 γ

tRt|(s, σ), µ]. We denote the skilltransition probability as

Pσs,s′ =

∞∑j=0

γjPr[k = j, st+j = s′|st = s, σ]

Under these definitions the optimal skill value function isgiven by the following equation (Stolle & Precup) as:

Q∗Σ(s, σ) = E[Rσs + γkmaxσ′∈Σ

Q∗Σ(s′, σ′)] . (3)

3. MethodologyFor each domain:

1. Learn : Train a DQN agent.

2. Evaluate : Run the agent, record visited states, neuralactivations and Q-values.

3. Reduce : Apply t-SNE on the neural activation to ob-tain a low dimensional representation.

4. Cluster : Apply clustering on the data.

5. Model : Fit an SMDP model. Estimate the transitionprobabilities and reward values.

6. Visualize : Visualize the SMDP on top of the t-SNEmap.

The rest of the paper is organized as follows: Section 4 ex-plains how we create the t-SNE maps from raw pixel statesusing the DQN algorithm. In section 5 we present the clus-tering methods that we developed and the quantitative eval-uation criteria. Section 6 explains our visualization methodand Section 7 presents examples for Atari2600 games usingour method. Finally Section 8 summarizes our work.

4. From DQN to t-SNEWe train DQN agents using the Vanilla DQN algorithm(Mnih et al., 2015). When training is done, we evaluate

the agent at multiple episodes, using an ε-greedy policy.We record all visited states and their neural activations, aswell as the Q-values and other manually extracted features.We keep the states in their original visitation order in orderto maintain temporal relations. Since the neural activationsare of high order we apply t-SNE dimensionality reductionso we are able to visualize it.t-SNE is a visualization technique for high dimensionaldata that assigns each data point a location in a two or three-dimensional map. It has been proven to outperform lin-ear dimensionality reduction methods and non-linear em-bedding methods such as ISOMAP (Tenenbaum et al.,2000) in several research fields including machine-learningbenchmark datasets and hyper-spectral remote sensing data(Lunga et al., 2014). At its core, The t-SNE algorithm de-fines two similarity measures from the euclidean distancesbetween points. The first measure: pi,j is defined over pairsof points xi, xj in the high dimension space using a Gaus-sian distribution:

pj|i =exp(−‖xi − xj‖2/2σ2

i )∑k 6=i exp(−‖xi − xk‖

2/2σ2

i )

pi,j =pi|j + pj|i

2

(4)

The second measure qi,j is defined over pairs of pointsyi, yj in the desired low-dimension space using a Student-tdistribution:

qi,j =(1 + ‖yi − yj‖2)−1∑k 6=i (1 + ‖yi − yj‖2)−1

(5)

The algorithm defines a cost function between the mea-sures:

Cost = KL(P ||Q) =∑i,j

pi,j logpi,jqi,j

(6)

And minimize it by following a gradient descent approach:

δCost

δyi= 4

∑j

(pij − qij)(yi − yj)(1 + ‖yi − yj‖2)−1

Y t ← Y t−1 + ηδCost

δY+ α(t)(Y t−1 − Y t−2)

(7)

The technique is relatively easy to optimize, and reducesthe tendency to crowd points together in the center of themap by using the heavy tailed Student-t distribution (Equa-tion 5) in the low dimensional space. It is known to be par-ticularly good at creating a single map that reveals structureat many different scales, which is particularly important forhigh-dimensional data that lie on several different, but re-lated, low-dimensional manifolds. Therefore we can vi-sualize the different sub-manifolds learned by the networkand interpret their meaning.


5. From t-SNE to SMDPIn this section we explain the clustering methods which wedeveloped for this work and show how to create an SMDPmodel. We define an SMDP model over the set of t-SNEpoints using a vector of cluster labels C, and a transitionprobability matrix P where Pi,j indicates the empiricalprobability of moving from cluster i to j. We define theentropy of a model by: e = −

∑i{|Ci| ·

∑j Pi,j logPi,j},

i.e., the average entropy over transition probability fromeach cluster weighted by its size.

We note that threw this entire paper, by an SMDP modelwe only refer to the induced Markov Reward Process ofthe DQN policy. Recall that the DQN agent is learninga deterministic policy, therefore, in deterministic environ-ments (e.g., the Atari2600 emulator), the underlying SMDPshould in fact be deterministic and an entropy minimizer.

The data that we collect from the DQN agent is highly cor-related since it was generated from an MDP. However, stan-dard clustering algorithms assume the data is drawn froman i.i.d distribution, and therefore result with clusters thatoverlook the temporal information. This results with high-entropy SMDP models that are too complicated to analayzeand are not consistent with the data. For this aim, we pro-pose 3 clustering methods that incorporate the data tempo-ral information. We present two variants of K-means and avariant of hierarchical clustering.

5.1. K-means based methods

K-means (MacQueen et al., 1967) is a method commonlyused to automatically partition a data set into k groups.Given a set of observations (x1, x2, · · · , xn), where eachobservation is a d-dimensional real vector, K-means clus-tering aims to partition the n observations into k(≤ n) setsC = (C1, C2, · · · , Ck) so as to minimize the within-clustersum of squares:

arg minC

k∑i=1

∑x∈Ci

‖x− µi‖2 (8)

where µi is the mean of points in Ci. It proceeds by se-lecting k initial cluster centers and then iteratively refiningthem as follows:

1. Assignment step, each observation xi is assigned toits closest cluster center:

C(t)i =

{xp :

∥∥xp − µ(t)i

∥∥2 ≤∥∥xp − µ(t)

j

∥∥2

∀j, 1 ≤ j ≤ k}.

(9)

2. Update step, each cluster center µj is updated to be

the mean of its constituent instances:

µ(t+1)i =

1

|C(t)i |

∑xj∈C(t)

i

xj .

The algorithm converges when there is no further changein the assignment of instances to clusters.

Spatio-Temporal Cluster Assignment.Our first K-means variant modifies the assignment step inthe vanilla K-means algorithm to include temporal infor-mation. Here, each observation xp is a t-SNE point withtime index p along the trajectory. The K-means algorithm,presented with the coordinates of the points, takes care ofclustering points with spatial proximity. In order to encour-age temporal coherency, we modify the assignment step inthe following way:

C(t)i =

{xp :

∥∥Xp−w:p+w − µ(t)i

∥∥2 ≤∥∥Xp−w:p+w − µ(t)

j

∥∥2,

∀j, 1 ≤ j ≤ k}

(10)

Where Xp−w:p+w is the set of 2w t-SNE points before andafter xp along the trajectory. In this way, a point xp is as-signed to a cluster µj , if its neighbours along the trajectoryare also close to µj .

Entropy Regularization Cluster Assignment.In order to create simpler models, we suggest to add an en-tropy regularization term for the K-mean assignment step:

C(t)i =

{xp :

∥∥xp − µ(t)i

∥∥2+ d · et−1

xp→i ≤∥∥xp − µ(t)j

∥∥2+ d · et−1

xp→j ,

∀j, 1 ≤ j ≤ k}.

(11)

Where d is the penalty weight, and et−1xp→i indicates the en-

tropy gain of changing xp assignment to cluster i in theSMDP obtained at iteration t − 1. This is equivalent tominimizing an energy function which is the sum of the K-means objective function (Equation 8) and an entropy term.

5.2. Agglomerative clustering approach

Our third method for creating an SMDP model is a vari-ant of hierarchical clustering. Agglomerative clustering isa bottom-up hierarchical approach. It begin when each ob-servation forms a unique cluster. Then, pairs of clustersare merged together so as to minimize some linkage cri-teria. Most popular here are the single-linkage criteria:c(A,B) = min

a,b{|xa − xb‖ : a ∈ A, b ∈ B}, and the

complete-linkage criteria: c(A,B) = maxa,b{|xa − xb‖ :

a ∈ A, b ∈ B}. In order to encourage temporal coherencyin cluster assignments we form a new linkage criteria based


on ward’s (Ward, 1963) criteria:

c(A,B) = (1−λ) ·mean{‖xa − xb‖ : a ∈ A, b ∈ B}+λ · e{A,B}→AB

(12)

where e{A,B}→AB measures the difference between the en-tropy of the corresponding SMDP before and after mergingclusters A,B.

5.3. Evaluation criteria

We follow the analysis of (Hallak et al., 2013) and definecriteria to measure the fitness of a model empirically. Wedefine the Value Mean Square Error(VMSE) as the nor-malized distance between two value estimations:

VMSE =‖vDQN − vSMDP ‖

‖vDQN‖.

The SMDP value is given by

VSMDP = (I + γkP )−1r (13)

and the DQN value is evaluated by averaging the DQNvalue estimates over all MDP states in a given cluster(SMDP state): vDQN (cj) = 1

|Cj |∑i:si∈cj v

DQN (si) . Fi-nally, the greedy policy with respect to the SMDP value isgiven by:

πgreedy(ci) = argmaxj{Rσi,j + γkσi,j vSMDP (cj)} (14)

The Minimum Description Length (MDL; (Rissanen,1978)) principle is a formalization of the celebrated Oc-cams Razor. It copes with the over-fitting problem for thepurpose of model selection. According to this principle, thebest hypothesis for a given data set is the one that leads tothe best compression of the data. Here, the goal is to finda model that explains the data well, but is also simple interms of the number of parameters. In our work we followa similar logic and look for a model that best fits the databut is still simple.Instead of considering ”simple” in terms of the numberof parameters, we measure the simplicity of the spatio-temporal state aggregation. For spatial simplicity we definethe Inertia: I =

∑ni=0 minµj∈C(||xj − µi||2) which mea-

sures the variance of MDP states inside a cluster (AMDPstate). For temporal simplicity we define the entropy:e = −

∑i{|Ci| ·

∑j Pi,j logPi,j} , and the Intensity Fac-

tor which measures the fraction of in/out cluster transitions:F =

∑j

Pjj∑i Pji

.

6. Visualization: fusing SMDP with t-SNEIn Section 4 we explained how to create a t-SNE map fromDQN’s neural activations and in Section 5 we showed how

to automatically design an SMDP model using temporal-ware clustering methods. In this section we explain how tofuse the SMDP model with the t-SNE map for a clear visu-alization of the dynamics.In our approach, an SMDP is represented by a directedgraph. Each SMDP state is represented by a node in thegraph and corresponds to a cluster of t-SNE points (gamestates). In addition, the transition probabilities between theSMDP states are represented by weighted edges betweenthe graph nodes. We draw the graph on top of the t-SNEmap such that it reveals the underlying dynamics. Choos-ing a good layout mechanism to represent a graph is a hardtask when dealing with high dimensional data (Tang et al.,2016). We consider different layout algorithms for the po-sition of the nodes, such as the spring layout that positionnodes by using the Fruchterman-Reingold force-directedalgorithm and the spectral layout that uses the eigenvectorsof the graph Laplacian (Hagberg et al., 2008). However, wefound out that simply positioning each node at the averagecoordinates of each t-SNE cluster gives a more clear visual-ization. The intuition behind it is that the t-SNE algorithmwas planned to solve the crowding problem and thereforeoutputs clusters that are well separated from each other.Finally, each node in the graph is represented by its cen-troid. For example, if each state is an image, then a nodeis represented using the mean or median images. Anotherapproach is to represent a node using its most significantfeatures. In this approach each node is annotated with afew features that are considered most distinct, e.g., by thefeature with lowest variance in the cluster.

7. ExperimentsExperimental set-up. We evaluated our method on twoAtari2600 games, Breakout and Pacman. For each gamewe collected 120k game states. We apply the t-SNE al-gorithm directly on the collected neural activations of thelast hidden layer, similar to Mnih et al. (2015). The in-put X ∈ R120k×512 consists of 120k game states with 512features each (the size of our DQN last layer). Since thisdata is relatively large, we pre-processed it using Princi-pal Component Analysis to dimensionality of 50 and usedthe Barnes Hut t-SNE approximation (Van Der Maaten,2014). All experiments were performed with 3000 itera-tions perplexity of 30. The input X ∈ R120k×3 to theclustering algorithm consists of 120k game states with 3features each (two t-SNE coordinates and the Value esti-mate). We applied the Spatio-Temporal Cluster Assign-ment with k=20 clusters and w=2 temporal window size(Equation 10). We run the algorithm for 160 iterations andchoose the best SMDP in terms of minimum entropy (wewill consider other measures in future work). Finally we vi-sualize the SMDP using the visualization method explainedin Section 6.


Figure 1. SMDP visualization for Breakout.

Simplicity. Looking at the resulted SMDPs it is interestingto note that the transition probability matrix is very sparse,i.e., the transition probability from each state is not zeroonly for a small subset of the states, thus, indicating that ourcluster are located in time. Inspecting the mean image ofeach cluster we can see that the clusters are also highly spa-tially located, meaning that the states in each cluster sharesimilar game position.

Figure 1 shows the SMDP for Breakout. The mean imageof each cluster shows us the ball location and direction (inred), thus characterizes the game situation in each cluster.We also observe that states with low entropy follow a welldefined skill policy. For example cluster 10 has one maintransition ans show a well defined skill of carving the lefttunnel (see the mean image). In contrast, clusters 6 and16 has transitions to more clusters (and therefore higherentropy) and a much less defined skill policy (presented byits relatively confusing mean state).

Figure 2 shows the SMDP for Pacman. The mean imageof each cluster shows us the agent’s location (in blue), thuscharacterizes the game situation in each cluster. We cansee that the agent is spending its time in a very definedareas in the state space at each cluster. For example, cluster19 it is located in the north-west part of the screen and incluster 9 it is located in south-east. We also observe thatclusters with more transitions, e.g., clusters 0 and 2, sufferfrom less defined mean state.

0 5 10 15 20 25Cluster index

2

0

2

4

Valu

e

DQN value vs. SAMDP value

DQN

SAMDP

0 5 10 15 20 25Cluster index

1

0

1

Corr

ela

tion

Correlation between greedy policy and trajectory reward

0.00 0.02 0.04 0.06 0.08 0.10 0.12 0.14Percentage of extermum trajectories used

0.2

0.3

0.4

0.5

Weig

ht

Greedy policy weight in good/bad trajectories

High reward trajs

Low reward trajs

Figure 3. Model Evaluation. Top: Value function consistency.Center: greedy policy correlation with trajectory reward. Bot-tom: top (blue), least (red) rewarded trajectories.

Model Evaluation. We evaluate our model using three dif-ferent methods. First, the VMSE criteria (Figure 3, top):high correlation between the DQN values and the SMDPvalues gives a clear indication to the fitness of the model tothe data. Second, we evaluate the correlation between thetransitions induced by the policy improvement step and thetrajectory reward Rj . To do so, we measure P ji : the em-pirical distribution of choosing the greedy policy at state ciin that trajectory. Finally we present the correlation coeffi-cients at each state: corri = corr(P ji , R

j) (Figure 3, cen-ter). Positive correlation indicates that following the greedypolicy leads to high reward. Indeed for most of the stateswe observe positive correlation, supporting the consistency


Figure 2. SMDP visualization for Pacman.

of the model. The third evaluation is close in spirit to thesecond one. We create two transition matrices T+, T− us-ing k top-rewarded trajectories and k least-rewarded tra-jectories respectively. We measure the correlation of thegreedy policy TG with each of the transition matrices fordifferent values of k (Figure 3 bottom). As clearly seen, thecorrelation of the greedy policy and the top trajectories ishigher than the correlation with the bad trajectories.

8. DiscussionIn this work we considered the problem of visualizing dy-namics. Starting with a t-SNE map of the neural activa-tions of a DQN and ending up with an SMDP model de-scribing the underlying dynamics. We developed cluster-ing algorithms that take into account the temporal aspectsof the data and defined quantitative criteria to rank candi-date SMDP models based on the likelihood of the data andan entropy simplicity term. Finally we showed in the exper-iments section that our method can successfully be appliedon two Atari2600 benchmarks, resulting in a clear interpre-tation for the agent policy.Our method is fully automatic and does nor require anymanual or game specific work. We note that this is a workin progress, it is mainly missing the quantitative results forthe different likelihood criteria. In future work we will fin-ish to implement the different criteria followed by the rele-vant simulations.

ReferencesDean, Thomas and Lin, Shieu-Hong. Decomposition tech-

niques for planning in stochastic domains. 1995.

Dietterich, Thomas G. Hierarchical reinforcement learningwith the MAXQ value function decomposition. J. Artif.Intell. Res.(JAIR), 13:227–303, 2000.

Duda, Richard O, Hart, Peter E, and Stork, David G. Pat-tern classification. John Wiley & Sons, 2012.

Erhan, Dumitru, Bengio, Yoshua, Courville, Aaron, andVincent, Pascal. Visualizing higher-layer features of adeep network. Dept. IRO, Universite de Montreal, Tech.Rep, 4323, 2009.

Francis, Bruce A and Wonham, William M. The internalmodel principle for linear multivariable regulators. Ap-plied mathematics and optimization, 2(2), 1975.

Hagberg, Aric A., Schult, Daniel A., and Swart, Pieter J.Exploring network structure, dynamics, and function us-ing NetworkX. In Proceedings of the 7th Python in Sci-ence Conference (SciPy2008), August 2008.

Hallak, Assaf, Di-Castro, Dotan, and Mannor, Shie. Modelselection in markovian processes. In Proceedings of the19th ACM SIGKDD international conference on Knowl-edge discovery and data mining. ACM, 2013.


Hauskrecht, Milos, Meuleau, Nicolas, Kaelbling,Leslie Pack, Dean, Thomas, and Boutilier, Craig.Hierarchical solution of Markov decision processesusing macro-actions. In Proceedings of the Fourteenthconference on Uncertainty in artificial intelligence, pp.220–229. Morgan Kaufmann Publishers Inc., 1998.

Krizhevsky, Alex, Sutskever, Ilya, and Hinton, Geoffrey E.Imagenet classification with deep convolutional neuralnetworks. In Advances in neural information processingsystems, pp. 1097–1105, 2012.

Kulkarni, Tejas D, Narasimhan, Karthik R, Saeedi, Arda-van, and Tenenbaum, Joshua B. Hierarchical deep rein-forcement learning: Integrating temporal abstraction andintrinsic motivation. arXiv preprint arXiv:1604.06057,2016.

Lin, Long-Ji. Reinforcement learning for robots using neu-ral networks. Technical report, DTIC Document, 1993.

Lunga, Dalton, Prasad, Santasriya, Crawford, Melba M,and Ersoy, Ozan. Manifold-learning-based feature ex-traction for classification of hyperspectral data: a reviewof advances in manifold learning. Signal ProcessingMagazine, IEEE, 31(1):55–66, 2014.

MacQueen, James et al. Some methods for classificationand analysis of multivariate observations. 1967.

Mnih, Volodymyr, Kavukcuoglu, Koray, Silver, David,Rusu, Andrei A, Veness, Joel, Bellemare, Marc G,Graves, Alex, Riedmiller, Martin, Fidjeland, Andreas K,Ostrovski, Georg, et al. Human-level control throughdeep reinforcement learning. Nature, 518(7540), 2015.

Parr, Ronald. Flexible decomposition algorithms forweakly coupled Markov decision problems. In Proceed-ings of the Fourteenth conference on Uncertainty in arti-ficial intelligence, pp. 422–430. Morgan Kaufmann Pub-lishers Inc., 1998.

Rissanen, Jorma. Modeling by shortest data description.Automatica, 14(5):465–471, 1978.

Rusu, Andrei A, Colmenarejo, Sergio Gomez, Gulcehre,Caglar, Desjardins, Guillaume, Kirkpatrick, James, Pas-canu, Razvan, Mnih, Volodymyr, Kavukcuoglu, Koray,and Hadsell, Raia. Policy distillation. arXiv preprintarXiv:1511.06295, 2015.

Sontag, Eduardo D. Adaptation and regulation with sig-nal detection implies internal model. Systems & controlletters, 50(2):119–126, 2003.

Stolle, Martin and Precup, Doina. Learning options in re-inforcement learning. Springer.

Sutton, Richard S, Precup, Doina, and Singh, Satinder. Be-tween MDPs and semi-MDPs: A framework for tempo-ral abstraction in reinforcement learning. Artificial Intel-ligence, 112(1), August 1999.

Tang, Jian, Liu, Jingzhou, Zhang, Ming, and Mei, Qiaozhu.Visualizing large-scale and high-dimensional data. InProceedings of the 25th International Conference onWorld Wide Web. International World Wide Web Con-ferences Steering Committee, 2016.

Tenenbaum, Joshua B, De Silva, Vin, and Langford,John C. A global geometric framework for nonlineardimensionality reduction. Science, 290(5500), 2000.

Tessler, Chen, Givony, Shahar, Zahavy, Tom, Mankowitz,Daniel J, and Mannor, Shie. A deep hierarchical ap-proach to lifelong learning in minecraft. arXiv preprintarXiv:1604.07255, 2016.

Van Der Maaten, Laurens. Accelerating t-SNE using tree-based algorithms. The Journal of Machine Learning Re-search, 15(1):3221–3245, 2014.

Van der Maaten, Laurens and Hinton, Geoffrey. Visual-izing data using t-SNE. Journal of Machine LearningResearch, 9(2579-2605):85, 2008.

Ward, Joe H. Hierarchical grouping to optimize an objec-tive function. Journal of the American Statistical Asso-ciation, 58(301):236–244, 1963.

Yi, Tau-Mu, Huang, Yun, Simon, Melvin I, and Doyle,John. Robust perfect adaptation in bacterial chemotaxisthrough integral feedback control. Proceedings of theNational Academy of Sciences, 97(9):4649–4653, 2000.

Yosinski, Jason, Clune, Jeff, Bengio, Yoshua, and Lipson,Hod. How transferable are features in deep neural net-works? In Advances in Neural Information ProcessingSystems, pp. 3320–3328, 2014.

Zahavy, Tom, Zrihem, Nir Ben, and Mannor, Shie. Gray-ing the black box: Understanding dqns. arXiv preprintarXiv:1602.02658, 2016.

Zeiler, Matthew D and Fergus, Rob. Visualizing and under-standing convolutional networks. In Computer Vision–ECCV 2014, pp. 818–833. Springer, 2014.

Visualizing Dynamics: from t-SNE to SEMI-MDPs › icmlviz2016 › assets › papers › 10.pdfusing a manual inspection of a t-Distributed Stochastic Neighbor Embedding (t-SNE) map

Documents