Network A/B Testing: From Sampling to EstimationNetwork A/B Testing: From Sampling to Estimation Huan Guiy Ya Xuz Anmol Bhasinz Jiawei Hany yUniversity of Illinois at Urbana-Champaign,

Network A/B Testing: From Sampling to Estimation

Huan Gui† Ya Xu‡ Anmol Bhasin‡ Jiawei Han†† University of Illinois at Urbana-Champaign, Urbana, IL 61801 USA

‡ LinkedIn Corporation, Mountain View, CA 94043 USA†{huangui2, hanj}@illinois.edu ‡{yaxu, abhasin}@linkedin.com

ABSTRACTA/B testing, also known as bucket testing, split testing, orcontrolled experiment, is a standard way to evaluate userengagement or satisfaction from a new service, feature, orproduct. It is widely used in online websites, including socialnetwork sites such as Facebook, LinkedIn, and Twitter tomake data-driven decisions. The goal of A/B testing is to es-timate the treatment effects of a new change, which becomesintricate when users are interacting, i.e., the treatment ef-fects of a user may spill over to other users via underlyingsocial connections. When conducting these online controlledexperiments, it is a common practice to make the StableUnit Treatment Value Assumption (SUTVA) that each in-dividual’s response is affected by their own treatment only.Though this assumption simplifies the estimation of treat-ment effects, it does not hold when network interference ispresent, and may even lead to wrong conclusion.

In this paper, we study the problem of network A/B test-ing in real networks, which have substantially different char-acteristics from the simulated random networks studied inprevious works. We first examine the existence of networkeffects in a recent online experiment conducted at LinkedIn;Secondly, we propose an efficient and effective estimator forAverage Treatment Effect (ATE) considering the interfer-ence between users in real online experiments; Finally, weapply our method in both simulations and a real world onlineexperiment. The simulation results show that our estimatorachieves better performance with respect to both bias andvariance reduction. The real world online experiment notonly demonstrates that large-scale network A/B test is fea-sible but also further validates many of our observations inthe simulation studies.

Categories and Subject DescriptorsG.3 [Probability and Statistics]: Experimental design

Copyright is held by the International World Wide Web Conference Com-mittee (IW3C2). IW3C2 reserves the right to provide a hyperlink to theauthor’s site if the Material is used in electronic media.WWW 2015, May 18–22, 2015, Florence, Italy.ACM 978-1-4503-3469-3/15/05.http://dx.doi.org/10.1145/2736277.2741081.

KeywordsA/B testing; controlled experiments; network A/ B testing;design of experiments; peer effects; network effects; balancedgraph partition.

1. INTRODUCTIONA/B testing, also called controlled experimentation, is

widely used in many consumer facing web technology com-panies to guide product development and data-driven de-cisions, including Amazon, eBay, Etsy, Facebook, Google,Groupon, LinkedIn, Microsoft, Netflix and Yahoo. It hasbecome the gold standard for testing out new product strate-gies and approaches [14, 13].

The theory of A/B test is simple and dates back to SirRonald A. Fisher’s experiments at the Rothamsted Agri-cultural Experimental Station in England in the 1920s [29].Rubin causal model [23], a standard machinery of testingframework, is usually adopted in conducting and analyzingA/B tests. A key assumption made in Rubin causal modelis the Stable Unit Treatment Value Assumption (SUTVA),which states that the behavior of each user in the experi-ment depends only on their own treatment and not on thetreatments of others.

However, in a social network setting, a user’s behavior islikely impacted by that of his/her social neighborhood [12,27, 28]. In most cases, a user would find a new feature morevaluable and hence more likely to adopt it if more of his/herneighbors adopt it. For example, video chat is a useless fea-ture unless one’s friends use it too. The phenomenon that anindividual’s behavior has a non-trivial effect on his/her so-cial neighborhood is called network effects, also known as so-cial interactions, peer influence, or social interference [2, 8].In an A/B experiment, this implies that if the treatment hasa significant impact on a user, the effect would spill over tohis/her social circles, regardless whether his/her neighborsare in treatment or control.

To understand how network effects introduces challengesfor A/B testing, consider the Average Treatment Effect (ATE),a primary quantity of interest defined as the difference of theaverage outcomes between applying treatment to the entireuser population and applying control to the entire user pop-ulation. Define U as the set of users and |U| = N , whereN is the total number of users. Let Z ∈ MN , a vector oflength N , as the experiment assignment vector for all users,where M is the set of variants. Without loss of generality,for the rest of the paper, we assume the experiment only hastwo variants thatM = {0, 1}, where 1 stands for treatmentand 0 for control. We let Yi(Z = z) be the response func-

399

tion of user i given Z = z. The response can be any usermetrics the experiment tries to optimize, such as number ofpageviews or clicks. The ATE can be expressed as

δ(1,0) =1

N·∑i

E[Yi(Z = 1)− Yi(Z = 0)

], (1)

where 1, and 0 are the treatment assignment where all usersreceive the control variant and treatment variant respec-tively.

Of course, a user can only receive one treatment at a timein reality. So in classical A/B experiments, we randomlyselect N0 users to receive the control variant, and N1 usersto receive treatment. The ATE can then be estimated by

δ =1

N1·∑

{i;zi=1}

Yi(Z = z)− 1

N0·∑

{i;zi=0}

Yi(Z = z), (2)

where zi is the experiment assignment of user i. UnderSUTVA and the Law of Large Numbers [11], we have δ →δ(1,0) whenN1, N0 →∞. However, when there are networkeffects, STUVA is no longer valid, and the convergence doesnot hold any more.

This poses a special challenge for running A/B tests inmany online social and professional networks like Facebook,Twitter and LinkedIn. Many features tested there throughA/B experiments are likely to have network effects. For ex-ample, a better recommendation algorithm in treatment forthe People You May Know module on LinkedIn encouragesa user to send more invitations. However, users who receivesuch invitations can be in the control variant and when theyvisit LinkedIn to accept the invitation they may discovermore people they know. If the primary metric of interest isthe total number of invitations sent, we would see a posi-tive gain in both the treatment and the control groups. TheATE estimated ignoring network effects would be biased andnot fully capturing the benefit of the new algorithm. Suchbias exists in testing almost any features that involve socialinteractions, which is truly ubiquitous in a social networkenvironment.

In this paper, we define the problem of A/B testing whenthere are network effects as Network A/B Testing. Specif-ically, we study the problem in large real social networks,which have substantially different properties and character-istics than simulated random networks studied in previousworks. Our work aims at bridging the gap among theoret-ical analysis on casual analysis in the Statistical literature,recent works on network bucket testing [3, 8, 28], and real-world applications of online controlled experiments. Ourmain contributions are as follows:

1. As far as we know, we are the first to extensively studythe problem in real social networks.

2. We propose a simple yet efficient network samplingalgorithm that is able to overcome an important chal-lenge presented in sampling real social networks.

3. We propose a new estimation model that not onlygeneralizes the existing methods, but also producessmaller bias and variance in our extensive simulationstudies.

4. We are the first to study how the overall traffic split be-tween treatment and control impacts the performanceof various ATE estimators.

5. We run a large-scale network A/B test with a real ap-plication at LinkedIn. The results from the experimentnot only help us compare the various models but alsofurther validate many of the observations from the sim-ulation studies.

The rest of the paper is organized as follows. Related workis discussed in Section 2. In Section 3 we take a close lookat possible network effects in a recent A/B experiment con-ducted at LinkedIn to motivate our study. Section 4 is wherewe introduce our framework for network A/B testing andpropose our new network sampling and estimation meth-ods. Extensive simulations and online experimental resultsare included in Section 5. Section 6 concludes our study,summarizing our approach and motivating future work inthis area.

2. RELATED WORKIn this section, we will introduce some related work on

network A/B testing, mainly including two parts, interfer-ence analysis in Statistics, and network bucket testing inComputer Science.

There are two parts of related work on interference, oneis based on group-level interference analysis, where thereis interference within each group, and no interference acrossgroups [10, 22, 24, 26]; the other one is based on unit-level in-terference analysis, where interference between any two unitsmay be non-trivial [2, 8, 16, 27, 28]. [22] explores methodsof inverting distribution-free randomization tests for Fisher’ssharp null hypothesis based on group-level interference anal-ysis. [24] considers the potential bias for ATE estimationwhen SUTVA is assumed, and further defines several causalestimands including direct and indirect effects that mightbe identifiable. This work is further extended by [10] bydefining direct, indirect, total and overall causal effects, andrelationship between these estimands are established. Hude-gens et al. [10] also propose unbiased estimators of the pro-posed estimands by a two-stage randomization procedureexperimental design that first perform randomization at thegroup level, then at the individual level within groups. [26]gives conservative variance estimators, and proposes a newframework of finite sample inference analysis and inverseprobability weighting estimators considering interference.

Regarding arbitrary interference of known form betweenunits, [2] proposes randomization based methods to estimateATE, and further refines the estimator by covariance ad-justment, besides analysis of conservative estimators of theATE estimators. [16] explores the identification of users’ re-sponses under different constraints and assumptions, consid-ering that social interference will influence users’ behaviors.To quantify the causal effect of peer influence, [27] intro-duces new randomization based causal estimand of peer in-fluence, by extending the potential responses with social in-terference. [8, 28] focus on estimating the ATE in networks,in which a new randomization scheme is proposed, calledgraph clustering randomization. Extensive simulations areconducted on random networks. We further this work bystudying the ATE estimation on real networks, which havedrastically different topologies and structures.

Another line of research that is quite related to our workis network bucket testing [3, 12], in which the new featurewill take effect only if some minimal number of treated users’social neighbors are also in treatment. [3] proposes a walk-

400

based sampling method to generate the core set of userswhich are internally well-connected while approximately uni-form over the whole network. This work is generalized by [12],which introduces a general framework for network A/B test-ing, which first generate the core set based on different algo-rithms, and then gives out the corresponding variance boundbased on the core set generation function.

Our work takes both the sampling (design) and estimation(analysis) into consideration, by proposing a realistic sam-pling method that works well in real networks and a newestimation framework that generalizes the existing ones.

3. NETWORK EFFECTS IN REAL EXPER-IMENT

There is a line of research studying social interference,also known as information diffusion [6, 9, 21]. Wheneverinformation can be propagated from a user to his/her so-cial neighborhood, there is likely to be network effects. Inparticular, we are interested in studying how such informa-tion propagates between the treatment and control groupsin an A/B test setting. With that in mind, we first take aclose look at a real A/B experiment recently conducted atLinkedIn on homepage Feeds. By incorporating componentsthat are specific to one’s social neighborhood, we show us-ing a linear model that there are significant network effectspresent in this experiment.

3.1 Feed ExperimentFeed is an important part of LinkedIn homepage experi-

ence. It provides users with stories that they may be in-terested in reading, updates from their network (such as ajob change), and more. One can click on a feed, or interactwith it through social gestures such as “like”, “comment” or“share”.

The Feed team strives to surface the most relevant itemsto users by constantly testing new recommendation algo-rithms. One important evaluation criterion is whether thenew algorithm has improved the total number of user in-teractions with the feeds. In a recent experiment, users arerandomly split into two equal groups, one receives the pro-duction algorithm (control) and the other receives a newexperimental algorithm (treatment) that is supposed to pro-vide more relevant feeds.

Because the sampling is uniformly random, users in treat-ment have neighbors in both treatment and control. There-fore, users in the control group will not only receive the setof recommended feeds from the control recommendation al-gorithm, but also the set of feeds shared/liked from theirneighbors in treatment group, and vice versa. If the newalgorithm is indeed better, and users are interacting morewith the more relevant content discovered in their feeds,those good content pieces will leak into the control groupand hence makes the control algorithm look better that itactually is.

In the following section, we will show that such concernsof information “leaking” is not vacuous and there is indeedsignificant network effects present in this feed experiment.We verify by extending the classical framework used to es-timate ATE to incorporate components that are specific toone’s social neighborhood.

3.2 Presence of Network EffectTwo-sample t-test is the most commonly used framework

to analyze online experiment [7]. Suppose we are interestedin some response metric Y (e.g. pageviews per user). Thenull hypothesis is that treatment and control have the samemean for this metric (i.e. ATE = 0) and the alternative isthat they do not (i.e. ATE 6= 0) . The t-test is based on thet-statistic

δ√var(δ) , (3)

and under SUTVA, δ defined in (2) is an unbiased estimatorof the ATE and the t-statistic is a normalized version ofthat estimator. This framework is in fact equivalent to thefollowing linear model

Yi(Z) = α+ βZi (4)

where Zi ∈ {0, 1} is user i’s experiment assignment and the

least square estimator for β turns out to be δ.In order to take into consideration of possible network ef-

fects, we introduce two additional components to the linearmodel above, social interference and homophily [15, 17]. Thesocial interference component aims at capturing the “spill-over”treatment effects from one’s neighbors, and is thereforemodeled based on the total number of treated neighbors, i.e.,A>·iZ, where A is the adjacency matrix, and A·i is the ithcolumn of A. The homophily refers to the observation thatone’s social network is homogeneous considering differentsociodemographic, behavioral, and intrapersonal traits, i.e.,people are more likely to connect with others who are sim-ilar (birds of a feather). To this end, a user’s behavior canbe partially explained by the behavior of his/her neighbors.Therefore, homophily is approximated by the average be-havior of i’s neighborhood, i.e., A>·iY /Dii, where D is the

diagonal matrix that Dii =∑Nj=1Aij . Putting the three

pieces together, we have

Yi(Z) = α+ βZi + γA>·iZ + ηA>·iY /Dii. (5)

Note that linear additive models have been widely used inother works on causal analysis [15, 27]. The three parame-ters β, γ and η aim at capturing treatment effects, networkeffects and homophily respectively. Under this simple lin-ear model, we can estimate the size of each effect and testwhether each effect is statistically significant. The resultsare shown in Table 1 where we can see that users’ responsesare positively correlated with treatment effects (primary ef-fects), network effects, and homophily.

Metric β γ η

# of interactions 0.0486 0.1252 0.0626

Table 1: The linear regression results of the feed experiment.All the p-value are < 10−16, and thus omitted.

It is important to point out that this experiment is con-ducted with fully uniform random sampling, and hence thenumber of treated neighbors and the neighborhood size arehighly correlated. This means that the network effects wemeasure here could be confounded with one’s popularity in

401

the network. To control for that, we have also tried a modelthat includes one’s neighborhood size as the fourth compo-nent, as follows:

Yi(Z) = α+ βZi + γA>·iZ + ηA>·iY /Dii + κDii.

The coefficient γ for the network effects stays significantlypositive. Also note that because the sampling is done uni-formly random, every user in the experiment has about thesame percentage of neighbors in treatment. This means thatwe cannot expect to remove such confounding effects by us-ing the percent of treated neighbors instead of the absolutenumber.

4. NETWORK A/B TESTINGThe framework of A/B testing usually involves two parts:

sampling, which determines who gets what treatment; andestimation, which provides a framework to compute the ATE.

Uniform random sampling is the most commonly usedsampling method in most web facing applications. It issimple and sufficient in most cases when the sample sizeis large. However, as we have mentioned in the last section,uniform random sampling makes it difficult to separate outthe contribution of the network effects from other confound-ing factors such as neighborhood size. To this end, we lookfor a sampling solution in Section 4.1 that is able to takeinto consideration the network structure. The idea is to re-move information diffusion between treatment and controlgroups as much as we can. However, as we will see, net-work sampling itself poses special challenges in real socialnetworks because of their sizes and structures. We proposea sampling method that is able to overcome several of thesechallenges in Section 4.1.2. With the new sampling algo-rithm, we then study the problem of estimation, where wepropose an exposure model and the corresponding estima-tors in Section 4.2 and show that they are a generalizationof the existing estimators.

4.1 Network SamplingSampling is the process where we decide which users get

assigned into what variant. At a high level, we try to splitusers into treatment and control so that there is as littleinformation flow between the variants as possible. The sam-pling scheme we adopt here is called cluster randomized sam-pling, which is a sampling method where clusters of users arerandomized together [18, 20]. It is a two-stage procedure:

1. Partition the users into clusters;

2. Treat each cluster as a unit and randomize at the clus-ter level, so that all the users in the same cluster areassigned to the same variant.

The advantages of cluster randomized sampling includethe ability to study the interference within the same clusterand the ability to control for “contamination” across clus-ters [8]. In this section, we will start with a recent proposalcalled ε-net [28]. We examine its performance in a real socialnetwork. We then propose a new sampling algorithm thatcan overcome an important limitation and hence reduce thebias of the ATE estimator.

4.1.1 Graph Cluster RandomizationGraph clustering, also known as community detection,

graph cutting, and graph partition, is a well-researched area [1,4, 19]. However, clustering real social networks is challeng-ing due to their special structure and topology. [28] proposesa local clustering algorithm, ε-net, that overcomes some ofthose challenges and the idea is as follows:

1. Find k cluster centers that are at least 2ε− 1 apart;-

2. Assign the nodes to their nearest center, and break thetie randomly if needed.

To study how this clustering algorithm works on real net-works, we extract a sub-network from LinkedIn’s social graphby taking all the users who have ever worked at a leadinginternet technology company with a global presence. Thesub-network (employee network) is constructed by extract-ing the connections among them. After removing nodes withno connections, we get a network of about 70K users. Thebasic statistics of this sub-network are given in Table 2. We

Nodes # Edges # max{di} mean {di} var{di}7.26e4 2.88e6 3997 39.67 1.27e4

Table 2: Basic statistics of a employee network.

then apply the 3-net clustering algorithm to partition thissub-network into 100 shards. The distribution of the clustersizes in logarithmic scale is given in Figure 1.

0 20 40 60 80 100Cluster Index

0

2

4

6

8

10

12

14

16

log(C

lust

er

Siz

e)

Figure 1: Cluster sizes of 3-net Network Clustering. Y axisspecifies the log2 of cluster sizes.

It is not surprising to find out that the cluster sizes varyquite a lot. Intuitively, the ε-net algorithm constructs clus-ters of radius at least ε− 1 around each center node, whichimplies that the size of each cluster is largely determined bythe degrees of the center node. Because degrees are hetero-geneous in real networks, so are the cluster sizes. But howwould this influence the estimation of ATE?

According to bias analysis in [18], the ATE estimatorgiven in (2) has the following bias

E[δ]− δ = −NcN

[1

|J1|Cov

(∑j∈J1

∑nji=1 Yij∑

j∈J1nj

,∑j∈J1

nj

)

− 1

|J0|Cov

(∑j∈J0

∑nji=1 Yij∑

j∈J0nj

,∑j∈J0

nj

)],

402

where J1 (J0) is the set of clusters that are assigned to treat-ment (control), Nc = |J0| + |J1| is the number of clusters,Yij is the response of i-th user in the j-th cluster, nj is thesize of cluster j, and Cov(·, ·) is the covariance function.

To get an unbiased estimator, we need to have E[δ]−δ = 0,i.e.,

Cov

(∑j∈J1

∑nji=1 Yij∑

j∈J1 nj,∑j∈J1

nj

)= 0,

Cov

(∑j∈J0

∑nji=1 Yij∑

j∈J0 nj,∑j∈J0

nj

)= 0.

(6)

There are two scenarios under which (6) can hold. Scenario(i): users’ responses Yij ’s are independent of cluster size nj ;Scenario (ii): nj ’s are constant. When there are networkeffects, we expect users in larger treatment clusters to getmore social influence, and hence scenario (i) is unlikely tohold. Therefore, we need a network sampling method thatis able to produce equal size clusters. Based on this ob-servation and the large scale of real online social networks,we propose a simple yet efficient balanced graph partitionalgorithm, called randomized balanced graph partition.

4.1.2 Randomized Balanced Graph PartitionIn this section, we introduce the randomized balanced

graph partition algorithm with the goal of producing clus-ters that are of equal size and that can be computationallyfeasible to be applied to real, large-scale social networks.

We start with an initial feasible partition where all theclusters are of the same size. Our goal is to maintain thebalance of the cluster sizes while maximize the number ofedges within each cluster. To do that, we iteratively up-date the cluster label of each node based on the majorityvote from its neighbors. The balance of the cluster sizes isachieved by swapping the cluster label of a pair of nodes,instead of simply changing them. The challenge, of course,is to decide which and how many pairs of labels should beswapped.

It is worth noting that the balanced graph partitioningproblem is proven to be NP-complete even for the case withonly two clusters [1]. We propose a greedy algorithm withrandomization, which is effective and efficient and can beeasily parallelizable. Our algorithm alternates the two stepsbelow till convergence:

1. Label Propagation. Define “gain” as an increase of thenumber of edges between nodes of the same cluster.For every pair of nodes, switch their cluster labels ina greedy way until the gain cannot be increased anymore.

2. Random Shuffling. Randomly select ζ% pairs of nodes,and swap their labels.

By randomly shuffling the cluster labels, we can breakthe local optimum, and increase the number of links withinthe same cluster. As shown in Table 3, we can improvethe baseline label propagation by 8.9% with shuffling. Inexperiments, we set ζ = 51. Since the sub-network is notvery large, we are also able to compare our proposal with

1The choice of ζ may be of independent interest. Howeverit is beyond the scope of this paper.

Methods LP RSLP MM

# of links in clusters (×106) 2.161 2.355 2.359

Table 3: Clustering results for Label Propagation (LP) andRandom Shuffling on Label Propagation (RSLP), as well asone based on Modularity Maximization (MM).

Modularity Maximization [5]2, which is known to have goodperformance but computationally expensive for large net-works. Our proposed method achieves comparable resultsyet is scalable and can be easily parallelized.

4.2 EstimationIn this section, we study the problem of how to estimate

ATE. To do that, we need to first decide how we model users’exposure to network effects. As we will see, the assumptionsof existing exposure models are hard to satisfy in real net-works and under the new sampling scheme we proposed inSection 4.1.2.

To address this problem, we propose a new exposure modelthat we describe in Section 4.2.4 and the corresponding es-timators. Through a linear additive model, we are able toshow that our exposure model and estimator is a general-ization of the existing ones.

4.2.1 FrameworkSuppose z = (z1, z2, · · · , zN )′ where zi ∈ {0, 1}, and

the probability of seeing a particular value of z is pz. LetZ = {z : pz > 0}, so that Z = (Z1, ...,ZN )′ is a random vec-tor with support Z and probability P(Z = z) = pz. Define aunit-specific onto function that maps an assignment vectorand unit specific traits ξi ∈ E (such as one’s local neighbor-hood structure) to an expected user response f : Z×E → ∆.The codomain of ∆ contains all the expected treatment re-sponses that might be induced in the experiment.

Instead of being determined by the entire treatment as-signment vector, users’ responses are determined by differenttreatment exposures, which are resulted from the interactionof sampling design (Z) and the traits of users ξi. The keystep of causal inference, as well as treatment effects estima-tion, is to specify f .

The ATE can be expressed as

δ =1

N·N∑i=1

fi(Z = 1, ξi)−1

N·N∑i=1

fi(Z = 0, ξi)

= τ1 − τ0

(7)

where τ1 is the expected response when treatment is appliedglobally, similarly for τ0. We will start with two existingexposure models and their corresponding ATE estimators.

4.2.2 SUTVAUnder the Stable Unit Treatment Value Assumption, also

known as Individual Treatment Response, f is defined as

fSi (Z, ξi) = I(Zi = 1) · τ1 + I(Zi = 0) · τ0. (8)

In other words, (i) user i’s treatment exposure depends onlyon his/her own treatment, regardless of the treatment as-

2The Modularity Maximization algorithm does not generateclusters with equal sizes, so that we constrained the maxi-mum cluster size during each iteration.

403

signments of other users in the network; (ii) user i’s expectedresponse under treatment (control) is the same as if everyonewere in treatment (control). It is easy to see that under thisresponse function fS(·), the ATE can be estimated basedon (2).

4.2.3 Neighborhood ExposureThere are two basic Neighborhood Exposure models, one

based on the percent of treated neighbors and the otherbased on the absolute number of treated neighbors. Weassume the former in our discussion here as it is more robustto the heterogeneity of users’ degrees, though the analysisfor the latter is similar.

Given a threshold θ ∈ [0, 1], the expected response func-tion f for the neighborhood exposure model is defined as

fNi (Z, ξi) = I(Zi = 1, σi ≥ θ) · τ1 + I(Zi = 0, σi ≤ 1− θ) · τ0+(I(Zi = 0, σi ≥ 1− θ) + I(Zi = 1, σi ≤ θ)

)· xi(Z, ξi),

where xi(Z, ξi) is an unknown function and σi is the percentof i’s neighbors in treatment. Define T θi = {Z|Zi = 1, σi ≥θ}. Then we have that ∀ Z ∈ T θi , i is neighborhood exposedto treatment. Similarly, define Cθi = {Z|Zi = 0, σi ≤ 1− θ}.We have that ∀ Z ∈ Cθi , i is neighborhood exposed to control.

Under the neighborhood exposure model, we have (i) useri’s treatment exposure depends on not just his/her owntreatment but also his/her neighbors’ treatment assignment;(ii) user i’s expected response when he/she is network ex-posed to treatment (control) is τ1 (τ0), the same as if every-one were in treatment (control).

For i to be neighborhood exposed to treatment, i shouldbe in treatment and at least θ fraction of i’s neighbors arealso in treatment. For users who are not neighborhood ex-posed to either treatment or control, their responses, xi(Z, ξi)are considered invalid and are hence removed from estima-tion. Again, we can estimate τ1 and τ0 based on the samplemeans of the network-exposed users and hence the ATE as

δN =1

Nθ1

·∑

{i;Z∈T θi }

Yi −1

Nθ0

·∑

{i;Z∈=Cθi }

Yi, (9)

where Nθ0 , Nθ

1 are the number of users that are network-exposed to control, treatment with threshold θ respectively.

A key step of applying Neighborhood Exposure model isto select θ, the threshold for σi that determines which nodesmeet the condition of being neighborhood exposed. To seehow θ affects the estimation, we first apply the randomizedgraph partition to the employee network in Table 2. Afterrandomly assigning 50 of the 100 clusters to treatment, wehave an empirical cumulative distribution function of σi forthe nodes in treatment (Figure 2).

By setting different θ’s, we have the following observa-tions.

• If we set θ = 0.9, we have Pr(σi ≤ θ) = 0.79. A largerθ means a weaker assumption for the exposure modeland hence smaller bias. However, about 79% observa-tions are “invalid” and cannot be used to estimate theATE, which leads to larger variance;

• If we set θ = 0.3, we have Pr(σi ≤ θ) = 0.07. 93% ofthe observations can be used in the estimation. How-ever, a smaller θ means a stronger assumption andhence a larger bias.

Figure 2: Empirical cumulative distribution function of σifor nodes in treatment.

The bias and variance trade-off presented in the choice ofθ leads us to propose a new estimator that is able to takeinto consideration the level of exposure, and hence utilize allobservations regardless whether they are network exposed.The new model also turns out to be a generalization of theexposure models and estimators discussed so far.

4.2.4 Fraction Neighborhood ExposureIn this section, we propose a new estimator based on what

we call the Fraction Neighborhood Exposure model. We de-fine f(·) the expected response function to be any functionthat depends only on the user’s experiment assignment andthe fraction of his treated neighbors σi:

fFi (Z, ξi) = g(Zi, σi). (10)

It is easy to see that the ATE in (7) becomes

δ = g(1, 1)− g(0, 0). (11)

Various response function g(·) can be chosen to modelusers’ behavior. We demonstrate using two linear additivemodels.

Linear Additive Model I.The first model we consider is as follows:

g(Zi, σi) = α+ βZi + γσi, (12)

where β captures the treatment effects and γ the networkeffects. α, β, γ can be estimated from users’ responses, asα, β, γ. The additive assumption of various effects is alsomade by previous works, such as the“linear-in-mean”model [15].By (11), the ATE can be estimated as

δLI = β + γ.

Linear Additive Model II.The linear model in (12) can be further generalized by con-

sidering different response functions for users in treatmentand control groups:

g(Zi, σi) =

{α0 + γ0σi, if Zi = 0α1 + γ1σi, if Zi = 1

(13)

404

where α0, γ0 are learned from observation data of usersin control group, while α1, γ1 are learned from observationdata of users in treatment group. By (11), the ATE can beestimated as

δLII = α1 + γ1 − α0.

4.2.5 Comparison of Exposure ModelsWhile comparing the three exposure models we discussed

so far, we show in this section that the SUTVA and neigh-borhood exposure model are special cases of the fractionneighborhood exposure model.

Lemma 1. SUTVA is a special case of fraction neighbor-hood exposure model in (12).

Proof. It can be trivially proved by setting γ = 0, i.e., setthe network effects to be zero.

For the case of neighborhood exposure model, we onlyneed to show that the assumption of the response functionis a special case of the linear model in (12).

Lemma 2. The neighborhood exposure model is a specialcase of the fraction neighborhood exposure model in (12).

Proof. The response function in neighborhood exposure modelcan be defined as follows:

fNi (Z, ξi) =

τ0 if Zi = 0, σi < 1− θτ1 if Zi = 1, σi > θinvalid otherwise

which is equivalent to

fNi (Z, ξi) =

α+ βZi if Zi = 0, σi < 1− θα+ βZi if Zi = 1, σi > θinvalid otherwise

,

by re-parameterizing α = τ0, β = τ1 − τ0. In other words,fNi (Z, ξi) = α+βZi restricted to only users who are network-exposed.

5. EXPERIMENTAL RESULTSIn order to demonstrate the effectiveness and efficiency

of fraction neighborhood model, we performed various ex-periments including extensive simulations and a real onlineexperiment. These two types of experiments are comple-mentary. We can obtain ground truth in the simulationstudy, so it is possible to evaluate methods under variousconditions based on their bias and variance. On the otherhand, real experiment offers special insights into how modelscompare in practice. Also note that all simulations here arebased on real social network, so we only need to simulateuser responses.

5.1 SimulationsConsidering the heterogeneity of degrees of nodes in real

social networks, we will use the Randomized Balanced GraphPartition algorithm proposed in Section 4.1.2 to determinethe experiment assignment and focus on comparing the es-timation models. To be more specific, we aim at answeringthe following questions based on the simulations:

1. How does the percentage of units under treatment (orcontrol) influence the estimation?

2. How do different estimation models compare with re-spect to both bias and variance?

All simulations are based on the employee network sum-marized in Table 2, which has a heterogeneous degree dis-tribution with a high variance. We have also run simi-lar simulation analysis on different networks extracted fromLinkedIn’s social network graph, but the results are similar.

5.1.1 Simulation ModelThe observed user response is generated based on the fol-

lowing probit model [8]3

Yi,t = λ0 + λ1 · Zi + λ2 ·A>·i · Yt−1

Dii+ Ui,t

Yi,t = I(Yi,t > 0),

(14)

where Yi,t is the response of i at time t and Ui,t ∼ N (0, 1)is a stochastic component capturing user specific traits. λ0

is a constant that λ0 = −1.5. λ1 captures the strengthof treatment effects, and λ2 is for the network effects. Weinitialize Yi,0 = 0 for all users, and then run the iterativeprocess to generate users’ responses for T steps with differentcombinations of λ1 and λ2 where λ1 ∈ {0.25, 0.5, 0.75, 1.0},and λ2 ∈ {0, 0.1, 0.5, 1.0}. The step length is set to T = 3.

It is worth noting that in this data generation model, one’sresponse depends directly on the behavior of his/her neigh-bors at the previous time stamp, not simply the neighbors’experiment assignments, which is quite realistic.

5.1.2 EstimatorsWe have discussed different estimators in Section 4.2.1

according to different exposure models. In addition to thesediscussed, we will also include in the comparison the Hajekestimator [2]. Specifically, we will study the following fiveestimators:

1. Under SUTVA, the sample mean estimator as in (2).

2. Under the Neighborhood Exposure Model, the samplemean estimator as in (9).

3. Under the Neighborhood Exposure Model, the Hajekestimator [2] defined as follows

τ0,H =

∑Ni=1 I(zi = 0, σi < 1− θ) · Yi/π(Cθi )∑Ni=1 I(zi = 0, σi < 1− θ) · 1/π(Cθi )

τ1,H =

∑Ni=1 I(zi = 1, σi > θ) · Yi/π(T θi )∑Ni=1 I(zi = 1, σi > θ) · 1/π(T θi )

δNH = τ1,H − τ0,H

(15)

where π(Cθi ) and π(T θi ) are the probabilities for user ito be neighborhood exposed to control and treatmentgiven the cluster level randomization and threshold θ.We set θ = 0.75.

4. Under the Fraction Neighborhood Exposure Model,the linear model I estimator (12).

5. Under the Fraction Neighborhood Exposure Model,the linear model II estimator (13).

3For fair comparison, the simulation process is exactly thesame as [8] except that we use real networks while [8] usessimulated random networks.

405

We did 1000 simulations to compute both bias and vari-ance for each of these estimators. For each simulation, thetrue ATE is estimated by putting all users in treatment orall in control.

5.1.3 Percentage of Users in TreatmentWhen running an A/B test, how to split the traffic be-

tween treatment and control is one of the key questions onehas to decide. This is even more crucial if there are networkeffects, as the overall percentage of users in treatment (ρ)will substantially influence the fraction of one’s neighbors intreatment, which leads to different level of treatment expo-sure. This is demonstrated by how the empirical cumulativedistribution function of σi changes with different ρ’s in Fig-ure 3.

Figure 3: By changing the percentage of units in treatmentρ, the distribution of σi changes significantly.

A more interesting question is how the performance ofdifferent estimators change with ρ. As far as we know, we arethe first one to study this problem and the results are shownin Figure 4. We have the following observations. (i) The biasof each estimator does not change much as ρ changes, but thevariance changes significantly and reaches the smallest whenρ = 0.5 (as expected since Var ∼ 1/

√1/N0 + 1/N1). (ii) For

ρ < 0.3 and ρ > 0.7, the variance of the Hajek estimator islarge because the scale factors πi’s (probabilities of beingneighborhood exposed) are small. (iii) The Hajek estimatorperforms worse than sample mean estimator under the sameneighborhood exposure model. It is interesting to note that,in random networks such difference is minimal [8], showingthe importance of the network structure when evaluatingestimators. (iv) Our linear estimators under the FractionNeighborhood Exposure model achieve the smallest bias andvariance. The LI estimator achieves smaller variance, whileLII estimator achieves smaller bias. The different betweenLI estimator and LII is minimal when ρ = 0.5.

5.1.4 Network EffectWe now compare the performance of different ATE esti-

mators under different levels of treatment effects and net-work effects, by changing the values of λ1 and λ2 in thesimulation model (14). We use ρ = 0.5 and the results are

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9percentage of units in treatment (½)

−0.04

−0.03

−0.02

−0.01

0.00

bia

s of

ATE e

stim

ato

r

Uniform, SUTVA, Sample Mean

Cluster, Neighborhood (0.75), Sample Mean

Cluster, Neighborhood (0.75), Hajek

Cluster, Fraction, Sigma (LI )

Cluster, Fraction, Sigma (LII)

(a) The bias of different estimators

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9percentage of units in treatment (½)

0

1e-4

2e-4

3e-4

4e-4vari

ance

of

ATE e

stim

ato

r

Uniform, SUTVA, Sample Mean

Cluster, Neighborhood (0.75), Sample Mean

Cluster, Neighborhood (0.75), Hajek

Cluster, Fraction, Sigma (LI )

Cluster, Fraction, Sigma (LII)

(b) The variance of different estimators

Figure 4: Behavior of different estimators with different per-centage of neighbors in treatment.

shown in Figure 5. The results with ρ = 0.1 are similar andhence omitted.

We have the following observations. (i) The bias of bothsample mean and Hajek estimators under the neighborhoodexposure model becomes larger as the network effects andtreatment effects become larger; (ii) The variance of the es-timators is fairly consistent across different levels of treat-ment effects and network effects. This is because the num-ber of observations used in estimation does not change asλ1 and λ2 change. In addition, the linear estimators underour fraction neighborhood exposure model achieve a smallervariance than the estimators under the neighborhood expo-sure model because all the observations are included in theestimation. (iii) Our linear estimators achieve the best biasand variance across different levels of network effects andtreatment effects.

406

0.25 0.75 0.25 0.75 0.25 0.75 0.25 0.75

treatment effects (¸1 )

−0.08

−0.06

−0.04

−0.02

0.00

0.02

bia

s of

ATE e

stim

ato

r

network effects¸2 = 0.0

network effects¸2 = 0.1

network effects¸2 = 0.5

network effects¸2 = 1.0

network effects¸2 = 0.0

network effects¸2 = 0.1

network effects¸2 = 0.5

network effects¸2 = 1.0

network effects¸2 = 0.0

network effects¸2 = 0.1

network effects¸2 = 0.5

network effects¸2 = 1.0

network effects¸2 = 0.0

network effects¸2 = 0.1

network effects¸2 = 0.5

network effects¸2 = 1.0

network effects¸2 = 0.0

network effects¸2 = 0.1

network effects¸2 = 0.5

network effects¸2 = 1.0

Uniform, SUTVA, Sample Mean

Cluster, Neighborhood (0.75), Sample Mean

Cluster, Neighborhood (0.75), Hajek

Cluster, Fraction, Sigma (LI )

Cluster, Fraction, Sigma (LII)

(a) The bias of different estimators

0.25 0.75 0.25 0.75 0.25 0.75 0.25 0.75

treatment effects (¸1 )

0

1e-5

2e-5

3e-5

4e-5

vari

ance

of

ATE e

stim

ato

r

network effects

¸2 = 0.0

network effects

¸2 = 0.1

network effects

¸2 = 0.5

network effects

¸2 = 1.0

network effects

¸2 = 0.0

network effects

¸2 = 0.1

network effects

¸2 = 0.5

network effects

¸2 = 1.0

network effects

¸2 = 0.0

network effects

¸2 = 0.1

network effects

¸2 = 0.5

network effects

¸2 = 1.0

network effects

¸2 = 0.0

network effects

¸2 = 0.1

network effects

¸2 = 0.5

network effects

¸2 = 1.0

network effects

¸2 = 0.0

network effects

¸2 = 0.1

network effects

¸2 = 0.5

network effects

¸2 = 1.0

Uniform, SUTVA, Sample Mean

Cluster, Neighborhood (0.75), Sample Mean

Cluster, Neighborhood (0.75), Hajek

Cluster, Fraction, Sigma (LI )

Cluster, Fraction, Sigma (LII)

(b) The variance of different estimators

Figure 5: Behavior of different estimators with different pa-rameters, which controls the strength of treatment effects(λ1) and network effects (λ2), in the simulation model. Theoverall percentage of nodes in treatment ρ = 0.5.

5.2 Real Online ExperimentIn addition to the extensive simulations, we have also con-

ducted a real online experiment at LinkedIn using the net-work A/B testing framework we have proposed. Specifically,we have done the following:

1. Select a country as the sub-network to experiment on.

2. Apply our randomized balanced graph partition algo-rithm to assign users from this country into treatmentand control groups.

3. Apply different Feed algorithms to the treatment andcontrol groups. Estimate the ATE after running theexperiment for two weeks.

Country NS (×106) RS dS

Brazil 19.9 0.932 41.6United States 119.3 0.910 54.3Netherlands 6.1 0.868 93.0

Chile 2.8 0.866 38.4New Zealand 1.3 0.654 29.4

Table 4: Basic statistics about several countries. NS isthe size of the subnetwork, selected by the correspondingcountry; RS is the self-containment measure defined in (16);dS is the average degree of the selected subnetwork definedin (17).

We note that unlike simulations, there is no ground truthfor this real world experiment. The Feed team has, however,compared these two Feed algorithms globally in a uniformlyrandomized A/B test, and the treatment Feed algorithm wassignificantly better than control.

Our goal for the real world experiment is two-fold. First,we would like to compare results from different estimatorsin a real application setting to complement the observationsfrom simulations. In particular, we want to compare re-sults with and without taking into consideration of the net-work effects, and further, how our fraction neighborhoodexposure model compares with the neighborhood exposuremodel. Second, as far as we know, we are the first to run areal network A/B test. We would like to establish a processfor running network A/B test in practice. As we have seenhow the conclusions can differ drastically in real networkscompared to simulated networks, we hope this can bridgethe gap and encourage more research focusing on real appli-cations in the area of network A/B testing.

5.2.1 Country SelectionWe would like to select a country that has a well self-

contained LinkedIn social network. Ideally, It should be anisolated sub-network that has as few connections to the out-side of the country as possible to prevent network influenceto and from users outside. We use the following ratio toquantify such “self-containment” for a set S of users:

RS =

∑i,j∈S Ai,j∑

i∈S∑j∈S∪Sc Ai,j

, (16)

where Sc is the complement of S in the population network.Remark that RS is the ratio between the edge count withinthe set S and all the edges with one end in S.

In addition, we consider the average internal degree suchthat the selected subnetwork is well connected. Consider-ing selected sub-network S, the average internal degree isdefined as

dS =

∑i,j∈S Ai,j

|S| . (17)

We calculate RS and dS for all countries on LinkedIn, andamong the ones with top RS ratios (shown in Table 4), wedecide to pick Netherlands as it has the highest average in-ternal degrees and a reasonably large sized network (around6 million users).

After selecting the Netherlands as the sub-network, weapplied the randomized balanced graph partition algorithm

407

to divide it into 600 shards and randomly picked 300 of themto receive treatment while the rest 300 to receive control.Before performing the A/B test, we have also conducted anA/A test, which is a controlled experiment where treatmentis identical to control. This was to confirm that no bias wasintroduced during the experiment assignment process.

5.2.2 Online ResultsWe let the experiment run for two weeks before we col-

lected data for analysis. The metric we use for evaluationis the average number of social gestures on Feed, such as“like”, “comment” or “share”.

We compute the ATE based on the various estimators de-scribed in Section 5.1.2. The results are shown in Table 5.We have the following observations. (i) The ATE estimatorswith consideration of network effects are all larger than theestimate under SUTVA. This is yet another good confirma-tion that there are indeed network effects presented in theA/B experiment. (ii) The choice of θ in the neighborhoodexposure model matters. The sample mean estimator almostdoubles when θ changes from 0.75 to 0.9. On the other hand,the Hajek estimator gives a smaller estimate when θ changesfrom 0.75 to 0.9, this is due to small values of πi’s when θ islarge. (iv) The fraction neighborhood exposure model giveslarger estimates than existing methods.

Method ATE for social gestures

SUTVA 0.168Neighbor. Exposure θ = 0.75 0.264Neighbor. Exposure θ = 0.9 0.520Hajek. Exposure θ = 0.75 0.625Hajek. Exposure θ = 0.9 0.133Fraction Exposure (I) 0.687Fraction Exposure (II) 0.714

Table 5: ATE estimates from different models for the onlineFeed experiment.

6. CONCLUSION AND FUTURE WORKIn this paper, we study the problem of network A/B test-

ing in real networks. We start by examining a recent A/Bexperiment conducted on LinkedIn without considering net-work structures, which motivates us to set up a frameworkto study both the sampling and the estimation aspects ofthe network A/B testing problem. To address the challengeof degree heterogeneity in real social networks, we come upwith a new randomization scheme based on balanced graphpartitioning, for which an efficient and distributed algorithmis proposed. Based on new sampling scheme, we propose anew method to estimate the average treatment effect (ATE)that is able to take into consideration of the level of networkexposure. Extensive simulations are conducted to evaluatethese methods and the results show that our new propos-als can achieve both a smaller bias and a smaller variance.We have also conducted a real online experiment under theframework we have proposed and the results further validatemany observations from simulations.

On the other hand, there are still many open problems inthe field of network A/B testing that remain to be addressed,

especially with respect to real world applications. First ofall, we did not consider the influence strength between pairsof nodes, which may have significant impact on determin-ing users exposure status; Secondly, real social networks aregrowing all the time, leading to rapid change of networkstructures, which makes network A/B testing even morechallenging considering the effects of newly added edges andnodes. To further complicate the problem, many real exper-iments on social networks are aiming at increasing networkdensity, making the temporal variability a real, noticeableissue. Thirdly, there are different forms of network interfer-ence to be considered. For instance, in discussion groups,information propagates from one user to all other users ofthe same group, so every group acts as a fully connectedsub-network. However each user can belong to multiplegroups. In this case, the graph clustering randomization canno longer split users into treatment and control under thenew information propagation structure. Lastly, our focushere has been on ATE estimation, and we have not touchedupon how virality works and how to preserve it in a networkA/B testing setting. Given the complex structure of realsocial networks and the way viral information propagates,the framework proposed here may not be sufficient.

A/B testing in general is widely used and also well studiedin the industry as it offers the best scientific approach tounderstand the causal impact of product changes on enduser behavior. However, the problem of A/B testing in asocial network setting is no where near solved. A lot ofwork still remains to be done to make it a well-understoodproblem in real world applications. We hope our work herecan bridge some of the gaps and encourage more research inthis area.

AcknowledgementFirstly, we wish to thank our colleagues and friends who haveheld many enlightening discussions with us: Deepak Agar-wal, Mathieu Bastian, Evion Kim, Wayne Tai Lee, HaishanLiu, Mitul Tiwari, Xiaolong Wang, and many members ofthe A/B testing team at LinkedIn. Secondly, we wish tothank Bee-Chung Chen, Liang Zhang, Pannagadatta Shiv-aswamy, Cory Hicks and Caroline Gaffney for working withus to run the network A/B test on LinkedIn Feeds. Lastly,we wish to thank the anonymous reviewers for their valu-able comments and suggestions to improve the quality ofthe paper.

References[1] K. Andreev and H. Racke. Balanced graph

partitioning. Theory of Computing Systems,39(6):929–939, 2006.

[2] P. M. Aronow and C. Samii. Estimating averagecausal effects under general interference. Citeseer,2012.

[3] L. Backstrom and J. Kleinberg. Network buckettesting. In Proceedings of the 20th internationalconference on World wide web, pages 615–624. ACM,2011.

[4] D. A. Bader, H. Meyerhenke, P. Sanders, andD. Wagner. Graph partitioning and graph clustering,volume 588. American Mathematical Soc., 2013.

408

[5] V. D. Blondel, J.-L. Guillaume, R. Lambiotte, andE. Lefebvre. Fast unfolding of communities in largenetworks. Journal of Statistical Mechanics: Theoryand Experiment, 2008(10):P10008, 2008.

[6] J. Cheng, L. Adamic, P. A. Dow, J. M. Kleinberg, andJ. Leskovec. Can cascades be predicted? InProceedings of the 23rd international conference onWorld wide web, pages 925–936. International WorldWide Web Conferences Steering Committee, 2014.

[7] N. Cressie and H. Whitford. How to use the twosample t-test. Biometrical Journal, 28(2):131–148,1986.

[8] D. Eckles, B. Karrer, and J. Ugander. Design andanalysis of experiments in networks: Reducing biasfrom interference. arXiv preprint arXiv:1404.7530,2014.

[9] H. Gui, Y. Sun, J. Han, and G. Brova. Modeling topicdiffusion in multi-relational bibliographic informationnetworks. In Proceedings of the 23rd ACMInternational Conference on Conference onInformation and Knowledge Management, pages649–658. ACM, 2014.

[10] M. G. Hudgens and M. E. Halloran. Toward causalinference with interference. Journal of the AmericanStatistical Association, 103(482), 2008.

[11] K. L. Judd. The law of large numbers with acontinuum of iid random variables. Journal ofEconomic theory, 35(1):19–25, 1985.

[12] L. Katzir, E. Liberty, and O. Somekh. Framework andalgorithms for network bucket testing. In Proceedingsof the 21st international conference on World WideWeb, pages 1029–1036. ACM, 2012.

[13] R. Kohavi, A. Deng, B. Frasca, T. Walker, Y. Xu, andN. Pohlmann. Online controlled experiments at largescale. In Proceedings of the 19th ACM SIGKDDinternational conference on Knowledge discovery anddata mining, pages 1168–1176. ACM, 2013.

[14] R. Kohavi, A. Deng, R. Longbotham, and Y. Xu.Seven rules of thumb for web site experimenters.Proceedings of the 14th ACM SIGKDD internationalconference on Knowledge discovery and data mining,2014.

[15] C. F. Manski. Identification of endogenous socialeffects: The reflection problem. The review ofeconomic studies, 60(3):531–542, 1993.

[16] C. F. Manski. Identification of treatment responsewith social interactions. The Econometrics Journal,16(1):S1–S23, 2013.

[17] M. McPherson, L. Smith-Lovin, and J. M. Cook.Birds of a feather: Homophily in social networks.Annual review of sociology, pages 415–444, 2001.

[18] J. A. Middleton and P. M. Aronow. Unbiasedestimation of the average treatment effect incluster-randomized experiments. Available at SSRN1803849, 2011.

[19] M. E. Newman. Modularity and community structurein networks. Proceedings of the National Academy ofSciences, 103(23):8577–8582, 2006.

[20] S. W. Raudenbush. Statistical analysis and optimaldesign for cluster randomized trials. PsychologicalMethods, 2(2):173, 1997.

[21] M. G. Rodriguez, J. Leskovec, D. Balduzzi, andB. Scholkopf. Uncovering the structure and temporaldynamics of information propagation. NetworkScience, 2(01):26–65, 2014.

[22] P. R. Rosenbaum. Interference between units inrandomized experiments. Journal of the AmericanStatistical Association, 102(477), 2007.

[23] D. B. Rubin. Estimating causal effects of treatmentsin randomized and nonrandomized studies. Journal ofeducational Psychology, 66(5):688, 1974.

[24] M. E. Sobel. What do randomized studies of housingmobility demonstrate? causal inference in the face ofinterference. Journal of the American StatisticalAssociation, 101(476):1398–1407, 2006.

[25] L. Tang, R. Rosales, A. Singh, and D. Agarwal.Automatic ad format selection via contextual bandits.In Proceedings of the 22nd ACM internationalconference on Conference on information & knowledgemanagement, pages 1587–1594. ACM, 2013.

[26] E. J. T. Tchetgen and T. J. VanderWeele. On causalinference in the presence of interference. StatisticalMethods in Medical Research, 21(1):55–75, 2012.

[27] P. Toulis and E. Kao. Estimation of causal peerinfluence effects. In Proceedings of The 30thInternational Conference on Machine Learning, pages1489–1497, 2013.

[28] J. Ugander, B. Karrer, L. Backstrom, andJ. Kleinberg. Graph cluster randomization: networkexposure to multiple universes. In Proceedings of the19th ACM SIGKDD international conference onKnowledge discovery and data mining, pages 329–337.ACM, 2013.

[29] F. Yates. Sir ronald fisher and the design ofexperiments. Biometrics, 20(2):307–321, 1964.

409