Preliminary draft: Please do not quote ALTERNATIVE ... ALTERNATIVE APPROACHES TO THE EX-POST EVALUATION OF INDIVIDUAL MERGER DECISIONS 1. INTRODUCTION A retrospective or ex-post evaluation

1

Preliminary draft: Please do not quote

ALTERNATIVE APPROACHES

TO THE EX-POST EVALUATION

OF INDIVIDUAL MERGER DECISIONS

Adriaan Dierx and Fabienne Ilzkovitz1

(European Commission, Directorate General for Competition)

ABSTRACT

Ex-post evaluations of competition policy decisions have the dual objective of

increasing our understanding of the impact of decisions taken and of informing

and improving future decision making practices. They can thereby contribute to

increasing the transparency of decision making processes within competition

authorities and defending the legitimacy of their market interventions.

Evaluations carried out thus far have led to a better understanding of the (price)

impact of individual merger decisions, but have been less effective in drawing

concrete lessons for competition policy enforcement. To help fill this gap, we

propose the use of a more descriptive and qualitative approach to test if: (1) key

assumptions about market characteristics made at the time of the deicions(e.g.

competitive constraints, countervailing market power, take-up of new

technologies) were well founded; and (2) ex-ante expectations about market

developments (e.g. entry of new competitors, viability of divested entity) turned

out to be true ex-post. This approach has already been used by competition

authorities in Canada, New Zealand and the UK. The proposed use of

descriptive and qualitative methods would complement the application of more

quantitative methods that are better suited to identify the causal relationship

between decisions taken and market outcomes.

1 The paper has been written under the sole responsibility of the authors and does not present the views

of the European Commission.

2

ALTERNATIVE APPROACHES

TO THE EX-POST EVALUATION

OF INDIVIDUAL MERGER DECISIONS

1. INTRODUCTION

A retrospective or ex-post evaluation can be defined as an evidence-based judgment of

the extent to which an action has been effective and efficient in achieving its expected

effects. Over the past decade, competition authorities worldwide have become

increasingly interested in conducting ex-post evaluations of their interventions.

Competition authorities perform ex-post evaluations in order to: (i) enhance decision

making and enforcement practices; (ii) improve effectiveness of competition law; (iii)

set internal priorities; and (iv) provide robust evidence to support the advocacy of

competition and competition policy.2

Within this context, this paper discusses alternative approaches to implement the

evaluation of individual decisions. Section 2 provides a short description of the

organisation of ex-post evaluation activities in the European Commission's Directorate

General for Competition (DG Competition). It also draws some lessons from DG

Competition's recent experience with ex-post evaluation. Section 3 presents the two

main approaches which have been used to analyse the impact of individual decisions: (i)

more sophisticated quantitative methodologies based on structural model simulations

and counterfactual impact analysis using the Difference-in-Differences (DiD) method;

(ii) qualitative and descriptive methodologies, based on desk research, surveys and

interviews. While the first approach aims at identifying the effects of individual

decisions, the second can be used to analyse whether market developments following

the decision are conform to the competition authority's expectations ex-ante. Section 3

also discusses the experience of three competition authorities having used the second

approach. Section 4 considers the pros and cons of using this qualitative and descriptive

approach. It also proposes different options to test its feasibility in a pilot study.

2. ORGANISATION AND SCOPE OF EVALUATION ACTIVITIES IN DG COMPETITION

2.1 Organisation

The organisation of the work on ex-post evaluations within DG Competition contains

elements of centralisation and decentralisation (see Figure 1) and combines internal and

external expertise. Part of the evaluation function is centralised, with an entity

responsible for the coordination of the multi-annual planning of evaluations and

evaluation-related studies as well as the provision of organisational and methodological

support, and the Principal Advisor and the Chief Economist Team taking charge of

2

For a detailed description of the objectives of ex-post economic evaluation of competition policy,

see Ilzkovitz, F. and Dierx, A. (2015), "Ex-post economic evaluation of completion policy

enforcement: A review of the literature", European Commission, ISBN 078-92-79-48350-9,

http://ec.europa.eu/competition/publications/reports/expost_evaluation_competition_policy_en.pdf.

3

individual projects (both outsourced and done in-house) while also providing

methodological support. Operational and policy units are involved in a decentralised

fashion, depending on the subject matter of the evaluation.

DG Competition uses internal and external expertise for the evaluations. Some ex-post

evaluations are conducted in-house but most are outsourced to consultancies and

academic experts. The aim of combining internal and external expertise is to achieve the

right balance between independence and effectiveness. On the one hand, outsourcing

the evaluations may increase the credibility of results if external evaluators are

perceived as more independent than the competition authority involved in the decision3;

it allows exploiting their knowledge on the design and implementation of 'state of the

art' methodologies and provides a fresh perspective. On the other hand, internal

evaluators are more familiar with the activities to be evaluated and have access to

important sources of information. A mixed team combining internal and external

evaluators could offer several advantages and this is the reason why outsourced studies

are closely monitored by DG Competition staff, which implies that in-house resources

are required even if the evaluation is outsourced. In general, a steering group including

different colleagues from DG Competition and other Commission services provides

close guidance to the study team and is consulted at key steps of the project.

As evaluation takes time and resources and requires good data, a careful planning of

evaluation activities is essential. Therefore, DG Competition has developed a

multiannual evaluation programme covering a period of five years which is revised

every year. This multiannual evaluation programme also guarantees a certain degree of

regularity in evaluations. Such regularity allows increasing the available number of ex-

post evaluations which can then progressively be used to derive more general

conclusions. It also allows better organising the collection of data which is essential to

get robust evaluation results.

The scope of ex-post evaluation activities is very broad. Evaluations may cover

individual decisions taken in the different areas of competition policy (mergers, cartels,

abuse of dominance and state aid control), the regulatory framework (legislation,

guidelines, notices) in these areas, and other activities such as sector inquiries, advocacy

actions and international collaboration with other competition authorities. The degree

of aggregation of the evaluation can range from the microeconomic impact of specific

decisions in a given market to the broader impact of competition policy enforcement at

sector and macroeconomic level.

A literature review published on the DG Competition website4 provides an overview of

the extent of the evaluation activities carried out by academics and competition

authorities in these different areas. It shows that most of the existing work has

3

This independence is not necessarily a given as some evaluators might have a tendency to avoid

providing bad evaluation results in order to not jeopardise future relationships with their clients.

4 See Footnote 1.

4

concentrated on the impact of merger and cartel decisions. Some evaluations have been

done to assess the macroeconomic impact of competition policy, focusing on the

customer savings resulting from key cartel and merger decisions. Within the context of

the State aid modernisation, investigations of the effectiveness of State aid schemes

have been launched. However, less work or no work at all has been done to evaluate

State aid control, decisions to halt the abuse of dominance and other activities of

competition authorities, including competition advocacy.

DG Competition started to seriously develop its ex-post evaluation programme in 2014.

Ex-post evaluations carried out since then cover a variety of areas: mergers, antitrust,

state aid or sector and macro-impact. Most of these evaluations were outsourced to

academics or consultancies. Two studies were done internally. The first one is an ex-

post evaluation of two merger decisions in the telecommunications sector. Its two main

objectives were: (i) to improve the understanding of the econometric techniques with a

view to using these techniques in other ex-post evaluations; and (ii) to assess the impact

of the Commission's decisions in a sector which is experiencing a wave of mergers in

the EU with a view to drawing lessons for future decisions. The second one is a study

on the macroeconomic impact of key competition policy decisions taken by the

Commission. Its two main objectives were: (1) to complement the customer savings

results reported in the Annual Activity Report of DG Competition, which only give a

partial view of the impact of competition policy; and (2) to obtain better evidence on the

broader impact of competition policy to be used for competition policy advocacy to a

wider public.

In the area of merger control, a study carried out for DG Competition gave a systematic

review of ex-post evaluations of the impact of merger decisions taken by EU

competition authorities. Its objectives were to replicate a study done on US merger

control and to evaluate the quality of merger decisions by EU competition authorities. A

recent study in the area of State aids analysed whether EU restructuring aids contribute

to restoring the long term viability of the aided firms. Such studies should contribute to

improving the regulatory framework for competition policy enforcement in the EU.

Another evaluation activity should be seen within the context of the State Aid

Modernisation implemented by the Commission in 2014. Member States have been

requested to conduct ex-post evaluations of large aid schemes to determine their impacts

on markets and competition. They are required to submit evaluation plans allowing

assessing whether the schemes have had the expected effects. DG Competition assesses

these plans and on that basis decides whether to prolong approval of the scheme. This

evaluation activity is different in nature from the others and aims at developing an

evaluation culture in the EU Member States.

Finally, DG Competition commissioned external evaluations to measure the impact of

its interventions at the sector level. A first pilot study concerned the energy sector. It

combined different methodological approaches (such as a broad econometric analysis of

the impact of competition policy interventions on competition and productivity and case

5

studies analysis using the DiD method) to assess the impact of our interventions on

sector performance. The objectives of this evaluation were to improve our knowledge of

methodologies and data requirements for this type of analysis and to get well-founded

evidence of the impact of specific competition policy interventions in the energy sector.

A second sector study evaluated competition policy interventions in the telecoms sector,

using a case study approach.

2.2 Lessons learnt

There are already some lessons to be learnt from DG Competition's experience with ex-

post evaluations.

First, an evaluation culture has been developing in international organisations (not only

in the European Commission but also the OECD and the World Bank) and EU Member

States in support of the better regulation agenda as well as a more effective and efficient

use of public resources. This not only concerns competition policy but also other policy

domains. In the area of competition policy, the US FTC and the UK competition

authorities have been the frontrunners.5 Nevertheless, a recent evaluation of the FTC's

Bureau of Economics6

recommends that "the Bureau of Economics should consider

dedicating more research and analysis to support the Bureau of Competition and the

Bureau of Consumer Protection in assessing the impact of FTC actions on consumers

(e.g., conduct and publish more retrospectives), apart from litigation support."

Second, there is a broad agreement that competition policy agency activities can be

informed and improved by a careful examination of past practices. Ex-post evaluation

of competition policy enforcement can also contribute to provide evidence about the

impact of the actions taken by competition authorities, which is essential in the

communication to the wider public. Nevertheless, it is not always clear what is the

added value of ex-post evaluations for the day-to-day work. How does a better

understanding of the impact of our merger decisions affect the future decision making

process? As discussed in the next section, the DiD approach generally used to carry out

quantitative ex-post evaluations of individual merger decisions is not necessarily

sufficient to inform merger enforcement.

Third, ex-post evaluations are complex and require time and resources. In a presentation

made at the second workshop organised by the OECD on the ex-post evaluation of

competition authorities' enforcement decisions, Daniel Hosken of the FTC, looking

back at the use of empirical analysis in US merger reviews, highlighted that such

analyses are expensive, relying not only on the availability of well-trained empirical

economists but also on "big data" infrastructure for the storage and security of data. The

5 See OECD (2012), "Evaluation of competition enforcement and advocacy activities: the results of

an OEDC survey", DAF/COMP/WP2(2012)7/FINAL. 6 See Federal Trade Commission (2015), "Evaluation of the Federal Trade Commission's Bureau of

Economics", Office of Inspection General Report n°15-03,

https://www.ftc.gov/system/files/documents/reports/evaluation-ftc-bureau-

economics/150630beevaluation.pdf

6

direct costs of performing an ex-post evaluation primarily relate to human resources

requirements if it is done internally and the study budget if the evaluation is outsourced.

If the ex-post evaluation is done internally, it would thus divert resources from

enforcement activities in the short term but could contribute to increase the efficiency

and effectiveness of competition authorities in the medium term. Even if ex-post

evaluations are outsourced, it is important to devote sufficient internal resources to the

in-house management of the study project and the analytical support of the study team.

Fourth, access to good data is essential for quantitative evaluations. This should be a

criterion for the selection of the cases to be evaluated and a budget for data acquisition

should be included in the offer made by consultants. Ideally, the collection of data to be

used for ex-post evaluation purposes could already commence at the time of the

decision. This would imply that the choice of cases to be subjected to an ex-post

evaluation should be made at a very early stage. Moreover, a sufficient time-gap

between the competition policy intervention and the analysis should be observed, which

also affects the choice of cases to be evaluated. This time gap depends on the type of

intervention being evaluated, on the effects being analysed and on the methodology

used. It is important to find the right balance between the time needed for the effects to

manifest themselves (in particularly efficiency driven effects) and the risk of new events

occurring and making the identification of these effects difficult.

Fifth, a sufficient interest and participation of key stakeholders is important, in

particular for qualitative studies including interviews or surveys. Stakeholders are more

likely to respond if the evaluation is done in-house and credible commitments of

confidentiality are made (as illustrated by the high response rate in the European

Commission's 2005 Merger Remedies Study7).

Sixth, independence of the evaluations should be ensured, ideally by not entrusting the

very unit that has prepared the case decision or developed the policy with carrying out

an evaluation of the respective decision or policy. Involvement of academic experts at

an early stage of evaluations carried out in-house by competition authorities is an

example of best practice. For example, the use of external referees providing comments

on the in-house evaluation of telecoms mergers contributed to improve the quality of the

analysis. Moreover, the sharing of experience with other competition authorities and

international organisations (e.g. via the presentation of the evaluations in workshops

organised by the OECD) has proven to be very valuable.

Finally, it is important to recognise the technical limitations of the different quantitative

analyses, such as the difficulty to distinguish the impact of remedies from that of

mergers and the methodological challenge of selecting an appropriate control group in

counterfactual impact analyses using the DiD method. Technical issues may have an

adverse impact of the reliability of the results of ex-post evaluations of past enforcement

7 Merger Remedies Study, Public version (2005), DG COMP, European Commission, October,

http://ec.europa.eu/competition/mergers/studies_reports/studies_reports.html.

7

decisions, which needs to be properly taken into account when interpreting the results.

These issues are described in more detail below. It is also not obvious how results of

retrospectives on the effects of past decisions can be translated into recommendations

for future enforcement decisions. This is the reason why the use of descriptive and

qualitative analyses can usefully complement quantitative ones and also contribute to

increase the robustness of results.

3. ALTERNATIVE APPROACHES TO EX-POST EVALUATION OF INDIVIDUAL

DECISIONS

This section starts by briefly describing the different approaches which have most

commonly been used to carry out ex-post evaluations of individual decisions.8 It also

presents a more qualitative and descriptive approach which consists in reviewing

whether certain market conditions used to support the decision have developed as

predicted. This approach has been used in the past by competition authorities in the

Canada, New Zealand and the UK, and their experience with this approach based on a

'review of market conditions' will be considered in more detail.

3.1 Description of different approaches

In this note, a distinction is made between two main approaches which can be used to

analyse individual decisions. The first one includes sophisticated quantitative analyses

such as market structure based analyses (i.e. structural model simulations) and

counterfactual impact analyses (including in particular the DiD methodology). The

second one comprises qualitative and descriptive analyses based on desk research,

surveys and interviews. As explained below, the type of issues which can be addressed

by these two approaches is different: more sophisticated quantitative methodologies aim

at identifying the causal effect of a decision while the qualitative and descriptive

methodologies are rather used to analyse market trends post-decision and to check

whether key assumptions and predictions taken at the time of the decision are verified

ex-post.

3.1.1 Quantitative methodologies

Ex-post evaluations have been most extensively used to assess the impact of horizontal

merger decisions. These evaluations aim at measuring the impact of the merger decision

on price levels. The standard methodology would compare the price before and after the

merger in markets where there is overlap of production or sales between the merging

parties. The difficulty is that other factors (changes in demand or costs, e.g.) may affect

price developments as well and the main challenge is to isolate the impact of the merger

from that of these other factors. There are two main quantitative techniques which have

8 A detailed description of these approaches can be found in Ilzkovitz and Dierx (2015), see Footnote

2, or in OECD (2016), "Reference guide on ex-post evaluation of competition agencies' enforcement

decisions", http://www.oecd.org/daf/competition/reference-guide-on-ex-post-evaluation-of-

enforcement-decisions.htm.

8

been used to address this issue: structural model simulations and the Difference-in-

Differences methodology.

Structural model simulations

Structural models are mainly utilised to predict ex-ante the effects of proposed mergers

but it can be adapted to assess ex-post the impact of merger decisions under alternative

scenarios. This methodology consists in building an economic model describing the

underlying demand and cost/supply forces in the industry and in simulating with this

model prices and quantities in a counterfactual scenario describing a situation without

the merger. As this method requires many data and is based on relatively sophisticated

econometrics, it has been rarely used for ex-post evaluations.

Difference-in-Differences (DiD) methodology

The most widely used methodology to perform ex-post evaluations of individual

decisions is the DiD methodology. This methodology avoids the need for a full

specification of demand and supply functions. Instead, it uses data from actual

transactions (or time periods) not affected by the decision to construct a counterfactual

scenario and compare it with actual market developments. It is called a DiD approach

because it compares the changes in the affected markets before and after the decision

(1st difference) with the changes that took place over the same period in a comparable

market not influenced by the decision (2nd

difference) (see Figure 1).

DiD studies are very popular because they can generally lead to reliable and statistically

robust estimates of the impact of a decision. Their main advantage is that they allow

identifying a causal relationship between a specific intervention of a competition

authority and market developments. Whereas the DiD framework is most often used to

assess the price effects of mergers, the same approach could also be used to investigate

the effect of a merger on other variables such as investments, product variety, product

quality etc. if suitable data are available. However, they also have serious of drawbacks

(see Box 1). They are sensitive to the underlying assumptions and require the definition

of an appropriate counterfactual and a good set of control variables. Ormosi et al.

(2015)9 and Werden (2015) stress the fact that DiD approach may lead to biased

estimates if the "underlying assumptions are not fully understood and mastered".

Moreover, reliable results come at the cost of increased data demands, which is why

sometimes DiD studies are difficult to realise in practice.

9 Ormosi P., F. Mariuzzo and R. Havell (2015), A review of merger decisions in the EU: What can we

learn from ex-post publications", European Commission, DG Competition, ISBN 978-92-79-51935-

2, http://ec.europa.eu/competition/publications/reports/kd0115715enn.pdf.

9

Figure 1: DiD methodology

Box 1: Drawbacks of DiD estimations in merger retrospectives

Ormosi et al. (2015) and Werden (2015) identify the following limitations of DiD estimations:

- The validity of the estimates is highly dependent on the quality of the control group: the effect of the

treatment should not be allowed to spill over into the control group and the only difference between

the two groups should be the treatment.

- DiD estimates of merger's price effect are unbiased only if the following conditions hold:

The time trends of prices should be the same for the treatment and the control group (parallel

time trends hypothesis);

Prices in the treatment and control groups should respond in the same way to supply and demand

variables;

Treatment and control markets should not differ in terms of degree of competition and degree of

substitutability among differentiated products; and

There should not be serial correlation in the price data.

- The decision to merge should be exogenous and not be affected by the variables influencing the

price. Sources of endogeneity can be omitted variables or reversed causality between the price and

the merger.

- Performing a DiD study allows for very different choices (concerning data sources, price measures

and time periods of the analysis, e.g.). As a consequence the same data can actually lead to very

different results according to the choices made.

- A merger retrospective could substantially mislead if the merger effects are complex. In fact, a

merger does not only produce price effects and could also affect the quality and range of products

available to consumers. A DiD study only analyses the price effects.

- A merger retrospective is likely to estimate only short-term merger effects, but a merger might have

effects that become clear only in the long run, simply because the process of merging two firms

might take years.

The DiD methodology is based on a

comparison of developments over time

between the treatment group

(affected by the decision) and the

control group. The improvement in

performance (i.e. the price reduction)

of the control group (A-D) is subtracted

from the improvement in performance

of the treatment group (B-C) to arrive

at the estimated impact of the

competition policy enforcement

measure ([B-C] - [A-D]).

10

3.1.2 Descriptive and qualitative methodologies

When limited quantitative data are available10, ex post evaluation can only be carried

out using descriptive, i.e. based on desk research using publicly available data, or

qualitative methods, i.e. interviews or surveys with the parties concerned, their

competitors, suppliers and customers. The range of respondents could occasionally be

widened to include lawyers and consultants advising the parties. Unfortunately, surveys

and peer reviews are by nature subjective measures of the effects of competition policy

enforcement activities and they may be manipulated by respondents for strategic

purposes. This is the reason why, ideally, surveys and peer reviews should be

complemented by quantitative assessments or the judgement of independent experts.

Moreover, when a descriptive or qualitative approach is used, the assessments can often

only provide an indication of the sign of the effects of a decision on market variables

and there is less certainty about the causal relationship between the decisions taken and

the effects identified. In order to make a correct assessment survey respondents would

need to form expectations on what would have happened in the absence of the merger

which can be very challenging. This is the reason why a reflection about the

counterfactual cannot be completely ignored even when using descriptive and

qualitative evaluation techniques.

On the other hand, a descriptive and qualitative approach has the advantage that it can

help to assess the impact of a decision on a broad range of variables, including those

that cannot be easily quantified. While the assessment of the price effect of a merger

would preferably be made using quantitative techniques, descriptive and qualitative

tools may be well suited to further corroborate quantitative results and provide the

context in which these effects take place11. A descriptive and qualitative assessment can

also be employed to test if key assumptions about market characteristics (e.g., take-up

of new technologies) and ex-ante expectations about market developments (e.g., entry

of new competitors, level of countervailing power of buyers) are well founded. An

approach based on a review of market conditions does not aim at directly assessing the

impact of the decision taken but rather at examining whether anticipated market

developments that were key to the decision took place as predicted or not. Such an

assessment can help to identify pitfalls, limitations and sometimes errors in the analysis

that underpinned decisions taken and can lead to concrete recommendations to improve

10

This can be due to several reasons: parties affected by the decision may not wish to share the data,

the data have not been collected and kept by the market players, the data are not available with a

sufficient level of disaggregation, the variables are difficult to quantify (e.g. on product quality), etc. 11

For example, the ex-post DiD evaluation of the abuse of dominance of the Polish telecommunication

operator showed that the abuse delayed the internet penetration on Polish markets. However, it could

not identify a significant effect of the abuse on broadband prices. One reason might be that the

countries in the control group might not represent fully competitive benchmarks. Another possible

explanation emerged from the descriptive and qualitative analysis, which showed that – in response

to the difficulties of obtaining access to the incumbent's DSL broadband network – alternative

operators had decided to invest in superior technologies (fibre, e.g.), which may have limited the

effect of the abuse on broadband prices.

11

future decisions. Over the recent years, such an approach has been advocated by Dennis

Carlton and Gregory Werden, both well-known economists who have held high

positions at the US DoJ.

3.2 Debate about the scope of ex-post evaluations

Ex-post evaluations of individual decisions have two main objectives: (1) to determine

the impact of enforcement decisions taken; and (2) to gather insights that can be used to

improve future enforcement. The achievement of the latter objective is particularly

challenging. Werden (2015)12 is doubtful that evaluations relying on quantitative

econometric analysis only can contribute to improved merger enforcement. He argues

inter alia that the number of merger retrospectives is still far too small and subject to

sampling error, making it impossible to draw any statistical inference on which market

characteristics are correlated with price effects. He considers that quantitative studies

that focus on the effects of consummated mergers cannot be used to test the efficacy of

the assessment process or to provide useful guidance to aid merger enforcement. For

these reasons, he suggests combining such quantitative analyses with a detailed case

study approach to test whether post-merger competition developments conform to the

competition authority's predictions at the time the decision was taken.

Similar to Werden, Carlton13 considers that merger retrospectives should also study of

the accuracy of predictions made by various methods and tools used by the agencies:

"For each merger that the agencies review closely…, they should record which

analytical tools they employed and what predictions they reached with each tool. Then,

for those mergers that are consummated, the antitrust agencies should undertake

retrospective reviews of actual market-place outcomes in comparisons with those

predictions. Only in that way – by combining a record of what tools were used and what

conclusions were drawn from each tool with a study of observed outcomes from

mergers – can systematic evidence on the efficacy of various methods used in merger

reviews".

Carlton regards merger retrospectives that only ask whether prices went up post-merger

as poor guides for analysing merger policy. A fundamental question facing enforcement

officials is whether their current policy is too lax or too stringent and this question is

different from whether a particular merger decision was correct. Carlton advocates

meta-studies covering a large number of mergers as a single retrospective study of an

individual merger would tell very little about the soundness of the agency's policy. He

also believes that a merger evaluation should combine pre- and post-merger data to be

able to compare the actual developments in the market with the agency's predictions

made at the time of the merger. These data should not only cover price developments

12

Werden, G.J. (2015), "Inconvenient truths on merger retrospectives studies", Journal of Antitrust

Enforcement, 3(2), 287-301. 13

See Carlton, D.W. (2010), "Revising the Horizontal Merger Guidelines", Journal of Competition

Law and Economics, 6(3), 619–652 and Carlton, D.W. (2009), "Why we need to measure the effect

of merger policy and how to do it", NBER Working Paper 14719.

12

but also other variables considered in the merger investigation, such as the likelihood of

entry, product repositioning, buyer power, … In order to be able to perform this

analysis, he suggests that at the end of each in-depth merger investigation, the

competition authority should fill out a data sheet summarising the main elements

playing a role in the decision, including data on price, entry, product characteristics and

predictions made by the authority. The study could then compare pre- and post- merger

data with the predictions of the authority.

However, the type of analysis proposed by Werden and Carlton is not straightforward.

When a retrospective study finds that what happened after a merger was not what the

authority had expected, the study should try to investigate and identify the sources of

error. Different types of error sources can be been distinguished14: faulty facts,

erroneous analysis, dominance of non-price effects and random variation.

Let us consider a merger found to have resulted in significant price increases although

the responsible competition authority had considered such adverse effects to be unlikely

on the basis of the argument that market entry would occur15. The agency could have

observed faulty facts by relying on too optimistic statements of a potential entrant. It

could have made an erroneous economic analysis by misjudging the impact of the

merger on the probability of entry or by underestimating the price increase of the

merger due to the misuse of a tool or to a bad interpretation of results. Finally, random

variation could have affected entry, for example, if a firm planning to enter had been

forced by a financial setback to abandon its plans. Analysing the sources of errors could

be very informative for future merger decisions. For example, if the error was found to

be due to faulty facts, the competition authority could identify ways to improve its fact

finding. If the mistake was due to the inappropriate application of an analytical tool, the

authority could consider consulting external experts on how to best apply state of the art

methodologies.

To conclude, this section suggests that a qualitative analysis examining whether the key

assumptions and predictions made by the competition authority proved correct ex-post

can be very informative for future decisions and may usefully complement the more

quantitative analyses currently used to estimate the effects of merger or antitrust

decisions. However such a qualitative analysis, which could in principle rely on simpler

methodologies, is not necessarily easy to implement if the sources of the errors made by

the competition authority have to be identified and investigated.

14

See Ormosi et al. (2015), opcit in Footnote 14 and Werden (2015), opcit in Footnote 16. 15

This is what happened in the merger between AstraZeneca Tica (AZT) and GlaxoSmithKline (GSK).

This merger was unconditionally cleared by the Swedish competition authority despite the fact that

it was estimated to significantly increase prices. In this case, the competition authority predicted that

the coming deregulation of the pharmaceutical industry would encourage new entry and

competition, which in the end did not happen.

13

3.3 Testing key assumptions and predictions underpinning merger decisions

Competition authorities in three countries (Canada, UK and New Zealand) have done

studies which systematically test key assumptions and expectations underpinning their

decisions. While their more descriptive and qualitative approach has only been used

thus far in the area of mergers, it could also be considered in the area of antitrust. This

section gives an overview of the experience of these three competition authorities.

Summary tables with more details about this experience can be found in Annex C.

Canadian Bureau of Competition (2007)

The Canadian Bureau of Competition asked CRA to undertake a review of three merger

decisions16: the Corus-WIC merger between two companies of commercial radio

broadcasting in 2000, the Fording merger between a number of coal companies in 2003

and the Carmeuse-Lafarge merger between two large suppliers of concrete and other

building materials (1999). These three mergers were considered to have occurred

sufficiently long before the review to allow the effects of the merger to materialise (a

minimum of three years before the review).

The three mergers selected were cleared by the Bureau despite the fact that they raised

significant competition concerns at the time of the merger investigation. However, these

concerns were considered as not substantial enough to justify remedies. The main

objective of the review was to determine whether the Bureau's expectations about

market conditions after the merger were borne out or not. The review was based on

information included in the case file, interviews with market participants, mainly

customers as merged parties and rivals were reluctant to participate in the review, and

information from other public sources. The focus of the review was on three issues: a

general comparison of outcomes versus predictions, entry and countervailing market

power of buyers.

The main conclusions of this review were that the predictions of future market

conditions had been quite accurate, given the information available at the time of the

merger investigation, that the economic reasoning was sensible and that there were no

indications of significant errors of logic. However, the review identified areas where

incremental improvements might be made. In particular, the Bureau had underestimated

the likelihood of entry when taking its decisions and it had underestimated the

countervailing market power of buyers. Finally, the review concluded that when dealing

with a merger where the economic aspects of the case are complex, it is productive to

review the most recent academic research and examine its relevance for the case.

16

See Neumann M. and M. Sanderson (2007), "Ex Post Merger Review: An Evaluation of Three

Competition Bureau Merger Assessments", Final Report of CRA, August.

14

UK competition authorities (2009)

A similar study was performed by Deloitte in collaboration with Stephen Davies for the

Office of Fair Trading (OFT), the Competition Commission (CC) and the Department

for Business, Enterprise and Regulatory Reform of the UK in 2009.17 The aim of the

study was to review eight merger decisions in the light of subsequent market

developments and to comment on the approaches taken by the OFT and the CC. The

reviewed mergers concerned decisions from the period 2004 to 2006. As was the case in

the study conducted for the Canadian Bureau of Competition, a minimum period of

three years had passed before the review of the decision. Three decisions had been

resolved at the OFT stage and five had been referred to the CC. The decisions included

four unconditional clearances, three conditional clearances and one prohibition.

The review was based on interviews with market participants relevant to the merger

decisions, written questionnaires to the merging parties and selected other market

participants, and desk research of publicly available information. In addition, a merger

simulation exercise was carried out to assess the price impact of four of the eight

mergers considered.

Overall, the review suggested that the analyses of the OFT and CC were broadly

consistent, coherent and transparent. The authorities were able to "ask the right

questions" and develop sensible theories of harm, even though the CC had a tendency to

overstate barriers to entry. The review also expressed some doubts concerning the

ability of the UK competition authorities to collect all data required for merger

simulations, such as data allowing the calculation of the market elasticity of demand

and price-cost margins at the product level.

Finally, the review concluded that subsequent market developments did not raise any

substantial doubts about the overall soundness of the decisions taken except in two

cases: an unconditional clearance decision in the corrugated cardboard packaging sector

and a conditional clearance decision in the rail freight maintenance services sector. In

the former, there were a series of price rises in one of the affected markets and many

customers considered these to be attributable to the merger. In the latter, there was some

doubt about the CC's assessment that the merger would (in the absence of a remedy)

cause a substantial lessening of competition.

New Zealand Commerce Commission (2014-2015)

Up to 2014, the New Zealand competition authority, the Commerce Commission,

reviewed a single merger clearance decision every year using a qualitative approach.

The main objective of this review, largely based on interviews with market participants,

was to determine whether the right decision had been taken in merger cases that had

17

Deloitte (2009), "Review of merger decisions under the Enterprise Act 2002", Report prepared for

the Competition Commission, Office of Fair Trading and the Department for Business, Enterprise

and Regulatory Reform, 18 March.

15

raised significant competition concerns. However, in 2014 the Commerce Commission

decided to change its approach to ex-post merger evaluations. The main reasons given

for this change of approach were (i) the empirical challenges of an accurate assessment

of the impact of the merger (due inter alia to a paucity of data); and (ii) the limited

general applicability of the findings: "A demonstration that prices did in fact rise and

that that increase can be attributed to the merger may tell us something about the

correctness of the decision in that transaction but not about decisions in other

transactions more generally".18

The goal of the new approach implemented from 2014 on was: (1) to review whether

certain market conditions had developed as predicted; and (2) to refine and improve the

tools and techniques used for making predictions about post-merger market conditions

such as entry, competitive constraints exercised by competitors in the market at the time

of the merger, impact of divestitures and countervailing market power of buyers. This

more focussed approach made it possible to increase the number of mergers reviewed.

Two exercises of this type were carried out: one in 2014 covering ten mergers having

taken place between May 2008 and June 2011 and one in 2015 covering eight mergers

consummated in the period July 2011 – July 2013. The mergers reviewed concerned a

variety of sectors including pharmaceuticals, food products and electronic equipment.

The review was based on internet research and phone interviews with the merged

entities and their customers and competitors.

The review showed that, overall, the Commerce Commission had been right about

future developments of the market. More precisely, entry had occurred in 12 out of the

17 markets in which the Commission had relied on it to discipline any substantial

lessening of competition and existing competitors had continued to exercise a

competitive constraint in 18 of the 20 markets where their role had been a key reason

for clearance. In the two clearance decisions where a business line was divested as part

of the merger, the divested businesses had managed to maintain their market share (in

comparison with the period when the businesses were still owned by the relevant

merging parties). Finally, in the one decision relying on buyer countervailing market

power, no entry had taken place but the buyer identified as able to sponsor entry had

indicated that it would encourage entry if prices were to become too high. A summary

of these results is given in Table 1 below.

The main lessons drawn by the Commerce Commission from this review are the

following:

When relying on imports as a source of competition, greater attention should be

paid to market conditions in the exporting markets and to possible variations in

18

See Csorgo L. and H. Chitale (2015), "Targeted ex post evaluations in a data poor world", OECD

DAF/COMP/WP2(2015)10.

16

these conditions. Factors to be taken into account could include the process by

which export decisions are made or the certainty of supply.

The competitive constraint of imports is not independent of the exchange rate.

Therefore, the import assessment should determine the range of exchange rates

over which imports by foreign companies are likely to occur.

When the merger results in a duopoly, greater consideration should be given to

the possibility of market exit of one of the two remaining players.

When considering entry, particularly by multinational companies, the

opportunity costs influencing investment decisions to enter New Zealand and the

circumstances in which these investment decisions are likely to be reconsidered

should be analysed.

As the review tends to confirm that markets with low sunk costs have a higher

likelihood of entry, it is vital to assess the degree to which entry costs are sunk.

It should be noted that some of these conclusions are particular to a small, open

economy such as that of New Zealand.

Table 1: Summary of the review of 18 merger clearance decisions entailing 40

markets over the period May 2008-July 2013

Key reason for clearance Number of markets for

which this was a key reason

for clearance

Number of markets for

which the predictions was

right

Entry 17 12

Competitive constraints

exercised by existing

competitors

20 18

Divestment became a

competitive alternative

2 2

Ability of buyers to sponsor

entry

1 1

Source: Csorgo and Chitale (2015)

4. PROS AND CONS OF THE TWO APPROACHES

As explained above, carrying out ex-post evaluation studies of competition authorities'

decisions is very useful, either to be more accountable and transparent towards the

different categories of stakeholders or to improve the design of future interventions. The

quantitative methods commonly used for such ex-post evaluations can be difficult and

costly to implement. Conducting a DiD study or simply trying to assess the effects of a

decision on the level of competition in a given market usually requires a lot of data.

17

Once data have been obtained, indicators of competition and market outcomes need to

be calculated or estimated, which often is no easy task. As set out in Section 3, studies

of this sort do not lead to conclusions which can be easily generalised and do not

necessarily provide very concrete recommendations helpful for future decision making.

However, if the study's objective is to obtain a sound and reliable analysis of the impact

of an individual competition decision, a quantitative analysis may be the only option.

Nevertheless, to draw lessons for future decision making, it may also be worthwhile to

analyse whether market developments following decisions taken are in line with what

was expected ex-ante, as has been done by the Canadian, UK and New Zealand

competition authorities. This alternative, more descriptive and qualitative approach,

entails some advantages:

First, the data requirements for this kind of analysis are less demanding than that

for a DiD study: the only requirement is to investigate market developments

post-merger to check whether they are consistent with the predictions made at

the time of the decision. Nevertheless, extra work is implied since case teams do

not normally continue tracking market developments after their cases have been

closed. Also, the identification of the predictions made at the time of the

decision is not necessarily straightforward.

Second, this type of analysis necessitates less technical expertise and could be

more easily done in-house. This would allow developing the expertise of case

teams in the area of ex-post evaluation on the basis of learning-by-doing.

Third, the application of this approach (depending on the scope of analysis

envisaged) might be more straightforward and require fewer resources than DiD

method used to estimate the effects of past decisions.

Nonetheless, there are some drawbacks to be taken into account:

In particular, despite the lower amount of resources that this kind of study might

require, it would still be necessary to free up the necessary resources. Ideally, a

team combining experts of the selected cases and economists should be

composed.

If it is decided to have structured interviews with the parties involved and other

market participants, the preparation of these interviews may be resource-

intensive. Also, it remains to be seen whether the parties and the other market

participants would be ready to respond to our inquiries. Moreover, replies may

be subjective and results may not be reliable and precise enough to draw

meaningful conclusions. Finally, conducting reliable interviews would impose

having a reflection about the counterfactual in order to avoid wrong conclusions

(see Section 3.1.2). For example, simply asking interviewees whether prices

have declined post-merger would wrongly lead to the conclusion that the merger

did not lead to a price decrease. It is necessary to investigate whether other post-

merger market developments did not contribute to the price decline and to

include these considerations in the questions discussed during the interview.

18

Finally, if the study would like to investigate the reasons why the competition

authority's ex-ante expectations did not materialise ex-post, it could become

much more complex.

In light of the increased importance of ex-post evaluation for policy making, it might be

useful to launch a pilot exercise to test the feasibility of a study using this alternative,

more qualitative approach.

Data gathering is one of the first issues to be tackled in such a pilot study. In particular,

before launching the project, the team involved should have a clear idea of where to find

the data necessary to perform the analysis. Having access to the comprehensiveness

case file related to the decision would be crucial to identify the key assumptions made at

the time of the decision. In addition, the experience of the three competition authorities

having implemented this type of approach shows that information from public sources

and market participants are key elements as well.

Another important issue is the selection of the decisions to be reviewed and the topics to

be analysed. Starting with merger decisions would allow benefiting from the existing

experience of other competition authorities. There are three possible options for the

selection of cases:

The first would be to consider one complex merger decision and complement the

quantitative analysis to make an in-depth assessment of whether all key

assumptions made and market developments anticipated at the time of the

decision are consistent with observed developments in the markets concerned.

The second would be to take a relatively limited number of decisions concerning

a selected sector (at most five) and to analyse a few issues on which these

decisions have been based. It would be interesting to select decisions for which a

quantitative analysis has already been made. The telecommunications industry,

for example, would be a good candidate as we already have access to four

quantitative ex-post evaluations19

. However, developments in this sector are

very complex and not necessarily representative of developments in other

sectors. A similar approach combining qualitative and quantitative analyses has

been used by the UK competition authorities.

The third option would be to consider a larger number (at least ten) of decisions

concerning different sectors but to focus on one or two key issues (like entry or

the role of existing competitors).

The benefit of Options 2 and 3 is that they offer a simpler analysis applied to a larger

number of cases, which could provide on average (based on the law of large numbers)

an appropriate description of the ability of competition authorities to predict key market

19

These are: the merger between T-Mobile and Tele.ring in Austria, the merger between Hutchison 3G

Austria and Orange Austria, the merger between T-Mobile and Orange in the Netherlands, and the

merger between T-Mobile and Orange in the United Kingdom.

19

developments. Option 2 would present the advantage of allowing the selection of a

sector for which a simpler analysis is more warranted. It also would permit a more in-

depth analysis of sector-specific issues. Option 3 – more in line with the approach used

by the New Zealand competition authority –would rather provide a snapshot analysing

the extent to which competition authorities' expectations in terms of one or two key

issues in its merger decisions were realised. If Option 1 were to be selected, it would be

better to combine the qualitative and descriptive approach with a traditional

(Difference-in Differences) estimation of the merger effects as, in this case, the

descriptive and qualitative analysis should be seen as supporting the more elaborated

quantitative approach.

When selecting the decisions to be reviewed, it is important to keep in mind that a

minimum gap of three years and a maximum gap of around 10 years should be observed

between the date of the competition policy intervention and the date of the ex-post

review. The time gap should be sufficiently large to let the relevant changes in the

market appear but not too large for fear that the impact of the decision would no longer

be observable. Moreover, it should be noted that the cases selected by the three

competition authorities were mostly borderline cases, such as cleared mergers raising

competition concerns.

5. CONCLUSIONS

This paper proposes continuing the quantitative analysis of the impact of individual

merger decisions. However, it suggests that it might be worthwhile to explore as well an

alternative, more descriptive and qualitative approach, to the ex-post evaluation of

individual competition policy decisions. The main reason to develop this alternative

approach is that while the DiD analysis proved useful to defend the legitimacy of

competition decisions, it was less beneficial in providing more concrete

recommendations to improve these decisions. This more descriptive and qualitative

approach would consist in determining whether market developments anticipated at the

time of the decision and which were key to the decision did in fact take place. Such an

approach would act a complement the more widely used quantitative ex-post evaluation

methods. The paper suggests testing the feasibility and usefulness of this approach in a

pilot study of merger decisions.

20

ANNEX

DETAILS ON THE STUDIES TESTING KEY ASSUMPTIONS IN

MERGER DECISIONS

1. COMPETITION BUREAU OF CANADA (2007)

Period analysed Mergers took place more than three years before the study (2007)

Internal or

outsourced study

Outsourced to CRA

N° of mergers

considered

3 merger decisions:

- Corus-WIC (2000)

- Fording (2003)

- Carmeuse-Lafarge (1999)

Sector concerned - The Corus-WIC merger is a merger between two companies active

in commercial radio broadcasting.

- The Fording merger is a merger between a number of coal

companies.

- Carmeuse-Lafarge is a joint venture between two large suppliers of

concrete and other building materials.

Types of decisions

selected

- Cleared mergers raising complex and significant competition issues

at the time of the review, but not substantial enough to justify

remedies.

- Mergers occurring sufficiently long ago to allow effects of the

merger to materialise (minimum three years before the review).

Main conclusions

from the Bureau

decisions

- Regarding the Corus-WIC merger, the bureau concluded that the

merger would lead to a significant increase in market shares in some

markets. However, other factors like entry attenuated the

competition concerns.

- In the Fording merger, the Bureau concluded that, given that there

appeared to be reasonably close substitution possibilities with US

coal suppliers, there was not sufficient evidence to challenge the

merger.

- The Bureau decided not to challenge the Carmeuse-Lafarge merger,

because it concluded that the combination of alternative US

suppliers and the countervailing market power of large buyers was

sufficient to maintain competition in the market.

Issues analysed CRA undertook an independent review of the Bureau's past merger

assessment process with the following goals:

- To determine if the Bureau's expectations about market

conditions after the merger were borne out;

- To evaluate the Bureau's merger investigation process in light of

this ex-post merger review and recommend improvements.

CRA mostly focused on a general comparison between outcomes and

predictions, entry and countervailing market power of buyers.

Methodology used - Interviews of market participants (which were mainly customers as

merged parties and rivals were reluctant to participate in the study)

- Information from other public sources was used to determine post-

merger market conditions

- Comparison of these market conditions with the predictions made by

the Bureau.

Main conclusions - The study considered the predictions of future market developments

made by the Bureau to be quite accurate, given the information

available at the time of the review. For example, in the Fording case,

21

the significant growth in demand in Asia was impossible to predict.

- The economic reasoning employed in the three cases was sensible

and there were no indications of significant errors in logic.

Lessons learned - Entry: In the Corus-WIC and Fording cases, the Bureau correctly

identified the likelihood of entry but underestimated its actual

impact. In the Corus-WIC case, it would have been useful to

investigate further the possible impact of the change in the Canadian

Radio-Television and Telecommunication Commission (CRCT)

policy. Interviews of past entrants and a review of CRTC decisions

would have been helpful in determining just how important the

relaxation of the CRTC entry policy would be. In the Fording case,

the rise in entry would have been more difficult to predict as it was

due to the rising coal prices induced by surging demand in China.

- Countervailing market power: When considering the importance of

countervailing market power of buyers, the size of the buyers is not

the only variable to consider. Another important factor to consider is

the availability of substitute sources of supply. This element could

have been further investigated, especially in the Fording case.

- Other issues: When dealing with a merger where the economic

aspects of the case are complex, it is productive to review the most

recent academic research and examine its relevance for the case.

22

2. UK COMPETITION AUTHORITIES (2009)

Period analysed 2004 to 2006

N° of mergers considered Eight merger decisions.

Internal or outsourced study Study outsourced to Deloitte and Steven Davies (for the

merger simulations)

Sector The eight transactions included companies in different

sectors, like healthcare, soft drinks or processed food

Types of decisions selected Three of the eight cases selected were resolved at the OFT

stage and five were referred to the CC. The cases analysed

included four unconditional clearances, three conditional

clearances and one prohibition.

Issues analysed For each of the selected cases the study considered:

- whether the competition authorities' decisions were

sound, given the information that was available at the

time of the decision;

- whether the reasoning behind the decisions was clear,

consistent and supported by evidence; and

- whether the competition authorities' predictions about

future market developments were borne out.

Methodology used - Interviews with market participants relevant to the

merger decisions, written questionnaires to the merger

parties and selected other market participants and desk

research based on publicly available information.

- In addition, a merger simulation exercise was carried out

Main conclusions Analysis of the OFT and CC: the study suggested that the

analyses of the OFT and CC were generally consistent,

coherent and transparent. The authorities were able to "ask

the right questions" and they had developed sensible theories

of harm.

Lessons learned - Soundness of decisions: The conclusion of the review

was that subsequent market developments had not raised

any substantial doubts about the overall soundness of the

decisions in six out of eight cases. Following the

unconditional clearance of the DS Smith/LINPAC

merger in the corrugated cardboard packaging sector,

there was a series of price rises in one of the affected

markets (sheet) and many customers (48% of those

surveyed) considered these to be attributable to the

merger. The conditional clearance of the merger between

English Welsh and Scottish railway Limited and Railway

Holdings Marcroft Engineering in the sector of rail

freight maintenance services had given rise to doubts

about the CC's assessment that the merger would (in the

absence of a remedy) result in a substantial lessening of

competition. For example, there was evidence to suggest

that self-supply should have been included in the

maintenance market, which would have imposed a

stronger competition constraint. Moreover, barriers to

entry and expansion may have been overstated.

- Entry: in the three cases decided by the OFT, market

entry had been in line with what was anticipated by the

OFT at the time of its decisions. However, in three of the

23

five cases referred to CC there was significant actual or

planned entry which has not been anticipated by the CC.

Consequently, there may have been a tendency for the

CC to overstate barriers to entry.

- Self-supply: A different assessment as to whether to

include self-supply in the product market was made in

two cases and in one case (the EWS/Marcroft merger),

there was some evidence that in-house supply should

have been considered as part of the market (see above).

Therefore, the review recommended including some

guidance on this issue in a future update of the merger

guidelines.

24

3. NEW ZEALAND COMMERCE COMMISSION (2014-2015)

Period analysed For the 2014 review: merger cases between May 2008 and

June 2011.

For the 2015 review: merger cases between July 2011 and

July 2013.

N° of mergers considered 18 mergers in total, 10 belonging to the first review and, 8

to the second.

Internal or outsourced study Internal study.

Sector The 18 transactions concerned 40 markets from various

sectors, such as pharmaceuticals (including also animal

health products), electronic equipment, food, ….

Types of decision All decisions were clearance decisions.

Issues analysed The review was focused on expectations regarding 4 major

issues: entry and expansion, competitive constraint

exercised by existing competitors, divestitures, and

countervailing market power of buyers.

Methodology used - Interviews by phone of merged entities, customers and

competitors

- Internet research.

Main conclusions - Entry was a key reason for clearance in 17 markets and

did effectively take place in 12 out of these 17 markets.

- Presence of competitors in the market was a reason for

clearance in 20 cases and competitors continued to

effectively constrain the merged entity in 18 cases:

In one of these markets, the sole remaining market

participant exited the market because of concerns

regarding its product. Two years after that exit, a

third party entered the market.

In the other market, the effectiveness of import

competition varied depending on fluctuations in the

exchange rate.

- In the two clearance decisions in which a business line

was divested as part of the merger, the divested

business managed to maintain a market share that was

similar to that when the business was owned by a

merging party.

- In the one matter where buyer sponsorship of entry was

relied upon, such entry had not occurred but the

identified sponsor indicated that it would still do so

were prices to become too high.

Lessons learned - Exchange rates: In some cases the price difference at

which imports can profitably be brought into New

Zealand is so large that it is very likely that imports will

continue to provide competitive constraints. However,

exchange rate fluctuations can have an impact on the

competitive constraint exercised by imports. In this

case, the determination of the range of exchange rates

over which imports by foreign companies are likely to

occur should form a standard part of the assessment.

- Conditions for exports: When relying on imports as an

important source of competition, greater attention

should be paid to local market conditions in the

25

exporting countries and to possible variations in these

conditions as well as in the conditions affecting the

decision to import by New-Zealand companies. Factors

to be considered could include the process by which

export decisions are made, the financial conditions of

the exporting company and the certainty of supply.

- Exit in duopoly markets: When a merger results in a

duopoly, greater consideration should be given to the

possibility of withdrawal of one player.

- Opportunity cost or cost of entry: When considering

entry, particularly by multinational companies, the

opportunity costs of investing in New Zealand in light

of its small size of its economy should be considered.