Can Agent-Based Models Probe Market Microstructure? · cies (Fabretti 2013), but this still remains largely unexplored territory in the case of intraday data. For this reason, we

Can Agent-Based Models Probe Market Microstructure?

D. F. Platt ∗1,2 and T. J. Gebbie1,2,3

1School of Computer Science and Applied Mathematics, University of the Witwatersrand, Johannesburg2QuERILab - Quantifying Emergence, Risk and Information

3Department of Statistical Sciences, University of Cape Town, Cape Town

Abstract

We extend prior evidence that naively using intraday agent-based models that involve realistic order-matching processes for modeling continuous-time double auction markets seems to fail to be able toprovide a robust link between data and many model parameters, even when these models are able toreproduce a number of well-known stylized facts of return time series. We demonstrate that while theparameters of intraday agent-based models rooted in market microstructure can be meaningfully calibrated,those exclusively related to agent behaviors and incentives remain problematic. This could simply be afailure of the calibration techniques used but we argue that the observed parameter degeneracies are mostlikely a consequence of the realistic matching processes employed in these models. This suggests thatalternative approaches to linking data, phenomenology and market structure may be necessary and that itis conceivable that one could construct a useful model that does not directly depend on the nuances ofagent behaviors, even when it is known that the real agents engage in complex behaviors.

Keywords: agent-based modeling, calibration, complexity, market microstructure, hierarchical causality

JEL Classification: C13 · C52 · G10

1. Introduction

Financial agent-based models (ABMs) haveseen increasing prevalence in quantitative fi-nance literature in recent years, with a widevariety of models emerging that are capable ofreplicating the stylized facts of financial returntime series, such as a fat-tailed distributionand volatility clustering (Barde 2016). This islargely due to their replacement of empiricallyinconsistent economic assumptions, such asthat of a Gaussian return distribution and theefficient market hypothesis, with assumptionsrooted in agent behaviors (LeBaron 2005).

Despite this, there still exists significantskepticism as to the value and validity of thisclass of models, particularly in relation to cur-rent validation techniques (Hamill and Gilbert2016). The vast majority of financial ABMs,particularly those replicating continuous dou-ble auction markets at an intraday time scale,are still validated by demonstrating an abil-ity to produce return time series replicating a

number of empirically-observed stylized facts(Panayi et al. 2013). A detailed survey ofsuch stylized facts is presented by Cont (2001).Though a solid starting point, this method ofvalidation has brought with it a number ofsignificant challenges.

A key concern is the fact that a very largenumber of models with very different designphilosophies are able to replicate the stylizedfacts of return time series equally well, leadingto difficulty in model comparisons and the pre-cise identification of which mechanisms lead tothe aforementioned stylized facts (Barde 2016).

This, combined with concerns that increasesin the mechanistic complexity of intradayABMs may not provide improved representa-tions of the dynamics and processes that gov-ern market behavior and structure on shortertime scales necessitates a more comprehensiveinvestigation into model validation.

This leads to a need to calibrate such mod-els to transaction data in an attempt to obtainparameters that allow the models in question

∗Corresponding author, [email protected]

1

arX

iv:1

611.

0851

0v3

[q-

fin.

CP]

31

Aug

201

7

to generate financial return time series withmoments and other statistical characteristicscomparable to empirical measurements, suchthat an empirically measured series and a sim-ulated series can then be said to come from thesame distribution (Fabretti 2013). Such investi-gations have been done in the work of Fabretti(2013) and Gilli and Winker (2003), among var-ious other studies. An exhaustive survey ispresented by Kukacka and Barunik (2016).

The majority of these prior investigationshave focused primarily on the calibration ofvery simple models that produce time seriesappropriate at a daily time scale and typicallymake use of closed-form approximations tocalculate market prices, in the vein of closingauctions occurring at the end of each tradingday in real financial markets. Models of thisclass include the Farmer and Joshi (2002) andKirman (1991) models. Overall, these experi-ments have proven relatively successful, withcalibration using these methods generating sat-isfactory results.

Despite this, the methods employed by Fab-retti (2013) and Gilli and Winker (2003) havenot been readily applied to intraday modelsemploying realistic order matching processesapproximating continuous double auction mar-kets, with the general calibration literature forthis class of models also being relatively sparse.

The underlying questions of when, and towhat extent, one can use aggregate propertiesof financial fluctuations or return time seriesphenomena to probe the complex hidden fea-tures and structures inherent in real financialmarkets remains open. ABMs have been usedto successfully extract hidden structure for ag-gregate return data on daily sampled frequen-cies (Fabretti 2013), but this still remains largelyunexplored territory in the case of intradaydata.

For this reason, we had previously appliedthe calibration framework of Fabretti (2013) inPlatt and Gebbie (2016), attempting to calibratethe Jacob Leal et al. (2015) model, a model rep-resenting both high- and low-frequency traderinteractions in the context of a continuous dou-ble auction market. This model has a variety of

features that are common in more complex in-traday models, such as matching orders mecha-nistically rather than market clearing at or nearan equilibrium price. Other examples of thisclass of models include the Chiarella and Iori(2002), Chiarella et al. (2009) and Preis et al.(2006) models.

In this prior investigation (Platt and Geb-bie 2016), we demonstrated and verified theability of the model to generate well-knownreturn time series stylized facts, while still suf-fering from parameter degeneracies when sub-jected to the established calibration proceduresof Fabretti (2013). We also found the dynamicsof the simulated price time series to be dom-inated by the underlying dynamics of orderprices within the model, with parameters re-lating to order price determination being theonly meaningfully calibrated parameters. Thissuggested that some parameters may not havethe meaning or effect that the model designerhad originally intended.

This can be regarded as problematic if suchagent-based modeling research focuses on thevarying of parameter values to determine theeffect of various phenomena in financial mar-kets when the link between the parameters andreal data from the markets being consideredis weak. If these types of models are used tomake both regulatory and structural inferencesabout market microstructure then the robust-ness of the ability to link parameters to realdata and the ability to understand how param-eters drive model behavior is essential.

The original investigation of Jacob Leal etal. (2015) attempts to determine the effect ofdifferent numbers of high-frequency traderson the prevalence of flash crashes in a simu-lated market by varying a suitable parameter.Our prior investigation found that the param-eter in question in the model of Jacob Leal etal. (2015) was degenerate and had a poorlybehaved effect on a number of moment andtest statistic values of the generated price timeseries; this included the mean, standard devia-tion, kurtosis and generalized Hurst exponent(Platt and Gebbie 2016). This is also in spiteof the model’s ability to replicate a number of

2

well-known stylized facts.The source of these degeneracies is difficult

to determine. They may emerge because theJacob Leal et al. (2015) model generates orderprices for most of its traders without consultingthe current state of the limit order book (LOB),and instead uses a geometric random walk thatreferences previous market prices. They mayalso be a consequence of the use of realisticorder matching processes rather than closed-form approximations to market prices, whichmay affect the link between the underlyingparameters and empirical measurements. Al-ternatively, it may be that the modeling frame-work is fundamentally incomplete and that keydrivers of measured price dynamics are absentfrom the models themselves.

This ultimately leads us to consider two keyquestions in the following investigation:

Firstly, since some success was found incalibrating parameters related to order pricedynamics in our previous investigation (Plattand Gebbie 2016), is it possible that despite theexistence of degeneracies in parameters relat-ing to agent behaviors, parameters which relateto the order book and market microstructurecan be meaningfully calibrated?

Secondly, can aspects of market microstruc-ture not explicitly modeled in an ABM, such asorder flow correlation, be detected through cal-ibration? In other words, can we probe modelincompleteness through calibration?

We attempt to shed light on these concernsby applying the calibration methodology pre-viously considered (Platt and Gebbie 2016) toanother well understood intraday ABM approx-imating a continuous double auction marketusing realistic matching procedures, but nowusing an order placement mechanism that ismore closely related to the current state ofthe order book, namely the Preis et al. (2006)model.

An important precedent in the literature isthat relating to the zero-intelligence agent ar-gument of Farmer et al. (2005), which suggeststhat very basic models with high parsimonyand clear falsifiability can match observations

surprisingly well. These types of models aredriven by simple relationships between orderflow and price process parameters with no ex-plicit need for parameters related to more so-phisticated agent behaviors. Our study there-fore considers a more sophisticated model thathas similar basic parameters as an importantsubset.

2. The Preis et al. Model

A. Model Overview

We consider the model first presented by Preiset al. (2006) and later elaborated upon by Preiset al. (2007).

Referring to Figure 1, the model consists oftwo agent types, liquidity providers, who sub-mit limit orders to buy or sell an asset to aLOB, and liquidity takers, who submit marketorders to buy or sell an asset to the same LOB,through a series of T Monte Carlo steps.

LOBLiquidity

TakersLiquidityProviders

MarketPrice Time

Series

Market OrdersLimit Orders

Match Orders, Determine New Mid Price

Results in

Figure 1: An illustration of the agent types and interac-tions within the model

During each of these Monte Carlo steps,active liquidity providers first submit limit or-ders, referencing a decision based on the cur-rent state of the LOB, after which liquidity tak-ers submit market orders, resulting in a num-ber of trades.

Based on the mid price1 of orders in theLOB following the execution of all trades in aMonte Carlo step, a new market price for the

1The average of the best bid (buy order with the highest price) and best ask (sell order with the lowest price) in the LOB.

3

asset can be determined, ultimately resultingin a series of market prices, one for each MonteCarlo step.

SimulationStart

ModelInitializa-tion, 10

MC Steps

LiquidityTakersPlace

MarketOrders

LiquidityProviders

PlaceLimit

Orders

LiquidityProviders

CancelLimit

Orders

DetermineNew Mid

Price

UpdateLiquidityTaker BuyProbability

UpdateLiquidityProvider

PlacementDepth

SimulationEnd

Enter Standard MC Steps

MC Step 1to T − 1

MC Step T, Exit Standard MC Steps

Figure 2: Flowchart describing a typical simulation of TMonte Carlo steps

We present an outline of the overall simula-tion in Figure 2, where the steps indicated areelaborated upon in subsequent subsections.

B. Liquidity Provider Agent Specifi-cation

Each simulation contains NA liquidityproviders, who submit limit orders to theLOB at a given frequency, α. This means thatfor any given Monte Carlo step, approximatelybαNAc limit orders are placed by liquidityproviders. Each of the aforementioned ordersis of size 1 and is a buy order with probabilityqprovider =

12 .

The price of each of these orders is deter-mined by the current placement depth param-eter, λ(t), and the current state of the orderbook, namely the current best bid, pb, and thecurrent best ask, pa, where the price of a limitbuy order is given by

p = pa − 1− η (1)

and the price of a limit sell order is given by

p = pb + 1 + η (2)

where η is an exponentially distributed ran-dom number, generated according to

η = b−λ(t) ln(u)c (3)

and u ∼ U(0, 1).The placement depth parameter is time

varying, and is directly related to the buy prob-ability of liquidity takers, qtaker(t), discussedin more detail in subsequent subsections, andthe initial placement depth parameter, λ0. It isgiven by

λ(t) = λ0

1 +

∣∣∣qtaker(t)− 12

∣∣∣√⟨[qtaker(t)− 1

2

]2⟩Cλ

(4)

where Cλ is an integer parameter and 〈[qtaker −12 ]

2〉 is determined by iterating qtaker(t) for 105

Monte Carlo steps separately before the mainsimulation and obtaining the mean value of[qtaker − 1

2 ]2.

It should be noted that the above orderplacement mechanism is entirely driven by thecurrent characteristics of the LOB. This is incontrast to the placement mechanism used byJacob Leal et al. (2015), which is based on theprevious history of market prices. A more de-tailed LOB-based order placement mechanismis presented by Mandes (2015).

During each Monte Carlo step, liquidityproviders may also cancel previously placedorders. This is implemented by randomly can-celling each limit order in the LOB with proba-bility δ.

Finally, the model requires some form ofinitialization to produce a viable order book be-fore actual trading can begin. This is done in aseries of 10 initial Monte Carlo steps, in whichliquidity providers place limit orders aroundan initial price, pa = pb = p0, with no liquid-ity taker activity present. This corresponds tothe placement of approximately b10αNAc limitorders.

4

C. Liquidity Taker Agent Specifica-tion

Each simulation contains NA liquidity takers,who submit market orders to the LOB at agiven frequency, µ. This means that for anygiven Monte Carlo step, approximately bµNAcmarket orders are placed by liquidity takers.Each of the aforementioned orders is of size1 and is a buy order with probability qtaker(t),with qtaker(0) = 1

2 .A sell market order simply executes and

results in the removal of the best bid from theLOB and a buy market order simply executesand results in the removal of the best ask fromthe LOB, since all orders are of size 1.

Unlike qprovider, qtaker(t) is not fixed, butrather evolves over time. It is implementedas a mean-reverting random walk, with meanqtaker(0), increment ∆S, and a mean reversionprobability of 1

2 + |qtaker(t)− 12 |.

3. Calibration Experiment Design

A. Data

The dataset used in all calibration experimentsis acquired in the TRTH (Thomson Reuters2016) format, presenting a tick-by-tick series oftrades, quotes and auction quotes.

We convert the dataset to a series of one-minute price bars, with each price correspond-ing to the final quote mid price for each minute,where the mid price of a quote is given as theaverage of the level 1 bid price and level 1 askprice associated with that quote, as was previ-ously done in Platt and Gebbie (2016). Fromthis series of prices, we may obtain a series oflog prices 2, which is the series we attempt tocalibrate the model to.

The transaction dataset often presentsevents occurring outside of standard tradinghours, 9:00 to 17:00, but we consider onlyquotes with a timestamp occurring in the pe-riod from 9:10 to 16:50 on any particular trad-

ing day. This is a result of the fact that theopening auction occurring from 8:30 to 9:00tends to produce erroneous data during thefirst 10 minutes of continuous trading and thefact that the period from 16:50 to 17:00 repre-sents a closing auction.

In all calibration experiments, we investi-gate a one-week period, corresponding to a to-tal of 2300 one-minute price bars, representing460 minutes of trading each day from Mondayto Friday.

Finally, we consider a single, liquid stocklisted on the Johannesburg Stock Exchange,Anglo American PLC, over the period begin-ning at 9:10 on 1 November 2013 and endingat 16:50 on 5 November 2013. This dataset wasalso considered in Platt and Gebbie (2016).

B. Calibration Framework

As previously discussed, we apply the calibra-tion framework described by Fabretti (2013) toan intraday ABM approximating a continuousdouble auction market through the use of re-alistic order matching procedures, as opposedto a model operating at a daily time scale andusing closed-form approximations to marketprices.

We make use of the method of simulatedmoments (Winker et al. 2007) to construct anobjective function that measures errors relat-ing to the mean, standard deviation, kurtosis,Kolmogorov-Smirnov (KS) test and generalizedHurst exponent when comparing a log pricetime series measured from the data and a logprice time series simulated using the Preis etal. (2006) model.

We aim to minimize this objective functionby employing the Nelder-Mead simplex algo-rithm combined with the threshold acceptingheuristic (Gilli and Winker 2003).

We reproduce the implementation we pre-viously described in Platt and Gebbie (2016)without any alteration, again making use of 5

2This is a technical requirement of the method of Fabretti (2013) and is discussed in detail in Platt and Gebbie (2016).3A single calibration experiment involving the Nelder-Mead simplex algorithm may take in excess of 7 hours when

parallelized over the 2× 16 = 32 workers of an HP Z840 workstation with 2 Intel Xeon E5-2683 CPUs, making more MonteCarlo replications computationally too expensive relative to the reduction in the variance of the estimates.

5

Monte Carlo replications, due to computationalconstraints 3.

4. Calibration Results

A. Free and Fixed Parameters

In all calibration experiments, we consider 6free parameters, with only NA set to be fixed.This particular simplification was made forthe purposes of maintaining computationaltractability, since increasing the value of NAcan have a detrimental impact on the speed ofsimulation. We have chosen NA = 250, as wasselected by Preis et al. (2006).

The remaining model parameters, namelyδ, λ0, Cλ, ∆S, α and µ, defined in Section 2, areset to be free parameters and we randomly gen-erate initial values for these parameters duringeach calibration experiment.

We initialize δ, ∆S and µ between 0 and 0.1,α between 0.1 and 0.5, λ0 between 0 and 200,and Cλ between 0 and 20.

B. Nelder-Mead Simplex Algorithm

Since we consider n = 6 free parameters inour calibration experiments, we begin withn + 1 = 7 simplex vertices, each consistingof randomly generated initial values for the 6free parameters.

In our experiments, it was noted that 100iterations of the Nelder-Mead simplex algo-rithm with threshold accepting almost alwaysproduced convergent behavior and we thusconducted 20 calibration experiments with thisfixed number of iterations.

Despite the presence of convergent behav-ior, we find that the obtained parameter sets aresomewhat different for various calibration ex-periments, with dependence on the set of initialvertices. We also observed this phenomenon inPlatt and Gebbie (2016), but find a greater aver-age reduction in the initial search space whenconsidering the parameter confidence intervals4 obtained for the Preis et al. (2006) model in

comparison to those obtained for the Jacob Lealet al. (2015) model. These confidence intervalsare shown in Table 1.

Table 1: Nelder-Mead Simplex Algorithm CalibrationResults

Parameter 95% Conf Int s√n

δ [0.0554, 0.0782] 0.0054λ0 [137.3818, 190.9182] 12.7892Cλ [11.0808, 21.3192] 2.4458∆S [0.0272, 0.0572] 0.0072α [0.1545, 0.2517] 0.0232µ [0.0692, 0.1014] 0.0077

95% confidence intervals for the set of free parameters,obtained from 20 independent calibration experimentsinvolving the Nelder-Mead simplex algorithm combinedwith the threshold accepting heuristic

This is indicative of convergence to localminima, even with the inclusion of methodsaiming to overcome this problem, namely thethreshold accepting heuristic. It would there-fore appear that there exists significant diffi-culty in identifying a unique, optimal parame-ter set.

As was discussed by Fabretti (2013), we alsofind that these different parameter sets forma region of feasible parameters, all producingsimilar objective function values, despite differ-ences in the parameter values themselves. InPlatt and Gebbie (2016), we found that such atrend existed, but not as prominently as in thiscase, where we now find that all the calibrationexperiments conform to this behavior.

It is also worth noting that, as was foundin Platt and Gebbie (2016), the majority of theobtained parameter sets again produce fits ofsimilar quality to that of Fabretti (2013), whichwe demonstrate in Section 4.C.

C. Comparison of Calibrated Modeland Empirical Data

It was mentioned in the preceding sectionsthat while no unique parameter set could be

4The intervals are calculated as: x̄± t∗ s√n , where x̄ is the sample mean, s is the sample standard deviation, n is the

sample size, and t∗ is the appropriate critical value for the t distribution.

6

established, all of the experiments converged todifferent parameter sets with similar objectivefunction values producing reasonable behaviorwhen compared to the simulated data. In thissection, we consider the best parameter set, asmeasured by the objective function, obtained inthe Nelder-Mead simplex algorithm calibrationexperiments.

For this parameter set, we simulate 20 pricepaths and obtain the 95% confidence intervalsfor the estimates of the mean, standard de-viation, kurtosis, and Hurst exponent of thesimulated log prices and compare this to theempirical moments obtained from the transac-tion data.

Table 2: Best Parameter Set Obtained Through Calibra-tion

Calibrated Parameter Parameter Valueδ 0.0733

λ0 180Cλ 33∆S 0.0328α 0.2129µ 0.0653

Best parameter set obtained through the implementationof Nelder-Mead simplex algorithm combined with thethreshold accepting heuristic

Table 3: Comparison of Empirical and Calibrated Mo-ments

Moment Simulated Value Empirical Valuem1 [5.4628, 5.5137] 5.5143m2 [0.0268, 0.0387] 0.0300m3 [2.3716, 3.3788] 1.2402m4 [0.5795, 0.5993] 0.5658

95% confidence intervals for the simulated moments ob-tained using the calibrated parameters compared to theempirically measured moments, with m1, ..., m4 corre-sponding to the mean, standard deviation, kurtosis andgeneralized Hurst exponent respectively

As seen in Table 3 and also demonstratedin prior work (Platt and Gebbie 2016), we ob-serve that the parameter sets obtained throughthe calibration procedure employed are indeed

able to produce fits of similar quality to thatof Fabretti (2013) for the mean and standarddeviation, while slightly overestimating theHurst exponent and severely overestimatingthe kurtosis. This is predominantly due to thefact that the weight matrix obtained throughthe method of Fabretti (2013) tends to assignvery large weights to errors on the mean andstandard deviation, while assigning very smallweights to errors on kurtosis, leading calibra-tion to either underestimate or overestimatekurtosis in the investigations of both Fabretti(2013) and Platt and Gebbie (2016).

We provide a detailed discussion on pos-sible shortcomings of the method of Fabretti(2013) in Section 6.

Despite the previously mentioned flaws,the above seems to suggest that members ofthis class of models, namely ABMs of contin-uous double auction markets using realisticorder matching processes, are indeed capableof reproducing a number of characteristics ofempirically-sampled data to some degree andthat the calibration methodology of Fabretti(2013) can indeed determine appropriate pa-rameter sets, but that the parameters produc-ing such fits are simply not unique. In thefollowing subsections, we detail possible rea-sons for this behavior through an investigationof the behavior of the objective function.

D. Parameter Analysis

In an attempt to illustrate the behavior of theobjective function with respect to changes in pa-rameter values, we generate surfaces represent-ing the objective function values correspond-ing to various parameter value pairs, as hasbeen done by Gilli and Winker (2003) and Fab-retti (2013). We repeat the process previouslyemployed in Platt and Gebbie (2016), wherewe consider 1000 points in a two-dimensionalspace generated using a two-dimensional Sobolsequence, corresponding to various possiblevalues of a parameter pair. We then obtain theobjective function value for each combinationof the values in the parameter pair using 5Monte Carlo replications, resulting in an ob-

7

jective function surface. The base parameterset used for these experiments is presented inTable 2.

-20.1

0

2

4

0.2

6

f(θ)

×105

0.08

8

10

α

0.3 0.06

12

µ

14

0.040.40.02

0

Figure 3: Type 1 (Platt and Gebbie 2016) parameter sur-face illustrating the effect of α and µ on theobjective function

As is clearly visible in Figures 3 and 4, wesee that δ, ∆S, α and µ all exhibit a similar ef-fect on the objective function. While there areindeed certain regions that produce particu-larly bad fits to the data, the vast majority ofthe surfaces are relatively flat, indicating a verywide range of feasible parameters producing areasonable fit to the data, with no clear way todistinguish an optimal global minimum. Thisis a similar insight to that expressed by Fabretti(2013).

0

1000

2000

0.08

3000

0.02

f(θ)

4000

0.06

5000

δ

0.04

6000

∆S

7000

0.04 0.060.02 0.08

Figure 4: Type 1 (Platt and Gebbie 2016) parameter sur-face illustrating the effect of δ and ∆S on theobjective function

This is the likely reason why parameter con-

vergence was not observed, despite the fact thatthe objective function values were similar foreach experiment, as a very large area of the pa-rameter space produces very similar dynamics.The effect of these parameters on the dynamicsof the obtained log price time series is thus notclearly defined, again indicative of parameterdegeneracies in a similar vein to those foundfor the Jacob Leal et al. (2015) model in Plattand Gebbie (2016).

100

200

300

400

500

505

600

f(θ)

700

800

λ0Cλ

900

10010

1000

15015

Figure 5: Type 3 (Platt and Gebbie 2016) parameter sur-face illustrating the effect of λ0 and Cλ on theobjective function

In contrast to this, however, Figure 5 indi-cates that both Cλ and λ0 produce a very well-defined effect on the objective function. This isan interesting observation. In Platt and Gebbie(2016), we observed a similar trend in the caseof the Jacob Leal et al. (2015) model, with allparameter surfaces behaving in a haphazardmanner, except for those dealing with parame-ters that drove the random walk which definedorder prices in the order book. It is thereforenot surprising that Cλ and λ0 happen to drivethe order placement depth process, λ(t), andhence order prices in this model.

In our previous investigation (Platt andGebbie 2016) we conjectured that this behaviorcould be attributed to the fact that the JacobLeal et al. (2015) model set order prices accord-ing to the previous history of market pricesonly and not the current state of the order book.This may have reduced the effect of order sizeand order frequency effects on the obtained logprice time series. This could, in turn, cause the

8

associated parameters to become degenerate.It was believed that consideration of the orderbook dynamics during order placement, whichwould be more directly affected by the otherparameters in the model, would lead to greaterparameter stability.

Despite this, it now seems that this wasnot the case. We have found that even withsuch considerations in the Preis et al. (2006)model, we have again produced similar find-ings, leading to a more unfortunate conclusionthat increased mechanistic complexity may notenhance the ability of models to be open tocredible calibration.

In this context, one should consider thework of Bookstaber et al. (2016), which pro-vides a number of significant mechanistic ex-tensions to the Preis et al. (2006) model, mostnotably the inclusion of heterogeneous deci-sion cycles and new agent types, such as largefirms. As with much of the existing ABM litera-ture, stylized fact-centric validation proceduresare employed exclusively, with the authors ex-plicitly stating that such validation proceduresare fairly rigorous. Our results indicate thatthis may in fact not be the case and that there isat least cause for some skepticism as to whethercertain of the extensions to the model result inparameter degeneracies not detected by styl-ized fact-centric validation.

It is important to note that in the case of theJacob Leal et al. (2015) model, a single parame-ter associated with one of the smooth objectivefunction surfaces previously mentioned didshow convergence, though neither λ0 nor Cλ

have demonstrated this behavior in this inves-tigation.

This is most likely due to the fact that otherparameters in the Jacob Leal et al. (2015) modelnever produced significant effects on the ob-jective function in comparison to the singlesuccessfully calibrated parameter.

In the Preis et al. (2006) model, however,there are regions where the objective functionvalue can become very large for certain val-ues of δ, ∆S, α and µ, whereas they typicallyhave a very small effect on the objective func-tion. This is because these regions typically

represent unstable model dynamics, for exam-ple α close to µ and a high order cancelationrate, δ, leading to an illiquid order book thatis frequently depleted of orders, which in turndrives unrealistic and unstable simulated pricedynamics.

These very high objective function valuesin these regions may have affected convergencein λ0 and Cλ, since their effect would not be asdominant as it would be for typical parametervalues, this dominance being the reason forthe observed convergence in Platt and Gebbie(2016). Rather than blindly accepting such rea-soning, however, we rigorously demonstratethat such convergence is indeed possible inSection 4.E.

It would therefore seem that regardless ofthe equations used to construct intraday ABMsof continuous double auction markets involv-ing realistic order matching processes, the dy-namics of the obtained market price time se-ries are predominantly driven by the equationsused to determine order prices in the model,with other parameters degenerating to a signif-icant extent.

This may suggest that prices are much morethan just the aggregation of predefined mech-anistic classes of agents (Wilcox and Gebbie2014).

E. Verification of λ0 and Cλ Conver-gence

In order to verify that λ0 and Cλ can beuniquely determined when the dynamics ofthe other parameters do not introduce noiseinto the calibration procedures, we consider aseparate calibration experiment involving onlyλ0 and Cλ.

Rather than using the Nelder-Mead sim-plex algorithm as before, we now consider agenetic algorithm, since 3 initial simplex ver-tices would be too small a parameter space torandomly initialize.

We begin with a randomly initialized initialpopulation of 100 individuals, using the sameparameter bounds as in the Nelder-Mead sim-plex calibration experiments, and iterate this

9

population for 50 generations, determined tobe sufficient for convergence. We then repeatthis process a total of 8 times in order to obtainconfidence intervals as in Section 4.B. All otherparameters are set according to the values inTable 2.

Table 4: Genetic Algorithm Calibration Results (λ0, Cλ)

Parameter 95% Conf Int s√n

λ0 [193.2596, 196.5567] 0.6972Cλ [18.7165, 19.4306] 0.1510

95% confidence intervals for λ0 and Cλ, obtained from 8independent calibration experiments involving a geneticalgorithm

Referring to Table 4, we see that when onlyλ0 and Cλ are considered in a calibration ex-periment, we observe meaningful convergencein both parameters. The identified region ofconvergence is also consistent with the visibleminimum in Figure 5.

Table 5: Genetic Algorithm Calibration Results (δ, ∆S,α, µ)

Parameter 95% Conf Int s√n

δ [0.0653, 0.0958] 0.0064∆S [0.0163, 0.0405] 0.0051α [0.1167, 0.1926] 0.0160µ [0.0608, 0.0924] 0.0067

95% confidence intervals for δ, ∆S, α and µ, obtainedfrom 8 independent calibration experiments involving agenetic algorithm

Finally, we repeat the genetic algorithm ex-periment employed for parameters λ0 and Cλ,but this time consider δ, ∆S, α and µ as freeparameters and as expected find that these pa-rameters cannot be uniquely identified usingthe genetic algorithm, as shown in Table 5.

5. Realistic Order Matching

Procedures and Price Dynamics

In both our current and previous investigations(Platt and Gebbie 2016), it appears that certain

features of the microstructure of a market, suchas order book depth and the associated dynam-ics of the prices of limit orders, can be reliablyprobed by the calibration of intraday ABMs totransaction data. In contrast to this, however,it seems that the parameters of these modelsassociated with the frequency at which traderssubmit orders, the specific number of traderagents participating in a particular market andvarious other phenomena that are rooted inagent behavior tend to be difficult to infer withsignificant confidence.

This naturally leads to a discussion on theeffect of various processes within an intradayABM on the simulated price time series whenthis time series is generated using realistic or-der matching processes. Bouchaud et al. (2008)argue that prices in real markets are driven bytwo main phenomena, the response of prices toindividual orders and the flow of orders arriv-ing in the market. We therefore investigate howthis may apply to a market simulated using anintraday ABM.

It is evident that the processes which de-termine order prices within an intraday ABMdirectly influence the first of these phenom-ena in a well-defined way. These processesdirectly determine the prices of limit orderswithin the LOB and these order prices definethe mid prices or trade prices (depending onthe model in question) that will eventually beset as market prices. This likely explains thefairly well-defined effect of the parameters in-fluencing order prices on both the simulatedprice time series and objective function in thecase of the Jacob Leal et al. (2015) and Preis etal. (2006) models.

The second mechanism, namely the orderflow, requires more careful consideration. Akey question to consider is whether the Preis etal. (2006) model does in fact produce a realistic,nontrivial order flow.

A well-known property of order flow in realfinancial markets is long memory (Bouchaudet al. 2008). This phenomenon can be observedby plotting the autocorrelation function of thetime series of trade signs for a particular mar-ket over some period (+1 for buyer initiated

10

trades and -1 for seller initiated trades), whichtypically demonstrates a slow decay in auto-correlation as the number of lags is increasedin most empirical investigations (Kuroda et al.2011).

0 10 20 30 40 50 60 70 80 90 100Lag

-0.05

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

Auto

corre

latio

n

Autocorrelation Function95% Confidence Bands

Figure 6: Autocorrelation function for trade signs in-ferred from our calibration dataset

Using the measured data and following themethod of Lee and Ready (1991), which wasalso applied to similar datasets in Harvey etal. (2017), we classified trades as either buyeror seller initiated and computed the associatedtrade sign autocorrelation function. This au-tocorrelation function is presented in Figure 6and confirms that long memory dynamics arecaptured in the order flow associated with thedata to which the model was calibrated.

Table 6: Default Model Parameter Set

Parameter Default Valueδ 0.0250

λ0 100Cλ 10∆S 0.0010α 0.1500µ 0.0250

Default model parameter set presented in Preis et al.(2006) and Preis et al. (2007)

Figure 7 shows the autocorrelation functionfor trade signs generated by the Preis et al.(2006) model, initialized with the default pa-

rameters described by Preis et al. (2006), whichwe present in Table 6. We see that the character-istic slow decay in autocorrelation associatedwith long memory is absent, indicative thatsuch dynamics are not captured by the modelwhen initialized with the default parametersof Preis et al. (2006).

0 10 20 30 40 50 60 70 80 90 100Lag

-0.05

-0.04

-0.03

-0.02

-0.01

0

0.01

0.02

0.03

0.04

0.05

Auto

corre

latio

n

Mean Autocorrelation Function95% Autocorrelation Function Mean Confidence Intervals95% Confidence Bands

Figure 7: Autocorrelation function for trade signs gen-erated over 50 Preis et al. (2006) model simu-lations initialized with the parameter set pre-sented in Table 6

Referring to Figure 8, where we repeat thisexercise using the calibrated parameters pre-sented in Table 2, we see that the slow decay inautocorrelation associated with long memoryprocesses is now present, indicating that thecalibration process has determined a parame-ter combination that results in a more realisticorder flow than the model’s default parame-ters.

On closer inspection, it was found that shift-ing the values of individual parameters in Ta-ble 2, with the exception of ∆S, and repeatingthis exercise did not cause these more realisticlong memory dynamics to disappear. Whenshifting ∆S, however, it was found that smallervalues of this parameter resulted in long mem-ory dynamics no longer being present in theorder flow.

This is in fact consistent with the surfacepresented in Figure 4:

For the δ value presented in Table 2, ap-proximately 0.07, we see that smaller ∆S val-

11

ues tend to result in poorer objective functionvalues, whereas values of ∆S roughly largerthan 0.03 tend to result in fairly similar andrelatively low objective function values. Notsurprisingly, this region of lower objective func-tion values in fact corresponds relatively wellto those in which long memory was observedin the simulated order flow.

0 10 20 30 40 50 60 70 80 90 100Lag

-0.05

-0.04

-0.03

-0.02

-0.01

0

0.01

0.02

0.03

0.04

0.05

Auto

corre

latio

n

Mean Autocorrelation Function95% Autocorrelation Function Mean Confidence Intervals95% Confidence Bands

Figure 8: Autocorrelation function for trade signs gen-erated over 50 Preis et al. (2006) model simu-lations initialized with the parameter set pre-sented in Table 2

Therefore, ∆S does indeed exert some effecton the order flow within the model, but a verywide range of possible values exists for ∆S thatproduces long memory dynamics. This is re-flected in the effect of ∆S on the obtained pricetime series, which is itself reflected in the effectof ∆S on the objective function, leading to alarge region where no unique minimum canbe determined. While the simulated order flowobtained through calibration is by no meansidentical to that observed in the data, likely dueto model limitations, calibration has resultedin an attempt to capture these dynamics.

In contrast to Cλ, λ0 and ∆S, parametersα, µ and δ relate to agent behaviors and thusdo not directly affect the individual orders ororder flow within the model. The fact that al-tering these parameters did not affect the pres-ence of long memory in the simulated orderflow provides evidence of the second aspectof the preceding statement. This implies that

these parameters do not directly relate to thetwo processes conjectured by Bouchaud et al.(2008) to be the driving forces behind prices inreal markets and this seems to be mirrored inthe realistic matching processes employed inintraday ABMs, where we observed that the ef-fect of these parameters on the simulated pricetime series was not well-defined.

We therefore conjecture that the encoun-tered parameter degeneracies emerge fromthe application of realistic matching processeswithin intraday ABMs. Parameters which exertwell-defined effects on either order flow or indi-vidual orders themselves produce well-definedeffects on the obtained simulated price timeseries and are therefore amenable to calibra-tion. These parameters tend to be rooted in themicrostructure of the market and the nature ofthe LOB. In contrast to this, parameters whichrelate more to agent behavior and do not tendto have a well-defined or direct effect on in-dividual order dynamics or order flow tendto produce poorly-defined effects on the ob-tained price time series, resulting in calibrationdifficulties.

This may explain why the calibration ofclosed-form, daily ABMs proved more success-ful in past experiments, such as those of Fab-retti (2013). In using a closed-form solution todetermine market prices, one directly enforcesa well-defined, albeit simplified, relationshipbetween parameter values and the obtainedmarket price time series. In contrast to this, therealistic matching processes in intraday modelsseem to be instead driven by order flow andthe nature of individual orders, leading param-eters which do not directly determine thesedynamics in a well-defined way to producehaphazard dynamics.

The recovery of trade sign autocorrelationthrough model calibration leads us to a dis-cussion of the latent order book. This is a keypotential source of top-down causation (Wilcoxand Gebbie 2014) that may render the type ofABM considered here as being fatally incom-plete.

As suggested by Bonart (2015), the LOBitself only represents a fraction of the actual

12

liquidity in a particular market. In real finan-cial markets, liquidity providers attempt tohide their intentions and thus tend to avoid theplacement of very large limit orders. This im-plies that liquidity takers attempting to acquireor liquidate large numbers of shares cannot doso through the placement of a single order andthus break a single parent order into a numberof child orders, resulting in the observed tradesign autocorrelation (Tóth et al. 2011). The trueintentions of liquidity providers and the truenature of supply and demand are therefore notcaptured in the order book itself, but ratherin a fictitious order book known as the latentorder book. The purpose of trading that drivesthe latent liquidity can operate on very differ-ent, and possibly much longer, time scales thanthose inherent in the short term dynamics ofthe order book (Bouchaud et al. 2008).

Therefore, it may be worthwhile to considerthe calibration of reaction-diffusion models ofthe latent order book to similar data to thatemployed in this investigation in future work.Possible candidate models include the work ofMastromatteo et al. (2014a), Mastromatteo etal. (2014b) and Gao and Deng (2014).

It is clear that the ABM considered, thoughnot explicitly considering the latent order book,has been able to reproduce some of the featuresof financial markets thought to originate fromit, suggesting that there may be some hiddenexplanatory potential embedded within ABMsof continuous double auction markets with re-gard to the underlying microstructure of themarket. The ability to uniquely determine theparameters associated with limit order pricegeneration processes in both the Jacob Leal etal. (2015) and Preis et al. (2006) models pro-vides further evidence of this possibility.

Therefore, while certain parameters couldnot be uniquely determined in our calibrationexperiments, this does not imply that suchmodels do not possess an ability to probe mar-ket microstructure.

If dynamics relating to possible sources oftop-down causation (Wilcox and Gebbie 2014),such as those resulting from the latent orderbook (Bouchaud et al. 2008; Bonart 2015), were

not readily recoverable through calibration,there would be concerns that a given modelcould not be used to faithfully represent thestructure of the market being modeled. Wehave been able to provide an example of theuse of calibration to determine parameters thatrecover reasonable order flow autocorrelationwithout a priori parameter tuning.

This does not resolve the concerns that thefeedbacks between order flow and the stateof the LOB may be dominant features of realintraday markets and that the behaviors of dif-ferent agent classes cannot be robustly resolvedthrough calibration. This leads to the conclu-sion that overly complex and mechanistic intra-day ABMs and market clearing specificationsmay be counterproductive in the search for par-simonious, yet useful model representations.

The derivation of models from fundamen-tal assumptions about the intelligent behaviorsof market participants often results in modelsthat are difficult to test (Ziliak and McCloskey2004), since a large variety of auxiliary assump-tions are often necessary to make model esti-mation meaningful. These assumptions mayenhance the formal acceptability of a particularmethod or approach, but often lead to littlenew insight into how best to model, explain orpredict phenomenology arising from real mar-kets. This can be seen as resulting in a situationwhere a model seems to have passed a varietyof tests but can still present model behaviorsthat can be recovered using alternative explana-tions (Farmer et al. 2005; Ziliak and McCloskey2004).

As stated in Farmer et al. (2005) and alludedto in Smith et al. (2003), it is conceivable to havea successful model that does not directly de-pend on the nuances of agent behaviors, evenwhen it is known that the real agents engage insophisticated behaviors. The fact that our cali-bration attempts point to a similar conclusion,though from a very different departure point,could be telling us something important aboutthe relationship between rational effects andstochastic effects – particularly where institu-tions can strongly shape behavior (Farmer etal. 2005). This points to the widely held view

13

(Bouchaud et al. 2008) that it is indeed the com-bination of order-flow and price specificationthat drive prices in financial markets. As statedin the Farmer et al. (2005), this does not answerwhy order flow varies as it does but seems topoint at the situation where ABMs of intradaydouble-auction markets can be sufficiently welldescribed by a reduced set of parameters, pa-rameters that are directly related to order flowand order price specification, independently ofcomplex agent behaviors.

6. Relevant Caveats

It is important to note that our investigationdoes not claim to be definitive or exhaustive,but rather aims to encourage new lines of in-vestigation that further evaluate the rigor ofcurrent research paradigms, such as the exclu-sive use of stylized fact-centric validation andthe continued increase of the mechanistic com-plexity of financial ABMs without compellingmodel calibration.

While a substantial part of this manuscripthas been devoted to the discussion of possibleflaws relating to current approaches to intradayfinancial ABM construction, it is also impor-tant to identify possible flaws associated withthe calibration methodology itself as a relevantcaveat to this investigation, which we alludedto in Platt and Gebbie (2016).

Although the method of Fabretti (2013) iswell-motivated for the most part and basedupon established literature, it does involve theinversion of near-singular matrices. We believe,however, that because we are able to obtainsmooth objective function surfaces for certainparameters (λ0, Cλ) in both our current andprevious investigations (Platt and Gebbie 2016),any noisy effects observed in the constructedobjective function surfaces most likely stemfrom model as opposed to calibration method-related concerns.

In addition to this and as previously dis-cussed, the weight matrices in both our inves-tigation and that of Fabretti (2013) assign sig-nificantly larger weights to errors on the meanand standard deviation in comparison to other

moments. Given that characteristics such askurtosis and the Hurst exponent are relativelyimportant features of financial return time se-ries and since there may be many differentdistributions that could be parameterized bythe same mean and standard deviation, this isalso an undesirable feature of the method.

Therefore, it is recommended that futureinvestigations involving the method of Fabretti(2013) or similar methods make attempts to ad-dress these particular problems to improve nu-merical stability and the veracity of any resultsobtained using such methods. In achievingsuch ends, one might also consider the replace-ment of the constructed objective function withan alternative benchmark, such as the informa-tion theoretic criterion introduced by Lamperti(2015).

While we have identified a number of rele-vant methodological concerns, we believe thatthe identification of relatively precise valuesfor λ0 and Cλ as well as the ability of themethod to find parameter sets that recover or-der flow correlations are sufficient to argue thatthe method of Fabretti (2013) is able to providevaluable insights in this context.

7. Conclusion

In both our current and previous investigations(Platt and Gebbie 2016), we have observed thatthere is some evidence indicating that intradayABMs of continuous double auction marketsemploying realistic order matching procedurestend to produce price time series with dynam-ics predominantly driven by the processes orequations defining order prices within them,with the remaining parameters and procedurestending to produce either relatively insignif-icant or poorly behaved effects on the sameprice time series. This is indicative of parame-ter degeneracies and is in spite of the fact thatthese models tend to reproduce the empirically-observed stylized facts of financial return timeseries relatively well.

We conjecture that this is likely due to thefact that like in real financial markets, simu-lated market prices in intraday ABMs employ-

14

ing realistic matching processes are driven byresponses to individual orders and the natureof order flow and therefore only parametersassociated with processes which affect thesephenomena in well-defined ways produce well-defined and significant effects on the marketprice time series.

This suggests that parameters related tothese phenomena, such as the nature of in-dividual orders in a market, can indeed beprobed by intraday ABMs, but that parame-ters rooted in agent behavior are difficult tocalibrate due to the fact that they do not ex-ert well-behaved effects on the simulated pricetime series when it is generated using realisticmatching processes.

Traditional stylized fact-centric validationseems unable to detect these potential prob-lems, suggesting that such methods of valida-tion are simply not sufficient for the validationof intraday ABMs employing realistic match-ing procedures. Therefore, simply increasingthe mechanistic complexity of agents in intra-day ABMs such that empirically-observed re-turn time series stylized facts are reproducedmay lead to flawed insights, simply becausethe relationship between the simulated pricetime series and agent behaviors has to be well-

defined.We thus argue that calibration and stylized-

fact replication should both be consideredwhen developing such ABMs, in order to en-sure a replication of reasonable model behavior,while still ensuring that parameters producethe intended effect of the model designer onthe simulated price time series.

This move towards more rigorous valida-tion would be an important paradigm shift,since our existing calibration experiments havealready shown evidence of dynamics, such asorder flow correlation, not being sufficientlywell-captured in the models themselves, butemerging as important concerns when attempt-ing to calibrate such models to data.

8. Acknowledgements

T.J. Gebbie acknowledges the financial supportof the National Research Foundation (NRF)of South Africa (grant number 87830). D.F.Platt acknowledges the financial support of theUniversity of the Witwatersrand, Johannesburgand NRF of South Africa. The conclusionsherein are due to the authors and the NRFaccepts no liability in this regard.

References

Barde S (2016) Direct calibration and comparison of agent-based herding models of financialmarkets. J Econ Dyn Control 73:329-353

Bonart J (2015) Mathematical aspects of delayed market clearing in order driven markets and itsapplications to non-Markovian price impact and optimal execution. SSRN:2659092

Bookstaber R, Foley MD, Tivnan BF (2016) Toward an understanding of market resilience: marketliquidity and heterogeneity in the investor decision cycle. J Econ Interact Coord 11:205-227

Bouchaud JP, Farmer JD, Lillo F (2008) How markets slowly digest changes in supply and demand.arXiv:0809.0822.

Cartea A, Jaimungal S, Penalva J (2015) Algorithmic and High-Frequency Trading. CambridgeUniversity Press, Cambridge

Chiarella C, Iori G (2002) A simulation analysis of the microstructure of double auction markets.Quant Financ 2:346-353

Chiarella C, Iori G, Perello J (2009) The impact of heterogeneous trading rules on the limit orderbook and order flows. J Econ Dyn Control 33:525-537

15

Cont R (2001) Empirical properties of asset returns: stylized facts and statistical issues. QuantFinanc 1:223-236

Easley D, Lopez de Prado M, O’Hara M (2012) The Volume Clock: Insights into the High-FrequencyParadigm. J Portfolio Manage 39:19-29

Fabretti A (2013) On the problem of calibrating an agent-based model for financial markets. J EconInteract Coord 8:277-293

Farmer JD, Joshi S (2002) The price dynamics of common trading strategies. J Econ Behav Organ49:149-171

Farmer JD, Patelli P, Zovko II (2005) The predictive power of zero intelligence in financial markets.Proc Natl Acad Sci USA 102(6):2254-2259

Gao X, Deng SJ (2014) Hydrodynamic limit of order book dynamics. arXiv:1411.7502

Gilli M, Winker P (2003) A global optimization heuristic for estimating agent based models.Comput Stat Data Anal 42:299-312

Hamill L, Gilbert N (2016) Agent-Based Modelling in Economics. John Wiley & Sons, Chichester

Harvey M, Hendricks D, Gebbie T, Wilcox D (2017) Deviations in expected price impact for smalltransaction volumes under fee restructuring. Physica A 471:416-426

Jacob Leal S, Napoletano M, Roventini A, Fagiolo G (2015) Rock Around the Clock: An Agent-Based Model of Low- and High-frequency Trading. J Evol Econ 25:1-25

Kirman A (1991) Epidemics of opinion and speculative bubbles in financial markets. In: Taylor M(ed) Money and Financial Markets. Blackwell, Oxford, pp 354-368

Kukacka J, Barunik J (2016) Estimation of Financial Agent-Based Models with Simulated MaximumLikelihood. SSRN:2783663

Kuroda K, Maskawa J, Murai J (2011) Stock price process and long memory in trade signs. AdvMath Econ 14:69-92

Lamperti F (2015) An Information Theoretic Criterion for Empirical Validation of Time SeriesModels. LEM Papers Series, Laboratory of Economics and Management, Sant’Anna School ofAdvanced Studies, 2-2015

LeBaron B (2005) Agent-based Computational Finance. In: Judd KL, Tesfatsion L (eds) TheHandbook of Computational Economics, Vol. 2. Elsevier, Amsterdam, pp 1187-1233

Lee CMC, Ready MJ (1991) Inferring Trade Direction from Intraday Data. J Finance 46(2):733-746

Mandes A (2015) Order placement in a continuous double auction agent based model. AlgorithmicFinance 4(3-4):105-125

Mastromatteo I, Tóth B, Bouchaud JP (2014a) Anomalous Impact in Reaction-Diffusion FinancialModels. Phys Rev Lett 113:268701

Mastromatteo I, Tóth B, Bouchaud JP (2014b) Agent-based models for latent liquidity and concaveprice impact. Phys Rev E 89:042805

16

Panayi E, Harman M, Wetherilt A (2013) Agent-based modelling of stock markets using existingorder book data. In: Giardini F, Amblard F (eds) Multi-Agent-Based Simulation XIII. Springer-Verlag, Berlin, pp 101-114

Platt DF, Gebbie TJ (2016) The Problem of Calibrating an Agent-Based Model of High-FrequencyTrading. arXiv:1606.01495

Preis T, Golke S, Paul W, Schneider (2006) Multi-agent-based Order Book Model of financialmarkets. Europhys Lett 75(3):510-516

Preis T, Golke S, Paul W, Schneider (2007) Statistical analysis of financial returns for a multiagentorder book model of asset trading. Phys Rev E 76:016108

Smith E, Farmer JD, Gillemot L, Krishnamurthy S (2003) Statistical theory of the continuous doubleauction. Quant Financ 3:481-514

Thomson Reuters (2016) Thomson Reuters Tick History. https://tickhistory.thomsonreuters.com.Accessed 23 May 2016

Tóth B, Lempérière Y, Deremble C, de Lataillade J, Kockelkoren J, Bouchaud JP (2011) AnomalousPrice Impact and the Critical Nature of Liquidity in Financial Markets. Phys Rev X 1:021006

Wilcox D, Gebbie T (2014) Hierarchical causality in financial economics. SSRN:2544327

Winker P, Gilli M, Jeleskovic V (2007) An Objective Function for Simulation Based Inference onExchange Rate Data. SSRN:964131

Ziliak ST, McCloskey DN (2004) Size matters: the standard error of regressions in the AmericanEconomic Review. J Socio Econ 33(5):527-546

17