Pseudo-Mathematics and Financial Charlatanism: The … ·  · 2014-04-01Pseudo-Mathematics and Financial Charlatanism: The Effects of Backtest Overfitting on ... rithmic investment

Pseudo-Mathematics and FinancialCharlatanism: The Effects ofBacktest Overfitting onOut-of-Sample PerformanceDavid H. Bailey, Jonathan M. Borwein,Marcos López de Prado, and Qiji Jim Zhu

Another thing I must point out is that you cannotprove a vague theory wrong. […] Also, if the processof computing the consequences is indefinite, thenwith a little skill any experimental result can bemade to look like the expected consequences.

—Richard Feynman [1964]

IntroductionA backtest is a historical simulation of an algo-rithmic investment strategy. Among other things,it computes the series of profits and losses thatsuch strategy would have generated had that al-gorithm been run over that time period. Popularperformance statistics, such as the Sharpe ratioor the Information ratio, are used to quantify thebacktested strategy’s return on risk. Investorstypically study those backtest statistics and thenallocate capital to the best performing scheme.

Regarding the measured performance of abacktested strategy, we have to distinguish betweentwo very different readings: in-sample (IS) and out-of-sample (OOS). The IS performance is the onesimulated over the sample used in the design ofthe strategy (also known as “learning period” or

David H. Bailey is retired from Lawrence Berkeley NationalLaboratory. He is a Research Fellow at the University of Cal-ifornia, Davis, Department of Computer Science. His emailaddress is [email protected].

Jonathan M. Borwein is Laureate Professor of Mathematicsat the University of Newcastle, Australia, and a Fellow of theRoyal Society of Canada, the Australian Academy of Science,and the AAAS. His email address is [email protected].

Marcos López de Prado is Senior Managing Director atGuggenheim Partners, New York, and Research Affiliate atLawrence Berkeley National Laboratory. His email addressis [email protected].

Qiji Jim Zhu is Professor of Mathematics at Western Michi-gan University. His email address is [email protected].

DOI: http://dx.doi.org/10.1090/noti1105

“training set” in the machine-learning literature).The OOS performance is simulated over a samplenot used in the design of the strategy (a.k.a. “testingset”). A backtest is realistic when the IS performanceis consistent with the OOS performance.

When an investor receives a promising backtestfrom a researcher or portfolio manager, one ofher key problems is to assess how realistic thatsimulation is. This is because, given any financialseries, it is relatively simple to overfit an investmentstrategy so that it performs well IS.

Overfitting is a concept borrowed from ma-chine learning and denotes the situation when amodel targets particular observations rather thana general structure. For example, a researchercould design a trading system based on someparameters that target the removal of specificrecommendations that she knows led to lossesIS (a practice known as “data snooping”). After afew iterations, the researcher will come up with“optimal parameters”, which profit from featuresthat are present in that particular sample but maywell be rare in the population.

Recent computational advances allow invest-ment managers to methodically search throughthousands or even millions of potential options fora profitable investment strategy. In many instances,that search involves a pseudo-mathematical ar-gument which is spuriously validated through abacktest. For example, consider a time series ofdaily prices for a stock X. For every day in thesample, we can compute one average price ofthat stock using the previous m observations xmand another average price using the previous nobservations xn, where m < n. A popular invest-ment strategy called “crossing moving averages”consists of owning X whenever xm > xn. Indeed,since the sample size determines a limited numberof parameter combinations thatm and n can adopt,

458 Notices of the AMS Volume 61, Number 5

it is relatively easy to determine the pair (m,n)that maximizes the backtest’s performance. Thereare hundreds of such popular strategies, marketedto unsuspecting lay investors as mathematicallysound and empirically tested.

In the context of econometric models severalprocedures have been proposed to determineoverfit in White [27], Romano et al. [23], andHarvey et al. [9]. These methods propose to adjustthe p-values of estimated regression coefficientsto account for the multiplicity of trials. Theseapproaches are valuable for dealing with tradingrules based on an econometric specification.

The machine-learning literature has devotedsignificant effort to studying the problem ofoverfitting. The proposed methods typically arenot applicable to investment problems for multiplereasons. First, these methods often require explicitpoint forecasts and confidence bands over adefined event horizon in order to evaluate theexplanatory power or quality of the prediction(e.g., “E-mini S&P500 is forecasted to be around1,600 with a one-standard deviation of 5 indexpoints at Friday’s close”). Very few investmentstrategies yield such explicit forecasts; instead,they provide qualitative recommendations (e.g.,“buy” or “strong buy”) over an undefined perioduntil another such forecast is generated, withrandom frequency. For instance, trading systems,like the crossing of moving averages explainedearlier, generate buy and sell recommendationswith little or no indication as to forecasted values,confidence in a particular recommendation, orexpected holding period.

Second, even if a particular investment strat-egy relies on such a forecasting equation, othercomponents of the investment strategy may havebeen overfitted, including entry thresholds, risksizing, profit taking, stop-loss, cost of capital, andso on. In other words, there are many ways tooverfit an investment strategy other than simplytuning the forecasting equation. Third, regressionoverfitting methods are parametric and involve anumber of assumptions regarding the underlyingdata which may not be easily ascertainable. Fourth,some methods do not control for the number oftrials attempted.

To illustrate this point, suppose that a researcheris given a finite sample and told that she needs tocome up with a strategy with an SR (Sharpe Ratio,a popular measure of performance in the presenceof risk) above 2.0, based on a forecasting equationfor which the AIC statistic (Akaike InformationCriterion, a standard of the regularization method)rejects the null hypothesis of overfitting with a95 percent confidence level (i.e., a false positiverate of 5 percent). After only twenty trials, theresearcher is expected to find one specification

that passes the AIC criterion. The researcher willquickly be able to present a specification that notonly (falsely) passes the AIC test but also gives anSR above 2.0. The problem is, AIC’s assessment didnot take into account the hundreds of other trialsthat the researcher neglected to mention. For thesereasons, commonly used regression overfittingmethods are poorly equipped to deal with backtestoverfitting.

Although there are many academic studies thatclaim to have identified profitable investmentstrategies, their reported results are almost alwaysbased on IS statistics. Only exceptionally do wefind an academic study that applies the “hold-out”method or some other procedure to evaluate per-formance OOS. Harvey, Liu, and Zhu [10] argue thatthere are hundreds of papers supposedly identify-ing hundreds of factors with explanatory powerover future stock returns. They echo Ioannidis[13] in concluding that “most claimed researchfindings are likely false.” Factor models are onlythe tip of the iceberg.1 The reader is probablyfamiliar with many publications solely discussingIS performance.

This situation is, quite frankly, depressing,particularly because academic researchers areexpected to recognize the dangers and practice ofoverfitting. One common criticism, of course, isthe credibility problem of “holding-out” when theresearcher had access to the full sample anyway.Leinweber and Sisk [15] present a meritoriousexception. They proposed an investment strategyin a conference and announced that six monthslater they would publish the results with thepure (yet to be observed) OOS data. They calledthis approach “model sequestration”, which is anextreme variation of “hold-out”.

Our Intentions

In this paper we shall show that it takes a relativelysmall number of trials to identify an investmentstrategy with a spuriously high backtested perfor-mance. We also compute the minimum backtestlength (MinBTL) that an investor should requiregiven the number of trials attempted. Although inour examples we always choose the Sharpe ratioto evaluate performance, our methodology can beapplied to any other performance measure.

We believe our framework to be helpful tothe academic and investment communities byproviding a benchmark methodology to assess thereliability of a backtested performance. We would

1We invite the reader to read specific instances of pseudo-mathematical financial advice at this website: http://www.m-a-f-f-i-a.org/. Also, Edesses (2007) provides numer-ous examples.

May 2014 Notices of the AMS 459

Figure 1. Overfitting a backtest’s results as thenumber of trials grows.

Figure 1 provides a graphical representation ofProposition 1. The blue (dotted) line shows the max-imum of a particular set of N independent randomnumbers, each following a Standard Normal distri-bution. The black (continuous) line is the expectedvalue of the maximum of that set of N randomnumbers. The red (dashed) line is an upper boundestimate of that maximum. The implication is that itis relatively easy to wrongly select a strategy on thebasis of a maximum Sharpe ratio when displayedIS.

feel sufficiently rewarded in our efforts if at leastthis paper succeeded in drawing the attentionof the mathematical community regarding thewidespread proliferation of journal publications,many of them claiming profitable investmentstrategies on the sole basis of IS performance. Thisis perhaps understandable in business circles, buta higher standard is and should be expected froman academic forum.

We would also like to raise the question ofwhether mathematical scientists should continueto tolerate the proliferation of investment productsthat are misleadingly marketed as mathematicallysound. In the recent words of Sir Andrew Wiles,

One has to be aware now that mathematicscan be misused and that we have to protectits good name. [29]

We encourage the reader to search the Internet forterms such as “stochastic oscillators”, “Fibonacciratios”, “cycles”, “Elliot wave”, “Golden ratio”,“parabolic SAR”, “pivot point”, “momentum”, andothers in the context of finance. Although suchterms clearly evoke precise mathematical concepts,in fact in almost all cases their usage is scientificallyunsound.

Historically, scientists have led the way in ex-posing those who utilize pseudoscience to extracta commercial benefit. As early as the eighteenthcentury, physicists exposed the nonsense of as-trologers. Yet mathematicians in the twenty-firstcentury have remained disappointingly silent withregard to those in the investment community who,knowingly or not, misuse mathematical techniquessuch as probability theory, statistics, and stochas-tic calculus. Our silence is consent, making usaccomplices in these abuses.

The rest of our study is organized as follows:The section “Backtest Overfitting” introduces theproblem in a more formal way. The section“Minimum Backtest Length (MinBTL)” defines theconcept of Minimum Backtest Length (MinBTL).The section “Model Complexity” argues how modelcomplexity leads to backtest overfitting. The section“Overfitting in Absence of Compensation Effects”analyzes overfitting in the absence of compensationeffects. The section “Overfitting in Presence ofCompensation Effects” studies overfitting in thepresence of compensation effects. The section“Is Backtest Overfitting a Fraud?” exposes howbacktest overfitting can be used to commit fraud.The section “A Practical Application” presentsa typical example of backtest overfitting. Thesection “Conclusions” lists our conclusions. Themathematical appendices supply proofs of thepropositions presented throughout the paper.

Backtest OverfittingThe design of an investment strategy usuallybegins with a prior or belief that a certain patternmay help forecast the future value of a financialvariable. For example, if a researcher recognizes alead-lag effect between various tenor bonds in ayield curve, she could design a strategy that bets ona reversion towards equilibrium values. This modelmight take the form of a cointegration equation,a vector-error correction model, or a system ofstochastic differential equations, just to name afew. The number of possible model configurations(or trials) is enormous, and naturally the researcherwould like to select the one that maximizes theperformance of the strategy. Practitioners often relyon historical simulations (also called backtests) todiscover the optimal specification of an investmentstrategy. The researcher will evaluate, amongother variables, what are the optimal sample sizes,signal update frequency, entry and profit-takingthresholds, risk sizing, stop losses, maximumholding periods, etc.

The Sharpe ratio is a statistic that evaluates aninvestment manager’s or strategy’s performance onthe basis of a sample of past returns. It is defined asthe ratio between average excess returns (in excessof the rate of return paid by a risk-free asset, such as

460 Notices of the AMS Volume 61, Number 5

a government note) and the standard deviation ofthe same returns. Intuitively, this can be interpretedas a “return on risk” (or as William Sharpe put it,“return on variability”). But the standard deviationof excess returns may be a misleading measureof variability when returns follow asymmetric orfat-tailed distributions or when returns are notindependent or identically distributed. Supposethat a strategy’s excess returns (or risk premiums),rt , are independent and identically distributed (IID)following a Normal law:

rt ∼N (µ,σ 2),(1)

where N represents a Normal distribution withmean µ and variance σ 2. The annualized Sharperatio (SR) can be computed as

SR = µσ√q,(2)

where q is the number of returns per year (see Lo[17] for a detailed derivation of this expression).Sharpe ratios are typically expressed in annualterms in order to allow for the comparison ofstrategies that trade with different frequency. Thegreat majority of financial models are built upon theIID Normal assumption, which may explain why theSharpe ratio has become the most popular statisticfor evaluating an investment’s performance.

Since µ,σ are usually unknown, the true valueSR cannot be known for certain. Instead, we canestimate the Sharpe ratio as SR = µ

σ√q, where µ

and σ are the sample mean and sample standarddeviation. The inevitable consequence is thatSR calculations are likely to be the subject ofsubstantial estimation errors (see Bailey and Lópezde Prado [2] for a confidence band and an extensionof the concept of Sharpe Ratio beyond the IIDNormal assumption).

From Lo [17] we know that the distribution of theestimated annualized Sharpe ratio SR convergesasymptotically (as y →∞) to

SR a-→N

SR, 1+ SR2

2q

y

,(3)

where y is the number of years used to estimateSR.2 As y increases without bound, the proba-bility distribution of SR approaches a Normal

distribution with mean SR and variance1+ SR

2

2qy . For a

2Most performance statistics assume IID Normal returnsand so are normally distributed. In the case of the Sharperatio, several authors have proved that its asymptotic dis-tribution follows a Normal law even when the returns arenot IID Normal. The same result applies to the InformationRatio. The only requirement is that the returns be ergodic.We refer the interested reader to Bailey and López de Prado[2].

Figure 2. Minimum Backtest Length needed toavoid overfitting, as a function of the number oftrials.

Figure 2 shows the tradeoff between the numberof trials (N) and the minimum backtest length(MinBTL) needed to prevent skill-less strategies to begenerated with a Sharpe ratio IS of 1. For instance,if only five years of data are available, no more thanforty-five independent model configurations shouldbe tried. For that number of trials, the expectedmaximum SR IS is 1, whereas the expected SR OOSis 0. After trying only seven independent strategyconfigurations, the expected maximum SR IS is 1 fora two-year long backtest, while the expected SR OOSis 0. The implication is that a backtest which doesnot report the number of trials N used to identifythe selected configuration makes it impossible toassess the risk of overfitting.

sufficiently large y , (3) provides an approximationof the distribution of SR.

Even for a small number N of trials, it isrelatively easy to find a strategy with a high Sharperatio IS but which also delivers a null Sharpe ratioOOS. To illustrate this point, consider N strategieswith T = yq returns distributed according to aNormal law with mean excess returns µ and withstandard deviation σ . Suppose that we would liketo select the strategy with optimal SR IS, basedon one year of observations. A risk we face ischoosing a strategy with a high Sharpe ratio IS butzero Sharpe ratio OOS. So we ask the question,how high is the expected maximum Sharpe ratio ISamong a set of strategy configurations where thetrue Sharpe ratio is zero?

Bailey and López de Prado [2] derived anestimate of the Minimum Track Record Length(MinTRL) needed to reject the hypothesis that anestimated Sharpe ratio is below a certain threshold

May 2014 Notices of the AMS 461

Figure 3. Performance IS vs. OOS beforeintroducing strategy selection.

Figure 3 shows the relation between SR IS (x-axis) and SR OOS (y-axis) for µ = 0, σ = 1, N =1000, T = 1000. Because the process follows arandom walk, the scatter plot has a circular shapecentered at the point (0,0). This illustrates the factthat, in absence of compensation effects, overfittingthe IS performance (x-axis) has no bearing on theOOS performance (y-axis), which remains aroundzero.

(let’s say zero). MinTRL was developed to evaluatea strategy’s track record (a single realized path,N = 1). The question we are asking now is different,because we are interested in the backtest lengthneeded to avoid selecting a skill-less strategyamong N alternative specifications. In other words,in this article we are concerned with overfittingprevention when comparing multiple strategies,not in evaluating the statistical significance ofa single Sharpe ratio estimate. Next, we willderive the analogue to MinTRL in the context ofoverfitting, which we will call Minimum BacktestLength (MinBTL), since it specifically addresses theproblem of backtest overfitting.

From (3), ifµ = 0 and y = 1, then SR a-→N (0,1).

Note that because SR = 0, increasing q does notreduce the variance of the distribution. The proofof the following proposition is left for the appendix.

Proposition 1. Given a sample of IID random vari-ables, xn ∼ Z, n = 1, . . . ,N, where Z is the CDFof the Standard Normal distribution, the expectedmaximum of that sample, E[maxN] = E[max{xn}],can be approximated for a large N as

(4)E[max

N] ≈ (1− γ)Z−1

[1− 1

N

]+ γZ−1

[1− 1

Ne−1

]

whereγ ≈ 0.5772156649 . . . is the Euler-Mascheroniconstant and N � 1.

An upper bound to (4) is√

2 ln[N].3 Figure 1plots, for various values of N (x-axis), the expectedSharpe ratio of the optimal strategy IS. For example,if the researcher tries only N = 10 alternativeconfigurations of an investment strategy, she isexpected to find a strategy with a Sharpe ratioIS of 1.57 despite the fact that all strategies areexpected to deliver a Sharpe ratio of zero OOS(including the “optimal” one selected IS).

Proposition 1 has important implications. Asthe researcher tries a growing number of strategyconfigurations, there will be a nonnull probabilityof selecting IS a strategy with null expectedperformance OOS. Because the hold-out methoddoes not take into account the number of trialsattempted before selecting a model, it cannotassess the representativeness of a backtest.

Minimum Backtest Length (MinBTL)Let us consider now the case that µ = 0 but y ≠ 1.Then, we can still apply Proposition 1 by rescalingthe expected maximum by the standard deviationof the annualized Sharpe ratio, y−1/2. Thus, theresearcher is expected to find an “optimal” strategywith an IS annualized Sharpe ratio of(5)E[max

N]

≈y−1/2((1− γ)Z−1

[1− 1

N

]+γZ−1

[1− 1

Ne−1

]).

Equation (5) says that the more independent theconfigurations a researcher tries (N), the morelikely she is to overfit, and therefore the higher theacceptance threshold should be for the backtestedresult to be trusted. This situation can be partiallymitigated by increasing the sample size (y). Bysolving (5) for y , we obtain the following statement.

Theorem 2. The Minimum Backtest Length(MinBTL, in years) needed to avoid selecting astrategy with an IS Sharpe ratio of E[maxN]among N independent strategies with an expectedOOS Sharpe ratio of zero is(6)

MinBTL

≈ (1−γ)Z−1

[1− 1

N

]+γZ−1

[1− 1

N e−1]

E[maxN]

2

<2 ln[N]

E[maxN]2 .

Equation (6) tells us that MinBTL must growas the researcher tries more independent modelconfigurations (N) in order to keep constant theexpected maximum Sharpe ratio at a given level

3See Example 3.5.4 of Embrechts et al. [5] for a detailedtreatment of the derivation of upper bounds on the maxi-mum of a Normal distribution.

462 Notices of the AMS Volume 61, Number 5

E[maxN]. Figure 2 shows how many years of back-test length (MinBTL) are needed so that E[maxN]is fixed at 1. For instance, if only five years of dataare available, no more than forty-five independentmodel configurations should be tried or we arealmost guaranteed to produce strategies with anannualized Sharpe ratio IS of 1 but an expectedSharpe ratio OOS of zero. Note that Proposition 1assumed the N trials to be independent, whichleads to a quite conservative estimate. If the trialsperformed were not independent, the number ofindependent trials N involved could be derivedusing a dimension-reduction procedure, such asPrincipal Component Analysis.

We will examine this tradeoff between N andT in greater depth later in the paper withoutrequiring such a strong assumption, but MinBTLgives us a first glance at how easy it is to overfit bymerely trying alternative model configurations. Asan approximation, the reader may find it helpfulto remember the upper bound to the minimumbacktest length (in years), MinBTL < 2 ln[N]

E[maxN]2 .

Of course, a backtest may be overfit even if it iscomputed on a sample greater than MinBTL. Fromthat perspective, MinBTL should be considereda necessary, nonsufficient condition to avoidoverfitting. We leave to Bailey et al. [1] the derivationof a more precise measure of backtest overfitting.

Model ComplexityHow does the previous result relate to modelcomplexity? Consider a one-parameter model thatmay adopt two possible values (like a switchthat generates a random sequence of trades) ona sample of T observations. Overfitting will bedifficult, because N = 2. Let’s say that we makethe model more complex by adding four moreparameters so that the total number of parametersbecomes 5, i.e., N = 25 = 32. Having thirty-twoindependent sequences of random trades greatlyincreases the possibility of overfitting.

While a greater N makes overfitting easier,it makes perfectly fitting harder. Modern super-computers can only perform around 250 rawcomputations per second, or less than 258 rawcomputations per year. Even if a trial could bereduced to a raw computation, searching N = 2100

will take us 242 supercomputer-years of compu-tation (assuming a 1 Pflop/s system, capable of1015 floating-point operations per second). Hence,a skill-less brute force search is certainly impos-sible. While it is hard to perfectly fit a complexskill-less strategy, Proposition 1 shows that thereis no need for that. Without perfectly fitting astrategy or making it overcomplex, a researchercan achieve high Sharpe ratios. A relatively simplestrategy with just seven binomial independent para-meters offersN = 27 = 128 trials, with an expected

Figure 4. Performance IS vs. performance OOSfor one path after introducing strategy selection.

Figure 4 provides a graphical representation ofwhat happens when we select the random walkwith highest SR IS. The performance of the firsthalf was optimized IS, and the performance of thesecond half is what the investor receives OOS. Thegood news is, in the absence of memory, there isno reason to expect overfitting to induce negativeperformance.

maximum Sharpe ratio above 2.6 (see Figure 1).We suspect, however, that backtested strategies

that significantly beat the market typically relyon some combination of valid insight, boostedby some degree of overfitting. Since believing insuch an artificially enhanced high-performancestrategy will often also lead to over leveraging,such overfitting is still very damaging. MostTechnical Analysis strategies rely on filters, whichare sets of conditions that trigger trading actions,like the random switches exemplified earlier.Accordingly, extra caution is warranted to guardagainst overfitting in using Technical Analysisstrategies, as well as in complex nonparametricmodeling tools, such as Neural Networks andKernel Estimators.

Here is a key concept that investors generallymiss:

A researcher that does not report the num-ber of trials N used to identify the selectedbacktest configuration makes it impossibleto assess the risk of overfitting.

Because N is almost never reported, themagnitude of overfitting in published backtests isunknown. It is not hard to overfit a backtest (indeed,the previous theorem shows that it is hard not to), sowe suspect that a large proportion of backtests pub-lished in academic journals may be misleading. The

May 2014 Notices of the AMS 463

Figure 5. Performance degradation afterintroducing strategy selection in absence of

compensation effects.

Figure 5 illustrates what happens once we add a“model selection” procedure. Now the SR IS rangesfrom 1.2 to 2.6, and it is centered around 1.7. Al-though the backtest for the selected model generatesthe expectation of a 1.7 SR, the expected SR OOSis unchanged and lies around 0.

situation is not likely to be better among practi-tioners.

In our experience, overfitting is pathologicalwithin the financial industry, where proprietaryand commercial software is developed to estimatethe combination of parameters that best fits (or,more precisely, overfits) the data. These tools allowthe user to add filters without ever reporting howsuch additions increase the probability of backtestoverfitting. Institutional players are not immuneto this pitfall. Large mutual fund groups typicallydiscontinue and replace poorly performing funds,introducing survivorship and selection bias. Whilethe motivation of this practice may be entirelyinnocent, the effect is the same as that of hidingexperiments and inflating expectations.

We are not implying that those technical ana-lysts, quantitative researchers, or fund managersare “snake oil salesmen”. Most likely they mostgenuinely believe that the backtested results arelegitimate or that adjusted fund offerings bet-ter represent future performance. Hedge fundmanagers are often unaware that most backtestspresented to them by researchers and analysts maybe useless, and so they unknowingly package faultyinvestment propositions into products. One goalof this paper is to make investors, practitioners,and academics aware of the futility of consideringbacktest without controlling for the probability ofoverfitting.

Overfitting in Absence of CompensationEffectsRegardless of how realistic the prior being testedis, there is always a combination of parametersthat is optimal. In fact, even if the prior is false, theresearcher is very likely to identify a combination ofparameters that happens to deliver an outstandingperformance IS. But because the prior is false, OOSperformance will almost certainly underperformthe backtest’s results. As we have described,this phenomenon, by which IS results tend tooutperform the OOS results, is called overfitting.It occurs because a sufficiently large number ofparameters are able to target specific data points—say by chance buying just before a rally andshorting a position just before a sell-off—ratherthan triggering trades according to the prior.

To illustrate this point, suppose we generateN Gaussian random walks by drawing from aStandard Normal distribution, each walk having asize T . Each performance pathmτ can be obtainedas a cumulative sum of Gaussian draws∆mτ = µ + σετ ,(7)

where the random shocks ετ are IID distributedετ ∼ Z,τ = 1, . . . , T . Suppose that each path hasbeen generated by a particular combination ofparameters, backtested by a researcher. Withoutloss of generality, assume that µ = 0, σ = 1, andT = 1000, covering a period of one year (with aboutfour observations per trading day). We divide thesepaths into two disjoint samples of equal size, 500,and call the first one IS and the second one OOS.

At the moment of choosing a particular pa-rameter combination as optimal, the researcherhad access to the IS series, not the OOS. For eachmodel configuration, we may compute the Sharperatio of the series IS and compare it with theSharpe ratio of the series OOS. Figure 3 shows theresulting scatter plot. The p-values associated withthe intercept and the IS performance (SR a priori)are respectively 0.6261 and 0.7469.

The problem of overfitting arises when theresearcher uses the IS performance (backtest) tochoose a particular model configuration, with theexpectation that configurations that performedwell in the past will continue to do so in thefuture. This would be a correct assumption if theparameter configurations were associated with atruthful prior, but this is clearly not the case of thesimulation above, which is the result of Gaussianrandom walks without trend (µ = 0).

Figure 4 shows what happens when we select themodel configuration associated with the randomwalk with highest Sharpe ratio IS. The performanceof the first half was optimized IS, and the per-formance of the second half is what the investorreceives OOS. The good news is that under these

464 Notices of the AMS Volume 61, Number 5

conditions, there is no reason to expect overfittingto induce negative performance. This is illustratedin Figure 5, which shows how the optimizationcauses the expected performance IS to range be-tween 1.2 and 2.6, while the OOS performance willrange between -1.5 and 1.5 (i.e., around µ, whichin this case is zero). The p-values associated withthe intercept and the IS performance (SR a priori)are respectively 0.2146 and 0.2131. Selecting anoptimal model IS had no bearing on the perfor-mance OOS, which simply equals the zero mean ofthe process. A positive mean (µ > 0) would leadto positive expected performance OOS, but suchperformance would nevertheless be inferior to theone observed IS.

Overfitting in Presence of CompensationEffectsMultiple causes create compensation effectsin practice, such as overcrowded investmentopportunities, major corrections, economic cycles,reversal of financial flows, structural breaks,bubble bursts, etc. Optimizing a strategy’s param-eters (i.e., choosing the model configuration thatmaximizes the strategy’s performance IS) doesnot necessarily lead to improved performance(compared to not optimizing) OOS, yet againleading to overfitting.

In some instances, when the strategy’s perfor-mance series lacks memory, overfitting leads to noimprovement in performance OOS. However, thepresence of memory in a strategy’s performanceseries induces a compensation effect, which in-creases the chances for that strategy to be selectedIS, only to underperform the rest OOS. Under thosecircumstances, IS backtest optimization is in factdetrimental to OOS performance.4

Global Constraint

Unfortunately, overfitting rarely has the neutralimplications discussed in the previous section. Ourprevious example was purposely chosen to exhibita globally unconditional behavior. As a result, theOOS data had no memory of what occurred IS.Centering each path to match a mean µ removesone degree of freedom:

(8) ∆mτ = ∆mτ + µ − 1T

T∑τ=1

∆mτ .

4Bailey et al. [1] propose a method to determine the degreeto which a particular backtest may have been compromisedby the risk of overfitting.

Figure 6. Performance degradation as a result ofstrategy selection under compensation effects(global constraint).

Figure 6 shows that adding a single global con-straint causes the OOS performance to be negativeeven though the underlying process was trendless.Also, a strongly negative linear relation betweenperformance IS and OOS arises, indicating that themore we optimize IS, the worse the OOS perfor-mance of the strategy.

We may rerun the same Monte Carlo experimentas before, this time on the recentered variables∆mτ . Somewhat scarily, adding this single globalconstraint causes the OOS performance to benegative even though the underlying process wastrendless. Moreover, a strongly negative linearrelation between performance IS and OOS arises,indicating that the more we optimize IS, theworse the OOS performance. Figure 6 displays thisdisturbing pattern. The p-values associated withthe intercept and the IS performance (SR a priori)are respectively 0.5005 and 0, indicating that thenegative linear relation between IS and OOS Sharperatios is statistically significant.

The following proposition is proven in theappendix.

Proposition 3. Given two alternative configura-tions (A and B) of the same model, where σAIS =σAOOS = σBIS = σBOOS imposing a global constraint

µA = µB , implies that

(9) SRAIS > SRBIS a SRAOOS < SR

BOOS .

May 2014 Notices of the AMS 465

Figure 7. Performance degradation as a result ofstrategy selection under compensation effects

(first-order serial correlation).

Figure 7 illustrates that a serially correlated perfor-mance introduces another form of compensationeffects, just as we saw in the case of a global con-straint. For example, if ϕ = 0.995, it takes about138 observations to recover half of the deviationfrom the equilibrium. We have rerun the previousMonte Carlo experiment, this time on an autore-gressive process with µ = 0, σ = 1,ϕ = 0.995, andhave plotted the pairs of performance IS vs. OOS.

Recentering a series is one way to introducememory into a process, because some data pointswill now compensate for the extreme outcomesfrom other data points. By optimizing a backtest,the researcher selects a model configuration thatspuriously works well IS and consequently is likelyto generate losses OOS.

Serial Dependence

Imposing a global constraint is not the onlysituation in which overfitting actually is detrimental.To cite another (less restrictive) example, the sameeffect happens if the performance series is seriallyconditioned, such as a first-order autoregressiveprocess,

(10) ∆mτ = (1−ϕ)µ + (ϕ − 1)ϕmτ−1 + σετor, analogously,

(11) mτ = (1−ϕ)µ +ϕmτ−1 + σετ ,where the random shocks are again IID distributedas ετ ∼ Z. The following proposition is proven inthe appendix. The number of observations thatit takes for a process to reduce its divergencefrom the long-run equilibrium by half is known asthe half-life period, or simply half-life (a familiarphysical concept introduced by Ernest Rutherfordin 1907).

Proposition 4. The half-life period of a first-orderautoregressive process with autoregressive coeffi-cient ϕ ∈ (0,1) occurs at

τ = − ln[2]ln[ϕ]

.(12)

For example, if ϕ = 0.995, it takes about 138observations to retrace half of the deviation fromthe equilibrium. This introduces another form ofcompensation effect, just as we saw in the case ofa global constraint. If we rerun the previous MonteCarlo experiment, this time for the autoregressiveprocess with µ = 0, σ = 1,ϕ = 0.995, and plotthe pairs of performance IS vs. OOS, we obtainFigure 7.

The p-values associated with the intercept andthe IS performance (SR a priori) are respectively0.4513 and 0, confirming that the negative linearrelation between IS and OOS Sharpe ratios is againstatistically significant. Such serial correlationis a well-known statistical feature, present inthe performance of most hedge fund strategies.Proposition 5 is proved in the appendix.

Proposition 5. Given two alternative configura-tions (A and B) of the same model, where σAIS =σAOOS = σBIS = σBOOS and the performance series fol-lows the same first-order autoregressive stationaryprocess,

(13) SRAIS > SRBIS a SRAOOS < SR

BOOS .

Proposition 5 reaches the same conclusion asProposition 3 (a compensation effect) withoutrequiring a global constraint.

Is Backtest Overfitting a Fraud?Consider an investment manager who emails hisstock market forecast for the next month to 2nxprospective investors, where x and n are positiveintegers. To half of them he predicts that marketswill go up, and to the other half that markets willgo down. After the month passes, he drops fromhis list the names to which he sent the incorrectforecast, and sends a new forecast to the remaining2n−1x names. He repeats the same procedure ntimes, after which only x names remain. These xinvestors have witnessed n consecutive infallibleforecasts and may be extremely tempted to give thisinvestment manager all of their savings. Of course,this is a fraudulent scheme based on randomscreening: The investment manager is hiding thefact that for every one of the x successful witnesses,he has tried 2n unsuccessful ones (see Harris [8, p.473] for a similar example).

To avoid falling for this psychologically com-pelling fraud, a potential investor needs to considerthe economic cost associated with manufacturingthe successful experiments, and require the invest-ment manager to produce a number n for which

466 Notices of the AMS Volume 61, Number 5

the scheme is uneconomic. One caveat is, even if nis too large for a skill-less investment manager, itmay be too low for a mediocre investment managerwho uses this scheme to inflate his skills.

Not reporting the number of trials (N) involvedin identifying a successful backtest is a similar kindof fraud. The investment manager only publicizesthe model that works but says nothing about all thefailed attempts, which as we have seen can greatlyincrease the probability of backtest overfitting.

An analogous situation occurs in medical re-search, where drugs are tested by treating hundredsor thousands of patients; however, only the bestoutcomes are publicized. The reality is that theselected outcomes may have healed in spite of(rather than thanks to) the treatment or due toa placebo effect (recall Proposition 1). Such be-havior is unscientific—not to mention dangerousand expensive—and has led to the launch of thealltrials.net project, which demands that all results(positive and negative) for every experiment aremade publicly available. A step forward in thisdirection is the recent announcement by Johnson &Johnson that it plans to open all of its clinical testresults to the public [14]. For a related discussionof reproducibility in the context of mathematicalcomputing, see Stodden et al. [25].

Hiding trials appears to be standard procedurein financial research and financial journals. Asan aggravating factor, we know from the section“Overfitting in Presence of Compensation Effects”that backtest overfitting typically has a detrimentaleffect on future performance due to the compensa-tion effects present in financial series. Indeed, thecustomary disclaimer “past performance is not anindicator of future results” is too optimistic in thecontext for backtest overfitting. When investmentadvisers do not control for backtest overfitting,good backtest performance is an indicator ofnegative future results.

A Practical ApplicationInstitutional asset managers follow certain in-vestment procedures on a regular basis, such asrebalancing the duration of a fixed income port-folio (PIMCO); rolling holdings on commodities(Goldman Sachs, AIG, JP Morgan, Morgan Stanley);investing or divesting as new funds flow at the endof the month (Fidelity, BlackRock); participating inthe regular U.S. Treasury Auctions (all major invest-ment banks); delevering in anticipation of payroll,FOMC or GDP releases; tax-driven effects aroundthe end of the year and mid-April; positioning forelectoral cycles, etc. There are a large number ofinstances where asset managers will engage insomewhat predictable actions on a regular basis.It should come as no surprise that a very popular

Figure 8. Backtested performance of a seasonalstrategy (Example 6).

We have generated a time series of 1000 daily prices(about four years) following a random walk. ThePSR-Stat of the optimal model configuration is 2.83,which implies a less-than 1 percent probability thatthe true Sharpe ratio is below 0. Consequently, wehave been able to identify a plausible seasonal strat-egy with an SR of 1.27 despite the fact that no trueseasonal effect exists.

investment strategy among hedge funds is to profitfrom such seasonal effects.

For example, a type of question often askedby hedge fund managers follows the form: “Isthere a time interval every [ ] when I wouldhave made money on a regular basis?” Youmay replace the blank space with a word likeday, week, month, quarter, auction, nonfarmpayroll (NFP) release, European Central Bank (ECB)announcement, presidential election year, . . . . Thevariations are as abundant as they are inventive.Doyle and Chen [4] study the “weekday effect” andconclude that it appears to “wander”.

The problem with this line of questioningis that there is always a time interval that isarbitrarily “optimal” regardless of the cause. Theanswer to one such question is the title of a verypopular investment classic, Do Not Sell Stocks onMonday , by Hirsch [12]. The same author wrote analmanac for stock traders that reached its forty-fifthedition in 2012 and he is also a proponent of the“Santa Claus Rally”, the quadrennial political/stockmarket cycle, and investing during the “BestSix Consecutive Months” of the year, Novemberthrough April. While these findings may indeedbe caused by some underlying seasonal effect,it is easy to demonstrate that any random datacontains similar patterns. The discovery of apattern IS typically has no bearing OOS, yet again isa result of overfitting. Running such experiments

May 2014 Notices of the AMS 467

without controlling for the probability of backtestoverfitting will lead the researcher to spuriousclaims. OOS performance will disappoint, and thereason will not be that “the market has foundout the seasonal effect and arbitraged away thestrategy’s profits.” Rather, the effect was neverthere; instead, it was just a random pattern thatgave rise to an overfitted trading rule. We willillustrate this point with an example.

Example 6. Suppose that we would like to iden-tify the optimal monthly trading rule given fourcustomary parameters: Entry day, Holding period,Stop loss, and Side. Side defines whether we willhold long or short positions on a monthly basis.Entry day determines the business day of themonth when we enter a position. Holding periodgives the number of days that the position is held.Stop loss determines the size of the loss (as a mul-tiple of the series’ volatility) that triggers an exitfor that month’s position. For example, we couldexplore all nodes that span the set {1, . . . ,22} forEntry day, the set {1, . . . ,20} for Holding period,the set {0, . . . ,10} for Stop loss, and {−1,1} forSide. The parameter combinations involved forma four-dimensional mesh of 8,800 elements. Theoptimal parameter combination can be discoveredby computing the performance derived by eachnode.

First, we generated a time series of 1,000 dailyprices (about four years), following a random walk.Figure 8 plots the random series, as well as theperformance associated with the optimal param-eter combination: Entry day = 11, Holding period= 4, Stop loss = −1 and Side = 1. The annualizedSharpe ratio is 1.27.

Given the elevated Sharpe ratio, we could con-clude that this strategy’s performance is signifi-cantly greater than zero for any confidence level.Indeed, the PSR-Stat is 2.83, which implies a lessthan 1 percent probability that the true Sharperatio is below 0.5 Several studies in the practition-ers and academic literature report similar results,which are conveniently justified with some ex-postexplanation (“the posterior gives rise to a prior”).What this analysis misses is an evaluation of theprobability that this backtest has been overfit tothe data, which is the subject of Bailey et al. [1].

In this practical application we have illustratedhow simple it is to produce overfit backtests whenanswering common investment questions, suchas the presence of seasonal effects. We refer thereader to the appendix section “Reproducing the

5The Probabilistic Sharpe Ratio (or PSR) is an extension tothe SR. Nonnormality increases the error of the varianceestimator, and PSR takes that into consideration when de-termining whether an SR estimate is statistically significant.See Bailey and López de Prado [2] for details.

Results in Example 6” for the implementation ofthis experiment in the Python language. Similarexperiments can be designed to demonstrateoverfitting in the context of other effects, suchas trend following, momentum, mean-reversion,event-driven effects, etc. Given the facility withwhich elevated Sharpe ratios can be manufacturedIS, the reader would be well advised to remainhighly suspicious of backtests and of researcherswho fail to report the number of trials attempted.

ConclusionsWhile the literature on regression overfitting isextensive, we believe that this is the first studyto discuss the issue of overfitting on the subjectof investment simulations (backtests) and itsnegative effect on OOS performance. On thesubject of regression overfitting, the great EnricoFermi once remarked (Mayer et al. [20]):

I remember my friend Johnny von Neumannused to say, with four parameters I can fitan elephant, and with five I can make himwiggle his trunk.

The same principle applies to backtesting, withsome interesting peculiarities. We have shown thatbacktest overfitting is difficult indeed to avoid. Anyperseverant researcher will always be able to find abacktest with a desired Sharpe ratio regardless ofthe sample length requested. Model complexity isonly one way that backtest overfitting is facilitated.Given that most published backtests do not reportthe number of trials attempted, many of themmay be overfitted. In that case, if an investorallocates capital to them, performance will vary: Itwill be around zero if the process has no memory,but it may be significantly negative if the processhas memory. The standard warning that “pastperformance is not an indicator of future results”understates the risks associated with investingon overfit backtests. When financial advisors donot control for overfitting, positive backtestedperformance will often be followed by negativeinvestment results.

We have derived the expected maximum Sharperatio as a function of the number of trials (N) andsample length. This has allowed us to determinethe Minimum Backtest Length (MinBTL) needed toavoid selecting a strategy with a given IS Sharperatio among N trials with an expected OOS Sharperatio of zero. Our conclusion is that the more trialsa financial analyst executes, the greater shouldbe the IS Sharpe ratio demanded by the potentialinvestor.

We strongly suspect that such backtest over-fitting is a large part of the reason why so manyalgorithmic or systematic hedge funds do not liveup to the elevated expectations generated by theirmanagers.

468 Notices of the AMS Volume 61, Number 5

We would feel sufficiently rewarded in our effortsif this paper succeeds in drawing the attention ofthe mathematical community to the widespreadproliferation of journal publications, many ofthem claiming profitable investment strategies onthe sole basis of in-sample performance. This isunderstandable in business circles, but a higherstandard is and should be expected from anacademic forum.

A depressing parallel can be drawn betweentoday’s financial academic research and the situa-tion denounced by economist and Nobel LaureateWassily Leontief writing in Science (see Leontief[16]):

A dismal performance. . . “What economists revealed most

clearly was the extent to which theirprofession lags intellectually.” This edi-torial comment by the leading economicweekly (on the 1981 annual proceedings ofthe American Economic Association) says,essentially, that the “king is naked.” But noone taking part in the elaborate and solemnprocession of contemporary U.S. academiceconomics seems to know it, and those whodo don’t dare speak up.[. . .][E]conometricians fit algebraic functions

of all possible shapes to essentially thesame sets of data without being able toadvance, in any perceptible way, a system-atic understanding of the structure and theoperations of a real economic system.[. . .]That state is likely to be maintained as

long as tenured members of leading eco-nomics departments continue to exercisetight control over the training, promotion,and research activities of their youngerfaculty members and, by means of peerreview, of the senior members as well.

We hope that our distinguished colleagues willfollow this humble attempt with ever-deeper andmore convincing analysis. We did not write thispaper to settle a discussion. On the contrary, ourwish is to ignite a dialogue among mathematiciansand a reflection among investors and regulators.We should do well also to heed Newton’s commentafter he lost heavily in the South Seas bubble; see[21]:

For those who had realized big losses orgains, the mania redistributed wealth. Thelargest honest fortune was made by ThomasGuy, a stationer turned philanthropist, whoowned £54,000 of South Sea stock in April1720 and sold it over the following six weeksfor £234,000. Sir Isaac Newton, scientist,

master of the mint, and a certifiably rationalman, fared less well. He sold his £7,000 ofstock in April for a profit of 100 percent.But something induced him to reenter themarket at the top, and he lost £20,000. “Ican calculate the motions of the heavenlybodies,” he said, “but not the madness ofpeople.”

AppendicesProof of Proposition 1

Embrechts et al. [5, pp. 138–147] show that themaximum value (or last order statistic) in a sampleof independent random variables following anexponential distribution converges asymptoticallyto a Gumbel distribution. As a particular case, theGumbel distribution covers the Maximum Domainof Attraction of the Gaussian distribution, andtherefore it can be used to estimate the expectedvalue of the maximum of several independentrandom Gaussian variables.

To see how, suppose there is a sample of IIDrandom variables, zn ∼ Z,n = 1, . . . ,N, where Z isthe CDF of the Standard Normal distribution. Toderive an approximation for the sample maximum,maxn = max{zn}, we apply the Fisher-Tippett-Gnedenko theorem to the Gaussian distributionand obtain that

(14) limN→∞

Prob[

maxN −αβ

≤ x]= G[x],

where

• G[x] = e−e−x is the CDF for the StandardGumbel distribution.

• α = Z−1[1− 1

N

], β = Z−1

[1− 1

N e−1]−α,

and Z−1 corresponds to the inverse of theStandard Normal’s CDF.

The normalizing constants (α,β) are derived inResnick [22] and Embrechts et al. [5]. The limit ofthe expectation of the normalized maxima from adistribution in the Gumbel Maximum Domain ofAttraction (see Proposition 2.1(iii) in Resnick [22])is

(15) limN→∞

E[

maxN −αβ

]= γ,

where γ is the Euler-Mascheroni constant, γ ≈0.5772156649 . . .. Hence, for N sufficiently large,the mean of the sample maximum of standardnormally distributed random variables can beapproximated by(16)E[max

N]

≈α+γβ = (1− γ)Z−1[

1− 1N

]+γZ−1

[1− 1

Ne−1]

where N > 1.

May 2014 Notices of the AMS 469

Proof of Proposition 3

Suppose there are two random samples (A andB) of the same process {∆mτ}, where A and Bare of equal size and have means and standarddeviations µA, µB , σA, σB . A fraction δ of eachsample is called IS, and the remainder is calledOOS, where for simplicity we have assumed thatσAIS = σAOOS = σBIS = σBOOS . We would like tounderstand the implications of a global constraintµA = µB .

First, we note that µA = δµAIS + (1 − δ)µAOOSand µB = δµBIS + (1−δ)µBOOS . Then, µAIS > µ

AOOS a

µAIS > µA a µAOOS < µA. Likewise, µBIS > µBOOS a

µBIS > µB a µBOOS < µB .Second, because of the global constraintµA = µB ,

µAIS + (1−δ)δ µAOOS = µBIS + (1−δ)

δ µBOOS and µAIS −µBIS =(1−δ)δ (µBOOS − µAOOS). Then, µAIS > µ

BIS a µAOOS <

µBOOS . We can divide this expression by σAIS > 0,with the implication that

(17) SRAIS > SRBIS a SRAOOS < SR

BOOS

where we have denoted SRAIS =µAISσAIS

, etc. Note that

we did not have to assume that ∆mτ is IID, thanksto our assumption of equal standard deviations.The same conclusion can be reached withoutassuming equality of standard deviations; however,the proof would be longer but no more revealing(the point of this proposition is the implication ofglobal constraints).

Proof of Proposition 4

This proposition computes the half-life of a first-order autoregressive process. Suppose there is arandom variablemτ that takes values of a sequenceof observations τ ∈ {1, . . . ,∞}, where

(18) mτ = (1−ϕ)µ +ϕmτ−1 + σετsuch that the random shocks are IID distributedas ετ ∼ N(0,1). Then

limτ→∞

E0[mτ] = µ

if and only ifϕ ∈ (−1,1). In particular, from Baileyand López de Prado [3] we know that the expectedvalue of this process at a particular observation τis

(19) E0[mτ] = µ(1−ϕτ)+ϕτm0.

Suppose that the process is initialized or resetat some value m0 ≠ µ. We ask the question, howmany observations must pass before

(20) E0[mτ] =µ +m0

2?

Inserting (20) into (19) and solving for τ , we obtain

(21) τ = − ln[2]ln[ϕ]

,

which implies the additional constraint that ϕ ∈(0,1).

Proof of Proposition 5

Suppose that we draw two samples (A and B) ofa first-order autoregressive process and generatetwo subsamples of each. The first subsample iscalled IS and is comprised of τ = 1, . . . , δT , and thesecond subsample is called OOS and is comprisedof τ = δT + 1, . . . , T , with δ ∈ (0,1) and T aninteger multiple of δ. For simplicity, let us assumethatσAIS = σAOOS = σBIS = σBOOS . From Proposition 4,(18), we obtain

(22) EδT [mT ]−mδT = (1−ϕT )(µ −mδT ).Because 1 −ϕT > 0, σAIS = σBIS , SRAIS > SRBIS amAδT > mBδT . This means that the OOS of Abegins with a seed that is greater than the seedthat initializes the OOS of B. Therefore, mAδT >mBδT a EδT [mAT ]−mAδT < EδT [mBT ]−mBδT . Because

σBIS = σBOOS , we conclude that

(23) SRAIS > SRBIS a SRAOOS < SR

BOOS

Reproducing the Results in Example 6

Python code implementing the experiment de-scribed in “A Practical Application” can be foundathttp://www.quantresearch.info/Software.htm and at http://www.financial-math.org/software/.

Acknowledgments

We are indebted to the editor and two anonymousreferees who peer-reviewed this article for theNotices of the American Mathematical Society. Weare also grateful to Tony Anagnostakis (MooreCapital), Marco Avellaneda (Courant Institute, NYU),Peter Carr (Morgan Stanley, NYU), Paul Embrechts(ETH Zürich), Matthew D. Foreman (Universityof California, Irvine), Jeffrey Lange (GuggenheimPartners), Attilio Meucci (KKR, NYU), Natalia Nolde(University of British Columbia ad ETH Zürich), andRiccardo Rebonato (PIMCO, University of Oxford).The opinions expressed in this article are theauthors’ and they do not necessarily reflect theviews of the Lawrence Berkeley National Laboratory,Guggenheim Partners, or any other organization.No particular investment or course of action isrecommended.

References[1] D. Bailey, J. Borwein, M. López de Prado

and J. Zhu, The probability of backtest over-fitting, 2013, working paper, available athttp://ssrn.com/abstract=2326253.

[2] D. Bailey and M. López de Prado, The Sharpe ratioefficient frontier, Journal of Risk 15(2) (2012), 3–44.Available at http://ssrn.com/abstract=1821643.

470 Notices of the AMS Volume 61, Number 5

[3] , Drawdown-based stop-outs and the triplepenance rule, 2013, working paper. Available athttp://ssrn.com/abstract=2201302.

[4] J. Doyle and C. Chen, The wandering weekday ef-fect in major stock markets, Journal of Banking andFinance 33 (2009), 1388–1399.

[5] P. Embrechts, C. Klueppelberg and T. Mikosch,Modelling Extremal Events, Springer-Verlag, New York,2003.

[6] R. Feynman, The Character of Physical Law, The MITPress, 1964.

[7] J. Hadar and W. Russell, Rules for ordering uncertainprospects, American Economic Review 59 (1969), 25–34.

[8] L. Harris, Trading and Exchanges: Market Mi-crostructure for Practitioners, Oxford University Press,2003.

[9] C. Harvey and Y. Liu, Backtesting, working paper,SSRN, 2013, http://ssrn.com/abstract=2345489.

[10] C. Harvey, Y. Liu, and H. Zhu, . . . and the cross-section of expected returns, working paper, SSRN,2013, http://ssrn.com/abstract=2249314.

[11] D. Hawkins, The problem of overfitting, Journal ofChemical Information and Computer Science 44 (2004),1–12.

[12] Y. Hirsch, Don’t Sell Stocks on Monday , Penguin Books,1st edition, 1987.

[13] J. Ioannidis, Why most published research findingsare false, PLoS Medicine 2(8), August 2005.

[14] H. Krumholz Give the data to the people, NewYork Times, February 2, 2014. Available athttp://www.nytimes.com/2014/02/03/opinion/give-the-data-to-the-people.html.

[15] D. Leinweber and K. Sisk, Event driven trading andthe “new news”, Journal of Portfolio Management 38(1)(2011), 110–124.

[16] W. Leontief, Academic economics, Science Magazine(July 9, 1982), 104–107.

[17] A. Lo, The statistics of Sharpe ratios, Financial An-alysts Journal 58 4 (Jul/Aug, 2002). Available athttp://ssrn.com/abstract=377260.

[18] M. López de Prado and A. Peijan, Measuring theloss potential of hedge fund strategies, Journal ofAlternative Investments 7(1), (2004), 7–31. Availableat http://ssrn.com/abstract=641702.

[19] M. López de Prado and M. Foreman, A mixtureof Gaussians approach to mathematical portfo-lio oversight: The EF3M algorithm, working paper,RCC at Harvard University, 2012. Available athttp://ssrn.com/abstract=1931734.

[20] J. Mayer, K. Khairy, and J. Howard, Drawing anelephant with four complex parameters, AmericanJournal of Physics 78(6) (2010).

[21] C. Reed, The damn’d South Sea, Harvard Magazine(May–June 1999).

[22] S. Resnick, Extreme Values, Regular Variation andPoint Processes, Springer, 1987.

[23] J. Romano and M. Wolf, Stepwise multiple testing asformalized data snooping, Econometrica 73(4) (2005),1273–1282.

[24] F. Schorfheide and K. Wolpin, On the use of hold-out samples for model selection, American EconomicReview 102(3) (2012), 477–481.

[25] V. Stodden, D. Bailey, J. Borwein, R. LeVeque,W. Rider, and W. Stein, Setting the default toreproducible: Reproducibility in computational and

experimental mathematics, February 2013. Availableat http://www.davidhbailey.com/dhbpapers/icerm-report.pdf.

[26] G. Van Belle and K. Kerr, Design and Analysis ofExperiments in the Health Sciences, John Wiley & Sons.

[27] H. White, A reality check for data snooping,Econometrica 68(5), 1097–1126.

[28] S. Weiss and C. Kulikowski, Computer Systems ThatLearn: Classification and Prediction Methods fromStatistics, Neural Nets, Machine Learning and ExpertSystems, 1st edition, Morgan Kaufman,1990.

[29] A. Wiles, Financial greed threatens the good nameof maths, The Times (04 Oct 2013). Available onlineat http://www.thetimes.co.uk/tto/education/article3886043.ece.

May 2014 Notices of the AMS 471