10 Ways Backtests Lie by Tucker Balch

10 WAYS BACKTESTS LIE

TUCKER BALCH, PH.D. PROFESSOR, GEORGIA TECH CO-FOUNDER AND CTO, LUCENA RESEARCH

OR… Mistakes quant developers make that cause backtests to be inaccurate.

WHAT I’LL COVER Introductions What is a backtest? How backtests lie: 1.  In sample backtesting 2.  Survivor bias 3.  Assume you can observe the close and trade at the close 4.  Ignoring market impact 5.  Assume you can buy $10M of a $1M company 6.  Data mining fallacy 7.  Stateful strategy luck 8.  Buy at the open 9.  Don’t trust complex models 10.  Don’t forward test

ABOUT THE SPEAKER •  Professor of Interactive Computing at

Georgia Institute of Technology. •  Teach courses in Artificial Intelligence and

Finance. •  Teach MOOCs on Machine Learning for

Trading. •  Published over 120 research publications

related to Robotics and Machine Learning. •  Co-founder of Lucena Research.

ABOUT THE SPEAKER •  Professor of Interactive Computing at

Georgia Institute of Technology. •  Teach courses in Artificial Intelligence and

Finance. •  Teach MOOCs on Machine Learning for

Trading. •  Published over 120 research publications

related to Robotics and Machine Learning. •  Co-founder of Lucena Research.

New book ->

ABOUT LUCENA RESEARCH •  We are a fin-tech company who

employ experts in Computational Finance, Quantitative Analysis, and Software Development.

•  We deliver investment decision support technology at a fraction of the cost of an in house quant shop.

•  Python-based infrastructure. •  Erez Katz, CEO •  Eric Davidson, VP http://lucenaresearch.com

LUCENA RESEARCH!

BUILDING A MODEL FROM DATA

BUILDING A MODEL FROM DATA

BUILDING A MODEL FROM DATA

BUILDING A MODEL FROM DATA

BUILDING A MODEL FROM DATA

BUILDING A MODEL FROM DATA

BUILDING A MODEL FROM DATA

BUILDING A MODEL FROM DATA

BACKTESTING TO VALIDATE THE MODEL

BACKTESTING TO VALIDATE THE MODEL

BACKTESTING TO VALIDATE THE MODEL

BACKTESTING TO VALIDATE THE MODEL

BACKTESTING TO VALIDATE THE MODEL

BACKTESTING TO VALIDATE THE MODEL

BACKTESTING TO VALIDATE THE MODEL

Roll forward cross validation Out of sample validation

10 WAYS BACKTESTS LIE

1. IN SAMPLE BACKTESTING Description: Backtesting over the same data you used to train your model.

1. IN SAMPLE BACKTESTING Description: Backtesting over the same data you used to train your model. This method is doomed to succeed spectacularly!

1. IN SAMPLE BACKTESTING Description: Backtesting over the same data you used to train your model.

Training

Testing

1. IN SAMPLE BACKTESTING How to avoid?

Training

Testing

1. IN SAMPLE BACKTESTING How to avoid? More generally, build safeguards and procedures to prevent testing over the same data you train over. E.g., Train over 2007, test over 2008-forward.

2. SURVIVOR BIAS Description: Selective use of data in a statistical study that emphasizes examples that are “alive” at the end of the study. The significance of the bias depends on how important survival is to the quantity being measured.

2. SURVIVOR BIAS Example: Company claims: “Our drug reduces the blood pressure of those who take the drug over time.” 5 year study:

•  Randomly select 500 cardiac patients •  Administer drug to them •  Measure their blood pressure monthly

Results: •  160/110 average first month •  135/80 average at end of study

Do you believe this is a good drug?

2. SURVIVOR BIAS Problem: 58 of the patients they started with have died since the start of the study.

Note: 58 of the members of the S&P 500 in 2008 are now delisted. Not just out of the S&P 500, but gone as companies. 11.6%

2. SURVIVOR BIAS

2. SURVIVOR BIAS

2. SURVIVOR BIAS

2. SURVIVOR BIAS

Green: Current S&P 500, Purple: Point in time S&P 500

--Lucena Research

2. SURVIVOR BIAS How to prevent?

•  Use historic index membership. •  Pair with SBF-free data. •  Use these indices as your universe for testing.

3. OBSERVING THE CLOSE Description: You assume you can observe information recorded at market close, and trade on it.

Examples:

•  Closing price/volume •  Technicals based on price/volume

3. OBSERVING THE CLOSE Description: You assume you can observe information recorded at market close, and trade on it.

Examples:

•  Closing price/volume •  Technicals based on price/volume

This is a specific case of “look ahead bias.” Other examples:

•  Earnings reports •  News feeds

3. OBSERVING THE CLOSE How to prevent? Ensure that information with timestamp X cannot be acted on until X+1.

Example: Data marked January 15 cannot be traded until the open on January 16.

4. IGNORING MARKET IMPACT Description: The act of trading affects price. Historical data does not include your trades and is therefore not an accurate representation of the price you would get.

4. IGNORING MARKET IMPACT

4. IGNORING MARKET IMPACT

4. IGNORING MARKET IMPACT

4. IGNORING MARKET IMPACT

Swetha Shivakumar, Georgia Tech

4. IGNORING MARKET IMPACT How to prevent? Include a “slippage” or “market impact” model in your backtests.

5. BUY $10M OF A $1M COMPANY Description: Backtest allows a strategy to buy (or short) as much of a symbol as it wants.

5. BUY $10M OF A $1M COMPANY Description: Backtest allows a strategy to buy (or short) as much of a symbol as it wants. There often is real alpha in thinly traded stocks.

5. BUY $10M OF A $1M COMPANY Description: Backtest allows a strategy to buy (or short) as much of a symbol as it wants. There often is real alpha in thinly traded stocks.

This is a specific example of the more general issue of capacity limitations.

5. BUY $10M OF A $1M COMPANY How to avoid? Ensure the backtester prohibits trading more dollar volume than actually was available on that day.

Add slippage/market impact models to penalize buying too much.

6. DATA MINING FALLACY Description: If you generate and test enough strategies you’ll eventually find one that “works” in a backtest. The quality of the strategy cannot be distinguished from random luck.

6. DATA MINING FALLACY Description: If you generate and test enough strategies you’ll eventually find one that “works” in a backtest. The quality of the strategy cannot be distinguished from random luck. Example: Look for “skilled” coin flipper among 10,000 candidates.

6. DATA MINING FALLACY How to avoid? You can’t! However you can and should forward test before committing significant capital.

7 THROUGH 10 7. Stateful strategy luck

8. Buy at the open

9. Trust complex models

10. Don’t forward test

THANK YOU! www.lucenaresearch.com