Backtesting That Doesn't Lie

Survivorship bias, look-ahead bias, slippage modeling, gas accounting, failed transactions, and the tools (Hummingbot, Jesse, vectorbt, LEAN) and data sources (Kaiko, Dune, Allium) that make honest backtests possible.

35 min · expert · part of Trading Algorithms & Market Microstructure

Why Most Crypto Backtests Are Wrong

Most backtests in crypto are wrong. Not slightly wrong — systematically, structurally, multiple-orders-of-magnitude wrong in the direction of overstating expected returns. The seasoned quant looks at a published backtest with a Sharpe ratio above 3 and assumes it is broken until proven otherwise, because the pitfalls are so numerous and so easy to fall into that producing an honest backtest is genuinely hard work. This lesson is a practical inventory of the ways backtests go wrong in on-chain markets, with concrete examples and the corrections that turn fantasy PnL into something resembling reality. The pitfalls cluster into a few families: data hygiene problems (survivorship and look-ahead biases), execution-quality problems (slippage and gas modeling), exclusion problems (ignoring failed transactions and oracle drift), and tooling problems (using libraries that silently make assumptions you have not validated). A useful mental discipline: before you ever look at the backtested equity curve, write down what you expect the strategy to earn on a per-trade basis. If your reasoning gives you, say, 5-10 basis points of expected edge per trade after costs, and your backtest reports 80bp per trade, the backtest is almost certainly wrong. Hunt for the bug before trusting the chart. Conversely, if you cannot articulate a per-trade economic story for where the edge comes from, the backtest is probably curve-fit noise no matter how good the equity curve looks. Honest backtesting is not just defensive accounting. It is the only path to a strategy that actually makes money in production. Every basis point of overstatement in backtest is a basis point of disappointment in live trading, and disappointments compound until you turn off the strategy and reasoning about why becomes guesswork.

Also in this lesson

Survivorship Bias: Delisted Tokens and Dead Pools
Look-Ahead Bias: Data You Could Not Have Had
Slippage, Gas, and Failed Transactions
Tooling: Hummingbot, Jesse, vectorbt, LEAN, CCXT
Data Sources: Kaiko, Dune, Allium, Tardis

Key terms

Survivorship bias: A data error where the universe of instruments is restricted to those that survived to the end of the period. In crypto this is severe because the failure rate of tokens is high and most data providers do not retain dead tokens.
Look-ahead bias: Using information in a backtest that would not have been available at the moment the strategy supposedly acted. Most common with block timestamps, daily closes, and adjusted historical data.
Slippage: The difference between expected execution price and realized execution price. In on-chain trading slippage has multiple components: price impact, MEV exposure, latency drift, and protocol fees.
Hummingbot: Open-source market-making and arbitrage framework supporting both CEX and DEX connectors. Common choice for inventory-aware market making strategies.
vectorbt: A vectorized Python backtesting library that uses NumPy operations for dramatic speedups, ideal for parameter sweeps and Monte Carlo testing.
QuantConnect LEAN: Open-source institutional-grade C# backtesting and live-trading platform supporting equities, futures, options, and crypto with rigorous handling of corporate actions and survivorship-bias-free datasets.
CCXT: A unified Python and JavaScript API client supporting 100+ centralized exchanges. Standard library for CEX live execution.
Kaiko: Institutional-grade market data provider covering 100+ CEXs and growing DEX coverage, with full order book snapshots and consolidated reference rates.
Dune Analytics: SQL interface on indexed on-chain data across many chains. Standard tool for on-chain historical research and protocol-level reconstruction.
Allium: Newer competitor to Dune providing indexed on-chain data with stronger low-latency feeds and institutional reliability.

Continue this lesson — 5 more sections in the CryptoBipto app.

Open lesson

Educational only — not financial advice.