Preventing Overfitting in Backtests: Walk-forward vs Purged K-Fold Comparison

How Overfitting Occurs

When backtest results look too good, it’s a red flag. A Sharpe ratio of 2.5 with an 8% MDD? Hardly ever replicates in live trading. Usually, the strategy is not learning the data patterns but memorizing the noise within that period.

Overfitting becomes particularly severe under three conditions: having many parameters, short validation periods, and excessive optimization attempts. This is also called “data mining bias”—trying hundreds of parameter combinations on the same data often results in coincidentally good fits.

Two common methods to prevent this are Walk-forward analysis and Purged K-Fold cross-validation.

---

Walk-forward Analysis

The concept is simple: iterate training and validation sequentially in chronological order. First, optimize the strategy over N months, then validate over the subsequent M months. Then move the window forward by M months and repeat.

results = []
for i in range(train_periods):
    train = data[i : i + train_size]
    test  = data[i + train_size : i + train_size + test_size]

    model.fit(train)
    preds = model.predict(test)
    results.append(evaluate(preds))

The biggest advantage is realism. Actual trading is also based on past data to predict the future, ensuring that future data does not leak into training.

The downside is that it consumes more data since training and testing periods do not overlap—large portions of data are used only once in validation. If your dataset spans only 3–5 years, validation samples may become quite small, reducing reliability.

---

Purged K-Fold Cross-Validation

This is a time-series adapted version of K-Fold CV. Applying standard K-Fold directly to financial data can be problematic. For example, testing fold 3 while training on folds 4 and 5 indirectly leaks future information into the past.

Purged K-Fold addresses this with two techniques:

Purging: Remove data points from the training set that are close to the test period, creating a buffer zone.

Embargo: Exclude data immediately following the test period from the training set to prevent information leakage.

kf = KFold(n_splits=5)
for train_idx, test_idx in kf.split(X):
    # Remove data within the gap around test set
    purged_train_idx = purge(train_idx, test_idx, gap=5)

    X_train, X_test = X[purged_train_idx], X[test_idx]
    y_train, y_test = y[purged_train_idx], y[test_idx]

    model.fit(X_train, y_train)
    score = model.score(X_test, y_test)

The key advantage is higher data utilization. Because the same data appears across multiple folds, the statistical power increases—especially useful when tuning hyperparameters in ML strategies.

---

Which method should you choose?

Criteria	Walk-forward	Purged K-Fold
Data efficiency	Low	High
Real-world similarity	High	Medium
Implementation complexity	Low	High
Suitability for parameter tuning	Low	High

When to prefer Walk-forward: If you have ample data (more than 10 years) and want to verify the strategy’s stability over time. Common in trend-following strategies where temporal consistency is crucial.

When Purged K-Fold is better: If your data is limited or you need to extensively tune hyperparameters. It compensates for small datasets by offering more robust validation, especially when splitting data with traditional methods results in too few samples.

In practice, it’s often best to combine both: use Purged K-Fold to narrow down parameters, then apply Walk-forward for final validation. Similar results between the two suggest low overfitting risk.

---

Checks when Suspecting Overfitting

These are practical steps to verify the reliability of your backtest results:

• Compare in-sample and out-of-sample performance. If Sharpe is 3 in-sample but 0.5 out-of-sample, it’s likely overfitted.
• Small changes in parameters dramatically alter results; perform sensitivity analysis.
• Watch out for lookahead bias, e.g., assuming entry signals are made at close prices and executed instantly. Check time indices meticulously.
• After accounting for trading costs and slippage, assess whether the strategy remains profitable. High-frequency strategies especially risk losing profits to fees.

---

Final Thoughts

A truly robust strategy works in both past and current markets. The stricter your validation methods, the lower the chance of being deceived in live trading. Whether using Walk-forward or Purged K-Fold, the core principle is preventing future information from leaking into past training data.

Related Articles

Factors and IC Analysis for Factor Combining Strategies

Quantitative Investing? An Introductory Guide for Individual Investors

Getting Started with Python Backtesting: How to Validate Your First Strategy