How to Minimize the Risk of Disappointment Once You Launch Your Trading System
What is Curve-Fitting?
Curve-fitting means you optimize your trading system to give nice stats in a backtest but with little chance of delivering the same great results in the future. It's like making dinner with Playdough; it will look amazing and give you lots of "likes" on Instagram, but it's guaranteed to disappoint anyone who tries it.
Why Do We Curve-Fit?
I like to say "incompetence", but it's more complicated than that. Even experienced traders (including yours truly) occasionally fall into the trap of tweaking numbers to give a backtest a false sense of promising performance. Is it because of vanity? The game-like experience of getting a high score in backtesting? The pursuit of the holy grail? Probably a bit of all. But it doesn't matter. What matters is that we recognize our flaws and do what we can to avoid them.
How to Make Your Trading System More Robust
There are many ways to test robustness, like K-fold cross-validation, walk-forward testing, and Monte Carlo simulations. This list below doesn't contain the most advanced methods but focuses on the basic things you can do, with regular backtesting software. It's a good start and will help you scrap a lot of systems. And, by that, save you a lot of money.
1. Minimize the Number of (Moving) Parts
The number one way to avoid curve-fitting is to keep your strategy simple. The more things your strategy consists of, the more you can go wrong. But there's a difference between indicators with parameters and what we can call "solid-state" indicators. Moving averages and indicators like MACD can have infinite possible values, but the open, close, and volume can only have one. Solid-state indicators are less prone to curve-fitting because they don't rely on adjustable parameters. Keep them as few as possible.
2. Stress-Test the Parameters
Despite your ambitions to slim your systems, you likely end up with a few parameters anyway. To ensure they're not over-optimized, try them with slightly different values. If your strategy is robust, it should perform well even with minor variations.
But what's a "minor" variation? For a 100-period moving average, a 105-period moving average should do just fine. That's a 5% difference. Changing the RSI value from 2 to 3, on the other hand, is a 50% increase in the lookback period. In that case, tweaking the limit for the indicator is more relevant than changing the indicator itself.
3. Know Your Indicators
If you have an indicator in your system that boosts the equity curve, but you're not sure how or why, chances are it will also make the system fail without you knowing why. By understanding how your indicators work and how they are calculated, you will know which parameters are more and less at risk of ruining future performance if you tweak them.
4. Count Out Extreme Winners
Does your backtest make most of its profits from a few trades? A small win rate and big winners can be the idea behind your strategy, but if not, you should see what happens if you remove the top 5% winners. You don't need special software to do this, just looking at the equity curve is usually enough. If it looks like a staircase (up, flat, up, flat, etc.) instead of a smooth slope upwards, your system is in danger.
5. Test on a Correlated Market
Some traders argue that a good trading system should work with any market and timeframe. While such a system would be amazing, we believe differently. Various markets and timeframes behave very differently, but to put your system to the test, it's worth running it on a highly correlated market. An SPX system usually performs well on the Nasdaq 100, and a strategy for DAX 40 often performs well on CAC 40, for example.
6. Use Sufficient Data
To avoid curve-fitting, ensure that your backtest uses a large enough dataset to include various market conditions. Testing your strategy on a dataset that spans multiple years or even decades helps capture different economic cycles, market crashes, bull markets, and periods of low volatility. This way, you can see how your strategy performs across various environments and avoid the pitfall of optimizing it for a non-recurring scenario.
What if you don't have decades' worth of data? This is often the case on smaller timeframes. One way to tackle this is to add a filter that only takes trades in the volatility and trend that your data includes. Another is to test the system on correlated markets, as discussed above. A good idea is to run the strategy on a demo account longer than normal (see point 8 below). But the truth is, your system is prone to be less reliable with less data in your backtest.
7. Use Out-of-Sample Data
Out-of-sample testing involves dividing your historical data into two sets: in-sample and out-of-sample. The in-sample data is used to develop and optimize your trading strategy, while the out-of-sample data is used to test the strategy's performance on unseen data. This helps ensure that your strategy is not simply overfitting the in-sample data.
Ideally, out-of-sample testing should be done with data that follows the in-sample period chronologically. For example, if you have 10 years of data, use the first 7 years for in-sample testing and the last 3 years for out-of-sample testing. This approach better simulates real trading scenarios where you don’t have future data.
8. Run the Strategy on a Demo Account
This is time-consuming, of course, but well worth the wait. Different from your out-of-sample test, this gives you a realistic idea of your strategy's true impact on your portfolio, including slippage, spreads, and overnight fees. This is especially important if your system includes stop or limit orders, or operates on a smaller timeframe with small profits per trade.
コメント