How to tell a real edge from a curve fit.
Anyone can torture data until it confesses. Out-of-sample validation is the set of disciplines that stops you fooling yourself — and it’s the difference between a signal you can trust and a backtest that evaporates the moment real money touches it.
Everything works in-sample
Give a clever person enough indicators and enough freedom, and they can build a model that “predicted” the past perfectly. That proves nothing — it’s memorising, not forecasting. The only question that matters is whether a signal works on data it has never seen. That’s out-of-sample, and most flashy crypto tools quietly skip it.
Only the past predicts the next
Ioxer’s engine is walk-forward: to score a given week, it is calibrated only on weeks before it. It never sees the outcome it’s predicting. Roll the window forward one step at a time and you get a track of honest, real-time-style predictions — exactly what the model would have produced had it been running live.
If a model needs to see the answer to look smart, it isn’t a model. It’s a mirror.
Non-overlap and multiple testing
- Non-overlapping windows. Reusing the same days across many tests fakes significance. Overlap inflated our own factor t-stats and fooled us for weeks — so we now measure significance only on independent windows.
- Multiple-testing control. Test enough ideas and one looks great by pure luck. We control the false-discovery rate (Benjamini–Hochberg), so a factor only ships if it beats that higher bar — not if it just happens to win one lottery.
- Distribution-free coverage. The uncertainty band is built with conformal prediction; on data the model never trained on, the 80% band covered roughly 80% of outcomes.
The live record is the final exam
Even a clean out-of-sample backtest is a hypothesis, not proof. The real test is the live, forward track record — predictions recorded point-in-time every day and checked against what actually happened. That takes months to become statistically meaningful, and until it does we call our backtests directional, not a performance claim. Honest validation is slow on purpose.
Common questions
What does out-of-sample mean?
It means testing a signal on data it has never seen — data from after the period it was tuned on. A model is only allowed to use the past to predict the next, never the answer it is trying to predict. In-sample results (tested on the same data you fit) are nearly worthless.
What is walk-forward testing?
You fit the model on weeks 1–N, score week N+1, then roll forward and repeat. The score for each week is produced only from data before it. It mimics how the model would actually have run in real time, with no peeking ahead.
Why do non-overlapping windows matter?
Overlapping windows reuse the same days in many tests, which makes results look far more statistically significant than they are — overlap inflated factor t-stats and fooled us for weeks before we enforced non-overlap. We now measure significance only on independent windows.
Ioxer is research, not investment advice. IOX is a crowding read — not a price prediction, not a buy/sell signal.