Walk-forward backtest · Platt scaling · split-conformal prediction
If it says 80%,
it should be right about
80% of the time.
Habit trackers show you what already happened. I wanted one that says something about tomorrow and shows its working, and the awkward part there is how little data you actually have to work with — one person logging for a few months gives you somewhere between 30 and 300 rows, and they are noisy. A model will still fit that. It will just be far too sure of itself.
So everything below is measured on days the model has not seen. About 20 features, an L2 logistic regression I wrote by hand, Platt scaling on a trailing holdout, and split-conformal prediction sets, all scored one day at a time so it only ever trains on days that came before the one it is predicting. It runs in your browser.
Booting CPython and numpy in your browser…
01 Run the backtest
This runs on a generated person, because I am not going to put someone's real gym log in a repo. The pattern the generator puts in is real but weak, and the noise on top of it is deliberate, so a model that works here has to beat both baselines and keep its probabilities close to honest at the same time. One without the other does not count.
02 When there is not enough history
Drag days of history down to 50 and run it again. The conformal sets widen to {0, 1} and the average set size climbs towards 2, which is the model saying it cannot separate the two outcomes at the coverage you asked for. Push it back up to 400 and they tighten again as the calibration slice grows.
That is the coverage guarantee working. A split-conformal set has to contain the truth at the rate you asked for. When there is very little calibration data left, the only way to keep that promise is to hand back both answers. So the set gets wider and the model says less.
The CLI puts a hard rule on top of this: nothing above 70% before 21 days of data, whatever the model thinks. A model that has seen two weeks of someone should not be making confident claims yet, even when the maths says it can.
03 How it works
-
01 Baselines first
Two baselines set the bar. Persistence predicts the same as this day last week and base rate predicts the average, and between them they cover most of what looks like a pattern in habit data. If the model cannot beat both, it has only learned that you are fairly consistent. Both are scored in the table above on every run.
-
02 Keep the model linear
L2 logistic regression, written by hand, trains in milliseconds on this much data. Keeping it linear means
coefficient × featureis already the explanation, so the top three drivers come straight off the model. There is no SHAP step. -
03 Recalibrate on a holdout
Probabilities from a small-sample fit come out over-confident. Platt scaling fits a one-dimensional logistic correction on a trailing calibration slice, and that correction is what pulls the reliability curve back onto the diagonal in the chart above. It costs one extra parameter.
-
04 Guarantee coverage separately
Split-conformal prediction gives a finite-sample coverage guarantee, and it does not assume the model is well specified, which matters here because on this much data you have no reliable way to check that it is. The guarantee holds even when the model turns out to be mediocre.
-
05 Score it walk-forward
Train on days 1 to k, predict day k+1, reveal the answer, repeat. This is how the app would actually run, retraining overnight and predicting tomorrow, and it also makes look-ahead leakage impossible because the future rows are simply never in the training set when the call is made.
04 What these numbers cover
- The person here is generated. This shows the machinery is correct. Whether it can predict a real human is a separate question, and answering it needs 45 to 60 days of real logs from one person.
- A generated person is easier than a real one. Real behaviour drifts when someone changes job or gets injured or moves house, and the generator I wrote never drifts at all, so that whole failure mode is invisible from here.
- Conformal coverage comes out conservative. Around 0.88 against a 0.80 target. It over-covers, and it pays for that by returning {0, 1} on some days. That direction is the safe one, but it costs something.
- ECE is coarse at this sample size. Ten bins over 143 predictions means some bins hold only a handful of days. The gap between the raw and calibrated numbers is the useful part. The third decimal place is noise.
- Persistence is a weak baseline for gym-going. It is the honest bar for a habit and it is a low one, so beating it is necessary and nowhere near sufficient.
All of this is written up in FAILURES.md.
Read the code
Built by Vatsal Vaghasiya. MIT licensed.