Which ignition deserves the second engine?

Where the models stood

Round 1's models were honest about their protocol and untouched by craft. Before improving anything, here is the inventory of what was actually wrong.

• Nothing had ever been tuned. The model factory accepted hyperparameters and was called with none, at all six call sites. Every published score came from defaults.
• The validation era was computed and thrown away. The 2015–2017 rows were counted, reported, and never used: a free tuning and calibration surface left idle.
• No interval anywhere. Every number was a bare point estimate. "0.870 beats 0.787" was asserted without asking whether the gap survives resampling.
• The growth-risk model was scored at a threshold that means nothing. On a 2% base rate, predict() at 0.5 is not a decision rule.
• The leak probe didn't bite (+0.0024), and I framed that as a demonstration of leakage. It wasn't. Section 10 explains why it was pointed at the wrong task.
• Aspect went in as raw degrees, so a slope facing 359° and one facing 1° looked maximally different to the model.

train  <= 2014          model fitting + hyperparameter search (spatial-block CV)
val    2015-2017        calibration mapping, model choice
test   2018-2020        scored once per model family FOR SELECTION (gated in code)

search CV = GroupKFold(3) over 1-degree spatial blocks, train era only
metric    = average precision (the base rate is the no-skill line)

TL;DR

Restating the question as a decision

"Predict the cause" is a classification exercise. "Which of these ignitions will become something" is a resource-allocation problem with a budget, and it is measured differently.

California reported 26,142 ignitions in the 2018–2020 test era. 538 of them reached 100 acres, 2.06%. That number governs everything downstream: it is the no-skill line for average precision, it makes accuracy meaningless (a model that says "no" to everything scores 97.9%), and it sets hard ceilings on what any ranking can achieve within a fixed response budget.

Those ceilings are arithmetic, not modeling: with 261 slots you cannot catch 538 fires. Every recall number in this article is reported against its ceiling, because a model that captures 45% of escalations inside a budget whose maximum is 49% is doing something very different from one that scores 45% out of a possible 100%.

The feature foundry

Going back to the 308-column source for what the pipeline never took, then building the variables fire scientists actually use, from 30 raw columns to 86.

Cache v2: 29 columns the project had been ignoring

The biggest single omission was wind direction. Wind speed was cached; direction was not, which makes the defining mechanism of California's most destructive fires unmodelable. A Diablo or Santa Ana event is not just "windy": it is wind blowing downslope, aligned with the terrain, dry and warming as it descends. With direction and aspect together that becomes a feature:

wind blows TOWARD  = wind_dir + 180
alignment          = cos(radians(wind_toward - aspect))
downslope_wind     = alignment x wind_speed x sin(radians(slope))

Also added: climatological normals (so today can be expressed as an anomaly), fine-fuel load (rangeland production, cheatgrass and exotic annual grass cover), vegetation structure (canopy cover, height, type), terrain ruggedness, and the remaining same-day weather needed for published fire-weather indices.

Collinearity is severe enough to make single-feature importance meaningless, which is why section 13 permutes in groups: cheatgrass and exotic annual grass are correlated at 1.00, canopy cover and vegetation height at 0.997, slope and terrain ruggedness at 0.971, and 1000-hour fuel moisture against 5-day maximum ERC at −0.96.

Three search stacks, one budget

The interesting question is not "what are the best hyperparameters" but "does the modern tuning stack beat a plain one when both get the same number of tries?"

Same spatial-block cross-validation, same metric, and the same trial count per track, corrected by the Round-4 review from an earlier "wall-clock" claim that was backwards. Two disclosed asymmetries survive: the tracks' actual wall clocks were 6.7 / 8.3 / 9.0 minutes (the winner consumed 35% more compute than the loser), and LightGBM's search space carries a row-subsampling dimension the two HGB tracks have no equivalent for. The bake-off compares stacks as practitioners use them, not perfectly matched estimators:

All three stacks installed and ran on Python 3.14, worth stating, since two of the three ship no version-specific wheel for it and the run depended on a stable-ABI build and a pure-Python package.

The zoo, with intervals

Nine configurations (seven learners and two baselines), each scored once on the test era, each with a spatial-block bootstrap interval, and paired tests that ask whether the gaps are real.

Two baselines are in that table deliberately. Climatology is the historical escalation rate for the fire's own place and month, the number you get from a groupby with no model at all. ERC-only ranks fires by a single fire-danger index. A model that cannot clear both is not earning its complexity.

The number I got backwards

This section exists because the main article shipped a claim that is not true, and the correction is more interesting than the original claim.

The cause is class_weight="balanced". Re-weighting the rare class is a reasonable way to make a model rank well on imbalanced data, and it worked: the ranking metrics were real. But it also tells the model that positives are as common as negatives, so its probabilities come out centred near 0.5 on a problem whose true rate is 2%. A roughly 24-fold systematic overstatement of risk.

The reliability curve I published did not catch it because I read the shape and never computed the skill. The curve bends in the right direction; the axis it bends on was wrong by an order of magnitude.

The fix uses the era Round 1 discarded: fit the model on training years, learn a mapping from raw scores to honest probabilities on the 2015–2017 validation era, apply it to the test years. The val-selected isotonic mapping moves the Brier skill score from +0.050 to +0.053 and the expected calibration error from 0.007 to 0.0035. It is not free: isotonic's step-function ties cost 0.013 of AP (0.124 → 0.111 on the seed-42 fit). Honest probabilities here carry a small ranking price, and both sides of that trade are printed. The calibration slope moves from 0.95 to 0.96 and the intercept from 0.267 to 0.057 (1 and 0 are ideal).

Decisions, not scores

With honest probabilities, the model can be evaluated the way it would be used: under a budget, against a cost ratio.

What the feature work actually bought

Every engineered family, added one rung at a time, measured with intervals, including the rungs that bought nothing.

Leakage, properly demonstrated

Round 1's leak probe moved the score by 0.0024 and I presented it as a warning. It deserves a better demonstration, and an explanation of why the first one was flat.

Why the original probe was right to do nothing

That probe added fire size to a model predicting cause. Fire size is not where a cause label comes from, and the model already knew "remote high-country lightning country" through elevation, ecoregion, vegetation and coordinates. Adding one weakly-informative column to a model that already had the information through other channels correctly changed almost nothing. The measurement was right; my framing of it was wrong.

A probe that bites

On the escalation task, fire size is the label source, so leakage can be demonstrated properly. The interesting rungs are not the obvious ones:

Does it hold up out of sample?

Two stress tests: across years, and across a completely different fire regime. (The third drift the audit flagged, cause-label availability, is answered in §03 by refusing to model that label at all.)

The transfer test is the harsh one. Florida's fire regime shares almost nothing with California's: it burns in spring, at high frequency, on flat terrain, with prescribed fire as a dominant land-management practice. A California-trained model applied to Florida is being asked whether it learned fire behaviour or California. The transfer result says California. Transferred AP is 0.180 against 0.293 for a Florida-trained model, still above Florida's 4.29% base rate, so some of what it knows is fire behaviour rather than geography, but the gap is the honest headline.

What generalizes, and what this model cannot answer

Two questions a reader is entitled to ask and this project had not answered: how much of the score is memorization, and can any of this predict a fire before it starts.

The in-sample gap, measured

Until now every argument here against overfitting was indirect. The direct measurement is one fit scored on all three eras, and it had never been run. Nothing else in this project quotes a training score, which is correct practice, but it also meant the gap was unknown rather than small.

The gap is large. Average precision falls from 0.538 in sample to 0.124 out of sample, a difference of 0.415. A boosted ensemble of 360 trees at 61 leaves each has ample capacity to memorize 200,967 rows, and it used it. That is the reason no score in this project is ever quoted from the training era.

What matters for deployment is the behaviour of the held-out estimates, and there the picture is stable. The validation era (2015–2017) and the test era (2018–2020) are independent future periods at almost identical base rates, and they score 0.813 and 0.817 ROC-AUC. A model that had fitted era-specific noise would decay as the evaluation moves further from training. This one does not: the later era scores slightly higher on both metrics. The drop happens once, at the boundary between in-sample and out-of-sample, and then it stops.

The pipeline can still fail

A held-out score means nothing if the evaluation code would produce it from noise. Refitting the same configuration on permuted training labels and scoring the real test era gives AP 0.0232 against a base rate of 0.0206, a lift of 1.13, and ROC-AUC 0.513. Destroying the labels destroys the skill, which is what should happen and had only ever been demonstrated for the Round-1 cause model, not for this one.

Can this predict fires before they start?

No, and the reason is structural rather than a matter of accuracy. This model estimates P(escalation | a fire was reported). Every quantity it uses is conditioned on an ignition that already happened and was written down. Predicting ignition is a different estimand: P(a fire starts | this place, this day), and answering it requires the days and places where no fire started.

Those rows do not exist in this project. Every cache is fire-conditional. Dividing California into 0.25° cells that recorded at least one fire in 29 years gives 740 cells over 10,593 days, or 7.8 million cell-days. The caches contain 251,882 of them. The other 7.6 million cell-days, the ones where nothing ignited, are absent, and they are exactly the negative class an ignition model would learn from.

Rarity is not the obstacle, which was the first thing worth checking. An ignition occurs on 3.2% of those cell-days, so the positive class would be about as common as escalation is among reported fires. The obstacles are that the negatives are missing, and that they cannot be manufactured: every weather column here is sampled at the ignition point on the ignition date, so a cell-day with no fire has no covariates to describe it. Building that dataset means pulling gridded daily weather rasters, which is a data acquisition problem rather than a modelling one.

A second measurement bounds the question further. California recorded at least one ignition on 96.1% of the 10,593 days in this record. The binary question "will a fire start in California today" is answered yes almost every day and is not worth modelling. A useful ignition product would estimate spatial intensity, how many and where, which is a point-process problem rather than the classification this project is built around.

What the model actually uses

With features correlated up to 1.00, permuting one at a time measures nothing. The reliable read permutes whole families together.

Scoring the fire that started this

The model is trained on data through 2014. The Tubbs Fire ignited on October 8, 2017. It is not in the training data, the validation data, or any tuning decision.

Negative results

Its own section, because "we tried it and it didn't help" is a finding, and because a stage that reports only its wins is incomplete. Every null below carries an interval, because "no effect" and "no power to detect an effect" are different sentences.

Red team: the hostile review

Before this page went anywhere near a stakeholder, a fresh-context adversarial reviewer audited every line of the Round-3 code with instructions to be merciless. It returned 35 findings. This section is the triage of what was broken and fixed, what was overstated and amended, and what was checked and held, with every new number produced by run_redteam.py and enforced by the verifier like everything else.

The four that mattered most

The experiments the review demanded

Every refit above is a single seed-42 run compared against its own seed-42 baseline (AP 0.124); the 5-seed system-of-record mean is 0.126.

Questions your boss will ask

The full accounting

Gated selection touches of the test era: 9 (one per zoo family). Diagnostic evaluations, reported in full and never selected on: calibration variants 3, threshold ladder 4, ablation 7, leak ladder 5, by-year 3, transfer 2, grouped permutation 40, single-feature permutation 258, SHAP 1, Tubbs 1, and this red team's own refits, same rule. The earned null the review demanded: Nadeau–Bengio corrected t on the bake-off fold vectors gives p = 0.50 for Optuna-vs-random, so the "the tuner didn't matter" claim now has a test behind it rather than a stamp. Of the 35 findings: the four blockers above fixed, the remainder amended in place (protocol text, budget wording, labels, captions) or checked and held, each traceable in the repository. The full list ships in the repository.

Reproduce, and limits

python reduce_raw.py            # cache v2: 29 added columns, overflow sentinels masked
python tuning.py --probe        # which of sklearn / optuna / lightgbm / shap are importable
python run_modeling.py          # the whole stage: bake-off, zoo, calibration, decisions
python run_redteam.py           # the hostile review: seed studies, refits, tripwire v2 (s15)
python run_final_figures.py     # re-render the calibration + precision-at-k panels (val-selected map)
python verify_claims.py         # every number on all three pages == the results JSONs

Limits, stated plainly.

• Daily weather cannot see gusts. The wind values are 4 km daily means. The nighttime downslope wind events that drive the most destructive fires are exactly what a daily mean smooths away. The wind-alignment feature is a direction-and-terrain proxy rather than a gust measurement.
• No suppression data. Whether a fire escalates depends heavily on what was sent to it and how fast. None of that is in this dataset, so part of what the model calls "risk" is really "risk given the response that actually happened."
• Five days is the whole drought memory. The antecedent window in this dataset stops at five days; multi-season drought enters only through the climatological normals.
• California only for the headline. The transfer test says what happens elsewhere; it does not make this a national model.
• Reported ignitions, not all ignitions: the standing caveat from the data audit, which applies to every model here.
• One analyst, one pipeline. These checks are the ones I thought to run. The code is in the repository so someone else can run the one I didn't.