Model Transparency
SENTINELWhat the risk index is, how it was validated out-of-sample, and what it does not establish.
An earlier version of this platform advertised an XGBoost model with AUC 0.984, precision 0.96, and recall 0.94, trained on ~54,000 samples. That number was a circular-validation artifact and it has been withdrawn.
The training script built its own target variable as a weighted sum of the same features it then trained on, plus small noise, thresholded at 0.35. An AUC against that label measures how well gradient boosting approximates arithmetic. It says nothing about Lyme disease. That script has since been deleted; the formula is quoted verbatim in the model card so the claim stays checkable without the script staying runnable.
There are no patient-level outcome labels anywhere in this project — no diagnoses, no case/control, no chart review. Nothing here could have produced a legitimate diagnostic AUC. The “54,000 samples” were 250 real state-year observations enumerated 216 times across months, symptom levels, and exposure levels, so a random split leaked on top of the circularity. And the model artifact was never present in the request path: production has always run the deterministic index below.
| Component | Weight | Source |
|---|---|---|
| Historical incidence | 0.30 | CDC county-coded Lyme surveillance, falling back to state NNDSS |
| Current exposure geography | 0.30 | Same resolved rate — county where CDC county-coded, else state |
| Seasonal vector activity | 0.20 | Month-of-year Ixodes activity curve |
| Symptom overlap | 0.20 | Fraction of Lyme's catalogued symptoms reported. Unvalidated. |
Geographic incidence is normalised against a fixed ceiling of 300 per 100,000 rather than the observed maximum, because Dukes County MA sits near 1,425 and normalising to it would flatten every other county to near zero. The output is an exposure risk index for a place and time of year — not a probability that anyone has Lyme disease.
- Temporal leakage control
- Ranked on 2022 only, scored against 2023 only. No 2023 information of any kind enters the ranking.
- Geographic leakage control
- Leave-one-state-out. For each state the decision threshold is the median of every other state’s counties, so a county never contributes to the threshold that judges it — and neither does any neighbour.
- Sample
- 424 counties. 140 excluded (not in both years), 124 excluded (low state completeness). Counties from states where CDC county-coded under 90% of cases are excluded: their rate is a floor of unknown depth, so ranking it against a near-complete state measures reporting practice rather than risk.
0.882
Spearman rank correlation — the honest headline
| Was high in 2023 | Was not | |
|---|---|---|
| Predicted high | 406 | 2 |
| Predicted not high | 10 | 6 |
- Sensitivity (recall)
- 0.976
- Specificity
- 0.750
- False positive rate
- 0.250
- Precision (PPV)
- 0.995
- Negative predictive value
- 0.375
- Balanced accuracy
- 0.863
- Tier exact agreement
- 76.8%
- Within one tier
- 99.0%
0.897
Median within-state Spearman ρ
| Was high in 2023 | Was not | |
|---|---|---|
| Predicted high | 209 | 0 |
| Predicted not high | 206 | 8 |
- Sensitivity (recall)
- 0.503
- Specificity
- 1.000
- False positive rate
- 0.000
- Precision (PPV)
- 1.000
- Negative predictive value
- 0.037
- Balanced accuracy
- 0.751
Read this as a threshold-selection finding, not a failure. A median-of-other-states threshold produced zero false positives across 423 counties and missed about half the true positives. Rank order holds within states; the absolute cut point does not transfer across state lines, which is exactly what a geographic-leakage control is built to reveal.
The temporal holdout produces AUC 0.987 for high-incidence classification. We are not headlining it, for the same reason we deleted the 0.984.
Prevalence in this sample is 98.1%. After the completeness floor, almost every eligible county is high-incidence, because states that county-code their cases are overwhelmingly states that have Lyme. An AUC against a 98%-positive class is dominated by base rate. Specificity rests on 8 negative counties, and negative predictive value is 0.375.
Spearman ρ = 0.882 is the number to quote. It says rank order is stable year over year. It does not say the index detects disease in a person.
| Tier assigned | n | Same tier | Still high-incidence |
|---|---|---|---|
| very high | 203 | 96.5% | 100.0% |
| high | 161 | 60.8% | 100.0% |
| moderate | 54 | 53.7% | 94.4% |
| low | 6 | 50.0% | 16.6% |
Very high is stable. High and moderate are not. A county labelled “high” has roughly a 39.1% chance of being in a different tier next year, though it stays above the high-incidence threshold. The tier boundary is softer than the tier name suggests, and that is an open item rather than a solved one.
- ×Whether an individual has Lyme disease. There are no patient-level labels anywhere in this project.
- ×The symptom-overlap or exposure components, which are deterministic and have no outcome data to validate against.
- ×Anything about the six non-Lyme conditions, for which no comparable federal county surveillance exists.
- ×True incidence. Both years are reported cases, which CDC estimates undercount by roughly 10x.
Full statement, including the open items, in docs/MODEL_CARD.md. Regenerate these numbers with python data/scripts/validate_geographic_risk.py — it is deterministic, with no fitted parameters and no seed. The evidence base covers the research behind the design decisions.