How spectrum-to-structure methods get fooled — a failure-mode catalogue by sub-population, and a CASMI 2026 validation protocol that does not lie

TL;DR. Every spectrum-to-structure method fails predictably in specific sub-populations: isomers that fragment identically (rutinoside vs neohesperidoside glycosides give literally identical MS/MS), formulas that go wrong above m/z 300–400 (SIRIUS single-spectrum error >50 % toward m/z 800; MIST-CF 50.5 % top-1), spectra at the wrong collision energy or from another instrument (cross-library sensitivity 88–92 % vs 99.5 % in-library), and — worst — leaky evaluation: a 2026 audit found problems in ≥17 of 26 MassSpecGym papers, with "state-of-the-art" hit@1 collapsing from 99.7 % to 8.2 % once a canonicalisation shortcut was removed. Our own audit of train.parquet shows the local traps: all 250 enveda-np-examples structures are also in public libraries (246 in RIKEN, 233 in GNPS), the visible test is verbatim enveda-180, 29 % of enveda-180 rows are dimer adducts that never occur in test, and a 400-molecule MRR has s.e. ≈ 0.02. A validation that will not lie needs three disjoint strata (timsTOF-synthetic, NP-other-instrument, NP-timsTOF-with-analogs-removed), a spectrum-blind baseline, production pool density and a paired bootstrap — and a gain counts only if it survives all three.

1. The catalogue — 21 documented ways methods are fooled

FP = wrong structure promoted; FN = correct structure lost; INFL = the evaluation number is inflated (the modeler is fooled). "local" = my audit of C:\Temp\casmi26\data\train.parquet / test.parquet (script audit_train.py, output audit_out.json, session scratchpad). Full table also in the .json.

Two softer modes: confidence miscalibration — on a 301-query Sciex QTOF set raw CSI:FingerID scores separated "worse than random"; COSMIC's authors say the confidence "must be treated as a score but not as the probability that the annotation is correct"; Jürgens et al. get relative AURC 0.347 for retrieval-level ensemble uncertainty vs ≈0.92 for fingerprint-level scores, and hit@1 selective prediction "collapses to minimal coverage" (PMC8926923; arXiv 2603.10950). Decoder memorisation — a de novo model returns the right answer for a Tanimoto-0.98 pretraining neighbour while its encoder fingerprint is only 0.61 to the truth (audit §4.1). No failure numbers were obtainable for MS-FINDER, CFM-ID 4.0 (abstract only: 4.0 added rules for flavonoid glycosides, a tell about where 3.0 failed), Spec2Vec, or any timsTOF small-molecule evaluation.

2. Isomer blindness (Q1)

The metric forgives stereo and tautomers, not constitution. Three layers: (a) physically identical — 1→6 vs 1→2 disaccharide linkage on the same aglycone (PMC2722651); sugar epimers presumably likewise (no source quantified it; low confidence). No model beats chance; k tied candidates give expected reciprocal rank (1/k)Σ1/i = 0.75, 0.61, 0.52 for k = 2, 3, 4. (b) Separable only by intensity ratios — glycosylation position; a dedicated classifier still misses 15 % at one CE (PMC11644359); MSNovelist concedes some isomers "fragment (almost) indistinguishably" (PMC9262714). (c) Fingerprint-identical candidates — no fingerprint scorer can separate them; MetFrag's tied scores are the fragment-count analogue (PMC4732001). The PNAS 2015 "fraction unresolvable" statistic could not be retrieved (Europe PMC 500 ×3) — unverified. NP pools are glycoside-rich, so the tie fraction will exceed PubChem-metabolite benchmarks; inside a tie only priors (database frequency, NP-likeness) help — legitimate as tie-breakers, dangerous as primary scores (mode 17).

3. Formula-error propagation (Q2)

Single-spectrum SIRIUS: <10 % error below m/z 300, >50 % toward 800; only sample-level joint analysis restores <10 % (PMC9262714) — the hidden test has no sample context. MIST-CF 50.5 / 73.2 % top-1/top-5 clean, 59.0–66.5 % when training touches test (audit Table 7); DiffMS quotes formula tools at 48–84 % (arXiv 2502.09571). Downstream, a hard filter makes a wrong formula a certain zero: MIST hit@1 51.45 % (true) → 26.64 % (top-1 predicted) → 28.23 % (top-5 element-wise max); random ranking with the right formula already scores 12.98 % vs 0.43 % — a ~30× likelihood ratio. No opened source stratifies formula accuracy by instrument (negative finding); SIRIUS #280 and #41 show element-set and adduct configuration, not only the scorer, generate formula errors.

4. Multi-CE spectra and intensity normalisation (Q3)

Only 53 % of MassSpecGym entries carry a CE; ICEBERG and MIST merge all CE scans (peaks within 1e-4 m/z, top-50 above 0.003), DreaMS keeps the 60/100 most intense peaks and never sees CE (arXiv 2304.13136v2; PMC13090125). Direction of effect: merged > 40 > 20 > 10 eV for CSI:FingerID ranking (PMC8926923); 15–80 % sensitivity loss when query and library CE windows do not overlap (PMC6359582); ~10 points of identification from interpolating missing CEs (PMC9400100). A direct single-CE-train/merged-test ablation for fingerprint models exists in no opened source, and MassSpecGym stratifies folds by CE so it cannot show one. Local: enveda-180 847,063 single-CE vs 304,780 three-CE rows; np-examples 773/213/82/101/15 for 1–5 CEs; public libraries 337,443 rows with no CE; visible test 915 single / 298 three-CE. Expect ~25 % merged spectra in the hidden set. Whether ms2_normalized_intensities is base-peak- or sum-normalised identically across the 11 ingest libraries is un-audited (low confidence).

5. Instrument transfer (Q4)

Plain library search loses 8–12 points of sensitivity across Orbitrap ↔ QqTOF (PMC6359582; PMC9400100). For learned models no cross-instrument number was found; MS2DeepScore 2.0 could not be fetched (bioRxiv 429 ×4, Springer 404), so only its abstract claim — similarity "both between different and the same ionization modes", plus a flag for spectra "dissimilar to the training data" — is cited. No timsTOF small-molecule evaluation exists in the opened literature; the one timsTOF-vs-Orbitrap fragmentation comparison surfaced is a peptide preprint (bioRxiv 2023.07.17.549401, not opened, low confidence). Local: timsTOF spectra exist only in the Enveda libraries; 2,538 of 183,191 timsTOF structures also have non-timsTOF spectra — the only instrument-shift probe we own.

6. Chemical-space shift (Q5)

ICEBERG: NIST20 0.727 vs NPLIB1 0.627 cosine, 29 % top-1 retrieval on NPLIB1; SPARC: a MassSpecGym-trained simulator drops to 0.517 on NPLIB1 and recovers to 0.630 with a few reference spectra (arXiv 2609.35649) — chemistry shift costs ~0.1 cosine and is partly recoverable. CSI:FingerID with the biomolecule DB: 34.9–49.2 % top-1 in CV, 41.8–60.0 % independent; PubChem "substantially worse" (PMC8926923) — the brief's "PubChem 90 M hurts" is the same effect. enveda-180 overlaps COCONUT ~0.03 %; np-examples are 99.6 % COCONUT and 100 % present in public spectral libraries (local), so the only NP-on-timsTOF training signal (1,184 spectra) is also the most leak-prone. A public write-up found NP hard sets predicted the LB better than instrument-matched timsTOF proxies (Isaackjoshua README): chemistry match beat instrument match.

7. Split inflation (Q6)

8. Local MRR@25 pitfalls for CASMI26 (Q7) and a protocol that does not lie

Local audit (2,539,608 rows, 275,810 InChIKey14; test 1,213 rows / 400 ids):

1. np-examples: 250 structures, 0 exclusive — RIKEN 246, GNPS 233, MoNA 231, MS-DIAL 227, Pluskal 224, MassBank 208, drug_plus 145. Holding out its spectra while the public spectra of the same structures stay in train is a library-hit test (0.445 local vs 0.063 LB).
2. enveda-180: 182,941 structures, 2,288 in public libraries, 180,653 exclusive, zero overlap with np-examples; timsTOF but synthetic.
3. Adducts: enveda-180 51 % [M+H]+, 16 % [2M+Na]+, 16 % [M-H]-, 8 % [2M+H]+, 5 % [2M-H]-; test 79 % [M+H]+, 16 % [M-H]-, 2.6 % [M+CH2O2-H]-, 1.8 % [M+Na]+, no dimers.
4. Mass (10/50/90 %): visible test 280/329/379; enveda-180 290/347/705; public 210/355/676; np-examples 223/388/677. The hidden set's range is unknown; if NP-like, a third of molecules sit in §3's >50 %-formula-error regime.
5. Duplicates: median 6 spectra per structure (99th pct 79), 17,033 singletons; a per-spectrum random split puts ~94 % of structures on both sides.
6. Noise: reciprocal ranks have sd ≈ 0.4, so 400 molecules → s.e. ≈ 0.02; LB deltas < 0.02–0.03 are unresolvable unless paired. A write-up's own rule: |Δ| ≤ 0.001 on 2,000 paired queries is noise (leesirleesirlee-star README).

Protocol. (i) Canonicalise truth and pool with the metric's routine (RDKit 2026.03.3 tautomer → InChIKey14) first. (ii) Three holdouts, each with all spectra of held-out structures removed plus every train structure with MCES < 10 (v1.5 settings: threshold 15, always_stronger_bound) or Tanimoto(ECFP4, 2048) ≥ 0.7 to any held-out structure: S-tims — ~2,000 enveda-180 structures resampled to the test adduct mix (no dimers) and mass range; S-np — ~1,000 COCONUT-member structures from GNPS/MassBank/MoNA/RIKEN (QTOF/Orbitrap); S-np-tims — the 250 np-examples structures, timsTOF spectra as queries, public spectra and analogs removed (s.e. ≈ 0.025, but the only stratum matching the hidden test on both axes); P-inst (diagnostic) — the 2,538 paired structures, timsTOF spectra scored by a model trained without them, drop vs their non-timsTOF spectra = pure instrument effect. (iii) Production pool and density (same COCONUT tiers, ±10 ppm, 25 slots); never shrink the pool to fake novelty — a density-mismatched proxy predicted +15 % and delivered +0.002 (Isaackjoshua README). (iv) Spectrum-blind baseline (pool order, NP-likeness, chiral count) and recall@25 reported in every stratum. (v) Paired molecule-level bootstrap, ≥ 10,000 resamples, per stratum × adduct × n-CE × mass tercile. (vi) Rule: accept a change only if Δ > 2 paired s.e. in S-np-tims, Δ ≥ 0 in S-tims and S-np, and the blind baseline's Δ < 20 % of the model's. Gain only in S-tims = instrument fitting; only in S-np = chemistry fitting; blind baseline moving = pool fitting.

Forced analogy: medical differential diagnosis — spectrum bias and external validity

Statement. Each scorer is a diagnostic test for "candidate X is the true structure": sensitivity = P(rank ≤ 25 | true), specificity = P(rank > 25 | decoy). Spectrum bias is the clinical-epidemiology fact that these are not constants but functions of case-mix; a test validated on its derivation cohort under-performs in a different population. The mapping is exact: class, adduct, n-CE, mass and instrument are the case-mix; the derivation cohort is "molecules with public spectra, mostly Orbitrap, mostly [M+H]+"; the deployment population is "unpublished NPs on timsTOF". Validating on np-examples with public spectra in train is validating on the derivation cohort, and it over-estimated 7× in public.

Testable idea — the spectrum-bias audit. Compute MRR@25 and recall@25 per cell of (NPClassifier superclass) × adduct × n-CE ∈ {1, 3} × mass tercile with paired bootstrap CIs, for each scorer and the blind baseline. Then standardise to the deployment case-mix as a clinician recomputes PPV at a new prevalence: weights from the inferred hidden-test adduct × n-CE mix (79/16 % [M+H]+/[M-H]-, ~3:1 single:merged) and the COCONUT class prior. Decision rule, mirroring "a test transports only if its likelihood ratios are stable across sub-populations": a gain is believable if its per-cell Δ has the same sign in ≥ 70 % of weighted cells; a gain concentrated in one cell (e.g. m/z > 600 or a rare adduct) is spectrum bias. One afternoon of pandas once the strata exist; the cell table is a dashboard panel.

Verdict. Holds for split inflation, instrument and adduct shift and pool bias (true case-mix effects with the right algebra). Partially holds for formula errors (a mis-specified disease, not a false negative). Breaks for isomers: medicine assumes one disease per patient, whereas several structures can be physically identical to the test — there the differential stays differential and the honest output is a prior-ordered tie worth (1/k)Σ1/i.

Gotchas / what could fool us

1. np-examples looks like the ideal validation set and is the worst: 250/250 structures have public spectra; it carries 4–5-CE rows that never occur in test; it is 0.05 % of train.
2. The visible test is enveda-180; every "test" statistic (mass 280–379, 75 % single-CE) is enveda-180's.
3. Exact-InChIKey14 holdouts remove ~1/6 of leakage; the rest is analogs (11.6 → 11.0 → 8.0).
4. 331 k enveda-180 dimer rows have precursor masses that are not the molecule's; a naïve mass-window pipeline builds wrong candidate sets for 29 % of the largest library and learns nothing transferable to a dimer-free test.
5. The formula is a 30× likelihood ratio and a single point of failure; a hard filter turns 50 % formula top-1 into a 50 % ceiling.
6. Spectrum-blind strength hides in aggregates (PubChem order = 50 % hit@1); mask the spectrum and re-score.
7. Batched-inference bugs inflate silently (+21 points in MIST); a write-up's five real bugs "all operated silently" — test the effect, not the log label.
8. Metric drift: ±0.08 Tanimoto, ±5 MCES on identical predictions; pin RDKit 2026.03.3 and the tautomer routine.
9. Confidence ≠ probability (COSMIC); never gate on a threshold tuned on one instrument.
10. 400 molecules → s.e. ≈ 0.02; LB 0.417 vs 0.471 is ~2.5 unpaired s.e.
11. CE-stratified benchmark folds never test single→merged transfer; our test does.
12. Isomer ties are physics; optimise within-tie priors and stop paying for separation the spectrum cannot provide.

Implications for the challenge

1. Build S-tims / S-np / S-np-tims with analog exclusion and production pool density before tuning anything; adopt the §8 rule.
2. Run P-inst once per scorer; if the timsTOF penalty exceeds ~10 points of recall@25, fine-tune on enveda-180 with analog-disjoint validation.
3. Drop or down-weight dimer-adduct rows; sample training batches to the test adduct × n-CE mix.
4. Use top-k formula predictions as calibrated log-priors, never a filter; expect ~50 % single-spectrum formula top-1 above m/z 400.
5. Put the blind baseline and the per-cell spectrum-bias table on the dashboard; a model that cannot beat pool order in the [M+H]+ × single-CE × 300–400 Da cell has learned the pool.
6. For glycoside-rich classes, invest in tie-breaking priors (linkage frequency in COCONUT, NP-likeness), budgeting against the (1/k)Σ1/i ceiling.
7. Treat local gains < 2 paired s.e. (≈ 0.01 on 2,000 queries, 0.025 on 250) as noise; replicate before submitting.

Open questions

• Hidden-test mass and class mix (q2): decides whether the formula regime hits 10 % or 40 % of molecules.
• Fraction of COCONUT same-formula sets with fingerprint-identical pairs (the unretrieved PNAS 2015 statistic) — computable locally.
• MS2DeepScore 2.0 / DreaMS degradation on P-inst timsTOF spectra — unpublished, measurable in an hour.
• Any single→merged CE ablation for fingerprint models — none found.
• Is ms2_normalized_intensities normalised identically across the 11 ingest libraries?

Sources

Opened and used:

• MassSpecGym, arXiv 2410.23326: https://arxiv.org/html/2410.23326
• MassSpecGym in the Wild, arXiv 2606.19624 (both halves): https://arxiv.org/html/2606.19624
• DreaMS, Nat Biotechnol 2025, PMC13090125: https://www.ebi.ac.uk/europepmc/webservices/rest/PMC13090125/fullTextXML
• ICEBERG, arXiv 2304.13136v2: https://arxiv.org/html/2304.13136v2
• DiffMS, arXiv 2502.09571: https://arxiv.org/html/2502.09571
• MSNovelist, Nat Methods 2022, PMC9262714: https://www.ebi.ac.uk/europepmc/webservices/rest/PMC9262714/fullTextXML
• COSMIC, Nat Biotechnol 2022, PMC8926923: https://www.ebi.ac.uk/europepmc/webservices/rest/PMC8926923/fullTextXML
• MetFrag relaunched, J Cheminform 2016, PMC4732001: https://www.ebi.ac.uk/europepmc/webservices/rest/PMC4732001/fullTextXML
• Two complementary tandem libraries (QqTOF vs Orbitrap), PMC6359582: https://www.ebi.ac.uk/europepmc/webservices/rest/PMC6359582/fullTextXML
• King et al., spectral interpolation, JCIM 2022, PMC9400100: https://pmc.ncbi.nlm.nih.gov/articles/PMC9400100/
• Flavonoid glycoside isomers / ViT, Plants 2024, PMC11644359: https://www.ebi.ac.uk/europepmc/webservices/rest/PMC11644359/fullTextXML
• Flavanone 7-glycoside isomers, PMC2722651: https://www.ebi.ac.uk/europepmc/webservices/rest/PMC2722651/fullTextXML
• Nearest-neighbour baselines, Khoo, arXiv 2610.02249: https://arxiv.org/html/2610.02249
• Selective prediction, Jürgens et al., arXiv 2603.10950: https://arxiv.org/html/2603.10950
• SPARC, arXiv 2609.35649: https://arxiv.org/html/2609.35649
• SIRIUS 4, Nat Methods 2019 (page text, abstract-level only): https://www.nature.com/articles/s41592-019-0344-8
• MIST-CF, DSpace@MIT record (abstract): https://dspace.mit.edu/handle/1721.1/165432
• Semantic Scholar abstracts only: CSI:FingerID PNAS 2015 (10.1073/pnas.1509788112); spectral entropy 2021 (10.1038/s41592-021-01331-z); CFM-ID 4.0 (10.1021/acs.analchem.1c01465); BUDDY (10.1038/s41592-023-01850-x)
• GitHub: samgoldman97/mist #17, #16, #15; sirius-ms/sirius #280, #41; matchms/ms2deepscore #133
• CASMI 2026 write-ups (Kaggle facts via GitHub mirrors): https://github.com/Isaackjoshua/casmi-2026; https://github.com/leesirleesirlee-star/Enveda_CASMI-2026_Molecule-ID-From-Mass-Spectra
• Local audit of C:\Temp\casmi26\data\train.parquet and test.parquet (audit_train.py → audit_out.json, session scratchpad)

Attempted, not retrieved: MS2DeepScore 2.0 (bioRxiv 10.1101/2024.03.25.586580, HTTP 429 ×4; Springer 404); MIST bioRxiv 10.1101/2022.12.30.522318 (429); "Confronting spurious evaluations…" bioRxiv 10.64898/2026.05.03.722532 (429); timsTOF peptide fragment-intensity preprint 10.1101/2023.07.17.549401 (429); Révész 2020 QTOF/Orbitrap CE (PDF unparseable); CSI:FingerID PNAS full text (Europe PMC 500).