Spectrum → fingerprint → retrieval: the FingerID / CSI:FingerID / SIRIUS / MIST lineage and what it buys (or costs) in CASMI26

TL;DR. Every strong CASMI entry since 2016 and every CASMI26 public write-up above ~0.1 MRR uses the same skeleton: predict a binary substructure fingerprint from the MS/MS spectrum, then rank mass-matched database candidates by how well their true fingerprint matches the prediction. CSI:FingerID (SIRIUS) is the reference implementation but it is a licensed web service that cannot run inside an offline Kaggle notebook; MIST (MIT licence, pretrained weights on Zenodo) is the open re-think. The numbers that matter: MIST reaches only ~10–15 % hit@1 on MassSpecGym's MCES-disjoint split, CSI:FingerID scored 26–29 % top-1 at CASMI 2022, and the CASMI26 FPNet channel alone scores ~0.48 Class-2 MRR locally yet pure-fingerprint submissions stall at 0.11–0.15 on the leaderboard — the fingerprint is necessary but the ranker, the candidate pool and leak control decide the score.

1. Lineage at a glance

2. Q1 — Published accuracy, by benchmark (methods × benchmark × metric)

Random vs InChIKey-disjoint vs MCES splits. All CSI:FingerID evaluations since 2015 are structure-disjoint (2018: even stereoisomers of a query are removed from training). MassSpecGym goes further: agglomerative clustering with MCES distance, linkage threshold 10, so no train/test pair is within 10 edge edits; its Fig. 2 shows this "significantly surpasses the commonly applied 2D InChIKey disjoint approach, used in nearly all related works". The 2026 audit quantifies why that matters: decoders pretrained with only exact-InChIKey exclusion memorise analogues (Tanimoto ≥ 0.7), and MIST+MolForge's bonus-task top-1 collapses from 31.75 % (inflated) to 10.73 % (corrected). The MIST paper's own split protocol could not be verified today (see Open questions). No paper I opened reports a random split for any of these methods — the 0.7–0.9 "local MRR" values in CASMI26 write-ups are the only random-ish numbers in circulation, and they are the known-structure (Class 1) regime.

3. Q2 — Which bits, and how they are scored

Fingerprint inventory.

"Biosynthetic" fingerprints: not found in any opened source for CSI:FingerID/SIRIUS; the only "biosynthetic" element is the 2018 scoring variant "Bayesian (biotransformations)", which uses 29 CHNO biotransformation formulas (CH2, C2H2O, CO2, H2O, …) to enlarge the covariance-estimation set, not as fingerprint bits.

Scoring functions (CSI:FingerID), with the measured deltas. Let p_i be the Platt (sigmoid-calibrated) posterior of bit i, x_i the candidate's bit.

• Unit score: count of matching bits. Accuracy-weighted (FingerID 2012): "does not perform better than unit scoring".
• Platt score: P(M|D) = ∏ p_i (if x_i=1) or (1−p_i) (if x_i=0) — a MAP estimator; ≈ +3.5 pp top-1 over unit score.
• Modified Platt (2015, "found by trial and error"): p_i^0.75·(1−sens_i)^0.25 when p_i ≥ 0.5 and x_i = 1; (1−p_i)^0.75 when p_i ≥ 0.5, x_i = 0; p_i^0.75 when p_i < 0.5, x_i = 1; (1−p_i)^0.75·(1−spec_i)^0.25 when p_i < 0.5, x_i = 0. ≈ +1.5 pp over Platt; in 2015 it gave "17.0 % (4.99 percentage points) more correct identifications" than accuracy scoring.
• Bayesian (fixed tree) (SIRIUS 4.0 default lineage): bit dependencies modelled as a tree Bayesian network whose structure is learned from 236,656 biological structures and whose edge covariances are estimated from training predictions; +2.85 pp over modified Platt. Individual tree recomputes the tree per molecular formula: 43.62 % top-1 (best).
• SIRIUS docs caution that Platt probabilities are posterior estimates: "even a high certainty (e.g., 99%) does not guarantee the presence of a molecular property."

COSMIC confidence (what it estimates). Features: the CSI:FingerID score calibrated to an E-value by kernel-density modelling of PubChem candidate scores as a mixture of log-normals; score gaps to other candidates; total MS/MS intensity explained by the fragmentation tree; fingerprint cardinality. A linear SVM "with enforced directionality of features" separates correct from incorrect top hits; decision values are Platt-scaled. The authors state the result "must be treated as a score but not as the probability that the annotation is correct"; its purpose is FDR control (q-values). Numbers: CASMI 2016 (123 queries, biomolecule DB) 57 hits at FDR < 10 % where the raw CSI:FingerID score yields "zero annotations at FDR 40 %"; ROC AUC 0.82 vs 0.40/0.55 for competing scores; on 3,013 queries at 5 % FDR, 1,701 COSMIC hits vs 1 for spectral-library search. SIRIUS 6 docs: a COSMIC score of 0.64 ≈ 10 % FDR; two modes, "exact" (is this the exact structure?) and "approximate" (is it correct or highly similar?). CSI:FingerID's training set at that time: 16,703 structures / 23,965 MS/MS measurements.

4. Q3 — Can any of this run inside the Kaggle notebook today?

The CASMI26 code competition runs offline (≤ 9 h per corentinlaudicina's README; nghianguyen's 4-channel notebook takes ~40 min with GPU), so every pretrained asset must be uploaded as a Kaggle dataset beforehand — including an RDKit 2026.03.3 wheel (aidensong123/casmi26-offline-rdkit-2026033).

5. Q4 — What SIRIUS actually requires as input

• Formula first. CSI:FingerID only scores candidates of formulas that SIRIUS ranked (fragmentation tree + isotope pattern). Chau 2025 note: when the true formula was absent from SIRIUS's list, those challenges fell back to fingerprint-only scores; CASMI 2017 data lacked isotope patterns and "in many cases SIRIUS was not able to find the correct molecular formula" (Jena blog, via search snippet; page returned 503).
• Multiple collision energies. CLI docs: "You can provide multiple MS/MS files … if you have several measurements of the same compound with different collision energies; SIRIUS will merge these spectra into a single spectrum"; the .ms format has >collision blocks; the GUI shows "a merged spectrum" by default. So a merged multi-CE timsTOF spectrum is the native SIRIUS input, not a problem.
• Profile vs centroid. "all data must be centroided"; formats .ms, .mgf, .msp/.mat, Agilent .cef, and .mzML/.mzXML for LC-MS runs; "SIRIUS does not support input of raw file formats" (no Bruker .d). timsTOF/ion mobility: no explicit statement anywhere in the v6 docs I read; SIRIUS is "vendor independent". MassSpecGym's training distribution is 98 % Orbitrap/QTOF, so timsTOF is a shift for every published model.
• Mass accuracy. Deviation "should be within 20 ppm" (usable to ~50 ppm); absolute tolerance below 200 Da; up to 60 fragment peaks are used and spectra "must not" be noise-filtered.
• Max m/z. The "~850 Da default limit" hypothesis is not confirmed: CLIRootOptions.java (stable branch) defines --maxmz with defaultValue = "Infinity", hidden and @Deprecated(forRemoval = true) // sirius 7; the FAQ only recommends excluding high-mass compounds and setting --compound-timeout. For > 600 Da without isotope data the prerequisites page suggests restricting to PubChem. SIRIUS 4 itself was evaluated on 3,965 compounds spanning 75–1,289 Da.

6. Q5 — What the CASMI26 "spectrum→fingerprint + COCONUT" write-ups did

Common findings across write-ups: (i) a bigger pool hurts (PubChem isomers drop Class-2 MRR 0.52 → 0.35; corentin 0.112 → 0.079); (ii) orthogonal evidence beats a better fingerprint ("four fingerprint variants within 0.007 LB"); (iii) SHAP: ~80 % of attribution from fragment peaks at m/z 60–250, with neutral losses 162 (hexose) and 308 (rutinoside) decisive for natural products; (iv) the public FPNet weights are leaky — 0.94 Class-2 MRR on seen sets vs 0.477 on the only held-out set (enveda-np-examples).

7. Forced analogy: medical differential diagnosis

Statement. A predicted fingerprint bit i with Platt probability p_i is a lab test with sensitivity sens_i and specificity spec_i; the candidate structures are the differential; a candidate's true bit vector is the pattern of test results a patient with that "disease" would produce; retrieval is Bayesian updating of a prior over the differential (candidate prior) by the likelihood ratios of each test.

Does SIRIUS already do this? Partly. The Platt score is exactly naive-Bayes updating with per-test posteriors. Modified Platt injects (1−sens_i) and (1−spec_i) factors, i.e. it discounts positive calls from low-sensitivity tests and negative calls from low-specificity tests — a clinician's instinct, admittedly "without statistical interpretation". The 2018 Bayesian tree is the diagnostic analogue of recognising that tests are correlated (ALT and AST), and the "individual tree" conditions the correlation structure on the molecular formula, the first step toward class-conditioning. What nobody does: estimate sens_i/spec_i per compound class (glycoside, terpenoid, alkaloid, flavonoid). CSI:FingerID selects bits by a single global F1 (≥ 0.25) and all public evaluations report global per-bit F1; the CASMI26 FPNets use global logits. Yet class dependence is plausible: sugar-related bits are inferred from neutral losses 162/308 that only glycosides exhibit, aromatic-ring bits are inferred from stable aromatic fragment ions, and aliphatic terpenoid skeletons fragment into uninformative CnHm ladders.

Testable idea (one experiment, ~1 GPU-day). Take the leak-free FPNet (or any spectrum→bit model) and a held-out natural-product set with class labels from NPClassifier or CANOPUS-style prediction. For each class c and bit i, estimate LR+{i,c} = sens{i,c}/(1−spec_{i,c}) and LR−{i,c} = (1−sens{i,c})/spec_{i,c} after thresholding p_i, with Laplace smoothing. Score candidate f under a mixture over classes: log Σ_c P(c | spectrum) · Π_i LR_{i,c}(f_i, p_i), where P(c | spectrum) comes from a small class head trained on the same encoder. Compare MRR@25 against the global f·z on the same ±10 ppm candidate sets, stratified by class. A second, cheaper variant: keep f·z but prune bits whose class-specific F1 < 0.25 before the dot product (the per-class analogue of CSI:FingerID's global filter). Success criterion: ≥ 0.02 MRR gain on Class 2 (above the measured ±0.01 seed noise) in at least glycosides.

Where it holds / breaks. Holds: calibration (Platt scaling, COSMIC's q-values ≈ post-test probability and FDR), test correlation, and the prior. Breaks: (1) the "diseases" are near-identical — 70.8 % of ±10 ppm candidates share the formula and isomers differ in a handful of bits, so the task is not picking one of a few diseases but ranking hundreds of near-duplicates; (2) the prior is dangerous — a spectrum-blind PubChem-order prior reaches 49.98 % hit@1 on MassSpecGym, and the gnshx popularity prior (µ = 0.25) bought only ≈ +0.005–0.01 on CASMI26, so "pre-test probability" can silently replace the measurement; (3) tests are nested, not merely correlated (ECFP4 ⊂ ECFP6 ⊂ RDKitFP paths), so independent-LR products double-count more severely than any lab panel; (4) the "lab" changes: per-bit sens/spec calibrated on Orbitrap/QTOF libraries drift on timsTOF merged multi-CE spectra, which is exactly the hidden-test instrument.

8. Gotchas / what could fool us

1. Benchmark numbers are not comparable across sources. CSI:FingerID at CASMI 2022 is 26–29 % top-1 on the organisers' page but 8–14 % in Chau 2025's re-run (MS/MS + precursor only, 5 ppm, PubChem candidates); CASMI 2016 shows the opposite (CSI:FingerID best at 48–54 %). Candidate DB, formula input and who ran it dominate.
2. CSI:FingerID is trained on much of the public data. Chau 2025 "found a few of the CASMI challenges in CSI:FingerID's training set" and could not guarantee disjointness; the same applies to any SIRIUS-derived label you might use for validation. ESP's claim that MIST was "the highest performance annotation tool at CASMI2022" conflicts with the organisers' table (MIST does not appear there) — treat as misattribution.
3. InChIKey-disjoint is not leak-free. MassSpecGym needed MCES ≥ 10; the 2026 audit shows analogue memorisation (Tanimoto ≥ 0.7) inflates results, and public CASMI26 FPNet weights score 0.94 on structures they saw vs 0.477 held-out. Any "local MRR 0.7–0.9" is Class-1 behaviour.
4. Candidate-set artefacts can score without a spectrum. SMILES canonicalisation mismatch: 82.41 % → 6.70 % hit@1 after fixing; PubChem default order: 49.98 %; chiral-atom count: on par with learned FFNs. In CASMI26 the analogous trap is pool provenance (in_train/in_coconut flags leak the label: "0.94 on a task that honestly scores 0.55").
5. Formula leakage. Random ranking jumps from 0.43 % to 12.98 % hit@1 just by knowing the formula on MassSpecGym; but on CASMI26's COCONUT pool 70.8 % of candidates already share the formula, so formula-first pipelines (SIRIUS, MIST-CF) buy far less here than in the literature.
6. Metric divergence. The audit lists InChIKey hit rate, MCES, Tanimoto and "close match" rates as implementation-dependent (fingerprint radius/bit length, stereo handling reorder leaderboards). CASMI26 scores on an RDKit tautomer-canonical key that disagrees with naive InChIKey14 slicing on 10.44 % of COCONUT; de-duplicate by the scorer key or waste slots.
7. Bit-level metrics ≠ retrieval. kabir0774: "a model can score well by getting common bits right and still never rank the correct molecule first"; nghianguyen: four fingerprint variants within 0.007 LB; averaging logits of independently trained models destroys the Bayes-log-likelihood interpretation.
8. Platt/COSMIC outputs are not probabilities for your query. SIRIUS docs: 99 % ≠ presence; COSMIC "must be treated as a score".
9. Silent preprocessing failures. Collision-energy strings like "20 V"/"30%" parsed to NaN produced NaN FPNet logits for 1,212 of 2,499 queries, turning the channel into noise below random; MIST's padding-mask bug (attn += attn_mask) affected training and batched inference in MIST and DiffMS codebases.
10. Local validation can invert leaderboard order. nghianguyen: a decomposition fitted to their own anchors gave a negative Class-1 weight; two public notebooks re-run byte-identically scored 0.011 and 0.105 below their advertised LB; seed noise ±0.01 locally, ~0.006 on LB.
11. Instrument shift. SIRIUS docs never mention timsTOF; MassSpecGym is 98 % Orbitrap/QTOF; per-bit calibration learned there is untested on merged multi-CE timsTOF spectra.
12. Licensing. The competition data is CC BY-NC 4.0 (kabir0774 README); CSI:FingerID's non-academic terms run through Bright Giant — even if network were available, routing competition spectra through the Jena/Bright Giant service is a legal question, not just a technical one.

9. Implications for the challenge

1. Do not plan on CSI:FingerID/SIRIUS structure search: it is a web service with licence servers and is unreachable from the offline notebook. SIRIUS's local parts (formula + fragmentation trees) are usable but give little here (70.8 % of candidates already share the formula).
2. Build (or reuse) a spectrum→fingerprint model trained with a decoy-ranking loss on the exact ±10 ppm candidate population (BCE + softmax over f·z against 32–63 same-window decoys); this is what every write-up found to beat plain BCE. Train structure-disjoint and additionally exclude all evaluation structures (and tautomer duplicates) — the FPNet leak costs 0.08 Class-1 MRR in the downstream ranker.
3. Fingerprint choice is second-order (four variants within 0.007); spend the effort on orthogonal channels (library, mass-shifted analogue propagation, fragment-explanation, forward-model rescoring) and a learned ranker — the lineage's own history says the same: Platt → modified Platt → Bayesian tree bought 1.5–3 pp each, while candidate-pool choices swing MRR by 0.1–0.2.
4. Keep the pool at COCONUT ∪ train; PubChem isomers measurably hurt (0.52 → 0.35 Class-2). Use a popularity/structure prior only as a feature in the ranker, never as a multiplicative Bayesian prior (spectrum-blind priors can dominate).
5. Score candidates with f·z on logits (exact Bayes log-likelihood), not Tanimoto on thresholded bits; average views of one model, not logits of different models.
6. Run the class-conditional likelihood-ratio experiment from §7 on glycosides/terpenoids/alkaloids; it is cheap and directly addresses the natural-product shift that corentin measured (0.138 vs 0.107 random on enveda-np-examples).
7. If MIST is used, use the MassSpecGym v1.5 corrected implementation and the Zenodo 11580401 weights, re-train on the challenge data with merged multi-CE timsTOF augmentation, and budget the subformula/MAGMa preprocessing for ~1,500 hidden spectra inside the notebook.
8. Validate on a held-out natural-product set with tautomer-canonical keys, report per-class and per-regime (Class 1/2/3) MRR, and treat any gain < 0.02 as noise.

10. Open questions

• MIST paper's own retrieval tables (NPLIB1 and csi2022 top-k, split protocol): the Nature page exposes only abstract and captions, bioRxiv rate-limited every attempt. Needs one clean fetch of bioRxiv 10.1101/2022.12.30.522318v2 or the PDF.
• Exact bit composition of SIRIUS 6's "5,000 molecular properties" and whether the F1 ≥ 0.25 filter still applies (docs page with "3,215" could not be opened; the number comes from a search snippet).
• What Dührkop's CASMI 2022 submission actually used (SIRIUS version, databases, COSMIC thresholds) — the method descriptions live in a Google Drive folder not opened.
• SIRIUS 4's ">70 %" headline: which database (biomolecule vs PubChem) and which k.
• Whether per-bit sensitivity/specificity really varies by natural-product class (no published stratification found).

11. Sources (opened)

• Dührkop K, Shen H, Meusel M, Rousu J, Böcker S. Searching molecular structure databases with tandem mass spectra using CSI:FingerID. PNAS 2015;112:12580. doi:10.1073/pnas.1509788112 (PMC4611636).
• Ludwig M, Dührkop K, Böcker S. Bayesian networks for mass spectrometric metabolite identification via molecular fingerprints. Bioinformatics 2018;34:i333. doi:10.1093/bioinformatics/bty245 (PMC6022630, full text).
• Dührkop K et al. SIRIUS 4. Nat Methods 2019;16:299. doi:10.1038/s41592-019-0344-8 (abstract + figure captions only).
• Hoffmann MA et al. High-confidence structural annotation of metabolites absent from spectral libraries (COSMIC). Nat Biotechnol 2022;40:411. doi:10.1038/s41587-021-01045-9 (PMC8926923, full text).
• Stravs MA et al. MSNovelist. Nat Methods 2022. PMC9262714 (full text).
• Goldman S et al. Annotating metabolite mass spectra with domain-inspired chemical formula transformers (MIST). Nat Mach Intell 2023. doi:10.1038/s42256-023-00708-3 (abstract/captions; preprint 10.1101/2022.12.30.522318).
• Goldman S, Xin J, Provenzano J, Coley CW. MIST-CF. J Chem Inf Model 2024. doi:10.1021/acs.jcim.3c01082 (abstract; github.com/samgoldman97/mist-cf README).
• github.com/samgoldman97/mist (README main_v2, LICENSE badge, quickstart/00_download_models.sh → zenodo.org/record/8316682).
• Bushuiev R et al. MassSpecGym. NeurIPS 2024. arXiv:2410.23326 (HTML + PDF incl. supplement); github.com/pluskal-lab/MassSpecGym README.
• Liu H, Bushuiev R, Lightheart I, … Healey D, Pluskal T, Coley CW. MassSpecGym in the Wild. arXiv:2606.19624 (PDF, 2026-06-17).
• Schymanski EL et al. CASMI 2016: automated methods. J Cheminform 2017;9:22. doi:10.1186/s13321-017-0207-1 (PMC5368104).
• casmi-contest.org/2017/results.shtml (fetched directly; TLS certificate mismatch noted).
• Fiehn Lab, CASMI 2022 results: fiehnlab.ucdavis.edu/casmi/casmi-2022-results.
• Chau HYK, Zhang X, Ressom HW. Deep learning-based molecular fingerprint prediction for metabolite annotation. Metabolites 2025;15:132. doi:10.3390/metabo15020132 (PMC11857613, full text).
• Li X, Chen YZ, Kalia A, Zhu H, Liu L-p, Hassoun S. ESP. Bioinformatics 2024. doi:10.1093/bioinformatics/btae490 (PMC11344591).
• SIRIUS 6 docs: v6.docs.sirius-ms.io/account-and-license, /io, /cli, /gui, /faq, /methods-background; boecker-lab.github.io/docs.sirius.github.io/prerequisites; github.com/sirius-ms/sirius README and sirius_cli/.../CLIRootOptions.java (stable).
• CASMI26 write-ups (GitHub READMEs/docs, Kaggle facts quoted second-hand from them): corentinlaudicina/enveda-casmi-2026; nghianguyen7171/casmi26 (README, docs/METHOD.md, docs/EXPERIMENT_LOG.md); gnshx/Enveda-CASMI-2026---Molecule-ID-From-Mass-Spectra (README, progress/ROADMAP_AND_PROGRESS.md); billylu24/Enveda-CASMI-2026---Molecule-ID-From-Mass-Spectra; AayanShabbir/casmi26; kabir0774/Enveda-Casmi-2026; mxm-1x/Enveda-CASMI-2026; Rythamo8055/enveda-casmi26-molecule-id.
• Search-snippet-only (page not openable, lower confidence): bio.informatik.uni-jena.de "CASMI 2017 results are out" and "How good is the new SIRIUS?"; SIRIUS docs statement "3,215 molecular properties … F1 at least 0.25".