Spectral foundation models (DreaMS, PRISM), the MassSpecGym benchmark and its audit, CASMI 2022 — plus a likelihood-ratio test of "Bayes vs rank-average"

expertconfidence mediumrelevance 4/525 sources4.9k words
Forced analogyMedical differential diagnosis / likelihood ratios — partially holds: calibrated log-LR fusion beats rank-averaging by 0.03–0.06 MRR@25 in simulation and class-weighting adds another ~0.03, but the independence assumption breaks for correlated scorers (ρ ≥ 0.6), where only a joint (correlation-aware) model keeps helping.

TL;DR. DreaMS (116 M-parameter transformer, pre-trained on up to 700 M GNPS spectra, MIT weights on Hugging Face) is the only open spectral foundation model; Enveda's PRISM (1.2 B spectra, half proprietary) has a blog post, no weights, no code and — as far as I could verify — no preprint. MassSpecGym is the leaderboard that ranks everyone, but its June-2026 audit found evaluation problems in at least 17 of 26 papers, with "state-of-the-art" retrieval numbers collapsing from 82 % to 7 % hit@1 once a SMILES-canonicalisation shortcut was removed; honest formula-known retrieval today tops out around 54 % hit@1 and formula-free around 27–28 %. A simulation calibrated to those numbers says: never rank-average two scorers; sum calibrated log-likelihood-ratios instead (+0.03–0.06 MRR@25), weight them by predicted compound class (+0.02–0.03 more), and expect every fusion rule except a jointly-trained one to stop helping once scorer errors are ≥ 60 % correlated.

Part A — Facts

A1. DreaMS

Item Value Source
Paper Bushuiev R, Bushuiev A, Samusevich R, Brungs C, Sivic J, Pluskal T. Self-supervised learning of molecular representations from millions of tandem mass spectra using DreaMS. Nat Biotechnol 44(4):630–640; online 2025-05-23; DOI 10.1038/s41587-025-02663-3; PMC13090125 (open access). Preprint title was "Emergence of molecular structures from repository-scale self-supervised learning on tandem mass spectra" (ChemRxiv). Europe PMC record; paper PDF
Pre-training corpus (GeMS) Mined from MassIVE/GNPS: 814 M raw MS/MS → 714 M after removing empty/corrupted ("up to 700 M"). Quality tiers: GeMS-A 42 M (≈4-decimal m/z accuracy; 97 % Orbitrap), GeMS-B 100 M, GeMS-C 201 M (includes QTOF). LSH-clustered variants, e.g. GeMS-A10 (24 M, the unlabeled set shipped with MassSpecGym) and GeMS-C1 (79 M clusters). HF dataset roman-bushuiev/GeMS, 528 GB, MIT. DreaMS Methods; MassSpecGym §3.3; HF card
Architecture 116 M parameters. Peak encoder (Fourier features of m/z + intensity, shallow FFN) → 7 transformer-encoder layers, 8 heads, d = 1,024, Graphormer-style attention biased by pair-wise mass differences; precursor token acts as master node; n most-intense peaks kept (n not captured in the text I read). Objectives: 30 % of m/z masked and predicted as a classification over m/z bins (cross-entropy mattered most in ablations) + chromatographic retention-order prediction. Output: 1,024-d spectrum embedding. DreaMS Fig. 3, Methods
Compute Final model pre-trained 48 h on 4×AMD MI250X or 8×NVIDIA A100 (DDP). Inference: ≈1.2 M embeddings/h on 8×A100; "1 M spectra ≈ 1 h on one A100, ≈ 7 h on an M1 MacBook". VRAM never stated; 116 M fp32 weights ≈ 0.46 GB, so single-GPU (even 8 GB) inference and fine-tuning are realistic. DreaMS Methods
Downstream results Zero-shot embedding cosine beats MS2DeepScore in correlation with Tanimoto and MCES; after contrastive fine-tuning on hard triplets (negatives within 0.05 Da) it beats MS2DeepScore, cosine and spectral entropy in library retrieval (same InChIKey14 within 10 ppm) and analog search (≤ 5 MCES edits); the Fig. 4 panels print AUCs of 0.85 (DreaMS) vs 0.82 (entropy) / 0.80 (cosine) / 0.78 (MS2DeepScore) and 0.77 vs 0.74 / 0.72 / 0.71 (modified cosine). Fingerprint prediction "on par with MIST" on the MIST (NPLIB1) benchmark — no number given in the text. Fluorine detection on 17,000 in-house test spectra at ≥ 90 % precision thresholds. DreaMS Atlas: 3-NN network over 201 M spectra. DreaMS Fig. 4 caption and text
MassSpecGym numbers DreaMS is not a baseline in the MassSpecGym paper (verified: no DreaMS row in Tables 2–4 of arXiv v3). The only DreaMS retrieval number on MassSpecGym is in the audit paper: "DreaMS + ChemBERTa Alignment" (7-layer DreaMS, last two layers unfrozen, InfoNCE to frozen ChemBERTa) — 82.41 % hit@1 inflated → 6.70 % corrected (formula track). MassSpecGym v1.5 ships a retrained "data-safe" DreaMS. HF also hosts embedding_model_massspecgym.ckpt. audit Table 3, §A.4; HF tree
Open weights HF roman-bushuiev/DreaMS (MIT): ssl_model.ckpt 1.39 GB, embedding_model.ckpt 1.24 GB, embedding_model_massspecgym.ckpt 1.24 GB, DreaMS_embedding_model_torchscript.pt 468 MB. Also Zenodo 10997887. Code github.com/pluskal-lab/DreaMS (MIT), docs dreams-docs.readthedocs.io; dreams.api.dreams_embeddings(mgf) → 1,024-d matrix. HF tree; README; docs

A2. PRISM (Enveda + Microsoft)

What is public is a single Enveda blog post (2024-05-28; authors Healey, Domingo-Fernández, Taylor, Krettler, Lightheart, Park, Kind, Allen, Colluru) plus press coverage. Claims: 1.2 B MS/MS spectra, 85 B tokens; ≈ 600 M from public repositories (GNPS, MetaboLights, Metabolomics Workbench) + ≈ 600 M from Enveda's own platform; BERT-family encoder; "masked peak modeling" masking 20 % of peak masses; downstream: 7–16 % relative R² improvement across 12 chemical properties vs a baseline, and "23 % relative improvement" in spectral library matching vs "standard ML models". Parameter count, layer count, tokeniser, benchmarks used and baselines are not disclosed. I found no weights, no code and no preprint (the search budget ran out before I could exhaust bioRxiv; treat "no preprint" as unverified, confidence low). Note that Rose Lightheart (PRISM author) is a co-author of the MassSpecGym audit (Liu, Bushuiev, Lightheart et al.) — the challenge host knows the shortcut literature intimately, which is consistent with the RDKit tautomer-canonicalisation step in the Kaggle metric.

A3. MassSpecGym (NeurIPS 2024 Datasets & Benchmarks; arXiv 2410.23326 v1 2024-10-30, v3 2025-02-14)

Dataset. 231,104 spectra (231 k), ≈ 29 k unique structures (≈ 10 k, 33 %, newly measured in-house); sources MoNA, MassBank, GNPS, matchms-filtered; adducts only [M+H]+ and [M+Na]+; Orbitrap or QTOF; m/z < 1,000; spectra with > 50 % unexplained intensity removed. HF roman-bushuiev/MassSpecGym, MIT, 52.8 GB. Split. Agglomerative clustering of the 29 k molecules on MCES distance, single linkage, threshold 10 — no two molecules in different folds are < 10 edge edits apart; folds stratified by instrument, collision energy, adduct, molecule frequency. The paper shows that the usual InChIKey-disjoint split leaks Tanimoto > 0.85 analogs. Retrieval candidates. |C| ≤ 256 per spectrum, sampled with the same mass ("within an acceptable experimental error range" — tolerance not given in the main text) first from 1 M bio/environmental molecules, then 4 M diverse, then all 118 M PubChem (2024-05-31 download). Bonus track: same formula instead of same mass. Metrics: hit@1/5/20, MCES@1, 99.9 % bootstrap CIs (20,000 resamples).

Original baselines (arXiv v3 Table 3; %):

Track Method Hit@1 Hit@5 Hit@20 MCES@1
mass Random 0.37 2.01 8.22 30.81
mass DeepSets 1.47 6.21 19.23 25.11
mass Fingerprint FFN 2.54 7.59 20.00 24.66
mass DeepSets + Fourier 5.24 12.58 28.21 22.13
mass MIST 14.64 34.87 59.15 15.37
formula Random 3.06 11.35 27.74 13.87
formula DeepSets 4.42 14.46 30.76 15.04
formula Fingerprint FFN 5.09 14.69 31.97 14.94
formula DeepSets + Fourier 6.56 16.46 33.46 14.14
formula MIST 9.57 22.11 41.12 12.75

De novo (Table 2): every baseline (SMILES/SELFIES transformer, random generation) scores 0.00 % top-1 and top-10 exact accuracy; best MCES@top-10 was random generation (18.25 formula track). Simulation (Table 4): FraGNNet cosine 0.52, forward-model retrieval hit@1 46.64 % (mass track).

The audit — "MassSpecGym in the Wild: Uncovering and Correcting Evaluation Pitfalls in AI-Driven Molecule Discovery" (Liu, Bushuiev, Lightheart et al., arXiv 2606.19624, 2026-06-17). 34 papers in the first year; 26 report benchmark numbers; ≥ 17 of those 26 have evaluation issues in three classes: (i) data leakage (decoder pre-training sets excluding only exact InChIKeys, not MCES ≥ 10 analogs; ICEBERG simulators trained on test molecules; formula annotators trained on test), (ii) shortcut learning from candidate artefacts, (iii) implementation bugs and metric divergence.

Corrected / updated retrieval leaderboard (audit Table 3; formula-known track, |C| ≤ 256, %):

Method Hit@1 Hit@5 Hit@20 MCES@1 Status
Random 3.06 11.35 27.74 13.87 clean
MIST vMassSpecGym 9.57 22.11 41.12 12.75 clean
Nearest Neighbor 9.58 22.26 39.92 9.00 clean
JESTR 11.82 33.48 61.46 11.71 clean
MVP 13.96 36.88 68.12 10.36 clean
FLARE 22.66 50.00 75.15 — clean
DiffMS (generative retrieval) 32.29 54.36 75.47 — uses MIST vDiffMS checkpoint (grey)
ICEBERG (forward-model retrieval) 36.18 60.70 78.83 6.67 clean (CE imputed for ~half of test)
MIST + MolForge (generative retrieval) 39.09 49.25 61.60 7.72 grey (buggy MIST checkpoint)
FRIGID (generative retrieval) 45.27 58.30 69.46 6.61 grey ("a few test structures leaked")
MIST vDiffMS 46.42 54.74 66.27 13.82 grey (unreproducible checkpoint)
MIST vFRIGID 53.76 65.32 74.71 5.16 grey (same leak flag as FRIGID)
DreaMS + ChemBERTa (inflated → corrected) 82.41 → 6.70 92.57 → 19.82 97.41 → 40.62 2.86 → 13.13 canonicalisation shortcut
ChemFormer two-stage (inflated → corrected) 63.89 → 7.60 70.52 → 19.12 75.86 → 39.31 5.39 → 13.23 canonicalisation shortcut
ChemBERTa SMILES-only classifier (inflated → corrected) 99.66 → 8.2 99.84 → 18.39 99.99 → 38.14 0.05 → 13.8 spectrum-blind
PubChem default ranking (spectrum-blind) 49.98 54.13 61.04 7.43 artefact baseline
Rank by number of chiral atoms (spectrum-blind) 4.88 16.85 37.86 14.4 artefact baseline

(Column alignment of the MCES values for FRIGID/vFRIGID in the PDF text is ambiguous; the 5.16/6.28 pair belongs to the two FRIGID rows, 6.61/8.73 to ICEBERG/MolForge — treat MCES assignments among those four rows as ± one row.)

Mass-based (formula-free) track after the audit (Table 2, 8; data-safe MIST retrained per FRIGID recipe): Random 0.43 %; MIST with data-safe MIST-CF top-1 formula 26.64 % hit@1 (38.47 / 54.90 @5/@20, MCES 16.41); with MIST-CF top-5 element-wise-max 28.23 %; with test-contaminated MIST-CF 31.39–35.15 % (the leak); with ground-truth formula 51.45 % (66.25 / 80.58). MIST-CF itself: 50.5 % top-1 / 73.2 % top-5 formula accuracy when clean, 66.5 / 84 % when trained on test. Simply filtering mass-candidates to the true formula lifts random from 0.43 % to 12.98 % hit@1 — a 30× likelihood ratio from the formula alone.

De novo (audit Table 9, formula track; top-1 acc / top-10 acc): MIST+MSNovelist 0 / 0; Spec2Mol 0 / 0; MIST+Neuraldecipher 0 / 0; MS-BART 1.07 / 1.11; MADGEN 1.31 / 1.54; DiffMS 2.30 / 4.25 (MCES 18.45 → audit prints 27.47 under its pinned MCES; see Table 10); FOAM 1.50 / 10.28; MIST+MolForge corrected 10.73 / 14.48 (inflated 31.75 / 40.55 from the batching bug); FRIGID 18.29 / 22.00. Metric divergence (Table 10, identical DiffMS predictions): Tanimoto 0.297 (Morgan r2, 1024 b) vs 0.214 (r3, 4096 b); MCES 21.21 vs 15.87 depending on solver bound. v1.5 pins InChIKey14, 2048-bit ECFP4, MCES threshold 15.

Unaudited claims to treat with suspicion: GLMR (Zhang et al., arXiv 2511.06259, Nov 2025) reports 64.17 % recall@1 mass-track / 68.48 % formula-track on MassSpecGym, with re-run MIST at 18.46 % and JESTR 17.62 % (mass track); the text never mentions candidate canonicalisation and the paper predates v1.5 — it is absent from the audited leaderboard. MARLIN (arXiv 2607.04774, July 2026), MS-GPT (2607.23607), SpecBridge (2601.17204), FlowMS (2603.18397) report on NPLIB1 or MassSpecGym with numbers I could not open; unverified. The live leaderboard massspecgym.onrender.com returned HTTP 503 on three attempts.

A4. CASMI 2022 (Fiehn lab, UC Davis)

500 compounds: 250 "priority" (not in public spectral libraries) + 250 "bonus"; raw LC-HRMS/MS in +ESI and −ESI (each challenge in one polarity only); inclusion list, 0.8 s dynamic exclusion; MS1, MS2 and RT provided; top-1 answer only, scored as exact InChIKey first-block match; deadline 2022-06-06; 16 groups. Instrument vendor/type not stated on the pages I could read (unverified).

2D-structure results (priority / bonus; correct / attempted):

Submission Priority Bonus
Duhrkop (SIRIUS/CSI:FingerID — MIST paper calls CSI:FingerID "the highest performing method at CASMI2022") 57/222 = 26 % 64/219 = 29 %
Nothias 58/228 = 25 % 62/219 = 28 %
Rutz 58/228 = 25 % 62/219 = 28 %
Nikolic_KUCA (subset only) 21/55 = 38 % —
Mehta 32/156 = 21 % 7/33 = 21 %
Xing 37/246 = 15 % 31/246 = 13 %
Tsugawa 30/241 = 12 % —
Namini/Mosley 25/235 = 11 % —
Wang 23/249 = 9 % 10/247 = 4 %
Wei_MSKU_1/2/3 18/14/12 of 247 = 7/6/5 % —
Alcazar 12/215 = 6 % —
Frigerio/Andujar (MS-DIAL / patRoon) 11/196 = 6 %; 1/22 = 5 % —
Ressom 10/234 = 4 % —
Hong (pub / pred+pub) 3/249 = 1 % 5/247 = 2 %; 15/247 = 6 %
Young; Elser 0/130; 0/97 0/122 (Elser)

Formula: Duhrkop 94 % (priority) / 95 % (bonus), Nothias 93 %; class: Duhrkop 69 % priority, Wang 77 % bonus; adducts: 93–96 % for the best. Tools per participant beyond Duhrkop are not stated on the results page (MetFrag and CFM-ID entries are not identifiable there; unverified). Take-away: on "not-in-any-library" compounds the best 2022 top-1 rate was ≈ 26 % — roughly where the 2026 public-LB MRR@25 of ≈ 0.42–0.47 sits once you account for partial credit at ranks 2–25.

Part B — Forced analogy: medical differential diagnosis with likelihood ratios

The analogy, precisely. A clinician starts from a pre-test probability (prevalence given the presentation), then multiplies the odds by the likelihood ratio of each test result, LR = P(result | disease) / P(result | no disease), and only multiplies when tests are conditionally independent given the disease. Mapping: a candidate structure = a diagnosis; the prior = 1/|C| or a database-frequency prior; each scorer (fingerprint cosine, spectral-library similarity, forward-simulation match, formula agreement) = a test; a scorer's output for a candidate = a test result; LR = how much more often that score arises for the true structure than for a decoy. Rank-averaging corresponds to the clinically discredited habit of "counting positive tests".

(i) Crude per-class sensitivities and LRs from published numbers. The only per-class, two-method table I could reach was none (MIST's bioRxiv full text returned HTTP 429 on every attempt; the Nature version is paywalled to the fetcher; paperclip is not installed on this box). So I calibrated the marginal sensitivity of two real methods on identical candidate sets and modelled class-dependence as an explicit assumption: MassSpecGym formula track, |C| ≤ 256 — MIST 9.57 / 22.11 / 41.12 % and JESTR 11.82 / 33.48 / 61.46 % hit@1/5/20. Fitting an equal-variance Gaussian score model (true ~ N(μ,1), decoys ~ N(0,1)) gives μ_MIST = 1.30 and μ_JESTR = 1.60. Then three latent "classes" with priors 0.5/0.3/0.2 and multiplicative signal factors (A: 1.35/1.0/0.55; B: 0.6/1.0/1.45) — i.e. A is good where B is weak — assumed, not measured. At 1,000 candidates the implied sensitivities P(true in top-25 | class) are A: 0.415 / 0.251 / 0.105 and B: 0.157 / 0.355 / 0.637; against a random top-25 rate of 2.5 % these are LRs of 16.6 / 10.1 / 4.2 (A) and 6.3 / 14.2 / 25.5 (B). A genuinely measured LR from the audit: a correct formula filter lifts random hit@1 from 0.43 % to 12.98 %, LR ≈ 30.

(ii) Simulation. 400 molecules × 1,000 candidates, two scorers with shared noise fraction ρ, calibration fitted on a separate 400-molecule simulated "training" set (logistic regression of P(true | score), global or per class), 5 seeds. Code (lr_sim.py, numpy + scipy only):

# score model: decoys ~ N(0,1); true ~ N(mu_scorer[class],1); shared noise fraction rho
def simulate(n_mol, n_cand, mu_A, mu_B, class_prior, rho, rng):
    cls = rng.choice(len(class_prior), size=n_mol, p=class_prior)
    shared = rng.normal(0, 1, (n_mol, n_cand))
    eA = np.sqrt(rho)*shared + np.sqrt(1-rho)*rng.normal(0, 1, (n_mol, n_cand))
    eB = np.sqrt(rho)*shared + np.sqrt(1-rho)*rng.normal(0, 1, (n_mol, n_cand))
    sA, sB = eA.copy(), eB.copy()
    sA[:, 0] += mu_A[cls]; sB[:, 0] += mu_B[cls]       # candidate 0 is the truth
    return cls, sA, sB

def mrr_at_25(score):
    rank = 1 + np.sum(score[:, 1:] >= score[:, [0]], axis=1)   # pessimistic ties
    return np.mean(np.where(rank <= 25, 1.0/rank, 0.0))

def rank_avg(sA, sB):      # lower mean rank = better
    rA = (-sA).argsort(1).argsort(1); rB = (-sB).argsort(1).argsort(1); return -(rA+rB)/2
def z_avg(sA, sB):
    z = lambda s: (s - s.mean(1, keepdims=True)) / s.std(1, keepdims=True); return z(sA)+z(sB)
def naive_bayes(sA, sB, cls, models, per_class, joint=False):
    # models[c] = (wA, wB, wAB): logistic fits of P(true|score) on a held-out simulated set.
    # logit = log LR + log prior-odds; the prior is constant within a query, so sum of logits ranks
    out = np.empty_like(sA)
    for i in range(len(cls)):
        wA, wB, wAB = models[cls[i] if per_class else None]
        out[i] = (wAB[0] + wAB[1]*sA[i] + wAB[2]*sB[i]) if joint \
                 else (wA[0] + wA[1]*sA[i]) + (wB[0] + wB[1]*sB[i])
    return out
# mu fitted so that hit@1/hit@20 at |C|=256 match MIST (9.57/41.12 %) and JESTR (11.82/61.46 %);
# classes: priors (0.5,0.3,0.2), factors A=(1.35,1.0,0.55), B=(0.6,1.0,1.45)

(iii) Results — MRR@25, mean of 5 seeds (SD ≤ 0.012 everywhere):

Scenario Method ρ = 0.0 ρ = 0.3 ρ = 0.6 ρ = 0.9
class known A only 0.103 0.100 0.098 0.095
class known B only 0.118 0.118 0.120 0.117
class known rank-average 0.161 0.113 0.085 0.061
class known z-score average 0.213 0.149 0.115 0.089
class known naive Bayes, global calibration 0.212 0.150 0.115 0.089
class known naive Bayes, class-weighted LR 0.241 0.183 0.145 0.117
class known joint logistic (class, correlation-aware) 0.242 0.193 0.185 0.339*
class 80 % correct rank-average 0.161 0.113 0.085 0.061
class 80 % correct z-score average 0.213 0.149 0.115 0.089
class 80 % correct naive Bayes, class-weighted LR 0.229 0.173 0.137 0.111
class 80 % correct joint logistic (class, correlation-aware) 0.229 0.179 0.166 0.301*
no class effect B only 0.115 0.117 0.117 0.116
no class effect rank-average 0.188 0.136 0.107 0.083
no class effect z-score average 0.223 0.158 0.120 0.094
no class effect naive Bayes (global = class-weighted) 0.224 0.161 0.123 0.097
no class effect joint logistic 0.223 0.160 0.125 0.123

* At ρ = 0.9 the joint model exploits the difference A − B, whose noise variance is 2(1 − ρ) = 0.2 while its signal μ_A − μ_B stays class-specific; that "common-mode rejection" is a property of my Gaussian shared-noise model, not something to expect from two real spectral scorers. Discount it.

Verdict. Bayesian (calibrated log-LR) combination beats rank averaging by 0.03–0.05 MRR@25 at every correlation, and class-weighting the LRs adds another 0.02–0.03; rank-averaging two correlated scorers is worse than the better scorer alone once ρ ≥ 0.3, and every independence-assuming combiner (rank, z, naive Bayes) stops beating the single best scorer around ρ ≈ 0.6. Z-averaging and global naive Bayes are indistinguishable here because, for equal-variance Gaussian scores, log LR is linear in the score — so the honest message is "calibrate, then add log-odds", not "Bayes is magic". The gain of class-weighting survives a 20 % class-misassignment rate (0.229 vs 0.213 at ρ = 0). Absolute values are lower than published hit rates because 1,000 candidates were used, not 256.

Does the analogy hold? Partially. It holds for the core prescription (prior × LR, not vote counting), for the role of the formula/mass filter as a high-LR "screening test" (LR ≈ 30 measured), and for class-conditional calibration (a test's LR depends on the patient subgroup = compound class). It breaks on conditional independence: fingerprint-cosine scorers trained on the same libraries share failure modes (same missing substructures, same adduct confusions), exactly the ρ ≥ 0.6 regime where the naive product over-counts evidence. It also breaks on the prior: clinicians have measured prevalences; here the natural prior (1/|C|) ignores that PubChem decoys are not drawn from "chemistry that gets measured" — the audit's spectrum-blind PubChem-order baseline reaching 49.98 % hit@1 is a prior, not a test, and should enter the fusion as log prior-odds, not as another LR.

Gotchas — what could silently fool us

  1. Canonicalisation leak in any locally built candidate set. If your true-structure SMILES come from the training parquet (normalized_smiles) and decoys from COCONUT/PubChem canonicalised differently, a learned re-ranker will reach 80–99 % locally and ≈ 7 % on the hidden test (audit Table 3). Canonicalise everything with the same RDKit version and tautomer routine as the metric (RDKit 2026.03.3).
  2. Formula leakage dressed as "mass-based". Any formula annotator (SIRIUS, MIST-CF, BUDDY) trained or tuned on molecules that appear in your validation set inflates hit@1 by 9 points absolute (26.6 → 35.2 %); a correct formula alone is a 30× LR. The hidden test molecules are "none previously published" — a formula model trained on all public libraries is fine for the test but will lie on any validation set drawn from those libraries.
  3. Spectrum-blind priors masquerade as skill. PubChem default order gives 50 % hit@1 on MassSpecGym; number of chiral centres gives 4.9 %. On a COCONUT-sourced pool, "is in COCONUT at all", NP-likeness, or deposition frequency will carry similar spurious signal — fine to use as a prior, dangerous to mistake for a spectral model.
  4. Batched-inference masks. The MIST/DiffMS attention mask added +1 instead of excluding padding; a reused checkpoint tripled de novo accuracy (10.7 → 31.8 %). Check that batched and single-spectrum outputs agree before trusting any borrowed encoder.
  5. Metric parameterisation. Tanimoto moves 0.30 → 0.21 across Morgan radius/bit choices and MCES 21.2 → 15.9 across solver bounds on the same predictions. The Kaggle metric is InChIKey14 after tautomer canonicalisation — a prediction that differs only by a tautomer is correct, one that differs by a stereocentre is also correct, one that differs by a double-bond position is wrong. Local MRR must use the same routine.
  6. "Formula track" is harder than "mass track" for fingerprint methods. MIST drops from 14.64 % (mass candidates) to 9.57 % (formula candidates) because same-formula isomers have near-identical fingerprints. A COCONUT pool with a tight mass window is effectively a formula-track problem for most queries.
  7. Collision energies missing. Roughly half of the MassSpecGym test split lacks CE; ICEBERG's numbers required imputing them. The hidden test gives CE lists (merged multi-CE spectra); models trained on single-CE library spectra see a distribution shift.
  8. Foundation-model benchmarks are mostly on Orbitrap. 97 % of GeMS-A is Orbitrap; the hidden test is all timsTOF. DreaMS's Fourier m/z features assume ~4-decimal accuracy; QTOF spectra are in GeMS-C only.
  9. The leaderboard you cite may be the uncorrected one. Published papers (GLMR 64 %, FRIGID 45 %, MIST vFRIGID 54 %) sit above the audited clean frontier (FLARE 22.7 %, ICEBERG 36.2 %, formula track). The clean formula-free number to beat is MIST + data-safe MIST-CF ≈ 27–28 % hit@1 at |C| ≤ 256.
  10. CASMI 2022 was top-1 only, one polarity per challenge, with RT and MS1 — its 26 % is not an MRR@25 and was obtained with information the 2026 challenge does not give (RT) and without the information it does (multi-CE merged spectra).
  11. Simulation caveats. Equal-variance Gaussian scores, a single shared-noise ρ for truth and decoys, three fictitious classes, calibration on simulated rather than real data; class factors are assumptions. The ordering of methods is robust across the three scenarios; the magnitudes are not to be quoted as predictions.

Implications for the challenge

  1. Fuse scorers by summing calibrated log-odds, never by averaging ranks. Fit per-scorer P(true | score) with logistic/isotonic regression on an MCES-disjoint, novel-molecule validation set; rank by Σ logit. Expected +0.03–0.05 MRR@25 over rank averaging in simulation.
  2. Add a class-conditional calibration layer. Predict NPClassifier/ClassyFire superclass from the spectrum (CANOPUS-style head or DreaMS fine-tune, 80 % accuracy suffices) and fit calibration per class; worth ≈ +0.02–0.03 in simulation and it is exactly where "scorer A is good on terpenoids, B on alkaloids" lives.
  3. Measure ρ between your scorers on validation (correlation of per-query true-candidate z-scores or of rank errors). If ρ ≥ 0.6 for two fingerprint models, drop one or replace the product with a jointly trained 2-feature logistic re-ranker; seek genuinely orthogonal evidence (forward-simulated spectrum match, spectral-library analog similarity, formula/adduct consistency, isotope pattern) rather than a third fingerprint.
  4. Treat formula/mass filtering as the highest-LR test and get it right first. Use a formula annotator trained on all public data (no leakage risk against unpublished test molecules), take its top-k formulas with calibrated probabilities, and feed them as log prior-odds over candidates, not as a hard filter — 51 % vs 27 % hit@1 is the stake.
  5. Use DreaMS as a feature extractor, not as a solved component: embedding_model_massspecgym.ckpt or the v1.5 data-safe DreaMS, 1,024-d embeddings, single-GPU; fine-tune to fingerprints with the Murcko-histogram split protocol. Its MassSpecGym retrieval number (6.7 % corrected, formula track) says the published alignment recipe is not a shortcut to a strong ranker.
  6. Validate exactly like the audit's v1.5: uniform RDKit canonicalisation of truth and decoys, MCES ≥ 10 (or at least Tanimoto < 0.7) between train and validation molecules, pinned metric code, and a spectrum-blind baseline (database order, NP-likeness, chiral-centre count) reported next to every model — if the blind baseline is within a few points of the model, the model has learned the pool, not the spectrum.
  7. Do not benchmark against PRISM: nothing is reproducible; its only transferable lessons are "masked-peak pre-training works at 20 % masking" and that Enveda's half of the corpus is proprietary — expect the hidden test to resemble enveda-180 acquisition conditions more than GNPS.
  8. Set expectations with the audited numbers: clean formula-free hit@1 ≈ 27 % at 256 candidates; at a COCONUT-scale pool with 10 ppm windows the public LB's ≈ 0.42–0.47 MRR@25 is consistent with that, so headroom is in candidate coverage (recall@25) and calibrated fusion, not in a bigger single encoder.

Open questions

  • MIST paper per-class (NPClassifier) MIST-vs-CSI:FingerID retrieval tables (Fig. 2/3 and SI) — needed to replace my assumed class factors with measured ones; bioRxiv 10.1101/2022.12.30.522318 and Nature 10.1038/s42256-023-00708-3 could not be opened here.
  • Whether a PRISM preprint exists (search budget exhausted); whether PRISM's "library matching +23 %" was measured on timsTOF data.
  • MassSpecGym candidate mass tolerance (ppm/mDa) — stated only in the SI; and whether v1.5 candidate JSONs change |C|.
  • Whether FRIGID/MIST-vFRIGID's "few leaked test structures" flag was resolved in a later revision; the live leaderboard (503) would show it.
  • CASMI 2022 instrument and per-participant tools beyond Duhrkop (results page lists names only).
  • Real ρ between fingerprint-cosine, DreaMS-similarity and forward-simulation scorers on timsTOF natural products — the single number that decides whether naive LR fusion or a joint re-ranker is right.

Sources