TL;DR. DreaMS (116 M-parameter transformer, pre-trained on up to 700 M GNPS spectra, MIT weights on Hugging Face) is the only open spectral foundation model; Enveda's PRISM (1.2 B spectra, half proprietary) has a blog post, no weights, no code and — as far as I could verify — no preprint. MassSpecGym is the leaderboard that ranks everyone, but its June-2026 audit found evaluation problems in at least 17 of 26 papers, with "state-of-the-art" retrieval numbers collapsing from 82 % to 7 % hit@1 once a SMILES-canonicalisation shortcut was removed; honest formula-known retrieval today tops out around 54 % hit@1 and formula-free around 27–28 %. A simulation calibrated to those numbers says: never rank-average two scorers; sum calibrated log-likelihood-ratios instead (+0.03–0.06 MRR@25), weight them by predicted compound class (+0.02–0.03 more), and expect every fusion rule except a jointly-trained one to stop helping once scorer errors are ≥ 60 % correlated.
Part A — Facts
A1. DreaMS
| Item | Value | Source |
|---|---|---|
| Paper | Bushuiev R, Bushuiev A, Samusevich R, Brungs C, Sivic J, Pluskal T. Self-supervised learning of molecular representations from millions of tandem mass spectra using DreaMS. Nat Biotechnol 44(4):630–640; online 2025-05-23; DOI 10.1038/s41587-025-02663-3; PMC13090125 (open access). Preprint title was "Emergence of molecular structures from repository-scale self-supervised learning on tandem mass spectra" (ChemRxiv). | Europe PMC record; paper PDF |
| Pre-training corpus (GeMS) | Mined from MassIVE/GNPS: 814 M raw MS/MS → 714 M after removing empty/corrupted ("up to 700 M"). Quality tiers: GeMS-A 42 M (≈4-decimal m/z accuracy; 97 % Orbitrap), GeMS-B 100 M, GeMS-C 201 M (includes QTOF). LSH-clustered variants, e.g. GeMS-A10 (24 M, the unlabeled set shipped with MassSpecGym) and GeMS-C1 (79 M clusters). HF dataset roman-bushuiev/GeMS, 528 GB, MIT. |
DreaMS Methods; MassSpecGym §3.3; HF card |
| Architecture | 116 M parameters. Peak encoder (Fourier features of m/z + intensity, shallow FFN) → 7 transformer-encoder layers, 8 heads, d = 1,024, Graphormer-style attention biased by pair-wise mass differences; precursor token acts as master node; n most-intense peaks kept (n not captured in the text I read). Objectives: 30 % of m/z masked and predicted as a classification over m/z bins (cross-entropy mattered most in ablations) + chromatographic retention-order prediction. Output: 1,024-d spectrum embedding. | DreaMS Fig. 3, Methods |
| Compute | Final model pre-trained 48 h on 4×AMD MI250X or 8×NVIDIA A100 (DDP). Inference: ≈1.2 M embeddings/h on 8×A100; "1 M spectra ≈ 1 h on one A100, ≈ 7 h on an M1 MacBook". VRAM never stated; 116 M fp32 weights ≈ 0.46 GB, so single-GPU (even 8 GB) inference and fine-tuning are realistic. | DreaMS Methods |
| Downstream results | Zero-shot embedding cosine beats MS2DeepScore in correlation with Tanimoto and MCES; after contrastive fine-tuning on hard triplets (negatives within 0.05 Da) it beats MS2DeepScore, cosine and spectral entropy in library retrieval (same InChIKey14 within 10 ppm) and analog search (≤ 5 MCES edits); the Fig. 4 panels print AUCs of 0.85 (DreaMS) vs 0.82 (entropy) / 0.80 (cosine) / 0.78 (MS2DeepScore) and 0.77 vs 0.74 / 0.72 / 0.71 (modified cosine). Fingerprint prediction "on par with MIST" on the MIST (NPLIB1) benchmark — no number given in the text. Fluorine detection on 17,000 in-house test spectra at ≥ 90 % precision thresholds. DreaMS Atlas: 3-NN network over 201 M spectra. | DreaMS Fig. 4 caption and text |
| MassSpecGym numbers | DreaMS is not a baseline in the MassSpecGym paper (verified: no DreaMS row in Tables 2–4 of arXiv v3). The only DreaMS retrieval number on MassSpecGym is in the audit paper: "DreaMS + ChemBERTa Alignment" (7-layer DreaMS, last two layers unfrozen, InfoNCE to frozen ChemBERTa) — 82.41 % hit@1 inflated → 6.70 % corrected (formula track). MassSpecGym v1.5 ships a retrained "data-safe" DreaMS. HF also hosts embedding_model_massspecgym.ckpt. |
audit Table 3, §A.4; HF tree |
| Open weights | HF roman-bushuiev/DreaMS (MIT): ssl_model.ckpt 1.39 GB, embedding_model.ckpt 1.24 GB, embedding_model_massspecgym.ckpt 1.24 GB, DreaMS_embedding_model_torchscript.pt 468 MB. Also Zenodo 10997887. Code github.com/pluskal-lab/DreaMS (MIT), docs dreams-docs.readthedocs.io; dreams.api.dreams_embeddings(mgf) → 1,024-d matrix. |
HF tree; README; docs |
A2. PRISM (Enveda + Microsoft)
What is public is a single Enveda blog post (2024-05-28; authors Healey, Domingo-Fernández, Taylor, Krettler, Lightheart, Park, Kind, Allen, Colluru) plus press coverage. Claims: 1.2 B MS/MS spectra, 85 B tokens; ≈ 600 M from public repositories (GNPS, MetaboLights, Metabolomics Workbench) + ≈ 600 M from Enveda's own platform; BERT-family encoder; "masked peak modeling" masking 20 % of peak masses; downstream: 7–16 % relative R² improvement across 12 chemical properties vs a baseline, and "23 % relative improvement" in spectral library matching vs "standard ML models". Parameter count, layer count, tokeniser, benchmarks used and baselines are not disclosed. I found no weights, no code and no preprint (the search budget ran out before I could exhaust bioRxiv; treat "no preprint" as unverified, confidence low). Note that Rose Lightheart (PRISM author) is a co-author of the MassSpecGym audit (Liu, Bushuiev, Lightheart et al.) — the challenge host knows the shortcut literature intimately, which is consistent with the RDKit tautomer-canonicalisation step in the Kaggle metric.
A3. MassSpecGym (NeurIPS 2024 Datasets & Benchmarks; arXiv 2410.23326 v1 2024-10-30, v3 2025-02-14)
Dataset. 231,104 spectra (231 k), ≈ 29 k unique structures (≈ 10 k, 33 %, newly measured in-house); sources MoNA, MassBank, GNPS, matchms-filtered; adducts only [M+H]+ and [M+Na]+; Orbitrap or QTOF; m/z < 1,000; spectra with > 50 % unexplained intensity removed. HF roman-bushuiev/MassSpecGym, MIT, 52.8 GB.
Split. Agglomerative clustering of the 29 k molecules on MCES distance, single linkage, threshold 10 — no two molecules in different folds are < 10 edge edits apart; folds stratified by instrument, collision energy, adduct, molecule frequency. The paper shows that the usual InChIKey-disjoint split leaks Tanimoto > 0.85 analogs.
Retrieval candidates. |C| ≤ 256 per spectrum, sampled with the same mass ("within an acceptable experimental error range" — tolerance not given in the main text) first from 1 M bio/environmental molecules, then 4 M diverse, then all 118 M PubChem (2024-05-31 download). Bonus track: same formula instead of same mass. Metrics: hit@1/5/20, MCES@1, 99.9 % bootstrap CIs (20,000 resamples).
Original baselines (arXiv v3 Table 3; %):
| Track | Method | Hit@1 | Hit@5 | Hit@20 | MCES@1 |
|---|---|---|---|---|---|
| mass | Random | 0.37 | 2.01 | 8.22 | 30.81 |
| mass | DeepSets | 1.47 | 6.21 | 19.23 | 25.11 |
| mass | Fingerprint FFN | 2.54 | 7.59 | 20.00 | 24.66 |
| mass | DeepSets + Fourier | 5.24 | 12.58 | 28.21 | 22.13 |
| mass | MIST | 14.64 | 34.87 | 59.15 | 15.37 |
| formula | Random | 3.06 | 11.35 | 27.74 | 13.87 |
| formula | DeepSets | 4.42 | 14.46 | 30.76 | 15.04 |
| formula | Fingerprint FFN | 5.09 | 14.69 | 31.97 | 14.94 |
| formula | DeepSets + Fourier | 6.56 | 16.46 | 33.46 | 14.14 |
| formula | MIST | 9.57 | 22.11 | 41.12 | 12.75 |
De novo (Table 2): every baseline (SMILES/SELFIES transformer, random generation) scores 0.00 % top-1 and top-10 exact accuracy; best MCES@top-10 was random generation (18.25 formula track). Simulation (Table 4): FraGNNet cosine 0.52, forward-model retrieval hit@1 46.64 % (mass track).
The audit — "MassSpecGym in the Wild: Uncovering and Correcting Evaluation Pitfalls in AI-Driven Molecule Discovery" (Liu, Bushuiev, Lightheart et al., arXiv 2606.19624, 2026-06-17). 34 papers in the first year; 26 report benchmark numbers; ≥ 17 of those 26 have evaluation issues in three classes: (i) data leakage (decoder pre-training sets excluding only exact InChIKeys, not MCES ≥ 10 analogs; ICEBERG simulators trained on test molecules; formula annotators trained on test), (ii) shortcut learning from candidate artefacts, (iii) implementation bugs and metric divergence.
Corrected / updated retrieval leaderboard (audit Table 3; formula-known track, |C| ≤ 256, %):
| Method | Hit@1 | Hit@5 | Hit@20 | MCES@1 | Status |
|---|---|---|---|---|---|
| Random | 3.06 | 11.35 | 27.74 | 13.87 | clean |
| MIST vMassSpecGym | 9.57 | 22.11 | 41.12 | 12.75 | clean |
| Nearest Neighbor | 9.58 | 22.26 | 39.92 | 9.00 | clean |
| JESTR | 11.82 | 33.48 | 61.46 | 11.71 | clean |
| MVP | 13.96 | 36.88 | 68.12 | 10.36 | clean |
| FLARE | 22.66 | 50.00 | 75.15 | — | clean |
| DiffMS (generative retrieval) | 32.29 | 54.36 | 75.47 | — | uses MIST vDiffMS checkpoint (grey) |
| ICEBERG (forward-model retrieval) | 36.18 | 60.70 | 78.83 | 6.67 | clean (CE imputed for ~half of test) |
| MIST + MolForge (generative retrieval) | 39.09 | 49.25 | 61.60 | 7.72 | grey (buggy MIST checkpoint) |
| FRIGID (generative retrieval) | 45.27 | 58.30 | 69.46 | 6.61 | grey ("a few test structures leaked") |
| MIST vDiffMS | 46.42 | 54.74 | 66.27 | 13.82 | grey (unreproducible checkpoint) |
| MIST vFRIGID | 53.76 | 65.32 | 74.71 | 5.16 | grey (same leak flag as FRIGID) |
| DreaMS + ChemBERTa (inflated → corrected) | 82.41 → 6.70 | 92.57 → 19.82 | 97.41 → 40.62 | 2.86 → 13.13 | canonicalisation shortcut |
| ChemFormer two-stage (inflated → corrected) | 63.89 → 7.60 | 70.52 → 19.12 | 75.86 → 39.31 | 5.39 → 13.23 | canonicalisation shortcut |
| ChemBERTa SMILES-only classifier (inflated → corrected) | 99.66 → 8.2 | 99.84 → 18.39 | 99.99 → 38.14 | 0.05 → 13.8 | spectrum-blind |
| PubChem default ranking (spectrum-blind) | 49.98 | 54.13 | 61.04 | 7.43 | artefact baseline |
| Rank by number of chiral atoms (spectrum-blind) | 4.88 | 16.85 | 37.86 | 14.4 | artefact baseline |
(Column alignment of the MCES values for FRIGID/vFRIGID in the PDF text is ambiguous; the 5.16/6.28 pair belongs to the two FRIGID rows, 6.61/8.73 to ICEBERG/MolForge — treat MCES assignments among those four rows as ± one row.)
Mass-based (formula-free) track after the audit (Table 2, 8; data-safe MIST retrained per FRIGID recipe): Random 0.43 %; MIST with data-safe MIST-CF top-1 formula 26.64 % hit@1 (38.47 / 54.90 @5/@20, MCES 16.41); with MIST-CF top-5 element-wise-max 28.23 %; with test-contaminated MIST-CF 31.39–35.15 % (the leak); with ground-truth formula 51.45 % (66.25 / 80.58). MIST-CF itself: 50.5 % top-1 / 73.2 % top-5 formula accuracy when clean, 66.5 / 84 % when trained on test. Simply filtering mass-candidates to the true formula lifts random from 0.43 % to 12.98 % hit@1 — a 30× likelihood ratio from the formula alone.
De novo (audit Table 9, formula track; top-1 acc / top-10 acc): MIST+MSNovelist 0 / 0; Spec2Mol 0 / 0; MIST+Neuraldecipher 0 / 0; MS-BART 1.07 / 1.11; MADGEN 1.31 / 1.54; DiffMS 2.30 / 4.25 (MCES 18.45 → audit prints 27.47 under its pinned MCES; see Table 10); FOAM 1.50 / 10.28; MIST+MolForge corrected 10.73 / 14.48 (inflated 31.75 / 40.55 from the batching bug); FRIGID 18.29 / 22.00. Metric divergence (Table 10, identical DiffMS predictions): Tanimoto 0.297 (Morgan r2, 1024 b) vs 0.214 (r3, 4096 b); MCES 21.21 vs 15.87 depending on solver bound. v1.5 pins InChIKey14, 2048-bit ECFP4, MCES threshold 15.
Unaudited claims to treat with suspicion: GLMR (Zhang et al., arXiv 2511.06259, Nov 2025) reports 64.17 % recall@1 mass-track / 68.48 % formula-track on MassSpecGym, with re-run MIST at 18.46 % and JESTR 17.62 % (mass track); the text never mentions candidate canonicalisation and the paper predates v1.5 — it is absent from the audited leaderboard. MARLIN (arXiv 2607.04774, July 2026), MS-GPT (2607.23607), SpecBridge (2601.17204), FlowMS (2603.18397) report on NPLIB1 or MassSpecGym with numbers I could not open; unverified. The live leaderboard massspecgym.onrender.com returned HTTP 503 on three attempts.
A4. CASMI 2022 (Fiehn lab, UC Davis)
500 compounds: 250 "priority" (not in public spectral libraries) + 250 "bonus"; raw LC-HRMS/MS in +ESI and −ESI (each challenge in one polarity only); inclusion list, 0.8 s dynamic exclusion; MS1, MS2 and RT provided; top-1 answer only, scored as exact InChIKey first-block match; deadline 2022-06-06; 16 groups. Instrument vendor/type not stated on the pages I could read (unverified).
2D-structure results (priority / bonus; correct / attempted):
| Submission | Priority | Bonus |
|---|---|---|
| Duhrkop (SIRIUS/CSI:FingerID — MIST paper calls CSI:FingerID "the highest performing method at CASMI2022") | 57/222 = 26 % | 64/219 = 29 % |
| Nothias | 58/228 = 25 % | 62/219 = 28 % |
| Rutz | 58/228 = 25 % | 62/219 = 28 % |
| Nikolic_KUCA (subset only) | 21/55 = 38 % | — |
| Mehta | 32/156 = 21 % | 7/33 = 21 % |
| 37/246 = 15 % | 31/246 = 13 % | |
| Tsugawa | 30/241 = 12 % | — |
| Namini/Mosley | 25/235 = 11 % | — |
| Wang | 23/249 = 9 % | 10/247 = 4 % |
| Wei_MSKU_1/2/3 | 18/14/12 of 247 = 7/6/5 % | — |
| Alcazar | 12/215 = 6 % | — |
| Frigerio/Andujar (MS-DIAL / patRoon) | 11/196 = 6 %; 1/22 = 5 % | — |
| Ressom | 10/234 = 4 % | — |
| Hong (pub / pred+pub) | 3/249 = 1 % | 5/247 = 2 %; 15/247 = 6 % |
| Young; Elser | 0/130; 0/97 | 0/122 (Elser) |
Formula: Duhrkop 94 % (priority) / 95 % (bonus), Nothias 93 %; class: Duhrkop 69 % priority, Wang 77 % bonus; adducts: 93–96 % for the best. Tools per participant beyond Duhrkop are not stated on the results page (MetFrag and CFM-ID entries are not identifiable there; unverified). Take-away: on "not-in-any-library" compounds the best 2022 top-1 rate was ≈ 26 % — roughly where the 2026 public-LB MRR@25 of ≈ 0.42–0.47 sits once you account for partial credit at ranks 2–25.
Part B — Forced analogy: medical differential diagnosis with likelihood ratios
The analogy, precisely. A clinician starts from a pre-test probability (prevalence given the presentation), then multiplies the odds by the likelihood ratio of each test result, LR = P(result | disease) / P(result | no disease), and only multiplies when tests are conditionally independent given the disease. Mapping: a candidate structure = a diagnosis; the prior = 1/|C| or a database-frequency prior; each scorer (fingerprint cosine, spectral-library similarity, forward-simulation match, formula agreement) = a test; a scorer's output for a candidate = a test result; LR = how much more often that score arises for the true structure than for a decoy. Rank-averaging corresponds to the clinically discredited habit of "counting positive tests".
(i) Crude per-class sensitivities and LRs from published numbers. The only per-class, two-method table I could reach was none (MIST's bioRxiv full text returned HTTP 429 on every attempt; the Nature version is paywalled to the fetcher; paperclip is not installed on this box). So I calibrated the marginal sensitivity of two real methods on identical candidate sets and modelled class-dependence as an explicit assumption: MassSpecGym formula track, |C| ≤ 256 — MIST 9.57 / 22.11 / 41.12 % and JESTR 11.82 / 33.48 / 61.46 % hit@1/5/20. Fitting an equal-variance Gaussian score model (true ~ N(μ,1), decoys ~ N(0,1)) gives μ_MIST = 1.30 and μ_JESTR = 1.60. Then three latent "classes" with priors 0.5/0.3/0.2 and multiplicative signal factors (A: 1.35/1.0/0.55; B: 0.6/1.0/1.45) — i.e. A is good where B is weak — assumed, not measured. At 1,000 candidates the implied sensitivities P(true in top-25 | class) are A: 0.415 / 0.251 / 0.105 and B: 0.157 / 0.355 / 0.637; against a random top-25 rate of 2.5 % these are LRs of 16.6 / 10.1 / 4.2 (A) and 6.3 / 14.2 / 25.5 (B). A genuinely measured LR from the audit: a correct formula filter lifts random hit@1 from 0.43 % to 12.98 %, LR ≈ 30.
(ii) Simulation. 400 molecules × 1,000 candidates, two scorers with shared noise fraction ρ, calibration fitted on a separate 400-molecule simulated "training" set (logistic regression of P(true | score), global or per class), 5 seeds. Code (lr_sim.py, numpy + scipy only):
# score model: decoys ~ N(0,1); true ~ N(mu_scorer[class],1); shared noise fraction rho
def simulate(n_mol, n_cand, mu_A, mu_B, class_prior, rho, rng):
cls = rng.choice(len(class_prior), size=n_mol, p=class_prior)
shared = rng.normal(0, 1, (n_mol, n_cand))
eA = np.sqrt(rho)*shared + np.sqrt(1-rho)*rng.normal(0, 1, (n_mol, n_cand))
eB = np.sqrt(rho)*shared + np.sqrt(1-rho)*rng.normal(0, 1, (n_mol, n_cand))
sA, sB = eA.copy(), eB.copy()
sA[:, 0] += mu_A[cls]; sB[:, 0] += mu_B[cls] # candidate 0 is the truth
return cls, sA, sB
def mrr_at_25(score):
rank = 1 + np.sum(score[:, 1:] >= score[:, [0]], axis=1) # pessimistic ties
return np.mean(np.where(rank <= 25, 1.0/rank, 0.0))
def rank_avg(sA, sB): # lower mean rank = better
rA = (-sA).argsort(1).argsort(1); rB = (-sB).argsort(1).argsort(1); return -(rA+rB)/2
def z_avg(sA, sB):
z = lambda s: (s - s.mean(1, keepdims=True)) / s.std(1, keepdims=True); return z(sA)+z(sB)
def naive_bayes(sA, sB, cls, models, per_class, joint=False):
# models[c] = (wA, wB, wAB): logistic fits of P(true|score) on a held-out simulated set.
# logit = log LR + log prior-odds; the prior is constant within a query, so sum of logits ranks
out = np.empty_like(sA)
for i in range(len(cls)):
wA, wB, wAB = models[cls[i] if per_class else None]
out[i] = (wAB[0] + wAB[1]*sA[i] + wAB[2]*sB[i]) if joint \
else (wA[0] + wA[1]*sA[i]) + (wB[0] + wB[1]*sB[i])
return out
# mu fitted so that hit@1/hit@20 at |C|=256 match MIST (9.57/41.12 %) and JESTR (11.82/61.46 %);
# classes: priors (0.5,0.3,0.2), factors A=(1.35,1.0,0.55), B=(0.6,1.0,1.45)
(iii) Results — MRR@25, mean of 5 seeds (SD ≤ 0.012 everywhere):
| Scenario | Method | ρ = 0.0 | ρ = 0.3 | ρ = 0.6 | ρ = 0.9 |
|---|---|---|---|---|---|
| class known | A only | 0.103 | 0.100 | 0.098 | 0.095 |
| class known | B only | 0.118 | 0.118 | 0.120 | 0.117 |
| class known | rank-average | 0.161 | 0.113 | 0.085 | 0.061 |
| class known | z-score average | 0.213 | 0.149 | 0.115 | 0.089 |
| class known | naive Bayes, global calibration | 0.212 | 0.150 | 0.115 | 0.089 |
| class known | naive Bayes, class-weighted LR | 0.241 | 0.183 | 0.145 | 0.117 |
| class known | joint logistic (class, correlation-aware) | 0.242 | 0.193 | 0.185 | 0.339* |
| class 80 % correct | rank-average | 0.161 | 0.113 | 0.085 | 0.061 |
| class 80 % correct | z-score average | 0.213 | 0.149 | 0.115 | 0.089 |
| class 80 % correct | naive Bayes, class-weighted LR | 0.229 | 0.173 | 0.137 | 0.111 |
| class 80 % correct | joint logistic (class, correlation-aware) | 0.229 | 0.179 | 0.166 | 0.301* |
| no class effect | B only | 0.115 | 0.117 | 0.117 | 0.116 |
| no class effect | rank-average | 0.188 | 0.136 | 0.107 | 0.083 |
| no class effect | z-score average | 0.223 | 0.158 | 0.120 | 0.094 |
| no class effect | naive Bayes (global = class-weighted) | 0.224 | 0.161 | 0.123 | 0.097 |
| no class effect | joint logistic | 0.223 | 0.160 | 0.125 | 0.123 |
* At ρ = 0.9 the joint model exploits the difference A − B, whose noise variance is 2(1 − ρ) = 0.2 while its signal μ_A − μ_B stays class-specific; that "common-mode rejection" is a property of my Gaussian shared-noise model, not something to expect from two real spectral scorers. Discount it.
Verdict. Bayesian (calibrated log-LR) combination beats rank averaging by 0.03–0.05 MRR@25 at every correlation, and class-weighting the LRs adds another 0.02–0.03; rank-averaging two correlated scorers is worse than the better scorer alone once ρ ≥ 0.3, and every independence-assuming combiner (rank, z, naive Bayes) stops beating the single best scorer around ρ ≈ 0.6. Z-averaging and global naive Bayes are indistinguishable here because, for equal-variance Gaussian scores, log LR is linear in the score — so the honest message is "calibrate, then add log-odds", not "Bayes is magic". The gain of class-weighting survives a 20 % class-misassignment rate (0.229 vs 0.213 at ρ = 0). Absolute values are lower than published hit rates because 1,000 candidates were used, not 256.
Does the analogy hold? Partially. It holds for the core prescription (prior × LR, not vote counting), for the role of the formula/mass filter as a high-LR "screening test" (LR ≈ 30 measured), and for class-conditional calibration (a test's LR depends on the patient subgroup = compound class). It breaks on conditional independence: fingerprint-cosine scorers trained on the same libraries share failure modes (same missing substructures, same adduct confusions), exactly the ρ ≥ 0.6 regime where the naive product over-counts evidence. It also breaks on the prior: clinicians have measured prevalences; here the natural prior (1/|C|) ignores that PubChem decoys are not drawn from "chemistry that gets measured" — the audit's spectrum-blind PubChem-order baseline reaching 49.98 % hit@1 is a prior, not a test, and should enter the fusion as log prior-odds, not as another LR.
Gotchas — what could silently fool us
- Canonicalisation leak in any locally built candidate set. If your true-structure SMILES come from the training parquet (
normalized_smiles) and decoys from COCONUT/PubChem canonicalised differently, a learned re-ranker will reach 80–99 % locally and ≈ 7 % on the hidden test (audit Table 3). Canonicalise everything with the same RDKit version and tautomer routine as the metric (RDKit 2026.03.3). - Formula leakage dressed as "mass-based". Any formula annotator (SIRIUS, MIST-CF, BUDDY) trained or tuned on molecules that appear in your validation set inflates hit@1 by 9 points absolute (26.6 → 35.2 %); a correct formula alone is a 30× LR. The hidden test molecules are "none previously published" — a formula model trained on all public libraries is fine for the test but will lie on any validation set drawn from those libraries.
- Spectrum-blind priors masquerade as skill. PubChem default order gives 50 % hit@1 on MassSpecGym; number of chiral centres gives 4.9 %. On a COCONUT-sourced pool, "is in COCONUT at all", NP-likeness, or deposition frequency will carry similar spurious signal — fine to use as a prior, dangerous to mistake for a spectral model.
- Batched-inference masks. The MIST/DiffMS attention mask added +1 instead of excluding padding; a reused checkpoint tripled de novo accuracy (10.7 → 31.8 %). Check that batched and single-spectrum outputs agree before trusting any borrowed encoder.
- Metric parameterisation. Tanimoto moves 0.30 → 0.21 across Morgan radius/bit choices and MCES 21.2 → 15.9 across solver bounds on the same predictions. The Kaggle metric is InChIKey14 after tautomer canonicalisation — a prediction that differs only by a tautomer is correct, one that differs by a stereocentre is also correct, one that differs by a double-bond position is wrong. Local MRR must use the same routine.
- "Formula track" is harder than "mass track" for fingerprint methods. MIST drops from 14.64 % (mass candidates) to 9.57 % (formula candidates) because same-formula isomers have near-identical fingerprints. A COCONUT pool with a tight mass window is effectively a formula-track problem for most queries.
- Collision energies missing. Roughly half of the MassSpecGym test split lacks CE; ICEBERG's numbers required imputing them. The hidden test gives CE lists (merged multi-CE spectra); models trained on single-CE library spectra see a distribution shift.
- Foundation-model benchmarks are mostly on Orbitrap. 97 % of GeMS-A is Orbitrap; the hidden test is all timsTOF. DreaMS's Fourier m/z features assume ~4-decimal accuracy; QTOF spectra are in GeMS-C only.
- The leaderboard you cite may be the uncorrected one. Published papers (GLMR 64 %, FRIGID 45 %, MIST vFRIGID 54 %) sit above the audited clean frontier (FLARE 22.7 %, ICEBERG 36.2 %, formula track). The clean formula-free number to beat is MIST + data-safe MIST-CF ≈ 27–28 % hit@1 at |C| ≤ 256.
- CASMI 2022 was top-1 only, one polarity per challenge, with RT and MS1 — its 26 % is not an MRR@25 and was obtained with information the 2026 challenge does not give (RT) and without the information it does (multi-CE merged spectra).
- Simulation caveats. Equal-variance Gaussian scores, a single shared-noise ρ for truth and decoys, three fictitious classes, calibration on simulated rather than real data; class factors are assumptions. The ordering of methods is robust across the three scenarios; the magnitudes are not to be quoted as predictions.
Implications for the challenge
- Fuse scorers by summing calibrated log-odds, never by averaging ranks. Fit per-scorer P(true | score) with logistic/isotonic regression on an MCES-disjoint, novel-molecule validation set; rank by Σ logit. Expected +0.03–0.05 MRR@25 over rank averaging in simulation.
- Add a class-conditional calibration layer. Predict NPClassifier/ClassyFire superclass from the spectrum (CANOPUS-style head or DreaMS fine-tune, 80 % accuracy suffices) and fit calibration per class; worth ≈ +0.02–0.03 in simulation and it is exactly where "scorer A is good on terpenoids, B on alkaloids" lives.
- Measure ρ between your scorers on validation (correlation of per-query true-candidate z-scores or of rank errors). If ρ ≥ 0.6 for two fingerprint models, drop one or replace the product with a jointly trained 2-feature logistic re-ranker; seek genuinely orthogonal evidence (forward-simulated spectrum match, spectral-library analog similarity, formula/adduct consistency, isotope pattern) rather than a third fingerprint.
- Treat formula/mass filtering as the highest-LR test and get it right first. Use a formula annotator trained on all public data (no leakage risk against unpublished test molecules), take its top-k formulas with calibrated probabilities, and feed them as log prior-odds over candidates, not as a hard filter — 51 % vs 27 % hit@1 is the stake.
- Use DreaMS as a feature extractor, not as a solved component:
embedding_model_massspecgym.ckptor the v1.5 data-safe DreaMS, 1,024-d embeddings, single-GPU; fine-tune to fingerprints with the Murcko-histogram split protocol. Its MassSpecGym retrieval number (6.7 % corrected, formula track) says the published alignment recipe is not a shortcut to a strong ranker. - Validate exactly like the audit's v1.5: uniform RDKit canonicalisation of truth and decoys, MCES ≥ 10 (or at least Tanimoto < 0.7) between train and validation molecules, pinned metric code, and a spectrum-blind baseline (database order, NP-likeness, chiral-centre count) reported next to every model — if the blind baseline is within a few points of the model, the model has learned the pool, not the spectrum.
- Do not benchmark against PRISM: nothing is reproducible; its only transferable lessons are "masked-peak pre-training works at 20 % masking" and that Enveda's half of the corpus is proprietary — expect the hidden test to resemble enveda-180 acquisition conditions more than GNPS.
- Set expectations with the audited numbers: clean formula-free hit@1 ≈ 27 % at 256 candidates; at a COCONUT-scale pool with 10 ppm windows the public LB's ≈ 0.42–0.47 MRR@25 is consistent with that, so headroom is in candidate coverage (recall@25) and calibrated fusion, not in a bigger single encoder.
Open questions
- MIST paper per-class (NPClassifier) MIST-vs-CSI:FingerID retrieval tables (Fig. 2/3 and SI) — needed to replace my assumed class factors with measured ones; bioRxiv 10.1101/2022.12.30.522318 and Nature 10.1038/s42256-023-00708-3 could not be opened here.
- Whether a PRISM preprint exists (search budget exhausted); whether PRISM's "library matching +23 %" was measured on timsTOF data.
- MassSpecGym candidate mass tolerance (ppm/mDa) — stated only in the SI; and whether v1.5 candidate JSONs change |C|.
- Whether FRIGID/MIST-vFRIGID's "few leaked test structures" flag was resolved in a later revision; the live leaderboard (503) would show it.
- CASMI 2022 instrument and per-participant tools beyond Duhrkop (results page lists names only).
- Real ρ between fingerprint-cosine, DreaMS-similarity and forward-simulation scorers on timsTOF natural products — the single number that decides whether naive LR fusion or a joint re-ranker is right.
Sources
- Bushuiev R et al. Self-supervised learning of molecular representations from millions of tandem mass spectra using DreaMS. Nat Biotechnol 44:630–640 (2026, online 2025-05-23). https://doi.org/10.1038/s41587-025-02663-3 (PDF read; Europe PMC record PMC13090125 https://www.ebi.ac.uk/europepmc/webservices/rest/search?query=DOI:%2210.1038/s41587-025-02663-3%22&format=json)
- DreaMS code/README: https://raw.githubusercontent.com/pluskal-lab/DreaMS/main/README.md ; docs https://dreams-docs.readthedocs.io/en/latest/ ; weights https://huggingface.co/roman-bushuiev/DreaMS/tree/main ; GeMS https://huggingface.co/datasets/roman-bushuiev/GeMS ; Zenodo 10997887
- Enveda. PRISM: A foundation model for life's chemistry (2024-05-28). https://www.enveda.com/prism-a-foundation-model-for-lifes-chemistry/
- Bushuiev R et al. MassSpecGym: A benchmark for the discovery and identification of molecules. arXiv:2410.23326 (v3, 2025-02-14), NeurIPS 2024 D&B. https://arxiv.org/abs/2410.23326 (HTML v3 and PDF read); README https://raw.githubusercontent.com/pluskal-lab/MassSpecGym/main/README.md ; dataset https://huggingface.co/datasets/roman-bushuiev/MassSpecGym
- Liu, Bushuiev, Lightheart et al. MassSpecGym in the Wild: Uncovering and Correcting Evaluation Pitfalls in AI-Driven Molecule Discovery. arXiv:2606.19624 (2026-06-17). https://arxiv.org/abs/2606.19624 (HTML and PDF read)
- Bohde M et al. DiffMS: Diffusion Generation of Molecules Conditioned on Mass Spectra. arXiv:2502.09571 (ICML 2025). https://arxiv.org/html/2502.09571v2
- Zhang Y et al. Breaking the Modality Barrier: Generative Modeling for Accurate Molecule Retrieval from Mass Spectra (GLMR). arXiv:2511.06259 (2025-11-09). https://arxiv.org/abs/2511.06259
- Che X, Du X, Xu D. MARLIN. arXiv:2607.04774 (2026-07-06). https://arxiv.org/abs/2607.04774 (abstract only)
- De Waele G et al. Small molecule retrieval from tandem mass spectrometry: what are we optimizing for? arXiv:2602.16507 (2026-02-18). https://arxiv.org/abs/2602.16507 (abstract only)
- Goldman S et al. Annotating metabolite mass spectra with domain-inspired chemical formula transformers. Nat Mach Intell (2023). https://doi.org/10.1038/s42256-023-00708-3 (abstract/summary only — full text not accessible); code README https://raw.githubusercontent.com/samgoldman97/mist/master/README.md
- Fiehn Lab CASMI 2022: https://fiehnlab.ucdavis.edu/casmi ; results https://fiehnlab.ucdavis.edu/casmi/casmi-2022-results ; FAQ https://fiehnlab.ucdavis.edu/casmi/casmi-faq
- Simulation code and raw output:
lr_sim.py,lr_sim_results.json(session scratchpad; code reproduced above; numpy 2.5.3, scipy 1.18.1). - Not reachable in this session: massspecgym.onrender.com (HTTP 503 ×3), ChemRxiv DreaMS preprint page (403), bioRxiv MIST full text (429 ×4), casmi-contest.org (TLS certificate mismatch).