TL;DR. Treating a spectrum as a sentence to be translated into SMILES has produced the same failures machine translation documented in 2006–2019: token-level cross-entropy models emit fluent-but-wrong molecules (on MassSpecGym a SMILES transformer scores 0.00 % exact matches and is worse than random on structural distance, MCES 53.8 vs 28.6), while the published objective that actually tracks MRR is a softmax cross-entropy over the formula/mass-matched candidate set (candidate-based InfoNCE: 47.5 % R@1 vs 19.5 % for in-batch InfoNCE on the same model, MSAlign 2026). Retrieval with candidates still beats de novo generation by a wide margin (≥47 % vs ≤24 % top-1 on the structure-disjoint split, and the de novo numbers assume the true formula), so for the ~45 % of CASMI molecules outside the pool the only realistic de novo use is MBR-style consensus (sample, group by InChIKey14, rank by frequency, re-score) as a slot filler. The stylometry analogy yields one testable reranker feature: a Burrows-Delta z-scored profile of common neutral-loss frequencies, predicted per candidate by a forward model and compared with the observed spectrum.
1. SMILES-as-language de novo models (Question 1)
All MassSpecGym numbers below are on the MCES ≥ 10 structure-disjoint test split, [M+H]+ / [M+Na]+ only, single spectrum. "Formula given" means the ground-truth molecular formula was supplied (MassSpecGym "bonus" setting).
| Model (year, venue) | Architecture | Training objective | Decoding / ranking of the k outputs | MassSpecGym top-1 / top-10 exact | Tanimoto top-1 | MCES@1 | Other benchmark | Beats retrieval? |
|---|---|---|---|---|---|---|---|---|
| SMILES Transformer (MassSpecGym baseline, NeurIPS 2024) | peak-token encoder → SMILES decoder | token cross-entropy | beam / sampling top-k | 0.00 / 0.00 (formula given: 0.00 / 0.00) | 0.07 (formula given: 0.03) | 53.80 (formula given: 79.39) — random generation: 28.59 / 21.11 | — | No |
| SELFIES Transformer (same) | same, SELFIES tokens (always valid) | token cross-entropy | same | 0.00 / 0.00 | 0.10 | 33.28 (formula given 38.88) | — | No |
| Spec2Mol (Litsa et al., Commun Chem 2023) | 1-D CNN on 4 spectra (±, 2 CE) → latent of a GRU SMILES autoencoder pretrained on ~135 M PubChem/ZINC molecules | RMSE between spectrum embedding and SMILES-encoder embedding (no token CE at the spectrum stage) | autoencoder decode | 0.00 / 0.00 (DiffMS re-run); valid SMILES 68.5 % (MSG) / 66.5 % (NPLIB1) | 0.12 | 37.76 | own NIST20 random split: 7.0 % exact (15.2 % when 3 spectra available), 39.3 % exact formula | No |
| MS2Mol (Butler et al., Enveda, ChemRxiv June 2023; closed) | BART-style seq2seq; m/z + intensity tokens → formula then SMILES | token cross-entropy (per abstract; PDF 403) | 25 beams + ranker (per sibling q5 file; unverified by me) | not on MassSpecGym | — | — | EnvedaDark (226 NPs absent from DBs): 21 % "close match" (Tanimoto ≥ 0.675), 62 % "meaningful" (≥ 0.4); top-10 % confident: 63 % / 98 % | Not tested against retrieval; exact-match rate not reported in anything I could open |
| MassGenie (Shrivastava et al., Biomolecules 2021) | not opened by me — see research/q5_methods_lineage/denovo_generation_*.md |
— | — | n/a | — | — | — | — |
| MADGEN (Wang, Chen, Liu, Hassoun, ICLR 2025) | stage 1: contrastive (InfoNCE) scaffold retrieval; stage 2: scaffold-conditioned Markov-bridge graph generation with classifier-free guidance, spectrum cross-attention | contrastive (stage 1) + bridge likelihood (stage 2) | — | 1.31 / 1.54 (oracle scaffold: 10.5 / 12.4); scaffold-retrieval accuracy 13.2 % (MSG), 17.7 % (CANOPUS), 40.3 % (NIST23) | 0.20 | 27.47 | CANOPUS 2.10 / 2.39; NIST23 4.6 / 7.3 | No |
| DiffMS (Bohde et al., ICML 2025) | MIST formula transformer encoder (pairwise neutral-loss features) → discrete graph diffusion over bonds with heavy atoms fixed by formula; decoder pretrained on 2.8 M fingerprint→molecule pairs (DSSTox, HMDB, COCONUT, MOSES) | cross-entropy between predicted and true adjacency (diffusion denoising) | sample 100 per spectrum, drop invalid/disconnected, rank by frequency | 2.30 / 4.25 (with MIST-CF formulae: 1.86 / 4.10); 100 % valid | 0.28 | 18.45 | NPLIB1 8.34 / 15.44 | No |
| MS-GPT (Zhao et al., arXiv 2607.23607, Jul 2026) | spectrum → fingerprint posterior → M = 20 calibrated bit-density queries × N = 5 samples (B = 100) from a SAFE-GPT molecule LM | molecule-LM likelihood conditioned on posterior queries | pool, group by InChIKey-14, rank by generation frequency; B 100 → 1500 keeps improving | 23.91 / 28.65 | 0.52 | — | NPLIB1 29.76 / 41.07 | No (vs candidate retrieval) |
| DualLGD (Che, Du, Xu, arXiv 2605.07048, May 2026) | dual-stream line-graph diffusion; atoms fixed by formula, bonds denoised | diffusion loss | 100 samples, ranked by model likelihood (no consensus step described) | 23.89 / 33.08 | — | — | NPLIB1 34.37 / 44.96 | No |
| FRIGID (Bohde et al., arXiv 2604.16648, Apr 2026) | masked-graph generation + iterative refinement; ICEBERG forward model flags "hallucinated fragments" for re-masking | — | ranks refinement candidates by Tanimoto to the MIST-predicted fingerprint; 100 % valid | 18.29 / 22.00 (number from sibling q5 file / audit Table 9; I did not re-open the table) | — | — | NPLIB1 25.03 / 33.37 | No |
Two facts stand out. (i) Until Feb 2025 every SMILES/SELFIES decoder scored exactly 0.00 % on the structure-disjoint split; the jump to ~24 % (DualLGD, MS-GPT) came from graph/posterior-query models that fix the formula and aggregate many samples, not from better token cross-entropy. (ii) None of them beats retrieval when candidates exist: on the same MassSpecGym formula setting, MSAlign reports 47.5 % R@1, GLMR 68.48 %, SpecBridge 84.7 % (the last two unaudited, see Gotchas). The de novo top-1 numbers also presuppose the true formula; DiffMS drops from 2.30 % to 1.86 % when MIST-CF supplies the formula.
2. The "grammar of fragmentation" and where the parse-tree analogy breaks (Question 2)
Formal content. Böcker & Rasche (Bioinformatics 2008) build, for one spectrum and one precursor formula, a directed graph whose vertices are molecular-formula explanations of peaks (vertices coloured by peak) and whose edges are fragmentation steps labelled by a neutral loss; the fragmentation tree is the maximum-weight colourful subtree (each colour/peak used at most once). Finding it is NP-hard; they gave three exact algorithms and two heuristics, and on 32 test compounds the correct formula was top-5 for all 32 and top-1 for 26 (exact algorithm). "Fragmentation trees reloaded" (Böcker & Dührkop, J Cheminform 2016, PMC4736045) makes the scoring explicitly Bayesian: P(T|D) ∝ P(D|T)·P(T), and the tree weight is a sum of log-odds terms, so log P(T)·P(D|T) = L(T) + c. The terms are (a) ~40 common losses with priors learned from training data (water 18.0106, CO 27.9949, NH3 17.0265 Da, H radical, …), (b) a log-normal prior on loss mass, P_loss-mass(Δ) = 1/(Δ√(2πσ²))·exp(−(ln Δ − μ)²/(2σ²)) with μ, σ estimated by weighted maximum likelihood over observed losses, (c) a radical-loss prior (implausible radicals 10⁻³), (d) the 40 most common fragments with a prior scaled by the 80th most common, plus mass-deviation and intensity terms. Formula identification rose from 31.1 % (SIRIUS 2) to 76.0 % top-1 on 2,046 Agilent + 2,005 GNPS compounds (84.8 % with isotope filtering; 93.3 % on 60 pesticides). SIRIUS 6 documentation calls the optimisation "a Maximum Aposteriori Estimator" over colourful subtrees.
As a grammar. A fragmentation tree is a derivation: each edge is a production "ion → smaller ion + loss", the loss lexicon is the ~40 learned common losses (closed-class, high-frequency) plus an open class governed only by the loss-mass prior, and the tree score is the log-likelihood of the derivation under a context-free-like model whose rule probabilities are the learned loss priors. Learned grammars replace the hand-built lexicon: SCARF (Goldman et al. 2023) decodes product subformulae atom-type by atom-type through a prefix tree and then assigns intensities; ICEBERG (Goldman et al. 2024) learns an autoregressive atom-removal policy (max depth 3 at training, top-100 fragments at inference, MAGMa-derived ground-truth DAGs) and a Set-Transformer scorer that predicts a separate intensity for each hydrogen shift δ ∈ ±0…±6 per fragment; it reaches top-1 retrieval 29.0 % on NPLIB1 (MassFormer 19.8 %, SCARF 13.5 %) and 18.9 % on NIST20, cosine 0.627 (NPLIB1). FraGNNet (Young et al., TMLR 2025) is explicitly probabilistic — P_θ(n) over DAG nodes (depth d ∈ {3,4}), P_θ(f|n) over formulae, ±j hydrogen tolerance, median DAG support 679 formulae on NIST20 — trained with NLL plus an out-of-support term, and ties ICEBERG at 36.0 % top-1 / beats it at top-5 (78.4 vs 75.4 %) on NIST20.
Where it breaks.
- No fixed lexicon. The long tail of losses is handled by a log-normal mass prior, not by grammar; the ~40-loss "vocabulary" is a frequency cutoff, and the same mass can be several formulae (the parse is over formulae inferred from m/z at ppm tolerance, so the "words" themselves are ambiguous).
- Rearrangements are ungrammatical but real. Hydrogen migrations are patched in as δ-shift channels (ICEBERG ±6 H, FraGNNet ±j), skeletal rearrangements that create fragments unreachable by bond cleavage are outside every learned grammar (ICEBERG's ground truth comes from MAGMa bond-breaking DAGs).
- Spurious ambiguity. SIRIUS's own docs list two systematic artefacts: "pull-ups" (the optimiser "tends to generate deep trees, favoring many small fragmentation steps over fewer larger ones") and "parallelograms" (consecutive losses that occur in either order, which a tree cannot represent). The tree is a maximum-likelihood derivation, not a mechanism.
- The parse is per formula, not per structure. A fragmentation tree discriminates molecular formulae; it says nothing about which of 256 isomers you have until CSI:FingerID/MIST turn tree features into fingerprint predictions. For MRR@25 the grammar is the formula gate, not the ranker.
3. Machine-translation pathologies and their mass-spec analogues (Question 3)
| MT pathology (source, numbers) | Documented spectrum→structure analogue | Status |
|---|---|---|
| Metric ≠ quality. Callison-Burch, Osborne & Koehn (EACL 2006): BLEU "vastly underestimates" the rule-based SYSTRAN system against human adequacy/fluency (R² = 0.002 for fluency when the outlier is included); BLEU unreliable "for comparing systems which employ different translation strategies". | MassSpecGym in the Wild (Liu, Bushuiev et al., arXiv 2606.19624, June 2026): ≥ 17 of 26 papers had evaluation issues; a ChemBERTa SMILES classifier fell from 99.66 % to 8.2 % once candidate SMILES were uniformly canonicalised (shortcut learning); metric divergence from Morgan radius/bit length and non-14-char InChIKey matching. | Documented |
| Fluent but inadequate. Koehn & Knowles (2017): out-of-domain NMT 3.9 BLEU vs SMT 10.2 (medical→law); NMT "completely sacrifice[s] adequacy for the sake of fluency". | SMILES Transformer MCES@1 53.80 vs random generation 28.59, and 79.39 vs 21.11 when the formula is given — the decoder produces well-formed molecules unrelated to the spectrum. | Documented (MassSpecGym Table 2) |
| Validity ≠ correctness. | SELFIES decoder is 100 % valid yet 0.00 % exact; Spec2Mol 66.5–68.5 % valid; MIST+Neuraldecipher 91.11 %, MSNovelist 98.60 %, DiffMS/FRIGID 100 % (graph). | Documented (DiffMS Table 4) |
| Beam-search curse. Koehn & Knowles: BLEU peaks at beam 4–8 then declines (Cs-En 30.4 → 29.9 at beam 1000; with length normalisation optimum shifts to 30–50). Stahlberg & Byrne (EMNLP 2019): the empty string is the global mode for 51.8 % of sentences; search errors 73.6 % (greedy), 57.7 % (beam-10), 53.62 % (beam-100); exact inference gives 2.1 BLEU (length ratio 0.06) vs 30.3 for beam-10; "vanilla NMT … requires just the right amount of beam search errors". | No paper reports exact-mode decoding of a spectrum→SMILES model; the closest evidence is that sample-based consensus scales the other way (MS-GPT improves steadily from B = 100 to 1500) because it is MBR, not MAP. Whether spectrum→SMILES models put their mode on trivially short molecules is untested. | Hypothesis |
| Exposure bias / train–test mismatch. Ranzato et al. (2016): word-level next-token training vs sequence-level generation, errors accumulate; sequence-level training on the test metric fixes greedy decoding. | Every SMILES decoder above is trained with next-token CE and evaluated with exact InChIKey14 match; no spectrum→SMILES paper I opened uses sequence-level (RL/MBR) training. MADGEN's two-stage design (scaffold by contrastive retrieval, then generation) is the nearest structural fix. | Gap |
| Calibration / label smoothing. Müller, Kornblith & Hinton (2019): label smoothing improves calibration and "can substantially enhance beam-search performance" but hurts distillation. | Contrastive temperature plays the same role: De Waele et al. (2026) sweep τ ∈ {1/16, 1/64, 1/256} and HR@1 moves 11.3 → 12.6; Jürgens et al. (2026) find fingerprint-level uncertainty a "poor proxy for retrieval success" while retrieval-level total uncertainty is the strongest rejection criterion. | Partially documented |
| Mode collapse to frequent outputs. | MADGEN's scaffold-retrieval ceiling (13.2 % on MSG) and DiffMS/MS-GPT frequency ranking (the mode of 100 samples) make the first slot the modal scaffold; no paper measures scaffold diversity of the 25-list. | Gap |
4. Why top-1 SMILES cross-entropy is the wrong objective for MRR@25, and what is closest (Questions 4–5)
MRR@25 rewards the reciprocal rank of the true InChIKey14 inside a list that we construct. Token cross-entropy on one reference SMILES (a) rewards exact string reproduction (stereo tokens, atom order) that the metric ignores, (b) says nothing about the relative order of the other 24 slots, and (c) is maximised by the mode of a distribution that, as in NMT, need not be adequate. The retrieval literature has converged on objectives defined over the candidate set:
| Method (year) | Loss | Negatives | Reported retrieval (MassSpecGym unless stated) | Source |
|---|---|---|---|---|
| CSI:FingerID (SIRIUS 4) | fingerprint SVMs + Bayes/modified Platt scoring | — | CANOPUS, 3 structural splits: top-1 43.00 %, top-20 87.23 % (CSI+Bayes); CSI+Cosine 38.24 / 83.20 | MIST SI Table S4 |
| MIST (Goldman et al., Nat Mach Intell 2023) | stage 1 cosine loss on 4096-bit fingerprint; stage 2 contrastive fine-tune | hard isomer decoys from PubChem (--negative-strategy hardisomer_tani_pickled, --num-decoys 64, --max-db-decoys 256, --contrastive-weight 0.6, --contrastive-scale 16; repo README) |
CANOPUS: MIST+Cosine 29.72 / 73.60 (top-1 / top-20) → MIST+Contrastive 34.45 / 85.75; 5× ensemble 37.39 / 88.20. Ablation (Table S8): FFN fingerprint 17.3 → FFN contrastive 20.6 top-1. MassSpecGym baseline: 14.64 / 34.87 / 59.15 (mass), 9.57 / 22.11 / 41.10 (formula) | MIST SI; MassSpecGym Table 3; JESTR Table 2 |
| MassSpecGym fingerprint FFN / DeepSets (2024) | BCE on fingerprint bits | none | hit@1 2.54 (FFN), 1.47 (DeepSets), 5.24 (DeepSets+Fourier), random 0.37 | MassSpecGym Table 3 |
| CMSSP (Chen et al., Anal Chem 2024, 10.1021/acs.analchem.4c03724) | contrastive joint space + discriminator | in-batch | own: +30 % top-1 on CASMI 2017; JESTR re-eval on MSG formula: 2.30 (retrained) / 3.61 (pretrained) rank@1 | CMSSP abstract; JESTR Table 2 |
| ESP (Li et al. 2024, mol-to-spec) | spectrum prediction, rank by similarity | — | MSG formula 11.05 / 27.42 / 52.20 | JESTR Table 2 |
| JESTR (Kalia, Chen, Krishnan, Hassoun, 2025) | NT-Xent/InfoNCE on cosine/τ between MLP spectrum and GNN molecule embeddings | in-batch, plus a "regularization" loss = mean cosine(spectrum, k_aug most Tanimoto-similar PubChem same-formula candidates), switched on for the last 3 % of epochs with β = 0.1 | NPLIB1 45.76 / 81.53 / 95.77 (without regularisation 41.06 / 81.12 / 96.13); MSG mass 15.13 / 36.75 / 60.32 (NR: 15.62 / 37.47 / 60.55); MSG formula 11.85 / 32.95 / 61.46 (NR: 11.82 / 33.48 / 61.46). SIRIUS on the same MSG formula subset: 19.04 % rank@1 overall, 10.41 % after removing SIRIUS-training overlap; CFM-ID 1.96 % | JESTR §2.3, Table 2, §3 |
| De Waele, Wydmuch, Dembczyński, Kotłowski, Waegeman (arXiv 2602.16507, Feb 2026) | systematic comparison: BCE, cosine, IoU, focal, contrastive Emb-Cos / FP-Cos / Fp-MLP | MassSpecGym candidate sets; Table 5 adds custom PubChem "hard equal-mass" negatives (1,024) | mass setting HR@1/5/20: BCE 8.94 / 18.14 / 32.45; cosine 7.49 / 15.82 / 29.39; IoU 5.58 / 13.01 / 26.51 (best fingerprint Tanimoto 19.42); Emb-Cos τ = 1/64 12.08 / 27.42 / 47.44; Fp-MLP τ = 1/16 12.60 / 27.84 / 47.88; formula setting Emb-Cos 13.62 / 26.33 / 46.50. Hard PubChem negatives: 11.64 (mass) / 14.10 (formula) vs standard MSG candidates 12.29 / 14.17 — no gain. Conclusion: "optimizing for more accurate fingerprint predictions typically worsens retrieval results, and vice versa" | §Results, Tables 1–5 |
| MSAlign (Krzakala et al., arXiv 2605.19752, May 2026) | candidate-based InfoNCE: L = −1/B Σ log exp(ρ(s,m⁺)/ε) / Σ_{m'∈C_K(s)} exp(ρ(s,m')/ε), with the bound 1 − R@1 ≤ L_cand / log 2 | K mass-matched PubChem candidates (10 ppm; 128 per query in Table 8, 256 elsewhere in the text) — frozen DreaMS + MolDeBERTa with MLP adapters | formula split R@1/5/20: candidate InfoNCE 47.5 ± 3.4 / 71.3 / 87.0 vs in-batch InfoNCE 19.5 ± 2.1 / 42.6 / 67.7 vs regression 22.8 / 42.0 / 60.3; late fusion of four scorers 58.2 / 80.7 / 92.8; MCES split 19.0 ± 3.2 vs JESTR 7.9 ± 1.1; in-batch loss shown "decorrelated from accuracy" (a trivial mass scorer reaches loss < 10⁻² with 0 % top-1) | §3, Appendix A.1, Tables 2, 8, 11 |
| FLARE (bioRxiv 10.64898/2026.01.27.702086, Jan 2026) | symmetric InfoNCE with fine-grained peak-node alignment (subformula tokens ↔ GCN atoms) | in-batch only (no hard-negative ablation) | rank@1 43.15 (mass), 22.66 (formula); audit-corrected formula track 22.66 / 50.00 / 75.15 | FLARE; audit Table 3 |
| GLMR (Zhang et al., AAAI 2026, arXiv 2511.06259) | stage 1 dual InfoNCE (mol→ms negatives by intensity perturbation, N = 1; ms→mol in-batch, M = 1); stage 2 autoregressive SMILES generation conditioned on spectrum + top-K candidates, then re-rank candidates by similarity to the generated molecule | in-batch | MSG formula 68.48 / 78.09 / 84.22; mass 64.17 / 72.96 / 78.78 (JESTR 11.77 / 33.26 / 61.01; MIST 10.94 / 23.82 / 44.63); MassRET-20k (NIST20-derived, 12 adducts) 54.04 / 64.35 / 72.98 | GLMR Tables |
| SpecBridge (Wang, Chen, Chen, Liu, Hassoun, arXiv 2601.17204, Jan 2026) | MSE/cosine regression of fine-tuned DreaMS output onto frozen ChemBERTa molecule embeddings; no negatives at all | none | MSG R@1/5/20 84.7 / 92.3 / 96.9, MRR 88.1, MCES@1 2.37 (GLMR 68.5 / 78.1 / 84.2, MRR 72.5); Spectraverse R@1 36.6 % (1,494 avg candidates); MSnLib 53.4 % (3,041 candidates) | SpecBridge |
| MADGEN stage 1 (ICLR 2025) | InfoNCE scaffold retrieval | in-batch | scaffold top-1 13.2 % (MSG), 17.7 % (CANOPUS), 40.3 % (NIST23) | MADGEN |
| Yoo, Kim, Yoon, Kim (arXiv 2602.00547, Jan 2026) | cross-modal alignment into a chemical-LM embedding | — | 42.2 % top-1 in fixed 256-way zero-shot retrieval on scaffold-disjoint benchmarks | abstract only |
| MS2MetGAN (Tsai et al., arXiv 2603.13342, Mar 2026) | autoencoder latents + GAN-generated decoy metabolite vectors as negatives | adversarial | "better overall performance"; no numbers in abstract | abstract only |
Direct comparisons of negatives (Question 5). Three papers isolate the variable:
- MSAlign: same encoders, same data — in-batch InfoNCE 19.5 % R@1 vs candidate-set InfoNCE 47.5 % (formula split), i.e. +28 points from using the chemistry-defined candidates as negatives.
- JESTR: adding Tanimoto-sorted same-formula PubChem candidates as a push-away regulariser gains +4.7 points on NPLIB1 (41.06 → 45.76) and an average "+5.72 % across all datasets", but nothing on MassSpecGym (15.62 → 15.13 mass; 11.82 → 11.85 formula).
- De Waele et al.: replacing the MassSpecGym candidate sets with 1,024 "hard equal-mass" PubChem negatives selected by similarity does not beat the standard candidate sets (11.64 vs 12.29 HR@1 mass; 14.10 vs 14.17 formula).
- MIST's contrastive stage with hard isomer decoys lifts CANOPUS top-1 from 29.72 to 34.45 and top-20 from 73.60 to 85.75, but this conflates "contrastive objective" with "hard decoys" (no random-negative arm). Net reading: negatives drawn from the actual retrieval candidate set are what matter (a listwise softmax over the candidates you will be ranked on), and mining harder negatives beyond that is not shown to help.
Closest published objective to MRR@25. Candidate-set softmax cross-entropy (MSAlign's L_cand) is, with a single relevant item, exactly a ListNet top-one-probability loss over the list we will submit, and its value upper-bounds 1 − R@1. No paper I found optimises reciprocal rank or a smoothed MRR directly (ListMLE/LambdaRank-style), nor an @25-truncated surrogate — that is an open slot. The MERT analogue (tune a log-linear fusion of orthogonal scorers on the evaluation metric, Och 2003: on the dev corpus MMI-trained weights give 12.2 BLEU vs 19.6 when trained on BLEU) is already paying off: MSAlign's temperature-selected late fusion of four scorers lifts R@1 from 47.5 to 58.2.
5. Minimum Bayes-risk decoding and n-best reranking: the mass-spec analogue (Question 6)
- Kumar & Byrne (HLT-NAACL 2004): rescoring 1000-best lists on the NIST 2001+2002 Chinese–English test set, MAP 31.2 BLEU → MBR under the BLEU loss 31.5 (70 % CI ± 0.3); each loss function wins under its own metric ("matched conditions"); the MAP decoder "is not optimal in any of the cases".
- Och (ACL 2003): minimum error rate training of log-linear feature weights on n-best lists; dev-corpus BLEU 12.2 (MMI) → 19.6 (trained on BLEU), 15.4 (trained on mWER), 18.9 (trained on NIST).
- Shen, Sarkar & Och (HLT-NAACL 2004): perceptron-style reranking of 1000-best lists; baseline 31.6 → 32.6 (their splitting algorithm), 32.9 with ~450 syntactic features (SMT Team 2003); "at least 1000 candidates need to be considered".
Mass-spec analogues that exist in print: DiffMS ("sample 100 molecules for each spectrum, remove invalid or disconnected molecules, and identify the top-k molecules based on frequency") and MS-GPT ("candidates are pooled across queries, grouped by InChIKey-14, and ranked by generation frequency"; B = 100 → 1500 "steadily improves recovery") are MBR with a 0/1 utility on InChIKey14 — the sample mode. DualLGD ranks its 100 samples by likelihood (MAP-style) and does worse at top-1 than MS-GPT on MassSpecGym for a similar top-10. GLMR is the reranking analogue: generate one spectrum-conditioned molecule and re-rank the DB candidates by similarity to it, which the authors credit with 46 %/56 % relative R@1 gains over JESTR. FRIGID ranks refinement candidates by Tanimoto to the MIST fingerprint. No published Tanimoto-expected-utility MBR (choose the sample maximising mean Tanimoto to the other samples) for spectra was found (searched "minimum Bayes risk" + molecule/Tanimoto/mass spectra; only NLP results).
Forced analogy: stylometry (Mosteller–Wallace function words, Burrows's Delta)
Statement. Mosteller & Wallace (1963/1964) attributed the 12 disputed Federalist papers to Madison by Bayesian inference on the rates of ~30 function words ("upon", "whilst", "while", "enough"), chosen because they are "stylistically neutral and less topic-dependent" (summary in Fang et al., arXiv 2503.01869, who find that LLM embeddings still do not beat this). Burrows's Delta (LLC 2002) uses "the relative frequencies of the very common words", z-scored against the corpus, with Delta = mean absolute z-difference (Manhattan distance); Evert et al. (DSH 2017) show the signal "lies in the profile of deviation across the most frequent words rather than in the extent of the deviation or in the deviation of specific words only", and that vector normalisation is "the decisive factor". Map: document → spectrum; author → structure (or structural class); content words → diagnostic high-mass fragment ions (the peaks an analyst reads); function words → the closed class of common neutral losses (H2O, CO, NH3, CH3•, C2H4, CO2, CH2O, …) and their relative frequencies.
Does SIRIUS already behave like a Delta model? Only half. Its ~40 learned common-loss priors plus log-normal loss-mass prior are a generative rate model of losses — the Mosteller–Wallace side of stylometry — used to pick the formula and tree, and MIST feeds pairwise neutral losses into attention. Neither computes a per-candidate loss profile, nor normalises each loss type by its corpus variance, nor compares profiles with a distance. That is the missing Delta step.
One testable experiment (MRR@25). For each query spectrum S and each candidate c in the ±5 ppm COCONUT/PubChem pool:
- Observed profile: within the lexicon of the n = 20–40 most frequent losses (take frequencies from a SIRIUS-style tree or, simpler, from all pairwise peak differences and precursor − peak differences matching a lexicon mass at 10 ppm), count intensity-weighted occurrences f_obs(ℓ).
- Candidate profile: simulate c's spectrum with a forward model trained strictly on the train split (ICEBERG or FraGNNet both emit fragment formulae, so losses are parent − fragment formulae), and compute f_c(ℓ) the same way.
- z-score each loss type against the corpus of train timsTOF spectra (mean, sd of f(ℓ)); Delta(S, c) = (1/n) Σ_ℓ |z_obs(ℓ) − z_c(ℓ)|; also the Cosine-Delta variant.
- Add −Delta as one feature in the reranker and measure MRR@25 against the same reranker without it. Hold-out: a scaffold-disjoint split of train (MCES ≥ 10 as in MassSpecGym, or Murcko-scaffold grouping), restricted to timsTOF rows, with the forward model, z-score statistics and reranker all fitted on the training side only; report also the Delta-only 256-way R@1 against random (0.37 %) and the gain restricted to isomer pairs that share their three most intense fragments (where "content words" cannot separate them). Pass criterion: +0.01 MRR@25 with 99.9 % bootstrap CI excluding zero (MassSpecGym's 20,000-resample convention).
Does the analogy hold? Partially. It holds in that losses are high-frequency, low-salience, and routinely discarded by human readers yet carried by every ranking model implicitly; it holds in that corpus-normalised profiles (Evert) rather than single-loss presence are the right representation; and it holds in the confounder structure — genre/period in stylometry corresponds to instrument/collision energy here (Koehn's domain mismatch), so z-scores must be computed per instrument class (timsTOF subset: 1,154,969 train rows). It breaks in three places: (1) losses are not content-independent — the number of H2O losses tracks the number of hydroxyls, so the "function words" are a compositional fingerprint in disguise and the feature is correlated with Morgan/MACCS bits already in the ranker; (2) losses are tree-dependent (sequential, parallelogram-ambiguous), not bag-of-words independent, so Mosteller–Wallace's independence assumption is violated more badly than for prose; (3) stylometry needs several documents per author to estimate rates, whereas each structure is a one-document author — the only way to get an "author corpus" is to attribute at class level (NPClassifier/ClassyFire), which is CANOPUS's job and does not pick the isomer. Expect a small, orthogonal gain, concentrated on isomer disambiguation; the analogy is worth one feature, not a model.
Gotchas / what could fool us
- Hit rate is conditional on presence. Every retrieval number (47.5 %, 68.5 %, 84.7 %) assumes the true molecule is in the ≤ 256-candidate set. CASMI's back-solved recall@25 ≈ 0.55 means MRR@25 ≈ P(in pool) × conditional MRR; a ranker gain of +10 points conditional is worth ≈ +0.05 unconditional.
- Formula-given de novo numbers. MS-GPT/DualLGD ~24 % assume the true formula; DiffMS loses 0.44 points top-1 with MIST-CF formulae; the formula step on MassSpecGym is itself a bottleneck (MIST-CF on MSG, see sibling q5 file: 48 % top-1).
- Worse than random is possible. A SMILES decoder trained with token CE scores MCES 53.80 vs 28.59 for random generation. Any seq2seq filler must be checked against the random-valid-molecule baseline, not just against zero.
- Benchmark leakage is the norm, not the exception. ≥ 17/26 audited papers had issues; ICEBERG trained on test molecules inflates retrieval to 67.75 % (53.76 % leak-free; MIST baseline 9.57 %); the original MIST padding-mask bug inflated MIST+MolForge from 10.73 % to 31.75 % top-1; a random baseline with the true formula scores 12.98 % vs 0.43 % without. SpecBridge (84.7 %) and GLMR (68.5 %) post-date the audit's corrected table in part and are unaudited; SpecBridge embeds candidate SMILES with ChemBERTa, the same family whose classifier collapsed 99.66 → 8.2 % under uniform canonicalisation — unverified whether SpecBridge is affected, but treat 84.7 % as unconfirmed.
- In-batch InfoNCE loss is decorrelated from retrieval accuracy (MSAlign A.1: a mass-only scorer reaches loss < 10⁻² with 0 % top-1). A falling contrastive loss curve proves nothing about MRR unless the negatives are the candidates.
- Fingerprint accuracy and retrieval trade off (De Waele: IoU loss best Tanimoto 19.42 but HR@1 5.58; contrastive 12.08). Do not select a fingerprint model on bit-Tanimoto.
- Duplicate fingerprints in candidate sets (σ_max = 1 cases) create unbreakable ties; MRR tie order then depends on candidate enumeration order. Dedupe by InChIKey14 after RDKit 2026.03.3 tautomer canonicalisation and add a tie-breaker (mass error, NP-likeness).
- Tautomers as false negatives. Candidate-set softmax treats every non-matching candidate as negative; tautomers of the true structure in PubChem (identical InChIKey14 after canonicalisation) become label noise unless collapsed before training.
- Regularisation gains are dataset-specific (JESTR +4.7 on NPLIB1, −0.5 on MassSpecGym); NPLIB1 is near-duplicate-rich. Quote MassSpecGym or your own scaffold-disjoint split, never NPLIB1, for novel NPs.
- Tool comparisons are contaminated by training overlap: SIRIUS 19.04 % → 10.41 % on MassSpecGym after removing spectra whose molecules were in its training set.
- Sample-frequency ranking has coarse resolution. With 100 samples most candidates appear once; slots 2–25 are effectively unordered unless re-scored by an independent scorer (forward model, fingerprint cosine).
- Temperature is a hyper-parameter of the metric, not just the loss (HR@1 11.3–12.6 across τ); report it and retune after any change of candidate-set size K.
- Instrument/CE is the "genre" confounder: z-score baselines, forward models and negatives must be computed on timsTOF-like data or the Delta/consensus features will measure instrument, not structure.
- Stereo tokens are wasted capacity: InChIKey14 ignores stereo; train and emit canonical non-isomeric SMILES.
Implications for the challenge
- Train the ranker with candidate-set softmax cross-entropy over the actual ±ppm pool (K = 128–256 COCONUT/PubChem candidates per spectrum, deduped by InChIKey14 after tautomer canonicalisation), not in-batch InfoNCE: the one clean ablation shows 19.5 → 47.5 % R@1.
- Do not spend time mining harder-than-pool negatives (no gain in De Waele; none on MSG in JESTR); spend it on making the training pool look like the test pool (NP-biased, timsTOF, same mass window).
- Add an MRR-shaped fine-tune: after candidate CE, fine-tune on a smoothed reciprocal-rank surrogate truncated at 25 (untested in the literature — measure it, it is the one objective that matches the metric exactly).
- MERT the fusion: tune the weights of orthogonal scorers (fingerprint cosine, contrastive embedding, forward-model spectral similarity, Delta profile, mass error) directly on dev MRR@25 (MSAlign fusion: 47.5 → 58.2).
- Use de novo only as MBR consensus filler: pool samples across all spectra of a molecule_id and across generators, group by InChIKey14, rank by frequency, break ties with a leak-free forward model; put these behind the DB candidates except where the pool is empty. Try Tanimoto-expected-utility selection as a variant (no published numbers — measure).
- GLMR-style anchor reranking: Tanimoto of each DB candidate to the top consensus de novo molecule as a reranker feature.
- Run the stylometry experiment of §Forced analogy as one feature; keep it only if the CI excludes zero on the scaffold-disjoint timsTOF hold-out.
- Validation protocol: MCES ≥ 10 or Murcko split of train, timsTOF-only queries, remove the true structure from the pool for ~45 % of hold-out molecules, report conditional and unconditional MRR@25 with 20,000-resample bootstrap CIs; pin RDKit 2026.03.3 and InChIKey14 matching.
- Never let a seq2seq SMILES model rank above DB candidates without a forward-model consistency check; its failure mode is fluent-but-wrong, invisible to validity filters.
- Use retrieval-level uncertainty (score gap / total uncertainty over the candidate ranking), not fingerprint-bit uncertainty, to decide how many of the 25 slots go to de novo candidates per molecule.
Open questions
- Does a smoothed-MRR or ListMLE fine-tune beat candidate softmax CE on MassSpecGym formula split? No paper tests it.
- Do SpecBridge's 84.7 % and GLMR's 68.5 % survive the audit's uniform-canonicalisation protocol and the v1.5 candidate sets?
- Where is the mode of a spectrum→SMILES decoder under exact inference — is there an "empty string" analogue (methane-sized molecules)? Untested.
- Does Tanimoto-expected-utility MBR over 100–1500 samples beat frequency ranking? No published numbers.
- Is the Delta profile of common losses informative beyond Morgan bits for isomer pairs sharing diagnostic fragments? Only the experiment above can say.
- MS2Mol's exact-match rate on EnvedaDark (PDF not reachable: ChemRxiv 403).
Sources
- Bushuiev et al., MassSpecGym, NeurIPS 2024 — https://arxiv.org/abs/2410.23326 (HTML v1 and PDF read; Tables 2–3)
- Liu, Bushuiev et al., "MassSpecGym in the Wild", arXiv 2606.19624 (abs + HTML v1)
- Bohde et al., DiffMS, ICML 2025 — https://arxiv.org/abs/2502.09571 (HTML + PDF; Tables 2–4)
- Wang, Chen, Liu, Hassoun, MADGEN, ICLR 2025 — https://arxiv.org/html/2501.01950v3
- Litsa et al., Spec2Mol, Commun Chem 2023 — https://pmc.ncbi.nlm.nih.gov/articles/PMC10290119
- MS2Mol: ChemRxiv landing pages 403; numbers from search snippets of the ChemRxiv abstract and BioSpace press release (2023-06-21) https://www.biospace.com/enveda-biosciences-unveils-new-ai-algorithm-to-unlock-nature-s-chemical-code
- Zhao et al., MS-GPT — https://arxiv.org/html/2607.23607v1
- Che, Du, Xu, DualLGD — https://arxiv.org/html/2605.07048
- Bohde et al., FRIGID — https://arxiv.org/html/2604.16648v1
- Goldman et al., MIST, Nat Mach Intell 2023, doi:10.1038/s42256-023-00708-3 — Supplementary PDF (Tables S4, S8) https://media.springernature.com/original/springer-static/esm/art%3A10.1038%2Fs42256-023-00708-3/MediaObjects/42256_2023_708_MOESM1_ESM.pdf ; repo README https://raw.githubusercontent.com/samgoldman97/mist/master/README.md
- Kalia, Chen, Krishnan, Hassoun, JESTR — https://arxiv.org/pdf/2411.14464
- Krzakala et al., MSAlign — https://arxiv.org/html/2605.19752
- De Waele et al., "what are we optimizing for?" — https://arxiv.org/html/2602.16507
- Zhang et al., GLMR, AAAI 2026 — https://arxiv.org/html/2511.06259
- Wang et al., SpecBridge — https://arxiv.org/html/2601.17204v1
- FLARE — https://www.biorxiv.org/content/10.64898/2026.01.27.702086v1.full
- Chen et al., CMSSP, Anal Chem 2024 — https://cib.cas.cn/kycg/lw/202410/t20241016_7400732.html (doi 10.1021/acs.analchem.4c03724)
- Yoo et al., contrastive domain generalization — https://arxiv.org/abs/2602.00547 ; Tsai et al., MS2MetGAN — https://arxiv.org/abs/2603.13342 ; Jürgens et al., selective prediction — https://arxiv.org/abs/2603.10950
- Böcker & Rasche 2008, Bioinformatics 24(16):i49 (abstract via Europe PMC REST, doi 10.1093/bioinformatics/btn270); Böcker & Dührkop 2016, "Fragmentation trees reloaded", J Cheminform 8:5 — https://pmc.ncbi.nlm.nih.gov/articles/PMC4736045/ ; SIRIUS docs https://v6.docs.sirius-ms.io/methods-background/ (SIRIUS 4 Nat Methods full text not reachable: idp redirect / Jena preprint 503)
- Goldman et al., ICEBERG — https://arxiv.org/html/2304.13136v2 ; SCARF — https://arxiv.org/abs/2303.06470 ; Young et al., FraGNNet — https://arxiv.org/html/2404.02360v2
- Callison-Burch, Osborne, Koehn 2006 — https://aclanthology.org/E06-1032.pdf ; Koehn & Knowles 2017 — https://ar5iv.labs.arxiv.org/html/1706.03872 ; Stahlberg & Byrne 2019 — https://ar5iv.labs.arxiv.org/html/1908.10090 ; Ranzato et al. 2016 — https://arxiv.org/abs/1511.06732 ; Müller, Kornblith, Hinton 2019 — https://arxiv.org/abs/1906.02629
- Kumar & Byrne 2004 — https://aclanthology.org/N04-1022.pdf ; Och 2003 — https://aclanthology.org/P03-1021.pdf ; Shen, Sarkar, Och 2004 — https://aclanthology.org/N04-1023.pdf
- Burrows 2002, LLC 17(3):267 — https://academic.oup.com/dsh/article/17/3/267/929277 (abstract) ; Evert et al. 2017, DSH 32(suppl 2):ii4 — https://academic.oup.com/dsh/article/32/suppl_2/ii4/3865676 (abstract) ; Fang et al., "From Small to Large Language Models: Revisiting the Federalist Papers" — https://arxiv.org/html/2503.01869 (Mosteller–Wallace summary; the 1963 JASA paper itself was 403)
- Sibling worker file (cross-checked, not re-verified):
research/q5_methods_lineage/denovo_generation_ms2mol_diffms_madgen_spec2mol_msnovelist.md