Analog retrieval by spectral similarity — cosine, modified cosine, spectral entropy, Spec2Vec, MS2DeepScore 1.0/2.0, MS2Query, DreaMS and MassSpecGym numbers, what Tanimoto an analog hit really delivers, how to turn a hit into candidate SMILES, and the compute budget for a 2.5 M-spectrum library

TL;DR. Exact-match similarity (cosine, spectral entropy) is a very narrow sensor: entropy similarity reaches <10 % false-discovery rate at a 0.75 cutoff for identical compounds, but ~30 % of its false positives are one-bond isomers and it says nothing about Tanimoto. For true analog retrieval the only like-for-like published numbers come from MS2Query's 302,514-spectrum GNPS benchmark: at 35 % recall the best modified-cosine analog averages Tanimoto 0.45, MS2Query (Spec2Vec + neighbourhood-averaged MS2DeepScore + mass features) averages 0.63, and Spec2Vec finds a top-10 hit with Tanimoto > 0.6 for ~60 % of unknown queries. The hidden test (~90 % novel molecules) will therefore mostly yield analogs at Tanimoto 0.4–0.7, which score zero under MRR@25 unless they are converted into candidate structures; the cheapest published converters are single-modification enumeration (ModiFinder-style, 8,033 of 12,909 single-modification pairs even have shifted peaks) and mass-window-filtered structural neighbourhoods of the analog, and no paper we could open reports a class-filtered (CANOPUS) re-ranking gain.

1. What the similarity measures are and what each was actually benchmarked on

Two negative findings are worth stating plainly: (i) no paper we could open reports the mean Tanimoto of the top-1 hit for plain cosine or spectral entropy on an analog task — the Li & Fiehn work is exact-match only, and Spec2Vec's cosine/modified-cosine comparison for unknown compounds is relegated to "Fig B in S4 Text" with only the qualitative statement that the high false-positive fraction "results in less accurate compound suggestions"; (ii) MassSpecGym evaluated no spectral-library-similarity baseline at all (only Random, Fingerprint FFN, DeepSets, DeepSets+Fourier, MIST), so there is no published MRR-like figure for "nearest-library-analog → candidate" on an MCES-split benchmark.

2. What Tanimoto an analog hit actually delivers

• Chance level. In the Spec2Vec AllPositive set the random probability of a pair having Tanimoto > 0.6 is 0.0103 and > 0.7 is 0.0034 (Huber 2021). Any method's "fraction of top pairs above 0.6" must be read against ~1 %.
• Modified cosine. At a recall of 35 % (i.e. the 35 % of analog queries that pass the score threshold), mean Tanimoto of the chosen analog is 0.45 (MS2Query benchmark). That is well above chance but means the median modified-cosine "analog" shares roughly half its Morgan bits with the truth — not a scaffold-preserving neighbour in most cases.
• Embedding / learned measures. MS2Query 0.63 at the same recall; Spec2Vec ~60 % of queries with a Tanimoto > 0.6 hit somewhere in the top 10 (not top-1), rising to mean > 0.8 for > 400 Da molecules — relevant because natural products skew heavy. MS2DeepScore's RMSE ≈ 0.15 on the Tanimoto scale is the noise floor of the sensor: a predicted 0.7 is a true 0.55–0.85.
• What "cosine ≥ 0.7 edge implies" in GNPS terms. Wang 2016 gives 35.2 % of unidentified nodes linked to related molecules at cosine 0.8, but related there is a curation judgement, not a Tanimoto threshold; Bittremieux 2022 quantified it on 10 million pairs and found modified cosine best but modification-type dependent. The precise precision-vs-mass-shift curves are in the body of that paper, which we could not open (bioRxiv rate-limited; ACS 403) — unverified.
• Known false-analog modes. (a) Entropy/cosine high-score false positives: "About 30 % of the false positives were one-bond isomers" (Li 2021) — these are excellent candidates for an MRR task even though they are wrong identities. (b) COSMIC: of nine high-confidence incorrect CSI:FingerID hits, "the true structure was highly similar to the corresponding hit" in all nine; three verified pairs had cosine 0.85–1.00 between their spectra. (c) ModiFinder: of 12,909 single-modification library pairs, only 8,033 (62 %) have any shifted peaks at all — i.e. for ~38 % of genuine one-modification analogs the modified-cosine δ-shift term contributes nothing, and the pair looks like a plain-cosine match or no match.

3. Mechanics

Modified cosine. GNPS: "Additional edges are added to the bipartite matching between library and query peaks which differ by a δ (as determined by their precursor mass difference δ) +/− the user defined fragment mass tolerance." Peaks are therefore matched either unshifted or shifted by exactly δ; a modification that changes which bonds break (not just adds a mass) produces neither kind of match. Bittremieux's abstract states the consequence: performance "depends on the location and type of the modification". ModiFinder's benchmark constraints show the practical operating window: precursor < 2,000 Da, Δ precursor < 50 % of the smaller precursor, same [M+H]+ adduct, exactly one modification site. MS2Query capped the analog window at 100 Da. Beyond that, the shifted term starts matching chance pairs because many m/z pairs differ by δ by accident (the "large mass difference" failure mode); Spec2Vec used min_match = 10 and tolerance 0.005 Da for modified cosine precisely to suppress this. matchms default tolerance/intensity power values were not retrievable this session — unverified.

Spectral entropy similarity (Li et al. 2021). Preprocessing: centroid, remove "low-abundance ions at < 1 % base peak intensity", 0.05 Da matching tolerance, precursor removal tested both ways. Weighting: with S the spectral entropy, I_new = I^w with w = 1 if S ≥ 3 and w = 0.25 + 0.25·S if S < 3. Low-entropy (few dominant peaks) spectra are flattened; merged multi-CE spectra have higher entropy and therefore sit closer to the unweighted regime. Flash entropy search (ms_entropy 1.5.3, PyPI upload 2024-12-19, Python ≥ 3.7) exposes identity, open (any precursor), neutral-loss (peaks re-expressed as precursor − fragment) and hybrid (fragment OR neutral-loss match) modes; hybrid is the entropy analogue of modified cosine. low_memory modes 0/1/2 (2 = numpy.memmap index), GPU via CuPy "benefits searches exceeding ~0.1 seconds", and the docs warn 1 M × 1 M all-vs-all needs ~4 TB if stored dense.

4. Analog hit → candidate structures: what is actually published

Nothing we opened reports "scaffold + substituent enumeration from an analog hit" as an evaluated pipeline with MRR or hit-rate numbers — that remains our experiment to run.

5. Practical throughput for 2.5 M library × ~1,500 queries

Kaggle offline notebook budget (9–12 h per the brief, unverified): feasible for Flash entropy and embedding methods; feasible for modified cosine only after structure-level de-duplication and precursor-window indexing (matchms 0.33.1, PyPI 2026-06-08, now stores sparse score arrays for "several hundred thousands of spectra"). matchms and ms_entropy are pure pip packages, so they can be shipped as wheels in a Kaggle dataset.

6. Does merging collision energies help or hurt similarity?

Evidence points both ways, and the direction depends on what the library looks like:

• Helps for learned embeddings. Giné et al. 2026 (JASMS 10.1021/jasms.5c00428, 71,561 compounds from NIST2023/METLIN/GNPS2/MS-DIAL) merged "spectra acquired at multiple collision energies into a single consolidated spectrum" and report that merged spectra "yielding predicted embeddings that more closely aligned with ground-truth Mol2Vec embeddings and improved compound ranking performance" versus individual-CE spectra.
• Standard practice for CSI:FingerID. COSMIC built "merged spectra … by combining these three spectra (pseudo-ramp spectra)" from 10/20/40 eV.
• Hurts for exact-match library scoring against single-CE references. Fu et al. 2025 (Anal Chem 10.1021/acs.analchem.5c01414; Orbitrap Exploris 120/480; ~180,000 spectra of > 4,500 metabolites; NCE 1–100 % plus stepped 20/40/60 %) found stepped-CE spectra scored cross-sectionally (entropy) reached average sensitivity 92.25 % / specificity 77.38 % at a 0.95 threshold, versus 98.89 % / 92.66 % for their multi-CE "longitudinal fragment profile" method, which improved identification by 25.9 % (C+), 12.6 % (C−), 8.7 % (A+), 16.9 % (A−) over single-spectrum entropy scoring.
• Benchmarks keep CEs separate. MassSpecGym keeps spectra at different collision energies as distinct entries and stratifies splits by instrument, CE and adduct; DreaMS's merging was across adducts, not CEs.

Net: the hidden test's merged multi-CE timsTOF spectra will match single-CE library entries worse under cosine/entropy than under embeddings, and the right counter-move is to merge the library per structure (sum or max intensities across CEs, then re-normalise and re-apply the 1 % noise cut) so both sides are pseudo-ramp.

Forced analogy: search-and-rescue sweep width

The analogy. In Koopman's search theory a sensor has a lateral-range curve P(detect | x), x = lateral distance, and sweep width W = ∫ P(x) dx. Map x → structural distance d = 1 − Tanimoto(true, retrieved library molecule). A similarity measure "detects" when the true molecule's nearest library relative at distance d is ranked (or scored) above a threshold that we would act on. Published numbers give a few points per curve:

These are sketches assembled from different benchmarks with different fingerprints; the shape, not the values, is the claim.

Orthogonality claim (cited). Cosine and entropy identity searches sweep the same strip (both are peak-overlap measures; entropy only re-weights intensities and the DreaMS 10-ppm benchmark ranks entropy as the best of the 44 conventional measures, which puts them on the same curve). MS2DeepScore and DreaMS likewise sweep the same broad strip (both regress Tanimoto/MCES; DreaMS beats MS2DeepScore on correlation but it is the same quantity). The most orthogonal pair is therefore modified cosine (or Flash hybrid search) × a Tanimoto-regression embedding (MS2DeepScore 2.0 or DreaMS): the first detects conserved-fragment, single-site modifications and is blind when fragmentation changes (Bittremieux: depends on modification location/type; ModiFinder: 38 % of single-modification pairs lack shifted peaks); the second detects global structural similarity even when no peaks align (rutin: MS2DeepScore 2.0 0.7 vs modified cosine 0.3 vs cosine 0.0) but cannot resolve isomers (RMSE 0.15 ≫ one-bond distance). The published evidence that the union widens the sweep is MS2Query's own ablation: starting from a modified-cosine search at 35 % recall (Tanimoto 0.45), the feature with "the biggest impact on the increased performance is the Average MS2Deepscore of multiple library molecules", lifting mean Tanimoto to 0.63 at the same recall.

Testable idea for MRR@25. Build a two-sensor candidate expander: for each test molecule, (1) run Flash hybrid search and MS2DeepScore 2.0/DreaMS nearest neighbours over the structure-level de-duplicated train library (≤ 275,810 merged pseudo-ramp spectra); (2) for the top-k analogs from each sensor, generate candidates as (a) all COCONUT/train structures within Tanimoto ≥ 0.5 of the analog and within 10 ppm of the query neutral mass, plus (b) single-site modifications of the analog whose mass change equals Δm (ModiFinder-style, when |Δm| < 50 % of the smaller precursor); (3) rank the union by the fingerprint model. Measure recall@25 on a held-out set of train structures with all their spectra removed (MS2Query-style split, or harsher MCES ≥ 10 as MassSpecGym). The prediction from the numbers above: analog seeds at Tanimoto ≈ 0.6 shrink the 10-ppm pool to a few dozen structures, which is where the current 0.55 recall@25 ceiling has room to move.

Does it hold? Partially. It holds for the lateral-range picture and for choosing orthogonal sensors. It breaks in two places: Koopman assumes a single target, whereas here several near-isomers sit inside every sweep and the metric pays only for the exact one — so a "detection" at d ≈ 0.1 is not a success but a candidate-generation event; and sweep width in Koopman is additive over independent sensors, whereas MS2DeepScore and DreaMS are trained on overlapping GNPS/NIST data, so their errors are correlated and their combined sweep is less than the sum.

Gotchas / what could fool us

1. The visible test is train. Any library-search score on test.parquet is leakage; the brief's 0.06–0.10 hidden-test figure is the only honest baseline.
2. Benchmarks are kinder than the hidden test. MS2Query/Spec2Vec/MS2DeepScore split on unique 2D structures but keep close analogs in the library; MassSpecGym's MCES ≥ 10 split drops MIST to 14.6 % hit@1. "None previously published" natural products are likely nearer the MassSpecGym regime.
3. Tanimoto is not one number. Papers use different Morgan radii/bit lengths (not retrievable for every paper this session); cross-paper comparison of 0.45 vs 0.63 is only valid within MS2Query, where both were computed identically.
4. Adduct errors poison δ. Modified cosine shifts by the precursor difference; with 7 adduct types in the test, a mis-assigned adduct makes δ wrong by 17.03/21.98/… Da and silently produces chance-level shifted matches.
5. Merged-CE query vs single-CE library. Fu 2025 shows stepped-CE spectra scored against references lose specificity (77 % vs 93 %); the entropy weight w = 0.25 + 0.25·S also changes regime when merging raises S above 3.
6. The 1 % noise cut is applied per spectrum. Merging before or after the cut changes peak counts and therefore min-match gates (6 or 10 peaks).
7. Instrument domain shift. Spec2Vec/MS2DeepScore were trained on GNPS (Orbitrap/QTOF dominated; MassSpecGym says 98 % Orbitrap or QTOF); timsTOF is a QTOF with ion mobility, so expect closer-to-QTOF behaviour but no published timsTOF-specific number.
8. Uncertainty filtering throws away most of the data. MS2DeepScore's 0.11 RMSE applies after discarding 75 % of pairs; the 0.15 figure is what a full-library search sees.
9. Library redundancy inflates compute and ranks. 2.5 M spectra / 275,810 structures ≈ 9 spectra per structure; without structure-level aggregation, the top-25 analog list fills with copies of one molecule.
10. Metric blind spot. An analog with Tanimoto 0.9 scores exactly 0 under MRR@25; only the converter step turns retrieval quality into score.
11. PubChem-scale pools hurt. COSMIC saw "a substantial drop of correct annotations for small FDR values" on 77 M PubChem vs 391,855 biomolecules, matching the Kaggle lesson that PubChem-90 M pools hurt.

Implications for the challenge

1. Build the library search at the structure level: merge all train spectra per InChIKey14 (and per adduct class) into pseudo-ramp spectra, re-normalise, re-apply the 1 % noise threshold; this both matches the merged-CE test format and cuts search cost ~9×.
2. Use Flash entropy hybrid search (ms_entropy 1.5.3) as the fast peak-alignment sensor and MS2DeepScore 2.0 (or DreaMS embeddings) as the Tanimoto-regression sensor; skip plain cosine and skip Spec2Vec unless time allows (same strip as entropy / MS2DeepScore respectively).
3. Treat every analog hit as a seed, not an answer: expand to the 10-ppm-consistent Tanimoto ≥ 0.5 neighbourhood in COCONUT + train structures, plus single-site Δm modifications when |Δm| < 50 % of the smaller precursor.
4. For exact-match hits (identity score ≥ 0.75 entropy, ≥ 0.7 cosine with ≥ 6 peaks) also enumerate one-bond isomers of the hit into slots 2–25: 30 % of high-score false positives are such isomers.
5. Gate the analog converter at a score where the published mean Tanimoto is ≥ 0.6 (MS2Query threshold 0.633 gave 35 % recall); below it the neighbourhood is too diffuse and wastes slots.
6. Validate on a split that removes all spectra of held-out structures and, as a stress test, an MCES ≥ 10 split; report recall@25 of the candidate pool separately from MRR so the ceiling is visible.
7. Budget: embeddings for 275,810 merged library spectra once (minutes on GPU), hybrid entropy index once, then ~1,500 queries — comfortably inside a 9–12 h offline notebook; avoid matchms ModifiedCosine over the raw 2.5 M rows (≈ 19 h single-process by MS2Query's rate).
8. Use CANOPUS/NPClassifier class predictions only as a soft multiplicative prior on candidates (sparse-class MCC ≈ 0.53); no paper shows a hard class gate helping.

Open questions

• The numeric lateral-range curves for DreaMS (AUROC vs MCES k = 0…7, Ext. Data Fig. 5) and MS2DeepScore 1.0's precision at Tanimoto > 0.6 by score percentile are in figures we could not read; a figure-level read would sharpen the sweep sketch.
• Bittremieux 2022's per-modification-type and per-mass-shift precision numbers (body of the paper) are the missing quantitative basis for "modified cosine fails beyond Δm = X".
• Flash entropy hybrid-search accuracy on analogs (vs modified cosine) is not in the abstract; the paper body or an in-house benchmark is needed.
• Whether matchms 0.33.1's default ModifiedCosine tolerance is still 0.1 Da and whether its Hungarian variant survives — unverified.
• Kaggle notebook wall-clock and RAM limits remain unverified (brief).
• No timsTOF-specific analog-retrieval benchmark exists in what we opened; the Fu 2025 inter-instrument validation is Orbitrap-to-Orbitrap only.

Sources

Opened this session (web search budget was exhausted early; everything below was fetched directly or via the Europe PMC REST API):

• Li Y, Kind T, Folz J, Vaniya A, Mehta SS, Fiehn O. Spectral entropy outperforms MS/MS dot product similarity for small-molecule compound identification. Nat Methods 2021. DOI 10.1038/s41592-021-01331-z; full text via PMC11492813.
• Li Y, Fiehn O. Flash entropy search to query all mass spectral libraries in real time. Nat Methods 2023. DOI 10.1038/s41592-023-02012-9 (abstract via Europe PMC REST; full text not reachable this session).
• MSEntropy README, https://github.com/YuanyueLi/MSEntropy ; docs https://msentropy.readthedocs.io/en/latest/entropy_search_advanced_usage.html ; PyPI metadata https://pypi.org/pypi/ms-entropy/json (1.5.3).
• Huber F et al. Spec2Vec. PLoS Comput Biol 2021. DOI 10.1371/journal.pcbi.1008724.
• Huber F, van der Burg S, van der Hooft JJJ, Ridder L. MS2DeepScore. J Cheminform 2021. DOI 10.1186/s13321-021-00558-4; PMC8556919.
• de Jonge NF et al. Cross ionization mode chemical similarity prediction between tandem mass spectra (MS2DeepScore 2.0). Nat Commun 2026. DOI 10.1038/s41467-026-69083-y; PMC12992594. Preprint DOI 10.1101/2024.03.25.586580 (listed, not opened).
• de Jonge NF et al. MS2Query. Nat Commun 2023. DOI 10.1038/s41467-023-37446-4; PMC10060387.
• Bushuiev R et al. DreaMS. Nat Biotechnol 2026. DOI 10.1038/s41587-025-02663-3; PMC13090125.
• Bushuiev R et al. MassSpecGym. arXiv 2410.23326 (HTML version).
• Wang M et al. GNPS. Nat Biotechnol 2016. PMC5321674.
• Bittremieux W et al. Comparison of cosine, modified cosine, and neutral loss based spectrum alignment. J Am Soc Mass Spectrom 2022. DOI 10.1021/jasms.2c00153 (abstract via Europe PMC; bioRxiv 10.1101/2022.06.01.494370 rate-limited).
• Shahneh MRZ et al. ModiFinder. bioRxiv 10.1101/2024.02.17.580849; JASMS 2024 DOI 10.1021/jasms.4c00061.
• Dührkop K et al. CANOPUS preprint. bioRxiv 10.1101/2020.04.17.046672 (journal version Nat Biotechnol 2021 DOI 10.1038/s41587-020-0740-8).
• Hoffmann MA et al. COSMIC. Nat Biotechnol 2022. DOI 10.1038/s41587-021-01045-9; PMC8926923.
• Fu X et al. Longitudinal fragment profiles based on multi-collision energy tandem mass spectra. Anal Chem 2025. DOI 10.1021/acs.analchem.5c01414; PMC12330898.
• Giné R et al. Benchmarking MS/MS featurization strategies for machine learning. JASMS 2026. DOI 10.1021/jasms.5c00428; PMC13329996.
• SIRIUS 6 documentation, Methods and algorithms: https://v6.docs.sirius-ms.io/methods-background/
• matchms README https://github.com/matchms/matchms ; PyPI https://pypi.org/project/matchms/ (0.33.1, 2026-06-08).