Ranking the true answer among 25 look-alikes: calibration and re-ranking tricks from differential diagnosis, transient astronomy, eDNA taxonomy and open-set recognition, applied to the CASMI 2026 open-set problem

expertconfidence mediumrelevance 4/510 sources7.4k words
Forced analogyForced lens was art connoisseurship (Rembrandt Research Project). It paid off as an evidence-fusion design: stylistic verdicts reversed when orthogonal physical measurements were added, which maps cleanly onto learned-similarity scores versus hard mass/formula/adduct constraints, and yields a concrete A/B/C triage. The Morelli sub-test also paid off: Li et al. 2021 state outright that look-alike isomers share the base peak and differ in low-abundance ions.

TL;DR. Four unrelated fields solve "rank the truth inside a short candidate list", and each has a published trick worth stealing: a context prior (host-galaxy features alone separate supernova types at ~70 % accuracy; a biological-source prior lifts rank-1 natural-product annotations from 10.2 % to ~72 %), a hubness correction (CSLS adds +2.1 to +6.5 precision@1 points on word translation), bootstrap-over-features confidence (IDTAXA mis-assigns 0.01 % of sequences whose genus is missing from the reference, against 17.3 % for a BLAST top hit), and an explicit "none of the above" training class (BirdCLEF 2024 winner: private score 0.655 → 0.671). The Morelli hypothesis survives contact with the MS/MS literature: every optimised intensity exponent published since 1994 is below 1 (0.25–0.6), and Li et al. show why — ADP and ATP share the adenine base peak and differ in the small ions. The silent killer for us is the opposite lesson: the single largest published re-ranking gain in metabolite ID (MetFrag, ~6 % → 89 % top-1) comes from literature-reference counts, a prior that is actively inverted on a test set defined as never previously published.

1. Five systems that rank a true answer among look-alikes

Field / system Published numbers The one transferable trick (and how to test it on held-out NPs) Open-set handling
Symptom checkers — 23 engines, 45 vignettes (Semigran 2015 BMJ 351:h3480) Correct diagnosis listed first 34 % (95 % CI 31–37), top-3 51 %, top-20 58 %; per-engine rank-1 spread 5 % (MEDoctor) to 50 % (DocResponse); triage appropriate 57 % overall — 80 % emergent, 55 % non-emergent, 33 % self-care, i.e. systematically risk-averse Accept precision loss to buy list membership. Going from 3 to 20 slots bought only 7 points of hit rate: the list filled with near-duplicates. Test: build the 25 by raw score, versus score + maximal-marginal-relevance on Bemis–Murcko scaffold (penalise a candidate by its max Tanimoto to already-selected ones); compare recall@25 and MRR@25 on a scaffold-disjoint hold-out None — closed world
GPT-4 differential — 70 NEJM CPCs (Kanjee 2023 JAMA 10.1001/jama.2023.8288); legacy generators in Bond 2012 JGIM Top-1 39 % (27/70); correct anywhere in the list 64 % (45/70); mean list 9.0 items; mean quality 4.2/5 (SD 1.3); correct answers sat at mean rank 2.5. Bond 2012 five-point means: Isabel 3.45, DXplain 3.45, Diagnosis Pro 2.65, PEPID 1.70 Hard pre-filter before scoring. Isabel restricts by age, sex and region before ranking and flags "don't-miss" conditions (vendor documentation; its "96 % top-10" is a vendor claim — unverified). Test: our pre-filter is the physical one — adduct-consistent neutral mass at ≤ 5 ppm across all of a molecule's 1–16 spectra, not just one; measure how much pool shrinkage that joint constraint buys over single-spectrum filtering None
Transient astronomy — GHOST (Gagliano 2021 ApJ 908:170), Foley & Mandel 2013 ApJ 778, RAPID (Muthukrishna 2019), Sherlock in ATLAS/Lasair (Smith 2020) GHOST: 16,175 classified supernovae; host-galaxy features plus radial offset alone give ~70 % Ia vs core-collapse accuracy with no photometry of the event. Foley & Mandel: host-only classification improves the figure of merit by >2×, morphology most informative (Ia fraction ~65–100 % in E/S0 hosts, ≲20 % in Sbc/Sb/Scd/Irr). RAPID: 12 classes, AUC 0.95 early / 0.98 late, host redshift supplied as an extra input row — no redshift ablation is reported, and it cites earlier work finding redshift did not help. Sherlock: 7 contextual classes, 1.5″ synonym radius, galaxy association rejected beyond 2.4× the semi-major axis or 50 kpc Deterministic context rules first, learned score second. Sherlock decides supernova versus nuclear transient by geometry before anything is learned. Test: add a per-molecule context channel (set of adducts observed, collision-energy span, number of spectra, precursor error spread) to the ranker and ablate it; the astronomy claim is that context alone is worth a large fraction of the answer Explicit "orphan" class for no cross-match; back off to a coarser class
eDNA / 16S taxonomy — RDP (Wang 2007), IDTAXA (Murali 2018 Microbiome 6:140), SINTAX (Edgar 2016) RDP closed-set: ≥ 89 % genus accuracy on near-full-length leave-one-out; 97.5 % of assignments carried ≥ 95 % bootstrap and 98 % of those were right. RDP open-set is the warning: in a mock community with the expected genera removed from training, the fraction still forced into a genus was IDTAXA 0.01 %, RDP 3.83 %, SINTAX 8.76 %, QIIME 13.5 %, BLAST top-hit 17.3 %, MAPSeq 24.6 %, SPINGO 26.7 %; spurious genera invented: IDTAXA 2, SINTAX 9, RDP 12, SPINGO 15, MAPSeq 20, BLAST 65. The coverage price: IDTAXA assigned 19.9 % of gut reads to genus versus RDP's 22.5 % Bootstrap over features. RDP resamples 1/8 of its 8-mer words 100× and calls the fraction of resamples agreeing on the top label its confidence. Test: resample 1/8 of peaks 100×, re-rank each time, record top-1 stability; then measure its AUC for separating hold-out molecules whose true structure is in the pool from those whose true structure we deliberately deleted Abstain and back off to the parent rank ("unclassified Enterobacteriaceae"); thresholds 60 % (IDTAXA), 80 % (RDP), 0.8 (SINTAX)
Open-set recognition / OOD — OpenMax (Bendale & Boult 2016, arXiv 1511.06233), energy score (Liu 2020, arXiv 2010.03759), BirdCLEF 2024 1st place OpenMax F-measure 0.595 versus 0.568 for thresholded softmax, on 50,000 ImageNet validation + 15,000 open-set + 15,000 fooling images; α = 10 top classes recalibrated, Weibull tail size 20 fitted to distances from the mean activation vector. Energy score E(x) = −T·log Σ exp(f_i/T): FPR95 on CIFAR-10 WideResNet 51.04 % → 33.01 %; with energy fine-tuning 8.53 % → 3.32 %. BirdCLEF 2024 winner added a 183rd "nocall" class at training and dropped it at inference: public score unchanged, private 0.655 → 0.671 Build the "none of the above" class from decoys and throw it away at inference. Test: for each hold-out molecule, delete its true structure from the pool for half the examples; train a small head on (spectrum embedding, top-25 score vector) to predict "truth absent"; report AUC and the MRR change from routing high-scoring-absent molecules to a generative branch Reject as unknown. None of these systems produces a replacement answer

Machine-readable version of this table: healthcare_diagnosis_astronomy_ecology_other_rank_25_problems.json.

2. What each field does when the answer is not in the reference — and what we can actually do

Field Their response Our analogue Does it earn MRR@25?
Taxonomy Abstain; emit the parent taxon Emit "an analog of compound X" / the scaffold No. Scoring is exact InChIKey14 after RDKit 2026.03.3 tautomer canonicalisation. A correct scaffold with a wrong structure is worth exactly zero
Astronomy "Orphan"; coarser class Flag the molecule as low-confidence Only if the flag changes which 25 structures are emitted
Open-set vision / audio Reject as unknown Same Same
Differential diagnosis Never arises — —
Nobody generates — Enumerate analogs of the best library hits by mass-difference rules (methyl +14.0157, hydroxyl +15.9949, acetyl +42.0106, hexose +162.0528, deoxyhexose +146.0579, pentose +132.0423) constrained to the inferred molecular formula; or run de novo structure generation conditioned on spectrum + formula Yes — this is the only branch that can score on the ~45 % of hidden molecules whose answer never enters the pool

The structural asymmetry is the whole lesson. Every field surveyed is rewarded for saying "unknown"; we are not permitted to. So the import from IDTAXA and OpenMax is not the abstention step — it is the detector that precedes it. Their contribution is a calibrated signal "the truth is probably not in my reference", which for us becomes a routing decision, not an output.

3. Hubness and popularity bias in nearest-neighbour retrieval

Radovanović, Nanopoulos & Ivanović (JMLR 11:2487, 2010) define hubness as right-skew in the distribution of k-occurrences N_k — how often a point appears in other points' k-nearest-neighbour lists. Across 50 real data sets the Spearman correlation between embedding dimensionality and the standardised skewness S_N10 is 0.62, and the correlation with intrinsic dimensionality (maximum-likelihood estimator) is stronger still; shuffling features to destroy dependencies raises skewness further, which is their evidence that hubness tracks intrinsic rather than embedding dimension. They separate N_k into "good" and "bad" k-occurrences by label agreement, and show that down-weighting a point's vote by exp(−h_B), where h_B is its standardised bad-hubness score, raises leave-one-out k-NN accuracy on six real data sets (their Fig. 11 — the deltas are plotted, not tabulated, so I cannot quote a number). Schnitzer et al. (JMLR 13:2871, 2012) report mutual proximity raising k = 5 accuracy on Corel-1000 from 65.2 % to 70.3 % (taken from the authors' project page, not the PDF — medium confidence).

CSLS (Conneau et al., ICLR 2018, arXiv 1710.04087) is the cheapest fix and the one with the cleanest numbers:

CSLS(Wx_s, y_t) = 2·cos(Wx_s, y_t) − r_T(Wx_s) − r_S(y_t), with K = 10

where r_T is the query's mean cosine to its K nearest gallery items and r_S is the gallery item's mean cosine to its K nearest queries. The paper's framing is exactly ours: "some vectors, dubbed hubs, are with high probability nearest neighbors of many other points, while others (anti-hubs) are not nearest neighbors of any point."

Pair Procrustes NN Procrustes CSLS Adv-Refine NN Adv-Refine CSLS
en-es 77.4 81.4 79.1 81.7
es-en 77.3 82.9 78.1 83.3
en-fr 74.9 81.1 78.1 82.3
fr-en 76.1 82.4 78.2 82.1
en-de 68.4 73.5 71.3 74.0
de-en 67.7 72.4 69.6 72.2
en-ru 47.0 51.7 37.3 44.0
ru-en 58.2 63.7 54.3 59.1
en-zh 40.6 42.7 30.9 32.5
zh-en 30.2 36.7 21.9 31.4

Gain from CSLS alone: +2.1 to +6.5 P@1 points in the supervised setting, largest on the hardest pairs. A 2026 preprint (arXiv 2605.26575) argues hubness rather than anisotropy drives cross-lingual retrieval asymmetry — hub mass carries a 49.5 % dominance share and partial R² 0.302 against 0.003 for anisotropy — and that CSLS closes 63.5 % of the worst-to-best reciprocity gap.

Nothing comparable has been published for spectrum→structure embeddings. I searched and did not find hubness or CSLS treatment in DreaMS, SpecBridge, FLARE or the contrastive spectrum-embedder literature; that is a genuine gap, and the experiment is a few lines of code. The mapping: queries are the ~1,500 hidden-test spectrum embeddings, the gallery is the candidate-structure embedding set, and r_S must be estimated against a query bank (the test spectra plus, say, 50,000 random training spectra) because the gallery outnumbers the queries by orders of magnitude. The predicted hubs are structurally generic molecules — simple flavonoid glycosides, small terpenoid acids — whose predicted-fingerprint vectors sit near the data mean, which is precisely Radovanović's mechanism. My honest expectation of the size of the effect here is small, because the per-molecule exact-mass window already reduces each ranking problem to isomers of one formula, and hubness is a global-neighbourhood pathology. The place to look for a real gain is inside the largest isomer families, where several hundred candidates compete and one or two of them are everybody's nearest neighbour.

4. Morelli secondary test: are the quiet peaks the informative ones?

Morelli attributed paintings from earlobes and fingernails on the theory that a copyist reproduces the celebrated features and betrays himself in the unconsidered ones. The mass-spectrometric translation: the base peak is the celebrated feature — a stable heteroatom ring, a dominant neutral loss — shared across an entire structural family, while the low-abundance ions carry what separates its members. Li et al. 2021 assert this directly: "highly abundant MS/MS fragment ions are often shared between structurally similar molecules … ADP and ATP both showing the adenine fragment m/z 136.062 as base peak. Hence, the difference between highly similar structures may best be discerned from low-abundant ions."

4.1 Evidence table: intensity transforms and what they were shown to do

Transform Source Data Reported effect Is it an identification-accuracy result?
Intensity scaling with m/z weighting; square root "nearly optimal", mass weight best at power 2–3 Stein & Scott 1994, JASMS 5:859 12,592 replicate EI spectra of ~8,000 NIST compounds Dot product best at 75 % rank-1; Euclidean 72 %, absolute-value 68 %, probability-based matching 65 %, Hertz 64 %. A composite term rewarding consistent intensity ratios among shared peaks gave "a modest improvement" Yes — but EI, unit resolution, closed library
I^0.5 (Sokolow, Horai/MassBank), I^0.53 (Kim 2012, exponents chosen by maximising the skewness/kurtosis ratio of library-internal scores), I^0.6 m/z^3 (Stein & Scott) as compiled in Hart et al. 2024 — Exponents are all sub-linear Secondary citation; I did not open Kim 2012 or Sokolow
Dynamic entropy weighting: I_new = I^w with w = 0.25 + 0.25·S for S < 3, w = 1 for S ≥ 3, then renormalise to sum 1 Li et al. 2021, Nat Methods, doi 10.1038/s41592-021-01331-z; implementation in the MSEntropy package 434,287 query spectra vs NIST20; repeated on 90,382 MassBank.us spectra; 37,299 natural-product spectra; 770 manually curated gut metabolites Best of 43 similarity measures, AUC 0.958; 27 measures beat the plain dot product. For information-poor spectra (S < 1) unweighted entropy gave FDR > 35 % at similarity 0.9 while the weighted form sat near 15 % — the authors call this "the importance of low-abundance ion weighting". On natural products, entropy similarity reached FDR < 10 % at a 0.75 threshold whereas dot product "barely reached 15 % FDR" above 0.95. On the gut data at 0.75: 26 false positives (5.8 % FDR) versus 50 (9.6 %) for dot product Yes, and it contains the only published ablation of the idea. Crucially it also reports where Morelli fails: "if such power transformations are used for ion-rich spectra with high entropy values, similarity scoring performance is decreased", which is why the rule is switched off above S = 3
Square-root intensities; peaks below 0.1 % of base removed; at most 1,000 peaks; 10,000 bins over m/z 10–1000 (9,948 non-empty) MS2DeepScore, Huber et al. 2021, J Cheminform 13:84 GNPS Tanimoto RMSE ≈ 0.15, falling to ≈ 0.10–0.11 when Monte-Carlo-dropout uncertainty filtering discards ~75 % of pairs. The stated reason for the square root is "to avoid a too strong focus on the highest intensity peaks only" No ablation of the transform is reported — the choice is asserted, not measured
Intensities normalised to base peak; peaks below 1 % of base discarded for the cosine baselines; two-decimal m/z binning Spec2Vec, Huber et al. 2021, PLoS Comput Biol GNPS Library matching up to 88 % accuracy; better true-to-false ratios than cosine at every threshold No intensity-transform ablation
I^0.25 multiplied by the m/z's relative frequency in the reference library, also to the power 0.25 Hart et al. 2024, JASMS (PMC10854760) 10 M GNPS spectrum pairs for alignment; 25,437 GNPS queries against NIST23 (2,776 unique molecules) for library search Alignment: modified-cosine F1 0.094 → 0.14 (optimal threshold moving 0.88 → 0.67). Library search top-1: Sokolow's weights best, the authors' close behind, Kim's third — and some historical weighting schemes fell below unweighted cosine. The per-scheme numbers live in a figure and supplementary tables, not the main text, so I cannot quote them Partly. The honest reading: intensity weighting is clearly good for finding analogs, and only marginal for picking the exact compound
FSIWsim (fragment-size and intensity weighting), top-1 45.96 % on CASMI 2017 search snippet only — — Unverified — I did not open the paper; low confidence

Verdict on the analogy: it holds, with a measured boundary. Every optimised exponent published since 1994 is below 1, which is the quantitative form of "attend to the quiet peaks". But Li et al. located the point where it reverses — in ion-rich, high-entropy spectra, amplifying small peaks amplifies noise instead of signal. Nobody has published the extreme Morellian version (delete the base peak; use only sub-10 % peaks) as an identification experiment, only as similarity-metric tuning. That is the gap worth filling, and it is cheap.

4.2 Proposed MRR@25 experiment

Hold-out construction. From train.parquet, select molecules whose Bemis–Murcko scaffold appears nowhere else in the training set (scaffold-disjoint), whose molecular formula has at least 20 isomers in the candidate pool, and which carry at least 3 spectra. Target ~2,000 molecules. Keep the true structure in the pool for this experiment — this test is about ranking, not coverage. Mimic the hidden-test distribution where possible: timsTOF rows, merged multi-collision-energy spectra, the 7 permitted adducts.

Arms. Score every pool candidate with the identical ranker under: (i) full spectrum, relative intensity; (ii) base peak deleted — and for merged multi-CE spectra, the surviving precursor ion deleted as well; (iii) only peaks below 10 % of base; (iv) I^0.5, I^0.25, and log(1 + 100·I); (v) the entropy-dynamic rule w = 0.25 + 0.25·S capped at S = 3; (vi) two-stage: full spectrum to shortlist 100, then re-rank that shortlist by arm (iii).

Reporting. MRR@25 and recall@25, stratified by spectral entropy (S < 1.5 / 1.5–3 / > 3), by isomer-family size, and by number of spectra per molecule. Report paired bootstrap confidence intervals over molecules, not spectra.

Prediction, stated before running it. Arms (iv) and (v) win overall and land within ~0.01 MRR of each other; (i) is a few points behind; (ii) is roughly neutral for S < 1.5 and clearly harmful above, because the base peak in a merged multi-CE timsTOF spectrum aggregates several genuinely informative channels; (iii) alone loses badly on MRR but wins the tie-break inside large isomer families, which is why (vi) is the arm to watch. If (vi) beats (iv) by more than 0.01 MRR, Morelli is confirmed for this data and the production ranker should be two-stage. If (iii) alone lands within 0.02 of (i), the base peak is carrying far less identity information than intuition suggests, which is itself a publishable negative result for the write-up.

5. How much does a prior actually buy in metabolite identification?

Prior System, data Without → with Why it may not transfer to CASMI 2026
Database restriction: 0.5 M "structures of biological interest" instead of PubChem CSI:FingerID / SIRIUS 4 (Dührkop 2019 Nat Methods 10.1038/s41592-019-0344-8), CASMI 2016 positive mode Top-1 39.4 % in PubChem (74.8 % top-10) → 74.0 % in the bio database This is pool shrinkage, not evidence. It is the same mechanism as the public-LB observation that a 90 M PubChem pool hurts — and it carries the same cost: any truth outside the small pool becomes unreachable
PubChem / ChemSpider literature reference and patent counts added as a weighted scoring term MetFrag 2.2 (Ruttkies 2016 J Cheminform 8:3), 473 merged high-resolution Eawag/MassBank spectra Fragmenter score alone 30 top-1 (~6 %) → 336 (71 %) with PubChem reference counts → 420 (89 %) with ChemSpider references; MetFrag + CFM-ID reached 343 (73 %) Inverted for us. The hidden molecules are described as "none previously published", so their reference count is ~0 by construction. Adding this prior would systematically demote the correct answer. This is the single most dangerous import in this whole document
Taxonomic (producing-organism) match, scored +1 / +2 / +3 at family / genus / species (optimum found at 0.81 / 1.62 / 2.55) Rutz et al. 2019 Front Plant Sci 10:1329, 2,107 characterised NPs with a unique biological source Rank-1 correct: ISDB-DNP 214 (10.2 %) → 1,510; SIRIUS 975 (46.3 %) → 1,508; MS-Finder 180 (8.5 %) → 546; F1 improved ~7× / 1.5× / 3× The largest published prior effect in natural-product annotation — and unusable here, because the hidden test gives no source organism. Worth noting as the measure of how much a true context prior is worth when you have one
NP-likeness (Ertl, Roggo, Schuffenhauer 2008 JCIM 48:68 — atom-centred HOSE-fragment naive Bayes, log-ratio contributions, score range about −5 to +5, ROC AUC 0.977, and 93.9 % of Novartis natural products absent from the Dictionary of Natural Products still correctly called NP at the ROC-optimal cutoff) Jayaseelan & Steinbeck 2014 BMC Bioinformatics 15:234 — 41 small natural products, 100 structure-generation runs each Correct structure retrieved in 1,096 / 4,100 runs → 1,470 / 4,100 (a 9.1 % improvement); ranked first in 9 of 34 → 17 of 35 recovered cases Important nuance: the gain came on generated candidates, where the prior deletes chemically implausible output. On a COCONUT pool every candidate is already natural-product-like, so expect little. But on our generative C-branch this is exactly the right filter, and Ertl's 93.9 % figure on structures absent from the reference dictionary is the direct evidence that it generalises to unseen chemistry
Calibrated confidence rather than raw score (COSMIC: a direction-constrained linear SVM over the calibrated score, score gaps to other candidates, fraction of fragmentation-tree intensity explained, and fingerprint cardinality; e-values from a log-normal mixture fitted to that query's own PubChem candidate-score distribution) Hoffmann et al. 2022 Nat Biotechnol 10.1038/s41587-021-01045-9 Mean rank of the correct annotation 56.5 → 33.9; AUC 0.82 against 0.40–0.55 for competing in-silico tools on 127 CASMI 2016 queries; 1,715 high-confidence annotations absent from spectral libraries across 17,400 public experiments Transfers cleanly, and it is the template for Section 6. The key design choice is that the score is calibrated per query against that query's own candidate pool, which makes confidences comparable across molecules with wildly different pool sizes

6. Forced analogy: art attribution and the Rembrandt Research Project

The analogy, stated precisely. The Rembrandt Research Project's Corpus (Bruyn, Haak, Levie, van Thiel, van de Wetering, from 1982) sorted paintings into three categories: A, by Rembrandt; B, authorship neither securely accepted nor rejected; C, not by Rembrandt. Two well-known verdicts moved when evidence of a different kind arrived. For The Polish Rider (Frick Collection), Josua Bruyn of the project suggested in 1984 that characteristics of Rembrandt's pupil Willem Drost could be seen in it; that view did not hold, Ernst van de Wetering argued for Rembrandt at the 1997 Melbourne symposium, and a 1998 project study concluded that a second hand had worked on the painting alongside Rembrandt — the doubt resolved into "Rembrandt, plus a collaborator", possibly a canvas he left unfinished. For The Man with the Golden Helmet (Gemäldegalerie Berlin), a Dutch curators' commission raised doubts in 1984, Jan Kelch announced a stylistic mismatch in November 1985, and in 1985–86 the Doerner Institut in Munich examined the painting by autoradiography, infrared and ultraviolet photography, X-radiography and paint-sample analysis, after which the museum concluded it was by someone in Rembrandt's circle rather than by Rembrandt. (Whether the project itself issued a formal C is not established by the sources I opened.)

The mapping to our problem: connoisseurial style corresponds to learned similarity — predicted fingerprints, contrastive spectrum–structure embeddings, library cosine. Technical examination corresponds to hard physical constraints, and the important property is that they are orthogonal to the learned score rather than better than it: the neutral mass implied by each of a molecule's 1–16 spectra under its own adduct, agreeing to within a few ppm; the ring-and-double-bond equivalent and element counts of the candidate formula; the fraction of total ion current assignable to subformulas of the candidate; whether fragment formulas stay consistent as collision energy rises. Dendrochronology cannot tell you who painted a panel — it can only tell you the wood was felled too late. Exact mass is the same kind of evidence: it excludes, it does not attribute.

The derived, testable idea: an A/B/C triage with an explicit routing rule.

Per-molecule features, all computable without knowing the answer:

  1. a COSMIC-style calibrated top-1 score — the top candidate's score expressed as an e-value against the score distribution of that molecule's own candidate pool, so molecules with 12 and 1,200 candidates are comparable;
  2. the score gap from rank 1 to rank 2, and from rank 1 to the median of the top 25;
  3. RDP-style bootstrap stability — resample 1/8 of the peaks 100 times, re-rank, record how often the same candidate stays top-1;
  4. an energy-style openness score, log Σ exp(s_i/T) over the top 25 unnormalised candidate scores, following Liu et al.'s point that softmax normalisation destroys exactly the magnitude information that separates in-distribution from out-of-distribution inputs;
  5. explained intensity — fraction of total ion current the top candidate can account for by subformula assignment;
  6. cross-spectrum agreement — whether the molecule's several spectra independently rank the same candidate first.

Tiers and routes. A — ranker output is kept as-is; the submission's 25 are the top 25. B — keep the top 25 but diversify the tail: the bottom 10 slots are filled by maximal-marginal-relevance over scaffolds rather than by score, on the symptom-checker logic that a near-duplicate in slot 18 buys nothing. C — the top 3–5 pool candidates are retained as insurance, and the remaining ~20 slots are filled from the generative branch: analogs of the best library hit generated by mass-difference rules consistent with the inferred formula, filtered by NP-likeness, de-duplicated by InChIKey14 after RDKit 2026.03.3 tautomer canonicalisation.

Proposed thresholds. Because the quantity we care about is a rate, set the thresholds by quantile rather than by absolute score: calibrate a single scalar triage score (logistic regression on features 1–6, trained on the hold-out described below) and route the bottom ~30 % to C and the next ~25 % to B, since the back-solved public-LB recall@25 of ≈ 0.55 implies roughly 45 % of molecules have no reachable answer. Then tune: the right C fraction is the one that maximises held-out MRR@25, and it should be smaller than 45 % because the generative branch is itself unreliable.

Validation protocol — the part that matters. Build a hold-out of ~1,000 natural products with timsTOF-like spectra and scaffolds disjoint from training. For a randomly chosen half, delete the true structure from the candidate pool, which manufactures exactly the condition we cannot otherwise observe. Report (a) the AUC of the triage score for separating truth-present from truth-deleted molecules — the directly comparable number from another field is OpenMax's F-measure of 0.595 versus 0.568, i.e. open-set detection is hard and the honest expectation is an AUC around 0.7–0.8, not 0.95; (b) MRR@25 with and without routing, separately on the truth-present and truth-deleted halves, because the test of a triage is that it does not damage the molecules it should have left alone; (c) the fraction of truth-deleted molecules for which the generative branch puts the true structure anywhere in the 25 — if this is below a few percent, the C route is not worth its risk and the correct decision is to use the triage only for tail diversification.

Where the analogy breaks — three places, honestly.

First and most importantly, the art historians were allowed to answer "C: not by Rembrandt" and stop. We cannot. A C verdict in our setting is not a conclusion, it is a request for a different generator, and nothing in the art-historical method tells us how to invent the painter. Second, the technical evidence in art attribution is often decisive on its own — a canvas woven after the artist's death ends the discussion — whereas our hard constraints are weak: thousands of COCONUT entries share a formula, so exact mass narrows the field and never closes it. Dendrochronology excludes one candidate; our mass filter excludes 10^7 and leaves 10^2. Third, the RRP had no metric and no cost function. It could spend twenty years on one panel, revise in 1998 what it wrote in 1984, and treat an unresolved B as an acceptable permanent state. We have MRR@25, a fixed submission deadline of 2026-12-14, and a scoring rule under which an honest "uncertain" is worth precisely zero. There is also a sampling caveat worth stating: The Polish Rider and The Man with the Golden Helmet are famous precisely because the verdicts moved, so reasoning from them overstates how often orthogonal evidence overturns a stylistic judgement — the same survivorship trap as reading only the Kaggle write-ups of teams whose trick worked.

Gotchas / what could silently fool us

  1. Popularity priors are inverted on this test set. MetFrag's ~6 % → 89 % top-1 gain comes from literature reference counts. The hidden molecules are "none previously published". Any feature correlated with how well-studied a structure is — reference counts, patent counts, PubChem CID low numbers, presence in HMDB/KEGG, number of GNPS spectra, even COCONUT occurrence counts — will push the truth down. A model trained on public libraries will learn this prior implicitly whether or not we add it as a feature; that is the subtle version of the trap.
  2. Validating on in-pool molecules measures the wrong thing. Nearly all published top-1 numbers (CSI:FingerID's 74.0 %, RDP's 89 %, GPT-4's 64 %) are closed-set. The eDNA field is the only one that systematically reports the open-set number, and the gap is enormous: RDP looks superb at 98 % precision in-set and forces 3.83 % of absent-genus sequences into a wrong genus, while a BLAST top hit does it to 17.3 % and invents 65 spurious genera. Our analogue of "BLAST top hit" is naive nearest-neighbour library search.
  3. Abstention has a coverage price that is invisible in error rates. IDTAXA's near-zero over-classification costs it genus-level coverage (19.9 % versus RDP's 22.5 % of gut reads). Any confidence threshold we introduce trades recall for precision, and MRR@25 only rewards precision at the top. A triage that abstains without a generative branch is strictly worse than no triage.
  4. Intensity transforms reverse sign with spectral entropy. Li et al. switch the weighting off above S = 3 because up-weighting small ions in ion-rich spectra degrades scoring. Merged multi-collision-energy timsTOF spectra are ion-rich by construction, so a weighting exponent tuned on single-energy library spectra may be mistuned for the actual test data. Always stratify the ablation by entropy.
  5. Isomers set a floor that no amount of weighting removes. In NIST20, roughly 30 % of Li et al.'s false positives at similarity > 0.95 were one-bond isomers and 21.5 % two-bond isomers. Since the metric is InChIKey14 after tautomer canonicalisation, a methoxy on the wrong ring carbon scores zero. Part of our 45 % miss is irreducible from MS/MS alone and is not a modelling failure to be optimised away.
  6. Multiple spectra per molecule are not independent samples. With 1–16 spectra (median 3) of the same compound, treating them as independent votes will overstate confidence — the same mistake as counting correlated catalogue cross-matches as independent evidence. Any bootstrap or ensemble confidence must resample at the molecule level.
  7. Hubness is a property of the metric, not of a molecule. Correcting it with CSLS changes every score, including the ones that were already right. The 2026 preprint's framing is explicit on this point, and a naive CSLS application can lower P@1 if the query bank is unrepresentative of the test distribution — ours must be drawn from timsTOF natural-product spectra, not from the full heterogeneous training mixture of 82 instrument types.
  8. The visible test set is a decoy (verbatim enveda-180 training rows), so any local score computed on it measures memorisation. Every number in this document should be re-derived on a scaffold-disjoint, timsTOF, natural-product hold-out before being believed.
  9. Open-set detectors degrade under the exact shift we face. OpenMax's margin over thresholded softmax is 0.595 versus 0.568 — real but small, on a benchmark designed for the task. Expect our truth-absent detector to be mediocre, and design the routing to be robust to a mediocre detector (keep the top pool candidates even in C) rather than betting on a good one.
  10. Vendor and secondary numbers leak into the literature. Isabel's "96 % top-10" is a vendor claim; the peer-reviewed numbers for the same class of system are 34 % top-1 / 58 % top-20 (Semigran) and 3.45/5 (Bond). Several intensity exponents in Section 4 (Sokolow, Kim, Horai) reach us through Hart et al.'s compilation rather than from the primary papers. Marked, and not to be quoted as primary.

Implications for the challenge

  1. Do not add any popularity or well-studiedness feature (reference counts, patent counts, database occurrence, spectrum counts). Actively audit the existing ranker for an implicit version of it: correlate the model's score on held-out molecules against the number of training spectra for structurally similar compounds, and if the correlation is strong, consider a de-biasing term.
  2. Build the truth-absent hold-out this week. ~1,000 scaffold-disjoint natural products; delete the true structure from the pool for a random half. This single artefact is what lets every other idea here be measured, and nothing in the public write-ups suggests other teams have one.
  3. Implement the A/B/C triage of Section 6 with the six features listed, thresholds set by quantile and then tuned for held-out MRR@25. Insist on the negative control: routing must not reduce MRR on the truth-present half.
  4. Spend the tail slots, not the head slots, on diversity. Slots 1–5 by score; slots 6–25 by maximal-marginal-relevance over Murcko scaffolds for B-tier, and by generated analogs for C-tier. Semigran's data says a list of near-duplicates wastes its length (3 → 20 slots bought 7 points).
  5. Adopt the entropy-dynamic intensity weighting (I^w, w = 0.25 + 0.25·S for S < 3, w = 1 above) as the default spectrum preprocessing, and run the arm comparison of Section 4.2 to confirm it on timsTOF merged spectra rather than assuming the NIST20 result transfers.
  6. Add bootstrap-over-peaks stability (resample 1/8 of peaks, 100 iterations) as a confidence feature. It is model-agnostic, costs one extra forward pass per resample on the candidate scoring only, and in taxonomy it is the single mechanism that distinguishes "found the right thing" from "found the nearest wrong thing".
  7. Use the unnormalised log-sum-exp of the top-25 candidate scores as the openness feature, not a softmax probability (Liu et al.: FPR95 51.04 % → 33.01 % from exactly this change).
  8. Test CSLS on the spectrum→structure retrieval step, with the query bank restricted to timsTOF natural-product spectra. Expected gain is small because the formula window already localises each problem, but the published effect in an analogous cross-modal setting is +2 to +6.5 P@1 and the implementation is a few lines. Measure it inside the largest isomer families specifically.
  9. Apply NP-likeness only to generated candidates, never as a re-ranker on a COCONUT pool where everything already scores high. Ertl's figure that 93.9 % of natural products absent from the reference dictionary were still called natural products is the evidence it will not systematically reject genuinely novel chemistry.
  10. Build the generative C-branch around mass-difference analog enumeration first (methyl +14.0157, hydroxyl +15.9949, acetyl +42.0106, hexose +162.0528, deoxyhexose +146.0579, pentose +132.0423, and their combinations consistent with the inferred formula), because it is cheap, interpretable and directly targets "analogs of known natural products" — the stated chemistry of the hidden set. De novo generation is the second-order option.
  11. Report every result with a paired bootstrap over molecules, not spectra, and stratify by isomer-family size. The 400-molecule hidden test means a 0.01 MRR difference is close to noise; do not chase it.

Open questions

  1. What is the actual AUC of a truth-absent detector on natural-product MS/MS? No one appears to have published this. COSMIC's AUC 0.82 for correct-versus-incorrect annotation is the closest proxy and is a different question.
  2. Does hubness measurably exist in spectrum→structure embedding spaces? Unanswered in the literature I opened. Measuring the skewness S_N10 of the candidate-pool embedding would settle it in an afternoon.
  3. What is the ceiling imposed by MS/MS-indistinguishable isomers on a COCONUT-sized pool under InChIKey14 matching? Li et al.'s one-bond/two-bond breakdown (30 % / 21.5 % of high-scoring false positives) suggests it is substantial but is specific to NIST20.
  4. Do the per-scheme top-1 numbers in Hart et al.'s supplementary tables show any intensity weighting beating unweighted cosine for exact identification, or only for analog alignment? The main text hints at the latter. Worth pulling the supplement.
  5. Would a Sherlock-style deterministic rule layer (hard constraints applied before any learned score, with explicit precedence) outperform feeding the same constraints as features? Astronomy chose rules-first; our field almost always chooses features.
  6. Is there any exploitable context prior analogous to Rutz's producing-organism prior — which was worth a 7× F1 improvement — recoverable from the hidden test's metadata (adduct set, collision-energy pattern, which molecules share a submission batch)? If molecules arrived in related batches, that is a weak taxonomic prior in disguise.

Sources

Opened and read (full text or extracted PDF text) unless noted:

  • Semigran HL et al. "Evaluation of symptom checkers for self diagnosis and triage: audit study." BMJ 2015;351:h3480. doi 10.1136/bmj.h3480 (via Europe PMC PMC4496786 full text).
  • Kanjee Z, Crowe B, Rodman A. "Accuracy of a generative artificial intelligence model in a complex diagnostic challenge." JAMA 2023. doi 10.1001/jama.2023.8288 (jamanetwork.com full article).
  • Bond WF et al. "Differential diagnosis generators: an evaluation of currently available computer programs." J Gen Intern Med 2012. doi 10.1007/s11606-011-1804-8 — numbers via search-result summary of the abstract, not the full text.
  • Isabel Healthcare product documentation (likelihood indicator, "don't-miss" flags, age/sex/region filtering) — vendor material; the "96 % top-10" claim is unverified.
  • Gagliano A et al. "GHOST: Using only host galaxy information to accurately associate and distinguish supernovae." arXiv:2008.09630 / ApJ 2021;908:170 (abstract page).
  • Foley RJ, Mandel K. "Classifying supernovae using only galaxy data." arXiv:1309.2630 / ApJ 2013;778 — via search-result summary of the abstract.
  • Muthukrishna D et al. "RAPID: early classification of explosive transients using deep learning." arXiv:1904.00014 (ar5iv full text; confusion-matrix values are in figures only).
  • Smith KW et al. "Design and operation of the ATLAS Transient Science Server." arXiv:2003.09052 (ar5iv full text; Sherlock catalogues, classes, 1.5″ radius, 2.4× semi-major-axis and 50 kpc rules).
  • Wang Q et al. "Naive Bayesian classifier for rapid assignment of rRNA sequences into the new bacterial taxonomy." Appl Environ Microbiol 2007 (Europe PMC PMC1950982 full text).
  • Murali A, Bhargava A, Wright ES. "IDTAXA: a novel approach for accurate taxonomic classification of microbiome sequences." Microbiome 2018;6:140. doi 10.1186/s40168-018-0521-5 (Europe PMC PMC6085705 full text).
  • Edgar RC. "SINTAX: a simple non-Bayesian taxonomy classifier." bioRxiv 074161, 2016 — the preprint itself returned HTTP 429 and was not read; the 0.8 cutoff equivalence to RDP at 80 % comes from the drive5 USEARCH manual page cmd_sintax.html, which I did open. Low confidence on all other SINTAX details.
  • Bendale A, Boult TE. "Towards open set deep networks" (OpenMax). arXiv:1511.06233 (ar5iv full text).
  • Liu W et al. "Energy-based out-of-distribution detection." arXiv:2010.03759 (ar5iv full text).
  • BirdCLEF 2024 1st-place write-up, summarised at zenn.dev/yuto_mo/articles/ad43c630729073 — a third-party write-up of a Kaggle solution, as the Kaggle pages are JS-rendered and unreadable here.
  • Radovanović M, Nanopoulos A, Ivanović M. "Hubs in space: popular nearest neighbors in high-dimensional data." JMLR 2010;11:2487 (PDF text extracted locally).
  • Schnitzer D et al. "Local and global scaling reduce hubs in space." JMLR 2012;13:2871 — Corel-1000 numbers taken from the authors' project page summary, not the PDF. Medium confidence.
  • Conneau A et al. "Word translation without parallel data." ICLR 2018, arXiv:1710.04087 (ar5iv full text, Table 1 and the CSLS definition).
  • "Hubness, not anisotropy, drives cross-lingual retrieval asymmetry in multilingual embedding models." arXiv:2605.26575 (abstract page only).
  • Stein SE, Scott DR. "Optimization and testing of mass spectral library search algorithms for compound identification." J Am Soc Mass Spectrom 1994;5:859. doi 10.1016/1044-0305(94)87009-8 — the publisher page was paywalled; the 12,592 spectra, 75 % rank-1 dot product, 72/68/65/64 % for the alternatives, and the square-root-intensity / power-2-or-3-mass findings come from the Zenodo record 1258660 abstract, which I opened.
  • Li Y, Kind T, Folz J, Vaniya A, Mehta SS, Fiehn O. "Spectral entropy outperforms MS/MS dot product similarity for small-molecule compound identification." Nat Methods 2021. doi 10.1038/s41592-021-01331-z (PDF text extracted locally from the eScholarship author manuscript, qt0249w6hk — equations 2 and 3, the 43-method AUC 0.958 benchmark, the FDR numbers, and the ADP/ATP base-peak argument).
  • MSEntropy implementation, apply_weight_to_intensity (github.com/YuanyueLi/MSEntropy) — confirms if entropy < 3: weight = 0.25 + 0.25 * entropy, power transform then renormalisation.
  • Huber F et al. "MS2DeepScore: a novel deep learning similarity measure to compare tandem mass spectra." J Cheminform 2021;13:84. doi 10.1186/s13321-021-00558-4 (Europe PMC PMC8556919 full text — 10,000 bins over m/z 10–1000, 9,948 non-empty, 0.1 % intensity floor, ≤1,000 peaks, square-root intensities, RMSE ≈ 0.15).
  • Huber F et al. "Spec2Vec: improved mass spectral similarity scoring through learning of structural relationships." PLoS Comput Biol 2021 (journal full text).
  • Hart M et al. "Weighting low-intensity MS/MS ions and m/z frequency for spectral library annotation." J Am Soc Mass Spectrom 2024 (Europe PMC PMC10854760 full text — 10 M GNPS pairs, 25,437 queries vs NIST23, F1 0.094 → 0.14; per-scheme top-1 values are in a figure and supplement I did not open).
  • Ruttkies C et al. "MetFrag relaunched: incorporating strategies beyond in silico fragmentation." J Cheminform 2016;8:3. doi 10.1186/s13321-016-0115-9 (Europe PMC PMC4732001 full text — the weighted-sum scoring equation, 473 spectra, 30 / 336 / 420 top-1 ranks).
  • Dührkop K et al. "SIRIUS 4: a rapid tool for turning tandem mass spectra into metabolite structure information." Nat Methods 2019. doi 10.1038/s41592-019-0344-8 — the 39.4 % PubChem / 74.0 % bio-database / 74.8 % top-10 figures for CASMI 2016 come from a search-result summary of the paper, not from the full text I opened.
  • Rutz A et al. "Taxonomically informed scoring enhances confidence in natural products annotation." Front Plant Sci 2019;10:1329 (Europe PMC PMC6824209 full text — 2,107 entries, 0.81/1.62/2.55 optimal weights, rank-1 counts before and after).
  • Ertl P, Roggo S, Schuffenhauer A. "Natural product-likeness score and its application for prioritization of compound libraries." J Chem Inf Model 2008;48:68 (PDF text extracted locally from peter-ertl.com reprint — equation 1, range −5 to 5, ROC AUC 0.977, 93.9 % of non-DNP Novartis natural products correctly classified).
  • Jayaseelan KV, Steinbeck C. "Building blocks for automated elucidation of metabolites: natural product-likeness for candidate ranking." BMC Bioinformatics 2014;15:234 (Europe PMC PMC4089557 full text — 41 compounds, 4,100 runs, 1,096 → 1,470, 9/34 → 17/35).
  • Hoffmann MA et al. "High-confidence structural annotation of metabolites absent from spectral libraries." Nat Biotechnol 2022. doi 10.1038/s41587-021-01045-9 (Europe PMC PMC8926923 full text — SVM features, mean rank 56.5 → 33.9, AUC 0.82, 1,715 annotations).
  • "The Rembrandt research project." Editorial, Burlington Magazine 1983;125(968):661 (locally extracted text — the Corpus's team authorship, Bruyn and van de Wetering's leading roles, the project's stated aim of establishing a "logical evolution" as the basis for attribution decisions).
  • Wikipedia, "The Polish Rider" and "The Man with the Golden Helmet"; rembrandtdatabase.org in-focus article 43 (Doerner Institut 1985–86 autoradiography, infrared, UV, X-radiography and sample analysis; Berlin's "circle of Rembrandt" conclusion). Encyclopaedic and museum-database sources, not primary art-historical scholarship; the A/B/C category definitions and the 1984 Drost suggestion are reported at second hand. The claim that van de Wetering abandoned A/B/C in the later Corpus volumes is my recollection and is NOT verified by anything I opened — treat as unverified.

Not obtained despite trying: the Stein & Scott full text (paywalled), the SINTAX preprint (rate-limited), Kim et al. 2012 Bioinformatics (paywalled; its I^0.53 exponent reaches this document only through Hart et al.), and Hart et al.'s supplementary tables with the per-scheme top-1 precisions.