Candidate ranking: features, learning-to-rank models and score fusion for MRR@25

TL;DR. Every published gain in MS candidate ranking comes from one of three things: a better spectrum-to-structure scorer (IOKR/Bayesian scoring/MIST each add 2-6 percentage points top-1), fusing orthogonal scorers (equal-weight fusion of six fragmenters beat the best single tool 82 vs 70 top-1 in CASMI 2016; MetFusion cut median rank 28 to 7), or "metadata" terms that are really database-popularity priors (PubChem/ChemSpider reference counts alone put 143/208 CASMI 2016 challenges at rank 1 - and will do nothing or worse for a hidden test of unpublished natural products). For the challenge: train a LightGBM lambdarank model with group = molecule_id over all in-window COCONUT candidates (10^6 rows trains in minutes on CPU), feed it per-ranker scores plus within-query context features (z-score, gap to runner-up, candidate count), validate structure-disjoint, and keep reciprocal-rank fusion (k = 60) as the no-training baseline the learned model must beat.

Web-search budget was exhausted for this session partway through, so coverage leans on primary papers reachable by DOI/PMC and on two PDFs read locally; items I could not open are flagged.

1. Features with published evidence

Reported deltas, in the papers' own units. "Top-1" = number or fraction of queries where the true structure is rank 1.

Two things jump out. First, the database-popularity terms (#3-#5) dominate every benchmark that contains known compounds, which is exactly why they are poison for a test set described as "none previously published": the correct answer has by construction a low or zero reference count, so the feature would push it down. Second, the genuinely spectral improvements are modest (2-6 pp top-1 each) and partly additive, which is why fusion matters.

2. Models with ablations

MassSpecGym (arXiv 2410.23326, read locally from the PDF) gives the cleanest modern retrieval baselines, with candidates capped at 256 per spectrum (87 % of molecules hit the cap by mass, 66 % by formula), MCES-based structure-disjoint split, 231,104 spectra:

Note the inversion: formula-restricted candidate sets are harder than mass-window sets (MIST 9.57 vs 14.64 %) because same-formula isomers are closer decoys. A +/-10 ppm COCONUT window with median 36 candidates is closer to the "mass" setting, but much smaller than 256, so published hit@1 figures do not transfer; only relative deltas do.

3. Score fusion of orthogonal rankers

Evidence that fusion beats the best single method:

• MetFusion (Gerlich & Neumann 2013, J Mass Spectrom 48:291, DOI 10.1002/jms.3123; abstract only, paywalled): MetFrag + MassBank spectral similarity on 1,062 spectra, median rank 28 -> 7, "even when correct compounds were absent from the spectral library".
• CASMI 2016 Combined_MS/MS: "combining six different fragmenters with equal weighting: CFM_orig, CSI:FID, CSI:IOKR_A, MAGMa+, MetFrag and MS-FINDER" -> 82 top-1 vs 70 (CSI:FID), 62 (IOKR), 46 (MS-FINDER), 39 (CFM), 24 (MAGMa+). MetFrag+CFM+RT+Refs 163, Combined_MS/MS+RT+Refs 164.
• Ruttkies 2016 + CFM-ID: adding CFM-ID to MetFrag's full score moved PubChem top-1 from 336 to 343 of 473.
• MIST hybrid: lambda_r = 0.3 blend of fingerprint-cosine and contrastive distance beat either alone from top-5 upward.
• RRF (Cormack, Clarke, Büttcher, SIGIR 2009): RRF(d) = sum_r 1/(k + r(d)), "k = 60 was fixed during a pilot investigation"; pilot MAP vs k: 0 -> .2072, 10 -> .2123, 60 -> .2145, 100 -> .2142, 500 -> .2098 (best individual .2016, Condorcet .2074, CombMNZ .2039). On TREC Robust/3/5/9 RRF beat Condorcet in all cases and CombMNZ in all but one; "outperforms Condorcet, CombMNZ and the best system by 4 % to 5 % on average". The constant k "mitigates the impact of high rankings by outlier systems" and the 1/(k+r) shape keeps lower ranks from vanishing as an exponential would.

Aggregation methods actually used in MS: MetFrag/CASMI 2016 use a weighted sum of scores each normalised to the maximum within the candidate list (equal weights for Combined_MS/MS); MIST uses a convex blend of two distances; COSMIC uses a linear SVM over score features with Platt scaling. I found no MS paper that benchmarks RRF or z-score fusion against these; the recommendation below therefore rests on IR evidence (Cormack) plus the consistent MS evidence that any reasonable fusion of orthogonal scorers beats the best single one.

4. Calibration: is the score monotone in P(correct)?

Within a query, every ranker is by definition monotone in its own estimate; across queries, raw scores are not. COSMIC quantifies this: with the raw CSI:FingerID score as confidence, 0 hits could be reported at 40 % FDR on CASMI 2016/ChemSpider, whereas COSMIC gives 16 hits at 0 % FDR; on the biomolecule DB, 57 hits at 10 % exact FDR vs 0 at 70 % FDR for MetFrag/MAGMa+/CFM-ID. COSMIC's recipe: (i) model the within-query score distribution of PubChem candidates as a log-normal mixture (kernel density) and derive a P-value/E-value for the top hit, (ii) a linear SVM with enforced feature directionality ("a high CSI:FingerID score of a hit should increase, but must never decrease, our confidence"), (iii) Platt scaling to a posterior. On independent QTOF data it gave 1,415 hits at 5 % FDR vs 52 for library search (20 eV) and 1,701 vs 1 (merged spectra); NIST structure-disjoint CV ~400 hits at 5 % FDR (n = 3,721), with "E-values only" substantially worse. Caveats they state: "Confidence score values must be treated as a score but not as the probability that the annotation is correct"; q-values were "highly conservative" on independent data; separation was inflated when training and evaluation structures overlapped.

For MRR@25 this matters less than it seems: the metric is purely within-query, so cross-query calibration only matters for fusion (deciding how much to trust each ranker on this query). That is exactly where COSMIC-style context features (score gap to runner-up, number of candidates, explained intensity) earn their keep as inputs to the fused model.

5. Negative sampling and hard negatives

Lesson: hard negatives help the embedding models, and the gain shrinks or reverses when the evaluation candidate set is already hard (MassSpecGym). For a GBDT re-ranker over in-window candidates there is no reason to subsample: all candidates (median 36, p90 204) are the natural training group, and the ranker then sees the exact decoy distribution it will face.

6. Practical: LightGBM / XGBoost ranking API and CPU time

• LightGBM: objective = lambdarank (or rank_xendcg, documented as "faster than and achieves the similar performance as lambdarank"); metric = ndcg or map; eval_at (alias ndcg_eval_at, default 1,2,3,4,5) - set [1, 25]; lambdarank_truncation_level default 30 ("controls the number of top-results to focus on", set slightly above the NDCG cutoff - 30 already covers @25); lambdarank_norm = true by default (normalises lambdas across queries, "improve the performance for unbalanced data"); label_gain default 0,1,3,7,... is fine for binary labels. Group sizes are passed as an array of consecutive group lengths (group=[27,18,67] style, or a .query file); rows must be contiguous per query. (https://lightgbm.readthedocs.io/en/latest/Parameters.html)
• XGBoost: rank:ndcg is the default and recommended fallback; rank:map for binary relevance; rank:pairwise is the unscaled RankNet loss. lambdarank_pair_method = mean (random lambdarank_num_pair_per_sample pairs per doc, better for small data) or topk (~k x |query| pairs); XGBRanker.fit(X, y, qid=...) with rows sorted by qid; metrics ndcg@k, map@k, pre@k. (https://xgboost.readthedocs.io/en/stable/tutorials/learning_to_rank.html)
• Binary relevance and MRR: with one positive per group, NDCG@25 with binary gains is 1/log2(1+rank), i.e. a softer version of MRR@25; map with one positive is exactly reciprocal rank (truncate with eval_at=[25]). Use map as the early-stopping metric.
• CPU time, measured here (LightGBM 4.7.0, 32-core Windows box, 8 threads, synthetic 1e6 rows, 40 float32 features, groups lognormal around median 36): see the benchmark line below (filled in from lgb_bench.py; if it reads "pending" the run had not finished when this file was written).

BENCHMARK_PLACEHOLDER

Forced analogy: multi-sensor data fusion in search-and-rescue

The analogy. SAR fuses sonar, side-scan, visual and drift-model evidence into one posterior over cells of a search grid: posterior(cell) proportional to prior(cell) x product over sensors of L_i(observation_i | cell). Taking logs, sensor evidence adds as log-likelihood ratios, and the operation is only correct if (a) each sensor's P(detection | target present) and P(false alarm) are calibrated and (b) sensors are conditionally independent given the target. Averaging raw sensor scores is wrong because a sonar "0.8" and a visual "0.8" carry different evidence.

Mapping: grid cell = candidate SMILES in the window; prior(cell) = database prior (COCONUT membership, NP-likeness, class-prior from CANOPUS); sensors = fingerprint-prediction scorer, combinatorial fragmenter, library-similarity, contrastive-embedding cosine, isotope/formula agreement.

What it would take to build the LR version.

1. For each ranker r and each query q, estimate the null distribution of r's score over the in-window candidates (COSMIC's KDE / log-normal-mixture trick, using the other candidates as decoys), giving P(s | wrong, q).
2. Estimate P(s | correct) from the held-out, structure-disjoint validation set, binned by the within-query context (number of candidates, spectrum quality). Isotonic regression of "is correct" on s within context bins is the practical form.
3. log LR_r(c) = log P(s_r(c) | correct) - log P(s_r(c) | wrong). CSI:FingerID's Bayesian score is already log P(predicted fingerprint | candidate) under a tree model, so for it step 3 is nearly free; MetFrag's max-normalised score and a raw cosine are not likelihoods and need steps 1-2.
4. Fuse: score(c) = log prior(c) + sum_r w_r log LR_r(c), with w_r = 1 being naive Bayes and w_r < 1 the correction for redundant sensors. Fitting w_r by logistic regression on the validation set is literally a 1-layer learned fusion; LambdaMART on the same log-LR columns plus context features is the non-linear generalisation.

Testable idea for MRR@25. Run three fusions on the same structure-disjoint validation (novel InChIKey14, timsTOF-like, multi-CE merged): (A) RRF with k = 60 over the per-ranker rank lists; (B) naive log-LR sum (w_r = 1) using isotonic-calibrated per-ranker scores; (C) LightGBM lambdarank over [log LR_r, within-query z-score_r, rank_r, gap-to-runner-up_r, n_candidates, log prior]. Prediction: C > B > A in MRR@25, with B-A positive only if the per-ranker calibration is done per candidate-count stratum; if B < A, the calibration is broken and the LTR gain in C is coming from context features, not from the LR transform.

Where the analogy holds, partially holds, breaks. It holds for the prior/likelihood decomposition and for the point that raw-score averaging is wrong (MetFrag's own max-normalisation and CASMI's equal-weight sum are ad hoc versions of z-scoring). It partially holds for calibration: COSMIC shows per-query decoy E-values are feasible and that Platt-scaled SVM output is still "a score, not a probability". It breaks on independence: every ranker reads the same spectrum, fingerprint-based rankers (CSI, MIST, FFN) share the same failure modes (same wrong bits), and the fragmenter and the fingerprinter both fail on the same low-information spectra (few peaks, high CE). Summing log-LRs then double-counts evidence and over-confidently places a wrong candidate at rank 1 - precisely the failure MRR punishes hardest. The learned weights in (C) are the standard fix for correlated sensors, which is why the recommendation is LTR over LR-shaped features rather than pure Bayesian summation.

Gotchas / what could fool us

1. Reference-count leakage. ChemSpider refs alone = 143/208 top-1, lowest-CSID alone = 113/208 (CASMI 2016); refs alone 58-77 % top-1 (Ruttkies 2016). Any feature that encodes "how known is this candidate" (PubChem CID age, citation counts, number of source databases in COCONUT, presence in the training spectra) will look spectacular in a random train/val split and invert on a test of unpublished molecules. The same holds for "has a library spectrum" features: ~90 % of hidden molecules have none.
2. Structure overlap inflates everything. CSI:FID dropped from 70 to 36 top-1 (34 -> 18 %) when training compounds overlapping the challenge were removed; COSMIC saw better score separation under overlap and warned about it. Split by InChIKey14 at minimum, by MCES/scaffold clusters preferably (MassSpecGym style).
3. Candidate-set construction is part of the model. Formula-restricted candidates are harder than mass-window ones (MIST 9.57 vs 14.64 % hit@1); random capping to 256 adds easy decoys. Build validation candidate sets with the exact inference pipeline (COCONUT +/-10 ppm after RDKit 2026.03.3 tautomer canonicalisation, deduplicated by InChIKey14) or your local MRR is a fiction.
4. MRR is within-query; calibration only matters for fusion. Do not spend effort calibrating a single ranker's score across queries; spend it on context features that tell the fusion which ranker to trust.
5. Hard-negative mining can hurt when the evaluation set is already hard (JESTR -0.49 pp on MassSpecGym-by-mass). For the GBDT re-ranker, use all in-window candidates and do not subsample.
6. Groups with no positive carry no ranking gradient. If the true structure is outside the window (recall ceiling ~55 % at LB 0.417 per the brief), the group contributes nothing to lambdarank training but still counts as 0 in MRR; track recall@window separately from rank-within-window.
7. Multiple spectra per molecule (median 3 in hidden test). MVP reports consensus-spectrum annotation at 36.0 % vs 26.4 % rank@1 for single-spectrum annotation and states it beats rank aggregation of per-spectrum results; fuse at the spectrum/feature level (merge CEs, average embeddings) before ranking, not at the rank level after.
8. Duplicate candidates waste the 25 slots. Stereoisomers and tautomers collapse to one InChIKey14; COSMIC explicitly does not re-rank candidates sharing formula + fingerprint. Deduplicate before fusion, then diversify (one slot per InChIKey14).
9. Polarity imbalance. In CASMI 2016, CSI:FID had 70 positive-mode top-1 and 0 negative-mode; MS-FINDER led negative mode (14). Train and validate the fusion weights per ionization_mode.
10. Max-normalised scores are not comparable across tools. MetFrag normalises each term "to the maximum value within the candidate result list"; a 1.0 in a 2-candidate list and a 1.0 in a 2,000-candidate list mean different things. Add n_candidates and the within-query z-score as explicit features.
11. LTR benchmarks say simple fusion is hard to beat. On LETOR 3, RRF (0.6051 MAP) beat ListNet, AdaRank, RankSVM and RankBoost; CombMNZ edged RRF (0.6107, p ~ .2). If a learned fusion does not clearly beat RRF on a clean validation, the features are not adding information.

Implications for the challenge

1. Build the ranker as LightGBM lambdarank (or rank_xendcg) with group = molecule_id over all in-window COCONUT candidates; eval_at=[1,25], early-stop on map@25 (= MRR@25 with one positive), lambdarank_truncation_level=30, lambdarank_norm=true. 10^6 rows is a minutes-scale CPU job (benchmark line above).
2. Feature block per candidate: fingerprint-prediction scores (Morgan r2/r3 + MACCS Tanimoto and a Bayesian/Platt log-likelihood if available), combinatorial-fragmentation score (MetFrag-style explained-peak fraction), contrastive-embedding cosine, library-similarity of nearest train spectrum to the query with the candidate's own train presence masked out, formula/isotope agreement; plus within-query context: rank under each scorer, z-score, gap to runner-up, n_candidates, log-prior (NP-likeness, COCONUT source count). Explicitly exclude PubChem/ChemSpider reference counts and "in training set" indicators.
3. Always run RRF(k = 60) over the same rankers as the control. Expect +4-5 % over the best single ranker for free; the LTR must beat it on the structure-disjoint validation or it is overfitting to context leakage.
4. Validation must mimic the hidden test: InChIKey14-disjoint (MCES-cluster split preferred), timsTOF / merged multi-CE spectra, natural-product chemistry, candidates generated by the production window. Expect single-ranker top-1 to fall roughly by half relative to a random split (CSI:FID 34 -> 18 % effect).
5. Fuse multiple spectra per molecule before ranking (merged CE spectrum and/or averaged embeddings); do not aggregate ranks afterwards.
6. Deduplicate by InChIKey14 after RDKit tautomer canonicalisation before ranking, then fill 25 slots with distinct keys; a COSMIC-style E-value of the top hit vs the in-window decoys is a useful feature for deciding how aggressively to trust the top ranker on each query.
7. If a contrastive model is trained in-house, mine hard negatives from the actual window (Tanimoto-sorted, temperature-sampled as in MIST) but verify on the hard validation, since the gain can vanish (JESTR on MassSpecGym).
8. Per-polarity models or a polarity feature with interaction depth - the historical negative-mode weakness of fingerprint methods is large.

Open questions

• Does log-LR fusion with isotonic calibration beat RRF on in-window candidate sets? No MS paper tests this; experiment (A/B/C) above settles it in a day once scorers exist.
• Does NP-likeness or CANOPUS-class agreement add anything once a fingerprint scorer is present? No published number; cheap to test as a feature.
• Dührkop 2015 PNAS full text (Platt vs unit vs maximum-likelihood deltas on GNPS/MassBank) and the Rousu-group ACML "magnitude-preserving ranking" paper were not openable this session; their numbers would sharpen rows 7-8.
• How large is the candidate-count effect on calibration (COSMIC conditions on it; we have median 36, p90 204)? Needs our own validation curves.

Sources

• Ruttkies C, Schymanski EL, Wolf S, Hollender J, Neumann S. MetFrag relaunched. J Cheminform 2016;8:3. DOI 10.1186/s13321-016-0115-9 (Europe PMC PMC4732001 full text)
• Schymanski EL et al. Critical Assessment of Small Molecule Identification 2016: automated methods. J Cheminform 2017;9:22. DOI 10.1186/s13321-017-0207-1 (PMC5368104 full text)
• Brouard C et al. Fast metabolite identification with Input Output Kernel Regression. Bioinformatics 2016;32:i28. DOI 10.1093/bioinformatics/btw246 (PMC4908330 full text)
• Ludwig M et al. Bayesian networks for mass spectrometric metabolite identification via molecular fingerprints. Bioinformatics 2018. DOI 10.1093/bioinformatics/bty245 (PMC6022630)
• Ruttkies C, Neumann S, Posch S. Improving MetFrag with statistical learning of fragment annotations. BMC Bioinformatics 2019;20:376. DOI 10.1186/s12859-019-2954-7 (PMC6612146)
• Hoffmann MA et al. High-confidence structural annotation of metabolites absent from spectral libraries (COSMIC). Nat Biotechnol 2022;40:411. DOI 10.1038/s41587-021-01045-9 (PMC8926923 full text)
• Gerlich M, Neumann S. MetFusion. J Mass Spectrom 2013;48:291. DOI 10.1002/jms.3123 (abstract via Europe PMC; paywalled)
• Goldman S et al. MIST. bioRxiv 2022, DOI 10.1101/2022.12.30.522318 (full text)
• JESTR. Bioinformatics 2025. DOI 10.1093/bioinformatics/btaf354 (PMC12233093 full text)
• MVP: Learning from All Views. Anal Chem 2026. DOI 10.1021/acs.analchem.5c05675 (abstract via Europe PMC)
• Bach E et al. LC-MS2Struct. bioRxiv DOI 10.1101/2022.02.11.480137 (abstract); Bach E et al. Bioinformatics 2021, DOI 10.1093/bioinformatics/btaa998 (abstract)
• Bushuiev R et al. MassSpecGym. arXiv:2410.23326 (PDF read locally, Tables 3 and candidate construction)
• Cormack GV, Clarke CLA, Büttcher S. Reciprocal Rank Fusion outperforms Condorcet and individual rank learning methods. SIGIR 2009 (PDF from cormack.uwaterloo.ca, read locally)
• Dührkop K et al. SIRIUS 4. Nat Methods 2019, DOI 10.1038/s41592-019-0344-8 (abstract only)
• Dührkop K et al. CANOPUS. Nat Biotechnol 2021, DOI 10.1038/s41587-020-0740-8 (abstract only)
• Ertl P, Roggo S, Schuffenhauer A. Natural product-likeness score. J Chem Inf Model 2008, DOI 10.1021/ci700286x (abstract only)
• LightGBM Parameters: https://lightgbm.readthedocs.io/en/latest/Parameters.html
• XGBoost Learning to Rank: https://xgboost.readthedocs.io/en/stable/tutorials/learning_to_rank.html
• Not opened (HTTP 500/403): Dührkop 2015 PNAS DOI 10.1073/pnas.1509788112 (PMC4611636).