Queen digest — candidate space and retrieval (pool, formula, analogs, ranking, coverage ceiling)

Resume-mode tick: all five worker files were complete with Sources sections; no workers re-dispatched.

TL;DR (5 bullets a chief of staff would read)

1. There is one natural-product map, and it is small. COCONUT Oct-2026 collapses from 738,794 rows to 479,690 InChIKey14 (recomputed from the download; q3's worker got the identical number). LOTUS v11 adds 254, NPAtlas ≤ 138, SuperNatural/NPASS/KNApSAcK ~0; the 29-source THEOBROMA union reaches only 486,215. "Union of NP databases" is not a lever.
2. PubChem wholesale is a proven loss; gated injection is a small win. Laudicina: train+COCONUT LB 0.112 → train+PubChem 0.079 (−30 %, 68 % of top-1 slots stolen by PubChem-only decoys). gnshx: PubChem top-50 injected only when library confidence < 0.7 → +0.011. Prior-density ratio COCONUT : PubChem-only is ~1,500–2,000×, so a PubChem candidate needs ~7 nats of spectral evidence to earn a slot.
3. Formula-first filtering is a net loss on MRR@25. Inside a ±10 ppm COCONUT window the true formula already covers 62–90 % of structures (median 2–4 formulas per window), while MS2-only formula top-1 is 50–77 % on realistic benchmarks (MIST-CF 50.5 % on MassSpecGym; 74–77 % NPLIB1). Break-even is P(formula) ≈ 0.75–0.9. Use the formula posterior as a soft multiplicative feature; hard-gate only on mass/adduct.
4. The binding term is pool coverage c, not the ranker. MRR@25 = c·r·e with the LB-0.417 regime back-solved to c ≈ 0.55–0.61, r ≈ 0.90, e ≈ 0.76. Headroom: c +0.27, e +0.13, r +0.05. The ~39 % of hidden molecules in no database (Class 3) is the only large lever, and every way of raising c (analog enumeration, PubChem) dilutes e — so extension must be gated and query-conditioned.
5. Rank by calibrated posterior, not by ranker quota. Stone's optimal-search theorem (AF447 search) reduces the 25-slot problem to: sort the union of channels by calibrated P(candidate = truth). Inserting a foreign candidate at slot 1 needs P ≥ 0.2, slot 5 ≥ 0.006; appending at slots 11–25 costs only 0.0001–0.0003 MRR per slot. Practical ranker: LightGBM lambdarank over all in-window candidates, with RRF(k = 60) as the no-training control it must beat; exclude PubChem/ChemSpider reference counts (they encode "already published", which the hidden test is not).

New concrete facts (numbered; file + source count; confidence)

1. COCONUT Oct-2026: 738,794 rows → 728,390 full InChIKeys → 479,690 IK14; LOTUS v11: 227,298 full keys → 151,885 IK14, 99.83 % inside COCONUT (+254); THEOBROMA v1.34: 1,133,004 compounds → 486,215 IK14 families; PubChem live count 124,717,116 CIDs (2026-10-07). candidate_databases_...md (18 sources + local computation). High.
2. ±10 ppm neutral-mass window in IK14-dedup COCONUT: median 36, mean 87, p90 204, max 1,761 candidates (max is a C6H5N3O curation artefact); 300–400 Da median 64, 400–500 Da 49, 500–600 Da 20. Stereo-expanded the median is 107. Same window in today's PubChem ≈ 10^4 (MetFrag 2009 figure scaled by CID growth). candidate_databases_...md; q3 independently: median 27 at ±5 ppm, p90 193. High.
3. Formulas per window (our enumeration, CHNOPS+Cl, golden rules): at 300/500/800 Da and 5 ppm, 21/175/829 de-novo formulas; inside COCONUT the same windows contain median 2/2/1 formulas and the true formula covers 0.90/0.76/0.86 of window structures (10 ppm: 0.83/0.62/0.76). formula_assignment_...md (local computation + Kind & Fiehn 2007). High for COCONUT rows; medium for enumeration.
4. MS2-only formula top-1 accuracy: SIRIUS 5.6.3 48 % (NPLIB1), MIST-CF 74.1–76.9 % (NPLIB1), 50.5 % on MassSpecGym MCES-split (59–66.5 % when leaked), 82–84 % on CASMI 2022 positive; FIDDLE 90.6 % ≤ 800 Da but 64.2 % > 800 Da on NIST23 QTOF; external top-5 average FIDDLE 75.1 %, BUDDY 69.9 %, SIRIUS 69.3 %, MIST-CF 66.5 %. formula_assignment_...md (6 primary papers). High.
5. Pool-size penalty evidence: MetFrag2010 338 candidates → median rank 8 vs 2,508 → 31.5; Laudicina 0.112 → 0.079 with PubChem; gnshx +0.011 with gated PubChem top-50; nghianguyen7171 ungated analog enumeration +0.000. candidate_databases_...md, coverage_ceiling_...md (4 write-ups + 2 papers). Medium (write-ups are public LB numbers, ±0.01 noise per q7).
6. One-step analog reachability (our RDKit experiment, 15 operators on COCONUT): 38.7 % of random COCONUT structures and 82.8 % of the 250 enveda-np-examples have a one-step neighbour in COCONUT; 33 products/parent mean; window-conditioned analog pool median ~1,015 vs 35 direct = ~28× dilution. Five operators (dehydroxylation, sp3/aromatic hydroxylation, O-methylation, O-hexosylation, O-acetylation) give most of the reach at ~1/5 of the dilution. coverage_ceiling_...md, scratchpad calc/onestep.py. High for the measurement; medium for transfer to hidden test.
7. MRR decomposition anchors: Class 1 (reference spectrum exists) ≈ 16 % at channel MRR 0.917; Class 2 (structure in some DB) ≈ 45 % at 0.465; Class 3 ≈ 39 % at 0 → 0.356, matching gnshx's "single pool ceiling ≈ 0.36". e = 0.758 is plausible: CSI:FingerID on a biodatabase gives E[1/rank | found] ≈ 0.80. coverage_ceiling_...md. Medium.
8. Analog-retrieval sensor numbers (MS2Query benchmark, 302,514 GNPS spectra): at 35 % recall the best modified-cosine analog averages Tanimoto 0.45, MS2Query 0.63; Spec2Vec finds a Tanimoto > 0.6 hit in the top-10 for ~60 % of unknown queries (chance ≈ 1 %); MS2DeepScore RMSE ≈ 0.15 on the Tanimoto scale; 30 % of entropy-similarity false positives are one-bond isomers; only 62 % of single-modification pairs show shifted peaks (ModiFinder). No paper reports MRR of "nearest analog → candidates" on an MCES split. library_search_...md (12 papers). High for the numbers; the gap is real.
9. Ranking evidence: fusion of six fragmenters 82 vs best single 70 top-1 (CASMI 2016); MetFusion median rank 28 → 7; Bayesian fingerprint scoring +2.85 pp; MIST contrastive +5.7 pp; JESTR candidate-regularisation hurts on MassSpecGym by-mass (15.62 → 15.13); MVP consensus-spectrum 36.0 % vs 26.4 % single spectra; RRF(k = 60) +4–5 % over best system (Cormack 2009). Reference-count priors alone give 69 % top-1 on CASMI 2016 — poison here. No MS paper with a LambdaMART ablation could be opened. candidate_ranking_...md (14 papers). Medium.
10. COCONUT is not all natural: 28.5 % of structures have NP-likeness ≤ 0 (ZINC NP, SuperNatural II, UNPD synthetics); licences: COCONUT CC0, LOTUS CC0, NPAtlas CC BY-NC, SuperNatural/NPASS NC. candidate_databases_...md. High.

Forced analogies — what the search-and-rescue lens produced

The King asked whether the files already tested the Bayesian-search lens. Yes — three of five workers used it explicitly; two paid off, one half-paid, and one angle died.

• Paid off (coverage worker): Stone's discrete optimal-search theorem. Slot allocation = sort by calibrated posterior across channels, never by ranker of origin. The gnshx library-confidence gate is literally a posterior update after an unsuccessful search of the dense cell. It yields the insertion break-even table above and explains why ungated enumeration scored +0.000: extending the map 28× spends prior mass in every existing cell.
• Paid off (database worker): Koopman density, not map size. The "PubChem hurts" result becomes a ~2,000× prior-density ratio and a scoring rule s' = s_ranker + λ·log(c_S/n_S) + μ·log(1+k), with the prediction that μ (multi-database attestation) helps on known-structure validation but is ≈ 0 or negative on never-published hidden molecules.
• Half-paid (formula worker): datum fixing. Holds for hard filtering (a wrong formula displaces the whole posterior → break-even P ≈ 0.75–0.9); breaks for soft weighting, because search theory integrates over the datum distribution — which is just "multiply by P(formula)".
• Half-paid (analog worker): sweep width per sensor. Peak-alignment sensors (entropy/modified cosine) and Tanimoto-regression sensors (MS2DeepScore 2.0, DreaMS) have the least-overlapping sweeps; MS2Query's 0.45 → 0.63 ablation is the published evidence. Breaks where Koopman assumes one target: several near-isomers sit inside the sweep and only the exact one scores.
• Died (ranking worker): multi-sensor log-LR summation with independence. Rankers share one spectrum, so independence fails; q5's simulation confirms rank-averaging is worse than the best single scorer once correlation ρ ≥ 0.3. Fix is a jointly fitted stacker (LambdaMART over log-LR features), not naive summation.
• Structural breaks noted: no per-cell feedback within one submission (sequential optimality collapses to one-shot top-k); the target may be in no cell (Class 3); reward is 1/rank, so ordering in the top 5 dominates membership beyond slot 10.

The one testable MRR@25 experiment: on the three-mask enveda-np-examples split (plus a larger masked-COCONUT set, MCES ≥ 10 from train), submit the union of {library channel, COCONUT ranker channel, gated one-step analog channel (5 operators), gated PubChem top-50} ordered by isotonic-calibrated posterior P(correct), versus the same channels under a fixed 15/7/3 quota and versus the COCONUT ranker alone. Report (c, r, e) per class. Prediction: posterior ordering beats quota by +0.01–0.03 and beats single pool by +0.03–0.05; ungated analog union is ≤ single pool.

Contradictions within my subdomain

1. enveda-np-examples ∩ COCONUT: 99.6 % (assumed by my coverage worker) vs "8 of 250" (q2 reports write-ups swinging between the two depending on keying). Every c-estimate hangs on which is right. Asked q2 to recompute with the metric's RDKit pipeline.
2. Analog reach 82.8 % (the 250 NPs) vs 38.7 % (random COCONUT) vs q3's 95.7 % formula-level "parent" figure. Different definitions, same direction: well-studied NPs sit in dense map regions; the hidden test's unpublished molecules probably do not. Treat 82.8 % as an upper bound by ~2×.
3. NPAtlas gain: my worker ~0 vs q3's +138 IK14. Immaterial (Δc < 0.001) but shows dedup pipelines differ slightly.
4. Formula selectivity: "70.8 % of in-window candidates share the true formula" (one write-up) vs our 62–90 % by mass bin. Consistent in range; the write-up's pool included train structures.
5. MassSpecGym inversion: formula-restricted candidate sets are harder than mass-window sets (MIST 9.57 vs 14.64 % hit@1) because same-formula isomers are closer decoys — so published hit@1 figures on 256-candidate pools do not transfer to our 36-median windows; only deltas do.

Gotchas for the modeler (what could silently fool us)

• Reference-count and "in training set" features put 143/208 CASMI 2016 challenges at rank 1 and will push a never-published true answer down. Exclude them; validate structure-disjoint.
• Train structures dilute the pool. enveda-180 is Enamine HLL-200 chemistry (51 of 182,941 in COCONUT); adding train IK14s raises the window median 36 → ~56 with ~zero coverage gain for novel NPs. Keep them only for the Class-1 channel with a per-source prior.
• Known-NP validation overstates both c and e. Any gain measured on enveda-np-examples is an upper bound (dense-region bias, ~2×).
• Tautomer canonicalisation is slow on polyphenols (180–360 ms, hits maxTransforms per q7): dedup pools on plain IK14, canonicalise only the final 25; the exposure is de-novo/enumerated SMILES, not database candidates.
• Mass-stratify formula decisions: formula top-1 ≈ 0.9 below 500 Da for [M+H]+ but 0.4–0.65 above 800 Da; the 245–460 m/z test band is both the formula-easy and the window-dense region (300–400 Da median 64).
• Adducts are hypotheses: build H↔Na-swapped (Δ 21.981944 Da) and NH4/NH3-ambiguous neutral masses and admit them at low prior in slots ~20–25 when the primary top score is weak.
• Spectrum-blind baselines (q5): PubChem default order scores 49.98 % hit@1 on MassSpecGym; always report a blind baseline next to any local score.
• COCONUT formula artefacts (C6H5N3O window of 1,761) and 38,771 revoked rows must be fixed/dropped before any per-formula statistic.

Questions for other queens

• q2_data_forensics: exact IK14 overlap of the 250 np-examples and of all 275,810 train structures with COCONUT under RDKit 2026.03.3 canonicalisation; p99 of precursor_error_ppm on timsTOF rows after the ≤ 20 ppm cleaning (to choose ±5 vs ±10 ppm).
• q3_natural_product_chemistry: which NPClassifier classes populate the dense tail of the 245–460 m/z windows (sesquiterpene/diterpene formulas with 600–1,300 isomers), so analog enumeration can be class-gated.
• q1_instrument_physics: can the precursor residual in the MS2 peak list give a per-query ppm offset ("candidate-conditional self-calibration") reliable enough to shrink the window to ±5 ppm?
• q5_methods_lineage: does any open implementation of CSI:FingerID-style Bayesian scoring or MIST run offline within Kaggle-kernel limits, so the "contrastive + fingerprint" channel is feasible rather than aspirational?
• q7_compute_execution: does one-step enumeration from in-window parents (~1,050 candidates/query × ~1,500 queries) fit the kernel time budget with fingerprint scoring?

Layman sentences

1. All the world's known natural-product structures fit in one list of about 480,000 entries, and for any measured molecule only a few dozen of them have the right weight — so the hard part is choosing among look-alikes, not finding the list.
2. Roughly four in ten test molecules are in no list at all; the only way to score on those is to guess "a close relative of something we know", and guessing too freely buries the right answer under thousands of wrong relatives.
3. The best strategy is the one used to find a lost aircraft at sea: rank every possible location by how likely it is, search the likeliest cells first, and only widen the search when the likely cells come up empty.

Gaps I could not fill

• No published MRR/recall of "nearest spectral analog → candidate structures" on a structure-disjoint benchmark; MassSpecGym ran no library-similarity baseline.
• No MS paper with a LambdaMART/XGBoost learning-to-rank ablation could be opened; LTR recommendation rests on IR evidence (Cormack 2009) plus MS fusion results.
• Dührkop 2015 PNAS (CSI:FingerID) and Bittremieux 2022 (modified-cosine precision by modification type) could not be opened (HTTP 500/403); numbers taken from secondary sources.
• The c-term is back-solved from public LB write-ups, not measured; direct measurement waits on q2's overlap computation.
• Class-3 reachability for novel-looking molecules (MCES ≥ 10 from everything in train) is unmeasured; the 38.7 % vs 82.8 % spread is the largest uncertainty in the whole subdomain.