Coverage ceiling: decomposing MRR@25 into pool-recall x in-pool-recall@25 x E[1/rank], and what Bayesian search theory says about spending 25 slots

TL;DR. Writing MRR@25 = c × r × e (c = P(truth in pool), r = P(rank ≤ 25 | in pool), e = E[1/rank | rank ≤ 25]), the public write-ups pin the LB-0.417 regime at roughly c ≈ 0.55–0.61, r ≈ 0.90, e ≈ 0.76 — i.e. the ranker already puts ~60–70 % of findable answers at rank 1 and the binding term is c, with ~0.4 of headroom versus ~0.1 for r and ~0.24 for e. Stone's optimal-search theorem (AF447 search, Statistical Science 2014) answers the slot question exactly: order the 25 slots by calibrated posterior P(candidate = truth), regardless of which ranker produced the candidate; a fixed "15/7/3" split is only a crude approximation of that, and inserting a second ranker's candidate above slot ~5 needs P(correct) > 0.006–0.03, while appending it at slots 11–25 costs ≈ 0.0001–0.0003 MRR per slot (nearly free). A direct measurement on COCONUT (this work) shows one-step analog enumeration can reach ~39 % of random COCONUT structures (83 % of the 250 enveda-np-examples) but inflates the ±10 ppm pool ~28× (median 35 → ~1,050), which is why ungated enumeration scored +0.000 on the board; it pays only as a gated second search phase.

1. Estimating the three terms for the known approaches

Definitions. For one hidden molecule: c = P(its RDKit-2026.03.3 tautomer-canonical InChIKey14 is in the submitted candidate universe ∩ mass window); r = P(final rank ≤ 25 | in pool); e = E[1/rank | rank ≤ 25]. MRR@25 = c·r·e exactly (law of total expectation; molecules outside the pool contribute 0). recall@25 = c·r.

Public anchors (all from GitHub mirrors of Kaggle write-ups; Kaggle itself is a JS shell):

Algebra for the brief's anchor: 0.417 = c·r·e with c·r = 0.55 ⇒ e = 0.758; with c = 0.61 ⇒ r = 0.90. Alternative reading: if class 3 is unreachable, 0.417 needs the class-2 channel at MRR = (0.417 − 0.16×0.917)/0.45 = 0.60 (vs nghianguyen7171's 0.465) and 0.471 needs 0.72 — or, holding class 2 at 0.465, a class-3 channel at 0.29. Either way the conditional ranking is already in the CSI:FingerID-biodatabase band and the open 0.39 of probability mass is the only large lever.

Which term binds. Headroom (max gain holding the others fixed): c: (1 − 0.61)×0.90×0.76 = +0.27; e: 0.61×0.90×(1 − 0.76) = +0.13; r: 0.61×0.10×0.76 = +0.05. Per-unit slopes are similar (∂MRR/∂c = r·e = 0.68; ∂MRR/∂e = c·r = 0.55), so c wins on headroom, not slope — and every way of raising c lowers e (§4–6).

2. Is e = 0.758 plausible? Published rank curves say yes

e is fixed by P(rank 1 | found) plus the tail shape. Fitting a geometric tail to published top-k curves (script in scratchpad calc/mrr.py; tail ratio 0.5–0.7):

So e = 0.758 ⇔ P(rank 1 | found) ≈ 0.6–0.65: what a CSI:FingerID-class ranker achieves on a ~0.5 M biodatabase with 36–107 isobars per query (sibling: median 36, p90 204 in IK14-deduplicated COCONUT at ±10 ppm), and not what a MassSpecGym-class FFN achieves (e ≈ 0.4 — the 0.11–0.25 LB tier). The library channel (class 1, MRR 0.917 ⇒ ~88 % rank 1) pulls the blend up. At 0.417 most found answers are at rank 1–2; the board is coverage-limited, not ordering-limited.

3. Bayesian search theory, the actual machinery (AF447) and the mapping

From Stone, Keller, Kratzke & Strumpfer, Statistical Science 29(1):69–80, 2014 (arXiv 1405.4720), read in full:

• Prior = scenario mixture. P = p₁D₁ + p₂D₂ + p₃D₃, subjective weights p₁ = p₂ = 0.35, p₃ = 0.30: D₁ uniform in the 40 NM circle, D₂ circular normal σ = 8 NM (nine loss-of-control crashes), D₃ reverse-drift of recovered bodies (current noise σ = 0.22 kt, wind 2.0 kt). N = 75,000 particles. Their retrospective: D₃ should have been a likelihood multiplying 0.5D₁ + 0.5D₂, not a mixture component.
• Objective. Cell j, effort z, detection b_j(z); maximise P(Z) = Σ b_j(z_j)p(j) s.t. Σ z_j ≤ T. Koopman's 1946 law b(z) = 1 − e^{−wz}; the optimum equalises marginal detection rate p(j)·b′_j(z_j) across searched cells (Stone 1975).
• Posterior after unsuccessful search (eq. 4.1). w̃ₙ = (1 − p_d(n))wₙ / Σₙ′(1 − p_d(n′))wₙ′; grid form: searched cell p′ = p(1−q)/(1−pq), others r′ = r/(1−pq).
• PODs used. Surface search: 1 − p_d = 0.7 + 0.3q(n) (only 30 % chance the search was effective). Towed pinger locator: POD capped at 0.9 within 1,730 m ("manufacturers' specifications and operator estimates tend to be optimistic"); beacon survival 0.8 each ⇒ 0.92 independent / 0.72 dependent, blended 0.25/0.75 ⇒ P_D = 0.77. Side-scan sonar: 0.90 inside searched rectangles, 0 outside (eq. 4.2). Four sequential updates P̃₁ → P̃₁₂₃₄.
• Outcome. The wreck lay inside the acoustically "searched" area; the recovered beacon was dead — both had failed. The posterior assuming beacon failure (their Fig. 8) placed it in a high-probability cell; the 2011 search "commenced in the center of the distribution and quickly found the wreckage". Lesson: model the sensor-failure hypothesis jointly; never trust a POD above 0.9.

USS Scorpion (Richardson & Stone 1971, NRLQ — not fetchable; Wikipedia, secondary): commander-elicited scenario priors, depth-dependent detection grid, hull found ~740 km SW of the Azores at > 3,000 m, October 1968. The greedy rule "search the cell maximising p_i a_i / c_i" is a Gittins-index special case.

Mapping to CASMI (one row per piece of machinery):

The one place the aircraft analogy must be extended rather than copied: the ocean search has feedback (a cell searched unsuccessfully is updated before the next cell). A CASMI submission has none within a molecule — all 25 cells are searched at once — so the sequential-update machinery applies only across channels in the pipeline (library miss ⇒ reweight classes) and across days on the leaderboard (5 probes/day, aggregate signal only).

4. Slot allocation: the theorem, the toy, the inequality

Theorem (discrete Stone, applied to MRR). With calibrated posteriors πᵢ = P(i = truth | spectrum), expected MRR of an ordering σ is Σᵢ πᵢ/σ(i), maximised (rearrangement inequality) by sorting on πᵢ — regardless of which ranker produced i. "Top-25 of one ranker" is optimal iff its score is monotone in πᵢ and its universe holds everything with non-negligible πᵢ; it fails when (a) another channel proposes candidates with πᵢ above the first ranker's k-th (coverage) or (b) scores are miscalibrated across sub-populations (Laudicina's "68 % of top-1 guesses were PubChem-only" at LB 0.079).

Toy case (calc/mrr.py). Ranker A (fingerprint on the DB pool): c = 0.61; conditional ranks p₁ = 0.62, p₂ = 0.11, p₃ = 0.05, p₄ = 0.03, p₅ = 0.02, 0.012 each for 6–10, 0.004 each for 11–25 (e = 0.78, the §2 regime). Ranker B (orthogonal: enumeration / class-prior draw) acts on the 0.39 out-of-pool molecules, contains the truth with g = 0.15, ranks it p₁ = 0.30, p₂ = 0.18, p₃ = 0.12, p₄ = 0.08, 0.06, 0.05, 0.04, remainder over 8–25.

Marginal probabilities behind the optimum: P(A's k-th is correct) = 0.378, 0.067, 0.031, 0.018, 0.012, 0.007 for k = 1..6; P(B's j-th is correct) = 0.0175, 0.0105, 0.0070, 0.0047 for j = 1..4. B1 belongs at slot 5, not slot 2 and not slot 23.

The inequality. Inserting one foreign candidate at slot s, shifting A's slots s..24 down, costs Δ(s) = Σ_{k=s}^{24} c·p_A(k)·(1/k − 1/(k+1)) + c·p_A(25)/25 and gains π_B/s. It pays iff π_B > s·Δ(s):

So: a foreign candidate needs ≥ 20 % posterior to displace A's top-1, ≥ 3 % for slot 2, ≥ 0.6 % for slot 5, ≥ 0.25 % for a tail slot. A B-first block of 3 costs 0.31 MRR. A fixed "15/7/3" split is safe only as a tail allocation (slots ≥ 11, cost ≤ 0.0003 per wrong slot); above slot ~5 the decision must come from calibrated probabilities. The best fusion gains only +0.004 over A-alone here because g = 0.15; the gain is linear in g, so coverage of the second channel, not the split ratio, is what matters.

Calibration recipe: per-channel isotonic score → P(correct) on the three-mask split (§5; the Class-3 mask calibrates B), merge by probability, dedupe by IK14. The gnshx lib_max gate then emerges automatically: a confident library hit has π ≈ 0.9 and nothing from PubChem reaches 0.2.

5. Measurement protocol: a held-out set that mimics the hidden test

Hidden test: ~400 NPs, timsTOF, "none previously published", 1–16 spectra each (median 3). Nothing public has all three properties, so the protocol must simulate novelty by masking.

(a) MassSpecGym MCES-disjoint split (arXiv 2410.23326, §2.5, Table 2, Algorithm 1). Pairwise myopic-MCES distances (bond-order-weighted edge edit distance, threshold 10, always_stronger_bound off), then AgglomerativeClustering(metric="precomputed", linkage="single", distance_threshold=10) so no molecule in one fold is within MCES < 10 of any molecule in another; clusters feed StratifiedGroupKFold on adduct × CE × instrument × spectra-per-molecule. Folds: train 194,119 spectra / 25,046 molecules / 3,061 clusters; val 19,429 / 3,386 / 2,221; test 17,556 / 3,170 / 2,202. Candidates: true molecule + molecules within ±10 ppm (or same formula) from 1 M bio → 4 M diverse → 118 M PubChem until |C| = 256 (87 % / 66 % hit the cap). Use the split procedure, not the numbers: MCES-10 is the right definition of "novel" for a Class-3 simulation; the spectra are not timsTOF and the candidate lists are random isobars.

(b) Three-mask protocol on enveda-np-examples (shiferaxa). 250 host-measured timsTOF structures (1,184 spectra; 249/250 in COCONUT at IK14 — verified here). Masks: (1) all visible = Class 1; (2) own spectra removed from train = Class 2; (3) structure also removed from the pool = Class 3. shiferaxa: "the only split that has predicted the public leaderboard"; random train hold-outs "predicted the opposite ordering of three rankers". Harden masks 2/3 by also dropping train spectra of anything with MCES < 10 to the target, or analog propagation leaks.

(c) GNPS deposits after a cutoff — correct idea; no count verifiable this session (search budget exhausted).

Computation. With k(m) = RDKit-2026.03.3 tautomer-canonical InChIKey14 (COCONUT's stored keys are not tautomer-canonical): recall@pool = mean 1[k(m) ∈ {k(x): x ∈ pool, |mass(x) − M(m)| ≤ 10 ppm}]; recall@25 = mean 1[rank ≤ 25]; MRR = mean 1[rank ≤ 25]/rank; report c, r = recall@25/recall@pool, e = MRR/recall@25 per class, blend with the class prior (0.16/0.45/0.39) to predict LB. 250 molecules ⇒ ±0.06 binomial half-width on c, so differences < 0.05 are noise; bootstrap, and use a larger spectrum-free masked COCONUT set for pool-side metrics.

6. Pool-expansion ideas, with cost

New measurement (this work, scratchpad calc/onestep.py, RDKit 2026.03.6, COCONUT CSV-lite Oct 2026, 479,690 IK14): 15 single-step RDKit reaction operators (sp3/aromatic hydroxylation, dehydroxylation, O-methyl/demethyl, ±H₂, O-acetyl/deacetyl, O-hexosyl/dehexosyl, C-prenyl/deprenyl, ketone↔alcohol), applied to 2,947 random COCONUT structures and to the 250 enveda-np-examples; a structure is "one-step reachable" if any product's IK14 is another COCONUT key.

Operator-level reachability on enveda-np-examples (of 250): dehydroxylate 113, sp3-hydroxylate 106, aromatic hydroxylate 83, O-methylate 83, O-hexosylate 79, O-acetylate 61, dehydrogenate 53, O-demethylate 47, hydrogenate 34, dehexosylate 26, alcohol→ketone 25, prenylate 22, ketone→alcohol 19, deacetylate 8, deprenylate 4. Oxidation-state and O-methyl/glycosyl edits dominate; five operators capture most of the reach at a fifth of the dilution.

Forced analogy: Bayesian search theory (Koopman / Stone / SAROPS)

Statement. The hidden molecule is a stationary target in a discrete space; each candidate structure is a cell with prior p(j) (database-membership density × class prior); each ranker is a sensor with a cell-dependent POD; the 25 slots are a fixed search budget with slot-dependent reward 1/s; and a "search increment" that returns nothing (library miss) updates the cell probabilities by w̃ ∝ (1 − p_d)w.

Testable derivations. (1) Posterior-sorted fusion (§4): on the three-mask split it beats any fixed k/m split by ≥ 0.003 and never loses; B-first loses ≥ 0.2. (2) Gate thresholds are derivable, not tuned: a library hit blocks foreign candidates from slot 1 iff π ≥ 0.2, from slot 2 iff ≥ 0.03 — so gnshx's lib_max = 0.7 is about right for slot 1 and far too strict for slots ≥ 3, where PubChem/analog candidates should be admitted behind a confident library hit (+0.005–0.01 LB expected). (3) Sensor-failure hypothesis: AF447's wreck was in a "searched" cell because the beacons had failed; the CASMI analogue is a Class-2 molecule whose timsTOF merged-CE spectrum the ranker cannot read. Prediction: the residual in-pool misses (1 − r ≈ 0.10) cluster by CE-merge pattern and adduct, and an instrument-conditioned second ranker on that subset beats any pool growth. (4) Across-day update: treat each of the 5 daily probes (±0.025) as an aggregate unsuccessful search and update class-prior weights, not per-molecule guesses.

Does it hold? Partially. Holds: sort-by-posterior allocation, posterior update after a channel miss, scenario-mixture prior, POD cap. Breaks: (a) no per-cell feedback inside a submission — all 25 cells are searched at once, so sequential optimality collapses to one-shot top-k selection; (b) the target may be in no cell (Class 3, ~39 %) — the fix is map extension, which costs prior mass in every existing cell (the 28× dilution above); (c) cells are not independent — near-isomers share fingerprint bits, so the effective cell count is < n and rank grows sub-linearly with pool size (MetFrag 2010: 7.4× candidates → 3.9× median rank); (d) reward is 1/s, not 1, so ordering inside the top 5 dominates membership beyond slot 10.

Gotchas / what could fool us

1. The class shares (0.16/0.45/0.39) are one team's estimate; "recall@25 ≈ 0.55" is back-solved from them. kabir0774 claims the opposite ("essentially every real test molecule is already in the curated pool") while scoring 0.245 against a local 0.445 — if true, the gap is pure ranking. Their inference (two submissions' implied reachable share) is weaker than gnshx's and nghianguyen7171's direct coverage analysis, but nobody has measured c; hold c ∈ [0.55, 0.75] until a Class-3 mask on timsTOF NPs is run.
2. Local MRR on known NPs overstates e and c together. enveda-np-examples: 99.6 % in COCONUT, 82.8 % with a one-step COCONUT neighbour vs 38.7 % for random COCONUT structures. Well-studied NPs sit in dense map regions; never-published ones do not. Any analog-propagation or enumeration gain measured on enveda-np-examples is an upper bound, plausibly by 2×.
3. E[1/rank] is a rank-1 statistic. Rank 2 → 1 is worth 0.5; rank 26 → 25 is worth 0.04. A coverage intervention that costs rank-1 hits loses unless its own hits land in the top 3.
4. Pool changes recalibrate the ranker. Laudicina's 68 % PubChem-only top-1 picks: scores are not comparable across sources; a channel calibrated on COCONUT decoys is over-confident on PubChem or enumerated near-isomers. Calibrate each channel on the pool it will face.
5. Tautomer/IK14 mismatch lowers measured c silently. COCONUT's standard_inchi_key is not tautomer-canonical; compute recall@pool with the metric's exact RDKit pipeline (this session had 2026.03.6, not 2026.03.3).
6. MassSpecGym numbers do not transfer. Candidates are 256 random isobars (random Hit@1 = 0.37 %); formula-bonus Hit@1 (9.57 %) is lower than mass-based (14.64 %) because isomers are harder decoys; PubChem default ordering alone scores ~50 % Hit@1 there (Liu et al. 2026, via sibling) — popularity priors may invert on never-published NPs.
7. Binomial noise. 400 hidden molecules ⇒ ±0.025 per probe; 250 validation molecules ⇒ ±0.06 on c. The toy's 0.004 fusion gain is invisible on the board; measure slot policies on thousands of masked molecules.
8. Dedup level changes every count: 36 (IK14) vs 107 (stereo-expanded) per query; the 28× dilution is relative to IK14.
9. AF447's warning: operator-estimated PODs are optimistic; cap any ranker's self-reported confidence at 0.9 before using it as a gate.

Implications for the challenge

1. Attack c first, but only with gated, query-conditioned extensions: phase 1 library (Class 1), phase 2 COCONUT ∪ train fingerprint ranker, phase 3 (only if phase-2 top-1 posterior < threshold) analog enumeration from in-window parents using the five dominant operators (dehydroxylation, hydroxylation, O-methylation, O-hexosylation, O-acetylation) and/or PubChem top-50; expect +0.03–0.05 from phase 3, not +0.2.
2. Fuse by calibrated posterior, not by ranker quota. Fit isotonic score→P(correct) per channel on the three-mask split; order the union by probability; dedupe by canonical IK14. Fixed splits (15/7/3) are acceptable only for slots ≥ 11.
3. Use the break-even table: foreign candidate into slot 1 needs P ≥ 0.2, slot 2 ≥ 0.03, slot 5 ≥ 0.006, tail ≥ 0.0025. Re-derive once real calibrated probabilities exist.
4. Implement the §5 protocol now: three-mask enveda-np-examples with MCES-10 leakage removal, reporting (c, r, e) per class; predict LB as the class-weighted blend and check the prediction against each probe.
5. Measure c directly on a second, larger masked set (COCONUT structures with their own key removed) to get the one-step reachability of novel-looking molecules (MCES ≥ 10 from everything in train), since the 38.7 % vs 82.8 % spread is the single largest uncertainty in the Δc estimate.
6. Treat the residual in-pool misses (1 − r ≈ 0.10) as a "sensor failure" subset: inspect them by adduct and CE-merge pattern before buying any pool growth.
7. Do not spend effort on LOTUS/NPAtlas/ChEBI/LIPID MAPS for coverage (Δc ≤ 0.001, measured +0.000).
8. Keep calc/onestep.py as the enumeration reference: 15 operators, 33 products/parent mean, 28× window dilution — any proposed enumeration scheme should report its own dilution factor against this baseline.

Open questions

• What is c on the actual hidden set? Only Enveda knows; the best public proxy is the Class-3 mask on NP-like timsTOF spectra, which no write-up has reported with MCES leakage control.
• Are the hidden "analogs" one step from COCONUT (Enveda's platform isolates analog series) or genuinely new scaffolds? The reachability of enveda-np-examples (83 %) vs random COCONUT (39 %) brackets the answer.
• Per-channel calibration curves on the three-mask split — needed before the §4 theorem can be applied; no team has published them.
• Richardson & Stone 1971 and Koopman 1946 could not be opened this session (search budget exhausted; Wiley/HathiTrust access); AF447 facts are primary, Scorpion facts are secondary (Wikipedia).
• GNPS post-cutoff NP deposits: count and timsTOF share unknown.

Sources

• Stone L. D., Keller C. M., Kratzke T. M., Strumpfer J. P., "Search for the Wreckage of Air France Flight AF 447", Statistical Science 29(1):69–80 (2014), DOI 10.1214/13-STS420, arXiv 1405.4720 — https://arxiv.org/pdf/1405.4720 (full text read; eqs. 3.1, 4.1, 4.2, PODs 0.77/0.9/0.7+0.3q)
• Bayesian search theory (Scorpion, Koopman 1946, Stone 1975, Gittins) — https://en.wikipedia.org/wiki/Bayesian_search_theory (secondary; Richardson & Stone 1971 NRLQ and Koopman OEG Report 56 not opened)
• Bushuiev R. et al., MassSpecGym, NeurIPS 2024, arXiv 2410.23326 — https://arxiv.org/pdf/2410.23326 and https://arxiv.org/html/2410.23326 (§2.5 split, Table 2, Table 3, Algorithm 1, Fig. 5)
• Dührkop K. et al., "Searching molecular structure databases with tandem mass spectra using CSI:FingerID", PNAS 112:12580 (2015), DOI 10.1073/pnas.1509788112 — https://pmc.ncbi.nlm.nih.gov/articles/PMC4611636/ (Fig 2A/2B, S2, S4, S6 top-k rates)
• Sibling worker file: research/q6_candidate_space_retrieval/candidate_databases_pubchem_coconut_lotus_npatlas_chembl_coverage_math.md (COCONUT IK14 counts, window statistics, density arithmetic)
• Kaggle write-ups via GitHub mirrors (Kaggle pages are JS shells): https://github.com/corentinlaudicina/enveda-casmi-2026 ; https://github.com/nghianguyen7171/casmi26 ; https://github.com/gnshx/Enveda-CASMI-2026---Molecule-ID-From-Mass-Spectra ; https://github.com/shiferaxa/casmi26 ; https://github.com/kabir0774/Enveda-Casmi-2026 ; https://github.com/AayanShabbir/casmi26 ; https://github.com/billylu24/Enveda-CASMI-2026---Molecule-ID-From-Mass-Spectra ; https://github.com/OfficialBhattacharya/kaggle_CASMI_2026 ; https://github.com/yasumorishima/kaggle-competitions/pull/32 (LB 0.275; "real class-2 score is about 0.47")
• This work: scratchpad/calc/mrr.py, calc/insert.py (decomposition, toy allocation, insertion costs); calc/onestep.py + onestep_out.json (one-step reachability and dilution on COCONUT CSV-lite Oct-2026 at C:\Temp\casmi26\npdb\coconut\ and train.parquet enveda-np-examples; RDKit 2026.03.6)