Sibling-project lessons translated to CASMI26

TL;DR. Two prior structure-prediction challenges run on this machine converged on one result: the model's own confidence does not rank its own candidates (within-item Spearman ρ ≈ −0.03, worse than random selection), while agreement across ≥4 genuinely distinct engines does (+0.0380, p = 3.4 × 10⁻⁶) — provided you deduplicate, because 39 % of the "independent" opinions were copies. For CASMI26 the same shape applies: the published metabolite-annotation literature shows top-1 accuracy roughly halving when the candidate pool grows from a biological database to PubChem (CSI:FingerID 68.5 % → 34.4 %; MetFrag fragmentation-only 22 % → 6 %), and a random ranker's MRR@25 falls exactly as H₂₅/n = 3.816/n, so pool size is a first-order multiplier on the score, not a detail. The fusion rule to use is quality-weighted Reciprocal Rank Fusion, not Borda — and Cormack's own data shows CombMNZ is a live competitor, so it must be measured, not assumed.

1. Translation table — every sibling lesson → a CASMI26 rule

Also at research/q7_compute_execution/sibling_project_lessons_consensus_confidence_poolsize.json (20 rows, array-of-objects). Abbreviated here; the JSON carries the full test_here text.

2. Pool-size sweet spot — what the published numbers actually say

The arithmetic first, because it bounds everything. If the truth sits at a uniformly random rank among n candidates and the ranker has no skill, then for n ≥ 25, E[MRR@25] = H₂₅/n where H₂₅ = 3.815958. Exactly:

(Derived here, not cited.) The public-LB rank-1 of ≈ 0.471 is therefore ~123× a zero-skill ranker on a 1,000-candidate pool. Because the relationship is 1/n, halving the pool doubles the no-skill component, and the marginal damage of adding candidates is largest when the pool is already small.

Published empirical anchors. No paper I opened publishes a clean accuracy-vs-candidate-count curve at 10 / 10² / 10⁴. What exists is database-to-database contrasts and one binned analysis:

Three things follow. First, the halving is real: CSI:FingerID's top-1 goes 68.5 % → 34.4 % and MetFrag's goes 22 % → 6 % when the pool grows. Second, it is not purely a size effect: ChemSpider is only a few-fold smaller than PubChem yet MetFrag's fragmentation-only top-1 is 3.7× higher, so content composition (how chemically plausible the decoys are) matters as much as count — which is exactly the eyewitness "fair fillers" problem in §4. Third, and most important as a gotcha, the one study that binned by actual candidate count found the relationship is NOT monotonic. CASMI 2016 reports verbatim: "Fewer Top 1 entries and higher median/mean ranks were observed for the challenges with moderate candidate numbers (200–1000 candidates); lower median ranks and more Top 1 entries were observed for lower and higher candidate numbers", and twice states "candidate numbers are not the only driver of performance." Per-cluster means were black 611 / green 1,603 / blue 1,019 / red 380 (negative) and magenta 5,297 / cyan 1,029 / red 886 / black 1,534 / blue 978 / green 722 (positive). The plausible mechanism: very large pools arise for large, heteroatom-rich formulas whose MS2 spectra are also more informative, so formula size confounds pool size.

CASMI 2016 category top-1s on ~1,114-candidate pools: Category 2 (in-silico only) CSI:FID 34 % (70/208), CSI:IOKR_A 30 % (62/208), MS-FINDER 22 % (46/208), CFM_orig top-10 59 % (123/208); Category 3 (metadata allowed) MS-FINDER+MD 70 % top-1, CFM_retrain+DB 58 % top-1, CFM top-10 87 %. The metadata jump (34 % → 70 %) is bigger than any model improvement in the table — and MetFrag's 6 % → 71 % with retention time + references says it twice. We have no retention time, but we do have adduct, polarity, collision energy and a declared natural-product provenance; our analogue of "metadata" is any prior that reweights the pool (NP-likeness, biosynthetic plausibility, taxonomic co-occurrence).

Sweet spot. The evidence supports the smallest pool whose recall@25 you can still prove, not a fixed number. The operational rule: sweep the ppm window and the database union, and plot recall@25 against mean pool size; stop at the knee where recall@25 stops rising faster than 1/n is falling.

3. Rank aggregation — which fusion rule for MRR@25

I read the primary paper in full. Cormack, Clarke & Büttcher, Reciprocal Rank Fusion outperforms Condorcet and individual Rank Learning Methods, SIGIR '09, pp. 758–759:

RRFscore(d ∈ D) = Σ_{r∈R} 1/(k + r(d)), where k = 60 was fixed during a pilot investigation and not altered during subsequent validation.

Their stated intuition is precisely our problem: "while highly-ranked documents are more important, the importance of lower-ranked documents does not vanish as it would were, say, an exponential function used. The constant k mitigates the impact of high rankings by outlier systems."

Reported numbers:

Sensitivity to k (pilot, TREC topics 351–400): MAP .2072 at k=0, .2134 at k=20, .2145 at k=60, .2147 at k=80, .2098 at k=500 — flat across 10–100, so k is not a tuning knob worth spending on. Significance: RRF beat Condorcet 7/7 (p ≈ 0.008) and CombMNZ 6/7 (p ≈ 0.04); "RRF outperforms Condorcet, CombMNZ and the best system by 4 % to 5 % on average."

Recommendation: quality-weighted RRF as the default, with CombSUM-on-within-molecule-z-scores and CombMNZ as mandatory comparators. The reasoning, in order of weight:

1. Scale immunity. Our scorers emit incommensurable quantities — a modified cosine in [0,1], a MetFrag/CFM-ID explained-intensity score, a Tanimoto to a predicted fingerprint, a formula-agreement count. Cormack names exactly this as CombMNZ's weakness: it "multiplies the sum of the uncalibrated scores… It is perhaps not surprising that its results have higher variance… by happenstance, some scores are more amenable than others." The sibling's z-hybrid solved the same problem by standardising within the item before summing — CombSUM on z-scores, the defensible score-level alternative.
2. Objective alignment. The competition metric is reciprocal rank. RRF optimises a surrogate of the thing being scored; Borda and CombMNZ do not.
3. Why Borda lost in PXR, mechanistically. Borda weights linearly in rank. On a 1,000-candidate pool with 25 output slots, one engine burying the truth at rank 400 subtracts 400 units and vetoes it, while the gap between rank 600 and rank 900 — information we will never use — carries 300 units. RRF's 1/(k+r) collapses everything past rank ~100 to near-zero, the correct shape when only the top 25 score.
4. Honest caveat: RRF was not uniformly best in its own paper. CombMNZ beat it on TREC 3 (.4381 vs .4350) and on LETOR 3 (.6107 vs .6051, p ≈ .2). Cormack's claim is "consistently equaled or bettered," not "always beat." Default to weighted RRF, but run the five-way bake-off (table row 16) with a permutation CI before fixing it.
5. Weighting. Sibling row 6 (esmfold2 halving the gain) says an unweighted fusion is actively dangerous. Use RRFscore = Σ_e w_e/(k + r_e(d)) with w_e from each engine's standalone holdout MRR@25, admitting an engine only if leave-one-out removal costs more than the permutation floor.

I did not open Fox & Shaw (1994), the CombSUM/CombMNZ primary; the CombMNZ definition and numbers quoted above are from the RRF paper's own formulation, CMNZscore(d) = |{r : r(d) ≤ c}| · Σ_{r: r(d)≤c} s_r(d). Treat the Fox & Shaw attribution as unverified here.

4. Forced analogy: jurisprudence of eyewitness line-ups

The analogy, precisely. A candidate pool for one molecule_id is a police line-up. The true structure is the suspect; every other candidate is a filler. The ranking model is the eyewitness; the MS/MS spectrum is the witness's memory trace of the perpetrator. MRR@25 is "how far down the witness's ordered list the true suspect appears." The sibling finding "confidence ≠ accuracy" is the eyewitness literature's Phase 1 conclusion, and Wixted & Wells (2017, Psychol Sci Public Interest, doi:10.1177/1529100616686966) overturn it: under what they call pristine conditions, confidence is highly diagnostic. Their Box 3 lists five, verbatim:

1. Include Only One Suspect per Lineup — 2. The Suspect Should Not "Stand Out" in the Lineup — 3. Caution that the Offender Might Not Be in the Lineup — 4. Use Double-Blind Testing — 5. Collect a Confidence Statement at the Time of the Identification

Their numbers: in fair line-ups "high-confidence accuracy is very high (95 % to 100 % correct)"; a field study of real robbery cases yielded 72 high-confidence suspect IDs and 17 high-confidence filler IDs, implying ≈ 96 % correct, with a signal-detection fit estimating ≈ 97 %. Under unfair line-ups the same high-confidence bin falls to 88 %, 85 %, 80 %/75 %, and in one study ~66 % versus ~86 % in the matched fair condition. The Phase-1 pessimism came from the point-biserial correlation (Sporer et al. 1995 reported .41, which Wixted & Wells argue answers the wrong question — it conditions on accuracy to predict confidence, when the legal system needs the reverse) and from pooling choosers with non-choosers.

Our five pristine conditions, derived.

1. One suspect per line-up. Deduplicate the pool on tautomer-canonical InChIKey14. If the truth appears three times as three SMILES and the metric collapses them, our ranker's "votes" are split across identical suspects and the pool statistics are wrong. Sibling row 5 is the same failure in pose space (39 % duplicates).
2. The suspect must not stand out. This is the mock-witness test, and it is the single most valuable idea the analogy yields. Wells' test asks whether a non-witness, given only the verbal description, can pick the suspect. Our version: can a model that never sees the MS2 spectrum — using only precursor m/z, formula and adduct — pick the true structure out of the pool above chance? If yes, the pool is biased (e.g. COCONUT's representation skew, or candidate lists where the truth is the only "drug-like" entry among 900 combinatorial decoys), and our measured accuracy is partly line-up bias, not spectral evidence. This is directly testable on the holdout and costs almost nothing. The dud effect follows: Wixted & Wells cite Charman, Wells & Joy (2011) showing that adding obviously-wrong fillers increases confidence in a wrong-but-plausible pick. Padding pools with implausible structures will inflate softmax confidence without improving accuracy.
3. Caution that the offender might not be in the line-up. Our pools are all-suspect line-ups with no "not present" option — the model must emit 25 guesses whether or not the truth is in the pool, and with recall@25 ≈ 0.55, ~45 % of line-ups are target-absent. Wixted & Wells' framing is exact: "a lineup that contains only suspects (no fillers) is like a multiple-choice test with no 'wrong' answer." We need an abstention head — a scored estimate of P(truth ∈ pool) — even though the submission format forbids abstaining, because it is what routes a molecule to the expensive tail pipeline (row 17).
4. Double-blind administration. The administrator must not know the answer. Our violations are leakage: tuning the ppm window, the pool source or the fusion weights against the public LB; or validating on test.parquet, whose rows are verbatim train. The visible test set is a non-blind administrator.
5. Confidence at the initial identification. Wixted & Wells are emphatic that confidence expressed later, after feedback, is worthless. Our analogue: record the ranker's confidence from the first, un-reranked pass. A confidence computed after consensus reranking is contaminated by the other engines' opinions — the statistical equivalent of post-identification feedback — and will look better calibrated than it is.

Is cross-engine consensus = multiple independent witnesses? Partly, and the independence assumption is violated — worse here than in the sibling. Chai-1's errors correlated with Boltz-2's at ρ = +0.45 and Protenix's at +0.474. Our engines share more: library search, fingerprint-prediction retrieval and in-silico fragmentation are all parameterised, directly or indirectly, by the same public GNPS / MassBank / MoNA spectra and by the same structure databases. In eyewitness terms these are co-witnesses who discussed the case before testifying, and the law treats that as contamination for exactly the reason that applies here: four correlated witnesses are not four witnesses. The sibling's measurement that a weaker fourth engine drove the gain from +0.0380 to +0.0178 is the quantitative version of adding a contaminated co-witness.

Where it breaks — three places, honestly.

• The base rate is inverted. Line-ups mostly contain the culprit; our pools fail to contain the truth ~45 % of the time. Wixted & Wells' high-confidence diagnosticity depends on a base rate of guilt; they show high-confidence accuracy "stays high (above 90 %) until the base rate drops below 35 %." Ours may be below that. Their asymmetric result survives though, and it is the one we should bank: "No matter how good or how bad the eyewitness identification procedure is, a low-confidence ID implies that the ID is error prone." That matches sibling row 18 exactly — our ranker will be a good failure-detector and a poor fine-ranker.
• There is no memory decay or post-event contamination. A spectrum is a fixed measurement; a witness's memory degrades and can be overwritten. The whole "collect confidence at the initial ID" machinery maps only metaphorically.
• Fillers are not adversarially chosen in our case, they are whatever the database contains. Line-up fairness is a controlled design variable; pool fairness is an accident of COCONUT's composition. That makes the mock-witness test a diagnostic for us, not a fix: if the pool is unfair we cannot simply re-select fillers without changing recall@25.

Verdict: partially holds. The pristine-conditions framework translates into five concrete, cheap, testable preconditions and one genuinely new experiment (the mock-witness probe). The multiple-witness part of the analogy holds structurally but its independence premise is more violated here than in the sibling project, so the expected consensus gain should be discounted, not extrapolated.

5. Engineering reuse — absolute paths

All under D:\Users\ashenoy00000\.windsurf\OpenADMET-cyp-structure. Read-only; port patterns, do not modify.

Gotchas / what could fool us

1. The visible test set is a non-blind administrator. test.parquet rows are verbatim enveda-180 train rows. Any score computed on it is library-search performance and will be 3–5× the hidden score.
2. Pooled-vs-within confounding. A ranker whose within-molecule ρ is zero can show a strongly positive pooled correlation purely because easy molecules have small pools and high scores. The sibling measured exactly this (+0.05 pooled, −0.033 within).
3. Pool size and formula size are confounded. CASMI 2016's non-monotonic binning (worst performance at 200–1,000 candidates, better both above and below) is the smoking gun. Any "smaller pool is better" experiment must control for molecular formula, or it will measure chemistry, not pool size.
4. Tautomer canonicalisation is a silent zero. A correct structure written as the wrong tautomer, or canonicalised with a different RDKit build, scores 0. The organisers pin RDKit 2026.03.3; so must we, in both the local harness and the notebook.
5. Duplicate InChIKey14 inside one row's 25 wastes slots. Twenty-five SMILES that collapse to eleven distinct InChIKey14s is an eleven-guess submission. Dedup after canonicalisation, not before.
6. "Four engines" that are one engine. Fingerprint-prediction models trained on the same GNPS dump, and library search over the same library, are correlated witnesses. Measure the error correlation; report an effective witness count.
7. The dud effect. Padding the pool with implausible decoys raises the model's apparent confidence without raising accuracy — documented in the eyewitness literature, and structurally identical to softmax over an easy-negatives-dominated candidate set.
8. Best-of-N illusion. The sibling drew ~20 features and six of them cleared +0.010, all noise (null sd 0.0087). With 5 submissions/day for ~90 days, the public LB offers ~450 draws. Two final selections chosen by public LB rank is close to picking the best of 450 random draws.
9. A metric reimplementation that is "close enough." The sibling's own scorer paired atoms by (resnum, atom_name) while the organisers' used sequence alignment; the discrepancy created a bug class that cost whole ligands. Reimplement MRR@25 against the published definition, with unit tests on adversarial tautomers.
10. Consensus is a catastrophe detector, not a fine-ranker. Its measured value scaled with how bad the pool's worst members were (ρ −0.763 where 19.2 % of poses were catastrophic, ρ −0.258 where 0.2 % were). If our pools are uniformly mediocre, expect the small regime, not the large one.
11. Oracle gains that selection cannot collect. Adding a candidate source raised the sibling's oracle +0.0375 and its selected score −0.0017. Always report both.
12. Collision-energy merging. Hidden test spectra carry a list of collision energies (merged multi-CE). A holdout built from single-CE train spectra tests a different measurement object.

Implications for the challenge

1. Instrument every run with two numbers: recall@25 and MRR@25. Attribute every change to the coverage term or the ranking term. Do this before any modelling work.
2. Build the hidden-test-mimicking holdout first (timsTOF only, scaffold-disjoint, NP-like, 1–16 spectra/molecule median 3, the 7 test adducts, merged multi-CE). Nothing else is measurable without it.
3. Run the mock-witness probe. Train a model on precursor m/z + formula + adduct only, no MS2, and measure its MRR@25 on our pools. Any score materially above H₂₅/n is pool bias we are accidentally harvesting — and it will not transfer to a differently-built hidden pool.
4. Compute a permutation null for MRR@25 on the holdout (≥2,000 draws of random per-candidate scores on the real pools) and refuse to ship any feature below the 99th percentile adjusted for the number of features tried.
5. Build exactly four orthogonal engines, gated on standalone recall@25: (i) library/entropy search, (ii) spectrum→fingerprint retrieval, (iii) in-silico fragmentation (MetFrag / CFM-ID) as an absolute scorer, (iv) formula/isotope/adduct agreement. Add de-novo generation as a fifth only for the low-margin tail.
6. Fuse with quality-weighted RRF, k = 60, Σ_e w_e/(k + r_e(d)), w_e from standalone holdout MRR@25 — but run the five-way bake-off (RRF / weighted RRF / within-molecule z-sum / CombMNZ / Borda) once, with a permutation CI, before freezing it.
7. Deduplicate candidates on tautomer-canonical InChIKey14 before fusion and again before writing the 25.
8. Sweep the ppm window and database union against recall@25 vs mean pool size and stop at the knee. The BRIEF's "1 mDa / 10 ppm is critical" and "PubChem HURTS" are both consistent with the 1/n arithmetic and with CSI:FingerID's 68.5 % → 34.4 %.
9. Port submission_audit.py on day one, not on drop day. Hard gates: column presence, nulls, duplicate molecule_id, >25 guesses, RDKit-2026.03.3 parse, canonical-form match.
10. Add a per-molecule wall-clock budget with a degrade path that still emits 25 valid SMILES, and score the degrade path so we know what a timeout costs.
11. Route the lowest-margin decile to an expensive second pipeline (tail rescue), pre-registering the decile size.
12. Pre-register the two final submissions' selection criterion now, on holdout MRR@25, with the public LB used only as a gross-failure check.

Open questions

• What is the actual Kaggle notebook compute/time limit for this competition? BRIEF marks it unverified; it determines whether a four-engine fusion is feasible at inference.
• Is there any published accuracy-vs-candidate-count curve at fixed molecular formula? Nothing I opened controls for formula, which is the main confounder in the CASMI 2016 binning.
• What exactly does the organisers' tautomer canonicalisation call look like (TautomerEnumerator defaults? a specific Canonicalize call?)? A one-line difference changes our local metric.
• Does enveda's PRISM, or any public checkpoint derived from it, count as an orthogonal engine or as another correlated witness trained on the same public half of its 1.2 B spectra?
• Can the mock-witness probe be run against the hidden pool indirectly, e.g. by checking whether public-LB score correlates with pool size across submissions? Probably not within 5 submissions/day without burning the budget.

Sources

Local (read, not modified):

• D:\Users\ashenoy00000\.devin\CASMI26_Mass_Spec\research\_meta\sibling_learnings.md
• D:\Users\ashenoy00000\.devin\CASMI26_Mass_Spec\BRIEF.md
• D:\Users\ashenoy00000\.windsurf\OpenADMET-cyp-structure\docs\FINDING_001_selection_is_the_whole_problem.md
• …\docs\FINDING_003_first_selector_that_works.md, FINDING_007_noise_floor.md, FINDING_011_cross_engine_agreement.md, FINDING_012_cross_engine_generalises.md, FINDING_013_union_pool_adds_oracle_not_score.md, FINDING_015_protenix_v2_is_deterministic.md, FINDING_016_sampler_sweep_is_a_renewable_reference.md, FINDING_041_reference_depth_value.md
• …\src\cypstruct\budget.py, …\scripts\submit\submission_audit.py (module docstrings)

Note: the digest records the cross-engine gain as "+0.038, p = 3.7e-6"; the primary document (FINDING_011) states +0.0380 at reference depth 7 with within-ligand ρ = −0.258, 75.9 % correct direction and Wilcoxon p = 3.4e−06. I report the primary.

Web (opened in full or in substantial part):

• Cormack, Clarke & Büttcher (2009), Reciprocal Rank Fusion outperforms Condorcet and individual Rank Learning Methods, SIGIR '09: 758–759 — http://cormack.uwaterloo.ca/cormacksigir09-rrf.pdf (full text extracted)
• Dührkop, Shen, Meusel, Rousu & Böcker (2015), Searching molecular structure databases with tandem mass spectra using CSI:FingerID, PNAS, doi:10.1073/pnas.1509788112 — https://pmc.ncbi.nlm.nih.gov/articles/PMC4611636/
• Ruttkies, Schymanski, Wolf, Hollender & Neumann (2016), MetFrag relaunched, J Cheminform 8:3, doi:10.1186/s13321-016-0115-9 — https://pmc.ncbi.nlm.nih.gov/articles/PMC4732001/
• Schymanski et al. (2017), Critical Assessment of Small Molecule Identification 2016: automated methods, J Cheminform, doi:10.1186/s13321-017-0207-1 — https://pmc.ncbi.nlm.nih.gov/articles/PMC5368104/
• Hoffmann et al. (2022), High-confidence structural annotation of metabolites absent from spectral libraries (COSMIC), Nat Biotechnol, doi:10.1038/s41587-021-01045-9 — https://pmc.ncbi.nlm.nih.gov/articles/PMC8926923/
• Ludwig et al. (2018), Bayesian networks for mass spectrometric metabolite identification via molecular fingerprints, Bioinformatics 34:i333, doi:10.1093/bioinformatics/bty245 — https://pmc.ncbi.nlm.nih.gov/articles/PMC6022630/ (top-1 43.62 % CV / 36.61 % CASMI 2016; no candidate-count analysis — a negative result for this brief)
• Wixted & Wells (2017), The Relationship Between Eyewitness Confidence and Identification Accuracy: A New Synthesis, Psychol Sci Public Interest 18(1), doi:10.1177/1529100616686966 — https://wixtedlab.ucsd.edu/publications/wixted2017/Wixted_Wells_2017.pdf (full text extracted; Box 3, pp. 26–27 and the fair/unfair CAC comparisons)
• Metadata confirmation for the above: https://api.semanticscholar.org/graph/v1/paper/DOI:10.1177/1529100616686966 ; https://pmc.ncbi.nlm.nih.gov/tools/idconv/

Not opened, flagged: Fox & Shaw (1994) CombSUM/CombMNZ primary — the CombMNZ formula and results quoted here come from the RRF paper's own formulation. The SIRIUS 4 book-chapter preprint at bio.informatik.uni-jena.de returned HTTP 503. The Kaggle competition page itself was not fetched (JS-rendered; per BRIEF, Kaggle facts here come from the BRIEF's own GitHub-write-up intelligence).