Forensic identification, the database-search problem and evidential independence, applied to MS/MS annotation confidence and candidate-pool size

TL;DR. Treat each MS/MS annotation as a forensic identification made by searching a suspect database: the posterior that the top candidate is right depends not only on its score (a likelihood ratio) but on how many same-formula "suspects" the pool contains and on the prior that the true molecule is in the pool at all. Written out, that formula says enlarging the pool from COCONUT to PubChem inserts ~10²–10³× more look-alike isomers whose spectral LRs are nowhere near zero, while barely raising the probability that a "never previously published" natural product is in the pool — which is the Kaggle observation that PubChem hurts. Forensic practice also supplies the two tools we lack: calibrated confidence from decoy/known-wrong candidates (Scheubert 2017, COSMIC 2022) and the discipline of multiplying only conditionally independent evidence channels.

1. How forensic fields rank and report partial matches

The hidden test is explicitly "none previously published" (BRIEF), so the exact-match route (CODIS "hit", AFIS "ident", library cosine ≥ 0.7) is closed for ~90 % of molecules and we are in the partial-match regime of every field below.

Three structural lessons. (i) Every field separates the search (automated, ranked, generous recall) from the identification (confirmatory, orthogonal: Y-STR, microscope, examiner). Our MRR@25 submission is the candidate list, not the identification — so the pipeline should be tuned for recall@25 first, then precision at rank 1. (ii) Partial-match protocols cap the candidate list ("manageable number") and then apply non-spectral priors (geography, age, non-forensic information) — the analogue is a database-membership / NP-likeness prior, not more spectral scoring. (iii) The familial-search error mode is not "unrelated stranger" (10⁻⁵–10⁻⁹) but "wrong relative" (cousin called sibling 3–18 %): for us, the dangerous wrong answer is the near-isomer, not the random molecule, exactly what Scheubert 2017 reports ("a structurally similar isomer with similar fragmentation spectrum ... will receive relatively high scores") and what Oberacher found in CASMI 2012 (1-aminoanthraquinone matched to 2-aminoanthraquinone/N-phenylphthalimide when no reference spectrum existed).

2. The database-search problem, and the pool-size formula for MS/MS

The controversy. NRC 1996 (I could not open the report text; the recommendation is characterised through Stockmarr 1999 and Storvik & Egeland 2007) advised the "np rule": when a suspect is found by searching a database of N profiles, report the random-match probability multiplied by N. Balding & Donnelly 1996 (J Forensic Sci 41:603) showed that the correct likelihood ratio is, if anything, slightly larger after a database search, because the N−1 non-matching profiles have been excluded from the population of possible sources; Stockmarr 1999 (Biometrics 55:671) defended an LR formulation that reproduces the NRC rule; Dawid 2001 (Biometrics 57:976), Balding 2002 ("I outline reasons why Stockmarr and the NRC report are wrong") and Storvik & Egeland 2007 ("the np rule should not be used to quantify evidence"; it approximates a frequentist P-value) sided with Balding; Meester & Sjerps 2003 argued the LR is ambiguous and the posterior odds are what matter; Sjerps & Meester 2009 generalised: selection through the evidence is compensated by small prior odds, "no further correction is necessary".

Derivation for our setting. For one spectrum S and a formula-matched pool of n candidates, hypotheses H_i ("the molecule is candidate i") and H_0 ("the molecule is not in the pool"). Priors P(H_i) = π_in·w_i (Σw_i = 1; w_i = 1/n if uniform), P(H_0) = 1 − π_in, where π_in is pool coverage. Likelihoods ℓ_i = P(S | H_i); ℓ_0 = P(S | H_0) is the likelihood under an out-of-pool molecule, which for MS/MS is not small, because an absent natural product usually has an in-pool near-isomer with an almost identical spectrum. With LR_i = ℓ_i/ℓ_0:

P(H_top | S) = π_in·w_top·LR_top / [ π_in·Σ_j w_j·LR_j + (1 − π_in) ].

Three readings. (a) With uniform w the ranking of candidates present in both pools is identical whatever n is: LR ordering is invariant to pool size. Enlarging the pool can only hurt by inserting candidates with LR_j > LR_true. (b) The posterior that the top hit is correct falls as Σ_j LR_j grows; with n_f same-formula isomers of mean wrong-candidate LR λ, the denominator term is ≈ π_in(LR_top + n_f·λ)/n. Expected rank of the true candidate ≈ 1 + n_f·P(LR_wrong > LR_true), so MRR ≈ π_in·E[1/rank] decays roughly as 1/(1 + n_f·p_beat). (c) The NRC "np" quantity reappears not as the evidence but as the expected number of wrong candidates above a score threshold, n_f·p(score ≥ s) — which is exactly the target–decoy FDR estimate of Scheubert 2017 (FDR = PIT × decoys above s / targets above s). So Balding is right for ordering and NRC/Stockmarr is right for calibrating a threshold; the two camps answer different questions, and we need both.

Why PubChem hurts. Going from COCONUT (~7×10⁵ structures) to PubChem (~9×10⁷) multiplies n_f for a typical NP formula by 10²–10³ (MetFrag relaunched, 473 spectra all present in both databases: MetFrag2.2 Top-1 105 with ChemSpider vs 30 with PubChem, Top-10 320 vs 226; "as PubChem is now double the size of the 2009 version, the candidate ranking becomes more challenging"). COSMIC reports the same: "the number of candidates in the structure database had a strong effect", with "a substantial drop of correct annotations for small FDR values" on PubChem vs the ~391 k biomolecule database. Against this, Δπ_in for a hidden set of unpublished natural products and "hypothesised NPs" is small: a molecule absent from COCONUT/LOTUS/NPAtlas is usually absent from PubChem too. The break-even condition is Δπ_in/π_in > 1 − E[1/rank_PubChem]/E[1/rank_COCONUT]; with the MetFrag numbers the right-hand side is ≈ 0.5–0.7, so the bigger pool pays only if it raises coverage by > 50–70 % relative — it does not. The forensic framing adds the fix: do not enlarge the suspect population uniformly; add candidates with a source prior w_i (COCONUT/LOTUS membership, NP-likeness, biosynthetic class), which is what familial searching does with geography and age. This is the Bayesian reading of q6's tiered pools.

3. Verbal scales and confidence levels: what is reachable on the hidden test

Consequence: the metric rewards a Level-3 list, judged as if it were Level 1 (InChIKey14 identity). The forensic word for scoring Level 3 output as Level 1 is the prosecutor's fallacy, and the correct defence is to report, per molecule, a calibrated P(correct in top-25) — useful internally for deciding where to spend de-novo or re-ranking effort, even though Kaggle never sees it.

4. Line-ups, fillers and the calibration of confidence

Findings used. Steblay, Dysart & Fulero 2001 (23 papers, 30 tests, 4,145 participants): sequential line-ups raise correct rejections, lower target-present hits. Wells, Steblay & Dysart 2015 field experiment: filler picks 11 % vs 18 %, suspect picks equal. Wixted, Mickes & Dunn 2016 (Houston PD, one year, three-point confidence): "confidence in an eyewitness identification from a fair lineup is a highly reliable indicator of accuracy" and any format difference "likely favors the simultaneous procedure" once analysed by signal detection/ROC rather than diagnosticity ratios. Wixted & Wells 2017: under pristine conditions (initial, uncontaminated test; fair line-up; no administrator influence; immediate confidence statement) "high-confidence suspect identifications are remarkably accurate", whereas "initial, low-confidence suspect identifications are error prone"; under unfair line-ups "even a high-confidence suspect ID is seriously compromised". Wixted, Mickes & Clark 2015: high-confidence misidentifications featured in > 70 % of > 300 DNA exonerations — confidence at trial, after contamination, not initial confidence. q7's channel note already records that the "sequential beats simultaneous" argument for a cascaded ranker collapses under ROC analysis; I do not repeat it.

Should fillers be similar or dissimilar? The line-up answer is precise: fillers must match the witness's description (so an innocent suspect does not stand out) but not be clones of the suspect (then nobody can be picked). A line-up where the suspect is the only member matching the description is "biased" and inflates both pick rate and confidence. Transposed: the decoy candidates against which a ranker's confidence is calibrated must match the spectrum's "description" (same formula, same adduct, NP-like), not random PubChem molecules of the same mass. Calibrating on easy fillers produces the "relative judgement" artefact — a large score gap to the runner-up that reflects a weak pool rather than a strong match. This is directly relevant because COSMIC's features include "score differences to other candidates" and the paper warns the score "must be treated as a score but not as the probability that the annotation is correct" and that performance depends strongly on database size. Features like score gap should be normalised by n_f, or the model should see n_f as an input (COSMIC does include the number of candidates).

MS equivalents. Scheubert 2017 is the formal decoy line-up: fragmentation-tree re-rooted decoys ("passatutto") give accurate q-values while "naive" random-fragment decoys "never result in accurate q-value estimates"; even the good decoys "tend to overestimate significance ... estimates are close to zero for true q-values below 0.05", and the stated reason is the near-isomer problem — wrong targets outscore decoys at the top of the list. COSMIC avoids decoy generation entirely ("we use candidates in PubChem as a proxy of decoys") and trains a discriminative confidence on known-wrong candidates — the fillers are real molecules matching the description. That is the right design for us: fillers = same-formula COCONUT/PubChem candidates that are known wrong on a labelled hold-out.

Confidence at first identification. The Wixted result has a modelling translation: the model's score on the untouched spectrum is the diagnostic quantity; any confidence obtained after test-time adaptation, pseudo-labelling on test spectra, or leaderboard-guided re-ranking is "contaminated memory" and should not be used to decide between final submissions.

5. Daubert: what a court would ask of our annotator

Daubert v. Merrell Dow, 509 U.S. 579 (1993) factors (Cornell LII): testability; peer review and publication; "known or potential error rate"; "standards controlling its operation"; general acceptance. The forensic field answered the error-rate demand with black-box studies (Ulery 2011: FP 0.1 %, FN 7.5 %, with the design tied to a 58-million-subject AFIS, i.e. error rate at the operating database size). Our analogue: we do not know our false-discovery rate per candidate-pool size, instrument or adduct. What exists: Scheubert's q-values (library search only; overconfident below q = 0.05); COSMIC's exact FDR on reference data (57 correct at 5 % FDR on CASMI 2016 n = 123 with the biomolecule DB; 16 at 0 % FDR with ChemSpider n = 127; 1,415 hits at 5 % FDR on 3,013 QTOF spectra vs 52 for library search), and its explicit caveat that FDR estimation for small molecules "will remain highly challenging" because wrong hits are not random. A Daubert-grade statement for this challenge would be: "on a hold-out of novel NP-like molecules on timsTOF, with pool P of size n_f distribution D, the rank-1 FDR is x % and recall@25 is y %", stratified by n_f bins. That table does not exist anywhere; we have to make it (Implication 2).

6. Chain of custody and reference-database contamination

Forensic DNA labs keep elimination databases of staff and run contamination checks because a reference database that contains the evidence source by accident produces a perfect "hit" that is evidence of nothing. The visible test.parquet is that scenario: its 1,213 spectra are verbatim enveda-180 train rows (BRIEF; q7 confirmed the tautomer/InChIKey side), so any local score on it measures library recall, not identification, and any model that memorises enveda-180 will look spectacular locally. The hidden test is "none previously published", which under this framing means: the evidence is guaranteed absent from the reference database, so a hit of library-grade similarity to a train spectrum on the hidden set is itself suspicious (a near-isomer or a contaminant in the hidden set), and the training signal that transfers is similarity structure, not identity. The custody rule to adopt: the local validation set must be constructed so that no InChIKey14, and ideally no Murcko scaffold, is shared with training, and the provenance of every hold-out molecule (which ingest_lib, which instrument) is logged with the score.

Forced analogy: technical art history as orthogonal evidence

The analogy, precisely. The Rembrandt Research Project (founded 1968; Bruyn, Haak, Levie, van Thiel, van de Wetering; older members resigned 1993; funding ceased 1998; board voted to terminate 2011; final Corpus volume 2014; Wikipedia) combined connoisseurial judgement with technical examination and, over its life, roughly halved the accepted self-portraits and reassigned many works to pupils. Its early volumes issued categorical verdicts (the A/B/C scheme and the later reversals of specific pictures are from general knowledge; not confirmed in the source I opened — unverified). The structural point is sound regardless: a technical channel such as an anachronistic pigment or a dendrochronological felling date later than the painter's death is exclusionary and independent of the stylistic judgement; a second stylistic opinion by the same committee is not independent of the first. Multiplying LRs is legitimate only for channels that are conditionally independent given the hypothesis.

Evidence channels for an MS/MS annotation, with independence status.

Truly independent of the MS2-derived scores: formula (filter), the prior, CCS/RT (absent). Everything else is one evidence channel read by several instruments — the RRP situation of five connoisseurs who trained together. Naive-Bayes multiplication over MS2-derived scores is double-counting.

One MRR@25 experiment. Hold-out: 2,000–4,000 molecules from enveda-np-examples and other NP-rich ingest_libs, scaffold-disjoint from the training split, spectra merged across CE as in test; pool = COCONUT (+LOTUS) formula-exact within 10 ppm. Channels: c1 fingerprint-MLP similarity, c2 in-silico fragmentation score, c3 nearest-train-spectrum similarity transferred to candidates via Tanimoto, c4 NP-likeness/db-membership prior, c5 multi-CE agreement, plus pool statistics n_f and score gap. Calibrate each channel to a log-LR by fitting score densities for correct vs wrong candidates on a separate calibration split (COSMIC's kernel-density E-value idea). Arms: (A) Σ log LR_c + log prior (naive Bayes); (B) logistic regression / gradient-boosted combiner on the same inputs plus n_f and gap, trained on correct-vs-wrong candidate pairs (COSMIC-style); (C) each channel alone; (D) A with the correlated pair (c1,c2) replaced by their mean log-LR. Readouts: MRR@25, recall@25, rank-1 FDR at confidence 0.5, and the diagnostic that decides the analogy — Pearson correlation of log-LR_c1 and log-LR_c2 among wrong candidates of the same spectrum. Prediction: if ρ < 0.2, A ≈ B and the product is safe; if ρ > 0.4, B beats A by ≥ 0.01 MRR and A's confidences are overconfident (reliability curve above the diagonal). Cost: one GPU-hour for the MLP scores, CPU for the rest.

Does the analogy hold? Partially. It holds for the distinction between exclusionary channels (formula, impossible fragment formulas, pigment anachronism) and holistic scores, and for the warning against multiplying correlated opinions. It breaks in one place: in art history the orthogonal channels (dendrochronology, pigments) are available; in this test the orthogonal physical channels (RT, CCS, isotope pattern, standards) are stripped, so our "technical art history" reduces to one channel and a prior. The honest consequence is that most of the achievable gain is in the prior and the candidate pool, not in cleverer combination of MS2 scores.

Gotchas / what could fool us

1. Verbal-scale fallacy. A "strong" LR (10³) against n_f = 10⁴ same-formula candidates is a posterior near 0.1; SWGDAM explicitly withholds its scale from database searches. Never report or threshold a raw score without n_f.
2. Pool enlargement is population enlargement. Balding–Donnelly's "search strengthens evidence" assumes a fixed population; adding PubChem adds new suspects with non-zero LR, so the evidence for the true candidate is unchanged while its rank falls. No LR correction fixes this — only a prior over sources does.
3. Decoys that are too easy. Naive decoys "never" calibrate (Scheubert); random-mass fillers inflate confidence; even good decoys are overconfident below q = 0.05. Fillers must match the description (formula, adduct, NP-likeness).
4. Score gap is a line-up property. A large gap to the runner-up on COCONUT may vanish on PubChem for the same spectrum; models that learn "gap ⇒ confident" learn pool sparsity.
5. Near-isomer error mode. Wrong answers are cousins, not strangers (3–18 % cousin-as-sibling; Oberacher's aminoanthraquinones; Scheubert's isomers). InChIKey14 scoring makes a 2-edge-different structure worth exactly zero.
6. Contaminated reference set. Visible test = train rows; local scores on it measure memorisation. Hidden molecules are by construction outside the library, so a library-grade hit on hidden data is a red flag, not a win.
7. Confidence after contamination. Any confidence computed after test-time adaptation or leaderboard-steered changes is the courtroom confidence Wixted warns about; use only first-pass scores to choose between the two final submissions. q7's measurement that 228 teams sit in one 0.01-wide public-LB band makes LB-steering doubly uninformative.
8. Data-dependent hypotheses. ENFSI §3.3: when results are known (a database search), "every effort should be made to avoid being led by the findings". Choosing the pool, mass window or re-ranker after probing the LB is the forensic "tailored hypothesis"; Sjerps & Meester say the prior must then be deflated — i.e. discount gains obtained that way.
9. Error rate is conditional. Ulery's 0.1 % is at a 58 M-subject AFIS; COSMIC's FDR moves with database size and instrument. A single global FDR for our ranker is meaningless; stratify by n_f, adduct and number of spectra per molecule.
10. Identity definition. The identification standard is InChIKey14 after RDKit 2026.03.3 tautomer canonicalisation; q7 found the enumerator silently truncates on polyphenols (MaxTransformsReached) — candidates written in unusual tautomers (de novo output) are the exposure.
11. Unavailable channels. No MS1 isotope pattern, RT or CCS columns in test; any pipeline that silently assumes them (e.g. SIRIUS formula scoring with isotopes) is running on defaults.
12. Paperclip and WebSearch were unavailable for this file (CLI not installed; search budget exhausted); ENFSI, SWGDAM, California and Sumner texts were read from PDFs; NRC 1996 and Schymanski 2014 originals could not be opened and are cited through secondary sources.

Implications for the challenge

1. Use the posterior formula, not the raw score, to choose pools. Measure n_f and p_beat per pool on the hold-out; enlarge the pool (COCONUT → LOTUS → NPAtlas → ChEMBL → PubChem subsets) only where Δπ_in/π_in exceeds the measured rank loss. Implement enlargement as a source prior w_i, never as a uniform merge.
2. Build the Daubert table. Rank-1 FDR and recall@25 stratified by n_f bin, adduct, number of spectra, on a scaffold-disjoint NP hold-out. This is the only local number allowed to steer decisions (the public LB band is 0.01 wide).
3. Calibrate confidence on description-matched fillers (same formula, same adduct, NP-like, known wrong), COSMIC-style, with n_f as an input; use it to decide per molecule whether to spend the 25 slots on pool candidates, on analogue expansion (edit the near-isomers), or on de-novo output.
4. Run the naive-Bayes vs learned-combiner experiment (Forced analogy section) and read the wrong-candidate correlation ρ before trusting any multiplicative ensemble; replace correlated MS2 channels by a single learned score.
5. Tune for recall@25 first. Every forensic search is recall-first with confirmation later; the back-solved recall@25 ≈ 0.55 at MRR 0.417 says ~45 % of molecules are lost before ranking matters. Slots 10–25 should be spent on diversity (distinct scaffolds/isomer classes) rather than on score-order depth.
6. Protect first-pass confidence. Freeze the scorer before any test-time adaptation; choose the two final submissions by hold-out metrics, not by LB probing.
7. Treat near-isomer confusion as the enemy. On the hold-out, log the InChIKey14-distance (edit count) between the top wrong hit and the truth; if most misses are 1–2 edits away, add an "isomer rescue" step that enumerates regio/positional isomers of the top-3 candidates into slots 4–25.

Open questions

• What is the empirical n_f distribution (same-formula candidates within 10 ppm) for COCONUT vs PubChem on NP-like formulas, and p_beat per scorer? (q6 can supply the counts; without them the break-even condition in §2 is qualitative.)
• Does the hidden set's "hypothesised NPs / analogs / synthetics plausibly in nature" sit in COCONUT at the 99.6 % rate of enveda-np-examples or near the 0.03 % of enveda-180? This single number sets π_in and therefore the whole pool strategy.
• Is the MS2 list in test ever contaminated with precursor isotope peaks that could recover an isotope-pattern channel?
• NRC 1996 Recommendation 5.1 exact wording, and whether later NRC/SWGDAM guidance withdrew the np rule, remain unverified here.
• Does a tempered product over multi-CE spectra (α ≈ 0.5) beat a plain sum of log-LRs? Fit α on the hold-out.

Sources

• ENFSI Guideline for Evaluative Reporting in Forensic Science (approved v3.0, STEOFRAE project; PDF opened): https://roebuckforensics.com.au/wp-content/uploads/2024/07/ENFSI-Guideline.pdf
• SWGDAM, Recommendations of the Ad Hoc Working Group on Genotyping Results Reported as Likelihood Ratios (approved 2018-07-12; PDF opened): https://swgdam.org/_files/ugd/4344b0_dd5221694d1448588dcd0937738c9e46.pdf
• California DOJ Information Bulletin 2008-BFS-01, DNA Partial Match Policy (PDF opened): https://oag.ca.gov/system/files/attachments/press_releases/n1548_08-bfs-01.pdf
• Myers SP et al. 2011, FSI Genetics 5:493–500, doi:10.1016/j.fsigen.2010.10.010 (abstract via Europe PMC)
• Rohlfs RV et al., "The influence of relatives on the efficiency and error rate of familial searching", arXiv:1304.3160 (ar5iv full text)
• Maguire CN et al. 2014, FSI Genetics 8:1–9, doi:10.1016/j.fsigen.2013.07.004 (abstract via Europe PMC)
• Balding DJ, Donnelly P 1996, J Forensic Sci 41:603–607, doi:10.1520/jfs13961j (abstract)
• Stockmarr A 1999, Biometrics 55:671–677, doi:10.1111/j.0006-341x.1999.00671.x (abstract)
• Dawid AP 2001, Biometrics 57:976–980, PMID 11550954 (record only)
• Balding DJ 2002, Biometrics 58:241–244, doi:10.1111/j.0006-341x.2002.00241.x (abstract)
• Meester R, Sjerps M 2003, Biometrics 59:727–732, doi:10.1111/1541-0420.00084 (abstract); Sjerps M, Meester R 2009, FSI 192:56–61, doi:10.1016/j.forsciint.2009.07.022 (abstract)
• Storvik G, Egeland T 2007, Biometrics 63:922–925, doi:10.1111/j.1541-0420.2007.00751.x (abstract)
• NRC 1996, The Evaluation of Forensic DNA Evidence — text not retrievable (NAP page images failed); characterised via Stockmarr 1999 and Storvik & Egeland 2007
• Ulery BT et al. 2011, PNAS 108:7733–7738, doi:10.1073/pnas.1018707108, PMC3093498 (abstract)
• Wikipedia, Combined DNA Index System; National Integrated Ballistic Information Network (secondary; opened)
• Daubert standard, Cornell LII: https://www.law.cornell.edu/wex/daubert_standard
• Steblay N, Dysart J, Fulero S 2001, Law Hum Behav 25:459–473, doi:10.1023/a:1012888715007 (abstract)
• Wells GL, Steblay NK, Dysart JE 2015, Law Hum Behav 39:1–14, doi:10.1037/lhb0000096 (abstract); Wells, Steblay, Dysart 2012, Perspect Psychol Sci 7:264 (abstract)
• Wixted JT, Mickes L, Dunn JC 2016, PNAS 113:304–309, doi:10.1073/pnas.1516814112 (abstract; full text not retrievable)
• Wixted JT, Wells GL 2017, Psychol Sci Public Interest 18:10–65, doi:10.1177/1529100616686966 (abstract); Wixted, Mickes, Clark 2015, Am Psychol 70:515 (abstract)
• Scheubert K et al. 2017, Nat Commun 8:1494, doi:10.1038/s41467-017-01318-5, PMC5684233 (full text via Europe PMC)
• Hoffmann MA et al. 2022, Nat Biotechnol 40:411, doi:10.1038/s41587-021-01045-9, PMC8926923 (full text via Europe PMC)
• Sumner LW et al. 2007, Metabolomics 3:211, doi:10.1007/s11306-007-0082-2 (PDF opened)
• Schymanski EL et al. 2014, Environ Sci Technol 48:2097–2098, doi:10.1021/es5002105 (original HTTP 403; levels paraphrased; structure confirmed via Charbonnet et al. 2022, ES&T Lett 9:473, PMC9202347)
• Ruttkies C et al. 2016, "MetFrag relaunched", J Cheminform 8:3, doi:10.1186/s13321-016-0115-9, PMC4732001 (full text)
• Schymanski EL et al. 2017, CASMI 2016 automated methods, J Cheminform, doi:10.1186/s13321-017-0207-1, PMC5368104 (full text); Oberacher H 2013, Metabolites 3:312, doi:10.3390/metabo3020312 (abstract)
• Wikipedia, Rembrandt Research Project (secondary; opened)
• Hive: research/_channels/serendipity_broadcast.md (q7 note, 2026-10-07); BRIEF.md