Validation as an eyewitness line-up

TL;DR. Our local score lies unless the "fillers" (decoy candidates), the "witness conditions" (timsTOF, natural-product chemistry, 1–16 merged multi-CE spectra per molecule) and the "administrator" (no leaked structures, tautomer twins, analogs or re-hosted duplicates) match the hidden test. Only 250 train structures satisfy "timsTOF + natural product" and every one of them also lives in another library, so the honest hold-out is a QTOF-sourced, scaffold-disjoint, NP-like block of ~400 structures (estimated 4,000–15,000 eligible) with three strata: truth-in-pool, truth-masked-from-pool ("target-absent", ~45 % of the hidden test), and sealed replicate. Cascade-ranking evidence (SIGIR 2011/2017) and the ROC rebuttal of "sequential superiority" in eyewitness research both say the same thing: a sequential "ask A, stop if confident" policy buys compute, not accuracy; fuse simultaneously, and use a calibrated absence detector, not ranker self-confidence, to route queries.

1. Protocol spec (implementable today)

Line-up mapping: query spectrum = witness memory; true InChIKey14 = culprit; candidate pool = line-up; ranker = witness; our split code = administrator; MRR@25 = graded verdict. The hidden test (BRIEF; Queen's measurements) is: ≈400 molecules / ~1,500 spectra, 1–16 spectra per molecule (median 3), all Bruker timsTOF, 7 adducts in both polarities, multi-CE spectra merged into one row with a collision_energy_ev list, natural products/analogs "none previously published", precursor-inconsistent spectra dropped by the host.

Hard facts from the official file EDA (research/_meta/train_eda.md, 2026-10-07): 2,539,608 spectra / 275,810 InChIKey14; timsTOF rows = 1,154,969 and all are enveda-180 (1,153,785; synthetic Enamine chemistry, ~0.03 % in COCONUT) + enveda-np-examples (1,184 rows, 250 NP structures, 0 exclusive — every one also appears in GNPS/MassBank/RIKEN/etc.). Therefore "timsTOF-only AND NP-only AND structure-disjoint" has exactly 250 members, and holding them out means deleting their rows from every library. The protocol below keeps that block as stratum T (timsTOF) and adds a QTOF-sourced block Q, because QTOF beam-type CID is the nearest public analogue to the timsTOF's collision cell (q2 provenance worker).

Filter sequence (pseudocode over the train columns; exact code in §2):

S0  rows ← train.parquet
S1  instrument: keep instrument_type matching /tims/i OR /q-?tof|tof/i, drop /orbitrap|qft|itft|ft|trap|qq|qqq|qit|fab/i and nulls
S2  adduct ∈ test.adduct.unique() (the 7) AND ionization_mode ∈ test.ionization_mode.unique()
S3  |precursor_error_ppm| ≤ 10 AND not null            # host applied this to the test; train ships it uncleaned
S4  precursor_mz within [min, max] of the hidden-test-like range (visible decoy: 245–460; hidden per q2: 157–1,159, unverified)
S5  structures ← group_by(inchikey14): first(normalized_smiles), n_unique(ingest_lib), count
    ik14_metric ← InChIKey14(TautomerEnumerator().Canonicalize(mol))   # recompute; raw inchikey14 is not metric-faithful
    collapse structures whose ik14_metric coincide (tautomer twins)
S6  NP membership: ik14_metric ∈ COCONUT_IK14 (recomputed with the same canonicalisation) OR NPClassifier returns a pathway
    (fallback proxy: RDKit NP_Score > 0; COCONUT has 28.5 % of entries ≤ 0, so a threshold of 0 is conservative)
S7  scaffold ← MurckoScaffold.MurckoScaffoldSmiles(mol, includeChirality=False) for ALL 275,810 train structures
    eligible ← NP structures whose scaffold occurs in exactly one train structure (and scaffold ≠ "" i.e. not acyclic)
S8  draw 3 × 400 structures (seed 0/1/2) from eligible, disjoint; block Q
    block T ← the 250 enveda-np-examples structures
S9  for each held-out structure, DELETE from the training set every row with the same ik14_metric in any ingest_lib,
    every row whose Morgan-r2 Tanimoto to it is ≥ 0.80 (report count), and every row whose spectrum is a
    ≥0.99-cosine duplicate of a deleted row (re-hosted copies in mona/spectraverse/msdial)
S10 spectra for held-out structures: keep only S1–S4-passing rows; if a structure has single-CE rows, merge
    (same structure, same adduct, same polarity) across CE with the SAME merge function used at inference
    on the test's `collision_energy_ev` lists; subsample to the hidden test's per-molecule count distribution
    (draw k ~ empirical{1..16, median 3}); one query = one molecule_id with k spectra, exactly like test.parquet
S11 candidate pool for scoring = the same pool the submission will use (COCONUT IK14 + train structures),
    with three masks: (a) truth present; (b) truth removed ("target-absent"); (c) truth and its ≥0.80 analogs removed
S12 report MRR@25, recall@25-of-pool, top-1, per stratum (T, Q) × mask (a, b, c), with paired bootstrap CIs

Why each step: S1/S2/S4 reproduce the witness conditions; S3 reproduces a filter the host applied to the test but not to train (q2 provenance: 62,834 train rows > 20 ppm); S5 is mandatory because write-ups found 82 of 250 np-examples listed under two inchikey14 strings (tautomer/charge variants) and 11 such pairs train-wide; S7 is the Bemis–Murcko requirement (RDKit MurckoScaffold.MurckoScaffoldSmiles(smiles=None, mol=None, includeChirality=False), GetScaffoldForMol(mol), MakeScaffoldGeneric(mol) for the atom-stripped variant); S9 is the administrator's job (§4); S10 matches the hidden test's spectrum multiplicity, which matters because the LB-leading pipelines aggregate across a molecule's spectra; S11(b) exists because back-solved recall@25 ≈ 0.55 at LB 0.417 means ~45 % of hidden molecules are "perpetrator not in the line-up".

2. Size estimate and counting script

All numbers below are ESTIMATES from the EDA tables and published library statistics; the script computes the truth once the parquet is on disk (download currently returns 403 until rules are accepted).

Standard error: with MRR ≈ 0.42 the per-query reciprocal rank has SD ≈ 0.45 (≈35 % at 1.0, ≈45 % at 0), so n = 400 gives SE ≈ 0.023 and n = 250 gives ≈ 0.029. The public LB's 228-team pile-up inside 0.42–0.43 is below one SE: only paired comparisons on identical queries (paired bootstrap of ΔMRR) can resolve a 0.01 change locally.

Script (count_lineup_holdout.py; smoke-tested on a synthetic parquet with the official columns, RDKit 2026.03.6, polars 2.0.0; pip install polars — it was not on this box):

"""
count_lineup_holdout.py -- size the "eyewitness line-up" validation set from train.parquet.
Usage: python count_lineup_holdout.py TRAIN.parquet TEST.parquet [COCONUT_IK14.txt] [--seed 0] [--n-holdout 400]
COCONUT_IK14.txt = one InChIKey14 per line, produced from COCONUT SMILES with the SAME RDKit tautomer
canonicalisation the metric uses (make_coconut_ik14 below). If omitted, NP membership falls back to NP_Score > 0.
"""
import sys, time, argparse, os
import numpy as np, polars as pl
from rdkit import Chem, DataStructs, RDLogger
from rdkit.Chem import AllChem, rdMolDescriptors
from rdkit.Chem.Scaffolds import MurckoScaffold
from rdkit.Chem.MolStandardize import rdMolStandardize
RDLogger.DisableLog("rdApp.*")
TIMS_RE, QTOF_RE = r"(?i)tims", r"(?i)(q-?tof|qtof|tof)"
NOT_QTOF = r"(?i)(orbitrap|qft|itft|\bft\b|hybrid ft|exactive|trap|\bqq|qqq|\bit\b|qit|fab|ebeb)"

def canon_ik14(smiles, enumerator):
    m = Chem.MolFromSmiles(smiles)
    if m is None: return None
    try: m = enumerator.Canonicalize(m)
    except Exception: pass
    ik = Chem.MolToInchiKey(m); return ik[:14] if ik else None

def scaffold_smiles(smiles):
    m = Chem.MolFromSmiles(smiles)
    if m is None: return None
    try: return MurckoScaffold.MurckoScaffoldSmiles(mol=m, includeChirality=False)
    except Exception: return None

def np_score_fn():
    try:
        from rdkit import RDConfig
        sys.path.append(os.path.join(RDConfig.RDContribDir, "NP_Score")); import npscorer
        f = npscorer.readNPModel(); return lambda m: npscorer.scoreMol(m, f)
    except Exception: return None

def main():
    ap = argparse.ArgumentParser(); ap.add_argument("train"); ap.add_argument("test")
    ap.add_argument("coconut_ik14", nargs="?"); ap.add_argument("--seed", type=int, default=0)
    ap.add_argument("--n-holdout", type=int, default=400); ap.add_argument("--tanimoto-leak", type=float, default=0.80)
    a = ap.parse_args(); t0 = time.time()
    test = pl.read_parquet(a.test)
    test_adducts = set(test["adduct"].unique().to_list()); test_modes = set(test["ionization_mode"].unique().to_list())
    mz_lo, mz_hi = float(test["precursor_mz"].min()), float(test["precursor_mz"].max())
    print(f"[test] rows={test.height} molecule_ids={test['molecule_id'].n_unique()} adducts={sorted(test_adducts)} "
          f"modes={sorted(test_modes)} precursor_mz=[{mz_lo:.3f},{mz_hi:.3f}]")
    lf = pl.scan_parquet(a.train)
    cols = ["ingest_lib","normalized_smiles","inchikey14","molecular_formula","ionization_mode","instrument_type",
            "adduct","precursor_mz","precursor_error_ppm","num_peaks","collision_energy_ev"]
    lf = lf.select([c for c in cols if c in lf.collect_schema().names()])
    def report(name, frame):
        n = frame.select(pl.len(), pl.col("inchikey14").n_unique()).collect().row(0)
        print(f"{name:<58s} spectra={n[0]:>10,d}  IK14={n[1]:>9,d}"); return n
    report("F0 all train", lf)
    f1 = lf.filter(pl.col("instrument_type").is_not_null()
                   & (pl.col("instrument_type").str.contains(TIMS_RE) | pl.col("instrument_type").str.contains(QTOF_RE))
                   & ~pl.col("instrument_type").str.contains(NOT_QTOF))
    report("F1 instrument in {timsTOF, QTOF/TOF}", f1)
    report("F1a   of which timsTOF only", f1.filter(pl.col("instrument_type").str.contains(TIMS_RE)))
    f2 = f1.filter(pl.col("adduct").is_in(list(test_adducts)) & pl.col("ionization_mode").is_in(list(test_modes)))
    report("F2 + adduct in test's set and polarity", f2)
    f3 = f2.filter(pl.col("precursor_error_ppm").is_not_null() & (pl.col("precursor_error_ppm").abs() <= 10.0))
    report("F3 + |precursor_error_ppm| <= 10", f3)
    f4 = f3.filter(pl.col("precursor_mz").is_between(mz_lo, mz_hi))
    report("F4 + precursor_mz within test range", f4)
    structs = (f4.group_by("inchikey14").agg(pl.col("normalized_smiles").first(),
               pl.col("ingest_lib").n_unique().alias("n_libs"), pl.len().alias("n_spectra")).collect())
    print(f"[struct] unique IK14 after F4: {structs.height:,d}  ({time.time()-t0:.0f}s)")
    enum = rdMolStandardize.TautomerEnumerator(); enum.SetMaxTransforms(1000)   # metric default; slow tail 0.2-0.4 s
    smi = structs["normalized_smiles"].to_list(); scaf = [scaffold_smiles(s) for s in smi]
    structs = structs.with_columns(pl.Series("scaffold", scaf))
    print(f"[struct] scaffolds computed; null={sum(s is None for s in scaf):,d}; acyclic={sum(s=='' for s in scaf):,d}")
    npfn = np_score_fn()
    if a.coconut_ik14:
        coco = set(l.strip() for l in open(a.coconut_ik14) if l.strip())
        is_np = [k in coco for k in structs["inchikey14"].to_list()]; tag = "in COCONUT (metric-canonical IK14)"
    elif npfn is not None:
        is_np = [(m := Chem.MolFromSmiles(s)) is not None and npfn(m) > 0.0 for s in smi]; tag = "NP_Score > 0 (proxy)"
    else:
        is_np = [True]*len(smi); tag = "NO NP FILTER"
    structs = structs.with_columns(pl.Series("is_np", is_np)); np_structs = structs.filter(pl.col("is_np"))
    print(f"F5 + natural-product: {tag:<40s} IK14={np_structs.height:>9,d}")
    all_structs = lf.group_by("inchikey14").agg(pl.col("normalized_smiles").first()).collect()
    all_structs = all_structs.with_columns(pl.Series("scaffold", [scaffold_smiles(s) for s in all_structs["normalized_smiles"].to_list()]))
    scaf_freq = all_structs.group_by("scaffold").agg(pl.len().alias("n_struct_with_scaffold"))
    np_structs = np_structs.join(scaf_freq, on="scaffold", how="left")
    eligible = np_structs.filter(pl.col("scaffold").is_not_null() & (pl.col("scaffold") != "") & (pl.col("n_struct_with_scaffold") == 1))
    print(f"F6 + scaffold unique in whole train (holdable):  IK14={eligible.height:>9,d}   (scaffold shared with 2-5 others: "
          f"{np_structs.filter(pl.col('n_struct_with_scaffold').is_between(2,5)).height:,d})")
    rng = np.random.default_rng(a.seed); hold = eligible[rng.permutation(eligible.height)[:a.n_holdout].tolist()]
    hold_keys = set(hold["inchikey14"].to_list())
    hold_spectra = f4.filter(pl.col("inchikey14").is_in(list(hold_keys))).collect()
    print(f"[holdout] {hold.height} structures, {hold_spectra.height:,d} spectra, spectra/structure median="
          f"{np.median(hold_spectra.group_by('inchikey14').len()['len'].to_numpy()):.0f} (hidden test: median 3, range 1-16)")
    fpg = AllChem.GetMorganGenerator(radius=2, fpSize=2048)
    def fps(ss): return [fpg.GetFingerprint(m) if (m := Chem.MolFromSmiles(s)) is not None else None for s in ss]
    rest = all_structs.filter(~pl.col("inchikey14").is_in(list(hold_keys)))
    rest_fps = [f for f in fps(rest["normalized_smiles"].to_list()) if f is not None]
    nn = np.array([max(DataStructs.BulkTanimotoSimilarity(f, rest_fps)) if f is not None else np.nan for f in fps(hold["normalized_smiles"].to_list())])
    print(f"[leak] Tanimoto(holdout -> nearest remaining train): median={np.nanmedian(nn):.2f} p90={np.nanpercentile(nn,90):.2f} "
          f"frac>={a.tanimoto_leak}: {np.nanmean(nn>=a.tanimoto_leak):.3f}")
    print(f"[leak] held-out structures present in >1 ingest_lib: {hold.filter(pl.col('n_libs')>1).height} / {hold.height}")
    masses = np.sort(np.array([rdMolDescriptors.CalcExactMolWt(m) for s in np_structs['normalized_smiles'].to_list() if (m := Chem.MolFromSmiles(s)) is not None]))
    hm = np.array([rdMolDescriptors.CalcExactMolWt(Chem.MolFromSmiles(s)) for s in hold["normalized_smiles"].to_list()]); tol = hm*10e-6
    cpq = np.searchsorted(masses, hm+tol, side="right") - np.searchsorted(masses, hm-tol)
    print(f"[fillers] candidates/query at +-10 ppm in the NP pool ({len(masses):,d}): median={np.median(cpq):.0f} mean={cpq.mean():.1f} "
          f"p90={np.percentile(cpq,90):.0f}  (q6 measured COCONUT IK14 pool: median 36, mean 87, p90 204)")
    print(f"done in {time.time()-t0:.0f}s")

def make_coconut_ik14(coconut_smiles_txt, out_txt):
    """Metric-canonical IK14 list for COCONUT (~0.48 M IK14): run once, cache; slow tail 0.2-0.4 s/molecule."""
    enum = rdMolStandardize.TautomerEnumerator()
    with open(out_txt, "w") as fo:
        for line in open(coconut_smiles_txt):
            k = canon_ik14(line.strip(), enum)
            if k: fo.write(k + "\n")

if __name__ == "__main__": main()

Synthetic smoke-test output (3,000 rows, 42 structures): all stages print; eligible scaffold-unique = 5 of 20 NP-like; Tanimoto-leak fraction 0.000; candidates/query median 1. Real numbers will replace these.

3. Filler difficulty — how the benchmarks build decoys and what each gets wrong

The eyewitness rule (NIJ: fillers who do not resemble the suspect make the suspect stand out) translates into: decoys must match the description the witness has — same neutral mass within the instrument's error, same polarity/adduct feasibility, same chemical population (NP-like), and similar size/heteroatom content. A line-up where the true molecule is the only flavonoid among 35 synthetic amides is unfair in our favour.

Filler-fairness profile (proposed, four numbers, computed per query, compared as distributions between the local pool and the hidden-test-like reference):

1. Effective line-up size E_q = number of IK14-distinct candidates inside ±10 ppm (and, separately, sharing the inferred formula). Reference: COCONUT IK14 pool around enveda-np-examples masses (median 36). A local pool with median E_q ≪ 36 is too easy; ≫ 200 is PubChem-like and too hard.
2. Suspect-stands-out score S_q = Tanimoto(Morgan r2, true, nearest decoy) and the rank of the truth under a spectrum-blind NP-likeness/size prior. Fair ⇔ the truth is not an outlier (S_q ≥ 0.4 for most queries; prior rank distribution ≈ uniform). This is the mock-witness test of the eyewitness literature (named measures such as effective size and E' exist; not opened here, unverified).
3. Near-duplicate load D_q = fraction of candidates with pairwise Tanimoto ≥ 0.85 to another candidate (De Waele's σ_max). High D_q wastes slots unless tautomer/stereo-collapsed; report before and after collapsing.
4. Target-absent rate A = fraction of queries whose truth is not in the pool at all. Hidden-test estimate ≈ 0.45 (recall@25 back-solve); local pool for known train structures gives A = 0 unless masked, which is why §1 S11(b) exists.

Report the Kolmogorov–Smirnov distance of E_q, S_q, D_q between local and reference, and A directly; a validation set is "fair" when all three KS < 0.15 and A matches within ±0.1.

4. Administrator leak — routes and the de-duplication recipe

Tautomer-aware dedup that respects the measured timing. Queen's numbers: TautomerEnumerator().Canonicalize() 0.2–6 ms for rigid molecules, 179 ms quercetin, 357 ms rutin, 287 ms tetracycline, all hitting maxTransforms=1000. Natural products are exactly the slow class (polyphenols, glycosides, polyketides). Recipe:

1. Never canonicalise rows. Canonicalise unique structures only: train 275,810; COCONUT ≈ 480 k IK14. Budget ≈ 0.28 M × 5 ms ≈ 23 min plus a slow tail (if 15 % of NPs take 0.3 s: 42 k × 0.3 s ≈ 3.5 h single-core; 16 cores ≈ 15 min). Cache {raw_smiles_sha1: ik14_metric} on disk; this is a one-time cost.
2. Two-level keying: fast key = InChIKey14 of the raw SMILES (no tautomer step) to collapse the >99 % of identical structures; run the tautomer step once per fast key.
3. Do not tautomer-canonicalise PubChem (≥124.7 M): 124.7 M × 5 ms ≈ 7 days before the tail. Instead canonicalise only candidates that survive the ±10 ppm window and a cheap pre-score (≤ 200 per query × 400 queries = 80 k ≈ minutes).
4. Do not lower maxTransforms/maxTautomers to speed up the final 25 × 400 submission keys: a reduced cap can yield a different canonical tautomer than the metric's default and silently mismatch the hidden key. Full-default canonicalisation of 10,000 submitted SMILES is ≤ 10,000 × 0.4 s ≈ 67 min worst case — affordable.
5. Pin RDKit 2026.03.3 in the scoring container (the metric's version); the local box runs 2026.03.6 — patch releases should not change tautomer rules, but verify on the 250 np-examples that both versions agree (cheap).

5. Sequential vs simultaneous — cascades, line-ups, and the MRR@25 analogue

Cascade ranking evidence. Wang, Lin & Metzler (SIGIR 2011) learn a cascade of stages that prune then re-rank with progressively costlier features. Table 2 (Wt10g / Gov2 / ClueWeb09-B): cascade NDCG@20 35.60 / 47.44 / 30.60 vs monolithic AdaRank 35.49 / 47.37 / 30.94, i.e. +0.3 % / +0.1 % / −1.1 % relative; query time 0.175 / 2.00 / 4.28 s vs 0.260 / 3.90 / 6.55 s (32.7 %, 48.7 %, 34.7 % faster); filter loss per stage 0.0–1.6 %. Chen, Gallagher, Blanco & Culpepper (SIGIR 2017) reach "state-of-the-art effectiveness while only using a subset of the features". Viola & Jones (CVPR 2001) built the canonical rejection cascade for real-time face detection. The literature's own verdict: cascades buy cost, and match — not beat — the full model's effectiveness. A sequential policy's upside is therefore purely that it lets us afford expensive rankers (in-silico fragmentation, large fingerprint ensembles) on the survivors.

Eyewitness evidence. Steblay, Dysart & Wells (2011, Psychology, Public Policy, and Law, doi:10.1037/a0021650, "Seventy-two tests of the sequential lineup superiority effect") concluded sequential line-ups beat simultaneous on the diagnosticity ratio (I could not open the abstract; its numbers are unverified here). Mickes, Flowe & Wixted (2012, doi:10.1037/a0030609) introduced ROC analysis and found "the sequential procedure appears to be inferior to the simultaneous procedure in discriminating between the presence versus absence of a guilty suspect". Wixted & Mickes (2018, PMC5849663) give the mechanism: the diagnosticity ratio HR/FAR ignores response bias — Lindsay & Wells (1985) sequential HR .50 / FAR .17 (DR 2.94) vs simultaneous .57 / .42 (DR 1.36) looks like sequential superiority, but a stricter criterion on the simultaneous procedure can reach FAR .02 with HR still above sequential; only the partial AUC (FAR capped at 1/6 for a fair six-person line-up) orders procedures, and "empirical ROC analyses … unanimously suggest that the simultaneous procedure yields a higher pAUC". The Houston field study (Wixted et al., PNAS 2016, 348 blind line-ups) and Wixted & Mickes 2025 (d′ 2.93 simultaneous vs 2.03 sequential, p = 0.025) agree; the IACP (2016) reversed its sequential recommendation and DOJ (2017) treats both as valid (as reported in Wixted & Mickes 2018; DOJ memo not opened). The Wells, Steblay & Dysart (2015) field experiment, for completeness, found sequential gave 11 % filler IDs vs 18 % simultaneous — a criterion shift, exactly the thing ROC discounts.

Translation. The sequential "advantage" was a conservative criterion: fewer picks of any kind. MRR@25 has no penalty for a wrong pick — every query submits 25 names, abstention earns nothing — so the one benefit sequential procedures demonstrably have (fewer false identifications) is worth exactly zero here. What remains is discriminability, and there the evidence points the other way: simultaneous presentation lets the witness discount features shared by all faces (Wixted & Mickes' diagnostic-feature-detection account), whose ML analogue is listwise/contrastive scoring over the actual candidate set, which De Waele et al. 2026 show beats pointwise fingerprint losses on retrieval (HR@1 13.62 % vs 5.58 %). The MRR@25 analogue of the ROC is the routing curve: sweep the gate threshold τ of "use ranker A if conf_A ≥ τ else ranker B", plot MRR@25 against the fraction routed to B, and compare with the single point of simultaneous fusion s = Σ w_i·z_i(score_i) (per-ranker z-normalised, weights fit on block Q). If fusion includes conf_A as a feature, the gate is a special case and fusion cannot lose in expectation; when a gate does win empirically — gnshx's "inject PubChem top-50 only when library confidence < 0.7" (LB 0.384 → 0.420, via q6 worker; not opened by me) — it is because the fused scores were not calibrated across sources (Laudicina: 68 % of top-1 picks were PubChem-only after merging). Verdict: use a cascade for cost (cheap mass/formula/fingerprint prefilter → expensive scorers on ≤ 200 survivors), fuse the survivors' scores simultaneously, and route to the de novo/analog channel only on a calibrated target-absent detector (§6), never on ranker A's self-confidence.

6. Calibration — the CAC analogue

Wixted & Wells (2017, PSPI, doi:10.1177/1529100616686966) argue confidence is highly diagnostic of accuracy under pristine conditions: one suspect per line-up with ≥ 5 physically similar fillers, blind administration, the "may or may not be present" instruction, memory tested only once, and confidence recorded at the time of the identification (verified via Wixted & Mickes 2025's restatement; the SAGE page returned 403). CAC curve: for each confidence bin, suspect-ID accuracy = correct suspect IDs / (correct + false suspect IDs), with innocent-suspect IDs estimated as filler IDs / (n − 1). Field numbers (Houston, PNAS 2016; Wixted & Mickes 2025): high-confidence IDs ≈ 97 % accurate, medium ≈ 85–87 %, low ≈ 64–70 %; lab (Ayala et al. 2025 via the same paper): 98 / 88 / 75 %.

Our CAC. Confidence c_q = calibrated probability that the top-1 candidate is correct (after temperature scaling fitted on block Q, replicate 1). Bins: 8 equal-count bins (≥ 50 queries each at n = 400), or Guo et al.'s M = 15 equal-width bins when n ≥ 1,500. Plot per bin: observed top-1 accuracy with Wilson 95 % intervals (y) against mean confidence (x), bin size as marker area, the diagonal as reference. Report ECE = Σ_m (|B_m|/n)·|acc(B_m) − conf(B_m)| (Guo, Pleiss, Sun & Weinberger, ICML 2017) and MCE = max_m |acc − conf|. Produce the plot three times: stratum (a) truth-in-pool, stratum (b) target-absent (accuracy is 0 by construction, so this panel shows the confidence histogram — it must sit in the low bins), and the pooled set at the hidden test's A ≈ 0.45. Add a fourth panel: ROC of c_q as a detector of "truth in pool" with AUROC — this is the real decision aid for routing to generation/analog enumeration.

A healthy picture: monotone CAC, top bin ≥ 0.90 accuracy (the eyewitness 97 % is the aspiration), ECE ≤ 0.05, target-absent histogram massed below c = 0.3, AUROC(absence) ≥ 0.8. Unhealthy and common: flat CAC (confidence uninformative — ranker margin reflects decoy count, not evidence), inverted top bin (over-confident on near-duplicate candidates, De Waele's σ_max ≈ 1), or a target-absent histogram that looks like the target-present one (the witness who confidently picks a filler — the ranker has no "not there" option, so its margin is meaningless when the truth is absent). Use c_q only after this plot exists; a sibling-project lesson (BRIEF) is that model confidence ≠ accuracy until shown otherwise.

Forced analogy: eyewitness line-ups

(a) Statement. A validation query is a line-up administered to a witness (the ranker) whose memory is the training set; the hold-out molecule is the suspect, pool candidates are fillers, "fair fillers" are mass/formula/population-matched decoys, the "may-not-be-present" instruction is the target-absent stratum, double-blind administration is leak removal plus a sealed replicate, the confidence statement is the calibrated top-1 probability, and ROC/CAC are the only admissible ways to compare procedures.

(b) Testable ideas. (1) Target-absent stratum: mask the truth from the pool for a random 45 % of block-Q queries; a pipeline whose MRR@25 on that stratum is > 0 (via analog enumeration or de novo) gains on the hidden test in proportion, and its absence-detector AUROC predicts how much. (2) Listwise = simultaneous: retrain the final scorer with a contrastive loss over the actual ±10 ppm candidate set and compare with the pointwise fingerprint model by paired bootstrap on block Q; De Waele's MassSpecGym result predicts a top-1 gain. (3) Filler-fairness gate: reject any local validation pool whose E_q/S_q/D_q KS distances to the COCONUT-around-np-examples reference exceed 0.15 before trusting its number.

(c) Does it hold? Holds for pool fairness, target-absence, calibration and the ROC critique of gating. Partially holds for "double-blind": leakage is the mugshot-exposure problem (the witness has seen the suspect before), and the fix — remove the exposure — is the same, but our witness can be retrained, which no court can do. Breaks in two places: MRR@25 charges nothing for false picks, so the entire protect-the-innocent motive behind sequential procedures is absent; and a six-person line-up produces one decision while our witness returns a full ordering over 36–10⁴ candidates, so pAUC must be replaced by routing curves and recall-at-k.

Gotchas / what could fool us

1. "timsTOF + NP" = 250 structures, all of them duplicated in other libraries. Any timsTOF-only hold-out that keeps the other libraries' rows measures cross-instrument replication (0.93), not identification (0.00–0.05).
2. Random IK14 splits are inflated twice (enveda-180 R-group siblings; cross-library replicates): 0.456 local → 0.112 LB in one write-up.
3. COCONUT membership is key-sensitive: the same 250 np-examples are "99.6 % in COCONUT" and "8 of 250" in two write-ups; train-wide exact overlap claimed at 545–708 is implausibly low given RIKEN plant metabolites and is probably a canonicalisation artefact. Recompute with the metric's canonicalisation before trusting S6.
4. Raw inchikey14 is not the metric's key: 82/250 np-examples had two keys. Splitting on the raw key leaks tautomer twins.
5. n = 400 has SE ≈ 0.023; the 228-team 0.01 band on the public LB is noise; use paired bootstrap of ΔMRR, never absolute differences.
6. Halogen and amide features are strongly predictive on any enveda-180-containing split and dead on the hidden test (NPs are rarely halogenated; the decoy test is 36 % halogenated).
7. Multi-CE merge function unknown: the hidden test ships merged rows; if our merge of QTOF single-CE rows differs (binning tolerance, intensity aggregation), block Q spectra will look unlike the test. Reverse-engineer it from enveda-180, where single-CE and "20,40,60" rows coexist for the same compound.
8. Spectra-per-molecule mismatch: riken averages 21.8 spectra/structure vs test median 3; without S10 subsampling, aggregation-based pipelines look better locally than they will on the test.
9. Instrument strings are free text (82 values; "qtof", "qTof", "ToF", "Maxis II HD Q-TOF Bruker"); a missed pattern silently drops a source. Print the matched/unmatched string table.
10. Formula artefacts in COCONUT (C6H5N3O with 1,761 IK14s; triglyceride blocks) inflate E_q for some masses; compute formulas from SMILES, not from the CSV column.
11. Pool-dependent calibration: scores calibrated on a COCONUT-only pool are miscalibrated once PubChem candidates enter (68 % PubChem-only top-1s). Calibrate on the exact pool composition you will submit with.
12. Target-absent queries make ranker confidence meaningless, but a ranker still emits a margin; do not interpret that margin without the absence detector.
13. RDKit version drift: the metric is pinned to 2026.03.3; verify tautomer agreement with the local 2026.03.6 on the 250 np-examples.
14. Precursor-consistency filter asymmetry: the host dropped inconsistent test spectra but train ships 62,834 rows > 20 ppm; a validation set without S3 contains adduct/charge confusions the test cannot have.

Implications for the challenge

1. Accept the rules, download, and run count_lineup_holdout.py today; publish the true per-step counts and the eligible-scaffold count. If eligible < 1,200, relax S7 to "scaffold shared with ≤ 2 other train structures" and report the relaxation.
2. Build block T (250 np-examples, all library rows removed) and block Q (3 × 400 QTOF-sourced NP-like scaffold-disjoint structures, seeds 0/1/2); seal replicate 2 until the final model choice.
3. Score every candidate pipeline on the stratum × mask grid (T/Q × present/absent/analog-removed) with paired bootstrap CIs; promote a change only if ΔMRR on Q-present and Q-absent are both ≥ 0 and the pooled Δ at A = 0.45 is positive.
4. Compute the filler-fairness profile (E_q, S_q, D_q, A) for the local pool against the COCONUT-around-np-examples reference; fix the pool before fixing the ranker.
5. Tautomer-canonicalise unique structures only (train 275,810; COCONUT ≈ 480 k), cache keys; never canonicalise PubChem wholesale; canonicalise the 10,000 submitted SMILES at full default settings.
6. Implement the cascade for cost only (prefilter → ≤ 200 survivors → expensive scorers) and fuse survivors' scores simultaneously with per-source z-normalisation; replace any confidence-gated source injection by a calibrated absence detector once the CAC/ROC panel exists.
7. Produce the four-panel CAC/absence plot for the final ensemble; require monotone CAC, ECE ≤ 0.05 and AUROC(absence) ≥ 0.8 before using confidence anywhere in slot allocation.
8. Reverse-engineer the multi-CE merge from enveda-180 single-vs-merged rows and apply the identical function to block Q.

Open questions

• The exact 7 adducts and the hidden test's m/z range (157–1,159 Da is from a sibling worker, unverified by me) — both are read from test.parquet by the script, but the hidden set may differ from the visible decoy.
• True COCONUT ∩ train overlap under the metric's canonicalisation (545 vs tens of thousands) — decides whether S6 should use COCONUT membership or NP_Score/NPClassifier.
• Enveda's multi-CE merge algorithm (peak binning tolerance, intensity aggregation).
• Whether the public/private LB split is per molecule and what fraction is public (affects how much of the 228-team pile-up is noise).
• Steblay, Dysart & Wells 2011 headline rates (abstract not retrievable here).

Sources

• Wixted JT, Wells GL (2017) The relationship between eyewitness confidence and identification accuracy: a new synthesis. Psychol Sci Public Interest 18(1):10–65, doi:10.1177/1529100616686966 (PMID 28395650; abstract via Europe PMC and Semantic Scholar; SAGE full text 403).
• Wixted JT, Mickes L (2025) Suspect identification accuracy from lineups, in the lab and in the field. Cogn Res Princ Implic 10(1), doi:10.1186/s41235-025-00670-1, https://pmc.ncbi.nlm.nih.gov/articles/PMC12449283/ (pristine conditions, CAC formula, 97/85/70 % and 98/88/75 %, d′ 2.93 vs 2.03).
• Wixted JT, Mickes L (2018) Theoretical vs. empirical discriminability: the application of ROC methods to eyewitness identification. Cogn Res Princ Implic 3:9, doi:10.1186/s41235-018-0093-8, https://pmc.ncbi.nlm.nih.gov/articles/PMC5849663/ (diagnosticity-ratio critique, Lindsay & Wells 1985 numbers, pAUC, IACP/DOJ).
• Mickes L, Flowe HD, Wixted JT (2012) ROC analysis of eyewitness memory: comparing the diagnostic accuracy of simultaneous versus sequential lineups. J Exp Psychol Appl 18(4):361–376, doi:10.1037/a0030609 (abstract via Semantic Scholar).
• Wixted JT, Mickes L, Dunn JC, Clark SE, Wells W (2016) Estimating the reliability of eyewitness identifications from police lineups. PNAS 113(2):304–309, doi:10.1073/pnas.1516814112, https://pmc.ncbi.nlm.nih.gov/articles/PMC4720310/.
• Wells GL, Steblay NK, Dysart JE (2015) Double-blind photo lineups using actual eyewitnesses. Law Hum Behav, doi:10.1037/lhb0000096 (abstract via Europe PMC).
• Steblay NK, Dysart JE, Wells GL (2011) Seventy-two tests of the sequential lineup superiority effect. doi:10.1037/a0021650 (title/authors/year via Semantic Scholar; abstract not retrievable — numbers unverified).
• NIJ, Eyewitness Identification (archived article), https://nij.ojp.gov/topics/articles/eyewitness-identification.
• Wang L, Lin J, Metzler D (2011) A cascade ranking model for efficient ranked retrieval. SIGIR 2011, doi:10.1145/2009916.2009934; PDF https://cs.uwaterloo.ca/~jimmylin/publications/Wang_etal_SIGIR2011.pdf (Tables 2–3).
• Chen R-C, Gallagher L, Blanco R, Culpepper JS (2017) Efficient cost-aware cascade ranking in multi-stage retrieval. SIGIR 2017, doi:10.1145/3077136.3080819 (abstract via Semantic Scholar).
• Viola P, Jones M (2001) Rapid object detection using a boosted cascade of simple features. CVPR 2001, doi:10.1109/CVPR.2001.990517 (metadata only).
• Guo C, Pleiss G, Sun Y, Weinberger KQ (2017) On calibration of modern neural networks. ICML 2017, arXiv:1706.04599 (ECE/MCE definitions, M = 15 bins, temperature scaling).
• Bushuiev R et al. (2024) MassSpecGym. arXiv:2410.23326, https://arxiv.org/html/2410.23326; repo README https://github.com/pluskal-lab/MassSpecGym.
• De Waele G, Wydmuch M, Dembczyński K, Kotłowski W, Waegeman W (2026) Small molecule retrieval from tandem mass spectrometry: what are we optimizing for? arXiv:2602.16507, https://arxiv.org/html/2602.16507.
• Dührkop K, Shen H, Meusel M, Rousu J, Böcker S (2015) Searching molecular structure databases with tandem mass spectra using CSI:FingerID. PNAS 112(41), https://pmc.ncbi.nlm.nih.gov/articles/PMC4611636/.
• Kim HW et al. (2021) NPClassifier. J Nat Prod, doi:10.1021/acs.jnatprod.1c00399 (abstract via Europe PMC).
• Chandrasekhar V et al. (2025) COCONUT 2.0. Nucleic Acids Res, doi:10.1093/nar/gkae1063 (abstract via Europe PMC).
• RDKit rdkit.Chem.Scaffolds.MurckoScaffold docs, https://www.rdkit.org/docs/source/rdkit.Chem.Scaffolds.MurckoScaffold.html; RDKit Contrib NP_Score (local install, 2026.03.6).
• Hive files reused (not re-derived): research/_meta/train_eda.md (official-download EDA), research/q2_data_forensics/* (provenance, enveda-180/np-examples anatomy, write-up numbers from Laudicina, shiferaxa, chiparon, mxm-1x, bobownYAO, leesirleesirlee, nghianguyen7171, gnshx — those GitHub write-ups were opened by the q2/q6 workers, not by me), research/q6_candidate_space_retrieval/candidate_databases_*.md (COCONUT IK14 = 479,690; ±10 ppm median 36), research/q4_challenge_intelligence/casmi_history_*.md (CASMI 2016/2022 rules from casmi-contest.org).
• Not reachable this session: casmi-contest.org (TLS certificate mismatch), journals.sagepub.com (403), psycnet.apa.org (JS shell), web search (session budget exhausted after the first call).