The visible test is a planted exemplar: decoy recovery, hidden-set inference from its construction, a validation-split recipe with real fold sizes, and the published numbers on MS/MS domain shift

TL;DR. The 400 visible-test molecules are a random draw of enveda-180 structures (every structure-level statistic sits inside the 95 % band of 2,000 random 400-structure draws), but the spectra were selected by two rules: every dimer adduct ([2M+X], 664 of 2,482 available rows) was discarded, and of the remaining monomer rows only 1,213 of 1,818 (66.7 %) were kept, with positive mode retained at 71 % and negative at 53 %. The celebrated "245–460 m/z window" is not an organiser filter at all: 99.67 % of all enveda-180 monomer rows already fall in it, because the Enamine HLL-200 library is mass-bounded. The decoys' chemistry is indistinguishable from random enveda-180 (NP_Score mean −1.47 vs −1.45; 0/400 in COCONUT; 36 % halogenated) and therefore says nothing about hidden-set chemistry; a validation fold that mimics the hidden set as described (timsTOF, NP-like, 1–16 spectra, no structure leakage) contains only 1,977 structures (250 np-examples + 1,727 NP-like enveda-180), and most of its "NP-likeness" is NP_Score > 0, not database membership.

All numbers labelled (computed) were computed from the local train.parquet (2,539,608 rows) and test.parquet (1,213 rows) at C:\Temp\casmi26\data\ (Kaggle download 2026-10-07) with pandas 3.0.6 / RDKit 2026.03.6; scripts live in the session scratchpad. JSON versions of every table are in the sibling .json file.

1. Decoy recovery and the fingerprint of the selection (Q1)

Join. Joining test rows to train on (round(precursor_mz,4), num_peaks, base_peak_intensity, adduct) matches 1,213 / 1,213 rows, each to exactly one train row, all in enveda-180; the collision_energy_orig string agrees for all 1,213; every one of the 400 molecule_ids resolves to exactly one inchikey14, no two molecule_ids share a structure (computed). 395 of the 400 structures exist only in enveda-180; 3 also appear in pluskal_ms2 and 2 in gnps+pluskal_ms2 (computed) — i.e. 1.25 % of decoy molecules have a non-timsTOF public spectrum, far below the ~10 % "in public libraries" share inferred for the hidden set by write-ups (corentinlaudicina README: first cosine-library submission 0.098 ⇒ "~9 in 10 molecules have no spectrum in train").

Structure selection is random. For 2,000 random draws of 400 enveda-180 structures I computed the same statistics as for the 400 decoy structures, using all their enveda-180 rows (2,482 rows) (computed):

Every "observed, all rows" value lies inside the null band: the organiser picked 400 structures at random (or by a criterion orthogonal to size, mass, adduct and CE). The deviations appear only after the row selection (bold).

Row selection rule 1 — monomer adducts only. Of the 2,482 enveda-180 rows belonging to the decoy structures, 664 are dimers ([2M+Na]+ 390, [2M+H]+ 230, [2M−H]− 41, [2M+CH2O2−H]− 3) and 0 of 664 were kept (computed). The visible test's "7 adducts" are therefore simply the monomer adducts that happened to occur among 400 structures; enveda-180 has two more monomer adducts ([M+C2H4O2−H]− 376 rows, [M+Br]− 25 rows) that would surface in a bigger draw.

(computed). Note that np-examples carries in-source water-loss adducts and 6–9 % each of [M+NH4]+ and [M+Na]+ — a very different adduct economy from enveda-180's monomer rows, where [M+H]+ dominates at 72 %.

Row selection rule 2 — a ~2/3 thinning that is not a deduplication. 1,213 of 1,818 monomer rows were kept (66.7 %). This is not "one spectrum per structure × adduct × CE": of 1,810 such groups, 1,802 contain a single row, 1,213 groups were kept whole and 597 groups were dropped whole (computed). Nor is it an m/z cut: the dropped monomer rows span 247.1–482.1 m/z and only 3 of them (all [M+CH2O2−H]− of one structure at 482.14) exceed 460. The keep rate varies with polarity and weakly with spectrum quality:

(computed). Kept rows have median 230 peaks vs 182 for dropped, median base peak 4.7e5 vs 2.6e5, median |ppm| 0.93 vs 1.09. Selection is neither all-or-nothing per polarity block (535 structure×polarity blocks: 234 fully kept, 38 fully dropped, 263 partial) nor per precursor species (704 blocks: 348 / 107 / 249). The pattern is consistent with a quality-weighted or random row sample at a ~2/3 rate with lower retention of negative-mode rows; I could not identify a deterministic rule (and the enveda-180 curation fields that could carry one — sirius_score, explained_intensity, num_merged_microscans — are not in train.parquet).

CE patterns are preserved, not filtered. Within polarity, the test's CE shares (positive: 60 29.3 %, 40 28.5 %, "20,40,60" 26.0 %, 20 16.2 %) track enveda-180's (25.8 / 27.5 / 27.0 / 19.5 %) (computed); the mild tilt toward 60 eV is the thinning's quality bias (higher-CE spectra have more peaks). The commonest per-molecule CE set is identical in test (76 molecules) and enveda-180 (3,788 of a 20,000-structure sample): {20, 40, 60, "20,40,60"}, i.e. the three single energies plus the merged spectrum for one precursor species. No test molecule has duplicate (adduct, CE) rows; enveda-180 structures commonly do (several precursor species per adduct), e.g. the pattern "20|20|20,40,60|20,40,60|40|40|60|60" occurs 1,470 times in the sample.

The m/z window is Enamine's, not the organiser's. The train-wide figure "64.6 % of precursors in 245–460" (train_eda) hides that enveda-180 monomer rows are 99.67 % inside 245–460.2 (5 %–95 %: 269.2–396.2; min 235.09; max 544.14), and 99.54 % of enveda-180 structures have all their monomer rows inside it (computed). The test's 245.09–460.16 range is what any 1,213 monomer rows would show. The two tails of the HLL-200 mass distribution (0.23 % of structures below 245, 0.23 % above 460) are simply absent from a 400-structure draw with probability ≈ e^(−400·0.0046) ≈ 16 %, i.e. unremarkable.

Spectra per molecule. Test: 1–9, median 3, mean 3.03 (hist 1:55, 2:97, 3:99, 4:106, 5:24, 6:14, 7:2, 8:2, 9:1). enveda-180 counting monomer rows only: 1–23, median 4, mean 4.58; × 0.667 thinning ⇒ ≈ 3.05, matching the test (computed). (mxm-1x DATA_AUDIT reports the same 1–9 / median 3 for the visible test.) The hidden set is described as 1–16 per molecule, median 3 (§3), so the decoy reproduces the hidden set's shape but not its tail.

2. Chemistry of the 400 decoy structures (Q2)

RDKit 2026.03.6 with the Contrib NP_Score model (npscorer.py, publicnp.model.gz, present in the user site-packages Contrib dir — verified) and COCONUT coconut_csv_lite-10-2026.csv (C:\Temp\casmi26\npdb\coconut\, 479,690 distinct InChIKey14) (computed):

COCONUT membership of train structures by library (computed): enveda-180 51 / 182,941 (0.03 %); enveda-np-examples 249 / 250 (99.6 %); gnps 37.0 %; mona 80.9 %; massbank 63.1 %; riken 79.0 %; msdial 80.3 %; drug_plus 76.8 %; masaryk 84.9 %; pluskal_ms2 17.4 %; spectraverse 21.7 %. The decoy is chemically a random Enamine screening-compound sample: nitrogen-rich amides/heteroaromatics, one stereocentre, a third halogenated. The np-examples slice is its mirror image — oxygen-rich, nitrogen-free in half the cases, 4.6 stereocentres. Anything learned from the decoy's chemistry transfers to the hidden set only if the hidden set is also Enamine-like, which contradicts every public description (§3).

3. Hidden-set inference (Q3)

What the organiser has said, as mirrored (Kaggle pages are JS shells; none of these quotes were read on kaggle.com itself):

Inference from construction. The decoy tells us the organiser's formatting pipeline: monomer adducts only; CE given as single values or a comma-merged "20,40,60"; collision_energy_orig_units = eV; a 1–9 spectra/molecule histogram produced by thinning. If the hidden set shares this formatter (strong inference — the decoy file was produced by the same code path that will ingest the hidden set, otherwise the schema would differ), then: no dimers; [M+H]+ majority; a merged-CE row for most positive-mode precursor species; and the 1–16 range implies that some hidden molecules carry several adducts and both polarities (max 16 ≈ 4 CE rows × 2 adducts × 2 polarities, or more precursor species). Over-reading would be: assuming the 79/16 % positive/negative split, the 7-adduct list, or the 245–460 window — all three are Enamine-library and thinning artefacts, not pipeline settings.

The press release's "2,500 spectra" vs Kaggle's "~1,500" is itself informative: 2,500/400 = 6.25 per molecule, exactly enveda-180's all-adduct mean (6.31); 1,500/400 = 3.75 is what a monomer-only, ~2/3-thinned set would give (4.58 × 0.667 × a slightly heavier tail). The simplest reading is that the hidden set was acquired with the enveda-180 protocol (all adducts, three nominal CEs + merged) and formatted like the decoy. Confidence medium: it rests on two secondhand numbers.

Chemistry: 157–1,159 Da (if real) spans a far wider range than enveda-180 monomers (235–544) and resembles np-examples (precursor m/z 132–1,154; MW p10–p90 208–612). The np-examples CE vocabulary, however, is not the enveda-180 one: "40" 18.0 %, "20;50" 17.7 %, "60" 16.9 %, "35" 10.5 %, "20" 9.6 %, "80" 9.3 %, "20;40;60;80" 8.5 %, "20;40;60" 6.8 %, "0;20;40;60;80" 1.3 % (semicolon-separated, with 35 and 80 eV steps) (computed). So np-examples is Enveda's NP platform data with a different acquisition method. Which of the two the hidden set follows is unknown; the decoy's comma format weakly favours the enveda-180 formatter, but "generated on Enveda's isolation and profiling platforms" favours the np-examples instrument method. Nearest siblings: (i) all 250 np-examples structures (the only timsTOF natural products; 249 in COCONUT; 215 with NP_Score > 0; 147 with monomer precursor in 245–460; 249 with ≤ 16 spectra) — note that all 250 also have non-timsTOF spectra elsewhere in train (gnps/mona/massbank…), so they are "novelty class 1" molecules, not class 2/3; (ii) the 1,727 enveda-180 structures with NP_Score > 0 or COCONUT membership (43 in COCONUT; 1,689 NP_Score > 0; 212 NP_Score > 0.5; only 31 > 1.0) — chemically marginal NP-likes, but the right instrument and formatter.

4. Validation-split recipe with fold sizes (Q4)

# pseudocode; every count computed on train.parquet 2026-10-07
S = structures(train, instrument_type == "timsTOF")            # 183,191 structures
monomer(s)   = rows of s with adduct not starting "[2M"
merged(s)    = any row of s whose collision_energy_orig has >1 value (',' OR ';' separated)
in_window(s) = 245 <= min(precursor_mz over monomer(s)) <= 460.2   # the Enamine band; relax to 157–1,159 if the hidden-mass claim is confirmed
np_like(s)   = NP_Score(s) > 0  OR  inchikey14(s) in COCONUT-10-2026
heldout_A = { s in S : libs(s) contains "enveda-np-examples" }                              # 250
          ∪ { s in S : enveda-180 only, in_window(s), 1 <= n_spectra(s) <= 16, merged(s), np_like(s) }   # 1,727
          # = 1,977 structures, 13,556 timsTOF spectra
heldout_B = np-examples (250) ∪ enveda-180 ∩ COCONUT (43)      # 293 structures: strict database-NP fold
control   = 400 random enveda-180 structures with in_window, 1–16, merged, NOT np_like         # pool 161,074
train'    = train minus EVERY row (any ingest_lib, any instrument) whose inchikey14 in heldout  # closes the class-1 leak
queries   = for each heldout structure keep its timsTOF monomer rows; subsample rows at p=0.71 (pos) / 0.53 (neg) to mimic the decoy thinning
candidates= COCONUT ∪ train' structures, within ±10 ppm of neutral mass from (precursor_mz, adduct)
metric    = MRR@25 on InChIKey14 after RDKit 2026.03.3 tautomer canonicalisation

Sizes (computed): timsTOF structures 183,191 → enveda-180 subset 182,941 → after window + ≤16 + merged-CE 162,801 → NP_Score > 0: 1,689 (1,667 of them with no non-timsTOF train spectrum); NP_Score > 0.5: 212; COCONUT: 43 (38 timsTOF-only); union 1,727. Across all timsTOF structures NP_Score has mean −1.47, p95 −0.43, p99 +0.05, max 3.81 — i.e. "NP-like enveda-180" is the top 1 % tail of a drug-like distribution, not a natural-product set. The 250 np-examples structures are the only true NP fold and they are all leaked into other libraries, so train' must drop their gnps/mona/massbank rows too (250 / 250 have non-timsTOF rows). The important caveat: merged(s) must test for ';' as well as ',' — with the literal "20,40,60" test, 0 of 250 np-examples pass.

Spectrum-blind baselines to report alongside any model: (a) rank candidates by PubChem CID order — 49.98 % hit@1 on MassSpecGym retrieval (Liu et al. 2026, §5.2), meaning the candidate set itself carries most of the signal; (b) candidate count within ±10 ppm as the random-rank expectation (MRR ≈ (1/n)·H_min(n,25)); (c) cosine library search, which scored 0.098 on the public LB (corentinlaudicina) — any local fold on which library search scores far above 0.1 is leaking structures.

Evidence that the fold matters: shiferaxa/casmi26 reports that the np-examples fold "predicted the public leaderboard" whereas "random held-out training structures predicted the opposite ordering of three rankers and were wrong on the board"; corentinlaudicina reports 0.867 MRR on training molecules vs 0.017 on novel ones for the same model.

5. Published numbers on domain shift (Q5)

Net: the published cross-instrument penalty for library search is small (a few points for QTOF↔Orbitrap when similarity is computed well); the cross-chemistry penalty for learned models is enormous (0.867 → 0.017 for seen → unseen structures; ~10 % hit@1 for the best MassSpecGym baselines on unseen structures). For this challenge the shift that matters is Enamine → natural products, not timsTOF → anything.

Forced analogy: forensic chain of custody

Statement. A planted exemplar betrays its workshop through choices that were not forced by the content: the forger's paper, not the forger's signature. The visible test is such an exemplar; its content (spectra) is copied, so only its selection and formatting carry information about the hidden set's construction.

Derived, testable ideas. (1) Build the local validation query set by applying the recovered selection operator — drop dimers, thin monomer rows at 0.71/0.53 by polarity — to held-out timsTOF structures; if a model's local MRR changes by more than noise under this operator versus using all rows, the model is sensitive to a formatting choice the hidden set shares, and that sensitivity should be fixed before anything else. (2) Because the formatter merges CE as "20,40,60" and never emits dimers, train-time augmentation should include merged spectra built from the three single-CE rows and should exclude dimer rows from the timsTOF fine-tuning slice (28.5 % of enveda-180). (3) Predict the hidden set's spectra/molecule tail: under enveda-180 all-adduct acquisition thinned to 2/3, P(n ≥ 10) ≈ 2 % of structures; a hidden max of 16 with ~400 molecules is consistent, so multi-adduct molecules are rare and per-molecule aggregation should be designed for 1–4 spectra, with the long tail handled by simple averaging.

Does it hold? Partially. It holds for the formatter and the adduct policy (strong: a different code path would produce a different schema). It breaks on chemistry: an exemplar copied from Enamine stock cannot reveal what plant extract the real set came from — the decoy's 0 % COCONUT, 36 % halogen, NP_Score −1.47 are the paper of the forger, not the hand. It also breaks on the m/z window, which looked like a fingerprint and turned out to be the library's own bounds (99.67 % of enveda-180 monomer rows in-window). And it is silent on the acquisition method (enveda-180's 20/40/60 eV vs np-examples' 20;50 / 35 / 80 eV vocabulary), which is the single biggest unknown.

Gotchas / what could fool us

1. "245–460 m/z" is not a filter. It is the HLL-200 mass band; building a fold on it excludes exactly the large NPs (np-examples: 42 % of structures lie outside it) that the hidden set is said to contain (157–1,159 Da, if that claim is real).
2. The 7-adduct list is a sample, not a whitelist. enveda-180 monomers also include [M+C2H4O2−H]− and [M+Br]−; np-examples include [M−H2O+H]+ (2 %) and [M−2H2O+H]+. A model with a hard 7-adduct one-hot will mis-handle unseen adducts; a neutral-mass computation keyed on the adduct string must cover all monomer adducts in train (121 strings).
3. Decoy spectra/molecule (1–9) understates the hidden tail (1–16). Aggregation code must not assume ≤ 9 rows per molecule.
4. Decoy chemistry is a trap. Any feature that separates the 400 decoys from the rest of train is Enamine chemistry; 0/400 are NPs. A model that scores well on a decoy-shaped fold has learned drug-like amides.
5. np-examples are fully leaked. All 250 structures have spectra in other libraries; a fold that removes only their timsTOF rows still lets the model see them on Orbitrap. Remove by inchikey14 across all libraries.
6. CE string formats differ between Enveda slices ("20,40,60" vs "20;40;60;80"); a parser keyed on commas will treat np-examples merged rows as single-CE, and the hidden set may use either.
7. The press release says ~2,500 spectra, Kaggle says ~1,500. If the hidden set has 2,500 rows including dimers, the "no dimers" inference is wrong and dimer spectra (28.5 % of enveda-180) become relevant again.
8. Row-level duplicates in enveda-180 (several precursor species per adduct, e.g. "20|20|40|40|60|60|20,40,60|20,40,60") are near-identical spectra of the same structure; a random row split leaks them. Split by structure, always.
9. PubChem default ordering alone gives 49.98 % hit@1 on MassSpecGym — candidate-set construction is where most of the score comes from; a spectrum-blind baseline must be reported with every fold or the model's contribution is unknowable.
10. The organiser's thinning is positive-biased. If the hidden set was thinned the same way, negative-mode spectra are ~1.3× under-represented relative to acquisition; weighting training by observed polarity frequencies in test would double-count that bias.
11. Precursor m/z, num_peaks and base_peak_intensity uniquely identify every decoy row — a reminder that the hidden set, if ever partially acquired with the same protocol as any train row, would be trivially matched by this join. Enveda says "none previously published"; the 0.098 library-search score says ~10 % still match something.
12. NP_Score is a weak proxy. Only 1.2 % of random enveda-180 structures exceed 0, but 14 % of true NPs (np-examples) fall below 0; COCONUT membership is specific but hits only 43 enveda-180 structures.

Implications for the challenge

1. Use heldout_A (1,977 structures, 13,556 spectra) as the primary local fold and heldout_B (293) as the strict NP fold; always report cosine-library-search and PubChem/COCONUT-default-order baselines on the same fold; discard any fold where library search exceeds ~0.15 MRR as leaked.
2. Remove held-out structures from every library by inchikey14 (closes the np-examples class-1 leak); split by structure, never by row.
3. Apply the recovered selection operator (no dimers; thin monomers 0.71 pos / 0.53 neg) to fold queries so local spectra-per-molecule (median 3) matches the decoy; keep a second variant with all rows in case the hidden set has 2,500 rows.
4. Do not filter candidates or folds to 245–460 m/z; use the adduct-derived neutral mass ±10 ppm over 157–1,159 Da.
5. Fine-tune the spectrum encoder on timsTOF monomer rows only (822,361 enveda-180 rows + 1,184 np-examples), with merged-CE rows synthesised from single-CE triples as augmentation; down-weight dimers.
6. Make the CE parser accept both "," and ";" and both signed (negative-mode "−40") and unsigned values; map 35/50/80 eV to nearest bins rather than failing.
7. Treat the decoy strictly as a schema/ingestion test; never tune on it, never derive chemistry priors from it.
8. Request (via the Kaggle discussion) the ambiguity between 1,500 and 2,500 spectra and whether dimer adducts are present in the hidden set — the answer changes the fine-tuning slice.

Open questions

• Which acquisition method does the hidden set use: enveda-180's 20/40/60 eV scheme or np-examples' 20;50 / 35 / 80 eV vocabulary? The decoy cannot say.
• Is the organiser's 2/3 thinning random or quality-weighted (e.g. by SIRIUS explained intensity)? Train.parquet lacks the curation fields needed to tell.
• Are "157–1,159 Da, median 348" and the three novelty classes verbatim from the Kaggle data page? No opened mirror carries them.
• How many hidden molecules fall in novelty class 1 (public spectra exist)? The 0.098 library-search score suggests ~10 %, but that score also depends on similarity thresholds.
• What are the actual MIST → CASMI 2022 performance magnitudes in the reusability report (full text inaccessible here)?

Sources

• Local data: C:\Temp\casmi26\data\train.parquet, test.parquet (Kaggle download 2026-10-07); C:\Temp\casmi26\npdb\coconut\coconut_csv_lite-10-2026.csv; RDKit 2026.03.6 Contrib NP_Score (npscorer.py, publicnp.model.gz). Scripts: q_decoy.py, q_chem.py, q_thin.py, build_json.py (session scratchpad).
• research/_meta/train_eda.md; research/q2_data_forensics/enveda180_and_np_examples_anatomy.md (first 60 lines).
• mxm-1x/Enveda-CASMI-2026 DATA_AUDIT.md — https://raw.githubusercontent.com/mxm-1x/Enveda-CASMI-2026/main/DATA_AUDIT.md
• corentinlaudicina/enveda-casmi-2026 README — https://raw.githubusercontent.com/corentinlaudicina/enveda-casmi-2026/main/README.md
• kabir0774/Enveda-Casmi-2026 README — https://raw.githubusercontent.com/kabir0774/Enveda-Casmi-2026/main/README.md
• shiferaxa/casmi26 README — https://raw.githubusercontent.com/shiferaxa/casmi26/main/README.md
• ibnmarzuk/casmi26-molecule-id README — https://raw.githubusercontent.com/ibnmarzuk/casmi26-molecule-id/main/README.md
• Isaackjoshua/casmi-2026 README — https://raw.githubusercontent.com/Isaackjoshua/casmi-2026/main/README.md
• Also opened (no hidden-set text): leesirleesirlee-star, tyofthestars11-lab, AmanDubal, chiparon READMEs; eganeganegan returned 404.
• Business Wire press release via Yahoo Finance, 15 Sept 2026 — https://finance.yahoo.com/healthcare/articles/enveda-launching-next-alphafold-contest-120700676.html
• Bushuiev et al., MassSpecGym, arXiv 2410.23326 — https://arxiv.org/html/2410.23326
• Liu et al., "MassSpecGym in the Wild", arXiv 2606.19624 (17 June 2026) — https://arxiv.org/html/2606.19624
• Reusability report on MIST, Nat. Mach. Intell. 6:1296–1302 (2024), 10.1038/s42256-024-00909-4 (abstract via search snippet; full text 403) — https://www.nature.com/articles/s42256-024-00909-4
• Li et al., Nat. Methods 2021, 10.1038/s41592-021-01331-z — https://pmc.ncbi.nlm.nih.gov/articles/PMC11492813/
• King et al., JCIM 2022, 10.1021/acs.jcim.2c00620 — https://pmc.ncbi.nlm.nih.gov/articles/PMC9400100/
• Kaggle competition page (JS shell; title only retrievable) — https://www.kaggle.com/competitions/enveda-CASMI26-molecule-id-mass-spectra