Public leaderboard anatomy, top kernels and open write-ups — CASMI 2026, snapshot 2026-10-07

TL;DR. Pulled twice from the Kaggle API today (05:20 and 13:59 UTC): 2,703 teams, rank 1 = 0.481, median 0.328, and a third of all teams (885) sit on one of sixteen exact scores produced by re-running public notebooks — the biggest pile-up is 149 teams at exactly 0.425 (ranks 79–227), which is also the derived silver-medal line. The public LB is scored on roughly a third of the hidden set (~130 molecules, second-hand), so one molecule moves it by up to 0.0076 and a byte-identical rerun of one notebook moved 0.335→0.328; nothing below ~0.01 is signal. Every open write-up that reports both numbers says the same thing: ranking inside a found list is good (within-hit MRR ≈ 0.75–0.95), and the missing half of the score is molecules whose truth never enters the 25.

1. Leaderboard anatomy (own API pull)

Source: kaggle competitions leaderboard enveda-CASMI26-molecule-id-mass-spectra --download (CLI 2.2.4), zip member …publicleaderboard-2026-10-07T13:59:36.csv, saved as lb_snapshot_2026-10-07.csv beside this file. Columns: Rank, TeamId, TeamName, LastSubmissionDate, Score, SubmissionCount, TeamMemberUserNames. A second pull from 05:20:19 UTC the same day (2,673 teams) is used for the intra-day diff.

The q7 digest figures (2,673 teams, max 0.480, 263 ≥ 0.42, 35 ≥ 0.43) are reproduced exactly by the 05:20 file; they were simply eight hours old. In 8.6 hours the board gained 30 net teams (31 new), 187 teams improved, the ≥ 0.43 club grew from 35 to 49 and the ≥ 0.42 club from 266 to 317; 580 teams had already submitted on 2026-10-07 by 14:00 UTC (383 on 10-06, 224 on 10-05) — the public-notebook wave is accelerating, not plateauing.

Medal cut-offs (Kaggle rule for competitions with > 1,000 teams: gold = top 10 + 0.2 % of teams, silver = top 5 %, bronze = top 10 %; I use floor on the percentages): gold = rank 15 → 0.436; silver = rank 135 → 0.425; bronze = rank 270 → 0.421. The silver line passes through the middle of the 149-team 0.425 cluster (ranks 79–227) and the bronze line sits on the 0.421 cluster (ranks 245–283); Kaggle breaks exact ties by earlier submission time, so which clone lands inside the medal zone is decided by the clock, not by the model.

Exact-score clusters (the "tutorial / public notebook" signature). 885 of 2,703 teams (32.7 %) score one of these sixteen values to three decimals:

Submissions vs score. Spearman(SubmissionCount, Score) = 0.54. Band medians: ≥ 0.44 → 57 submissions (n = 12); 0.43–0.44 → 36 (n = 37); 0.42–0.43 → 20 (n = 268); 0.40–0.42 → 14; 0.35–0.40 → 11; 0.30–0.35 → 7; < 0.20 → 3; zero scores → 2. 377 teams have exactly one submission, 842 have ≤ 3, 123 have ≥ 50; the busiest team (kglctf, 113 submissions) sits at 0.411, i.e. volume without a new idea buys a clone score. The top-12 average 55 submissions in 23 days (~2.4/day of the 5 allowed).

2. Top-voted public kernels (API, kernels list --sort-by voteCount, 200 rows saved as kernels_2026-10-07.csv)

Method-tool census over the 200 listed titles plus 14 pulled notebooks: DreaMS — used as a tail filler (bobthebot369 V18, uninhibitedscholar own153), as a demo port of the HF space with internet ON (hengck23/dreams-structure-from-spectrum-retrival-demo, 7 votes, MassSpecGym embeddings), and in three zero-vote titles (rajrajak99 "v115 DreaMS Foundation", haideptry "Mass DreaMS Champion", imranarif536 "V42 DreaMS ICE Cap10"); two independent negatives (prvsiyan, megayak; corentinlaudicina on GitHub: 0.811 vs 0.867 cosine). MIST: one title (franciscoangulo/enveda-casmi-v60-qtof-msbuddy-mist-dedup, 1 vote). FIORA: lehau007 "V32 FIORA Tri-Forward 0420" (1 vote). ICEBERG/GLACIER (MIT ms-pred): in every ≥ 0.37 kernel. MS2DeepScore, SIRIUS, PRISM: zero kernels in 200 titles or pulled code. Library search of enveda-180: every retrieval kernel indexes it (it is 45 % of train); only the host tutorial excludes it. Decoy exploitation: none scores against it; the ≥ 0.38 family detects it (MD5 of spectrum_id = 654e030e…, or SHA-256 of test.parquet = 2fa29c4a…, or env KAGGLE_IS_COMPETITION_RERUN) to run a 12-molecule smoke commit and the full 400 only on the scoring rerun — the decoy is exploited for wall-clock, not for score. Forks: Kaggle's kernel_sources field is empty on 19 of 20 pulled kernels, so lineage is only recoverable from markdown attribution (the V44/"Forensics Lab" text names ten ancestors); the AmanDubal repo is the only explicit fork of the inversion tutorial I found, and it fixes the tutorial's "same-token reconstruction objective" and "generation passes only the most recent token without a KV cache".

3. GitHub write-ups (raw READMEs fetched; Kaggle facts therein are mirrors)

4. Organiser statements and contested facts (all second-hand; forum unreachable by API)

• Decoy thread 741404: cited by mxm-1x README as "host clarification" that the visible file "is a dummy test file" replaced at rerun. No quote of the host's text survives in any README or kernel I opened. The Data-page mirror (dalloliogm; same text in dreaddevelopment kernel) says the visible file "is comprised of examples from the training dataset; it will be replaced by the hidden test set during re-run, and will be approximately the same size."
• Public/private fraction: kitopl kernel table: "Public LB ~33 % of the hidden set … roughly 130 molecules; a single molecule moves it by up to 0.0076, so differences around 0.005 are noise." Attribution is not stated (reads like the Kaggle LB-page banner); treat as plausible but unverified. Corroborated by prvsiyan's identical-code rerun 0.335 → 0.328 (Δ 0.007).
• Rerun limit: "CPU/GPU notebook ≤ 9 h, internet off" appears in NoName7n11, corentinlaudicina, yasumorishima; seyitkaangunes reports actual hidden-test runtime 3.5–4.5 h on T4 ×2; leesirleesirlee-star measured ~18 s/molecule end-to-end.
• ~1,500 vs ~1,213 spectra: Data page says "~1,500 spectra of ~400 molecules" for the hidden set; the visible decoy has 1,213 spectra/400 molecules; leesirleesirlee-star notes the graded submission file is 530,716 bytes vs 393,908 bytes for the public run — the hidden set has different molecule_ids (not the same 400). The ten-adduct list on the Data page vs seven in the decoy (kitopl) is a related discrepancy. I found no mention of "~2,500" anywhere.
• Pretrained weights: prvsiyan: "The host has confirmed that open pretrained weights are allowed provided they are distributed in compliance with the licence of the data they were trained on." Instrument: mohammadhmozafary: "the competition organizer has mentioned that the test set only contains [timsTOF]." Both unquoted paraphrases.
• Class shares: nobody has them; the forum back-solved class 1 ≈ 16–17 % from library-only = 0.151 (0.151/0.873 = 17.3 %, kitopl). nghianguyen7171 estimates 16/45/39; yasumorishima 16/27/55 — the class-2/3 split is genuinely contested (Δ 18 points), and it decides whether retrieval caps at ~0.45 or ~0.61.

5. How fast has the top moved? (dated snapshots, all second-hand except today's)

Rank 1 moved 0.425 → 0.471 → 0.481 in 11 days (+0.056), the bronze line 0.342 → 0.401 → 0.411 → 0.421 (+0.079 in 10 days), while the median has been frozen at 0.328 since at least 10-05 — the prvsiyan-rerun cluster. The same 0.417 score was rank 111 on 10-05 and is ranks 344–417 today: a public-notebook score decays ~30 ranks/day.

Forced analogy: law of evidence — standard of proof and burden allocation

Statement. A submission row is a 25-seat jury that must convict exactly one defendant; MRR pays 1/seat-number of the first correct conviction and nothing for acquittals. The "primary hypothesis" H (chosen formula / curated pool / adduct) is the prosecution's theory; "hedges" A (second formula, other adduct, PubChem-only isomer, derivative) are alternative theories. The burden question is: how many seats must the prosecution's theory fill before an alternative theory earns a seat, as a function of P = P(H true)?

Derivation. Let f(k) be the probability that, given its theory is true and the truth is in its list, a theory's ranker puts the truth at list-rank k. Fit f(k) ∝ k^−s on 1..25 to the one measured within-hit curve we have (leesirleesirlee-star: ranking-only MRR 0.765) → s = 2.073, which implies 64.4 % of hits at rank 1, 15.3 % at rank 2, 6.6 % at rank 3, 92.3 % within top-5. Expected MRR of any seating is Σ_i π_i / i where π_i is the probability the item in seat i is the truth; because exactly one item is true, the rearrangement inequality makes the optimum simply sort every candidate from every theory by its posterior π. For block seating (m primary seats then hedges) the m-th primary seat is worth P·R_H·f(m)/m and the first hedge seat (1−P)·q·R_A·f(1)/m, so the hedge enters at seat m* = ⌊(P·R_H / ((1−P)·q·R_A))^{1/s}⌋ + 1 (q = P(alternative theory true | H false); R = within-theory recall@25). Numerically with R_H = R_A, q = 1 (ev_model.py in scratch):

If the alternative theory's list is only half as good (q·R_A/R_H = 0.5): best m = 2 / 4 / 11 at P = 0.5 / 0.7 / 0.9, EV 0.431 / 0.553 / 0.691 vs all-primary 0.383 / 0.536 / 0.689. Rule: unless the primary theory is > 90 % certain, the second seat belongs to the best alternative theory, and the primary's own rank-2 candidate goes to seat 3; "preponderance" (P ≈ 0.5–0.7) means strict alternation; only "beyond reasonable doubt" (P ≥ 0.95) justifies four or more consecutive primary seats. The hedge is cheap because f(k) falls as k^−2: the primary's third candidate carries 6.6 % of its mass, the alternative's first carries 64 % of its. Testable: on the enveda-np-examples backtest, replace block seating (curated top-25, PubChem only in slots 4/8/12 or 5 tail slots — exactly what V44 does) with posterior-sorted interleave; the model predicts +0.02–0.04 when the gate P is miscalibrated below 0.9, and ≈ 0 when the library lock (lib_max ≥ 0.9) is already correct.

Back-solve check against write-ups (within-hit MRR = MRR / recall@25; implied top-1 share from the fitted Zipf):

Consistency: the Zipf fit reproduces chiparon's measured top-1 (0.90 predicted vs 0.896 measured) — the only place the shape can be checked — and leesirleesirlee's independent "head p ≈ 0.30–0.35" rank-1 share matches the 0.35 implied here. The public LB's 0.765 lies between prvsiyan's class-2 ranking (0.52–0.73, recall forced to 1) and the near-perfect class-1 numbers (0.93), which is what a 17 % class-1 / 83 % class-2+3 mixture should give. So the claim "~45 % of hidden molecules never reach the 25" is arithmetically consistent across three independent write-ups, though it rests on one simulated 0.765. Random-ranker reference: MRR@25 = H₂₅/n = 0.153 (n = 25), 0.068 (56), 0.036 (107), 0.0008 (4,652) — the PubChem window of Isaackjoshua is a jury of 4,652 with no judge.

Where the analogy holds, partially holds, breaks. Holds: seats are scarce, early seats are worth more, and the burden shifts with the prior exactly as the odds formula says. Partially: a jury convicts once; here a later correct seat still pays (1/k), so the "standard of proof" is continuous, not a threshold. Breaks: (i) there is no cost to a wrong conviction — Isaackjoshua's "tiered" PubChem fill "cannot lose ground" because empty seats score zero anyway, so every row should carry 25 distinct InChIKey14s (377 single-submission teams and many baselines leave seats empty); (ii) a duplicate InChIKey14 (two tautomer spellings, stereoisomers, Kekulé vs aromatic) is a juror who has already voted — dariushafshar measured 10.42 % of COCONUT keys change under the scorer's canonicalisation, 2.30 % can disagree, costing 0.0004 MRR when keyed naively on COCONUT but +0.105 on the affected molecules; (iii) the "jurors" are not independent — same-formula isomers are the entire problem (70.8 % of window candidates share the true formula; median 46 isomers), so hedging across formulas is cheap while hedging across isomers within one formula is where f(k) is flat and expensive.

Gotchas — what could silently fool a modeller or a reader

1. The public LB is a 130-molecule sample. One molecule = up to 0.0076; identical code reran 0.335 → 0.328; nghianguyen7171's byte-identical reruns landed 0.011 and 0.105 below advertised scores. Any delta < 0.01 is noise; the top-50 (0.429–0.481) spans 7 "molecules".
2. Exact-score clusters are clones, not convergence. 32.7 % of teams sit on sixteen values; the silver line is the 0.425 clone score. A team count or "rank 100 = 0.425" tells you about notebook popularity, not about the difficulty of the problem.
3. Submission count correlates with score (ρ = 0.54) because clones need ≥ 1 submission and real work needs 50 — not because iteration on the public LB works; yasumorishima's log shows 30+ submissions moving 0.275 → 0.303 while local CV moved 0.52 → 0.66.
4. Local validation inverts on the LB. Three independent reports: random held-out train structures predicted the opposite ranker order (shiferaxa); LGB CV 0.65 → LB 0.279 < no-LGB 0.294 (yasumorishima); "a decomposition fitted to our own anchors yields a negative class-1 weight" (nghianguyen7171). Only the 250 enveda-np-examples under three masks has predicted LB order.
5. The decoy detector is a landmine. Kernels switch to full runs on sha256(test.parquet) != 2fa29c4a… or MD5 654e030e…; if Kaggle ever changes the placeholder file, those notebooks silently run the 9-h path on commit.
6. ICEBERG/GLACIER "worth ≈ 0" was a budget artefact: ICE scored 71 of 400 molecules (18 %) with budget 300 s and the score still read 0.417; the 0.421 notebook uses 5,400 s. Fixed wall-clock budgets do not scale with molecule count — a backtest with 2,000 queries starves them.
7. Bigger pool ≠ more recall in practice: PubChem top-25 isomers took prvsiyan's class-2 sim 0.732 → 0.379; Enamine tier (92.3 % of enveda-180) cost −0.013 LB; +62,744 ChEBI/LIPID MAPS = +0.000; popularity priors are +0.004 to +0.011 in one lineage and monotonically negative in another (μ 0.2 → −0.075).
8. Advertised titles lie by version: "0.421" notebook scored 0.420 on v1; "V18 Sovereign Zenith [Target 0.440+]" replays at 0.425; "Protected Bio-DB Tail" says "new score unknown". Use the LB cluster, not the title.
9. Kaggle's kernel_sources is empty on 19/20 pulled kernels — fork lineage must be read from markdown; the ten-ancestor list in the V44 text is the best provenance map available and is itself a competitor's claim.
10. Keying candidates by the library's own InChIKey14 instead of the scorer's canonicalisation costs ~0 on COCONUT overall but 0.105 on the 2.3 % of structures where it disagrees (dariushafshar) — and more on de-novo SMILES in odd tautomeric forms.
11. Ties are broken by submission time: inside the 149-team 0.425 block, medal/no-medal is decided by who ran the clone first.
12. Team-merge arithmetic: 95 % of teams are solo; the 12 teams ≥ 0.44 include five-person teams (Arsokan) and a three-person team with 96 submissions (Sls2) — merging pools submission budget (5/day/team) and GPU quota (30 h/week/account, the real bottleneck per leesirleesirlee-star: "~6 variants per week").

Implications for the challenge

1. Treat the public LB as a ±0.01 instrument and never select between variants on it alone; use the 250 enveda-np-examples three-mask backtest (shiferaxa, leesirleesirlee-star Tier 1, paired s.e. ±0.002) as the primary selector and the LB as a sanity check.
2. Clone-floor is 0.425 today (V18 Sovereign Zenith replay) and decays ~30 ranks/day; our first real submission should reproduce 0.425 to anchor our environment (as leesirleesirlee-star did with V44), then measure every change as a paired delta on the backtest.
3. Fill all 25 seats with distinct scorer-canonical InChIKey14s, every row — zero-cost, and the posterior-sorted interleave rule above says the second seat goes to the best alternative theory unless P(primary) > 0.9.
4. Target recall@25 explicitly: it is ≈ 0.545 at 0.417 and nobody in the public tree optimises it; the mxm-1x and Isaackjoshua numbers show ranking inside a found list already converts 58–77 % of hits to rank-1 share.
5. Budget forward models by molecule count, not wall-clock; run ICEBERG/GLACIER on the library-less molecules first (the 0.421 notebook's ordering) and verify coverage (n_mols_scored) in logs on every scoring rerun.
6. Do not add PubChem as a pool; join ≤ 50 PubChem-only candidates into the main ranker under a library gate (lib_max < 0.7) — the one pool change with a reproducible +0.011.
7. Keep two final picks on different theories (e.g., one library-locked, one interleaved) — the private 67 % can reorder a 0.01-wide band of 300 teams.
8. Settle the class-2/3 split (16/45/39 vs 16/27/55) with a library-free probe submission (all-COCONUT retrieval, no train spectra) — it fixes the retrieval ceiling (≈0.45 vs ≈0.61) and therefore how much to invest in de novo.

Open questions

• Is the public/private split really 33/67 and ~130 molecules? (kitopl, unattributed) — readable only on the Kaggle LB page.
• What does thread 741404 actually say, and did the host address the 1,213 vs ~1,500 and seven-vs-ten-adduct discrepancies?
• What produces the 77-team 0.411 and 14-team 0.146 clusters?
• Does any team > 0.44 use anything outside the prvsiyan → v4n → fusion lineage? Rank 2 (MarvinTMB, 0.466 with 12 submissions) is the strongest hint of an independent method.
• Does the k^−2 within-list rank curve hold per class, or is class-2 flatter (prvsiyan's 0.52 at recall 1 implies s ≈ 1.3)? A per-class curve changes the interleave rule.

Sources

Kaggle API (own pulls, 2026-10-07): kaggle competitions leaderboard … --download at 05:20:19 and 13:59:36 UTC (lb_snapshot_2026-10-07.csv); kaggle kernels list --competition enveda-CASMI26-molecule-id-mass-spectra --sort-by voteCount|scoreDescending|dateRun --page-size 100 -v pages 1–2 (kernels_2026-10-07.csv); kaggle kernels pull of: inversion/casmi-denovo-tutorial-notebook, haideptry/enveda-casmi-2026-fast-spectral-cosine-baseline, evgendvorkin/enveda-casmi-2026, prvsiyan/analog-propagation-casmi-2026-baseline, bobthebot369/enveda-casmi-2026-v18-sovereign-zenith, dmitriigluzdov/casmi-26-from-spectra-to-structures, seyitkaangunes/casmi26-v4n-engine-fusion-union-lb-0-399, huseyinemreaksoy/casmi26-v4n-fusion-pubchem-on-public-0-421, imranarif536/casmi26-v44-pairtail-locked-top1, uninhibitedscholar/enveda-casmi-2026-own153-public-lb-0-417, hengck23/dreams-structure-from-spectrum-retrival-demo, denpugovkin/casmi-2026-protected-bio-db-tail, kozykappa/enveda-0-37-lb-score, dariushafshar/casmi26-10-4-of-keys-change-0-48-cost-you, plus earlier pulls of kitopl/casmi-2026-data-pitfalls-that-break-your-cv, josefreitasalvesneto/casmi-2026-a-complete-statistical-tour, mohammadhmozafary/casmi-2026-domain-knowledge-for-non-chemists, dreaddevelopment/casmi-2026-data-metric and the official metric notebook metric/casmi-mean-reciprocal-rank (read, never executed). GitHub (raw READMEs/docs, mirrors of Kaggle facts): https://github.com/leesirleesirlee-star/Enveda_CASMI-2026_Molecule-ID-From-Mass-Spectra (README, docs/LEADERBOARD.md, docs/PLAYBOOK.md); https://github.com/gnshx/Enveda-CASMI-2026---Molecule-ID-From-Mass-Spectra (README, progress/ROADMAP_AND_PROGRESS.md); https://github.com/nghianguyen7171/casmi26; https://github.com/Isaackjoshua/casmi-2026; https://github.com/corentinlaudicina/enveda-casmi-2026; https://github.com/yasumorishima/kaggle-competitions (enveda-casmi26/CLAUDE.md); https://github.com/billylu24/Enveda-CASMI-2026---Molecule-ID-From-Mass-Spectra; https://github.com/mxm-1x/Enveda-CASMI-2026; https://github.com/chiparon/enveda-casmi26; https://github.com/shiferaxa/casmi26; https://github.com/AmanDubal/Enveda-CASMI-2026---Molecule-ID-From-Mass-Spectra; https://github.com/kabir0774/Enveda-Casmi-2026; https://github.com/AayanShabbir/casmi26; https://github.com/NoName7n11/Enveda_CASMI_2026 (OVERVIEW.md, WORKFLOW.md; README 404); https://github.com/dalloliogm/kaggle_competitions (Data-page mirror); https://github.com/OfficialBhattacharya/kaggle_CASMI_2026; https://github.com/Clyde0513/CASMI-2026; https://github.com/bobownYAO/Enveda-CASMI-2026-Molecular_Formula; https://github.com/ibnmarzuk/casmi26-molecule-id; https://github.com/tyofthestars11-lab/casmi26-molecule-id. Not reachable: https://www.kaggle.com/competitions/enveda-CASMI26-molecule-id-mass-spectra/discussion/741404 (JS shell; title only). Kaggle medal rule as applied: gold = top 10 + 0.2 %, silver = top 5 %, bronze = top 10 % for > 1,000 teams (Kaggle progression rules, applied from memory of the published table — verify on kaggle.com/progression).