CASMI 2012-2022: every edition's winners, methods, scoring rules and the lessons that carry to MRR@25

expertconfidence highrelevance 4/515 sources5k words
Forced analogyLaw of evidence / precedent: each CASMI scoring rule is a standard of proof; MRR@25 is an absolute-but-graded standard sitting between 2017's F1-points (more lenient at rank 2, cuts off at 10) and 2022's single-answer top-1 (strictest). The analogy held well enough to yield one quantitative rule (sort by calibrated posterior; hedge formula/adduct hypotheses in slots 6-25) and one precedent (every edition without a provided candidate list collapsed) - it breaks where 'evidence' is reusable across cases (training-set memorisation has no courtroom analogue).

TL;DR. Across six CASMI editions (2012, 2013, 2014, 2016, 2017, 2022) the pattern never changed: when the true structure was in a provided candidate list or in a spectral library, machine learning (CSI:FingerID/IOKR) or metadata (reference counts, library hits) solved 40-90 % of challenges at rank 1; when the compound was absent from libraries and no candidate list was given (2014 "unknowns", 2022), top-1 collapsed to 0-38 %. Winners were never decided the way MRR@25 scores: 2012-2016 used a per-challenge "best rank among participants" tally (a rank-100 answer could win), 2017 used Formula-1 points over ranks 1-10, 2022 used single-answer exact match with no ranking at all. The 2026 metric is therefore a stricter standard than 2012-2016, more lenient at ranks 11-25 than 2017, and far more lenient than 2022 - and every precedent says the candidate pool, not the ranker, sets the ceiling.

1. Edition-by-edition table

All numbers are from the official casmi-contest.org results pages (fetched 2026-10-07 via curl; the site's TLS certificate is a SourceForge wildcard, so browsers warn) and the cited papers. "Top-1" = correct InChIKey block-1 ranked first. The JSON twin of this table is at casmi_history_2012_2022_winners_methods_lessons.json.

Edition (organiser) Challenges / data Categories Winner (method) Reported numbers Scoring rule
2012 (Schymanski & Neumann, Eawag/IPB Halle) 14 LC-HRMS/MS (Bruker micrOTOF-Q, <=5 ppm, 12k res; Elbe-river environmental + plant compounds) + 16 GC-MS; only LC categories got external entries Cat 1 formula, Cat 2 structure (LC); Cat 3/4 GC Cat 1: Dunn et al., Birmingham (manual + KEGG/ChemSpider/MetFrag). Cat 2: H. Oberacher (spectral library search: MassBank/METLIN/NIST/Wiley) Cat 1: Dunn 11 submitted, 9 correct rank-1; Shen (FingerID) 14 subm., 8 correct, avg rank 2.88; Duhrkop (SIRIUS) 14 subm., 8 correct, avg rank 2.25. Cat 2: Oberacher 5 submitted, 3 correct, all rank 1 (avg 1.0); Dunn 3/11; Shen 4/14. Internal MetFrag/MetFusion/MOLGEN not eligible Per challenge, best rank among participants wins; overall = most wins
2013 (MSSJ Japan, Nishioka et al.) 16 molecules, 12 formula + 16 structure challenges; mixed instruments (Shimadzu IT-TOF, LTQ Orbitrap Velos, Bruker FT-ICR, Waters Xevo QTOF, MALDI-QIT), <=10 ppm; metabolites, agrochemicals, synthetics Cat 1 formula, Cat 2 structure; manual vs automatic flagged separately Both: Newsome & Nikolic, UIC (manual: fragment/neutral-loss formula analysis + ChemSpider, SciFinder, Reaxys, Google Scholar). Best automated tool: MAGMa (Ridder) Newsome/Nikolic 12/12 formulas, 14/15 structures rank 1 (+1 "highly commended"). SIRIUS (Duhrkop/Bocker) 10/12 formulas. MAGMa 8/8 formulas, 7/12 structures. MathSpec (Sweeney) 8(1)/11 automated. Other automatic 7(1)/16, 2/12 Same per-challenge win rule; one submission per challenge
2014 (Nikolic/UIC, Vaniya, Fiehn et al.) 48 challenges: 1-15 mammalian biofluids (Orbitrap Velos, Q Exactive), 16-31 natural products, 32-42 synthetics (Waters SYNAPT QTOF, Shimadzu IT-TOF), 43-48 "unknowns" (no SciFinder hit); winners decided on 1-42 Cat 1 formula, Cat 2 structure Cat 1: Lars Ridder (MAGMa) - 34/42 rank-1, tie with Vaniya and Allen broken by two rank-2 answers; post-contest incl. 43-48: Ridder 40/48, Allen 38/48, Vaniya 36/48. Cat 2: Felicity Allen (CFM-ID) - 29 challenge wins, 24/42 rank-1 Unknowns 43-48: nobody ranked one first; only Ridder (rank 2) and Allen (rank 6) hit one (page text says ch. 47; the table lists it as "challenge46(isomer2)") Per-challenge win; ties use worst-case rank
2016 (Allard/Wolfender Geneva + Schymanski/Neumann) Cat 1: 19 natural-product challenges (Agilent 6540 / Waters QToF; ch. 3 removed). Cat 2/3: 208 challenges (127 pos [M+H]+, 81 neg [M-H]-) from 188 substances, Q Exactive Plus, stepped HCD 20/35/50, plus 312 training spectra (285 substances); ChemSpider candidate lists (+-5 ppm) containing the correct answer provided, mean 816-1114 candidates Cat 1 manual NP; Cat 2 in-silico only; Cat 3 full information (RT, libraries, patents, reference counts) Cat 1: Dejan Nikolic (manual) 15 wins; Vaniya 14; Kind 12. Cat 2: Brouard (CSI:IOKR) 86 gold; Duhrkop CSI:FingerID 1.0.1 82 gold (positive mode only); Vaniya MS-FINDER 1.62 70. Cat 3: Kind (MS-FINDER 1.60 + NIST/MoNA search + STOFF-IDENT) 159 gold; Allen CFM-ID retrained + DB 156 Cat 2 Top-1: CSI:FID 70 (of 127 pos), IOKR 62, MS-FINDER 46, CFM 39, MAGMa+ 24; Top-10: 100-123. Cat 3 Top-1: Kind 146/208 (70 %), Allen 120; Top-10 174/182 Gold/silver/bronze per challenge, Olympic tally; ties = average rank; stereo collapsed to InChIKey block 1; also reported Top-1/3/10, RRP, Formula-1 score, up to 3 entries per team
2017 (Nikolic, Schymanski, Neumann et al.) 243 natural-product challenges, Waters SYNAPT QTOF (1-45: MS+MS/MS, plant NPs, a few PubChem/SciFinder-absent analogs; 46-243: MSE MS/MS only, 46-131 neg, 132-243 pos); Cat 4 added June 2017 with fixed PubChem candidate lists (6 ppm, 6 adducts) for 46-243 Cat 1 manual (1-45), Cat 2 in-silico only, Cat 3 full info, Cat 4 fixed-candidate ranking No winner is declared on the page ("And the results are ..."). Highest F1: Cat 1 kai = Duhrkop CSI:FingerID/SIRIUS 3.5 (F1 469, 11 Top-1/45); Cat 2 kai_iso (F1 2775, 77 Top-1, 147 Top-10, 4 misses of 243); Cat 3 tkind1 = Kind MS-FINDER 2.14 + plant target DB (F1 3707, 91 Top-1, 193 Top-10, 34 misses); Cat 4 kai_iso (66 Top-1, 119 Top-10 of 198) Cat 4 others: IOKR-transfer 36 Top-1; Li/Kuhn/Bork (EMBL) ML spectrum prediction 10; MetFrag 7; MetFrag+ 6. Cat 3 Tsugawa+Yamamoto MS-FINDER x CSI:FingerID rank-sum voting 28 Top-1 Formula-1 points 25/18/15/12/10/8/6/4/2/1 for ranks 1-10, summed
2022 (Fiehn / WCMC UC Davis, Vaniya) 500 compounds (250 priority + 250 bonus; 81, 282, 432, 476 voided), raw LC-MS/MS mzML/.RAW, +ESI and -ESI (each challenge in one mode), RT + precursor m/z given, inclusion list, 0.8 s dynamic exclusion; "metabolites and exposome chemicals" not in public spectral libraries; no candidate lists; MassIVE MSV000089239 4 tasks: adduct, formula, structure class, 2D structure; single answer per challenge No formal winner ("trial ... to gauge which methods are used"). Best 2D-structure rate: Nikolic_KUCA (manual, 55 attempts) 21/55 = 38 %; best large-scale: Duhrkop (SIRIUS/CSI:FingerID) 57/222 = 26 % priority, 64/219 = 29 % bonus; Nothias and Rutz 58/228 = 25 %; Tsugawa (MS-DIAL/MS-FINDER) 30/241 = 12 % Formula: Duhrkop 208/222 = 94 %, Nothias 93 %, Rutz 91 %. Adducts: Nikolic 94 %, Duhrkop 93 %, Nothias 92 %. Classes: Duhrkop 116/168 = 69 %, Wang 67 % Exact InChIKey block-1 match of the single submitted answer; percent of attempted

Participant counts: 2012 three external teams per LC category; 2013 seven teams; 2014 seven teams (six in Cat 2); 2016 "more participants than any CASMI before" (six Cat 2 teams incl. internal MetFrag); 2017 ~8 participant IDs; 2022 16 groups (Toronto, Geneva x2, UBC x2, US EPA, Strasbourg, UIC, Alberta, Luxembourg, TUAT, Jena, Nanjing, oloBion, OmicsCraft, Indiana).

2. CASMI 2016 (Schymanski et al. 2017, J Cheminform 9:22, DOI 10.1186/s13321-017-0207-1): automatic vs metadata

Headline: "The best methods were able to achieve over 30% Top 1 ranks in Category 2, with all methods ranking the correct candidate in the Top 10 in around 50% of challenges. This success rate rose to 70% Top 1 ranks in Category 3, with candidates in the Top 10 in over 80% of the challenges."

What the post-contest decomposition showed (Table 5 of the paper, 208 challenges, Top-1 counts):

Entry Top-1
Reference count alone (ChemSpider reference count) 143
Lowest ChemSpider ID alone ("gut feeling" prior) 113
MoNA spectral-library score alone 70
Retention-time prediction alone 1
Combined MS/MS (six fragmenters equally weighted: CFM, CSI:FID, CSI:IOKR, MAGMa+, MetFrag, MS-FINDER) 82
Combined MS/MS + RT + Refs (+MoNA) ~164 (from fetched table summary; medium confidence on exact value)

The single most repeated lesson is metadata/prior-knowledge ranking, conditioned on a candidate set that contains the answer: "The references outperformed all individual metadata categories and even the combined fragmenters clearly." And the authors' own caveat, which is the one that matters for 2026: "for new (emerging) anthropogenic substances and transformation products ... these strategies would not work so well ... This situation is also likely to be drastically different for natural products and metabolites, where many more closely-related substances or even isomers could be expected."

Other quoted lessons:

  • Memorisation: Team Duhrkop "observed that the top-scoring candidate matched a compound from the CSI:FID training set in 67 challenges"; the true leave-out retrain fell from 70 to 36 Top-1 in positive mode, yet "still had more Top 1 ranks than any other non-CSI method". On the globally-excluded subset "the percentage of maximum Top 1 ranks dropped from 34 to 18%", while Top-10 "still remained around 40-50%".
  • Negative mode: Brouard trained on 7300 positive vs 816 negative spectra; MS-FINDER (14 Top-1) and CFM (12) beat IOKR (9) in negative mode.
  • Complementarity: "The Combined MS/MS entry outperformed all of the individual Category 2 entries".
  • Candidate-list size is non-monotone: "fewer Top 1 entries and higher median/mean ranks were observed for the challenges with moderate candidate numbers (200-1000 candidates)".
  • Spectral-library sanity check: a library built from CSI:FingerID's own training data returned candidates for 91/127 positive challenges, correct for 63 of 69 present, but the mean cosine between challenge and training spectrum of the same compound was only 0.76 (one below 0.01).
  • Scoring-rule critique: "the winner could have rank 100, if all other participants were worse ... it creates cross-dependencies between participants".
  • Future: "Challenges for future contests remain true unknowns ... distinguishing (positional) isomers is likely sufficient challenge for the next few years."

Blazenovic et al. 2017 (J Cheminform 9:32, DOI 10.1186/s13321-017-0219-x) re-ran four open tools on the same 312/208 spectra: stand-alone Top-1 on the challenge set MetFragCL 25 %, CFM-ID 14 %, MAGMa+ and MS-FINDER <14 %; adding database-presence (MS-FINDER's 220,213-entry local DB, 2x boost for STOFF-IDENT) and MS/MS-library hits (NIST14 + MassBank, 4x boost) lifted CFM-ID to 86 % and the best consensus to 87 % Top-1 (93 % on training); library matching alone gave 60 %; "boosting alone yielded 286 correct hits" of 312 on training vs max 69 for any in-silico consensus. Their own verdict: "in silico algorithms alone are still far away from practical use for identification of true unknowns".

Note on the brief's citation: the "Nikolic et al. 2017" paper is "CASMI 2016: A manual approach for dereplication of natural products using tandem mass spectrometry", Phytochemistry Letters (Feb 2017), not J Mass Spectrom; the Cat 2/3 summary is Schymanski et al. 2017. I could not open the Phytochemistry Letters full text (publisher block); the official page credits Nikolic with 15 wins.

3. CASMI 2017 specifics (Category 2/3/4 pages)

  • 243 challenges, all natural products, Waters SYNAPT QTOF (MS mode <=10 ppm, MS/MS ~10 ppm / 2 mDa). Only challenges 1-45 had MS1 spectra; 46-243 were MS/MS only (MSE), and ion species were not given, so Category 4 candidate lists were pulled from PubChem (mirror 2017-02-03, MetFrag 2.4.2) at 6 ppm for [M+H]+/[M+Na]+/[M+NH4]+ and [M-H]-/[M+Cl]-/[M+FA-H]-. There was no MS1-only category; the "formula" category of 2013/2014 (Cat 1 "Best Molecular Formula") was retired in 2016.
  • CSI:FingerID (SIRIUS 3.5, kai_iso) results: Cat 2 Top-1 77/243 (32 %), Top-3 110, Top-10 147 (60 %), 4 misses, median rank 5; split 55 positive / 22 negative Top-1. Cat 4 (fixed candidates, 198 challenges): Top-1 66 (33 %), Top-10 119 (60 %). The kai112 variant (no hard formula threshold, NIST-augmented training) scored 38 Top-1 with 102 misses in Cat 2 but 55 Top-1 / 0 misses in Cat 4 - a formula-selection failure mode, which Duhrkop attributed to "no isotope patterns for the precursor were given".
  • Cat 3 winner-equivalent tkind1 (MS-FINDER 2.14, target DB = UNPD + plant metabolite list only, 5 min 17 s batch run) reached 91 Top-1 / 193 Top-10 but with 34 misses: restricting the database to plant metabolites is the whole trick, and it costs recall.
  • IOKR variants ranked 31-36 Top-1 in Cat 4 with median ranks 201-396 (poor tails), MetFrag 7, and the EMBL spectrum-prediction ML approach 10 - in 2017 a fingerprint-prediction method beat spectrum-prediction methods by 6x on Top-1.
  • Challenge 15 was excluded after a data fix; evaluation used Formula-1 points with ties averaged and non-integer ranks rounded up.

4. CASMI 2022 (Fiehn/WCMC)

  • Scale: 500 compounds (priority 1-250, bonus 251-500), raw LC-MS/MS in both polarities; each challenge in one polarity; RT and precursor m/z supplied; evaluation by exact InChIKey block-1 match of a single answer ("there is no candidate list. We are only looking for your best-ranked compound"). Results due 2022-06-06; workshop at Metabolomics Society 2022, Valencia; MANA Sept 2022.
  • Instrument: not stated on any page I could open (results, rules, FAQ, MassIVE MSV000089239 summary). Evidence: participants could request ".RAW instead of .mzml" (Thermo format) and file names such as A_M24_posPFP_01 (PFP column); the WCMC primarily runs Thermo Orbitrap-class LC-MS/MS. Treat "Orbitrap" as likely but unverified.
  • How "winners" were decided: they were not. The page calls it a "trial ... to gauge which methods are used by the community" and reports four tables of percent-correct-of-attempted. Attempt counts range from 22 (patRoon) to 249, so percentages are not comparable across groups.
  • Top results (priority set): 2D structure - Nikolic_KUCA 38 % (21/55), Duhrkop 26 % (57/222), Nothias 25 %, Rutz 25 %, Mehta 21 %, Nikolic_POSO 19 %, Xing 15 %, Tsugawa 12 %, Namini/Mosley (EPA) 11 %, Wang 9 %, Wei_MSKU 5-7 %, Young 0/130, Elser 0/97, Hong 1 %. Formula - Duhrkop 94 %, Nothias 93 %, Rutz 91 %, Nikolic_POSO 87 %. Bonus set 2D: Duhrkop 29 %, Rutz/Nothias 28 %.
  • Reliance on NIST20/library hits: by construction the priority compounds had no public spectral-library spectra (FAQ: "neither deposited nor are there any corresponding fragmentation spectra ... in public ... libraries"). The methods excerpts live in a Google Drive folder (16 files dated 2022-07-06) that I could list but not read, so per-team library use is unverified. Vaniya's 2022 Metabolomics Society talk was titled "Comparison of CFM-ID, MS-FINDER, and SIRIUS+CSI:FingerID with LC-high resolution-MS/MS data from 500 new reference compounds".
  • Published summary: none found. A Europe PMC full-text search for "CASMI 2022" (2026-10-07) returns 17 papers that use the data (e.g. MetFID, IEEE EMBC 2023, PMC12143282: "CASMI 2016 and CASMI 2022 ... 208 and 250 challenges"; its own Top-1 50 % vs 32 %), but no organiser paper, despite the 2022 call promising "a manuscript". C&EN (2026-09-17) credits Fiehn with running 2022 and quotes his caveat that the 2026 set has no "overlapping coeluting entities, chimeric spectra, weird adducts".

5. What the editions did NOT measure, and database presence of the answers

  • Chimeric/co-eluting spectra: never. 2016 spectra were cleaned with RMassBank (recalibrated, subformula-filtered, SPLASH-deduplicated, min intensity 1e5); 2022 used an inclusion list on pure standards; 2026 is likewise isolated compounds (Fiehn, C&EN).
  • Stereo/positional isomers: evaluation always collapsed to InChIKey block 1; Schymanski 2017 explicitly deferred isomer discrimination.
  • Novel compounds absent from structure DBs: only 2014 (ch. 43-48, zero Top-1) and 2017's "few" unknown analogs among 1-45. 2022's "not in public libraries" means spectral libraries; the structures were known reference compounds.
  • Database presence of the answers: 2012 all confirmed with reference standards; 2013 all solvable by database search (Newsome: ChemSpider/SciFinder/Reaxys); 2014 42/48 known, 6 SciFinder-absent; 2016 and 2017 Cat 4 100 % by construction (candidate lists from ChemSpider/PubChem containing the answer). McEachran et al. 2020 (Metabolites 10:260, PMC7345619) report DSSTox coverage before curation of 9/14 (2012), 8/18 (2013), 29/42 (2014), 54/243 (2017) - i.e. the 2017 natural products were mostly outside the EPA environmental database, and their CFM-ID + data-source-count workflow got 144/243 in Top-5 with MS/MS alone but fell to 130 when metadata was added, the one documented case where "importance" priors hurt.
  • Uncontrolled training sets: Schymanski 2017 recommends an "extensive, open training dataset ... with all CASMI challenges excluded" and InChIKey lists of training sets with submissions - exactly what Enveda's 2026 setup finally does.

6. Scoring rules vs MRR@25 and the strategies they induced

Rule Credit at rank 1 / 2 / 5 / 10 / 25 (normalised) Strategy it induced
2012-2016 per-challenge win tally relative: whoever is best wins; rank 100 can score Submit something for every challenge; long lists cost nothing (2016 MS-FINDER exported top 500; CSI added +10000 to make scores positive); cross-dependency between teams
2016 reported Top-1/3/10, RRP absolute step functions Teams tuned on the 312-spectrum training set; "leave-out" entries appear for the first time
2017 Formula-1 points 1.00 / 0.72 / 0.40 / 0.04 / 0 Rewards putting the answer anywhere in top 3; 2017 kai112 traded Top-1 for fewer misses in Cat 4
2022 single answer, exact match 1 / 0 / 0 / 0 / 0 Teams attempted subsets (Nikolic 55 of 250), raising their percentages; no hedging possible
2026 MRR@25 1.00 / 0.50 / 0.20 / 0.10 / 0.04 Rank-2 is punished harder than in 2017 (0.50 vs 0.72) but ranks 11-25 still pay; expected score is maximised by sorting candidates by calibrated P(correct) (rearrangement inequality), so hedging across formula/adduct hypotheses in slots 6-25 is cheap insurance

2014's "score-based rank" (sum of normalised scores an analyst must inspect before reaching the answer) was announced but, as far as the pages show, never used - the closest any edition came to a calibration metric.

Forced analogy: law of evidence - precedent / case law

The analogy stated. Each CASMI edition is a reported decision that fixed a standard of proof for "identification". The submitter is the prosecution, the hidden InChIKey block 1 is the fact to be proven, the candidate pool is the court's jurisdiction (no defendant outside it can be convicted), and the scoring rule is the standard the fact-finder applies.

Edition Implicit standard of proof Courtroom analogue
2012-2016 "best rank among participants wins" comparative, no absolute threshold Civil preponderance between the parties present: you win if your case is less bad than the other side's; a rank-100 "win" is a verdict by default
2013/2014 manual flag; 2016 Cat 1 / Cat 3 "any additional information" admits hearsay and prior record Literature, SciFinder, reference counts and "lowest ChemSpider ID" are reputation and prior-conviction evidence; Cat 2's ban on library lookup is an exclusionary rule
2017 Formula-1 points graded standard, zero credit beyond rank 10 Comparative-negligence apportionment: partial recovery scaled by how close you came
2022 single answer beyond reasonable doubt, top-1 only Criminal verdict: guilty or not, no partial credit
2026 MRR@25 absolute, graded, 25-deep Apportioned damages with a long tail: 1/rank is the court's discount on a weaker case

Do the Schymanski et al. 2014 levels (ES&T 48:2097, DOI 10.1021/es5002105) function as a formal standard of proof? Partially. The scale (Level 1 confirmed by reference standard; 2a library match / 2b diagnostic evidence; 3 tentative candidate; 4 unequivocal formula; 5 exact mass) is an evidentiary hierarchy - it grades the kind of evidence, like the distinction between DNA, fingerprint, eyewitness and circumstantial evidence - but it never states a probability threshold, so it is not a standard of proof in the legal sense. CASMI scoring inverts it: the contest demands Level-1-equivalent correctness (InChIKey match against a reference-standard truth) while the submitter can only ever possess Level 2b/3 evidence. MRR@25 formalises that gap: a tentative (Level 3) list is admitted and paid at 1/rank. (I could not open the ES&T full text - publisher 403 - so the level definitions are quoted from the abstract-level description widely reproduced in the CASMI papers; confidence medium-high.)

Is MRR@25 a lower or higher standard than before? Higher than 2012-2016 (absolute, not comparative; nothing beyond rank 25 counts), lower than 2022 (hedging restored), and split relative to 2017: stricter at rank 2 (0.50 vs 0.72) and ranks 3-4, more lenient from rank 6 on (rank 10: 0.10 vs 0.04; ranks 11-25: 0.09-0.04 vs 0).

One concrete, testable rule derived from the precedents - "jurisdiction before trial". In every edition where the court's jurisdiction was fixed to include the defendant (2016, 2017 Cat 4: candidate list contains the answer), ranking engines delivered 32-70 % Top-1 and 60-85 % Top-10; in every edition where it was not (2014 unknowns: 0 Top-1; 2022 no list, not in libraries: 26-29 % Top-1 for the best automated entry), results collapsed. The 2026 public-LB back-solve (recall@25 ~0.55 at MRR 0.417) is the same precedent restated. Rule: spend the first effort budget on the candidate pool, then fill slots by calibrated posterior. Concretely and testably: (a) build a held-out set of enveda-180 structures with the same InChIKey14 removed from training (the 2016 "leave-out" precedent) and measure recall@25 of the pool alone (no ranker) within +-10 ppm / 1 mDa for each of the 7 test adducts; (b) measure MRR@25 of the ranker conditional on the answer being in the pool; (c) MRR@25 ~= recall@25 x conditional-MRR, so improving recall from 0.55 to 0.70 at a fixed conditional-MRR of 0.76 raises the score from 0.42 to 0.53, which no ranker improvement at 0.55 recall can reach. A second, cheaper test: since MRR is maximised by sorting by calibrated probability, reserve slots 6-25 for the top candidates under the second and third molecular-formula/adduct hypotheses (the 2017 kai112 lesson: formula errors produced 102 misses; the test set gives precursor_mz but no MS1 isotope pattern) and measure whether the 0.04-0.17 tail credit exceeds what those slots earn under the top hypothesis.

Where the analogy holds, partially holds, or breaks. Holds: standards of proof, admissibility of priors, jurisdiction = candidate pool, apportioned credit. Partially: Schymanski levels are evidence types, not thresholds. Breaks: (1) in law, evidence from one case does not change the verdict in another; in CASMI the training set is reused evidence, and 2016 showed the "witness had seen the defendant before" in 67 of 208 cases (memorisation), which has no clean courtroom analogue; (2) courts do not reward submitting 25 defendants - reputational priors (reference counts, 143/208 Top-1 in 2016) are in legal terms inadmissible character evidence, yet in CASMI they were the single strongest predictor, and in 2026 they are useless for novel natural products (COCONUT overlap 0.03 % for enveda-180).

Gotchas / what could fool us

  1. "Winner" is not comparable across editions. 2012-2016 wins are relative to who showed up; 2016 Cat 2 "was very sensitive to the method of declaring the winner" (CSI:FID had more Top-1 than IOKR, 70 vs 62, but fewer golds). 2017 declared no winner; 2022 declared none. Any dashboard line "CSI:FingerID won CASMI 2016/2017" is an interpretation.
  2. Reported Top-1 rates are conditional on a candidate list containing the answer (2016, 2017 Cat 4). The 70 % Cat 3 and 87 % Blazenovic numbers vanish without that list; 2022 with no list gave 26-29 %.
  3. Metadata priors are dataset-specific. Reference count alone gave 143/208 Top-1 in 2016 because the set was "predominantly well-known anthropogenic substances"; the same prior lowered McEachran's 2017 natural-product Top-5 from 144 to 130. For 2026 ("none previously published") a popularity prior is at best a tie-breaker.
  4. Memorisation inflates everything. 2016: 67/208 challenge compounds in CSI:FingerID's training set; leave-out cut Top-1 from 70 to 36. The 2026 visible test is verbatim train rows - a local score on it is the 2016 pre-leave-out illusion.
  5. Spectral-library "hits" of the same compound can have cosine 0.14-0.76 (2016 analysis) - library search succeeds by precursor filtering, not spectral similarity; do not trust a cosine threshold as a correctness proxy.
  6. Negative mode is systematically under-trained (816 vs 7300 spectra in 2016; 5776 vs 14181 in 2017 IOKR). Check the polarity split of enveda-180 and the hidden test's 7 adducts before trusting a single model.
  7. Formula selection is the silent killer when MS1 is absent. 2017 challenges 46-243 had no isotope pattern; kai112's 102 misses vs kai_iso's 4 came from formula handling, not fingerprint quality. The 2026 test gives precursor_mz only.
  8. Tie handling changes Top-1 counts. 2014 used worst-case rank, 2016/2017 average rank rounded up; MAGMa+ had 1098 candidates but only 616 distinct scores. In 2026 duplicate InChIKey14 after tautomer canonicalisation would waste slots.
  9. Percent-of-attempted (2022) rewards cherry-picking: Nikolic_KUCA's 38 % is on 55 self-selected challenges; Duhrkop's 26 % is on 222.
  10. Instrument drift across editions: micrOTOF-Q (2012), mixed IT-TOF/Orbitrap/FT-ICR (2013), Orbitrap Velos/Q Exactive/SYNAPT (2014), Q Exactive Plus HCD 20/35/50 (2016), SYNAPT MSE (2017), likely Orbitrap (2022, unverified), Bruker timsTOF (2026). No prior edition tested timsTOF; CE scales (NCE vs eV ramp) differ.
  11. Candidate-count effect is non-monotone (worst at 200-1000 candidates in 2016); pool size tuning should be measured, not assumed.
  12. Page-level inconsistencies: 2014 text says unknown "challenge 47" was hit, the table lists "challenge46(isomer2)"; 2016 Cat 1 had 19 challenges on the page vs "18" in Nikolic's paper abstract (challenge 3 removed); 2016 Table 5 "164" combined Top-1 is from a fetch summary, not read directly.

Implications for the challenge

  1. Treat recall@25 of the candidate pool as the primary KPI; the precedents (2014 unknowns, 2022, 2026 LB back-solve) all show the pool, not the ranker, sets the ceiling. Measure it on a leave-out split by InChIKey14 and by Bemis-Murcko scaffold.
  2. Reserve slots for alternative formula/adduct hypotheses (2017 kai112 lesson; no MS1 in test); the MRR tail (0.04-0.17 for ranks 6-25) is free money if those slots are otherwise filled with same-formula near-duplicates.
  3. Fuse orthogonal engines: every edition with a consensus entry (2016 Combined MS/MS 82 vs best single 70; Blazenovic +5 %; 2017 Tsugawa rank-sum voting) beat its components; CSI-style fingerprint prediction and MS-FINDER-style chemistry were explicitly complementary.
  4. Do not import popularity priors (reference counts, lowest ID, "in 13 databases") as hard features; test them only as tie-breakers, because 2026 chemistry is novel (COCONUT overlap 0.03 %) and McEachran 2020 showed they can hurt on natural products.
  5. Build a true leave-out validation (remove test-like InChIKey14 from training) and expect Top-1 to roughly halve (70 -> 36 in 2016; 34 % -> 18 % on the globally excluded subset); local scores on the visible test are meaningless.
  6. Train/validate polarity-specific models and check adduct-specific recall; negative mode was the weakest link in every edition.
  7. Sort final lists by calibrated probability, not raw score; MRR's 1/rank weighting makes the rank-1/rank-2 decision worth 0.5 points per molecule - the costliest single decision in the metric.
  8. Benchmark against the 2022 ceiling: 26-29 % exact 2D for the best automated system on library-absent known compounds with RT given; a 2026 MRR@25 of 0.47 implies Top-1 somewhere around 0.3-0.35 plus tail, consistent with that precedent - and tells you the realistic headroom is in recall, not Top-1 polish.
  9. Use the CASMI 2016 (208 + 312 spectra, Q Exactive) and 2017 (243, SYNAPT) data, both public with solutions, as instrument-shift sanity sets for a timsTOF-trained model.

Open questions

  • Instrument model and CE scheme for CASMI 2022 (likely Thermo Orbitrap; unverified) - the 16 method files in the organisers' Google Drive would settle it and per-team library use.
  • Whether any CASMI 2022 summary manuscript ever appeared (none found in Europe PMC as of 2026-10-07).
  • Exact Table 4/5 values of Schymanski 2017 for the combined metadata entries (reported here from a fetch summary as ~164 Top-1).
  • Nikolic 2017 Phytochemistry Letters: full method and how many of the 14-15 Cat 1 solutions came from SciFinder/literature rather than spectra.
  • Whether the 2014 "score-based rank" (calibration-like) evaluation was ever computed - no page or paper I opened reports it.

Sources

  • casmi-contest.org results, rules, category and challenge pages (fetched 2026-10-07 with curl -k; TLS cert is *.sf.net): /2013/results.shtml, /2013/results-cat1.shtml, /2013/results-cat2.shtml, /2013/rules.shtml, /2013/challenges.shtml, /2014/results.shtml, /2014/results-cat1.shtml, /2014/results-cat2.shtml, /2014/rules.shtml, /2014/challenges_main.shtml, /2016/results.shtml, /2016/results-cat1.shtml, /2016/results-cat2.shtml, /2016/results-cat3.shtml, /2016/rules.shtml, /2016/rules-categories.shtml, /2016/challenges-cat1.shtml, /2016/challenges-cat2+3.shtml, /2017/results.shtml, /2017/results-cat1..4.shtml, /2017/rules-categories.shtml, /2017/rules-submit_evaluate.shtml, /2017/challenges_1-45.shtml, /2017/challenges_46-243.shtml, /2022/index.shtml, /2026/index.shtml, /index2012.shtml (2012 results page returns 404).
  • Schymanski EL, Neumann S. CASMI: And the Winner is ... Metabolites 2013;3:412-415. DOI 10.3390/metabo3020412 (PMC3901266).
  • Schymanski EL, Neumann S. The CASMI: Challenges and Solutions. Metabolites 2013;3:517-538. DOI 10.3390/metabo3030517 (PMC3901296).
  • Ruttkies C, Gerlich M, Neumann S. Tackling CASMI 2012: Solutions from MetFrag and MetFusion. Metabolites 2013;3:623-636. DOI 10.3390/metabo3030623 (PMC3901291).
  • Nishioka T et al. Winners of CASMI2013: Automated Tools and Challenge Data. Mass Spectrom (Tokyo) 2014;3:S0039. DOI 10.5702/massspectrometry.S0039 (J-STAGE).
  • Schymanski EL et al. Critical Assessment of Small Molecule Identification 2016: automated methods. J Cheminform 2017;9:22. DOI 10.1186/s13321-017-0207-1 (PMC5368104, full text read).
  • Blazenovic I et al. Comprehensive comparison of in silico MS/MS fragmentation tools of the CASMI contest: database boosting is needed to achieve 93% accuracy. J Cheminform 2017;9:32. DOI 10.1186/s13321-017-0219-x (PMC5445034, full text read).
  • McEachran AD et al. Revisiting Five Years of CASMI Contests with EPA Identification Tools. Metabolites 2020;10:260. DOI 10.3390/metabo10060260 (PMC7345619).
  • Duhrkop K et al. Searching molecular structure databases with tandem mass spectra using CSI:FingerID. PNAS 2015;112:12580. DOI 10.1073/pnas.1509788112 (PMC4611636; no CASMI results inside).
  • Ruttkies C et al. MetFrag relaunched. J Cheminform 2016;8:3. DOI 10.1186/s13321-016-0115-9 (PMC4732001): 473 Orbitrap XL spectra, MetFrag 30 / CFM 43 Top-1 on PubChem, 336 (71 %) with RT + references.
  • Schymanski EL et al. Identifying small molecules via HRMS: communicating confidence. Environ Sci Technol 2014;48:2097. DOI 10.1021/es5002105 (full text not accessible; levels quoted as commonly reproduced).
  • Fiehn Lab, Revisiting CASMI (rules), CASMI 2022 Results, CASMI FAQ: https://fiehnlab.ucdavis.edu/casmi , https://fiehnlab.ucdavis.edu/casmi/casmi-2022-results , https://fiehnlab.ucdavis.edu/casmi/casmi-faq ; methods folder https://drive.google.com/drive/folders/12cCAdUjV_mXZ7tavdz9A_r18B6rDk437 (file names only).
  • MassIVE MSV000089239 "GNPS - 2022 CASMI Contest - Raw Data".
  • MetFID: Deep Learning Based Metabolite Annotation, IEEE EMBC 2023 (PMC12143282) - uses CASMI 2022 as 250 challenges.
  • Oldach L. Metabolomics has a data problem. Could this competition fix it? C&EN 2026-09-17, DOI 10.47287/cen.612552.newsarticle.
  • Europe PMC REST searches (DOI/title lookups, "CASMI 2022" full-text query, 2026-10-07).
  • Not opened (search summaries only, low confidence): Nikolic DS. CASMI 2016: A manual approach ... Phytochemistry Letters 2017; Nikolic et al. CASMI 2014: Challenges, Solutions and Results, Current Metabolomics (Bentham) - both publisher-blocked.