MRR@25 identity protocol: InChIKey14 after RDKit TautomerEnumerator.Canonicalize — exact procedure, empirical failure modes, pre-canonicalisation verdict

TL;DR. The grader decides "same molecule" by InChIKey[:14] of TautomerEnumerator().Canonicalize(MolFromSmiles(s)) with RDKit 2026.03.3, applied to both our guess and the answer; no salt stripping, no neutralisation, no stereo (block 1 of the key ignores stereo and protonation anyway). I re-implemented this locally and ran 73 curated cases, 14 stereo cases and 969 natural-product SMILES from Wikidata/LOTUS: canonicalisation changes the InChIKey14 of 12–17 % of NP structures (mostly enols, acyloins/sugars, 1,3-dicarbonyls, enones), is identical between RDKit 2026.03.3 and 2026.03.6 on every molecule tested, but is stereo-dependent for ~0.45 % of stereo-bearing NPs and non-idempotent for ~0.3 %, and one third of NPs > 40 heavy atoms hit the 1000-tautomer/transform cap. Verdict: yes, pre-canonicalise and de-duplicate our 25 guesses with exactly this protocol (pinned RDKit 2026.03.3), and for the rare candidates whose stereo and flat forms give different keys, submit both forms back-to-back.

1. The exact protocol (what is verified, what is not)

The Kaggle page is a JS shell (fetch returned only the title), so the protocol below is reconstructed from the competition text as reproduced in GitHub write-ups and from their re-implementations. Three independent write-ups agree on the wording and the code shape:

Reconstructed procedure (confidence medium; items marked ? are unverified against the official notebook):

key(s) = MolToInchiKey( TautomerEnumerator().Canonicalize( MolFromSmiles(s) ) )[:14]
rr(mol) = 1/i for the smallest i ≤ 25 with key(guess_i) == key(answer), else 0      (? behaviour on unparsable guess: most likely skipped = wasted slot)
MRR@25 = mean over hidden molecule_ids

Unknowns to flag: (a) whether the grader strips counter-ions/neutralises (every community re-implementation says no; InChI itself does not); (b) whether the answer SMILES carries stereo (matters — see §3.3); (c) whether an unparsable guess aborts the row or is skipped; (d) whether the official notebook catches Canonicalize exceptions as kabir0774 does.

What InChIKey block 1 already does, independently of RDKit (Wikipedia InChI article, confirmed empirically): the first 14 characters hash the main layer (formula, connections, H positions in mobile-H form) plus the /q charge sublayer; the /p protonation sublayer is not in block 1. Hence, with no RDKit step at all: glycine neutral = zwitterion = cation (DHMQDGOQFOQNFH), pyridine = pyridinium, 2-pyridone = 2-hydroxypyridine, adenine 7H = 9H, guanine lactam = lactim, imidazole/triazole/tetrazole NH isomers, amide = imidic acid, uracil diketo = dihydroxy. Standard InChI does not treat keto–enol (C–H shifts) as mobile H (the KET/15T options are non-standard), which is exactly where RDKit's step changes keys.

What RDKit's step adds. TautomerEnumerator applies 37 transform SMARTS (Code/GraphMol/MolStandardize/TautomerCatalog/tautomerTransforms.in, header: "Derived from the transforms in Sitzmann et al.", doi 10.1007/s10822-010-9346-4), including 1,3- and 1,5-(thio)keto/enol, aliphatic imine, 1,3/1,5/1,7/1,9/1,11 heteroatom H shifts, furanone, keten/ynol, nitro/aci, oxime/nitroso, cyano/isocyanic acid, phosphonic acid. Defaults (rdMolStandardize docs): maxTautomers = 1000, maxTransforms = 1000, tautomerRemoveSp3Stereo = True, tautomerRemoveBondStereo = True, tautomerReassignStereo = True. Scoring (Tautomer.cpp, Release_2026_03 branch): aromatic ring +100 (+150 if all-carbon), benzoquinone +25, oxime +4, C=O/N=O/P=O +2, C(=hetero)–hetero +2, C=hetero +1, methyl +1, guanidine terminal =N +1 / endocyclic +2, aromatic C=exocyclic N −1, aci-nitro −4, −1 per H on P/S/Se/Te. Tie-break in 2026.03.x: lexicographically smallest isomeric canonical SMILES (if (t.first < bestSmiles)), i.e. the winner can depend on stereo tags because @ sorts differently from letters. If a limit is hit, Canonicalize still returns the best of what was enumerated (status MaxTautomersReached/MaxTransformsReached), silently.

2. Empirical results (RDKit 2026.03.3, Python 3.12, this machine)

Full table (87 rows) in the companion .json. Highlights:

2.1 Curated pairs: who is "the same person"

2.2 "Convicts the innocent": correct structure, key moved (and merged with other compounds)

• Open-chain aldoses/ketoses collapse: D-glucose (GZCGUPFRVQAUEE) and D-fructose (BJHIKXHVCXFQLS) open-chain forms both canonicalise to O=C(C(O)CO)C(O)C(O)CO (a 3-hexulose), key DWJZKGYQNOQQEZ. Lactaldehyde → hydroxyacetone (XLSMFKSTNGKWQX). Ascorbic acid enediol and its keto form both go to an aromatic furan-triol OC[C@H](O)c1oc(O)c(O)c1O (LPYXWGMUVRGUOY; +100 for the aromatic ring wins). Mechanism: 1,3-keto/enol run through an enediol. Acetoin and benzoin are not moved (symmetric). Implication: α-hydroxy-carbonyl positional isomers (acyloins, open-chain sugars, some polyketide fragments) are one "person" under the metric — a guess with the carbonyl on the wrong carbon is scored correct, and the ring form is scored wrong.
• Tetracycline OFVLGDICTFRJMM → MUHBBHLZPHKTTR (canonical form has four C=O and no enol; chemically unrealistic but symmetric). Taxol RCINICONZNJXQF → NDUAXKHGCJIYEQ (an enol-acetate tautomer is chosen). Progesterone RJKFOVLPORLFTN → MNRHZPCIEGLWGK (enone deconjugated to β,γ). Nootkatone XKBRFTTZONSFNM → FNSBGCSZGIIKCQ. These are harmless only if the grader applies the identical protocol to the answer — which it does — and if everyone's RDKit agrees (§2.5).
• Keys that must never be compared across protocols: the train inchikey14 column, COCONUT/PubChem InChIKeys and the metric's canonical keys are three different partitions of chemical space (12–17 % of NPs differ, §2.4).

2.3 Stereo-dependence (issue #7499 class) — the serious one

On 660 LOTUS NP SMILES with ≥ 1 stereo element (median 30 heavy atoms): 3/660 = 0.45 % get a different canonical InChIKey14 depending on whether stereo is present in the input (a sesquiterpene lactone, a triterpenoid diglucuronide with 19 stereocentres, and a bilin with E/Z bonds). Root cause is the lexicographic-SMILES tie-break on isomeric SMILES — RDKit issue #9518 ("Canonical tautomer tie-break is systematically biased against stereochemistry (acetoin enantiomers merge)"), fixed only in Release_2026_09_1, i.e. not in the pinned 2026.03.3. Scaled to ~400 hidden molecules, expect ~2 molecules whose correctness flips with the stereo content of the submitted string, and we do not know whether the answer SMILES carry stereo (train normalized_smiles would tell; check).

2.4 Population statistics (Wikidata/LOTUS NPs; wdt:P703 found-in-taxon)

By size (P233 set): 8–20 heavy atoms: 15 % keys change, 3 % hit a cap, median 1.3 ms; 30–40: 20 %, 13 %, 14 ms; 40–60: 8 %, 42 %, 265 ms; > 60: 24 %, 94 %, 799 ms (peptides/depsipeptides and ellagitannins saturate at 1000 tautomers). Distinct raw keys collapsing onto one canonical key within this single-source sample: 0 — a curated database stores one tautomer per compound, so collapse appears only when pools are merged from several sources or when tautomers are enumerated as candidates.

The one non-idempotent case (canon(canon(x)) ≠ canon(x), JZYLAQXKOBFJTP → IGUAMTWXXKKBDL) is a cyclodepsipeptide that hits the cap — issue #3755 (closed 2021) describes the same symptom for serine, which now passes; cap-limited enumeration re-opens it.

2.5 Version sensitivity

• 2026.03.3 vs 2026.03.6: identical canonical SMILES and keys on all 73 + 14 + 309 + 660 molecules. Release notes 2026_03_4–_6 list only one tautomer item, #9338 (precondition violation with SetRemoveBondStereo(False) on atropisomers — not the default path). 2026_03_2 added "Tautomer insensitive hash v2" (a different API, not Canonicalize).
• 2026.09.1 (published 2026-10-03) changes results by design: #9518 tie-break fix, #9134 "Speed-up tautomer canonicalization", #9583 macrocycle STEREOANY fix. As of 2026-10-07 PyPI offers only rdkit<=2026.3.6 wheels (uv resolution failed for >=2026.9), so I could not measure the delta; conda-forge availability unverified. Any environment that silently upgrades to 2026.09 will dedupe/score differently from the grader on an unknown fraction of molecules.
• Explicit-H graphs (issue #6856, closed not-planned): Canonicalize(AddHs(mol)) leaves CC1=C(C)C=NC(=O)C1 unchanged and gives a different key than the implicit-H path; SMILES containing [H] are fine because MolFromSmiles folds them in. Never canonicalise hydrogen-added mols.
• Charges/salts: [NH3+], [nH+], [O-] forms give the same block-1 key as neutral (protonation is /p); but counter-ions and permanent cations are in block 1: sodium acetate VMHLLURERBWHNL ≠ acetate QTBSBXVTEAMEQO; berberine YBHILYKTIRIUTE ≠ berberine chloride VKJGBAJNNALVAF; choline ≠ choline chloride. No write-up's metric strips fragments.

3. Known failures/complaints in RDKit (issues opened)

No GitHub issue about non-determinism across runs was found; my repeated runs were deterministic.

4. Forced analogy: forensic identity standards (fingerprint ACE-V)

(a) Statement. In latent-print examination the conclusion "same source" is not a property of the finger; it is the output of a protocol (Analysis–Comparison–Evaluation–Verification, agreed minutiae rules, a tolerance for distortion). Here "same molecule" is the output of key14 ∘ Canonicalize ∘ MolFromSmiles, version-pinned. Two drawings that look different (keto/enol, open-chain glucose/fructose) are "the same person"; one drawing that is physically the same compound but carries a different protocol-relevant feature (stereo tags that change a SMILES sort order) can be "a different person".

(b) Concrete, testable ideas.

1. Run the examiner yourself, with the examiner's tools. Install rdkit==2026.3.3 (wheel bundled as a Kaggle dataset for the offline notebook), compute key14 for every candidate once, and dedupe the 25 slots on that key. Expected gain: every duplicate slot removed is a free rank position for the next candidate; with recall@25 ≈ 0.55 at the 0.417 level, any slot wasted on a duplicate is a direct loss.
2. Hedge where the protocol is unstable. For each candidate compute key14(canon(stereo SMILES)) and key14(canon(flat SMILES)); when they differ (~0.45 % of stereo-bearing NPs) emit both strings as consecutive guesses. Cost ≈ 0.1 slot per molecule; benefit: immunity to not knowing whether the answer SMILES has stereo. Testable offline on train: count molecules where the two keys differ and whether normalized_smiles carries stereo.
3. Minutiae-count analogue → protocol-stable keys. Flag candidates whose enumeration returns MaxTautomersReached/MaxTransformsReached (33 % above 40 heavy atoms): their canonical key depends on enumeration order and could, in principle, differ between the grader's run and ours if transform ordering changes between versions. Keep the pin; do not "improve" with SetMaxTautomers(10000) — that changes the identity, not the accuracy.
4. Known-collision classes as a data-cleaning tool: build candidate pools, then group by canonical key; groups with > 1 distinct raw key are tautomer duplicates that fingerprint models currently see as distinct training labels (lactaldehyde vs hydroxyacetone, glucose vs fructose open-chain). Merging them before training removes label noise that no model can resolve by spectrum anyway.

(c) Does it hold? Partially. It holds for the core point (identity is protocol-defined, version-pinned, with documented "examiner" rules) and for protocol-instability as the failure mode to audit. It breaks on noise: a fingerprint comparison has measurement noise and examiner judgement; the key is deterministic bit-for-bit given the version, so "verification by a second examiner" is pointless here — the only second examiner that matters is the grader's RDKit build. It also breaks on hash collisions: two different molecules sharing a 14-character block is possible in principle but I have no verified collision rate (unverified; ignore in practice).

5. Gotchas / what could fool us

1. Three incompatible key partitions: train inchikey14, external database InChIKeys, and metric keys differ for 12–17 % of NP structures. Any "is this candidate in train?" or "does candidate X equal candidate Y?" logic must use metric keys. Whether train's inchikey14 was computed post-canonicalisation is unknown — recompute and diff on a sample.
2. Stereo in the submitted string can change the verdict (0.45 % of stereo-bearing NPs; testosterone-type enones). Hedge (§4b2) or at least be consistent with the answer convention.
3. Open-chain sugar / acyloin merging: glucose = fructose = any hexulose when drawn open-chain; the ring form is a different key. Candidate pools that list both ring and chain forms carry one false "distinct" candidate per sugar.
4. Salts and counter-ions are not stripped by any known implementation: submit single-fragment SMILES; keep permanent cations (berberine-type quaternary alkaloids) as the cation, not as the chloride; but do not neutralise quaternary N+ (it cannot be neutralised). Protonated amines/deprotonated acids are harmless.
5. Invalid SMILES silently waste a slot (my own hand-written hypericin failed MolFromSmiles). Validate every guess with MolFromSmiles and the key function before writing the CSV; also never emit ; inside a SMILES or > 25 guesses — the row/submission is rejected.
6. Timing: canonicalisation costs 33 ms median, 250 ms mean, up to 1.7 s per NP, ~0.8 s median above 60 heavy atoms. 10 000 guesses ≈ 40 min single-core; a 700 k COCONUT pool ≈ 50 CPU-hours — precompute keys offline and ship them as a dataset; do not canonicalise inside the 9-hour offline notebook per spectrum.
7. Version drift: 2026.03.3 ≡ .6 (tested), but 2026.09.1 changes the tie-break and the algorithm; pin rdkit==2026.3.3 everywhere (local CV, pool keys, notebook). Kaggle's default image version is unverified.
8. Cap-limited enumeration (33 % of NPs > 40 heavy atoms) makes the key depend on transform order; it is stable within a version but is where non-idempotence lives (1/309). Never feed an already-canonicalised SMILES back through the protocol expecting the same key.
9. Canonicalize may throw (e.g. #9338 class); kabir0774's metric keeps the un-canonicalised mol on exception, the official notebook's behaviour is unknown — a guess that throws could be scored by raw key.
10. MRR arithmetic illusion: a rank-25 hit is worth 1/25 of a rank-1 hit; moving one molecule from rank 2 to 1 (+0.00125 on 400 molecules) beats moving twelve molecules from rank 25 to 24 combined (+0.000004 each). Re-ranking the top-3 is where score lives; adding candidates at ranks 20–25 is recall insurance worth ≈ 0.0001 each.

6. MRR@25 arithmetic

Reciprocal ranks: 1 → 1.000, 2 → 0.500, 3 → 0.333, 5 → 0.200, 10 → 0.100, 25 → 0.040. One rank-1 hit = 2 rank-2 = 5 rank-5 = 25 rank-25 hits. Over 400 molecules one rank-1 hit = +0.0025, one rank-25 hit = +0.0001.

Expected MRR = recall@25 × E[1/rank | hit]:

Reading the public leaderboard: 0.417 with R@25 ≈ 0.55 (BRIEF) implies E[1/rank|hit] ≈ 0.76, i.e. roughly 65–70 % of recovered molecules already sit at rank 1; the remaining lever is recall. 0.471 (rank 1) at the same recall would need E[1/rank|hit] ≈ 0.86.

Implications for the challenge

1. Re-implement the key function exactly (MolFromSmiles → TautomerEnumerator().Canonicalize → MolToInchiKey[:14], rdkit==2026.3.3), wrap Canonicalize in try/except, and use it for all dedup, train/test overlap checks and local CV. Yes: pre-canonicalise and dedupe the 25 guesses by this key before submission — duplicates score nothing and cost a slot.
2. Precompute metric keys for the whole candidate pool offline (≈ 50 CPU-h for 700 k NPs; parallelise on Explorer), ship as a dataset; expect 12–17 % of pool keys to differ from the database InChIKeys and some pool entries to merge.
3. For candidates whose stereo and flat canonical keys differ (~0.45 %), submit both strings adjacently; run the train-set check of whether normalized_smiles carries stereo to decide which goes first.
4. Submit single-fragment SMILES; strip counter-ions; keep quaternary cations as cations; leave protonation as is.
5. Validate every guess with MolFromSmiles and the key function; reject ;, nulls, > 25.
6. Pin RDKit 2026.03.3 in the Kaggle notebook (bundle the wheel); verify the Kaggle image version at runtime and log it.
7. Merge training labels by metric key before training fingerprint models (removes tautomer/acyloin label splits).
8. Spend modelling effort on top-3 re-ranking and recall@25, not on ordering ranks 10–25.

Open questions

• Content of the official Kaggle metric notebook: fragment handling, exception handling, stereo in answers (needs a Kaggle login/browser session).
• Was train inchikey14 computed after canonicalisation? (diff on 1 000 rows settles it.)
• RDKit version in Kaggle's current offline image (not on PyPI beyond 2026.3.6 today; conda-forge unknown).
• Fraction of a real merged candidate pool (COCONUT + PubChem-lite + enumerated analogs) that collapses under the metric key (my single-source sample: 0 %).
• Magnitude of 2026.09.1 result changes vs 2026.03.3 (could not install).

Sources

• kabir0774/Enveda-Casmi-2026, casmi26/metric.py — https://raw.githubusercontent.com/kabir0774/Enveda-Casmi-2026/main/casmi26/metric.py
• OfficialBhattacharya/kaggle_CASMI_2026, src/casmi/chem.py — https://raw.githubusercontent.com/OfficialBhattacharya/kaggle_CASMI_2026/main/src/casmi/chem.py
• corentinlaudicina/enveda-casmi-2026 README — https://raw.githubusercontent.com/corentinlaudicina/enveda-casmi-2026/main/README.md
• AmanDubal/Enveda-CASMI-2026 README — https://raw.githubusercontent.com/AmanDubal/Enveda-CASMI-2026---Molecule-ID-From-Mass-Spectra/main/README.md
• Isaackjoshua/casmi-2026 README; ibnmarzuk/casmi26-molecule-id README; shiferaxa/casmi26 README; yasumorishima/kaggle-competitions PR #29 (Kaggle facts via GitHub mirrors; the Kaggle page itself returned only its title)
• RDKit rdMolStandardize docs — https://www.rdkit.org/docs/source/rdkit.Chem.MolStandardize.rdMolStandardize.html
• RDKit Tautomer.cpp (Release_2026_03 branch and master) — https://raw.githubusercontent.com/rdkit/rdkit/Release_2026_03/Code/GraphMol/MolStandardize/Tautomer.cpp ; https://raw.githubusercontent.com/rdkit/rdkit/master/Code/GraphMol/MolStandardize/Tautomer.cpp
• RDKit tautomerTransforms.in — https://raw.githubusercontent.com/rdkit/rdkit/master/Code/GraphMol/MolStandardize/TautomerCatalog/tautomerTransforms.in
• RDKit releases (API listing with dates) — https://api.github.com/repos/rdkit/rdkit/releases ; Release_2026_03_5 — https://github.com/rdkit/rdkit/releases/tag/Release_2026_03_5 ; Release_2026_09_1 — https://github.com/rdkit/rdkit/releases/tag/Release_2026_09_1
• RDKit issues #3755, #3710, #6856, #6920, #7499 and discussion #6852 — https://github.com/rdkit/rdkit/issues/{3755,3710,6856,6920,7499}, https://github.com/rdkit/rdkit/discussions/6852
• Sitzmann M, Ihlenfeldt WD, Nicklaus MC. Tautomerism in large databases. J Comput Aided Mol Des 2010;24:521–551, doi:10.1007/s10822-010-9346-4 (verified via Europe PMC REST)
• InChI layers and InChIKey block composition — https://en.wikipedia.org/wiki/International_Chemical_Identifier (inchi-trust.org returned 403)
• Test SMILES: Wikidata SPARQL (wdt:P703 + wdt:P233 / wdt:P2017, LIMIT 600/700), fetched 2026-10-07 — https://query.wikidata.org/sparql
• Local experiments: scripts exp_cases.py, exp_stereo.py, exp_sample.py, exp_stereo_sample.py in the session scratchpad; RDKit 2026.03.3 and 2026.03.6 via uv venvs, Python 3.12.13, Windows Server 2022.