Infer/The plan

68-day execution plan: week-by-week milestones, dated kill criteria, submission cadence, final-two rule, risk register

intermediateconfidence mediumrelevance 4/521 sources4.7k words
Forced analogyEyewitness line-up jurisprudence (DOJ 2017 blind administration + contemporaneous confidence statement + fixed protocol) -> a pre-registered final-two rule and a 'culprit-absent line-up' detector; the pre-registration half pays off, the blindness half only partially (we are one human plus agents, and we can resubmit).

TL;DR. We have 68 days (2026-10-07 to 2026-12-14), at most 5 Kaggle submissions per day, and a hidden test of ~400 novel timsTOF natural products where candidate coverage (recall@25 ~0.55 at the 0.417 public-LB tier) is the ceiling. The plan front-loads the slow things (COCONUT/LOTUS/NPAtlas pool, Explorer environment, offline wheel vendoring, a structure-disjoint validation set) so a scoring library-search submission exists by 2026-10-14, pins eleven dated, measurable kill criteria, and pre-registers the final-two selection rule before anyone looks at more LB numbers. Key correction to the brief: the public LB is not scored on the decoy test.parquet (library search scores 0.06-0.10, not ~1.0), so it is a noisy but unbiased sample of the private set; the trap is noise (+/-0.02-0.04 per score) and local over-fitting to decoy rows, not domain shift between public and private.

1. Calendar anchors and the arithmetic that constrains everything

Fact Value Source
Today Wed 2026-10-07 -
Entry / team-merge deadline 2026-12-07 BRIEF.md; ibnmarzuk README
Final submission deadline 2026-12-14 23:59 UTC ibnmarzuk README
Days remaining 68 (to Dec 14); 61 to the merge deadline arithmetic
Daily submissions / final picks 5 / 2 BRIEF.md
Theoretical max submissions 340; realistic 100-150 arithmetic; see cadence section
Kaggle notebook rerun ceiling ~9 h ("9-hour ceiling allows ~1,800 queries" at ~18 s/molecule) leesirleesirlee-star README (competitor write-up; the official rules page is JS-rendered and was not readable)
Kaggle GPU quota 30 h/week (free tier) NVIDIA/kaggle docs via search; leesirleesirlee notes "~6 variants testable per week (GPU quota bottleneck)"
Internet during rerun off; output must be /kaggle/working/submission.csv ibnmarzuk README
Grader RDKit 2026.03.3 (PyPI: released 2026-06-05; current is 2026.3.6, 2026-09-03) pypi.org/project/rdkit
Explorer gpu partition default/max 4 h / 8 h, 1 GPU per job, 4 GPUs per user, multigpu 24 h needs a ServiceNow ticket; /scratch purged monthly rc-docs.northeastern.edu (gpuoverview, scratchpurge)
Modal H100 $0.001097/s (~$3.95/h), A100-80GB ~$2.50/h, A10 ~$1.10/h; Starter $30/month credits modal.com/pricing

The 9 h ceiling is the single hardest constraint: a 1,500-spectra / 400-molecule hidden set at 18 s/molecule is already 2 h, and anything that forward-simulates spectra per candidate (ICEBERG/CFM-style) scales with pool size x molecules. Every week's plan below keeps inference time as a first-class metric.

2. Week-by-week plan (10 weeks)

Wk Dates Workstream Owner-role Deliverable (definition of done) The one metric that must move
1 Oct 07-11 data + infra Data Eng train.parquet/test.parquet in C:\Temp\casmi26\data; COCONUT CSV-lite (191 MB, CC0, Oct-2026 release) + SDF-lite (287.6 MB) downloaded; LOTUS v11 frozen CSV (20.6 MB + 90.3 MB metadata, CC-BY-4.0, 2026-04-13); NPAtlas v2024_09 TSV (33.3 MB, CC-BY-NC-4.0); rdkit==2026.3.3 env on Windows + Explorer; wheel directory vendored as a Kaggle Dataset Bytes on disk + python -c "import rdkit; assert rdkit.__version__=='2026.03.3'" passes in all 3 environments
2 Oct 12-18 submission-infra + MVS Submission Lead Minimum viable submission (library search: +/-10 ppm precursor filter, modified cosine + entropy similarity, RRF across spectra, mass-nearest backfill to 25) submitted by Mon 2026-10-12, scored by Wed 2026-10-14; metric.py reproduces MRR@25 with RDKit tautomer canonicalisation; a structure-disjoint validation split by inchikey14 (never by spectrum, per kabir0774) that holds out timsTOF NP-like molecules Public LB > 0 (expected 0.06-0.10)
3 Oct 19-25 retrieval (candidate pool) Retrieval Lead Unified pool = train structures U COCONUT U LOTUS U NPAtlas, deduplicated on tautomer-canonical InChIKey14, indexed by neutral monoisotopic mass with adduct table for the 7 test adducts; recall report at +/-5/10/20 ppm Pool recall@25 on hidden-proxy val (target >= 0.60; see K2)
4 Oct 26-Nov 01 model (spectrum -> fingerprint) Model Lead Spectrum->fingerprint MLP/transformer (Morgan r2 + r3 + MACCS heads) trained on Explorer; candidate scoring = Tanimoto to predicted fingerprint within mass window; wired into the notebook Local structure-disjoint MRR@25 (K3 reads this on Nov 01)
5 Nov 02-08 ranking Ranking Lead LambdaRank/LightGBM re-ranker over [fingerprint sim, spectral sim to analogs, formula agreement, mass error, pool provenance] features; paired-bootstrap significance harness (n = val molecules, 1,000 resamples) Local MRR@25 delta vs. fingerprint-only, with 95 % CI
6 Nov 09-15 submission-infra (stress) Submission Lead Notebook end-to-end on a synthetic 1,600-spectra / 450-molecule stress set in <= 6 h on Kaggle GPU; OOM guard (chunked candidate scoring); schema-robust loader (hidden test may carry columns/values absent from decoy) Wall-clock (K5)
7 Nov 16-22 model + ranking (ensemble) Model + Ranking Second orthogonal scorer (e.g. analog/library-propagated or in-silico fragmentation score within fixed time budget) fused by RRF or learned weights; "culprit-absent" gate (section 6) Ensemble - best single >= +0.01 local MRR@25 (K7)
8 Nov 23-29 ranking (polish) Ranking Lead Ablations; per-adduct / per-CE behaviour; tautomer-canonical dedupe of the 25 slots (duplicate InChIKey14 in one row wastes a slot) Local MRR@25; slot-waste rate = 0
9 Nov 30-Dec 06 submission-infra (freeze) Submission Lead + PI Code freeze 2026-12-04; the two candidate final notebooks versioned, each scored on the public LB at least once with no error; team roster final by Dec 06 12:00 UTC Both candidates scored; zero errored reruns
10 Dec 07-14 submission (select) PI Final two selected in the Kaggle UI by 2026-12-11 23:59 UTC per the pre-registered rule (section 5); post-mortem notes; code packaged for open-sourcing (winners must) Selection recorded in PREREG.md with hashes

Front-loading rationale. The three longest lead times are (i) pool construction (download + RDKit standardisation of ~0.7 M COCONUT structures at ~1-2 ms each is minutes, but tautomer canonicalisation with rdMolStandardize is 10-100x slower and must be done once, offline, on Explorer CPU nodes), (ii) Explorer environment build behind the proxy (http://10.99.0.130:3128, scp not rsync, CRLF breaks sbatch), and (iii) Kaggle offline packaging (every wheel, model and index must be a Kaggle Dataset; a single missing wheel is a 9 h delay). All three are week-1/2 items.

3. Kill criteria (dated, with the measurement each is read from)

Each is phrased so a reader can answer yes/no from one named artefact. "Local val" means the structure-disjoint hidden-proxy validation set (held-out inchikey14, timsTOF-like NP molecules) whose score is written by metric.py to val/<run_id>.json.

# Date (23:59 UTC) Criterion Measurement Action if triggered
K1 2026-10-14 No submission has a non-error public LB score > 0 Kaggle "My Submissions" page All modelling stops; only submission-infra work until a score exists
K2 2026-10-25 Pool recall@25 (truth InChIKey14 inside the +/-10 ppm candidate list) on local val < 0.50 val/recall_report.json, field recall_at_pool Week 4 is re-assigned entirely to pool construction (add formula-based enumeration of analogs, widen sources); model work deferred one week
K3 2026-11-01 Local structure-disjoint MRR@25 < 0.15 for any de novo generation branch val/<run_id>.json, mrr_at_25 Drop de novo generation; all-in on retrieval + rerank (seed criterion from brief)
K4 2026-11-08 Any model architecture not yet present in the val leaderboard with a score val/leaderboard.csv No new architecture may be started after this date; unlisted ones are abandoned
K5 2026-11-15 Kaggle stress-set rerun (1,600 spectra / 450 molecules, GPU) wall-clock > 6 h 00 m or OOM Kaggle notebook version log, "Run time" field Halve every per-molecule compute budget (forward-scoring budgets, pool width) until <= 6 h; no feature may be added that costs > 10 min on the stress set
K6 Rolling, cap by 2026-11-30 Modal cumulative billed spend >= $120 (lesson: sibling project hit a $37.58 spend limit mid-run) Modal billing dashboard, "billed to date" All remaining training moves to Explorer; Modal used only for < $5 inference tests
K7 2026-11-22 Ensemble/rerank does not beat best single scorer by >= +0.010 MRR@25 with paired-bootstrap 95 % CI excluding 0 val/ablation_<date>.json, fields delta, ci_low Ship the single scorer; ensemble work stops
K8 2026-12-04 Code freeze: any commit after this date touching ranking/scoring logic git log on src/rank*, src/score* Reverted unless it fixes a crash on the stress set (crash fixes allowed with a reviewer note)
K9 2026-12-06 12:00 UTC Any intended teammate has not accepted rules and merged Kaggle team page Proceed solo; no merge attempts in the final 36 h before Dec 07
K10 2026-12-11 Final two not selected in the Kaggle UI Kaggle "My Submissions", "Selected" ticks PI selects per section 5 immediately; no further experiments
K11 Standing A claimed improvement < 0.010 MRR@25 or with paired-bootstrap CI containing 0 val/ablation_*.json Not merged; logged as null result
K12 Standing from 2026-10-20 A change improves local val but drops public LB by > 0.04 (> ~1.5 SE, section 4) Kaggle score vs. val/<run_id>.json Investigate for a val/hidden mismatch before any further LB probes; do not simply revert

4. Submission cadence and what the public LB can and cannot tell us

Correcting the "decoy means meaningless" assumption. The visible test.parquet (1,213 spectra, 400 molecule_ids) is verbatim train rows (enveda-180). If the public LB were computed on those rows, a library search would score close to 1.0. Competitor write-ups instead report library search at ~0.06-0.10 and heavily engineered pipelines at 0.416-0.418 (leesirleesirlee-star, 2026-10-06). The only consistent reading is the standard Kaggle code-competition mechanism: the notebook is re-run on the real hidden test (~1,500 spectra / ~400 molecules), and the public LB is a subset of that. Consequences:

  1. Local evaluation on test.parquet is worthless (it measures memorisation). The decoy's only legitimate uses are schema and timing rehearsal.
  2. Public and private share one distribution (same instrument, same chemistry batch, same era). Shake-up from distribution shift between public and private should be small; shake-up from sampling noise is not.
  3. Noise quantified. Per-molecule reciprocal rank at a mean of ~0.4 has SD ~0.45 (mass at 0 and 1). Independent-sample SE = 0.45/sqrt(n): n = 400 -> 0.022; n = 200 -> 0.032; n = 120 -> 0.041. The public fraction is unverified (rules page unreadable), so assume 0.02-0.04. Two submissions that differ by < 0.03 on the public LB are indistinguishable; leesirleesirlee's three variants (0.416 / 0.417 / 0.418) are one sample. Paired differences on the same molecules have smaller SE (depends on how many molecules flip), so an A/B on the LB can detect ~0.015-0.02 only if the change is localised; broad re-rankers that shuffle many rows are noisier.
  4. What the public LB can tell us: gross pipeline validity (0 vs. 0.07 vs. 0.3), timing/OOM survival, and whether a category of change (new pool source, new scorer) moves the needle by >= 0.05. What it cannot: choose between variants within 0.03, or validate a val set that was built badly.

Daily slot policy (5/day, resets 00:00 UTC; assume an errored rerun also consumes a slot - unverified).

Slot Purpose Rule
1 Infra canary Any notebook with changed packaging (new Dataset, new wheel, new index) goes here first. Never bundle a modelling change with a packaging change.
2-3 Hypothesis probes At most one category of change per submission; the predicted local-val delta is written in the commit message before submission (pre-registration in miniature).
4 Confirmation Re-run of the current best candidate after any environment change (Kaggle image update, RDKit wheel refresh) to detect silent drift.
5 Reserve Unused until 18:00 UTC; then available for a crash retry.

Practical throughput is bounded by the 30 h/week GPU quota: a 2-4 h rerun each, so ~6-10 scored GPU submissions per week, not 35. Weeks 1-2 use CPU-only notebooks (library search) to preserve quota.

5. Final-two selection rule (to be pre-registered; see section 6 for the artefact)

Community practice, from write-ups we read: Montgomerie (2022) lost a gold medal in chaii by not selecting his best-CV submission because it looked weak publicly, then won gold in Jigsaw Toxic Severity by ignoring a public LB that was only 5 % of the test and jumping from "one thousand five hundred and somethingth" to 14th; Marmerola (Santander Customer Satisfaction, 5,123 teams) gained 470 places by "trust your CV" with a repeated 7-fold CV and a 0.003-AUC stability target that he equated to ~1,350 leaderboard places. The general lesson: pick on validation, use the LB as a sanity check, and diversify the two picks. Our situation differs in one way that strengthens the LB: public and private come from the same batch, so the LB is an unbiased (if noisy) sample. Hence:

Rule (fixed 2026-10-20, applied 2026-12-11):

  • Eligibility: a notebook version is eligible only if (a) it scored on the public LB without error, (b) its stress-set rerun took <= 6 h, (c) its code is at or before the 2026-12-04 freeze.
  • Pick A (val-pick): the eligible version with the highest local structure-disjoint MRR@25 whose public LB is within 0.05 of the best eligible public LB (the band excludes versions whose val/LB disagreement signals a broken val).
  • Pick B (LB-pick with diversity): the eligible version with the highest public LB. If that is Pick A, Pick B becomes the eligible version from a different scorer family (different pool composition or ranker) with local val within 0.02 of Pick A.
  • Ties (< 0.005 apart on either axis) break toward the shorter rerun time.
  • No one may change this rule after 2026-10-20 without a dated entry in PREREG.md stating what information prompted the change.

6. Forced analogy: jurisprudence of eyewitness line-ups (pre-registration, blind administration, the two-pick rule)

The analogy, precisely. On 2017-01-06 Deputy Attorney General Sally Yates issued the first department-wide DOJ policy on photo arrays (FBI, DEA, ATF, US Marshals). As reported by Federal Times and NPR/KPBS, it requires investigators to "document or record an eyewitness's confidence in an identification at the very moment the ID is made", by audio/video recording or verbatim transcription, and encourages "blind" administration (the agent does not know what the suspect looks like) or "blinded" administration (the agent knows the suspect but not the order of the photos), because an administrator who does not know the answer finds it "virtually impossible verbally or nonverbally ... to cue the witness". It implements the 2014 National Academies report Identifying the Culprit (doi:10.17226/18891), which called for standardised, written procedures. Map: the witness is our selection process; the photo array is the set of candidate final notebooks; the administrator is whoever has been staring at the public LB; the confidence statement is the predicted score written before the LB reveals anything; post-identification feedback (which inflates witness confidence) is the LB score itself.

Derived artefact: hive/PREREG.md, committed by 2026-10-20. Fields: (1) rule_version and commit hash; (2) the final-two rule verbatim (section 5); (3) the kill-criteria table (section 3) with artefact paths; (4) val_set_hash - SHA-256 of the validation molecule list, so the val set cannot be quietly re-cut after a bad LB result; (5) predicted_private_mrr with a 90 % interval for each candidate, written before its public score is seen (the contemporaneous confidence statement); (6) amendments - append-only, each with date, reason and the information that triggered it; (7) selection_record - filled on 2026-12-11 with the two version ids and which clause chose each. A template is at research/q7_compute_execution/prereg_template.md.

Derived testable idea: the culprit-absent line-up. Eyewitness research's central finding is that witnesses asked to pick from a line-up where the culprit is absent still pick someone - the best relative match - and the DOJ instruction "the perpetrator may or may not be present" exists to blunt that. Our analogue is exact: ~45 % of hidden molecules are not in the candidate pool at all (back-solved recall@25 ~0.55 at 0.417), yet the ranker still fills 25 slots with the best relative matches, all of which score 0. Idea: train a culprit-present classifier on val (features: top-1 fingerprint similarity, gap between top-1 and top-5, spectral-entropy similarity to nearest library analog, number of pool candidates in the mass window, formula-assignment confidence). For queries predicted culprit-absent with probability > p*, re-allocate some of the 25 slots to candidates produced by a different generator (analog edits of the nearest library hit: +/-CH2, +/-O, de-/glycosylation, methylation; or a de novo model if K3 did not kill it). Test: MRR@25 on val with vs. without re-allocation, paired bootstrap; prediction: gain if the second generator's hit rate on absent cases exceeds the loss on present cases mis-flagged as absent. The slot arithmetic is favourable because a slot at rank 15-25 is worth at most 1/15 even when correct.

Where the analogy holds, partially holds, breaks. Holds: pre-commitment against post-hoc rationalisation (the LB is exactly the kind of feedback that inflates confidence), and the contemporaneous prediction as a calibration record. Partially holds: blindness. A witness is one person; we are one human (Amit) plus agents, so the LB-watcher and the rule-author are the same mind. Procedural blindness substitutes for personnel blindness: the rule is hashed in git before the relevant LB results exist, and agents applying it are given the rule and the artefact paths, not the LB history. Breaks: (i) we can resubmit ~100 times and see a graded score each time; a witness views the array once and gets no score, so our risk is gradual over-fitting to the LB, not a single cued pick - which is why K12 limits reactions to LB drops rather than forbidding them; (ii) the witness picks one face, we pick 25 ranked faces, so the relevant literature (relative-judgment, sequential vs. simultaneous presentation) is about a different decision; (iii) a mis-identification in court is irreversible, whereas our worst case is a lower rank - the DOJ's level of procedural rigidity is proportionate to its stakes, not ours.

7. Risk register

# Risk P Impact Early-warning signal Mitigation Owner
R1 Kaggle rerun OOM/timeout on the hidden set (larger than decoy; per-candidate forward scoring scales with pool) H Errored final submission = 0 Stress-set run > 6 h or RSS > 12 GB; leesirleesirlee's ICEBERG budget initially covered only 18 % of molecules K5; chunked scoring; fixed wall-clock budgets per stage; write partial submission.csv early and overwrite Submission Lead
R2 RDKit version mismatch (grader 2026.03.3; PyPI default now 2026.3.6; tautomer canonicalisation can differ across versions) M Silent InChIKey14 mismatches, lost ranks metric.py self-test on 10k train SMILES disagrees between envs Pin rdkit==2026.3.3 in all envs; vendor the wheel as a Kaggle Dataset; CI test compares canonical keys across envs Data Eng
R3 COCONUT download/licence issues L Pool missing 0.7 M NPs 403/throttling from coconut.s3.uni-jena.de; schema change COCONUT is CC0 (no restriction); mirror Zenodo 10.5281/zenodo.13692394; LOTUS CC-BY-4.0 and NPAtlas CC-BY-NC-4.0 are fine for a non-commercial competition entry but record attribution in the open-sourced code Data Eng
R4 Explorer queue wait / 8 h gpu limit / monthly /scratch purge M Training slips a week; data loss squeue pending > 6 h; purge notice Use gyorilab partition when available (4xH200); checkpoint every epoch to /projects or /home (75 GB limit); request multigpu ticket in week 1 Model Lead
R5 Modal credit exhaustion mid-run M Lost training run, no budget for inference tests Billed-to-date > $80 K6 cap at $120; budget guard reads billed dollars (sibling lesson); Explorer as fallback Model Lead
R6 Hidden-test domain shift vs. our val (hidden = novel NPs on timsTOF; train is 82 instrument types) H Local val lies; wrong model chosen Val MRR@25 vs. public LB diverge by > 0.10 consistently Build val from held-out inchikey14, restricted to NP-like, timsTOF/QTOF spectra; K12 Ranking Lead
R7 Team/entry deadline missed (2026-12-07) L Teammate cannot join; solo entry Teammate has not accepted rules by Dec 01 K9: roster final by Dec 06 12:00 UTC PI
R8 LB shake-up (public subset noise) M Rank drop of tens of places Variants within 0.03 of each other on LB Section 5 rule: one val-pick, one LB-pick, diversity tie-break PI
R9 Local over-fitting to the decoy test.parquet (train rows) M Inflated local numbers; wrong decisions Any local score > 0.6 Never evaluate on test.parquet; use it only for schema/timing Ranking Lead
R10 Schema drift between decoy and hidden test (new instrument_type strings, missing CE values, adducts outside the 7) M Crash on hidden rerun None visible before scoring Loader tolerates unknown categories; default adduct handling; unit test with mutated decoy rows Submission Lead
R11 Duplicate InChIKey14 among the 25 guesses (tautomers/stereoisomers collapse) H Wasted slots; recall@25 lower than pool recall Slot-waste rate > 0 on val Dedupe on tautomer-canonical InChIKey14 before writing the row Ranking Lead
R12 Silent bugs (leesirleesirlee found five, "all silent rather than crashed") H Weeks lost Val score flat after a change that should move it Every change ships with a predicted delta; unexpected zeros trigger a diff review before another submission All
R13 Kaggle GPU quota exhausted by reruns M Cannot score experiments in a given week Quota < 6 h remaining on Wednesday CPU-only notebooks for library search; schedule GPU submissions Mon-Wed Submission Lead

8. Minimum viable submission and definitions of done

MVS (first non-zero submission): submitted 2026-10-12, scored by 2026-10-14. Library search only: +/-10 ppm precursor filter against train spectra of matching polarity, modified cosine + spectral-entropy similarity on cleaned top-64 peaks, RRF across the 1-16 spectra per molecule, dedupe on InChIKey14, mass-nearest backfill to 25 from the train structure list. CPU only, < 30 min. Expected public LB 0.06-0.10. Done = a score appears and submission.csv passes the format validator (no nulls, no duplicate molecule_id, <= 25 SMILES).

Milestone done-definitions: W1 done = version-asserted RDKit in three environments and all four database files checksummed on disk; W3 done = recall report exists and K2 is evaluated; W4 done = a fingerprint model's val score is in val/leaderboard.csv; W6 done = stress-set run time recorded in the notebook log; W9 done = two frozen versions each with an LB score; W10 done = selection_record filled.

Gotchas / what could fool us

  1. The decoy is a memorisation test. Any local number computed on test.parquet is meaningless; it can also leak into "validation" if someone de-duplicates by spectrum_id rather than inchikey14.
  2. LB resolution. Differences < 0.03 on the public LB are noise at any plausible public fraction; do not promote a variant on a 0.001 gain (0.417 -> 0.418 is exactly this).
  3. Pool size is not free. PubChem-scale pools hurt ranking (BRIEF) and multiply inference time; the pool metric is recall@25 and seconds per molecule.
  4. Tautomer canonicalisation drift across RDKit versions can silently turn a correct answer into a miss; the grader pin is 2026.03.3 and the current wheel is 2026.3.6.
  5. Slot waste. Stereoisomers and tautomers of one structure share an InChIKey14; 25 guesses that collapse to 12 keys throw away 13 slots.
  6. Rerun time scales with the hidden set, not the decoy. Budget for 1,600 spectra / 450 molecules, not 1,213 / 400.
  7. GPU-quota accounting. Reruns appear to consume the weekly quota (competitor write-up, unverified against rules); a week of careless probing can leave no quota for the confirmation run.
  8. Explorer scratch is purged monthly - checkpoints left there vanish.
  9. Val/LB disagreement is information, not an error - K12 exists so we investigate rather than flip-flop.
  10. Licence hygiene for open-sourcing: NPAtlas is CC-BY-NC-4.0, LOTUS CC-BY-4.0, COCONUT CC0; attribution must ride with any released pool.

Implications for the challenge

  1. Submit the library-search MVS on 2026-10-12 so K1 is never in play; it also calibrates our val against the LB on day 7.
  2. Spend weeks 1-3 on the pool and the val set, not on models: recall@25 is the ceiling and K2 is the first real gate.
  3. Freeze hive/PREREG.md (rule, kill criteria, val hash, predicted scores) by 2026-10-20, before the first fingerprint-model LB results arrive.
  4. Treat every LB submission as a hypothesis test with a pre-written predicted delta; 5 slots/day is a budget, not a target.
  5. Implement the culprit-absent gate (section 6) in week 7 as the one "serendipity" experiment, judged by K7's paired-bootstrap standard.
  6. Keep the inference wall-clock on every val report; a feature that adds > 10 min on the stress set needs to buy >= 0.01 MRR@25.
  7. Final picks: one val-pick, one LB-pick from a different scorer family, chosen on 2026-12-11 by the written rule.

Open questions

  • What fraction of the hidden set feeds the public LB? (Rules page unreadable; determines the SE in section 4.)
  • Do errored reruns count toward the 5/day limit and toward the GPU quota? (Assumed yes; unverified.)
  • Exact Kaggle rerun limit for this competition (9 h is from a competitor write-up) and whether GPU is enabled for the rerun.
  • Does the grader apply tautomer canonicalisation to our SMILES, the truth, or both? (Determines whether we should canonicalise before writing.)
  • Current COCONUT (Oct 2026) compound count; the paper's figure is 695,133 for the Sept 2024 release.

Sources