Hot data: C:\Temp\casmi26 (train/test parquet, COCONUT lite CSV, work/ artefacts). Python: C:\Temp\casmi26\venv
(3.12, rdkit==2026.03.3 — the version the organiser's scorer pins). Cluster copy: explorer:/scratch/shenoy.am/casmi26.
Run from this directory with the venv: C:\Temp\casmi26\venv\Scripts\python.exe scripts\<name>.py.
Package casmi/
| module | what |
|---|---|
keys.py |
metric identity: SMILES → tautomer-canonical InChIKey14 (metric_key), batch ≤ 4 procs |
metric.py |
MRR@25 exactly as scored, submission validator, bootstrap / paired-bootstrap CIs, H25 |
data.py |
train metadata loader, SpectrumStore (row-group random access), adduct ↔ neutral-mass table |
pool.py |
candidate pool = COCONUT ∪ train structures (729,389 keys), ±ppm window index |
search.py |
packed top-128 peak arrays, cosine library search with precursor-region stripping |
val.py |
block T = 250 enveda-np-examples (NPs on timsTOF) under masks class1 / class2 / class3 |
Scripts (in order)
| script | output | status |
|---|---|---|
00_build_pool.py |
work/pool.parquet, pool_stats_npexamples.json |
done — median 61 candidates at ±10 ppm, truth in window 98.8 %, chance MRR 0.077 |
04_pack_spectra.py |
work/train_{mz,it}_top128.npy |
running |
02_mock_witness.py |
work/mock_witness_T.json |
running — no-MS2 orderings vs chance |
03_baseline_library_search.py |
work/baseline_library_T*.{json,csv} |
next — first (c, r, e) numbers per class |
01_metric_keys.py |
work/pool_metric_keys.parquet |
done locally in 2.0 h (4 procs): 729,389 → 723,475 distinct metric keys (5,914 tautomer merges, 4 unparsable) |
fp/prep_labels.py |
work/pool_fp_bits.npy (Morgan r2+r3+MACCS, 4,263 bits), holdout_keys.json (T 250, E 1,000) |
done |
fp/train_fp.py → fp/score_holdout.py |
work/fp_v1_* |
done: T 0.387 / E 0.565 class-2 (scorer vectorised: 20 s for both blocks) |
fp/train_fp_v2.py |
work/fp_v2_* |
done: T 0.510 / E 0.838 class-2 (Explorer job 10977942, 97 min on H200) |
kaggle/build_dataset.py → kaggle/submit_baseline.py |
Kaggle dataset casmi26-pool, kernel casmi26-baseline-library-search |
dataset uploaded 10-10; kernel push next |
Results log (block T = 250 NPs on timsTOF unless stated)
| date | experiment | class1 | class2 | class3 | note |
|---|---|---|---|---|---|
| 10-10 | mock witness, no MS2, popularity ordering | — | 0.830 (hit@25 0.98) | — | popularity alone solves block T — famous molecules; inverted on the hidden set. Popularity banned as a feature. |
| 10-10 | mock witness, no MS2, random / | ppm | / NP-likeness | — | 0.119 / 0.162 / 0.169 |
| 10-10 | library search (cosine, ±10 ppm, 0.02 Da) + NP-likeness fallback | 0.932 (top1 0.89) | 0.115 (hit@25 0.50) | 0.054 | class3 non-zero = tautomer spellings of the truth in the pool (metric keys pending) |
| 10-10 | fp_v1: spectrum→(Morgan r2+r3+MACCS) MLP, 12 epochs H200 (BCE 0.072), log-likelihood ranking over ±10 ppm pool, no popularity | — | 0.387 [0.35–0.43] (hit@25 0.83, top1 0.24) | — | block E pending; structure-disjoint (T and E keys held out); analogs NOT removed — optimistic for scaffolds with close train relatives |
| 10-11 | fp_v1 on block E (1,000 enveda-180-only structures, timsTOF; class 2) | — | 0.565 [0.545–0.585] (hit@25 0.943, top1 0.416) | — | same chemistry family as 45 % of training (enveda-180) → optimistic for NPs; block T (0.387) is the NP number · Morgan-r2 bits only: T 0.371, E 0.528 (multi-fingerprint gain = all − r2) |
| 10-11 | fp_v2: fp_v1 + candidate-set softmax CE (K=31 mass-window decoys, lambda=1, tau=20), 6 epochs, analog-removed training (1,915 structures with Tanimoto >= 0.7 to a hold-out dropped) | — | T 0.510 [0.467–0.549] (hit@25 0.904, top1 0.364); E 0.838 (hit@25 0.995, top1 0.731) | — | paired delta vs fp_v1: T +0.123 [+0.092,+0.154], E +0.273 [+0.254,+0.292]; bit-BCE rose 0.072→0.130 (the ranking objective reshapes the probabilities — they are no longer calibrated bit posteriors) |
| 10-11 | pipeline = library gate + fp_v2 on block T (gate = promote library hits with cosine >= g to the top) | g=0.7: 0.917 · g=0.8: 0.869 · g=0.9: 0.788 · g=0.95: 0.600 · off: 0.510 | g=0.7: 0.455 · g=0.8: 0.476 · g=0.9: 0.498 · g=0.95: 0.509 · off: 0.510 | — | in class 2, 49 % of molecules have a cosine >= 0.7 hit to a different structure (library analog) that the gate wrongly promotes; expected hidden-set value (pi1 0.16, pi2 0.45) is ~0.35 for g = 0.7–0.9, 0.31 with no gate → keep g = 0.7. A "consistency" rule (promote only if also in fp top-K) loses 0.15–0.22 on class 1 and gains < 0.02 on class 2 — rejected. Proper fix = calibrated fusion (fp score + cosine as features), not a gate. |
| 10-11 | fp_v3: fp_v2 continued 6 epochs with K=63 decoys (lr 7e-5), analog-removed | — | T 0.501 [0.459–0.540]; E 0.845 | — | paired delta vs fp_v2: T -0.009 [-0.024,+0.007], E +0.007 [+0.000,+0.014] — no significant change; more/harder decoys are saturated. Keep fp_v2. Next levers: fusion reranker (cosine + fp score + window features) and class-3 coverage. |
| 10-11 | fp_v4: fp_v2 fine-tuned 3 epochs on timsTOF rows only (test-instrument adaptation) | — | T 0.502; E 0.844 | — | paired delta vs fp_v2: T -0.007 [-0.016,+0.001], E +0.006 [+0.001,+0.011] — no gain; the model is not instrument-limited on these blocks. Keep fp_v2. |
| 10-11 | fusion reranker v0 (LightGBM lambdarank on fp score/rank/gap + library cosine + window features), trained on block R = 1,500 training structures | T class1 0.342 (gate: 0.909) | T class2 0.557 (gate: 0.465, fp-only 0.509) | — | invalid: block R structures were in fp_v2's training set (fp-only MRR 0.895 on R vs 0.509 on T), so the reranker learned from leaked fp confidence (fp_gap = 85 % of gain) and under-weights the library. Retraining on fp hold-outs (E + T) with grouped CV. |
| 10-11 | fusion reranker v1: LightGBM lambdarank on [fp score, z, rank, gap, library cosine, n library spectra compared, ppm, NP-likeness, in_coconut, heavy atoms, window size, n spectra], trained ONLY on fp hold-outs (E 1,000 class-2 + T 250 class-1/2), class-1 rows weighted x8, 5-fold grouped CV | T class1 0.845 (gate 0.7: 0.909; fp-only 0.509) | T class2 0.537 (gate: 0.465; fp-only 0.509); E class2 0.987 (gate 0.626; fp-only 0.838) | — | expected hidden-set value (pi1 0.16, pi2 0.45): reranker 0.377 vs gate 0.355 vs fp-only 0.310; wins for any pi1 <= 0.5. Dominant features: fp_gap 67 %, then lib_ncmp / lib_cos / np_likeness. Caveat: E's 0.987 leans on 'truth has no library spectra' — weaker on COCONUT-dense windows (T shows the realistic +0.07 on class 2). → Kaggle kernel v3 pushed 10-11 (validated: reranker loaded, 400 rows, decoy 0.901 vs gate 0.998 — expected cost on exact library duplicates). |
| 10-11 | class-3 lever started: one-step analog enumeration from top-K fp candidates (17 reaction-SMARTS operators, mass-matched ±10 ppm), scored by fp log-likelihood; evaluated on T class3 (truth removed from pool) and the cost on class2 | — | — | class3: hit@25 0.076 → 0.200, MRR 0.045 → 0.055 | truth reachable by one edit of a top-5 parent in 64.8 % of class-3 queries (17 operators, parents searched at target − Δmass); when enumerated, the truth is rank 1 among enumerated candidates 40 % of the time, top-5 60 %, top-25 85 % — the ranking is fine; the 'tail' slot policy (enumerated after all pool candidates) is what buried it, since the pool window usually exceeds 25. Operators producing the truth: dehydroxylation 66/162, hydroxylation 54, O-methylation 11, O-demethylation/hexosylation/dehexosylation/hydrogenation 5 each. Class-2 cost of 'tail' = 0. Slot policy 'mix' (enumerated candidates compete by fp score): class3 MRR 0.055 → 0.299, hit@25 0.576; interleave5 0.097, interleave10 0.079 — rationing slots loses; candidates must compete on score. Class-2 cost of mix pending; enumeration reranker trained on the dumped candidates. fp_v5 training on gyorilab. |
| 10-11 | enumeration reranker (LightGBM over enumerated candidates: product score, parent score/rank/z, product−parent, product−pool-top, operator, NP-likeness of parent, window sizes), 5-fold CV on T class3 | — | — | truth MRR within the enumerated set 0.499 → 0.574 (top-1 0.40 → 0.46) | operator prior matters: dehydroxylation 41 %, aromatic/aliphatic hydroxylation 17/16 %, O-methylation 7 %. Full-pipeline evaluator (pool ∪ enumerated → fusion reranker) written; class-2 cost of 'mix' pending. |
| 10-11 | fp_v5: trained from scratch with the joint BCE + candidate-CE objective (12 epochs, lambda 0.5) | — | T 0.341; E 0.772 | — | paired delta vs fp_v2: T -0.168 [-0.201,-0.135], E -0.067 — clearly worse: BCE pre-training then CE fine-tuning (fp_v2) beats joint training from scratch. Keep fp_v2. |
| 10-11 | full pipeline with enumeration (pool ∪ one-step enumerated candidates, all scored by the fusion reranker; enumerated get lib_cos = lib_ncmp = 0) | 0.885 (no-enum 0.845) | pen 0: 0.425 · pen 1.0: 0.528 · pen 2.0: 0.584 (no-enum 0.537) | pen 0: 0.349 · pen 1.0: 0.320 · pen 2.0: 0.288 (no-enum ≈ 0.05) | expected hidden-set value (0.16/0.45/0.39): no-enum 0.396 → pen 0 0.469 → pen 1 0.504 → pen 2 0.517. pen 3.0: class2 0.620 / class3 0.249 → 0.518; pen 4.0: 0.637 / 0.223 → 0.516 — flat optimum over penalties 2–4 under this class mix; penalty 2.0 kept (most class-3 upside). Caveat: class 2 keeps rising with the penalty (0.537 → 0.637) although enumerated candidates are demoted — the fusion reranker's window-relative features (z, gap, n_window) shift when ~115 enumerated near-isomers are added; mechanism unexplained → next step is to retrain the fusion reranker with enumerated candidates present in its training features. Kernel v4 pushed 10-11. |
| 10-11 | Kaggle kernel v5 = v4 + exact-match guard (no enumeration when best library cosine ≥ 0.95) | decoy 0.901 (= v3); T class1 0.964 | T 0.584 (pen 2) | T 0.288 (pen 2) | validated (reranker loaded, 400 rows, 14.5 min). Recommended submission. Expected hidden-set ≈ 0.52 vs v3 0.40. |
| 10-11 | fusion reranker v2 (trained on pool ∪ enumerated with an enum feature; E class2 + T class0/1/2; grouped CV) |
T class0 0.977, class1 0.868 | T 0.542; E 0.988 | pending (T class3 dump running) | not better than reranker v1 + fixed penalty 2 on class 1/2 (expected 0.383 vs 0.404 on those two classes); lib_cos now carries 52 % of gain. Decision after class 3. |
| 10-11 | Kaggle kernel v7 = fp_v2 + analog-augmented fusion reranker (rerank3) + one-step enumeration (pen 2) + exact-match guard | decoy 0.914 (v5 0.901) | T class2 (reranker CV, no enumeration) 0.634 | — | validated: 'loaded (analog-augmented)', 400 rows, 26.8 min. Recommended submission. (v6 ran against the previous dataset version and is identical to v5 — ignore.) Expected hidden-set ≈ 0.55–0.56 on the block-T class mix. |
| 10-10 | Kaggle kernel v4 casmi26-baseline-library-search (library + NP-likeness fallback) |
decoy test 0.9975 (sanity) | — | — | 12 min on Kaggle CPU; submission.csv valid (400 rows); awaiting UI submit |
Validation protocol (from hive q7 / Emperor tick 1)
- Primary selector: block T, three masks. Promote a change only if Δ on class2 is ≥ 0 with a paired-bootstrap 90 % CI that excludes 0, and class1 does not fall.
- Public LB deltas < 0.01 are noise (SE ≈ 0.04 on ~130 public molecules). Never select on them.
- Pre-registration (
hive/PREREG.md) must be frozen by 2026-10-20 with the val-set hash and the final-two rule. - Block Q (≈1,000 scaffold-disjoint QTOF NPs, analogs removed) is the second, larger hold-out — to build.
Kill criteria carried from the 90-day plan
- K2 (2026-10-25): pool recall@25 on class2 < 0.50 ⇒ week 4 goes entirely to pool construction.
- Fingerprint model must beat library-search+popularity fallback on class2 by ≥ 0.02 before it earns GPU time.
Next levers (not yet tried)
- Two-step enumeration restricted to the five dominant operators (reachability beyond 65 % for class 3), with the enumeration reranker (within-set MRR 0.50 → 0.57) applied inside the fusion.
- A class-membership detector (truth-in-pool vs not) to set the enumeration penalty per query — pool-confidence proxies failed; try features from the analog engine (max analog cosine, analog agreement).
- Retrain the fusion reranker with enumerated candidates present and analog features (the current v7 reranker never saw enumerated rows).
- Telluric-mask retraining (train with isotope satellites removed) — test-time masking was mixed.
- Calibration of fusion scores to probabilities for principled slot allocation.
Breakthrough-idea sweep (10-11) — cross-domain imports and competitor ideas, each tested on the hold-outs
Baseline = fp_v2, mean-logit aggregation, ±10 ppm: T class2 0.510, E class2 0.838 (fp-only); kernel v5 pipeline EV ≈ 0.517.
| idea | origin | result (T class2 / E class2 unless stated) | verdict |
|---|---|---|---|
| Evidence accumulation: sum of per-spectrum log-likelihoods | forensics / physics | identical to mean-logit (0.510 / 0.838); max-over-spectra 0.536 / 0.813 | reject (sum ≡ mean; max is mixed) |
| Burrows' Delta / candidate-conditional IDF (weight bits by window disagreement) | stylometry, cryptanalysis | 0.389 / 0.777 (sqrt variant 0.439 / 0.830) | reject — unanimous bits carry the "typical member" prior |
| Known-plaintext calibration: asymmetric / narrower windows | cryptanalysis + hive's +1.9 ppm bias | −4/+5: 0.507 / 0.834; ±5: 0.511 / 0.844; ±15: 0.499 / 0.833 | no robust gain; keep ±10 |
| Telluric mask (drop 13C / M+1 / M+2 satellites at inference) | astronomy | 0.518 / 0.818 | mixed; would need retraining with the mask to judge |
| Morelli (drop the base peak at inference) | art attribution | 0.470 / 0.808 | reject at test time (model trained with it) |
| matchms-style low-intensity cut (< 1 % of base) at inference | tool-default check (hive gotcha) | 0.502 / 0.818 | reject — the tiny peaks carry information, as the hive predicted |
| wider precursor-region strip (−2.1 Da) | instrument physics | 0.509 / 0.838 | no effect |
| Culprit-absent triage: confidence-adaptive enumeration penalty | eyewitness line-ups, Rembrandt A/B/C | with class 3: fixed 2 → 0.980/0.964/0.584/0.288, EV 0.529; best rule (cos+gap) → 0.986/0.974/0.623/0.252, EV 0.534; fixed 3 EV 0.532 | reject (+0.005 < bar): class 2 wants a high penalty, class 3 a low one, and pool-confidence proxies (score gap, top score, max cosine) cannot separate the two classes — a real absence detector would need other evidence |
| Analog propagation (nearest library analog at shifted mass × Tanimoto to candidates) as an orthogonal reranker feature | CASMI-2016 consensus lesson; molecular networking | fusion reranker CV: T class2 0.554 → 0.665 (Δ +0.111 [+0.080, +0.143]), T class1 0.847 → 0.913, E class2 +0.002 | ADOPTED — honest re-run with analog features on every block: T class2 0.537 → 0.634 (Δ +0.097 [+0.065, +0.129]), class1 +0.006, class0 −0.008, E +0.000. First orthogonal engine since the fp model; the nearest analog's scaffold tells the ranker which window candidates are plausible. → kernel v6 (rerank3 + enumeration pen 2 + guard). Open: effect on class 3 (enumerated candidates also receive analog features) not yet measured locally. |
| Two-step enumeration reachability for class 3 | class-3 ceiling analysis | aborted: 144 operator pairs × shifted windows ≈ minutes per query (hours per block) | too expensive in this form; revisit with the 5 dominant operators only (dehydroxylation, hydroxylations, O-methylation/demethylation) |
Breakthrough round 2 (10-12) — new data, new representations, mechanistic / biosynthetic inductive bias, chemistry-aware cleaning
Reference for reranker experiments = v7 fusion reranker (base + analog features), grouped 5-fold CV: E class2 0.987, T class0 0.973, T class1 0.867, T class2 0.634. All deltas are paired bootstraps (90 % CI). Scripts: fp/formula_feature.py, fp/frag_feature.py, fp/chem_feature.py, fp/npclass_prior.py, fp/train_rerank4.py, fp/ce_weight_probe.py, fp/clean_pack.py, fp/clean_holdout.py, fp/ensemble_probs.py, fp/analog_enum.py, fp/score_full2.py.
Data facts established first. No ion-mobility / CCS, retention time or MS1 isotope columns exist in train or test (a lack of data: those filters are impossible here). Windows (±10 ppm) hold a median of 4 (T) / 7 (E) distinct molecular formulas; the truth's formula is the most populous one only 88 % (T) / 62 % (E) of the time. Fragment mass accuracy on the true formula is intensity-dependent: intensity-weighted 69 % of peaks lie within 10 ppm of a valid sub-formula; peaks under 50 absolute ion counts only 20 %, above 5,000 counts 77 %; m/z < 100 peaks 38 %; the positive-mode offset is +1.9 ppm (negative −0.7). COCONUT already contains NPAtlas and LOTUS almost entirely (324 of 185 k structures missing from the pool) — the pool cannot be grown from those sources.
| idea | inductive bias | result | verdict |
|---|---|---|---|
| Sub-formula explanation engine (SIRIUS-style): every fragment ion must be a sub-formula of the precursor-ion formula with fragment- and loss-DBE rules; scored as a target-decoy LLR against each formula's chance rate on a bank of real peaks from unrelated spectra | mechanistic + stochastic | raw explained fraction favours element-rich formulas (truth formula rank-1 in only 28 % of multi-formula T queries); the decoy-null LLR fixes it (63 % on T, 31 % on E). In the fusion reranker: E +0.001 [−0.001, +0.003], T class1 +0.017 [+0.004, +0.030], T class2 −0.010 [−0.025, +0.005] | reject as a ranking feature — composition swaps inside a 10 ppm window (CH4 ↔ O etc.) are below the fragment tolerance and the fp model already encodes composition. Kept as the cleaning/noise engine. |
| In-silico bond-cleavage fragmentation (MetFrag-style: 1 acyclic cut, 2 acyclic cuts, ring openings, ±2 H), decoy-null LLR, top-40 fp candidates | mechanistic (generative direction) | alone: MRR 0.78 among the top-40 on E, 0.39 on T; in the fusion: E +0.002 [−0.001, +0.006], T class2 +0.013 [−0.013, +0.038], class1 +0.008, class0 +0.001 | mildly positive everywhere but inside the noise; EV ≈ +0.006 — below the promotion bar; keep as a candidate for the v8 reranker if other gains stack |
| Ionisation chemistry × adduct profile (which adducts/polarities the lab recorded for the molecule, crossed with acid/phenol/amine/sugar/quaternary-N counts of the candidate) | intuitive chemistry; "absence of a negative-mode spectrum is information" | E +0.002, T class2 −0.003, class1 +0.007 | reject — the fp model's adduct one-hot and bits already carry it |
| Biosynthetic-class prior from a new data source: COCONUT's NPClassifier pathway/superclass/glycoside labels, a multi-task head on the predicted fingerprint (structure-disjoint val: superclass top-1 0.65, pathway 0.83, glycoside 0.95); candidate feature = the query's log-posterior of its class | biological ontology | T class2 −0.007, class1 +0.002, class0 +0.003 (E not usable: enveda-180-only truths carry no COCONUT label, which would leak) | reject — an ontology projection of the same fingerprint evidence adds no new information |
| Collision-energy breakdown (probe for a multi-channel CE model): weight a molecule's spectra by CE bucket when fusing logits | mechanistic (survival yield) | T wants high-CE down-weighted (+0.014, n.s.), E wants it up-weighted (+0.003) | contradictory → no consistent signal; the plain mean over spectra stays; the multi-channel model was not built |
| Absolute-count noise floor at inference (drop peaks < 300 / < 1,000 ion counts) | detector physics | 0.496 / 0.837 and 0.491 / 0.825 vs 0.510 / 0.838 | reject at test time (as with the relative cut: the model was trained on the noise) |
| Window-consensus formula cleaning at inference (keep a peak only if some window formula explains it; + 100-count floor): 31 % of peaks and 18 % of intensity removed from hold-out spectra | chemistry-aware denoising | fp_v2: 0.506 / 0.840 vs 0.510 / 0.838 | neutral on the noisy-trained model — the fair test is the model trained on cleaned spectra (below) |
Train on chemically cleaned spectra (clean_pack.py on Explorer: true-formula sub-formula filter + 100-count floor removed 32 % of peaks, 16 % of intensity, 2.14 M of 2.54 M rows cleaned; fp_v2 recipe retrained as fp_v2clean, Explorer job 11000809, V100, 6 epochs ≈ 1.6 h; test-time cleaning by window consensus) |
the same bias at train and test time | fp_v2clean: T 0.502 / E 0.841 (true-formula-cleaned hold-outs), 0.501 / 0.842 with the honest window-consensus mask — vs fp_v2 0.510 / 0.838 | no gain — the MLP already learns to ignore the noise it was trained on; the removed 16 % of intensity was not masquerading as signal |
| Ensemble of fp_v2 + fp_v3 + fp_v4 (logit mean) | variance reduction | 0.511 / 0.844 vs 0.510 / 0.838 | +0.001 / +0.006 — v3/v4 are near-copies of v2 |
| Ensemble of fp_v2 + fp_v2clean (different training data) | variance reduction | 0.511 / 0.847 | +0.001 on T, +0.009 on E — real but small; costs +374 MB and 2× inference; not deployed (hidden set is T-like) |
Audit finding (10-12): the full-pipeline class-2 numbers in the sweep above (e.g. 0.584 with enumeration, penalty sweeps) were scored with rerank_fp_v2.txt, which was trained on the block-T dumps themselves — in-sample (pool-only class 2 scores 0.987 with it). Class-3 numbers are unaffected (truth rows unseen). All full-pipeline comparisons below use rerank_E.txt (trained on block E only, score_full2.py --rerank rerank_E), which is out-of-sample for T: pool-only T class2 = 0.569 [0.528, 0.609]. (That model never saw a truth with lib_cos > 0, so its class-1 number is meaningless; class 1 stays covered by the deployed reranker + exact-match guard.) |
validation hygiene | — | protocol fixed |
| Analog-driven enumeration for class 3: start from library spectra that resemble the query at a shifted mass (the analog-propagation hits), apply only the operator(s) whose Δmass equals the shift, score products by fp LL + analog cosine | mechanistic + evidence-grounded generator | T class3, enumerated list alone: reachable 56 % with 17 candidates/query (fp-parent enumeration: 65 % with ~115), MRR 0.376 [0.329, 0.421]; truth produced by dehydroxylation 25, dehexosylation 19, O-demethylation 18, hydroxylation 24, O-methylation 11… | promising: a compact, spectrally justified candidate set; full-pipeline test below |
Honest full-pipeline generator comparison (block T, out-of-fold rerankers rerank_cv_{k} from train_rerank_cv.py — 5 folds by query over E class2 + T class0/1/2 + T class3 with enumerated rows; score_full2.py --rerank cv, exact-match guard 0.95, features as deployed). EV = 0.16 × class1 (0.96) + 0.45 × class2 + 0.39 × class3.
| generator | penalty / bonus | T class2 | T class3 | enumerated / query | EV (39 % class 3) | EV if class 3 = 20 % |
|---|---|---|---|---|---|---|
| none (pool only) | — | 0.530 [0.487, 0.570] | ≈ 0.045 | 0 | 0.410 | 0.502 |
| fp-parent one-step (= v7) | 2 / 0 | 0.416 [0.375, 0.458] | 0.286 [0.244, 0.327] | 112 | 0.452 | 0.477 |
| analog-driven | 2 / 0 | 0.488 [0.447, 0.529] | 0.212 [0.174, 0.251] | 15 | 0.456 | 0.508 |
| analog-driven | 2 / 2·cos | 0.451 [0.411, 0.492] | 0.264 [0.224, 0.305] | 15 | 0.460 | 0.496 |
| union | 2 / 0 | 0.416 [0.374, 0.458] | 0.284 [0.244, 0.324] | 121 | 0.452 | 0.477 |
| union | 2 / 2·cos | 0.380 [0.339, 0.419] | 0.354 [0.311, 0.398] | 122 | 0.463 | 0.468 |
| union | 3 / 2·cos | 0.460 [0.417, 0.502] | 0.315 [0.274, 0.357] | 121 | 0.484 | 0.512 |
| union | 4 / 3·cos | 0.481 [0.438, 0.523] | 0.299 [0.259, 0.339] | 121 | 0.487 | 0.521 |
Reading: the v7 enumeration's class-2 cost is far larger than the in-sample sweep suggested (−0.11, not −0.05), because the ~110 fingerprint-tailored near-isomers push the truth down the reranker's rank/gap features. The analog-driven generator alone reaches the same EV with 15 candidates and a quarter of the class-2 cost. Combining both generators with a stiffer penalty (4) and a cosine bonus (3 × cosine of the analog hit, i.e. enumerated candidates only win a slot when a library spectrum supports that specific edit) recovers most of class 2 (0.481) while keeping class 3 at 0.30 — EV +0.035 over the v7 configuration on the 39 % class-3 mix and +0.044 if class 3 is 20 %.
| 10-12 | Kaggle kernel v8 = v7 + analog-driven generator only (GEN="analog", penalty 2, no bonus) | decoy 0.9167 (v7 re-scored 0.9167 — identical, the guard skips enumeration on class-0 decoys) | honest T class2 0.488 / class3 0.212 | 23.3 min | validated; robust option if class 3 is rare |
| 10-12 | Kaggle kernel v9 = union generator, penalty 4, bonus 3·cos (GEN="both") | decoy 0.9167 (= v7/v8; guard skips enumeration on exact matches) | honest T class2 0.481 / class3 0.299 — best EV (+0.035 over v7 at 39 % class 3, +0.044 at 20 %) | 27.9 min, 400 valid rows, 'loaded (analog-augmented)' | Submitted 2026-10-12 via Kaggle CLI (competitions submit -k ... -v 9, submission 57118690; public LB 0.237 — rank 2175 of 3166; the public-notebook plateau is 0.440 (316 teams), top 0.491). LB probes 10-12: no generators 0.276, no generators + no reranker (fp + library gate) 0.249 → generators cost −0.04 on the hidden set, reranker +0.03; core fp+pool path ≈ 0.25 vs public v4b engine 0.354. Kernel fp-only path replayed locally on block T = 0.5107 (= local 0.510) → no inference bug; the hidden chemistry is harder than np-examples for fp_v2 and the public stack's extra evidence channels (analog propagation, MetFrag, subformula, 100-feature ranker on simulated regimes, PubChem gate, simulators) carry its 0.44 Public-base fork (unchanged flexonafft V83, kernel casmi26-public-base v1, submission 57121760) scored 0.415, not 0.440: V83's last cell overwrites submission.csv with an untested "joint common ranking" heuristic for weak-library molecules and keeps the real E006 (0.440) result only as submission_e006_control.csv, which the competition refuses as a submission file name. Fork v2 = same notebook minus that overlay cell (E006 reproduction) pushed 10-13 03:55 UTC, submission 57130021 scored 0.440 = exact reproduction of the public plateau under our account (hidden rerun ~3.7 h); commit run on the 12-molecule smoke set takes ~18 min (ICEBERG 11/12 molecules, GLACIER [M+H]+ only). First layered variant: kernel casmi26-public-plus-kgl v1 (folder C:\Temp\casmi26\kaggle_kernel_pubkgl) = E006 + Engine 3 = our probe-a engine (eng/kgl_engine.py: fp_v2 likelihood + fusion reranker, GEN none, top-40 lists as JSON, run in a subprocess right after the smoke switch) whose candidates join the ICEBERG/GLACIER input and the final reciprocal-rank fusion as a third list with weight KGL_BETA 0.3 (engine 2 uses 0.6). Local 12-molecule smoke of the engine script: 31 s, 12/12 lists. plus-kgl v1 (submission 57130512) turned out to add our list only at the fusion cell, which the later metric-identity/E006 overlay cells rebuild from the saved Engine 2 lists → it tests only the ICE_UNION simulator-input union. plus-kgl v3 (v2 failed: merge cell before Engine 2) merges our top-40 lists into the Engine 2 lists by reciprocal rank (offset 3, ours at relative weight 0.5) right after Engine 2, so every downstream stage sees them; smoke commit: 41 new candidates in 12 top-40 lists, top-1 unchanged; Results 10-13: plus-kgl v1 (simulator-input union only) 0.440 = no change; plus-kgl v3 (our lists merged into Engine 2 at relative weight 0.5) 0.433 (−0.007): our fp_v2 ranking adds no complementary information at the list level and dilutes the stronger Engine 2 list. Next: tail-only insertion of novel analog-derived candidates (class-3 generator) for weak-library molecules, which cannot displace pool candidates above slot 22 → plus-kgl v4 (submission 57140620) 0.440 = no change: either few hidden molecules are weak-library with an analog hit ≥ 0.5, or the novel candidates are never the truth; hidden-run counts are not visible. Net 10-13: four variants of our channels on the public base, none above 0.440. |
Intermediate-representation / auxiliary-learning track (10-15, user request)
Hand inspection of 41 annotated flavone-family timsTOF spectra: research/_synthesis/flavone_inspection.md (atlas
https://claude.ai/artifact/EvepwVH8vQfhj9AdroHK3D). Scripts: fp/prep_targets.py (count-Morgan, FCFP, atom-pair, torsion,
E-state, physchem, element counts for all 729 k pool structures; work/pool_tgt_*.npy), fp/train_fp_aux.py
(multi-head trunk, MLP or peak-set transformer, structure heads + auxiliary heads), fp/score_targets.py (per-head and
fused window ranking, paired delta vs the sweep base), fp/train_dreams_fp.py + explorer/setup_dreams.sbatch +
explorer/dreams_fp.sbatch (DreaMS foundation model, frozen and fine-tuned), explorer/sweep_aux.sbatch
(sequential 10-config sweep, 600 k-spectra budget, 8 epochs, analog-removed, structure-disjoint).
| date | experiment | T class2 | E class2 | verdict |
|---|---|---|---|---|
| 10-15 | fp_v2 probabilities scored by fuzzy Tanimoto / Dice / cosine instead of bit log-likelihood (score_variants.py --sim) |
0.423 / 0.423 / 0.441 (loglik 0.510) | 0.757 / 0.757 / 0.767 (0.838) | reject — the likelihood scoring is the right comparison for a calibrated bit predictor; Tanimoto discards the model's confidence |
| 10-16 | aux_base = Morgan r2+r3+MACCS bits, MLP, 600 k spectra / 8 epochs (the sweep's reference; fp_v1 on full data was 0.387 / 0.565) | 0.211 [0.183, 0.239] | 0.198 [0.183, 0.213] | reference (MACCS head alone on E 0.232 > fused 0.198: window-z fusion is not free) |
| 10-16 | aux_m2 = Morgan r2 head only | 0.187 (Δ −0.024 [−0.043, −0.005]) | 0.126 (Δ −0.073) | the extra r3 + MACCS heads help even at this budget |
| 10-16 | aux_count2 = count-Morgan regression (MSE on z-scored log counts, Gaussian-LL ranking) | 0.098 (Δ −0.113) | 0.070 (Δ −0.129) | reject as a sole target: regression + distance ranking is far weaker than bit likelihoods |
| 10-16 | aux_fcfp = Morgan r2 + FCFP (functional-class) bits | 0.146 (Δ −0.065) | 0.110 (Δ −0.089) | worse; the morgan2 head itself degraded (0.169 vs 0.205 in base) — run-to-run variance at this budget is being measured with aux_base_rep |
| 10-16 | aux_aptt = Morgan r2 + atom-pair + torsion bits | 0.140 (Δ −0.071) | 0.114 (Δ −0.084) | worse; topology-at-distance bits are not more predictable from fragments |
| 10-16 | LightGBM probe fp/physchem_probe.py (1-Da bins → descriptor, 200 k spectra; R² on hold-out spectra, vs precursor-mass-only trees) |
T: logP 0.32 (mass-only 0.07), rotatable bonds 0.25 (−0.10), aromatic rings 0.25 (0.00), Fsp3 0.43 (0.23), HBD 0.58 (0.43), N count 0.19 (−0.38), S −0.17 (−0.55); MolWt/heavy atoms/ASA ≈ 0.96 from mass alone | E: everything low (narrow mass window): aromatic rings 0.30 (0.11), Fsp3 0.29 (0.07), halogens 0.29–0.36 (0.04) | the fragments carry saturation / aromaticity / heteroatom information beyond the mass — exactly the "composition per ring" a human reads; whether that regularises the fingerprint trunk is what aux_base_pc / aux_base_es test |
| 10-16 | DreaMS frozen (pre-trained 116 M-param encoder, Bushuiev 2024; precursor-token embedding + 2-layer head, same 600 k / analog-removed protocol, 2 epochs) | 0.315 [0.279, 0.351] (hit@25 0.84) | 0.283 [0.265, 0.300] | +0.10 on T, +0.085 on E over aux_base at equal data budget: a repository-scale pre-trained spectrum embedding beats our from-scratch trunk when data are scarce. Next: cache embeddings for all 2.5 M spectra (fp/dreams_embed_all.py) and train the fp_v2 recipe on them (fp/train_emb_head.py, job dreams_head.sbatch) for the full-data comparison with fp_v2 (0.510 / 0.838) and an ensemble. |
| 10-16 | aux_base_rep = exact replicate of aux_base (different init / shuffle) | 0.217 (Δ +0.006 [−0.004, +0.015]) | 0.198 (Δ −0.000 [−0.004, +0.003]) | run-to-run noise ≈ ±0.01 on T, ±0.003 on E at this budget — the losses above are real, not seed noise |
| 10-16 | aux_phys = E-state + physchem + element-count regression heads only (no bits) | 0.121 (Δ −0.090) | 0.122 (Δ −0.076) | reject as targets: descriptor space is far too coarse to rank a ±10 ppm window |
| 10-16 | aux_base_pc = bits + physchem/elements auxiliary heads (weight 0.3) | 0.199 (Δ −0.012 [−0.022, −0.003]) | 0.195 (Δ −0.003 n.s.) | auxiliary descriptor regression does not regularise the trunk; slightly hurts T |
| 10-16 | aux_base_es = bits + E-state/count-Morgan auxiliary heads | 0.219 (Δ +0.008 n.s.) | 0.198 (Δ 0.000) | neutral — within replicate noise |
| 10-16 | aux_all = all seven structure heads + three auxiliary heads | 0.154 (Δ −0.057) | 0.155 (Δ −0.043) | more heads dilute the trunk; the bit heads themselves drop (morgan2 0.170 vs 0.205) |
| 10-16 | aux_xf = same bit targets, peak-set transformer trunk (Fourier m/z + neutral-loss tokens, d 256, 4 layers; 40 min vs 12 min on V100) | 0.317 [0.281, 0.352] (Δ +0.106 [+0.081, +0.133]) | 0.418 [0.398, 0.438] (Δ +0.220 [+0.204, +0.235]) | the single biggest architectural gain seen in this project at equal data; per-head MACCS 0.437 on E. The MLP over 0.1-Da bins is the bottleneck, not the target. Full-data run xf_full queued (job 11126913, BCE only, compare with fp_v1 0.387 / 0.565) |
| 10-16 | DreaMS fine-tuned (all 106 M params, 3 epochs, 600 k, BCE 0.076; dreams_ft) |
0.450 [0.411, 0.490] (hit@25 0.90, top1 0.29) | 0.572 [0.551, 0.591] (hit@25 0.95, top1 0.42) | at 600 k spectra it already exceeds fp_v1 on the full 2.5 M (0.387 / 0.565) — pre-training on 700 M unlabeled spectra is worth ~4× our labelled data. Full-data fine-tune with the fp_v2 recipe (candidate-CE, warm start from dreams_ft, fp16, checkpoint/resume) queued as dreams_full (job 11126914); embedding-head + fp_v2 ensemble as dreams_head (job 11126913) |
| 10-16 | dreams_head = head on cached frozen-DreaMS embeddings, full 2.2 M analog-removed spectra, fp_v2 recipe (10 BCE + 4 candidate-CE epochs; fp/train_emb_head.py) |
after BCE 0.274; after CE 0.357 (Δ vs fp_v2 −0.153 [−0.187, −0.120]) | 0.404 → 0.770 (Δ −0.068 [−0.082, −0.054]) | candidate-CE nearly doubles E again (as fp_v1→fp_v2). But full-data frozen embeddings (0.357 on T) lose to the 600 k fine-tuned encoder (0.450): fine-tuning the encoder beats 4× more data through a frozen one |
| 10-16 | ens_v2_dreams = logit mean of fp_v2 and dreams_head (equal weights) | 0.482 (Δ −0.027 [−0.046, −0.011]) | 0.847 (Δ +0.008 [+0.000, +0.017]) | equal weighting hurts T, marginal on E; weighted variants running (explorer/ens_eval.sbatch) |
| 10-16 | xf_full = peak-set transformer, full data, 8 epochs BCE only (fp_v1-comparable; 2.4 h V100) | 0.373 (Δ vs aux_base +0.162) | 0.651 (Δ +0.453) | vs fp_v1 (binned MLP, BCE, full data) 0.387 / 0.565: same on T, +0.09 on E with no candidate-CE yet. xf_full_ce (warm start + 3 epochs BCE + candidate-CE, job 11130597) is the fp_v2-comparable run |
| 10-16 | dreams_full attempt 1 (fp16 autocast, bs 128) | 0.112 | 0.049 | NaN from step 0: the DreaMS encoder overflows under fp16 autocast on V100 (not a divergence — the first forward is already NaN). Run archived in work/nan_run/; resubmitted in fp32, bs 64 (job 11130565, ~6.5 h, checkpoint/resume) |
| 10-16 | weighted ensembles fp_v2 + dreams_head (logit mean) | 0.7/0.3: 0.502 (Δ −0.008 n.s.); 0.6/0.4: 0.492 (Δ −0.018) | 0.7/0.3: 0.848 (Δ +0.009); 0.6/0.4: 0.851 (Δ +0.012 [+0.005, +0.019]) | the frozen-embedding head adds a little on E, costs T — not worth shipping on its own |
| 10-16 | xf_full_ce = full-data peak transformer, fp_v2 recipe (warm start from xf_full + 3 epochs BCE + candidate-CE; 1.9 h V100; 68 MB weights) | 0.459 [0.418, 0.497] (Δ vs fp_v2 −0.051 [−0.078, −0.021]) | 0.861 [0.847, 0.874] (Δ +0.022 [+0.010, +0.035]) | first single model to beat fp_v2 on E; candidate-CE took it from 0.667 → 0.861 (same jump as fp_v1 → fp_v2). Per-head: morgan2 0.835 > morgan3 0.773 > maccs 0.704 on E |
| 10-16 | ens_v2_xfce = logit mean fp_v2 + xf_full_ce (equal weights) | 0.519 (Δ +0.009 [−0.008, +0.025]) | 0.878 (Δ +0.040 [+0.031, +0.049]) | new best on both blocks; top-1 0.794 on E (fp_v2 0.746). The binned MLP and the peak transformer make different errors. Weight sweep + 3-way with dreams_head running (explorer/ens_eval2.sbatch) |
| 10-16 | weight sweep fp_v2 : xf_full_ce = 0.6/0.4, 0.5/0.5, 0.4/0.6, 0.3/0.7; 3-way + dreams_head (0.5/0.4/0.1 best) | 0.517, 0.519, 0.517, 0.511; 3-way 0.512 | 0.876, 0.878, 0.874, 0.872; 3-way 0.879 | flat within ±0.005 — equal weights, two models is the pick; the frozen-embedding head adds nothing once the transformer is in |
| 10-16 | Kaggle probe casmi26-fp-probe-xf = probe-a (fp_v2 likelihood + fusion reranker, GEN none; LB 0.276) with the fp logits replaced by the equal-weight logit mean of fp_v2 and xf_full_ce (fp/xf_infer.py, standalone port validated against the cluster predictions to fp16 precision; weights shipped in dataset casmi26-pool) |
local ens 0.519 | local ens 0.878 | hidden-set A/B of the transformer against probe-a 0.276: pushed 10-16 (folder C:\Temp\casmi26\kaggle_kernel_probe_xf); public LB 0.290 (submission 57275601, 10-16) vs probe-a 0.276: +0.014 on the hidden set, same direction as local but a fraction of the local gain (E +0.04, T +0.01). The hidden chemistry stays far below block T (0.52 local) for both fp models; the transformer helps a little, the deficit is mostly elsewhere (candidate generation / evidence channels), as the public-base layering showed |
| 10-16 | DreaMS deployment route: fp/export_dreams_ts.py traces encoder + head to TorchScript (needs the Lightning _jit_is_scripting flag and de-duplication of one shared tensor); fp/dreams_infer.py runs it with plain torch |
traced vs eager max diff 0 on new inputs; torch 2.2 (cluster) → 2.14 CPU (local) reproduces the hold-out probabilities to fp16 | 426 MB file; 65 ms / spectrum on 4 CPU threads → ~80 s for the 1,213 test spectra | the kernel can carry fine-tuned DreaMS without the dreams package. Probe kernel casmi26-fp-probe-dreams (3-way logit mean) prepared, waiting for dreams_full |
| 10-16 | dreams_full = DreaMS (116M, SSL-pretrained) fully fine-tuned on all 2.4M spectra, fp_v2 recipe (BCE + candidate-CE from epoch 1, K 31, τ 20; fp32, bs 64, lr 2e-5 / head 2e-4; 3 epochs, 7.4 h V100; job 11130565) | 0.504 [0.463, 0.543] (Δ vs fp_v2 −0.006 [−0.043, +0.031] n.s.) | 0.853 [0.838, 0.866] (Δ +0.014 [+0.002, +0.026]) | the strongest single model after fp_v2: matches it on T, beats it on E. cand-ce 0.85 → 0.51 → 0.36 over the 3 epochs (still falling: more epochs would help). 426 MB weights |
| 10-16 | ens_v2_dfull = logit mean fp_v2 + dreams_full (equal weights) | 0.544 [0.503, 0.581] (Δ vs fp_v2 +0.034 [+0.009, +0.059]; vs ens_v2_xfce +0.025 [+0.002, +0.048]) | 0.869 [0.856, 0.882] (Δ vs fp_v2 +0.031 [+0.022, +0.040]; vs ens_v2_xfce −0.009 [−0.018, −0.000]) | first model to beat fp_v2 significantly on block T (the honest NP block) and the new best on T; ens_v2_xfce keeps a sliver on E. The three models make different errors → 3-way ensemble (explorer/ens_eval3.sbatch, job 11141377) + TorchScript export of dreams_full queued |
| 10-16 | ensembles with dreams_full (logit mean; explorer/ens_eval3.sbatch): fp_v2+dfull 0.6/0.4; xfce+dfull 0.5/0.5; 3-way fp_v2/xfce/dfull 0.34/0.33/0.33, 0.4/0.4/0.2, 0.25/0.35/0.4 |
0.546 (Δ vs fp_v2 +0.036 [+0.013, +0.059]); 0.508; 3-way 0.535 (Δ +0.026 [+0.000, +0.051]), 0.526, 0.527 | 0.867 (Δ +0.029); 0.869; 3-way 0.875 (Δ +0.036 [+0.027, +0.046]), 0.878, 0.874 | fp_v2 + DreaMS is the T winner (the transformer adds nothing on T once DreaMS is in: xfce+dfull 0.508), the 3-way is the balanced pick (−0.008 T / +0.006 E vs the 2-way; vs ens_v2_xfce +0.017 T n.s. / −0.004 E n.s.). Both go to the hidden-set A/B |
| 10-16 | dreams_full TorchScript export + local check |
traced vs eager diff 0; cluster → local torch 2.14 CPU: probabilities match on 3 hold-out spectra to 1e-4 | 73 ms / spectrum (4 threads) | dreams_full_ts.pt (426 MB, md5 461ddfc2…) + fp/dreams_infer.py added to dataset casmi26-pool; kernels casmi26-fp-probe-dreams (3-way 0.33/0.33/0.34) and casmi26-fp-probe-dreams2 (fp_v2 0.6 + DreaMS 0.4) prepared for the hidden-set A/B vs probe-xf 0.290 |
| 10-16 | Kaggle probes pushed + submitted: casmi26-fp-probe-dreams (3-way 0.33/0.33/0.34; submission 57280081) and casmi26-fp-probe-dreams2 (fp_v2 0.6 + DreaMS 0.4; submission 57280082); both kernels 10.5 min, 400 rows, DreaMS loaded from TorchScript |
local 0.535 / 0.546 | local 0.875 / 0.867 | hidden-set A/B vs probe-xf 0.290 / probe-a 0.276; scores pending at write time (see next row) |
| 10-16 | dreams_full2 = continuation of dreams_full, 3 more epochs at lr 1e-5 / head 1e-4 (job 11141795, explorer/dreams_full2.sbatch; scores + ens_v2_dfull2 + 3-way + TorchScript export chained) |
– | – | cand-ce was still falling at epoch 3 (0.36); checking whether more epochs keep paying |
| 10-16 | hidden-set A/B results (public LB): probe-a (fp_v2 only) 0.276 → probe-xf (fp_v2+transformer) 0.290 → probe-dreams (3-way) 0.287 → probe-dreams2 (fp_v2 0.6 + DreaMS 0.4) 0.292 | local T: 0.510 / 0.519 / 0.535 / 0.546 | local E: 0.838 / 0.878 / 0.875 / 0.867 | the hidden set rewards the encoder upgrades by only +0.011…+0.016 over fp_v2 alone, with the three ensembles within 0.005 of each other (≈ noise at 400 molecules). Rank order on the hidden set follows block T (NP, timsTOF), not block E. Conclusion: the fp-prediction encoder is no longer where the hidden-set deficit lives; it sits in candidate generation / spectral-library evidence (public-base layering gets 0.440). Ship fp_v2 + DreaMS (0.6/0.4) as the fp channel in the recommended pipeline and move effort back to the candidate side |
| 10-16 | dreams_full2 = dreams_full + 3 more epochs at lr 1e-5 / head 1e-4 (7.4 h V100, job 11141795) | 0.485 (dreams_full 0.504); ens with fp_v2 0.530 (0.544); 3-way 0.529 (0.535) | 0.840 (0.853); ens 0.865 (0.869); 3-way 0.876 (0.875) | overfits: train cand-ce fell 0.36 → 0.24 but every hold-out number dropped. Three epochs at 2e-5 is the right budget for DreaMS; dreams_full stays the shipped encoder. (Chained TorchScript export failed on a GPU node with a cuda/cpu device mix: export_dreams_ts.py now forces CPU) |
10-17 — six creative approaches (user: "at least five distinct new highly creative approaches … go meta")
All measured on the honest hold-outs (block T = 250 NPs on timsTOF, block E = 1,000 enveda-180-only structures; class 2) with
paired-bootstrap deltas; GPU runs use the sweep protocol (600 k spectra, 8 epochs, peak-set transformer trunk) and are compared to
xf_ce600 (same protocol, one-hot candidate CE). Code: fp/prep_common.py (shared prep: hold-out split, analog removal, random and
hard decoy tables), fp/chan_eval.py (shared window evaluation, candidate tables, paired deltas), explorer/cre_*.sbatch.
| # | approach | what is new | script |
|---|---|---|---|
| 1 | two-tower contrastive retrieval | no fixed intermediate representation: a learned molecule encoder (MLP over the structure bits) and the spectrum trunk meet on a unit sphere; InfoNCE over the truth, 31 mass-window decoys and all in-batch candidates (same-structure columns masked); auxiliary bit head | fp/train_twotower.py, fp/score_twotower.py |
| 2 | forward model (structure → spectrum) | the reverse problem: an MLP predicts the fragment-m/z and neutral-loss distributions (0.1 Da bins) from the candidate's bits + acquisition meta; a candidate is scored by the likelihood of the observed spectrum under its predicted spectrum; trained on real spectra + candidate-set listwise CE | fp/train_forward.py, fp/score_forward.py |
| 3 | cross-attention verifier | substructure tokens (the candidate's on-bits) attend to the peak tokens through a transformer decoder → a match score; listwise over truth + 15 decoys, half of them hard (most Tanimoto-similar window candidates); reranks the top-50 of the fp ranker | fp/train_verifier.py, fp/score_verifier.py |
| 4 | Tanimoto-tempered soft targets | new loss: the candidate CE target is q_c ∝ exp((Tanimoto(c, truth) − 1)/τ) instead of one-hot, so near-isomers of the truth get partial credit | fp/train_fp_aux.py --soft-tau |
| 5 | formula-explained intensity | deterministic chemistry channel: a peak is explained by a candidate iff a valence-valid sub-formula ion (with a valence-valid complementary loss) matches it within 5 ppm; sqrt-intensity-weighted explained fraction, averaged over spectra | fp/formula_channel.py |
| 6 | meta-scorer and stacking/gating | learn the ranking function: per-bit evidence weights + bit priors (meta), then listwise stacking of all channels with query-dependent weights (meta²/meta³), cross-fit T ↔ E | fp/meta_scorer.py, fp/stack_channels.py |
| date | experiment | T | E | notes |
|---|---|---|---|---|
| 10-17 | window formula diversity (diagnostic) | 87 % of windows mix formulas, median 4, 71 % of candidates share the truth formula | 98 %, median 7, 50 % | the formula channel has room, especially on E |
| 10-17 | #5 formula channel v1 (unconstrained sub-formulas, 10 ppm) fused into fp_v2 (z-sum, α 0.25) | −0.008 [−0.015, −0.003] | +0.002 [+0.000, +0.004] | too permissive: almost every peak is explainable by every candidate |
| 10-17 | #5 formula channel v2 (valence-valid ion + complementary loss, 5 ppm), fp_v2 + α z(frac) | α 0.1: −0.001 n.s.; 0.25: −0.005 n.s. | α 0.25: +0.005 [+0.002, +0.008]; 0.1: +0.002; 0.5: +0.003 | real but tiny: the fingerprints already carry most formula information (element counts via MACCS / Morgan). Alone: T 0.118 / E 0.114 (isomers tie). Kept as a channel for the stacker |
| 10-17 | #6a meta-scorer, per-bit w, v (L2 1.0), cross-fit | fit on E → T: −0.008 n.s. | fit on T → E: −0.033 [−0.043, −0.023] | 8.5 k parameters from 250 / 1,000 queries overfit |
| 10-17 | #6a meta-scorer, 3 group weights + 3 group priors, cross-fit | fit on E → T: +0.014 [+0.004, +0.024] (weights r2 0.08 / r3 0.056 / maccs 0.085: morgan3 evidence down-weighted; priors +r2 +r3 −maccs) | fit on T → E: +0.000 n.s. | the listwise fit mostly learns a temperature (sum of 4,263 log-likelihoods is over-confident) and a mild re-weighting of the radius-3 bits |
| 10-17 | #6a meta-scorer, per-bit with strong L2 (20), 20 epochs, cross-fit | fit on E → T: −0.005 n.s. | fit on T → E: −0.017 [−0.026, −0.009] | still overfits → per-bit weights rejected; the group-level variant is the only usable one |
| 10-17 | #6b stacker (listwise, cross-fit) over fp_v2 + formula channel (frac, top5), linear / query-gated | −0.003 n.s. / −0.003 n.s. | +0.003 [+0.001, +0.005] / +0.003 [+0.001, +0.005] | learned weights fp 3.7 : frac 0.4 : top5 0.5 (E-fit); gating on query features adds nothing. The stacker reproduces the hand-tuned α, as it should; its value will be in combining the GPU channels |
| 10-17 | xf_ce600 = sweep-protocol reference for the GPU approaches (peak transformer, 600 k / 8 epochs, BCE + one-hot candidate CE from epoch 3; job 11161806) | 0.362 [0.325, 0.396] | 0.750 [0.733, 0.768] | class 2: T 0.369 / E 0.755. All GPU rows below are paired against this (same data, epochs, trunk) |
| 10-17 | #4 soft targets τ 0.1 (--soft-tau 0.1) |
0.351 (Δ −0.011 [−0.031, +0.010] n.s.) | 0.761 (Δ +0.011 [−0.002, +0.024] n.s.) | no significant change either way; train cand-ce ends 2.55 vs 2.5-ish for one-hot |
| 10-17 | #4 soft targets τ 0.2 | 0.303 (Δ −0.059 [−0.081, −0.037]) | 0.635 (Δ −0.115 [−0.131, −0.098]) | hurts badly: at τ 0.2 most of the decoy mass (Tanimoto 0.3–0.6 to the truth) gets credit and the model stops separating isomers. Verdict: rejected — partial credit for near-isomers is the wrong prior when the metric only rewards the exact tautomer-canonical key |
| 10-17 | #1 two-tower tt600, cosine alone (no bits) (job 11161818, 75 min V100) | 0.345 (Δ vs xf_ce600 −0.018 [−0.053, +0.020] n.s.) | 0.830 (Δ +0.079 [+0.063, +0.097]) | the learned molecule encoder beats the 4,263-bit log-likelihood by a wide margin on E with the same spectrum trunk and data; its own bit head is weak (T 0.301 / E 0.439: the InfoNCE gradient dominates the trunk) and z-fusing both ("all") is worse than cosine alone |
| 10-17 | #1 two-tower cosine z-fused into fp_v2 (α 0.5 / 1 / 2) | +0.000 n.s. / −0.021 n.s. / −0.058 [−0.090, −0.027] | +0.041 [+0.033, +0.049] / +0.042 / +0.040 | fp_v2 + cos0.5: T 0.510 / E 0.879 — the 600 k-spectra retrieval model adds as much to fp_v2 on E as the full-data DreaMS ensemble did (0.869) |
| 10-17 | #1 two-tower cosine z-fused into ens_v2_dfull (α 0.25 / 0.5 / 1) | −0.002 n.s. / −0.009 n.s. / −0.039 [−0.064, −0.014] | +0.014 [+0.009, +0.019] / +0.022 [+0.016, +0.028] / +0.024 [+0.016, +0.031] | ens + cos0.5: T 0.534 / E 0.891 (best E ever). The usual T / E split: the retrieval embedding is tuned to the Orbitrap-heavy training distribution and does not transfer to timsTOF NPs at all. Full-data version tt_full queued (cre_twotower_full.sbatch) |
| 10-17 | #2 forward model fwd600 alone (recon-nll 13.3 → 9.4, train cand-top1 0.93; job 11164411, 75 min) | 0.173 (Δ vs ens_v2_dfull −0.370) | 0.755 (Δ −0.114) | a structure → spectrum MLP that scores by Σ t·log p over 10,000 fragment / loss bins reaches 0.755 on E alone — half-way to the fp ranker — but 0.17 on T: the predicted spectra are instrument-specific |
| 10-17 | #2 forward model z-fused into ens_v2_dfull (α 0.25 / 0.5 / 1) | −0.032 [−0.049, −0.015] / −0.061 / −0.109 | +0.005 [+0.003, +0.008] / +0.009 / +0.011 [+0.006, +0.016] | into fp_v2: T −0.021 [−0.038, −0.005], E +0.019 [+0.013, +0.025] at α 1. Verdict: a weak, instrument-bound channel — it hurts T significantly at every α, so it can only enter through a gated stacker (and the hidden set tracks T) |
| 10-17 | #3 verifier ver600 (3.0 M params, listwise over truth + 15 half-hard decoys; job 11164412) | — | — | ~1 h / epoch at bs 48 (cross-attention over 200 peak tokens × up-to-L substructure tokens is the bottleneck): the 6 h job was cancelled at the time limit after epoch 5/8 (train listwise-ce 2.49 → 0.89, top1 0.15 → 0.68). The epoch-5 checkpoint is scored by cre_verifier_score.sbatch (job 11178022) and a 5-epoch annealed retrain (cre_verifier2.sbatch, tag ver600e5) is queued behind tt_full; the stack (cre_stack.sbatch, job 11178025) runs once the epoch-5 scores exist |
| 10-17 | #3 verifier ver600 epoch-5/8 checkpoint, reranking the top-50 of ens_v2_dfull: alone / fused α 0.5 / 1 / 2 (job 11178022) | alone 0.322 (Δ −0.221); fused −0.040 [−0.060, −0.019] / −0.069 / −0.116 | alone 0.754 (Δ −0.115); fused +0.018 [+0.011, +0.024] / +0.021 [+0.012, +0.029] / +0.013 | on fp_v2: T −0.027 [−0.047, −0.007] at α 0.5, E +0.040 [+0.030, +0.050] at α 1. Same picture as the forward model and the two-tower: a real E gain, a T loss. Verdict: not worth the 5-epoch retrain (cre_verifier2.sbatch left unsubmitted); the GPU slot goes to the full-data two-tower |
| 10-17 | #5 formula channel v2 on ens_v2_dfull (α 0.1 / 0.25 / 0.5, and log-count fusion) | +0.000 / +0.000 / −0.002, all n.s. | +0.002 [+0.001, +0.003] / +0.003 [+0.001, +0.005] / +0.002 | same tiny E-only gain as on fp_v2 |
| 10-17 | #6a group meta-scorer on ens_v2_dfull (3 evidence weights + 3 bit priors, listwise, cross-fit) | fit on E → T: +0.034 [+0.019, +0.050] (0.5435 → 0.578) | fit on T → E: +0.014 [+0.007, +0.021] (0.869 → 0.883) | the only approach that helps both blocks. E-fit: w morgan2 0.084 / morgan3 0.059 / maccs 0.107 (≈ temperature 0.08 with morgan3 down-weighted ×0.7 and MACCS up ×1.3), priors +0.036 / +0.074 / −0.063 per on-bit (a mild bonus for candidates with more radius-3 substructure bits, a penalty per MACCS on-bit). T-fit: w 0.122 / 0.120 / 0.188, v +0.079 / +0.319 / −0.222 — same signs. Six numbers, no data needed at inference → shipped as Kaggle probe casmi26-fp-probe-meta (E-fit weights normalised by mean w so the reranker features keep their scale) |
| 10-17 | #6b stacker over ens_v2_dfull + {tt600 cos, fwd600, formula frac, ver600} (job 11178025, cross-fit): linear / query-gated | fit on E → T: −0.062 [−0.086, −0.038] / −0.084 | fit on T → E: +0.012 [+0.007, +0.017] / +0.013 | E-fit weights fp 2.6 : cos 1.49 : fwd 0.39 : frac 0.28 : ver 1.05 — what is right for E (every channel in) is wrong for T. T-fit weights fp 2.17 : cos 0.56 : fwd −0.48 : frac 0.15 : ver 0.04 → +0.012 on E: the T-optimal mixture keeps only the two-tower (α ≈ 0.26) and subtracts the forward model. Gating on query features changes nothing |
| 10-17 | #6b stacker subsets: {cos, fwd} / {frac, ver} | −0.034 [−0.056, −0.011] / −0.042 [−0.065, −0.021] | +0.012 [+0.007, +0.017] / +0.003 [+0.001, +0.005] | the E-fit always over-trusts the new channels on T; a stacker fitted on 250 T queries is the honest way to weight them for the hidden set, and it says: fp + a little two-tower cosine, nothing else |
| 10-17 | Kaggle probes pushed (hidden-set A/B vs probe-dreams2 0.292): probe-tt (= probe-dreams2 + tt600 cosine z-fused α 0.5; local T 0.534 / E 0.891), probe-tt2 (α 0.25; T 0.542 / E 0.883), probe-meta (= probe-dreams2 + E-fit group meta-scorer; local T +0.034 / E +0.014 cross-fit) |
— | — | dataset casmi26-pool new version with tt600_spec.pt (36 MB spectrum tower), tt600_pool_emb.npy (373 MB float16 pool embeddings), fp/tt_infer.py (standalone port, cosine to the cluster embeddings 1.0000 ± fp16); kernels in C:\Temp\casmi26\kaggle_kernel_probe_{tt,tt2,meta}; submitted 10-17 23:49 / 23:50 EDT (2 of 5 daily submissions left after probe-meta) |
| 10-18 | hidden-set results of the three probes (public LB, vs probe-dreams2 0.292): probe-tt (tt600 cosine α 0.5) 0.263, probe-tt2 (α 0.25) 0.278, probe-meta (group meta-scorer) 0.265 | local: T 0.534 / 0.542 / 0.578 | local: E 0.891 / 0.883 / 0.883 | all three lose on the hidden set, by more than the ≈ 0.005 noise band, including the meta-scorer that gained on both honest blocks. The one thing the three share that the local evaluation does not have: the kernel feeds the modified fp_score into the fusion reranker (rerank3, trained on the plain fp_v2 log-likelihood), so fp_score / fp_z / fp_rank / fp_gap leave the reranker's training distribution. Tested locally with fp/rerank_shift_test.py (reranker CV, SHIFT = train on original / test on modified, RETRAIN = train and test on modified) — see the next row |
| 10-18 | reranker feature-shift test (fp/rerank_shift_test.py: rerank3 CV on the reranker tables; SHIFT = reranker trained on the plain fp_v2 score, test fold fed the modified score; RETRAIN = trained on the modified score). fp-only sanity: meta T +0.014 [+0.005, +0.025] / E +0.000; tt025 T +0.003 n.s. / E +0.031; tt05 T −0.002 / E +0.041 (reproduces chan_eval) |
under the reranker, T class 1: SHIFT meta −0.018 [−0.036, −0.002], tt025 −0.020 [−0.037, −0.004], tt05 −0.016 [−0.033, −0.001]; RETRAIN meta +0.003 n.s., tt025 +0.011 n.s., tt05 +0.003 n.s. T class 2: SHIFT meta +0.003 n.s., tt ≈ −0.004 n.s.; RETRAIN meta +0.001 n.s., tt025 +0.018 [−0.007, +0.043] n.s., tt05 +0.009 n.s. | E class 2 is saturated under the reranker (0.987) | (1) the hidden-set losses are reproduced only for the library-hit (class-1-like) molecules: a shifted fp_score / fp_gap / fp_rank confuses the trained reranker where it trusts the fp features against library evidence; (2) the meta-scorer's fp-only gain disappears under the reranker — the reranker (fp_z, fp_rank, fp_gap, non-linear) already learns the temperature / re-weighting; (3) the two-tower channel survives only as a retrained-reranker deployment, +0.01…+0.02 on T, not significant. Reranker retrained on fp_v2 + 0.25·sd·z(tt600 cos) over all rows (fp/train_rerank_tt.py → rerank3_tt025.txt; gain: lib_cos 0.43, fp_z 0.19, fp_gap 0.16) and shipped as probe casmi26-fp-probe-tt3 = probe-dreams2 + tt600 α 0.25 + that reranker |
| 10-18 | probe-tt3 hidden-set result (= probe-dreams2 + tt600 cosine α 0.25 + reranker retrained on that fused score, rerank3_tt025.txt; submission 57339541, 01:45 EDT) |
public LB 0.266 vs probe-dreams2 0.292 (probe-tt2, same α with the old reranker: 0.278) | local prediction from the RETRAIN arm of the feature-shift test: T class 2 +0.018 n.s., class 1 +0.011 n.s. | the retrained reranker does not rescue the channel — it is worse than the shifted deployment. So the feature shift explains at most part of the probe losses; the two-tower cosine itself degrades the hidden-set ranking by ≈ 0.02–0.03 even though it is neutral on the 250 honest T NPs and +0.02 on E. Verdict for the whole creative track on the hidden set: no fp-side channel from approaches 1–6 ships (two-tower ×3 probes, meta-scorer ×1; forward / verifier / soft targets rejected locally; formula ≈ 0). The 250-query T block cannot resolve the ±0.02 differences the LB shows (its 90 % CI is ≈ ±0.02 to ±0.03), so LB probes remain the arbiter and are capped at 5/day — 1 left for the 10-18 UTC day. Kernel of record stays probe-dreams2 (0.292) |
| 10-18 | #1 two-tower, full data tt_full (2,407,689 spectra / 272,483 structures, 8 epochs, 31.2 M params; job 11178770, 4.9 h on a V100; final in-batch top-1 0.92, scale 93) — cosine alone, and z-fused into fp_v2 / ens_v2_dfull (α 0.25 / 0.5 / 1) |
alone 0.369 (tt600: 0.344). Into fp_v2: +0.003 / +0.005 / −0.007, all n.s. Into ens: +0.005 n.s. / −0.001 n.s. / −0.020 n.s. | alone 0.832 (tt600: 0.830). Into fp_v2: +0.030 / +0.039 / +0.046 [+0.036, +0.056]. Into ens: +0.014 / +0.021 [+0.014, +0.029] / +0.021 | 4× the training data changes nothing: the retrieval channel saturates at the 600 k sweep (E 0.83 alone, +0.02 into the ensemble, T flat), so the two-tower's ceiling is a property of the representation, not of the data budget. Stackers on tt_full:cos, fwd600:fwd, formula_ens:frac (linear / gated, cross-fit): E-fit → T −0.027 [−0.049, −0.005] / −0.068; T-fit → E +0.011 / +0.011 — identical to the tt600 stack. Given probe-tt3 (above), not deployed; no submission spent. Closes the 10-17 creative track: nothing from approaches 1–6 reaches the kernel |