TL;DR. train.parquet is 3.03 GB on disk but only 6.45 GB in pyarrow (401.8 M peaks, mean 158 / median 42 peaks per spectrum, measured here, not assumed), and a flattened float32/float16 offset layout clipped to the top-100 peaks is 0.80 GB — small enough to live on any GPU. A 25 M-parameter peak-token transformer (DreaMS/MassSpecGym style, 61 tokens per spectrum) costs about 2.7×10^16 FLOPs per epoch: ~2 min on one H200, ~6 min on a Modal A100 (about $10 for 50 epochs), ~15 min on a Colab L4, ~35 min on one Kaggle T4; a 36 M-parameter binned-fingerprint MLP is I/O-bound at 10–60 s per epoch anywhere. No public write-up ablates enveda-180; the only public data point (a model trained on enveda-180 alone scoring 0.138 vs 0.107 random on natural products) and our own finding that enveda-180 supplies 1,153,785 of the 1,154,969 timsTOF spectra in train together say: do not drop it, re-weight it and separate its instrument signal from its chemistry signal — the ablation grid below (R0–R6) is written as exact runs.
1. Memory and I/O — measured on the real file
All numbers below were computed on C:\Temp\casmi26\data\train.parquet (3,033,286,496 bytes, downloaded 2026-10-07 via the Kaggle CLI) with pyarrow 25.0.1; the script and raw JSON are in the worker scratchpad.
File facts. 2,539,608 rows, 21 row groups of 131,072 rows, ZSTD. ms2_mzs and ms2_normalized_intensities are list<double>; they are 98 % of the file: m/z 1.20 GB compressed / 3.21 GB raw, intensities 1.78 GB compressed / 3.13 GB raw (intensities compress worse because m/z values carry only 4 decimals — median and max decimals = 4 in a 20,000-peak sample — while intensities are full double precision). All metadata columns together are < 60 MB; reading them takes 0.3 s.
Peak statistics (whole file, num_peaks column, verified equal to len(ms2_mzs) on 1,048,576 sampled rows).
| statistic | value |
|---|---|
| total peaks N | 401,806,233 |
| mean / median peaks per spectrum | 158.2 / 42 |
| p10 / p90 / p99 / max | 5 / 416 / 1,347 / 73,318 |
| share of spectra with > 60 / > 100 / > 128 / > 300 peaks | 42.4 % / 33.4 % / 29.2 % / 15.0 % |
| N after clipping to top-60 / top-100 | 94,445,825 / 132,580,562 |
| peaks with relative intensity < 0.01 / < 0.001 (sample) | 85.5 % / 66.5 % |
| peaks in m/z 50–1000 / < 50 / > 1000 (sample) | 95.2 % / 4.0 % / 0.8 % |
| precursors with m/z > 1000 | 0.86 % (max precursor_mz = 2,430,333 — a corrupt row, see Gotchas) |
Per-library mean/median peaks: enveda-180 246.5/138 (1,153,785 spectra), enveda-np-examples 256.6/141.5 (1,184), gnps 218.5/46 (220,849), mona 293.6/57 (92,416), pluskal_ms2 35.1/21 (527,581), riken 36.7/9 (347,171), massbank 26.0/11 (101,727), msdial 43.0/11, spectraverse 113.6/47, drug_plus 112.3/108, masaryk 11.1/9. The timsTOF sources are the peak-rich ones (they are merged multi-CE spectra with a long noise tail: two-thirds of all peaks are below 0.1 % of base peak), so a top-k truncation throws away almost no information but ~67 % of the bytes. For comparison, published libraries cap much lower: MassSpecGym discards spectra with > 300 peaks after noise removal (arXiv 2410.23326, App. 2.3) and DreaMS keeps n = 60 peaks for pretraining, 100 for fine-tuning (Bushuiev et al., Nat Biotechnol 2025, Methods).
In-RAM size formulas (S = 2,539,608 spectra, N = 401,806,233 peaks).
| layout | formula | GB |
|---|---|---|
pyarrow list<double> × 2 columns |
16 B·N + 2·4 B·(S+1) | 6.45 (measured 16.05 B/peak) |
pandas, one ndarray per cell (default to_pandas()) |
≈ 16 B·N + 2·(112 B + 8 B)·S | 6.94 (measured 17.27 B/peak) |
pandas / Python lists of floats (.tolist(), JSON-style loaders) |
2·(8 B ptr + 24 B float)·N | 25.7 — this is what OOMs 16 GB notebooks |
| flattened offset arrays, float32 m/z + float16 intensity, all peaks | 6 B·N + 8 B·(S+1) | 2.43 |
| same, clipped to top-100 / top-60 | 6 B·N_clip | 0.80 / 0.57 |
The flattened layout is the one to train from: mz.npy (float32), inten.npy (float16, store sqrt- or log-intensity so float16 keeps dynamic range — 66 % of raw values are < 1e-3), offsets.npy (int64, S+1), plus meta.parquet. A 4-decimal m/z in float32 is exact to ±0.00003 Da below 1,000 Da (24-bit mantissa), fine for 0.01 Da bins; do not use float16 for m/z (3 significant digits).
Pre-binned dense tensors over m/z 50–1000 (float16; uint8 log-quantised halves it):
| bin width | bins | bytes/spectrum | full train (dense) | mean occupancy |
|---|---|---|---|---|
| 1 Da | 950 | 1,900 | 4.83 GB | 158/950 ≈ 17 % |
| 0.1 Da | 9,500 | 19,000 | 48.3 GB | ≈ 1.7 % |
| 0.01 Da | 95,000 | 190,000 | 482.5 GB | ≈ 0.17 % |
Consequence: 1 Da dense fits any GPU; 0.1 Da dense fits the 128 GB RAM request on Explorer (and the 135 GB local box) as a memmap but is 48 GB of traffic per epoch; 0.01 Da must be sparse — and a sparse 0.01 Da representation (int32 bin + float16 value = 6 B/nonzero ≤ 2.4 GB) is just the peak list again. The fastest epoch for an MLP is therefore: keep the 0.8 GB top-100 peak list on the GPU and scatter_reduce (amax) into the bin vector per batch; nothing is read from disk after startup. For a peak-token transformer, pad per batch to the batch max (bucket by num_peaks so padding waste is < 10 %). WebDataset shards or LMDB only pay off when the dataset does not fit in RAM; here it does on every target except Kaggle (16 GB system RAM, where the 0.8 GB file still fits).
2. Tokenisation: what the published models actually do
| model | input representation | params | training hardware | training time | reported metric | source |
|---|---|---|---|---|---|---|
| MassFormer (Young, Röst, Wang) | molecular graph → Graphormer (molecule-side); spectrum output as 1,000-bin 1 Da vector up to m/z 1000 (repo also has 0.1 Da configs) | 56 M (48.3 M Graphormer + 7.8 M MLP, from repo demo log) | tested on Tesla T4 16 GB / Quadro RTX 6000 24 GB, ≥ 12 GB VRAM recommended (repo); GPU for paper runs not disclosed | 20 epochs, wall-clock not reported | NIST-Scaffold cosine 0.65 (WLN 0.56, fingerprint 0.59, CFM ≈ 0.50); CASMI 2016 avg rank 27.1 ± 5.3, top-5 0.61 | arXiv 2111.04824 (ar5iv), github.com/Roestlab/massformer README |
| MIST (Goldman et al., Nat Mach Intell 2023) | peaks replaced by SIRIUS/internal subformula labels, sinusoidal formula embeddings, pairwise neutral-loss features; "chemical formula transformer" → unfolded fingerprint | not stated in sources opened; config hidden 256, 8 heads, 2 peak-attention + 4 refinement layers (small, likely < 10 M — estimate, unverified) | 1 GPU (config gpus: 1); on MassSpecGym: 1× A100 + 2× EPYC 7452 |
on MassSpecGym 231 K spectra: 32 epochs ≈ 5 h (early-stop patience 20) | own paper: fingerprint better than CSI:FingerID on > 70 % of standards, retrieval equal/better on 66 %; on MassSpecGym retrieval hit@1 5.24 %, hit@20 28.21 % (formula bonus 14.64 / 59.15) | Nat Mach Intell s42256-023-00708-3 (abstract-level only — full text 429/paywalled today), repo configs/canopus_compare/canopus_fp_mist.yaml (batch 128, lr 7.7e-4, morgan4096, cosine loss, aug p = 0.5), MassSpecGym App. 3.2 |
| DreaMS (Bushuiev et al., Nat Biotechnol 2025) | each peak = 2-D continuous token; m/z → 6,000 Fourier features (1,000 low + 5,000 high frequencies) → 4-layer FFN; intensity linear; extra never-masked precursor token; top-60 peaks (pretrain) / top-100 (fine-tune), m/z 0–1,000 | 116 M (7 layers, 1,024-d, 8 heads) | 4× AMD MI250X (8 devices) or 8× NVIDIA A100, DDP | pre-trained 48 h on GeMS-A10 (24 M spectra); 30 % m/z masking (loss 0.8 mass + 0.2 retention order, c = 20,000 mass bins) | fluorine detection precision 0.91 / recall 0.57; fine-tuned fingerprint retrieval "on par with MIST" on the CANOPUS benchmark (8 K spectra); inference 1 M spectra/h on one A100 | PMC13090125 full text |
| MassSpecGym Fingerprint FFN | 1 Da bins, max m/z 1,005 (1,005-d) → MLP → 4,096-bit Morgan r2, cosine loss | small (hidden 16–1,024 grid), not stated | 4× MI250X (8 devices) DDP | best model 4 epochs = 6.5 h (231 K spectra — slow because of per-epoch validation over full candidate sets) | hit@1 1.47 %, hit@5 6.21 %, hit@20 19.23 % | arXiv 2410.23326 App. 3.2 |
| MassSpecGym DeepSets (+ Fourier) | set of (m/z, intensity) pairs; Fourier variant uses 6,000 sin/cos wavelengths on m/z (d1 = 0.8 d), intensity linear (d2) | small | same | 3 epochs = 10.5 h | DeepSets hit@1 0.37 % → + Fourier 2.54 %; hit@20 8.22 % → 20.00 % | same |
| MassSpecGym SMILES/SELFIES Transformer (de novo) | 2-D continuous peak tokens + precursor token with dummy intensity 1.1, linear projection; encoder–decoder, BPE on 4 M molecules | small (hidden 256/512, 4–8 enc layers) | same | 24–25 epochs = 1 h | top-1 accuracy 0.00; MCES@1 28.59 (SMILES) | same |
Three facts matter for us. (i) The Fourier m/z embedding is worth ~7× in hit@1 on an otherwise identical set model (0.37 → 2.54 %) — raw-float m/z is the single most expensive mistake a peak-token model can make. (ii) Every peak-token model caps peaks at 60–100; our data says the cap costs < 1 % of intensity mass. (iii) MIST's formula-labelled tokens beat raw peak tokens (5.24 vs 2.54 hit@1) but need a per-spectrum subformula decomposition (SIRIUS or MIST's internal labeller) — on 2.5 M spectra that pre-processing, not the GPU, is the budget item; the MassSpecGym team called MIST training "computationally costly" and ran only 5 hyperparameter combinations.
3. Throughput, wall-clock and dollars
Assumptions (all stated so the arithmetic can be checked).
- Transformer-S: 25 M params (d = 512, 8 layers, FFN 2,048, 6,000→512 Fourier projection, 4,096-bit fingerprint head), T = 61 tokens (top-60 + precursor). Train FLOPs/spectrum ≈ 6·P·T + 6·6000·512·T + 12·L·T²·d = 9.15e9 + 1.12e9 + 0.18e9 ≈ 1.05e10; epoch = 2.54e6 × 1.05e10 = 2.66e16 FLOPs.
- Transformer-L: 50 M params, T = 101 → ≈ 3.3e10/spectrum, 8.4e16 FLOPs/epoch.
- Fingerprint MLP: 9,500 (0.1 Da) → 2,048 → 2,048 → 2,048 → 4,096 = 36.3 M params → 6·P = 2.2e8 FLOPs/spectrum, 5.5e14/epoch; I/O-bound, not compute-bound.
- Dense tensor-core peaks used: H200 SXM 989 TFLOPS BF16 (NVIDIA quotes 1,979 "with sparsity"; Wikipedia Hopper table), A100 312 TFLOPS BF16 (Wikipedia Ampere table), L4 ≈ 121 TFLOPS FP16 (derived: NVIDIA L4 page states FP8 485 TFLOPS with sparsity and "half without sparsity"; FP16 is half of FP8 on Ada — the 121 figure itself is not on the page I opened), T4 65 TFLOPS FP16 (nvidia.com T4 page; no BF16 → fp16 AMP + loss scaling).
- MFU: 25 % for transformers (short sequences, small d — expect 20–35 %; measure it with a 200-step benchmark before trusting any row), 20 % on T4, 15 % for the MLP. DDP scaling efficiency 0.85 (4× H200), 0.8 (2× T4).
- Prices from modal.com/pricing (per-second, fetched 2026-10-07): H200 $0.001261/s = $4.54/h; H100 $3.95/h; A100-80 GB $2.50/h; A100-40 GB $0.000583/s = $2.10/h; L40S $1.95/h; L4 $0.80/h; T4 $0.59/h; B200 $6.25/h; CPU $0.0472/core-h; RAM $0.008/GiB-h; $30/month free on Starter. Note the brief's "~$1.4/GPU-h" is below any current Modal GPU price — treat it as stale.
| hardware | Transformer-S epoch | 50 ep | Transformer-L epoch | 50 ep | MLP epoch | 50 ep | $ for 50 ep (S / L / MLP) |
|---|---|---|---|---|---|---|---|
| 1× H200 (Explorer gyorilab / gpu) | 108 s | 1.5 h | 340 s | 4.7 h | 4 s compute, ~15 s I/O-bound | 12 min | $0 (cluster) |
| 4× H200 DDP (gyorilab node d4079) | 32 s | 26 min | 100 s | 1.4 h | ~10 s | 8 min | $0 |
| Modal A100-40 GB | 341 s | 4.7 h | 1,077 s | 15 h | 12 s compute, ~20 s | 17 min | ≈ $10 / $31 / $0.6 (+ ~$0.4/h CPU+RAM) |
| Modal H200 (for reference) | 108 s | 1.5 h | 340 s | 4.7 h | ~15 s | 12 min | ≈ $7 / $21 / $1 |
| Colab L4 (Pro+) | 880 s | 12.2 h | 46 min | 39 h | 30 s compute, ~30 s | 25 min | compute units (rate unverified — Colab pricing page is a JS shell) |
| Kaggle 1× T4 | 2,046 s | 28 h | 105 min | 87 h | 56 s | 47 min | free quota |
| Kaggle 2× T4 DDP | 1,280 s | 18 h | 65 min | 54 h | ~40 s | 35 min | free quota |
Memory check for Transformer-S at batch 1,024: activations ≈ 1,024 × 61 × 512 × 8 layers × ~34 B/element ≈ 8.7 GB — fits a 16 GB T4 at batch 512 and an L4 at 1,024; H200 can take 8,192 with no loss of MFU (use a LR warm-up and sqrt scaling).
How to split the work.
- Explorer
gyorilab(one node, 4× H200, 96 cores, 773 GB RAM, default 7 d, max 30 d, PriorityTier 10, group-restricted) is where every transformer run and the whole ablation grid of §4 goes: the full grid (7 runs × 50 epochs of Transformer-S) is ~10 h on one H200, or run 4 jobs concurrently with--gres=gpu:1each. - Explorer
gpupartition (max 8 h, default 4 h, 105 GPUs incl. 4 nodes × 8 H200, 512 GB RAM, billing weight GPU = 100 CPU) for anything when gyorilab is busy — must checkpoint and chain (template below). - Modal A100-40 GB for embarrassingly parallel sweeps that need to finish tonight (10 concurrent GPUs on Starter): ~$10 per 50-epoch Transformer-S, so a 20-run sweep is ~$200 — set the budget guard.
- Colab L4: interactive debugging, attention/embedding visualisation, nothing > 2 h.
- Kaggle T4×2: only the inference notebook (the submission must run there offline) and MLP training if a Kaggle-native retrain is required; never 50-epoch transformer training (18 h vs session limits).
- Local box (32 cores, 135 GB RAM): parquet → flattened arrays conversion, candidate-pool building, scoring. Do not train here.
4. enveda-180 ablation
What is public. I opened seven CASMI-2026 GitHub write-ups (corentinlaudicina, kabir0774, billylu24, OfficialBhattacharya, vicky0718, AmanDubal — mxm-1x and AmanDubal README fetches 404'd). None reports a train-with vs train-without enveda-180 ablation. The single relevant number is in corentinlaudicina/enveda-casmi-2026 README: "Baseline MLP on natural products (enveda-np-examples, train+COCONUT pool) | 0.138 vs 0.107 random: a model trained on enveda-180 alone barely transfers to natural products" (their MLP: 100 strongest peaks, 0.1 Da / 10,000 bins, 2,048-bit Morgan, BCE + 32 same-mass-decoy ranking loss; public LB 0.112 with COCONUT pool, 0.079 with PubChem pool). Kaggle discussion threads could not be read (JS shell; web-search budget for this session was exhausted after the first query), so "nothing public" is a statement about GitHub mirrors, not the Kaggle forum. Verdict: no public with/without ablation exists; design it.
What the file says that changes the design. Instrument type is timsTOF for 1,154,969 training rows and 1,153,785 of them are enveda-180; the other 1,184 are enveda-np-examples (250 molecules). Dropping enveda-180 therefore removes 99.9 % of the hidden-test instrument domain from training. enveda-180 also carries adduct chemistry the public libraries lack: [2M+Na]+ 188,158, [2M+H]+ 89,487, [2M−H]− 52,933, [M+CH2O2−H]− 24,556 spectra. Its 182,941 InChIKey14 overlap the other sources by only 2,288 molecules, so it is also structurally disjoint from everything else. A plain "remove it" ablation confounds chemistry-domain with instrument-domain.
Validation set (one for all runs). NP-holdout: all 250 enveda-np-examples molecules (1,184 spectra, timsTOF — the only in-domain data we have) + 1,750 further InChIKey14 drawn from non-enveda-180 sources whose InChIKey14 is in COCONUT, sampled stratified by precursor m/z decile; every spectrum of a held-out InChIKey14 is removed from training, whatever its source. Candidate pool = COCONUT (Oct 2026 lite, local copy) within ±10 ppm of the neutral mass implied by the adduct. Metrics: MRR@25 (primary), recall@25, and the two slices "timsTOF NP (n = 250)" and "other-instrument NP (n = 1,750)". Seed-triplicate every run; the 0.03 differences that matter here need ±0.01 CIs.
| run | training data | mechanism | reads |
|---|---|---|---|
| R0 | all 11 sources, uniform | baseline | reference |
| R1 | all minus enveda-180 | hard ablation | does synthetic chemistry hurt or does timsTOF signal help? (confounded — see R3/R6) |
| R2 | all, per-sample weight w = 0.25 on enveda-180 (its share drops from 45 % to ~17 % of gradient mass) | importance re-weighting | monotonic response → sweep w ∈ {0.1, 0.25, 0.5, 1} |
| R3 | stage 1: all sources, 40 epochs; stage 2: non-enveda-180 sources + enveda-np-examples-like NP spectra only, 5 epochs at 0.1× LR, EMA weights | pretrain-then-finetune | best cheap fix if R1 < R0 < R3 |
| R4 | all, plus a 2-way domain head (enveda-180 vs rest) on the pooled embedding through a gradient-reversal layer, λ = 0.1 | domain-adversarial (DANN) | checks whether removing "which library" information from the embedding costs or gains MRR |
| R5 | enveda-180 only | lower bound; replicates the public 0.138 datapoint under our metric | calibration against corentinlaudicina |
| R6 | enveda-180 used only for masked-peak self-supervised pretraining (DreaMS objective, 30 % masking, 3 epochs over 1.15 M spectra ≈ 20 min on H200), then supervised fine-tune on non-enveda-180 sources | separates instrument prior from chemistry labels | if R6 ≈ R0 > R1, the value of enveda-180 is the instrument/adduct prior, not its structures |
Cost: 7 runs × 3 seeds × ~1.5 h (Transformer-S, 1× H200) ≈ 32 GPU-h ≈ 1.3 days on the 4-GPU node; the MLP version of the grid is < 2 h total and should run first to decide whether the transformer grid is worth it.
Decision rule. If R1 − R0 > +0.01 on the "other-instrument NP" slice but < 0 on the timsTOF slice, enveda-180 is a biased filler for chemistry and a necessary one for instrument — ship R3 or R6. If R2 at w = 0.25 ≥ R0 on both slices, ship R2 (simplest). If R4 beats everything on the timsTOF slice, the model was using library identity as a shortcut.
Forced analogy: jurisprudence of eyewitness line-ups (filler selection)
The analogy, precisely. In a line-up the witness has given a description; fillers must match that description so that the suspect does not stand out for reasons unrelated to memory. Here the hidden test's "description" is: natural product or NP-like, measured on a timsTOF with one of seven adducts. Our candidate pool (COCONUT ± 10 ppm) is the line-up; the true structure is the suspect; the model is the witness. enveda-180 is a set of 182,941 fillers that match half the description — instrument and adduct, yes; chemistry, no (0.03 % COCONUT, 2,288-molecule overlap with the rest of train). A witness trained to tell apart synthetic-looking fillers may at test time "pick out" the candidate that looks least like the training distribution, i.e. the most NP-like one, regardless of the spectrum — or, conversely, push NP-typical substructures (glycosides, polyketide chains) toward low fingerprint probabilities because it never saw them lit up.
Testable derivation — the synthetic-preference index (SPI). For every NP-holdout spectrum build a mixed line-up: the COCONUT candidates within ±10 ppm plus an equal number of mass-matched synthetic decoys drawn from enveda-180 structures held out of training (they exist for every mass window; enveda-180 spans the same precursor range). Rank the mixed pool with model M. SPI(M) = mean rank-percentile of the synthetic decoys (0.5 = unbiased; < 0.5 = the model promotes synthetic-looking structures; > 0.5 = it demotes them). Compute for R0, R1, R2, R6. A second, cheaper probe: predicted-fingerprint drift Δ = mean cos(f̂, f_true) on NP-holdout minus the same on a held-out enveda-180 slice, for each run; if Δ(R0) ≪ Δ(R1) the fingerprint head has specialised to synthetic bit patterns. Both are one evaluation pass each (~2 min on an H200) and give a number before any leaderboard submission.
Does the analogy hold? Partially. It holds for the evaluation side: a biased line-up (a pool whose fillers are systematically unlike the suspect) makes any ranking uninformative, and that is exactly why the public write-ups found PubChem pools (90 M fillers that do not match the description) score 0.079 vs COCONUT 0.112 — filler selection is the candidate-pool question. It breaks on the training side: fillers never train the witness; enveda-180 does. The closer legal concept is mugshot-exposure / "unconscious transference": a witness who browsed thousands of faces before the line-up tends to pick faces that feel familiar for the wrong reason. The practical lesson survives the break: measure the witness's prior (SPI) separately from the line-up's fairness (pool recall@25), because the two failures look identical on MRR.
Gotchas / what could fool us
num_peaksmax = 73,318 and precursor_mz max = 2,430,333. Row-level garbage exists (profile-mode spectra, unit errors). Filter: precursor_mz ≤ 1,000 (drops 0.86 %), num_peaks ≤ 2,000, no peak > precursor + 3 Da with > 20 % intensity (MassSpecGym's rule). Otherwise a 73 K-peak row blows up a padded batch.- Two-thirds of peaks are noise-floor (< 0.1 % base peak). A model with raw-intensity inputs effectively sees ~50 meaningful tokens; a binned MLP that does not sqrt/log-transform intensities sees a one-hot base peak. Use sqrt or log1p(1e4·I).
- Library-identity shortcut. Peak count, intensity floor and adduct distribution identify enveda-180 almost perfectly (median 138 peaks vs 9–21 in riken/pluskal/massbank). A model can learn "timsTOF-looking ⇒ synthetic chemistry prior" — this is exactly what R4 tests and what the visible (decoy) test set rewards.
- The visible test set is verbatim enveda-180 rows, so any local score on it is a memorisation score; the NP-holdout above is the only honest local number.
- Dimer adducts. 24 % of enveda-180 spectra are [2M+·]± adducts; if the hidden test's seven adducts exclude dimers (unverified here), those 330 K spectra contribute nothing but a wrong mass prior — consider dropping them before the ablation, or the R1/R2 effect will be partly a dimer effect.
- Padding waste and MFU. With median 42 and p90 416 peaks, naive padding to the batch max wastes > 70 % of compute; bucket by length or clip to 100. The throughput table assumes clipping.
- MFU is assumed, not measured. Every wall-clock in §3 scales linearly with the 25 % guess; run a 200-step benchmark on each target and rewrite the table — expect T4 to be worse than modelled (no BF16, 320 GB/s).
- Modal price drift. The brief's $1.4/GPU-h does not match the page today ($2.10 for A100-40 GB); budgets should be set from the fetched per-second rates.
- Explorer
gpupartition is 8 h max; a 50-epoch Transformer-L (4.7 h) fits, but a Transformer-S hyperparameter sweep does not unless chained. gyorilab is group-restricted (AllowGroups=rc,gyorilab) — confirm membership withsacctmgr show assoc user=shenoy.ambefore relying on it. - Proxy.
http_proxy/https_proxy=http://10.99.0.130:3128is already exported on explorer-01's login shell; compute nodes need it exported inside the job forpip, HF, Kaggle downloads. Nothing in the rc-docs mentions it; it comes from the environment and the brief. - CRLF. Any
.shedited on Windows and copied up fails withsbatch: error: Batch script contains DOS line breaks— runsed -i 's/\r$//' job.sh(dos2unix is not guaranteed on the login node). - rsync exists on both ends (
/usr/bin/rsyncon explorer-01; rtools45 rsync locally) but the local one is a Cygwin build whose path mapping and ssh invocation differ from Git Bash'sscp; the "scp not rsync" rule is a Windows-side problem. Usescporrclone/Globus viaxfer.discovery.neu.edu(the documented transfer node), notexplorer-01. - /scratch is 91 % full (1.4 of 1.5 PB) and not backed up; stage 1 GB of flattened arrays, not 48 GB of binned tensors, and copy checkpoints home.
Explorer sbatch template (copy-paste; strip CR before submitting)
#!/bin/bash
#SBATCH --job-name=casmi_tfS
#SBATCH --partition=gyorilab,gpu # gyorilab: 4xH200, 30 d max; gpu: 8 h max — whichever starts first
#SBATCH --gres=gpu:1
#SBATCH --cpus-per-task=12
#SBATCH --mem=128G
#SBATCH --time=07:50:00 # under the gpu partition's 8:00:00 MaxTime; harmless on gyorilab
#SBATCH --array=1-6%1 # chain: 6 sequential 8-h slots = 48 h budget, resume from checkpoint
#SBATCH --signal=B:USR1@600 # 10 min before the limit, let the script checkpoint
#SBATCH --output=/scratch/shenoy.am/casmi26/logs/%x_%A_%a.out
#SBATCH --error=/scratch/shenoy.am/casmi26/logs/%x_%A_%a.err
set -euo pipefail
export http_proxy=http://10.99.0.130:3128 https_proxy=http://10.99.0.130:3128 ftp_proxy=http://10.99.0.130:3128
export HF_HOME=/scratch/shenoy.am/hf TOKENIZERS_PARALLELISM=false OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK
module load explorer anaconda3/2024.06 cuda/12.8.0 # cuda modules present: 12.1.1 12.3.0 12.8.0 13.2.0
source activate casmi26
RUN=/scratch/shenoy.am/casmi26/runs/${SLURM_JOB_NAME}_${SLURM_ARRAY_JOB_ID}
mkdir -p "$RUN"
CKPT=$(ls -t "$RUN"/ckpt_*.pt 2>/dev/null | head -1 || true)
trap 'echo "USR1: checkpointing"; kill -USR1 $PID; wait $PID' USR1 # train.py saves on SIGUSR1 and exits 0
python -u train.py --data /scratch/shenoy.am/casmi26/flat_top100 --out "$RUN" \
--epochs 50 --batch 2048 --bf16 ${CKPT:+--resume "$CKPT"} &
PID=$!; wait $PID; RC=$?
# done? cancel the remaining array tasks so the chain stops
if [ -f "$RUN/DONE" ]; then scancel --state=PENDING "$SLURM_ARRAY_JOB_ID"; fi
exit $RC
Rules of use: the script must write ckpt_<epoch>.pt (model + optimizer + scheduler + RNG + epoch) every ~15 min (rc-docs recommends 10–15 min) and on SIGUSR1, and touch DONE after the last epoch. For 4-GPU DDP on gyorilab change to --gres=gpu:4 --ntasks-per-node=4 --cpus-per-task=12 and launch with srun torchrun --nproc_per_node=4 .... Submit with sed -i 's/\r$//' job.sh && sbatch job.sh; stage data with scp -r C:/Temp/casmi26/flat_top100 shenoy.am@xfer.discovery.neu.edu:/scratch/shenoy.am/casmi26/. Monitor with squeue -u shenoy.am and sacct -j <id> --format=JobID,Elapsed,MaxRSS,State.
Implications for the challenge
- Convert train.parquet once to the flattened top-100 layout (0.80 GB) and put that, not parquet, on every machine; it removes the data loader from the critical path on all five targets.
- Use Fourier m/z features (6,000-d, DreaMS/MassSpecGym recipe) in any peak-token model; the published 7× hit@1 gain is the cheapest known win.
- Run the MLP version of the R0–R6 grid first (< 2 h on one H200) and read the NP-holdout MRR@25 on the timsTOF and other-instrument slices before spending 32 GPU-h on the transformer grid.
- Do not simply drop enveda-180: it is 99.9 % of the timsTOF spectra in train. Prefer R2 (w = 0.25), R3 (pretrain-then-finetune) or R6 (self-supervised use only) and let the data choose.
- Report SPI and recall@25 beside MRR for every model so "biased witness" and "biased line-up" failures are told apart.
- Keep all 50-epoch transformer training on Explorer (gyorilab first,
gpuwith the chained-array template second); spend Modal (~$10 per Transformer-S run) only on parallel sweeps with a hard budget; Kaggle T4s are for the inference notebook. - Filter the 0.86 % precursor > 1,000 Da rows and cap num_peaks before any batch padding; the 73 K-peak row will OOM a padded batch.
- Benchmark MFU on each target for 200 steps and overwrite the §3 table with measured epoch times before planning the final week.
Open questions
- Kaggle code-competition limits (max notebook runtime, GPU allowed, RAM) are unverified — the Kaggle pages are JS shells; a GitHub mirror of the rules is needed.
- Colab L4 compute-unit burn rate and Pro+ monthly allocation: unverified (pricing page not fetchable).
- Are the hidden test's seven adducts free of dimers? If so, 24 % of enveda-180 is deadweight and should be excluded before the ablation.
- Is
shenoy.amin thegyorilabSlurm group? (AllowGroups=rc,gyorilab.) Onesacctmgrcall settles it. - MIST parameter count and its own paper's hardware/time: not retrievable today (Nature paywall, bioRxiv 429); the A100/5 h figure is MassSpecGym's reproduction, not Goldman's.
- True MFU of a 25 M-param, 61-token model on H200/T4 — only a benchmark will tell; the table could be off by 2× either way.
Sources
- MassSpecGym paper, arXiv 2410.23326v3 (PDF, appendices 2.3, 3.1–3.3, Tables 2–6): https://arxiv.org/pdf/2410.23326
- MassSpecGym README: https://raw.githubusercontent.com/pluskal-lab/MassSpecGym/main/README.md ; HF dataset card https://huggingface.co/datasets/roman-bushuiev/MassSpecGym
- DreaMS, Bushuiev et al., Nat Biotechnol 2025, doi:10.1038/s41587-025-02663-3, full text via Europe PMC PMC13090125: https://www.ebi.ac.uk/europepmc/webservices/rest/PMC13090125/fullTextXML ; README https://raw.githubusercontent.com/pluskal-lab/DreaMS/main/README.md ; model code https://raw.githubusercontent.com/pluskal-lab/DreaMS/main/dreams/models/dreams/dreams.py
- MIST: Goldman et al., Nat Mach Intell 2023, doi:10.1038/s42256-023-00708-3 (abstract-level access only); repo README (master, main_v2) and config https://raw.githubusercontent.com/samgoldman97/mist/main_v2/configs/canopus_compare/canopus_fp_mist.yaml , canopus_fp_ffn.yaml
- MassFormer: arXiv 2111.04824 (ar5iv HTML https://ar5iv.labs.arxiv.org/html/2111.04824 ); repo README https://raw.githubusercontent.com/Roestlab/massformer/master/README.md
- Modal pricing: https://modal.com/pricing (fetched 2026-10-07)
- NVIDIA pages: H200 https://www.nvidia.com/en-us/data-center/h200/ ; H100 https://www.nvidia.com/en-us/data-center/h100/ ; L4 https://www.nvidia.com/en-gb/data-center/l4/ ; T4 https://www.nvidia.com/en-us/data-center/tesla-t4/ ; A100 https://www.nvidia.com/en-us/data-center/a100/ ; Wikipedia Hopper and Ampere microarchitecture tables (dense BF16 figures)
- NEU RC docs: https://rc-docs.northeastern.edu/en/latest/gpus/quickstart-h200.html , gpus/gpujobsubmission.html , gpus/multigpu-partition-access.html , best-practices/checkpointing.html , datamanagement/transferringdata.html , runningjobs/interactiveandbatch.html , runningjobs/slurmarray.html
- Explorer live queries (2026-10-07,
ssh explorer):sinfo -p gyorilab,gpu,...,scontrol show partition gyorilab|gpu,env | grep proxy,module avail cuda,df -h /scratch - CASMI-2026 public write-ups opened: https://github.com/corentinlaudicina/enveda-casmi-2026 (README), https://github.com/kabir0774/Enveda-Casmi-2026 , https://github.com/billylu24/Enveda-CASMI-2026---Molecule-ID-From-Mass-Spectra , https://github.com/OfficialBhattacharya/kaggle_CASMI_2026 , https://github.com/vicky0718/Enveda-CASMI2026 (mxm-1x and AmanDubal READMEs returned 404)
- Local measurements:
C:\Temp\casmi26\data\train.parquet(Kaggle CLI, 2026-10-07), scriptsstats.py,stats2.pyin the worker scratchpad; outputs reproduced in §1 and §4.