Brain2Qwerty v1 — MEG Brain-to-Text Reproduction

Upstream: github.com/facebookresearch/brain2qwerty · Paper: Brain-to-Text Decoding: A Non-invasive Approach via Typing (arXiv 2502.17480)

Why reproduce a brain-to-text model

Meta FAIR's Brain2Qwerty decodes the sentences a person types from non-invasive brain recordings — no surgery, just a MEG scanner reading magnetic fields off the scalp while they type memorized sentences on a QWERTY keyboard. The v1 system reports a character error rate (CER) of 32% with MEG.

A published number and public data is a standing invitation: can you actually land it? Reproduction is an underrated skill — it forces you to handle real scientific data, provision real GPUs, and be honest about the gap between "the paper says X" and "I measured X on my own run." This project is that exercise, done end to end and reported without spin.

What I set out to reproduce (and the number that actually matters)

Brain2Qwerty v1 has two stages:

1. A neural network — a convolutional encoder feeding a sentence-level Transformer (623M parameters) — that maps a MEG recording to per-keystroke character predictions.
2. A character-level n-gram language model that rescores those predictions with beam search.

These produce two different numbers, and conflating them is the classic reproduction trap:

• ≈0.38 CER — the neural network alone, greedy-decoded, no language model. The repository documents this as the expected MEG result.
• 0.32 CER — the full system, after language-model rescoring. This is the paper's headline (macro-averaged across the 19 participants; best subject 19%, worst 45%).

This write-up reproduces the first number. The language-model stage is a separate decoding step I have not yet run — so I claim the ~0.38, not the 0.32, and I say so plainly.

The result

Trained from scratch on the public SpanishBCBL MEG dataset and evaluated on the held-out test split (521 sentences across 19 participants):

The micro number lands directly on the documented ≈0.38 target. The macro number is ~1 point higher because it weights each participant equally, so the handful of harder, lower-sentence-count subjects pull the mean up — the same per-subject averaging the paper uses for its error bars.

The decoded text is legible

Even without the language model, the neural network recovers recognizable Spanish. A few held-out examples (true → model-decoded):

• el pedido merece las atenciones → el ilg co oereve las atenciones
• la fuerza atrae el electron primero → la fnerza teae tl elestranelai es
• los ninos mayores acaban la escuela → los hilos iaopredantan nsl eecieos

Whole words survive intact; the errors are the kind a language model is specifically designed to clean up — which is exactly why the LM stage buys the jump from ~0.38 to 0.32.

The engineering

The science is Meta's; the reproduction is an infrastructure problem.

Real, gated, large data. The SpanishBCBL MEG dataset is ~250GB of .fif recordings behind a Hugging Face access gate. I scripted the gated download (working around a flaky transfer backend that throttled to sub-MB/s by disabling it and falling back to plain HTTPS) and the exca-cached preprocessing.

Single-GPU cloud training. The pinned environment (torch 2.6 / CUDA 12.4) was incompatible with my local Blackwell GPU, so the faithful run went on a GCP L4 spot-then-on-demand instance. Provisioning was its own runbook: the base image shipped the wrong Python, the toolchain for building KenLM and psutil from source was missing, and a 1TB data disk needed careful mount/persistence handling across stop/start cycles. Training ran ~2.5 days (~16 min/epoch, ~200 epochs to plateau) under early stopping.

A fidelity-preserving OOM fix. At the end of epoch 0 the run crashed with a CUDA out-of-memory error — the validation batch is 2048 segments (a value tuned for the paper's 80GB A100s), which doesn't fit the L4's 22GB alongside a 623M model and its optimizer state. The tempting fix — shrink the validation batch — would have inflated CER, because the repo groups keystrokes into sentences within a batch, so a smaller batch splits more sentences across boundaries and feeds the Transformer partial sentences. Instead I enabled PyTorch's expandable-segments allocator, which let the 2048-segment batch fit with zero change to fidelity or code. Diagnosing why the naive fix was wrong mattered more than the fix itself.

Honest reporting. The reproduction report documents every deviation from paper defaults — the allocator setting, an HDD-vs-SSD data disk (I/O only), and the sentence-batch-boundary caveat — plus the full cost ($115 of GCP credit) and a teardown to a ~$0.15/month archived footprint. All artifacts (checkpoint, logs, per-sentence predictions) are archived to GCS.

What I'd do next

• Run the language-model stage to chase the paper's 0.32 — build a character-level ARPA n-gram and run the beam-search rescoring. There's a suspected bug to fix first: the rescorer appears to add a base-10 KenLM score to a natural-log neural term, which would mis-weight the language model.
• Contribute upstream — the sentence-batch-boundary behavior is a real fidelity caveat worth raising with the maintainers as a possible improvement (a sampler that packs whole sentences to a segment budget), framed as a suggestion, not a bug report.

Takeaway

The headline isn't a leaderboard score — it's the discipline. Reproducing a research model on real scientific data means handling gated multi-hundred-GB datasets, provisioning GPUs, debugging failures without quietly breaking the science, and then stating clearly which number you hit and which you didn't. A reproduction you can trust is worth more than a number you can't.