VantaLLM.

01 · the problem

During my research internship at SwiftSafe I was handed the kind of brief you don't get twice as a third-year: build the company an LLM stack. Not fine-tune something, not wrap an API. A framework that could train decoder-only models from random weights, on security-domain data, on whatever hardware we could get.

The honest version of why this was hard: I had never trained anything past a toy model. Most people at that point reach for a fine-tune. I decided that if I was going to own an LLM, I wanted to own all of it, which meant starting from an empty repo.

02 · the constraints

• One A100 with 20GB usable. Not a cluster. Every architecture decision had to survive that budget.
• The same codebase had to scale from a 567M dev model to a 125B bring-up profile by editing a JSON config. No forks, no "big model branch".
• Security-native from day one: the tokenizer had to round-trip SQLi payloads and CVE strings exactly, because a security model that shreds its own domain vocabulary is useless.
• Everything testable. If a component couldn't be verified against a reference implementation, it didn't ship.

03 · the architecture

  ┌────────────────────────────────────────────┐
  │ VantaLLM · decoder-only · 567,068,416 params│
  ├────────────────────────────────────────────┤
  │ tok(151,003) → emb → 12× [ RMSNorm         │
  │   → GQA + RoPE (shared lazy cache)         │
  │   → dense FFN ⇄ MoE 16e top-2              │
  │     (batched grouped-GEMM, cap=1.25) ]     │
  │ → RMSNorm → lm_head                        │
  ├────────────────────────────────────────────┤
  │ train: DDP / DeepSpeed ZeRO-2·3 · bf16     │
  │ serve: FastAPI · SSE · persona routing     │
  └────────────────────────────────────────────┘

Decoder-only, 12 layers, with alternating dense FFN and Mixture-of-Experts blocks: 16 experts, top-2 routing, capacity factor 1.25. Grouped-query attention with RoPE and a shared lazy cache keeps the KV memory sane on 20GB. RMSNorm and SwiGLU because they earn their place, not because a paper said so. The tokenizer is a custom 151,003-token BPE trained on security corpora.

Training runs through DDP or DeepSpeed ZeRO-2/3 in bf16, picked per hardware profile. Serving is FastAPI with SSE streaming and persona routing.

04 · what I owned

The architecture and every technical call in it: the MoE layout, the routing strategy, the tokenizer decision, the scaling-by-config design, the test plan. I used AI tooling in the loop the way modern teams do, and I reviewed and debugged every line that mattered, because when a training run hangs at 3am the tooling doesn't debug it for you. I can explain every layer because I had to debug every layer.

05 · failures & fixes

• ZeRO-3 param-gather hang

  A Python loop over 128 experts deadlocked DeepSpeed's parameter gathering. The fix replaced the loop with three batched tensor ops, a grouped-GEMM formulation. Verified equivalent to the naive loop within 5.22e-08. That debugging week taught me more about distributed training than any course.

• OOM crashes, repeatedly

  20GB is unforgiving at 567M params with optimizer states. Activation checkpointing choices, bf16 everywhere it's safe, and honest measurements of what each block actually costs. Posted the crashes on X as they happened.

• multi-node launches hanging at 0% GPU

  NAT and duplicate-IP setups made NCCL hang silently. The launcher now fails fast with a real error message instead. Paranoia as a design principle.

06 · results

• ▸ 30,000-step training run completed on a single A100 20GB
• ▸ param_count = 567,068,416, exact, not estimated
• ▸ 86/86 tests: 41 unit · 20 integration · 25 live serving, green in CI
• ▸ KV-cache decode bit-identical to no-cache decode
• ▸ tokenizer round-trips SQLi payloads and CVE strings exactly
• ▸ the build-in-public thread crossed 7,800 impressions