jingyaogong · Train An LLM Cheaply

MiniMind

Trains a language model from scratch on hardware you already own: a 64M dense model in roughly two hours on a single RTX 3090 for about three yuan. Decoder-only transformer with RMSNorm pre-normalisation, SwiGLU and RoPE with YaRN extrapolation, plus a MoE variant at 198M total and 64M active over 4 experts routed top-1. The full pipeline covers pretrain, SFT, LoRA, DPO, PPO and GRPO, and agentic tool-use training. Reported comparable to SmolLM2 and TinyLlama on C-Eval and CMMLU, weaker on English. Apache-2.0.