A single-file, nanoGPT-style language model trainer: train_gpt.py (~1k lines, internally split into logical sections). Author and version are defined at the top of the script as __author__ and __version__, or run python train_gpt.py --version.
- Configuration:
TrainConfigdefaults → optionalconfigs/default.yaml→ CLI overrides win; merged config is written toout_dir/config_resolved.yamlat startup. - Environment: Staged checks before training (Python version, required packages,
devicevs CUDA/MPS, data paths, optionaltensorboard/wandb/tiktoken). - Data: supports raw UTF-8 text mode and pretokenized binary mode. Text mode reads a single
.txtfile or directory of.txtfiles; bin mode openstrain.bin/val.binwithnp.memmapso host memory stays roughly constant even for very large corpora. - Tokenizer: Character-level by default; also supports
--tokenizer tiktokenand externaltokenizer.jsonfiles. Char tokenization in bin mode can reuse a saved vocab via--vocab_file. - Model: Causal Transformer (nanoGPT-style blocks),
dropout/bias, init, and parameter counts printed at startup. - Training:
AdamWby default, optionalbitsandbytesAdamW8bitvia--optimizer_name adamw8bit, warmup + cosine LR schedule down tomin_lr, gradient clipping, configurable gradient accumulation via--accumulation_steps, optional CUDA AMP; L2 gradient norm (grad_norm) logged each eval. - Memory saving: optional gradient checkpointing (
--gradient_checkpointing) reduces activation memory during training by recomputing each Transformer block in backward; optional 8-bit optimizer reduces optimizer-state memory on supported CUDA setups. - Evaluation & logging:
val_loss/ perplexity on a validation split everyeval_interval;log_backendisnone|tensorboard|wandb. - Console progress line (each eval): fixed-width
Step …/max_iters | percentage, ASCII bar (e.g.[====>---------------]),train_loss/val_loss/grad_norm/ppl/lr;d_tr/d_vaare loss deltas vs the previous eval (n/aon the first line; improvement is negative);s/itapproximates seconds per training step from wall time between evals. TensorBoard/WandB also gettrain/loss_delta,val/loss_delta,train/secs_per_iter. The finalEval historyblock uses the same column layout without the progress bar to keep lines shorter. - Checkpoints:
checkpoint_interval→latest.pt; improved validation →best.pt; includes model, optimizer, scaler, RNG state (including NumPy);torch.load(..., weights_only=False)for full checkpoints on PyTorch 2.6+. - Resume:
--resumeloadsout_dir/latest.ptby default, or--checkpoint path. - Early stopping: if
early_stop_patience > 0, stop after that many evals without validation improvement. - Sampling: periodic text samples at
sample_interval; optional append toout_dir/samples.txt. - Summary: after training, prints
=== Eval history (every eval_interval) ===. - Lightweight help:
python train_gpt.py -h/--versionexit before importing PyTorch (works without a full ML env).
- Python 3.9+ (enforced in-script).
- See
requirements.txt. Install PyTorch for your CPU or CUDA stack from the official site.
python3 -m venv .venv
source .venv/bin/activate # Windows: .venv\Scripts\activate
pip install -r requirements.txt
# For GPU, follow PyTorch docs for the CUDA build you needOptional extras:
pip install tiktoken tokenizers tensorboard wandb bitsandbytesNotes:
bitsandbytesAdamW8bitis typically for CUDA training. On unsupported devices likecpuormps, the script can either fall back to standardAdamWor fail early, depending on--optimizer_fallback.- Gradient checkpointing saves memory but usually increases compute time because activations are recomputed during backward.
-
Prepare either raw text data or pretokenized binary data.
mkdir -p data # text mode: put *.txt under data/ or point --data_dir at one file # bin mode: create data/train.bin and data/val.bin with uint16/int32/int64 token ids
-
Run (example overrides):
python train_gpt.py \ --config configs/default.yaml \ --data_dir data \ --out_dir runs/exp01 \ --device auto \ --num_threads 8
-
For large corpora, switch to memmap-backed bin mode:
python train_gpt.py \ --data_dir data \ --data_format bin \ --tokenizer tokenizer.json \ --batch_size 2 \ --accumulation_steps 32 \ --gradient_checkpointing \ --amp
-
Help and version (no PyTorch required):
python train_gpt.py --help python train_gpt.py --version
| Area | Examples | Meaning |
|---|---|---|
| Device | --device auto|cpu|cuda|mps |
auto: CUDA → MPS → CPU |
| Training | --batch_size --lr --max_iters --weight_decay |
Batch size, LR, total steps, weight decay |
| Effective batch | --accumulation_steps |
Number of microbatches per optimizer update |
| Schedule | --warmup_iters --min_lr |
Warmup steps and cosine floor LR |
| Model | --n_layer --n_head --n_embd --block_size |
Depth, heads, width, context length |
| Data | --data_dir --data_format text|bin --train_bin --val_bin --token_dtype |
Raw text or memmap-backed token files |
| Tokenizer | --tokenizer char|tiktoken|path/to/tokenizer.json --vocab_file |
Tokenizer backend and optional saved char vocab |
| I/O | --out_dir --eval_interval --checkpoint_interval |
Output dir and log/checkpoint cadence |
| Resume | --resume --checkpoint path/to.pt |
Continue from checkpoint |
| Logging | --log_backend none|tensorboard|wandb |
Metrics backend |
| Optimizer | --optimizer_name adamw|adamw8bit --optimizer_fallback fallback|strict |
Choose optimizer backend and unsupported-env behavior |
| Memory | --gradient_checkpointing |
Recompute Transformer blocks in backward to reduce activation memory |
| Other | --amp --early_stop_patience N --grad_clip |
AMP, early stop, clip norm |
The full list is in python train_gpt.py --help. YAML keys match TrainConfig field names (snake_case).
| Path | Description |
|---|---|
config_resolved.yaml |
Final merged configuration |
startup_log.txt |
Startup report (same as stdout) |
latest.pt / best.pt |
Latest / best validation checkpoint |
samples.txt |
If sampling is on and append_samples_to_file |
tb/ |
TensorBoard logs when log_backend=tensorboard |
The startup report now also shows the requested optimizer, resolved optimizer after fallback, optimizer warning text, and whether gradient checkpointing is active.
Use the legacy workflow when corpora are small enough to tokenize in memory:
python train_gpt.py \
--data_dir data \
--data_format text \
--tokenizer chartrain_ratio and val_ratio are only used in this mode.
Use this for large datasets or limited RAM. The script opens train.bin and val.bin lazily with np.memmap.
python train_gpt.py \
--data_dir data \
--data_format bin \
--token_dtype uint16 \
--tokenizer tokenizer.jsonIf you prefer explicit paths:
python train_gpt.py \
--data_format bin \
--train_bin /path/to/train.bin \
--val_bin /path/to/val.bin \
--tokenizer tiktokenFor char tokenization in bin mode, pass a saved vocab file:
python train_gpt.py \
--data_format bin \
--tokenizer char \
--vocab_file char_vocab.txtchar_vocab.txt should contain the character set in token-id order. A JSON file with a top-level chars list is also accepted.
Laptop CUDA example with aggressive memory savings:
python train_gpt.py \
--data_dir ./dataset \
--data_format bin \
--tokenizer ./tokenizer_v1/tokenizer.json \
--n_layer 24 \
--n_embd 2048 \
--batch_size 2 \
--accumulation_steps 32 \
--gradient_checkpointing \
--optimizer_name adamw8bit \
--ampCPU compatibility example:
python train_gpt.py \
--device cpu \
--data_dir ./dataset \
--data_format text \
--n_layer 6 \
--n_embd 384 \
--batch_size 16model_training_script/
README.md
train_gpt.py # single entrypoint and implementation
requirements.txt
configs/default.yaml # optional default hyperparameters
plan.md # design notes (if kept)
Artifacts (runs, .venv, etc.) should stay ignored via .gitignore—do not commit them.
If no LICENSE file is present, all rights are reserved; add a license if you open-source the project.
Canonical version and author values live at the top of train_gpt.py (__version__, __author__).