Small vs Big - NanoChat vs GPT-OSS 20B

nanochat, Andrej Karpathy's successor to nanogpt, is a community effort to build a full chat-enabled LLM training pipeline from pre-training to alignment.

It aims for a compact, efficient design using small, GPT-2 level models (561M parameters). The goal is to optimize cost and performance, to reach a conversational model under $100 in total GPU hours, using modern hardware like the NVIDIA H100.

Before nanochat, nanogpt was a particularly interesting development in the AI community. Created by Andrej Karpathy, nanoGPT distilled the complexity of large-scale GPT architectures into a minimalist, transparent codebase that fit into just a few hundred lines. It challenged assumptions that only massive corporations could build systems like GPT-2 or GPT-3.

Modern transformer with strategic optimizations

nanochat implements a dense transformer decoder with several architectural refinements that have become mainstream since the original GPT-2 paper.

Core specifications

The default configuration, which you can run via the speedrun.sh script, produces a 561 million parameter model (20 layers, called d20 in nanochat) for only $100.

Model configuration

Architectural decisions

Rotary Position Embeddings (RoPE)

nanochat employs RoPE, which encodes position information through rotation matrices applied to query and key vectors.

This provides improved length generalization by making the dot-product between tokens a function of their relative distance rather than absolute position in the sequence.

You can check RoPE implementation on nanochat here.

RMSNorm without learnable parameters

Replaces traditional LayerNorm and improves training stability while reducing computational overhead. RMSNorm has proven particularly effective in modern LLM architectures.

You can check the RMSNorm implementation in nanochat norm .

Given a̅ as the activation input to RMSNorm, the output is a̅ᵢ according to:

Multi-Query Attention (MQA)

Shares key-value projections across attention heads, dramatically reducing memory bandwidth requirements during inference.

Squared ReLU activation

nanochat uses squared ReLU in the feedforward networks rather than GELU, providing faster convergence to target validation loss.

It is mainly used in the MLP layer.

QK normalization

Normalizes query and key vectors before attention computation, mitigating the exploding attention logit issue which can destabilize training at higher learning rates. QK Normalization is applied after RoPE.

Logit softcap

Logit Softcap limits the logit value range to stabilize training. This technique is also used in models like Gemma 2 which is based on Gemini.

When logit softcap is combined with QK norm, learning rates can be increased by approximately 1.5× while mitigating model divergence.

Training pipeline

Stage 1: tokenization

The pipeline begins by a Byte-Pair Encoding-based tokenizer written from scratch. The tokenizer encodes approximately 4.8 characters per token with a vocab size of 65,536.

Stage 2: pre-training on FineWeb-EDU

The base model is pre-trained on the FineWeb-EDU dataset, which is a 1.3 trillion token dataset filtered from Common Crawl using a classifier trained to identify educational content.

Stage 3: Mid-training

Following general pre-training, the model is tuned on more specialized datasets.

This stage bridges the gap between general language modeling and more specialized capabilities. The model is exposed to everything in a conversation format, roughly following the OpenAI Harmony format, and starts to adapt to the new special tokens that give the multi-turn conversational structure.

Stage 4: Supervised Fine-Tuning (SFT)

The SFT stage adapts the model's behavior to function as a helpful chat assistant. Using the the nicest or best subset of data from the mid-training mix, the model learns to follow instructions more closely and to maintain context across multi-turn dialogues. At this stage it can also be tuned for specific properties, such as in safety training. At this size however the model is unlikely to have much in the way of forbidden knowledge.

Stage 5: Reinforcement learning with GRPO

This stage is optional, it is for users that are willing to invest additional compute. nanochat implements Group Relative Policy Optimization (GRPO) which is a reinforcement learning algorithm designed for mathematical and coding tasks.

Unlike traditional RLHF (Reinforcement Learning from Human Feedback) which requires expensive human annotations, GRPO uses programmable reward functions with verifiable outcomes.

Muon optimizer

One of nanochat's most significant technical innovations lies in its optimization strategy. The training pipeline employs a hybrid approach combining Muon for linear layers and AdamW for embeddings and biases.

Muon (Matrix Orthogonalization Optimizer) is a relatively new optimizer that applies a post-processing step to gradient updates.

KV caching

storing previously computed key and value matrices to avoid redundant computation during token generation. For each new token, rather than recomputing attention over the entire sequence, the system:

This optimization reduces computational complexity from O(n²) to O(n) per token, though at the cost of linear memory growth with sequence length.

Scaling across GPUs

While designed to run on a single 8×H100 node, nanochat's training scripts demonstrate production-ready distributed data parallel (DDP) training using PyTorch's torchrun utility.

Comparison to 2025 main stream architecture case study: GPT-OSS

We will be mainly focusing on the 20 billion variant of GPT-OSS. GPT-OSS and nanoChat represent fundamentally different architectural approaches to language modeling, with GPT-OSS employing a sophisticated Mixture-of-Experts (MoE) design for production deployment while nanoChat uses a traditional dense transformer optimized for educational accessibility and minimal compute budgets.

Model scale and parameters

GPT-OSS-20B contains 20.9 billion total parameters but uses sparse activation through its MoE architecture, activating only 3.6 billion parameters per token (approximately 17.3% of total parameters). This sparse activation strategy enables the model to maintain the knowledge capacity of a much larger system while operating with significantly reduced computational requirements.

In contrast, nanoChat is a dense transformer with 560.9 million parameters where all parameters are active during each forward pass. The model uses a depth-based scaling approach where a single depth parameter controls the entire architecture: with depth=20, the model has 1,280 channels, 10 attention heads of dimension 128 each, and 20 transformer layers.

Mixture-of-Experts vs Dense Architecture

The most fundamental architectural distinction lies in GPT-OSS-20B's MoE design. Each of its 24 transformer layers contains 32 expert networks, with a lightweight linear router that selects the top-4 experts per token via softmax scoring.

nanoChat uses a standard dense transformer where every parameter participates in processing every token. Its feed-forward networks use ReLU² activation (squared ReLU) rather than SwiGLU, chosen for improved expressiveness compared to standard activations.

Position encoding and normalization

Both models use Rotary Position Embeddings (RoPE) rather than learned positional encodings, providing better length generalization and more efficient position representation.

For normalization, both use RMSNorm instead of LayerNorm. This "Pre-LN" placement (normalization before each attention and MoE/FFN block) improves training stability.

Context window and scaling

GPT-OSS-20B supports a 131,072-token context window (128K tokens) using YaRN (Yet another RoPE extensioN) for context extension. This massive context capability enables processing of lengthy documents and extended conversations.

nanochat uses a much smaller 2,048-token context window, sufficient for its educational demonstrations and conversational use cases but far more limited than GPT-OSS-20B.

Embedding strategy

GPT-OSS-20B's embedding details are not extensively documented in available sources, though the model uses a standard tokenizer compatible with GPT architectures.

nanochat uses untied embeddings—separate input and output embedding matrices, providing additional model capacity without significantly increasing parameter count.

Evaluations

Types of evaluation

Code execution safety

Sandboxed Python execution for HumanEval:

Limitations of nanochat

As we said before, nanochat makes deliberate trade-offs to achieve its educational mission:

Main takeaways

What’s next?

If you are still curious about nanochat, Andrej made a more detailed overview about the project that I recommend checking.