# Small vs Big - NanoChat vs GPT-OSS 20B

- URL: https://energyai.berlin/thinking/nanochat-vs-gptoss
- Published: 2025-11-04
- Updated: 2026-01-02
- Authors: Muhammad N. ElNokrashy
- Publisher: EnergyAI Berlin GmbH

> nanochat implements a dense transformer decoder with several architectural refinements that have become mainstream since the original GPT-2 paper.

**nanochat**, Andrej Karpathy's successor to **nanogpt**, is a community effort to build a **full chat-enabled LLM training pipeline** from pre-training to alignment.

It aims for a compact, efficient design using small, GPT-2 level models (**561M parameters**). The goal is to optimize cost and performance, to reach a  conversational model **under $100 in total GPU hours**, using modern hardware like the NVIDIA H100.

Before **nanochat**, **nanogpt** was a particularly interesting development in the AI community. Created by Andrej Karpathy, nanoGPT distilled the complexity of large-scale GPT architectures into a minimalist, transparent codebase that fit into just a few hundred lines. It challenged assumptions that only massive corporations could build systems like GPT-2 or GPT-3.

## Modern transformer with strategic optimizations

**nanochat** implements a **dense transformer decoder** with several architectural refinements that have become mainstream since the original GPT-2 paper.

## Core specifications

The default configuration, which you can run via the [`speedrun.sh`](https://github.com/karpathy/nanochat/blob/master/speedrun.sh) script, produces a 561 million parameter model (20 layers, called `d20` in nanochat) for only $100.

### Model configuration

- Model Dimension: 1,280 channels
- Attention Heads: 10 (128 dimensions each)
- Vocabulary: 65,536 tokens
- Context Length: 2,048 tokens
- Precision: BFloat16

## Architectural decisions

**Rotary Position Embeddings (RoPE)**

nanochat employs RoPE, which encodes position information through rotation matrices applied to query and key vectors.

{% figure image="https://firebasestorage.googleapis.com/v0/b/gridconnect-1udkh.firebasestorage.app/o/cms%2Fposts%2Fnanochat-vs-gptoss%2Fimage-1-mv56cf57.webp?alt=media&token=36339965-46e2-423f-b1e4-2863e89b7f52" alt="Figure" caption="" /%}

This provides improved length generalization by making the dot-product between tokens a function of their relative distance rather than absolute position in the sequence.

You can check RoPE implementation on nanochat [here](https://github.com/karpathy/nanochat/blob/b6da6982f6cd81a6769d89353860f458a4c204b9/nanochat/gpt.py#L41).

**RMSNorm without learnable parameters**

Replaces traditional LayerNorm and improves training stability while reducing computational overhead. RMSNorm has proven particularly effective in modern LLM architectures.

You can check the RMSNorm implementation in nanochat [norm](https://github.com/karpathy/nanochat/blob/b6da6982f6cd81a6769d89353860f458a4c204b9/nanochat/gpt.py#L36) .

Given _a̅_ as the activation input to RMSNorm, the output is _a̅ᵢ_ according to:

{% figure image="https://firebasestorage.googleapis.com/v0/b/gridconnect-1udkh.firebasestorage.app/o/cms%2Fposts%2Fnanochat-vs-gptoss%2Fimage-2-mv56cflm.webp?alt=media&token=ad01a138-8cca-4d36-a8b5-5a4c937ec5e0" alt="Figure" caption="" /%}

**Multi-Query Attention (MQA)**

Shares key-value projections across attention heads,
dramatically reducing memory bandwidth requirements during inference.

{% figure image="https://firebasestorage.googleapis.com/v0/b/gridconnect-1udkh.firebasestorage.app/o/cms%2Fposts%2Fnanochat-vs-gptoss%2Fimage-3-mv56cfvn.webp?alt=media&token=ec1d71f3-df1c-4368-8f46-d5011f3b30bd" alt="Figure" caption="" /%}

Squared **ReLU activation**

{% figure image="https://firebasestorage.googleapis.com/v0/b/gridconnect-1udkh.firebasestorage.app/o/cms%2Fposts%2Fnanochat-vs-gptoss%2Fimage-4-mv56cgef.webp?alt=media&token=b33b5c37-d94e-4f1b-a804-a6b93e8db1ce" alt="Figure" caption="" /%}

nanochat uses squared ReLU in the feedforward networks rather than GELU, providing faster convergence to target validation loss.

It is mainly used in the [MLP](https://github.com/karpathy/nanochat/blob/b6da6982f6cd81a6769d89353860f458a4c204b9/nanochat/gpt.py#L121) layer. 

**QK normalization**

Normalizes query and key vectors before attention computation, mitigating the exploding attention logit issue which can destabilize training at higher learning rates. QK Normalization is [applied](https://github.com/karpathy/nanochat/blob/b6da6982f6cd81a6769d89353860f458a4c204b9/nanochat/gpt.py#L77) after RoPE.

**Logit softcap**

Logit Softcap limits the logit value range to stabilize training. This technique is also used in models like [Gemma 2](https://huggingface.co/blog/gemma2) which is based on Gemini.

When logit softcap is combined with QK norm, [learning rates can be increased by approximately 1.5× while mitigating model divergence](https://arxiv.org/abs/2410.16682).

## Training pipeline

{% figure image="https://firebasestorage.googleapis.com/v0/b/gridconnect-1udkh.firebasestorage.app/o/cms%2Fposts%2Fnanochat-vs-gptoss%2Fimage-5-mv56cgng.webp?alt=media&token=cd975b3b-3afd-4799-b01e-ac31e885da4d" alt="Figure" caption="" /%}

## Stage 1: tokenization

The pipeline begins by a Byte-Pair Encoding-based tokenizer written from scratch. The tokenizer encodes approximately 4.8 characters per token with a vocab size of 65,536.

## Stage 2: pre-training on FineWeb-EDU

The base model is pre-trained on the FineWeb-EDU dataset, which is a 1.3 trillion token dataset filtered from Common Crawl using a classifier trained to identify educational content.

## Stage 3: Mid-training

Following general pre-training, the model is tuned on more specialized datasets.

This stage bridges the gap between general language modeling and more specialized capabilities. The model is exposed to everything in a conversation format, roughly following the OpenAI [Harmony](https://github.com/openai/harmony) format, and starts to adapt to the new special tokens that give the multi-turn conversational structure.

- **S****molTalk**: A 1 million sample synthetic dataset of user-assistant conversations designed for supervised fine-tuning of small language models
- **Multiple-choice question datasets**: Using 100k samples from the `auxiliary_train` split of MMLU. Exposes the model to the structure and expectations of multiple-choice questions, which it couldn't learn at such a small scale from just web data.
- **Mathematical reasoning:** GSM8K problems to teach simple math and calculation tool use.

## Stage 4: Supervised Fine-Tuning (SFT)

The SFT stage adapts the model's behavior to function as a helpful chat assistant. Using the the nicest or best subset of data from the mid-training mix, the model learns to follow instructions more closely and to maintain context across multi-turn dialogues. At this stage it can also be tuned for specific properties, such as in safety training. At this size however the model is unlikely to have much in the way of forbidden knowledge.

## Stage 5: Reinforcement learning with GRPO

This stage is optional, it is for users that are willing to invest additional compute. nanochat implements **Group Relative Policy Optimization (GRPO)** which is a reinforcement learning algorithm designed for mathematical and coding tasks.

Unlike traditional **RLHF (Reinforcement Learning from Human Feedback)** which requires expensive human annotations, GRPO uses programmable reward functions with verifiable outcomes.

## Muon optimizer

One of nanochat's most significant technical innovations lies in its optimization strategy. The training pipeline employs a hybrid approach combining Muon for linear layers and AdamW for embeddings and biases.

Muon (Matrix Orthogonalization Optimizer) is a relatively new optimizer that applies a post-processing step to gradient updates.

## KV caching

storing previously computed key and value matrices to avoid redundant computation during token generation. For each new token, rather than recomputing attention over the entire sequence, the system:

1. Retrieves cached KV sectors from all previous tokens
2. Computes QKV values for only the new tokens
3. Performs the attention using current Q and cached KV
4. Updates the cache with the new KV vectors

This optimization reduces computational complexity from O(n²) to O(n) per token, though at the **cost of linear memory growth** with sequence length.

## Scaling across GPUs

While designed to run on a single 8×H100 node, nanochat's training scripts demonstrate production-ready distributed data parallel (DDP) training using PyTorch's [torchrun](https://energyai.berlin/thinking/dspy-multiagent) utility.

## Comparison to 2025 main stream architecture case study: GPT-OSS

{% figure image="https://firebasestorage.googleapis.com/v0/b/gridconnect-1udkh.firebasestorage.app/o/cms%2Fposts%2Fnanochat-vs-gptoss%2Fimage-6-mv56che1.webp?alt=media&token=7e5c208e-4afc-4f2d-8fa6-f1638d4699a1" alt="Figure" caption="" /%}

We will be mainly focusing on the 20 billion variant of GPT-OSS. GPT-OSS and nanoChat represent fundamentally different architectural approaches to language modeling, with GPT-OSS employing a sophisticated Mixture-of-Experts (MoE) design for production deployment while nanoChat uses a traditional dense transformer optimized for educational accessibility and minimal compute budgets.

## Model scale and parameters

GPT-OSS-20B contains **20.9 billion total parameters** but uses sparse activation through its MoE architecture, activating only **3.6 billion parameters per token** (approximately 17.3% of total parameters). This sparse activation strategy enables the model to maintain the knowledge capacity of a much larger system while operating with significantly reduced computational requirements.

In contrast, nanoChat is a **dense transformer with 560.9 million parameters** where all parameters are active during each forward pass. The model uses a depth-based scaling approach where a single **`depth`** parameter controls the entire architecture: with **`depth=20`**, the model has 1,280 channels, 10 attention heads of dimension 128 each, and 20 transformer layers.

## Mixture-of-Experts vs Dense Architecture

The most fundamental architectural distinction lies in GPT-OSS-20B's MoE design. Each of its **24 transformer layers** contains **32 expert networks**, with a lightweight linear router that selects the **top-4 experts per token** via softmax scoring.

nanoChat uses a standard dense transformer where every parameter participates in processing every token. Its feed-forward networks use **ReLU² activation** (squared ReLU) rather than SwiGLU, chosen for improved expressiveness compared to standard activations.

## Position encoding and normalization

Both models use **Rotary Position Embeddings (RoPE)** rather than learned positional encodings, providing better length generalization and more efficient position representation.

For normalization, both use **RMSNorm** instead of LayerNorm. This "Pre-LN" placement (normalization before each attention and MoE/FFN block) improves training stability.

## Context window and scaling

GPT-OSS-20B supports a **131,072-token context window** (128K tokens) using YaRN (Yet another RoPE extensioN) for context extension. This massive context capability enables processing of lengthy documents and extended conversations.

nanochat uses a much smaller **2,048-token context window**, sufficient for its educational demonstrations and conversational use cases but far more limited than GPT-OSS-20B.

## Embedding strategy

GPT-OSS-20B's embedding details are not extensively documented in available sources, though the model uses a standard tokenizer compatible with GPT architectures.

nanochat uses **untied embeddings**—separate input and output embedding matrices, providing additional model capacity without significantly increasing parameter count.

## Evaluations

### Types of evaluation

- **Loss metrics**
   - Bits-Per-Byte (BPB): Vocabulary-size-independent metric
   - Accounts for special tokens and masked positions
- **CORE metric**
   - Multiple-choice tasks: MMLU, ARC, BoolQ, etc.
   - Reading comprehension: SQuAD
   - Schema-based and LM-based evaluation modes
   - Centered scoring (relative to random baseline)
- **Chat tasks**
   - **MMLU**: 57 domains, multiple choice
   - **ARC**: Science reasoning (Challenge + Easy)
   - **GSM8K**: Grade school math (generative)
   - **HumanEval**: Code generation with sandboxed execution
   - **SmolTalk**: Conversation quality

### Code execution safety

Sandboxed Python execution for HumanEval:

- Isolated subprocess with memory limits (256MB)
- Timeout protection (10s default)
- Stdout/stderr capture
- Follows OpenAI's HumanEval methodology

## Limitations of nanochat

As we said before, nanochat makes deliberate trade-offs to achieve its educational mission:

- Not a framework: The repository has readability in mind over abstractions, there are not giant configurable object or model factories.
- Single Node Focus: While the training scales across 8 GPUs, the codebase doesn’t implement multi-node distributed training.
- Limited Model Diversity: nanochat implements a specific architectural configuration so trying other architectures will require code changes.
- Model Performance: Even the $1,000 tier produces models that are far below frontier ones after all, we are not fine-tuning some pre-existing model but we are training it from scratch. The project explicitly acknowledges this, the goal is education not State-of-the-art (SOTA)  performance.
- [English only](https://x.com/karpathy/status/1980665134415802554): The training data focuses on English text

## Main takeaways

- nanochat is an educational, single-file baseline for building a conversational LLM on a very small budget (~$100), prioritizing simplicity and auditability over production-scale complexity.
- Architectural optimizations emphasize practical efficiency: RoPE positional embeddings, RMSNorm, Multi-Query Attention (MQA), QK normalization, Logit Softcap, KV caching, and ReLU² in the MLP.
- Training pipeline is staged: tokenizer (BPE), large-scale pretraining on FineWeb-EDU, mid-training on specialized datasets (SmolTalk, MMLU, ARC, GSM8K), supervised fine-tuning for chat, and optional GRPO reinforcement learning.
- Uses a hybrid optimizer strategy (Muon for linear layers, AdamW for embeddings/biases) to boost convergence efficiency.
- Contrasts with GPT-OSS-20B: nanochat is dense and fully active per token; GPT-OSS-20B is MoE-based (20.9B params, sparse activation ≈3.6B active/token), supports huge context (128K), and uses expert routing with SwiGLU.

## What’s next?

If you are still curious about nanochat, Andrej made a more detailed [overview](https://github.com/karpathy/nanochat/discussions/1) about the project that I recommend checking.