Build Your Own Local LLM From Scratch

What You're Building

A character-level GPT. It reads your text files one character at a time, learns which characters tend to follow which, and generates new text in the same style. Every line of logic is code you can read below, running entirely on your own machine with no external models, API calls, or offloading.

It won't be clever, and that's the point. Because you built all of it, its limits aren't abstract. By the end you'll have seen for yourself why LLMs are fluent, why they hallucinate, and why "knowing" is the wrong word for what they do.

You'll end up with three small files:

• train.py: reads your text, teaches the model, saves the result
• model.py: the transformer itself, where all the actual "intelligence" lives
• generate.py: loads the trained model and produces text, with or without a prompt

Setup

You need Python and PyTorch. A GPU helps but isn't required, and a small model trains fine on CPU, just slower.

In your project folder, create a virtual environment and install PyTorch into it (modern Linux distributions require this, since a bare pip install outside a venv will refuse to run):

python3 -m venv .venv
source .venv/bin/activate
pip install torch

The activate line is needed once per terminal session, and it makes python and pip point at the venv. If a later python train.py complains that torch isn't installed (or python isn't found), you've opened a new terminal and forgotten to activate.

Make a project folder with a data/ subfolder inside it. Drop any .txt files you want to train on into data/. Shakespeare, your own writing, a pile of recipes, educational articles, whatever you like. More text is better, but even a single megabyte will produce something.

Step 1: Read the Files and Build the Vocabulary (train.py)

This is where "training on your own data" actually happens. The model's entire vocabulary is just the set of characters that appear in your files.

import glob
import torch
from model import GPT, block_size

# training knobs — smaller numbers train faster and learn less
batch_size = 32          # sequences processed in parallel
max_iters = 12000        # training steps
eval_interval = 500
eval_iters = 50
learning_rate = 3e-4
device = 'cuda' if torch.cuda.is_available() else 'cpu'
# GPUs do 16-bit maths much faster than 32-bit; autocast applies it where it's safe
amp = torch.autocast('cuda', dtype=torch.bfloat16, enabled=(device == 'cuda'))
torch.manual_seed(1337)

# read every .txt file in data/
files = sorted(glob.glob('data/*.txt'))
if not files:
    raise SystemExit("no .txt files found in data/ — add some text to train on")
text = ""
for f in files:
    with open(f, encoding='utf-8') as fh:
        text += fh.read()

# the vocabulary is every unique character in your text
chars = sorted(set(text))
vocab_size = len(chars)

# map characters to integers and back
stoi = {ch: i for i, ch in enumerate(chars)}
itos = {i: ch for i, ch in enumerate(chars)}
encode = lambda s: [stoi[c] for c in s]
decode = lambda l: ''.join(itos[i] for i in l)

# turn all the text into one long tensor of integers
data = torch.tensor(encode(text), dtype=torch.long)
if len(data) < block_size + 1:
    raise SystemExit("not enough text to fill even one context window")
n = int(0.9 * len(data))
train_data = data[:n]
val_data = data[n:]
print(f"{len(text):,} characters, vocabulary of {vocab_size}")

Notice there's no clever tokeniser here: each character is one token, which keeps the code honest and readable. It also caps how much the model can learn, and that's part of the lesson.

One warning: the model reproduces everything about your files, including their junk. Stray formatting, runs of whitespace and boilerplate headers all get learned just as faithfully as the prose, and they'll show up in its output. Data cleaning is a huge, unglamorous part of training real LLMs, and you're about to find out why.

The last 10% of the text is held back as validation data, text the model never trains on. Watching how the model performs on text it has seen versus text it hasn't is how you tell learning apart from memorising. More on that when we train.

Step 2: Batching (train.py)

The model learns by looking at a chunk of text and trying to predict the next character at every position. This function serves up random chunks.

def get_batch(split):
    d = train_data if split == 'train' else val_data
    ix = torch.randint(len(d) - block_size, (batch_size,))
    x = torch.stack([d[i:i+block_size] for i in ix])
    y = torch.stack([d[i+1:i+block_size+1] for i in ix])
    return x.to(device), y.to(device)

@torch.no_grad()
def estimate_loss():
    out = {}
    model.eval()
    for split in ['train', 'val']:
        losses = torch.zeros(eval_iters)
        for k in range(eval_iters):
            X, Y = get_batch(split)
            with amp:
                _, loss = model(X, Y)
            losses[k] = loss.item()
        out[split] = losses.mean()
    model.train()
    return out

@torch.no_grad()
def sample(n_chars=200):
    """generate a short sample so we can watch the model improve"""
    model.eval()
    context = torch.zeros((1, 1), dtype=torch.long, device=device)  # token 0 — in a sorted vocab this is usually '\n'
    out = decode(model.generate(context, max_new_tokens=n_chars)[0].tolist())
    model.train()
    return out

The model.eval() / model.train() dance matters: the model uses dropout during training (randomly silencing parts of itself so it can't lean on any single connection), and that must be switched off whenever you want its real output.

Step 3: The Model (model.py)

This is the heart of it, and it's worth reading slowly rather than pasting blind. Attention is the mechanism that lets each character's prediction depend on the characters before it, and everything else is plumbing around that idea.

This whole file is the "intelligence". There is nothing else.

import torch
import torch.nn as nn
from torch.nn import functional as F

# architecture knobs — the shape of the model itself
block_size = 256         # how many characters of context the model sees
n_embd = 384             # size of each token's vector
n_head = 6               # number of attention heads
n_layer = 6              # number of transformer blocks
dropout = 0.2


class Head(nn.Module):
    """one head of self-attention"""
    def __init__(self, head_size):
        super().__init__()
        self.key = nn.Linear(n_embd, head_size, bias=False)
        self.query = nn.Linear(n_embd, head_size, bias=False)
        self.value = nn.Linear(n_embd, head_size, bias=False)
        self.register_buffer('tril', torch.tril(torch.ones(block_size, block_size)))
        self.dropout = nn.Dropout(dropout)

    def forward(self, x):
        B, T, C = x.shape
        k = self.key(x)
        q = self.query(x)
        # how much each position should attend to every earlier position
        wei = q @ k.transpose(-2, -1) * k.shape[-1]**-0.5
        wei = wei.masked_fill(self.tril[:T, :T] == 0, float('-inf'))
        wei = F.softmax(wei, dim=-1)
        wei = self.dropout(wei)
        v = self.value(x)
        return wei @ v


class MultiHeadAttention(nn.Module):
    """several attention heads running in parallel"""
    def __init__(self, num_heads, head_size):
        super().__init__()
        self.heads = nn.ModuleList([Head(head_size) for _ in range(num_heads)])
        self.proj = nn.Linear(n_embd, n_embd)
        self.dropout = nn.Dropout(dropout)

    def forward(self, x):
        out = torch.cat([h(x) for h in self.heads], dim=-1)
        return self.dropout(self.proj(out))


class FeedForward(nn.Module):
    """a simple layer that lets the model 'think' on what attention gathered"""
    def __init__(self, n_embd):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(n_embd, 4 * n_embd),
            nn.ReLU(),
            nn.Linear(4 * n_embd, n_embd),
            nn.Dropout(dropout),
        )

    def forward(self, x):
        return self.net(x)


class Block(nn.Module):
    """one transformer block: attention followed by feed-forward"""
    def __init__(self, n_embd, n_head):
        super().__init__()
        head_size = n_embd // n_head
        self.sa = MultiHeadAttention(n_head, head_size)
        self.ffwd = FeedForward(n_embd)
        self.ln1 = nn.LayerNorm(n_embd)
        self.ln2 = nn.LayerNorm(n_embd)

    def forward(self, x):
        x = x + self.sa(self.ln1(x))
        x = x + self.ffwd(self.ln2(x))
        return x


class GPT(nn.Module):
    def __init__(self, vocab_size):
        super().__init__()
        self.token_embedding_table = nn.Embedding(vocab_size, n_embd)
        self.position_embedding_table = nn.Embedding(block_size, n_embd)
        self.blocks = nn.Sequential(*[Block(n_embd, n_head) for _ in range(n_layer)])
        self.ln_f = nn.LayerNorm(n_embd)
        self.lm_head = nn.Linear(n_embd, vocab_size)

    def forward(self, idx, targets=None):
        B, T = idx.shape
        tok_emb = self.token_embedding_table(idx)
        pos_emb = self.position_embedding_table(torch.arange(T, device=idx.device))
        x = tok_emb + pos_emb
        x = self.blocks(x)
        x = self.ln_f(x)
        logits = self.lm_head(x)

        if targets is None:
            loss = None
        else:
            B, T, C = logits.shape
            logits = logits.view(B * T, C)
            targets = targets.view(B * T)
            loss = F.cross_entropy(logits, targets)
        return logits, loss

    @torch.no_grad()
    def generate(self, idx, max_new_tokens, temperature=1.0):
        for _ in range(max_new_tokens):
            idx_cond = idx[:, -block_size:]        # crop to context window
            logits, _ = self(idx_cond)
            logits = logits[:, -1, :] / temperature  # focus on the last position
            probs = F.softmax(logits, dim=-1)
            idx_next = torch.multinomial(probs, num_samples=1)
            idx = torch.cat((idx, idx_next), dim=1)
        return idx

Note what generate does: predict a probability for every possible next character, pick one, append it, repeat. That loop, one character at a time with each choice a roll of weighted dice, is all any GPT ever does. temperature scales how adventurous the dice are, and we'll play with it in Step 6.

Step 4: Train It (train.py, continued)

model = GPT(vocab_size).to(device)
print(f"{sum(p.numel() for p in model.parameters())/1e6:.2f}M parameters on {device}")

optimizer = torch.optim.AdamW(model.parameters(), lr=learning_rate)

for iteration in range(max_iters):
    if iteration % eval_interval == 0 or iteration == max_iters - 1:
        losses = estimate_loss()
        print(f"step {iteration}: train loss {losses['train']:.4f}, val loss {losses['val']:.4f}")
        print(f"--- sample ---{sample()}\n--------------")

    xb, yb = get_batch('train')
    with amp:
        _, loss = model(xb, yb)
    optimizer.zero_grad(set_to_none=True)
    loss.backward()
    optimizer.step()

torch.save({'model': model.state_dict(), 'chars': chars}, 'model.pt')
print("saved model.pt")

Run python train.py. On a decent GPU this takes 15–20 minutes. On CPU, drop to something like n_layer = 4, n_embd = 128, block_size = 64, max_iters = 2000 first or you'll be waiting a while.

One line in the setup deserves a closer look: amp. Modern GPUs have dedicated hardware that does 16-bit arithmetic far faster than 32-bit, and torch.autocast runs the safe parts of the maths in bfloat16 (half-size floating-point numbers) while keeping the delicate parts like the loss in full precision. Nothing about the model changes: same weights, same results to within noise, and the arithmetic is just cheaper. On our small model that one line cuts training time by about 30%, and on billion-parameter models the same trick gets closer to 2×, which is why every serious LLM is trained this way.

It's also a window into something bigger. Because compute is the dominant cost of an LLM (Step 7 makes this concrete), tricks that squeeze more model out of the same silicon are being discovered all the time: mixed precision, FlashAttention (identical attention maths, cleverly reordered to slash memory traffic), better optimizers, better learning-rate schedules, better data curation. A surprising share of LLM progress is not new theory about intelligence but engineering discoveries like this one, quietly compounding. When a lab announces a model matching last year's frontier at a tenth of the cost, this is usually what happened.

Watch the samples. This is the single most instructive part of the whole tutorial. At step 0 the model emits uniform noise. Within a few hundred steps it discovers that spaces exist and words have plausible lengths. Then real short words appear, then bigger word-shaped inventions, then punctuation lands in sensible places. Nobody programmed any of those rules; they are the statistics of your text, absorbed one weight-nudge at a time. When people say an LLM "learned" something, this is the entire mechanism.

Watch the two losses, too. Train loss measures how well the model predicts text it's trained on, and val loss measures text it has never seen. If they fall together, the model is learning genuine patterns. If train keeps falling while val flattens or rises, the model has started memorising its training data rather than generalising, which is the same overfitting problem that frontier labs fight at billion-dollar scale, now visible on your laptop.

Note that the checkpoint saves the vocabulary alongside the weights. The weights are meaningless without knowing which character each number stands for, a small preview of why "downloading a model" always means downloading its tokeniser too.

Step 5: Generate (generate.py)

A separate script, so you can generate any time without retraining. This mirrors how real LLMs work: training happens once at great cost, then the frozen weights are used for inference over and over.

import sys
import torch
from model import GPT

device = 'cuda' if torch.cuda.is_available() else 'cpu'

ckpt = torch.load('model.pt', map_location=device)
chars = ckpt['chars']
stoi = {ch: i for i, ch in enumerate(chars)}
itos = {i: ch for i, ch in enumerate(chars)}

model = GPT(len(chars)).to(device)
model.load_state_dict(ckpt['model'])
model.eval()   # switch off dropout — we want the model's best output now

prompt = sys.argv[1] if len(sys.argv) > 1 else '\n'
temperature = float(sys.argv[2]) if len(sys.argv) > 2 else 0.8

unknown = set(prompt) - set(chars)
if unknown:
    raise SystemExit(f"prompt contains characters the model has never seen: {unknown}")

idx = torch.tensor([[stoi[c] for c in prompt]], dtype=torch.long, device=device)
out = model.generate(idx, max_new_tokens=500, temperature=temperature)[0].tolist()
text = ''.join(itos[i] for i in out)
# a blank line separates conversations in the training data, so the model
# emits one when it feels a conversation has ended — treat that as a stop signal
print(text.strip().split('\n\n')[0])

python generate.py

Early in training this produces noise. After training, it produces text that has the shape of your source material: real-looking word structures, sensible spacing, punctuation in roughly the right places, and no actual meaning.

Notice the last line. Nothing in the model knows how to stop, and it would happily predict the next character forever. But our training data separates conversations with a blank line, so the model learned to emit one when a conversation feels finished, and we simply cut the output there. Real LLMs end their answers the exact same way: they learn a special stop token from their training data, and the surrounding software halts generation when it appears.

Step 6: Prompt It

Here's the experiment that demystifies chat models:

python generate.py "The most important thing about "

The model continues you rather than answering you. Your prompt is just characters it treats as text-so-far, and it predicts what plausibly comes next. This is genuinely what ChatGPT-style systems do underneath. Their training data and some fine-tuning make "plausible continuation of a conversation" look like answering, but the mechanism is exactly the loop you're running. There's no question-understanding step, and there never was.

Also try characters your model has never seen (an emoji, say). It refuses, because it literally has no number for that character. Commercial models have the same hard boundary, just with a vastly bigger vocabulary.

Then play with temperature:

python generate.py "The " 0.5    # cautious: picks likelier characters
python generate.py "The " 1.5    # reckless: gives unlikely characters a real chance

Low temperature produces safer, more repetitive text, and high temperature produces creative garbage. Notice there's no setting that produces truth: temperature only trades predictability against surprise. Every answer a commercial LLM gives you sits somewhere on this same dial.

Step 7: Scale It Up (And Watch What It Costs)

The knobs at the top of model.py and train.py aren't decoration; they are the difference between a bad model and a better one. Here's the same code trained twice. The first run used a smaller model on less data: block_size = 128, n_embd = 192 (2.7M parameters), ~6MB of conversations, about 3 minutes on a consumer GPU. The second is the configuration printed above: block_size = 256, n_embd = 384 (10.8M parameters), ~40MB of conversations, about 17 minutes on the same GPU.

The small run:

$ python generate.py "Hello, how are you doing?"
Hello, how are you doing?
How much me?
I'm sure. I am very sure.
What will you do that?
I please with the that Paron in and Chinese out four day. He seemed to go
to the way. I will catch us you say that this has day.

The big run:

$ python generate.py "Hello, how are you doing?"
Hello, how are you doing?
I'm doing well, thanks. I'm just wondering if it would make a difference.
I can't wait to see why you're constantly being looking forward to succeed
in everything.
Yeah, I am. I don't know if I can start looking for a while now.

Same architecture, same training loop, same generation code, not one line changed. The second model greets you back, holds a grammatical sentence together, and stays on topic for a couple of turns, purely because it had roughly 4× the parameters, 2× the context, 7× the data, and 6× the training compute. Its val loss fell from 1.21 to 0.82, and you can see what that number means in the output.

This experiment, writ large, is the economics of the entire LLM industry. Every leap in quality you've felt between model generations was bought the same way you just bought yours: more parameters, more data, more compute, and compute is money. Your upgrade cost a quarter of an hour extra on a gaming GPU, while the frontier labs run the same trade at warehouse scale, which is why training a state-of-the-art model costs hundreds of millions of dollars and why they charge you per token. Notice the other thing your experiment shows, though: the second model is better, but it still doesn't understand anything, because scale bought fluency rather than comprehension. Keep both halves of that lesson.

Now Look At What You Built, And What It Can't Do

This is the payoff. You now have a working language model whose every behaviour you can trace to code you wrote, so its limits aren't abstract.

It has no idea what words mean. It learned that certain characters follow other characters. Fluency and understanding are completely separate things, and this model has the first without a trace of the second.

Its whole world is your text files. It cannot know anything you didn't feed it. Whatever isn't in that folder simply doesn't exist to it, so there's no hidden knowledge, no reasoning, and no common sense arriving from somewhere else. When a giant commercial model seems to "know" things, that's the same mechanism scaled up across most of the internet, not a different kind of intelligence.

It confidently produces nonsense. Your model generates confident, well-formed, meaningless text, because it optimises for plausible rather than true, and nothing in the loss function you wrote ever mentioned truth. The big models do a subtler version of exactly this, which is why they hallucinate. You've now seen the root cause with your own eyes.

It only ever continues. As Step 6 showed, prompting supplies the start of a document for the model to finish rather than asking it anything. Keep that picture in mind next time a chatbot "gets confused": it hasn't misunderstood you, because understanding was never part of the mechanism.

Scale is doing almost all the work in real LLMs. The architecture you built is genuinely close to what runs GPT-style models. The difference is billions of parameters instead of your few million, a dataset the size of a library instead of your folder, and word-piece tokens instead of characters. Same machine at wildly different scale, and that gap explains both what they can do and where they fall over.

Train it on a few different text files and watch how completely its "personality" is dictated by the input. That single experiment teaches more about how these systems actually behave than any amount of reading.