What is an LLM?
A Large Language Model is a neural network trained on massive text data to predict the next token. That's it — the entire magic comes from next-token prediction at scale.
How LLMs Generate Text
python
class="text-purple-400 font-medium">import torch
class="text-purple-400 font-medium">from transformers class="text-purple-400 font-medium">import AutoTokenizer, AutoModelForCausalLM
model_name = class="text-emerald-class="text-amber-300">400">"gpt2"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name)
class="text-purple-400 font-medium">def generate_text(prompt, max_tokens=class="text-amber-300">50, temperature=class="text-amber-300">0.7):
inputs = tokenizer(prompt, return_tensors=class="text-emerald-class="text-amber-300">400">"pt")
with torch.no_grad():
outputs = model.generate(
**inputs,
max_new_tokens=max_tokens,
temperature=temperature,
do_sample=True,
top_p=class="text-amber-300">0.9
)
return tokenizer.decode(outputs[class="text-amber-300">0], skip_special_tokens=True)
print(generate_text(class="text-emerald-class="text-amber-300">400">"The future of AI is"))Temperature and Sampling
python
class=class="text-emerald-class="text-amber-300">400">"text-gray-class="text-amber-300">500 italic"># Temperature controls randomness
class=class="text-emerald-class="text-amber-300">400">"text-gray-class="text-amber-300">500 italic"># Low (class="text-amber-300">0.1): Deterministic, focused
class=class="text-emerald-class="text-amber-300">400">"text-gray-class="text-amber-300">500 italic"># High (class="text-amber-300">1.5): Creative, random
prompt = class="text-emerald-class="text-amber-300">400">"The meaning of life is"
for temp in [class="text-amber-300">0.1, class="text-amber-300">0.7, class="text-amber-300">1.0, class="text-amber-300">1.5]:
print(class="text-emerald-class="text-amber-300">400">f"\nTemperature {temp}:")
print(generate_text(prompt, temperature=temp, max_tokens=class="text-amber-300">30))Tokenization
python
tokenizer = AutoTokenizer.from_pretrained(class="text-emerald-class="text-amber-300">400">"gpt2")
text = class="text-emerald-class="text-amber-300">400">"Hello, how are you?"
tokens = tokenizer.encode(text)
print(class="text-emerald-class="text-amber-300">400">f"Tokens: {tokens}") class=class="text-emerald-class="text-amber-300">400">"text-gray-class="text-amber-300">500 italic"># [class="text-amber-300">15496, class="text-amber-300">11, class="text-amber-300">703, class="text-amber-300">389, class="text-amber-300">345, class="text-amber-300">30]
print(class="text-emerald-class="text-amber-300">400">f"Decoded: {[tokenizer.decode([t]) for t in tokens]}")
class=class="text-emerald-class="text-amber-300">400">"text-gray-class="text-amber-300">500 italic"># [class="text-emerald-class="text-amber-300">400">'Hello', class="text-emerald-class="text-amber-300">400">',', class="text-emerald-class="text-amber-300">400">' how', class="text-emerald-class="text-amber-300">400">' are', class="text-emerald-class="text-amber-300">400">' you', class="text-emerald-class="text-amber-300">400">'?']The Three Stages of LLM Training
python
class=class="text-emerald-class="text-amber-300">400">"text-gray-class="text-amber-300">500 italic"># Stage class="text-amber-300">1: Pre-training (next token prediction on huge corpus)
class=class="text-emerald-class="text-amber-300">400">"text-gray-class="text-amber-300">500 italic"># - Learns language, facts, reasoning patterns
class=class="text-emerald-class="text-amber-300">400">"text-gray-class="text-amber-300">500 italic"># - Cost: millions of dollars
class=class="text-emerald-class="text-amber-300">400">"text-gray-class="text-amber-300">500 italic"># Stage class="text-amber-300">2: Supervised Fine-Tuning (SFT)
class=class="text-emerald-class="text-amber-300">400">"text-gray-class="text-amber-300">500 italic"># - Learns to follow instructions
class=class="text-emerald-class="text-amber-300">400">"text-gray-class="text-amber-300">500 italic"># - Uses high-quality instruction-response pairs
class=class="text-emerald-class="text-amber-300">400">"text-gray-class="text-amber-300">500 italic"># - Cost: thousands of dollars
class=class="text-emerald-class="text-amber-300">400">"text-gray-class="text-amber-300">500 italic"># Stage class="text-amber-300">3: RLHF / RLVR
class=class="text-emerald-class="text-amber-300">400">"text-gray-class="text-amber-300">500 italic"># - Aligns with human preferences or verifiable rewards
class=class="text-emerald-class="text-amber-300">400">"text-gray-class="text-amber-300">500 italic"># - Makes outputs helpful, harmless, honest
class=class="text-emerald-class="text-amber-300">400">"text-gray-class="text-amber-300">500 italic"># - Cost: moderate compute + reward modelContext Window
python
class=class="text-emerald-class="text-amber-300">400">"text-gray-class="text-amber-300">500 italic"># The context window is how much text the model can class="text-emerald-class="text-amber-300">400">"see"
class=class="text-emerald-class="text-amber-300">400">"text-gray-class="text-amber-300">500 italic"># GPT-class="text-amber-300">4: 128K tokens (~class="text-amber-300">300 pages)
class=class="text-emerald-class="text-amber-300">400">"text-gray-class="text-amber-300">500 italic"># Llama class="text-amber-300">3: 8K-128K tokens
class=class="text-emerald-class="text-amber-300">400">"text-gray-class="text-amber-300">500 italic"># Claude class="text-amber-300">3: 200K tokens
class=class="text-emerald-class="text-amber-300">400">"text-gray-class="text-amber-300">500 italic"># Practical limit: shorter context = faster, cheaper
class=class="text-emerald-class="text-amber-300">400">"text-gray-class="text-amber-300">500 italic"># Longer context = more information but slower inference
class="text-purple-400 font-medium">def chunk_long_text(text, max_tokens=class="text-amber-300">4000):
class="text-emerald-class="text-amber-300">400">""class="text-emerald-class="text-amber-300">400">"Split long text for processing"class="text-emerald-class="text-amber-300">400">""
tokens = tokenizer.encode(text)
chunks = []
for i in range(class="text-amber-300">0, len(tokens), max_tokens):
chunk = tokens[i:i + max_tokens]
chunks.append(tokenizer.decode(chunk))
return chunksKey Takeaways
- ●LLMs are next-token prediction machines trained at massive scale
- ●Temperature controls the creativity vs determinism tradeoff
- ●Training happens in three stages: pre-training, SFT, then RLHF/RLVR
- ●Context window limits how much text the model can process at once
- ●Understanding tokenization helps with prompt engineering and cost optimization