What is RLVR?
RLVR (Reinforcement Learning from Verifiable Rewards) is a training method where an LLM learns through trial and error, with rewards based on whether its output is objectively verifiable — like correct math answers or working code.
This is different from RLHF (Reinforcement Learning from Human Feedback), which relies on human preferences.
Why RLVR Matters
- ●No human labelers needed — rewards are automated
- ●Scalable — can train on millions of problems
- ●Objective — answers are either right or wrong
- ●Chain-of-thought emerges — models learn to reason step-by-step
The RLVR Loop
python
class=class="text-emerald-class="text-amber-300">400">"text-gray-class="text-amber-300">500 italic"># Simplified RLVR concept
class="text-purple-400 font-medium">import random
class="text-purple-400 font-medium">def rlvr_training_loop(model, problems, num_iterations):
for i in range(num_iterations):
problem = random.choice(problems)
class=class="text-emerald-class="text-amber-300">400">"text-gray-class="text-amber-300">500 italic"># class="text-amber-300">1. Model generates a solution with reasoning
response = model.generate(problem.question)
class=class="text-emerald-class="text-amber-300">400">"text-gray-class="text-amber-300">500 italic"># class="text-amber-300">2. Extract the final answer
answer = extract_answer(response)
class=class="text-emerald-class="text-amber-300">400">"text-gray-class="text-amber-300">500 italic"># class="text-amber-300">3. Verify against ground truth
reward = class="text-amber-300">1.0 if answer == problem.answer else class="text-amber-300">0.0
class=class="text-emerald-class="text-amber-300">400">"text-gray-class="text-amber-300">500 italic"># class="text-amber-300">4. Update model using reward signal
class=class="text-emerald-class="text-amber-300">400">"text-gray-class="text-amber-300">500 italic"># (In practice: GRPO or PPO algorithm)
model.update(
prompt=problem.question,
response=response,
reward=reward
)GRPO: The Algorithm Behind DeepSeek R1
Group Relative Policy Optimization (GRPO) is a variant of PPO that's simpler and more stable:
python
class=class="text-emerald-class="text-amber-300">400">"text-gray-class="text-amber-300">500 italic"># Conceptual GRPO
class="text-purple-400 font-medium">def grpo_step(model, prompt, num_samples=class="text-amber-300">8):
class=class="text-emerald-class="text-amber-300">400">"text-gray-class="text-amber-300">500 italic"># class="text-amber-300">1. Generate multiple responses
responses = [model.generate(prompt) for _ in range(num_samples)]
class=class="text-emerald-class="text-amber-300">400">"text-gray-class="text-amber-300">500 italic"># class="text-amber-300">2. Score each response
rewards = [score(response) for response in responses]
class=class="text-emerald-class="text-amber-300">400">"text-gray-class="text-amber-300">500 italic"># class="text-amber-300">3. Compute relative advantage
mean_reward = sum(rewards) / len(rewards)
advantages = [(r - mean_reward) for r in rewards]
class=class="text-emerald-class="text-amber-300">400">"text-gray-class="text-amber-300">500 italic"># class="text-amber-300">4. Update model to increase probability of high-advantage responses
class=class="text-emerald-class="text-amber-300">400">"text-gray-class="text-amber-300">500 italic"># Advantage = how much better/worse than averageWhen to Use RLVR
python
class=class="text-emerald-class="text-amber-300">400">"text-gray-class="text-amber-300">500 italic"># Good use cases for RLVR:
verifiable_tasks = {
class="text-emerald-class="text-amber-300">400">"math": class="text-emerald-class="text-amber-300">400">"Answer is correct/incorrect (verify against solution)",
class="text-emerald-class="text-amber-300">400">"code": class="text-emerald-class="text-amber-300">400">"Code passes/fails test cases",
class="text-emerald-class="text-amber-300">400">"logic": class="text-emerald-class="text-amber-300">400">"Conclusion follows/doesnclass="text-emerald-class="text-amber-300">400">'t follow class="text-purple-400 font-medium">from premises",
class="text-emerald-class="text-amber-300">400">"retrieval": class="text-emerald-class="text-amber-300">400">"Cited fact exists/doesn't exist in source"
}
class=class="text-emerald-class="text-amber-300">400">"text-gray-class="text-amber-300">500 italic"># Bad use cases:
non_verifiable = {
class="text-emerald-class="text-amber-300">400">"creative_writing": class="text-emerald-class="text-amber-300">400">"Subjective quality",
class="text-emerald-class="text-amber-300">400">"summarization": class="text-emerald-class="text-amber-300">400">"No single correct answer",
class="text-emerald-class="text-amber-300">400">"conversation": class="text-emerald-class="text-amber-300">400">"Depends on user preference"
}Key Takeaways
- ●RLVR trains models using objective, automated rewards
- ●It's how DeepSeek R1 achieved strong reasoning without massive compute
- ●Works best for tasks with verifiable correct answers (math, code, logic)
- ●The model learns to show its work because reasoning chains help find correct answers