LLM Post-Training

When computing labs release new foundation models, people often focus on the large GPU clusters and the pre-training phase. While pre-training teaches the model language structure and world knowledge,post-training is what turns it from a raw text predictor into a usable, instruction-following assistant.

In this post, we will dive into data preparation, Supervised Fine-Tuning (SFT), Reward Modeling, environments, and the evaluation required to deploy an LLM.

1. Data Preparation & Synthetic Data Generation

The foundation of post-training is data. You cannot align a model without high-quality examples of what "good" behavior looks like. Historically, this meant hiring thousands of human annotators. Today, the meta has heavily shifted toward synthetic data generation using frontier models.

Scenario / Dataset Design

Turning a complex research problem into actionable training data requires formulating the capability into a structured taxonomy. If you simply ask GPT-4, "Generate 100 Python coding problems," you get a generic, useless dataset. Instead, you must engage in strict Scenario Design. You prompt the frontier model to create highly specific constraints: e.g., "Create a Python script that fails because of a deeply nested circular import, then simulate a terminal traceback, then write the thought process fixing it." This ensures your dataset covers the long-tail edge-cases the target model actually needs to learn.

Rubric Design

A Rubric is a set of grading criteria. How do you design a good rubric? You must define independent grading dimensions: conciseness, safety, logical soundness, formatting constraints, and instruction-following constraint checks. A strong rubric ensures that diverse tasks are defined, penalizing repetitive or lazy completions. This filters for high-variance, high-quality data, which helps the target model learn the exact desired behaviors during training rather than mimicking a generic AI "tone". High-quality rubric design is the difference between a capable reasoning model and a babbling one.

Data Quality: LLM-as-a-Judge

Once you have generated synthetic data against your scenarios, you need a scalable data quality filter to prevent garbage from entering your SFT pipeline. This is whereLLM-as-a-Judge comes in. (For a deeper dive into crafting rubrics and alignment prompts, check out LLM as a Judge.)

2. Training

Supervised Fine-Tuning (SFT)

SFT is the process of behavior cloning. The base model is trained on your synthetic dataset to learn the exact formatting and tone expected of an assistant.

Parameter Fine-Tuning with TRL

Training an LLM requires heavily abstracted, highly optimized loops. The industry standard is Hugging Face's TRL (Transformers Reinforcement Learning) library. Parameter fine-tuning using TRL's SFTTrainer abstracts away the complex batching of large context windows, gradient accumulation, and learning rate scheduling necessary to prevent forgetting.

Optimization: LoRA and Quantization

Full-parameter fine-tuning of a 70-billion parameter model is computationally prohibitive for most teams. To solve this, we use the following two parameter-efficient fine-tuning (PEFT) strategies:

• LoRA (Low-Rank Adaptation): Instead of updating billions of parameters, you freeze the base model and inject small, trainable rank-decomposition matrices into the self-attention layers. This cuts memory requirements.
• Quantization (QLoRA): To further save VRAM, the frozen base model weights are mathematically downcast from 16-bit to 8-bit or 4-bit precision, allowing you to fine-tune large models on a single consumer GPU.

The Three Strategies: RLHF, RLVR, and DPO

Once your data is ready, there are three primary strategies to align the policy: RLHF (Reinforcement Learning from Human Feedback), RLVR (Reinforcement Learning from Verifiable Rewards), and DPO (Direct Preference Optimization).

DPO maps preference pairs directly into the language modeling loss function, increasing the probability of the preferred response and heavily penalizing the rejected one. DPO can be a preferred choice when you are constrained by GPU VRAM and don't require advanced reasoning capabilities. It folds alignment into supervised training without the instability of multi-model RL.

RLHF trains a model by sampling outputs and optimizing them using scalar rewards from a reward model. Unlike supervised fine-tuning, it adjusts behavior based on preference signals rather than fixed targets.

RLVR replaces the reward model with a programmatic verifier (e.g., tests, symbolic solvers) that assigns rewards based on correctness. These rewards can be binary or graded. It works well for tasks where correctness can be checked. While it reduces reliance on reward models, it still depends on the verifier's robustness.

Different RL Algorithms:

• 1. TRPO (Trust Region Policy Optimization)

  TRPO restricts how much the policy can change in a single update using a mathematical constraint on the KL divergence. It calculates second-order derivatives (using the Fisher Information Matrix) to guarantee the new model stays within a safe "trust region" of the old model, ensuring stable improvement.

  When to use: Rarely used for LLMs today. The second-order math is computationally brutal and far too slow for models containing billions of parameters.

  
• 2. PPO (Proximal Policy Optimization)

  PPO is the industry standard powering models like ChatGPT. It abandons the computationally expensive strict math of TRPO in favor of efficient first-order derivatives. To maintain training stability and estimate future rewards, it requires running a separate "Value Model" (the Critic) alongside the active policy.

  The primary innovation of PPO lies in its "clipping" objective function. Without any constraints, a gradient update formula scales the advantage by the probability ratio:

  L(θ) = E[r_t(θ)A_t]

  With clipping, PPO prevents the policy from changing too much from its prior state:

  L_PPO(θ) = E[min(r_t(θ)A_t, clip(r_t(θ), 1−ϵ, 1+ϵ)A_t)]

  Here, r_t(θ) is the ratio of new versus old policy probabilities, and A_t is the advantage, i.e., how much better this specific action is than expected. The clip range ϵ strictly controls the update size. PPO improves the policy when the new response is better, but caps the update if it moves too far from the previous model.

  However, optimizing strictly for reward is dangerous. If left unconstrained, the policy will quickly discover edge cases where it can "game" the reward model without actually improving quality, a classic phenomenon known as reward hacking. To counter this, PPO introduces a penalty based on KL Divergence:

  L = L_PPO − β D_KL(π_θ ∥ π_ref)

  L is the final objective function, L_PPO is the reward output, and D_KL(π_θ ∥ π_ref) is the KL divergence between the new policy and the reference policy.

  In this objective, the reward term pushes the model toward preferred outputs, while the KL penalty acts as an elastic band, pulling it back toward the original SFT behavior. The coefficient β dictates the strength of that penalty. Token by token, the model produces a probability distribution. If the model starts outputting unusual or low-quality tokens just because they trick the reward model, the KL divergence will spike, safely reducing the gradient update that caused the shift.

  This penalty is vital because reward models are trained on finite preference data and have blind spots. It prevents reward hacking, preserves the core language intuition and grammar built during pre-training, and acts as a regularizer to stabilize the otherwise high variance of RL updates. However, getting this balance right is difficult. If the penalty is too strong (high β), the model suffers from alignment inertia, it barely deviates from the SFT baseline and reward improvements stall. If it is too weak (low β), the model drifts rapidly, producing responses that are brittle, verbose, or unnatural just to exploit the rubric. Without proper KL balancing, you commonly see a length bias, where the model learns to inflate word count because the reward model correlates verbosity with quality.

  To mitigate these issues, standard practice now involves Adaptive KL control. Instead of a constant coefficient, teams dynamically adjust it: increasing β if KL grows too large, and dropping β if the model starts stalling on reward improvements.

• 3. GRPO (Group Relative Policy Optimization)

  GRPO is the algorithm powering models like DeepSeek-R1. PPO’s biggest flaw is the memory footprint of the Value Model. GRPO eliminates the Value Model by ranking responses relative to each other within a batch. Instead, it generates a group of 4 to 8 responses for the exact same prompt, scores all of them with the Reward Model, and normalizes the scores relative to each other (e.g., using z-scores). Good responses in the group get positive advantages, bad ones get negative advantages.

  A_i = (r_i − μ) / σ
  • r_i: reward for response i
  • μ, σ: mean and standard deviation over the generated group

PPO vs GRPO

Their fundamental difference lies in memory architecture. PPO requires four models simultaneously loaded in memory (Actor, Critic, Reward, and Reference), creating the "RLHF VRAM Wall." GRPO sidesteps this by eliminating the Critic model entirely, instead generating a group of responses and substituting the Critic's baseline with the statistical average of that group.

3. Evaluation & Benchmarks

Evaluation is the systematic process of scoring a model's capabilities across varied, unseen prompts. This step is critical because fine-tuning risks catastrophic forgetting, a scenario where a model successfully masters a new behavioral constraint but loses its underlying reasoning or coding capabilities. Without rigorous evaluation, you are flying blind. Standardized benchmarks provide the rigid tests necessary to prove your model is actually adapting rather than just memorizing its SFT data, and they empirically dictate exactly what new synthetic examples must be generated in the next training cycle to fix newly discovered blind spots.