DeepSeek-R1 PPO Reinforcement Learning & Reward Model Architecture

You are a Principal Reinforcement Learning Scientist and RLHF Research Director.

Architect and mathematically derive a complete Proximal Policy Optimization (PPO) reinforcement learning engine with {PPO with Generalized Advantage Estimation (GAE-Lambda) & Adaptive KL Penalty} using PyTorch 2.x for {LLM Reasoning Alignment (RLHF) for Mathematical Proof Generation with verifiable verifier feedback}.

### MANDATORY DEEPSEEK-R1 REASONING PROTOCOL:
1. <think>
   - Formulate the clipped surrogate objective function: L^CLIP(θ) = E_t[min(r_t(θ)·Â_t, clip(r_t(θ), 1-ε, 1+ε)·Â_t)].
   - Derive Generalized Advantage Estimation (GAE): Â_t^GAE(γ, λ) = Σ_{l=0}^∞ (γ·λ)^l · δ_{t+l}^V, where δ_t^V = r_t + γ·V(s_{t+1}) - V(s_t).
   - Model the Kullback-Leibler (KL) divergence penalty to constrain policy drift from reference policy π_ref: D_KL(π_θ || π_ref).
   - Implement the Bradley-Terry preference model for pairwise reward estimation: P(y_w > y_l | x) = σ(r(x, y_w) - r(x, y_l)).
   - Construct vectorized mini-batch rollout buffers, value loss clipping, and entropy regularization to encourage exploration.
</think>

### TECHNICAL DELIVERABLES:
1. **Mathematical Derivations**: Step-by-step proofs of policy gradient clipped updates and variance reduction via GAE.
2. **Vectorized PyTorch 2.x Actor-Critic Implementation**: `ActorCritic` neural network module with shared or decoupled backbones.
3. **Full PPO Epoch Training Step**: Clean, production training loop handling advantage normalization, clipped value loss, and gradient clipping.

Action & State Space:
{Discrete autoregressive token actions (vocab size 152,000) with state represented by transformer hidden states (dim 4096)}

Reward Function & Shaping:
{Binary outcome verification (+1.0 for valid Lean 4 / Python code, -1.0 for failure) with length penalty}