UGTC: Uncertainty-Gated Temporal Credit --- A Modular Advantage Estimator for Actor-Critic RL

Abstract

The Generalized Advantage Estimator fixes a single bias--variance tradeoff ($\lambda$) for every state and training phase. We introduce UGTC (Uncertainty-Gated Temporal Credit), a plug-in module that replaces the advantage estimator with a state-dependent blend of a fast critic ($\lambda=0.80$) and a slow critic ensemble ($\lambda=0.99$, $M=3$), gated by ensemble disagreement. UGTC composes with PPO, TD3, SAC, and DreamerV3 via backbone-specific insertion points. Across six benchmarks (64 tasks, 10 seeds, bootstrap 95% CIs), UGTC-PPO converges $2.7\times$ faster on Hopper ($1.9\times$ wall-clock); UGTC-DreamerV3 peaks at $466.7$ vs. $191.8$ on MetaWorld ML45 ($2.4\times$); UGTC-PPO improves Procgen Hard by $+19.9\%$. We report two negative results---Crafter ($-23.3\%$) and Ant peak return ($-14\%$ vs. TD3)---with deep analysis showing the Ant regression stems from persistent conservative bias in high-dimensional state spaces with spatially uniform critic reliability. A systematic ablation including meta-gradient $\lambda$, REDQ, a learned gate, and a per-state $\lambda$ network confirms that calibrated epistemic uncertainty is the critical ingredient. A held-out sensitivity analysis on Procgen and ML45 validates that defaults transfer without retuning. We derive a practical applicability criterion (reward density $\times$ critic reliability variation, $R^2=0.85$) predicting when uncertainty-gated credit helps versus hurts.