FQE Improvement Without Bellman Completeness

This paper addresses a key limitation of Fitted Q-Evaluation (FQE), a core technique in off-policy reinforcement learning. FQE typically requires Bellman completeness, a difficult condition to satisfy. The authors identify a norm mismatch as the root cause and propose a simple reweighting strategy using the stationary density ratio. This allows for strong evaluation guarantees without the restrictive Bellman completeness assumption, improving the robustness and practicality of FQE.