How to Stop Shipping Low-Quality RL Environments (with Examples)
English summary
Auriel Wright discusses common failures in reinforcement learning training harnesses that produce garbage data. She identifies three major error classes: stale cache, reward hacking, and false resolution. The post emphasizes that a flaky environment corrupts model training and advocates for traditional software engineering practices in RL research. It provides practical advice for building robust harnesses and suggests that teams should fix harness issues before addressing model problems.
Chinese summary
Auriel Wright讨论了强化学习训练环境中常见的故障，这些故障会产生垃圾数据。她识别了三大错误类别：陈旧缓存、奖励黑客和虚假解决。文章强调，不稳定的环境会破坏模型训练，并倡导在RL研究中采用传统软件工程实践。它提供了构建健壮训练环境的实用建议，并建议团队在解决模型问题之前先修复训练环境的问题。
Key points
In RL, the environment is the data generator; a flaky harness systematically produces garbage data that poisons model training.
在强化学习中，环境是数据生成器；不稳定的训练环境会系统地产生垃圾数据，污染模型训练。
Three major error classes: stale cache (returns old data), reward hack (agent games the metric), and false resolution (status change without solving the problem).
三大错误类别：陈旧缓存（返回旧数据）、奖励黑客（代理欺骗指标）和虚假解决（状态改变但未解决问题）。
Additional failures include silent timeouts, non-deterministic resets, reward clipping, mismatched mock data, and action space drift.
其他故障包括静默超时、非确定性重置、奖励裁剪、模拟数据不匹配以及动作空间漂移。
If environment failure rate exceeds 5%, it is a harness problem, not a model problem; fix the harness first.
如果环境失败率超过5%，则是训练环境问题而非模型问题；应优先修复训练环境。
Adopt traditional software engineering best practices in RL research to build robust, production-quality training harnesses.
在强化学习研究中采用传统软件工程最佳实践，构建稳健、生产级别的训练环境。