When Synchronous RL Actually Matters: My Take on A2C vs A3C
I was debugging a reinforcement learning pipeline at 2 AM last month when I realized I'd spent the entire week trying to get A3C working in production. The asynchronous nature of the algorithm, which should've been a feature, had become my nightmare. Race conditions in the gradient...