The Cache Consistency Problem I Didn't Know I Had (Until It Cost Us)
Three months into production, our ReplicaSet controller started behaving like a ghost, taking actions based on information that didn't actually exist anymore. A pod would be deleted, our controller would see the old state in its cache, and we'd recreate it seconds later. Users rep...