How we made PlanetScale’s background jobs self-healing
When building PlanetScale, we knew that we would be using background jobs extensively for tasks like creating databases, branching, and deploying schema changes. Early on, we decided we had two hard requirements for this system. If we lose all data in the queues at any time, we can recover without any loss in functionality. If a single job fails, it will be automatically re-run. These requirements came about from our past experience with background job systems. We knew that it was "not if, but w
