A job should be able to be un-scheduled at any time, based on a unique identifier issued by the queuing system. The client library should be designed to promote best practices such as exponential backoffs and rate limit handling, rather than to expose the internal of the scheduling service. The fact that the job is scheduled with a specific job scheduler and transported in a message queue should be transparent to developers. For example, the job queue should support individual message success/fail acknowledgment (failing to process a single message should not affect the others), dead letter queues (messages that have failed to be processed for a number of times should be moved to a separate queue), and separate worker pools for each individual consumer (each service should run its own worker pool, rather than a shared worker pool).
Besides scheduling the jobs reliably, the system should also offer an efficient queuing interface. The scheduler should have a p95 scheduling deviation lower than 10 seconds. In lots of our use cases, applications require jobs to run within seconds of their scheduled time. A single application’s queue being overwhelmed should not affect job processing in other services. The system should be able to isolate jobs for each application. The scheduling system should be horizontally scalable, to allow for capacity planning as the Airbnb community grows. Airbnb believes in long-term investments, and our queuing system should be able to scale and support our needs in the future without significant scaling efforts. It should guarantee at-least-once delivery of every single job. The system should not lose data if the system fails or restarts. After talking to teams at Airbnb, we decided that the scheduling system has to provide the following abilities: In those cases, a reliable and easy-to-use jobs scheduling system would be extremely useful and also necessary. These jobs can often be scheduled months or even years into the future, and they must be delivered reliably to ensure a good experience for the Airbnb community.
In another example, before the scheduled check-in time for a reservation, we send guests a reminder that it’s time to get ready for their trip. The matching process is quite complicated and takes intensive compute resources, so we put the matching jobs on a job queue to ensure Airbnb’s reliability and responsiveness to those in need. For example, when Airbnb’s community of generous hosts join our Open Homes program, Airbnb will match them with non-profits or evacuees in need.
Many systems at Airbnb take advantage of a job queue. In this article, we will walk through the history of job queuing systems at Airbnb, explain why we built Dynein, describe how we were able to achieve its high scalability, and finally, open-source the highly scalable scheduler we built for Dynein. Since its introduction, the system has become a very important component of our architecture, powering use cases from delivering in-app messaging to dynamic pricing, all with very high number of transactions per second. Having a background queue helps the web servers handle incoming web requests promptly, and reduces the likelihood of performance issues that occur when requests become backlogged.Īt Airbnb, we built a job scheduling system called Dynein for very critical use cases. Asynchronous background jobs can often dramatically improve the scalability of web applications by moving time-consuming, resource-intensive tasks to the background. These tasks are often prone to failures, and retrying mechanisms often make it even more expensive to operate applications with such jobs.
0 kommentar(er)
