Job Queues Are Deceptively Tricky: The Hidden Complexity of Scheduling
I discovered that job queues are far more complex than they appear, especially when balancing throughput and latency. While trying to optimize git repo repacking at my workplace, I realized that simple scheduling logic often fails due to hidden semantic choices like how to handle overlapping jobs. Understanding fault models and explicit limits is crucial to avoiding surprising system behaviors.
Reality has a surprising amount of detail.
- groundzeros2015
The UNIX pipe really is an incredible concurrency invention which is not well understood, and attempts to work around its features turn into bugs.
A buffer to accumulate data that blocks when it’s full allows you to handle bursty loads. It solves the back pressure problem of readers and writers operating at different speeds. It doesn’t over consume resources.
It also solves the architecture problem of when to trigger work. Both consumer and producer act on the pipe imperatively, rather than one end being imperative and the other being a declarative graph of callbacks (all those “reactive” libraries).
Even using a term like “back pressure” is a tell to me that someone is confused snd doing something architecturally wrong.
- theamk
I think the the second part was kinda obvious? The moment I read this:
> If you’re anything like me, you would probably have said Parallel Spawn, Prefer New, and Wait are perhaps defensible, whereas Prefer Old feels weird/backward.
it was pretty obvious I was not anything like him. My intuitive answers are pretty different.
- Parallel Spawn is useful, but it's orthogonal to the rest. Even if you have 4 workers, you'll still have to worry about hitting concurrency limit once you have enough jobs. What is it even doing in this list?
- Wait is very useful for non-scheduled tasks: if user uploaded 100 files to process, you better process them all. Sometimes you need limit, sometimes you do not (let them queue for a while until devops notices and either allocate more workers or clear them and has some harsh words with consumer).
For scheduled tasks, "Wait" seems much less useful. I can come up with a reason but they are all somewhat convoluted - perhaps you are hitting 3rd-party service, and it has a quota, so you've decided to use scheduler to avoid hitting ratelimits?
- "Prefer Old" is normally the best way. You repack takes 3-8 hours, so you set your timer to "every 1 hours, skip if running already" and you can be sure that your job finishes and the next one will start.
- "Prefer New" seems almost useless. You've already spent all this effort doing the job, why are you cancelling it and throwing away the results? If you want to add a timeout, add a timeout, preferably to specif […]
- nosefrog
Major lesson from when I worked on Google Search indexing is that queues have a lot of hidden complexity and can make your outages much longer than they need to be. We had a big project to get rid of a bunch of queues by just scaling up our synchronous backends and making them faster.
- teleforce
The most popular resource manager for job submission and queueing system is Slurm. It's being used in majority of TOP500 supercomputers, and overwhelming majority of the world HPCs [1].
SchedMD the leading developer of Slurm has recently being acquired by Nvidia, while Slurm remaining free and open source, but somehow it's Wikipedia entry is not yet updated accordingly.
[1] Slurm Workload Manager:
https://en.wikipedia.org/wiki/Slurm_Workload_Manager
[2] Nvidia Acquires SchedMD (7 comments):
- wewewedxfgdf
You solve this simply with two cron jobs, one for weekend and one weekdays.