A Bash command queue that runs shell jobs in parallel with a concurrency limit

Concurrent Shell Command Queues

▲ 37speckx16 HN discussion
A Bash command queue that runs shell jobs in parallel with a concurrency limit

The author, a self-described Bash enthusiast, builds a small set of shell functions that create a FIFO-backed command queue managed by xargs -P. Commands can be submitted over time rather than all up front, and the queue is inherited by subshells. It works in Bash and Zsh, but the author concedes that for most people Python would be the wiser choice.

You'd be right to ask this question, and my answer is: because I have a fondness for Bash, and I wanted the challenge.
Read original →HN discussionRSS

HN discussion

16
  1. js2

    > How to run shell commands in parallel while limiting their concurrency.

    Depending upon the task, generate a Makefile and run it with:

    make -j <concurrency>

    > My original use case had to do with making a large number of web requests

    This is in fact the use case that caused me to think about using a Makefile for concurrency probably 20 years ago. A rule for downloading each URL to a file. Then a simple `make -j <jobs>` with whatever concurrency I desired and bonus: automatically skipping already downloaded files. e.g.:

    BASE := https://example.com/files

    FILES := a.txt b.txt c.txt d.txt e.txt

    .DELETE_ON_ERROR:

    all: $(FILES)

    $(FILES):

    curl -sS --fail --max-time 30 -o $@ $(BASE)/$@

    Or, to emulate one of the examples in the blog post:

    SHELL := bash

    .SHELLFLAGS := -o pipefail -c

    N := 1 2 3 4 5

    REQS := $(N:%=req-%)

    .PHONY: all $(REQS)

    all: $(REQS)

    $(REQS): req-%:

    @curl -sS --fail-with-body --max-time 10 -X POST -H 'Content-Type: application/json' \

    -d '{"request": $*}' https://postman-echo.com/post | jq -c .json

    Run with:

    $ make -O -j 3

    > You do not need to know all the commands that you want executed up-front, you can submit commands to it over time.

    Ah, yeah, that does put Make in the same bucket as `xargs -P` / GNU `parallel` and requires something more clever.

  2. reconquestio

    There is also GNU Parallel's sem: https://www.gnu.org/software/parallel/sem.html

    sem --id jobs -j 2 'sleep 1; echo FIRST'

    sem --id jobs -j 2 'echo SECOND'

    sem --id jobs -j 2 'echo THIRD'

    sem --id jobs --wait

    SECOND

    THIRD

    FIRST

  3. 0cf8612b2e1e

    I really like pueue for this. Like a light weight slurm or parallel without the nagging citation notice.

    https://github.com/Nukesor/pueue

More from this day

2026-11-13