Run commands in parallel with xargs
Process many files concurrently with xargs -P, safely handling spaces and newlines in names.
- Bash
- GreenSnippets
- MIT
# Compress every .log file in a directory using 4 processes
find ./logs -type f -name '*.log' -print0 \
| xargs -0 -n 1 -P 4 gzip -9
# Convert images, naming each output after its input
find ./photos -type f -name '*.png' -print0 \
| xargs -0 -n 1 -P "$(nproc)" -I {} sh -c 'convert "$1" "${1%.png}.jpg"' _ {}Example / expected output
# nproc gives the number of CPU cores (use `sysctl -n hw.ncpu` on macOS).How it works
xargs -P N runs up to N processes at once and -n 1 hands each one a single argument. Pair find -print0 with xargs -0 so unusual filenames survive. Output from parallel jobs can interleave, so write to per-file outputs or use a tool such as GNU parallel when ordering matters.
This snippet has been reviewed but was not executed by an automated test. Test it in your own environment before relying on it.
Related snippets
- Python Tested
Time a block of code with a context manager
Measure elapsed wall-clock time of a with-block using time.perf_counter.
- PHP Tested
Read a large CSV file with a generator
Stream CSV rows as associative arrays using fgetcsv, without loading the whole file into memory.
- JavaScript Tested
Run async tasks with a concurrency limit
Process a large list with at most N promises in flight at once, preserving result order.
- JavaScript Tested
Memoize a function
Cache results by argument so expensive pure functions only run once per distinct input.