Skip to content
wadeghcpPublic

About

GNU dd fanned out to up to 100 output devices in parallel (coreutils 8.27 patch set, GPLv3), with a reproducible build and a 100-output round-trip test

Resources

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

5 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

gangdd

dd that writes one input to many block devices at once. It is GNU coreutils 8.27's dd.c with the output side widened: instead of one of=, it opens up to 100 removable drives and writes every block to all of them in parallel, one thread per device. Written in 2017 to image a stack of SD cards in a single pass rather than N sequential dd runs; published as-is in 2026.

What it does

gangdd accepts the full dd operand set (if=, bs=, count=, conv=, iflag=, oflag=, status=, ...). Two output names are special:

of= value behaviour
gangSDListDevicesOnly print the removable drives that would be written, then exit
gangSD write the input to every removable drive found (careful)
anything else ordinary dd behaviour, one output

Discovery shells out to lshw -class disk -class storage, keeps entries marked removable, and takes their /dev/sd* logical names. It insists on USER=root. Always run the ListDevicesOnly form first and read the list: if a removable drive you care about is plugged in, it will be overwritten.

Internally, each input block is handed to one pthread per device; the threads write the same buffer to their own descriptor (fd 3 upward, sorted by device name) and are joined before the next block is read, so every device sees the identical byte stream. With conv=fsync each device is synced by its own thread and reported as <dev> fully synced. A progress line (written to each device: N MiB) is printed to stdout per block.

# see what would be written
sudo gangdd if=image.img of=gangSDListDevicesOnly

# write it to all of them, 4 MiB blocks, sync each card at the end
sudo gangdd if=image.img of=gangSD bs=4M conv=fsync

# without discovery: name the outputs yourself, one path per line
printf '/dev/sdc\n/dev/sdd\n' > cards
GANGDD_DEVICE_LIST=cards gangdd if=image.img of=gangSD bs=4M conv=fsync

GANGDD_DEVICE_LIST was added at import time (patches/0003) so the parallel path can be tested against plain files; with it set, discovery and the root check are skipped and the listed paths are used verbatim.

Build

Nothing from coreutils is vendored. build.sh downloads the coreutils 8.27 release tarball from ftp.gnu.org (SHA-256 pinned; the tarball's .sig verifies against the GNU keyring), unpacks it under build/, applies patches/*.patch, regenerates the autotools files, runs configure, and builds only src/gangdd:

./build.sh          # -> build/gangdd, about a minute on a modern machine
./test.sh           # 8 outputs, 5,000,001 random bytes; prints PASS or FAIL
./test.sh 100       # any N up to 100

Requires gcc, make, autoconf, automake, autopoint (gettext), perl, xz and curl or wget. Verified on Ubuntu 24.04 with gcc 13.3, glibc 2.39, autoconf 2.71, automake 1.16.5.

Patches

file what
patches/0001-gangdd.patch the 2017 work: src/gangdd.c, man/gangdd.x, and the one-line hunks that register gangdd in coreutils' program lists and link it with pthreads (build-aux/gen-lists-of-programs.sh, m4/cu-progs.m4, src/cu-progs.mk, src/local.mk, src/single-binary.mk, man/local.mk, cfg.mk)
patches/0002-gnulib-modern-toolchain-fixes.patch what coreutils 8.27's bundled gnulib needs to compile today: <limits.h> in lib/xdectoint.c; O_BINARY defaulting to 0 in lib/binary-io.h; the glibc 2.28 libio change (_IO_ftrylockfile gone) in lib/stdio-impl.h and the f*.c stdio helpers, as upstream gnulib fixed it in 2018
patches/0003-gangdd-device-list-env.patch the GANGDD_DEVICE_LIST override described above
patches/0004-gangdd-device-name-buffer-lifetime.patch bug fix, found by test.sh: the device-name buffer was a block-scoped VLA that died before the outputs were opened, so with more than about 16 devices the names were overwritten and the run failed with failed to open ''. Now a static 4 KiB buffer (256 bytes held ~25 names, short of the 100 limit), with a bounds check on the device count

src/gangdd.c and man/gangdd.x are also in the tree as plain files for reading; src/gangdd.c is the state after all four patches, and the import commit holds the untouched 2017 version.

Test

test.sh N BYTES writes BYTES random bytes to N plain files through the gangSD path and checks that the dry run lists all N and writes nothing, that the copy exits 0, and that every output has the input's size and sha256. It needs no root and no devices. Passes at N = 8, 20, 50 and 100.

Known limitations

Inherited from the 2017 code and left as they are:

  • Discovery depends on lshw's text output and on /dev/sd* naming, and checks the USER environment variable rather than the effective uid.
  • Write errors are only detected on the first device: iwrite returns the first thread's byte count, and the per-device status array is set only by the conv=fsync path. A card that fails mid-write will not by itself fail the run; use conv=fsync and compare afterwards.
  • The per-device byte counters are int.
  • The progress line is printed regardless of status=none.
  • The man page stub still says "convert and copy a file".

Licence

GNU General Public License v3 or later (see LICENSE, coreutils' COPYING). gangdd is a derivative of GNU coreutils' dd and carries the FSF's copyright alongside the author's; anyone distributing binaries must make the corresponding source available under the same terms. build.sh downloads the unmodified coreutils source rather than redistributing it.

Provenance

Written in May 2017 against coreutils 8.27. The private archive held one 190,346-line diff -ruN of the whole patched tree, 98% of it regenerated configure, Makefile.in and autom4te.cache output. Published in September 2026 with the hand-written parts extracted, a reproducible build, a test, and the two fixes noted above.

About

GNU dd fanned out to up to 100 output devices in parallel (coreutils 8.27 patch set, GPLv3), with a reproducible build and a 100-output round-trip test

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages