Backpressure in media workflows
How Wrangler's copy engine schedules parallel writes across multiple drives without deadlocks, data corruption, or unnecessary re-reads.
A camera card fails on set. The footage exists nowhere else. The DIT pulls the card, plugs it into a reader, and starts copying to three drives: the edit shuttle, the on-set backup, and the lab delivery. Thirty minutes later, the AD wants the card back in the camera. The copy to the shuttle drive finished ten minutes ago. The backup finished five. The lab drive, connected over a slower bus, still has eight minutes left.
Nothing about this is unusual. It happens on every shoot, every day, on every continent where someone points a camera at something expensive. The question is what the software does about it.
The sequential approach is the safe default: copy to drive A, verify, copy to drive B, verify, copy to drive C, verify. The source card gets read three times. On a 512 GB CFexpress card over USB 3.2, that is roughly 45 minutes of unnecessary reads. Multiply by the number of cards in a shooting day. The maths gets uncomfortable fast.
Wrangler solves this with a streaming tee: one source read, parallel writes to all destinations, with backpressure ensuring the slowest writer sets the pace. The card gets read once. All three destinations complete in the time it takes the slowest drive to finish. But getting from “parallel writes” to “correct parallel writes with verification” requires careful thinking about drive scheduling, failure isolation, and evidence tracking.
The five-phase pipeline
Wrangler breaks a copy job into five phases, each with clear entry/exit conditions:
Phase 1: Source hash. Read the source and compute a cryptographic hash. This can be skipped if an MHL manifest already provides a trusted hash, or if we’ll cross-verify in a later pass.
Phase 2: Stream and write. Single source read, streaming through a hash function, teed to all primary destinations simultaneously. Each destination writer independently buffers and flushes. The tee applies backpressure: if destination B’s drive is slower than destination A’s, the source read slows to match B.
Phase 3: Drive scheduling. When multiple jobs run concurrently, the scheduler prevents circular-wait deadlock. Rule: serialise operations on the same physical drive, parallelise across drives. A job writing to drives A and B doesn’t block a job reading from drive C.
Phase 4: Read-back verification. Each destination is read back and its hash compared against the source hash from Phase 1 or 2. This catches silent write corruption: bit-rot, firmware bugs, cable faults.
Phase 5: Cascade verification. A verified primary destination becomes the source for secondary copies. This enables workflows like “shuttle drive to the lab, make archive copies there” without re-reading the original card.
Why backpressure matters
Without backpressure, the tee writes fill up buffers and either: the OS starts swapping (slow), the process runs out of memory (crash), or writes are silently dropped (corruption). With backpressure, the fastest writer simply waits. The overall throughput is determined by the slowest destination, but since all destinations complete in a single pass, the total time is still less than sequential copies.
The key design decision: backpressure is per-pass, not per-block. Within a single pass, all destinations are teed from the same source read. But if destination C needs independent verification (because it’s a cascade target), it gets its own pass with its own source read. This keeps the tee simple while supporting complex verification topologies.
The verification ledger
Every destination accumulates evidence. The ledger tracks what kind of verification each destination received:
read_back: destination was re-read and hash compared to sourcecascade: destination was verified by being used as a source for further copieswrite_hash: hash was computed during the write pass (Hedge/OffShoot model)cross_pass: hash from an independent source re-read matched the write-pass hashmhl: an existing MHL manifest provided trusted hashes
A destination can have multiple evidence types. The ledger records not just “verified” but how it was verified, which matters when presenting evidence to a client or studio.
| Level | Evidence | Proves | Gap |
|---|---|---|---|
| Write hash | Hash computed during write | Software wrote correct bytes to write buffer | Doesn't catch hardware write errors |
| Read-back | Destination re-read, hash matched source | What landed on disk matches what was read from source | Doesn't prove source was good before copy |
| Independent source | Source hashed before copy began | Source was healthy at time of first observation | Doesn't prove source was healthy before that moment |
| Cascade | Verified copy used as source for further copies | Chain extends without re-reading original | Trusts the original verification |
| Cross-pass | Independent second read matched first | Source is consistently readable | Doesn't catch corruption that happened after both reads |
Crash recovery
The entire pipeline is journaled. Every phase transition, every hash result, every verification outcome is appended to an NDJSON journal file. If the process crashes mid-copy, Wrangler reads the journal on restart and resumes from the last completed phase boundary.
The journal is append-only and each event is self-contained. A partial final line (from a crash during write) is safely ignored. This is simpler and more robust than checkpoint files or database transactions. The journal IS the state. Before tools like this existed, DITs built their own pipelines in bash, arriving at the same phases through hard-won experience.