What the benchmarks actually taught us

Plain-English notes on the measurements in bench/RESULTS.md and the decisions they forced in engine/tasks.md. Written for someone who has not read either.

Everything here is a number somebody ran on real hardware. Where a belief turned out to be wrong, that is written down too — those are the most useful entries. Machine unless stated otherwise: MacBook Pro (M1 Max, 10 core), 64 GB, macOS 26.6.1, APFS on internal NVMe.

1. The fastest copy is the one that copies nothing

APFS is copy-on-write, so clonefile(2) makes a second name for the same bytes and only writes when someone modifies them.

A single 4 GB file clones in 106 µs against 3.71 s for copyfile — roughly 35,000x.

The subtle part. Per-file COPYFILE_CLONE and tree-level clonefile clone exactly the same bytes, but one is 2.7x and the other is 22x. The entire difference is per-file syscall overhead. When you are doing nothing per item, doing it 10,000 times is still the whole cost. The rule that came out of this: always try clonefile at the highest common directory first, and only decompose into per-file work when that fails.

Also worth noticing: cp -R and ditto are both slower than a plain sequential copyfile loop. The system tools were not a shortcut worth taking.

2. Moving a folder should be one syscall

Same volume, 100,000 files. No bytes need to move — only a directory entry changes.

Projected to a 1.32M-file corpus: 25 ms versus ~102 seconds versus ~9 minutes.

Two things a junior dev should take from this.

Parallelism is not a general-purpose speedup. Eight threads of renameat moved 13k/s to 14k/s. APFS serializes directory metadata mutation, so the threads queue behind a lock instead of running side by side. When your bottleneck is a lock rather than CPU or I/O, adding workers adds context switches and nothing else. Contrast this with §1, where parallel copying was worth 2.4x on the same machine — same hardware, different bottleneck.

Read the failure modes of your fast path. rename() on a directory fails with ENOTEMPTY when the destination already exists and has contents, which drops you into per-file work: 25 ms becomes 102 seconds. So if a file manager ever felt slow moving files within one disk, it was almost certainly a merge into an existing folder — not a slow computer.

A review of the strategy table caught a related bug worth repeating. An empty destination directory was being treated like a non-empty one, but POSIX rename(2) happily replaces an empty directory (measured on APFS at 0.12 ms). That mistake cost 25.6 ms → 7.76 s, a 303x loss, on the extremely common "make the folder first, then move things into it" flow.

3. The standard library was the slow part

Listing a flat directory with metadata for every entry, 100k entries:

The plan going in claimed getattrlistbulk would be "5–20x". It is 1.75x. That claim was wrong and got corrected in writing.

But the real finding is the third row. FileManager with prefetched resource keys — the documented fast path, the thing nearly every Swift app uses — is half the speed of naive readdir + stat and 3.2x slower than the bulk syscall. The win is real, but it is a win against Foundation, not against the kernel. Know which baseline you are beating.

Two smaller lessons: buffer size barely mattered (64 KiB ≈ 1 MiB — don't tune what you haven't measured), and the index beat every syscall strategy by 12x while returning rows already sorted. Choosing a better syscall is a second-order optimization; not doing the work at all is first-order.

4. Synthetic corpora lie, and so does linear extrapolation

Walking the real home directory — 9.4M files, 1.8M directories — ran at 38–43k entries/s. The synthetic flat corpus ran at 473k/s. A 10x gap, because real trees are directory-open bound, not entry bound. A benchmark on one big flat directory tells you almost nothing about a real tree.

Related: an early target said a cold snapshot of 871k entries should complete in under 2 seconds. That number was extrapolated from the 100k measurement. When someone actually built the 871k fixture, raw enumeration alone took 11.7–13.2 s and the full snapshot took 25.21 s. No implementation layered on that syscall could have hit 2 s. The target was arithmetic, not measurement, and it was replaced rather than quietly missed.

What the app ships instead: first visible batch of rows in 87.8 ms, and an app-cold load from the persisted index in 189 ms — you just cannot have the complete first-ever scan quickly, because the filesystem will not give it to you.

5. The "files don't show up" bug was a contract violation

The complaint being chased: files created during heavy activity never appear in the file manager. The assumption was that the change-notification API (FSEvents) was too slow, and polling would fix it.

FSEvents is 11x faster to notice a change. Replacing it with polling would have made the app feel worse. But then the flood test — 40,000 files created in 1.56 s:

DROP FLAGS RAISED: ["MustScanSubDirs": 16, "UserDropped": 16]

FSEvents is lossy under load. It also says so, sixteen times, in flags the caller receives. It never silently lost anything; it reported "I dropped events in this subtree, go rescan it."

So the bug was never in the kernel. It is in any client that treats a hint stream as a reliable change log. Ignore the drop flags and your directory listing goes permanently stale after any burst — an unzip, a build, a large download — until something forces a re-read. That is an exact match for the symptom.

This is the most transferable lesson in the whole file. The API told the truth, in the documented way, and the bug was on the reading side. Before concluding an API is broken, check whether it is reporting a condition you are throwing away.

6. Local and remote have opposite bottlenecks

Locally, parallelism was worth 2.4x and batching was irrelevant. Over ssh, it inverts completely.

10,000 files (39 MB), wired, to a Linux host:

Projected to 1.32M files: sftp ≈ 4.3–5.8 hours; framed ≈ 3.7–4.4 minutes.

sftp is not badly written — it is doing a round trip per file. At any real network latency, a per-file protocol is dead on arrival no matter how fast either machine is. Wrapping the files into one stream is worth 20–80x. Adding parallel streams on top of that is worth 1.0–1.6x, and the measurements are noisy enough that the project kept a single stream as the default.

That became a standing decision: framing beats parallel streams; stream count is a tunable, never an architecture concern. The local result — where parallelism is everything and framing is meaningless — is the exact inverse. Same codebase, opposite advice, because the bottleneck moved from disk to latency.

7. ZFS vs ext4, and the result that did not survive a re-run

This one is a lesson about benchmarking itself, not about filesystems.

Two destination hosts: freya (ZFS, 32-core x86_64) and gentoo-rpi5 (ext4, 4-core Raspberry Pi 5). Over Wi-Fi, the Pi beat the server:

The conclusion written down at the time: destination filesystem beats destination CPU — ZFS's metadata and sync behaviour punishes many-small-file creation hard enough that a 4-core Pi on ext4 wins. Which is a genuinely interesting claim.

Then the same benchmark was re-run wired:

The result flipped. freya's rsync went from losing at 2,459 to winning at 5,422 — the Pi's numbers barely moved. The Wi-Fi link, not the filesystem, was the thing being measured. Note that the original write-up already flagged the danger: freya's single-stream figure ranged from 1,901 to 5,014 files/s across runs, a 2.6x spread. A 2.6x spread in your measurement means you cannot support a 1.7x conclusion.

If you take one habit from this document: when run-to-run variance is larger than the effect you are claiming, you have not measured the effect. Control the noisy variable — here, replace Wi-Fi with Ethernet — and re-run before writing the conclusion down.

A direct wired host-to-host run (mars → freya, no laptop in the middle) reached 29,139 files/s with 8 streams, which tells you how much the laptop's link was costing in every earlier number.

8. Why BLAKE3, and why it took four rewrites

The honest version of this story is more interesting than "we picked the fast hash".

The engine needs a content identity hash: prove the bytes that arrived are the bytes that were sent, before deleting anything from the source. The requirement that decides the algorithm is not speed — it is that both ends must compute the same digest. The sender is Swift on macOS; the receiving agent is a Go binary on Linux (github.com/zeebo/blake3). Apple's CryptoKit, which gets hardware-accelerated SHA-256 on Apple silicon, does not exist on the Linux side. BLAKE3 also hashes as a tree, which is what lets a transfer verify individual chunks and resume mid-file instead of re-reading everything.

So the choice was made on portability and structure. Then came the performance work — four attempts at the same function, hashing 256 MiB:

Two things worth sitting with. First, the portable C backend was still slower than hardware SHA-256 on this machine — the record says so plainly. BLAKE3 only becomes competitive once the NEON (ARM SIMD) backend is switched on, which is a 2.2x jump over portable C from the same algorithm and the same library. Choosing a fast algorithm buys you nothing if you run a portable implementation of it.

Second, this story is not finished. The acceptance criterion is that hashing adds ≤ 5% to the streaming copy path, and E5.5 is still an open story. Four rewrites, a 26x total improvement, and the gate has not been met — because 0.152 s of hashing against ~0.15 s of copying is still roughly doubling the work. Real optimization work looks like this far more often than it looks like one clever trick.

9. Smaller findings that will save you an afternoon

Sample the data; don't guess from the file extension. A compression sweep took a bounded 1 MiB sample and chose from the observed ratio: zstd for XML (ratio 0.323), no compression for random media (ratio 1.000). It got every case right without ever looking at a filename. On the Pi, gzipping incompressible media was slower than not compressing at all (6.43 s vs 4.49 s) — a slow CPU turns compression into a tax.

SSH connection setup dominates small requests. Ten hello requests: a fresh SSH connection each time cost 2.295 s (freya) and 3.626 s (mars); one persistent process cost 0.367 s and 0.220 s. That is up to 16x, entirely in handshakes.

macOS tar silently doubles your file count on Linux. The OS stamps com.apple.provenance on files, so bsdtar emits an AppleDouble ._ sidecar for each one: a 2,000-file corpus landed as 4,001 files. COPYFILE_DISABLE=1 did not suppress it; --no-mac-metadata did. Any existing tar | ssh workflow on a Mac is probably littering ._ files on the far end right now.

Label your baselines. Projections in f2 plan were carrying 2,500 files/s as the parallel-copy rate. That number was actually the sequential copy+delete rate, so every estimate built on it was wrong. It was replaced with measured rates (clonefile 70k entries/s, parallel copy 7.7k/s at width 16).

Watch for constants that were measured once. The large-file threshold is labelled PROVISIONAL in the source because only a single 4 GB file was ever measured. Naming that honestly is better than letting a magic number look authoritative.

10. The measurement that was almost a wrong feature

An indexing cost model started with: 891,382 files changed in the last 24 hours — about 30% of all text and code on the machine, every day. That makes continuous indexing a permanent tax and argues for a very different design.

Attributing the churn by directory showed 871,244 of those files in a single tree, and there the age buckets looked like this:

chg 24h = chg 7d = chg 30d = 871,886

Identical. Files do not change on a schedule that produces identical 1-day, 7-day and 30-day counts. That equality is the signature of a bulk write — one import — not of churn. stat confirmed discrete import batches on specific dates.

Corrected: real daily churn is a few hundred authored files, not 891k. The recurring cost tracks bytes changed, not files stored. The tool now detects this shape directly (chg24h / chg30d > 0.95 over a subtree) and reports bulk separately.

The same walk turned up 367,618 dataless files — iCloud placeholders whose contents are not on disk. Reading those to index them would trigger 367,000 cloud downloads. Enumeration has to check the SF_DATALESS flag and skip, and the cost is bandwidth and battery, not CPU.

The lesson: when a number is surprising, look at its shape before you design around it. The first number was real, correctly computed, and would have driven a worse product.

11. What is still not measured

Reading a benchmark file well means knowing where it is silent.

• NFS: no data at all. Nothing in the repo measures it.
• SMB: a mount, but no numbers. The two reports under bench/f2bench-reports/network/ record a mounted share from freya.local and the build output — and nothing else. RESULTS.md still says every SMB/NFS claim is unvalidated, and that is still true.
• Everything is warm-cache. Purging the page cache needs sudo purge, which was not run. "Cold" in the tables means first repetition, not cold kernel caches. This understates the bulk-syscall advantage, because readdir+fstatat benefits from a warm cache more than getattrlistbulk does.
• The corpora are synthetic and uniform. Real directories have varied sizes, xattrs and compression, all of which move these numbers — see §4 for how much.
• The project's own abort criterion is only half satisfied. "Beat the incumbent 3x on transfers" is met for same-volume copy (22x tree clone, 2.45x parallel small-files) and unvalidated for network — which is the case that motivated the project in the first place.

That last point is the reason the harness fingerprints hardware, OS and filesystem on every run, and treats results as advisory when the fingerprint does not match. A benchmark you cannot reproduce is a story, not a gate.