Chunking and deduplication
FastCDC finds content-dependent boundaries; BLAKE3 identities let byte-identical chunks share one local stored object.
Why content-defined boundaries
Fixed-size blocks shift after an insertion, making every later block look different. Content-defined chunking chooses boundaries from a rolling view of the bytes, so boundaries often resynchronize after a local edit. This can preserve reuse without computing a pairwise delta against every prior version.
Exact reuse
Each chunk is hashed. If the exact bytes already exist under the same identifier, the manifest can reference the existing object. Collision resistance and verification protect identity; similarity is a separate indexing problem.
Where it tends to help
- Local insertions, deletions, or append-heavy changes.
- Repeated assets embedded or copied across paths.
- Container changes that leave large payload regions byte-identical.
- Histories with stable encoded regions.
Where it may not help
- Opaque compression or encryption that randomizes later bytes.
- Full video, image, or audio re-encodes.
- Diffuse numeric updates across model checkpoints.
- Small files where metadata and hashing overhead exceed reuse.
Current limitations
The local alpha still needs stronger bounded-memory ingest, packfiles, high-object-count indexes, and public workload comparisons. The committed FastCDC rate is a component microbenchmark, not repository throughput.