Question 1

What is Dits for AI?

Accepted Answer

Dits for AI is content-addressed version control for the heaviest, least version-controlled data in machine learning and research — model weights, checkpoints, training datasets, and the scientific data behind them. It splits every artifact into content-defined chunks, addresses each chunk by its BLAKE3 hash, and stores each unique chunk once. You get real commits, diffs, and lineage over multi-gigabyte files instead of a folder full of model-final-v3.safetensors.

Question 2

Is this only for AI, or for scientific data too?

Accepted Answer

Both. AI is the sharpest example of a much wider problem: any field whose most valuable data is large, binary, and constantly re-versioned. The same engine handles genomic reads, geospatial rasters, simulation output, and microscopy stacks. If your data is big, changes incrementally, and currently lives in a bucket with version numbers in the filename, Dits is built for it.

Question 3

How is it related to Dits for media?

Accepted Answer

It is the same open-core engine pointed at a different set of enormous files. Dits for media versions video, photos, and 3D assets; Dits for AI versions models, datasets, and research data. Identical primitives (chunks, manifests, commits), different heaviest files. Nothing is forked — improvements to the engine benefit both surfaces.

Question 4

Who is it for?

Accepted Answer

ML engineers tracking checkpoints and fine-tunes, data teams versioning training sets, research labs that need reproducible data lineage, and platform teams tired of paying full storage and bandwidth for files that barely changed. If you have ever asked 'which weights actually shipped?' or 'which exact dataset produced this result?', that is the gap Dits closes.

Question 5

Do I need to know Git?

Accepted Answer

It helps but is not required. The commands mirror Git — add, commit, diff, log, branch, checkout — so anyone comfortable with Git is productive immediately. The mental model is the same; only the size and type of the files are different.

Question 6

How is this different from Git LFS?

Accepted Answer

Git LFS stores a complete copy of every version of a large file — change one layer of a 30 GB checkpoint and you store all 30 GB again. Dits chunks files and stores only the chunks that actually changed, deduplicated across every file and version. It also verifies every read against its BLAKE3 hash, where LFS just trusts the pointer.

Question 7

How does it compare to Hugging Face Xet?

Accepted Answer

Xet pioneered content-defined chunking for model and dataset storage, and Dits builds on the same foundation (FastCDC + BLAKE3 over a content-addressed store). Where Dits aims to go further is two addressing layers beyond exact-match: similarity-addressing (dedupe near-duplicate samples and re-encodes) and derivation-addressing (store the recipe for a reproducible artifact instead of its bytes). Those are on the roadmap, and clearly labeled as such.

Question 8

How does it compare to DVC?

Accepted Answer

DVC versions data by storing file-level pointers in Git and the bytes in a remote cache — dedup is per-file, not sub-file. Dits dedupes at the chunk level across everything, so a dataset that grows by appending shards, or a model and its quantized variant, share storage automatically. Dits is also a first-class store with integrity-verified reads, rather than a layer that orchestrates Git plus a bucket.

Question 9

What about LakeFS and other 'Git for data' tools?

Accepted Answer

Those give you Git-like branching over object storage, which is genuinely useful for workflow. Dits's difference is the storage substrate itself: content-defined chunking with sub-file dedup and byte-exact, hash-verified reconstruction. Think of LakeFS as versioning the catalog and Dits as versioning the bytes efficiently underneath.

Question 10

Why not just keep everything in S3?

Accepted Answer

You can — Dits can even use S3 as its chunk backend. But plain S3 stores a new object on every write and has no idea that two checkpoints or two dataset snapshots are 99% identical. You pay full storage and full transfer for changes that are a fraction of the bytes, and you get no commits, diffs, or lineage. Dits adds those on top of (or instead of) the bucket.

Question 11

How does it compare to Perforce?

Accepted Answer

Perforce is centralized, proprietary, and priced accordingly; dedup requires explicit setup. Dits is open-source, distributed, content-addressed, and deduplicates automatically across versions. For teams that adopted Perforce only because Git could not handle their binaries, Dits is the open alternative.

Question 12

Why not my cloud provider's bucket versioning?

Accepted Answer

Bucket versioning keeps a full copy per version — it is a safety net, not efficient version control. It cannot diff, cannot show lineage across related artifacts, and bills you for every full copy. Dits stores only the delta and gives you real history.

Question 13

Does chunk dedup actually work on model weights?

Accepted Answer

It depends entirely on how the weights changed. Where it wins big: a base model vs. its fine-tune, a model vs. its quantized or LoRA variant, and appended or partially-edited artifacts — these share large runs of identical bytes. Where plain byte-level chunking struggles is two consecutive training checkpoints, because a gradient step nudges almost every weight a little and the bytes change nearly everywhere.

Question 14

So checkpoints every N steps won't dedupe?

Accepted Answer

Honestly, not well with byte-level chunking alone — diffuse float updates defeat it. You still get full commit history, integrity, and dedup against base models and variants, but consecutive checkpoints are the hard case. Tensor-aware chunking that survives diffuse updates is on the roadmap, and we label it as roadmap everywhere rather than implying it works today.

Question 15

What about safetensors, GGUF, or PyTorch .pt files?

Accepted Answer

They are all just files to the content store — Dits chunks and dedupes them the same way. Formats with stable layout (like safetensors) tend to dedupe better across variants than opaque pickled formats, because identical tensors land in identical byte ranges. Format-aware parsing that understands tensor boundaries directly is the roadmap improvement.

Question 16

Does it help with datasets and shards?

Accepted Answer

Yes — datasets are often the best case. Adding samples, appending shards, or re-snapshotting a set that mostly stayed the same dedupes extremely well, because the unchanged shards produce identical chunks and are stored once. You keep every version of the dataset without paying for it more than once.

Question 17

Does it work for genomics data (BAM/CRAM/FASTQ/VCF)?

Accepted Answer

As universal files, yes — they chunk and dedupe today. Dedup pays off where versions share structure: re-aligning against the same reference, appending samples to a cohort, or re-running a pipeline that changes only part of the output. The honest caveat is the same as elsewhere — a transform that perturbs the whole file (such as re-compressing an entire archive) dedupes poorly, and format-aware parsing for genomic containers is roadmap, not shipped.

Question 18

Geospatial rasters and point clouds (GeoTIFF/COG/LAS)?

Accepted Answer

Tiled and pyramided formats are a good fit: when only some tiles or zoom levels change between versions, the untouched tiles dedupe away. Point-cloud and mosaic updates that append or revise regions also benefit. Whole-file recompression is the weak case, as always.

Question 19

Simulation and HPC output (HDF5/NetCDF/Zarr/Parquet)?

Accepted Answer

Chunked array formats like Zarr and Parquet are well-suited because they are already laid out as independent blocks — appending timesteps or revising a subset of chunks dedupes cleanly. Monolithic HDF5/NetCDF files dedupe well when edits are localized and less well when a write touches the whole file.

Question 20

When does dedup NOT help?

Accepted Answer

When a change perturbs the entire file: re-encoding or re-compressing a whole archive, an encryption pass over everything, or a training step that nudges every float. In those cases you still get history, integrity, and dedup against related artifacts — just not against the immediately-previous full version. We would rather tell you that up front than imply magic.

Question 21

What are the three addressing layers?

Accepted Answer

Three ways to give data an address. L1 Exact: address a chunk by the BLAKE3 hash of its bytes — identical bytes stored once. L2 Similar: address near-duplicates by a perceptual or semantic fingerprint and store a small delta against the closest existing chunk. L3 Derived: address a reproducible artifact by its recipe (data refs + config + seed) and recompute it on demand instead of storing the bytes.

Question 22

Which layers work today?

Accepted Answer

L1, exact content-addressing, ships today on the open engine — chunking, dedup, integrity-verified reads, and byte-exact reconstruction. L2 (similarity) and L3 (derivation) are the roadmap that takes Dits past the exact-match ceiling every other tool is built on. They are labeled as roadmap throughout the site.

Question 23

What is derivation-addressing / recompute?

Accepted Answer

If an artifact is reproducible — a quantized model from a base, a LoRA from (data + config + seed), a cache from a deterministic build — you can store the few-hundred-byte recipe instead of the terabytes of output, and recompute it when needed. It trades storage for compute on artifacts that are cheap to regenerate. This is L3, and it is on the roadmap.

Question 24

How much storage will I actually save?

Accepted Answer

It depends on your edit patterns, and we won't quote a universal number. The mechanism: anything two versions share is stored once. Teams with many variants of a base model, slowly-growing datasets, or appended shards see the largest wins; teams storing only diffuse, unrelated checkpoints see the least. The honest framing is 'you stop paying for the bytes that didn't change.'

Question 25

What's the difference between logical and physical size?

Accepted Answer

Logical size is what you think you have — the sum of every file's size. Physical size is what is actually stored after dedup — the unique chunks. Dits reports both, so you can see your real footprint. For repos with shared structure, physical can be a fraction of logical.

Question 26

Does syncing cost full bandwidth?

Accepted Answer

That is the point of content-addressing: sync transfers only chunks the other side is missing, and resumes where it dropped. Moving a checkpoint between nodes that already share 90% of the bytes uploads only the 10% delta. Note the networked sync engine is in active development; the content-addressed store it builds on works today.

Question 27

Where are the chunks actually stored?

Accepted Answer

Locally by default — Dits is local-first and works fully offline. For remotes, it supports S3-compatible backends (and self-hosted MinIO), with metadata coordinated separately. Your bytes can stay entirely on infrastructure you control.

Question 28

What commands does it use?

Accepted Answer

The Git-familiar set: dits init, add, commit, status, diff, log, branch, checkout, and more. If you can use Git, you can use Dits — the difference is what is underneath, not the interface.

Question 29

How do I version a dataset?

Accepted Answer

Point Dits at the dataset directory, then add and commit it. Each new snapshot commits again; unchanged shards dedupe automatically and only new or changed data is stored. You get a full, diffable history of the dataset and can check out any past version byte-for-byte.

Question 30

Can I hook it into my training loop?

Accepted Answer

Yes — commit checkpoints and dataset versions as part of your pipeline the same way you would script Git. Because commits are content-addressed, each one pins the exact data and artifacts that produced a result, which is what makes a run reproducible later.

Question 31

How do I migrate from DVC or Git LFS?

Accepted Answer

The data models differ (LFS and DVC use pointers; Dits uses chunk manifests), so migration means re-importing files into Dits so they can be chunked. First-class migration tooling is on the roadmap; today you import the files and Dits takes over chunking and dedup from there.

Question 32

Does it track lineage and reproducibility?

Accepted Answer

That is a core reason it exists. Every commit references content by hash, so the history is tamper-evident: a commit tells you exactly which bytes of data and which artifacts existed at that point. 'Which dataset produced this model?' becomes a question with a precise answer instead of a guess.

Question 33

Can I diff two checkpoints or dataset versions?

Accepted Answer

Yes. Dits compares manifests to show which chunks changed between versions — so you can see, at a glance, how much of an artifact actually changed, and track it across runs, variants, and shards rather than just storing more copies.

Question 34

Is there networked sync today?

Accepted Answer

The content-addressed store, local commits, and byte-exact reconstruction work today. Networked delta sync over QUIC (push/pull/fetch across machines) is on the roadmap and in active development — the current network commands are scaffolding, and we say so rather than implying they are done.

Question 35

Multi-node and multi-region?

Accepted Answer

That is the target for the sync engine: move only deltas between nodes, registries, and regions, and resume on drop. The design is content-addressed end to end so the same chunk is never shipped twice. Available once networked sync lands; today the savings are real for local and single-store workflows.

Question 36

Can it back a model or dataset registry?

Accepted Answer

That is a natural fit — a content-addressed store is exactly what a registry wants underneath, so variants and versions share storage. It is part of where the hosted layer is headed; the open engine gives you the substrate today.

Question 37

How does it handle concurrent edits to the same file?

Accepted Answer

Like Git, Dits uses a commit DAG; for binary files that cannot be merged, it supports explicit locking so two people do not clobber the same artifact. Lock a file before editing and others see it as read-only until you are done.

Question 38

How big can a repo or file get?

Accepted Answer

There is no hard file-size limit — the engine streams chunking so memory stays bounded, and files past 100 GB have been tested. Repos scale with your storage; dedup means the physical footprint grows with unique content, not with the number of versions.

Question 39

Is it open source?

Accepted Answer

Yes. Dits is open-core: the CLI, engine, formats, and protocol are open, inspectable, and self-hostable — the same engine behind Dits for media. An optional hosted layer adds managed storage and compute, but everything it does is possible with the self-hosted core.

Question 40

Is it self-hostable? Does my data leave my infrastructure?

Accepted Answer

Fully self-hostable and local-first. You can init, commit, diff, and check out entirely offline; nothing is uploaded unless you configure a remote. Your weights and datasets never have to leave infrastructure you control — which matters for proprietary models and regulated data.

Question 41

Is data encrypted?

Accepted Answer

The engine supports convergent encryption (AES-256-GCM), which preserves dedup while keeping chunks confidential. Full client-side encryption with role-based key management is a later roadmap phase; for now, the most important guarantee is that local-first means data does not leave your machine unless you send it.

Question 42

Can it run air-gapped or meet compliance needs?

Accepted Answer

Local-first and self-hostable makes air-gapped operation straightforward — no account or phone-home is required to use the core. Content addressing also gives tamper-evident history, which helps with audit and provenance requirements. Specific compliance certifications would sit with the hosted offering, not the open core.

Question 43

What does it cost?

Accepted Answer

The open-core engine is free and open-source — run it locally or self-host with no account. The optional hosted layer (managed storage, cloud compute for recompute-heavy workflows, collaboration) is where paid plans live, on the same open protocol so you are never locked in.

Question 44

Is it production-ready?

Accepted Answer

It is early, and honest about it. The local content engine — chunking, dedup, integrity, local history — works today; networked sync, tensor-aware chunking, and the similarity and derivation layers are in progress. Treat it as a strong foundation for early adopters, not a finished hosted product, and check the roadmap before depending on a specific feature.

Question 45

What works today vs. what's on the roadmap?

Accepted Answer

Working today: content-addressed storage with BLAKE3 verification, FastCDC chunking and exact dedup, byte-exact reconstruction, and local commit/checkout/diff/log. Roadmap: networked delta sync, tensor-aware chunking, similarity-addressing (L2), and derivation/recompute (L3). Every roadmap item is labeled as such across the site.

Question 46

How do I follow progress or contribute?

Accepted Answer

Dits is developed in the open on GitHub — follow the repo, open issues, and join discussions. The open-problems list is long and genuinely unsolved, so research, prototypes, and design ideas are as welcome as code.

Question 47

How do I get early access?

Accepted Answer

Use the 'Request early access' link on the AI page, or just clone the open-source engine from GitHub and start locally today — no account needed. Early adopters who share workloads and feedback directly shape the roadmap.

Questions, answered honestly

The basics

How it compares

Does dedup actually work on my data?

The three addressing layers

Storage, bandwidth & cost

Workflow & integration

Sync, scale & teams

Security, privacy & licensing

Status & roadmap

Still have a question?

Read the engine

See the numbers

Dits for media