Skip to main content
D
Dits
Documentation

Dits Wire Protocol

The Dits wire protocol handles efficient, resumable transfer of chunks and metadata over QUIC, with built-in integrity verification and compression.

Overview

The Dits wire protocol is a binary protocol optimized for transferring large binary chunks and metadata between Dits clients and servers. Built on QUIC for reliable, multiplexed transport, it features:

  • Content-defined chunking - Variable-size chunks based on content patterns
  • BLAKE3 integrity - Cryptographic verification of all data
  • Resumable transfers - Continue interrupted uploads/downloads
  • Compression - Optional zstd compression for metadata
  • Multiplexing - Concurrent chunk transfers over single connection

Transport Layer

QUIC Transport
Modern UDP-based transport protocol
Advantages:
  • Built-in multiplexing (streams)
  • Connection migration
  • Forward error correction
  • TLS 1.3 encryption
Security Features
End-to-end encryption and verification
Built-in:
  • TLS 1.3 encryption
  • Content integrity (BLAKE3)
  • Authentication (JWT/bearer tokens)
  • Optional end-to-end encryption

Message Framing

All wire protocol messages follow this binary frame structure:

Magic (4 bytes)
DITS
Version (1 byte)
0x01
Type (1 byte)
0x01-0xFF
Flags (1 byte)
Bitfield
Length (4 bytes, big-endian)
Payload size
Payload (variable)
Message data

Message Types

TypeNameDescriptionDirection
0x01HELLOProtocol handshake and version negotiationBidirectional
0x02AUTHAuthentication request/responseBidirectional
0x10HAVE_WANTBloom filter sync for chunk discoveryBidirectional
0x11CHUNK_REQUESTRequest specific chunks by hashClient → Server
0x12CHUNK_DATAChunk data with integrity verificationServer → Client
0x13CHUNK_UPLOADUpload chunk to serverClient → Server
0x20MANIFEST_PUSHPush manifest (file metadata)Client → Server
0x21MANIFEST_PULLRequest manifest by commit hashClient → Server
0x30LOCK_ACQUIRERequest file lockClient → Server
0x31LOCK_RELEASERelease file lockClient → Server
0xFFERRORError response with detailsBidirectional

Flags

Message flags are encoded as a bitfield in the flags byte:

BitFlagDescription
0COMPRESSEDPayload is zstd-compressed
1ENCRYPTEDPayload is additionally encrypted
2STREAMMessage is part of a stream
3FINALLast message in stream
4PRIORITY_HIGHHigh priority message
5-7RESERVEDReserved for future use

Chunk Transfer Protocol

Have/Want Sync

Dits uses Bloom filters to efficiently determine which chunks need to be transferred. This avoids sending lists of thousands of chunk hashes.

Bloom Filter Sync Flow:

Client creates Bloom filter from local chunk hashes (~1KB)
Server checks which chunks client probably has
Server sends list of chunks to upload/download
Only missing chunks are transferred

Resumable Transfers

Large chunk transfers can be resumed after network interruptions. The protocol tracks progress and can restart from any byte offset.

Connection Management

Connection Parameters
Max concurrent streams:
100
Stream data limit:
16 MB
Connection timeout:
30s idle
Keep-alive interval:
10s
Performance Tuning
Chunk size (avg):
64-256 KB
Compression:
zstd level 3
Parallel transfers:
Up to 32 streams
Bloom filter size:
1-4 KB

Implementation Considerations

Error Handling

All protocol errors include:

  • Error code: Machine-readable error identifier
  • Message: Human-readable error description
  • Context: Additional error context (chunk hash, etc.)
  • Retry advice: Whether the operation can be retried

Security Considerations

  • Transport encryption: All traffic encrypted with TLS 1.3
  • Content verification: BLAKE3 hashes prevent tampering
  • Authentication: Bearer tokens for session management
  • Rate limiting: Built-in protection against abuse

Protocol Benefits Summary

Efficiency

  • Minimal bandwidth overhead
  • Resumable large file transfers
  • Parallel chunk streaming
  • Compression for metadata

Reliability

  • Content integrity verification
  • Automatic error recovery
  • Connection migration support
  • Forward compatibility

Reference Implementation

The reference implementation is available in the Dits CLI codebase. See the protocol architecture docs for detailed implementation notes and the source code for examples.