Skip to main content
D
Dits
Documentation

Performance Tuning

Optimize Dits for maximum performance in your specific use case. Learn about tuning chunking algorithms, storage backends, and network settings.

Chunking Performance

Chunk Size Optimization
Balance between deduplication and overhead

Smaller chunks increase deduplication but add metadata overhead. Larger chunks reduce overhead but may miss deduplication opportunities.

Small files (<1MB)4KB-16KB chunks
Large files (>100MB)64KB-256KB chunks
Video/Media1MB-4MB chunks
Algorithm Selection
Choose the right chunking algorithm
Fixed-size chunking

Fastest, best for static content

Content-defined chunking

Better deduplication, handles insertions well

FastCDC

Recommended default - balanced performance

Storage Backend Tuning

Local Storage Optimization

Filesystem Choices

  • XFS: Best for large files and high concurrency
  • ext4: Good general purpose choice
  • Btrfs: Built-in compression and snapshots
  • ZFS: Maximum data integrity

I/O Settings

  • Use SSD/NVMe for hot data
  • Enable write caching
  • Tune I/O scheduler for workload
  • Consider RAID for throughput

Configuration Examples

High-Performance Configuration

# dits.toml - High performance settings
[chunks]
algorithm = "fastcdc"
min_size = "32KB"
avg_size = "64KB" 
max_size = "256KB"
parallel_workers = 8

[storage]
type = "local"
path = "/fast-ssd/dits"
cache_size = "8GB"
write_buffer = "256MB"

[network]
max_connections = 1000
buffer_size = "4MB"
compression = "lz4"

[cache]
enabled = true
size = "4GB"
eviction = "lru"

Low-Memory Configuration

# dits.toml - Memory-constrained settings
[chunks]
algorithm = "fixed"
size = "64KB"
parallel_workers = 2

[storage]
type = "local"
cache_size = "256MB"
write_buffer = "32MB"

[cache]
enabled = true
size = "512MB"
aggressive_eviction = true

Monitoring Performance

Latency Metrics
  • Chunk read/write time
  • Hash computation time
  • Network round-trip
  • Database query time
Throughput Metrics
  • MB/s upload rate
  • MB/s download rate
  • Chunks processed/sec
  • Deduplication ratio
Resource Usage
  • CPU utilization
  • Memory consumption
  • Disk I/O wait
  • Network bandwidth

Benchmarking

Use the built-in benchmarking tool to measure performance:

# Run comprehensive benchmark
dits benchmark --all

# Test specific operations
dits benchmark --chunking --file large-file.bin
dits benchmark --network --remote origin
dits benchmark --storage --size 10GB