BitVault Storage Subsystem — Design

Audience: distributed-systems / storage engineers. Scope: the deep design of BitVault’s storage subsystem — the part that turns bytes into durable, deduplicated, content-addressed, verifiable, tiered storage across many cloud providers, at millions of files / billions of chunks / PB scale.

This set refines and elaborates the system-level decisions already recorded: ADR-0004 Postgres as truth, ADR-0005 storage abstraction, ADR-0011 presigned data plane, 04 bounded contexts, 05 service boundaries, 08 data model. Where this set deepens the high-level BLOB abstraction into Object → Chunk → Pack, that refinement is called out explicitly.

No implementation code — architecture and documentation only. Every decision states Tradeoffs / Alternatives / Scaling; the discrete contested choices are crystallized as ADRs 0016–0021.

0. Reading order

1. Design tenets (the rules everything else obeys)

1. Metadata is truth; bytes are replaceable. Postgres records what exists and where; object storage holds opaque, immutable, content-addressed bytes. A byte object with no metadata reference does not exist and is reclaimable (ADR-0004).
2. Immutability by content address. Stored units are named by the hash of their content. Identical content has one name, is written once, and never mutated — only referenced, dereferenced, or copied. This is what makes dedup, caching, integrity, and safe concurrency tractable.
3. The data plane never flows through the control plane. User bytes move directly between client and object storage via scoped presigned URLs (ADR-0011). Our compute touches bytes only for asynchronous maintenance (packing, scrubbing, tier migration), never on the user’s hot path.
4. Separate the dedup unit from the transfer unit from the storage unit. These are three different sizes for three different reasons (see §3). Conflating them is the most common storage-design mistake.
5. Everything derivable is rebuildable. Indexes, caches, and placement maps are reconstructible from manifests + object listings. Only the chunk bytes and the manifests are irreplaceable.
6. Tenant isolation is a storage boundary, not just a DB boundary. Dedup, placement, and keys are tenant-scoped by default (ADR-0007, ADR-0018) — closing the cross-tenant dedup side channel (Harnik 2010).
7. Design for the long tail. Millions of files means billions of small chunks. Per-object overhead, request cost, and index cardinality — not raw capacity — are the binding constraints. Pack small things into big things.

2. The layered data model (terminology — used everywhere)

A file’s bytes pass through four named layers. Get this vocabulary straight or the rest of the docs are ambiguous.

flowchart TB
    classDef ns fill:#dbeafe,stroke:#1e40af,color:#111827;
    classDef log fill:#fde68a,stroke:#b45309,color:#111827;
    classDef phys fill:#bbf7d0,stroke:#15803d,color:#111827;

    subgraph NS["Namespace context (File & Metadata) — see 08-data-model"]
      node["Node (file/folder)"]:::ns
      ver["Version"]:::ns
    end

    subgraph LOGICAL["Storage logical model (content-addressed)"]
      obj["Object = content of one version<br/>id = content_hash (BLAKE3 root)"]:::log
      man["Manifest = ordered list of chunk refs<br/>{chunk_hash, offset, length}"]:::log
      chunk["Chunk = unit of dedup & CAS<br/>id = chunk_hash (BLAKE3), variable size (CDC)"]:::log
    end

    subgraph PHYSICAL["Storage physical model (provider objects)"]
      pack["Pack = large container object<br/>(~256MiB–1GiB) aggregating many chunks"]:::phys
      pobj["Provider object<br/>(MinIO/S3/R2/GCS/Azure key)"]:::phys
    end

    node --> ver --> obj --> man --> chunk
    chunk -->|"packed (long tail)"| pack --> pobj
    chunk -->|"standalone (hot/large)"| pobj

Refinement note: 08-data-model modeled a single content-addressed BLOB. The storage subsystem refines BLOB into Object (manifest) + Chunks + Packs. A whole-file-only deployment (no chunking) is the degenerate case where each Object = one Chunk = one provider object — supported as a config profile for small self-host installs.

3. Three sizes, three reasons (the tenet that prevents the classic mistake)

A 64 MiB multipart part may contain dozens of chunks; a 1 GiB pack aggregates hundreds. These layers move independently. (Grounding: Dropbox stores 4 MiB blocks aggregated into 1 GiB buckets; restic packs CDC chunks into pack files.)

4. Internal service boundaries

The Storage bounded context (05) decomposes into these components. In v1 they are modules inside bitvaultd (ADR-0001); the workers are the first extraction candidates because their load profile (bulk, bursty, CPU/IO-bound) differs sharply from the control-plane API.

flowchart TB
    classDef api fill:#c7d2fe,stroke:#3730a3,color:#111827;
    classDef idx fill:#fde68a,stroke:#b45309,color:#111827;
    classDef wk fill:#fed7aa,stroke:#c2410c,color:#111827;
    classDef store fill:#bbf7d0,stroke:#15803d,color:#111827;
    classDef ext fill:#e5e7eb,stroke:#6b7280,color:#111827;

    client["Client (smart/simple)"]:::ext

    subgraph CP["Control plane (sync, stateless)"]
      coord["Storage Coordinator API (gRPC)<br/>init · negotiate chunks · presign · commit · resolve"]:::api
      place["Placement Service<br/>provider/region/tier/bucket policy"]:::api
      adpt["Provider Adapters<br/>MinIO·S3·R2·GCS·Azure (capability-flagged)"]:::api
    end

    subgraph META["Storage metadata (Postgres, sharded by tenant)"]
      cidx[("Chunk Index<br/>chunk_hash → refcount, location, size, ckSum, tier, state")]:::idx
      mstore[("Manifest Store<br/>content_hash → ordered chunk refs")]:::idx
      pidx[("Pack Index<br/>pack_id → provider object, members")]:::idx
    end

    subgraph WK["Async maintenance workers (extractable)"]
      packer["Packer / Compactor"]:::wk
      gc["Garbage Collector"]:::wk
      scrub["Integrity Scrubber"]:::wk
      life["Lifecycle Engine"]:::wk
      mig["Tier / Migration Worker"]:::wk
    end

    subgraph DP["Data plane (bytes; bypasses compute)"]
      obj[("Object Storage (per provider)<br/>chunk objects · pack objects")]:::store
    end

    client -->|"REST/gRPC: control"| coord
    client -. "presigned PUT/GET: bytes" .-> obj
    coord --> cidx & mstore & pidx
    coord --> place --> adpt --> obj
    packer --> cidx & pidx & obj
    gc --> cidx & mstore & pidx & obj
    scrub --> cidx & obj
    life --> mstore & cidx
    mig --> place & cidx & pidx & obj

Component responsibilities

5. Data ownership (one writer per datum — the rule that enables extraction)

Critical ownership nuance: the Chunk Index row has multiple legitimate writers (create, relocate, delete) at different lifecycle stages. We serialize them through an explicit per-chunk state machine (writing → committed → packed → unreferenced → deleting) guarded by compare-and-swap on the row, not by “one service owns the table.” This is the single most concurrency-sensitive datum in the subsystem and is treated as such in 11.

6. Key invariants (correctness — tested, not hoped)

• SI-1 (no orphan reference): a manifest is committed only after all its chunks are durably present and checksum-verified (05).
• SI-2 (no premature delete): a chunk’s bytes are deleted only after it is unreferenced by every manifest and a grace period has elapsed — closing the dedup-vs-GC race (11).
• SI-3 (content address is honest): stored bytes always hash to their name; violated bytes are quarantined and repaired (04).
• SI-4 (tenant containment): a chunk is only ever referenced by manifests in the tenant that created it (per-tenant dedup) (03, ADR-0018).
• SI-5 (rebuildable indexes): the Pack/Chunk location indexes can be rebuilt by scanning manifests + provider listings; loss of an index is an availability event, not a durability event.

Related ADRs

Plus inherited: 0004, 0005, 0011, 0007, 0014.

References (research grounding)

• FastCDC (USENIX ATC ‘16) — content-defined chunking: https://www.usenix.org/system/files/conference/atc16/atc16-paper-xia.pdf
• BLAKE3 — parallel Merkle-tree hash & verified streaming: https://github.com/BLAKE3-team/BLAKE3
• Harnik, Pinkas, Shulman-Peleg, “Side Channels in Cloud Services: Deduplication in Cloud Storage” (2010): https://eprint.iacr.org/2016/977.pdf
• AWS S3 multipart upload limits: https://docs.aws.amazon.com/AmazonS3/latest/userguide/qfacts.html
• tus resumable upload protocol: https://tus.io/protocols/resumable-upload
• restic prune (mark-sweep over CDC chunks): https://restic.readthedocs.io/en/stable/060_forget.html
• Dropbox Magic Pocket (content-addressed blocks + 1 GiB buckets): https://dropbox.tech/infrastructure/inside-the-magic-pocket