04 — Integrity & Checksums

Topics: checksums, data integrity verification. The promise behind a storage product: the bytes you get back are exactly the bytes you put in — forever, across five providers, despite bitrot, truncation, and provider bugs.

Integrity in BitVault is defense in depth: the content hash is the spine, and checksums are verified at every hop so a corruption is caught at the earliest possible point and attributed to the right layer.

1. The integrity stack (verify at every hop)

flowchart LR
    classDef step fill:#fde68a,stroke:#b45309,color:#111827;
    c["Client<br/>computes chunk_hash"]:::step --> t["In transit<br/>TLS + provider PUT checksum"]:::step
    t --> commit["On commit<br/>server Head + verify size/checksum"]:::step
    commit --> rest["At rest<br/>name == BLAKE3(content)"]:::step
    rest --> scrub["Background scrub<br/>periodic read-verify"]:::step
    scrub --> dl["On download<br/>client re-verifies hash"]:::step
Hop Mechanism Catches
In transit (up) TLS + provider checksum on PUT (CRC32C/SHA-256 where available) network corruption, truncated upload
On commit server Head (size) + checksum match before manifest write (SI-1) missing/short/garbled chunk before it becomes referencable
At rest the object’s name is its BLAKE3 hash any later mutation/bitrot is detectable by recompute
Background scrubber read-verifies on a schedule silent bitrot, provider-side loss/corruption, missing objects
On download client recomputes hash over received bytes corruption anywhere downstream incl. cache/CDN

End-to-end property: the client both produces and verifies the hash, so no intermediate layer (our compute, the provider, a CDN) is trusted to be correct — it is checked.

2. Checksums: ours + the provider’s (belt and suspenders)

Why both: our hash proves content; the provider checksum lets the provider reject bad transfers and lets us cheaply audit via Head/metadata without egress.

Verified streaming (BLAKE3 superpower)

Because BLAKE3 is a Merkle tree internally, we can verify a byte range without the whole chunk. This makes range downloads (06) and partial scrubbing verifiable, and lets the scrubber sample-verify large packs efficiently. SHA-256 (compliance mode) loses this; there we verify whole chunks.

3. The scrubber (background data integrity verification)

A continuous worker that reads stored bytes and confirms they still hash correctly — the only defense against silent corruption (bitrot, firmware bugs, provider incidents) that no request-path check would ever notice.

4. Durability & repair

BitVault leans on provider durability (S3-class ≈ 11 nines) as the baseline and adds detection + optional cross-provider redundancy on top.

Redundancy options (policy-driven, 09)

| Mode | How | Durability | Cost | |—|—|—|—| | Single provider (default) | rely on provider’s internal erasure coding | provider’s (≈11 nines) | 1× | | Cross-provider replication | important data mirrored to a 2nd provider/region | survives a whole-provider loss | ~2× + egress | | Erasure across providers | (advanced) split with parity across providers | high, lower overhead than 2× | complex; deferred |

We do not reimplement intra-datacenter erasure coding — providers already do it far better than we could. Our value-add is detection (scrub) + policy- driven cross-provider redundancy for data that warrants it, plus the content hash that makes any copy verifiable.

Repair flow

flowchart TB
    classDef bad fill:#fecaca,stroke:#b91c1c,color:#111827;
    classDef ok fill:#bbf7d0,stroke:#15803d,color:#111827;
    d["Scrub/Download detects<br/>hash mismatch or missing"]:::bad --> q["Quarantine chunk<br/>mark suspect in index"]:::bad
    q --> f{"another good copy?<br/>(replica / other provider / other tier)"}
    f -- yes --> r["Re-copy good bytes →<br/>rewrite object, clear suspect"]:::ok
    f -- no --> l["Mark chunk LOST<br/>flag affected manifests/versions degraded<br/>alert + audit"]:::bad

5. Tradeoffs / Alternatives / Scaling

Tradeoffs. Verifying at every hop costs CPU (hashing) and IO (scrub egress). We accept ingest hashing (cheap with BLAKE3) and budget/throttle scrubbing. Skipping scrub would save cost but reintroduce silent-bitrot risk — unacceptable for a storage product.

Alternatives.

Scaling.

References