File & Metadata

Purpose

File & Metadata is the spine of BitVault — the source of truth for the entire file namespace. It owns the commit protocol that defeats dual-write: a namespace mutation and its event are one atomic transaction. No version can exist without verified bytes; no event can be missed without a visible gap.

Data owned

Internal API

Files.* gRPC methods:

Commit protocol

The commit protocol is the central correctness invariant of BitVault. It ensures bytes are verified before any metadata is written, and that the event is never lost.

sequenceDiagram
    autonumber
    participant C as Client
    participant GW as API Gateway
    participant F as File & Metadata
    participant S as Storage
    participant O as Object Store
    participant B as Event Bus
    participant IX as Indexer

    C->>GW: POST /v1/files (init: path, size, hash)
    GW->>F: CreateUpload(node draft) [gRPC]
    F->>S: PresignPut(staging key, size-range, ttl)
    S-->>F: presigned URL (+ uploadId if multipart)
    F-->>GW: uploadId + presigned URL(s)
    GW-->>C: 201 {uploadId, url}

    C->>O: PUT bytes (direct, presigned)
    O-->>C: 200 (ETag)

    C->>GW: POST /v1/files/{uploadId}/commit (etag, hash)
    GW->>F: CommitUpload(uploadId, hash)
    F->>S: HeadObject(staging key) — verify size/etag/hash
    S->>O: HEAD staging key
    O-->>S: metadata (size, etag)
    S-->>F: verified ✔ (blob refcount++)
    Note over F: BEGIN TX<br/>insert/replace node + version<br/>insert outbox(NodeChanged)<br/>COMMIT
    F-->>GW: 200 (node, version)
    GW-->>C: 200 committed

    F->>B: publish NodeChanged (drained from outbox)
    B->>IX: NodeChanged → index (eventual)

Failure paths:

• Client uploads bytes but never commits → staging blob has refcount = 0 → GC reclaims it after TTL.
• Commit fails verification (hash/size mismatch) → no metadata written → client retries safely.
• Transaction fails after verification → staging blob is rolled back from committed to staging → GC reclaims it.
• Outbox decouples publish from the transaction → at-least-once delivery; consumers are idempotent.

:::danger Invariant I1: A VERSION row must only be created after the BLOB’s bytes have been verified (size / etag / hash match). Never commit without verification. A VERSION that references unverified bytes breaks content integrity, dedup accounting, and GC safety. :::

Node model

• Types: file or folder. Folders contain nodes; files hold versions.
• Materialized path: each node stores the full path string to enable efficient subtree queries (WHERE path LIKE '/tenant/a/b/%') without recursive CTEs.
• Namespace ops are byte-free: move, rename, and copy mutate NODE rows (and path), never BLOB objects. Copy creates a new NODE/VERSION referencing the same BLOB with refcount++.

Versioning

Each CommitUpload creates a new VERSION row referencing the content-addressed BLOB. NODE.current_version_id points to the head version. The full version history is accessible via Files.GetVersions. Deleting a version decrements the BLOB’s refcount; GC reclaims the object when refcount reaches zero.

Trash and restoration

Trash is a soft-delete: NODE.deleted_at is set. The row is retained so restore is a metadata-only operation. Hard purge removes the row and triggers BLOB refcount decrements for all associated versions.

Events emitted