System Overview

BitVault v1 ships as a modular monolith: one bitvaultd binary containing all logical services as in-process modules. The internal seams are drawn exactly where future service boundaries would be cut — extraction is a deployment change, not a rewrite (ADR-0001). The diagrams on this page show the logical architecture, which is identical whether deployed as one process or many.

See Service Boundaries for per-module ownership detail and Data Model for the entity schema.

System Context (C4 Level 1)

Who and what interacts with BitVault, and what external systems it depends on.

flowchart TB
    classDef person fill:#dbeafe,stroke:#1e40af,color:#111827;
    classDef system fill:#fde68a,stroke:#b45309,color:#111827;
    classDef ext fill:#e5e7eb,stroke:#6b7280,color:#111827;
    user["End User\n(web / mobile)"]:::person
    dev["Developer / Operator\n(CLI, automation)"]:::person
    admin["Tenant Admin"]:::person
    bv["BitVault Platform\nfile storage · sync · sharing · search"]:::system
    obj["Object Storage\nMinIO / S3 / R2 / GCS / Azure"]:::ext
    idp["External IdP\n(OIDC / SSO)"]:::ext
    smtp["Email / SMTP"]:::ext
    hook["Tenant Webhook Endpoints"]:::ext
    user -->|"REST / HTTPS"| bv
    dev -->|"REST + gRPC (CLI)"| bv
    admin -->|"REST / HTTPS"| bv
    bv -->|"presigned PUT/GET"| obj
    bv -->|"authn"| idp
    bv -->|"notifications"| smtp
    bv -->|"signed events"| hook
    user <-. "direct byte transfer (presigned)" .-> obj

The dashed line is load-bearing: users transfer bytes directly to and from object storage. BitVault issues the presigned URL but the payload never traverses its compute (ADR-0011).

Container View (C4 Level 2)

Internal building blocks and the stores they own. All boxes are modules in bitvaultd in v1.

flowchart TB
    classDef edge fill:#c7d2fe,stroke:#3730a3,color:#111827;
    classDef svc fill:#fde68a,stroke:#b45309,color:#111827;
    classDef worker fill:#fed7aa,stroke:#c2410c,color:#111827;
    classDef store fill:#bbf7d0,stroke:#15803d,color:#111827;
    classDef bus fill:#fbcfe8,stroke:#be185d,color:#111827;

    subgraph clients[Clients]
        web["Next.js Web"]:::edge
        cli["Go CLI"]:::edge
        mob["React Native\n(future)"]:::edge
    end

    gw["API Gateway / BFF\nREST edge · authn · rate-limit · REST↔gRPC"]:::edge

    subgraph control[Control plane services - gRPC]
        id["Identity"]:::svc
        fm["File & Metadata\n(source of truth + outbox)"]:::svc
        st["Storage\n(presign · multipart · commit)"]:::svc
        sh["Sharing"]:::svc
        sy["Sync"]:::svc
        srch["Search (query)"]:::svc
    end

    subgraph workers[Async workers]
        idx["Search Indexer"]:::worker
        ntf["Notifier"]:::worker
        gc["Finalizer / GC"]:::worker
        mtr["Meter"]:::worker
        aud["Audit Sink"]:::worker
    end

    bus{{"NATS JetStream\n(in-proc bus in v1)"}}:::bus

    pg[("PostgreSQL\nmetadata · outbox")]:::store
    rd[("Redis\ncache · sessions · locks · rate-limit")]:::store
    os[("OpenSearch\n(optional; PG-FTS fallback)")]:::store
    obj[("Object Storage\nblobs")]:::store

    web & cli & mob --> gw
    gw --> id & fm & st & sh & sy & srch

    id --> pg
    fm --> pg
    sh --> pg
    sy --> pg
    st --> obj
    id & gw --> rd
    srch --> os

    fm -->|outbox→publish| bus
    st -->|outbox→publish| bus
    sh -->|outbox→publish| bus
    bus --> idx & ntf & gc & mtr & aud & sy
    idx --> os
    gc --> obj
    mtr --> pg
    aud --> pg

    web -. presigned .-> obj
    cli -. presigned .-> obj

Layers

Clients — Next.js web, Go CLI, future React Native mobile, and third-party integrations. All speak REST to the gateway and transfer bytes directly to object storage.
Edge / Gateway — TLS termination, authn, per-tenant rate limiting, REST↔gRPC translation, BFF aggregation. Stateless; the primary horizontal-scale unit.
Control-plane services (gRPC) — Identity, File & Metadata, Storage, Sharing, Sync, Search-query. Strong consistency; each owns its Postgres tables.
Event backbone — transactional outbox → NATS JetStream (in-proc bus in v1). The published language between core and consumers.
Async workers — indexer, notifier, GC/finalizer, meter, audit. Eventually consistent; independently scalable; failures isolated from the control plane.
Data stores — Postgres (truth + outbox), Redis (cache/sessions/locks/rate-limit), OpenSearch (optional derived index), object storage (blobs).

The Three Planes

Plane	Consistency	Components	Scales with
Control	Strong, synchronous	Gateway, Identity, File & Metadata, Sharing, Storage presign/commit	Read replicas + stateless replicas
Data	High-throughput, compute-bypass	Client ⇄ Object Store via presigned URLs	Object store, independently of control
Async / Derivation	Eventual	Sync projector, Search indexer, Notifier, Meter, Audit, GC	Per-worker; failures never block control

:::tip Data Plane Scalability Bytes never traverse BitVault compute. The presigned URL model means the data plane scales with the object store, independently of the control plane. :::

Upload Flow

The commit protocol is the correctness heart of the system. It defeats the dual-write problem: the namespace row and its event are written in one transaction, and bytes are verified before commit.

sequenceDiagram
    autonumber
    participant C as Client
    participant GW as API Gateway
    participant F as File & Metadata
    participant S as Storage
    participant O as Object Store
    participant B as Event Bus
    participant IX as Indexer

    C->>GW: POST /v1/files (init upload: path, size, hash)
    GW->>F: CreateUpload(node draft) [gRPC]
    F->>S: PresignPut(staging key, size-range, ttl)
    S-->>F: presigned URL (+ uploadId if multipart)
    F-->>GW: uploadId + presigned URL(s)
    GW-->>C: 201 {uploadId, url}

    C->>O: PUT bytes (direct, presigned)
    O-->>C: 200 (ETag)

    C->>GW: POST /v1/files/{uploadId}/commit (etag, hash)
    GW->>F: CommitUpload(uploadId, hash)
    F->>S: HeadObject(staging key) — verify size/etag/hash
    S->>O: HEAD staging key
    O-->>S: metadata (size, etag)
    S-->>F: verified ✔ (+ blob refcount++)
    Note over F: BEGIN TX<br/>insert/replace node version<br/>insert outbox(NodeChanged)<br/>COMMIT
    F-->>GW: 200 (node, version)
    GW-->>C: 200 committed

    F->>B: publish NodeChanged (from outbox)
    B->>IX: NodeChanged
    IX->>IX: index name+metadata (eventual)
    Note over C,IX: UI may show "indexing…" until IX catches up

Failure handling:

Client uploads bytes but never commits → staging blob has refcount = 0 → GC reclaims it after TTL. No orphan, no dangling reference.
Commit fails verification (hash/size mismatch) → no metadata written → safe retry.
Outbox decouples publish from the transaction → at-least-once delivery; consumers are idempotent (ADR-0006).

Download Flow

Authz happens before URL issuance; the URL is scoped to an exact key with a short TTL. Bytes never touch BitVault compute.

sequenceDiagram
    autonumber
    participant C as Client
    participant GW as API Gateway
    participant SH as Sharing
    participant F as File & Metadata
    participant S as Storage
    participant O as Object Store

    C->>GW: GET /v1/files/{id}/content
    GW->>SH: CheckAccess(principal, node, read)
    SH-->>GW: allow
    GW->>F: ResolveVersion(node) → blob hash/key
    F->>S: PresignGet(key, ttl, response-headers)
    S-->>F: presigned GET URL
    F-->>GW: redirect URL
    GW-->>C: 302 (or {url})
    C->>O: GET bytes (direct, presigned)
    O-->>C: 200 bytes

Sync Flow

The sync journal is a per-tenant monotonic sequence of CHANGE rows. Devices hold a cursor; delta pull is WHERE seq > cursor. A stale base version never overwrites — it becomes a conflicted copy. Both histories survive.

sequenceDiagram
    autonumber
    participant D as Device
    participant GW as API Gateway
    participant SY as Sync
    participant F as File & Metadata
    participant S as Storage

    Note over SY: Sync projects NodeChanged events into a<br/>per-tenant monotonic change journal
    D->>GW: GET /v1/sync/changes?cursor=N
    GW->>SY: PullDeltas(cursor=N)
    SY-->>GW: changes[N+1..M] + new cursor M
    GW-->>D: deltas (created/updated/moved/deleted)

    loop for each changed file to fetch
        D->>GW: GET content URL (download flow above)
    end

    Note over D: local edit while offline →
    D->>GW: POST /v1/sync/push (node, baseVersion, hash)
    GW->>SY: Push(node, baseVersion)
    SY->>F: CompareAndCommit(baseVersion)
    alt baseVersion is current
        F-->>SY: committed new version
        SY-->>D: ok (no conflict)
    else baseVersion stale (concurrent change)
        F-->>SY: conflict (current != base)
        SY->>F: CreateConflictedCopy(node, device, ts)
        SY-->>D: conflict resolved as copy (both versions kept)
    end

See ADR-0008, ADR-0022, and ADR-0024 for the full conflict and cursor model.

Event Flow

All producers write to the outbox in the same transaction as the aggregate change. The outbox drainer publishes to NATS JetStream. All consumers are idempotent (dedup on event id), per-aggregate ordered, with DLQ for poison messages. Derived stores are rebuildable from the journal.

flowchart LR
    classDef src fill:#fde68a,stroke:#b45309,color:#111827;
    classDef bus fill:#fbcfe8,stroke:#be185d,color:#111827;
    classDef cons fill:#fed7aa,stroke:#c2410c,color:#111827;
    classDef sink fill:#bbf7d0,stroke:#15803d,color:#111827;

    FM["File & Metadata\n(outbox)"]:::src
    ST["Storage\n(outbox)"]:::src
    SH["Sharing\n(outbox)"]:::src

    J{{"NATS JetStream\nsubjects: node.*, blob.*, share.*"}}:::bus

    SY["Sync projector"]:::cons
    IX["Search indexer"]:::cons
    NT["Notifier"]:::cons
    MT["Meter"]:::cons
    AU["Audit"]:::cons
    GC["GC / finalizer"]:::cons

    OS[("OpenSearch")]:::sink
    PG[("Postgres: journal/meters/audit")]:::sink
    HK[("Webhooks / Email")]:::sink
    OB[("Object Store")]:::sink

    FM & ST & SH -->|"at-least-once"| J
    J --> SY --> PG
    J --> IX --> OS
    J --> NT --> HK
    J --> MT --> PG
    J --> AU --> PG
    J --> GC --> OB

Next: Data Model · Service Boundaries