Skip to content

feat: record statements to a hash-chained access ledger - #4

Open
mickamy wants to merge 14 commits into
mainfrom
feat/ledger
Open

feat: record statements to a hash-chained access ledger#4
mickamy wants to merge 14 commits into
mainfrom
feat/ledger

Conversation

@mickamy

@mickamy mickamy commented Aug 27, 2026

Copy link
Copy Markdown
Owner

Summary

Adds the access ledger: every statement a session handles becomes a record of who ran it, when, what shape it had, what the proxy decided, and how many rows it returned. Records carry no SQL literals and no result values, and are hash-chained so tampering shows. This is the foundation the data-subject index (identifier HMACs) will build on.

What is in here

  • internal/sqlscan: Fingerprint returns a statement with its literals removed (string, dollar-quoted, and numeric literals become ?), so statements of the same shape share a fingerprint and no literal is stored.
  • internal/ledger: the Record, a JSON-lines Sink that hash-chains each record to the previous, and a Guard that wraps another guard to record each session's statements.
  • internal/wire: Principal (agent, purpose, user, database, application), Decision, and the Recorder/Result interfaces. Enforcement now carries the principal and an optional recorder; Session.Frontend takes the recorder.
  • internal/pg: the backend parses RowDescription/DataRow/CommandComplete to feed the result, and the frontend records each decision. Result.Columns/Row are wired so subject capture can be added without touching the protocol code.
  • internal/policy: sets the Principal from the matched role.
  • internal/cli: a -ledger PATH flag that appends records to a JSON-lines file.

Recording model

  • One record per forwarded request. A simple query records its full row count; an extended-protocol statement records its decision at Parse (row counts to follow).
  • The fingerprint, not the SQL, is stored. Result values are never stored.
  • The chain: each record's hash covers its content and the previous hash.

Testing

  • make test and make lint pass.
  • internal/sqlscan: fingerprints mask literals and drop comments while preserving identifiers.
  • internal/ledger: records carry the principal, kind, fingerprint, decision, and row count; the fingerprint holds no literal; hashes chain.
  • internal/pg: a recorder observes a simple query's row count and a denial.
  • End-to-end against postgres:16: allowed and denied statements are recorded with agent, purpose, kind, masked fingerprint, row count, and a linked hash chain; the file contains no literal from the queries (verified by inspecting the output).

Not in this PR

Identifier extraction from result columns into keyed HMACs (the data-subject index), extended-protocol row counts, at-least-once spooling, and the limited/approved decisions.

Review round: accuracy, privacy, and the write path

  • Extended-protocol recording is accurate: statements are recorded at Sync, so a statement rejected mid-batch is no longer recorded as "allowed"; a single-statement extended batch captures its row count. A multi-statement query records each distinct kind (a SELECT 1; DELETE is not a plain SELECT).
  • Escape-string literals no longer leak: the tokenizer and fingerprint handle E'...' with backslash escapes, so E'O\'Brien' does not surface words as identifiers.
  • The ledger is written off the response path: a single writer goroutine drains a queue, so a slow disk never blocks a session's response or serializes sessions against each other. A session that ends before ReadyForQuery finalizes its in-flight records.
  • The chain survives restarts and can be keyed: a new Sink resumes from the file's last hash, and uses an HMAC when ROLLCALL_LEDGER_KEY is set (plain SHA-256 otherwise).
  • No result is materialized when nothing records: RowDescription/DataRow/CommandComplete stream straight through when the session has no recorder.

Verified against postgres:16: simple and extended (\bind) queries record row counts, a mid-batch denial records only the denial, an E'...' literal does not leak, and the chain stays intact across a proxy restart.

Review round: ledger size, key identity, and startup cost

  • The fingerprint is bounded (4 KiB, with a hash suffix when truncated), so a connected agent cannot grow the ledger one large statement at a time, and no record can exceed the tail-read window on the next startup.
  • The chain records its key: each record carries a key_id (a prefix of the key's hash, empty when unkeyed), so a verifier knows which key signed each record and can see where the key changed across restarts.
  • Startup reads only the tail: resuming the chain reads the last window of the file and parses the last complete record, instead of scanning the whole ledger, so a multi-GB ledger still starts quickly. A partial trailing write is skipped.
  • Records are finalized off the session lock: readyForQuery writes the client's ReadyForQuery and releases the lock before finalizing the record, so a full ledger queue never stalls the response path in the other direction. The Sink comment no longer overstates this — records are queued and never dropped, and Write applies backpressure when the queue fills.
  • Numeric literals are fully masked: 0x1F, 1_000, and scientific notation no longer leave fragments as identifiers in the fingerprint.

Covered by unit tests including a CLI-flow test that resumes the chain across a key change. (Docker was unavailable on this run for a full-binary end-to-end; the full pipeline was verified against postgres:16 in the previous rounds.)

Review round: prepared statements and key hardening

  • Prepared-statement executions are now recorded — the essential gap. The extended protocol records one entry per Execute, resolving a re-executed statement's SQL (Bind/Execute with no Parse) through a name→SQL and portal→name map. This is exactly what driver statement caches (pgx, JDBC prepareThreshold, npgsql auto-prepare) produce, so their second-and-later executions are no longer invisible. A prepare-only batch (Parse without Execute) records nothing rather than a phantom rows-0 execution.
  • Oversized simple-query denials are recorded, matching the Parse path.
  • LastHash skips a complete-but-hashless line ({"foo":1}) instead of silently starting a new chain from empty.
  • The key id resists a dictionary hash: it is a domain-separated HMAC of the key, not a plain SHA-256; the README should still call for a high-entropy key.

Verified with unit tests reproducing the exact driver wire pattern (prepare then re-execute with no Parse → two records; prepare-only → none), plus a full-stack CLI test that drives the real proxy with -ledger and reads back two chained records. (Docker was unavailable on this run for a live driver end-to-end; the wire behavior is covered directly.)

Review round: Execute attribution and map growth

  • Result sets are aligned per Execute. Every Execute in a batch keeps a slot in the record queue — a placeholder when its portal cannot be resolved or when an oversized Bind/Execute is forwarded uninspected — so a result set is always credited to its own record and never bleeds into a later batch. The index into the queue is bounded by the batch's own Execute count.
  • Unknown portals no longer fall back to the unnamed statement. Resolution checks that the portal exists before looking up its statement, so an Execute of an unbound portal records nothing instead of the unnamed prepared statement's SQL.
  • The prepared-statement maps are bounded. They are tracked only when a ledger is configured, capped by count and by total SQL bytes, and pruned on Close, so a long-lived connection preparing statements without end cannot exhaust proxy memory.

Covered by unit tests: an unknown-portal Execute is not misattributed; an unrecorded Execute's result does not steal the next record's rows; a statement reused after Close is not recorded. Full suite passes under -race -count=30 on the prepared-statement paths.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant