Full Run: Charon + Lethe Memory Git

This document walks one complete run of the system that was built: from a cold machine to deployed services, generated credentials, three agent roles, a reviewed merge into protected shared memory, and the operational drills (restart, rotation, backup) that prove the guarantees.

It is also the release smoke-test script. Every phase ends with an expected result — if any phase deviates, stop and investigate before continuing.

What was built

• Lethe (git mode) — the canonical, versioned memory store. Semantic changesets, refs with compare-and-swap, protected refs, context projection with manifests, conflict records. Every changeset is validated per operation (memory_git/v1 contract) before it becomes immutable history.
• Charon — the MCP authorization and governance gateway in front of Lethe. Principals with scoped grants, owned refs, proposals, independent review, protected merges authorized by an expiring single-use signed envelope (memory-git-merge/v2), fail-closed audit ledger, OAuth (S256 PKCE) and Obol credentials.

The guarantees the run below exercises:

• exactly-once writes (idempotency-keyed changesets, digest-bound replays);
• no malformed operation ever enters immutable history (validation at Charon and again at Lethe);
• no principal reviews or merges its own proposal;
• protected refs move only through a signed, expiring, single-use, proposal-digest-bound authorization — replays are rejected even if the ref cycles back to the same head;
• every mutation is ledgered; consumed merge nonces survive restarts;
• coordinated backup and verified restore of both databases.

Topology and ports

Both Charon gateways share one policy database (charon.db in the charon-data named volume): principals, grants, proposals, findings, Obol digests, idempotency records, and the audit ledger. Lethe Git owns changesets, refs, accepted memory, manifests, and conflicts in its own SQLite database on a host bind mount.

Roles

Three roles cover everything an agent (or operator) does with this system. The role skills under skills/ are the per-role playbooks; this document is the shared narrative.

A fourth profile, readonly (reader), gets memory.read, memory.search, thread.read and is provisioned by the maintainer for consumers that must never write.

Separation of duties is enforced in code, not by convention: a proposal originator cannot review (denied: self review) or merge (denied: self approval) its own proposal, wildcard project grants are read-only, and refs/shared/* rejects direct commits from everyone.

Credentials: what exists, who makes it, what a restart does

The operator generates credentials for both services at bootstrap. Nothing else in the system can mint authority.

So: a container restart gives you a new browser pairing key by design — and only that. API keys, HMAC keys, Obols, and OAuth access tokens all persist. You never need to regenerate the deployment after a restart; you only need the new pairing key the next time a browser OAuth flow is run.

Phase 0 — Prerequisites

• Docker with Compose v2, openssl, and both repositories checked out side by side (lethe/ and charon/).
• Nothing from a previous experiment may reuse these data directories. Never point Lethe Git at the OpenLethe data directory.

Phase 1 — Start Lethe Git (maintainer)

In the lethe repository:

sh scripts/prepare-local-memory-git-env.sh   # writes .env.git (mode 0600); refuses to overwrite
docker compose -f docker-compose.git.yml --env-file .env.git up -d

The script generates LETHE_API_KEY and CHARON_MERGE_HMAC_KEY (openssl rand -hex 32 each) without printing them. The compose stack binds 127.0.0.1:18485 and fails closed if either value is missing. The data directory must be owned by UID 1000 (the container user); Lethe refuses to run against a group/other-accessible database directory.

Verify (a 401 proves the HTTP stack is up and the key is enforced):

curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:18485/api/health
# expect: 401
set -a; . ./.env.git; set +a
curl -s -H "Authorization: Bearer $LETHE_API_KEY" http://127.0.0.1:18485/api/health
# expect: {"status":"ok"} (or equivalent 200)

Expected result: container lethe-git-local healthy; unauthenticated request rejected; keyed request succeeds.

Phase 2 — Bootstrap Charon (maintainer)

In the charon repository, with the Lethe values present in the environment:

set -a; . ../lethe/.env.git; set +a          # provides LETHE_API_KEY + CHARON_MERGE_HMAC_KEY
export CHARON_OAUTH_REDIRECT_URIS="https://chatgpt.com/connector/oauth/*"   # required in OAuth mode
./setup.sh

setup.sh (no secret value is ever printed):

1. writes a mode-0600 .env with freshly generated CHARON_OBOL_HMAC_KEY, CHARON_OAUTH_HMAC_KEY, and a pairing-secret placeholder, carrying over LETHE_API_KEY and CHARON_MERGE_HMAC_KEY from the environment;
2. builds the hardened image (non-root UID 10001, cap_drop: ALL, read-only root filesystem);
3. reconciles three principals for the project — Local Memory Author (propose), Local Memory Reviewer (review), Local Memory Reader (readonly) — and records their IDs in secrets/principal-ids.txt;
4. sets CHARON_OAUTH_DEFAULT_USER to the author principal;
5. mints 7-day Obols into mode-0600 files under secrets/;
6. launches both gateways via start.sh.

Verify:

docker compose ps                                  # both services healthy
curl -s http://127.0.0.1:18484/livez               # primary (OAuth) gateway
curl -s http://127.0.0.1:18486/livez               # reviewer (Obol) gateway
docker compose logs charon 2>&1 | grep -i -A6 'authorization'   # the current pairing-key banner

Expected result: two healthy gateways; .env and secrets/ all mode 0600; a current pairing key visible in the primary gateway's startup banner.

Minting working local credentials (important)

Obols are audience-bound: a token minted against one gateway's public URL is rejected by another listener. The default primary gateway speaks OAuth only, so local agents (author, reviewer, reader) authenticate through the Obol gateway on 18486 — and their Obols must be minted through that gateway:

set -a; . secrets/principal-ids.txt; set +a
docker compose exec charon-reviewer charon obol mint --expires 7d "$AUTHOR_ID"   > secrets/author-obol.txt
docker compose exec charon-reviewer charon obol mint --expires 7d "$REVIEWER_ID" > secrets/reviewer-obol.txt
docker compose exec charon-reviewer charon obol mint --expires 7d "$READER_ID"   > secrets/reader-obol.txt
chmod 600 secrets/*-obol.txt

Phase 3 — Connect the agents

Local MCP clients (OpenClaw example; any streamable-http MCP client works). Extract tokens without leaving them in shell history:

AUTHOR_TOKEN=$(grep -oE 'Token: obol_[A-Za-z0-9_-]+' secrets/author-obol.txt | sed 's/Token: //')
REVIEWER_TOKEN=$(grep -oE 'Token: obol_[A-Za-z0-9_-]+' secrets/reviewer-obol.txt | sed 's/Token: //')

openclaw mcp add charon-author \
  --url http://127.0.0.1:18486/mcp \
  --transport streamable-http \
  --header "Authorization=Bearer ${AUTHOR_TOKEN}"

openclaw mcp add charon-reviewer \
  --url http://127.0.0.1:18486/mcp \
  --transport streamable-http \
  --header "Authorization=Bearer ${REVIEWER_TOKEN}"

openclaw mcp reload
openclaw mcp probe charon-author
openclaw mcp probe charon-reviewer

Remote browser clients (ChatGPT connector) instead point at the primary gateway through the HTTPS tunnel and complete the S256 PKCE flow, pasting the current pairing key from the startup banner. Their tokens are automatically capped to memory.read, memory.search, thread.read, and memory.propose — no branch, commit, review, merge, or write — because CHARON_OAUTH_ALLOW_AUTHOR_SCOPES=false by default.

Expected result: both probes list the thirteen Memory Git tools; no client has a direct route to Lethe Git.

Phase 4 — Propose memory (proposer)

As the author principal, through charon-author:

1. memory_repo_init({"project":"<project>"}) — once per project; creates the synthetic root and protected refs/shared/main.
2. memory_context_at({"project":"<project>","ref":"refs/shared/main",...}) — orient on accepted memory.
3. memory_branch_create — one owned ref, e.g. refs/agents/<actor>/main or refs/topics/<topic>.
4. memory_changeset_create — one CAS commit with sequential ordinals, a unique idempotency_key, and semantic operations (add_memory, correct_memory, attach_evidence, …). Payloads are closed-key validated at Charon and again at Lethe before anything becomes immutable.
5. memory_merge_propose — propose the owned head into refs/shared/main.

Retry the identical request with the same idempotency key after a network failure — you get the original changeset back, never a duplicate. Change any field and the same key fails closed instead of silently returning stale work.

Expected result: changeset committed on the owned ref (verify with memory_show); proposal pending; refs/shared/main unchanged.

Phase 5 — Review and merge (reviewer)

As the reviewer principal, through charon-reviewer:

1. Discover pending proposals (maintainer shell: docker compose exec charon charon proposal ls) and resolve exact state.
2. Inspect with memory_show, memory_log, and memory_diff against the exact changeset IDs — never a branch name.
3. Apply the review criteria (durable content, no credentials or private reasoning, valid ordinals, no blocking conflicts, not stale).
4. memory_merge_review with approve / request_changes / reject. The verdict is bound to the exact proposal snapshot: if the proposal changes, the approval no longer applies.
5. memory_merge — Charon re-validates that the approval is current, computes the strategy (fast-forward / two-parent / cherry-pick), signs a memory-git-merge/v2 envelope (2-minute TTL, single-use nonce, bound to project, ref, heads, proposal digest, reviewer, and merger), and Lethe independently verifies it and consumes the nonce atomically with the CAS.
6. Verify: proposal completed, refs/shared/main advanced to the returned changeset, memory_context_at serves the new accepted memory, and docker compose exec charon charon ledger tail shows the review and merge.

Negative checks worth running once: the author calling memory_merge_review on its own proposal is denied (self review); calling memory_merge is denied (self approval); replaying a captured merge authorization is rejected by Lethe's consumed-nonce table.

Expected result: protected ref advanced via a reviewed merge; both databases and the ledger agree.

Phase 6 — Read accepted memory (reader / any role)

memory_context_at on refs/shared/main from any principal with read grants — including an OAuth-capped ChatGPT connector — returns exactly the accepted state at the resolved head. Optional manifest pinning requires exact-project commit/write authority.

Expected result: readers see the merged memory; the author's unmerged branch work is invisible on refs/shared/main.

Phase 7 — Restart persistence (maintainer)

docker compose -f docker-compose.git.yml --env-file .env.git restart   # in the lethe repo
docker compose restart                                                 # in the charon repo

Then confirm:

• refs/shared/main head, changesets, proposals, and ledger are intact;
• author and reviewer Obols still authenticate (digests persisted);
• a previously consumed merge nonce is still rejected (replay protection survives restarts);
• the OAuth pairing key in the startup banner changed — this is expected and is the only credential that rotates on restart.

Expected result: full state persistence; new pairing key; nothing else regenerated.

Phase 8 — Rotation drill (maintainer)

./scripts/rotate-tokens.sh

Revokes every active author/reviewer/reader Obol and mints fresh 7-day tokens into secrets/ (values never printed). Old tokens are rejected immediately. Key-level rotation semantics (Obol key, OAuth key, merge key — which must rotate on both services together) are in docs/operations.md.

Expected result: old Obol rejected with 401; new Obol works; MCP clients re-registered with the new token.

Phase 9 — Backup and restore drill (maintainer)

Coordinated snapshot of both databases with the CLI (both database files must be reachable by the one charon process that runs it):

charon backup --charon-db <path-to-charon.db> --lethe-db <path-to-lethe.db> --out <backup-dir>

In the Compose deployment charon.db lives inside a named volume, so use the zero-downtime runbook instead: sqlite3 .backup against the host-mounted Lethe database, and docker cp the Charon database out after stopping both gateways — exact commands in docs/local-memory-reviewer.md. Validate before trusting a restore:

docker compose exec charon charon ledger verify
charon reconcile --charon-data <restored-charon-data-dir> --lethe-db <restored-lethe.db>

CHARON_RECOVERY_READONLY=1 keeps a restored instance inspectable until reconciliation passes.

Expected result: integrity checks pass; restore procedure documented and rehearsed before it is needed.

Where to go next

• docs/operations.md — runbook: health, rotation, backup/restore, incidents.
• docs/observability.md — metrics, SLOs, alert thresholds.
• docs/local-memory-reviewer.md — concrete reviewer deployment example.
• skills/charon-maintainer/SKILL.md — operator playbook.
• skills/charon-proposer/SKILL.md — author playbook.
• skills/charon-reviewer/SKILL.md — reviewer playbook.
• skills/charon/SKILL.md — deployment-neutral tool reference for agents.
• Lethe repository: docs/memory-git-v1.md (protocol), docs/local-memory-git.md (boundary rules), docs/memory-context-bridge.md (projection).