JSONL ledgers in git as the state layer for an autonomous agent: patterns that survive crashes and retries

Our autonomous agent operates a small publishing business, posting and replying to articles while tracking every decision made. The state layer powering this operation is not a traditional database like Postgres or SQLite, but rather a directory of JSONL files committed to a git repository. This unique choice has led to several benefits that make it resilient to crashes, retries, concurrent writers, and the tendency of an LLM to re-run tasks it has already completed.

The primary reason for using files in git is that each state change is recorded as a diff within the git log, complete with timestamps and author information. This provides an easy-to-understand audit trail for the autonomous system. Scheduled jobs and interactive sessions both share state without the need for a centralized server.

GitHub Actions jobs check out the repository, read the ledgers, perform actions, and then commit the changes. The interactive session, meanwhile, pulls the latest state before making any decisions. The merge boundary is handled by git, which is a well-established process.

One remarkable aspect of this approach is that the LLM can read its own state directly, without requiring a separate query layer. This allows the agent to have a comprehensive understanding of its decision history, making it significantly smarter than a system that would rely on an external query system. Two patterns emerge from this design:

1. Append-only ledgers, with one exception: Almost every ledger is append-only, with each line representing a new JSON object containing facts. This ensures that a crashed write only corrupts the final line, making recovery a simple drop of the broken tail instead of a full system restore. The sole exception is consumption ledgers, which require a consumedAt stamp on existing rows.

These files are small, so loading, modifying, and rewriting the entire file is acceptable, with the rule that a consumed mark is never overwritten. This refusal to overwrite a row with a set consumedAt timestamp contributes to the system's retry-safety.

2. Idempotency keys from the outside world: Every ledger row that mirrors an external event carries the external system's identifier, such as a post URI, article ID, or comment permalink. During ingestion, the system deduplicates on this key, ensuring that the same feedback is recorded only once. This allows the system to handle retries or repeated runs without creating duplicate entries.

Furthermore, the LLM should never be given raw identifiers to input, as even a single typo can lead to incorrect records that will remain in the history forever. All identifiers must be mechanically copied from a previous command's output.

The third pattern involves two-phase validation, where commands that act on the world validate the entire batch before performing any actions. If any entry in a reply batch is malformed, the whole batch is rejected before any action is taken, preventing the creation of a half-executed batch. This ensures that the ledger's state aligns with the actual world state, reducing potential inconsistencies.

The fourth pattern highlights the ledger as the gatekeeper for all state changes. Test suites can read production state directly from the ledgers, allowing for comprehensive testing of invariants. For instance, the tests verify that stocked posts are within length limits, that article titles don't collide, and that open TODO items haven't exceeded their grace period.

Any corrupt or contradictory state cannot be committed, as the tests run on every commit. This ensures that state bugs are caught during the write process rather than at an inconvenient time, such as during a scheduled job.