{
  "id": "page:docs-user-guide-features-journal-system",
  "_kind": "Page",
  "_file": "wiki/docs/user-guide/features/journal-system.md",
  "_cluster": "wiki",
  "attributes": {
    "nodeKind": "Page",
    "sourcePath": "docs/user-guide/features/journal-system.md",
    "sourceKind": "repo-docs",
    "title": "Journal System: Event Sourcing and Audit Trail",
    "displayName": "Journal System: Event Sourcing and Audit Trail",
    "slug": "docs/user-guide/features/journal-system",
    "articlePath": "wiki/docs/user-guide/features/journal-system.md",
    "article": "\n# Journal System: Event Sourcing and Audit Trail\n\n**Version:** 1.1\n**Last Updated:** 2026-01-26\n**Category:** Feature Guide\n\n---\n\n## In Plain English\n\n**The journal is a diary of everything that happens during a run.**\n\nEvery time the AI does something - writes code, runs tests, asks for approval - it gets recorded in the journal. This means:\n\n- **Nothing is lost**: If your computer crashes, you can pick up where you left off\n- **You can see what happened**: \"Why did it do that?\" → check the journal\n- **You have proof**: For compliance or debugging, you have a complete record\n\n**Where is it?** Look in `.a5c/runs/<your-run-id>/journal/` - each file is one event.\n\n**Tip for beginners:** You don't need to understand the journal to use Babysitter. It works automatically. But when something goes wrong, the journal helps you figure out why.\n\n---\n\n## Overview\n\nThe journal system is Babysitter's event-sourced persistence layer. Every state change in a workflow is recorded as an immutable event in the journal. This append-only log serves as the single source of truth, enabling deterministic replay, debugging, audit trails, and resumption from any point.\n\n### Why Use the Journal System\n\n- **Complete Audit Trail**: Every action and decision is recorded permanently\n- **Deterministic Replay**: Reproduce exact execution by replaying events\n- **Time-Travel Debugging**: Inspect state at any point in history\n- **Git-Friendly**: One event per file minimizes merge conflicts\n- **Compliance**: Meets audit requirements for regulated environments\n- **Resumption**: Rebuild state from journal to continue interrupted workflows\n\n---\n\n## Use Cases and Scenarios\n\n### Scenario 1: Debugging a Failed Run\n\nInvestigate what happened when a workflow failed.\n\n```bash\n# View all events\nbabysitter run:events 01KFFTSF8TK8C9GT3YM9QYQ6WG --json | jq '.events[]'\n\n# Find the failure event\nbabysitter run:events 01KFFTSF8TK8C9GT3YM9QYQ6WG --filter-type RUN_FAILED --json\n\n# See events leading up to failure\nbabysitter run:events 01KFFTSF8TK8C9GT3YM9QYQ6WG --limit 10 --reverse --json\n```\n\n### Scenario 2: Audit Compliance Reporting\n\nGenerate an audit report of all human approvals in a workflow.\n\n```bash\n# Extract all breakpoint-related events (breakpoints use EFFECT_REQUESTED with kind: \"breakpoint\")\njq 'select(.type == \"EFFECT_REQUESTED\" and .data.kind == \"breakpoint\") // select(.type == \"EFFECT_RESOLVED\")' \\\n  .a5c/runs/*/journal/*.json \\\n  > audit-report.json\n```\n\n### Scenario 3: Effect Status Tracking\n\nTrack effect resolution statuses across a run for trend analysis.\n\n```bash\n# Extract effect statuses from all resolved effects\njq 'select(.type == \"EFFECT_RESOLVED\") | {effectId: .data.effectId, status: .data.status}' \\\n  .a5c/runs/01KFFTSF8TK8C9GT3YM9QYQ6WG/journal/*.json\n```\n\n### Scenario 4: Performance Analysis\n\nAnalyze task execution times to identify bottlenecks.\n\n```bash\n# Calculate task durations\njq -s '\n  [.[] | select(.type == \"EFFECT_RESOLVED\")] |\n  map({effectId: .data.effectId, duration: (.data.finishedAt | fromdateiso8601) - (.data.startedAt | fromdateiso8601)}) |\n  sort_by(.duration) |\n  reverse\n' .a5c/runs/01KFFTSF8TK8C9GT3YM9QYQ6WG/journal/*.json\n```\n\n---\n\n## Step-by-Step Instructions\n\n### Step 1: Understand the Journal Structure\n\nThe journal is stored in the `journal/` directory within each run.\n\n```\n.a5c/runs/<runId>/\n├── journal/\n│   ├── 000001.<ulid>.json    # Event 1\n│   ├── 000002.<ulid>.json    # Event 2\n│   ├── 000003.<ulid>.json    # Event 3\n│   └── ...\n├── state/\n│   └── state.json            # Derived cache (gitignored)\n├── tasks/\n│   └── <effectId>/\n│       ├── task.json\n│       └── result.json\n└── run.json\n```\n\n**File naming convention:**\n```\n<sequence>.<ulid>.json\n```\n\n- `sequence`: 6-digit zero-padded number (000001, 000002, ...)\n- `ulid`: Unique Lexicographically Sortable Identifier\n- Example: `000042.01HJKMNPQR3STUVWXYZ012345.json`\n\n### Step 2: View Journal Events\n\nUse the CLI or read files directly.\n\n**Via CLI (recommended):**\n```bash\n# View all events\nbabysitter run:events 01KFFTSF8TK8C9GT3YM9QYQ6WG --json\n\n# View last 10 events\nbabysitter run:events 01KFFTSF8TK8C9GT3YM9QYQ6WG --limit 10 --reverse\n\n# Filter by event type\nbabysitter run:events 01KFFTSF8TK8C9GT3YM9QYQ6WG --filter-type EFFECT_RESOLVED --json\n```\n\n**Via file system:**\n```bash\n# List all event files\nls -la .a5c/runs/01KFFTSF8TK8C9GT3YM9QYQ6WG/journal/\n\n# Read a specific event\ncat .a5c/runs/01KFFTSF8TK8C9GT3YM9QYQ6WG/journal/000001.*.json | jq .\n```\n\n### Step 3: Understand Event Types\n\nLearn the different event types recorded in the journal.\n\n**Run lifecycle events:**\n- `RUN_CREATED` - Run initialized\n- `RUN_COMPLETED` - Run finished successfully\n- `RUN_FAILED` - Run terminated with error\n\n**Effect (task) events:**\n- `EFFECT_REQUESTED` - Task requested for execution\n- `EFFECT_RESOLVED` - Task completed (success or error)\n\n**Breakpoint events (subset of effect events):**\n- Breakpoints use `EFFECT_REQUESTED` with `kind: \"breakpoint\"` - Human approval requested\n- Breakpoints are resolved via `EFFECT_RESOLVED` - Approval granted or denied\n\n> **Note:** Quality scoring is handled at the application level and does not have a dedicated journal event type.\n\n### Step 4: Query the Journal\n\nUse jq or other tools to query and analyze events.\n\n```bash\n# Count events by type\njq -s 'group_by(.type) | map({type: .[0].type, count: length})' \\\n  .a5c/runs/01KFFTSF8TK8C9GT3YM9QYQ6WG/journal/*.json\n\n# Find all failed tasks\njq 'select(.type == \"EFFECT_RESOLVED\" and .data.status == \"error\")' \\\n  .a5c/runs/01KFFTSF8TK8C9GT3YM9QYQ6WG/journal/*.json\n\n# Get timeline of events (note: seq is derived from filename, not stored in event body)\njq '{type, recordedAt}' .a5c/runs/01KFFTSF8TK8C9GT3YM9QYQ6WG/journal/*.json\n```\n\n### Step 5: Rebuild State from Journal\n\nIf the state cache is corrupted, rebuild it from the journal.\n\n```bash\n# Delete corrupted state\nrm .a5c/runs/01KFFTSF8TK8C9GT3YM9QYQ6WG/state/state.json\n\n# Any CLI command will rebuild state automatically\nbabysitter run:status 01KFFTSF8TK8C9GT3YM9QYQ6WG\n```\n\n---\n\n## Configuration Options\n\n### Event Schema\n\nAll events share a common base structure:\n\n```typescript\ntype JournalEventBase = {\n  type: string;        // Event type\n  recordedAt: string;  // ISO 8601 timestamp\n  data: object;        // Event-specific data (see below)\n  checksum: string;    // SHA-256 hex digest for integrity verification\n};\n// Note: seq (sequence number) is derived from the filename ({SEQ}.{ULID}.json),\n// not stored in the event body.\n```\n\n### RUN_CREATED Event\n\n```typescript\ntype RunCreated = JournalEventBase & {\n  type: \"RUN_CREATED\";\n  data: {\n    runId: string;\n    processId: string;\n    processRevision?: string;\n    entrypoint: {\n      importPath: string;\n      exportName: string;\n    };\n    inputsRef?: string;\n  };\n};\n```\n\n### EFFECT_REQUESTED Event\n\n```typescript\ntype EffectRequested = JournalEventBase & {\n  type: \"EFFECT_REQUESTED\";\n  data: {\n    effectId: string;\n    invocationKey: string;\n    stepId: string;\n    taskId: string;\n    kind: string;           // \"node\", \"breakpoint\", \"orchestrator_task\"\n    label?: string;\n    taskDefRef: string;\n    inputsRef?: string;\n  };\n};\n```\n\n### EFFECT_RESOLVED Event\n\n```typescript\ntype EffectResolved = JournalEventBase & {\n  type: \"EFFECT_RESOLVED\";\n  data: {\n    effectId: string;\n    status: \"ok\" | \"error\";\n    resultRef?: string;\n    error?: {\n      name: string;\n      message: string;\n      stack?: string;\n      data?: any;\n    };\n    stdoutRef?: string;\n    stderrRef?: string;\n    startedAt?: string;\n    finishedAt?: string;\n  };\n};\n```\n\n### RUN_COMPLETED / RUN_FAILED Events\n\n```typescript\ntype RunCompleted = JournalEventBase & {\n  type: \"RUN_COMPLETED\";\n  data: {\n    outputRef?: string;\n  };\n};\n\ntype RunFailed = JournalEventBase & {\n  type: \"RUN_FAILED\";\n  data: {\n    error: {\n      name: string;\n      message: string;\n      stack?: string;\n      data?: any;\n    };\n  };\n};\n```\n\n---\n\n## Code Examples and Best Practices\n\n### Example 1: Read All Events in a Run\n\n```bash\n#!/bin/bash\nRUN_ID=\"01KFFTSF8TK8C9GT3YM9QYQ6WG\"\nJOURNAL_DIR=\".a5c/runs/$RUN_ID/journal\"\n\n# Read and sort events by recordedAt timestamp\n# (seq ordering is already guaranteed by the filename sort order)\nfor f in \"$JOURNAL_DIR\"/*.json; do\n  cat \"$f\"\ndone | jq -s 'sort_by(.recordedAt)'\n```\n\n### Example 2: Extract Audit Trail\n\nGenerate a human-readable audit trail:\n\n```bash\njq -r '\n  \"\\(.recordedAt) [\\(.type)] \" +\n  (if .type == \"RUN_CREATED\" then \"Run started: \\(.data.processId)\"\n   elif .type == \"EFFECT_REQUESTED\" then \"Task requested: \\(.data.taskId) (\\(.data.kind))\"\n   elif .type == \"EFFECT_RESOLVED\" then \"Task completed: \\(.data.effectId) - \\(.data.status)\"\n   elif .type == \"RUN_COMPLETED\" then \"Run completed successfully\"\n   elif .type == \"RUN_FAILED\" then \"Run failed: \\(.data.error.message)\"\n   else .type end)\n' .a5c/runs/01KFFTSF8TK8C9GT3YM9QYQ6WG/journal/*.json\n```\n\n**Output:**\n```\n2026-01-25T10:15:30.123Z [RUN_CREATED] Run started: feature/auth\n2026-01-25T10:15:31.456Z [EFFECT_REQUESTED] Task requested: plan (agent)\n2026-01-25T10:15:45.789Z [EFFECT_RESOLVED] Task completed: effect-001 - ok\n2026-01-25T10:15:46.012Z [EFFECT_REQUESTED] Task requested: breakpoint (breakpoint)\n2026-01-25T10:20:12.345Z [EFFECT_RESOLVED] Task completed: breakpoint-001 - ok\n2026-01-25T10:20:13.678Z [EFFECT_REQUESTED] Task requested: implement (agent)\n2026-01-25T10:25:34.901Z [RUN_COMPLETED] Run completed successfully\n```\n\n### Example 3: Effect Resolution Summary\n\n> **Note:** Quality scoring is handled at the application level, not as a journal event type.\n> This example shows how to summarize effect resolutions instead.\n\n```bash\njq -s '\n  [.[] | select(.type == \"EFFECT_RESOLVED\")] |\n  map({effectId: .data.effectId, status: .data.status, recordedAt: .recordedAt}) |\n  sort_by(.recordedAt)\n' .a5c/runs/01KFFTSF8TK8C9GT3YM9QYQ6WG/journal/*.json\n```\n\n**Output:**\n```json\n[\n  {\"effectId\": \"effect-001\", \"status\": \"ok\", \"recordedAt\": \"2026-01-25T10:15:45.123Z\"},\n  {\"effectId\": \"effect-002\", \"status\": \"ok\", \"recordedAt\": \"2026-01-25T10:18:23.456Z\"},\n  {\"effectId\": \"effect-003\", \"status\": \"error\", \"recordedAt\": \"2026-01-25T10:21:34.789Z\"}\n]\n```\n\n### Example 4: Find Long-Running Tasks\n\n```bash\njq -s '\n  [.[] | select(.type == \"EFFECT_RESOLVED\" and .data.startedAt and .data.finishedAt)] |\n  map({\n    effectId: .data.effectId,\n    durationSec: ((.data.finishedAt | fromdateiso8601) - (.data.startedAt | fromdateiso8601))\n  }) |\n  sort_by(.durationSec) |\n  reverse |\n  .[0:5]\n' .a5c/runs/01KFFTSF8TK8C9GT3YM9QYQ6WG/journal/*.json\n```\n\n### Example 5: Export Journal for External Analysis\n\n```bash\n# Export to single JSON file\njq -s '.' .a5c/runs/01KFFTSF8TK8C9GT3YM9QYQ6WG/journal/*.json > events.json\n\n# Export to CSV for spreadsheet analysis (note: seq is derived from filename, not event body)\njq -r '\n  [.type, .recordedAt, .data.effectId // \"\", .data.status // \"\"] |\n  @csv\n' .a5c/runs/01KFFTSF8TK8C9GT3YM9QYQ6WG/journal/*.json > events.csv\n```\n\n### Best Practices\n\n1. **Never Manually Edit Journal Files**: The journal is append-only and immutable\n2. **Use CLI Commands for Queries**: The CLI handles edge cases and formats output properly\n3. **Archive Old Runs**: Completed runs can be archived to save disk space\n4. **Monitor Journal Size**: Very long runs may generate large journals\n5. **Use Filters for Large Journals**: Filter by event type for efficiency\n6. **Back Up Before Cleanup**: Always backup before deleting old runs\n\n---\n\n## Common Pitfalls and Troubleshooting\n\n### Pitfall 1: Manual Journal Editing\n\n**Symptom:** Run behaves unexpectedly or fails to resume.\n\n**Cause:** Direct edits to journal files.\n\n**Solution:**\n- Never edit journal files directly\n- If corrupted, you may need to start a new run\n- For state cache issues, delete `state/state.json` and let it rebuild\n\n### Pitfall 2: Journal Too Large\n\n**Symptom:** Slow performance when reading events.\n\n**Cause:** Very long runs with many iterations.\n\n**Solution:**\n- Use `--limit` flag when querying\n- Filter by specific event types\n- Consider archiving completed runs\n\n```bash\n# Query with limits\nbabysitter run:events \"$RUN_ID\" --limit 100 --json\n\n# Filter to specific type\nbabysitter run:events \"$RUN_ID\" --filter-type EFFECT_RESOLVED --limit 50 --json\n```\n\n### Pitfall 3: Missing State Cache\n\n**Symptom:** CLI commands are slow on first access.\n\n**Cause:** State cache doesn't exist or was deleted.\n\n**Solution:**\n- This is normal - state is derived from journal\n- First CLI command will rebuild the cache\n- Cache is stored in `state/state.json`\n\n```bash\n# Trigger state rebuild\nbabysitter run:status \"$RUN_ID\"\n```\n\n### Pitfall 4: Event Order Confusion\n\n**Symptom:** Events appear out of order.\n\n**Cause:** File system listing order differs from sequence order.\n\n**Solution:**\n- Always use the CLI or sort by `recordedAt` field\n- File names include sequence numbers for proper ordering (seq is derived from filename, not stored in event body)\n\n```bash\n# Correct: sorted by timestamp\njq -s 'sort_by(.recordedAt)' .a5c/runs/\"$RUN_ID\"/journal/*.json\n\n# Or use CLI which handles ordering\nbabysitter run:events \"$RUN_ID\" --json\n```\n\n### Pitfall 5: Journal Conflicts in Git\n\n**Symptom:** Merge conflicts in journal files.\n\n**Cause:** Multiple writers to same run (should not happen in normal use).\n\n**Solution:**\n- Babysitter uses single-writer model\n- If conflicts occur, the later events are typically valid\n- Consider using separate runs for parallel work\n\n---\n\n## How the Journal Works\n\n### Event Sourcing Model\n\n```\nUser Request\n    │\n    ▼\n┌─────────────┐\n│   Process   │ ──▶ Decisions (tasks, breakpoints, etc.)\n└─────────────┘\n    │\n    ▼\n┌─────────────┐\n│   Journal   │ ◀── Append events (immutable)\n│  (Events)   │\n└─────────────┘\n    │\n    ▼\n┌─────────────┐\n│    State    │ ◀── Derived by replaying events\n│  (Cache)    │\n└─────────────┘\n```\n\n### State Reconstruction\n\nOn each iteration:\n1. Load all journal events\n2. Replay events to build state index\n3. Process function starts from beginning\n4. Intrinsics check state index:\n   - If result exists: return cached result (short-circuit)\n   - If pending: throw exception\n   - If new: record event and throw exception\n\n### Git-Friendly Design\n\n- **One file per event**: Minimizes merge conflicts\n- **Deterministic naming**: Sequence + ULID ensures ordering\n- **Append-only**: No modifications to existing files\n- **State cache gitignored**: Derived, not source of truth\n\n---\n\n## Related Documentation\n\n- [Run Resumption](./run-resumption.md) - Understand how journal enables resumption\n- [Process Definitions](./process-definitions.md) - How events are generated\n- [Breakpoints](./breakpoints.md) - Breakpoint events in the journal\n\n---\n\n## Summary\n\nThe journal system is the foundation of Babysitter's reliability and auditability. Every action is recorded as an immutable event, enabling deterministic replay, debugging, compliance reporting, and seamless resumption. Use the CLI to query events, never edit journal files directly, and leverage the complete audit trail for debugging and analysis.\n",
    "documents": []
  },
  "outgoingEdges": [],
  "incomingEdges": [
    {
      "from": "page:docs-user-guide-features",
      "to": "page:docs-user-guide-features-journal-system",
      "kind": "contains_page"
    }
  ]
}