Files
crank/_bmad-output/implementation-artifacts/1-15-invocation-history-usage.md
T

21 KiB
Raw Blame History

story_id, story_key, requirement, status, baseline_commit, created
story_id story_key requirement status baseline_commit created
1.15 1-15-invocation-history-usage FR-6 done 767428436d 2026-08-23T01:47:15+03:00

Story 1.15: Предоставить безопасные Invocation History и Usage

Status: done

Story

As an оператор Crank, I want видеть фильтруемую историю и показатели выполнения, so that могу диагностировать ошибки и оценивать качество работы Agents и Operations без доступа к чувствительным данным.

Operational Brief

Story 1.15 закрывает FR-6 и завершает hardened core перед onboarding Story 1.16.

Предыдущие stories уже дали основу:

  • 1.5 сохраняет Request ID и Trace ID независимо от telemetry sampling;
  • 1.7 и 1.13 гарантируют immutable Operation/Agent published references;
  • 1.8 и 1.9 ввели typed execution taxonomy и honest side-effect uncertainty;
  • 1.14 гарантирует atomic approvals и не заменяет product result при history-loss.

Теперь нужно сделать Invocation History и Usage полноценным операторским инструментом:

  • history record должен быть typed, scoped и безопасно редактированным;
  • Logs API/UI должны фильтровать по bounded period, Agent, Operation и outcome;
  • details должны показывать stage, safe previews, duration, Agent, Operation Version, Request ID и Trace ID;
  • Usage должен считать counts, outcomes и latency percentiles без смешивания upstream/client/schema failures с Crank-caused failures;
  • CSV export обязан наследовать filters/auth/redaction/bounds и защищать spreadsheet consumers;
  • retention должен быть typed/observable и не повреждать audit/evidence;
  • production-size query path не должен грузить 1 000 000 records в память приложения или браузера.

Requirement / Architecture Trace

  • Direct requirement: FR-6.
  • Primary architecture: AD-11 correlation identity; AD-13 typed metrics/local product analytics; AD-24 retention.
  • Supporting architecture: AD-1, AD-2, AD-4, AD-12, AD-16, AD-19, AD-20.
  • Applicable NFR: NFR-P3, NFR-P5, NFR-R2, NFR-S1, NFR-S2, NFR-S4, NFR-S6, NFR-S8, NFR-O1, NFR-O4, NFR-O5, NFR-C1, NFR-C4.

Scope

In scope:

  • Invocation History record contract and persistence validation.
  • Safe preview redaction/limits/truncation evidence.
  • Logs list/detail Admin API filters, pagination and scoped access.
  • Usage aggregation by workspace, Agent and Operation with outcome groups and percentiles.
  • UTC half-open period boundary semantics.
  • CSV export for filtered history/usage dataset with redaction and formula-injection protection.
  • Retention cleanup policy and observable outcome.
  • Logs/Usage UI loading, empty, error, retry, export and stale-response guards in RU/EN.
  • Registry/Admin/UI/Playwright tests, docs, capability inventory and baseline evidence for FR-6.

Out of scope:

  • GlitchTip trace deep links/search UI beyond showing/copying local Request ID/Trace ID.
  • Product analytics DAU/adoption rollups (Epic 5 / FR-35).
  • Prometheus/Grafana packaged dashboards (Epic 5).
  • Background Tasks history (Epic 4), Load Runs (Epic 6), Resources/Prompts usage (Epic 3).
  • Changing Operation/Agent/approval semantics from Stories 1.7/1.13/1.14.

Acceptance Criteria

  1. Typed history record. Given завершённая или отклонённая Execution attempt, when формируется Invocation History record, then сохраняются workspace, Agent, Operation/version, execution ID, outcome taxonomy, timestamps, duration, Request ID и Trace ID, and запись не содержит raw credential, arbitrary headers/body или unbounded error text.
  2. Safe previews. Given execution input и output, when создаются previews, then централизованная redaction применяет byte/field/depth limits, сохраняет валидный UTF-8/JSON and oversized content получает явный truncation marker.
  3. History-loss honesty. Given failure записи Invocation History после фактического outcome, when persistence недоступен или record отклонён, then product result не заменяется ложным execution failure, and typed crank-metrics увеличивает low-cardinality history-loss counter.
  4. Logs filters and pagination. Given авторизованный Admin UI caller, when он открывает Logs, then history фильтруется по bounded period, Agent, Operation и outcome с deterministic opaque pagination, and foreign-scope query не раскрывает counts, names или records.
  5. Log details. Given конкретная Invocation History record, when администратор открывает details, then UI показывает stage, safe previews, duration, Agent, точную Operation Version, Request ID и Trace ID, and raw internal cause остаётся server-side.
  6. Usage aggregation. Given Usage query за период, when данные агрегируются, then UI получает counts, success/failure outcomes и latency percentiles по Agent и Operation, and upstream/client/schema outcomes отображаются отдельно от Crank-caused failures.
  7. UTC period boundaries. Given period boundary, when агрегируются records на границе дня или timezone UI, then storage/query contract использует UTC instants and one record cannot be counted twice by inclusive overlap.
  8. CSV export. Given CSV export, when администратор экспортирует отфильтрованный dataset, then export applies same authorization, filters, redaction and bounded row/size policy as UI, and spreadsheet formula injection values are escaped safely.
  9. Retention. Given configured Invocation History retention, when record reaches retention boundary, then archive/delete uses typed observable policy/outcome, and deletion does not damage Published Version audit, aggregate Usage or immutable release evidence.
  10. Production-size query plan. Given 1 000 000 history records, when Logs/Usage screens query data, then query path is bounded and does not load full dataset into app/browser memory; local evidence records plan/row-count/proxy timing.
  11. Logs/Usage UI RU/EN. Given Logs/Usage UI на RU и EN, when данные загружаются, отсутствуют, фильтруются, экспортируются или завершаются ошибкой, then loading/empty/error/retry/stale-response states exist, and workspace/request generation guard prevents stale response overwrite.
  12. Verification. Given automated verification, when history and usage suites run, then redaction, bounded previews, persistence loss, period boundaries, percentiles, CSV injection, retention, cross-scope access and production-size query plans are covered, and registry/Admin API/Playwright tests plus just verify pass.

Tasks / Subtasks

  • Task 1 — Freeze history/usage oracle and RED tests (AC: 1–12)

    • Characterize current invocation_logs, Admin Logs/Usage API and UI behavior.
    • Add RED tests for filters, pagination, details, period boundaries, CSV injection, retention and production-size plan.
    • Add RED tests for redaction, bounded previews, cross-scope denial and history-loss product-result preservation.
  • Task 2 — Invocation History persistence contract (AC: 1–3, 5, 9)

    • Validate all new history records include operation version, execution taxonomy, request_id and trace_id.
    • Ensure safe previews are produced by one central sanitizer and stored bounded.
    • Add retention API/policy with typed outcome and metrics; do not delete Published Version audit/evidence.
  • Task 3 — Logs list/detail API (AC: 4–5, 7–8, 12)

    • Add bounded query DTO fields for period start/end, Agent, Operation, outcome and opaque cursor.
    • Implement deterministic pagination without leaking foreign counts or names.
    • Add safe details projection including stage, previews, duration, Agent, Operation Version, Request ID and Trace ID.
    • Add CSV export endpoint using same filters/auth/redaction/bounds and formula escaping.
  • Task 4 — Usage aggregation API (AC: 6–7, 10, 12)

    • Implement UTC half-open window semantics and prevent boundary double-counting.
    • Aggregate counts, success/failure outcomes and p50/p95/p99 by Agent and Operation.
    • Separate upstream/client/schema outcomes from Crank-caused failures.
    • Add production-size query-plan/index evidence and bounded memory proof.
  • Task 5 — Logs/Usage UI and Playwright (AC: 4–8, 11–12)

    • Add Agent/Operation/outcome filters, pagination, retry/export states and Request ID/Trace ID details.
    • Add stale-response guards for workspace/filter/request generation.
    • Add RU/EN strings together and avoid raw HTML sinks.
    • Extend Playwright logs-usage.spec.js for filters, details, CSV export, empty/error and stale-response behavior.
  • Task 6 — Docs, inventory, gates and handoff (AC: 12)

    • Update docs/admin-api.md, docs/data-model.md, docs/manual-regression-checklist.md and capability inventory for FR-6 only after evidence passes.
    • Update baseline manifest/results without raw browser reports, secrets, payloads or host paths.
    • Run registry/Admin/UI targeted suites, capability gates, SQLx checks and final just verify.
    • Update Dev Agent Record, File List, Change Log and story/sprint status to review only after gates pass.

Review Findings

  • [Review][Patch][High] Safe preview sanitizer does not enforce field/depth limits [crates/crank-core/src/observability.rs:12]
  • [Review][Patch][High] Logs detail UI omits exact Operation Version [apps/ui/js/logs.js:126]
  • [Review][Patch][High] Usage CSV export remains client-side snapshot instead of server-authoritative bounded export [apps/ui/js/usage.js:556]
  • [Review][Patch][High] Retention remains raw timestamp delete without typed observable policy/outcome and preserved aggregate usage semantics [crates/crank-registry/src/postgres/observability.rs:19]
  • [Review][Patch][Medium] Usage stale-response guard mutates state before validating workspace/request generation [apps/ui/js/usage.js:527]
  • [Review][Patch][Medium] Logs filter changes do not invalidate in-flight responses immediately [apps/ui/js/logs.js:727]
  • [Review][Patch][Medium] Logs CSV export inherits page limit 100 instead of export dataset bound [apps/ui/js/logs.js:443]
  • [Review][Patch][Medium] Logs outcome filter supports only ok/error status, not execution outcome groups [apps/admin-api/src/dto.rs:307]
  • [Review][Patch][Medium] Usage outcome grouping defaults uncovered typed failures to crank [crates/crank-registry/src/postgres/observability.rs:633]
  • [Review][Patch][Medium] Admin Logs/Usage API lacks explicit start/end UTC window filters claimed by the story [apps/admin-api/src/dto.rs:307]

Dev Notes

Current State / Repository Findings

  • InvocationLog already contains workspace, optional Agent, Operation, optional operation version, source, level, status, request_id, trace_id, status_code, duration, execution stage/error/retryability/certainty and request/response previews.
  • Registry create_invocation_log already validates request_id, trace_id, operation_version and execution taxonomy for new records and sanitizes previews before insert.
  • Current list_invocation_logs supports workspace, level, source, operation_id, agent_id, created_after, search and limit, but does not provide cursor pagination, explicit start/end period, outcome filter or CSV export.
  • Current get_invocation_log scopes by workspace and ID, but details projection must explicitly stay redacted and expose trace_id in UI.
  • Current usage aggregation supports workspace/source/created_after and p50/p95/p99, but lacks explicit Agent/Operation endpoint evidence, outcome group separation and half-open start/end boundary contract.
  • Current UI Logs supports level/search/period/live mode, but does not include Agent/Operation/outcome filters, opaque pagination, CSV export, trace_id in details or robust logs stale-response guard.
  • Current UI Usage renders summary/timeline/operation breakdown and client-side CSV snapshot, but export is not server-authoritative and does not prove same filters/auth/redaction/bounds as UI.
  • Existing history-loss contract exists in Admin/MCP DC08 tests and crank-metrics history-loss metric; Story 1.15 must keep product result unchanged and add visible evidence, not invent a new failure path.
  • Existing retention method delete_invocation_logs_before(cutoff) is raw and timestamp-only; Story 1.15 must wrap it in typed policy/outcome and prove it does not damage immutable Operation/Agent evidence.

Architecture and Safety Constraints

  • PostgreSQL remains source of truth. Admin API/MCP startup must remain read-only; schema changes, if needed, go through MigrationAuthority append-only version.
  • History records are diagnostic evidence, not user payload storage. Do not store raw credentials, arbitrary headers/body or unbounded error text.
  • Request ID and Trace ID are support identifiers and must never become metric labels.
  • Logs/Usage filters must include workspace scope in every query and must not disclose foreign object existence through counts, names or different error shapes.
  • Pagination cursor must be opaque, deterministic and bounded; no offset pagination over production-size dataset if it requires scanning skipped rows at scale.
  • CSV export is an exfiltration surface: same auth/filter/redaction as UI, bounded rows/bytes, formula injection escaping, no raw host paths/secrets.
  • Usage/telemetry must not leave the installation; CSV remains local Admin response only.
  • UI updates must use textContent/DOM APIs and discard stale responses when workspace/filter/request generation changes.
  • No dependency/toolchain/MSRV changes; no broad frontend framework migration.

Likely File Map

Likely UPDATE:

  • crates/crank-core/src/observability.rs
  • crates/crank-registry/src/model.rs
  • crates/crank-registry/src/postgres/observability.rs
  • crates/crank-registry/src/postgres/mod.rs
  • crates/crank-registry/tests/integration/observability.rs
  • crates/crank-registry/tests/integration/common.rs
  • apps/admin-api/src/dto.rs
  • apps/admin-api/src/app.rs
  • apps/admin-api/src/routes/observability.rs
  • apps/admin-api/src/service/observability.rs
  • apps/admin-api/src/service.rs
  • apps/admin-api/tests/integration/community_access_usage.rs
  • apps/admin-api/tests/dc08.rs
  • apps/mcp-server/tests/dc08.rs
  • apps/ui/js/api.js
  • apps/ui/js/logs.js
  • apps/ui/js/usage.js
  • apps/ui/js/i18n.js
  • apps/ui/html/logs.html
  • apps/ui/html/usage.html
  • apps/ui/tests/e2e/logs-usage.spec.js
  • docs/admin-api.md
  • docs/data-model.md
  • docs/manual-regression-checklist.md
  • docs/capability-inventory.json
  • docs/capability-baseline/results.json
  • docs/capability-baseline/manifest.json

Likely NEW:

  • crates/crank-registry/tests/integration/observability_scale.rs only if keeping ignored production-size evidence separate.
  • Admin API CSV/export tests if existing community_access_usage.rs would exceed maintainability limits.

Required Test Matrix

  • New history record rejects missing/invalid request_id, trace_id, operation_version and unknown taxonomy.
  • Secret canaries in request/response/error material are absent from stored previews, API details, UI DOM and CSV.
  • Oversized nested previews preserve valid UTF-8/JSON and include explicit truncation markers.
  • History-loss after product success/error keeps product response and emits low-cardinality metric.
  • Logs filters: period, Agent, Operation, outcome, source/search and cursor; foreign objects return empty/safe error without counts.
  • Details show stage, duration, Agent, Operation Version, Request ID and Trace ID; raw internal cause absent.
  • Usage: counts, success/failure, p50/p95/p99 by Agent/Operation; upstream/client/schema groups separate from Crank-caused failures.
  • Period boundary: records exactly at start included, exactly at end excluded; adjacent windows do not double-count.
  • CSV: same filters as list, bounded rows/bytes, redaction, formula escaping for =, +, -, @, tab and CR/LF-leading values.
  • Retention: deletes only eligible records, reports typed outcome, preserves published versions, Agent snapshots and release evidence.
  • Scale: 1 000 000-row seed/plan evidence or ignored heavy test with deterministic local proxy budget.
  • UI: loading, empty, error, retry, filters, details, export, stale workspace/filter response, RU/EN.

Verification Commands

cargo test -p crank-core observability -- --test-threads=1
cargo test -p crank-registry --test integration observability -- --test-threads=1
cargo test -p crank-registry --test integration observability -- --test-threads=1 --ignored
cargo test -p admin-api --test dc08 -- --test-threads=1
cargo test -p mcp-server --test dc08 -- --test-threads=1
cargo test -p admin-api --test integration community_access_usage -- --test-threads=1
cargo test -p crank-observability --test incidents -- --test-threads=1
just ui-build
cd apps/ui && npx playwright test tests/e2e/logs-usage.spec.js --project=chromium
just capability-inventory-check
just sqlx-check
python3 scripts/check-community-scope.py --root . --files <all NEW/UPDATE paths>
git diff --check
just verify

Dev Agent Record

Agent Model Used

GPT-5 Codex

Debug Log References

  • 2026-08-23T01:47:15+03:00 — Created Story 1.15 from Epic 1 FR-6 after Story 1.14 moved to done.
  • 2026-08-23T02:50:00+03:00 — Implemented Logs/Usage API, UI, CSV export, outcome aggregation, cursor pagination, 1M-row bounded plan evidence, docs/inventory updates and final gates.
  • 2026-08-23T10:51:37+03:00 — Applied all code-review patch findings: bounded preview depth/field/array sanitization, operation version in log detail, server-authoritative Usage CSV export, typed retention outcome, immediate stale-response invalidation, execution outcome groups and explicit UTC windows. Re-ran targeted suites, Playwright Logs/Usage, capability/scope gates and full just verify.

Completion Notes List

  • Implemented bounded Invocation History list/detail API with status, Agent, Operation, period/search filters and opaque cursor pagination.
  • Added server-side CSV export using the same filter path with row/size bounds and spreadsheet formula escaping.
  • Added Usage outcome breakdown groups plus half-open UTC period aggregation for workspace/Agent/Operation usage queries.
  • Added Logs UI status/Agent/Operation filters, load-more pagination, CSV export, Request ID/Trace ID details and stale-response guards; Usage UI renders outcome groups and safer CSV cells.
  • Added registry/admin/UI tests including real test-run evidence, retention/boundary assertions and an ignored heavy 1,000,000-row query-plan test that passed locally.
  • Updated FR-6 docs, manual checklist, capability inventory and baseline manifest checksum.
  • Closed 10/10 code-review patch findings and moved Story 1.15 from review to done after just verify passed.

File List

  • _bmad-output/implementation-artifacts/1-15-invocation-history-usage.md
  • _bmad-output/implementation-artifacts/sprint-status.yaml
  • apps/admin-api/src/app.rs
  • apps/admin-api/src/dto.rs
  • apps/admin-api/src/main.rs
  • apps/admin-api/src/routes/observability.rs
  • apps/admin-api/src/service.rs
  • apps/admin-api/src/service/agents.rs
  • apps/admin-api/src/service/demo.rs
  • apps/admin-api/src/service/observability.rs
  • apps/admin-api/tests/integration.rs
  • apps/admin-api/tests/integration/community_access_usage.rs
  • apps/admin-api/tests/integration/logs_usage.rs
  • apps/ui/css/logs.css
  • apps/ui/html/logs.html
  • apps/ui/html/usage.html
  • apps/ui/js/api.js
  • apps/ui/js/i18n.js
  • apps/ui/js/logs.js
  • apps/ui/js/usage.js
  • apps/ui/tests/e2e/logs-usage.spec.js
  • crates/crank-core/src/observability.rs
  • crates/crank-metrics/src/schema.rs
  • crates/crank-metrics/tests/schema_v1.rs
  • crates/crank-registry/src/lib.rs
  • crates/crank-registry/src/model.rs
  • crates/crank-registry/src/postgres/mod.rs
  • crates/crank-registry/src/postgres/observability.rs
  • crates/crank-registry/tests/integration/agents_usage.rs
  • crates/crank-registry/tests/integration/common.rs
  • crates/crank-registry/tests/integration/observability.rs
  • docs/admin-api.md
  • docs/capability-baseline/manifest.json
  • docs/capability-inventory.json
  • docs/data-model.md
  • docs/manual-regression-checklist.md
  • docs/observability.md
  • docs/schemas/metrics-registry-v1.json

Change Log

  • 2026-08-23 — Created Story 1.15 with FR-6 history/usage implementation context.
  • 2026-08-23 — Implemented FR-6 Invocation History/Usage hardening and moved Story 1.15 to review.
  • 2026-08-23 — Closed code-review patch findings, re-ran full verification and moved Story 1.15 to done.