59 KiB
workflowStatus, totalSteps, stepsCompleted, lastStep, nextStep, lastSaved, inputDocuments
| workflowStatus | totalSteps | stepsCompleted | lastStep | nextStep | lastSaved | inputDocuments | |||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| completed | 5 |
|
step-05-generate-output | 2026-08-08 |
|
Test Design Progress — crank
Step 1: Mode and Prerequisites
- Mode: System-Level.
- Reason: одновременно доступны PRD, Architecture Spine/architecture decisions и полные Epics/Stories; по правилу workflow системный test design предшествует epic-level plans.
- PRD:
_bmad-output/planning-artifacts/prds/prd-crank-2026-08-03/prd.md. - Architecture:
_bmad-output/planning-artifacts/architecture/architecture-crank-2026-08-03/ARCHITECTURE-SPINE.md. - Story coverage:
_bmad-output/planning-artifacts/epics.md— 8 эпиков, 99 Story (25 исторически done, 74 backlog), 54 FR. Изменение декомпозиции не меняет факт или commit ранее выполненных проверок. - Prerequisite result: PASS; обязательные входы найдены.
Step 2: Loaded Context and Knowledge Base
- Configuration:
tea_use_playwright_utils=true,tea_use_pactjs_utils=false,tea_pact_mcp=none,tea_browser_automation=auto,test_stack_type=auto,test_artifacts=_bmad-output/test-artifacts. - Detected stack:
fullstack. Backend indicators: Rust workspace/Cargo, Axum/Tokio/SQLx/PostgreSQL. Frontend indicators:apps/ui/package.json, Playwright config and existing browser specs usingpage.goto/locators. - Product inputs: PRD plus addendum, Architecture Spine/ADRs, complete approved 8-epic/99-story plan and project-context rules. No required input is missing.
- Pinned technology: Rust 1.96.1 / Edition 2024 / MSRV 1.96, Axum 0.8, Tokio 1, SQLx 0.9, Reqwest 0.12, PostgreSQL 16, optional Valkey 8, Node 22, Alpine.js 3.15.12 and Playwright 1.61.1.
- Primary integration boundaries: Admin UI/API; version-isolated MCP server; REST upstream and OpenAPI external refs; PostgreSQL authority; optional Valkey acceleration; durable Task Worker; isolated Load Runner; artifact storage; GlitchTip/Sentry-compatible telemetry; Prometheus/Grafana; backup/PITR and signed offline release bundle.
- Test-design scope: all 54 FR, 30 NFR and 34 architecture requirements, including current REST→MCP regression, real-world OpenAPI corpus, Tools/Resources/Prompts and MCP revision matrix, synchronous/background execution, trace correlation, technical/business metrics, built-in Load Runs, deployment/recovery and one-RC release evidence.
Extracted NFR thresholds
- Reference workload: 100 RPS for 30 minutes and 250 RPS for 5 minutes; per-minute product success ≥99.95%, saturation rejection ≤0.1%.
- Capacity: ≥128 concurrent executions, 10,000 queued Tasks, and production dataset of 2,000 Operations / 200 Agents / 500 keys / 1,000,000 history records.
- Latency: platform overhead ≤50 ms p95 / ≤150 ms p99 against a 10 ms stub; discovery ≤150 ms p95; principal UI screens ≤2 s p95 over local LAN.
- Reliability: product success ≥99.95%, control-plane availability ≥99.9%; graceful shutdown ≤30 s; Task start ≤5 s p95; lease recovery ≤60 s; RPO ≤5 min and RTO ≤30 min.
- Observability: IDs on 100% of requests/records; traces generated for 100%; delivery ≥99.99% within 60 s and 100% completeness within 25 h for outages ≤24 h; 30 h spool capacity; overhead ≤5% p95 latency and ≤10% CPU.
- Security: default deny, encrypted/non-returnable secrets, per-connection SSRF enforcement, bounded resources/cardinality, fail-closed safety stores, no Critical/High vulnerabilities or findings, and no telemetry egress without opt-in.
- Compatibility/release: latest stable MCP at feature freeze plus N-1/N-2; ≥50 OpenAPI fixtures with ≥95% actionable Drafts and zero silent loss; 72 h soak; non-accelerable 30-day/≥1M-event burn-in; complete manual regression and signed evidence for one immutable RC.
Threshold completeness and deferred parameters
Critical release thresholds are measurable and sufficiently defined for system test design. Remaining values are implementation-time tuning parameters rather than missing product targets: PostgreSQL partition/index thresholds, Task polling/lease/batch/fairness values, exact Grafana panel layout/derived alert thresholds, digest-pinned observability images and feature-freeze MCP revisions. They must be frozen before the corresponding epic qualification tests and cannot weaken the NFRs above.
Loaded testing guidance
- System-level readiness uses the 8-category/29-criterion ADR checklist and objective PASS/CONCERNS/FAIL evidence.
- Risk scoring is probability × impact (1–9); score ≥6 requires owner/mitigation and score 9 blocks the gate.
- Coverage follows the lowest effective level: Rust unit/property tests for pure invariants, PostgreSQL/container integration tests for persistence/concurrency, protocol/API contract tests for boundaries, Playwright only for critical UI journeys, and dedicated load/security/recovery tools for NFRs.
- Tests must be deterministic, isolated, parallel-safe, self-cleaning, without hard waits, with explicit assertions, focused files and controlled test data.
- Contract testing is relevant to Admin/OpenAPI/MCP/runtime boundaries, but Pact.js Utils and Pact MCP are disabled; contracts must be verified against authoritative Rust handlers/types or OpenAPI rather than consumer assumptions.
- Browser exploration is not required in this system-level step. With automation=
auto, Playwright CLI remains the preferred lightweight selector/trace investigation path for later epic-level work, with trace artifacts retained on failure.
Step 3: Testability and Risk Assessment
🚨 Testability concerns
- Deterministic control plane is not yet a complete test contract. The repository has
crank-test-support, PostgreSQL Testcontainers, in-memory runtime stores and some seeded UI scenarios, but the target architecture still needs reusable builders/fixtures for every aggregate, controllable clocks/UUIDs, lifecycle transitions, cleanup and production-size datasets. Without these, Tasks, retention, leases and parallel runs will be slow or flaky. Required before Epic 4 qualification. - Fault injection is not an explicit shared facility. We need deterministic doubles/proxies for DNS rebinding, redirects, partial/oversized bodies, upstream timeouts/resets, PostgreSQL/Valkey interruptions, worker crash points, telemetry endpoint outage/corruption and disk-full/fsync failures. Each injected fault must be named and observable. Required before reliability/security evidence.
- Time-dependent Task semantics need virtual-time seams. Lease expiry, retry/backoff, deadlines, input-required timeout, 30-hour spool, 25-hour completeness and retention cannot depend on wall-clock sleeps. Ports for clock/scheduler and persisted timestamps must be controllable while production wiring remains monotonic/UTC-correct. Required during foundation/Task implementation.
- End-to-end observability assertions need queryable sinks. The architecture defines IDs, traces, GlitchTip, local records and metrics, but tests need bounded APIs/helpers to query local history, captured OTLP/Sentry envelopes, Prometheus exposition and Grafana provisioning by
request_id/trace_id. Direct dependence on a live external GlitchTip is insufficiently deterministic. Required before Epic 5. - Load generation must be independently measurable. The built-in runner shares the host with the SUT in the reference profile; its own CPU/memory/network use, heartbeat and kill path must be measured separately so generator saturation cannot be misreported as product failure. A stub upstream and externally cross-checked reference load tool are required. Required before Epic 6 evidence.
- Release evidence needs executable provenance from the start. Corpus/matrix/load/recovery/manual outputs must use stable schemas, content-addressed storage, deterministic commands and one-RC identity. Retrofitting this at Epic 8 risks unrepeatable or stale evidence. Establish schemas and collectors in Epic 1, extend per epic.
- Browser tests need parallel-safe state and selectors. Existing Playwright coverage proves some flows, but production qualification requires API-first seeding, per-worker namespaces, self-cleanup, RU/EN coverage, stable role/test-id selectors and retained traces. Avoid UI setup for non-UI behavior. Required before full manual/automated regression.
✅ Testability assessment summary
- Hexagonal boundaries and framework-free core permit low-cost unit/property/state-machine tests and replaceable adapters.
- PostgreSQL is the single correctness authority and Valkey is explicitly rebuildable/optional, enabling PG-only reference assertions and differential cache tests.
- Existing
crank-test-support, PostgreSQL 16 Testcontainers, runtime in-memory stores, Rust integration suites and Playwright tests provide a usable brownfield foundation. - Typed errors, events, metric registry, identity conventions and immutable revisions create deterministic assertion surfaces.
- The OpenAPI NormalizedIR/ImportPlan pipeline, version-isolated MCP adapters and unified runtime execution path expose natural corpus, contract and differential-test boundaries.
- Quantified NFRs, immutable evidence artifacts and a fail-closed release manifest allow objective gates rather than subjective sign-off.
Architecturally Significant Requirements (ASRs)
| ASR | Classification | Test-design consequence |
|---|---|---|
| PostgreSQL-authoritative Tasks with transactional dispatch, CAS leases and fencing | ACTIONABLE | Model/state-machine, crash-point, concurrency and real-PG tests; prove no stale terminal writes and explicit outcome_unknown. |
| Unified outbound pipeline with per-connection DNS/redirect policy and hard bounds | ACTIONABLE | Hermetic malicious-network lab; verify every ingress path uses identical policy. |
| Canonical W3C context across browser/MCP/runtime/worker and durable telemetry spool | ACTIONABLE | One cross-boundary golden E2E plus outage/replay/corruption/disk-capacity tests. |
| Immutable OpenAPI ParsedDocument→IR→Plan→Apply pipeline | ACTIONABLE | ≥50-spec corpus, metamorphic/determinism tests, transactional rollback and zero-silent-loss oracle. |
| Modern MCP adapters: stable N/N-1/N-2, Tools/Resources/Prompts and Tasks | ACTIONABLE | Protocol harness plus ≥3 real clients; adapter contract snapshots and negative negotiation tests. |
| Signed LoadAuthorizationPlan and isolated Load Runner | ACTIONABLE | Authorization/kill-switch tests and calibrated load/overload/soak evidence on reference hardware. |
| One migration authority, N/N-1 data compatibility, backup manifest and product-probe restore | ACTIONABLE | Upgrade/rollback matrix and three independent PITR restores measuring RPO/RTO. |
| One-time admin bootstrap, fail-closed security, encrypted secrets and air-gap bundle | ACTIONABLE | Abuse/negative tests, secret canaries, configuration matrix, SAST/SCA/container scans and default-deny egress test. |
| Alpine/vanilla JS remains the UI model | FYI | Use current bundle/build conventions; test async revision guards, safe DOM sinks, localization and accessibility without framework migration. |
| Single-instance drain/restart; HA/rolling upgrades deferred | FYI | Qualify claimed topology only and explicitly prevent HA claims in docs/evidence. |
Risk register
Scoring is Probability (P) × Impact (I), each 1–3. Scores ≥6 are high and require owned mitigation; score 9 blocks release while open.
| ID | Category | Risk | P | I | Score | Mitigation / evidence | Owner | Timeline |
|---|---|---|---|---|---|---|---|---|
| R-001 | TECH | Brownfield regressions remain hidden because current core and UI are not fully manually/automatically baselined. | 3 | 3 | 9 | Capability inventory, characterization tests and complete guided baseline before semantic changes. | QA Lead + Core | Epic 1, before feature branches |
| R-002 | DATA | OpenAPI normalization silently drops or mis-maps real constructs, security or schema branches. | 3 | 3 | 9 | Versioned ≥50-spec corpus, finding oracle, round-trip/metamorphic tests, immutable preview and atomic apply rollback. | Import Lead | Epic 2 gate |
| R-003 | TECH | Concurrent workers duplicate side effects or accept stale writes after crash/lease expiry. | 3 | 3 | 9 | Model-based Task lifecycle, deterministic crash points, PG contention suite, fencing/idempotency invariants and reconciliation evidence. | Task Lead | Epic 4 gate |
| R-004 | SEC | SSRF, redirect/proxy bypass or secret leakage crosses the closed-contour boundary. | 3 | 3 | 9 | Malicious DNS/HTTP lab on every execution path; secret canaries across logs/traces/metrics/UI/artifacts; fail-closed configuration tests. | Security Owner | Per-epic + RC security gate |
| R-005 | OPS | Trace delivery outage, spool overflow/corruption or unsafe replay makes UI/GlitchTip correlation incomplete while UI claims success. | 2 | 3 | 6 | 30 h capacity, 24 h outage/replay, corruption and disk-pressure tests; loss records; golden trace query/deep-link E2E. | Observability Lead | Epic 5 gate |
| R-006 | PERF | Reference workload breaches latency/success due to DB contention, metric cardinality, telemetry or load-runner interference. | 3 | 3 | 9 | Calibrated stub, baseline/telemetry differential, production-size query plans, 100/250 RPS runs, generator resource accounting and ceilings. | Performance Owner | Epics 5–6; RC repeat |
| R-007 | SEC | Admin/session/bootstrap/recovery paths allow unauthorized state changes or metadata disclosure. | 2 | 3 | 6 | Auth state-machine tests, CSRF/CORS/cookie/proxy/TLS matrix, rate-limit tests, cross-scope negative corpus and recovery revocation proof. | Security + Admin API | Epic 7 gate |
| R-008 | OPS | Upgrade, rollback or PITR restore yields schema/artifact/key mismatch despite services becoming healthy. | 2 | 3 | 6 | N/N-1 expand/migrate/contract matrix, consistency manifest and three timed restores ending in encrypted-auth MCP probe. | Release/SRE | Epic 7 gate |
| R-009 | BUS | Resources, Prompts or current/previous MCP clients observe inconsistent catalog revisions or unsupported semantics. | 2 | 3 | 6 | Immutable catalog snapshots, adapter golden contracts, negotiation/change-signal tests and ≥3-client protocol matrix. | MCP Lead | Epic 3 gate |
| R-010 | OPS | Built-in load test can overload the control plane, escape its target scope or fail to stop. | 2 | 3 | 6 | Signed authorization bounds, reserved control capacity, target allowlist, heartbeat/watchdog, Unix kill socket and forced-stop chaos tests. | Load Lead + Security | Epic 6 gate |
| R-011 | DATA | Metrics/business rollups double count or expose high-cardinality/user identifiers. | 2 | 3 | 6 | Typed registry lint, label allowlist, idempotent rollup replay/differential tests, cardinality budget and exemplar validation. | Metrics Lead | Epic 5 gate |
| R-012 | TECH | Flaky or over-broad E2E suite delays feedback and hides low-level defects. | 2 | 2 | 4 | Test pyramid, per-worker data, network-first waits, no hard sleeps, burn-in/quarantine policy and trace-on-failure. | QA Automation | Continuous |
| R-013 | OPS | Evidence from different builds/configurations is combined or signed despite being stale/incomplete. | 2 | 3 | 6 | Schema-validated manifest binds RC/image/config/topology/commands/digests; freshness checks and fail-closed verifier. | Release Owner | Framework Epic 1; final Epic 8 |
| R-014 | SEC | Air-gapped deployment contains vulnerable/untrusted images or performs undeclared egress. | 2 | 3 | 6 | Digest/signature/SBOM/license verification, fresh install/N-1 upgrade in disconnected lab and default-deny egress capture. | Supply-chain Owner | Epic 7/8 |
| R-015 | OPS | 30-day burn-in exposes leaks or corruption too late for planned publication. | 2 | 3 | 6 | Begin continuous nightly/burn-in runs as soon as foundations exist; 72 h soak precheck; trend memory, handles, queues, spool and DB growth. | QA/SRE | Start Epic 4; final Epic 8 |
NFR validation plan
| Category | Planned evidence sources | Threshold status |
|---|---|---|
| Security / privacy | Rust negative integration tests, malicious upstream lab, auth/browser E2E, secret-canary scans, SAST/SCA/SBOM/container/config scans, air-gap egress capture | Defined; no Critical/High and all default-deny/leak invariants are release blocking. |
| Performance / scalability | Criterion/k6-equivalent external calibration plus built-in runner, Prometheus snapshots, production-size DB plans, telemetry-on/off differential, 100/250 RPS and 128/10k concurrency variants | Defined numerically; hardware/topology fixed by Reference Deployment. |
| Reliability / recovery | Task model/crash tests, kill/restart/drain chaos, telemetry outage/replay, N/N-1 upgrade/rollback, three WAL/PITR restores with real MCP probe, 72 h soak and 30-day burn-in | Defined numerically; exact internal lease/poll tuning remains implementation-owned within limits. |
| Observability | Correlation contract tests, golden UI→API/MCP→runtime→upstream→history→GlitchTip flow, OTLP/Sentry capture, Prometheus schema/cardinality lint, Grafana provisioning/alert-rule tests | Defined; 100% identity/trace and delivery/completeness/overhead budgets apply. |
| Maintainability / compatibility | cargo fmt/clippy/test, architecture dependency checks, unsafe/panic lint, wire round-trips, OpenAPI corpus, MCP/client/browser/PG/Valkey matrices, localization lint and regression-per-defect rule |
Version matrix is frozen at feature freeze; no unsupported compatibility claim before evidence. |
| Compliance / release provenance | Content-addressed raw evidence, JSON-Schema manifest validation, signature/trust/rotation/revocation tests, RC/config/topology digest checks and manual sign-off | Defined as fail-closed Habr/production release gate; no waivers for mandatory capabilities or Critical/High failures. |
Highest-priority mitigation order
- Close R-001 with an immutable current-product baseline and establish reusable deterministic fixtures/evidence schemas.
- Develop R-002, R-003 and R-004 tests alongside OpenAPI, Task and outbound-security code; these are score-9 design drivers, not post-implementation QA.
- Build observability and performance measurement seams before optimizing (R-005/R-006/R-011), including independent load-generator accounting.
- Exercise upgrade/restore/security/offline paths continuously before the final RC (R-007/R-008/R-014), and start long-duration testing early enough to absorb R-015.
- The release gate remains FAIL while any score-9 risk is open or required evidence cannot be tied to the same immutable RC.
Step 4: Coverage Plan and Execution Strategy
Coverage matrix
The indicated level is the primary oracle. Defense-in-depth E2E is reserved for cross-boundary user outcomes; it does not repeat every branch already proven by unit/integration tests.
| ID | Requirements / risks | Atomic validation scenario | Primary level / tool | Priority | Planned automated cases |
|---|---|---|---|---|---|
| CP-01 | FR-1..6, R-001 | Characterize existing manual REST Operation → test → publish Agent → MCP discovery/call → history, including validation/upstream/store failures | API integration + 2 golden Playwright/MCP E2E | P0 | 30–45 |
| CP-02 | FR-1..6, AR-2/16/17 | Immutable revisions, publish snapshot atomicity, config bounds and migration compatibility/locking | Rust unit + PostgreSQL integration | P0 | 35–55 |
| CP-03 | FR-4/6, NFR-S1/S7 | Approval/idempotency state machines, deduplication, expiry, scope isolation and fail-closed store outage | Rust model/property + PostgreSQL contention | P0 | 25–40 |
| CP-04 | FR-5, NFR-S2 | Secret create/use/rotate/revoke lifecycle; ciphertext versioning; canary absent from every output channel | API/integration + artifact scanner | P0 | 25–40 |
| CP-05 | FR-7..12, R-002 | Parse/normalize/preview deterministic across YAML/JSON OAS 3.0/3.1/Swagger 2; explicit unsupported findings and zero silent loss | Rust corpus/property tests | P0 | 150–250 corpus assertions |
| CP-06 | FR-7..12 | Upload/encoding/size/ref limits, conflict selection, explicit partial mode and all-or-nothing apply/retry | API + PostgreSQL integration | P0 | 35–55 |
| CP-07 | FR-7..12 | User uploads, reviews findings, edits, applies and publishes representative OpenAPI specs | Playwright critical journey (RU/EN) | P1 | 8–14 |
| CP-08 | FR-013..16, NFR-C2 | MCP initialize/negotiation, list/call Tools, pagination/change signals, stable/N-1/N-2 wire contracts | Protocol harness + snapshots | P0 | 45–70 |
| CP-09 | FR-17..18 | Static/dynamic Resource draft/publish/list/read/templates; read-safe pinned Operation and sensitivity enforcement | Rust/API contract + MCP harness | P0 | 35–55 |
| CP-10 | FR-19..20 | Prompt draft/publish/list/get/render/completion; DAG validation, argument errors and side-effect freedom | Rust unit/property + MCP harness | P0 | 30–45 |
| CP-11 | FR-013..20, R-009 | ≥3 representative clients discover/use identical immutable catalog revisions across negotiated MCP versions | Cross-client E2E matrix | P0 | 18–30 |
| CP-12 | FR-21..26, R-003 | Task lifecycle model permits only legal/versioned/idempotent transitions and immutable terminal result | Rust model/property tests | P0 | 50–80 |
| CP-13 | FR-21..26 | Transactional enqueue, bounded SKIP LOCKED claim, CAS lease/fencing, retries, cancellation/input and outcome_unknown under contention/crash |
PostgreSQL integration/chaos | P0 | 45–70 |
| CP-14 | FR-21..26, NFR-P2/R5 | Sync/background fairness, 128 active + 10k queue, deadline propagation, overload/retry guidance and ≤60 s recovery | Load/concurrency harness | P0 | 12–20 profiles |
| CP-15 | FR-21..26 | Client starts/observes/cancels/resumes a Task through disconnect/restart; fallback on revisions without Tasks | MCP E2E | P1 | 10–16 |
| CP-16 | FR-27..31, R-005 | Canonical IDs/W3C propagation through browser/Admin/MCP/runtime/upstream/worker retry/history; safe boundary errors | Rust/API contracts | P0 | 30–45 |
| CP-17 | FR-27..31 | Golden request is searchable/copied in UI and deep-links to captured GlitchTip trace; loss record suppresses false claim | Playwright + MCP + telemetry-sink E2E | P0 | 8–12 |
| CP-18 | FR-27..31, NFR-O2/O3 | Telemetry outage, 30 h spool sizing, encrypted persistence, restart, replay ordering/idempotency, corruption/disk pressure and overhead | Soak/chaos/performance | P0 | 15–25 profiles |
| CP-19 | FR-32..35, R-011 | Metric registry names/types/units/labels/buckets stable; cardinality rejects IDs/arbitrary values | Rust unit/schema lint | P0 | 25–40 |
| CP-20 | FR-32..35 | Product events/UTC rollups exactly-once under replay; Prometheus exposition and exemplar correlation agree with source records | PostgreSQL/API differential | P0 | 25–40 |
| CP-21 | FR-32..35 | Grafana dashboards/recording rules/SLO denominators and fast/slow burn alerts provision and fire against synthetic series | Config-as-code test + Prometheus/Grafana integration | P1 | 18–28 |
| CP-22 | FR-36..40, R-010 | Load plan validates/authenticates target and ceilings; start/status/stop/forced kill are safe and auditable | Rust/API integration | P0 | 25–40 |
| CP-23 | FR-36..40, NFR-P1/P4 | Built-in runner results cross-check external generator at 100 RPS/30 m and 250 RPS/5 m; error classes and percentiles are accurate | Calibrated performance test | P0 | 8–12 profiles |
| CP-24 | FR-36..40 | Operator launches RPS/DAU run, sees live status, safely stops and inspects correlated results | Playwright critical journey | P1 | 6–10 |
| CP-25 | FR-41..45, R-007 | Fresh bootstrap/login/session/recovery and TLS/proxy/cookie/CSRF/CORS/default-deny matrix | API/browser security suite | P0 | 35–55 |
| CP-26 | FR-41..45, R-008 | Fresh install, preflight, N/N-1 migration, drain/restart, rollback compatibility and three PITR restores ending in real encrypted-auth MCP probe | Deployment/recovery harness | P0 | 18–28 workflows |
| CP-27 | FR-41..45, R-014 | Signed digest-pinned offline bundle installs/updates in air gap; SBOM/license/signature valid and undeclared egress absent | Supply-chain/air-gap system test | P0 | 12–20 |
| CP-28 | FR-46..50 | Complete regression, OpenAPI/MCP/client/browser/PG/Valkey matrices, static/security scans and production-size dataset | CI matrix | P0 | Matrix-defined |
| CP-29 | FR-49, R-015 | 72 h soak and non-accelerable 30-day ≥1M-event burn-in show no leak, corruption, queue/spool drift or SLO breach | Long-duration system test | P0 | 2 qualification runs + probes |
| CP-30 | FR-51..53 | Fresh user/operator follows quickstart, upgrade, recovery, alert and incident runbooks in connected and air-gapped profiles | Guided manual + docs executable checks | P1 | 20–35 checks |
| CP-31 | FR-54, R-013 | Manifest rejects missing/stale/cross-RC/digest-mismatched/untrusted/revoked evidence and accepts only complete signed RC | Rust/CLI schema and signature tests | P0 | 30–45 |
| CP-32 | NFR-C1/C3/C4, AR-31..34 | Dependency direction, MSRV, unsafe/panic rules, wire compatibility, Gitea CI and simultaneous RU/EN strings | Static/contract CI | P1 | 15–25 gates |
| CP-33 | UX/accessibility | Loading/error/empty/stale-response states, keyboard/focus/labels, responsive critical screens and safe DOM sinks | Component/Playwright + axe-equivalent | P1 | 30–50 |
| CP-34 | Exploratory | Unscripted real OpenAPI/client combinations, operator diagnostics and usability under partial failures | Time-boxed exploratory/manual | P2 | 15–25 charters |
NFR coverage and evidence artifacts
| NFR category | Validation scenarios | Evidence consumed later by nfr-assess |
|---|---|---|
| Security/privacy | CP-03/04/19/25/27/28 plus malicious-network and secret-canary suites | JUnit/JSON results, scanner SARIF, SBOM/signature reports, egress capture, redaction canary report and config digest |
| Performance/scalability | CP-14/18/20/21/23/28/29 on the checksummed Reference Workload | Raw request samples/histograms, generator and SUT resource series, Prometheus snapshot, query plans, run metadata and threshold verdict JSON |
| Reliability/recovery | CP-02/12/13/14/18/26/29 | State-model coverage, crash schedule/seed, Task invariants, telemetry accounting, upgrade/rollback logs, BackupManifest, measured RPO/RTO and product-probe output |
| Observability | CP-16..21 | Trace/ID correlation map, captured redacted envelopes, spool ledger/loss records, metric schema/cardinality report, dashboard/alert test results and overhead differential |
| Maintainability/compatibility | CP-01/05/08/11/28/31/32 | Rust/UI coverage, lint/architecture reports, corpus and protocol matrices, browser traces, signed manifest verifier results and regression index |
| Release provenance/manual quality | CP-28..31/34 | Content-addressed evidence tree for one RC, signed manifest, manual checklist with zero blocked/not-run, docs execution logs and defect disposition |
No critical NFR threshold is UNKNOWN. Feature-freeze versions, image digests and implementation tuning values are controlled gate inputs; absence at their freeze point blocks the corresponding qualification run.
Execution strategy
- PR (target ≤15 minutes, fail fast): formatting/lints/architecture checks; all changed-crate unit/property tests; targeted PostgreSQL integration; contract/corpus delta; P0 API/MCP smoke; Chromium critical UI smoke; manifest/schema tests. A full small suite may replace selection only while it stays below 15 minutes.
- Nightly: complete Rust and PostgreSQL/Valkey suites; full OpenAPI corpus; MCP adapters and representative-client matrix; Chromium full UI plus Firefox/WebKit critical paths; security negative suite; production-size dataset smoke; telemetry outage/replay short profile; flaky-test burn-in.
- Weekly: full browser/client matrices, dependency/container/security scans, high-contention Task chaos, migration/rollback, dashboard/alert integration, performance baseline and 100/250 RPS qualification rehearsal, spool/disk-pressure and backup/restore rotation.
- Release candidate: immutable RC only; all automated/manual matrices, three restore runs, air-gap fresh/N-1 flows, 72-hour soak, then non-accelerable 30-day burn-in. Any code/config/topology change creates a new RC and restarts freshness-dependent evidence.
Resource estimates
These ranges estimate design and implementation of the automated/manual test capability and qualification assets, not feature development or the unavoidable 30-day wall-clock gate.
| Priority | Estimated effort |
|---|---|
| P0 | ~900–1,400 engineering hours |
| P1 | ~300–520 engineering hours |
| P2 | ~80–160 engineering hours |
| P3 / optional benchmarks | ~30–70 engineering hours |
| Total | ~1,310–2,150 engineering hours |
With 3–4 effective engineers spanning QA automation, Rust/backend, SRE/performance and security, implementation is roughly 12–22 working weeks, followed/overlapped by the mandatory 30-day burn-in. A single engineer should plan roughly 8–13 person-months; long-running environments and manual qualification add calendar time but not equivalent active hours.
Quality gates
- P0 pass rate is 100% on every applicable PR and RC; no quarantine or waiver for mandatory P0 release evidence.
- P1 pass rate is ≥95% for development/nightly signal and 100% for RC scenarios mapped to FR-1..FR-54 or critical user journeys. Any remaining P1 failure requires explicit non-release classification; it cannot mask an NFR or mandatory capability.
- All score-9 risks must be mitigated/closed; every score 6–8 risk needs verified mitigation and no expired waiver. Critical/High security risks cannot be waived for release.
- Rust line coverage target is ≥80% workspace-wide, ≥90% for safety/domain state-machine crates, with branch/property/mutation evidence on Task, import, auth, admission, redaction and manifest logic. Coverage percentage never substitutes for requirement traceability.
- 100% of FRs, NFR categories and ASRs map to planned scenarios and later immutable evidence; no manual item may be blocked or not-run.
- OpenAPI corpus has zero silent loss and ≥95% actionable Drafts; all required MCP/client/browser/database/cache matrices pass.
- All stated performance, availability, recovery, observability and spool thresholds must have raw machine-readable evidence from the same RC/reference topology.
- Full NFR PASS/CONCERNS/FAIL decision is intentionally deferred to
bmad-testarch-nfrafter implementation evidence exists. - Habr/production release gate is fail-closed on missing/stale/cross-RC evidence, invalid signature/schema/digest, any open Critical/High defect or any mandatory SLO failure.
Step 5: Final Outputs and Validation
- Mode: System-Level; resolved execution mode: sequential.
- Architecture output:
_bmad-output/test-artifacts/test-design-architecture.md. - QA output:
_bmad-output/test-artifacts/test-design-qa.md. - BMAD handoff:
_bmad-output/test-artifacts/test-design/crank-handoff.md. - Validation: workflow checklist reviewed; template-required sections, risk IDs R-001..R-015, priority separation, NFR planning boundary, PR/nightly/weekly strategy, ranges, fenced code and placeholders checked. No browser session was opened and no temporary artifact exists outside the test-artifacts tree.
- Open controlled inputs: feature-freeze MCP revisions and image digests, implementation tuning values, named owners where the repository does not identify a person, and provision of reference/performance/security environments. Missing any of these at its freeze/qualification point is a blocker, not an implicit default.
- Next workflow: epic-level Test Design or ATDD for the first implementation story; final NFR verdict waits for implementation evidence.
Epic-Level Test Design Progress — Epic 1
Run 2 / Step 1: Mode and Prerequisites
- Mode: Epic-Level.
- Scope: Epic 1 — «Надёжный базовый REST → MCP продукт»; выбран первым по утверждённому dependency order.
- Reason: system-level TD завершён, полный Epic 1 со Story и acceptance criteria доступен в
_bmad-output/planning-artifacts/epics.md. - Architecture context: Architecture Spine, PRD, project context and system-level test-design outputs are available.
- Prerequisite result: PASS; дополнительных материалов от пользователя не требуется.
Run 2 / Step 2: Epic 1 Context and Existing Coverage
- Configuration: Playwright Utils enabled; Pact.js Utils disabled; Pact MCP disabled; browser automation
auto; detected stackfullstack; artifacts root_bmad-output/test-artifacts. - Loaded product scope: Epic 1 Stories 1.1–1.16 with all acceptance criteria; PRD/Architecture Spine/project context; completed system architecture/QA TD and handoff.
- Epic requirements: FR-1..FR-6, FR-27, FR-32, FR-33, FR-43, FR-45 and FR-46; inherited security, performance, reliability, observability and compatibility NFRs.
- Integration boundaries: Community scope validator/docs; typed configuration and process startup; PostgreSQL migration authority; Admin/MCP identity propagation; metric registry; Operation/Agent lifecycle; runtime/REST adapter; DNS/network boundary; Auth Profiles/machine/master keys; Admin auth/session; approvals/idempotency; history/usage/CSV/retention; real MCP onboarding.
Existing coverage and reusable patterns
crank-test-supportsupplies PostgreSQL 16 Testcontainers and unique schemas; registry, Admin API and MCP suites already use real PostgreSQL.- Existing Admin API integration proves create/test/publish/archive Operations, Agent binding/publish/unpublish/archive, Auth Profile import/rotation, keys/revocation, Logs/Usage and request context.
- MCP integration proves published-tool discovery/call, catalog refresh/scope, direct/search behavior, transport negotiation/session, approvals, rate limits, request/trace context and uncertain approval recovery.
- Runtime tests cover confirmation single-use, idempotency contention/conflict, fail-closed coordination and uncertain outcomes; REST adapter tests cover private-target defaults, redirects and response size limits.
crank-metrics,crank-observabilityandcrank-tracehave contract tests for closed labels, cardinality, redaction, propagation, Prometheus protection, OTLP config and history-loss signals.- Existing Playwright specs cover Operations, Agents, keys, secrets, login, Logs/Usage and observability, but use Chromium only, one worker, retries only in CI and
trace: on-first-retry.
Confirmed coverage gaps
- No executable Capability Inventory schema/validator/evidence mapping or frozen exploratory baseline for Story 1.1–1.2.
- No demonstrated single
crank-configauthority covering every process, alias/deprecation conflict, generated docs and boundary scan (1.3). - Current migration coverage does not yet prove the full append-only checksummed authority, legacy consolidation, concurrent migrator/failure/resume and N/N-1 contract (1.4).
- Correlation tests exist, but the full 100% UI/Admin/MCP/runtime/history contract and hostile header matrix are incomplete (1.5).
- Metrics foundation exists, but million-distinct-value cardinality budget, direct-declaration workspace scan and measured overhead remain to prove (1.6).
- Operation/Agent happy lifecycle exists, while immutable revision conflict/YAML semantic roundtrip/stale UI/cross-scope/production-size discovery cases remain incomplete (1.7/1.13).
- Outbound tests do not yet form the complete DNS-rebinding/actual-connect/IPv4+IPv6/redirect credential stripping/proxy/encoded-host/deadline/concurrency matrix (1.9).
- Auth Profile/key basics exist; one-time UI disclosure persistence, wrong-purpose/cross-scope/cache-loss matrix and master-key epoch/re-encryption/crash/abort/revocation lifecycle are incomplete (1.10–1.11).
- Login/rate-limit coverage exists, but bootstrap race/replay, session fixation, CSRF/Origin/trusted-proxy/TLS, recovery misuse and password-rotation revocation need system proof (1.12).
- Approval coverage is substantial; canonical full-scope fingerprint, conditional races, exact single-use dispatch and store-readiness coupling need explicit real-PG assertions (1.14).
- History/Usage basic flows exist; bounded preview redaction, CSV injection, UTC boundary/no-double-count, retention and 1M-record/≤2s evidence are missing (1.15).
- Getting Started checklist and first-call authoritative/resumable/accessibility flow are not present (1.16).
- Browser suite is not parallel-safe by configuration and does not yet retain trace on first failure; this blocks the eventual fast PR target until per-worker data/ports exist.
Knowledge and exploration
- Applied required epic-level risk governance, probability-impact, test-level, priority and NFR guidance, plus prior system TD.
- Contract testing remains relevant for Admin/MCP/runtime boundaries; provider Rust handlers/types are authoritative.
playwright-cliis not installed and no bound-browser MCP is configured, so browser exploration was skipped per workflow fallback. No browser session or exploration artifact was created.- Loaded-input result: complete; no user-supplied material is missing.
Run 2 / Step 3: Epic 1 Risk and NFR Planning
Epic 1 risk register
| ID | Category | Real failure risk | P | I | Score | Mitigation / evidence | Owner | Timeline |
|---|---|---|---|---|---|---|---|---|
| E1-R-001 | TECH | Capability Inventory/baseline marks gaps or flaky/manual flows as production-ready, allowing regressions to propagate through every later epic. | 3 | 3 | 9 | Strict schema/validator negative matrix, checksum baseline, traceable automated/manual evidence and blocked Critical/High flows. | QA Lead + Release | Stories 1.1–1.2 |
| E1-R-002 | OPS | Multiple env readers/defaults diverge across Admin/MCP/worker, causing unsafe or unreproducible startup. | 3 | 3 | 9 | One typed schema, cross-process golden config, alias-conflict/redaction/boundary tests and fingerprint comparison. | Config/Core Lead | Story 1.3 |
| E1-R-003 | DATA | Migration authority changes baseline, races, partially applies or prevents N/N-1 recovery. | 2 | 3 | 6 | Fresh/baseline/current/concurrent/failure PG matrix, immutable checksum and expand/migrate/contract fixtures. | Registry Lead | Story 1.4 |
| E1-R-004 | OPS | Request/Trace IDs are missing, attacker-controlled, conflated or lost between UI/Admin/MCP/runtime/history. | 2 | 3 | 6 | Boundary contract/property tests plus one real UI/Admin/MCP/upstream/history correlation chain at sampling off/on. | Observability Lead | Story 1.5 |
| E1-R-005 | PERF | Dynamic metric labels/cardinality or instrumentation overhead exhausts resources before later observability work. | 2 | 3 | 6 | Compile/API restriction, workspace scan, million-value stress, schema snapshots and overhead baseline. | Metrics Lead | Story 1.6 |
| E1-R-006 | DATA | Editing/importing/archiving an Operation mutates published Agent behavior or historical evidence. | 2 | 3 | 6 | Revision/state model, PG conflicts, YAML semantic roundtrip and Agent snapshot regression. | Operation Lead | Story 1.7 |
| E1-R-007 | TECH | Admin Test and MCP execute different targets/pipelines or classify the same failure differently. | 3 | 3 | 9 | Table-driven parity contracts for success/all stages, dependency-boundary guardrails and outbound call counters. | Runtime Lead | Story 1.8 |
| E1-R-008 | SEC | DNS/redirect/proxy/header/retry bypass enables SSRF, credential forwarding or resource exhaustion. | 3 | 3 | 9 | Hermetic malicious-network matrix against both Admin and MCP paths; actual-connect validation and no-secret outputs. | Security + REST Adapter | Story 1.9 |
| E1-R-009 | SEC | Credentials/master-key material leak, wrong-purpose/cross-scope keys work, or rotation makes stored secrets unreadable. | 3 | 3 | 9 | Secret canaries, full credential matrix, ciphertext corruption and crash-resumable epoch rotation with verified revocation. | Security + Registry | Stories 1.10–1.11 |
| E1-R-010 | SEC | Remote bootstrap takeover, session fixation, CSRF/proxy spoofing or recovery bypass compromises Admin UI. | 2 | 3 | 6 | Auth state-model/concurrency tests, web-security config matrix, browser flow and local-only recovery proof. | Auth/Security Lead | Story 1.12 |
| E1-R-011 | DATA | Agent publication/cache exposes Draft/foreign/stale Tools or changes exact published versions. | 2 | 3 | 6 | Atomic PG publish/conflict, exact-version snapshots, cache-loss differential, scope negatives and dataset discovery test. | Agent/MCP Lead | Story 1.13 |
| E1-R-012 | BUS | Approval races/replay/retries produce duplicate or unauthorized side effects or falsely claim exactly-once. | 3 | 3 | 9 | Real-PG concurrency model, single-use admission, scope fingerprint, store-outage and uncertain-dispatch assertions. | Approval/Runtime Lead | Story 1.14 |
| E1-R-013 | SEC | History/CSV/UI exposes sensitive data or foreign metadata; aggregation double-counts and large datasets time out. | 2 | 3 | 6 | Redaction/property/cross-scope/CSV injection/UTC boundary/retention tests and 1M-record query plans/UI timings. | History/Admin Lead | Story 1.15 |
| E1-R-014 | BUS | Onboarding uses browser flags or test bypass, loses one-time key, duplicates mutations or never proves a real MCP call. | 2 | 2 | 4 | Server-authoritative checklist, real public endpoint E2E, refresh/tab/stale/lost-key/recovery and accessibility/usability run. | Product/UI Lead | Story 1.16 |
| E1-R-015 | TECH | Serial, shared-port/state browser tests mask races and cannot meet PR feedback goals. | 2 | 2 | 4 | Per-worker data/ports, API-first fixtures, trace-on-failure, sharding and deterministic burn-in before raising workers. | QA Automation | Epic 1 continuous |
NFR planning for Epic 1
| Category | Applicable threshold / invariant | Planned evidence source | Status |
|---|---|---|---|
| Security/privacy | Default deny; no foreign metadata; secrets encrypted/non-returnable and absent from logs/traces/metrics/reports/DOM; SSRF policy on every actual connection; no Critical/High findings | Rust/API/MCP security suites, malicious network, secret-canary artifact scan, Playwright auth/key flows, SAST/SCA | Defined; P0 |
| Performance/scale | Tool discovery ≤150 ms p95 at 200 Agents/2,000 Operations; Logs/Usage interactive ≤2 s p95 at 1M history records; bounded resource use/cardinality | Criterion/external timing harness, PG plans, browser timings, metric-series/heap/CPU measurements | Defined except exact metric series budget |
| Reliability/data | Published Operation/Agent revisions, approvals and config changes durable; migration transactional/checksummed; safety stores fail closed | PostgreSQL contention/failure/restart tests, checksum audit, cache-loss differential and execution call counters | Defined invariants |
| Observability | 100% Epic 1 ingress and Execution/history records have separate Request/Trace IDs; bounded labels; no secret payloads | Correlation contract/E2E, persistence queries, metric schema/cardinality and redaction reports | Defined |
| Compatibility/maintainability | Rust/MSRV/boundaries unchanged; existing wire behavior retained; new user text RU+EN; regression per defect | just verify, boundary/health scripts, wire snapshots, localization/static checks, baseline inventory |
Defined |
| UX/business | Fresh manual REST path reaches real first MCP call in ≤15 minutes; all async flows expose loading/error/empty/recovery and stale-response protection | Guided usability timing and real-stack Playwright RU/EN/accessibility suite | Defined |
Controlled UNKNOWN inputs
- Metric cardinality budget: Story 1.6 says series count stays “in budget” but does not give a numeric maximum. Before Story 1.6 implementation, the Metrics owner must freeze per-process/per-metric series ceilings and measurement window; E1-R-005 remains open until then.
- Session/bootstrap/backoff durations and history retention: exact values are typed configuration decisions. Tests must validate declared bounds/defaults/precedence and boundary behavior; absence from the schema is a blocker under E1-R-002/E1-R-010/E1-R-013, but values are not guessed here.
- Reference usability sample/protocol: the ≤15-minute outcome is defined, while participant count and timing procedure must be frozen before Story 1.16 qualification; tracked under E1-R-014.
Highest-priority mitigation order
- E1-R-001 and E1-R-002 first: truthful baseline and one config contract are prerequisites for meaningful evidence from every later Story.
- E1-R-003/004/005 establish durable schema, identity and metrics before capability state changes.
- E1-R-006/007/008 lock immutable execution semantics and the outbound security boundary before adding credentials and Agents.
- E1-R-009/010 close secret/admin takeover risks before production exposure.
- E1-R-011/012/013 prove catalog, side-effect and diagnostic integrity; E1-R-014 validates first-user value only after those authoritative contracts exist.
- Epic 1 gate is FAIL while any score-9 risk is open; implementation evidence, not existing partial tests, determines mitigation status.
Run 2 / Step 4: Epic 1 Coverage and Execution Plan
Atomic coverage matrix
The primary level owns the detailed oracle. E2E validates only cross-boundary user outcomes and does not repeat domain branches.
| Test group | Story | Atomic scenario / oracle | Primary level | Priority | Risk | New/extended cases |
|---|---|---|---|---|---|---|
| E1-01 | 1.1 | Inventory schema accepts every required field/status and rejects duplicate/missing/unknown/broken evidence deterministically | Unit/CLI contract | P0 | E1-R-001 | 15–25 |
| E1-02 | 1.1 | Community scope positive/negative matrix includes target capabilities, rejects enterprise markers and explicitly scans untracked inputs | Python/unit + boundary CI | P0 | E1-R-001 | 10–18 |
| E1-03 | 1.2 | Inventory maps every existing UI/Admin/MCP flow and never promotes failed/flaky/skipped/manual-only/gap evidence | CLI/integration | P0 | E1-R-001 | 12–20 |
| E1-04 | 1.2 | Current REST Operation→test→publish→Agent→key→public MCP call→history passes with safe checksummed evidence | Real-stack API/MCP E2E | P0 | E1-R-001 | 3–6 |
| E1-05 | 1.2 | Bounded exploratory RU/EN pass captures loading/empty/error/recovery/stale/unsafe-output gaps with severity/owner | Guided manual/Playwright support | P1 | E1-R-001 | 10–16 charters |
| E1-06 | 1.3 | Typed config default/min/max/invalid/cross-field/unknown and alias precedence/conflict are identical across processes | Rust unit/property + app contract | P0 | E1-R-002 | 35–55 |
| E1-07 | 1.3 | Generated env/docs/effective fingerprint are stable; diagnostics redact every secret; direct env reads fail boundary scan | Snapshot/static/integration | P0 | E1-R-002 | 18–30 |
| E1-08 | 1.4 | Fresh, immutable baseline, sequential current, already-current and checksum mismatch states are deterministic and preserve data | PostgreSQL integration | P0 | E1-R-003 | 18–28 |
| E1-09 | 1.4 | Concurrent migrators, failed transaction, incompatible/partial legacy state and bounded resumable backfill fail safely | PostgreSQL contention/fault | P0 | E1-R-003 | 12–20 |
| E1-10 | 1.5 | Missing/malformed/oversized/multiple/untrusted IDs create separate UUIDv7 Request ID and valid W3C Trace ID with safe boundary response | Unit/API/MCP contract | P0 | E1-R-004 | 20–32 |
| E1-11 | 1.5 | System correlation overrides user mappings and persists through Admin/MCP/runtime/upstream/history for success/auth/retry/failure and sampling-off | Cross-boundary integration/E2E | P0 | E1-R-004 | 10–16 |
| E1-12 | 1.6 | Typed metric schema names/types/units/help/labels/buckets snapshot is closed; direct declarations/aliases fail workspace scan | Unit/static contract | P0 | E1-R-005 | 20–30 |
| E1-13 | 1.6 | Million distinct hostile values stay inside frozen series budget; exemplars do not alter aggregates; overhead is measured | Stress/benchmark | P0 | E1-R-005 | 8–14 profiles |
| E1-14 | 1.7 | Operation state/revision model rejects illegal/stale/cross-workspace mutations and keeps Published Version immutable | Unit/property + PostgreSQL | P0 | E1-R-006 | 30–45 |
| E1-15 | 1.7 | Draft test, publish/edit/archive, YAML semantic roundtrip/upsert/conflict/malformed/oversized and history preservation work through Admin API | API/PG integration | P0 | E1-R-006 | 25–40 |
| E1-16 | 1.7 | RU/EN lifecycle UI proves stale-response guards, confirmations, safe IDs and minimal immutable-publish/YAML journey | Playwright | P1 | E1-R-006 | 8–14 |
| E1-17 | 1.8 | Admin Draft and MCP exact Published snapshot enter one RuntimeExecutionRequest path and make only expected adapter calls | Rust/API/MCP integration | P0 | E1-R-007 | 18–28 |
| E1-18 | 1.8 | Success and schema/mapping/auth/persistence/adapter/upstream/runtime/timeout/cancel/uncertain failures have parity in stage/code/retryability/IDs | Table-driven contract | P0 | E1-R-007 | 25–40 |
| E1-19 | 1.8 | Architecture checks reject app/runtime/adapter/persistence dependency bypass through alias/re-export/helper | Static boundary tests | P0 | E1-R-007 | 8–14 |
| E1-20 | 1.9 | IPv4/IPv6 forbidden classes, encoded/embedded hosts, DNS rebinding and actual retry addresses are blocked before bytes leave | Malicious-network integration | P0 | E1-R-008 | 30–50 |
| E1-21 | 1.9 | Redirect hop revalidation/credential stripping, proxy env, reserved headers, byte/deadline/concurrency limits and write retry rules are fail closed | REST adapter + Admin/MCP contract | P0 | E1-R-008 | 30–45 |
| E1-22 | 1.10 | Auth Profile/key create/list/get/export/rotate/revoke enforce encryption, one-time disclosure, exact purpose/scope and immediate revocation/cache loss | API/PG/MCP security | P0 | E1-R-009 | 30–50 |
| E1-23 | 1.10 | RU/EN key UI does not restore raw secret via refresh/back/DOM/storage/telemetry and requires destructive confirmation | Playwright + artifact canary | P0 | E1-R-009 | 10–16 |
| E1-24 | 1.11 | Process fingerprint/epoch readiness, ciphertext version/integrity/corruption and non-secret backup reference behave safely | Rust/PG integration | P0 | E1-R-009 | 20–35 |
| E1-25 | 1.11 | Rotation preflight/re-encrypt/checkpoint/crash/resume/verify/promote/abort/old-key revoke is idempotent with concurrent writes | State-machine + PostgreSQL fault | P0 | E1-R-009 | 25–40 |
| E1-26 | 1.12 | Bootstrap one-time/race/replay, login constant shape/backoff, session entropy/rotation/expiry/revocation and password change are correct | Rust/API/PG concurrency | P0 | E1-R-010 | 30–50 |
| E1-27 | 1.12 | TLS/trusted proxy/spoofed headers/cookies/CSRF/CORS/hostile Origin and local recovery matrix is default deny | API/security integration | P0 | E1-R-010 | 25–40 |
| E1-28 | 1.12 | RU/EN browser login/logout/throttle/expiry/recovery states use shared guard and reveal no account/security details | Playwright | P1 | E1-R-010 | 10–16 |
| E1-29 | 1.13 | Agent draft/publish transaction, exact bindings/revision, concurrent conflict, pinning/unpublish/archive and stale search result preserve catalog | PostgreSQL + MCP integration | P0 | E1-R-011 | 30–45 |
| E1-30 | 1.13 | Cross-Agent/workspace/key scope and cache-loss differential expose no metadata; direct/search discovery ≤150 ms p95 at 200/2k | Security integration + performance | P0 | E1-R-011 | 20–30 + 4 profiles |
| E1-31 | 1.13 | RU/EN Agent UI displays exact Draft/Published/revision and handles conflicts/stale responses safely | Playwright | P1 | E1-R-011 | 8–14 |
| E1-32 | 1.14 | Canonical full-scope fingerprint deduplicates pending approval; concurrent decision/expiry/recovery permits one legal transition | Rust model + PostgreSQL race | P0 | E1-R-012 | 30–45 |
| E1-33 | 1.14 | Approval/key scope/replay/revalidation/store outage/idempotency mismatch and uncertain dispatch never produce duplicate/false exactly-once side effect | Runtime/MCP integration | P0 | E1-R-012 | 25–40 |
| E1-34 | 1.14 | RU/EN approval UI has safe summary/IDs, expiry/conflict/stale/confirmation states | Playwright | P1 | E1-R-012 | 8–12 |
| E1-35 | 1.15 | History redaction/bounds/UTF-8 marker/persistence-loss metric and cross-scope filters/details are correct | Property + API/PG | P0 | E1-R-013 | 25–40 |
| E1-36 | 1.15 | UTC ranges/percentiles/no overlap, CSV formula escaping/limits and retention preserve audit/aggregates/evidence | Unit/API/PG integration | P0 | E1-R-013 | 20–35 |
| E1-37 | 1.15 | 1M-record plans and Logs/Usage UI meet ≤2 s p95 with pagination, stale guards and RU/EN states | Performance + Playwright | P0 | E1-R-013 | 6 profiles + 8–12 |
| E1-38 | 1.16 | Server-authoritative checklist derives each step, survives refresh/workspace/tab/revision changes and never duplicates publish/key creation | API/component/Playwright | P1 | E1-R-014 | 18–30 |
| E1-39 | 1.16 | Fresh user reaches first real public MCP call ≤15 min; lost/revoked key and validation/auth/upstream failure recover safely | Real-stack Playwright + guided usability | P1 | E1-R-014 | 8–14 + usability sample |
| E1-40 | 1.16 | Keyboard/focus/screen-reader/reduced-motion, RU/EN and bounded ProductEvent denominators/opt-in pass | Accessibility/component/API | P1 | E1-R-014 | 15–25 |
| E1-41 | All | Existing test fixtures are deterministic, isolated and traceable; selected Playwright suites pass repeated burn-in without order/state dependence | Test-quality audit/burn-in | P1 | E1-R-015 | 8–12 harness checks |
NFR evidence artifacts
| Category | Coverage groups | Evidence for later NFR assessment |
|---|---|---|
| Security/privacy | E1-10/11, E1-20..28, E1-30, E1-32..37 | JUnit/JSON, malicious-network capture, secret-canary scan, web-security matrix, SAST/SCA and browser traces |
| Performance/scale | E1-13, E1-30, E1-37 | Metric-series/CPU/memory report, raw discovery/UI timings, PG query plans and checksummed dataset/topology |
| Reliability/data | E1-08/09, E1-14/15, E1-24/25, E1-29, E1-32/33/35/36 | Migration/checksum logs, state-model/concurrency seeds, call counters, ciphertext/rotation audit and cache-loss differential |
| Observability | E1-10..13, E1-18, E1-35 | Correlation map, persistence queries, trace capture, metric schema/cardinality and redaction/history-loss reports |
| Compatibility/maintainability | E1-01..07, E1-19, E1-41 | Inventory/boundary/just verify results, wire/config snapshots, test-quality and flake report |
| UX/business | E1-05/16/23/28/31/34/37..40 | Playwright traces/screens, accessibility report, RU/EN coverage and timed usability protocol/results |
Execution strategy
- PR, target ≤15 minutes: all changed-crate unit/property/static tests; targeted real-PG/API/MCP integration; inventory/scope/config/boundary contracts; changed Playwright spec plus shared auth/API smoke. Run all functional coverage when it remains within the target; otherwise select by changed contract and always include its P0 boundary cases.
- Nightly: full Epic 1 Rust/PG/Valkey/Admin/MCP suites, all current Chromium Playwright, full security negative subset, correlation chain, million-value cardinality stress, migration/rotation/approval contention and repeated flake burn-in.
- Weekly: malicious-network matrix, production-size 200/2k/1M performance profiles, browser security configuration matrix, master-key crash/recovery, fresh-user guided usability/accessibility and full exploratory baseline refresh where changes require it.
- Epic qualification: all 1.1–1.16 acceptance criteria traced; complete Inventory and manual checklist; all P0/P1 evidence bound to one Epic candidate digest. Full product RC soak/burn-in remains Epic 8, not duplicated here.
Incremental resource estimate
Estimate covers closing Epic 1 test gaps and integrating evidence, reusing existing suites. It excludes feature implementation and later full-product qualification.
| Priority | Effort range |
|---|---|
| P0 | ~420–700 engineering hours |
| P1 | ~180–320 engineering hours |
| P2 exploratory extensions | ~40–90 engineering hours |
| P3 optional benchmarks | ~15–40 engineering hours |
| Total | ~655–1,150 engineering hours |
For 2–3 effective contributors across Rust/security/SRE/QA, this is approximately 7–14 working weeks, overlapped with Story implementation.
Epic 1 quality gates
- P0 = 100%; P1 ≥95% during development and 100% for Story acceptance/epic qualification.
- Every Story 1.1–1.16 has bidirectional requirement→test→evidence mapping; no acceptance criterion is silently manual or skipped.
- E1-R-001/002/007/008/009/012 (score 9) are closed by evidence; all score 6 risks have verified mitigation/owner.
- No Critical/High defect or security finding; secret-canary and scope-isolation scans are clean.
- Workspace coverage remains ≥80%, and newly introduced critical state/security/config/migration code targets ≥90% branch/contract evidence; coverage cannot replace traceability.
- Catalog discovery and Logs/Usage thresholds pass on checksummed production-size datasets; metric budget is frozen and passed before Story 1.6 closes.
- Existing REST→MCP baseline remains green, and no toolchain/dependency/wire change is mixed into the Epic without its own compatibility change.
- Final NFR PASS/CONCERNS/FAIL remains deferred to the NFR assessment after implementation evidence exists.
Run 2 / Step 5: Epic 1 Output and Validation
- Mode: Epic-Level; Epic 1; single-worker sequential output as required for epic-level mode.
- Output:
_bmad-output/test-artifacts/test-design-epic-1.md. - Validation: template/checklist sections, E1-R-001..E1-R-015, E1-01..E1-41, priorities, NFR planning boundary, estimates, quality gates, placeholders and Markdown structure checked.
- Browser hygiene: Playwright CLI was unavailable, no browser session was opened, and no temporary exploration artifact was created.
- Open controlled inputs: numeric metric series budget, typed security/retention configuration values and Story 1.16 usability protocol must be frozen by their owning Story; no values were guessed.
- Recommended next workflow: ATDD for Story 1.1, followed by its implementation; Epic-level trace/NFR verdict waits for implementation evidence.