29 KiB
workflowStatus, totalSteps, stepsCompleted, lastStep, nextStep, lastSaved, lastEdited, editScope, auditedRevision, workflowType, inputDocuments
| workflowStatus | totalSteps | stepsCompleted | lastStep | nextStep | lastSaved | lastEdited | editScope | auditedRevision | workflowType | inputDocuments | |||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| completed | 5 |
|
step-05-generate-output | 2026-09-13 | 2026-09-13 | system-level inventory and evidence-aware strategy refresh | 083974c27f |
testarch-test-design |
|
Test Design for QA: Crank Community Production Readiness
Purpose: Execution recipe for proving Crank Community production readiness. Architectural requirements and production-code mitigations are in test-design-architecture.md.
Date: 2026-09-13 (original design: 2026-08-08)
Author: BMad TEA / Bsodfather
Status: System-level design updated; current evidence inventory precedes epic-level planning
Project: crank
Executive Summary
Scope covers all 54 FRs, 30 NFRs and the eight production-readiness epics. There are 15 risks: 14 high (five score 9), one medium and none low. Dominant categories are operations, security, technical/concurrency, data integrity and performance.
The plan retains 34 scenario families, not a promise to write 850–1,250 additional tests. Existing tests must be reused. P0 denotes a security/data-integrity/core-operation failure with no safe workaround; ordinary feature variants are P1, not automatically P0. E2E checks cross-boundary outcomes rather than repeating parser or state-machine branch tests.
Audit boundary: main 083974c27f3a21a15807dd248bb6bc4fe04a5cd8. Story 2.9 candidate f8ee7c9bfeaa8a3f73a4850ff3f82b2bc2155fe8 is recorded separately and is not integrated into main. This update is a source/configuration audit, not a fresh product test run.
Not in Scope
| Item | Reason | Mitigation |
|---|---|---|
| HA/multi-instance rolling upgrade | Explicit post-launch scope | Document single-instance claim and reject HA evidence claims |
| S3-compatible artifacts, SSO/RBAC, non-REST adapters | Deferred by architecture | Preserve ports/invariants; revisit in later test design |
| Arbitrary distributed load targets or loads above built-in ceilings | Unsafe/outside Community goal | Load runner allowlists only Crank MCP and enforces signed ceilings |
| MCP revisions published after feature freeze | Unbounded moving target | Enter next adapter/compatibility cycle |
Dependencies and Test Blockers
Backend/architecture dependencies
- B-001 deterministic control plane: typed factories/builders, scoped cleanup, controllable time/IDs/scheduler.
- B-002 named fault injection: network, stores, worker, telemetry and disk failure controls.
- B-003 queryable observability sinks: captured trace/envelope/metric/dashboard state searchable by IDs.
- B-004 evidence contract: stable artifact schemas and RC provenance collectors.
- B-005 measurable load runner: calibrated stub/external generator, separate resource metrics and reliable kill path.
QA infrastructure setup
- Extend
crank-test-supportwith PostgreSQL 16/Valkey/telemetry/malicious-upstream fixtures and per-worker schemas. - Provide seeded production-size and minimal datasets with deterministic seeds and automatic cleanup.
- Create MCP protocol/client matrix harness, OpenAPI corpus runner, browser storage-state fixtures and artifact collectors.
- Provision PR ephemeral dependencies, nightly full topology, weekly reference hardware/air-gap lab and immutable RC qualification environment.
- Retain JUnit/JSON, Playwright traces, raw metrics, logs and content-addressed evidence under
_bmad-output/test-artifacts/or the release evidence store.
Actual Tooling vs Configuration
tea_use_playwright_utils=true and tea_use_pactjs_utils=true are workflow preferences, not installed dependencies. The UI package currently contains @playwright/test 1.61.1 and esbuild; neither Utils nor Pact is installed. This design does not install or mandate a wholesale test rewrite. Retain working fixtures; evaluate additional tooling only for an identified gap.
Existing Test Inventory
Presence below means inspected test source, not current pass or a coverage percentage.
| Surface / FR | Representative existing source | What it exercises | Missing qualification / next evidence |
|---|---|---|---|
| Config, FR-45 | crates/crank-config/tests/{contract,generation,redaction}.rs; app tests/config_startup.rs |
Config bounds, deterministic contracts, read-only startup, safe failures | Production-profile matrix on a bound artifact |
| Auth/secrets, FR-3/5 | apps/admin-api/tests/integration/{auth_rate_limit,credential_lifecycle,secrets_import_auth}.rs; runtime tests/integration/{confirmation,idempotency}.rs |
CSRF/origin/rate-limit, lifecycle, encrypted credentials, approval/idempotency | Whole web-security and deployed secret-canary matrix |
| Operation/Agent lifecycle, FR-1/4/6 | Admin tests/integration/{operation_lifecycle,operations_agents}.rs; registry tests/integration/operations_artifacts.rs |
Versions, publish, concurrency, immutable snapshots, history | Same-revision UI→API→MCP proof and complete manual pass |
| REST runtime, FR-2 | crates/crank-runtime/tests/stages.rs; adapter tests/integration/{client,outbound_security}.rs |
Validation, retries/deadlines, SSRF/proxy/DNS, bounded execution | Calibrated performance and deployed network-fault profiles |
| OpenAPI, FR-7..12 | crates/crank-import/tests/{preview,coverage,normalization_details,reference_resolution}.rs; Admin tests/integration/openapi_import.rs and openapi_import/external_references.rs |
Parser/IR/ref limits, provenance, preview/apply/replay, atomicity | ≥50-spec independent oracle corpus, remaining security/re-import semantics; candidate 2.9 tracked separately |
| Artifacts / recovery foundations | crates/crank-artifacts/tests/{artifact_store,artifact_store_io,reconciliation}.rs; registry migration tests |
Digest integrity, symlink protection, crash/disk-full, reconciliation, schema drift/rollback | Full deployment restore with DB/artifacts/keys and measured RPO/RTO |
| MCP Tools, FR-13/14/20 | MCP app tests/integration/{transport_protocol,catalog_access,tool_search}.rs; crates/crank-community-mcp/tests/ |
JSON/SSE sessions, list/call, scope/revocation, stale catalog | Frozen revision matrix and ≥3 actual clients |
| Resources/Prompts/Tasks, FR-15..19/21..26 | Capability inventory marks planned flows | No implemented feature-level suite found | Model/contract tests plus public MCP/UI outcomes after features exist; internal tokio tasks are not MCP Tasks |
| Correlation / metrics, FR-27..35 | crates/crank-observability/tests/; app tests/{request_context,product_metrics}.rs; crates/crank-metrics/tests/ |
Local IDs, propagation, captured OTLP, redaction, metric contracts | Actual GlitchTip delivery/spool truth; browser correlation; Grafana/alerts; full overhead workload |
| Load Runs, FR-36..40 | Local metric overhead and concurrency probes only | Narrow budgets, not a built-in load capability | Bounded runner, external calibration, safe stop and report accuracy |
| Browser | apps/ui/tests/e2e/, playwright.config.js |
56 discovered tests in 13 files; selected RU/EN and mobile viewport cases | Firefox/WebKit; complete locale/critical-viewport matrix; isolation before parallelism |
| CI / tooling, FR-46/50 | tests/unit/, justfile, .gitea/workflows/{ci,release}.yml |
138 Python test methods; Rust/UI guards, dependency checks, image smoke, scan/evidence collectors | Coverage reports, scheduled long-run lanes, full release evidence and retention policy |
Rust inventory method: git ls-tree -r --name-only HEAD -- apps crates filtered for /tests/.*\\.rs$ or /src/.*_tests\\.rs$ yields 116 files. git grep -h -E '#\\[(tokio::test|test)' HEAD -- apps crates | wc -l yields 768 attributes (including inline tests). These static counts are not executed cases or line/branch coverage.
Playwright discovery was actually run: ./node_modules/.bin/playwright test --list --reporter=line → 56 tests / 13 files, exit 0. No browsers or product services were started by this command. Real-stack scenarios include onboarding.spec.js public tools/call and OpenAPI upload; many failure/race UI tests use page.route and prove browser handling only. A local controlled upstream is appropriate for deterministic wire assertions, but does not qualify arbitrary external APIs.
Available Execution Evidence and Exclusions
- Historical baseline
docs/capability-baseline/results.json(2026.08.24.1) contains revision-bound accepted runs, historical failures and a not-run manual item. Do not transplant its verdicts to current main; validate the baseline as a historical contract only. - Candidate 2.9 has a prior successful full
just verify, documented in_bmad-output/implementation-artifacts/spec-2-9-http-contract.md; local log/tmp/crank-24-verify-final-gate.logis non-durable supporting evidence. It includes admin integration 81 and MCP integration 40 passes. It does not prove browser E2E, deployment or public readiness. - Its review also leaves two explicitly deferred pre-existing cases: undeclared URL placeholders and composite object/array defaults. Preserve them in the future corpus/gap triage; a green candidate gate does not close these cases.
crates/crank-registry/tests/integration/observability.rshas ignoredproduction_size_invocation_history_queries_stay_bounded, inserting 1,000,000 rows. Standard workspace tests do not execute it; it belongs in an explicitly provisioned performance lane.- No measured workspace line/branch coverage report was found. The existing ≥80%/≥90% targets below remain design targets, not observed values.
- Presence of restore scripts, manual checklists or SLO thresholds is not evidence that restores, manual qualification, 72-hour soak or 30-day burn-in passed.
Risk Assessment
| Risk | Score | QA coverage |
|---|---|---|
| R-001 brownfield regressions | 9 | CP-01..04 characterization/manual baseline |
| R-002 OpenAPI semantic loss | 9 | CP-05..07 corpus, API and representative UI |
| R-003 Task duplicate/stale writes | 9 | CP-12..15 model, PG contention, chaos and MCP E2E |
| R-004 SSRF/secret leakage | 9 | CP-03/04/19/25/27 plus malicious-network/canary scans |
| R-005 trace spool/replay false claims | 6 | CP-16..18 outage/replay and golden correlation |
| R-006 SLO breach under contention | 9 | CP-14/18/20/23/29 calibrated performance |
| R-007 admin/session/recovery auth | 6 | CP-25 web-security state/config matrix |
| R-008 inconsistent upgrade/restore | 6 | CP-26 three restores and N/N-1 workflows |
| R-009 MCP catalog/client inconsistency | 6 | CP-08..11 protocol and real-client matrix |
| R-010 unsafe/uncontrollable Load Run | 6 | CP-22..24 authorization/kill/UI tests |
| R-011 metric double count/cardinality | 6 | CP-19..21 registry/differential/dashboard tests |
| R-013 mixed/stale evidence | 6 | CP-28..31 manifest negative matrix |
| R-014 untrusted/leaky offline bundle | 6 | CP-27 supply-chain and air-gap capture |
| R-015 late soak defect | 6 | CP-29 early trends + 72 h/30 d qualification |
| R-012 flaky E2E | 4 | Parallel-safety burn-in, trace-on-failure and quality lint |
NFR Test Coverage Plan
| Category | Requirement / threshold | Planned validation | Tool / level | Evidence artifact | Priority |
|---|---|---|---|---|---|
| Security/privacy | Default deny, encrypted/no-leak secrets, per-connection SSRF, no Critical/High, opt-in egress | Auth/config abuse, malicious DNS/redirect/proxy, secret canaries, supply-chain scans | Rust/API/Playwright + SAST/SCA/SBOM | SARIF/JUnit, SBOM/signature, canary and egress reports | P0 |
| Performance/scale | 100 RPS/30 m, 250 RPS/5 m, ≥99.95%, reject ≤0.1%, 128/10k, overhead 50/150 ms | Calibrated built-in/external load, contention, production dataset and telemetry differential | External load tool + built-in runner + Prometheus | Raw samples, SUT/generator resource series, plans, verdict JSON | P0 |
| Reliability/recovery | drain ≤30 s, Task start ≤5 s p95, lease recovery ≤60 s, RPO≤5 m/RTO≤30 m | Crash schedules, restart/drain, N/N-1, three PITR restores, 72 h/30 d | PG integration/chaos/recovery/soak | State invariants, manifests, probe logs and time series | P0 |
| Observability | 100% IDs/traces, delivery ≥99.99%/60 s, completeness 100%/25 h, 30 h spool, overhead limits | Golden correlation, outage/replay/corruption/disk, metrics/cardinality/dashboard/alerts | API/MCP/Playwright + captured telemetry + Prom/Grafana | Correlation graph, envelopes, loss ledger, dashboard tests | P0 |
| Compatibility | MCP stable/N-1/N-2, ≥3 clients; OpenAPI ≥50/≥95%/zero silent loss; browser/PG/Valkey matrices | Corpus, protocol/clients, browser and dependency matrices | Rust corpus/protocol/CI matrix | Machine-readable matrices and raw failures | P0 |
| Maintainability/provenance | ≥80% overall, ≥90% critical crates; immutable one-RC evidence; RU/EN | Coverage/lint/architecture/wire/localization and manifest negative tests | CI/static/contract | Coverage, lint and signed manifest reports | P1/P0 manifest |
Threshold source is PRD §5/§10; exact client/revision/image selections are freeze inputs, not guessed current versions. Missing evidence sources: reference performance host, calibrated generator, durable report storage, GlitchTip/Prometheus/Grafana test topology, recovery/air-gap environment and release signing trust.
Measure steady and burst acceptance per minute (NFR-P1a/P1b), not only whole-run averages. Use PRD SLI denominators for handled upstream/client errors, product failures, saturation and missing probes. Run the required workload three consecutive times and retain raw inputs/config/resource series. Measure platform overhead separately from the controlled upstream delay. A million-row query-plan test alone cannot prove the mixed 100/250 RPS workload or UI p95.
Entry Criteria
- Requirements/architecture/epics are versioned and the selected story has ATDD-ready acceptance criteria.
- Applicable B-001..B-005 seams are implemented and production-disabled where required.
- Test environment, deterministic factories, cleanup and artifact retention are operational.
- Feature is deployed with exact code/config/topology digest and migrations compatible with the test matrix.
- Test oracle (corpus expected result, protocol version, SLO formula or recovery checkpoint) is frozen.
Exit Criteria
- P0 = 100%; development P1 ≥95%; RC P1 mapped to mandatory FR/critical journeys = 100%.
- No open Critical/High defect, security finding or score-9 risk; score 6–8 mitigations have evidence.
- Workspace coverage ≥80% and critical state/security crates ≥90%, without requirement gaps.
- All required corpus/client/browser/store matrices and manual checks pass; no manual item blocked/not-run.
- Every NFR has same-RC raw evidence and later NFR assessment; all numerical thresholds pass.
- Signed manifest validates and binds the exact published RC, config, topology and artifacts.
Test Coverage Plan
P0/P1/P2/P3 indicate priority, not execution timing. The table is the corrected current system mapping; older CP mappings in the historical progress/handoff files are not authoritative. Each family will be decomposed only when its epic is planned. “Partial” means some source exists, not that the complete FR passes.
P0 — Critical
Security, data integrity and core-operation invariants without a safe workaround. Negative subcases of P1 families are P0 where stated.
P1 — High
Required feature variants, compatibility and operator outcomes with material reach. Mandatory release requirements still require 100% accepted evidence even when P1.
P2 — Medium
Supplementary exploratory cases with an acceptable workaround; they do not replace the mandatory manual regression pass.
P3 — Low
Optional cosmetic/experimental probes only; not mandatory release evidence.
| ID | Requirement | Priority | Primary level | Risk | Required oracle / outcome | Main inventory |
|---|---|---|---|---|---|---|
| CP-01 | FR-1..6 | P0 | API + golden UI/MCP | R-001/R-004 | Create→test→publish→call→history; exact result and safe failure | Partial |
| CP-02 | FR-1/4/43/45 | P0 | Unit + PG | R-001/R-008 | Immutable versions, atomic publish, compatible migration and config | Partial |
| CP-03 | FR-5; NFR-S1/S7 | P0 | State/PG + boundary | R-004 | Approval/idempotency cannot duplicate effects or fail open | Partial |
| CP-04 | FR-3; NFR-S2 | P0 | API/PG + canary | R-004 | Create/use/rotate/revoke; no plaintext in output channels | Partial |
| CP-05 | FR-7/8/9/10/12 | P0 | Pure corpus + wire contract | R-002 | Canonical parse/replay; exact blockers; no silent semantic loss | Partial; 2.9 candidate separate |
| CP-06 | FR-7/11 | P0 | API/PG | R-002 | Preview no mutation; scoped, atomic, idempotent apply; re-import conflicts | Partial |
| CP-07 | FR-7..12 | P1 | Playwright real stack | R-002 | Upload→findings→selection→draft→publish; cancellation/recovery | Partial |
| CP-08 | FR-13/14/19 | P1 | MCP protocol | R-009 | Negotiation, Tools schema/results, pagination and notifications | Partial |
| CP-09 | FR-15/16 | P1 | Unit/API + public MCP | R-009/R-004 | Static/dynamic resources, templates, pinned read-safe operation; negative access P0 | Planned |
| CP-10 | FR-17/18/19 | P1 | Unit/API + public MCP | R-009/R-004 | Prompt arguments/render/DAG/completion; forbidden side effects P0 | Planned |
| CP-11 | FR-20; NFR-C2 | P1 | Actual-client matrix | R-009 | ≥3 clients × frozen stable/N-1/N-2; identical versioned semantics | Gap |
| CP-12 | FR-21/22 | P0 | Model/property | R-003 | Only legal Task states; immutable terminal result and pinned versions | Planned |
| CP-13 | FR-23/25 | P0 | PG crash/concurrency | R-003 | Enqueue/claim/fencing/retry/cancel; no stale writes or false exactly-once claim | Planned |
| CP-14 | FR-24; NFR-P2/R5 | P0 | Controlled load | R-003/R-006 | 128 active/10k queued; bounded wait/fairness and ≤60s recovery | Planned |
| CP-15 | FR-21..26 | P1 | Public MCP + UI | R-003/R-009 | Observe/cancel/resume through disconnect/restart and supported fallback | Planned |
| CP-16 | FR-27/30; NFR-O1/O5 | P0 | Contract/API | R-005/R-004 | Request/trace identities survive boundaries; redaction and invalid-input replacement | Partial |
| CP-17 | FR-28/29/30 | P1 | Browser→MCP→receiver | R-005 | Search/copy/deep-link actual delivered trace; missing/lost is never shown as delivered | Gap |
| CP-18 | FR-29/31; NFR-O2/O2a/O3 | P0 | Telemetry fault/capacity | R-005/R-006 | Outage/restart/replay/disk corruption; delivery/loss and resource budgets | Gap |
| CP-19 | FR-32; NFR-O4 | P0 | Unit/contract | R-011 | Names/units/buckets/labels bounded; no user identifiers as labels | Partial |
| CP-20 | FR-33 | P0 | PG differential | R-011 | UTC DAU/usage events and rollups agree; replay cannot double count | Partial |
| CP-21 | FR-34/35 | P1 | Prometheus/Grafana integration | R-011 | Provision dashboards/rules; known synthetic series trigger correct SLO alerts | Gap |
| CP-22 | FR-36/37/38/39 | P0 | API/state/security | R-010 | Validate target/auth/ceilings; reserve stop path; reject scope escape | Planned |
| CP-23 | FR-40; NFR-P1/P4 | P1 | Calibrated performance | R-006 | Report classifications/percentiles match external generator at reference workload | Planned |
| CP-24 | FR-36..40 | P1 | Playwright real stack | R-010 | RPS/DAU configuration→launch→live statistics→stop→report | Planned |
| CP-25 | FR-41/42/44/45 | P0 | API/browser security | R-007/R-004 | Bootstrap/session/recovery, TLS/proxy/cookie/CSRF/CORS/default-deny matrix | Partial |
| CP-26 | FR-43 | P0 | Deployment/restore | R-008 | Fresh/N-1/rollback + 3 PITR restores; real encrypted-auth MCP probe | Partial foundations |
| CP-27 | FR-42/44/50 | P0 | Supply-chain/air-gap | R-014 | Trusted pinned bundle/SBOM; zero undeclared egress and Critical/High findings | Partial scans |
| CP-28 | FR-46/48/50 | P1 | CI matrix | R-001/R-009 | Required suites, browsers, clients, PG/cache modes; retain failures too | Partial |
| CP-29 | FR-49 | P0 | Soak/recovery | R-006/R-015 | 72h then supervised 30-day qualification on immutable RC, no loss/drift | Gap |
| CP-30 | FR-47/51/52/53 | P1 | Guided manual/docs | R-001/R-008 | Operator can follow runbooks; readiness page reflects evidence, not promises | Partial docs; qualification missing |
| CP-31 | FR-54 | P0 | Manifest negative contract | R-013 | Reject missing/stale/mixed/untrusted evidence; FR-54 cannot prove itself | Partial collectors only |
| CP-32 | NFR-C1/C3/C4 | P1 | Static/contract | R-001/R-012 | Dependency/MSRV/wire rules; simultaneous RU/EN; coverage report | Partial |
| CP-33 | UI across FR-1..53 | P1 | Component/browser | R-012 | Loading/empty/error/stale states; keyboard/focus/viewport; no hidden controls | Partial |
| CP-34 | FR-47/51 | P2 | Time-boxed exploratory | R-001/R-002 | Real-spec/operator/client combinations beyond deterministic corpus | Not evidenced |
This is family-level traceability, not a measured AC-to-test coverage matrix. Use the lowest effective test for exhaustive variants, then one representative cross-boundary success and failure for each critical journey. Retain explicit negative cases for authorization, secret leakage, corruption, concurrent mutation, cancellation and limits. Do not generate duplicate E2E tests for every unit-level combination.
Execution Strategy
Current reality: Gitea runs full functional Rust tests, UI build/audit, Chromium E2E and image smoke on configured PR/push paths; release adds artifact/scanning steps. just verify does not run UI E2E, Docker image smoke or deployment. Playwright currently uses one worker and CI retry=1, not sharding. No scheduled nightly/weekly workflows or measured coverage gate were found.
Target: run functional regression in PRs; move only explicitly expensive load/chaos/long-run qualification to provisioned lanes. The 10–15-minute feedback target below is aspirational, not measured. Do not weaken assertions, omit a failing test or increase timeouts merely to meet it.
- Every PR (10–15 min feedback target, not a current duration claim): existing full functional Rust/PG suite, format/lint/boundaries, dependency/UI build checks, Chromium E2E, contract/evidence validation and applicable image smoke. Keep selected local tests for fast iteration, not as a substitute for the integration gate. Parallelism requires isolation evidence first.
- Nightly (duration to measure after provisioning): expanded browser/client/cache matrices, production-size history, bounded contention, telemetry outage/replay and repeat-run flake detection. Ordinary functional failures must not be moved here merely to unblock PRs.
- Weekly (hours): full browser/client matrices, scans, Task contention/chaos, upgrade/rollback, dashboard/alerts, 100/250 RPS rehearsal, spool/disk pressure and backup/restore rotation.
- RC: exact immutable artifact; all matrices/manual work, three restores, air-gap flows, 72-hour soak and then non-accelerable 30-day/≥1M-event burn-in. Any digest-changing fix creates a new RC.
Before enabling Playwright sharding, prove per-worker data/ports/storage/session isolation; the existing single-worker setting stays until that evidence exists. Current traces are on-first-retry; screenshots/videos retain failures. A retry pass is flaky evidence, not a clean pass: existing OpenAPI accepted-run collection already rejects flaky/missing required cases.
Before integration: targeted regression → affected subsystem → full Rust gate plus affected UI/deploy gate. Reuse a completed gate only for the identical source tree and relevant environment; record subsequent documentation-only changes. Capture subprocess fault tests through pipes (not direct regular-file stdout under RLIMIT_FSIZE). Keep a bounded log and the complete machine-readable outcome so failed attempts remain diagnosable.
Provision explicitly for expensive probes: PostgreSQL16 and disk headroom for the million-row test; a PRD reference host for load; isolated backup/restore destinations and keys; test-only telemetry sinks; no production endpoints or customer secrets. Scheduled execution and provisioning are follow-up implementation work, not changes made by this design.
QA Effort Estimate
The August estimate (~42–71 engineer-weeks, ~850–1,250 cases) predates the inventory and is not a forecast of remaining work. Remaining automation effort is UNKNOWN until each family separates reusable tests from missing assertions and unavailable product features. No new test-count quota is introduced.
Estimate each epic delta as a range after system-design acceptance, based on fixture reuse and actual test durations; exclude production-feature implementation and infrastructure operations from QA effort. The 30-day burn-in is a PRD calendar constraint, not 30 days of full-time test-authoring effort.
Implementation Planning Handoff
| Work item | Owner | Target | Dependencies |
|---|---|---|---|
| Deterministic fixtures/factories and per-worker cleanup | QA + Core | Epic 1 | B-001 |
| OpenAPI corpus oracle/runner | Import + QA | Epic 2 | Stable IR/findings |
| MCP protocol/representative-client harness | MCP + QA | Epic 3 | Version adapters |
| Task model/PG chaos harness | Task + QA/SRE | Epic 4 | B-001/B-002 |
| Telemetry/metrics capture and assertions | Observability + QA | Epic 5 | B-003 |
| Calibrated load and reference topology | Performance/SRE | Epic 6 | B-005 |
| Upgrade/restore/air-gap automation | Release/Security/QA | Epic 7 | B-002/B-004 |
| Evidence collector/manifest verifier and RC qualification | Release + QA | Epic 1→8 | B-004, all suites |
Tooling and Access
| Tool/service | Purpose | Access | Status |
|---|---|---|---|
| PostgreSQL 16/Valkey Testcontainers | Storage/coordination integration | Container runtime | Existing foundation; extend |
| Playwright 1.61.1 | UI and real-stack/mocked-boundary browser tests | Node + Chromium | Installed; Utils/Pact are preferences only, absent from package.json |
| MCP protocol harness + ≥3 clients | Compatibility matrix | Pinned clients | Pending feature-freeze selection |
| Captured OTLP/Sentry endpoint, Prometheus/Grafana/GlitchTip | Observability assertions | Self-hosted test profile | Pending Epic 5 |
| External calibrated load/chaos tooling | Cross-check and fault injection | Reference host/lab | Pending |
| cargo-deny, npm audit, Trivy / SBOM | Dependency/container policy | CI toolchain/images | Configured; current scan reports and complete SAST/secret/signature evidence not established by this audit |
Interworking and Regression
| Component | Impact | Regression scope | Validation |
|---|---|---|---|
| Core/runtime/REST adapter | Shared execution semantics/security | Existing REST→MCP + all ingress paths | CP-01..04, CP-16, malicious upstream |
| Import/artifacts/registry | New parser/transactional plans | OpenAPI and persisted revisions | CP-05..07 |
| MCP server/catalog | New capabilities and adapters | Stable/N-1/N-2 clients | CP-08..11 |
| Task worker/PostgreSQL/Valkey | New durable concurrency | State, contention, restart, PG-only | CP-12..15 |
| UI/Admin API | New operator flows/security | RU/EN, async state, auth, browser matrix | CP-07/17/24/25/33 |
| Observability/metrics stack | Cross-cutting data path | Correlation, redaction, cardinality, SLO | CP-16..21 |
| Load/deploy/release assets | Production operations | Safety, recovery, air gap, evidence | CP-22..31 |
Every fixed defect adds the lowest-level regression and a boundary case if the failure crossed a contract.
Appendix A: Reproducible Discovery and Evidence
# From apps/ui: enumerate only; does not prove a pass.
./node_modules/.bin/playwright test --list --reporter=line
# From the repository: static test attributes, not coverage.
git grep -h -E '#\[(tokio::test|test)' HEAD -- apps crates | wc -l
For each future accepted run retain source revision/tree, image/config/topology identifiers, dependency/client/browser versions, command, timestamps, cases passed/failed/skipped/flaky, raw report digest and sanitized diagnostics. Source files and generated “completed” planning metadata cannot serve as execution evidence. Keep secrets, credentials and unbounded payloads out of retained reports.
Current tests do not have a complete @P0/@P1 tag taxonomy; selecting tags before annotating the suite may run zero tests. Use known files/test names now, and make empty mandatory selections fail. Report retention must cover ordinary failing runs as well as successful sanitized OpenAPI release evidence.
Edit Verification — 2026-09-13
Read source/configuration and previous plans; ran Playwright discovery only. Product tests, benchmarks, browser runs and deploy were not rerun. Document checks confirm 34 unique CP families covering all FR-1..54 at family level, shared risk identifiers, preserved PRD thresholds and explicit missing evidence. This is not an assertion-level coverage proof or final NFR PASS/CONCERNS/FAIL verdict.
Appendix B: Knowledge Base References
risk-governance.mdandtest-priorities-matrix.mdtest-levels-framework.mdandtest-quality.mdnfr-criteria.mdandadr-quality-readiness-checklist.mdplaywright-cli.mdandcontract-testing.md
Generated by: BMad TEA Agent
Workflow: bmad-testarch-test-design