Files
crank/_bmad-output/test-artifacts/test-design-qa.md
T

29 KiB
Raw Blame History

workflowStatus, totalSteps, stepsCompleted, lastStep, nextStep, lastSaved, lastEdited, editScope, auditedRevision, workflowType, inputDocuments
workflowStatus totalSteps stepsCompleted lastStep nextStep lastSaved lastEdited editScope auditedRevision workflowType inputDocuments
completed 5
step-01-detect-mode
step-02-load-context
step-03-risk-and-testability
step-04-coverage-plan
step-05-generate-output
step-05-generate-output 2026-09-13 2026-09-13 system-level inventory and evidence-aware strategy refresh 083974c27f testarch-test-design
_bmad-output/test-artifacts/test-design-architecture.md
_bmad-output/planning-artifacts/epics.md

Test Design for QA: Crank Community Production Readiness

Purpose: Execution recipe for proving Crank Community production readiness. Architectural requirements and production-code mitigations are in test-design-architecture.md.

Date: 2026-09-13 (original design: 2026-08-08)
Author: BMad TEA / Bsodfather
Status: System-level design updated; current evidence inventory precedes epic-level planning
Project: crank

Executive Summary

Scope covers all 54 FRs, 30 NFRs and the eight production-readiness epics. There are 15 risks: 14 high (five score 9), one medium and none low. Dominant categories are operations, security, technical/concurrency, data integrity and performance.

The plan retains 34 scenario families, not a promise to write 850–1,250 additional tests. Existing tests must be reused. P0 denotes a security/data-integrity/core-operation failure with no safe workaround; ordinary feature variants are P1, not automatically P0. E2E checks cross-boundary outcomes rather than repeating parser or state-machine branch tests.

Audit boundary: main 083974c27f3a21a15807dd248bb6bc4fe04a5cd8. Story 2.9 candidate f8ee7c9bfeaa8a3f73a4850ff3f82b2bc2155fe8 is recorded separately and is not integrated into main. This update is a source/configuration audit, not a fresh product test run.

Not in Scope

Item Reason Mitigation
HA/multi-instance rolling upgrade Explicit post-launch scope Document single-instance claim and reject HA evidence claims
S3-compatible artifacts, SSO/RBAC, non-REST adapters Deferred by architecture Preserve ports/invariants; revisit in later test design
Arbitrary distributed load targets or loads above built-in ceilings Unsafe/outside Community goal Load runner allowlists only Crank MCP and enforces signed ceilings
MCP revisions published after feature freeze Unbounded moving target Enter next adapter/compatibility cycle

Dependencies and Test Blockers

Backend/architecture dependencies

  1. B-001 deterministic control plane: typed factories/builders, scoped cleanup, controllable time/IDs/scheduler.
  2. B-002 named fault injection: network, stores, worker, telemetry and disk failure controls.
  3. B-003 queryable observability sinks: captured trace/envelope/metric/dashboard state searchable by IDs.
  4. B-004 evidence contract: stable artifact schemas and RC provenance collectors.
  5. B-005 measurable load runner: calibrated stub/external generator, separate resource metrics and reliable kill path.

QA infrastructure setup

  • Extend crank-test-support with PostgreSQL 16/Valkey/telemetry/malicious-upstream fixtures and per-worker schemas.
  • Provide seeded production-size and minimal datasets with deterministic seeds and automatic cleanup.
  • Create MCP protocol/client matrix harness, OpenAPI corpus runner, browser storage-state fixtures and artifact collectors.
  • Provision PR ephemeral dependencies, nightly full topology, weekly reference hardware/air-gap lab and immutable RC qualification environment.
  • Retain JUnit/JSON, Playwright traces, raw metrics, logs and content-addressed evidence under _bmad-output/test-artifacts/ or the release evidence store.

Actual Tooling vs Configuration

tea_use_playwright_utils=true and tea_use_pactjs_utils=true are workflow preferences, not installed dependencies. The UI package currently contains @playwright/test 1.61.1 and esbuild; neither Utils nor Pact is installed. This design does not install or mandate a wholesale test rewrite. Retain working fixtures; evaluate additional tooling only for an identified gap.

Existing Test Inventory

Presence below means inspected test source, not current pass or a coverage percentage.

Surface / FR Representative existing source What it exercises Missing qualification / next evidence
Config, FR-45 crates/crank-config/tests/{contract,generation,redaction}.rs; app tests/config_startup.rs Config bounds, deterministic contracts, read-only startup, safe failures Production-profile matrix on a bound artifact
Auth/secrets, FR-3/5 apps/admin-api/tests/integration/{auth_rate_limit,credential_lifecycle,secrets_import_auth}.rs; runtime tests/integration/{confirmation,idempotency}.rs CSRF/origin/rate-limit, lifecycle, encrypted credentials, approval/idempotency Whole web-security and deployed secret-canary matrix
Operation/Agent lifecycle, FR-1/4/6 Admin tests/integration/{operation_lifecycle,operations_agents}.rs; registry tests/integration/operations_artifacts.rs Versions, publish, concurrency, immutable snapshots, history Same-revision UI→API→MCP proof and complete manual pass
REST runtime, FR-2 crates/crank-runtime/tests/stages.rs; adapter tests/integration/{client,outbound_security}.rs Validation, retries/deadlines, SSRF/proxy/DNS, bounded execution Calibrated performance and deployed network-fault profiles
OpenAPI, FR-7..12 crates/crank-import/tests/{preview,coverage,normalization_details,reference_resolution}.rs; Admin tests/integration/openapi_import.rs and openapi_import/external_references.rs Parser/IR/ref limits, provenance, preview/apply/replay, atomicity ≥50-spec independent oracle corpus, remaining security/re-import semantics; candidate 2.9 tracked separately
Artifacts / recovery foundations crates/crank-artifacts/tests/{artifact_store,artifact_store_io,reconciliation}.rs; registry migration tests Digest integrity, symlink protection, crash/disk-full, reconciliation, schema drift/rollback Full deployment restore with DB/artifacts/keys and measured RPO/RTO
MCP Tools, FR-13/14/20 MCP app tests/integration/{transport_protocol,catalog_access,tool_search}.rs; crates/crank-community-mcp/tests/ JSON/SSE sessions, list/call, scope/revocation, stale catalog Frozen revision matrix and ≥3 actual clients
Resources/Prompts/Tasks, FR-15..19/21..26 Capability inventory marks planned flows No implemented feature-level suite found Model/contract tests plus public MCP/UI outcomes after features exist; internal tokio tasks are not MCP Tasks
Correlation / metrics, FR-27..35 crates/crank-observability/tests/; app tests/{request_context,product_metrics}.rs; crates/crank-metrics/tests/ Local IDs, propagation, captured OTLP, redaction, metric contracts Actual GlitchTip delivery/spool truth; browser correlation; Grafana/alerts; full overhead workload
Load Runs, FR-36..40 Local metric overhead and concurrency probes only Narrow budgets, not a built-in load capability Bounded runner, external calibration, safe stop and report accuracy
Browser apps/ui/tests/e2e/, playwright.config.js 56 discovered tests in 13 files; selected RU/EN and mobile viewport cases Firefox/WebKit; complete locale/critical-viewport matrix; isolation before parallelism
CI / tooling, FR-46/50 tests/unit/, justfile, .gitea/workflows/{ci,release}.yml 138 Python test methods; Rust/UI guards, dependency checks, image smoke, scan/evidence collectors Coverage reports, scheduled long-run lanes, full release evidence and retention policy

Rust inventory method: git ls-tree -r --name-only HEAD -- apps crates filtered for /tests/.*\\.rs$ or /src/.*_tests\\.rs$ yields 116 files. git grep -h -E '#\\[(tokio::test|test)' HEAD -- apps crates | wc -l yields 768 attributes (including inline tests). These static counts are not executed cases or line/branch coverage.

Playwright discovery was actually run: ./node_modules/.bin/playwright test --list --reporter=line → 56 tests / 13 files, exit 0. No browsers or product services were started by this command. Real-stack scenarios include onboarding.spec.js public tools/call and OpenAPI upload; many failure/race UI tests use page.route and prove browser handling only. A local controlled upstream is appropriate for deterministic wire assertions, but does not qualify arbitrary external APIs.

Available Execution Evidence and Exclusions

  • Historical baseline docs/capability-baseline/results.json (2026.08.24.1) contains revision-bound accepted runs, historical failures and a not-run manual item. Do not transplant its verdicts to current main; validate the baseline as a historical contract only.
  • Candidate 2.9 has a prior successful full just verify, documented in _bmad-output/implementation-artifacts/spec-2-9-http-contract.md; local log /tmp/crank-24-verify-final-gate.log is non-durable supporting evidence. It includes admin integration 81 and MCP integration 40 passes. It does not prove browser E2E, deployment or public readiness.
  • Its review also leaves two explicitly deferred pre-existing cases: undeclared URL placeholders and composite object/array defaults. Preserve them in the future corpus/gap triage; a green candidate gate does not close these cases.
  • crates/crank-registry/tests/integration/observability.rs has ignored production_size_invocation_history_queries_stay_bounded, inserting 1,000,000 rows. Standard workspace tests do not execute it; it belongs in an explicitly provisioned performance lane.
  • No measured workspace line/branch coverage report was found. The existing ≥80%/≥90% targets below remain design targets, not observed values.
  • Presence of restore scripts, manual checklists or SLO thresholds is not evidence that restores, manual qualification, 72-hour soak or 30-day burn-in passed.

Risk Assessment

Risk Score QA coverage
R-001 brownfield regressions 9 CP-01..04 characterization/manual baseline
R-002 OpenAPI semantic loss 9 CP-05..07 corpus, API and representative UI
R-003 Task duplicate/stale writes 9 CP-12..15 model, PG contention, chaos and MCP E2E
R-004 SSRF/secret leakage 9 CP-03/04/19/25/27 plus malicious-network/canary scans
R-005 trace spool/replay false claims 6 CP-16..18 outage/replay and golden correlation
R-006 SLO breach under contention 9 CP-14/18/20/23/29 calibrated performance
R-007 admin/session/recovery auth 6 CP-25 web-security state/config matrix
R-008 inconsistent upgrade/restore 6 CP-26 three restores and N/N-1 workflows
R-009 MCP catalog/client inconsistency 6 CP-08..11 protocol and real-client matrix
R-010 unsafe/uncontrollable Load Run 6 CP-22..24 authorization/kill/UI tests
R-011 metric double count/cardinality 6 CP-19..21 registry/differential/dashboard tests
R-013 mixed/stale evidence 6 CP-28..31 manifest negative matrix
R-014 untrusted/leaky offline bundle 6 CP-27 supply-chain and air-gap capture
R-015 late soak defect 6 CP-29 early trends + 72 h/30 d qualification
R-012 flaky E2E 4 Parallel-safety burn-in, trace-on-failure and quality lint

NFR Test Coverage Plan

Category Requirement / threshold Planned validation Tool / level Evidence artifact Priority
Security/privacy Default deny, encrypted/no-leak secrets, per-connection SSRF, no Critical/High, opt-in egress Auth/config abuse, malicious DNS/redirect/proxy, secret canaries, supply-chain scans Rust/API/Playwright + SAST/SCA/SBOM SARIF/JUnit, SBOM/signature, canary and egress reports P0
Performance/scale 100 RPS/30 m, 250 RPS/5 m, ≥99.95%, reject ≤0.1%, 128/10k, overhead 50/150 ms Calibrated built-in/external load, contention, production dataset and telemetry differential External load tool + built-in runner + Prometheus Raw samples, SUT/generator resource series, plans, verdict JSON P0
Reliability/recovery drain ≤30 s, Task start ≤5 s p95, lease recovery ≤60 s, RPO≤5 m/RTO≤30 m Crash schedules, restart/drain, N/N-1, three PITR restores, 72 h/30 d PG integration/chaos/recovery/soak State invariants, manifests, probe logs and time series P0
Observability 100% IDs/traces, delivery ≥99.99%/60 s, completeness 100%/25 h, 30 h spool, overhead limits Golden correlation, outage/replay/corruption/disk, metrics/cardinality/dashboard/alerts API/MCP/Playwright + captured telemetry + Prom/Grafana Correlation graph, envelopes, loss ledger, dashboard tests P0
Compatibility MCP stable/N-1/N-2, ≥3 clients; OpenAPI ≥50/≥95%/zero silent loss; browser/PG/Valkey matrices Corpus, protocol/clients, browser and dependency matrices Rust corpus/protocol/CI matrix Machine-readable matrices and raw failures P0
Maintainability/provenance ≥80% overall, ≥90% critical crates; immutable one-RC evidence; RU/EN Coverage/lint/architecture/wire/localization and manifest negative tests CI/static/contract Coverage, lint and signed manifest reports P1/P0 manifest

Threshold source is PRD §5/§10; exact client/revision/image selections are freeze inputs, not guessed current versions. Missing evidence sources: reference performance host, calibrated generator, durable report storage, GlitchTip/Prometheus/Grafana test topology, recovery/air-gap environment and release signing trust.

Measure steady and burst acceptance per minute (NFR-P1a/P1b), not only whole-run averages. Use PRD SLI denominators for handled upstream/client errors, product failures, saturation and missing probes. Run the required workload three consecutive times and retain raw inputs/config/resource series. Measure platform overhead separately from the controlled upstream delay. A million-row query-plan test alone cannot prove the mixed 100/250 RPS workload or UI p95.

Entry Criteria

  • Requirements/architecture/epics are versioned and the selected story has ATDD-ready acceptance criteria.
  • Applicable B-001..B-005 seams are implemented and production-disabled where required.
  • Test environment, deterministic factories, cleanup and artifact retention are operational.
  • Feature is deployed with exact code/config/topology digest and migrations compatible with the test matrix.
  • Test oracle (corpus expected result, protocol version, SLO formula or recovery checkpoint) is frozen.

Exit Criteria

  • P0 = 100%; development P1 ≥95%; RC P1 mapped to mandatory FR/critical journeys = 100%.
  • No open Critical/High defect, security finding or score-9 risk; score 6–8 mitigations have evidence.
  • Workspace coverage ≥80% and critical state/security crates ≥90%, without requirement gaps.
  • All required corpus/client/browser/store matrices and manual checks pass; no manual item blocked/not-run.
  • Every NFR has same-RC raw evidence and later NFR assessment; all numerical thresholds pass.
  • Signed manifest validates and binds the exact published RC, config, topology and artifacts.

Test Coverage Plan

P0/P1/P2/P3 indicate priority, not execution timing. The table is the corrected current system mapping; older CP mappings in the historical progress/handoff files are not authoritative. Each family will be decomposed only when its epic is planned. “Partial” means some source exists, not that the complete FR passes.

P0 — Critical

Security, data integrity and core-operation invariants without a safe workaround. Negative subcases of P1 families are P0 where stated.

P1 — High

Required feature variants, compatibility and operator outcomes with material reach. Mandatory release requirements still require 100% accepted evidence even when P1.

P2 — Medium

Supplementary exploratory cases with an acceptable workaround; they do not replace the mandatory manual regression pass.

P3 — Low

Optional cosmetic/experimental probes only; not mandatory release evidence.

ID Requirement Priority Primary level Risk Required oracle / outcome Main inventory
CP-01 FR-1..6 P0 API + golden UI/MCP R-001/R-004 Create→test→publish→call→history; exact result and safe failure Partial
CP-02 FR-1/4/43/45 P0 Unit + PG R-001/R-008 Immutable versions, atomic publish, compatible migration and config Partial
CP-03 FR-5; NFR-S1/S7 P0 State/PG + boundary R-004 Approval/idempotency cannot duplicate effects or fail open Partial
CP-04 FR-3; NFR-S2 P0 API/PG + canary R-004 Create/use/rotate/revoke; no plaintext in output channels Partial
CP-05 FR-7/8/9/10/12 P0 Pure corpus + wire contract R-002 Canonical parse/replay; exact blockers; no silent semantic loss Partial; 2.9 candidate separate
CP-06 FR-7/11 P0 API/PG R-002 Preview no mutation; scoped, atomic, idempotent apply; re-import conflicts Partial
CP-07 FR-7..12 P1 Playwright real stack R-002 Upload→findings→selection→draft→publish; cancellation/recovery Partial
CP-08 FR-13/14/19 P1 MCP protocol R-009 Negotiation, Tools schema/results, pagination and notifications Partial
CP-09 FR-15/16 P1 Unit/API + public MCP R-009/R-004 Static/dynamic resources, templates, pinned read-safe operation; negative access P0 Planned
CP-10 FR-17/18/19 P1 Unit/API + public MCP R-009/R-004 Prompt arguments/render/DAG/completion; forbidden side effects P0 Planned
CP-11 FR-20; NFR-C2 P1 Actual-client matrix R-009 ≥3 clients × frozen stable/N-1/N-2; identical versioned semantics Gap
CP-12 FR-21/22 P0 Model/property R-003 Only legal Task states; immutable terminal result and pinned versions Planned
CP-13 FR-23/25 P0 PG crash/concurrency R-003 Enqueue/claim/fencing/retry/cancel; no stale writes or false exactly-once claim Planned
CP-14 FR-24; NFR-P2/R5 P0 Controlled load R-003/R-006 128 active/10k queued; bounded wait/fairness and ≤60s recovery Planned
CP-15 FR-21..26 P1 Public MCP + UI R-003/R-009 Observe/cancel/resume through disconnect/restart and supported fallback Planned
CP-16 FR-27/30; NFR-O1/O5 P0 Contract/API R-005/R-004 Request/trace identities survive boundaries; redaction and invalid-input replacement Partial
CP-17 FR-28/29/30 P1 Browser→MCP→receiver R-005 Search/copy/deep-link actual delivered trace; missing/lost is never shown as delivered Gap
CP-18 FR-29/31; NFR-O2/O2a/O3 P0 Telemetry fault/capacity R-005/R-006 Outage/restart/replay/disk corruption; delivery/loss and resource budgets Gap
CP-19 FR-32; NFR-O4 P0 Unit/contract R-011 Names/units/buckets/labels bounded; no user identifiers as labels Partial
CP-20 FR-33 P0 PG differential R-011 UTC DAU/usage events and rollups agree; replay cannot double count Partial
CP-21 FR-34/35 P1 Prometheus/Grafana integration R-011 Provision dashboards/rules; known synthetic series trigger correct SLO alerts Gap
CP-22 FR-36/37/38/39 P0 API/state/security R-010 Validate target/auth/ceilings; reserve stop path; reject scope escape Planned
CP-23 FR-40; NFR-P1/P4 P1 Calibrated performance R-006 Report classifications/percentiles match external generator at reference workload Planned
CP-24 FR-36..40 P1 Playwright real stack R-010 RPS/DAU configuration→launch→live statistics→stop→report Planned
CP-25 FR-41/42/44/45 P0 API/browser security R-007/R-004 Bootstrap/session/recovery, TLS/proxy/cookie/CSRF/CORS/default-deny matrix Partial
CP-26 FR-43 P0 Deployment/restore R-008 Fresh/N-1/rollback + 3 PITR restores; real encrypted-auth MCP probe Partial foundations
CP-27 FR-42/44/50 P0 Supply-chain/air-gap R-014 Trusted pinned bundle/SBOM; zero undeclared egress and Critical/High findings Partial scans
CP-28 FR-46/48/50 P1 CI matrix R-001/R-009 Required suites, browsers, clients, PG/cache modes; retain failures too Partial
CP-29 FR-49 P0 Soak/recovery R-006/R-015 72h then supervised 30-day qualification on immutable RC, no loss/drift Gap
CP-30 FR-47/51/52/53 P1 Guided manual/docs R-001/R-008 Operator can follow runbooks; readiness page reflects evidence, not promises Partial docs; qualification missing
CP-31 FR-54 P0 Manifest negative contract R-013 Reject missing/stale/mixed/untrusted evidence; FR-54 cannot prove itself Partial collectors only
CP-32 NFR-C1/C3/C4 P1 Static/contract R-001/R-012 Dependency/MSRV/wire rules; simultaneous RU/EN; coverage report Partial
CP-33 UI across FR-1..53 P1 Component/browser R-012 Loading/empty/error/stale states; keyboard/focus/viewport; no hidden controls Partial
CP-34 FR-47/51 P2 Time-boxed exploratory R-001/R-002 Real-spec/operator/client combinations beyond deterministic corpus Not evidenced

This is family-level traceability, not a measured AC-to-test coverage matrix. Use the lowest effective test for exhaustive variants, then one representative cross-boundary success and failure for each critical journey. Retain explicit negative cases for authorization, secret leakage, corruption, concurrent mutation, cancellation and limits. Do not generate duplicate E2E tests for every unit-level combination.

Execution Strategy

Current reality: Gitea runs full functional Rust tests, UI build/audit, Chromium E2E and image smoke on configured PR/push paths; release adds artifact/scanning steps. just verify does not run UI E2E, Docker image smoke or deployment. Playwright currently uses one worker and CI retry=1, not sharding. No scheduled nightly/weekly workflows or measured coverage gate were found.

Target: run functional regression in PRs; move only explicitly expensive load/chaos/long-run qualification to provisioned lanes. The 10–15-minute feedback target below is aspirational, not measured. Do not weaken assertions, omit a failing test or increase timeouts merely to meet it.

  • Every PR (10–15 min feedback target, not a current duration claim): existing full functional Rust/PG suite, format/lint/boundaries, dependency/UI build checks, Chromium E2E, contract/evidence validation and applicable image smoke. Keep selected local tests for fast iteration, not as a substitute for the integration gate. Parallelism requires isolation evidence first.
  • Nightly (duration to measure after provisioning): expanded browser/client/cache matrices, production-size history, bounded contention, telemetry outage/replay and repeat-run flake detection. Ordinary functional failures must not be moved here merely to unblock PRs.
  • Weekly (hours): full browser/client matrices, scans, Task contention/chaos, upgrade/rollback, dashboard/alerts, 100/250 RPS rehearsal, spool/disk pressure and backup/restore rotation.
  • RC: exact immutable artifact; all matrices/manual work, three restores, air-gap flows, 72-hour soak and then non-accelerable 30-day/≥1M-event burn-in. Any digest-changing fix creates a new RC.

Before enabling Playwright sharding, prove per-worker data/ports/storage/session isolation; the existing single-worker setting stays until that evidence exists. Current traces are on-first-retry; screenshots/videos retain failures. A retry pass is flaky evidence, not a clean pass: existing OpenAPI accepted-run collection already rejects flaky/missing required cases.

Before integration: targeted regression → affected subsystem → full Rust gate plus affected UI/deploy gate. Reuse a completed gate only for the identical source tree and relevant environment; record subsequent documentation-only changes. Capture subprocess fault tests through pipes (not direct regular-file stdout under RLIMIT_FSIZE). Keep a bounded log and the complete machine-readable outcome so failed attempts remain diagnosable.

Provision explicitly for expensive probes: PostgreSQL16 and disk headroom for the million-row test; a PRD reference host for load; isolated backup/restore destinations and keys; test-only telemetry sinks; no production endpoints or customer secrets. Scheduled execution and provisioning are follow-up implementation work, not changes made by this design.

QA Effort Estimate

The August estimate (~42–71 engineer-weeks, ~850–1,250 cases) predates the inventory and is not a forecast of remaining work. Remaining automation effort is UNKNOWN until each family separates reusable tests from missing assertions and unavailable product features. No new test-count quota is introduced.

Estimate each epic delta as a range after system-design acceptance, based on fixture reuse and actual test durations; exclude production-feature implementation and infrastructure operations from QA effort. The 30-day burn-in is a PRD calendar constraint, not 30 days of full-time test-authoring effort.

Implementation Planning Handoff

Work item Owner Target Dependencies
Deterministic fixtures/factories and per-worker cleanup QA + Core Epic 1 B-001
OpenAPI corpus oracle/runner Import + QA Epic 2 Stable IR/findings
MCP protocol/representative-client harness MCP + QA Epic 3 Version adapters
Task model/PG chaos harness Task + QA/SRE Epic 4 B-001/B-002
Telemetry/metrics capture and assertions Observability + QA Epic 5 B-003
Calibrated load and reference topology Performance/SRE Epic 6 B-005
Upgrade/restore/air-gap automation Release/Security/QA Epic 7 B-002/B-004
Evidence collector/manifest verifier and RC qualification Release + QA Epic 1→8 B-004, all suites

Tooling and Access

Tool/service Purpose Access Status
PostgreSQL 16/Valkey Testcontainers Storage/coordination integration Container runtime Existing foundation; extend
Playwright 1.61.1 UI and real-stack/mocked-boundary browser tests Node + Chromium Installed; Utils/Pact are preferences only, absent from package.json
MCP protocol harness + ≥3 clients Compatibility matrix Pinned clients Pending feature-freeze selection
Captured OTLP/Sentry endpoint, Prometheus/Grafana/GlitchTip Observability assertions Self-hosted test profile Pending Epic 5
External calibrated load/chaos tooling Cross-check and fault injection Reference host/lab Pending
cargo-deny, npm audit, Trivy / SBOM Dependency/container policy CI toolchain/images Configured; current scan reports and complete SAST/secret/signature evidence not established by this audit

Interworking and Regression

Component Impact Regression scope Validation
Core/runtime/REST adapter Shared execution semantics/security Existing REST→MCP + all ingress paths CP-01..04, CP-16, malicious upstream
Import/artifacts/registry New parser/transactional plans OpenAPI and persisted revisions CP-05..07
MCP server/catalog New capabilities and adapters Stable/N-1/N-2 clients CP-08..11
Task worker/PostgreSQL/Valkey New durable concurrency State, contention, restart, PG-only CP-12..15
UI/Admin API New operator flows/security RU/EN, async state, auth, browser matrix CP-07/17/24/25/33
Observability/metrics stack Cross-cutting data path Correlation, redaction, cardinality, SLO CP-16..21
Load/deploy/release assets Production operations Safety, recovery, air gap, evidence CP-22..31

Every fixed defect adds the lowest-level regression and a boundary case if the failure crossed a contract.

Appendix A: Reproducible Discovery and Evidence

# From apps/ui: enumerate only; does not prove a pass.
./node_modules/.bin/playwright test --list --reporter=line
# From the repository: static test attributes, not coverage.
git grep -h -E '#\[(tokio::test|test)' HEAD -- apps crates | wc -l

For each future accepted run retain source revision/tree, image/config/topology identifiers, dependency/client/browser versions, command, timestamps, cases passed/failed/skipped/flaky, raw report digest and sanitized diagnostics. Source files and generated “completed” planning metadata cannot serve as execution evidence. Keep secrets, credentials and unbounded payloads out of retained reports.

Current tests do not have a complete @P0/@P1 tag taxonomy; selecting tags before annotating the suite may run zero tests. Use known files/test names now, and make empty mandatory selections fail. Report retention must cover ordinary failing runs as well as successful sanitized OpenAPI release evidence.

Edit Verification — 2026-09-13

Read source/configuration and previous plans; ran Playwright discovery only. Product tests, benchmarks, browser runs and deploy were not rerun. Document checks confirm 34 unique CP families covering all FR-1..54 at family level, shared risk identifiers, preserved PRD thresholds and explicit missing evidence. This is not an assertion-level coverage proof or final NFR PASS/CONCERNS/FAIL verdict.

Appendix B: Knowledge Base References

  • risk-governance.md and test-priorities-matrix.md
  • test-levels-framework.md and test-quality.md
  • nfr-criteria.md and adr-quality-readiness-checklist.md
  • playwright-cli.md and contract-testing.md

Generated by: BMad TEA Agent
Workflow: bmad-testarch-test-design