Compare commits

...
Author SHA1 Message Date
bsodfather 5488eb73ec chore(migrations): зафиксировать каноническую последовательность v26–v28
CI / Rust Checks (pull_request) Successful in 22m6s
CI / Rust Feature Validation (pull_request) Skipped
CI / Frontend E2E (pull_request) Skipped
CI / Community Image Smoke (pull_request) Skipped
CI / Required Checks (pull_request) Successful in 1s
CI / Deploy (pull_request) Skipped
CI / UI Checks (pull_request) Successful in 12s
Обязательный CI-гейт «Check canonical migration contract» сравнивает
`crank-migrate plan` с docs/schemas/migration-sequence.json. Принятая дельта
Story 4.3 добавила миграции v26–v28, но снимок контракта не был регенерирован,
поэтому exact-head CI падал на точном PR head с contract_drift (plan.compare).

Файл получен машинной регенерацией из того же дерева:
`crank-migrate plan > docs/schemas/migration-sequence.json`; проверка
`crank-migrate plan --check` возвращает {"status":"contract_current"}.
Содержимое меняется только добавлением трёх записей 26/27/28; существующие
записи 1–25 и их checksum не изменены, поведение продукта не меняется.
2026-10-29 22:43:05 +00:00
bsodfather f9f8c625d1 feat(epic4): исполнять durable Task через PG lease/fence и восстанавливать после сбоев (Story 4.3)
CI / UI Checks (pull_request) Successful in 15s
CI / Rust Checks (pull_request) Failing after 4m58s
CI / Rust Feature Validation (pull_request) Skipped
CI / Frontend E2E (pull_request) Skipped
CI / Community Image Smoke (pull_request) Skipped
CI / Required Checks (pull_request) Failing after 1s
CI / Deploy (pull_request) Skipped
mcp-server запускает сохранённые фоновые задачи: захват по PostgreSQL lease с
fence-токеном, зашифрованный input и повторная проверка доступа до сетевого
вызова, checkpoint с AttemptID до dispatch и восстановление только по PG.
Устаревший worker не может зафиксировать результат.

Добавлены миграции v26–v28 (V25 не меняется), guard их отсутствия в старой
схеме, метрики/трассы попыток (attempt_trace) и остановка до отправки запроса:
стоп в окне после checkpoint и до permit завершает задачу как
failed(shutdown_before_dispatch) без сетевых вызовов, соседние queued-задачи
остаются доступными. Retry-попытки получают отдельные дочерние spans с durable
current_attempt_id. Shared admission/fairness из Story 4.1 сохранены.

Проверки: durable_tasks 13/13, stages 11/11 (retry 5), runtime lib 47,
bin shutdown 1, registry tasks 16, admission 11, migrations 58 (независимая QA
и исходный runtime-прогон), точный exact-head CI остаётся обязательным.
2026-10-29 22:26:52 +00:00
bsodfather 6df6ffac4e Merge pull request 'Story 4.2 — подтверждать durable Task после атомарного PostgreSQL commit (FR-22, FR-23)' (#109) from delivery/epic4-story-4.2 into main
CI / UI Checks (push) Successful in 13s
CI / Rust Checks (push) Successful in 21m12s
CI / Rust Feature Validation (push) Skipped
CI / Frontend E2E (push) Skipped
CI / Community Image Smoke (push) Skipped
CI / Required Checks (push) Successful in 1s
CI / Deploy (push) Skipped
Community Release / release (push) Successful in 36m42s
Merge accepted Story 4.2 durable Task acceptance
2026-10-27 07:33:30 +00:00
bsodfather 2219a7217d feat(epic4): подтверждать durable Task после атомарного PG-commit (Story 4.2)
CI / UI Checks (pull_request) Successful in 8s
CI / Rust Checks (pull_request) Successful in 19m34s
CI / Rust Feature Validation (pull_request) Skipped
CI / Frontend E2E (pull_request) Skipped
CI / Community Image Smoke (pull_request) Skipped
CI / Required Checks (pull_request) Successful in 1s
CI / Deploy (pull_request) Skipped
Application entry выдаёт Task ID только после атомарного PostgreSQL commit
Task+intent+capacity; replay/conflict, owner scope, bounded envelope,
terminal CAS и fail-closed readiness проверены на реальном PG.

Также включён принятый план перехода на rmcp (только документ, миграция не
запускается).
2026-10-27 00:09:36 +00:00
bsodfather cc612516a2 Merge pull request 'Story 4.1 — публиковать execution mode и применять admission limits (FR-21, FR-24)' (#104) from delivery/epic4-story-4.1 into main
CI / Rust Checks (push) Successful in 20m52s
CI / Rust Feature Validation (push) Skipped
CI / Frontend E2E (push) Skipped
CI / UI Checks (push) Successful in 10s
CI / Community Image Smoke (push) Skipped
CI / Required Checks (push) Successful in 1s
CI / Deploy (push) Skipped
2026-10-26 21:38:06 +00:00
bsodfather 599dc1867b Merge pull request 'ci: безопасно интегрировать принятый Crank и отделить production deploy' (#105) from ops/separate-production-deploy into main
CI / UI Checks (push) Successful in 12s
CI / Rust Checks (push) Successful in 16m0s
CI / Rust Feature Validation (push) Skipped
CI / Frontend E2E (push) Skipped
CI / Community Image Smoke (push) Skipped
CI / Required Checks (push) Successful in 2s
CI / Deploy (push) Skipped
2026-10-26 21:36:09 +00:00
Codex 4e7440f25e ci: separate protected source integration from explicit production deployment
CI / UI Checks (pull_request) Successful in 8s
CI / Rust Checks (pull_request) Successful in 12m54s
CI / Rust Feature Validation (pull_request) Skipped
CI / Frontend E2E (pull_request) Skipped
CI / Community Image Smoke (pull_request) Skipped
CI / Required Checks (pull_request) Successful in 3s
CI / Deploy (pull_request) Skipped
2026-10-27 00:21:37 +03:00
bsodfather e80d30ddb0 feat(epic4): публиковать execution mode и применять общий admission (Story 4.1, FR-21, FR-24)
CI / UI Checks (pull_request) Successful in 11s
CI / Rust Checks (pull_request) Successful in 12m45s
CI / Rust Feature Validation (pull_request) Skipped
CI / Frontend E2E (pull_request) Skipped
CI / Community Image Smoke (pull_request) Skipped
CI / Required Checks (pull_request) Successful in 1s
CI / Deploy (pull_request) Skipped
Community Release / release (push) Successful in 40m35s
Tool Draft фиксирует synchronous/background/adaptive, deadline и fallback;
runtime применяет единый снимок политики (concurrency, очередь, approval,
deadline) до внешнего действия. Насыщение отклоняется typed overload с
retry guidance, mandatory safety store fail closed, background/adaptive
используют опубликованный synchronous/unsupported fallback.

Refs: #29
2026-10-26 21:08:08 +00:00
bsodfather 44774883a0 fix(epic3): выводить Tools-сигнал из клиентской проекции tools/list (Story 3.5, FR-14)
CI / UI Checks (pull_request) Successful in 8s
CI / Rust Checks (pull_request) Successful in 10m49s
CI / Rust Feature Validation (pull_request) Skipped
CI / Frontend E2E (pull_request) Skipped
CI / Community Image Smoke (pull_request) Skipped
CI / Required Checks (pull_request) Successful in 1s
CI / Deploy (pull_request) Skipped
Community Release / release (push) Successful in 25m35s
QA F1: CatalogFingerprint сравнивал только Tool-bindings, а tools/list
строится из catalog_tool_definitions, которая ещё зависит от
tool_selection_policy (mode, groups, search.max_results); в Search-режиме
отдаваемый список вообще не зависит от bindings. Из-за этого commit,
меняющий только политику, не порождал notifications/tools/list_changed
(клиент оставался со stale-каталогом), а binding-only commit в Search
порождал ложный сигнал.

Теперь Tools-часть отпечатка — канонизированные определения, которые
реально отдаёт tools/list (отсортированы, чтобы порядок чтения не был
ложным изменением): сигнал есть ровно тогда, когда меняется видимая
клиенту проекция.

ATDD: новый сценарий на обеих ревизиях — policy-only commit (Direct ->
Search) даёт ровно один Tools-кадр и авторитетный tools/list с
search_tools/call_tool, а последующий binding-only commit в Search-режиме
не сигналит. Юнит-оракулы переведены на реальную проекцию: policy-only
change, невидимое поле политики, version bump с идентичным определением и
binding change в Search-режиме — silent.
2026-10-26 17:22:07 +00:00
hermes-dev a0365bb80b feat(epic3): отправлять Tools catalog change signals после commit (Story 3.5, FR-14)
Клиент, согласовавший tools list-changed, получает bounded
`notifications/tools/list_changed` после коммита видимой Tool-проекции
опубликованного Agent snapshot; `tools/list` остаётся авторитетным для
клиента без notification и после reconnect.

- signals.rs: третий тип сигнала в существующем stream-local watcher
  (один tick = максимум один кадр на тип, fixed-capacity queue, без
  cross-process bus и без post-commit hook); fingerprint сравнивает
  tool_name/operation.version/title/description, поэтому Tool-only
  commit не даёт Resource/Prompt сигналов и наоборот.
- registry.rs: tools.listChanged=true (реально отправляется в обеих
  revision); modern filter toolsListChanged теперь подтверждается
  acknowledge и не шлётся вне согласованного набора.
- legacy GET-стрим несёт Tools-сигнал вместе с Resource/Prompt.
- ATDD: RED на baseline (ack пуст, Tools-кадра нет, capability false),
  GREEN — 8 integration-сценариев × обе revision, unit-оракулы wire,
  representatives official MCP SDK 1.32.2 smoke со штатными headers.
- capability inventory/baseline manifest и frozen projection обновлены
  под tools.listChanged=true; ci-story-checks.sh дополнен AC 3.5.
2026-10-26 17:22:07 +00:00
bsodfather 2499016dc7 fix(epic3): принимать поток progress при JSON-first Accept официального SDK (Story 3.4, FR-14)
CI / Rust Checks (pull_request) Successful in 9m12s
CI / Rust Feature Validation (pull_request) Skipped
CI / Frontend E2E (pull_request) Skipped
CI / Community Image Smoke (pull_request) Skipped
CI / Required Checks (pull_request) Successful in 1s
CI / Deploy (pull_request) Skipped
CI / UI Checks (pull_request) Successful in 10s
Community Release / release (push) Successful in 24m39s
Официальный MCP SDK перечисляет медиатипы в `Accept` JSON-first
(`application/json, text/event-stream`). Прежняя реализация выбирала
negotiated JSON-тело, и клиент, которому нужен progress, не получал
ни одного кадра.

Коррекция переиспользует уже провалидированные шаги: для `tools/call`
с допустимым progress token и предложенным event-stream режим ответа
переключается на SSE. Admission/авторизация не двигались; другой метод,
отсутствующий или непригодный token и JSON-only `Accept` сохраняют
negotiated режим.

ATDD: тот же JSON-first `Accept` даёт строго монотонные progress 1/2/3 и
прежний result в обеих supported revisions, а негативные контроли
(без token и с непригодным token) остаются на JSON-теле.
2026-10-26 16:03:48 +00:00
bsodfather f613250fd3 fix(epic3): сделать ATDD отмены чувствительным к регрессии и покрыть bounds токена в обеих revision (Story 3.4, FR-14)
- CANCEL_GRACE_MS 1200 -> 2500 (> LONG_HOLD_MS) и const-assert на этот порядок:
  при незавершённой отмене вызов успевает завершиться и записать исход внутри
  окна проверки, поэтому тест доказывает остановку, а не только «ещё не готово».
- malformed/oversized/empty/object/array token теперь проверяются и в modern
  revision, не только в legacy (обе читают одно поле RequestMetaObject).
- комментарий cancelled_response приведён к фактическому поведению: у
  остановленного вызова строки Invocation History нет вообще.
2026-10-26 13:07:14 +00:00
hermes_dev 862cde8559 feat(epic3): progress и безопасная отмена Tool calls (Story 3.4, FR-14)
- request-scoped lifecycle одного tools/call: валидированный `_meta.progressToken`
  (строковый/числовой, 1..=128 байт) и bounded очередь реальных этапов
  (tool_resolved → authorization_resolved → execution_dispatched) в собственный
  SSE-stream запроса; без token или без stream-carrier результат не меняется;
- сериализация `notifications/progress` по pinned схеме обеих revisions
  (progressToken/progress/message, total не изобретается);
- отмена по revision: 2025-11-25 — `notifications/cancelled` через bounded
  in-flight таблицу (владение = session + platform credential, не progress token);
  2026-07-28 — закрытие response stream; в обоих случаях drop будущего вызова =
  кооперативная остановка, terminal-сообщение не отправляется;
- streamed ответ сохраняет проекцию Story 3.18: closed `notifications/message`
  перед финальным кадром (перенос logging-scope в тело стрима);
- ATDD: 12 end-to-end тестов на реальном endpoint/каталоге/counted upstream
  (обе revisions, malformed/oversized/reused token, owner/foreign/late cancel,
  disconnect, ambiguous outbound, representative client smoke) + 7 unit-тестов
  границ lifecycle; entry в scripts/ci-story-checks.sh.
2026-10-26 12:02:11 +00:00
bsodfather 49c8af24ed fix(epic3): обновить checksum inventory в capability baseline (Story 3.19)
CI / Rust Checks (pull_request) Successful in 8m5s
CI / Rust Feature Validation (pull_request) Skipped
CI / UI Checks (pull_request) Successful in 31m50s
CI / Frontend E2E (pull_request) Skipped
CI / Community Image Smoke (pull_request) Skipped
CI / Required Checks (pull_request) Successful in 1s
CI / Deploy (pull_request) Skipped
Community Release / release (push) Successful in 22m34s
Story 3.19 меняет `docs/capability-inventory.json` (статусы `listChanged`,
`subscriptions/listen`, `notifications/*/list_changed`), поэтому
`docs/capability-baseline/manifest.json` должен ссылаться на новый SHA-256 того
же артефакта: `85970569…` → `10ca3538aa1b6b25868f80e19932c7ef918398c4817c52bc1aae52f18c31522b`,
ровно как в `tests/fixtures/mcp-revisions/*.provenance.json` этого же delta.

Поймано Required CI на `a965bc4` (run 887, job 2491): шаг
`Check Capability Baseline` падал с `CHECKSUM_MISMATCH
pointer=/artifacts/0/sha256`; остальные шаги того же job'а (tooling unit tests
305 OK, capability inventory `status=valid protocol_items=50`, community scope)
были зелёными. Локально после правки:
`validate-capability-baseline.py` → `baseline_version=2026.08.24.1 flows=35`,
`validate-capability-inventory.py` → `status=valid`, community scope (1263
файла), runtime-config/config/metrics/rust boundaries и code health — 0.

Содержательных изменений продукта здесь нет: только checksum-указатель на уже
изменённый в предыдущем коммите артефакт.
2026-10-25 21:08:20 +00:00
bsodfather a965bc4137 feat(epic3): уведомления об изменении Resource/Prompt каталога MCP (Story 3.19)
CI / UI Checks (pull_request) Successful in 9s
CI / Rust Checks (pull_request) Failing after 4m10s
CI / Rust Feature Validation (pull_request) Skipped
CI / Frontend E2E (pull_request) Skipped
CI / Community Image Smoke (pull_request) Skipped
CI / Required Checks (pull_request) Failing after 1s
CI / Deploy (pull_request) Skipped
Story 3.19 / FR-19 на принятой базе Story 3.18 (`47a6e54`, PR #99): после
commit опубликованного каталога Agent клиент получает
`notifications/resources/list_changed` и/или
`notifications/prompts/list_changed`, обе pinned revisions, а обычное чтение
каталога остаётся авторитетным при потере сигнала. Scope ограничен
list-change: `resources/subscribe`/`unsubscribe` остаются `-32601`, значение
`resources.subscribe` остаётся `false` (одобренный владельцем вариант A).

Публикация переносит только semantic delta Story 3.19 из
`/home/codex/code/crank-worktrees/story31-worktree`
(`1b6db09` + cumulative overlay) на принятый head 3.18, а не копирует
worktree:
- новый `protocol/signals.rs` (watcher, отпечаток каталога, bounded filter,
  revision-correct encoders, bounded legacy stream registry) и
  `protocol/signals_tests.rs` (17 unit oracles);
- `subscriptions/listen` (modern-only dispatch registration) + честные
  `listChanged: true` объявления; `app/endpoint.rs` получает legacy GET
  producer/DELETE retire, `app/dispatch.rs` — modern listen stream;
- ATDD `mcp_signals` 5 сценариев на реальном endpoint/PostgreSQL/artifact
  store; capability inventory и frozen/provenance проекции обновлены
  (`b90b59f3…`, inventory `10ca3538…`);
- `scripts/ci-story-checks.sh`: добавлены 3.19-проверки, записи Story 3.18
  сохранены (гейт не ослаблен).

Принятая правка code-health `206ca78` сохранена: `app/endpoint.rs` 402 строки
(было 381), `app/legacy_dispatch.rs` 731 без изменений, т.е. split не
переписан. Новый ATDD-файл `mcp_signals.rs` до 1138 строк превышал лимит
`scripts/check-rust-code-health.sh`, поэтому его сценарии механически
вынесены в `mcp_signals/scenarios.rs` без изменения проверок.

Наследованный дефект tooling-гейта: `tests/unit/test_validate_capability_inventory.py`
требует, чтобы `Run tooling unit tests` шёл раньше
`scripts/validate-capability-inventory.py`; в принятом `ci.yml` (3.18) шаг
оказался в условном feature-job. Шаг перенесён в обязательный `rust` job перед
проверкой inventory — исполняемый tooling-гейт теперь проверяется там же, где
применяется; ни одна проверка не отключена и не ослаблена.

Проверки на этом коммите: `ci-story-checks.sh` exit 0 — 4/9/1/12/10/3/5(+1
manual ignored)/17 passed; `crank-community-mcp --lib` 102/0; transport 20/0,
mcp_prompts 15/0, revision_matrix::negotiation 23/0, era_isolation 5/0;
`cargo fmt --check` и `cargo clippy -p crank-community-mcp -p mcp-server
--all-targets -D warnings` — 0; `check-rust-code-health.sh` — pass;
capability inventory `status=valid protocol_items=50 revisions=2`;
`python3 -m unittest discover -s tests/unit` — 305 OK.

NOT_RUN (честно, не PASS): полный `just verify`, межкомпонентная интеграция,
нагрузка и квалификация ≥3 реальных клиентов — остаются на собранной фиче 3.20
по validation-policy.md.
2026-10-25 21:00:51 +00:00
hermes-dev 47a6e54660 fix(ui): bump nginx runtime base to 1.30.6 for the image scan gate
CI / Rust Checks (pull_request) Successful in 6m33s
CI / Rust Feature Validation (pull_request) Skipped
CI / UI Checks (pull_request) Successful in 16m36s
CI / Frontend E2E (pull_request) Skipped
CI / Community Image Smoke (pull_request) Skipped
CI / Required Checks (pull_request) Successful in 1s
CI / Deploy (pull_request) Skipped
Community Release / release (push) Failing after 16m35s
2026-10-25 16:19:11 +00:00
bsodfather 206ca784f7 fix(epic3): уложить endpoint.rs Story 3.18 в лимит 1000 строк
CI / UI Checks (pull_request) Successful in 11s
CI / Rust Checks (pull_request) Successful in 9m2s
CI / Rust Feature Validation (pull_request) Skipped
CI / Frontend E2E (pull_request) Skipped
CI / Community Image Smoke (pull_request) Skipped
CI / Required Checks (pull_request) Successful in 1s
CI / Deploy (pull_request) Skipped
The Story 3.18 logging arm pushed `crates/crank-community-mcp/src/app/endpoint.rs`
to 1058 lines, so `scripts/check-rust-code-health.sh` failed the Required CI
(Rust Checks, run 870 / job 2408) at the pinned 1000-line default.

Move the whole legacy-era dispatch of `mcp_post_response` verbatim into
`app/legacy_dispatch.rs` (`handle_legacy_request`), leaving the endpoint as the
transport/admission layer it documents itself to be. endpoint.rs is now 381
lines and the new module 731, both inside the limit; no baseline pin and no
check was touched.

Behaviour is unchanged: the moved match keeps every arm, envelope, status code
and safe detail as-is (legacy-only logging/setLevel, -32602 INVALID_LEVEL,
EVENT_LEVEL_APPLIED Notice acknowledgment, per-session level isolation), the
pre-dispatch legacy level read stays in `mcp_post_response`, and the only edits
to the moved text are the mechanical `&credential` -> `credential` (the
credential now arrives as a reference) and `protocol_version` already being
`Option<&str>`.
2026-10-25 15:45:01 +00:00
bsodfather a438c1df42 feat(epic3): безопасное structured logging MCP (Story 3.18)
CI / UI Checks (pull_request) Successful in 11s
CI / Rust Checks (pull_request) Failing after 4m11s
CI / Rust Feature Validation (pull_request) Skipped
CI / Frontend E2E (pull_request) Skipped
CI / Community Image Smoke (pull_request) Skipped
CI / Required Checks (pull_request) Failing after 1s
CI / Deploy (pull_request) Skipped
Story 3.18 (#25) surface: клиент получает только закрытые structured
`notifications/message` выбранного уровня, без запросов, секретов и
internal causes.

- `logging` capability advertising в обеих поддерживаемых revisions;
  `logging/setLevel` — legacy-only dispatch (modern — `_meta` per request);
- закрытый словарь LoggingLevel (RFC 5424), malformed → -32602 с одним
  safe detail; закрытый набор safe events, task-local request scope;
  bounded in-memory opt-in, без миграций;
- `notifications/message` идут перед ответом в SSE; JSON-режим их не несёт;
- ATDD в `scripts/ci-story-checks.sh`: mcp_logging (10), inventory projection (3),
  protocol::logging unit (12);
- проекция capability inventory/frozen oracle переведена на `logging`.

Story ATDD и affected проверки пройдены на этом дереве (fmt, clippy,
python oracle 56 OK, capability inventory valid). Полный `just verify` —
NOT_RUN на story по validation-policy, остаётся на собранной фиче.
2026-10-25 12:50:00 +00:00
156 changed files with 21451 additions and 1113 deletions

No files matched your search

+10 -3
View File
@@ -11,6 +11,12 @@ on:
options: [story, feature]
default: story
required: true
deploy_target:
description: Deployment target for this run
type: choice
options: [none, production]
default: none
required: true
push:
branches:
- main
@@ -81,6 +87,9 @@ jobs:
cargo run -p crank-metrics --bin crank-metrics-contract -- --check
python3 scripts/check-metrics-boundaries.py --root .
- name: Run tooling unit tests
run: python3 -m unittest discover -s tests/unit
- name: Check Capability Inventory
run: |
required_args=""
@@ -203,8 +212,6 @@ jobs:
docker info
- name: Install dependency policy tool
run: cargo install cargo-deny --version 0.20.2 --locked
- name: Run tooling unit tests
run: python3 -m unittest discover -s tests/unit
- name: Check dependency licenses and advisories
run: cargo deny --locked check advisories bans licenses sources
- name: Run offline OpenAPI support corpus gate
@@ -498,7 +505,7 @@ jobs:
runs-on: ubuntu-latest
needs:
- required-checks
if: ${{ gitea.event_name == 'push' && gitea.ref == 'refs/heads/main' }}
if: ${{ gitea.event_name == 'workflow_dispatch' && gitea.ref == 'refs/heads/main' && inputs.validation_level == 'feature' && inputs.deploy_target == 'production' }}
env:
REGISTRY: git.itexp.me
IMAGE_TAG: ${{ gitea.sha }}
Generated
+1
View File
@@ -826,6 +826,7 @@ dependencies = [
name = "crank-registry"
version = "0.3.1"
dependencies = [
"async-trait",
"crank-artifacts",
"crank-core",
"crank-mapping",
+99 -87
View File
@@ -2,6 +2,10 @@
mod admin_auth_command;
#[path = "crank_migrate/db_connect.rs"]
mod db_connect;
#[path = "crank_migrate/reporting.rs"]
mod reporting;
use reporting::{active_rotation, ensure_no_options, read_required_key_file, rotation_json};
use crank_config::{ConfigSource, parse_migrator};
use crank_registry::{
@@ -11,7 +15,7 @@ use crank_registry::{
};
use crank_runtime::SecretCrypto;
use serde_json::json;
use std::{path::Path, process::ExitCode};
use std::process::ExitCode;
use time::OffsetDateTime;
const MASTER_KEY_ROTATION_PAGE_SIZE: i64 = 1_000;
#[tokio::main]
@@ -765,6 +769,59 @@ async fn process_rotation_batch(
processed += 1;
}
}
let mut after_task_id: Option<String> = None;
loop {
let tasks = registry
.list_tasks_for_master_key_epoch_page(
rotation.source_epoch,
after_task_id.as_deref(),
MASTER_KEY_ROTATION_PAGE_SIZE,
)
.await
.map_err(CliError::from_registry)?;
if tasks.is_empty() {
break;
}
for task in tasks {
after_task_id = Some(task.task_id.clone());
if task.target_epoch == Some(rotation.target_epoch) {
continue;
}
if max_versions.is_some_and(|limit| processed as usize >= limit) {
return Ok(processed);
}
let plaintext = current_crypto
.decrypt_for_epoch(&task.key_version, task.epoch, &task.ciphertext)
.map_err(|_| {
CliError::new(
"master_key_rotation_verification_failed",
"master_key.rotation",
"rerun_rotation_or_abort",
)
})?;
let target_ciphertext = target_crypto.encrypt(&plaintext).map_err(|_| {
CliError::new(
"master_key_rotation_verification_failed",
"master_key.rotation",
"rerun_rotation_or_abort",
)
})?;
let now = OffsetDateTime::now_utc();
registry
.stage_master_key_rotation_task_ciphertext(
&rotation.id,
&task.task_id,
rotation.source_epoch,
&target_ciphertext,
target_crypto.key_version(),
rotation.target_epoch,
&now,
)
.await
.map_err(CliError::from_registry)?;
processed += 1;
}
}
Ok(processed)
}
@@ -818,6 +875,47 @@ async fn verify_staged_targets(
verified += 1;
}
}
let mut after_task_id: Option<String> = None;
loop {
let tasks = registry
.list_target_tasks_for_master_key_rotation_page(
rotation.target_epoch,
after_task_id.as_deref(),
MASTER_KEY_ROTATION_PAGE_SIZE,
)
.await
.map_err(CliError::from_registry)?;
if tasks.is_empty() {
break;
}
for task in tasks {
after_task_id = Some(task.task_id.clone());
let ciphertext = task.target_ciphertext.as_deref().ok_or_else(|| {
CliError::new(
"master_key_rotation_verification_failed",
"master_key.rotation",
"rerun_rotation_or_abort",
)
})?;
let key_version = task.target_key_version.as_deref().ok_or_else(|| {
CliError::new(
"master_key_rotation_verification_failed",
"master_key.rotation",
"rerun_rotation_or_abort",
)
})?;
target_crypto
.decrypt_for_epoch(key_version, rotation.target_epoch, ciphertext)
.map_err(|_| {
CliError::new(
"master_key_rotation_verification_failed",
"master_key.rotation",
"rerun_rotation_or_abort",
)
})?;
verified += 1;
}
}
if verified != rotation.total_secret_versions {
return Err(CliError::new(
"master_key_rotation_verification_failed",
@@ -877,89 +975,3 @@ fn decrypt_current(
)
})
}
async fn active_rotation(
registry: &PostgresRegistry,
allowed_states: &[&str],
) -> Result<Option<MasterKeyRotationRecord>, CliError> {
let status = registry
.master_key_rotation_status()
.await
.map_err(CliError::from_registry)?;
Ok(status
.rotations
.into_iter()
.find(|rotation| allowed_states.contains(&rotation.state.as_str())))
}
fn rotation_json(rotation: &MasterKeyRotationRecord) -> serde_json::Value {
json!({
"id": rotation.id,
"source_epoch": rotation.source_epoch,
"target_epoch": rotation.target_epoch,
"state": rotation.state,
"backup_ref": rotation.backup_ref.as_ref().map(|_| "configured"),
"checkpoint_secret_id": rotation.checkpoint_secret_id,
"total_secret_versions": rotation.total_secret_versions,
"processed_secret_versions": rotation.processed_secret_versions,
"verified_secret_versions": rotation.verified_secret_versions,
"failure_code": rotation.failure_code,
})
}
fn read_required_key_file(path: Option<&str>) -> Result<String, CliError> {
let path = path.ok_or_else(|| {
CliError::new("invalid_command", "cli.arguments", "run_master_key_status")
})?;
if path.len() > 512 || path.bytes().any(|byte| byte.is_ascii_control()) {
return Err(CliError::new(
"master_key_input_invalid",
"master_key.input",
"verify_operator_input",
));
}
let metadata = std::fs::metadata(Path::new(path)).map_err(|_| {
CliError::new(
"master_key_input_invalid",
"master_key.input",
"verify_operator_input",
)
})?;
if !metadata.is_file() || metadata.len() > 16_384 {
return Err(CliError::new(
"master_key_input_invalid",
"master_key.input",
"verify_operator_input",
));
}
let value = std::fs::read_to_string(Path::new(path)).map_err(|_| {
CliError::new(
"master_key_input_invalid",
"master_key.input",
"verify_operator_input",
)
})?;
if value.trim().is_empty() {
return Err(CliError::new(
"master_key_input_invalid",
"master_key.input",
"verify_operator_input",
));
}
Ok(value)
}
fn ensure_no_options(options: &MasterKeyOptions) -> Result<(), CliError> {
if options.current_key_file.is_some()
|| options.target_key_file.is_some()
|| options.backup_ref.is_some()
|| options.rotation_id.is_some()
|| options.max_versions.is_some()
{
return Err(CliError::new(
"invalid_command",
"cli.arguments",
"run_master_key_status",
));
}
Ok(())
}
@@ -0,0 +1,98 @@
//! Leaf/reporting helpers of the migrator CLI.
//!
//! Extracted from `crank-migrate.rs` without behaviour changes so the binary entry stays within
//! the code-health line limit.
use std::path::Path;
use crank_registry::{MasterKeyRotationRecord, PostgresRegistry};
use serde_json::json;
use super::{CliError, MasterKeyOptions};
pub(super) async fn active_rotation(
registry: &PostgresRegistry,
allowed_states: &[&str],
) -> Result<Option<MasterKeyRotationRecord>, CliError> {
let status = registry
.master_key_rotation_status()
.await
.map_err(CliError::from_registry)?;
Ok(status
.rotations
.into_iter()
.find(|rotation| allowed_states.contains(&rotation.state.as_str())))
}
pub(super) fn rotation_json(rotation: &MasterKeyRotationRecord) -> serde_json::Value {
json!({
"id": rotation.id,
"source_epoch": rotation.source_epoch,
"target_epoch": rotation.target_epoch,
"state": rotation.state,
"backup_ref": rotation.backup_ref.as_ref().map(|_| "configured"),
"checkpoint_secret_id": rotation.checkpoint_secret_id,
"total_secret_versions": rotation.total_secret_versions,
"processed_secret_versions": rotation.processed_secret_versions,
"verified_secret_versions": rotation.verified_secret_versions,
"failure_code": rotation.failure_code,
})
}
pub(super) fn read_required_key_file(path: Option<&str>) -> Result<String, CliError> {
let path = path.ok_or_else(|| {
CliError::new("invalid_command", "cli.arguments", "run_master_key_status")
})?;
if path.len() > 512 || path.bytes().any(|byte| byte.is_ascii_control()) {
return Err(CliError::new(
"master_key_input_invalid",
"master_key.input",
"verify_operator_input",
));
}
let metadata = std::fs::metadata(Path::new(path)).map_err(|_| {
CliError::new(
"master_key_input_invalid",
"master_key.input",
"verify_operator_input",
)
})?;
if !metadata.is_file() || metadata.len() > 16_384 {
return Err(CliError::new(
"master_key_input_invalid",
"master_key.input",
"verify_operator_input",
));
}
let value = std::fs::read_to_string(Path::new(path)).map_err(|_| {
CliError::new(
"master_key_input_invalid",
"master_key.input",
"verify_operator_input",
)
})?;
if value.trim().is_empty() {
return Err(CliError::new(
"master_key_input_invalid",
"master_key.input",
"verify_operator_input",
));
}
Ok(value)
}
pub(super) fn ensure_no_options(options: &MasterKeyOptions) -> Result<(), CliError> {
if options.current_key_file.is_some()
|| options.target_key_file.is_some()
|| options.backup_ref.is_some()
|| options.rotation_id.is_some()
|| options.max_versions.is_some()
{
return Err(CliError::new(
"invalid_command",
"cli.arguments",
"run_master_key_status",
));
}
Ok(())
}
+1
View File
@@ -181,6 +181,7 @@ mod tests {
idempotency: None,
safety: None,
approval_policy: None,
execution_policy: None,
auth_profile_ref: None,
headers: BTreeMap::new(),
},
+18 -1
View File
@@ -3,7 +3,7 @@ use std::{io, net::SocketAddr, process::ExitCode, time::Duration};
use admin_api::{
app::build_app,
auth::{AuthSettings, BootstrapAdminConfig},
pool_metrics::spawn_postgres_pool_metrics,
pool_metrics::{spawn_admission_metrics, spawn_postgres_pool_metrics},
reconciliation::{open_reconciliation_store, spawn_artifact_reconciliation},
service::AdminServiceBuilder,
state::AppState,
@@ -169,6 +169,7 @@ async fn run(
};
if metrics_enabled {
spawn_postgres_pool_metrics(registry.pool().clone());
spawn_admission_metrics(registry.pool().clone());
}
let base_url = config
.runtime
@@ -214,6 +215,16 @@ async fn run(
.with_limits(runtime_limits)
.with_response_cache(cache_stores.response.clone())
.with_coordination_store(cache_stores.coordination.clone())
.with_admission(std::sync::Arc::new(
crank_registry::PostgresExecutionAdmission::new(
registry.pool().clone(),
crank_registry::AdmissionLimits {
global_limit: runtime_limits.max_concurrent_unary,
..crank_registry::AdmissionLimits::default()
},
crank_core::AdmissionSchedulerConfig::default(),
)?,
))
.build();
let identity_provider =
PasswordIdentityProvider::new(registry.clone(), auth_settings.password_pepper.clone());
@@ -321,6 +332,12 @@ fn preflight_config(config: &AdminProcessConfig) -> Result<(), crank_config::Con
config.runtime.max_concurrent_sessions,
)
.map_err(|_| invalid("runtime.limits"))?;
crank_registry::AdmissionLimits {
global_limit: config.runtime.max_concurrent_unary,
..crank_registry::AdmissionLimits::default()
}
.validate()
.map_err(|_| invalid("runtime.admission"))?;
runtime_cache_config(config).map_err(|_| invalid("cache"))?;
RequestRateLimitConfig::new(
config.rate_limit.requests_per_second,
+14
View File
@@ -11,3 +11,17 @@ pub fn spawn_postgres_pool_metrics(pool: PgPool) -> tokio::task::JoinHandle<()>
}
})
}
/// Story 4.1: samples the authoritative admission queue so the oldest-waiting age grows (and returns
/// to zero) without waiting for a new request.
pub fn spawn_admission_metrics(pool: PgPool) -> tokio::task::JoinHandle<()> {
tokio::spawn(async move {
let mut interval = tokio::time::interval(Duration::from_secs(5));
loop {
interval.tick().await;
if let Ok(snapshot) = crank_registry::admission_snapshot(&pool).await {
crank_observability::record_admission_oldest_waiting(snapshot.oldest_waiting_ms);
}
}
})
}
+27 -2
View File
@@ -37,13 +37,22 @@ pub async fn readiness(State(state): State<AppState>) -> impl IntoResponse {
} else {
"not_ready"
};
let execution_admission = if checks.execution_admission {
"ready"
} else {
"not_ready"
};
if checks.is_ready() {
(
StatusCode::OK,
Json(json!({
"service": "admin-api",
"status": "ready",
"checks": { "postgres": postgres, "artifact_storage": artifact_storage }
"checks": {
"postgres": postgres,
"artifact_storage": artifact_storage,
"execution_admission": execution_admission
}
})),
)
} else {
@@ -52,7 +61,11 @@ pub async fn readiness(State(state): State<AppState>) -> impl IntoResponse {
Json(json!({
"service": "admin-api",
"status": "not_ready",
"checks": { "postgres": postgres, "artifact_storage": artifact_storage }
"checks": {
"postgres": postgres,
"artifact_storage": artifact_storage,
"execution_admission": execution_admission
}
})),
)
}
@@ -67,6 +80,7 @@ mod tests {
let checks = ReadinessChecks {
postgres: false,
artifact_storage: true,
execution_admission: true,
};
assert!(!checks.is_ready());
assert_eq!(
@@ -92,6 +106,7 @@ mod tests {
let checks = ReadinessChecks {
postgres: true,
artifact_storage: false,
execution_admission: true,
};
assert!(!checks.is_ready());
assert_eq!(
@@ -111,4 +126,14 @@ mod tests {
"not_ready"
);
}
#[test]
fn readiness_checks_identify_an_admission_state_failure() {
let checks = ReadinessChecks {
postgres: true,
artifact_storage: true,
execution_admission: false,
};
assert!(!checks.is_ready());
}
}
+9 -3
View File
@@ -55,8 +55,8 @@ use crate::{
storage::LocalArtifactStorage,
};
use operation_validation::{
validate_approval_policy, validate_execution_timeout, validate_idempotency_policy,
validate_protocol_target, validate_response_cache_policy,
validate_approval_policy, validate_execution_policy, validate_execution_timeout,
validate_idempotency_policy, validate_protocol_target, validate_response_cache_policy,
};
#[derive(Clone)]
@@ -104,11 +104,12 @@ pub use crate::dto::*;
pub struct ReadinessChecks {
pub postgres: bool,
pub artifact_storage: bool,
pub execution_admission: bool,
}
impl ReadinessChecks {
pub const fn is_ready(self) -> bool {
self.postgres && self.artifact_storage
self.postgres && self.artifact_storage && self.execution_admission
}
}
@@ -150,6 +151,8 @@ pub(super) struct CredentialAuditRecord<'a> {
impl AdminService {
pub async fn readiness(&self) -> ReadinessChecks {
let postgres = self.registry.ping().await.is_ok();
// Story 4.1: the mandatory admission state is a readiness dependency of every execution.
let execution_admission = self.registry.execution_admission_health().await.is_ok();
let store = Arc::clone(&self.artifact_store);
let artifact_storage = tokio::task::spawn_blocking(move || store.check_health())
.await
@@ -157,6 +160,7 @@ impl AdminService {
ReadinessChecks {
postgres,
artifact_storage,
execution_admission,
}
}
@@ -725,6 +729,7 @@ impl AdminService {
validate_response_cache_policy(&payload.target, &payload.execution_config)?;
validate_idempotency_policy(&payload.target, &payload.execution_config)?;
validate_approval_policy(&payload.execution_config)?;
validate_execution_policy(&payload.execution_config)?;
payload.input_mapping.validate_paths()?;
payload.output_mapping.validate_paths()?;
Ok(())
@@ -738,6 +743,7 @@ impl AdminService {
validate_response_cache_policy(&operation.target, &operation.execution_config)?;
validate_idempotency_policy(&operation.target, &operation.execution_config)?;
validate_approval_policy(&operation.execution_config)?;
validate_execution_policy(&operation.execution_config)?;
operation.input_mapping.validate_paths()?;
operation.output_mapping.validate_paths()?;
Ok(())
+1
View File
@@ -441,6 +441,7 @@ fn demo_rest_operation_payload() -> OperationPayload {
idempotency: None,
safety: None,
approval_policy: None,
execution_policy: None,
auth_profile_ref: None,
headers: BTreeMap::new(),
},
+1
View File
@@ -635,6 +635,7 @@ impl AdminService {
idempotency: None,
safety: None,
approval_policy: None,
execution_policy: None,
auth_profile_ref: None,
headers: BTreeMap::new(),
},
@@ -610,6 +610,7 @@ impl AdminService {
idempotency: None,
safety: None,
approval_policy: None,
execution_policy: None,
auth_profile_ref,
headers: BTreeMap::new(),
},
@@ -160,6 +160,23 @@ pub(super) fn validate_approval_policy(
Ok(())
}
pub(super) fn validate_execution_policy(
execution_config: &crank_core::ExecutionConfig,
) -> Result<(), ApiError> {
let Some(policy) = execution_config.execution_policy.as_ref() else {
return Ok(());
};
if let Err(error) = policy.validate() {
return Err(ApiError::validation_with_context(
error.to_string(),
json!({
"field": "execution_config.execution_policy.deadline_ms",
}),
));
}
Ok(())
}
#[cfg(test)]
mod tests {
use std::collections::BTreeMap;
@@ -171,8 +188,8 @@ mod tests {
};
use super::{
validate_approval_policy, validate_execution_timeout, validate_idempotency_policy,
validate_response_cache_policy,
validate_approval_policy, validate_execution_policy, validate_execution_timeout,
validate_idempotency_policy, validate_response_cache_policy,
};
fn cacheable_execution_config() -> ExecutionConfig {
@@ -183,6 +200,7 @@ mod tests {
idempotency: None,
safety: None,
approval_policy: None,
execution_policy: None,
auth_profile_ref: None,
headers: BTreeMap::new(),
}
@@ -201,6 +219,7 @@ mod tests {
}),
safety: None,
approval_policy: None,
execution_policy: None,
auth_profile_ref: None,
headers: BTreeMap::new(),
}
@@ -365,4 +384,36 @@ mod tests {
"approval ttl must be between 1 and 300 seconds"
);
}
#[test]
fn accepts_published_execution_policy_and_absent_legacy_policy() {
let mut config = cacheable_execution_config();
validate_execution_policy(&config).unwrap();
config.execution_policy = Some(crank_core::ToolExecutionPolicy {
mode: crank_core::ToolExecutionMode::Adaptive,
deadline_ms: 120_000,
no_tasks_fallback: crank_core::NoTasksFallback::Unsupported,
});
validate_execution_policy(&config).unwrap();
}
#[test]
fn rejects_zero_and_overflowing_execution_policy_deadline() {
let mut config = cacheable_execution_config();
config.execution_policy = Some(crank_core::ToolExecutionPolicy {
mode: crank_core::ToolExecutionMode::Synchronous,
deadline_ms: 0,
no_tasks_fallback: crank_core::NoTasksFallback::Synchronous,
});
let error = validate_execution_policy(&config).unwrap_err();
assert!(matches!(error, crate::error::ApiError::Validation { .. }));
config.execution_policy = Some(crank_core::ToolExecutionPolicy {
mode: crank_core::ToolExecutionMode::Background,
deadline_ms: crank_core::MAX_TOOL_DEADLINE_MS + 1,
no_tasks_fallback: crank_core::NoTasksFallback::Synchronous,
});
assert!(validate_execution_policy(&config).is_err());
}
}
@@ -495,6 +495,7 @@ pub(super) fn test_operation_payload(base_url: &str, name: &str) -> OperationPay
idempotency: None,
safety: None,
approval_policy: None,
execution_policy: None,
auth_profile_ref: None,
headers: BTreeMap::new(),
},
+2 -1
View File
@@ -29,7 +29,7 @@ sha2.workspace = true
sqlx.workspace = true
thiserror.workspace = true
time.workspace = true
tokio = { workspace = true, features = ["sync"] }
tokio = { workspace = true, features = ["sync", "signal"] }
tracing.workspace = true
tracing-subscriber.workspace = true
uuid.workspace = true
@@ -45,5 +45,6 @@ opentelemetry_sdk.workspace = true
prost.workspace = true
reqwest.workspace = true
tower.workspace = true
tokio = { workspace = true, features = ["net", "time"] }
tracing-opentelemetry.workspace = true
metrics-util = "0.20.4"
+193 -11
View File
@@ -1,7 +1,7 @@
use std::{io, process::ExitCode, time::Duration};
use crank_community_mcp::{
auth::CommunityMachineCredentialVerifier, build_app_with_artifacts,
auth::CommunityMachineCredentialVerifier, build_app_with_artifacts_and_tasks,
session::PostgresTransportSessionStore,
};
use crank_config::{
@@ -14,13 +14,14 @@ use crank_observability::{
OtlpTraceConfig, RedactionLimits, SentryConfig, ServiceIdentity, capture_critical_error,
};
use crank_registry::{
MASTER_KEY_CIPHER_CONTRACT, MasterKeyIdentityCandidate, PostgresPoolConfig, PostgresRegistry,
AdmissionLimits, MASTER_KEY_CIPHER_CONTRACT, MasterKeyIdentityCandidate,
PostgresExecutionAdmission, PostgresPoolConfig, PostgresRegistry, PostgresTaskStore,
};
use crank_runtime::{
RequestRateLimitConfig, RequestRateLimiter, RuntimeCacheConfig, RuntimeCacheStores,
RuntimeLimits, SecretCrypto,
};
use mcp_server::pool_metrics::spawn_postgres_pool_metrics;
use mcp_server::pool_metrics::{spawn_admission_metrics, spawn_postgres_pool_metrics};
use sqlx::postgres::PgConnectOptions;
use tokio::net::TcpListener;
use tracing::info;
@@ -175,6 +176,7 @@ async fn run(
};
if metrics_enabled {
spawn_postgres_pool_metrics(registry.pool().clone());
spawn_admission_metrics(registry.pool().clone());
}
let session_store = PostgresTransportSessionStore::from_pool(registry.pool().clone()).await?;
let secret_crypto =
@@ -190,12 +192,55 @@ async fn run(
.with_limits(runtime_limits)
.with_response_cache(cache_stores.response.clone())
.with_coordination_store(cache_stores.coordination.clone())
.with_admission(execution_admission(
registry.pool().clone(),
runtime_limits,
)?)
.build();
// Published static Resources are read from the same immutable content-addressed root the
// admin-api publishes into. The compatibility constructor keeps an unprovisioned/unsafe root
// fail-closed per read instead of preventing startup; nothing is ever written from this process.
let artifact_store = crank_artifacts::ArtifactStore::new(config.storage_root.clone());
let app = build_app_with_artifacts(
// Story 4.2/4.3: the durable Task application entry shares the registry pool, the published
// catalog refresh interval and the exact admission limits of the runtime; the public wire stays
// synchronous/unsupported while a fenced in-process worker executes accepted Tasks.
let task_store = std::sync::Arc::new(PostgresTaskStore::new(
registry.pool().clone(),
admission_limits(runtime_limits),
crank_core::AdmissionSchedulerConfig::default(),
)?);
let tasks = std::sync::Arc::new(crank_community_mcp::TaskApplication::new(
registry.clone(),
Duration::from_millis(config.refresh_ms),
cache_stores.coordination.clone(),
crank_runtime::TaskService::with_secret_crypto(
task_store.clone(),
std::sync::Arc::new(secret_crypto.clone()),
),
));
// Story 4.3: the durable Task worker runs in this process over the same PostgreSQL store,
// published catalog, executor and encrypted-input authority as the accepting application.
let task_handler =
std::sync::Arc::new(crank_community_mcp::TaskWorkerHandler::from_application(
&tasks,
registry.clone(),
runtime.clone(),
secret_crypto.clone(),
));
let task_worker = std::sync::Arc::new(crank_runtime::TaskWorker::new(
task_store,
task_handler,
crank_runtime::TaskWorkerConfig {
owner: format!("mcp-server-{}", std::process::id()),
..crank_runtime::TaskWorkerConfig::default()
},
)?);
let (task_shutdown_tx, task_shutdown_rx) = tokio::sync::watch::channel(false);
let task_worker_handle = tokio::spawn({
let worker = task_worker.clone();
async move { worker.run_until_shutdown(task_shutdown_rx).await }
});
let app = build_app_with_artifacts_and_tasks(
registry,
Duration::from_millis(config.refresh_ms),
config.runtime.base_url.clone(),
@@ -211,6 +256,7 @@ async fn run(
std::sync::Arc::new(CommunityMachineCredentialVerifier),
runtime_limits.max_concurrent_sessions,
artifact_store,
tasks,
);
let listener = TcpListener::bind(config.bind_addr).await?;
@@ -231,15 +277,77 @@ async fn run(
info!(name: "mcp.server.listening", bind_address = %config.bind_addr, "mcp-server listening");
*startup_completed = true;
if let Some(metrics_server) = metrics_server {
tokio::select! {
result = axum::serve(listener, app) => result?,
result = metrics_server.serve() => result?,
let serve_result: Result<(), Box<dyn std::error::Error>> = async {
if let Some(metrics_server) = metrics_server {
tokio::select! {
result = serve_with_worker_shutdown(app, listener, shutdown_signal(), task_shutdown_tx.clone()) => result?,
result = metrics_server.serve() => result?,
}
} else {
serve_with_worker_shutdown(app, listener, shutdown_signal(), task_shutdown_tx.clone())
.await?;
}
} else {
axum::serve(listener, app).await?;
Ok(())
}
.await;
// Stop new claims; active attempts checkpoint/drain or are cancelled inside the bounded worker
// shutdown timeout (<=30s), and queued PostgreSQL work is left for recovery.
task_shutdown_tx.send(true).ok();
if tokio::time::timeout(Duration::from_secs(30), task_worker_handle)
.await
.is_err()
{
tracing::warn!(
name: "mcp.task_worker.shutdown_timeout",
"durable Task worker did not stop within the story shutdown budget"
);
}
serve_result
}
/// Gracefully stop the durable Task worker as soon as shutdown is requested, while still
/// draining in-flight HTTP connections before returning.
async fn serve_with_worker_shutdown<S>(
app: axum::Router,
listener: tokio::net::TcpListener,
signal: S,
worker_stop: tokio::sync::watch::Sender<bool>,
) -> std::io::Result<()>
where
S: std::future::Future<Output = ()> + Send + 'static,
{
let worker_stop_graceful = worker_stop.clone();
let graceful = async move {
signal.await;
let _ = worker_stop_graceful.send(true);
};
let result = axum::serve(listener, app)
.with_graceful_shutdown(graceful)
.await;
let _ = worker_stop.send(true);
result
}
/// Process shutdown trigger for the HTTP surface and the durable Task worker.
async fn shutdown_signal() {
let ctrl_c = async {
let _ = tokio::signal::ctrl_c().await;
};
#[cfg(unix)]
let terminate = async {
if let Ok(mut signal) =
tokio::signal::unix::signal(tokio::signal::unix::SignalKind::terminate())
{
signal.recv().await;
}
};
#[cfg(not(unix))]
let terminate = std::future::pending::<()>();
tokio::select! {
_ = ctrl_c => {}
_ = terminate => {}
}
Ok(())
}
fn init_observability(
@@ -287,6 +395,25 @@ fn init_observability(
)?)
}
fn admission_limits(runtime_limits: RuntimeLimits) -> AdmissionLimits {
AdmissionLimits {
global_limit: runtime_limits.max_concurrent_unary,
..AdmissionLimits::default()
}
}
fn execution_admission(
pool: sqlx::PgPool,
runtime_limits: RuntimeLimits,
) -> Result<std::sync::Arc<dyn crank_core::ExecutionAdmission>, Box<dyn std::error::Error>> {
let store = PostgresExecutionAdmission::new(
pool,
admission_limits(runtime_limits),
crank_core::AdmissionSchedulerConfig::default(),
)?;
Ok(std::sync::Arc::new(store))
}
fn preflight_config(config: &McpProcessConfig) -> Result<(), crank_config::ConfigError> {
let invalid = |field| crank_config::ConfigError::single(DiagnosticCode::InvalidType, field);
database_options(&config.database).map_err(|_| invalid("database.source"))?;
@@ -307,6 +434,12 @@ fn preflight_config(config: &McpProcessConfig) -> Result<(), crank_config::Confi
config.runtime.max_concurrent_sessions,
)
.map_err(|_| invalid("runtime.limits"))?;
crank_registry::AdmissionLimits {
global_limit: config.runtime.max_concurrent_unary,
..crank_registry::AdmissionLimits::default()
}
.validate()
.map_err(|_| invalid("runtime.admission"))?;
runtime_cache_config(config).map_err(|_| invalid("cache"))?;
RequestRateLimitConfig::new(
config.rate_limit.requests_per_second,
@@ -460,3 +593,52 @@ fn runtime_cache_config(
.map(|value| value.expose_secret().to_owned()),
)
}
#[cfg(test)]
mod tests {
use super::*;
use axum::{Router, routing::get};
use std::time::Duration;
use tokio::sync::{oneshot, watch};
async fn slow_handler() -> &'static str {
tokio::time::sleep(Duration::from_millis(600)).await;
"ok"
}
#[tokio::test]
async fn worker_stops_before_http_drain_completes() {
let app = Router::new().route("/slow", get(slow_handler));
let listener = TcpListener::bind("127.0.0.1:0").await.unwrap();
let addr = listener.local_addr().unwrap();
let (stop_tx, mut stop_rx) = watch::channel(false);
let (signal_tx, signal_rx) = oneshot::channel::<()>();
let handle = tokio::spawn(serve_with_worker_shutdown(
app,
listener,
async {
let _ = signal_rx.await;
},
stop_tx,
));
let client = reqwest::Client::new();
let request = client.get(format!("http://{}/slow", addr));
let in_flight = tokio::spawn(async move { request.send().await });
tokio::time::sleep(Duration::from_millis(150)).await;
let _ = signal_tx.send(());
tokio::time::timeout(Duration::from_millis(250), stop_rx.changed())
.await
.expect("worker stop signal should arrive before HTTP drain")
.expect("watch changed");
assert!(*stop_rx.borrow());
let response = in_flight.await.unwrap().unwrap();
assert!(response.status().is_success());
let serve_result = handle.await.unwrap();
assert!(serve_result.is_ok());
}
}
+14
View File
@@ -11,3 +11,17 @@ pub fn spawn_postgres_pool_metrics(pool: PgPool) -> tokio::task::JoinHandle<()>
}
})
}
/// Story 4.1: samples the authoritative admission queue so the oldest-waiting age grows (and returns
/// to zero) without waiting for a new request.
pub fn spawn_admission_metrics(pool: PgPool) -> tokio::task::JoinHandle<()> {
tokio::spawn(async move {
let mut interval = tokio::time::interval(Duration::from_secs(5));
loop {
interval.tick().await;
if let Ok(snapshot) = crank_registry::admission_snapshot(&pool).await {
crank_observability::record_admission_oldest_waiting(snapshot.oldest_waiting_ms);
}
}
})
}
+874
View File
@@ -0,0 +1,874 @@
//! ATDD Story 4.1 (FR-21/FR-24): the runtime asks one authoritative store for admission.
//!
//! Real PostgreSQL state, two independent `RuntimeExecutor` instances (the multi-process shape) and a
//! counted adapter: one shared global budget, typed overload with retry guidance, typed unsupported
//! fallback, deadline that includes queue waiting and zero outbound calls on every refusal.
//!
//! Lives in the `mcp-server` app crate because the runtime must not depend on the registry crate; this
//! app already wires both and this is the real deployment shape.
use std::{
collections::BTreeMap,
sync::{
Arc, Mutex,
atomic::{AtomicUsize, Ordering},
},
time::{Duration, Instant},
};
use async_trait::async_trait;
use crank_core::{
AdapterResponse, AdmissionMode, AdmissionRequest, AdmissionScope, ApprovalEvidence,
ConfirmationPolicy, ExecutionAdmission, ExecutionConfig, ExecutionErrorCode, ExecutionMode,
ExecutionOrigin, ExecutionStage, HttpMethod, InvocationSource, NoTasksFallback, Operation,
OperationId, OperationSafetyClass, OperationSafetyPolicy, OperationSecurityLevel,
OperationStatus, OutcomeCertainty, Protocol, ProtocolAdapter, RestTarget,
RuntimeRequestContext, Target, ToolDescription, ToolExecutionMode, ToolExecutionPolicy,
WorkspaceId,
};
use crank_mapping::MappingSet;
use crank_registry::{AdmissionLimits, PostgresExecutionAdmission};
use crank_runtime::{RuntimeExecutor, RuntimeExecutorBuilder, RuntimeLimits};
use crank_schema::{Schema, SchemaKind};
use serde_json::json;
use sqlx::PgPool;
use time::OffsetDateTime;
struct CountingAdapter {
calls: Arc<AtomicUsize>,
current: Arc<AtomicUsize>,
peak: Arc<AtomicUsize>,
}
#[async_trait]
impl ProtocolAdapter for CountingAdapter {
fn protocol(&self) -> Protocol {
Protocol::Rest
}
fn supports_mode(&self, mode: ExecutionMode) -> bool {
mode == ExecutionMode::Unary
}
async fn invoke_unary(
&self,
_target: &Target,
_prepared: &crank_core::PreparedRequest,
context: &RuntimeRequestContext,
) -> Result<AdapterResponse, crank_core::ProtocolAdapterError> {
context.mark_dispatch_started();
self.calls.fetch_add(1, Ordering::SeqCst);
let current = self.current.fetch_add(1, Ordering::SeqCst) + 1;
self.peak.fetch_max(current, Ordering::SeqCst);
tokio::time::sleep(Duration::from_millis(150)).await;
self.current.fetch_sub(1, Ordering::SeqCst);
Ok(AdapterResponse {
status_code: 200,
headers: BTreeMap::new(),
body: json!({"ok": true}),
data: json!({"ok": true}),
})
}
}
fn schema(kind: SchemaKind) -> Schema {
Schema {
kind,
format: None,
description: None,
required: false,
nullable: false,
default_value: None,
fields: BTreeMap::new(),
items: None,
enum_values: Vec::new(),
variants: Vec::new(),
}
}
fn operation(policy: Option<ToolExecutionPolicy>) -> Operation<Schema, MappingSet> {
Operation {
id: OperationId::new("op_admission_budget"),
name: "admission_budget".to_owned(),
display_name: "Admission budget".to_owned(),
category: "general".to_owned(),
protocol: Protocol::Rest,
security_level: OperationSecurityLevel::Standard,
status: OperationStatus::Published,
version: 7,
target: Target::Rest(RestTarget {
base_url: "https://api.example.com".to_owned(),
method: HttpMethod::Get,
path_template: "/v1/items".to_owned(),
static_headers: BTreeMap::new(),
request_contract: None,
response_contract: Vec::new(),
multipart_field_mime_types: BTreeMap::new(),
}),
input_schema: schema(SchemaKind::Object),
output_schema: schema(SchemaKind::Object),
input_mapping: MappingSet::default(),
output_mapping: MappingSet::default(),
execution_config: ExecutionConfig {
timeout_ms: 5_000,
execution_policy: policy,
..ExecutionConfig::default()
},
tool_description: ToolDescription {
title: "Admission".to_owned(),
description: "Admission budget probe".to_owned(),
tags: Vec::new(),
examples: Vec::new(),
},
samples: None,
generated_draft: None,
config_export: None,
wizard_state: None,
created_at: OffsetDateTime::now_utc(),
updated_at: OffsetDateTime::now_utc(),
published_at: None,
}
}
async fn execute(
executor: &RuntimeExecutor,
operation: &crank_runtime::RuntimeOperation,
input: &serde_json::Value,
) -> Result<serde_json::Value, crank_core::ExecutionFailure> {
let workspace = WorkspaceId::new("ws_test");
let context = crank_runtime::RuntimeRequestContext::from_correlation(
&crank_core::CorrelationContext::generate(),
)
.with_metering_context(workspace.clone(), None, InvocationSource::AdminTestRun);
execute_in(executor, operation, input, &context).await
}
async fn execute_in(
executor: &RuntimeExecutor,
operation: &crank_runtime::RuntimeOperation,
input: &serde_json::Value,
context: &crank_runtime::RuntimeRequestContext,
) -> Result<serde_json::Value, crank_core::ExecutionFailure> {
let workspace = context
.metering_context()
.map(|metering| metering.workspace_id.clone())
.unwrap_or_else(|| WorkspaceId::new("ws_test"));
let request = crank_runtime::RuntimeExecutionRequest::try_new(
&workspace,
ExecutionOrigin::AdminDraft,
None,
operation,
input,
crank_runtime::ExecutionAuthorization::Authorized,
None,
context,
Instant::now() + Duration::from_secs(30),
)
.map_err(|_| {
crank_core::ExecutionFailure::new(
ExecutionErrorCode::RuntimeInternal,
crank_core::CorrelationContext::new(
context.request_id.clone(),
context.trace_context.clone(),
),
)
})?;
executor
.execute_outcome(request)
.await
.map(|success| success.output)
}
async fn schema_pool(schema_url: &str) -> PgPool {
let pool = PgPool::connect(schema_url)
.await
.expect("admission schema pool");
crank_registry::MigrationAuthority::apply(&pool)
.await
.expect("migration authority applies the admission schema");
pool
}
fn admission_store(pool: PgPool, global: usize, queue: usize) -> Arc<PostgresExecutionAdmission> {
Arc::new(
PostgresExecutionAdmission::new(
pool,
AdmissionLimits {
global_limit: global,
agent_limit: 8,
tool_limit: 8,
queue_limit: queue,
},
crank_core::AdmissionSchedulerConfig::default(),
)
.expect("valid admission limits"),
)
}
fn executor(
store: Arc<PostgresExecutionAdmission>,
calls: Arc<AtomicUsize>,
current: Arc<AtomicUsize>,
peak: Arc<AtomicUsize>,
) -> RuntimeExecutor {
RuntimeExecutorBuilder::new()
.register_adapter(Arc::new(CountingAdapter {
calls,
current,
peak,
}))
.with_limits(RuntimeLimits::try_new(8, 2).expect("local limits"))
.with_admission(store)
.build()
}
#[tokio::test]
async fn two_executors_share_one_global_budget() {
let schema_url = crank_test_support::postgres_schema_url("admission_budget").await;
let pool = schema_pool(&schema_url).await;
let store = admission_store(pool.clone(), 2, 8);
let calls = Arc::new(AtomicUsize::new(0));
let current = Arc::new(AtomicUsize::new(0));
let peak = Arc::new(AtomicUsize::new(0));
let first = executor(
Arc::clone(&store),
Arc::clone(&calls),
Arc::clone(&current),
Arc::clone(&peak),
);
let second = executor(
Arc::clone(&store),
Arc::clone(&calls),
Arc::clone(&current),
Arc::clone(&peak),
);
let operation: crank_runtime::RuntimeOperation = operation(None).into();
let mut tasks = Vec::new();
for index in 0..4 {
let executor = if index % 2 == 0 {
first.clone()
} else {
second.clone()
};
let operation = operation.clone();
tasks.push(tokio::spawn(async move {
execute(&executor, &operation, &json!({})).await
}));
}
let mut completed = 0;
for task in tasks {
match task.await.expect("execution task") {
Ok(_) => completed += 1,
Err(failure) => panic!("admission must grant every request: {failure:?}"),
}
}
assert_eq!(completed, 4);
assert_eq!(calls.load(Ordering::SeqCst), 4);
assert!(
peak.load(Ordering::SeqCst) <= 2,
"peak concurrency {} exceeded the shared global limit",
peak.load(Ordering::SeqCst)
);
assert_eq!(store.snapshot().await.unwrap().executing, 0);
}
#[tokio::test]
async fn saturation_returns_typed_overload_without_an_outbound_call() {
let schema_url = crank_test_support::postgres_schema_url("admission_overload").await;
let pool = schema_pool(&schema_url).await;
let store = admission_store(pool, 1, 1);
let calls = Arc::new(AtomicUsize::new(0));
let current = Arc::new(AtomicUsize::new(0));
let peak = Arc::new(AtomicUsize::new(0));
let runtime = executor(
Arc::clone(&store),
Arc::clone(&calls),
Arc::clone(&current),
Arc::clone(&peak),
);
// Another process already holds the only global permit.
let held = match store
.admit(AdmissionRequest {
scope: AdmissionScope {
workspace_id: "ws_test".to_owned(),
agent_key: "admin".to_owned(),
operation_id: "op_admission_budget".to_owned(),
operation_version: 7,
},
mode: AdmissionMode::Synchronous,
policy_mode: ToolExecutionMode::Synchronous,
no_tasks_fallback: NoTasksFallback::Synchronous,
approval: ApprovalEvidence::NotRequired,
fingerprint: "fp_holder".to_owned(),
expires_at_unix_ms: crank_core::current_unix_ms() + 10_000,
})
.await
.expect("admission store is reachable")
{
crank_core::AdmissionOutcome::Permitted(permit) => permit,
other => panic!("expected a permit, got {other:?}"),
};
let waiting_operation: crank_runtime::RuntimeOperation = operation(Some(ToolExecutionPolicy {
mode: ToolExecutionMode::Synchronous,
deadline_ms: 250,
no_tasks_fallback: NoTasksFallback::Synchronous,
}))
.into();
let waiting_runtime = runtime.clone();
let first_waiting_operation = waiting_operation.clone();
let waiting = tokio::spawn(async move {
execute(&waiting_runtime, &first_waiting_operation, &json!({})).await
});
// Let the first request occupy the bounded queue before the next one arrives.
while store.snapshot().await.unwrap().waiting == 0 {
tokio::time::sleep(Duration::from_millis(5)).await;
}
let overloaded = execute(&runtime, &waiting_operation, &json!({})).await;
let failure = overloaded.expect_err("a full queue must refuse the request");
assert_eq!(
failure.error_code(),
ExecutionErrorCode::ExecutionOverloaded
);
assert_eq!(failure.stage(), ExecutionStage::Admission);
assert_eq!(failure.retry_after_ms(), Some(250));
// The queued request hits its absolute deadline instead of dispatching. That refusal happened
// inside admission (nothing was ever dispatched), so the client must see an admission-stage
// overload with retry guidance — not an `upstream_timeout`.
let expired = waiting
.await
.expect("waiting task")
.expect_err("deadline must refuse");
assert_eq!(
expired.error_code(),
ExecutionErrorCode::ExecutionOverloaded,
"queue waiting counts against the published deadline"
);
assert_eq!(expired.stage(), ExecutionStage::Admission);
assert_eq!(expired.retry_after_ms(), Some(250));
assert_eq!(expired.outcome_certainty(), OutcomeCertainty::Certain);
assert_eq!(
calls.load(Ordering::SeqCst),
0,
"no outbound call on refusal"
);
assert!(store.release(&held).await.unwrap());
}
#[tokio::test]
async fn background_policy_without_a_task_transport_is_typed_unsupported() {
let schema_url = crank_test_support::postgres_schema_url("admission_unsupported").await;
let pool = schema_pool(&schema_url).await;
let store = admission_store(pool, 2, 4);
let calls = Arc::new(AtomicUsize::new(0));
let current = Arc::new(AtomicUsize::new(0));
let peak = Arc::new(AtomicUsize::new(0));
let runtime = executor(
Arc::clone(&store),
Arc::clone(&calls),
Arc::clone(&current),
Arc::clone(&peak),
);
let operation: crank_runtime::RuntimeOperation = operation(Some(ToolExecutionPolicy {
mode: ToolExecutionMode::Background,
deadline_ms: 2_000,
no_tasks_fallback: NoTasksFallback::Unsupported,
}))
.into();
let failure = execute(&runtime, &operation, &json!({}))
.await
.expect_err("no Task transport exists in Story 4.1");
assert_eq!(
failure.error_code(),
ExecutionErrorCode::ExecutionModeUnsupported
);
assert_eq!(calls.load(Ordering::SeqCst), 0);
let snapshot = store.snapshot().await.unwrap();
assert_eq!(snapshot.waiting, 0);
assert_eq!(snapshot.executing, 0);
}
#[tokio::test]
async fn background_policy_with_a_synchronous_fallback_keeps_the_sync_response_shape() {
let schema_url = crank_test_support::postgres_schema_url("admission_fallback").await;
let pool = schema_pool(&schema_url).await;
let store = admission_store(pool, 2, 4);
let calls = Arc::new(AtomicUsize::new(0));
let current = Arc::new(AtomicUsize::new(0));
let peak = Arc::new(AtomicUsize::new(0));
let runtime = executor(
Arc::clone(&store),
Arc::clone(&calls),
Arc::clone(&current),
Arc::clone(&peak),
);
let operation: crank_runtime::RuntimeOperation = operation(Some(ToolExecutionPolicy {
mode: ToolExecutionMode::Adaptive,
deadline_ms: 2_000,
no_tasks_fallback: NoTasksFallback::Synchronous,
}))
.into();
let output = execute(&runtime, &operation, &json!({}))
.await
.expect("adaptive resolves through the published synchronous fallback");
// The response shape is unchanged by admission: an operation without output mapping still
// returns the same mapped output object it returned before Story 4.1.
assert_eq!(output, json!({}));
assert_eq!(calls.load(Ordering::SeqCst), 1);
assert_eq!(store.snapshot().await.unwrap().executing, 0);
}
#[tokio::test]
async fn admission_store_outage_fails_closed_before_dispatch() {
let schema_url = crank_test_support::postgres_schema_url("admission_outage").await;
let pool = schema_pool(&schema_url).await;
let store = admission_store(pool.clone(), 2, 4);
let calls = Arc::new(AtomicUsize::new(0));
let current = Arc::new(AtomicUsize::new(0));
let peak = Arc::new(AtomicUsize::new(0));
let runtime = executor(
Arc::clone(&store),
Arc::clone(&calls),
Arc::clone(&current),
Arc::clone(&peak),
);
// The mandatory state disappears after the store was built: every execution must fail closed.
sqlx::query("drop table execution_admission_entries")
.execute(&pool)
.await
.expect("drop the mandatory admission state");
let operation: crank_runtime::RuntimeOperation = operation(None).into();
let failure = execute(&runtime, &operation, &json!({}))
.await
.expect_err("a missing mandatory store must fail closed");
assert_eq!(
failure.error_code(),
ExecutionErrorCode::SafetyStoreUnavailable
);
assert_eq!(calls.load(Ordering::SeqCst), 0);
assert!(store.health().await.is_err());
}
/// Admission store that records the snapshot it was asked to decide on (Story 4.1, AC2/F2).
struct RecordingAdmission {
inner: Arc<PostgresExecutionAdmission>,
requests: Mutex<Vec<AdmissionRequest>>,
permits: AtomicUsize,
releases: AtomicUsize,
}
impl RecordingAdmission {
fn new(inner: Arc<PostgresExecutionAdmission>) -> Self {
Self {
inner,
requests: Mutex::new(Vec::new()),
permits: AtomicUsize::new(0),
releases: AtomicUsize::new(0),
}
}
fn requests(&self) -> Vec<AdmissionRequest> {
self.requests.lock().expect("admission log").clone()
}
fn permits(&self) -> usize {
self.permits.load(Ordering::SeqCst)
}
fn releases(&self) -> usize {
self.releases.load(Ordering::SeqCst)
}
}
#[async_trait]
impl ExecutionAdmission for RecordingAdmission {
async fn health(&self) -> Result<(), crank_core::AdmissionStoreError> {
self.inner.health().await
}
async fn admit(
&self,
request: AdmissionRequest,
) -> Result<crank_core::AdmissionOutcome, crank_core::AdmissionStoreError> {
self.requests
.lock()
.expect("admission log")
.push(request.clone());
let outcome = self.inner.admit(request).await?;
if matches!(outcome, crank_core::AdmissionOutcome::Permitted(_)) {
self.permits.fetch_add(1, Ordering::SeqCst);
}
Ok(outcome)
}
async fn release(
&self,
permit: &crank_core::AdmissionPermit,
) -> Result<bool, crank_core::AdmissionStoreError> {
let released = self.inner.release(permit).await?;
if released {
self.releases.fetch_add(1, Ordering::SeqCst);
}
Ok(released)
}
async fn cancel(
&self,
ticket_id: u64,
owner_token: &str,
) -> Result<bool, crank_core::AdmissionStoreError> {
self.inner.cancel(ticket_id, owner_token).await
}
async fn snapshot(
&self,
) -> Result<crank_core::AdmissionQueueSnapshot, crank_core::AdmissionStoreError> {
self.inner.snapshot().await
}
}
fn confirmation_operation(version: u32) -> crank_runtime::RuntimeOperation {
let mut source = operation(None);
source.id = OperationId::new("op_admission_confirmation");
source.name = "admission_confirmation".to_owned();
source.version = version;
let Target::Rest(target) = &mut source.target;
target.method = HttpMethod::Delete;
source.execution_config.safety = Some(OperationSafetyPolicy {
class: OperationSafetyClass::Destructive,
confirmation: Some(ConfirmationPolicy { ttl_ms: 60_000 }),
});
source.into()
}
fn confirmation_runtime(
admission: Arc<RecordingAdmission>,
coordination: Option<Arc<dyn crank_core::CoordinationStateStore>>,
calls: Arc<AtomicUsize>,
) -> RuntimeExecutor {
let store: Arc<dyn ExecutionAdmission> = admission;
let builder = RuntimeExecutorBuilder::new()
.register_adapter(Arc::new(CountingAdapter {
calls,
current: Arc::new(AtomicUsize::new(0)),
peak: Arc::new(AtomicUsize::new(0)),
}))
.with_limits(RuntimeLimits::try_new(8, 2).expect("local limits"))
.with_admission(store);
match coordination {
Some(coordination) => builder.with_coordination_store(coordination).build(),
None => builder.build(),
}
}
fn confirmation_context() -> crank_runtime::RuntimeRequestContext {
crank_runtime::RuntimeRequestContext::from_correlation(
&crank_core::CorrelationContext::generate(),
)
.with_response_cache_scope("ws_test", "agent_a")
.with_metering_context(
WorkspaceId::new("ws_test"),
None,
InvocationSource::AdminTestRun,
)
}
#[tokio::test]
async fn confirmation_requiring_call_takes_a_permit_only_with_verified_approval() {
let schema_url = crank_test_support::postgres_schema_url("admission_confirmation").await;
let pool = schema_pool(&schema_url).await;
let recording = Arc::new(RecordingAdmission::new(Arc::new(
PostgresExecutionAdmission::new(
pool,
AdmissionLimits {
global_limit: 4,
agent_limit: 4,
tool_limit: 4,
queue_limit: 8,
},
crank_core::AdmissionSchedulerConfig::default(),
)
.expect("valid admission limits"),
)));
let coordination: Arc<dyn crank_core::CoordinationStateStore> =
Arc::new(crank_runtime::InMemoryCoordinationStateStore::default());
let calls = Arc::new(AtomicUsize::new(0));
let runtime = confirmation_runtime(
Arc::clone(&recording),
Some(Arc::clone(&coordination)),
Arc::clone(&calls),
);
let operation = confirmation_operation(3);
let context = confirmation_context();
let input = json!({ "reason": "remove the stale item" });
// 1. No approval yet: the existing challenge is returned and no execution capacity is taken.
let challenge = execute_in(&runtime, &operation, &input, &context)
.await
.expect_err("an unapproved destructive call must only stage confirmation");
assert_eq!(
challenge.error_code(),
ExecutionErrorCode::ConfirmationRequired
);
let token = challenge
.confirmation()
.expect("expected the confirmation challenge")
.token()
.to_owned();
assert_eq!(
recording.permits(),
0,
"an unapproved request must not take an execution permit"
);
assert_eq!(calls.load(Ordering::SeqCst), 0);
let requests = recording.requests();
assert_eq!(
requests.len(),
1,
"the one admission snapshot is consulted once"
);
assert_eq!(
requests[0].approval,
ApprovalEvidence::NotGranted,
"admission must see the real approval state, not a fixed NotRequired"
);
assert_eq!(requests[0].scope.operation_version, 3);
let snapshot = recording.snapshot().await.expect("admission snapshot");
assert_eq!((snapshot.executing, snapshot.waiting), (0, 0));
// 2. An approval issued for another published version never opens this execution.
let other_version = confirmation_operation(4);
let foreign = execute_in(
&runtime,
&other_version,
&input,
&context.clone().with_confirmation_token(token.clone()),
)
.await
.expect_err("a foreign approval must not authorise this version");
assert_eq!(
foreign.error_code(),
ExecutionErrorCode::ConfirmationInvalid
);
assert_eq!(recording.permits(), 0);
assert_eq!(
recording.requests().len(),
1,
"an unusable approval is refused before any capacity request"
);
// 3. An approval is bound to its exact input as well.
let mismatched_input = json!({ "reason": "remove another item" });
let mismatched = execute_in(
&runtime,
&operation,
&mismatched_input,
&context.clone().with_confirmation_token(token.clone()),
)
.await
.expect_err("a confirmation token must not authorise different input");
assert_eq!(
mismatched.error_code(),
ExecutionErrorCode::ConfirmationInvalid
);
assert_eq!(recording.permits(), 0);
assert_eq!(recording.requests().len(), 1);
// 4. The matching approval passes admission with truthful evidence and executes exactly once.
let executed = execute_in(
&runtime,
&operation,
&input,
&context.clone().with_confirmation_token(token.clone()),
)
.await
.expect("a verified approval executes");
assert!(executed.is_object());
assert_eq!(calls.load(Ordering::SeqCst), 1);
let requests = recording.requests();
assert_eq!(requests.len(), 2);
assert_eq!(requests[1].approval, ApprovalEvidence::Granted);
assert_eq!(requests[1].scope.operation_version, 3);
assert_eq!(recording.permits(), 1);
assert_eq!(
recording.releases(),
1,
"the permit is released after execution"
);
assert_eq!(recording.snapshot().await.expect("snapshot").executing, 0);
// 5. Replay of the consumed confirmation is refused before capacity and without a side effect.
let replay = execute_in(
&runtime,
&operation,
&input,
&context.clone().with_confirmation_token(token),
)
.await
.expect_err("a confirmation token is single-use");
assert_eq!(replay.error_code(), ExecutionErrorCode::ConfirmationInvalid);
assert_eq!(calls.load(Ordering::SeqCst), 1);
assert_eq!(recording.permits(), 1);
assert_eq!(
recording.requests().len(),
2,
"a replayed approval never reaches admission"
);
// 6. The trusted approved-recovery path still passes the same gate with real evidence.
let approved = execute_in(
&runtime,
&operation,
&json!({ "reason": "approved by an operator" }),
&context.with_approval_granted(),
)
.await;
assert!(approved.is_ok());
assert_eq!(calls.load(Ordering::SeqCst), 2);
let requests = recording.requests();
assert_eq!(requests.len(), 3);
assert_eq!(requests[2].approval, ApprovalEvidence::Granted);
assert_eq!(recording.permits(), 2);
assert_eq!(recording.snapshot().await.expect("snapshot").executing, 0);
}
#[tokio::test]
async fn mandatory_safety_store_outage_fails_closed_before_admission() {
let schema_url = crank_test_support::postgres_schema_url("admission_confirmation_outage").await;
let pool = schema_pool(&schema_url).await;
let recording = Arc::new(RecordingAdmission::new(Arc::new(
PostgresExecutionAdmission::new(
pool,
AdmissionLimits {
global_limit: 4,
agent_limit: 4,
tool_limit: 4,
queue_limit: 8,
},
crank_core::AdmissionSchedulerConfig::default(),
)
.expect("valid admission limits"),
)));
let calls = Arc::new(AtomicUsize::new(0));
// Confirmation is mandatory for this Tool, but no safety store is wired: the request must fail
// closed instead of reserving capacity and hoping for an in-memory fallback.
let runtime = confirmation_runtime(Arc::clone(&recording), None, Arc::clone(&calls));
let operation = confirmation_operation(3);
let failure = execute_in(
&runtime,
&operation,
&json!({ "reason": "no store" }),
&confirmation_context(),
)
.await
.expect_err("a mandatory safety store outage must fail closed");
assert_eq!(
failure.error_code(),
ExecutionErrorCode::SafetyStoreUnavailable
);
assert_eq!(calls.load(Ordering::SeqCst), 0);
assert_eq!(recording.permits(), 0);
let after_missing_store = recording.requests().len();
// The same Tool with a claimed approval and a *failing* safety store never reaches the capacity
// store at all: the approval claim stays unverified, so no permit and no outbound call exist.
let failing = confirmation_runtime(
Arc::clone(&recording),
Some(Arc::new(UnavailableSafetyStore)),
Arc::clone(&calls),
);
let outage = execute_in(
&failing,
&operation,
&json!({ "reason": "failing store" }),
&confirmation_context().with_confirmation_token("ct_unavailable"),
)
.await
.expect_err("an unreachable safety store must refuse the claimed approval");
assert_eq!(
outage.error_code(),
ExecutionErrorCode::SafetyStoreUnavailable
);
assert_eq!(
recording.requests().len(),
after_missing_store,
"an unverified approval never becomes a capacity request"
);
assert_eq!(recording.permits(), 0);
assert_eq!(calls.load(Ordering::SeqCst), 0);
}
/// Coordination store whose mandatory backend is down: every operation fails closed.
struct UnavailableSafetyStore;
impl UnavailableSafetyStore {
fn error() -> crank_core::CacheStoreError {
crank_core::CacheStoreError::Unavailable {
message: "test backend unavailable".to_owned(),
}
}
}
#[async_trait]
impl crank_core::CoordinationStateStore for UnavailableSafetyStore {
async fn get_value(
&self,
_scope: crank_core::CacheScope,
_key: &str,
) -> Result<Option<crank_core::CoordinationStateValue>, crank_core::CacheStoreError> {
Err(Self::error())
}
async fn put_value(
&self,
_scope: crank_core::CacheScope,
_key: &str,
_value: crank_core::CoordinationStateValue,
_ttl: Duration,
) -> Result<(), crank_core::CacheStoreError> {
Err(Self::error())
}
async fn delete_value(
&self,
_scope: crank_core::CacheScope,
_key: &str,
) -> Result<(), crank_core::CacheStoreError> {
Err(Self::error())
}
async fn take_value(
&self,
_scope: crank_core::CacheScope,
_key: &str,
) -> Result<Option<crank_core::CoordinationStateValue>, crank_core::CacheStoreError> {
Err(Self::error())
}
async fn reserve_value(
&self,
_scope: crank_core::CacheScope,
_key: &str,
_value: crank_core::CoordinationStateValue,
_ttl: Duration,
) -> Result<crank_core::CoordinationStateReservation, crank_core::CacheStoreError> {
Err(Self::error())
}
async fn compare_and_set_value(
&self,
_scope: crank_core::CacheScope,
_key: &str,
_expected: &crank_core::CoordinationStateValue,
_value: crank_core::CoordinationStateValue,
_ttl: Duration,
) -> Result<bool, crank_core::CacheStoreError> {
Err(Self::error())
}
}
+4
View File
@@ -1,16 +1,20 @@
mod integration {
mod catalog_access;
mod common;
mod durable_tasks;
mod era_isolation;
mod execution_certainty;
mod execution_stages;
mod inventory_projection;
mod jsonrpc_correlation;
mod mcp_completion;
mod mcp_logging;
mod mcp_prompts;
mod mcp_signals;
mod request_context;
mod revision_matrix;
mod static_resources;
mod tool_search;
mod tools_lifecycle;
mod transport_protocol;
}
@@ -60,6 +60,31 @@ impl SharedLogWriter {
}
}
/// Story 3.4: a capturing writer plus the subscriber that records into it, so a test can prove that a
/// request-scoped progress token never reaches the log projection. Install it with
/// `tracing::subscriber::set_default` around the request on the `current_thread` runtime.
pub(super) fn capturing_subscriber() -> (
CapturedLogs,
impl tracing::Subscriber + Send + Sync + 'static,
) {
let writer = SharedLogWriter::default();
let subscriber = tracing_subscriber::fmt()
.with_writer(writer.clone())
.with_max_level(LevelFilter::TRACE)
.with_ansi(false)
.finish();
(CapturedLogs(writer), subscriber)
}
/// The captured log text of one scoped subscriber.
pub(super) struct CapturedLogs(SharedLogWriter);
impl CapturedLogs {
pub(super) fn output(&self) -> String {
self.0.output()
}
}
impl<'a> MakeWriter<'a> for SharedLogWriter {
type Writer = SharedLogGuard;
@@ -596,6 +621,7 @@ pub(super) fn test_operation(base_url: &str, name: &str) -> Operation<Schema, Ma
idempotency: None,
safety: None,
approval_policy: None,
execution_policy: None,
auth_profile_ref: None,
headers: BTreeMap::new(),
},
@@ -0,0 +1,413 @@
//! Story 4.3 ATDD: durable Attempt ID reaches the adapter attempt span (PF-1).
//!
//! End-to-end: a real durable Task worker dispatches a retrying GET through a real local
//! upstream, and the `adapter_attempt` spans carry the same durable Attempt ID that is
//! persisted in `tasks.current_attempt_id` on both attempts.
use super::super::common;
use axum::Router;
use axum::http::StatusCode;
use axum::routing::get;
use crank_community_mcp::TaskWorkerHandler;
use crank_core::{
AgentOperationBinding, CorrelationContext, RetryPolicy, Target, ToolExecutionPolicy,
};
use crank_runtime::{
OutboundHttpPolicy, SecretCrypto, TaskWorker, TaskWorkerConfig, community_with_outbound_policy,
};
use std::sync::atomic::{AtomicUsize, Ordering};
use std::sync::{Arc, Mutex};
use time::OffsetDateTime;
use tracing::{Id, Subscriber, field::Visit};
use tracing_subscriber::{Layer, layer::SubscriberExt, registry::LookupSpan};
const AGENT_SLUG: &str = "reports";
const TOOL_NAME: &str = "attempt_trace";
const KEY: &str = "attempt-trace-key-1";
async fn publish(
registry: &crank_registry::PostgresRegistry,
operation_id: &crank_core::OperationId,
version: u32,
) {
registry
.publish_operation(crank_registry::PublishRequest {
workspace_id: &common::test_workspace_id(),
operation_id,
version,
published_at: &OffsetDateTime::parse(
"2026-03-26T10:00:00Z",
&time::format_description::well_known::Rfc3339,
)
.unwrap(),
published_by: Some("alice"),
})
.await
.unwrap();
}
async fn application_with_store(
registry: &crank_registry::PostgresRegistry,
) -> (
crank_community_mcp::TaskApplication,
std::sync::Arc<crank_registry::PostgresTaskStore>,
SecretCrypto,
) {
let crypto = SecretCrypto::new("test-master-key-00000000000000000000000000000000").unwrap();
let observed_at = OffsetDateTime::now_utc();
registry
.verify_or_register_master_key_identity(crank_registry::MasterKeyIdentityCandidate {
epoch: crypto.master_key_epoch(),
fingerprint: crypto.master_key_fingerprint(),
cipher_contract: crank_registry::MASTER_KEY_CIPHER_CONTRACT,
observed_at: &observed_at,
})
.await
.expect("test master-key identity registered");
let store = std::sync::Arc::new(
crank_registry::PostgresTaskStore::new(
registry.pool().clone(),
crank_registry::AdmissionLimits {
global_limit: 4,
agent_limit: 4,
tool_limit: 4,
queue_limit: 8,
},
Default::default(),
)
.expect("valid task store configuration"),
);
let tasks = crank_community_mcp::TaskApplication::new(
registry.clone(),
std::time::Duration::ZERO,
std::sync::Arc::new(crank_runtime::InMemoryCoordinationStateStore::default()),
crank_runtime::TaskService::with_secret_crypto(
store.clone(),
std::sync::Arc::new(crypto.clone()),
),
);
(tasks, store, crypto)
}
async fn principal(
registry: &crank_registry::PostgresRegistry,
name: &str,
scopes: &[crank_core::PlatformApiKeyScope],
) -> crank_community_mcp::TaskPrincipal {
let token = common::create_platform_api_key(registry, AGENT_SLUG, name, scopes).await;
let secret_hash = common::hash_access_secret(&token);
let api_key = registry
.get_platform_api_key_by_secret_for_agent_slug(
common::test_workspace_slug(),
AGENT_SLUG,
&secret_hash,
)
.await
.unwrap()
.expect("an active platform key resolves");
registry
.touch_platform_api_key(
&api_key.api_key.workspace_id,
&api_key.api_key.id,
&OffsetDateTime::now_utc(),
)
.await
.unwrap();
crank_community_mcp::TaskPrincipal::Agent {
workspace_slug: common::test_workspace_slug().to_owned(),
agent_slug: AGENT_SLUG.to_owned(),
credential: crank_core::VerifiedMachineCredential {
machine_access_mode: crank_core::MachineAccessMode::StaticAgentKey,
max_security_level: crank_core::OperationSecurityLevel::Standard,
scopes: api_key.api_key.scopes.clone(),
platform_api_key_id: Some(api_key.api_key.id.clone()),
},
}
}
#[tokio::test]
async fn adapter_attempt_span_carries_durable_attempt_id() {
let registry = common::test_registry().await;
let calls = Arc::new(AtomicUsize::new(0));
let upstream_app = Router::new().route(
"/crm/leads",
get({
let calls = Arc::clone(&calls);
move || {
let calls = Arc::clone(&calls);
async move {
let n = calls.fetch_add(1, Ordering::SeqCst);
if n == 0 {
return (
StatusCode::INTERNAL_SERVER_ERROR,
axum::Json(serde_json::json!({"error":"boom"})),
);
}
(
StatusCode::OK,
axum::Json(serde_json::json!({"id":"lead-1"})),
)
}
}
}),
);
let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
let address = listener.local_addr().unwrap();
let server = tokio::spawn(async move { axum::serve(listener, upstream_app).await.unwrap() });
let upstream = format!("http://{address}");
let mut operation = common::test_operation(&upstream, TOOL_NAME);
let Target::Rest(target) = &mut operation.target;
target.method = crank_core::HttpMethod::Get;
operation.execution_config.execution_policy = Some(ToolExecutionPolicy {
mode: crank_core::ToolExecutionMode::Background,
deadline_ms: 60_000,
no_tasks_fallback: crank_core::NoTasksFallback::Unsupported,
});
operation.execution_config.retry_policy = Some(RetryPolicy { max_attempts: 2 });
registry
.create_operation(&common::test_workspace_id(), &operation, Some("alice"))
.await
.unwrap();
publish(&registry, &operation.id, 1).await;
common::publish_agent_with_bindings(
&registry,
AGENT_SLUG,
vec![AgentOperationBinding {
operation_id: operation.id.clone(),
operation_version: 1,
tool_name: operation.name.clone(),
tool_title: operation.tool_description.title.clone(),
tool_description_override: None,
enabled: true,
}],
)
.await;
let caller = principal(
&registry,
"attempt-trace",
&[
crank_core::PlatformApiKeyScope::Read,
crank_core::PlatformApiKeyScope::Write,
],
)
.await;
let (tasks, store, crypto) = application_with_store(&registry).await;
let acceptance = tasks
.accept_background(
&caller,
TOOL_NAME,
None,
KEY,
&serde_json::json!({"email": "user@example.com"}),
&CorrelationContext::generate(),
)
.await
.expect("durable acceptance");
let runtime =
community_with_outbound_policy(OutboundHttpPolicy::allowing_hosts(["127.0.0.1"])).build();
let handler = Arc::new(TaskWorkerHandler::from_application(
&tasks,
registry.clone(),
runtime,
crypto,
));
let mut config = TaskWorkerConfig::default();
config.owner = "runtime-worker-attempt-trace".to_owned();
config.batch_limit = 1;
config.lease_ms = 10_000;
config.poll_interval = std::time::Duration::from_millis(50);
config.max_inflight = 1;
config.shutdown_timeout = std::time::Duration::from_secs(5);
let worker = TaskWorker::new(store.clone(), handler, config).expect("valid worker");
let stop_flag = worker.stop_flag();
let capture = TraceCapture::default();
let subscriber = tracing_subscriber::registry().with(capture.clone());
let _guard = tracing::subscriber::set_default(subscriber);
let worker_task = tokio::spawn(async move { worker.run().await });
let mut state = None;
for _ in 0..200 {
tokio::time::sleep(std::time::Duration::from_millis(50)).await;
let current: Option<String> =
sqlx::query_scalar("select state from tasks where task_id = $1")
.bind(acceptance.task_id.as_uuid())
.fetch_optional(registry.pool())
.await
.unwrap();
state = current;
if state.as_deref() == Some("completed") {
break;
}
}
stop_flag.store(true, Ordering::SeqCst);
tokio::time::timeout(std::time::Duration::from_secs(10), worker_task)
.await
.expect("worker stops inside the budget")
.expect("worker exits cleanly")
.expect("worker reports no error");
server.abort();
let durable_attempt_id: Option<String> = sqlx::query_scalar::<_, Option<String>>(
"select current_attempt_id from tasks where task_id = $1",
)
.bind(acceptance.task_id.as_uuid())
.fetch_one(registry.pool())
.await
.unwrap();
assert_eq!(state.as_deref(), Some("completed"));
let spans = capture.snapshot();
let attempt_spans: Vec<_> = spans
.iter()
.filter(|s| s.name == "adapter_attempt")
.collect();
assert_eq!(attempt_spans.len(), 2, "two adapter_attempt spans");
for span in &attempt_spans {
let aid = span.fields.get("attempt_id").cloned().unwrap_or_default();
assert!(!aid.is_empty(), "attempt_id must not be empty");
assert!(
!aid.contains("traceparent"),
"attempt_id must not contain traceparent"
);
assert_eq!(
Some(aid.as_str()),
durable_attempt_id.as_deref(),
"span attempt_id must equal durable attempt id"
);
}
let ids: Vec<String> = attempt_spans
.iter()
.map(|s| s.fields.get("attempt_id").cloned().unwrap_or_default())
.collect();
assert_eq!(ids[0], ids[1], "same durable id on both retries");
assert_eq!(
spans
.iter()
.filter(|span| span.name == "durable_task_attempt")
.count(),
1,
"one durable attempt span per dispatch"
);
let runtime_span = spans
.iter()
.find(|span| span.name == "runtime.execute")
.expect("runtime.execute span");
assert_eq!(
runtime_span.parent_name,
Some("durable_task_attempt"),
"the execution continues the durable attempt trace"
);
for span in &attempt_spans {
assert_eq!(
span.parent_name,
Some("runtime.execute"),
"each retry is a child attempt span under the durable execution"
);
}
let attempts: Vec<String> = attempt_spans
.iter()
.map(|s| s.fields.get("attempt").cloned().unwrap_or_default())
.collect();
assert!(attempts.contains(&"1".to_string()));
assert!(attempts.contains(&"2".to_string()));
for span in &attempt_spans {
assert_eq!(
span.fields.get("max_attempts").cloned().unwrap_or_default(),
"2"
);
}
}
// TraceCapture minimal port from crates/crank-runtime/tests/stages.rs
#[derive(Clone, Default)]
struct TraceCapture {
spans: Arc<Mutex<Vec<CapturedSpan>>>,
}
impl TraceCapture {
fn snapshot(&self) -> Vec<CapturedSpan> {
self.spans.lock().expect("span lock").clone()
}
}
#[derive(Clone, Debug)]
struct CapturedSpan {
name: &'static str,
parent_name: Option<&'static str>,
fields: std::collections::BTreeMap<String, String>,
}
impl<S> Layer<S> for TraceCapture
where
S: Subscriber + for<'lookup> LookupSpan<'lookup>,
{
fn on_new_span(
&self,
attributes: &tracing::span::Attributes<'_>,
id: &Id,
context: tracing_subscriber::layer::Context<'_, S>,
) {
let parent = attributes
.parent()
.and_then(|parent| context.span(parent))
.or_else(|| {
attributes
.is_contextual()
.then(|| context.lookup_current())
.flatten()
});
let mut visitor = FieldVisitor::default();
attributes.record(&mut visitor);
let mut spans = self.spans.lock().expect("span lock");
let index = spans.len();
spans.push(CapturedSpan {
name: attributes.metadata().name(),
parent_name: parent.map(|span| span.metadata().name()),
fields: visitor.fields,
});
context
.span(id)
.expect("span exists")
.extensions_mut()
.insert(index);
}
fn on_record(
&self,
id: &Id,
values: &tracing::span::Record<'_>,
context: tracing_subscriber::layer::Context<'_, S>,
) {
let mut visitor = FieldVisitor::default();
values.record(&mut visitor);
let span = context.span(id).expect("span exists");
let index = *span.extensions().get::<usize>().expect("capture index");
self.spans.lock().expect("span lock")[index]
.fields
.extend(visitor.fields);
}
}
#[derive(Default)]
struct FieldVisitor {
fields: std::collections::BTreeMap<String, String>,
}
impl Visit for FieldVisitor {
fn record_str(&mut self, field: &tracing::field::Field, value: &str) {
self.fields
.insert(field.name().to_owned(), value.to_owned());
}
fn record_debug(&mut self, field: &tracing::field::Field, value: &dyn std::fmt::Debug) {
self.fields
.insert(field.name().to_owned(), format!("{value:?}"));
}
}
@@ -0,0 +1,610 @@
//! ATDD Story 4.2 (FR-22/FR-23): the durable Task application entry over a real published catalog
//! and a real PostgreSQL schema.
//!
//! These oracles drive the production application entry (`TaskApplication`) that the mcp-server
//! composition root constructs: a verified machine credential is checked against the published Tool,
//! the Task is accepted durably, a freshly constructed application instance reads the same identity
//! and pins, replay/version conflicts never rebind, foreign callers receive no metadata, and a missing
//! Task store fails closed and reports readiness as not ready. The public `tools/call` wire stays
//! synchronous/unsupported: this suite never calls it.
use super::common;
use std::time::Duration;
use crank_community_mcp::{TaskApplication, TaskApplicationError, TaskPrincipal};
use crank_core::{
AgentOperationBinding, CorrelationContext, NoTasksFallback, PlatformApiKeyScope, RequestId,
TaskAcceptRequest, TaskEnvelope, TaskId, TaskOwnerKind, TaskPins, TaskState, TaskStore,
TaskSupportIds, ToolExecutionMode, ToolExecutionPolicy,
};
use crank_registry::{
AdmissionLimits, CreateVersionRequest, PostgresRegistry, PostgresTaskStore, PublishRequest,
};
use crank_runtime::{InMemoryCoordinationStateStore, SecretCrypto, TaskService};
use serde_json::json;
use sqlx::{PgPool, Row};
use time::{OffsetDateTime, format_description::well_known::Rfc3339};
const AGENT_SLUG: &str = "reports";
const TOOL_NAME: &str = "report_gen";
const KEY: &str = "retry-key-1";
fn operation(policy: ToolExecutionPolicy) -> crank_registry::RegistryOperation {
let mut operation = common::test_operation("https://api.example.com", TOOL_NAME);
operation.execution_config.execution_policy = Some(policy);
operation
}
fn background_policy() -> ToolExecutionPolicy {
ToolExecutionPolicy {
mode: ToolExecutionMode::Background,
deadline_ms: 60_000,
no_tasks_fallback: NoTasksFallback::Unsupported,
}
}
async fn seed_tool(registry: &PostgresRegistry, policy: ToolExecutionPolicy) {
let operation = operation(policy);
registry
.create_operation(&common::test_workspace_id(), &operation, Some("alice"))
.await
.unwrap();
publish(registry, &operation.id, 1).await;
// A second, explicitly synchronous Tool of the same Agent: the durable entry must refuse it.
let synchronous = common::test_operation("https://api.example.com", "report_sync");
registry
.create_operation(&common::test_workspace_id(), &synchronous, Some("alice"))
.await
.unwrap();
publish(registry, &synchronous.id, 1).await;
let binding = AgentOperationBinding {
operation_id: operation.id.clone(),
operation_version: 1,
tool_name: operation.name.clone(),
tool_title: operation.tool_description.title.clone(),
tool_description_override: None,
enabled: true,
};
let synchronous_binding = AgentOperationBinding {
operation_id: synchronous.id.clone(),
operation_version: 1,
tool_name: synchronous.name.clone(),
tool_title: synchronous.tool_description.title.clone(),
tool_description_override: None,
enabled: true,
};
common::publish_agent_with_bindings(registry, AGENT_SLUG, vec![binding, synchronous_binding])
.await;
}
async fn publish(
registry: &PostgresRegistry,
operation_id: &crank_core::OperationId,
version: u32,
) {
registry
.publish_operation(PublishRequest {
workspace_id: &common::test_workspace_id(),
operation_id,
version,
published_at: &OffsetDateTime::parse("2026-03-26T10:00:00Z", &Rfc3339).unwrap(),
published_by: Some("alice"),
})
.await
.unwrap();
}
async fn publish_tool_version(registry: &PostgresRegistry, version: u32) {
let mut next = operation(background_policy());
next.version = version;
next.status = crank_core::OperationStatus::Draft;
next.published_at = None;
registry
.create_version(CreateVersionRequest {
workspace_id: &common::test_workspace_id(),
snapshot: &next,
change_note: Some("next publication"),
created_by: Some("alice"),
})
.await
.unwrap();
publish(registry, &next.id, version).await;
// The agent catalog must bind the published Tool version for the new publication to be addressed.
let agent_id = common::test_agent_id(AGENT_SLUG);
let draft = crank_core::AgentVersion {
agent_id: agent_id.clone(),
version: 2,
status: crank_core::AgentStatus::Draft,
instructions: json!({}),
tool_selection_policy: Default::default(),
created_at: OffsetDateTime::parse("2026-03-26T10:01:00Z", &Rfc3339).unwrap(),
};
registry
.create_agent_draft_version(crank_registry::CreateAgentDraftVersionRequest {
workspace_id: &common::test_workspace_id(),
agent_id: &agent_id,
version: &draft,
bindings: &[AgentOperationBinding {
operation_id: next.id.clone(),
operation_version: version,
tool_name: next.name.clone(),
tool_title: next.tool_description.title.clone(),
tool_description_override: None,
enabled: true,
}],
updated_at: &OffsetDateTime::parse("2026-03-26T10:01:00Z", &Rfc3339).unwrap(),
})
.await
.unwrap();
registry
.publish_agent(crank_registry::PublishAgentRequest {
workspace_id: &common::test_workspace_id(),
agent_id: &agent_id,
version: 2,
published_at: &OffsetDateTime::parse("2026-03-26T10:02:00Z", &Rfc3339).unwrap(),
published_by: Some("alice"),
expected_state: None,
})
.await
.unwrap();
}
const TEST_MASTER_KEY: &str = "test-master-key-00000000000000000000000000000000";
fn task_service(pool: PgPool, crypto: SecretCrypto) -> TaskService {
let store = PostgresTaskStore::new(
pool,
AdmissionLimits {
global_limit: 4,
agent_limit: 4,
tool_limit: 4,
queue_limit: 8,
},
Default::default(),
)
.expect("valid task store configuration");
TaskService::with_secret_crypto(std::sync::Arc::new(store), std::sync::Arc::new(crypto))
}
/// The application entry under the same encrypted-input authority the production process uses.
async fn application(registry: &PostgresRegistry) -> TaskApplication {
let crypto = SecretCrypto::new(TEST_MASTER_KEY).expect("test master key");
let observed_at = OffsetDateTime::now_utc();
registry
.verify_or_register_master_key_identity(crank_registry::MasterKeyIdentityCandidate {
epoch: crypto.master_key_epoch(),
fingerprint: crypto.master_key_fingerprint(),
cipher_contract: crank_registry::MASTER_KEY_CIPHER_CONTRACT,
observed_at: &observed_at,
})
.await
.expect("test master-key identity registered");
TaskApplication::new(
registry.clone(),
Duration::from_millis(0),
std::sync::Arc::new(InMemoryCoordinationStateStore::default()),
task_service(registry.pool().clone(), crypto),
)
}
async fn principal(
registry: &PostgresRegistry,
name: &str,
scopes: &[PlatformApiKeyScope],
) -> TaskPrincipal {
let token = common::create_platform_api_key(registry, AGENT_SLUG, name, scopes).await;
// The production machine-access path resolves the presented secret against the active platform
// key of the addressed Agent and derives the scope identity from that verified record.
let secret_hash = common::hash_access_secret(&token);
let api_key = registry
.get_platform_api_key_by_secret_for_agent_slug(
common::test_workspace_slug(),
AGENT_SLUG,
&secret_hash,
)
.await
.unwrap()
.expect("an active platform key resolves");
registry
.touch_platform_api_key(
&api_key.api_key.workspace_id,
&api_key.api_key.id,
&OffsetDateTime::now_utc(),
)
.await
.unwrap();
TaskPrincipal::Agent {
workspace_slug: common::test_workspace_slug().to_owned(),
agent_slug: AGENT_SLUG.to_owned(),
credential: crank_core::VerifiedMachineCredential {
machine_access_mode: crank_core::MachineAccessMode::StaticAgentKey,
max_security_level: crank_core::OperationSecurityLevel::Standard,
scopes: api_key.api_key.scopes.clone(),
platform_api_key_id: Some(api_key.api_key.id.clone()),
},
}
}
fn unsampled_correlation() -> CorrelationContext {
CorrelationContext::new(
crank_core::RequestId::generate(),
crank_core::TraceContext::parse("00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-00")
.unwrap(),
)
}
async fn counts(pool: &PgPool) -> (i64, i64) {
let row = sqlx::query(
"select
(select count(*)::bigint from tasks) as tasks,
(select count(*)::bigint from execution_admission_entries where task_id is not null)
as intents",
)
.fetch_one(pool)
.await
.unwrap();
(row.get("tasks"), row.get("intents"))
}
#[tokio::test]
async fn a_verified_caller_receives_a_durable_task_that_a_new_instance_reads() {
let registry = common::test_registry().await;
seed_tool(&registry, background_policy()).await;
let caller = principal(
&registry,
"task-accept",
&[PlatformApiKeyScope::Read, PlatformApiKeyScope::Write],
)
.await;
let correlation = unsampled_correlation();
let app = application(&registry).await;
let acceptance = app
.accept_background(
&caller,
TOOL_NAME,
None,
KEY,
&json!({"report": "q3"}),
&correlation,
)
.await
.expect("acceptance over the real publication");
assert_eq!(acceptance.state, TaskState::Queued);
assert!(!acceptance.replayed);
assert_eq!(acceptance.operation_version, 1);
assert_eq!(acceptance.support.request_id, *correlation.request_id());
assert_eq!(acceptance.support.trace_id, *correlation.trace_id());
assert!(!acceptance.support.trace_sampled);
assert_eq!(
uuid::Uuid::parse_str(&acceptance.task_id.to_string())
.unwrap()
.get_version_num(),
7,
"the durable Task id is UUIDv7"
);
// Task, dispatch intent and queue slot committed together.
assert_eq!(counts(registry.pool()).await, (1, 1));
// A freshly constructed application + service reads the same durable identity and pins.
let reader = application(&registry).await;
let record = reader
.status(&caller, TOOL_NAME, acceptance.task_id)
.await
.unwrap();
assert_eq!(record.state, TaskState::Queued);
assert_eq!(record.pins.operation_version, 1);
assert_eq!(record.pins.policy_mode, ToolExecutionMode::Background);
assert_eq!(record.support.request_id, acceptance.support.request_id);
assert_eq!(record.support.traceparent, acceptance.support.traceparent);
assert!(!record.support.trace_sampled);
let outcome = reader
.result(&caller, TOOL_NAME, acceptance.task_id)
.await
.unwrap();
assert!(outcome.result.is_none() && outcome.error.is_none());
// Replay of the same scoped key and input returns the accepted Task instead of a new one.
let replay = app
.accept_background(
&caller,
TOOL_NAME,
None,
KEY,
&json!({"report": "q3"}),
&unsampled_correlation(),
)
.await
.unwrap();
assert_eq!(replay.task_id, acceptance.task_id);
assert!(replay.replayed);
assert_eq!(counts(registry.pool()).await, (1, 1));
}
#[tokio::test]
async fn a_new_publication_never_rebinds_an_accepted_task() {
let registry = common::test_registry().await;
seed_tool(&registry, background_policy()).await;
let caller = principal(
&registry,
"task-rebind",
&[PlatformApiKeyScope::Read, PlatformApiKeyScope::Write],
)
.await;
let app = application(&registry).await;
let accepted = app
.accept_background(
&caller,
TOOL_NAME,
None,
KEY,
&json!({"report": "q3"}),
&unsampled_correlation(),
)
.await
.unwrap();
let before = app
.status(&caller, TOOL_NAME, accepted.task_id)
.await
.unwrap();
publish_tool_version(&registry, 2).await;
// The same retry keeps the accepted Task and its v1 pin; it never becomes a v2 Task.
let replay = app
.accept_background(
&caller,
TOOL_NAME,
None,
KEY,
&json!({"report": "q3"}),
&unsampled_correlation(),
)
.await
.unwrap();
assert_eq!(replay.task_id, accepted.task_id);
assert_eq!(replay.operation_version, 1);
let after = app
.status(&caller, TOOL_NAME, accepted.task_id)
.await
.unwrap();
assert_eq!(after.pins.operation_version, 1);
assert_eq!(after.support.request_id, before.support.request_id);
assert_eq!(counts(registry.pool()).await, (1, 1));
// An explicit other exact version is a typed conflict, not a rebind.
let explicit = app
.accept_background(
&caller,
TOOL_NAME,
Some(2),
KEY,
&json!({"report": "q3"}),
&unsampled_correlation(),
)
.await
.expect_err("a different explicit pin cannot rebind the accepted Task");
assert_eq!(explicit, TaskApplicationError::VersionUnavailable);
// The original explicit pin still resolves to the accepted Task.
let pinned = app
.accept_background(
&caller,
TOOL_NAME,
Some(1),
KEY,
&json!({"report": "q3"}),
&unsampled_correlation(),
)
.await
.unwrap();
assert_eq!(pinned.task_id, accepted.task_id);
assert_eq!(counts(registry.pool()).await, (1, 1));
}
#[tokio::test]
async fn foreign_and_insufficient_callers_receive_no_task_metadata() {
let registry = common::test_registry().await;
seed_tool(&registry, background_policy()).await;
let owner = principal(
&registry,
"task-owner",
&[PlatformApiKeyScope::Read, PlatformApiKeyScope::Write],
)
.await;
let app = application(&registry).await;
let accepted = app
.accept_background(
&owner,
TOOL_NAME,
None,
KEY,
&json!({"report": "q3"}),
&unsampled_correlation(),
)
.await
.unwrap();
// Positive control: the owner reads its Task.
assert!(
app.status(&owner, TOOL_NAME, accepted.task_id)
.await
.is_ok()
);
// A second verified key of the same Agent is another authorization scope: no payload, no metadata.
let foreign_scope =
principal(&registry, "task-other-scope", &[PlatformApiKeyScope::Read]).await;
match app
.status(&foreign_scope, TOOL_NAME, accepted.task_id)
.await
{
Err(TaskApplicationError::NotFound) => {}
other => panic!("foreign scope must be not-found without metadata, got {other:?}"),
}
assert!(matches!(
app.result(&foreign_scope, TOOL_NAME, accepted.task_id)
.await,
Err(TaskApplicationError::NotFound)
));
// A synchronous publication is not accepted as background work at all.
let synchronous_app = application(&registry).await;
let refused = synchronous_app
.accept_background(
&owner,
"report_sync",
None,
"sync-key",
&json!({"report": "q3"}),
&unsampled_correlation(),
)
.await
.expect_err("a synchronous policy has no durable acceptance");
assert_eq!(refused, TaskApplicationError::NotBackground);
assert_eq!(counts(registry.pool()).await, (1, 1));
}
#[tokio::test]
async fn admin_owned_tasks_are_scoped_to_that_admin_and_workspace() {
let registry = common::test_registry().await;
seed_tool(&registry, background_policy()).await;
let app = application(&registry).await;
let admin = TaskPrincipal::Admin {
workspace_slug: common::test_workspace_slug().to_owned(),
agent_slug: AGENT_SLUG.to_owned(),
user_id: "user_admin_1".to_owned(),
};
let accepted = app
.accept_background(
&admin,
TOOL_NAME,
None,
KEY,
&json!({"report": "q3"}),
&unsampled_correlation(),
)
.await
.expect("an authenticated administrator accepts durable work");
assert_eq!(accepted.state, TaskState::Queued);
// Positive control: the same administrator reads its own Task.
assert!(
app.status(&admin, TOOL_NAME, accepted.task_id)
.await
.is_ok()
);
// An administrator is never a wildcard owner: another admin user and the owning Agent of the
// same workspace both see nothing.
let other_admin = TaskPrincipal::Admin {
workspace_slug: common::test_workspace_slug().to_owned(),
agent_slug: AGENT_SLUG.to_owned(),
user_id: "user_admin_2".to_owned(),
};
assert_eq!(
app.status(&other_admin, TOOL_NAME, accepted.task_id)
.await
.unwrap_err(),
TaskApplicationError::NotFound
);
assert_eq!(
app.result(&other_admin, TOOL_NAME, accepted.task_id)
.await
.unwrap_err(),
TaskApplicationError::NotFound
);
let agent = principal(&registry, "task-admin-cross", &[PlatformApiKeyScope::Read]).await;
assert_eq!(
app.status(&agent, TOOL_NAME, accepted.task_id)
.await
.unwrap_err(),
TaskApplicationError::NotFound
);
// An admin identity addressed to another workspace cannot resolve the publication at all.
let foreign_workspace = TaskPrincipal::Admin {
workspace_slug: "other-workspace".to_owned(),
agent_slug: AGENT_SLUG.to_owned(),
user_id: "user_admin_1".to_owned(),
};
assert_eq!(
app.accept_background(
&foreign_workspace,
TOOL_NAME,
None,
"foreign-key",
&json!({"report": "q3"}),
&unsampled_correlation(),
)
.await
.unwrap_err(),
TaskApplicationError::NotFound
);
assert_eq!(counts(registry.pool()).await, (1, 1));
}
#[tokio::test]
async fn a_missing_task_store_fails_closed_and_readiness_reports_not_ready() {
let database_url = crank_test_support::postgres_schema_url("durable_tasks_outage").await;
let pool = PgPool::connect(&database_url).await.unwrap();
// No migration: the durable Task state does not exist.
let outage_registry = PostgresRegistry::connect(&database_url).await;
assert!(
outage_registry.is_err(),
"an unmigrated registry cannot be used"
);
crank_registry::MigrationAuthority::apply(&pool)
.await
.unwrap();
let registry = PostgresRegistry::connect(&database_url).await.unwrap();
seed_tool(&registry, background_policy()).await;
let caller = principal(
&registry,
"task-outage",
&[PlatformApiKeyScope::Read, PlatformApiKeyScope::Write],
)
.await;
let app = application(&registry).await;
assert!(app.health().await.is_ok());
// Fail closed: without the Task schema the entry accepts nothing and reports dependency failure.
sqlx::query("drop table tasks cascade")
.execute(&pool)
.await
.unwrap();
assert!(app.health().await.is_err());
let error = app
.accept_background(
&caller,
TOOL_NAME,
None,
KEY,
&json!({"report": "q3"}),
&unsampled_correlation(),
)
.await
.expect_err("no durable Task store, no acceptance");
assert!(matches!(
error,
TaskApplicationError::Task(crank_core::TaskError::Unavailable { .. })
));
// Readiness of the app that wires the same store reports the durable Task dependency as not ready.
let router = common::build_test_app(registry.clone(), Duration::from_millis(0), None);
let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
let address = listener.local_addr().unwrap();
let server = tokio::spawn(async move {
axum::serve(listener, router).await.unwrap();
});
let response = reqwest::get(format!("http://{address}/ready"))
.await
.unwrap();
assert_eq!(response.status(), reqwest::StatusCode::SERVICE_UNAVAILABLE);
let body: serde_json::Value = response.json().await.unwrap();
assert_eq!(body["checks"]["durable_tasks"], json!("not_ready"));
server.abort();
}
mod attempt_trace;
mod shutdown_window;
mod task_worker;
@@ -0,0 +1,190 @@
//! Story 4.3 QA caveat proof: the narrow post-checkpoint / pre-permit stop window.
//!
//! Split from `task_worker.rs` to keep both integration files inside the code-health limit.
use super::*;
/// A stop that lands after the durable dispatch checkpoint but before the sealed dispatch permit
/// exists finalizes the Task as failed(`shutdown_before_dispatch`) without any outbound call, and
/// does not consume work that is still queued.
///
/// The window is made deterministic instead of raced: a `BEFORE UPDATE` trigger delays only the
/// durable dispatch checkpoint, so the stop flag is set while that checkpoint statement is still in
/// flight (before `TaskDispatchPermit`/`TaskHandler::execute` can exist).
#[tokio::test]
async fn task_worker_stop_between_checkpoint_and_dispatch_never_reaches_upstream() {
use crank_core::{TaskAccess, TaskLease, TaskOwnerKind};
use crank_runtime::{
TaskAttemptOutcome, TaskDispatchPermit, TaskHandler, TaskWorker, TaskWorkerConfig,
TaskWorkerError,
};
use std::sync::Arc;
use std::sync::atomic::{AtomicUsize, Ordering};
let registry = common::test_registry().await;
// The test registry exposes a single pooled connection: delaying the checkpoint inside its own
// statement keeps the window deterministic without competing for a second connection.
sqlx::raw_sql(sqlx::AssertSqlSafe(
"create function delay_task_dispatch_checkpoint() returns trigger language plpgsql as $$
begin
if new.current_attempt_id is not null and old.current_attempt_id is null then
perform pg_sleep(1.5);
end if;
return new;
end $$;
create trigger delay_task_dispatch_checkpoint before update on tasks
for each row execute function delay_task_dispatch_checkpoint();",
))
.execute(registry.pool())
.await
.expect("checkpoint delay trigger");
let store = Arc::new(
PostgresTaskStore::new(
registry.pool().clone(),
AdmissionLimits {
global_limit: 4,
agent_limit: 4,
tool_limit: 4,
queue_limit: 8,
},
Default::default(),
)
.expect("valid task store configuration"),
);
let mut accepted = Vec::new();
for (seed, scoped_key) in [("q", "a"), ("r", "c")] {
let request = TaskAcceptRequest {
task_id: TaskId::generate(),
workspace_id: common::test_workspace_id().to_string(),
owner_kind: TaskOwnerKind::Agent,
owner_id: "agent_a".to_owned(),
authorization_scope: "credential:pk_stop_window".to_owned(),
platform_api_key_id: None,
operation_id: "op_report".to_owned(),
pins: TaskPins {
operation_version: 1,
policy_mode: ToolExecutionMode::Background,
execution_deadline_unix_ms: 1_900_000_000_000,
retain_until_unix_ms: 1_900_000_000_000 + 60_000,
approval_reference: None,
},
scoped_key: scoped_key.repeat(64),
input_fingerprint: scoped_key.repeat(64),
input: TaskEnvelope::try_redacted(&json!({ "report": seed })).unwrap(),
support: TaskSupportIds::new(
RequestId::generate(),
&crank_core::TraceContext::parse(
"00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-00",
)
.unwrap(),
),
executable_input_ciphertext: None,
executable_input_key_version: None,
executable_input_master_key_epoch: None,
};
accepted.push(store.accept(request).await.expect("PG accepts"));
}
// The first revalidation waits for the test; the second Task is never claimed (batch_limit = 1).
let revalidate_entered = Arc::new(tokio::sync::Semaphore::new(0));
let revalidate_release = Arc::new(tokio::sync::Semaphore::new(0));
let executed = Arc::new(AtomicUsize::new(0));
struct GatedHandler {
entered: Arc<tokio::sync::Semaphore>,
release: Arc<tokio::sync::Semaphore>,
executed: Arc<AtomicUsize>,
}
#[async_trait::async_trait]
impl TaskHandler for GatedHandler {
async fn revalidate(&self, _lease: &TaskLease) -> Result<(), TaskWorkerError> {
self.entered.add_permits(1);
let _ = self.release.acquire().await;
Ok(())
}
async fn execute(
&self,
_lease: &TaskLease,
_attempt_id: &str,
_cancelled: Arc<std::sync::atomic::AtomicBool>,
_permit: TaskDispatchPermit,
) -> TaskAttemptOutcome {
self.executed.fetch_add(1, Ordering::SeqCst);
TaskAttemptOutcome::Completed(json!({ "ok": true }))
}
}
let handler: Arc<dyn TaskHandler> = Arc::new(GatedHandler {
entered: revalidate_entered.clone(),
release: revalidate_release.clone(),
executed: executed.clone(),
});
let mut config = TaskWorkerConfig::default();
config.owner = "runtime-worker-stop-window".to_owned();
config.batch_limit = 1;
config.lease_ms = 30_000;
config.poll_interval = Duration::from_millis(50);
config.renewal_interval = Duration::from_secs(20);
config.max_inflight = 1;
config.shutdown_timeout = Duration::from_secs(20);
let worker = TaskWorker::new(store.clone(), handler, config).expect("valid worker");
let stop_flag = worker.stop_flag();
let worker_task = tokio::spawn(async move { worker.run().await });
tokio::time::timeout(Duration::from_secs(5), revalidate_entered.acquire())
.await
.expect("worker must claim the first Task and enter revalidation")
.expect("revalidation gate is open")
.forget();
// Let revalidation continue; the checkpoint now runs while the trigger holds it for 1.5s.
revalidate_release.add_permits(1);
tokio::time::sleep(Duration::from_millis(300)).await;
// The stop now lands while the durable checkpoint statement is still in flight: the worker may
// finalize the Task, but no permit exists yet, so no outbound call can start.
let stop_requested = std::time::Instant::now();
stop_flag.store(true, Ordering::SeqCst);
tokio::time::timeout(Duration::from_secs(30), worker_task)
.await
.expect("worker stops inside the 30s budget")
.expect("worker exits cleanly")
.expect("worker reports no error");
assert!(
stop_requested.elapsed() >= Duration::from_millis(500),
"the checkpoint must have still been in flight when the stop arrived"
);
assert_eq!(
executed.load(Ordering::SeqCst),
0,
"a stop between checkpoint and permit must never reach the upstream adapter"
);
let access = TaskAccess {
workspace_id: common::test_workspace_id().to_string(),
owner_kind: TaskOwnerKind::Agent,
owner_id: "agent_a".to_owned(),
authorization_scope: "credential:pk_stop_window".to_owned(),
};
let record = store.read(&access, accepted[0].task_id).await.unwrap();
assert_eq!(record.state, TaskState::Failed);
assert_eq!(record.attempts, 1, "the durable checkpoint did commit");
assert!(record.current_attempt_id.is_some());
assert_eq!(
record
.error
.as_ref()
.and_then(|error| error.value().get("code"))
.and_then(|code| code.as_str()),
Some("shutdown_before_dispatch")
);
// Work that was still queued is not consumed by the stopped worker.
let queued = store.read(&access, accepted[1].task_id).await.unwrap();
assert_eq!(queued.state, TaskState::Queued);
let recovered = store
.claim_batch("worker_after_stop", 4, 10_000)
.await
.expect("queued work stays claimable");
assert_eq!(recovered.len(), 1);
assert_eq!(recovered[0].record.task_id, accepted[1].task_id);
}
@@ -0,0 +1,822 @@
//! Story 4.3 ATDD: fenced durable Task worker leases, recovery and real upstream dispatch.
//!
//! Split from `durable_tasks.rs` to keep both integration files inside the code-health limit.
use super::*;
/// Store-level positive control only: PostgreSQL leases/fencing/recovery work without Valkey or a
/// task-worker process. This does not exercise TaskWorker startup, a bounded authoritative scan,
/// graceful shutdown, outbound dispatch, or runtime recovery timing, which remain unimplemented.
#[tokio::test]
async fn task_worker_store_lease_recovery_contract() {
let registry = common::test_registry().await;
let store = PostgresTaskStore::new(
registry.pool().clone(),
AdmissionLimits {
global_limit: 4,
agent_limit: 4,
tool_limit: 4,
queue_limit: 8,
},
Default::default(),
)
.expect("valid task store configuration");
// Positive control: no Valkey or worker process is involved; PostgreSQL alone accepts and
// leases durable work, preserving the same admission-backed intent.
let request = TaskAcceptRequest {
task_id: TaskId::generate(),
workspace_id: common::test_workspace_id().to_string(),
owner_kind: TaskOwnerKind::Agent,
owner_id: "agent_a".to_owned(),
authorization_scope: "credential:pk_1".to_owned(),
platform_api_key_id: None,
operation_id: "op_report".to_owned(),
pins: TaskPins {
operation_version: 1,
policy_mode: ToolExecutionMode::Background,
execution_deadline_unix_ms: 1_900_000_000_000,
retain_until_unix_ms: 1_900_000_000_000 + 60_000,
approval_reference: None,
},
scoped_key: "a".repeat(64),
input_fingerprint: "b".repeat(64),
input: TaskEnvelope::try_redacted(&json!({"report": "q3"})).unwrap(),
support: TaskSupportIds::new(
RequestId::generate(),
&crank_core::TraceContext::parse(
"00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-00",
)
.unwrap(),
),
// This store-level control exercises lease/fence behavior, not the encrypted acceptance path.
executable_input_ciphertext: None,
executable_input_key_version: None,
executable_input_master_key_epoch: None,
};
let accepted = store.accept(request).await.expect("PG accepts");
let leases = store
.claim_batch("worker_1", 4, 10_000)
.await
.expect("claim");
assert_eq!(leases.len(), 1);
assert_eq!(leases[0].record.task_id, accepted.task_id);
let lease = leases[0].clone();
// A stale or competing owner cannot win CAS with the issued fencing token.
assert!(
store
.renew_lease("worker_2", lease.record.task_id, lease.fence, 10_000)
.await
.is_err()
);
assert!(
store
.write_progress(
"worker_1",
lease.record.task_id,
lease.fence + 1,
TaskEnvelope::try_redacted(&json!({"progress": 1})).unwrap(),
)
.await
.is_err()
);
// A stale fencing token can never checkpoint dispatch, even under the issuing owner.
assert!(
store
.checkpoint_dispatch(
"worker_1",
lease.record.task_id,
lease.fence + 1,
"attempt-stale".to_owned()
)
.await
.is_err()
);
// The valid owner still commits the dispatch checkpoint before any outbound work.
assert!(
store
.checkpoint_dispatch(
"worker_1",
lease.record.task_id,
lease.fence,
"attempt-1".to_owned(),
)
.await
.unwrap()
.current_attempt_id
.as_deref()
== Some("attempt-1")
);
}
#[tokio::test]
async fn task_worker_runtime_dispatch_and_recovery_contract() {
use async_trait::async_trait;
use crank_core::{TaskAccess, TaskLease, TaskOwnerKind};
use crank_runtime::{
TaskAttemptOutcome, TaskDispatchPermit, TaskHandler, TaskWorker, TaskWorkerConfig,
TaskWorkerError,
};
use std::sync::Arc;
use std::sync::atomic::{AtomicUsize, Ordering};
let registry = common::test_registry().await;
let pool = registry.pool().clone();
let store = Arc::new(
PostgresTaskStore::new(
pool.clone(),
AdmissionLimits {
global_limit: 4,
agent_limit: 4,
tool_limit: 4,
queue_limit: 8,
},
Default::default(),
)
.expect("valid task store configuration"),
);
let observed = Arc::new(AtomicUsize::new(0));
// Positive control: PostgreSQL alone accepts the durable Task.
let request = TaskAcceptRequest {
task_id: TaskId::generate(),
workspace_id: common::test_workspace_id().to_string(),
owner_kind: TaskOwnerKind::Agent,
owner_id: "agent_a".to_owned(),
authorization_scope: "credential:pk_runtime".to_owned(),
platform_api_key_id: None,
operation_id: "op_report".to_owned(),
pins: TaskPins {
operation_version: 1,
policy_mode: ToolExecutionMode::Background,
execution_deadline_unix_ms: 1_900_000_000_000,
retain_until_unix_ms: 1_900_000_000_000 + 60_000,
approval_reference: None,
},
scoped_key: "r".repeat(64),
input_fingerprint: "b".repeat(64),
input: TaskEnvelope::try_redacted(&json!({"email": "q3"})).unwrap(),
support: TaskSupportIds::new(
RequestId::generate(),
&crank_core::TraceContext::parse(
"00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-00",
)
.unwrap(),
),
executable_input_ciphertext: None,
executable_input_key_version: None,
executable_input_master_key_epoch: None,
};
let accepted = store.accept(request).await.expect("PG accepts");
let task_id = accepted.task_id;
struct RuntimeHandler {
observed: Arc<AtomicUsize>,
}
#[async_trait]
impl TaskHandler for RuntimeHandler {
async fn revalidate(&self, _lease: &TaskLease) -> Result<(), TaskWorkerError> {
Ok(())
}
async fn execute(
&self,
_lease: &TaskLease,
_attempt_id: &str,
_cancelled: Arc<std::sync::atomic::AtomicBool>,
_permit: TaskDispatchPermit,
) -> TaskAttemptOutcome {
self.observed.fetch_add(1, Ordering::SeqCst);
TaskAttemptOutcome::Completed(serde_json::json!({"ok": true}))
}
}
let handler: Arc<dyn TaskHandler> = Arc::new(RuntimeHandler {
observed: observed.clone(),
});
// PG-only authoritative dispatch: no Valkey or upstream process is involved.
let mut config = TaskWorkerConfig::default();
config.owner = "runtime-worker-1".to_owned();
config.batch_limit = 1;
config.lease_ms = 10_000;
config.poll_interval = Duration::from_millis(100);
config.renewal_interval = Duration::from_millis(1_000);
config.max_inflight = 1;
config.shutdown_timeout = Duration::from_secs(5);
let worker = TaskWorker::new(store.clone(), handler, config).expect("valid worker");
let stop_flag = worker.stop_flag();
let worker_task = tokio::spawn(async move { worker.run().await });
let access = TaskAccess {
workspace_id: common::test_workspace_id().to_string(),
owner_kind: TaskOwnerKind::Agent,
owner_id: "agent_a".to_owned(),
authorization_scope: "credential:pk_runtime".to_owned(),
};
let mut state = None;
for _ in 0..100 {
tokio::time::sleep(Duration::from_millis(100)).await;
if let Ok(record) = store.read(&access, task_id).await {
state = Some(record.state);
if record.state == TaskState::Completed {
break;
}
}
}
stop_flag.store(true, Ordering::SeqCst);
let _ = tokio::time::timeout(Duration::from_secs(10), worker_task).await;
assert_eq!(state, Some(TaskState::Completed));
assert_eq!(
observed.load(Ordering::SeqCst),
1,
"exactly one outbound attempt"
);
// Terminal state is not duplicated.
let record = store.read(&access, task_id).await.expect("terminal read");
assert_eq!(record.state, TaskState::Completed);
// Bounded shutdown with no fresh work returns promptly.
let mut idle_config = TaskWorkerConfig::default();
idle_config.owner = "runtime-worker-idle".to_owned();
idle_config.poll_interval = Duration::from_millis(50);
idle_config.shutdown_timeout = Duration::from_secs(2);
let idle_worker = TaskWorker::new(
store.clone(),
handler_for_no_work(store.clone()),
idle_config,
)
.expect("valid idle worker");
idle_worker.stop();
let started = std::time::Instant::now();
idle_worker.run().await.expect("idle worker stops");
assert!(started.elapsed() <= Duration::from_secs(30));
}
/// Story 4.3 ATDD: failed pre-dispatch revalidation finishes the attempt without any outbound call.
#[tokio::test]
async fn task_worker_revalidation_denial_never_dispatches() {
use crank_core::{TaskAccess, TaskError, TaskLease, TaskOwnerKind};
use crank_runtime::{
TaskAttemptOutcome, TaskDispatchPermit, TaskHandler, TaskWorker, TaskWorkerConfig,
TaskWorkerError,
};
use std::sync::Arc;
use std::sync::atomic::{AtomicUsize, Ordering};
let registry = common::test_registry().await;
let store = Arc::new(
PostgresTaskStore::new(
registry.pool().clone(),
AdmissionLimits {
global_limit: 4,
agent_limit: 4,
tool_limit: 4,
queue_limit: 8,
},
Default::default(),
)
.expect("valid task store configuration"),
);
let request = TaskAcceptRequest {
task_id: TaskId::generate(),
workspace_id: common::test_workspace_id().to_string(),
owner_kind: TaskOwnerKind::Agent,
owner_id: "agent_a".to_owned(),
authorization_scope: "credential:pk_denied".to_owned(),
platform_api_key_id: None,
operation_id: "op_report".to_owned(),
pins: TaskPins {
operation_version: 1,
policy_mode: ToolExecutionMode::Background,
execution_deadline_unix_ms: 1_900_000_000_000,
retain_until_unix_ms: 1_900_000_000_000 + 60_000,
approval_reference: None,
},
scoped_key: "d".repeat(64),
input_fingerprint: "b".repeat(64),
input: TaskEnvelope::try_redacted(&json!({"report": "q3"})).unwrap(),
support: TaskSupportIds::new(
RequestId::generate(),
&crank_core::TraceContext::parse(
"00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01",
)
.unwrap(),
),
executable_input_ciphertext: None,
executable_input_key_version: None,
executable_input_master_key_epoch: None,
};
let accepted = store.accept(request).await.expect("PG accepts");
struct DeniedHandler {
observed: Arc<AtomicUsize>,
}
#[async_trait::async_trait]
impl TaskHandler for DeniedHandler {
async fn revalidate(&self, _lease: &TaskLease) -> Result<(), TaskWorkerError> {
Err(TaskWorkerError::Store(TaskError::invalid(
"publication or credential revoked",
)))
}
async fn execute(
&self,
_lease: &TaskLease,
_attempt_id: &str,
_cancelled: Arc<std::sync::atomic::AtomicBool>,
_permit: TaskDispatchPermit,
) -> TaskAttemptOutcome {
self.observed.fetch_add(1, Ordering::SeqCst);
TaskAttemptOutcome::Completed(json!({"ok": true}))
}
}
let observed = Arc::new(AtomicUsize::new(0));
let handler: Arc<dyn TaskHandler> = Arc::new(DeniedHandler {
observed: observed.clone(),
});
let mut config = TaskWorkerConfig::default();
config.owner = "runtime-worker-denied".to_owned();
config.batch_limit = 1;
config.lease_ms = 10_000;
config.poll_interval = Duration::from_millis(50);
config.max_inflight = 1;
config.shutdown_timeout = Duration::from_secs(5);
let worker = TaskWorker::new(store.clone(), handler, config).expect("valid worker");
let stop_flag = worker.stop_flag();
let worker_task = tokio::spawn(async move { worker.run().await });
let access = TaskAccess {
workspace_id: common::test_workspace_id().to_string(),
owner_kind: TaskOwnerKind::Agent,
owner_id: "agent_a".to_owned(),
authorization_scope: "credential:pk_denied".to_owned(),
};
let mut state = None;
for _ in 0..100 {
tokio::time::sleep(Duration::from_millis(50)).await;
if let Ok(record) = store.read(&access, accepted.task_id).await {
state = Some(record.state);
if record.state == TaskState::Failed {
break;
}
}
}
stop_flag.store(true, Ordering::SeqCst);
let _ = tokio::time::timeout(Duration::from_secs(10), worker_task).await;
assert_eq!(state, Some(TaskState::Failed));
assert_eq!(
observed.load(Ordering::SeqCst),
0,
"invalidated work never reaches the upstream"
);
}
fn handler_for_no_work(
_store: std::sync::Arc<PostgresTaskStore>,
) -> std::sync::Arc<dyn crank_runtime::TaskHandler> {
struct NoWorkHandler;
#[async_trait::async_trait]
impl crank_runtime::TaskHandler for NoWorkHandler {
async fn revalidate(
&self,
_lease: &crank_core::TaskLease,
) -> Result<(), crank_runtime::TaskWorkerError> {
Ok(())
}
async fn execute(
&self,
_lease: &crank_core::TaskLease,
_attempt_id: &str,
_cancelled: std::sync::Arc<std::sync::atomic::AtomicBool>,
_permit: crank_runtime::TaskDispatchPermit,
) -> crank_runtime::TaskAttemptOutcome {
crank_runtime::TaskAttemptOutcome::Completed(serde_json::json!({"ok": true}))
}
}
std::sync::Arc::new(NoWorkHandler)
}
/// Story 4.3 end-to-end ATDD: the production durable-Task handler, the real runtime executor and a
/// real local HTTP upstream observe exactly one dispatch after the fenced checkpoint, while the
/// accepted parent trace is continued on the wire.
#[tokio::test]
async fn task_worker_dispatches_one_real_upstream_attempt_with_parent_trace() {
use axum::Router;
use axum::http::{HeaderMap, StatusCode};
use axum::routing::post;
use crank_community_mcp::TaskWorkerHandler;
use crank_core::{AgentOperationBinding, PlatformApiKeyScope};
use crank_runtime::{SecretCrypto, TaskWorker, TaskWorkerConfig};
use std::sync::atomic::{AtomicUsize, Ordering};
use std::sync::{Arc, Mutex};
let registry = common::test_registry().await;
let calls = Arc::new(AtomicUsize::new(0));
let seen_traceparent = Arc::new(Mutex::new(None::<String>));
let app = Router::new().route(
"/crm/leads",
post({
let calls = Arc::clone(&calls);
let seen = Arc::clone(&seen_traceparent);
move |headers: HeaderMap| {
let calls = Arc::clone(&calls);
let seen = Arc::clone(&seen);
async move {
calls.fetch_add(1, Ordering::SeqCst);
if let Some(value) = headers
.get("traceparent")
.and_then(|value| value.to_str().ok())
{
*seen.lock().unwrap() = Some(value.to_owned());
}
(StatusCode::OK, axum::Json(json!({"id": "lead-1"})))
}
}
}),
);
let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
let address = listener.local_addr().unwrap();
let server = tokio::spawn(async move { axum::serve(listener, app).await.unwrap() });
let upstream = format!("http://{address}");
let mut operation = common::test_operation(&upstream, TOOL_NAME);
operation.execution_config.execution_policy = Some(background_policy());
registry
.create_operation(&common::test_workspace_id(), &operation, Some("alice"))
.await
.unwrap();
publish(&registry, &operation.id, 1).await;
common::publish_agent_with_bindings(
&registry,
AGENT_SLUG,
vec![AgentOperationBinding {
operation_id: operation.id.clone(),
operation_version: 1,
tool_name: operation.name.clone(),
tool_title: operation.tool_description.title.clone(),
tool_description_override: None,
enabled: true,
}],
)
.await;
let caller = principal(
&registry,
"task-worker-e2e",
&[PlatformApiKeyScope::Read, PlatformApiKeyScope::Write],
)
.await;
let correlation = unsampled_correlation();
let crypto = SecretCrypto::new("test-master-key-00000000000000000000000000000000").unwrap();
let observed_at = OffsetDateTime::now_utc();
registry
.verify_or_register_master_key_identity(crank_registry::MasterKeyIdentityCandidate {
epoch: crypto.master_key_epoch(),
fingerprint: crypto.master_key_fingerprint(),
cipher_contract: crank_registry::MASTER_KEY_CIPHER_CONTRACT,
observed_at: &observed_at,
})
.await
.expect("test master-key identity registered");
let store = Arc::new(
PostgresTaskStore::new(
registry.pool().clone(),
AdmissionLimits {
global_limit: 4,
agent_limit: 4,
tool_limit: 4,
queue_limit: 8,
},
Default::default(),
)
.expect("valid task store configuration"),
);
let tasks = TaskApplication::new(
registry.clone(),
Duration::ZERO,
Arc::new(InMemoryCoordinationStateStore::default()),
TaskService::with_secret_crypto(store.clone(), Arc::new(crypto.clone())),
);
let acceptance = tasks
.accept_background(
&caller,
TOOL_NAME,
None,
"worker-e2e-key",
&json!({"email": "user@example.com"}),
&correlation,
)
.await
.expect("durable acceptance with executable input");
let runtime = crank_runtime::community_with_outbound_policy(
crank_runtime::OutboundHttpPolicy::allowing_hosts(["127.0.0.1"]),
)
.build();
let handler = Arc::new(TaskWorkerHandler::from_application(
&tasks,
registry.clone(),
runtime,
crypto,
));
let mut config = TaskWorkerConfig::default();
config.owner = "runtime-worker-e2e".to_owned();
config.batch_limit = 1;
config.lease_ms = 10_000;
config.poll_interval = Duration::from_millis(50);
config.max_inflight = 1;
config.shutdown_timeout = Duration::from_secs(5);
let worker = TaskWorker::new(store.clone(), handler, config).expect("valid worker");
let stop_flag = worker.stop_flag();
let worker_task = tokio::spawn(async move { worker.run().await });
let mut state = None;
for _ in 0..200 {
tokio::time::sleep(Duration::from_millis(50)).await;
let current: Option<String> =
sqlx::query_scalar("select state from tasks where task_id = $1")
.bind(acceptance.task_id.as_uuid())
.fetch_optional(registry.pool())
.await
.unwrap();
state = current;
if state.as_deref() == Some("completed") {
break;
}
}
stop_flag.store(true, Ordering::SeqCst);
let _ = tokio::time::timeout(Duration::from_secs(10), worker_task).await;
server.abort();
assert_eq!(state.as_deref(), Some("completed"));
assert_eq!(calls.load(Ordering::SeqCst), 1, "exactly one upstream call");
assert_eq!(
seen_traceparent.lock().unwrap().as_deref(),
Some(correlation.trace_context().traceparent()),
"the durable attempt continues the accepted parent trace"
);
}
/// Story 4.3 end-to-end ATDD: an ambiguous post-checkpoint upstream failure is persisted as
/// `outcome_unknown` (not a retryable failure) and is not redispatched automatically.
#[tokio::test]
async fn task_worker_ambiguous_upstream_error_becomes_outcome_unknown() {
use axum::Router;
use axum::http::StatusCode;
use axum::routing::post;
use crank_community_mcp::TaskWorkerHandler;
use crank_core::{AgentOperationBinding, PlatformApiKeyScope};
use crank_runtime::{SecretCrypto, TaskWorker, TaskWorkerConfig};
use std::sync::Arc;
use std::sync::atomic::Ordering;
let registry = common::test_registry().await;
let upstream_app = Router::new().route(
"/crm/leads",
post(|| async { (StatusCode::SERVICE_UNAVAILABLE, "upstream unavailable") }),
);
let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
let address = listener.local_addr().unwrap();
let server = tokio::spawn(async move { axum::serve(listener, upstream_app).await.unwrap() });
let upstream = format!("http://{address}");
let mut operation = common::test_operation(&upstream, TOOL_NAME);
operation.execution_config.execution_policy = Some(background_policy());
registry
.create_operation(&common::test_workspace_id(), &operation, Some("alice"))
.await
.unwrap();
publish(&registry, &operation.id, 1).await;
common::publish_agent_with_bindings(
&registry,
AGENT_SLUG,
vec![AgentOperationBinding {
operation_id: operation.id.clone(),
operation_version: 1,
tool_name: operation.name.clone(),
tool_title: operation.tool_description.title.clone(),
tool_description_override: None,
enabled: true,
}],
)
.await;
let caller = principal(
&registry,
"task-worker-uncertain",
&[PlatformApiKeyScope::Read, PlatformApiKeyScope::Write],
)
.await;
let correlation = unsampled_correlation();
let crypto = SecretCrypto::new("test-master-key-00000000000000000000000000000000").unwrap();
let observed_at = OffsetDateTime::now_utc();
registry
.verify_or_register_master_key_identity(crank_registry::MasterKeyIdentityCandidate {
epoch: crypto.master_key_epoch(),
fingerprint: crypto.master_key_fingerprint(),
cipher_contract: crank_registry::MASTER_KEY_CIPHER_CONTRACT,
observed_at: &observed_at,
})
.await
.expect("test master-key identity registered");
let store = Arc::new(
PostgresTaskStore::new(
registry.pool().clone(),
AdmissionLimits {
global_limit: 4,
agent_limit: 4,
tool_limit: 4,
queue_limit: 8,
},
Default::default(),
)
.expect("valid task store configuration"),
);
let tasks = TaskApplication::new(
registry.clone(),
Duration::ZERO,
Arc::new(InMemoryCoordinationStateStore::default()),
TaskService::with_secret_crypto(store.clone(), Arc::new(crypto.clone())),
);
let acceptance = tasks
.accept_background(
&caller,
TOOL_NAME,
None,
"worker-uncertain-key",
&json!({"email": "user@example.com"}),
&correlation,
)
.await
.expect("durable acceptance with executable input");
let runtime = crank_runtime::community_with_outbound_policy(
crank_runtime::OutboundHttpPolicy::allowing_hosts(["127.0.0.1"]),
)
.build();
let handler = Arc::new(TaskWorkerHandler::from_application(
&tasks,
registry.clone(),
runtime,
crypto,
));
let mut config = TaskWorkerConfig::default();
config.owner = "runtime-worker-uncertain".to_owned();
config.batch_limit = 1;
config.lease_ms = 10_000;
config.poll_interval = Duration::from_millis(50);
config.max_inflight = 1;
config.shutdown_timeout = Duration::from_secs(5);
let worker = TaskWorker::new(store.clone(), handler, config).expect("valid worker");
let stop_flag = worker.stop_flag();
let worker_task = tokio::spawn(async move { worker.run().await });
let mut state = None;
for _ in 0..200 {
tokio::time::sleep(Duration::from_millis(50)).await;
let current: Option<String> =
sqlx::query_scalar("select state from tasks where task_id = $1")
.bind(acceptance.task_id.as_uuid())
.fetch_optional(registry.pool())
.await
.unwrap();
state = current;
if state.as_deref() == Some("outcome_unknown") {
break;
}
}
stop_flag.store(true, Ordering::SeqCst);
let _ = tokio::time::timeout(Duration::from_secs(10), worker_task).await;
server.abort();
assert_eq!(state.as_deref(), Some("outcome_unknown"));
// No automatic retry: another authoritative scan never redispaches the uncertain attempt.
assert!(
store
.claim_batch("worker_after", 4, 10_000)
.await
.unwrap()
.is_empty()
);
}
/// Story 4.3 ATDD: graceful stop stops new claims but lets an active attempt reach its fenced
/// terminal commit within the 30s budget.
#[tokio::test]
async fn task_worker_graceful_stop_drains_active_attempt() {
use crank_core::{TaskAccess, TaskLease, TaskOwnerKind};
use crank_runtime::{
TaskAttemptOutcome, TaskDispatchPermit, TaskHandler, TaskWorker, TaskWorkerConfig,
TaskWorkerError,
};
use std::sync::Arc;
use std::sync::atomic::{AtomicUsize, Ordering};
let registry = common::test_registry().await;
let store = Arc::new(
PostgresTaskStore::new(
registry.pool().clone(),
AdmissionLimits {
global_limit: 4,
agent_limit: 4,
tool_limit: 4,
queue_limit: 8,
},
Default::default(),
)
.expect("valid task store configuration"),
);
let request = TaskAcceptRequest {
task_id: TaskId::generate(),
workspace_id: common::test_workspace_id().to_string(),
owner_kind: TaskOwnerKind::Agent,
owner_id: "agent_a".to_owned(),
authorization_scope: "credential:pk_drain".to_owned(),
platform_api_key_id: None,
operation_id: "op_report".to_owned(),
pins: TaskPins {
operation_version: 1,
policy_mode: ToolExecutionMode::Background,
execution_deadline_unix_ms: 1_900_000_000_000,
retain_until_unix_ms: 1_900_000_000_000 + 60_000,
approval_reference: None,
},
scoped_key: "s".repeat(64),
input_fingerprint: "d".repeat(64),
input: TaskEnvelope::try_redacted(&json!({"report": "q3"})).unwrap(),
support: TaskSupportIds::new(
RequestId::generate(),
&crank_core::TraceContext::parse(
"00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-00",
)
.unwrap(),
),
executable_input_ciphertext: None,
executable_input_key_version: None,
executable_input_master_key_epoch: None,
};
let accepted = store.accept(request).await.expect("PG accepts");
struct SlowHandler {
started: Arc<AtomicUsize>,
}
#[async_trait::async_trait]
impl TaskHandler for SlowHandler {
async fn revalidate(&self, _lease: &TaskLease) -> Result<(), TaskWorkerError> {
Ok(())
}
async fn execute(
&self,
_lease: &TaskLease,
_attempt_id: &str,
_cancelled: Arc<std::sync::atomic::AtomicBool>,
_permit: TaskDispatchPermit,
) -> TaskAttemptOutcome {
self.started.fetch_add(1, Ordering::SeqCst);
tokio::time::sleep(Duration::from_millis(500)).await;
TaskAttemptOutcome::Completed(json!({"ok": true}))
}
}
let started = Arc::new(AtomicUsize::new(0));
let handler: Arc<dyn TaskHandler> = Arc::new(SlowHandler {
started: started.clone(),
});
let mut config = TaskWorkerConfig::default();
config.owner = "runtime-worker-drain".to_owned();
config.batch_limit = 1;
config.lease_ms = 10_000;
config.poll_interval = Duration::from_millis(50);
config.max_inflight = 1;
config.shutdown_timeout = Duration::from_secs(20);
let worker = TaskWorker::new(store.clone(), handler, config).expect("valid worker");
let stop_flag = worker.stop_flag();
let worker_task = tokio::spawn(async move { worker.run().await });
for _ in 0..100 {
if started.load(Ordering::SeqCst) > 0 {
break;
}
tokio::time::sleep(Duration::from_millis(20)).await;
}
assert_eq!(started.load(Ordering::SeqCst), 1, "attempt started");
let stop_requested = std::time::Instant::now();
stop_flag.store(true, Ordering::SeqCst);
tokio::time::timeout(Duration::from_secs(30), worker_task)
.await
.expect("worker stops inside the 30s budget")
.expect("worker exits cleanly")
.expect("worker reports no error");
assert!(stop_requested.elapsed() <= Duration::from_secs(30));
let access = TaskAccess {
workspace_id: common::test_workspace_id().to_string(),
owner_kind: TaskOwnerKind::Agent,
owner_id: "agent_a".to_owned(),
authorization_scope: "credential:pk_drain".to_owned(),
};
let record = store.read(&access, accepted.task_id).await.unwrap();
assert_eq!(record.state, TaskState::Completed);
}
@@ -241,6 +241,11 @@ async fn inventory_projection_sidecar_is_provenance_only_and_names_the_projectio
assert_eq!(
triples,
vec![
(
"server".to_owned(),
"capability".to_owned(),
"logging".to_owned()
),
(
"server".to_owned(),
"capability".to_owned(),
@@ -262,7 +267,7 @@ async fn inventory_projection_sidecar_is_provenance_only_and_names_the_projectio
"tools".to_owned()
)
],
"the advertised server capabilities of the current scope are `completions`, `prompts`, `resources` and `tools`"
"the advertised server capabilities of the current scope are `logging`, `completions`, `prompts`, `resources` and `tools`"
);
// The sidecar is not the payload: byte comparison is against the frozen file only.
@@ -0,0 +1,856 @@
//! Story 3.18 acceptance tests: safe MCP server logging (AT-3.18-01..03, FR-19).
//!
//! Oracles: `docs/acceptance-test-plan.md` AT-3.18-01..03, `epics.md` Story 3.18 and the pinned
//! `logging`/`notifications/message`/`LoggingLevel` schemas of both supported revisions
//! (`2026-07-28` carries the level per request in `_meta`; `2025-11-25` serves `logging/setLevel`).
//! Every request travels through the real HTTP endpoint built by `build_test_app` over a real
//! PostgreSQL registry; the level is set through the production request path.
//!
//! The delivered events are observed on the SSE response stream, which is the only carrier the
//! pinned transport gives a server notification: a JSON body has no place for one. Where a pending
//! notification exists and the client offered `text/event-stream`, the negotiated mode is the stream.
//!
//! No external MCP client is exercised here: these are local harness results against the pinned wire
//! shapes, and the story handoff states that honestly.
use std::time::Duration;
use crank_core::PlatformApiKeyScope;
use crank_registry::PostgresRegistry;
use serde_json::{Value, json};
use super::common::{
agent_mcp_url, build_test_app, create_platform_api_key, initialize_session,
publish_agent_with_bindings, spawn_mcp_server, test_registry,
};
const MODERN_REVISION: &str = "2026-07-28";
const LEGACY_REVISION: &str = "2025-11-25";
const META_PROTOCOL_VERSION: &str = "io.modelcontextprotocol/protocolVersion";
const META_LOG_LEVEL: &str = "io.modelcontextprotocol/logLevel";
const META_CLIENT_CAPABILITIES: &str = "io.modelcontextprotocol/clientCapabilities";
const META_CLIENT_INFO: &str = "io.modelcontextprotocol/clientInfo";
const AGENT_SLUG: &str = "logging-agent";
const OTHER_AGENT_SLUG: &str = "logging-other";
/// The one safe wire detail of a malformed logging level.
const INVALID_LEVEL: &str = "invalid logging level";
/// The fixed logger name of a delivered notification.
const LOGGER_NAME: &str = "crank-mcp-server";
/// A value that must never appear in a delivered notification. It is distinctive on purpose.
const CANARY: &str = "crank-canary-9f7620e5-aa41-4b0d";
struct LoggingEndpoint {
client: reqwest::Client,
base_url: String,
api_key: String,
other_api_key: String,
registry: PostgresRegistry,
}
impl LoggingEndpoint {
fn url(&self, agent_slug: &str) -> String {
agent_mcp_url(&self.base_url, agent_slug)
}
/// One request whose response is negotiated as the SSE stream, so a pending notification really
/// has a carrier. The body is split into its notifications and the final JSON-RPC response.
async fn post_stream(
&self,
agent_slug: &str,
api_key: &str,
headers: &[(&str, &str)],
payload: Value,
) -> StreamReply {
let mut request = self
.client
.post(self.url(agent_slug))
.header(reqwest::header::ACCEPT, "text/event-stream")
.header(reqwest::header::AUTHORIZATION, format!("Bearer {api_key}"));
for (name, value) in headers {
request = request.header(*name, *value);
}
let response = request
.json(&payload)
.send()
.await
.expect("the MCP endpoint answers");
let status = response.status();
let content_type = response
.headers()
.get(reqwest::header::CONTENT_TYPE)
.and_then(|value| value.to_str().ok())
.unwrap_or_default()
.to_owned();
let body = response.text().await.expect("the stream body is readable");
StreamReply::parse(status, content_type, body)
}
/// One legacy request inside the initialized session of `agent_slug`.
async fn legacy_post(
&self,
agent_slug: &str,
api_key: &str,
session_id: &str,
payload: Value,
) -> StreamReply {
self.post_stream(
agent_slug,
api_key,
&[
("MCP-Protocol-Version", LEGACY_REVISION),
("MCP-Session-Id", session_id),
],
payload,
)
.await
}
/// One `logging/setLevel` request of the legacy era.
async fn set_level(&self, session_id: &str, level: Value) -> StreamReply {
self.legacy_post(
AGENT_SLUG,
&self.api_key,
session_id,
json!({
"jsonrpc": "2.0",
"id": 41,
"method": "logging/setLevel",
"params": {"level": level}
}),
)
.await
}
/// One modern request of an arbitrary method with the required per-request identity.
async fn modern_stream(&self, method: &str, params: Value) -> StreamReply {
let mut params = params;
params["_meta"] = json!({
META_PROTOCOL_VERSION: MODERN_REVISION,
META_CLIENT_CAPABILITIES: {},
META_CLIENT_INFO: {"name": "crank-logging-atdd", "version": "0.1.0"}
});
self.modern_stream_with_params(method, params).await
}
/// The same, with the caller's `params` (including a chosen `_meta`).
async fn modern_stream_with_params(&self, method: &str, params: Value) -> StreamReply {
self.post_stream(
AGENT_SLUG,
&self.api_key,
&[
("MCP-Protocol-Version", MODERN_REVISION),
("Mcp-Method", method),
],
json!({"jsonrpc": "2.0", "id": 51, "method": method, "params": params}),
)
.await
}
async fn legacy_session(&self, agent_slug: &str, api_key: &str) -> String {
initialize_session(&self.client, &self.url(agent_slug), api_key).await
}
}
struct StreamReply {
status: reqwest::StatusCode,
content_type: String,
/// The JSON-RPC response, which is the last event of the stream.
response: Value,
/// The `notifications/message` bodies delivered before it.
notifications: Vec<Value>,
/// The raw body, so a test can search it for a canary value.
raw: String,
}
impl StreamReply {
/// Parse a reply that may arrive either as an SSE stream (a pending notification needs the
/// stream) or as the plain JSON body of a method this era does not serve.
fn parse(status: reqwest::StatusCode, content_type: String, body: String) -> Self {
let mut messages = Vec::new();
for block in body.split("\n\n") {
for line in block.lines() {
let Some(data) = line.strip_prefix("data:") else {
continue;
};
let data = data.trim_start();
if data.is_empty() {
continue;
}
let value: Value = serde_json::from_str(data)
.unwrap_or_else(|error| panic!("every stream event is JSON: {data:?} {error}"));
messages.push(value);
}
}
if messages.is_empty() {
// An era-correct rejection of an unserved method is answered as a plain JSON body: it
// never carries a notification, so there is nothing to stream.
let response: Value = serde_json::from_str(&body)
.unwrap_or_else(|error| panic!("expected a JSON-RPC body, got {body:?}: {error}"));
return Self {
status,
content_type,
response,
notifications: Vec::new(),
raw: body,
};
}
let response = messages
.pop()
.unwrap_or_else(|| panic!("the stream must end with the JSON-RPC response: {body:?}"));
Self {
status,
content_type,
response,
notifications: messages,
raw: body,
}
}
fn result(&self) -> &Value {
self.response
.get("result")
.unwrap_or_else(|| panic!("expected a JSON-RPC result, got {}", self.response))
}
fn error(&self) -> (i64, String) {
let code = self
.response
.pointer("/error/code")
.and_then(Value::as_i64)
.unwrap_or_else(|| panic!("expected a JSON-RPC error, got {}", self.response));
let message = self
.response
.pointer("/error/message")
.and_then(Value::as_str)
.unwrap_or_default()
.to_owned();
(code, message)
}
fn assert_stream(&self) {
assert_eq!(self.status, reqwest::StatusCode::OK);
assert!(
self.content_type.starts_with("text/event-stream"),
"a streamed answer must carry the SSE content type, got {:?}",
self.content_type
);
}
}
/// The exact closed body of one delivered event. Nothing else is allowed to be carried.
fn expected_notification(level: &str, event: &str, outcome: &str) -> Value {
json!({
"jsonrpc": "2.0",
"method": "notifications/message",
"params": {
"level": level,
"logger": LOGGER_NAME,
"data": {"event": event, "outcome": outcome}
}
})
}
async fn logging_endpoint() -> LoggingEndpoint {
let registry = test_registry().await;
publish_agent_with_bindings(&registry, AGENT_SLUG, Vec::new()).await;
publish_agent_with_bindings(&registry, OTHER_AGENT_SLUG, Vec::new()).await;
let api_key = create_platform_api_key(
&registry,
AGENT_SLUG,
"mcp-logging",
&[PlatformApiKeyScope::Read, PlatformApiKeyScope::Write],
)
.await;
let other_api_key = create_platform_api_key(
&registry,
OTHER_AGENT_SLUG,
"mcp-logging-other",
&[PlatformApiKeyScope::Read, PlatformApiKeyScope::Write],
)
.await;
let base_url = spawn_mcp_server(build_test_app(
registry.clone(),
Duration::from_millis(0),
Some("https://crank.example.com".to_owned()),
))
.await;
LoggingEndpoint {
client: reqwest::Client::new(),
base_url,
api_key,
other_api_key,
registry,
}
}
/// AT-3.18-01/03: the advertisement equals the implemented surface on both revisions.
#[tokio::test]
async fn advertisement_declares_the_logging_capability_on_both_revisions() {
let endpoint = logging_endpoint().await;
let discover = endpoint
.client
.post(endpoint.url(AGENT_SLUG))
.header(
reqwest::header::ACCEPT,
"application/json, text/event-stream",
)
.header(
reqwest::header::AUTHORIZATION,
format!("Bearer {}", endpoint.api_key),
)
.header("MCP-Protocol-Version", MODERN_REVISION)
.header("Mcp-Method", "server/discover")
.json(&json!({
"jsonrpc": "2.0",
"id": 1,
"method": "server/discover",
"params": {"_meta": {
META_PROTOCOL_VERSION: MODERN_REVISION,
META_CLIENT_CAPABILITIES: {},
META_CLIENT_INFO: {"name": "crank-logging-atdd", "version": "0.1.0"}
}}
}))
.send()
.await
.expect("the endpoint answers");
let discover: Value = discover.json().await.expect("a JSON body");
assert_eq!(
discover["result"]["capabilities"]["logging"],
json!({}),
"the modern advertisement must declare the implemented logging capability: {discover}"
);
let initialize = endpoint
.client
.post(endpoint.url(AGENT_SLUG))
.header(
reqwest::header::ACCEPT,
"application/json, text/event-stream",
)
.header(
reqwest::header::AUTHORIZATION,
format!("Bearer {}", endpoint.api_key),
)
.json(&json!({
"jsonrpc": "2.0",
"id": 2,
"method": "initialize",
"params": {"protocolVersion": LEGACY_REVISION, "capabilities": {}}
}))
.send()
.await
.expect("the endpoint answers");
let initialize: Value = initialize.json().await.expect("a JSON body");
assert_eq!(
initialize["result"]["capabilities"]["logging"],
json!({}),
"the legacy advertisement must declare the implemented logging capability: {initialize}"
);
// Control: a capability this story does not implement is still absent, so the assertion above is
// not a blanket "advertise everything" change.
assert!(
initialize["result"]["capabilities"]
.get("io.modelcontextprotocol/tasks")
.is_none(),
"an unimplemented extension must not appear: {initialize}"
);
endpoint.registry.ping().await.expect("the registry is up");
}
/// AT-3.18-01: setting a valid legacy level is acknowledged with one safe structured event.
#[tokio::test]
async fn legacy_set_level_acknowledges_with_a_closed_safe_event() {
let endpoint = logging_endpoint().await;
let session = endpoint.legacy_session(AGENT_SLUG, &endpoint.api_key).await;
let reply = endpoint.set_level(&session, json!("debug")).await;
reply.assert_stream();
assert_eq!(
reply.result(),
&json!({}),
"the pinned `logging/setLevel` result is empty: {}",
reply.response
);
assert_eq!(
reply.notifications,
vec![
expected_notification("notice", "mcp.logging.level_applied", "applied"),
expected_notification("info", "mcp.request.served", "ok"),
],
"the acknowledgment carries the applied level and the served outcome, nothing else"
);
assert!(
reply.raw.len() < 4_096,
"one notification is a few hundred bytes, never a payload echo"
);
}
/// AT-3.18-01: after the opt-in, a served request delivers its own safe event only.
#[tokio::test]
async fn legacy_served_request_delivers_only_the_closed_safe_event() {
let endpoint = logging_endpoint().await;
let session = endpoint.legacy_session(AGENT_SLUG, &endpoint.api_key).await;
endpoint.set_level(&session, json!("debug")).await;
let reply = endpoint
.legacy_post(
AGENT_SLUG,
&endpoint.api_key,
&session,
json!({"jsonrpc": "2.0", "id": 61, "method": "tools/list", "params": {}}),
)
.await;
reply.assert_stream();
assert!(
reply.result().get("tools").is_some(),
"the business result is unchanged: {}",
reply.response
);
assert_eq!(
reply.notifications,
vec![expected_notification("info", "mcp.request.served", "ok")],
"a served request reports its outcome and nothing about the request"
);
}
/// AT-3.18-01 (redaction): a value the request carries can never reach a notification, while the
/// positive control proves the same search does find a reflected value elsewhere in the body.
#[tokio::test]
async fn a_request_value_can_never_reach_a_notification() {
let endpoint = logging_endpoint().await;
let session = endpoint.legacy_session(AGENT_SLUG, &endpoint.api_key).await;
endpoint.set_level(&session, json!("debug")).await;
let reply = endpoint
.legacy_post(
AGENT_SLUG,
&endpoint.api_key,
&session,
json!({
"jsonrpc": "2.0",
"id": 62,
"method": "tools/call",
"params": {
"name": CANARY,
"arguments": {"secret": CANARY},
"_meta": {"progressToken": CANARY}
}
}),
)
.await;
reply.assert_stream();
// Positive control: the existing product answer echoes the unknown tool name verbatim, so the
// canary is provably present in this very body — a "not found" search here would be meaningless.
let (code, message) = reply.error();
assert_eq!(code, -32602, "an unknown tool is a safe client error");
assert!(
message.contains(CANARY),
"the positive control must reflect the canary in the error message, got {message:?}"
);
// Positive control on delivery: the event is really delivered and its body is exactly the
// closed one for a rejected request, so the canary assertions below cannot pass vacuously.
assert_eq!(
reply.notifications,
vec![expected_notification(
"warning",
"mcp.request.rejected",
"invalid_params"
)],
"a rejected request must deliver exactly one safe warning event"
);
// The delivered notification is exactly the closed safe body: no canary, no argument value.
for notification in &reply.notifications {
assert_eq!(
notification["params"]["data"]
.as_object()
.map(|data| {
let mut keys: Vec<&str> = data.keys().map(String::as_str).collect();
keys.sort_unstable();
keys
})
.unwrap_or_default(),
vec!["event", "outcome"],
"a notification data object carries exactly the two closed members: {notification}"
);
assert!(
!notification.to_string().contains(CANARY),
"a request value must never reach a notification: {notification}"
);
assert!(
notification["params"].get("id").is_none(),
"a notification is not a response: {notification}"
);
}
}
/// AT-3.18-02/03: the modern era opts in per request, and only from the pinned `_meta` key.
#[tokio::test]
async fn modern_level_is_per_request_and_requires_an_explicit_opt_in() {
let endpoint = logging_endpoint().await;
let opted_in = endpoint
.modern_stream_with_params(
"tools/list",
json!({"_meta": {
META_PROTOCOL_VERSION: MODERN_REVISION,
META_CLIENT_CAPABILITIES: {},
META_CLIENT_INFO: {"name": "crank-logging-atdd", "version": "0.1.0"},
META_LOG_LEVEL: "debug"
}}),
)
.await;
opted_in.assert_stream();
assert!(opted_in.result().get("tools").is_some());
assert_eq!(
opted_in.notifications,
vec![expected_notification("info", "mcp.request.served", "ok")],
"a request that opted in receives its safe event"
);
// The pinned modern schema: without the key the server MUST NOT send any notification.
let not_opted_in = endpoint.modern_stream("tools/list", json!({})).await;
assert_eq!(not_opted_in.status, reqwest::StatusCode::OK);
assert!(
not_opted_in.result().get("tools").is_some(),
"the business answer is unchanged without an opt-in: {}",
not_opted_in.response
);
assert!(
not_opted_in.notifications.is_empty(),
"no notification without an explicit opt-in, got {:?}",
not_opted_in.notifications
);
// An unknown level is treated as "not opted in" rather than as an error of the business request,
// and it never widens the accepted vocabulary.
let unknown = endpoint
.modern_stream_with_params(
"tools/list",
json!({"_meta": {
META_PROTOCOL_VERSION: MODERN_REVISION,
META_CLIENT_CAPABILITIES: {},
META_CLIENT_INFO: {"name": "crank-logging-atdd", "version": "0.1.0"},
META_LOG_LEVEL: "verbose"
}}),
)
.await;
assert!(unknown.result().get("tools").is_some());
assert!(
unknown.notifications.is_empty(),
"an unpinned level must not opt in, got {:?}",
unknown.notifications
);
}
/// AT-3.18-03: the delivered severities follow the ranged level of the legacy session.
#[tokio::test]
async fn legacy_severity_filter_is_honoured() {
let endpoint = logging_endpoint().await;
let session = endpoint.legacy_session(AGENT_SLUG, &endpoint.api_key).await;
// `error` is more severe than both a served request and a rejected one: neither is delivered.
endpoint.set_level(&session, json!("error")).await;
let served = endpoint
.legacy_post(
AGENT_SLUG,
&endpoint.api_key,
&session,
json!({"jsonrpc": "2.0", "id": 71, "method": "tools/list", "params": {}}),
)
.await;
assert!(
served.notifications.is_empty(),
"a served event is below the `error` level, got {:?}",
served.notifications
);
let rejected = endpoint
.legacy_post(
AGENT_SLUG,
&endpoint.api_key,
&session,
json!({"jsonrpc": "2.0", "id": 72, "method": "prompts/get", "params": {}}),
)
.await;
assert_eq!(
rejected.error().0,
-32602,
"a malformed request is a client error"
);
assert!(
rejected.notifications.is_empty(),
"a warning event is below the `error` level, got {:?}",
rejected.notifications
);
// `warning` admits the rejected event, and still not the served one.
endpoint.set_level(&session, json!("warning")).await;
let rejected = endpoint
.legacy_post(
AGENT_SLUG,
&endpoint.api_key,
&session,
json!({"jsonrpc": "2.0", "id": 73, "method": "prompts/get", "params": {}}),
)
.await;
assert_eq!(
rejected.notifications,
vec![expected_notification(
"warning",
"mcp.request.rejected",
"invalid_params"
)],
"the rejection is delivered at the `warning` level"
);
let served = endpoint
.legacy_post(
AGENT_SLUG,
&endpoint.api_key,
&session,
json!({"jsonrpc": "2.0", "id": 74, "method": "tools/list", "params": {}}),
)
.await;
assert!(
served.notifications.is_empty(),
"an info event is still below the `warning` level, got {:?}",
served.notifications
);
}
/// AT-3.18-02: a malformed level is a revision-correct safe error with bounded work, and it never
/// changes the level of the client or of anybody else.
#[tokio::test]
async fn a_malformed_level_is_a_safe_error_without_state_change() {
let endpoint = logging_endpoint().await;
let session = endpoint.legacy_session(AGENT_SLUG, &endpoint.api_key).await;
endpoint.set_level(&session, json!("error")).await;
// A value outside the pinned union, a non-string shape, a missing member and an oversized value
// all share one safe detail: the client cannot tell them apart and nothing is echoed back.
for level in [
json!("verbose"),
json!("INFO"),
json!(7),
json!(null),
json!({}),
json!("i".repeat(64)),
] {
let reply = endpoint.set_level(&session, level.clone()).await;
assert_eq!(
reply.error(),
(-32602, INVALID_LEVEL.to_owned()),
"a malformed level is one safe invalid-params detail for {level}"
);
if let Some(text) = level.as_str() {
assert!(
!reply.raw.contains(text),
"the offending value must never be echoed back"
);
}
assert!(
reply.notifications.is_empty(),
"a rejected level is not an opt-in, got {:?}",
reply.notifications
);
}
// An oversized value is refused by the fixed bound: the answer stays a short safe detail and the
// value is not reflected, which is the observable half of "bounded work".
let oversized = "i".repeat(65_536);
let reply = endpoint.set_level(&session, json!(oversized)).await;
assert_eq!(reply.error(), (-32602, INVALID_LEVEL.to_owned()));
assert!(
!reply.raw.contains(&oversized),
"an oversized level must not be reflected"
);
assert!(
reply.raw.len() < 4_096,
"the answer to an oversized level is bounded, got {} bytes",
reply.raw.len()
);
let missing = endpoint
.legacy_post(
AGENT_SLUG,
&endpoint.api_key,
&session,
json!({"jsonrpc": "2.0", "id": 81, "method": "logging/setLevel", "params": {}}),
)
.await;
assert_eq!(missing.error(), (-32602, INVALID_LEVEL.to_owned()));
// The level of the session is unchanged (`error` was set before the malformed attempts), so a
// served request still delivers nothing...
let served = endpoint
.legacy_post(
AGENT_SLUG,
&endpoint.api_key,
&session,
json!({"jsonrpc": "2.0", "id": 82, "method": "tools/list", "params": {}}),
)
.await;
assert!(
served.notifications.is_empty(),
"a malformed level must not change the active level, got {:?}",
served.notifications
);
// ...and a valid level still works, which proves the assertions above are not vacuous.
endpoint.set_level(&session, json!("debug")).await;
let served = endpoint
.legacy_post(
AGENT_SLUG,
&endpoint.api_key,
&session,
json!({"jsonrpc": "2.0", "id": 83, "method": "tools/list", "params": {}}),
)
.await;
assert_eq!(served.notifications.len(), 1);
}
/// AT-3.18-02 (era matrix): the modern schema declares no `logging/setLevel`, so the same wire name
/// is a method-not-found there and never reaches the legacy service.
#[tokio::test]
async fn the_modern_era_has_no_set_level_method() {
let endpoint = logging_endpoint().await;
let reply = endpoint
.modern_stream("logging/setLevel", json!({"level": "debug"}))
.await;
let (code, message) = reply.error();
assert_eq!(
code, -32601,
"the modern revision has no logging/setLevel: {}",
reply.response
);
assert!(
!message.contains("debug"),
"the rejection must not echo the level: {message:?}"
);
// And the level was not applied through a wrong-era route: a modern request stays silent.
let after = endpoint.modern_stream("tools/list", json!({})).await;
assert!(after.notifications.is_empty());
}
/// AT-3.18-01/03: a method this era does not implement is a pinned client error, so the safe event
/// of an opted-in client describes a client-side rejection, never an internal server failure.
#[tokio::test]
async fn an_unknown_method_is_a_rejection_not_an_internal_failure() {
let endpoint = logging_endpoint().await;
let session = endpoint.legacy_session(AGENT_SLUG, &endpoint.api_key).await;
endpoint.set_level(&session, json!("info")).await;
// `server/discover` is a modern-era method, so the legacy adapter answers it with -32601.
let reply = endpoint
.legacy_post(
AGENT_SLUG,
&endpoint.api_key,
&session,
json!({
"jsonrpc": "2.0",
"id": 71,
"method": "server/discover",
"params": {}
}),
)
.await;
assert_eq!(
reply.error().0,
-32601,
"an unimplemented method is a pinned client error: {}",
reply.response
);
assert_eq!(
reply.notifications,
vec![expected_notification(
"warning",
"mcp.request.rejected",
"invalid_params"
)],
"a pinned client error must not be reported as an internal failure"
);
}
/// AT-3.18-01/03 (independence): a level belongs to one client context, so a second session and a
/// second Agent keep their own (or no) level.
#[tokio::test]
async fn a_level_never_leaks_to_another_client() {
let endpoint = logging_endpoint().await;
let session = endpoint.legacy_session(AGENT_SLUG, &endpoint.api_key).await;
endpoint.set_level(&session, json!("debug")).await;
// Another initialized session of the same Agent never opted in.
let other_session = endpoint.legacy_session(AGENT_SLUG, &endpoint.api_key).await;
assert_ne!(
session, other_session,
"each initialize mints its own session"
);
let other = endpoint
.legacy_post(
AGENT_SLUG,
&endpoint.api_key,
&other_session,
json!({"jsonrpc": "2.0", "id": 91, "method": "tools/list", "params": {}}),
)
.await;
assert!(
other.notifications.is_empty(),
"a session that never opted in must stay silent, got {:?}",
other.notifications
);
assert!(other.result().get("tools").is_some());
// The opted-in session still receives its own events...
let opted_in = endpoint
.legacy_post(
AGENT_SLUG,
&endpoint.api_key,
&session,
json!({"jsonrpc": "2.0", "id": 92, "method": "tools/list", "params": {}}),
)
.await;
assert_eq!(opted_in.notifications.len(), 1);
// ...and another Agent's session is unaffected by this Agent's level, both before and after it
// sets its own.
let other_agent_session = endpoint
.legacy_session(OTHER_AGENT_SLUG, &endpoint.other_api_key)
.await;
let other_agent = endpoint
.legacy_post(
OTHER_AGENT_SLUG,
&endpoint.other_api_key,
&other_agent_session,
json!({"jsonrpc": "2.0", "id": 93, "method": "tools/list", "params": {}}),
)
.await;
assert!(
other_agent.notifications.is_empty(),
"another Agent must not inherit a level, got {:?}",
other_agent.notifications
);
endpoint
.legacy_post(
OTHER_AGENT_SLUG,
&endpoint.other_api_key,
&other_agent_session,
json!({"jsonrpc": "2.0", "id": 94, "method": "logging/setLevel", "params": {"level": "error"}}),
)
.await;
let other_agent_error = endpoint
.legacy_post(
OTHER_AGENT_SLUG,
&endpoint.other_api_key,
&other_agent_session,
json!({"jsonrpc": "2.0", "id": 95, "method": "tools/list", "params": {}}),
)
.await;
assert!(
other_agent_error.notifications.is_empty(),
"the other Agent's own `error` level admits no served event"
);
}
@@ -23,7 +23,7 @@ async fn advertisement_declares_prompts_on_both_revisions() {
let discover = endpoint.modern("server/discover", json!({}), None).await;
assert_eq!(
discover["result"]["capabilities"]["prompts"],
json!({"listChanged": false}),
json!({"listChanged": true}),
"the modern advertisement must declare the prompts capability: {discover}"
);
@@ -50,7 +50,7 @@ async fn advertisement_declares_prompts_on_both_revisions() {
let initialize = json_body(initialize_response).await;
assert_eq!(
initialize["result"]["capabilities"]["prompts"],
json!({"listChanged": false}),
json!({"listChanged": true}),
"the legacy advertisement must declare the prompts capability: {initialize}"
);
}
@@ -0,0 +1,928 @@
//! Story 3.19 acceptance tests (AT-3.19-01..03, FR-19): committed Resource/Prompt catalog changes
//! are announced to a client that asked for the signal of its revision, and the ordinary list
//! methods stay authoritative when a notification is lost.
//!
//! Everything here is the production path: the real loopback MCP endpoint, the real PostgreSQL
//! registry, the real immutable artifact store and real commits through the registry API. No stubbed
//! route, no faked notification, no seeded notification table.
#![allow(dead_code)]
mod scenarios;
use std::{
fs,
path::PathBuf,
sync::atomic::{AtomicU64, Ordering},
time::Duration,
};
use axum::http::header;
use crank_artifacts::ArtifactStore;
use crank_core::{
Agent, AgentId, AgentStatus, AgentVersion, PlatformApiKeyScope, ToolAccessMode, ToolGroup,
ToolSearchSettings, ToolSelectionPolicy,
ids::{PromptId, ResourceId},
prompt::{PromptDefinition, PromptSensitivity},
};
use crank_registry::model::{
AgentPromptBinding, CreatePromptDraftRequest, PublishPromptRequest,
SaveAgentCatalogConfigRequest,
resource::{CreateResourceDraftRequest, PublishResourceVersionRequest},
};
use crank_registry::{
AgentResourceBinding, CreateAgentDraftVersionRequest, CreateAgentRequest, PostgresRegistry,
PublishAgentRequest, PublishRequest,
};
use serde_json::{Value, json};
use time::OffsetDateTime;
use super::common::*;
const MODERN_REVISION: &str = "2026-07-28";
const LEGACY_REVISION: &str = "2025-11-25";
const META_PROTOCOL_VERSION: &str = "io.modelcontextprotocol/protocolVersion";
const META_CLIENT_CAPABILITIES: &str = "io.modelcontextprotocol/clientCapabilities";
const META_CLIENT_INFO: &str = "io.modelcontextprotocol/clientInfo";
const RESOURCE_LIST_CHANGED: &str = "notifications/resources/list_changed";
const PROMPT_LIST_CHANGED: &str = "notifications/prompts/list_changed";
const TOOL_LIST_CHANGED: &str = "notifications/tools/list_changed";
const SUBSCRIPTIONS_ACKNOWLEDGED: &str = "notifications/subscriptions/acknowledged";
const SUBSCRIPTION_ID_KEY: &str = "io.modelcontextprotocol/subscriptionId";
const AGENT_SLUG: &str = "sales-signals";
/// A second published Agent of the same workspace: its catalog must never reach another Agent.
const OTHER_AGENT_SLUG: &str = "sales-other";
static NEXT_ROOT: AtomicU64 = AtomicU64::new(0);
/// Test-owned artifact root (0700), removed on drop.
struct TestRoot(PathBuf);
impl TestRoot {
fn new() -> Self {
let path = std::env::temp_dir().join(format!(
"crank-mcp-signals-{}-{}",
std::process::id(),
NEXT_ROOT.fetch_add(1, Ordering::Relaxed)
));
fs::create_dir_all(&path).expect("artifact root must be creatable");
let mut permissions = fs::metadata(&path).unwrap().permissions();
std::os::unix::fs::PermissionsExt::set_mode(&mut permissions, 0o700);
fs::set_permissions(&path, permissions).expect("artifact root must be private");
Self(path)
}
}
impl Drop for TestRoot {
fn drop(&mut self) {
let _ = fs::remove_dir_all(&self.0);
}
}
/// The published fixture: one Tool, two bound Prompts, one published but initially unbound Resource.
struct SignalsEndpoint {
client: reqwest::Client,
base_url: String,
mcp_url: String,
api_key: String,
registry: PostgresRegistry,
artifact_store: ArtifactStore,
root: TestRoot,
agent_id: AgentId,
operation: crank_core::Operation<crank_schema::Schema, crank_mapping::MappingSet>,
second_operation: crank_core::Operation<crank_schema::Schema, crank_mapping::MappingSet>,
greeting: PromptId,
review: PromptId,
extra_prompt: PromptId,
notice: ResourceId,
}
impl SignalsEndpoint {
fn modern_meta() -> Value {
json!({
META_PROTOCOL_VERSION: MODERN_REVISION,
META_CLIENT_CAPABILITIES: {},
META_CLIENT_INFO: { "name": "crank-signals-atdd", "version": "0.1.0" }
})
}
async fn modern_post(&self, payload: Value) -> reqwest::Response {
self.client
.post(&self.mcp_url)
.header(header::ACCEPT, "application/json, text/event-stream")
.header(header::AUTHORIZATION, format!("Bearer {}", self.api_key))
.json(&payload)
.send()
.await
.unwrap()
}
/// One modern request whose result is a JSON body (never a stream).
async fn modern(&self, method: &str, params: Value) -> Value {
let mut params = params;
params["_meta"] = Self::modern_meta();
let response = self
.client
.post(&self.mcp_url)
.header(header::ACCEPT, "application/json")
.header(header::AUTHORIZATION, format!("Bearer {}", self.api_key))
.header("MCP-Protocol-Version", MODERN_REVISION)
.header("Mcp-Method", method)
.json(&json!({"jsonrpc": "2.0", "id": 5, "method": method, "params": params}))
.send()
.await
.unwrap();
let status = response.status();
let body = response.text().await.unwrap();
serde_json::from_str(&body).unwrap_or_else(|error| {
panic!("expected JSON for {method}, {status} {body:?}: {error}")
})
}
/// One modern request that offers both response modes and is expected to answer a JSON body
/// (an admission or validation rejection, never a stream).
async fn modern_offering_stream(&self, method: &str, params: Value) -> Value {
let mut params = params;
params["_meta"] = Self::modern_meta();
let response = self
.client
.post(&self.mcp_url)
.header(header::ACCEPT, "application/json, text/event-stream")
.header(header::AUTHORIZATION, format!("Bearer {}", self.api_key))
.header("MCP-Protocol-Version", MODERN_REVISION)
.header("Mcp-Method", method)
.json(&json!({"jsonrpc": "2.0", "id": 9, "method": method, "params": params}))
.send()
.await
.unwrap();
assert_eq!(response.status(), reqwest::StatusCode::OK);
let body = response.text().await.unwrap();
serde_json::from_str(&body).unwrap_or_else(|error| {
panic!("expected a JSON rejection for {method}: {body:?} {error}")
})
}
/// One legacy request inside the initialized session of this Agent.
async fn legacy(&self, session: &str, method: &str, params: Value) -> Value {
let response = self
.client
.post(&self.mcp_url)
.header(header::ACCEPT, "application/json, text/event-stream")
.header(header::AUTHORIZATION, format!("Bearer {}", self.api_key))
.header("MCP-Protocol-Version", LEGACY_REVISION)
.header("MCP-Session-Id", session)
.json(&json!({"jsonrpc": "2.0", "id": 6, "method": method, "params": params}))
.send()
.await
.unwrap();
let status = response.status();
let body = response.text().await.unwrap();
serde_json::from_str(&body).unwrap_or_else(|error| {
panic!("expected JSON for {method}, {status} {body:?}: {error}")
})
}
async fn legacy_session(&self) -> String {
initialize_session(&self.client, &self.mcp_url, &self.api_key).await
}
/// The legacy notification stream (`GET`), consumed as an SSE response.
async fn legacy_stream(&self, session: &str) -> SseResponse {
let response = self
.client
.get(&self.mcp_url)
.header(header::ACCEPT, "text/event-stream")
.header(header::AUTHORIZATION, format!("Bearer {}", self.api_key))
.header("MCP-Protocol-Version", LEGACY_REVISION)
.header("MCP-Session-Id", session)
.send()
.await
.unwrap();
SseResponse::assert_open(response).await
}
/// Open the modern `subscriptions/listen` stream with an explicit filter.
async fn listen(&self, request_id: Value, filter: Value) -> SseResponse {
let params = json!({ "notifications": filter, "_meta": Self::modern_meta() });
let response = self
.client
.post(&self.mcp_url)
.header(header::ACCEPT, "text/event-stream")
.header(header::AUTHORIZATION, format!("Bearer {}", self.api_key))
.header("MCP-Protocol-Version", MODERN_REVISION)
.header("Mcp-Method", "subscriptions/listen")
.json(&json!({
"jsonrpc": "2.0",
"id": request_id,
"method": "subscriptions/listen",
"params": params
}))
.send()
.await
.unwrap();
SseResponse::assert_open(response).await
}
fn agent_version(&self, version: u32) -> AgentVersion {
AgentVersion {
agent_id: self.agent_id.clone(),
version,
status: AgentStatus::Draft,
instructions: json!({}),
tool_selection_policy: ToolSelectionPolicy::default(),
created_at: OffsetDateTime::now_utc(),
}
}
async fn open_draft(&self, version: u32) {
self.registry
.create_agent_draft_version(CreateAgentDraftVersionRequest {
workspace_id: &test_workspace_id(),
agent_id: &self.agent_id,
version: &self.agent_version(version),
bindings: &[],
updated_at: &OffsetDateTime::now_utc(),
})
.await
.expect("a new Agent draft version opens");
}
/// Write the exact catalog config of one draft version: Tool bindings, Resource bindings and
/// Prompt bindings in the single accepted transition.
async fn configure_draft(
&self,
version: u32,
tool_bindings: &[crank_core::AgentOperationBinding],
resource_bindings: &[AgentResourceBinding],
prompt_bindings: &[AgentPromptBinding],
) {
self.configure_draft_with_policy(
version,
tool_bindings,
&ToolSelectionPolicy::default(),
resource_bindings,
prompt_bindings,
)
.await;
}
/// The same transition with an explicit Tool selection policy (Story 3.5: the policy is part of
/// the served `tools/list` projection).
async fn configure_draft_with_policy(
&self,
version: u32,
tool_bindings: &[crank_core::AgentOperationBinding],
tool_selection_policy: &ToolSelectionPolicy,
resource_bindings: &[AgentResourceBinding],
prompt_bindings: &[AgentPromptBinding],
) {
self.registry
.save_agent_catalog_config_editable_with_prompts(
SaveAgentCatalogConfigRequest {
workspace_id: &test_workspace_id(),
agent_id: &self.agent_id,
agent_version: version,
bindings: tool_bindings,
tool_selection_policy,
resource_bindings,
expected_state: None,
},
prompt_bindings,
)
.await
.expect("the exact catalog config of the draft version saves");
}
async fn publish_version(&self, version: u32) {
self.registry
.publish_agent(PublishAgentRequest {
workspace_id: &test_workspace_id(),
agent_id: &self.agent_id,
version,
published_at: &OffsetDateTime::now_utc(),
published_by: Some("alice"),
expected_state: None,
})
.await
.expect("the draft Agent version publishes");
}
fn tool_bindings(&self) -> Vec<crank_core::AgentOperationBinding> {
vec![binding_for_operation(&self.operation)]
}
fn both_tool_bindings(&self) -> Vec<crank_core::AgentOperationBinding> {
vec![
binding_for_operation(&self.operation),
binding_for_operation(&self.second_operation),
]
}
fn base_prompts(&self) -> Vec<AgentPromptBinding> {
vec![
AgentPromptBinding {
prompt_id: self.greeting.clone(),
prompt_version: 1,
},
AgentPromptBinding {
prompt_id: self.review.clone(),
prompt_version: 1,
},
]
}
fn extended_prompts(&self) -> Vec<AgentPromptBinding> {
let mut prompts = self.base_prompts();
prompts.push(AgentPromptBinding {
prompt_id: self.extra_prompt.clone(),
prompt_version: 1,
});
prompts
}
fn notice_binding(&self) -> AgentResourceBinding {
AgentResourceBinding {
resource_id: self.notice.clone(),
resource_version: 1,
}
}
/// The draft version this Agent would open next, so the commit helpers do not depend on the
/// order the test calls them in.
async fn next_draft_version(&self) -> u32 {
self.registry
.get_agent_summary(&test_workspace_id(), &self.agent_id)
.await
.expect("the Agent summary reads")
.expect("the fixture Agent exists")
.current_draft_version
+ 1
}
/// The committed catalog change of AT-3.19-01: one Resource binding and one Prompt binding join
/// the published Agent snapshot in a single publication.
async fn commit_resource_and_prompt_change(&self) {
let version = self.next_draft_version().await;
self.open_draft(version).await;
self.configure_draft(
version,
&self.tool_bindings(),
&[self.notice_binding()],
&self.extended_prompts(),
)
.await;
self.publish_version(version).await;
}
/// A commit that only changes the Tool catalog: the visible Resource/Prompt projection is
/// identical, so it must not produce a Resource or Prompt signal.
async fn commit_tool_only_change(&self) {
let version = self.next_draft_version().await;
self.open_draft(version).await;
self.configure_draft(
version,
&self.both_tool_bindings(),
&[self.notice_binding()],
&self.extended_prompts(),
)
.await;
self.publish_version(version).await;
}
/// A commit that only changes the Resource catalog: the visible Prompt projection is identical,
/// so it must produce a Resource signal only.
async fn commit_resource_only_change(&self) {
let version = self.next_draft_version().await;
self.open_draft(version).await;
self.configure_draft(
version,
&self.tool_bindings(),
&[self.notice_binding()],
&self.base_prompts(),
)
.await;
self.publish_version(version).await;
}
/// A commit that only changes the Prompt catalog: the visible Resource projection is identical,
/// so it must produce a Prompt signal only.
async fn commit_prompt_only_change(&self) {
let version = self.next_draft_version().await;
self.open_draft(version).await;
self.configure_draft(
version,
&self.tool_bindings(),
&[self.notice_binding()],
&self.extended_prompts(),
)
.await;
self.publish_version(version).await;
}
/// A draft-only edit: nothing is published, so no committed catalog state changed.
async fn edit_draft_without_publishing(&self) {
let version = self.next_draft_version().await;
self.open_draft(version).await;
self.configure_draft(
version,
&self.both_tool_bindings(),
&[self.notice_binding()],
&self.base_prompts(),
)
.await;
}
/// The reverse Tool-only commit: the second Tool binding leaves the published snapshot while the
/// visible Resource/Prompt projection stays identical.
async fn commit_tool_only_revert(&self) {
let version = self.next_draft_version().await;
self.open_draft(version).await;
self.configure_draft(
version,
&self.tool_bindings(),
&[self.notice_binding()],
&self.extended_prompts(),
)
.await;
self.publish_version(version).await;
}
/// The search-mode Tool selection policy of this Agent: one group over the bound Tool.
fn search_policy(&self) -> ToolSelectionPolicy {
ToolSelectionPolicy {
mode: ToolAccessMode::Search,
groups: vec![ToolGroup {
id: "crm".to_owned(),
name: "CRM".to_owned(),
description: "CRM operations".to_owned(),
tool_names: vec![self.operation.name.clone()],
}],
search: ToolSearchSettings::default(),
}
}
/// A commit that changes only the Tool selection policy: same Tool, Resource and Prompt
/// bindings, direct mode -> search mode. The served `tools/list` changes from the bound Tool to
/// `search_tools`/`call_tool`, so this is a visible Tool catalog change.
async fn commit_policy_only_change(&self) {
let version = self.next_draft_version().await;
self.open_draft(version).await;
self.configure_draft_with_policy(
version,
&self.tool_bindings(),
&self.search_policy(),
&[],
&self.base_prompts(),
)
.await;
self.publish_version(version).await;
}
/// A search-mode commit that only adds a Tool binding: `tools/list` still serves exactly
/// `search_tools`/`call_tool`, so it must not announce a Tools change.
async fn commit_search_mode_binding_change(&self) {
let version = self.next_draft_version().await;
self.open_draft(version).await;
self.configure_draft_with_policy(
version,
&self.both_tool_bindings(),
&self.search_policy(),
&[],
&self.base_prompts(),
)
.await;
self.publish_version(version).await;
}
/// Publish an independent Agent of the same workspace, bound to the first fixture operation, and
/// return its read-only MCP client key.
async fn publish_other_agent(&self, slug: &str) -> String {
let agent_id = test_agent_id(slug);
let now = OffsetDateTime::now_utc();
self.registry
.create_agent(CreateAgentRequest {
agent: &Agent {
id: agent_id.clone(),
workspace_id: test_workspace_id(),
slug: slug.to_owned(),
display_name: slug.to_owned(),
description: "Story 3.5 isolation fixture".to_owned(),
status: AgentStatus::Draft,
current_draft_version: 1,
latest_published_version: None,
created_at: now,
updated_at: now,
published_at: None,
},
version: &AgentVersion {
agent_id: agent_id.clone(),
version: 1,
status: AgentStatus::Draft,
instructions: json!({}),
tool_selection_policy: ToolSelectionPolicy::default(),
created_at: now,
},
bindings: &[binding_for_operation(&self.operation)],
})
.await
.expect("the second Agent is created");
self.registry
.publish_agent(PublishAgentRequest {
workspace_id: &test_workspace_id(),
agent_id: &agent_id,
version: 1,
published_at: &OffsetDateTime::now_utc(),
published_by: Some("alice"),
expected_state: None,
})
.await
.expect("the second Agent publishes");
create_platform_api_key(
&self.registry,
slug,
"mcp-signals-other",
&[PlatformApiKeyScope::Read],
)
.await
}
/// A committed Tool-only change of the independent Agent: it binds the second fixture Tool.
async fn commit_other_agent_tool_change(&self, slug: &str) {
let agent_id = test_agent_id(slug);
let version = AgentVersion {
agent_id: agent_id.clone(),
version: 2,
status: AgentStatus::Draft,
instructions: json!({}),
tool_selection_policy: ToolSelectionPolicy::default(),
created_at: OffsetDateTime::now_utc(),
};
self.registry
.create_agent_draft_version(CreateAgentDraftVersionRequest {
workspace_id: &test_workspace_id(),
agent_id: &agent_id,
version: &version,
bindings: &[],
updated_at: &OffsetDateTime::now_utc(),
})
.await
.expect("a second draft version of the other Agent opens");
self.registry
.save_agent_catalog_config_editable_with_prompts(
SaveAgentCatalogConfigRequest {
workspace_id: &test_workspace_id(),
agent_id: &agent_id,
agent_version: 2,
bindings: &self.both_tool_bindings(),
tool_selection_policy: &ToolSelectionPolicy::default(),
resource_bindings: &[],
expected_state: None,
},
&[],
)
.await
.expect("the other Agent's second catalog config saves");
self.registry
.publish_agent(PublishAgentRequest {
workspace_id: &test_workspace_id(),
agent_id: &agent_id,
version: 2,
published_at: &OffsetDateTime::now_utc(),
published_by: Some("alice"),
expected_state: None,
})
.await
.expect("the other Agent's second version publishes");
}
}
/// A live SSE response, read event by event.
struct SseResponse {
response: reqwest::Response,
buffer: String,
}
impl SseResponse {
async fn assert_open(response: reqwest::Response) -> Self {
assert_eq!(
response.status(),
reqwest::StatusCode::OK,
"the notification stream must be served"
);
let content_type = response
.headers()
.get(header::CONTENT_TYPE)
.and_then(|value| value.to_str().ok())
.unwrap_or_default()
.to_owned();
assert!(
content_type.starts_with("text/event-stream"),
"the notification stream must be an SSE response, got {content_type}"
);
Self {
response,
buffer: String::new(),
}
}
/// The payload of the next `data:` frame, or `None` when nothing arrived within `timeout`.
async fn next_data(&mut self, timeout: Duration) -> Option<Value> {
loop {
if let Some(index) = self.buffer.find("\n\n") {
let block: String = self.buffer.drain(..index + 2).collect();
for line in block.lines() {
if let Some(data) = line.strip_prefix("data:") {
return Some(
serde_json::from_str(data.trim_start())
.unwrap_or_else(|error| panic!("SSE frame must be JSON: {error}")),
);
}
}
continue;
}
let chunk = tokio::time::timeout(timeout, self.response.chunk())
.await
.ok()?
.ok()??;
self.buffer.push_str(&String::from_utf8_lossy(&chunk));
}
}
/// The next frame within the watcher's bound.
async fn expect_frame(&mut self, what: &str) -> Value {
self.next_data(Duration::from_secs(6))
.await
.unwrap_or_else(|| panic!("expected {what} on the notification stream"))
}
/// Exactly one safe negative window: no frame within `wait`.
async fn expect_silence(&mut self, wait: Duration, what: &str) {
if let Some(frame) = self.next_data(wait).await {
panic!("expected no {what}, got {frame}");
}
}
}
async fn publish_prompt(registry: &PostgresRegistry, prompt_id: &PromptId, name: &str) -> PromptId {
let definition = PromptDefinition::parse(&json!({
"name": name,
"description": format!("{name} fixture"),
"arguments": [],
"messages": [{"role": "user", "content": format!("Use {name}.")}]
}))
.expect("the fixture Prompt definition is valid");
let created = registry
.create_prompt_draft(CreatePromptDraftRequest {
workspace_id: test_workspace_id(),
prompt_id: prompt_id.clone(),
sensitivity: PromptSensitivity::Internal,
definition,
created_at: OffsetDateTime::now_utc(),
})
.await
.expect("the Prompt Draft is created");
registry
.publish_prompt_draft(PublishPromptRequest {
workspace_id: test_workspace_id(),
prompt_id: prompt_id.clone(),
expected_updated_at: created.updated_at,
published_at: OffsetDateTime::now_utc(),
})
.await
.expect("the Prompt version publishes");
prompt_id.clone()
}
/// Publish one static Resource through the production 3.6/3.7 registry transition.
async fn publish_resource(
registry: &PostgresRegistry,
store: &ArtifactStore,
resource_id: &ResourceId,
bytes: &[u8],
) {
let artifact = store
.put_registered(bytes)
.expect("the fixture bytes are registered durably");
registry
.create_resource_draft(CreateResourceDraftRequest {
workspace_id: test_workspace_id(),
resource_id: resource_id.clone(),
uri: format!("crank://resources/{}", resource_id.as_str()),
name: resource_id.as_str().to_owned(),
description: "Story 3.19 signal fixture".to_owned(),
kind: "text".to_owned(),
mime_type: "text/plain".to_owned(),
sensitivity: "internal".to_owned(),
artifact,
created_at: OffsetDateTime::now_utc(),
})
.await
.expect("the Resource Draft is created");
let current = registry
.get_resource_draft(test_workspace_id().as_str(), resource_id)
.await
.expect("the Resource Draft reads");
registry
.publish_resource_version(PublishResourceVersionRequest {
workspace_id: test_workspace_id(),
resource_id: resource_id.clone(),
expected_updated_at: current.resource.updated_at,
expected_version: current.version.version,
expected_digest: current.version.digest.clone(),
expected_size_bytes: current.version.size_bytes,
artifact: store
.put_registered(bytes)
.expect("the published bytes are registered durably"),
published_at: OffsetDateTime::now_utc(),
})
.await
.expect("the Resource version publishes");
}
async fn signals_endpoint() -> SignalsEndpoint {
let root = TestRoot::new();
let artifact_store = ArtifactStore::open(root.0.clone()).expect("the artifact store opens");
let registry = test_registry().await;
let upstream_base_url = spawn_upstream_server().await;
let operation = test_operation(&upstream_base_url, "crm_create_lead");
registry
.create_operation(&test_workspace_id(), &operation, Some("alice"))
.await
.unwrap();
registry
.publish_operation(PublishRequest {
workspace_id: &test_workspace_id(),
operation_id: &operation.id,
version: 1,
published_at: &OffsetDateTime::now_utc(),
published_by: Some("alice"),
})
.await
.unwrap();
let second_operation = test_operation(&upstream_base_url, "crm_create_second");
registry
.create_operation(&test_workspace_id(), &second_operation, Some("alice"))
.await
.unwrap();
registry
.publish_operation(PublishRequest {
workspace_id: &test_workspace_id(),
operation_id: &second_operation.id,
version: 1,
published_at: &OffsetDateTime::now_utc(),
published_by: Some("alice"),
})
.await
.unwrap();
let greeting = publish_prompt(&registry, &PromptId::new("prm_a_greeting"), "greeting").await;
let review = publish_prompt(&registry, &PromptId::new("prm_b_review"), "review").await;
let extra_prompt = publish_prompt(&registry, &PromptId::new("prm_c_extra"), "extra").await;
let notice = ResourceId::new("res_a_notice");
publish_resource(&registry, &artifact_store, &notice, b"first notice").await;
let agent_id = AgentId::new(format!("agent_{AGENT_SLUG}"));
let now = OffsetDateTime::now_utc();
registry
.create_agent(CreateAgentRequest {
agent: &Agent {
id: agent_id.clone(),
workspace_id: test_workspace_id(),
slug: AGENT_SLUG.to_owned(),
display_name: AGENT_SLUG.to_owned(),
description: "Story 3.19 signal fixture".to_owned(),
status: AgentStatus::Draft,
current_draft_version: 1,
latest_published_version: None,
created_at: now,
updated_at: now,
published_at: None,
},
version: &AgentVersion {
agent_id: agent_id.clone(),
version: 1,
status: AgentStatus::Draft,
instructions: json!({}),
tool_selection_policy: ToolSelectionPolicy::default(),
created_at: now,
},
bindings: &[binding_for_operation(&operation)],
})
.await
.expect("the draft Agent is created");
registry
.save_agent_catalog_config_editable_with_prompts(
SaveAgentCatalogConfigRequest {
workspace_id: &test_workspace_id(),
agent_id: &agent_id,
agent_version: 1,
bindings: &[binding_for_operation(&operation)],
tool_selection_policy: &ToolSelectionPolicy::default(),
resource_bindings: &[],
expected_state: None,
},
&[
AgentPromptBinding {
prompt_id: greeting.clone(),
prompt_version: 1,
},
AgentPromptBinding {
prompt_id: review.clone(),
prompt_version: 1,
},
],
)
.await
.expect("the exact bound Prompt versions are accepted");
registry
.publish_agent(PublishAgentRequest {
workspace_id: &test_workspace_id(),
agent_id: &agent_id,
version: 1,
published_at: &OffsetDateTime::now_utc(),
published_by: Some("alice"),
expected_state: None,
})
.await
.expect("the bound Agent publishes");
let api_key = create_platform_api_key(
&registry,
AGENT_SLUG,
"mcp-signals",
&[PlatformApiKeyScope::Read, PlatformApiKeyScope::Write],
)
.await;
let base_url = spawn_mcp_server(build_test_app_with_artifacts(
registry.clone(),
Duration::from_millis(0),
None,
artifact_store.clone(),
))
.await;
SignalsEndpoint {
client: reqwest::Client::new(),
mcp_url: agent_mcp_url(&base_url, AGENT_SLUG),
base_url,
api_key,
registry,
artifact_store,
root,
agent_id,
operation,
second_operation,
greeting,
review,
extra_prompt,
notice,
}
}
fn prompt_names(list: &Value) -> Vec<String> {
list["result"]["prompts"]
.as_array()
.unwrap_or_else(|| panic!("prompts/list must return a page: {list}"))
.iter()
.map(|prompt| prompt["name"].as_str().unwrap().to_owned())
.collect()
}
fn resource_uris(list: &Value) -> Vec<String> {
list["result"]["resources"]
.as_array()
.unwrap_or_else(|| panic!("resources/list must return a page: {list}"))
.iter()
.map(|resource| resource["uri"].as_str().unwrap().to_owned())
.collect()
}
fn tool_names(list: &Value) -> Vec<String> {
let mut names: Vec<String> = list["result"]["tools"]
.as_array()
.unwrap_or_else(|| panic!("tools/list must return a page: {list}"))
.iter()
.map(|tool| tool["name"].as_str().unwrap().to_owned())
.collect();
names.sort();
names
}
/// Write a stand file that only the current user may read (credential files live here).
fn write_private(path: &PathBuf, contents: &str) {
use std::io::Write as _;
use std::os::unix::fs::OpenOptionsExt as _;
let mut file = fs::OpenOptions::new()
.create(true)
.truncate(true)
.write(true)
.mode(0o600)
.open(path)
.unwrap_or_else(|error| panic!("stand file {path:?} must be writable: {error}"));
file.write_all(contents.as_bytes())
.expect("the stand file must be written");
}
@@ -0,0 +1,791 @@
//! Story 3.19 acceptance scenarios (AT-3.19-01..03, FR-19) on the harness in the parent module.
//!
//! Moved out of `mcp_signals.rs` verbatim (code-health split, 1000-line limit); no scenario,
//! assertion or fixture changed.
use super::*;
/// Loopback disposable stand for the AC-3 representative-client evidence. This is deliberately not
/// part of the automated ATDD: it publishes the fixture catalog, serves the real MCP endpoint on
/// `127.0.0.1`, and then performs one commit per command file so an external MCP client can observe
/// the signals on its own stream.
///
/// Run it explicitly (the credential file is `0600` and never printed):
///
/// ```text
/// MCP_SIGNALS_STAND_DIR=/tmp/crank-signals-stand \
/// cargo test -p mcp-server --test integration mcp_signals::scenarios::representative_client_stand \
/// -- --ignored --exact --nocapture
/// ```
///
/// Commands (create the file, wait for `<command>.done`): `commit-both`, `commit-prompt`,
/// `commit-resource`, `commit-tool`, `draft`, `stop`.
#[tokio::test]
#[ignore = "manual loopback stand for the representative MCP Inspector client evidence"]
async fn representative_client_stand() {
let directory = PathBuf::from(
std::env::var("MCP_SIGNALS_STAND_DIR")
.expect("MCP_SIGNALS_STAND_DIR names the disposable stand directory"),
);
fs::create_dir_all(&directory).expect("the stand directory is creatable");
let endpoint = signals_endpoint().await;
write_private(
&directory.join("stand.json"),
&json!({ "url": endpoint.mcp_url, "token": endpoint.api_key }).to_string(),
);
println!("stand ready: {}", directory.join("stand.json").display());
let deadline = tokio::time::Instant::now() + Duration::from_secs(900);
while tokio::time::Instant::now() < deadline {
for command in [
"commit-both",
"commit-prompt",
"commit-resource",
"commit-tool",
"draft",
"stop",
] {
let marker = directory.join(command);
if !marker.exists() {
continue;
}
fs::remove_file(&marker).expect("the command file is consumed once");
if command == "stop" {
println!("stand stopped: {command}");
return;
}
match command {
"commit-both" => endpoint.commit_resource_and_prompt_change().await,
"commit-prompt" => endpoint.commit_prompt_only_change().await,
"commit-resource" => endpoint.commit_resource_only_change().await,
"commit-tool" => endpoint.commit_tool_only_change().await,
_ => endpoint.edit_draft_without_publishing().await,
}
write_private(&directory.join(format!("{command}.done")), "done");
println!("stand committed: {command}");
}
tokio::time::sleep(Duration::from_millis(100)).await;
}
}
/// AT-3.19-01 (legacy): an open notification stream receives the Resource and Prompt list-change
/// notifications only after the catalog commit, and `resources/list` / `prompts/list` serve the new
/// state afterwards.
#[tokio::test]
async fn legacy_stream_announces_a_committed_catalog_change_after_the_commit() {
let endpoint = signals_endpoint().await;
let session = endpoint.legacy_session().await;
let mut stream = endpoint.legacy_stream(&session).await;
// The watcher takes its baseline before it can report anything; let it settle before the commit.
tokio::time::sleep(Duration::from_millis(1_500)).await;
stream
.expect_silence(Duration::from_millis(1_200), "signal before any commit")
.await;
// A draft-only edit is not a commit: nothing is announced.
endpoint.edit_draft_without_publishing().await;
stream
.expect_silence(Duration::from_millis(2_500), "signal for a draft-only edit")
.await;
endpoint.commit_resource_and_prompt_change().await;
let mut methods = Vec::new();
for _ in 0..2 {
let frame = stream
.expect_frame("the committed list-change signals")
.await;
assert_eq!(frame["jsonrpc"], json!("2.0"));
assert!(
frame.get("params").is_none(),
"a legacy list-change notification carries no parameters: {frame}"
);
methods.push(frame["method"].as_str().unwrap().to_owned());
}
methods.sort();
assert_eq!(
methods,
vec![
PROMPT_LIST_CHANGED.to_owned(),
RESOURCE_LIST_CHANGED.to_owned()
],
"one signal per changed catalog type"
);
// The ordinary list methods are authoritative for the same committed state.
let prompts = endpoint.legacy(&session, "prompts/list", json!({})).await;
assert!(
prompt_names(&prompts).contains(&"extra".to_owned()),
"prompts/list must serve the newly bound Prompt: {prompts}"
);
let resources = endpoint.legacy(&session, "resources/list", json!({})).await;
assert!(
resource_uris(&resources)
.iter()
.any(|uri| uri == &format!("crank://resources/{}", endpoint.notice.as_str())),
"resources/list must serve the newly bound Resource: {resources}"
);
// A Tool-only commit keeps the Resource/Prompt projection: only a Tools frame follows (Story 3.5).
endpoint.commit_tool_only_change().await;
let frame = stream
.expect_frame("the committed Tools list-change signal")
.await;
assert_eq!(frame["method"], TOOL_LIST_CHANGED);
assert!(
frame.get("params").is_none(),
"a legacy list-change notification carries no parameters: {frame}"
);
stream
.expect_silence(
Duration::from_millis(2_500),
"a Resource or Prompt signal for a Tool-only commit",
)
.await;
}
/// AT-3.19-01/03 (modern): `subscriptions/listen` acknowledges the honoured filter with the request
/// id, then delivers coalesced list-change notifications carrying the subscription id.
#[tokio::test]
async fn modern_listen_stream_acknowledges_then_announces_coalesced_changes() {
let endpoint = signals_endpoint().await;
let mut stream = endpoint
.listen(
json!(41),
json!({
"resourcesListChanged": true,
"promptsListChanged": true,
"toolsListChanged": true,
"resourceSubscriptions": ["crank://resources/res_a_notice"]
}),
)
.await;
let acknowledgement = stream.expect_frame("the acknowledgment").await;
assert_eq!(acknowledgement["method"], SUBSCRIPTIONS_ACKNOWLEDGED);
assert_eq!(
acknowledgement["params"]["_meta"][SUBSCRIPTION_ID_KEY],
json!(41)
);
assert_eq!(
acknowledgement["params"]["notifications"],
json!({
"promptsListChanged": true,
"resourcesListChanged": true,
"toolsListChanged": true
}),
"only the implemented list-change types are acknowledged: {acknowledgement}"
);
tokio::time::sleep(Duration::from_millis(1_500)).await;
// Two commits in quick succession coalesce into one signal per changed type.
endpoint.commit_resource_and_prompt_change().await;
endpoint.commit_tool_only_change().await;
let mut methods = Vec::new();
for _ in 0..3 {
let frame = stream
.expect_frame("the coalesced list-change signal")
.await;
assert_eq!(frame["params"]["_meta"][SUBSCRIPTION_ID_KEY], json!(41));
methods.push(frame["method"].as_str().unwrap().to_owned());
}
methods.sort();
assert_eq!(
methods,
vec![
PROMPT_LIST_CHANGED.to_owned(),
RESOURCE_LIST_CHANGED.to_owned(),
TOOL_LIST_CHANGED.to_owned()
]
);
stream
.expect_silence(
Duration::from_millis(2_500),
"a duplicate or unrequested signal",
)
.await;
// A client that missed every notification still recovers the authoritative catalog.
let prompts = endpoint.modern("prompts/list", json!({})).await;
assert!(
prompt_names(&prompts).contains(&"extra".to_owned()),
"{prompts}"
);
}
/// AT-3.19-01/03 (modern): the acknowledged filter is the delivery contract. A prompts-only stream
/// stays silent for a Resource-only commit, delivers exactly one Prompt frame with the subscription
/// id for a Prompt commit, and never delivers an unrequested type.
#[tokio::test]
async fn modern_listen_delivers_only_the_negotiated_signal_types() {
let endpoint = signals_endpoint().await;
let mut stream = endpoint
.listen(
json!("prompts-only"),
json!({
"promptsListChanged": true,
"toolsListChanged": true,
"resourceSubscriptions": ["crank://resources/res_a_notice"]
}),
)
.await;
let acknowledgement = stream.expect_frame("the acknowledgment").await;
assert_eq!(acknowledgement["method"], SUBSCRIPTIONS_ACKNOWLEDGED);
assert_eq!(
acknowledgement["params"]["_meta"][SUBSCRIPTION_ID_KEY],
json!("prompts-only")
);
assert_eq!(
acknowledgement["params"]["notifications"],
json!({ "promptsListChanged": true, "toolsListChanged": true }),
"only the implemented and requested types are acknowledged: {acknowledgement}"
);
// The watcher takes its baseline before it can report anything.
tokio::time::sleep(Duration::from_millis(1_500)).await;
// A Resource-only commit is outside the negotiated filter: a server that ignored the filter would
// emit a Resource frame here.
endpoint.commit_resource_only_change().await;
stream
.expect_silence(
Duration::from_millis(2_500),
"signal for a Resource-only commit on a prompts-only stream",
)
.await;
// A Prompt-only commit is inside it: exactly one frame, correlated by the subscription id.
endpoint.commit_prompt_only_change().await;
let frame = stream
.expect_frame("the negotiated Prompt list-change signal")
.await;
assert_eq!(frame["method"], PROMPT_LIST_CHANGED);
assert_eq!(
frame["params"]["_meta"][SUBSCRIPTION_ID_KEY],
json!("prompts-only")
);
// No Tool commit happened, so no Tools frame follows; `resourceSubscriptions` was not honoured.
stream
.expect_silence(
Duration::from_millis(2_500),
"a duplicate, Tools or URI-content signal",
)
.await;
}
/// AT-3.19-03: the unsupported URI-content subscription surface is never acknowledged, and the
/// legacy revision has no `subscriptions/listen` request at all.
#[tokio::test]
async fn unsupported_subscription_surfaces_are_refused_rather_than_faked() {
let endpoint = signals_endpoint().await;
for method in ["resources/subscribe", "resources/unsubscribe"] {
let reply = endpoint
.modern(method, json!({ "uri": "crank://resources/res_a_notice" }))
.await;
assert_eq!(
reply["error"]["code"],
json!(-32601),
"{method} must stay unimplemented: {reply}"
);
}
// The legacy revision opens its notification stream with GET only.
let session = endpoint.legacy_session().await;
let legacy_listen = endpoint
.legacy(
&session,
"subscriptions/listen",
json!({ "notifications": {} }),
)
.await;
assert_eq!(
legacy_listen["error"]["code"],
json!(-32601),
"subscriptions/listen is a modern-only request: {legacy_listen}"
);
// A malformed filter is a bounded invalid-params answer with no echoed value.
let malformed = endpoint
.modern_offering_stream(
"subscriptions/listen",
json!({ "notifications": ["promptsListChanged"] }),
)
.await;
assert_eq!(malformed["error"]["code"], json!(-32602));
assert_eq!(
malformed["error"]["message"],
json!("invalid subscriptions/listen notification filter")
);
// Without an SSE-capable client the stream cannot be served at all.
let without_sse = endpoint
.client
.post(&endpoint.mcp_url)
.header(header::ACCEPT, "application/json")
.header(
header::AUTHORIZATION,
format!("Bearer {}", endpoint.api_key),
)
.header("MCP-Protocol-Version", MODERN_REVISION)
.header("Mcp-Method", "subscriptions/listen")
.json(&json!({
"jsonrpc": "2.0",
"id": 42,
"method": "subscriptions/listen",
"params": {
"notifications": { "promptsListChanged": true },
"_meta": SignalsEndpoint::modern_meta()
}
}))
.send()
.await
.unwrap();
assert_eq!(without_sse.status(), reqwest::StatusCode::NOT_ACCEPTABLE);
}
/// AT-3.19-02: a missed notification never loses committed state — a legacy client re-initializes
/// (the retired session is a safe `404`) and reads the authoritative catalog, while the modern
/// stateless client recovers through `server/discover` and the list methods.
#[tokio::test]
async fn a_lost_notification_is_recovered_through_the_authoritative_methods() {
let endpoint = signals_endpoint().await;
// Modern recovery: the commit happens with no stream open at all.
endpoint.commit_resource_and_prompt_change().await;
let discover = endpoint.modern("server/discover", json!({})).await;
assert_eq!(
discover["result"]["capabilities"]["prompts"]["listChanged"],
json!(true)
);
assert_eq!(
discover["result"]["capabilities"]["resources"]["listChanged"],
json!(true)
);
assert_eq!(
discover["result"]["capabilities"]["tools"]["listChanged"],
json!(true),
"the Tools list-change capability is advertised: {discover}"
);
let resources = endpoint.modern("resources/list", json!({})).await;
assert!(
resource_uris(&resources)
.iter()
.any(|uri| uri == &format!("crank://resources/{}", endpoint.notice.as_str())),
"the committed Resource is served to a client that missed the signal: {resources}"
);
// Legacy recovery: the negotiated session can be dropped, and the revision-correct answer is a
// safe `404` that makes the client re-initialize instead of resuming a stale stream.
let session = endpoint.legacy_session().await;
let mut stream = endpoint.legacy_stream(&session).await;
tokio::time::sleep(Duration::from_millis(1_500)).await;
let deleted = endpoint
.client
.delete(&endpoint.mcp_url)
.header(
header::AUTHORIZATION,
format!("Bearer {}", endpoint.api_key),
)
.header("MCP-Protocol-Version", LEGACY_REVISION)
.header("MCP-Session-Id", &session)
.send()
.await
.unwrap();
assert_eq!(deleted.status(), reqwest::StatusCode::NO_CONTENT);
stream
.expect_silence(
Duration::from_millis(2_500),
"a signal on the retired session",
)
.await;
let stale_get = endpoint
.client
.get(&endpoint.mcp_url)
.header(header::ACCEPT, "text/event-stream")
.header(
header::AUTHORIZATION,
format!("Bearer {}", endpoint.api_key),
)
.header("MCP-Protocol-Version", LEGACY_REVISION)
.header("MCP-Session-Id", &session)
.send()
.await
.unwrap();
assert_eq!(
stale_get.status(),
reqwest::StatusCode::NOT_FOUND,
"a retired session is never resumed"
);
let fresh = endpoint.legacy_session().await;
let prompts = endpoint.legacy(&fresh, "prompts/list", json!({})).await;
assert!(
prompt_names(&prompts).contains(&"extra".to_owned()),
"re-initialization serves the authoritative catalog: {prompts}"
);
}
// ------------------------------------------------------------------------------------------------
// Story 3.5 (FR-14): Tool catalog change signals. A client that negotiated the Tools list-changed
// notification learns about a *committed* Tool catalog change, and `tools/list` stays the
// authoritative source for a client without the notification or after a reconnect.
// ------------------------------------------------------------------------------------------------
/// AT-3.5-01/02 (modern): the Tools signal is delivered only to the stream that negotiated it, once
/// per committed change, and `tools/list` serves the same committed snapshot.
#[tokio::test]
async fn modern_tools_list_changed_is_delivered_only_to_a_client_that_asks_for_it() {
let endpoint = signals_endpoint().await;
let mut tools = endpoint
.listen(json!(51), json!({ "toolsListChanged": true }))
.await;
let mut prompts = endpoint
.listen(json!(52), json!({ "promptsListChanged": true }))
.await;
let acknowledgement = tools.expect_frame("the Tools acknowledgment").await;
assert_eq!(acknowledgement["method"], SUBSCRIPTIONS_ACKNOWLEDGED);
assert_eq!(
acknowledgement["params"]["notifications"],
json!({ "toolsListChanged": true }),
"the Tools list-change type must be honoured and acknowledged: {acknowledgement}"
);
let other = prompts.expect_frame("the Prompts acknowledgment").await;
assert_eq!(
other["params"]["notifications"],
json!({ "promptsListChanged": true }),
"a stream that did not ask for Tools is acknowledged without it: {other}"
);
// The watchers take their baseline before they can report anything.
tokio::time::sleep(Duration::from_millis(1_500)).await;
// A draft-only edit is not a commit: nothing is announced.
endpoint.edit_draft_without_publishing().await;
tools
.expect_silence(
Duration::from_millis(2_500),
"Tools signal for a draft-only edit",
)
.await;
// A committed Resource/Prompt change is not a Tools change: the Tools stream stays silent while
// the Prompts stream receives exactly its own type.
endpoint.commit_resource_and_prompt_change().await;
let prompt_frame = prompts
.expect_frame("the committed Prompt list-change signal")
.await;
assert_eq!(prompt_frame["method"], PROMPT_LIST_CHANGED);
tools
.expect_silence(
Duration::from_millis(2_500),
"Tools signal for a Resource/Prompt commit",
)
.await;
// One committed Tool catalog change is announced exactly once.
endpoint.commit_tool_only_change().await;
let frame = tools
.expect_frame("the committed Tools list-change signal")
.await;
assert_eq!(frame["jsonrpc"], json!("2.0"));
assert_eq!(frame["method"], TOOL_LIST_CHANGED);
assert_eq!(frame["params"]["_meta"][SUBSCRIPTION_ID_KEY], json!(51));
tools
.expect_silence(Duration::from_millis(2_500), "a repeated Tools signal")
.await;
// The prompts-only stream was never told about Tools, even though the commit happened after it
// opened.
prompts
.expect_silence(Duration::from_millis(1_200), "unrequested Tools signal")
.await;
// `tools/list` is authoritative for the same committed state, notification or not.
let list = endpoint.modern("tools/list", json!({})).await;
assert_eq!(
tool_names(&list),
vec!["crm_create_lead".to_owned(), "crm_create_second".to_owned()]
);
}
/// AT-3.5-01/02 (legacy): the legacy `GET` stream carries the Tools signal after the commit; a
/// Resource/Prompt commit produces no Tools frame; a committed change of another Agent is never
/// disclosed on this stream, and `tools/list` stays authoritative.
#[tokio::test]
async fn legacy_stream_announces_a_committed_tools_change_and_never_another_agents() {
let endpoint = signals_endpoint().await;
let other_key = endpoint.publish_other_agent(OTHER_AGENT_SLUG).await;
let session = endpoint.legacy_session().await;
let mut stream = endpoint.legacy_stream(&session).await;
tokio::time::sleep(Duration::from_millis(1_500)).await;
stream
.expect_silence(Duration::from_millis(1_200), "signal before any commit")
.await;
// A committed Resource/Prompt change keeps the Tool binding set: no Tools frame.
endpoint.commit_resource_and_prompt_change().await;
let mut methods = Vec::new();
for _ in 0..2 {
let frame = stream.expect_frame("the committed catalog signals").await;
assert!(
frame.get("params").is_none(),
"a legacy list-change notification carries no parameters: {frame}"
);
methods.push(frame["method"].as_str().unwrap().to_owned());
}
methods.sort();
assert_eq!(
methods,
vec![
PROMPT_LIST_CHANGED.to_owned(),
RESOURCE_LIST_CHANGED.to_owned()
],
"a Resource/Prompt commit must not announce Tools"
);
// A committed Tool change of another Agent of the same workspace never reaches this stream or
// this Agent's `tools/list`.
endpoint
.commit_other_agent_tool_change(OTHER_AGENT_SLUG)
.await;
stream
.expect_silence(
Duration::from_millis(2_500),
"Tools signal for another Agent's commit",
)
.await;
let isolated = endpoint.legacy(&session, "tools/list", json!({})).await;
assert_eq!(
tool_names(&isolated),
vec!["crm_create_lead".to_owned()],
"another Agent's commit must not change this Agent's catalog: {isolated}"
);
// The commit really happened: the other Agent serves its own two-Tool catalog.
let other_url = agent_mcp_url(&endpoint.base_url, OTHER_AGENT_SLUG);
let other_session = initialize_session(&endpoint.client, &other_url, &other_key).await;
let other_list = endpoint
.client
.post(&other_url)
.header(header::ACCEPT, "application/json, text/event-stream")
.header(header::AUTHORIZATION, format!("Bearer {other_key}"))
.header("MCP-Protocol-Version", LEGACY_REVISION)
.header("MCP-Session-Id", &other_session)
.json(&json!({
"jsonrpc": "2.0",
"id": 7,
"method": "tools/list",
"params": {}
}))
.send()
.await
.unwrap();
let other_list: Value = serde_json::from_str(&other_list.text().await.unwrap()).unwrap();
assert_eq!(
tool_names(&other_list),
vec!["crm_create_lead".to_owned(), "crm_create_second".to_owned()],
"the other Agent's committed catalog changed: {other_list}"
);
// This Agent's committed Tool change is announced after the commit, exactly once.
endpoint.commit_tool_only_change().await;
let frame = stream
.expect_frame("the committed Tools list-change signal")
.await;
assert_eq!(frame["method"], TOOL_LIST_CHANGED);
assert!(
frame.get("params").is_none(),
"a legacy list-change notification carries no parameters: {frame}"
);
stream
.expect_silence(
Duration::from_millis(2_500),
"a repeated or unrequested signal",
)
.await;
let list = endpoint.legacy(&session, "tools/list", json!({})).await;
assert_eq!(
tool_names(&list),
vec!["crm_create_lead".to_owned(), "crm_create_second".to_owned()],
"tools/list must serve the committed catalog without the notification: {list}"
);
}
/// AT-3.5-04: the Tools signal follows the same projection `tools/list` serves, so a committed
/// publication that only changes the Tool selection policy is announced on both revisions, while a
/// committed binding change that leaves `tools/list` unchanged is silent.
#[tokio::test]
async fn a_policy_only_commit_is_announced_on_both_revisions_and_an_invisible_change_is_silent() {
let endpoint = signals_endpoint().await;
let direct_projection = vec!["crm_create_lead".to_owned()];
// `tool_names` sorts the served page.
let search_projection = vec!["call_tool".to_owned(), "search_tools".to_owned()];
// Both revisions negotiate the Tools signal and take their baseline before anything can arrive.
let mut modern = endpoint
.listen(json!(61), json!({ "toolsListChanged": true }))
.await;
let acknowledgement = modern.expect_frame("the Tools acknowledgment").await;
assert_eq!(acknowledgement["method"], SUBSCRIPTIONS_ACKNOWLEDGED);
assert_eq!(
acknowledgement["params"]["notifications"],
json!({ "toolsListChanged": true })
);
let session = endpoint.legacy_session().await;
let mut legacy = endpoint.legacy_stream(&session).await;
tokio::time::sleep(Duration::from_millis(1_500)).await;
// The served projection before the commit is the direct-mode binding itself.
assert_eq!(
tool_names(&endpoint.modern("tools/list", json!({})).await),
direct_projection
);
// A policy-only commit: identical Tool/Resource/Prompt bindings, direct -> search mode. The
// served list changes, so each negotiated stream receives exactly one Tools frame.
endpoint.commit_policy_only_change().await;
let modern_frame = modern
.expect_frame("the committed Tools list-change signal (modern)")
.await;
assert_eq!(modern_frame["method"], TOOL_LIST_CHANGED);
assert_eq!(
modern_frame["params"]["_meta"][SUBSCRIPTION_ID_KEY],
json!(61)
);
let legacy_frame = legacy
.expect_frame("the committed Tools list-change signal (legacy)")
.await;
assert_eq!(legacy_frame["method"], TOOL_LIST_CHANGED);
assert!(
legacy_frame.get("params").is_none(),
"a legacy list-change notification carries no parameters: {legacy_frame}"
);
modern
.expect_silence(
Duration::from_millis(2_500),
"a repeated modern Tools signal",
)
.await;
legacy
.expect_silence(
Duration::from_millis(2_500),
"a repeated legacy Tools signal",
)
.await;
// `tools/list` is authoritative for the new projection on both revisions.
assert_eq!(
tool_names(&endpoint.modern("tools/list", json!({})).await),
search_projection
);
assert_eq!(
tool_names(&endpoint.legacy(&session, "tools/list", json!({})).await),
search_projection
);
// The inverse direction: in search mode a binding-only commit does not change `tools/list`, so
// neither stream may claim a visible Tool change.
endpoint.commit_search_mode_binding_change().await;
modern
.expect_silence(
Duration::from_millis(3_500),
"a Tools signal for an unchanged modern tools/list",
)
.await;
legacy
.expect_silence(
Duration::from_millis(2_500),
"a Tools signal for an unchanged legacy tools/list",
)
.await;
assert_eq!(
tool_names(&endpoint.modern("tools/list", json!({})).await),
search_projection
);
}
/// AT-3.5-02/03: a client that missed the notification recovers Tool discovery from the authoritative
/// state — the modern stateless client after a reconnect, and the legacy client after re-initializing
/// a retired session — and a reconnected stream never replays the missed change.
#[tokio::test]
async fn a_reconnected_client_recovers_tools_discovery_from_the_authoritative_state() {
let endpoint = signals_endpoint().await;
// Modern: the commit happens while no stream is open at all, then a client reconnects.
endpoint.commit_tool_only_change().await;
let mut reconnected = endpoint
.listen(
json!("after-reconnect"),
json!({ "toolsListChanged": true }),
)
.await;
let acknowledgement = reconnected.expect_frame("the Tools acknowledgment").await;
assert_eq!(
acknowledgement["params"]["notifications"],
json!({ "toolsListChanged": true })
);
// The baseline of a new stream is the committed state: the missed change is not replayed.
reconnected
.expect_silence(
Duration::from_millis(2_500),
"a replayed Tools signal after reconnect",
)
.await;
let list = endpoint.modern("tools/list", json!({})).await;
assert_eq!(
tool_names(&list),
vec!["crm_create_lead".to_owned(), "crm_create_second".to_owned()],
"a client without the notification recovers the committed catalog: {list}"
);
// Legacy: one more committed Tool change is announced on the live stream, then the session is
// retired and the client re-initializes.
let session = endpoint.legacy_session().await;
let mut stale = endpoint.legacy_stream(&session).await;
tokio::time::sleep(Duration::from_millis(1_500)).await;
endpoint.commit_tool_only_revert().await;
let frame = stale
.expect_frame("the committed Tools list-change signal")
.await;
assert_eq!(frame["method"], TOOL_LIST_CHANGED);
let deleted = endpoint
.client
.delete(&endpoint.mcp_url)
.header(
header::AUTHORIZATION,
format!("Bearer {}", endpoint.api_key),
)
.header("MCP-Protocol-Version", LEGACY_REVISION)
.header("MCP-Session-Id", &session)
.send()
.await
.unwrap();
assert_eq!(deleted.status(), reqwest::StatusCode::NO_CONTENT);
let fresh = endpoint.legacy_session().await;
let list = endpoint.legacy(&fresh, "tools/list", json!({})).await;
assert_eq!(
tool_names(&list),
vec!["crm_create_lead".to_owned()],
"re-initialization serves the committed catalog: {list}"
);
// A stream reopened after the reconnect starts from the committed baseline, not from a replay.
let mut reopened = endpoint.legacy_stream(&fresh).await;
reopened
.expect_silence(
Duration::from_millis(2_500),
"a replayed Tools signal on a reconnected stream",
)
.await;
}
@@ -0,0 +1,868 @@
//! Story 3.4 (FR-14) ATDD: progress and cancellation of one synchronous `tools/call`.
//!
//! Every case drives the real HTTP endpoint of the real application against a real (counted, slow)
//! upstream and real published catalog/registry state; nothing here is a fake client of the server
//! internals. The oracles are the pinned MCP shapes: `notifications/progress` (monotonic `progress`,
//! the request's own `progressToken`, no invented `total`) and the terminal `tools/call` result.
//!
//! Cancellation is revision-correct: `2025-11-25` cancels with `notifications/cancelled`, and
//! `2026-07-28` cancels by closing the request's own response stream. Both are proven to stop the
//! work, to release the call, and to never produce a `completed` record.
use std::{
sync::{
Arc,
atomic::{AtomicUsize, Ordering},
},
time::Duration,
};
use axum::{Json, Router, routing::post};
use crank_core::{InvocationStatus, OperationId, PlatformApiKeyScope};
use crank_registry::{ListInvocationLogsQuery, PostgresRegistry, PublishRequest};
use serde_json::{Value, json};
use time::{OffsetDateTime, format_description::well_known::Rfc3339};
use tokio::net::TcpListener;
use super::common::{
agent_mcp_url, build_test_app, capturing_subscriber, create_platform_api_key,
initialize_session, post_jsonrpc, post_jsonrpc_with_headers, publish_agent_for_operation,
spawn_mcp_server, test_operation, test_registry, test_workspace_id,
};
const LEGACY: &str = "2025-11-25";
const MODERN: &str = "2026-07-28";
const META_PROTOCOL_VERSION: &str = "io.modelcontextprotocol/protocolVersion";
const META_CLIENT_CAPABILITIES: &str = "io.modelcontextprotocol/clientCapabilities";
const META_CLIENT_INFO: &str = "io.modelcontextprotocol/clientInfo";
const TOOL: &str = "crm_lifecycle_lead";
/// A call that is long enough for the client to observe real progress, short enough for a suite.
const HOLD_MS: u64 = 250;
/// A call that is still running when the client cancels it.
const LONG_HOLD_MS: u64 = 1_500;
/// The grace period after a cancellation. It must exceed [`LONG_HOLD_MS`]: a call that was *not*
/// stopped completes its upstream request and records the outcome inside this window, so the checks
/// after it prove that the cancellation stopped the call rather than only that it had not finished
/// yet.
const CANCEL_GRACE_MS: u64 = 2_500;
/// The suite depends on that order: the constant must not silently drop back below the hold.
const _: () = assert!(CANCEL_GRACE_MS > LONG_HOLD_MS);
#[derive(Clone, Copy, Debug)]
enum Revision {
Legacy,
Modern,
}
impl Revision {
const fn label(self) -> &'static str {
match self {
Self::Legacy => LEGACY,
Self::Modern => MODERN,
}
}
}
struct Upstream {
base_url: String,
started: Arc<AtomicUsize>,
completed: Arc<AtomicUsize>,
}
/// A real upstream that records when a request starts and when it finishes. A cancelled call must
/// increment `started` exactly once and never `completed`: the request really was dispatched (so the
/// outcome is not `certain`-not-sent) and its response really was abandoned (so it is not retried).
async fn spawn_counted_slow_upstream(hold_ms: u64) -> Upstream {
let started = Arc::new(AtomicUsize::new(0));
let completed = Arc::new(AtomicUsize::new(0));
let handler_started = Arc::clone(&started);
let handler_completed = Arc::clone(&completed);
let app = Router::new().route(
"/crm/lifecycle-leads",
post(move |Json(payload): Json<Value>| {
let started = Arc::clone(&handler_started);
let completed = Arc::clone(&handler_completed);
async move {
started.fetch_add(1, Ordering::SeqCst);
tokio::time::sleep(Duration::from_millis(hold_ms)).await;
completed.fetch_add(1, Ordering::SeqCst);
Json(json!({ "id": "lead_123", "email": payload["email"] }))
}
}),
);
let listener = TcpListener::bind("127.0.0.1:0").await.unwrap();
let address = listener.local_addr().unwrap();
tokio::spawn(async move {
axum::serve(listener, app).await.unwrap();
});
Upstream {
base_url: format!("http://{address}"),
started,
completed,
}
}
struct Fixture {
client: reqwest::Client,
mcp_url: String,
api_key: String,
other_api_key: String,
registry: PostgresRegistry,
operation_id: OperationId,
upstream: Upstream,
}
async fn fixture(agent_slug: &str, hold_ms: u64, timeout_ms: u64) -> Fixture {
let upstream = spawn_counted_slow_upstream(hold_ms).await;
let registry = test_registry().await;
let mut operation = test_operation(&upstream.base_url, TOOL);
let crank_core::Target::Rest(target) = &mut operation.target;
target.path_template = "/crm/lifecycle-leads".to_owned();
operation.execution_config.timeout_ms = timeout_ms;
registry
.create_operation(&test_workspace_id(), &operation, Some("alice"))
.await
.unwrap();
registry
.publish_operation(PublishRequest {
workspace_id: &test_workspace_id(),
operation_id: &operation.id,
version: 1,
published_at: &OffsetDateTime::parse("2026-03-26T10:00:00Z", &Rfc3339).unwrap(),
published_by: Some("alice"),
})
.await
.unwrap();
publish_agent_for_operation(&registry, &operation, agent_slug).await;
let scopes = [PlatformApiKeyScope::Read, PlatformApiKeyScope::Write];
let api_key = create_platform_api_key(&registry, agent_slug, "mcp-lifecycle", &scopes).await;
let other_api_key =
create_platform_api_key(&registry, agent_slug, "mcp-lifecycle-other", &scopes).await;
let base_url = spawn_mcp_server(build_test_app(
registry.clone(),
Duration::ZERO,
Some("https://crank.example.com".to_owned()),
))
.await;
Fixture {
client: reqwest::Client::new(),
mcp_url: agent_mcp_url(&base_url, agent_slug),
api_key,
other_api_key,
registry,
operation_id: operation.id,
upstream,
}
}
/// One parsed answer: the HTTP status, the negotiated media type and every JSON-RPC frame in order.
struct Reply {
status: reqwest::StatusCode,
content_type: String,
frames: Vec<Value>,
raw: String,
}
impl Reply {
fn parse(status: reqwest::StatusCode, content_type: String, body: String) -> Self {
let mut frames = Vec::new();
for block in body.split("\n\n") {
for line in block.lines() {
let Some(data) = line.strip_prefix("data:") else {
continue;
};
let data = data.trim_start();
if data.is_empty() {
continue;
}
frames.push(
serde_json::from_str(data)
.unwrap_or_else(|error| panic!("every frame is JSON: {data:?} {error}")),
);
}
}
if frames.is_empty() && !body.trim().is_empty() {
frames.push(
serde_json::from_str(body.trim()).unwrap_or_else(|error| {
panic!("expected a JSON-RPC body, got {body:?}: {error}")
}),
);
}
Self {
status,
content_type,
frames,
raw: body,
}
}
fn assert_stream(&self) {
assert_eq!(self.status, reqwest::StatusCode::OK);
assert!(
self.content_type.starts_with("text/event-stream"),
"a progress answer must be the request's SSE stream, got {:?}: {}",
self.content_type,
self.raw
);
}
fn progress_frames(&self) -> Vec<&Value> {
self.frames
.iter()
.filter(|frame| frame["method"] == json!("notifications/progress"))
.collect()
}
fn terminal(&self) -> &Value {
let terminal = self
.frames
.last()
.unwrap_or_else(|| panic!("the stream must carry a terminal frame: {}", self.raw));
assert!(
terminal.get("method").is_none(),
"the terminal frame must be the JSON-RPC response, got {terminal}"
);
terminal
}
}
impl Fixture {
fn call_params(&self, token: Option<Value>, revision: Revision) -> Value {
let mut params = json!({
"name": TOOL,
"arguments": { "email": "lifecycle@example.com" },
});
let mut meta = match revision {
Revision::Modern => json!({
META_PROTOCOL_VERSION: MODERN,
META_CLIENT_CAPABILITIES: {},
META_CLIENT_INFO: { "name": "crank-3.4-atdd", "version": "0.1.0" },
}),
Revision::Legacy => json!({}),
};
if let Some(token) = token {
meta["progressToken"] = token;
}
if meta.as_object().is_some_and(|object| !object.is_empty()) {
params["_meta"] = meta;
}
params
}
fn headers(&self, revision: Revision, session: Option<&str>) -> Vec<(&'static str, String)> {
let mut headers = vec![("MCP-Protocol-Version", revision.label().to_owned())];
match revision {
Revision::Legacy => {
if let Some(session) = session {
headers.push(("MCP-Session-Id", session.to_owned()));
}
}
Revision::Modern => {
headers.push(("Mcp-Method", "tools/call".to_owned()));
headers.push(("Mcp-Name", TOOL.to_owned()));
}
}
headers
}
fn request(
&self,
accept: &'static str,
key: &str,
headers: &[(&'static str, String)],
payload: Value,
) -> reqwest::RequestBuilder {
let mut request = self
.client
.post(&self.mcp_url)
.header(reqwest::header::ACCEPT, accept)
.header(reqwest::header::AUTHORIZATION, format!("Bearer {key}"));
for (name, value) in headers {
request = request.header(*name, value);
}
request.json(&payload)
}
/// One `tools/call` whose answer is negotiated as the request's own SSE stream.
async fn call_stream(
&self,
revision: Revision,
id: u64,
params: Value,
session: Option<&str>,
request_id: &str,
) -> Reply {
self.call_stream_with_key(revision, &self.api_key, id, params, session, request_id)
.await
}
async fn call_stream_with_key(
&self,
revision: Revision,
key: &str,
id: u64,
params: Value,
session: Option<&str>,
request_id: &str,
) -> Reply {
self.send(
"text/event-stream",
key,
revision,
id,
params,
session,
Some(request_id),
)
.await
}
/// One `tools/call` whose answer is negotiated as a single JSON body.
async fn call_json(
&self,
revision: Revision,
id: u64,
params: Value,
session: Option<&str>,
) -> Reply {
self.send(
"application/json",
&self.api_key,
revision,
id,
params,
session,
None,
)
.await
}
#[allow(clippy::too_many_arguments)]
async fn send(
&self,
accept: &'static str,
key: &str,
revision: Revision,
id: u64,
params: Value,
session: Option<&str>,
request_id: Option<&str>,
) -> Reply {
let headers = self.headers(revision, session);
let payload =
json!({ "jsonrpc": "2.0", "id": id, "method": "tools/call", "params": params });
let mut request = self.request(accept, key, &headers, payload);
if let Some(request_id) = request_id {
request = request.header("x-request-id", request_id);
}
let response = request.send().await.expect("the MCP endpoint answers");
let status = response.status();
let content_type = response
.headers()
.get(reqwest::header::CONTENT_TYPE)
.and_then(|value| value.to_str().ok())
.unwrap_or_default()
.to_owned();
let body = response.text().await.expect("the body is readable");
Reply::parse(status, content_type, body)
}
async fn cancel(&self, key: &str, session: &str, request_id: Value) -> reqwest::StatusCode {
let response = self
.request(
"application/json",
key,
&[
("MCP-Protocol-Version", LEGACY.to_owned()),
("MCP-Session-Id", session.to_owned()),
],
json!({
"jsonrpc": "2.0",
"method": "notifications/cancelled",
"params": { "requestId": request_id, "reason": "user requested cancellation" }
}),
)
.send()
.await
.expect("the cancellation is acknowledged");
response.status()
}
/// Every Invocation History row of this operation and request id, oldest first.
async fn history(&self, request_id: &str) -> Vec<crank_core::InvocationLog> {
self.registry
.list_invocation_logs(ListInvocationLogsQuery {
workspace_id: &test_workspace_id(),
level: None,
status: None,
outcome_group: None,
search_text: None,
source: Some(crank_core::InvocationSource::AgentToolCall),
operation_id: Some(&self.operation_id),
agent_id: None,
created_after: None,
created_before: None,
cursor_created_at: None,
cursor_id: None,
limit: 50,
})
.await
.unwrap()
.into_iter()
.map(|record| record.log)
.filter(|log| log.request_id.as_deref() == Some(request_id))
.collect()
}
/// Wait for the counted upstream to be entered, so a cancellation happens while it really runs.
async fn wait_until_dispatched(&self) {
for _ in 0..200 {
if self.upstream.started.load(Ordering::SeqCst) > 0 {
return;
}
tokio::time::sleep(Duration::from_millis(5)).await;
}
panic!("the upstream was never dispatched");
}
}
fn progress_values(frames: &[&Value]) -> Vec<u64> {
frames
.iter()
.map(|frame| {
frame["params"]["progress"]
.as_u64()
.unwrap_or_else(|| panic!("progress must be a number, got {frame}"))
})
.collect()
}
// ---------------------------------------------------------------------------------------------
// AT-3.4-01: a negotiated stream carries bounded monotonic progress of the real execution, and a
// missing token (or a missing carrier) never changes the execution result.
// ---------------------------------------------------------------------------------------------
#[tokio::test]
async fn at_3_4_01_legacy_progress_stream_is_monotonic_and_ends_with_the_real_result() {
let fixture = fixture("lifecycle-legacy-progress", HOLD_MS, 5_000).await;
let session = initialize_session(&fixture.client, &fixture.mcp_url, &fixture.api_key).await;
let reply = fixture
.call_stream(
Revision::Legacy,
501,
fixture.call_params(Some(json!("tok-legacy-progress")), Revision::Legacy),
Some(&session),
"req-legacy-progress",
)
.await;
reply.assert_stream();
let progress = reply.progress_frames();
assert!(
!progress.is_empty(),
"a token with a stream carrier must produce progress, got {}",
reply.raw
);
for frame in &progress {
assert_eq!(frame["jsonrpc"], json!("2.0"));
assert_eq!(
frame["params"]["progressToken"],
json!("tok-legacy-progress")
);
assert!(
frame["params"].get("total").is_none(),
"this server knows no total and must not invent one: {frame}"
);
assert!(
frame["params"]["message"].is_string(),
"every real step names itself: {frame}"
);
}
let values = progress_values(&progress);
assert!(
values.windows(2).all(|pair| pair[0] < pair[1]),
"progress must be strictly monotonic, got {values:?}"
);
// One event per real execution step this server actually crosses: resolved, authorized,
// dispatched. Never a timer-driven percentage and never an unbounded stream.
assert_eq!(values, vec![1, 2, 3], "got {values:?}");
let terminal = reply.terminal();
assert_eq!(terminal["id"], json!(501));
assert_eq!(terminal["error"], Value::Null);
assert_eq!(terminal["result"]["isError"], json!(false));
assert_eq!(
terminal["result"]["structuredContent"],
json!({ "id": "lead_123" })
);
assert_eq!(fixture.upstream.started.load(Ordering::SeqCst), 1);
assert_eq!(fixture.upstream.completed.load(Ordering::SeqCst), 1);
}
#[tokio::test]
async fn at_3_4_01_modern_progress_stream_is_monotonic_and_ends_with_the_real_result() {
let fixture = fixture("lifecycle-modern-progress", HOLD_MS, 5_000).await;
let reply = fixture
.call_stream(
Revision::Modern,
502,
fixture.call_params(Some(json!(77)), Revision::Modern),
None,
"req-modern-progress",
)
.await;
reply.assert_stream();
let progress = reply.progress_frames();
assert!(!progress.is_empty(), "got {}", reply.raw);
for frame in &progress {
assert_eq!(frame["params"]["progressToken"], json!(77));
}
let values = progress_values(&progress);
assert!(
values.windows(2).all(|pair| pair[0] < pair[1]),
"{values:?}"
);
let terminal = reply.terminal();
assert_eq!(terminal["result"]["isError"], json!(false));
assert_eq!(
terminal["result"]["structuredContent"],
json!({ "id": "lead_123" })
);
}
// ---------------------------------------------------------------------------------------------
// AT-3.4-01 (TL correction): the official SDK sends the dual `Accept` JSON-first. A `tools/call`
// that carries a validated token and offers `text/event-stream` in that same header must still be
// answered on its own stream, wherever the media types are ordered.
// ---------------------------------------------------------------------------------------------
#[tokio::test]
async fn at_3_4_01_json_first_dual_accept_keeps_progress_for_a_valid_token() {
/// Exactly what the official MCP SDK sends when it can read a stream.
const SDK_ACCEPT: &str = "application/json, text/event-stream";
let fixture = fixture("lifecycle-json-first", HOLD_MS, 5_000).await;
let session = initialize_session(&fixture.client, &fixture.mcp_url, &fixture.api_key).await;
for revision in [Revision::Legacy, Revision::Modern] {
let carrier = match revision {
Revision::Legacy => Some(session.as_str()),
Revision::Modern => None,
};
let token = json!("tok-json-first");
let reply = fixture
.send(
SDK_ACCEPT,
&fixture.api_key,
revision,
550,
fixture.call_params(Some(token.clone()), revision),
carrier,
Some("req-json-first"),
)
.await;
reply.assert_stream();
let progress = reply.progress_frames();
assert!(
!progress.is_empty(),
"{revision:?}: a valid token with an acceptable stream must produce progress, got {}",
reply.raw
);
for frame in &progress {
assert_eq!(frame["params"]["progressToken"], token, "{frame}");
assert!(
frame["params"].get("total").is_none(),
"this server knows no total: {frame}"
);
}
let values = progress_values(&progress);
assert!(
values.windows(2).all(|pair| pair[0] < pair[1]),
"{revision:?}: progress must be strictly monotonic, got {values:?}"
);
assert_eq!(values, vec![1, 2, 3], "{revision:?}: got {values:?}");
let terminal = reply.terminal();
assert_eq!(terminal["id"], json!(550), "{revision:?}");
assert_eq!(terminal["result"]["isError"], json!(false), "{revision:?}");
assert_eq!(
terminal["result"]["structuredContent"],
json!({ "id": "lead_123" }),
"{revision:?}"
);
}
// The correction is scoped to a call that can actually carry progress: the same JSON-first
// header without a token, and with an unusable token, keeps the negotiated single JSON body.
for token in [None, Some(json!("x".repeat(4096)))] {
let reply = fixture
.send(
SDK_ACCEPT,
&fixture.api_key,
Revision::Legacy,
551,
fixture.call_params(token.clone(), Revision::Legacy),
Some(&session),
None,
)
.await;
assert_eq!(reply.status, reqwest::StatusCode::OK);
assert!(
reply.content_type.starts_with("application/json"),
"token {token:?} must keep the negotiated JSON body, got {:?}: {}",
reply.content_type,
reply.raw
);
assert!(reply.progress_frames().is_empty(), "{token:?}");
assert_eq!(
reply.terminal()["result"]["structuredContent"],
json!({ "id": "lead_123" }),
"token {token:?}"
);
}
}
#[tokio::test]
async fn at_3_4_01_no_token_and_no_carrier_keep_the_result_identical() {
let fixture = fixture("lifecycle-no-token", HOLD_MS, 5_000).await;
let session = initialize_session(&fixture.client, &fixture.mcp_url, &fixture.api_key).await;
// Without a token the very same call is one plain JSON body: no notification exists.
let without_token = fixture
.call_json(
Revision::Legacy,
510,
fixture.call_params(None, Revision::Legacy),
Some(&session),
)
.await;
assert_eq!(without_token.status, reqwest::StatusCode::OK);
assert!(without_token.progress_frames().is_empty());
let plain = without_token.terminal()["result"].clone();
assert_eq!(plain["isError"], json!(false));
// With a token but only the `application/json` carrier the result is the same and progress is
// simply not sent (the receiver is never obligated to send it).
let token_without_carrier = fixture
.call_json(
Revision::Legacy,
511,
fixture.call_params(Some(json!("tok-no-carrier")), Revision::Legacy),
Some(&session),
)
.await;
assert_eq!(token_without_carrier.status, reqwest::StatusCode::OK);
assert!(token_without_carrier.progress_frames().is_empty());
assert_eq!(token_without_carrier.terminal()["result"], plain);
// In the stream mode without a token the only frame is the terminal response itself.
let stream_without_token = fixture
.call_stream(
Revision::Legacy,
512,
fixture.call_params(None, Revision::Legacy),
Some(&session),
"req-no-token-stream",
)
.await;
assert!(stream_without_token.progress_frames().is_empty());
assert_eq!(stream_without_token.frames.len(), 1);
assert_eq!(stream_without_token.terminal()["result"], plain);
}
/// The streamed answer keeps the Story 3.18 projection: a client that opted in to logging gets the
/// same single closed `notifications/message` of this call before the terminal frame, next to the
/// progress events of Story 3.4.
#[tokio::test]
async fn at_3_4_01_a_progress_stream_keeps_the_logging_projection() {
let fixture = fixture("lifecycle-progress-logging", HOLD_MS, 5_000).await;
let session = initialize_session(&fixture.client, &fixture.mcp_url, &fixture.api_key).await;
let set_level = fixture
.request(
"application/json",
&fixture.api_key,
&[
("MCP-Protocol-Version", LEGACY.to_owned()),
("MCP-Session-Id", session.clone()),
],
json!({
"jsonrpc": "2.0", "id": 900, "method": "logging/setLevel",
"params": { "level": "debug" }
}),
)
.send()
.await
.expect("the level is accepted");
assert_eq!(set_level.status(), reqwest::StatusCode::OK);
let reply = fixture
.call_stream(
Revision::Legacy,
901,
fixture.call_params(Some(json!("tok-logging")), Revision::Legacy),
Some(&session),
"req-progress-logging",
)
.await;
reply.assert_stream();
assert!(!reply.progress_frames().is_empty(), "got {}", reply.raw);
let logged: Vec<&Value> = reply
.frames
.iter()
.filter(|frame| frame["method"] == json!("notifications/message"))
.collect();
assert_eq!(
logged.len(),
1,
"exactly one closed log event per response, got {}",
reply.raw
);
assert_eq!(
logged[0]["params"]["data"]["event"],
json!("mcp.request.served")
);
assert_eq!(logged[0]["params"]["data"]["outcome"], json!("ok"));
assert_eq!(reply.terminal()["result"]["isError"], json!(false));
}
// ---------------------------------------------------------------------------------------------
// AT-3.4-02: a malformed, reused-in-another-scope or oversized token disables progress, and the
// token never becomes a metric label, a log field or an authorization scope.
// ---------------------------------------------------------------------------------------------
#[tokio::test]
async fn at_3_4_02_malformed_and_oversized_tokens_disable_progress_without_changing_the_result() {
let fixture = fixture("lifecycle-bad-token", HOLD_MS, 5_000).await;
let session = initialize_session(&fixture.client, &fixture.mcp_url, &fixture.api_key).await;
// Both supported revisions read the token from the same pinned `RequestMetaObject` field, so an
// unusable token must behave identically in each of them: no progress, unchanged result.
for revision in [Revision::Legacy, Revision::Modern] {
let carrier = match revision {
Revision::Legacy => Some(session.as_str()),
Revision::Modern => None,
};
let tokens = [
json!(""),
json!({ "not": "a token" }),
json!([1, 2]),
json!(true),
json!("x".repeat(4096)),
];
let mut observed = Vec::new();
for token in tokens {
let reply = fixture
.call_stream(
revision,
520,
fixture.call_params(Some(token.clone()), revision),
carrier,
"req-bad-token",
)
.await;
observed.push((token, reply));
}
for (token, reply) in observed {
assert!(
reply.progress_frames().is_empty(),
"{revision:?}: token {token} must not produce progress, got {}",
reply.raw
);
let terminal = reply.terminal();
assert_eq!(
terminal["result"]["structuredContent"],
json!({ "id": "lead_123" }),
"{revision:?}: token {token} must not change the execution result, got {terminal}"
);
assert_eq!(terminal["result"]["isError"], json!(false));
assert!(terminal["error"].is_null() || terminal.get("error").is_none());
}
}
}
#[tokio::test]
async fn at_3_4_02_a_reused_token_is_scoped_to_its_own_request() {
let fixture = fixture("lifecycle-reused-token", HOLD_MS, 5_000).await;
let first = initialize_session(&fixture.client, &fixture.mcp_url, &fixture.api_key).await;
let second = initialize_session(&fixture.client, &fixture.mcp_url, &fixture.api_key).await;
let token = json!("tok-shared");
let (first_reply, second_reply) = tokio::join!(
fixture.call_stream(
Revision::Legacy,
530,
fixture.call_params(Some(token.clone()), Revision::Legacy),
Some(&first),
"req-first",
),
fixture.call_stream(
Revision::Legacy,
531,
fixture.call_params(Some(token.clone()), Revision::Legacy),
Some(&second),
"req-second",
)
);
for (reply, id) in [(&first_reply, 530), (&second_reply, 531)] {
reply.assert_stream();
let values = progress_values(&reply.progress_frames());
// Each request counts its own real steps: a reused token never inherits another call's count.
assert_eq!(values, vec![1, 2, 3], "request {id} got {values:?}");
assert_eq!(reply.terminal()["id"], json!(id));
}
}
#[tokio::test]
async fn at_3_4_02_a_progress_token_never_reaches_logs_or_invocation_history() {
const CANARY: &str = "crank-progress-canary-4f1c2d";
let fixture = fixture("lifecycle-token-leak", HOLD_MS, 5_000).await;
let session = initialize_session(&fixture.client, &fixture.mcp_url, &fixture.api_key).await;
let (writer, subscriber) = capturing_subscriber();
let reply = {
let _guard = tracing::subscriber::set_default(subscriber);
fixture
.call_stream(
Revision::Legacy,
540,
fixture.call_params(Some(json!(CANARY)), Revision::Legacy),
Some(&session),
"req-token-leak",
)
.await
};
reply.assert_stream();
let logs = writer.output();
assert!(
logs.contains("mcp request received"),
"the capture must really see the endpoint's own log projection, got {logs}"
);
assert!(
!logs.contains(CANARY),
"the progress token is client data and must never be logged, got {logs}"
);
let history = fixture.history("req-token-leak").await;
assert_eq!(history.len(), 1, "got {history:?}");
let row = &history[0];
let serialized = serde_json::to_string(row).unwrap();
assert!(
!serialized.contains(CANARY),
"the progress token must never reach Invocation History, got {serialized}"
);
assert_eq!(row.status, InvocationStatus::Ok);
}
// AT-3.4-03: a cancellation stops a call that can still be stopped, releases it and never yields a
// `completed` result or record.
mod cancellation;
// AT-3.4-05: the client-visible smoke of both supported revisions.
mod smoke;
@@ -0,0 +1,294 @@
//! AT-3.4-03: a cancellation stops a call that can still be stopped, releases it, and never yields a
//! `completed` result or history row. The harness of the suite lives in `super`.
use super::*;
// ---------------------------------------------------------------------------------------------
// AT-3.4-03: a cancellation stops a call that can still be stopped, releases it, and never yields a
// `completed` result or history row.
// ---------------------------------------------------------------------------------------------
#[tokio::test]
async fn at_3_4_03_legacy_cancellation_stops_the_call_and_never_completes_it() {
let fixture = fixture("lifecycle-legacy-cancel", LONG_HOLD_MS, 30_000).await;
let session = initialize_session(&fixture.client, &fixture.mcp_url, &fixture.api_key).await;
let call_session = session.clone();
let call = tokio::spawn({
let fixture_client = fixture.client.clone();
let mcp_url = fixture.mcp_url.clone();
let api_key = fixture.api_key.clone();
let params = fixture.call_params(Some(json!("tok-cancel")), Revision::Legacy);
async move {
let response = fixture_client
.post(mcp_url)
.header(reqwest::header::ACCEPT, "text/event-stream")
.header(reqwest::header::AUTHORIZATION, format!("Bearer {api_key}"))
.header("MCP-Protocol-Version", LEGACY)
.header("MCP-Session-Id", call_session)
.header("x-request-id", "req-legacy-cancel")
.json(&json!({
"jsonrpc": "2.0",
"id": 601,
"method": "tools/call",
"params": params,
}))
.send()
.await
.unwrap();
(response.status(), response.text().await.unwrap())
}
});
fixture.wait_until_dispatched().await;
assert_eq!(
fixture.cancel(&fixture.api_key, &session, json!(601)).await,
reqwest::StatusCode::ACCEPTED,
"an accepted cancellation is acknowledged without a body"
);
let (status, body) = tokio::time::timeout(Duration::from_secs(10), call)
.await
.expect("the cancelled call is released, it does not hang")
.expect("the request task joins");
assert_eq!(status, reqwest::StatusCode::OK);
let reply = Reply::parse(status, String::new(), body);
assert!(
reply
.frames
.iter()
.all(|frame| frame.get("method").is_some()),
"a cancelled request gets no terminal message, only the progress it already saw: {}",
reply.raw
);
assert_eq!(
reply.progress_frames().len(),
3,
"the client really saw the real steps before the stop: {}",
reply.raw
);
tokio::time::sleep(Duration::from_millis(CANCEL_GRACE_MS)).await;
assert_eq!(
fixture.upstream.completed.load(Ordering::SeqCst),
0,
"the abandoned upstream response must never complete the call"
);
assert_eq!(
fixture.upstream.started.load(Ordering::SeqCst),
1,
"the write really was dispatched and must not be retried automatically"
);
let history = fixture.history("req-legacy-cancel").await;
assert!(
history.iter().all(|row| row.status != InvocationStatus::Ok),
"a cancelled call must never be recorded as completed, got {history:?}"
);
// The record contract admits a non-`ok` row only with a closed execution error code, and a
// cancellation is not one of them: the honest terminal state of this call is "no record" instead
// of an invented error code. Either way nothing claims that the call completed.
assert!(
history.is_empty(),
"a stopped call has no terminal outcome to record, got {history:?}"
);
}
#[tokio::test]
async fn at_3_4_03_a_foreign_or_unknown_cancellation_never_touches_the_active_call() {
let fixture = fixture("lifecycle-foreign-cancel", HOLD_MS, 5_000).await;
let session = initialize_session(&fixture.client, &fixture.mcp_url, &fixture.api_key).await;
let foreign_session =
initialize_session(&fixture.client, &fixture.mcp_url, &fixture.other_api_key).await;
let call = tokio::spawn({
let client = fixture.client.clone();
let mcp_url = fixture.mcp_url.clone();
let api_key = fixture.api_key.clone();
let call_session = session.clone();
let params = fixture.call_params(Some(json!("tok-owner")), Revision::Legacy);
async move {
let response = client
.post(mcp_url)
.header(reqwest::header::ACCEPT, "text/event-stream")
.header(reqwest::header::AUTHORIZATION, format!("Bearer {api_key}"))
.header("MCP-Protocol-Version", LEGACY)
.header("MCP-Session-Id", call_session)
.header("x-request-id", "req-owner")
.json(&json!({
"jsonrpc": "2.0", "id": 602, "method": "tools/call", "params": params,
}))
.send()
.await
.unwrap();
response.text().await.unwrap()
}
});
fixture.wait_until_dispatched().await;
// A foreign credential of the same agent, and an unknown request id: neither may cancel this call.
assert_eq!(
fixture
.cancel(&fixture.other_api_key, &session, json!(602))
.await,
reqwest::StatusCode::ACCEPTED
);
assert_eq!(
fixture
.cancel(&fixture.api_key, &session, json!(999_999))
.await,
reqwest::StatusCode::ACCEPTED
);
// A foreign session of the same credential cannot reach this call either.
assert_eq!(
fixture
.cancel(&fixture.api_key, &foreign_session, json!(602))
.await,
reqwest::StatusCode::ACCEPTED
);
let body = tokio::time::timeout(Duration::from_secs(10), call)
.await
.expect("the untouched call completes")
.expect("the request task joins");
let reply = Reply::parse(reqwest::StatusCode::OK, String::new(), body);
assert_eq!(
reply.terminal()["result"]["structuredContent"],
json!({ "id": "lead_123" }),
"a foreign cancellation must not stop the owner's call, got {}",
reply.raw
);
let history = fixture.history("req-owner").await;
assert_eq!(history.len(), 1, "got {history:?}");
assert_eq!(history[0].status, InvocationStatus::Ok);
}
#[tokio::test]
async fn at_3_4_03_a_late_cancellation_after_the_result_is_ignored() {
let fixture = fixture("lifecycle-late-cancel", HOLD_MS, 5_000).await;
let session = initialize_session(&fixture.client, &fixture.mcp_url, &fixture.api_key).await;
let reply = fixture
.call_stream(
Revision::Legacy,
603,
fixture.call_params(Some(json!("tok-late")), Revision::Legacy),
Some(&session),
"req-late-cancel",
)
.await;
reply.assert_stream();
assert_eq!(reply.terminal()["result"]["isError"], json!(false));
assert_eq!(
fixture.cancel(&fixture.api_key, &session, json!(603)).await,
reqwest::StatusCode::ACCEPTED
);
tokio::time::sleep(Duration::from_millis(200)).await;
let history = fixture.history("req-late-cancel").await;
assert_eq!(
history.len(),
1,
"a cancellation of a finished call must not add a second record, got {history:?}"
);
assert_eq!(history[0].status, InvocationStatus::Ok);
}
#[tokio::test]
async fn at_3_4_03_a_plain_json_call_is_cancellable_too() {
let fixture = fixture("lifecycle-json-cancel", LONG_HOLD_MS, 30_000).await;
let session = initialize_session(&fixture.client, &fixture.mcp_url, &fixture.api_key).await;
let call = tokio::spawn({
let client = fixture.client.clone();
let mcp_url = fixture.mcp_url.clone();
let api_key = fixture.api_key.clone();
let call_session = session.clone();
let params = fixture.call_params(None, Revision::Legacy);
async move {
let response = client
.post(mcp_url)
.header(reqwest::header::ACCEPT, "application/json")
.header(reqwest::header::AUTHORIZATION, format!("Bearer {api_key}"))
.header("MCP-Protocol-Version", LEGACY)
.header("MCP-Session-Id", call_session)
.header("x-request-id", "req-json-cancel")
.json(&json!({
"jsonrpc": "2.0", "id": 605, "method": "tools/call", "params": params,
}))
.send()
.await
.unwrap();
let status = response.status();
(status, response.text().await.unwrap())
}
});
fixture.wait_until_dispatched().await;
assert_eq!(
fixture.cancel(&fixture.api_key, &session, json!(605)).await,
reqwest::StatusCode::ACCEPTED
);
let (status, body) = tokio::time::timeout(Duration::from_secs(10), call)
.await
.expect("the cancelled call is released")
.expect("the request task joins");
assert_eq!(status, reqwest::StatusCode::OK);
let reply = Reply::parse(status, String::new(), body);
assert!(
reply.frames.is_empty(),
"a cancelled request gets no terminal message, got {}",
reply.raw
);
tokio::time::sleep(Duration::from_millis(CANCEL_GRACE_MS)).await;
assert_eq!(fixture.upstream.completed.load(Ordering::SeqCst), 0);
let history = fixture.history("req-json-cancel").await;
assert!(
history.iter().all(|row| row.status != InvocationStatus::Ok),
"a cancelled call must never be recorded as completed, got {history:?}"
);
}
#[tokio::test]
async fn at_3_4_03_closing_the_modern_response_stream_cancels_the_call() {
let fixture = fixture("lifecycle-modern-disconnect", LONG_HOLD_MS, 30_000).await;
let response = fixture
.request(
"text/event-stream",
&fixture.api_key,
&fixture.headers(Revision::Modern, None),
json!({
"jsonrpc": "2.0",
"id": 604,
"method": "tools/call",
"params": fixture.call_params(Some(json!("tok-disconnect")), Revision::Modern),
}),
)
.header("x-request-id", "req-modern-disconnect")
.send()
.await
.expect("the stream opens before the call finishes");
assert_eq!(response.status(), reqwest::StatusCode::OK);
fixture.wait_until_dispatched().await;
// The pinned `2026-07-28` cancellation signal: closing the request's own response stream.
drop(response);
tokio::time::sleep(Duration::from_millis(CANCEL_GRACE_MS)).await;
assert_eq!(
fixture.upstream.completed.load(Ordering::SeqCst),
0,
"the closed stream must stop the dispatched call instead of letting it finish"
);
assert_eq!(fixture.upstream.started.load(Ordering::SeqCst), 1);
let history = fixture.history("req-modern-disconnect").await;
assert!(
history.iter().all(|row| row.status != InvocationStatus::Ok),
"a cancelled call must never be recorded as completed, got {history:?}"
);
}
@@ -0,0 +1,79 @@
//! AT-3.4-05: the client-visible smoke of both supported revisions.
//!
//! A representative client of each revision performs the whole flow a real MCP client performs —
//! discovery, then a `tools/call` that asks for progress and reads the request's own stream to the
//! terminal response — against the real endpoint. The harness of the suite lives in `super`.
use super::*;
#[tokio::test]
async fn at_3_4_05_representative_client_smoke_of_both_revisions() {
let fixture = fixture("lifecycle-client-smoke", HOLD_MS, 5_000).await;
// Legacy client: initialize → tools/list → tools/call with progress.
let session = initialize_session(&fixture.client, &fixture.mcp_url, &fixture.api_key).await;
let tools = post_jsonrpc(
&fixture.client,
&fixture.mcp_url,
&fixture.api_key,
Some(&session),
json!({ "jsonrpc": "2.0", "id": 700, "method": "tools/list", "params": {} }),
)
.await;
assert_eq!(tools["result"]["tools"][0]["name"], json!(TOOL));
let legacy_call = fixture
.call_stream(
Revision::Legacy,
701,
fixture.call_params(Some(json!("tok-smoke-legacy")), Revision::Legacy),
Some(&session),
"req-smoke-legacy",
)
.await;
legacy_call.assert_stream();
assert_eq!(
legacy_call.terminal()["result"]["structuredContent"],
json!({ "id": "lead_123" })
);
// Modern client: `tools/list` and `tools/call` of the stateless revision, progress on the stream.
let modern_list = post_jsonrpc_with_headers(
&fixture.client,
&fixture.mcp_url,
&fixture.api_key,
&[
("MCP-Protocol-Version", MODERN),
("Mcp-Method", "tools/list"),
],
json!({
"jsonrpc": "2.0",
"id": 710,
"method": "tools/list",
"params": { "_meta": {
META_PROTOCOL_VERSION: MODERN,
META_CLIENT_CAPABILITIES: {},
META_CLIENT_INFO: { "name": "crank-3.4-smoke", "version": "0.1.0" },
}}
}),
)
.await;
let modern_list = modern_list.json::<Value>().await.unwrap();
assert_eq!(modern_list["result"]["tools"][0]["name"], json!(TOOL));
let modern_call = fixture
.call_stream(
Revision::Modern,
711,
fixture.call_params(Some(json!("tok-smoke-modern")), Revision::Modern),
None,
"req-smoke-modern",
)
.await;
modern_call.assert_stream();
assert!(!modern_call.progress_frames().is_empty());
assert_eq!(
modern_call.terminal()["result"]["structuredContent"],
json!({ "id": "lead_123" })
);
}
+1 -1
View File
@@ -10,7 +10,7 @@ COPY crank-community.png ./crank-community.png
RUN npm run build
FROM nginx:1.30.4-alpine3.24-slim
FROM nginx:1.30.6-alpine3.24-slim
RUN apk add --no-cache --upgrade \
'pcre2>=10.49-r0' \
+2
View File
@@ -1036,6 +1036,7 @@ var TRANSLATIONS = {
'wizard.error.secret_name_required': 'Secret name is required.',
'wizard.error.secret_value_required': 'Secret value is required.',
'wizard.error.save_upstream_first': 'Save the API host after creating a new authorization profile before continuing.',
'wizard.error.execution_policy_invalid': 'Execution policy is invalid: mode, deadline from 1 to 3600000 ms and no-Tasks fallback are required.',
'wizard.toast.auth_profile_created_title': 'Authorization profile created',
'wizard.toast.auth_profile_created_body': 'New authorization profile is ready and selected.',
'wizard.toast.auth_profile_error_title': 'Authorization profile creation failed',
@@ -2353,6 +2354,7 @@ var TRANSLATIONS = {
'wizard.error.secret_name_required': 'Нужно указать имя секрета.',
'wizard.error.secret_value_required': 'Нужно указать значение секрета.',
'wizard.error.save_upstream_first': 'Сначала сохраните API-хост после создания нового профиля авторизации.',
'wizard.error.execution_policy_invalid': 'Политика выполнения некорректна: нужны режим, deadline от 1 до 3600000 мс и поведение без Tasks.',
'wizard.toast.auth_profile_created_title': 'Профиль авторизации создан',
'wizard.toast.auth_profile_created_body': 'Новый профиль авторизации готов и уже выбран.',
'wizard.toast.auth_profile_error_title': 'Не удалось создать профиль авторизации',
+30
View File
@@ -214,10 +214,33 @@ function parseExecutionConfig(text) {
auth_profile_ref: authProfileRef,
headers: headers,
};
var executionPolicy = parseExecutionPolicy(value.execution_policy);
if (executionPolicy) {
config.execution_policy = executionPolicy;
}
return config;
}
// Story 4.1: the published execution policy must survive a wizard save. The editor is a free-form
// document, so unknown/legacy shapes stay untouched: only a fully valid policy is projected, and an
// explicit invalid value is reported instead of being silently replaced by defaults.
function parseExecutionPolicy(raw) {
if (raw === undefined || raw === null) return null;
if (typeof raw !== 'object' || Array.isArray(raw)) {
throw new Error(tKey('wizard.error.execution_policy_invalid'));
}
var mode = raw.mode === undefined ? 'synchronous' : raw.mode;
var fallback = raw.no_tasks_fallback === undefined ? 'synchronous' : raw.no_tasks_fallback;
var deadline = Number(raw.deadline_ms);
if (['synchronous', 'background', 'adaptive'].indexOf(mode) < 0
|| ['synchronous', 'unsupported'].indexOf(fallback) < 0
|| !Number.isInteger(deadline) || deadline < 1 || deadline > 3600000) {
throw new Error(tKey('wizard.error.execution_policy_invalid'));
}
return { mode: mode, deadline_ms: deadline, no_tasks_fallback: fallback };
}
function currentUpstream() {
var selected = upstreams.find(function(item) { return item.id === selectedUpstreamId; });
if (selected) return selected;
@@ -415,6 +438,13 @@ function executionConfigToEditorValue(config) {
var value = {
timeout_ms: config.timeout_ms,
};
if (config.execution_policy) {
value.execution_policy = {
mode: config.execution_policy.mode,
deadline_ms: config.execution_policy.deadline_ms,
no_tasks_fallback: config.execution_policy.no_tasks_fallback,
};
}
if (config.retry_policy) {
value.retry = { max_attempts: config.retry_policy.max_attempts };
}
@@ -0,0 +1,139 @@
const { test, expect } = require('@playwright/test');
const { getCurrentWorkspace, login } = require('./helpers');
// Story 4.1 (FR-21): the published execution policy is part of the Tool version and must survive a
// wizard save. The editor is a free-form document, so a no-op edit must not erase it and an explicit
// invalid value must be reported instead of being silently replaced by defaults.
function draftSnapshot(operationId) {
return {
id: operationId,
name: operationId,
display_name: 'Policy rates',
category: 'general',
protocol: 'rest',
target: {
kind: 'rest',
base_url: 'https://api.example.test',
method: 'GET',
path_template: '/rates',
static_headers: {},
},
input_schema: {
type: 'object',
description: null,
required: false,
nullable: false,
default_value: null,
fields: {},
items: null,
enum_values: [],
variants: [],
},
output_schema: {
type: 'object',
description: null,
required: false,
nullable: false,
default_value: null,
fields: {},
items: null,
enum_values: [],
variants: [],
},
input_mapping: { rules: [] },
output_mapping: { rules: [{ source: '$.response.body.rate', target: '$.output.rate', required: true }] },
execution_config: {
timeout_ms: 10000,
retry_policy: null,
auth_profile_ref: null,
headers: {},
approval_policy: null,
execution_policy: {
mode: 'adaptive',
deadline_ms: 45000,
no_tasks_fallback: 'unsupported',
},
},
tool_description: { title: 'Policy rates', description: 'Get rates.', tags: [], examples: [] },
wizard_state: { input_sample: {}, output_sample: {}, test_input: {}, import_findings: [] },
};
}
test('wizard round-trips the published execution policy and rejects an invalid edit', async ({ page }) => {
await login(page);
const workspace = await getCurrentWorkspace(page);
const operationId = 'policy_rates_41';
const operationPath = `/api/admin/workspaces/${encodeURIComponent(workspace.id)}/operations/${operationId}`;
const savedPayloads = [];
await page.route(operationPath, async (route) => {
if (route.request().method() === 'PATCH') {
savedPayloads.push(route.request().postDataJSON());
await route.fulfill({
contentType: 'application/json',
body: JSON.stringify({ operation_id: operationId, workspace_id: workspace.id, version: 2, status: 'draft' }),
});
return;
}
await route.fulfill({
contentType: 'application/json',
body: JSON.stringify({
id: operationId,
workspace_id: workspace.id,
name: operationId,
display_name: 'Policy rates',
category: 'general',
protocol: 'rest',
status: 'draft',
current_draft_version: 1,
draft_version_ref: { version: 1, status: 'draft' },
published_version_ref: null,
agent_refs: [],
}),
});
});
await page.route(`${operationPath}/versions/*`, async (route) => {
await route.fulfill({
contentType: 'application/json',
body: JSON.stringify({
operation_id: operationId,
workspace_id: workspace.id,
version: 1,
status: 'draft',
snapshot: draftSnapshot(operationId),
}),
});
});
await page.route('**/operations/analyze-quality', async (route) => {
await route.fulfill({ contentType: 'application/json', body: JSON.stringify({ blocking: false, findings: [] }) });
});
page.on('dialog', (dialog) => dialog.accept());
await page.goto(`/wizard/?mode=edit&operationId=${operationId}`);
await expect(page.locator('#tool-display-name')).toHaveValue('Policy rates');
const editorValue = await page.locator('#tool-exec-config').inputValue();
expect(editorValue).toContain('execution_policy');
expect(editorValue).toContain('adaptive');
expect(editorValue).toContain('45000');
expect(editorValue).toContain('unsupported');
// A real edit still saves the pinned policy unchanged.
await page.evaluate(() => window.CrankWizardShell.doGoToStep(4));
await page.locator('#tool-display-name').fill('Policy rates edited');
await page.locator('.btn-save-draft').click();
await expect.poll(() => savedPayloads.length).toBe(1);
expect(savedPayloads[0].execution_config.execution_policy).toEqual({
mode: 'adaptive',
deadline_ms: 45000,
no_tasks_fallback: 'unsupported',
});
// An explicit invalid policy is refused instead of being silently rewritten to defaults.
await page.evaluate(() => window.CrankWizardShell.doGoToStep(5));
await expect(page.locator('#tool-exec-config')).toBeVisible();
await page.locator('#tool-exec-config').fill('timeout_ms: 10000\nexecution_policy:\n mode: background\n deadline_ms: 0\n');
await page.locator('.btn-save-draft').click();
await expect(page.locator('#tool-exec-config')).toHaveValue(/deadline_ms: 0/);
expect(savedPayloads.length).toBe(1);
});
+4 -1
View File
@@ -272,7 +272,10 @@ fn record_unknown_approval_access_denial(reason: &'static str) {
);
}
fn allows_scope(scopes: &[PlatformApiKeyScope], required_scope: PlatformApiKeyScope) -> bool {
pub(super) fn allows_scope(
scopes: &[PlatformApiKeyScope],
required_scope: PlatformApiKeyScope,
) -> bool {
match required_scope {
PlatformApiKeyScope::Read => scopes.iter().any(|scope| {
matches!(
+120 -16
View File
@@ -30,6 +30,7 @@ mod approval_policy;
#[cfg(test)]
pub(crate) mod dispatch_observer;
mod invocation_history;
pub(crate) mod lifecycle;
mod metrics;
mod request;
mod response;
@@ -49,13 +50,16 @@ mod approvals;
mod auth_resolution;
mod dispatch;
mod endpoint;
mod legacy_dispatch;
mod task_worker;
mod tasks;
use self::approvals::{
approve_request, deny_request, get_approval_request, list_pending_approvals,
};
pub(super) use self::auth_resolution::resolve_operation_auth;
use self::endpoint::{mcp_delete, mcp_get, mcp_post};
pub use self::task_worker::TaskWorkerHandler;
pub use self::tasks::{TaskApplication, TaskApplicationError, TaskPrincipal};
// Test-only import surface: the BUILD-internal `app/tests.rs` module reaches these two
// through `super`, and the plain lib build does not use them (same pattern as
// `#[cfg(test)] use invocation_history::observe_invocation_history_outcome;` above).
@@ -82,6 +86,17 @@ pub(super) struct AppState {
session_slots: Arc<Semaphore>,
pub(super) credential_verifier: SharedMachineCredentialVerifier,
allowed_origins: AllowedOrigins,
/// Story 3.18: the bounded in-memory `logging/setLevel` opt-in of the legacy sessions. It is a
/// client-scoped level, never a process-global one, and it needs no schema migration.
pub(super) logging_levels: Arc<crate::protocol::LogLevelStore>,
/// Story 3.19: the bounded table of the legacy session that currently owns the notification
/// stream, so one session never broadcasts the same list-change signal into two streams.
pub(super) signal_streams: Arc<crate::protocol::StreamRegistry>,
/// Story 3.4: the bounded table of the `tools/call` requests a legacy client can still cancel.
pub(super) in_flight: Arc<self::lifecycle::InFlightCalls>,
/// Story 4.2: durable background Tasks over the same registry pool. The public wire stays
/// synchronous/unsupported; readiness depends on this mandatory store.
tasks: Arc<TaskApplication>,
}
#[derive(Clone)]
@@ -127,6 +142,7 @@ pub fn build_app(
DEFAULT_MAX_CONCURRENT_SESSIONS,
false,
None,
None,
)
}
@@ -182,6 +198,7 @@ pub fn build_app_with_background_workers_and_limits(
max_concurrent_sessions,
true,
None,
None,
)
}
@@ -217,6 +234,41 @@ pub fn build_app_with_artifacts(
max_concurrent_sessions,
true,
Some(artifact_store),
None,
)
}
/// Production launcher entry point with the durable Task application entry constructed by the
/// process composition root from the same registry pool and admission limits as the runtime.
#[allow(clippy::too_many_arguments)]
pub fn build_app_with_artifacts_and_tasks(
registry: PostgresRegistry,
refresh_interval: Duration,
public_base_url: Option<String>,
secret_crypto: SecretCrypto,
runtime: RuntimeExecutor,
api_rate_limiter: RequestRateLimiter,
coordination_store: Arc<dyn CoordinationStateStore>,
sessions: SharedSessionStore,
credential_verifier: SharedMachineCredentialVerifier,
max_concurrent_sessions: usize,
artifact_store: crank_artifacts::ArtifactStore,
tasks: Arc<TaskApplication>,
) -> Router {
build_app_inner(
registry,
refresh_interval,
public_base_url,
secret_crypto,
runtime,
api_rate_limiter,
coordination_store,
sessions,
credential_verifier,
max_concurrent_sessions,
true,
Some(artifact_store),
Some(tasks),
)
}
@@ -234,8 +286,23 @@ fn build_app_inner(
max_concurrent_sessions: usize,
start_background_workers: bool,
artifact_store: Option<crank_artifacts::ArtifactStore>,
tasks: Option<Arc<TaskApplication>>,
) -> Router {
let session_metrics = ActiveSessionMetrics::start(Arc::clone(&sessions));
let tasks = tasks.unwrap_or_else(|| {
let store = crank_registry::PostgresTaskStore::new(
registry.pool().clone(),
crank_registry::AdmissionLimits::default(),
crank_core::AdmissionSchedulerConfig::default(),
)
.expect("default admission limits and scheduler weights are valid");
Arc::new(TaskApplication::new(
registry.clone(),
refresh_interval,
Arc::clone(&coordination_store),
crank_runtime::TaskService::new(Arc::new(store)),
))
});
let state = Arc::new(AppState {
registry: registry.clone(),
catalog: PublishedToolCatalog::new(registry, refresh_interval, coordination_store),
@@ -248,6 +315,12 @@ fn build_app_inner(
session_slots: Arc::new(Semaphore::new(max_concurrent_sessions)),
credential_verifier,
allowed_origins: AllowedOrigins::new(public_base_url),
logging_levels: Arc::new(crate::protocol::LogLevelStore::new()),
signal_streams: Arc::new(crate::protocol::StreamRegistry::new()),
in_flight: Arc::new(lifecycle::InFlightCalls::new(
lifecycle::MAX_IN_FLIGHT_CALLS,
)),
tasks,
});
if start_background_workers {
spawn_approval_recovery(Arc::clone(&state));
@@ -296,24 +369,55 @@ async fn health() -> Json<Value> {
}
async fn readiness(State(state): State<Arc<AppState>>) -> Response {
match state.registry.ping().await {
Ok(()) => Json(json!({
let postgres_ready = state.registry.ping().await.is_ok();
// Story 4.1: admission state is a mandatory dependency of every execution, so readiness must
// reflect it rather than reporting ready on a bare `select 1`.
let admission_ready = state.registry.execution_admission_health().await.is_ok();
// Story 4.2: the durable Task store is mandatory for a durable acceptance acknowledgement.
let durable_tasks_ready = state.tasks.health().await.is_ok();
if postgres_ready && admission_ready && durable_tasks_ready {
return Json(json!({
"service": "mcp-server",
"status": "ready",
"checks": { "postgres": "ready" }
"checks": {
"postgres": "ready",
"execution_admission": "ready",
"durable_tasks": "ready"
}
}))
.into_response(),
Err(_) => (
StatusCode::SERVICE_UNAVAILABLE,
Json(json!({
"service": "mcp-server",
"status": "not_ready",
"checks": { "postgres": "not_ready" },
"error": "database is unavailable"
})),
)
.into_response(),
.into_response();
}
let mut checks = serde_json::Map::new();
checks.insert(
"postgres".to_owned(),
json!(if postgres_ready { "ready" } else { "not_ready" }),
);
checks.insert(
"execution_admission".to_owned(),
json!(if admission_ready {
"ready"
} else {
"not_ready"
}),
);
checks.insert(
"durable_tasks".to_owned(),
json!(if durable_tasks_ready {
"ready"
} else {
"not_ready"
}),
);
(
StatusCode::SERVICE_UNAVAILABLE,
Json(json!({
"service": "mcp-server",
"status": "not_ready",
"checks": checks,
"error": "a mandatory dependency is unavailable"
})),
)
.into_response()
}
#[cfg(test)]
@@ -19,18 +19,31 @@ pub(crate) async fn resolve_operation_auth(
workspace_id: &crank_core::WorkspaceId,
execution_config: &crank_core::ExecutionConfig,
) -> Result<Option<ResolvedAuth>, RuntimeError> {
if execution_config.auth_profile_ref.is_none() {
return Ok(None);
}
let span = Stage::AuthResolve.span();
let result = resolve_runtime_auth_for_task(
resolve_operation_auth_with(
&state.registry,
&state.secret_crypto,
workspace_id,
execution_config,
)
.instrument(span.clone())
.await;
.await
}
/// Same resolution for callers that hold the authoritative registry/secret-crypto pair without the
/// HTTP application state (the durable Task worker).
pub(crate) async fn resolve_operation_auth_with(
registry: &PostgresRegistry,
secret_crypto: &SecretCrypto,
workspace_id: &crank_core::WorkspaceId,
execution_config: &crank_core::ExecutionConfig,
) -> Result<Option<ResolvedAuth>, RuntimeError> {
if execution_config.auth_profile_ref.is_none() {
return Ok(None);
}
let span = Stage::AuthResolve.span();
let result =
resolve_runtime_auth_for_task(registry, secret_crypto, workspace_id, execution_config)
.instrument(span.clone())
.await;
match &result {
Ok(_) => StageOutcome::Success.record(&span),
Err(_) => {
+108 -20
View File
@@ -23,7 +23,9 @@ use crate::{
},
session::SessionState,
tool_error::execution_locale,
transport::{ResponseMode, json_response, session_id_from_headers, transport_response},
transport::{
ResponseMode, json_response, session_id_from_headers, sse_response, transport_response,
},
};
use axum::{
http::{HeaderMap, StatusCode},
@@ -37,8 +39,7 @@ use time::OffsetDateTime;
use super::response::{
completion_result, internal_jsonrpc_error, prompt_arguments_error_response, prompt_get_result,
prompts_list_result, resource_error_response, resource_read_result,
resource_templates_list_result, resources_list_result, tool_command_response,
tools_list_result,
resource_templates_list_result, resources_list_result, tools_list_result,
};
use super::{AgentRoutePath, AppState, TRANSPORT_SESSION_TTL_MS};
use crate::protocol::{ResourceTemplatesListError, list_resource_templates};
@@ -67,6 +68,12 @@ pub(super) async fn handle_modern_request(
.as_deref()
.unwrap_or(crate::jsonrpc::MODERN_PROTOCOL_VERSION);
// Story 3.18: the modern revision carries the log level of this request in
// `_meta["io.modelcontextprotocol/logLevel"]` (SEP-2577); the former `logging/setLevel` request
// does not exist here. Absent, non-string or unknown values mean "not opted in", so no
// `notifications/message` is emitted for this request.
crate::protocol::set_active_level(crate::protocol::modern_request_level(message));
// A notification carries no `id`. It is acknowledged before the request-only header rules: the
// mandatory `Mcp-Method`/`Mcp-Name` pair applies to requests, not to notifications (protocol era
// matrix row 18). The envelope, identity and mixed-carrier checks — and authentication — have
@@ -148,6 +155,8 @@ pub(super) async fn handle_modern_request(
locale: execution_locale(message),
},
ToolsCommand::List(list_command),
// A list command has no progress or cancellation lifecycle (Story 3.4).
None,
)
.await;
match outcome {
@@ -212,30 +221,35 @@ pub(super) async fn handle_modern_request(
// session-less, so its calls cannot advertise elicitation support.
let mut arguments = tool_call_params.arguments.unwrap_or_else(|| json!({}));
let confirmation_token = take_confirmation_token(&mut arguments);
let outcome = handle_tools_command(
state,
ServiceContext {
workspace_slug: &path.workspace_slug,
agent_slug: &path.agent_slug,
credential,
correlation: transport_correlation,
// Story 3.4: the client's own `_meta.progressToken` of this request, validated and bounded.
let progress = crate::app::lifecycle::token_from_meta(tool_call_params.meta.as_ref());
super::lifecycle::run_tool_call(
Arc::clone(state),
crate::app::lifecycle::ToolCallRequest {
workspace_slug: path.workspace_slug.clone(),
agent_slug: path.agent_slug.clone(),
credential: credential.clone(),
correlation: transport_correlation.clone(),
continuation: ContinuationSupport::ElicitationUnsupported,
locale: execution_locale(message),
},
ToolsCommand::Call(ToolCallCommand {
crate::app::lifecycle::ToolCallWire {
message: message.clone(),
protocol_version: version.to_owned(),
response_mode,
progress,
session_id: None,
// The modern revision cancels by closing this request's response stream, so no
// session-scoped cancellation table is involved.
in_flight: None,
},
ToolCallCommand {
tool_name: tool_call_params.name,
arguments,
confirmation_token,
}),
)
.await;
tool_command_response(
message,
response_mode,
version,
transport_correlation,
outcome,
},
)
.await
}
DispatchDecision::Allowed {
handler: MethodHandler::ResourceList,
@@ -652,6 +666,80 @@ pub(super) async fn handle_modern_request(
}
}
}
// Story 3.19: the modern long-lived notification stream. The response is itself the SSE
// stream that stays open, so it is built here instead of going through the one-shot
// response helpers: the acknowledgment is its first frame, then the coalesced list-change
// signals, and closing the HTTP stream ends the producer.
DispatchDecision::Allowed {
handler: MethodHandler::SubscriptionsListen,
..
} => {
// A notification stream is an SSE body by construction: a client that did not offer
// `text/event-stream` could not receive it.
if !crate::transport::accepts_event_stream(headers) {
return StatusCode::NOT_ACCEPTABLE.into_response();
}
let filter: crate::protocol::ListenParams =
match serde_json::from_value(params(message)) {
Ok(filter) => filter,
Err(_) => {
return transport_response(
StatusCode::OK,
jsonrpc_error(
request_id(message),
-32602,
crate::protocol::INVALID_LISTEN_FILTER,
),
response_mode,
None,
Some(version),
);
}
};
let accepted = match filter.notifications.accepted() {
crate::protocol::FilterOutcome::Accepted(accepted) => accepted,
crate::protocol::FilterOutcome::Invalid(detail) => {
return transport_response(
StatusCode::OK,
jsonrpc_error(request_id(message), -32602, detail),
response_mode,
None,
Some(version),
);
}
};
// The subscription id is the JSON-RPC id of this request; a value whose encoding is not
// bounded cannot be a frame identity carrier and is refused before the stream opens.
let Some(subscription_id) = crate::protocol::subscription_id(&request_id(message))
else {
return transport_response(
StatusCode::OK,
jsonrpc_error(request_id(message), -32600, "invalid JSON-RPC message"),
response_mode,
None,
Some(version),
);
};
let Ok(permit) = super::metrics::ActiveStreamGuard::try_acquire(&state.session_slots)
else {
return StatusCode::TOO_MANY_REQUESTS.into_response();
};
let acknowledged = crate::protocol::acknowledged_frame(&subscription_id, accepted);
let receiver = crate::protocol::spawn_signal_watch(
Arc::clone(state),
crate::protocol::Watch {
workspace_slug: path.workspace_slug.clone(),
agent_slug: path.agent_slug.clone(),
signals: accepted,
wire: crate::protocol::Wire::Modern(subscription_id),
legacy_stream: None,
deadline: None,
},
);
let stream = crate::protocol::frames_stream(vec![acknowledged], receiver, permit);
sse_response(StatusCode::OK, stream, None, Some(version))
}
// The standard Prompt requests are registered capabilities now; any other
// unregistered/wrong-era method goes straight to the era-correct unsupported answer.
_ => crate::protocol::modern_method_not_found_response(message),
+92 -659
View File
@@ -3,55 +3,37 @@
//!
//! Moved verbatim out of `app.rs` (Story 3.6, kanban t_987bf309); `mcp_post_response` is reached from
//! `build_app_inner` via `mcp_post`, hence `pub(super)`. No behaviour changed.
//!
//! The legacy-era dispatch itself lives in `super::legacy_dispatch` (Story 3.18 code-health, kanban
//! t_d1b45ddc) to keep this transport entry file inside the 1000-line limit.
use super::metrics::{ActiveStreamGuard, McpRequestMetrics};
use super::request::ToolCallParams;
use super::response::take_confirmation_token;
use super::stages::{enforce_traced_rate_limit, require_traced_machine_access};
use crate::{
jsonrpc::{
LEGACY_PROTOCOL_VERSION, is_notification, is_request, jsonrpc_error, jsonrpc_result,
method_name, params, request_id,
},
jsonrpc::{jsonrpc_error, method_name, request_id},
protocol::{
CompletionCommand, CompletionOutcome, ContinuationSupport, DispatchDecision, EnvelopeKind,
Kind, MethodHandler, PromptGetCommand, PromptsCommand, PromptsListCommand, PromptsOutcome,
RequestEra, ResourceReadCommand, ResourcesCommand, ResourcesListCommand, ResourcesOutcome,
ServiceContext, ToolCallCommand, ToolsCommand, ToolsListCommand, ToolsOutcome,
ambiguous_session_carrier_response, handle_completion_command, handle_prompts_command,
handle_resources_command, handle_tools_command, resolve_request_identity,
EnvelopeKind, RequestEra, ambiguous_session_carrier_response, resolve_request_identity,
},
rate_limit::{rate_limited_jsonrpc_response, rate_limited_status_response},
request_context::RequestContext,
tool_error::execution_locale,
transport::{
negotiate_post_response_mode, session_id_from_headers, sse_response, transport_response,
validate_get_accept_header, validate_origin, validate_session_protocol_version,
with_request_id_header,
ResponseMode, accepts_event_stream, negotiate_post_response_mode, session_id_from_headers,
sse_response, transport_response, validate_get_accept_header, validate_origin,
validate_session_protocol_version, with_request_id_header,
},
};
use axum::{
Json,
extract::{Extension, Path, State, rejection::JsonRejection},
http::{HeaderMap, StatusCode},
response::{IntoResponse, Response, sse::Event},
response::{IntoResponse, Response},
};
use crank_core::{CorrelationContext, PlatformApiKeyScope};
use futures_util::stream;
use serde_json::{Value, json};
use std::{convert::Infallible, sync::Arc, time::Duration};
use serde_json::Value;
use std::{sync::Arc, time::Duration, time::Instant};
use tracing::info;
use super::dispatch::{
handle_initialize, handle_initialized_notification, handle_modern_request,
require_initialized_session,
};
use super::response::{
completion_result, internal_jsonrpc_error, prompt_arguments_error_response, prompt_get_result,
prompts_list_result, resource_error_response, resource_read_result,
resource_templates_list_result, resources_list_result, tool_command_response,
tools_list_result,
};
use super::dispatch::handle_modern_request;
use super::{AgentRoutePath, AppState, TRANSPORT_SESSION_TTL_MS};
pub(super) async fn mcp_get(
Path(path): Path<AgentRoutePath>,
@@ -120,11 +102,32 @@ pub(super) async fn mcp_get(
return StatusCode::TOO_MANY_REQUESTS.into_response();
};
let stream = stream::unfold(permit, |permit| async move {
tokio::time::sleep(Duration::from_millis(TRANSPORT_SESSION_TTL_MS)).await;
drop(permit);
None::<(Result<Event, Infallible>, _)>
});
// Story 3.19: the pinned legacy transport forbids broadcasting one JSON-RPC message across the
// SSE streams of a session, so exactly one admitted `GET` receives the Resource/Prompt/Tool
// list-change signals. A newer `GET` takes them over and the older stream simply ends; a
// `DELETE` retires the producer, so a stale generation never writes into a reconnect.
let Some(generation) = state.signal_streams.admit(&session_id) else {
// The bounded owner table is contended: refuse this admission instead of waiting.
return StatusCode::TOO_MANY_REQUESTS.into_response();
};
let receiver = crate::protocol::spawn_signal_watch(
Arc::clone(&state),
crate::protocol::Watch {
workspace_slug: path.workspace_slug.clone(),
agent_slug: path.agent_slug.clone(),
signals: crate::protocol::Signals::ALL,
wire: crate::protocol::Wire::Legacy,
legacy_stream: Some((
Arc::clone(&state.signal_streams),
session_id.clone(),
generation,
)),
deadline: Some(Instant::now() + Duration::from_millis(TRANSPORT_SESSION_TTL_MS)),
},
);
// The stream permit lives in the response stream: a client that closes the SSE connection ends
// the producer and releases the slot, and no signal is queued for a reader that is gone.
let stream = crate::protocol::frames_stream(Vec::new(), receiver, permit);
sse_response(
StatusCode::OK,
@@ -177,6 +180,7 @@ pub(super) async fn mcp_delete(
match state.sessions.delete(&session_id).await {
Ok(true) => {
state.signal_streams.retire(&session_id);
state.session_metrics.refresh();
StatusCode::NO_CONTENT.into_response()
}
@@ -224,8 +228,14 @@ pub(super) async fn mcp_post(
"mcp request received"
);
let response =
mcp_post_response(&path, state, &headers, &message, &transport_correlation).await;
let response = crate::protocol::logging_scope(mcp_post_response(
&path,
state,
&headers,
&message,
&transport_correlation,
))
.await;
request_metrics.complete(&response);
with_request_id_header(response, &transport_request_id)
}
@@ -246,10 +256,17 @@ async fn mcp_post_response(
return status.into_response();
}
let response_mode = match negotiate_post_response_mode(headers) {
let mut response_mode = match negotiate_post_response_mode(headers) {
Ok(mode) => mode,
Err(status) => return status.into_response(),
};
// Story 3.18: a server notification is delivered only on the SSE stream, so the logging
// projection records whether this request offered it. This is an admission fact of the same
// `Accept` header the mode above was negotiated from.
if accepts_event_stream(headers) {
crate::protocol::set_sse_acceptable();
}
// A conflicting duplicate session carrier is an ambiguous identity: it must not be resolved to
// one of its values. The carrier itself cannot tell which era the request belongs to, so the
// diagnostic is read from the non-ambiguous part of the request (matrix row 17) — a consumer of
@@ -327,6 +344,17 @@ async fn mcp_post_response(
}
}
// Story 3.18: the legacy logging opt-in belongs to this session, so it is read before dispatch.
// A session that never sent `logging/setLevel` has no level, which is what the pinned schema
// describes as "not opted in" (no notification is delivered).
if matches!(identity.era, RequestEra::Legacy) {
crate::protocol::set_active_level(
request_session_id
.as_deref()
.and_then(|session_id| state.logging_levels.get(session_id)),
);
}
let required_scope = match method_name(message) {
Some("tools/call") => PlatformApiKeyScope::Write,
_ => PlatformApiKeyScope::Read,
@@ -337,6 +365,19 @@ async fn mcp_post_response(
Err(error) => return error.into_response(),
};
// Story 3.4 correction: the official SDK lists the media types JSON-first. For a `tools/call`
// that carries a validated progress token and offered `text/event-stream`, the stream is the only
// carrier that can deliver the progress the client asked for, so the media-type order is not a
// preference for the JSON body here. Every other request — another method, no token, an unusable
// token, a JSON-only `Accept` — keeps the negotiated mode, and no admission check moved.
if matches!(response_mode, ResponseMode::Json)
&& method_name(message) == Some("tools/call")
&& accepts_event_stream(headers)
&& crate::app::lifecycle::token_from_meta(message.pointer("/params/_meta")).is_some()
{
response_mode = ResponseMode::Sse;
}
if matches!(identity.era, RequestEra::Modern) {
// ERA_MODERN_ENTER (spec section 7.2): this request entered the modern era-adapter.
#[cfg(test)]
@@ -358,625 +399,17 @@ async fn mcp_post_response(
#[cfg(test)]
crate::app::dispatch_observer::era_legacy_enter();
// R5/F5: the owning registration decides *which handler* serves this method in the request's
// revision. The adapter holds no method list of its own — the method-to-handler mapping lives in
// `protocol::registry` and is read through `dispatch_decision`, so a newly registered method
// really reaches its handler and removing a descriptor removes the dispatch.
let registration_revision = protocol_version
.as_deref()
.unwrap_or(LEGACY_PROTOCOL_VERSION);
let dispatch = crate::protocol::dispatch_decision(
method_name(message).unwrap_or_default(),
registration_revision,
);
match dispatch {
// G2: the *registered kind* of the matched method, not the envelope shape, decides which
// legacy lifecycle action runs. `initialize` is a request and `notifications/initialized` is a
// notification: the two share one handler but are not interchangeable, so an `initialize`
// notification may not initialize an existing session and a `notifications/initialized`
// request may not create one.
DispatchDecision::Allowed {
handler: MethodHandler::Lifecycle,
kind: Kind::Method,
} if is_request(message) => handle_initialize(state, path, message, response_mode).await,
DispatchDecision::Allowed {
handler: MethodHandler::Lifecycle,
kind: Kind::Notification,
} if is_notification(message) => {
handle_initialized_notification(state, path, headers).await
}
DispatchDecision::Allowed {
handler: MethodHandler::Ping,
..
} if is_request(message) => {
let session = match require_initialized_session(&state, path, headers, message).await {
Ok(session) => session,
Err(response) => return response,
};
transport_response(
StatusCode::OK,
jsonrpc_result(
request_id(message),
json!({ "protocolVersion": session.protocol_version }),
),
response_mode,
None,
Some(&session.protocol_version),
)
}
DispatchDecision::Allowed {
handler: MethodHandler::ToolList,
..
} if is_request(message) => {
let session = match require_initialized_session(&state, path, headers, message).await {
Ok(session) => session,
Err(response) => return response,
};
// The adapter parses the wire form and maps the neutral outcome back; the shared Tools
// service never sees the envelope, the response mode or the protocol revision.
let list_command: ToolsListCommand = match serde_json::from_value(params(message)) {
Ok(value) => value,
Err(_) => {
return transport_response(
StatusCode::OK,
jsonrpc_error(request_id(message), -32602, "invalid tools/list parameters"),
response_mode,
None,
Some(&session.protocol_version),
);
}
};
let outcome = handle_tools_command(
&state,
ServiceContext {
workspace_slug: &session.workspace_slug,
agent_slug: &session.agent_slug,
credential: &credential,
correlation: transport_correlation,
continuation: ContinuationSupport::from_supported(session.supports_elicitation),
locale: execution_locale(message),
},
ToolsCommand::List(list_command),
)
.await;
match outcome {
ToolsOutcome::List { tools, next_cursor } => transport_response(
StatusCode::OK,
jsonrpc_result(request_id(message), tools_list_result(tools, next_cursor)),
response_mode,
None,
Some(&session.protocol_version),
),
ToolsOutcome::InvalidParams(message_text) => transport_response(
StatusCode::OK,
jsonrpc_error(request_id(message), -32602, message_text),
response_mode,
None,
Some(&session.protocol_version),
),
ToolsOutcome::CatalogUnavailable => {
internal_jsonrpc_error(message, "published catalog is unavailable")
}
// A list command never produces a call outcome; it is answered with the adapter's
// internal error instead of being silently ignored.
ToolsOutcome::Call(_) => internal_jsonrpc_error(
message,
"unexpected shared tools outcome for a tools/list",
),
}
}
DispatchDecision::Allowed {
handler: MethodHandler::ToolCall,
..
} if is_request(message) => {
let session = match require_initialized_session(&state, path, headers, message).await {
Ok(session) => session,
Err(response) => return response,
};
let tool_call_params: ToolCallParams = match serde_json::from_value(params(message)) {
Ok(value) => value,
Err(_) => {
return transport_response(
StatusCode::OK,
jsonrpc_error(request_id(message), -32602, "invalid tool call parameters"),
response_mode,
None,
Some(&session.protocol_version),
);
}
};
// The adapter owns the wire form: it parses the call parameters, removes the reserved
// confirmation marker and the shared Tools service never sees the envelope, the
// response mode or the protocol revision (spec section 6.2).
let mut arguments = tool_call_params.arguments.unwrap_or_else(|| json!({}));
let confirmation_token = take_confirmation_token(&mut arguments);
let outcome = handle_tools_command(
&state,
ServiceContext {
workspace_slug: &session.workspace_slug,
agent_slug: &session.agent_slug,
credential: &credential,
correlation: transport_correlation,
continuation: ContinuationSupport::from_supported(session.supports_elicitation),
locale: execution_locale(message),
},
ToolsCommand::Call(ToolCallCommand {
tool_name: tool_call_params.name,
arguments,
confirmation_token,
}),
)
.await;
tool_command_response(
message,
response_mode,
&session.protocol_version,
transport_correlation,
outcome,
)
}
DispatchDecision::Allowed {
handler: MethodHandler::ResourceList,
..
} if is_request(message) => {
let session = match require_initialized_session(&state, path, headers, message).await {
Ok(session) => session,
Err(response) => return response,
};
let list_command: ResourcesListCommand = match serde_json::from_value(params(message)) {
Ok(value) => value,
Err(_) => {
return transport_response(
StatusCode::OK,
jsonrpc_error(
request_id(message),
-32602,
"invalid resources/list parameters",
),
response_mode,
None,
Some(&session.protocol_version),
);
}
};
let outcome = handle_resources_command(
&state,
ServiceContext {
workspace_slug: &session.workspace_slug,
agent_slug: &session.agent_slug,
credential: &credential,
correlation: transport_correlation,
continuation: ContinuationSupport::from_supported(session.supports_elicitation),
locale: execution_locale(message),
},
&session.protocol_version,
ResourcesCommand::List(list_command),
)
.await;
match outcome {
ResourcesOutcome::Listed {
resources,
next_cursor,
} => transport_response(
StatusCode::OK,
jsonrpc_result(
request_id(message),
resources_list_result(resources, next_cursor),
),
response_mode,
None,
Some(&session.protocol_version),
),
ResourcesOutcome::InvalidParams(detail) => transport_response(
StatusCode::OK,
jsonrpc_error(request_id(message), -32602, detail),
response_mode,
None,
Some(&session.protocol_version),
),
_ => internal_jsonrpc_error(message, "published resources are unavailable"),
}
}
DispatchDecision::Allowed {
handler: MethodHandler::ResourceRead,
..
} if is_request(message) => {
let session = match require_initialized_session(&state, path, headers, message).await {
Ok(session) => session,
Err(response) => return response,
};
let read_command: ResourceReadCommand = match serde_json::from_value(params(message)) {
Ok(value) => value,
Err(_) => {
return transport_response(
StatusCode::OK,
jsonrpc_error(
request_id(message),
-32602,
"invalid resources/read parameters",
),
response_mode,
None,
Some(&session.protocol_version),
);
}
};
let outcome = handle_resources_command(
&state,
ServiceContext {
workspace_slug: &session.workspace_slug,
agent_slug: &session.agent_slug,
credential: &credential,
correlation: transport_correlation,
continuation: ContinuationSupport::from_supported(session.supports_elicitation),
locale: execution_locale(message),
},
&session.protocol_version,
ResourcesCommand::Read(read_command),
)
.await;
match outcome {
ResourcesOutcome::Read { contents } => transport_response(
StatusCode::OK,
jsonrpc_result(request_id(message), resource_read_result(contents)),
response_mode,
None,
Some(&session.protocol_version),
),
ResourcesOutcome::NotAvailable => transport_response(
StatusCode::OK,
jsonrpc_error(
request_id(message),
-32602,
"the requested resource is not available",
),
response_mode,
None,
Some(&session.protocol_version),
),
ResourcesOutcome::InvalidParams(detail) => transport_response(
StatusCode::OK,
jsonrpc_error(request_id(message), -32602, detail),
response_mode,
None,
Some(&session.protocol_version),
),
ResourcesOutcome::ExecutionError(contract) => resource_error_response(
message,
response_mode,
&session.protocol_version,
contract,
),
_ => internal_jsonrpc_error(message, "published resources are unavailable"),
}
}
// Story 3.11: the shared service returns exact bound Published templates; the legacy adapter
// retains its earlier result shape and still requires the initialized session.
DispatchDecision::Allowed {
handler: MethodHandler::ResourceTemplateList,
..
} if is_request(message) => {
let session = match require_initialized_session(&state, path, headers, message).await {
Ok(session) => session,
Err(response) => return response,
};
let command: ResourcesListCommand = match serde_json::from_value(params(message)) {
Ok(command) => command,
Err(_) => {
return transport_response(
StatusCode::OK,
jsonrpc_error(
request_id(message),
-32602,
"invalid resources/templates/list parameters",
),
response_mode,
None,
Some(&session.protocol_version),
);
}
};
let (templates, next_cursor) = match crate::protocol::list_resource_templates(
&state,
&path.workspace_slug,
&path.agent_slug,
&session.protocol_version,
command,
)
.await
{
Ok(page) => page,
Err(crate::protocol::ResourceTemplatesListError::InvalidCursor) => {
return transport_response(
StatusCode::OK,
jsonrpc_error(
request_id(message),
-32602,
"invalid resources/templates/list cursor",
),
response_mode,
None,
Some(&session.protocol_version),
);
}
Err(crate::protocol::ResourceTemplatesListError::Unavailable) => {
return internal_jsonrpc_error(message, "published templates are unavailable");
}
};
transport_response(
StatusCode::OK,
jsonrpc_result(
request_id(message),
resource_templates_list_result(templates, next_cursor),
),
response_mode,
None,
Some(&session.protocol_version),
)
}
// Story 3.14: `prompts/list` / `prompts/get` of the initialized legacy session, served from
// the same exact bound Published Prompt Versions as the modern era.
DispatchDecision::Allowed {
handler: MethodHandler::PromptList,
..
} if is_request(message) => {
let session = match require_initialized_session(&state, path, headers, message).await {
Ok(session) => session,
Err(response) => return response,
};
let command: PromptsListCommand = match serde_json::from_value(params(message)) {
Ok(command) => command,
Err(_) => {
return transport_response(
StatusCode::OK,
jsonrpc_error(
request_id(message),
-32602,
"invalid prompts/list parameters",
),
response_mode,
None,
Some(&session.protocol_version),
);
}
};
let outcome = handle_prompts_command(
&state,
ServiceContext {
workspace_slug: &session.workspace_slug,
agent_slug: &session.agent_slug,
credential: &credential,
correlation: transport_correlation,
continuation: ContinuationSupport::from_supported(session.supports_elicitation),
locale: execution_locale(message),
},
&session.protocol_version,
PromptsCommand::List(command),
)
.await;
match outcome {
PromptsOutcome::Listed {
prompts,
next_cursor,
} => transport_response(
StatusCode::OK,
jsonrpc_result(
request_id(message),
prompts_list_result(prompts, next_cursor),
),
response_mode,
None,
Some(&session.protocol_version),
),
PromptsOutcome::InvalidParams(detail) => transport_response(
StatusCode::OK,
jsonrpc_error(request_id(message), -32602, detail),
response_mode,
None,
Some(&session.protocol_version),
),
_ => internal_jsonrpc_error(message, "published prompts are unavailable"),
}
}
DispatchDecision::Allowed {
handler: MethodHandler::PromptGet,
..
} if is_request(message) => {
let session = match require_initialized_session(&state, path, headers, message).await {
Ok(session) => session,
Err(response) => return response,
};
let command: PromptGetCommand = match serde_json::from_value(params(message)) {
Ok(command) => command,
Err(_) => {
return transport_response(
StatusCode::OK,
jsonrpc_error(
request_id(message),
-32602,
"invalid prompts/get parameters",
),
response_mode,
None,
Some(&session.protocol_version),
);
}
};
let outcome = handle_prompts_command(
&state,
ServiceContext {
workspace_slug: &session.workspace_slug,
agent_slug: &session.agent_slug,
credential: &credential,
correlation: transport_correlation,
continuation: ContinuationSupport::from_supported(session.supports_elicitation),
locale: execution_locale(message),
},
&session.protocol_version,
PromptsCommand::Get(command),
)
.await;
match outcome {
PromptsOutcome::Got {
description,
messages,
} => transport_response(
StatusCode::OK,
jsonrpc_result(
request_id(message),
prompt_get_result(description, messages),
),
response_mode,
None,
Some(&session.protocol_version),
),
PromptsOutcome::NotAvailable => transport_response(
StatusCode::OK,
jsonrpc_error(
request_id(message),
-32602,
"the requested prompt is not available",
),
response_mode,
None,
Some(&session.protocol_version),
),
PromptsOutcome::ArgumentsRejected { reason, argument } => {
prompt_arguments_error_response(
message,
response_mode,
&session.protocol_version,
reason,
argument,
)
}
PromptsOutcome::DependencyUnavailable { position } => transport_response(
StatusCode::OK,
jsonrpc_error(
request_id(message),
-32603,
format!("the pinned dependency at dependencies[{position}] is unavailable"),
),
response_mode,
None,
Some(&session.protocol_version),
),
PromptsOutcome::InvalidParams(detail) => transport_response(
StatusCode::OK,
jsonrpc_error(request_id(message), -32602, detail),
response_mode,
None,
Some(&session.protocol_version),
),
_ => internal_jsonrpc_error(message, "published prompts are unavailable"),
}
}
// Story 3.17: `completion/complete` of the initialized legacy session. The shared service is
// the same one the modern era uses; only the result envelope differs (no `resultType`).
DispatchDecision::Allowed {
handler: MethodHandler::Completion,
..
} if is_request(message) => {
let session = match require_initialized_session(&state, path, headers, message).await {
Ok(session) => session,
Err(response) => return response,
};
let command: CompletionCommand = match serde_json::from_value(params(message)) {
Ok(command) => command,
Err(_) => {
return transport_response(
StatusCode::OK,
jsonrpc_error(
request_id(message),
-32602,
"invalid completion/complete parameters",
),
response_mode,
None,
Some(&session.protocol_version),
);
}
};
let outcome = handle_completion_command(
&state,
ServiceContext {
workspace_slug: &session.workspace_slug,
agent_slug: &session.agent_slug,
credential: &credential,
correlation: transport_correlation,
continuation: ContinuationSupport::from_supported(session.supports_elicitation),
locale: execution_locale(message),
},
&session.protocol_version,
command,
)
.await;
match outcome {
CompletionOutcome::Completed {
values,
total,
has_more,
} => transport_response(
StatusCode::OK,
jsonrpc_result(
request_id(message),
completion_result(values, total, has_more),
),
response_mode,
None,
Some(&session.protocol_version),
),
CompletionOutcome::InvalidParams(detail) => transport_response(
StatusCode::OK,
jsonrpc_error(request_id(message), -32602, detail),
response_mode,
None,
Some(&session.protocol_version),
),
CompletionOutcome::NotAvailable => transport_response(
StatusCode::OK,
jsonrpc_error(
request_id(message),
-32602,
"the completion reference is not available",
),
response_mode,
None,
Some(&session.protocol_version),
),
CompletionOutcome::CatalogUnavailable => {
internal_jsonrpc_error(message, "published completion sources are unavailable")
}
}
}
// A notification whose method the registration does not map to this adapter's handlers is
// acknowledged without being dispatched, exactly as before.
_ if is_notification(message) => StatusCode::ACCEPTED.into_response(),
// Every remaining decision (unregistered, wrong era, or a handler this adapter does not
// serve) is a method-level rejection: no handler, no service entry, no outbound call.
_ => match method_name(message) {
Some(method) => transport_response(
StatusCode::OK,
jsonrpc_error(
request_id(message),
-32601,
format!("method {method} is not supported"),
),
response_mode,
None,
protocol_version.as_deref(),
),
None => transport_response(
StatusCode::BAD_REQUEST,
jsonrpc_error(Value::Null, -32600, "invalid JSON-RPC message"),
response_mode,
None,
protocol_version.as_deref(),
),
},
}
// R5/F5: the owning registration decides which handler serves this method in the request's
// revision; that legacy dispatch lives in `super::legacy_dispatch`.
super::legacy_dispatch::handle_legacy_request(
state,
path,
headers,
message,
response_mode,
&identity,
&credential,
transport_correlation,
)
.await
}
@@ -0,0 +1,794 @@
//! Legacy-era JSON-RPC dispatch: the registered method-to-handler mapping, the
//! initialized-session guard and every legacy envelope the transport endpoint answers.
//!
//! Moved verbatim out of `app/endpoint.rs` (Story 3.18 code-health, kanban t_d1b45ddc) so the
//! transport entry file stays inside the 1000-line limit; the entry point is `pub(super)`.
//! No behaviour changed.
use std::sync::Arc;
use axum::{
http::{HeaderMap, StatusCode},
response::{IntoResponse, Response},
};
use crank_core::CorrelationContext;
use serde_json::{Value, json};
use super::dispatch::{
handle_initialize, handle_initialized_notification, require_initialized_session,
};
use super::request::ToolCallParams;
use super::response::{
completion_result, internal_jsonrpc_error, prompt_arguments_error_response, prompt_get_result,
prompts_list_result, resource_error_response, resource_read_result,
resource_templates_list_result, resources_list_result, take_confirmation_token,
tools_list_result,
};
use super::{AgentRoutePath, AppState};
use crate::{
auth::VerifiedMachineCredential,
jsonrpc::{
LEGACY_PROTOCOL_VERSION, is_notification, is_request, jsonrpc_error, jsonrpc_result,
method_name, params, request_id,
},
protocol::{
CompletionCommand, CompletionOutcome, ContinuationSupport, DispatchDecision,
EVENT_LEVEL_APPLIED, INVALID_LEVEL, Kind, LoggingOutcome, MethodHandler, PromptGetCommand,
PromptsCommand, PromptsListCommand, PromptsOutcome, RequestIdentity, ResourceReadCommand,
ResourcesCommand, ResourcesListCommand, ResourcesOutcome, ServiceContext, SetLevelCommand,
ToolCallCommand, ToolsCommand, ToolsListCommand, ToolsOutcome, handle_completion_command,
handle_logging_command, handle_prompts_command, handle_resources_command,
handle_tools_command,
},
tool_error::execution_locale,
transport::{ResponseMode, session_id_from_headers, transport_response},
};
// The transport invariants of one request travel together here; a parameter struct would only
// duplicate them (the same shape every other dispatch entry point in this crate uses).
#[allow(clippy::too_many_arguments)]
pub(super) async fn handle_legacy_request(
state: Arc<AppState>,
path: &AgentRoutePath,
headers: &HeaderMap,
message: &Value,
response_mode: ResponseMode,
identity: &RequestIdentity,
credential: &VerifiedMachineCredential,
transport_correlation: &CorrelationContext,
) -> Response {
let protocol_version = identity.version.as_deref();
// R5/F5: the owning registration decides *which handler* serves this method in the request's
// revision. The adapter holds no method list of its own — the method-to-handler mapping lives in
// `protocol::registry` and is read through `dispatch_decision`, so a newly registered method
// really reaches its handler and removing a descriptor removes the dispatch.
let registration_revision = protocol_version.unwrap_or(LEGACY_PROTOCOL_VERSION);
let dispatch = crate::protocol::dispatch_decision(
method_name(message).unwrap_or_default(),
registration_revision,
);
match dispatch {
// G2: the *registered kind* of the matched method, not the envelope shape, decides which
// legacy lifecycle action runs. `initialize` is a request and `notifications/initialized` is a
// notification: the two share one handler but are not interchangeable, so an `initialize`
// notification may not initialize an existing session and a `notifications/initialized`
// request may not create one.
DispatchDecision::Allowed {
handler: MethodHandler::Lifecycle,
kind: Kind::Method,
} if is_request(message) => handle_initialize(state, path, message, response_mode).await,
DispatchDecision::Allowed {
handler: MethodHandler::Lifecycle,
kind: Kind::Notification,
} if is_notification(message) => {
handle_initialized_notification(state, path, headers).await
}
DispatchDecision::Allowed {
handler: MethodHandler::Ping,
..
} if is_request(message) => {
let session = match require_initialized_session(&state, path, headers, message).await {
Ok(session) => session,
Err(response) => return response,
};
transport_response(
StatusCode::OK,
jsonrpc_result(
request_id(message),
json!({ "protocolVersion": session.protocol_version }),
),
response_mode,
None,
Some(&session.protocol_version),
)
}
DispatchDecision::Allowed {
handler: MethodHandler::ToolList,
..
} if is_request(message) => {
let session = match require_initialized_session(&state, path, headers, message).await {
Ok(session) => session,
Err(response) => return response,
};
// The adapter parses the wire form and maps the neutral outcome back; the shared Tools
// service never sees the envelope, the response mode or the protocol revision.
let list_command: ToolsListCommand = match serde_json::from_value(params(message)) {
Ok(value) => value,
Err(_) => {
return transport_response(
StatusCode::OK,
jsonrpc_error(request_id(message), -32602, "invalid tools/list parameters"),
response_mode,
None,
Some(&session.protocol_version),
);
}
};
let outcome = handle_tools_command(
&state,
ServiceContext {
workspace_slug: &session.workspace_slug,
agent_slug: &session.agent_slug,
credential,
correlation: transport_correlation,
continuation: ContinuationSupport::from_supported(session.supports_elicitation),
locale: execution_locale(message),
},
ToolsCommand::List(list_command),
// A list command has no progress or cancellation lifecycle (Story 3.4).
None,
)
.await;
match outcome {
ToolsOutcome::List { tools, next_cursor } => transport_response(
StatusCode::OK,
jsonrpc_result(request_id(message), tools_list_result(tools, next_cursor)),
response_mode,
None,
Some(&session.protocol_version),
),
ToolsOutcome::InvalidParams(message_text) => transport_response(
StatusCode::OK,
jsonrpc_error(request_id(message), -32602, message_text),
response_mode,
None,
Some(&session.protocol_version),
),
ToolsOutcome::CatalogUnavailable => {
internal_jsonrpc_error(message, "published catalog is unavailable")
}
// A list command never produces a call outcome; it is answered with the adapter's
// internal error instead of being silently ignored.
ToolsOutcome::Call(_) => internal_jsonrpc_error(
message,
"unexpected shared tools outcome for a tools/list",
),
}
}
DispatchDecision::Allowed {
handler: MethodHandler::ToolCall,
..
} if is_request(message) => {
let session = match require_initialized_session(&state, path, headers, message).await {
Ok(session) => session,
Err(response) => return response,
};
let tool_call_params: ToolCallParams = match serde_json::from_value(params(message)) {
Ok(value) => value,
Err(_) => {
return transport_response(
StatusCode::OK,
jsonrpc_error(request_id(message), -32602, "invalid tool call parameters"),
response_mode,
None,
Some(&session.protocol_version),
);
}
};
// The adapter owns the wire form: it parses the call parameters, removes the reserved
// confirmation marker and the shared Tools service never sees the envelope, the
// response mode or the protocol revision (spec section 6.2).
let mut arguments = tool_call_params.arguments.unwrap_or_else(|| json!({}));
let confirmation_token = take_confirmation_token(&mut arguments);
// Story 3.4: the client's own `_meta.progressToken`, validated and bounded, and the
// session/credential ownership of the in-flight cancellation table.
let progress = crate::app::lifecycle::token_from_meta(tool_call_params.meta.as_ref());
super::lifecycle::run_tool_call(
Arc::clone(&state),
crate::app::lifecycle::ToolCallRequest {
workspace_slug: session.workspace_slug.clone(),
agent_slug: session.agent_slug.clone(),
credential: credential.clone(),
correlation: transport_correlation.clone(),
continuation: ContinuationSupport::from_supported(session.supports_elicitation),
locale: execution_locale(message),
},
crate::app::lifecycle::ToolCallWire {
message: message.clone(),
protocol_version: session.protocol_version.clone(),
response_mode,
progress,
session_id: Some(session.id.clone()),
in_flight: Some(crate::app::lifecycle::InFlightScope {
session_id: session.id.clone(),
owner: in_flight_owner(credential),
}),
},
ToolCallCommand {
tool_name: tool_call_params.name,
arguments,
confirmation_token,
},
)
.await
}
DispatchDecision::Allowed {
handler: MethodHandler::Cancelled,
kind: Kind::Notification,
} if is_notification(message) => {
handle_cancelled_notification(&state, path, headers, message, credential).await
}
DispatchDecision::Allowed {
handler: MethodHandler::ResourceList,
..
} if is_request(message) => {
let session = match require_initialized_session(&state, path, headers, message).await {
Ok(session) => session,
Err(response) => return response,
};
let list_command: ResourcesListCommand = match serde_json::from_value(params(message)) {
Ok(value) => value,
Err(_) => {
return transport_response(
StatusCode::OK,
jsonrpc_error(
request_id(message),
-32602,
"invalid resources/list parameters",
),
response_mode,
None,
Some(&session.protocol_version),
);
}
};
let outcome = handle_resources_command(
&state,
ServiceContext {
workspace_slug: &session.workspace_slug,
agent_slug: &session.agent_slug,
credential,
correlation: transport_correlation,
continuation: ContinuationSupport::from_supported(session.supports_elicitation),
locale: execution_locale(message),
},
&session.protocol_version,
ResourcesCommand::List(list_command),
)
.await;
match outcome {
ResourcesOutcome::Listed {
resources,
next_cursor,
} => transport_response(
StatusCode::OK,
jsonrpc_result(
request_id(message),
resources_list_result(resources, next_cursor),
),
response_mode,
None,
Some(&session.protocol_version),
),
ResourcesOutcome::InvalidParams(detail) => transport_response(
StatusCode::OK,
jsonrpc_error(request_id(message), -32602, detail),
response_mode,
None,
Some(&session.protocol_version),
),
_ => internal_jsonrpc_error(message, "published resources are unavailable"),
}
}
DispatchDecision::Allowed {
handler: MethodHandler::ResourceRead,
..
} if is_request(message) => {
let session = match require_initialized_session(&state, path, headers, message).await {
Ok(session) => session,
Err(response) => return response,
};
let read_command: ResourceReadCommand = match serde_json::from_value(params(message)) {
Ok(value) => value,
Err(_) => {
return transport_response(
StatusCode::OK,
jsonrpc_error(
request_id(message),
-32602,
"invalid resources/read parameters",
),
response_mode,
None,
Some(&session.protocol_version),
);
}
};
let outcome = handle_resources_command(
&state,
ServiceContext {
workspace_slug: &session.workspace_slug,
agent_slug: &session.agent_slug,
credential,
correlation: transport_correlation,
continuation: ContinuationSupport::from_supported(session.supports_elicitation),
locale: execution_locale(message),
},
&session.protocol_version,
ResourcesCommand::Read(read_command),
)
.await;
match outcome {
ResourcesOutcome::Read { contents } => transport_response(
StatusCode::OK,
jsonrpc_result(request_id(message), resource_read_result(contents)),
response_mode,
None,
Some(&session.protocol_version),
),
ResourcesOutcome::NotAvailable => transport_response(
StatusCode::OK,
jsonrpc_error(
request_id(message),
-32602,
"the requested resource is not available",
),
response_mode,
None,
Some(&session.protocol_version),
),
ResourcesOutcome::InvalidParams(detail) => transport_response(
StatusCode::OK,
jsonrpc_error(request_id(message), -32602, detail),
response_mode,
None,
Some(&session.protocol_version),
),
ResourcesOutcome::ExecutionError(contract) => resource_error_response(
message,
response_mode,
&session.protocol_version,
contract,
),
_ => internal_jsonrpc_error(message, "published resources are unavailable"),
}
}
// Story 3.11: the shared service returns exact bound Published templates; the legacy adapter
// retains its earlier result shape and still requires the initialized session.
DispatchDecision::Allowed {
handler: MethodHandler::ResourceTemplateList,
..
} if is_request(message) => {
let session = match require_initialized_session(&state, path, headers, message).await {
Ok(session) => session,
Err(response) => return response,
};
let command: ResourcesListCommand = match serde_json::from_value(params(message)) {
Ok(command) => command,
Err(_) => {
return transport_response(
StatusCode::OK,
jsonrpc_error(
request_id(message),
-32602,
"invalid resources/templates/list parameters",
),
response_mode,
None,
Some(&session.protocol_version),
);
}
};
let (templates, next_cursor) = match crate::protocol::list_resource_templates(
&state,
&path.workspace_slug,
&path.agent_slug,
&session.protocol_version,
command,
)
.await
{
Ok(page) => page,
Err(crate::protocol::ResourceTemplatesListError::InvalidCursor) => {
return transport_response(
StatusCode::OK,
jsonrpc_error(
request_id(message),
-32602,
"invalid resources/templates/list cursor",
),
response_mode,
None,
Some(&session.protocol_version),
);
}
Err(crate::protocol::ResourceTemplatesListError::Unavailable) => {
return internal_jsonrpc_error(message, "published templates are unavailable");
}
};
transport_response(
StatusCode::OK,
jsonrpc_result(
request_id(message),
resource_templates_list_result(templates, next_cursor),
),
response_mode,
None,
Some(&session.protocol_version),
)
}
// Story 3.14: `prompts/list` / `prompts/get` of the initialized legacy session, served from
// the same exact bound Published Prompt Versions as the modern era.
DispatchDecision::Allowed {
handler: MethodHandler::PromptList,
..
} if is_request(message) => {
let session = match require_initialized_session(&state, path, headers, message).await {
Ok(session) => session,
Err(response) => return response,
};
let command: PromptsListCommand = match serde_json::from_value(params(message)) {
Ok(command) => command,
Err(_) => {
return transport_response(
StatusCode::OK,
jsonrpc_error(
request_id(message),
-32602,
"invalid prompts/list parameters",
),
response_mode,
None,
Some(&session.protocol_version),
);
}
};
let outcome = handle_prompts_command(
&state,
ServiceContext {
workspace_slug: &session.workspace_slug,
agent_slug: &session.agent_slug,
credential,
correlation: transport_correlation,
continuation: ContinuationSupport::from_supported(session.supports_elicitation),
locale: execution_locale(message),
},
&session.protocol_version,
PromptsCommand::List(command),
)
.await;
match outcome {
PromptsOutcome::Listed {
prompts,
next_cursor,
} => transport_response(
StatusCode::OK,
jsonrpc_result(
request_id(message),
prompts_list_result(prompts, next_cursor),
),
response_mode,
None,
Some(&session.protocol_version),
),
PromptsOutcome::InvalidParams(detail) => transport_response(
StatusCode::OK,
jsonrpc_error(request_id(message), -32602, detail),
response_mode,
None,
Some(&session.protocol_version),
),
_ => internal_jsonrpc_error(message, "published prompts are unavailable"),
}
}
DispatchDecision::Allowed {
handler: MethodHandler::PromptGet,
..
} if is_request(message) => {
let session = match require_initialized_session(&state, path, headers, message).await {
Ok(session) => session,
Err(response) => return response,
};
let command: PromptGetCommand = match serde_json::from_value(params(message)) {
Ok(command) => command,
Err(_) => {
return transport_response(
StatusCode::OK,
jsonrpc_error(
request_id(message),
-32602,
"invalid prompts/get parameters",
),
response_mode,
None,
Some(&session.protocol_version),
);
}
};
let outcome = handle_prompts_command(
&state,
ServiceContext {
workspace_slug: &session.workspace_slug,
agent_slug: &session.agent_slug,
credential,
correlation: transport_correlation,
continuation: ContinuationSupport::from_supported(session.supports_elicitation),
locale: execution_locale(message),
},
&session.protocol_version,
PromptsCommand::Get(command),
)
.await;
match outcome {
PromptsOutcome::Got {
description,
messages,
} => transport_response(
StatusCode::OK,
jsonrpc_result(
request_id(message),
prompt_get_result(description, messages),
),
response_mode,
None,
Some(&session.protocol_version),
),
PromptsOutcome::NotAvailable => transport_response(
StatusCode::OK,
jsonrpc_error(
request_id(message),
-32602,
"the requested prompt is not available",
),
response_mode,
None,
Some(&session.protocol_version),
),
PromptsOutcome::ArgumentsRejected { reason, argument } => {
prompt_arguments_error_response(
message,
response_mode,
&session.protocol_version,
reason,
argument,
)
}
PromptsOutcome::DependencyUnavailable { position } => transport_response(
StatusCode::OK,
jsonrpc_error(
request_id(message),
-32603,
format!("the pinned dependency at dependencies[{position}] is unavailable"),
),
response_mode,
None,
Some(&session.protocol_version),
),
PromptsOutcome::InvalidParams(detail) => transport_response(
StatusCode::OK,
jsonrpc_error(request_id(message), -32602, detail),
response_mode,
None,
Some(&session.protocol_version),
),
_ => internal_jsonrpc_error(message, "published prompts are unavailable"),
}
}
// Story 3.17: `completion/complete` of the initialized legacy session. The shared service is
// the same one the modern era uses; only the result envelope differs (no `resultType`).
DispatchDecision::Allowed {
handler: MethodHandler::Completion,
..
} if is_request(message) => {
let session = match require_initialized_session(&state, path, headers, message).await {
Ok(session) => session,
Err(response) => return response,
};
let command: CompletionCommand = match serde_json::from_value(params(message)) {
Ok(command) => command,
Err(_) => {
return transport_response(
StatusCode::OK,
jsonrpc_error(
request_id(message),
-32602,
"invalid completion/complete parameters",
),
response_mode,
None,
Some(&session.protocol_version),
);
}
};
let outcome = handle_completion_command(
&state,
ServiceContext {
workspace_slug: &session.workspace_slug,
agent_slug: &session.agent_slug,
credential,
correlation: transport_correlation,
continuation: ContinuationSupport::from_supported(session.supports_elicitation),
locale: execution_locale(message),
},
&session.protocol_version,
command,
)
.await;
match outcome {
CompletionOutcome::Completed {
values,
total,
has_more,
} => transport_response(
StatusCode::OK,
jsonrpc_result(
request_id(message),
completion_result(values, total, has_more),
),
response_mode,
None,
Some(&session.protocol_version),
),
CompletionOutcome::InvalidParams(detail) => transport_response(
StatusCode::OK,
jsonrpc_error(request_id(message), -32602, detail),
response_mode,
None,
Some(&session.protocol_version),
),
CompletionOutcome::NotAvailable => transport_response(
StatusCode::OK,
jsonrpc_error(
request_id(message),
-32602,
"the completion reference is not available",
),
response_mode,
None,
Some(&session.protocol_version),
),
CompletionOutcome::CatalogUnavailable => {
internal_jsonrpc_error(message, "published completion sources are unavailable")
}
}
}
// Story 3.18: the pinned legacy `logging/setLevel`. The level belongs to the initialized
// session of this request, so setting it can never change another client's level. The request
// answers an empty result; the acknowledgment event is filtered by the level it just set.
DispatchDecision::Allowed {
handler: MethodHandler::LoggingSetLevel,
..
} if is_request(message) => {
let session = match require_initialized_session(&state, path, headers, message).await {
Ok(session) => session,
Err(response) => return response,
};
let command: SetLevelCommand = match serde_json::from_value(params(message)) {
Ok(command) => command,
// A non-string, missing or otherwise unparsable level shares the one safe detail of a
// value outside the pinned union: nothing the client sent is echoed back.
Err(_) => {
return transport_response(
StatusCode::OK,
jsonrpc_error(request_id(message), -32602, INVALID_LEVEL),
response_mode,
None,
Some(&session.protocol_version),
);
}
};
match handle_logging_command(&state.logging_levels, &session.id, command) {
LoggingOutcome::Applied(level) => {
crate::protocol::set_active_level(Some(level));
crate::protocol::note(
crate::protocol::LogLevel::Notice,
EVENT_LEVEL_APPLIED,
"applied",
);
transport_response(
StatusCode::OK,
jsonrpc_result(request_id(message), json!({})),
response_mode,
None,
Some(&session.protocol_version),
)
}
LoggingOutcome::InvalidParams(detail) => transport_response(
StatusCode::OK,
jsonrpc_error(request_id(message), -32602, detail),
response_mode,
None,
Some(&session.protocol_version),
),
}
}
// A notification whose method the registration does not map to this adapter's handlers is
// acknowledged without being dispatched, exactly as before.
_ if is_notification(message) => StatusCode::ACCEPTED.into_response(),
// Every remaining decision (unregistered, wrong era, or a handler this adapter does not
// serve) is a method-level rejection: no handler, no service entry, no outbound call.
_ => match method_name(message) {
Some(method) => transport_response(
StatusCode::OK,
jsonrpc_error(
request_id(message),
-32601,
format!("method {method} is not supported"),
),
response_mode,
None,
protocol_version,
),
None => transport_response(
StatusCode::BAD_REQUEST,
jsonrpc_error(Value::Null, -32600, "invalid JSON-RPC message"),
response_mode,
None,
protocol_version,
),
},
}
}
/// The credential half of the cancellation ownership of one call (Story 3.4). The progress token is
/// deliberately not part of it: a token never authorizes a cancellation.
fn in_flight_owner(credential: &VerifiedMachineCredential) -> String {
credential
.platform_api_key_id
.as_ref()
.map_or_else(String::new, |id| id.as_str().to_owned())
}
/// Route one legacy `notifications/cancelled` to the call it names, then acknowledge it.
///
/// The pinned rules of a cancelled request are all "ignore": an unknown or already finished request,
/// a foreign session or credential, a malformed `requestId` and an unknown session are acknowledged
/// without any effect. That is also why this handler never fails the transport — a client that
/// cancels must not learn anything about another client's calls.
async fn handle_cancelled_notification(
state: &Arc<AppState>,
path: &AgentRoutePath,
headers: &HeaderMap,
message: &Value,
credential: &VerifiedMachineCredential,
) -> Response {
let request_id = params(message).get("requestId").cloned();
let session_id = session_id_from_headers(headers).ok().flatten();
if let (Some(request_id), Some(session_id)) = (request_id, session_id) {
// Only a string or a number is a pinned `RequestId`; anything else cannot name a call.
if matches!(request_id, Value::String(_) | Value::Number(_)) {
let owned_by_session = match state.sessions.get(&session_id).await {
Ok(Some(session)) => {
session.workspace_slug == path.workspace_slug
&& session.agent_slug == path.agent_slug
}
_ => false,
};
if owned_by_session {
state.in_flight.cancel(
&crate::app::lifecycle::InFlightScope {
session_id,
owner: in_flight_owner(credential),
},
&request_id,
);
}
}
}
StatusCode::ACCEPTED.into_response()
}
@@ -0,0 +1,566 @@
//! Story 3.4 (FR-14): request-scoped progress and cancellation of one synchronous `tools/call`.
//!
//! The pinned `RequestMetaObject` of *both* supported revisions carries an optional opaque
//! `progressToken`; a request that has one asks for out-of-band `notifications/progress` messages
//! about that request. Such a message only has a carrier on the request's own SSE response stream, so
//! a token is honoured only when the client negotiated `text/event-stream`. Without a token — or
//! without the stream — the call runs exactly as before: the receiver of a progress token is never
//! obligated to send progress, so the execution result cannot change.
//!
//! Cancellation follows the revision-specific rule of the pinned specification:
//! * `2025-11-25`: the client POSTs `notifications/cancelled` with the id of the request it issued.
//! The bounded in-flight table routes it to that call. Ownership is the session *and* the platform
//! credential of the call, never the progress token: a foreign session or key cancels nothing.
//! * `2026-07-28`: closing the request's SSE response stream *is* the cancellation signal, so there is
//! no notification to route; dropping the response stream drops the call future.
//!
//! Either way a cancelled call never becomes a false `completed`: no further message is sent for it
//! (the pinned rule for a cancelled request), and the call never reaches its terminal outcome, so no
//! Invocation History row claims a completion. The record contract only admits a *closed* execution
//! error code for a non-`ok` row and a cancellation is not one of them, so the honest state of a
//! stopped call is "no terminal record" rather than an invented error code; a dedicated cancellation
//! code would be a schema/vocabulary decision of its own.
use std::{
collections::{HashMap, VecDeque},
future::Future,
pin::Pin,
sync::{Arc, Mutex},
};
use axum::{
http::StatusCode,
response::{IntoResponse, Response, sse::Event},
};
use crank_core::{CorrelationContext, ExecutionLocale, VerifiedMachineCredential};
use futures_util::stream;
use serde_json::{Number, Value, json};
use tokio::sync::{mpsc, watch};
use super::{
AppState,
response::{ToolCommandPayload, tool_command_payload, tool_command_response},
};
use crate::{
jsonrpc::request_id,
protocol::{
ContinuationSupport, ServiceContext, ToolCallCommand, ToolsCommand, ToolsOutcome,
handle_tools_command,
},
transport::{ResponseMode, sse_response},
};
/// The bounded progress queue of one call. A client that reads its stream slowly can never block the
/// call: a full queue drops the event instead of waiting, and the remaining events stay monotonic.
pub(crate) const PROGRESS_CAPACITY: usize = 8;
/// The pinned `ProgressToken` bound this server accepts. A longer token is not rejected as an error:
/// progress is simply disabled for that request, exactly as for a malformed one.
pub(crate) const MAX_PROGRESS_TOKEN_LEN: usize = 128;
/// The bounded table of calls that can still be cancelled. A call that finds the table full runs
/// without cancellation instead of growing the table.
pub(crate) const MAX_IN_FLIGHT_CALLS: usize = 256;
/// A validated `RequestMetaObject.progressToken`: the pinned opaque token is a string or a number.
#[derive(Clone, Debug, PartialEq, Eq)]
pub(crate) enum ProgressToken {
Text(String),
Number(Number),
}
impl ProgressToken {
/// `None` means "disable progress", never "fail the call": the pinned schema declares the token
/// as an opaque client value the receiver may simply ignore.
fn parse(value: &Value) -> Option<Self> {
match value {
Value::String(text)
if !text.is_empty()
&& text.len() <= MAX_PROGRESS_TOKEN_LEN
&& !text.chars().any(char::is_control) =>
{
Some(Self::Text(text.clone()))
}
Value::Number(number) => Some(Self::Number(number.clone())),
_ => None,
}
}
fn as_value(&self) -> Value {
match self {
Self::Text(text) => Value::String(text.clone()),
Self::Number(number) => Value::Number(number.clone()),
}
}
}
/// The progress token of one request parameters object, already validated and bounded. The adapters
/// read the `_meta` object they already parsed ([`token_from_meta`]); this is the test seam of the
/// same rule.
#[cfg(test)]
pub(crate) fn progress_token(params: &Value) -> Option<ProgressToken> {
token_from_meta(params.get("_meta"))
}
/// The validated token of one `RequestMetaObject`. A non-object `_meta` has no token.
pub(crate) fn token_from_meta(meta: Option<&Value>) -> Option<ProgressToken> {
meta.and_then(|meta| meta.get("progressToken"))
.and_then(ProgressToken::parse)
}
/// One real, already reached execution step: the shared Tools service emits these at the stage
/// boundaries it actually crosses, so no percentage is invented and no timer drives them.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub(crate) enum Phase {
Resolved,
Authorized,
Dispatching,
}
impl Phase {
const fn message(self) -> &'static str {
match self {
Self::Resolved => "tool_resolved",
Self::Authorized => "authorization_resolved",
Self::Dispatching => "execution_dispatched",
}
}
}
#[derive(Clone, Debug)]
pub(crate) struct ProgressEvent {
progress: u64,
message: &'static str,
}
/// The pinned `notifications/progress` frame. Both supported revisions declare the same parameters,
/// so the projection is shared; the token is echoed verbatim and never validated as a credential.
fn progress_frame(token: &ProgressToken, event: &ProgressEvent) -> Value {
json!({
"jsonrpc": "2.0",
"method": "notifications/progress",
"params": {
"progressToken": token.as_value(),
"progress": event.progress,
"message": event.message,
}
})
}
#[derive(Clone)]
struct ProgressSink {
events: mpsc::Sender<ProgressEvent>,
produced: Arc<std::sync::atomic::AtomicU64>,
}
impl ProgressSink {
fn emit(&self, message: &'static str) {
let progress = self
.produced
.fetch_add(1, std::sync::atomic::Ordering::SeqCst)
+ 1;
let _ = self.events.try_send(ProgressEvent { progress, message });
}
}
/// A cooperative cancellation signal of one call. It is set at most once; a signal whose only sender
/// is gone simply never fires.
#[derive(Clone)]
pub(crate) struct CancelSignal {
receiver: watch::Receiver<bool>,
}
impl CancelSignal {
fn never() -> Self {
let (_sender, receiver) = watch::channel(false);
Self { receiver }
}
pub(crate) async fn cancelled(&mut self) {
if self
.receiver
.wait_for(|cancelled| *cancelled)
.await
.is_err()
{
// The owning call is gone, so nothing can cancel it any more.
std::future::pending::<()>().await;
}
}
}
/// The request-scoped lifecycle of one `tools/call`: its progress sink (when the client asked for
/// progress *and* offered the stream carrier) and its cancellation signal.
#[derive(Clone)]
pub(crate) struct ToolCallLifecycle {
progress: Option<ProgressSink>,
cancel: CancelSignal,
}
impl ToolCallLifecycle {
pub(crate) fn new(
token: Option<ProgressToken>,
) -> (Self, Option<mpsc::Receiver<ProgressEvent>>) {
match token {
Some(_) => {
let (sender, receiver) = mpsc::channel(PROGRESS_CAPACITY);
(
Self {
progress: Some(ProgressSink {
events: sender,
produced: Arc::new(std::sync::atomic::AtomicU64::new(0)),
}),
cancel: CancelSignal::never(),
},
Some(receiver),
)
}
None => (
Self {
progress: None,
cancel: CancelSignal::never(),
},
None,
),
}
}
pub(crate) fn with_cancel(mut self, cancel: CancelSignal) -> Self {
self.cancel = cancel;
self
}
/// Record one real execution step. Without a progress sink this is a no-op, so the shared service
/// does not branch on the presence of a token.
pub(crate) fn advance(&self, phase: Phase) {
if let Some(progress) = &self.progress {
progress.emit(phase.message());
}
}
pub(crate) fn cancelled(&mut self) -> impl Future<Output = ()> + '_ {
self.cancel.cancelled()
}
}
/// The owned identity of one call, so the call future can be driven by the response stream itself.
pub(crate) struct ToolCallRequest {
pub(crate) workspace_slug: String,
pub(crate) agent_slug: String,
pub(crate) credential: VerifiedMachineCredential,
pub(crate) correlation: CorrelationContext,
pub(crate) continuation: ContinuationSupport,
pub(crate) locale: ExecutionLocale,
}
/// The wire facts the adapter owns: the envelope, the negotiated revision and response mode, the
/// validated token and — for the legacy revision — the session-scoped in-flight ownership.
pub(crate) struct ToolCallWire {
pub(crate) message: Value,
pub(crate) protocol_version: String,
pub(crate) response_mode: ResponseMode,
pub(crate) progress: Option<ProgressToken>,
pub(crate) session_id: Option<String>,
pub(crate) in_flight: Option<InFlightScope>,
}
/// Who owns a cancellable call. The session and the platform credential are the ownership of the
/// legacy revision; the progress token is deliberately not part of it.
#[derive(Clone, Debug, PartialEq, Eq)]
pub(crate) struct InFlightScope {
pub(crate) session_id: String,
pub(crate) owner: String,
}
/// The bounded table of the calls a legacy client can still cancel.
pub(crate) struct InFlightCalls {
entries: Mutex<HashMap<String, watch::Sender<bool>>>,
capacity: usize,
}
impl InFlightCalls {
pub(crate) fn new(capacity: usize) -> Self {
Self {
entries: Mutex::new(HashMap::new()),
capacity,
}
}
fn entries(&self) -> std::sync::MutexGuard<'_, HashMap<String, watch::Sender<bool>>> {
// A panic of another request must not make the cancellation table unusable.
self.entries
.lock()
.unwrap_or_else(|poisoned| poisoned.into_inner())
}
fn key(scope: &InFlightScope, request_id: &Value) -> String {
// The pinned `RequestId` is a string or a number; the two must not collapse into one key.
format!(
"{}\u{1f}{}\u{1f}{request_id}",
scope.session_id, scope.owner
)
}
/// Register one call as cancellable. `None` means the table is full (or the id is already live),
/// and the call then runs without cancellation instead of evicting someone else's entry.
pub(crate) fn register(
self: &Arc<Self>,
scope: &InFlightScope,
request_id: &Value,
) -> Option<InFlightEntry> {
let key = Self::key(scope, request_id);
let (sender, receiver) = watch::channel(false);
{
let mut entries = self.entries();
if entries.len() >= self.capacity || entries.contains_key(&key) {
return None;
}
entries.insert(key.clone(), sender);
}
Some(InFlightEntry {
registry: Arc::clone(self),
key,
cancel: CancelSignal { receiver },
})
}
/// Route one `notifications/cancelled` to the call it names. `false` means there is no such
/// in-flight call of this owner, which the specification requires to be ignored.
pub(crate) fn cancel(&self, scope: &InFlightScope, request_id: &Value) -> bool {
let entries = self.entries();
entries
.get(&Self::key(scope, request_id))
.is_some_and(|sender| sender.send(true).is_ok())
}
}
/// The live registration of one call: dropping it (terminal response, cancellation or a dropped
/// response stream) releases the table slot.
pub(crate) struct InFlightEntry {
registry: Arc<InFlightCalls>,
key: String,
cancel: CancelSignal,
}
impl InFlightEntry {
pub(crate) fn cancel_signal(&self) -> CancelSignal {
self.cancel.clone()
}
}
impl Drop for InFlightEntry {
fn drop(&mut self) {
self.registry.entries().remove(&self.key);
}
}
/// Run one `tools/call` through the shared Tools service with the lifecycle of this request.
///
/// Without a progress token — or without the SSE carrier the client negotiated — the response is the
/// unchanged single JSON (or single-event SSE) response. With both, the response is the request's own
/// bounded stream: the real progress events of this call, then the terminal response as the last
/// event. Dropping that stream (a `2026-07-28` cancellation) or setting the signal of a legacy
/// `notifications/cancelled` drops the call future, which is the cooperative stop.
pub(crate) async fn run_tool_call(
state: Arc<AppState>,
request: ToolCallRequest,
wire: ToolCallWire,
command: ToolCallCommand,
) -> Response {
// A token is only honoured when the client offered the only carrier a notification has: the
// request's own SSE stream. Otherwise the call runs unchanged (a receiver is never obligated).
let token = match (&wire.response_mode, &wire.progress) {
(ResponseMode::Sse, Some(token)) => Some(token.clone()),
_ => None,
};
let (mut lifecycle, progress_events) = ToolCallLifecycle::new(token.clone());
let in_flight = wire.in_flight.as_ref().and_then(|scope| {
let entry = state.in_flight.register(scope, &request_id(&wire.message));
if let Some(entry) = &entry {
lifecycle = lifecycle.clone().with_cancel(entry.cancel_signal());
}
entry
});
let message = wire.message.clone();
let correlation = request.correlation.clone();
let protocol_version = wire.protocol_version.clone();
let response_mode = wire.response_mode;
let mut call = Box::pin(call_future(
Arc::clone(&state),
request,
lifecycle.clone(),
command,
));
let Some(progress_events) = progress_events else {
// No notification carrier for this call: run it as before, still cancellable.
let outcome = tokio::select! {
biased;
outcome = &mut call => outcome,
_ = lifecycle.cancelled() => return cancelled_response(),
};
return tool_command_response(
&message,
response_mode,
&protocol_version,
&correlation,
outcome,
);
};
debug_assert!(matches!(response_mode, ResponseMode::Sse));
let token = token.expect("a progress receiver implies a validated token");
let stream = stream::unfold(
CallStream {
call,
events: progress_events,
cancel: lifecycle.cancel.clone(),
token,
message,
protocol_version: protocol_version.clone(),
correlation,
pending: VecDeque::new(),
finished: false,
logging: crate::protocol::carry_scope(),
_in_flight: in_flight,
},
next_event,
);
sse_response(
StatusCode::OK,
stream,
wire.session_id.as_deref(),
Some(&protocol_version),
)
}
async fn call_future(
state: Arc<AppState>,
request: ToolCallRequest,
lifecycle: ToolCallLifecycle,
command: ToolCallCommand,
) -> ToolsOutcome {
handle_tools_command(
&state,
ServiceContext {
workspace_slug: &request.workspace_slug,
agent_slug: &request.agent_slug,
credential: &request.credential,
correlation: &request.correlation,
continuation: request.continuation,
locale: request.locale,
},
ToolsCommand::Call(command),
Some(lifecycle),
)
.await
}
/// A cancelled call is not answered: the pinned rule for a cancelled request is that no further
/// message is sent for it, and the client that cancelled it ignores a late response anyway. Nothing
/// here claims that an outbound side effect did or did not happen — and nothing else claims the
/// opposite either: the call is stopped before its terminal outcome, so its Invocation History holds
/// no row for this call at all (see the module docs for why there is no invented cancellation code).
fn cancelled_response() -> Response {
StatusCode::OK.into_response()
}
struct CallStream {
call: Pin<Box<dyn Future<Output = ToolsOutcome> + Send>>,
events: mpsc::Receiver<ProgressEvent>,
cancel: CancelSignal,
token: ProgressToken,
message: Value,
protocol_version: String,
correlation: CorrelationContext,
pending: VecDeque<Event>,
finished: bool,
/// The request's logging context, so the terminal event of the stream is projected exactly as the
/// single response of the same call would be (Story 3.18 carried across the stream, Story 3.4).
logging: crate::protocol::CarriedScope,
/// Held for the lifetime of the stream: the slot is released when the call is over, however it ends.
_in_flight: Option<InFlightEntry>,
}
enum Step {
Progress(ProgressEvent),
Outcome(ToolsOutcome),
Cancelled,
}
/// One stream item: a queued frame, the next real progress event, the terminal response, or the end of
/// a cancelled stream. A cancelled call sends no terminal frame at all.
async fn next_event(
call: CallStream,
) -> Option<(Result<Event, std::convert::Infallible>, CallStream)> {
let scope = call.logging;
crate::protocol::in_carried_scope(scope, next_event_in_scope(call)).await
}
async fn next_event_in_scope(
mut call: CallStream,
) -> Option<(Result<Event, std::convert::Infallible>, CallStream)> {
if let Some(frame) = call.pending.pop_front() {
return Some((Ok(frame), call));
}
if call.finished {
return None;
}
let step = {
let CallStream {
call,
events,
cancel,
..
} = &mut call;
tokio::select! {
biased;
Some(event) = events.recv() => Step::Progress(event),
outcome = call => Step::Outcome(outcome),
_ = cancel.cancelled() => Step::Cancelled,
}
};
match step {
Step::Progress(event) => {
let frame = progress_frame(&call.token, &event);
Some((Ok(Event::default().data(frame.to_string())), call))
}
Step::Outcome(outcome) => {
call.finished = true;
let payload = match tool_command_payload(
&call.message,
&call.protocol_version,
&call.correlation,
outcome,
) {
ToolCommandPayload::Ok(payload) => payload,
ToolCommandPayload::Internal(_) => jsonrpc_internal_error(&call.message),
};
// The same projection point the single response uses, so a streamed answer delivers the
// same closed `notifications/message` (Story 3.18) before its terminal frame.
crate::transport::note_logging_outcome(&payload);
for notification in crate::protocol::take_notifications() {
call.pending
.push_back(Event::default().data(notification.to_string()));
}
call.pending
.push_back(Event::default().data(payload.to_string()));
Some((Ok(call.pending.pop_front().expect("queued frame")), call))
}
Step::Cancelled => {
call.finished = true;
call.pending.clear();
None
}
}
}
fn jsonrpc_internal_error(message: &Value) -> Value {
crate::jsonrpc::jsonrpc_error(request_id(message), -32603, "internal server error")
}
#[cfg(test)]
#[path = "lifecycle_tests.rs"]
mod tests;
@@ -0,0 +1,170 @@
//! Story 3.4 bounds that do not need the HTTP surface: token validation, the monotonic bounded
//! progress queue, the pinned frame shape and the ownership of the in-flight cancellation table.
use super::*;
fn token(value: Value) -> Option<ProgressToken> {
progress_token(&json!({ "_meta": { "progressToken": value } }))
}
#[test]
fn a_pinned_progress_token_is_a_bounded_string_or_a_number() {
assert_eq!(
token(json!("tok-1")),
Some(ProgressToken::Text("tok-1".to_owned()))
);
assert!(matches!(token(json!(7)), Some(ProgressToken::Number(_))));
assert!(matches!(token(json!(1.5)), Some(ProgressToken::Number(_))));
// Malformed and out-of-bound tokens disable progress instead of failing the call.
assert_eq!(token(json!("")), None);
assert_eq!(token(json!("x".repeat(MAX_PROGRESS_TOKEN_LEN + 1))), None);
assert_eq!(token(json!("line\nbreak")), None);
assert_eq!(token(json!(null)), None);
assert_eq!(token(json!(true)), None);
assert_eq!(token(json!({ "nested": 1 })), None);
assert_eq!(token(json!([1, 2])), None);
// A token at the bound is still accepted.
assert!(token(json!("x".repeat(MAX_PROGRESS_TOKEN_LEN))).is_some());
// Absent `_meta`, absent token and a non-object `_meta` are all "no progress".
assert_eq!(progress_token(&json!({})), None);
assert_eq!(progress_token(&json!({ "_meta": { "other": 1 } })), None);
assert_eq!(progress_token(&json!({ "_meta": "not-an-object" })), None);
}
#[tokio::test]
async fn real_steps_produce_strictly_increasing_bounded_progress_events() {
let (lifecycle, events) = ToolCallLifecycle::new(token(json!("tok-monotonic")));
let mut events = events.expect("a validated token opens the progress queue");
for phase in [Phase::Resolved, Phase::Authorized, Phase::Dispatching] {
lifecycle.advance(phase);
}
drop(lifecycle);
let mut observed = Vec::new();
while let Some(event) = events.recv().await {
observed.push((event.progress, event.message));
}
assert_eq!(
observed,
vec![
(1, "tool_resolved"),
(2, "authorization_resolved"),
(3, "execution_dispatched"),
]
);
assert!(
observed.windows(2).all(|pair| pair[0].0 < pair[1].0),
"progress must increase monotonically, got {observed:?}"
);
}
#[tokio::test]
async fn a_full_progress_queue_drops_events_instead_of_blocking_the_call() {
let (lifecycle, events) = ToolCallLifecycle::new(token(json!(9)));
let mut events = events.expect("a validated token opens the progress queue");
// Many more steps than the queue holds: the call must never wait for the reader.
for _ in 0..(PROGRESS_CAPACITY * 4) {
lifecycle.advance(Phase::Resolved);
}
let mut delivered = 0;
while events.try_recv().is_ok() {
delivered += 1;
}
assert_eq!(delivered, PROGRESS_CAPACITY);
}
#[tokio::test]
async fn without_a_token_no_progress_is_produced_at_all() {
let (lifecycle, events) = ToolCallLifecycle::new(None);
assert!(events.is_none());
lifecycle.advance(Phase::Resolved);
lifecycle.advance(Phase::Dispatching);
}
/// The pinned `notifications/progress` frame (schema `ProgressNotification`, both supported
/// revisions): `progressToken` echoed verbatim, `progress` increasing, `message` optional and
/// `total` absent because this server never invents one.
#[test]
fn the_progress_frame_matches_the_pinned_notification_shape() {
let event = ProgressEvent {
progress: 2,
message: Phase::Authorized.message(),
};
assert_eq!(
progress_frame(&ProgressToken::Text("tok-2".to_owned()), &event),
json!({
"jsonrpc": "2.0",
"method": "notifications/progress",
"params": {
"progressToken": "tok-2",
"progress": 2,
"message": "authorization_resolved"
}
})
);
assert_eq!(
progress_frame(&ProgressToken::Number(7.into()), &event)["params"]["progressToken"],
json!(7)
);
}
#[tokio::test]
async fn a_cancellation_only_reaches_the_call_of_the_same_session_and_credential() {
let registry = Arc::new(InFlightCalls::new(MAX_IN_FLIGHT_CALLS));
let scope = InFlightScope {
session_id: "sess-1".to_owned(),
owner: "pk_a".to_owned(),
};
let entry = registry
.register(&scope, &json!(42))
.expect("the table has room");
let mut signal = entry.cancel_signal();
// A foreign credential, a foreign session and a different request id cancel nothing.
for foreign in [
InFlightScope {
session_id: "sess-1".to_owned(),
owner: "pk_b".to_owned(),
},
InFlightScope {
session_id: "sess-2".to_owned(),
owner: "pk_a".to_owned(),
},
] {
assert!(!registry.cancel(&foreign, &json!(42)));
}
assert!(!registry.cancel(&scope, &json!(43)));
assert!(
tokio::time::timeout(std::time::Duration::from_millis(50), signal.cancelled())
.await
.is_err(),
"a foreign cancellation must not fire the signal"
);
assert!(registry.cancel(&scope, &json!(42)));
tokio::time::timeout(std::time::Duration::from_millis(50), signal.cancelled())
.await
.expect("the owner's cancellation fires the signal");
}
#[tokio::test]
async fn a_released_call_releases_its_slot_and_the_table_is_bounded() {
let registry = Arc::new(InFlightCalls::new(1));
let scope = InFlightScope {
session_id: "sess-1".to_owned(),
owner: "pk_a".to_owned(),
};
let entry = registry.register(&scope, &json!(1)).expect("first slot");
assert!(
registry.register(&scope, &json!(2)).is_none(),
"the table refuses a call beyond its bound instead of evicting the live one"
);
assert!(registry.cancel(&scope, &json!(1)));
drop(entry);
assert!(
!registry.cancel(&scope, &json!(1)),
"a terminal call is no longer cancellable"
);
assert!(registry.register(&scope, &json!(2)).is_some());
}
@@ -121,3 +121,53 @@ impl ActiveStreamGuard {
})
}
}
#[cfg(test)]
mod tests {
use crank_metrics::McpMethod;
use serde_json::json;
use super::normalized_mcp_method;
/// AT-3.18-02: the metric label of the logging request is a member of the closed label enum, so
/// neither the level nor any other client-supplied identifier can reach the label space. There is
/// no mapping from a request value to a label anywhere in this file.
#[test]
fn a_logging_request_can_only_produce_a_fixed_metric_label() {
for level in [
json!("debug"),
json!("verbose"),
json!("crank-canary-9f7620e5-aa41-4b0d"),
json!(7),
json!(null),
] {
let message = json!({
"jsonrpc": "2.0",
"id": 1,
"method": "logging/setLevel",
"params": {"level": level}
});
assert_eq!(
normalized_mcp_method(&message),
McpMethod::Unsupported,
"the method label is a fixed enum member, never the request value"
);
}
assert_eq!(McpMethod::Unsupported.as_str(), "unsupported");
// Every label value of the enum is a pinned literal, so the label domain cannot be widened by
// a request at all.
assert!(
McpMethod::VALUES.iter().all(|value| value
.chars()
.all(|character| character.is_ascii_lowercase() || character == '_')),
"every method label is a fixed lowercase literal: {:?}",
McpMethod::VALUES
);
// A malformed envelope has its own fixed label as well.
assert_eq!(
normalized_mcp_method(&json!({"level": "debug"})),
McpMethod::Invalid
);
}
}
@@ -14,6 +14,11 @@ pub(super) struct ToolCallParams {
pub(super) name: String,
#[serde(default, deserialize_with = "deserialize_arguments_object")]
pub(super) arguments: Option<Value>,
/// Story 3.4: the pinned `RequestMetaObject` of this request. Only the validated
/// `progressToken` of it is read, and only to decide whether the client asked for out-of-band
/// progress; a malformed or foreign `_meta` never fails the call.
#[serde(default, rename = "_meta")]
pub(super) meta: Option<Value>,
}
/// The pinned `CallToolRequest.params.arguments` is an **optional object**. Omitting it stays valid
+92 -66
View File
@@ -23,71 +23,75 @@ pub(crate) fn take_confirmation_token(arguments: &mut Value) -> Option<String> {
.filter(|value| !value.trim().is_empty())
}
pub(crate) fn success_tool_response(
/// The era-neutral success payload of one `tools/call` (Story 3.4: the streamed response needs the
/// payload itself, because the terminal response is the last event of the request's stream).
pub(super) fn success_tool_payload(
message: &Value,
response_mode: ResponseMode,
protocol_version: &str,
output: Value,
) -> Response {
transport_response(
StatusCode::OK,
jsonrpc_result(
request_id(message),
project_modern_result(
protocol_version,
json!({
"content": [
{
"type": "text",
"text": serde_json::to_string_pretty(&output).unwrap_or_else(|_| "{}".to_owned())
}
],
"structuredContent": output,
"isError": false
}),
),
) -> Value {
jsonrpc_result(
request_id(message),
project_modern_result(
protocol_version,
json!({
"content": [
{
"type": "text",
"text": serde_json::to_string_pretty(&output).unwrap_or_else(|_| "{}".to_owned())
}
],
"structuredContent": output,
"isError": false
}),
),
response_mode,
None,
Some(protocol_version),
)
}
#[cfg(test)]
pub(crate) fn tool_error_response(
message: &Value,
response_mode: ResponseMode,
protocol_version: &str,
error: ToolErrorContract,
) -> Response {
let error_message = tool_error_text(&error);
let error_value = tool_error_value(&error);
transport_response(
StatusCode::OK,
jsonrpc_result(
request_id(message),
project_modern_result(
protocol_version,
json!({
"content": [
{
"type": "text",
"text": error_message
}
],
"structuredContent": {
"error": error_value
},
"isError": true
}),
),
),
tool_error_payload(message, protocol_version, error),
response_mode,
None,
Some(protocol_version),
)
}
pub(super) fn tool_error_payload(
message: &Value,
protocol_version: &str,
error: ToolErrorContract,
) -> Value {
let error_message = tool_error_text(&error);
let error_value = tool_error_value(&error);
jsonrpc_result(
request_id(message),
project_modern_result(
protocol_version,
json!({
"content": [
{
"type": "text",
"text": error_message
}
],
"structuredContent": {
"error": error_value
},
"isError": true
}),
),
)
}
/// The era-neutral `tools/list` result shape built from the shared command outcome. Object keys end
/// up byte-lexicographically ordered, exactly as before the extraction.
pub(super) fn tools_list_result(tools: Vec<Value>, next_cursor: Option<String>) -> Value {
@@ -193,28 +197,35 @@ pub(super) fn prompt_arguments_error_response(
)
}
/// Map one shared Tools-command outcome back to the era-specific wire response (spec section 6.2).
/// One `tools/call` wire frame, or the adapter's own internal-error detail.
///
/// This adapter function is the only place that turns a neutral outcome into a response: the shared
/// Story 3.4: the same projection serves the single response and the terminal event of a streamed
/// response, so both forms of one outcome can never drift apart.
pub(super) enum ToolCommandPayload {
Ok(Value),
Internal(&'static str),
}
/// Map one shared Tools-command outcome back to its era-specific wire payload (spec section 6.2).
///
/// This adapter function is the only place that turns a neutral outcome into a wire value: the shared
/// service never sees `ResponseMode`, the protocol revision, the request envelope or an
/// `axum::Response`. `ApprovalPending` is a success for the client (it still has to answer), while
/// an unavailable elicitation and a failing call keep their era-correct tool-error shape.
pub(super) fn tool_command_response(
pub(super) fn tool_command_payload(
message: &Value,
response_mode: ResponseMode,
protocol_version: &str,
correlation: &CorrelationContext,
outcome: ToolsOutcome,
) -> Response {
) -> ToolCommandPayload {
match outcome {
ToolsOutcome::Call(ToolCallOutcome::Completed(output))
| ToolsOutcome::Call(ToolCallOutcome::ApprovalPending(output)) => {
success_tool_response(message, response_mode, protocol_version, output)
ToolCommandPayload::Ok(success_tool_payload(message, protocol_version, output))
}
ToolsOutcome::Call(ToolCallOutcome::ApprovalUnavailable(control_error)) => {
tool_error_response(
ToolCommandPayload::Ok(tool_error_payload(
message,
response_mode,
protocol_version,
control_tool_error_contract(
control_error,
@@ -222,34 +233,49 @@ pub(super) fn tool_command_response(
correlation.trace_id().as_str(),
execution_locale(message),
),
)
))
}
ToolsOutcome::Call(ToolCallOutcome::InvalidParams(detail)) => {
ToolCommandPayload::Ok(jsonrpc_error(request_id(message), -32602, detail))
}
ToolsOutcome::Call(ToolCallOutcome::InvalidParams(detail)) => transport_response(
StatusCode::OK,
jsonrpc_error(request_id(message), -32602, detail),
response_mode,
None,
Some(protocol_version),
),
ToolsOutcome::Call(ToolCallOutcome::Error(contract)) => {
tool_error_response(message, response_mode, protocol_version, contract)
ToolCommandPayload::Ok(tool_error_payload(message, protocol_version, contract))
}
// The pending approval could not be persisted: the adapter answers its internal error shape,
// as the previous wire path did.
ToolsOutcome::Call(ToolCallOutcome::InternalFailure) => internal_jsonrpc_error(
message,
"the pending approval request could not be persisted",
),
ToolsOutcome::Call(ToolCallOutcome::InternalFailure) => {
ToolCommandPayload::Internal("the pending approval request could not be persisted")
}
// A call command never produces these variants; they are answered with the adapter's
// internal error instead of being silently ignored.
ToolsOutcome::List { .. }
| ToolsOutcome::InvalidParams(_)
| ToolsOutcome::CatalogUnavailable => {
internal_jsonrpc_error(message, "unexpected shared tools outcome for a tools/call")
ToolCommandPayload::Internal("unexpected shared tools outcome for a tools/call")
}
}
}
/// The single-response form of [`tool_command_payload`].
pub(super) fn tool_command_response(
message: &Value,
response_mode: ResponseMode,
protocol_version: &str,
correlation: &CorrelationContext,
outcome: ToolsOutcome,
) -> Response {
match tool_command_payload(message, protocol_version, correlation, outcome) {
ToolCommandPayload::Ok(payload) => transport_response(
StatusCode::OK,
payload,
response_mode,
None,
Some(protocol_version),
),
ToolCommandPayload::Internal(detail) => internal_jsonrpc_error(message, detail),
}
}
pub(super) fn resource_error_response(
message: &Value,
response_mode: ResponseMode,
@@ -0,0 +1,282 @@
//! Application-level durable Task handler for Story 4.3.
use std::sync::Arc;
use std::time::{Duration, Instant};
use async_trait::async_trait;
use crank_core::{
AgentId, ExecutionOrigin, MachineAccessMode, OperationSecurityLevel, PlatformApiKeyScope,
TaskError, TaskLease, TaskOwnerKind, TraceContext, VerifiedMachineCredential, WorkspaceId,
current_unix_ms,
};
use crank_registry::PostgresRegistry;
use crank_registry::PublishedAgentTool;
use crank_runtime::{
ExecutionAuthorization, ExecutionRequestError, RuntimeExecutionRequest, RuntimeExecutor,
SecretCrypto, TaskDispatchPermit, TaskHandler, TaskService, TaskWorkerError,
};
use super::TaskApplication;
use super::auth_resolution::resolve_operation_auth_with;
use super::tool_resolution::runtime_operation;
use crate::access::{allows_scope, credential_allows_security_level};
use crate::catalog::PublishedToolCatalog;
pub struct TaskWorkerHandler {
registry: PostgresRegistry,
catalog: PublishedToolCatalog,
runtime: RuntimeExecutor,
secret_crypto: SecretCrypto,
service: TaskService,
}
impl TaskWorkerHandler {
/// Production construction from the same application entry, registry and runtime the MCP
/// process serves requests with. The worker needs no HTTP state, only these authoritative
/// dependencies.
pub fn from_application(
tasks: &TaskApplication,
registry: PostgresRegistry,
runtime: RuntimeExecutor,
secret_crypto: SecretCrypto,
) -> Self {
Self {
registry,
catalog: tasks.catalog(),
runtime,
secret_crypto,
service: tasks.task_service(),
}
}
/// Authoritative pre-dispatch authorization of one leased Task (Story 4.3).
///
/// Resolves the exact pinned Published Version in the current publication, repeats the approval
/// policy and revalidates the typed persisted Platform API key identity against the registry:
/// the key must still be active, unexpired and bound to this workspace/agent, and its exact
/// scopes/security level must still authorize the operation. Any denial is a typed error, so the
/// worker finishes the attempt without dispatch (fail closed before any network use).
///
/// An administrator Task has no authoritative Task revalidation API in the registry, so it stays
/// fail-closed here instead of inventing authority.
async fn authorize(&self, lease: &TaskLease) -> Result<PublishedAgentTool, TaskError> {
if lease.record.owner_kind != TaskOwnerKind::Agent {
return Err(TaskError::invalid(
"admin task authority revalidation is not implemented",
));
}
let Some(key_id) = lease.record.platform_api_key_id.as_ref() else {
return Err(TaskError::invalid("task authority reference is missing"));
};
let workspace_id = WorkspaceId::new(lease.record.workspace_id.as_str());
let agent_id = AgentId::new(lease.record.owner_id.as_str());
let workspace = self
.registry
.get_workspace(&workspace_id)
.await
.map_err(|_| TaskError::unavailable("published catalog unavailable"))?
.ok_or_else(|| TaskError::invalid("task workspace no longer exists"))?;
let agents = self
.registry
.list_agents(&workspace_id)
.await
.map_err(|_| TaskError::unavailable("published catalog unavailable"))?;
let agent = agents
.into_iter()
.find(|agent| agent.id.as_str() == lease.record.owner_id.as_str())
.ok_or_else(|| TaskError::invalid("task agent no longer exists"))?;
let catalog = self
.catalog
.get_catalog(&workspace.workspace.slug, &agent.slug)
.await
.map_err(|_| TaskError::unavailable("published catalog unavailable"))?;
let tool = catalog
.tools
.into_iter()
.find(|tool| {
tool.operation.id.as_str() == lease.record.operation_id.as_str()
&& tool.operation.version == lease.record.pins.operation_version
})
.ok_or_else(|| TaskError::invalid("task published operation pin no longer matches"))?;
if tool.workspace_id.as_str() != lease.record.workspace_id.as_str() {
return Err(TaskError::invalid("task workspace binding mismatch"));
}
if tool.agent_id.as_str() != lease.record.owner_id.as_str() {
return Err(TaskError::invalid("task agent binding mismatch"));
}
if tool
.operation
.execution_config
.approval_policy
.as_ref()
.is_some_and(|policy| policy.required)
{
return Err(TaskError::invalid(
"missing approval proof for task operation",
));
}
// Authoritative revalidation of the exact immutable key identity. A revoked, expired,
// foreign-workspace or foreign-agent key is absent here and never reaches the network.
let key = self
.registry
.revalidate_platform_api_key(&workspace_id, &agent_id, key_id)
.await
.map_err(|_| TaskError::unavailable("task authority store unavailable"))?
.ok_or_else(|| TaskError::invalid("task authority is no longer active"))?;
let credential = VerifiedMachineCredential {
machine_access_mode: MachineAccessMode::StaticAgentKey,
max_security_level: OperationSecurityLevel::Standard,
scopes: key.api_key.scopes.clone(),
platform_api_key_id: Some(key.api_key.id.clone()),
};
if !allows_scope(&credential.scopes, PlatformApiKeyScope::Write) {
return Err(TaskError::invalid("task authority scope is insufficient"));
}
if !credential_allows_security_level(&credential, tool.operation.security_level) {
return Err(TaskError::invalid(
"task authority security level is insufficient",
));
}
Ok(tool)
}
}
#[async_trait]
impl TaskHandler for TaskWorkerHandler {
async fn revalidate(&self, lease: &TaskLease) -> Result<(), TaskWorkerError> {
self.authorize(lease)
.await
.map_err(TaskWorkerError::Store)?;
Ok(())
}
async fn execute(
&self,
lease: &TaskLease,
_attempt_id: &str,
cancelled: Arc<std::sync::atomic::AtomicBool>,
permit: TaskDispatchPermit,
) -> crank_runtime::TaskAttemptOutcome {
use crank_runtime::TaskAttemptOutcome;
use std::sync::atomic::Ordering;
// The lease may have been lost between the checkpoint and the outbound call; never dispatch
// on behalf of a worker that no longer owns the Task.
if cancelled.load(Ordering::Acquire) {
return TaskAttemptOutcome::OutcomeUnknown;
}
let tool = match self.authorize(lease).await {
Ok(tool) => tool,
Err(error) => {
return TaskAttemptOutcome::Failed(serde_json::json!({"code": error.to_string()}));
}
};
let input = match self.service.executable_input(lease).await {
Ok(input) => input,
Err(error) => {
return TaskAttemptOutcome::Failed(serde_json::json!({"code": error.to_string()}));
}
};
let operation = runtime_operation(&tool);
let now_ms = current_unix_ms();
let deadline_ms = lease.record.pins.execution_deadline_unix_ms;
if deadline_ms <= now_ms {
return TaskAttemptOutcome::Failed(
serde_json::json!({"code": "task deadline elapsed"}),
);
}
let deadline = Instant::now()
+ Duration::from_millis((deadline_ms - now_ms).min(i64::MAX as i64) as u64);
let workspace_id = WorkspaceId::new(lease.record.workspace_id.as_str());
let agent_id = AgentId::new(lease.record.owner_id.as_str());
let resolved_auth = match resolve_operation_auth_with(
&self.registry,
&self.secret_crypto,
&workspace_id,
&operation.execution_config,
)
.await
{
Ok(auth) => auth,
Err(error) => {
return TaskAttemptOutcome::Failed(
serde_json::json!({ "code": error.to_string() }),
);
}
};
// Continue the parent trace persisted with the accepted Task; the worker's attempt span is a
// child of it, and the executor records the attempt identity. Trace data stays out of labels.
let trace_context = match TraceContext::parse(&lease.record.support.traceparent) {
Ok(context) => context,
Err(_) => {
return TaskAttemptOutcome::Failed(
serde_json::json!({ "code": "task trace parent is invalid" }),
);
}
};
let request_context = crank_runtime::RuntimeRequestContext::new(
lease.record.support.request_id.clone(),
trace_context,
)
.with_metering_context(
workspace_id.clone(),
Some(agent_id.clone()),
crank_core::InvocationSource::AgentToolCall,
);
let request = match RuntimeExecutionRequest::try_new(
&workspace_id,
ExecutionOrigin::AgentSnapshot,
Some(&agent_id),
&operation,
&input,
ExecutionAuthorization::Authorized,
resolved_auth.as_ref(),
&request_context,
deadline,
) {
Ok(request) => request,
Err(error) => {
return TaskAttemptOutcome::Failed(match error {
ExecutionRequestError::InvalidOrigin => {
serde_json::json!({"code": "invalid execution origin"})
}
ExecutionRequestError::InvalidOperationVersion => {
serde_json::json!({"code": "invalid operation version"})
}
ExecutionRequestError::DeadlineElapsed => {
serde_json::json!({"code": "task deadline elapsed"})
}
ExecutionRequestError::InvalidMeteringScope => {
serde_json::json!({"code": "invalid metering scope"})
}
});
}
};
if cancelled.load(Ordering::Acquire) {
return TaskAttemptOutcome::OutcomeUnknown;
}
match self.runtime.execute_task_outcome(request, permit).await {
Ok(success) => TaskAttemptOutcome::Completed(success.output),
Err(failure) => {
// An ambiguous outbound failure is recorded as uncertainty, never as a retryable
// failure: the durable checkpoint already exists, so Story 4.4 may reconcile it.
if failure.outcome_certainty() == crank_core::OutcomeCertainty::OutcomeUnknown {
return TaskAttemptOutcome::OutcomeUnknown;
}
TaskAttemptOutcome::Failed(
serde_json::json!({ "code": format!("{:?}", failure.error_code()) }),
)
}
}
}
}
+322
View File
@@ -0,0 +1,322 @@
//! Application entry of durable background Tasks (Story 4.2, FR-22/FR-23).
//!
//! The entry takes an already authenticated principal (a verified machine credential, or an
//! administrator identity verified by the calling admin surface), resolves the exact published Tool,
//! repeats the caller's authorization, pins the Published Version/policy/deadline and delegates the
//! durable acceptance to the runtime Task service. The repository owns the transaction, so the
//! returned Task ID exists only after Task + dispatch intent + queue slot committed.
//!
//! Deliberately not wired: the public MCP `tools/call`/Tasks wire (Story 4.7) and the worker/leases
//! (Story 4.3). The existing synchronous/unsupported `tools/call` path is unchanged, and this entry
//! never spawns an ephemeral execution future.
use std::sync::Arc;
use std::time::{Duration, Instant};
use crank_core::{
CorrelationContext, PlatformApiKeyId, TaskAcceptance, TaskAccess, TaskError, TaskId,
TaskOwnerKind, TaskRecord, ToolExecutionMode,
};
use crank_registry::{PostgresRegistry, PublishedAgentTool};
use crank_runtime::{TaskAcceptDraft, TaskOutcome, TaskService};
use serde_json::Value;
use crate::access::credential_allows_security_level;
use crate::catalog::PublishedToolCatalog;
/// Authenticated principal of one Task operation. The authorization scope identity is derived here
/// from the verified credential/admin identity, never from a caller-supplied request string.
#[derive(Clone, Debug)]
pub enum TaskPrincipal {
Agent {
workspace_slug: String,
agent_slug: String,
credential: crank_core::VerifiedMachineCredential,
},
/// Administrator identity that the admin surface already authenticated. It is never a wildcard:
/// owner, workspace and authorization scope all participate in the stored predicate.
Admin {
workspace_slug: String,
agent_slug: String,
user_id: String,
},
}
impl TaskPrincipal {
fn workspace_slug(&self) -> &str {
match self {
Self::Agent { workspace_slug, .. } | Self::Admin { workspace_slug, .. } => {
workspace_slug
}
}
}
fn agent_slug(&self) -> &str {
match self {
Self::Agent { agent_slug, .. } | Self::Admin { agent_slug, .. } => agent_slug,
}
}
fn owner(&self, tool: &PublishedAgentTool) -> (TaskOwnerKind, String) {
match self {
Self::Agent { .. } => (TaskOwnerKind::Agent, tool.agent_id.as_str().to_owned()),
Self::Admin { user_id, .. } => (TaskOwnerKind::Admin, user_id.clone()),
}
}
/// Typed immutable Platform API key identity of this principal. A verified agent credential
/// without a key ID cannot be revalidated by a worker, so it is refused instead of accepted;
/// an administrator has no machine key and is stored as `None` (the worker fails closed).
fn platform_api_key_id(&self) -> Result<Option<PlatformApiKeyId>, TaskApplicationError> {
match self {
Self::Agent { credential, .. } => credential
.platform_api_key_id
.clone()
.map(Some)
.ok_or(TaskApplicationError::Denied),
Self::Admin { .. } => Ok(None),
}
}
/// Bounded server-derived authorization scope identity of this principal.
fn authorization_scope(&self) -> String {
let identity = match self {
Self::Agent { credential, .. } => {
let mut scopes = credential
.scopes
.iter()
.map(|scope| format!("{scope:?}"))
.collect::<Vec<_>>();
scopes.sort();
format!(
"key:{}|mode:{:?}|level:{:?}|scopes:{}",
credential
.platform_api_key_id
.as_ref()
.map(|id| id.as_str().to_owned())
.unwrap_or_else(|| "-".to_owned()),
credential.machine_access_mode,
credential.max_security_level,
scopes.join(",")
)
}
Self::Admin { user_id, .. } => format!("admin:{user_id}"),
};
identity.chars().take(160).collect()
}
}
/// Typed refusal of the application entry. `NotFound`/`Denied` never reveal whether a foreign Task
/// exists, and no variant carries Task payload or fingerprint.
#[derive(Clone, Debug, PartialEq, Eq, thiserror::Error)]
pub enum TaskApplicationError {
#[error("task request was not found")]
NotFound,
#[error("task request is not authorized")]
Denied,
#[error("the published Tool policy does not accept durable background work")]
NotBackground,
#[error("the requested published Tool version is not available")]
VersionUnavailable,
#[error("this Tool requires a verified approval before acceptance")]
ApprovalRequired,
#[error(transparent)]
Task(#[from] TaskError),
}
#[derive(Clone)]
pub struct TaskApplication {
catalog: PublishedToolCatalog,
service: TaskService,
}
impl TaskApplication {
/// Production construction of the application entry over the process registry, the published
/// catalog and the durable Task service.
pub fn new(
registry: PostgresRegistry,
refresh_interval: Duration,
coordination_store: Arc<dyn crank_core::CoordinationStateStore>,
service: TaskService,
) -> Self {
Self {
catalog: PublishedToolCatalog::new(registry, refresh_interval, coordination_store),
service,
}
}
/// Runtime task service carrying encrypted executable input.
pub fn task_service(&self) -> TaskService {
self.service.clone()
}
/// Published catalog of the same application entry, for the durable Task worker's pre-dispatch
/// revalidation.
pub(super) fn catalog(&self) -> PublishedToolCatalog {
self.catalog.clone()
}
/// Readiness of the durable Task store this entry depends on.
pub async fn health(&self) -> Result<(), TaskError> {
self.service.health().await
}
/// Accepts one background request for an already authenticated principal.
///
/// `expected_version` is the caller's explicit Published Version pin. Without it the current
/// publication is pinned for new work and a retry of the same scoped key returns the stored Task
/// even after a newer publication. An explicit pin may only retry the Task accepted with that
/// exact version: a different pin is a typed conflict, never a rebind.
pub async fn accept_background(
&self,
principal: &TaskPrincipal,
tool_name: &str,
expected_version: Option<u32>,
scoped_key: &str,
input: &Value,
correlation: &CorrelationContext,
) -> Result<TaskAcceptance, TaskApplicationError> {
let tool = self.resolve_authorized_tool(principal, tool_name).await?;
let current_version = tool.operation.version;
let Some(policy) = tool.operation.execution_config.execution_policy else {
return Err(TaskApplicationError::NotBackground);
};
if policy.mode == ToolExecutionMode::Synchronous {
return Err(TaskApplicationError::NotBackground);
}
if tool
.operation
.execution_config
.approval_policy
.as_ref()
.is_some_and(|approval| approval.required)
{
// Without a verified, non-consuming approval reference this entry fails closed instead
// of accepting work a worker could never safely execute.
return Err(TaskApplicationError::ApprovalRequired);
}
let platform_api_key_id = principal.platform_api_key_id()?;
let draft = |operation_version: u32| {
let (owner_kind, owner_id) = principal.owner(&tool);
TaskAcceptDraft {
workspace_id: tool.workspace_id.as_str().to_owned(),
owner_kind,
owner_id,
authorization_scope: principal.authorization_scope(),
platform_api_key_id: platform_api_key_id.clone(),
operation_id: tool.operation.id.as_str().to_owned(),
operation_version,
policy_mode: policy.mode,
scoped_key: scoped_key.to_owned(),
input: input.clone(),
correlation: correlation.clone(),
deadline: Instant::now() + Duration::from_millis(policy.deadline_ms.max(1)),
retention_ms: TaskService::default_retention_ms(),
approval_reference: None,
}
};
let Some(requested) = expected_version else {
return Ok(self.service.accept(draft(current_version)).await?);
};
let draft = draft(requested);
match self.service.lookup(&draft).await? {
Some(record) => {
if record.input_fingerprint != TaskService::input_fingerprint(input)? {
return Err(TaskApplicationError::Task(TaskError::FingerprintConflict));
}
if record.pins.operation_version != requested {
return Err(TaskApplicationError::VersionUnavailable);
}
Ok(TaskAcceptance {
task_id: record.task_id,
state: record.state,
operation_version: record.pins.operation_version,
support: record.support,
created_at_unix_ms: record.created_at_unix_ms,
retain_until_unix_ms: record.pins.retain_until_unix_ms,
replayed: true,
})
}
None if requested == current_version => Ok(self.service.accept(draft).await?),
// A new Task can only be accepted against the currently published version.
None => Err(TaskApplicationError::VersionUnavailable),
}
}
/// Owner-scoped status read of one Task of the addressed published Tool.
pub async fn status(
&self,
principal: &TaskPrincipal,
tool_name: &str,
task_id: TaskId,
) -> Result<TaskRecord, TaskApplicationError> {
let access = self.task_access(principal, tool_name).await?;
match self.service.status(&access, task_id).await {
Ok(record) => Ok(record),
Err(TaskError::NotFound) => Err(TaskApplicationError::NotFound),
Err(error) => Err(TaskApplicationError::Task(error)),
}
}
/// Owner-scoped result projection of one Task of the addressed published Tool.
pub async fn result(
&self,
principal: &TaskPrincipal,
tool_name: &str,
task_id: TaskId,
) -> Result<TaskOutcome, TaskApplicationError> {
let access = self.task_access(principal, tool_name).await?;
match self.service.result(&access, task_id).await {
Ok(outcome) => Ok(outcome),
Err(TaskError::NotFound) => Err(TaskApplicationError::NotFound),
Err(error) => Err(TaskApplicationError::Task(error)),
}
}
async fn task_access(
&self,
principal: &TaskPrincipal,
tool_name: &str,
) -> Result<TaskAccess, TaskApplicationError> {
let tool = self.resolve_authorized_tool(principal, tool_name).await?;
let (owner_kind, owner_id) = principal.owner(&tool);
Ok(TaskAccess {
workspace_id: tool.workspace_id.as_str().to_owned(),
owner_kind,
owner_id,
authorization_scope: principal.authorization_scope(),
})
}
/// Resolves the addressed Tool of the principal's published catalog and repeats its authorization
/// check. A foreign workspace, an unknown Tool and an insufficient security level are typed
/// refusals without metadata.
async fn resolve_authorized_tool(
&self,
principal: &TaskPrincipal,
tool_name: &str,
) -> Result<PublishedAgentTool, TaskApplicationError> {
let catalog = self
.catalog
.get_catalog(principal.workspace_slug(), principal.agent_slug())
.await
.map_err(|_| TaskApplicationError::NotFound)?;
let tool = catalog
.tools
.into_iter()
.find(|tool| tool.tool_name == tool_name)
.ok_or(TaskApplicationError::NotFound)?;
if let TaskPrincipal::Admin { workspace_slug, .. } = principal
&& tool.workspace_slug != *workspace_slug
{
return Err(TaskApplicationError::Denied);
}
if let TaskPrincipal::Agent { credential, .. } = principal
&& !credential_allows_security_level(credential, tool.operation.security_level)
{
return Err(TaskApplicationError::Denied);
}
Ok(tool)
}
}
@@ -550,6 +550,7 @@ pub(super) fn test_operation(base_url: &str) -> Operation<Schema, MappingSet> {
idempotency: None,
safety: None,
approval_policy: None,
execution_policy: None,
auth_profile_ref: None,
headers: BTreeMap::new(),
},
@@ -18,6 +18,9 @@ pub const CURRENT_PROTOCOL_VERSION: &str = LEGACY_PROTOCOL_VERSION;
pub const META_PROTOCOL_VERSION_KEY: &str = "io.modelcontextprotocol/protocolVersion";
pub const META_CLIENT_CAPABILITIES_KEY: &str = "io.modelcontextprotocol/clientCapabilities";
pub const META_CLIENT_INFO_KEY: &str = "io.modelcontextprotocol/clientInfo";
/// `_meta` key of the per-request log level (Story 3.18). SEP-2577 replaces the former
/// `logging/setLevel` request of the legacy era with this per-request opt-in.
pub const META_LOG_LEVEL_KEY: &str = "io.modelcontextprotocol/logLevel";
/// Result `_meta` key carrying the self-reported server identity (pinned `DiscoverResult`).
pub const META_SERVER_INFO_KEY: &str = "io.modelcontextprotocol/serverInfo";
+3 -2
View File
@@ -16,6 +16,7 @@ mod tool_search;
mod transport;
pub use app::{
build_app, build_app_with_artifacts, build_app_with_background_workers,
build_app_with_background_workers_and_limits,
TaskApplication, TaskApplicationError, TaskPrincipal, TaskWorkerHandler, build_app,
build_app_with_artifacts, build_app_with_artifacts_and_tasks,
build_app_with_background_workers, build_app_with_background_workers_and_limits,
};
@@ -33,6 +33,7 @@ use crate::{
access::credential_allows_security_level,
app::{
AppState, ApprovalPolicyContext, ApprovalPolicyDecision, InvocationRecord,
lifecycle::{Phase, ToolCallLifecycle},
maybe_handle_approval_policy, persist_invocation_for_key, resolve_generated_tool,
resolve_operation_auth, runtime_operation, take_confirmation_token,
},
@@ -140,10 +141,15 @@ pub(crate) enum ToolCallOutcome {
/// Handle one shared Tools command. No wire envelope, `ResponseMode`, `axum::Response`, session or
/// protocol revision crosses this boundary.
///
/// `lifecycle` is the request-scoped progress/cancellation handle of a `tools/call` (Story 3.4);
/// a command that cannot have one (`tools/list`) passes `None`, and the service then simply never
/// produces a progress event.
pub(crate) async fn handle_tools_command(
state: &Arc<AppState>,
context: ServiceContext<'_>,
command: ToolsCommand,
lifecycle: Option<ToolCallLifecycle>,
) -> ToolsOutcome {
// One marker per entered owning Tools service/handler (Story 3.1 section 7.2). It is a
// `#[cfg(test)]`-only, request-scoped counter and has no production effect.
@@ -166,9 +172,9 @@ pub(crate) async fn handle_tools_command(
Err(message) => ToolsOutcome::InvalidParams(message.to_owned()),
}
}
ToolsCommand::Call(call) => {
ToolsOutcome::Call(execute_tool_command(state, &context, call, &catalog).await)
}
ToolsCommand::Call(call) => ToolsOutcome::Call(
execute_tool_command(state, &context, call, &catalog, lifecycle.as_ref()).await,
),
}
}
@@ -178,6 +184,7 @@ async fn execute_tool_command(
context: &ServiceContext<'_>,
command: ToolCallCommand,
catalog: &PublishedAgentCatalog,
lifecycle: Option<&ToolCallLifecycle>,
) -> ToolCallOutcome {
match catalog.tool_selection_policy.mode {
ToolAccessMode::Direct => {
@@ -188,6 +195,7 @@ async fn execute_tool_command(
&command.tool_name,
command.arguments,
command.confirmation_token,
lifecycle,
)
.await
}
@@ -222,6 +230,7 @@ async fn execute_tool_command(
&proxy.name,
arguments,
confirmation_token,
lifecycle,
)
.await
}
@@ -240,6 +249,7 @@ async fn execute_resolved_tool(
tool_name: &str,
arguments: Value,
confirmation_token: Option<String>,
lifecycle: Option<&ToolCallLifecycle>,
) -> ToolCallOutcome {
let correlation = context.correlation;
let transport_request_id = correlation.request_id().as_str();
@@ -250,6 +260,10 @@ async fn execute_resolved_tool(
Some(resolved) => {
StageOutcome::Success.record(&resolve_span);
drop(resolve_span);
// First real execution step: the addressed tool of the published catalog is resolved.
if let Some(lifecycle) = lifecycle {
lifecycle.advance(Phase::Resolved);
}
resolved.tool
}
None => {
@@ -349,8 +363,14 @@ async fn execute_resolved_tool(
let resolved_auth =
resolve_operation_auth(state, &tool.workspace_id, &operation.execution_config).await;
// A cancellation can only stop the call before it reaches its terminal outcome: nothing is
// persisted for a stopped call, so Invocation History can never claim a false `completed`.
let result = match resolved_auth {
Ok(resolved_auth) => {
// Second real step: the operation's own authorization is resolved, so the runtime may run.
if let Some(lifecycle) = lifecycle {
lifecycle.advance(Phase::Authorized);
}
match RuntimeExecutionRequest::try_new(
&tool.workspace_id,
ExecutionOrigin::AgentSnapshot,
@@ -365,7 +385,15 @@ async fn execute_resolved_tool(
operation.execution_config.timeout_ms.max(1),
),
) {
Ok(request) => state.runtime.execute_outcome(request).await,
Ok(request) => {
// Third real step: the request is about to enter the runtime, so from here on an
// outbound side effect may happen and a cancellation can no longer promise that
// nothing was sent.
if let Some(lifecycle) = lifecycle {
lifecycle.advance(Phase::Dispatching);
}
state.runtime.execute_outcome(request).await
}
Err(_) => Err(ExecutionFailure::new(
ExecutionErrorCode::RuntimeInternal,
correlation.clone(),
@@ -0,0 +1,378 @@
//! Story 3.18: safe MCP server logging (FR-19).
//!
//! Scope of this module is the server side of the pinned `logging` capability of the two approved
//! revisions, and nothing else:
//!
//! * `2025-11-25` (legacy session era) declares `logging/setLevel` as a client request with one
//! `level` of the pinned `LoggingLevel` union, and `logging?: object` in `ServerCapabilities`.
//! The request sets the severity a client wants to receive for its own session.
//! * `2026-07-28` (modern stateless era) has no `logging/setLevel` method at all: SEP-2577 replaces
//! it with the per-request `_meta["io.modelcontextprotocol/logLevel"]`, and the schema says that
//! when that key is absent "the server MUST NOT send any notifications/message notifications for
//! this request". The `logging` capability and `notifications/message` themselves stay in the
//! pinned schema as deprecated for at least twelve months.
//!
//! The answer of this story is therefore the narrow one the contract offers: an opt-in level that
//! belongs to the current negotiated client context (one legacy session, or one modern request —
//! never a process-global level shared by every agent), and `notifications/message` events that are
//! built **only** from the closed safe vocabulary below.
//!
//! Redaction is structural, not a filter: no request payload, credential, internal error cause,
//! Prompt/Resource/Tool name, URI, argument value or any other user-supplied value is ever accepted
//! by this module, so none of them can reach the wire. A `notifications/message` body is exactly
//! `{"level": <severity of the event>, "logger": "crank-mcp-server", "data": {"event": <closed name>,
//! "outcome": <closed label>}}` — every member of it is a `&'static str` from this file.
//!
//! Bounds (issue requirement: bounded frequency/queues/bytes, logging never blocks execution and
//! never replaces the audit trail):
//! * at most one event per response is recorded, and at most [`MAX_PENDING_NOTIFICATIONS`] are
//! drained into one response — a fixed, per-response frequency bound;
//! * every event has a fixed size and no variable input, so the byte bound follows from the count
//! bound without a separate accounting path;
//! * recording is a bounded in-memory push that drops the event when the buffer is full, the active
//! level is unset, or the task has no request scope. No I/O, no lock acquisition that can wait, and
//! no error path: logging can never fail or slow down the request it describes.
//! * the legacy opt-in table is a bounded in-memory map read/written with `try_lock` only.
use std::{
cell::RefCell,
collections::HashMap,
future::Future,
sync::{Mutex, MutexGuard},
};
use serde::Deserialize;
use serde_json::{Value, json};
use tokio::task_local;
/// The pinned `LoggingLevel` union: RFC 5424 severities, ordered from the most verbose to the most
/// severe. The ordering is the filtering contract — a client is sent the events at or above its
/// level.
#[derive(Clone, Copy, Debug, PartialEq, Eq, PartialOrd, Ord)]
pub(crate) enum LogLevel {
Debug,
Info,
Notice,
Warning,
Error,
Critical,
Alert,
Emergency,
}
impl LogLevel {
/// The exact wire literal of this level.
pub(crate) const fn as_str(self) -> &'static str {
match self {
Self::Debug => "debug",
Self::Info => "info",
Self::Notice => "notice",
Self::Warning => "warning",
Self::Error => "error",
Self::Critical => "critical",
Self::Alert => "alert",
Self::Emergency => "emergency",
}
}
/// Exact parse of the eight pinned literals.
///
/// Nothing is coerced: no case folding, no aliases (`warn`), no numeric severities. A value that
/// is not one of the pinned strings is malformed, which is what the pinned `LoggingLevel` type
/// says and what `InvalidParamsError` is for.
pub(crate) fn parse(value: &str) -> Option<Self> {
match value {
"debug" => Some(Self::Debug),
"info" => Some(Self::Info),
"notice" => Some(Self::Notice),
"warning" => Some(Self::Warning),
"error" => Some(Self::Error),
"critical" => Some(Self::Critical),
"alert" => Some(Self::Alert),
"emergency" => Some(Self::Emergency),
_ => None,
}
}
}
/// Bound of `params.level` before it is compared: the longest pinned literal is nine bytes, so a
/// value beyond this cannot be a level and must not be scanned further.
pub(crate) const MAX_LEVEL_BYTES: usize = 16;
/// The one safe detail of a malformed `logging/setLevel`: it never echoes the offending value, so an
/// oversized or hostile value cannot be reflected back or widened into an error payload.
pub(crate) const INVALID_LEVEL: &str = "invalid logging level";
/// Notifications drained into one response (frequency bound).
pub(crate) const MAX_PENDING_NOTIFICATIONS: usize = 4;
/// Bound of the in-memory legacy opt-in table (memory bound; see [`LogLevelStore`]).
pub(crate) const MAX_TRACKED_SESSIONS: usize = 1_024;
/// The fixed logger name carried by every notification. It is a server constant, never a
/// client-supplied or catalog-supplied value.
const LOGGER_NAME: &str = "crank-mcp-server";
/// Closed safe vocabulary: the server served the request.
pub(crate) const EVENT_SERVED: &str = "mcp.request.served";
/// Closed safe vocabulary: the request was rejected as invalid before any service work.
pub(crate) const EVENT_REJECTED: &str = "mcp.request.rejected";
/// Closed safe vocabulary: the request failed on the server side.
pub(crate) const EVENT_FAILED: &str = "mcp.request.failed";
/// Closed safe vocabulary: this client set its logging level.
pub(crate) const EVENT_LEVEL_APPLIED: &str = "mcp.logging.level_applied";
/// The `params` of one `logging/setLevel` request as the legacy adapter parses it from the wire form.
///
/// Only the single pinned member exists here: an unknown member is ignored by serde, and a `level`
/// that is not a string is a deserialization error, which the adapter answers with the same safe
/// [`INVALID_LEVEL`] detail instead of a serde diagnostic.
#[derive(Debug, Deserialize)]
pub(crate) struct SetLevelCommand {
pub(crate) level: String,
}
impl SetLevelCommand {
/// The bounded input contract of AC2: the level is length-bounded and compared exactly, so a
/// malformed or oversized value is refused with bounded work and one safe detail.
pub(crate) fn validate(&self) -> Result<LogLevel, &'static str> {
if self.level.is_empty() || self.level.len() > MAX_LEVEL_BYTES {
return Err(INVALID_LEVEL);
}
LogLevel::parse(&self.level).ok_or(INVALID_LEVEL)
}
}
/// A neutral outcome of the shared `logging/setLevel` service.
pub(crate) enum LoggingOutcome {
/// The level of this client context was accepted.
Applied(LogLevel),
/// Malformed or oversized level: one safe detail, no echoed input.
InvalidParams(&'static str),
}
/// One recorded event of the closed vocabulary.
#[derive(Debug, Clone, Copy)]
struct SafeEvent {
level: LogLevel,
event: &'static str,
outcome: &'static str,
}
/// The request-scoped logging state: the level this client opted in to, whether the client offered
/// the SSE response mode that is the only carrier of a server notification, and the bounded events
/// this response will carry.
#[derive(Default)]
struct RequestLog {
active: Option<LogLevel>,
sse_acceptable: bool,
pending: Vec<SafeEvent>,
}
task_local! {
/// Per-request logging scope. Installed by the HTTP entry point of the MCP endpoint, so two
/// concurrently served requests never share a level or a buffer. Code outside such a scope
/// (unit tests, background workers) sees a no-op.
static REQUEST_LOG: RefCell<RequestLog>;
}
/// Run one request inside its own logging scope.
pub(crate) async fn scoped<F: Future>(future: F) -> F::Output {
REQUEST_LOG
.scope(RefCell::new(RequestLog::default()), future)
.await
}
/// Activate (or clear) the level of the current client context. Outside a scope this is a no-op.
pub(crate) fn set_active_level(level: Option<LogLevel>) {
let _ = REQUEST_LOG.try_with(|log| {
if let Ok(mut log) = log.try_borrow_mut() {
log.active = level;
}
});
}
/// The request's logging context, carried into a streamed response body (Story 3.4).
///
/// A streamed `tools/call` response is assembled after the request handler returned, so its terminal
/// event would be projected outside the task-local scope of the request. This snapshot carries the
/// only two facts the projection needs — the level the client opted in to and whether the request
/// offered the SSE carrier — and never a request value.
#[derive(Clone, Copy, Default)]
pub(crate) struct CarriedScope {
active: Option<LogLevel>,
sse_acceptable: bool,
}
/// Snapshot the scope of the request that is being served now. Outside a scope the snapshot is
/// "not opted in", which delivers nothing.
pub(crate) fn carry_scope() -> CarriedScope {
REQUEST_LOG
.try_with(|log| {
let log = log.borrow();
CarriedScope {
active: log.active,
sse_acceptable: log.sse_acceptable,
}
})
.unwrap_or_default()
}
/// Run one step of the streamed response inside a scope carrying the request's logging context.
pub(crate) async fn in_carried_scope<F: Future>(scope: CarriedScope, future: F) -> F::Output {
REQUEST_LOG
.scope(
RefCell::new(RequestLog {
active: scope.active,
sse_acceptable: scope.sse_acceptable,
pending: Vec::new(),
}),
future,
)
.await
}
/// Remember that this request offered `text/event-stream`, the only carrier of a notification.
pub(crate) fn set_sse_acceptable() {
let _ = REQUEST_LOG.try_with(|log| {
if let Ok(mut log) = log.try_borrow_mut() {
log.sse_acceptable = true;
}
});
}
/// Record one event of the closed vocabulary for the level this client opted in to.
///
/// Everything below the active level, every event without an active level (the client did not opt
/// in), and every event beyond the fixed buffer bound is dropped silently: the caller never branches
/// on logging.
pub(crate) fn note(level: LogLevel, event: &'static str, outcome: &'static str) {
let _ = REQUEST_LOG.try_with(|log| {
let Ok(mut log) = log.try_borrow_mut() else {
return;
};
let Some(active) = log.active else {
return;
};
if level < active || log.pending.len() >= MAX_PENDING_NOTIFICATIONS {
return;
}
log.pending.push(SafeEvent {
level,
event,
outcome,
});
});
}
/// The pending events as wire `notifications/message` bodies, and whether this request can carry
/// them at all. Draining is the last step of the response projection, so a notification can never be
/// delivered twice or survive into another request.
pub(crate) fn take_notifications() -> Vec<Value> {
REQUEST_LOG
.try_with(|log| {
let Ok(mut log) = log.try_borrow_mut() else {
return Vec::new();
};
if !log.sse_acceptable {
// A JSON body has no place for a server notification: the event is not delivered
// instead of being smuggled into the result object.
log.pending.clear();
return Vec::new();
}
std::mem::take(&mut log.pending)
.into_iter()
.map(|event| {
json!({
"jsonrpc": "2.0",
"method": "notifications/message",
"params": {
"level": event.level.as_str(),
"logger": LOGGER_NAME,
"data": {
"event": event.event,
"outcome": event.outcome,
}
}
})
})
.collect()
})
.unwrap_or_default()
}
/// The per-session opt-in of the legacy era.
///
/// The login belongs to the session the client negotiated, so a level set by one agent's client can
/// never apply to another. It is deliberately **in-memory only**: persisting it would need a schema
/// migration of the transport session store, and the deprecated capability does not justify that.
///
/// shortcut: the table is per process — a level set on one replica is not seen by another replica
/// serving the same session, and a restart clears it; upgrade to the session store if a durable or
/// cross-replica level is ever required.
#[derive(Default)]
pub(crate) struct LogLevelStore {
levels: Mutex<HashMap<String, LogLevel>>,
}
impl LogLevelStore {
pub(crate) fn new() -> Self {
Self::default()
}
/// Store the level of one session. Never waits and never fails: on lock contention the update is
/// skipped rather than blocking the request.
pub(crate) fn set(&self, session_id: &str, level: LogLevel) {
let Some(mut levels) = self.lock() else {
return;
};
if levels.len() >= MAX_TRACKED_SESSIONS && !levels.contains_key(session_id) {
// Bounded table: the oldest opt-ins are dropped instead of growing without a limit. A
// dropped client simply stops receiving notifications until it sets its level again.
levels.clear();
}
levels.insert(session_id.to_owned(), level);
}
/// The level of one session, if this process still remembers it.
pub(crate) fn get(&self, session_id: &str) -> Option<LogLevel> {
self.lock()?.get(session_id).copied()
}
fn lock(&self) -> Option<MutexGuard<'_, HashMap<String, LogLevel>>> {
self.levels.try_lock().ok()
}
}
/// Serve one `logging/setLevel` request of the legacy era (AC1/AC2).
///
/// The command has already been parsed from the wire form, so this function only validates and
/// records; it never reads the registry, the catalog or any stored value.
pub(crate) fn handle_logging_command(
store: &LogLevelStore,
session_id: &str,
command: SetLevelCommand,
) -> LoggingOutcome {
match command.validate() {
Ok(level) => {
store.set(session_id, level);
LoggingOutcome::Applied(level)
}
Err(detail) => LoggingOutcome::InvalidParams(detail),
}
}
/// The level a modern request asked for in `_meta`.
///
/// The pinned schema says the server must not send `notifications/message` for a request that does
/// not carry the key, so an absent, non-string or unknown value is simply "not opted in": it is not
/// an error of the business request and never widens the accepted vocabulary.
pub(crate) fn modern_request_level(message: &Value) -> Option<LogLevel> {
let meta = crate::jsonrpc::params(message).get("_meta")?.clone();
let value = meta.get(crate::jsonrpc::META_LOG_LEVEL_KEY)?.as_str()?;
LogLevel::parse(value)
}
#[cfg(test)]
#[path = "logging_tests.rs"]
mod tests;
@@ -0,0 +1,316 @@
//! Unit contract of Story 3.18: the pinned level vocabulary, the bounded input contract of
//! `logging/setLevel`, the request-scoped bounded event buffer and the closed safe notification
//! shape.
use serde_json::json;
use super::*;
/// Parse one `logging/setLevel` params object and validate its level, or `None` when the shape itself
/// is malformed (which the adapter answers with the same safe detail).
fn validate(level: &str) -> Option<Result<LogLevel, &'static str>> {
let command: SetLevelCommand = serde_json::from_value(json!({ "level": level })).ok()?;
Some(command.validate())
}
#[test]
fn every_pinned_level_is_accepted_exactly_once_and_in_severity_order() {
// The pinned `LoggingLevel` union of both revisions, most verbose first.
let pinned = [
("debug", LogLevel::Debug),
("info", LogLevel::Info),
("notice", LogLevel::Notice),
("warning", LogLevel::Warning),
("error", LogLevel::Error),
("critical", LogLevel::Critical),
("alert", LogLevel::Alert),
("emergency", LogLevel::Emergency),
];
for (index, (literal, expected)) in pinned.iter().enumerate() {
assert_eq!(
LogLevel::parse(literal),
Some(*expected),
"{literal} is a pinned level"
);
assert_eq!(
expected.as_str(),
*literal,
"the wire literal of a level is exactly the pinned string"
);
assert_eq!(
*expected as usize, index,
"the severity ordering of {literal} must follow the RFC 5424 order"
);
assert_eq!(
validate(literal),
Some(Ok(*expected)),
"{literal} must be accepted through the wire shape too"
);
}
assert!(
pinned.windows(2).all(|pair| pair[0].1 < pair[1].1),
"severity must be strictly increasing so the filter is `event >= level`"
);
}
#[test]
fn nothing_outside_the_pinned_union_is_a_level() {
for value in [
"",
"warn",
"WARN",
"Info",
"information",
" debug",
"debug ",
"0",
"7",
"emergency!",
"debug\u{0}",
"информация",
] {
assert_eq!(
LogLevel::parse(value),
None,
"{value:?} is not one of the pinned literals and must never be coerced into one"
);
assert_eq!(
validate(value),
Some(Err(INVALID_LEVEL)),
"a value outside the union is a bounded invalid-params detail, got {value:?}"
);
}
}
#[test]
fn malformed_level_shapes_share_one_safe_detail() {
// A `level` the schema types as a string, sent as something else: a deserialization error that the
// adapter answers with the same safe detail. It never echoes what the client sent.
for value in [json!(7), json!(1.5), json!(0)] {
assert!(
serde_json::from_value::<SetLevelCommand>(json!({ "level": value })).is_err(),
"a numeric level is not a pinned LoggingLevel"
);
}
for value in [json!(null), json!({}), json!(["info"]), json!(true)] {
assert!(
serde_json::from_value::<SetLevelCommand>(json!({ "level": value })).is_err(),
"a non-string level is not a pinned LoggingLevel"
);
}
assert!(
serde_json::from_value::<SetLevelCommand>(json!({})).is_err(),
"the pinned request requires `level`"
);
// Control: the same safe detail is what an unknown string produces, so both malformed shapes are
// indistinguishable on the wire and neither discloses what was sent.
assert_eq!(validate("verbose"), Some(Err(INVALID_LEVEL)));
assert!(
INVALID_LEVEL.len() < 64,
"the detail stays a fixed short text"
);
}
#[test]
fn an_oversized_level_is_refused_before_any_comparison() {
let oversized = "i".repeat(MAX_LEVEL_BYTES + 1);
assert_eq!(validate(&oversized), Some(Err(INVALID_LEVEL)));
// Positive control: the bound is not so tight that a pinned literal is refused.
assert_eq!(validate("emergency"), Some(Ok(LogLevel::Emergency)));
}
#[test]
fn the_session_opt_in_is_per_session_and_bounded() {
let store = LogLevelStore::new();
store.set("session-a", LogLevel::Error);
store.set("session-b", LogLevel::Debug);
assert_eq!(store.get("session-a"), Some(LogLevel::Error));
assert_eq!(store.get("session-b"), Some(LogLevel::Debug));
assert_eq!(
store.get("session-c"),
None,
"a session that never opted in must not inherit another session's level"
);
// The table stays bounded: filling it past the cap drops the previous opt-ins instead of growing.
for index in 0..MAX_TRACKED_SESSIONS {
store.set(&format!("filler-{index}"), LogLevel::Info);
}
store.set("session-after-cap", LogLevel::Alert);
assert_eq!(store.get("session-after-cap"), Some(LogLevel::Alert));
}
#[tokio::test]
async fn an_event_is_delivered_only_at_or_above_the_opted_in_level() {
// One closed-vocabulary event per severity, recorded inside a request scope that offered SSE.
for (active, expected) in [
(LogLevel::Debug, 4usize),
(LogLevel::Info, 4),
(LogLevel::Notice, 3),
(LogLevel::Warning, 2),
(LogLevel::Error, 1),
(LogLevel::Critical, 0),
(LogLevel::Alert, 0),
(LogLevel::Emergency, 0),
] {
let delivered = scoped(async move {
set_active_level(Some(active));
set_sse_acceptable();
note(LogLevel::Info, EVENT_SERVED, "ok");
note(LogLevel::Notice, EVENT_LEVEL_APPLIED, "applied");
note(LogLevel::Warning, EVENT_REJECTED, "invalid_params");
note(LogLevel::Error, EVENT_FAILED, "internal");
take_notifications()
})
.await;
assert_eq!(
delivered.len(),
expected,
"level {} must deliver exactly the events at or above it",
active.as_str()
);
}
}
#[tokio::test]
async fn without_an_opt_in_nothing_is_even_recorded() {
// Outside a scope (no request, e.g. a background worker): a no-op.
note(LogLevel::Info, EVENT_SERVED, "ok");
assert!(take_notifications().is_empty());
// Inside a scope without an opted-in level.
let no_level = scoped(async {
set_sse_acceptable();
note(LogLevel::Error, EVENT_FAILED, "internal");
take_notifications()
})
.await;
assert!(
no_level.is_empty(),
"the pinned modern schema forbids notifications for a client that did not opt in"
);
}
#[tokio::test]
async fn a_level_is_present_but_the_client_offered_no_stream() {
let delivered = scoped(async {
set_active_level(Some(LogLevel::Debug));
note(LogLevel::Error, EVENT_FAILED, "internal");
take_notifications()
})
.await;
assert!(
delivered.is_empty(),
"a JSON body has no carrier for notifications/message; the event is dropped, not smuggled in"
);
}
#[tokio::test]
async fn the_buffer_is_bounded_and_drains_once() {
let (first, second) = scoped(async {
set_active_level(Some(LogLevel::Debug));
set_sse_acceptable();
for _ in 0..(MAX_PENDING_NOTIFICATIONS * 4) {
note(LogLevel::Info, EVENT_SERVED, "ok");
}
let first = take_notifications();
let second = take_notifications();
(first, second)
})
.await;
assert_eq!(
first.len(),
MAX_PENDING_NOTIFICATIONS,
"the per-response frequency bound"
);
assert!(
second.is_empty(),
"a delivered event must not survive into a later response"
);
}
#[tokio::test]
async fn the_notification_body_is_exactly_the_closed_safe_shape() {
let delivered = scoped(async {
set_active_level(Some(LogLevel::Debug));
set_sse_acceptable();
note(LogLevel::Warning, EVENT_REJECTED, "invalid_params");
take_notifications()
})
.await;
assert_eq!(
delivered,
vec![json!({
"jsonrpc": "2.0",
"method": "notifications/message",
"params": {
"level": "warning",
"logger": "crank-mcp-server",
"data": {"event": "mcp.request.rejected", "outcome": "invalid_params"}
}
})],
"`params` carries only the severity, the fixed logger name and the closed data object"
);
// A notification has no `id`, so it can never be mistaken for a response.
assert!(delivered[0].get("id").is_none());
}
#[test]
fn the_modern_level_is_read_from_the_pinned_meta_key_only() {
let modern = |meta: Value| json!({"jsonrpc": "2.0", "id": 1, "method": "tools/list", "params": {"_meta": meta}});
assert_eq!(
modern_request_level(&modern(
json!({ "io.modelcontextprotocol/logLevel": "error" })
)),
Some(LogLevel::Error)
);
// Not opted in: an absent key, a non-string value and an unknown literal are all "no level".
assert_eq!(modern_request_level(&modern(json!({}))), None);
assert_eq!(
modern_request_level(&modern(json!({ "io.modelcontextprotocol/logLevel": 7 }))),
None
);
assert_eq!(
modern_request_level(&modern(
json!({ "io.modelcontextprotocol/logLevel": "verbose" })
)),
None
);
// A level under another key is not the pinned carrier.
assert_eq!(
modern_request_level(&modern(json!({ "logLevel": "error" }))),
None
);
assert_eq!(
modern_request_level(&json!({"jsonrpc": "2.0", "id": 1, "method": "tools/list"})),
None
);
}
#[test]
fn the_command_records_the_level_of_its_own_session_only() {
let store = LogLevelStore::new();
store.set("other-session", LogLevel::Debug);
let command: SetLevelCommand =
serde_json::from_value(json!({ "level": "critical" })).expect("a pinned level parses");
let outcome = handle_logging_command(&store, "this-session", command);
assert!(matches!(
outcome,
LoggingOutcome::Applied(LogLevel::Critical)
));
assert_eq!(store.get("this-session"), Some(LogLevel::Critical));
assert_eq!(
store.get("other-session"),
Some(LogLevel::Debug),
"setting one session's level must never change another client's level"
);
// A malformed level changes nothing.
let malformed: SetLevelCommand =
serde_json::from_value(json!({ "level": "verbose" })).expect("the shape parses");
assert!(matches!(
handle_logging_command(&store, "this-session", malformed),
LoggingOutcome::InvalidParams(INVALID_LEVEL)
));
assert_eq!(store.get("this-session"), Some(LogLevel::Critical));
}
@@ -26,15 +26,23 @@ use crate::transport::{json_response, protocol_version_from_headers, singleton_h
mod command;
mod completion;
mod logging;
mod prompts;
mod registry;
mod resources;
mod signals;
pub(crate) use command::{
ContinuationSupport, ServiceContext, ToolCallCommand, ToolCallOutcome, ToolsCommand,
ToolsListCommand, ToolsOutcome, handle_tools_command,
};
pub(crate) use completion::{CompletionCommand, CompletionOutcome, handle_completion_command};
pub(crate) use logging::{
CarriedScope, EVENT_FAILED, EVENT_LEVEL_APPLIED, EVENT_REJECTED, EVENT_SERVED, INVALID_LEVEL,
LogLevel, LogLevelStore, LoggingOutcome, SetLevelCommand, carry_scope, handle_logging_command,
in_carried_scope, modern_request_level, note, scoped as logging_scope, set_active_level,
set_sse_acceptable, take_notifications,
};
pub(crate) use prompts::{
PromptGetCommand, PromptsCommand, PromptsListCommand, PromptsOutcome, handle_prompts_command,
};
@@ -42,6 +50,10 @@ pub(crate) use resources::{
ResourceReadCommand, ResourceTemplatesListError, ResourcesCommand, ResourcesListCommand,
ResourcesOutcome, handle_resources_command, list_resource_templates,
};
pub(crate) use signals::{
FilterOutcome, INVALID_LISTEN_FILTER, ListenParams, Signals, StreamRegistry, Watch, Wire,
acknowledged_frame, frames_stream, spawn as spawn_signal_watch, subscription_id,
};
/// Resolved era of an incoming POST request.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
@@ -22,6 +22,20 @@ pub(crate) enum CapabilityId {
Prompts,
/// Bounded argument completion for bound Published Prompts and Resource templates (Story 3.17).
Completions,
/// `subscriptions/listen` of the modern revision: a *client-sent* request that opens the
/// notification stream (Story 3.19). It is a dispatch-only registration without an advertisement
/// fragment because the pinned modern schema has no server `subscriptions` capability object;
/// there is no such method in the legacy revision at all.
Subscriptions,
/// `logging` server capability: safe structured `notifications/message` events of the negotiated
/// client context (Story 3.18). Advertised in both revisions; the pinned modern revision carries
/// the level per request in `_meta` and marks the capability deprecated (SEP-2577).
Logging,
/// `logging/setLevel` as the *legacy* client request that sets a session level. It is a
/// dispatch-only registration without an advertisement fragment because the pinned modern
/// revision has no such method: there the per-request `_meta` key replaces it, so the same wire
/// name is a wrong-era request and must never be served.
LoggingSetLevel,
Ping,
Discovery,
LegacyLifecycle,
@@ -81,9 +95,20 @@ pub(crate) enum MethodHandler {
/// `completion/complete`: bounded suggestions from the typed schema of one exact bound Prompt
/// argument or one exact bound Resource template parameter (Story 3.17).
Completion,
/// `subscriptions/listen`: open the long-lived modern notification stream and deliver the
/// Resource/Prompt list-change signals the client opted in to (Story 3.19).
SubscriptionsListen,
/// `logging/setLevel`: validate the pinned level and record it for this legacy session, so the
/// adapter can deliver safe structured `notifications/message` events of that context
/// (Story 3.18). The handler never widens the accepted vocabulary and never echoes the input.
LoggingSetLevel,
Ping,
Discover,
Lifecycle,
/// `notifications/cancelled`: the legacy client cancels a `tools/call` it already issued
/// (Story 3.4). The handler only routes the id to the owner's in-flight call; it never widens
/// what may be cancelled and never authorizes by the progress token.
Cancelled,
}
#[derive(Clone, Copy, Debug)]
@@ -348,7 +373,9 @@ pub(crate) const CAPABILITY_DESCRIPTORS: &[CapabilityIdDescriptor] = &[
extension: None,
deprecated: None,
sub_features: FeatureSet(&["listChanged"]),
advertised: Some(("tools", &[("listChanged", false)])),
// Story 3.5: the adapter emits `notifications/tools/list_changed` from the committed
// published Tool bindings of the Agent snapshot, so the declaration is now true.
advertised: Some(("tools", &[("listChanged", true)])),
revisions: MODERN_AND_LEGACY,
},
CapabilityIdDescriptor {
@@ -358,7 +385,9 @@ pub(crate) const CAPABILITY_DESCRIPTORS: &[CapabilityIdDescriptor] = &[
extension: None,
deprecated: None,
sub_features: FeatureSet(&["subscribe", "listChanged"]),
advertised: Some(("resources", &[("subscribe", false), ("listChanged", false)])),
// Story 3.19: the server really emits `notifications/resources/list_changed`; URI-content
// subscriptions (`subscribe`) remain unimplemented and are still advertised as false.
advertised: Some(("resources", &[("subscribe", false), ("listChanged", true)])),
revisions: MODERN_AND_LEGACY,
},
CapabilityIdDescriptor {
@@ -368,7 +397,8 @@ pub(crate) const CAPABILITY_DESCRIPTORS: &[CapabilityIdDescriptor] = &[
extension: None,
deprecated: None,
sub_features: FeatureSet(&["listChanged"]),
advertised: Some(("prompts", &[("listChanged", false)])),
// Story 3.19: the server really emits `notifications/prompts/list_changed`.
advertised: Some(("prompts", &[("listChanged", true)])),
revisions: MODERN_AND_LEGACY,
},
CapabilityIdDescriptor {
@@ -382,6 +412,46 @@ pub(crate) const CAPABILITY_DESCRIPTORS: &[CapabilityIdDescriptor] = &[
advertised: Some(("completions", &[])),
revisions: MODERN_AND_LEGACY,
},
CapabilityIdDescriptor {
// Story 3.18: the standard `logging` declaration of both approved revisions. The pinned
// modern revision marks it deprecated (SEP-2577) but keeps it in the schema for at least
// twelve months, and `notifications/message` remains the only log channel there; the legacy
// revision additionally serves `logging/setLevel`.
id: CapabilityId::Logging,
roster: Roster::Server,
kind: Kind::Capability,
extension: None,
deprecated: None,
// The pinned schema declares `logging` as an open object with no named sub-features.
sub_features: FeatureSet(&[]),
advertised: Some(("logging", &[])),
revisions: MODERN_AND_LEGACY,
},
CapabilityIdDescriptor {
// Dispatch-only registration of the legacy `logging/setLevel` request: it is a client
// request, not an advertised server capability, and it does not exist in the modern schema.
id: CapabilityId::LoggingSetLevel,
roster: Roster::Server,
kind: Kind::Method,
extension: None,
deprecated: None,
sub_features: FeatureSet(&[]),
advertised: None,
revisions: LEGACY_ONLY,
},
CapabilityIdDescriptor {
// Story 3.19: dispatch-only registration of the modern `subscriptions/listen` request. The
// client sends it; the pinned modern schema declares no server `subscriptions` capability
// object, so it carries no advertisement fragment. The legacy revision has no such method.
id: CapabilityId::Subscriptions,
roster: Roster::Client,
kind: Kind::Method,
extension: None,
deprecated: None,
sub_features: FeatureSet(&[]),
advertised: None,
revisions: MODERN_ONLY,
},
CapabilityIdDescriptor {
id: CapabilityId::Ping,
roster: Roster::Server,
@@ -486,6 +556,27 @@ pub(crate) const METHOD_DESCRIPTORS: &[MethodDescriptor] = &[
required_headers: HEADERS_METHOD_ONLY,
handler: MethodHandler::Completion,
},
MethodDescriptor {
// Story 3.18: the legacy client request that sets the log level of its session. It declares
// the level, not a server-side name, so only the method header applies; the pinned modern
// revision has no such method at all (per-request `_meta` replaces it).
name: "logging/setLevel",
capability: CapabilityId::LoggingSetLevel,
kind: Kind::Method,
required_headers: HEADERS_METHOD_ONLY,
handler: MethodHandler::LoggingSetLevel,
},
MethodDescriptor {
// Story 3.19: the modern long-lived notification stream. It declares the notification filter
// in its params, not a server-side name, so only the method header applies. The legacy
// revision opens its notification stream with `GET` instead, so this name is a wrong-era
// request there and must never be served.
name: "subscriptions/listen",
capability: CapabilityId::Subscriptions,
kind: Kind::Method,
required_headers: HEADERS_METHOD_ONLY,
handler: MethodHandler::SubscriptionsListen,
},
MethodDescriptor {
name: "ping",
capability: CapabilityId::Ping,
@@ -514,6 +605,17 @@ pub(crate) const METHOD_DESCRIPTORS: &[MethodDescriptor] = &[
required_headers: HEADERS_NONE,
handler: MethodHandler::Lifecycle,
},
MethodDescriptor {
// Story 3.4: the legacy revision's client-side cancellation of a request it issued. The
// pinned modern revision has no client-to-server notification over Streamable HTTP (closing
// the response stream is its cancellation signal), so this name exists in the legacy era
// only. It is a dispatch-only notification registration and advertises nothing.
name: "notifications/cancelled",
capability: CapabilityId::LegacyLifecycle,
kind: Kind::Notification,
required_headers: HEADERS_NONE,
handler: MethodHandler::Cancelled,
},
];
#[cfg(test)]
@@ -36,9 +36,10 @@ fn static_registration_is_unique_and_advertises_only_the_registered_capability()
registry.advertised_capabilities(revision),
json!({
"completions": {},
"prompts": {"listChanged": false},
"resources": {"listChanged": false, "subscribe": false},
"tools": {"listChanged": false}
"logging": {},
"prompts": {"listChanged": true},
"resources": {"listChanged": true, "subscribe": false},
"tools": {"listChanged": true}
}),
"the registered advertisement must be the same reviewed object in {revision}"
);
@@ -46,9 +47,10 @@ fn static_registration_is_unique_and_advertises_only_the_registered_capability()
advertised_server_capabilities(revision),
json!({
"completions": {},
"prompts": {"listChanged": false},
"resources": {"listChanged": false, "subscribe": false},
"tools": {"listChanged": false}
"logging": {},
"prompts": {"listChanged": true},
"resources": {"listChanged": true, "subscribe": false},
"tools": {"listChanged": true}
}),
"the production advertisement entry point must serve {revision}"
);
@@ -259,9 +261,10 @@ fn an_inert_test_descriptor_changes_both_projections_and_is_never_registered_in_
without.advertised_capabilities(MODERN_PROTOCOL_VERSION),
json!({
"completions": {},
"prompts": {"listChanged": false},
"resources": {"listChanged": false, "subscribe": false},
"tools": {"listChanged": false}
"logging": {},
"prompts": {"listChanged": true},
"resources": {"listChanged": true, "subscribe": false},
"tools": {"listChanged": true}
})
);
assert_eq!(
@@ -269,9 +272,10 @@ fn an_inert_test_descriptor_changes_both_projections_and_is_never_registered_in_
json!({
"inert_test_capability": {"enabled": true},
"completions": {},
"prompts": {"listChanged": false},
"resources": {"listChanged": false, "subscribe": false},
"tools": {"listChanged": false}
"logging": {},
"prompts": {"listChanged": true},
"resources": {"listChanged": true, "subscribe": false},
"tools": {"listChanged": true}
})
);
assert_eq!(
@@ -279,9 +283,10 @@ fn an_inert_test_descriptor_changes_both_projections_and_is_never_registered_in_
json!({
"inert_test_capability": {"enabled": true},
"completions": {},
"prompts": {"listChanged": false},
"resources": {"listChanged": false, "subscribe": false},
"tools": {"listChanged": false}
"logging": {},
"prompts": {"listChanged": true},
"resources": {"listChanged": true, "subscribe": false},
"tools": {"listChanged": true}
})
);
@@ -462,9 +467,10 @@ fn a_single_era_descriptor_never_leaks_into_the_other_era_advertisement() {
other_era_advertisement,
json!({
"completions": {},
"prompts": {"listChanged": false},
"resources": {"listChanged": false, "subscribe": false},
"tools": {"listChanged": false}
"logging": {},
"prompts": {"listChanged": true},
"resources": {"listChanged": true, "subscribe": false},
"tools": {"listChanged": true}
}),
"the other era advertisement stays the reviewed object"
);
@@ -654,6 +660,56 @@ fn story317_completion_is_advertised_and_dispatchable_in_both_revisions() {
);
}
#[test]
fn story318_logging_is_advertised_on_both_revisions_and_set_level_is_legacy_only() {
let registry = production_registry();
for revision in [MODERN_PROTOCOL_VERSION, LEGACY_PROTOCOL_VERSION] {
assert!(
registry.advertise_for(CapabilityId::Logging, revision),
"the implemented logging surface must be advertised in {revision}"
);
assert_eq!(
registry.advertised_capabilities(revision)["logging"],
json!({}),
"the pinned advertisement is an empty `logging` object in {revision}"
);
}
// `logging/setLevel` is a legacy request only: the pinned modern revision replaces it with the
// per-request `_meta` log level, so the modern era must never serve it.
assert_eq!(
registry.dispatch_decision("logging/setLevel", LEGACY_PROTOCOL_VERSION),
DispatchDecision::Allowed {
handler: MethodHandler::LoggingSetLevel,
kind: Kind::Method,
}
);
assert_eq!(
registry.dispatch_decision("logging/setLevel", MODERN_PROTOCOL_VERSION),
DispatchDecision::WrongEra {
handler: MethodHandler::LoggingSetLevel,
kind: Kind::Method,
},
"a method the modern schema does not declare must never reach a handler there"
);
let descriptor = METHOD_DESCRIPTORS
.iter()
.find(|descriptor| descriptor.name == "logging/setLevel")
.expect("logging/setLevel is a registered method");
assert_eq!(
descriptor.required_headers.0,
&["Mcp-Method"],
"the request declares a level, not a server-side name"
);
assert!(
CAPABILITY_DESCRIPTORS
.iter()
.filter(|descriptor| descriptor.id == CapabilityId::LoggingSetLevel)
.all(|descriptor| descriptor.advertised.is_none()),
"the client request must never be advertised as a server capability"
);
}
#[test]
fn adding_and_removing_a_method_descriptor_changes_the_dispatch_decision() {
// The inert capability owns one inert method. With the method descriptor registered the
@@ -702,9 +758,10 @@ fn adding_and_removing_a_method_descriptor_changes_the_dispatch_decision() {
json!({
"inert_test_capability": {"enabled": true},
"completions": {},
"prompts": {"listChanged": false},
"resources": {"listChanged": false, "subscribe": false},
"tools": {"listChanged": false}
"logging": {},
"prompts": {"listChanged": true},
"resources": {"listChanged": true, "subscribe": false},
"tools": {"listChanged": true}
})
);
}
@@ -0,0 +1,501 @@
//! Story 3.19: Resource and Prompt list-change signals (FR-19); Story 3.5 adds the Tools signal of
//! the same seam (FR-14).
//!
//! The positive result of the story is that a client which asked for the list-change signal of its
//! revision learns that the *committed* Resource, Prompt or Tool catalog changed, and that the
//! ordinary `resources/list`, `resources/read`, `resources/templates/list`, `prompts/list`,
//! `prompts/get` and `tools/list` stay authoritative for a client that misses the notification.
//!
//! Shape of the chosen seam (ARC decision, kanban t_c12863fb): a **stream-local watcher**. One open
//! notification stream owns one bounded periodic read of the published Agent catalog the stream is
//! scoped to; a change of the visible Resource/Prompt/Tool projection of that snapshot (the Resource
//! and Prompt bindings plus the exact Tool definitions `tools/list` serves) produces the revision's
//! list-change notification. There is no post-commit hook, no cross-process bus and no durable
//! queue: the committed catalog is the source of truth, so a signal can never be produced before its
//! commit, and a failed read can never roll the commit back — it only skips that tick and stays
//! observable in tracing.
//!
//! Bounds (issue requirement: bounded queues/frequency/bytes):
//! * one in-flight read per stream, one fixed poll interval, a read timeout and skipped ticks;
//! * at most one frame per change type per tick, so a burst coalesces by the latest observed
//! revision instead of queueing every intermediate commit;
//! * the queue is a fixed-capacity channel; a slow reader drops the newest frame instead of blocking
//! the producer or growing memory, and recovers through the authoritative list methods;
//! * every frame has a fixed shape — a closed method name plus the request id of the subscription —
//! so the byte bound follows from the id bound ([`MAX_SUBSCRIPTION_ID_BYTES`]).
//!
//! Scope boundary: this module serves list-change signals only. URI-content subscriptions
//! (`resources/subscribe` / `notifications/resources/updated`) stay unimplemented and are never
//! acknowledged or emitted here.
use std::{
collections::{HashMap, VecDeque},
future::Future,
pin::Pin,
sync::{
Arc, Mutex, MutexGuard,
atomic::{AtomicU64, Ordering},
},
time::{Duration, Instant},
};
use axum::response::sse::Event;
use crank_registry::PublishedAgentCatalog;
use futures_util::stream;
use serde::Deserialize;
use serde_json::{Map, Value, json};
use tokio::sync::mpsc;
use tracing::warn;
use crate::app::AppState;
/// The pinned key that correlates every message of one subscription stream with its
/// `subscriptions/listen` request id.
pub(crate) const SUBSCRIPTION_ID_KEY: &str = "io.modelcontextprotocol/subscriptionId";
/// `notifications/resources/list_changed` (both approved revisions).
pub(crate) const RESOURCE_LIST_CHANGED: &str = "notifications/resources/list_changed";
/// `notifications/prompts/list_changed` (both approved revisions).
pub(crate) const PROMPT_LIST_CHANGED: &str = "notifications/prompts/list_changed";
/// `notifications/tools/list_changed` (both approved revisions, Story 3.5).
pub(crate) const TOOL_LIST_CHANGED: &str = "notifications/tools/list_changed";
/// `notifications/subscriptions/acknowledged` (modern revision only).
pub(crate) const SUBSCRIPTIONS_ACKNOWLEDGED: &str = "notifications/subscriptions/acknowledged";
/// Engineering default of the watcher tick. It is not a measured latency SLO: the story only needs
/// "after commit", and the client always keeps the authoritative list methods for recovery.
pub(crate) const POLL_INTERVAL: Duration = Duration::from_secs(1);
/// Upper bound of one watcher read. A read that exceeds it is a skipped tick, exactly like a failed
/// read: no signal, no rollback, one safe tracing event.
pub(crate) const READ_TIMEOUT: Duration = Duration::from_secs(5);
/// Frames a slow reader may fall behind before the newest frame is dropped (`coalescing`).
pub(crate) const SIGNAL_QUEUE_CAPACITY: usize = 4;
/// Bound of the encoded `subscriptions/listen` id carried by every frame of one stream.
pub(crate) const MAX_SUBSCRIPTION_ID_BYTES: usize = 64;
/// Bound of the legacy session stream table (memory bound; see [`StreamRegistry`]).
pub(crate) const MAX_TRACKED_STREAMS: usize = 1_024;
/// Bound of the URIs a modern filter may name. The server does not implement URI subscriptions, so
/// this many names is already far more than a well-behaved client asks for.
pub(crate) const MAX_FILTER_URIS: usize = 32;
/// Bound of one URI in the modern filter: longer values are rejected without being echoed back.
pub(crate) const MAX_FILTER_URI_BYTES: usize = 2_048;
/// The one safe detail of a malformed or oversized `subscriptions/listen` filter.
pub(crate) const INVALID_LISTEN_FILTER: &str = "invalid subscriptions/listen notification filter";
/// The three list-change signal types this server implements.
#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)]
pub(crate) struct Signals {
pub(crate) resources: bool,
pub(crate) prompts: bool,
pub(crate) tools: bool,
}
impl Signals {
/// All types: the legacy revision defines no client opt-in, so its stream carries every type.
pub(crate) const ALL: Self = Self {
resources: true,
prompts: true,
tools: true,
};
#[cfg_attr(not(test), allow(dead_code))] // read by the unit oracle of the projection
pub(crate) fn any(self) -> bool {
self.resources || self.prompts || self.tools
}
fn intersect(self, other: Self) -> Self {
Self {
resources: self.resources && other.resources,
prompts: self.prompts && other.prompts,
tools: self.tools && other.tools,
}
}
}
/// The pinned modern `SubscriptionFilter`, as far as this server reads it.
///
/// Unknown members are ignored and a malformed shape is a deserialization error, so a client cannot
/// reach the watcher with anything but the reviewed field set.
#[derive(Debug, Default, Deserialize)]
#[serde(rename_all = "camelCase")]
pub(crate) struct ListenFilter {
#[serde(default)]
tools_list_changed: bool,
#[serde(default)]
prompts_list_changed: bool,
#[serde(default)]
resources_list_changed: bool,
#[serde(default)]
resource_subscriptions: Vec<String>,
}
/// The pinned `SubscriptionsListenRequestParams`: the notification filter is a required member.
///
/// Only the reviewed member is read; `_meta` and any other member of the request params is not part
/// of this contract.
#[derive(Debug, Deserialize)]
pub(crate) struct ListenParams {
pub(crate) notifications: ListenFilter,
}
/// The bounded input contract of a `subscriptions/listen` filter.
pub(crate) enum FilterOutcome {
/// The list-change subset this server honours for the stream.
Accepted(Signals),
/// One safe detail: no URI, name or value the client sent is echoed back.
Invalid(&'static str),
}
impl ListenFilter {
/// Validate the bounded inputs and keep only the honoured types.
///
/// `resourceSubscriptions` is not implemented in the current scope, which is what the
/// acknowledgment reports: it is simply not honoured, and the pinned schema forbids the server
/// from sending a type it did not agree to.
pub(crate) fn accepted(&self) -> FilterOutcome {
if self.resource_subscriptions.len() > MAX_FILTER_URIS
|| self
.resource_subscriptions
.iter()
.any(|uri| uri.len() > MAX_FILTER_URI_BYTES)
{
return FilterOutcome::Invalid(INVALID_LISTEN_FILTER);
}
FilterOutcome::Accepted(Signals {
resources: self.resources_list_changed,
prompts: self.prompts_list_changed,
tools: self.tools_list_changed,
})
}
}
/// The `notifications` object of the acknowledgment: exactly the subset that will be delivered.
fn acknowledged_filter_wire(accepted: Signals) -> Value {
let mut filter = Map::new();
if accepted.prompts {
filter.insert("promptsListChanged".to_owned(), Value::Bool(true));
}
if accepted.resources {
filter.insert("resourcesListChanged".to_owned(), Value::Bool(true));
}
if accepted.tools {
filter.insert("toolsListChanged".to_owned(), Value::Bool(true));
}
Value::Object(filter)
}
/// The exact bound Resource/Prompt/Tool projection of one published Agent catalog.
///
/// The projection is derived from the same committed catalog read that already feeds
/// `resources/list`, `resources/templates/list`, `prompts/list`, `prompts/get` and `tools/list`: the
/// exact Resource and Prompt bindings of the published Agent snapshot, in deterministic binding
/// order, and the exact Tool definitions that `tools/list` would serve. The Tool part is taken from
/// [`crate::manifest::catalog_tool_definitions`] rather than from the bindings, because that is the
/// projection the client sees: it also depends on the Tool selection policy (mode, groups, search
/// settings), and in `Search` mode it does not depend on the bindings at all. `catalog_revision`
/// alone is *not* the comparison key — a commit that only changes Tools must not produce a Resource
/// or Prompt signal, and vice versa — so the three projections are compared on their own.
///
/// shortcut: a Resource revocation invalidates a bound publication without a new Agent snapshot, so
/// it does not change this projection and produces no list-change signal; the revocation gate at
/// read time (Story 3.8) still fails closed, and a faithful revocation signal needs a bounded
/// registry read that the current scope does not have.
#[derive(Clone, Debug, Default, PartialEq, Eq)]
pub(crate) struct CatalogFingerprint {
resources: Vec<(String, u32)>,
prompts: Vec<(String, u32)>,
/// Canonical JSON of every definition `tools/list` serves, sorted so a re-ordered read is not a
/// false change.
tools: Vec<String>,
}
impl CatalogFingerprint {
pub(crate) fn of(catalog: &PublishedAgentCatalog) -> Self {
fn sorted<T: Ord>(values: impl Iterator<Item = T>) -> Vec<T> {
let mut sorted = values.collect::<Vec<_>>();
sorted.sort_unstable();
sorted
}
Self {
resources: sorted(catalog.resources.iter().map(|binding| {
(
binding.resource_id.as_str().to_owned(),
binding.resource_version,
)
})),
prompts: sorted(catalog.prompts.iter().map(|binding| {
(
binding.prompt_id.as_str().to_owned(),
binding.prompt_version,
)
})),
tools: sorted(
crate::manifest::catalog_tool_definitions(catalog)
.into_iter()
.map(|definition| definition.to_string()),
),
}
}
/// The list-change types whose visible projection differs between `previous` and `self`.
pub(crate) fn changes_since(&self, previous: &Self) -> Signals {
Signals {
resources: self.resources != previous.resources,
prompts: self.prompts != previous.prompts,
tools: self.tools != previous.tools,
}
}
}
/// The wire contract of one stream.
#[derive(Clone, Debug)]
pub(crate) enum Wire {
/// `2025-11-25`: a bare server notification without parameters.
Legacy,
/// `2026-07-28`: every message carries the subscription id of its `subscriptions/listen` request.
Modern(Value),
}
impl Wire {
fn signal_frames(&self, signals: Signals) -> Vec<String> {
let mut frames = Vec::with_capacity(3);
for (changed, method) in [
(signals.resources, RESOURCE_LIST_CHANGED),
(signals.prompts, PROMPT_LIST_CHANGED),
(signals.tools, TOOL_LIST_CHANGED),
] {
if !changed {
continue;
}
let frame = match self {
Wire::Legacy => json!({ "jsonrpc": "2.0", "method": method }),
Wire::Modern(subscription_id) => json!({
"jsonrpc": "2.0",
"method": method,
"params": { "_meta": { SUBSCRIPTION_ID_KEY: subscription_id } }
}),
};
frames.push(frame.to_string());
}
frames
}
}
/// The first modern frame: the accepted subset, before any list-change notification of the stream.
pub(crate) fn acknowledged_frame(subscription_id: &Value, accepted: Signals) -> String {
json!({
"jsonrpc": "2.0",
"method": SUBSCRIPTIONS_ACKNOWLEDGED,
"params": {
"_meta": { SUBSCRIPTION_ID_KEY: subscription_id },
"notifications": acknowledged_filter_wire(accepted),
}
})
.to_string()
}
/// The subscription id of a `subscriptions/listen` request, or `None` when its encoding is not a
/// bounded identity carrier.
pub(crate) fn subscription_id(request_id: &Value) -> Option<Value> {
let encoded = request_id.to_string();
(encoded.len() <= MAX_SUBSCRIPTION_ID_BYTES).then(|| request_id.clone())
}
/// Bounded table of the legacy session that currently owns the notification stream.
///
/// The pinned legacy transport forbids broadcasting one message across several SSE streams of the
/// same session, and the community scope keeps exactly one producer per session: the newest admitted
/// `GET` takes the signals over and the older stream ends. A retired stream (session `DELETE`) or a
/// superseded generation never writes into a reconnect.
#[derive(Default)]
pub(crate) struct StreamRegistry {
next_generation: AtomicU64,
owners: Mutex<HashMap<String, u64>>,
}
impl StreamRegistry {
pub(crate) fn new() -> Self {
Self::default()
}
/// Admit the notification stream of one session and return its generation.
pub(crate) fn admit(&self, session_id: &str) -> Option<u64> {
let generation = self.next_generation.fetch_add(1, Ordering::Relaxed) + 1;
let mut owners = self.lock()?;
if owners.len() >= MAX_TRACKED_STREAMS && !owners.contains_key(session_id) {
// Bounded table: forget the whole owner set instead of growing without a limit. Each
// running producer still holds its own generation, so the worst case is a stream that
// can no longer be told apart from a newer one — never unbounded memory.
owners.clear();
}
owners.insert(session_id.to_owned(), generation);
Some(generation)
}
/// Whether `generation` still owns the stream of `session_id`.
pub(crate) fn is_current(&self, session_id: &str, generation: u64) -> bool {
self.lock()
.and_then(|owners| owners.get(session_id).copied())
.is_some_and(|current| current == generation)
}
/// Stop the producer of one session (`DELETE`, or a session that no longer exists).
pub(crate) fn retire(&self, session_id: &str) {
if let Some(mut owners) = self.lock() {
owners.remove(session_id);
}
}
fn lock(&self) -> Option<MutexGuard<'_, HashMap<String, u64>>> {
self.owners.try_lock().ok()
}
}
/// Everything one watcher needs.
pub(crate) struct Watch {
pub(crate) workspace_slug: String,
pub(crate) agent_slug: String,
/// The signal types this stream may deliver.
pub(crate) signals: Signals,
pub(crate) wire: Wire,
/// The legacy session lease: the producer stops when the stream is superseded or retired.
pub(crate) legacy_stream: Option<(Arc<StreamRegistry>, String, u64)>,
/// Lifetime bound inherited from the legacy transport session TTL.
pub(crate) deadline: Option<Instant>,
}
/// One committed-catalog read of a watcher tick.
///
/// The reader is a closure so the failure path is driveable in a unit oracle without a live
/// registry: production passes [`read`] over the shared [`AppState`], and the oracle passes a
/// scripted sequence.
pub(crate) type ReadFuture =
Pin<Box<dyn Future<Output = Result<CatalogFingerprint, ReadError>> + Send>>;
/// A source of committed-catalog reads. One call per tick.
pub(crate) type Reader = Box<dyn FnMut() -> ReadFuture + Send>;
/// Start the stream-local watcher and return the bounded frame queue of the response stream.
///
/// The first read is the baseline taken **before** the acknowledgment is written; it is never reset
/// afterwards, so a publication between the baseline and the acknowledgment is still delivered.
pub(crate) fn spawn(state: Arc<AppState>, watch: Watch) -> mpsc::Receiver<String> {
let scope = (watch.workspace_slug.clone(), watch.agent_slug.clone());
let reader: Reader = Box::new(move || {
let state = Arc::clone(&state);
let scope = scope.clone();
Box::pin(async move { read(&state, &scope).await })
});
let (sender, receiver) = mpsc::channel(SIGNAL_QUEUE_CAPACITY);
tokio::spawn(run(watch, sender, reader));
receiver
}
async fn run(watch: Watch, sender: mpsc::Sender<String>, mut read: Reader) {
let mut baseline = match read().await {
Ok(fingerprint) => Some(fingerprint),
Err(ReadError::Gone) => return,
Err(ReadError::Retry) => {
observe_failure();
None
}
};
loop {
if !still_owned(&watch) || sender.is_closed() {
return;
}
if watch
.deadline
.is_some_and(|deadline| Instant::now() >= deadline)
{
return;
}
tokio::time::sleep(POLL_INTERVAL).await;
if !still_owned(&watch) {
return;
}
match read().await {
Ok(fingerprint) => {
if let Some(previous) = baseline.as_ref() {
let changes = fingerprint.changes_since(previous).intersect(watch.signals);
for frame in watch.wire.signal_frames(changes) {
// A slow reader coalesces: the newest frame is dropped instead of blocking
// the producer or growing a queue, and the client recovers through the
// authoritative list methods.
let _ = sender.try_send(frame);
}
}
baseline = Some(fingerprint);
}
// The Agent is no longer published: fail closed by ending the stream instead of serving a
// catalog nobody may read any more.
Err(ReadError::Gone) => return,
// The committed catalog is untouched: the failure is observable and this tick is skipped.
Err(_) => observe_failure(),
}
}
}
pub(crate) enum ReadError {
/// The published Agent behind this stream is gone: the producer ends.
Gone,
/// The catalog could not be read this tick: skip it, keep the committed state.
Retry,
}
fn still_owned(watch: &Watch) -> bool {
match &watch.legacy_stream {
Some((registry, session_id, generation)) => registry.is_current(session_id, *generation),
None => true,
}
}
async fn read(state: &AppState, scope: &(String, String)) -> Result<CatalogFingerprint, ReadError> {
let catalog = tokio::time::timeout(READ_TIMEOUT, state.catalog.get_catalog(&scope.0, &scope.1))
.await
.map_err(|_| ReadError::Retry)?
.map_err(|error| match error {
crank_registry::RegistryError::PublishedAgentNotFound { .. } => ReadError::Gone,
_ => ReadError::Retry,
})?;
Ok(CatalogFingerprint::of(&catalog))
}
/// One safe, closed-vocabulary observation of a skipped tick: no URI, Agent name, credential or raw
/// error text is read here.
fn observe_failure() {
warn!(
name: "mcp.signals.read_failed",
error_category = "catalog",
"catalog change signal read failed"
);
}
/// The bounded SSE body of one signal stream: the leading frames (acknowledgment), then the queue,
/// ending when the producer ends. The caller's stream permit lives in `guard` and is released when
/// the response stream is dropped.
pub(crate) fn frames_stream<G: Send + 'static>(
first: Vec<String>,
receiver: mpsc::Receiver<String>,
guard: G,
) -> impl stream::Stream<Item = Result<Event, std::convert::Infallible>> {
let pending: VecDeque<String> = first.into();
stream::unfold(
(pending, receiver, guard),
|(mut pending, mut receiver, guard)| async move {
loop {
if let Some(frame) = pending.pop_front() {
return Some((Ok(Event::default().data(frame)), (pending, receiver, guard)));
}
let frame = receiver.recv().await?;
pending.push_back(frame);
}
},
)
}
#[cfg(test)]
#[path = "signals_tests.rs"]
mod tests;
@@ -0,0 +1,741 @@
//! Unit oracles of the Story 3.19 signal seam: the catalog projection, the bounded filter, the exact
//! wire frames of both approved revisions and the bounded legacy stream ownership.
use std::{
collections::BTreeMap,
io,
sync::{Arc, Mutex},
time::{Duration, Instant},
};
use crank_core::ids::{PromptId, ResourceId};
use crank_core::{
ExecutionConfig, HttpMethod, Operation, OperationId, OperationSecurityLevel, OperationStatus,
Protocol, RestTarget, Target, ToolAccessMode, ToolDescription, ToolGroup, ToolSearchSettings,
ToolSelectionPolicy, WorkspaceId,
};
use crank_mapping::MappingSet;
use crank_registry::PublishedAgentCatalog;
use crank_registry::model::{AgentPromptBinding, AgentResourceBinding, PublishedAgentTool};
use crank_schema::{Schema, SchemaKind};
use serde_json::{Value, json};
use time::OffsetDateTime;
use tokio::sync::mpsc;
use super::{
CatalogFingerprint, FilterOutcome, INVALID_LISTEN_FILTER, ListenFilter, MAX_FILTER_URIS,
MAX_SUBSCRIPTION_ID_BYTES, PROMPT_LIST_CHANGED, RESOURCE_LIST_CHANGED, ReadError, Reader,
SIGNAL_QUEUE_CAPACITY, SUBSCRIPTION_ID_KEY, SUBSCRIPTIONS_ACKNOWLEDGED, Signals,
StreamRegistry, TOOL_LIST_CHANGED, Watch, Wire, acknowledged_frame, run, subscription_id,
};
fn catalog(
resources: &[(&str, u32)],
prompts: &[(&str, u32)],
catalog_revision: &str,
) -> PublishedAgentCatalog {
catalog_with(
ToolSelectionPolicy::default(),
Vec::new(),
resources,
prompts,
catalog_revision,
)
}
fn catalog_with(
tool_selection_policy: ToolSelectionPolicy,
tools: Vec<PublishedAgentTool>,
resources: &[(&str, u32)],
prompts: &[(&str, u32)],
catalog_revision: &str,
) -> PublishedAgentCatalog {
PublishedAgentCatalog {
agent_version: 1,
catalog_revision: catalog_revision.to_owned(),
tool_selection_policy,
tools,
resources: resources
.iter()
.map(|(id, version)| AgentResourceBinding {
resource_id: ResourceId::new(*id),
resource_version: *version,
})
.collect(),
prompts: prompts
.iter()
.map(|(id, version)| AgentPromptBinding {
prompt_id: PromptId::new(*id),
prompt_version: *version,
})
.collect(),
}
}
fn search_policy(tool_names: &[&str]) -> ToolSelectionPolicy {
ToolSelectionPolicy {
mode: ToolAccessMode::Search,
groups: vec![ToolGroup {
id: "crm".to_owned(),
name: "CRM".to_owned(),
description: "CRM operations".to_owned(),
tool_names: tool_names.iter().map(|name| (*name).to_owned()).collect(),
}],
search: ToolSearchSettings::default(),
}
}
fn empty_object_schema() -> Schema {
Schema {
kind: SchemaKind::Object,
format: None,
description: None,
required: false,
nullable: false,
default_value: None,
fields: BTreeMap::new(),
items: None,
enum_values: Vec::new(),
variants: Vec::new(),
}
}
/// One published Tool binding whose served definition is exactly `title`/`description`.
fn published_tool(name: &str, title: &str, description: &str, version: u32) -> PublishedAgentTool {
PublishedAgentTool {
workspace_id: WorkspaceId::new("ws_test"),
workspace_slug: "ws-test".to_owned(),
agent_id: "agent_test".into(),
agent_slug: "agent".to_owned(),
operation: Operation {
id: OperationId::new(format!("op_{name}")),
name: name.to_owned(),
display_name: title.to_owned(),
category: "test".to_owned(),
protocol: Protocol::Rest,
security_level: OperationSecurityLevel::Standard,
status: OperationStatus::Published,
version,
target: Target::Rest(RestTarget {
base_url: "https://example.test".to_owned(),
method: HttpMethod::Get,
path_template: "/leads".to_owned(),
static_headers: BTreeMap::new(),
request_contract: None,
response_contract: Vec::new(),
multipart_field_mime_types: BTreeMap::new(),
}),
input_schema: empty_object_schema(),
output_schema: empty_object_schema(),
input_mapping: MappingSet { rules: Vec::new() },
output_mapping: MappingSet { rules: Vec::new() },
execution_config: ExecutionConfig {
timeout_ms: 10_000,
retry_policy: None,
response_cache: None,
idempotency: None,
safety: None,
approval_policy: None,
execution_policy: None,
auth_profile_ref: None,
headers: BTreeMap::new(),
},
tool_description: ToolDescription {
title: title.to_owned(),
description: description.to_owned(),
tags: Vec::new(),
examples: Vec::new(),
},
samples: None,
generated_draft: None,
config_export: None,
wizard_state: None,
created_at: OffsetDateTime::UNIX_EPOCH,
updated_at: OffsetDateTime::UNIX_EPOCH,
published_at: None,
},
tool_name: name.to_owned(),
tool_title: title.to_owned(),
tool_description: description.to_owned(),
}
}
#[test]
fn a_committed_resource_binding_is_a_resource_change_only() {
let before = CatalogFingerprint::of(&catalog(&[("res_a", 1)], &[("prm_a", 1)], "rev-1"));
let after = CatalogFingerprint::of(&catalog(
&[("res_a", 1), ("res_b", 1)],
&[("prm_a", 1)],
"rev-2",
));
assert_eq!(
after.changes_since(&before),
Signals {
resources: true,
prompts: false,
tools: false
}
);
assert!(!before.changes_since(&after).prompts);
assert_eq!(
before.changes_since(&after),
Signals {
resources: true,
prompts: false,
tools: false
}
);
}
#[test]
fn a_committed_prompt_binding_is_a_prompt_change_only() {
let before = CatalogFingerprint::of(&catalog(&[], &[("prm_a", 1)], "rev-1"));
let after = CatalogFingerprint::of(&catalog(&[], &[("prm_a", 1), ("prm_b", 2)], "rev-2"));
assert_eq!(
after.changes_since(&before),
Signals {
resources: false,
prompts: true,
tools: false
}
);
}
#[test]
fn a_pinned_version_bump_is_a_change_even_when_the_binding_names_are_stable() {
let before = CatalogFingerprint::of(&catalog(&[("res_a", 1)], &[("prm_a", 1)], "rev-1"));
let after = CatalogFingerprint::of(&catalog(&[("res_a", 2)], &[("prm_a", 1)], "rev-1"));
assert_eq!(
after.changes_since(&before),
Signals {
resources: true,
prompts: false,
tools: false
}
);
}
#[test]
fn a_committed_tool_definition_change_is_a_tool_change_only() {
let lead = || published_tool("crm_create_lead", "Create Lead", "Creates a lead", 1);
let before = CatalogFingerprint::of(&catalog_with(
ToolSelectionPolicy::default(),
vec![lead()],
&[("res_a", 1)],
&[("prm_a", 1)],
"rev-1",
));
let added = CatalogFingerprint::of(&catalog_with(
ToolSelectionPolicy::default(),
vec![
lead(),
published_tool(
"crm_create_second",
"Create Second",
"Creates a second lead",
1,
),
],
&[("res_a", 1)],
&[("prm_a", 1)],
"rev-2",
));
let retitled = CatalogFingerprint::of(&catalog_with(
ToolSelectionPolicy::default(),
vec![published_tool(
"crm_create_lead",
"Create a lead",
"Creates a lead",
1,
)],
&[("res_a", 1)],
&[("prm_a", 1)],
"rev-2",
));
let tools_only = Signals {
resources: false,
prompts: false,
tools: true,
};
assert_eq!(added.changes_since(&before), tools_only);
assert_eq!(
retitled.changes_since(&before),
tools_only,
"a retitled binding changes the visible Tool projection without a new Operation version"
);
// The reverse direction is a Tool change as well: a removed binding is a real change.
assert!(before.changes_since(&added).tools);
}
#[test]
fn a_republished_operation_version_with_an_identical_definition_is_silent() {
// `tools/list` serves the definition, not the Operation version: a version bump that leaves the
// served definition unchanged is not a client-visible Tool catalog change.
let before = CatalogFingerprint::of(&catalog_with(
ToolSelectionPolicy::default(),
vec![published_tool(
"crm_create_lead",
"Create Lead",
"Creates a lead",
1,
)],
&[("res_a", 1)],
&[("prm_a", 1)],
"rev-1",
));
let after = CatalogFingerprint::of(&catalog_with(
ToolSelectionPolicy::default(),
vec![published_tool(
"crm_create_lead",
"Create Lead",
"Creates a lead",
2,
)],
&[("res_a", 1)],
&[("prm_a", 1)],
"rev-2",
));
assert!(!after.changes_since(&before).any());
}
#[test]
fn a_policy_only_commit_that_changes_tools_list_is_a_tool_change_only() {
// The Tool selection policy is part of the served `tools/list` projection: switching the same
// bindings from direct to search mode replaces the served tools with `search_tools`/`call_tool`.
let tool = published_tool("crm_create_lead", "Create Lead", "Creates a lead", 1);
let direct = CatalogFingerprint::of(&catalog_with(
ToolSelectionPolicy::default(),
vec![tool.clone()],
&[("res_a", 1)],
&[("prm_a", 1)],
"rev-1",
));
let search = CatalogFingerprint::of(&catalog_with(
search_policy(&["crm_create_lead"]),
vec![tool],
&[("res_a", 1)],
&[("prm_a", 1)],
"rev-2",
));
let tools_only = Signals {
resources: false,
prompts: false,
tools: true,
};
assert_eq!(search.changes_since(&direct), tools_only);
assert_eq!(direct.changes_since(&search), tools_only);
}
#[test]
fn a_policy_field_invisible_to_tools_list_is_silent() {
// `search.max_results` is only served inside the search-mode definitions: in direct mode it
// changes nothing the client receives.
let tool = || published_tool("crm_create_lead", "Create Lead", "Creates a lead", 1);
let before = CatalogFingerprint::of(&catalog_with(
ToolSelectionPolicy::default(),
vec![tool()],
&[],
&[],
"rev-1",
));
let after = CatalogFingerprint::of(&catalog_with(
ToolSelectionPolicy {
search: ToolSearchSettings { max_results: 5 },
..ToolSelectionPolicy::default()
},
vec![tool()],
&[],
&[],
"rev-2",
));
assert!(!after.changes_since(&before).any());
}
#[test]
fn a_search_mode_binding_change_is_silent_when_tools_list_is_unchanged() {
// In search mode the served list is `search_tools`/`call_tool`; the bindings behind it are not
// part of `tools/list`, so a binding-only commit must not claim a visible Tool change.
let policy = search_policy(&["crm_create_lead"]);
let before = CatalogFingerprint::of(&catalog_with(
policy.clone(),
vec![published_tool(
"crm_create_lead",
"Create Lead",
"Creates a lead",
1,
)],
&[],
&[],
"rev-1",
));
let after = CatalogFingerprint::of(&catalog_with(
policy,
vec![
published_tool("crm_create_lead", "Create Lead", "Creates a lead", 1),
published_tool(
"crm_create_second",
"Create Second",
"Creates a second lead",
1,
),
],
&[],
&[],
"rev-2",
));
assert!(!after.changes_since(&before).any());
}
#[test]
fn a_resource_or_prompt_change_is_not_a_tool_change() {
let tool = || published_tool("crm_create_lead", "Create Lead", "Creates a lead", 1);
let before = CatalogFingerprint::of(&catalog_with(
ToolSelectionPolicy::default(),
vec![tool()],
&[("res_a", 1)],
&[("prm_a", 1)],
"rev-1",
));
let changed = CatalogFingerprint::of(&catalog_with(
ToolSelectionPolicy::default(),
vec![tool()],
&[("res_a", 2)],
&[("prm_a", 1), ("prm_b", 1)],
"rev-1",
));
assert_eq!(
changed.changes_since(&before),
Signals {
resources: true,
prompts: true,
tools: false
}
);
}
#[test]
fn a_catalog_revision_bump_without_a_visible_change_is_silent() {
// `catalog_revision` moved, but no visible Resource/Prompt/Tool projection did: no signal.
let before = CatalogFingerprint::of(&catalog(&[("res_a", 1)], &[("prm_a", 1)], "rev-1"));
let after = CatalogFingerprint::of(&catalog(&[("res_a", 1)], &[("prm_a", 1)], "rev-2"));
assert!(!after.changes_since(&before).any());
}
#[test]
fn the_projection_is_order_independent_and_has_no_duplicate_growth() {
let forward = CatalogFingerprint::of(&catalog(
&[("res_a", 1), ("res_b", 2)],
&[("prm_a", 1), ("prm_b", 2)],
"rev-1",
));
let reversed = CatalogFingerprint::of(&catalog(
&[("res_b", 2), ("res_a", 1)],
&[("prm_b", 2), ("prm_a", 1)],
"rev-1",
));
assert!(!reversed.changes_since(&forward).any());
}
#[test]
fn the_legacy_frames_carry_a_closed_method_name_and_no_parameters() {
let frames = Wire::Legacy.signal_frames(Signals::ALL);
assert_eq!(
frames,
vec![
json!({"jsonrpc": "2.0", "method": RESOURCE_LIST_CHANGED}).to_string(),
json!({"jsonrpc": "2.0", "method": PROMPT_LIST_CHANGED}).to_string(),
json!({"jsonrpc": "2.0", "method": TOOL_LIST_CHANGED}).to_string(),
]
);
// No URI, name or payload of a changed object is part of a list-change notification.
assert!(frames.iter().all(|frame| !frame.contains("uri")));
}
#[test]
fn the_modern_frames_carry_the_subscription_id_of_their_listen_request() {
let id = json!(41);
let frames = Wire::Modern(id.clone()).signal_frames(Signals::ALL);
assert_eq!(
frames,
vec![
json!({
"jsonrpc": "2.0",
"method": RESOURCE_LIST_CHANGED,
"params": { "_meta": { SUBSCRIPTION_ID_KEY: 41 } }
})
.to_string(),
json!({
"jsonrpc": "2.0",
"method": PROMPT_LIST_CHANGED,
"params": { "_meta": { SUBSCRIPTION_ID_KEY: 41 } }
})
.to_string(),
json!({
"jsonrpc": "2.0",
"method": TOOL_LIST_CHANGED,
"params": { "_meta": { SUBSCRIPTION_ID_KEY: 41 } }
})
.to_string(),
]
);
assert_eq!(
serde_json::from_str::<Value>(&frames[0]).unwrap()["params"]["_meta"][SUBSCRIPTION_ID_KEY],
id
);
}
#[test]
fn only_the_requested_and_honoured_types_are_encoded() {
let frames = Wire::Modern(json!(1)).signal_frames(Signals {
resources: false,
prompts: true,
tools: false,
});
assert_eq!(frames.len(), 1);
assert!(frames[0].contains(PROMPT_LIST_CHANGED));
}
#[test]
fn the_acknowledgment_reports_exactly_the_honoured_subset() {
let frame: Value = serde_json::from_str(&acknowledged_frame(
&json!(7),
Signals {
resources: true,
prompts: false,
tools: true,
},
))
.unwrap();
assert_eq!(frame["method"], SUBSCRIPTIONS_ACKNOWLEDGED);
assert_eq!(frame["params"]["_meta"][SUBSCRIPTION_ID_KEY], json!(7));
assert_eq!(
frame["params"]["notifications"],
json!({ "resourcesListChanged": true, "toolsListChanged": true })
);
}
#[test]
fn an_empty_honoured_subset_is_an_empty_filter_object() {
let frame: Value =
serde_json::from_str(&acknowledged_frame(&json!("sub-1"), Signals::default())).unwrap();
assert_eq!(frame["params"]["notifications"], json!({}));
}
#[test]
fn the_filter_keeps_only_the_implemented_list_change_types() {
let filter: ListenFilter = serde_json::from_value(json!({
"toolsListChanged": true,
"promptsListChanged": true,
"resourcesListChanged": false,
"resourceSubscriptions": ["file:///project/config.json"]
}))
.unwrap();
match filter.accepted() {
FilterOutcome::Accepted(signals) => assert_eq!(
signals,
Signals {
resources: false,
prompts: true,
tools: true
},
"the honoured set is the requested list-change subset: {signals:?}"
),
FilterOutcome::Invalid(detail) => panic!("a bounded filter must be accepted: {detail}"),
}
}
#[test]
fn an_absent_filter_field_means_not_subscribed() {
let filter: ListenFilter = serde_json::from_value(json!({})).unwrap();
match filter.accepted() {
FilterOutcome::Accepted(signals) => assert!(!signals.any()),
FilterOutcome::Invalid(detail) => panic!("an empty filter must be accepted: {detail}"),
}
}
#[test]
fn an_oversized_uri_filter_is_refused_without_echoing_it() {
let filter: ListenFilter = serde_json::from_value(json!({
"resourceSubscriptions": (0..=MAX_FILTER_URIS)
.map(|index| format!("file:///resource-{index}"))
.collect::<Vec<_>>()
}))
.unwrap();
match filter.accepted() {
FilterOutcome::Invalid(detail) => assert_eq!(detail, INVALID_LISTEN_FILTER),
FilterOutcome::Accepted(_) => panic!("an oversized filter must be refused"),
}
let oversized: ListenFilter = serde_json::from_value(json!({
"resourceSubscriptions": ["u".repeat(super::MAX_FILTER_URI_BYTES + 1)]
}))
.unwrap();
assert!(matches!(
oversized.accepted(),
FilterOutcome::Invalid(INVALID_LISTEN_FILTER)
));
}
#[test]
fn a_non_object_filter_is_a_deserialization_error() {
assert!(serde_json::from_value::<ListenFilter>(json!(["toolsListChanged"])).is_err());
assert!(serde_json::from_value::<ListenFilter>(json!({"promptsListChanged": "yes"})).is_err());
}
#[test]
fn the_subscription_id_is_bound_by_its_encoded_size() {
assert_eq!(subscription_id(&json!(1)), Some(json!(1)));
assert_eq!(subscription_id(&json!("sub-1")), Some(json!("sub-1")));
assert_eq!(
subscription_id(&json!("x".repeat(MAX_SUBSCRIPTION_ID_BYTES))),
None,
"an id longer than the frame bound must not be used as a subscription id"
);
}
#[test]
fn a_newer_legacy_stream_supersedes_the_previous_one() {
let streams = StreamRegistry::new();
let first = streams.admit("session-1").unwrap();
let second = streams.admit("session-1").unwrap();
assert!(!streams.is_current("session-1", first));
assert!(streams.is_current("session-1", second));
// Another session is unaffected.
let other = streams.admit("session-2").unwrap();
assert!(streams.is_current("session-2", other));
streams.retire("session-1");
assert!(!streams.is_current("session-1", second));
assert!(streams.is_current("session-2", other));
}
#[derive(Clone, Default)]
struct SharedLogWriter {
buffer: Arc<Mutex<Vec<u8>>>,
}
impl SharedLogWriter {
fn output(&self) -> String {
String::from_utf8(self.buffer.lock().unwrap().clone()).unwrap()
}
}
impl<'a> tracing_subscriber::fmt::MakeWriter<'a> for SharedLogWriter {
type Writer = SharedLogGuard;
fn make_writer(&'a self) -> Self::Writer {
SharedLogGuard {
buffer: Arc::clone(&self.buffer),
}
}
}
struct SharedLogGuard {
buffer: Arc<Mutex<Vec<u8>>>,
}
impl io::Write for SharedLogGuard {
fn write(&mut self, bytes: &[u8]) -> io::Result<usize> {
self.buffer.lock().unwrap().extend_from_slice(bytes);
Ok(bytes.len())
}
fn flush(&mut self) -> io::Result<()> {
Ok(())
}
}
/// AC-2: a failed committed-catalog read is observable and loses nothing — the tick is skipped (no
/// fabricated signal, no write path that could roll the catalog back) while the baseline survives, so
/// the next successful read still reports the change of the committed catalog.
#[tokio::test]
async fn a_failed_catalog_read_is_observable_and_loses_no_committed_change() {
let writer = SharedLogWriter::default();
let subscriber = crank_observability::build_subscriber(
crank_observability::ObservabilityConfig::new(
crank_observability::ServiceIdentity::try_new("mcp-server", "test", "test").unwrap(),
"info",
crank_observability::RedactionLimits::default(),
),
writer.clone(),
)
.unwrap();
let dispatch = tracing::Dispatch::new(subscriber);
let _guard = tracing::dispatcher::set_default(&dispatch);
let baseline = CatalogFingerprint::of(&catalog(&[], &[("prm_a", 1)], "rev-1"));
let committed = CatalogFingerprint::of(&catalog(&[], &[("prm_a", 1), ("prm_b", 1)], "rev-2"));
let script = Arc::new(Mutex::new(
vec![Ok(baseline), Err(ReadError::Retry), Ok(committed)].into_iter(),
));
let reader: Reader = Box::new(move || {
let next = script
.lock()
.unwrap()
.next()
.unwrap_or(Err(ReadError::Retry));
Box::pin(async move { next })
});
let (sender, mut receiver) = mpsc::channel(SIGNAL_QUEUE_CAPACITY);
tokio::spawn(run(
Watch {
workspace_slug: "workspace-canary".to_owned(),
agent_slug: "agent-canary".to_owned(),
signals: Signals::ALL,
wire: Wire::Legacy,
legacy_stream: None,
deadline: Some(Instant::now() + Duration::from_millis(2_100)),
},
sender,
reader,
));
let mut frames = Vec::new();
while let Some(frame) = receiver.recv().await {
frames.push(frame);
}
// The failed tick produced no frame; the committed change of the next successful read did.
assert_eq!(
frames,
vec![json!({"jsonrpc": "2.0", "method": PROMPT_LIST_CHANGED}).to_string()]
);
let output = writer.output();
let failures: Vec<Value> = output
.lines()
.filter_map(|line| serde_json::from_str::<Value>(line).ok())
.filter(|event| event["event"] == "mcp.signals.read_failed")
.collect();
assert!(
!failures.is_empty(),
"a skipped tick must be observable: {output}"
);
for event in &failures {
assert_eq!(event["fields"]["error_category"], "catalog");
}
// Closed vocabulary: no workspace, Agent, URI or raw database text is read into the event.
assert!(!output.contains("workspace-canary"));
assert!(!output.contains("agent-canary"));
}
+59 -1
View File
@@ -116,6 +116,19 @@ fn accepted_media_types(headers: &HeaderMap) -> Option<Vec<String>> {
)
}
/// Whether this request offered `text/event-stream`.
///
/// Story 3.18: the SSE response stream is the only carrier of a server `notifications/message`, so
/// the logging projection asks this before it promises a notification. It is the same admission rule
/// the response-mode negotiation already applies, not a second interpretation of the header.
pub(super) fn accepts_event_stream(headers: &HeaderMap) -> bool {
accepted_media_types(headers).is_some_and(|media_types| {
media_types
.iter()
.any(|media_type| media_type == "text/event-stream")
})
}
pub(super) fn negotiate_post_response_mode(
headers: &HeaderMap,
) -> Result<ResponseMode, StatusCode> {
@@ -266,8 +279,26 @@ pub(super) fn transport_response(
protocol_version: Option<&str>,
) -> Response {
let outcome = payload_outcome(&payload);
// Story 3.18: this is the single projection point of the adapter's own safe structured event.
// The severity is derived from the already-decided outcome of the payload, so no request payload,
// credential, error cause or user value is read here.
note_logging_outcome(&payload);
let notifications = crate::protocol::take_notifications();
// A pending `notifications/message` has no carrier in a JSON body. When the client offered both
// media types and opted in to logging, the negotiated mode is the SSE stream, so the notification
// precedes the response instead of being silently dropped.
let response_mode = if notifications.is_empty() {
response_mode
} else {
ResponseMode::Sse
};
if status == StatusCode::OK && matches!(response_mode, ResponseMode::Sse) {
let stream = stream::iter([Ok(Event::default().data(payload.to_string()))]);
let mut messages = Vec::with_capacity(notifications.len() + 1);
for notification in notifications {
messages.push(Ok(Event::default().data(notification.to_string())));
}
messages.push(Ok(Event::default().data(payload.to_string())));
let stream = stream::iter(messages);
let mut response = sse_response(status, stream, session_id, protocol_version);
if let Some(outcome) = outcome {
@@ -279,6 +310,33 @@ pub(super) fn transport_response(
json_response(status, payload, session_id, protocol_version)
}
/// Record the closed safe event of one response: served, rejected as invalid, failed, or a tool-level
/// error. Every label is a fixed string of the closed vocabulary; nothing from the payload or the
/// request is copied into it.
pub(super) fn note_logging_outcome(payload: &Value) {
use crate::protocol::{EVENT_FAILED, EVENT_REJECTED, EVENT_SERVED, LogLevel, note};
if payload.get("error").is_some() {
match payload.pointer("/error/code").and_then(Value::as_i64) {
// The pinned client-error codes: a malformed request, a method that this era does not
// implement, and invalid params. The client can correct all three, so they are a
// rejection rather than a server failure; the closed label stays the one client-error
// label of the vocabulary instead of growing a new one.
Some(-32602..=-32600) => note(LogLevel::Warning, EVENT_REJECTED, "invalid_params"),
_ => note(LogLevel::Error, EVENT_FAILED, "internal"),
}
return;
}
if payload
.pointer("/result/isError")
.and_then(Value::as_bool)
.unwrap_or(false)
{
note(LogLevel::Warning, EVENT_REJECTED, "tool_error");
return;
}
note(LogLevel::Info, EVENT_SERVED, "ok");
}
fn payload_outcome(payload: &Value) -> Option<McpOutcome> {
if payload.get("error").is_some() {
return Some(McpOutcome::JsonRpcError);
@@ -284,6 +284,7 @@ fn operation() -> RegistryOperation {
idempotency: None,
safety: None,
approval_policy: None,
execution_policy: None,
auth_profile_ref: None,
headers: BTreeMap::new(),
},
+531
View File
@@ -0,0 +1,531 @@
//! Typed execution-admission seam (Story 4.1, FR-21/FR-24).
//!
//! Admission is the single gate in front of every real Tool execution: it decides whether the
//! caller receives a scoped permit or a durable enqueue decision, and it owns the deployment-wide
//! global/per-Agent/per-Tool capacity and the bounded waiting queue. The authoritative state lives
//! in PostgreSQL (`crank-registry`), so several processes share one budget.
//!
//! Deliberately absent from this contract: credentials, connection or protocol revision (they must
//! never widen a quota), request bodies and prepared headers (payload ownership is Story 4.2).
use async_trait::async_trait;
use serde::{Deserialize, Serialize};
use crate::{NoTasksFallback, ToolExecutionMode};
pub const ADMISSION_FINGERPRINT_MAX_BYTES: usize = 128;
pub const ADMISSION_OWNER_TOKEN_MAX_BYTES: usize = 64;
pub const MAX_RETRY_AFTER_MS: u64 = 60_000;
/// Bounded retry guidance for an admission refusal (capacity or queue wait exhausted). One value for
/// the store and every runtime so the client contract stays stable.
pub const ADMISSION_OVERLOAD_RETRY_AFTER_MS: u64 = 250;
/// Current wall clock in unix milliseconds (absolute deadlines are persisted as unix ms).
pub fn current_unix_ms() -> i64 {
i64::try_from(
std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.map(|elapsed| elapsed.as_millis())
.unwrap_or_default(),
)
.unwrap_or(i64::MAX)
}
/// Bucket used for an execution that has no Agent (admin draft/test-run): an explicit admin origin
/// bucket that is still bounded by the global limit.
pub const ADMIN_ORIGIN_KEY: &str = "admin";
/// Exact execution identity pinned at admission time.
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
pub struct AdmissionScope {
pub workspace_id: String,
/// Agent id, or [`ADMIN_ORIGIN_KEY`] for an Agent-less administrative execution.
pub agent_key: String,
pub operation_id: String,
/// Quota keys exclude the version; the version is pinned for execution semantics only.
pub operation_version: u32,
}
impl AdmissionScope {
/// Quota key of the Agent bucket: `(workspace, agent)`.
pub fn agent_bucket(&self) -> String {
format!("{}:{}", self.workspace_id, self.agent_key)
}
/// Quota key of the Tool bucket: `(workspace, operation)` without version.
pub fn tool_bucket(&self) -> String {
format!("{}:{}", self.workspace_id, self.operation_id)
}
}
#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize, Deserialize)]
#[serde(rename_all = "snake_case")]
pub enum AdmissionMode {
/// The caller waits for a scoped permit and dispatches inside it.
Synchronous,
/// The store commits a durable enqueue decision and reserves queue capacity.
Background,
}
impl AdmissionMode {
pub const fn as_str(self) -> &'static str {
match self {
Self::Synchronous => "synchronous",
Self::Background => "background",
}
}
}
/// Non-consuming approval evidence gathered by the runtime before admission.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum ApprovalEvidence {
/// The operation does not require confirmation.
NotRequired,
/// A verified approval exists for this exact scope and input.
Granted,
/// Confirmation is required and no verified approval exists yet.
NotGranted,
}
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct AdmissionRequest {
pub scope: AdmissionScope,
pub mode: AdmissionMode,
/// Published policy mode that produced this request.
pub policy_mode: ToolExecutionMode,
pub no_tasks_fallback: NoTasksFallback,
pub approval: ApprovalEvidence,
/// Bounded request identity used only to correlate one admission entry.
pub fingerprint: String,
/// Absolute deadline (unix milliseconds), already including queue waiting time.
pub expires_at_unix_ms: i64,
}
impl AdmissionRequest {
pub fn validate(&self) -> Result<(), AdmissionStoreError> {
if self.fingerprint.is_empty()
|| self.fingerprint.len() > ADMISSION_FINGERPRINT_MAX_BYTES
|| self
.fingerprint
.bytes()
.any(|byte| byte.is_ascii_control() || byte == b' ')
{
return Err(AdmissionStoreError::invalid(
"admission fingerprint is outside its allowed bounds",
));
}
if self.scope.workspace_id.is_empty() || self.scope.workspace_id.len() > 160 {
return Err(AdmissionStoreError::invalid(
"admission workspace is invalid",
));
}
if self.scope.agent_key.is_empty() || self.scope.agent_key.len() > 160 {
return Err(AdmissionStoreError::invalid("admission agent is invalid"));
}
if self.scope.operation_id.is_empty() || self.scope.operation_id.len() > 160 {
return Err(AdmissionStoreError::invalid(
"admission operation is invalid",
));
}
if self.scope.operation_version == 0 {
return Err(AdmissionStoreError::invalid(
"admission operation version must be positive",
));
}
if self.expires_at_unix_ms <= 0 {
return Err(AdmissionStoreError::invalid(
"admission deadline is invalid",
));
}
Ok(())
}
}
/// Scoped permit. Held for the whole local execution future (including retries) and released
/// explicitly; a dropped permit is cleaned up by a bounded owner-token CAS, never reused by TTL.
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct AdmissionPermit {
pub ticket_id: u64,
pub owner_token: String,
pub scope: AdmissionScope,
pub mode: AdmissionMode,
pub policy_mode: ToolExecutionMode,
}
/// Durable enqueue decision: a reserved queue slot with exact Published Version semantics.
/// It is not a public Task id and it never authorises a worker dispatch (Story 4.2).
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct AdmissionDecision {
pub ticket_id: u64,
pub scope: AdmissionScope,
pub expires_at_unix_ms: i64,
pub policy_mode: ToolExecutionMode,
}
#[derive(Clone, Debug, PartialEq, Eq)]
pub enum AdmissionOutcome {
Permitted(AdmissionPermit),
Enqueued(AdmissionDecision),
/// Capacity or queue bounds were exhausted before any unbounded allocation.
Overloaded {
retry_after_ms: u64,
},
/// The queue wait reached the absolute deadline; nothing was dispatched.
DeadlineElapsed,
/// Confirmation is mandatory for this operation and no verified approval exists.
ApprovalRequired,
/// The published policy does not advertise durable Tasks and refuses the fallback.
Unsupported,
}
#[derive(Clone, Debug, Default, PartialEq, Eq)]
pub struct AdmissionQueueSnapshot {
pub waiting: u64,
pub executing: u64,
/// Age of the oldest waiting request in the authoritative queue, in milliseconds.
pub oldest_waiting_ms: Option<u64>,
}
#[derive(Clone, Debug, PartialEq, Eq, thiserror::Error)]
pub enum AdmissionStoreError {
/// The mandatory safety/capacity store cannot confirm state; every dependent path fails closed.
#[error("execution admission store is unavailable: {message}")]
Unavailable { message: String },
/// The store rejected a malformed admission request.
#[error("execution admission request is invalid: {message}")]
Invalid { message: String },
}
impl AdmissionStoreError {
pub fn unavailable(message: impl Into<String>) -> Self {
Self::Unavailable {
message: message.into(),
}
}
pub fn invalid(message: impl Into<String>) -> Self {
Self::Invalid {
message: message.into(),
}
}
}
#[async_trait]
pub trait ExecutionAdmission: Send + Sync {
/// Proves the mandatory admission state is usable (schema and operations reachable).
async fn health(&self) -> Result<(), AdmissionStoreError>;
/// Waits (bounded by `expires_at_unix_ms`) for a scoped permit, or commits a durable enqueue
/// decision for a background request, or returns a typed refusal.
async fn admit(
&self,
request: AdmissionRequest,
) -> Result<AdmissionOutcome, AdmissionStoreError>;
/// Releases the caller's own permit. Idempotent: a repeated release returns `false`.
async fn release(&self, permit: &AdmissionPermit) -> Result<bool, AdmissionStoreError>;
/// Removes the caller's own waiting ticket. Idempotent and never touches a foreign ticket.
async fn cancel(&self, ticket_id: u64, owner_token: &str) -> Result<bool, AdmissionStoreError>;
/// Bounded authoritative queue readings for low-cardinality metrics.
async fn snapshot(&self) -> Result<AdmissionQueueSnapshot, AdmissionStoreError>;
}
/// One waiting entry as seen by the fairness scheduler.
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct AdmissionQueueEntry {
pub ticket_id: u64,
/// Authoritative insertion order inside `(Agent, Tool)`; never rewritten.
pub sequence: u64,
pub agent_key: String,
pub operation_id: String,
}
/// Weighted round-robin state persisted with the authoritative queue.
#[derive(Clone, Debug, Default, PartialEq, Eq)]
pub struct AdmissionCursor {
pub agent: Option<String>,
pub agent_credit: u32,
pub tool: Option<String>,
pub tool_credit: u32,
}
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub struct AdmissionSchedulerConfig {
pub agent_weight: u32,
pub tool_weight: u32,
}
impl Default for AdmissionSchedulerConfig {
fn default() -> Self {
Self {
agent_weight: 1,
tool_weight: 1,
}
}
}
impl AdmissionSchedulerConfig {
pub fn validate(&self) -> Result<(), AdmissionStoreError> {
if !(1..=64).contains(&self.agent_weight) || !(1..=64).contains(&self.tool_weight) {
return Err(AdmissionStoreError::invalid(
"admission weights must be between 1 and 64",
));
}
Ok(())
}
}
/// Result of one fair-selection step: the queue index to grant and the cursor to persist when the
/// grant actually happens. The cursor is not mutated on a blocked head, so a blocked head keeps its
/// turn instead of being skipped by newer arrivals.
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct AdmissionSelection {
pub index: usize,
pub cursor: AdmissionCursor,
}
/// Hierarchical weighted round-robin: Agents first, then the Tools of the chosen Agent, FIFO inside
/// `(Agent, Tool)`. A new waiter never resets the cursor and never inherits idle credit.
pub fn select_next_grant(
queue: &[AdmissionQueueEntry],
cursor: &AdmissionCursor,
config: AdmissionSchedulerConfig,
) -> Option<AdmissionSelection> {
if queue.is_empty() {
return None;
}
let agent_weight = config.agent_weight.max(1);
let tool_weight = config.tool_weight.max(1);
let mut agents: Vec<&str> = queue.iter().map(|entry| entry.agent_key.as_str()).collect();
agents.sort_unstable();
agents.dedup();
let mut next = cursor.clone();
let agent = match cursor.agent.as_deref() {
Some(current) if cursor.agent_credit > 0 && agents.contains(&current) => current.to_owned(),
_ => {
let chosen = match cursor.agent.as_deref() {
None => agents[0].to_owned(),
Some(current) => agents
.iter()
.copied()
.find(|candidate| *candidate > current)
.unwrap_or(agents[0])
.to_owned(),
};
if cursor.agent.as_deref() != Some(chosen.as_str()) {
next.tool = None;
next.tool_credit = 0;
}
next.agent = Some(chosen.clone());
next.agent_credit = agent_weight;
chosen
}
};
let mut tools: Vec<&str> = queue
.iter()
.filter(|entry| entry.agent_key == agent)
.map(|entry| entry.operation_id.as_str())
.collect();
tools.sort_unstable();
tools.dedup();
let tool = match cursor.tool.as_deref() {
Some(current) if cursor.tool_credit > 0 && tools.contains(&current) => current.to_owned(),
_ => {
let chosen = match cursor.tool.as_deref() {
None => tools[0].to_owned(),
Some(current) => tools
.iter()
.copied()
.find(|candidate| *candidate > current)
.unwrap_or(tools[0])
.to_owned(),
};
next.tool = Some(chosen.clone());
next.tool_credit = tool_weight;
chosen
}
};
let index = queue
.iter()
.position(|entry| entry.agent_key == agent && entry.operation_id == tool)?;
next.agent_credit = next.agent_credit.saturating_sub(1);
next.tool_credit = next.tool_credit.saturating_sub(1);
Some(AdmissionSelection {
index,
cursor: next,
})
}
#[cfg(test)]
mod tests {
use super::{
ADMIN_ORIGIN_KEY, AdmissionCursor, AdmissionMode, AdmissionRequest,
AdmissionSchedulerConfig, AdmissionScope, AdmissionStoreError, ApprovalEvidence,
select_next_grant,
};
use crate::{NoTasksFallback, ToolExecutionMode};
fn entry(ticket_id: u64, agent: &str, tool: &str) -> super::AdmissionQueueEntry {
super::AdmissionQueueEntry {
ticket_id,
sequence: ticket_id,
agent_key: agent.to_owned(),
operation_id: tool.to_owned(),
}
}
fn request() -> AdmissionRequest {
AdmissionRequest {
scope: AdmissionScope {
workspace_id: "ws_default".to_owned(),
agent_key: ADMIN_ORIGIN_KEY.to_owned(),
operation_id: "op_leads".to_owned(),
operation_version: 3,
},
mode: AdmissionMode::Synchronous,
policy_mode: ToolExecutionMode::Synchronous,
no_tasks_fallback: NoTasksFallback::Synchronous,
approval: ApprovalEvidence::NotRequired,
fingerprint: "fp_0123456789".to_owned(),
expires_at_unix_ms: 1_790_000_000_000,
}
}
#[test]
fn quota_keys_exclude_version_and_keep_agent_bucket() {
let scope = request().scope;
assert_eq!(scope.agent_bucket(), "ws_default:admin");
assert_eq!(scope.tool_bucket(), "ws_default:op_leads");
}
#[test]
fn request_validation_rejects_zero_deadline_and_oversized_fingerprint() {
let mut invalid = request();
invalid.expires_at_unix_ms = 0;
assert!(matches!(
invalid.validate(),
Err(AdmissionStoreError::Invalid { .. })
));
let mut invalid = request();
invalid.fingerprint = "x".repeat(super::ADMISSION_FINGERPRINT_MAX_BYTES + 1);
assert!(invalid.validate().is_err());
let mut invalid = request();
invalid.scope.operation_version = 0;
assert!(invalid.validate().is_err());
assert!(request().validate().is_ok());
}
#[test]
fn weights_are_bounded_positive_configuration() {
assert!(
AdmissionSchedulerConfig {
agent_weight: 0,
tool_weight: 1
}
.validate()
.is_err()
);
assert!(
AdmissionSchedulerConfig {
agent_weight: 1,
tool_weight: 65
}
.validate()
.is_err()
);
assert!(AdmissionSchedulerConfig::default().validate().is_ok());
}
#[test]
fn weighted_round_robin_serves_each_agent_its_quantum() {
let mut queue = vec![
entry(1, "agent_a", "op_1"),
entry(2, "agent_a", "op_1"),
entry(3, "agent_a", "op_1"),
entry(4, "agent_b", "op_1"),
entry(5, "agent_b", "op_1"),
];
let config = AdmissionSchedulerConfig {
agent_weight: 2,
tool_weight: 1,
};
let mut cursor = AdmissionCursor::default();
let mut order = Vec::new();
while !queue.is_empty() {
let selection = select_next_grant(&queue, &cursor, config).unwrap();
order.push(queue.remove(selection.index).agent_key.clone());
cursor = selection.cursor;
}
assert_eq!(
order,
vec!["agent_a", "agent_a", "agent_b", "agent_b", "agent_a"]
);
}
#[test]
fn fifo_inside_one_tool_and_round_robin_between_tools() {
let mut queue = vec![
entry(1, "agent_a", "op_1"),
entry(2, "agent_a", "op_1"),
entry(3, "agent_a", "op_2"),
];
let config = AdmissionSchedulerConfig::default();
let mut cursor = AdmissionCursor::default();
let mut order = Vec::new();
while !queue.is_empty() {
let selection = select_next_grant(&queue, &cursor, config).unwrap();
// A granted ticket leaves the waiting queue: granted entries are never re-granted.
order.push(queue.remove(selection.index).ticket_id);
cursor = selection.cursor;
}
// One grant per Tool per round; FIFO is preserved inside each `(Agent, Tool)` queue.
assert_eq!(order, vec![1, 3, 2]);
}
#[test]
fn new_arrival_does_not_reset_cursor_or_receive_idle_credit() {
let queue = vec![entry(1, "agent_a", "op_1")];
let config = AdmissionSchedulerConfig {
agent_weight: 2,
tool_weight: 1,
};
let cursor = AdmissionCursor::default();
let first = select_next_grant(&queue, &cursor, config).unwrap();
assert_eq!(first.cursor.agent.as_deref(), Some("agent_a"));
assert_eq!(first.cursor.agent_credit, 1);
// A later Agent joins; the cursor keeps its remaining quantum for `agent_a`.
let queue = vec![entry(1, "agent_a", "op_1"), entry(2, "agent_b", "op_9")];
let second = select_next_grant(&queue, &first.cursor, config).unwrap();
assert_eq!(queue[second.index].agent_key, "agent_a");
assert_eq!(second.cursor.agent_credit, 0);
let third = select_next_grant(&queue, &second.cursor, config).unwrap();
assert_eq!(queue[third.index].agent_key, "agent_b");
assert_eq!(third.cursor.agent_credit, 1);
}
#[test]
fn empty_queue_keeps_cursor_untouched() {
let cursor = AdmissionCursor {
agent: Some("agent_b".to_owned()),
agent_credit: 1,
tool: Some("op_1".to_owned()),
tool_credit: 0,
};
assert_eq!(
select_next_grant(&[], &cursor, AdmissionSchedulerConfig::default()),
None
);
assert_eq!(cursor.agent.as_deref(), Some("agent_b"));
}
}
+38 -9
View File
@@ -1,4 +1,5 @@
pub mod access;
pub mod admission;
pub mod agent;
pub mod approval;
pub mod auth;
@@ -17,6 +18,7 @@ pub mod prompt_composition;
pub mod protocol;
pub mod resource;
pub mod secret;
pub mod task;
pub mod tool_catalog;
pub mod tool_quality;
pub mod tool_search;
@@ -60,10 +62,11 @@ pub mod domain {
pub use crate::onboarding::{OnboardingProjection, OnboardingStep, OnboardingStepId};
pub use crate::operation::{
ConfigExport, ConfirmationPolicy, ExecutionConfig, GeneratedDraft, GeneratedDraftStatus,
IdempotencyMode, IdempotencyPolicy, Operation, OperationApprovalMode,
IdempotencyMode, IdempotencyPolicy, NoTasksFallback, Operation, OperationApprovalMode,
OperationApprovalPayloadPreviewMode, OperationApprovalPolicy, OperationApprovalRiskLevel,
OperationSafetyClass, OperationSafetyPolicy, OperationStatus, ResponseCachePolicy,
RestTarget, RetryPolicy, Samples, Target, ToolDescription, ToolExample, WizardState,
RestTarget, RetryPolicy, Samples, Target, ToolDescription, ToolExample, ToolExecutionMode,
ToolExecutionPolicy, WizardState,
};
pub use crate::product_event::{
OnboardingMilestone, PRODUCT_EVENT_IDEMPOTENCY_KEY_MAX_BYTES, PRODUCT_EVENT_SCHEMA_VERSION,
@@ -71,6 +74,15 @@ pub mod domain {
};
pub use crate::protocol::{AuthKind, ExportMode, HttpMethod, Protocol};
pub use crate::secret::{Secret, SecretKind, SecretStatus, SecretVersion};
pub use crate::task::{
TASK_APPROVAL_REFERENCE_MAX_BYTES, TASK_DEFAULT_RETENTION_MS, TASK_ENVELOPE_MAX_BYTES,
TASK_ENVELOPE_MAX_DEPTH, TASK_ENVELOPE_MAX_NODES,
TASK_EXECUTABLE_INPUT_CIPHERTEXT_MAX_BYTES, TASK_EXECUTABLE_INPUT_MAX_BYTES,
TASK_IDENTITY_MAX_BYTES, TASK_MAX_RETENTION_MS, TASK_PURGE_BATCH_LIMIT,
TASK_SCOPED_KEY_MAX_BYTES, TaskAcceptRequest, TaskAcceptance, TaskAccess, TaskEnvelope,
TaskError, TaskFinishRequest, TaskId, TaskOwnerKind, TaskPins, TaskPurgeReport, TaskRecord,
TaskState, TaskSupportIds,
};
pub use crate::tool_catalog::{ToolCatalogAnalysis, ToolCatalogBudget};
pub use crate::tool_quality::{
ToolQualityCatalogTool, ToolQualityFinding, ToolQualityMappingRule, ToolQualityMappingSet,
@@ -108,12 +120,20 @@ pub mod ports {
PreparedRequest, ProtocolAdapter, ProtocolAdapterError, ResponseCacheScope,
RuntimeRequestContext, SharedProtocolAdapter,
};
pub use crate::task::TaskStore;
}
pub use access::{
InvitationStatus, InvitationToken, Membership, MembershipRole, PlatformApiKey,
PlatformApiKeyKind, PlatformApiKeyScope, PlatformApiKeyStatus, User, UserStatus,
};
pub use admission::{
ADMIN_ORIGIN_KEY, ADMISSION_FINGERPRINT_MAX_BYTES, ADMISSION_OVERLOAD_RETRY_AFTER_MS,
AdmissionCursor, AdmissionDecision, AdmissionMode, AdmissionOutcome, AdmissionPermit,
AdmissionQueueEntry, AdmissionQueueSnapshot, AdmissionRequest, AdmissionSchedulerConfig,
AdmissionScope, AdmissionSelection, AdmissionStoreError, ApprovalEvidence, ExecutionAdmission,
MAX_RETRY_AFTER_MS, current_unix_ms, select_next_grant,
};
pub use agent::{
Agent, AgentOperationBinding, AgentStatus, AgentVersion, ToolAccessMode, ToolGroup,
ToolSearchSettings, ToolSelectionPolicy, ToolSelectionPolicyError,
@@ -168,13 +188,14 @@ pub use observability::{
pub use onboarding::{OnboardingProjection, OnboardingStep, OnboardingStepId};
pub use operation::{
ConfigExport, ConfirmationPolicy, ExecutionConfig, GeneratedDraft, GeneratedDraftStatus,
IdempotencyMode, IdempotencyPolicy, Operation, OperationApprovalMode,
OperationApprovalPayloadPreviewMode, OperationApprovalPolicy, OperationApprovalRiskLevel,
OperationAvailability, OperationLifecycle, OperationLifecycleAction, OperationLifecycleError,
OperationSafetyClass, OperationSafetyPolicy, OperationStatus, OperationVersionState,
ResponseCachePolicy, RestRequestContract, RestRequestContractVersion, RestRequestDescriptor,
RestResponseDescriptor, RestTarget, RetryPolicy, Samples, Target, ToolDescription, ToolExample,
WizardState,
IdempotencyMode, IdempotencyPolicy, MAX_TOOL_DEADLINE_MS, NoTasksFallback, Operation,
OperationApprovalMode, OperationApprovalPayloadPreviewMode, OperationApprovalPolicy,
OperationApprovalRiskLevel, OperationAvailability, OperationLifecycle,
OperationLifecycleAction, OperationLifecycleError, OperationSafetyClass, OperationSafetyPolicy,
OperationStatus, OperationVersionState, ResponseCachePolicy, RestRequestContract,
RestRequestContractVersion, RestRequestDescriptor, RestResponseDescriptor, RestTarget,
RetryPolicy, Samples, Target, ToolDescription, ToolExample, ToolExecutionMode,
ToolExecutionPolicy, ToolExecutionPolicyError, WizardState,
};
pub use product_event::{
OnboardingMilestone, PRODUCT_EVENT_IDEMPOTENCY_KEY_MAX_BYTES, PRODUCT_EVENT_SCHEMA_VERSION,
@@ -182,6 +203,14 @@ pub use product_event::{
};
pub use protocol::{AuthKind, ExportMode, HttpMethod, Protocol};
pub use secret::{Secret, SecretKind, SecretStatus, SecretVersion};
pub use task::{
TASK_DEFAULT_RETENTION_MS, TASK_ENVELOPE_MAX_BYTES, TASK_EXECUTABLE_INPUT_CIPHERTEXT_MAX_BYTES,
TASK_EXECUTABLE_INPUT_MAX_BYTES, TASK_MAX_RETENTION_MS, TASK_PURGE_BATCH_LIMIT,
TASK_SCOPED_KEY_MAX_BYTES, TaskAcceptRequest, TaskAcceptance, TaskAccess, TaskEnvelope,
TaskError, TaskFinishRequest, TaskId, TaskLease, TaskOwnerKind, TaskPins, TaskPurgeReport,
TaskRecord, TaskState, TaskStore, TaskSupportIds, contains_sensitive_task_keys,
redact_task_envelope, sensitive_task_key, validate_task_envelope,
};
pub use tool_catalog::{
RECOMMENDED_TOOL_CATALOG_CONTEXT_TOKENS, ToolCatalogAnalysis, ToolCatalogAnalysisError,
ToolCatalogBudget, analyze_tool_catalog,
+144 -2
View File
@@ -270,6 +270,72 @@ pub struct OperationApprovalPolicy {
pub elicitation_message: Option<String>,
}
/// Published execution mode of one Tool (Story 4.1, FR-21).
///
/// This is a scheduling policy of the Published Tool version. It is deliberately separate from
/// `ext::protocol::ExecutionMode`, which describes the protocol adapter (streaming vs unary).
#[derive(Clone, Copy, Debug, Default, PartialEq, Eq, Serialize, Deserialize)]
#[serde(rename_all = "snake_case")]
pub enum ToolExecutionMode {
#[default]
Synchronous,
Background,
Adaptive,
}
/// Behaviour promised to a client that does not negotiate durable Tasks (Story 4.1).
#[derive(Clone, Copy, Debug, Default, PartialEq, Eq, Serialize, Deserialize)]
#[serde(rename_all = "snake_case")]
pub enum NoTasksFallback {
/// Execute through the same synchronous admission path.
#[default]
Synchronous,
/// Reject with a typed unsupported-execution-mode failure; never a Task ID.
Unsupported,
}
pub const MAX_TOOL_DEADLINE_MS: u64 = 3_600_000;
/// Execution policy pinned by a Published Tool version.
///
/// Absence of the policy keeps the pre-4.1 behaviour: synchronous execution bounded by
/// `ExecutionConfig::timeout_ms`, with no background path.
#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize, Deserialize)]
#[serde(deny_unknown_fields)]
pub struct ToolExecutionPolicy {
#[serde(default)]
pub mode: ToolExecutionMode,
/// Absolute budget of one execution, counted from the start of admission.
pub deadline_ms: u64,
#[serde(default)]
pub no_tasks_fallback: NoTasksFallback,
}
impl Default for ToolExecutionPolicy {
fn default() -> Self {
Self {
mode: ToolExecutionMode::Synchronous,
deadline_ms: 30_000,
no_tasks_fallback: NoTasksFallback::Synchronous,
}
}
}
impl ToolExecutionPolicy {
pub fn validate(&self) -> Result<(), ToolExecutionPolicyError> {
if !(1..=MAX_TOOL_DEADLINE_MS).contains(&self.deadline_ms) {
return Err(ToolExecutionPolicyError::DeadlineOutOfRange);
}
Ok(())
}
}
#[derive(Clone, Copy, Debug, PartialEq, Eq, thiserror::Error)]
pub enum ToolExecutionPolicyError {
#[error("tool execution deadline must be between 1 and {MAX_TOOL_DEADLINE_MS} ms")]
DeadlineOutOfRange,
}
#[derive(Clone, Debug, Default, PartialEq, Serialize, Deserialize)]
#[serde(deny_unknown_fields)]
pub struct ExecutionConfig {
@@ -285,6 +351,8 @@ pub struct ExecutionConfig {
#[serde(skip_serializing_if = "Option::is_none")]
pub approval_policy: Option<OperationApprovalPolicy>,
#[serde(skip_serializing_if = "Option::is_none")]
pub execution_policy: Option<ToolExecutionPolicy>,
#[serde(skip_serializing_if = "Option::is_none")]
pub auth_profile_ref: Option<AuthProfileId>,
#[serde(default, skip_serializing_if = "BTreeMap::is_empty")]
pub headers: BTreeMap<String, String>,
@@ -444,9 +512,10 @@ mod tests {
edition::OperationSecurityLevel,
ids::{AuthProfileId, OperationId},
operation::{
ConfigExport, ExecutionConfig, Operation, OperationAvailability, OperationLifecycle,
OperationLifecycleAction, OperationLifecycleError, OperationStatus,
ConfigExport, ExecutionConfig, NoTasksFallback, Operation, OperationAvailability,
OperationLifecycle, OperationLifecycleAction, OperationLifecycleError, OperationStatus,
OperationVersionState, RestTarget, Samples, Target, ToolDescription, ToolExample,
ToolExecutionMode, ToolExecutionPolicy,
},
protocol::{AuthKind, ExportMode, HttpMethod, Protocol},
};
@@ -664,6 +733,7 @@ updated_at: 2026-03-25T08:10:00Z
idempotency: None,
safety: None,
approval_policy: None,
execution_policy: None,
auth_profile_ref: Some(AuthProfileId::new("auth_01")),
headers: BTreeMap::new(),
},
@@ -684,4 +754,76 @@ updated_at: 2026-03-25T08:10:00Z
published_at: None,
}
}
#[test]
fn execution_policy_round_trips_all_modes_and_fallbacks() {
for (mode, fallback) in [
(ToolExecutionMode::Synchronous, NoTasksFallback::Synchronous),
(ToolExecutionMode::Background, NoTasksFallback::Synchronous),
(ToolExecutionMode::Background, NoTasksFallback::Unsupported),
(ToolExecutionMode::Adaptive, NoTasksFallback::Unsupported),
] {
let config = ExecutionConfig {
execution_policy: Some(ToolExecutionPolicy {
mode,
deadline_ms: 45_000,
no_tasks_fallback: fallback,
}),
..ExecutionConfig::default()
};
let encoded = serde_json::to_value(&config).unwrap();
assert_eq!(encoded["execution_policy"]["deadline_ms"], json!(45_000));
let decoded: ExecutionConfig = serde_json::from_value(encoded).unwrap();
assert_eq!(decoded, config);
assert_eq!(
decoded.execution_policy.unwrap().mode,
mode,
"the published mode must survive the JSON round-trip"
);
}
}
#[test]
fn legacy_execution_config_without_policy_stays_synchronous() {
let legacy: ExecutionConfig =
serde_json::from_value(json!({ "timeout_ms": 12_000 })).unwrap();
assert_eq!(legacy.execution_policy, None);
assert_eq!(legacy.timeout_ms, 12_000);
}
#[test]
fn execution_policy_rejects_explicit_invalid_deadline() {
assert!(
ToolExecutionPolicy {
mode: ToolExecutionMode::Background,
deadline_ms: 0,
no_tasks_fallback: NoTasksFallback::Synchronous,
}
.validate()
.is_err()
);
assert!(
ToolExecutionPolicy {
mode: ToolExecutionMode::Adaptive,
deadline_ms: crate::operation::MAX_TOOL_DEADLINE_MS + 1,
no_tasks_fallback: NoTasksFallback::Unsupported,
}
.validate()
.is_err()
);
assert!(ToolExecutionPolicy::default().validate().is_ok());
}
#[test]
fn execution_policy_rejects_unknown_fields() {
assert!(
serde_json::from_value::<ToolExecutionPolicy>(json!({
"mode": "background",
"deadline_ms": 1000,
"no_tasks_fallback": "synchronous",
"task_queue": "default"
}))
.is_err()
);
}
}
+874
View File
@@ -0,0 +1,874 @@
//! Durable background Task seam (Story 4.2, FR-22/FR-23).
//!
//! A Task is the internal, protocol-independent record of one accepted background request. This
//! module owns the domain vocabulary only: the lifecycle states, the accepted identity/pins, the
//! bounded persisted envelopes and the `TaskStore` port. The protocol adapter (Story 4.7) and the
//! worker (Story 4.3) do not define these semantics; the authoritative store lives in PostgreSQL
//! (`crank-registry`), so an accepted Task survives a crash or restart.
//!
//! Deliberately absent: credentials, prepared headers, approval bearer tokens and any claim that a
//! Task was executed. A terminal state is written by a conditional transition only.
use async_trait::async_trait;
use serde::{Deserialize, Serialize};
use serde_json::Value;
use uuid::Uuid;
use crate::correlation::{RequestId, TraceContext, TraceId};
use crate::ids::PlatformApiKeyId;
use crate::operation::ToolExecutionMode;
/// Byte budget of every persisted Task envelope (input, progress, result, error, schema metadata).
pub const TASK_ENVELOPE_MAX_BYTES: usize = 32_768;
/// Maximum byte budget of the decrypted executable Task input.
pub const TASK_EXECUTABLE_INPUT_MAX_BYTES: usize = TASK_ENVELOPE_MAX_BYTES;
/// Maximum byte budget of the stored encrypted executable Task input ciphertext.
pub const TASK_EXECUTABLE_INPUT_CIPHERTEXT_MAX_BYTES: usize = 98_304;
/// Maximum nesting depth of a persisted Task envelope.
pub const TASK_ENVELOPE_MAX_DEPTH: usize = 8;
/// Maximum number of JSON members/nodes of a persisted Task envelope.
pub const TASK_ENVELOPE_MAX_NODES: usize = 256;
/// Bound of every scope/owner/operation identity column.
pub const TASK_IDENTITY_MAX_BYTES: usize = 160;
/// Bound of the caller-supplied idempotency key before it becomes a scoped digest.
pub const TASK_SCOPED_KEY_MAX_BYTES: usize = 128;
/// Bound of a saved approval reference (an identifier, never a bearer token).
pub const TASK_APPROVAL_REFERENCE_MAX_BYTES: usize = 160;
/// Default retention of a terminal Task after its execution deadline (internal bounded default).
pub const TASK_DEFAULT_RETENTION_MS: i64 = 86_400_000;
/// Upper bound on the retention an accepted Task may pin.
pub const TASK_MAX_RETENTION_MS: i64 = 30 * 86_400_000;
/// Upper bound of one retention/housekeeping batch.
pub const TASK_PURGE_BATCH_LIMIT: u32 = 128;
/// Stable identity of one accepted Task. Generated as UUIDv7 and returned only after the accepting
/// transaction committed, so the ID never exists without its durable record.
#[derive(Clone, Copy, Debug, PartialEq, Eq, PartialOrd, Ord, Hash, Serialize, Deserialize)]
#[serde(transparent)]
pub struct TaskId(Uuid);
impl TaskId {
pub fn generate() -> Self {
Self(Uuid::now_v7())
}
pub fn parse(value: &str) -> Result<Self, TaskError> {
Uuid::parse_str(value)
.map(Self)
.map_err(|_| TaskError::invalid("task identity is not a UUID"))
}
pub fn as_uuid(self) -> Uuid {
self.0
}
}
impl std::fmt::Display for TaskId {
fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
self.0.fmt(formatter)
}
}
/// Closed Task lifecycle vocabulary. `outcome_unknown` is a non-terminal state: a value commit that
/// failed to confirm its dispatch is never collapsed into a plain failure.
#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize, Deserialize)]
#[serde(rename_all = "snake_case")]
pub enum TaskState {
Queued,
Working,
InputRequired,
OutcomeUnknown,
Completed,
Failed,
Cancelled,
}
impl TaskState {
pub const fn as_str(self) -> &'static str {
match self {
Self::Queued => "queued",
Self::Working => "working",
Self::InputRequired => "input_required",
Self::OutcomeUnknown => "outcome_unknown",
Self::Completed => "completed",
Self::Failed => "failed",
Self::Cancelled => "cancelled",
}
}
pub fn parse(value: &str) -> Result<Self, TaskError> {
match value {
"queued" => Ok(Self::Queued),
"working" => Ok(Self::Working),
"input_required" => Ok(Self::InputRequired),
"outcome_unknown" => Ok(Self::OutcomeUnknown),
"completed" => Ok(Self::Completed),
"failed" => Ok(Self::Failed),
"cancelled" => Ok(Self::Cancelled),
_ => Err(TaskError::invalid("unknown task state")),
}
}
pub const fn is_terminal(self) -> bool {
matches!(self, Self::Completed | Self::Failed | Self::Cancelled)
}
/// One conditional transition may only move along this table. A terminal state has no successor,
/// so a repeated terminal write can never rewrite an accepted result.
pub const fn allows_transition_to(self, next: Self) -> bool {
match self {
Self::Queued => matches!(next, Self::Working | Self::Failed | Self::Cancelled),
Self::Working => matches!(
next,
Self::InputRequired
| Self::OutcomeUnknown
| Self::Completed
| Self::Failed
| Self::Cancelled
),
Self::InputRequired => matches!(next, Self::Working | Self::Failed | Self::Cancelled),
Self::OutcomeUnknown => {
matches!(next, Self::Completed | Self::Failed | Self::Cancelled)
}
Self::Completed | Self::Failed | Self::Cancelled => false,
}
}
}
/// Who owns the accepted Task. An administrator is never a wildcard owner of every Task: the owner
/// identity and the authorization scope are separate, both-pinned columns.
#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize, Deserialize)]
#[serde(rename_all = "snake_case")]
pub enum TaskOwnerKind {
Agent,
Admin,
}
impl TaskOwnerKind {
pub const fn as_str(self) -> &'static str {
match self {
Self::Agent => "agent",
Self::Admin => "admin",
}
}
pub fn parse(value: &str) -> Result<Self, TaskError> {
match value {
"agent" => Ok(Self::Agent),
"admin" => Ok(Self::Admin),
_ => Err(TaskError::invalid("unknown task owner kind")),
}
}
}
/// Support identities captured at acceptance. They are stored and returned independently of
/// sampling, and a replay never rewrites them.
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct TaskSupportIds {
pub request_id: RequestId,
pub trace_id: TraceId,
pub traceparent: String,
pub trace_sampled: bool,
}
impl TaskSupportIds {
pub fn new(request_id: RequestId, trace_context: &TraceContext) -> Self {
Self {
request_id,
trace_id: trace_context.trace_id().clone(),
traceparent: trace_context.traceparent().to_owned(),
trace_sampled: trace_context.is_sampled(),
}
}
/// Re-validates both the trace identity and the parent context: a Task must never persist an
/// inconsistent or malformed support pair, and an unsampled context is still a valid parent.
pub fn validate(&self) -> Result<(), TaskError> {
if self.trace_id.as_str().is_empty() {
return Err(TaskError::invalid("task trace identity is missing"));
}
let context = TraceContext::parse(&self.traceparent)
.map_err(|_| TaskError::invalid("task trace parent is invalid"))?;
if context.trace_id() != &self.trace_id || context.is_sampled() != self.trace_sampled {
return Err(TaskError::invalid("task trace parent is inconsistent"));
}
Ok(())
}
}
/// Execution semantics pinned at acceptance: a later publication never rewrites them.
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct TaskPins {
pub operation_version: u32,
pub policy_mode: ToolExecutionMode,
pub execution_deadline_unix_ms: i64,
/// Retention deadline of the record; never the same thing as the execution deadline.
pub retain_until_unix_ms: i64,
/// Verified approval evidence reference (identifier only, never a bearer token).
pub approval_reference: Option<String>,
}
impl TaskPins {
pub fn validate(&self) -> Result<(), TaskError> {
if self.operation_version == 0 {
return Err(TaskError::invalid(
"task operation version must be positive",
));
}
if self.execution_deadline_unix_ms <= 0 {
return Err(TaskError::invalid("task execution deadline is invalid"));
}
if self.retain_until_unix_ms < self.execution_deadline_unix_ms {
return Err(TaskError::invalid(
"task retention deadline precedes execution",
));
}
if self.retain_until_unix_ms - self.execution_deadline_unix_ms > TASK_MAX_RETENTION_MS {
return Err(TaskError::invalid("task retention exceeds its bound"));
}
if let Some(reference) = self.approval_reference.as_deref()
&& (reference.is_empty()
|| reference.len() > TASK_APPROVAL_REFERENCE_MAX_BYTES
|| reference.bytes().any(|byte| byte.is_ascii_control()))
{
return Err(TaskError::invalid("task approval reference is invalid"));
}
Ok(())
}
}
/// Bounded, classified payload of a Task. Construction validates the byte/depth/node budget and
/// redacts secret-bearing fields before anything is hashed or persisted.
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct TaskEnvelope {
value: Value,
}
impl TaskEnvelope {
/// Validates the caller-provided value, then stores a redacted copy of it. The caller hashes the
/// original value separately, so redaction never collapses two different requests into one.
pub fn try_redacted(input: &Value) -> Result<Self, TaskError> {
validate_task_envelope(input)?;
Ok(Self {
value: redact_task_envelope(input),
})
}
/// Accepts a value that was already validated and redacted at a trust boundary.
pub fn from_persisted(value: Value) -> Result<Self, TaskError> {
validate_task_envelope(&value)?;
Ok(Self { value })
}
pub fn value(&self) -> &Value {
&self.value
}
pub fn into_value(self) -> Value {
self.value
}
}
/// Validates the size, nesting depth and node count of one Task envelope. Every byte, collection and
/// node is counted before the value is cloned, hashed or persisted.
pub fn validate_task_envelope(value: &Value) -> Result<(), TaskError> {
let encoded = serde_json::to_vec(value)
.map_err(|_| TaskError::invalid("task envelope is not serializable"))?;
if encoded.len() > TASK_ENVELOPE_MAX_BYTES {
return Err(TaskError::invalid("task envelope exceeds its byte budget"));
}
let mut nodes = 0usize;
check_budget(value, 1, &mut nodes)?;
Ok(())
}
fn check_budget(value: &Value, depth: usize, nodes: &mut usize) -> Result<(), TaskError> {
if depth > TASK_ENVELOPE_MAX_DEPTH {
return Err(TaskError::invalid("task envelope exceeds its depth budget"));
}
*nodes += 1;
if *nodes > TASK_ENVELOPE_MAX_NODES {
return Err(TaskError::invalid("task envelope exceeds its node budget"));
}
match value {
Value::Object(object) => {
for (key, child) in object {
check_budget_name(key)?;
check_budget(child, depth + 1, nodes)?;
}
}
Value::Array(values) => {
for child in values {
check_budget(child, depth + 1, nodes)?;
}
}
_ => {}
}
Ok(())
}
fn check_budget_name(key: &str) -> Result<(), TaskError> {
if key.len() > TASK_IDENTITY_MAX_BYTES {
return Err(TaskError::invalid("task envelope key exceeds its budget"));
}
Ok(())
}
/// Field-name classification of secret-bearing values. A matching field keeps its position but never
/// its plaintext in the persisted envelope.
pub fn sensitive_task_key(name: &str) -> bool {
let lowered = name.to_ascii_lowercase();
[
"password",
"passwd",
"secret",
"token",
"api_key",
"apikey",
"authorization",
"credential",
"private_key",
"cookie",
"signature",
]
.iter()
.any(|marker| lowered.contains(marker))
}
/// Redacted projection of an already validated envelope.
pub fn redact_task_envelope(value: &Value) -> Value {
match value {
Value::Object(object) => Value::Object(
object
.iter()
.map(|(key, child)| {
let value = if sensitive_task_key(key) {
Value::String("[redacted]".to_owned())
} else {
redact_task_envelope(child)
};
(key.clone(), value)
})
.collect(),
),
Value::Array(values) => Value::Array(values.iter().map(redact_task_envelope).collect()),
other => other.clone(),
}
}
/// Recursive check used to fail closed on sensitive fields before encryption.
pub fn contains_sensitive_task_keys(value: &Value) -> bool {
match value {
Value::Object(object) => object
.iter()
.any(|(key, child)| sensitive_task_key(key) || contains_sensitive_task_keys(child)),
Value::Array(values) => values.iter().any(contains_sensitive_task_keys),
_ => false,
}
}
/// Write request of a new durable Task. The store owns the transaction: Task, dispatch intent and
/// queue capacity commit together, and no acknowledgement exists before that commit.
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct TaskAcceptRequest {
pub task_id: TaskId,
pub workspace_id: String,
pub owner_kind: TaskOwnerKind,
pub owner_id: String,
pub authorization_scope: String,
/// Typed identity of the Platform API key that authorized the acceptance (Story 4.3). Only the
/// immutable key ID is stored, never the bearer secret; a worker revalidates it authoritatively.
/// `None` means the accepting principal has no machine key (administrator) and fail-closes.
pub platform_api_key_id: Option<PlatformApiKeyId>,
pub operation_id: String,
pub pins: TaskPins,
/// Digest of the caller key over the full owner/authorization/Tool scope.
pub scoped_key: String,
/// Digest of the canonical validated executable input.
pub input_fingerprint: String,
pub input: TaskEnvelope,
pub support: TaskSupportIds,
/// Bounded encrypted executable input, persisted alongside the redacted input projection.
pub executable_input_ciphertext: Option<String>,
pub executable_input_key_version: Option<String>,
pub executable_input_master_key_epoch: Option<i64>,
}
impl TaskAcceptRequest {
pub fn validate(&self) -> Result<(), TaskError> {
for (field, value) in [
("workspace", self.workspace_id.as_str()),
("owner", self.owner_id.as_str()),
("authorization scope", self.authorization_scope.as_str()),
("operation", self.operation_id.as_str()),
] {
if value.is_empty() || value.len() > TASK_IDENTITY_MAX_BYTES {
return Err(TaskError::invalid(format!("task {field} is invalid")));
}
}
if let Some(key_id) = self.platform_api_key_id.as_ref()
&& (key_id.as_str().is_empty() || key_id.as_str().len() > TASK_IDENTITY_MAX_BYTES)
{
return Err(TaskError::invalid(
"task platform api key reference is invalid",
));
}
if self.scoped_key.len() < 16 || self.scoped_key.len() > TASK_SCOPED_KEY_MAX_BYTES {
return Err(TaskError::invalid("task scoped key digest is invalid"));
}
if self.input_fingerprint.len() != 64
|| !self
.input_fingerprint
.bytes()
.all(|byte| byte.is_ascii_hexdigit() && !byte.is_ascii_uppercase())
{
return Err(TaskError::invalid("task input fingerprint is invalid"));
}
self.pins.validate()?;
self.support.validate()?;
let has_ciphertext = self.executable_input_ciphertext.is_some();
let has_key = self.executable_input_key_version.is_some();
let has_epoch = self.executable_input_master_key_epoch.is_some();
match (has_ciphertext, has_key, has_epoch) {
(false, false, false) => {}
(true, true, true) => {
let ciphertext = self.executable_input_ciphertext.as_deref().unwrap();
if ciphertext.is_empty()
|| ciphertext.len() > TASK_EXECUTABLE_INPUT_CIPHERTEXT_MAX_BYTES
{
return Err(TaskError::invalid(
"task executable input ciphertext is invalid",
));
}
let key_version = self.executable_input_key_version.as_deref().unwrap();
if key_version.is_empty() || key_version.len() > 64 {
return Err(TaskError::invalid(
"task executable input ciphertext is invalid",
));
}
if self.executable_input_master_key_epoch.unwrap_or_default() < 1 {
return Err(TaskError::invalid(
"task executable input ciphertext is invalid",
));
}
}
_ => {
return Err(TaskError::invalid(
"task executable input ciphertext is invalid",
));
}
}
Ok(())
}
}
/// Successful acceptance, returned only after the accepting transaction committed. A replay returns
/// the stored identity and support IDs of the original acceptance.
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct TaskAcceptance {
pub task_id: TaskId,
pub state: TaskState,
/// Exact Published Version pinned by this acceptance; a replay returns the stored pin.
pub operation_version: u32,
pub support: TaskSupportIds,
pub created_at_unix_ms: i64,
pub retain_until_unix_ms: i64,
pub replayed: bool,
}
/// Owner-side access proof of a read or a terminal write. It is built by the application layer from
/// the verified principal, never from a caller-supplied marker, and every store query carries the
/// whole predicate so a foreign row is never fetched.
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct TaskAccess {
pub workspace_id: String,
pub owner_kind: TaskOwnerKind,
pub owner_id: String,
pub authorization_scope: String,
}
impl TaskAccess {
pub fn validate(&self) -> Result<(), TaskError> {
for (field, value) in [
("workspace", self.workspace_id.as_str()),
("owner", self.owner_id.as_str()),
("authorization scope", self.authorization_scope.as_str()),
] {
if value.is_empty() || value.len() > TASK_IDENTITY_MAX_BYTES {
return Err(TaskError::invalid(format!("task {field} is invalid")));
}
}
Ok(())
}
}
/// Read model of one Task. Lifecycle fields, pins and support IDs stay readable together; the
/// protocol adapter only maps them.
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct TaskRecord {
pub task_id: TaskId,
pub state: TaskState,
pub workspace_id: String,
pub owner_kind: TaskOwnerKind,
pub owner_id: String,
pub authorization_scope: String,
/// Typed immutable identity of the Platform API key that authorized the acceptance; the worker
/// revalidates this exact key against the registry before any outbound dispatch.
pub platform_api_key_id: Option<PlatformApiKeyId>,
pub operation_id: String,
/// Canonical digest of the accepted executable input (compare-and-conflict identity).
pub input_fingerprint: String,
pub pins: TaskPins,
pub support: TaskSupportIds,
pub input: TaskEnvelope,
pub progress: Option<TaskEnvelope>,
pub result: Option<TaskEnvelope>,
pub error: Option<TaskEnvelope>,
pub attempts: u32,
/// Monotonic lease/fencing token. It never resets and protects Task state mutations.
pub fence: u64,
/// Random process-instance lease owner, if a worker currently owns the Task.
pub lease_owner: Option<String>,
/// Lease expiry epoch milliseconds, if the Task is claimed.
pub lease_until_unix_ms: Option<i64>,
/// Current dispatch attempt identity, committed before any outbound adapter call.
pub current_attempt_id: Option<String>,
/// Dispatch checkpoint epoch milliseconds, if an outbound attempt was committed.
pub dispatch_started_at_unix_ms: Option<i64>,
/// Monotonic lifecycle version used by the terminal CAS.
pub version: u32,
pub created_at_unix_ms: i64,
pub updated_at_unix_ms: i64,
}
/// One conditional terminal transition. Exactly one concurrent writer wins; the losers observe the
/// accepted state and never rewrite the result.
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct TaskFinishRequest {
pub access: TaskAccess,
pub task_id: TaskId,
pub expected_state: TaskState,
pub expected_version: u32,
pub target_state: TaskState,
pub result: Option<TaskEnvelope>,
pub error: Option<TaskEnvelope>,
}
impl TaskFinishRequest {
pub fn validate(&self) -> Result<(), TaskError> {
self.access.validate()?;
if !self.target_state.is_terminal() {
return Err(TaskError::invalid("task finish target is not terminal"));
}
if self.expected_state.is_terminal() {
return Err(TaskError::invalid("task finish source is already terminal"));
}
if !self.expected_state.allows_transition_to(self.target_state) {
return Err(TaskError::invalid("task state transition is not allowed"));
}
if self.target_state == TaskState::Completed && self.result.is_none() {
return Err(TaskError::invalid("a completed task requires a result"));
}
Ok(())
}
}
/// Bounded retention outcome.
#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)]
pub struct TaskPurgeReport {
/// Queued Tasks that passed their execution deadline and were failed with their intent removed.
pub expired_queued: u64,
/// Terminal Tasks that passed their retention deadline.
pub expired_terminal: u64,
}
/// Typed Task failure. A denial and a missing Task share one outward shape, so a foreign caller can
/// never distinguish "not yours" from "does not exist".
#[derive(Clone, Debug, PartialEq, Eq, thiserror::Error)]
pub enum TaskError {
#[error("task request is invalid: {message}")]
Invalid { message: String },
#[error("task store is unavailable: {message}")]
Unavailable { message: String },
#[error("task was not found")]
NotFound,
#[error("task request conflicts with the accepted request")]
FingerprintConflict,
#[error("task lifecycle transition was rejected")]
StateConflict,
#[error("background task capacity is exhausted")]
Overloaded { retry_after_ms: u64 },
}
impl TaskError {
pub fn invalid(message: impl Into<String>) -> Self {
Self::Invalid {
message: message.into(),
}
}
pub fn unavailable(message: impl Into<String>) -> Self {
Self::Unavailable {
message: message.into(),
}
}
/// Whether the caller may tell this failure apart from an absent Task.
pub fn is_not_found(&self) -> bool {
matches!(self, Self::NotFound)
}
}
/// Leased view of a claimed Task: its durable record plus lease fencing identity.
#[derive(Clone, Debug, PartialEq)]
pub struct TaskLease {
pub record: TaskRecord,
pub owner: String,
pub fence: u64,
pub lease_until: i64,
}
/// Authoritative durable Task port.
#[async_trait]
pub trait TaskStore: Send + Sync {
/// Proves the durable Task state is usable (schema and operations reachable).
async fn health(&self) -> Result<(), TaskError>;
/// Atomically accepts a Task: Task record, dispatch intent and one queue slot commit together.
async fn accept(&self, request: TaskAcceptRequest) -> Result<TaskAcceptance, TaskError>;
/// Reads one Task inside the caller's owner scope.
async fn read(&self, access: &TaskAccess, task_id: TaskId) -> Result<TaskRecord, TaskError>;
/// Scoped lookup of an already accepted Task by its full-scope idempotency digest. Used to keep a
/// retry of an explicitly pinned version from becoming new work; `None` means "not accepted yet".
async fn find(
&self,
access: &TaskAccess,
operation_id: &str,
scoped_key: &str,
) -> Result<Option<TaskRecord>, TaskError>;
/// Applies exactly one conditional terminal transition and releases its intent atomically.
async fn finish(&self, request: TaskFinishRequest) -> Result<TaskRecord, TaskError>;
/// Reads encrypted executable input only under a valid, unexpired current lease.
async fn read_executable_input(
&self,
owner: &str,
task_id: TaskId,
fence: u64,
) -> Result<(String, String, i64), TaskError>;
/// Bounded retention...
async fn purge_expired(&self, batch: u32) -> Result<TaskPurgeReport, TaskError>;
/// Claims a bounded batch of queued or safely recoverable Tasks under leases.
async fn claim_batch(
&self,
owner: &str,
batch_limit: u32,
lease_ms: i64,
) -> Result<Vec<TaskLease>, TaskError>;
/// Renews a lease owned by `owner` when fence and expiry still match.
async fn renew_lease(
&self,
owner: &str,
task_id: TaskId,
fence: u64,
lease_ms: i64,
) -> Result<TaskLease, TaskError>;
/// Persists redacted progress under a valid lease.
async fn write_progress(
&self,
owner: &str,
task_id: TaskId,
fence: u64,
progress: TaskEnvelope,
) -> Result<TaskRecord, TaskError>;
/// Commits an outbound attempt identity and dispatch timestamp before network use.
async fn checkpoint_dispatch(
&self,
owner: &str,
task_id: TaskId,
fence: u64,
attempt_id: String,
) -> Result<TaskRecord, TaskError>;
/// Applies a terminal transition only under a valid lease and releases intent atomically.
async fn finish_leased(
&self,
request: TaskFinishRequest,
owner: &str,
fence: u64,
) -> Result<TaskRecord, TaskError>;
}
#[cfg(test)]
mod tests {
use super::*;
use serde_json::json;
fn request() -> TaskAcceptRequest {
TaskAcceptRequest {
task_id: TaskId::generate(),
workspace_id: "ws_default".to_owned(),
owner_kind: TaskOwnerKind::Agent,
owner_id: "agent_a".to_owned(),
authorization_scope: "credential:pk_1".to_owned(),
platform_api_key_id: Some(PlatformApiKeyId::new("pk_1")),
operation_id: "op_report".to_owned(),
pins: TaskPins {
operation_version: 3,
policy_mode: ToolExecutionMode::Background,
execution_deadline_unix_ms: 1_790_000_000_000,
retain_until_unix_ms: 1_790_000_000_000 + 1_000,
approval_reference: Some("appr_1".to_owned()),
},
scoped_key: "a".repeat(64),
input_fingerprint: "b".repeat(64),
input: TaskEnvelope::try_redacted(&json!({"report": "q3"})).unwrap(),
support: TaskSupportIds::new(
RequestId::generate(),
&TraceContext::parse("00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-00")
.unwrap(),
),
executable_input_ciphertext: None,
executable_input_key_version: None,
executable_input_master_key_epoch: None,
}
}
#[test]
fn uuid_v7_task_ids_are_unique_and_round_trip() {
let first = TaskId::generate();
let second = TaskId::generate();
assert_ne!(first, second);
assert_eq!(TaskId::parse(&first.to_string()).unwrap(), first);
}
#[test]
fn terminal_states_have_no_successor() {
for terminal in [
TaskState::Completed,
TaskState::Failed,
TaskState::Cancelled,
] {
assert!(terminal.is_terminal());
for next in [
TaskState::Queued,
TaskState::Working,
TaskState::Completed,
TaskState::Failed,
] {
assert!(!terminal.allows_transition_to(next));
}
}
assert!(TaskState::Queued.allows_transition_to(TaskState::Working));
assert!(TaskState::Working.allows_transition_to(TaskState::OutcomeUnknown));
assert!(!TaskState::Queued.allows_transition_to(TaskState::Completed));
assert!(!TaskState::InputRequired.allows_transition_to(TaskState::Completed));
}
#[test]
fn envelope_bounds_reject_bytes_depth_and_nodes() {
assert!(TaskEnvelope::try_redacted(&json!({"a": 1})).is_ok());
assert!(
TaskEnvelope::try_redacted(&json!({"a": "x".repeat(TASK_ENVELOPE_MAX_BYTES)})).is_err()
);
let mut deep = json!(1);
for _ in 0..=TASK_ENVELOPE_MAX_DEPTH {
deep = json!({"n": deep});
}
assert!(TaskEnvelope::try_redacted(&deep).is_err());
let wide = Value::Array((0..=TASK_ENVELOPE_MAX_NODES).map(|_| json!(1)).collect());
assert!(TaskEnvelope::try_redacted(&wide).is_err());
}
#[test]
fn secret_bearing_fields_are_redacted_and_keep_their_shape() {
let envelope = TaskEnvelope::try_redacted(&json!({
"query": "leads",
"auth": {"api_key": "sk_live_1", "nested": [{"password": "p"}]}
}))
.unwrap();
let value = envelope.value();
assert_eq!(value["query"], json!("leads"));
assert_eq!(value["auth"]["api_key"], json!("[redacted]"));
assert_eq!(value["auth"]["nested"][0]["password"], json!("[redacted]"));
}
#[test]
fn unsampled_support_ids_are_valid_and_consistent() {
let support = TaskSupportIds::new(
RequestId::generate(),
&TraceContext::parse("00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-00")
.unwrap(),
);
assert!(!support.trace_sampled);
assert!(support.traceparent.ends_with("-00"));
support.validate().unwrap();
let mut inconsistent = support.clone();
inconsistent.traceparent = inconsistent.traceparent.replace("-00", "-01");
assert!(inconsistent.validate().is_err());
}
#[test]
fn accept_request_validation_holds_the_identity_and_pin_bounds() {
assert!(request().validate().is_ok());
let mut oversized = request();
oversized.workspace_id = "w".repeat(TASK_IDENTITY_MAX_BYTES + 1);
assert!(oversized.validate().is_err());
let mut bad_fingerprint = request();
bad_fingerprint.input_fingerprint = "XYZ".to_owned();
assert!(bad_fingerprint.validate().is_err());
let mut bad_retention = request();
bad_retention.pins.retain_until_unix_ms = bad_retention.pins.execution_deadline_unix_ms - 1;
assert!(bad_retention.validate().is_err());
// The persisted authority reference is bounded and never empty.
let mut empty_key_reference = request();
empty_key_reference.platform_api_key_id = Some(PlatformApiKeyId::new(""));
assert!(empty_key_reference.validate().is_err());
}
#[test]
fn finish_requires_a_terminal_reachable_transition() {
let access = TaskAccess {
workspace_id: "ws_default".to_owned(),
owner_kind: TaskOwnerKind::Agent,
owner_id: "agent_a".to_owned(),
authorization_scope: "credential:pk_1".to_owned(),
};
let base = TaskFinishRequest {
access,
task_id: TaskId::generate(),
expected_state: TaskState::Working,
expected_version: 1,
target_state: TaskState::Completed,
result: Some(TaskEnvelope::try_redacted(&json!({"ok": true})).unwrap()),
error: None,
};
assert!(base.validate().is_ok());
let mut non_terminal = base.clone();
non_terminal.target_state = TaskState::Working;
assert!(non_terminal.validate().is_err());
let mut terminal_source = base.clone();
terminal_source.expected_state = TaskState::Completed;
assert!(terminal_source.validate().is_err());
let mut completed_without_result = base;
completed_without_result.result = None;
assert!(completed_without_result.validate().is_err());
}
}
@@ -231,6 +231,7 @@ fn default_execution_config() -> ExecutionConfig {
idempotency: None,
safety: None,
approval_policy: None,
execution_policy: None,
auth_profile_ref: None,
headers: BTreeMap::new(),
}
+1
View File
@@ -564,6 +564,7 @@ fn default_execution_config() -> ExecutionConfig {
idempotency: None,
safety: None,
approval_policy: None,
execution_policy: None,
auth_profile_ref: None,
headers: BTreeMap::new(),
}
+28
View File
@@ -217,6 +217,34 @@ string_enum! {
Concurrency => "concurrency",
RateLimit => "rate_limit",
McpStream => "mcp_stream",
Admission => "admission",
}
}
string_enum! {
pub enum RejectionReason {
Limit => "limit",
Capacity => "capacity",
QueueFull => "queue_full",
Deadline => "deadline",
Unsupported => "unsupported",
DependencyUnavailable => "dependency_unavailable",
}
}
string_enum! {
pub enum ScopeKind {
Process => "process",
Global => "global",
Agent => "agent",
Tool => "tool",
}
}
string_enum! {
pub enum AdmissionModeLabel {
Synchronous => "synchronous",
Background => "background",
}
}
+6 -5
View File
@@ -16,14 +16,15 @@ mod schema;
pub use exemplar::{ExemplarObservation, ExemplarTraceId, exemplar_snapshot};
pub use labels::{
CacheOutcome, ConfirmationOutcome, DbPoolState, Exporter, HttpMethod, HttpRoute,
HttpStatusClass, IdempotencyOutcome, InvocationSource, LimitStage, McpMethod, McpOutcome,
McpResponseMode, SignalType, ToolErrorKind, ToolOutcome, UpstreamOperationKind,
UpstreamOutcome,
AdmissionModeLabel, CacheOutcome, ConfirmationOutcome, DbPoolState, Exporter, HttpMethod,
HttpRoute, HttpStatusClass, IdempotencyOutcome, InvocationSource, LimitStage, McpMethod,
McpOutcome, McpResponseMode, RejectionReason, ScopeKind, SignalType, ToolErrorKind,
ToolOutcome, UpstreamOperationKind, UpstreamOutcome,
};
pub use record::{
InFlightGuard, ToolInvocationMetrics, UpstreamRequestMetrics, bind_metric_service,
bound_metric_service, initialize_gauges, normalize_http_route, record_cache_outcome,
bound_metric_service, initialize_gauges, normalize_http_route, record_admission_oldest_waiting,
record_admission_queue_wait, record_admission_rejection, record_cache_outcome,
record_confirmation_outcome, record_export_failure, record_http_request,
record_http_request_with_exemplar, record_idempotency_outcome, record_invocation_history_lost,
record_limit_rejection, record_mcp_request, record_mcp_request_with_exemplar,
+41 -5
View File
@@ -5,10 +5,10 @@ use metrics::Gauge;
use crate::schema::{MetricDefinition, metric_schema};
use crate::{
CacheOutcome, ConfirmationOutcome, DbPoolState, ExemplarTraceId, Exporter, HttpMethod,
HttpRoute, HttpStatusClass, IdempotencyOutcome, InvocationSource, LimitStage, McpMethod,
McpOutcome, McpResponseMode, MetricBindingError, MetricService, SignalType, ToolErrorKind,
ToolOutcome, UpstreamOperationKind, UpstreamOutcome,
AdmissionModeLabel, CacheOutcome, ConfirmationOutcome, DbPoolState, ExemplarTraceId, Exporter,
HttpMethod, HttpRoute, HttpStatusClass, IdempotencyOutcome, InvocationSource, LimitStage,
McpMethod, McpOutcome, McpResponseMode, MetricBindingError, MetricService, RejectionReason,
ScopeKind, SignalType, ToolErrorKind, ToolOutcome, UpstreamOperationKind, UpstreamOutcome,
};
static BINDING: OnceLock<MetricService> = OnceLock::new();
@@ -352,11 +352,47 @@ pub fn record_limit_rejection(stage: LimitStage) {
}
metrics::counter!(
"crank_runtime_limit_rejections_total",
"stage" => stage.as_str()
"stage" => stage.as_str(),
"reason" => RejectionReason::Limit.as_str(),
"scope_kind" => ScopeKind::Process.as_str()
)
.increment(1);
}
/// Typed admission refusal with finite, low-cardinality labels only: no workspace, Agent, Tool,
/// request id, fingerprint or arbitrary error text ever becomes a label.
pub fn record_admission_rejection(reason: RejectionReason, scope_kind: ScopeKind) {
if bound_family("crank_runtime_limit_rejections_total").is_none() {
return;
}
metrics::counter!(
"crank_runtime_limit_rejections_total",
"stage" => LimitStage::Admission.as_str(),
"reason" => reason.as_str(),
"scope_kind" => scope_kind.as_str()
)
.increment(1);
}
pub fn record_admission_queue_wait(mode: AdmissionModeLabel, waited_ms: u64) {
if bound_family("crank_runtime_admission_queue_wait_seconds").is_none() {
return;
}
metrics::gauge!(
"crank_runtime_admission_queue_wait_seconds",
"mode" => mode.as_str()
)
.set(waited_ms as f64 / 1000.0);
}
pub fn record_admission_oldest_waiting(oldest_waiting_ms: Option<u64>) {
if bound_family("crank_runtime_admission_oldest_waiting_seconds").is_none() {
return;
}
let seconds = oldest_waiting_ms.unwrap_or_default() as f64 / 1000.0;
metrics::gauge!("crank_runtime_admission_oldest_waiting_seconds").set(seconds);
}
pub fn record_cache_outcome(outcome: CacheOutcome) {
if bound_family("crank_runtime_cache_total").is_none() {
return;
+38 -8
View File
@@ -3,16 +3,17 @@
use super::routes::{BOTH, HTTP_ROUTE_DOMAIN, MCP};
use crate::labels::{
CacheOutcome, ConfirmationOutcome, DbPoolState, Exporter, HttpMethod, HttpStatusClass,
IdempotencyOutcome, InvocationSource, LimitStage, McpMethod, McpOutcome, McpResponseMode,
SignalType, ToolErrorKind, ToolOutcome, UpstreamOperationKind, UpstreamOutcome,
AdmissionModeLabel, CacheOutcome, ConfirmationOutcome, DbPoolState, Exporter, HttpMethod,
HttpStatusClass, IdempotencyOutcome, InvocationSource, LimitStage, McpMethod, McpOutcome,
McpResponseMode, RejectionReason, ScopeKind, SignalType, ToolErrorKind, ToolOutcome,
UpstreamOperationKind, UpstreamOutcome,
};
pub const METRIC_SCHEMA_VERSION: u32 = 2;
pub const MAX_METRIC_FAMILIES: usize = 128;
pub const MAX_PRODUCT_LABELS_PER_FAMILY: usize = 4;
pub const MAX_LOGICAL_SERIES_PER_PROCESS: usize = 6_285;
pub const MAX_RENDERED_SERIES_PER_PROCESS: usize = 19_515;
pub const MAX_LOGICAL_SERIES_PER_PROCESS: usize = 6_381;
pub const MAX_RENDERED_SERIES_PER_PROCESS: usize = 19_611;
pub const MAX_EXEMPLAR_SLOTS_PER_PROCESS: usize = 12_348;
pub const MAX_EXPOSITION_BYTES: usize = 8 * 1024 * 1024;
@@ -179,6 +180,9 @@ const TOOL_ERRORS: &[&str] = ToolErrorKind::VALUES;
const OPERATION_KINDS: &[&str] = UpstreamOperationKind::VALUES;
const UPSTREAM_OUTCOMES: &[&str] = UpstreamOutcome::VALUES;
const LIMIT_STAGES: &[&str] = LimitStage::VALUES;
const REJECTION_REASONS: &[&str] = RejectionReason::VALUES;
const SCOPE_KINDS: &[&str] = ScopeKind::VALUES;
const ADMISSION_MODES: &[&str] = AdmissionModeLabel::VALUES;
const CACHE_OUTCOMES: &[&str] = CacheOutcome::VALUES;
const IDEMPOTENCY_OUTCOMES: &[&str] = IdempotencyOutcome::VALUES;
const CONFIRMATION_OUTCOMES: &[&str] = ConfirmationOutcome::VALUES;
@@ -403,13 +407,39 @@ pub(crate) const METRIC_SCHEMA: &[MetricDefinition] = &[
"crank_runtime_limit_rejections_total",
MetricKind::Counter,
MetricUnit::Count,
&["stage"],
&[domain("stage", LIMIT_STAGES)],
&["stage", "reason", "scope_kind"],
&[
domain("stage", LIMIT_STAGES),
domain("reason", REJECTION_REASONS),
domain("scope_kind", SCOPE_KINDS),
],
&[],
BOTH,
3,
96,
"Runtime executions rejected by a bounded limit.",
),
metric(
"crank_runtime_admission_queue_wait_seconds",
MetricKind::Gauge,
MetricUnit::Seconds,
&["mode"],
&[domain("mode", ADMISSION_MODES)],
&[],
BOTH,
2,
"Wait of the most recently admitted execution before it received a scoped permit.",
),
metric(
"crank_runtime_admission_oldest_waiting_seconds",
MetricKind::Gauge,
MetricUnit::Seconds,
&[],
&[],
&[],
BOTH,
1,
"Age of the oldest waiting admission request.",
),
metric(
"crank_runtime_cache_total",
MetricKind::Counter,
+13 -6
View File
@@ -1,12 +1,13 @@
use std::{collections::BTreeSet, time::Duration};
use crank_metrics::{
CacheOutcome, ConfirmationOutcome, DbPoolState, Exporter, HttpMethod, HttpRoute,
HttpStatusClass, IdempotencyOutcome, InFlightGuard, InvocationSource, LimitStage, McpMethod,
McpOutcome, McpResponseMode, MetricService, SignalType, ToolErrorKind, ToolInvocationMetrics,
ToolOutcome, UpstreamOperationKind, UpstreamOutcome, UpstreamRequestMetrics,
bind_metric_service, initialize_gauges, metric_schema, record_cache_outcome,
record_confirmation_outcome, record_export_failure, record_http_request,
AdmissionModeLabel, CacheOutcome, ConfirmationOutcome, DbPoolState, Exporter, HttpMethod,
HttpRoute, HttpStatusClass, IdempotencyOutcome, InFlightGuard, InvocationSource, LimitStage,
McpMethod, McpOutcome, McpResponseMode, MetricService, RejectionReason, ScopeKind, SignalType,
ToolErrorKind, ToolInvocationMetrics, ToolOutcome, UpstreamOperationKind, UpstreamOutcome,
UpstreamRequestMetrics, bind_metric_service, initialize_gauges, metric_schema,
record_admission_oldest_waiting, record_admission_queue_wait, record_admission_rejection,
record_cache_outcome, record_confirmation_outcome, record_export_failure, record_http_request,
record_idempotency_outcome, record_invocation_history_lost, record_limit_rejection,
record_mcp_request, record_tool_invocation, record_upstream_request, set_catalog,
set_db_pool_connections, set_mcp_active_sessions, set_mcp_session_metrics_fresh,
@@ -71,6 +72,9 @@ fn schema_names_are_unique_and_labels_are_closed() {
"state",
"signal_type",
"exporter",
"reason",
"scope_kind",
"mode",
];
for metric in schema {
assert!(
@@ -265,6 +269,9 @@ fn every_recording_api_matches_the_declared_schema() {
Duration::from_millis(1),
);
record_limit_rejection(LimitStage::Concurrency);
record_admission_rejection(RejectionReason::Capacity, ScopeKind::Global);
record_admission_queue_wait(AdmissionModeLabel::Synchronous, 5);
record_admission_oldest_waiting(Some(5));
record_cache_outcome(CacheOutcome::Hit);
record_idempotency_outcome(IdempotencyOutcome::Replay);
record_confirmation_outcome(ConfirmationOutcome::Required);
+8 -8
View File
@@ -48,10 +48,10 @@ fn schema_v1_is_complete_deterministic_and_inside_the_frozen_budget() {
#[test]
fn frozen_budget_edges_are_exact() {
assert!(validate_budget(6_284, 19_514).is_ok());
assert!(validate_budget(6_285, 19_515).is_ok());
assert!(validate_budget(6_286, 16_000).is_err());
assert!(validate_budget(6_000, 19_516).is_err());
assert!(validate_budget(6_380, 19_610).is_ok());
assert!(validate_budget(6_381, 19_611).is_ok());
assert!(validate_budget(6_382, 19_000).is_err());
assert!(validate_budget(6_000, 19_612).is_err());
assert_eq!(
crank_metrics::max_exemplar_slots(MetricService::AdminApi),
12_348
@@ -79,15 +79,15 @@ fn canonical_schema_validation_reports_its_category() {
assert_eq!(
schema_budget(MetricService::AdminApi),
Ok(crank_metrics::SchemaBudget {
logical_series: 6_285,
rendered_series: 19_515
logical_series: 6_381,
rendered_series: 19_611
})
);
assert_eq!(
schema_budget(MetricService::McpServer),
Ok(crank_metrics::SchemaBudget {
logical_series: 1_083,
rendered_series: 3_558
logical_series: 1_179,
rendered_series: 3_654
})
);
}
+11 -11
View File
@@ -543,22 +543,22 @@ fn root_budgets_are_per_process_only() {
"root обязан объявлять ровно один ключ budgets"
);
assert!(rendered.ends_with(
",\"budgets\":[{\"process\":\"admin_api\",\"families\":15,\"logical_series\":6285,\
\"rendered_series\":19515,\"exemplar_slots\":12348},{\"process\":\"mcp_server\",\
\"families\":23,\"logical_series\":1083,\"rendered_series\":3558,\"exemplar_slots\":2310}]}\n"
",\"budgets\":[{\"process\":\"admin_api\",\"families\":17,\"logical_series\":6381,\
\"rendered_series\":19611,\"exemplar_slots\":12348},{\"process\":\"mcp_server\",\
\"families\":25,\"logical_series\":1179,\"rendered_series\":3654,\"exemplar_slots\":2310}]}\n"
));
assert_eq!(
schema_budget(MetricService::AdminApi),
Ok(crank_metrics::SchemaBudget {
logical_series: 6_285,
rendered_series: 19_515
logical_series: 6_381,
rendered_series: 19_611
})
);
assert_eq!(
schema_budget(MetricService::McpServer),
Ok(crank_metrics::SchemaBudget {
logical_series: 1_083,
rendered_series: 3_558
logical_series: 1_179,
rendered_series: 3_654
})
);
}
@@ -566,11 +566,11 @@ fn root_budgets_are_per_process_only() {
/// TD-M3 — frozen caps и границы `validate_exemplar_budget` точны.
#[test]
fn frozen_caps_and_edges_are_exact() {
assert_eq!(MAX_LOGICAL_SERIES_PER_PROCESS, 6_285);
assert_eq!(MAX_RENDERED_SERIES_PER_PROCESS, 19_515);
assert_eq!(MAX_LOGICAL_SERIES_PER_PROCESS, 6_381);
assert_eq!(MAX_RENDERED_SERIES_PER_PROCESS, 19_611);
assert_eq!(MAX_EXEMPLAR_SLOTS_PER_PROCESS, 12_348);
assert!(validate_budget(6_285, 19_515).is_ok());
assert!(validate_budget(6_286, 19_515).is_err());
assert!(validate_budget(6_381, 19_611).is_ok());
assert!(validate_budget(6_382, 19_611).is_err());
assert!(validate_exemplar_budget(12_347).is_ok());
assert!(validate_exemplar_budget(12_348).is_ok());
assert!(validate_exemplar_budget(12_349).is_err());
+1 -1
View File
@@ -13,7 +13,7 @@ mod schema;
pub use config::{ObservabilityConfig, ObservabilityConfigError, ServiceIdentity};
pub use crank_metrics::{
DURATION_BUCKETS_SECONDS, MetricDefinition, MetricKind, MetricService, MetricUnit,
bind_metric_service, bound_metric_service, metric_schema,
bind_metric_service, bound_metric_service, metric_schema, record_admission_oldest_waiting,
};
pub use error_reporting::{
CriticalErrorCategory, SentryConfig, SentryConfigError, capture_critical_error,
+1
View File
@@ -7,6 +7,7 @@ publish.workspace = true
version.workspace = true
[dependencies]
async-trait = "0.1"
crank-artifacts = { path = "../crank-artifacts" }
crank-core = { path = "../crank-core" }
crank-mapping = { path = "../crank-mapping" }
+3 -2
View File
@@ -116,6 +116,7 @@ pub use model::{
YamlImportJob, YamlImportJobCompletion, YamlImportJobId, YamlImportJobStatus,
};
pub use postgres::{
AuthorizedResourceRead, BoundResourceAdmission, PostgresPoolConfig, PostgresPoolConfigError,
PostgresRegistry,
AdmissionLimits, AuthorizedResourceRead, BoundResourceAdmission, PostgresExecutionAdmission,
PostgresPoolConfig, PostgresPoolConfigError, PostgresRegistry, PostgresTaskStore,
admission_health, admission_snapshot, tasks_health,
};
+10
View File
@@ -6,6 +6,11 @@ mod artifact_cleanup_indexes_v13;
mod artifact_metadata_v12;
mod authority;
mod baseline_v1;
mod durable_tasks_v25;
mod durable_tasks_v26;
mod durable_tasks_v27;
mod durable_tasks_v28;
mod execution_admission_v24;
mod execution_outcome_v5;
mod import_plan_v14;
mod master_key_identity_v7;
@@ -35,6 +40,11 @@ mod schema_guard_v19;
mod schema_guard_v20;
mod schema_guard_v22;
mod schema_guard_v23;
mod schema_guard_v24;
mod schema_guard_v25;
mod schema_guard_v26;
mod schema_guard_v27;
mod schema_guard_v28;
mod schema_guard_v7;
mod schema_guard_v8;
mod schema_guard_v9;
@@ -4,6 +4,11 @@ use super::agent_resource_bindings_v18;
use super::approval_side_effects_v10;
use super::artifact_cleanup_indexes_v13;
use super::artifact_metadata_v12;
use super::durable_tasks_v25;
use super::durable_tasks_v26;
use super::durable_tasks_v27;
use super::durable_tasks_v28;
use super::execution_admission_v24;
use super::execution_outcome_v5;
use super::import_plan_v14;
use super::master_key_identity_v7;
@@ -39,9 +44,10 @@ use preflight::*;
use contract::baseline_source_digest;
use contract::validate_descriptors;
const MIGRATION_LOCK_ID: i64 = 0x4352_414E_4B4D_4947;
const CURRENT_VERSION: i64 = 23;
const CURRENT_VERSION: i64 = 28;
const IMPLEMENTED_VERSIONS: &[i64] = &[
1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23,
1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26,
27, 28,
];
const BASELINE_SOURCE_SHA256: &str =
"eb1656fc5b4b5be9ee390d237d1d58e4b2274ae5ba9b7ba06a2f3f860dfda675";
@@ -330,6 +336,36 @@ impl MigrationAuthority {
prompt_dependencies_v23::SOURCE_SHA256,
22,
),
expand_descriptor(
24,
execution_admission_v24::NAME,
execution_admission_v24::SOURCE_SHA256,
23,
),
expand_descriptor(
25,
durable_tasks_v25::NAME,
durable_tasks_v25::SOURCE_SHA256,
24,
),
expand_descriptor(
26,
durable_tasks_v26::NAME,
durable_tasks_v26::SOURCE_SHA256,
25,
),
expand_descriptor(
27,
durable_tasks_v27::NAME,
durable_tasks_v27::SOURCE_SHA256,
26,
),
expand_descriptor(
28,
durable_tasks_v28::NAME,
durable_tasks_v28::SOURCE_SHA256,
27,
),
]
}
pub fn validate_sequence() -> Result<(), MigrationError> {
@@ -489,6 +525,21 @@ impl MigrationAuthority {
if from < 23 {
prompt_dependencies_v23::apply(&mut transaction, &Self::sequence()[22]).await?;
}
if from < 24 {
execution_admission_v24::apply(&mut transaction, &Self::sequence()[23]).await?;
}
if from < 25 {
durable_tasks_v25::apply(&mut transaction, &Self::sequence()[24]).await?;
}
if from < 26 {
durable_tasks_v26::apply(&mut transaction, &Self::sequence()[25]).await?;
}
if from < 27 {
durable_tasks_v27::apply(&mut transaction, &Self::sequence()[26]).await?;
}
if from < 28 {
durable_tasks_v28::apply(&mut transaction, &Self::sequence()[27]).await?;
}
match inspect(&mut transaction).await? {
MigrationPreflight::Current {
version: CURRENT_VERSION,
@@ -140,6 +140,8 @@ pub(super) fn validate_descriptors(
!= agent_resource_bindings_v18::SOURCE_SHA256
|| sha256_hex(resource_security_revocations_v19::SOURCE.as_bytes())
!= resource_security_revocations_v19::SOURCE_SHA256
|| sha256_hex(execution_admission_v24::SOURCE.as_bytes())
!= execution_admission_v24::SOURCE_SHA256
{
return Err(MigrationError::new(
"invalid_contract",
@@ -148,6 +148,7 @@ pub(super) async fn inspect(
owned_relations::PROMPTS,
owned_relations::PROMPT_PUBLICATIONS,
owned_relations::PROMPT_DEPENDENCIES,
owned_relations::EXECUTION_ADMISSION,
] {
validate_required_relations(connection, relations, CURRENT_VERSION).await?;
}
@@ -0,0 +1,45 @@
use sqlx::{Postgres, Transaction, query};
use super::authority::{MigrationDescriptor, MigrationError};
pub(super) const NAME: &str = "durable-tasks-v25";
pub(super) const SOURCE: &str = include_str!("durable_tasks_v25.sql");
pub(super) const SOURCE_SHA256: &str =
"ada48bfb7d7130c84ab38217795608cd368ccb28af66faafbdfc4fc7609f67cd";
pub(super) async fn apply(
transaction: &mut Transaction<'_, Postgres>,
descriptor: &MigrationDescriptor,
) -> Result<(), MigrationError> {
sqlx::raw_sql(SOURCE)
.execute(&mut **transaction)
.await
.map_err(|_| {
MigrationError::new(
"apply_failed",
"apply.durable_tasks_v25",
Some(25),
"restore_known_good_backup",
)
})?;
query(
"insert into __crank_migrations (version, name, checksum, phase, compatibility)
values ($1, $2, $3, $4, $5)",
)
.bind(descriptor.version)
.bind(descriptor.name)
.bind(&descriptor.checksum)
.bind(descriptor.phase)
.bind(descriptor.compatibility)
.execute(&mut **transaction)
.await
.map_err(|_| {
MigrationError::new(
"apply_failed",
"apply.canonical_ledger",
Some(25),
"restore_known_good_backup",
)
})?;
Ok(())
}
@@ -0,0 +1,90 @@
-- V25: durable background Tasks (Story 4.2, FR-22, FR-23).
--
-- Одна строка `tasks` — принятая durable работа. Идентификатор Task (UUIDv7), owner scope,
-- authorization scope, запрошенный Tool и точная published-версия, canonical fingerprint входа,
-- support IDs (Request/Trace/parent TraceContext независимо от sampling) и bounded envelopes
-- закрепляются в той же PostgreSQL transaction, что и dispatch intent
-- (`execution_admission_entries.task_id`) и его queue slot. До commit наружу не выдаётся ни ID, ни ack.
--
-- `scoped_key` — digest caller-ключа по полному (workspace, owner kind/id, authorization scope,
-- operation) scope; UNIQUE по этому набору и есть защита идемпотентности. Повтор того же ключа с тем
-- же canonical fingerprint возвращает сохранённую Task, другой fingerprint — typed conflict.
--
-- `retain_until` — срок хранения записи, а не execution deadline: пока запись в retention, replay
-- находит тот же Task. Удаление terminal-записей возможно только после retain_until; queued/working/
-- input_required/outcome_unknown не удаляются никогда ради лимита.
--
-- Строка не хранит credentials, bearer/prepared headers и confirmation tokens: секретные поля входа
-- редактируются до записи, а approval хранится безопасной ссылкой.
create table tasks (
task_id uuid not null,
workspace_id text not null,
owner_kind text not null,
owner_id text not null,
authorization_scope text not null,
operation_id text not null,
operation_version integer not null,
scoped_key text not null,
input_fingerprint text not null,
request_id text not null,
trace_id text not null,
traceparent text not null,
trace_sampled boolean not null,
policy_mode text not null,
approval_reference text,
state text not null,
version integer not null default 1,
attempts integer not null default 0,
input_envelope jsonb not null,
progress_envelope jsonb,
result_envelope jsonb,
error_envelope jsonb,
execution_deadline timestamptz not null,
retain_until timestamptz not null,
created_at timestamptz not null default now(),
updated_at timestamptz not null default now(),
constraint tasks_pkey primary key (task_id),
constraint tasks_workspace_check check (octet_length(workspace_id) between 1 and 160),
constraint tasks_owner_kind_check check (owner_kind in ('agent', 'admin')),
constraint tasks_owner_check check (octet_length(owner_id) between 1 and 160),
constraint tasks_authorization_scope_check check (
octet_length(authorization_scope) between 1 and 160
),
constraint tasks_operation_check check (octet_length(operation_id) between 1 and 160),
constraint tasks_operation_version_check check (operation_version >= 1),
constraint tasks_scoped_key_check check (octet_length(scoped_key) between 16 and 128),
constraint tasks_fingerprint_check check (octet_length(input_fingerprint) = 64),
constraint tasks_request_check check (octet_length(request_id) between 1 and 128),
constraint tasks_trace_check check (octet_length(trace_id) = 32),
constraint tasks_traceparent_check check (octet_length(traceparent) = 55),
constraint tasks_policy_mode_check check (
policy_mode in ('synchronous', 'background', 'adaptive')
),
constraint tasks_state_check check (
state in ('queued', 'working', 'input_required', 'outcome_unknown',
'completed', 'failed', 'cancelled')
),
constraint tasks_lifecycle_check check (version >= 1 and attempts >= 0),
constraint tasks_retention_check check (retain_until >= execution_deadline),
constraint tasks_approval_check check (
approval_reference is null or octet_length(approval_reference) between 1 and 160
)
);
create unique index tasks_scoped_key_uidx
on tasks (workspace_id, owner_kind, owner_id, authorization_scope, operation_id, scoped_key);
create index tasks_retention_idx on tasks (state, retain_until, task_id);
create index tasks_deadline_idx on tasks (state, execution_deadline, task_id);
-- Dispatch intent: прежняя reserved-строка admission становится точкой связи Task и capacity.
-- FK без cascade: удаление принятой Task не маскируется housekeeping-удалением intent.
alter table execution_admission_entries
add column task_id uuid,
add constraint execution_admission_entries_task_fkey
foreign key (task_id) references tasks (task_id);
create unique index execution_admission_entries_task_uidx
on execution_admission_entries (task_id) where task_id is not null;
@@ -0,0 +1,45 @@
use sqlx::{Postgres, Transaction, query};
use super::authority::{MigrationDescriptor, MigrationError};
pub(super) const NAME: &str = "durable-tasks-v26";
pub(super) const SOURCE: &str = include_str!("durable_tasks_v26.sql");
pub(super) const SOURCE_SHA256: &str =
"6dcbf9990deabb3984a333acbfc3cb490cfc886c2ec30093e3b46d9113616089";
pub(super) async fn apply(
transaction: &mut Transaction<'_, Postgres>,
descriptor: &MigrationDescriptor,
) -> Result<(), MigrationError> {
sqlx::raw_sql(SOURCE)
.execute(&mut **transaction)
.await
.map_err(|_| {
MigrationError::new(
"apply_failed",
"apply.durable_tasks_v26",
Some(26),
"restore_known_good_backup",
)
})?;
query(
"insert into __crank_migrations (version, name, checksum, phase, compatibility)
values ($1, $2, $3, $4, $5)",
)
.bind(descriptor.version)
.bind(descriptor.name)
.bind(&descriptor.checksum)
.bind(descriptor.phase)
.bind(descriptor.compatibility)
.execute(&mut **transaction)
.await
.map_err(|_| {
MigrationError::new(
"apply_failed",
"apply.durable_tasks_v26",
Some(26),
"restore_known_good_backup",
)
})?;
Ok(())
}
@@ -0,0 +1,6 @@
-- V26: PostgreSQL lease/fencing/CAS additions for durable background Tasks (Story 4.3).
alter table tasks add column if not exists fence bigint not null default 0;
alter table tasks add column if not exists lease_owner text;
alter table tasks add column if not exists lease_until timestamptz;
alter table tasks add column if not exists current_attempt_id text;
alter table tasks add column if not exists dispatch_started_at timestamptz;
@@ -0,0 +1,45 @@
use sqlx::{Postgres, Transaction, query};
use super::authority::{MigrationDescriptor, MigrationError};
pub(super) const NAME: &str = "durable-tasks-v27";
pub(super) const SOURCE: &str = include_str!("durable_tasks_v27.sql");
pub(super) const SOURCE_SHA256: &str =
"a3c5335cbf81c244359faaee993487ae132d8b3be6b9c63fa322788021273dc8";
pub(super) async fn apply(
transaction: &mut Transaction<'_, Postgres>,
descriptor: &MigrationDescriptor,
) -> Result<(), MigrationError> {
sqlx::raw_sql(SOURCE)
.execute(&mut **transaction)
.await
.map_err(|_| {
MigrationError::new(
"apply_failed",
"apply.durable_tasks_v27",
Some(27),
"restore_known_good_backup",
)
})?;
query(
"insert into __crank_migrations (version, name, checksum, phase, compatibility)
values ($1, $2, $3, $4, $5)",
)
.bind(descriptor.version)
.bind(descriptor.name)
.bind(&descriptor.checksum)
.bind(descriptor.phase)
.bind(descriptor.compatibility)
.execute(&mut **transaction)
.await
.map_err(|_| {
MigrationError::new(
"apply_failed",
"apply.durable_tasks_v27",
Some(27),
"restore_known_good_backup",
)
})?;
Ok(())
}
@@ -0,0 +1,19 @@
-- V27: encrypted executable Task input (Story 4.3).
alter table tasks add column if not exists executable_input_ciphertext text;
alter table tasks add column if not exists executable_input_key_version text;
alter table tasks add column if not exists executable_input_master_key_epoch bigint;
alter table tasks add constraint tasks_executable_input_key_version_check check (
executable_input_ciphertext is null
or (executable_input_key_version is not null and octet_length(executable_input_key_version) between 1 and 64)
);
alter table tasks add column if not exists target_executable_input_ciphertext text;
alter table tasks add column if not exists target_executable_input_key_version text;
alter table tasks add column if not exists target_executable_input_master_key_epoch bigint;
alter table tasks add constraint tasks_executable_input_epoch_check check (
executable_input_ciphertext is null
or (executable_input_master_key_epoch is not null
and executable_input_master_key_epoch >= 1)
);
@@ -0,0 +1,45 @@
use sqlx::{Postgres, Transaction, query};
use super::authority::{MigrationDescriptor, MigrationError};
pub(super) const NAME: &str = "durable-tasks-v28";
pub(super) const SOURCE: &str = include_str!("durable_tasks_v28.sql");
pub(super) const SOURCE_SHA256: &str =
"4dc77739c2ea1e32a4b00abd70df0f00fd82b7cf7933372f869f2c1e997a3676";
pub(super) async fn apply(
transaction: &mut Transaction<'_, Postgres>,
descriptor: &MigrationDescriptor,
) -> Result<(), MigrationError> {
sqlx::raw_sql(SOURCE)
.execute(&mut **transaction)
.await
.map_err(|_| {
MigrationError::new(
"apply_failed",
"apply.durable_tasks_v28",
Some(28),
"restore_known_good_backup",
)
})?;
query(
"insert into __crank_migrations (version, name, checksum, phase, compatibility)
values ($1, $2, $3, $4, $5)",
)
.bind(descriptor.version)
.bind(descriptor.name)
.bind(&descriptor.checksum)
.bind(descriptor.phase)
.bind(descriptor.compatibility)
.execute(&mut **transaction)
.await
.map_err(|_| {
MigrationError::new(
"apply_failed",
"apply.durable_tasks_v28",
Some(28),
"restore_known_good_backup",
)
})?;
Ok(())
}
@@ -0,0 +1,8 @@
-- V28: typed Platform API key authority reference of a durable Task (Story 4.3).
-- Only the immutable key identifier is stored, never the bearer secret.
alter table tasks add column if not exists platform_api_key_id text;
alter table tasks add constraint tasks_platform_api_key_id_check check (
platform_api_key_id is null
or octet_length(platform_api_key_id) between 1 and 160
);
@@ -0,0 +1,45 @@
use sqlx::{Postgres, Transaction, query};
use super::authority::{MigrationDescriptor, MigrationError};
pub(super) const NAME: &str = "execution-admission-v24";
pub(super) const SOURCE: &str = include_str!("execution_admission_v24.sql");
pub(super) const SOURCE_SHA256: &str =
"efe021ebcd68bbed64edaa1e90c3a851f7b167b94ba65d88f525d58160f02681";
pub(super) async fn apply(
transaction: &mut Transaction<'_, Postgres>,
descriptor: &MigrationDescriptor,
) -> Result<(), MigrationError> {
sqlx::raw_sql(SOURCE)
.execute(&mut **transaction)
.await
.map_err(|_| {
MigrationError::new(
"apply_failed",
"apply.execution_admission_v24",
Some(24),
"restore_known_good_backup",
)
})?;
query(
"insert into __crank_migrations (version, name, checksum, phase, compatibility)
values ($1, $2, $3, $4, $5)",
)
.bind(descriptor.version)
.bind(descriptor.name)
.bind(&descriptor.checksum)
.bind(descriptor.phase)
.bind(descriptor.compatibility)
.execute(&mut **transaction)
.await
.map_err(|_| {
MigrationError::new(
"apply_failed",
"apply.canonical_ledger",
Some(24),
"restore_known_good_backup",
)
})?;
Ok(())
}
@@ -0,0 +1,92 @@
-- V24: authoritative execution admission state (Story 4.1, FR-21, FR-24).
--
-- Один deployment-wide admission: PostgreSQL хранит cursor/fairness state, ожидающие тикеты,
-- выданные permits и durable enqueue decisions, поэтому несколько процессов делят общий бюджет.
--
-- `execution_admission_entries` — bounded набор: waiting (ожидает capacity), executing (scoped
-- permit выдан, держится до явного release) и reserved (durable enqueue decision с зарезервированным
-- queue slot; не исполняется и не рекламируется как Task до Story 4.2).
--
-- Строка не хранит ни credentials, ни connection/session id, ни prepared headers, ни тело запроса:
-- quota key — (workspace, agent) и (workspace, operation) без version. `operation_version` пишется
-- только как pin исполняемых semantics и не участвует в квотах.
--
-- `expires_at` — абсолютный deadline. Queue decisions без dispatch можно expire по deadline; активный
-- permit НЕ освобождается по TTL (иначе paused/crashed owner обойдёт global limit): его снимает только
-- явный release/cancel владельца.
create table execution_admission_controllers (
workspace_id text not null,
cursor_agent text,
cursor_agent_credit integer not null default 0,
cursor_tool text,
cursor_tool_credit integer not null default 0,
updated_at timestamptz not null default now(),
constraint execution_admission_controllers_pkey primary key (workspace_id),
constraint execution_admission_controllers_workspace_check check (
octet_length(workspace_id) between 1 and 160
),
constraint execution_admission_controllers_agent_credit_check check (
cursor_agent_credit between 0 and 64
),
constraint execution_admission_controllers_tool_credit_check check (
cursor_tool_credit between 0 and 64
)
);
create table execution_admission_entries (
ticket_id bigint generated always as identity,
workspace_id text not null,
agent_key text not null,
operation_id text not null,
operation_version integer not null,
mode text not null,
policy_mode text not null,
state text not null,
owner_token text not null,
fingerprint text not null,
created_at timestamptz not null default now(),
expires_at timestamptz not null,
granted_at timestamptz,
constraint execution_admission_entries_pkey primary key (ticket_id),
constraint execution_admission_entries_workspace_check check (
octet_length(workspace_id) between 1 and 160
),
constraint execution_admission_entries_agent_check check (
octet_length(agent_key) between 1 and 160
),
constraint execution_admission_entries_operation_check check (
octet_length(operation_id) between 1 and 160
),
constraint execution_admission_entries_version_check check (operation_version >= 1),
constraint execution_admission_entries_mode_check check (
mode in ('synchronous', 'background')
),
constraint execution_admission_entries_policy_mode_check check (
policy_mode in ('synchronous', 'background', 'adaptive')
),
constraint execution_admission_entries_state_check check (
state in ('waiting', 'executing', 'reserved')
),
constraint execution_admission_entries_owner_check check (
octet_length(owner_token) between 1 and 64
),
constraint execution_admission_entries_fingerprint_check check (
octet_length(fingerprint) between 1 and 128
),
constraint execution_admission_entries_reserved_mode_check check (
state <> 'reserved' or mode = 'background'
)
);
create unique index execution_admission_entries_owner_uidx
on execution_admission_entries (owner_token);
create index execution_admission_entries_state_idx
on execution_admission_entries (state, workspace_id, ticket_id);
create index execution_admission_entries_agent_idx
on execution_admission_entries (state, workspace_id, agent_key);
create index execution_admission_entries_operation_idx
on execution_admission_entries (state, workspace_id, operation_id);
@@ -55,6 +55,11 @@ pub(super) const PROMPT_PUBLICATIONS: &[&str] = &["prompt_publications", "agent_
pub(super) const PROMPT_DEPENDENCIES: &[&str] =
&["prompt_dependencies", "prompt_message_resource_refs"];
pub(super) const EXECUTION_ADMISSION: &[&str] = &[
"execution_admission_controllers",
"execution_admission_entries",
];
pub(super) const ARTIFACT_METADATA: &[&str] = &["artifact_blobs", "artifact_sources"];
pub(super) const IMPORT_PLAN: &[&str] = &["import_plans", "import_plan_applications"];
@@ -468,6 +468,53 @@ pub(super) async fn validate_schema_fingerprint(
} else {
super::schema_guard_v23::validate_v23_prompt_dependencies(connection).await?;
}
if current_version < 24 {
if !super::schema_guard_v24::validate_v24_absent(connection).await? {
return Err(schema_error(current_version));
}
} else {
super::schema_guard_v24::validate_v24_execution_admission(connection, current_version)
.await?;
}
if current_version < 25 {
if !super::schema_guard_v25::validate_v25_absent(connection).await? {
return Err(schema_error(current_version));
}
if !super::schema_guard_v26::validate_v26_absent(connection).await? {
return Err(schema_error(current_version));
}
if !super::schema_guard_v27::validate_v27_absent(connection).await? {
return Err(schema_error(current_version));
}
if !super::schema_guard_v28::validate_v28_absent(connection).await? {
return Err(schema_error(current_version));
}
} else if current_version < 26 {
super::schema_guard_v25::validate_v25_durable_tasks(connection).await?;
if !super::schema_guard_v26::validate_v26_absent(connection).await? {
return Err(schema_error(current_version));
}
if !super::schema_guard_v28::validate_v28_absent(connection).await? {
return Err(schema_error(current_version));
}
} else if current_version < 27 {
super::schema_guard_v26::validate_v26_durable_tasks(connection).await?;
if !super::schema_guard_v27::validate_v27_absent(connection).await? {
return Err(schema_error(current_version));
}
if !super::schema_guard_v28::validate_v28_absent(connection).await? {
return Err(schema_error(current_version));
}
} else if current_version < 28 {
super::schema_guard_v26::validate_v26_base_columns(connection).await?;
super::schema_guard_v27::validate_v27_durable_tasks(connection).await?;
if !super::schema_guard_v28::validate_v28_absent(connection).await? {
return Err(schema_error(current_version));
}
} else {
super::schema_guard_v26::validate_v26_base_columns(connection).await?;
super::schema_guard_v28::validate_v28_durable_tasks(connection).await?;
}
Ok(())
}
@@ -51,6 +51,8 @@ pub(crate) const OWNED_RELATIONS: &[&str] = &[
"prompt_messages",
"prompt_dependencies",
"prompt_message_resource_refs",
"execution_admission_controllers",
"execution_admission_entries",
];
pub(crate) const REQUIRED_COLUMNS: &[(&str, &[&str])] = &[
@@ -0,0 +1,277 @@
//! V24 schema guard: authoritative execution admission state (Story 4.1, FR-21/FR-24).
//!
//! Схема V24 либо полностью отсутствует (миграция ещё не применена), либо присутствует ровно в
//! ожидаемой форме: controller-row с weighted round-robin cursor и bounded entries с закрытым
//! словарём mode/policy_mode/state, уникальным owner token и индексами квот.
use sqlx::{PgConnection, Row, query};
use super::authority::MigrationError;
use super::schema_guard::relation_exists;
const CONTROLLERS: &str = "execution_admission_controllers";
const ENTRIES: &str = "execution_admission_entries";
const OWNED_RELATIONS: &[&str] = &[CONTROLLERS, ENTRIES];
const CONTROLLER_COLUMNS: &[(&str, &str, bool)] = &[
("workspace_id", "text", false),
("cursor_agent", "text", true),
("cursor_agent_credit", "integer", false),
("cursor_tool", "text", true),
("cursor_tool_credit", "integer", false),
("updated_at", "timestamp with time zone", false),
];
const ENTRY_COLUMNS: &[(&str, &str, bool)] = &[
("ticket_id", "bigint", false),
("workspace_id", "text", false),
("agent_key", "text", false),
("operation_id", "text", false),
("operation_version", "integer", false),
("mode", "text", false),
("policy_mode", "text", false),
("state", "text", false),
("owner_token", "text", false),
("fingerprint", "text", false),
("created_at", "timestamp with time zone", false),
("expires_at", "timestamp with time zone", false),
("granted_at", "timestamp with time zone", true),
];
/// V25 adds the nullable Task dispatch-intent link to the same reserved row. At version 24 the
/// column must not exist yet, so the guard validates the shape of the version it describes.
fn entry_columns(current_version: i64) -> Vec<(&'static str, &'static str, bool)> {
let mut columns = ENTRY_COLUMNS.to_vec();
if current_version >= 25 {
columns.push(("task_id", "uuid", true));
}
columns
}
fn constraint_count(table: &str, current_version: i64) -> Option<usize> {
match table {
CONTROLLERS => Some(4),
ENTRIES if current_version >= 25 => Some(12),
ENTRIES => Some(11),
_ => None,
}
}
const UNIQUE_INDEXES: &[&str] = &["execution_admission_entries_owner_uidx"];
fn v24_error(stage: &'static str) -> MigrationError {
MigrationError::new(
"partial_sequence",
stage,
Some(24),
"restore_known_good_backup",
)
}
/// Схема V24 отсутствует целиком (для current_version < 24).
pub(super) async fn validate_v24_absent(
connection: &mut PgConnection,
) -> Result<bool, MigrationError> {
for relation in OWNED_RELATIONS {
if relation_exists(connection, relation).await? {
return Ok(false);
}
}
for index in UNIQUE_INDEXES {
if index_exists(connection, index).await? {
return Ok(false);
}
}
Ok(true)
}
/// Схема V24 присутствует в ожидаемой форме (для current_version >= 24).
pub(super) async fn validate_v24_execution_admission(
connection: &mut PgConnection,
current_version: i64,
) -> Result<(), MigrationError> {
for relation in OWNED_RELATIONS {
if !relation_exists(connection, relation).await? {
return Err(v24_error("v24.relations"));
}
}
let tables = [
(CONTROLLERS, CONTROLLER_COLUMNS.to_vec()),
(ENTRIES, entry_columns(current_version)),
];
for (table, columns) in tables {
validate_columns(connection, table, &columns).await?;
}
for table in [CONTROLLERS, ENTRIES] {
let expected = constraint_count(table, current_version).unwrap_or_default();
let count: i64 = query(
"select count(*)::bigint as count
from pg_catalog.pg_constraint c
join pg_catalog.pg_class t on t.oid = c.conrelid
join pg_catalog.pg_namespace n on n.oid = t.relnamespace
where n.nspname = current_schema() and t.relname = $1",
)
.bind(table)
.fetch_one(&mut *connection)
.await
.map_err(|_| MigrationError::storage("preflight.schema"))?
.try_get("count")
.map_err(|_| MigrationError::storage("preflight.schema"))?;
if count != i64::try_from(expected).unwrap_or_default() {
return Err(v24_error("v24.constraints"));
}
}
for (table, constraint) in [
(CONTROLLERS, "execution_admission_controllers_pkey"),
(ENTRIES, "execution_admission_entries_pkey"),
(ENTRIES, "execution_admission_entries_state_check"),
] {
if !constraint_exists(connection, table, constraint).await? {
return Err(v24_error("v24.constraint_missing"));
}
}
if !normalized_expression(
connection,
ENTRIES,
"execution_admission_entries_state_check",
)
.await?
.is_some_and(|expression| {
super::schema_guard::enum_constraint_matches(
&expression,
"state",
&["waiting", "executing", "reserved"],
)
}) {
return Err(v24_error("v24.state_vocabulary"));
}
for index in UNIQUE_INDEXES {
let unique = index_unique(connection, index).await?;
if unique != Some(true) {
return Err(v24_error("v24.indexes"));
}
}
Ok(())
}
async fn validate_columns(
connection: &mut PgConnection,
table: &str,
columns: &[(&str, &str, bool)],
) -> Result<(), MigrationError> {
let count: i64 = query(
"select count(*)::bigint as count from information_schema.columns
where table_schema = current_schema() and table_name = $1",
)
.bind(table)
.fetch_one(&mut *connection)
.await
.map_err(|_| MigrationError::storage("preflight.schema"))?
.try_get("count")
.map_err(|_| MigrationError::storage("preflight.schema"))?;
if count != i64::try_from(columns.len()).unwrap_or_default() {
return Err(v24_error("v24.columns"));
}
for (column, data_type, nullable) in columns {
let row = query(
"select data_type, is_nullable from information_schema.columns
where table_schema = current_schema() and table_name = $1 and column_name = $2",
)
.bind(table)
.bind(column)
.fetch_optional(&mut *connection)
.await
.map_err(|_| MigrationError::storage("preflight.schema"))?;
let valid = row.is_some_and(|row| {
row.try_get::<String, _>("data_type").ok().as_deref() == Some(*data_type)
&& row.try_get::<String, _>("is_nullable").ok().as_deref()
== Some(if *nullable { "YES" } else { "NO" })
});
if !valid {
return Err(v24_error("v24.columns"));
}
}
Ok(())
}
async fn constraint_exists(
connection: &mut PgConnection,
table: &str,
constraint: &str,
) -> Result<bool, MigrationError> {
query(
"select exists (
select 1 from pg_catalog.pg_constraint c
join pg_catalog.pg_class t on t.oid = c.conrelid
join pg_catalog.pg_namespace n on n.oid = t.relnamespace
where n.nspname = current_schema() and t.relname = $1 and c.conname = $2
) as present",
)
.bind(table)
.bind(constraint)
.fetch_one(&mut *connection)
.await
.map_err(|_| MigrationError::storage("preflight.schema"))?
.try_get::<bool, _>("present")
.map_err(|_| MigrationError::storage("preflight.schema"))
}
async fn normalized_expression(
connection: &mut PgConnection,
table: &str,
constraint: &str,
) -> Result<Option<String>, MigrationError> {
let row = query(
"select pg_get_expr(c.conbin, c.conrelid) as expression
from pg_catalog.pg_constraint c
join pg_catalog.pg_class t on t.oid = c.conrelid
join pg_catalog.pg_namespace n on n.oid = t.relnamespace
where n.nspname = current_schema() and t.relname = $1 and c.conname = $2
and c.contype = 'c'",
)
.bind(table)
.bind(constraint)
.fetch_optional(&mut *connection)
.await
.map_err(|_| MigrationError::storage("preflight.schema"))?;
Ok(row.and_then(|row| row.try_get::<String, _>("expression").ok()))
}
async fn index_exists(connection: &mut PgConnection, index: &str) -> Result<bool, MigrationError> {
query(
"select exists (
select 1 from pg_catalog.pg_class idx
join pg_catalog.pg_namespace n on n.oid = idx.relnamespace
where n.nspname = current_schema() and idx.relname = $1
) as present",
)
.bind(index)
.fetch_one(&mut *connection)
.await
.map_err(|_| MigrationError::storage("preflight.schema"))?
.try_get::<bool, _>("present")
.map_err(|_| MigrationError::storage("preflight.schema"))
}
async fn index_unique(
connection: &mut PgConnection,
index: &str,
) -> Result<Option<bool>, MigrationError> {
let row = query(
"select i.indisunique, i.indisvalid
from pg_catalog.pg_index i
join pg_catalog.pg_class idx on idx.oid = i.indexrelid
join pg_catalog.pg_namespace n on n.oid = idx.relnamespace
where n.nspname = current_schema() and idx.relname = $1",
)
.bind(index)
.fetch_optional(&mut *connection)
.await
.map_err(|_| MigrationError::storage("preflight.schema"))?;
Ok(row.and_then(|row| {
let unique = row.try_get::<bool, _>("indisunique").ok()?;
let valid = row.try_get::<bool, _>("indisvalid").ok()?;
Some(unique && valid)
}))
}
@@ -0,0 +1,247 @@
//! V25 schema guard: durable background Tasks (Story 4.2, FR-22/FR-23).
//!
//! Схема V25 либо полностью отсутствует (миграция ещё не применена), либо присутствует ровно в
//! ожидаемой форме: таблица `tasks` с закрытым словарём state/policy_mode и scoped uniqueness, плюс
//! nullable `task_id` на существующей reserved-строке admission с FK и UNIQUE-индексом.
use sqlx::{PgConnection, Row, query};
use super::authority::MigrationError;
use super::schema_guard::relation_exists;
const TASKS: &str = "tasks";
const ENTRIES: &str = "execution_admission_entries";
const SCOPED_INDEX: &str = "tasks_scoped_key_uidx";
const TASK_INDEX: &str = "execution_admission_entries_task_uidx";
const TASK_FK: &str = "execution_admission_entries_task_fkey";
const TASK_COLUMNS: &[(&str, &str, bool)] = &[
("task_id", "uuid", false),
("workspace_id", "text", false),
("owner_kind", "text", false),
("owner_id", "text", false),
("authorization_scope", "text", false),
("operation_id", "text", false),
("operation_version", "integer", false),
("scoped_key", "text", false),
("input_fingerprint", "text", false),
("request_id", "text", false),
("trace_id", "text", false),
("traceparent", "text", false),
("trace_sampled", "boolean", false),
("policy_mode", "text", false),
("approval_reference", "text", true),
("state", "text", false),
("version", "integer", false),
("attempts", "integer", false),
("input_envelope", "jsonb", false),
("progress_envelope", "jsonb", true),
("result_envelope", "jsonb", true),
("error_envelope", "jsonb", true),
("execution_deadline", "timestamp with time zone", false),
("retain_until", "timestamp with time zone", false),
("created_at", "timestamp with time zone", false),
("updated_at", "timestamp with time zone", false),
];
fn v25_error(stage: &'static str) -> MigrationError {
MigrationError::new(
"partial_sequence",
stage,
Some(25),
"restore_known_good_backup",
)
}
/// Схема V25 отсутствует целиком (для current_version < 25).
pub(super) async fn validate_v25_absent(
connection: &mut PgConnection,
) -> Result<bool, MigrationError> {
if relation_exists(connection, TASKS).await? {
return Ok(false);
}
if super::schema_guard::column_exists(connection, ENTRIES, "task_id").await? {
return Ok(false);
}
for index in [SCOPED_INDEX, TASK_INDEX] {
if index_exists(connection, index).await? {
return Ok(false);
}
}
Ok(true)
}
/// Схема V25 присутствует в ожидаемой форме (для current_version >= 25).
pub(super) async fn validate_v25_durable_tasks(
connection: &mut PgConnection,
) -> Result<(), MigrationError> {
if !relation_exists(connection, TASKS).await? {
return Err(v25_error("v25.relations"));
}
let count: i64 = query(
"select count(*)::bigint as count from information_schema.columns
where table_schema = current_schema() and table_name = $1",
)
.bind(TASKS)
.fetch_one(&mut *connection)
.await
.map_err(|_| MigrationError::storage("preflight.schema"))?
.try_get("count")
.map_err(|_| MigrationError::storage("preflight.schema"))?;
if count != i64::try_from(TASK_COLUMNS.len()).unwrap_or_default() {
return Err(v25_error("v25.columns"));
}
for (column, data_type, nullable) in TASK_COLUMNS {
let row = query(
"select data_type, is_nullable from information_schema.columns
where table_schema = current_schema() and table_name = $1 and column_name = $2",
)
.bind(TASKS)
.bind(column)
.fetch_optional(&mut *connection)
.await
.map_err(|_| MigrationError::storage("preflight.schema"))?;
let valid = row.is_some_and(|row| {
row.try_get::<String, _>("data_type").ok().as_deref() == Some(*data_type)
&& row.try_get::<String, _>("is_nullable").ok().as_deref()
== Some(if *nullable { "YES" } else { "NO" })
});
if !valid {
return Err(v25_error("v25.columns"));
}
}
if !super::schema_guard::column_exists(connection, ENTRIES, "task_id").await? {
return Err(v25_error("v25.intent_column"));
}
for (constraint, vocabulary) in [
(
"tasks_state_check",
[
"queued",
"working",
"input_required",
"outcome_unknown",
"completed",
"failed",
"cancelled",
]
.as_slice(),
),
(
"tasks_policy_mode_check",
["synchronous", "background", "adaptive"].as_slice(),
),
] {
let expression = normalized_expression(connection, TASKS, constraint).await?;
if !expression.is_some_and(|expression| {
super::schema_guard::enum_constraint_matches(
&expression,
column_of(constraint),
vocabulary,
)
}) {
return Err(v25_error("v25.vocabulary"));
}
}
for (index, table) in [(SCOPED_INDEX, TASKS), (TASK_INDEX, ENTRIES)] {
if index_unique(connection, table, index).await? != Some(true) {
return Err(v25_error("v25.indexes"));
}
}
if !constraint_exists(connection, ENTRIES, TASK_FK).await? {
return Err(v25_error("v25.intent_fk"));
}
Ok(())
}
fn column_of(constraint: &str) -> &'static str {
if constraint.contains("policy_mode") {
"policy_mode"
} else {
"state"
}
}
async fn constraint_exists(
connection: &mut PgConnection,
table: &str,
constraint: &str,
) -> Result<bool, MigrationError> {
query(
"select exists (
select 1 from pg_catalog.pg_constraint c
join pg_catalog.pg_class t on t.oid = c.conrelid
join pg_catalog.pg_namespace n on n.oid = t.relnamespace
where n.nspname = current_schema() and t.relname = $1 and c.conname = $2
) as present",
)
.bind(table)
.bind(constraint)
.fetch_one(&mut *connection)
.await
.map_err(|_| MigrationError::storage("preflight.schema"))?
.try_get::<bool, _>("present")
.map_err(|_| MigrationError::storage("preflight.schema"))
}
async fn normalized_expression(
connection: &mut PgConnection,
table: &str,
constraint: &str,
) -> Result<Option<String>, MigrationError> {
let row = query(
"select pg_get_expr(c.conbin, c.conrelid) as expression
from pg_catalog.pg_constraint c
join pg_catalog.pg_class t on t.oid = c.conrelid
join pg_catalog.pg_namespace n on n.oid = t.relnamespace
where n.nspname = current_schema() and t.relname = $1 and c.conname = $2
and c.contype = 'c'",
)
.bind(table)
.bind(constraint)
.fetch_optional(&mut *connection)
.await
.map_err(|_| MigrationError::storage("preflight.schema"))?;
Ok(row.and_then(|row| row.try_get::<String, _>("expression").ok()))
}
async fn index_exists(connection: &mut PgConnection, index: &str) -> Result<bool, MigrationError> {
query(
"select exists (
select 1 from pg_catalog.pg_class idx
join pg_catalog.pg_namespace n on n.oid = idx.relnamespace
where n.nspname = current_schema() and idx.relname = $1
) as present",
)
.bind(index)
.fetch_one(&mut *connection)
.await
.map_err(|_| MigrationError::storage("preflight.schema"))?
.try_get::<bool, _>("present")
.map_err(|_| MigrationError::storage("preflight.schema"))
}
async fn index_unique(
connection: &mut PgConnection,
table: &str,
index: &str,
) -> Result<Option<bool>, MigrationError> {
let row = query(
"select i.indisunique, i.indisvalid
from pg_catalog.pg_index i
join pg_catalog.pg_class idx on idx.oid = i.indexrelid
join pg_catalog.pg_class t on t.oid = i.indrelid
join pg_catalog.pg_namespace n on n.oid = idx.relnamespace
where n.nspname = current_schema() and idx.relname = $1 and t.relname = $2",
)
.bind(index)
.bind(table)
.fetch_optional(&mut *connection)
.await
.map_err(|_| MigrationError::storage("preflight.schema"))?;
Ok(row.and_then(|row| {
let unique = row.try_get::<bool, _>("indisunique").ok()?;
let valid = row.try_get::<bool, _>("indisvalid").ok()?;
Some(unique && valid)
}))
}
@@ -0,0 +1,120 @@
//! V26 schema guard: lease/fencing columns for durable background Tasks (Story 4.3).
use sqlx::{PgConnection, Row, query};
use super::authority::MigrationError;
use super::schema_guard::relation_exists;
const TASKS: &str = "tasks";
const TASK_COLUMNS: &[(&str, &str, bool)] = &[
("task_id", "uuid", false),
("workspace_id", "text", false),
("owner_kind", "text", false),
("owner_id", "text", false),
("authorization_scope", "text", false),
("operation_id", "text", false),
("operation_version", "integer", false),
("scoped_key", "text", false),
("input_fingerprint", "text", false),
("request_id", "text", false),
("trace_id", "text", false),
("traceparent", "text", false),
("trace_sampled", "boolean", false),
("policy_mode", "text", false),
("approval_reference", "text", true),
("state", "text", false),
("version", "integer", false),
("attempts", "integer", false),
("input_envelope", "jsonb", false),
("progress_envelope", "jsonb", true),
("result_envelope", "jsonb", true),
("error_envelope", "jsonb", true),
("execution_deadline", "timestamp with time zone", false),
("retain_until", "timestamp with time zone", false),
("created_at", "timestamp with time zone", false),
("updated_at", "timestamp with time zone", false),
("fence", "bigint", false),
("lease_owner", "text", true),
("lease_until", "timestamp with time zone", true),
("current_attempt_id", "text", true),
("dispatch_started_at", "timestamp with time zone", true),
];
fn v26_error(stage: &'static str) -> MigrationError {
MigrationError::new(
"partial_sequence",
stage,
Some(26),
"restore_known_good_backup",
)
}
pub(super) async fn validate_v26_absent(
connection: &mut PgConnection,
) -> Result<bool, MigrationError> {
if relation_exists(connection, TASKS).await? {
// delegate to v25 absent for relation mostly; V26 columns must not exist.
for column in [
"fence",
"lease_owner",
"lease_until",
"current_attempt_id",
"dispatch_started_at",
] {
if super::schema_guard::column_exists(connection, TASKS, column).await? {
return Ok(false);
}
}
}
Ok(true)
}
pub(super) async fn validate_v26_base_columns(
connection: &mut PgConnection,
) -> Result<(), MigrationError> {
if !relation_exists(connection, TASKS).await? {
return Err(v26_error("v26.relations"));
}
for (column, data_type, nullable) in TASK_COLUMNS {
let row = query(
"select data_type, is_nullable from information_schema.columns
where table_schema = current_schema() and table_name = $1 and column_name = $2",
)
.bind(TASKS)
.bind(column)
.fetch_optional(&mut *connection)
.await
.map_err(|_| MigrationError::storage("preflight.schema"))?;
let valid = row.is_some_and(|row| {
row.try_get::<String, _>("data_type").ok().as_deref() == Some(*data_type)
&& row.try_get::<String, _>("is_nullable").ok().as_deref()
== Some(if *nullable { "YES" } else { "NO" })
});
if !valid {
return Err(v26_error("v26.columns"));
}
}
Ok(())
}
pub(super) async fn validate_v26_durable_tasks(
connection: &mut PgConnection,
) -> Result<(), MigrationError> {
validate_v26_base_columns(connection).await?;
let count: i64 = query(
"select count(*)::bigint as count from information_schema.columns
where table_schema = current_schema() and table_name = $1",
)
.bind(TASKS)
.fetch_one(&mut *connection)
.await
.map_err(|_| MigrationError::storage("preflight.schema"))?
.try_get("count")
.map_err(|_| MigrationError::storage("preflight.schema"))?;
if count != i64::try_from(TASK_COLUMNS.len()).unwrap_or_default() {
return Err(v26_error("v26.columns"));
}
// Constraints/index checks inherited conceptually from V25; V26 adds columns only.
Ok(())
}
@@ -0,0 +1,85 @@
//! V27 schema guard: encrypted executable Task input columns (Story 4.3).
use sqlx::{PgConnection, Row, query};
use super::authority::MigrationError;
use super::schema_guard::relation_exists;
use super::schema_guard_v26::validate_v26_base_columns;
const TASKS: &str = "tasks";
pub(super) async fn validate_v27_absent(
connection: &mut PgConnection,
) -> Result<bool, MigrationError> {
if relation_exists(connection, TASKS).await? {
for column in [
"executable_input_ciphertext",
"executable_input_key_version",
"executable_input_master_key_epoch",
] {
if super::schema_guard::column_exists(connection, TASKS, column).await? {
return Ok(false);
}
}
}
Ok(true)
}
pub(super) async fn validate_v27_durable_tasks(
connection: &mut PgConnection,
) -> Result<(), MigrationError> {
validate_v26_base_columns(connection).await?;
// Additional V27 columns.
let extra = [
("executable_input_ciphertext", "text", true),
("executable_input_key_version", "text", true),
("executable_input_master_key_epoch", "bigint", true),
("target_executable_input_ciphertext", "text", true),
("target_executable_input_key_version", "text", true),
("target_executable_input_master_key_epoch", "bigint", true),
];
for (column, data_type, nullable) in extra {
let row = query(
"select data_type, is_nullable from information_schema.columns
where table_schema = current_schema() and table_name = $1 and column_name = $2",
)
.bind(TASKS)
.bind(column)
.fetch_optional(&mut *connection)
.await
.map_err(|_| MigrationError::storage("preflight.schema"))?;
let valid = row.is_some_and(|row| {
row.try_get::<String, _>("data_type").ok().as_deref() == Some(data_type)
&& row.try_get::<String, _>("is_nullable").ok().as_deref()
== Some(if nullable { "YES" } else { "NO" })
});
if !valid {
return Err(MigrationError::new(
"partial_sequence",
"v27.columns",
Some(27),
"restore_known_good_backup",
));
}
}
// Exact total column count for version 27: 43 V26 columns + 3 V27 columns.
let count: i64 = query(
"select count(*)::bigint as count from information_schema.columns
where table_schema = current_schema() and table_name = $1",
)
.bind(TASKS)
.fetch_one(&mut *connection)
.await
.map_err(|_| MigrationError::storage("preflight.schema"))?
.try_get("count")
.map_err(|_| MigrationError::storage("preflight.schema"))?;
if count != i64::try_from(37).unwrap_or_default() {
return Err(MigrationError::new(
"partial_sequence",
"v27.columns",
Some(27),
"restore_known_good_backup",
));
}
Ok(())
}
@@ -0,0 +1,89 @@
//! V28 schema guard: typed Platform API key authority reference of a durable Task (Story 4.3).
use sqlx::{PgConnection, Row, query};
use super::authority::MigrationError;
use super::schema_guard::relation_exists;
const TASKS: &str = "tasks";
pub(super) async fn validate_v28_absent(
connection: &mut PgConnection,
) -> Result<bool, MigrationError> {
if relation_exists(connection, TASKS).await? {
if super::schema_guard::column_exists(connection, TASKS, "platform_api_key_id").await? {
return Ok(false);
}
}
Ok(true)
}
pub(super) async fn validate_v28_durable_tasks(
connection: &mut PgConnection,
) -> Result<(), MigrationError> {
// The V27 column set is still required and unchanged; V28 only appends one column.
validate_v27_base_columns(connection).await?;
let row = query(
"select data_type, is_nullable from information_schema.columns
where table_schema = current_schema() and table_name = $1 and column_name = $2",
)
.bind(TASKS)
.bind("platform_api_key_id")
.fetch_optional(&mut *connection)
.await
.map_err(|_| MigrationError::storage("preflight.schema"))?;
let valid = row.is_some_and(|row| {
row.try_get::<String, _>("data_type").ok().as_deref() == Some("text")
&& row.try_get::<String, _>("is_nullable").ok().as_deref() == Some("YES")
});
if !valid {
return Err(MigrationError::new(
"partial_sequence",
"v28.columns",
Some(28),
"restore_known_good_backup",
));
}
// Exact total column count for version 28: the 37 V27 columns + 1 V28 column.
let count: i64 = query(
"select count(*)::bigint as count from information_schema.columns
where table_schema = current_schema() and table_name = $1",
)
.bind(TASKS)
.fetch_one(&mut *connection)
.await
.map_err(|_| MigrationError::storage("preflight.schema"))?
.try_get("count")
.map_err(|_| MigrationError::storage("preflight.schema"))?;
if count != i64::try_from(38).unwrap_or_default() {
return Err(MigrationError::new(
"partial_sequence",
"v28.columns",
Some(28),
"restore_known_good_backup",
));
}
Ok(())
}
/// V27 columns must still be exactly present before V28 is considered applied.
async fn validate_v27_base_columns(connection: &mut PgConnection) -> Result<(), MigrationError> {
for column in [
"executable_input_ciphertext",
"executable_input_key_version",
"executable_input_master_key_epoch",
"target_executable_input_ciphertext",
"target_executable_input_key_version",
"target_executable_input_master_key_epoch",
] {
if !super::schema_guard::column_exists(connection, TASKS, column).await? {
return Err(MigrationError::new(
"partial_sequence",
"v28.columns",
Some(28),
"restore_known_good_backup",
));
}
}
Ok(())
}
Loaded 100 of 156 files, more files were not shown because too many files have changed in this diff. Show more