docs: add refactoring implementation plan

This commit is contained in:
a.tolmachev
2026-04-11 23:01:21 +03:00
parent d08ee8ce88
commit f337bd007e
3 changed files with 945 additions and 6 deletions
+1
View File
@@ -17,3 +17,4 @@ apps/ui/test-results
apps/ui/vite.config.js
apps/ui/vite.config.d.ts
*.log
__*.md
+12 -6
View File
@@ -2,21 +2,27 @@
## Current
### `feat/runtime-env-simplification`
### `feat/refactoring-plan`
Status: in_progress
DoD:
- runtime uses component-based PostgreSQL config as the primary model
- `CRANK_BASE_URL` replaces split public URL variables
- deployment uses one GitHub secret per runtime env variable
- deprecated env variables are removed from compose, examples, and docs
- `__REFACTORING.md` documents the full remediation plan for registry modularization, sqlx compile-time validation, KDF correction, typed timestamps, ID display support, pool tuning, and error normalization
- `.gitignore` ignores local `__*.md` working notes and refactoring specs
- `TASKS.md` points to the refactoring track as the active work item
## Next
- `feat/websocket-test-run-polish`
- `feat/postgres-registry-modularization`
## Backlog
- `feat/postgres-registry-modularization`
- `feat/sqlx-compile-time-verification`
- `feat/typed-timestamps`
- `feat/secret-crypto-hkdf`
- `feat/id-display-support`
- `feat/postgres-pool-config`
- `feat/error-structure-normalization`
- `feat/websocket-test-run-polish`
- `feat/live-authenticated-staging-entry`
+932
View File
@@ -0,0 +1,932 @@
# Refactoring Plan
## Purpose
This document defines the refactoring track required to improve maintainability, safety, and production-readiness in the current codebase. It is intentionally implementation-oriented: each area is split into concrete slices, target modules, function ownership, migration order, and acceptance criteria.
The refactoring scope covers:
1. splitting `crates/crank-registry/src/postgres.rs` into domain modules;
2. migrating static SQL to `sqlx::query!` / `sqlx::query_as!` where possible;
3. replacing SHA-256 key derivation in `SecretCrypto` with HKDF;
4. replacing string timestamps in domain models with typed timestamps;
5. adding `Display` support to `define_id!`;
6. introducing explicit PostgreSQL pool configuration;
7. removing duplicated SQL timestamp formatting;
8. making runtime and API error payloads more structured.
## Guiding Rules
- Keep domain logic separate from SQL storage concerns.
- Preserve current behavior while shrinking file size and coupling.
- Move in vertical slices: storage module split first, then SQL typing, then timestamp migration.
- Every slice must be independently reviewable and mergeable.
- Prefer compatibility shims during migration instead of breaking multiple crates at once.
- Add or update tests in the same slice as the refactoring.
## Refactoring Tracks
### Track A: Registry Modularization
#### Problem
`crates/crank-registry/src/postgres.rs` currently concentrates almost all database logic:
- bootstrap and auth;
- workspaces and memberships;
- invitations;
- operations and versions;
- agents and bindings;
- samples and descriptors;
- API keys;
- secrets;
- stream sessions and async jobs;
- invocation logs and usage;
- YAML import jobs.
This creates:
- a permanent git conflict hotspot;
- slow code navigation;
- oversized review diffs;
- accidental coupling between unrelated domains.
#### Target Layout
`crates/crank-registry/src/`
```text
error.rs
lib.rs
migrations.rs
model.rs
postgres/
mod.rs
connect.rs
auth.rs
workspace.rs
invitation.rs
secret.rs
stream.rs
api_key.rs
operation.rs
agent.rs
observability.rs
yaml_import.rs
shared.rs
```
#### Ownership by Module
##### `postgres/mod.rs`
Responsibilities:
- define `PostgresRegistry`;
- wire submodules;
- expose the public impl surface;
- keep only high-level constructor entry points.
Functions staying here:
- `new`
- `connect`
- `connect_with_options`
- `migrate`
Functions moving out:
- everything domain-specific.
##### `postgres/connect.rs`
Responsibilities:
- connection setup;
- pool initialization;
- migration invocation;
- pool option translation from config to `PgPoolOptions`.
Functions:
- `connect`
- `connect_with_options`
- `migrate`
- internal helpers:
- `build_pool_options`
- `build_connect_options`
##### `postgres/auth.rs`
Responsibilities:
- auth users;
- user sessions;
- membership access checks;
- auth profiles.
Functions:
- `upsert_bootstrap_user`
- `get_auth_user_by_email`
- `get_auth_user_by_id`
- `update_user_profile`
- `update_user_password`
- `create_user_session`
- `get_user_session`
- `touch_user_session`
- `revoke_user_session`
- `set_user_session_current_workspace`
- `user_has_workspace_access`
- `save_auth_profile`
- `get_auth_profile`
- `list_auth_profiles`
- `list_auth_profiles_referencing_secret`
Internal helpers to extract:
- `map_auth_user_row`
- `map_session_row`
- `map_auth_profile_row`
##### `postgres/workspace.rs`
Responsibilities:
- workspaces;
- memberships;
- workspace CRUD.
Functions:
- `list_workspaces`
- `list_workspaces_for_user`
- `ensure_membership`
- `list_memberships`
- `update_membership_role`
- `delete_membership`
- `get_workspace`
- `create_workspace`
- `update_workspace`
- `delete_workspace`
Internal helpers:
- `workspace_exists`
- `map_workspace_row`
- `map_membership_row`
##### `postgres/invitation.rs`
Responsibilities:
- invitations lifecycle.
Functions:
- `list_invitations`
- `create_invitation`
- `delete_invitation`
##### `postgres/secret.rs`
Responsibilities:
- secrets;
- secret versions;
- secret touch/rotation.
Functions:
- `list_secrets`
- `get_secret`
- `get_current_secret_version`
- `create_secret`
- `rotate_secret`
- `delete_secret`
- `touch_secret`
##### `postgres/stream.rs`
Responsibilities:
- stream sessions;
- async jobs;
- state transitions;
- cleanup.
Functions:
- `create_stream_session`
- `get_stream_session`
- `update_stream_session_state`
- `close_stream_session`
- `list_stream_sessions`
- `delete_expired_stream_sessions`
- `create_async_job`
- `get_async_job`
- `update_async_job_status`
- `cancel_async_job`
- `list_async_jobs`
- `delete_expired_async_jobs`
Internal helpers:
- `assert_stream_session_transition`
- `assert_async_job_transition`
- `map_stream_session_row`
- `map_async_job_row`
##### `postgres/api_key.rs`
Responsibilities:
- platform API key lifecycle.
Functions:
- `list_platform_api_keys`
- `get_platform_api_key_by_secret_for_workspace_slug`
- `create_platform_api_key`
- `revoke_platform_api_key`
- `delete_platform_api_key`
- `touch_platform_api_key`
##### `postgres/operation.rs`
Responsibilities:
- operations;
- versions;
- published operations;
- samples;
- descriptors.
Functions:
- `create_operation`
- `create_version`
- `update_operation_draft`
- `archive_operation`
- `delete_operation`
- `has_published_agent_bindings_for_operation`
- `list_operations`
- `list_operation_usage_summaries`
- `list_operation_agent_refs`
- `get_operation_summary`
- `get_operation_version`
- `list_operation_versions`
- `publish_operation`
- `get_published_operation`
- `list_published_operations`
- `save_sample_metadata`
- `list_sample_metadata`
- `save_descriptor_metadata`
- `list_descriptor_metadata`
- `insert_version_row`
Internal helpers:
- `map_operation_summary_row`
- `map_operation_version_row`
- `map_published_operation_row`
- `serialize_optional_json`
##### `postgres/agent.rs`
Responsibilities:
- agents;
- agent versions;
- bindings;
- publish lifecycle.
Functions:
- `list_agents`
- `get_agent_summary`
- `create_agent`
- `get_agent_version`
- `save_agent_bindings`
- `update_agent_summary`
- `delete_agent`
- `publish_agent`
- `unpublish_agent`
- `archive_agent`
- `get_published_agent_tools_by_slug`
- `list_agent_bindings`
- `insert_agent_version_row`
- `replace_agent_bindings_rows`
##### `postgres/observability.rs`
Responsibilities:
- invocation logs;
- usage rollups and breakdowns.
Functions:
- `create_invocation_log`
- `list_invocation_logs`
- `get_invocation_log`
- `summarize_usage`
- `list_usage_timeline`
- `list_usage_by_operation`
- `get_usage_for_operation`
- `list_usage_by_agent`
- `get_usage_for_agent`
##### `postgres/yaml_import.rs`
Responsibilities:
- YAML import jobs.
Functions:
- `create_yaml_import_job`
- `finish_yaml_import_job`
- `get_yaml_import_job`
##### `postgres/shared.rs`
Responsibilities:
- SQL snippets reused by several modules;
- common timestamp select fragments during the migration period;
- small row-mapping helpers;
- pagination helpers.
Planned content:
- `Page<T>` helpers if they remain storage-local;
- constants for table-independent fragments;
- transitional helpers until typed timestamps remove string formatting entirely.
#### Slice Plan for Track A
##### Slice A1: File layout and constructors
- create `postgres/` directory;
- move `connect`, `connect_with_options`, `migrate`, and struct wiring;
- keep all public signatures unchanged.
##### Slice A2: Workspace/Auth/API-key split
- move `auth.rs`, `workspace.rs`, `invitation.rs`, `api_key.rs`;
- leave behavior unchanged;
- add targeted module tests if not present.
##### Slice A3: Operation/Agent split
- move `operation.rs`, `agent.rs`, `yaml_import.rs`;
- keep SQL unchanged for this slice;
- ensure published-operation and published-agent behavior is unchanged.
##### Slice A4: Secret/Stream/Observability split
- move `secret.rs`, `stream.rs`, `observability.rs`;
- keep transition logic and cleanup behavior unchanged.
##### Slice A5: Remove legacy monolith
- remove remaining methods from root `postgres.rs`;
- rename module structure so `postgres/mod.rs` is the only entry point.
#### Done Criteria for Track A
- no `postgres.rs` monolith remains;
- no single registry source file exceeds roughly 8001200 lines;
- all existing registry tests pass unchanged;
- `lib.rs` still re-exports `PostgresRegistry` without downstream breakage.
## Track B: Compile-Time SQL Verification
### Problem
Most static SQL currently uses `sqlx::query(...)` / `query_as::<_, T>(...)` with runtime validation only. That leaves:
- column name typos;
- mismatched types;
- bad nullability assumptions;
- schema drift
to runtime instead of compile time.
### Goal
Use `sqlx::query!` and `sqlx::query_as!` for every static query with stable shape.
Keep runtime builders only where the query shape is truly dynamic.
### Query Classification
#### Must migrate to `query!` / `query_as!`
Use macros for:
- single-row fetches by ID;
- list queries with static columns;
- insert/update/delete with static statement shape;
- existence checks;
- count queries;
- cleanup queries with fixed predicates.
Expected modules:
- `auth.rs`
- `workspace.rs`
- `invitation.rs`
- `secret.rs`
- `stream.rs`
- `api_key.rs`
- most of `agent.rs`
- most of `yaml_import.rs`
#### May stay runtime-based
Keep dynamic APIs only when necessary:
- optional filters that materially change SQL shape;
- optional sort columns chosen from a whitelist;
- bulk dynamic `IN` lists where macro usage becomes unreadable;
- `QueryBuilder`-based multi-row inserts/updates.
Expected modules:
- `operation.rs` list queries with optional filters;
- `observability.rs` usage aggregations and breakdowns;
- some search/filter endpoints.
### Required Infrastructure
#### Add SQLx offline support
Repository-level requirements:
- use `sqlx prepare` to generate checked metadata;
- keep checked metadata in version control if the team wants deterministic CI;
- wire verification into CI or `just verify`.
Concrete additions:
- `sqlx-data.json` or the current SQLx metadata equivalent for the chosen version;
- `DATABASE_URL` or test DB wiring for `cargo sqlx prepare`.
#### Introduce row structs where needed
For `query_as!`, add storage-local row structs for cases where domain structs should not be coupled directly to SQL column shape.
Examples:
- `WorkspaceRow`
- `AuthUserRow`
- `OperationSummaryRow`
- `InvocationLogRow`
- `UsageTimelineRow`
Rule:
- domain structs remain domain-oriented;
- row structs remain SQL-oriented.
### Function-Level Migration Rules
#### For `query_as!`
Use `query_as!` when:
- the query result maps cleanly to a typed row struct;
- all returned columns are explicit;
- nullability is known.
#### For `query!`
Use `query!` when:
- no reusable row struct is needed;
- the query is a simple `insert`, `update`, `delete`, or `exists` fetch.
#### For dynamic list queries
Use:
- `sqlx::QueryBuilder<Postgres>` with explicit allowed sort/filter sets;
- storage-local mapper functions from row to domain object.
### Migration Order
##### Slice B1: `auth`, `workspace`, `invitation`
Low-risk, fixed-shape queries.
##### Slice B2: `secret`, `stream`, `api_key`, `yaml_import`
Still mostly fixed-shape.
##### Slice B3: `operation`, `agent`
Introduce mixed strategy: macros for fixed queries, builder for dynamic lists.
##### Slice B4: `observability`
Likely the most dynamic area. Convert only static pieces first, leave complex aggregations to `QueryBuilder`.
### Done Criteria for Track B
- all fixed-shape SQL uses compile-time checked macros;
- dynamic SQL is explicitly documented and isolated;
- CI fails on SQL drift before runtime tests;
- no new static query is accepted through `sqlx::query(...)` without justification.
## Track C: SecretCrypto KDF Correction
### Problem
Current code in `crates/crank-runtime/src/secret_crypto.rs` derives the AES key by hashing the master key with SHA-256:
- no KDF semantics;
- no labeled derivation context;
- no future rotation/versioning path.
### Goal
Replace direct hashing with HKDF-SHA256.
### Target Design
`SecretCrypto::new(master_key: &str)` will:
1. validate non-empty master key;
2. derive a 32-byte AES key using `Hkdf<Sha256>`;
3. use explicit `info` context, for example:
- `b"crank.secret-envelope.v1"`
4. initialize `Aes256Gcm` from the derived bytes.
### Module Changes
File:
- `crates/crank-runtime/src/secret_crypto.rs`
Changes:
- replace `Sha256::digest(trimmed.as_bytes())` with HKDF expansion;
- preserve the public constructor signature;
- keep ciphertext format unchanged if possible;
- if ciphertext format must change later, introduce envelope `version` before doing so.
### Required Tests
- `encrypt_decrypt_roundtrip_still_works`
- `empty_master_key_is_rejected`
- `same_master_key_derives_stable_key`
- `different_master_keys_produce_different_ciphertexts`
### Done Criteria for Track C
- no direct SHA-256 KDF remains;
- `SecretCrypto` uses HKDF-SHA256 with explicit context;
- all existing secret encrypt/decrypt flows remain compatible.
## Track D: Typed Timestamps
### Problem
Domain and storage structs use `String` timestamps. SQL converts timestamps via repeated `to_char(...)`.
That causes:
- repeated formatting logic in SQL;
- weak type guarantees;
- inability to do time arithmetic without reparsing;
- broad surface for formatting drift.
### Goal
Use `time::OffsetDateTime` in domain and storage models, with RFC 3339 serialization at the transport boundary.
### Target Type Strategy
#### Domain layer
Use:
```rust
#[serde(with = "time::serde::rfc3339")]
pub created_at: OffsetDateTime
```
and the same for `updated_at`, `expires_at`, `used_at`, and similar fields.
#### Storage row layer
Also use `OffsetDateTime`.
Do not format timestamps in SQL except for transitional compatibility shims.
#### Transport layer
JSON still emits RFC 3339 strings through Serde.
### Files Affected
#### Domain crates
- `crates/crank-core/src/workspace.rs`
- `crates/crank-core/src/access.rs`
- `crates/crank-core/src/auth.rs`
- `crates/crank-core/src/agent.rs`
- `crates/crank-core/src/operation.rs`
- `crates/crank-core/src/observability.rs`
- `crates/crank-core/src/secret.rs`
- `crates/crank-core/src/stream_session.rs`
#### Registry models
- `crates/crank-registry/src/model.rs`
#### Admin API transport/service DTOs
- `apps/admin-api/src/service.rs`
- any route DTO files that mirror time fields directly
### Migration Strategy
##### Slice D1: storage row types
- change storage row structs to `OffsetDateTime`;
- remove `to_char(...)` from fixed-shape SQL;
- keep service DTOs as strings temporarily if needed.
##### Slice D2: domain models
- convert `crank-core` models to typed timestamps with Serde helpers;
- update constructors and tests.
##### Slice D3: API/service transport
- remove string timestamp shims from service DTOs where possible;
- ensure JSON remains stable.
##### Slice D4: cleanup
- remove transitional formatting helpers;
- remove `now_string()` helpers in favor of typed timestamps.
### Acceptance Criteria
- no `created_at: String` or `updated_at: String` remains in domain/storage models;
- no `to_char(created_at...)` / `to_char(updated_at...)` remains in registry SQL except perhaps in legacy compatibility code slated for removal in the same slice;
- JSON contract stays RFC 3339.
## Track E: `Display` for ID Newtypes
### Problem
`define_id!` currently requires `.as_str()` everywhere for formatting and messages.
### Goal
Add `std::fmt::Display` support to every generated ID newtype.
### File
- `crates/crank-core/src/ids.rs`
### Change
Add inside `define_id!`:
```rust
impl std::fmt::Display for $name {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
write!(f, "{}", self.0)
}
}
```
### Follow-Up Cleanup
After macro support lands:
- replace formatting sites like `workspace_id.as_str()` inside messages;
- keep `.as_str()` only where an actual `&str` API requires it.
### Acceptance Criteria
- all ID newtypes implement `Display`;
- formatting code no longer uses `.as_str()` inside `format!`, `println!`, or error messages unless `&str` is actually required.
## Track F: PostgreSQL Pool Configuration
### Problem
The pool is currently created via default `PgPoolOptions`, which means the runtime behavior is implicit and under-configured.
### Goal
Make pool behavior explicit and configurable from environment.
### Target Config Model
Add a typed config struct, for example:
```rust
pub struct PostgresPoolConfig {
pub max_connections: u32,
pub min_connections: u32,
pub acquire_timeout_ms: u64,
pub idle_timeout_ms: u64,
pub max_lifetime_ms: u64,
}
```
### Runtime Environment
Add env variables:
- `POSTGRES_MAX_CONNECTIONS`
- `POSTGRES_MIN_CONNECTIONS`
- `POSTGRES_ACQUIRE_TIMEOUT_MS`
- `POSTGRES_IDLE_TIMEOUT_MS`
- `POSTGRES_MAX_LIFETIME_MS`
### Files
- `crates/crank-registry/src/postgres/connect.rs`
- `apps/admin-api/src/main.rs`
- `apps/mcp-server/src/main.rs`
- config docs and examples
### Behavior
`connect_with_options` should accept either:
- direct `PgConnectOptions` plus explicit pool config;
- or an internal default config built from env.
Defaults should be conservative and explicit, for example:
- `max_connections = 20`
- `min_connections = 2`
- `acquire_timeout = 5s`
- `idle_timeout = 10m`
- `max_lifetime = 30m`
Exact values can be tuned later, but they must no longer be hidden defaults.
### Acceptance Criteria
- pool limits and timeouts are explicit;
- docs reflect the new variables;
- startup logs or config inspection make active pool settings visible.
## Track G: Timestamp Formatting Deduplication
### Problem
The same `to_char(... at time zone 'UTC', 'YYYY-MM-DD"T"HH24:MI:SS"Z"')` fragment is copied across many queries.
### Goal
Eliminate duplicated formatting logic.
### Strategy
This track is mostly resolved by Track D, but there is a transition window.
During transition:
- centralize any remaining formatting in `postgres/shared.rs`;
- define named SQL fragments only once;
- remove inline duplicated snippets.
After Track D:
- remove formatting fragments entirely.
### Acceptance Criteria
- duplicated inline timestamp formatting is no longer scattered across query strings;
- final state has typed timestamps instead of SQL formatting.
## Track H: Error Structure Normalization
### Problem
Some runtime and API error variants carry only a `details: String`, which makes programmatic handling and consistent observability harder.
### Goal
Introduce structured error payloads where the current model is too string-heavy.
### Scope
Priority types:
- `RuntimeError`
- registry storage errors where context is stringly typed;
- API error mapping around secret/runtime/streaming faults.
### Design Rules
- keep `thiserror`;
- use structured fields in variants whenever stable context exists;
- reserve free-form `details` only for opaque third-party messages.
Examples:
Instead of:
```rust
RuntimeError::SecretCrypto { details: String }
```
prefer:
```rust
RuntimeError::SecretCrypto {
operation: &'static str,
details: String,
}
```
or dedicated variants such as:
- `InvalidSecretEnvelope`
- `SecretDecryptFailed`
- `SecretSerializeFailed`
### Slice Plan
##### Slice H1
- normalize `SecretCrypto` and streaming/session/job errors.
##### Slice H2
- normalize adapter errors for REST/gRPC/WebSocket/SOAP where stable categories exist.
##### Slice H3
- normalize API mapping and error codes.
### Acceptance Criteria
- core runtime error variants expose structured context;
- API can map key failure classes deterministically;
- logs and telemetry can group errors by category instead of by ad hoc message text.
## Delivery Order
Recommended order:
1. `feat/refactoring-plan`
2. `feat/id-display-support`
3. `feat/postgres-pool-config`
4. `feat/postgres-registry-modularization`
5. `feat/sqlx-compile-time-verification`
6. `feat/secret-crypto-hkdf`
7. `feat/typed-timestamps`
8. `feat/error-structure-normalization`
9. `feat/post-refactor-cleanup`
Reasoning:
- `Display` is tiny and immediately reduces noise.
- pool config is operational and isolated.
- module split should happen before large query migrations, otherwise the macro migration lands into the monolith and becomes harder to review.
- HKDF is isolated and low-risk.
- typed timestamps should land after storage modules are split and query migration is underway.
## Per-Slice Verification
Every slice must run:
- `just fmt-check`
- `just check`
- `just clippy`
- targeted tests for the touched area
Additionally:
- SQL macro migration slices must run SQLx prepare/verification;
- timestamp slices must run serialization regression tests;
- secret crypto slice must run compatibility and roundtrip tests.
## Success Definition
The refactoring track is complete when:
- registry storage code is split into domain modules;
- static SQL is compile-time validated;
- `SecretCrypto` uses HKDF instead of SHA-256 as a KDF;
- domain/storage timestamps are typed;
- ID newtypes implement `Display`;
- PostgreSQL pool settings are explicit and configurable;
- timestamp formatting duplication is removed;
- error variants carry structured context where it matters.
At that point, maintainability improves materially without changing the product model or feature scope.