crank/__REFACTORING.md

# Refactoring Plan

## Purpose

This document defines the refactoring track required to improve maintainability, safety, and production-readiness in the current codebase. It is intentionally implementation-oriented: each area is split into concrete slices, target modules, function ownership, migration order, and acceptance criteria.

The refactoring scope covers:

1. splitting `crates/crank-registry/src/postgres.rs` into domain modules;
2. migrating static SQL to `sqlx::query!` / `sqlx::query_as!` where possible;
3. replacing SHA-256 key derivation in `SecretCrypto` with HKDF;
4. replacing string timestamps in domain models with typed timestamps;
5. adding `Display` support to `define_id!`;
6. introducing explicit PostgreSQL pool configuration;
7. removing duplicated SQL timestamp formatting;
8. making runtime and API error payloads more structured.

## Guiding Rules

- Keep domain logic separate from SQL storage concerns.
- Preserve current behavior while shrinking file size and coupling.
- Move in vertical slices: storage module split first, then SQL typing, then timestamp migration.
- Every slice must be independently reviewable and mergeable.
- Prefer compatibility shims during migration instead of breaking multiple crates at once.
- Add or update tests in the same slice as the refactoring.

## Refactoring Tracks

### Track A: Registry Modularization

#### Problem

`crates/crank-registry/src/postgres.rs` currently concentrates almost all database logic:

- bootstrap and auth;
- workspaces and memberships;
- invitations;
- operations and versions;
- agents and bindings;
- samples and descriptors;
- API keys;
- secrets;
- stream sessions and async jobs;
- invocation logs and usage;
- YAML import jobs.

This creates:

- a permanent git conflict hotspot;
- slow code navigation;
- oversized review diffs;
- accidental coupling between unrelated domains.

#### Target Layout

`crates/crank-registry/src/`

```text
error.rs
lib.rs
migrations.rs
model.rs
postgres/
  mod.rs
  connect.rs
  auth.rs
  workspace.rs
  invitation.rs
  secret.rs
  stream.rs
  api_key.rs
  operation.rs
  agent.rs
  observability.rs
  yaml_import.rs
  shared.rs
```

#### Ownership by Module

##### `postgres/mod.rs`

Responsibilities:

- define `PostgresRegistry`;
- wire submodules;
- expose the public impl surface;
- keep only high-level constructor entry points.

Functions staying here:

- `new`
- `connect`
- `connect_with_options`
- `migrate`

Functions moving out:

- everything domain-specific.

##### `postgres/connect.rs`

Responsibilities:

- connection setup;
- pool initialization;
- migration invocation;
- pool option translation from config to `PgPoolOptions`.

Functions:

- `connect`
- `connect_with_options`
- `migrate`
- internal helpers:
  - `build_pool_options`
  - `build_connect_options`

##### `postgres/auth.rs`

Responsibilities:

- auth users;
- user sessions;
- membership access checks;
- auth profiles.

Functions:

- `upsert_bootstrap_user`
- `get_auth_user_by_email`
- `get_auth_user_by_id`
- `update_user_profile`
- `update_user_password`
- `create_user_session`
- `get_user_session`
- `touch_user_session`
- `revoke_user_session`
- `set_user_session_current_workspace`
- `user_has_workspace_access`
- `save_auth_profile`
- `get_auth_profile`
- `list_auth_profiles`
- `list_auth_profiles_referencing_secret`

Internal helpers to extract:

- `map_auth_user_row`
- `map_session_row`
- `map_auth_profile_row`

##### `postgres/workspace.rs`

Responsibilities:

- workspaces;
- memberships;
- workspace CRUD.

Functions:

- `list_workspaces`
- `list_workspaces_for_user`
- `ensure_membership`
- `list_memberships`
- `update_membership_role`
- `delete_membership`
- `get_workspace`
- `create_workspace`
- `update_workspace`
- `delete_workspace`

Internal helpers:

- `workspace_exists`
- `map_workspace_row`
- `map_membership_row`

##### `postgres/invitation.rs`

Responsibilities:

- invitations lifecycle.

Functions:

- `list_invitations`
- `create_invitation`
- `delete_invitation`

##### `postgres/secret.rs`

Responsibilities:

- secrets;
- secret versions;
- secret touch/rotation.

Functions:

- `list_secrets`
- `get_secret`
- `get_current_secret_version`
- `create_secret`
- `rotate_secret`
- `delete_secret`
- `touch_secret`

##### `postgres/stream.rs`

Responsibilities:

- stream sessions;
- async jobs;
- state transitions;
- cleanup.

Functions:

- `create_stream_session`
- `get_stream_session`
- `update_stream_session_state`
- `close_stream_session`
- `list_stream_sessions`
- `delete_expired_stream_sessions`
- `create_async_job`
- `get_async_job`
- `update_async_job_status`
- `cancel_async_job`
- `list_async_jobs`
- `delete_expired_async_jobs`

Internal helpers:

- `assert_stream_session_transition`
- `assert_async_job_transition`
- `map_stream_session_row`
- `map_async_job_row`

##### `postgres/api_key.rs`

Responsibilities:

- platform API key lifecycle.

Functions:

- `list_platform_api_keys`
- `get_platform_api_key_by_secret_for_workspace_slug`
- `create_platform_api_key`
- `revoke_platform_api_key`
- `delete_platform_api_key`
- `touch_platform_api_key`

##### `postgres/operation.rs`

Responsibilities:

- operations;
- versions;
- published operations;
- samples;
- descriptors.

Functions:

- `create_operation`
- `create_version`
- `update_operation_draft`
- `archive_operation`
- `delete_operation`
- `has_published_agent_bindings_for_operation`
- `list_operations`
- `list_operation_usage_summaries`
- `list_operation_agent_refs`
- `get_operation_summary`
- `get_operation_version`
- `list_operation_versions`
- `publish_operation`
- `get_published_operation`
- `list_published_operations`
- `save_sample_metadata`
- `list_sample_metadata`
- `save_descriptor_metadata`
- `list_descriptor_metadata`
- `insert_version_row`

Internal helpers:

- `map_operation_summary_row`
- `map_operation_version_row`
- `map_published_operation_row`
- `serialize_optional_json`

##### `postgres/agent.rs`

Responsibilities:

- agents;
- agent versions;
- bindings;
- publish lifecycle.

Functions:

- `list_agents`
- `get_agent_summary`
- `create_agent`
- `get_agent_version`
- `save_agent_bindings`
- `update_agent_summary`
- `delete_agent`
- `publish_agent`
- `unpublish_agent`
- `archive_agent`
- `get_published_agent_tools_by_slug`
- `list_agent_bindings`
- `insert_agent_version_row`
- `replace_agent_bindings_rows`

##### `postgres/observability.rs`

Responsibilities:

- invocation logs;
- usage rollups and breakdowns.

Functions:

- `create_invocation_log`
- `list_invocation_logs`
- `get_invocation_log`
- `summarize_usage`
- `list_usage_timeline`
- `list_usage_by_operation`
- `get_usage_for_operation`
- `list_usage_by_agent`
- `get_usage_for_agent`

##### `postgres/yaml_import.rs`

Responsibilities:

- YAML import jobs.

Functions:

- `create_yaml_import_job`
- `finish_yaml_import_job`
- `get_yaml_import_job`

##### `postgres/shared.rs`

Responsibilities:

- SQL snippets reused by several modules;
- common timestamp select fragments during the migration period;
- small row-mapping helpers;
- pagination helpers.

Planned content:

- `Page<T>` helpers if they remain storage-local;
- constants for table-independent fragments;
- transitional helpers until typed timestamps remove string formatting entirely.

#### Slice Plan for Track A

##### Slice A1: File layout and constructors

- create `postgres/` directory;
- move `connect`, `connect_with_options`, `migrate`, and struct wiring;
- keep all public signatures unchanged.

##### Slice A2: Workspace/Auth/API-key split

- move `auth.rs`, `workspace.rs`, `invitation.rs`, `api_key.rs`;
- leave behavior unchanged;
- add targeted module tests if not present.

##### Slice A3: Operation/Agent split

- move `operation.rs`, `agent.rs`, `yaml_import.rs`;
- keep SQL unchanged for this slice;
- ensure published-operation and published-agent behavior is unchanged.

##### Slice A4: Secret/Stream/Observability split

- move `secret.rs`, `stream.rs`, `observability.rs`;
- keep transition logic and cleanup behavior unchanged.

##### Slice A5: Remove legacy monolith

- remove remaining methods from root `postgres.rs`;
- rename module structure so `postgres/mod.rs` is the only entry point.

#### Done Criteria for Track A

- no `postgres.rs` monolith remains;
- no single registry source file exceeds roughly 800–1200 lines;
- all existing registry tests pass unchanged;
- `lib.rs` still re-exports `PostgresRegistry` without downstream breakage.

## Track B: Compile-Time SQL Verification

### Problem

Most static SQL currently uses `sqlx::query(...)` / `query_as::<_, T>(...)` with runtime validation only. That leaves:

- column name typos;
- mismatched types;
- bad nullability assumptions;
- schema drift

to runtime instead of compile time.

### Goal

Use `sqlx::query!` and `sqlx::query_as!` for every static query with stable shape.

Keep runtime builders only where the query shape is truly dynamic.

### Query Classification

#### Must migrate to `query!` / `query_as!`

Use macros for:

- single-row fetches by ID;
- list queries with static columns;
- insert/update/delete with static statement shape;
- existence checks;
- count queries;
- cleanup queries with fixed predicates.

Expected modules:

- `auth.rs`
- `workspace.rs`
- `invitation.rs`
- `secret.rs`
- `stream.rs`
- `api_key.rs`
- most of `agent.rs`
- most of `yaml_import.rs`

#### May stay runtime-based

Keep dynamic APIs only when necessary:

- optional filters that materially change SQL shape;
- optional sort columns chosen from a whitelist;
- bulk dynamic `IN` lists where macro usage becomes unreadable;
- `QueryBuilder`-based multi-row inserts/updates.

Expected modules:

- `operation.rs` list queries with optional filters;
- `observability.rs` usage aggregations and breakdowns;
- some search/filter endpoints.

### Required Infrastructure

#### Add SQLx offline support

Repository-level requirements:

- use `sqlx prepare` to generate checked metadata;
- keep checked metadata in version control if the team wants deterministic CI;
- wire verification into CI or `just verify`.

Concrete additions:

- `sqlx-data.json` or the current SQLx metadata equivalent for the chosen version;
- `DATABASE_URL` or test DB wiring for `cargo sqlx prepare`.

#### Introduce row structs where needed

For `query_as!`, add storage-local row structs for cases where domain structs should not be coupled directly to SQL column shape.

Examples:

- `WorkspaceRow`
- `AuthUserRow`
- `OperationSummaryRow`
- `InvocationLogRow`
- `UsageTimelineRow`

Rule:

- domain structs remain domain-oriented;
- row structs remain SQL-oriented.

### Function-Level Migration Rules

#### For `query_as!`

Use `query_as!` when:

- the query result maps cleanly to a typed row struct;
- all returned columns are explicit;
- nullability is known.

#### For `query!`

Use `query!` when:

- no reusable row struct is needed;
- the query is a simple `insert`, `update`, `delete`, or `exists` fetch.

#### For dynamic list queries

Use:

- `sqlx::QueryBuilder<Postgres>` with explicit allowed sort/filter sets;
- storage-local mapper functions from row to domain object.

### Migration Order

##### Slice B1: `auth`, `workspace`, `invitation`

Low-risk, fixed-shape queries.

##### Slice B2: `secret`, `stream`, `api_key`, `yaml_import`

Still mostly fixed-shape.

##### Slice B3: `operation`, `agent`

Introduce mixed strategy: macros for fixed queries, builder for dynamic lists.

##### Slice B4: `observability`

Likely the most dynamic area. Convert only static pieces first, leave complex aggregations to `QueryBuilder`.

### Done Criteria for Track B

- all fixed-shape SQL uses compile-time checked macros;
- dynamic SQL is explicitly documented and isolated;
- CI fails on SQL drift before runtime tests;
- no new static query is accepted through `sqlx::query(...)` without justification.

## Track C: SecretCrypto KDF Correction

### Problem

Current code in `crates/crank-runtime/src/secret_crypto.rs` derives the AES key by hashing the master key with SHA-256:

- no KDF semantics;
- no labeled derivation context;
- no future rotation/versioning path.

### Goal

Replace direct hashing with HKDF-SHA256.

### Target Design

`SecretCrypto::new(master_key: &str)` will:

1. validate non-empty master key;
2. derive a 32-byte AES key using `Hkdf<Sha256>`;
3. use explicit `info` context, for example:
   - `b"crank.secret-envelope.v1"`
4. initialize `Aes256Gcm` from the derived bytes.

### Module Changes

File:

- `crates/crank-runtime/src/secret_crypto.rs`

Changes:

- replace `Sha256::digest(trimmed.as_bytes())` with HKDF expansion;
- preserve the public constructor signature;
- keep ciphertext format unchanged if possible;
- if ciphertext format must change later, introduce envelope `version` before doing so.

### Required Tests

- `encrypt_decrypt_roundtrip_still_works`
- `empty_master_key_is_rejected`
- `same_master_key_derives_stable_key`
- `different_master_keys_produce_different_ciphertexts`

### Done Criteria for Track C

- no direct SHA-256 KDF remains;
- `SecretCrypto` uses HKDF-SHA256 with explicit context;
- all existing secret encrypt/decrypt flows remain compatible.

## Track D: Typed Timestamps

### Problem

Domain and storage structs use `String` timestamps. SQL converts timestamps via repeated `to_char(...)`.

That causes:

- repeated formatting logic in SQL;
- weak type guarantees;
- inability to do time arithmetic without reparsing;
- broad surface for formatting drift.

### Goal

Use `time::OffsetDateTime` in domain and storage models, with RFC 3339 serialization at the transport boundary.

### Target Type Strategy

#### Domain layer

Use:

```rust
#[serde(with = "time::serde::rfc3339")]
pub created_at: OffsetDateTime
```

and the same for `updated_at`, `expires_at`, `used_at`, and similar fields.

#### Storage row layer

Also use `OffsetDateTime`.

Do not format timestamps in SQL except for transitional compatibility shims.

#### Transport layer

JSON still emits RFC 3339 strings through Serde.

### Files Affected

#### Domain crates

- `crates/crank-core/src/workspace.rs`
- `crates/crank-core/src/access.rs`
- `crates/crank-core/src/auth.rs`
- `crates/crank-core/src/agent.rs`
- `crates/crank-core/src/operation.rs`
- `crates/crank-core/src/observability.rs`
- `crates/crank-core/src/secret.rs`
- `crates/crank-core/src/stream_session.rs`

#### Registry models

- `crates/crank-registry/src/model.rs`

#### Admin API transport/service DTOs

- `apps/admin-api/src/service.rs`
- any route DTO files that mirror time fields directly

### Migration Strategy

##### Slice D1: storage row types

- change storage row structs to `OffsetDateTime`;
- remove `to_char(...)` from fixed-shape SQL;
- keep service DTOs as strings temporarily if needed.

##### Slice D2: domain models

- convert `crank-core` models to typed timestamps with Serde helpers;
- update constructors and tests.

##### Slice D3: API/service transport

- remove string timestamp shims from service DTOs where possible;
- ensure JSON remains stable.

##### Slice D4: cleanup

- remove transitional formatting helpers;
- remove `now_string()` helpers in favor of typed timestamps.

### Acceptance Criteria

- no `created_at: String` or `updated_at: String` remains in domain/storage models;
- no `to_char(created_at...)` / `to_char(updated_at...)` remains in registry SQL except perhaps in legacy compatibility code slated for removal in the same slice;
- JSON contract stays RFC 3339.

## Track E: `Display` for ID Newtypes

### Problem

`define_id!` currently requires `.as_str()` everywhere for formatting and messages.

### Goal

Add `std::fmt::Display` support to every generated ID newtype.

### File

- `crates/crank-core/src/ids.rs`

### Change

Add inside `define_id!`:

```rust
impl std::fmt::Display for $name {
    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
        write!(f, "{}", self.0)
    }
}
```

### Follow-Up Cleanup

After macro support lands:

- replace formatting sites like `workspace_id.as_str()` inside messages;
- keep `.as_str()` only where an actual `&str` API requires it.

### Acceptance Criteria

- all ID newtypes implement `Display`;
- formatting code no longer uses `.as_str()` inside `format!`, `println!`, or error messages unless `&str` is actually required.

## Track F: PostgreSQL Pool Configuration

### Problem

The pool is currently created via default `PgPoolOptions`, which means the runtime behavior is implicit and under-configured.

### Goal

Make pool behavior explicit and configurable from environment.

### Target Config Model

Add a typed config struct, for example:

```rust
pub struct PostgresPoolConfig {
    pub max_connections: u32,
    pub min_connections: u32,
    pub acquire_timeout_ms: u64,
    pub idle_timeout_ms: u64,
    pub max_lifetime_ms: u64,
}
```

### Runtime Environment

Add env variables:

- `POSTGRES_MAX_CONNECTIONS`
- `POSTGRES_MIN_CONNECTIONS`
- `POSTGRES_ACQUIRE_TIMEOUT_MS`
- `POSTGRES_IDLE_TIMEOUT_MS`
- `POSTGRES_MAX_LIFETIME_MS`

### Files

- `crates/crank-registry/src/postgres/connect.rs`
- `apps/admin-api/src/main.rs`
- `apps/mcp-server/src/main.rs`
- config docs and examples

### Behavior

`connect_with_options` should accept either:

- direct `PgConnectOptions` plus explicit pool config;
- or an internal default config built from env.

Defaults should be conservative and explicit, for example:

- `max_connections = 20`
- `min_connections = 2`
- `acquire_timeout = 5s`
- `idle_timeout = 10m`
- `max_lifetime = 30m`

Exact values can be tuned later, but they must no longer be hidden defaults.

### Acceptance Criteria

- pool limits and timeouts are explicit;
- docs reflect the new variables;
- startup logs or config inspection make active pool settings visible.

## Track G: Timestamp Formatting Deduplication

### Problem

The same `to_char(... at time zone 'UTC', 'YYYY-MM-DD"T"HH24:MI:SS"Z"')` fragment is copied across many queries.

### Goal

Eliminate duplicated formatting logic.

### Strategy

This track is mostly resolved by Track D, but there is a transition window.

During transition:

- centralize any remaining formatting in `postgres/shared.rs`;
- define named SQL fragments only once;
- remove inline duplicated snippets.

After Track D:

- remove formatting fragments entirely.

### Acceptance Criteria

- duplicated inline timestamp formatting is no longer scattered across query strings;
- final state has typed timestamps instead of SQL formatting.

## Track H: Error Structure Normalization

### Problem

Some runtime and API error variants carry only a `details: String`, which makes programmatic handling and consistent observability harder.

### Goal

Introduce structured error payloads where the current model is too string-heavy.

### Scope

Priority types:

- `RuntimeError`
- registry storage errors where context is stringly typed;
- API error mapping around secret/runtime/streaming faults.

### Design Rules

- keep `thiserror`;
- use structured fields in variants whenever stable context exists;
- reserve free-form `details` only for opaque third-party messages.

Examples:

Instead of:

```rust
RuntimeError::SecretCrypto { details: String }
```

prefer:

```rust
RuntimeError::SecretCrypto {
    operation: &'static str,
    details: String,
}
```

or dedicated variants such as:

- `InvalidSecretEnvelope`
- `SecretDecryptFailed`
- `SecretSerializeFailed`

### Slice Plan

##### Slice H1

- normalize `SecretCrypto` and streaming/session/job errors.

##### Slice H2

- normalize adapter errors for REST/gRPC/WebSocket/SOAP where stable categories exist.

##### Slice H3

- normalize API mapping and error codes.

### Acceptance Criteria

- core runtime error variants expose structured context;
- API can map key failure classes deterministically;
- logs and telemetry can group errors by category instead of by ad hoc message text.

## Delivery Order

Recommended order:

1. `feat/refactoring-plan`
2. `feat/id-display-support`
3. `feat/postgres-pool-config`
4. `feat/postgres-registry-modularization`
5. `feat/sqlx-compile-time-verification`
6. `feat/secret-crypto-hkdf`
7. `feat/typed-timestamps`
8. `feat/error-structure-normalization`
9. `feat/post-refactor-cleanup`

Reasoning:

- `Display` is tiny and immediately reduces noise.
- pool config is operational and isolated.
- module split should happen before large query migrations, otherwise the macro migration lands into the monolith and becomes harder to review.
- HKDF is isolated and low-risk.
- typed timestamps should land after storage modules are split and query migration is underway.

## Per-Slice Verification

Every slice must run:

- `just fmt-check`
- `just check`
- `just clippy`
- targeted tests for the touched area

Additionally:

- SQL macro migration slices must run SQLx prepare/verification;
- timestamp slices must run serialization regression tests;
- secret crypto slice must run compatibility and roundtrip tests.

## Success Definition

The refactoring track is complete when:

- registry storage code is split into domain modules;
- static SQL is compile-time validated;
- `SecretCrypto` uses HKDF instead of SHA-256 as a KDF;
- domain/storage timestamps are typed;
- ID newtypes implement `Display`;
- PostgreSQL pool settings are explicit and configurable;
- timestamp formatting duplication is removed;
- error variants carry structured context where it matters.

At that point, maintainability improves materially without changing the product model or feature scope.