Runtime kernel: the verified persistence, messaging and tracing path

Every generated backend includes a small, framework-neutral runtime next to its domain kernel. It is the code path that ghk verifies against real infrastructure: PostgreSQL and an event broker (RabbitMQ, Google Cloud Pub/Sub, Azure Service Bus or Apache Kafka), with OpenTelemetry trace context. Each stack implements it with that language's standard driver, broker client and OpenTelemetry SDK. The behavior and the tests are the same in every language and on every broker.

Brokers

The outbox relay and the inbox consumer use broker-neutral ports (publisher, subscription, broker). The adapter is chosen when the service starts: GHK_BROKER if set, otherwise the broker whose variables are present, otherwise stack.messaging. One image can therefore run on any cloud.

Rust, Ruby, Elixir and PHP have no official Service Bus or Kafka SDK: with those settings they keep RabbitMQ. On Kafka, the exchange and the DLX are topics, and the queue and the DLQ are consumer groups on them; publishing waits for every in-sync replica (acks=all, idempotent producer), and an ack commits the offset. The variables of each broker are in DEPLOYMENT.md.

Running the integration suite

ghk verify --services starts PostgreSQL, RabbitMQ and the broker of stack.messaging (the Pub/Sub emulator, the Service Bus emulator with its database, or a single-node Kafka, as in the golden builds), runs the stack's integration command in its toolchain image and removes everything. The integration tests read DATABASE_URL (postgres://user:pass@host:5432/db) and AMQP_URL (amqp://user:pass@host:5672). The golden builds start postgres:17-alpine and rabbitmq:4-alpine for them. To run them locally:

docker run -d --name ghk-pg -e POSTGRES_PASSWORD=ghk -p 5432:5432 postgres:17-alpine
docker run -d --name ghk-mq -e RABBITMQ_DEFAULT_USER=ghk -e RABBITMQ_DEFAULT_PASS=ghk -p 5672:5672 rabbitmq:4-alpine
export DATABASE_URL=postgres://postgres:ghk@localhost:5432/postgres AMQP_URL=amqp://ghk:ghk@localhost:5672

Schema

All tables live in a configurable PostgreSQL schema. The tests create a fresh schema per test, which keeps them isolated.

The service creates them when it starts, in public, before it serves a request or relays an event: migrate runs from the generated startup code of every stack (and from the separate relay process of Django, Laravel and Rails). It is idempotent, and it runs as one statement that first takes a transaction-scoped advisory lock, so replicas starting together do not race on PostgreSQL's catalog. Set GHK_MIGRATE=false when a migration job or another tool owns the schema and the service connects without DDL rights; then create the tables below before it starts.

CREATE TABLE ghk_aggregates (tenant_id TEXT, aggregate_type TEXT, id TEXT, state TEXT NOT NULL, version INT NOT NULL,
                             updated_at TIMESTAMPTZ NOT NULL DEFAULT now(), PRIMARY KEY (tenant_id, aggregate_type, id));
CREATE TABLE ghk_outbox     (id TEXT PRIMARY KEY, tenant_id TEXT NOT NULL, aggregate_id TEXT NOT NULL, event_type TEXT NOT NULL,
                             payload TEXT NOT NULL, traceparent TEXT, created_at TIMESTAMPTZ NOT NULL DEFAULT clock_timestamp(),
                             claimed_by TEXT, claimed_until TIMESTAMPTZ, published_at TIMESTAMPTZ, attempts INT NOT NULL DEFAULT 0,
                             last_error TEXT, failed_at TIMESTAMPTZ);
CREATE TABLE ghk_idempotency(tenant_id TEXT, key TEXT, status TEXT NOT NULL, response TEXT,
                             created_at TIMESTAMPTZ NOT NULL DEFAULT now(), PRIMARY KEY (tenant_id, key));
CREATE TABLE ghk_inbox      (consumer TEXT, message_id TEXT, processed_at TIMESTAMPTZ NOT NULL DEFAULT now(),
                             PRIMARY KEY (consumer, message_id));
CREATE TABLE ghk_sagas      (id TEXT PRIMARY KEY, tenant_id TEXT NOT NULL, status TEXT NOT NULL,
                             completed_steps TEXT NOT NULL DEFAULT '', updated_at TIMESTAMPTZ NOT NULL DEFAULT now());

Every read and write is filtered by tenant_id, which is a mandatory argument of each operation. The schema also documents the PostgreSQL row-level-security policy to enable as defense in depth.

Components and guarantees

1. Command service handle(tenant, aggregateId, command, payload, idempotencyKey?, traceparent?, aggregateType?). It holds every aggregate of the service (one per feature whose domain kernel is generated); aggregateType selects one and defaults to the feature generated last. In one transaction it:

   1. claims the idempotency key (INSERT … ON CONFLICT DO NOTHING);
   2. loads the aggregate FOR UPDATE;
   3. runs the domain command;
   4. saves the new state and version;
   5. inserts the event into the outbox together with the current traceparent;
   6. stores the response under the key.

   A domain error rolls everything back, including the key. A replayed key returns the stored response without running the command again (status: replayed).

2. Outbox relay publishBatch(workerId, limit) claims rows with FOR UPDATE SKIP LOCKED and a 30 s lease. It publishes each one to the topic exchange <service>.events (service = projectName, shared by every aggregate; routing key = event type) with publisher confirms, message_id = outbox id, and the traceparent and tenant_id headers, then marks it published. A failed publish releases the lease and records the error. After 5 attempts the row is set aside (failed_at). On Pub/Sub, Service Bus and Kafka the same names become topics and subscriptions, and the row is marked published only after the broker accepted the message.

3. Inbox consumer consumes <service>.consumer. It records message_id in ghk_inbox in the same transaction as the handler, so a redelivered message is acknowledged without running the handler again. A handler error rejects the message without requeue, so RabbitMQ dead-letters it to <service>.dlq.

4. Saga orchestrator runs ordered steps. It persists progress after each step. When a step fails, it runs the compensations of the completed steps in reverse order and ends COMPENSATED; otherwise it ends COMPLETED.

5. Telemetry. W3C trace context is propagated from the incoming traceparent to:

   • the command span;
   • the outbox row;
   • the published message header;
   • the consumer span.

   The spans are INTERNAL (command), PRODUCER (publish) and CONSUMER (consume), all in the same trace. Exporters are configured with the standard OTEL_* environment variables.

Integration tests (identical in every stack)

Application layer (wired to the kernel)

Every backend exposes the command service over one HTTP contract:

The tenant comes only from the validated token. X-Tenant-Id is never read and there is no default tenant. An outbox relay worker publishes the outbox every second when a broker is configured (AMQP_URL, or the Pub/Sub or Service Bus variables of DEPLOYMENT.md). It runs in-process, or as manage.py outbox_relay / artisan outbox:relay / rails outbox:relay.

The generated openapi.json (and contracts/<feature>/openapi.json) describes exactly this route:

• command is an enum of the feature's commands;
• Idempotency-Key and traceparent are header parameters;
• the responses are CommandResult (status, aggregateId, eventType, version) and Error ({ "detail": ... }).

HTTP codes written in the scenarios are not copied into the contract, because the route answers with the codes above.

Each backend has unit tests of this contract with a fake command port, and an HTTP integration test against PostgreSQL. It proves that X-Tenant-Id alone gets 401, a command writes one outbox row under the token's tenant, and a replay returns 200. The golden builds generate a two-feature service and run both features' tests.

Backend-for-frontend (stack.role: "bff")

A backend-for-frontend owns no data and publishes no events, so it gets none of the above: no runtime kernel, database, outbox, aggregate or seed.sql, and no asyncapi.json. It is generated for Express, NestJS (11 and 12) and Spring Boot.

{
  "stack": { "language": "java", "framework": "spring-boot", "orm": "none", "role": "bff" },
  "bff": {
    "upstreams": {
      "enrollment": { "url": "http://enrollment-service:8080" },
      "academic": { "urlEnv": "ACADEMIC_API_URL" }
    },
    "timeoutMs": 10000
  }
}

Each feature names the service that owns its commands with @upstream:<name> (the first upstream when it has none). The BFF exposes POST /bff/v1/<feature>/{id}/{command} and forwards it to POST /api/v1/<feature>/{id}/{command}, the route of the service's application layer above, at the URL in the upstream's environment variable (ENROLLMENT_URL by default).

GET /health answers without calling the services. The generated tests run each of these cases against a stub upstream; the goldens express-bff, nestjs-bff, nestjs-12-bff and spring-bff build and run them on every change. docker-compose.yml and .env.example hold only the BFF and the upstream URLs. Terraform, Helm and the CI/CD pipeline are not generated for a BFF yet (the cloud targets would provision a database it does not use): deploy its image next to the services.

MySQL (stack.kernelDatabase: "mysql", NestJS and Express)

The TypeScript kernel also runs on MySQL 8.4 when stack.kernelDatabase is "mysql". It is a separate key on purpose: until 3.2.1 the kernel ignored stack.database, so projects that declare "mysql" there run on PostgreSQL, and they keep doing so. Everything above holds, with these differences:

DATABASE_URL is mysql://user:password@host:3306/database. The service's user needs table privileges in its own database, not global CREATE. The goldens express-mysql and nestjs-mysql run IT1–IT7 against MySQL 8.4. The other runtimes (JVM, Python, .NET, Go, ...) generate contracts and configuration only on MySQL until 3.3.