Verification evidence — gherkin-ai 3.2.2

This document lists what is verified for each release, how it is verified, and how to reproduce it. Every result below comes from a command you can run yourself; nothing is scored by hand.

Run of record: 2026-10-30, gherkin-ai 3.2.2 (branch fix/3.2.2). The CLI under test is the branch's build (npm run build, dist/). CI runs the same goldens against the packaged tarball (npm pack, installed in a clean directory and passed as GHK_CLI), the file release.yml publishes. The 3.2.1 and earlier reports are in the git history of this file.

Environment:

• Ubuntu on WSL 2 with Docker Engine 29.9.2 (Linux, as on the CI runners), Node.js 26.13;
• the official toolchain images listed below;
• PostgreSQL postgres:17-alpine (MySQL mysql:8.4 for the MySQL kernel) and RabbitMQ rabbitmq:4-alpine for the integration suites, plus the official emulators or brokers of the broker variants: the Pub/Sub emulator (google-cloud-cli:emulators), the Service Bus emulator with Azure SQL Edge, and apache/kafka:4.1.0 (single-node KRaft).

1. Golden builds: every stable stack

scripts/golden-build.js takes the following steps for each stack in scripts/golden-stacks.js:

1. generates a project with the CLI, from scripts/golden-feature.feature (and, for backends and frontends, a second feature in the same service);
2. builds it inside the stack's official image;
3. runs its generated unit, contract and BDD tests;
4. for backends, runs the runtime integration suite (IT1–IT7) and the HTTP integration tests against real PostgreSQL and the stack's broker;
5. for backends, builds the production image (Dockerfile), starts it against a fresh database (the service must create the runtime kernel's tables itself, as in a first deployment) next to an OpenTelemetry collector, requires its liveness and readiness probes to answer 200, sends one command with a JWT (HTTP 202), and requires the service's spans to reach the collector over OTLP.

Until 3.2.1 the image started against the database the integration suite had just migrated, which hid that no backend created its tables at startup (CHANGELOG 3.2.1).

A stack passes only if every step exits with code 0. Pending BDD steps are allowed; failing or erroring tests are not. Deployment and CI goldens run their tools instead (sections 1.4 and 1.5).

1.1 Backends

The MySQL kernel (stack.kernelDatabase: "mysql", TypeScript runtimes only) runs IT1–IT7 against MySQL 8.4, and its production image creates its tables in a fresh MySQL database.

In the full run FastAPI and Django (and the three FastAPI broker variants) failed their BDD tests: pytest 9.2, released after 3.2.1, removed an internal that pytest-bdd 9.0 uses. The generated pyproject.toml now requires pytest<9.2 (commit 5aa629c); the times above are from the rerun. The two MySQL goldens were rerun on the final build (29332e0 makes the MySQL kernel opt-in).

BFF role

stack.role: "bff" (new in 3.2.2) generates a backend-for-frontend without a runtime kernel. Its golden builds it, runs its tests, and checks that it has no database seed and no AsyncAPI file and that its routes are under /bff/v1/.

1.2 Broker adapters

The same backend, build and tests, with the runtime kernel on another broker. IT1–IT7 run through the broker ports.

In the full run the .NET Kafka golden failed IT6 once, while the broker logged Coordinator load in progress; it passed on the unchanged rerun (140 s). It is recorded here as an intermittent failure, not as fixed.

1.3 Frontends and contracts

1.4 Deployment targets

Nothing is applied. Terraform runs init -backend=false, validate and fmt -check; Helm runs lint --strict and template for every environment.

1.5 CI/CD pipelines

Result: 70 of 70 goldens pass: 21 backends (2 new, on MySQL), 4 BFF stacks (new), 20 broker variants, 9 frontend and contract stacks, 12 deployment (4 new) and 4 CI goldens. All 41 backend configurations (21 backends and 20 broker variants) pass IT1–IT7, and every backend image creates its tables in a fresh database. Eight goldens failed in the full run and passed on the rerun: five Python goldens after the pytest cap, the two MySQL goldens on the final build, and .NET Kafka without changes (sections 1.1 and 1.2). CI repeats this on every change against the packaged tarball, with one job per stack (.github/workflows/ci.yml, jobs package and golden-builds), and release.yml publishes only what that pipeline verified.

1.6 Database over TLS

Not rerun for 3.2.2 (results of the 3.2.1 run). node scripts/golden-build.js --db-tls reruns the backends against a PostgreSQL that accepts only TLS connections (self-signed certificate, hostssl only), with DATABASE_URL=...?sslmode=require, as managed databases enforce. The integration suites and the production image all connect over TLS.

With 3.2.1 the production image also creates its tables over TLS, in a fresh database, when it starts. In the 3.2.0 run, the first pass found that the generated TypeScript tests and the Prisma 7 adapter passed the URL to node-postgres unchanged, which treats sslmode=require as verify-full; they add uselibpqcompat=true since then.

What the runtime integration suite proves

The same seven tests run in every backend and on every broker. The contract is in RUNTIME-KERNEL.md.

2. Reference projects

Sixteen reference projects (ghk-test-projects) cover realistic multi-stack combinations: backend plus frontend, contracts, a real broker, an adversarial security harness, and a Quarkus banking service created with ghk init --enterprise --industry banking. On 2026-10-30 every feature of each was regenerated with the 3.2.2 build. Each was then built and tested with the golden-build recipes (golden-build.js <stack> --project <dir>, from a clean tree), including the runtime integration suite (IT1–IT7 and the HTTP API) for its backend. Last, its production image had to start against a fresh database, create its tables, answer its liveness and readiness probes with 200 and accept a command.

Result: 16 of 16 projects pass, with 28 of 28 stack checks. ts04 (Django) was regenerated once more after the pytest cap (section 1.1) before this run.

A second ghk generate over each regenerated project changes no file, .ghe/inventory.json included (with 3.2.1 every run rewrote its dates), and ghk validate passes in all 16. 14 of the 16 projects declare stack.database: "mysql": they stay on PostgreSQL, because the MySQL kernel is opt-in (stack.kernelDatabase). ghk lint scores are the same as with 3.2.1. ghk converge is the same in 15 projects; in ts08 it goes from 68% to 77%, on the same tree, because Traceability now finds the feature's domain names shorter than 6 characters in the code (50% to 100%). No project reports contract drift. Each project's README lists its own results.

In the 3.2.0 run of record, ghk verify --services was also run on a copy of each of the 16 projects, the way a team runs it, and all 16 exited 0 (between 16 s and 438 s). It was not repeated for 3.2.1 or 3.2.2.

3. Regenerating an implemented service

The goldens above generate new projects. tests/golden/implemented-service.test.ts (CI job golden-implemented) covers the other case: a service that a team has already implemented on top of generated code. Its fixture is an LMS service as it was before its 3.2 upgrade (.ghk-implemented, an edited 3.0 kernel, its own aggregate, step definitions and application.yml), with the inventory of its last generation. The table is from the 3.2.1 run; the CI job repeats it on every change.

The same upgrade was repeated on the real service: mvn clean test passes right after upgrade --apply (24 of 24), and its integration suite passes (7 of 7 kernel, 2 of 2 service).

4. The CLI itself

5. What this report does not claim

• Agent outcome quality. Whether an agent implements a feature better with gherkin-ai than without it depends on the model, the agent and the task. gherkin-ai ships a harness to measure this on your own tasks (ghk bench, ENTERPRISE.md §18) and does not publish benchmark numbers of its own.
• Production readiness of generated code. Generated projects are verified starting points. They build, their tests pass, the runtime kernel passes its integration contract and the image starts and exports traces. The business logic in the BDD steps is left pending, for your team to implement.
• Cloud deployments. The Terraform and Helm targets are validated and linted, not applied: no golden creates cloud resources. The CI/CD pipelines are checked against actionlint and the official schemas, not run in the providers.
• Stacks outside the matrix. Combinations not listed in section 1 generate contracts and prompts only, and are labeled experimental.

6. Reproducing

npm ci && npm run build
node scripts/golden-build.js                 # all stacks (Docker required)
node scripts/golden-build.js java rust       # selected stacks
node scripts/golden-build.js --db-tls java   # against a TLS-only PostgreSQL
node scripts/golden-build.js nestjs --keep   # keep the generated project for inspection
node scripts/golden-build.js --no-image go   # skip the production image step
node scripts/golden-build.js --list          # stacks and images
npm test                                     # the CLI's own suite

Each stack's full log is written to the system temp directory as ghk-golden-<stack>.log.