Skip to main content

Stripe Testing Harness (21-Tier Coupon Validation)

Summary​

End-to-end validation of every paid tier in the ChurchWiseAI lineup — 21 priceKeys across chat / voice / bundle, monthly + annual — by spinning synthetic Stripe subscriptions using 100% coupons, running the real checkout flow against production, and asserting the webhook inbox provisions the right entitlements. Catches regressions before they reach real customers.

Two cooperating subsystems make this work:

  1. Webhook inbox pattern (PRODUCTION) — the live webhook handler is a thin ack-and-enqueue: verify signature, insert into stripe_webhook_inbox, return 200. A cron worker (/api/cron/process-stripe-webhooks, every 30s) claims pending rows, calls processStripeEvent(), handles exponential backoff retries, and files a P0 founder action item if an event abandons.
  2. Monthly synthetic validator — GH Action (stripe-e2e-validation-monthly.yml, 1st of month, 5am UTC, or manual dispatch) spins 21 test checkouts against the deployed production URL, verifies each webhook round-trips through the inbox, and asserts DB state. Metadata test_customer=true lets the webhook handler short-circuit customer-facing side effects (MailerLite, directory visibility, real emails) while still exercising full provisioning.

Why this exists​

On 2026-04-14 an agent shipped a Stripe harness refactor. Build passed, types passed, CI went green. The webhook started returning 200 on silent provisioning failures — Stripe saw success, a customer paid, nothing provisioned, founder spent the night diagnosing instead of onboarding. The harness + inbox pattern

  • critical-path gate together prevent that failure class from recurring. See knowledge/processes/critical-path-definition.md for the gate, and knowledge/decisions/2026-04-14-stripe-webhook-inbox.md for the refactor.

Flow​

Phase tracker​

  • Phase 0 — Spike: 100%-coupon real-mode validation proved the concept
  • Phase 1 — Validator library (src/lib/stripe-synthetic-validator.ts) + monthly GH Action
  • Phase 1.5 — Webhook inbox pattern (2026-04-14) — handler is thin ack-and-enqueue, cron worker processes with retries + P0 alerts
  • Phase 2 — Full 21-tier Playwright coverage running green monthly (in progress — some tiers still flaky)
  • Phase 3 — Baseline ingestion + automated drift alerting (ingest route scaffolded at /api/cron/stripe-e2e-validation-ingest)
  • Phase 4 — Founder-facing status page for harness health

Code files​

Authoritative list in frontmatter. Quick map:

  • Live webhook handler — src/app/api/stripe/webhook/route.ts (thin inbox enqueue; exports processStripeEvent() imported by cron worker)
  • Inbox processor — src/app/api/cron/process-stripe-webhooks/route.ts (claims + retries, 30s cron, exponential backoff 0/30s/2m/10m/30m/hourly up to 10 attempts)
  • Founder inbox UI — src/app/founder/[token]/webhook-inbox/page.tsx
    • api/founder/webhook-inbox/*
  • Harness library — src/lib/stripe-synthetic-validator.ts
  • Harness Playwright specs — e2e/stripe-e2e-validator.spec.ts, e2e/stripe-synthetic-validator.spec.ts (gated by RUN_COUPON_VALIDATION=1 so they don't run in normal CI)
  • Monthly GH Action — .github/workflows/stripe-e2e-validation-monthly.yml (cron 0 5 1 * *, workflow_dispatch, BASE_URL=https://churchwiseai.com)
  • Critical-path gate — .github/workflows/critical-path-gate.yml (blocks PRs touching stripe files without artifact or override)
  • Ingest cron — src/app/api/cron/stripe-e2e-validation-ingest/route.ts (reads harness results into baselines — Phase 3, scaffolded)

Tests​

See knowledge/tests/registry.yaml entries:

  • stripe-e2e-validation-monthly — monthly cron + manual dispatch
  • stripe-live-checkout — critical-path gate entry (requires artifact on PR)

Decisions​

  • 2026-04-14-stripe-webhook-inbox — move from inline provisioning inside the webhook handler to inbox + cron worker pattern. Forced by a live-customer provisioning loss that day.
  • 2026-04-14-critical-path-gate — add GH Action that blocks PR merges touching Stripe files without a live-flow Playwright artifact.

Gotchas​

  • test_customer=true metadata: the live webhook handler detects this on any event and short-circuits customer-facing side effects (no MailerLite sync, no real welcome email, no directory flip). Full provisioning logic still runs. Remove or alter the check and you will start mailing real addresses from harness runs.
  • processStripeEvent() is imported across files. Renaming or changing its signature breaks the cron worker silently. The critical-path gate will block the PR — do not try to merge around it.
  • 21 priceKeys means 21 subscriptions per run. Always clean up test customers after the harness — otherwise Stripe Dashboard fills with noise. The validator library does this, but a crashed run may leave orphans.
  • Monthly validator targets production, not a preview deploy. Anything the harness asserts must be true on https://churchwiseai.com. If a PR is unmerged but the validator claims red, suspect production drift not code.
  • Inbox backoff is durable but finite. 10 attempts ≈ 6 hours. A P0 row in founder_actions is created on abandon — the founder-facing inbox UI surfaces these. Do not silently delete abandoned rows.
  • The harness only validates happy-path provisioning. Cancel, upgrade, downgrade, failed payment, and tax edge cases are not yet in the matrix. Those are tracked in Phase 2.

Recent activity​

Reconciler (Phase 4 of the doc system) will populate this section automatically from merged PRs that touch any code-files: entry above.