Knowledge > Runbooks > Technical Ops > Running Tests

Running Tests

Test suites across all active codebases. Each codebase uses Playwright for E2E and either Vitest or Node's built-in test runner for unit/integration.

All E2E tests run against production URLs by default (or localhost with a dev server). There is no staging environment — see feedback_test_against_production.md.

Quick Reference

Suite	Command	Runtime	When to run
Knowledge integrity	`pnpm test` in `C:\dev\knowledge\`	~5s	Before every knowledge commit
CWA unit/integration	`pnpm test:all` in `churchwiseai-web/`	~10s	Before every PR
CWA E2E (fast)	`pnpm test:e2e` in `churchwiseai-web/`	~2min	Before every PR
CWA delivers (lifecycle)	`npx playwright test e2e/delivers/` in `churchwiseai-web/`	~5min	Pre-launch, weekly
CWA agent sim	`pnpm test:sim` in `churchwiseai-web/`	~10min	Before voice/chatbot changes
PewSearch E2E	`pnpm test:e2e` in `pewsearch/web/`	~2min	Before every PR
ITW E2E	`pnpm test:e2e` in `sermon-illustrations/`	~2min	Before every PR

1. Knowledge Tests (`C:\dev\knowledge\`)

Runner: Vitest | Count: 99 tests across 10 test files

cd C:\dev\knowledge
pnpm test

Test files

File	What it checks
`tests/index-completeness.test.ts`	Every doc has required frontmatter (title, description, last-verified). Hard-fails on missing `description:`.
`tests/links-integrity.test.ts`	All relative markdown links resolve to real files. INDEX.md is generateable.
`tests/secret-detection.test.ts`	No actual API keys, tokens, or secrets in doc content.
`tests/derive-e2e.test.ts`	Derive pipeline runs without errors for all sources.

2. ChurchWiseAI Web (`C:\dev\churchwiseai-web\`)

Unit & Integration Tests

Runner: Node built-in test runner (tsx) | Location: src/test/

cd C:\dev\churchwiseai-web
pnpm test:all          # all unit + integration tests
pnpm test:chatbot      # same (alias)
pnpm test:sermon       # sermon generation only

File	What it tests
`unit/moderation.test.ts`	Content moderation blocklist patterns
`unit/content-moderation.test.ts`	Full moderation pipeline
`unit/faq-matcher.test.ts`	Canned response semantic matching
`unit/denomination-packs.test.ts`	Denomination-specific response packs
`unit/lens-vocabulary.test.ts`	Theological lens vocabulary filters
`unit/tool-config.test.ts`	Chatbot tool access by tier
`unit/sermon-generation.test.ts`	SermonWise AI generation
`unit/social-*.test.ts`	ShareWiseAI — auth, crypto, OAuth, platforms, pricing, AI
`integration/chatbot-tools.test.ts`	Chatbot tool call integration (hits Supabase)
`e2e/multi-turn-chat.test.ts`	Multi-turn chatbot conversation flow

E2E Tests

Runner: Playwright | Location: e2e/ | Config: playwright.config.ts

cd C:\dev\churchwiseai-web
pnpm test:e2e                      # all specs, auto-starts dev server on :3002
pnpm test:e2e:ui                   # Playwright UI mode (visual debugging)
pnpm test:e2e:update-snapshots     # Refresh visual snapshots

# Run against production (skip dev server)
BASE_URL=https://churchwiseai.com pnpm test:e2e

Browsers: Desktop Chrome, iPad Mini, iPhone 12 (3 parallel projects)

Root-level specs

Spec	What it tests
`admin.spec.ts`	Admin dashboard tabs, RBAC, data display
`api.spec.ts` + `api-new-routes.spec.ts`	API route availability and responses
`care.spec.ts`	Care messaging subscribe/broadcast
`crud-admin-*.spec.ts`	Admin CRUD: care, settings, training
`crud-chatbot.spec.ts`	Chatbot config CRUD
`crud-contact.spec.ts`	Contact form submission
`crud-social-*.spec.ts`	ShareWiseAI: campaigns, platforms, posts, settings
`edge-auth-rbac.spec.ts`	Role-based access edge cases
`edge-chatbot.spec.ts`	Chatbot edge cases
`edge-cross-site.spec.ts`	Cross-site script injection, header checks
`edge-input-validation.spec.ts`	Form input validation
`edge-rate-limiting.spec.ts`	Rate limit enforcement
`env-integration.spec.ts`	Required env vars are set
`journey-*.spec.ts`	End-to-end user journeys
`legal-pages.spec.ts`	Privacy policy, terms pages
`onboard.spec.ts`	Church onboarding flow
`p2-security.spec.ts`	Security headers, auth gates
`p4-seo-metadata.spec.ts`	SEO meta tags, structured data
`p5-content-accuracy.spec.ts`	Marketing copy accuracy vs DB
`p7-branding.spec.ts`	Brand consistency
`production-smoke-all-sites.spec.ts`	Smoke test all 4 properties
`security.spec.ts`	Security hardening checks

`e2e/delivers/` — Product Delivers (Persona-Based)

Philosophy: Test OUTCOMES not STATUS CODES. "Did the pastor find the right plan?" not "Did /pricing return 200?"

# Fast lifecycle (no live API calls) — run every PR
npx playwright test e2e/delivers/ --grep-invert "@live" --project=desktop

# Theological chatbot (weekly / pre-launch — hits real AI APIs)
npx playwright test e2e/delivers/theological/ --grep "@live" --project=desktop

# Known-bug specs only (check which are now fixed)
npx playwright test e2e/delivers/known-bugs/ --project=desktop

# Cross-property (production URLs required)
TEST_PRODUCTION=1 npx playwright test e2e/delivers/cross-property/ --project=desktop

Folder	Specs	What
`delivers/churchwiseai/`	01–08	Full CWA lifecycle: discovery → understanding → signup → onboarding → dashboard → chatbot config → theological setup → upgrade
`delivers/personas/`	9 personas	Skeptical/edge-case buyers: Pastor Ezekiel (AI skeptic), Karen (privacy paranoid), Deacon Bob (board evaluation), committee buyer, Mark (mega-church IT), Pastor Maria (Catholic), Pastor Ruth (tiny church), Pastor Steve (burned before), Youth Pastor Jake (mobile)
`delivers/theological/`	7 specs	Vocabulary safety for Baptist, Catholic, Orthodox, Pentecostal, Reformed; crisis escalation; landing page claims
`delivers/cross-property/`	2 specs	PewSearch→CWA upsell, ITW→SermonWise cross-sell
`delivers/consistency/`	4 specs	Feature claims accuracy, price matching, product_knowledge alignment, stale copy detection
`delivers/known-bugs/`	6 specs	Documented UX bugs (`test.fixme`) — become regression guards when fixed

`e2e/experience/` and `e2e/quality/`

experience/cta-clarity.spec.ts — CTA button copy and placement
experience/error-messages.spec.ts — User-facing error message quality
quality/chatbot-denomination-accuracy.spec.ts — Chatbot responses per denomination

Agent Simulation Tests

Runner: Custom tsx runner | Location: tests/agent-sim/

Drives real conversations against the live chatbot or voice agent runtime, records transcripts + tool traces, and asserts against outcomes.

cd C:\dev\churchwiseai-web
pnpm test:sim            # all cases (chat + voice)
pnpm test:sim:chat       # chat agent only
pnpm test:sim:voice      # voice agent only
pnpm test:sim:dry        # dry run — parse cases only, no execution

# Single case
npx tsx tests/agent-sim/runner.ts --case chat_prayer_confidential

# By tag
npx tsx tests/agent-sim/runner.ts --tag safety

Cases defined in YAML files under tests/agent-sim/cases/
Rubrics in tests/agent-sim/rubrics/
Results saved to tests/agent-sim/results/
Global invariant: admin tokens and service keys must NEVER appear in agent responses

3. PewSearch (`C:\dev\pewsearch\web\`)

Runner: Playwright | Location: e2e/

cd C:\dev\pewsearch\web
pnpm test:e2e
pnpm test:e2e:ui
pnpm test:e2e:update-snapshots

Spec	What
`admin.spec.ts`	PewSearch admin dashboard
`chatbot.spec.ts`	Chatbot widget on church pages
`checkout.spec.ts`	Premium Page checkout
`church-detail.spec.ts`	Church listing page
`claim.spec.ts`	Church claiming flow
`crud-*.spec.ts`	Admin CRUD: add church, settings, training
`demo.spec.ts`	Demo page
`directory.spec.ts`	Search, filter, pagination
`legal.spec.ts`	Legal pages
`delivers/01–07`	Full lifecycle: discovery → browse → find listing → claim → premium → admin → upsell

4. IllustrateTheWord (`C:\dev\sermon-illustrations\`)

Runner: Playwright | Location: e2e/

cd C:\dev\sermon-illustrations
pnpm test:e2e
pnpm test:e2e:ui
pnpm test:e2e:update-snapshots

Spec	What
`auth.spec.ts`	Signup, login, magic link
`browse.spec.ts`	Topic/scripture/emotion browse
`checkout.spec.ts`	Premium subscription checkout
`content-gating.spec.ts`	Free vs premium content access
`illustration-detail.spec.ts`	Illustration detail page
`favorites.spec.ts`	Save/unsave illustrations
`lenslines.spec.ts`	Theological lens filtering
`responsive.spec.ts`	Mobile/tablet layout
`crud-*.spec.ts`	Auth, checkout, contact, favorites, profile, search CRUD
`audit.spec.ts`	Content audit checks
`delivers/01–05`	Full lifecycle: discovery → browse → free use → premium → daily use

Tag Filtering (CWA E2E)

npx playwright test --grep "@theological"
npx playwright test --grep "@lifecycle"
npx playwright test --grep "@live"           # requires real API calls
npx playwright test --grep-invert "@live"    # skip AI API calls (faster)
npx playwright test --grep "@known-bug"      # expected failures
npx playwright test --grep "@cross-property" # multi-domain

Known-Bugs Specs

Tests in delivers/known-bugs/ across all codebases use test.fixme(). They document confirmed UX issues and serve as regression guards — when a bug is fixed, the test should be changed to test() and moved out of known-bugs/.

Current documented bugs:

cwa-demo-phone-labels.spec.ts — Demo page phone label copy issue
cwa-no-post-signup-explanation.spec.ts — No explanation after onboard signup
cwa-pricing-paralysis.spec.ts — Pricing page causes decision paralysis
itw-irrelevant-on-this-day.spec.ts — "On This Day" content not always relevant
pewsearch-search-returns-all.spec.ts — Search with empty query returns all results
sermonwise-homily-missing.spec.ts — Homily content type missing from SermonWise

Quick Reference​

1. Knowledge Tests (C:\dev\knowledge\)​

Test files​

2. ChurchWiseAI Web (C:\dev\churchwiseai-web\)​

Unit & Integration Tests​

E2E Tests​

Root-level specs​

e2e/delivers/ — Product Delivers (Persona-Based)​

e2e/experience/ and e2e/quality/​

Agent Simulation Tests​

3. PewSearch (C:\dev\pewsearch\web\)​

4. IllustrateTheWord (C:\dev\sermon-illustrations\)​

Tag Filtering (CWA E2E)​

Known-Bugs Specs​

See Also​