Skip to main content

Manual Test Path: Pastor Dave -- Protestant Pastor

Persona: Pastor Dave Thompson, 47, senior pastor at a 150-member non-denominational church in suburban Ohio. Not tech-savvy but willing to try tools that save time. Uses Planning Center for attendance tracking. Gets 10-15 calls/week to the church office, half go to voicemail. Wife handles the bulletin. Youth pastor left last year and hasn't been replaced.

Products to test: CWA Starter Chat ($14.95/mo), ITW Premium ($9.95/mo), SermonWise Pro ($19.95/mo) Payment: Live Stripe, real credit card Email: Use a real email you can check (for magic links, MailerLite, etc.)

LESSONS LEARNED (2026-03-30 CEO Walk-Through)​

The CEO tested this exact journey on 2026-03-30 and found 40+ issues that all automated testing missed. Key lessons have been incorporated into this document as [LESSON] annotations. See knowledge/processes/manual-testing-retrospective-2026-03-30.md for the full retrospective.

Critical insight: Every previous test asked "does the code work?" The CEO asked "would Pastor Dave succeed?" These are different questions. When walking this journey, you must think like Pastor Dave, not like an engineer.

Rules for Running This Test​

  1. Use an incognito/private browser window -- no cookies, no cached state
  2. Do NOT look at the codebase before or during the test
  3. Start from Google or the homepage -- do NOT type direct URLs
  4. Read EVERY word on every page as if you have never seen it
  5. Note every moment of confusion -- even brief ones
  6. Check emails within 60 seconds of receiving them
  7. Compare email promises to actual dashboard features
  8. Try to USE the product -- not just verify it loads
  9. Time each step -- if anything takes more than 2 minutes, note it

Part 1: ChurchWiseAI Starter Chat​

Step 1 -- Discovery (churchwiseai.com)​

  • Open churchwiseai.com in an incognito/private window
  • As Pastor Dave, can you tell within 5 seconds what this product does?
  • Is there a pricing CTA above the fold? (Spec says there should be -- may be missing)
  • Click through to /pricing
  • Does the pricing page default to "Chat Only"? (Spec says it should)
  • Can you see the Starter Chat plan at $14.95/mo without scrolling?
  • Click "Get Started" on the Starter Chat card

[LESSON] Cross-Page Consistency Checks (NEW):

  • Count the tool number shown -- is it "39" everywhere? (was stale at "33")
  • Count the agent number shown -- is it correct for each tier? (Starter=2, Pro=4, Suite=4)
  • Check church size language -- does any plan say "50-200 member churches"? (pigeonholes)
  • Check for voice mentions -- does any element say "Voice + Chatbot" for the chat-only plan?
  • Look for upsells -- is there a "Pro Website" upsell on the Starter card? (conversion leak)
  • Check "Book a Strategy Call" -- is it prominent for a $14.95 plan? (wastes founder time)
  • Check founder pricing badge -- is it repeated confusingly on each card?
  • Read every FAQ answer -- does any FAQ mention the wrong phone number? The wrong features?
  • Check crisis FAQ -- does it imply system integration that does not exist ("triggers" vs "shares")?

Step 2 -- Onboarding (/onboard)​

  • Does the plan show "Starter Chat -- $14.95/mo"?
  • Enter church name (use a real-sounding name: "Grace Fellowship Church" or similar)
  • Enter city, state
  • Does the form say "Find Church" or "Search"? (Known issue -- should say "Your Church" or "Church Details")
  • Enter your name, email, phone
  • Is the 14-day free trial mentioned anywhere on the form?
  • Toggle Annual -- does it show $149.50/yr (save $30)?
  • Toggle back to Monthly
  • Check Terms of Service box
  • Is the marketing opt-in checkbox pre-checked? (Should be for US)
  • Click "Continue to Checkout"

[LESSON] Payment Architecture Check (NEW):

  • BEFORE clicking "Continue to Checkout": Open a new tab and check the database -- are there already premium_churches/organization_settings records? (There should NOT be)
  • If you ABANDON checkout at this point: Check DB again. Any orphan records? Any emails sent? (Both should be empty)

Step 3 -- Stripe Checkout​

  • Does it say "14 days free" on the Stripe checkout?
  • Is the product named "ChurchWiseAI Starter Chat" (not "CWA Chat Starter")?
  • Does the description mention voice? (It should NOT for chat-only)
  • [LESSON] Check the CURRENCY -- is it USD? (Stripe Adaptive Pricing may show CAD or other local currency; must be forced to USD)
  • Enter real credit card
  • Complete payment
  • Note the exact charges shown -- screenshot if possible

Step 4 -- Post-Checkout​

  • Do you see a success page?
  • Is there a "Go to Your Dashboard" button? (Spec says there should be)
  • Is there any PewSearch bonus mentioned? (There should NOT be for Starter)
  • Click through to the dashboard

[LESSON] Founder Notification Check (NEW):

  • Did the founder receive a notification (email, Slack, or other) about this new trial/sale? (There should be one)

Step 5 -- Email Verification​

  • Check your email -- did you receive a magic link / welcome email?
  • Who sent it? (From address, brand name)
  • Does the email mention the 14-day trial?
  • Does the email link work and go to the dashboard?
  • Check MailerLite -- were you added to a subscriber list? Which one?

[LESSON] Email Content vs Features (NEW -- CRITICAL):

  • Read every feature mentioned in the welcome email -- is each one actually available at Starter tier?
  • Look for "FAQ management" references -- Starter does NOT have this (was found in Starter Kit email)
  • Check for duplicate content sections -- the welcome email had a duplicate "3 things" section
  • Check for PDF download link -- if the Starter Kit email mentions a PDF, is the link present and working?
  • Click EVERY link in the email -- do they all return 200? (Magic link /auth/magic was returning 505)
  • Look for plain-text fallback URLs -- are they present for users whose email clients strip HTML?

Step 6 -- Dashboard First Impressions​

  • Does the dashboard load without errors?
  • What tabs do you see? Expected for Starter Chat: Overview, Requests, Care, Settings, Upgrade
  • What tabs should be HIDDEN? Calls (voice-only), Analytics (Pro+), Training (Pro+)
  • Is the church name correct?
  • Is the plan shown correctly (Starter Chat)?
  • Click through each visible tab -- do they load?

[LESSON] Tier-Gating Element-Level Check (NEW -- CRITICAL):

  • On EVERY page, look for voice-related elements:
    • Voice badge on agent cards? (SHOULD NOT be visible)
    • Voice greeting field? (SHOULD NOT be visible)
    • "Voice + Chatbot" text anywhere? (SHOULD NOT be visible)
    • SMS phone number field? (SHOULD NOT be visible)
  • Check Training Progress:
    • Does the progress bar include voice-related steps? (Should NOT)
    • Is the denominator based on chat-only steps? (Should be)
    • Does "Customize agents" mark itself complete just by visiting the tab? (Should NOT)
  • Check Document Upload:
    • Is it visible? (Should show lock icon for Starter)
    • Does the lock explain how to upgrade? (Should say "Upgrade to Pro" not just a lock icon)
  • Check Getting Started:
    • Can each step actually be completed at this tier?
    • Do steps track real completion or just tab visits?
  • Check Sharing Links:
    • Are embed code, hosted page link, and share URLs all in one place? (Were scattered across 3 tabs)
  • Check Compliance/Safety section:
    • Does the compliance checklist use reassuring language? (Was scaring users with "Insurance provider notified")

Step 7 -- Chatbot Setup​

[LESSON] Jargon and Comprehensibility (NEW -- CRITICAL):

  • Read every label, heading, and description as Pastor Dave -- note ANY term you would not immediately understand:

    • "Ministry tools" -- is there a tooltip explaining what tools are?
    • "Care Agent" -- is there an explanation of what it does?
    • "Coordinator Agent" -- is there an explanation of what it does?
    • "2 personas" or "specialization areas" -- does it make sense?
    • "Theological lens" -- would Pastor Dave know what this means?
    • "Handoff rules" -- does it imply complex configuration?
    • "Hero Photo URL" -- would a pastor know what to enter?
    • "Human escalation" -- is it findable or buried?

  • Is there a chatbot widget or embed code visible?

  • Can you access the embed widget? (Should be available at all tiers)

  • Is FAQ management visible? (Should be HIDDEN -- Pro+ only)

  • Is document upload visible? (Should be HIDDEN -- Pro+ only)

  • Is widget customization visible? (Should be HIDDEN -- Pro+ only)

  • How many agents do you see? (Should be 2: Care + Coordinator)

  • Are Discipleship and Stewardship agents HIDDEN? (Should be)

[LESSON] Content Quality Checks (NEW):

  • Check doctrinal position labels -- are they human-readable? (Was showing raw variable names like "believers_baptism_only")
  • Check custom practice examples -- do they vary? (Were all showing the same baptism text)
  • Check sermon section -- is it denomination-aware? (Should say "homily" for Catholic, "sermon" for Protestant)
  • Check Safety Guide framing -- is it helpful or scary? (Was framed as legal requirement)

Step 8 -- Chatbot Testing​

  • Open the hosted chat page for your church
  • Send a message: "What time are Sunday services?"
  • Send: "I'd like someone to pray with me"
  • Send: "I'm new and thinking about visiting"
  • Do responses make sense for a church that was just created (no KB yet)?
  • Does the "Powered by ChurchWiseAI" badge show? (Should show -- Starter can't remove it)

Step 9 -- Settings​

  • Can you update church info (name, address, phone)?
  • Can you add team members?
  • Is theological lens/denomination selector visible?
  • Can you set service times?

[LESSON] Suggested Questions Check (NEW):

  • Set suggested questions in settings, then open the chat page -- do the saved values load? (Were not loading)

Part 2: IllustrateTheWord Premium ($9.95/mo)​

Step 10 -- Discovery (illustratetheword.com)​

  • Open illustratetheword.com in the same incognito window
  • Browse some free illustrations
  • Find the signup/premium CTA
  • Is the value proposition clear? Why should Pastor Dave pay $9.95/mo?

Step 11 -- Signup​

  • Click signup / create account
  • What auth method is offered? (Supabase Auth -- email signup)
  • Enter the SAME email as CWA signup (test cross-product experience)
  • Complete signup
  • Check email for confirmation

Step 12 -- Payment​

  • Find the upgrade/premium option
  • Does it show $9.95/mo?
  • Complete Stripe checkout with real card
  • Verify confirmation page

Step 13 -- Premium Features​

  • Can you access premium illustrations?
  • Can you download/copy illustrations?
  • Is there a clear difference between free and premium content?
  • Test search functionality

Part 3: SermonWise Pro ($19.95/mo)​

Step 14 -- Discovery (sermonwise.ai)​

  • Open sermonwise.ai
  • Does it load correctly? (This is a hostname rewrite to churchwiseai.com/sermons)
  • Is the product clear -- what does SermonWise do for Pastor Dave?
  • Find pricing / signup

Step 15 -- Signup​

  • Create account (Supabase Auth)
  • Use the SAME email again
  • Complete signup flow
  • Check email

Step 16 -- Payment​

  • Find upgrade to Pro ($19.95/mo)
  • Complete Stripe checkout
  • Verify confirmation

Step 17 -- Product Testing​

  • Can you create/prepare a sermon?
  • Test the core sermon tools
  • Does the interface make sense for a pastor?

Post-Test Verification​

Stripe Dashboard Check​

  • Log into Stripe -- do you see 3 subscriptions for the same email?
  • Are all 3 showing as active trials (if applicable)?
  • Are the product names correct and professional?
  • [LESSON] Are all charges in USD? (Not localized currency)

MailerLite Check​

  • Is the subscriber in MailerLite?
  • Which lists/groups are they in?
  • Were any automation emails triggered?

Database Check (agent will verify)​

  • premium_churches row created correctly?
  • Correct plan, status, trial dates?
  • Church linked to PewSearch directory (if match found)?
  • [LESSON] Were records created AFTER payment (not during form submission)?
  • [LESSON] Are there any orphan records from abandoned checkout attempts?

Cross-Page Consistency Audit (NEW -- added 2026-03-30)​

After completing the journey, go back and check these consistency items across all pages visited:

ClaimHomepagePricingChatbot PageFAQEmailsDashboard
Tool count (should be 39)
Agent count per tier
Pricing (Starter $14.95)
Trial length (14 days)
Demo phone number
Feature descriptions

Fill in the actual value found at each location. Any inconsistency is a SPEC VIOLATION.

Email Content Audit (NEW -- added 2026-03-30)​

For each email received, fill in this table:

EmailFeature MentionedAvailable at Starter?Link Works?Content Accurate?
Welcome
Starter Kit
Stripe Receipt
Magic Link
MailerLite Welcome

Every "No" in the "Available at Starter?" column for a feature the email explicitly mentions is a SPEC VIOLATION.

Jargon Audit (NEW -- added 2026-03-30)​

Note every term encountered that Pastor Dave would NOT immediately understand:

Term FoundPage/LocationWould Pastor Dave Understand?Has Tooltip?Suggested Alternative

Any customer-facing term without a tooltip that a non-technical pastor would not understand is a PERSONA RISK.

Notes During Testing​

Use this space to capture bugs, confusion, and impressions as you go:

StepWhat HappenedBug/Issue?SeverityRoot Cause (A/B/C/D/E)

Root Cause Key:

  • A = Marketing drift (inconsistent claims across pages)
  • B = Payment flow architecture (sequence/lifecycle issue)
  • C = Tier-gating leakage (feature from wrong tier visible)
  • D = Jargon / comprehensibility (pastor would not understand)
  • E = Email content mismatch (email promises vs actual features)