Voice Line Provisioning Runbook

Last updated: 2026-03-27 Owner: Voice Agent Engineer

This runbook covers how to provision a new customer's voice agent phone number. The voice agent runs on LiveKit Cloud with Telnyx providing phone numbers for customer lines.

Infrastructure Overview (One-Time Setup, Already Done)

Telnyx Account

Field	Value
Account	churchwiseai
Status	Identity verified, paid
API Key	Stored in `knowledge/.env` as `TELNYX_API_KEY`

Telnyx FQDN Connection

Field	Value
Connection ID	`2925081061861885519`
Name	"LiveKit ChurchWiseAI"
FQDN	`5u9xu5ysoly.sip.livekit.cloud:5060`
FQDN Record ID	`2925083349418510207`
Transport	TCP (critical — UDP will not work)
Auth Username	`churchwiseai`
Auth Password	`CWA-livekit-2026!`
default_primary_fqdn_id	Set to the FQDN record ID above

Important: The Auth Username and Password above are credentials Telnyx uses to authenticate outbound SIP registrations from Telnyx to LiveKit. They are configured on the Telnyx side only. LiveKit SIP inbound trunks must NOT have authentication set — see the note in Step 3.

Telnyx Outbound Voice Profile

Field	Value
Profile ID	`2925086295615079772`
Name	"ChurchWiseAI"
Whitelisted	US + CA

LiveKit Cloud

Field	Value
Project	`cwa-voice-9x077mph` (project ID `p_5u9xu5ysoly`)
CLI	`C:\dev\lk.exe` (authenticated, config at `~/.livekit/cli-config.yaml`)
Agent ID	`CA_pX3Me4NK6qK8`
Agent Region	`us-east`
SIP Endpoint	`5u9xu5ysoly.sip.livekit.cloud`
Agent Name	`churchwiseai-voice`

Automated Provisioning (New Default)

As of 2026-04-04, provisioning is fully automated. When a customer buys any voice plan:

The Stripe webhook creates a church_voice_agents stub (status: 'pending_setup')
It immediately calls provisionVoiceLine() from src/lib/voice-provision.ts
The function searches Telnyx for a "friendly" local number (tries patterns: 0000, 5000, 4000, 000, 500, 400, 200, 100... then first available)
Buys the number, creates LiveKit SIP trunk + dispatch rule, updates DB to status: 'active'
If auto-provisioning fails, falls back to sending the founder a manual provisioning email

Manual override (if you need a specific number or area code):

curl -X POST https://churchwiseai.com/api/admin/provision-number \
  -H "Content-Type: application/json" \
  -d '{"token": "$FOUNDER_TOKEN", "churchId": "...", "areaCode": "734"}'

Response includes phoneNumber, trunkId, dispatchRuleId.

Per-Customer Provisioning Steps (Manual / Reference)

Step 1: Search for Available Numbers

Find a local number in the customer's area code:

curl -s "https://api.telnyx.com/v2/available_phone_numbers?filter[country_code]=US&filter[national_destination_code]=414&filter[limit]=5" \
  -H "Authorization: Bearer $TELNYX_API_KEY" | python -m json.tool

Replace 414 with the desired area code. For Canadian numbers, use filter[country_code]=CA.

Step 2: Buy the Telnyx Number

curl -X POST "https://api.telnyx.com/v2/number_orders" \
  -H "Authorization: Bearer $TELNYX_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "phone_numbers": [{"phone_number": "+1XXXXXXXXXX"}],
    "connection_id": "2925081061861885519"
  }'

The connection_id assigns the number to our FQDN connection, which routes calls to LiveKit's SIP endpoint.

Step 3: Create LiveKit SIP Inbound Trunk

Each customer number gets its own SIP trunk in LiveKit.

Critical: Do NOT set auth_username or auth_password on inbound trunks.

Telnyx FQDN connections route PSTN calls as standard SIP INVITEs without attaching SIP credential headers. If the LiveKit trunk has authentication configured, LiveKit challenges the incoming INVITE for credentials it will never receive and rejects every call — callers hear "not assigned." Security comes from Telnyx owning the DID (only calls from Telnyx's SIP infrastructure reach LiveKit), not from SIP passwords. This matches the working Twilio trunk pattern (ST_Xa3Bp9aixRFP), which has no auth set.

from livekit.api import LiveKitAPI
from livekit.protocol.sip import (
    CreateSIPInboundTrunkRequest,
    SIPInboundTrunkInfo,
)

lk = LiveKitAPI(url=LIVEKIT_URL, api_key=LIVEKIT_API_KEY, api_secret=LIVEKIT_API_SECRET)

trunk = await lk.sip.create_sip_inbound_trunk(
    CreateSIPInboundTrunkRequest(
        trunk=SIPInboundTrunkInfo(
            name='Church Name',
            numbers=['+1XXXXXXXXXX'],
            krisp_enabled=True,
            # DO NOT set auth_username or auth_password here.
            # Telnyx does not send SIP credentials on inbound PSTN calls.
        ),
    )
)
print(f"Trunk ID: {trunk.sip_trunk_id}")

Or via the setup_sip.py script in voice-agent-livekit/scripts/.

Step 4: Create Dispatch Rule for the Trunk

from livekit.protocol.sip import (
    CreateSIPDispatchRuleRequest,
    SIPDispatchRule,
    SIPDispatchRuleIndividual,
)
from livekit.protocol.room import RoomConfiguration, RoomAgentDispatch

dispatch = await lk.sip.create_sip_dispatch_rule(
    CreateSIPDispatchRuleRequest(
        trunk_ids=[trunk.sip_trunk_id],
        rule=SIPDispatchRule(
            dispatch_rule_individual=SIPDispatchRuleIndividual(room_prefix='call-')
        ),
        name='Church Name Dispatch',
        room_config=RoomConfiguration(
            agents=[RoomAgentDispatch(
                agent_name='churchwiseai-voice',
                metadata='{"source": "phone"}',
            )]
        ),
    )
)
print(f"Dispatch Rule ID: {dispatch.sip_dispatch_rule_id}")

Step 5: Update Database

UPDATE church_voice_agents
SET twilio_phone_number = '+1XXXXXXXXXX',
    twilio_phone_sid = 'telnyx',
    updated_at = now()
WHERE church_id = '<church_uuid>';

Note: The column is named twilio_phone_number for historical reasons. The value works regardless of provider.

Step 6: (Optional) Add to PHONE_REGISTRY in session.py

Not required — the DB fallback lookup in resolve_route() works automatically. But for faster routing (avoids a DB query per call), add to the PHONE_REGISTRY dict in voice-agent-livekit/session.py:

"+1XXXXXXXXXX": "<church_uuid>",  # Church Name (Telnyx)

Step 7: Deploy (If Code Changed)

C:\dev\lk.exe agent deploy --project cwa-voice --silent

If only DB changes were made (Steps 5-6 skipped the code change), no deploy is needed.

Step 8: Post-Provisioning Checklist

Run through every item before marking provisioning complete:

Verify trunk has no authentication set. Run C:\dev\lk.exe sip inbound list --project cwa-voice and confirm the new trunk's Authentication column is blank. A non-blank value means every inbound call will be rejected. If auth is set, clear it via the LiveKit REST API (see Troubleshooting below).
Verify dispatch rule exists for the new trunk in the LiveKit Cloud dashboard or via C:\dev\lk.exe sip dispatch list --project cwa-voice.
Verify the DB record — church_voice_agents.twilio_phone_number matches the provisioned number and status = 'active'.
Founder tests a call from Canada. Canadian callers dial the number from a Canadian phone (or use a VoIP service) to confirm cross-border PSTN routing works. Telnyx sometimes requires carrier-specific configuration for CA→US routing. This test must be done by a human — automated tests cannot confirm PSTN connectivity from Canada.
Verify correct church greeting plays — the right church name is spoken in the greeting.
Check voice_call_logs in Supabase for a new record after the test call.
Verify the call summary and transcript are populated in the call log.

Current Phone Numbers

Number	Church	Provider	LiveKit Trunk ID
+14144007103	Medhanialem Ethiopian Evangelical Church	Telnyx	ST_LWgmiBSwTk7P
+17473897673	Melvindale Church of God (Pastor Deb Moelker)	Telnyx	ST_3KMJ5YEjF2fF
+18886030316	Sales (toll-free)	Twilio	ST_jKtes8Md7dEZ
+14696152221	Demo (US)	Twilio	ST_jKtes8Md7dEZ
+13658254095	Demo (CA)	Twilio	ST_jKtes8Md7dEZ
+13658253552	Spare (unassigned)	Twilio	ST_hvf5m2RXfEiS
+14697288326	Company fallback (unroutable calls)	Telnyx	N/A (TeXML app)

Note: Twilio numbers route via Twilio SIP trunk to LiveKit. New customer numbers use Telnyx (cheaper, simpler direct SIP).

Company Fallback Number (+14697288326)

This number is NOT routed through LiveKit. It uses a Telnyx TeXML application that plays a professional message and hangs up. Used when a call can't be routed to a specific church agent.

Field	Value
Number	`+14697288326`
Telnyx Phone ID	`2927180561589994835`
TeXML App ID	`2927180743153026408`
TeXML App Name	"ChurchWiseAI Fallback Greeting"
Webhook URL	`https://churchwiseai.com/api/voice/fallback`
Vercel Env Var	`TELNYX_FALLBACK_DIAL_NUMBER` (set on churchwiseai-web production)

The webhook endpoint is at churchwiseai-web/src/app/api/voice/fallback/route.ts. It returns TeXML (TwiML-compatible XML) that says a brief message and hangs up.

Outbound SIP Trunk — X-Telnyx-Username Header (Day 3 Action)

The outbound SIP trunk ST_X3n9jxR55VrB (provisioned Day 1, 2026-04-28) needs a X-Telnyx-Username header set to force credential-based authentication on the Telnyx side. Without this header, Telnyx authenticates by IP whitelist only, which is weaker and can fail if LiveKit's egress IP shifts.

Source: tmp/research-livekit-outbound.md section "Telnyx outbound trunk gotchas" Decision: Worktree A Day 2 spec — Day 3 integration runs this with founder present.

Why this matters

Telnyx outbound credential connections authenticate via:

IP allowlist (default) — fragile when LiveKit Cloud changes egress IPs
X-Telnyx-Username SIP header — explicitly presents the credential connection username on every outbound INVITE, stable regardless of IP

The X-Telnyx-Username header approach is preferred for production.

Step A — Identify the outbound credential username

The Telnyx outbound credential connection 2948197312620398250 ("CWA Verticals Platform Outbound") was provisioned on Day 1. The username is stored in:

tmp/telnyx-outbound-creds.env  (NOT committed to git)

If that file is unavailable, retrieve via Telnyx API:

curl -s "https://api.telnyx.com/v2/credential_connections/2948197312620398250" \
  -H "Authorization: Bearer $TELNYX_API_KEY" | python -m json.tool | grep username

Step B — Set the header on the LiveKit outbound trunk

Run from any clean checkout of voice-agent-livekit/:

# Load the username from the creds env file
USERNAME=$(grep TELNYX_OUTBOUND_USERNAME /c/dev/churchwiseai-web/voice-agent-livekit/tmp/telnyx-outbound-creds.env 2>/dev/null | cut -d'=' -f2)

if [ -z "$USERNAME" ]; then
  echo "ERROR: TELNYX_OUTBOUND_USERNAME not found — retrieve from Telnyx API (see Step A above)"
  exit 1
fi

# Set the header on the outbound trunk
/c/dev/lk.exe sip outbound update \
  --project cwa-voice \
  --id ST_X3n9jxR55VrB \
  --header "X-Telnyx-Username=$USERNAME"

If lk sip outbound update does not support --header (verify via lk sip outbound update --help), use the LiveKit REST API directly:

TOKEN=$(/c/dev/lk.exe sip outbound list --project cwa-voice --curl 2>&1 | grep "Bearer " | head -1 | sed "s/.*Bearer //" | sed "s/'.*//")

curl -s -X POST \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  --data "{
    \"sipTrunkId\": \"ST_X3n9jxR55VrB\",
    \"replace\": {
      \"headers\": {\"X-Telnyx-Username\": \"$USERNAME\"}
    }
  }" \
  "https://cwa-voice-9x077mph.livekit.cloud/twirp/livekit.SIP/UpdateSIPOutboundTrunk"

Step C — Verify

/c/dev/lk.exe sip outbound list --project cwa-voice

Confirm the ST_X3n9jxR55VrB trunk shows X-Telnyx-Username in its headers field.

Step D — Post-verification smoke test

After setting the header, perform a smoke-test outbound dial to confirm Telnyx accepts the credential. See the Day 3 verification checklist in knowledge/specs/day2-verticals-platform/05-DAY3-INTEGRATION.md.

Troubleshooting

"Number not assigned" error

Check that the number is assigned to the FQDN connection (2925081061861885519) in Telnyx
Verify default_primary_fqdn_id is set on the connection (this was the root cause of initial setup failure)
Confirm transport is TCP (not UDP)

Agent not answering calls

Run C:\dev\lk.exe agent status --project cwa-voice — agent may be sleeping (auto-wakes on call) or crashed
Check LiveKit Cloud dashboard for agent logs
Redeploy: C:\dev\lk.exe agent deploy --project cwa-voice --silent

Wrong church greeting plays

Check church_voice_agents.welcome_greeting for that church_id
Verify PHONE_REGISTRY in session.py maps to the correct church UUID
If not in PHONE_REGISTRY, check church_voice_agents.twilio_phone_number matches the called number

Call not routing to the agent

Verify the SIP inbound trunk exists in LiveKit with the correct number
Verify a dispatch rule exists for that trunk

Verify the trunk has NO authentication set. Run C:\dev\lk.exe sip inbound list --project cwa-voice and check the Authentication column for the trunk. If it shows credentials, clear them via the LiveKit REST API — the lk.exe CLI cannot clear auth (it omits empty strings from the update payload). Use this curl pattern (replace trunk ID, name, and number):

TOKEN=$(C:/dev/lk.exe sip inbound update --project cwa-voice --id ST_XXXX --curl --auth-user "x" --auth-pass "x" 2>&1 | grep "Bearer " | sed "s/.*Bearer //" | sed "s/'.*//")
curl -s -X POST \
  -H "Authorization: Bearer $TOKEN" \
  -H 'Content-Type: application/json' \
  --data '{"sipTrunkId":"ST_XXXX", "replace":{"name":"Church Name", "numbers":["+1XXXXXXXXXX"], "authUsername":"", "authPassword":"", "krispEnabled":true}}' \
  https://cwa-voice-9x077mph.livekit.cloud/twirp/livekit.SIP/UpdateSIPInboundTrunk

Toll-free numbers not connecting reliably

Toll-free numbers (~20% connect rate on direct SIP trunk). Use TwiML-to-SIP bridge as fallback: Twilio webhook calls /api/voice/twiml which generates a <Dial><Sip> response pointing to LiveKit's SIP endpoint. Set VOICE_PIPELINE=livekit on the Twilio number's webhook.

Cost Reference

Item	Cost
Telnyx DID (local US/CA)	~$1/mo + $0.005/min inbound
Twilio DID (local)	$1.15/mo + $0.0085/min
Twilio toll-free	$2.15/mo + $0.0130/min
LiveKit Cloud	Per-minute agent compute (see LiveKit pricing)
Cartesia TTS	Included in LiveKit agent (Cartesia plugin)
Deepgram STT	Included in LiveKit agent (Deepgram plugin)

Telnyx is preferred for new customer numbers — cheaper and simpler (direct SIP, no TwiML bridge needed).

Which Numbers Go Where (Telnyx vs Twilio)

Provider	Used for	Credential connection	LiveKit trunk IDs
Telnyx	All new customer numbers (2026-04-04+)	`2925081061861885519` (FQDN inbound) / `2948197312620398250` (outbound transfers)	Per-customer (created by `provisionVoiceLine()`)
Twilio	Legacy numbers only — demo lines + toll-free	SIP trunk via Twilio elastic SIP	`ST_Xa3Bp9aixRFP` (main, locked) / `ST_hvf5m2RXfEiS` (test)

Rule: A newly provisioned church or funeral home ALWAYS gets a Telnyx number. Twilio numbers are read-only — no new Twilio numbers are purchased. The locked inbound trunk ST_Xa3Bp9aixRFP is PROTECTED — do not touch it (see churchwiseai-web/CLAUDE.md).

Outbound-Trunk First-Dial Certification

Source: memory/feedback_telnyx_outbound_three_requirements.md (2026-04-29 Day 3 P0 #7)
Background: The outbound credential connection 2948197312620398250 had outbound_voice_profile_id: null. Telnyx silently rejected every INVITE with an internal 403/D35 — no CDR generated, no SIP error visible to LiveKit, no MDR record. From LiveKit's side: CreateSIPParticipant returned a participant_id, but the participant never joined the room. ~3 hours lost diagnosing this because no certification step existed.

Before declaring a new outbound trunk production-ready, ALL THREE checks below must pass. Do not trust a 200 OK from CreateSIPParticipant alone — that only confirms LiveKit accepted the request, not that Telnyx carried the call.

The Three Requirements

Requirement 1 — Credential connection active + credentials set

# Replace $TELNYX_API_KEY with the key from knowledge/.env (do NOT log/commit the key).
# Replace $CONN_ID with the credential connection ID.

CONN_ID="2948197312620398250"
curl -H "Authorization: Bearer $TELNYX_API_KEY" \
  "https://api.telnyx.com/v2/credential_connections/$CONN_ID" \
  | jq '{active: .data.active, has_user: (.data.user_name != null)}'
# Both must be true: {"active": true, "has_user": true}

Requirement 2 — Outbound voice profile bound + active + region whitelisted

curl -H "Authorization: Bearer $TELNYX_API_KEY" \
  "https://api.telnyx.com/v2/credential_connections/$CONN_ID" \
  | jq '.data.outbound.outbound_voice_profile_id'
# Must NOT be null. Expected value: "2925086295615079772" (ChurchWiseAI profile, US+CA).
# If null: PATCH with nested object (see PATCH gotcha below).

To verify the profile is active and covers the right regions:

PROFILE_ID="2925086295615079772"
curl -H "Authorization: Bearer $TELNYX_API_KEY" \
  "https://api.telnyx.com/v2/outbound_voice_profiles/$PROFILE_ID" \
  | jq '{enabled: .data.enabled, whitelisted: .data.whitelisted_destinations}'
# enabled must be true; whitelisted must include "US" and "CA".

Requirement 3 — Outbound DID bound to THIS credential connection

DID="+12268830000"  # replace with your outbound DID
curl -H "Authorization: Bearer $TELNYX_API_KEY" \
  "https://api.telnyx.com/v2/phone_numbers?filter[phone_number]=$DID" \
  | jq '.data[0].connection_id'
# Must equal $CONN_ID exactly.
# If null or different: the DID is on the wrong connection. Telnyx will D35-reject the INVITE.

The PATCH-Needs-Nested-Objects Gotcha

When updating outbound_voice_profile_id, the PATCH body MUST be nested under "outbound". A flat PATCH returns HTTP 200 but silently ignores the field:

# WRONG — HTTP 200 returned but field is silently ignored
curl -X PATCH "https://api.telnyx.com/v2/credential_connections/$CONN_ID" \
  -H "Authorization: Bearer $TELNYX_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"outbound_voice_profile_id": "2925086295615079772"}'

# CORRECT — nested under "outbound"
curl -X PATCH "https://api.telnyx.com/v2/credential_connections/$CONN_ID" \
  -H "Authorization: Bearer $TELNYX_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"outbound": {"outbound_voice_profile_id": "2925086295615079772"}}'

# Always re-GET after PATCH to verify the write took effect — do NOT trust the 200 alone.
curl -H "Authorization: Bearer $TELNYX_API_KEY" \
  "https://api.telnyx.com/v2/credential_connections/$CONN_ID" \
  | jq '.data.outbound.outbound_voice_profile_id'
# Must now equal "2925086295615079772"

Required: ONE End-to-End Test Dial Before Production

After confirming all three API checks pass, dial a Telnyx echo/test number from the new trunk before routing any customer calls through it. This is the only ground-truth proof that the full SIP path (LiveKit → Telnyx → carrier → callee) works end-to-end.

# This is now automated via the daily cron. To run it manually:
curl -H "Authorization: Bearer $CRON_SECRET" \
  https://churchwiseai.com/api/cron/outbound-trunk-cert
# Expected: {"status": "pass", "elapsed_ms": <N>, ...}
# Failure: check "failure_detail" in the response and the Telnyx Mission Control SIP Call Flow Tool.

Automated guard against drift: src/app/api/cron/outbound-trunk-cert/route.ts runs this certification dial daily at 04:00 ET. If it fails, a P0 founder_action_items row is created and reportError fires. Monitor for OUTBOUND-TRUNK-CERT-* action items in the founder dashboard.

Do NOT Trust `lk sip outbound list` Alone

Critical: lk sip outbound list --project cwa-voice shows only the LiveKit half of the outbound wiring — the trunk exists, the trunk ID is correct, the headers are set. The Telnyx half (credential connection state, outbound voice profile binding, DID-to-connection binding) is completely invisible from LiveKit-side.

Treat them as two separate systems requiring independent verification:

What `lk sip outbound list` tells you	What it does NOT tell you
LiveKit trunk `ST_X3n9jxR55VrB` exists	Whether Telnyx accepted the credential
Trunk headers (e.g. `X-Telnyx-Username`) are set	Whether `outbound_voice_profile_id` is set on the credential connection
`sip_trunk_id` is correct	Whether the outbound DID is bound to the connection
The trunk is configured	Whether Telnyx will actually carry the call

Never declare an outbound trunk "working" based solely on LiveKit-side state. Always verify Telnyx-side state from the Telnyx API or Mission Control Portal.

Diagnosing Silent-Drop (No MDR Record)

If CreateSIPParticipant returns a participant_id but the participant never joins the room:

Check Telnyx Mission Control → Account Activity → Debugging → SIP Call Flow Tool. This is the only ground-truth source for carrier-level rejection codes (D35, 403, etc.). MDR endpoints filter to call-control connections — credential connection traffic does NOT appear in MDR.
Run all three API checks above. outbound_voice_profile_id: null is the most common cause.
Check the PATCH-needs-nested-objects gotcha if you recently updated the profile.
Verify OUTBOUND_TRUNK_ID env var is set correctly in LiveKit Cloud secrets — empty string causes LiveKit to use its default outbound trunk (which may not be configured for Telnyx).

The automated voice-health cron (/api/cron/voice-health) now also validates all three Telnyx requirements every 15 minutes. Check the WatchTower dashboard for telnyx_outbound check failures.

Infrastructure Overview (One-Time Setup, Already Done)​

Telnyx Account​

Telnyx FQDN Connection​

Telnyx Outbound Voice Profile​

LiveKit Cloud​

Automated Provisioning (New Default)​

Per-Customer Provisioning Steps (Manual / Reference)​

Step 1: Search for Available Numbers​

Step 2: Buy the Telnyx Number​

Step 3: Create LiveKit SIP Inbound Trunk​

Step 4: Create Dispatch Rule for the Trunk​

Step 5: Update Database​

Step 6: (Optional) Add to PHONE_REGISTRY in session.py​

Step 7: Deploy (If Code Changed)​

Step 8: Post-Provisioning Checklist​

Current Phone Numbers​

Company Fallback Number (+14697288326)​

Outbound SIP Trunk — X-Telnyx-Username Header (Day 3 Action)​

Why this matters​

Step A — Identify the outbound credential username​

Step B — Set the header on the LiveKit outbound trunk​

Step C — Verify​

Step D — Post-verification smoke test​

Troubleshooting​

"Number not assigned" error​

Agent not answering calls​

Wrong church greeting plays​

Call not routing to the agent​

Toll-free numbers not connecting reliably​

Cost Reference​

Which Numbers Go Where (Telnyx vs Twilio)​

Outbound-Trunk First-Dial Certification​

The Three Requirements​

The PATCH-Needs-Nested-Objects Gotcha​

Required: ONE End-to-End Test Dial Before Production​

Do NOT Trust lk sip outbound list Alone​

Diagnosing Silent-Drop (No MDR Record)​