Skip to main content

Next.js 16 — Streaming Soft-404s and ISR Blockers

Verified: 2026-06-11. Two related Next.js 16 App Router production failures, found and fixed across PewSearch, ITW, and churchwiseai-web on 2026-06-09/10. Both are invisible in dev and in "build passes" checks — they only show in HTTP status codes and the prerender manifest.

Problem 1 — Streaming locks the status at 200 (soft-404)​

When ANY Suspense boundary is in the render path (loading.tsx, a layout <Suspense>, or the streaming shell of a complex layout), a dynamically-rendered route STREAMS. The HTTP status is locked at 200 before notFound() / redirect() / permanentRedirect() resolve, so every invalid dynamic URL returns 200 + a "Not Found" body. permanentRedirect() is demoted to a client-side NEXT_REDIRECT digest inside the 200 body, which Googlebot ignores.

Consequence: Googlebot records soft-404s, keeps junk URLs, wastes crawl budget, and the junk volume drags down sitewide quality — real pages get deindexed. Found via the founder's GSC Pages report on ITW (46 Soft 404 + 7,270 crawled-not-indexed); same shapes confirmed live on PewSearch. Refs: vercel/next.js discussion #76501.

Disproven non-fixes (each tested on ITW via rebuild + local prod-server curl)​

  • Removing loading.tsx (8 files) — still 200.
  • Removing the layout <Suspense> — still 200.
  • Disabling the proxy — still 200.
  • export const dynamic = 'force-dynamic' — PewSearch [vanity] already had it and still 200'd.
  • "Call notFound() before any await" only works if NO streaming boundary exists anywhere upstream — unreliable in real layouts.

The canonical fix — validate in proxy/middleware BEFORE render​

Rewrite certain misses to a tiny static route with an explicit status (static = no streaming = the 404 sticks). From pewsearch/web/src/middleware.ts:

function rewriteTo404(request: NextRequest): NextResponse {
  return NextResponse.rewrite(new URL("/_not-found-404", request.url), {
    status: 404,
  });
}

// Verdict handling — ONLY 404 on a CERTAIN miss:
switch (verdict.verdict) {
  case "missing":  return rewriteTo404(request);
  case "redirect": return NextResponse.redirect(new URL(verdict.to, request.url), 308); // retired slug
  case "exists":
  case "unknown":  // any DB error / ambiguity → fall through, never wrong-404
  default:         return null;
}

/_not-found-404/page.tsx is a tiny static page that calls notFound() (real 404 + styled UI). Existence checks (pewsearch/web/src/lib/proxy-existence.ts) use the SAME slug resolution the page uses — set-based routes via pure functions (unslugifyBook, getTypeBySlug, …), DB-backed routes via one scoped Supabase REST HEAD + count=exact (no supabase-js in the edge bundle).

Safety contract (218K+ real church pages): only 404 when CERTAIN invalid; on ANY query error or ambiguity return unknown and fall through to normal rendering. Retired slugs → 308 to canonical, never 404. Scope the middleware to exact path shapes; deeper/unrelated paths fall through untouched.

Problem 2 — ISR blockers (X-Vercel-Cache: MISS everywhere)​

Three independent gotchas, proven on PewSearch PR #55 then re-confirmed on churchwiseai-web PR #846:

  1. A dynamic API (headers() / cookies()) in the ROOT layout forces EVERY route dynamic. Prerender manifest ends up with ZERO routes; all revalidate exports AND next.config s-maxage header rules become silently inert. Both repos had await headers() in RootLayout for brand/subdomain chrome toggling. Fix: remove it; default export const revalidate = 3600 in the root layout (pages with their own config override).
  2. A dynamic-params route with export const revalidate does NOT enter the prerender manifest without generateStaticParams. An empty export function generateStaticParams() { return []; } registers on-demand ISR (see pewsearch/web/src/app/denominations/[slug]/page.tsx).
  3. usePathname() reports the BROWSER URL on middleware-rewritten requests, not the rewritten path — so a pathname-based chrome-hider works for direct /s/foo but NOT for {slug}.pewsearch.com → /s/{slug} rewrites (SSR renders the chrome). Fix: a marker element rendered by the rewritten segment's LAYOUT (layouts DO follow the rewrite) + CSS — PewSearch: body:has(.pro-website-page) .site-chrome { display: none }; CWA: <BrandBareMarker> in 7 brand segment layouts + body:has(.brand-bare) .cwa-chrome { display: none }. Do NOT pair with a window.location check in the component — hydration mismatch.

Secondary traps when pages flip from dynamic to static:

  • Any client component calling useSearchParams() without a Suspense boundary crashes the build of newly-prerendered pages — self-wrap the component (CWA's only unwrapped consumer was BrandSalesChatbot).
  • Client-only host-gated widgets (ChatWidget, CookieConsent) must move behind a client host gate (<ApexOnly>) since the server no longer knows the host.

ISR blocker checklist (run when cache is MISS everywhere)​

  1. Get-Content .next/prerender-manifest.json after a build — count routes / dynamicRoutes. Zero = something forces global dynamic.
  2. Grep the root layout (and every layout above the route) for headers() / cookies() / connection().
  3. Dynamic-params route with revalidate missing from the manifest → add empty generateStaticParams.
  4. After the fix builds: any useSearchParams() without Suspense will fail the build — fix, don't suppress.
  5. If chrome/widgets depended on request headers, convert to layout-marker + :has() CSS or a client host gate.

How to verify​

  • Local prod server (pnpm build && pnpm start) — it reproduces the streaming bug AND runs middleware. Vercel previews are 401-protected, so curl can't verify them. curl junk URLs (expect real 404) AND a broad sweep of valid URLs (expect 200 — never wrong-404 a real page).
  • prerender-manifest.json, not response headers — local next start still emits s-maxage for revalidate-tagged routes even when Vercel will treat them as dynamic.
  • Prod ISR: repeat-request the same URL → X-Vercel-Cache: MISS → HIT + rising Age. Beware: a 200 right after merge can be the OLD deployment; gate on response CONTENT (real <title>), never status/size alone.
  • Chrome hiding on rewrites: Playwright computed-style check (display: none) against the REAL subdomain/brand host — preview hosts can't exercise hostname middleware. Local subdomain testing of endsWith() host checks is impossible (local Host carries :port); --host-resolver-rules tests silently leak to real prod if a redirect target domain isn't mapped.

Who fixed what​

RepoChangeWhere
sermon-illustrations (ITW)Soft-404 proxy fix (first proof of the pattern)src/proxy.ts, commit e7f9ce2, 2026-06-09
pewsearchSoft-404 middleware fix (churches/directory/vanity/care, slug_redirects → 308)commit 31a75a4, 2026-06-09
pewsearchCoverage extended to /directory/<state> + /denominations/<slug>; ISR root-cause foundPR #54, 2026-06-09/10
pewsearchISR unblocked: 0 → 92 prerender routes + 6 on-demand patterns; chrome via :has() markerPR #55, 2026-06-10
churchwiseai-webSoft-404: audit found CWA/wiseaiagency already clean; only SermonWise /templates fixed via scoped middlewarePR #836, 2026-06-09
churchwiseai-webISR unblocked: 0 → 900 prerender routes + 13 dynamic patterns; BrandBareMarker + ApexOnly; BrandSalesChatbot Suspense self-wrapPR #846, 2026-06-10
sermon-illustrations (ITW)ISR: root layout audited clean (no headers()), no change needed—

All live-verified 2026-06-09/10: junk-URL probes return hard 404 on all three prod sites; PewSearch and CWA serve X-Vercel-Cache: HIT / PRERENDER. Related memory: project_nextjs16_streaming_soft404, project_nextjs_isr_blockers_and_rewrite_pathname. DECISION_LOG entries: 2026-06-09 (ITW soft-404 root cause, fixes shipped) and 2026-06-10 (PR #54/#55/#846).