Skip to content

Eng Review: AI Page Builder for Raklet (Hybrid)

Generated by /plan-eng-review on 2026-05-22 Branch: master Repo: rakletv3 Status: COMPLETE (all 4 sections + outside voice integrations locked) Design doc: rakletai-master-design-20260521-103533-ai-page-builder.md

Step 0 — Scope Challenge

Validation gate skipped (acknowledged risk). Design doc's Assignment said run a 5-customer demand pass BEFORE eng review. CEO chose to proceed anyway. If validation surfaces unexpected signal, large parts of this plan may need re-cutting. Per feedback_review_chain_scope_creep.

Complexity check fires. Hybrid touches: onboarding flow, Social Page editor, new LLM service layer, new template-token rendering layer, prompt management, eval harness, cost controls, scraping pool. Well past 8-file / 2-new-service threshold. Justified by the explicit CEO choice; not silently expanded.

What already exists (reuse, don't rebuild): - Social Page storage (HTML in existing table) — V1 reuses, no new schema. - Onboarding funnel — V1 inserts steps, no new framework. - Member / event / payment data APIs — exist, need audit for AI consumability. - Existing prod deploy pipeline (master → test → prod promotion). - browse-tests Playwright DNA — reusable as a base for production scraping pool (but NOT the same worker pool; see Section 1). - Existing admin T&S workflow — extends for AI generation audit log.

Section 1 — Architecture (LOCKED)

Pipeline diagram

INTAKE                STAGE 1            STAGE 2          STAGE 3              ASSEMBLY
URL  ────┐         ┌─ Design system  ─┬─ Page structure ─┬─ Per-page content ─┬─ Token-wire pages
PDF  ────┼─Input───┤  inference       │  planning        │  generation        │  + nav + theme
Desc ────┘         │  (colors/tone)   │  (which pages,   │  (HTML for each    │  → save as Social
                   │                  │  nav items)      │  page)             │    Page rows
                   │                  │                  │                    │
                LLM call #1        LLM call #2        LLM calls #3..#N    No LLM
                ~2-5 sec           ~2-5 sec           ~5-15 sec each      ~1 sec
                                                      (parallel)

DATA-AWARE TOKENS (resolved server-side at page render time, NOT generation):
  {{members.directory}}  → MembersService at render
  {{events.upcoming}}    → EventsService at render
  {{membership.plans}}   → MembershipService at render

Decisions

D2 — LLM provider: Anthropic Claude + Azure OpenAI (dual provider). - Thin IAiGeneration interface with two implementations: AnthropicGenerationProvider, AzureOpenAIGenerationProvider. - Config flag picks provider per generation. Outputs + cost + latency logged for offline comparison. - Quality eval harness (offline) compares outputs from both providers on the same prompts. Real $/portal comparison, not $/token (provider pricing models differ; Anthropic prompt caching savings don't translate). - Azure OpenAI Service is region-pinned, has separate rate limits, needs explicit model deployments per region. Provisioning is its own setup step.

D3 — Execution model: Streaming progressive reveal. - Server-Sent Events on .NET MVC. EventSource on client. - Stage 1 + Stage 2 emit JSON events ("design system inferred", "page list: Home, About, Members, Events"). Stage 3 streams page-by-page HTML. - Client-side partial HTML parsing: defer paint until each page's HTML is fully streamed (chunk boundaries are not safe DOM boundaries). - Abort = HTTP cancellation cascades to LLM stream cancellation. Stop billing for unreceived tokens. - Backpressure: client closes mid-generation → server cancels LLM stream.

D4 — Cost ceiling + abuse: - Onboarding (all signups, free trial + paid): no per-user cap; bounded by signup volume (~300/mo). Worst case ~$60/mo. - In-product chat: PAID CUSTOMERS ONLY (gated by Raklet plan). - Per-user: 10 generations/hour, 50/day. - Per-org: 200 generations/day hard cap (configurable). - Provider-side: monthly hard cap at Anthropic + Azure of $N (TBD with finance) that auto-throttles. Alerts at 50% / 80% of cap. - New spec addition: onboarding step "what's your website?" captures URL during signup. Fallback to PDF or description if no website.

D5 — URL scraping: Headless browser (Playwright), separate production pool. - Reuse Playwright DNA from browse-tests but run on a SEPARATE production worker pool (queue + workers). Do NOT share with CI fixture pool. - Scrape returns clean HTML + screenshots + extracted text → all three fed to multimodal LLM (Claude or GPT-4o). - Hard timeout 15s per scrape. Retry once on transient failure. - Failure path: scrape fails (404, JS crash, CAPTCHA) → UI says "we couldn't fetch your site, paste a PDF or describe instead." - robots.txt: honor disallow (rare on org sites, but check). - Cost: ~$0.01 per scrape on internal infra.

D6 — Content safety + IP policy: Two-layer. - Layer 1 — Domain-match guardrail (CEO's idea): - Custom-domain email + URL on same domain → scrape allowed. - Free-provider email (Gmail/Hotmail/Outlook/Yahoo) OR domain mismatch → URL scraping disabled. Fall back to PDF or description. - Layer 2 — Provider safety classifiers: - Anthropic + Azure built-in classifiers refuse adult/illegal/violent content. - Log every generation (input + output + user ID + provider) for reactive ban workflow. - Reactive ban via existing admin T&S workflow extension.

D7 — Pipeline staging: Invisible to user. - Stages run sequentially server-side. User sees streaming portal output (nav → pages), not labeled checkpoints. - "Confirmation gate" + "full transparency" + "advanced mode" all deferred to V1.5 or killed; V1 = clean magic.

Eng-only architectural picks (no CEO decision needed)

E1 — Token rendering layer placement: Razor partials. - Raklet is .NET MVC. Tokens like {{members.directory}} get pre-processed server-side at page render into Razor partial invocations (e.g. @Html.Partial("_MembersDirectory")). Existing Raklet partial infra reused. - New IAiTokenResolver service maps token string → partial name + model. - No new templating engine (Liquid/Handlebars). Stay on Razor.

E2 — Portal-level state model: OrgPortalConfig table. - New small table: OrgPortalConfig (one row per org) stores theme colors, nav structure, AI-generation metadata (provider used, timestamps, source-of-truth: URL/PDF/desc). - Per-page HTML still lives in existing Social Page table; the OrgPortalConfig row is the umbrella concept. - Schema change goes in its own migration branch per feedback_migration_branch_first.

E3 — Generation prompt management: Versioned in source. - Prompts live in Raklet.Services/AiGeneration/Prompts/*.txt (or similar). - Each prompt has a version number embedded. Generation log records prompt version used. - Eval harness compares prompt versions across providers.

E4 — Watermark metadata (TODO candidate): - Add <meta name="generator" content="Raklet AI YYYY-MM-DD"> to every AI-generated page. Audit trail for any later IP dispute. ~1 hour add.

E5 — Production scraping pool ≠ browse-tests CI pool. - Separate worker process and queue. Conflating them risks CI traffic flooding the AI flow and vice versa. Per feedback_test_helpers_not_in_bundled_tree principle.

Section 2 — Code Quality (LOCKED)

Eng-only picks (no CEO decision needed)

CQ1 — Cancellation tokens end-to-end. Every async path (SSE → LLM stream → Playwright scrape → DB writes) takes a CancellationToken. Wire from the HTTP request token down to provider SDK calls. Abort = whole chain stops cleanly.

CQ2 — Token resolver defensive defaults. Every token ({{members.directory}}, {{events.upcoming}}, {{membership.plans}}) renders an empty-state component, not a crash, when underlying data is empty/deleted. Test each token against zero, one, many, and deleted-source-row cases.

CQ3 — Prompt versioning in source. Prompts live in Raklet.Services/AiGeneration/Prompts/*.txt. Each file has a version comment at the top. Generation log records the version used. No DB-backed prompt management in V1.

CQ4 — Eval harness project: separate. New Raklet.AiEval console project. NOT under main test suite (different lifecycle, different reviewers, different runtime needs). Runs offline against captured generation fixtures.

CQ5 — Error UX inventory: define before ship. Every failure mode gets a defined user-facing message before V1 ships. Failure modes inventory: - LLM provider 5xx → "Our AI is taking a break, try again in a minute." - LLM refused content → "We couldn't generate this content. Try a different description." - Scrape failed (404/CAPTCHA/JS error) → "We couldn't reach your site. Paste a PDF or describe your org instead." - Cost cap hit → "You've reached today's generation limit. Upgrade or retry tomorrow." - Browser closed mid-generation → resume option ("you have a draft in progress") on next login. - Network drop mid-stream → graceful client-side reconnect, server picks up from last completed page. - No UI gets a raw 500 page.

D8 — Audit log: Azure Storage Table + Blob (LOCKED)

Schema: - Storage Table AiGenerationLog: PartitionKey = org_id, RowKey = {ISO timestamp}_{guid}, columns: user_id, provider (anthropic|azure-openai), model, prompt_version, cost_usd, latency_ms, status (success|refused|error|cancelled), input_kind (url|pdf|desc), surface (onboarding|in-product), input_blob_url, output_blob_url. - Blob container ai-generation-payloads: full input + full output HTML. One blob per generation, URL referenced from table row. - SQL remains source of truth for AI-derived insights that flow back into customer data (org theme colors, member tag suggestions, etc.). A separate analytics job consumes the audit log to write back to SQL — decoupled from the live generation path.

Why this shape: - Lookup pattern is by org or by date, partition+row key fits perfectly. - Storage Table is ~10× cheaper than CosmosDB at this volume, ~3× cheaper than indexed SQL rows for this access pattern. - Blob keeps fat payload off the indexed store; cheap; auditable. - SQL stays clean of high-churn audit data.

Eng note (capture in PR description, not a CEO decision): Storage Table has a 1 MB row limit. Generated HTML can exceed this for multi-page portals → always put HTML in Blob, never inline in the row. The table row is metadata only.

Section 3 — Tests (LOCKED)

Coverage diagram

68 code/flow paths identified, all GAP (plan-stage). 4 CRITICAL regressions (must-have tests, IRON RULE). 8 LLM eval categories. Full diagram in conversation transcript above and in test plan artifact.

D9 — Eval scope: EN-only at V1 (LOCKED)

  • V1 eval harness covers English-only generation.
  • Other locales (DE/FR/SV/ZH) generate via prompt locale variable but are NOT quality-gated by automated eval.
  • Per-locale errors surface from audit log; V2 expands eval matrix based on signup volume.

Test strategy summary

  • Unit + integration (.NET xUnit/NUnit): every component in the coverage diagram with a unit test for happy path + each error branch. Mocked providers via IAiGeneration test doubles.
  • E2E (browse-tests Playwright): 22 flows covering the user journeys in the diagram. New payment-regression-style suite: ai-portal-regression.
  • LLM eval (new Raklet.AiEval console project):
  • 8 eval categories: HTML validity, XSS safety, portal-quality vs human baseline, Anthropic vs Azure comparison, regression on prompt changes.
  • Fixtures: 20 archetype org descriptions (chess club, alumni group, professional association, nonprofit, etc.) in English.
  • Regressions (IRON RULE — no AUQ):
  • Editing an existing (non-AI) Social Page still works after chat UI replaces textarea.
  • Rendering an existing Social Page returns byte-identical output after token resolver layer lands.
  • Interrupted signup resumes cleanly through new onboarding steps.
  • Mobile app reads and renders AI-generated pages correctly.

Test plan artifact

Separate file at ~/.gstack/projects/rakletv3/rakletai-master-eng-review-test-plan-20260522.md for consumption by /qa and /qa-only.

Section 4 — Performance (LOCKED)

Eng-only picks (no CEO decision needed)

P1 — Token resolver batching. Members/events/membership tokens fetch their source data in a single batched query, not N+1. Defensive pattern: every resolver receives an IEnumerable<int> of IDs or a filter, returns the batch.

P2 — Playwright pool sizing. Start with 4 worker processes; scale up if queue depth > 10. Use existing browse-tests pool sizing as a starting point but as a SEPARATE production pool.

P3 — SSE connection pool ceiling. .NET MVC holds a thread per open SSE stream. Cap concurrent in-flight generations at 50 server-wide via semaphore; queue overflow returns "try again in a minute" UX.

P4 — Cost-cap middleware caching. Cache cap status in-memory per request scope; don't re-query Storage Table per LLM call. Invalidate on generation completion.

D10 — Stage 3 timing: Sequential per-page (LOCKED)

  • Pages generate one at a time. User sees page 1 at ~5s, page 2 at ~10s, etc.
  • Wall-clock ~30s for 6 pages. Feels alive.
  • Failure isolation: one page fails → others continue. Failed page can be retried without re-generating the whole portal.
  • Rate-limit safer: smaller per-second token burst.

Production latency budget

  • Stage 1 (design system): 2–5s
  • Stage 2 (page structure): 2–5s
  • Stage 3 (per-page, sequential ×6): ~30s
  • Assembly + DB writes: ~1s
  • Total onboarding portal generation: ~40s p50, ~60s p95.
  • In-product single page: ~5–8s.

Section 5 — Outside Voice Integrations (LOCKED)

Independent Claude subagent surfaced 10 findings against the locked plan. Six were material. Two were strategic tensions for the CEO; four were must-have additions that I'm applying directly.

D12 — Data-aware moat at onboarding: Accept empty states as CTAs (LOCKED)

  • Empty {{members.directory}} → "Invite your first members" CTA + button.
  • Empty {{events.upcoming}} → "Add your first event" CTA + button.
  • Empty {{membership.plans}} → "Set up your first plan" CTA + button.
  • Reframes the data-aware moat: not instant demo wow, but a portal that grows into itself as the org populates Raklet. Empty-state CTAs onboard new orgs into Raklet's core features.
  • Design doc framing for V1 demo material should explicitly NOT promise "look at your live members directory" before the user has members.

D13 — Onboarding latency: Background WebJob + notification (REVISES D3)

  • Onboarding generation moves from synchronous SSE streaming to BACKGROUND WebJob. D3 streaming UX revised: applies only to in-product chat surface.
  • New onboarding flow: user submits intake → "We're building your portal — explore your dashboard while we work, you'll get an email + in-app notification when it's ready (~60 sec)." → user lands on dashboard → notification fires → portal appears.
  • In-product chat surface KEEPS streaming SSE (single-page, ~5-8s, alive reveal makes sense at that timescale).
  • Cost cancellation: if user closes browser mid-onboarding, generation completes in background (already-paid tokens). Output saved, available on next login.
  • New components: WebJob/Hangfire job runner (or existing Raklet job infra — needs eng audit), in-app notification system (audit existing Raklet notification UX), email template.

Must-have additions (applied directly, no CEO decision needed)

OV1 — Token resolver SSTI allowlist (security). IAiTokenResolver ships with an explicit whitelist of approved token names: members.directory, events.upcoming, events.past, events.calendar, membership.plans, membership.signup, org.contact, org.about. LLM-emitted tokens not on the list render as inert text (not Razor partial invocations). New tokens added by explicit code review.

OV2 — Prompt injection hardening (security). - All scraped HTML + extracted PDF text wrapped in XML tags (<untrusted_input>...</untrusted_input>) before being concatenated into the LLM prompt. - System prompt explicitly instructs the LLM to treat content inside <untrusted_input> as data to summarize, not instructions to follow. - Anthropic prompt-caching keys exclude the untrusted-input section so cached prefix can't be poisoned across users. - Adversarial corpus added to eval harness (20+ prompt-injection variants in the XSS safety eval category).

OV3 — Feature flag + cohort split (validation gate). - V1 ships behind a cohort flag: 50% of new signups get the AI flow, 50% continue with current onboarding. Existing customers get the in-product chat for paid orgs only (no cohort split needed — they self-select). - Cohort assignment is sticky per org (consistent experience across sessions). - Kill switch: feature flag can disable AI flow entirely for all signups. - Conversion lift now measurable: AI-cohort signup→first-published-page vs control-cohort signup→first-published-page.

OV4 — Cost ceiling math correction. - Revised model: each portal generation includes initial + average 3 regenerations (per-page edits, theme tweaks, content rewrites). - 300 onboarding signups/mo × ~$0.20 base × 3 (regenerate multiplier) = ~$180/mo onboarding cost. - In-product chat: paid customers × estimated 5 generations/customer/month × $0.05 = bounded by paid count, expect $500–$1500/mo at current scale. - Revised V1 total estimate: $700–$2000/mo, not the original $300–$600. - Monthly hard cap at provider level: target $3000/mo (~50% buffer over high estimate) — confirm with finance before V1 ship.

OV5 — Eval baseline: AI vs current Raklet textarea, not just provider A/B. - New eval category in Raklet.AiEval: side-by-side comparison of AI-generated output vs a hand-built reference page using the existing textarea editor. - Reference pages: 5 archetype orgs, each built once via current textarea (sampled from existing high-quality Raklet customer pages), once via AI generation from the same org's PDF/description. - LLM-judge scores both; humans spot-check the judge's calls. Target: AI output scores ≥ reference 80% of the time. - Without this baseline, "AI passes design QA ≥80%" is unmeasurable.

OV6 — Mobile sequencing fix. - Mobile rendering audit happens BEFORE V1 onboarding ships, not during. - Audit task: render 5 existing Social Pages from the Raklet mobile app; confirm tokens (when added) also render mobile-correctly. - If mobile audit surfaces gap: V1 AI-generated pages are flagged "web-only preview" until mobile rendering ships in V1.1. NOT a CRITICAL regression blocker for V1.

Outside voice findings noted but not changing the plan

  • #3 (2barevents is N=1): Acknowledged. The 5-customer demand pass (deferred by CEO at Step 0) remains the right counter to this risk. Re-flagging it as the highest-priority follow-on action.
  • "Cut V1 in half": CEO already made this call at Step 0 with full knowledge of the scope-creep risk. Not re-litigating. The locked decisions D12 + D13 + OV3 materially reduce the V1 surface anyway (background generation + empty-state CTAs + cohort flag = smaller bet than the original Hybrid as written).

T1 — 5-customer demand pass (CRITICAL pre-ship). What: 2-minute Loom showing 2barevents portal generation; show to 5 named Raklet customers across org types; collect specific quotes about whether they'd re-do their portal with this and what would make it a no-brainer. Why: counters the "2barevents is N=1" risk surfaced by the outside voice and matches the design doc's deferred Assignment. Five quotes also feed V1 marketing copy. Pros: Validates demand with named evidence before maximalist engineering investment. Catches mismatches between CEO's instinct and customer reality. Cons: ~2 days of CEO time. Risks surfacing data that forces V1 re-cut. Context: Was the original deferred Assignment from the design doc; CEO chose to proceed with full eng review anyway at Step 0. Now an explicit TODO so it doesn't drop. Depends on: nothing.

T2 — Mobile rendering audit (V1 pre-ship gate). What: render 5 existing Social Pages from the Raklet mobile app; confirm the token resolver layer (once it lands) also renders mobile-correctly. Document any rendering gaps. Why: per OV6, mobile rendering for AI pages is currently an open question that was inappropriately listed as a CRITICAL regression. Audit first, then we know what V1 ships with. Pros: Catches mobile gap before V1 ships; allows "web-only preview" fallback if needed. Cons: ~1 day of mobile team time. Context: User added Claude access to the mobile repo during /office-hours; that's where this audit lives. Depends on: nothing. Can run in parallel with backend implementation.

T3 — HTML <meta> generator watermark. What: every AI-generated page gets <meta name="generator" content="Raklet AI YYYY-MM-DD"> in its <head>. Generation source (URL / PDF / desc) tagged as a separate <meta> for audit trail. Why: provides legal / IP audit trail for any later content dispute. ~1 hour of eng work for permanent provenance. Pros: Audit trail at near-zero cost. Disambiguates AI-generated content from human-created Social Pages. Cons: Negligible. Single line in the assembly stage. Context: Surfaced as a TODO candidate during eng review; CEO accepted. Depends on: assembly stage of generation pipeline being implemented.

V2 candidates (parked in this doc, not yet filed)

  • In-product portal-level regenerate (existing customers redo whole portal).
  • Drag-and-drop power-user editor (gated on V1 demand signal).
  • Multi-language eval expansion to DE/FR/SV/ZH (currently EN-only at V1).
  • Provider failover (Anthropic ⇄ Azure OpenAI on 5xx).
  • Confirmation gate UX (Stage 1 design preview before Stage 2/3).
  • Streaming progressive reveal at onboarding (revisit if background path underperforms).

Completion Summary

  • Step 0: Scope Challenge — CEO chose full eng review on Hybrid plan (acknowledged scope-creep risk; not re-litigated).
  • Architecture Review: 6 strategic decisions locked (D2–D7). 5 eng-only picks captured (E1–E5).
  • Code Quality Review: 1 strategic decision locked (D8). 5 eng-only picks captured (CQ1–CQ5).
  • Test Review: 68 paths in coverage diagram, all GAP (plan-stage). 4 CRITICAL regressions. 8 LLM eval categories. 1 strategic decision (D9). Test plan artifact written.
  • Performance Review: 1 strategic decision locked (D10). 4 eng-only picks captured (P1–P4).
  • Outside Voice: ran (Claude subagent, codex CLI unavailable). 10 findings; 2 cross-model tensions resolved (D12 + D13 — both REVISED the plan); 4 must-have additions applied (OV1–OV6); 1 finding noted but not re-litigated; 1 ("cut V1 in half") already addressed at Step 0.
  • NOT in scope: written (7 items deferred).
  • What already exists: written (6 reuse points identified).
  • TODOs proposed: 3 V1 pre-ship items captured (T1–T3); 6 V2 candidates parked.
  • Failure modes / critical gaps: 0 ungaurded (every failure mode has defined UX in CQ5).
  • Parallelization: see below.
  • Unresolved decisions: 0.
  • Lake Score: 13/14 recommendations chose complete option (only D5 used existing infra over fancier vendor — appropriate).

Worktree parallelization

Implementation lanes (module-level):

Step Modules touched Depends on
L1 — LLM provider plumbing Raklet.Services/AiGeneration/ (new), Raklet.Eval/ (new) —
L2 — Storage Table audit log Raklet.Services/AiGeneration/, Azure infra —
L3 — Playwright prod scraping Raklet.Services/AiScraping/ (new), Azure WebJobs —
L4 — Token resolver (Razor) Raklet.Web/, Raklet.Services/AiGeneration/ L1
L5 — OrgPortalConfig schema migration EF migrations —
L6 — Cost-cap middleware + paid-gate Raklet.Web/Middleware/ L2
L7 — Onboarding flow changes (background WebJob) Raklet.Web/Onboarding/, Hangfire L1, L5
L8 — In-product chat UI (SSE) Raklet.Backend/Content/scripts/, Raklet.Web/ L1
L9 — Cohort feature flag Raklet.Web/, existing flag infra —

Lanes that can run in parallel from day 1: L1, L2, L3, L5, L9 (all independent). After those land: - L4 (after L1) — depends on provider interface. - L6 (after L2) — depends on audit log shape. - L7 (after L1 + L5) — needs both LLM service and schema. - L8 (after L1) — needs LLM service interface.

Conflict flag: L4 and L7 both touch Raklet.Web/ views — coordinate on PR sequencing or accept rebase friction.

NOT in scope (V1)

  • Drag-and-drop editor (deferred to possible V2; gated on V1 demand signal).
  • In-product portal-level regenerate (existing customers redo whole portal via Wizard) — V2.
  • Custom-LLM-per-org / BYO-key. V2.
  • Multi-LLM router beyond Anthropic + Azure OpenAI. V2.
  • "Confirmation gate" UX between Stage 1 and 2. Killed for V1.
  • Advanced/transparency stage view. V2 at earliest.
  • Marketing landing page for the feature. Separate marketing plan.

Open questions (still)

  • LLM-provider monthly cost ceiling $ — needs finance input.
  • Mobile app surface — is generated portal viewable from Raklet mobile? Audit existing mobile rendering for HTML pages from Social Page table.
  • Multi-language: default to org account locale for LLM generation. Prompt needs locale variant for DE/EN/FR/SV/ZH. Eng to design.
  • Data API consumability audit — do existing Members/Events/Membership APIs return shapes consumable by Razor partial models? Likely needs adapter.
  • Onboarding-step UX wireframe — where exactly does "what's your website" appear? Belongs in /plan-design-review.