Skip to content

MRR Dashboard: Redis-backed cache + Scheduled WebJob pre-warm

Context

The /Customer/MrrTransactionSummary endpoint (consumed by crm.raklet.com/Customers) timed out on production after we shipped the dashboard. The prior commit on this branch (fix/mrr-dashboard-perf, PR #13705) deduped and parallelised the Stripe lookups, dropping cold-compute time roughly 10×. That fixes the acute symptom (a request now completes within the request timeout) but the dashboard still recomputes from scratch on every cache miss because of two latent bugs:

  1. The existing cache key is effectively unique per request. MrrTransactionSummary builds it as crm-mrr-summary:{since:O}:{until:O} where since/until are DateTime.UtcNow-derived with millisecond precision. Two requests one ms apart get two different keys; the 10-minute HttpRuntime.Cache insert is essentially write-only.
  2. HttpRuntime.Cache is per-AppDomain. Raklet.Crm runs on multiple Azure instances. Each instance maintains its own cache, and the cache is wiped on every app-pool recycle. There is no shared, persistent warm cache.

Goal: every user request to the default 30-day dashboard view hits a pre-warmed Redis entry, so cold-compute work happens only inside a WebJob — never on the user request path.

Approach

Three coordinated changes:

  1. Extract the dashboard compute into Services/MrrDashboardService.cs so both the controller and a WebJob can invoke it. The controller becomes thin (parse inputs → load rows → filter/sort/paginate → return JSON). The service owns the Stripe enrichment plus a single WarmDefaultCache() entrypoint.
  2. Switch the cache from HttpRuntime.Cache to CacheService (Redis) with a stable key parameterised by days only, so the warmer and the controller agree on the slot.
  3. Add a new Scheduled5mins trigger + handler in Raklet.WebJobs.Secondary that calls MrrDashboardService.WarmDefaultCache() every 10 minutes. Only the 30-day window is pre-warmed (per scope decision).

The synchronous fallback stays: if Redis is empty (first deploy, WebJob failed, Redis cold), the controller still computes inline so the dashboard is never dead.

Files to modify

New files

  • Services/MrrDashboardService.cs — owns:
  • All private helpers currently in CustomerController.cs (GetApplicationFeeMap, GetOrganisationStripeConnectAccountMap, GetConnectAccountMetricsMap, GetConnectAccountLifetimeTxnMap, GetPlanMap, GetSubscriptionAggregateMaps, GetCustomerLifetimeRevenueMap, BuildMrrTxRows, ApplySorting, and the model types MrrTxOrganisationRow, MrrTxCachePayload, MrrTxPlanData, MrrTxRowViewModel, MrrTxSubscriptionSnapshot, MrrTxConnectMetrics, MrrTxSubscriptionAggregate).
  • New public API:
    • MrrDashboardService.GetOrBuildSummary(int days, DateTime? customStartUtc, DateTime? customEndUtc, bool bypassCache) → MrrTxCachePayload
    • MrrDashboardService.WarmDefaultCache() → computes the 30-day payload and writes it to Redis under the canonical key
    • MrrDashboardService.GetCanonicalCacheKey(int days) → string (e.g. "crm-mrr-summary:days:30") — used by both warmer and reader
  • Raklet.WebJobs.Secondary/CrmDashboard/MrrDashboardWarmFunctions.cs (placement mirrors the existing Raklet.WebJobs.Secondary/Email/EmailFunctions.cs convention):
  • public static async Task TriggerMrrDashboardWarm([QueueTrigger("trigger-mrr-dashboard-warm")] string message, TextWriter log) — guards with WebJobUtil.IsDeploymentTime, acquires a 12-minute Redis lock isProcessingMrrDashboardWarm (matching the pattern in EmailFunctions.TriggerEmailStats), then calls new MrrDashboardService(new RakletDb()).WarmDefaultCache().

Edited files

  • Raklet.Crm/Controllers/CustomerController.cs
  • MrrTransactionSummary shrinks to: parse query params → var payload = MrrDashboardService.GetOrBuildSummary(days, startUtc, endUtc, bypassCache) → apply plan / txOnly / search filters, sort, paginate → return Json(...). The filtering/sorting/pagination logic stays in the controller because it's pure presentation and varies per request.
  • All the static helpers / model types move out (deletions in this file).
  • Raklet.WebJobs.Scheduled5mins/Functions.cs (dictionary at lines 13–50)
  • Add one entry: {"trigger-mrr-dashboard-warm", new[] { 0, 10, 20, 30, 40, 50 }} — fires every 10 minutes.
  • Services/Services.csproj — add <Compile Include="MrrDashboardService.cs" /> (per CLAUDE.md: non-SDK csproj, manual registration required).
  • Raklet.WebJobs.Secondary/<csproj> — add <Compile Include="CrmDashboard\MrrDashboardWarmFunctions.cs" />.

Cache key + TTL design

  • Default days view (the only view we pre-warm): key = "crm-mrr-summary:days:30". The warmer overwrites every 10 minutes; Redis TTL = 12 minutes (one warm-cycle of grace). The cached MrrTxCachePayload carries CachedAtUtc, SinceUtc, UntilUtc so the response can show freshness, even though the key itself is parameterless beyond days.
  • Custom days other than 30 (e.g. 7, 60, 90): same key shape "crm-mrr-summary:days:{days}", 12-minute Redis TTL. Not pre-warmed, so the first hit per days value is a cold compute; subsequent hits within 12 minutes are warm. Acceptable per scope decision.
  • Explicit startUtc / endUtc (custom range): key = "crm-mrr-summary:range:{since:yyyyMMddHHmm}:{until:yyyyMMddHHmm}" rounded to the minute, 5-minute TTL. Not pre-warmed.
  • bypassCache=true: deletes the relevant key, then recomputes and stores.

Reused utilities

  • Services/CacheService.cs — Get<T>, Set<T>(key, value, TimeSpan expiry), Purge(key). JSON-serialised via Newtonsoft; lazy ConnectionMultiplexer already initialised. MrrTxCachePayload is a POCO of simple types — serialises fine.
  • Raklet.WebJobs.Secondary/Email/EmailFunctions.cs (TriggerEmailStats, ~line 1817) — reference implementation for the new handler (lock-guard + direct Service invocation, no HTTP).
  • Services/WebJobUtil — IsDeploymentTime(log) re-queues the message during deploy windows. Used in every existing handler; copy that pattern.
  • Services/QueueService — already used by Scheduled5mins/Functions.cs to drop trigger messages; no change required there beyond the dictionary entry.

Verification

After implementing, validate end-to-end:

  1. Local build: scripts/dev/build-fast.ps1 should build Services, Raklet.Crm, Raklet.WebJobs.Scheduled5mins, Raklet.WebJobs.Secondary (or whichever the dependency analyser picks up).
  2. Controller smoke test (local Raklet.Crm, IIS):
  3. Cold call: https://crm.raklet.local/Customer/MrrTransactionSummary?days=30&bypassCache=true → completes, response shows CachedAtUtc ≈ now.
  4. Warm call: same URL without bypassCache → completes in well under a second, same payload, CachedAtUtc unchanged.
  5. Confirm Redis key exists (redis-cli GET crm-mrr-summary:days:30, or Azure Portal) — returns a JSON blob.
  6. WebJob smoke test (local Scheduled5mins console):
  7. Wait for a 0/10/20/30/40/50 minute boundary OR manually enqueue: QueueService.SendMessage(new QueueType("trigger-mrr-dashboard-warm")).
  8. Watch Secondary WebJob console: handler picks up the message, logs lock acquisition, runs WarmDefaultCache, logs duration.
  9. Confirm Redis key is refreshed (CachedAtUtc advances).
  10. Multi-instance verification (post-deploy, prod-like): hit /Customer/MrrTransactionSummary against two CRM instances behind the load balancer; both should return the same CachedAtUtc (proves shared Redis read), and the response should be sub-second from both even immediately after one instance recycles.
  11. Failure modes:
  12. Stop the WebJob → user request still works (synchronous fallback computes inline, writes Redis).
  13. Stop Redis → user request still works (the controller logs the Redis failure via the existing CacheService swallowed-exception path and falls through to compute; do not introduce a hard dependency).
  14. Confirm via a deliberate connection-string break in a local config.
  15. Stripe load: in prod, watch the Stripe dashboard for request volume. With one WebJob warming every 10 minutes, the per-customer subscription enrichment runs 6× per hour total across the entire cluster (instead of N-instances × per-user request). Should be well under Stripe rate limits.

Out of scope (defer to a later PR)

  • Pre-warming for days ≠ 30. Add only if dashboard analytics show meaningful usage.
  • Per-customer Stripe data caching with longer TTL (would let the WebJob refresh incrementally). The current full-payload cache is simpler and covers the cold-start goal.
  • A /CustomerWebhook/InvalidateMrrCache endpoint to purge the warm cache when a subscription state change happens. Not needed at 10-minute staleness target.