Skip to content

Browse-tests fixture cleanup

A delayed-message queue + WebJob handler that purges orphaned test fixtures from the QA org after a browse-tests run finishes.

Why this exists

Browse-tests (smoke, admin-regression, payment-regression) all run against a single shared org per environment:

Env Domain Permalink Login
dev .raklet.net secret BROWSE_TEST_RAKLETNET_PERMALINK secret BROWSE_TEST_EMAIL
test .raklet.org secret BROWSE_TEST_PERMALINK secret BROWSE_TEST_EMAIL

That org (currently gercek on both) is reused for every run; the tests never sign up a new tenant. They create resources inside it — directories, custom fields, events, draft emails, sender identities, application-form fields, navigation menu items, API tokens — and clean them up in finally blocks. If the browse daemon crashes mid-test (which happens — recovery shows "Recovering: restarting browse daemon ..." in the run log), the finally block never runs and the row leaks.

Leaked rows accumulate. Once the org hits any plan cap (10 directories on premium-50K, etc.), the next run paywalls instead of creating fixtures, and the test reports a selector timeout that looks like a regression but is actually quota exhaustion. The 2026-05-22 run #26306200826 hit this on five clusters at once (3 directory tests, 2 custom-field tests, 3 messages tests, 2 events tests, 1 settings test).

The accumulation is older and bigger than the browse-test era — the Selenium-era social/jobs tests have been running against .raklet.net for years, and as of 2026-05-22 gercek.raklet.net carries 1244 pending Social Network posts and 5923 pending Jobs from old Selenium runs that crashed mid-test. The cleanup queue covers both eras: anything matching a documented test prefix on a permalink-allowlisted org gets purged.

The cleanup queue is the fix. It runs after every dispatch, regardless of whether the suite passed, failed, or timed out, and deletes only rows that match the test-prefix allowlist on the test org.

How it works

browse-tests/run.ps1            POST /v2/internal/test-fixture-cleanup/enqueue
        │                                       │
        │                                       │
        ▼                                       ▼
   GitHub Actions step ─────► Raklet.Api (IP-restricted to GHA + ci-vm-1)
   (pre-suite, always runs)                     │
                                                ▼
                                  QueueService.SendMessage(
                                    QueueType.TestFixtureCleanup,
                                    payload,
                                    delay: MaxRunDuration + 30min
                                  )
                                                │
                                                ▼
                                  Azure Storage Queue
                                  (delayed visibility ≤ 7d)
                                                │
                                                ▼  message becomes visible
                                  Raklet.WebJobs.Secondary
                                  TestFixtureCleanupFunction
                                                │
                                                ▼
                                  Validate → delete rows matching
                                  permalink+prefix allowlist
                                                │
                                                ▼
                                  ActionLog summary
                                  (counts per entity type)

Message payload

{
  "Permalink": "gercek",
  "Domain": ".raklet.org",
  "RunId": "26306200826",
  "EnqueuedAt": "2026-05-22T18:53:55Z",
  "DelayMinutes": 150
}

RunId is the GitHub Actions run id (or local:<host> for pr-ready-check invocations). EnqueuedAt is the cutoff for "row created before this run started" — anything newer is skipped, so a long-running follow-up dispatch doesn't get its own fixtures wiped.

Delay calculation

Default: suite-runtime budget + 30 min buffer. The buffer absorbs CI queue time, daemon-recovery loops, and short upward drift in test runtime without a code change. The current ceiling is the admin-regression job's 2h step timeout, so default DelayMinutes is 150.

Override per-dispatch with a cleanup_delay_minutes workflow_dispatch input. Azure caps initial visibility delay at 7 days, which is the hard upper bound.

Guardrails

This handler deletes data. The guardrails are defense-in-depth — each one on its own should be enough to prevent a runaway delete; together they make the failure mode "nothing happens" rather than "real data lost."

  1. Permalink allowlist (hard-coded, compile-time). The handler refuses any permalink not in BrowseTestOrgPermalinks:
  2. gercek (the QA org on both .raklet.net and .raklet.org)
  3. Stripe / iyzico sandbox sub-orgs configured via app settings

Any other permalink → log + ActionLog + drop the message.

  1. Domain allowlist. Only .raklet.net and .raklet.org are accepted. .com (prod) → reject + alert.

  2. Prefix allowlist (hard-coded). The handler will only delete rows whose Name / Title matches one of the documented test patterns. The list is split by era because each test framework has its own convention:

Browse-tests (literal prefix): - am1- — contacts / contact-details - am3- — settings tests - am6- — admin / messages / directory / job tests - browse2026… — Get-BrowseRunPrefix timestamp prefix - browse-events-tag- — event-tag tests - browse-social-settings- — social settings tests - browse-test-coupon- — coupon tests - browse-clone-online-, browse-clone-venue- — events-clone tests

Selenium-era (regex, anchored to start of name): - ^\d{4}-\d{2}-\d{2}-\d{2}-\d{2}-\d{2}(?!\d) — the RakletGeneralExtensions.getDateTime() timestamp emitted by every Selenium helper. Always followed by a feature-specific suffix (Post for Comments Tests, Job Position, etc.). The regex is anchored to ^, so a row with a date inside the name is never matched. The trailing (?!\d) guard prevents chaining into longer numeric IDs that happen to start with a 19-char timestamp shape.

Required TestFixtureCleanupFunctionTests cases for the Selenium regex (any change to the regex updates these): | Input | Match | | --- | --- | | 2026-05-01-19-58-30Post for Comments Tests | ✓ | | 2026-05-01-19-58-30 Job Position | ✓ | | 2026-05-01 board meeting | ✗ — wrong separator | | Meeting on 2026-05-01-19-58-30 | ✗ — not anchored to start | | 2026-05-01-19-58-301234567 | ✗ — trailing-digit guard |

The mixed strategy (literal prefixes for browse-tests, regex for Selenium) is deliberate: we own browse-tests so a literal-prefix convention is reliable; Selenium is being retired (RAK-342) but not yet gone, and per-suffix enumeration would silently break as new Selenium helpers land before retirement.

Any literal prefix shorter than 4 characters or any regex that doesn't anchor to ^ is rejected at compile time. Adding a new prefix or regex requires a code change AND this doc updated AND a unit test exercising the new shape.

  1. Per-entity-type caps. A single cleanup invocation will not delete more than:
  2. 200 directories
  3. 500 custom fields
  4. 500 events
  5. 1000 draft emails / templates / sender identities
  6. 2000 social posts (higher cap — gercek.raklet.net already carries 1244)
  7. 1000 social comments
  8. 6000 jobs (higher cap — gercek.raklet.net already carries 5923)
  9. 200 navigation menu items
  10. 200 API tokens
  11. 200 application-form fields

Anything beyond the cap is logged for human review and left in place. The first run on the historical backlog will need either a one-off manual purge or several runs to clear; that's intentional — the cap is the brake.

  1. CreatedAt < EnqueuedAt filter. The handler only deletes rows created before the message was enqueued. Concurrent in-flight tests on the same org keep their fixtures.

  2. Idempotent. Replaying the same message is a no-op (matching rows are gone). Two parallel dispatches enqueueing overlapping cleanups is safe.

  3. Soft-delete where possible. Entities with an IsDeleted flag are soft-deleted (recoverable for 30 days via the existing restore path). Hard-delete is reserved for entities without that flag (custom fields, API tokens, navigation menu items).

  4. ActionLog summary per run. Every invocation writes one ActionLog entry with the counts per entity type. Easy to spot a misfire.

Operating procedures

Inspect the queue

# Approximate message count (does not consume)
.\scripts\dev\queue-peek.ps1 -QueueName test-fixture-cleanup

The Storage account is AzureWebJobsStorage per environment.

Manually trigger a cleanup right now

.\scripts\dev\enqueue-fixture-cleanup.ps1 `
  -Domain '.raklet.org' `
  -Permalink 'gercek' `
  -DelayMinutes 1

This calls the same /v2/internal/test-fixture-cleanup/enqueue endpoint, with a 1-minute delay so the handler picks it up almost immediately.

Manually purge a single resource type

If a specific entity type bloats and you don't want to wait for the queue, the same handler is exposed as a controller action for one-shot use. Do not script this in CI — the queued path is the contract.

Disable cleanup

Set app setting BrowseTestFixtureCleanupEnabled = false on the target environment. The handler will read each message, log "disabled," and drop it. The enqueue endpoint also short-circuits and returns 200. Re-enable when you've reviewed the orphan state and want auto-purge back.

Add a new test entity type

  1. Add a hard-coded prefix-matched delete to TestFixtureCleanupFunction for the new entity, including the per-entity cap.
  2. Update the prefix list in Guardrails → 3 above.
  3. Add a unit test under Raklet.UnitTests/WebJobs/ exercising the new prefix branch with IsDeleted and CreatedAt guards.
  4. Bump the per-suite default DelayMinutes only if the suite's job-step timeout grew.

Slack notifications

The handler posts a single message to #release-manager after each run that actually did something — using the same incoming webhook Deployment.Scripts/SlackMessage.ps1 uses, so cleanup updates land next to the deploy notes the team already watches.

Posts as Test Fixture Cleanup with a :broom: emoji so they're visually distinct from Release Manager deploy posts.

When the handler posts

Condition Posts? Sample header
Allowlist rejection yes :no_entry_sign: Test fixture cleanup REJECTED — domain-not-allowed (...)
Any branch failure yes :x: Test fixture cleanup FAILED (...)
Any cap hit yes :warning: Test fixture cleanup cap hit — backlog draining (...)
Anything deleted yes :broom: Test fixture cleanup ok (...)
Clean no-op (everything zero, no warnings, not rejected) no —

The no-op skip keeps the channel quiet once the historical backlog is drained. Once a day's nightly fires against a clean org, nothing posts; the next time a test creates a fixture that ends up orphaned, the post returns. Source of truth is still the WebJob log and App Insights — Slack is a convenience signal, not the audit trail.

Sample messages

Healthy run after a draining round: 

:broom: Test fixture cleanup ok (gercek on .raklet.net, runId=gha:26306200826)
• custom-fields: 3 deleted
• jobs: 47 deleted
• social-posts: 12 deleted

Backlog mid-drain (cap hit): 

:warning: Test fixture cleanup cap hit — backlog draining (gercek on .raklet.net, runId=gha:26306500000)
• custom-fields: 0 deleted
• jobs: 6000 deleted
• social-posts: 1244 deleted
• TestFixtureCleanup: jobs matched 7842 > cap 6000; deleting cap-many oldest, re-run for the remainder.

Misconfigured caller: 

:no_entry_sign: Test fixture cleanup REJECTED — permalink-not-allowed (foo on .raklet.org, runId=...)

This last shape is the highest-signal alert in the channel: it means somebody (or something) hit /v2/internal/test-fixture-cleanup/enqueue with a permalink that's not in the allowlist. Worth triaging — either a legitimate sandbox sub-org needs adding to BrowseTestFixtureCleanupExtraPermalinks, or the call shouldn't have been made.

Configuration

App setting Default Purpose
BrowseTestFixtureCleanupSlackWebhook release-manager incoming webhook Override per env; blank disables Slack entirely (the handler still runs, just silently — WebJob log captures everything)
BrowseTestFixtureCleanupSlackUsername Test Fixture Cleanup Override the bot's display name

To disable Slack on a specific env (e.g. local dev) without removing the app setting, set the webhook value to empty string.

Failure modes

Scenario Mitigation
Enqueue endpoint unreachable (Raklet.Api down) GHA step logs warning, run continues. Nightly orphan sweep (next row) catches the bloat.
Browse daemon crashes before PowerShell enqueue GHA step enqueues from the workflow YAML directly, before invoking PowerShell.
Queue backlog (handler stalled) Approximate count surfaces via QueueService.GetApproximateMessageCount. Alert if count > 50.
Misconfigured permalink in allowlist Handler rejects + ActionLog. No deletion.
Two suites overlap on the same org CreatedAt < EnqueuedAt filter keeps each suite's in-flight rows.
Prefix collides with real customer data on a test org Hard-coded prefix allowlist plus org allowlist makes this impossible without two simultaneous code changes.

Nightly fallback

A GitHub Actions cron — .github/workflows/test-fixture-nightly-sweep.yml — calls the same /v2/internal/test-fixture-cleanup/enqueue endpoint at 04:00 UTC every day, once for each domain in the allowlist, with delay_minutes=1 so the handler picks it up immediately. This is the safety net for "queue message never arrived" — e.g. matrix-init failure, GHA runner force-kill, or the dispatch step dying before its own in-line enqueue fired.

We use a GHA cron rather than a TimerTrigger WebJob because:

  • Same code path as the suite-level enqueue (one handler, one set of guardrails, one place to audit).
  • No new WebJobs SDK extension or host.json schedule wiring.
  • The cron lives next to the smoke workflow; engineers reviewing the test infra see both in one folder.

Runner pin. The nightly runs on [self-hosted, windows, raklet-ci-parallel], the same pool browse-ui-tests uses — not ubuntu-latest. The internal endpoint is IP-allowlisted via InternalController's FilterIP attribute, and GitHub-hosted runners have rotating IPs that won't be in the allowlist. From ubuntu-latest the enqueue would fail the IP gate, the helper would swallow the error, and the safety net would stay silently green.

Failure visibility. The nightly invokes the helper with the -ThrowOnFailure switch, and the workflow step has no continue-on-error. A failed enqueue surfaces as a red job in the Actions tab. The suite-start helper call from run.ps1 keeps the default swallow-and-warn behavior — a transient enqueue blip should not fail an otherwise green test run, since the nightly will catch up.

Manual one-off purge: open the Test Fixture Nightly Sweep workflow in Actions and click Run workflow; pick the domain + permalink + delay. Same allowlist guardrails enforce on the server side.

Phase 1 scope (this PR)

Entity coverage that ships now, sized to unblock today's gercek paywalls and the 1244 social posts + 5923 jobs already piled up on gercek.raklet.net:

Entity Source Cutoff applies? Delete path
Social Network posts (Posts table) Post.Title, joined Group.OrganisationId == orgId ✓ CreatedOn < cutoff _rakletDb.Posts.Remove (no IsDeleted to soft-set)
Job postings (JobPostings table) JobPosting.Position, JobPosting.OrganisationId == orgId ✓ CreatedOn < cutoff _rakletDb.JobPostings.Remove
Custom fields (CustomFields table) CustomField.Name, CustomField.OrganisationId == orgId ✗ no CreatedOn on entity (see below) CustomFieldService.RemoveAsync so the per-org sparse column on customfields.CustomFieldTable_{orgId} is dropped too

Each branch is wrapped in SafeRun so one entity-type failure doesn't block the others.

Implementation notes

  • SQL pre-filter, then C# final-verify. Each branch issues a single EF query that pushes the literal-prefix StartsWith chain and the Selenium-shape SqlFunctions.PatIndex test into SQL, ordered by CreatedOn ASC and bounded by Take(cap + 1). The matched candidates come back to C# where TestFixturePrefixMatcher.IsTestFixture does the final regex check (the SQL PatIndex pattern is over-broad relative to the (?!\d) guard, so the C# layer catches the trailing-digit false-positives). The cap is then applied to the matched set, not to pre-filter candidates, so non-fixture noise can never fill the page and hide older fixtures.
  • Oldest-first deletion. ORDER BY CreatedOn ASC means the historical backlog (1244 social posts on gercek.raklet.net, 5923 jobs) gets drained in order, not at random.
  • Custom field cleanup uses CustomFieldService.RemoveAsync. A raw _rakletDb.CustomFields.Remove leaves the per-org sparse column on customfields.CustomFieldTable_{orgId} orphaned. The service path drops the metadata row AND the column.
  • CustomField has no CreatedOn. The cutoff guardrail (§5 above) does not apply to that branch. A concurrent test that creates a custom field between enqueue and handler execution could see its field deleted. The 150-min default delay is much larger than a single custom-field test's lifetime, so realistic races are rare, and the per-entity cap (500) bounds the blast radius. Phase 2 follow-up: add a CreatedOn to CustomField, or track creation timestamps in a side table, then enable the cutoff here.

Phase 2 follow-ups (not in this PR)

Tracked under [RAK-367] as remaining work; each is a small additive branch in TestFixtureCleanupService once Phase 1 ships clean:

  • Directories (lives in CosmosDB — needs CosmosDbDirectoryService path, not RakletDb)
  • Events (Events table)
  • Draft emails / templates / sender identities (Mails, EmailIdentities, ...)
  • Social comments (Posts with ParentId != null or dedicated table)
  • Navigation menu items
  • API tokens (ApiTokens)
  • Application-form fields

The Phase 2 caps are already listed in Guardrails → 4 so the allowlist contract is set; the missing piece is just the per-entity Clean* method.

What this is not

  • Not a general cleanup utility. It only handles test-prefixed rows on the test orgs. Use the regular admin UI / /v2/internal/contacts / etc. for everything else.
  • Not a substitute for per-test finally cleanup. Tests should still delete what they create. The queue is the backstop for daemon crashes and timeouts, not the primary cleanup path.
  • Not a customer-facing feature. No UI, no API surface outside the internal endpoint. The PR description should call out the surface area explicitly.

References