Skip to content

Browse-test failures — status board

Living index of browse-ui-tests matrix failures on .raklet.net and the PRs that address them. Update this file when a new matrix run reveals a fresh set of failures, or when a fix lands.

Two environments, two failure sets. The HTML reports below track the .raklet.net browse-ui-tests matrix. The Quarantine register directly below tracks ci-vm-1's pr-ci.yml against .raklet.org — a different environment with different fixtures, so it fails a different set of tests (e.g. .raklet.net chronically fails 04/14/16/34/35; .raklet.org fails the 6 quarantined below). Don't assume a green matrix run means a green pr-ci run.

Each report is a self-contained HTML brief uploaded to Azure Blob (rakletlocalfiles/customcodes/browse-test-reports/<date>/) so the team can share and review without cloning the repo. The HTML is generated by-hand from $B-validated investigation and includes per-failure cards (error, screenshot evidence, root cause, fix options, recommendations).

2026-08-01 — 8 tests un-quarantined; every umbrella ticket is closed

Audited the register against three consecutive pr-ci runs (30665522745, 30648866101, 30628459620). Quarantined tests still run (once, non-gating), so the artifacts already say which ones pass. The failure set was identical across all three runs — these are deterministic, not flaky.

Un-quarantined (passed 3/3): admin-regression 10-directory-read-only, 19-application-form-update-sections, 21-application-form-fields-requiredness, 22-application-form-create-custom-field, 26-contacts-dashboard-count, 27-social-content-mutation; payment-regression 06-fundraising-campaign-crud, 14-contact-donations-crud. Most were waiting on "ENG-63 per-shard isolation", which shipped via the per-runner ci1..ci4 orgs (ENG-283 / ENG-293). Register: 32 → 24.

Not quarantined but failing 3/3 — payment-regression is red on every PR: 12-membership-digital-card-fields and 20-membership-subscriptions-filtering-sorting. Deliberately left un-quarantined: hiding a consistent failure is worse than a red check. These need owners.

Bookkeeping is stale: every umbrella the remaining 24 point at is closed — ENG-62 (Done), ENG-63/RAK-748 (Done), ENG-186 (Done). Tracked on RAK-740, along with a proposed CI guard that fails the summary step when known-failures.json references a completed ticket — the manual sweep clearly doesn't hold on its own.

Quarantine register (ci-vm-1 / .raklet.org pr-ci)

These admin-regression tests are quarantined via browse-tests/known-failures.json since 2026-06-07, tracked under ENG-62 (the last two of the original cohort, the ENG-53 contact-mutation pair, were added after the first live pr-ci run still cancelled payment-regression — see below). The register was expanded on 2026-06-22 with 9 more admin-regression + 2 more payment-regression entries — see Batch added 2026-06-22 below.

Single source of truth (2026-06-12): known-failures.json is the one registry. run.ps1 reads it to decide which tests run-once/non-gating, and post-regression-summary-to-pr.ps1 reads it to classify each failure as KNOWN vs NEW in the PR comment — so the two can't drift. It also carries the payment-regression known-failures (ENG-4). To un-quarantine a test, delete its entry from the JSON.

Why quarantine, mechanically: each ran twice (retry) and triggered a daemon restart + re-login recovery on every failure. Six of them per run added enough wall-clock that the ci job blew past its timeout-minutes: 90 cap, so payment-regression never ran and the job was cancelled. (Note pr-ci.yml already marks both regression suites continue-on-error: true, so the failing verdict was the timeout/cancel, not a red gate.) Quarantined tests now run once, do not trigger the recovery cycle, and do not count toward the suite exit code — but they still run, still report red, and still post failure screenshots, so no canary signal is lost. Reversible: delete the test's entry from known-failures.json. Root cause evidence below is from the pr-ci screenshot comments on PRs #13944 / #13945 / #13953.

20-social-content-read-only — Times out after 30s waiting for the New Post link .SocialAnnouncements a[ui-sref='manager.social.postcreate'] on the social posts list. The list shell (.SocialAnnouncements) renders but the action toolbar's New Post anchor never appears within the window — consistent with the admin-SPA cold-load / unknown-route "Loading… Please Wait." hang tracked in ENG-26, or a ui-sref selector drift. Read-only test, so this is not shared-state contamination. Re-enable once the toolbar reliably renders (or the selector is corrected).

21-application-form-fields-requiredness — Apply submit did not reach Success/CreditCard (url: about:blank). … Debug: no form. The public apply form never loads/submits — the page is left at about:blank with no form detected, so the requiredness assertion can't run. Sits in the apply-form validation area tracked by ENG-29 and was cluster F of the ENG-4 .raklet.org sweep. Re-enable once the apply form renders deterministically on the ci-vm-1 org.

26-contacts-dashboard-count — Contact count mismatch: contacts grid=213, dashboard=206. The contacts grid and the dashboard counter disagree because other mutation tests in the same suite add/remove contacts in the shared org and the dashboard count lags the grid (read-side caching). This is shared-org data accumulation — the ENG-4 (RAK-367) nightly fixture-cleanup territory. Re-enable once the count is computed from a stable snapshot or the test tolerates the known lag.

27-social-content-mutation — ManagerSocialService.createAnnouncement(...) → page.evaluate: Couldn't update!. The backend rejects the post-create with "Couldn't update!", most likely because the hard-coded category id / org state the test posts into has drifted on the shared, repeatedly-mutated org. Adjacent to the social mutation matrix gap in ENG-46. Re-enable once the test seeds its own category / the shared org state is reset per run.

28-social-settings-mutation — page.evaluate: Error: portal about us title missing at Assert-PortalAboutUs. The test mutates social settings and then asserts the About Us title/text on the portal — but the About Us feature was retired (CEO confirmed 2026-05-19; the source Selenium row is already [disabled] in the migration checklist). The assertion targets a feature that no longer exists, so it can never pass as written. Was cluster G of ENG-4. The real fix is removing the About Us assertion (follow-on, tracked in ENG-62) — quarantined until then so the remaining settings-mutation coverage isn't blanket-deleted.

36-messages-sender-identities — Times out after 60s waiting for form, .EmailSenderProfile, table — the Email Sender Profiles settings page never renders on the ci-vm-1 test org. This is the messages-module render gate family (sibling of the activation/draft gate in ENG-23); the row-lookup logic was already hardened in #13882, so the current blocker is the page itself not loading, not the test's selectors. Per ENG-23's "leave it red as the canary" stance we do not SKIP it — quarantine keeps it red and visible while removing the retry cost. Re-enable once the sender settings page renders on the test org.

28-contact-details-mutation (added 2026-06-07) — get file failed on the contact detail Files tab. Landed with the ENG-53 contact detail-mutation work (#13945) and fails both attempts on ci-vm-1, so it was retrying and burning budget. The first live pr-ci run of this quarantine (#13956, run #27102604900) still cancelled payment-regression at the 90-min cap partly because this pair was not yet quarantined; added here so it runs once. Root cause tracked under ENG-53.

28-contact-segment-mutation (added 2026-06-07) — Expected body contains 'am1-segment-browse… Contacts' to be False: the segment still contains a contact from a prior run, i.e. shared-org state contamination (the same fixture-cleanup family as 26). Also from ENG-53 / #13945, failing both attempts. Quarantined to stop the retry budget bleed; root cause tracked under ENG-53.

Batch added 2026-06-22 — ENG-161 regression-baseline cleanup

The pr-ci regression summary was flagging 13 "NEW" failures on every PR targeting master, including PRs that touch neither the failing code paths nor the test harness. They are not PR-caused: the same 13 with byte-identical error signatures appeared on three unrelated PRs back-to-back —

Of the 13, 11 are quarantined here (env / shared-org / harness baseline) and 2 are deliberately left red — 34-messages-email-draft-lifecycle and 35-messages-template are the customer-impact canary for the Urgent, in-progress ENG-23 ("drafts all show 'Empty'"). Quarantining those would hide a real product bug, so they stay NEW/red on purpose (per ENG-23's "leave them red as the canary").

The three unrelated PRs: #14183 (messages fix), #14185/ #14192 (a C# null-Data fix), and #14190. Each was triaged against the failure screenshot + the test source; none is a product regression. Per-test root cause now lives in known-failures.json; the buckets:

  • Shared-org state pollution / read-side lag (admin-regression 42-social-jobseeker-profile-mutation, 44-social-comments-mutation, 45-membership-settings-general-mutation, 46-membership-debts-mutation, 10-directory-read-only; payment-regression 10-contact-scheduled-payments-crud, 13-contact-scheduled-donations-crud). The shared gercek org accumulates state across runs/legs, so created records aren't found, toggles read back a foreign value, or list/dashboard reads lag. Same family as the original cohort (26/27/28-segment) and the ENG-4 contact-CRUD pair (11/14). Durable fix is per-shard org isolation (ENG-63). 44-social-comments-mutation fails at the exact svc.createAnnouncement → "Couldn't update!" step as already-quarantined 27-social-content-mutation.
  • Admin-SPA load hang (38-membership-applications-search-read-only, 47-membership-applications-export) — "page did not finish loading", the ENG-26 SPA cold-load family already represented by 20-social-content-read-only.
  • Harness/env, not product (48-digital-card-scan-actions-mutation, 49-social-only-member-access-mutation). 48 hits a PowerShell-5.1 native-invoke/console-handle quirk on ci-vm-1 under NT AUTHORITY\SYSTEM (StandardOutputEncoding is only supported when standard output is redirected at helpers/browse.ps1:26); the test sets no encoding. 49 runs right after and inherits the mid-restart daemon (browse failed: restart). The durable fix is hardening the native invocation in browse.ps1, but it is not reproducible without a live ci-vm-1 repro, so it is not blind-fixed here (a shared-helper change must be validated via a real CI dispatch first). Re-enable both by deleting their entries once the harness is hardened.

These ran twice with daemon-restart recovery on each failure and counted as "NEW", which both burned the ci job's time budget and masked the canary by crying wolf on unrelated PRs. As with the original cohort they still run once, still report red, and still post failure screenshots — they just no longer gate or mislabel. Re-enable any test by deleting its entry from known-failures.json.

Latest report

2026-05-22 — browse-test-failures-2026-05-22.html

Based on matrix run #26236942007 (first full dispatch after the test-env deploy carrying PRs #13843–#13848). Covers 18 failures, classified into four categories:

Category Tests Status
A — Test-org data accumulation AR 22, AR 24, AR 28-directory AR 24 fixed in #13854; AR 22 + 28-directory need manual cleanup + broader-prefix cleanup helpers
B — Test design / infrastructure AR 25, AR 26, AR 35 AR 25/26 fixed in #13854; AR 35 (template title not persisted) needs $B walkthrough
C — Sandbox credentials needed PM 06, PM 07, PM 15 Blocked on Stripe sandbox sub-org password
D — Product bug PM 16 Blocked on RAK-349

Fixes already merged: PR #13851 (7 tests) + PR #13854 (3 tests).

How to publish a new report

  1. Generate a self-contained HTML brief (no external CSS/JS so it survives offline / email).
  2. Upload to Azure Blob via the snippet in .agents/local/ (or the PowerShell function below).
  3. Append a new entry at the top of the "Latest report" section here; keep the prior entries for trend reference.
# Reads the storage account key from Application/Web.config and uploads
# the HTML via the Azure Blob REST API (Shared Key auth). No Azure CLI
# required. The customcodes container is public-blob-read, so the URL
# works for anyone on the team.
function Upload-RakletBlob {
    param(
        [string]$AccountName = 'rakletlocalfiles',
        [string]$AccountKey,
        [string]$Container,
        [string]$BlobPath,
        [string]$LocalFile,
        [string]$ContentType = 'text/html; charset=utf-8'
    )
    $fileBytes = [System.IO.File]::ReadAllBytes($LocalFile)
    $now = [DateTime]::UtcNow.ToString('R')
    $version = '2020-04-08'
    $canonicalizedHeaders = "x-ms-blob-content-type:$ContentType`nx-ms-blob-type:BlockBlob`nx-ms-date:$now`nx-ms-version:$version`n"
    $stringToSign = "PUT`n`n`n$($fileBytes.Length)`n`n$ContentType`n`n`n`n`n`n`n${canonicalizedHeaders}/$AccountName/$Container/$BlobPath"
    $hmac = New-Object System.Security.Cryptography.HMACSHA256
    $hmac.Key = [Convert]::FromBase64String($AccountKey)
    $sig = [Convert]::ToBase64String($hmac.ComputeHash([Text.Encoding]::UTF8.GetBytes($stringToSign)))
    $headers = @{
        'x-ms-blob-type'         = 'BlockBlob'
        'x-ms-blob-content-type' = $ContentType
        'x-ms-date'              = $now
        'x-ms-version'           = $version
        'Authorization'          = "SharedKey ${AccountName}:$sig"
    }
    $uri = "https://$AccountName.blob.core.windows.net/$Container/$BlobPath"
    Invoke-WebRequest -Uri $uri -Method PUT -Headers $headers -Body $fileBytes -ContentType $ContentType -UseBasicParsing | Out-Null
    $uri
}

$key = ([xml](Get-Content 'Application\Web.config')).configuration.appSettings.add |
       Where-Object { $_.key -eq 'StorageConnectionString' } |
       ForEach-Object { ($_.value -split ';' | Where-Object { $_ -like 'AccountKey=*' }) -replace '^AccountKey=' }

Upload-RakletBlob -AccountKey $key -Container 'customcodes' `
    -BlobPath "browse-test-reports/$(Get-Date -Format yyyy-MM-dd)/browse-test-failures.html" `
    -LocalFile '<path-to-html>'

The container is the same one used for PR-evidence uploads (see feedback-pr-evidence-hosting memory). Container ACL is public-blob-read; no SAS token required.