Skip to content

AzDO Release Pipeline 31 → browse-ui-tests integration

The browse-tests migration is complete (15/15 batches merged 2026-05-20). This doc spells out how to wire the GitHub Actions browse-ui-tests.yml workflow into Azure DevOps Release Pipeline 31 as a post-deploy gate, replacing the 4-hour Selenium stages with a 30-minute GitHub-runner suite.

Goal

After every release deploys to .raklet.net:

  1. AzDO post-deploy stage triggers browse-ui-tests.yml on GitHub against .raklet.net.
  2. AzDO polls until the run terminates.
  3. Smoke green ⇒ release stage marked successful; downstream stages (currently Selenium) run.
  4. Smoke red ⇒ release stage fails; alert + investigation.

After one or two green release cycles, we delete the Selenium stages (and Raklet.UI.Test.{1..5}).

Pieces

1. PowerShell dispatcher

scripts/ci/dispatch-browse-ui-tests.ps1 — calls the GitHub REST API:

  • POST /actions/workflows/browse-ui-tests.yml/dispatches with the chosen suite + target. Default Suite=all runs smoke + admin-regression + payment-regression sequentially inside one GitHub Actions workflow run (one Playwright/bun setup).
  • Identifies the run-id by snapshotting recent runs before dispatch.
  • Polls /actions/runs/<id> until status=completed.
  • Downloads the test-results artifact (per-suite JUnit XML + failure screenshots) and emits one ##vso[task.publishtestresults type=JUnit ...] logging command per XML, so each suite renders as its own run in the Tests tab of the AzDO release stage.
  • Exits 0 on conclusion=success, 1 otherwise — after publishing results, so red runs still surface in the Tests tab.

Works on any AzDO agent that has PowerShell 7 (Microsoft-hosted ubuntu-latest or windows-latest both qualify). Doesn't need Playwright, bun, or any browse-test machinery on the AzDO side.

2. GitHub fine-grained PAT

Owner: organisation rakletadmin. Scopes:

Permission Access
Actions Read and write
Contents Read
Metadata Read

Repository access: rakletadmin/rakletv3 only.

Set expiry to 1 year (renewal reminder in calendar). Copy the token into AzDO Pipelines → Library → variable group → add secret variable GH_PAT.

3. AzDO stage YAML

If Pipeline 31 is YAML-defined (recommended) — add this stage between Deploy and Selenium:

- stage: BrowseSmoke
  displayName: 'Browse smoke (GitHub Actions, .raklet.net)'
  dependsOn: Deploy
  variables:
    - group: 'github-integration'   # contains GH_PAT secret
  jobs:
    - job: dispatch
      pool:
        vmImage: 'windows-latest'
      timeoutInMinutes: 100
      steps:
        - checkout: self
          fetchDepth: 1
        - pwsh: |
            ./scripts/ci/dispatch-browse-ui-tests.ps1 `
              -GitHubToken "$(GH_PAT)" `
              -Suite all `
              -TestDomain ".raklet.net" `
              -AdminUrl "https://admin.raklet.net/ng/"
          displayName: 'Dispatch browse-ui-tests + wait'

If Pipeline 31 is classic (UI-defined) — Pipeline 31 is classic. The existing "Browse smoke (GitHub Actions)" stage runs an inline PowerShell@2 task (no source checkout). The release stage's agent doesn't have the repo, so the task can't call the .ps1 by path — paste the script body inline. After this PR lands, paste the full body of scripts/ci/dispatch-browse-ui-tests.ps1 into the inline task and set:

  • Task: PowerShell@2
  • Inputs:
  • targetType: inline
  • pwsh: true
  • script: <paste of dispatch-browse-ui-tests.ps1> appended with the invocation below
  • Environment variables: GH_PAT: $(GH_PAT) (the inline param(...) block reads -GitHubToken, so pass it on the invocation line, not as an env var; the env mapping just keeps the secret available if you tweak the script)
  • Timeout: 120 minutes

Append this single invocation line at the very end of the inline script (after the existing param(...) block has been preserved):

& $MyInvocation.MyCommand.ScriptBlock -GitHubToken $env:GH_PAT -Suite 'all' -TestDomain '.raklet.net' -AdminUrl 'https://admin.raklet.net/ng/'

…or, simpler: just paste everything below the param(...) block of dispatch-browse-ui-tests.ps1 into the inline task, hard-coding $GitHubToken = $env:GH_PAT, $Suite = 'all', $TestDomain = '.raklet.net', $AdminUrl = 'https://admin.raklet.net/ng/', $PollSeconds = 30, $TimeoutMinutes = 120, $Repo = 'rakletadmin/rakletv3', $Ref = 'master', $ArtifactDownloadPath = '' at the top. AzDO's classic-UI doesn't accept paramaterised inline scripts the way YAML pipelines do.

The 2020-character inline task currently in Pipeline 31 (rev 13) is the old smoke-only dispatcher with no artifact-publish step. After this PR, swap that content for the new file's body. The PR description includes the exact paste block — see Pipeline 31 inline task section there.

4. Selenium dependency

Once the BrowseSmoke stage is in:

  • Keep Selenium running for 2 release cycles alongside browse-ui-tests. Compare results.
  • When confident the browse suite catches the same regressions (and ideally more, with fewer flakes), set the existing Selenium stage's dependsOn to require BrowseSmoke success, and downgrade Selenium to condition: failed() so it only runs as a fallback comparison.
  • After 2 more cycles of green BrowseSmoke, remove Selenium stages entirely + open a PR deleting Raklet.UI.Test.{1..5}.

Which suite(s) to run

Suite Wall-clock What it covers Run on every release?
smoke ~5 min Login + manager shell + module page-loads + portal account + apply/signup Yes (gating)
admin-regression ~25 min Read-only role-fixture tests + admin mutations (settings, fields, application forms, social, contacts, jobs, messages, directory) Yes (gating, after smoke); already proven against .raklet.net via current dispatches
payment-regression ~15 min Stripe + Iyzico sandbox flows (fundraising, billing, contacts, events, membership) Yes (gating) — requires the sandbox secrets we already wired

Recommend: dispatch with Suite=all — one workflow run does smoke + admin-regression + payment-regression sequentially, sharing the bun/Playwright/browse-binary setup (~10 min saved over three separate dispatches). Each suite emits its own JUnit XML and shows up as a separate run in the AzDO Tests tab.

Rollout plan

Day Action Owner
0 Land [this PR] introducing the dispatcher + doc engineering
0 Create GH PAT, store as GH_PAT in AzDO library release-mgmt (you)
1 Add BrowseSmoke stage to Pipeline 31 (smoke only first) release-mgmt
1 Trigger a release to .raklet.net, watch BrowseSmoke result release-mgmt + me
2–3 Add admin-regression + payment-regression stages release-mgmt
Week 1–2 2 green release cycles with both Browse + Selenium running observe
Week 3 Selenium → condition: failed() fallback, then deletion PR engineering
Week 3 Delete Raklet.UI.Test.{1..5} projects from solution engineering

Troubleshooting

"Dispatch POST failed: 401" — PAT expired or wrong scopes. Regenerate with the scopes above.

"Could not find new run within 2 minutes" — GitHub Actions billing paused, or runner unavailable. Check Settings → Billing & Plans on rakletadmin org. (We hit this once today; fix is to pay the bill or reduce parallel non-essential workflows.)

Run timeout — bump -TimeoutMinutes past 90. Cold ci-vm-1 starts can stretch admin-regression toward 50 min; we set the workflow's own timeout-minutes: 90 after #13774 v3. If your dispatch caller times out at exactly 90 min, raise both.

Skip noise — runs against .raklet.net should not SKIP much since the fixtures exist there. Watch for Skipped: K in the summary; if a test SKIPs because a secret is missing, populate it in Settings → Secrets and variables → Actions (see browse-ui-tests.yml for the full env-var list).