Skip to content

Release test-result visibility

Goal: trustworthy, frictionless, low-human-touch releases with durable, visible test results. This doc covers the two CEO complaints and what we did about each.

Acting on a red gate? This doc is about where results show up. For how to diagnose and fix a failing browse gate (test bug vs product bug), follow the release browse-gate triage runbook. The consolidated comment below auto-links it whenever a browse signal is red.

# Complaint Where it's fixed
B AzDO release Tests tab is empty for the browse + integration suites This doc, "Option A" — needs a human AzDO change (runbook below). Mitigated today by Option B.
C GitHub PR checks summary disappears after merge, so a promotion leaves no durable record Done — consolidated sticky comment (Option B below). Sticky comments persist on a merged PR.

C — durable post-merge record (implemented, GitHub-side)

scripts/ci/post-release-test-summary-to-pr.ps1, wired into .github/workflows/azdo-results-to-pr.yml.

For every open Test→Prod PR it upserts one consolidated sticky comment (HTML marker <!-- release-test-results -->) holding every release-gating signal with a clickable link:

  • AzDO Unit + integration tests stage result (release def 31).
  • AzDO Browse smoke stage result.
  • The linked GitHub Browse UI Tests run conclusion + per-suite pass/fail (smoke / admin-regression / payment-regression), pulled from the run's jobs.
  • Sentry net-new unresolved issue count for the deploying environment.

Sticky comments persist on a merged PR, so this comment is the permanent record.

How the GitHub browse run is linked

The AzDO Browse-smoke stage dispatches browse-ui-tests.yml on ref=master, so the dispatched run's head_sha is the master tip at dispatch time, not the test-tip PR head — it can't be matched by SHA. We correlate by dispatch time instead: the linked run is the browse-ui-tests.yml run whose created_at is the earliest value at/after the AzDO Browse-smoke stage's deploy-phase startedOn, within a tolerance window (default 30 min).

Verified against release 6053 (Release-85, PR #14109): stage phase start 2026-06-18T21:27:00Z → run 27790363519 created 2026-06-18T21:27:07Z (10 s later). The per-suite breakdown correctly showed admin-regression green while smoke + payment-regression were cancelled — which is exactly why the stage was red.

Safety / idempotency

  • Every external read is wrapped; an API blip degrades a single row to "unavailable" rather than failing. The script always exit 0, and its workflow step is continue-on-error: true.
  • The comment is matched by HTML marker and edited in place, so re-running on the 20-min poll / on PR events upserts rather than duplicates.

Relationship to the existing mirrors

This extends, not replaces: - post-azdo-results-to-pr.ps1 still posts the per-stage commit statuses (azdo / unit-tests, azdo / browse-smoke) that the pre-merge checks UI uses. - post-sentry-new-issues-to-pr.ps1 still posts its detailed Sentry table. - The new script adds the single consolidated comment that is the durable, one-glance post-merge artifact.

B — populate the native AzDO release Tests tab (runbook, needs a human)

The browse suite runs in GitHub Actions (browse-ui-tests.yml), so its results live in GitHub, not in the AzDO release — hence the empty Tests tab.

There are two ways to address this. We recommend Option B as the canonical record, and Option A only if you specifically want the native AzDO tab populated. Doing nothing further is acceptable: Option B already gives a durable, linkable record.

No AzDO change. The sticky comment above is the single source of truth for "what passed for this promotion", it links straight into the AzDO release and the GitHub run, and it survives the merge. Cheapest and lowest-risk.

Option A (optional): publish the GitHub JUnit into the AzDO Tests tab

This makes the browse suite's results render natively in the release stage's Tests tab. It requires editing the classic release definition 31 in the AzDO portal.

⚠️ Classic release definitions snapshot when edited and are delicate. An in-flight release keeps the definition revision it started with; only new releases pick up your edit. The Browse-smoke stage already runs an inlined copy of dispatch-browse-ui-tests.ps1 (currently rev 22); see .agents/context/ci-runners.md → "AzDO Pipeline 31 dispatch step". Make this change in a single edit window, then trigger one throwaway release to .raklet.net and confirm the Tests tab populates before relying on it.

The dispatcher already does 90% of the work: after the GitHub run terminates it downloads each test-results-<suite> artifact (per-suite JUnit XML + failure screenshots), de-duplicates cross-leg JUnit leakage, and emits

##vso[task.setvariable variable=BrowseTestResultsRoot;]<download-root>

So the only missing piece is a PublishTestResults@2 task in the same deploy phase, after the dispatch task, that reads $(BrowseTestResultsRoot).

Exact AzDO portal steps

  1. Pipelines → Releases → definition 31 ("Test with Unit") → Edit.
  2. Open the Browse smoke (GitHub Actions) stage → its agent job. Confirm the existing inline PowerShell task is the current body of dispatch-browse-ui-tests.ps1 (rev 22). If a rename/body change is pending, re-inline it in the same edit (see the canonical-source note at the top of that script).
  3. Add task → Publish Test Results (PublishTestResults@2), placed immediately after the inline dispatch task in the same job, with:
  4. Test result format: JUnit
  5. Test results files: **/*.junit.xml
  6. Search folder: $(BrowseTestResultsRoot)
  7. Merge test results: unchecked (keep one run per suite so smoke / admin-regression / payment-regression appear as distinct results)
  8. Test run title: Browse UI Tests ($(Release.ReleaseName))
  9. Fail if there are test failures: unchecked — the stage's pass/fail is already governed by the dispatcher's exit code; this task only publishes.
  10. Control Options → Run this task: Even if a previous task has failed, unless the deployment was canceled (succeededOrFailed() equivalent) — so red runs still surface their failed cases in the tab.
  11. Upload test results files: leave default (uploads the XML as attachments).
  12. Save the definition (this creates a new revision — note the rev number).
  13. Create a release (or re-run the deploy) to .raklet.net. After the Browse-smoke stage completes, open the release → Tests tab and confirm the three suites appear.

Why a real task and not a logging command

##vso[task.publishtestresults …] is not a real AzDO logging command (it is rejected with "not a recognized command for task command extension"), so publishing must be done by a real PublishTestResults@2 task. The dispatcher deliberately only sets BrowseTestResultsRoot and lets this task do the publish.

Caveats

  • Per-leg JUnit leakage: the three matrix legs share a clean: false workspace, so each test-results-<suite> artifact carries copies of its siblings' XMLs. The dispatcher already prunes the cross-leg copies before setting BrowseTestResultsRoot, so the **/*.junit.xml glob is safe. If you see triple-counted results, confirm you're on dispatcher rev ≥ the one that added the de-dup loop.
  • Integration tests: the AzDO "Unit Tests" stage runs unit + integration tests inside AzDO and already (or should already) publish via its own VSTest/PublishTestResults task; that is independent of this browse wiring.

Files