"Test with Unit" — Release Pipeline 31¶

AzDO Release Definition 31 is the gate between the Test and Production environments. Every Dev→Test merge triggers it, and Prod gets no deploy unless this pipeline passes end-to-end.

Definition URL: https://raklet-git.visualstudio.com/raklet-git/_releaseDefinition?definitionId=31&_a=definition-pipeline

Stage sequence¶

flowchart LR
    A([Release starts]) --> FP[Flight Preparation\nrank 1]
    FP --> BE[Backend\nrank 2]
    BE --> AD[Admin\nrank 3]
    BE --> LO[Login\nrank 4]
    BE --> V3[v3\nrank 5]
    BE --> AP[api\nrank 6]
    BE --> CR[Crm\nrank 7]
    AD & LO & V3 & AP & CR --> UT[Unit Tests\nrank 8]
    AD & LO & V3 & AP & CR --> SW[Stop PREP Webjobs\nrank 9]
    AD & LO & V3 & AP & CR & BE --> BS[Browse smoke\nGitHub Actions\nrank 10]
    UT & SW & BS --> MS[Message to Slack\nrank 11]
    MS --> TEST([TEST stage\nrank 12])

Backend must finish before any app deploy stage starts. All five app stages (Admin, Login, v3, api, Crm) run in parallel once Backend succeeds. Unit Tests and Stop PREP Webjobs both start as soon as all five app stages succeed — they run in parallel with each other. Browse smoke has an additional dependency on Backend itself (it hits the test site end-to-end, so all six deploys must be done).

How each deploy stage works — the slot-swap pattern¶

Every service (Backend included) uses the same blue/green deploy pattern via Azure App Service deployment slots:

flowchart TD
    A[Deploy new code\nto **prep** slot] --> B[Start prep slot\n+ start its WebJobs]
    B --> C[WarmUp PREP task\nHTTP requests to heat JIT]
    C --> D[Swap prep → production\natomic — zero downtime]
    D --> E{Is this Backend?}
    E -- No --> F[Stop prep slot\nold code stays idle]
    E -- Yes --> G[Leave prep slot running\nWebjobs still active]

After the swap, the prep slot holds the previous version of the code. For Admin / Login / v3 / api / Crm the prep slot is stopped immediately (task 5 in each stage). For Backend it is intentionally left running — this is what the "Stop PREP Webjobs" stage cleans up.

Stop PREP Webjobs — why and when¶

The problem it solves¶

After the Backend slot swap, raklet-test-backend's prep slot contains the old backend code and its continuous WebJobs (PollClose, email-link summaries, scheduled hourly tasks) are still active and hitting the same TEST SQL databases:

raklet-test-sqldb-rakletv3
raklet-test-sqldb-email
raklet-test-sqldb-login
raklet-test-sqldb-sms

The Unit Tests stage runs integration tests (TestCategory=Integration) against those same databases. An active old-version WebJob writing to the same tables mid-test produces non-deterministic results (for example, CloseExpiredPollsAsync updates dbo.Announcements at the same time the PollClosePollAsyncTest is asserting on it).

What the stage does¶

Task	Detail
Wait 10 minutes	Lets the new production slot fully stabilize and handle real traffic before the prep slot is touched
Stop `prep` slot	`az webapp stop --name raklet-test-backend --slot prep`
Stop all continuous WebJobs	Kills any WebJob processes that survived the slot stop
Redis cache clear	Removes the `$(DeploymentRedisKey)` entry so stale cached data from the old build doesn't leak into the new slot's responses

Timing relative to Unit Tests¶

Both stages (rank 8 and rank 9) start at the same moment. Unit Tests runs immediately; Stop PREP Webjobs waits 10 minutes first. This means the old prep-slot WebJobs are still running for the first ~10 minutes of the test suite.

gantt
    title After all 5 app stages succeed
    dateFormat  mm:ss
    axisFormat  %M:%S

    section Unit Tests
    Variable substitution  : 00:00, 30s
    Run integration + unit tests : 00:30, 10m

    section Stop PREP Webjobs
    Wait 10 min            : 00:00, 10m
    Stop slot + webjobs    : 10:00, 1m
    Redis cache clear      : 11:00, 30s

For most tests this overlap is harmless — the WebJobs don't touch the same rows. If a test becomes flaky and the failure correlates with WebJob activity, shorten or remove the 10-minute wait in the pipeline definition.

Unit Tests stage¶

Runs the test assembly against the TEST environment. Key settings (from the VSTest task):

Setting	Value
Assembly	`*\Raklet.UnitTests*.dll`
Filter	`TestCategory=Integration \\| TestCategory=Unit`
SQL target	`raklet-test-sqlserver.database.windows.net`
Connection substitution	Variable Substitution task rewrites `App.config` connection strings from release variables before the test run

Important: TestCategory=Integration is included here but not in GitHub Actions (tests.yml / pr-ci.yml / run-local-tests.ps1). This means integration tests only fail in a full AzDO release, not in PR CI. If an integration test starts failing, this pipeline will catch it — but the PR that introduced the regression will already be on master.

Known integration test failure sources (see [ENG-82]):

RakletEmailContext connection string must be present as a release variable or email integration tests fail with "error 26 — instance not found"
PollClosePollAsyncTest.CloseExpiredPollsAsync can time out (30 s) if the dbo.Announcements table is large and the IX_Announcements_PollClose index is missing

Browse smoke stage¶

Dispatches the GitHub Actions workflow browse-ui-tests.yml against .raklet.net and waits for it to finish. Three test suites run in parallel on the raklet-ci-parallel runner pool (r2/r3/r4 on ci-vm-1):

Suite	JUnit artifact
Admin Regression	`admin-regression.junit.xml`
Payment Regression	`payment-regression.junit.xml`
Smoke	`smoke.junit.xml`

Results are published to the AzDO Tests tab via three separate PublishTestResults@2 tasks (one per suite) so each suite gets its own named run.

See ci-runners.md for the full dispatch mechanics and the "what to do if you rename the workflow file" rule.

Operational notes¶

Renaming `browse-ui-tests.yml` or `dispatch-browse-ui-tests.ps1`¶

Pipeline 31 holds an inline copy of scripts/ci/dispatch-browse-ui-tests.ps1. If you rename either the workflow file or the canonical script, you must also re-inline the updated script into Pipeline 31 in the same change window. Failure to do so causes every subsequent release to 404 on the dispatch POST — silently, because the runners have no work and log nothing (burned us 2026-05-23 → 05-25).

Classic release snapshot behaviour¶

AzDO classic releases freeze the pipeline definition at release-creation time. A fix pushed to the definition only takes effect in newly queued releases. In-flight releases keep running the old definition body.

Connection string substitution¶

The Variable Substitution task rewrites **/Raklet.UnitTests.dll.config from release-definition variables. The variable name must match the name attribute of the <connectionStrings> entry exactly. Current required variables:

Variable	Target DB
`RakletV3`	`raklet-test-sqldb-rakletv3`
`RakletEmailContext`	`raklet-test-sqldb-email`
`AzureWebJobsStorage`	WebJob storage account

If a new integration test references a connection string not in this list, add the release variable and verify the substitution log in the next release (look for the connection string name in the "Variable Substitution" task output).