[bradystroud]

Description

✏️

Phase 1 of #2801. This pull request changes 4 workflow files and .dockerignore. It does not change application code, the Dockerfile, package.json or scripts/.

No workflow has run with these changes. Each value in the "Expected" column is an estimate. The estimates come from production run 42969510432 (11 Nov, 42.1 minutes) and from 5 to 13 other runs.

#	Change	File	Expected (estimate)
1	Run the disk cleanup only when free disk space is less than 30 GB. Remove the post-build cleanup.	build-artifacts.yml	2.0 to 3.6 minutes less
2	Build the image at the same time as the infrastructure job	build-and-deploy.yml, pr-deploy.yml	2.1 to 3.3 minutes less
3	Send only the MDX and JSON files of the content checkout to Docker	.dockerignore	0.5 to 0.8 minutes less
4	Run the sitemap crawler without the dependency install, and at 20 requests each second after a deploy	crawl-sitemap.yml, build-and-deploy.yml	9 minutes less, after the site is live

For run 42969510432, the estimate is: site live after 25 minutes (now 30), run complete after approximately 28 minutes (now 42.1).

1. Disk cleanup (build-artifacts.yml)

• A new step, "Check if the disk cleanup is necessary", reads the free space of the file system that holds the Docker data directory. It prints the value in each run. The steps "Aggressive disk space cleanup (pre-build)" and "Check disk usage after cleanup" run only when the free space is less than 30 GB, or when the step cannot read the free space. The cleanup script is not changed.
• The step "Post-build cleanup" is removed. GitHub discards the runner after the job. A new step prints df -h after the image build, so that each run shows how much space the build used.
• docker/setup-buildx-action gets cleanup: false. With the default value, the action runs docker buildx rm at the end of the job. That command deletes the same build cache that the post-build cleanup deleted. In a local test, it took 8.5 seconds for a cache of 18.7 GB (821,000 files). On the runner, docker buildx prune took 42 to 64 seconds for the same quantity of data.

Evidence from 6 build logs (9 to 12 Nov, runner image ubuntu-24.04 20261101.360.1):

	Value
Free space before the cleanup	80 GB of 145 GB, in 6 of 6 runs
Build cache of one image build	18.2 to 18.3 GB
Pre-build cleanup	79 to 156 seconds
Post-build cleanup	42 to 64 seconds
Docker data directory	/var/lib/docker, on /dev/root (mounted on /). The runner has no other data disk.

Side effect. The cleanup deletes /opt/hostedtoolcache, and that folder contains the Python of actions/setup-python. Thus, in each run until now, the steps "Install Algolia Python dependencies" and "Sync Algolia index" used the system Python 3.12 of the runner. When the cleanup does not run, these two steps use Python 3.15.2 from setup-python. A local test ran the unchanged algolia_sync.py with the two Python versions against a stand-in for the Algolia API (3,933 rules). The requests and the records were identical.

2. Build at the same time as the infrastructure job (build-and-deploy.yml, pr-deploy.yml)

• The build job no longer needs the infrastructure job. In the last 14 successful runs, the build started 2.1 to 3.3 minutes after the infrastructure job started.
• The build job used only one output of the infrastructure job, acrName. It now gets the name from the naming rule of infra/deploy.ps1 (acr + sswrules + environment, where production becomes prod): acrsswrulesprod for production, acrsswrulesstaging for staging and for pull request slots. deploy-infrastructure.yml already calculates the name in this way, and pr-close.yml already contains the staging name.
• The deploy job continues to need the two jobs, so a failed infrastructure job continues to stop the deploy. The if: conditions and the concurrency groups are not changed.
• The first step of the deploy job is new. It compares the image that the build job pushed with the registry that the infrastructure job reports (acrLoginServer). If they are different, the step fails with a message that names infra/deploy.ps1, and the deploy does not start.

Does the build need a resource that the infrastructure job creates in the same run?

• Environment that exists (staging, production): No. The registry exists, and the AcrPush role assignment of the pipeline has a fixed name, so each run writes the same assignment again. The infrastructure job on main also pushes its own image to the registry before its Bicep step.
• New pull request slot: No. The infrastructure job creates the slot and the AcrPull assignment of the slot. The build only pushes ssw-rules:pr-<number> to the staging registry. The deploy job uses the slot, and it waits for the two jobs.

3. Docker build context (.dockerignore)

The workflow moves the content checkout to the content folder before the image build, so the entry content-temp did not exclude it. The context was 3.33 GB: 1.7 GB of git data, 1.5 GB of images and the repository. Docker took 14 to 20 seconds to receive it and 11 to 30 seconds for COPY . ..

The new rules keep only the .mdx and .json files of content. In a local test, the context was 31 MB (4,635 files).

The rules do not exclude the full folder, because one step of the image build reads it. styles.css uses the automatic source detection of Tailwind, which reads all text files in the build folder. That includes the rule text. A local test compared the output of tinacms build and next build (compile mode, 4,522 files) for three contexts:

Context	Result
Full content checkout (as now)	Reference
MDX and JSON files only (this pull request)	Identical to the reference, except the 80 files that also differ between two builds of the same input (build ID, keys)
No content folder	The CSS has 10 fewer utility rules (716 bytes): .invisible, .sticky, .capitalize, .blur, .invert, .zoom-in, .w-17, .grid-cols-15, .fixed\!, .block\!

Those 10 class names are words in the rule text. A search of the content, the application code and 6 saved pages found no element that uses them. With the MDX and JSON files in the context, the CSS does not change, so this question does not occur. This is also the form that phase 3 needs ("Send only the MDX and JSON files of the content checkout to Docker").

No other step reads the folder:

• Dockerfile: COPY . . is the only instruction that reads the context after the lock files. git lfs pull fails with "git: not found" in 6 of 6 logs, and the Dockerfile ignores that.
• tinacms build --content=local: LOCAL_CONTENT_RELATIVE_PATH is not set in the image build, so the content root of Tina is /app (resolveContentRootPath in @tinacms/cli 4.0.4). The collections are in public/uploads/rules, categories, global and homepage. None of these folders exists in /app, with or without this change. The CLI prints "Using separate content repo" when it uses a different root, and none of the 6 logs has that line.
• next build: the file trace of the standalone output has no entry in content (53 trace files, 25,491 entries). The content checkout has no .ts or .tsx file.
• postbuild: scripts/generate-archived-rules.js reads TinaCloud. next-sitemap.config.js reads archived-rules.json.

4. Sitemap crawl (crawl-sitemap.yml, build-and-deploy.yml)

• The steps "Install pnpm" and "Install dependencies" are removed (1 + 20 seconds). scripts/crawl-sitemap.js imports only fs, url and path.
• The job calls node ./scripts/crawl-sitemap.js. This is the command that pnpm crawl-sitemap ran.
• fetch-depth: 0 is removed from the checkout (2.4 seconds for the fetch). No step reads the git history.
• The script is an ES module in a package that has no "type" field. It thus needs Node 22.7 or later. A local test ran it with 22.6.0 (fails), 22.7.0 and 22.23.4 (the two work). .nvmrc contains 22, and the runner resolved that to 22.23.4. This condition is the same as before.

Request rate. The crawler has three settings. It sends SITEMAP_BATCH_SIZE requests at the same time and waits for all of them. It then waits SITEMAP_DELAY_MS, or longer if that is necessary to stay below SITEMAP_MAX_RPS. With the defaults (5, 250 ms, 20), a batch took 0.66 to 0.80 seconds in run 42969510432, and each batch then waited 250 ms. That is 5.3 requests each second, and 11.8 minutes for 3,707 URLs.

• crawl-sitemap.yml gets two optional workflow_call inputs, batch_size and delay_ms. The crawl step gives them to the crawler. When they are empty, the crawler uses its defaults.
• build-and-deploy.yml passes 20 and 0. SITEMAP_MAX_RPS stays 20. The crawler then sends 20 requests, and starts the next 20 one second after the start of the batch. That is 186 batches, or 3.1 minutes, if a batch of 20 completes in one second or less. With the job setup, the estimate for the crawl job is 3.3 minutes (now 12.3). If a batch of 20 takes longer than one second, the crawl takes longer (4.7 minutes at 1.5 seconds for each batch).
• A crawl that a person starts (workflow_dispatch of crawl-sitemap.yml) has no such inputs. It keeps 5 requests and 250 ms.

Reason for the two rates. The build gives each rule page a revalidate period of 6 hours (/[filename] 6h 1y in the route table of run 43165212271). The crawl of a deploy starts minutes after the build. Each page is then newer than that period, the server answers from its cache, and no render starts. Thus the fast crawl does not load TinaCloud. A crawl that a person starts can run more than 6 hours after a deploy. Each request for a rule page (also a HEAD request) then starts a render that reads TinaCloud, which is up to 3,700 renders. At the fast rate, that load can cause TinaCloud throttling, which is the cause of the broken pages in #2801.

A local test with a test site of 200 pages gave: with 20 and 0, 20 requests at the same time and 9.2 seconds. With empty inputs, 5 requests at the same time, 250 ms delay and 14.2 seconds.

5. A comment no longer cancels a /deploy (pr-deploy.yml)

Each comment on a pull request starts pr-deploy.yml, and GitHub applies the workflow concurrency group before the validate job checks for /deploy. With the group pr-<number>, any comment cancelled a /deploy of that pull request. For example, run 43323425588 on #2802 stopped at page 1,921 of 4,211. Only a /deploy comment now shares the group of its pull request. This takes effect after the merge, because issue_comment uses the workflow files of main.

How to measure

1. /deploy on this pull request tests only item 3. pr-deploy.yml starts on issue_comment. For that event, GitHub uses the workflow file of the last commit on the default branch. A reusable workflow that is called with a local path comes from the same commit as its caller. All 40 recent runs of pr-deploy.yml have main as their branch. The build checks out the head commit of the pull request, so it uses the new .dockerignore. In the step "Build & push image", read the line transferring context (now 3.33GB) and the time of COPY . .. Do not add a comment to the pull request while that run is in progress: pr-deploy.yml has one concurrency group for each pull request, so each new comment cancels the run (example: run 26620005887).
2. Items 1 and 2 need one manual run from this branch. This step needs approval, because it deploys this branch to staging:
   gh workflow run build-and-deploy.yml --repo SSWConsulting/SSW.Rules --ref bs/2801-deploy-workflow-time -f environment=staging
   Start it when no other build is in progress. A second build makes the two builds slower (see ⚡ Performance - Decrease the production deploy time from 42 minutes to 20 minutes #2801). In that run, read:
   • the start times of "Deploy Infrastructure" and "Build and Push Container" (they must start together),
   • the output of "Check if the disk cleanup is necessary", and that the two cleanup steps are skipped,
   • the time of "Post Set up Docker Buildx",
   • the steps "Install Algolia Python dependencies" and "Sync Algolia index",
   • the first step of the deploy job.
3. Item 4 runs only in a production deploy. Production deploys use the workflow files of the release branch, so the first measurement is the first deploy of the next release branch. In the step "Run sitemap crawler", read the lines below "Throttling configuration" (20 and 0ms), the line "Duration", and the status counts. A manual run of crawl-sitemap.yml from this branch tests only the removal of the install steps. It uses the slow rate, and it can start one render for each page that is older than 6 hours.

Risks

1. A failed infrastructure job no longer stops the build. The build continues for approximately 20 minutes and pushes the image with the tag of the environment. The deploy job does not run. The registry then has an image that the site does not run yet, and an App Service restart can pull it. The same condition occurs now when the deploy job fails. In the last 150 runs of build-and-deploy.yml, the infrastructure job failed in 2 runs and the deploy job in 9.
2. The registry name is in the workflow files and in infra/deploy.ps1. If one of them changes, the new first step of the deploy job fails before the deploy starts. The build time of that run is lost.
3. az acr login of the build now occurs during the Bicep deployment of the registry. The deployment writes the same registry properties and role assignments again. Not tested.
4. The Algolia steps use a different Python when the cleanup does not run (see item 1). Tested with a local stand-in for the Algolia API, not with Algolia. If these steps fail, the build job fails and the deploy does not start.
5. The 30 GB minimum leaves approximately 12 GB of margin for a build that writes 18 GB. If a new runner image has less free space, the cleanup runs as before.
6. cleanup: false is correct only for runners that GitHub discards after the job.
7. The crawl job cannot fail the deploy. If the crawler does not start, the job summary shows "Sitemap crawl report not generated".
8. The crawl after a deploy sends 20 requests at the same time to production, 4 times the current number. Not tested on the site. If the site answers slowly, the crawler reports a timeout (10 seconds, 3 attempts) as an error in its report, and the deploy stays green. The fact that these requests start no render comes from the route table and from a separate test with single requests, not from a crawl at this rate.

Relates to #2801.

Screenshot (optional)

✏️ Not applicable.

🤖 Generated with Claude Code

https://claude.ai/code/session_01UTtNDpWnuaUUQ2winSSgMb

[bradystroud]

🎬 Explainer video for this pull request (about 40 seconds, no sound): the problem, the change, and how it helps.

🤖 Generated with Claude Code

pr-2806.mp4