[bradystroud]

Description

✏️

Relates to #2801. This is the fourth task of phase 2.

• generateStaticParams in app/[filename]/page.tsx reads the list of rules from rule-folders.json. Before, it sent 20 TinaCloud requests of 200 rules each.
• The workflow step "Validate rule categories" writes the file before the Docker build. No workflow file changes.
• If the file is missing or not valid, generateStaticParams gets the list from Tina as before. Local development needs no new step.
• generateStaticParams does not send the generated categoryConnection query. TinaCloud answered this query with status 500 in 7 of 7 build logs from 10 to 12 Nov. This is a separate commit. See "The removed category request".
• A new log line shows the source of the list and the number of rules.
• The paging code moved from page.tsx to lib/ruleFolders.ts with no change in behavior, so that unit tests can cover the two sources.
• README.md says that the validation script writes the file.

Why

• In CI, the time from Collecting page data to Generating static pages was 80 to 121 seconds in 7 build logs from 10 to 12 Nov. In production run 42969510432 it was 1.5 minutes (06:10:37 to 06:12:09). In that time, the build sends the 20 rule requests, the category request that fails, and 6 small category requests.
• A rule request that failed ended the paging with no error. The build then prerendered only the rules of the earlier pages, and the sitemap of that build did not list the other rules.

How the file gets to the build

1. scripts/validate-rule-categories.js writes the names of all rule folders of the content checkout to rule-folders.json in the repository root when its check passes. The step "Validate rule categories" in build-artifacts.yml and pnpm prepare:content both run this script. It is the only Node content preparation that the two run. The script prints one line with the path and the number of names.
   • A /deploy comment runs the workflow files of main and the scripts of the pull request. main already has this step, so /deploy can measure this change before the merge. A new workflow step would not run in that test.
2. .gitignore has an entry for the file. .dockerignore has no entry that matches it, so the file is in the Docker build context. The file is not in the content checkout folder.
3. COPY . . in the builder stage of the Dockerfile copies the file to /app. next build runs in /app and reads the file from its working directory.

A test image with the same COPY . . line and the .dockerignore of the repository has /app/rule-folders.json with 3,933 entries.

What generateStaticParams returns

	Before	After
Rule identifier	The folder name (first part of _sys.relativePath). Not the uri field.	The folder name: each folder in public/uploads/rules that has a rule.mdx file.
Archived rules	Included	Included
Categories	Included, from 3 sources: categoryConnection, topCategoryWithIndexQuery, category-uri-title-map.json	Included, from 2 sources: topCategoryWithIndexQuery, category-uri-title-map.json
Order	Rules sorted by file path, then categories	Rules sorted by folder name, then categories
Duplicates	Removed (Set)	Removed (Set)
A rule request fails	Warning, then a list that is not complete	The file needs no request. The fallback keeps the old behavior.
Content 2d41451	3,933 rules + 239 categories = 4,172	3,933 rules + 239 categories = 4,172

The removed category request

generateStaticParams sent the generated categoryConnection query (50 categories for each request) to get the file names of all categories.

Measured:

• CI: each of the 7 build logs from 10 to 12 Nov has the line Error fetching category: Server responded with status code 500, 0.1 to 0.2 seconds before the line with the total. After that error, the code stops the paging of this request.
• Local Tina server, content 2d41451: the first request fails after 7.4 seconds with Unable to find record public/uploads/rules/avoid-redundant-links.mdx. avoid-repetition/rule.mdx has that path in its related list, and the file is avoid-redundant-links/rule.mdx. This pull request does not show that the status 500 of TinaCloud has the same cause.

The static params are the same set before and after:

Source of category names	Names for content 2d41451
categoryConnection (removed): each category file but not index	239, counted from the files in categories, because the request fails
topCategoryWithIndexQuery: the children of the top categories	239 (the same 239 names)
category-uri-title-map.json	238 (all are in the 239)

• Before, with the request (failed or not): 3,933 rules + 239 categories = 4,172.
• After: 3,933 rules + 239 categories = 4,172.
• The CI logs of 12 Nov show total=4172 and 4,211 static pages. The local production build of this branch shows the same two numbers.

A category file that no top category lists is the only name that the request could add. Such a file is now not prerendered. dynamicParams is not set, so it has its default value (true), and the first request renders the URL. The page finds a category only in the index of a top category, so it does not find this category. It then looks for a rule with that name, finds none, and renders the fallback page, which shows NOT FOUND to visitors. Before, the build prerendered that fallback page when the request did not fail. The content has no such file.

Proof that the list is the same

1. Local Tina server, content 2d41451. The query gives 3,933 names and the file gives 3,933 names. 0 names are only in the query. 0 names are only in the file.
   • 46 names have a different position. The query sorts by file path (<folder>/rule.mdx) and the file sorts by folder name. Example: the query has automated-ui-testing-sparingly before automated-ui-testing, because - sorts before /. The order has no effect, because Next.js prerenders each entry.
2. Production sitemap (the list from TinaCloud). The sitemap of the production build of 12 Nov (file date 06:36 UTC) has 3,709 URLs: 3,454 rules, 239 categories and 16 other pages. The checkout 2d41451 (12 Nov 05:15 UTC) has 3,454 rules that are not archived and 239 categories. The two sets are the same. The sitemap does not list archived rules, so this check does not cover the 479 archived rules.
3. Local production build with the file (local Tina server). The build generates 4,211 static pages. That is the number in the CI logs of 12 Nov. prerender-manifest.json has 4,172 /[filename] routes. They are the same set as the list that the code on main makes from the same content (the same requests, sent to the local Tina server): 0 differences. The build log has no [ClientFallbackPage] line.

One known difference: the Tina collection matches rule.mdx at each depth (match.include: "**/rule"). The query thus also returns the first folder of a path such as a/b/rule.mdx. The file has only folders with a direct rule.mdx, because the page loads <folder>/rule.mdx. The content has no such file.

When the file and TinaCloud differ

The file comes from the content checkout of the job. The pages get their data from TinaCloud, which has the head of the content branch. The head can be some commits newer than the checkout. Before next build starts, tinacms build waits until TinaCloud reports that its index of the branch is complete.

Case	Possible cause	Result with this change	Result before
A rule is in the file but not in TinaCloud	A newer commit deleted or renamed the rule folder. Or TinaCloud did not get the commit of the checkout.	The build prerenders the URL. getRuleData gets "Unable to find record", so the page is the fallback page, which shows NOT FOUND to visitors. The sitemap lists the URL. The page stays until the Tina webhook revalidates the path, or until the next deploy.	The URL was not prerendered and was not in the sitemap. The first request rendered the page: the same fallback page, or the rule if TinaCloud had it at that time.
A rule is in TinaCloud but not in the file	A newer commit added the rule or renamed its folder.	The build does not prerender the URL. dynamicParams is not set, so it has its default value (true): the first request renders the page from TinaCloud and Next.js stores it. The sitemap of this build does not list the URL.	The build prerendered the URL if TinaCloud had the commit before Collecting page data.

For production, the job "Check Content Freshness & Retrigger if Stale" starts a new deploy when the head of the content branch is not the commit of the checkout. That deploy uses the newer checkout. For staging and pull request slots, the difference stays until the next deploy.

Phase 3 of #2801 removes this difference, because the build then reads the pages from the same checkout.

Times

CI (TinaCloud):

• Before: 80 to 121 s from Collecting page data to Generating static pages (7 logs: 84.5, 91.6, 91.8, 80.3, 120.8, 92.8 and 104.5 s). That is approximately 1.5 minutes.
• After: less than 15 s (estimate, not measured). The 6 topCategoryWithIndexQuery requests stay. They load approximately 500 documents. The 20 rule requests loaded 3,933 documents. Measure the time with /deploy on this pull request.

Local Tina server, content 2d41451 (measured):

	Before	After
20 allRulesPaths requests	5.6 s (second run), 14.5 s (first run)	No request
categoryConnection request	7.4 s, then it fails	No request
6 topCategoryWithIndexQuery requests	0.3 s	0.3 s
generateStaticParams in next dev	13.9 s	Not measured
Collecting page data in next build	Not measured	0.7 s and 2.3 s (2 runs, the second with a high machine load)
"Validate rule categories" command	0.6 s in CI (11 Nov)	1.0 s locally, with the list

How to check a CI run

• The step "Validate rule categories" prints 📋 validate-rule-categories: wrote <N> rule folders to .../rule-folders.json.
• The Docker build prints 📋 generateStaticParams: rules=<N> source=rule-folders.json. The two numbers must be the same.
• If the build prints source=Tina (rule-folders.json: <cause>), the file did not get to the build, and the build used the paged requests.
• 🚀 generateStaticParams: rules + categories total=<M> and the number of static pages must agree with a build of the same content before this change.
• The log must not have the line Error fetching category: Server responded with status code 500.

Checks

• pnpm exec tinacms build --local --skip-indexing --skip-cloud-checks (with --port 4102 --datalayer-port 9102), then git diff --quiet -- tina/tina-lock.json: no difference.
• pnpm lint: 344 files, no problem.
• pnpm test: 30 suites passed and 1 skipped. 260 tests passed and 8 skipped. 28 tests are new.
• tsc --noEmit: no error.
• pnpm run validate:rule-categories <content> (the CI command) and pnpm prepare:content: each wrote 3,933 names. With content that fails the check, the command stops with exit code 1 and writes no file.
• Local production build with the local Tina server: tinacms build --local --skip-cloud-checks -c "NODE_ENV=production next build" (with the same ports): success, 4,211 static pages.

Notes

• Next.js file tracing copies rule-folders.json (170 KB) to .next/standalone, so the file is also in the final image. The production server does not read it: Next.js 16.2.4 calls generateStaticParams only in the build and in next dev.

Not verified

• A CI run. No workflow was started for this pull request.
• The time that TinaCloud needs for the categoryConnection request, and for the 6 topCategoryWithIndexQuery requests that stay.
• The Docker build with the Dockerfile of the repository. It needs the build secrets.
• The 479 archived rules in the list from TinaCloud. The local Tina server returns them, but the production sitemap does not list archived rules.

Merge order

#2802 changes the imports, removes lines 19 to 159 and changes generateMetadata in app/[filename]/page.tsx. A three-way merge of the two changes to that file (git merge-file) has no conflict.

Screenshot (optional)

✏️

Log lines from local runs with content 2d41451:

# Command of the step "Validate rule categories": pnpm run validate:rule-categories <content>
✅ validate-rule-categories: every rule belongs to an existing category
📋 validate-rule-categories: wrote 3933 rule folders to <repository>/rule-folders.json

# next build, with the file
  Collecting page data using 15 workers ...
📋 generateStaticParams: rules=3933 source=rule-folders.json
🚀 generateStaticParams: rules + categories total=4172
  Generating static pages using 15 workers (0/4211) ...
✓ Generating static pages using 15 workers (4211/4211) in 5.4min

# next dev, without the file
📋 generateStaticParams: rules=3933 source=Tina (rule-folders.json: file not found)
🚀 generateStaticParams: rules + categories total=4172

🤖 Generated with Claude Code

https://claude.ai/code/session_01UTtNDpWnuaUUQ2winSSgMb

[bradystroud]

🎬 Explainer video for this pull request (about 40 seconds, no sound): the problem, the change, and how it helps.

🤖 Generated with Claude Code

pr-2803.mp4