[Flo0807]

Stacked on #2455.

Problem

Backpex.Adapters.Ecto joined every group of relationship aggregates as an uncorrelated subquery (LEFT JOIN (SELECT root.pk, agg(...) FROM root JOIN related GROUP BY root.pk), DISTINCT ON for :first). It aggregated the related items of all items on every read, also when the aggregate was only selected for the 15 items of a page or for the single item of the show view.

Change

For a query with a limit that selects aggregates, the adapter now finds the primary keys of the page first and restricts every aggregate subquery to them:

WITH backpex_page AS (
  SELECT u.id AS key FROM users u WHERE ... ORDER BY u.username LIMIT 15 OFFSET 0
)
SELECT u0.*, coalesce(s1.value_1, 0)
FROM users u0
LEFT JOIN (
  SELECT su.id AS key, count(*) AS value_1
  FROM users su JOIN posts sp ON sp.user_id = su.id
  WHERE su.id IN (SELECT key FROM backpex_page)
  GROUP BY su.id
) s1 ON s1.key = u0.id
WHERE u0.id IN (SELECT key FROM backpex_page)
ORDER BY u0.username

• backpex_page applies the filter, the search, the sorts, the offset and the limit. It joins only the aggregates these read, as before over all items (sorting or filtering by an aggregate needs the values of all items).
• The outer query selects the items of these keys in the order of the sorts. Its aggregates are still one subquery per group (aggregates that differ only in :value share it), named from the fixed pool of atoms, now with WHERE pk IN (SELECT key FROM backpex_page).
• Unchanged, grouped over all items: queries without a limit (e.g. an export via ecto_query/1), the aggregates of metrics (aggregate/1), preload queries of Backpex.Query.related/1, aggregates in exists or in another aggregate, and base queries that may return an item more than once or are limited themselves (an :options_query with a join, select, distinct, group or limit).
• No API change. Where.join_aggregates/3 takes of: keys.

I chose the common table expression over LEFT JOIN LATERAL and correlated subqueries in the select list: it does not depend on the planner postponing the subqueries behind sort, limit and offset, needs no named parent binding, and keeps one subquery per group.

Measurements

PostgreSQL 18.7, test app schema, 100,000 users, 2,000,000 posts (skewed, about 20 per user), page of 15, EXPLAIN (ANALYZE, BUFFERS) execution time of the second run. "FK index" is an index on posts.user_id.

Query	before, FK index	after, FK index	before, no FK index	after, no FK index
page sorted by username, aggregate(:count, :posts) selected	705 ms	0.4 ms	1356 ms	373 ms
page 500 (offset 7485) of the same	729 ms	2.6 ms	1540 ms	361 ms
same, filtered by a username prefix	565 ms	22 ms	1255 ms	387 ms
page, aggregate(:first, :posts, value: title, sort: [likes: :desc]) selected	2021 ms	0.7 ms	1807 ms	365 ms
page, six computed values of UserStatsLive (four groups)	5706 ms	1.1 ms	5787 ms	1378 ms
page sorted by the count	615 ms	758 ms	1304 ms	1756 ms
page filtered by count > 100	667 ms	602 ms	1342 ms	1644 ms
count query, count only selected	14 ms	13 ms	14 ms	13 ms
count query, filtered by count > 100	630 ms	613 ms	1268 ms	1304 ms

Shared buffers of the first row: 291,197 before, 184 after (FK index).

Not improved, and worth knowing:

• Sorting or filtering by an aggregate still aggregates the whole table; the differences in these rows with the FK index are run-to-run noise.
• Without an index on the foreign key, every group of selected aggregates costs one sequential scan of the related table. For a page that is sorted or filtered by an aggregate, this scan comes on top (about +30 % in the two rows above). The guide now recommends the index.

Tests

• test/adapters/ecto_test.exs: the shape of the query with a limit (keys without aggregates, restricted subqueries, sort without limit in the outer query), with a sort and a filter by aggregates, and the cases that keep the grouped join (no limit, aggregate only filtered by, options query with a join, metric).
• integration_test/options/computed_test.exs: pages read the same aggregates as the query without a limit, for three sorts and three filters.
• MIX_ENV=test mix ci passes.

Surfaces

Views: index and show/edit (Resource.get/4 has limit: 1) go through the new query; metrics and options keep the old one. Adapter boundary: the change is inside Backpex.Adapters.Ecto. User docs: cost notes in guides/fields/computed-fields.md and guides/upgrading/v0.25.md. No texts, markup or fixtures changed.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!