Skip to content

avoid counting all siblings for :first-child and :nth-*() bounds - #223

Merged
wRAR merged 1 commit into
scrapy:masterfrom
itecz26:nth-child-positional
Nov 9, 2026
Merged

wRAR merged 1 commit into
scrapy:masterfrom
itecz26:nth-child-positional

Conversation

@itecz26

@itecz26 itecz26 commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

:first-child, :last-child, :only-child, the -of-type variants and the bounds of :nth-*() are translated to count(preceding-sibling::*) (or following-sibling) compared with a constant, and count() walks every sibling for every element it's evaluated on, so matching is quadratic in the number of siblings. on lxml 6.1.3 li:first-child, li:last-child, li:nth-child(2) and li:nth-child(-n+3) each take ~3.1 s of CPU on a 281 KiB page of 32000 <li> (0.76 s at 16000) and li:only-of-type 4.1 s, so a page with one long flat list is a cheap way to stall a scrapy/parsel spider that runs any of these (an algorithmic-complexity DoS where the attacker only has to serve the page). testing count >= k as ***-sibling::***[k] lets libxml2 stop at the kth sibling, e.g. :first-child becomes not(preceding-sibling::*[1]), and the same selectors take 4.5 to 11 ms on that page. only the modulo of :nth-*(an+b) with |a| > 1 still needs count(), XPath 1.0 has no sibling index to use instead. old and new translations select the same elements on ~35k random selectors (every one of these forms, namespaced types, inside :not()/:has()/:is() and after combinators) and test_nth_child still passes; :nth-child(0) now short-circuits to [0] like -n does.

:first-child, :last-child, :only-child, the -of-type variants and the
bounds of :nth-*() compared count(preceding-sibling::*) (or
following-sibling) with a constant. count() walks every sibling for
every element it is evaluated on, so matching was quadratic in the
number of siblings: li:first-child took about 3 s on a list of 32000
items. Test count >= k as ***-sibling::***[k] instead, which stops at
the kth sibling. Only the modulo of :nth-*(an+b) with |a| > 1 still
needs count().
@codecov

codecov Bot commented Nov 3, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 100.00%. Comparing base (2834f2c) to head (261c381).

Additional details and impacted files
@@            Coverage Diff            @@
##            master      #223   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files            3         3           
  Lines         1019      1022    +3     
  Branches       170       171    +1     
=========================================
+ Hits          1019      1022    +3     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@wRAR
wRAR merged commit fc8cb51 into scrapy:master Nov 9, 2026
23 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants