Repository navigation
github: unserved searches return 422 for missing q or repository_id ahead of credential - #1447
duhanguler wants to merge 1 commit into
Conversation
|
Closing this one: until one of your pull requests here is merged, one may be open at a time, a draft included, and three after that (Pull requests, item 7). Open now: #1438. The branch stays in your fork; open a pull request from it again when that leaves room. |
nuriyoo
left a comment
There was a problem hiding this comment.
Thanks for this. I measured what counts as blank against api.github.com on 2026-11-21 between 19:51 and 19:56 UTC, each target sent once with no Authorization. One thing differs from real:
- What counts as blank (inline, in
_an_unserved_search_refusal).str.strip()strips U+001C, U+001D, U+001E and U+001F, and real reads each as a value: on/search/repositoriesaqof each of the four, as of U+200B, was the search's 200, and on/search/commitsU+001F was the 200 where U+00A0 was the 422. On/search/labels,q=%1Fwith norepository_idwas the#search-labels422, where this head answers theq422 with the/v3/searchbody, and withq=xarepository_idof U+001F was the 404 "Not Found" where a tab and U+00A0 were the 422. A space, a tab, LF, VT, FF, CR, U+0085, U+00A0, U+1680, U+2000, U+2028, U+202F, U+205F and U+3000 were each the 422 on/search/repositories. That is the linebacklot.odata.strip_blanksdraws, Python's whitespace but U+001C to U+001F, so the suggestion reads both values with it, and aqof U+001F on the other four searches, or arepository_idof it, goes on to the 401 or the 404 a non-blank value gets here. The import is outside this PR's lines and is in the diff below.
Smaller:
- The order in the docstrings (inline). The summary line puts the 422 "ahead of the credential check", but a credential that does not resolve is the 401 first, as the helper's measurement and the test have it. The paragraph in
refuse_an_unserved_operationand the first test's docstring restate the echo and thex-ratelimit-*from the helper's docstring; the suggestions state the order and point at the helper. if request.method != "GET": return Nonedoes not return (inline).refuse_an_unserved_operationcalls the helper only for whatis_an_unserved_operationmatches, which on the five paths isGETalone, and aHEADreaches it as theGETthroughbacklot.main.answer_head_as_the_get_without_its_body:HEAD /github/search/repositories?q=is the 422 withcontent-length: 164with and without the check.
Nit: tests/test_github.py:3137-3138 splits the code span 401 Bad credentials across the line break; item 2's suggestion there rewraps it.
With the suggestions and the diff applied, tests/test_github.py passes, 577 tests, and the U+001F rows fail on this head (five of the six test cases; the sixth has no such row). If you take them, the description's "Why" can name the 2026-11-21 rows.
Not in this PR. The served /search/issues and /search/code read a blank q with q.strip() too, and differ from real in other ways; filed as #1449.
--- a/backlot/routers/github.py
+++ b/backlot/routers/github.py
@@ -24,7 +24,7 @@
from pydantic import BaseModel, ConfigDict
from starlette.types import Scope
-from backlot import auth, published, store, synth
+from backlot import auth, odata, published, store, synth
from backlot.acl import Caller
from backlot.config import get_settings
from backlot.graphql import body as request_body| """Real's 422 for a missing or blank `q` on the five searches Backlot does not serve, or a | ||
| missing or blank `repository_id` on `/search/labels`, ahead of the credential check. |
There was a problem hiding this comment.
Items 1 and 2: a credential that does not resolve is the 401 ahead of this 422, so the line names what blank means instead of an order.
| """Real's 422 for a missing or blank `q` on the five searches Backlot does not serve, or a | |
| missing or blank `repository_id` on `/search/labels`, ahead of the credential check. | |
| """Real's 422 for a missing or blank `q` on the five searches Backlot does not serve, or a | |
| missing or blank `repository_id` on `/search/labels`, a value being blank when | |
| :func:`backlot.odata.strip_blanks` leaves nothing of it. |
| rules. | ||
| """ | ||
| if request.method != "GET": | ||
| return None | ||
| path = request.scope["path"].removeprefix("/github") | ||
| if path not in _UNSERVED_SEARCHES: | ||
| return None | ||
| if not request.query_params.get("q", "").strip(): | ||
| return _search_validation_failed("q") | ||
| if path == "/search/labels" and not request.query_params.get("repository_id", "").strip(): |
There was a problem hiding this comment.
Items 1 and 3: the 2026-11-21 rows, odata.strip_blanks for both values, and no method check. The import is in the review's diff.
| rules. | |
| """ | |
| if request.method != "GET": | |
| return None | |
| path = request.scope["path"].removeprefix("/github") | |
| if path not in _UNSERVED_SEARCHES: | |
| return None | |
| if not request.query_params.get("q", "").strip(): | |
| return _search_validation_failed("q") | |
| if path == "/search/labels" and not request.query_params.get("repository_id", "").strip(): | |
| rules. On 2026-11-21, each target sent once with no `Authorization`, a `q` of a space, a tab, | |
| LF, VT, FF, CR, U+0085, U+00A0, U+1680, U+2000, U+2028, U+202F, U+205F or U+3000 was the 422 on | |
| `/search/repositories`, where U+001C, U+001D, U+001E, U+001F and U+200B were each the search's | |
| 200, as U+001F was on `/search/commits` where U+00A0 was the 422. On `/search/labels`, `q` of | |
| U+001F was the 422 for the missing `repository_id`, and with `q=x` a `repository_id` of a tab | |
| or U+00A0 was that 422, where U+001F was the 404 "Not Found". | |
| """ | |
| path = request.scope["path"].removeprefix("/github") | |
| if path not in _UNSERVED_SEARCHES: | |
| return None | |
| if not odata.strip_blanks(request.query_params.get("q", "")): | |
| return _search_validation_failed("q") | |
| if path == "/search/labels" and not odata.strip_blanks( | |
| request.query_params.get("repository_id", "") | |
| ): |
| On the five unserved searches (:data:`_UNSERVED_SEARCHES`), a missing or blank `q` — and on | ||
| `/search/labels`, a missing or blank `repository_id` — is answered with real's 422 ahead of | ||
| :func:`_require` (:func:`_an_unserved_search_refusal`), carrying the version echo and the five | ||
| `x-ratelimit-*` counted in `search`. |
There was a problem hiding this comment.
Item 2: the order as the code runs it, with what the 422 answers left to the helper's docstring.
| On the five unserved searches (:data:`_UNSERVED_SEARCHES`), a missing or blank `q` — and on | |
| `/search/labels`, a missing or blank `repository_id` — is answered with real's 422 ahead of | |
| :func:`_require` (:func:`_an_unserved_search_refusal`), carrying the version echo and the five | |
| `x-ratelimit-*` counted in `search`. | |
| On the five searches of :data:`_UNSERVED_SEARCHES`, :func:`_an_unserved_search_refusal`'s 422 | |
| comes after a credential that does not resolve and a version real does not have, and ahead of | |
| :func:`_require`. |
| """Real answers a missing or blank `q` on the five unserved searches with its 422 ahead of a | ||
| credential, echoing the API version and counting against `search` | ||
| (`backlot.routers.github._an_unserved_search_refusal`), while `Bearer nope` stays `401 Bad | ||
| credentials` and a non-blank `q` on the four non-label searches keeps `401` without a token.""" |
There was a problem hiding this comment.
Item 2: the echo and the search count are the helper's docstring's; the test docstring keeps the pointer and the order the rows fix.
| """Real answers a missing or blank `q` on the five unserved searches with its 422 ahead of a | |
| credential, echoing the API version and counting against `search` | |
| (`backlot.routers.github._an_unserved_search_refusal`), while `Bearer nope` stays `401 Bad | |
| credentials` and a non-blank `q` on the four non-label searches keeps `401` without a token.""" | |
| """`backlot.routers.github._an_unserved_search_refusal`'s 422 for a missing or blank `q` comes | |
| ahead of the 401 for no token, where `Bearer nope` is the 401 `Bad credentials` and a non-blank | |
| `q` on the four non-label searches goes on to the 401 without a token.""" |
| assert (r.headers["x-ratelimit-resource"], r.headers["x-ratelimit-limit"]) == ( | ||
| "search", | ||
| limit, | ||
| ), (query, headers) |
There was a problem hiding this comment.
Item 1: a tab and U+00A0 stay the 422. They have a loop of their own because with a Digest x caller in it the Bearer<TAB>nope row below would be the eleventh request in the unparseable search window and get the 403.
| ), (query, headers) | |
| ), (query, headers) | |
| for query in ("?q=%09", "?q=%C2%A0"): | |
| for headers in ({}, gh_admin_h): | |
| r = c.get(path + query, headers=headers) | |
| assert (r.status_code, r.json()) == (422, _MISSING_Q_BODY), (query, headers) |
| for headers in ({}, {"Authorization": "Digest x"}): | ||
| r = c.get(path + "?q=x", headers=headers) | ||
| assert (r.status_code, r.json()["message"]) == (401, "Requires authentication"), headers |
There was a problem hiding this comment.
Item 1: a q of U+001F is a value and goes on as q=x does. These rows fail on this head.
| for headers in ({}, {"Authorization": "Digest x"}): | |
| r = c.get(path + "?q=x", headers=headers) | |
| assert (r.status_code, r.json()["message"]) == (401, "Requires authentication"), headers | |
| for query in ("?q=x", "?q=%1F"): | |
| for headers in ({}, {"Authorization": "Digest x"}): | |
| r = c.get(path + query, headers=headers) | |
| assert (r.status_code, r.json()["message"]) == ( | |
| 401, | |
| "Requires authentication", | |
| ), (query, headers) |
| path = "/github/search/labels" | ||
| bad_version = {"X-GitHub-Api-Version": "1999-01-01"} | ||
|
|
||
| for query in ("?q=x", "?q=x&repository_id=", "?q=x&repository_id=%20"): |
There was a problem hiding this comment.
Item 1: q=%1F with no repository_id is the #search-labels 422, and a repository_id of U+00A0 is blank.
| for query in ("?q=x", "?q=x&repository_id=", "?q=x&repository_id=%20"): | |
| for query in ( | |
| "?q=x", | |
| "?q=%1F", | |
| "?q=x&repository_id=", | |
| "?q=x&repository_id=%20", | |
| "?q=x&repository_id=%C2%A0", | |
| ): |
| for headers in ({}, {"Authorization": "Digest x"}): | ||
| r = c.get(path + "?q=x&repository_id=1", headers=headers) | ||
| assert (r.status_code, r.json()["message"]) == (401, "Requires authentication"), headers |
There was a problem hiding this comment.
Item 1: a repository_id of U+001F is a value and goes on as repository_id=1 does.
| for headers in ({}, {"Authorization": "Digest x"}): | |
| r = c.get(path + "?q=x&repository_id=1", headers=headers) | |
| assert (r.status_code, r.json()["message"]) == (401, "Requires authentication"), headers | |
| for query in ("?q=x&repository_id=1", "?q=x&repository_id=%1F"): | |
| for headers in ({}, {"Authorization": "Digest x"}): | |
| r = c.get(path + query, headers=headers) | |
| assert (r.status_code, r.json()["message"]) == ( | |
| 401, | |
| "Requires authentication", | |
| ), (query, headers) |
What changed
refuse_an_unserved_operationinbacklot/routers/github.pynow checks_an_unserved_search_refusalon the five search endpoints GitHub publishes that Backlot does not serve (GET /search/commits,/search/labels,/search/repositories,/search/topics,/search/users):q(absent,q=, or whitespace such asq=%20) on all five searches returns real's422 Validation Failed(errors: [{"resource": "Search", "field": "q", "code": "missing"}],documentation_url: "https://docs.github.com/v3/search") ahead of_require, echoingX-GitHub-Api-Version-Selectedand counting againstx-ratelimit-resource: search(limit: 10for unauthenticated / unparseableAuthorizationcallers,limit: 30for a token).GET /search/labelswith a non-blankq, a missing or blankrepository_id(absent,repository_id=, orrepository_id=%20) returns real's422 Validation Failed(documentation_url: "https://docs.github.com/rest/search/search#search-labels") under the same credential, version, and rate-limit rules._validate_bad_credential) and API version validation (_validate_api_version) still run ahead of the 422, soBearer nopeand token-shapedBasicvalues return401 Bad credentialswithout echo or rate-limit headers,Bearer<TAB>nopereturns401 Bad credentialswithout echo and counted insearch, andX-GitHub-Api-Version: 1999-01-01returns400 Bad Requestwithout echo and counted insearch.qon the four non-label searches (and with bothqandrepository_idpresent on/search/labels), requests without a token keep the401 Requires authenticationand requests with a token keep the router's404until those searches are served.test_github_unserved_searches_refuse_missing_query_ahead_of_credentialandtest_github_search_labels_refuses_missing_repository_id_ahead_of_credentialintests/test_github.py.Closes #1182.
Why this is what the real API does
Measured against
api.github.comon 2026-10-31 and re-measured on 2026-11-21 (UTC), each request on its own cache-busted URL:/search/commits,/search/labels,/search/repositories,/search/topics, and/search/users, noq,q=, andq=%20all returned422 Validation Failed(errors: [{"resource":"Search","field":"q","code":"missing"}],documentation_url: "https://docs.github.com/v3/search",status: "422"), carryingX-GitHub-Api-Version-Selected: 2022-11-28and the fivex-ratelimit-*headers counted insearch(limit: 10for noAuthorization,Digest x,nope, emptyAuthorization,Basic eDp5, andBasic<TAB>!!!!;limit: 30for a token).X-GitHub-Api-Version: 1999-01-01returned400 Bad Requestwith no version echo and counted insearch(for noAuthorization,Digest x,Basic eDp5, and a token);Bearer nopeand a token-shapedBasicvalue returned401 Bad credentialswith no echo and nox-ratelimit-*;Bearer<TAB>nopereturned401 Bad credentialswith no echo and counted insearch(limit: 10)./search/labels?q=x, absentrepository_id,repository_id=, andrepository_id=%20returned422 Validation Failed(documentation_url: "https://docs.github.com/rest/search/search#search-labels",status: "422") withX-GitHub-Api-Version-Selected: 2022-11-28and counted insearch(limit: 10for noAuthorizationandDigest x,limit: 30for a token), and400 Bad Requestwhen pinned to1999-01-01.Verification
Tests —
PYTHONWARNDEFAULTENCODING=1 pytest -rson a.[dev]install acrosstests/test_github.py,tests/test_openapi.py,tests/test_packaging.py,tests/test_docs.py, andtests/test_prose_check.py: all passing. Confirmed all 6 new test cases (test_github_unserved_searches_refuse_missing_query_ahead_of_credential×5 +test_github_search_labels_refuses_missing_repository_id_ahead_of_credential) fail without the router change and pass with it.Optional surfaces that ran rather than skipped —
gittests intests/test_github.pyandtests/test_prose_check.py;.[official-sdk],.[mcp],.[llamaindex],.[mirage]and.[fsspec]skipped (not installed in.[dev]).Lint —
ruff check . && ruff format --check .andruff check --preview --select PLW1514 .: all checks passed (206 files formatted);python scripts/prose_check.py: 0 findings.A test fails without this change
A new endpoint serving corpus content is ACL-scoped, proved for both an admin and a scoped token
Comments state what was measured, not the history of the fix