Skip to content

gmail: an operator inside a quoted run is phrase text, not the operator - #1446

Open
aryankalantri wants to merge 1 commit into
brekkylab:mainfrom
aryankalantri:gmail-quoted-operator-fidelity
Open

aryankalantri wants to merge 1 commit into
brekkylab:mainfrom
aryankalantri:gmail-quoted-operator-fidelity

Conversation

@aryankalantri

Copy link
Copy Markdown

What changed

Backlot now reads a Gmail query the way Gmail does when a quoted run or a bracket touches an operator. An operator inside a quoted run is no operator, so the run is the phrase; a bracket glued to an operator's lone quote opens no group, so what follows the quote is the operator's value; and a quoted subject: value starting with a bracket reads as the word, the bracket being punctuation around the value, not part of it.

Why this is what the real API does

The table was measured against gmail.googleapis.com on 2026-10-25: "probe from:me", "from:me probe", "NOT from:me", "(NOT from:me)", "(from:me) probe" and "(probe from:me)" answered nothing, where a quoted run holding the operator alone still reads as that operator; subject:"(from:me and subject:"(from:me) answered nothing, where subject:" (from:me) answered what from:me answers; subject:"(probe" answered what subject:probe answers. "" and "()" already answered every message on main, so that row needed no code change.

Closes #1030

Verification

  • Tests — pytest tests/test_search.py tests/test_google.py -q on a .[dev] install assembled offline from wheels fetched via the PyPI JSON API (no index access): 3146 passed, 15 failed, 235 errors. The 15 failures and 235 errors are pre-existing environment gaps in this offline venv, none of them gmail: tests that call httpx directly fail with httpx InvalidURL, and other surfaces error at fixture setup on missing optional dependencies. A sample of the failing tests fails identically on pristine main. Every gmail test passes, and the issue's 9-query repro over the wire on the issue's corpus is all OK.

  • Optional surfaces that ran rather than skipped — the Gmail surface only (tests/test_search.py, tests/test_google.py); other surfaces were not run, their optional dependencies are not installed in this offline venv.

  • Lint — ruff check . all checks passed; ruff format --check . 206 files already formatted; python scripts/prose_check.py no findings.

  • A test fails without this change

  • A new endpoint serving corpus content is ACL-scoped, proved for both an admin and a scoped token

  • Comments state what was measured, not the history of the fix

@nuriyoo nuriyoo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this. I measured the spellings around the two bracket rules against gmail.googleapis.com on 2026-11-21, with a Workspace member's gmail.readonly token on its own mailbox of 840 messages, comparing messages.list id sets between 19:31 and 19:36 UTC. On this head K:"(from:me and K:"(from:me) answer nothing for all seven operators, as measured. Three things differ from the measurements, and the first is a row of the issue:

  1. A quoted subject: value with a closing bracket (inline, in _take). The issue's 2026-11-10 row has subject:"(probe)" answering what subject:probe answers; on this head it searches subjects for probe) and answers nothing. On 2026-11-21 subject:"probe)", subject:"( probe", subject:"(probe )", subject:"[probe]" and subject:"probe]" answered what subject:probe answers too, and the same held for label: and in:: label:"(inbox)", label:"(inbox", label:"[inbox]", label:"( inbox )", label:"((inbox))", in:"(inbox)" and in:"(sent)" answered what label:inbox and in:sent answer. The suggestion drops brackets and spaces at both ends of a quoted subject:, label: or in: value. The address operators already read through them: from:"(me)" and to:"(me)" answered what from:me and to:me answer, and this head agrees.
  2. label:"(inbox) with a lone quote (inline, in _gmail_group_items). Real answered what label:inbox answers, and -label:"(inbox) its negation. main answers label:"(inbox) that way; this head answers nothing, since the glued bracket now stays in the label's value. Item 1's suggestion covers it: the lone quote takes the rest of the search as the value, (inbox), and the brackets at its ends drop. The suggestion on the comment names the 2026-11-21 rows for the other six operators.
  3. A lone quote glued to a bracket that no ) closes (body diff, _GMAIL_LONE_QUOTE_UNCLOSED). Real answered nothing for subject:"(probe, subject:"((probe), label:"(inbox, in:"(inbox, from:"(me, to:"(me, cc:"(me, bcc:"(me and from:me subject:"(probe, and every message for -subject:"(probe, -label:"(inbox and NOT subject:"(probe: the rule _GMAIL_LONE_QUOTE_UNCLOSED records with whitespace after the quote holds without it. On this head subject:"(probe answers what subject:probe answers, from:"(bob what from:bob answers, and -subject:"(probe what -subject:probe answers. The diff below lets the whitespace before the bracket be empty; the pattern is outside this PR's lines, so the change is here rather than inline. With it subject:"(from:me reaches that rule before _gmail_group_items.

Smaller:

  1. "still reads as" in the _take comment and "as before" in the _gmail_group_items comment describe the change rather than the rule (inline).
  2. The two test comments restate the code comments; the suggestions point at them instead. The over-the-wire test checks subject:"(probe" against subject:probe, which is empty in that corpus, so both sides are empty on main as well; the suggestion uses rollout, which Rollout plan holds, and adds the label and open-bracket rows (inline).

Nit: the short lines at backlot/routers/google.py:1687-1688 and tests/test_search.py:801 are inside the suggestions, which rewrap them.

With the suggestions and the diff applied, tests/test_search.py and tests/test_google.py pass, and the new rows fail on this head. If you take them, the description's "Why" can name the 2026-11-10 and 2026-11-21 rows.

--- a/backlot/routers/google.py
+++ b/backlot/routers/google.py
@@ -852,16 +852,20 @@
     r'^[ \t]*"[ \t]+-*(?=(?i:from|to|cc|bcc|subject|label|in|has|after|before|newer|older'
     r"|newer_than|older_than):\S+[ \t]*$)"
 )
-# A lone quote after `subject:`, `label:`, `in:` or an address operator, then whitespace and a
-# `(` that no `)` closes. On 2026-10-25 (the record at `_GMAIL_NOT`): `subject:" (from:me`,
-# `subject:" ( from:me`, `subject:" (probe`, `subject:" ((from:me)`, `subject:" (from:me) (to:me`,
-# `label:" (from:me`, `from:" (from:me`, `to:" (from:me` and `probe subject:" (from:me` answered
-# nothing, and `-subject:" (from:me`, `-subject:" (probe`, `-label:" (from:me` and
-# `NOT subject:" (from:me` every message, where `subject:" (from:me)`, with the `)`, answered the
-# ids of `from:me`. `_parse_gmail_q` reads it before `_gmail_groups` can read the group.
+# A lone quote after `subject:`, `label:`, `in:` or an address operator, then a `(` that no `)`
+# closes, with or without whitespace in between. On 2026-10-25 (the record at `_GMAIL_NOT`):
+# `subject:" (from:me`, `subject:" ( from:me`, `subject:" (probe`, `subject:" ((from:me)`,
+# `subject:" (from:me) (to:me`, `label:" (from:me`, `from:" (from:me`, `to:" (from:me`,
+# `probe subject:" (from:me` and `subject:"(from:me` answered nothing, and `-subject:" (from:me`,
+# `-subject:" (probe`, `-label:" (from:me` and `NOT subject:" (from:me` every message, where
+# `subject:" (from:me)`, with the `)`, answered the ids of `from:me`. On 2026-11-21
+# `subject:"(probe`, `subject:"((probe)`, `label:"(inbox`, `in:"(inbox`, `from:"(me`, `to:"(me`,
+# `cc:"(me`, `bcc:"(me`, `label:"(from:me` and `from:me subject:"(probe` answered nothing, and
+# `-subject:"(probe`, `-label:"(inbox` and `NOT subject:"(probe` every message. `_parse_gmail_q`
+# reads it before `_gmail_groups` can read the group.
 _GMAIL_LONE_QUOTE_UNCLOSED = re.compile(
     r'(?<!\S)((?:NOT\s+)*)(-*)(?:subject|label|in|from|to|cc|bcc):\s*(?:"[ \t]*"\s*)*'
-    r'"(?![^"]*")\s+(.*)',
+    r'"(?![^"]*")\s*(.*)',
     re.IGNORECASE,
 )
 # A quoted run holding one operator and its value, with nothing around them but whitespace,

Comment thread backlot/routers/google.py
Comment on lines +1218 to +1223
# A bracket glued to an operator's lone quote is no group: the quote belongs to
# the operator, and what follows it is the operator's value, brackets and all.
# Measured against gmail.googleapis.com on 2026-10-25:
# `subject:"(from:me` and `subject:"(from:me)` answered nothing, where
# `subject:" (from:me)`, with a space after the quote, answered what `from:me`
# answers. A bracket after a space still opens a group, as before.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Item 2: the comment names the 2026-11-21 rows for all seven operators and drops "as before". With the body diff applied, a bracket left open never reaches this branch, so the rows here are the closed ones.

Suggested change
# A bracket glued to an operator's lone quote is no group: the quote belongs to
# the operator, and what follows it is the operator's value, brackets and all.
# Measured against gmail.googleapis.com on 2026-10-25:
# `subject:"(from:me` and `subject:"(from:me)` answered nothing, where
# `subject:" (from:me)`, with a space after the quote, answered what `from:me`
# answers. A bracket after a space still opens a group, as before.
# A bracket glued to an operator's lone quote is no group: the quote takes the rest
# of the search as the operator's value, brackets and all (`_GMAIL_LONE_QUOTE`).
# Measured against gmail.googleapis.com: on 2026-10-25 `subject:"(from:me)`
# answered nothing, where `subject:" (from:me)`, with a space after the quote,
# answered what `from:me` answers; on 2026-11-21 `label:"(from:me)`,
# `in:"(from:me)`, `from:"(from:me)`, `to:"(from:me)`, `cc:"(from:me)`,
# `bcc:"(from:me)` and `subject:"(probe) from:me` answered nothing, and
# `subject:"(probe)` and `label:"(inbox)` what `subject:probe` and `label:inbox`
# answer.

Comment thread backlot/routers/google.py
Comment on lines +1604 to +1605
# still reads as `label:inbox`. The closing quote is looked for after the
# operator's colon, not after the match: the value itself can swallow it.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Item 4: "still" describes the change; the rule reads the same without it.

Suggested change
# still reads as `label:inbox`. The closing quote is looked for after the
# operator's colon, not after the match: the value itself can swallow it.
# reads as `label:inbox`. The closing quote is looked for after the operator's
# colon, not after the match: the value itself can swallow it.

Comment thread backlot/routers/google.py
Comment on lines +1686 to +1691
# A quoted `subject:` value starting with a bracket reads as the word: the bracket
# is punctuation around the value, not part of it. Measured against
# gmail.googleapis.com on 2026-10-25: `subject:"(probe"` answered
# what `subject:probe` answers.
if key == "subject" and value.startswith('"'):
value = '"' + value[1:].lstrip("(")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Item 1: brackets and spaces drop at both ends of a quoted subject:, label: or in: value, as measured on 2026-10-25, 2026-11-10 and 2026-11-21.

Suggested change
# A quoted `subject:` value starting with a bracket reads as the word: the bracket
# is punctuation around the value, not part of it. Measured against
# gmail.googleapis.com on 2026-10-25: `subject:"(probe"` answered
# what `subject:probe` answers.
if key == "subject" and value.startswith('"'):
value = '"' + value[1:].lstrip("(")
# Brackets and spaces at the ends of a quoted `subject:`, `label:` or `in:` value are
# no part of it. Measured against gmail.googleapis.com: `subject:"(probe"` answered
# what `subject:probe` answers on 2026-10-25, and `subject:"(probe)"` on 2026-11-10;
# on 2026-11-21 `subject:"((probe"`, `subject:"probe)"`, `subject:"( probe"`,
# `subject:"(probe )"`, `subject:"[probe]"` and `subject:"probe]"` did too, and
# `label:"(inbox)"`, `label:"(inbox"`, `label:"[inbox]"`, `label:"( inbox )"`,
# `label:"((inbox))"`, `in:"(inbox)"` and `in:"(sent)"` answered what `label:inbox`
# and `in:sent` answer.
if key in _GMAIL_NEGATABLE and value.startswith('"'):
value = '"' + value.strip('"').strip(" ()[]") + '"'

Comment thread tests/test_search.py
Comment on lines +801 to +804
# A quoted run holding an operator and another term is a phrase, not the
# operator; a bracket glued to an operator's lone quote is no group, so the value
# keeps its bracket; and a quoted `subject:` value starting with a bracket reads as
# the word. A quoted run holding the operator alone still reads as that operator.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Item 5: a pointer to the comments that hold the measurements.

Suggested change
# A quoted run holding an operator and another term is a phrase, not the
# operator; a bracket glued to an operator's lone quote is no group, so the value
# keeps its bracket; and a quoted `subject:` value starting with a bracket reads as
# the word. A quoted run holding the operator alone still reads as that operator.
# The measurements in `_parse_gmail_q`'s `_take`, at `_GMAIL_LONE_QUOTE_UNCLOSED` and in
# `_gmail_group_items`.

Comment thread tests/test_search.py
Comment on lines +814 to +816
('subject:"(from:me', ("", {"subject": ["from:me"]})),
('subject:"(from:me)', ("", {"subject": ["from:me"]})),
('subject:"(probe"', ("", {"subject": ["probe"]})),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Items 1 and 3: subject:"(from:me reads at _GMAIL_LONE_QUOTE_UNCLOSED with the body diff, and rows for a closing bracket, […], label: and the negated open bracket.

Suggested change
('subject:"(from:me', ("", {"subject": ["from:me"]})),
('subject:"(from:me)', ("", {"subject": ["from:me"]})),
('subject:"(probe"', ("", {"subject": ["probe"]})),
('subject:"(from:me', ("", {"": ['subject:"(from:me']})),
('-subject:"(probe', ("", {})),
('subject:"(from:me)', ("", {"subject": ["from:me"]})),
('subject:"(probe"', ("", {"subject": ["probe"]})),
('subject:"(probe)"', ("", {"subject": ["probe"]})),
('subject:"[probe]"', ("", {"subject": ["probe"]})),
('label:"(inbox)', ("", {"label": ["inbox"]})),

Comment thread tests/test_search.py
Comment on lines +937 to +948
# An operator inside a quoted run is no operator, so the run is a phrase
# that matches nothing here; a bracket glued to an operator's lone quote is no group.
for q in (
'"probe from:ava"',
'"NOT from:ava"',
'"(from:ava) probe"',
'subject:"(from:ava',
'subject:"(from:ava)',
):
assert ids(q) == set(), q
assert ids('""') == ids('"()"') == every
assert ids('subject:"(probe"') == ids("subject:probe") == set()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Item 5: subject:rollout is not empty in this corpus, so the bracket rows tell the two readings apart; the label and open-bracket rows run over the wire too.

Suggested change
# An operator inside a quoted run is no operator, so the run is a phrase
# that matches nothing here; a bracket glued to an operator's lone quote is no group.
for q in (
'"probe from:ava"',
'"NOT from:ava"',
'"(from:ava) probe"',
'subject:"(from:ava',
'subject:"(from:ava)',
):
assert ids(q) == set(), q
assert ids('""') == ids('"()"') == every
assert ids('subject:"(probe"') == ids("subject:probe") == set()
for q in (
'"probe from:ava"',
'"NOT from:ava"',
'"(from:ava) probe"',
'subject:"(from:ava',
'subject:"(from:ava)',
'subject:"(rollout',
'from:"(bob',
'subject:"(rollout) from:bob',
):
assert ids(q) == set(), q
assert ids('""') == ids('"()"') == ids('-subject:"(rollout') == every
rollout = ids("subject:rollout")
assert rollout and rollout != every
for q in (
'subject:"(rollout"',
'subject:"(rollout)"',
'subject:"[rollout]"',
'subject:"(rollout)',
):
assert ids(q) == rollout, q
assert ids('label:"(inbox)"') == ids('label:"(inbox)') == ids("label:inbox")
assert ids('-label:"(inbox)') == every - ids("label:inbox")

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants