Repository navigation
fix(chat): parse $…$ spans according to pandoc's tex_math_dollars rules - #3300
Merged
tpae merged 1 commit intoNov 14, 2026
Merged
Conversation
A rejected `$…$` span emitted its opening `$` as text but left the closing
`$` in the stream, where it was retested as an *opening* delimiter. When
that closer was followed by punctuation it passed the opener tests, paired
with the next `$` whose predecessor was non-space — skipping the real math
span's opener, which prose always space-precedes — and every word in
between was handed to KaTeX. Math mode drops whitespace, so the sentence
became unreadable and a legitimate span on the same line vanished.
The content guards that caused the rejection (`looksLikeLatex`,
`looksLikeDollarMath`: contains LaTeX syntax / does not start with a digit /
shorter than 64 scalars) are replaced by pandoc's `tex_math_dollars`
delimiter rules:
- the opening `$` must have a non-space character immediately to its right;
- the closing `$` must have a non-space character immediately to its left,
and must not be followed immediately by a digit;
- the closer is the first unescaped `$` after the opener, so a span never
contains another `$`, and a rejected pair cannot skip over a real opener;
- `\$` is a literal dollar; code spans are never math.
Currency stays literal for positional reasons rather than content sniffs:
`$5 to $10` fails because the candidate closer is space-preceded, and
`$5$10` fails because the closer is followed by a digit. Digit-leading
spans also stop leaking raw delimiters — `$0.5$`, `$1/2$`, `$50/50$` and
`$0.399$` now typeset, matching pandoc.
Verified: the scanner reproduces pandoc's own output for every line of a
29-input corpus (currency, code spans, escapes, digit-leading spans,
`$a$b$c$`, `$20,000 and $30,000`, and the reported paragraph) and passes
the existing suite assertions plus new regression tests for the prose-swallow
and cascade cases.
fix(chat): pair inline `$…$` spans with pandoc's tex_math_dollars rules
A rejected `$…$` span emitted its opening `$` as text but left the closing
`$` in the stream, where it was retested as an *opening* delimiter. When
that closer was followed by punctuation it passed the opener tests, paired
with the next `$` whose predecessor was non-space — skipping the real math
span's opener, which prose always space-precedes — and every word in
between was handed to KaTeX. Math mode drops whitespace, so the sentence
became unreadable and a legitimate span on the same line vanished.
The content guards that caused the rejection (`looksLikeLatex`,
`looksLikeDollarMath`: contains LaTeX syntax / does not start with a digit /
shorter than 64 scalars) are replaced by pandoc's `tex_math_dollars`
delimiter rules:
- the opening `$` must have a non-space character immediately to its right;
- the closing `$` must have a non-space character immediately to its left,
and must not be followed immediately by a digit;
- the closer is the first unescaped `$` after the opener, so a span never
contains another `$`, and a rejected pair cannot skip over a real opener;
- `\$` is a literal dollar; code spans are never math.
Currency stays literal for positional reasons rather than content sniffs:
`$5 to $10` fails because the candidate closer is space-preceded, and
`$5$10` fails because the closer is followed by a digit. Digit-leading
spans also stop leaking raw delimiters — `$0.5$`, `$1/2$`, `$50/50$` and
`$0.399$` now typeset, matching pandoc.
Verified: the scanner reproduces pandoc's own output for every line of a
29-input corpus (currency, code spans, escapes, digit-leading spans,
`$a$b$c$`, `$20,000 and $30,000`, and the reported paragraph) and passes
the existing suite assertions plus new regression tests for the prose-swallow
and cascade cases.
tpae
approved these changes
Nov 14, 2026
tpae
left a comment
tpae
left a comment
Contributor
There was a problem hiding this comment.
Thanks for this, and for grounding it in pandoc's tex_math_dollars rules. That's the right call for model output.
I verified it merged onto current main (it merges cleanly):
InlineMathScannerTestsandStreamingMarkdownParserTestspass, 23/23.- The removed helpers (
looksLikeLatex,looksLikeDollarMath) have no other callers.MarkdownBlockParsingkeeps its own privatelooksLikeLatexfor$$blocks. - All of the existing currency and code-span negatives are still pinned.
I also ran a scratch probe of everyday currency phrasing. These all stay literal:
$5 to $10$5, $10, or $20$5-$10$5/$10US$5 or AU$7$1.2M to $3.4M$100 (was $150)- amounts in separate sentences
$2x=20$ and the sentence from #3298 now render as math.
One known trade-off, also present in pandoc, so no change needed: trailing-dollar notation like $5 each (total 20$) now pairs as math. Previously it stayed literal because the content started with a digit.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes #3298
Update the inline latex detection to follow pandoc's inline math detection rules:
$needs a non-space character to its right$needs a non-space character to its left and must not be followed by a digit$after the openerThis changes the heuristic of "dollar sign followed by a digit is never a latex span" to a more sophisticated one, which allows for expressions starting with a digit.
Changes
Test Plan
Run InlineMathScannerTests on a local build.
Screenshots
Before
After
Checklist
CONTRIBUTING.md