Skip to content

fix: map English inflections to real lemmas instead of non-words - #189

Merged
adbar merged 2 commits into
adbar:mainfrom
adrims:fix/en-inflection-lemmas
Oct 23, 2026
Merged

adbar merged 2 commits into
adbar:mainfrom
adrims:fix/en-inflection-lemmas

Conversation

@adrims

@adrims adrims commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

Some common English forms point at dictionary entries that are not words:

form before after
playing playe play
thinking thinke think
thanked, thanking thanke thank
developed develope develop
growing growe grow
mixed, mixes, mixing mixe mix
fixed, fixes, fixing fixe fix
singing singe sing
determining determin determine
cancelled, cancelling cancell cancel
travelling travell travel
gelled, gelling gell gel
envied, envies, envying envie envy
fantasies fantasie fantasy

The British -ll- spellings were the most visible group of the first batch: "travelled" was already correct, but "travelling" wasn't. All these errors were also present in previous versions.

Change:

  • 110 reviewed rows in training/overrides/en.tsv, with en.plzma rebuilt using dictionary_builder en --in-place.
    • 23 from the first batch (the table above plus budging, labelled, modelled, signalled, dispelled, exemplified, curries, assailed).
    • 87 from a review of every entry whose lemma is a self-mapping headword plus e, ll or ie (653 candidate lemmas). Most are real words (hope/hop, stage/stag, bathe/bath, install/instal, belie/bely) and stay as they are; each candidate was reviewed by hand.
  • The obsolete headwords (playe, thinke, cancell, …) and their own obsolete forms (thinkes, cancells) stay as they are; only the modern inflected forms are remapped.
  • download_eval_data now creates parent folders (mkdir(parents=True)), so it works in a fresh clone.

Checks:

  • The rebuilt en.plzma differs from the shipped one in exactly these 110 keys.
  • dictionary_builder --check: all 54 dictionaries recompose byte-identically.
  • eval_gate.gate("en", …): PASS on all 13 English treebanks, with no token or type accuracy decrease.
  • On the dev and test splits, accuracy stays the same or rises slightly, for example EWT test 0.9425 → 0.9432, GUM test 0.9460 → 0.9471, PUD test 0.9570 → 0.9582.
  • test_en_inflection_lemmas covers one form per pattern. pytest, ruff and mypy pass.

Not included: a few entries are wrong but don't fit the stem pattern, because the correct lemma isn't the bare stem: bade → bede (bid), belay → belie (belay), reed → ree (reed). They can go in a follow-up.

Some common English forms pointed at entries that are not words, e.g.
playing -> playe, cancelled -> cancell, determining -> determin,
envied -> envie. Most are British -ll- spellings and verbs ending in -e
or -y.

Add 23 reviewed rows to training/overrides/en.tsv and rebuild en.plzma.
The rebuilt dictionary differs from the shipped one in exactly these 23
keys. Obsolete headwords such as playe or cancell are left as they are.
@codecov

codecov Bot commented Oct 21, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 100.00%. Comparing base (4a5d20a) to head (0d69c01).

Additional details and impacted files
@@            Coverage Diff            @@
##              main      #189   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files           48        48           
  Lines         1139      1139           
=========================================
  Hits          1139      1139           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@adbar

adbar commented Oct 22, 2026

Copy link
Copy Markdown
Owner

@adrims Thanks, that's correct and your side note is right too. You could add the latter to the PR so that it's fixed.

The 23-case test only rechecks the dictionary data so you could trim it to the minimum.

I still see potential for further fixes in this PR, the same pattern is present elsewhere in the English dictionary. The most frequent case seems to be thinking -> thinke (think expected). Look for entries where the lemma is the form's stem plus e, ll or ie, and the bare stem is a headword that maps to itself.
If the additional batch needs manual review we could do it in a separate PR.

Review every English entry whose lemma is a self-mapping headword plus
e, ll or ie (653 candidate lemmas). Most are real words (hope, stage,
install, belie) and stay. 87 forms pointed at non-words or obsolete
spellings and now map to the stem, e.g. thinking -> thinke -> think,
growing -> growe -> grow, mixed -> mixe -> mix, fantasies -> fantasie
-> fantasy, gelled -> gell -> gel. Obsolete headwords and their own
obsolete forms (thinkes, cancells) are left as they are.

The rebuilt en.plzma differs from the previous one in exactly these 87
keys.

Also trim the lemma test to one form per pattern, and create parent
folders in download_eval_data so it works in a fresh clone.
@adrims

adrims commented Oct 22, 2026

Copy link
Copy Markdown
Contributor Author

Thanks! Done:

  • mkdir(parents=True) in download_eval_data (confirmed it now works on a fresh clone).
  • Trimmed the test to one form per pattern.
  • Reviewed the wider pattern you described (lemma = stem + e/ll/ie with a self-mapping stem). That gives 653 candidate lemmas; each was reviewed manually, since most are real words (hope/hop, stage/stag, install/instal, belie/bely). 87 forms are remapped, e.g. thinking → think, growing → grow, mixed → mix, singing → sing, fantasies → fantasy, gelled → gel. The gate passes on all 13 English treebanks with no decrease, and dev/test rise slightly (EWT test 0.9427 → 0.9432, GUM 0.9462 → 0.9471).

A few entries are wrong but outside this pattern (bade → bede, belay → belie, reed → ree); I left them for a follow-up.

@adbar

adbar commented Oct 23, 2026

Copy link
Copy Markdown
Owner

Thanks!

@adbar
adbar merged commit da61986 into adbar:main Oct 23, 2026
21 checks passed
@adrims

adrims commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor Author

Thanks a lot for reviewing and merging this!

Is there a release planned that would include it? We'd like to pick up the fix from PyPI rather than pinning a commit.

No rush at all, and I understand the next version also carries the breaking changes from #188. Just asking so we can plan for it.

@adbar

adbar commented Oct 27, 2026

Copy link
Copy Markdown
Owner

I'm going to check first if there are other similar cases in English or other languages.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants