Repository navigation
fix: map English inflections to real lemmas instead of non-words - #189
Conversation
Some common English forms pointed at entries that are not words, e.g. playing -> playe, cancelled -> cancell, determining -> determin, envied -> envie. Most are British -ll- spellings and verbs ending in -e or -y. Add 23 reviewed rows to training/overrides/en.tsv and rebuild en.plzma. The rebuilt dictionary differs from the shipped one in exactly these 23 keys. Obsolete headwords such as playe or cancell are left as they are.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #189 +/- ##
=========================================
Coverage 100.00% 100.00%
=========================================
Files 48 48
Lines 1139 1139
=========================================
Hits 1139 1139 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
@adrims Thanks, that's correct and your side note is right too. You could add the latter to the PR so that it's fixed. The 23-case test only rechecks the dictionary data so you could trim it to the minimum. I still see potential for further fixes in this PR, the same pattern is present elsewhere in the English dictionary. The most frequent case seems to be thinking -> thinke (think expected). Look for entries where the lemma is the form's stem plus e, ll or ie, and the bare stem is a headword that maps to itself. |
Review every English entry whose lemma is a self-mapping headword plus e, ll or ie (653 candidate lemmas). Most are real words (hope, stage, install, belie) and stay. 87 forms pointed at non-words or obsolete spellings and now map to the stem, e.g. thinking -> thinke -> think, growing -> growe -> grow, mixed -> mixe -> mix, fantasies -> fantasie -> fantasy, gelled -> gell -> gel. Obsolete headwords and their own obsolete forms (thinkes, cancells) are left as they are. The rebuilt en.plzma differs from the previous one in exactly these 87 keys. Also trim the lemma test to one form per pattern, and create parent folders in download_eval_data so it works in a fresh clone.
|
Thanks! Done:
A few entries are wrong but outside this pattern (bade → bede, belay → belie, reed → ree); I left them for a follow-up. |
|
Thanks! |
|
Thanks a lot for reviewing and merging this! Is there a release planned that would include it? We'd like to pick up the fix from PyPI rather than pinning a commit. No rush at all, and I understand the next version also carries the breaking changes from #188. Just asking so we can plan for it. |
|
I'm going to check first if there are other similar cases in English or other languages. |
Some common English forms point at dictionary entries that are not words:
The British
-ll-spellings were the most visible group of the first batch: "travelled" was already correct, but "travelling" wasn't. All these errors were also present in previous versions.Change:
training/overrides/en.tsv, withen.plzmarebuilt usingdictionary_builder en --in-place.e,llorie(653 candidate lemmas). Most are real words (hope/hop, stage/stag, bathe/bath, install/instal, belie/bely) and stay as they are; each candidate was reviewed by hand.playe,thinke,cancell, …) and their own obsolete forms (thinkes,cancells) stay as they are; only the modern inflected forms are remapped.download_eval_datanow creates parent folders (mkdir(parents=True)), so it works in a fresh clone.Checks:
en.plzmadiffers from the shipped one in exactly these 110 keys.dictionary_builder --check: all 54 dictionaries recompose byte-identically.eval_gate.gate("en", …): PASS on all 13 English treebanks, with no token or type accuracy decrease.test_en_inflection_lemmascovers one form per pattern. pytest, ruff and mypy pass.Not included: a few entries are wrong but don't fit the stem pattern, because the correct lemma isn't the bare stem:
bade → bede(bid),belay → belie(belay),reed → ree(reed). They can go in a follow-up.