Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Correction doesn't prioritize bigram. #107

Open
xcTorres opened this issue Dec 5, 2021 · 3 comments
Open

Correction doesn't prioritize bigram. #107

xcTorres opened this issue Dec 5, 2021 · 3 comments

Comments

@xcTorres
Copy link

xcTorres commented Dec 5, 2021

There are sengkerang and selatan in gram dictionary, the frequency are 500.
And there is tangkerang selatan in bigram dictionary, the frequency is 1200.

When correcting the address "Jalan Surabaya No.17, Tengkerang Selatan, Bukit Raya".
The output is 'jalan surabaya no 17 sengkerang selatan bukit raya',
But I expect Tengkerang Selatan to be corrected as Tangkerang Selatan bigram.
Is it possible to do this?

@mammothb
Copy link
Owner

mammothb commented Dec 6, 2021

Is this similar to #92?

I believe this is because bigrams are only used when a term from the input phrase is split up, e.g., when tengkerang is split to maybe tengke rang. So it doesn't actually look up a correction for the corresponding bigram in the input phrase, i.e., it doesn't find a correction for tengkerang selatan but for its individual words and maybe followed by some split. Relevant lines of code:
Split individual input term
Join suggestions for the splits into a bigram
Compare and see if the bigram exists

The quickest workaround for this particular example would be to have tangkerang in the dictionary with a higher frequency than sengkerang. Perhaps you can use the frequency of the bigram to decide the frequency of tangkerang.

@xcTorres
Copy link
Author

xcTorres commented Dec 6, 2021

Thanks for your explanation. Will have a try.

@ierezell
Copy link

ierezell commented Aug 1, 2022

@mammothb Do you plan to add this feature in symspellpy ? Or do we continue to have this "custom frequency logic"?

As we now understand were it comes from I might do a PR to fix that. For my usecase it's almost vital as I cannot afford false negative and bi-grams helps to solve that.

Thanks again for this nice lib,
Have a great day :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants