Split Candidates

I. Introduction

A token is a single word (term without spaces). If a token is corrected to multiple tokens (a multiword - a term with spaces), this correction is called split correction. Space(s) are inserted to the input token to generate the split candidates. Split correction is used to correct agglutination errors.

II. Algorithm

• Get all possible splits from the spelling error words within 1 edit distance by:
  • Insert a space: abc -> {{a bc}, {ab c}}
• Get all possible split combinations from the spelling error words within N splits by:
  • Recursively apply the same algorithm on the result of above (1 split)
  • Sum all combinations

  • Also replace '-' by space for split. For example, "neck-lesion" is converted to "neck lesion". However, this feature increase recall, decrease precision and F1. So, it is currently disable. Need more study on the data to find a better algorithm for this feature. Maybe check with spelling variants as filter first.
• Check all N splits:
  • Check if the whole split term is multiwords (multiword dictionary):
    • If yes, they are suggested candidates
      => such as "perse" to "per se"
    • If not, check if all split words are valid split words from split word dictionary. Split word dictionary includes English words and proper noun, excludes abbreviations/acronyms with small (specified) length
      => Used dictionary without acronym and abbreviations to reduce noise (such as 'a', 'ab', ...)
      => may add exceptions (such as digit, unit, ...)

III. Split Test Examples

IV. TBD

• split + 1-to-1 correction (TBD)
• Examples:

  Input	Split	1-to-1 Correction	Notes
  menimgtisneef	menimgtis neef	meningitis needs
  anorexia?Thank-you	anorexia? Thank-you	anorexia? Thank you	2.txt
  Shuntfrom2007.How	Shunt from 2007. How	Shut from 2007. How
  polipsremoved	polips removed	polyps removed
  problem,I-am	problem, I-am	problem, I am
  gooddog	good dog	good dog	split
  goooddog	goood dog	good dog	split to none-word correction
  goddog	god dog	good dog	split to real-word correction