Dictionary Functions - Check Valid Word

I. Introduction

In cSpell, all tokens that are used for spelling error detection are single words. Thus, only single words are needed to be in the dictionary. This page described which dictionary should be used for the spelling erroor detection.

II. Algorithm

Both the whole token and core-term for the token are checked for the valid spellina (Is-Valid-Word):

• Check the whole token and core-term for dictioary word (non-case snesitive)
• Remove possessive ('s) to check the original word for dictioary word (non-case snesitive)
• Check all words in the or-slash-term for valid word

III. Results

Test cSpell with different dictionaries:

• Is Valid (Spell Checker):
  • Dictionary.IsValidWord
    • Possesses
    • slash or
    • hyphen words (TBD)
  • Exceptions:
    • IsDigit
    • IsPunc
    • IsDigitPunc
    • IsMeasurements (units, digit + units)
    • IsUrl
    • IsEmail
    • IsEmptyString
    • IsProperNoun (dictionary based)
    • IsAbbAcr(dictionary based)
• Candidates:
  • 1-to-1: mainDic
  • Split:
    • if whole split is a multiword
    • if all split words are digit, unit, or noAaDic*

* noAaDic: En + Pn

eng_medical.dic:

• element words from UMLS Strings
• does not inlcude Proper noun
• does not include Abbreviation or acronym
• does not include spelling varaints

Lexicon.dic:

• all: all words
• sw: single words
• mw: multiwords
• ew: element words (unigram)
• aa: abbreviations and acronyms
• pn: proper noun
• sv: spelling variants
• noAa: english words (all exclude aa)
• en: english words (all exclude aa and pn)
• swEn: single word and english words (single word exclude aa and pn)

IV. Tests:

Test-1: Tests on Baseline + Lexicon (not used, result are included from above)

• These tests use only 1 dictionary for check and suggestion
• The pn, aa, sv check are implemented in the algorithm
• The result show element word (ew: for better spelling error detection) with English word (en: for better candidate suggestion) are the best.
• The result leads us to have two dictionaries: check and suggest

Test 2: Tests on split Dictionaries

• Implemented two dictinoary: check and suggestion
• Results shows:
  • check dictionary should be validated element words (Lexicon.ew)
  • suggestion dictinoary should be the one in the testing domain, exclude pn, aa, sv? (more tests on UMLS/MedLine and TBD)

Test-3: test on the suggestion dictionary

• Need a dictionary that focuses on the domain
• The performance is coupled with the candidate and ranking algorithm, need a analysis tool for better understanding.