Implementation

• Lazy Implementation of tokenizer

• Non-Dictionary-based Correction Module:
  • XML/HTML Handler
  • Informal Expression Handler

  • Leading Digit Splitter
  • Ending Digit Splitter
  • Leading Punctuation Splitter
  • Ending Punctuation Splitter

  

• Dictionary-based Correction Module:
  • Non-word Merge
  • Non-word Split
  • Non-word Spelling (1-To-1)

  • Real-word Merge
  • Real-word Split
  • Real-word Spelling (1-To-1)

  

• Rankning Knowledge Sources:
  • Orthographic Score
    • Token similarity score
    • Phonetic similarity score
    • Overlap (leading/trailing character) similarity score
  • Frequency Score
  • Context Score

  • Noisy Channel - Combined
  • Ensemble - Combined
  • CSpell - Combined

• Other Code Implementation:
  • Add new configurable parameters