Non-word Merge

I. Introduction

This page describes the processes for non-word merge detection and correction.

II. Processes

• Detector:
  NonWordMergeDetector.java
  • non-word: invalid word (not in splitDic)
    • Check both tokenStr and rmEndPuncStr (which remove the ending punctuation, such as ?|..., to avoid merge happen at the end of sentence).
    • Pure abbreviations or acronyms are considered as non-words. They are stored in the Lexicon.noAa.Dic, which is excluded in splitDic
      => so that pure abbreviations or acronyms are considered as non-word.
    • Example: dur ing, where dur matches "DUR|E0446524" for "drug use review", while ing matches "ING|E0439350" for "isotope nephrogram". When we exclude abbreviations and acronyms, dur is a non-word and starts the merge process to during.
  • Not Exceptions:
    • Exceptions include digit, punctuation, digit/punctuation, email, url, empty string, upperCase, 1Char, measurement
    • No merge on exceptions
• Candidate Generator:
  MergeCandidates.java
  • Find the merged candidates by merging the target word and adjacent words within the specified number of spaces in both directions (before and after).
  • MergeNo <= 2 (configurable: CS_CAN_NW_MAX_MERGE_NO)
  • merge with hyphen = true (configurable: CS_CAN_NW_MERGE_WITH_HYPHEN)
    merge with space (" ") and hyphen ("-")
    Example: non prescription is merge to nonprescription and non-prescription
  • exceptions (url, email, ...) are not merge, (target word or adjacent words)
  • orgWords (before merge words) is not a multiword (in mwDic)
  • candidate is in suggDic (valid word), not in aaDic (not Abbreviations or acronyms)
    => Use the merged word as candidate if it is in the dictionary
  • MergeObj is used for the merge operation
    Example: If the input is A big disap point ment comes, disap is detected as an non-word for merge case. A merge size of 2 generates 5 possible candidates:
    • abigdisap
    • bigdisappoint
    • disappointment
    • bigdisap
    • disappoint
    
    Only disappointment and disappoint are in the suggestDic and used as candidates
    • The candidates must be in the suggestion Dic
    • the original term is not a known multiword (such as "non clinical")
    • words in the original term are not known abbreviations or acronyms (so "c d" does not merge to "cd")
• Ranker:
  RankNonWordMergeByContext.java,
  uses the top ranked candidate for correction for the following cases:
  • frequency (better recall)
  • word embedding (better precision)
    • context radius = 2 (configurable, CS_NW_MERGE_CONTEXT_RADIUS)

    • topScore != 0
    
    Please note that as for implementation, use word embedding for ranking merge candidates is a much more complicated than 1-to-1 or split because different merge candidates have different context.
  • CSpell ranking: word embedding, then frequency

  • We observed that there were not too many merge cases in the training set. Most cases have only 1 merge candidate. Thus, no ranking is needed.
  • TBD: noisy channel for merge cases is not implemented and tested. It is because current ranking method is good enough and not too many merge cases (no need to spend more efforts)
• Corrector:
  MergeCorrector.java
  • Reconstrcut the whole text (inTokenList) by going through all mergeObjs
  • Update process history to non-word-merge

III. Development Test

• True-Positive Non-word Merge:

  Id	Source	Original Words	Merged Word
  TP-1	11	neuro transmissions	neurotransmissions
  TP-2	12	tricho rhino phalangeal	trichorhinophalangeal
  TP-3	73	dur ing	during
  TP-4 (RW)	11579	meth amphetamines	methamphetamines
  TP-5 (RW)	13645	non prescription	nonprescription
  TP-6 (RW)	16974	non drug	nondrug

  • Case 4,5,6 are corrected as non-word merge (non) while it was annotated as real-word merge in the gold standard.
• False-Positive Non-word Merge:

  Id	Source	Original Words	Merged Word
  FP-1	42	senior loken	senior-loken
  FP-2	80	pallido ponto nigral	pallidopontonigral
  FP-3	13423	3rd stage	3rd-stage

  • Case 1 is "Senior Loken Syndrome" are not in the Lexicon.
  • Case 2 is "pallido ponto nigral" are not in the Lexicon (multiword dictioanry).
  • Case 3 is fixed by adding order number to the splitDictionary
• False-Negative Non-word Merge:

  Id	Source	Original Words	Merged Word
  FN-1	53	rs 12934922	rs12934922
  FN-2	53	rs 4889294	rs4889294
  FN-3	13082	as nd	and
  FN-4	16247	long gevity	longevity

  • Case 1 & 2 are Genome, which is not in the current scope of CSpell
  • Case 3 & 4 involves more than pure merge operations (spelling + merge)

IV. Spelling Variants and Merge
Spelling variants, that include space, hyphen, and no space, are very tough to handle for merge cases. Please see the following table for the result from our merge algorithm: