Consumer Health Corpus - The Distilled N-gram Set

I. Introduction

The raw n-gram set include many invalid LMWs. These invalid LMWs are filtered out through a series of exclsudive filters. These filters are designed to increase precision while keep the same (similar) recall in terms of LMWs. In oother words, these filter only filter our invalidLMWs without removing valid LMWs. The filtering processes filter out about 2/3 of n-grams and results in the distilled n-gram set. This distilled n-gram set is used for further processes (matchers) to generate LMW candidates.

II. Process

• program:
  ${LMW_DIR}/bin/21.CSpellHealthCorpus
2017

  Option	Description	Inputs - ${IN_DIR}/${OUT_DIR}	Outputs - ${OUT_DIR}	Notes
  Generate the raw n-gram set
  10	Sort nGrams by DC|WC|Term	21.CSpellHealthCorpus/nGrams/nGramSet.${YEAR}.${MIN_WC}	21.CSpellHealthCorpus/nGrams/nGramSet.${YEAR}.${MIN_WC}.dwt Format: DC|WC|n-grm	Sort by DC, WC, alphabetic order of n-gram
  11	Generate the distilled n-gram set	21.CSpellHealthCorpus/nGrams/nGramSet.${YEAR}.${MIN_WC}.dwt	21.CSpellHealthCorpus/nGrams/ distilledNGramSet.data distilledNGramSet.rpt distilledNGramSet.trap distilledNGramSet.exp	Go through exclusive filters process time: ~3 hr.
  12	TBD ...Sort all raw n-gram files	21.CSpellHealthCorpus/nGrams/nGram.${N}.data	21.CSpellHealthCorpus/nGrams/nGram.${N}.data.dwt 21.CSpellHealthCorpus/nGrams/nGram.${N}.data.tdw

II. Filter Details
TBD...

IV. Results