Exclusive Filter: A Term is in Lexicon

• Description:
  If a term is in Lexicon, it is a valid (multi)word. However, they are not not multiword cndidates (because they are already in Lexicon). This filter has several filter levels as described follows. Any of these levels are considered as "true" match.

  Filter Type	Description	Example
  FT_LEX_EM	exact match	skin disease
  FT_LEX_LC	match after lowercase	Skin disease
  FT_LEX_LE_PUNC	mathc after strip leading and ending punctuation	:skin disease
  FT_LEX_LC_LE_PUNC	mathc after lowercase & strip leading and ending punctuation	:SKIN DISEASE
  FT_LEX_ALL_PUNC	mathc after removing all punctuation	:skin-disease,
  FT_LEX_LC_ALL_PUNC	match after lowercase and removing all punctuation	:SKIN-DISEASE,

• Filter Algorithm:
  • Logics:

    Description	FilterType	Notes
    Init	FT_TBD
    Check exact match	FT_LEX_EM	filtered terms in Leixcon
    Check match after lowercase	FT_LEX_LC	filtered terms in Leixcon
    Check match after removing lead-end punctuation	FT_LEX_LE_PUNC	filtered terms in Leixcon
    Check match after lowercase and removing lead-end punctuation	FT_LEX_LC_LE_PUNC	filtered terms in Leixcon
    Check match after removing all punctuation	FT_LEX_ALL_PUNC	filtered terms in Leixcon
    Check match after lowercase and removing all punctuation	FT_LEX_LC_ALL_PUNC	filtered terms in Leixcon

  • source code: FilterLexicon.java
  • FilterType: FilterType.FT_LEX_EM

• Accuracy Test on Lexicon:
  • InFile:
    • ${OUT_DATA}/03.LeadEndTerm/lexWords.data
    • ${IN_DATA}/inflVars.data
  • Result:

    Lexicon	Filter	Sample No	Pass No	Trap No	Exp No	Pass-Rate
    2023	FT_LEX_EM	1001867	1001867	0	0	100.0000%
    2022	FT_LEX_EM	998845	998845	0	0	100.0000%
    2021	FT_LEX_EM	992545	992545	0	0	100.0000%
    2020	FT_LEX_EM	983420	983420	0	0	100.0000%
    2019	FT_LEX_EM	972721	972721	0	0	100.0000%
    2018	FT_LEX_EM	955564	955564	0	0	100.0000%
    2017	FT_LEX_EM	935276	935276	0	0	100.0000%
    2016	FT_LEX_EM	915583	915583	0	0	100.0000%
    2015	FT_LEX_EM	896213	896213	0	0	100.0000%
    2014	FT_LEX_EM	875090	875090	0	0	100.0000%