Exclusive Filter: A Term contains pattern of disallowed characters

• Description:
  If a term contains pattern of disallowed characters, it is an invalid MWE. Disallowed characters includes:

  Ascii disallowed chars	_@\~=|$`^{<#*}!;?>"	Derived from the Lexicon, see details at disallowed punctuation
  Unicode disallowed chars	=⁺⁻×÷⁼⅀∀∑−≅≠≤≥≦≧±¼½¾⅓⅔⅕⅖⅗⅘⅙⅚⅛⅜⅝⅞°	Derived from CSpell Consumer Health Corpus

• Examples:
  • (ps> 0.05)
  • group (n=6) received
  • US$

  • cooked, ½ cup
  • 101°f (38.3°c)
  • 179 ± 16

• Input Term: original term
• Filter Algorithm:
  • Logics:

    Description	FilterType	Notes
    Check if word contains disallowed characters	FT_CHAR_DISALLOW	filtered invalid terms - contain disallowed characters :

  • source code: FilterPuncDisallow.java
  • FilterType: FilterType.FT_CHAR_DISALLOW

• Accuracy Test on Lexicon:
  • InFile:
    • ${OUT_DATA}/03.LeadEndTerm/lexWords.data
  • Result:

    Lexicon	Filter	Sample No	Pass No	Trap No	Exp No	Pass-Rate
    2023	FT_CHAR_DISALLOW	1001867	1001851	16	0	99.9984%
    2022	FT_CHAR_DISALLOW	998845	998829	16	0	99.9984%
    2021	FT_CHAR_DISALLOW	992545	992529	16	0	99.9984%
    2020	FT_CHAR_DISALLOW	983420	983406	14	0	99.9986%
    2019	FT_CHAR_DISALLOW	972721	972707	14	0	99.9986%
    2018	FT_CHAR_DISALLOW	955564	955550	14	0	99.9985%
    2017	FT_CHAR_DISALLOW	935276	935263	13	0	99.9986%
    2016	FT_CHAR_DISALLOW	915583	915570	13	0	99.9986%
    2015	FT_CHAR_DISALLOW	896213	896194	19	0	99.9979%
    2014	FT_CHAR_DISALLOW	875090	875071	19	0	99.9978%