Exclusive Filter: A Term is punctuation or space

• Description:
  If a term contains nothing but punctuation or space, it is not a valid multiword.

• Examples:
  • "-"
  • $
  • +/-
  • %
  • (+)
  • [...]

• Input Term: core-term.lc
• Filter Algorithm:
  • Logics:

    Description	FilterType	Notes
    Norm: Strip punctuation	FT_TBD
    Norm: Trim	FT_TBD
    Check if an empty string	FT_PUNCTUATION	filtered invalid terms - punctuation and space

  • source code: FilterPunctuation.java
  • FilterType: FilterType.FT_PUNC_SPACE

• Accuracy Test on Lexicon:
  • InFile:
    • ${OUT_DATA}/03.LeadEndTerm/lexWords.data
  • Result:

    Lexicon	Filter	Sample No	Pass No	Trap No	Exp No	Pass-Rate
    2023	FT_PUNC_SPACE	1001867	1001867	0	0	100.0000%
    2022	FT_PUNC_SPACE	998845	998845	0	0	100.0000%
    2021	FT_PUNC_SPACE	992545	992545	0	0	100.0000%
    2020	FT_PUNC_SPACE	983420	983420	0	0	100.0000%
    2019	FT_PUNC_SPACE	972721	972721	0	0	100.0000%
    2018	FT_PUNC_SPACE	955564	955564	0	0	100.0000%
    2017	FT_PUNC_SPACE	935276	935276	0	0	100.0000%
    2016	FT_PUNC_SPACE	915583	915583	0	0	100.0000%
    2015	FT_PUNC_SPACE	896213	896213	0	0	100.0000%
    2014	FT_PUNC_SPACE	875090	875090	0	0	100.0000%