Exclusive Filter: A Term is digit or stopword

• Description:
  If a term contains nothing but digit or stopword, it is not a valid multiword. The default stopwords include: of, and, with, for, nos, to, in, by, on, the, (non mesh).

• Examples:
  • in the
  • the 8:2
  • 1, 2, and
  • 2003 to 2007
  • 50% of

• Input Term: core-term.lc
• Filter Algorithm:
  • Logics:

    Description	FilterType	Notes
    Get words from inTerm	FT_TBD
    Norm: lowercase adn strip punctuation	FT_TBD
    Check words if digit or stopword	FT_DIGIT_STOPWORD	filtered invalid terms - composed of digit and stopwords only

  • source code: FilterPipe.java
  • FilterType: FilterType.FT_DIGIT_STOPWORD

• Accuracy Test on Lexicon:
  • InFile:
    • ${OUT_DATA}/03.LeadEndTerm/lexWords.data
    • ${IN_DATA}/stopWords.data
  • Result:

    Lexicon	Filter	Sample No	Pass No	Trap No	Exp No	Pass-Rate
    2023	FT_DIGIT_STOPWORD	1001867	1001858	9	0	99.9991%
    2022	FT_DIGIT_STOPWORD	998845	998836	9	0	99.9991%
    2021	FT_DIGIT_STOPWORD	992545	992536	9	0	99.9991%
    2020	FT_DIGIT_STOPWORD	983420	983411	9	0	99.9991%
    2019	FT_DIGIT_STOPWORD	972721	972712	9	0	99.9991%
    2018	FT_DIGIT_STOPWORD	955564	955555	9	0	99.9991%
    2017	FT_DIGIT_STOPWORD	935276	935268	8	0	99.9991%
    2016	FT_DIGIT_STOPWORD	915583	915575	8	0	99.9991%
    2015	FT_DIGIT_STOPWORD	896213	896205	8	0	99.9991%
    2014	FT_DIGIT_STOPWORD	875090	875085	6	0	99.9993%