Exclusive Filter: A Term Ends with a Valid End-Unit (VEU) matches Pattern of no SpVar

• Description:
  If a term ends with a valid end-unit (VEU) and has no spelling variants co-exist in the n-gram set, it is not a valid multiword. These terms are filtered out from the MEDLINE n-gram set. The spelling variants pattern includes hypen (built in|built-in), non-space (built in|builtin), case (Built in|built in), and combination of above cases (Built-In|built in).

• Examples:
  • analysis framework for
  • channel blockers should be
  • lotion in the treatment of
  • design was used to
  • lymph nodes up

  The valid-end-units are derived from Lexicon. Some end-units from the invalid lead-end-unit candidate list are valid-end-units and used to checked in the spVar pattern, such as "after", "for", "worth", etc.. N-grams end with any of these pattern valid end-units and does not have spelling variant co-exist in n-gram set are most likely not valid multiwords. In 2014, there are 37 valid end-units found from program. 10 of them are removed and only 27 valid end-units are used for the pattern of no spVar. Terms - "I", "W", "all", "bar", "may", "mine", "minus", "need", "one", and "other" have valid MWE in Lexicon without spVar and thus they are removed as well. Please refer to design documents of End-Unit Types for details.

• Input Term: core-term
• Filter Algorithm:
  • Logics:

    Description	FilterType	Notes
    Get invalid terms	FT_TBD	EndTermPatObj.java HashSet<String> endTermSet_ HashSet<String> natureTermSet_ HashSet<String> orgTermSet_ collect all terms that ends with valid-end-unit Save all terms with same normSpVar (hyphened or non-spaced) lowercase terms if the valid-end-unit is not upper case lowercase terms if terms are not all upper case save to a hashMap(normSpVar, EndTermPatObj) if nature-unit ends with valid-end-unit Convert hashMap(normSpVar, EndTermPatObj) to invalid term list Find EndTermPatObj matches valid-end-unit pattern no spVar (checks EndTermPatObj.natureTermSet_.size() <= 1) not an invalid-end-unit candidates
    Check if an invalid-unit	FT_END_TERM_INV_PAT	Use invalid-unit-list from above step

  • source code: FilterEndTermPat.java
  • FilterType: FilterType.FT_END_TERM_INV_PAT

• Accuracy Test on Lexicon:
  • InFile:
    • ${OUT_DATA}/03.LeadEndTerm/lexWords.data
    • ${OUT_DATA}/03.LeadEndTerm/validEndTerms.data.pat
    • ${OUT_DATA}/03.LeadEndTerm/invalidLeadEndTermCandidates.data
  • Result:

    Lexicon	Filter	Sample No	Pass No	Trap No	Exp No	Pass-Rate
    2023	FT_END_TERM_INV_PATTERN	1001867	1001838	29	0	99.9971%
    2022	FT_END_TERM_INV_PATTERN	998845	998816	29	0	99.9971%
    2021	FT_END_TERM_INV_PATTERN	992545	992516	29	0	99.9971%
    2020	FT_END_TERM_INV_PATTERN	983420	983391	29	0	99.9971%
    2019	FT_END_TERM_INV_PATTERN	972721	972692	29	0	99.9970%
    2018	FT_END_TERM_INV_PATTERN	955564	955535	29	0	99.9970%
    2017	FT_END_TERM_INV_PATTERN	935276	935247	29	0	99.9969%
    2016	FT_END_TERM_INV_PATTERN	915583	915554	29	0	99.9968%
    2015	FT_END_TERM_INV_PATTERN	896213	896190	23	0	99.9974%
    2014	FT_END_TERM_INV_PATTERN	875090	875068	22	0	99.9975%