Exclusive Filter: A Term with min. Document Count and Word Count

• Description:
  If a term has DC or WC less than the specified minimum DC or WC, it is not a valid (or commonly-used) multiword. These terms are filtered out from the MEDLINE n-gram set. For examples, the following terms are invalid multiwords:
  • n2oniliforme
  • m2per
  • protocol, recruitment
  • embryos (61.9%)

  The MEDLINE n-gram set is used to retrieve the DC and WC. It uses 30 as the minimum WC and 1 and the min. DC. There are lots of multiwords in Lexcion are not in the n-gram set due to:

  • They are spelling variants - meaning their spVars exist in n-gram set, they don't have enough WC (30).
  • n-gram might in different forms, such as case and punctuation, thus they don't have enough WC (30).
  • Lexicon records some multiwords which has small occurance (WC).
  Thus, we think this is not an right (frequency) filter to used in generating multiwords. Instead, we should use normalized word count (NWC) along with DC|WC for a better result from the frequncy test. Please note that normalized document count (NDC) can't be calculated.

• Filter Algorithm:
  • Logics:

    Description	FilterType	Notes
    get DC|WC from n-gram	FT_TBD
    if not in the MEDLINE n-gram set	FT_WC_DC_NOT_FOUND	Exceptions: valid terms not found DC|WC in n-gram
    Check if (dc < minDc) or (wc < minWc)	FT_WC_DC_INV_LES	filtered invalid terms with DC|WC less than minimum

  • source code: FilterDcWc.java
  • FilterType: FilterType.FT_WC_DC_INV_LESS

• Accuracy Test on Lexicon (DC >= 1; WC >= 30):
  • InFile:
    • ${OUT_DATA}/03.LeadEndTerm/lexWords.data
    • ${OUT_DATA}/02.NGram/nGrams/n-gram.${YEAR}
  • Result:

    Lexicon	Filter	Sample No	Pass No	Trap No	Exp No	Pass-Rate
    2023	FT_WC_DC_INV_LESS	1001867	1001867	0	621582	100.0000%
    2022	FT_WC_DC_INV_LESS	998845	998845	0	623591	100.0000%
    2021	FT_WC_DC_INV_LESS	992545	992545	0	626830	100.0000%
    2020	FT_WC_DC_INV_LESS	983420	983420	0	629088	100.0000%
    2019	FT_WC_DC_INV_LESS	972721	972721	0	630890	100.0000%
    2018	FT_WC_DC_INV_LESS	955564	955564	0	625175	100.0000%
    2017	FT_WC_DC_INV_LESS	935276	935276	0	618346	100.0000%
    2016	FT_WC_DC_INV_LESS	915583	915583	0	618966	100.0000%
    2015	FT_WC_DC_INV_LESS	896213	896213	0	612316	100.0000%
    2014	FT_WC_DC_INV_LESS	875090	875090	0	603592	100.0000%