Exclusive Filter: A Term contains pattern of disallowed punctuation

• Description:
  If a term contains pattern of disallowed punctuation, it is an invalid MWE and. Disallowed punctuation includes {}_!@#*\;"?~=|<>$`^. The count of valid terms in Lexicon.2014 is shown as table below:

  Punctuation	Count in Lexicon.2014
  {	0
  }	0
  _	0
  !	1
  @	0
  #	0
  *	4
  \	0
  ;	0
  "	10
  ?	0
  ~	0
  =	0
  |	0
  <	0
  >	0
  $	0
  `	4
  ^	0

  Among above punctuation, some are allowed to be the leading or ending characters. They should be stripped first:

  • Allowed leading-char-punctuation:

    {

    <

    "

    space

    tab

    *

    #

  • Allowed ending-char-punctuation:

    }

    >

    "

    space

    tab

    ?

    !

    ;

  This filter is replaced by disallowed characters filter after 2020+.

• Examples:
  • (ps> 0.05)
  • group (n=6) received
  • US$

• Input Term: original term
• Filter Algorithm:
  • Logics:

    Description	FilterType	Notes
    Norm: strip allowed-leading-char-punctuation	FT_TBD
    Norm: strip allowed-ending-char-punctuation	FT_TBD
    Check if word contains disallowed punctuation	FT_PUNC_DISALLOW	filtered invalid terms - contain disallowed punctuation :

  • source code: FilterPuncDisallow.java
  • FilterType: FilterType.FT_PUNC_DISALLOW

• Accuracy Test on Lexicon:
  • InFile:
    • ${OUT_DATA}/03.LeadEndTerm/lexWords.data
  • Result:

    Lexicon	Filter	Sample No	Pass No	Trap No	Exp No	Pass-Rate
    2023	FT_PUNC_DISALLOW	1001867	1001851	16	0	99.9984%
    2022	FT_PUNC_DISALLOW	998845	998829	16	0	99.9984%
    2021	FT_PUNC_DISALLOW	992545	992529	16	0	99.9984%
    2020	FT_PUNC_DISALLOW	983420	983406	14	0	99.9986%
    2019	FT_PUNC_DISALLOW	972721	972707	14	0	99.9986%
    2018	FT_PUNC_DISALLOW	955564	955550	14	0	99.9985%
    2017	FT_PUNC_DISALLOW	935276	935263	13	0	99.9986%
    2016	FT_PUNC_DISALLOW	915583	915570	13	0	99.9986%
    2015	FT_PUNC_DISALLOW	896213	896194	19	0	99.9979%
    2014	FT_PUNC_DISALLOW	875090	875071	19	0	99.9978%