Because of a lapse in government funding, the information on this website may not be up to date, transactions submitted via the website may not be processed, and the agency may not be able to respond to inquiries until appropriations are enacted. The NIH Clinical Center (the research hospital of NIH) is open. For more details about its operating status, please visit cc.nih.gov. Updates regarding government operating status and resumption of normal operations can be found at OPM.gov.

The SPECIALIST Lexicon

Matcher - SpVar Pattern

I. Introduction

From our osverbation, if n-grams of a term that matches spVar pattern in the n-gram set, they are good candidates of MWEs.Spelling variants are usually looks alike and sounds alike as shown in the following example:

EUIExamplesDifference
E0055858
  • sideeffect
  • side effect
  • side-effect
Spaces and hyphens
E001353
  • bloodpressure
  • blood pressure
  • blood-pressure
Spaces and hyphens
E0000237
  • Alzheimers disease
  • Alzheimer's disease
  • Alzheimers' disease
Genitive
E0000862
  • Behcets disease
  • Behcet's disease
  • Behçets disease
  • Behçet's disease
Genitive and unicode
E0000679
  • BM
  • B.M.
Punctuation
E0008797
  • analyse
  • analyze
Spelling
E0017903
  • color
  • colour
Spelling
E0000857
  • Beduin
  • beduin
  • Bedouin
  • bedouin
Cases and spelling
E0223708
  • type 2 diabetes
  • type II diabetes
  • type-2 diabetes
  • type-II diabetes
Spaces, hyphens, and numbers.

A spVar model, including over 10 spelling variant types, has been developed to identify spVars in a corpus. This model was tested on LRSPL (recall), tested on Lexicon (recall and precision), and then apply on MEDLINE.

II. Models
The following algorithm is used to retrieve N-grams that match spVar patterns:

  • AMIA.2016 paper (May, 2016)
    • The distrilled MEDLINE n-gram set, WC > 150

    • Group SpVar ByNorm
    • Group by MES: maxEditDist = 2
    • Group by ES: maxEditDist = 1
    • Group by MES: maxEditDist = 3
    • Group by ES: maxEditDist = 2
    • Group by MES: maxEditDist = 4

    • This model work fine. However, more developments on this model are needed to improve the processing time and performance (precison and recall on Lexicon).
  • HealthInf.2017 paper
    • The distrilled MEDLINE n-gram set, WC > 150

    • Group SpVar ByNorm
    • Group by M2CES: maxEditDist = 2

    • Lower the WC,

III. Processes

  • directory: ${MULTIWORDS_DIR}/bin
  • Run program: shell> ./08.MatcherSpVar ${YEAR}
  • Processes:

    StepDescriptionInputsOutputsNotes - Examples
    Unit Tests for SpVar software compoments
    1Unit Test: SpVar Norm
    SpVarNorm.java     		NonenormStr of sample StrUnit test on spVar norm
    • No need to run for candidate list
    2Unit Test: Metaphone
    Metaphone.java     		Nonemetaphone of sample StrUnit test on metaphone
    • No need to run for candidate list
    3Unit Test: Edit Distance
    EditDistance.java     		Noneedit distance of 2 sample StrsUnit test on edit distance
    • No need to run for candidate list
    4Unit Test: Sorted Distance
    SortedDistance.java     		Nonesorted distance between sample Strs in a set of input termsUnit test on sorted distance
    • No need to run for candidate list
    Utility Tests for SpVar software compoments
    10SpVar on norm|Metaphone|Edit distance ..
    TestSpVarOnNormMpEd.java     		Nonenorm|Metaphone|EditDistanceShow results of two inStrs
    • Not needed for candidate list
    11GroupSpVarByNorm ..
    GetStdAndSpVarsFromLRSPL.java
    GroupSpVarByNorm.java
    • ./inData/LRSPL
    • LRSPL.std
      SpVar-1|SpVar-2|SpVar-3|...
    • LRSPL.data
      SpVar-1
      SpVar-2
      SpVar-3
      ...

    • ./unitTest/LRSPL.data.1.byNorm.out
    		Retrieve LMW from LRSPL by spVarNorm
    • Not needed for candidate list
    12GroupSpVarByMES ..
    GroupSpVarByMES.java
    • ./unitTest/LRSPL.data.1.byNorm.out
    • ./unitTest/LRSPL.data.2.byMES.2.out
    		Retrieve LMW from results of step-11 by MES
    • Not needed for candidate list
    13GroupSpVarByES ..
    GroupSpVarByES.java
    • ./unitTest/LRSPL.data.2.byMES.2.out
    • ./unitTest/LRSPL.data.3.byES.1.out
    		Retrieve LMW from results of step-12 by ES
    • Not needed for candidate list
    14PrintOutSpVars ..
    PrintOutSpVars.java
    • ./unitTest/LRSPL.data.3.byES.1.out
    		Split and print out spVars to single and spVars
    • ./unitTest/LRSPL.data.3.byES.1.out.std (= single + spVars)
    • ./unitTest/LRSPL.data.3.byES.1.out.notSpVars (single without spVar)
    • ./unitTest/LRSPL.data.3.byES.1.out.spVars (have spVars)
    • Not needed for candidate list
    15GetNormSpVarsTable ..
    GetNormSpVarsTable.java
    • ./unitTest/LRSPL.data
    • ./unitTest/LRSPL.data.notSpVar
    • ./unitTest/LRSPL.data.spVar
    		Get a norm spVar|spVars table from a file (terms or n-grams)
    • Not needed for candidate list
    Analysis: Test on LRSPL
    20Convert file format from LRSP ..
    21Get SpVar from LRSPL - ByNorm ..
    22Get SpVar from LRSPL - ByMES (ED:2) ..
    23Get SpVar from LRSPL - ByES (ED:1) ..
    24Get SpVar from LRSPL - ByMES (ED:3) ..
    25Get SpVar from LRSPL - ByES (ED:2) ..
    26Get SpVar from LRSPL - ByMES (ED:4) ..
    27Print result of step 26 - ByMES (ED:4) ..
    28Test SpVar matcher on LRSPL - (Steps: 21-26) ..
    29Analysis: GetSpVarTypeFromLRSPL ..
    GetSpVarTypeFromLRSPL.java
    • ./inData/LRSPL
    • spVars.type
      = GENITIVE + NON_GENITIVE

    • spVars.type.GENITIVE
    • spVars.type.NON_GENITIVE
    • spVars.type.NON_GENITIVE.GENITIVE
    		Analyze types of spVars:
      TypeExamples
      SVT_SPACElookup|look up
      SVT_CASEAcG|ACG
      SVT_PUNC_DASHlookup|look-up
      SVT_PUNC_PERIODAAMD|A.A.M.D.
      SVT_PUNC_OTHERSanti-HB(s)|antiHBs
      SVT_GENITIVEAddisons|Addison's
      SVT_GENITIVE_SAlzheimer|Alzheimer's
      SVT_GENITIVE_PAlzheimer|Alzheimers'
      SVT_GENITIVE_SSAlzheimer|Alzheimer'S
      SVT_GENITIVE_PPAAlzheimer|lzheimerS'
      SVT_NUMBER3|three
      SVT_RANK2nd|second
      SVT_SYNONYMSt.|Saint
      SVT_SPVARantitumour|antitumor
      SVT_UNICODEæcidium|aecidium
      SVT_TBDadvertize|advertise

    • No need to run for candidate list
    Analysis: Test on Lexicon (inflVars.data) for AMIA paper Table-2
    30Get inflectional SpVar from Lexicon
    • GetInflSpVarsFromLexicon.java
    31Get gold standard for SpVars from Lexicon
    • GetGoldStdFromLex.java
    32Get SpVar from Lex - ByNorm
    • GroupSpVarByNorm.java
    33Get SpVar from Lex - ByMES (ED:2)
    • GroupSpVarByMES.java
    • ED = 2
    • Time: hr.
    33A,B,C,D,EGet SpVar from Lex - ByM2ES, M3ES, C2ES, M2CES, M3CES (ED:2)
    • GroupSpVarByXXXX.java
    • ED = 2
    • Time: ~ 2 hr.
    34Get SpVar from Lex - ByES (ED:1)
    • GroupSpVarByES.java
    • ED = 1
    • Time: hr.
    35Get SpVar from Lex - ByMES (ED:3)
    • GroupSpVarByMES.java
    • ED = 3
    • Time: hr.
    36Get SpVar from Lex - ByES (ED:2)
    • GroupSpVarByES.java
    • ED = 2
    • Time: hr.
    37Get SpVar from Lex - ByMES (ED:4)
    • GroupSpVarByMES.java
    • ED = 4
    • Time: hr.
    37Get PRF for above tests (Must complete steps: 31-36)
    Pre-Process
    40Group nGrams by core-term
    GroupByCoreTerm.java
    Same as use otpion 11 of 6.NGramUtil
    • ${NGRAM_DIR}distilledNGram.${YEAR}
    • ./Candidates/distilledNGram.${YEAR}.core
    • ./Candidates/distilledNGram.${YEAR}.detail
    • Group distilled n-gram by core-term
    • Must finish the distilled n-gram set
    41Get terms from nGramSet (filtered by WC, sorted)
    NGramWcTermFilter.java
    • distilledNGram.${YEAR}.core
    • distilledNGram.${YEAR}.core.${WC}.sort
    • distilledNGram.${YEAR}.core.${WC}.sort.term
    		Filter by WC (default 150) and sort
    Process: Apply spVar on MEDLINE
    50Apply SpVar on Medline - ByNorm
    • 1 min (AMIA.init, WC: 150)
    • 2 min (HealthInf, WC: 100)
    • 3 min (HealthInf, WC: 50)
    • 5 min (HealthInf, WC: 30)
    • distilledNGram.${YEAR}.core.WC.sort (manual remove lines less than WC)
    • distilledNGram.${YEAR}.core.WC.sort.term (flds 2 of above file)
    		./${YEAR}/outData/08.MatcherSpVar/Medline/
    • medline.1.byNorm.out.150
    • medline.1.byNorm.out.100
    • medline.1.byNorm.out.50
    • medline.1.byNorm.out.30
    51Apply SpVar on Medline - ByM2CES
    • 2 hr (Amia, WC:150)
    • 23 hr (2015, WC:100)
    • 4 days 18 hr (2015, WC:50)
    • 14 days 8 hr (2015, WC:30)
    • 16 days 23 hr (2016, WC:30)
    		./${YEAR}/outData/08.MatcherSpVar/Medline/
    • medline.2.byM2CES.2.out.150
    • medline.2.byM2CES.2.out.100
    • medline.2.byM2CES.2.out.50
    • medline.2.byM2CES.2.out.30
    51APrint out SpVarClass results of Step-51 to files
    • 1 min (HealthInf)
    		./${YEAR}/outData/08.MatcherSpVar/Medline/
    • medline.2.byM2CES.2.out.30.notSpVars
    • medline.2.byM2CES.2.out.30.spVars
    • medline.2.byM2CES.2.out.30.std
    52Apply SpVar on Medline - ByMES (ED:2)
    • 11 hr (AMIA-lexdev1)
    53Apply SpVar on Medline - ByES (ED:1)
    • 5 days 6 hr (AMIA-lexdev1)
    54Apply SpVar on Medline - ByMES (ED:3)
    • 30 min. (AMIA-lexdev1)
    55Apply SpVar on Medline - ByES (ED:2)
    • 5 days 5 hr (AMIA-lexdev1)
    56Apply SpVar on Medline - ByMES (ED:4)
    • 20 min. (AMIA-lexdev1)
    		Results
    57Apply SpVar on Medline (Step 51 - 56)
    • 11~12 days

    GetSpVarClassByTerm.java
    • distilledNGram.${YEAR}.core.${WC}.sort.term
    		It takes 12 days to generate spVar class (WC >=150), they are LMW candidates:
    • norm
    • MES 1, maxEditDist=2
    • ES 1, maxEditDist=1
    • MES 2: maxEditDist=3
    • ES 2, maxEditDist=2
    • MES 3: maxEditDist=4
    • PrintOut results
    58Print out SpVarClass results to files
    • distilledNGram.${YEAR}.core.${WC}.sort.term.std
    • distilledNGram.${YEAR}.core.${WC}.sort.term.notSpVars
    • distilledNGram.${YEAR}.core.${WC}.sort.term.spVars
    		Results
    Process: Generate LMW candidates from SpVar, Cui, Wc
    60Get LMS candidates from spVar file with CUI
    • GetCanFromSpVarCui.java
    • 90 min.
    • ./Medline/medline.2.byM2CES.2.out.30.spVars
    • ${MED_DIR}/distilledNGram.2015.core.30 (for WC)
    • ${STMT_DIR}/data/Config/smt.properties (for SMT to get CUI)
    • ./Candidates/medline.2.byM2CES.2.out.30.spVars.cui
    • All fields are tokenized into terms.
    • Check if the term has CUI information from SMT
    • Filter out terms without CUI in the spVar class
    • Remove the spVar class if only one term in the class
    • Retrieve WC information
    61Apply WC to LMW candidates from above spVar-CUI file
    • ApplyWcToSpVarCuiCanFile.java
    • ./Candidates/medline.2.byM2CES.2.out.30.spVars.cui
    • WC_BASE
    • UP_LIMIT
    • DOWN_LIMIT
    • ./Candidates/medline.2.byM2CES.2.out.30.spVars.cui.raw
    • ./*.spVars.cui.raw.WC_BASE.UP_LIMIT.DOWN_LIMIT
    61AApply 5 WC sets to LMW candidates
    • ApplyWcToSpVarCuiCanFile.java
    • 15 sec.
    • medline.2.byM2CES.2.out.30.spVars.cui
    • *.raw.100.0.500
    • *.raw.1000.0.500
    • *.raw.10000.0.500
    • *.raw.100000.0.500
    • *.raw.1000000.0.500
    • WC: 1000K, 100K, 10K, 1K, 100
    • UP_LIMIT: 0
    • DOWN_LIMIT: 500
    62Auto tag spVar-CUI-WC file
    • TagSpVarWcCandidateFile.java
    • Use 62A instead
    62AAuto tag 5 WC sets of spVar-CUI files
    • TagSpVarWcCandidateFile.java
    • 15 sec.
    • *.raw.100.0.500
    • *.raw.100.50.50.tag.${YEAR}
    • *.raw.100.50.50.tag.${YEAR}.no
    • *.raw.100.50.50.tag.${YEAR}.tbd
    • *.raw.100.50.50.tag.${YEAR}.yes
    • *.raw.100.50.50.tag.${YEAR}.yesNo
    • *.raw.100.50.50.tag.${YEAR}.can (candidate file)
      => Same as tag file except:
      • remvoe tag if tag is TBD
      • remove the spVar class if all candidates are auto-tagged
    • WC:1000K, 100K, 10K, 1K, 100
    Post-Process: TBD
    70Add word count to SpVar class
    AddWcToSpVarClass.java
    • distilledNGram.${YEAR}.core.${WC}.sort.term.spVars
    • distilledNGram.${YEAR}.core.${WC}.sort
    • distilledNGram.${YEAR}.core.${WC}.sort.term.spVars.wc
    		Add WC back to spVar1|spVar2|..
    71Sort by reversed string ... (request by Lynn)