Distilled MEDLINE N-Gram Set

I. Introduction

The MEDLINE n-gram set includes many invalid LMWs that are not needed for most NLP research. LSG developed a set of exclusive filters that filter out these invalid LMWs. The filtering process filtered out about 2/3 of n-grams from MEDLINE n-gram set release. This enhanced/filtered N-Gram set is called the distilled MEDLINE n-gram set.

II. Precision and Recall

This distilled MEDLINE n-gram set has higher precision and same (similar) recall rate in terms of valid multiwords. LSG performs the accuracy test on all developed exclusive filters by applying these filters on Lexicon (valid LMW). The minimum passing rate is 99.99%. In other words, these filters only filter out invalid LMWs without removing valid LMWs. A simple calculation is described as below:

• The n-gram set include valid N LMW (TP0) and M invalid LMW (FP0)
• A serial filters filter out X valid LMW (TP1) and Y invalid LMW (FP1)
• The distilled N-Gram set have (N-X) valid LMW (TP2) and (M-Y) invalid LMW (FP1)
• If the accuracy test is very high (99.99%), then
  • X is a very small number (almost 0)
  • Y is a large number (almost 2/3 of N+M)

• The precision
  = (retrieved and relevant)/(total retrieved)
  = TP2/(TP2+FP2)
  = (N-X)/(N-X + M-Y)
  = N/(N+M-Y) (> N/N+M)

• The recall
  = (retrieved and relevant)/(total relevant)
  = TP2/(TP2 + FN0)
  = (N-X)/(N-X + FN0)
  = N/(N + FN0) (= TP0/ (TP0 + FN0))

III. Conclusion

The distilled MEDLINE n-gram Set vs. MEDLINE n-gram Set

• All exclusive filters have accuracy rate above 99.99% (tested on Lexicon)
• smaller data set (about 1/3)
• better precision
• similar recall
• cab be used as baseline for further analysis

IV. Release Processes

• Dir: ${MULTIWORD_DIR}
• Script: manually add n-gram number for ${YEAR} to ${MULTIWORD_DIR}/bin/05.ApplyFilters
  shell>cd ${MULTIWORDS}/data/${YEAR}/outData/02.NGram/nGrams
shell>wc -l nGramSet.${YEAR}.30

  Year	nGram Number
  2014	17,023,819
  2015	18,148,692
  2016	19,325,338
  2017	21,963,037
  2018	23,171,133
  2019	24,666,816
  2020	26,310,808
  2021	28,103,252
  2022	30,090,771
  2023	32,107,061
  2024	34,160,908
  2025	36,370,468

  
  The nGram number is a must to get the correct pass-rate (percentage). Update this number with associated year in the 05.ApplyFilters script before run the distilled n-gram set and update the log file.
• Input Data:
  Need to setup all the following files before runnning the program (05.ApplyFilters)
  In practice, we test all filters with the latest Lexicon (04.TestFilters) by setting up the following files before we proceed to 05.ApplyFilters.
  For the Lead-End-Unit: they should run Step 1-4 on the program of 03.LeadEndUnit ${YEAR} to update th data. However, use the previous data is OK.
  • n-gram.${YEAR} (Step 1):
    shell>mkdir ${MULTIWORD_DIR}/data/${YEAR}/outData/05.ApplyFilters
shell>cd ${MULTIWORD_DIR}/data/${YEAR}/outData/05.ApplyFilters
shell>ln -sf ../02.NGram/nGrams/nGramSet.${YEAR}.30 nGram.${YEAR}
  • NRVAR (Step 13):
    shell>cd ${MULTIWORD_DIR}/data/${YEAR}/inData
shell>ln -sf nfsvol/lex/Lu/Backup/Releases/UMLS/${YEAR}_AA_release/LEX/NUMBERS/NRVAR NRVAR
  • stopWords.data (Step 14):
    shell>cd ${MULTIWORD_DIR}/data/${YEAR}/inData
shell>cp -p ../../${PREV_YEAR}/inData/stopWords.data.${PREV_YEAR} stopWords.data.${YEAR}
shell>ln -sf ./stopWords.data.${YEAR} stopWords.data
  • unit.data (Step 24):
    shell>cp -p ../../${PREV_YEAR}/inData/unit.data.${PREV_YEAR} unit.data.${YEAR}
shell>ln -sf ./unit.data.${YEAR} unit.data
  • invalidLeadTerms.data.abs (Step 30):
    shell>cd ${MULTIWORD_DIR}/data/${YEAR}/outData/03.LeadEndTerm
shell>cat invalidLeadTerms.data invalidLeadTerms.data.append > invalidLeadTerms.data.${YEAR}
    or
    shell>cp -p ../../../${PREV_YEAR}/outData/03.LeadEndTerm/invalidLeadTerms.data.${PREV_YEAR} invalidLeadTerms.data.${YEAR}
shell>ln -sf ./invalidLeadTerms.data.${YEAR} invalidLeadTerms.data.abs
  • invalidEndTerms.data.abs (Step 31):
    shell>cd ${MULTIWORD_DIR}/data/${YEAR}/outData/03.LeadEndTerm
shell>mv invalidEndTerms.data invalidEndTerms.data.${YEAR}
    or
    shell>cp -p ../../../${PREV_YEAR}/outData/03.LeadEndTerm/invalidEndTerms.data.${PREV_YEAR} invalidEndTerms.data.${YEAR}
shell>ln -sf ./invalidEndTerms.data.${YEAR} invalidEndTerms.data.abs
  • invalidLeadEndTermCandidates.data (Step 32):
    shell>cd ${MULTIWORD_DIR}/data/${YEAR}/outData/03.LeadEndTerm
shell>cp -p ../../../${PREV_YEAR}/outData/03.LeadEndTerm/invalidLeadEndTermCandidates.data .
    This file could be the same if you run the 03.LeadEndTerm
  • validLeadTerms.data.pat (Step 33):
    shell>cd ${MULTIWORD_DIR}/data/${YEAR}/outData/03.LeadEndTerm
shell>cp -p ../../../${PREV_YEAR}/outData/03.LeadEndTerm/validLeadTerms.data.pat.${PREV_YEAR} validLeadTerms.data.pat.${YEAR}
shell>ln -sf ./validLeadTerms.data.pat.${YEAR} validLeadTerms.data.pat
  • validEndTerms.data.pat (Step 34):
    shell>cd ${MULTIWORD_DIR}/data/${YEAR}/outData/03.LeadEndTerm
shell>cp -p ../../../${PREV_YEAR}/outData/03.LeadEndTerm/validEndTerms.data.pat.${PREV_YEAR} validEndTerms.data.pat.${YEAR}
shell>ln -sf ./validEndTerms.data.pat.${YEAR} validEndTerms.data.pat
• Run Program:
  • shell>cd ${MULTIWORDS}/bin/05.ApplyFilters ${YEAR}
1
10-14
20-25
30-34
40

    or

  • shell>cd 05.ApplyFiltersAll
  • shell>runApplyFilersAll ${YEAR}
• Output Data:
  • Dir: /${MULTIWORD}/data/${YEAR}/outData/05.ApplyFilters
  • ApplyFilters.rpt (use this file to update release log file below)
    => shell> cp -p ApplyFilters.rpt ApplyFilters.rpt.${YEAR}
  • nGram.${YEAR}.${STEP}.${NAME}
  • nGram.${YEAR}.${STEP}.${NAME}.exp
  • nGram.${YEAR}.${STEP}.${NAME}.trap

  • Use nGram.${YEAR}.34.invEndTermPat (the last fitlered one) for the distilled n-gram set
  • Distribute it
    =>shell> cp -p nGram.2018.34.invEndTermPat ../02.NGram/nGrams/distilledNGram.${YEAR}
  • Backup it
    =>shell> gtar -czvf distilledNGram.${YEAR}.tgz distilledNGram.${YEAR}
  • Get Core files
    =>shell> cd ${MULTIWORDS}/bin
    =>shell> 06.NGramUtil ${YEAR}
20 for the MEDLINE nGram set
    21 for the distilled nGram set
    22 for nGrams (N = 3 ~ 5)
    3
4
5

V. Release Logs

• 2025
• 2024
• 2023
• 2022
• 2021
• 2020
• 2019
• 2018
• 2017
• 2016
• 2015
• 2014

VI. Run the Test data on the Lexicon

• Must run 03.LeadEndTerm ${YEAR} for lexWords.data
• run 04.TestFilters ${YEAR} to update test result on each filter