Words from Medline

This page describes the details of adding words (multiword) from MEDLINE to The SPECIALIST Lexicon:

• The SPECIALIST Lexicon
  • Build The SPECIALIST Lexicon
  • Definition of word count

• Approaches
  • Element Words Approach (2014-)
  • N-gram approaches, 2014+

• Source Corpus
  • MEDLINE N-gram Set
  • Consumer Health Corpus N-gram Set

• Software Components
  • Multiwords

  • Lead-End-Unit Model
  • Acronyms
  • Spelling Variants Model

  • N-gram Utilities
  • Normalization
  • core-term
  • Rule Types for all words from n-grams
  • Lexicon Words

• Filters (Exclusive Filters)
  • Exclusive filter: examples used in the paper of "using element words to generate multiwords with predictive N-gram"
    • Exclusive filter: Rule-based Filter
    • Example: Candidate Multiwords from MEDLINE n-gram

  • Current Enhanced Filters (design & test)
    • Program: 04.TestFilters
    • Add input files to ./inData:
      • NRVARa (from Lexicon release)
      • stopWords.data (from previous year)
      • unit.data (from previous year)
    • update links on these files in ../outData/03.LeadEndTerm:
      • invalidEndTerms.data.abs
      • invalidLeadTerms.data.abs
      • validEndTerms.data.pat
      • validLeadTerms.data.pat
    • shell> 04.TestFilters ${YEAR}
1
10
-----
2
11-42
    • Output files: use ${outdata}/04.TestFilters/TestFilters.rpt to update the results of following links.
  • General Filters:
  • Filter: Pipe
  • Filter: Punctuation or space
  • Filter: Digit
  • Filter: Number word
  • Filter: Digit and Stopword

  • Pattern Filters:
  • Filter: Parenthetic Acronym Pattern (PAP)
  • Filter: Leading character a
  • Filter: UPPERCASE colon
  • Filter: Disallowed punctuation (replaced by disallowed characters after 2020+)
  • Filter: Disallowed character
  • Filter: Measurement pattern
  • Filter: Incomplete pattern

  • Lead-End-Unit Filters:
  • Filter: Absolute invalid lead-unit (AILU)
  • Filter: Absolute invalid end-unit (AIEU)
  • Filter: Invalid lead-end-unit (ILEU)
  • Filter: Valid lead-unit no SpVar (VLU)
  • Filter: Valid end-unit no SpVar (VEU)

  • Filter: Ends with a Parenthetic Abbreviation (not used, replaced by (ACR))

  • Project Domain Filters:
  • Filter: Lexicon
  • Filter: SingleWord
  • Filter: Frenquency - Doc Count and Word Count

• The Distilled MEDLINE n-gram set - Apply filters on the MEDLINE n-gram set
  • 2026
  • 2025
  • 2024
  • 2023
  • 2022
  • 2021
  • 2020
  • 2019
  • 2018
  • 2017
  • 2016
  • 2015
  • 2014