Words from Medline

This page describes the details of adding words (multiword) from MEDLINE to The SPECIALIST Lexicon:

• The SPECIALIST Lexicon
  • Build The SPECIALIST Lexicon
  • Definition of word count

• Approaches
  • Element Words Approach (2014-)
  • N-gram approaches, 2014+

• Source Corpus
  • MEDLINE N-gram Set
  • Consumer Health Corpus N-gram Set

• Software Components
  • Multiwords

  • Lead-End-Unit Model
  • Acronyms
  • Spelling Variants Model

  • N-gram Utilities
  • Normalization
  • core-term
  • Rule Types for all words from n-grams
  • Lexicon Words

• Filters (Exclusive Filters)
  • Exclusive filter: examples used in the paper of "using element words to generate multiwords with predictive N-gram"
    • Exclusive filter: Rule-based Filter
    • Example: Candidate Multiwords from MEDLINE n-gram

  • Current Enhanced Filters (design & test)
    • General Filters:
    • Filter: Pipe
    • Filter: Punctuation or space
    • Filter: Digit
    • Filter: Number word
    • Filter: Digit and Stopword

    • Pattern Filters:
    • Filter: Parenthetic Acronym Pattern (PAP)
    • Filter: Leading character a
    • Filter: UPPERCASE colon
    • Filter: Disallowed punctuation (replaced by disallowed characters after 2020+)
    • Filter: Disallowed character
    • Filter: Measurement pattern
    • Filter: Incomplete pattern

    • Lead-End-Unit Filters:
    • Filter: Absolute invalid lead-unit (AILU)
    • Filter: Absolute invalid end-unit (AIEU)
    • Filter: Invalid lead-end-unit (ILEU)
    • Filter: Valid lead-unit no SpVar (VLU)
    • Filter: Valid end-unit no SpVar (VEU)

    • Filter: Ends with a Parenthetic Abbreviation (not used, replaced by (ACR))

    • Project Domain Filters:
    • Filter: Lexicon
    • Filter: SingleWord
    • Filter: Frenquency - Doc Count and Word Count

  • The Distilled MEDLINE n-gram set - Apply filters on the MEDLINE n-gram set
    • 2023
    • 2022
    • 2021
    • 2020
    • 2019
    • 2018
    • 2017
    • 2016
    • 2015
    • 2014

• Matchers (Inclusive Filters)
  • Matcher: Acronym expansion Pattern (AEP)
    => I. Annual LMW candidate list (acronymExp.tag.data.tag.final.tbd.${YEAR}), must generated after the previous year file is done
    • Precision Analysis for new LMW candidates from (AEP) model
    • PRF Analysis for all LMW candidates from (AEP) model - Paper on (AEP) and Distilled MEDLINE N-Gram set
  • Matcher: Spelling Variant Pattern (SVP)
    => III. Also LMW Candidate list, use spVar (DMNS) + CUI + Frequency + Distrilled nGram (tedious process, done in 07/2016).
    • PRF Analysis for all LMW candidates from SPV model - Paper on SpVar
  • Matcher: Concept Mapping Pattern (CMP) - with CUI(s)
    => II. Annual LMW candidate list (distilled nGrams + + coreTerm.Lc + CUI + MW + 33 endWord list + sort, step 30-34/35)
  • Matcher: End-Word Pattern (EWP)

• Analysis
  • Frequency Analysis for AMIA poster Paper
  • Frequency Analysis on 5 WC ranges: 100, 1K, 10K, 100K, 1M

• Lexicon MultiWord Candidate List

• Previous MultiWord Candidate Lists

• References