MEDLINE N-Gram Set

The MEDLINE N-gram Set 2024: by Split, Group, Filter, Combine and Sort Algorithm

The MEDLINE n-gram set (generated by split, group, filter, combine and sort - SGFCS algorithm) is listed as bellows. For each MEDLINE record, title and abstract are used as the source of n-grams. They are combined, tokenized into sentences, and then tokenized into tokens (words use space as word boundary). Finally, n-grams are generated by filtering out terms with more than 50 characters or the total word count is less than 30. The specifications of generating these n-grams are listed as follows:

  • MEDLINE: 2024 - TI and AB (from MEDLINE Baseline Repository - MBR, pubmed24nXXXX.xml -> PmidTiAbS24nXXXX.txt: 1 ~ 1219)
  • Method: Split, Group, Filter, Combine and Sort Algorithm
  • Max. Character Size: 50
  • Min. word count: 30
  • Min. document count: 1

  • Total document count: 36,555,430
  • Total sentence count: 253,923,392
  • Total token count: 5,326,576,788

  • N-gram files
    • File format - 3 fields:
      Document countWord countN-gram
    • Sorted by document count, word count, then alphabetic order of n-grams. N-gram set is not sorted. It can be sorted by nGramUtil package.

  • Download:
    N-gramsFileZip SizeActual SizeNo. of n-grams
    Unigrams1-gram.2024.tgz9.2 MB23 MB1,374,878
    Bigrams2-gram.2024.tgz61 MB179 MB8,369,463
    Trigrams3-gram.2024.tgz97 MB323 MB12,511,710
    Four-grams4-gram.2024.tgz72 MB251 MB8,226,169
    Five-grams5-gram.2024.tgz37 MB131 MB3,678,688
    N-gram SetnGramSet.2024.30.tgz275 MB905 MB34,160,908
    Distilled N-gram SetdistilledNGram.2024.tgz112 MB366 MB13,775,979