SMT: Synonym Files

• Descriptions:

  This page describes how to convert your own synonym file to a SMT standard normalized synonym file and then loaded into SMT corpus tree.

• Standard normalized synonyms file
  • A Java class, SynonymsTable, in STMT package can be used to:
    • read in synonym file(s) in different format
    • store normTerm|synonym in a Hashtable
    • convert to standard format normTerm|synonym

  • normalize key:
    • use SynonymNorm: -f:g:rs:Ct:o:l (also trim spaces)

    Variations	Standardization	Lvg flow	Notes and Examples
    genitive	remove genitive	-f:g	down's|mongoloid bristowe's|split brain cowper's|bulbourethral
    parenthetic plural from	remove (s), (es), (ies)	-f:rs	This is needed to standardize the term (not the synonym)
    spelling variants	get citation forms of all words in the term	-f:Ct
    inflectional variants
    punctuation	replace with space	-f:o	trim multiple spaces % is removed
    Upper, lower, mixed cases	lower case	-f:l
    stopWord	Not implemented	-f:t

  • User's input synonym file to SMT standard normalized synonym file

    Features	Descriptions	User's Input	SMT Standard Normalized
    # for comments	A line starts with # is a comment	# This is a comment
    Duplicates	All duplicated synonym pairs are removed	TB|TUBERCULOSIS TB|TUBERCULOSIS	tb|TUBERCULOSIS tuberculosis|TB
    Redundancy	Remove the synonym pair if the synonym terms are the same	decal|decal
    Norm Redundancy	Remove the synonym pair if the normed synonym terms are the same. This is an optional because no new CUI will be found for such substitution	Fecal|fecal
    Bi-direction	Generate symmetric synonym pairs	AB|ANTIBODY	ab|ANTIBODY antibody|AB
    Multiple synonym pairs	Convert to all possible double pairs	AF|ATRIAL FIBRILLATION|A FIB	af|ATRIAL FIBRILLATION af|A FIB atrial fibrillation|A FIB atrial fibrillation|AF a fib|AF a fib|ATRIAL FIBRILLATION
    Recursive synonyms	Not implemented (manually add them)	abut|adjoin abut|to touch ... anaemia|anemia chlorosis|chlorotic anemia	... adjoin|to touch ... chlorosis|chlorotic anaemia ...
    suffixes	Not implemented (manually remove them)	Knife|-tome

  • Java class to use
    • GenSynonymsFileFromAFile:
      Convert a user's synonym file to the SMT standard normalized synonym file
    • GenSynonymsFileFromFiles
      Add a user's synonym file to the SMT standard normalized synonym file

• Load synonyms file to SMT corpus tree

  Set the SYNONYM_FILE to the standard normalized synonyms file generated from above in the SMT configuration file (${STMT}/data/Config/stm.properties)