Synonym Test on Precision and Recall

Synonym tables have been grown since the new enhanced system was implemented in 2017 to generate synonyms. This page details a test model for the new enahcned synonyms.

I. Test Data

The terms in the UMLS CORE (Clinical Observations Recording and Encoding) problem list of SNOMED CT are used. They are controlled terminologies to encode clinical information at a summary level, such as the problem list, discharge diagnosis, or reason for encounter secton of an EHR. They are from 8 large scale healthcare insitutions:

• KA: Kaiser Permanente
• MA: Mayo Clinic
• NU: University of Nebraska Medical Center
• HA: Hong Kong Authority
• IH: Intermountain Healthcare
• RI: Regenstrief Institute
• BI: Beth Israel Deaconess Nedical Center
• VA: Veterans Adminstration

References:

• Fung KW and Xu J. "An exploration of the properties of the CORE problem list subset and how it facilitates the implementationof SNOMED CT, JAMIA 2015; 22:649-658
• K.W. Fung , C. McDonald, S. Srinivasan, “The UMLS-CORE project: a study of the problem list terminologies used in large healthcare institutions", JAMIA, 2013, June, Vol. 17, p.675-680

Data details:

• Total: 98,337 terms are in this data set
  • 88,614 of them are unique terms (some terms are duplicated among healthcare insitutions
• 13,076 are selected from the top 95% usage and mapped to SNOEMED CT
  • They are manually assigned with 13,113 CUIs (37 terms have multiple CUIs).

  • Use the UMLS Metathesaurs.2016AB for CUI mapping (see test model below):
    • 10,321 terms have direct mapped CUI through Norm.
    • 2,755 terms don't have direct mapped CUI through Norm.
      => These 2,755 terms are used as test set for synonym performance test.
      => have 2,756 CUIs (1 term have multiple CUI):
      • Term: [Closed fracture of phalanx or phalanges of hand]:
      • CUI: [C0159791|closed fracture of one or more phalanges of hand]
      • CUI: [C0272698|closed fracture of phalanx of finger]

II. Testing Model

SMT (Synonym Mapping Tools) in STMT package are used to test the performance of synonym.2017. This tool allows users to easily configure different synonym list for subterm sustitution and found the mapped CUI through the expanded terms. The scripts of prgrams are at ${PROJECTS}/STMT/stmt2015/bin/. Several configurations are set up as follows:

Model details:

• top directory: ${DEVELOPMENT}/STMT/Applications/UmlsCore
• program: ${TOP_DIR}/bin
  • shell> 1.UmlsCore.stmt2015 < ./Inputs/in.stmt2015.lexSyn.2016
  • shell> 1.UmlsCore.stmt2015 < ./Inputs/in.stmt2015.lexSyn.2017
  • shell> 1.UmlsCore.stmt2015 < ./Inputs/in.stmt2015.2016AB
  • shell> 1.UmlsCore.stmt2015 < ./Inputs/in.stmt2015.2017

  • shell> 2.GetPRFAmia2017
out.LexSynonym.2016AB
0 (Apply SNOMED CT fitler, only work on subNo != 0, Query Expansion)
    ...
• data: ${TOP_DIR}/data/2017/LexSynonym/outputs.2017Amia

III. Test Results

IV. Discussion

• Compare the 2016AB and 2017:
  • the recall is increased 4.97% (meet the purpose of query expansion)
  • the precision is about the same (slightly increased 0.26%)
    the slight increase on precision is because the precision of pure lexSynonym.2017 is 71.04% (high than the 65.87% in smt.2016AB). In other words, the retrieval from LexSynonym have higher precision.
• Example of TP:
  • [Varicosities of leg]
    => [varicosity of leg] (synonym norm)
    => [varicose vein of leg] (synonym substitution)
    => [varicose vein leg] (norm)
    => [C0155778|varicose veins of leg]
    => [C0155778|Varicose veins of lower extremity] (preferred term)
  • [calcaneal fracture]
    => [heel bone fracture]
    => [calcaneus fracture]
    => [C0281926|Fracture of calcaneus]
  • [cancer of main bronchus]
    => [malignant tumor of main bronchus]
    => [C0153490|Malignant Neoplasm of Main Bronchus]
  • [history of breast cancer]
    => [history of breast malignant neoplasm]
    => [C1997028|History of malignant neoplasm of breast]
  • [cardiac disease screening]
    => [heart disease screening]
    => [C0420042|Heart disease screening]
  • [nasal contusion]
    => [nose contusion]
    => [C0274210|Contusion of nose]
  • [closed nasal fracture]
    => [closed nose fracture]
    => [C0159322|Closed fracture of nasal bones]
• Examples of FP:
  • Legit substitutions leads to FP
    KP30975|CA OF NECK|ca cervical|C4048328|cervical cancer|1
    Gold Std: [Malignant tumor of neck (disorder)|C0746787]
    Query Expansion:
    • [CA of NECK] -> [cancer of cervical]
    • [C4048328|cervical cancer] is usually referred to [uterine cervical cancer]

• 6 LVG flow components are affected by the change of LexSynonym.2017:
  • furiful variants: G, Ge, Gn, v
  • synonyms: y, r