LexAccess

Annual DataBase Update Procedures

This page describes annual release procedures for updating database with new set of THE SPECIALIST LEXICON data in lexAccess tools. This is the only procedure needed after release 2013+.

  1. LA PreProc
    • Generate Db table files (UTF-8) 
      shell> cd ${LA_PRE_PROC}/PreDatabase/bin
shell> 1.GenerateDbFiles ${YEAR}
--------------------------------------
Which Program ?
--------------------------------------
1) Copy LEXICON
2) Copy inflVars.data
3) Generate lexicon.data
4) Generate eui.data
----------
5) Copy above 4 files to LexAccessDbData2013+
=> Go to LexAccessDbData2013+ to load files to DB tables
----------
6) Copy Db tables from LexAccessDbData2013+ to laData
7) Back Db tables from laData to Backup/Releases/LA/laData
----------
10) All (default: Step 1- 5)
----------

      Steps 1-5: 4 files are copied and generated:
      OptionFile nameAction
      1LEXICON copy ${LEX_DIR}/data/${YEAR}/tables.frozen/LEXICON to ${LA_PRE_DB_DIR}/data/tables/LEXICON
      2inflVars.data copy ${LEX_DIR}/data/${YEAR}/tables.frozen/inflVars.data to ${LA_PRE_DB_DIR}/data/tables/inflVars.data
      3lexicon.data Generate ${LA_DIR}/data/tables/lexicon.data (from LEXICON)
      shell> ${LA}/PreDatabase/bin/ModifyLexicon -f:d -i:LEXICON -o:lexicon.data
      4eui.data Generate eui.data. The total number of EUI should be equals to total lexical records in LEXICON because this file include deleted records if using lexicon.data from LB.
      5Above 4 filescopy to ${DEV}/LA/lexAccessDbData2013+/data.${YEAR}

    • Check Max. length of fields in Lexicon.data with database tables 
      shell> cd ${LA_PRE_PROC}/PreDatabase/bin
shell> 2.AnalyzeDbFiles ${YEAR}

  2. LoadDb Program (After Step-5)
    • Update db.cfg
      • update ${lexAccessDbData2013+}/loadDb/sources/gov/nih/nlm/nls/lexAccess/loadDb/HSqlDb/db.cfg
        • DB_NAME=lexAccess${YEAR}
        • DB_DATA_DIR=data.${YEAR}/HSqlDb

    • Setup directory 
      shell> cd ${lexAccessDbData2013+}/loadDb/bin
shell> 1.Setup ${YEAR}
      • mkdir ${LA_DB_DATA_DIR}/data.${YEAR}
      • ln -s ${LA_DB_DATA_DIR}/data.${YEAR} ${LA_DB_DATA_DIR}/data => Use this script to create Db data directory. The database name and username are defined in the cfg.data file.

    • Load data into HSqlDb (skip MySql)

      Manully link tables 

      shell> cd ${lexAccessDbData2013+}/data.${YEAR}
shell> ln -sf ./tables.utf8 tables


      shell> cd ${lexAccessDbData2013+}/loadDb/bin
shell> 2.LoadDb
1
3
      • Update DB:
        add "readonly=true" in *.properties after tables are created/loaded

    • Check loaded data in Db 
      shell> cd ${lexAccessDbData2013+}/loadDb/bin
shell> 3.TestDb
1
1

1
2
      => The total number of lexicon does not includes deleted records.

      => Manully move HSqlDb to HSqlDb.utf8 (${lexAccessDbData2013+}/data.${YEAR}/

  3. laData and backup 
    shell> cd ${LA_PRE_PROC}/PreDatabase/bin
shell> 1.GenerateDbFiles ${YEAR}
--------------------------------------
Which Program ?
--------------------------------------
...
----------
6) Copy Db tables from LexAccessDbData2013+ to laData
7) Back Db tables from laData to Backup/Releases/LA/laData
----------
    • laData 
      shell> cd ${lexAccess2013+}/data.${YEAR}
shell> mv HSqlDb to HSqlDb.utf8
shell> 1.GenerateDbFiles ${YEAR}
6
      => Copy -rp ${lexAccess2013+}/data.${YEAR}/HSqlDb.utf8 to ${DEV}/LA/laData/HSqlDb.${YEAR}

    • Backup 
      shell> 1.GenerateDbFiles ${YEAR}
7
      => Copy -rp ${LA_DATA}/HSqlDb.${YEAR} ${BACKUP}Releases/LA/laData/.
      => gtar -czvf HSqlDb.${YEAR}.tgz HSqlDb.${YEAR}

      => Manully mv HSqlDb.${YEAR}* to ${BACKUP}/Releases/LA/laData/HSqlDb.2.3.2/.

  4. Generate pure ASCII Database (for MetaMap Project)
    • generate and copy 4 ASCII files to ${LA_DIR}/data/tables.ascii 
      shell> cd ${LA_PRE_DIR}/bin
shell> 3.GenerateAsciiDbFiles ${YEAR}
1
2
3
4
5
    • LoadDb Program (silimar to Step in UTF-8):
      • Update ${lexAccessDbData2013+}/loadDb/sources/gov/nih/nlm/nls/lexAccess/loadDb/HSqlDb/db.cfg
        • DB_NAME=lexAccess${YEAR}ASCII 
      • shell> cd ${lexAccessDbData2013+}/data.${YEAR}
shell> ln -s ./tables.ascii tables
shell> cd ${lexAccessDbData2013+}/loadDb/bin
shell> 2.LoadDb
	1
	3
      • change "readonly=true" in *.properties after tables are loaded
      • Rename HSqlDb to HSqbDb.ascii in ${LA_DIR}/data/
    • laData (silimar to Step in UTF-8):
      shell> 3.GenerateAsciiDbFiles ${YEAR}
      6
    • Backup (silimar to Step in UTF-8):
      shell> 3.GenerateAsciiDbFiles ${YEAR}
      7

      => Manully mv HSqlDb.${YEAR}ASCII* to ${BACKUP}Releases/LA/laData/HSqlDb.2.3.2/.

  5. Add new Database to ${PROJECT}/lexAccess
    • LexAccess2023 & lagt:
      • ${LA_DIR} is ${PROJECTS}/LA/lexAccess2023
      • Update config file: ${LA_DIR}/data/config/lexAccess.properties
        • Update LATEST_VERSION=${YEAR} for the lastest version
        • Update DB_NAME=lexAccess${YEAR} for default Db table
      • Copy HSqlDb files to ${PROJECTS}/LA/laData/HSqlDb
        => link to /nfsvol/lex/Lu/Development/LA/laData/HSqlDb
        • lexAccess${YEAR}.data,
        • lexAccess${YEAR}.properties
        • lexAccess${YEAR}.script

        • lexAccess${YEAR}ASCII.data,
        • lexAccess${YEAR}ASCII.properties
        • lexAccess${YEAR}ASCII.script

  6. go through unit test

  7. Update WebSite
    • Update ${YEAR}, LexRecord, InflVars, Max. EUI, etc. in HsqlDb Database table on ${LA_WEB}/web/download.html
    • upload downloadable files to LHC-Download site:
      • DB data files: from ${BACKUP}/Releases/LA/laData/HSqlDb.2.3.2/ to https://data.lhncbc.nlm.nih.gov/public/lsg/lexAccess/release/HSqlDb2.3.2/
        • HSqlDb.${YEAR}.tgz
        • HSqlDb.${YEAR}ASCII.tgz
      • Release jat files:from ${Nexus}/lhc-lexicon-raw/www/lexAccess/2023/ to https://data.lhncbc.nlm.nih.gov/public/lsg/lexAccess/release/
        • lexAccess${YEAR}.tgz
        • lexAccess${YEAR}lite.tgz