Tools to Generate LEXICON

This page include the tools under ${LEXICON}/bin. Some other tools from ${LEXBUILD}/tools/ are used during the LEXICON generation to check Cross-reference, Irreg base, TradeMark, etc. are not included here.

I. FinalizeLexicon

• This tool modifies the LEXICON.freeze to LEXICON.release. It removes annotation, signature & fixes incompliant non-ASCII characters.
• input: LEXICON.freeze
• output: LEXICON.release
• To run:
shell> FinalizeLexicon <year>
• Some manual modification might be needed after this. Such as trim extra empty space, correct typo in non-ASCII Unicode, etc..

II. ValidateLexicon

• This tool validates syntax and Eui of LEXICON.release
• input: LEXICON.release
• To run:
shell> ValidateLexicon <year>
• Check if there error messages on the screen

III. GenerateTables

• This tool automatic generates Lexicon tables
• input: LEXICON.release
• outputs: table files under tables directory
• To run:
shell> GenerateTables <year>
• Check error messages on the screen and modify LEXICON.release accordingly. Then, repeat the procedures.

IV. ReleaseLexicon

• This tool generates and copies Lexicon files to release directory
• To run:
shell> ReleaseLexicon <year>
• Make sure the lvg release is the latest
• Make sure the lvg/data/tables/derivation.data is updated because it is used as ./LEX_DB/DM.DB
• Go through every single file to ensure the file are correct

V. Other Tools

• Location: "${LEXICON}/bin"

  Program Names	Usage	Short Description	Notes
  CompareTables	TableTest <year>	Test all generated tables	Run it at the end to compare data
  GenerateLexiconTerms	GenerateLexiconTerms <year>	Generate all unique terms in LEXICON
  GenerateStatistics	GenerateStatistics <year>	Generate statistics for a given LEXICON	Input: ${LEXICON}/data/${YEAR}/data/LEXICON.release
  GetUniqueEui	GetUniqueEui <field number> <inFile> <outFile>	Generate unique EUI from a specified field in a file