Derivations Procedures - orgD

Retrieved and verify dPairs from original Lexical Tools DM.DB and add them to derivation table. This should be done after nomD, prefixD, suffixD, and zeroD. All orgD with EUIs (in Lexicon EUIs) and valid dPairs and not included in our system are added to our final derivation table. We don't expect too many valid dPair from orgD because only new LexRecords in orgD will be added. However, this procedure requires many manually updates (in Steps: 2,5,6,7,8,81)

I. Directory:

• ${DERIVATION}/0.orgD

II. Input Files (./data/${YEAR}/dataOrg/):
shell> ${ORG_D}/bin/GetOrgD ${YEAR}
0

• Copy following 5 files from ${PREV_YEAR}:
  • convers.fct
  • dm.fct
  • etc.fct
  • nomiz.fct
  • pd.fct

• Copy ${SRC_DIR}/orgD.yes.data.{YEAR} from previous year
  =>ln -sf ${SRC_DIR}/orgD.yes.data.final to orgD.yes.data.{YEAR}

III. Final files for allD (release)

• ${TAR_DIR}/orgD.yes.${YEAR}

IV. Summary of GetOrgD

IV. Processes details:

• shell>cd ${DERIVATION}/0.orgD/bin
• shell>GetOrgD ${YEAR}

  1) Combine Original Lvg Fact dPairs from 5 files
  Combine above five files:
  => Generate ./data/${YEAR}/data/orgD.raw.data

  2) Reformat: remove comments, uSort, empty line
  Reformat: remove comments, uSort, empty lines:
  => Generate ./data/${YEAR}/data/orgD.yes.data
  => Remove 1st (empty) line in ./data/${YEAR}/data/orgD.yes.data

  3) Add EUI to orgD.yes.data.final (use E0000000 for no EUI)
  Add EUI to dPairs in orgD.yes.data, use E0000000 if no EUI found
  => Prepare:

  • cd ./data/${YEAR}/dataOrg/
  • link LEXICON.${YEAR} to /nfsvol/lex/Lu/Development/Lexicon/data/${YEAR}/data/LEXICON.release
  • link orgD.yes.data.final to ../../${PRE_YEAR}/data/orgD.yes.data.${PRE_YEAR}

  => Generate:

  • ./data/${YEAR}/data/orgD.yes.data.final.all
  • ./data/${YEAR}/data/orgD.yes.data.final.noEui
    => Send to linguists to add in new record with E0000000
  • ./data/${YEAR}/data/orgD.yes.data.final.yesEui
    => go to step 4.

  4) Add dType of orgD.yes.data.final.yesEui
  Add dType to orgD.yes.data.final.yesEui to zeroD, suffixD, and prefixD:
  generate:

  • orgD.yes.data.final.yesEui.P
    => go to step 6
  • orgD.yes.data.final.yesEui.S
    => go to step 7
  • orgD.yes.data.final.yesEui.Z
    => go to step 8
  • orgD.yes.data.final.yesEui.PS
    => ignore because PDs by SpVars are excluded
  • orgD.yes.data.final.yesEui.SS
    => ignore because SDs by SpVars are excluded
  • orgD.yes.data.final.yesEui.ZS
    => ignore because ZDs by SpVars are excluded
  • orgD.yes.data.final.yesEui.U
    => Manually review and assign dTag and dType
    • If valid dPairs:
      => manually add to P, S, Z
      => add to ${ALL_D}/data/${YEAR}/dataOrg/dTypeStr.data
    • If invalid dPairs
      => add to ./dataOrg/orgD.tag.txt

  5) Add tag to prefixD: orgD.yes.data.final.yesEui.type.P
  Generates:

  • orgD.yes.data.final.yesEui.type.P.raw
  • orgD.yes.data.final.yesEui.type.P.meta
    • orgD.yes.data.final.yesEui.type.P.yes.data
    • orgD.yes.data.final.yesEui.type.P.no.data
    • orgD.yes.data.final.yesEui.type.P.tbd.data
      => send to linguist, then add all "yes" of it to prefixD
    • orgD.yes.data.final.yesEui.type.P.tbt.data

  6) Add tag to suffixD: orgD.yes.data.final.yesEui.type.S
  Generates:

  • orgD.yes.data.final.yesEui.type.S.raw
  • orgD.yes.data.final.yesEui.type.S.meta
    • orgD.yes.data.final.yesEui.type.S.yesNo.data
    • orgD.yes.data.final.yesEui.type.S.yes.data
    • orgD.yes.data.final.yesEui.type.S.no.data
    • orgD.yes.data.final.yesEui.type.S.tbd.data
      => send to linguist, then add all "yes" of it to suffixD

  7) Add tag to zeroD: orgD.yes.data.final.yesEui.type.Z
  Generates:

  • orgD.yes.data.final.yesEui.type.Z.raw
  • orgD.yes.data.final.yesEui.type.Z.meta
    • orgD.yes.data.final.yesEui.type.Z.yes.data
    • orgD.yes.data.final.yesEui.type.Z.no.data
    • orgD.yes.data.final.yesEui.type.Z.tbd.data
      should be 0 because all zeroZ.raw are generated automatically.
      Some of these might be acronym or abbreviation, which is not legal in zeroD.
      => send to linguist, then add all "yes" of it to zeroD

Ideally, all orgD should be automatically generated by our new derivations generation processes by adding: more prefix (for prefixD) and SD candidate rules (for suffixD). No new zeroD should be found because our system should cover all possible zeroD (please notes that acronyms or abbreviations can't be zeroD). In 2014 release, we manually verify and add orgD into derivational table. Please see the reports on orgD, 2014 for detail.

Please refer to derivation design documents in Lexical Tools for details.