Lexical Tools

Optimizing 2024 SD-Rule Set - Optimum Log

I. Criteria:

  • Total valid SD-Pairs from baseline (parent only rules - baseline) is 59,911
  • Candidate child rules are:
    • Decompose occurrence rate >= 40% (default)
    • Candidate child rules: occurrence rate >= 25% (default)
    • Candidate child rules: precision is decided by the methodology of optimization
    • Find the one with max. precision and recall
    • If the child rule has lower precision and recall than parents in the decomposed file, it is not a good candidate even its recall is over 25%!
      • Such as in Cases 15 and 16, the system performance is worse because both precision and recall are lower than parents in the decompose file. No need to run the program for these two cases. Ideally, the CHILD rules should be more precise and less coverage.
      • If the performance of a child rule is worse, then the next generation child rules will be worse. No need to run through the following generations (we iused to run through them to keep the log completed and confirmed this observation)!
  • Find the best set by comparing PARENT vs. CHILD rules (in different generations):
    • Only apply when child rules precision is better than parent rule
    • Higher system performance (F1)
    • If System performance (F1) is the same, use the priority as follows:
      • Precision
      • Recall
      • Linguistic knowledge

      • parent rule to replace child rules.
      • if no parent-child rules involved, use more rules

II. Iterative Optimization Log:

Source:

  • Dir: ${SUFFIX_DIR}/data/${YEAR}/dataR/SdRulesOptimum/*/
  • File: sdRules.stats.out.html

IDRank: Parent-RuleRank: Candidate Child-RulesCutoff SD-Rules
Rank|Accu. Rate|Occr.|Yes|No|TBD|SD-Rule|Precision|Recall|F1|Accu. Yes|Accu. Occu 		Notes
0 Rank in Baseline (all Rank)
Parent-rule only - Baseline 		Rank
No child-Rule 		102|73.13%|67|49|18|0|$|verb|per$|noun|2024|WORDNET|SELF|95.26%|87.24%|1.8251|52268|54867 baseline
1.1 0|98.99%|2086|2065|21|0|$|adj|ally$|adv|100.00%|99.49 1|99.95%|1966|1965|1|0|c$|adj|cally$|adv|94.25%|97.01 102|73.13%|67|49|18|0|$|verb|per$|noun|2024|WORDNET|SELF|95.29%|87.08%|1.8237|52168|54747 Worse
1.2 2|99.95%|1961|1960|1|0|ic$|adj|ically$|adv|94.01%|96.89 102|73.13%|67|49|18|0|$|verb|per$|noun|2024|WORDNET|SELF|95.29%|87.07%|1.8236|52163|54742 Worse
2.1 0|97.85%|2095|2050|45|0|$|adj|ity$|noun|100.00%|98.91 1|99.26%|950|943|7|0|c$|adj|city$|noun|45.35%|62.25 1|97.69%|736|719|17|0|l$|adj|lity$|noun|35.13%|51.68 F1 of CHILD rules is more than 5% lower than PARENT, past result is worse
=> No need to run evaluation program 		Worse
2.2 2|99.37%|949|943|6|0|ic$|adj|icity$|noun|45.30%|62.23 1|97.69%|736|719|17|0|l$|adj|lity$|noun|35.13%|51.68 F1 of CHILD rules is more than 5% lower than PARENT, past result is worse
=> No need to run evaluation program 		Worse
3.1 0|72.89%|1376|1003|373|0|$|noun|al$|adj|100.00%|84.32 1|82.60%|684|565|119|0|n$|noun|nal$|adj|49.71%|62.07 F1 of CHILD rules is more than 5% lower than PARENT, past result is worse => No need to run evaluation program Worse
3.2 2|85.74%|631|541|90|0|on$|noun|onal$|adj|45.86%|59.75 F1 of CHILD rules is more than 5% lower than PARENT, past result is worse => No need to run evaluation program Worse
3.3 3|86.30%|584|504|80|0|ion$|noun|ional$|adj|42.44%|56.90 F1 of CHILD rules is more than 5% lower than PARENT, past result is worse => No need to run evaluation program Worse
3.4 4|86.40%|478|413|65|0|tion$|noun|tional$|adj|34.74%|49.55 F1 of CHILD rules is more than 5% lower than PARENT, past result is worse => No need to run evaluation program Worse
4.1 0|51.22%|697|357|340|0|$|noun|y$|noun|100.00%|67.74 1|91.05%|257|234|23|0|h$|noun|hy$|noun|36.87%|52.49 F1 of CHILD rules is more than 5% lower than PARENT, past result is worse
=> No need to run evaluation program 		Worse
5.1 0|93.29%|581|542|39|0|$|verb|ion$|noun|100.00%|96.53 1|96.48%|454|438|16|0|t$|verb|tion$|noun|78.14%|86.35 F1 of CHILD rules is more than 5% lower than PARENT, past result is worse
=> No need to run evaluation program 		Worse
5.2 2|99.38%|324|322|2|0|ct$|verb|ction$|noun|55.77%|71.44 F1 of CHILD rules is more than 5% lower than PARENT, past result is worse
=> No need to run evaluation program 		Worse
5.3 3|100.00%|186|186|0|0|ect$|verb|ection$|noun|32.01%|48.50 F1 of CHILD rules is more than 5% lower than PARENT, past result is worse
=> No need to run evaluation program 		Worse
6.1 0|86.54%|312|270|42|0|a$|noun|an$|adj|100.00%|92.78 No candidate CHILD rule found! No candidate child rules found => No need to run evaluation program Same
7.1 0|10.26%|312|32|280|0|a$|noun|an$|noun|100.00%|18.60 1|11.69%|154|18|136|0|ia$|noun|ian$|noun|49.36%|18.90 102|73.13%|67|49|18|0|$|verb|per$|noun|2024|WORDNET|SELF|95.26%|87.24%|1.8251|52268|54867 Same
8.1 0|87.14%|140|122|18|0|a$|noun|ar$|adj|100.00%|93.13 1|91.45%|117|107|10|0|la$|noun|lar$|adj|83.57%|87.33 F1 of CHILD rules is more than 5% lower than PARENT, past result is worse
=> No need to run evaluation program 		Worse
8.2 2|94.37%|71|67|4|0|ula$|noun|ular$|adj|50.71%|65.97 F1 of CHILD rules is more than 5% lower than PARENT, past result is worse
=> No need to run evaluation program 		Worse
9.1 0|99.18%|2560|2539|21|0|ation$|noun|e$|verb|100.00%|99.59 1|99.91%|1080|1079|1|0|sation$|noun|se$|verb|42.19%|59.32 1|100.00%|1261|1261|0|0|zation$|noun|ze$|verb|49.26%|66.00 F1 of CHILD rules is more than 5% lower than PARENT, past result is worse
=> No need to run evaluation program 		Worse
9.2 2|100.00%|1056|1056|0|0|isation$|noun|ise$|verb|41.25%|58.41 2|100.00%|1254|1254|0|0|ization$|noun|ize$|verb|48.98%|65.76 F1 of CHILD rules is more than 5% lower than PARENT, past result is worse
=> No need to run evaluation program 		Worse
9.3 2|100.00%|1056|1056|0|0|isation$|noun|ise$|verb|41.25%|58.41 1|100.00%|1261|1261|0|0|zation$|noun|ze$|verb|49.26%|66.00 F1 of CHILD rules is more than 5% lower than PARENT, past result is worse
=> No need to run evaluation program 		Worse
10.1 0|87.71%|301|264|37|0|c$|adj|s$|noun|100.00%|93.45 1|91.29%|287|262|25|0|ic$|adj|is$|noun|95.35%|93.27 102|73.13%|67|49|18|0|$|verb|per$|noun|2024|WORDNET|SELF|95.28%|87.24%|1.8252|52266|54853 Better
10.2 2|94.82%|193|183|10|0|tic$|adj|tis$|noun|64.12%|76.50 102|73.13%|67|49|18|0|$|verb|per$|noun|2024|WORDNET|SELF|95.30%|87.11%|1.8241|52187|54759 Worse
10.3 3|97.71%|175|171|4|0|itic$|adj|itis$|noun|58.14%|72.90 102|73.13%|67|49|18|0|$|verb|per$|noun|2024|WORDNET|SELF|95.31%|87.09%|1.8240|52175|54741 Worse
11.1 0|22.39%|527|118|409|0|c$|adj|sm$|noun|100.00%|36.59 1|22.43%|526|118|408|0|ic$|adj|ism$|noun|99.81%|36.63 102|73.13%|67|49|18|0|$|verb|per$|noun|2024|WORDNET|SELF|95.28%|87.24%|1.8252|52266|54853 Same
12.1 0|96.60%|911|880|31|0|ce$|noun|t$|adj|100.00%|98.27 1|97.89%|899|880|19|0|nce$|noun|nt$|adj|98.68%|98.28 102|73.13%|67|49|18|0|$|verb|per$|noun|2024|WORDNET|SELF|95.30%|87.24%|1.8254|52266|54841 Better
12.2 2|98.27%|346|340|6|0|ance$|noun|ant$|adj|37.98%|54.79 2|97.65%|553|540|13|0|ence$|noun|ent$|adj|60.70%|74.87 103|73.13%|67|49|18|0|$|verb|per$|noun|2024|WORDNET|SELF|95.30%|87.24%|1.8254|52266|54841 Same
13.1 0|98.61%|431|425|6|0|cy$|noun|t$|adj|100.00%|99.30 1|98.60%|430|424|6|0|ncy$|noun|nt$|adj|99.77%|99.18 102|73.13%|67|49|18|0|$|verb|per$|noun|2024|WORDNET|SELF|95.30%|87.24%|1.8254|52265|54840 Same
14.1 0|98.77%|2365|2336|29|0|e$|verb|ion$|noun|100.00%|99.38 1|99.64%|2227|2219|8|0|te$|verb|tion$|noun|94.16%|96.83 102|73.13%|67|49|18|0|$|verb|per$|noun|2024|WORDNET|SELF|95.33%|87.04%|1.8238|52149|54703 Worse
14.2 2|99.81%|2123|2119|4|0|ate$|verb|ation$|noun|89.77%|94.52 102|73.13%|67|49|18|0|$|verb|per$|noun|2024|WORDNET|SELF|95.33%|86.88%|1.8221|52049|54599 Worse
15.1 0|93.24%|148|138|10|0|e$|verb|is$|noun|100.00%|96.50 1|96.50%|143|138|5|0|se$|verb|sis$|noun|96.62%|96.56 102|73.13%|67|49|18|0|$|verb|per$|noun|2024|WORDNET|SELF|95.31%|87.24%|1.8255|52266|54836 Better
15.2 2|96.30%|54|52|2|0|ose$|verb|osis$|noun|36.49%|52.92 2|100.00%|59|59|0|0|yse$|verb|ysis$|noun|39.86%|57.00 103|73.13%|67|49|18|0|$|verb|per$|noun|2024|WORDNET|SELF|95.32%|87.19%|1.8251|52239|54806 Worse
16.1 0|92.41%|224|207|17|0|esis$|noun|ic$|adj|100.00%|96.06 1|98.56%|209|206|3|0|nesis$|noun|nic$|adj|93.30%|95.86 102|73.13%|67|49|18|0|$|verb|per$|noun|2024|WORDNET|SELF|95.34%|87.24%|1.8258|52265|54821 Better
16.2 2|99.52%|207|206|1|0|enesis$|noun|enic$|adj|92.41%|95.83 102|73.13%|67|49|18|0|$|verb|per$|noun|2024|WORDNET|SELF|95.34%|87.24%|1.8258|52265|54819 Same
16.3 3|99.52%|207|206|1|0|genesis$|noun|genic$|adj|92.41%|95.83 102|73.13%|67|49|18|0|$|verb|per$|noun|2024|WORDNET|SELF|95.34%|87.24%|1.8258|52265|54819 Same
16.4 4|100.00%|181|181|0|0|ogenesis$|noun|ogenic$|adj|80.80%|89.38 F1 of CHILD rules is more than 5% lower than PARENT, past result is worse
=> No need to run evaluation program 		Worse
17.1 0|99.88%|1646|1644|2|0|ility$|noun|le$|adj|100.00%|99.94 1|99.94%|1644|1643|1|0|bility$|noun|ble$|adj|99.88%|99.91 102|73.13%|67|49|18|0|$|verb|per$|noun|2024|WORDNET|SELF|95.34%|87.24%|1.8258|52264|54819 Same
17.2 2|99.92%|1304|1303|1|0|ability$|noun|able$|adj|79.22%|88.38 2|100.00%|317|317|0|0|ibility$|noun|ible$|adj|19.26%|32.30 F1 of CHILD rules is more than 5% lower than PARENT, past result is worse
=> No need to run evaluation program 		Worse
18.1 0|99.13%|1029|1020|9|0|sis$|noun|tic$|adj|100.00%|99.56 1|98.26%|345|339|6|0|esis$|noun|etic$|adj|33.53%|50.00 1|99.73%|369|368|1|0|osis$|noun|otic$|adj|35.86%|52.75 1|100.00%|217|217|0|0|ysis$|noun|ytic$|adj|21.09%|34.83 F1 of CHILD rules is more than 5% lower than PARENT, past result is worse
=> No need to run evaluation program 		Worse
19.1 0|89.42%|104|93|11|0|sity$|noun|us$|adj|100.00%|94.42 1|90.29%|103|93|10|0|osity$|noun|ous$|adj|99.04%|94.46 102|73.13%|67|49|18|0|$|verb|per$|noun|2024|WORDNET|SELF|95.34%|87.24%|1.8258|52265|54820 Same
20.1 0|95.83%|24|23|1|0|$|noun|free$|adj|100.00%|97.87 No suggested decompsoed CHILD rule found No need to run evaluation program Same
21.1 0|68.94%|425|293|132|0|e$|verb|ive$|adj|100.00%|81.62 1|68.65%|386|265|121|0|te$|verb|tive$|adj|90.82%|78.20 102|73.13%|67|49|18|0|$|verb|per$|noun|2024|WORDNET|SELF|95.34%|87.24%|1.8258|52265|54821 Same
22.1 0|36.11%|36|13|23|0|asm$|noun|astic$|adj|100.00%|53.06 No suggested decompsoed CHILD rule found No need to run evaluation program Same
23.1 0|75.74%|136|103|33|0|$|noun|ly$|adj|100.00%|86.19 1|82.22%|45|37|8|0|r$|noun|rly$|adj|33.09%|47.19 102|73.13%|67|49|18|0|$|verb|per$|noun|2024|WORDNET|SELF|95.38%|87.13%|1.8250|52199|54730
=> Not previously evaluated (even F1 is more than 5% lower) 		Worse
24.1 0|69.55%|243|169|74|0|$|noun|ian$|adj|100.00%|82.04 No suggested decompsoed CHILD rule found No need to run evaluation program Same
25.1 0|59.68%|248|148|100|0|$|verb|ive$|adj|100.00%|74.75 No suggested decompsoed CHILD rule found No need to run evaluation program Same
26.1 0|57.77%|251|145|106|0|e$|verb|ory$|adj|100.00%|73.23 1|58.44%|243|142|101|0|te$|verb|tory$|adj|96.81%|72.88 102|73.13%|67|49|18|0|$|verb|per$|noun|2024|WORDNET|SELF|95.34%|87.24%|1.8258|52265|54821 Same
26.2 2|58.95%|229|135|94|0|ate$|verb|atory$|adj|91.24%|71.62 102|73.13%|67|49|18|0|$|verb|per$|noun|2024|WORDNET|SELF|95.34%|87.24%|1.8258|52265|54821 Same
27.1 0|67.86%|196|133|63|0|ize$|verb|y$|noun|100.00%|80.85 1|87.37%|95|83|12|0|mize$|verb|my$|noun|48.47%|62.35 103|73.13%|67|49|18|0|$|verb|per$|noun|2024|WORDNET|SELF|95.32%|87.38%|1.8270|52348|54916
F1 is lower than 5%, yet not previous reviewed. 		better
27.2 2|86.96%|92|80|12|0|omize$|verb|omy$|noun|46.94%|60.97 103|73.13%|67|49|18|0|$|verb|per$|noun|2024|WORDNET|SELF|95.32%|87.37%|1.8269|52345|54913 Worse
28.1 0|65.61%|410|269|141|0|$|noun|ise$|verb|100.00%|79.23 No suggested decompsoed CHILD rule found No need to run evaluation program Same
29.1 0|93.33%|75|70|5|0|ier$|noun|y$|verb|100.00%|96.55 1|95.74%|47|45|2|0|fier$|noun|fy$|verb|62.67%|75.75 103|73.13%|67|49|18|0|$|verb|per$|noun|2024|WORDNET|SELF|95.33%|87.33%|1.8266|52323|54888 worse
29.2 2|95.56%|45|43|2|0|ifier$|noun|ify$|verb|60.00%|73.71 103|73.13%|67|49|18|0|$|verb|per$|noun|2024|WORDNET|SELF|95.33%|87.33%|1.8266|52321|54886 Worse
30.1 0|19.51%|164|32|132|0|$|noun|ical$|adj|100.00%|32.65 No suggested decompsoed CHILD rule found No need to run evaluation program Same
31.1 0|47.11%|121|57|64|0|$|noun|an$|adj|100.00%|64.04 No suggested decompsoed CHILD rule found No need to run evaluation program Same
32.1 0|9.09%|264|24|240|0|al$|adj|s$|noun|100.00%|16.67 No suggested decompsoed CHILD rule found No need to run evaluation program Same
33.1 0|73.13%|67|49|18|0|$|verb|per$|noun|100.00%|84.48 1|85.96%|57|49|8|0|p$|verb|pper$|noun|85.07%|85.52 103|73.17%|41|30|11|0|e$|noun|ery$|noun|2013|ORG_RULE|SELF|95.34%|87.38%|1.8272|52348|54906 Better
33.2 2|95.24%|21|20|1|0|ap$|verb|apper$|noun|31.34%|47.16 2|88.89%|18|16|2|0|ip$|verb|ipper$|noun|26.87%|41.26 104|73.17%|41|30|11|0|e$|noun|ery$|noun|2013|ORG_RULE|SELF|95.35%|87.35%|1.8270|52335|54888 Worse
34.1 0|54.88%|82|45|37|0|$|verb|ter$|noun|100.00%|70.87 1|80.00%|55|44|11|0|t$|verb|tter$|noun|67.07%|72.97 104|73.17%|41|30|11|0|e$|noun|ery$|noun|2013|ORG_RULE|SELF|95.33%|87.45%|1.8278|52392|54961 Better
35.1 0|58.12%|1237|719|518|0|$|noun|er$|noun|100.00%|73.5 No suggested decompsoed CHILD rule found No need to run evaluation program Same
36.0 0|94.00%|50|47|3|0|$|verb|nce$|noun|100.00%|96.91 0|94.00%|50|47|3|0|$|verb|nce$|noun|100.00%|96.91 104|73.17%|41|30|11|0|e$|noun|ery$|noun|2013|ORG_RULE|SELF|95.33%|87.50%|1.8283|52422|54991
test parent from previous better rules. 		Better
36.1 1|94.00%|50|47|3|0|e$|verb|ence$|noun|100.00%|96.91 104|73.17%|41|30|11|0|e$|noun|ery$|noun|2013|ORG_RULE|SELF|95.33%|87.50%|1.8283|52422|54991 Same
36.2 2|100.00%|13|13|0|0|ce$|verb|cence$|noun|26.00%|41.27 2|85.00%|20|17|3|0|ge$|verb|gence$|noun|40.00%|54.40 105|73.17%|41|30|11|0|e$|noun|ery$|noun|2013|ORG_RULE|SELF|95.33%|87.47%|1.8280|52405|54974 Worse
37.1 0|67.19%|64|43|21|0|ity$|noun|y$|adj|100.00%|80.37 1|73.68%|57|42|15|0|rity$|noun|ry$|adj|89.06%|80.65 105|73.17%|41|30|11|0|e$|noun|ery$|noun|2013|ORG_RULE|SELF|95.31%|87.57%|1.8288|52464|55048 Best
37.2 2|76.36%|55|42|13|0|arity$|noun|ary$|adj|85.94%|80.87 105|73.17%|41|30|11|0|e$|noun|ery$|noun|2013|ORG_RULE|SELF|95.31%|87.57%|1.8288|52464|55046 Same
37.3 3|90.48%|21|19|2|0|narity$|noun|nary$|adj|32.81%|48.16 105|73.17%|41|30|11|0|e$|noun|ery$|noun|2013|ORG_RULE|SELF|95.33%|87.53%|1.8286|52441|55012 Worse