TI Report (Difference)
This page describes the difference in words tokenization for TI between Java and lisp versions. There are 159 different occurrences in three types of causes in 2004 training set. They are details as follows:
- Expansion errors (2)
Original string | New Words (Java) | Old Words (Lisp) | Source
|
---|
...don't.. | DO NOT | DON | 0294: 10859841
|
...can't... | CANNOT | CAN | 0316: 11673724
|
- Tokenization errors (4)
Original string | New Words (Java) | Old Words (Lisp) | Source
|
---|
...retroplasmid[... | ...retroplasmid | retroplasmi | 0302: 11001704
|
...(C2O4)Cl2[[Fe(C2O4)(OH)4] | ...C204 CL2 C204 | ..C204 | 0303: 11205012
|
...[[Na(dibenzo-18-crown-6)(MeCN)]3[PMo12O40]] | ...DIBENZO CROWN MECN PMO12O40 | | 0315: 11578196
|
...2-[substituted acetyl]-amino-5-alkyl-1,3,4-thiadiazoles | SUBSTITUTED ACETYL AMINO ALKYL THIADIAZOLES | | 0404: 9868551
|
- Remove ...[correction of...]... (153)
- 0284: 10535671, 10550444
- 0285: 10583726, 10593612
- 0286: 10602121, 10611632
- 0287: 10632073, 10649694, 10656876
- 0288: 10664895, 10665342, 10675514, 10676662
- 0289: 10700132, 10717712
- 0290: 10726856, 10746397, 10750193
- 0291: 10777771
- 0292: 10797379, 10809817, 10820389, 10821636
- 0293: 10821812, 10854880
- 0294: 10866041, 10870307, 10871161, 10877549
- 0295: 10917489
- 0296: 10930074, 10931436, 10933152, 10945166
- 0297: 10982000
- 0299: 11027856, 11030661
- 0300: 11081738
- 0301: 11098039, 11101655
- 0302: 11125550, 11138534
- 0303: 11029803
- 0304: 11158807, 11163336, 11166953, 11170754, 11176048, 11181838, 11181900, 11215731
- 0305: 11222723, 11234902, 11237328, 11238899, 11239663, 11245836, 11246679
- 0306: 11249995, 11260312, 11275357, 11284188
- 0307: 11284733, 11286145, 11286811, 11294464, 11297467, 11301119, 11301232
- 0308: 11322494, 11324784, 11337257
- 0309: 11348132, 11350315, 11356562, 11380813, 11385484
- 0310: 11388169, 11392354, 11399089, 11401187, 11412876, 11418412, 11418735
- 0311: 11419882, 11423299, 11429314, 11441186, 11447130
- 0312: 11451607, 11451651, 11479400
- 0313: 11489943, 11490709
- 0314: 11517527, 11523083, 11524446, 11528428, 11555872
- 0315: 11558588, 11581170
- 0316: 11589894, 11592084
- 0317: 11678668, 11689277, 11693709, 11704776
- 0318: 11730929
- 0319: 11585831, 11741255, 11747347, 11748200, 11765341
- 0400: 9746215
- 0402: 9793965, 9811196
- 0404: 9861054
- 0405: 9892879, 9914379, 9916133
- 0406: 9949746, 9950324, 9972251, 10036782
- 0407: 10051468, 10064675, 10076631, 10079018, 10085512, 10085613
- 0408: 10096609, 10102289, 10103218, 10190525, 10193221
- 0409: 10220432, 10231178
- 0410: 10347636, 10348046, 10352226
- 0411: 10378459
- 0412: 10389455, 10399325, 10403116
- 0413: 10421131, 10421592, 10441466
- 0414: 10452735, 10455375, 10465100
- 0415: 10484306, 10497146
- 0416: 10530778