Because of a lapse in government funding, the information on this website may not be up to date, transactions submitted via the website may not be processed, and the agency may not be able to respond to inquiries until appropriations are enacted. The NIH Clinical Center (the research hospital of NIH) is open. For more details about its operating status, please visit cc.nih.gov. Updates regarding government operating status and resumption of normal operations can be found at OPM.gov.

Text Categorization

TI Report (Difference)

This page describes the difference in words tokenization for TI between Java and lisp versions. There are 159 different occurrences in three types of causes in 2004 training set. They are details as follows:

  • Expansion errors (2)
    Original stringNew Words (Java)Old Words (Lisp)Source
    ...don't..DO NOTDON0294: 10859841
    ...can't...CANNOTCAN0316: 11673724

  • Tokenization errors (4)
    Original stringNew Words (Java)Old Words (Lisp)Source
    ...retroplasmid[......retroplasmidretroplasmi0302: 11001704
    ...(C2O4)Cl2[[Fe(C2O4)(OH)4]...C204 CL2 C204..C2040303: 11205012
    ...[[Na(dibenzo-18-crown-6)(MeCN)]3[PMo12O40]]...DIBENZO CROWN MECN PMO12O40 0315: 11578196
    ...2-[substituted acetyl]-amino-5-alkyl-1,3,4-thiadiazolesSUBSTITUTED ACETYL AMINO ALKYL THIADIAZOLES 0404: 9868551

  • Remove ...[correction of...]... (153)
    • 0284: 10535671, 10550444
    • 0285: 10583726, 10593612
    • 0286: 10602121, 10611632
    • 0287: 10632073, 10649694, 10656876
    • 0288: 10664895, 10665342, 10675514, 10676662
    • 0289: 10700132, 10717712
    • 0290: 10726856, 10746397, 10750193
    • 0291: 10777771
    • 0292: 10797379, 10809817, 10820389, 10821636
    • 0293: 10821812, 10854880
    • 0294: 10866041, 10870307, 10871161, 10877549
    • 0295: 10917489
    • 0296: 10930074, 10931436, 10933152, 10945166
    • 0297: 10982000
    • 0299: 11027856, 11030661
    • 0300: 11081738
    • 0301: 11098039, 11101655
    • 0302: 11125550, 11138534
    • 0303: 11029803
    • 0304: 11158807, 11163336, 11166953, 11170754, 11176048, 11181838, 11181900, 11215731
    • 0305: 11222723, 11234902, 11237328, 11238899, 11239663, 11245836, 11246679
    • 0306: 11249995, 11260312, 11275357, 11284188
    • 0307: 11284733, 11286145, 11286811, 11294464, 11297467, 11301119, 11301232
    • 0308: 11322494, 11324784, 11337257
    • 0309: 11348132, 11350315, 11356562, 11380813, 11385484
    • 0310: 11388169, 11392354, 11399089, 11401187, 11412876, 11418412, 11418735
    • 0311: 11419882, 11423299, 11429314, 11441186, 11447130
    • 0312: 11451607, 11451651, 11479400
    • 0313: 11489943, 11490709
    • 0314: 11517527, 11523083, 11524446, 11528428, 11555872
    • 0315: 11558588, 11581170
    • 0316: 11589894, 11592084
    • 0317: 11678668, 11689277, 11693709, 11704776
    • 0318: 11730929
    • 0319: 11585831, 11741255, 11747347, 11748200, 11765341
    • 0400: 9746215
    • 0402: 9793965, 9811196
    • 0404: 9861054
    • 0405: 9892879, 9914379, 9916133
    • 0406: 9949746, 9950324, 9972251, 10036782
    • 0407: 10051468, 10064675, 10076631, 10079018, 10085512, 10085613
    • 0408: 10096609, 10102289, 10103218, 10190525, 10193221
    • 0409: 10220432, 10231178
    • 0410: 10347636, 10348046, 10352226
    • 0411: 10378459
    • 0412: 10389455, 10399325, 10403116
    • 0413: 10421131, 10421592, 10441466
    • 0414: 10452735, 10455375, 10465100
    • 0415: 10484306, 10497146
    • 0416: 10530778