Word Tokenizer Algorithm (Java)
Word Tokenizer is used to tokenize and filter out words and characters in TI and AB fields from citations. The algorithm used in the Java version is slice different than the Lisp version. Please see TI report and AB report for details.
The procedures and criteria are described as follows:
- Remove matched string (case sensitive)
| Beginning string | Ending string | References
|
|---|
| [correction | ] | ?
|
| (abstracts were not | included) | 0289-10695616
|
| [J. Neuroimmunol. 104, | 85-91] | ?
|
| (abstracts presented at recent scientific meetings | package inserts) | 0306-11261533
|
| (Japanese Association of Intellectual Copyright | #130,591) | 0306-11276498
|
- remove matched ending string (case sensitive)
| Beginning string | References
|
|---|
| CopyrightCopyright | ?
|
| Copyright Copyright | ?
|
| Copyright | ?
|
| .Copyright | ?
|
| )Copyright | ?
|
| (abstract | ?
|
| (ABSTRACT | ?
|
| ? Copyright | ?
|
| ) Copyright | ?
|
| Copyright 2001 Wiley-Liss, Inc. | 0310-11391771
|
- remove matched ending string (case insensitive)
| Beginning string | Ending string | Exceptions | References
|
|---|
| [ | ] | [ | ?
|
| [ | .] | [These syndromes can be a contributory | 0408-10199143
|
| [published erratum | ] | None | ?
|
| [forensic science international | ] | None | ?
|
| (abstract truncated | ) | None | ?
|
| (published erratum | ) | None | ?
|
| (comments | )] | None | ?
|
- remove exact matched ending string (case insensitive)
| Match string | References
|
|---|
| [see comments] | ?
|
| (see comments) | ?
|
| [seecomments] | ?
|
| [ see comments] | ?
|
| [in process citation] | ?
|
| (in process citation) | ?
|
| [corrected] | ?
|
| [correction of artistic] | ?
|
| (letter) | ?
|
| (letter)] | ?
|
| (editorial)] | ?
|
- remove [title]
- remove non-alpha-num char (beginning and ending) from all words
- expand contraction
- replace punctuation with space
- remove words with less 3 characters
- remove words begins with digit
