Lexical Tools

Token Types

Token type can be different depends on the applications. Table below shows a common use for these token types.

Token Type	Characters example	condition	Clean-up Action
Token	N/A	All	add token into list Update last type
Space Delimiter	space, tab	> lastType == Token lastType == Restore Delimiter lastType == Stripping Delimiter lastType == Strippable Delimiter	add token into list Update last type
Stripped	N/A	All	add token into list Update last type
Restore Delimiter	( { [ < " '	All	add token into list Update last type
Striping Delimiter	) } ] ! ? >	All	add token into list Update last type Strip previous token if they are in conflict list or stripped type
Strippable Delimiter	, . : ;	lastType == Stripped	add token into list Update last type
Conflict List	- , : ; space	N/A	N/A
Non-Delimiter Punctuation List	*- _ @ # % & \ ~ + = \| $ ` ^ /**	N/A	N/A

Notes: Punctuations: " ' and / are not handled gracefully.