Some tokenization "test" to do !  We are checking so-so and so - so and
--- also...  Especially we are interested in abrs e.g.  Ave.  which are
very special.  Single char abrs like John C.  Mills.  This is zyz. but
not known as abbreviation.  This is zyz. BUT not known as abbreviation.
Another case is .  in a sentence???  Or .Net .12 or so.  Numbers 9.23 1,23
$12 22% #2 and so on !!!  Parentheses (which are important) and [numeric]
{expressions} ((*)) like (3 - 5) + 2 * -1 / 12 or 1/2 must work too.
Also mark@twain.com and 9.4.124.8 and
www.ibm-research.com @are also@ ### $$ @@ -checked.  Commas, and semicolons; and colons:
are ::: interesting,,, ,too?  Apostrophes ''' are' 'interesting as well: L'Oreal
Tom's 'don't' 1'2'3 8''. Also 'used' as 'quotations'. Let's go to the internet-cafe and
chat with foo-bar.

The next lines are paragraph boundary tests:

tok1 tok2

tok3 tok4

tok5

- tokX tokY

tok6