opennlp.tools.tokenize
Class TokenSample

java.lang.Object
  opennlp.tools.tokenize.TokenSample

public class TokenSample
extends java.lang.Object

A TokenSample is text with token spans.

Field Summary

static java.lang.String

DEFAULT_SEPARATOR_CHARS

Constructor Summary

TokenSample(Detokenizer detokenizer, java.lang.String[] tokens)

TokenSample(java.lang.String text, Span[] tokenSpans)
Initializes the current instance.

Method Summary

java.lang.String	getText() Retrieves the text.
Span[]	getTokenSpans() Retrieves the token spans.
static TokenSample	parse(java.lang.String sampleString, java.lang.String separatorChars)
java.lang.String	toString()

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

DEFAULT_SEPARATOR_CHARS

public static final java.lang.String DEFAULT_SEPARATOR_CHARS

See Also:

Constant Field Values

Constructor Detail

TokenSample

public TokenSample(java.lang.String text,
                   Span[] tokenSpans)

Initializes the current instance.

Parameters:

text - the text which contains the tokens.

tokenSpans - the spans which mark the begin and end of the tokens.

TokenSample

public TokenSample(Detokenizer detokenizer,
                   java.lang.String[] tokens)

Method Detail

getText

public java.lang.String getText()

Retrieves the text.

getTokenSpans

public Span[] getTokenSpans()

Retrieves the token spans.

toString

public java.lang.String toString()

Overrides:

toString in class java.lang.Object

parse

public static TokenSample parse(java.lang.String sampleString,
                                java.lang.String separatorChars)