Package opennlp.tools.tokenize

# Class TokenSample

java.lang.Object
opennlp.tools.tokenize.TokenSample

All Implemented Interfaces:
  `Serializable`, `Sample`

public class TokenSample extends Object implements Sample
A `TokenSample` is text with token spans.

See Also:
  - Serialized Form

- ## Field Summary

  Fields
  Modifier and Type
  Field
  Description
  `static final String`
  `DEFAULT_SEPARATOR_CHARS`

- ## Constructor Summary

  Constructors
  Constructor
  Description
  `TokenSample(String text, Span[] tokenSpans)`
  Initializes a `instance`.
  `TokenSample(Detokenizer detokenizer, String[] tokens)`
  Initializes a `instance` via a `Detokenizer`.

- ## Method Summary

  Modifier and Type
  Method
  Description
  `boolean`
  `equals(Object obj)`
  `String`
  `getText()`
  `Span[]`
  `getTokenSpans()`
  `int`
  `hashCode()`
  `static TokenSample`
  `parse(String sampleString, String separatorChars)`
  Parses a string sample.
  `String`
  `toString()`

  ### Methods inherited from class java.lang.Object

  `getClass, notify, notifyAll, wait, wait, wait`

- ## Field Details

  - ### DEFAULT_SEPARATOR_CHARS

    public static final String DEFAULT_SEPARATOR_CHARS
    See Also:
      - Constant Field Values

- ## Constructor Details

  - ### TokenSample

    public TokenSample(String text, Span[] tokenSpans)
    Initializes a `instance`.
    Parameters:
      `text` - The text which contains the tokens. Must not be `null`.
      `tokenSpans` - The spans which mark the start and end of the tokens. Must not be `null`.

  - ### TokenSample

    public TokenSample(Detokenizer detokenizer, String[] tokens)
    Initializes a `instance` via a `Detokenizer`.
    Parameters:
      `detokenizer` - The text which contains the tokens. Must not be `null`.
      `tokens` - The tokens to be processed. Must not be `null`.

- ## Method Details

  - ### getText

    public String getText()
    Returns:
      Retrieves the text.

  - ### getTokenSpans

    public Span[] getTokenSpans()
    Returns:
      Retrieves the token `spans`.

  - ### toString

    public String toString()
    Overrides:
      `toString` in class `Object`

  - ### parse

    public static TokenSample parse(String sampleString, String separatorChars)
    Parses a string sample.
    Parameters:
      `sampleString` - The sample to be parsed. Must not be `null`.
      `separatorChars` - The characters to be considered separators. See `DEFAULT_SEPARATOR_CHARS`. Must not be `null`.
    Returns:
      A valid `TokenSample` instance.

  - ### hashCode

    public int hashCode()
    Overrides:
      `hashCode` in class `Object`

  - ### equals

    public boolean equals(Object obj)
    Overrides:
      `equals` in class `Object`