Package opennlp.tools.tokenize

# Class ThreadSafeTokenizerME

java.lang.Object
opennlp.tools.tokenize.ThreadSafeTokenizerME

All Implemented Interfaces:
  `AutoCloseable`, `Probabilistic`, `Tokenizer`

@ThreadSafe public class ThreadSafeTokenizerME extends Object implements Tokenizer, Probabilistic, AutoCloseable
A thread-safe version of `TokenizerME`. Using it is completely transparent. You can use it in a single-threaded context as well, it only incurs a minimal overhead.

Note:
This implementation uses a `ThreadLocal`. Although the implementation is lightweight because the model is not duplicated, if you have many long-running threads, you may run into memory problems.

Be careful when using this in a Jakarta EE application, for example.

The user is responsible for clearing the `ThreadLocal` via calling `close()`.

See Also:
  - `Probabilistic`
  - `Tokenizer`
  - `TokenizerME`

- ## Constructor Summary

  Constructors
  Constructor
  Description
  `ThreadSafeTokenizerME(String language)`
  Initializes a `ThreadSafeTokenizerME` by downloading a default model for a given `language`.
  `ThreadSafeTokenizerME(TokenizerModel model)`
  Initializes a `ThreadSafeTokenizerME` with the specified `model`.
  `ThreadSafeTokenizerME(TokenizerModel model, Dictionary abbDict)`
  Instantiates a `ThreadSafeTokenizerME` with an existing `TokenizerModel`.

- ## Method Summary

  Modifier and Type
  Method
  Description
  `void`
  `close()`
  `double[]`
  `getProbabilities()`
  Deprecated, for removal: This API element is subject to removal in a future version.
  Use `probs()` instead.
  `double[]`
  `probs()`
  Retrieves the probabilities of the last decoded sequence.
  `String[]`
  `tokenize(String s)`
  Splits a string into its atomic parts.
  `Span[]`
  `tokenizePos(String s)`
  Finds the boundaries of atomic parts in a string.

  ### Methods inherited from class java.lang.Object

  `equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

- ## Constructor Details

  - ### ThreadSafeTokenizerME

    public ThreadSafeTokenizerME(String language) throws IOException
    Initializes a `ThreadSafeTokenizerME` by downloading a default model for a given `language`.
    Parameters:
      `language` - An ISO conform language code.
    Throws:
      `IOException` - Thrown if the model could not be downloaded or saved.

  - ### ThreadSafeTokenizerME

    public ThreadSafeTokenizerME(TokenizerModel model)
    Initializes a `ThreadSafeTokenizerME` with the specified `model`.
    Parameters:
      `model` - A valid `TokenizerModel`.

  - ### ThreadSafeTokenizerME

    public ThreadSafeTokenizerME(TokenizerModel model, Dictionary abbDict)
    Instantiates a `ThreadSafeTokenizerME` with an existing `TokenizerModel`.
    Parameters:
      `model` - The `TokenizerModel` to be used.
      `abbDict` - The `Dictionary` to be used. It must fit the language of the `model`.

- ## Method Details

  - ### tokenize

    public String[] tokenize(String s)
    Description copied from interface: `Tokenizer`
    Splits a string into its atomic parts.
    Specified by:
      `tokenize` in interface `Tokenizer`
    Parameters:
      `s` - The string to be tokenized.
    Returns:
      The String[] with the individual tokens as the array elements.

  - ### tokenizePos

    public Span[] tokenizePos(String s)
    Description copied from interface: `Tokenizer`
    Finds the boundaries of atomic parts in a string.
    Specified by:
      `tokenizePos` in interface `Tokenizer`
    Parameters:
      `s` - The string to be tokenized.
    Returns:
      The `spans (offsets into `s`)` for each token as the individuals array elements.

  - ### probs

    public double[] probs()
    Description copied from interface: `Probabilistic`
    Retrieves the probabilities of the last decoded sequence.
    Specified by:
      `probs` in interface `Probabilistic`
    Returns:
      An array with the same number of probabilities as tokens were sent to the computational method when it was last called.

  - ### getProbabilities

    @Deprecated(forRemoval=true, since="2.5.5") public double[] getProbabilities()
    Deprecated, for removal: This API element is subject to removal in a future version.
    Use `probs()` instead.

  - ### close

    public void close()
    Specified by:
      `close` in interface `AutoCloseable`