# Uses of Package
opennlp.tools.tokenize

Packages that use opennlp.tools.tokenize
Package
Description
opennlp.tools.cmdline.parser
opennlp.tools.cmdline.tokenizer
opennlp.tools.formats
Experimental package related to converting various corpora to OpenNLP Format.
opennlp.tools.formats.ad
Experimental package related to the `Arvores Deitadas corpus` format.
opennlp.tools.formats.brat
Experimental package related to the corpus format used by the "brat rapid annotation tool" (brat).
opennlp.tools.formats.conllu
Experimental package related to the CoNNL-U format.
opennlp.tools.formats.convert
opennlp.tools.formats.irishsentencebank
Experimental package related to the Irish Sentence Bank format.
opennlp.tools.formats.letsmt
Experimental package related to the `letsmt` corpus format.
opennlp.tools.formats.masc
Experimental package related to the `MASC` corpus format.
opennlp.tools.formats.muc
Experimental package related to the `MUC` corpus format.
opennlp.tools.sentdetect
Package related to identifying sentence boundaries.
opennlp.tools.tokenize
Contains classes related to finding token or words in a string.
opennlp.tools.tokenize.lang
opennlp.tools.tokenize.lang.en
opennlp.tools.util.featuregen
This package contains classes for generating sequence features.

- Classes in opennlp.tools.tokenize used by opennlp.tools.cmdline.parser
  Class
  Description
  Tokenizer
  The interface for tokenizers, which segment a string into its tokens.
- Classes in opennlp.tools.tokenize used by opennlp.tools.cmdline.tokenizer
  Class
  Description
  TokenizerEvaluationMonitor
  A marker interface for evaluating `tokenizers`.
  TokenizerModel
  The `TokenizerModel` is the model used by a learnable `Tokenizer`.
  TokenSample
  A `TokenSample` is text with token spans.
- Classes in opennlp.tools.tokenize used by opennlp.tools.formats
  Class
  Description
  Detokenizer
  A `Detokenizer` merges tokens back to their detokenized representation.
  TokenSample
  A `TokenSample` is text with token spans.
- Classes in opennlp.tools.tokenize used by opennlp.tools.formats.ad
  Class
  Description
  TokenSample
  A `TokenSample` is text with token spans.
- Classes in opennlp.tools.tokenize used by opennlp.tools.formats.brat
  Class
  Description
  Tokenizer
  The interface for tokenizers, which segment a string into its tokens.
  TokenizerModel
  The `TokenizerModel` is the model used by a learnable `Tokenizer`.
- Classes in opennlp.tools.tokenize used by opennlp.tools.formats.conllu
  Class
  Description
  TokenSample
  A `TokenSample` is text with token spans.
- Classes in opennlp.tools.tokenize used by opennlp.tools.formats.convert
  Class
  Description
  Detokenizer
  A `Detokenizer` merges tokens back to their detokenized representation.
  TokenSample
  A `TokenSample` is text with token spans.
- Classes in opennlp.tools.tokenize used by opennlp.tools.formats.irishsentencebank
  Class
  Description
  TokenSample
  A `TokenSample` is text with token spans.
- Classes in opennlp.tools.tokenize used by opennlp.tools.formats.letsmt
  Class
  Description
  Detokenizer
  A `Detokenizer` merges tokens back to their detokenized representation.
- Classes in opennlp.tools.tokenize used by opennlp.tools.formats.masc
  Class
  Description
  TokenSample
  A `TokenSample` is text with token spans.
- Classes in opennlp.tools.tokenize used by opennlp.tools.formats.muc
  Class
  Description
  Tokenizer
  The interface for tokenizers, which segment a string into its tokens.
- Classes in opennlp.tools.tokenize used by opennlp.tools.sentdetect
  Class
  Description
  Detokenizer
  A `Detokenizer` merges tokens back to their detokenized representation.
- Classes in opennlp.tools.tokenize used by opennlp.tools.tokenize
  Class
  Description
  DetokenizationDictionary
  DetokenizationDictionary.Operation
  Detokenizer
  A `Detokenizer` merges tokens back to their detokenized representation.
  Detokenizer.DetokenizationOperation
  This enum contains an operation for every token to merge the tokens together to their detokenized form.
  SimpleTokenizer
  A basic `Tokenizer` implementation which performs tokenization using character classes.
  TokenContextGenerator
  Interface for context generators required for `TokenizerME`.
  Tokenizer
  The interface for tokenizers, which segment a string into its tokens.
  TokenizerEvaluationMonitor
  A marker interface for evaluating `tokenizers`.
  TokenizerFactory
  The factory that provides `Tokenizer` default implementation and resources.
  TokenizerModel
  The `TokenizerModel` is the model used by a learnable `Tokenizer`.
  TokenSample
  A `TokenSample` is text with token spans.
  WhitespaceTokenizer
  A basic `Tokenizer` implementation which performs tokenization using white spaces.
- Classes in opennlp.tools.tokenize used by opennlp.tools.tokenize.lang
  Class
  Description
  TokenContextGenerator
  Interface for context generators required for `TokenizerME`.
- Classes in opennlp.tools.tokenize used by opennlp.tools.tokenize.lang.en
  Class
  Description
  TokenSample
  A `TokenSample` is text with token spans.
- Classes in opennlp.tools.tokenize used by opennlp.tools.util.featuregen
  Class
  Description
  Tokenizer
  The interface for tokenizers, which segment a string into its tokens.