Package opennlp.tools.tokenize

# Interface Detokenizer

All Known Implementing Classes:
  `DictionaryDetokenizer`

public interface Detokenizer
A `Detokenizer` merges tokens back to their detokenized representation.

- ## Nested Class Summary

  Nested Classes
  Modifier and Type
  Interface
  Description
  `static enum`
  `Detokenizer.DetokenizationOperation`
  This enum contains an operation for every token to merge the tokens together to their detokenized form.

- ## Method Summary

  Modifier and Type
  Method
  Description
  `Detokenizer.DetokenizationOperation[]`
  `detokenize(String[] tokens)`
  Detokenizes the collection of tokens.
  `String`
  `detokenize(String[] tokens, String splitMarker)`
  Detokenizes the input `tokens` into a String.

- ## Method Details

  - ### detokenize

    Detokenizer.DetokenizationOperation[] detokenize(String[] tokens)
    Detokenizes the collection of tokens.
    Parameters:
      `tokens` - The elements which should be detokenized.
    Returns:
      The `merge operations` to handle given `tokens`.

  - ### detokenize

    String detokenize(String[] tokens, String splitMarker)
    Detokenizes the input `tokens` into a String. Tokens which are connected without a `whitespace` character in between can be separated by a given `splitMarker`.
    Parameters:
      `tokens` - The elements which should be concatenated.
      `splitMarker` - The split marker or `null`.
    Returns:
      The concatenated tokens as a single string.