Package opennlp.tools.tokenize
Class SimpleTokenizer
java.lang.Object
opennlp.tools.tokenize.SimpleTokenizer
- All Implemented Interfaces:
Tokenizer
-
Field Summary
FieldsModifier and TypeFieldDescriptionstatic final SimpleTokenizerUse this static reference to retrieve an instance of theSimpleTokenizer. -
Method Summary
Modifier and TypeMethodDescriptionvoidsetKeepNewLines(boolean keepNewLines) Switches whether to keep new lines or not.String[]Splits a string into its atomic parts.Span[]Finds the boundaries of atomic parts in a string.
-
Field Details
-
INSTANCE
Use this static reference to retrieve an instance of theSimpleTokenizer.
-
-
Method Details
-
tokenizePos
Description copied from interface:TokenizerFinds the boundaries of atomic parts in a string.- Parameters:
s- The string to be tokenized.- Returns:
- The
spans (offsets intofor each token as the individuals array elements.s)
-
tokenize
Description copied from interface:TokenizerSplits a string into its atomic parts. -
setKeepNewLines
public void setKeepNewLines(boolean keepNewLines) Switches whether to keep new lines or not.- Parameters:
keepNewLines-Trueif new lines are kept,falseotherwise.
-