Package opennlp.tools.tokenize
Class TokenSampleStream
- All Implemented Interfaces:
AutoCloseable,ObjectStream<TokenSample>
This class is a
stream filter which reads in string encoded
samples and creates samples out of them.
The input string sample is tokenized if a whitespace or the special separator chars occur.
Sample:
"token1 token2 token3<SPLIT>token4"
The tokens token1 and token2 are separated by a whitespace,
token3 and token4 are separated by the special character sequence.
In this case, the default split sequence applies.
Note: The sequence must be unique in the input string and is not escaped.
-
Constructor Summary
ConstructorsConstructorDescriptionTokenSampleStream(ObjectStream<String> sentences) Initializes ainstance.TokenSampleStream(ObjectStream<String> samples, String separatorChars) Initializes ainstance. -
Method Summary
Methods inherited from class opennlp.tools.util.FilterObjectStream
close, reset
-
Constructor Details
-
TokenSampleStream
Initializes ainstance.- Parameters:
samples- A plain textline stream. Must not benull.separatorChars- The characters to be considered separators. SeeTokenSample.DEFAULT_SEPARATOR_CHARS. Must not benull.
-
TokenSampleStream
Initializes ainstance.- Parameters:
sentences- A plain textline stream. Must not benull.
-
-
Method Details
-
read
Description copied from interface:ObjectStreamReturns the nextObjectStreamobject. Calling this method repeatedly until it returnsnullwill return each object from the underlying source exactly once.- Returns:
- The next object or
nullto signal that the stream is exhausted. - Throws:
IOException- Thrown if there is an error during reading.
-