Package org.apache.lucene.analysis.fa
Class PersianAnalyzer
java.lang.Object
org.apache.lucene.analysis.Analyzer
org.apache.lucene.analysis.StopwordAnalyzerBase
org.apache.lucene.analysis.fa.PersianAnalyzer
- All Implemented Interfaces:
Closeable,AutoCloseable
Analyzer for Persian.
This Analyzer uses PersianCharFilter which implies tokenizing around zero-width
non-joiner in addition to whitespace. Some persian-specific variant forms (such as farsi yeh and
keheh) are standardized. "Stemming" is accomplished via stopwords.
- Since:
- 3.1
-
Nested Class Summary
Nested classes/interfaces inherited from class org.apache.lucene.analysis.Analyzer
Analyzer.ReuseStrategy, Analyzer.TokenStreamComponents -
Field Summary
FieldsModifier and TypeFieldDescriptionstatic final StringFile containing default Persian stopwords.static final StringThe comment character in the stopwords file.Fields inherited from class org.apache.lucene.analysis.StopwordAnalyzerBase
stopwordsFields inherited from class org.apache.lucene.analysis.Analyzer
GLOBAL_REUSE_STRATEGY, PER_FIELD_REUSE_STRATEGY -
Constructor Summary
ConstructorsConstructorDescriptionBuilds an analyzer with the default stop words:DEFAULT_STOPWORD_FILE.PersianAnalyzer(CharArraySet stopwords) Builds an analyzer with the given stop wordsPersianAnalyzer(CharArraySet stopwords, CharArraySet stemExclusionSet) Builds an analyzer with the given stop word. -
Method Summary
Modifier and TypeMethodDescriptionprotected Analyzer.TokenStreamComponentscreateComponents(String fieldName) CreatesAnalyzer.TokenStreamComponentsused to tokenize all the text in the providedReader.static CharArraySetReturns an unmodifiable instance of the default stop-words set.protected ReaderinitReader(String fieldName, Reader reader) Wraps the Reader withPersianCharFilterprotected TokenStreamnormalize(String fieldName, TokenStream in) Methods inherited from class org.apache.lucene.analysis.StopwordAnalyzerBase
getStopwordSet, loadStopwordSet, loadStopwordSetMethods inherited from class org.apache.lucene.analysis.Analyzer
attributeFactory, close, getOffsetGap, getPositionIncrementGap, getReuseStrategy, initReaderForNormalization, normalize, tokenStream, tokenStream
-
Field Details
-
DEFAULT_STOPWORD_FILE
File containing default Persian stopwords.Default stopword list is from http://members.unine.ch/jacques.savoy/clef/index.html The stopword list is BSD-Licensed.
- See Also:
-
STOPWORDS_COMMENT
The comment character in the stopwords file. All lines prefixed with this will be ignored- See Also:
-
-
Constructor Details
-
PersianAnalyzer
public PersianAnalyzer()Builds an analyzer with the default stop words:DEFAULT_STOPWORD_FILE. -
PersianAnalyzer
Builds an analyzer with the given stop words- Parameters:
stopwords- a stopword set
-
PersianAnalyzer
Builds an analyzer with the given stop word. If a none-empty stem exclusion set is provided this analyzer will add aSetKeywordMarkerFilterbeforePersianStemFilter.- Parameters:
stopwords- a stopword setstemExclusionSet- a set of terms not to be stemmed
-
-
Method Details
-
getDefaultStopSet
Returns an unmodifiable instance of the default stop-words set.- Returns:
- an unmodifiable instance of the default stop-words set.
-
createComponents
CreatesAnalyzer.TokenStreamComponentsused to tokenize all the text in the providedReader.- Specified by:
createComponentsin classAnalyzer- Returns:
Analyzer.TokenStreamComponentsbuilt from aStandardTokenizerfiltered withLowerCaseFilter,DecimalDigitFilter,ArabicNormalizationFilter,PersianNormalizationFilter, Persian Stop words, andPersianStemFilter.
-
normalize
-
initReader
Wraps the Reader withPersianCharFilter- Overrides:
initReaderin classAnalyzer
-