org.dkpro.tc.features.pair.core.ngram

Class LuceneKeywordPFE

java.lang.Object

org.apache.uima.resource.Resource_ImplBase

org.apache.uima.fit.component.Resource_ImplBase

org.dkpro.tc.api.features.FeatureExtractorResource_ImplBase

org.dkpro.tc.features.ngram.base.NGramFeatureExtractorBase

org.dkpro.tc.features.ngram.base.LuceneFeatureExtractorBase

org.dkpro.tc.features.pair.core.ngram.meta.LucenePFEBase

org.dkpro.tc.features.pair.core.ngram.LuceneKeywordPFE

All Implemented Interfaces:

org.apache.uima.fit.component.ExternalResourceAware, org.apache.uima.resource.Resource, MetaDependent, PairFeatureExtractor

Direct Known Subclasses:

LuceneKeywordCPFE

public class LuceneKeywordPFE
extends LucenePFEBase
implements PairFeatureExtractor

Pair keyword ngram feature extractor for KeywordNGramDFE Can be used to extract ngrams from one or both documents in the pair, and parameters for each document (view 1's, view 2's) can be set separately, or both documents can be treated together as one extended document.
Note that ngram features created by this class are each from a single document, i.e., not combinations of ngrams from the pair of documents.

Field Summary

Fields

Modifier and Type	Field and Description
protected boolean	includeCommas
static String	KEYWORD_NGRAM_FIELD
static String	KEYWORD_NGRAM_FIELD1
static String	KEYWORD_NGRAM_FIELD2
protected int	keywordMaxN
protected int	keywordMinN
protected int	keywordNgramUseTopK
protected Set<String>	keywords
protected String	keywordsFile
protected boolean	markSentenceBoundary
protected boolean	markSentenceLocation
protected boolean	markViewBlindNgramsWithLocalView
protected int	ngramMaxN1
protected int	ngramMaxN2
protected int	ngramMinN1
protected int	ngramMinN2
static String	PARAM_KEYWORD_NGRAM_INCLUDE_COMMAS
static String	PARAM_KEYWORD_NGRAM_MARK_SENTENCE_BOUNDARY
static String	PARAM_KEYWORD_NGRAM_MARK_SENTENCE_LOCATION
static String	PARAM_KEYWORD_NGRAM_MAX_N
static String	PARAM_KEYWORD_NGRAM_MAX_N_VIEW1 Maximum size n of ngrams from View 1's.
static String	PARAM_KEYWORD_NGRAM_MAX_N_VIEW2 Maximum size n of ngrams from View 2's.
static String	PARAM_KEYWORD_NGRAM_MIN_N
static String	PARAM_KEYWORD_NGRAM_MIN_N_VIEW1 Minimum size n of ngrams from View 1's.
static String	PARAM_KEYWORD_NGRAM_MIN_N_VIEW2 Minimum size n of ngrams from View 2's.
static String	PARAM_KEYWORD_NGRAM_USE_TOP_K
static String	PARAM_MARK_VIEWBLIND_KEYWORD_NGRAMS_WITH_LOCAL_VIEW This option collects a FrequencyDistribution of ngrams across both documents of all pairs, but when writing features, the view where a particular ngram is found is recorded with the ngram.
static String	PARAM_NGRAM_KEYWORDS_FILE
static String	PARAM_USE_VIEW1_KEYWORD_NGRAMS_AS_FEATURES Each ngram from View 1 documents added to the document pair instance as a feature.
static String	PARAM_USE_VIEW2_KEYWORD_NGRAMS_AS_FEATURES Each ngram from View 1 documents added to the document pair instance as a feature.
static String	PARAM_USE_VIEWBLIND_KEYWORD_NGRAMS_AS_FEATURES All qualifying ngrams from anywhere in either document are used as features.
protected boolean	useView1NgramsAsFeatures
protected boolean	useView2NgramsAsFeatures
protected boolean	useViewBlindNgramsAsFeatures

Fields inherited from class org.dkpro.tc.features.pair.core.ngram.meta.LucenePFEBase

fieldOfTheMoment, kngramUseTopK, ngramBinaryFeatureValuesCombos, ngramUseTopK1, ngramUseTopK2, PARAM_NGRAM_BINARY_FEATURE_VALUES_COMBO, PARAM_NGRAM_USE_TOP_K_VIEW1, PARAM_NGRAM_USE_TOP_K_VIEW2, topKSetView1, topKSetView2, topNOfTheMoment

Fields inherited from class org.dkpro.tc.features.ngram.base.LuceneFeatureExtractorBase

forceRereadFromIndex, LUCENE_NGRAM_FIELD, luceneDir, PARAM_SOURCE_LOCATION

Fields inherited from class org.dkpro.tc.features.ngram.base.NGramFeatureExtractorBase

dfStore, filterPartialStopwordMatches, ngramFreqThreshold, ngramLowerCase, ngramStopwordsFile, ngramUseTopK, PARAM_FILTER_PARTIAL_STOPWORD_MATCHES, PARAM_NGRAM_FREQ_THRESHOLD, PARAM_NGRAM_LOWER_CASE, PARAM_NGRAM_MAX_N, PARAM_NGRAM_MIN_N, PARAM_NGRAM_STOPWORDS_FILE, PARAM_NGRAM_USE_TOP_K, PARAM_TF_IDF_CALCULATION, prefix, stopwords, tfIdfCalculation, topKSet

Fields inherited from class org.dkpro.tc.api.features.FeatureExtractorResource_ImplBase

featureExtractorName, PARAM_UNIQUE_EXTRACTOR_NAME

Fields inherited from interface org.apache.uima.resource.Resource

PARAM_AGGREGATE_SOFA_MAPPINGS, PARAM_CONFIG_MANAGER, PARAM_CONFIG_PARAM_SETTINGS, PARAM_EXTERNAL_OVERRIDE_SETTINGS, PARAM_PERFORMANCE_TUNING_SETTINGS, PARAM_RESOURCE_MANAGER, PARAM_UIMA_CONTEXT

Constructor Summary

Constructors

Constructor and Description
LuceneKeywordPFE()

Method Summary

Modifier and Type	Method and Description
Set<Feature>	extract(org.apache.uima.jcas.JCas view1, org.apache.uima.jcas.JCas view2)
protected String	getFeaturePrefix()
protected String	getFieldName()
List<MetaCollectorConfiguration>	getMetaCollectorClasses(Map<String,Object> parameterSettings)
protected int	getTopN()
protected de.tudarmstadt.ukp.dkpro.core.api.frequency.util.FrequencyDistribution<String>	getViewNgrams(org.apache.uima.jcas.JCas view1, org.apache.uima.jcas.JCas view2)
boolean	initialize(org.apache.uima.resource.ResourceSpecifier aSpecifier, Map<String,Object> aAdditionalParams)

Methods inherited from class org.dkpro.tc.features.pair.core.ngram.meta.LucenePFEBase

addToFeatureArray

Methods inherited from class org.dkpro.tc.features.ngram.base.LuceneFeatureExtractorBase

getTopNgrams, logSelectionProcess, passesScreening

Methods inherited from class org.apache.uima.fit.component.Resource_ImplBase

afterResourcesInitialized, getLogger, getResourceName

Methods inherited from class org.apache.uima.resource.Resource_ImplBase

destroy, getCasManager, getMetaData, getRelativePathResolver, getResourceManager, getUimaContext, getUimaContextAdmin, setLogger, setMetaData

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

KEYWORD_NGRAM_FIELD

public static final String KEYWORD_NGRAM_FIELD

See Also:

Constant Field Values

PARAM_KEYWORD_NGRAM_MIN_N

public static final String PARAM_KEYWORD_NGRAM_MIN_N

See Also:

Constant Field Values

keywordMinN

protected int keywordMinN

PARAM_KEYWORD_NGRAM_MAX_N

public static final String PARAM_KEYWORD_NGRAM_MAX_N

See Also:

Constant Field Values

keywordMaxN

protected int keywordMaxN

PARAM_NGRAM_KEYWORDS_FILE

public static final String PARAM_NGRAM_KEYWORDS_FILE

See Also:

Constant Field Values

keywordsFile

protected String keywordsFile

PARAM_KEYWORD_NGRAM_MARK_SENTENCE_BOUNDARY

public static final String PARAM_KEYWORD_NGRAM_MARK_SENTENCE_BOUNDARY

See Also:

Constant Field Values

markSentenceBoundary

protected boolean markSentenceBoundary

PARAM_KEYWORD_NGRAM_MARK_SENTENCE_LOCATION

public static final String PARAM_KEYWORD_NGRAM_MARK_SENTENCE_LOCATION

See Also:

Constant Field Values

markSentenceLocation

protected boolean markSentenceLocation

PARAM_KEYWORD_NGRAM_INCLUDE_COMMAS

public static final String PARAM_KEYWORD_NGRAM_INCLUDE_COMMAS

See Also:

Constant Field Values

includeCommas

protected boolean includeCommas

PARAM_KEYWORD_NGRAM_USE_TOP_K

public static final String PARAM_KEYWORD_NGRAM_USE_TOP_K

See Also:

Constant Field Values

keywordNgramUseTopK

protected int keywordNgramUseTopK

keywords

protected Set<String> keywords

PARAM_KEYWORD_NGRAM_MIN_N_VIEW1

public static final String PARAM_KEYWORD_NGRAM_MIN_N_VIEW1

Minimum size n of ngrams from View 1's.

See Also:

Constant Field Values

ngramMinN1

protected int ngramMinN1

PARAM_KEYWORD_NGRAM_MIN_N_VIEW2

public static final String PARAM_KEYWORD_NGRAM_MIN_N_VIEW2

Minimum size n of ngrams from View 2's.

See Also:

Constant Field Values

ngramMinN2

protected int ngramMinN2

PARAM_KEYWORD_NGRAM_MAX_N_VIEW1

public static final String PARAM_KEYWORD_NGRAM_MAX_N_VIEW1

Maximum size n of ngrams from View 1's.

See Also:

Constant Field Values

ngramMaxN1

protected int ngramMaxN1

PARAM_KEYWORD_NGRAM_MAX_N_VIEW2

public static final String PARAM_KEYWORD_NGRAM_MAX_N_VIEW2

Maximum size n of ngrams from View 2's.

See Also:

Constant Field Values

ngramMaxN2

protected int ngramMaxN2

PARAM_USE_VIEW1_KEYWORD_NGRAMS_AS_FEATURES

public static final String PARAM_USE_VIEW1_KEYWORD_NGRAMS_AS_FEATURES

Each ngram from View 1 documents added to the document pair instance as a feature. E.g. Feature: view1NG_Dear

See Also:

Constant Field Values

useView1NgramsAsFeatures

protected boolean useView1NgramsAsFeatures

PARAM_USE_VIEW2_KEYWORD_NGRAMS_AS_FEATURES

public static final String PARAM_USE_VIEW2_KEYWORD_NGRAMS_AS_FEATURES

Each ngram from View 1 documents added to the document pair instance as a feature. E.g. Feature: view2NG_Dear

See Also:

Constant Field Values

useView2NgramsAsFeatures

protected boolean useView2NgramsAsFeatures

PARAM_USE_VIEWBLIND_KEYWORD_NGRAMS_AS_FEATURES

public static final String PARAM_USE_VIEWBLIND_KEYWORD_NGRAMS_AS_FEATURES

All qualifying ngrams from anywhere in either document are used as features. Feature does not specify which view the ngram came from. E.g. Feature: allNG_Dear

See Also:

Constant Field Values

useViewBlindNgramsAsFeatures

protected boolean useViewBlindNgramsAsFeatures

PARAM_MARK_VIEWBLIND_KEYWORD_NGRAMS_WITH_LOCAL_VIEW

public static final String PARAM_MARK_VIEWBLIND_KEYWORD_NGRAMS_WITH_LOCAL_VIEW

This option collects a FrequencyDistribution of ngrams across both documents of all pairs, but when writing features, the view where a particular ngram is found is recorded with the ngram. For example, using a PARAM_NGRAM_USE_TOP_K value of 500, 400 of the ngrams in the top 500 might happen to be from View 2's; and whenever an ngram from the 500 is seen in any document, view 1 or 2, the document's view is recorded.
E.g., Feature: view2allNG_Dear
In order to use this option, PARAM_USE_VIEWBLIND_KEYWORD_NGRAMS_AS_FEATURES must also be set to true.

See Also:

Constant Field Values

markViewBlindNgramsWithLocalView

protected boolean markViewBlindNgramsWithLocalView

KEYWORD_NGRAM_FIELD1

public static final String KEYWORD_NGRAM_FIELD1

See Also:

Constant Field Values

KEYWORD_NGRAM_FIELD2

public static final String KEYWORD_NGRAM_FIELD2

See Also:

Constant Field Values

Constructor Detail

LuceneKeywordPFE

public LuceneKeywordPFE()

Method Detail

getMetaCollectorClasses

public List<MetaCollectorConfiguration> getMetaCollectorClasses(Map<String,Object> parameterSettings)
                                                         throws org.apache.uima.resource.ResourceInitializationException

Specified by:

getMetaCollectorClasses in interface MetaDependent

Throws:

org.apache.uima.resource.ResourceInitializationException

initialize

public boolean initialize(org.apache.uima.resource.ResourceSpecifier aSpecifier,
                          Map<String,Object> aAdditionalParams)
                   throws org.apache.uima.resource.ResourceInitializationException

Specified by:

initialize in interface org.apache.uima.resource.Resource

Overrides:

initialize in class LucenePFEBase

Throws:

org.apache.uima.resource.ResourceInitializationException

extract

public Set<Feature> extract(org.apache.uima.jcas.JCas view1,
                            org.apache.uima.jcas.JCas view2)
                     throws org.dkpro.tc.api.exception.TextClassificationException

Specified by:

extract in interface PairFeatureExtractor

Throws:

org.dkpro.tc.api.exception.TextClassificationException

getViewNgrams

protected de.tudarmstadt.ukp.dkpro.core.api.frequency.util.FrequencyDistribution<String> getViewNgrams(org.apache.uima.jcas.JCas view1,
                                                                                                       org.apache.uima.jcas.JCas view2)

getFieldName

protected String getFieldName()

Specified by:

getFieldName in class LuceneFeatureExtractorBase

getTopN

protected int getTopN()

Specified by:

getTopN in class LuceneFeatureExtractorBase

getFeaturePrefix

protected String getFeaturePrefix()

Specified by:

getFeaturePrefix in class NGramFeatureExtractorBase