org.dkpro.tc.features.pair.core.ngram

Class LuceneNGramPFE

java.lang.Object

org.apache.uima.resource.Resource_ImplBase

org.apache.uima.fit.component.Resource_ImplBase

org.dkpro.tc.api.features.FeatureExtractorResource_ImplBase

org.dkpro.tc.features.ngram.base.NGramFeatureExtractorBase

org.dkpro.tc.features.ngram.base.LuceneFeatureExtractorBase

org.dkpro.tc.features.pair.core.ngram.LuceneNGramPFE

All Implemented Interfaces:

org.apache.uima.fit.component.ExternalResourceAware, org.apache.uima.resource.Resource, MetaDependent, PairFeatureExtractor

Direct Known Subclasses:

LuceneNGramCPFE

public class LuceneNGramPFE
extends LuceneFeatureExtractorBase
implements PairFeatureExtractor

Pair ngram feature extractor for document pair classification. Can be used to extract ngrams from one or both documents in the pair, and parameters for each document (view 1's, view 2's) can be set separately, or both documents can be treated together as one extended document.
Note that ngram features created by this class are each from a single document, i.e., not combinations of ngrams from the pair of documents. To make combinations of ngrams across both documents, please use LuceneNGramCPFE.

Field Summary

Fields

Modifier and Type	Field and Description
static String	LUCENE_NGRAM_FIELD1
static String	LUCENE_NGRAM_FIELD2
protected boolean	markViewBlindNgramsWithLocalView
protected boolean	ngramBinaryFeatureValuesCombos
protected int	ngramMaxN1
protected int	ngramMaxN2
protected int	ngramMinN1
protected int	ngramMinN2
protected int	ngramUseTopK1
protected int	ngramUseTopK2
static String	PARAM_MARK_VIEWBLIND_NGRAMS_WITH_LOCAL_VIEW This option collects a FrequencyDistribution of ngrams across both documents of all pairs, but when writing features, the view where a particular ngram is found is recorded with the ngram.
static String	PARAM_NGRAM_BINARY_FEATURE_VALUES_COMBO Whether features should be marked with binary (occurs, doesn't occur in this document pair) values, versus the document count of the feature.
static String	PARAM_NGRAM_MAX_N_VIEW1 Maximum size n of ngrams from View 1's.
static String	PARAM_NGRAM_MAX_N_VIEW2 Maximum size n of ngrams from View 2's.
static String	PARAM_NGRAM_MIN_N_VIEW1 Minimum size n of ngrams from View 1's.
static String	PARAM_NGRAM_MIN_N_VIEW2 Minimum size n of ngrams from View 2's.
static String	PARAM_NGRAM_USE_TOP_K_VIEW1 Use this number of most frequent ngrams from View 1's.
static String	PARAM_NGRAM_USE_TOP_K_VIEW2 Use this number of most frequent ngrams from View 2's.
static String	PARAM_USE_VIEW1_NGRAMS_AS_FEATURES Each ngram from View 1 documents added to the document pair instance as a feature.
static String	PARAM_USE_VIEW2_NGRAMS_AS_FEATURES Each ngram from View 1 documents added to the document pair instance as a feature.
static String	PARAM_USE_VIEWBLIND_NGRAMS_AS_FEATURES All qualifying ngrams from anywhere in either document are used as features.
protected de.tudarmstadt.ukp.dkpro.core.api.frequency.util.FrequencyDistribution<String>	topKSetView1
protected de.tudarmstadt.ukp.dkpro.core.api.frequency.util.FrequencyDistribution<String>	topKSetView2
protected boolean	useView1NgramsAsFeatures
protected boolean	useView2NgramsAsFeatures
protected boolean	useViewBlindNgramsAsFeatures

Fields inherited from class org.dkpro.tc.features.ngram.base.LuceneFeatureExtractorBase

forceRereadFromIndex, LUCENE_NGRAM_FIELD, luceneDir, PARAM_SOURCE_LOCATION

Fields inherited from class org.dkpro.tc.features.ngram.base.NGramFeatureExtractorBase

dfStore, filterPartialStopwordMatches, ngramFreqThreshold, ngramLowerCase, ngramMaxN, ngramMinN, ngramStopwordsFile, ngramUseTopK, PARAM_FILTER_PARTIAL_STOPWORD_MATCHES, PARAM_NGRAM_FREQ_THRESHOLD, PARAM_NGRAM_LOWER_CASE, PARAM_NGRAM_MAX_N, PARAM_NGRAM_MIN_N, PARAM_NGRAM_STOPWORDS_FILE, PARAM_NGRAM_USE_TOP_K, PARAM_TF_IDF_CALCULATION, prefix, stopwords, tfIdfCalculation, topKSet

Fields inherited from class org.dkpro.tc.api.features.FeatureExtractorResource_ImplBase

featureExtractorName, PARAM_UNIQUE_EXTRACTOR_NAME

Fields inherited from interface org.apache.uima.resource.Resource

PARAM_AGGREGATE_SOFA_MAPPINGS, PARAM_CONFIG_MANAGER, PARAM_CONFIG_PARAM_SETTINGS, PARAM_EXTERNAL_OVERRIDE_SETTINGS, PARAM_PERFORMANCE_TUNING_SETTINGS, PARAM_RESOURCE_MANAGER, PARAM_UIMA_CONTEXT

Constructor Summary

Constructors

Constructor and Description
LuceneNGramPFE()

Method Summary

Modifier and Type	Method and Description
protected Set<Feature>	addToFeatureArray(de.tudarmstadt.ukp.dkpro.core.api.frequency.util.FrequencyDistribution<String> viewNgrams, de.tudarmstadt.ukp.dkpro.core.api.frequency.util.FrequencyDistribution<String> topKSet, Set<Feature> features)
Set<Feature>	extract(org.apache.uima.jcas.JCas view1, org.apache.uima.jcas.JCas view2)
protected String	getFeaturePrefix()
protected String	getFieldName()
List<MetaCollectorConfiguration>	getMetaCollectorClasses(Map<String,Object> parameterSettings)
protected int	getTopN()
protected de.tudarmstadt.ukp.dkpro.core.api.frequency.util.FrequencyDistribution<String>	getTopNgrams()
protected de.tudarmstadt.ukp.dkpro.core.api.frequency.util.FrequencyDistribution<String>	getTopNgramsView1()
protected de.tudarmstadt.ukp.dkpro.core.api.frequency.util.FrequencyDistribution<String>	getTopNgramsView2()
protected de.tudarmstadt.ukp.dkpro.core.api.frequency.util.FrequencyDistribution<String>	getViewNgrams(org.apache.uima.jcas.JCas view1, org.apache.uima.jcas.JCas view2)
boolean	initialize(org.apache.uima.resource.ResourceSpecifier aSpecifier, Map<String,Object> aAdditionalParams)

Methods inherited from class org.dkpro.tc.features.ngram.base.LuceneFeatureExtractorBase

logSelectionProcess, passesScreening

Methods inherited from class org.apache.uima.fit.component.Resource_ImplBase

afterResourcesInitialized, getLogger, getResourceName

Methods inherited from class org.apache.uima.resource.Resource_ImplBase

destroy, getCasManager, getMetaData, getRelativePathResolver, getResourceManager, getUimaContext, getUimaContextAdmin, setLogger, setMetaData

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

PARAM_NGRAM_MIN_N_VIEW1

public static final String PARAM_NGRAM_MIN_N_VIEW1

Minimum size n of ngrams from View 1's.

See Also:

Constant Field Values

ngramMinN1

protected int ngramMinN1

PARAM_NGRAM_MIN_N_VIEW2

public static final String PARAM_NGRAM_MIN_N_VIEW2

Minimum size n of ngrams from View 2's.

See Also:

Constant Field Values

ngramMinN2

protected int ngramMinN2

PARAM_NGRAM_MAX_N_VIEW1

public static final String PARAM_NGRAM_MAX_N_VIEW1

Maximum size n of ngrams from View 1's.

See Also:

Constant Field Values

ngramMaxN1

protected int ngramMaxN1

PARAM_NGRAM_MAX_N_VIEW2

public static final String PARAM_NGRAM_MAX_N_VIEW2

Maximum size n of ngrams from View 2's.

See Also:

Constant Field Values

ngramMaxN2

protected int ngramMaxN2

PARAM_NGRAM_USE_TOP_K_VIEW1

public static final String PARAM_NGRAM_USE_TOP_K_VIEW1

Use this number of most frequent ngrams from View 1's.

See Also:

Constant Field Values

ngramUseTopK1

protected int ngramUseTopK1

PARAM_NGRAM_USE_TOP_K_VIEW2

public static final String PARAM_NGRAM_USE_TOP_K_VIEW2

Use this number of most frequent ngrams from View 2's.

See Also:

Constant Field Values

ngramUseTopK2

protected int ngramUseTopK2

PARAM_USE_VIEW1_NGRAMS_AS_FEATURES

public static final String PARAM_USE_VIEW1_NGRAMS_AS_FEATURES

Each ngram from View 1 documents added to the document pair instance as a feature. E.g. Feature: view1NG_Dear

See Also:

Constant Field Values

useView1NgramsAsFeatures

protected boolean useView1NgramsAsFeatures

PARAM_USE_VIEW2_NGRAMS_AS_FEATURES

public static final String PARAM_USE_VIEW2_NGRAMS_AS_FEATURES

Each ngram from View 1 documents added to the document pair instance as a feature. E.g. Feature: view2NG_Dear

See Also:

Constant Field Values

useView2NgramsAsFeatures

protected boolean useView2NgramsAsFeatures

PARAM_USE_VIEWBLIND_NGRAMS_AS_FEATURES

public static final String PARAM_USE_VIEWBLIND_NGRAMS_AS_FEATURES

All qualifying ngrams from anywhere in either document are used as features. Feature does not specify which view the ngram came from. E.g. Feature: allNG_Dear

See Also:

Constant Field Values

useViewBlindNgramsAsFeatures

protected boolean useViewBlindNgramsAsFeatures

PARAM_MARK_VIEWBLIND_NGRAMS_WITH_LOCAL_VIEW

public static final String PARAM_MARK_VIEWBLIND_NGRAMS_WITH_LOCAL_VIEW

This option collects a FrequencyDistribution of ngrams across both documents of all pairs, but when writing features, the view where a particular ngram is found is recorded with the ngram. For example, using a PARAM_NGRAM_USE_TOP_K_ALL value of 500, 400 of the ngrams in the top 500 might happen to be from View 2's; and whenever an ngram from the 500 is seen in any document, view 1 or 2, the document's view is recorded.
E.g., Feature: view2allNG_Dear
In order to use this option, PARAM_USE_VIEWBLIND_NGRAMS_AS_FEATURES must also be set to true.

See Also:

Constant Field Values

markViewBlindNgramsWithLocalView

protected boolean markViewBlindNgramsWithLocalView

PARAM_NGRAM_BINARY_FEATURE_VALUES_COMBO

public static final String PARAM_NGRAM_BINARY_FEATURE_VALUES_COMBO

Whether features should be marked with binary (occurs, doesn't occur in this document pair) values, versus the document count of the feature. In combo ngrams this is (doc1freq * doc2freq). Note this only applies to feature values; frequency selection of features is based on frequency across documents, not within documents.

See Also:

Constant Field Values

ngramBinaryFeatureValuesCombos

protected boolean ngramBinaryFeatureValuesCombos

LUCENE_NGRAM_FIELD1

public static final String LUCENE_NGRAM_FIELD1

See Also:

Constant Field Values

LUCENE_NGRAM_FIELD2

public static final String LUCENE_NGRAM_FIELD2

See Also:

Constant Field Values

topKSetView1

protected de.tudarmstadt.ukp.dkpro.core.api.frequency.util.FrequencyDistribution<String> topKSetView1

topKSetView2

protected de.tudarmstadt.ukp.dkpro.core.api.frequency.util.FrequencyDistribution<String> topKSetView2

Constructor Detail

LuceneNGramPFE

public LuceneNGramPFE()

Method Detail

getMetaCollectorClasses

public List<MetaCollectorConfiguration> getMetaCollectorClasses(Map<String,Object> parameterSettings)
                                                         throws org.apache.uima.resource.ResourceInitializationException

Specified by:

getMetaCollectorClasses in interface MetaDependent

Throws:

org.apache.uima.resource.ResourceInitializationException

initialize

public boolean initialize(org.apache.uima.resource.ResourceSpecifier aSpecifier,
                          Map<String,Object> aAdditionalParams)
                   throws org.apache.uima.resource.ResourceInitializationException

Specified by:

initialize in interface org.apache.uima.resource.Resource

Overrides:

initialize in class NGramFeatureExtractorBase

Throws:

org.apache.uima.resource.ResourceInitializationException

extract

public Set<Feature> extract(org.apache.uima.jcas.JCas view1,
                            org.apache.uima.jcas.JCas view2)
                     throws org.dkpro.tc.api.exception.TextClassificationException

Specified by:

extract in interface PairFeatureExtractor

Throws:

org.dkpro.tc.api.exception.TextClassificationException

addToFeatureArray

protected Set<Feature> addToFeatureArray(de.tudarmstadt.ukp.dkpro.core.api.frequency.util.FrequencyDistribution<String> viewNgrams,
                                         de.tudarmstadt.ukp.dkpro.core.api.frequency.util.FrequencyDistribution<String> topKSet,
                                         Set<Feature> features)

getTopNgrams

protected de.tudarmstadt.ukp.dkpro.core.api.frequency.util.FrequencyDistribution<String> getTopNgrams()
                                                                                               throws org.apache.uima.resource.ResourceInitializationException

Overrides:

getTopNgrams in class LuceneFeatureExtractorBase

Throws:

org.apache.uima.resource.ResourceInitializationException

getTopNgramsView1

protected de.tudarmstadt.ukp.dkpro.core.api.frequency.util.FrequencyDistribution<String> getTopNgramsView1()
                                                                                                    throws org.apache.uima.resource.ResourceInitializationException

Throws:

org.apache.uima.resource.ResourceInitializationException

getTopNgramsView2

protected de.tudarmstadt.ukp.dkpro.core.api.frequency.util.FrequencyDistribution<String> getTopNgramsView2()
                                                                                                    throws org.apache.uima.resource.ResourceInitializationException

Throws:

org.apache.uima.resource.ResourceInitializationException

getViewNgrams

protected de.tudarmstadt.ukp.dkpro.core.api.frequency.util.FrequencyDistribution<String> getViewNgrams(org.apache.uima.jcas.JCas view1,
                                                                                                       org.apache.uima.jcas.JCas view2)
                                                                                                throws org.dkpro.tc.api.exception.TextClassificationException

Throws:

org.dkpro.tc.api.exception.TextClassificationException

getFieldName

protected String getFieldName()

Specified by:

getFieldName in class LuceneFeatureExtractorBase

getTopN

protected int getTopN()

Specified by:

getTopN in class LuceneFeatureExtractorBase

getFeaturePrefix

protected String getFeaturePrefix()

Specified by:

getFeaturePrefix in class NGramFeatureExtractorBase