org.deeplearning4j.spark.canova

Class CanovaSequencePairDataSetFunction

java.lang.Object

org.deeplearning4j.spark.canova.CanovaSequencePairDataSetFunction

All Implemented Interfaces:

Serializable, org.apache.spark.api.java.function.Function<scala.Tuple2<Collection<Collection<org.canova.api.writable.Writable>>,Collection<Collection<org.canova.api.writable.Writable>>>,org.nd4j.linalg.dataset.DataSet>

public class CanovaSequencePairDataSetFunction
extends Object
implements org.apache.spark.api.java.function.Function<scala.Tuple2<Collection<Collection<org.canova.api.writable.Writable>>,Collection<Collection<org.canova.api.writable.Writable>>>,org.nd4j.linalg.dataset.DataSet>, Serializable

Map Tuple2<Collection<Collection<Writable>>,Collection<Collection<Writable>> objects (out of a TWO canova-spark sequence record reader functions) to DataSet objects for Spark training. Analogous to SequenceRecordReaderDataSetIterator, but in the context of Spark. Supports loading data from a TWO sources only; hence supports many-to-one and one-to-many situations. see CanovaSequenceDataSetFunction for the single file version

Author:

Alex Black

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	CanovaSequencePairDataSetFunction.AlignmentMode Alignment mode for dealing with input/labels of differing lengths (for example, one-to-many and many-to-one type situations).

Constructor Summary

Constructors

Constructor and Description
CanovaSequencePairDataSetFunction() Constructor for equal length and no conversion of labels (i.e., regression or already in one-hot representation).
CanovaSequencePairDataSetFunction(int numPossibleLabels, boolean regression) Constructor for equal length, no data set preprocessor or writable converter
CanovaSequencePairDataSetFunction(int numPossibleLabels, boolean regression, CanovaSequencePairDataSetFunction.AlignmentMode alignmentMode) Constructor for data with a specified alignment mode, no data set preprocessor or writable converter
CanovaSequencePairDataSetFunction(int numPossibleLabels, boolean regression, CanovaSequencePairDataSetFunction.AlignmentMode alignmentMode, org.nd4j.linalg.dataset.api.DataSetPreProcessor preProcessor, org.canova.api.io.WritableConverter converter)

Method Summary

Modifier and Type	Method and Description
org.nd4j.linalg.dataset.DataSet	call(scala.Tuple2<Collection<Collection<org.canova.api.writable.Writable>>,Collection<Collection<org.canova.api.writable.Writable>>> input)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

CanovaSequencePairDataSetFunction

public CanovaSequencePairDataSetFunction()

Constructor for equal length and no conversion of labels (i.e., regression or already in one-hot representation). No data set proprocessor or writable converter

CanovaSequencePairDataSetFunction

public CanovaSequencePairDataSetFunction(int numPossibleLabels,
                                         boolean regression)

Constructor for equal length, no data set preprocessor or writable converter

See Also:

CanovaSequencePairDataSetFunction(int, boolean, AlignmentMode, DataSetPreProcessor, WritableConverter)

CanovaSequencePairDataSetFunction

public CanovaSequencePairDataSetFunction(int numPossibleLabels,
                                         boolean regression,
                                         CanovaSequencePairDataSetFunction.AlignmentMode alignmentMode)

Constructor for data with a specified alignment mode, no data set preprocessor or writable converter

See Also:

CanovaSequencePairDataSetFunction(int, boolean, AlignmentMode, DataSetPreProcessor, WritableConverter)

CanovaSequencePairDataSetFunction

public CanovaSequencePairDataSetFunction(int numPossibleLabels,
                                         boolean regression,
                                         CanovaSequencePairDataSetFunction.AlignmentMode alignmentMode,
                                         org.nd4j.linalg.dataset.api.DataSetPreProcessor preProcessor,
                                         org.canova.api.io.WritableConverter converter)

Parameters:

numPossibleLabels - Number of classes for classification (not used if regression = true)

regression - False for classification, true for regression

alignmentMode - Alignment mode for data. See CanovaSequencePairDataSetFunction.AlignmentMode

preProcessor - DataSetPreprocessor (may be null)

converter - WritableConverter (may be null)

Method Detail

call

public org.nd4j.linalg.dataset.DataSet call(scala.Tuple2<Collection<Collection<org.canova.api.writable.Writable>>,Collection<Collection<org.canova.api.writable.Writable>>> input)
                                     throws Exception

Specified by:

call in interface org.apache.spark.api.java.function.Function<scala.Tuple2<Collection<Collection<org.canova.api.writable.Writable>>,Collection<Collection<org.canova.api.writable.Writable>>>,org.nd4j.linalg.dataset.DataSet>

Throws:

Exception