org.deeplearning4j.spark.canova

Class CanovaDataSetFunction

java.lang.Object

org.deeplearning4j.spark.canova.CanovaDataSetFunction

All Implemented Interfaces:

Serializable, org.apache.spark.api.java.function.Function<Collection<org.canova.api.writable.Writable>,org.nd4j.linalg.dataset.DataSet>

public class CanovaDataSetFunction
extends Object
implements org.apache.spark.api.java.function.Function<Collection<org.canova.api.writable.Writable>,org.nd4j.linalg.dataset.DataSet>, Serializable

Map Collection<Writable> objects (out of a canova-spark record reader function) to DataSet objects for Spark training. Analogous to RecordReaderDataSetIterator, but in the context of Spark.

Author:

Alex Black

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
protected int	batchSize

Constructor Summary

Constructors

Constructor and Description
CanovaDataSetFunction(int labelIndex, int numPossibleLabels, boolean regression)
CanovaDataSetFunction(int labelIndex, int numPossibleLabels, boolean regression, org.nd4j.linalg.dataset.api.DataSetPreProcessor preProcessor, org.canova.api.io.WritableConverter converter)

Method Summary

Modifier and Type	Method and Description
org.nd4j.linalg.dataset.DataSet	call(Collection<org.canova.api.writable.Writable> writables)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

batchSize

protected int batchSize

Constructor Detail

CanovaDataSetFunction

public CanovaDataSetFunction(int labelIndex,
                             int numPossibleLabels,
                             boolean regression)

CanovaDataSetFunction

public CanovaDataSetFunction(int labelIndex,
                             int numPossibleLabels,
                             boolean regression,
                             org.nd4j.linalg.dataset.api.DataSetPreProcessor preProcessor,
                             org.canova.api.io.WritableConverter converter)

Parameters:

labelIndex - Index of the label column

numPossibleLabels - Number of classes for classification (not used if regression = true)

regression - False for classification, true for regression

preProcessor - DataSetPreprocessor (may be null)

converter - WritableConverter (may be null)

Method Detail

call

public org.nd4j.linalg.dataset.DataSet call(Collection<org.canova.api.writable.Writable> writables)
                                     throws Exception

Specified by:

call in interface org.apache.spark.api.java.function.Function<Collection<org.canova.api.writable.Writable>,org.nd4j.linalg.dataset.DataSet>

Throws:

Exception