org.deeplearning4j.spark.impl.multilayer

Class SparkDl4jMultiLayer

java.lang.Object

org.deeplearning4j.spark.impl.multilayer.SparkDl4jMultiLayer

All Implemented Interfaces:

Serializable

public class SparkDl4jMultiLayer
extends Object
implements Serializable

Master class for spark

Author:

Adam Gibson

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
static String	ACCUM_GRADIENT
static String	AVERAGE_EACH_ITERATION
static int	DEFAULT_EVAL_SCORE_BATCH_SIZE
static String	DIVIDE_ACCUM_GRADIENT

Constructor Summary

Constructors

Constructor and Description
SparkDl4jMultiLayer(org.apache.spark.api.java.JavaSparkContext sc, MultiLayerConfiguration conf) Training constructor.
SparkDl4jMultiLayer(org.apache.spark.api.java.JavaSparkContext javaSparkContext, MultiLayerNetwork network)
SparkDl4jMultiLayer(org.apache.spark.SparkContext sparkContext, MultiLayerConfiguration conf) Training constructor.
SparkDl4jMultiLayer(org.apache.spark.SparkContext sparkContext, MultiLayerNetwork network) Instantiate a multi layer spark instance with the given context and network.

Method Summary

Modifier and Type	Method and Description
double	calculateScore(org.apache.spark.api.java.JavaRDD<org.nd4j.linalg.dataset.DataSet> data, boolean average)
Evaluation	evaluate(org.apache.spark.api.java.JavaRDD<org.nd4j.linalg.dataset.DataSet> data) Evaluate the network (classification performance) in a distributed manner on the provided data
Evaluation	evaluate(org.apache.spark.api.java.JavaRDD<org.nd4j.linalg.dataset.DataSet> data, List<String> labelsList) Evaluate the network (classification performance) in a distributed manner, using default batch size and a provided list of labels
Evaluation	evaluate(org.apache.spark.api.java.JavaRDD<org.nd4j.linalg.dataset.DataSet> data, List<String> labelsList, int evalBatchSize) Evaluate the network (classification performance) in a distributed manner, using specified batch size and a provided list of labels
MultiLayerNetwork	fit(org.apache.spark.api.java.JavaRDD<org.apache.spark.mllib.regression.LabeledPoint> rdd, int batchSize) Fit the given rdd given the context.
MultiLayerNetwork	fit(org.apache.spark.api.java.JavaSparkContext sc, org.apache.spark.api.java.JavaRDD<org.apache.spark.mllib.regression.LabeledPoint> rdd) Fit the given rdd given the context.
MultiLayerNetwork	fit(String path, int labelIndex, org.canova.api.records.reader.RecordReader recordReader) Train a multi layer network based on data loaded from a text file + RecordReader.
MultiLayerNetwork	fit(String path, int labelIndex, org.canova.api.records.reader.RecordReader recordReader, int examplesPerFit, int numPartitions) Train a multi layer network based on data loaded from a text file + RecordReader.
MultiLayerNetwork	fit(String path, int labelIndex, org.canova.api.records.reader.RecordReader recordReader, int examplesPerFit, int totalExamples, int numPartitions) Train a multi layer network based on data loaded from a text file + RecordReader.
MultiLayerNetwork	fitDataSet(org.apache.spark.api.java.JavaRDD<org.nd4j.linalg.dataset.DataSet> rdd) Fit the dataset rdd
MultiLayerNetwork	fitDataSet(org.apache.spark.api.java.JavaRDD<org.nd4j.linalg.dataset.DataSet> rdd, int examplesPerFit, int numPartitions) Equivalent to fitDataSet(JavaRDD, int, int, int), but persist and count the size of the data set first, instead of requiring the data set size to be provided externally.
MultiLayerNetwork	fitDataSet(org.apache.spark.api.java.JavaRDD<org.nd4j.linalg.dataset.DataSet> rdd, int examplesPerFit, int totalExamples, int numPartitions) Fit the data, splitting into smaller data subsets if necessary.
MultiLayerNetwork	getNetwork()
double	getScore() Gets the last (average) minibatch score from calling fit
protected void	invokeListeners(MultiLayerNetwork network, int iteration)
org.apache.spark.mllib.linalg.Matrix	predict(org.apache.spark.mllib.linalg.Matrix features) Predict the given feature matrix
org.apache.spark.mllib.linalg.Vector	predict(org.apache.spark.mllib.linalg.Vector point) Predict the given vector
protected void	runIteration(org.apache.spark.api.java.JavaRDD<org.nd4j.linalg.dataset.DataSet> rdd)
<K> org.apache.spark.api.java.JavaPairRDD<K,Double>	scoreExamples(org.apache.spark.api.java.JavaPairRDD<K,org.nd4j.linalg.dataset.DataSet> data, boolean includeRegularizationTerms) Score the examples individually, using the default batch size DEFAULT_EVAL_SCORE_BATCH_SIZE.
<K> org.apache.spark.api.java.JavaPairRDD<K,Double>	scoreExamples(org.apache.spark.api.java.JavaPairRDD<K,org.nd4j.linalg.dataset.DataSet> data, boolean includeRegularizationTerms, int batchSize) Score the examples individually, using a specified batch size.
org.apache.spark.api.java.JavaDoubleRDD	scoreExamples(org.apache.spark.api.java.JavaRDD<org.nd4j.linalg.dataset.DataSet> data, boolean includeRegularizationTerms) Score the examples individually, using the default batch size DEFAULT_EVAL_SCORE_BATCH_SIZE.
org.apache.spark.api.java.JavaDoubleRDD	scoreExamples(org.apache.spark.api.java.JavaRDD<org.nd4j.linalg.dataset.DataSet> data, boolean includeRegularizationTerms, int batchSize) Score the examples individually, using a specified batch size.
void	setListeners(Collection<IterationListener> listeners) This method allows you to specify IterationListeners for this model.
void	setNetwork(MultiLayerNetwork network)
static MultiLayerNetwork	train(org.apache.spark.api.java.JavaRDD<org.apache.spark.mllib.regression.LabeledPoint> data, MultiLayerConfiguration conf) Train a multi layer network

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

DEFAULT_EVAL_SCORE_BATCH_SIZE

public static final int DEFAULT_EVAL_SCORE_BATCH_SIZE

See Also:

Constant Field Values

AVERAGE_EACH_ITERATION

public static final String AVERAGE_EACH_ITERATION

See Also:

Constant Field Values

ACCUM_GRADIENT

public static final String ACCUM_GRADIENT

See Also:

Constant Field Values

DIVIDE_ACCUM_GRADIENT

public static final String DIVIDE_ACCUM_GRADIENT

See Also:

Constant Field Values

Constructor Detail

SparkDl4jMultiLayer

public SparkDl4jMultiLayer(org.apache.spark.SparkContext sparkContext,
                           MultiLayerNetwork network)

Instantiate a multi layer spark instance with the given context and network. This is the prediction constructor

Parameters:

sparkContext - the spark context to use

network - the network to use

SparkDl4jMultiLayer

public SparkDl4jMultiLayer(org.apache.spark.api.java.JavaSparkContext javaSparkContext,
                           MultiLayerNetwork network)

SparkDl4jMultiLayer

public SparkDl4jMultiLayer(org.apache.spark.SparkContext sparkContext,
                           MultiLayerConfiguration conf)

Training constructor. Instantiate with a configuration

Parameters:

sparkContext - the spark context to use

conf - the configuration of the network

SparkDl4jMultiLayer

public SparkDl4jMultiLayer(org.apache.spark.api.java.JavaSparkContext sc,
                           MultiLayerConfiguration conf)

Training constructor. Instantiate with a configuration

Parameters:

sc - the spark context to use

conf - the configuration of the network

Method Detail

fit

public MultiLayerNetwork fit(String path,
                             int labelIndex,
                             org.canova.api.records.reader.RecordReader recordReader)

Train a multi layer network based on data loaded from a text file + RecordReader. This method splits the entire data set at once

Parameters:

path - the path to the text file

labelIndex - the label index

recordReader - the record reader to parse results

Returns:

MultiLayerNetwork

See Also:

fit(String, int, RecordReader, int, int, int)

fit

public MultiLayerNetwork fit(String path,
                             int labelIndex,
                             org.canova.api.records.reader.RecordReader recordReader,
                             int examplesPerFit,
                             int numPartitions)

Train a multi layer network based on data loaded from a text file + RecordReader. This method splits the data into approximately examplesPerFit sized splits, and trains on each split. one after the other. See fitDataSet(JavaRDD, int, int, int) for further details.
Note: Compared to fit(String, int, RecordReader, int, int, int), this method persists and then counts the data set size directly. This is usually OK, though if the data set does not fit in memory, this can result in some overhead due to the data being loaded multiple times (once for count, once for fitting), as compared to providing the data set size to the fit(String, int, RecordReader, int, int, int) method

Parameters:

path - the path to the text file

labelIndex - the label index

recordReader - the record reader to parse results

examplesPerFit - Number of examples to fit on at each iteration

numPartitions - Number of partitions to divide each subset of the data into (for best results, this should be equal to the number of executors)

Returns:

MultiLayerNetwork

fit

public MultiLayerNetwork fit(String path,
                             int labelIndex,
                             org.canova.api.records.reader.RecordReader recordReader,
                             int examplesPerFit,
                             int totalExamples,
                             int numPartitions)

Train a multi layer network based on data loaded from a text file + RecordReader. This method splits the data into approximately examplesPerFit sized splits, and trains on each split. one after the other. See fitDataSet(JavaRDD, int, int, int) for further details.

Parameters:

path - the path to the text file

labelIndex - the label index

recordReader - the record reader to parse results

examplesPerFit - Number of examples to fit on at each iteration (divided between all executors)

numPartitions - Number of partitions to divide each subset of the data into (for best results, this should be equal to the number of executors)

Returns:

MultiLayerNetwork

See Also:

fit(String, int, RecordReader, int, int)

getNetwork

public MultiLayerNetwork getNetwork()

setNetwork

public void setNetwork(MultiLayerNetwork network)

predict

public org.apache.spark.mllib.linalg.Matrix predict(org.apache.spark.mllib.linalg.Matrix features)

Predict the given feature matrix

Parameters:

features - the given feature matrix

Returns:

the predictions

predict

public org.apache.spark.mllib.linalg.Vector predict(org.apache.spark.mllib.linalg.Vector point)

Predict the given vector

Parameters:

point - the vector to predict

Returns:

the predicted vector

fit

public MultiLayerNetwork fit(org.apache.spark.api.java.JavaRDD<org.apache.spark.mllib.regression.LabeledPoint> rdd,
                             int batchSize)

Fit the given rdd given the context. This will convert the labeled points to the internal dl4j format and train the model on that

Parameters:

rdd - the rdd to fitDataSet

Returns:

the multi layer network that was fitDataSet

fit

public MultiLayerNetwork fit(org.apache.spark.api.java.JavaSparkContext sc,
                             org.apache.spark.api.java.JavaRDD<org.apache.spark.mllib.regression.LabeledPoint> rdd)

Fit the given rdd given the context. This will convert the labeled points to the internal dl4j format and train the model on that

Parameters:

sc - the org.deeplearning4j.spark context

rdd - the rdd to fitDataSet

Returns:

the multi layer network that was fitDataSet

fitDataSet

public MultiLayerNetwork fitDataSet(org.apache.spark.api.java.JavaRDD<org.nd4j.linalg.dataset.DataSet> rdd,
                                    int examplesPerFit,
                                    int numPartitions)

Equivalent to fitDataSet(JavaRDD, int, int, int), but persist and count the size of the data set first, instead of requiring the data set size to be provided externally. Note: In some cases, it may be more efficient to count the size of the data set earlier in the pipeline and provide this count to the fitDataSet(JavaRDD, int, int, int) method, as counting on the JavaRDD<DataSet> requires a full pass of the data pipeline. In cases where the entire JavaRDD<DataSet> does not fit in memory, this approach can result in multiple passes being done over the data, potentially degrading performance

Parameters:

rdd - Data to train on

examplesPerFit - Number of examples to learn on (between averaging) across all executors. For example, if set to 1000 and rdd.count() == 10k, then we do 10 sets of learning, each on 1000 examples. To use all examples, set maxExamplesPerFit to Integer.MAX_VALUE

numPartitions - number of partitions to divide the data in to. For best results, this should be equal to the number of executors

Returns:

Trained network

fitDataSet

public MultiLayerNetwork fitDataSet(org.apache.spark.api.java.JavaRDD<org.nd4j.linalg.dataset.DataSet> rdd,
                                    int examplesPerFit,
                                    int totalExamples,
                                    int numPartitions)

Fit the data, splitting into smaller data subsets if necessary. This allows large JavaRDD<DataSet>s) to be trained as a set of smaller steps instead of all together.
Using this method, training progresses as follows:
train on examplesPerFit examples -> average parameters -> train on examplesPerFit -> average parameters etc until entire data set has been processed
Note: The actual number of splits for the input data is based on rounding up.
Suppose examplesPerFit=1000, with rdd.count()=1200. Then, we round up to 2000 examples, and the network will then be fit in two steps (as 2000/1000=2), with 1200/2=600 examples at each step. These 600 examples will then be distributed approximately equally (no guarantees) amongst each executor/core for training.

Parameters:

rdd - Data to train on

examplesPerFit - Number of examples to learn on (between averaging) across all executors. For example, if set to 1000 and rdd.count() == 10k, then we do 10 sets of learning, each on 1000 examples. To use all examples, set maxExamplesPerFit to Integer.MAX_VALUE

totalExamples - total number of examples in the data RDD

numPartitions - number of partitions to divide the data in to. For best results, this should be equal to the number of executors

Returns:

Trained network

fitDataSet

public MultiLayerNetwork fitDataSet(org.apache.spark.api.java.JavaRDD<org.nd4j.linalg.dataset.DataSet> rdd)

Fit the dataset rdd

Parameters:

rdd - the rdd to fitDataSet

Returns:

the multi layer network

runIteration

protected void runIteration(org.apache.spark.api.java.JavaRDD<org.nd4j.linalg.dataset.DataSet> rdd)

train

public static MultiLayerNetwork train(org.apache.spark.api.java.JavaRDD<org.apache.spark.mllib.regression.LabeledPoint> data,
                                      MultiLayerConfiguration conf)

Train a multi layer network

Parameters:

data - the data to train on

conf - the configuration of the network

Returns:

the fit multi layer network

setListeners

public void setListeners(@NonNull
                         Collection<IterationListener> listeners)

This method allows you to specify IterationListeners for this model. PLEASE NOTE: 1. These iteration listeners should be configured to use remote UiServer 2. Remote UiServer should be accessible via network from Spark master node.

Parameters:

listeners -

invokeListeners

protected void invokeListeners(MultiLayerNetwork network,
                               int iteration)

getScore

public double getScore()

Gets the last (average) minibatch score from calling fit

calculateScore

public double calculateScore(org.apache.spark.api.java.JavaRDD<org.nd4j.linalg.dataset.DataSet> data,
                             boolean average)

scoreExamples

public org.apache.spark.api.java.JavaDoubleRDD scoreExamples(org.apache.spark.api.java.JavaRDD<org.nd4j.linalg.dataset.DataSet> data,
                                                             boolean includeRegularizationTerms)

Score the examples individually, using the default batch size DEFAULT_EVAL_SCORE_BATCH_SIZE. Unlike calculateScore(JavaRDD, boolean), this method returns a score for each example separately. If scoring is needed for specific examples use either scoreExamples(JavaPairRDD, boolean) or scoreExamples(JavaPairRDD, boolean, int) which can have a key for each example.

Parameters:

data - Data to score

includeRegularizationTerms - If true: include the l1/l2 regularization terms with the score (if any)

Returns:

A JavaDoubleRDD containing the scores of each example

See Also:

MultiLayerNetwork.scoreExamples(DataSet, boolean)

scoreExamples

public org.apache.spark.api.java.JavaDoubleRDD scoreExamples(org.apache.spark.api.java.JavaRDD<org.nd4j.linalg.dataset.DataSet> data,
                                                             boolean includeRegularizationTerms,
                                                             int batchSize)

Score the examples individually, using a specified batch size. Unlike calculateScore(JavaRDD, boolean), this method returns a score for each example separately. If scoring is needed for specific examples use either scoreExamples(JavaPairRDD, boolean) or scoreExamples(JavaPairRDD, boolean, int) which can have a key for each example.

Parameters:

data - Data to score

includeRegularizationTerms - If true: include the l1/l2 regularization terms with the score (if any)

batchSize - Batch size to use when doing scoring

Returns:

A JavaDoubleRDD containing the scores of each example

See Also:

MultiLayerNetwork.scoreExamples(DataSet, boolean)

scoreExamples

public <K> org.apache.spark.api.java.JavaPairRDD<K,Double> scoreExamples(org.apache.spark.api.java.JavaPairRDD<K,org.nd4j.linalg.dataset.DataSet> data,
                                                                         boolean includeRegularizationTerms)

Score the examples individually, using the default batch size DEFAULT_EVAL_SCORE_BATCH_SIZE. Unlike calculateScore(JavaRDD, boolean), this method returns a score for each example separately
Note: The provided JavaPairRDD has a key that is associated with each example and returned score.
Note: The DataSet objects passed in must have exactly one example in them (otherwise: can't have a 1:1 association between keys and data sets to score)

Type Parameters:

K - Key type

Parameters:

data - Data to score

includeRegularizationTerms - If true: include the l1/l2 regularization terms with the score (if any)

Returns:

A JavaPairRDD<K,Double> containing the scores of each example

See Also:

MultiLayerNetwork.scoreExamples(DataSet, boolean)

scoreExamples

public <K> org.apache.spark.api.java.JavaPairRDD<K,Double> scoreExamples(org.apache.spark.api.java.JavaPairRDD<K,org.nd4j.linalg.dataset.DataSet> data,
                                                                         boolean includeRegularizationTerms,
                                                                         int batchSize)

Score the examples individually, using a specified batch size. Unlike calculateScore(JavaRDD, boolean), this method returns a score for each example separately
Note: The provided JavaPairRDD has a key that is associated with each example and returned score.
Note: The DataSet objects passed in must have exactly one example in them (otherwise: can't have a 1:1 association between keys and data sets to score)

Type Parameters:

K - Key type

Parameters:

data - Data to score

includeRegularizationTerms - If true: include the l1/l2 regularization terms with the score (if any)

Returns:

A JavaPairRDD<K,Double> containing the scores of each example

See Also:

MultiLayerNetwork.scoreExamples(DataSet, boolean)

evaluate

public Evaluation evaluate(org.apache.spark.api.java.JavaRDD<org.nd4j.linalg.dataset.DataSet> data)

Evaluate the network (classification performance) in a distributed manner on the provided data

Parameters:

data - Data to evaluate on

Returns:

Evaluation object; results of evaluation on all examples in the data set

evaluate

public Evaluation evaluate(org.apache.spark.api.java.JavaRDD<org.nd4j.linalg.dataset.DataSet> data,
                           List<String> labelsList)

Evaluate the network (classification performance) in a distributed manner, using default batch size and a provided list of labels

Parameters:

data - Data to evaluate on

labelsList - List of labels used for evaluation

Returns:

Evaluation object; results of evaluation on all examples in the data set

evaluate

public Evaluation evaluate(org.apache.spark.api.java.JavaRDD<org.nd4j.linalg.dataset.DataSet> data,
                           List<String> labelsList,
                           int evalBatchSize)

Evaluate the network (classification performance) in a distributed manner, using specified batch size and a provided list of labels

Parameters:

data - Data to evaluate on

labelsList - List of labels used for evaluation

evalBatchSize - Batch size to use when conducting evaluations

Returns:

Evaluation object; results of evaluation on all examples in the data set