org.deeplearning4j.spark.impl.computationgraph

Class SparkComputationGraph

java.lang.Object

org.deeplearning4j.spark.impl.computationgraph.SparkComputationGraph

All Implemented Interfaces:

Serializable

public class SparkComputationGraph
extends Object
implements Serializable

Main class for training ComputationGraph networks using Spark

Author:

Alex Black

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
static String	ACCUM_GRADIENT
static String	AVERAGE_EACH_ITERATION
static int	DEFAULT_EVAL_SCORE_BATCH_SIZE
static String	DIVIDE_ACCUM_GRADIENT

Constructor Summary

Constructors

Constructor and Description
SparkComputationGraph(org.apache.spark.api.java.JavaSparkContext javaSparkContext, ComputationGraph network)
SparkComputationGraph(org.apache.spark.api.java.JavaSparkContext sparkContext, ComputationGraphConfiguration conf)
SparkComputationGraph(org.apache.spark.SparkContext sparkContext, ComputationGraph network) Instantiate a ComputationGraph instance with the given context and network.
SparkComputationGraph(org.apache.spark.SparkContext sparkContext, ComputationGraphConfiguration conf)

Method Summary

Modifier and Type	Method and Description
double	calculateScore(org.apache.spark.api.java.JavaRDD<org.nd4j.linalg.dataset.api.MultiDataSet> data, boolean average)
double	calculateScoreDataSet(org.apache.spark.api.java.JavaRDD<org.nd4j.linalg.dataset.DataSet> data, boolean average)
ComputationGraph	fit(String path, int labelIndex, org.canova.api.records.reader.RecordReader recordReader, int examplesPerFit, int totalExamples, int numPartitions) Train a ComputationGraph network based on data loaded from a text file + RecordReader.
ComputationGraph	fitDataSet(org.apache.spark.api.java.JavaRDD<org.nd4j.linalg.dataset.DataSet> rdd, int examplesPerFit, int totalExamples, int numPartitions) DataSet version of fitMultiDataSet(JavaRDD, int, int, int).
ComputationGraph	fitDataSet(org.apache.spark.api.java.JavaRDD<org.nd4j.linalg.dataset.api.MultiDataSet> rdd) Fit the dataset rdd
ComputationGraph	fitMultiDataSet(org.apache.spark.api.java.JavaRDD<org.nd4j.linalg.dataset.api.MultiDataSet> rdd, int examplesPerFit, int totalExamples, int numPartitions) Fit the data, splitting into smaller data subsets if necessary.
ComputationGraph	getNetwork()
double	getScore() Gets the last (average) minibatch score from calling fit
protected void	invokeListeners(ComputationGraph network, int iteration)
protected void	runIteration(org.apache.spark.api.java.JavaRDD<org.nd4j.linalg.dataset.api.MultiDataSet> rdd)
<K> org.apache.spark.api.java.JavaPairRDD<K,Double>	scoreExamples(org.apache.spark.api.java.JavaPairRDD<K,org.nd4j.linalg.dataset.api.MultiDataSet> data, boolean includeRegularizationTerms) Score the examples individually, using the default batch size DEFAULT_EVAL_SCORE_BATCH_SIZE.
<K> org.apache.spark.api.java.JavaPairRDD<K,Double>	scoreExamples(org.apache.spark.api.java.JavaPairRDD<K,org.nd4j.linalg.dataset.api.MultiDataSet> data, boolean includeRegularizationTerms, int batchSize) Score the examples individually, using a specified batch size.
org.apache.spark.api.java.JavaDoubleRDD	scoreExamples(org.apache.spark.api.java.JavaRDD<org.nd4j.linalg.dataset.api.MultiDataSet> data, boolean includeRegularizationTerms) Score the examples individually, using the default batch size DEFAULT_EVAL_SCORE_BATCH_SIZE.
org.apache.spark.api.java.JavaDoubleRDD	scoreExamples(org.apache.spark.api.java.JavaRDD<org.nd4j.linalg.dataset.api.MultiDataSet> data, boolean includeRegularizationTerms, int batchSize) Score the examples individually, using a specified batch size.
<K> org.apache.spark.api.java.JavaPairRDD<K,Double>	scoreExamplesDataSet(org.apache.spark.api.java.JavaPairRDD<K,org.nd4j.linalg.dataset.DataSet> data, boolean includeRegularizationTerms) DataSet version of scoreExamples(JavaPairRDD, boolean)
<K> org.apache.spark.api.java.JavaPairRDD<K,Double>	scoreExamplesDataSet(org.apache.spark.api.java.JavaPairRDD<K,org.nd4j.linalg.dataset.DataSet> data, boolean includeRegularizationTerms, int batchSize) DataSet version of scoreExamples(JavaPairRDD, boolean,int)
org.apache.spark.api.java.JavaDoubleRDD	scoreExamplesDataSet(org.apache.spark.api.java.JavaRDD<org.nd4j.linalg.dataset.DataSet> data, boolean includeRegularizationTerms) DataSet version of scoreExamples(JavaRDD, boolean)
org.apache.spark.api.java.JavaDoubleRDD	scoreExamplesDataSet(org.apache.spark.api.java.JavaRDD<org.nd4j.linalg.dataset.DataSet> data, boolean includeRegularizationTerms, int batchSize) DataSet version of scoreExamples(JavaPairRDD, boolean, int)
void	setListeners(Collection<IterationListener> listeners) This method allows you to specify IterationListeners for this model.
void	setNetwork(ComputationGraph network)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

DEFAULT_EVAL_SCORE_BATCH_SIZE

public static final int DEFAULT_EVAL_SCORE_BATCH_SIZE

See Also:

Constant Field Values

AVERAGE_EACH_ITERATION

public static final String AVERAGE_EACH_ITERATION

See Also:

Constant Field Values

ACCUM_GRADIENT

public static final String ACCUM_GRADIENT

See Also:

Constant Field Values

DIVIDE_ACCUM_GRADIENT

public static final String DIVIDE_ACCUM_GRADIENT

See Also:

Constant Field Values

Constructor Detail

SparkComputationGraph

public SparkComputationGraph(org.apache.spark.SparkContext sparkContext,
                             ComputationGraph network)

Instantiate a ComputationGraph instance with the given context and network.

Parameters:

sparkContext - the spark context to use

network - the network to use

SparkComputationGraph

public SparkComputationGraph(org.apache.spark.api.java.JavaSparkContext javaSparkContext,
                             ComputationGraph network)

SparkComputationGraph

public SparkComputationGraph(org.apache.spark.SparkContext sparkContext,
                             ComputationGraphConfiguration conf)

SparkComputationGraph

public SparkComputationGraph(org.apache.spark.api.java.JavaSparkContext sparkContext,
                             ComputationGraphConfiguration conf)

Method Detail

fit

public ComputationGraph fit(String path,
                            int labelIndex,
                            org.canova.api.records.reader.RecordReader recordReader,
                            int examplesPerFit,
                            int totalExamples,
                            int numPartitions)

Train a ComputationGraph network based on data loaded from a text file + RecordReader. This method splits the data into approximately examplesPerFit sized splits, and trains on each split. one after the other. See fitDataSet(JavaRDD, int, int, int) for further details.
NOTE: This method can only be used with ComputationGraph instances that have a single input and single output

Parameters:

path - the path to the text file

labelIndex - the label index

recordReader - the record reader to parse results

examplesPerFit - Number of examples to fit on at each iteration

totalExamples - total number of examples

numPartitions - Number of partitions. Usually set to number of executors

Returns:

MultiLayerNetwork

getNetwork

public ComputationGraph getNetwork()

setNetwork

public void setNetwork(ComputationGraph network)

fitMultiDataSet

public ComputationGraph fitMultiDataSet(org.apache.spark.api.java.JavaRDD<org.nd4j.linalg.dataset.api.MultiDataSet> rdd,
                                        int examplesPerFit,
                                        int totalExamples,
                                        int numPartitions)

Fit the data, splitting into smaller data subsets if necessary. This allows large JavaRDD<DataSet>s) to be trained as a set of smaller steps instead of all together.
Using this method, training progresses as follows:
train on examplesPerFit examples -> average parameters -> train on examplesPerFit -> average parameters etc until entire data set has been processed
Note: The actual number of splits for the input data is based on rounding up.
Suppose examplesPerFit=1000, with rdd.count()=1200. Then, we round up to 2000 examples, and the network will then be fit in two steps (as 2000/1000=2), with 1200/2=600 examples at each step. These 600 examples will then be distributed approximately equally (no guarantees) amongst each executor/core for training.

Parameters:

rdd - Data to train on

examplesPerFit - Number of examples to learn on (between averaging) across all executors. For example, if set to 1000 and rdd.count() == 10k, then we do 10 sets of learning, each on 1000 examples. To use all examples, set maxExamplesPerFit to Integer.MAX_VALUE

totalExamples - total number of examples in the data RDD

numPartitions - number of partitions to divide the data in to

Returns:

Trained network

fitDataSet

public ComputationGraph fitDataSet(org.apache.spark.api.java.JavaRDD<org.nd4j.linalg.dataset.DataSet> rdd,
                                   int examplesPerFit,
                                   int totalExamples,
                                   int numPartitions)

DataSet version of fitMultiDataSet(JavaRDD, int, int, int). Handles conversion from DataSet to MultiDataSet internally.

fitDataSet

public ComputationGraph fitDataSet(org.apache.spark.api.java.JavaRDD<org.nd4j.linalg.dataset.api.MultiDataSet> rdd)

Fit the dataset rdd

Parameters:

rdd - the rdd to fitDataSet

Returns:

the ComputationGraph after parameter averaging

runIteration

protected void runIteration(org.apache.spark.api.java.JavaRDD<org.nd4j.linalg.dataset.api.MultiDataSet> rdd)

setListeners

public void setListeners(@NonNull
                         Collection<IterationListener> listeners)

This method allows you to specify IterationListeners for this model. PLEASE NOTE: 1. These iteration listeners should be configured to use remote UiServer 2. Remote UiServer should be accessible via network from Spark master node.

Parameters:

listeners -

invokeListeners

protected void invokeListeners(ComputationGraph network,
                               int iteration)

getScore

public double getScore()

Gets the last (average) minibatch score from calling fit

calculateScoreDataSet

public double calculateScoreDataSet(org.apache.spark.api.java.JavaRDD<org.nd4j.linalg.dataset.DataSet> data,
                                    boolean average)

calculateScore

public double calculateScore(org.apache.spark.api.java.JavaRDD<org.nd4j.linalg.dataset.api.MultiDataSet> data,
                             boolean average)

scoreExamplesDataSet

public org.apache.spark.api.java.JavaDoubleRDD scoreExamplesDataSet(org.apache.spark.api.java.JavaRDD<org.nd4j.linalg.dataset.DataSet> data,
                                                                    boolean includeRegularizationTerms)

DataSet version of scoreExamples(JavaRDD, boolean)

scoreExamplesDataSet

public org.apache.spark.api.java.JavaDoubleRDD scoreExamplesDataSet(org.apache.spark.api.java.JavaRDD<org.nd4j.linalg.dataset.DataSet> data,
                                                                    boolean includeRegularizationTerms,
                                                                    int batchSize)

DataSet version of scoreExamples(JavaPairRDD, boolean, int)

scoreExamplesDataSet

public <K> org.apache.spark.api.java.JavaPairRDD<K,Double> scoreExamplesDataSet(org.apache.spark.api.java.JavaPairRDD<K,org.nd4j.linalg.dataset.DataSet> data,
                                                                                boolean includeRegularizationTerms)

DataSet version of scoreExamples(JavaPairRDD, boolean)

scoreExamplesDataSet

public <K> org.apache.spark.api.java.JavaPairRDD<K,Double> scoreExamplesDataSet(org.apache.spark.api.java.JavaPairRDD<K,org.nd4j.linalg.dataset.DataSet> data,
                                                                                boolean includeRegularizationTerms,
                                                                                int batchSize)

DataSet version of scoreExamples(JavaPairRDD, boolean,int)

scoreExamples

public org.apache.spark.api.java.JavaDoubleRDD scoreExamples(org.apache.spark.api.java.JavaRDD<org.nd4j.linalg.dataset.api.MultiDataSet> data,
                                                             boolean includeRegularizationTerms)

Score the examples individually, using the default batch size DEFAULT_EVAL_SCORE_BATCH_SIZE. Unlike calculateScore(JavaRDD, boolean), this method returns a score for each example separately. If scoring is needed for specific examples use either scoreExamples(JavaPairRDD, boolean) or scoreExamples(JavaPairRDD, boolean, int) which can have a key for each example.

Parameters:

data - Data to score

includeRegularizationTerms - If true: include the l1/l2 regularization terms with the score (if any)

Returns:

A JavaDoubleRDD containing the scores of each example

See Also:

ComputationGraph.scoreExamples(MultiDataSet, boolean)

scoreExamples

public org.apache.spark.api.java.JavaDoubleRDD scoreExamples(org.apache.spark.api.java.JavaRDD<org.nd4j.linalg.dataset.api.MultiDataSet> data,
                                                             boolean includeRegularizationTerms,
                                                             int batchSize)

Score the examples individually, using a specified batch size. Unlike calculateScore(JavaRDD, boolean), this method returns a score for each example separately. If scoring is needed for specific examples use either scoreExamples(JavaPairRDD, boolean) or scoreExamples(JavaPairRDD, boolean, int) which can have a key for each example.

Parameters:

data - Data to score

includeRegularizationTerms - If true: include the l1/l2 regularization terms with the score (if any)

batchSize - Batch size to use when doing scoring

Returns:

A JavaDoubleRDD containing the scores of each example

See Also:

ComputationGraph.scoreExamples(MultiDataSet, boolean)

scoreExamples

public <K> org.apache.spark.api.java.JavaPairRDD<K,Double> scoreExamples(org.apache.spark.api.java.JavaPairRDD<K,org.nd4j.linalg.dataset.api.MultiDataSet> data,
                                                                         boolean includeRegularizationTerms)

Score the examples individually, using the default batch size DEFAULT_EVAL_SCORE_BATCH_SIZE. Unlike calculateScore(JavaRDD, boolean), this method returns a score for each example separately
Note: The provided JavaPairRDD has a key that is associated with each example and returned score.
Note: The DataSet objects passed in must have exactly one example in them (otherwise: can't have a 1:1 association between keys and data sets to score)

Type Parameters:

K - Key type

Parameters:

data - Data to score

includeRegularizationTerms - If true: include the l1/l2 regularization terms with the score (if any)

Returns:

A JavaPairRDD<K,Double> containing the scores of each example

See Also:

MultiLayerNetwork.scoreExamples(DataSet, boolean)

scoreExamples

public <K> org.apache.spark.api.java.JavaPairRDD<K,Double> scoreExamples(org.apache.spark.api.java.JavaPairRDD<K,org.nd4j.linalg.dataset.api.MultiDataSet> data,
                                                                         boolean includeRegularizationTerms,
                                                                         int batchSize)

Score the examples individually, using a specified batch size. Unlike calculateScore(JavaRDD, boolean), this method returns a score for each example separately
Note: The provided JavaPairRDD has a key that is associated with each example and returned score.
Note: The DataSet objects passed in must have exactly one example in them (otherwise: can't have a 1:1 association between keys and data sets to score)

Type Parameters:

K - Key type

Parameters:

data - Data to score

includeRegularizationTerms - If true: include the l1/l2 regularization terms with the score (if any)

Returns:

A JavaPairRDD<K,Double> containing the scores of each example

See Also:

MultiLayerNetwork.scoreExamples(DataSet, boolean)