org.deeplearning4j.spark.impl.multilayer

Class IterativeReduceFlatMap

java.lang.Object

org.deeplearning4j.spark.impl.multilayer.IterativeReduceFlatMap

All Implemented Interfaces:

Serializable, org.apache.spark.api.java.function.FlatMapFunction<Iterator<org.nd4j.linalg.dataset.DataSet>,scala.Tuple3<org.nd4j.linalg.api.ndarray.INDArray,Updater,ScoreReport>>

public class IterativeReduceFlatMap
extends Object
implements org.apache.spark.api.java.function.FlatMapFunction<Iterator<org.nd4j.linalg.dataset.DataSet>,scala.Tuple3<org.nd4j.linalg.api.ndarray.INDArray,Updater,ScoreReport>>

Iterative reduce with flat map using map partitions

Author:

Adam Gibson

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
protected org.apache.spark.Accumulator<Double>	best_score_acc
protected String	json
protected static org.slf4j.Logger	log
protected org.apache.spark.broadcast.Broadcast<org.nd4j.linalg.api.ndarray.INDArray>	params
protected org.apache.spark.broadcast.Broadcast<Updater>	updater

Constructor Summary

Constructors

Constructor and Description

IterativeReduceFlatMap(String json, org.apache.spark.broadcast.Broadcast<org.nd4j.linalg.api.ndarray.INDArray> params, org.apache.spark.broadcast.Broadcast<Updater> updater, org.apache.spark.Accumulator<Double> bestScoreAcc) Pass in network configuration as json, broadcast parameters, broadcast updater and the bestScore to fit mapped data set and update and return parameters, updater and best score.

Method Summary

Modifier and Type	Method and Description
Iterable<scala.Tuple3<org.nd4j.linalg.api.ndarray.INDArray,Updater,ScoreReport>>	call(Iterator<org.nd4j.linalg.dataset.DataSet> dataSetIterator)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

log

protected static org.slf4j.Logger log

json

protected String json

params

protected org.apache.spark.broadcast.Broadcast<org.nd4j.linalg.api.ndarray.INDArray> params

updater

protected org.apache.spark.broadcast.Broadcast<Updater> updater

best_score_acc

protected final org.apache.spark.Accumulator<Double> best_score_acc

Constructor Detail

IterativeReduceFlatMap

public IterativeReduceFlatMap(String json,
                              org.apache.spark.broadcast.Broadcast<org.nd4j.linalg.api.ndarray.INDArray> params,
                              org.apache.spark.broadcast.Broadcast<Updater> updater,
                              org.apache.spark.Accumulator<Double> bestScoreAcc)

Pass in network configuration as json, broadcast parameters, broadcast updater and the bestScore to fit mapped data set and update and return parameters, updater and best score.

Parameters:

json - newtork string configuration

params - broadcasted parameters to reload into network

updater - broadcasted updaters to reload into network

bestScoreAcc - accumulator which tracks best score seen

Method Detail

call

public Iterable<scala.Tuple3<org.nd4j.linalg.api.ndarray.INDArray,Updater,ScoreReport>> call(Iterator<org.nd4j.linalg.dataset.DataSet> dataSetIterator)
                                                                                      throws Exception

Specified by:

call in interface org.apache.spark.api.java.function.FlatMapFunction<Iterator<org.nd4j.linalg.dataset.DataSet>,scala.Tuple3<org.nd4j.linalg.api.ndarray.INDArray,Updater,ScoreReport>>

Throws:

Exception