org.deeplearning4j.spark.impl.multilayer.evaluation

Class EvaluateFlatMapFunction

java.lang.Object

org.deeplearning4j.spark.impl.multilayer.evaluation.EvaluateFlatMapFunction

All Implemented Interfaces:

Serializable, org.apache.spark.api.java.function.FlatMapFunction<Iterator<org.nd4j.linalg.dataset.DataSet>,Evaluation>

public class EvaluateFlatMapFunction
extends Object
implements org.apache.spark.api.java.function.FlatMapFunction<Iterator<org.nd4j.linalg.dataset.DataSet>,Evaluation>

Function to evaluate data (classification), in a distributed manner Flat map function used to batch examples for computational efficiency + reduce number of Evaluation objects returned for network efficiency.

Author:

Alex Black

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
protected int	evalBatchSize
protected org.apache.spark.broadcast.Broadcast<String>	json
protected org.apache.spark.broadcast.Broadcast<List<String>>	labels
protected static org.slf4j.Logger	log
protected org.apache.spark.broadcast.Broadcast<org.nd4j.linalg.api.ndarray.INDArray>	params

Constructor Summary

Constructors

Constructor and Description
EvaluateFlatMapFunction(org.apache.spark.broadcast.Broadcast<String> json, org.apache.spark.broadcast.Broadcast<org.nd4j.linalg.api.ndarray.INDArray> params, int evalBatchSize, org.apache.spark.broadcast.Broadcast<List<String>> labels)

Method Summary

Modifier and Type	Method and Description
Iterable<Evaluation>	call(Iterator<org.nd4j.linalg.dataset.DataSet> dataSetIterator)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

log

protected static org.slf4j.Logger log

json

protected org.apache.spark.broadcast.Broadcast<String> json

params

protected org.apache.spark.broadcast.Broadcast<org.nd4j.linalg.api.ndarray.INDArray> params

labels

protected org.apache.spark.broadcast.Broadcast<List<String>> labels

evalBatchSize

protected int evalBatchSize

Constructor Detail

EvaluateFlatMapFunction

public EvaluateFlatMapFunction(org.apache.spark.broadcast.Broadcast<String> json,
                               org.apache.spark.broadcast.Broadcast<org.nd4j.linalg.api.ndarray.INDArray> params,
                               int evalBatchSize,
                               org.apache.spark.broadcast.Broadcast<List<String>> labels)

Parameters:

json - Network configuration (json format)

params - Network parameters

evalBatchSize - Max examples per evaluation. Do multiple separate forward passes if data exceeds this. Used to avoid doing too many at once (and hence memory issues)

labels - list of string labels

Method Detail

call

public Iterable<Evaluation> call(Iterator<org.nd4j.linalg.dataset.DataSet> dataSetIterator)
                          throws Exception

Specified by:

call in interface org.apache.spark.api.java.function.FlatMapFunction<Iterator<org.nd4j.linalg.dataset.DataSet>,Evaluation>

Throws:

Exception