org.deeplearning4j.spark.util

Class MLLibUtil

java.lang.Object

org.deeplearning4j.spark.util.MLLibUtil

public class MLLibUtil
extends Object

Dl4j <----> MLLib

Author:

Adam Gibson

Constructor Summary

Constructors

Constructor and Description
MLLibUtil()

Method Summary

Modifier and Type	Method and Description
static org.apache.spark.api.java.JavaRDD<org.apache.spark.mllib.regression.LabeledPoint>	fromBinary(org.apache.spark.api.java.JavaPairRDD<String,org.apache.spark.input.PortableDataStream> binaryFiles, org.canova.api.records.reader.RecordReader reader) Convert a traditional sc.binaryFiles in to something usable for machine learning
static org.apache.spark.api.java.JavaRDD<org.apache.spark.mllib.regression.LabeledPoint>	fromBinary(org.apache.spark.api.java.JavaRDD<scala.Tuple2<String,org.apache.spark.input.PortableDataStream>> binaryFiles, org.canova.api.records.reader.RecordReader reader) Convert a traditional sc.binaryFiles in to something usable for machine learning
static org.apache.spark.api.java.JavaRDD<org.apache.spark.mllib.regression.LabeledPoint>	fromDataSet(org.apache.spark.api.java.JavaSparkContext sc, org.apache.spark.api.java.JavaRDD<org.nd4j.linalg.dataset.DataSet> data) Convert an rdd of data set in to labeled point
static org.apache.spark.api.java.JavaRDD<org.nd4j.linalg.dataset.DataSet>	fromLabeledPoint(org.apache.spark.api.java.JavaRDD<org.apache.spark.mllib.regression.LabeledPoint> data, int numPossibleLabels, int batchSize) Convert an rdd of labeled point based on the specified batch size in to data set
static org.apache.spark.api.java.JavaRDD<org.nd4j.linalg.dataset.DataSet>	fromLabeledPoint(org.apache.spark.api.java.JavaSparkContext sc, org.apache.spark.api.java.JavaRDD<org.apache.spark.mllib.regression.LabeledPoint> data, int numPossibleLabels) From labeled point
static org.apache.spark.mllib.regression.LabeledPoint	pointOf(Collection<org.canova.api.writable.Writable> writables) Returns a labeled point of the writables where the final item is the point and the rest of the items are features
static double	toClassifierPrediction(org.apache.spark.mllib.linalg.Vector vector) This is for the edge case where you have a single output layer and need to convert the output layer to an index
static org.apache.spark.mllib.linalg.Matrix	toMatrix(org.nd4j.linalg.api.ndarray.INDArray arr) Convert an ndarray to a matrix.
static org.nd4j.linalg.api.ndarray.INDArray	toMatrix(org.apache.spark.mllib.linalg.Matrix arr) Convert an ndarray to a matrix.
static org.apache.spark.mllib.linalg.Vector	toVector(org.nd4j.linalg.api.ndarray.INDArray arr) Convert an ndarray to a vector
static org.nd4j.linalg.api.ndarray.INDArray	toVector(org.apache.spark.mllib.linalg.Vector arr) Convert an ndarray to a vector

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

MLLibUtil

public MLLibUtil()

Method Detail

toClassifierPrediction

public static double toClassifierPrediction(org.apache.spark.mllib.linalg.Vector vector)

This is for the edge case where you have a single output layer and need to convert the output layer to an index

Parameters:

vector - the vector to get the classifier prediction for

Returns:

the prediction for the given vector

toMatrix

public static org.nd4j.linalg.api.ndarray.INDArray toMatrix(org.apache.spark.mllib.linalg.Matrix arr)

Convert an ndarray to a matrix. Note that the matrix will be con

Parameters:

arr - the array

Returns:

an mllib vector

toVector

public static org.nd4j.linalg.api.ndarray.INDArray toVector(org.apache.spark.mllib.linalg.Vector arr)

Convert an ndarray to a vector

Parameters:

arr - the array

Returns:

an mllib vector

toMatrix

public static org.apache.spark.mllib.linalg.Matrix toMatrix(org.nd4j.linalg.api.ndarray.INDArray arr)

Convert an ndarray to a matrix. Note that the matrix will be con

Parameters:

arr - the array

Returns:

an mllib vector

toVector

public static org.apache.spark.mllib.linalg.Vector toVector(org.nd4j.linalg.api.ndarray.INDArray arr)

Convert an ndarray to a vector

Parameters:

arr - the array

Returns:

an mllib vector

fromBinary

public static org.apache.spark.api.java.JavaRDD<org.apache.spark.mllib.regression.LabeledPoint> fromBinary(org.apache.spark.api.java.JavaPairRDD<String,org.apache.spark.input.PortableDataStream> binaryFiles,
                                                                                                           org.canova.api.records.reader.RecordReader reader)

Convert a traditional sc.binaryFiles in to something usable for machine learning

Parameters:

binaryFiles - the binary files to convert

reader - the reader to use

Returns:

the labeled points based on the given rdd

fromBinary

public static org.apache.spark.api.java.JavaRDD<org.apache.spark.mllib.regression.LabeledPoint> fromBinary(org.apache.spark.api.java.JavaRDD<scala.Tuple2<String,org.apache.spark.input.PortableDataStream>> binaryFiles,
                                                                                                           org.canova.api.records.reader.RecordReader reader)

Convert a traditional sc.binaryFiles in to something usable for machine learning

Parameters:

binaryFiles - the binary files to convert

reader - the reader to use

Returns:

the labeled points based on the given rdd

pointOf

public static org.apache.spark.mllib.regression.LabeledPoint pointOf(Collection<org.canova.api.writable.Writable> writables)

Returns a labeled point of the writables where the final item is the point and the rest of the items are features

Parameters:

writables - the writables

Returns:

the labeled point

fromLabeledPoint

public static org.apache.spark.api.java.JavaRDD<org.nd4j.linalg.dataset.DataSet> fromLabeledPoint(org.apache.spark.api.java.JavaRDD<org.apache.spark.mllib.regression.LabeledPoint> data,
                                                                                                  int numPossibleLabels,
                                                                                                  int batchSize)

Convert an rdd of labeled point based on the specified batch size in to data set

Parameters:

data - the data to convert

numPossibleLabels - the number of possible labels

batchSize - the batch size

Returns:

the new rdd

fromLabeledPoint

public static org.apache.spark.api.java.JavaRDD<org.nd4j.linalg.dataset.DataSet> fromLabeledPoint(org.apache.spark.api.java.JavaSparkContext sc,
                                                                                                  org.apache.spark.api.java.JavaRDD<org.apache.spark.mllib.regression.LabeledPoint> data,
                                                                                                  int numPossibleLabels)

From labeled point

Parameters:

sc - the org.deeplearning4j.spark context used for creating the rdd

data - the data to convert

numPossibleLabels - the number of possible labels

Returns:

fromDataSet

public static org.apache.spark.api.java.JavaRDD<org.apache.spark.mllib.regression.LabeledPoint> fromDataSet(org.apache.spark.api.java.JavaSparkContext sc,
                                                                                                            org.apache.spark.api.java.JavaRDD<org.nd4j.linalg.dataset.DataSet> data)

Convert an rdd of data set in to labeled point

Parameters:

sc - the spark context to use

data - the dataset to convert

Returns:

an rdd of labeled point