org.deeplearning4j.nn.layers.feedforward.embedding

Class EmbeddingLayer

java.lang.Object

org.deeplearning4j.nn.layers.AbstractLayer<LayerConfT>

org.deeplearning4j.nn.layers.BaseLayer<EmbeddingLayer>

org.deeplearning4j.nn.layers.feedforward.embedding.EmbeddingLayer

All Implemented Interfaces:

Serializable, Cloneable, Layer, Model

public class EmbeddingLayer
extends BaseLayer<EmbeddingLayer>

Embedding layer: feed-forward layer that expects single integers per example as input (class numbers, in range 0 to numClass-1) as input. This input has shape [numExamples,1] instead of [numExamples,numClasses] for the equivalent one-hot representation. Mathematically, EmbeddingLayer is equivalent to using a DenseLayer with a one-hot representation for the input; however, it can be much more efficient with a large number of classes (as a dense layer + one-hot input does a matrix multiply with all but one value being zero).
Note: can only be used as the first layer for a network
Note 2: For a given example index i, the output is activationFunction(weights.getRow(i) + bias), hence the weight rows can be considered a vector/embedding for each example.

Author:

Alex Black

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from interface org.deeplearning4j.nn.api.Layer

Layer.TrainingMode, Layer.Type

Field Summary

Fields inherited from class org.deeplearning4j.nn.layers.BaseLayer

gradient, gradientsFlattened, gradientViews, optimizer, params, paramsFlattened, score, solver, weightNoiseParams

Fields inherited from class org.deeplearning4j.nn.layers.AbstractLayer

cacheMode, conf, dropoutApplied, dropoutMask, epochCount, index, input, iterationCount, iterationListeners, maskArray, maskState, preOutput

Constructor Summary

Constructors

Constructor and Description
EmbeddingLayer(NeuralNetConfiguration conf)

Method Summary

Modifier and Type	Method and Description
org.nd4j.linalg.api.ndarray.INDArray	activate(boolean training) Trigger an activation with the last specified input
protected void	applyDropOutIfNecessary(boolean training)
org.nd4j.linalg.primitives.Pair<Gradient,org.nd4j.linalg.api.ndarray.INDArray>	backpropGradient(org.nd4j.linalg.api.ndarray.INDArray epsilon) Calculate the gradient relative to the error in the next layer
boolean	hasBias() Does this layer have no bias term? Many layers (dense, convolutional, output, embedding) have biases by default, but no-bias versions are possible via configuration
boolean	isPretrainLayer() Returns true if the layer can be trained in an unsupervised/pretrain manner (AE, VAE, etc)
org.nd4j.linalg.api.ndarray.INDArray	preOutput(boolean training)

Methods inherited from class org.deeplearning4j.nn.layers.BaseLayer

accumulateScore, activate, activate, calcL1, calcL2, clear, clearNoiseWeightParams, clone, computeGradientAndScore, fit, fit, getGradientsViewArray, getOptimizer, getParam, getParamWithNoise, gradient, initParams, iterate, layerConf, numParams, params, paramTable, paramTable, preOutput, score, setBackpropGradientsViewArray, setParam, setParams, setParams, setParamsViewArray, setParamTable, setScoreWithZ, toString, transpose, update, update

Methods inherited from class org.deeplearning4j.nn.layers.AbstractLayer

activate, activate, activate, addListeners, applyConstraints, applyMask, batchSize, conf, feedForwardMaskArray, getIndex, getInput, getInputMiniBatchSize, getListeners, getMaskArray, gradientAndScore, init, input, layerId, migrateInput, numParams, preOutput, preOutput, setCacheMode, setConf, setIndex, setInput, setInputMiniBatchSize, setListeners, setListeners, setMaskArray, type, validateInput

Methods inherited from class java.lang.Object

equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface org.deeplearning4j.nn.api.Layer

getEpochCount, getIterationCount, setEpochCount, setIterationCount

Constructor Detail

EmbeddingLayer

public EmbeddingLayer(NeuralNetConfiguration conf)

Method Detail

backpropGradient

public org.nd4j.linalg.primitives.Pair<Gradient,org.nd4j.linalg.api.ndarray.INDArray> backpropGradient(org.nd4j.linalg.api.ndarray.INDArray epsilon)

Description copied from interface: Layer

Calculate the gradient relative to the error in the next layer

Specified by:

backpropGradient in interface Layer

Overrides:

backpropGradient in class BaseLayer<EmbeddingLayer>

Parameters:

epsilon - w^(L+1)*delta^(L+1). Or, equiv: dC/da, i.e., (dC/dz)*(dz/da) = dC/da, where C is cost function a=sigma(z) is activation.

Returns:

Pair where Gradient is gradient for this layer, INDArray is epsilon needed by next layer, but before element-wise multiply by sigmaPrime(z). So for standard feed-forward layer, if this layer is L, then return.getSecond() == (w^(L)*(delta^(L))^T)^T

preOutput

public org.nd4j.linalg.api.ndarray.INDArray preOutput(boolean training)

Overrides:

preOutput in class BaseLayer<EmbeddingLayer>

activate

public org.nd4j.linalg.api.ndarray.INDArray activate(boolean training)

Description copied from interface: Layer

Trigger an activation with the last specified input

Specified by:

activate in interface Layer

Overrides:

activate in class BaseLayer<EmbeddingLayer>

Parameters:

training - training or test mode

Returns:

the activation of the last specified input

hasBias

public boolean hasBias()

Description copied from class: BaseLayer

Does this layer have no bias term? Many layers (dense, convolutional, output, embedding) have biases by default, but no-bias versions are possible via configuration

Overrides:

hasBias in class BaseLayer<EmbeddingLayer>

Returns:

True if a bias term is present, false otherwise

isPretrainLayer

public boolean isPretrainLayer()

Description copied from interface: Layer

Returns true if the layer can be trained in an unsupervised/pretrain manner (AE, VAE, etc)

Returns:

true if the layer can be pretrained (using fit(INDArray), false otherwise

applyDropOutIfNecessary

protected void applyDropOutIfNecessary(boolean training)

Overrides:

applyDropOutIfNecessary in class AbstractLayer<EmbeddingLayer>