FeedForward Net

Nerual net class for constructing the nets using layers and providing access functions for net info, e.g., parameters.

Example usages:

from singa import net as ffnet
from singa import metric
from singa import loss
from singa import layer
from singa import device

# create net and add layers
net = ffnet.FeedForwardNet(loss.SoftmaxCrossEntropy(), metric.Accuracy())
net.add(layer.Conv2D('conv1', 32, 5, 1, input_sample_shape=(3,32,32,)))
net.add(layer.Activation('relu1'))
net.add(layer.MaxPooling2D('pool1', 3, 2))
net.add(layer.Flatten('flat'))
net.add(layer.Dense('dense', 10))

# init parameters
for p in net.param_values():
    if len(p.shape) == 0:
        p.set_value(0)
    else:
        p.gaussian(0, 0.01)

# move net onto gpu
dev = device.create_cuda_gpu()
net.to_device(dev)

# training (skipped)

# do prediction after training
x = tensor.Tensor((2, 3, 32, 32), dev)
x.uniform(-1, 1)
y = net.predict(x)
print tensor.to_numpy(y)
class singa.net.FeedForwardNet(loss=None, metric=None)

Bases: object

to_device(dev)

Move the net onto the given device, including all parameters and intermediate data.

add(lyr, src=None)

Append a layer into the layer list.

This function will get the sample shape from the src layers to setup the newly added layer. For the first layer, it is setup outside. The calling function should ensure the correctness of the layer order. If src is None, the last layer is the src layer. If there are multiple src layers, the src is a list of the src layers.

Parameters

  • lyr (Layer) – the layer to be added

  • src (Layer) – the source layer of lyr

param_values()

Return a list of tensors for all parameters

param_specs()

Return a list of ParamSpec for all parameters

param_names()

Return a list for the names of all params

train(x, y)

Run BP for one iteration. This method is deprecated. It is only kept for backward compatibility. The name of this method is confusing since it does not update parameters. Please use backprob() instead. The back progagation algorithm computes gradients but it does not train.

backprob(x, y)

Run BP for one iteration.

Currently only support nets with a single output layer, and a single loss objective and metric. For multiple outputs (with multiple loss/metric), please manually call forward, compute loss/metric and call backward. backward() is also more memory efficient than this function.

Parameters

  • x – input data, a single input Tensor or a dict: layer name -> Tensor

  • y – label data, a single input Tensor.

Returns

gradients of parameters and the loss and metric values.

evaluate(x, y)

Evaluate the loss and metric of the given data.

Currently only support nets with a single output layer, and a single loss objective and metric. TODO(wangwei) consider multiple loss objectives and metrics.

Parameters

  • x – input data, a single input Tensor or a dict: layer name -> Tensor

  • y – label data, a single input Tensor.

predict(x)

Forward the input data through each layer to get the values of the output layers.

Currently only support nets with a single output layer

Parameters

x – input data, a single input Tensor or a dict: layer name -> Tensor

Returns

a single output tensor as the prediction result.

topo_sort(layers, src_of_layer)

Topology sort of layers.

It would try to preserve the orders of the input layers.

Parameters

  • layers – a list of layers; the layers from the output of the same layer (e.g., slice layer) should be added by users in correct order; This function would not change their order.

  • src_of_layer – a dictionary: src layer name -> a list of src layers

Returns

A list of ordered layer

forward(flag, x, output=[], freeze=None)

Forward the input(s) through every layer.

Parameters

  • flag – True for training; False for evaluation; could also be model_pb2.kTrain or model_pb2.kEval, or other values for future use.

  • x – a single SINGA tensor if there is a single input; otherwise, a dictionary: layer name-> singa tensor, for each layer accepting input data. Do not associate a layer with input tensor if it is connected from another layer. For such case, use a Dummy() layer to accept the input data and connect the dummy layer to this layer.

  • output (list) – a list of layer names whose output would be returned in addition to the default output.

  • freeze (str) – layer name, freeze all layers before this layer; flag is set to false for these layers.

Returns

if there is only one output layer and output arg is empty, return

the result from the single output layer; otherwise, return a dictionary: layer name -> output tensor(s)

backward(dy, output=[], freeze=None)

Run back-propagation after forward-propagation.

Parameters

  • dy – a single tensor if there is a single loss function; otherwise, a dictionary maps the name of the layer connecting to the loss function -> gradient from the loss function. Do not associate a layer with gradient tensor if it is connecting to another layer. For such case, connect this layer to a Dummy() layer and use the dummy layer to accept the gradient.

  • output (list) – a list of layer names whose output gradient would be returned in addition to the param gradient

  • freeze (str) – layer name, stop backward after this layer.

Returns

a geneartor iterator that generates (param_names, param_values, param_grads, layer_grads) after processing each layer h, where the first three lists are for h and the last item is a dictionary which maps layer name -> its output gradient tensor(s). At the end of this function, the key set includes all layers in the output arg.

save(f, buffer_size=10, use_pickle=False)

Save model parameters using io/snapshot.

Parameters

  • f – file name

  • buffer_size – size (MB) of the IO, default setting is 10MB; Please make sure it is larger than any single parameter object.

  • use_pickle (Boolean) – if true, it would use pickle for dumping; otherwise, it would use protobuf for serialization, which uses less space.

load(f, buffer_size=10, use_pickle=False)

Load model parameters using io/snapshot.

Please refer to the argument description in save().