Neural Net

Feed-forward models

Figure 2 - Net structure of a MLP model.

Feed-forward models, e.g., CNN and MLP, can easily get configured as their layer connections are undirected without circles. The configuration for the MLP model shown in Figure 1 is as follows,

net {
  layer {
    name : 'data"
    type : kData
  }
  layer {
    name : 'image"
    type : kImage
    srclayer: 'data'
  }
  layer {
    name : 'label"
    type : kLabel
    srclayer: 'data'
  }
  layer {
    name : 'hidden"
    type : kHidden
    srclayer: 'image'
  }
  layer {
    name : 'softmax"
    type : kSoftmaxLoss
    srclayer: 'hidden'
    srclayer: 'label'
  }
}

Energy models

Figure 3 - Convert connections in RBM and RNN.

For energy models including RBM, DBM, etc., their connections are undirected (i.e., Category B). To represent these models using NeuralNet, users can simply replace each connection with two directed connections, as shown in Figure 3a. In other words, for each pair of connected layers, their source layer field should include each other’s name. The full RBM example has detailed neural net configuration for a RBM model, which looks like

net {
  layer {
    name : "vis"
    type : kVisLayer
    param {
      name : "w1"
    }
    srclayer: "hid"
  }
  layer {
    name : "hid"
    type : kHidLayer
    param {
      name : "w2"
      share_from: "w1"
    }
    srclayer: "vis"
  }
}

RNN models

For recurrent neural networks (RNN), users can remove the recurrent connections by unrolling the recurrent layer. For example, in Figure 3b, the original layer is unrolled into a new layer with 4 internal layers. In this way, the model is like a normal feed-forward model, thus can be configured similarly. The RNN example has a full neural net configuration for a RNN model.

Batch and feature dimension

Figure 4 - Partitioning of a fully connected layer.

Every layer’s feature blob is considered a matrix whose rows are feature vectors. Thus, one layer can be split on two dimensions. Partitioning on dimension 0 (also called batch dimension) slices the feature matrix by rows. For instance, if the mini-batch size is 256 and the layer is partitioned into 2 sub-layers, each sub-layer would have 128 feature vectors in its feature blob. Partitioning on this dimension has no effect on the parameters, as every Param object is replicated in the sub-layers. Partitioning on dimension 1 (also called feature dimension) slices the feature matrix by columns. For example, suppose the original feature vector has 50 units, after partitioning into 2 sub-layers, each sub-layer would have 25 units. This partitioning may result in Param object being split, as shown in Figure 4. Both the bias vector and weight matrix are partitioned into two sub-layers.

Partitioning configuration

There are 4 partitioning schemes, whose configurations are give below,

1. Partitioning each singe layer into sub-layers on batch dimension (see below). It is enabled by configuring the partition dimension of the layer to 0, e.g.,

     # with other fields omitted
     layer {
       partition_dim: 0
     }
   

2. Partitioning each singe layer into sub-layers on feature dimension (see below). It is enabled by configuring the partition dimension of the layer to 1, e.g.,

     # with other fields omitted
     layer {
       partition_dim: 1
     }
   

3. Partitioning all layers into different subsets. It is enabled by configuring the location ID of a layer, e.g.,

     # with other fields omitted
     layer {
       location: 1
     }
     layer {
       location: 0
     }
   

1. Hybrid partitioning of strategy 1, 2 and 3. The hybrid partitioning is useful for large models. An example application is to implement the idea proposed by Alex. Hybrid partitioning is configured like,

     # with other fields omitted
     layer {
       location: 1
     }
     layer {
       location: 0
     }
     layer {
       partition_dim: 0
       location: 0
     }
     layer {
       partition_dim: 1
       location: 0
     }
   

Currently SINGA supports strategy-2 well. Other partitioning strategies are are under test and will be released in later version.

Creation

static shared_ptr<NeuralNet> NeuralNet::Create(const NetProto& np, Phase phase, int num);

The above function creates a NeuralNet for a given phase, and returns a shared pointer to the NeuralNet instance. The phase is in {kTrain, kValidation, kTest}. num is used for net partitioning which indicates the number of partitions. Typically, a training job includes three neural nets for training, validation and test phase respectively. The three neural nets share most layers except the data layer, loss layer or output layer, etc.. The Create function takes in the full net configuration including layers for training, validation and test. It removes layers for phases other than the specified phase based on the exclude field in layer configuration:

layer {
  ...
  exclude : kTest # filter this layer for creating test net
}

The filtered net configuration is passed to the constructor of NeuralNet:

NeuralNet::NeuralNet(NetProto netproto, int npartitions);

The constructor creates a graph representing the net structure firstly in

Graph* NeuralNet::CreateGraph(const NetProto& netproto, int npartitions);

Next, it creates a layer for each node and connects layers if their nodes are connected.

void NeuralNet::CreateNetFromGraph(Graph* graph, int npartitions);

Since the NeuralNet instance may be shared among multiple workers, the Create function returns a shared pointer to the NeuralNet instance .

Parameter sharing

Param sharing is enabled by first sharing the Param configuration (in NeuralNet::Create) to create two similar (e.g., the same shape) Param objects, and then calling (in NeuralNet::CreateNetFromGraph),

void Param::ShareFrom(const Param& from);

It is also possible to share Params of two nets, e.g., sharing parameters of the training net and the test net,

void NeuralNet:ShareParamsFrom(shared_ptr<NeuralNet> other);

It will call Param::ShareFrom for each Param object.

Access functions

NeuralNet provides a couple of access function to get the layers and params of the net:

const std::vector<Layer*>& layers() const;
const std::vector<Param*>& params() const ;
Layer* name2layer(string name) const;
Param* paramid2param(int id) const;

Partitioning