Data preparation

Before using SINGA, you need to write a program to pre-process the dataset you use to a format that SINGA can read. Please refer to the Data Preparation to get details about preparing this MNIST dataset.

Neural net

Figure 1 - Net structure of the MLP example.

Figure 1 shows the structure of the simple MLP model, which is constructed following Ciresan’s paper. The dashed circle contains two layers which represent one feature transformation stage. There are 6 such stages in total. They sizes of the InnerProductLayers in these circles decrease from 2500->2000->1500->1000->500->10.

Next we follow the guide in neural net page and layer page to write the neural net configuration.

• We configure a data layer to read the training/testing Records from DataShard.

  layer {
      name: "data"
      type: kShardData
      sharddata_conf {
        path: "examples/mnist/mnist_train_shard"
        batchsize: 1000
      }
      exclude: kTest
    }
  
  layer {
      name: "data"
      type: kShardData
      sharddata_conf {
        path: "examples/mnist/mnist_test_shard"
        batchsize: 1000
      }
      exclude: kTrain
    }
  

• We configure two parser layers to extract the image feature and label from Recordss loaded by the data layer. The MnistLayer will normalize the pixel values into [-1,1].

  layer{
      name:"mnist"
      type: kMnist
      srclayers: "data"
      mnist_conf {
        norm_a: 127.5
        norm_b: 1
      }
    }
  
  layer{
      name: "label"
      type: kLabel
      srclayers: "data"
    }
  

• All InnerProductLayers are configured similarly as,

  layer{
    name: "fc1"
    type: kInnerProduct
    srclayers:"mnist"
    innerproduct_conf{
      num_output: 2500
    }
    param{
      name: "w1"
      ...
    }
    param{
      name: "b1"
      ..
    }
  }
  

  with the num_output decreasing from 2500 to 10.

• A STanhLayer is connected to every InnerProductLayer except the last one. It transforms the feature via scaled tanh function.

  layer{
    name: "tanh1"
    type: kSTanh
    srclayers:"fc1"
  }
  

• The final Softmax loss layer connects to LabelLayer and the last STanhLayer.

  layer{
    name: "loss"
    type:kSoftmaxLoss
    softmaxloss_conf{ topk:1 }
    srclayers:"fc6"
    srclayers:"label"
  }
  

Updater

The normal SGD updater is selected. The learning rate shrinks by 0.997 every 60 steps (i.e., one epoch).

updater{
  type: kSGD
  learning_rate{
    base_lr: 0.001
    type : kStep
    step_conf{
      change_freq: 60
      gamma: 0.997
    }
  }
}

TrainOneBatch algorithm

The MLP model is a feed-forward model, hence Back-propagation algorithm is selected.

train_one_batch {
  alg: kBP
}

Cluster setting

The following configuration set a single worker and server for training. Training frameworks page introduces configurations of a couple of distributed training frameworks.

cluster {
  nworker_groups: 1
  nserver_groups: 1
}