MLP Example

Data preparation

Before using SINGA, you need to write a program to pre-process the dataset you use to a format that SINGA can read. Please refer to the Data Preparation to get details about preparing this MNIST dataset.

Neural net

Figure 1 - Net structure of the MLP example.

Figure 1 shows the structure of the simple MLP model, which is constructed following Ciresan’s paper. The dashed circle contains two layers which represent one feature transformation stage. There are 6 such stages in total. They sizes of the InnerProductLayers in these circles decrease from 2500->2000->1500->1000->500->10.

Next we follow the guide in neural net page and layer page to write the neural net configuration.

• We configure an input layer to read the training/testing records from a disk file.

  layer {
      name: "data"
      type: kRecordInput
      store_conf {
        backend: "kvfile"
        path: "examples/mnist/train_data.bin"
        random_skip: 5000
        batchsize: 64
        shape: 784
        std_value: 127.5
        mean_value: 127.5
       }
       exclude: kTest
    }
  
  layer {
      name: "data"
      type: kRecordInput
      store_conf {
        backend: "kvfile"
        path: "examples/mnist/test_data.bin"
        batchsize: 100
        shape: 784
        std_value: 127.5
        mean_value: 127.5
       }
       exclude: kTrain
    }
  

• All InnerProductLayers are configured similarly as,

  layer{
    name: "fc1"
    type: kInnerProduct
    srclayers:"data"
    innerproduct_conf{
      num_output: 2500
    }
    param{
      name: "w1"
      ...
    }
    param{
      name: "b1"
      ..
    }
  }
  

  with the num_output decreasing from 2500 to 10.

• A STanhLayer is connected to every InnerProductLayer except the last one. It transforms the feature via scaled tanh function.

  layer{
    name: "tanh1"
    type: kSTanh
    srclayers:"fc1"
  }
  

• The final Softmax loss layer connects to LabelLayer and the last STanhLayer.

  layer{
    name: "loss"
    type:kSoftmaxLoss
    softmaxloss_conf{ topk:1 }
    srclayers:"fc6"
    srclayers:"data"
  }
  

Updater

The normal SGD updater is selected. The learning rate shrinks by 0.997 every 60 steps (i.e., one epoch).

updater{
  type: kSGD
  learning_rate{
    base_lr: 0.001
    type : kStep
    step_conf{
      change_freq: 60
      gamma: 0.997
    }
  }
}

TrainOneBatch algorithm

The MLP model is a feed-forward model, hence Back-propagation algorithm is selected.

train_one_batch {
  alg: kBP
}

Cluster setting

The following configuration set a single worker and server for training. Training frameworks page introduces configurations of a couple of distributed training frameworks.

cluster {
  nworker_groups: 1
  nserver_groups: 1
}