CNN Example

Data preparation

Before using SINGA, you need to write a program to convert the dataset into a format that SINGA can read. Please refer to the Data Preparation to get details about preparing this CIFAR10 dataset.

Neural net

Figure 1 shows the net structure of the CNN model we used in this example, which is set following Alex The dashed circle represents one feature transformation stage, which generally has four layers as shown in the figure. Sometimes the rectifier layer and normalization layer are omitted or swapped in one stage. For this example, there are 3 such stages.

Next we follow the guide in neural net page and layer page to write the neural net configuration.

Figure 1 - Net structure of the CNN example.

• We configure a data layer to read the training/testing Records from DataShard.

  layer{
    name: "data"
    type: kShardData
    sharddata_conf {
      path: "examples/cifar10/cifar10_train_shard"
      batchsize: 16
      random_skip: 5000
    }
    exclude: kTest  # exclude this layer for the testing net
  }
  layer{
    name: "data"
    type: kShardData
    sharddata_conf {
      path: "examples/cifar10/cifar10_test_shard"
      batchsize: 100
    }
    exclude: kTrain # exclude this layer for the training net
  }
  

• We configure two parser layers to extract the image feature and label from Recordss loaded by the data layer.

  layer{
    name:"rgb"
    type: kRGBImage
    srclayers: "data"
    rgbimage_conf {...}
  }
  layer{
    name: "label"
    type: kLabel
    srclayers: "data"
  }
  

• We configure layers for the feature transformation as follows (all layers are built-in layers in SINGA; hyper-parameters of these layers are set according to Alex’s setting).

  layer {
    name: "conv1"
    type: kConvolution
    srclayers: "rgb"
    convolution_conf {... }
    ...
  }
  layer {
    name: "pool1"
    type: kPooling
    srclayers: "conv1"
    pooling_conf {... }
  }
  layer {
    name: "relu1"
    type: kReLU
    srclayers:"pool1"
  }
  layer {
    name: "norm1"
    type: kLRN
    lrn_conf {... }
    srclayers:"relu1"
  }
  

The configurations for another 2 stages are omitted here.

• There is an inner product layer after the 3 transformation stages, which is configured with 10 output units, i.e., the number of total labels. The weight matrix Param is configured with a large weight decay scale to reduce the over-fitting.

  layer {
    name: "ip1"
    type: kInnerProduct
    srclayers:"pool3"
    innerproduct_conf {
      num_output: 10
    }
    param {
      name: "w4"
      wd_scale:250
      ...
    }
    param {
      name: "b4"
      ...
    }
  }
  

• The last layer is a Softmax loss layer

  layer{
    name: "loss"
    type: kSoftmaxLoss
    softmaxloss_conf{ topk:1 }
    srclayers:"ip1"
    srclayers: "label"
  }
  

Updater

The normal SGD updater is selected. The learning rate is changed like going down stairs, and is configured using the kFixedStep type.

    updater{
      type: kSGD
      weight_decay:0.004
      learning_rate {
        type: kFixedStep
        fixedstep_conf:{
          step:0             # lr for step 0-60000 is 0.001
          step:60000         # lr for step 60000-65000 is 0.0001
          step:65000         # lr for step 650000- is 0.00001
          step_lr:0.001
          step_lr:0.0001
          step_lr:0.00001
        }
      }
    }

TrainOneBatch algorithm

The CNN model is a feed forward model, thus should be configured to use the Back-propagation algorithm.

train_one_batch {
  alg: kBP
}

Cluster setting

The following configuration set a single worker and server for training. Training frameworks page introduces configurations of a couple of distributed training frameworks.

cluster {
  nworker_groups: 1
  nserver_groups: 1
}