# Hybrid Parallelism

---

## User Guide

SINGA supports different parallelism options for distributed training.
Users just need to configure it in the job configuration.

Both `NetProto` and `LayerProto` have a field `partition_dim` to control the parallelism option:

  * `partition_dim=0`: neuralnet/layer is partitioned on data dimension, i.e., each worker processes a subset of data records.
  * `partition_dim=1`: neuralnet/layer is partitioned on feature dimension, i.e., each worker maintains a subset of feature parameters.

`partition_dim` field in `NetProto` will be applied to all layers, unless a layer has its own `partition_dim` field set.

If we want data parallelism for the whole model, just leave `partition_dim` as default (which is 0), or configure the job.conf like:

```
neuralnet {
  partition_dim: 0
  layer {
    name: ... 
    type: ...
  }
  ...
}
```

With the hybrid parallelism, we can have layers either partitioned on data dimension or feature dimension.
For example, if we want a specific layer partitioned on feature dimension, just configure like:

```
neuralnet {
  partition_dim: 0
  layer {
    name: "layer1_partition_on_data_dimension"
    type: ...
  }
  layer {
    name: "layer2_partition_on_feature_dimension"
    type: ...
    partition_dim: 1
  }
  ...
}
```

## Developer Guide

To support hybrid parallelism, after singa read users' model and paration configuration, a set of connection layers are automatically added between layers when needed:

* `BridgeSrcLayer` & `BridgeDstLayer` are added when two connected layers are not in the same machine. They are paired and are responsible for sending data/gradient to the other side during each iteration.

* `ConcateLayer` is added when there are multiple source layers. It combines their feature blobs along a given dimension.

* `SliceLayer` is added when there are mutliple dest layers, each of which only needs a subset(slice) of this layers' feature blob.

* `SplitLayer` is added when there are multiple dest layers, each of which needs the whole feature blob.

Following is the logic used in our code to add connection layers:

```
Add Slice, Concate, Split Layers for Hybrid Partition

All cases are as follows:
src_pdim | dst_pdim | connection_type | Action
    0    |     0    |     OneToOne    | Direct Connection
    1    |     1    |     OneToOne    | Direct Connection
    0    |     0    |     OneToAll    | Direct Connection
    1    |     0    |     OneToOne    | Slice -> Concate
    0    |     1    |     OneToOne    | Slice -> Concate
    1    |     0    |     OneToAll    | Slice -> Concate
    0    |     1    |     OneToAll    | Split -> Concate
    1    |     1    |     OneToAll    | Split -> Concate

Logic:
dst_pdim = 1 && OneToAll ?
  (YES) Split -> Concate
  (NO)  src_pdim = dst_pdim ?
          (YES) Direct Connection
          (NO)  Slice -> Concate
```