Parameters¶
A Param
object in SINGA represents a set of parameters, e.g., a weight matrix
or a bias vector. Basic user guide describes how to configure for a Param
object, and Advanced user guide provides details on implementing users’
parameter initialization methods.
Basic user guide¶
The configuration of a Param object is inside a layer configuration, as the
Param
are associated with layers. An example configuration is like
layer {
...
param {
name : "p1"
init {
type : kConstant
value: 1
}
}
}
The SGD algorithm starts with initializing all
parameters according to user specified initialization method (the init
field).
For the above example,
all parameters in Param
“p1” will be initialized to constant value 1. The
configuration fields of a Param object is defined in ParamProto:
- name, an identifier string. It is an optional field. If not provided, SINGA will generate one based on layer name and its order in the layer.
- init, field for setting initialization methods.
- share_from, name of another
Param
object, from which thisParam
will share configurations and values. - lr_scale, float value to be multiplied with the learning rate when updating the parameters
- wd_scale, float value to be multiplied with the weight decay when updating the parameters
There are some other fields that are specific to initialization methods.
Initialization methods¶
Users can set the type
of init
use the following built-in initialization
methods,
kConst
, set all parameters of the Param object to a constant valuetype: kConst value: float # default is 1
kGaussian
, initialize the parameters following a Gaussian distribution.type: kGaussian mean: float # mean of the Gaussian distribution, default is 0 std: float # standard variance, default is 1 value: float # default 0
kUniform
, initialize the parameters following an uniform distributiontype: kUniform low: float # lower boundary, default is -1 high: float # upper boundary, default is 1 value: float # default 0
kGaussianSqrtFanIn
, initializeParam
objects with two dimensions (i.e., matrix) usingkGaussian
and then multiple each parameter with1/sqrt(fan_in)
, wherefan_in
is the number of columns of the matrix.kUniformSqrtFanIn
, the same askGaussianSqrtFanIn
except that the distribution is an uniform distribution.kUniformFanInOut
, initialize matrixParam
objects usingkUniform
and then multiple each parameter withsqrt(6/(fan_in + fan_out))
, wherefan_in + fan_out
sums up the number of columns and rows of the matrix.
For all above initialization methods except kConst
, if their value
is not
1, every parameter will be multiplied with value
. Users can also implement
their own initialization method following the Advanced user guide.
Advanced user guide¶
This sections describes the details on implementing new parameter initialization methods.
Base ParamGenerator¶
All initialization methods are implemented as
subclasses of the base ParamGenerator
class.
class ParamGenerator {
public:
virtual void Init(const ParamGenProto&);
void Fill(Param*);
protected:
ParamGenProto proto_;
};
Configurations of the initialization method is in ParamGenProto
. The Fill
function fills the Param
object (passed in as an argument).
New ParamGenerator subclass¶
Similar to implement a new Layer subclass, users can define a configuration protocol message,
# in user.proto
message FooParamProto {
optional int32 x = 1;
}
extend ParamGenProto {
optional FooParamProto fooparam_conf =101;
}
The configuration of Param
would be
param {
...
init {
user_type: 'FooParam" # must use user_type for user defined methods
[fooparam_conf] { # must use brackets for configuring user defined messages
x: 10
}
}
}
The subclass could be declared as,
class FooParamGen : public ParamGenerator {
public:
void Fill(Param*) override;
};
Users can access the configuration fields in Fill
by
int x = proto_.GetExtension(fooparam_conf).x();
To use the new initialization method, users need to register it in the main function.
driver.RegisterParamGenerator<FooParamGen>("FooParam") # must be consistent with the user_type in configuration
{% comment %}
Base Param class¶
Members¶
int local_version_;
int slice_start_;
vector<int> slice_offset_, slice_size_;
shared_ptr<Blob<float>> data_;
Blob<float> grad_;
ParamProto proto_;
Each Param object has a local version and a global version (inside the data
Blob). These two versions are used for synchronization. If multiple Param
objects share the same values, they would have the same data_
field.
Consequently, their global version is the same. The global version is updated
by the stub thread. The local version is
updated in Worker::Update
function which assigns the global version to the
local version. The Worker::Collect
function is blocked until the global
version is larger than the local version, i.e., when data_
is updated. In
this way, we synchronize workers sharing parameters.
In Deep learning models, some Param objects are 100 times larger than others.
To ensure the load-balance among servers, SINGA slices large Param objects. The
slicing information is recorded by slice_*
. Each slice is assigned a unique
ID starting from 0. slice_start_
is the ID of the first slice of this Param
object. slice_offset_[i]
is the offset of the i-th slice in this Param
object. slice_size_[i]
is the size of the i-th slice. These slice information
is used to create messages for transferring parameter values or gradients to
different servers.
Each Param object has a grad_
field for gradients. Param objects do not share
this Blob although they may share data_
. Because each layer containing a
Param object would contribute gradients. E.g., in RNN, the recurrent layers
share parameters values, and the gradients used for updating are averaged from all recurrent
these recurrent layers. In SINGA, the stub thread will aggregate local
gradients for the same Param object. The server will do a global aggregation
of gradients for the same Param object.
The proto_
field has some meta information, e.g., name and ID. It also has a
field called owner
which is the ID of the Param object that shares parameter
values with others.
Functions¶
The base Param class implements two sets of functions,
virtual void InitValues(int version = 0); // initialize values according to `init_method`
void ShareFrom(const Param& other); // share `data_` from `other` Param
--------------
virtual Msg* GenGetMsg(bool copy, int slice_idx);
virtual Msg* GenPutMsg(bool copy, int slice_idx);
... // other message related functions.
Besides the functions for processing the parameter values, there is a set of
functions for generating and parsing messages. These messages are for
transferring parameter values or gradients between workers and servers. Each
message corresponds to one Param slice. If copy
is false, it means the
receiver of this message is in the same process as the sender. In such case,
only pointers to the memory of parameter value (or gradient) are wrapped in
the message; otherwise, the parameter values (or gradients) should be copied
into the message.
Implementing Param subclass¶
Users can extend the base Param class to implement their own parameter initialization methods and message transferring protocols. Similar to implementing a new Layer subclasses, users can create google protocol buffer messages for configuring the Param subclass. The subclass, denoted as FooParam should be registered in main.cc,
dirver.RegisterParam<FooParam>(kFooParam); // kFooParam should be different to 0, which is for the base Param type
- type, an integer representing the
Param
type. Currently SINGA provides oneParam
implementation with type 0 (the default type). If users want to use their own Param implementation, they should extend the base Param class and configure this field withkUserParam
{% endcomment %}