singa::Optimizer Class Reference

The base class for gradient descent algorithms used to update the model parameters in order to optimize the objective (loss) function. More...

#include <optimizer.h>

Inheritance diagram for singa::Optimizer:

Collaboration diagram for singa::Optimizer:

Public Member Functions
void	Setup (const string &str)
	Setup the optimzier using configurations from serialized string (for binding languages). More...
virtual void	Setup (const OptimizerConf &conf)
	Setup the meta fields of the optimizer.
virtual void	Register (const string &name, const ParamSpec &specs)
	Register the parameter, e.g., create Constraint and Regularizers. More...
virtual void	ApplyRegularizerConstraint (int epoch, const string &name, const Tensor &value, Tensor &grad, int step=-1)
virtual void	Apply (int epoch, float lr, const string &name, Tensor &grad, Tensor &value, int step=-1)=0
	Apply the updating algorithm if the gradient is not empty. More...
void	Apply (int epoch, const string &name, Tensor &grad, Tensor &value, int step=-1)
	Apply the updating algorithm if the gradient is not empty. More...
void	SetLearningRateGenerator (function< float(int)> func)
	The argument is a function that returns the learning rate given the current step (i.e., curren running iteration). More...
float	GetLearningRate (int step)

Protected Attributes
function< float(int)>	learning_rate_generator_
std::unordered_map< std::string, float >	learning_rate_multplier_
std::unordered_map< std::string, Constraint * >	constraints_
std::unordered_map< std::string, Regularizer * >	regularizers_
Constraint *	constraint_ = nullptr
Regularizer *	regularizer_ = nullptr
OptimizerConf	conf_

Detailed Description

The base class for gradient descent algorithms used to update the model parameters in order to optimize the objective (loss) function.

It updates parameters based on the gradients of the loss w.r.t each parameter. Most sub-classes uses first order gradients. An overview of gradient descent algorithms, http://sebastianruder.com/optimizing-gradient-descent/

Member Function Documentation

Apply() [1/2]

virtual void singa::Optimizer::Apply ( int  epoch, float  lr, const string &  name, Tensor &  grad, Tensor &  value, int  step = -1  )

pure virtual

Apply the updating algorithm if the gradient is not empty.

No learning rate scaling, gradient constraints/regularization will be conducted. It assumes all these operations are done either by users or by Apply(int, const string&, Tensor*, Tensor*). All sub-classes should override this function.

Implemented in singa::RMSProp, singa::AdaGrad, singa::Nesterov, and singa::SGD.

Apply() [2/2]

void singa::Optimizer::Apply	(	int	epoch,
		const string &	name,
		Tensor &	grad,
		Tensor &	value,
		int	step = -1
	)

Apply the updating algorithm if the gradient is not empty.

It will apply regularization and constraint to the parameters if configured during Register(). If will also scale the learning rate if configured in ParamSpecs (see Register).

Register()

virtual void singa::Optimizer::Register ( const string &  name, const ParamSpec &  specs  )

virtual

Register the parameter, e.g., create Constraint and Regularizers.

If there is no constraint or regularizer, then no need to register the parameter.

SetLearningRateGenerator()

void singa::Optimizer::SetLearningRateGenerator ( function< float(int)>  func )

inline

The argument is a function that returns the learning rate given the current step (i.e., curren running iteration).

Setup()

void singa::Optimizer::Setup ( const string &  str )

inline

Setup the optimzier using configurations from serialized string (for binding languages).

---

The documentation for this class was generated from the following file:

• /home/moaz/incubator-singa/include/singa/model/optimizer.h