#include <itkAdaptiveStochasticPreconditionedGradientDescentOptimizer.h>

Detailed Description

This class implements a gradient descent optimizer with adaptive gain.

If is a cost function that has to be minimized, the following iterative algorithm is used to find the optimal parameters :

$x(k+1) = x(k) - a(t_k) P dC/dx$

The gain at each iteration is defined by:

$a(t_k) = a / (A + t_k + 1)^alpha$

.

And the time is updated according to:

$t_{k+1} = [ t_k + sigmoid( -g_k^T P g_{k-1} ) ]^+$

where equals at iteration . For the InitialTime is used, which is defined in the the superclass (StandardGradientDescentOptimizer). Whereas in the superclass this parameter is superfluous, in this class it makes sense.

This method is described in the following references:

[1] Y. Qiao, B.P.F. Lelieveldt, M. Staring An efficient preconditioner for stochastic gradient descent optimization of image registration IEEE Transactions on Medical Imaging, 2019 https://doi.org/10.1109/TMI.2019.2897943

[2] P. Cruz Almost sure convergence and asymptotical normality of a generalization of Kesten's stochastic approximation algorithm for multidimensional case Technical Report, 2005. http://hdl.handle.net/2052/74

[3] S. Klein, J.P.W. Pluim, M. Staring, M.A. Viergever Adaptive stochastic gradient descent optimisation for image registration International Journal of Computer Vision, vol. 81, no. 3, pp. 227-239, 2009 http://dx.doi.org/10.1007/s11263-008-0168-y

It is very suitable to be used in combination with a stochastic estimate of the gradient . For example, in image registration problems it is often advantageous to compute the metric derivative ( ) on a new set of randomly selected image samples in each iteration. You may set the parameter NewSamplesEveryIteration to "true" to achieve this effect. For more information on this strategy, you may have a look at:

See also: StochasticPreconditionedGradientDescent, AdaptiveStochasticGradientDescentOptimizer

Definition at line 78 of file itkAdaptiveStochasticPreconditionedGradientDescentOptimizer.h.

Inheritance diagram for itk::AdaptiveStochasticPreconditionedGradientDescentOptimizer:

Public Types
typedef SmartPointer< const Self >	ConstPointer

typedef Superclass::CostFunctionType	CostFunctionType

typedef Superclass::DerivativeType	DerivativeType

typedef Superclass::MeasureType	MeasureType

typedef Superclass::ParametersType	ParametersType

typedef SmartPointer< Self >	Pointer

typedef Superclass::PreconditionType	PreconditionType

typedef Superclass::PreconditionValueType	PreconditionValueType

typedef Superclass::ScaledCostFunctionPointer	ScaledCostFunctionPointer

typedef Superclass::ScaledCostFunctionType	ScaledCostFunctionType

typedef Superclass::ScalesType	ScalesType

typedef AdaptiveStochasticPreconditionedGradientDescentOptimizer	Self

typedef Superclass::StopConditionType	StopConditionType

typedef StochasticPreconditionedGradientDescentOptimizer	Superclass

Public Types inherited from itk::StochasticPreconditionedGradientDescentOptimizer
typedef SmartPointer< const Self >	ConstPointer

typedef Superclass::CostFunctionType	CostFunctionType

typedef Superclass::DerivativeType	DerivativeType

typedef Superclass::MeasureType	MeasureType

typedef Superclass::ParametersType	ParametersType

typedef SmartPointer< Self >	Pointer

typedef Superclass::PreconditionType	PreconditionType

typedef Superclass::PreconditionValueType	PreconditionValueType

typedef Superclass::ScaledCostFunctionPointer	ScaledCostFunctionPointer

typedef Superclass::ScaledCostFunctionType	ScaledCostFunctionType

typedef Superclass::ScalesType	ScalesType

typedef StochasticPreconditionedGradientDescentOptimizer	Self

typedef Superclass::StopConditionType	StopConditionType

typedef PreconditionedGradientDescentOptimizer	Superclass

Public Types inherited from itk::PreconditionedGradientDescentOptimizer
typedef SmartPointer< const Self >	ConstPointer

typedef Superclass::CostFunctionType	CostFunctionType

typedef Superclass::DerivativeType	DerivativeType

typedef Superclass::MeasureType	MeasureType

typedef Superclass::ParametersType	ParametersType

typedef SmartPointer< Self >	Pointer

typedef vnl_sparse_matrix< PreconditionValueType >	PreconditionType

typedef DerivativeType::ValueType	PreconditionValueType

typedef Superclass::ScaledCostFunctionPointer	ScaledCostFunctionPointer

typedef Superclass::ScaledCostFunctionType	ScaledCostFunctionType

typedef Superclass::ScalesType	ScalesType

typedef PreconditionedGradientDescentOptimizer	Self

enum	StopConditionType { MaximumNumberOfIterations , MetricError , MinimumStepSize }

typedef ScaledSingleValuedNonLinearOptimizer	Superclass

Public Types inherited from itk::ScaledSingleValuedNonLinearOptimizer
typedef SmartPointer< const Self >	ConstPointer

typedef Superclass::CostFunctionType	CostFunctionType

typedef Superclass::DerivativeType	DerivativeType

typedef Superclass::MeasureType	MeasureType

typedef Superclass::ParametersType	ParametersType

typedef SmartPointer< Self >	Pointer

typedef ScaledCostFunctionType::Pointer	ScaledCostFunctionPointer

typedef ScaledSingleValuedCostFunction	ScaledCostFunctionType

typedef NonLinearOptimizer::ScalesType	ScalesType

typedef ScaledSingleValuedNonLinearOptimizer	Self

typedef SingleValuedNonLinearOptimizer	Superclass

Public Member Functions
virtual const char *	GetClassName () const

virtual double	GetSigmoidMax () const

virtual double	GetSigmoidMin () const

virtual double	GetSigmoidScale () const

virtual bool	GetUseAdaptiveStepSizes () const

virtual void	SetSigmoidMax (double _arg)

virtual void	SetSigmoidMin (double _arg)

virtual void	SetSigmoidScale (double _arg)

virtual void	SetUseAdaptiveStepSizes (bool _arg)

Public Member Functions inherited from itk::StochasticPreconditionedGradientDescentOptimizer
virtual void	AdvanceOneStep (void)

virtual const char *	GetClassName () const

virtual double	GetCurrentTime () const

virtual double	GetInitialTime () const

virtual double	GetParam_a () const

virtual double	GetParam_A () const

virtual double	GetParam_alpha () const

virtual void	SetInitialTime (double _arg)

virtual void	SetParam_a (double _arg)

virtual void	SetParam_A (double _arg)

virtual void	SetParam_alpha (double _arg)

virtual void	StartOptimization (void)

Public Member Functions inherited from itk::PreconditionedGradientDescentOptimizer
virtual void	AdvanceOneStep (void)

const cholmod_common *	GetCholmodCommon (void) const

const cholmod_factor *	GetCholmodFactor (void) const

virtual const char *	GetClassName () const

virtual double	GetConditionNumber () const

virtual unsigned int	GetCurrentIteration () const

virtual double	GetDiagonalWeight () const

virtual const DerivativeType &	GetGradient ()

virtual double	GetLargestEigenValue () const

virtual const double &	GetLearningRate ()

virtual double	GetMinimumGradientElementMagnitude () const

virtual const unsigned long &	GetNumberOfIterations ()

virtual const DerivativeType &	GetSearchDirection ()

virtual double	GetSparsity () const

virtual const StopConditionType &	GetStopCondition ()

virtual const double &	GetValue ()

virtual void	MetricErrorResponse (ExceptionObject &err)

virtual void	ResumeOptimization (void)

virtual void	SetDiagonalWeight (double _arg)

virtual void	SetLearningRate (double _arg)

virtual void	SetMinimumGradientElementMagnitude (double _arg)

virtual void	SetNumberOfIterations (unsigned long _arg)

virtual void	SetPreconditionMatrix (PreconditionType &precondition)

virtual void	StartOptimization (void)

virtual void	StopOptimization (void)

Public Member Functions inherited from itk::ScaledSingleValuedNonLinearOptimizer
virtual const char *	GetClassName () const

const ParametersType &	GetCurrentPosition (void) const override

virtual bool	GetMaximize () const

virtual const ScaledCostFunctionType *	GetScaledCostFunction ()

virtual const ParametersType &	GetScaledCurrentPosition ()

bool	GetUseScales (void) const

virtual void	InitializeScales (void)

virtual void	MaximizeOff ()

virtual void	MaximizeOn ()

void	SetCostFunction (CostFunctionType *costFunction) override

virtual void	SetMaximize (bool _arg)

virtual void	SetUseScales (bool arg)

Static Public Member Functions
static Pointer	New ()

Static Public Member Functions inherited from itk::StochasticPreconditionedGradientDescentOptimizer
static Pointer	New ()

Static Public Member Functions inherited from itk::PreconditionedGradientDescentOptimizer
static Pointer	New ()

Static Public Member Functions inherited from itk::ScaledSingleValuedNonLinearOptimizer
static Pointer	New ()

Protected Member Functions
	AdaptiveStochasticPreconditionedGradientDescentOptimizer ()

virtual void	UpdateCurrentTime (void)

virtual	~AdaptiveStochasticPreconditionedGradientDescentOptimizer ()

Protected Member Functions inherited from itk::StochasticPreconditionedGradientDescentOptimizer
virtual double	Compute_a (double k) const

	StochasticPreconditionedGradientDescentOptimizer ()

virtual void	UpdateCurrentTime (void)

virtual	~StochasticPreconditionedGradientDescentOptimizer ()

Protected Member Functions inherited from itk::PreconditionedGradientDescentOptimizer
virtual void	CholmodSolve (const DerivativeType &gradient, DerivativeType &searchDirection, int solveType=CHOLMOD_A)

	PreconditionedGradientDescentOptimizer ()

void	PrintSelf (std::ostream &os, Indent indent) const

virtual	~PreconditionedGradientDescentOptimizer ()

Protected Member Functions inherited from itk::ScaledSingleValuedNonLinearOptimizer
virtual void	GetScaledDerivative (const ParametersType &parameters, DerivativeType &derivative) const

virtual MeasureType	GetScaledValue (const ParametersType &parameters) const

virtual void	GetScaledValueAndDerivative (const ParametersType &parameters, MeasureType &value, DerivativeType &derivative) const

void	PrintSelf (std::ostream &os, Indent indent) const override

	ScaledSingleValuedNonLinearOptimizer ()

void	SetCurrentPosition (const ParametersType &param) override

virtual void	SetScaledCurrentPosition (const ParametersType &parameters)

	~ScaledSingleValuedNonLinearOptimizer () override

Protected Attributes
DerivativeType	m_PreviousSearchDirection

Protected Attributes inherited from itk::StochasticPreconditionedGradientDescentOptimizer
double	m_CurrentTime

Protected Attributes inherited from itk::PreconditionedGradientDescentOptimizer
cholmod_common *	m_CholmodCommon

cholmod_factor *	m_CholmodFactor

cholmod_sparse *	m_CholmodGradient

double	m_ConditionNumber

DerivativeType	m_Gradient

double	m_LargestEigenValue

double	m_LearningRate

DerivativeType	m_SearchDirection

double	m_Sparsity

StopConditionType	m_StopCondition

Protected Attributes inherited from itk::ScaledSingleValuedNonLinearOptimizer
ScaledCostFunctionPointer	m_ScaledCostFunction

ParametersType	m_ScaledCurrentPosition

Private Member Functions
	AdaptiveStochasticPreconditionedGradientDescentOptimizer (const Self &)

void	operator= (const Self &)

Private Attributes
double	m_SigmoidMax

double	m_SigmoidMin

double	m_SigmoidScale

bool	m_UseAdaptiveStepSizes

Additional Inherited Members
Protected Types inherited from itk::PreconditionedGradientDescentOptimizer
typedef int	CInt

Member Typedef Documentation

◆ ConstPointer

typedef SmartPointer<const Self> itk::AdaptiveStochasticPreconditionedGradientDescentOptimizer::ConstPointer

Definition at line 88 of file itkAdaptiveStochasticPreconditionedGradientDescentOptimizer.h.

◆ CostFunctionType

typedef Superclass::CostFunctionType itk::AdaptiveStochasticPreconditionedGradientDescentOptimizer::CostFunctionType

Definition at line 101 of file itkAdaptiveStochasticPreconditionedGradientDescentOptimizer.h.

◆ DerivativeType

typedef Superclass::DerivativeType itk::AdaptiveStochasticPreconditionedGradientDescentOptimizer::DerivativeType

Definition at line 100 of file itkAdaptiveStochasticPreconditionedGradientDescentOptimizer.h.

◆ MeasureType

typedef Superclass::MeasureType itk::AdaptiveStochasticPreconditionedGradientDescentOptimizer::MeasureType

Typedefs inherited from the superclass.

Definition at line 98 of file itkAdaptiveStochasticPreconditionedGradientDescentOptimizer.h.

◆ ParametersType

typedef Superclass::ParametersType itk::AdaptiveStochasticPreconditionedGradientDescentOptimizer::ParametersType

Definition at line 99 of file itkAdaptiveStochasticPreconditionedGradientDescentOptimizer.h.

◆ Pointer

typedef SmartPointer<Self> itk::AdaptiveStochasticPreconditionedGradientDescentOptimizer::Pointer

Definition at line 87 of file itkAdaptiveStochasticPreconditionedGradientDescentOptimizer.h.

◆ PreconditionType

typedef Superclass::PreconditionType itk::AdaptiveStochasticPreconditionedGradientDescentOptimizer::PreconditionType

Definition at line 109 of file itkAdaptiveStochasticPreconditionedGradientDescentOptimizer.h.

◆ PreconditionValueType

typedef Superclass::PreconditionValueType itk::AdaptiveStochasticPreconditionedGradientDescentOptimizer::PreconditionValueType

Some typedefs for computing the SelfHessian

Definition at line 108 of file itkAdaptiveStochasticPreconditionedGradientDescentOptimizer.h.

◆ ScaledCostFunctionPointer

typedef Superclass::ScaledCostFunctionPointer itk::AdaptiveStochasticPreconditionedGradientDescentOptimizer::ScaledCostFunctionPointer

Definition at line 104 of file itkAdaptiveStochasticPreconditionedGradientDescentOptimizer.h.

◆ ScaledCostFunctionType

typedef Superclass::ScaledCostFunctionType itk::AdaptiveStochasticPreconditionedGradientDescentOptimizer::ScaledCostFunctionType

Definition at line 103 of file itkAdaptiveStochasticPreconditionedGradientDescentOptimizer.h.

◆ ScalesType

typedef Superclass::ScalesType itk::AdaptiveStochasticPreconditionedGradientDescentOptimizer::ScalesType

Definition at line 102 of file itkAdaptiveStochasticPreconditionedGradientDescentOptimizer.h.

◆ Self

typedef AdaptiveStochasticPreconditionedGradientDescentOptimizer itk::AdaptiveStochasticPreconditionedGradientDescentOptimizer::Self

Standard ITK.

Definition at line 84 of file itkAdaptiveStochasticPreconditionedGradientDescentOptimizer.h.

◆ StopConditionType

typedef Superclass::StopConditionType itk::AdaptiveStochasticPreconditionedGradientDescentOptimizer::StopConditionType

Definition at line 105 of file itkAdaptiveStochasticPreconditionedGradientDescentOptimizer.h.

◆ Superclass

typedef StochasticPreconditionedGradientDescentOptimizer itk::AdaptiveStochasticPreconditionedGradientDescentOptimizer::Superclass

Definition at line 85 of file itkAdaptiveStochasticPreconditionedGradientDescentOptimizer.h.

Constructor & Destructor Documentation

◆ AdaptiveStochasticPreconditionedGradientDescentOptimizer() [1/2]

itk::AdaptiveStochasticPreconditionedGradientDescentOptimizer::AdaptiveStochasticPreconditionedGradientDescentOptimizer ( )

protected

◆ ~AdaptiveStochasticPreconditionedGradientDescentOptimizer()

virtual itk::AdaptiveStochasticPreconditionedGradientDescentOptimizer::~AdaptiveStochasticPreconditionedGradientDescentOptimizer ( )

inlineprotectedvirtual

Definition at line 134 of file itkAdaptiveStochasticPreconditionedGradientDescentOptimizer.h.

◆ AdaptiveStochasticPreconditionedGradientDescentOptimizer() [2/2]

itk::AdaptiveStochasticPreconditionedGradientDescentOptimizer::AdaptiveStochasticPreconditionedGradientDescentOptimizer ( const Self & )

private

Member Function Documentation

◆ GetClassName()

virtual const char * itk::AdaptiveStochasticPreconditionedGradientDescentOptimizer::GetClassName ( ) const

virtual

Run-time type information (and related methods).

Reimplemented from itk::StochasticPreconditionedGradientDescentOptimizer.

Reimplemented in elastix::PreconditionedGradientDescent< TElastix >.

◆ GetSigmoidMax()

virtual double itk::AdaptiveStochasticPreconditionedGradientDescentOptimizer::GetSigmoidMax ( ) const

virtual

◆ GetSigmoidMin()

virtual double itk::AdaptiveStochasticPreconditionedGradientDescentOptimizer::GetSigmoidMin ( ) const

virtual

◆ GetSigmoidScale()

virtual double itk::AdaptiveStochasticPreconditionedGradientDescentOptimizer::GetSigmoidScale ( ) const

virtual

◆ GetUseAdaptiveStepSizes()

virtual bool itk::AdaptiveStochasticPreconditionedGradientDescentOptimizer::GetUseAdaptiveStepSizes ( ) const

virtual

◆ New()

static Pointer itk::AdaptiveStochasticPreconditionedGradientDescentOptimizer::New ( )

static

Method for creation through the object factory.

◆ operator=()

void itk::AdaptiveStochasticPreconditionedGradientDescentOptimizer::operator= ( const Self & )

private

◆ SetSigmoidMax()

virtual void itk::AdaptiveStochasticPreconditionedGradientDescentOptimizer::SetSigmoidMax ( double _arg )

virtual

Set/Get the maximum of the sigmoid. Should be >0. Default: 1.0

◆ SetSigmoidMin()

virtual void itk::AdaptiveStochasticPreconditionedGradientDescentOptimizer::SetSigmoidMin ( double _arg )

virtual

Set/Get the maximum of the sigmoid. Should be <0. Default: -0.8

◆ SetSigmoidScale()

virtual void itk::AdaptiveStochasticPreconditionedGradientDescentOptimizer::SetSigmoidScale ( double _arg )

virtual

Set/Get the scaling of the sigmoid width. Large values cause a more wide sigmoid. Default: 1e-8. Should be >0.

◆ SetUseAdaptiveStepSizes()

virtual void itk::AdaptiveStochasticPreconditionedGradientDescentOptimizer::SetUseAdaptiveStepSizes ( bool _arg )

virtual

Set/Get whether the adaptive step size mechanism is desired. Default: true

◆ UpdateCurrentTime()

virtual void itk::AdaptiveStochasticPreconditionedGradientDescentOptimizer::UpdateCurrentTime ( void )

protectedvirtual

Function to update the current time If UseAdaptiveStepSizes is false this function just increments the CurrentTime by $E_0 = (sigmoid_{max} + sigmoid_{min})/2$ . Else, the CurrentTime is updated according to:
time = max[ 0, time + sigmoid( -gradient*previoussearchdirection) ]
In that case, also the m_PreviousSearchDirection is updated.

Reimplemented from itk::StochasticPreconditionedGradientDescentOptimizer.

Field Documentation

◆ m_PreviousSearchDirection

DerivativeType itk::AdaptiveStochasticPreconditionedGradientDescentOptimizer::m_PreviousSearchDirection

protected

The Previous search direction = P g, necessary for the CruzAcceleration

Definition at line 146 of file itkAdaptiveStochasticPreconditionedGradientDescentOptimizer.h.

◆ m_SigmoidMax

double itk::AdaptiveStochasticPreconditionedGradientDescentOptimizer::m_SigmoidMax

private

Definition at line 155 of file itkAdaptiveStochasticPreconditionedGradientDescentOptimizer.h.

◆ m_SigmoidMin

double itk::AdaptiveStochasticPreconditionedGradientDescentOptimizer::m_SigmoidMin

private

Definition at line 156 of file itkAdaptiveStochasticPreconditionedGradientDescentOptimizer.h.

◆ m_SigmoidScale

double itk::AdaptiveStochasticPreconditionedGradientDescentOptimizer::m_SigmoidScale

private

Definition at line 157 of file itkAdaptiveStochasticPreconditionedGradientDescentOptimizer.h.

◆ m_UseAdaptiveStepSizes

bool itk::AdaptiveStochasticPreconditionedGradientDescentOptimizer::m_UseAdaptiveStepSizes

private

Settings

Definition at line 154 of file itkAdaptiveStochasticPreconditionedGradientDescentOptimizer.h.

Detailed Description

Public Types

Public Member Functions

Static Public Member Functions

Protected Member Functions

Protected Attributes

Private Member Functions

Private Attributes

Additional Inherited Members

Member Typedef Documentation

◆ ConstPointer

◆ CostFunctionType

◆ DerivativeType

◆ MeasureType

◆ ParametersType

◆ Pointer

◆ PreconditionType

◆ PreconditionValueType

◆ ScaledCostFunctionPointer

◆ ScaledCostFunctionType

◆ ScalesType

◆ Self

◆ StopConditionType

◆ Superclass

Constructor & Destructor Documentation

◆ AdaptiveStochasticPreconditionedGradientDescentOptimizer() [1/2]

◆ ~AdaptiveStochasticPreconditionedGradientDescentOptimizer()

◆ AdaptiveStochasticPreconditionedGradientDescentOptimizer() [2/2]

Member Function Documentation

◆ GetClassName()

◆ GetSigmoidMax()

◆ GetSigmoidMin()

◆ GetSigmoidScale()

◆ GetUseAdaptiveStepSizes()

◆ New()

◆ operator=()

◆ SetSigmoidMax()

◆ SetSigmoidMin()

◆ SetSigmoidScale()

◆ SetUseAdaptiveStepSizes()

◆ UpdateCurrentTime()

Field Documentation

◆ m_PreviousSearchDirection

◆ m_SigmoidMax

◆ m_SigmoidMin

◆ m_SigmoidScale

◆ m_UseAdaptiveStepSizes