#include <itkStandardStochasticVarianceReducedGradientDescentOptimizer.h>

Detailed Description

This class implements a gradient descent optimizer with a decaying gain.

If is a cost function that has to be minimised, the following iterative algorithm is used to find the optimal parameters :

$x(k+1) = x(k) - a(k) dC/dx$

The gain at each iteration is defined by:

$a(k) = a / (A + k + 1)^alpha$

.

It is very suitable to be used in combination with a stochastic estimate of the gradient . For example, in image registration problems it is often advantageous to compute the metric derivative ( ) on a new set of randomly selected image samples in each iteration. You may set the parameter NewSamplesEveryIteration to "true" to achieve this effect. For more information on this strategy, you may have a look at:

S. Klein, M. Staring, J.P.W. Pluim, "Evaluation of Optimization Methods for Nonrigid Medical Image Registration using Mutual Information and B-Splines" IEEE Transactions on Image Processing, 2007, nr. 16(12), December.

This class also serves as a base class for other StochasticVarianceReducedGradient type algorithms, like the AcceleratedStochasticVarianceReducedGradientOptimizer.

See also: StandardStochasticVarianceReducedGradient, AcceleratedStochasticVarianceReducedGradientOptimizer

Definition at line 56 of file itkStandardStochasticVarianceReducedGradientDescentOptimizer.h.

Inheritance diagram for itk::StandardStochasticVarianceReducedGradientOptimizer:

Public Types
typedef SmartPointer< const Self >	ConstPointer

typedef Superclass::CostFunctionType	CostFunctionType

typedef Superclass::DerivativeType	DerivativeType

typedef Superclass::MeasureType	MeasureType

typedef Superclass::ParametersType	ParametersType

typedef SmartPointer< Self >	Pointer

typedef Superclass::ScaledCostFunctionPointer	ScaledCostFunctionPointer

typedef Superclass::ScaledCostFunctionType	ScaledCostFunctionType

typedef Superclass::ScalesType	ScalesType

typedef StandardStochasticVarianceReducedGradientOptimizer	Self

typedef Superclass::StopConditionType	StopConditionType

typedef StochasticVarianceReducedGradientDescentOptimizer	Superclass

Public Types inherited from itk::StochasticVarianceReducedGradientDescentOptimizer
typedef SmartPointer< const Self >	ConstPointer

typedef Superclass::CostFunctionType	CostFunctionType

typedef Superclass::DerivativeType	DerivativeType

typedef Superclass::MeasureType	MeasureType

typedef Superclass::ParametersType	ParametersType

typedef SmartPointer< Self >	Pointer

typedef Superclass::ScaledCostFunctionPointer	ScaledCostFunctionPointer

typedef Superclass::ScaledCostFunctionType	ScaledCostFunctionType

typedef Superclass::ScalesType	ScalesType

typedef StochasticVarianceReducedGradientDescentOptimizer	Self

enum	StopConditionType { MaximumNumberOfIterations , MetricError , MinimumStepSize , InvalidDiagonalMatrix , GradientMagnitudeTolerance , LineSearchError }

typedef ScaledSingleValuedNonLinearOptimizer	Superclass

Public Types inherited from itk::ScaledSingleValuedNonLinearOptimizer
typedef SmartPointer< const Self >	ConstPointer

typedef Superclass::CostFunctionType	CostFunctionType

typedef Superclass::DerivativeType	DerivativeType

typedef Superclass::MeasureType	MeasureType

typedef Superclass::ParametersType	ParametersType

typedef SmartPointer< Self >	Pointer

typedef ScaledCostFunctionType::Pointer	ScaledCostFunctionPointer

typedef ScaledSingleValuedCostFunction	ScaledCostFunctionType

typedef NonLinearOptimizer::ScalesType	ScalesType

typedef ScaledSingleValuedNonLinearOptimizer	Self

typedef SingleValuedNonLinearOptimizer	Superclass

Public Member Functions
void	AdvanceOneStep (void) override

virtual const char *	GetClassName () const

virtual double	GetCurrentTime () const

virtual double	GetInitialTime () const

virtual double	GetParam_a () const

virtual double	GetParam_A () const

virtual double	GetParam_alpha () const

virtual double	GetParam_beta () const

virtual void	ResetCurrentTimeToInitialTime (void)

virtual void	SetInitialTime (double _arg)

virtual void	SetParam_a (double _arg)

virtual void	SetParam_A (double _arg)

virtual void	SetParam_alpha (double _arg)

virtual void	SetParam_beta (double _arg)

void	StartOptimization (void) override

Public Member Functions inherited from itk::StochasticVarianceReducedGradientDescentOptimizer
virtual void	AdvanceOneStep (void)

virtual const char *	GetClassName () const

virtual unsigned int	GetCurrentInnerIteration () const

virtual unsigned int	GetCurrentIteration () const

virtual const DerivativeType &	GetGradient ()

virtual unsigned int	GetLBFGSMemory () const

virtual const double &	GetLearningRate ()

virtual const unsigned long &	GetNumberOfInnerIterations ()

virtual const unsigned long &	GetNumberOfIterations ()

virtual const DerivativeType &	GetPreviousGradient ()

virtual const ParametersType &	GetPreviousPosition ()

virtual const DerivativeType &	GetSearchDir ()

virtual const StopConditionType &	GetStopCondition ()

virtual const double &	GetValue ()

virtual void	MetricErrorResponse (ExceptionObject &err)

virtual void	ResumeOptimization (void)

virtual void	SetLearningRate (double _arg)

virtual void	SetNumberOfIterations (unsigned long _arg)

void	SetNumberOfWorkUnits (ThreadIdType numberOfThreads)

virtual void	SetPreviousGradient (DerivativeType _arg)

virtual void	SetPreviousPosition (ParametersType _arg)

virtual void	SetUseEigen (bool _arg)

virtual void	SetUseMultiThread (bool _arg)

virtual void	SetUseOpenMP (bool _arg)

void	StartOptimization (void) override

virtual void	StopOptimization (void)

Public Member Functions inherited from itk::ScaledSingleValuedNonLinearOptimizer
virtual const char *	GetClassName () const

const ParametersType &	GetCurrentPosition (void) const override

virtual bool	GetMaximize () const

virtual const ScaledCostFunctionType *	GetScaledCostFunction ()

virtual const ParametersType &	GetScaledCurrentPosition ()

bool	GetUseScales (void) const

virtual void	InitializeScales (void)

virtual void	MaximizeOff ()

virtual void	MaximizeOn ()

void	SetCostFunction (CostFunctionType *costFunction) override

virtual void	SetMaximize (bool _arg)

virtual void	SetUseScales (bool arg)

Static Public Member Functions
static Pointer	New ()

Static Public Member Functions inherited from itk::StochasticVarianceReducedGradientDescentOptimizer
static Pointer	New ()

Static Public Member Functions inherited from itk::ScaledSingleValuedNonLinearOptimizer
static Pointer	New ()

Protected Member Functions
virtual double	Compute_a (double k) const

virtual double	Compute_beta (double k) const

	StandardStochasticVarianceReducedGradientOptimizer ()

virtual void	UpdateCurrentTime (void)

	~StandardStochasticVarianceReducedGradientOptimizer () override

Protected Member Functions inherited from itk::StochasticVarianceReducedGradientDescentOptimizer
void	PrintSelf (std::ostream &os, Indent indent) const override

	StochasticVarianceReducedGradientDescentOptimizer ()

	~StochasticVarianceReducedGradientDescentOptimizer () override

Protected Member Functions inherited from itk::ScaledSingleValuedNonLinearOptimizer
virtual void	GetScaledDerivative (const ParametersType &parameters, DerivativeType &derivative) const

virtual MeasureType	GetScaledValue (const ParametersType &parameters) const

virtual void	GetScaledValueAndDerivative (const ParametersType &parameters, MeasureType &value, DerivativeType &derivative) const

void	PrintSelf (std::ostream &os, Indent indent) const override

	ScaledSingleValuedNonLinearOptimizer ()

void	SetCurrentPosition (const ParametersType &param) override

virtual void	SetScaledCurrentPosition (const ParametersType &parameters)

	~ScaledSingleValuedNonLinearOptimizer () override

Protected Attributes
double	m_CurrentTime

bool	m_UseConstantStep

Protected Attributes inherited from itk::StochasticVarianceReducedGradientDescentOptimizer
unsigned long	m_CurrentInnerIteration

unsigned long	m_CurrentIteration

DerivativeType	m_Gradient

unsigned long	m_LBFGSMemory

double	m_LearningRate

ParametersType	m_MeanSearchDir

unsigned long	m_NumberOfInnerIterations

unsigned long	m_NumberOfIterations

DerivativeType	m_PreviousGradient

ParametersType	m_PreviousPosition

ParametersType	m_PreviousSearchDir

ParametersType	m_SearchDir

bool	m_Stop

StopConditionType	m_StopCondition

ThreaderType::Pointer	m_Threader

double	m_Value

Protected Attributes inherited from itk::ScaledSingleValuedNonLinearOptimizer
ScaledCostFunctionPointer	m_ScaledCostFunction

ParametersType	m_ScaledCurrentPosition

Private Member Functions
void	operator= (const Self &)

	StandardStochasticVarianceReducedGradientOptimizer (const Self &)

Private Attributes
double	m_InitialTime

double	m_Param_a

double	m_Param_A

double	m_Param_alpha

double	m_Param_beta

Additional Inherited Members
Protected Types inherited from itk::StochasticVarianceReducedGradientDescentOptimizer
typedef itk::PlatformMultiThreader	ThreaderType

typedef ThreaderType::WorkUnitInfo	ThreadInfoType

Member Typedef Documentation

◆ ConstPointer

typedef SmartPointer<const Self> itk::StandardStochasticVarianceReducedGradientOptimizer::ConstPointer

Definition at line 66 of file itkStandardStochasticVarianceReducedGradientDescentOptimizer.h.

◆ CostFunctionType

typedef Superclass::CostFunctionType itk::StandardStochasticVarianceReducedGradientOptimizer::CostFunctionType

Definition at line 78 of file itkStandardStochasticVarianceReducedGradientDescentOptimizer.h.

◆ DerivativeType

typedef Superclass::DerivativeType itk::StandardStochasticVarianceReducedGradientOptimizer::DerivativeType

Definition at line 77 of file itkStandardStochasticVarianceReducedGradientDescentOptimizer.h.

◆ MeasureType

typedef Superclass::MeasureType itk::StandardStochasticVarianceReducedGradientOptimizer::MeasureType

Typedefs inherited from the superclass.

Definition at line 75 of file itkStandardStochasticVarianceReducedGradientDescentOptimizer.h.

◆ ParametersType

typedef Superclass::ParametersType itk::StandardStochasticVarianceReducedGradientOptimizer::ParametersType

Definition at line 76 of file itkStandardStochasticVarianceReducedGradientDescentOptimizer.h.

◆ Pointer

typedef SmartPointer<Self> itk::StandardStochasticVarianceReducedGradientOptimizer::Pointer

Definition at line 65 of file itkStandardStochasticVarianceReducedGradientDescentOptimizer.h.

◆ ScaledCostFunctionPointer

typedef Superclass::ScaledCostFunctionPointer itk::StandardStochasticVarianceReducedGradientOptimizer::ScaledCostFunctionPointer

Definition at line 81 of file itkStandardStochasticVarianceReducedGradientDescentOptimizer.h.

◆ ScaledCostFunctionType

typedef Superclass::ScaledCostFunctionType itk::StandardStochasticVarianceReducedGradientOptimizer::ScaledCostFunctionType

Definition at line 80 of file itkStandardStochasticVarianceReducedGradientDescentOptimizer.h.

◆ ScalesType

typedef Superclass::ScalesType itk::StandardStochasticVarianceReducedGradientOptimizer::ScalesType

Definition at line 79 of file itkStandardStochasticVarianceReducedGradientDescentOptimizer.h.

◆ Self

typedef StandardStochasticVarianceReducedGradientOptimizer itk::StandardStochasticVarianceReducedGradientOptimizer::Self

Standard ITK.

Definition at line 62 of file itkStandardStochasticVarianceReducedGradientDescentOptimizer.h.

◆ StopConditionType

typedef Superclass::StopConditionType itk::StandardStochasticVarianceReducedGradientOptimizer::StopConditionType

Definition at line 82 of file itkStandardStochasticVarianceReducedGradientDescentOptimizer.h.

◆ Superclass

typedef StochasticVarianceReducedGradientDescentOptimizer itk::StandardStochasticVarianceReducedGradientOptimizer::Superclass

Definition at line 63 of file itkStandardStochasticVarianceReducedGradientDescentOptimizer.h.

Constructor & Destructor Documentation

◆ StandardStochasticVarianceReducedGradientOptimizer() [1/2]

itk::StandardStochasticVarianceReducedGradientOptimizer::StandardStochasticVarianceReducedGradientOptimizer ( )

protected

◆ ~StandardStochasticVarianceReducedGradientOptimizer()

itk::StandardStochasticVarianceReducedGradientOptimizer::~StandardStochasticVarianceReducedGradientOptimizer ( )

inlineoverrideprotected

Definition at line 130 of file itkStandardStochasticVarianceReducedGradientDescentOptimizer.h.

◆ StandardStochasticVarianceReducedGradientOptimizer() [2/2]

itk::StandardStochasticVarianceReducedGradientOptimizer::StandardStochasticVarianceReducedGradientOptimizer ( const Self & )

private

Member Function Documentation

◆ AdvanceOneStep()

void itk::StandardStochasticVarianceReducedGradientOptimizer::AdvanceOneStep ( void )

overridevirtual

Sets a new LearningRate before calling the Superclass' implementation, and updates the current time.

Reimplemented from itk::StochasticVarianceReducedGradientDescentOptimizer.

◆ Compute_a()

virtual double itk::StandardStochasticVarianceReducedGradientOptimizer::Compute_a ( double k ) const

protectedvirtual

Function to compute the step size for SGD at time/iteration k.

◆ Compute_beta()

virtual double itk::StandardStochasticVarianceReducedGradientOptimizer::Compute_beta ( double k ) const

protectedvirtual

Function to compute the step size for SQN at time/iteration k.

◆ GetClassName()

virtual const char * itk::StandardStochasticVarianceReducedGradientOptimizer::GetClassName ( ) const

virtual

Run-time type information (and related methods).

Reimplemented from itk::StochasticVarianceReducedGradientDescentOptimizer.

Reimplemented in elastix::AdaptiveStochasticVarianceReducedGradient< TElastix >, and itk::AdaptiveStochasticVarianceReducedGradientOptimizer.

◆ GetCurrentTime()

virtual double itk::StandardStochasticVarianceReducedGradientOptimizer::GetCurrentTime ( ) const

virtual

Get the current time. This equals the CurrentIteration in this base class but may be different in inheriting classes, such as the AccelerateStochasticVarianceReducedGradient

◆ GetInitialTime()

virtual double itk::StandardStochasticVarianceReducedGradientOptimizer::GetInitialTime ( ) const

virtual

◆ GetParam_a()

virtual double itk::StandardStochasticVarianceReducedGradientOptimizer::GetParam_a ( ) const

virtual

◆ GetParam_A()

virtual double itk::StandardStochasticVarianceReducedGradientOptimizer::GetParam_A ( ) const

virtual

◆ GetParam_alpha()

virtual double itk::StandardStochasticVarianceReducedGradientOptimizer::GetParam_alpha ( ) const

virtual

◆ GetParam_beta()

virtual double itk::StandardStochasticVarianceReducedGradientOptimizer::GetParam_beta ( ) const

virtual

◆ New()

static Pointer itk::StandardStochasticVarianceReducedGradientOptimizer::New ( )

static

Method for creation through the object factory.

◆ operator=()

void itk::StandardStochasticVarianceReducedGradientOptimizer::operator= ( const Self & )

private

◆ ResetCurrentTimeToInitialTime()

virtual void itk::StandardStochasticVarianceReducedGradientOptimizer::ResetCurrentTimeToInitialTime ( void )

inlinevirtual

Set the current time to the initial time. This can be useful to 'reset' the optimisation, for example if you changed the cost function while optimisation. Be careful with this function.

Definition at line 122 of file itkStandardStochasticVarianceReducedGradientDescentOptimizer.h.

◆ SetInitialTime()

virtual void itk::StandardStochasticVarianceReducedGradientOptimizer::SetInitialTime ( double _arg )

virtual

Set/Get the initial time. Should be >=0. This function is superfluous, since Param_A does effectively the same. However, in inheriting classes, like the AcceleratedStochasticVarianceReducedGradient the initial time may have a different function than Param_A. Default: 0.0

◆ SetParam_a()

virtual void itk::StandardStochasticVarianceReducedGradientOptimizer::SetParam_a ( double _arg )

virtual

Set/Get a.

◆ SetParam_A()

virtual void itk::StandardStochasticVarianceReducedGradientOptimizer::SetParam_A ( double _arg )

virtual

Set/Get A.

◆ SetParam_alpha()

virtual void itk::StandardStochasticVarianceReducedGradientOptimizer::SetParam_alpha ( double _arg )

virtual

Set/Get alpha.

◆ SetParam_beta()

virtual void itk::StandardStochasticVarianceReducedGradientOptimizer::SetParam_beta ( double _arg )

virtual

Set/Get beta.

◆ StartOptimization()

void itk::StandardStochasticVarianceReducedGradientOptimizer::StartOptimization ( void )

override

Set current time to 0 and call superclass' implementation.

◆ UpdateCurrentTime()

virtual void itk::StandardStochasticVarianceReducedGradientOptimizer::UpdateCurrentTime ( void )

protectedvirtual

Function to update the current time This function just increments the CurrentTime by 1. Inheriting functions may implement something smarter, for example, dependent on the progress.

Reimplemented in itk::AdaptiveStochasticVarianceReducedGradientOptimizer.

Field Documentation

◆ m_CurrentTime

double itk::StandardStochasticVarianceReducedGradientOptimizer::m_CurrentTime

protected

The current time, which serves as input for Compute_a

Definition at line 146 of file itkStandardStochasticVarianceReducedGradientDescentOptimizer.h.

◆ m_InitialTime

double itk::StandardStochasticVarianceReducedGradientOptimizer::m_InitialTime

private

Settings

Definition at line 163 of file itkStandardStochasticVarianceReducedGradientDescentOptimizer.h.

◆ m_Param_a

double itk::StandardStochasticVarianceReducedGradientOptimizer::m_Param_a

private

Parameters, as described by Spall.

Definition at line 157 of file itkStandardStochasticVarianceReducedGradientDescentOptimizer.h.

◆ m_Param_A

double itk::StandardStochasticVarianceReducedGradientOptimizer::m_Param_A

private

Definition at line 159 of file itkStandardStochasticVarianceReducedGradientDescentOptimizer.h.

◆ m_Param_alpha

double itk::StandardStochasticVarianceReducedGradientOptimizer::m_Param_alpha

private

Definition at line 160 of file itkStandardStochasticVarianceReducedGradientDescentOptimizer.h.

◆ m_Param_beta

double itk::StandardStochasticVarianceReducedGradientOptimizer::m_Param_beta

private

Definition at line 158 of file itkStandardStochasticVarianceReducedGradientDescentOptimizer.h.

◆ m_UseConstantStep

bool itk::StandardStochasticVarianceReducedGradientOptimizer::m_UseConstantStep

protected

Constant step size or others, different value of k.

Definition at line 149 of file itkStandardStochasticVarianceReducedGradientDescentOptimizer.h.

Detailed Description

Public Types

Public Member Functions

Static Public Member Functions

Protected Member Functions

Protected Attributes

Private Member Functions

Private Attributes

Additional Inherited Members

Member Typedef Documentation

◆ ConstPointer

◆ CostFunctionType

◆ DerivativeType

◆ MeasureType

◆ ParametersType

◆ Pointer

◆ ScaledCostFunctionPointer

◆ ScaledCostFunctionType

◆ ScalesType

◆ Self

◆ StopConditionType

◆ Superclass

Constructor & Destructor Documentation

◆ StandardStochasticVarianceReducedGradientOptimizer() [1/2]

◆ ~StandardStochasticVarianceReducedGradientOptimizer()

◆ StandardStochasticVarianceReducedGradientOptimizer() [2/2]

Member Function Documentation

◆ AdvanceOneStep()

◆ Compute_a()

◆ Compute_beta()

◆ GetClassName()

◆ GetCurrentTime()

◆ GetInitialTime()

◆ GetParam_a()

◆ GetParam_A()

◆ GetParam_alpha()

◆ GetParam_beta()

◆ New()

◆ operator=()

◆ ResetCurrentTimeToInitialTime()

◆ SetInitialTime()

◆ SetParam_a()

◆ SetParam_A()

◆ SetParam_alpha()

◆ SetParam_beta()

◆ StartOptimization()

◆ UpdateCurrentTime()

Field Documentation

◆ m_CurrentTime

◆ m_InitialTime

◆ m_Param_a

◆ m_Param_A

◆ m_Param_alpha

◆ m_Param_beta

◆ m_UseConstantStep