go home Home | Main Page | Modules | Namespace List | Class Hierarchy | Alphabetical List | Data Structures | File List | Namespace Members | Data Fields | Globals | Related Pages
Loading...
Searching...
No Matches
Public Types | Public Member Functions | Static Public Member Functions | Protected Member Functions | Protected Attributes | Private Attributes
itk::AdaptiveStochasticVarianceReducedGradientOptimizer Class Reference
Components » Optimizers

#include <itkAdaptiveStochasticVarianceReducedGradientOptimizer.h>

Detailed Description

This class implements a gradient descent optimizer with adaptive gain.

If $C(x)$ is a cost function that has to be minimized, the following iterative algorithm is used to find the optimal parameters $x$:

\[ x(k+1) = x(k) - a(t_k) dC/dx \]

The gain $a(t_k)$ at each iteration $k$ is defined by:

\[ a(t_k) = a / (A + t_k + 1)^alpha \]

.

And the time $t_k$ is updated according to:

\[ t_{k+1} = [ t_k + sigmoid( -g_k^T g_{k-1} ) ]^+ \]

where $g_k$ equals $dC/dx$ at iteration $k$. For $t_0$ the InitialTime is used, which is defined in the the superclass (StandardGradientDescentOptimizer). Whereas in the superclass this parameter is superfluous, in this class it makes sense.

This method is described in the following references:

[1] P. Cruz, "Almost sure convergence and asymptotical normality of a generalization of Kesten's stochastic approximation algorithm for multidimensional case." Technical Report, 2005. http://hdl.handle.net/2052/74

[2] S. Klein, J.P.W. Pluim, and M. Staring, M.A. Viergever, "Adaptive stochastic gradient descent optimisation for image registration," International Journal of Computer Vision, vol. 81, no. 3, pp. 227-239, 2009. http://dx.doi.org/10.1007/s11263-008-0168-y It is very suitable to be used in combination with a stochastic estimate of the gradient $dC/dx$. For example, in image registration problems it is often advantageous to compute the metric derivative ( $dC/dx$) on a new set of randomly selected image samples in each iteration. You may set the parameter NewSamplesEveryIteration to "true" to achieve this effect. For more information on this strategy, you may have a look at:

See also
AdaptiveStochasticVarianceReducedGradient, StandardGradientDescentOptimizer

If $C(x)$ is a costfunction that has to be minimised, the following iterative algorithm is used to find the optimal parameters $x$:

\[ x(k+1) = x(k) - a(t_k) dC/dx \]

The gain $a(t_k)$ at each iteration $k$ is defined by:

\[ a(t_k) = a / (A + t_k + 1)^alpha \]

.

And the time $t_k$ is updated according to:

\[ t_{k+1} = [ t_k + sigmoid( -g_k^T g_{k-1} ) ]^+ \]

where $g_k$ equals $dC/dx$ at iteration $k$. For $t_0$ the InitialTime is used, which is defined in the the superclass (StandardGradientDescentOptimizer). Whereas in the superclass this parameter is superfluous, in this class it makes sense.

This method is described in the following references:

[1] P. Cruz, "Almost sure convergence and asymptotical normality of a generalization of Kesten's stochastic approximation algorithm for multidimensional case." Technical Report, 2005. http://hdl.handle.net/2052/74

[2] S. Klein, J.P.W. Pluim, and M. Staring, M.A. Viergever, "Adaptive stochastic gradient descent optimisation for image registration," International Journal of Computer Vision, vol. 81, no. 3, pp. 227-239, 2009. http://dx.doi.org/10.1007/s11263-008-0168-y

It is very suitable to be used in combination with a stochastic estimate of the gradient $dC/dx$. For example, in image registration problems it is often advantageous to compute the metric derivative ( $dC/dx$) on a new set of randomly selected image samples in each iteration. You may set the parameter NewSamplesEveryIteration to "true" to achieve this effect. For more information on this strategy, you may have a look at:

See also
AdaptiveStochasticVarianceReducedGradient, StandardGradientDescentOptimizer

Definition at line 70 of file itkAdaptiveStochasticVarianceReducedGradientOptimizer.h.

+ Inheritance diagram for itk::AdaptiveStochasticVarianceReducedGradientOptimizer:

Public Types

using ConstPointer = SmartPointer< const Self >
 
using Pointer = SmartPointer< Self >
 
using Self = AdaptiveStochasticVarianceReducedGradientOptimizer
 
using Superclass = StandardStochasticVarianceReducedGradientOptimizer
 
- Public Types inherited from itk::StandardStochasticVarianceReducedGradientOptimizer
using ConstPointer = SmartPointer< const Self >
 
using Pointer = SmartPointer< Self >
 
using Self = StandardStochasticVarianceReducedGradientOptimizer
 
enum  StopConditionType
 
using Superclass = StochasticVarianceReducedGradientDescentOptimizer
 
- Public Types inherited from itk::StochasticVarianceReducedGradientDescentOptimizer
using ConstPointer = SmartPointer< const Self >
 
using Pointer = SmartPointer< Self >
 
using ScaledCostFunctionPointer = ScaledCostFunctionType::Pointer
 
using ScaledCostFunctionType = ScaledSingleValuedCostFunction
 
using ScalesType = NonLinearOptimizer::ScalesType
 
using Self = StochasticVarianceReducedGradientDescentOptimizer
 
enum  StopConditionType {
  MaximumNumberOfIterations , MetricError , MinimumStepSize , InvalidDiagonalMatrix ,
  GradientMagnitudeTolerance , LineSearchError
}
 
using Superclass = ScaledSingleValuedNonLinearOptimizer
 
- Public Types inherited from itk::ScaledSingleValuedNonLinearOptimizer
using ConstPointer = SmartPointer< const Self >
 
using Pointer = SmartPointer< Self >
 
using ScaledCostFunctionPointer = ScaledCostFunctionType::Pointer
 
using ScaledCostFunctionType = ScaledSingleValuedCostFunction
 
using ScalesType = NonLinearOptimizer::ScalesType
 
using Self = ScaledSingleValuedNonLinearOptimizer
 
using Superclass = SingleValuedNonLinearOptimizer
 

Public Member Functions

virtual const char * GetClassName () const
 
virtual double GetSigmoidMax () const
 
virtual double GetSigmoidMin () const
 
virtual double GetSigmoidScale () const
 
virtual bool GetUseAdaptiveStepSizes () const
 
 ITK_DISALLOW_COPY_AND_MOVE (AdaptiveStochasticVarianceReducedGradientOptimizer)
 
virtual void SetSigmoidMax (double _arg)
 
virtual void SetSigmoidMin (double _arg)
 
virtual void SetSigmoidScale (double _arg)
 
virtual void SetUseAdaptiveStepSizes (bool _arg)
 
- Public Member Functions inherited from itk::StandardStochasticVarianceReducedGradientOptimizer
void AdvanceOneStep () override
 
virtual double GetCurrentTime () const
 
virtual double GetInitialTime () const
 
virtual double GetParam_a () const
 
virtual double GetParam_A () const
 
virtual double GetParam_alpha () const
 
virtual double GetParam_beta () const
 
 ITK_DISALLOW_COPY_AND_MOVE (StandardStochasticVarianceReducedGradientOptimizer)
 
virtual void ResetCurrentTimeToInitialTime ()
 
virtual void SetInitialTime (double _arg)
 
virtual void SetParam_a (double _arg)
 
virtual void SetParam_A (double _arg)
 
virtual void SetParam_alpha (double _arg)
 
virtual void SetParam_beta (double _arg)
 
void StartOptimization () override
 
- Public Member Functions inherited from itk::StochasticVarianceReducedGradientDescentOptimizer
virtual unsigned int GetCurrentInnerIteration () const
 
virtual unsigned int GetCurrentIteration () const
 
virtual const DerivativeType & GetGradient ()
 
virtual unsigned int GetLBFGSMemory () const
 
virtual const double & GetLearningRate ()
 
virtual const unsigned long & GetNumberOfInnerIterations ()
 
virtual const unsigned long & GetNumberOfIterations ()
 
virtual const DerivativeType & GetPreviousGradient ()
 
virtual const ParametersType & GetPreviousPosition ()
 
virtual const DerivativeType & GetSearchDir ()
 
virtual const StopConditionType & GetStopCondition ()
 
virtual const double & GetValue ()
 
 ITK_DISALLOW_COPY_AND_MOVE (StochasticVarianceReducedGradientDescentOptimizer)
 
virtual void MetricErrorResponse (ExceptionObject &err)
 
virtual void ResumeOptimization ()
 
virtual void SetLearningRate (double _arg)
 
virtual void SetNumberOfIterations (unsigned long _arg)
 
void SetNumberOfWorkUnits (ThreadIdType numberOfThreads)
 
virtual void SetPreviousGradient (DerivativeType _arg)
 
virtual void SetPreviousPosition (ParametersType _arg)
 
virtual void SetUseEigen (bool _arg)
 
virtual void SetUseMultiThread (bool _arg)
 
void StartOptimization () override
 
virtual void StopOptimization ()
 
- Public Member Functions inherited from itk::ScaledSingleValuedNonLinearOptimizer
const ParametersType & GetCurrentPosition () const override
 
virtual bool GetMaximize () const
 
virtual const ScaledCostFunctionType * GetScaledCostFunction ()
 
virtual const ParametersType & GetScaledCurrentPosition ()
 
bool GetUseScales () const
 
virtual void InitializeScales ()
 
 ITK_DISALLOW_COPY_AND_MOVE (ScaledSingleValuedNonLinearOptimizer)
 
virtual void MaximizeOff ()
 
virtual void MaximizeOn ()
 
void SetCostFunction (CostFunctionType *costFunction) override
 
virtual void SetMaximize (bool _arg)
 
virtual void SetUseScales (bool arg)
 

Static Public Member Functions

static Pointer New ()
 
- Static Public Member Functions inherited from itk::StandardStochasticVarianceReducedGradientOptimizer
static Pointer New ()
 
- Static Public Member Functions inherited from itk::StochasticVarianceReducedGradientDescentOptimizer
static Pointer New ()
 
- Static Public Member Functions inherited from itk::ScaledSingleValuedNonLinearOptimizer
static Pointer New ()
 

Protected Member Functions

 AdaptiveStochasticVarianceReducedGradientOptimizer ()
 
void UpdateCurrentTime () override
 
 ~AdaptiveStochasticVarianceReducedGradientOptimizer () override=default
 
- Protected Member Functions inherited from itk::StandardStochasticVarianceReducedGradientOptimizer
virtual double Compute_a (double k) const
 
virtual double Compute_beta (double k) const
 
 StandardStochasticVarianceReducedGradientOptimizer ()
 
 ~StandardStochasticVarianceReducedGradientOptimizer () override=default
 
- Protected Member Functions inherited from itk::StochasticVarianceReducedGradientDescentOptimizer
void PrintSelf (std::ostream &os, Indent indent) const override
 
 StochasticVarianceReducedGradientDescentOptimizer ()
 
 ~StochasticVarianceReducedGradientDescentOptimizer () override=default
 
- Protected Member Functions inherited from itk::ScaledSingleValuedNonLinearOptimizer
virtual void GetScaledDerivative (const ParametersType &parameters, DerivativeType &derivative) const
 
virtual MeasureType GetScaledValue (const ParametersType &parameters) const
 
virtual void GetScaledValueAndDerivative (const ParametersType &parameters, MeasureType &value, DerivativeType &derivative) const
 
void PrintSelf (std::ostream &os, Indent indent) const override
 
 ScaledSingleValuedNonLinearOptimizer ()
 
void SetCurrentPosition (const ParametersType &param) override
 
virtual void SetScaledCurrentPosition (const ParametersType &parameters)
 
 ~ScaledSingleValuedNonLinearOptimizer () override=default
 

Protected Attributes

DerivativeType m_PreviousGradient {}
 
- Protected Attributes inherited from itk::StandardStochasticVarianceReducedGradientOptimizer
double m_CurrentTime { 0.0 }
 
bool m_UseConstantStep {}
 
- Protected Attributes inherited from itk::StochasticVarianceReducedGradientDescentOptimizer
unsigned long m_CurrentInnerIteration {}
 
unsigned long m_CurrentIteration { 0 }
 
DerivativeType m_Gradient {}
 
unsigned long m_LBFGSMemory { 0 }
 
double m_LearningRate { 1.0 }
 
ParametersType m_MeanSearchDir {}
 
unsigned long m_NumberOfInnerIterations {}
 
unsigned long m_NumberOfIterations { 100 }
 
DerivativeType m_PreviousGradient {}
 
ParametersType m_PreviousPosition {}
 
ParametersType m_PreviousSearchDir {}
 
ParametersType m_SearchDir {}
 
bool m_Stop { false }
 
StopConditionType m_StopCondition { MaximumNumberOfIterations }
 
MultiThreaderBase::Pointer m_Threader { MultiThreaderBase::New() }
 
double m_Value { 0.0 }
 
- Protected Attributes inherited from itk::ScaledSingleValuedNonLinearOptimizer
ScaledCostFunctionPointer m_ScaledCostFunction {}
 
ParametersType m_ScaledCurrentPosition {}
 

Private Attributes

double m_SigmoidMax { 1.0 }
 
double m_SigmoidMin { -0.8 }
 
double m_SigmoidScale { 1e-8 }
 
bool m_UseAdaptiveStepSizes { true }
 

Additional Inherited Members

- Protected Types inherited from itk::StochasticVarianceReducedGradientDescentOptimizer
using ThreadInfoType = MultiThreaderBase::WorkUnitInfo
 

Member Typedef Documentation

◆ ConstPointer

using itk::AdaptiveStochasticVarianceReducedGradientOptimizer::ConstPointer = SmartPointer<const Self>

Definition at line 80 of file itkAdaptiveStochasticVarianceReducedGradientOptimizer.h.

◆ Pointer

using itk::AdaptiveStochasticVarianceReducedGradientOptimizer::Pointer = SmartPointer<Self>

Definition at line 79 of file itkAdaptiveStochasticVarianceReducedGradientOptimizer.h.

◆ Self

using itk::AdaptiveStochasticVarianceReducedGradientOptimizer::Self = AdaptiveStochasticVarianceReducedGradientOptimizer

Standard ITK.

Definition at line 76 of file itkAdaptiveStochasticVarianceReducedGradientOptimizer.h.

◆ Superclass

using itk::AdaptiveStochasticVarianceReducedGradientOptimizer::Superclass = StandardStochasticVarianceReducedGradientOptimizer

Definition at line 77 of file itkAdaptiveStochasticVarianceReducedGradientOptimizer.h.

Constructor & Destructor Documentation

◆ AdaptiveStochasticVarianceReducedGradientOptimizer()

itk::AdaptiveStochasticVarianceReducedGradientOptimizer::AdaptiveStochasticVarianceReducedGradientOptimizer ( )
protected

◆ ~AdaptiveStochasticVarianceReducedGradientOptimizer()

itk::AdaptiveStochasticVarianceReducedGradientOptimizer::~AdaptiveStochasticVarianceReducedGradientOptimizer ( )
overrideprotecteddefault

Member Function Documentation

◆ GetClassName()

virtual const char * itk::AdaptiveStochasticVarianceReducedGradientOptimizer::GetClassName ( ) const
virtual

Run-time type information (and related methods).

Reimplemented from itk::StandardStochasticVarianceReducedGradientOptimizer.

Reimplemented in elastix::AdaptiveStochasticVarianceReducedGradient< TElastix >.

◆ GetSigmoidMax()

virtual double itk::AdaptiveStochasticVarianceReducedGradientOptimizer::GetSigmoidMax ( ) const
virtual

◆ GetSigmoidMin()

virtual double itk::AdaptiveStochasticVarianceReducedGradientOptimizer::GetSigmoidMin ( ) const
virtual

◆ GetSigmoidScale()

virtual double itk::AdaptiveStochasticVarianceReducedGradientOptimizer::GetSigmoidScale ( ) const
virtual

◆ GetUseAdaptiveStepSizes()

virtual bool itk::AdaptiveStochasticVarianceReducedGradientOptimizer::GetUseAdaptiveStepSizes ( ) const
virtual

◆ ITK_DISALLOW_COPY_AND_MOVE()

itk::AdaptiveStochasticVarianceReducedGradientOptimizer::ITK_DISALLOW_COPY_AND_MOVE ( AdaptiveStochasticVarianceReducedGradientOptimizer  )

◆ New()

static Pointer itk::AdaptiveStochasticVarianceReducedGradientOptimizer::New ( )
static

Method for creation through the object factory.

◆ SetSigmoidMax()

virtual void itk::AdaptiveStochasticVarianceReducedGradientOptimizer::SetSigmoidMax ( double  _arg)
virtual

Set/Get the maximum of the sigmoid. Should be >0. Default: 1.0

◆ SetSigmoidMin()

virtual void itk::AdaptiveStochasticVarianceReducedGradientOptimizer::SetSigmoidMin ( double  _arg)
virtual

Set/Get the maximum of the sigmoid. Should be <0. Default: -0.8

◆ SetSigmoidScale()

virtual void itk::AdaptiveStochasticVarianceReducedGradientOptimizer::SetSigmoidScale ( double  _arg)
virtual

Set/Get the scaling of the sigmoid width. Large values cause a more wide sigmoid. Default: 1e-8. Should be >0.

◆ SetUseAdaptiveStepSizes()

virtual void itk::AdaptiveStochasticVarianceReducedGradientOptimizer::SetUseAdaptiveStepSizes ( bool  _arg)
virtual

Set/Get whether the adaptive step size mechanism is desired. Default: true

◆ UpdateCurrentTime()

void itk::AdaptiveStochasticVarianceReducedGradientOptimizer::UpdateCurrentTime ( )
overrideprotectedvirtual

Function to update the current time If UseAdaptiveStepSizes is false this function just increments the CurrentTime by $E_0 = (sigmoid_{max} + sigmoid_{min})/2$. Else, the CurrentTime is updated according to:
time = max[ 0, time + sigmoid( -gradient*previousgradient) ]
In that case, also the m_PreviousGradient is updated.

Reimplemented from itk::StandardStochasticVarianceReducedGradientOptimizer.

Field Documentation

◆ m_PreviousGradient

DerivativeType itk::AdaptiveStochasticVarianceReducedGradientOptimizer::m_PreviousGradient {}
protected

The PreviousGradient, necessary for the CruzAcceleration

Definition at line 132 of file itkAdaptiveStochasticVarianceReducedGradientOptimizer.h.

◆ m_SigmoidMax

double itk::AdaptiveStochasticVarianceReducedGradientOptimizer::m_SigmoidMax { 1.0 }
private

Definition at line 137 of file itkAdaptiveStochasticVarianceReducedGradientOptimizer.h.

◆ m_SigmoidMin

double itk::AdaptiveStochasticVarianceReducedGradientOptimizer::m_SigmoidMin { -0.8 }
private

Definition at line 138 of file itkAdaptiveStochasticVarianceReducedGradientOptimizer.h.

◆ m_SigmoidScale

double itk::AdaptiveStochasticVarianceReducedGradientOptimizer::m_SigmoidScale { 1e-8 }
private

Definition at line 139 of file itkAdaptiveStochasticVarianceReducedGradientOptimizer.h.

◆ m_UseAdaptiveStepSizes

bool itk::AdaptiveStochasticVarianceReducedGradientOptimizer::m_UseAdaptiveStepSizes { true }
private

Settings

Definition at line 136 of file itkAdaptiveStochasticVarianceReducedGradientOptimizer.h.


Generated on 1739326392 for elastix by doxygen 1.9.8 elastix logo