Package org.apache.commons.math.stat.descriptive.moment

Class Variance

  • All Implemented Interfaces:
    java.io.Serializable, StorelessUnivariateStatistic, UnivariateStatistic, WeightedEvaluation
    public class Variance
extends AbstractStorelessUnivariateStatistic
implements java.io.Serializable, WeightedEvaluation
    Computes the variance of the available values. By default, the unbiased "sample variance" definitional formula is used:

    variance = sum((x_i - mean)^2) / (n - 1)

    where mean is the Mean and n is the number of sample observations.

    The definitional formula does not have good numerical properties, so this implementation does not compute the statistic using the definitional formula.

    • The getResult method computes the variance using updating formulas based on West's algorithm, as described in Chan, T. F. and J. G. Lewis 1979, Communications of the ACM, vol. 22 no. 9, pp. 526-531.
    • The evaluate methods leverage the fact that they have the full array of values in memory to execute a two-pass algorithm. Specifically, these methods use the "corrected two-pass algorithm" from Chan, Golub, Levesque, Algorithms for Computing the Sample Variance, American Statistician, vol. 37, no. 3 (1983) pp. 242-247.
    Note that adding values using increment or incrementAll and then executing getResult will sometimes give a different, less accurate, result than executing evaluate with the full array of values. The former approach should only be used when the full array of values is not available.

    The "population variance" ( sum((x_i - mean)^2) / n ) can also be computed using this statistic. The isBiasCorrected property determines whether the "population" or "sample" value is returned by the evaluate and getResult methods. To compute population variances, set this property to false.

    Note that this implementation is not synchronized. If multiple threads access an instance of this class concurrently, and at least one of the threads invokes the increment() or clear() method, it must be synchronized externally.

    Version:
    $Revision: 1006299 $ $Date: 2010-10-10 16:47:17 +0200 (dim. 10 oct. 2010) $
    See Also:
    Serialized Form

    • Field Summary

      Fields 
      Modifier and Type Field Description
      protected boolean incMoment
      Boolean test to determine if this Variance should also increment the second moment, this evaluates to false when this Variance is constructed with an external SecondMoment as a parameter.
      protected SecondMoment moment
      SecondMoment is used in incremental calculation of Variance

    • Constructor Summary

      Constructors 
      Constructor Description
      Variance()
      Constructs a Variance with default (true) isBiasCorrected property.
      Variance​(boolean isBiasCorrected)
      Constructs a Variance with the specified isBiasCorrected property
      Variance​(boolean isBiasCorrected, SecondMoment m2)
      Constructs a Variance with the specified isBiasCorrected property and the supplied external second moment.
      Variance​(SecondMoment m2)
      Constructs a Variance based on an external second moment.
      Variance​(Variance original)
      Copy constructor, creates a new Variance identical to the original

    • Method Summary

      All Methods Static Methods Instance Methods Concrete Methods 
      Modifier and Type Method Description
      void clear()
      Clears the internal state of the Statistic
      Variance copy()
      Returns a copy of the statistic with the same internal state.
      static void copy​(Variance source, Variance dest)
      Copies source to dest.
      double evaluate​(double[] values)
      Returns the variance of the entries in the input array, or Double.NaN if the array is empty.
      double evaluate​(double[] values, double mean)
      Returns the variance of the entries in the input array, using the precomputed mean value.
      double evaluate​(double[] values, double[] weights)
      Returns the weighted variance of the entries in the the input array.
      double evaluate​(double[] values, double[] weights, double mean)
      Returns the weighted variance of the values in the input array, using the precomputed weighted mean value.
      double evaluate​(double[] values, double[] weights, double mean, int begin, int length)
      Returns the weighted variance of the entries in the specified portion of the input array, using the precomputed weighted mean value.
      double evaluate​(double[] values, double[] weights, int begin, int length)
      Returns the weighted variance of the entries in the specified portion of the input array, or Double.NaN if the designated subarray is empty.
      double evaluate​(double[] values, double mean, int begin, int length)
      Returns the variance of the entries in the specified portion of the input array, using the precomputed mean value.
      double evaluate​(double[] values, int begin, int length)
      Returns the variance of the entries in the specified portion of the input array, or Double.NaN if the designated subarray is empty.
      long getN()
      Returns the number of values that have been added.
      double getResult()
      Returns the current value of the Statistic.
      void increment​(double d)
      Updates the internal state of the statistic to reflect the addition of the new value.
      boolean isBiasCorrected()  
      void setBiasCorrected​(boolean biasCorrected)  

      • Methods inherited from class java.lang.Object

        clone, finalize, getClass, notify, notifyAll, toString, wait, wait, wait

    • Field Detail

      • moment

        protected SecondMoment moment
        SecondMoment is used in incremental calculation of Variance

      • incMoment

        protected boolean incMoment
        Boolean test to determine if this Variance should also increment the second moment, this evaluates to false when this Variance is constructed with an external SecondMoment as a parameter.

    • Constructor Detail

      • Variance

        public Variance()
        Constructs a Variance with default (true) isBiasCorrected property.

      • Variance

        public Variance​(SecondMoment m2)
        Constructs a Variance based on an external second moment.
        Parameters:
        m2 - the SecondMoment (Third or Fourth moments work here as well.)

      • Variance

        public Variance​(boolean isBiasCorrected)
        Constructs a Variance with the specified isBiasCorrected property
        Parameters:
        isBiasCorrected - setting for bias correction - true means bias will be corrected and is equivalent to using the argumentless constructor

      • Variance

        public Variance​(boolean isBiasCorrected,
                SecondMoment m2)
        Constructs a Variance with the specified isBiasCorrected property and the supplied external second moment.
        Parameters:
        isBiasCorrected - setting for bias correction - true means bias will be corrected
        m2 - the SecondMoment (Third or Fourth moments work here as well.)

      • Variance

        public Variance​(Variance original)
        Copy constructor, creates a new Variance identical to the original
        Parameters:
        original - the Variance instance to copy

    • Method Detail

      • increment

        public void increment​(double d)
        Updates the internal state of the statistic to reflect the addition of the new value.

        If all values are available, it is more accurate to use evaluate(double[]) rather than adding values one at a time using this method and then executing getResult(), since evaluate leverages the fact that is has the full list of values together to execute a two-pass algorithm. See Variance.

        Specified by:
        increment in interface StorelessUnivariateStatistic
        Specified by:
        increment in class AbstractStorelessUnivariateStatistic
        Parameters:
        d - the new value.

      • getN

        public long getN()
        Returns the number of values that have been added.
        Specified by:
        getN in interface StorelessUnivariateStatistic
        Returns:
        the number of values.

      • evaluate

        public double evaluate​(double[] values)
        Returns the variance of the entries in the input array, or Double.NaN if the array is empty.

        See Variance for details on the computing algorithm.

        Returns 0 for a single-value (i.e. length = 1) sample.

        Throws IllegalArgumentException if the array is null.

        Does not change the internal state of the statistic.

        Specified by:
        evaluate in interface UnivariateStatistic
        Overrides:
        evaluate in class AbstractStorelessUnivariateStatistic
        Parameters:
        values - the input array
        Returns:
        the variance of the values or Double.NaN if length = 0
        Throws:
        java.lang.IllegalArgumentException - if the array is null
        See Also:
        UnivariateStatistic.evaluate(double[])

      • evaluate

        public double evaluate​(double[] values,
                       int begin,
                       int length)
        Returns the variance of the entries in the specified portion of the input array, or Double.NaN if the designated subarray is empty.

        See Variance for details on the computing algorithm.

        Returns 0 for a single-value (i.e. length = 1) sample.

        Does not change the internal state of the statistic.

        Throws IllegalArgumentException if the array is null.

        Specified by:
        evaluate in interface UnivariateStatistic
        Overrides:
        evaluate in class AbstractStorelessUnivariateStatistic
        Parameters:
        values - the input array
        begin - index of the first array element to include
        length - the number of elements to include
        Returns:
        the variance of the values or Double.NaN if length = 0
        Throws:
        java.lang.IllegalArgumentException - if the array is null or the array index parameters are not valid
        See Also:
        UnivariateStatistic.evaluate(double[], int, int)

      • evaluate

        public double evaluate​(double[] values,
                       double[] weights,
                       int begin,
                       int length)

        Returns the weighted variance of the entries in the specified portion of the input array, or Double.NaN if the designated subarray is empty.

        Uses the formula 

           Σ(weights[i]*(values[i] - weightedMean)2)/(Σ(weights[i]) - 1)
        where weightedMean is the weighted mean

        This formula will not return the same result as the unweighted variance when all weights are equal, unless all weights are equal to 1. The formula assumes that weights are to be treated as "expansion values," as will be the case if for example the weights represent frequency counts. To normalize weights so that the denominator in the variance computation equals the length of the input vector minus one, use 

           evaluate(values, MathUtils.normalizeArray(weights, values.length));

        Returns 0 for a single-value (i.e. length = 1) sample.

        Throws IllegalArgumentException if any of the following are true:

        • the values array is null
        • the weights array is null
        • the weights array does not have the same length as the values array
        • the weights array contains one or more infinite values
        • the weights array contains one or more NaN values
        • the weights array contains negative values
        • the start and length arguments do not determine a valid array

        Does not change the internal state of the statistic.

        Throws IllegalArgumentException if either array is null.

        Specified by:
        evaluate in interface WeightedEvaluation
        Parameters:
        values - the input array
        weights - the weights array
        begin - index of the first array element to include
        length - the number of elements to include
        Returns:
        the weighted variance of the values or Double.NaN if length = 0
        Throws:
        java.lang.IllegalArgumentException - if the parameters are not valid
        Since:
        2.1

      • evaluate

        public double evaluate​(double[] values,
                       double[] weights)

        Returns the weighted variance of the entries in the the input array.

        Uses the formula 

           Σ(weights[i]*(values[i] - weightedMean)2)/(Σ(weights[i]) - 1)
        where weightedMean is the weighted mean

        This formula will not return the same result as the unweighted variance when all weights are equal, unless all weights are equal to 1. The formula assumes that weights are to be treated as "expansion values," as will be the case if for example the weights represent frequency counts. To normalize weights so that the denominator in the variance computation equals the length of the input vector minus one, use 

           evaluate(values, MathUtils.normalizeArray(weights, values.length));

        Returns 0 for a single-value (i.e. length = 1) sample.

        Throws IllegalArgumentException if any of the following are true:

        • the values array is null
        • the weights array is null
        • the weights array does not have the same length as the values array
        • the weights array contains one or more infinite values
        • the weights array contains one or more NaN values
        • the weights array contains negative values

        Does not change the internal state of the statistic.

        Throws IllegalArgumentException if either array is null.

        Specified by:
        evaluate in interface WeightedEvaluation
        Parameters:
        values - the input array
        weights - the weights array
        Returns:
        the weighted variance of the values
        Throws:
        java.lang.IllegalArgumentException - if the parameters are not valid
        Since:
        2.1

      • evaluate

        public double evaluate​(double[] values,
                       double mean,
                       int begin,
                       int length)
        Returns the variance of the entries in the specified portion of the input array, using the precomputed mean value. Returns Double.NaN if the designated subarray is empty.

        See Variance for details on the computing algorithm.

        The formula used assumes that the supplied mean value is the arithmetic mean of the sample data, not a known population parameter. This method is supplied only to save computation when the mean has already been computed.

        Returns 0 for a single-value (i.e. length = 1) sample.

        Throws IllegalArgumentException if the array is null.

        Does not change the internal state of the statistic.

        Parameters:
        values - the input array
        mean - the precomputed mean value
        begin - index of the first array element to include
        length - the number of elements to include
        Returns:
        the variance of the values or Double.NaN if length = 0
        Throws:
        java.lang.IllegalArgumentException - if the array is null or the array index parameters are not valid

      • evaluate

        public double evaluate​(double[] values,
                       double mean)
        Returns the variance of the entries in the input array, using the precomputed mean value. Returns Double.NaN if the array is empty.

        See Variance for details on the computing algorithm.

        If isBiasCorrected is true the formula used assumes that the supplied mean value is the arithmetic mean of the sample data, not a known population parameter. If the mean is a known population parameter, or if the "population" version of the variance is desired, set isBiasCorrected to false before invoking this method.

        Returns 0 for a single-value (i.e. length = 1) sample.

        Throws IllegalArgumentException if the array is null.

        Does not change the internal state of the statistic.

        Parameters:
        values - the input array
        mean - the precomputed mean value
        Returns:
        the variance of the values or Double.NaN if the array is empty
        Throws:
        java.lang.IllegalArgumentException - if the array is null

      • evaluate

        public double evaluate​(double[] values,
                       double[] weights,
                       double mean,
                       int begin,
                       int length)
        Returns the weighted variance of the entries in the specified portion of the input array, using the precomputed weighted mean value. Returns Double.NaN if the designated subarray is empty.

        Uses the formula 

           Σ(weights[i]*(values[i] - mean)2)/(Σ(weights[i]) - 1)

        The formula used assumes that the supplied mean value is the weighted arithmetic mean of the sample data, not a known population parameter. This method is supplied only to save computation when the mean has already been computed.

        This formula will not return the same result as the unweighted variance when all weights are equal, unless all weights are equal to 1. The formula assumes that weights are to be treated as "expansion values," as will be the case if for example the weights represent frequency counts. To normalize weights so that the denominator in the variance computation equals the length of the input vector minus one, use 

           evaluate(values, MathUtils.normalizeArray(weights, values.length), mean);

        Returns 0 for a single-value (i.e. length = 1) sample.

        Throws IllegalArgumentException if any of the following are true:

        • the values array is null
        • the weights array is null
        • the weights array does not have the same length as the values array
        • the weights array contains one or more infinite values
        • the weights array contains one or more NaN values
        • the weights array contains negative values
        • the start and length arguments do not determine a valid array

        Does not change the internal state of the statistic.

        Parameters:
        values - the input array
        weights - the weights array
        mean - the precomputed weighted mean value
        begin - index of the first array element to include
        length - the number of elements to include
        Returns:
        the variance of the values or Double.NaN if length = 0
        Throws:
        java.lang.IllegalArgumentException - if the parameters are not valid
        Since:
        2.1

      • evaluate

        public double evaluate​(double[] values,
                       double[] weights,
                       double mean)

        Returns the weighted variance of the values in the input array, using the precomputed weighted mean value.

        Uses the formula 

           Σ(weights[i]*(values[i] - mean)2)/(Σ(weights[i]) - 1)

        The formula used assumes that the supplied mean value is the weighted arithmetic mean of the sample data, not a known population parameter. This method is supplied only to save computation when the mean has already been computed.

        This formula will not return the same result as the unweighted variance when all weights are equal, unless all weights are equal to 1. The formula assumes that weights are to be treated as "expansion values," as will be the case if for example the weights represent frequency counts. To normalize weights so that the denominator in the variance computation equals the length of the input vector minus one, use 

           evaluate(values, MathUtils.normalizeArray(weights, values.length), mean);

        Returns 0 for a single-value (i.e. length = 1) sample.

        Throws IllegalArgumentException if any of the following are true:

        • the values array is null
        • the weights array is null
        • the weights array does not have the same length as the values array
        • the weights array contains one or more infinite values
        • the weights array contains one or more NaN values
        • the weights array contains negative values

        Does not change the internal state of the statistic.

        Parameters:
        values - the input array
        weights - the weights array
        mean - the precomputed weighted mean value
        Returns:
        the variance of the values or Double.NaN if length = 0
        Throws:
        java.lang.IllegalArgumentException - if the parameters are not valid
        Since:
        2.1

      • isBiasCorrected

        public boolean isBiasCorrected()
        Returns:
        Returns the isBiasCorrected.

      • setBiasCorrected

        public void setBiasCorrected​(boolean biasCorrected)
        Parameters:
        biasCorrected - The isBiasCorrected to set.

      • copy

        public static void copy​(Variance source,
                        Variance dest)
        Copies source to dest.

        Neither source nor dest can be null.

        Parameters:
        source - Variance to copy
        dest - Variance to copy to
        Throws:
        java.lang.NullPointerException - if either source or dest is null