pyFAI.engines package
pyFAI.engines.CSR_engine module
CSR rebinning engine implemented in pure python (with bits of scipy !)
- class pyFAI.engines.CSR_engine.CSRIntegrator(image_size, lut=None, empty=0.0)
Bases:
object- __init__(image_size, lut=None, empty=0.0)
Constructor of the abstract class
- Parameters:
size – input image size
lut – tuple of 3 arrays with data, indices and indptr, index of the start of line in the CSR matrix
empty – value for empty pixels
- integrate(signal, variance=None, poissonian=None, dummy=None, delta_dummy=None, dark=None, flat=None, solidangle=None, polarization=None, absorption=None, normalization_factor=1.0)
Actually perform the CSR matrix multiplication after preprocessing.
- Parameters:
signal – array of the right size with the signal in it.
variance – Variance associated with the signal
poissonian – set to use signal as variance (minimum 1), set to False to use azimuthal model.
dummy – values which have to be discarded (dynamic mask)
delta_dummy – precision for dummy values
dark – noise to be subtracted from signal
flat – flat-field normalization array
flat – solidangle normalization array
polarization – :solidangle normalization array
absorption – :absorption normalization array
normalization_factor – scale all normalization with this scalar
- Returns:
the preprocessed data integrated as array nbins x 4 which contains: regrouped signal, variance, normalization and pixel count
Nota: all normalizations are grouped in the preprocessing step.
- set_matrix(data, indices, indptr)
Actually set the CSR sparse matrix content
- Parameters:
data – the non zero values NZV
indices – the column number of the NZV
indptr – the index of the start of line
- class pyFAI.engines.CSR_engine.CsrIntegrator1d(image_size, lut=None, empty=0.0, unit=None, bin_centers=None, mask_checksum=None)
Bases:
CSRIntegrator- __init__(image_size, lut=None, empty=0.0, unit=None, bin_centers=None, mask_checksum=None)
Constructor of the abstract class for 1D integration
- Parameters:
image_size – size of the image
lut – (data, indices, indptr) of the CSR matrix
empty – value for empty pixels
unit – the kind of radial units
bin_center – position of the bin center
mask_checksum – just a place-holder to track which mask was used
Nota: bins value is deduced from the dimentionality of bin_centers
- property check_mask
- integrate(signal, variance=None, poissonian=None, dummy=None, delta_dummy=None, dark=None, flat=None, solidangle=None, polarization=None, absorption=None, normalization_factor=1.0)
Actually perform the 1D integration
- Parameters:
signal – array of the right size with the signal in it.
variance – Variance associated with the signal
poissonian – set to use signal as variance (minimum 1), set to False to use azimuthal model.
dummy – values which have to be discarded (dynamic mask)
delta_dummy – precision for dummy values
dark – noise to be subtracted from signal
flat – flat-field normalization array
flat – solidangle normalization array
polarization – :solidangle normalization array
absorption – :absorption normalization array
normalization_factor – scale all normalization with this scalar
- Returns:
Integrate1dResult or Integrate1dWithErrorResult object depending on variance
- integrate_ng(signal, variance=None, poissonian=None, dummy=None, delta_dummy=None, dark=None, flat=None, solidangle=None, polarization=None, absorption=None, normalization_factor=1.0)
Actually perform the 1D integration
- Parameters:
signal – array of the right size with the signal in it.
variance – Variance associated with the signal
poissonian – set to use signal as variance (minimum 1), set to False to use azimuthal model.
dummy – values which have to be discarded (dynamic mask)
delta_dummy – precision for dummy values
dark – noise to be subtracted from signal
flat – flat-field normalization array
flat – solidangle normalization array
polarization – :solidangle normalization array
absorption – :absorption normalization array
normalization_factor – scale all normalization with this scalar
- Returns:
Integrate1dResult or Integrate1dWithErrorResult object depending on variance
- set_matrix(data, indices, indptr)
Actually set the CSR sparse matrix content
- Parameters:
data – the non zero values NZV
indices – the column number of the NZV
indptr – the index of the start of line
- sigma_clip(data, dark=None, dummy=None, delta_dummy=None, variance=None, dark_variance=None, flat=None, solidangle=None, polarization=None, absorption=None, safe=True, error_model=None, normalization_factor=1.0, cutoff=4.0, cycle=5)
Perform a sigma-clipping iterative filter within each along each row. see the doc of scipy.stats.sigmaclip for more descriptions.
If the error model is “azimuthal”: the variance is the variance within a bin, which is refined at each iteration, can be costly !
Else, the error is propagated according to:
\[signal = (raw - dark) variance = variance + dark_variance normalization = normalization_factor*(flat * solidangle * polarization * absortoption) count = number of pixel contributing\]Integration is performed using the CSR representation of the look-up table on all arrays: signal, variance, normalization and count
Formula for azimuthal variance from: https://dbs.ifi.uni-heidelberg.de/files/Team/eschubert/publications/SSDBM18-covariance-authorcopy.pdf
- Parameters:
dark – array of same shape as data for pre-processing
dummy – value for invalid data
delta_dummy – precesion for dummy assessement
variance – array of same shape as data for pre-processing
dark_variance – array of same shape as data for pre-processing
flat – array of same shape as data for pre-processing
solidangle – array of same shape as data for pre-processing
polarization – array of same shape as data for pre-processing
safe – Unused in this implementation
normalization_factor – divide raw signal by this value
cutoff – discard all points with |value - avg| > cutoff * sigma. 3-4 is quite common
cycle – perform at maximum this number of cycles. 5 is common.
- Returns:
namedtuple with “position intensity error signal variance normalization count”
- class pyFAI.engines.CSR_engine.CsrIntegrator2d(image_size, lut=None, empty=0.0, bin_centers0=None, bin_centers1=None, checksum=None, mask_checksum=None)
Bases:
CSRIntegrator- __init__(image_size, lut=None, empty=0.0, bin_centers0=None, bin_centers1=None, checksum=None, mask_checksum=None)
Constructor of the abstract class for 2D integration
- Parameters:
size – input image size
lut – tuple of 3 arrays with data, indices and indptr, index of the start of line in the CSR matrix
empty – value for empty pixels
bin_center – position of the bin center
checksum – checksum for the LUT, if not provided, recalculated
mask_checksum – just a place-holder to track which mask was used
Nota: bins are deduced from bin_centers0, bin_centers1
- property check_mask
- integrate(signal, variance=None, poissonian=False, dummy=None, delta_dummy=None, dark=None, flat=None, solidangle=None, polarization=None, absorption=None, normalization_factor=1.0)
Actually perform the 2D integration
- Parameters:
signal – array of the right size with the signal in it.
variance – Variance associated with the signal
poissonian – set to True to variance=max(signal,1), False will implement azimuthal variance
dummy – values which have to be discarded (dynamic mask)
delta_dummy – precision for dummy values
dark – noise to be subtracted from signal
flat – flat-field normalization array
flat – solidangle normalization array
polarization – :solidangle normalization array
absorption – :absorption normalization array
normalization_factor – scale all normalization with this scalar
- Returns:
Integrate2dtpl namedtuple: “radial azimuthal intensity error signal variance normalization count”
- integrate_ng(signal, variance=None, poissonian=False, dummy=None, delta_dummy=None, dark=None, flat=None, solidangle=None, polarization=None, absorption=None, normalization_factor=1.0)
Actually perform the 2D integration
- Parameters:
signal – array of the right size with the signal in it.
variance – Variance associated with the signal
poissonian – set to True to variance=max(signal,1), False will implement azimuthal variance
dummy – values which have to be discarded (dynamic mask)
delta_dummy – precision for dummy values
dark – noise to be subtracted from signal
flat – flat-field normalization array
flat – solidangle normalization array
polarization – :solidangle normalization array
absorption – :absorption normalization array
normalization_factor – scale all normalization with this scalar
- Returns:
Integrate2dtpl namedtuple: “radial azimuthal intensity error signal variance normalization count”
- set_matrix(data, indices, indptr)
Actually set the CSR sparse matrix content
- Parameters:
data – the non zero values NZV
indices – the column number of the NZV
indptr – the index of the start of line
pyFAI.engines.histogram_engine module
simple histogram rebinning engine implemented in pure python (with the help of numpy !)
- pyFAI.engines.histogram_engine.histogram1d_engine(radial, npt, raw, dark=None, flat=None, solidangle=None, polarization=None, absorption=None, mask=None, dummy=None, delta_dummy=None, normalization_factor=1.0, empty=None, split_result=False, variance=None, dark_variance=None, poissonian=False, radial_range=None)
Implementation of rebinning engine using pure numpy histograms
- Parameters:
radial – radial position 2D array (same shape as raw)
npt – number of points to integrate over
raw – 2D array with the raw signal
dark – array containing the value of the dark noise, to be subtracted
flat – Array containing the flatfield image. It is also checked for dummies if relevant.
solidangle – the value of the solid_angle. This processing may be performed during the rebinning instead. left for compatibility
polarization – Correction for polarization of the incident beam
absorption – Correction for absorption in the sensor volume
mask – 2d array of int/bool: non-null where data should be ignored
dummy – value of invalid data
delta_dummy – precision for invalid data
normalization_factor – final value is divided by this
empty – value to be given for empty bins
variance – provide an estimation of the variance
dark_variance – provide an estimation of the variance of the dark_current,
poissonian – set to “True” for assuming the detector is poissonian and variance = raw + dark
NaN are always considered as invalid values
if neither empty nor dummy is provided, empty pixels are left at 0.
- Nota: “azimuthal_range” has to be integrated into the
mask prior to the call of this function
- Returns:
Integrate1dtpl named tuple containing: position, average intensity, std on intensity, plus the various histograms on signal, variance, normalization and count.
- pyFAI.engines.histogram_engine.histogram2d_engine(radial, azimuthal, npt, raw, dark=None, flat=None, solidangle=None, polarization=None, absorption=None, mask=None, dummy=None, delta_dummy=None, normalization_factor=1.0, empty=None, split_result=False, variance=None, dark_variance=None, poissonian=False, radial_range=None, azimuth_range=None)
Implementation of 2D rebinning engine using pure numpy histograms
- Parameters:
radial – radial position 2D array (same shape as raw)
azimuthal – azimuthal position 2D array (same shape as raw)
npt – number of points to integrate over in (radial, azimuthal) dimensions
raw – 2D array with the raw signal
dark – array containing the value of the dark noise, to be subtracted
flat – Array containing the flatfield image. It is also checked for dummies if relevant.
solidangle – the value of the solid_angle. This processing may be performed during the rebinning instead. left for compatibility
polarization – Correction for polarization of the incident beam
absorption – Correction for absorption in the sensor volume
mask – 2d array of int/bool: non-null where data should be ignored
dummy – value of invalid data
delta_dummy – precision for invalid data
normalization_factor – final value is divided by this
empty – value to be given for empty bins
variance – provide an estimation of the variance
dark_variance – provide an estimation of the variance of the dark_current,
poissonian – set to “True” for assuming the detector is poissonian and variance = raw + dark
NaN are always considered as invalid values
if neither empty nor dummy is provided, empty pixels are left at 0.
- Nota: “azimuthal_range” has to be integrated into the
mask prior to the call of this function
- Returns:
Integrate1dtpl named tuple containing: position, average intensity, std on intensity, plus the various histograms on signal, variance, normalization and count.
pyFAI.engines.preproc module
Module providing common pixel-wise pre-processing of data.
- pyFAI.engines.preproc.preproc(raw, dark=None, flat=None, solidangle=None, polarization=None, absorption=None, mask=None, dummy=None, delta_dummy=None, normalization_factor=1.0, empty=None, split_result=False, variance=None, dark_variance=None, poissonian=False, dtype=<class 'numpy.float32'>)
Common preprocessing step for all integration engines
- Parameters:
data – raw value, as a numpy array, 1D or 2D
mask – array non null where data should be ignored
dummy – value of invalid data
delta_dummy – precision for invalid data
dark – array containing the value of the dark noise, to be subtracted
flat – Array containing the flatfield image. It is also checked for dummies if relevant.
solidangle – the value of the solid_angle. This processing may be performed during the rebinning instead. left for compatibility
polarization – Correction for polarization of the incident beam
absorption – Correction for absorption in the sensor volume
normalization_factor – final value is divided by this
empty – value to be given for empty bins
split_result – set to true to separate signal from normalization and return an array of float2, float3 (with variance) ot float4 (including counts)
variance – provide an estimation of the variance, enforce split_result=True and return an float3 array with variance in second position.
dark_variance – provide an estimation of the variance of the dark_current, enforce split_result=True and return an float3 array with variance in second position.
poissonian – set to “True” for assuming the detector is poissonian and variance = max(1, raw + dark)
dtype – dtype for all processing
All calculation are performed in single precision floating point (32 bits).
NaN are always considered as invalid values
if neither empty nor dummy is provided, empty pixels are 0. Empty pixels are always zero in “split_result” mode.
When set to False, i.e the default, the pixel-wise operation is:
\[I = \frac{raw - dark}{flat \cdot solidangle \cdot polarization \cdot absorption}\]Invalid pixels are set to the dummy or empty value.
When split_result is set to True, each result is a float2 or a float3 (with an additional value for the variance) as such:
I = [\(raw - dark\), \(variance\), \(flat \cdot solidangle \cdot polarization \cdot absorption\)]
If split_result is 4, then the count of pixel is appended to the list, i.e. 1 or 0 for masked pixels Empty pixels will have all their 2 or 3 or 4 values to 0 (and not to dummy or empty value)
If poissonian is set to True, the variance is evaluated as raw + dark, with a minimum of 1.
Module contents
This sub-module contains various rebinning and pre-processing engines defined at the Python level.