Performing the azimutal integration from shell scripts
There are three main tools for performing azimuthal integration from the command line. Two of them specicalized for SAXS and WAXS and the last which is common to the GUI but we will see how to use it in command line mode (CLI).
They all require 3 mandatory parameters:
The PONI-file containing the geometry of the experimental setup
The number of bins used for the histogram
The list of input files
The three tools differ only by the default options selected to fit the best the needs of a given user community.
Azimuthal integration for powder diffraction
The pyFAI-waxs which stands for wide angle scattering produces integrated patterns in \(2\theta\) units by default.
Note: while this cookbook is performed in a Jupyter notebook, all commands are run in the command line interpreter (Bash), not in Python.
[1]:
pyFAI-waxs -p LaB6_29.4keV.poni -n 1000 LaB6_29.4keV.tif
Detector Detector Spline= /users/kieffer/workspace-400/pyFAI/doc/source/usage/cookbook/F_K4320T_Cam43_30012013_distorsion.spline PixelSize= 5.168e-05, 5.126e-05 m
Wavelength= 4.217150e-11m
SampleDetDist= 1.182208e-01m PONI= 5.394843e-02, 5.551600e-02m rot1=0.006974 rot2= -0.003313 rot3= -0.000000 rad
DirectBeamDist= 118.224mm Center: x=1066.839, y=1036.336 pix Tilt=0.442 deg tiltPlanRotation= -154.594 deg
Mask: None Method: splitpixel
Integrating LaB6_29.4keV.tif --> LaB6_29.4keV.xy, reading: 0.022s 1D integration: 1.789s.
[2]:
#Let's have a look at the header of the produces ascii file:
head -n 50 LaB6_29.4keV.xy
# == pyFAI calibration ==
# Distance Sample to Detector: 0.118220810284 m
# PONI: 5.395e-02, 5.552e-02 m
# Rotations: 0.006974 -0.003313 -0.000000 rad
#
# == Fit2d calibration ==
# Distance Sample-beamCenter: 118.224 mm
# Center: x=1066.839, y=1036.336 pix
# Tilt: 0.442 deg TiltPlanRot: -154.594 deg
#
# Detector Detector Spline= /users/kieffer/workspace-400/pyFAI/doc/source/usage/cookbook/F_K4320T_Cam43_30012013_distorsion.spline PixelSize= 5.168e-05, 5.126e-05 m
# Detector has a mask: False
# Detector has a dark current: False
# detector has a flat field: False
#
# Wavelength: 4.21714957131e-11 m
# Mask applied: False
# Dark current applied: False
# Flat field applied: False
# Polarization factor: None
# Normalization factor: 1.0
#
# Headers of the input frame:
# {
# "info": {},
# "photometricInterpretation": 1,
# "rowsPerStrip": 2048,
# "nColumns": 2048,
# "compression": false,
# "sampleFormat": 3,
# "imageDescription": "{\nEDF_DataBlockID=0.Image.Psd\nEDF_BinarySize=16777216\nEDF_HeaderSize=1536\nByteOrder=LowByteFirst\nDataType=FloatValue\nDim_1=2048\nDim_2=2048\nImage=0\nHeaderID=EH:000000:000000:000000\nSize=16777216\nmethod=max\nnframes=21\ncutoff=None\nmerged_file_00=ref_lab6_0001.edf\nmerged_file_01=ref_lab6_0002.edf\nmerged_file_02=ref_lab6_0003.edf\nmerged_file_03=ref_lab6_0004.edf\nmerged_file_04=ref_lab6_0005.edf\nmerged_file_05=ref_lab6_0006.edf\nmerged_file_06=ref_lab6_0007.edf\nmerged_file_07=ref_lab6_0008.edf\nmerged_file_08=ref_lab6_0009.edf\nmerged_file_09=ref_lab6_0010.edf\nmerged_file_10=ref_lab6_0011.edf\nmerged_file_11=ref_lab6_0012.edf\nmerged_file_12=ref_lab6_0013.edf\nmerged_file_13=ref_lab6_0014.edf\nmerged_file_14=ref_lab6_0015.edf\nmerged_file_15=ref_lab6_0016.edf\nmerged_file_16=ref_lab6_0017.edf\nmerged_file_17=ref_lab6_0018.edf\nmerged_file_18=ref_lab6_0019.edf\nmerged_file_19=ref_lab6_0020.edf\nmerged_file_20=ref_lab6_0021.edf\nfilename=LaB6_29.4keV.tif\n",
# "nRows": 2048,
# "colormap": null,
# "nBits": 32,
# "date": "Fri Mar 2 10:02:31 2018",
# "software": "fabio.tifimage",
# "compression_type": 1,
# "stripOffsets": [
# 1143
# ],
# "stripByteCounts": [
# 16777216
# ]
# }
# --> LaB6_29.4keV.xy
# 2th_deg I
1.668855e-02 2.425633e+00
5.006564e-02 2.804243e+00
8.344272e-02 2.121904e+00
1.168198e-01 2.734025e+00
[3]:
pyFAI-waxs -h
usage: pyFAI-waxs [options] -p ponifile file1.edf file2.edf ...
Azimuthal integration for powder diffraction.
positional arguments:
FILE Image files to integrate
optional arguments:
-h, --help show this help message and exit
-v, --version show program's version number and exit
-p PONIFILE PyFAI parameter file (.poni)
-n NPT, --npt NPT Number of points in radial dimension
-w WAVELENGTH, --wavelength WAVELENGTH
wavelength of the X-Ray beam in Angstrom
-e ENERGY, --energy ENERGY
energy of the X-Ray beam in keV
(hc=12.3984197396keV.A)
-u DUMMY, --dummy DUMMY
dummy value for dead pixels
-U DELTA_DUMMY, --delta_dummy DELTA_DUMMY
delta dummy value
-m MASK, --mask MASK name of the file containing the mask image
-d DARK, --dark DARK name of the file containing the dark current
-f FLAT, --flat FLAT name of the file containing the flat field
-P POLARIZATION_FACTOR, --polarization POLARIZATION_FACTOR
Polarization factor, from -1 (vertical) to +1
(horizontal), default is None for no correction,
synchrotrons are around 0.95
--error-model ERROR_MODEL
Error model to use. Currently on 'poisson' is
implemented
--unit UNIT unit for the radial dimension: can be q_nm^-1, q_A^-1,
2th_deg, 2th_rad or r_mm
--ext EXT extension of the regrouped filename (.xy)
--method METHOD Integration method
--multi Average out all frame in a file before integrating
extracting variance, otherwise treat every single
frame
--average AVERAGE Method for averaging out: can be 'mean' (default),
'min', 'max' or 'median
--do-2D Perform 2D integration in addition to 1D
pyFAI-waxs is the script of pyFAI that allows data reduction (azimuthal
integration) for Wide Angle Scattering to produce X-Ray Powder Diffraction
Pattern with output axis in 2-theta space.
One options which may be of some interest is the –do-2D which allow to perform caking.
Azimuthal integration for small angle data
Small angle scattering usually prefer integration in a radial unit independant of the wavelength, typically \(q = \frac{4 \pi sin(2\theta/2)}{\lambda}\) in inverse nanometers (\(nm^{-1}\)).
[4]:
pyFAI-saxs -p LaB6_29.4keV.poni -n 1000 LaB6_29.4keV.tif
Detector Detector Spline= /users/kieffer/workspace-400/pyFAI/doc/source/usage/cookbook/F_K4320T_Cam43_30012013_distorsion.spline PixelSize= 5.168e-05, 5.126e-05 m
Wavelength= 4.217150e-11m
SampleDetDist= 1.182208e-01m PONI= 5.394843e-02, 5.551600e-02m rot1=0.006974 rot2= -0.003313 rot3= -0.000000 rad
DirectBeamDist= 118.224mm Center: x=1066.839, y=1036.336 pix Tilt=0.442 deg tiltPlanRotation= -154.594 deg
Mask: None Method: splitpixel
Integrating LaB6_29.4keV.tif --> LaB6_29.4keV.dat, reading: 0.022s 1D integration: 1.792s.
[5]:
head -n 50 LaB6_29.4keV.dat
# == pyFAI calibration ==
# Distance Sample to Detector: 0.118220810284 m
# PONI: 5.395e-02, 5.552e-02 m
# Rotations: 0.006974 -0.003313 -0.000000 rad
#
# == Fit2d calibration ==
# Distance Sample-beamCenter: 118.224 mm
# Center: x=1066.839, y=1036.336 pix
# Tilt: 0.442 deg TiltPlanRot: -154.594 deg
#
# Detector Detector Spline= /users/kieffer/workspace-400/pyFAI/doc/source/usage/cookbook/F_K4320T_Cam43_30012013_distorsion.spline PixelSize= 5.168e-05, 5.126e-05 m
# Detector has a mask: False
# Detector has a dark current: False
# detector has a flat field: False
#
# Wavelength: 4.21714957131e-11 m
# Mask applied: False
# Dark current applied: False
# Flat field applied: False
# Polarization factor: None
# Normalization factor: 1.0
#
# Headers of the input frame:
# {
# "info": {},
# "photometricInterpretation": 1,
# "rowsPerStrip": 2048,
# "nColumns": 2048,
# "compression": false,
# "sampleFormat": 3,
# "imageDescription": "{\nEDF_DataBlockID=0.Image.Psd\nEDF_BinarySize=16777216\nEDF_HeaderSize=1536\nByteOrder=LowByteFirst\nDataType=FloatValue\nDim_1=2048\nDim_2=2048\nImage=0\nHeaderID=EH:000000:000000:000000\nSize=16777216\nmethod=max\nnframes=21\ncutoff=None\nmerged_file_00=ref_lab6_0001.edf\nmerged_file_01=ref_lab6_0002.edf\nmerged_file_02=ref_lab6_0003.edf\nmerged_file_03=ref_lab6_0004.edf\nmerged_file_04=ref_lab6_0005.edf\nmerged_file_05=ref_lab6_0006.edf\nmerged_file_06=ref_lab6_0007.edf\nmerged_file_07=ref_lab6_0008.edf\nmerged_file_08=ref_lab6_0009.edf\nmerged_file_09=ref_lab6_0010.edf\nmerged_file_10=ref_lab6_0011.edf\nmerged_file_11=ref_lab6_0012.edf\nmerged_file_12=ref_lab6_0013.edf\nmerged_file_13=ref_lab6_0014.edf\nmerged_file_14=ref_lab6_0015.edf\nmerged_file_15=ref_lab6_0016.edf\nmerged_file_16=ref_lab6_0017.edf\nmerged_file_17=ref_lab6_0018.edf\nmerged_file_18=ref_lab6_0019.edf\nmerged_file_19=ref_lab6_0020.edf\nmerged_file_20=ref_lab6_0021.edf\nfilename=LaB6_29.4keV.tif\n",
# "nRows": 2048,
# "colormap": null,
# "nBits": 32,
# "date": "Fri Mar 2 10:02:31 2018",
# "software": "fabio.tifimage",
# "compression_type": 1,
# "stripOffsets": [
# 1143
# ],
# "stripByteCounts": [
# 16777216
# ]
# }
# --> LaB6_29.4keV.dat
# q_nm^-1 I
4.278568e-02 2.427221e+00
1.283570e-01 2.821874e+00
2.139284e-01 2.101271e+00
2.994998e-01 2.708885e+00
[6]:
pyFAI-saxs -h
usage: pyFAI-saxs [options] -n 1000 -p ponifile file1.edf file2.edf ...
Azimuthal integration for SAXS users.
positional arguments:
FILE Image files to integrate
optional arguments:
-h, --help show this help message and exit
-v, --version show program's version number and exit
-p PONIFILE PyFAI parameter file (.poni)
-n NPT, --npt NPT Number of points in radial dimension
-w WAVELENGTH, --wavelength WAVELENGTH
wavelength of the X-Ray beam in Angstrom
-e ENERGY, --energy ENERGY
energy of the X-Ray beam in keV
(hc=12.3984197396keV.A)
-u DUMMY, --dummy DUMMY
dummy value for dead pixels
-U DELTA_DUMMY, --delta_dummy DELTA_DUMMY
delta dummy value
-m MASK, --mask MASK name of the file containing the mask image
-d DARK, --dark DARK name of the file containing the dark current
-f FLAT, --flat FLAT name of the file containing the flat field
-P POLARIZATION_FACTOR, --polarization POLARIZATION_FACTOR
Polarization factor, from -1 (vertical) to +1
(horizontal), default is None for no correction,
synchrotrons are around 0.95
--error-model ERROR_MODEL
Error model to use. Currently on 'poisson' is
implemented
--unit UNIT unit for the radial dimension: can be q_nm^-1, q_A^-1,
2th_deg, 2th_rad or r_mm
--ext EXT extension of the regrouped filename (.dat)
--method METHOD Integration method
pyFAI-saxs is the SAXS script of pyFAI that allows data reduction (azimuthal
integration) for Small Angle Scattering with output axis in q space.
Common tool for azimuthal intration
pyFAI-integrate offers a graphical user interface for integration which exposes most options from pyFAI. Once all options have been configured in the the windows, one can either launch the processing or save the configuration into a JSON file to be able to create script with all the options.
To run pyFAI-integrate as a script, i.e. without the graphical interface, the –no-gui option is required:
[7]:
pyFAI-integrate --no-gui -j azimint.json -o LaB6_29.4keV.cli LaB6_29.4keV.tif
INFO:root:Enter, port=54321.
INFO:root:Enter.
start config ...
INFO:pyFAI.worker:Detector Detector Spline= /users/kieffer/workspace-400/pyFAI/doc/source/usage/cookbook/F_K4320T_Cam43_30012013_distorsion.spline PixelSize= 5.168e-05, 5.126e-05 m
Wavelength= 4.217150e-11m
SampleDetDist= 1.182208e-01m PONI= 5.394843e-02, 5.551600e-02m rot1=0.006974 rot2= -0.003313 rot3= -0.000000 rad
DirectBeamDist= 118.224mm Center: x=1066.839, y=1036.336 pix Tilt=0.442 deg tiltPlanRotation= -154.594 deg
INFO:pyFAI.azimuthalIntegrator:AI.integrate1d: Resetting integrator because init
INFO:pyFAI.opencl.processing:234.570MB are needed on device which has 12644.188MB
INFO:pyFAI.opencl.processing:Compiling file ['kahan.cl', 'preprocess.cl', 'memset.cl', 'ocl_azim_CSR.cl'] with options -D NBINS=1000 -D NIMAGE=4194304 -D WORKGROUP_SIZE=32
INFO:pyFAI:Processing done in 2.433s !
[8]:
pyFAI-integrate --help
INFO:root:Enter, port=54321.
INFO:root:Enter.
usage: pyFAI-integrate [options] file1.edf file2.edf ...
PyFAI-integrate is a graphical interface (based on Python/Qt4) to perform
azimuthal integration on a set of files. It exposes most of the important
options available within pyFAI and allows you to select a GPU (or an openCL
platform) to perform the calculation on.
positional arguments:
FILE Files to be integrated
optional arguments:
-h, --help show this help message and exit
-V, --version show program's version number and exit
-v, --verbose switch to verbose/debug mode
-o OUTPUT, --output OUTPUT
Directory or file where to store the output data
-f FORMAT, --format FORMAT
output data format (can be HDF5)
-s SLOW, --slow-motor SLOW
Dimension of the scan on the slow direction (makes
sense only with HDF5)
-r RAPID, --fast-motor RAPID
Dimension of the scan on the fast direction (makes
sense only with HDF5)
--no-gui Process the dataset without showing the user
interface.
-j JSON, --json JSON Configuration file containing the processing to be
done
--monitor-name MONITOR_KEY
Name of the monitor in the header of each input files.
If defined the contribution of each input file is
divided by the monitor. If the header does not contain
or contains a wrong value, the contribution of the
input file is ignored. On EDF files, values from
'counter_pos' can accessed by using the expected
mnemonic. For example 'counter/bmon'.
PyFAI-integrate saves all parameters in a .azimint.json (hidden) file. This
JSON file is an ascii file which can be edited and used to configure online
data analysis using the LImA plugin of pyFAI. Nota: there is bug in debian6
making the GUI crash (to be fixed inside pyqt) http://bugs.debian.org/cgi-
bin/bugreport.cgi?bug=697348
[9]:
head -n 50 LaB6_29.4keV.cli
# == pyFAI calibration ==
# Distance Sample to Detector: 0.118220810284 m
# PONI: 5.395e-02, 5.552e-02 m
# Rotations: 0.006974 -0.003313 -0.000000 rad
#
# == Fit2d calibration ==
# Distance Sample-beamCenter: 118.224 mm
# Center: x=1066.839, y=1036.336 pix
# Tilt: 0.442 deg TiltPlanRot: -154.594 deg
#
# Detector Detector Spline= /users/kieffer/workspace-400/pyFAI/doc/source/usage/cookbook/F_K4320T_Cam43_30012013_distorsion.spline PixelSize= 5.168e-05, 5.126e-05 m
# Detector has a mask: False
# Detector has a dark current: False
# detector has a flat field: False
#
# Wavelength: 4.21714957131e-11 m
# Mask applied: False
# Dark current applied: False
# Flat field applied: False
# Polarization factor: None
# Normalization factor: 1.0
# --> /users/kieffer/workspace-400/pyFAI/doc/source/usage/cookbook/LaB6_29.4keV.cli
# r_mm I
3.913197e-02 2.562033e+00
1.173959e-01 2.491645e+00
1.956599e-01 2.354185e+00
2.739238e-01 2.858854e+00
3.521877e-01 2.796831e+00
4.304517e-01 2.496695e+00
5.087156e-01 2.796695e+00
5.869795e-01 2.742193e+00
6.652435e-01 2.834705e+00
7.435074e-01 2.855904e+00
8.217714e-01 2.906582e+00
9.000353e-01 3.021892e+00
9.782992e-01 3.171136e+00
1.056563e+00 3.010916e+00
1.134827e+00 3.027250e+00
1.213091e+00 3.020955e+00
1.291355e+00 2.935973e+00
1.369619e+00 3.207142e+00
1.447883e+00 2.983296e+00
1.526147e+00 3.007866e+00
1.604411e+00 3.052688e+00
1.682675e+00 2.898143e+00
1.760939e+00 3.027955e+00
1.839203e+00 3.147613e+00
1.917466e+00 3.243755e+00
1.995730e+00 3.213144e+00
2.073994e+00 2.948822e+00
Conclusion
This cookbook exposed the way to perfrom azimuthal integration in scripts with pyFAI using three tools: pyFAI-waxs, pyFAI-saxs and pyFAI-integrate. Using shell scripts is probably not as efficient as writing Python scripts but they may be easier to interface with batch queueing system managing compute-clusters.