astropy.io.fits FAQ¶
General Questions¶
What is PyFITS and how does it relate to astropy?¶
PyFITS is a library written in, and for use with the Python programming language for reading, writing, and manipulating FITS formatted files. It includes a high-level interface to FITS headers with the ability for high- and low-level manipulation of headers, and it supports reading image and table data as Numpy arrays. It also supports more obscure and nonstandard formats found in some FITS files.
The astropy.io.fits module is identical to PyFITS but with the names changed.
When the development of astropy began, it was clear that one of the core
requirements would be a FITS reader. Rather than starting from scratch,
PyFITS — being the most flexible FITS reader available for Python — was ported
into astropy. There are plans to gradually phase out PyFITS as a stand-alone
module and deprecate it in favor of astropy.io.fits. See more about this in
the next question.
Although PyFITS is written mostly in Python, it includes an optional module written in C that is required to read/write compressed image data. However, the rest of PyFITS functions without this extension module.
What is the development status of PyFITS?¶
PyFITS was written and maintained by the Science Software Branch at the Space Telescope Science Institute, and is licensed by AURA under a 3-clause BSD license.
It is now exclusively developed as a component of astropy
(astropy.io.fits) rather than as a stand-alone module. There are a few
reasons for this: The first is simply to reduce development effort; the
overhead of maintaining both PyFITS and astropy.io.fits in separate code
bases is nontrivial. The second is that there are many features of astropy
(units, tables, etc.) from which the astropy.io.fits module can benefit
greatly. Since PyFITS is already integrated into astropy, it makes more
sense to continue development there rather than make astropy a dependency
of PyFITS.
PyFITS’ past primary developer and active maintainer was Erik Bray. There is a GitHub project for PyFITS, but PyFITS is not actively developed anymore so patches and issue reports should be posted on the Astropy issue tracker.
The current (and last) stable release is 3.4.0.
Usage Questions¶
Something did not work as I expected. Did I do something wrong?¶
Possibly. But if you followed the documentation and things still did not work
as expected, it is entirely possible that there is a mistake in the
documentation, a bug in the code, or both. So feel free to report it as a bug.
There are also many, many corner cases in FITS files, with new ones discovered
almost every week. astropy.io.fits is always improving, but does not support
all cases perfectly. There are some features of the FITS format (scaled data,
for example) that are difficult to support correctly and can sometimes cause
unexpected behavior.
For the most common cases, however, such as reading and updating FITS headers,
images, and tables, astropy.io.fits is very stable and well-tested. Before
every astropy release it is ensured that all of its tests pass on a variety
of platforms, and those tests cover the majority of use cases (until new corner
cases are discovered).
astropy crashed and output a long string of code. What do I do?¶
This listing of code is what is known as a stack trace (or in Python parlance a “traceback”). When an unhandled exception occurs in the code causing the program to end, this is a way of displaying where the exception occurred and the path through the code that led to it.
As astropy is meant to be used as a piece in other software projects, some
exceptions raised by astropy are by design. For example, one of the most
common exceptions is a KeyError when an attempt is made to read
the value of a nonexistent keyword in a header:
>>> from astropy.io import fits
>>> h = fits.Header()
>>> h['NAXIS']
Traceback (most recent call last):
...
KeyError: "Keyword 'NAXIS' not found."
This indicates that something was looking for a keyword called “NAXIS” that
does not exist. If an error like this occurs in some other software that uses
astropy, it may indicate a bug in that software, in that it expected to
find a keyword that did not exist in a file.
Most “expected” exceptions will output a message at the end of the traceback
giving some idea of why the exception occurred and what to do about it. The
more vague and mysterious the error message in an exception appears, the more
likely that it was caused by a bug in astropy. So if you are getting an
exception and you really do not know why or what to do about it, feel free to
report it as a bug.
Why does opening a file work in CFITSIO, ds9, etc., but not in astropy?¶
As mentioned elsewhere in this FAQ, there are many unusual corner cases when
dealing with FITS files. It is possible that a file should work, but is not
supported due to a bug. Sometimes it is even possible for a file to work in an
older version of astropy, but not a newer version due to a regression
that has not been tested for yet.
Another problem with the FITS format is that, as old as it is, there are many
conventions that appear in files from certain sources that do not meet the FITS
standard. And yet they are so commonplace that it is necessary to support
them in any FITS readers. CONTINUE cards are one such example. There are
nonstandard conventions supported by astropy that are not supported by
CFITSIO and possibly vice versa. You may have hit one of those cases.
If astropy is having trouble opening a file, a good way to rule out whether
not the problem is with astropy is to run the file through the fitsverify
program. For smaller files you can even use the online FITS verifier.
These use CFITSIO under the hood, and should give a good indication of whether
or not there is something erroneous about the file. If the file is
malformatted, fitsverify will output errors and warnings.
If fitsverify confirms no problems with a file, and astropy is still having
trouble opening it (especially if it produces a traceback), then it is possible
there is a bug in astropy.
How do I turn off the warning messages astropy outputs to my console?¶
astropy uses Python’s built-in warnings subsystem for informing about
exceptional conditions in the code that are recoverable, but that the user may
want to be informed of. One of the most common warnings in astropy.io.fits
occurs when updating a header value in such a way that the comment must be
truncated to preserve space:
Card is too long, comment is truncated.
Any console output generated by astropy can be assumed to be from the
warnings subsystem. See Astropy’s documentation on the Python warnings system
for more information on how to control and quiet warnings.
What convention does astropy use for indexing, such as of image coordinates?¶
All arrays and sequences in astropy use a zero-based indexing scheme. For
example, the first keyword in a header is header[0], not header[1].
This is in accordance with Python itself, as well as C, on which Python is
based.
This may come as a surprise to veteran FITS users coming from IRAF, where 1-based indexing is typically used, due to its origins in Fortran.
Likewise, the top-left pixel in an N x N array is data[0,0]. The indices
for 2-dimensional arrays are row-major order, in that the first index is the
row number, and the second index is the column number. Or put in terms of
axes, the first axis is the y-axis, and the second axis is the x-axis. This is
the opposite of column-major order, which is used by Fortran and hence FITS.
For example, the second index refers to the axis specified by NAXIS1 in the
FITS header.
In general, for N-dimensional arrays, row-major orders means that the right-most axis is the one that varies the fastest while moving over the array data linearly. For example, the 3-dimensional array:
[[[1, 2],
[3, 4]],
[[5, 6],
[7, 8]]]
is represented linearly in row-major order as:
[1, 2, 3, 4, 5, 6, 7, 8]
Since 2 immediately follows 1, you can see that the right-most (or inner-most) axis is the one that varies the fastest.
The discrepancy in axis-ordering may take some getting used to, but it is a
necessary evil. Since most other Python and C software assumes row-major
ordering, trying to enforce column-major ordering in arrays returned by
astropy is likely to cause more difficulties than it is worth.
How do I open a very large image that will not fit in memory?¶
astropy.io.fits.open has an option to access the data portion of an
HDU by memory mapping using mmap. In astropy this is used by default.
What this means is that accessing the data as in the example above only reads
portions of the data into memory on demand. For example, if we request just a
slice of the image, such as hdul[0].data[100:200], then only rows 100-200
will be read into memory. This happens transparently, as though the entire
image were already in memory. This works the same way for tables. For most
cases this is your best bet for working with large files.
To ensure use of memory mapping, add the memmap=True argument to
fits.open. Likewise, using memmap=False will
force data to be read entirely into memory.
The default can also be controlled through a configuration option called
USE_MEMMAP. Setting this to 0 will disable mmap by default.
Unfortunately, memory mapping does not currently work as well with scaled image data, where BSCALE and BZERO factors need to be applied to the data to yield physical values. Currently this requires enough memory to hold the entire array, though this is an area that will see improvement in the future.
An alternative, which currently only works for image data (that is, non-tables) is the sections interface. It is largely replaced by the better support for mmap, but may still be useful on systems with more limited virtual memory space, such as on 32-bit systems. Support for scaled image data is flaky with sections too, though that will be fixed. See the documentation on image sections for more details on using this interface.
How can I create a very large FITS file from scratch?¶
See Create a very large FITS file from scratch.
For creating very large tables, this method may also be used, though it can be
difficult to determine ahead of time how many rows a table will need. In
general, use of the astropy.io.fits module is currently discouraged for the
creation and manipulation of large tables. The FITS format itself is not
designed for efficient on-disk or in-memory manipulation of table structures.
For large, heavy-duty table data it might be better too look into using HDF5
through the PyTables library. The Astropy Table
interface can provide an abstraction layer between different on-disk table
formats as well (for example, for converting a table between FITS and HDF5).
PyTables makes use of NumPy under the hood, and can be used to write binary table data to disk in the same format required by FITS. It is then possible to serialize your table to the FITS format for distribution. At some point this FAQ might provide an example of how to do this.
How do I create a multi-extension FITS file from scratch?¶
Why is an image containing integer data being converted unexpectedly to floats?¶
If the header for your image contains nontrivial values for the optional BSCALE and/or BZERO keywords (that is, BSCALE != 1 and/or BZERO != 0), then the raw data in the file must be rescaled to its physical values according to the formula:
physical_value = BZERO + BSCALE * array_value
As BZERO and BSCALE are floating point values, the resulting value must be a float as well. If the original values were 16-bit integers, the resulting values are single-precision (32-bit) floats. If the original values were 32-bit integers, the resulting values are double-precision (64-bit floats).
This automatic scaling can easily catch you off guard if you are not expecting it, because it does not happen until the data portion of the HDU is accessed (to allow for things like updating the header without rescaling the data). For example:
>>> fits_scaledimage_filename = fits.util.get_testdata_filepath('scale.fits')
>>> hdul = fits.open(fits_scaledimage_filename)
>>> image = hdul[0]
>>> image.header['BITPIX']
16
>>> image.header['BSCALE']
0.045777764213996
>>> data = image.data # Read the data into memory
>>> data.dtype.name # Got float32 despite BITPIX = 16 (16-bit int)
'float32'
>>> image.header['BITPIX'] # The BITPIX will automatically update too
-32
>>> 'BSCALE' in image.header # And the BSCALE keyword removed
False
The reason for this is that once a user accesses the data they may also manipulate it and perform calculations on it. If the data were forced to remain as integers, a great deal of precision is lost. So it is best to err on the side of not losing data, at the cost of causing some confusion at first.
If the data must be returned to integers before saving, use the
scale method:
>>> image.scale('int32')
>>> image.header['BITPIX']
32
>>> hdul.close()
Alternatively, if a file is opened with mode='update' along with the
scale_back=True argument, the original BSCALE and BZERO scaling will
be automatically reapplied to the data before saving. Usually this is
not desirable, especially when converting from floating point values back to
unsigned integer values. But this may be useful in cases where the raw
data needs to be modified corresponding to changes in the physical values.
To prevent rescaling from occurring at all (which is good for updating headers
— even if you do not intend for the code to access the data, it is good to err
on the side of caution here), use the do_not_scale_image_data argument when
opening the file:
>>> hdul = fits.open(fits_scaledimage_filename, do_not_scale_image_data=True)
>>> image = hdul[0]
>>> image.data.dtype.name
'int16'
>>> hdul.close()
Why am I losing precision when I assign floating point values in the header?¶
The FITS standard allows two formats for storing floating point numbers in a header value. The “fixed” format requires the ASCII representation of the number to be in bytes 11 through 30 of the header card, and to be right-justified. This leaves a standard number of characters for any comment string.
The fixed format is not wide enough to represent the full range of values that can be stored in a 64-bit float with full precision. So FITS also supports a “free” format in which the ASCII representation can be stored anywhere, using the full 70 bytes of the card (after the keyword).
Currently astropy only supports writing fixed format (it can read both
formats), so all floating point values assigned to a header are stored in the
fixed format. There are plans to add support for more flexible formatting.
In the meantime, it is possible to add or update cards by manually formatting the card image from a string, as it should appear in the FITS file:
>>> c = fits.Card.fromstring('FOO = 1234567890.123456789')
>>> h = fits.Header()
>>> h.append(c)
>>> h
FOO = 1234567890.123456789
As long as you do not assign new values to ‘FOO’ via h['FOO'] = 123, will
maintain the header value exactly as you formatted it (as long as it is valid
according to the FITS standard).
Why is reading rows out of a FITS table so slow?¶
Underlying every table data array returned by astropy.io.fits is a numpy
recarray which is a numpy array type specifically for representing
structured array data (i.e., a table). As with normal image arrays, astropy
accesses the underlying binary data from the FITS file via mmap (see the
question “What performance differences are there between astropy.io.fits and
fitsio?” for a deeper explanation of this). The underlying mmap is then
exposed as a recarray and in general this is a very efficient way to
read the data.
However, for many (if not most) FITS tables it is not all that simple. For
many columns there are conversions that have to take place between the actual
data that is “on disk” (in the FITS file) and the data values that are returned
to the user. For example, FITS binary tables represent boolean values
differently from how numpy expects them to be represented, “Logical” columns
need to be converted on the fly to a format numpy (and hence the user) can
understand. This issue also applies to data that is linearly scaled via the
TSCALn and TZEROn header keywords.
Supporting all of these “FITS-isms” introduces a lot of overhead that might
not be necessary for all tables, but are still common nonetheless. That is
not to say it cannot be faster even while supporting the peculiarities of
FITS — CFITSIO, for example, supports all of the same features but is orders of
magnitude faster. astropy could do much better here too, and there are many
known issues causing slowdown. There are plenty of opportunities for speedups,
and patches are welcome. In the meantime, for high-performance applications
with FITS tables some users might find the fitsio library more to their
liking.
I am opening many FITS files in a loop and getting OSError: Too many open files¶
Say you have some code like:
from astropy.io import fits
for filename in filenames:
with fits.open(filename) as hdul:
for hdu in hdul:
hdu_data = hdul.data
# Do some stuff with the data
The details may differ, but the qualitative point is that the data to many HDUs and/or FITS files are being accessed in a loop. This may result in an exception like:
Traceback (most recent call last):
File "<stdin>", line 2, in <module>
OSError: [Errno 24] Too many open files: 'my_data.fits'
As explained in the note on working with large files,
because astropy uses mmap by default to read the data in a FITS file, even
if you correctly close a file with HDUList.close a handle is kept open to that file so
that the memory-mapped data array can still continue to be read transparently.
The way numpy supports mmap is such that the file mapping is not closed
until the overlying ndarray object has no references to it and is freed
memory. However, when looping over a large number of files (or even just HDUs)
rapidly, this may not happen immediately. Or in some cases if the HDU object
persists, the data array attached to it may persist too. The recommended
workaround is to manually delete the .data attribute on the HDU object so
that the ndarray reference is freed and the mmap can be closed:
from astropy.io import fits
for filename in filenames:
with fits.open(filename) as hdul:
for hdu in hdul:
hdu_data = hdul.data
# Do some stuff with the data
# ...
# Don't need the data anymore; delete all references to it
# so that it can be garbage collected
del hdu_data
del hdu.data
In some extreme cases files are opened and closed fast enough that Python’s
garbage collector does not free them (and hence free the file handles) often
enough. To mitigate this, your code can manually force a garbage collection
by calling gc.collect() at the end of the loop.
In a future release it will be more convenient to automatically perform this sort of cleanup when closing FITS files, where needed.
Using header[‘NAXIS2’] += 1 does not add another row to my Table¶
NAXIS and similar keywords are FITS structural keywords and should not be
modified by the user. They are automatically updated by astropy.io.fits
when checking the validity of the data and headers. See Structural Keywords
for more information.
To add rows to a table, you can modify the actual data.
Comparison with Other FITS Readers¶
What is the difference between astropy.io.fits and fitsio?¶
The astropy.io.fits module (originally PyFITS) is a “pure Python” FITS
reader in that all of the code for parsing the FITS file format is in Python,
though numpy is used to provide access to the FITS data via the
ndarray interface. astropy.io.fits currently also accesses the
CFITSIO to support the
FITS Tile Compression convention, but this feature is optional. It does not
use CFITSIO outside of reading compressed images.
fitsio, on the other hand, is a Python
wrapper for the CFITSIO library. All of the heavy lifting of reading the FITS
format is handled by CFITSIO, while fitsio provides a better way to use
object-oriented API, including providing a numpy interface to FITS files
read from CFITSIO. Much of it is written in C (to provide the interface between
Python and CFITSIO), and the rest is in Python. The Python end mostly
provides the documentation and user-level API.
Because fitsio wraps CFITSIO it inherits most of its strengths and
weaknesses, though it has an added strength of providing a more convenient
API than if one were to use CFITSIO directly.
Why did Astropy adopt PyFITS as its FITS reader instead of fitsio?¶
When the Astropy Project was first started it was clear from the start that
one of its core components should be a submodule for reading and writing FITS
files, as many other components would be likely to depend on this
functionality. At the time, the fitsio package was in its infancy (it
goes back to roughly 2011) while PyFITS had already been established (going
back to before the year 2000). It was already a mature package with support
for the vast majority of FITS files found in the wild, including outdated
formats such as “Random Groups” FITS files still used extensively in the
radio astronomy community.
Although many aspects of PyFITS’ interface have evolved over the years, much of it has also remained the same, and is already familiar to astronomers working with FITS files in Python. Most of if not all existing training materials were also based around PyFITS. PyFITS was developed at STScI, which also put forward significant resources to develop Astropy, with an eye toward integrating Astropy into STScI’s own software stacks. As most of the Python software at STScI uses PyFITS, it was the only practical choice for making that transition.
Finally, although CFITSIO (and by extension fitsio) can read any FITS files
that conform to the FITS standard, it does not support all of the nonstandard
conventions that have been added to FITS files in the wild. While it does have
some support for some of these conventions (such as CONTINUE cards and, to a
limited extent, HIERARCH cards), it is not easy to add support for other
conventions to a large and complex C codebase.
PyFITS’ object-oriented design makes supporting nonstandard conventions somewhat easier in most cases, and as such PyFITS can be more flexible in the types of FITS files it can read and return useful data from. This includes better support for files that fail to meet the FITS standard, but still contain useful data that should be readable enough to correct any violations of the FITS standard. For example, a common error in non-English speaking regions is to insert non-ASCII characters into FITS headers. This is not a valid FITS file, but should still be readable in some sense. Supporting structural errors such as this is more difficult in CFITSIO which assumes a more rigid structure.
What performance differences are there between astropy.io.fits and fitsio?¶
There are two main performance areas to look at: reading/parsing FITS headers and reading FITS data (image-like arrays as well as tables).
In the area of headers, fitsio is significantly faster in most cases. This
is due in large part to the (almost) pure C implementation (due to the use of
CFITSIO), but also due to fact that it is more rigid and does not support as
many local conventions and other special cases as astropy.io.fits tries to
support in its pure Python implementation.
That said, the difference is small and only likely to be a bottleneck either when opening files containing thousands of HDUs, or reading the headers out of thousands of FITS files in succession (in either case the difference is not even an order of magnitude).
Where data is concerned the situation is a little more complicated, and
requires some understanding of how astropy.io.fits is implemented versus
CFITSIO and fitsio. First, it is important to understand how they differ in
terms of memory management.
astropy.io.fits uses mmap, by default, to provide access to the raw
binary data in FITS files. Mmap is a system call (or in most cases these days
a wrapper in your libc for a lower-level system call) which allows user-space
applications to essentially do the same thing your OS is doing when it uses a
pagefile (swap space) for virtual memory: it allows data in a file on disk to
be paged into physical memory one page (or in practice usually several pages)
at a time on an as-needed basis. These cached pages of the file are also
accessible from all processes on the system, so multiple processes can read
from the same file with little additional overhead. In the case of reading
over all of the data in the file, the performance difference between using mmap
versus reading the entire data into physical memory at once can vary widely
between systems, hardware, and depending on what else is happening on the
system at the moment, but mmap is almost always going to be better.
In principle, it requires more overhead since accessing each page will result in a page fault and the system requires more requests to the disk. But in practice, the OS will optimize this pretty aggressively, especially for the most common case of sequential access — also in reality, reading the entire thing into memory is still going to result in a whole lot of page faults too. For random access, having all of the data in physical memory is always going to be best, though with mmap it is usually going to be pretty good too. (Most users do not normally access all of the data in a file in a totally random order — usually a few sections of it will be accessed most frequently, so the OS will keep those pages in physical memory as best it can.) For the most general case of reading FITS files (or most large data on disk) this is therefore the best choice, especially for casual users, and is hence enabled by default.
CFITSIO/fitsio, on the other hand, does not assume the existence of
technologies like mmap and page caching. Thus it implements its own LRU cache
of I/O buffers that store sections of FITS files read from disk in memory in
FITS’ famous 2880 byte chunk size. The I/O buffers are used heavily in
particular for keeping the headers in memory. Though for large data reads (for
example, reading an entire image from a file), it does bypass the cache and
instead does a read directly from disk into a user-provided memory buffer.
However, even when CFITSIO reads direct from the file, this is still largely less efficient than using mmap. Normally when your OS reads a file from disk, it caches as much of that read as it can in physical memory (in its page cache) so that subsequent access to those same pages does not require a subsequent expensive disk read. This happens when using mmap too, since the data has to be copied from disk into RAM at some point. The difference is that when using mmap to access the data, the program is able to read that data directly out of the OS’s page cache (as long as it is only being read). On the other hand, when reading data from a file into a local buffer such as with fread(), the data is first read into the page cache (if not already present) and then copied from the page cache into the local buffer. So every read performs at least one additional memory copy per page read (requiring twice as much physical memory, and possibly lots of paging if the file is large and pages need to dropped from the cache).
The user API for CFITSIO usually works by having the user allocate a memory buffer large enough to hold the image/table they want to read (or at least the section they are interested in). There are some helper functions for determining the appropriate amount of space to allocate. Then you pass in a pointer to your buffer and CFITSIO handles all of the reading (usually using the process described above), and copies the results into your user buffer. For large reads, it reads directly from the file into your buffer, though if the data needs to be scaled it makes a stop in CFITSIO’s own buffer first, then writes the rescaled values out to the user buffer (if rescaling has been requested). Regardless, this means that if your program wishes to hold an entire image in memory at once it will use as much RAM as the size of the data. For most applications it is better (and sufficient) to work on smaller sections of the data, but this requires extra complexity. Using mmap on the other hand makes managing this complexity more efficient.
An informal test demonstrates this difference. This test was performed on four
simple FITS images (one of which is a cube) of dimensions 256x256, 1024x1024,
4096x4096, and 256x1024x1024. Each image was generated before the test and
filled with randomized 64-bit floating point values. A similar test was
performed using both astropy.io.fits and fitsio. A handle to the FITS
file is opened using each library’s basic semantics, and then the entire data
array of the files is copied into a temporary array in memory (for example, if
we were blitting the image to a video buffer). For astropy the test is
written:
def read_test_astropy(filename):
with fits.open(filename, memmap=True) as hdul:
data = hdul[0].data
c = data.copy()
The test was timed in IPython on a Linux system with kernel version 2.6.32, a 6-core Intel Xeon X5650 CPU clocked at 2.67 GHz per core, and 11.6 GB of RAM using:
for filename in filenames:
print(filename)
%timeit read_test_astropy(filename)
where filenames is just a list of the aforementioned generated sample
files. The results were:
256x256.fits
1000 loops, best of 3: 1.28 ms per loop
1024x1024.fits
100 loops, best of 3: 4.24 ms per loop
4096x4096.fits
10 loops, best of 3: 60.6 ms per loop
256x1024x1024.fits
1 loops, best of 3: 1.15 s per loop
For fitsio the test was:
def read_test_fitsio(filename):
with fitsio.FITS(filename) as f:
data = f[0].read()
c = data.copy()
This was also run in a loop over all of the sample files, producing the results:
256x256.fits
1000 loops, best of 3: 476 µs per loop
1024x1024.fits
100 loops, best of 3: 12.2 ms per loop
4096x4096.fits
10 loops, best of 3: 136 ms per loop
256x1024x1024.fits
1 loops, best of 3: 3.65 s per loop
It should be made clear that the sample files were rewritten with new random
data between the astropy test and the fitsio test, so they were not reading
the same data from the OS’s page cache. Fitsio was much faster on the small
(256x256) image because in that case the time is dominated by parsing the
headers. As already explained, this is much faster in CFITSIO. However, as
the data size goes up and the header parsing no longer dominates the time,
astropy.io.fits using mmap is roughly twice as fast. This discrepancy is
almost entirely due to it requiring roughly half as many in-memory copies
to read the data, as explained earlier. That said, more extensive benchmarking
could be very interesting.
This is also not to say that astropy.io.fits does better in all cases. There
are some cases where it is currently blown away by fitsio. See the subsequent
question.
Why is fitsio so much faster than astropy at reading tables?¶
In many cases it is not: there is either no difference, or it may be a little
faster in astropy depending on what you are trying to do with the table and
what types of columns or how many columns the table has. There are some
cases, however, where fitsio can be radically faster, mostly for reasons
explained above in “Why is reading rows out of a FITS table so slow?”
In principle a table is no different from, say, an array of pixels. But instead of pixels each element of the array is some kind of record structure (for example, two floats, a boolean, and a 20-character string field). Just as a 64-bit float is an 8 byte record in an array, a row in such a table can be thought of as a 37 byte (in the case of the previous example) record in a 1D array of rows. So in principle everything that was explained in the answer to the question “What performance differences are there between astropy.io.fits and fitsio?” applies just as well to tables as it does to any other array.
However, FITS tables have many additional complexities that sometimes preclude streaming the data directly from disk, and instead require transformation from the on-disk FITS format to a format more immediately useful to the user. A common example is how FITS represents boolean values in binary tables. Another significantly more complicated example, is variable length arrays.
As explained in “Why is reading rows out of a FITS table so slow?”,
astropy.io.fits does not currently handle some of these cases as
efficiently as it could, in particular in cases where a user only wishes to
read a few rows out of a table. Fitsio, on the other hand, has a better
interface for copying one row at a time out of a table and performing the
necessary transformations on that row only, rather than on the entire column
or columns that the row is taken from. As such, for many cases fitsio gets
much better performance and should be preferred for many performance-critical
table operations.
Fitsio also exposes a microlanguage (implemented in CFITSIO) for making
efficient SQL-like queries of tables (single tables only though — no joins or
anything like that). This format, described in the CFITSIO documentation can
in some cases perform more efficient selections of rows than might be possible
with numpy alone, which requires creating an intermediate mask array in
order to perform row selection.