Utility Functions for Handling Bit Masks and Mask Arrays

It is common to use bit fields, such as integer variables whose individual bits represent some attributes, to characterize the state of data. For example, Hubble Space Telescope (HST) uses arrays of bit fields to characterize data quality (DQ) of HST images. See, for example, DQ field values for WFPC2 image data (see Table 3.3) and WFC3 image data (see Table 3.3). As you can see, the meaning assigned to various bit flags for the two instruments is generally different.

Bit fields can be thought of as tightly packed collections of bit flags. Using masking we can “inspect” the status of individual bits.

One common operation performed on bit field arrays is their conversion to boolean masks, for example, by assigning boolean True (in the boolean mask) to those elements that correspond to non-zero-valued bit fields (bit fields with at least one bit set to 1) or, oftentimes, by assigning True to elements whose corresponding bit fields have only specific fields set (to 1). This more sophisticated analysis of bit fields can be accomplished using bit masks and the aforementioned masking operation.

The bitmask module provides two functions that facilitate conversion of bit field arrays (i.e., DQ arrays) to boolean masks: bitfield_to_boolean_mask converts an input bit field array to a boolean mask using an input bit mask (or list of individual bit flags) and interpret_bit_flags creates a bit mask from an input list of individual bit flags.

Creating Boolean Masks

Overview

bitfield_to_boolean_mask by default assumes that all input bit fields that have at least one bit turned “ON” corresponds to “bad” data (i.e., pixels) and converts them to boolean True in the output boolean mask (otherwise output boolean mask values are set to False).

Often, for specific algorithms and situations, some bit flags are okay and can be ignored. bitfield_to_boolean_mask accepts lists of bit flags that by default must be ignored in the input bit fields when creating boolean masks.

Fundamentally, by default, bitfield_to_boolean_mask performs the following operation:

(1)    boolean_mask = (bitfield & ~bit_mask) != 0

(Here & is bitwise and while ~ is the bitwise not operation.) In the previous formula, bit_mask is a bit mask created from individual bit flags that need to be ignored in the bit field.

Example

Table 1: Examples of Boolean Mask Computations (default parameters and 8-bit data type)

Bit Field

Bit Mask

~(Bit Mask)

Bit Field & ~(Bit Mask)

Boolean Mask

11011001 (217)

01010000 (80)

10101111 (175)

10001001 (137)

True

11011001 (217)

10101111 (175)

01010000 (80)

01010000 (80)

True

00001001 (9)

01001001 (73)

10110110 (182)

00000000 (0)

False

00001001 (9)

00000000 (0)

11111111 (255)

00001001 (9)

True

00001001 (9)

11111111 (255)

00000000 (0)

00000000 (0)

False

Specifying Bit Flags

bitfield_to_boolean_mask accepts either an integer bit mask or lists of bit flags. Lists of bit flags will be combined into a bit mask and can be provided either as a Python list of integer bit flag values or as a comma-separated (or +-separated) list of integer bit flag values. Consider the bit mask from the first example in Table 1. In this case ignore_flags can be set either to:

  • An integer value bit mask 80

  • A Python list indicating individual non-zero bit flag values: [16, 64]

  • A string of comma-separated bit flag values or mnemonic names: '16,64', 'CR,WARM'

  • A string of +-separated bit flag values or mnemonic names: '16+64', 'CR+WARM'

Example

To specify bit flags:

>>> from astropy.nddata import bitmask
>>> import numpy as np
>>> bitmask.bitfield_to_boolean_mask(217, ignore_flags=80)
array(True...)
>>> bitmask.bitfield_to_boolean_mask(217, ignore_flags='16,64')
array(True...)
>>> bitmask.bitfield_to_boolean_mask(217, ignore_flags=[16, 64])
array(True...)
>>> bitmask.bitfield_to_boolean_mask(9, ignore_flags=[1, 8, 64])
array(False...)
>>> bitmask.bitfield_to_boolean_mask([9, 10, 73, 217], ignore_flags='1,8,64')
array([False,  True, False,  True]...)

It is also possible to specify the type of the output mask:

>>> bitmask.bitfield_to_boolean_mask([9, 10, 73, 217], ignore_flags='1,8,64', dtype=np.uint8)
array([0, 1, 0, 1], dtype=uint8)

In order to use lists of mnemonic bit flags names, one must provide a map, a subclass of BitFlagNameMap, that can be used to map mnemonic names to bit flag values. Normally these maps should be provided by a third-party package supporting a specific instrument. Each bit flag in the map may also contain a string comment following the flag value. In the example below we define a simple mask map:

>>> from astropy.nddata.bitmask import BitFlagNameMap
>>> class ST_DQ(BitFlagNameMap):
...     CR = 1
...     CLOUDY = 4
...     RAINY = 8, 'Dome closed'
...     HOT = 32
...     DEAD = 64
>>> bitmask.bitfield_to_boolean_mask([9, 10, 73, 217], ignore_flags='CR,RAINY,DEAD',
...                                  dtype=np.uint8, flag_name_map=ST_DQ)
array([0, 1, 0, 1], dtype=uint8)

Using Bit Flags Name Maps

In order to allow the use of mnemonic bit flag names to describe the flags to be taken into consideration or ignored when creating a boolean mask, we use bit flag name maps. These maps perform case-insensitive translation of mnemonic bit flag names to the corresponding integer value.

Bit flag name maps are subclasses of BitFlagNameMap and can be constructed in two ways, either by directly subclassing BitFlagNameMap, e.g.,

>>> from astropy.nddata.bitmask import BitFlagNameMap
>>> class ST_DQ(BitFlagNameMap):
...     CR = 1
...     CLOUDY = 4
...     RAINY = 8
...
>>> class ST_CAM1_DQ(ST_DQ):
...     HOT = 16
...     DEAD = 32

or by using the extend_bit_flag_map class factory:

>>> from astropy.nddata.bitmask import extend_bit_flag_map
>>> ST_DQ = extend_bit_flag_map('ST_DQ', CR=1, CLOUDY=4, RAINY=8)
>>> ST_CAM1_DQ = extend_bit_flag_map('ST_CAM1_DQ', ST_DQ, HOT=16, DEAD=32)

Note

Bit flag values must be integer numbers that are powers of 2.

Once constructed, bit flag values of a map cannot be modified, deleted, or added. Adding flags to a map is allowed only through subclassing using one of the two methods shown above or by adding lists of tuples of the form ('NAME', value) to the class. This will create a new map class subclassed from the original map but containing the additional flags

>>> ST_CAM1_DQ = ST_DQ + [('HOT', 16), ('DEAD', 32)]

would result in an equivalent map as in the subclassing or class factory examples shown above.

Once a bit flag name map was created, the bit flag values can be accessed either as case-insensitive class attributes or keys in a dictionary:

>>> ST_CAM1_DQ.cloudy
4
>>> ST_CAM1_DQ['Rainy']
8

Modifying the Formula for Creating Boolean Masks

bitfield_to_boolean_mask provides several parameters that can be used to modify the formula used to create boolean masks.

Inverting Bit Masks

Sometimes it is more convenient to be able to specify those bit flags that must be considered when creating the boolean mask, and all other flags should be ignored.

Example

In bitfield_to_boolean_mask specifying bit flags that must be considered when creating the boolean mask can be accomplished by setting the parameter flip_bits to True. This effectively modifies equation (1) to:

(2)    boolean_mask = (bitfield & bit_mask) != 0

So, instead of:

>>> bitmask.bitfield_to_boolean_mask([9, 10, 73, 217], ignore_flags=[1, 8, 64])
array([False,  True, False,  True]...)

You can obtain the same result as:

>>> bitmask.bitfield_to_boolean_mask(
...     [9, 10, 73, 217], ignore_flags=[2, 4, 16, 32, 128], flip_bits=True
... )
array([False,  True, False,  True]...)

Note however, when ignore_flags is a comma-separated list of bit flag values, flip_bits cannot be set to either True or False. Instead, to flip bits of the bit mask formed from a string list of comma-separated bit flag values, you can prepend a single ~ to the list:

>>> bitmask.bitfield_to_boolean_mask([9, 10, 73, 217], ignore_flags='~2+4+16+32+128')
array([False,  True, False,  True]...)

Inverting Boolean Masks

Other times, it may be more convenient to obtain an inverted mask in which flagged data are converted to False instead of True:

(3)    boolean_mask = (bitfield & ~bit_mask) == 0

This can be accomplished by changing the good_mask_value parameter from its default value (False) to True.

Example

To obtain an inverted mask in which flagged data are converted to False instead of True:

>>> bitmask.bitfield_to_boolean_mask([9, 10, 73, 217], ignore_flags=[1, 8, 64],
...                                  good_mask_value=True)
array([ True, False,  True, False]...)