What’s new in h5py 2.0

HDF5 for Python (h5py) 2.0 represents the first major refactoring of the h5py codebase since the project’s launch in 2008. Many of the most important changes are behind the scenes, and include changes to the way h5py interacts with the HDF5 library and Python. These changes have substantially improved h5py’s stability, and make it possible to use more modern versions of HDF5 without compatibility concerns. It is now also possible to use h5py with Python 3.

Enhancements unlikely to affect compatibility

  • HDF5 1.8.3 through 1.8.7 now work correctly and are officially supported.

  • Python 3.2 is officially supported by h5py! Thanks especially to Darren Dale for getting this working.

  • Fill values can now be specified when creating a dataset. The fill time is H5D_FILL_TIME_IFSET for contiguous datasets, and H5D_FILL_TIME_ALLOC for chunked datasets.

  • On Python 3, dictionary-style methods like Group.keys() and Group.values() return view-like objects instead of lists.

  • Object and region references now work correctly in compound types.

  • Zero-length dimensions for extendable axes are now allowed.

  • H5py no longer attempts to auto-import ipython on startup.

  • File format bounds can now be given when opening a high-level File object (keyword “libver”).

Changes which may break existing code

Supported HDF5/Python versions

  • HDF5 1.6.X is no longer supported on any platform; following the release of 1.6.10 some time ago, this branch is no longer maintained by The HDF Group.

  • Python 2.6 or later is now required to run h5py. This is a consequence of the numerous changes made to h5py for Python 3 compatibility.

  • On Python 2.6, unittest2 is now required to run the test suite.

Group, Dataset and Datatype constructors have changed

In h5py 2.0, it is no longer possible to create new groups, datasets or named datatypes by passing names and settings to the constructors directly. Instead, you should use the standard Group methods create_group and create_dataset.

The File constructor remains unchanged and is still the correct mechanism for opening and creating files.

Code which manually creates Group, Dataset or Datatype objects will have to be modified to use create_group or create_dataset. File-resident datatypes can be created by assigning a NumPy dtype to a name (e.g. mygroup[“name”] = numpy.dtype(‘S10’)).

Unicode is now used for object names

Older versions of h5py used byte strings to represent names in the file. Starting with version 2.0, you may use either byte or unicode strings to create objects, but object names (obj.name, etc) will generally be returned as Unicode.

Code which may be affected:

  • Anything which uses “isinstance” or explicit type checks on names, expecting “str” objects. Such checks should be removed, or changed to compare to “basestring” instead.

  • In Python 2.X, other parts of your application may complain if they are handed Unicode data which can’t be encoded down to ascii. This is a general problem in Python 2.

File objects must be manually closed

With h5py 1.3, when File objects (or low-level FileID) objects went out of scope, the corresponding HDF5 file was closed. This led to surprising behavior, especially when files were opened with the H5F_CLOSE_STRONG flag; “losing” the original File object meant that all open groups and datasets suddenly became invalid.

Beginning with h5py 2.0, files must be manually closed, by calling the “close” method or by using the file object as a context manager. If you forget to close a file, the HDF5 library will try to close it for you when the application exits.

Please note that opening the same file multiple times (i.e. without closing it first) continues to result in undefined behavior.

Changes to scalar slicing code

When a scalar dataset was accessed with the syntax dataset[()], h5py incorrectly returned an ndarray. H5py now correctly returns an array scalar. Using dataset[...] on a scalar dataset still returns an ndarray.

Array scalars now always returned when indexing a dataset

When using datasets of compound type, retrieving a single element incorrectly returned a tuple of values, rather than an instance of numpy.void_ with the proper fields populated. Among other things, this meant you couldn’t do things like dataset[index][field]. H5py now always returns an array scalar, except in the case of object dtypes (references, vlen strings).

Reading object-like data strips special type information

In the past, reading multiple data points from dataset with vlen or reference type returned a Numpy array with a “special dtype” (such as those created by h5py.special_dtype()). In h5py 2.0, all such arrays now have a generic Numpy object dtype (numpy.dtype('O')). To get a copy of the dataset’s dtype, always use the dataset’s dtype property directly (mydataset.dtype).

The selections module has been removed

Only numpy-style slicing arguments remain supported in the high level interface. Existing code which uses the selections module should be refactored to use numpy slicing (and numpy.s_ as appropriate), or the standard C-style HDF5 dataspace machinery.

The H5Error exception class has been removed (along with h5py.h5e)

All h5py exceptions are now native Python exceptions, no longer inheriting from H5Error. RuntimeError is raised if h5py can’t figure out what exception is appropriate… every instance of this behavior is considered a bug. If you see h5py raising RuntimeError please report it so we can add the correct mapping!

The old errors module (h5py.h5e) has also been removed. There is no public error-management API.

File .mode property is now either ‘r’ or ‘r+

Files can be opened using the same mode arguments as before, but now the property File.mode will always return ‘r’ (read-only) or ‘r+’ (read-write).

Long-deprecated dict methods have been removed

Certain ancient aliases for Group/AttributeManager methods (e.g. listnames) have been removed. Please use the standard Python dict interface (Python 2 or Python 3 as appropriate) to interact with these objects.

Known issues

  • Thread support has been improved in h5py 2.0. However, we still recommend that for your own sanity you use locking to serialize access to files.

  • There are reports of crashes related to storing object and region references. If this happens to you, please post on the mailing list or contact the h5py author directly.