The built-in extension

The ability to serialize the following types is provided by asdf’s built-in extension:

  • dict

  • list

  • str

  • int

  • float

  • complex

  • numpy.ndarray

The built-in extension is packaged with asdf and is automatically used when reading and writing files. Users can not control the use of the built-in extension and in general they need not concern themselves with the details of its implementation. However, it is useful to be aware that the built-in extension is always in effect when reading and writing ASDF files.

Custom types

For the purposes of this documentation, a “custom type” is any data type that can not be serialized by the built-in extension.

In order for a particular custom type to be serialized, a special class called a “converter” must be implemented. Each converter defines how the corresponding custom type will be serialized and deserialized. More details on how converters are implemented can be found in Converters. Users should never have to refer to converter implementations directly; they simply enable asdf to recognize and process custom types.

In addition to converters, each custom type may have a corresponding schema, which is used for validation. The definition of the schema if present is closely tied to the definition of the converter. More details on schema validation can be found in Schema validation.

Schemas are generally versioned and change in sync with their associated converters. The version number will increase whenever a schema (and therefore the converter implementation) changes.

Extensions

In order for the converters and schemas to be used by asdf, they must be packaged into an extension class. In general, the details of extensions are irrelevant to users of asdf. However, users need to be aware of extensions in the following two scenarios:

  • when storing custom data types to files to be written

  • when reading files that contain custom data types

These scenarios require the use of custom extensions (the built-in extension is always used). There are two ways to use custom extensions, which are detailed below in Extensions from other packages and Explicit use of extensions.

Writing custom types to files

asdf is not capable of serializing any custom type unless an extension is provided that defines how to serialize that type. Attempting to do so will cause an error when trying to write the file. For details on developing support for custom types and extensions, see Extensions.

Reading files with custom types

The asdf software is capable of reading files that contain custom data types even if the extension that was used to create the file is not present. However, the extension is required in order to properly deserialize the original type.

If the necessary extension is not present, the custom data types will simply appear in the tree as a nested combination of basic data types. The structure of this data will mirror the structure of the YAML objects in the ASDF file.

In this case, a warning will occur by default to indicate to the user that the custom type in the file was not recognized and can not be deserialized. To suppress these warnings, users should pass ignore_unrecognized_tag=True to asdf.open.

Even if an extension for the custom type is present, it does not guarantee that the type can be deserialized successfully. Instantiating the custom type may involve additional software dependencies, which, if not present, will cause an error when the type is deserialized. Users should be aware of the dependencies that are required for instantiating custom types when reading ASDF files.

Custom types, extensions, and versioning

Tags and schemas that follow best practices are versioned. This allows changes to tags and schemas to be recorded, and it allows asdf to define behavior with respect to version compatibility.

Tag and schema versions may change for several reasons. One common reason is to reflect a change to the API of the custom type that a tag represents. This typically corresponds to an update to the version of the software that defines that custom type.

Since ASDF is designed to be an archival file format, extension authors are encouraged to maintain backwards compatibility with all older tag versions.

Reading files

When asdf encounters a tagged object in a file, it will compare the URI of the tag in the file with the list of tags handled by available converters. The first matching converter will be selected to deserialize the object. If no such converters exist, the library will emit a warning and the object will be presented to the user in its primitive form.

If multiple converters are present that both handle the same tag, the first found by the library will be used. Users may disable a converter by removing its extension with the remove_extension method.

Writing files

When writing a object to a file, asdf compares the object’s type to the list of types handled by available converters. The first matching converter will be selected to serialize the object. If no such converters exist, the library will raise an error.

If multiple converters are present that both handle the same type, the first found by the library will be used. Users may disable a converter by removing its extension with the remove_extension method.

Extensions from other packages

Some external packages may define extensions that allow asdf to recognize some or all of the types that are defined by that package. Such packages may install the extension class as part of the package itself (details for developers can be found in Installing extensions via entry points).

If the package installs its extension, then asdf will automatically detect the extension and use it when processing any files. No specific action is required by the user in order to successfully read and write custom types defined by the extension for that particular package.

Users can use the extensions command of the asdftool command line tool in order to determine which packages in the current Python environment have installed ASDF extensions:

$ asdftool extensions -s
Extension Name: 'bizbaz' (from bizbaz 1.2.3) Class: bizbaz.io.asdf.extension.BizbazExtension
Extension Name: 'builtin' (from asdf 2.0.0) Class: asdf.extension.BuiltinExtension

The output will always include the built-in extension, but may also display other extensions from other packages, depending on what is installed.

Explicit use of extensions

Sometimes no packaged extensions are provided for the types you wish to serialize. In this case, it is necessary to explicitly install any necessary extension classes when reading and writing files that contain custom types.

The config object returned from asdf.get_config offers an add_extension method that can be used to install an extension for the remainder of the current Python session.

Consider the following example where there exists a custom type MyCustomType that needs to be written to a file. An extension is defined MyCustomExtension that contains a converter that can serialize and deserialize MyCustomType. Since MyCustomExtension is not installed by any package, we will need to manually install it:

import asdf

...

asdf.get_config().add_extension(MyCustomExtension())
af = asdf.AsdfFile()
af.tree = {"thing": MyCustomType("foo")}
# This call would cause an error if the proper extension was not
# provided to the constructor
af.write_to("custom.asdf")

Note that the extension class must actually be instantiated when it is passed to add_extension.

To read the file (in a new session) we again need to install the extension first:

import asdf

asdf.get_config().add_extension(MyCustomExtension())
af = asdf.open("custom.asdf")

Extension checking

When writing ASDF files using this software, metadata about the extensions that were used to create the file will be added to the file itself. This includes the extension’s URI, which uniquely identifies a particular version of the extension.

When reading files with extension metadata, asdf can check whether the required extensions are present before processing the file. If a required extension is not present, asdf will issue a warning.

It is possible to turn these warnings into errors by using the strict_extension_check parameter of asdf.open. If this parameter is set to True, then opening the file will fail if any of the required extensions are missing.