|
GDAL
|
The VRT driver is a format driver for GDAL that allows a virtual GDAL dataset to be composed from other GDAL datasets with repositioning, and algorithms potentially applied as well as various kinds of metadata altered or added. VRT descriptions of datasets can be saved in an XML format normally given the extension .vrt.
The VRT format can also describe warping operations and pansharpening operations.
An example of a simple .vrt file referring to a 512x512 dataset with one band loaded from utm.tif might look like this:
Many aspects of the VRT file are a direct XML encoding of the GDAL Data Model which should be reviewed for understanding of the semantics of various elements.
VRT files can be produced by translating to VRT format. The resulting file can then be edited to modify mappings, add metadata or other purposes. VRT files can also be produced programmatically by various means.
This tutorial will cover the .vrt file format (suitable for users editing .vrt files), and how .vrt files may be created and manipulated programmatically for developers.
A XML schema of the GDAL VRT format is available.
Virtual files stored on disk are kept in an XML format with the following elements.
VRTDataset: This is the root element for the whole GDAL dataset. It must have the attributes rasterXSize and rasterYSize describing the width and height of the dataset in pixels. It may have a subClass attributes with values VRTWarpedDataset (Warped VRT) or VRTPansharpenedDataset (Pansharpened VRT). It may have SRS, GeoTransform, GCPList, Metadata, MaskBand and VRTRasterBand subelements.
The allowed subelements for VRTDataset are :
SRS: This element contains the spatial reference system (coordinate system) in OGC WKT format. Note that this must be appropriately escaped for XML, so items like quotes will have the ampersand escape sequences substituted. As as well WKT, and valid input to the SetFromUserInput() method (such as well known GEOGCS names, and PROJ.4 format) is also allowed in the SRS element.
GeoTransform: This element contains a six value affine geotransformation for the dataset, mapping between pixel/line coordinates and georeferenced coordinates.
GCPList: This element contains a list of Ground Control Points for the dataset, mapping between pixel/line coordinates and georeferenced coordinates. The Projection attribute should contain the SRS of the georeferenced coordinates in the same format as the SRS element.
Metadata: This element contains a list of metadata name/value pairs associated with the VRTDataset as a whole, or a VRTRasterBand. It has <MDI> (metadata item) subelements which have a "key" attribute and the value as the data of the element. The Metadata element can be repeated multiple times, in which case it must be accompanied with a "domain" attribute to indicate the name of the metadata domain.
MaskBand: (GDAL >= 1.8.0) This element represents a mask band that is shared between all bands on the dataset (see GMF_PER_DATASET in RFC 15). It must contain a single VRTRasterBand child element, that is the description of the mask band itself.
VRTRasterBand: This represents one band of a dataset. It will have a dataType attribute with the type of the pixel data associated with this band (use names Byte, UInt16, Int16, UInt32, Int32, Float32, Float64, CInt16, CInt32, CFloat32 or CFloat64) and the band this element represents (1 based). This element may have Metadata, ColorInterp, NoDataValue, HideNoDataValue, ColorTable, GDALRasterAttributeTable, Description and MaskBand subelements as well as the various kinds of source elements such as SimpleSource, ComplexSource, etc. A raster band may have many "sources" indicating where the actual raster data should be fetched from, and how it should be mapped into the raster bands pixel space.
The allowed subelements for VRTRasterBand are :
ColorInterp: The data of this element should be the name of a color interpretation type. One of Gray, Palette, Red, Green, Blue, Alpha, Hue, Saturation, Lightness, Cyan, Magenta, Yellow, Black, or Unknown.
NoDataValue: If this element exists a raster band has a nodata value associated with, of the value given as data in the element.
HideNoDataValue: If this value is 1, the nodata value will not be reported. Essentially, the caller will not be aware of a nodata pixel when it reads one. Any datasets copied/translated from this will not have a nodata value. This is useful when you want to specify a fixed background value for the dataset. The background will be the value specified by the NoDataValue element.
Default value is 0 when this element is absent.
ColorTable: This element is parent to a set of Entry elements defining the entries in a color table. Currently only RGBA color tables are supported with c1 being red, c2 being green, c3 being blue and c4 being alpha. The entries are ordered and will be assumed to start from color table entry 0.
GDALRasterAttributeTable: (GDAL >=2.3) This element is parent to a set of FieldDefn elements defining the columns of a raster attribute table, followed by a set of Row element defining the values of the columns of each row.
Description: This element contains the optional description of a raster band as its text value.
UnitType: This optional element contains the vertical units for elevation band data. One of "m" for meters or "ft" for feet. Default assumption is meters.
Offset: This optional element contains the offset that should be applied when computing "real" pixel values from scaled pixel values on a raster band. The default is 0.0.
Scale: This optional element contains the scale that should be applied when computing "real" pixel values from scaled pixel values on a raster band. The default is 1.0.
Overview: This optional element describes one overview level for the band. It should have a child SourceFilename and SourceBand element. The SourceFilename may have a relativeToVRT boolean attribute. Multiple elements may be used to describe multiple overviews.
CategoryNames: This optional element contains a list of Category subelements with the names of the categories for classified raster band.
SimpleSource: The SimpleSource indicates that raster data should be read from a separate dataset, indicating the dataset, and band to be read from, and how the data should map into this bands raster space. The SimpleSource may have the SourceFilename, SourceBand, SrcRect, and DstRect subelements. The SrcRect element will indicate what rectangle on the indicated source file should be read, and the DstRect element indicates how that rectangle of source data should be mapped into the VRTRasterBands space.
The relativeToVRT attribute on the SourceFilename indicates whether the filename should be interpreted as relative to the .vrt file (value is 1) or not relative to the .vrt file (value is 0). The default is 0.
The shared attribute, added in GDAL 2.0.0, on the SourceFilename indicates whether the dataset should be shared (value is 1) or not (value is 0). The default is 1. If several VRT datasets referring to the same underlying sources are used in a multithreaded context, shared should be set to 0. Alternatively, the VRT_SHARED_SOURCE configuration option can be set to 0 to force non-shared mode.
Some characteristics of the source band can be specified in the optional SourceProperties tag to enable the VRT driver to differ the opening of the source dataset until it really needs to read data from it. This is particularly useful when building VRTs with a big number of source datasets. The needed parameters are the raster dimensions, the size of the blocks and the data type. If the SourceProperties tag is not present, the source dataset will be opened at the same time as the VRT itself.
Starting with GDAL 1.8.0, the content of the SourceBand subelement can refer to a mask band. For example mask,1 means the mask band of the first band of the source.
Starting with GDAL 2.0, a OpenOptions subelement can be added to specify the open options to apply when opening the source dataset. It has <OOI> (open option item) subelements which have a "key" attribute and the value as the data of the element.
Starting with GDAL 2.0, a resampling attribute can be specified on a SimpleSource or ComplexSource element to specified the resampling algorithm used when the size of the destination rectangle is not the same as the size of the source rectangle. The values allowed for that attribute are : nearest,bilinear,cubic, cubicspline,lanczos,average,mode.
AveragedSource: The AveragedSource is derived from the SimpleSource and shares the same properties except that it uses an averaging resampling instead of a nearest neighbour algorithm as in SimpleSource, when the size of the destination rectangle is not the same as the size of the source rectangle. Note: starting with GDAL 2.0, a more general mechanism to specify resampling algorithms can be used. See above paragraph about the 'resampling' attribute.
ComplexSource: The ComplexSource is derived from the SimpleSource (so it shares the SourceFilename, SourceBand, SrcRect and DestRect elements), but it provides support to rescale and offset the range of the source values. Certain regions of the source can be masked by specifying the NODATA value.
Starting with GDAL 1.11, alternatively to linear scaling, non-linear scaling using a power function can be used by specifying the Exponent, SrcMin, SrcMax, DstMin and DstMax elements. If SrcMin and SrcMax are not specified, they are computed from the source minimum and maximum value (which might require analyzing the whole source dataset). Exponent must be positive. (Those 5 values can be set with the -exponent and -scale options of gdal_translate.)
The ComplexSource supports adding a custom lookup table to transform the source values to the destination. The LUT can be specified using the following form:
The intermediary values are calculated using a linear interpolation between the bounding destination values of the corresponding range.
The ComplexSource supports fetching a color component from a source raster band that has a color table. The ColorTableComponent value is the index of the color component to extract : 1 for the red band, 2 for the green band, 3 for the blue band or 4 for the alpha band.
When transforming the source values the operations are executed in the following order:
Non-linear scaling:
KernelFilteredSource: This is a pixel source derived from the Simple Source (so it shares the SourceFilename, SourceBand, SrcRect and DestRect elements, but it also passes the data through a simple filtering kernel specified with the Kernel element. The Kernel element should have two child elements, Size and Coefs and optionally the boolean attribute normalized (defaults to false=0). The size must always be an odd number, and the Coefs must have Size * Size entries separated by spaces. For now kernel is not applied to sub-sampled or over-sampled data.
Starting with GDAL 2.3, a separable kernel may also be used. In this case the number of Coefs entries should correspond to the Size. The Coefs specify a one-dimensional kernel which is applied along each axis in succession, resulting in far quicker execution. Many common image-processing filters are separable. For example, a Gaussian blur:
MaskBand: (GDAL >= 1.8.0) This element represents a mask band that is specific to the VRTRasterBand it contains. It must contain a single VRTRasterBand child element, that is the description of the mask band itself.
GDAL can make efficient use of overviews available in the sources that compose the bands when dealing with RasterIO() requests that involve downsampling. But in the general case, the VRT bands themselves will not expose overviews.
Except if (from top priority to lesser priority) :
So far we have described how to derive new virtual datasets from existing files supports by GDAL. However, it is also common to need to utilize raw binary raster files for which the regular layout of the data is known but for which no format specific driver exists. This can be accomplished by writing a .vrt file describing the raw file.
For example, the following .vrt describes a raw raster file containing floating point complex pixels in a file called l2p3hhsso.img. The image data starts from the first byte (ImageOffset=0). The byte offset between pixels is 8 (PixelOffset=8), the size of a CFloat32. The byte offset from the start of one line to the start of the next is 9376 bytes (LineOffset=9376) which is the width (1172) times the size of a pixel (8).
Some things to note are that the VRTRasterBand has a subClass specifier of "VRTRawRasterBand". Also, the VRTRawRasterBand contains a number of previously unseen elements but no "source" information. VRTRawRasterBands may never have sources (i.e. SimpleSource), but should contain the following elements in addition to all the normal "metadata" elements previously described which are still supported.
SourceFilename: The name of the raw file containing the data for this band. The relativeToVRT attribute can be used to indicate if the SourceFilename is relative to the .vrt file (1) or not (0).
ImageOffset: The offset in bytes to the beginning of the first pixel of data of this image band. Defaults to zero.
PixelOffset: The offset in bytes from the beginning of one pixel and the next on the same line. In packed single band data this will be the size of the dataType in bytes.
LineOffset: The offset in bytes from the beginning of one scanline of data and the next scanline of data. In packed single band data this will be PixelOffset * rasterXSize.
ByteOrder: Defines the byte order of the data on disk. Either LSB (Least Significant Byte first) such as the natural byte order on Intel x86 systems or MSB (Most Significant Byte first) such as the natural byte order on Motorola or Sparc systems. Defaults to being the local machine order.
A few other notes:
The image data on disk is assumed to be of the same data type as the band dataType of the VRTRawRasterBand.
All the non-source attributes of the VRTRasterBand are supported, including color tables, metadata, nodata values, and color interpretation.
The VRTRawRasterBand supports in place update of the raster, whereas the source based VRTRasterBand is always read-only.
The OpenEV tool includes a File menu option to input parameters describing a raw raster file in a GUI and create the corresponding .vrt file.
Multiple bands in the one .vrt file can come from the same raw file. Just ensure that the ImageOffset, PixelOffset, and LineOffset definition for each band is appropriate for the pixels of that particular band.
Another example, in this case a 400x300 RGB pixel interleaved image.
The VRT driver supports several methods of creating VRT datasets. As of GDAL 1.2.0 the vrtdataset.h include file should be installed with the core GDAL include files, allowing direct access to the VRT classes. However, even without that most capabilities remain available through standard GDAL interfaces.
To create a VRT dataset that is a clone of an existing dataset use the CreateCopy() method. For example to clone utm.tif into a wrk.vrt file in C++ the following could be used:
Note the use of GDALOpenShared() when opening the source dataset. It is advised to use GDALOpenShared() in this situation so that you are able to release the explicit reference to it before closing the VRT dataset itself. In other words, in the previous example, you could also invert the 2 last lines, whereas if you open the source dataset with GDALOpen(), you'd need to close the VRT dataset before closing the source dataset.
To create a virtual copy of a dataset with some attributes added or changed such as metadata or coordinate system that are often hard to change on other formats, you might do the following. In this case, the virtual dataset is created "in memory" only by virtual of creating it with an empty filename, and then used as a modified source to pass to a CreateCopy() written out in TIFF format.
In the above example the nodata value is set as -999. You can set the HideNoDataValue element in the VRT dataset's band using SetMetadataItem() on that band.
In this example a virtual dataset is created with the Create() method, and adding bands and sources programmatically, but still via the "generic" API. A special attribute of VRT datasets is that sources can be added to the VRTRasterBand (but not to VRTRawRasterBand) by passing the XML describing the source into SetMetadata() on the special domain target "new_vrt_sources". The domain target "vrt_sources" may also be used, in which case any existing sources will be discarded before adding the new ones. In this example we construct a simple averaging filter source instead of using the simple source.
A more general form of this that will produce a 3x3 average filtered clone of any input datasource might look like the following. In this case we deliberately set the filtered datasource as in the "vrt_sources" domain to override the SimpleSource created by the CreateCopy() method. The fact that we used CreateCopy() ensures that all the other metadata, georeferencing and so forth is preserved from the source dataset ... the only thing we are changing is the data source for each band.
The VRTDataset class is one of the few dataset implementations that supports the AddBand() method. The options passed to the AddBand() method can be used to control the type of the band created (VRTRasterBand, VRTRawRasterBand, VRTDerivedRasterBand), and in the case of the VRTRawRasterBand to set its various parameters. For standard VRTRasterBand, sources should be specified with the above SetMetadata() / SetMetadataItem() examples.
A specialized type of band is a 'derived' band which derives its pixel information from its source bands. With this type of band you must also specify a pixel function, which has the responsibility of generating the output raster. Pixel functions are created by an application and then registered with GDAL using a unique key.
Using derived bands you can create VRT datasets that manipulate bands on the fly without having to create new band files on disk. For example, you might want to generate a band using four source bands from a nine band input dataset (x0, x3, x4, and x8):
You could write the pixel function to compute this value and then register it with GDAL with the name "MyFirstFunction". Then, the following VRT XML could be used to display this derived band:
In addition to the subclass specification (VRTDerivedRasterBand) and the PixelFunctionType value, there is another new parameter that can come in handy: SourceTransferType. Typically the source rasters are obtained using the data type of the derived band. There might be times, however, when you want the pixel function to have access to higher resolution source data than the data type being generated. For example, you might have a derived band of type "Float", which takes a single source of type "CFloat32" or "CFloat64", and returns the imaginary portion. To accomplish this, set the SourceTransferType to "CFloat64". Otherwise the source would be converted to "Float" prior to calling the pixel function, and the imaginary portion would be lost.
Starting with GDAL 2.2, GDAL provides a set of default pixel functions that can be used without writing new code:
To register this function with GDAL (prior to accessing any VRT datasets with derived bands that use this function), an application calls GDALAddDerivedBandPixelFunc with a key and a GDALDerivedPixelFunc:
A good time to do this is at the beginning of an application when the GDAL drivers are registered.
GDALDerivedPixelFunc is defined with a signature similar to IRasterIO:
| papoSources | A pointer to packed rasters; one per source. The datatype of all will be the same, specified in the eSrcType parameter. |
| nSources | The number of source rasters. |
| pData | The buffer into which the data should be read, or from which it should be written. This buffer must contain at least nBufXSize * nBufYSize words of type eBufType. It is organized in left to right, top to bottom pixel order. Spacing is controlled by the nPixelSpace, and nLineSpace parameters. |
| nBufXSize | The width of the buffer image into which the desired region is to be read, or from which it is to be written. |
| nBufYSize | The height of the buffer image into which the desired region is to be read, or from which it is to be written. |
| eSrcType | The type of the pixel values in the papoSources raster array. |
| eBufType | The type of the pixel values that the pixel function must generate in the pData data buffer. |
| nPixelSpace | The byte offset from the start of one pixel value in pData to the start of the next pixel value within a scanline. If defaulted (0) the size of the datatype eBufType is used. |
| nLineSpace | The byte offset from the start of one scanline in pData to the start of the next. |
The following is an implementation of the pixel function:
Starting with GDAL 2.2, in addition to pixel functions written in C/C++ as documented in the Using Derived Bands (with pixel functions in C/C++) section, it is possible to use pixel functions written in Python. Both CPython and NumPy are requirements at run-time.
The subelements for VRTRasterBand (whose subclass specification must be set to VRTDerivedRasterBand) are :
PixelFunctionType (required): Must be set to a function name that will be defined as a inline Python module in PixelFunctionCode element or as the form "module_name.function_name" to refer to a function in an external Python module
PixelFunctionLanguage (required): Must be set to Python.
PixelFunctionArguments (optional): It is possible to pass arguments to the Python pixel function by defining attributes in the PixelFunctionArguments element.
PixelFunctionCode (required if PixelFunctionType is of the form "function_name", ignored otherwise). The in-lined code of a Python module, that must be at least have a function whose name is given by PixelFunctionType.
BufferRadius (optional, defaults to 0): Amount of extra pixels, with respect to the original RasterIO() request to satisfy, that are fetched at the left, right, bottom and top of the input and output buffers passed to the pixel function. Note that the values of the output buffer in this buffer zone will be ignored.
The signature of the Python pixel function must have the following arguments:
VRT that multiplies the values of the source file by a factor of 1.5
VRT that adds 2 (or more) rasters
VRT that computes hillshading using an external library
with hillshading.py:
The ability to run Python code potentially opens the door to many potential vulnerabilities if the user of GDAL may process untrusted datasets. To avoid such issues, by default, execution of Python pixel function will be disabled. The execution policy can be controlled with the GDAL_VRT_ENABLE_PYTHON configuration option, which can accept 3 values:
Currently only CPython - 2.6, 2.7 and 3.x - is supported. The GDAL shared object is not explicitly linked at build time to any of the CPython library. When GDAL will need to run Python code, it will first determine if the Python interpreter is loaded in the current process (which is the case if the program is a Python interpreter itself, or if another program, e.g. QGIS, has already loaded the CPython library). Otherwise it will look if the PYTHONSO configuration option is defined. This option can be set to point to the name of the Python library to use, either as a shortname like "libpython2.7.so" if it is accessible through the Linux dynamic loader (so typically in one of the paths in /etc/ld.so.conf or LD_LIBRARY_PATH) or as a full path name like "/usr/lib/x86_64-linux-gnu/libpython2.7.so". The same holds on Windows will shortnames like "python27.dll" if accessible through the PATH or full path names like "c:\python27\python27.dll". If the PYTHONSO configuration option is not defined, it will look for a "python" binary in the directories of the PATH and will try to determine the related shared object (it will retry with "python3" if no "python" has been found). If the above was not successful, then a predefined list of shared objects names will be tried. At the time of writing, the order of versions searched is 2.7, 2.6, 3.4, 3.5, 3.6, 3.3, 3.2. Enabling debug information (CPL_DEBUG=VRT) will show which Python version is used.
The use of a just-in-time compiler may significantly speed up execution times. Numba has been successfully tested. For better performance, it is recommended to use a offline pixel function so that the just-in-time compiler may cache its compilation.
Given the following mandelbrot.py file :
the following VRT file can be used (to be opened with QGIS for example)
A warped VRT is a VRTDataset with subClass="VRTWarpedDataset". It has a GDALWarpOptions element which describe the warping options.
(Since GDAL 2.1)
A VRT can describe a dataset resulting from a pansharpening operation. The pansharpening VRT combines a panchromatic band with several spectral bands of lower resolution to generate output spectral bands of the same resolution as the panchromatic band.
VRT pansharpening assumes that the panchromatic and spectral bands have the same projection (or no projection). If that is not the case, reprojection must be done in a prior step. Bands might have different geotransform matrices, in which case, by default, the resulting dataset will have as extent the union of all extents.
Currently the only supported pansharpening algorithm is a "weighted" Brovey algorithm. The general principle of this algorithm is that, after resampling the spectral bands to the resolution of the panchromatic band, a pseudo panchromatic intensity is computed from a weighted average of the spectral bands. Then the output value of the spectral band is its input value multiplied by the ratio of the real panchromatic intensity over the pseudo panchromatic intensity.
Corresponding pseudo code:
A valid pansharpened VRT must declare subClass="VRTPansharpenedDataset" as an attribute of the VRTDataset top element. The VRTDataset element must have a child PansharpeningOptions element. This PansharpeningOptions element must have a PanchroBand child element and one of several SpectralBand elements. PanchroBand and SpectralBand elements must have at least a SourceFilename child element to specify the name of the dataset. They may also have a SourceBand child element to specify the number of the band in the dataset (starting with 1). If not specify, the first band will be assumed.
The SpectralBand element must generally have a dstBand attribute to specify the number of the output band (starting with 1) to which the input spectral band must be mapped. If the attribute is not specified, the spectral band will be taken into account in the computation of the pansharpening, but not exposed as an output band.
Panchromatic and spectral bands should generally come from different datasets, since bands of a GDAL dataset are assumed to have all the same dimensions. Spectral bands themselves can come from one or several datasets. The only constraint is that they have all the same dimensions.
An example of a minimalist working VRT is the following. It will generates a dataset with 3 output bands corresponding to the 3 input spectral bands of multispectral.tif, pansharpened with panchromatic.tif.
In the above example, 3 output pansharpend bands will be created from the 3 declared input spectral bands. The weights will be 1/3. Cubic resampling will be used. The projection and geotransform from the panchromatic band will be reused for the VRT dataset.
It is possible to create more explicit and declarative pansharpened VRT, allowing for example to only output part of the input spectral bands (e.g. only RGB when the input multispectral dataset is RGBNir). It is also possible to add "classic" VRTRasterBands, in addition to the pansharpened bands.
In addition to the above mentioned required PanchroBand and SpectralBand elements, the PansharpeningOptions element may have the following children elements :
The below examples creates a VRT dataset with 4 bands. The first band is the panchromatic band. The 3 following bands are than red, green, blue pansharpened bands computed from a multispectral raster with red, green, blue and near-infrared bands. The near-infrared bands is taken into account for the computation of the pseudo panchromatic intensity, but not bound to an output band.
The below section applies to GDAL <= 2.2. Starting with GDAL 2.3, the use of VRT datasets is subject to the standard GDAL dataset multi-threaded rules (that is a VRT dataset handle may only be used by a same thread at a time, but you may open several dataset handles on the same VRT file and use them in different threads)
When using VRT datasets in a multi-threading environment, you should be careful to open the VRT dataset by the thread that will use it afterwards. The reason for that is that the VRT dataset uses GDALOpenShared when opening the underlying datasets. So, if you open twice the same VRT dataset by the same thread, both VRT datasets will share the same handles to the underlying datasets.
The shared attribute, added in GDAL 2.0.0, on the SourceFilename indicates whether the dataset should be shared (value is 1) or not (value is 0). The default is 1. If several VRT datasets referring to the same underlying sources are used in a multithreaded context, shared should be set to 0. Alternatively, the VRT_SHARED_SOURCE configuration option can be set to 0 to force non-shared mode.
A VRT can reference many (hundreds, thousands, or more) datasets. Due to operating system limitations, and for performance at opening time, it is not reasonable/possible to open them all at the same time. GDAL has a "pool" of datasets opened by VRT files whose maximum limit is 100 by default. When it needs to access a dataset referenced by a VRT, it checks if it is already in the pool of open datasets. If not, when the pool has reached its limit, it closes the least recently used dataset to be able to open the new one. This maximum limit of the pool can be increased by setting the GDAL_MAX_DATASET_POOL_SIZE configuration option to a bigger value. Note that a typical user process on Linux is limited to 1024 simultaneously opened files, and you should let some margin for shared libraries, etc... As of GDAL 2.0, gdal_translate and gdalwarp, by default, increase the pool size to 450.
1.8.13.