ElasticSearch: Geographically Encoded Objects for ElasticSearch

(Driver available in GDAL 1.10 or later)
Driver is read-write starting with GDAL 2.1 (was write only in GDAL 2.0 or earlier)
As of GDAL 2.1, ElasticSearch 1.X and, partially, 2.X versions are supported (5.0 known not to work). GDAL 2.2 adds supports for ElasticSearch 2.X and 5.X

ElasticSearch is an Enterprise-level search engine for a variety of data sources. It supports full-text indexing and geospatial querying of those data in a fast and efficient manor using a predefined REST API.

Opening dataset name syntax

Starting with GDAL 2.1, the driver supports reading existing indices from a ElasticSearch host. There are two main possible syntaxes to open a dataset: The open options available are :

ElasticSearch vs OGR concepts

Each mapping type inside a ElasticSearch index will be considered as a OGR layer. A ElasticSearch document is considered as a OGR feature.

Field definitions

Fields are dynamically mapped from the input OGR data source. However, the driver will take advantage of advanced options within ElasticSearch as defined in a field mapping file.

The mapping file allows you to modify the mapping according to the ElasticSearch field-specific types. There are many options to choose from, however, most of the functionality is based on all the different things you are able to do with text fields within ElasticSearch. 

ogr2ogr -progress --config ES_WRITEMAP /path/to/file/map.txt -f "ElasticSearch" http://localhost:9200 my_shapefile.shp

The ElasticSearch writer supports the following Configuration Options. Starting with GDAL 2.1, layer creation options are also available and should be preferred:

Geometry types

In GDAL 2.0 and earlier, the driver was limited in the geometry it handles: even if polygons were provided as input, they were stored as geo point and the "center" of the polygon is used as value of the point. Starting with GDAL 2.1, geo_shape is used to store all geometry types (except curve geometries that are not handled by ElasticSearch and will be approximated to their linear equivalents).

Filtering

The driver will forward any spatial filter set with SetSpatialFilter() to the server.

Starting with GDAL 2.2, SQL attribute filters set with SetAttributeFilter() are converted to ElasticSearch filter syntax. They will be combined with the potentially defined spatial filter.

It is also possible to directly use a ElasticSearch filter by setting the string passed to SetAttributeFilter() as a JSon serialized object, e.g. 

{ "post_filter": { "term": { "properties.EAS_ID": 169 } } }

Note: if defining directly an Elastic Search JSon filter, the spatial filter specified through SetSpatialFilter() will be ignored, and must thus be included in the JSon filter if needed.

Paging

Features are retrieved from the server by chunks of 100. This can be altered with the BATCH_SIZE open option.

Schema

When reading a Elastic Search index/type, OGR must establish the schema of attribute and geometry fields, since OGR has a fixed schema concept.

In the general case, OGR will read the mapping definition and the first 100 documents (can be altered with the FEATURE_COUNT_TO_ESTABLISH_FEATURE_DEFN open option) of the index/type and build the schema that best fit to the found fields and values.

It is also possible to set the JSON_FIELD=YES open option so that a _json special field is added to the OGR schema. When reading Elastic Search documents as OGR features, the full JSon version of the document will be stored in the _json field. This might be useful in case of complex documents or with data types that do not translate well in OGR data types. On creation/update of documents, if the _json field is present and set, its content will be used directly (other fields will be ignored).

Feature ID

Elastic Search have a special _id field that contains the unique ID of the document. This field is returned as an OGR field, but cannot be used as the OGR special FeatureID field, which must be of integer type. By default, OGR will try to read a potential 'ogc_fid' field to set the OGR FeatureID. The name of this field to look up can be set with the FID open option. If the field is not found, the FID returned by OGR will be a sequential number starting at 1, but it is not guaranteed to be stable at all.

ExecuteSQL() interface

Starting with GDAL 2.2, SQL requests, involving a single layer, with WHERE and ORDER BY statements will be translated as ElasticSearch queries.

Otherwise, if specifying "ES" as the dialect of ExecuteSQL(), a JSon string with a serialized Elastic Search filter can be passed. The search will be done on all indices and types, unless the filter itself restricts the search. The returned layer will be a union of the types returned by the FEATURE_COUNT_TO_ESTABLISH_FEATURE_DEFN first documents. It will also contain the _index and _type special fields to indicate the provenance of the features.

The following filter can be used to restrict the search to the "poly" index and its "FeatureCollection" type mapping (ElasticSearch 1.X and 2.X) 

{ "filter": {
    "indices" : {
        "no_match_filter": "none",
        "index": "poly",
        "filter": {
           "and" : [
             { "type": { "value": "FeatureCollection" } },
             { "term" : { "properties.EAS_ID" : 158.0 } }
           ]
        }
      }
    }
}
For ElasticSearch 5.X (works also with 2.X) : 
{ "post_filter": {
    "indices" : {
        "no_match_query": "none",
        "index": "poly",
        "query": {
          "bool": {
            "must" : [
              { "type": { "value": "FeatureCollection" } },
              { "term" : { "properties.EAS_ID" : 158.0 } }
            ]
          }
        }
      }
    }
}
Aggregations are not supported.

Getting metadata

Getting feature count is efficient.

Getting extent is efficient, only on geometry columns mapped to ElasticSearch type geo_point. On geo_shape fields, feature retrieval of the whole layer is done, which might be slow.

Write support

Index/type creation and deletion is possible.

Write support is only enabled when the datasource is opened in update mode.

When inserting a new feature with CreateFeature() in non-bulk mode, and if the command is successful, OGR will fetch the returned _id and use it for the SetFeature() operation.

Spatial reference system

Geometries stored in Elastic Search are supposed to be referenced as longitude/latitude over WGS84 datum (EPSG:4326). On creation, the driver will automatically reproject from the layer (or geometry field) SRS to EPSG:4326, provided that the input SRS is set and that is not already EPSG:4326.

Layer creation options

Starting with GDAL 2.1, the driver supports the following layer creation options:

Examples

Open the local store:
ogrinfo ES:
Open a remote store:
ogrinfo ES:http://example.com:9200
Filtering on a Elastic Search field:
ogrinfo -ro ES: my_type -where '{ "post_filter": { "term": { "properties.EAS_ID": 168 } } }'
Using "match" query on Windows:
On Windows the query must be between double quotes and double quotes inside the query must be escaped. 
C:\GDAL_on_Windows>ogrinfo ES: my_type -where "{\"query\": { \"match\": { \"properties.NAME\": \"Helsinki\" } } }"
Load an ElasticSearch index with a shapefile: 
ogr2ogr -f "ElasticSearch" http://localhost:9200 my_shapefile.shp
Create a Mapping File:
The mapping file allows you to modify the mapping according to the ElasticSearch field-specific types. There are many options to choose from, however, most of the functionality is based on all the different things you are able to do with text fields. 
ogr2ogr -progress --config ES_WRITEMAP /path/to/file/map.txt -f "ElasticSearch" http://localhost:9200 my_shapefile.shp
or (GDAL >= 2.1): 
ogr2ogr -progress -lco WRITE_MAPPING=/path/to/file/map.txt -f "ElasticSearch" http://localhost:9200 my_shapefile.shp
Read the Mapping File:
Reads the mapping file during the transformation
ogr2ogr -progress --config ES_META /path/to/file/map.txt -f "ElasticSearch" http://localhost:9200 my_shapefile.shp
or (GDAL >= 2.1): 
ogr2ogr -progress -lco MAPPING=/path/to/file/map.txt -f "ElasticSearch" http://localhost:9200 my_shapefile.shp
Bulk Uploading (for larger datasets):
Bulk loading helps when uploading a lot of data. The integer value is the number of bytes that are collected before being inserted. Bulk size considerations 
ogr2ogr -progress --config ES_BULK 5000000 -f "ElasticSearch" http://localhost:9200 PG:"host=localhost user=postgres dbname=my_db password=password" "my_table" -nln thetable
or (GDAL >= 2.1): 
ogr2ogr -progress -lco BULK_SIZE=5000000 -f "ElasticSearch" http://localhost:9200 my_shapefile.shp
Overwrite the current Index:
If specified, this will overwrite the current index. Otherwise, the data will be appended. 
ogr2ogr -progress --config ES_OVERWRITE 1 -f "ElasticSearch" http://localhost:9200 PG:"host=localhost user=postgres dbname=my_db password=password" "my_table" -nln thetable
or (GDAL >= 2.1): 
ogr2ogr -progress -overwrite ES:http://localhost:9200 PG:"host=localhost user=postgres dbname=my_db password=password" "my_table" -nln thetable

See Also