[gdal-dev] Loading GeoJSON to ES 7

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

[gdal-dev] Loading GeoJSON to ES 7

xbartolone
Hi All,

I'm trying to investigate this issue https://github.com/geopython/pygeoapi/issues/379 in the pygeoapi project. I'm using an instance of ES7 and if I understand well the command below:

docker run --rm --network=host -u $(id -u ${USER}):$(id -g ${USER}) \
-v $(pwd):/data \
-e OGR_GEOJSON_MAX_OBJ_SIZE=500MB \
osgeo/gdal:alpine-small-latest \
ogr2ogr -lco INDEX_NAME=pygeoapi ES:localhost:9200 /data/tests/data/ne_110m_populated_places_simple.geojson

should create a Feature object in the index or am I wrong? Actually I don't have a geojson object:

{
    "took"3,
    "timed_out"false,
    "_shards": {
        "total"1,
        "successful"1,
        "skipped"0,
        "failed"0
    },
    "hits": {
        "total": {
            "value"243,
            "relation""eq"
        },
        "max_score"1.0,
        "hits": [
            {
                "_index""pygeoapi",
                "_type""_doc",
                "_id""EfwD6nAB8v4-ZVfFzLGS",
                "_score"1.0,
                "_source": {
                    "geonameid"6691831.0,
                    "geometry": {
                        "type""POINT",
                        "coordinates": [
                            12.453386545,
                            41.90328218
                        ]
                    },
                    "scalerank"8,
                    "natscale"10,
                    "labelrank"3,
                    "featurecla""Admin-0 capital",
                    "name""Vatican City",
                    "namepar"null,
                    "namealt"null,
                    "diffascii"0,
                    "nameascii""Vatican City",
                    "adm0cap"1.0,
                    "capalt"null,
                    "capin"null,
                    "worldcity"1.0,
                    "megacity"0,
                    "sov0name""Vatican (Holy Sea)",
                    "sov_a3""VAT",
                    "adm0name""Vatican (Holy See)",
                    "adm0_a3""VAT",
                    "adm1name""Lazio",
                    "iso_a2""VA",
                    "note"null,
                    "latitude"41.900012226400001,
                    "longitude"12.4478083889,
                    "changed"4.0,
                    "namediff"0,
                    "diffnote""Changed scale rank.",
                    "pop_max"832,
                    "pop_min"832,
                    "pop_other"562430,
                    "rank_max"2,
                    "rank_min"2,
                    "meganame"null,
                    "ls_name""Vatican City",
                    "ls_match"1,
                    "checkme"0,
                    "min_zoom"7.0
                }
            },
...
}

Thanks,
Francesco

_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Loading GeoJSON to ES 7

Even Rouault-2
Francesco,

The behaviour in ES7 has been changed. Now only flat structure is supported.
This was discussed in https://github.com/OSGeo/gdal/issues/1246 and I've just
made the doc of the driver clearer:

-  **MAPPING_NAME=**\ =name. (Elasticsearch < 7) Name of the mapping type
within the index.
   By default, the mapping name is "FeatureCollection" and the documents
   will be written as GeoJSON Feature objects. If another mapping name
   is chosen, a more "flat" structure will be used.  This option is
   ignored when converting to Elasticsearch >=7 (see `Removal of mapping types
<https://www.elastic.co/guide/en/elasticsearch/reference/current/removal-of-types.html>`__).
   With ElasticSearch 7 or later, a "flat" structure is always used.

Even

--
Spatialys - Geospatial professional services
http://www.spatialys.com
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Loading GeoJSON to ES 7

xbartolone
Thanks Even, I was misleading the behavior. However, is there a way to ingest and set the value of a unique identifier, for instance geonameid, in the key _id of each item of the index?

Francesco
Il 17 mar 2020, 22:13 +0100, Even Rouault <[hidden email]>, ha scritto:
Francesco,

The behaviour in ES7 has been changed. Now only flat structure is supported.
This was discussed in https://github.com/OSGeo/gdal/issues/1246 and I've just
made the doc of the driver clearer:

- **MAPPING_NAME=**\ =name. (Elasticsearch < 7) Name of the mapping type
within the index.
By default, the mapping name is "FeatureCollection" and the documents
will be written as GeoJSON Feature objects. If another mapping name
is chosen, a more "flat" structure will be used. This option is
ignored when converting to Elasticsearch >=7 (see `Removal of mapping types
<https://www.elastic.co/guide/en/elasticsearch/reference/current/removal-of-types.html>`__).
With ElasticSearch 7 or later, a "flat" structure is always used.

Even

--
Spatialys - Geospatial professional services
http://www.spatialys.com

_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Loading GeoJSON to ES 7

Even Rouault-2
On mercredi 18 mars 2020 09:05:39 CET Francesco Bartoli wrote:
> Thanks Even, I was misleading the behavior. However, is there a way to
> ingest and set the value of a unique identifier, for instance geonameid, in
> the key _id of each item of the index?

Yes, you need to set the "_id" field of the feature before calling
CreateFeature()

Even

>
> Francesco
>
> Il 17 mar 2020, 22:13 +0100, Even Rouault <[hidden email]>, ha
scritto:

> > Francesco,
> >
> > The behaviour in ES7 has been changed. Now only flat structure is
> > supported. This was discussed in
> > https://github.com/OSGeo/gdal/issues/1246 and I've just made the doc of
> > the driver clearer:
> >
> > - **MAPPING_NAME=**\ =name. (Elasticsearch < 7) Name of the mapping type
> > within the index.
> > By default, the mapping name is "FeatureCollection" and the documents
> > will be written as GeoJSON Feature objects. If another mapping name
> > is chosen, a more "flat" structure will be used. This option is
> > ignored when converting to Elasticsearch >=7 (see `Removal of mapping
> > types
> > <https://www.elastic.co/guide/en/elasticsearch/reference/current/removal-o
> > f-types.html>`__). With ElasticSearch 7 or later, a "flat" structure is
> > always used.
> >
> > Even
> >
> > --
> > Spatialys - Geospatial professional services
> > http://www.spatialys.com


--
Spatialys - Geospatial professional services
http://www.spatialys.com
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Loading GeoJSON to ES 7

xbartolone
Does it mean that can be done only programmatically? How would it be translated into a -lco of ogr2ogr
Il 18 mar 2020, 10:36 +0100, Even Rouault <[hidden email]>, ha scritto:
On mercredi 18 mars 2020 09:05:39 CET Francesco Bartoli wrote:
Thanks Even, I was misleading the behavior. However, is there a way to
ingest and set the value of a unique identifier, for instance geonameid, in
the key _id of each item of the index?

Yes, you need to set the "_id" field of the feature before calling
CreateFeature()

Even


Francesco

Il 17 mar 2020, 22:13 +0100, Even Rouault <[hidden email]>, ha
scritto:
Francesco,

The behaviour in ES7 has been changed. Now only flat structure is
supported. This was discussed in
https://github.com/OSGeo/gdal/issues/1246 and I've just made the doc of
the driver clearer:

- **MAPPING_NAME=**\ =name. (Elasticsearch < 7) Name of the mapping type
within the index.
By default, the mapping name is "FeatureCollection" and the documents
will be written as GeoJSON Feature objects. If another mapping name
is chosen, a more "flat" structure will be used. This option is
ignored when converting to Elasticsearch >=7 (see `Removal of mapping
types
<https://www.elastic.co/guide/en/elasticsearch/reference/current/removal-o
f-types.html>`__). With ElasticSearch 7 or later, a "flat" structure is
always used.

Even

--
Spatialys - Geospatial professional services
http://www.spatialys.com


--
Spatialys - Geospatial professional services
http://www.spatialys.com

_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Loading GeoJSON to ES 7

Even Rouault-2
On mercredi 18 mars 2020 10:49:48 CET Francesco Bartoli wrote:
> Does it mean that can be done only programmatically? How would it be
> translated into a -lco of ogr2ogr?

With ogr2ogr, you can do something like

ogr2ogr [...] {src_ds_name} -sql "select geonameid as _id, ... from ..."

Even

--
Spatialys - Geospatial professional services
http://www.spatialys.com
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev