[gdal-dev] Handle geojson bigger than 200MB

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

[gdal-dev] Handle geojson bigger than 200MB

xbartolone
Dear devs,

In the pygeoapi project we are experiencing this issue https://github.com/geopython/pygeoapi/issues/371. I've just started a PR https://github.com/geopython/pygeoapi/pull/372 hich basically tries to catch if all properties of the feature have null values and raise an exception ourself.
Catching null values is due because GDAL doesn't throw the exception if OGR_GEOJSON_MAX_OBJ_SIZE has the default value. I'm wondering if this is a bug or an expected behavior. If it is then I will file an issue.

Best,
Francesco

_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Handle geojson bigger than 200MB

Even Rouault-2
On vendredi 28 février 2020 15:10:46 CET Francesco Bartoli wrote:

> Dear devs,
>
> In the pygeoapi project we are experiencing this
> issue https://github.com/geopython/pygeoapi/issues/371. I've just started a
> PR https://github.com/geopython/pygeoapi/pull/372 hich basically tries to
> catch if all properties of the feature have null values and raise an
> exception ourself. Catching null values is due because GDAL doesn't throw
> the exception if OGR_GEOJSON_MAX_OBJ_SIZE has the default value. I'm
> wondering if this is a bug or an expected behavior. If it is then I will
> file an issue.

Use ogr.UseExceptions() (instead of gdal.UseExceptions())

Enabling exceptions is controlled separately in the 2 Python modules.

Even

--
Spatialys - Geospatial professional services
http://www.spatialys.com
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Handle geojson bigger than 200MB

xbartolone
Ahh I didn’t know, thanks Even

Francesco
Il 28 feb 2020, 15:19 +0100, Even Rouault <[hidden email]>, ha scritto:
On vendredi 28 février 2020 15:10:46 CET Francesco Bartoli wrote:
Dear devs,

In the pygeoapi project we are experiencing this
issue https://github.com/geopython/pygeoapi/issues/371. I've just started a
PR https://github.com/geopython/pygeoapi/pull/372 hich basically tries to
catch if all properties of the feature have null values and raise an
exception ourself. Catching null values is due because GDAL doesn't throw
the exception if OGR_GEOJSON_MAX_OBJ_SIZE has the default value. I'm
wondering if this is a bug or an expected behavior. If it is then I will
file an issue.

Use ogr.UseExceptions() (instead of gdal.UseExceptions())

Enabling exceptions is controlled separately in the 2 Python modules.

Even

--
Spatialys - Geospatial professional services
http://www.spatialys.com

_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Handle geojson bigger than 200MB

xbartolone
In reply to this post by Even Rouault-2
However, we are using both modules so how can we handle exceptions properly for both of them?

Using ogr.UseExceptions() the call https://github.com/geopython/pygeoapi/pull/372/files#diff-b105ae3f7a36059b16bdc16a08071016R361 gives the same result silently without any message to system output this time.

Thanks,
Francesco
Il 28 feb 2020, 15:19 +0100, Even Rouault <[hidden email]>, ha scritto:
On vendredi 28 février 2020 15:10:46 CET Francesco Bartoli wrote:
Dear devs,

In the pygeoapi project we are experiencing this
issue https://github.com/geopython/pygeoapi/issues/371. I've just started a
PR https://github.com/geopython/pygeoapi/pull/372 hich basically tries to
catch if all properties of the feature have null values and raise an
exception ourself. Catching null values is due because GDAL doesn't throw
the exception if OGR_GEOJSON_MAX_OBJ_SIZE has the default value. I'm
wondering if this is a bug or an expected behavior. If it is then I will
file an issue.

Use ogr.UseExceptions() (instead of gdal.UseExceptions())

Enabling exceptions is controlled separately in the 2 Python modules.

Even

--
Spatialys - Geospatial professional services
http://www.spatialys.com

_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Handle geojson bigger than 200MB

Even Rouault-2
On vendredi 28 février 2020 17:22:28 CET Francesco Bartoli wrote:
> However, we are using both modules so how can we handle exceptions properly
> for both of them?

You can use both gdal.UseExceptions() and ogr.UseExceptions()

>
> Using ogr.UseExceptions() the
> call https://github.com/geopython/pygeoapi/pull/372/files#diff-b105ae3f7a36
> 059b16bdc16a08071016R361 gives the same result silently without any message
> to system output this time.

On a quick test, ogr.UseExceptions() does the job for me:

$ OGR_GEOJSON_MAX_OBJ_SIZE=0 python
>>> from osgeo import ogr
>>> ogr.UseExceptions()
>>> ds = ogr.Open('poly.geojson')
>>> lyr = ds.GetLayer(0)
>>> lyr.GetNextFeature()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/even/gdal/git/gdal/gdal/swig/python/build/lib.linux-x86_64-2.7/
osgeo/ogr.py", line 1339, in GetNextFeature
    return _ogr.Layer_GetNextFeature(self, *args)
RuntimeError: GeoJSON object too complex, please see the
OGR_GEOJSON_MAX_OBJ_SIZE environment option
>>>

--
Spatialys - Geospatial professional services
http://www.spatialys.com
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Handle geojson bigger than 200MB

Just van den Broecke-2
On 28-02-20 17:29, Even Rouault wrote:
> On vendredi 28 février 2020 17:22:28 CET Francesco Bartoli wrote:
>> However, we are using both modules so how can we handle exceptions properly
>> for both of them?
AFAIK we only use OGR (vector) currently within pygeoapi, no raster/GDAL
yet.
>
> You can use both gdal.UseExceptions() and ogr.UseExceptions()
My bad: in the initial version of the pygeoapi OGRProvider I added
gdal.UseExceptions() from example code (py-gdalogr-cookbook). Was not
aware of ogr.UseExceptions()!

>
>>
>> Using ogr.UseExceptions() the
>> call https://github.com/geopython/pygeoapi/pull/372/files#diff-b105ae3f7a36
>> 059b16bdc16a08071016R361 gives the same result silently without any message
>> to system output this time.
>
> On a quick test, ogr.UseExceptions() does the job for me:
>
> $ OGR_GEOJSON_MAX_OBJ_SIZE=0 python
>>>> from osgeo import ogr
>>>> ogr.UseExceptions()
>>>> ds = ogr.Open('poly.geojson')
>>>> lyr = ds.GetLayer(0)
>>>> lyr.GetNextFeature()
> Traceback (most recent call last):
>    File "<stdin>", line 1, in <module>
>    File "/home/even/gdal/git/gdal/gdal/swig/python/build/lib.linux-x86_64-2.7/
> osgeo/ogr.py", line 1339, in GetNextFeature
>      return _ogr.Layer_GetNextFeature(self, *args)
> RuntimeError: GeoJSON object too complex, please see the
> OGR_GEOJSON_MAX_OBJ_SIZE environment option
>>>>
>

_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Handle geojson bigger than 200MB

xbartolone
Il 28 feb 2020, 17:49 +0100, Just van den Broecke <[hidden email]>, ha scritto:
On 28-02-20 17:29, Even Rouault wrote:
On vendredi 28 février 2020 17:22:28 CET Francesco Bartoli wrote:
However, we are using both modules so how can we handle exceptions properly
for both of them?
AFAIK we only use OGR (vector) currently within pygeoapi, no raster/GDAL
yet. 


You can use both gdal.UseExceptions() and ogr.UseExceptions()
My bad: in the initial version of the pygeoapi OGRProvider I added
gdal.UseExceptions() from example code (py-gdalogr-cookbook). Was not
aware of ogr.UseExceptions()!



Using ogr.UseExceptions() the
call https://github.com/geopython/pygeoapi/pull/372/files#diff-b105ae3f7a36
059b16bdc16a08071016R361 gives the same result silently without any message
to system output this time.

On a quick test, ogr.UseExceptions() does the job for me:

$ OGR_GEOJSON_MAX_OBJ_SIZE=0 python
from osgeo import ogr
ogr.UseExceptions()
ds = ogr.Open('poly.geojson')
lyr = ds.GetLayer(0)
lyr.GetNextFeature()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/even/gdal/git/gdal/gdal/swig/python/build/lib.linux-x86_64-2.7/
osgeo/ogr.py", line 1339, in GetNextFeature
return _ogr.Layer_GetNextFeature(self, *args)
RuntimeError: GeoJSON object too complex, please see the
OGR_GEOJSON_MAX_OBJ_SIZE environment option



_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev

_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Handle geojson bigger than 200MB

xbartolone
In reply to this post by Even Rouault-2
Thanks Even, it works but I have strange result with the size value. I wouldn’t have been expected the exception from this test:

OGR_GEOJSON_MAX_OBJ_SIZE=500MB ogr2ogr -f "GeoJSON" indonesia.geojson "WFS:geonode.wfp.org/geoserver/wfs?request=GetFeature&typename=geonode:glb_bnd_adm0_1&outputformat=json" -where “adm0_id=‘272’" 

indonesia.geojson is around 35MB so the exception shouldn’t raise if OGR_GEOJSON_MAX_OBJ_SIZE=50 AFAIK:

$ python
Python 3.7.6 (default, Jan 28 2020, 15:35:48)
[Clang 11.0.0 (clang-1100.0.33.17)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> os.environ["OGR_GEOJSON_MAX_OBJ_SIZE"] = “50”
>>> from osgeo import ogr
>>> ogr.UseExceptions()
>>> ds = ogr.Open('/Users/geobart/Downloads/indonesia.geojson’)
>>> lyr = ds.GetLayer(0)
>>> lyr.GetNextFeature()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/geobart/.pyenv/versions/pygeoapi/lib/python3.7/site-packages/osgeo/ogr.py", line 1632, in GetNextFeature
    return _ogr.Layer_GetNextFeature(self, *args)
RuntimeError: GeoJSON object too complex, please see the OGR_GEOJSON_MAX_OBJ_SIZE environment option

Is it a bug?
Il 28 feb 2020, 17:29 +0100, Even Rouault <[hidden email]>, ha scritto:
On vendredi 28 février 2020 17:22:28 CET Francesco Bartoli wrote:
However, we are using both modules so how can we handle exceptions properly
for both of them?

You can use both gdal.UseExceptions() and ogr.UseExceptions()


Using ogr.UseExceptions() the
call https://github.com/geopython/pygeoapi/pull/372/files#diff-b105ae3f7a36
059b16bdc16a08071016R361 gives the same result silently without any message
to system output this time.

On a quick test, ogr.UseExceptions() does the job for me:

$ OGR_GEOJSON_MAX_OBJ_SIZE=0 python
from osgeo import ogr
ogr.UseExceptions()
ds = ogr.Open('poly.geojson')
lyr = ds.GetLayer(0)
lyr.GetNextFeature()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/even/gdal/git/gdal/gdal/swig/python/build/lib.linux-x86_64-2.7/
osgeo/ogr.py", line 1339, in GetNextFeature
return _ogr.Layer_GetNextFeature(self, *args)
RuntimeError: GeoJSON object too complex, please see the
OGR_GEOJSON_MAX_OBJ_SIZE environment option


--
Spatialys - Geospatial professional services
http://www.spatialys.com

_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Handle geojson bigger than 200MB

Momtchil Momtchev


On 29/02/2020 20:34, Francesco Bartoli wrote:
Thanks Even, it works but I have strange result with the size value. I wouldn’t have been expected the exception from this test:

OGR_GEOJSON_MAX_OBJ_SIZE=500MB ogr2ogr -f "GeoJSON" indonesia.geojson "WFS:geonode.wfp.org/geoserver/wfs?request=GetFeature&typename=geonode:glb_bnd_adm0_1&outputformat=json" -where “adm0_id=‘272’" 

indonesia.geojson is around 35MB so the exception shouldn’t raise if OGR_GEOJSON_MAX_OBJ_SIZE=50 AFAIK:


OGR_GEOJSON_MAX_OBJ_SIZE is not a limit on the file size, but on the maximum memory allocation for a single GeoJSON feature which is estimated before the feature is read. As country borders are usually a single GeoJSON feature, a country with a complex coastline is the typical case where you need larger features.


-- 
Momtchil Momtchev [hidden email]

_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Handle geojson bigger than 200MB

xbartolone
thanks Momtchil, it makes sense now
Il 2 mar 2020, 10:12 +0100, Momtchil Momtchev <[hidden email]>, ha scritto:


On 29/02/2020 20:34, Francesco Bartoli wrote:
Thanks Even, it works but I have strange result with the size value. I wouldn’t have been expected the exception from this test:

OGR_GEOJSON_MAX_OBJ_SIZE=500MB ogr2ogr -f "GeoJSON" indonesia.geojson "WFS:geonode.wfp.org/geoserver/wfs?request=GetFeature&typename=geonode:glb_bnd_adm0_1&outputformat=json" -where “adm0_id=‘272’" 

indonesia.geojson is around 35MB so the exception shouldn’t raise if OGR_GEOJSON_MAX_OBJ_SIZE=50 AFAIK:


OGR_GEOJSON_MAX_OBJ_SIZE is not a limit on the file size, but on the maximum memory allocation for a single GeoJSON feature which is estimated before the feature is read. As country borders are usually a single GeoJSON feature, a country with a complex coastline is the typical case where you need larger features.


--  
Momtchil Momtchev [hidden email]
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev

_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev