[gdal-dev] Exporting large geojsons to postgis using ogr2ogr

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

[gdal-dev] Exporting large geojsons to postgis using ogr2ogr

Hemant Bist-2
Hi,
ogr2ogr is repeatedly failing for me with the following error when I try to import a large geojson (5gb) to postgres. The geojson itself was created by ogr2ogr from a ESRI gdb file.
I would appreciate any pointers for workarounds. I only found 1] which indicates that I may have to manually(using custom script) split the geojson file.

 ogr2ogr  -f "PostgreSQL" -append -update -nlt GEOMETRY -t_srs EPSG:4326 PG:"dbname=mydb  tables=dummy" vol2/myuser/delme/a.json'  -nln tmpx -lco FID=gid -lco PRECISION=NO

ERROR 1: GeoJSON parsing error: buffer size overflow (at offset 0)
ERROR 4: Failed to read GeoJSON data
FAILURE:
Unable to open datasource `/vol2/myuser/delme/a.json' with the following drivers.
  -> PCIDSK
  -> JPEG2000
  -> PDF
  -> ESRI Shapefile


_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Exporting large geojsons to postgis using ogr2ogr

Even Rouault-2

On jeudi 2 novembre 2017 13:39:56 CET Hemant Bist wrote:

> Hi,

> ogr2ogr is repeatedly failing for me with the following error when I try to

> import a large geojson (5gb) to postgres. The geojson itself was created by

> ogr2ogr from a ESRI gdb file.

> I would appreciate any pointers for workarounds. I only found 1] which

> indicates that I may have to manually(using custom script) split the

> geojson file.

 

I've fixed this a couple days ago in GDAL trunk to be able to read arbitrarily large GeoJSON files with little RAM.

 

--

Spatialys - Geospatial professional services

http://www.spatialys.com


_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Exporting large geojsons to postgis using ogr2ogr

dagibbs22
In reply to this post by Hemant Bist-2
Hi,
I'm trying to use gdal_rasterize on a 20 GB geojson but it's not working
(the geojson field I want to include in the raster is growth). I assume it's
because the geojson is too large. What can I do to rasterize the geojson
into tiles of 10x10 degrees?

ubuntu@ip-192-168-80-222:~$ gdal_rasterize -tr 0.00025 0.00025 -co
COMPRESS=LZW plantations.geojson test.tif -te -60 -30 -50 -20 -a growth
ERROR 1: JSON parsing error: buffer size overflow (at offset 0)
ERROR 4: Failed to read GeoJSON data




--
Sent from: http://osgeo-org.1560.x6.nabble.com/GDAL-Dev-f3742093.html
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev