[gdal-dev] OGR C API PostGIS geometry with wrong field entries

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

[gdal-dev] OGR C API PostGIS geometry with wrong field entries

Markus Metz-3
In the GRASS module v.in.ogr we follow pretty much the vector API tutorial. The only difference is that we first fetch the geometry of a feature, then the fields. When input is a PG database, sometimes the wrong field entries are associated with the geometries, as if geometries and field entries were mixed up. This happens when several different v.in.ogr processes are trying to read the same features from the same PG database at the same time. When instead using several different ogr2ogr processes, this mixing up of geometries and field entries does not happen. I could not find any hints in the OGR PostGIS documentation about how to avoid this problem.

Any hints are highly appreciated!

Markus M

_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: OGR C API PostGIS geometry with wrong field entries

Even Rouault-2
Markus,

> In the GRASS module v.in.ogr we follow pretty much the vector API tutorial.
> The only difference is that we first fetch the geometry of a feature, then
> the fields. When input is a PG database, sometimes the wrong field entries
> are associated with the geometries, as if geometries and field entries were
> mixed up. This happens when several different v.in.ogr processes are trying
> to read the same features from the same PG database at the same time. When
> instead using several different ogr2ogr processes, this mixing up of
> geometries and field entries does not happen. I could not find any hints in
> the OGR PostGIS documentation about how to avoid this problem.

This is indeed a weird problem. I don't have a theory. The PG driver
implements a lot of lazy loading strategy to run efficiently with databases
with many layers, but I wouldn't expect potential issues to be triggered by
the fact that several v.in.org processes run in parallel... And I checked that
fields returned from the PG system tables are sorted by attnum, so they should
always be returned in the same order.

Are you sure there's no latent memory corruption in v.in.ogr ? Did you check
with Valgrind ?

Even

--
Spatialys - Geospatial professional services
http://www.spatialys.com
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: OGR C API PostGIS geometry with wrong field entries

Markus Metz-3
Even,

On Mon, Dec 2, 2019 at 7:46 PM Even Rouault <[hidden email]> wrote:
>
> Markus,
> [...]
>
> Are you sure there's no latent memory corruption in v.in.ogr ? Did you check
> with Valgrind ?

Yes, I checked with valgrind, no memory corruption. I tried v.in.ogr with other OGR formats, no problem. I am out of ideas, that's why I am asking here. I will prick the users reporting the problem for more details.

Thanks a lot for your quick response!

Markus M


_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: OGR C API PostGIS geometry with wrong field entries

Andreas Oxenstierna
In reply to this post by Markus Metz-3
I am no Postgre expert but we have experienced similar cases when running complex transactions with large geometries, i.e. faulty results.
When splitting the transaction into several pieces, i.e. enforcing COMMITs (Postgre doesn't support COMMIT inside functions), the correct results are given.
One guess is that this may happen for large geometries which are pushed to TOAST tables.

Check Postgre memory parameters and trace exactly what is happening in the database.
Test also with EXTERNAL STORAGE for the geometry if they are really large.


In the GRASS module v.in.ogr we follow pretty much the vector API tutorial. The only difference is that we first fetch the geometry of a feature, then the fields. When input is a PG database, sometimes the wrong field entries are associated with the geometries, as if geometries and field entries were mixed up. This happens when several different v.in.ogr processes are trying to read the same features from the same PG database at the same time. When instead using several different ogr2ogr processes, this mixing up of geometries and field entries does not happen. I could not find any hints in the OGR PostGIS documentation about how to avoid this problem.

Any hints are highly appreciated!

Markus M

_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev


-- 
Hälsningar

Andreas Oxenstierna
T-Kartor Geospatial AB
Olof Mohlins väg 12 Kristianstad
mobile: +46 733 206831
mailto: [hidden email]
http://www.t-kartor.com

_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: OGR C API PostGIS geometry with wrong field entries

Markus Metz-3
Thanks for your feedback! I think I figured it out. The GRASS module v.in.ogr goes through the input features twice, but only for polygons. This is needed to properly convert non-topological polygons to topological areas in GRASS. The order of features can be different between the first and second pass through the input features. We are now using FID to map features in the second pass to features in the first pass. This solution is not ideal because in between the two passes, features might be added or deleted. Ultimately, we will go through the input features only once and make a local copy for the second pass.

On Tue, Dec 3, 2019 at 8:32 AM Andreas Oxenstierna <[hidden email]> wrote:
I am no Postgre expert but we have experienced similar cases when running complex transactions with large geometries, i.e. faulty results.
When splitting the transaction into several pieces, i.e. enforcing COMMITs (Postgre doesn't support COMMIT inside functions), the correct results are given.
One guess is that this may happen for large geometries which are pushed to TOAST tables.

Check Postgre memory parameters and trace exactly what is happening in the database.
Test also with EXTERNAL STORAGE for the geometry if they are really large.


In the GRASS module v.in.ogr we follow pretty much the vector API tutorial. The only difference is that we first fetch the geometry of a feature, then the fields. When input is a PG database, sometimes the wrong field entries are associated with the geometries, as if geometries and field entries were mixed up. This happens when several different v.in.ogr processes are trying to read the same features from the same PG database at the same time. When instead using several different ogr2ogr processes, this mixing up of geometries and field entries does not happen. I could not find any hints in the OGR PostGIS documentation about how to avoid this problem.

Any hints are highly appreciated!

Markus M

_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev


-- 
Hälsningar

Andreas Oxenstierna
T-Kartor Geospatial AB
Olof Mohlins väg 12 Kristianstad
mobile: +46 733 206831
mailto: [hidden email]
http://www.t-kartor.com
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev

_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev