shp2pgsql

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

shp2pgsql

dahaynes
CONTENTS DELETED
The author has deleted this message.
Reply | Threaded
Open this post in threaded view
|

Re: shp2pgsql

Stephen Woodbridge
On 2/18/2015 10:21 AM, David Haynes II wrote:

> Hello,
>
> I am working on a project that conducts high performance spatial
> analysis using postgresql and PostGIS. We have been searching for
> platforms that allow PostGIS to run in parallel. We have evaluated
> Postgres-xc and postgres-xl and determined that they do not meet our use
> case. Currently, I am evaluating CitusDB, but there seem to be some
> obstacles there as well. However, we are working on developing our own
> platform using Stado, which is dervied from GridSQL.
>
> We have had some initial results that look pretty promising and we are
> looking to modify the source code of the shp2pgsql to work with our new
> platform. If someone could provide the location of the code we would
> greatly appreciate it. Currently we are creating are geometries using
> the ST_GeomFromTxt function.

http://trac.osgeo.org/postgis/browser/trunk/loader

_______________________________________________
postgis-devel mailing list
[hidden email]
http://lists.osgeo.org/cgi-bin/mailman/listinfo/postgis-devel
Reply | Threaded
Open this post in threaded view
|

Re: shp2pgsql

Rémi Cura
Hey,
I find this research type very interesting.
Do you have something published about your experiment and test of various cluster-postgres flavours?

About loading geometry into your cluster, you may prefer to change OGR, which can read a lot more formats
(and is however slower than postgis shp2pgsql).

Cheers,
Rémi-C

2015-02-18 16:37 GMT+01:00 Stephen Woodbridge <[hidden email]>:
On 2/18/2015 10:21 AM, David Haynes II wrote:
Hello,

I am working on a project that conducts high performance spatial
analysis using postgresql and PostGIS. We have been searching for
platforms that allow PostGIS to run in parallel. We have evaluated
Postgres-xc and postgres-xl and determined that they do not meet our use
case. Currently, I am evaluating CitusDB, but there seem to be some
obstacles there as well. However, we are working on developing our own
platform using Stado, which is dervied from GridSQL.

We have had some initial results that look pretty promising and we are
looking to modify the source code of the shp2pgsql to work with our new
platform. If someone could provide the location of the code we would
greatly appreciate it. Currently we are creating are geometries using
the ST_GeomFromTxt function.

http://trac.osgeo.org/postgis/browser/trunk/loader

_______________________________________________
postgis-devel mailing list
[hidden email]
http://lists.osgeo.org/cgi-bin/mailman/listinfo/postgis-devel


_______________________________________________
postgis-devel mailing list
[hidden email]
http://lists.osgeo.org/cgi-bin/mailman/listinfo/postgis-devel
Reply | Threaded
Open this post in threaded view
|

Re: shp2pgsql

dahaynes
CONTENTS DELETED
The author has deleted this message.
Reply | Threaded
Open this post in threaded view
|

Re: shp2pgsql

Brent Wood
You might look at the capabilities of systems like Netezza & Teradata. They have also adapted various tools around Postgres/Postgis to support parallelisation.

This comment on Netezza might be of interest:

Cheers

Brent Wood


From: David Haynes II <[hidden email]>
To: PostGIS Development Discussion <[hidden email]>
Sent: Thursday, February 19, 2015 5:15 AM
Subject: Re: [postgis-devel] shp2pgsql

Nothing published yet, but we are developing a working document.
Mostly we are testing this viability of spatial datasets and if a platform supports complex spatial queries (i.e. vector raster overlay distributed across a series of nodes). In addition we plan to conducting testing on tile size and raster density in relation to these complex queries. 



On Wed, Feb 18, 2015 at 9:48 AM, Rémi Cura <[hidden email]> wrote:
Hey,
I find this research type very interesting.
Do you have something published about your experiment and test of various cluster-postgres flavours?

About loading geometry into your cluster, you may prefer to change OGR, which can read a lot more formats
(and is however slower than postgis shp2pgsql).

Cheers,
Rémi-C

2015-02-18 16:37 GMT+01:00 Stephen Woodbridge <[hidden email]>:
On 2/18/2015 10:21 AM, David Haynes II wrote:
Hello,

I am working on a project that conducts high performance spatial
analysis using postgresql and PostGIS. We have been searching for
platforms that allow PostGIS to run in parallel. We have evaluated
Postgres-xc and postgres-xl and determined that they do not meet our use
case. Currently, I am evaluating CitusDB, but there seem to be some
obstacles there as well. However, we are working on developing our own
platform using Stado, which is dervied from GridSQL.

We have had some initial results that look pretty promising and we are
looking to modify the source code of the shp2pgsql to work with our new
platform. If someone could provide the location of the code we would
greatly appreciate it. Currently we are creating are geometries using
the ST_GeomFromTxt function.

http://trac.osgeo.org/postgis/browser/trunk/loader

_______________________________________________
postgis-devel mailing list
[hidden email]
http://lists.osgeo.org/cgi-bin/mailman/listinfo/postgis-devel


_______________________________________________
postgis-devel mailing list
[hidden email]
http://lists.osgeo.org/cgi-bin/mailman/listinfo/postgis-devel



--
David Haynes, Ph.D.
Research Associate Terra Populus
Minnesota Population Center

_______________________________________________
postgis-devel mailing list
[hidden email]
http://lists.osgeo.org/cgi-bin/mailman/listinfo/postgis-devel



_______________________________________________
postgis-devel mailing list
[hidden email]
http://lists.osgeo.org/cgi-bin/mailman/listinfo/postgis-devel
Reply | Threaded
Open this post in threaded view
|

Re: shp2pgsql

Stefan Keller
David, PostGIS developers

David: Is there any update on parallelizing shp2pgsql?

Devs (and David): Are there any plans to complement shp2pgsql with
something like GeoPackage-to-PostGIS (gpkg2pgsql) ?

I sometimes refer to vector GeoPackage as the "Shapefile of the
future". Rémi's pointer to OGR is also valuable. But I'd expect that a
standalone tool like gpkg2pgsql could eventually be easier to be
realized and tuned.

:Stefan


2015-02-18 17:37 GMT+01:00 Brent Wood <[hidden email]>:

> You might look at the capabilities of systems like Netezza & Teradata. They
> have also adapted various tools around Postgres/Postgis to support
> parallelisation.
>
> This comment on Netezza might be of interest:
> http://geothought.blogspot.co.nz/2008/09/netezza-spatial.html
>
> Cheers
>
> Brent Wood
>
> ________________________________
> From: David Haynes II <[hidden email]>
> To: PostGIS Development Discussion <[hidden email]>
> Sent: Thursday, February 19, 2015 5:15 AM
> Subject: Re: [postgis-devel] shp2pgsql
>
> Nothing published yet, but we are developing a working document.
> Mostly we are testing this viability of spatial datasets and if a platform
> supports complex spatial queries (i.e. vector raster overlay distributed
> across a series of nodes). In addition we plan to conducting testing on tile
> size and raster density in relation to these complex queries.
>
>
>
> On Wed, Feb 18, 2015 at 9:48 AM, Rémi Cura <[hidden email]> wrote:
>
> Hey,
> I find this research type very interesting.
> Do you have something published about your experiment and test of various
> cluster-postgres flavours?
>
> About loading geometry into your cluster, you may prefer to change OGR,
> which can read a lot more formats
> (and is however slower than postgis shp2pgsql).
>
> Cheers,
> Rémi-C
>
> 2015-02-18 16:37 GMT+01:00 Stephen Woodbridge <[hidden email]>:
>
> On 2/18/2015 10:21 AM, David Haynes II wrote:
>
> Hello,
>
> I am working on a project that conducts high performance spatial
> analysis using postgresql and PostGIS. We have been searching for
> platforms that allow PostGIS to run in parallel. We have evaluated
> Postgres-xc and postgres-xl and determined that they do not meet our use
> case. Currently, I am evaluating CitusDB, but there seem to be some
> obstacles there as well. However, we are working on developing our own
> platform using Stado, which is dervied from GridSQL.
>
> We have had some initial results that look pretty promising and we are
> looking to modify the source code of the shp2pgsql to work with our new
> platform. If someone could provide the location of the code we would
> greatly appreciate it. Currently we are creating are geometries using
> the ST_GeomFromTxt function.
>
>
> http://trac.osgeo.org/postgis/browser/trunk/loader
>
> _______________________________________________
> postgis-devel mailing list
> [hidden email]
> http://lists.osgeo.org/cgi-bin/mailman/listinfo/postgis-devel
>
>
>
> _______________________________________________
> postgis-devel mailing list
> [hidden email]
> http://lists.osgeo.org/cgi-bin/mailman/listinfo/postgis-devel
>
>
>
>
> --
> David Haynes, Ph.D.
> Research Associate Terra Populus
> Minnesota Population Center
> www.terrapop.org
>
> _______________________________________________
> postgis-devel mailing list
> [hidden email]
> http://lists.osgeo.org/cgi-bin/mailman/listinfo/postgis-devel
>
>
>
> _______________________________________________
> postgis-devel mailing list
> [hidden email]
> http://lists.osgeo.org/cgi-bin/mailman/listinfo/postgis-devel
_______________________________________________
postgis-devel mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/postgis-devel
Reply | Threaded
Open this post in threaded view
|

Re: shp2pgsql

Paul Norman
On 1/24/2016 4:38 PM, Stefan Keller wrote:
> David, PostGIS developers
>
> David: Is there any update on parallelizing shp2pgsql?

Having used shp2pgsql with large (>100GB) data loads, I'm not sure
there's a huge gain to be had by parallelizing shp2pgsql. Every case
I've seen where it was being used for a large data load involved a data
source distributed as multiple shapefiles which could be loaded in
parallel, and dealing with parallelization on that level is generally
more efficient.

This isn't to say that there wouldn't be any benefit, just that it's
limited in usefulness.
_______________________________________________
postgis-devel mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/postgis-devel
Reply | Threaded
Open this post in threaded view
|

Re: shp2pgsql

dahaynes
In reply to this post by Stefan Keller
CONTENTS DELETED
The author has deleted this message.