[gdal-dev] Improve clipping of shapefiles

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

[gdal-dev] Improve clipping of shapefiles

Paul Meems
I have a large shapefile with over 2.8 million shapes (fishnet) and I have a border file with only 1 multipolygon.

I'm trying to clip the fishnet with the border.
Using code is takes about 5 min. using command line it takes even longer.

This is my command:
ogr2ogr fishnetClipped.shp fishnet.shp -clipsrc border.shp

And this is my C# code:
using (var dsFishnet = Ogr.Open(fishnetFilename, 0))
{
    // Select first layer:
    using (var layerFishnet = dsFishnet.GetLayerByIndex(0))
    {
        // Open border:
        using (var dsBorder = Ogr.Open(borderFilename, 0))
        {
            // Select first layer:
            using (var layerBorder = dsBorder.GetLayerByIndex(0))
            {
                // Get ESRI driver:
                using (var driver = Ogr.GetDriverByName("ESRI Shapefile"))
                {
                    // Create new shapefile:
                    using (var dsClipped = driver.CreateDataSource(outputFilename, new string[] { }))
                    {
                        // Create new layer:
                        var layerName = Path.GetFileNameWithoutExtension(outputFilename);
                        using (var layerClipped = dsClipped.CreateLayer(layerName, layerFishnet.GetSpatialRef(), layerFishnet.GetGeomType(), new string[] { }))
                        {
                            if (layerClipped == null)
                            {
                                throw new Exception("Layer creation failed.");
                            }

                            // Clip the layer:
                            layerFishnet.Clip(layerBorder, layerClipped, null, null, null);
                        }
                    }
                }
            }
        }
    }
}

I'm using GDAL 2.1.3

Can I somehow improve the speed?
I read something about putting one of the layers in memory or using a spatial filter.
But I can't find an example of how to do this.

Please advise.

Thanks,
Paul

_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Improve clipping of shapefiles

janhec
Can't really help you with the OGR part, however, I wrote utilities do solve this (really) fast for calculating grids based on this type of intersection.
Speed is often a matter of using specific tools utilizing the characteristics of the problem, i.c. the fishnet regularities.
Can you share the shapesfiles so I can try?
Regards,
Jan

On Mon, Sep 11, 2017 at 11:09 AM, Paul Meems <[hidden email]> wrote:
I have a large shapefile with over 2.8 million shapes (fishnet) and I have a border file with only 1 multipolygon.

I'm trying to clip the fishnet with the border.
Using code is takes about 5 min. using command line it takes even longer.

This is my command:
ogr2ogr fishnetClipped.shp fishnet.shp -clipsrc border.shp

And this is my C# code:
using (var dsFishnet = Ogr.Open(fishnetFilename, 0))
{
    // Select first layer:
    using (var layerFishnet = dsFishnet.GetLayerByIndex(0))
    {
        // Open border:
        using (var dsBorder = Ogr.Open(borderFilename, 0))
        {
            // Select first layer:
            using (var layerBorder = dsBorder.GetLayerByIndex(0))
            {
                // Get ESRI driver:
                using (var driver = Ogr.GetDriverByName("ESRI Shapefile"))
                {
                    // Create new shapefile:
                    using (var dsClipped = driver.CreateDataSource(outputFilename, new string[] { }))
                    {
                        // Create new layer:
                        var layerName = Path.GetFileNameWithoutExtension(outputFilename);
                        using (var layerClipped = dsClipped.CreateLayer(layerName, layerFishnet.GetSpatialRef(), layerFishnet.GetGeomType(), new string[] { }))
                        {
                            if (layerClipped == null)
                            {
                                throw new Exception("Layer creation failed.");
                            }

                            // Clip the layer:
                            layerFishnet.Clip(layerBorder, layerClipped, null, null, null);
                        }
                    }
                }
            }
        }
    }
}

I'm using GDAL 2.1.3

Can I somehow improve the speed?
I read something about putting one of the layers in memory or using a spatial filter.
But I can't find an example of how to do this.

Please advise.

Thanks,
Paul

_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev


_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Improve clipping of shapefiles

Mike Toews-2
In reply to this post by Paul Meems
On 11 September 2017 at 21:09, Paul Meems <[hidden email]> wrote:

> I have a large shapefile with over 2.8 million shapes (fishnet) and I have a
> border file with only 1 multipolygon.
>
> I'm trying to clip the fishnet with the border.
> Using code is takes about 5 min. using command line it takes even longer.
>
> This is my command:
> ogr2ogr fishnetClipped.shp fishnet.shp -clipsrc border.shp
> ...
> Can I somehow improve the speed?

If the fishnet shapefile does not have a spatial index, try creating
one. There are a few ways to do this:
http://www.gdal.org/drv_shapefile.html

I'm not sure if -clipsrc datasource will use the spatial index, but
it's worth trying.
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Improve clipping of shapefiles

Paul Meems
Thanks Mike for your reply.

After clipping I'm doing a dissolve. The clipping takes 5 min, which is more or less acceptable for 2.8 million shapes.
But the dissolve is taking over 30 min. for 680k shapes.

I've added a spatial index using
dsShapefile.ExecuteSQL("CREATE SPATIAL INDEX ON " + layerName, null, "sqlite");
but this didn't help with the clipping.

Before dissolving I add an attribute index:
dsShapefile.ExecuteSQL($"CREATE INDEX ON {layerName} USING Avg1", null, null);
and to dissolve I use 
.OpenEx(input, 4, null, null, null);
.wrapper_GDALVectorTranslateDestName();

with these options:
var options = new[]
{
    "-f", "ESRI Shapefile",
    "-overwrite",
    "-explodecollections",
    "-dialect", "sqlite",
    "-sql", "SELECT ST_Union(geometry), Avg1 FROM {layerName} GROUP BY Avg1"
};

When I import the shapefile in PostGIS and do the same query it just takes 6 min.
So I keep thinking I'm not doing it right with GDAL/OGR.

As Jan Heckman suggested I tried using the Clipper library. Luckily it is implemented in MapWinGIS as well.
The dissolve using Clipper is just taking 4 min.
I prefer not to use yet another library.

So I'm still open for more suggestions.




Paul

Paul Meems 
Release manager, configuration manager
and forum moderator of MapWindow GIS.
www.mapwindow.org

Owner of MapWindow.nl - Support for
Dutch speaking users.
www.mapwindow.nl


The MapWindow GIS project has moved to GitHub!


Download the latest MapWinGIS mapping engine.

Download the latest MapWindow 5 open source desktop application.


2017-09-11 23:34 GMT+02:00 Mike Toews <[hidden email]>:
On 11 September 2017 at 21:09, Paul Meems <[hidden email]> wrote:
> I have a large shapefile with over 2.8 million shapes (fishnet) and I have a
> border file with only 1 multipolygon.
>
> I'm trying to clip the fishnet with the border.
> Using code is takes about 5 min. using command line it takes even longer.
>
> This is my command:
> ogr2ogr fishnetClipped.shp fishnet.shp -clipsrc border.shp
> ...
> Can I somehow improve the speed?

If the fishnet shapefile does not have a spatial index, try creating
one. There are a few ways to do this:
http://www.gdal.org/drv_shapefile.html

I'm not sure if -clipsrc datasource will use the spatial index, but
it's worth trying.


_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev