[gdal-dev] GDAL GeoPackage raster support

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

[gdal-dev] GDAL GeoPackage raster support

Even Rouault-2
Hi,

I've committed in GDAL trunk the final pieces of the code needed to support
GeoPackage rasters in read/creation/update modes. Feedback from early testers
would be much appreciated, particularly regarding interoperability with other
software stacks (1), as I could basically only test round-tripping with GDAL.

Documentation can be found at :
http://www.gdal.org/drv_geopackage_raster.html

Small samples generated by GDAL can be found at:
http://download.osgeo.org/gdal/data/geopackage/

This also includes the script that generated them.

Unless I've missed something, we now have support for all of the GeoPackage
specification regarding regarding raster, including its extensions.

On the vector part, the support is also rather complete, with spatial index
and curve geometries added some time ago.
What is missing :
- reading/writing vector layer level metadata. This should now be possible
with the GDAL-OGR unification. Would require an update in ogrinfo to display
such metadata, and in ogr2ogr to define and propagate them.
- use of schema/column constraints in vector layers. We could likely check
that the constraints are met when creating/update a feature, but we don't have
yet the matching concepts in the OGR abstraction to create the constraints
(although this can be done at hand with simple SQL inserts)
- creating the triggers that correspond to the gpkg_geometry_type_trigger and
gpkg_srs_id_trigger extensions (we should already behave OK with them if those
triggers are defined, since we have the support SQL functions)


Best regards,

Even

(1) A report of my experiments with the GeoServer community GeoPackage plugin
: http://sourceforge.net/p/geoserver/mailman/message/33136750/

--
Spatialys - Geospatial professional services
http://www.spatialys.com
_______________________________________________
gdal-dev mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: GDAL GeoPackage raster support

pcav
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi Even,

Il 10/12/2014 23:52, Even Rouault ha scritto:
> I've committed in GDAL trunk the final pieces of the code needed to
> support GeoPackage rasters in read/creation/update modes. Feedback
> from early testers would be much appreciated, particularly
> regarding interoperability with other software stacks (1), as I
> could basically only test round-tripping with GDAL.

unfortunately, QGIS does not seem yet able to load .gpkg, either
through load vector or through the Browser, so I cannot thest this.
I think I've already opened tickets about it.
All the best.

- --
Paolo Cavallini - www.faunalia.eu
QGIS & PostGIS courses: http://www.faunalia.eu/training.html
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iEYEARECAAYFAlSJuNwACgkQ/NedwLUzIr5mhACgitpy9z4m4+SdpfVdpmaq7zwJ
2jkAoLEurJbwLyaruFbmNxcv60AnN1Zo
=ee81
-----END PGP SIGNATURE-----
_______________________________________________
gdal-dev mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: GDAL GeoPackage raster support

Even Rouault-2
Le jeudi 11 décembre 2014 16:31:41, Paolo Cavallini a écrit :

> Hi Even,
>
> Il 10/12/2014 23:52, Even Rouault ha scritto:
> > I've committed in GDAL trunk the final pieces of the code needed to
> > support GeoPackage rasters in read/creation/update modes. Feedback
> > from early testers would be much appreciated, particularly
> > regarding interoperability with other software stacks (1), as I
> > could basically only test round-tripping with GDAL.
>
> unfortunately, QGIS does not seem yet able to load .gpkg, either
> through load vector or through the Browser, so I cannot thest this.
> I think I've already opened tickets about it.

Paolo,

I'm confused by your comment :
- are you trying to load a raster geopackage through "load vector"... ? That
will not do it. You should open it with "load raster". That works with my
antique QGIS 1.8 linked against GDAL trunk ;-)
- are you using GDAL trunk ? This is necessary for raster support. Vector
support should mostly work with GDAL 1.11

Even

--
Spatialys - Geospatial professional services
http://www.spatialys.com
_______________________________________________
gdal-dev mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: GDAL GeoPackage raster support

pcav
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi Even,

Il 11/12/2014 16:37, Even Rouault ha scritto:

> I'm confused by your comment :

sorry I was unclear:

> - are you trying to load a raster geopackage through "load
> vector"... ? That will not do it. You should open it with "load
> raster". That works with my antique QGIS 1.8 linked against GDAL
> trunk ;-) - are you using GDAL trunk ? This is necessary for raster
> support. Vector support should mostly work with GDAL 1.11

I'm saying that QGIS cannot load .gpkg, therefore I cannot do more
tests. When fixed, I'm available for testing.

All the best.

- --
Paolo Cavallini - www.faunalia.eu
QGIS & PostGIS courses: http://www.faunalia.eu/training.html
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iEYEARECAAYFAlSJuvYACgkQ/NedwLUzIr5YrACgsVW3i48tsZqJTeFfA8iWPjhj
OXYAnRZWgxCAUZWm9qmstCyjbUOw7XMa
=qq+j
-----END PGP SIGNATURE-----
_______________________________________________
gdal-dev mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: GDAL GeoPackage raster support

Stefan Ziegler-3
Hi

I'm playing with the new Geopackage raster support which is really nice. Thanks for that. Some feedback so far:

* Loading in QGIS is instant when overviews are added with gdaladdo [1] but really slow when there are no  overviews [2].
* Adding overviews to some areal images is really slow: gdaladdo only eats < 10 % of the cpu. Is this some slow storage medium issue (harddisk)?

best regards
Stefan




On Thu, Dec 11, 2014 at 4:40 PM, Paolo Cavallini <[hidden email]> wrote:
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi Even,

Il 11/12/2014 16:37, Even Rouault ha scritto:

> I'm confused by your comment :

sorry I was unclear:

> - are you trying to load a raster geopackage through "load
> vector"... ? That will not do it. You should open it with "load
> raster". That works with my antique QGIS 1.8 linked against GDAL
> trunk ;-) - are you using GDAL trunk ? This is necessary for raster
> support. Vector support should mostly work with GDAL 1.11

I'm saying that QGIS cannot load .gpkg, therefore I cannot do more
tests. When fixed, I'm available for testing.

All the best.

- --
Paolo Cavallini - www.faunalia.eu
QGIS & PostGIS courses: http://www.faunalia.eu/training.html
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iEYEARECAAYFAlSJuvYACgkQ/NedwLUzIr5YrACgsVW3i48tsZqJTeFfA8iWPjhj
OXYAnRZWgxCAUZWm9qmstCyjbUOw7XMa
=qq+j
-----END PGP SIGNATURE-----
_______________________________________________
gdal-dev mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/gdal-dev

_______________________________________________
gdal-dev mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: GDAL GeoPackage raster support

Even Rouault-2
Stefan,

>
> I'm playing with the new Geopackage raster support which is really nice.
> Thanks for that. Some feedback so far:
>
> * Loading in QGIS is instant when overviews are added with gdaladdo [1] but
> really slow when there are no  overviews [2].

As will other formats with a dataset of the same size I guess ?

> * Adding overviews to some areal images is really slow: gdaladdo only eats
> < 10 % of the cpu. Is this some slow storage medium issue (harddisk)?

In the update scenario like gdaladdo, an optimization that is used when
creating the file from scratch (gdal_translate) isn't applied. You can try
adding "--config OGR_SQLITE_SYNCHRONOUS OFF" on the gdaladdo command line. I
suspect this must be the issue. I was a bit hesitant to apply it by default in
update cases since it means that a interruption of the process (ctrl-c) could
damage the database. But this is generally the case for other formats, so we
probably could apply it too.

Even

>
> [1]: http://catais.org/tmp/hoehen.gpkg
> [2]: http://catais.org/tmp/hoehen_ohne.gpkg
>
> best regards
> Stefan
>
>
>
>
> On Thu, Dec 11, 2014 at 4:40 PM, Paolo Cavallini <[hidden email]>
>
> wrote:
> > -----BEGIN PGP SIGNED MESSAGE-----
> > Hash: SHA1
> >
> > Hi Even,
> >
> > Il 11/12/2014 16:37, Even Rouault ha scritto:
> > > I'm confused by your comment :
> > sorry I was unclear:
> > > - are you trying to load a raster geopackage through "load
> > > vector"... ? That will not do it. You should open it with "load
> > > raster". That works with my antique QGIS 1.8 linked against GDAL
> > > trunk ;-) - are you using GDAL trunk ? This is necessary for raster
> > > support. Vector support should mostly work with GDAL 1.11
> >
> > I'm saying that QGIS cannot load .gpkg, therefore I cannot do more
> > tests. When fixed, I'm available for testing.
> >
> > All the best.
> >
> > - --
> > Paolo Cavallini - www.faunalia.eu
> > QGIS & PostGIS courses: http://www.faunalia.eu/training.html
> > -----BEGIN PGP SIGNATURE-----
> > Version: GnuPG v1
> >
> > iEYEARECAAYFAlSJuvYACgkQ/NedwLUzIr5YrACgsVW3i48tsZqJTeFfA8iWPjhj
> > OXYAnRZWgxCAUZWm9qmstCyjbUOw7XMa
> > =qq+j
> > -----END PGP SIGNATURE-----
> > _______________________________________________
> > gdal-dev mailing list
> > [hidden email]
> > http://lists.osgeo.org/mailman/listinfo/gdal-dev

--
Spatialys - Geospatial professional services
http://www.spatialys.com
_______________________________________________
gdal-dev mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: GDAL GeoPackage raster support

Pepijn Van Eeckhoudt
On 17/12/14 11:18, Even Rouault wrote:
> In the update scenario like gdaladdo, an optimization that is used when
> creating the file from scratch (gdal_translate) isn't applied. You can try
> adding "--config OGR_SQLITE_SYNCHRONOUS OFF" on the gdaladdo command line. I
> suspect this must be the issue. I was a bit hesitant to apply it by default in
> update cases since it means that a interruption of the process (ctrl-c) could
> damage the database. But this is generally the case for other formats, so we
> probably could apply it too.
Not sure if this is feasible in GDAL, but what I did in my own code is
to use transactions to speed things up. All the tile inserts are batched
in transactions of 100 tiles or so.
I had a look at the driver code, but it wasn't immediately clear to me
if this is already happening or not. Doesn't look like it though.

Pepijn
_______________________________________________
gdal-dev mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: GDAL GeoPackage raster support

Even Rouault-2
Le mercredi 17 décembre 2014 13:03:13, Pepijn Van Eeckhoudt a écrit :

> On 17/12/14 11:18, Even Rouault wrote:
> > In the update scenario like gdaladdo, an optimization that is used when
> > creating the file from scratch (gdal_translate) isn't applied. You can
> > try adding "--config OGR_SQLITE_SYNCHRONOUS OFF" on the gdaladdo command
> > line. I suspect this must be the issue. I was a bit hesitant to apply it
> > by default in update cases since it means that a interruption of the
> > process (ctrl-c) could damage the database. But this is generally the
> > case for other formats, so we probably could apply it too.
>
> Not sure if this is feasible in GDAL, but what I did in my own code is
> to use transactions to speed things up. All the tile inserts are batched
> in transactions of 100 tiles or so.
> I had a look at the driver code, but it wasn't immediately clear to me
> if this is already happening or not. Doesn't look like it though.

No, it doesn't for now. Could indeed be a potential optimization.

>
> Pepijn
> _______________________________________________
> gdal-dev mailing list
> [hidden email]
> http://lists.osgeo.org/mailman/listinfo/gdal-dev

--
Spatialys - Geospatial professional services
http://www.spatialys.com
_______________________________________________
gdal-dev mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: GDAL GeoPackage raster support

Even Rouault-2
Le mercredi 17 décembre 2014 13:18:16, Even Rouault a écrit :

> Le mercredi 17 décembre 2014 13:03:13, Pepijn Van Eeckhoudt a écrit :
> > On 17/12/14 11:18, Even Rouault wrote:
> > > In the update scenario like gdaladdo, an optimization that is used when
> > > creating the file from scratch (gdal_translate) isn't applied. You can
> > > try adding "--config OGR_SQLITE_SYNCHRONOUS OFF" on the gdaladdo
> > > command line. I suspect this must be the issue. I was a bit hesitant
> > > to apply it by default in update cases since it means that a
> > > interruption of the process (ctrl-c) could damage the database. But
> > > this is generally the case for other formats, so we probably could
> > > apply it too.
> >
> > Not sure if this is feasible in GDAL, but what I did in my own code is
> > to use transactions to speed things up. All the tile inserts are batched
> > in transactions of 100 tiles or so.
> > I had a look at the driver code, but it wasn't immediately clear to me
> > if this is already happening or not. Doesn't look like it though.
>
> No, it doesn't for now. Could indeed be a potential optimization.

Implemented now. Greatly improves performance of overview creation indeed.

>
> > Pepijn
> > _______________________________________________
> > gdal-dev mailing list
> > [hidden email]
> > http://lists.osgeo.org/mailman/listinfo/gdal-dev

--
Spatialys - Geospatial professional services
http://www.spatialys.com
_______________________________________________
gdal-dev mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/gdal-dev