[gdal-dev] Encoding of keys and values returned by GDALGetMetadata?

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

[gdal-dev] Encoding of keys and values returned by GDALGetMetadata?

Sean Gillies-3
Hi all,

Little is said at https://www.gdal.org/gdal_8h.html#a25929a66e99d92c2788485ecf13e2e82. Are these values UTF-8 (as was the intent in https://trac.osgeo.org/gdal/wiki/rfc5_unicode) or does it depend on the format driver? In https://github.com/mapbox/rasterio/issues/1663 a rasterio user is encountering HDF5 metadata items that can't be decoded using Python's UTF-8 codec and I've love to be able to sort this out.

Thanks,

--
Sean Gillies

_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Encoding of keys and values returned by GDALGetMetadata?

Even Rouault-2
Sean,

> Little is said at
> https://www.gdal.org/gdal_8h.html#a25929a66e99d92c2788485ecf13e2e82. Are
> these values UTF-8 (as was the intent in
> https://trac.osgeo.org/gdal/wiki/rfc5_unicode) or does it depend on the
> format driver?

The key/value mechanism itself just assumes NUL-terminated strings. The
general aim of GDAL should be to have UTF-8 everywhere, but the reality is
that not all drivers enforce this, so you might easily end up with 'random'
encodings.

> In https://github.com/mapbox/rasterio/issues/1663 a rasterio
> user is encountering HDF5 metadata items that can't be decoded using
> Python's UTF-8 codec and I've love to be able to sort this out.

OK, looking at the details of this particular case, this is an inappropriate
handling by the HDF5 driver of some types of attributes.
Fixed by
https://github.com/OSGeo/gdal/commit/9640098fc79b6ba6bb135f8715fa3e7dd2705691
which I've backported to 2.4 as well

Even

--
Spatialys - Geospatial professional services
http://www.spatialys.com
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Encoding of keys and values returned by GDALGetMetadata?

Sean Gillies-3
Even,

On Tue, Mar 26, 2019 at 2:57 PM Even Rouault <[hidden email]> wrote:
Sean,

> Little is said at
> https://www.gdal.org/gdal_8h.html#a25929a66e99d92c2788485ecf13e2e82. Are
> these values UTF-8 (as was the intent in
> https://trac.osgeo.org/gdal/wiki/rfc5_unicode) or does it depend on the
> format driver?

The key/value mechanism itself just assumes NUL-terminated strings. The
general aim of GDAL should be to have UTF-8 everywhere, but the reality is
that not all drivers enforce this, so you might easily end up with 'random'
encodings.

I'll watch out for this.
 

> In https://github.com/mapbox/rasterio/issues/1663 a rasterio
> user is encountering HDF5 metadata items that can't be decoded using
> Python's UTF-8 codec and I've love to be able to sort this out.

OK, looking at the details of this particular case, this is an inappropriate
handling by the HDF5 driver of some types of attributes.
Fixed by
https://github.com/OSGeo/gdal/commit/9640098fc79b6ba6bb135f8715fa3e7dd2705691
which I've backported to 2.4 as well

Even

Thanks for the quick fix!
 
--
Sean Gillies

_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev