[gdal-dev] GDAL 2.3.1 on linux disregarding nodata values

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

[gdal-dev] GDAL 2.3.1 on linux disregarding nodata values

TomvanTilburg
I have a fresh gdal 2.3.1 build on a ubuntu 17.10 that tells me:

> gdalinfo dem.tif -stats -mm
[..snip...]
Band 1 Block=2200x1 Type=Float32, ColorInterp=Gray
  Min=-2.233 Max=340282346638529993179660072199368212480.000   Computed Min/Max=-2.233,340282346638528859811704183484516925440.000
  Minimum=-2.233, Maximum=340282346638529993179660072199368212480.000, Mean=42029510023671999325552505715632898048.000, StdDev=111961692493879995008202507419807383552.000
  NoData Value=3.40282346638529011e+38
  Metadata:
    STATISTICS_MAXIMUM=3.4028234663853e+38
    STATISTICS_MEAN=4.2029510023672e+37
    STATISTICS_MINIMUM=-2.2330000400543
    STATISTICS_STDDEV=1.1196169249388e+38


Obviously, cells with nodata value are included in the stats, although rounded. This seems wrong to me and I know gdal propagates the values with for instance gdal_translate.

When doing the same thing on my osgeo4w64 with gdal 2.2.4 or gdal 2.4.0 (dev) it tells me:

> gdalinfo dem.tif -stats -mm
[..snip...]
Band 1 Block=2200x1 Type=Float32, ColorInterp=Gray
  Min=-2.233 Max=2.176   Computed Min/Max=-2.233,2.176
  Minimum=-2.233, Maximum=2.176, Mean=0.169, StdDev=0.316
  NoData Value=3.40282346638529011e+38
  Metadata:
    STATISTICS_MAXIMUM=2.1760001182556
    STATISTICS_MEAN=0.16871771630616
    STATISTICS_MINIMUM=-2.2330000400543
    STATISTICS_STDDEV=0.31600319455862

This seems to be the correct answer.

Worth noticing: there is another gdal libs from package on the linux machine:
gdal-data                                  2.2.1+dfsg-2build3
libgdal20                                  2.2.1+dfsg-2build3
python3-gdal                               2.2.1+dfsg-2build3

Any thoughts?

Best,
 Tom

_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: GDAL 2.3.1 on linux disregarding nodata values

Even Rouault-2
On jeudi 5 juillet 2018 14:22:16 CEST Tom van Tilburg wrote:

> I have a fresh gdal 2.3.1 build on a ubuntu 17.10 that tells me:
> > gdalinfo dem.tif -stats -mm
>
> [..snip...]
> Band 1 Block=2200x1 Type=Float32, ColorInterp=Gray
>   Min=-2.233 Max=340282346638529993179660072199368212480.000   Computed
> Min/Max=-2.233,340282346638528859811704183484516925440.000
>   Minimum=-2.233, Maximum=340282346638529993179660072199368212480.000,
> Mean=42029510023671999325552505715632898048.000,
> StdDev=111961692493879995008202507419807383552.000
>   NoData Value=3.40282346638529011e+38
>   Metadata:
>     STATISTICS_MAXIMUM=3.4028234663853e+38
>     STATISTICS_MEAN=4.2029510023672e+37
>     STATISTICS_MINIMUM=-2.2330000400543
>     STATISTICS_STDDEV=1.1196169249388e+38
>
> Obviously, cells with nodata value are included in the stats, although
> rounded. This seems wrong to me and I know gdal propagates the values with
> for instance gdal_translate.

could you share the file ? (possibly in private if needed)

--
Spatialys - Geospatial professional services
http://www.spatialys.com
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: GDAL 2.3.1 on linux disregarding nodata values

Even Rouault-2
In reply to this post by TomvanTilburg
On jeudi 5 juillet 2018 14:22:16 CEST Tom van Tilburg wrote:

> I have a fresh gdal 2.3.1 build on a ubuntu 17.10 that tells me:
> > gdalinfo dem.tif -stats -mm
>
> [..snip...]
> Band 1 Block=2200x1 Type=Float32, ColorInterp=Gray
>   Min=-2.233 Max=340282346638529993179660072199368212480.000   Computed
> Min/Max=-2.233,340282346638528859811704183484516925440.000
>   Minimum=-2.233, Maximum=340282346638529993179660072199368212480.000,
> Mean=42029510023671999325552505715632898048.000,
> StdDev=111961692493879995008202507419807383552.000
>   NoData Value=3.40282346638529011e+38
>   Metadata:
>     STATISTICS_MAXIMUM=3.4028234663853e+38
>     STATISTICS_MEAN=4.2029510023672e+37
>     STATISTICS_MINIMUM=-2.2330000400543
>     STATISTICS_STDDEV=1.1196169249388e+38

ok, the issue is that nodata value (stored as text in GeoTIFF) is slightly
above the maximum value of a float32. Presumably due to rounding issues when
formatting it. I've pushed a fix to detect that situation and clamp it to the
max value of float32. A bit strange that the issue wasn't found on Windows
though.

--
Spatialys - Geospatial professional services
http://www.spatialys.com
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: GDAL 2.3.1 on linux disregarding nodata values

TomvanTilburg

I just did a fresh build of trunk (2.4.0) but noticed 2 issues:
1. The installed version in /usr/local/bin is still version 2.3.1:
> ls /usr/local/bin/gdalinfo -lrth
-rwxr-xr-x 1 root root 40K Jul  6 13:51 /usr/local/bin/gdalinfo
> /usr/local/bin/gdalinfo --version
GDAL 2.3.1, released 2018/06/22

(I did an ldconfig to be sure, but no to avail)

2. The newly build version in the git repo segfaults on:
/var/data/git_repos/gdal/gdal/apps/gdalinfo dem.tif
[..snip..]
Origin = (171950.000000000000000,437050.000000000000000)
Pixel Size = (0.500000000000000,-0.500000000000000)
Metadata:
  AREA_OR_POINT=Area
Image Structure Metadata:
  INTERLEAVE=BAND
Corner Coordinates:
ERROR 1: illegal axis orientation combination
Upper Left  (  171950.000,  437050.000)
ERROR 1: illegal axis orientation combination
Lower Left  (  171950.000,  435950.000)
ERROR 1: illegal axis orientation combination
Upper Right (  173050.000,  437050.000)
ERROR 1: illegal axis orientation combination
Lower Right (  173050.000,  435950.000)
ERROR 1: illegal axis orientation combination
Center      (  172500.000,  436500.000)
Segmentation fault (core dumped)


Since there is a second version of proj on the system, I have used ./configure --with-static-proj4

Thanks for dealing with this. 

Best,
 Tom

On Fri, Jul 6, 2018 at 12:14 PM, Even Rouault <[hidden email]> wrote:
On jeudi 5 juillet 2018 14:22:16 CEST Tom van Tilburg wrote:
> I have a fresh gdal 2.3.1 build on a ubuntu 17.10 that tells me:
> > gdalinfo dem.tif -stats -mm
>
> [..snip...]
> Band 1 Block=2200x1 Type=Float32, ColorInterp=Gray
>   Min=-2.233 Max=340282346638529993179660072199368212480.000   Computed
> Min/Max=-2.233,340282346638528859811704183484516925440.000
>   Minimum=-2.233, Maximum=340282346638529993179660072199368212480.000,
> Mean=42029510023671999325552505715632898048.000,
> StdDev=111961692493879995008202507419807383552.000
>   NoData Value=3.40282346638529011e+38
>   Metadata:
>     STATISTICS_MAXIMUM=3.4028234663853e+38
>     STATISTICS_MEAN=4.2029510023672e+37
>     STATISTICS_MINIMUM=-2.2330000400543
>     STATISTICS_STDDEV=1.1196169249388e+38

ok, the issue is that nodata value (stored as text in GeoTIFF) is slightly
above the maximum value of a float32. Presumably due to rounding issues when
formatting it. I've pushed a fix to detect that situation and clamp it to the
max value of float32. A bit strange that the issue wasn't found on Windows
though.


 
--
Spatialys - Geospatial professional services
http://www.spatialys.com


_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: GDAL 2.3.1 on linux disregarding nodata values

Even Rouault-2
On vendredi 6 juillet 2018 14:00:16 CEST Tom van Tilburg wrote:
> I just did a fresh build of trunk (2.4.0) but noticed 2 issues:
>
> 1. The installed version in /usr/local/bin is still version 2.3.1:

You may have to defined LD_LIBRARY_PATH since it probably links against your
system libgdal

Otherwise you don't need to install the master, but just make and source the
scripts/setdevenv.sh that will set PATH, LD_LIBRARY_PATH, GDAL_DATA and
PYTHONPATH to appropriate values in the built tree.

--
Spatialys - Geospatial professional services
http://www.spatialys.com
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: GDAL 2.3.1 on linux disregarding nodata values

TomvanTilburg
Got myself a build that has the correct proj version now (./configure --with-proj) so it doesn't segfault.
I can confirm that the nodata values are correctly dealt with now :)

Still battling the linked gdal version though, will post when I dealt with that.

Tom



On Fri, Jul 6, 2018 at 2:09 PM, Even Rouault <[hidden email]> wrote:
On vendredi 6 juillet 2018 14:00:16 CEST Tom van Tilburg wrote:
> I just did a fresh build of trunk (2.4.0) but noticed 2 issues:
>
> 1. The installed version in /usr/local/bin is still version 2.3.1:

You may have to defined LD_LIBRARY_PATH since it probably links against your
system libgdal

Otherwise you don't need to install the master, but just make and source the
scripts/setdevenv.sh that will set PATH, LD_LIBRARY_PATH, GDAL_DATA and
PYTHONPATH to appropriate values in the built tree.

--
Spatialys - Geospatial professional services
http://www.spatialys.com


_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: GDAL 2.3.1 on linux disregarding nodata values

Craig Bruce
In reply to this post by Even Rouault-2
On 07/06/2018 07:14 AM, Even Rouault wrote:
  NoData Value=3.40282346638529011e+38
  Metadata:
    STATISTICS_MAXIMUM=3.4028234663853e+38
ok, the issue is that nodata value (stored as text in GeoTIFF) is slightly 
above the maximum value of a float32. Presumably due to rounding issues when 
formatting it. I've pushed a fix to detect that situation and clamp it to the 
max value of float32. A bit strange that the issue wasn't found on Windows 
though.
A wider issue is that if the NoData value is not a fully-representable integer, you can't expect a direct comparison to work every time.  You also can't expect the text-encoded NoData value to match the binary value exactly, since people do silly things like truncate the precision or generate 'float' values using 'double' computations.  The following C program:

#include <stdio.h>
#include <stdlib.h>
#include <math.h>
#include <float.h>
int main() {
    float f = (float) atof("3.40282346638529011e+38");
    double d = 3.40282346638529011e+38;
    long double e = 3.40282346638529011e+38L;
    printf( "float:   %.18g  (%.18g, eq=%d)\n", f, (float) FLT_MAX,
             f == FLT_MAX );
    printf( "double:  %.18g (%.18g)\n", d, (double) (float) FLT_MAX );
    printf( "ldouble: %.19Lg (%.19Lg)\n", e, powl(2, 128) - 1 );
    return( 0 );
}

produces the output on Linux/GCC of:

float:   3.4028234663852886e+38  (3.4028234663852886e+38, eq=1)
double:  3.40282346638529011e+38 (3.4028234663852886e+38)
ldouble: 3.40282346638529011e+38 (3.402823669209384635e+38)


Which means the given NoData value doesn't actually overflow the 'float' representation and I get an equality when executing the comparison using 'float's.  The given value here is presumably meant to be 2^128-1, though it doesn't match this at any precision.  The text is representable as a 'double' but not as 'float', which indicates some kind of tortured origin story.

I don't know how GDAL does the comparison, but it needs to be done carefully.  In general, 'float's only keep 6 digits of accuracy when converting from decimal → binary → decimal and need 9 decimal digits to represent every conversion of binary → decimal → binary.  I do comparisons in my own code using 'double's after running the NoData value through 'float' and back again in this case and then use a comparison tolerance of ±(NoData × 1e-7) for 'float' samples (maybe it should be 1e-6 to handle more arbitrary precision truncations).

With the available information, if GDAL did the comparison using 'float's, the (float) STATISTICS_MAXIMUM == FLT_MAX == (float) 3.40282346638529011e+38 and the NoData value would be detected properly.  If, OTOH, GDAL is representing NoData as 'double' and comparing this directly to 'float' samples, then the values wouldn't match, because the NoData text value does not match (double) FLT_MAX.

--
Dr. Craig S. Bruce
Senior Software Developer
CubeWerx Inc.
http://www.cubewerx.com

_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: GDAL 2.3.1 on linux disregarding nodata values

TomvanTilburg
In reply to this post by Even Rouault-2
A follow up on the previous problem with versions (see below):

I removed all old libgdal versions that were in /usr/local/lib and that has solved the issue.

Best,
Tom


------------------------------
I thought LD_LIBRARY_PATH could be untouched since I'm doing a system-install according to: https://trac.osgeo.org/gdal/wiki/BuildingOnUnix
Anyway, even when setting LD_LIBRARY_PATH to /usr/local/lib and rebuilding, It shows the same issue.

It seems the linking is correct according to:

> ldd /usr/local/bin/gdalinfo |grep libgdal
        libgdal.so.20 => /usr/local/lib/libgdal.so.20 (0x00007fb64a7d4000)

> ls -lrt /usr/local/lib/libgdal*
-rwxr-xr-x 1 root root  95610360 Nov 10  2016 /usr/local/lib/libgdal.so.20.1.0
-rwxr-xr-x 1 root root  96245776 Jan  5  2017 /usr/local/lib/libgdal.so.20.1.2
-rwxr-xr-x 1 root root  95857208 Jan  5  2017 /usr/local/lib/libgdal.so.20.1.1
-rwxr-xr-x 1 root root  96494352 Mar 13  2017 /usr/local/lib/libgdal.so.20.1.3
-rwxr-xr-x 1 root root 120481920 May 16  2017 /usr/local/lib/libgdal.so.20.2.0
-rwxr-xr-x 1 root root 135179608 Aug 17  2017 /usr/local/lib/libgdal.so.20.3.0
-rwxr-xr-x 1 root root 136095424 Jul  5 13:58 /usr/local/lib/libgdal.so.20.3.2
-rwxr-xr-x 1 root root 156131272 Jul  5 14:01 /usr/local/lib/libgdal.so.20.4.1
-rwxr-xr-x 1 root root 160334560 Jul  6 14:00 /usr/local/lib/libgdal.so.20.4.0
lrwxrwxrwx 1 root root        17 Jul  6 14:00 /usr/local/lib/libgdal.so -> libgdal.so.20.4.0
-rwxr-xr-x 1 root root      1525 Jul  6 14:00 /usr/local/lib/libgdal.la
-rw-r--r-- 1 root root 406912300 Jul  6 14:00 /usr/local/lib/libgdal.a
lrwxrwxrwx 1 root root        17 Jul  6 14:00 /usr/local/lib/libgdal.so.20 -> libgdal.so.20.4.1

> /usr/local/bin/gdalinfo --version
GDAL 2.3.1, released 2018/06/22

> /usr/local/bin/gdal-config --version
2.4.0


I'm a bit lost, this version 2.3.1 was the previous custom build I did and it all resides in /usr/local
The only caveat I have is that there is an older libgdal20 2.2.1 package installed on the same system, but that doesn't seem to interfere.

Any hints on where to dig deeper?

Best,
 Tom



On Fri, Jul 6, 2018 at 2:44 PM, Tom van Tilburg <[hidden email]> wrote:
I thought LD_LIBRARY_PATH could be untouched since I'm doing a system-install according to: https://trac.osgeo.org/gdal/wiki/BuildingOnUnix
Anyway, even when setting LD_LIBRARY_PATH to /usr/local/lib and rebuilding, It shows the same issue.

It seems the linking is correct according to:

> ldd /usr/local/bin/gdalinfo |grep libgdal
        libgdal.so.20 => /usr/local/lib/libgdal.so.20 (0x00007fb64a7d4000)

> ls -lrt /usr/local/lib/libgdal*
-rwxr-xr-x 1 root root  95610360 Nov 10  2016 /usr/local/lib/libgdal.so.20.1.0
-rwxr-xr-x 1 root root  96245776 Jan  5  2017 /usr/local/lib/libgdal.so.20.1.2
-rwxr-xr-x 1 root root  95857208 Jan  5  2017 /usr/local/lib/libgdal.so.20.1.1
-rwxr-xr-x 1 root root  96494352 Mar 13  2017 /usr/local/lib/libgdal.so.20.1.3
-rwxr-xr-x 1 root root 120481920 May 16  2017 /usr/local/lib/libgdal.so.20.2.0
-rwxr-xr-x 1 root root 135179608 Aug 17  2017 /usr/local/lib/libgdal.so.20.3.0
-rwxr-xr-x 1 root root 136095424 Jul  5 13:58 /usr/local/lib/libgdal.so.20.3.2
-rwxr-xr-x 1 root root 156131272 Jul  5 14:01 /usr/local/lib/libgdal.so.20.4.1
-rwxr-xr-x 1 root root 160334560 Jul  6 14:00 /usr/local/lib/libgdal.so.20.4.0
lrwxrwxrwx 1 root root        17 Jul  6 14:00 /usr/local/lib/libgdal.so -> libgdal.so.20.4.0
-rwxr-xr-x 1 root root      1525 Jul  6 14:00 /usr/local/lib/libgdal.la
-rw-r--r-- 1 root root 406912300 Jul  6 14:00 /usr/local/lib/libgdal.a
lrwxrwxrwx 1 root root        17 Jul  6 14:00 /usr/local/lib/libgdal.so.20 -> libgdal.so.20.4.1

> /usr/local/bin/gdalinfo --version
GDAL 2.3.1, released 2018/06/22

> /usr/local/bin/gdal-config --version
2.4.0


I'm a bit lost, this version 2.3.1 was the previous custom build I did and it all resides in /usr/local
The only caveat I have is that there is an older libgdal20 2.2.1 package installed on the same system, but that doesn't seem to interfere.

Any hints on where to dig deeper?

Best,
 Tom

On Fri, Jul 6, 2018 at 2:09 PM, Even Rouault <[hidden email]> wrote:
On vendredi 6 juillet 2018 14:00:16 CEST Tom van Tilburg wrote:
> I just did a fresh build of trunk (2.4.0) but noticed 2 issues:
>
> 1. The installed version in /usr/local/bin is still version 2.3.1:

You may have to defined LD_LIBRARY_PATH since it probably links against your
system libgdal

Otherwise you don't need to install the master, but just make and source the
scripts/setdevenv.sh that will set PATH, LD_LIBRARY_PATH, GDAL_DATA and
PYTHONPATH to appropriate values in the built tree.

--
Spatialys - Geospatial professional services
http://www.spatialys.com



_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev