[gdal-dev] Infinite loop Netcdf/HDF5 -geoloc when warping (version 2.4.4)

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

[gdal-dev] Infinite loop Netcdf/HDF5 -geoloc when warping (version 2.4.4)

mvscheers

Dear list,

 

While in the process of upgrading from 2.2.4 to 2.4.4 we encountered an issue with a specific NETCDF file which can no longer be warped.
Sample files can be found at :
ftp://msgcpp-ogc-realtime.knmi.nl/

 

The file is a multiband dataset from which we want to extract the precip data, it contains 2 bands for georeferencing Lat/Lon.
So in 2.2.4 we used the -geoloc warp parameter to correctly warp it to a Webmercator or Mercator projection, this was done with the c-sharp swig bindings.

 

With 2.4.4 the program just halts and does not continue, the same can be reproduced with the commandline gdalwarp tool :
gdalwarp -geoloc -r cubic -t_srs "+proj=merc +a=6378137 +b=6378137 +lat_ts=0.0 +lon_0=0.0 +x_0=0.0 +y_0=0 +k=1.0 +units=m +nadgrids=@null +wktext  +no_defs" NETCDF:"samplefile ":precip "output.tiff"

 

When interrupting this process  HDF5 gives an error :

 

HDF5: infinite loop closing library

      D,T,FD,P,FD,P,FD,P,E,E,SL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL

Unsure if this is the actual problem or a result of the interrupt.

Best Regards,

Menno Van Scheers


_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Infinite loop Netcdf/HDF5 -geoloc when warping (version 2.4.4)

Even Rouault-2
Menno,

did you just update GDAL or GDAL+libnetcdf/libhdf5 ? I can't replicate the
error you see with GDAL 2.4 or master, and libhdf5 1.8.16 from Ubuntu 16.04

I've tried on ftp://msgcpp-ogc-realtime.knmi.nl/
SEVIR_OPER_R___MSGCPP__L2__20200305T000000_20200305T001500_0001.nc

By the way the georeferencing of that product is broken (but that's unrelated
to the issue you get):

                projection:semi_major_axis = 6356584.f ;
                projection:semi_minor_axis = 6378169.f ;

semi_minor_axis should be <= semi_major_axis, not greater

Even

--
Spatialys - Geospatial professional services
http://www.spatialys.com
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Infinite loop Netcdf/HDF5 -geoloc when warping (version 2.4.4)

mvscheers
Hi Even,

First thanks for the speedy  reply.

We have used a couple of different version and packages to test this (So a complete gdal update with the associated libhdf5 and netcdf).
In the project itself (.NET) we used 2 different version from the gdal package https://www.nuget.org/packages/GDAL/ (2.4.4 and 2.43)

For testing on commandline I downloaded the prebuild package from http://www.gisinternals.com/release.php

This was also reproduced on a completely clean machine with no prior GDAL or other dependencies installed.


Best Regards,

Menno Van Scheers


-----Oorspronkelijk bericht-----
Van: Even Rouault <[hidden email]>
Verzonden: donderdag 12 maart 2020 15:31
Aan: [hidden email]
CC: Menno van Scheers - HUSS <[hidden email]>; Mauricio Dominguez Diaz Del Castillo <[hidden email]>
Onderwerp: Re: [gdal-dev] Infinite loop Netcdf/HDF5 -geoloc when warping (version 2.4.4)

Menno,

did you just update GDAL or GDAL+libnetcdf/libhdf5 ? I can't replicate the error you see with GDAL 2.4 or master, and libhdf5 1.8.16 from Ubuntu 16.04

I've tried on ftp://msgcpp-ogc-realtime.knmi.nl/
SEVIR_OPER_R___MSGCPP__L2__20200305T000000_20200305T001500_0001.nc

By the way the georeferencing of that product is broken (but that's unrelated to the issue you get):

                projection:semi_major_axis = 6356584.f ;
                projection:semi_minor_axis = 6378169.f ;

semi_minor_axis should be <= semi_major_axis, not greater

Even

--
Spatialys - Geospatial professional services http://www.spatialys.com
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Infinite loop Netcdf/HDF5 -geoloc when warping (version 2.4.4)

jmckenna
Administrator
In reply to this post by mvscheers
Hi Menno,

I have also tested one of your files on Windows here, with MS4W 4.0.3
(GDAL 2.4.4, NetCDF 4.7.3, HDF5 1.10.5), and it also gives me errors:

gdalinfo
--------

gdalinfo --format netcdf

     GDAL_HAS_HDF4=YES
     GDAL_HAS_HDF5=YES
     NETCDF_CONVENTIONS=CF-1.5
     NETCDF_HAS_NC2=YES
     NETCDF_VERSION=4.7.3 of Jan 23 2020 15:16:48 $

gdalwarp
-------

gdalwarp -geoloc -r cubic -t_srs "+proj=merc +a=6378137 +b=6378137
+lat_ts=0.0 +lon_0=0.0 +x_0=0.0 +y_0=0 +k=1.0 +units=m +nadgrids=@null
+wktext  +no_defs"
NETCDF:"SEVIR_OPER_R___MSGCPP__L2__20200305T000000_20200305T001500_0001.nc":precip
"output.tiff"

output
------

Warning 1: NetCDF driver detected file type=5, but libnetcdf detected type=3
ERROR 5: OSRCalcInvFlattening(): Wrong input values
Warning 1: NetCDF driver detected file type=5, but libnetcdf detected type=3
ERROR 5: OSRCalcInvFlattening(): Wrong input values
Warning 1: NetCDF driver detected file type=5, but libnetcdf detected type=3
ERROR 5: OSRCalcInvFlattening(): Wrong input values
Creating output file that is 2790P x 4447L.
Processing
NETCDF:SEVIR_OPER_R___MSGCPP__L2__20200305T000000_20200305T001500_0001.nc:precip
[1/1] : 0Using internal nodata values (e.g. -1) for image
NETCDF:SEVIR_OPER_R___MSGCPP__L2__20200305T000000_20200305T001500_0001.nc:precip.
Copying nodata values from source
NETCDF:SEVIR_OPER_R___MSGCPP__L2__20200305T000000_20200305T001500_0001.nc:precip
to destination output.tiff.
...10...20...30...40...50...60...70...80...90...100 - done.


thoughts
--------

MS4W does include C Sharp GDAL bindings but I did not test them with
your data.


-jeff



--
Jeff McKenna
MapServer Consulting and Training Services
https://gatewaygeomatics.com/




On 2020-03-12 9:44 a.m., Menno van Scheers - HUSS wrote:

> Dear list,
>
> While in the process of upgrading from 2.2.4 to 2.4.4 we encountered an
> issue with a specific NETCDF file which can no longer be warped.
> Sample files can be found at : ftp://msgcpp-ogc-realtime.knmi.nl/
>
> The file is a multiband dataset from which we want to extract the
> *precip* data, it contains 2 bands for georeferencing Lat/Lon.
> So in 2.2.4 we used the -geoloc warp parameter to correctly warp it to a
> Webmercator or Mercator projection, this was done with the c-sharp swig
> bindings.
>
> With 2.4.4 the program just halts and does not continue, the same can be
> reproduced with the commandline gdalwarp tool :
> gdalwarp -geoloc -r cubic -t_srs "+proj=merc +a=6378137 +b=6378137
> +lat_ts=0.0 +lon_0=0.0 +x_0=0.0 +y_0=0 +k=1.0 +units=m +nadgrids=@null
> +wktext  +no_defs" NETCDF:"samplefile ":precip "output.tiff"
>
> When interrupting this process  HDF5 gives an error :
>
> HDF5: infinite loop closing library
>
>        
> D,T,FD,P,FD,P,FD,P,E,E,SL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL
>
> Unsure if this is the actual problem or a result of the interrupt.
>
> Best Regards,
>
> Menno Van Scheers
>
>





_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Infinite loop Netcdf/HDF5 -geoloc when warping (version 2.4.4)

Even Rouault-2
Jeff,

> I have also tested one of your files on Windows here, with MS4W 4.0.3
> (GDAL 2.4.4, NetCDF 4.7.3, HDF5 1.10.5), and it also gives me errors:

They look unrelated to Menno's error.

> Warning 1: NetCDF driver detected file type=5, but libnetcdf detected type=3

That one is likely related to MS4W nmake.opt lacking NETCDF_HAS_NC4 = yes

> ERROR 5: OSRCalcInvFlattening(): Wrong input values

Yep, that one is related to the oddity of the semi_minor_radius vs
semi_major_radius I mentionned before.

> MS4W does include C Sharp GDAL bindings but I did not test them with
> your data.

That's completely unrelated.

Even


--
Spatialys - Geospatial professional services
http://www.spatialys.com
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Infinite loop Netcdf/HDF5 -geoloc when warping (version 2.4.4)

jmckenna
Administrator
On 2020-03-12 3:16 p.m., Even Rouault wrote:

> Jeff,
>
>> I have also tested one of your files on Windows here, with MS4W 4.0.3
>> (GDAL 2.4.4, NetCDF 4.7.3, HDF5 1.10.5), and it also gives me errors:
>
> They look unrelated to Menno's error.
>
>> Warning 1: NetCDF driver detected file type=5, but libnetcdf detected type=3
>
> That one is likely related to MS4W nmake.opt lacking NETCDF_HAS_NC4 = yes
>
>

Thanks Even, confirmed, will fix this for next release.

-jeff





_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Infinite loop Netcdf/HDF5 -geoloc when warping (version 2.4.4)

mvscheers
Hi Even, Jeff

I checked both the 3.0.x and 2.4.4 release from  http://www.gisinternals.com/
They both report :

    GDAL_HAS_HDF4=YES
    GDAL_HAS_HDF5=YES
    NETCDF_CONVENTIONS=CF-1.5
    NETCDF_HAS_NC2=YES
    NETCDF_VERSION=4.3.2 of Sep  3 2017 18:13:37

Are the gdal nuget package based on these builds ?
And am I safe to assume the issue is due to this old NETCDF version

Best Regards,
Menno Van Scheers

-----Oorspronkelijk bericht-----
Van: gdal-dev <[hidden email]> Namens Jeff McKenna
Verzonden: donderdag 12 maart 2020 19:30
Aan: [hidden email]
Onderwerp: Re: [gdal-dev] Infinite loop Netcdf/HDF5 -geoloc when warping (version 2.4.4)

On 2020-03-12 3:16 p.m., Even Rouault wrote:

> Jeff,
>
>> I have also tested one of your files on Windows here, with MS4W 4.0.3
>> (GDAL 2.4.4, NetCDF 4.7.3, HDF5 1.10.5), and it also gives me errors:
>
> They look unrelated to Menno's error.
>
>> Warning 1: NetCDF driver detected file type=5, but libnetcdf detected
>> type=3
>
> That one is likely related to MS4W nmake.opt lacking NETCDF_HAS_NC4 =
> yes
>
>

Thanks Even, confirmed, will fix this for next release.

-jeff





_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev