[gdal-dev] Motion: adopt RFC 75: Multidimensional array

classic Classic list List threaded Threaded
18 messages Options
Reply | Threaded
Open this post in threaded view
|

[gdal-dev] Motion: adopt RFC 75: Multidimensional array

Even Rouault-2
Hi,

I motion to adopt RFC 75: Multidimensional array

        https://github.com/OSGeo/gdal/pull/1580

-----------

Starting with my +1

Even

--
Spatialys - Geospatial professional services
http://www.spatialys.com
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Motion: adopt RFC 75: Multidimensional array

Norman Barker-2
Even,

I am in favour of the work on multi-dimensional arrays. There are two open PRs https://github.com/rouault/gdal/pulls with changes to the text. Though I and my employer would like TileDB included in the list of formats (though happy to wait until I have added the code to support this) the issues around dimension ordering and defining NULL are two things I would like to discuss.

I am +1 for the PR

Norman

On Thu, Jul 18, 2019 at 4:37 AM Even Rouault <[hidden email]> wrote:
Hi,

I motion to adopt RFC 75: Multidimensional array

        https://github.com/OSGeo/gdal/pull/1580

-----------

Starting with my +1

Even

--
Spatialys - Geospatial professional services
http://www.spatialys.com
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev

_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Motion: adopt RFC 75: Multidimensional array

Even Rouault-2
Hi,

>
> I am in favour of the work on multi-dimensional arrays. There are two open
> PRs https://github.com/rouault/gdal/pulls with changes to the text.

Oh I didn't see them before. I've merged the first one from Edzer. Answering
yours now:

> Though
> I and my employer would like TileDB included in the list of formats (though
> happy to wait until I have added the code to support this)

I'd prefer to limit the scope of the RFC to the drivers I'll take care of.
Other contributors are welcome to implement it for their own drivers of
interest once this has landed into master.

> the issues
> around dimension ordering

https://github.com/rouault/gdal/blob/rfc75/gdal/doc/source/user/
multidim_raster_data_model.rst mentions

"""Most drivers use the row-major convention for dimensions: that is, when
considering that the array elemnts are stored consecutively in memory, the
first dimension is the slowest varying one (in a 2D image, the row), and the
last dimension the fastest varying one (in a 2D image, the colum). That
convention is the default convention used for NumPy arrays, the MEM driver and
the HDF5 and netCDF APIs. The GDAL API is mostly agnostic about that
convention, except when passing a NULL array as the stride parameter for the
:cpp:func:`GDALAbstractMDArray::Read` and
:cpp:func:`GDALAbstractMDArray::Write` methods. You can refer to NumPy
documentation about multidimensional array indeing order issues:
https://docs.scipy.org/doc/numpy/reference/internals.html#multidimensional-array-indexing-order-issues """

Does that answer your concern / question ?

> and defining NULL are two things I would like to
> discuss.

Yes, nodata support is there. Mentionned in
https://github.com/rouault/gdal/blob/rfc75/gdal/doc/source/user/
multidim_raster_data_model.rst#multidimensional-array

and in the C++ API:
https://github.com/rouault/gdal/blob/rfc75/gdal/gcore/gdal_priv.h#L2257

so 2 virtual methods that can be implemented by drivers:
const void* GetRawNoDataValue() const;
bool SetRawNoDataValue(const void* pRawNoData);

and 2 helpers that get/set as double for the common cases.

Even

--
Spatialys - Geospatial professional services
http://www.spatialys.com
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Motion: adopt RFC 75: Multidimensional array

Norman Barker-2
Hi Even,

thanks for the clarification around indexing and nodata. This works great and when I have availability and your changes last in master I will upgrade the tiledb driver.

+1 from me.

Norman

On Thu, Jul 18, 2019 at 10:48 AM Even Rouault <[hidden email]> wrote:
Hi,

>
> I am in favour of the work on multi-dimensional arrays. There are two open
> PRs https://github.com/rouault/gdal/pulls with changes to the text.

Oh I didn't see them before. I've merged the first one from Edzer. Answering
yours now:

> Though
> I and my employer would like TileDB included in the list of formats (though
> happy to wait until I have added the code to support this)

I'd prefer to limit the scope of the RFC to the drivers I'll take care of.
Other contributors are welcome to implement it for their own drivers of
interest once this has landed into master.

> the issues
> around dimension ordering

https://github.com/rouault/gdal/blob/rfc75/gdal/doc/source/user/
multidim_raster_data_model.rst mentions

"""Most drivers use the row-major convention for dimensions: that is, when
considering that the array elemnts are stored consecutively in memory, the
first dimension is the slowest varying one (in a 2D image, the row), and the
last dimension the fastest varying one (in a 2D image, the colum). That
convention is the default convention used for NumPy arrays, the MEM driver and
the HDF5 and netCDF APIs. The GDAL API is mostly agnostic about that
convention, except when passing a NULL array as the stride parameter for the
:cpp:func:`GDALAbstractMDArray::Read` and
:cpp:func:`GDALAbstractMDArray::Write` methods. You can refer to NumPy
documentation about multidimensional array indeing order issues:
https://docs.scipy.org/doc/numpy/reference/internals.html#multidimensional-array-indexing-order-issues """

Does that answer your concern / question ?

> and defining NULL are two things I would like to
> discuss.

Yes, nodata support is there. Mentionned in
https://github.com/rouault/gdal/blob/rfc75/gdal/doc/source/user/
multidim_raster_data_model.rst#multidimensional-array

and in the C++ API:
https://github.com/rouault/gdal/blob/rfc75/gdal/gcore/gdal_priv.h#L2257

so 2 virtual methods that can be implemented by drivers:
const void* GetRawNoDataValue() const;
bool SetRawNoDataValue(const void* pRawNoData);

and 2 helpers that get/set as double for the common cases.

Even

--
Spatialys - Geospatial professional services
http://www.spatialys.com

_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Motion: adopt RFC 75: Multidimensional array

Howard Butler-3
+1 Howard

On Jul 18, 2019, at 11:34 AM, Norman Barker <[hidden email]> wrote:

Hi Even,

thanks for the clarification around indexing and nodata. This works great and when I have availability and your changes last in master I will upgrade the tiledb driver.

+1 from me.

Norman

On Thu, Jul 18, 2019 at 10:48 AM Even Rouault <[hidden email]> wrote:
Hi,

>
> I am in favour of the work on multi-dimensional arrays. There are two open
> PRs https://github.com/rouault/gdal/pulls with changes to the text.

Oh I didn't see them before. I've merged the first one from Edzer. Answering
yours now:

> Though
> I and my employer would like TileDB included in the list of formats (though
> happy to wait until I have added the code to support this)

I'd prefer to limit the scope of the RFC to the drivers I'll take care of.
Other contributors are welcome to implement it for their own drivers of
interest once this has landed into master.

> the issues
> around dimension ordering

https://github.com/rouault/gdal/blob/rfc75/gdal/doc/source/user/
multidim_raster_data_model.rst mentions

"""Most drivers use the row-major convention for dimensions: that is, when
considering that the array elemnts are stored consecutively in memory, the
first dimension is the slowest varying one (in a 2D image, the row), and the
last dimension the fastest varying one (in a 2D image, the colum). That
convention is the default convention used for NumPy arrays, the MEM driver and
the HDF5 and netCDF APIs. The GDAL API is mostly agnostic about that
convention, except when passing a NULL array as the stride parameter for the
:cpp:func:`GDALAbstractMDArray::Read` and
:cpp:func:`GDALAbstractMDArray::Write` methods. You can refer to NumPy
documentation about multidimensional array indeing order issues:
https://docs.scipy.org/doc/numpy/reference/internals.html#multidimensional-array-indexing-order-issues """

Does that answer your concern / question ?

> and defining NULL are two things I would like to
> discuss.

Yes, nodata support is there. Mentionned in
https://github.com/rouault/gdal/blob/rfc75/gdal/doc/source/user/
multidim_raster_data_model.rst#multidimensional-array

and in the C++ API:
https://github.com/rouault/gdal/blob/rfc75/gdal/gcore/gdal_priv.h#L2257

so 2 virtual methods that can be implemented by drivers:
const void* GetRawNoDataValue() const;
bool SetRawNoDataValue(const void* pRawNoData);

and 2 helpers that get/set as double for the common cases.

Even

--
Spatialys - Geospatial professional services
http://www.spatialys.com
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev


_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Motion: adopt RFC 75: Multidimensional array

Even Rouault-2
In reply to this post by Even Rouault-2
On jeudi 18 juillet 2019 11:37:08 CEST Even Rouault wrote:
> Hi,
>
> I motion to adopt RFC 75: Multidimensional array
>
>         https://github.com/OSGeo/gdal/pull/1580
>
> -----------

I declare this motion passed with +1 from PSC members NormanB, HowardB and
myself.

Even

--
Spatialys - Geospatial professional services
http://www.spatialys.com
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Motion: adopt RFC 75: Multidimensional array

Joe Lee
I'm glad that this motion has passed!

I got one question after reviewing Even's latest code and RFC.

Does it consider a case that subsets 1d dataset from 2d dataset [1]?
The organization of MOP02J (1-D SWATH) and its shape is like below:

dset[2][n_points]
lat[n_points]
lon[n_points]

I wish I could be able to extract 1D from 2D using dset[n_points][0] and 
make a GeoTIFF grid from the scattered dset/lat/lon using a tool like [2] or 
gdalwarp with 3 VRTs that correspond to 1D lat/lon/dset.

To help you understand the case better, 
the equivalent Python code is available in [3].

If this workflow can be already done, 
can anyone give me a series of commands to be executed?

Best regards,


On Mon, Jul 22, 2019 at 10:10 AM Even Rouault <[hidden email]> wrote:
On jeudi 18 juillet 2019 11:37:08 CEST Even Rouault wrote:
> Hi,
>
> I motion to adopt RFC 75: Multidimensional array
>
>         https://github.com/OSGeo/gdal/pull/1580
>
> -----------

I declare this motion passed with +1 from PSC members NormanB, HowardB and
myself.

Even

--
Spatialys - Geospatial professional services
http://www.spatialys.com
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev

_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Motion: adopt RFC 75: Multidimensional array

Even Rouault-2
On jeudi 25 juillet 2019 19:12:02 CEST Joe Lee wrote:

> I'm glad that this motion has passed!
>
> I got one question after reviewing Even's latest code and RFC.
>
> Does it consider a case that subsets 1d dataset from 2d dataset [1]?
> The organization of MOP02J (1-D SWATH) and its shape is like below:
>
> dset[2][n_points]
> lat[n_points]
> lon[n_points]
>
> I wish I could be able to extract 1D from 2D using dset[n_points][0] and
> make a GeoTIFF grid from the scattered dset/lat/lon using a tool like [2]
> or
 gdalwarp with 3 VRTs that correspond to 1D lat/lon/dset.
>
> To help you understand the case better,
> the equivalent Python code is available in [3].
>
> If this workflow can be already done,
> can anyone give me a series of commands to be executed?

As far as I understand your need, this could probably be done with a gdalmdimtranslate invokation
(or possibly several ones and assemble manually a VRT)

https://gdal.org/programs/gdalmdimtranslate.html#gdalmdimtranslate

to extract 1D from 2D, you would do something like

gdalmdimtranslate in out -array name=dset,view=[:,0]

The view argument can be used to use numpy-style indexing & slicing:
https://gdal.org/api/gdalmdarray_cpp.html#_CPPv4NK11GDALMDArray7GetViewERKNSt6stringE 

Even

--
Spatialys - Geospatial professional services
http://www.spatialys.com
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Motion: adopt RFC 75: Multidimensional array

Joe Lee
Thanks, Even for the quick answer!

I tried it but I get "ERROR 6":

[nene:gdal/gdal/apps] hyoklee%  ./gdalmdimtranslate -array name='/HDFEOS/SWATHS/MOP02/Data Fields/RetrievedSurfaceTemperature',view=\[:,0\]  ~/NASAHDF/MOP02J-20131129-L2V17.8.3.he5  test.tif
ERROR 6: Unsupported number of dimensions


On 7/25/19, 2:39 PM, "Even Rouault" <[hidden email]> wrote:

    On jeudi 25 juillet 2019 19:12:02 CEST Joe Lee wrote:
    > I'm glad that this motion has passed!
    >
    > I got one question after reviewing Even's latest code and RFC.
    >
    > Does it consider a case that subsets 1d dataset from 2d dataset [1]?
    > The organization of MOP02J (1-D SWATH) and its shape is like below:
    >
    > dset[2][n_points]
    > lat[n_points]
    > lon[n_points]
    >
    > I wish I could be able to extract 1D from 2D using dset[n_points][0] and
    > make a GeoTIFF grid from the scattered dset/lat/lon using a tool like [2]
    > or
     gdalwarp with 3 VRTs that correspond to 1D lat/lon/dset.
    >
    > To help you understand the case better,
    > the equivalent Python code is available in [3].
    >
    > If this workflow can be already done,
    > can anyone give me a series of commands to be executed?
   
    As far as I understand your need, this could probably be done with a gdalmdimtranslate invokation
    (or possibly several ones and assemble manually a VRT)
   
    https://gdal.org/programs/gdalmdimtranslate.html#gdalmdimtranslate
   
    to extract 1D from 2D, you would do something like
   
    gdalmdimtranslate in out -array name=dset,view=[:,0]
   
    The view argument can be used to use numpy-style indexing & slicing:
    https://gdal.org/api/gdalmdarray_cpp.html#_CPPv4NK11GDALMDArray7GetViewERKNSt6stringE 
   
    Even
   
    --
    Spatialys - Geospatial professional services
    http://www.spatialys.com
   

_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Motion: adopt RFC 75: Multidimensional array

Even Rouault-2
On jeudi 25 juillet 2019 19:57:44 CEST Joe Lee wrote:
> Unsupported number of dimensions

Ah ok, when exporting to GeoTIFF or a "classic" 2D driver, the
multidimensional-to-classic bridge currently checks that the array is a 2D
one. Here you reduced it to a 1D-slice, hence it rejects it. The bridge could
probably be made a bit more flexible by accepting 1D arrays as well. If you
export to netCDF, this should work fine though (as this won't go through the
bridge, but use directly multidimensional netCDF output capabilities)

Even

--
Spatialys - Geospatial professional services
http://www.spatialys.com
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Motion: adopt RFC 75: Multidimensional array

Joe Lee

Hi, Even!

  1-D dataset handling in the new HDF5 multi-dimension code is fantastic!
  I could convert 1-d swath into GeoTIFF using the new Python HDF5 multidimensional API.
  Please see python code at the bottom of email.

  However, I can't run it as an AWS lambda function.
  The below is the error message:


'NoneType' object has no attribute 'GetRootGroup': AttributeError
Traceback (most recent call last):
File "/var/task/lambda.py", line 42, in handler
rg = ds.GetRootGroup()
AttributeError: 'NoneType' object has no attribute 'GetRootGroup'


  The same code worked fine under AWS lambda Docker before it is deployed.
  I even tested the Python code on Google Colab and could access data on S3 using vsis3.
  Thus, I am quite puzzled. 
   Has anyone successfully tested with HDF5 multidimensional with AWS Lambda?


    # Open dataset using vsis3 interface. fname is obtained from S3 bucket trigger.
    path = '/vsis3/sdt-data/' + fname
    ds = gdal.OpenEx(path, gdal.OF_MULTIDIM_RASTER)
    rg = ds.GetRootGroup()
    hg = rg.OpenGroup('HDFEOS')
    sg = hg.OpenGroup('SWATHS')
    mg = sg.OpenGroup('MOP02')
    gg = mg.OpenGroup('Geolocation Fields')
    dg = mg.OpenGroup('Data Fields')
    
    ar_lat = gg.OpenMDArray('Latitude')
    lat_data = ar_lat.Read(buffer_datatype = gdal.ExtendedDataType.Create(gdal.GDT_Float32))
    lats = np.frombuffer(lat_data, dtype=np.float32)
        ...

On Fri, Jul 26, 2019 at 9:34 AM Even Rouault <[hidden email]> wrote:
On jeudi 25 juillet 2019 19:57:44 CEST Joe Lee wrote:
> Unsupported number of dimensions

Ah ok, when exporting to GeoTIFF or a "classic" 2D driver, the
multidimensional-to-classic bridge currently checks that the array is a 2D
one. Here you reduced it to a 1D-slice, hence it rejects it. The bridge could
probably be made a bit more flexible by accepting 1D arrays as well. If you
export to netCDF, this should work fine though (as this won't go through the
bridge, but use directly multidimensional netCDF output capabilities)

Even

--
Spatialys - Geospatial professional services
http://www.spatialys.com

_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Motion: adopt RFC 75: Multidimensional array

Joe Lee

I found the error. It was my token key in gdal.SetConfigOption(). 
GDAL Multidimensional support is quite amazing and works well with AWS Lambda & HDF5 data on S3! 

I can't wait to see HDF4 multidimensional support. 

Thank you, GDAL development team for the new driver!


On Tue, Jul 30, 2019 at 12:17 PM Joe Lee <[hidden email]> wrote:

Hi, Even!

  1-D dataset handling in the new HDF5 multi-dimension code is fantastic!
  I could convert 1-d swath into GeoTIFF using the new Python HDF5 multidimensional API.
  Please see python code at the bottom of email.

  However, I can't run it as an AWS lambda function.
  The below is the error message:


'NoneType' object has no attribute 'GetRootGroup': AttributeError
Traceback (most recent call last):
File "/var/task/lambda.py", line 42, in handler
rg = ds.GetRootGroup()
AttributeError: 'NoneType' object has no attribute 'GetRootGroup'


  The same code worked fine under AWS lambda Docker before it is deployed.
  I even tested the Python code on Google Colab and could access data on S3 using vsis3.
  Thus, I am quite puzzled. 
   Has anyone successfully tested with HDF5 multidimensional with AWS Lambda?


    # Open dataset using vsis3 interface. fname is obtained from S3 bucket trigger.
    path = '/vsis3/sdt-data/' + fname
    ds = gdal.OpenEx(path, gdal.OF_MULTIDIM_RASTER)
    rg = ds.GetRootGroup()
    hg = rg.OpenGroup('HDFEOS')
    sg = hg.OpenGroup('SWATHS')
    mg = sg.OpenGroup('MOP02')
    gg = mg.OpenGroup('Geolocation Fields')
    dg = mg.OpenGroup('Data Fields')
    
    ar_lat = gg.OpenMDArray('Latitude')
    lat_data = ar_lat.Read(buffer_datatype = gdal.ExtendedDataType.Create(gdal.GDT_Float32))
    lats = np.frombuffer(lat_data, dtype=np.float32)
        ...

On Fri, Jul 26, 2019 at 9:34 AM Even Rouault <[hidden email]> wrote:
On jeudi 25 juillet 2019 19:57:44 CEST Joe Lee wrote:
> Unsupported number of dimensions

Ah ok, when exporting to GeoTIFF or a "classic" 2D driver, the
multidimensional-to-classic bridge currently checks that the array is a 2D
one. Here you reduced it to a 1D-slice, hence it rejects it. The bridge could
probably be made a bit more flexible by accepting 1D arrays as well. If you
export to netCDF, this should work fine though (as this won't go through the
bridge, but use directly multidimensional netCDF output capabilities)

Even

--
Spatialys - Geospatial professional services
http://www.spatialys.com
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev

_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Motion: adopt RFC 75: Multidimensional array

Even Rouault-2
On mardi 30 juillet 2019 18:10:15 CEST Joe Lee wrote:
> I found the error. It was my token key in gdal.SetConfigOption().
> GDAL Multidimensional support is quite amazing and works well with AWS
> Lambda & HDF5 data on S3!
 
> I can't wait to see HDF4 multidimensional support.

This has just landed in master (as well as exporting 1D arrays to GeoTIFF)

Even

--
Spatialys - Geospatial professional services
http://www.spatialys.com
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Motion: adopt RFC 75: Multidimensional array

Joe Lee

Hi, Even!

  I tested it and your new code works very well. 
  Thank you for adding this new feature so quickly!

  Now I wish /vsis3 work on HDF4 like HDF5. 
  That would be perfect because I tried /vsicurl as you suggested in [1] but it did not work.





On Tue, Aug 6, 2019 at 12:14 PM Even Rouault <[hidden email]> wrote:
On mardi 30 juillet 2019 18:10:15 CEST Joe Lee wrote:
> I found the error. It was my token key in gdal.SetConfigOption().
> GDAL Multidimensional support is quite amazing and works well with AWS
> Lambda & HDF5 data on S3!

> I can't wait to see HDF4 multidimensional support.

This has just landed in master (as well as exporting 1D arrays to GeoTIFF)

Even

--
Spatialys - Geospatial professional services
http://www.spatialys.com

_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Motion: adopt RFC 75: Multidimensional array

Even Rouault-2
On mercredi 7 août 2019 20:36:48 CEST Joe Lee wrote:
> Hi, Even!
>
>   I tested it and your new code works very well.
>   Thank you for adding this new feature so quickly!
>
>   Now I wish /vsis3 work on HDF4 like HDF5.
>   That would be perfect because I tried /vsicurl as you suggested in [1] but
> it did not work.

Unfortunately libhdf has no pluggable I/O layer, so only the vsipreload trick can be used.
Using a solution like https://github.com/s3fs-fuse/s3fs-fuse should also work.

I just successully tried on Ubunty 16.04 the following:

g++ -std=c++11 -Wall -fPIC port/vsipreload.cpp -shared -o vsipreload.so -Iport -L. -L.libs -lgdal
LD_PRELOAD=vsipreload.so gdalmdiminfo /vsicurl/https://download.osgeo.org/gdal/data/hdf4/REANALYSIS_1999217.hdf
LD_PRELOAD=vsipreload.so gdalmdimtranslate /vsicurl/https://download.osgeo.org/gdal/data/hdf4/REANALYSIS_1999217.hdf out.nc
LD_PRELOAD=vsipreload.so gdalmdiminfo /vsis3/{my_bucket}/REANALYSIS_1999217.hdf

Even

--
Spatialys - Geospatial professional services
http://www.spatialys.com
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Motion: adopt RFC 75: Multidimensional array

Even Rouault-2
On mercredi 7 août 2019 20:58:54 CEST Joe Lee wrote:
> Hi, Even!
>
>   Is there a configure flag for enabling it by default without relying on
> LD_PRELOAD=...? (e.g., --with-vsipreload)

No,
- this needs to be loaded with LD_PRELOAD because it overrides very common
symbols of the libc (fopen, etc..) and thus it needs to be loaded in a special
way so that the symbols of vsipreload.so are actually used and not the libc
ones.
- it is better to keep it in a separate library, because I can easily imagine
that having fopen() and the like re-defined in libgdal.so could cause serious
mess.

So all in all, this is a convenient trick, but this remains an experimental &
somewhat dangerous hack to be used in last resort.

Even

--
Spatialys - Geospatial professional services
http://www.spatialys.com
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Motion: adopt RFC 75: Multidimensional array

Even Rouault-2
In reply to this post by Even Rouault-2
Hi,

The last piece of the work planned per RFC 75 has now been pushed to master
with multidimensionnal support in the GRIB driver (time-based grouping for
GRIB messages of the same variable concatenated in a same file).

Even

--
Spatialys - Geospatial professional services
http://www.spatialys.com
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Motion: adopt RFC 75: Multidimensional array

James Klassen-2
In reply to this post by Even Rouault-2
There could be a vsipreload script that sets the environment variable
and then execs its parameters similar to how eatmydata [1] works.

[1] https://manpages.debian.org/testing/eatmydata/eatmydata.1.en.html

On 8/7/19 4:10 PM, Even Rouault wrote:

> On mercredi 7 août 2019 20:58:54 CEST Joe Lee wrote:
>> Hi, Even!
>>
>>    Is there a configure flag for enabling it by default without relying on
>> LD_PRELOAD=...? (e.g., --with-vsipreload)
> No,
> - this needs to be loaded with LD_PRELOAD because it overrides very common
> symbols of the libc (fopen, etc..) and thus it needs to be loaded in a special
> way so that the symbols of vsipreload.so are actually used and not the libc
> ones.
> - it is better to keep it in a separate library, because I can easily imagine
> that having fopen() and the like re-defined in libgdal.so could cause serious
> mess.
>
> So all in all, this is a convenient trick, but this remains an experimental &
> somewhat dangerous hack to be used in last resort.
>
> Even
>
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev