[gdal-dev] Gdal and Open Data Cube

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

[gdal-dev] Gdal and Open Data Cube

Julien Michel-2
Dear all,

There is a lot of activity around this Open Data Cube [1] initiative,
which seems promising. As far as I understand it however, what we can do
with data in the datacube is limited to what the python API allows you
to do. Is there any plan (and does it even make sense ?) to have a gdal
driver that allows to address data ingested in a datacube instance ?

Thanks a lot,

Regards,

Julien

[1] https://github.com/opendatacube/datacube-core

--
Julien MICHEL
CNES - DSO/SI/2A

_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Gdal and Open Data Cube

Even Rouault-2

On vendredi 2 juin 2017 14:28:18 CEST Julien Michel wrote:

> Dear all,

>

> There is a lot of activity around this Open Data Cube [1] initiative,

> which seems promising. As far as I understand it however, what we can do

> with data in the datacube is limited to what the python API allows you

> to do. Is there any plan (and does it even make sense ?) to have a gdal

> driver that allows to address data ingested in a datacube instance ?

 

I've just discovered this project, and browsed quickly through the docs at

http://datacube-core.readthedocs.io/en/latest/, so the following might be not really relevant.

 

According to

http://datacube-core.readthedocs.io/en/latest/user/intro.html , it seems that ingested data is put in tiled netCDF files, and referenced in a PostgreSQL database (not clear if the files are blob in the DB or in the filesystem)

 

So if the data itself is stored in the filesystem, a driver could probably read the metadata in the DB and create some sort of virtual mosaic around the tiles. If their database schema is stable enough (it doesn't seem to be formally documented or I didn't find where), the driver could probably read it directly. Otherwise if the DB schema may change, then yes you'd need to use their Python API. Could be a potential use case for my experimental concept of GDAL drivers written in Python (*). What would be fun here is that given they use GDAL, their Python API probably uses GDAL under the hood, so you would have GDAL core -> OpenDataCube driver in Python -> OpenDataCube Python API -> GDAL SWIG Python bindings -> GDAL core -> netCDF driver ;-)

 

Even

 

(*) https://lists.osgeo.org/pipermail/gdal-dev/2017-April/046526.html

 

--

Spatialys - Geospatial professional services

http://www.spatialys.com


_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Gdal and Open Data Cube

Julien Michel-2
Le 02/06/2017 à 15:20, Even Rouault a écrit :

>
> On vendredi 2 juin 2017 14:28:18 CEST Julien Michel wrote:
>
> > Dear all,
>
> >
>
> > There is a lot of activity around this Open Data Cube [1] initiative,
>
> > which seems promising. As far as I understand it however, what we can do
>
> > with data in the datacube is limited to what the python API allows you
>
> > to do. Is there any plan (and does it even make sense ?) to have a gdal
>
> > driver that allows to address data ingested in a datacube instance ?
>
> I've just discovered this project, and browsed quickly through the docs at
>
> http://datacube-core.readthedocs.io/en/latest/, so the following might
> be not really relevant.
>
> According to
>
> http://datacube-core.readthedocs.io/en/latest/user/intro.html , it
> seems that ingested data is put in tiled netCDF files, and referenced
> in a PostgreSQL database (not clear if the files are blob in the DB or
> in the filesystem)
>
> So if the data itself is stored in the filesystem, a driver could
> probably read the metadata in the DB and create some sort of virtual
> mosaic around the tiles. If their database schema is stable enough (it
> doesn't seem to be formally documented or I didn't find where), the
> driver could probably read it directly. Otherwise if the DB schema may
> change, then yes you'd need to use their Python API. Could be a
> potential use case for my experimental concept of GDAL drivers written
> in Python (*). What would be fun here is that given they use GDAL,
> their Python API probably uses GDAL under the hood, so you would have
> GDAL core -> OpenDataCube driver in Python -> OpenDataCube Python API
> -> GDAL SWIG Python bindings -> GDAL core -> netCDF driver ;-)
>
Thanks a lot for your response. I am still trying to figure out if this
is of any interest or not (I mean the gdal driver for ODC). It seems to
me that if you spent all the time and storage to build such a datacube,
it would be better to still be able to read back data with standard
tools. Plus the datacube could offer a nice abstraction of image
collections. Maybe the simplest way is to ask them.
>
> (*) https://lists.osgeo.org/pipermail/gdal-dev/2017-April/046526.html
>
--
Julien MICHEL
CNES - DSO/SI/2A - BPI 1219
18, avenue Edouard Belin
31401 Toulouse Cedex 09 - France
Tel: +33 561 282 894 - Fax: +33 561 283 109

_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Gdal and Open Data Cube

Baumann, Peter
In reply to this post by Julien Michel-2
Hi Julien,

already seen rasdaman (www.rasdaman.org)? Fully implemented nD datacube engine,
proven on massive data, in operational use (not just planning phase)...and
proudly using GDAL.

As I am editor of the OGC datacube standards I'll be glad to give advice on
standards-based coupling. Just le me know your plans.

cheers,

Peter


On 06/02/2017 02:28 PM, Julien Michel wrote:

> Dear all,
>
> There is a lot of activity around this Open Data Cube [1] initiative, which
> seems promising. As far as I understand it however, what we can do with data
> in the datacube is limited to what the python API allows you to do. Is there
> any plan (and does it even make sense ?) to have a gdal driver that allows to
> address data ingested in a datacube instance ?
>
> Thanks a lot,
>
> Regards,
>
> Julien
>
> [1] https://github.com/opendatacube/datacube-core
>

--
Dr. Peter Baumann
 - Professor of Computer Science, Jacobs University Bremen
   www.faculty.jacobs-university.de/pbaumann
   mail: [hidden email]
   tel: +49-421-200-3178, fax: +49-421-200-493178
 - Executive Director, rasdaman GmbH Bremen (HRB 26793)
   www.rasdaman.com, mail: [hidden email]
   tel: 0800-rasdaman, fax: 0800-rasdafax, mobile: +49-173-5837882
"Si forte in alienas manus oberraverit hec peregrina epistola incertis ventis dimissa, sed Deo commendata, precamur ut ei reddatur cui soli destinata, nec preripiat quisquam non sibi parata." (mail disclaimer, AD 1083)


_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Gdal and Open Data Cube

Julien Michel-2
Hi Peter,

I know about rasdaman, I even attended the rasdaman tutorial at FOSS4GEU
2015. But my point is not to choose or compare implementations : I am
working with a project that uses the this ODC,  I want to be able to do
some OTB processing with it, and OTB reads data through gdal. And from a
gdal perspective, it makes sense to have drivers for every possible
(open-source) data source. Hence my question. I am a bit worried about
the overhead of what Even sketched ;)

Regards,

Julien


Le 04/06/2017 à 15:09, Peter Baumann a écrit :

> Hi Julien,
>
> already seen rasdaman (www.rasdaman.org)? Fully implemented nD datacube engine,
> proven on massive data, in operational use (not just planning phase)...and
> proudly using GDAL.
>
> As I am editor of the OGC datacube standards I'll be glad to give advice on
> standards-based coupling. Just le me know your plans.
>
> cheers,
>
> Peter
>
>
> On 06/02/2017 02:28 PM, Julien Michel wrote:
>> Dear all,
>>
>> There is a lot of activity around this Open Data Cube [1] initiative, which
>> seems promising. As far as I understand it however, what we can do with data
>> in the datacube is limited to what the python API allows you to do. Is there
>> any plan (and does it even make sense ?) to have a gdal driver that allows to
>> address data ingested in a datacube instance ?
>>
>> Thanks a lot,
>>
>> Regards,
>>
>> Julien
>>
>> [1] https://github.com/opendatacube/datacube-core
>>


--
Julien MICHEL
CNES - DSO/SI/2A - BPI 1219
18, avenue Edouard Belin
31401 Toulouse Cedex 09 - France
Tel: +33 561 282 894 - Fax: +33 561 283 109

_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Gdal and Open Data Cube

Baumann, Peter
Hi Julien,

understood, a particular project will usually have particular constraints, no
problem. Actually, should you be willing to undertake this ODC endeavour maybe
you can post a summary of functionality and performance comparison here
afterwards? Might be interesting for some.

best,

Peter



On 06/12/2017 11:55 AM, Julien Michel wrote:

> Hi Peter,
>
> I know about rasdaman, I even attended the rasdaman tutorial at FOSS4GEU 2015.
> But my point is not to choose or compare implementations : I am working with a
> project that uses the this ODC,  I want to be able to do some OTB processing
> with it, and OTB reads data through gdal. And from a gdal perspective, it
> makes sense to have drivers for every possible (open-source) data source.
> Hence my question. I am a bit worried about the overhead of what Even sketched ;)
>
> Regards,
>
> Julien
>
>
> Le 04/06/2017 à 15:09, Peter Baumann a écrit :
>> Hi Julien,
>>
>> already seen rasdaman (www.rasdaman.org)? Fully implemented nD datacube engine,
>> proven on massive data, in operational use (not just planning phase)...and
>> proudly using GDAL.
>>
>> As I am editor of the OGC datacube standards I'll be glad to give advice on
>> standards-based coupling. Just le me know your plans.
>>
>> cheers,
>>
>> Peter
>>
>>
>> On 06/02/2017 02:28 PM, Julien Michel wrote:
>>> Dear all,
>>>
>>> There is a lot of activity around this Open Data Cube [1] initiative, which
>>> seems promising. As far as I understand it however, what we can do with data
>>> in the datacube is limited to what the python API allows you to do. Is there
>>> any plan (and does it even make sense ?) to have a gdal driver that allows to
>>> address data ingested in a datacube instance ?
>>>
>>> Thanks a lot,
>>>
>>> Regards,
>>>
>>> Julien
>>>
>>> [1] https://github.com/opendatacube/datacube-core
>>>
>
>

--
Dr. Peter Baumann
 - Professor of Computer Science, Jacobs University Bremen
   www.faculty.jacobs-university.de/pbaumann
   mail: [hidden email]
   tel: +49-421-200-3178, fax: +49-421-200-493178
 - Executive Director, rasdaman GmbH Bremen (HRB 26793)
   www.rasdaman.com, mail: [hidden email]
   tel: 0800-rasdaman, fax: 0800-rasdafax, mobile: +49-173-5837882
"Si forte in alienas manus oberraverit hec peregrina epistola incertis ventis dimissa, sed Deo commendata, precamur ut ei reddatur cui soli destinata, nec preripiat quisquam non sibi parata." (mail disclaimer, AD 1083)


_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev