[gdal-dev] Don't we have any ideas for GSoC 2017?

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

[gdal-dev] Don't we have any ideas for GSoC 2017?

jratike80
Hi,

OSGeo is accepted as an organization for the Google Summer of Code 2017 but there are no GDAL related ideas on https://wiki.osgeo.org/wiki/Google_Summer_of_Code_2017_Ideas. Could we raise some of the old ideas or discover some brand new ones?

-Jukka Rahkonen-
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Don't we have any ideas for GSoC 2017?

Casper Børgesen
Hi,

I don't know if the idea is too simple, but implementing indexing on VRT files would be great to have. I thinks it's currently a feature request.


Regards, Casper


-----Original Message-----
From: gdal-dev [mailto:[hidden email]] On Behalf Of Rahkonen Jukka (MML)
Sent: 28. februar 2017 09:07
To: '[hidden email]' ([hidden email])
Subject: [gdal-dev] Don't we have any ideas for GSoC 2017?

Hi,

OSGeo is accepted as an organization for the Google Summer of Code 2017 but there are no GDAL related ideas on https://wiki.osgeo.org/wiki/Google_Summer_of_Code_2017_Ideas. Could we raise some of the old ideas or discover some brand new ones?

-Jukka Rahkonen-
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Don't we have any ideas for GSoC 2017?

Mateusz Loskot
On 28 February 2017 at 11:16, Casper Børgesen (CABO) <[hidden email]> wrote:
> Hi,
>
> I don't know if the idea is too simple, but implementing indexing on VRT files would be great to have. I thinks it's currently a feature request.

Experiences show, that it's better to choose a simple idea feasible
for a student to complete,
than choose a fancy challenge that doesn't move even half way through.

The indexing sounds like a good idea.

Best regards,
--
Mateusz Loskot, http://mateusz.loskot.net
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Don't we have any ideas for GSoC 2017?

Hare, Trent
I would be a fan of indexing a VRT also.

There have been some good apps written in AMES Stereo Pipeline which could be "ported" to GDAL trunk. They work within their application so perhaps they are not needed within GDAL...?

1.) dem_merge (which works for images too). It has a very nice exponential blend edge option and works for non-square edges. This is a large program though.
2.) the also have a C++ version of Frank's hsv_merge.py which is much faster. This is much simpler. 

I would love to see any methods for tonal matching across images. Simple matching histograms and then running a script through a gdal_translate with a defined "stretch"..? Or here is a cool method to tonal match using another layer as the driver. There might not be code is not available (but it might be explained enough?). Also the author could be contacted. http://www.sciencedirect.com/science/article/pii/S0032063315300477

just a couple more ideas,
Trent




On Tue, Feb 28, 2017 at 3:26 AM, Mateusz Loskot <[hidden email]> wrote:
On 28 February 2017 at 11:16, Casper Børgesen (CABO) <[hidden email]> wrote:
> Hi,
>
> I don't know if the idea is too simple, but implementing indexing on VRT files would be great to have. I thinks it's currently a feature request.

Experiences show, that it's better to choose a simple idea feasible
for a student to complete,
than choose a fancy challenge that doesn't move even half way through.

The indexing sounds like a good idea.

Best regards,
--
Mateusz Loskot, http://mateusz.loskot.net
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev


_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

[gdal-dev] FW: Don't we have any ideas for GSoC 2017?

jratike80
In reply to this post by jratike80

Forwarding to the list because Jordan’s trial  was not successful.

Jordan Bess wrote:

Re: [gdal-dev] Don't we have any ideas for GSoC 2017?

 

Vector tiles! Mbtiles or folder of pbf

It would be great if OGR supported creation of vector tiles in protocol buff format

Get Outlook for Android

 


From: gdal-dev <[hidden email]> on behalf of Casper Børgesen (CABO) <[hidden email]>
Sent: Tuesday, February 28, 2017 5:16:19 AM
To: Rahkonen Jukka (MML); '[hidden email]' (
[hidden email])
Subject: Re: [gdal-dev] Don't we have any ideas for GSoC 2017?

 

Hi,

I don't know if the idea is too simple, but implementing indexing on VRT files would be great to have. I thinks it's currently a feature request.


Regards, Casper


-----Original Message-----
From: gdal-dev [
[hidden email]] On Behalf Of Rahkonen Jukka (MML)
Sent: 28. februar 2017 09:07
To: '[hidden email]' (
[hidden email])
Subject: [gdal-dev] Don't we have any ideas for GSoC 2017?

Hi,

OSGeo is accepted as an organization for the Google Summer of Code 2017 but there are no GDAL related ideas on https://wiki.osgeo.org/wiki/Google_Summer_of_Code_2017_Ideas. Could we raise some of the old ideas or discover some brand new ones?

-Jukka Rahkonen-
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev


_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: FW: Don't we have any ideas for GSoC 2017?

Jan Hartmann
Hi all,

What would you devs think of a project to add to the interpolation algorithms in gdal_grid, e.g. TPS. It's something I do now with R, but it would nice to have this native in Gdal. If you think this is a good idea, I could try to set up something from the University of Amsterdam ,together with Edzer Pebesma, of the University of Muenster, the leading specialist of spatial programming in R.

Regards,

Jan Hartmann

http://www.uni-muenster.de/Geoinformatics/en/institute/staff/index.php/119/Edzer_Pebesma
http://www.uva.nl/over-de-uva/organisatie/medewerkers/content/h/a/j.l.h.hartmann/j.l.h.hartmann.html



On 3/1/2017 12:43 PM, Rahkonen Jukka (MML) wrote:

Forwarding to the list because Jordan’s trial  was not successful.

Jordan Bess wrote:

Re: [gdal-dev] Don't we have any ideas for GSoC 2017?

 

Vector tiles! Mbtiles or folder of pbf

It would be great if OGR supported creation of vector tiles in protocol buff format

Get Outlook for Android

 


From: gdal-dev <[hidden email]> on behalf of Casper Børgesen (CABO) <[hidden email]>
Sent: Tuesday, February 28, 2017 5:16:19 AM
To: Rahkonen Jukka (MML); '[hidden email]' (
[hidden email])
Subject: Re: [gdal-dev] Don't we have any ideas for GSoC 2017?

 

Hi,

I don't know if the idea is too simple, but implementing indexing on VRT files would be great to have. I thinks it's currently a feature request.


Regards, Casper


-----Original Message-----
From: gdal-dev [
[hidden email]] On Behalf Of Rahkonen Jukka (MML)
Sent: 28. februar 2017 09:07
To: '[hidden email]' (
[hidden email])
Subject: [gdal-dev] Don't we have any ideas for GSoC 2017?

Hi,

OSGeo is accepted as an organization for the Google Summer of Code 2017 but there are no GDAL related ideas on https://wiki.osgeo.org/wiki/Google_Summer_of_Code_2017_Ideas. Could we raise some of the old ideas or discover some brand new ones?

-Jukka Rahkonen-
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev



_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev


_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: FW: Don't we have any ideas for GSoC 2017?

Even Rouault-2
In reply to this post by jratike80

On mercredi 1 mars 2017 11:43:12 CET Rahkonen Jukka (MML) wrote:

> Forwarding to the list because Jordan's trial was not successful.

 

Sounds a good idea. I think potential students should be more pro-active in proposing and developing their own ideas.

 

Regarding "indexing on VRT files", I guess this is about spatial indexing of sources in a VRT file. My bet would be that the size of VRT files that would need such indexing would be so large than their parsing would be the most limiting factor. But I can be wrong.

Some people who want to make a mosaic from a huge number of tiles do that by making VRTs of VRTs of VRTs with a pyramid structure to avoid parsing a lot of XML and still having good performance.

Another option that has been mentionned in the past would be to have a variation of the VRT syntax that would reference a tile index (shapefile or any OGR datasource) and would thus benefit from its spatial indexing & filtering capabilities

 

Something like:

 

<VRTDataset rasterXSize="20000" rasterYSize="20000">

<SRS>EPSG:26711</SRS>

<GeoTransform> 4.4072000000000000e+05, 6.0000000000000000e+01, 0.0000000000000000e+00, 3.7513200000000000e+06, 0.0000000000000000e+00, -6.0000000000000000e+01</GeoTransform>

<TileIndex>

<Filename relativeToVRT="1">index.shp</Filename>

<!-- <Layername>index</Layername> -->

</Tileindex>

<Overview rasterXSize="10000" rasterYSize="10000">

<TileIndex>

<Filename relativeToVRT="1">index_ovr_2.shp</Filename>

</Tileindex>

</Overview>

<Overview rasterXSize="5000" rasterYSize="5000">

<TileIndex>

<Filename relativeToVRT="1">index_ovr_4.shp</Filename>

</Tileindex>

</Overview>

<VRTRasterBand dataType="Byte" band="1">

<ColorInterp>Red</ColorInterp>

</VRTRasterBand>

<VRTRasterBand dataType="Byte" band="2">

<ColorInterp>Green</ColorInterp>

</VRTRasterBand>

<VRTRasterBand dataType="Byte" band="3">

<ColorInterp>Blue</ColorInterp>

</VRTRasterBand>

</VRTDataset>

 

Even

 

> Jordan Bess wrote:

> Re: [gdal-dev] Don't we have any ideas for GSoC 2017?

>

>

> Vector tiles! Mbtiles or folder of pbf

>

> It would be great if OGR supported creation of vector tiles in protocol buff

> format

>

> Get Outlook for Android<https://aka.ms/ghei36>

>

> ________________________________

> From: gdal-dev

> <[hidden email]<mailto:[hidden email]>>

> on behalf of Casper Børgesen (CABO) <[hidden email]<mailto:[hidden email]>>

> Sent: Tuesday, February 28, 2017 5:16:19 AM

> To: Rahkonen Jukka (MML); '[hidden email]'

> ([hidden email]<mailto:[hidden email]>) Subject: Re:

> [gdal-dev] Don't we have any ideas for GSoC 2017?

>

> Hi,

>

> I don't know if the idea is too simple, but implementing indexing on VRT

> files would be great to have. I thinks it's currently a feature request.

>

>

> Regards, Casper

>

>

> -----Original Message-----

> From: gdal-dev [mailto:[hidden email]] On Behalf Of

> Rahkonen Jukka (MML) Sent: 28. februar 2017 09:07

> To: '[hidden email]'

> ([hidden email]<mailto:[hidden email]>) Subject:

> [gdal-dev] Don't we have any ideas for GSoC 2017?

>

> Hi,

>

> OSGeo is accepted as an organization for the Google Summer of Code 2017 but

> there are no GDAL related ideas on

> https://wiki.osgeo.org/wiki/Google_Summer_of_Code_2017_Ideas. Could we

> raise some of the old ideas or discover some brand new ones?

>

> -Jukka Rahkonen-

> _______________________________________________

> gdal-dev mailing list

> [hidden email]<mailto:[hidden email]>

> https://lists.osgeo.org/mailman/listinfo/gdal-dev

> _______________________________________________

> gdal-dev mailing list

> [hidden email]<mailto:[hidden email]>

> https://lists.osgeo.org/mailman/listinfo/gdal-dev

 

 

--

Spatialys - Geospatial professional services

http://www.spatialys.com


_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: FW: Don't we have any ideas for GSoC 2017?

Casper Børgesen

Yes this is about spatial indexing J

 

Without looking at the actual VRT code I would guess that parsing the VRT file is one thing, but having a VRT file with +100,000 tiles would require GDAL to look through each tile every time a pixel is requested?

Even, if your idea below could be implemented into the VRT driver to be able to build the index file and gdaladdo to build the overview index files, it would be a huge step forward.

 

Is it possible to build the index on the fly into memory when opening the VRT file? Is it too memory or time consuming?

 

/Casper

 

From: gdal-dev [mailto:[hidden email]] On Behalf Of Even Rouault
Sent: 1. marts 2017 13:25
To: [hidden email]
Cc: Rahkonen Jukka (MML)
Subject: Re: [gdal-dev] FW: Don't we have any ideas for GSoC 2017?

 

On mercredi 1 mars 2017 11:43:12 CET Rahkonen Jukka (MML) wrote:

> Forwarding to the list because Jordan's trial was not successful.

 

Sounds a good idea. I think potential students should be more pro-active in proposing and developing their own ideas.

 

Regarding "indexing on VRT files", I guess this is about spatial indexing of sources in a VRT file. My bet would be that the size of VRT files that would need such indexing would be so large than their parsing would be the most limiting factor. But I can be wrong.

Some people who want to make a mosaic from a huge number of tiles do that by making VRTs of VRTs of VRTs with a pyramid structure to avoid parsing a lot of XML and still having good performance.

Another option that has been mentionned in the past would be to have a variation of the VRT syntax that would reference a tile index (shapefile or any OGR datasource) and would thus benefit from its spatial indexing & filtering capabilities

 

Something like:

 

<VRTDataset rasterXSize="20000" rasterYSize="20000">

<SRS>EPSG:26711</SRS>

<GeoTransform> 4.4072000000000000e+05, 6.0000000000000000e+01, 0.0000000000000000e+00, 3.7513200000000000e+06, 0.0000000000000000e+00, -6.0000000000000000e+01</GeoTransform>

<TileIndex>

<Filename relativeToVRT="1">index.shp</Filename>

<!-- <Layername>index</Layername> -->

</Tileindex>

<Overview rasterXSize="10000" rasterYSize="10000">

<TileIndex>

<Filename relativeToVRT="1">index_ovr_2.shp</Filename>

</Tileindex>

</Overview>

<Overview rasterXSize="5000" rasterYSize="5000">

<TileIndex>

<Filename relativeToVRT="1">index_ovr_4.shp</Filename>

</Tileindex>

</Overview>

<VRTRasterBand dataType="Byte" band="1">

<ColorInterp>Red</ColorInterp>

</VRTRasterBand>

<VRTRasterBand dataType="Byte" band="2">

<ColorInterp>Green</ColorInterp>

</VRTRasterBand>

<VRTRasterBand dataType="Byte" band="3">

<ColorInterp>Blue</ColorInterp>

</VRTRasterBand>

</VRTDataset>

 

Even


_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: FW: Don't we have any ideas for GSoC 2017?

Even Rouault-2

On mercredi 1 mars 2017 12:44:29 CET Casper Børgesen (CABO) wrote:

> Yes this is about spatial indexing :)

>

> Without looking at the actual VRT code I would guess that parsing the VRT

> file is one thing, but having a VRT file with +100,000 tiles

 

Let's say at least 100 bytes for each source declaration, 100 000 tiles would be at least 10 MB for the VRT. Big, but perhaps not that much, if it is left open in a quite long section

 

> would require

> GDAL to look through each tile every time a pixel is requested?

 

Yes, if you do single pixel queries, that might have bad performance. If you do more significant extractions, the cost might decrease.

 

> Even, if

> your idea below could be implemented into the VRT driver to be able to

> build the index file and gdaladdo to build the overview index files, it

> would be a huge step forward.

 

Building the index file is the job of gdaltindex. Regarding the overviews, you could have several variants: either you have a set of overview tiles, or you can also build the overview from the full resolution .vrt

 

>

> Is it possible to build the index on the fly into memory when opening the

> VRT file? Is it too memory or time consuming?

 

If one keeps the current syntax with many XML sources, yes on-the-fly indexing would be the only reasonable solution. I don't anticipate it would be particularly memory or time consuming for a "reasonable" number of tiles (let's say less than 10 millions). There's already infrastructure for that in GDAL, port/cpl_quad_tree.h. Used by the OpenFileGDB driver for example since it cannot currently use the native spatia index.

 

>

> /Casper

>

> From: gdal-dev [mailto:[hidden email]] On Behalf Of Even

> Rouault Sent: 1. marts 2017 13:25

> To: [hidden email]

> Cc: Rahkonen Jukka (MML)

> Subject: Re: [gdal-dev] FW: Don't we have any ideas for GSoC 2017?

>

> On mercredi 1 mars 2017 11:43:12 CET Rahkonen Jukka (MML) wrote:

> > Forwarding to the list because Jordan's trial was not successful.

>

> Sounds a good idea. I think potential students should be more pro-active in

> proposing and developing their own ideas.

>

>

>

> Regarding "indexing on VRT files", I guess this is about spatial indexing of

> sources in a VRT file. My bet would be that the size of VRT files that

> would need such indexing would be so large than their parsing would be the

> most limiting factor. But I can be wrong.

>

> Some people who want to make a mosaic from a huge number of tiles do that by

> making VRTs of VRTs of VRTs with a pyramid structure to avoid parsing a lot

> of XML and still having good performance.

>

> Another option that has been mentionned in the past would be to have a

> variation of the VRT syntax that would reference a tile index (shapefile or

> any OGR datasource) and would thus benefit from its spatial indexing &

> filtering capabilities

>

>

>

> Something like:

>

>

>

> <VRTDataset rasterXSize="20000" rasterYSize="20000">

>

> <SRS>EPSG:26711</SRS>

>

> <GeoTransform> 4.4072000000000000e+05, 6.0000000000000000e+01,

> 0.0000000000000000e+00, 3.7513200000000000e+06, 0.0000000000000000e+00,

> -6.0000000000000000e+01</GeoTransform>

>

> <TileIndex>

>

> <Filename relativeToVRT="1">index.shp</Filename>

>

> <!-- <Layername>index</Layername> -->

>

> </Tileindex>

>

> <Overview rasterXSize="10000" rasterYSize="10000">

>

> <TileIndex>

>

> <Filename relativeToVRT="1">index_ovr_2.shp</Filename>

>

> </Tileindex>

>

> </Overview>

>

> <Overview rasterXSize="5000" rasterYSize="5000">

>

> <TileIndex>

>

> <Filename relativeToVRT="1">index_ovr_4.shp</Filename>

>

> </Tileindex>

>

> </Overview>

>

> <VRTRasterBand dataType="Byte" band="1">

>

> <ColorInterp>Red</ColorInterp>

>

> </VRTRasterBand>

>

> <VRTRasterBand dataType="Byte" band="2">

>

> <ColorInterp>Green</ColorInterp>

>

> </VRTRasterBand>

>

> <VRTRasterBand dataType="Byte" band="3">

>

> <ColorInterp>Blue</ColorInterp>

>

> </VRTRasterBand>

>

> </VRTDataset>

>

>

>

> Even

 

 

--

Spatialys - Geospatial professional services

http://www.spatialys.com


_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev