[gdal-dev] Tiled GeoTiff that reduce the amount of reads in XYZ maps

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

[gdal-dev] Tiled GeoTiff that reduce the amount of reads in XYZ maps

Guy Doulberg
Hi all,

I am trying to use cloud optimized geotiff(cog) to reduce the block fetched when accessing a raster.

The use case I am trying to build, is a TileServer that serves tile in openlayer map.
The rasters are to be stored on a remote location, in my case azure blob storage.

We were able to do that, and you can see the code here:
https://github.com/satellogic/telluric/blob/master/telluric/georaster.py#L1497

The problem is, when running in verbose mode I can see that there are many requests to the blob storage,

I am responsible of creating the raster, is there a way in your opinion I can create the raster aligned to the XYZ tiling system so when I try to fetch a tile from the raster I will be able to do that with a single or close to that call?

In the current raster preparation code that I have, get_tile can reach even 10 requests.


Thanks







_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Tiled GeoTiff that reduce the amount of reads in XYZ maps

Even Rouault-2
On dimanche 17 juin 2018 10:19:34 CEST Guy Doulberg wrote:

> Hi all,
>
> I am trying to use cloud optimized geotiff(cog) to reduce the block fetched
> when accessing a raster.
>
> The use case I am trying to build, is a TileServer that serves tile in
> openlayer map.
> The rasters are to be stored on a remote location, in my case azure blob
> storage.
>
> We were able to do that, and you can see the code here:
> https://github.com/satellogic/telluric/blob/master/telluric/
> georaster.py#L1497
>
> The problem is, when running in verbose mode I can see that there are many
> requests to the blob storage,
>
> I am responsible of creating the raster, is there a way in your opinion I
> can create the raster aligned to the XYZ tiling system so when I try to
> fetch a tile from the raster I will be able to do that with a single or
> close to that call?

If your raster source is not already in WebMercator projection, you could use
gdalwarp with the appropriate -te and -tr to align on the boundaries of XYZ
tiling scheme at a given zoom level.

If your raster is already in WebMercator, gdal_translate -projwin -tr would do

If you don't want to compute the bounds, you could use gdal_translate to
MBTiles that will use gdalwarp internally to reproject and align on XYZ tiling
scheme automatically, followed by gdal_translate to TIFF with -oo
USE_BOUNDS=NO so that the extent used aligns on tile boundaries.

Even

--
Spatialys - Geospatial professional services
http://www.spatialys.com
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Tiled GeoTiff that reduce the amount of reads in XYZ maps

jratike80
In reply to this post by Guy Doulberg
Guy Doulberg wrote

> Hi all,
>
> I am trying to use cloud optimized geotiff(cog) to reduce the block
> fetched
> when accessing a raster.
>
> The use case I am trying to build, is a TileServer that serves tile in
> openlayer map.
> The rasters are to be stored on a remote location, in my case azure blob
> storage.
>
> We were able to do that, and you can see the code here:
> https://github.com/satellogic/telluric/blob/master/telluric/
> georaster.py#L1497
>
> The problem is, when running in verbose mode I can see that there are many
> requests to the blob storage,
>
> I am responsible of creating the raster, is there a way in your opinion I
> can create the raster aligned to the XYZ tiling system so when I try to
> fetch a tile from the raster I will be able to do that with a single or
> close to that call?
>
> In the current raster preparation code that I have, get_tile can reach
> even
> 10 requests.

Hi,

I wonder if something re-usable is implemented in the GeoTIFF cache type of
MapCache http://mapserver.org/mapcache/caches.html.

-Jukka Rahkonen-



--
Sent from: http://osgeo-org.1560.x6.nabble.com/GDAL-Dev-f3742093.html
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Tiled GeoTiff that reduce the amount of reads in XYZ maps

Guy Doulberg
In reply to this post by Even Rouault-2
Thanks Even

I tried both ways and in both I am still getting a lot of requests, I am exepcting only few, one for the headers and one for the tile:


The resolution I used is driven from https://wiki.openstreetmap.org/wiki/Zoom_levels

and the bounds I calculated using mercantile

so an example of what I ran is:

gdal_translate source.tif target.tif -co TILED=YES -co COPY_SRC_OVERVIEWS=YES -co COMPRESS=LZW -tr 0.596 0.596   -projwin 675091.8338146766 4706074.957461731 684875.7734351791 4696291.017841227

One more thing, the raster I got is 1/3 of the size of the original raster

On Sun, Jun 17, 2018 at 12:00 PM, Even Rouault <[hidden email]> wrote:
On dimanche 17 juin 2018 10:19:34 CEST Guy Doulberg wrote:
> Hi all,
>
> I am trying to use cloud optimized geotiff(cog) to reduce the block fetched
> when accessing a raster.
>
> The use case I am trying to build, is a TileServer that serves tile in
> openlayer map.
> The rasters are to be stored on a remote location, in my case azure blob
> storage.
>
> We were able to do that, and you can see the code here:
> https://github.com/satellogic/telluric/blob/master/telluric/
> georaster.py#L1497
>
> The problem is, when running in verbose mode I can see that there are many
> requests to the blob storage,
>
> I am responsible of creating the raster, is there a way in your opinion I
> can create the raster aligned to the XYZ tiling system so when I try to
> fetch a tile from the raster I will be able to do that with a single or
> close to that call?

If your raster source is not already in WebMercator projection, you could use
gdalwarp with the appropriate -te and -tr to align on the boundaries of XYZ
tiling scheme at a given zoom level.

If your raster is already in WebMercator, gdal_translate -projwin -tr would do

If you don't want to compute the bounds, you could use gdal_translate to
MBTiles that will use gdalwarp internally to reproject and align on XYZ tiling
scheme automatically, followed by gdal_translate to TIFF with -oo
USE_BOUNDS=NO so that the extent used aligns on tile boundaries.

Even

--
Spatialys - Geospatial professional services
http://www.spatialys.com


_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Tiled GeoTiff that reduce the amount of reads in XYZ maps

Guy Doulberg
it is 1/3 because the original was not compressed,

anyhow I am still getting too many requests when fetching a tile, maybe my read code is not doing what I think it is doing



On Sun, Jun 17, 2018 at 4:51 PM, Guy Doulberg <[hidden email]> wrote:
Thanks Even

I tried both ways and in both I am still getting a lot of requests, I am exepcting only few, one for the headers and one for the tile:


The resolution I used is driven from https://wiki.openstreetmap.org/wiki/Zoom_levels

and the bounds I calculated using mercantile

so an example of what I ran is:

gdal_translate source.tif target.tif -co TILED=YES -co COPY_SRC_OVERVIEWS=YES -co COMPRESS=LZW -tr 0.596 0.596   -projwin 675091.8338146766 4706074.957461731 684875.7734351791 4696291.017841227

One more thing, the raster I got is 1/3 of the size of the original raster

On Sun, Jun 17, 2018 at 12:00 PM, Even Rouault <[hidden email]> wrote:
On dimanche 17 juin 2018 10:19:34 CEST Guy Doulberg wrote:
> Hi all,
>
> I am trying to use cloud optimized geotiff(cog) to reduce the block fetched
> when accessing a raster.
>
> The use case I am trying to build, is a TileServer that serves tile in
> openlayer map.
> The rasters are to be stored on a remote location, in my case azure blob
> storage.
>
> We were able to do that, and you can see the code here:
> https://github.com/satellogic/telluric/blob/master/telluric/
> georaster.py#L1497
>
> The problem is, when running in verbose mode I can see that there are many
> requests to the blob storage,
>
> I am responsible of creating the raster, is there a way in your opinion I
> can create the raster aligned to the XYZ tiling system so when I try to
> fetch a tile from the raster I will be able to do that with a single or
> close to that call?

If your raster source is not already in WebMercator projection, you could use
gdalwarp with the appropriate -te and -tr to align on the boundaries of XYZ
tiling scheme at a given zoom level.

If your raster is already in WebMercator, gdal_translate -projwin -tr would do

If you don't want to compute the bounds, you could use gdal_translate to
MBTiles that will use gdalwarp internally to reproject and align on XYZ tiling
scheme automatically, followed by gdal_translate to TIFF with -oo
USE_BOUNDS=NO so that the extent used aligns on tile boundaries.

Even

--
Spatialys - Geospatial professional services
http://www.spatialys.com



_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Tiled GeoTiff that reduce the amount of reads in XYZ maps

Guy Doulberg
Even suggestions worked for me,

I am working on code that can ease the process of doing that


I will update when it is done 

After working with that I realized that now I have a new question. How deep(low) to go in alignments.

If I align the raster only in the zoom level the raster is at then I might fetch more then one tile of overviews image data when overviews are used, but to align the raster to zoom level that is very low doesn't make sense either since the headers will be huge.

How do you guys deal with this case?



On Sun, Jun 17, 2018 at 5:25 PM, Guy Doulberg <[hidden email]> wrote:
it is 1/3 because the original was not compressed,

anyhow I am still getting too many requests when fetching a tile, maybe my read code is not doing what I think it is doing



On Sun, Jun 17, 2018 at 4:51 PM, Guy Doulberg <[hidden email]> wrote:
Thanks Even

I tried both ways and in both I am still getting a lot of requests, I am exepcting only few, one for the headers and one for the tile:


The resolution I used is driven from https://wiki.openstreetmap.org/wiki/Zoom_levels

and the bounds I calculated using mercantile

so an example of what I ran is:

gdal_translate source.tif target.tif -co TILED=YES -co COPY_SRC_OVERVIEWS=YES -co COMPRESS=LZW -tr 0.596 0.596   -projwin 675091.8338146766 4706074.957461731 684875.7734351791 4696291.017841227

One more thing, the raster I got is 1/3 of the size of the original raster

On Sun, Jun 17, 2018 at 12:00 PM, Even Rouault <[hidden email]> wrote:
On dimanche 17 juin 2018 10:19:34 CEST Guy Doulberg wrote:
> Hi all,
>
> I am trying to use cloud optimized geotiff(cog) to reduce the block fetched
> when accessing a raster.
>
> The use case I am trying to build, is a TileServer that serves tile in
> openlayer map.
> The rasters are to be stored on a remote location, in my case azure blob
> storage.
>
> We were able to do that, and you can see the code here:
> https://github.com/satellogic/telluric/blob/master/telluric/
> georaster.py#L1497
>
> The problem is, when running in verbose mode I can see that there are many
> requests to the blob storage,
>
> I am responsible of creating the raster, is there a way in your opinion I
> can create the raster aligned to the XYZ tiling system so when I try to
> fetch a tile from the raster I will be able to do that with a single or
> close to that call?

If your raster source is not already in WebMercator projection, you could use
gdalwarp with the appropriate -te and -tr to align on the boundaries of XYZ
tiling scheme at a given zoom level.

If your raster is already in WebMercator, gdal_translate -projwin -tr would do

If you don't want to compute the bounds, you could use gdal_translate to
MBTiles that will use gdalwarp internally to reproject and align on XYZ tiling
scheme automatically, followed by gdal_translate to TIFF with -oo
USE_BOUNDS=NO so that the extent used aligns on tile boundaries.

Even

--
Spatialys - Geospatial professional services
http://www.spatialys.com




_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Tiled GeoTiff that reduce the amount of reads in XYZ maps

Even Rouault-2
On mardi 3 juillet 2018 09:16:34 CEST Guy Doulberg wrote:

> Even suggestions worked for me,
>
> I am working on code that can ease the process of doing that
>
> https://github.com/satellogic/telluric/blob/get_tile_aligned/telluric/georas
> ter.py#L1344
>
> I will update when it is done
>
> After working with that I realized that now I have a new question. How
> deep(low) to go in alignments.
>
> If I align the raster only in the zoom level the raster is at then I might
> fetch more then one tile of overviews image data when overviews are used,
> but to align the raster to zoom level that is very low doesn't make sense
> either since the headers will be huge.
>
> How do you guys deal with this case?

There's no good way of solving this with a single GeoTIFF File. You would need
each overview level to have possibly a different extent than the main overview
level. While you could put several images in the same GeoTIFF file with each
their GeoTIFF keys, that would no longer be overviews in the TIFF meaning (and
there is no support in GDAL for that).

--
Spatialys - Geospatial professional services
http://www.spatialys.com
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev