[gdal-dev] Scale dependent VRT for overviews

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

[gdal-dev] Scale dependent VRT for overviews

Rahkonen Jukka (Tike)
Hi,

This is a long letter and finally it will lead to a suggestion about
implementing some kind of a Virtual Overview (VOV) system for organising a
number of GDAL datasets into a one top level dataset in such a way that GDAL can
automatically select the best source for the requested scale. Same thing is used
in Mapserver through a layer group and scale dependent layers.
---

I noticed today that I had same problem than Andrea Peri some time ago
http://article.gmane.org/gmane.comp.gis.gdal.devel/31177

I had built an biggish image mosaic as VRT. Size of the mosaic was 288000 by
480000 pixels and it was combined fron 400 paletted raster maps (8-bit with a
palette. Naturally I wanted to build some overviews for the VRT and naturally I
started to create them with gdaladdo.  After two hours it was still showing me
zero percant progress. I stopped the job, read Andrea's mail and concluded that
perhaps gdaladdo does not work very well with big VRT files.

I started to think about alternative methods. ECW and JPEG2000 would be pretty
good because they offer overviews for free. Unfortunately ECW itself is not free
and free GDAL JPEG2000 drivers are not fast enough. And even I can mysself write
JPEG2000 with a good licenced SDK our users do not have licensed readers and
slow JPEG2000 drivers are slow in reading too. Thus I will keep on waiting for a
JPEG2000 driver that is at the same time flaming fast AND open.

Then I had a try with dropping the resolution of VRT into half by using
gdalwarp. It managed to convert the VRT into GeoTIFF with double sized pixels in
three hours. Gdalwarp gives also an option to select the resampling method which
is nice sometimes. Gdaladdo has that option too but you may remember that the
program itself seems to be unusable for this purpose.

Three hours felt still rather slow. Then I measured how long it would take to
convert the individual images forming up the VRT with gdal_translate and
-outsize 50% 50%. It took only half on hour. Running gdalbuildvrt took a few
seconds more and I had the first reduced resolution level ready.

First run with gdal_translate gave reasonably good quality but the next reduced
level looked very bad. Several nearest neighbour samplings in a row is obviously
not so good method. I had now paletted raster maps and they do not tolerate
subsampling well with any method. I made some more tests and found that
converting images one be one with gdalwarp was a suitable compromise for me. It
was slower than gdal_translate but gave better looking result and converting all
individual files one by one was faster than converting the VRT as a whole.

After playing with a whole many variations with gdal_translate and gdalwarp,
including some convertions of subareas frpom the VRT for getting bigger tiles
and less files for low resolution scales I had a nice set in my hands. I had no
idea about how to ulilise them as a one GDAL data set but because I have some
experience on Mapserver I just gathered the images for each resolution level
with gdaltindex and then build a scale dependent WMS layer group and all the GIS
programs consuming WMS are happy. However it would be awfull to teach the
process for someone and even I can make a well working WMS service for our
customers I cannot deliver the data which is an odd mess now and only usable as
a one data source for Mapserver.

Now finally to my suggestion. To me GDAL virtual format is quite a lot similar
than Mapserver tileindex, though much more feature rich. But with Mapserver it
is very easy to combine layers into scale dependent layer groups be setting
suitable MINSCALEDENOM and MAXSCALEDENOM values for the individual layers.

Dear GDAL developers, I hope you can find some nice place inside the VRT XML
file for setting MINSCALEDEMON and MAXSCALEDENOM for "SimpleSource" or for some
new "SimpleSourceGroup" or whatever you feel suitable.

Regards,


-Jukka Rahkonen-




_______________________________________________
gdal-dev mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

[gdal-dev] Re: Scale dependent VRT for overviews

Rahkonen Jukka (Tike)
Hi,

I am not sure if it is already possible to do what I want with .VRT file but I
just do not know how. Therefore I prepared a test set if someone happens to be
interested in having a look.

Zip file is at http://latuviitta.org/documents/scale_dependent_gdal_meta_vrt.zip

The package contains:
- two images with 1 m pixel size (left_100.tif and right_100.tif)
- a .vrt file "100_percent.vrt" combining full resolution images

- two images with 2 m pixel size (left_50.tif and right_50.tif)
- a .vrt file "50_percent.vrt" combining half resolution images

- two images with 4 m pixel size (left_25.tif and right_25.tif)
- a .vrt file "25_percent.vrt" combining quarter resolution images

- overviews.vrt file which is created as
  gdalbuildvrt -resolution highest overviews.vrt *.vrt

I can open overview.vrt with Quantum GIS and it shows the rasters OK. Obviously
is is drawing all the layers from the VRT file on top of each other. The 50
percent layer is last in the overview.vrt file and it is drawn last by QGis and
it is the one Qgis user sees.

What I am dreaming of is having a control for setting reasonable scale ranges
for the layers included in overviews.vrt. I have been thinking that it could be
doen with something like "PixelSizeRange" directive. The following VRT would
lead to use
- 100_percent.vrt if output pixel size is less than 2 m
- 50_percent.vrt between 2 an 4 m output pixels
- 25_percent.vrt if output pixels are 4 m or bigger

<SimpleSource>
  <PixelSizeRange MaxXPixelSize="2" MaxYPixelSize="2" </PixelSizeRange>
      <SourceFilename relativeToVRT="1">100_percent.vrt</SourceFilename>
      <SourceBand>1</SourceBand>
      <SourceProperties RasterXSize="24000" RasterYSize="12000" DataType="Byte"
BlockXSize="128" BlockYSize="128" />
      <SrcRect xOff="0" yOff="0" xSize="24000" ySize="12000" />
      <DstRect xOff="0" yOff="0" xSize="24000" ySize="12000" />
    </SimpleSource>

    <SimpleSource>
<PixelSizeRange MinXPixelSize="2" MinYPixelSize="2" MaxXPixelSize="4"
MaxXPixelSize="4" </PixelSizeRange>
      <SourceFilename relativeToVRT="1">50_percent.vrt</SourceFilename>
      <SourceBand>1</SourceBand>
      <SourceProperties RasterXSize="12000" RasterYSize="6000" DataType="Byte"
BlockXSize="128" BlockYSize="128" />
      <SrcRect xOff="0" yOff="0" xSize="12000" ySize="6000" />
      <DstRect xOff="0" yOff="0" xSize="24000" ySize="12000" />
    </SimpleSource>

    <SimpleSource>
<PixelSizeRange MinXPixelSize="4" MinYPixelSize="4" </PixelSizeRange>
      <SourceFilename relativeToVRT="1">25_percent.vrt</SourceFilename>
      <SourceBand>1</SourceBand>
      <SourceProperties RasterXSize="6000" RasterYSize="3000" DataType="Byte"
BlockXSize="128" BlockYSize="128" />
      <SrcRect xOff="0" yOff="0" xSize="6000" ySize="3000" />
      <DstRect xOff="0" yOff="0" xSize="24000" ySize="12000" />
    </SimpleSource>

The result could be pretty convenient and flexible sometimes. For raster map
datasets each overview could contain maps which are rendered to suit best for
just that scale range. Dataset could contain a couple of lossless TIFF layers
first and one lossy JPEG2000 layer for taking care of all the rest resolution
levels. Resampling methods could be selected in the most flexible way, each
resolution layer could be updated individually and so on.

-Jukka Rahkonen-

_______________________________________________
gdal-dev mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Scale dependent VRT for overviews

Even Rouault
Jukka,

I think you could achieve what you want by adding a <Overview> element (see
http://www.gdal.org/gdal_vrttut.html ) inside the <VRTRasterBand> element. The
dataset pointed by the <SourceFilename> in <Overview> can be a .vrt.

So, schematically, you would have something like 100_percent.vrt :

<VRTDataset>
   <VRTRasterBand>
       <Overview>
           <SourceFilename relativeToVRT="1">50_percent.vrt</SourceFilename>
           <SourceBand>1</SourceBand>
       </Overview>
   </VRTRasterBand>
</VRTDataset>

and in 50_percent.vrt :

<VRTDataset>
   <VRTRasterBand>
       <Overview>
           <SourceFilename relativeToVRT="1">25_percent.vrt</SourceFilename>
           <SourceBand>1</SourceBand>
       </Overview>
   </VRTRasterBand>
</VRTDataset>

and so on

>
> -Jukka Rahkonen-
>
> _______________________________________________
> gdal-dev mailing list
> [hidden email]
> http://lists.osgeo.org/mailman/listinfo/gdal-dev
_______________________________________________
gdal-dev mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Scale dependent VRT for overviews

Rahkonen Jukka (Tike)
Even Rouault <even.rouault <at> mines-paris.org> writes:

>
> Jukka,
>
> I think you could achieve what you want by adding a <Overview> element (see
> http://www.gdal.org/gdal_vrttut.html ) inside the <VRTRasterBand> element. The
> dataset pointed by the <SourceFilename> in <Overview> can be a .vrt.
>
> So, schematically, you would have something like 100_percent.vrt :
>
> <VRTDataset>
>    <VRTRasterBand>
>        <Overview>
>            <SourceFilename relativeToVRT="1">50_percent.vrt</SourceFilename>
>            <SourceBand>1</SourceBand>
>        </Overview>
>    </VRTRasterBand>
> </VRTDataset>
>
> and in 50_percent.vrt :
>
> <VRTDataset>
>    <VRTRasterBand>
>        <Overview>
>            <SourceFilename relativeToVRT="1">25_percent.vrt</SourceFilename>
>            <SourceBand>1</SourceBand>
>        </Overview>
>    </VRTRasterBand>
> </VRTDataset>

I finally tried how it goes with VRT overviews.  The result is rather well
usable with QGis. The VRT file containing an overview chain opens a bit slow but
once it is open it works totally transparently for the user. I had luck in my
trial because all my images do not have same extents which relealed one trouble.
Finland gets squeezed when zoomed far out because two images with smallest
scales are wider than the others.  For preventing wrong scaling I guess I should
clip all overview images to suit the extents of the primary VRT. Anyway, I
consider that this kind of system could be useful sometimes.

Test set is downloadable from
http://latuviitta.org/documents/GDAL_Scale_dependent_VRT_test.zip

Archive contains 5 original tiffs, each with a different pixel size and
rendering styles, and corresponding .VRT files.  Unzip and open the "t0320.vrt"
file with QGis and that's all.  QGis and user believes it is just one map even
all five originals are used one by one depending on the scale.

The native SRID is EPSG:3067.

-Jukka Rahkonen-





_______________________________________________
gdal-dev mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/gdal-dev