[gdal-dev] gdal_retile.py killed when dealing with large .vrt + pixel function

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

[gdal-dev] gdal_retile.py killed when dealing with large .vrt + pixel function

Wee Sritippho
Hi,

I have to retile about 100,000 images of size about 2667x2667 pixels for geoserver. The source images' nodata area were badly damaged due to compression so I decided to use VRT pixel function max to make the result tiles seamless.

However, the output was quite weird -- Killed appeared and the script exited:

Building internal Index for 1 tile(s) ... finished
Filename: test_sum.vrt
File Size: 1178083x2179667x1
Pixel Size: 0.780877 x -0.780877
UL:(10841319.553138,2329047.665115)   LR:(11761257.511933,626995.763142)
tileWidth:   2048
tileHeight:  2048
countTilesX: 576
countTilesY: 1065
overlap:     0
Killed

This is the command that I used:

mkdir pyramid
gdal_retile.py -v -r bilinear -levels 5 -ps 2048 2048 -co "TILED=YES" -co "COMPRESS=JPEG" -targetDir pyramid test_sum.vrt

And the pixel function that I used:

<VRTDataset rasterXSize="1178083" rasterYSize="2179667">
  <SRS>...</SRS>
  <GeoTransform>...</GeoTransform>
  <VRTRasterBand dataType="Byte" band="1" subClass="VRTDerivedRasterBand">
    <PixelFunctionType>max</PixelFunctionType>
    <PixelFunctionLanguage>Python</PixelFunctionLanguage>
    <PixelFunctionCode><![CDATA[
import numpy as np
def max(in_ar, out_ar, xoff, yoff, xsize, ysize, raster_xsize, raster_ysize, buf_radius, gt, **kwargs):
    np.round_(np.clip(np.max(in_ar, axis = 0),0,255), out = out_ar)
]]>
    </PixelFunctionCode>
    <ColorInterp>Gray</ColorInterp>
    <SimpleSource>...</SimpleSource>
    ...
  </VRTRasterBand>
</VRTDataset>

This is what I tried:

- retile the VRT file without pixel function --> OK, but the result isn't seamless (black background of each images are visible)
- retile smaller VRT file (just 5 images) with pixel function --> OK, but incomplete

How can I retile the whole dataset seamlessly?

Software used:

GDAL 2.4.1, released 2019/03/15
Python 2.7.15rc1

Thank you,
Wee.


_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: gdal_retile.py killed when dealing with large .vrt + pixel function

Even Rouault-2
I cannot reproduce with a test_sum.vrt relying on a tiny 20x20 raster:

<VRTDataset rasterXSize="1178083" rasterYSize="2179667">
  <VRTRasterBand dataType="Byte" band="1" subClass="VRTDerivedRasterBand">
    <PixelFunctionType>max</PixelFunctionType>
    <PixelFunctionLanguage>Python</PixelFunctionLanguage>
    <PixelFunctionCode><![CDATA[
import numpy as np
def max(in_ar, out_ar, xoff, yoff, xsize, ysize, raster_xsize, raster_ysize, buf_radius, gt, **kwargs):
    np.round_(np.clip(np.max(in_ar, axis = 0),0,255), out = out_ar)
]]>
    </PixelFunctionCode>
    <SimpleSource>
      <SourceFilename relativeToVRT="1">byte.tif</SourceFilename>
      <SourceBand>1</SourceBand>
      <SourceProperties RasterXSize="20" RasterYSize="20" DataType="Byte" BlockXSize="20" BlockYSize="20" />
      <SrcRect xOff="0" yOff="0" xSize="20" ySize="20" />
      <DstRect xOff="0" yOff="0" xSize="1178083" ySize="2179667" />
    </SimpleSource>
  </VRTRasterBand>
</VRTDataset>


$ GDAL_VRT_ENABLE_PYTHON=YES python swig/python/scripts/gdal_retile.py -v -r bilinear -levels 5 -ps 2048 2048 -co "TILED=YES" -co "COMPRESS=JPEG"  -targetDir pyramid test_sum.vrt
Building internal Index for 1 tile(s) ... finished
Filename: test_sum.vrt
File Size: 1178083x2179667x1
Pixel Size: 1.000000 x -1.000000
UL:(0.000000,0.000000)   LR:(1178083.000000,-2179667.000000)
tileWidth:   2048
tileHeight:  2048
countTilesX: 576
countTilesY: 1065
overlap:     0
pyramid/test_sum_0001_0001.tif : 0|0-->2048-2048
pyramid/test_sum_0001_0002.tif : 2048|0-->2048-2048
pyramid/test_sum_0001_0003.tif : 4096|0-->2048-2048
pyramid/test_sum_0001_0004.tif : 6144|0-->2048-2048
pyramid/test_sum_0001_0005.tif : 8192|0-->2048-2048
pyramid/test_sum_0001_0006.tif : 10240|0-->2048-2048
pyramid/test_sum_0001_0007.tif : 12288|0-->2048-2048
pyramid/test_sum_0001_0008.tif : 14336|0-->2048-2048
pyramid/test_sum_0001_0009.tif : 16384|0-->2048-2048
pyramid/test_sum_0001_0010.tif : 18432|0-->2048-2048
[...]

You'll have to run under a debugger and see where this blows up.

Even

--
Spatialys - Geospatial professional services
http://www.spatialys.com
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: gdal_retile.py killed when dealing with large .vrt + pixel function

Wee Sritippho
Hi Even,

Thanks for your kindly attention. I'm sad that I couldn't provide you a proper material to reproduce the issue (since I didn't own them).

But I think I just found the cause of this problem - lack of memory.

From your suggestion about using a debugger, I searched the internet and found out that I could pass '--debug ON' to every gdal commands. After I passed the option, more info are displayed:

GDAL: GDALDriver::Create(Memory,TileIndex,0,0,0,Unknown,(nil))
GDAL: GDALOpen(test_sum.vrt, this=0x2b10bc0) succeeds as VRT.
GDAL: GDALClose(test_sum.vrt, this=0x2b10bc0)
Building internal Index for 1 tile(s) ... finished
GDAL: GDALOpen(test_sum.vrt, this=0x2b10bc0) succeeds as VRT.
Filename: test_sum.vrt
File Size: 1178083x2179667x1
Pixel Size: 0.780877 x -0.780877
UL:(10841319.553138,2329047.665115)   LR:(11761257.511933,626995.763142)
tileWidth:   2048
tileHeight:  2048
countTilesX: 576
countTilesY: 1065
overlap:     0
GDAL: GDALDriver::Create(Memory,TileResult_0,0,0,0,Unknown,(nil))
GDAL: GDALDriver::Create(MEM,TEMP,2048,2048,1,Byte,(nil))
Killed

The script was killed after some message about 'memory', so I monitored my system's memory while running the script. The script gradually consumed my 8GB RAM then my 4GB swap before being killed.

I'll try to increase the memory/swap and report back.

Thank you,
Wee.
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev