[gdal-dev] Problem when creating overviews for large input files with gdaladdo

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[gdal-dev] Problem when creating overviews for large input files with gdaladdo

Armin Burger
Dear all

I ran into a problem when trying to create overviews for large
LZW-compressed GeoTIFF images using gdaladdo. The TIFF's are internally
tiled, the file type is Byte. GDAL versions tested are 2.1.1 and 1.11,
running on 64bit Linux.

The first try was using an input file of size 1038336 x 565248 pixels.
Setting CPL_DEBUG=ON the processing goes up to the message

...10...20...30...40...50...60...70..GDAL: Potential thrashing on band 1
of tileindex_nodsm_otsu_2

and then it remains at this "70.." % forever, with no further log
output. This happened for all files of similar size, on multiple
machines, on both GDAL versions. Always hangs at this "70.." state, the
process remains alive but nothing is written any more to the output
file. I used the command like

gdaladdo -ro --config COMPRESS_OVERVIEW LZW  INPUT_FILE 2 4 8 16 32 64

Trying to specify BIGTIFF or not for the overview did not make any
difference.

Using a smaller file or converting the large input file to 50% size and
running the gdaladdo on this smaller file finished successfully, but
also here I got some warnings of a similar kind, and the process got
stuck for a while after the ".70..GDAL: Potential thrashing..." warning,
but then continued to the end. The full log is attached.

An additional observation I made was that the overview file stopping at
70% was already bigger than the input file.

I was thinking to create single lower resolution files and merging them
to an overview file, but there's no built-in functionality for this, so
did not follow it up further.

Could there be a sort of limitation of maximum pixels x*y, or with the
LZW compression? Any ideas how this could be solved?

Thanks in advance for any hint
Armin

_______________________________________________
gdal-dev mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/gdal-dev

gdaladdo.log (5K) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Problem when creating overviews for large input files with gdaladdo

Even Rouault-2

Armin,

 

This is a "well known" performance issue with the geotiff driver / libtiff when switching back and forth between IFD (image file directories). At ~ 75%, you start generating overview factor 4 and that implies switching regularly between the one previously computed (read operations) and the new one (write operations). Each time you switch, you need to read and write the TileOffsets and TileByteCounts tags (those can be 10's of megabytes large on huge file, and are the root cause for the performance problem). With infinite patience, this would eventually complete.

 

A trick you can use is :

 

gdaladdo -ro --config COMPRESS_OVERVIEW LZW INPUT_FILE 2

gdaladdo -ro --config COMPRESS_OVERVIEW LZW INPUT_FILE.ovr 2

gdaladdo -ro --config COMPRESS_OVERVIEW LZW INPUT_FILE.ovr.ovr 2

gdaladdo -ro --config COMPRESS_OVERVIEW LZW INPUT_FILE.ovr.ovr.ovr 2

 

Even

 

 

 

--

Spatialys - Geospatial professional services

http://www.spatialys.com


_______________________________________________
gdal-dev mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Problem when creating overviews for large input files with gdaladdo

Armin Burger
Hi Even

thanks a lot for the fast reply and the explanation! I will try your
workaround. The patience needed to wait for the completion of the
"normal" method seems to be a bit too high, I stopped the last test
after 4 days...

Best
Armin



On 19/12/16 22:26, Even Rouault wrote:

> Armin,
>
>
>
> This is a "well known" performance issue with the geotiff driver /
> libtiff when switching back and forth between IFD (image file
> directories). At ~ 75%, you start generating overview factor 4 and that
> implies switching regularly between the one previously computed (read
> operations) and the new one (write operations). Each time you switch,
> you need to read and write the TileOffsets and TileByteCounts tags
> (those can be 10's of megabytes large on huge file, and are the root
> cause for the performance problem). With infinite patience, this would
> eventually complete.
>
>
>
> A trick you can use is :
>
>
>
> gdaladdo -ro --config COMPRESS_OVERVIEW LZW INPUT_FILE 2
>
> gdaladdo -ro --config COMPRESS_OVERVIEW LZW INPUT_FILE.ovr 2
>
> gdaladdo -ro --config COMPRESS_OVERVIEW LZW INPUT_FILE.ovr.ovr 2
>
> gdaladdo -ro --config COMPRESS_OVERVIEW LZW INPUT_FILE.ovr.ovr.ovr 2
>
>
>
> Even
>
>
>
>
>
>
>
> --
>
> Spatialys - Geospatial professional services
>
> http://www.spatialys.com
>
_______________________________________________
gdal-dev mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/gdal-dev
Loading...