Quantcast

[gdal-dev] Read error on large VRT files

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[gdal-dev] Read error on large VRT files

Armin Burger
Hi all

I sometimes get read errors when trying to convert large input VRT files
into a single BigTIFF file, typically applying an outsize of e.g. 50% or
25%. Errors are like

ERROR 1: TIFFFillTile:Read error at row 11264, col 11264; got 46 by
tes, expected 84
ERROR 1: TIFFReadEncodedTile() failed.

ERROR 1: 2352/2592_BUCNFD.tif, band 1: IReadBlock failed at X offset 45,
Y offset 47

The tile that returns the error can be converted alone without problems.
All together the number of tiles of the VRT can be towards 6000, having
100 GB in total, tiles are LZW or Deflate compressed TIFF's.


Is there a possible source of errors when trying to read and convert
VRT's of compressed TIFF of that size?

Thanks for any hint

Armin
_______________________________________________
gdal-dev mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Read error on large VRT files

Armin Burger
Margherita
 
thanks, now that you say this, I remeber a special GDAL option "GDAL_MAX_DATASET_POOL_SIZE" which by default is 100. The operating system limit itself is high enough on our server (65k). When running the processing again with less memory cache (and hence likely less concurrent opened files) it worked without errors. I will set this GDAL option and see if I ever run into this errors again.
 
armin
 
Gesendet: Mittwoch, 21. Dezember 2016 um 07:29 Uhr
Von: "Margherita Di Leo" <[hidden email]>
An: [hidden email]
Betreff: Re: [gdal-dev] Read error on large VRT files
Hey Armin,
 
Could it be a limitation in number of open files at same time? See https://grasswiki.osgeo.org/wiki/Large_raster_data_processing#Troubleshooting
 
Il giorno mar 20 dic 2016 alle 22:21 Armin Burger <[hidden email]> ha scritto:
Hi all



I sometimes get read errors when trying to convert large input VRT files

into a single BigTIFF file, typically applying an outsize of e.g. 50% or

25%. Errors are like



ERROR 1: TIFFFillTile:Read error at row 11264, col 11264; got 46 by

tes, expected 84

ERROR 1: TIFFReadEncodedTile() failed.



ERROR 1: 2352/2592_BUCNFD.tif, band 1: IReadBlock failed at X offset 45,

Y offset 47



The tile that returns the error can be converted alone without problems.

All together the number of tiles of the VRT can be towards 6000, having

100 GB in total, tiles are LZW or Deflate compressed TIFF's.





Is there a possible source of errors when trying to read and convert

VRT's of compressed TIFF of that size?



Thanks for any hint



Armin

_______________________________________________

gdal-dev mailing list

[hidden email]

http://lists.osgeo.org/mailman/listinfo/gdal-dev

_______________________________________________
gdal-dev mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Read error on large VRT files

Even Rouault-2

On mercredi 21 décembre 2016 10:29:27 CET Armin Burger wrote:

> Margherita

>  

> thanks, now that you say this, I remeber a special GDAL option

> "GDAL_MAX_DATASET_POOL_SIZE" which by default is 100. The operating system

> limit itself is high enough on our server (65k). When running the

> processing again with less memory cache (and hence likely less concurrent

> opened files) it worked without errors. I will set this GDAL option and see

> if I ever run into this errors again.

 

On which option did you play exactly ?

 

The TIFF error you get is a rather low level one (really trying to read a range of bytes from a file), and I can't explain why GDAL options would play a role on it.

 

Even

 

--

Spatialys - Geospatial professional services

http://www.spatialys.com


_______________________________________________
gdal-dev mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Read error on large VRT files

Armin Burger
I meant the GDAL_MAX_DATASET_POOL_SIZE option. Don't know if it really
has any influence, but these read errors happened multiple times in the
past when trying to translate VRT's with lots of input files, for tiles
that were actually readable. It could be of course also a problem of the
network file system, short network time-outs etc.

Armin

On 21/12/16 10:49, Even Rouault wrote:

> On mercredi 21 décembre 2016 10:29:27 CET Armin Burger wrote:
>
>> Margherita
>
>>
>
>> thanks, now that you say this, I remeber a special GDAL option
>
>> "GDAL_MAX_DATASET_POOL_SIZE" which by default is 100. The operating system
>
>> limit itself is high enough on our server (65k). When running the
>
>> processing again with less memory cache (and hence likely less concurrent
>
>> opened files) it worked without errors. I will set this GDAL option
> and see
>
>> if I ever run into this errors again.
>
>
>
> On which option did you play exactly ?
>
>
>
> The TIFF error you get is a rather low level one (really trying to read
> a range of bytes from a file), and I can't explain why GDAL options
> would play a role on it.
>
>
>
> Even
>
>
>
> --
>
> Spatialys - Geospatial professional services
>
> http://www.spatialys.com
>
_______________________________________________
gdal-dev mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/gdal-dev
Loading...