[gdal-dev] Error when trying to compress a TIFF file

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

[gdal-dev] Error when trying to compress a TIFF file

Hanlie Pretorius
Hi,

I'm trying to convert a 16 bit image to 8 bits and at the same time
add JPEG compression, but gdal reports errors during the process.

gdal_info for the input file is:

Driver: GTiff/GeoTIFF

Files: D:/SPOT 6/PS_Envi/Test_8bit.TIF

Size is 30717, 28629

Coordinate System is:

PROJCS["WGS 84 / UTM zone 34S",

GEOGCS["WGS 84",

DATUM["WGS_1984",

SPHEROID["WGS 84",6378137,298.257223563,

AUTHORITY["EPSG","7030"]],

AUTHORITY["EPSG","6326"]],

PRIMEM["Greenwich",0],

UNIT["degree",0.0174532925199433],

AUTHORITY["EPSG","4326"]],

PROJECTION["Transverse_Mercator"],

PARAMETER["latitude_of_origin",0],

PARAMETER["central_meridian",21],

PARAMETER["scale_factor",0.9996],

PARAMETER["false_easting",500000],

PARAMETER["false_northing",10000000],

UNIT["metre",1,

AUTHORITY["EPSG","9001"]],

AUTHORITY["EPSG","32734"]]

Origin = (537234.000000000000000,6296730.000000000000000)

Pixel Size = (1.500000000000000,-1.500000000000000)

Metadata:

AREA_OR_POINT=Area

Band_1=CN Sharp R (Band 3:IMG_SPOT6_MS_201304220815274_ORT_742338101_R1C1.TIF)

Band_2=CN Sharp G (Band 2:IMG_SPOT6_MS_201304220815274_ORT_742338101_R1C1.TIF)

Band_3=CN Sharp B (Band 1:IMG_SPOT6_MS_201304220815274_ORT_742338101_R1C1.TIF)

Image Structure Metadata:

INTERLEAVE=PIXEL

Corner Coordinates:

Upper Left ( 537234.000, 6296730.000) ( 21d24' 2.56"E, 33d28' 5.67"S)

Lower Left ( 537234.000, 6253786.500) ( 21d24' 9.04"E, 33d51'20.01"S)

Upper Right ( 583309.500, 6296730.000) ( 21d53'47.54"E, 33d27'56.33"S)

Lower Right ( 583309.500, 6253786.500) ( 21d54' 2.03"E, 33d51'10.53"S)

Center ( 560271.750, 6275258.250) ( 21d39' 0.30"E, 33d39'39.04"S)

Band 1 Block=30717x1 Type=Byte, ColorInterp=Red

Description = CN Sharp R (Band
3:IMG_SPOT6_MS_201304220815274_ORT_742338101_R1C1.TIF)

Band 2 Block=30717x1 Type=Byte, ColorInterp=Green

Description = CN Sharp G (Band
2:IMG_SPOT6_MS_201304220815274_ORT_742338101_R1C1.TIF)

Band 3 Block=30717x1 Type=Byte, ColorInterp=Blue

Description = CN Sharp B (Band
1:IMG_SPOT6_MS_201304220815274_ORT_742338101_R1C1.TIF)

The command I'm using is:
gdal_translate -ot Byte -of GTiff -scale 0 4905 0 255 -stats -co
COMPRESS=JPEG "D:/SPOT 6/PS_Envi/Test.TIF" "D:/SPOT
6/PS_Envi/Test_8bit_JPEG_Compr.TIF"

The last part of the error message is:
ERROR 1: IReadBlock failed at X offset 18, Y offset 2
ERROR 1: VRTSourceRasterBand::iRatserIO() called recursively on the
same band. It looks like the VRT is referencing itself.
ERROR 1: IReadBlock failed at X offset 19, Y offset 2
ERROR 1: VRTSourceRasterBand::iRatserIO() called recursively on the
same band. It looks like the VRT is referencing itself.
More than 1000 errors or warnings have been reported. No more will be
reported from now.
0...10...20...30...40...50...60...70...80...90...100 - done.

gdal_translate does complete the process and I get an output file that
looks usable, but I think something is wrong with the file because
working with it in QGIS takes longer than the original image, which is
two orders of magnitude bigger in file size.

gdal_info for the output file is:
Driver: GTiff/GeoTIFF

Files: D:/SPOT 6/PS_Envi/Test_8bit_JPEG_Compr.TIF

D:/SPOT 6/PS_Envi/Test_8bit_JPEG_Compr.TIF.aux.xml

Size is 30717, 28629

Coordinate System is:

PROJCS["WGS 84 / UTM zone 34S",

GEOGCS["WGS 84",

DATUM["WGS_1984",

SPHEROID["WGS 84",6378137,298.257223563,

AUTHORITY["EPSG","7030"]],

AUTHORITY["EPSG","6326"]],

PRIMEM["Greenwich",0],

UNIT["degree",0.0174532925199433],

AUTHORITY["EPSG","4326"]],

PROJECTION["Transverse_Mercator"],

PARAMETER["latitude_of_origin",0],

PARAMETER["central_meridian",21],

PARAMETER["scale_factor",0.9996],

PARAMETER["false_easting",500000],

PARAMETER["false_northing",10000000],

UNIT["metre",1,

AUTHORITY["EPSG","9001"]],

AUTHORITY["EPSG","32734"]]

Origin = (537234.000000000000000,6296730.000000000000000)

Pixel Size = (1.500000000000000,-1.500000000000000)

Metadata:

AREA_OR_POINT=Area

Band_1=CN Sharp R (Band 3:IMG_SPOT6_MS_201304220815274_ORT_742338101_R1C1.TIF)

Band_2=CN Sharp G (Band 2:IMG_SPOT6_MS_201304220815274_ORT_742338101_R1C1.TIF)

Band_3=CN Sharp B (Band 1:IMG_SPOT6_MS_201304220815274_ORT_742338101_R1C1.TIF)

Image Structure Metadata:

COMPRESSION=JPEG

INTERLEAVE=PIXEL

Corner Coordinates:

Upper Left ( 537234.000, 6296730.000) ( 21d24' 2.56"E, 33d28' 5.67"S)

Lower Left ( 537234.000, 6253786.500) ( 21d24' 9.04"E, 33d51'20.01"S)

Upper Right ( 583309.500, 6296730.000) ( 21d53'47.54"E, 33d27'56.33"S)

Lower Right ( 583309.500, 6253786.500) ( 21d54' 2.03"E, 33d51'10.53"S)

Center ( 560271.750, 6275258.250) ( 21d39' 0.30"E, 33d39'39.04"S)

Band 1 Block=30717x16 Type=Byte, ColorInterp=Red

Description = CN Sharp R (Band
3:IMG_SPOT6_MS_201304220815274_ORT_742338101_R1C1.TIF)

Band 2 Block=30717x16 Type=Byte, ColorInterp=Green

Description = CN Sharp G (Band
2:IMG_SPOT6_MS_201304220815274_ORT_742338101_R1C1.TIF)

Band 3 Block=30717x16 Type=Byte, ColorInterp=Blue

Description = CN Sharp B (Band
1:IMG_SPOT6_MS_201304220815274_ORT_742338101_R1C1.TIF)

Does anybody perhaps know what is going wrong or can suggest another
method to accomplishing what I wish to?

I have successfully created an 8 bit image without using compression,
so I'm pretty sure the error stems from the compression operation.

Regards
Hanlie
_______________________________________________
gdal-dev mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Error when trying to compress a TIFF file

Even Rouault-2
Hanlie,

The warning "VRTSourceRasterBand::iRatserIO() called recursively on the same
band" is bug http://trac.osgeo.org/gdal/ticket/5468
It is due to the statistics computation on the intermediate VRT file
(implementation detail), but shouldn't influence on the image itself
You can run afterward "gdalinfo -stats Test_8bit_JPEG_Compr.TIF" to compute
statistics on the result TIF.

The slowness in QGIS is not surprising since JPEG takes time to decompressing,
especially for big images, that are not tiled and have no overviews. What could
help is to add -co TILED=YES on the gdal_translate command line.
And then run "gdaladdo --config COMPRESS_OVERVIEW JPEG Test_8bit_JPEG_Compr.TIF
2 4 8 16" on the result to create overviews.

Even

> Hi,
>
> I'm trying to convert a 16 bit image to 8 bits and at the same time
> add JPEG compression, but gdal reports errors during the process.
>
> gdal_info for the input file is:
>
> Driver: GTiff/GeoTIFF
>
> Files: D:/SPOT 6/PS_Envi/Test_8bit.TIF
>
> Size is 30717, 28629
>
> Coordinate System is:
>
> PROJCS["WGS 84 / UTM zone 34S",
>
> GEOGCS["WGS 84",
>
> DATUM["WGS_1984",
>
> SPHEROID["WGS 84",6378137,298.257223563,
>
> AUTHORITY["EPSG","7030"]],
>
> AUTHORITY["EPSG","6326"]],
>
> PRIMEM["Greenwich",0],
>
> UNIT["degree",0.0174532925199433],
>
> AUTHORITY["EPSG","4326"]],
>
> PROJECTION["Transverse_Mercator"],
>
> PARAMETER["latitude_of_origin",0],
>
> PARAMETER["central_meridian",21],
>
> PARAMETER["scale_factor",0.9996],
>
> PARAMETER["false_easting",500000],
>
> PARAMETER["false_northing",10000000],
>
> UNIT["metre",1,
>
> AUTHORITY["EPSG","9001"]],
>
> AUTHORITY["EPSG","32734"]]
>
> Origin = (537234.000000000000000,6296730.000000000000000)
>
> Pixel Size = (1.500000000000000,-1.500000000000000)
>
> Metadata:
>
> AREA_OR_POINT=Area
>
> Band_1=CN Sharp R (Band
> 3:IMG_SPOT6_MS_201304220815274_ORT_742338101_R1C1.TIF)
>
> Band_2=CN Sharp G (Band
> 2:IMG_SPOT6_MS_201304220815274_ORT_742338101_R1C1.TIF)
>
> Band_3=CN Sharp B (Band
> 1:IMG_SPOT6_MS_201304220815274_ORT_742338101_R1C1.TIF)
>
> Image Structure Metadata:
>
> INTERLEAVE=PIXEL
>
> Corner Coordinates:
>
> Upper Left ( 537234.000, 6296730.000) ( 21d24' 2.56"E, 33d28' 5.67"S)
>
> Lower Left ( 537234.000, 6253786.500) ( 21d24' 9.04"E, 33d51'20.01"S)
>
> Upper Right ( 583309.500, 6296730.000) ( 21d53'47.54"E, 33d27'56.33"S)
>
> Lower Right ( 583309.500, 6253786.500) ( 21d54' 2.03"E, 33d51'10.53"S)
>
> Center ( 560271.750, 6275258.250) ( 21d39' 0.30"E, 33d39'39.04"S)
>
> Band 1 Block=30717x1 Type=Byte, ColorInterp=Red
>
> Description = CN Sharp R (Band
> 3:IMG_SPOT6_MS_201304220815274_ORT_742338101_R1C1.TIF)
>
> Band 2 Block=30717x1 Type=Byte, ColorInterp=Green
>
> Description = CN Sharp G (Band
> 2:IMG_SPOT6_MS_201304220815274_ORT_742338101_R1C1.TIF)
>
> Band 3 Block=30717x1 Type=Byte, ColorInterp=Blue
>
> Description = CN Sharp B (Band
> 1:IMG_SPOT6_MS_201304220815274_ORT_742338101_R1C1.TIF)
>
> The command I'm using is:
> gdal_translate -ot Byte -of GTiff -scale 0 4905 0 255 -stats -co
> COMPRESS=JPEG "D:/SPOT 6/PS_Envi/Test.TIF" "D:/SPOT
> 6/PS_Envi/Test_8bit_JPEG_Compr.TIF"
>
> The last part of the error message is:
> ERROR 1: IReadBlock failed at X offset 18, Y offset 2
> ERROR 1: VRTSourceRasterBand::iRatserIO() called recursively on the
> same band. It looks like the VRT is referencing itself.
> ERROR 1: IReadBlock failed at X offset 19, Y offset 2
> ERROR 1: VRTSourceRasterBand::iRatserIO() called recursively on the
> same band. It looks like the VRT is referencing itself.
> More than 1000 errors or warnings have been reported. No more will be
> reported from now.
> 0...10...20...30...40...50...60...70...80...90...100 - done.
>
> gdal_translate does complete the process and I get an output file that
> looks usable, but I think something is wrong with the file because
> working with it in QGIS takes longer than the original image, which is
> two orders of magnitude bigger in file size.
>
> gdal_info for the output file is:
> Driver: GTiff/GeoTIFF
>
> Files: D:/SPOT 6/PS_Envi/Test_8bit_JPEG_Compr.TIF
>
> D:/SPOT 6/PS_Envi/Test_8bit_JPEG_Compr.TIF.aux.xml
>
> Size is 30717, 28629
>
> Coordinate System is:
>
> PROJCS["WGS 84 / UTM zone 34S",
>
> GEOGCS["WGS 84",
>
> DATUM["WGS_1984",
>
> SPHEROID["WGS 84",6378137,298.257223563,
>
> AUTHORITY["EPSG","7030"]],
>
> AUTHORITY["EPSG","6326"]],
>
> PRIMEM["Greenwich",0],
>
> UNIT["degree",0.0174532925199433],
>
> AUTHORITY["EPSG","4326"]],
>
> PROJECTION["Transverse_Mercator"],
>
> PARAMETER["latitude_of_origin",0],
>
> PARAMETER["central_meridian",21],
>
> PARAMETER["scale_factor",0.9996],
>
> PARAMETER["false_easting",500000],
>
> PARAMETER["false_northing",10000000],
>
> UNIT["metre",1,
>
> AUTHORITY["EPSG","9001"]],
>
> AUTHORITY["EPSG","32734"]]
>
> Origin = (537234.000000000000000,6296730.000000000000000)
>
> Pixel Size = (1.500000000000000,-1.500000000000000)
>
> Metadata:
>
> AREA_OR_POINT=Area
>
> Band_1=CN Sharp R (Band
> 3:IMG_SPOT6_MS_201304220815274_ORT_742338101_R1C1.TIF)
>
> Band_2=CN Sharp G (Band
> 2:IMG_SPOT6_MS_201304220815274_ORT_742338101_R1C1.TIF)
>
> Band_3=CN Sharp B (Band
> 1:IMG_SPOT6_MS_201304220815274_ORT_742338101_R1C1.TIF)
>
> Image Structure Metadata:
>
> COMPRESSION=JPEG
>
> INTERLEAVE=PIXEL
>
> Corner Coordinates:
>
> Upper Left ( 537234.000, 6296730.000) ( 21d24' 2.56"E, 33d28' 5.67"S)
>
> Lower Left ( 537234.000, 6253786.500) ( 21d24' 9.04"E, 33d51'20.01"S)
>
> Upper Right ( 583309.500, 6296730.000) ( 21d53'47.54"E, 33d27'56.33"S)
>
> Lower Right ( 583309.500, 6253786.500) ( 21d54' 2.03"E, 33d51'10.53"S)
>
> Center ( 560271.750, 6275258.250) ( 21d39' 0.30"E, 33d39'39.04"S)
>
> Band 1 Block=30717x16 Type=Byte, ColorInterp=Red
>
> Description = CN Sharp R (Band
> 3:IMG_SPOT6_MS_201304220815274_ORT_742338101_R1C1.TIF)
>
> Band 2 Block=30717x16 Type=Byte, ColorInterp=Green
>
> Description = CN Sharp G (Band
> 2:IMG_SPOT6_MS_201304220815274_ORT_742338101_R1C1.TIF)
>
> Band 3 Block=30717x16 Type=Byte, ColorInterp=Blue
>
> Description = CN Sharp B (Band
> 1:IMG_SPOT6_MS_201304220815274_ORT_742338101_R1C1.TIF)
>
> Does anybody perhaps know what is going wrong or can suggest another
> method to accomplishing what I wish to?
>
> I have successfully created an 8 bit image without using compression,
> so I'm pretty sure the error stems from the compression operation.
>
> Regards
> Hanlie
> _______________________________________________
> gdal-dev mailing list
> [hidden email]
> http://lists.osgeo.org/mailman/listinfo/gdal-dev
>


--
Spatialys - Geospatial professional services
http://www.spatialys.com
_______________________________________________
gdal-dev mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Error when trying to compress a TIFF file

Hanlie Pretorius
Hi Even,

Thanks, the gdaldo command made a huge difference to the speed with
which QGIS works with the raster.

The image looks a bit shoddy, though, even when zoomed out. It
improved when I set the quality to 100 and the file is still a tenth
of the original size.

Would it help to create more levels in the overview and, if so, at
which end should I add the levels?

Hanlie

2014-08-26 16:25 MGT+02:00, Even Rouault <[hidden email]>:

> Hanlie,
>
> The warning "VRTSourceRasterBand::iRatserIO() called recursively on the
> same
> band" is bug http://trac.osgeo.org/gdal/ticket/5468
> It is due to the statistics computation on the intermediate VRT file
> (implementation detail), but shouldn't influence on the image itself
> You can run afterward "gdalinfo -stats Test_8bit_JPEG_Compr.TIF" to compute
> statistics on the result TIF.
>
> The slowness in QGIS is not surprising since JPEG takes time to
> decompressing,
> especially for big images, that are not tiled and have no overviews. What
> could
> help is to add -co TILED=YES on the gdal_translate command line.
> And then run "gdaladdo --config COMPRESS_OVERVIEW JPEG
> Test_8bit_JPEG_Compr.TIF
> 2 4 8 16" on the result to create overviews.
>
> Even
>
_______________________________________________
gdal-dev mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Error when trying to compress a TIFF file

Even Rouault-2
Selon Hanlie Pretorius <[hidden email]>:

> Hi Even,
>
> Thanks, the gdaldo command made a huge difference to the speed with
> which QGIS works with the raster.
>
> The image looks a bit shoddy, though, even when zoomed out. It
> improved when I set the quality to 100 and the file is still a tenth
> of the original size.

You can try adding -r average or -r cubic when computing overviews with gdaladdo
to improve a bit their quality.

>
> Would it help to create more levels in the overview and, if so, at
> which end should I add the levels?

Your full-res image is around 30000 x 30000.
With a max overview of factor 16 as in the below command, you have an image of
approximate size 1800x1800, which is still a bit big
Targetting 256x256 for the smallest overview would bring you instantaneous
opening in QGIS. So 30000 / 256 = 117.XXXX, round that to 128. So use "2 4 8 16
32 64 128".

>
> Hanlie
>
> 2014-08-26 16:25 MGT+02:00, Even Rouault <[hidden email]>:
> > Hanlie,
> >
> > The warning "VRTSourceRasterBand::iRatserIO() called recursively on the
> > same
> > band" is bug http://trac.osgeo.org/gdal/ticket/5468
> > It is due to the statistics computation on the intermediate VRT file
> > (implementation detail), but shouldn't influence on the image itself
> > You can run afterward "gdalinfo -stats Test_8bit_JPEG_Compr.TIF" to compute
> > statistics on the result TIF.
> >
> > The slowness in QGIS is not surprising since JPEG takes time to
> > decompressing,
> > especially for big images, that are not tiled and have no overviews. What
> > could
> > help is to add -co TILED=YES on the gdal_translate command line.
> > And then run "gdaladdo --config COMPRESS_OVERVIEW JPEG
> > Test_8bit_JPEG_Compr.TIF
> > 2 4 8 16" on the result to create overviews.
> >
> > Even
> >
>


--
Spatialys - Geospatial professional services
http://www.spatialys.com
_______________________________________________
gdal-dev mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Error when trying to compress a TIFF file

Hanlie Pretorius
Thanks, the r- average option improved the quality quite a bit.

2014-08-26 17:34 MGT+02:00, Even Rouault <[hidden email]>:

> Selon Hanlie Pretorius <[hidden email]>:
>
>> Hi Even,
>>
>> Thanks, the gdaldo command made a huge difference to the speed with
>> which QGIS works with the raster.
>>
>> The image looks a bit shoddy, though, even when zoomed out. It
>> improved when I set the quality to 100 and the file is still a tenth
>> of the original size.
>
> You can try adding -r average or -r cubic when computing overviews with
> gdaladdo
> to improve a bit their quality.
>
>>
>> Would it help to create more levels in the overview and, if so, at
>> which end should I add the levels?
>
> Your full-res image is around 30000 x 30000.
> With a max overview of factor 16 as in the below command, you have an image
> of
> approximate size 1800x1800, which is still a bit big
> Targetting 256x256 for the smallest overview would bring you instantaneous
> opening in QGIS. So 30000 / 256 = 117.XXXX, round that to 128. So use "2 4 8
> 16
> 32 64 128".
>
>>
_______________________________________________
gdal-dev mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/gdal-dev