[gdal-dev] Just want to do simple batch compression...

[srygonic]

I have a lot of geotiffs which are quite large at times (meaning they have
quite a bit of unnecessary white space between) and I would like to run a
batch script to just do a simple compression on them as I believe that they
are not compressed because the files that just contain a lot of white space
are still unnecessarily huge.

I've been trying for quite a bit (totally new to GDAL) and reading around.
But I can't seem to produce any usable results. Most of the time my image
turns totally black (when opened in QGIS).

I've read that the images just needs to scale up but I'm either getting
thrown with 16 bit errors or still produces a black image.

I've been trying to piece together some commands via
http://www.gdal.org/frmt_gtiff.html and other blogs / discussion forums
online but I'm not too sure I'm heading in the right direction. As
mentioned... This is all pretty new to me. Just starting looking into GDAL
48 hours ago.

Below are the commands that I've tried. I'm using OSGeo4W Shell command line
to run this.

cd /d e:
cd E:\TIFFs_Test
gdal_translate GTIFF_RAW:GAPS_2017-02-23.tif test.tif -co PHOTOMETRIC=CMYK
-co TFW=YES -co TILED=YES -co BIGTIFF=IF_SAFER -co COMPRESS=JPEG

gdal_translate GTIFF_RAW:GAPS_2017-02-23.tif test1.tif -co PHOTOMETRIC=CMYK
-co TFW=YES -co TILED=YES -co BIGTIFF=IF_SAFER -co COMPRESS=JPEG -a_srs
EPSG:3414

gdal_translate GTIFF_RAW:GAPS_2017-02-23.tif test2.tif -co PHOTOMETRIC=YCBCR
-co TFW=YES -co TILED=YES -co BIGTIFF=IF_SAFER -co COMPRESS=JPEG -a_srs
EPSG:3414

gdal_translate GTIFF_RAW:GAPS_2017-02-23.tif test4.tif -co PHOTOMETRIC=YCBCR
-co TFW=YES -co TILED=YES -co BIGTIFF=IF_SAFER -co COMPRESS=JPEG -ot Int16
-scale 0 255 0 32767

gdal_translate GTIFF_RAW:GAPS_2017-02-23.tif test6.tif -co PHOTOMETRIC=YCBCR
-co TFW=YES -co TILED=YES -co BIGTIFF=IF_SAFER -co COMPRESS=JPEG -scale 0
255 0 32767

The image in question is uploaded here.
https://www.dropbox.com/s/vol0bflu5avw7xp/TIFFs_Test.rar?dl=0

The file is about 444MB but can be compress via zip to about 81MB. So there
is obviously still room for compression. Can anybody point me in the right
direction on how to go about this? I would like to eventually run a batch
script on a folder to auto compress all the TIFF files.



--
Sent from: http://osgeo-org.1560.x6.nabble.com/GDAL-Dev-f3742093.html
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev

[jratike80]

srygonic wrote
> I have a lot of geotiffs which are quite large at times (meaning they have
> quite a bit of unnecessary white space between) and I would like to run a
> batch script to just do a simple compression on them as I believe that
> they
> are not compressed because the files that just contain a lot of white
> space
> are still unnecessarily huge.

Hi,

If you run gdalinfo about your image you will notice that it is already
compressed:

Metadata:
  TIFFTAG_SOFTWARE=OrbitGIS 17.0.0
Image Structure Metadata:
  COMPRESSION=YCbCr JPEG
  INTERLEAVE=PIXEL
  SOURCE_COLOR_SPACE=YCbCr

However, one of your commands which is using actually the same parameters
does shrink the image from 465 MB into 170 MB.

gdal_translate GTIFF_RAW:GAPS_2017-02-23.tif test2.tif -co
PHOTOMETRIC=YCBCR -co TFW=YES -co TILED=YES -co BIGTIFF=IF_SAFER -co
COMPRESS=JPEG -a_srs EPSG:3414

For saving disk space I would consider cutting those areas which do have
real data into individual files and then combine them into a virtual mosaic
with gdalbuildvrt.  Empty space does compress rather well but not having any
pixels from areas with no data is most savvy option of all.

-Jukka Rahkonen-




--
Sent from: http://osgeo-org.1560.x6.nabble.com/GDAL-Dev-f3742093.html
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev

[srygonic]

The issue is that the image turns out to be completely black for me... not
too sure what I'm doing wrong here.

I understand that it might be better to cut up the files but that would take
too long (as there are too many files) and I'm wondering if there is a
faster solution here to reduce the size before taking that route.

Will look into this gdalbuildrvt as well once I'm back at the computer
tomorrow... any good tutorials for me to start off with?

Thanks for the assistance and the help Jukka!



--
Sent from: http://osgeo-org.1560.x6.nabble.com/GDAL-Dev-f3742093.html
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev

[jratike80]

srygonic wrote

> The issue is that the image turns out to be completely black for me... not
> too sure what I'm doing wrong here.
>
> I understand that it might be better to cut up the files but that would
> take
> too long (as there are too many files) and I'm wondering if there is a
> faster solution here to reduce the size before taking that route.
>
> Will look into this gdalbuildrvt as well once I'm back at the computer
> tomorrow... any good tutorials for me to start off with?
>
> Thanks for the assistance and the help Jukka!

Hi,

I had another look at the GAPS_2017-02-23.tif file and I would recommend
that do not waste your time for improving it as one image. It is a 3-band
image of sixe 128478 by 133220 pixels and thus as uncompressed it would make
about 50 gigabytes. The current file size 450 GB is not bad especially
because the image includes a good set of overviews. Image also opens and
works fine with QGIS.

I have no ready made recipe for automatic selection and extracting of areas
which contain data. I guess that I would have a try by vectorizing the
raster files with http://www.gdal.org/gdal_polygonize.html first and build
windows for gdal_translate and -projwin based on the vector data. It could
be interesting as an exercise but probably there should be hundreds of
images before it really saves enough resources for being a good investment.

-Jukka Rahkonen-





--
Sent from: http://osgeo-org.1560.x6.nabble.com/GDAL-Dev-f3742093.html
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev