[gdal-dev] Compression Artifacts After Merging Imagery

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

[gdal-dev] Compression Artifacts After Merging Imagery

Jacob Adams
Oh ye great minds of the GDAL world, I've run into a problem that I've not been able to solve.

I have aerial imagery comprised of about 1500 jpegs that I am trying to merge together into three large (~18gb) jpeg-compressed geotiffs to be served by Geoserver. The images are tiled such that they shift down about 100 pixels (out of ~7000) every so often. With a pure gdal_merge or gdalwarp mosaic, this leaves compression artifacts in the nodata spaces along the borders. The cause of these artifacts seems to be explained in Evan's comments on this ticket: https://trac.osgeo.org/gdal/ticket/5102).  

I tried pre-processing a handful of the tilesinto geotiffs with the following three commands based on a similar discussion in February (http://lists.osgeo.org/pipermail/gdal-dev/2017-February/046027.html):
    nearblack -co TILED=YES -of Gtiff -nb 0 -near 0 -setmask -o temp1.tif Tile1.jpeg
    gdalwarp -t_srs EPSG:3560 -dstalpha -co TILED=YES temp1.tif temp2.tif
    gdal_translate -co TILED=YES -co COMPRESS=JPEG -co PHOTOMETRIC=YCBCR -b 1 -b 2 -b 3 -mask 4 --config GDAL_TIFF_INTERNAL_MASK YES temp2.tif Tile1_out.tif

However, I still get the artifacts when I then try to mosaic the resulting tiffs using either gdalwarp or gdal_merge.py:
    python gdal_merge.py -co TILED=YES -co COMPRESS=JPEG -co PHOTOMETRIC=YCBCR --config GDAL_TIFF_INTERNAL_MASK YES -o merged.tif Tile1_out.tif Tile2_out.tif (etc)

Leaving the compression out of the gdal_translate call (leaving all the compression to the gdal_merge command) does not make a difference in the artifacts.

I've also tried to merge the images and cut the resulting compressed tiffs with cutlines,:
    python gdal_merge.py -ot Byte -of Gtiff -co COMPRESS=JPEG -co PHOTOMETRIC=YCBCR -co TILED=YES -co BIGTIFF=YES -o "f:\imagery\trimmed_merged_jpegs\0.tif" --optfile f:\imagery\0.txt
    gdalwarp.exe -co COMPRESS=JPEG -co PHOTOMETRIC=YCBCR -co TILED=YES -co BIGTIFF=YES -cutline f:\imagery_footprints\threeinch0.shp -crop_to_cutline "f:\imagery\trimmed_merged_jpegs\0.tif" "f:\imagery\trimmed_merged_jpegs\trimmed0.tif"

But this also fails to remove the artifacts.

My understanding of the problem is that the jpeg compression occurs as the last step of any command, and thus even when fed a properly-masked file the final compression still mangles the output.

Is there a way to remove these artifacts in a format amenable to Geoserver while preserving the compression levels achievable with ycbcr-based jpeg compression?

Thanks,

Jacob Adams
GIS Specialist, Cache County
 


_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Compression Artifacts After Merging Imagery

Even Rouault-2

On lundi 25 septembre 2017 16:16:26 CEST Jacob Adams wrote:

> Oh ye great minds of the GDAL world, I've run into a problem that I've not

> been able to solve.

>

> I have aerial imagery comprised of about 1500 jpegs that I am trying to

> merge together into three large (~18gb) jpeg-compressed geotiffs to be

> served by Geoserver. The images are tiled such that they shift down about

> 100 pixels (out of ~7000) every so often. With a pure gdal_merge or

> gdalwarp mosaic, this leaves compression artifacts in the nodata spaces

> along the borders. The cause of these artifacts seems to be explained in

> Evan's comments on this ticket: https://trac.osgeo.org/gdal/ticket/5102).

>

> I tried pre-processing a handful of the tilesinto geotiffs with the

> following three commands based on a similar discussion in February

> (http://lists.osgeo.org/pipermail/gdal-dev/2017-February/046027.html):

 

> nearblack -co TILED=YES -of Gtiff -nb 0 -near 0 -setmask -o temp1.tif Tile1.jpeg

 

I think one issue is in the above nearblack invocation. Did you check temp1.tif is "clean" by displaying it ?

Basically by setting -near 0, you say that nearblack should expect a very clean frontier between nodata areas and valid areas. Whereas JPEG decompression artifacts in such a situation will result in intermediate values in a few pixels beyond the border.

I'd suggest trying with the default of nearblack, -near 15 and -nb 2, and tune them if needed.

 

 

> gdal_translate -co TILED=YES -co COMPRESS=JPEG -co PHOTOMETRIC=YCBCR -b 1 -b 2 -b 3 -mask 4 --config GDAL_TIFF_INTERNAL_MASK YES temp2.tif Tile1_out.tif

 

You shouldn't JPEG compress intermediate products. This will decrease image quality. Only apply it to the final stage of your workflow.

 

 

Even

 

--

Spatialys - Geospatial professional services

http://www.spatialys.com


_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Compression Artifacts After Merging Imagery

Jacob Adams
Evan,

Thanks for your help. Adjusting the nearblack parameters and using
-setalpha instead of -setmask got the mask working as expected.

I'd normally agree with you on mid-process compression, and if I were
using the data for any kind of analysis I would do as little compression
as possible. However, with this project I'm pressed for space and I'm
only trying to create an aerial imagery basemap. I did a quick test
compressing a single tile 10 times with gdal_translate and the output
was adequate for my project. There was an identifiable softening to the
image at the pixel-level, but the imagery was gathered at 3" per pixel.


On a different topic altogether, how would I go about contributing to
the doxygen-created documentation? In playing around with gdal_rasterize
I discovered that it's not clear in the current docs what options
trigger creation of a new raster instead of overwriting an existing
raster. My C++ skills are very rusty, but if I'm reading the source code
properly, the options -of, -init, -a_nodata, -a_srs, -te, -a_ullr, -co,
-ot, -ts, -outsize, -tr, and -tap set the bCreateOutput flag True. I'd
like to add a note explaining this so that someone else doesn't make the
same beginner's mistake I did of trying to specify creation options on
an existing dataset ;) Is this something I open a ticket for, or is
there a more direct way?

Thanks,

Jacob Adams
GIS Specialist, Cache County
 


>>> Even Rouault <[hidden email]> 9/25/2017 5:01 PM >>>
On lundi 25 septembre 2017 16:16:26 CEST Jacob Adams wrote:
> Oh ye great minds of the GDAL world, I've run into a problem that
I've not
> been able to solve.
>
> I have aerial imagery comprised of about 1500 jpegs that I am trying
to
> merge together into three large (~18gb) jpeg‑compressed geotiffs to
be
> served by Geoserver. The images are tiled such that they shift down
about
> 100 pixels (out of ~7000) every so often. With a pure gdal_merge or
> gdalwarp mosaic, this leaves compression artifacts in the nodata
spaces
> along the borders. The cause of these artifacts seems to be explained
in
> Evan's comments on this ticket:
https://trac.osgeo.org/gdal/ticket/5102).
>
> I tried pre‑processing a handful of the tilesinto geotiffs with the
> following three commands based on a similar discussion in February
>
(http://lists.osgeo.org/pipermail/gdal‑dev/2017‑February/046027.html):


> nearblack ‑co TILED=YES ‑of Gtiff ‑nb 0 ‑near 0 ‑setmask ‑o temp1.tif
 Tile1.jpeg

I think one issue is in the above nearblack invocation. Did you check
temp1.tif is "clean" by
displaying it ?
Basically by setting ‑near 0, you say that nearblack should expect a
very clean frontier
between nodata areas and valid areas. Whereas JPEG decompression
artifacts in such a
situation will result in intermediate values in a few pixels beyond the
border.
I'd suggest trying with the default of nearblack, ‑near 15 and ‑nb 2,
and tune them if needed.


>     gdal_translate ‑co TILED=YES ‑co COMPRESS=JPEG ‑co
PHOTOMETRIC=YCBCR ‑b 1 ‑b 2 ‑b
3 ‑mask 4 ‑‑config GDAL_TIFF_INTERNAL_MASK YES temp2.tif Tile1_out.tif


You shouldn't JPEG compress intermediate products. This will decrease
image quality. Only
apply it to the final stage of your workflow.


Even

‑‑
Spatialys ‑ Geospatial professional services
http://www.spatialys.com
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Compression Artifacts After Merging Imagery

Even Rouault-2

> On a different topic altogether, how would I go about contributing to

> the doxygen-created documentation? In playing around with gdal_rasterize

> I discovered that it's not clear in the current docs what options

> trigger creation of a new raster instead of overwriting an existing

> raster. My C++ skills are very rusty, but if I'm reading the source code

> properly, the options -of, -init, -a_nodata, -a_srs, -te, -a_ullr, -co,

> -ot, -ts, -outsize, -tr, and -tap set the bCreateOutput flag True. I'd

> like to add a note explaining this so that someone else doesn't make the

> same beginner's mistake I did of trying to specify creation options on

> an existing dataset ;) Is this something I open a ticket for, or is

> there a more direct way?

 

A ticket is fine. The changes are to be done in apps/gdal_utilities.dox

 

Even

 

--

Spatialys - Geospatial professional services

http://www.spatialys.com


_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev