[gdal-dev] Problem with black edges to DOQQs using JPEG in Tiff compression

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

[gdal-dev] Problem with black edges to DOQQs using JPEG in Tiff compression

Stephen Woodbridge
Hi All,

I need your wisdom. I'm downloading NAIP DOQQs in GTiff format and I
have a processing chain something like the following:

gdalwarp -t_srs EPSG:4326 -dstalpha -r "bilinear -multi -co TILED=YES
-dstnodata '0 0 0' srctiff tmpfile

nearblack -nb 15 -q tmpfile

gdal_translate -co TILED=YES -co JPEG_QUALITY=90 -co COMPRESS=JPEG -co
PHOTOMETRIC=YCBCR -b 1 -b 2 -b 3 -mask auto --config
GDAL_TIFF_INTERNAL_MASK YES  tmpfile, target

nearblack -nb 5 -q target

gdaladdo -clean -r bilinear  target 2 4 8 16 32 64 128 512

And create a tileindex for mapserver of all the tiffs

If I skip the gdal_translate (ie: JPEG compression) and the 2nd
nearblack, the doqq tiles are perfect with no nearblack edges between
the doqq tiles. But when a JPEG compress them, I get edges between the
doqqs like this:

http://imaptools.com:8080/dl/doqq-issue.jpg

I've never used the JPEG in tiff compression and I'm very impressed by
the amount of size reduction there and how good the image remains, but I
have not been able to figure out the magic trick to clearing the edge
artifacts.

Any help would be appreciated.

Thanks,
   -Steve

---
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus

_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Problem with black edges to DOQQs using JPEG in Tiff compression

Even Rouault-2

On samedi 11 février 2017 18:18:35 CET Stephen Woodbridge wrote:

> Hi All,

>

> I need your wisdom. I'm downloading NAIP DOQQs in GTiff format and I

> have a processing chain something like the following:

>

> gdalwarp -t_srs EPSG:4326 -dstalpha -r "bilinear -multi -co TILED=YES

> -dstnodata '0 0 0' srctiff tmpfile

>

> nearblack -nb 15 -q tmpfile

>

> gdal_translate -co TILED=YES -co JPEG_QUALITY=90 -co COMPRESS=JPEG -co

> PHOTOMETRIC=YCBCR -b 1 -b 2 -b 3 -mask auto --config

> GDAL_TIFF_INTERNAL_MASK YES tmpfile, target

>

> nearblack -nb 5 -q target

>

> gdaladdo -clean -r bilinear target 2 4 8 16 32 64 128 512

>

> And create a tileindex for mapserver of all the tiffs

>

> If I skip the gdal_translate (ie: JPEG compression) and the 2nd

> nearblack, the doqq tiles are perfect with no nearblack edges between

> the doqq tiles. But when a JPEG compress them, I get edges between the

> doqqs like this:

>

> http://imaptools.com:8080/dl/doqq-issue.jpg

>

> I've never used the JPEG in tiff compression and I'm very impressed by

> the amount of size reduction there and how good the image remains, but I

> have not been able to figure out the magic trick to clearing the edge

> artifacts.

 

Steve,

 

I managed to replicate something similar to the above with an image I've at hand.

The cause of the issue is the nearblack invokation after the gdal_translate. The effect of this nearblack is to "eat" some pixels at the border of the validity / invalidity transition, but that doesn't update the existing mask.

If you add -setmask to this nearblack, that should fix it.

Even better, I think you can just remove this second nearblack.

 

And you probably must add -setalpha for the first nearblack invokation as well.

 

Even

 

--

Spatialys - Geospatial professional services

http://www.spatialys.com


_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Problem with black edges to DOQQs using JPEG in Tiff compression

Stephen Woodbridge
On 2/12/2017 9:54 AM, Even Rouault wrote:

> On samedi 11 février 2017 18:18:35 CET Stephen Woodbridge wrote:
>
>> Hi All,
>
>>
>
>> I need your wisdom. I'm downloading NAIP DOQQs in GTiff format and I
>
>> have a processing chain something like the following:
>
>>
>
>> gdalwarp -t_srs EPSG:4326 -dstalpha -r "bilinear -multi -co TILED=YES
>
>> -dstnodata '0 0 0' srctiff tmpfile
>
>>
>
>> nearblack -nb 15 -q tmpfile
>
>>
>
>> gdal_translate -co TILED=YES -co JPEG_QUALITY=90 -co COMPRESS=JPEG -co
>
>> PHOTOMETRIC=YCBCR -b 1 -b 2 -b 3 -mask auto --config
>
>> GDAL_TIFF_INTERNAL_MASK YES tmpfile, target
>
>>
>
>> nearblack -nb 5 -q target
>
>>
>
>> gdaladdo -clean -r bilinear target 2 4 8 16 32 64 128 512
>
>>
>
>> And create a tileindex for mapserver of all the tiffs
>
>>
>
>> If I skip the gdal_translate (ie: JPEG compression) and the 2nd
>
>> nearblack, the doqq tiles are perfect with no nearblack edges between
>
>> the doqq tiles. But when a JPEG compress them, I get edges between the
>
>> doqqs like this:
>
>>
>
>> http://imaptools.com:8080/dl/doqq-issue.jpg
>
>>
>
>> I've never used the JPEG in tiff compression and I'm very impressed by
>
>> the amount of size reduction there and how good the image remains, but I
>
>> have not been able to figure out the magic trick to clearing the edge
>
>> artifacts.
>
>
>
> Steve,
>
>
>
> I managed to replicate something similar to the above with an image I've
> at hand.
>
> The cause of the issue is the nearblack invokation after the
> gdal_translate. The effect of this nearblack is to "eat" some pixels at
> the border of the validity / invalidity transition, but that doesn't
> update the existing mask.
>
> If you add -setmask to this nearblack, that should fix it.
>
> Even better, I think you can just remove this second nearblack.
>
>
>
> And you probably must add -setalpha for the first nearblack invokation
> as well.

Even,

Thanks for the quick feedback. I'll will give these suggestions and try
today and let you know how it goes.

Thanks,
   -Steve


---
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus

_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Problem with black edges to DOQQs using JPEG in Tiff compression

Brian Case
In reply to this post by Stephen Woodbridge
Steve,

first, if your data is non lossy you need to nearblack -nb 0 -near 15

however your DOQQ's dont have a collar so there is no reason to
nearblack

gdalwarp -dstalpha will crate an alpha band to mask out the collar after
gdalwarp rotates the image. no nead to nearblack the output of gdal
warp.

when using gdal_translate to create a tiff with a mask band, and you
warped file has a alpha band, gdal_translate -b 1 -b 2 -b 3 -mask 4
this will crate a mask band from the alpha band

brian

On Sat, 2017-02-11 at 18:18 -0500, Stephen Woodbridge wrote:

> Hi All,
>
> I need your wisdom. I'm downloading NAIP DOQQs in GTiff format and I
> have a processing chain something like the following:
>
> gdalwarp -t_srs EPSG:4326 -dstalpha -r "bilinear -multi -co TILED=YES
> -dstnodata '0 0 0' srctiff tmpfile
>
> nearblack -nb 15 -q tmpfile
>
> gdal_translate -co TILED=YES -co JPEG_QUALITY=90 -co COMPRESS=JPEG -co
> PHOTOMETRIC=YCBCR -b 1 -b 2 -b 3 -mask auto --config
> GDAL_TIFF_INTERNAL_MASK YES  tmpfile, target
>
> nearblack -nb 5 -q target
>
> gdaladdo -clean -r bilinear  target 2 4 8 16 32 64 128 512
>
> And create a tileindex for mapserver of all the tiffs
>
> If I skip the gdal_translate (ie: JPEG compression) and the 2nd
> nearblack, the doqq tiles are perfect with no nearblack edges between
> the doqq tiles. But when a JPEG compress them, I get edges between the
> doqqs like this:
>
> http://imaptools.com:8080/dl/doqq-issue.jpg
>
> I've never used the JPEG in tiff compression and I'm very impressed by
> the amount of size reduction there and how good the image remains, but I
> have not been able to figure out the magic trick to clearing the edge
> artifacts.
>
> Any help would be appreciated.
>
> Thanks,
>    -Steve
>
> ---
> This email has been checked for viruses by Avast antivirus software.
> https://www.avast.com/antivirus
>
> _______________________________________________
> gdal-dev mailing list
> [hidden email]
> https://lists.osgeo.org/mailman/listinfo/gdal-dev


_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Problem with black edges to DOQQs using JPEG in Tiff compression

Brian Case
steve and all

after reading my reply  I noticed a major error

"first, if your data is non lossy you need to nearblack -nb 0 -near 15"



this should read "nearblack -nb 0 -near 0"


Brian



On Sun, 2017-02-12 at 13:16 -0800, Brian Case wrote:

> Steve,
>
> first, if your data is non lossy you need to nearblack -nb 0 -near 15
>
> however your DOQQ's dont have a collar so there is no reason to
> nearblack
>
> gdalwarp -dstalpha will crate an alpha band to mask out the collar after
> gdalwarp rotates the image. no nead to nearblack the output of gdal
> warp.
>
> when using gdal_translate to create a tiff with a mask band, and you
> warped file has a alpha band, gdal_translate -b 1 -b 2 -b 3 -mask 4
> this will crate a mask band from the alpha band
>
> brian
>
> On Sat, 2017-02-11 at 18:18 -0500, Stephen Woodbridge wrote:
> > Hi All,
> >
> > I need your wisdom. I'm downloading NAIP DOQQs in GTiff format and I
> > have a processing chain something like the following:
> >
> > gdalwarp -t_srs EPSG:4326 -dstalpha -r "bilinear -multi -co TILED=YES
> > -dstnodata '0 0 0' srctiff tmpfile
> >
> > nearblack -nb 15 -q tmpfile
> >
> > gdal_translate -co TILED=YES -co JPEG_QUALITY=90 -co COMPRESS=JPEG -co
> > PHOTOMETRIC=YCBCR -b 1 -b 2 -b 3 -mask auto --config
> > GDAL_TIFF_INTERNAL_MASK YES  tmpfile, target
> >
> > nearblack -nb 5 -q target
> >
> > gdaladdo -clean -r bilinear  target 2 4 8 16 32 64 128 512
> >
> > And create a tileindex for mapserver of all the tiffs
> >
> > If I skip the gdal_translate (ie: JPEG compression) and the 2nd
> > nearblack, the doqq tiles are perfect with no nearblack edges between
> > the doqq tiles. But when a JPEG compress them, I get edges between the
> > doqqs like this:
> >
> > http://imaptools.com:8080/dl/doqq-issue.jpg
> >
> > I've never used the JPEG in tiff compression and I'm very impressed by
> > the amount of size reduction there and how good the image remains, but I
> > have not been able to figure out the magic trick to clearing the edge
> > artifacts.
> >
> > Any help would be appreciated.
> >
> > Thanks,
> >    -Steve
> >
> > ---
> > This email has been checked for viruses by Avast antivirus software.
> > https://www.avast.com/antivirus
> >
> > _______________________________________________
> > gdal-dev mailing list
> > [hidden email]
> > https://lists.osgeo.org/mailman/listinfo/gdal-dev
>
>
> _______________________________________________
> gdal-dev mailing list
> [hidden email]
> https://lists.osgeo.org/mailman/listinfo/gdal-dev


_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Problem with black edges to DOQQs using JPEG in Tiff compression

jratike80
In reply to this post by Stephen Woodbridge
Stephen Woodbridge wrote
Even,
Thanks for the quick feedback. I'll will give these suggestions and try
today and let you know how it goes.

Thanks,
   -Steve
Hi Steve,

They are probably good suggestions but folks did not bother to think what is your ultimate target. It is not to improve you commands and hide the black pixels but simply to make a jpeg-in-tiff compressed mosaic without seams for Mapserver, perhaps following some other route if it is easier.

So, my suggestion is:

1) Skip gdalwarp
2) Compress your images in native projection
gdal_translate -co TILED=YES -co JPEG_QUALITY=90 -co COMPRESS=JPEG -co PHOTOMETRIC=YCBCR
3) Create jpeg compressed overviews
gdaladdo -r average --config COMPRESS_OVERVIEW JPEG --config PHOTOMETRIC_OVERVIEW YCBCR --config INTERLEAVE_OVERVIEW PIXEL 2 4 8 16 32 64
4) Create tileindex and utilize the super powerful "Tileindexes with tiles in different projections" feature http://www.mapserver.org/optimization/tileindex.html

You did not mention if your DOQQs come in several different projections but I suppose that they do. Otherwise there is even less point to use gdalwarp, Mapserver is so fast with on-the-fly re-projecting.

-Jukka Rahkonen-

Reply | Threaded
Open this post in threaded view
|

Re: Problem with black edges to DOQQs using JPEG in Tiff compression

Stephen Woodbridge
Jukka, Thanks for your suggestion. We have a couple of requirements
mapserver is only one, the other is some additional processing, but I
like the simplicity of your suggestion.

Brian C. spend a lot of time on irc with me and I think we finally got
the processing chain straightened out and it looks like:

# first preclip any collar (probably not needed)
nearblack -co TILED=YES -of GTiff -nb 0 -near 0 -setmask -q -o tmpfile1
filepath

gdalwarp -t_srs EPSG:4326 -dstalpha -co TILED=YES tmpfile1 tmpfile2

gdal_translate -co TILED=YES -co JPEG_QUALITY=90 -co COMPRESS=JPEG -co
PHOTOMETRIC=YCBCR -b 1 -b 2 -b 3 -mask 4 --config
GDAL_TIFF_INTERNAL_MASK YES tmpfile2 target

gdaladdo -clean -r bilinear target 2 4 8 16 32 64 128

This is pretty fast taking about 9-10 sec per doqq to process.

Thank you for all the suggestions and help.

-Steve

On 2/13/2017 5:02 AM, jratike80 wrote:

> Stephen Woodbridge wrote
>> Even,
>> Thanks for the quick feedback. I'll will give these suggestions and try
>> today and let you know how it goes.
>>
>> Thanks,
>>    -Steve
>
> Hi Steve,
>
> They are probably good suggestions but folks did not bother to think what is
> your ultimate target. It is not to improve you commands and hide the black
> pixels but simply to make a jpeg-in-tiff compressed mosaic without seams for
> Mapserver, perhaps following some other route if it is easier.
>
> So, my suggestion is:
>
> 1) Skip gdalwarp
> 2) Compress your images in native projection
>
> 3) Create jpeg compressed overviews
>
> 4) Create tileindex and utilize the super powerful "Tileindexes with tiles
> in different projections" feature
> http://www.mapserver.org/optimization/tileindex.html
>
> You did not mention if your DOQQs come in several different projections but
> I suppose that they do. Otherwise there is even less point to use gdalwarp,
> Mapserver is so fast with on-the-fly re-projecting.
>
> -Jukka Rahkonen-
>
>
>
>
>
> --
> View this message in context: http://osgeo-org.1560.x6.nabble.com/gdal-dev-Problem-with-black-edges-to-DOQQs-using-JPEG-in-Tiff-compression-tp5307551p5307662.html
> Sent from the GDAL - Dev mailing list archive at Nabble.com.
> _______________________________________________
> gdal-dev mailing list
> [hidden email]
> https://lists.osgeo.org/mailman/listinfo/gdal-dev
>


---
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus

_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev