Mixed Raw and Overview data compression ?

Next Topic
 
classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Mixed Raw and Overview data compression ?

Vincent Sarago
What do I mean about `mixed` compression ?
JPEG (or WEBP) compression are great for storage saving (and also for small http request) but they are `bad` because they are not lossless (WEBP can be set to be lossless but compression is then not as great) so my idea is to package both lossless and non-lossless in one COG. e.g keeping high resolution data to a ZSTD/DEFLATE compression but the overview in JPEG.

Why ?
For web application we mostly want fast tile serving while for processing we will mostly want to read full resolution lossless data, a file with mixed compression could make both world happy ?

The gain won’t be astonishing but might be worth a try ?

Vincent
_______________________________________________
COG mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/cog
Reply | Threaded
Open this post in threaded view
|

Re: Mixed Raw and Overview data compression ?

Kirill Kouzoubov
This would be very useful, reducing storage costs while simultaneously allowing faster preview functionality. One issue to consider is `dtype` support/conversion. My understanding is that JPEG is best supported at 8 bit, and can not go beyond 12, while many sensors have more than 8 bits worth of data per band, and even more than 12. So one would need bit-depth reduction as well. To complicated things further, one might have 14-bit data stored in 16-bit image, so bit-depth reduction is not a trivial operation. Also `nodata` comes into play as well.


Kirill 

On Mon, Sep 17, 2018 at 10:39 AM Vincent Sarago <[hidden email]> wrote:
What do I mean about `mixed` compression ?
JPEG (or WEBP) compression are great for storage saving (and also for small http request) but they are `bad` because they are not lossless (WEBP can be set to be lossless but compression is then not as great) so my idea is to package both lossless and non-lossless in one COG. e.g keeping high resolution data to a ZSTD/DEFLATE compression but the overview in JPEG.

Why ?
For web application we mostly want fast tile serving while for processing we will mostly want to read full resolution lossless data, a file with mixed compression could make both world happy ?

The gain won’t be astonishing but might be worth a try ?

Vincent
_______________________________________________
COG mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/cog

_______________________________________________
COG mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/cog
Reply | Threaded
Open this post in threaded view
|

Re: Mixed Raw and Overview data compression ?

Vincent Sarago

Le 16 sept. 2018 à 21:26, Kirill Kouzoubov <[hidden email]> a écrit :

This would be very useful, reducing storage costs while simultaneously allowing faster preview functionality.


One issue to consider is `dtype` support/conversion. My understanding is that JPEG is best supported at 8 bit, and can not go beyond 12, while many sensors have more than 8 bits worth of data per band, and even more than 12.

Yeah absolutely, so this would be only for 8bit/12bit RGB dataset (noting that enabling different photogrammetric configuration could save more space too for YCBCR+JPEG). 

 So one would need bit-depth reduction as well. To complicated things further, one might have 14-bit data stored in 16-bit image, so bit-depth reduction is not a trivial operation.

Yeah I don’t want to overthink this, and create complex dataset (data difference between overview and high res, just thinking about compression) 

Also `nodata` comes into play as well.

I think using internal bitmask is a better solution than nodata IMO but that’s another subject :D . 

Vincent 


Kirill 

On Mon, Sep 17, 2018 at 10:39 AM Vincent Sarago <[hidden email]> wrote:
What do I mean about `mixed` compression ?
JPEG (or WEBP) compression are great for storage saving (and also for small http request) but they are `bad` because they are not lossless (WEBP can be set to be lossless but compression is then not as great) so my idea is to package both lossless and non-lossless in one COG. e.g keeping high resolution data to a ZSTD/DEFLATE compression but the overview in JPEG.

Why ?
For web application we mostly want fast tile serving while for processing we will mostly want to read full resolution lossless data, a file with mixed compression could make both world happy ?

The gain won’t be astonishing but might be worth a try ?

Vincent
_______________________________________________
COG mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/cog


_______________________________________________
COG mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/cog
Reply | Threaded
Open this post in threaded view
|

Re: Mixed Raw and Overview data compression ?

Pfister, Marc
In reply to this post by Vincent Sarago
On Sun, 16 Sep 2018 20:39:32 -0400 Vincent Sarago <[hidden email]> wrote:

> JPEG (or WEBP) compression are great for storage saving (and also for small http request)
> but they are `bad` because they are not lossless (WEBP can be set to be lossless but compression
> is then not as great) so my idea is to package both lossless and non-lossless in one COG. e.g
> keeping high resolution data to a ZSTD/DEFLATE compression but the overview in JPEG.

Setting aside the issues of bit depth, can the current tools (namely GDAL) handle mixed compression types?

I've also been pondering how far can you "abuse" the concept of overviews with different compression, bit depth, band combinations, and so on. So I am interested to see how this works out even at a basic level.

- Marc
_______________________________________________
COG mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/cog
Reply | Threaded
Open this post in threaded view
|

Re: Mixed Raw and Overview data compression ?

Even Rouault-2
On mardi 18 septembre 2018 14:45:49 CEST Pfister, Marc wrote:
> On Sun, 16 Sep 2018 20:39:32 -0400 Vincent Sarago <[hidden email]>
wrote:
> > JPEG (or WEBP) compression are great for storage saving (and also for
> > small http request) but they are `bad` because they are not lossless
> > (WEBP can be set to be lossless but compression is then not as great) so
> > my idea is to package both lossless and non-lossless in one COG. e.g
> > keeping high resolution data to a ZSTD/DEFLATE compression but the
> > overview in JPEG.
> Setting aside the issues of bit depth, can the current tools (namely GDAL)
> handle mixed compression types?

It should have no issue in reading a COG with mixed compression types.
But it cannot currently generate a GeoTIFF with different compression types
for full resolution imagery and overviews. There are no particular fundamental
obstacles to do it if it had to be implemented.

>
> I've also been pondering how far can you "abuse" the concept of overviews
> with different compression, bit depth, band combinations, and so on. So I
> am interested to see how this works out even at a basic level.

GDAL will require all overview levels to have identical band count and bit
depth.

Even

--
Spatialys - Geospatial professional services
http://www.spatialys.com
_______________________________________________
COG mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/cog