Tiff and JPEG2000 - COW

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Tiff and JPEG2000 - COW

Norman Barker-2
I am interested in the discussions comparing COG with JPEG2000.

To start with I (with the Mapbox Satellite team, in particular, Vincent Sarago) wrote https://github.com/mapbox/COGDumper to refresh our memory of the TIFF spec. This highlighted an issue with JPEG compressed tiles and masks inside a COG. In that to be effective over the wire the masks needs to be adjacent to the tile data.


When I first saw COG I thought  why not Compressed Optimized Wavelets (COW)? However you are not really comparing apples with apples here.

ISO JPEG2000 part 9 is JPIP, an internet protocol for stateful transfer of JPEG2000 data over potentially choppy network connections (very low bandwidth, contention, bursts, dropout etc). This standard also include stateless requests but is centred around a server model.

JPEG2000 Wavelets are progressive in resolution, spatial (X/Y) and quality layers. Due the nature of the stream they are also fault tolerant to missing bytes in transmission. This isn't the case with TIFF, TIFF is progressive in X/Y and if you are missing bytes then the tile isn't going to render. It is to be seen at what (low, or high number of users) bandwidths COG is effective.

COG is about tiled data; JPEG2000 has a concept of a tile, but to be effective over limited networks uses precincts (a specified group of code blocks). As precinct edges overlap with adjacent edges you will get a fuzzy edge until the adjacent precinct is retrieved, which is fine as it is still less data than retrieving a complete tile.

I have more to come on this as I am currently creating a library with the Satellite team at Mapbox to explore COW/COG in more detail. In a similar way to COG it should be possible to create a JPEG2000 profile that specifies the codestream layout in a way that the JPEG2000 data can be retrieved efficiently with HTTP requests and does not require a server.

While this work is still experimental, we see a number of clear benefits to COW over HTTP:

* Progressive in spatial position, resolution and quality
* Region of Interest encoding
* Bandwidth savings
* Finer control over region access

The biggest stumbling block with JPEG2000 has always been understanding the core spec which is very complex. I hope to expand on this over the next few months with some examples.

Norman



_______________________________________________
COG mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/cog
Reply | Threaded
Open this post in threaded view
|

Re: Tiff and JPEG2000 - COW

Even Rouault-2

Norman,

 

> While this work is still experimental, we see a number of clear benefits to

> COW over HTTP:

>

> * Progressive in spatial position, resolution and quality

 

As far as I understand JPEG2000, the issue is that you can't have at the same time fast spatial sub-window access and fast overview access in the same file layout. I believe the former is obtained with PCRL (Position-Component-Resolution-Layer) progression order (but even with that, I believe you'd need several non-contiguous ranges as the precincts corresponding to the small overview level are common to several spatial regions), and the later with RPCL (Resolution-Position-Component-Layer)

 

> * Region of Interest encoding

 

Is there a real use case for this ? If so, that could also be done with JPEG-in-TIFF. You can use different JPEG quality tables for each TIFF tile.

 

>

> The biggest stumbling block with JPEG2000 has always been understanding the

> core spec which is very complex.

 

Indeed. Having spent several weeks trying to improve OpenJPEG performance last summer, this involves huge headaches. Currently one big limitation of openjpeg regarding to what you mention above is that the lib needs to ingest the whole codestream of a JPEG2000 tile (so the whole file for a single-tiled JPEG2000 file). This could perhaps be enhanced, but this is a non trivial work. But this would have to be done since COW without an open source implementation wouldn't get much traction.

 

To check quickly if COW is something that has a potential, I'd suggest you try with the GDAL JP2KAK driver and /vsicurl/ and look at the range requests done.

 

Another issue is that even with the best implementation, the CPU time needed to decode JPEG2000 is way higher than with JPEG.

 

Even

 

--

Spatialys - Geospatial professional services

http://www.spatialys.com


_______________________________________________
COG mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/cog
Reply | Threaded
Open this post in threaded view
|

Re: Tiff and JPEG2000 - COW

Norman Barker-2
Hi Even,

thanks for the comments. I have put some more detail below.


On Fri, May 18, 2018 at 12:58 PM, Even Rouault <[hidden email]> wrote:

Norman,

 

> While this work is still experimental, we see a number of clear benefits to

> COW over HTTP:

>

> * Progressive in spatial position, resolution and quality

 

As far as I understand JPEG2000, the issue is that you can't have at the same time fast spatial sub-window access and fast overview access in the same file layout. I believe the former is obtained with PCRL (Position-Component-Resolution-Layer) progression order (but even with that, I believe you'd need several non-contiguous ranges as the precincts corresponding to the small overview level are common to several spatial regions), and the later with RPCL (Resolution-Position-Component-Layer)


I spent a lot of time working with JPIP and in my opinion it comes down to how you define fast. One contiguous download isn't necessarily fast. Several small downloads of packets may be faster. Add to that for each zoom level with COG you are downloading a new tile as opposed to incremental updates.

However I think there are a narrower subset of use cases for the complexity of COW vs COG.
 

 

> * Region of Interest encoding

 

Is there a real use case for this ? If so, that could also be done with JPEG-in-TIFF. You can use different JPEG quality tables for each TIFF tile.


ROI with J2K is sub-tile and tile boundaries with TIFF may cause artifacts. There are two use cases, obscuration and region enhancement at lower resolution levels. An example is the display use case of drawing the eye to a region of an image to zoom in on. 

 

>

> The biggest stumbling block with JPEG2000 has always been understanding the

> core spec which is very complex.

 

Indeed. Having spent several weeks trying to improve OpenJPEG performance last summer, this involves huge headaches. Currently one big limitation of openjpeg regarding to what you mention above is that the lib needs to ingest the whole codestream of a JPEG2000 tile (so the whole file for a single-tiled JPEG2000 file). This could perhaps be enhanced, but this is a non trivial work. But this would have to be done since COW without an open source implementation wouldn't get much traction.

 

To check quickly if COW is something that has a potential, I'd suggest you try with the GDAL JP2KAK driver and /vsicurl/ and look at the range requests done.


right, I did some quick experiments with the kakadu demo executables and creating different codestreams as I no longer have the kakadu sdk. It was some time ago now but I wrote the JPIPKAK driver.

I am currently evaluating this with Python and Numpy, but no commitment there on whether this will be a full library, as you say J2K is full of headaches.
  

 

Another issue is that even with the best implementation, the CPU time needed to decode JPEG2000 is way higher than with JPEG.


Understood, I think this needs to be measured and is one of my goals. The use case of progressive updates might be preferable for a user in an interactive use case.

 

 

Even

 

--

Spatialys - Geospatial professional services

http://www.spatialys.com



_______________________________________________
COG mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/cog
Reply | Threaded
Open this post in threaded view
|

Re: Tiff and JPEG2000 - COW

Even Rouault-2

> right, I did some quick experiments with the kakadu demo executables and

> creating different codestreams as I no longer have the kakadu sdk.

 

With the vsipreload hack [1] from GDAL, you can also use the kakadu demo executables with /vsicurl

 

Quick recipee (Linux only):

 

Checkout and build GDAL

cd port/

g++ -fPIC -std=c++11 vsipreload.cpp -shared -o vsipreload.so -lgdal

CPL_CURL_VERBOSE=YES LD_PRELOAD=vsipreload.so kdu_expand -i /vsicurl/http://download.osgeo.org/gdal/data/jpeg2000/utm.jp2 -o out.tif

 

Even

 

[1] https://lists.osgeo.org/pipermail/gdal-dev/2013-May/036324.html

 

--

Spatialys - Geospatial professional services

http://www.spatialys.com


_______________________________________________
COG mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/cog
Reply | Threaded
Open this post in threaded view
|

Cloud-ready GeoTiff Basic Functionality

Brian M Hamlin
Great topic - ready for good discussion here --
 
 For an HTTP-visible GeoTIFF today, is this roughly the idea ?
 
* prepare the GeoTIFF with embedded overviews, for example
  gdaladdo file0.tif  ....  2 4 8 16
 
* process the GeoTIFF to tiled
  gdal_translate -lco TILED=YES -lco COPY_SRC_OVERVIEWS=YES
-lco COMPRESS=whichever src dst
 
Reliable HTTP client CURL to test byte-range access
 
  curl -v -X GET -H "range: bytes=-11" <file_URL>
 
more?
 
  best regards from Berkeley, California
   --Brian
  

_______________________________________________
COG mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/cog
Reply | Threaded
Open this post in threaded view
|

Re: Cloud-ready GeoTiff Basic Functionality

Brad Hards
> For an HTTP-visible GeoTIFF today, is this roughly the idea ?
See https://trac.osgeo.org/gdal/wiki/CloudOptimizedGeoTIFF for the current definition (and recommendations, especially on the COMPRESS layer creation option).

Brad

_______________________________________________
COG mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/cog