Quantcast

[gdal-dev] Question on block size in the kakadu driver

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[gdal-dev] Question on block size in the kakadu driver

Kurt Schwehr-2
Frank, Even or anyone with experience with jp2k block sizes,

Any chance you could comment on the change I proposed here?


Is it reasonable to change this:

    if( nRasterXSize >= 2048 )
        nBlockXSize = 2048;
    else
        nBlockXSize = nRasterXSize;

    if( nRasterYSize >= 256 )  // <- 256
        nBlockYSize = 128;       // <- 128 ???
    else
        nBlockYSize = nRasterYSize;

Into:

         nBlockXSize = std::min(nRasterXSize, 2048);
         nBlockYSize = std::min(nRasterYSize, 128)

or

         nBlockXSize = std::min(nRasterXSize, 2048);
         nBlockYSize = std::min(nRasterYSize, 256)

Neither of those is the same behavior and the the jump from 256 back to 128 is not explained in the revision log.  Which behavior will have better performance for a average case or will it matter?   The difference will be between 262K cells or 524K cells.  With doubles, that will be 2 or 4 MB for a single block from one band.  I'm thinking 128 is likely going to work better with RAM caches.

_______________________________________________
gdal-dev mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Question on block size in the kakadu driver

Even Rouault-2

On jeudi 5 janvier 2017 07:18:45 CET Kurt Schwehr wrote:

> Frank, Even or anyone with experience with jp2k block sizes,

>

> Any chance you could comment on the change I proposed here?

>

> https://trac.osgeo.org/gdal/ticket/6764

>

> Is it reasonable to change this:

>

> if( nRasterXSize >= 2048 )

> nBlockXSize = 2048;

> else

> nBlockXSize = nRasterXSize;

>

> if( nRasterYSize >= 256 ) // <- 256

> nBlockYSize = 128; // <- 128 ???

> else

> nBlockYSize = nRasterYSize;

>

> Into:

>

> nBlockXSize = std::min(nRasterXSize, 2048);

> nBlockYSize = std::min(nRasterYSize, 128)

>

> or

>

> nBlockXSize = std::min(nRasterXSize, 2048);

> nBlockYSize = std::min(nRasterYSize, 256)

>

> Neither of those is the same behavior and the the jump from 256 back to 128

> is not explained in the revision log. Which behavior will have better

> performance for a average case or will it matter?

 

The code is a bit weird indeed. But the block size of smallish datasets has no real importance, so if you find it more reasable, switching the condition to if ( nRasterYSize >= 128 ) (or the equivalent std::min()) should have no adverse consequences.

 

Block size in the case of JPEG 2000 is really hard to come with. A smarter block size logic would depend on JPEG 2000 tiling, presence of precincts, etc... This also could depend on the read pattern. Would deserve extensive testing to come out with the ideal formula... In the case of JP2KAK, there's also a IRasterIO() implementation that causes the block size to not be used at all in typical read scenarios.

 

 

> The difference will be

> between 262K cells or 524K cells. With doubles, that will be 2 or 4 MB for

> a single block from one band.

 

FWIIW, JPEG2000 doesn't support double (at least not in the base standard)

 

 

--

Spatialys - Geospatial professional services

http://www.spatialys.com


_______________________________________________
gdal-dev mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Question on block size in the kakadu driver

Kurt Schwehr-2
Even,

Thanks!  I'm working on a change that will do:

nBlockXSize = std::min(nRasterXSize, 2048);

nBlockYSize = std::min(nRasterYSize, 128);


Getting a performance monitoring system setup is definitely on my todo list, but sadly still down a ways.


On Thu, Jan 5, 2017 at 7:40 AM, Even Rouault <[hidden email]> wrote:

On jeudi 5 janvier 2017 07:18:45 CET Kurt Schwehr wrote:

> Frank, Even or anyone with experience with jp2k block sizes,

>

> Any chance you could comment on the change I proposed here?

>

> https://trac.osgeo.org/gdal/ticket/6764

>

> Is it reasonable to change this:

>

> if( nRasterXSize >= 2048 )

> nBlockXSize = 2048;

> else

> nBlockXSize = nRasterXSize;

>

> if( nRasterYSize >= 256 ) // <- 256

> nBlockYSize = 128; // <- 128 ???

> else

> nBlockYSize = nRasterYSize;

>

> Into:

>

> nBlockXSize = std::min(nRasterXSize, 2048);

> nBlockYSize = std::min(nRasterYSize, 128)

>

> or

>

> nBlockXSize = std::min(nRasterXSize, 2048);

> nBlockYSize = std::min(nRasterYSize, 256)

>

> Neither of those is the same behavior and the the jump from 256 back to 128

> is not explained in the revision log. Which behavior will have better

> performance for a average case or will it matter?

 

The code is a bit weird indeed. But the block size of smallish datasets has no real importance, so if you find it more reasable, switching the condition to if ( nRasterYSize >= 128 ) (or the equivalent std::min()) should have no adverse consequences.

 

Block size in the case of JPEG 2000 is really hard to come with. A smarter block size logic would depend on JPEG 2000 tiling, presence of precincts, etc... This also could depend on the read pattern. Would deserve extensive testing to come out with the ideal formula... In the case of JP2KAK, there's also a IRasterIO() implementation that causes the block size to not be used at all in typical read scenarios.

 

 

> The difference will be

> between 262K cells or 524K cells. With doubles, that will be 2 or 4 MB for

> a single block from one band.

 

FWIIW, JPEG2000 doesn't support double (at least not in the base standard)

 

 

--

Spatialys - Geospatial professional services

http://www.spatialys.com




--

_______________________________________________
gdal-dev mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/gdal-dev
Loading...