[gdal-dev] Fwd: [OpenJPEG] OpenJPEG 2.2.0 is out ! Faster and safer.

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
15 messages Options
Reply | Threaded
Open this post in threaded view
|

[gdal-dev] Fwd: [OpenJPEG] OpenJPEG 2.2.0 is out ! Faster and safer.

N. Farah

Great news. Thanks for the work.


Any plans to integrate this new version in GDAL ?


Thanks

Noureddine Farah


_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Fwd: [OpenJPEG] OpenJPEG 2.2.0 is out ! Faster and safer.

Even Rouault-2

On dimanche 13 août 2017 15:57:54 CEST N. Farah wrote:

> Great news. Thanks for the work.

>

>

> Any plans to integrate this new version in GDAL ?

 

Well, there's nothing to change in the GDAL source code. The existing JP2OpenJPEG driver will work with OpenJPEG 2.2. It is just a matter of recompiling libopenjp2.

 

Even

 

--

Spatialys - Geospatial professional services

http://www.spatialys.com


_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Fwd: [OpenJPEG] OpenJPEG 2.2.0 is out ! Faster and safer.

N. Farah

Right...since GDAL does not ship the 3rd party lib(s).


Thanks again



From: Even Rouault <[hidden email]>
Sent: Monday, August 14, 2017 4:49:09 AM
To: [hidden email]
Cc: N. Farah
Subject: Re: [gdal-dev] Fwd: [OpenJPEG] OpenJPEG 2.2.0 is out ! Faster and safer.
 

On dimanche 13 août 2017 15:57:54 CEST N. Farah wrote:

> Great news. Thanks for the work.

>

>

> Any plans to integrate this new version in GDAL ?

 

Well, there's nothing to change in the GDAL source code. The existing JP2OpenJPEG driver will work with OpenJPEG 2.2. It is just a matter of recompiling libopenjp2.

 

Even

 

--

Spatialys - Geospatial professional services

http://www.spatialys.com


_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Fwd: [OpenJPEG] OpenJPEG 2.2.0 is out ! Faster and safer.

Vincent Schut-4
In reply to this post by Even Rouault-2


On 08/14/2017 10:49 AM, Even Rouault wrote:

On dimanche 13 août 2017 15:57:54 CEST N. Farah wrote:

> Great news. Thanks for the work.

>

>

> Any plans to integrate this new version in GDAL ?

 

Well, there's nothing to change in the GDAL source code. The existing JP2OpenJPEG driver will work with OpenJPEG 2.2. It is just a matter of recompiling libopenjp2.

 

Even


Well, there might be some small change necessary. Apparently the configure script only checks path/include/openjpeg-2.0 and path/include/openjpeg-2.1 subdirectories for openjpeg.h, while my openjpeg is now in path/include/openjpeg-2.2 :

configure: error: openjpeg.h not found in /usr/include/openjpeg-2.0 or /usr/include/openjpeg-2.1

Vincent.

 

--

Spatialys - Geospatial professional services

http://www.spatialys.com



_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev


_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Fwd: [OpenJPEG] OpenJPEG 2.2.0 is out ! Faster and safer.

Even Rouault-2

Vincent,

 

> Well, there might be some small change necessary. Apparently the

> configure script only checks path/include/openjpeg-2.0 and

> path/include/openjpeg-2.1 subdirectories for openjpeg.h, while my

> openjpeg is now in path/include/openjpeg-2.2 :

>

> configure: error: openjpeg.h not found in /usr/include/openjpeg-2.0 or

> /usr/include/openjpeg-2.1

 

Indeed. I actually only tested without rebuilding GDAL and just linking to the new lib...

I just added support for it in GDAL trunk and branches/2.2 branch per

https://trac.osgeo.org/gdal/ticket/7002

 

Even

 

--

Spatialys - Geospatial professional services

http://www.spatialys.com


_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Fwd: [OpenJPEG] OpenJPEG 2.2.0 is out ! Faster and safer.

N. Farah
In reply to this post by N. Farah

Hi,


I rebuilt GDAL (older version: 2.1.2) using this new open jpeg 2.2.0 (https://github.com/uclouvain/openjpeg/releases/download/v2.2.0/openjpeg-v2.2.0-linux-x86_64.tar.gz)

and tried to convert two jp2 dataset to tiff (gdal_translate):

- bluemarble_4km.jp2: (10800 x 5400 pixels). 37.3 mb file size.

- usgsLanSat.jp2: (8627 x 7745 pixels). 48 mb file size.


Tests were run on virtual machine with ‘Ubuntu 16.04’ operation system, 18 gb RAM and 2 CPU processors.


gdal_translate ‘bluemarble_4km.jp2’ to tiff:

- OpenJpeg 2.1.2:

13m34s: first run

13m35s: second run


- OpenJpeg 2.2.0:

13m45: first run

14m20s: second run


gdal_translate ‘usgsLanSat.jp2’ to tiff:

- OpenJpeg 2.1.2:

16m52s: first run

17m04s: second run


- OpenJpeg 2.2.0:

18m13s: first run

17m54s: second run


These results were not expected since the new openJpeg should be faster than the previous one. Could this be cause by 'gdal_translate' not using threads that the new openjpeg uses for performance boost ?


Or could it one of GDAL configuration or the built version of OpenJpeg need to be rebuilt using different config ?


Thanks

Noureddine Farah




From: N. Farah <[hidden email]>
Sent: Monday, August 14, 2017 11:42:31 AM
To: Even Rouault; [hidden email]
Subject: Re: [gdal-dev] Fwd: [OpenJPEG] OpenJPEG 2.2.0 is out ! Faster and safer.
 

Right...since GDAL does not ship the 3rd party lib(s).


Thanks again



From: Even Rouault <[hidden email]>
Sent: Monday, August 14, 2017 4:49:09 AM
To: [hidden email]
Cc: N. Farah
Subject: Re: [gdal-dev] Fwd: [OpenJPEG] OpenJPEG 2.2.0 is out ! Faster and safer.
 

On dimanche 13 août 2017 15:57:54 CEST N. Farah wrote:

> Great news. Thanks for the work.

>

>

> Any plans to integrate this new version in GDAL ?

 

Well, there's nothing to change in the GDAL source code. The existing JP2OpenJPEG driver will work with OpenJPEG 2.2. It is just a matter of recompiling libopenjp2.

 

Even

 

--

Spatialys - Geospatial professional services

http://www.spatialys.com


_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Fwd: [OpenJPEG] OpenJPEG 2.2.0 is out ! Faster and safer.

Even Rouault-2

Hi,

 

>

>

> I rebuilt GDAL (older version: 2.1.2) using this new open jpeg 2.2.0

> (https://github.com/uclouvain/openjpeg/releases/download/v2.2.0/openjpeg-v2

> .2.0-linux-x86_64.tar.gz)

>

> and tried to convert two jp2 dataset to tiff (gdal_translate):

>

> - bluemarble_4km.jp2: (10800 x 5400 pixels). 37.3 mb file size.

>

> - usgsLanSat.jp2: (8627 x 7745 pixels). 48 mb file size.

 

Do you know if they use JPEG2000 tiling ?

 

>

>

> Tests were run on virtual machine with ‘Ubuntu 16.04’ operation system, 18

> gb RAM and 2 CPU processors.

>

 

>

> These results were not expected since the new openJpeg should be faster than

> the previous one. Could this be cause by 'gdal_translate' not using threads

> that the new openjpeg uses for performance boost ?

 

By default, GDAL will now enable openjpeg thread support. So you need to set the OPJ_NUM_THREADS environment variable. That said, you should already get significant performance improvements even without enabling multi-threading. I'm not sure why you don't get them.

 

I'd be interested if you could share those images.

 

Even

 

 

--

Spatialys - Geospatial professional services

http://www.spatialys.com


_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Fwd: [OpenJPEG] OpenJPEG 2.2.0 is out ! Faster and safer.

N. Farah

Thanks Even for the quick response. I'll give it a try with setting OPJ_NUM_THREADS env variable to 8. Does it need to be used with GDAL_NUM_THREADS ? Any url to read about those two env variables ?


The two dataset i used:


p.s: i'm using GDAL 2.1.2


Noureddine


From: Even Rouault <[hidden email]>
Sent: Monday, August 28, 2017 1:40:31 PM
To: N. Farah
Cc: [hidden email]
Subject: Re: [gdal-dev] Fwd: [OpenJPEG] OpenJPEG 2.2.0 is out ! Faster and safer.
 

Hi,

 

>

>

> I rebuilt GDAL (older version: 2.1.2) using this new open jpeg 2.2.0

> (https://github.com/uclouvain/openjpeg/releases/download/v2.2.0/openjpeg-v2

> .2.0-linux-x86_64.tar.gz)

>

> and tried to convert two jp2 dataset to tiff (gdal_translate):

>

> - bluemarble_4km.jp2: (10800 x 5400 pixels). 37.3 mb file size.

>

> - usgsLanSat.jp2: (8627 x 7745 pixels). 48 mb file size.

 

Do you know if they use JPEG2000 tiling ?

 

>

>

> Tests were run on virtual machine with ‘Ubuntu 16.04’ operation system, 18

> gb RAM and 2 CPU processors.

>

 

>

> These results were not expected since the new openJpeg should be faster than

> the previous one. Could this be cause by 'gdal_translate' not using threads

> that the new openjpeg uses for performance boost ?

 

By default, GDAL will now enable openjpeg thread support. So you need to set the OPJ_NUM_THREADS environment variable. That said, you should already get significant performance improvements even without enabling multi-threading. I'm not sure why you don't get them.

 

I'd be interested if you could share those images.

 

Even

 

 

--

Spatialys - Geospatial professional services

http://www.spatialys.com


_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Fwd: [OpenJPEG] OpenJPEG 2.2.0 is out ! Faster and safer.

Even Rouault-2

On lundi 28 août 2017 18:00:03 CEST N. Farah wrote:

> Thanks Even for the quick response. I'll give it a try with setting

> OPJ_NUM_THREADS env variable to 8. Does it need to be used with

> GDAL_NUM_THREADS ? Any url to read about those two env variables ?

>

>

 

I've added some documentation per

https://trac.osgeo.org/gdal/changeset/39954

 

 

> The two dataset i used:

>

> <http://data.opengee.org/FusionTutorial-Full.tar.gz>

>

> https://1drv.ms/u/s!AkFDoDHsFe7_hkBS6GbG4xMqnk6t<http://data.opengee.org/Fus

> ionTutorial-Full.tar.gz>

> https://1drv.ms/u/s!AkFDoDHsFe7_hkG1-XDo7UztsPoC<http://data.opengee.org/Fu

> sionTutorial-Full.tar.gz>

 

Those images are single-tiled, so the GDAL_NUM_THREADS optimization will not help for them, and current openjpeg versions are super inefficient on those use cases, as GDAL expose a pseudo tiling to avoid a super big tile size, which cause the whole image to be decoded multiple times. I'm currently working on improving subtile decoding, that will substantially improve performance for such images (first steps are already in openjpeg master, next to follow)

 

I didn't have the patience to convert the whole image, so just took 3 GDAL blocks of 1024x1024

Without threading support, on hot runs:

 

gdal_translate bluemarble_4km.jp2 out.tif -srcwin 0 0 3072 1024

 

35.1 s GDAL 2.2 / openjpeg 2.1.2

26.2 s GDAL 2.2 / openjpeg 2.2.0

 

So I do see an improvement. Not sure why you don't.

 

Hardware: Intel(R) Core(TM) i7-6700HQ CPU @ 2.60GHz

OS: Ubuntu 16.04, 64bit build

 

 

and as a teaser:

3.1 s GDAL 2.2 / openjpeg master

1.5 s GDAL 2.2 / openjpeg (my work-in-progress branch)

 

With 8 threads:

9.9 s GDAL 2.2 / openjpeg 2.2.0

2.3 s GDAL 2.2 / openjpeg master

0.9 s GDAL 2.2 / openjpeg (my work-in-progress branch)

 

 

 

With usgsLanSat.jp2

 

gdal_translate usgsLanSat.jp2 out.tif -srcwin 0 0 3072 1024

 

40.2 s GDAL 2.2 / openjpeg 2.1.2

32.3 s GDAL 2.2 / openjpeg 2.2.0

 

>

> p.s: i'm using GDAL 2.1.2

 

GDAL 2.1 or 2.2 should have comparable performances on this, I think

 

--

Spatialys - Geospatial professional services

http://www.spatialys.com


_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Fwd: [OpenJPEG] OpenJPEG 2.2.0 is out ! Faster and safer.

N. Farah

Thanks Even for updating the doc and also running some perf testing using those datasets.


The results you get when enabling multi-threading are very interesting: 26.s to 9.9 s. Then using the work in progress you end up with 1s. Basically a 26 times speed improvement.


Is there a timeline for releasing next version of openJpeg (either master or work in progress).


I'm very interested in the performance improvement since they looks promising and may get closer or beat to Kakadu performance.


Are there shared jp2 dataset you're using for performance benchmark ?


Noureddine




From: Even Rouault <[hidden email]>
Sent: Monday, August 28, 2017 2:55 PM
To: N. Farah
Cc: [hidden email]
Subject: Re: [gdal-dev] Fwd: [OpenJPEG] OpenJPEG 2.2.0 is out ! Faster and safer.
 

On lundi 28 août 2017 18:00:03 CEST N. Farah wrote:

> Thanks Even for the quick response. I'll give it a try with setting

> OPJ_NUM_THREADS env variable to 8. Does it need to be used with

> GDAL_NUM_THREADS ? Any url to read about those two env variables ?

>

>

 

I've added some documentation per

https://trac.osgeo.org/gdal/changeset/39954

 

 

> The two dataset i used:

>

> <http://data.opengee.org/FusionTutorial-Full.tar.gz>

>

> https://1drv.ms/u/s!AkFDoDHsFe7_hkBS6GbG4xMqnk6t<http://data.opengee.org/Fus

> ionTutorial-Full.tar.gz>

> https://1drv.ms/u/s!AkFDoDHsFe7_hkG1-XDo7UztsPoC<http://data.opengee.org/Fu

> sionTutorial-Full.tar.gz>

 

Those images are single-tiled, so the GDAL_NUM_THREADS optimization will not help for them, and current openjpeg versions are super inefficient on those use cases, as GDAL expose a pseudo tiling to avoid a super big tile size, which cause the whole image to be decoded multiple times. I'm currently working on improving subtile decoding, that will substantially improve performance for such images (first steps are already in openjpeg master, next to follow)

 

I didn't have the patience to convert the whole image, so just took 3 GDAL blocks of 1024x1024

Without threading support, on hot runs:

 

gdal_translate bluemarble_4km.jp2 out.tif -srcwin 0 0 3072 1024

 

35.1 s GDAL 2.2 / openjpeg 2.1.2

26.2 s GDAL 2.2 / openjpeg 2.2.0

 

So I do see an improvement. Not sure why you don't.

 

Hardware: Intel(R) Core(TM) i7-6700HQ CPU @ 2.60GHz

OS: Ubuntu 16.04, 64bit build

 

 

and as a teaser:

3.1 s GDAL 2.2 / openjpeg master

1.5 s GDAL 2.2 / openjpeg (my work-in-progress branch)

 

With 8 threads:

9.9 s GDAL 2.2 / openjpeg 2.2.0

2.3 s GDAL 2.2 / openjpeg master

0.9 s GDAL 2.2 / openjpeg (my work-in-progress branch)

 

 

 

With usgsLanSat.jp2

 

gdal_translate usgsLanSat.jp2 out.tif -srcwin 0 0 3072 1024

 

40.2 s GDAL 2.2 / openjpeg 2.1.2

32.3 s GDAL 2.2 / openjpeg 2.2.0

 

>

> p.s: i'm using GDAL 2.1.2

 

GDAL 2.1 or 2.2 should have comparable performances on this, I think

 

--

Spatialys - Geospatial professional services

http://www.spatialys.com


_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Fwd: [OpenJPEG] OpenJPEG 2.2.0 is out ! Faster and safer.

Even Rouault-2

On lundi 28 août 2017 19:36:16 CEST N. Farah wrote:

> Thanks Even for updating the doc and also running some perf testing using

> those datasets.

>

>

> The results you get when enabling multi-threading are very interesting: 26.s

> to 9.9 s. Then using the work in progress you end up with 1s. Basically a

> 26 times speed improvement.

>

>

> Is there a timeline for releasing next version of openJpeg (either master or

> work in progress).

 

Probably in a few weeks

 

>

>

> Are there shared jp2 dataset you're using for performance benchmark ?

>

 

There's a basic benchmarking suite here

https://github.com/uclouvain/openjpeg/tree/master/tests/performance

referencing datasets in

https://github.com/uclouvain/openjpeg-data

but it is not necessarily complete.

 

>

> Noureddine

>

>

> ________________________________

> From: Even Rouault <[hidden email]>

> Sent: Monday, August 28, 2017 2:55 PM

> To: N. Farah

> Cc: [hidden email]

> Subject: Re: [gdal-dev] Fwd: [OpenJPEG] OpenJPEG 2.2.0 is out ! Faster and

> safer.

> On lundi 28 août 2017 18:00:03 CEST N. Farah wrote:

> > Thanks Even for the quick response. I'll give it a try with setting

> >

> > OPJ_NUM_THREADS env variable to 8. Does it need to be used with

> >

> > GDAL_NUM_THREADS ? Any url to read about those two env variables ?

>

> I've added some documentation per

>

> https://trac.osgeo.org/gdal/changeset/39954

>

> > The two dataset i used:

> >

> >

> >

> > <http://data.opengee.org/FusionTutorial-Full.tar.gz>

> >

> >

> >

> > https://1drv.ms/u/s!AkFDoDHsFe7_hkBS6GbG4xMqnk6t<http://data.opengee.org/F

> > us

> >

> > ionTutorial-Full.tar.gz>

> >

> > https://1drv.ms/u/s!AkFDoDHsFe7_hkG1-XDo7UztsPoC<http://data.opengee.org/F

> > u

> >

> > sionTutorial-Full.tar.gz>

>

> Those images are single-tiled, so the GDAL_NUM_THREADS optimization will not

> help for them, and current openjpeg versions are super inefficient on those

> use cases, as GDAL expose a pseudo tiling to avoid a super big tile size,

> which cause the whole image to be decoded multiple times. I'm currently

> working on improving subtile decoding, that will substantially improve

> performance for such images (first steps are already in openjpeg master,

> next to follow)

>

>

>

> I didn't have the patience to convert the whole image, so just took 3 GDAL

> blocks of 1024x1024

>

> Without threading support, on hot runs:

>

>

>

> gdal_translate bluemarble_4km.jp2 out.tif -srcwin 0 0 3072 1024

>

>

>

> 35.1 s GDAL 2.2 / openjpeg 2.1.2

>

> 26.2 s GDAL 2.2 / openjpeg 2.2.0

>

>

>

> So I do see an improvement. Not sure why you don't.

>

>

>

> Hardware: Intel(R) Core(TM) i7-6700HQ CPU @ 2.60GHz

>

> OS: Ubuntu 16.04, 64bit build

>

>

>

>

>

> and as a teaser:

>

> 3.1 s GDAL 2.2 / openjpeg master

>

> 1.5 s GDAL 2.2 / openjpeg (my work-in-progress branch)

>

>

>

> With 8 threads:

>

> 9.9 s GDAL 2.2 / openjpeg 2.2.0

>

> 2.3 s GDAL 2.2 / openjpeg master

>

> 0.9 s GDAL 2.2 / openjpeg (my work-in-progress branch)

>

>

>

>

>

>

>

> With usgsLanSat.jp2

>

>

>

> gdal_translate usgsLanSat.jp2 out.tif -srcwin 0 0 3072 1024

>

>

>

> 40.2 s GDAL 2.2 / openjpeg 2.1.2

>

> 32.3 s GDAL 2.2 / openjpeg 2.2.0

>

> > p.s: i'm using GDAL 2.1.2

>

> GDAL 2.1 or 2.2 should have comparable performances on this, I think

>

>

>

> --

>

> Spatialys - Geospatial professional services

>

> http://www.spatialys.com

 

 

--

Spatialys - Geospatial professional services

http://www.spatialys.com


_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Fwd: [OpenJPEG] OpenJPEG 2.2.0 is out ! Faster and safer.

jratike80
In reply to this post by N. Farah
N. Farah wrote
The results you get when enabling multi-threading are very interesting: 26.s to 9.9 s. Then using the work in progress you end up with 1s. Basically a 26 times speed improvement.
Hi,

As always when you read benchmarks, you must keep the results and the context together so it is clear what was tested. In this case the important part is that the test applies to single tiled images. They are what Kakadu prefers (also in the usage examples) and it handles them splendidly by utilizing the precincts. OpenJPEG has not been able to use precincts (despite in a fork https://lists.osgeo.org/pipermail/gdal-dev/2016-February/043650.html) and therefore it has slow with single tiled images and actually useless if the tile size is something like 20000 by 20000 pixels.

Even can say how much faster the new branch is with JPEG2000 images which use 1024x1024 tiles but I suppose that the speed-up factor is much less than 26.

-Jukka Rahkonen-
Reply | Threaded
Open this post in threaded view
|

Re: Fwd: [OpenJPEG] OpenJPEG 2.2.0 is out ! Faster and safer.

Even Rouault-2

On mardi 29 août 2017 01:42:21 CEST jratike80 wrote:

> N. Farah wrote

>

> > The results you get when enabling multi-threading are very interesting:

> > 26.s to 9.9 s. Then using the work in progress you end up with 1s.

> > Basically a 26 times speed improvement.

>

> Hi,

>

> As always when you read benchmarks, you must keep the results and the

> context together so it is clear what was tested. In this case the important

> part is that the test applies to single tiled images. They are what Kakadu

> prefers (also in the usage examples) and it handles them splendidly by

> utilizing the precincts. OpenJPEG has not been able to use precincts

> (despite in a fork

> https://lists.osgeo.org/pipermail/gdal-dev/2016-February/043650.html) and

> therefore it has slow with single tiled images and actually useless if the

> tile size is something like 20000 by 20000 pixels.

>

> Even can say how much faster the new branch is with JPEG2000 images which

> use 1024x1024 tiles but I suppose that the speed-up factor is much less than

> 26.

 

For tiled images, the GDAL OpenJPEG driver currently still read whole tiles (perhaps this should be re-considered at some point), so sub-tile decoding improvements will not help here, and you won't get more than the improvements done during 2.2.0, so something like 20% for openjpeg single-threaded (your mileage may vary), or more if you enable OPJ_NUM_THREADS.

 

Even

 

--

Spatialys - Geospatial professional services

http://www.spatialys.com


_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Fwd: [OpenJPEG] OpenJPEG 2.2.0 is out ! Faster and safer.

nicholas.g.lawrence-2-2

I’m curious, why is OpenJpeg not compiled into GDAL by default? Is this a policy?

 

Thanks,

Nick

 

From: gdal-dev [mailto:[hidden email]] On Behalf Of Even Rouault
Sent: Tuesday, 29 August 2017 7:37 PM
To: [hidden email]
Cc: jratike80 <[hidden email]>
Subject: Re: [gdal-dev] Fwd: [OpenJPEG] OpenJPEG 2.2.0 is out ! Faster and safer.

 

On mardi 29 août 2017 01:42:21 CEST jratike80 wrote:

> N. Farah wrote

>

> > The results you get when enabling multi-threading are very interesting:

> > 26.s to 9.9 s. Then using the work in progress you end up with 1s.

> > Basically a 26 times speed improvement.

>

> Hi,

>

> As always when you read benchmarks, you must keep the results and the

> context together so it is clear what was tested. In this case the important

> part is that the test applies to single tiled images. They are what Kakadu

> prefers (also in the usage examples) and it handles them splendidly by

> utilizing the precincts. OpenJPEG has not been able to use precincts

> (despite in a fork

> https://lists.osgeo.org/pipermail/gdal-dev/2016-February/043650.html) and

> therefore it has slow with single tiled images and actually useless if the

> tile size is something like 20000 by 20000 pixels.

>

> Even can say how much faster the new branch is with JPEG2000 images which

> use 1024x1024 tiles but I suppose that the speed-up factor is much less than

> 26.

 

For tiled images, the GDAL OpenJPEG driver currently still read whole tiles (perhaps this should be re-considered at some point), so sub-tile decoding improvements will not help here, and you won't get more than the improvements done during 2.2.0, so something like 20% for openjpeg single-threaded (your mileage may vary), or more if you enable OPJ_NUM_THREADS.

 

Even

 

--

Spatialys - Geospatial professional services

http://www.spatialys.com


***********************************************************************
WARNING: This email (including any attachments) may contain legally
privileged, confidential or private information and may be protected by
copyright. You may only use it if you are the person(s) it was
intended to be sent to and if you use it in an authorised way. No one
is allowed to use, review, alter, transmit, disclose, distribute, print
or copy this email without appropriate authority.

If this email was not intended for you and was sent to you by mistake,
please telephone or email me immediately, destroy any hardcopies of
this email and delete it and any copies of it from your computer
system. Any right which the sender may have under copyright law, and
any legal privilege and confidentiality attached to this email is not
waived or destroyed by that mistake.

It is your responsibility to ensure that this email does not contain
and is not affected by computer viruses, defects or interference by
third parties or replication problems (including incompatibility with
your computer system).

Opinions contained in this email do not necessarily reflect the
opinions of the Department of Transport and Main Roads,
or endorsed organisations utilising the same infrastructure.
***********************************************************************


_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Fwd: [OpenJPEG] OpenJPEG 2.2.0 is out ! Faster and safer.

Even Rouault-2

On mercredi 30 août 2017 00:25:46 CEST Nicholas G Lawrence wrote:

> I'm curious, why is OpenJpeg not compiled into GDAL by default? Is this a

> policy?

>

 

This is mostly a technical constraint. The GDAL OpenJPEG driver requires linking against libopenjp2, which is the result of the compilation of the OpenJPEG source code, which is in its own source code repository. So this is mostly an issue of packaging choice: which optional dependencies you install before building GDAL. Like all other GDAL drivers that depend on third party libraries. Most popular binary distributions of GDAL have the openjpeg driver built (OSGeo4W, gisinternals, UbuntuGIS) (not necessarily against the latest libopenjp2 version)

 

The GDAL source code repository has historically a few copies of common dependencies (libz, libtiff, etc...) for conveniency on platforms where installing packaged dependencies isn't easy (Windows typically), but we cannot extend this practice to all dependencies, because of the increased maintainance cost.

 

Even

 

 

> Thanks,

> Nick

>

> From: gdal-dev [mailto:[hidden email]] On Behalf Of Even

> Rouault Sent: Tuesday, 29 August 2017 7:37 PM

> To: [hidden email]

> Cc: jratike80 <[hidden email]>

> Subject: Re: [gdal-dev] Fwd: [OpenJPEG] OpenJPEG 2.2.0 is out ! Faster and

> safer.

> On mardi 29 août 2017 01:42:21 CEST jratike80 wrote:

> > N. Farah wrote

> >

> > > The results you get when enabling multi-threading are very interesting:

> > >

> > > 26.s to 9.9 s. Then using the work in progress you end up with 1s.

> > >

> > > Basically a 26 times speed improvement.

> >

> > Hi,

> >

> >

> >

> > As always when you read benchmarks, you must keep the results and the

> >

> > context together so it is clear what was tested. In this case the

> > important

> >

> > part is that the test applies to single tiled images. They are what Kakadu

> >

> > prefers (also in the usage examples) and it handles them splendidly by

> >

> > utilizing the precincts. OpenJPEG has not been able to use precincts

> >

> > (despite in a fork

> >

> > https://lists.osgeo.org/pipermail/gdal-dev/2016-February/043650.html) and

> >

> > therefore it has slow with single tiled images and actually useless if the

> >

> > tile size is something like 20000 by 20000 pixels.

> >

> >

> >

> > Even can say how much faster the new branch is with JPEG2000 images which

> >

> > use 1024x1024 tiles but I suppose that the speed-up factor is much less

> > than

> >

> > 26.

>

> For tiled images, the GDAL OpenJPEG driver currently still read whole tiles

> (perhaps this should be re-considered at some point), so sub-tile decoding

> improvements will not help here, and you won't get more than the

> improvements done during 2.2.0, so something like 20% for openjpeg

> single-threaded (your mileage may vary), or more if you enable

> OPJ_NUM_THREADS.

>

>

>

> Even

>

>

>

> --

>

> Spatialys - Geospatial professional services

>

> http://www.spatialys.com

>

>

> ***********************************************************************

> WARNING: This email (including any attachments) may contain legally

> privileged, confidential or private information and may be protected by

> copyright. You may only use it if you are the person(s) it was

> intended to be sent to and if you use it in an authorised way. No one

> is allowed to use, review, alter, transmit, disclose, distribute, print

> or copy this email without appropriate authority.

>

> If this email was not intended for you and was sent to you by mistake,

> please telephone or email me immediately, destroy any hardcopies of

> this email and delete it and any copies of it from your computer

> system. Any right which the sender may have under copyright law, and

> any legal privilege and confidentiality attached to this email is not

> waived or destroyed by that mistake.

>

> It is your responsibility to ensure that this email does not contain

> and is not affected by computer viruses, defects or interference by

> third parties or replication problems (including incompatibility with

> your computer system).

>

> Opinions contained in this email do not necessarily reflect the

> opinions of the Department of Transport and Main Roads,

> or endorsed organisations utilising the same infrastructure.

> ***********************************************************************

 

 

--

Spatialys - Geospatial professional services

http://www.spatialys.com


_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev