[gdal-dev] Help with idiomatic GDAL solution for raster algebra benchmark

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
16 messages Options
Reply | Threaded
Open this post in threaded view
|

[gdal-dev] Help with idiomatic GDAL solution for raster algebra benchmark

Alex HighViz

Dear all,

 

I few times I have posted to the list trying to promote the idea of providing iterators over pixels in a raster band , and more generally to make raster data accessible using (future) standard conforming ranges. It would make implementing algorithms on raster data a lot more intuitive.  These ideas are implemented in Pronto Raster which is an OSGeo Community C++ library. Feedback in this list has been that such a solution would incur costly computational overheads.

 

I now have some preliminary benchmark results, where I compare Pronto Raster solutions for a simple raster overlay operation (OUT = 3 * A + B * C) to a direct and idiomatic GDAL implementation. The results seem to indicate that overheads can be negligible, depending on which Pronto Raster functions are used. I would very much appreciate it if some more experienced GDAL C++ users could look at my “idiomatic GDAL implementation” to see if it really is what it claims to be and I am not overstating the results. I can use your help.

 

I’d also be interested to hear any opinion about these results and the costs / benefits associated with providing pixel ranges for raster bands.

 

For details on the benchmark see here:    https://github.com/ahhz/raster/blob/master/docs/_posts/2018-06-14-Preliminary-benchmark-results-are-promising.md

or here: http://ahhz.github.io/raster/Preliminary-benchmark-results-are-promising/

 

Many thanks,  Alex


_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Help with idiomatic GDAL solution for raster algebra benchmark

Mateusz Loskot
On 14 June 2018 at 14:17, Alex HighViz <[hidden email]> wrote:
>
> I few times I have posted to the list trying to promote the idea of
> providing iterators over pixels in a raster band , and more generally to
> make raster data accessible using (future) standard conforming ranges.

In case you are not aware, Boost.GIL provides such abstractions [1]
It's not yet pure C++20 ranges, there will be at some point.
Although Boost.GIL design is a complex one, it's IMHO not trivial to beat
its feature-completness, flexibility, extensibility and performance it
can generate.

FYI, I have half-baked IO extension for GDAL.
I'm going to continue/finish it as soon as next Boost version is released
which should include completely new version of IO layer.
Then, it can become another one to compare in your benchmark.

[1] http://boostorg.github.io/gil/develop/doc/html/design_guide.html
[2] https://lists.boost.org/boost-announce/2011/01/0281.php

Best regards,
--
Mateusz Loskot, http://mateusz.loskot.net
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Help with idiomatic GDAL solution for raster algebra benchmark

Alex HighViz
Hi Mateusz,

Yes, I am aware of Boost.GIL, but perhaps didn't study it well enough when it became part of Boost. The section on usage on image views (http://boostorg.github.io/gil/develop/doc/html/design_guide.html#image-view) illustrates exactly the kind of flexibility I would like to have when working with geographical raster data, and is what motivates me in Pronto Raster. The image view concept in GIL  serves exactly the same function as the raster concept in Pronto, but seems a bit more robust in adherence to standards.

I remember when looking at it at the time that I felt that using Boost.GIL would give me more difficulty than reward. I would have to make a lot of effort to wrap GDALRasterband's up to conform to GIL concepts and then would still need to implement the functionality that I was after then (generalized moving windows, i.e. convolutions, with a custom function, https://www.sciencedirect.com/science/article/pii/S0303243415300337  ). Now with hindsight, it is perhaps not that much *extra* work to allign Pronto's Raster View with GIL's Image View; it would put existing GIL features in scope and allow for the best of both worlds.

Have you looked at Pronto's gdal_raster_view class? It seems to meet most of the Image View requirement and might be of help when finishing the IO extension for GDAL.

Would you by any chance be able (and willing) to conjure up a simple example of "OUT = 3 * A + B * C" using Boost.GIL?

Thank you for your input.

Alex


p.s. I do remember a blog post or message on the boost mailing list by you saying how great it would be to have Boost.GIL compatible with GDAL.



-----Original Message-----
From: Mateusz Loskot [mailto:[hidden email]]
Sent: 14 June 2018 16:34
To: Alex HighViz <[hidden email]>
Cc: [hidden email]; Hagen-Zanker AH Dr (Civil & Env. Eng.) <[hidden email]>
Subject: Re: [gdal-dev] Help with idiomatic GDAL solution for raster algebra benchmark

On 14 June 2018 at 14:17, Alex HighViz <[hidden email]> wrote:
>
> I few times I have posted to the list trying to promote the idea of
> providing iterators over pixels in a raster band , and more generally
> to make raster data accessible using (future) standard conforming ranges.

In case you are not aware, Boost.GIL provides such abstractions [1] It's not yet pure C++20 ranges, there will be at some point.
Although Boost.GIL design is a complex one, it's IMHO not trivial to beat its feature-completness, flexibility, extensibility and performance it can generate.

FYI, I have half-baked IO extension for GDAL.
I'm going to continue/finish it as soon as next Boost version is released which should include completely new version of IO layer.
Then, it can become another one to compare in your benchmark.

[1] http://boostorg.github.io/gil/develop/doc/html/design_guide.html
[2] https://lists.boost.org/boost-announce/2011/01/0281.php

Best regards,
--
Mateusz Loskot, http://mateusz.loskot.net
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Help with idiomatic GDAL solution for raster algebra benchmark

Even Rouault-2
In reply to this post by Alex HighViz
Hi,

Interesting work.

A few thoughts:

- the benchmark_3_rasters_reference() could receive a few comments for non-
advanced users to mention that it is correct only if all the rasters use Byte
datatype, and all have the same block size.

- It would be interesting to see a variant of this reference benchmark that
does no computation at all, that is to say remove the for (int iY = 0; iY <
nYValid; iY++) loop . That would show how much CPU computation time is really
involved

- 1000x1000 is somewhat small. Perhaps benchmark on larger rasters

- with some tricks, it should probably be possible to have a compiler generate
SSE2/AVX2 instructions for the computation loop of the reference benchmark,
since it looks like an excellent candidate for vectorization. Or that could be
done at hand with a few _mm_ intrinsincs. Do you think that Pronto could use
that ?

- it would be interested if you could also benchmark a VRT with Python numpy-
based pixel function, and enable Numba. See
http://www.gdal.org/gdal_vrttut.html#gdal_vrttut_derived_python (or just pure
numpy based computation + Numba)

- the benchmark only involve integer computations. What if you do linear
combination of bands with floating-point coefficients for example ?

Even

> Dear all,
>
> I few times I have posted to the list trying to promote the idea of
> providing iterators over pixels in a raster band , and more generally to
> make raster data accessible using (future) standard conforming ranges. It
> would make implementing algorithms on raster data a lot more intuitive.
> These ideas are implemented in Pronto Raster which is an OSGeo Community
> C++ library. Feedback in this list has been that such a solution would
> incur costly computational overheads.
>
> I now have some preliminary benchmark results, where I compare Pronto Raster
> solutions for a simple raster overlay operation (OUT = 3 * A + B * C) to a
> direct and idiomatic GDAL implementation. The results seem to indicate that
> overheads can be negligible, depending on which Pronto Raster functions are
> used. I would very much appreciate it if some more experienced GDAL C++
> users could look at my "idiomatic GDAL implementation" to see if it really
> is what it claims to be and I am not overstating the results. I can use
> your help.
>
> I'd also be interested to hear any opinion about these results and the costs
> / benefits associated with providing pixel ranges for raster bands.
>
> For details on the benchmark see here:  
> https://github.com/ahhz/raster/blob/master/docs/_posts/2018-06-14-Prelimina
> ry-benchmark-results-are-promising.md or here:
> http://ahhz.github.io/raster/Preliminary-benchmark-results-are-promising/
>
> Many thanks,  Alex


--
Spatialys - Geospatial professional services
http://www.spatialys.com
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Help with idiomatic GDAL solution for raster algebra benchmark

Mateusz Loskot
In reply to this post by Alex HighViz
On 14 June 2018 at 18:47, Alex HighViz <[hidden email]> wrote:
> I remember when looking at it at the time that I felt that using Boost.GIL would give me more difficulty than reward.

It is not easy indeed. Things had not been improved for long time,
esp. documenation. It's changing and improving though.

> I would have to make a lot of effort to wrap GDALRasterband's up to conform to GIL concepts and then would still need to implement the functionality that I was after then (generalized moving windows, i.e. convolutions, with a custom function, https://www.sciencedirect.com/science/article/pii/S0303243415300337  ).

This is an interesting topic, I may review Boost.GIL features in this context.

> Now with hindsight, it is perhaps not that much *extra* work to allign Pronto's Raster View with GIL's Image View; it would put existing GIL features in scope and allow for the best of both worlds.

An interoperability certainly is possible with some adaptation in one
or both directions.

> Have you looked at Pronto's gdal_raster_view class? It seems to meet most of the Image View requirement and might be of help when finishing the IO extension for GDAL.

Yes, briefly. It looks like gdal_raster_view  offers one mode of iterations.

Boost.GIL allows operations on views which are:
- no iteartion (eg. single memmove)
- 1D iteration, usually fastest
- 2D iteration
Algorithms auto-choose depending on traversable mode available for
particular view(s).
This should not be an issue for adaptation though.

By the way, for example, a raw array of bytes is also easily adaptable
as a view, two arrays (colour table and image bytes) can also be
adapted as an indexed view/image.

> Would you by any chance be able (and willing) to conjure up a simple example of "OUT = 3 * A + B * C" using Boost.GIL?

I may give it a go, perhaps after new Boost.GIL is out.

BTW, if you have any GIL-specific questions, there is
https://lists.boost.org/mailman/listinfo.cgi/boost-gil  (low traffic).

> p.s. I do remember a blog post or message on the boost mailing list by you saying how great it would be to have Boost.GIL compatible with GDAL.

I don't remember exactly, but might got excited about this idea :-)

Best regards,
--
Mateusz Loskot, http://mateusz.loskot.net
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Help with idiomatic GDAL solution for raster algebra benchmark

Mateusz Loskot
In reply to this post by Alex HighViz
On 14 June 2018 at 18:47, Alex HighViz <[hidden email]> wrote:
> Would you by any chance be able (and willing) to conjure up a simple example of "OUT = 3 * A + B * C" using Boost.GIL?

Quick update: out of the box, GIL provides related algorithm that
accepts one or two sources: transform_pixels
Here is example on channel-wise pixel addition:

How to combine images with boost gil?
https://stackoverflow.com/a/47623314/151641

The transform_pixels could be extended, using the binary
channel_multiply internally, etc.

Best regards,
--
Mateusz Loskot, http://mateusz.loskot.net
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Help with idiomatic GDAL solution for raster algebra benchmark

Alex HighViz
In reply to this post by Even Rouault-2
Thanks for the suggestions. These extra benchmarks should be feasible in the next week. Especially if I manage to get started properly with Google benchmark. I need to look into sse2/ave2. My feeling is that will be easier and more effective for the reference case than for Pronto, because pronto is geared to express all operations at the pixel level.

From: Even Rouault <[hidden email]>
Sent: 14 June 2018 21:22:36
To: [hidden email]
Cc: Alex HighViz; Hagen-Zanker AH Dr (Civil & Env. Eng.)
Subject: Re: [gdal-dev] Help with idiomatic GDAL solution for raster algebra benchmark
 
Hi,

Interesting work.

A few thoughts:

- the benchmark_3_rasters_reference() could receive a few comments for non-
advanced users to mention that it is correct only if all the rasters use Byte
datatype, and all have the same block size.

- It would be interesting to see a variant of this reference benchmark that
does no computation at all, that is to say remove the for (int iY = 0; iY <
nYValid; iY++) loop . That would show how much CPU computation time is really
involved

- 1000x1000 is somewhat small. Perhaps benchmark on larger rasters

- with some tricks, it should probably be possible to have a compiler generate
SSE2/AVX2 instructions for the computation loop of the reference benchmark,
since it looks like an excellent candidate for vectorization. Or that could be
done at hand with a few _mm_ intrinsincs. Do you think that Pronto could use
that ?

- it would be interested if you could also benchmark a VRT with Python numpy-
based pixel function, and enable Numba. See
http://www.gdal.org/gdal_vrttut.html#gdal_vrttut_derived_python (or just pure
numpy based computation + Numba)

- the benchmark only involve integer computations. What if you do linear
combination of bands with floating-point coefficients for example ?

Even

> Dear all,
>
> I few times I have posted to the list trying to promote the idea of
> providing iterators over pixels in a raster band , and more generally to
> make raster data accessible using (future) standard conforming ranges. It
> would make implementing algorithms on raster data a lot more intuitive.
> These ideas are implemented in Pronto Raster which is an OSGeo Community
> C++ library. Feedback in this list has been that such a solution would
> incur costly computational overheads.
>
> I now have some preliminary benchmark results, where I compare Pronto Raster
> solutions for a simple raster overlay operation (OUT = 3 * A + B * C) to a
> direct and idiomatic GDAL implementation. The results seem to indicate that
> overheads can be negligible, depending on which Pronto Raster functions are
> used. I would very much appreciate it if some more experienced GDAL C++
> users could look at my "idiomatic GDAL implementation" to see if it really
> is what it claims to be and I am not overstating the results. I can use
> your help.
>
> I'd also be interested to hear any opinion about these results and the costs
> / benefits associated with providing pixel ranges for raster bands.
>
> For details on the benchmark see here:  
> https://github.com/ahhz/raster/blob/master/docs/_posts/2018-06-14-Prelimina
> ry-benchmark-results-are-promising.md or here:
> http://ahhz.github.io/raster/Preliminary-benchmark-results-are-promising/
>
> Many thanks,  Alex


--
Spatialys - Geospatial professional services
http://www.spatialys.com

_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Help with idiomatic GDAL solution for raster algebra benchmark

Mateusz Loskot
On 15 June 2018 at 01:22, Alex HighViz <[hidden email]> wrote:
> I need to look into sse2/ave2. My feeling is that will be easier
> and more effective for the reference case than for Pronto, because pronto is
> geared to express all operations at the pixel level.

You may try with auto vectorization first (could be interesting to
compare auto vectorized benchmark).
It's fairly easy to enable for MSVC, and GCC too, with a bunch of
compilation flags.

Best regards,
--
Mateusz Loskot, http://mateusz.loskot.net
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Help with idiomatic GDAL solution for raster algebra benchmark

Alex HighViz
Great, will try and let you know.
From: Mateusz Loskot <[hidden email]>
Sent: 15 June 2018 00:27:34
To: Alex HighViz
Cc: Even Rouault; [hidden email]; Hagen-Zanker AH Dr (Civil & Env. Eng.)
Subject: Re: [gdal-dev] Help with idiomatic GDAL solution for raster algebra benchmark
 
On 15 June 2018 at 01:22, Alex HighViz <[hidden email]> wrote:
> I need to look into sse2/ave2. My feeling is that will be easier
> and more effective for the reference case than for Pronto, because pronto is
> geared to express all operations at the pixel level.

You may try with auto vectorization first (could be interesting to
compare auto vectorized benchmark).
It's fairly easy to enable for MSVC, and GCC too, with a bunch of
compilation flags.

Best regards,
--
Mateusz Loskot, http://mateusz.loskot.net

_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Help with idiomatic GDAL solution for raster algebra benchmark

Alex HighViz
In reply to this post by Even Rouault-2
> Even Rouault wrote
> A few thoughts:

[ ]

> 1000x1000 is somewhat small. Perhaps benchmark on larger rasters

I am still working on your other suggestions, but thought it would be fair to report on my progress.

Regrettably I overstated the performance in my previous post, due to a bug that is now solved. With the solved bug, Pronto is about 50% slower than GDAL directly. I believe this is still good considering the benefits, but not as glamorous as I first thought. I have also applied it on a set of 100000 x 1000 rasters and the results are consistent.

The related post has been updated (and I still consider it "promising"): https://github.com/ahhz/raster/blob/master/docs/_posts/2018-06-14-Preliminary-benchmark-results-are-promising.md 
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Help with idiomatic GDAL solution for raster algebra benchmark

Mateusz Loskot
On 19 June 2018 at 11:22, Alex HighViz <[hidden email]> wrote:
>
> Regrettably I overstated the performance in my previous post, due to a bug that is now solved. With the solved bug, Pronto is about 50% slower than GDAL directly. I believe this is still good considering the benefits, but not as glamorous as I first thought. I have also applied it on a set of 100000 x 1000 rasters and the results are consistent.
>
> The related post has been updated (and I still consider it "promising"): https://github.com/ahhz/raster/blob/master/docs/_posts/2018-06-14-Preliminary-benchmark-results-are-promising.md

I might have missed that, but there seem to be no information about used
- compiler(s) and their versions
- compilation flags

Best regards,
--
Mateusz Loskot, http://mateusz.loskot.net
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Help with idiomatic GDAL solution for raster algebra benchmark

Alex HighViz


On 19/06/2018 10:28, Mateusz Loskot wrote:
> On 19 June 2018 at 11:22, Alex HighViz <[hidden email]> wrote:
>> Regrettably I overstated the performance in my previous post, due to a bug that is now solved. With the solved bug, Pronto is about 50% slower than GDAL directly. I believe this is still good considering the benefits, but not as glamorous as I first thought. I have also applied it on a set of 100000 x 1000 rasters and the results are consistent.
>>
>> The related post has been updated (and I still consider it "promising"): https://github.com/ahhz/raster/blob/master/docs/_posts/2018-06-14-Preliminary-benchmark-results-are-promising.md
> I might have missed that, but there seem to be no information about used
> - compiler(s) and their versions
> - compilation flags
>
> Best regards,

No you're right, that information is missing. I was using Visual Studio 15.2

/GS /GL /analyze- /W3 /Gy /Zc:wchar_t /Zi /Gm- /O2 /sdl
/Fd"Release\vc141.pdb" /Zc:inline /fp:precise /D "_MBCS"
/errorReport:prompt /WX- /Zc:forScope /Gd /Oy- /Oi /MD /std:c++latest
/Fa"Release\" /EHsc /nologo /Fo"Release\" /Fp"Release\raster.pch"
/diagnostics:classic
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Help with idiomatic GDAL solution for raster algebra benchmark

Mateusz Loskot
On 19 June 2018 at 11:50, Alex HighViz <[hidden email]> wrote:

> On 19/06/2018 10:28, Mateusz Loskot wrote:
>> On 19 June 2018 at 11:22, Alex HighViz <[hidden email]> wrote:
>>> Regrettably I overstated the performance in my previous post, due to a bug that is now solved. With the solved bug, Pronto is about 50% slower than GDAL directly. I believe this is still good considering the benefits, but not as glamorous as I first thought. I have also applied it on a set of 100000 x 1000 rasters and the results are consistent.
>>>
>>> The related post has been updated (and I still consider it "promising"): https://github.com/ahhz/raster/blob/master/docs/_posts/2018-06-14-Preliminary-benchmark-results-are-promising.md
>> I might have missed that, but there seem to be no information about used
>> - compiler(s) and their versions
>> - compilation flags
>>
>> Best regards,
>
> No you're right, that information is missing. I was using Visual Studio 15.2
>
> /GS /GL /analyze- /W3 /Gy /Zc:wchar_t /Zi /Gm- /O2 /sdl
> /Fd"Release\vc141.pdb" /Zc:inline /fp:precise /D "_MBCS"
> /errorReport:prompt /WX- /Zc:forScope /Gd /Oy- /Oi /MD /std:c++latest
> /Fa"Release\" /EHsc /nologo /Fo"Release\" /Fp"Release\raster.pch"
> /diagnostics:classic

Looks OK, close to or same as the defaults for VS optimised builds.

I was wondering if the slow performance may be due to the standard iterators
debugging enabled.


Best regards,
--
Mateusz Loskot, http://mateusz.loskot.net
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Help with idiomatic GDAL solution for raster algebra benchmark

Alex HighViz
On 19/06/2018 18:41, Mateusz Loskot wrote:

> On 19 June 2018 at 11:50, Alex HighViz <[hidden email]> wrote:
>> On 19/06/2018 10:28, Mateusz Loskot wrote:
>>> On 19 June 2018 at 11:22, Alex HighViz <[hidden email]> wrote:
>>>> Regrettably I overstated the performance in my previous post, due to a bug that is now solved. With the solved bug, Pronto is about 50% slower than GDAL directly. I believe this is still good considering the benefits, but not as glamorous as I first thought. I have also applied it on a set of 100000 x 1000 rasters and the results are consistent.
>>>>
>>>>
>>>>
>>>> I was wondering if the slow performance may be due to the standard iterators
>>>> debugging enabled.
>>>>
>>>>
>>>> Best regards,

I have looked at the benchmark a bit further, and considered that it
might have to do with the way in which the different solutions access
the data.

The Pronto Raster code uses band->GetLockedBlockRef(...), whereas my the
GDAL-only reference uses band->ReadBlock(...).

This means that Pronto uses the cache and the GDAL-only reference
doesn't.  My next thought was to use band->RasterIO(...) as the
reference because that is more appropriate, considering that it also
uses the cache. And finally, I used band->GetLockedBlockRef() in a GDAL
only solution.

I ended up with the following timings:

GDAL-Only using ReadBlock(...)                         :   7.7 seconds
GDAL-Only using RasterIO(...)                            : 30.1 seconds
GDAL-Only using GetLockedBlockRef(...)         :   8.9 seconds
Pronto Raster using GetLockedBlockRef(...)     : 10.6 seconds

I suppose the relatively poor performance of RasterIO is known
(https://lists.osgeo.org/pipermail/gdal-dev/2008-March/016357.html and
https://trac.osgeo.org/gdal/ticket/2266), although I don't understand
the reason for it.






_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Help with idiomatic GDAL solution for raster algebra benchmark

Even Rouault-2
> I ended up with the following timings:
>
> GDAL-Only using ReadBlock(...)                         :   7.7 seconds
> GDAL-Only using RasterIO(...)                            : 30.1 seconds
> GDAL-Only using GetLockedBlockRef(...)         :   8.9 seconds
> Pronto Raster using GetLockedBlockRef(...)     : 10.6 seconds
>
> I suppose the relatively poor performance of RasterIO is known
> (https://lists.osgeo.org/pipermail/gdal-dev/2008-March/016357.html and
> https://trac.osgeo.org/gdal/ticket/2266), although I don't understand
> the reason for it.

The main reason is that RasterIO() involves an extra memcpy() from the block
cache buffer to the user provided buffer.

--
Spatialys - Geospatial professional services
http://www.spatialys.com
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: Help with idiomatic GDAL solution for raster algebra benchmark

Alex HighViz
In reply to this post by Even Rouault-2

From: Alex Highviz <[hidden email]>
Sent: 22 June 2018 18:27:14
To: [hidden email]
Subject: Re: [gdal-dev] Help with idiomatic GDAL solution for raster algebra benchmark
 


On 14/06/2018 21:22, Even Rouault wrote:
>
> [ ]
> - 1000x1000 is somewhat small. Perhaps benchmark on larger rasters
>
> [ ]
> - it would be interested if you could also benchmark a VRT with Python numpy-
> based pixel function, and enable Numba. See
> http://www.gdal.org/gdal_vrttut.html#gdal_vrttut_derived_python (or just pure
> numpy based computation + Numba)
>
> [ ]
> Even
>
Hi Even,

I tried a few of your suggestions and am pleasantly surprised. These
results are for 10000  x 10000 matrices

VRT with C++ function : 10.1 seconds

VRT with NUMPY function (no Numba): 9.2 seconds

GDAL Direct using GetLockedBlockRef: 8.7 seconds

GDAL Direct using RasterIO: 30.7 seconds

Pronto Raster: 10.8 seconds

Just FYI

_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev