gdal_rasterize increase memory usage

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

gdal_rasterize increase memory usage

jramm
gdal_rasterize is limited to use just 10MB of memory (line ~640 of gdalrasterize.cpp).

Is there anyway to change this (without having to recompile?)

I'm noticing that changing the output data format from Byte to Int16 or even Int32 drastically reduces performance. This must be because the strict memory limits imposed by gdal_rasterize means that the amount of data it reads in one go is exponentially reduce each time I increase the integer size. Thus it has to loop through the polygon set many more times.

I'm seeing drastic slow down by just changing from Byte to Int16 when using gdal_rasterize (8-10 times slower)

Given that even the cheapest desktop has at least 1GB of RAM, isn't the scanline buffer size over conservative?
Reply | Threaded
Open this post in threaded view
|

Re: gdal_rasterize increase memory usage

Nicolas Cadieux
Hi,

I had the same problem with gdal_warp.  I recompiled the code using the latest ms visual studio community and the buffer was increased from 15 Mb to 1.5 GB.   It's was a first for me because I  usually work in python. It's easy to do.

That fixed all the issue.  I am told that buffer size is arbitrary and what may be good for one person may cause problems to another if, for example, a person launches many instances of the same program in parallel.

Best to recompile if the option is there. There is a page on the gdal web site explaining how to do it.  

Good luck

Nicolas Cadieux M.Sc.
Les Entreprises Archéotec inc. 
8548, rue Saint-Denis Montréal H2P 2H2
Téléphone: 514.381.5112  Fax: 514.381.4995
www.archeotec.ca

On Jul 16, 2015 05:15, jramm <[hidden email]> wrote:

>
> gdal_rasterize is limited to use just 10MB of memory (line ~640 of
> gdalrasterize.cpp).
>
> Is there anyway to change this (without having to recompile?)
>
> I'm noticing that changing the output data format from Byte to Int16 or even
> Int32 drastically reduces performance. This must be because the strict
> memory limits imposed by gdal_rasterize means that the amount of data it
> reads in one go is exponentially reduce each time I increase the integer
> size. Thus it has to loop through the polygon set many more times.
>
> I'm seeing drastic slow down by just changing from Byte to Int16 when using
> gdal_rasterize (8-10 times slower)
>
> Given that even the cheapest desktop has at least 1GB of RAM, isn't the
> scanline buffer size over conservative?
>
>
>
> --
> View this message in context: http://osgeo-org.1560.x6.nabble.com/gdal-rasterize-increase-memory-usage-tp5215862.html 
> Sent from the GDAL - Dev mailing list archive at Nabble.com.
> _______________________________________________
> gdal-dev mailing list
> [hidden email]
> http://lists.osgeo.org/mailman/listinfo/gdal-dev 
_______________________________________________
gdal-dev mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/gdal-dev