[gdal-dev] Memory utilization and GDAL_HTTP_MULTIRANGE

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

[gdal-dev] Memory utilization and GDAL_HTTP_MULTIRANGE

Peter Schmitt
Hi,

I have been experimenting with the GDAL_HTTP_MULTIRANGE config option added in gdal-2.3.  I have noticed that setting this to YES requires fewer curl requests, but utilizes a more memory compared to SERIAL. From a very simple test (details below):

1. gdal-2.2.4:
        Maximum resident set size (kbytes): 45544
2. gdal-2.3.0 with GDAL_HTTP_MULTIRANGE=SERIAL
        Maximum resident set size (kbytes): 46120
3. gdal-2.3.0 with GDAL_HTTP_MULTIRANGE=YES
        Maximum resident set size (kbytes): 83828

The biggest thing I noticed from the log is that multiple HTTP connections are left intact when GDAL_HTTP_MULTIRANGE=YES. I'm guessing this is where the additional memory utilization comes from.  Is there any way to reduce the memory footprint when GDAL_HTTP_MULTIRANGE=YES?

I came across this while trying to understand why I was seeing a much higher memory footprint with a MapServer built with gdal >= 2.3 compared to MapServer built with gdal <= 2.2.4, detailed here: https://lists.osgeo.org/pipermail/mapserver-users/2019-April/081121.html

Thanks,
Pete

PS:  Here are my test details.  I ran a test with the following config options:

GDAL_DISABLE_READDIR_ON_OPEN=YES
CPL_VSIL_CURL_ALLOWED_EXTENSIONS=tif
GDAL_CACHEMAX=0
VSI_CACHE=FALSE
CPL_VSIL_CURL_NON_CACHED="/vsicurl/https://s3.amazonaws.com" CPL_CURL_VERBOSE=ON 

and the following command:
/usr/bin/time -v gdal_translate -outsize 256 256 -projwin_srs EPSG:3857 -projwin 13894417.251632 -1012637.7505811 13895028.747858 -1013249.2468073 /vsicurl/https://s3.amazonaws.com/pschmitt-test-public/b378087a-c2a5-43a0-abec-71fcfb051150_cog.tif /out.tif | grep "Maximum resident set size"

_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev