Higher Memory Usage with vsicurl and gdal > 2.2.4

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Higher Memory Usage with vsicurl and gdal > 2.2.4

Peter Schmitt
Hi,

I run MapServer in a Docker container. I noticed its memory utilization climbed much higher than expected. This seems to happen only under the following conditions:

* Files are accessed network-based filesystems (I tried both /vsicurl/ and /vsis3/)
* Using gdal  > 2.2.4 (tested with 2.3.0, 2.3.2 & 2.4.1.  I see the same behavior with both MapServer-7.0.7 and MapServer-7.2.2.

I had a hard time isolating this to GDAL and could not reproduce with similar gdal_translate commands.  So I put together a little example to try and demonstrate the problem.

Does anyone have any ideas as to what might be going on? I tried various cache settings, which do not seem to reduce memory utilization:

_______________________________________________
mapserver-users mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/mapserver-users
Reply | Threaded
Open this post in threaded view
|

Re: Higher Memory Usage with vsicurl and gdal > 2.2.4

Peter Schmitt
I solved my problem... I found a way to reduce memory utilization and allocations when using MapServer with GDAL >= 2.3.0: 

     CONFIG "GDAL_HTTP_MULTIRANGE"       "SERIAL"

With this setting, the memory footprint matches that of GDAL <= 2.2.4.

With the default (YES) has mapserv allocates much more memory. Running mapserv with supervisor & configured over FCGI results in a much higher memory footprint over time. My best guess is that with the default parallel mode has many HTTP connections kept alive and each connection consumes some non-trivial amount of memory.  With CONFIG "CPL_CURL_VERBOSE" "ON", I see 9 HTTP connections kept around only with the default GDAL_HTTP_MULTIRANGE=ON setting:

* Curl_http_done: called premature == 0
* Connection #17 to host s3.amazonaws.com left intact
* Curl_http_done: called premature == 0
* Connection #11 to host s3.amazonaws.com left intact
* Curl_http_done: called premature == 0
* Connection #13 to host s3.amazonaws.com left intact
* Curl_http_done: called premature == 0
* Connection #14 to host s3.amazonaws.com left intact
* Curl_http_done: called premature == 0
* Connection #15 to host s3.amazonaws.com left intact
* Curl_http_done: called premature == 0
* Connection #19 to host s3.amazonaws.com left intact
* Curl_http_done: called premature == 0
* Connection #16 to host s3.amazonaws.com left intact
* Curl_http_done: called premature == 0
* Connection #18 to host s3.amazonaws.com left intact
* Curl_http_done: called premature == 0
* Connection #12 to host s3.amazonaws.com left intact

Side-note: the comments in cpl_vsil_curl.cpp for GDAL_HTTP_MULTIRANGE do not match the ConfigOptions documentation 
The valid settings are actually SINGLE_GET, SERIAL or _anything_ else for "parallel" mode

It's great to see all the vsicurl improvements to GDAL which MapServer can leverage!  Time for me to experiment with the other gdal-2.3 config options for libcurl >= 7.33 built with nghttp2.

Cheers,
Pete

On Fri, Mar 29, 2019 at 3:02 PM Peter Schmitt <[hidden email]> wrote:
Hi,

I run MapServer in a Docker container. I noticed its memory utilization climbed much higher than expected. This seems to happen only under the following conditions:

* Files are accessed network-based filesystems (I tried both /vsicurl/ and /vsis3/)
* Using gdal  > 2.2.4 (tested with 2.3.0, 2.3.2 & 2.4.1.  I see the same behavior with both MapServer-7.0.7 and MapServer-7.2.2.

I had a hard time isolating this to GDAL and could not reproduce with similar gdal_translate commands.  So I put together a little example to try and demonstrate the problem.

Does anyone have any ideas as to what might be going on? I tried various cache settings, which do not seem to reduce memory utilization:


--
Pete

_______________________________________________
mapserver-users mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/mapserver-users