[gdal-dev] GDAL_CACHMAX on a docker container

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

[gdal-dev] GDAL_CACHMAX on a docker container

Guy Doulberg
Hi guys

I wanted to share with you a GOTCHA I struggled with in the passed few days, maybe it will be helpful for more people

I am using GDAL on a docker cluster, since there are more services running on the host I limit the container to use only 1GB of ram

The default behavior of GDAL_CACHMAX is 5% of the available memory, but apparently when running in above setup it sees the System memory, which 16GB and not 1GB. 

On that container I ran 3 process of GDAL which means it calculated it can use 2.4 GB approximately, but the container limitation was only 1GB

The result was that the OOM killer of the operating system killed those processes because they exceeded memory.

To solve to issue (to control the memory the process is going to use) I explictly set the GDAL_CACHMAX to an absolute value



_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: GDAL_CACHMAX on a docker container

Mateusz Loskot
On 23 May 2018 at 12:30, Guy Doulberg <[hidden email]> wrote:
>
> To solve to issue (to control the memory the process is going to use) I
> explictly set the GDAL_CACHMAX to an absolute value

The option name is GDAL_CACHEMAX

https://trac.osgeo.org/gdal/wiki/ConfigOptions

Best regards,
--
Mateusz Loskot, http://mateusz.loskot.net
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: GDAL_CACHMAX on a docker container

Even Rouault-2
In reply to this post by Guy Doulberg
On mercredi 23 mai 2018 13:30:30 CEST Guy Doulberg wrote:

> Hi guys
>
> I wanted to share with you a GOTCHA I struggled with in the passed few
> days, maybe it will be helpful for more people
>
> I am using GDAL on a docker cluster, since there are more services running
> on the host I limit the container to use only 1GB of ram
>
> The default behavior of GDAL_CACHMAX is 5% of the available memory, but
> apparently when running in above setup it sees the System memory, which
> 16GB and not 1GB.
>
> On that container I ran 3 process of GDAL which means it calculated it can
> use 2.4 GB approximately, but the container limitation was only 1GB
>
> The result was that the OOM killer of the operating system killed those
> processes because they exceeded memory.
>
> To solve to issue (to control the memory the process is going to use) I
> explictly set the GDAL_CACHMAX to an absolute value

Guy,

can you create an issue about that in https://github.com/OSGeo/gdal/issues ?

There are some hints to how this could be done:
https://github.com/dotnet/coreclr/blob/master/src/gc/unix/cgroup.cpp
https://stackoverflow.com/questions/42187085/check-mem-limit-within-a-docker-container

I just tried

$ docker run --memory 512m --rm -it ubuntu bash

root@0ae673d96789:/# cat /sys/fs/cgroup/memory/memory.limit_in_bytes
536870912

Can you check in your docker container that
cat /sys/fs/cgroup/memory/memory.limit_in_bytes
returns the maximum memory amount you specified ?

Even

--
Spatialys - Geospatial professional services
http://www.spatialys.com
_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: GDAL_CACHMAX on a docker container

Robert Coup
Hi,

On 23 May 2018 at 11:48, Even Rouault <[hidden email]> wrote:

There are some hints to how this could be done:
https://github.com/dotnet/coreclr/blob/master/src/gc/unix/cgroup.cpp
https://stackoverflow.com/questions/42187085/check-mem-limit-within-a-docker-container

I just tried

$ docker run --memory 512m --rm -it ubuntu bash

root@0ae673d96789:/# cat /sys/fs/cgroup/memory/memory.limit_in_bytes
536870912

Can you check in your docker container that
cat /sys/fs/cgroup/memory/memory.limit_in_bytes
returns the maximum memory amount you specified ?

Was just posting the same thing.

Yes, that approach works, and that's how the JVM does it now (https://bugs.openjdk.java.net/browse/JDK-8170888), and others.

If there's no memory limit set you get (on 64-bit linux) 9223372036854771712, currently recommendation is to take the minimum of the total system memory (eg. /proc/meminfo) and the cgroups memory.limit_in_bytes
 
Rob :)

_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Reply | Threaded
Open this post in threaded view
|

Re: GDAL_CACHMAX on a docker container

Guy Doulberg
Can you check in your docker container that
cat /sys/fs/cgroup/memory/memory.limit_in_bytes
returns the maximum memory amount you specified ?

yes it returns the memory I specified

On Wed, May 23, 2018 at 1:53 PM, Robert Coup <[hidden email]> wrote:
Hi,

On 23 May 2018 at 11:48, Even Rouault <[hidden email]> wrote:

There are some hints to how this could be done:
https://github.com/dotnet/coreclr/blob/master/src/gc/unix/cgroup.cpp
https://stackoverflow.com/questions/42187085/check-mem-limit-within-a-docker-container

I just tried

$ docker run --memory 512m --rm -it ubuntu bash

root@0ae673d96789:/# cat /sys/fs/cgroup/memory/memory.limit_in_bytes
536870912

Can you check in your docker container that
cat /sys/fs/cgroup/memory/memory.limit_in_bytes
returns the maximum memory amount you specified ?

Was just posting the same thing.

Yes, that approach works, and that's how the JVM does it now (https://bugs.openjdk.java.net/browse/JDK-8170888), and others.

If there's no memory limit set you get (on 64-bit linux) 9223372036854771712, currently recommendation is to take the minimum of the total system memory (eg. /proc/meminfo) and the cgroups memory.limit_in_bytes
 
Rob :)

_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev


_______________________________________________
gdal-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/gdal-dev