Mapserver causes too many open files error

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

Mapserver causes too many open files error

Tamas Szekeres
Hi Devs,

We experienced an issue with mapserver 7.x (comparing to 6.x) that it causes too many open file handles in a multi-threaded environment. Actually the process is using the ASP.NET backend where the web requests are served through the ASP.NET worker process by using a thread pool, and the process should be considered as a long running process, where msCleanup is not being called regularly.

By looking into the code the issue should probably be dedicated to the global font cache approach implemented in mapserver 7. Since freetype keeps the font files open, we get the number of fonts used in the mapfile multiplied with the thread pool size which may exceed the number of stdio open file handle limit as per https://msdn.microsoft.com/en-us/library/kdfaxaay(v=vs.140).aspx

With the former approach (ie. 6.x) the font cache was implemented in the renderers and the cache cleanup was happening on the renderer cleanup in msFreeOutputFormat/msFreeMap.

Do you have any suggestions about a proper workaround to this?

I might think that we should either bind the font cache to each map objects (like earlier) or use a single common font cache for the entire process. I would probably prefer the first one as soon as we want to clean up every resources what a rendering session allocates.

Thanks,

Tamas



_______________________________________________
mapserver-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/mapserver-dev
Reply | Threaded
Open this post in threaded view
|

Re: Mapserver causes too many open files error

Lime, Steve D (MNIT)

The former is certainly simpler. How expensive is setting up and clearing out the cache? Probably should look at Thomas’ RFC on the topic.

 

From: mapserver-dev [mailto:[hidden email]] On Behalf Of Tamas Szekeres
Sent: Sunday, November 26, 2017 2:01 PM
To: [hidden email]
Subject: [mapserver-dev] Mapserver causes too many open files error

 

Hi Devs,

 

We experienced an issue with mapserver 7.x (comparing to 6.x) that it causes too many open file handles in a multi-threaded environment. Actually the process is using the ASP.NET backend where the web requests are served through the ASP.NET worker process by using a thread pool, and the process should be considered as a long running process, where msCleanup is not being called regularly.

 

By looking into the code the issue should probably be dedicated to the global font cache approach implemented in mapserver 7. Since freetype keeps the font files open, we get the number of fonts used in the mapfile multiplied with the thread pool size which may exceed the number of stdio open file handle limit as per https://msdn.microsoft.com/en-us/library/kdfaxaay(v=vs.140).aspx

 

With the former approach (ie. 6.x) the font cache was implemented in the renderers and the cache cleanup was happening on the renderer cleanup in msFreeOutputFormat/msFreeMap.

 

Do you have any suggestions about a proper workaround to this?

 

I might think that we should either bind the font cache to each map objects (like earlier) or use a single common font cache for the entire process. I would probably prefer the first one as soon as we want to clean up every resources what a rendering session allocates.

 

Thanks,

 

Tamas

 

 


_______________________________________________
mapserver-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/mapserver-dev
Reply | Threaded
Open this post in threaded view
|

Re: Mapserver causes too many open files error

Tamas Szekeres
We did some investigation in this regard and the following 2 changes could solve this specific problem on Windows and ASP.NET.

1. Adding the ability to use a single cache instead of a per-thread cache. 

2. Increase the number of open stdio file handle limit for Windows. Default is 512 the upper limit is 2048 accoding to this documentation: https://msdn.microsoft.com/en-us/library/6e3b887c.aspx  


These settings should probably be controlled with compile options or environment variables.

Any objections to apply such changes?

Best regards,

Tamas






2017-11-27 16:42 GMT+01:00 Lime, Steve D (MNIT) <[hidden email]>:

The former is certainly simpler. How expensive is setting up and clearing out the cache? Probably should look at Thomas’ RFC on the topic.

 

From: mapserver-dev [mailto:[hidden email]] On Behalf Of Tamas Szekeres
Sent: Sunday, November 26, 2017 2:01 PM
To: [hidden email]
Subject: [mapserver-dev] Mapserver causes too many open files error

 

Hi Devs,

 

We experienced an issue with mapserver 7.x (comparing to 6.x) that it causes too many open file handles in a multi-threaded environment. Actually the process is using the ASP.NET backend where the web requests are served through the ASP.NET worker process by using a thread pool, and the process should be considered as a long running process, where msCleanup is not being called regularly.

 

By looking into the code the issue should probably be dedicated to the global font cache approach implemented in mapserver 7. Since freetype keeps the font files open, we get the number of fonts used in the mapfile multiplied with the thread pool size which may exceed the number of stdio open file handle limit as per https://msdn.microsoft.com/en-us/library/kdfaxaay(v=vs.140).aspx

 

With the former approach (ie. 6.x) the font cache was implemented in the renderers and the cache cleanup was happening on the renderer cleanup in msFreeOutputFormat/msFreeMap.

 

Do you have any suggestions about a proper workaround to this?

 

I might think that we should either bind the font cache to each map objects (like earlier) or use a single common font cache for the entire process. I would probably prefer the first one as soon as we want to clean up every resources what a rendering session allocates.

 

Thanks,

 

Tamas

 

 



_______________________________________________
mapserver-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/mapserver-dev
Reply | Threaded
Open this post in threaded view
|

Re: Mapserver causes too many open files error

Lime, Steve D (MNIT)

What does #1 entail? The other change is Windows specific so seems ok – I don’t know what the right value should be of course.

 

From: Tamas Szekeres [mailto:[hidden email]]
Sent: Friday, December 01, 2017 1:53 AM
To: Lime, Steve D (MNIT) <[hidden email]>
Cc: [hidden email]
Subject: Re: [mapserver-dev] Mapserver causes too many open files error

 

We did some investigation in this regard and the following 2 changes could solve this specific problem on Windows and ASP.NET.

 

1. Adding the ability to use a single cache instead of a per-thread cache. 

 

2. Increase the number of open stdio file handle limit for Windows. Default is 512 the upper limit is 2048 accoding to this documentation: https://msdn.microsoft.com/en-us/library/6e3b887c.aspx  

 

 

These settings should probably be controlled with compile options or environment variables.

 

Any objections to apply such changes?

 

Best regards,

 

Tamas

 

 

 

 

 

 

2017-11-27 16:42 GMT+01:00 Lime, Steve D (MNIT) <[hidden email]>:

The former is certainly simpler. How expensive is setting up and clearing out the cache? Probably should look at Thomas’ RFC on the topic.

 

From: mapserver-dev [mailto:[hidden email]] On Behalf Of Tamas Szekeres
Sent: Sunday, November 26, 2017 2:01 PM
To: [hidden email]
Subject: [mapserver-dev] Mapserver causes too many open files error

 

Hi Devs,

 

We experienced an issue with mapserver 7.x (comparing to 6.x) that it causes too many open file handles in a multi-threaded environment. Actually the process is using the ASP.NET backend where the web requests are served through the ASP.NET worker process by using a thread pool, and the process should be considered as a long running process, where msCleanup is not being called regularly.

 

By looking into the code the issue should probably be dedicated to the global font cache approach implemented in mapserver 7. Since freetype keeps the font files open, we get the number of fonts used in the mapfile multiplied with the thread pool size which may exceed the number of stdio open file handle limit as per https://msdn.microsoft.com/en-us/library/kdfaxaay(v=vs.140).aspx

 

With the former approach (ie. 6.x) the font cache was implemented in the renderers and the cache cleanup was happening on the renderer cleanup in msFreeOutputFormat/msFreeMap.

 

Do you have any suggestions about a proper workaround to this?

 

I might think that we should either bind the font cache to each map objects (like earlier) or use a single common font cache for the entire process. I would probably prefer the first one as soon as we want to clean up every resources what a rendering session allocates.

 

Thanks,

 

Tamas

 

 

 


_______________________________________________
mapserver-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/mapserver-dev
Reply | Threaded
Open this post in threaded view
|

Re: Mapserver causes too many open files error

Tamas Szekeres
This change demonstrates what we need in #1


For #2 this is the example:


Both of them should be switchable which is not yet added to this code example.

Best regards,

Tamas



2017-12-01 15:48 GMT+01:00 Lime, Steve D (MNIT) <[hidden email]>:

What does #1 entail? The other change is Windows specific so seems ok – I don’t know what the right value should be of course.

 

From: Tamas Szekeres [mailto:[hidden email]]
Sent: Friday, December 01, 2017 1:53 AM
To: Lime, Steve D (MNIT) <[hidden email]>
Cc: [hidden email]
Subject: Re: [mapserver-dev] Mapserver causes too many open files error

 

We did some investigation in this regard and the following 2 changes could solve this specific problem on Windows and ASP.NET.

 

1. Adding the ability to use a single cache instead of a per-thread cache. 

 

2. Increase the number of open stdio file handle limit for Windows. Default is 512 the upper limit is 2048 accoding to this documentation: https://msdn.microsoft.com/en-us/library/6e3b887c.aspx  

 

 

These settings should probably be controlled with compile options or environment variables.

 

Any objections to apply such changes?

 

Best regards,

 

Tamas

 

 

 

 

 

 

2017-11-27 16:42 GMT+01:00 Lime, Steve D (MNIT) <[hidden email]>:

The former is certainly simpler. How expensive is setting up and clearing out the cache? Probably should look at Thomas’ RFC on the topic.

 

From: mapserver-dev [mailto:[hidden email]] On Behalf Of Tamas Szekeres
Sent: Sunday, November 26, 2017 2:01 PM
To: [hidden email]
Subject: [mapserver-dev] Mapserver causes too many open files error

 

Hi Devs,

 

We experienced an issue with mapserver 7.x (comparing to 6.x) that it causes too many open file handles in a multi-threaded environment. Actually the process is using the ASP.NET backend where the web requests are served through the ASP.NET worker process by using a thread pool, and the process should be considered as a long running process, where msCleanup is not being called regularly.

 

By looking into the code the issue should probably be dedicated to the global font cache approach implemented in mapserver 7. Since freetype keeps the font files open, we get the number of fonts used in the mapfile multiplied with the thread pool size which may exceed the number of stdio open file handle limit as per https://msdn.microsoft.com/en-us/library/kdfaxaay(v=vs.140).aspx

 

With the former approach (ie. 6.x) the font cache was implemented in the renderers and the cache cleanup was happening on the renderer cleanup in msFreeOutputFormat/msFreeMap.

 

Do you have any suggestions about a proper workaround to this?

 

I might think that we should either bind the font cache to each map objects (like earlier) or use a single common font cache for the entire process. I would probably prefer the first one as soon as we want to clean up every resources what a rendering session allocates.

 

Thanks,

 

Tamas

 

 

 



_______________________________________________
mapserver-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/mapserver-dev
Reply | Threaded
Open this post in threaded view
|

Re: Mapserver causes too many open files error

Lime, Steve D (MNIT)

Seem relatively minor. Couple of questions:

 

·         Since changes for item 1 would apply more generally, is there any downside for FastCGI or should there be a 3rd state where you have threads AND are using Windows (or is that what you mean by being switchable).

·         Why the symbol change from USE_THREAD to USE_THREAD_NOTSET?

 

Steve

 

From: Tamas Szekeres [mailto:[hidden email]]
Sent: Friday, December 01, 2017 9:51 AM
To: Lime, Steve D (MNIT) <[hidden email]>
Cc: [hidden email]
Subject: Re: [mapserver-dev] Mapserver causes too many open files error

 

This change demonstrates what we need in #1

 

 

For #2 this is the example:

 

 

Both of them should be switchable which is not yet added to this code example.

 

Best regards,

 

Tamas

 

 

 

2017-12-01 15:48 GMT+01:00 Lime, Steve D (MNIT) <[hidden email]>:

What does #1 entail? The other change is Windows specific so seems ok – I don’t know what the right value should be of course.

 

From: Tamas Szekeres [mailto:[hidden email]]
Sent: Friday, December 01, 2017 1:53 AM
To: Lime, Steve D (MNIT) <[hidden email]>
Cc: [hidden email]
Subject: Re: [mapserver-dev] Mapserver causes too many open files error

 

We did some investigation in this regard and the following 2 changes could solve this specific problem on Windows and ASP.NET.

 

1. Adding the ability to use a single cache instead of a per-thread cache. 

 

2. Increase the number of open stdio file handle limit for Windows. Default is 512 the upper limit is 2048 accoding to this documentation: https://msdn.microsoft.com/en-us/library/6e3b887c.aspx  

 

 

These settings should probably be controlled with compile options or environment variables.

 

Any objections to apply such changes?

 

Best regards,

 

Tamas

 

 

 

 

 

 

2017-11-27 16:42 GMT+01:00 Lime, Steve D (MNIT) <[hidden email]>:

The former is certainly simpler. How expensive is setting up and clearing out the cache? Probably should look at Thomas’ RFC on the topic.

 

From: mapserver-dev [mailto:[hidden email]] On Behalf Of Tamas Szekeres
Sent: Sunday, November 26, 2017 2:01 PM
To: [hidden email]
Subject: [mapserver-dev] Mapserver causes too many open files error

 

Hi Devs,

 

We experienced an issue with mapserver 7.x (comparing to 6.x) that it causes too many open file handles in a multi-threaded environment. Actually the process is using the ASP.NET backend where the web requests are served through the ASP.NET worker process by using a thread pool, and the process should be considered as a long running process, where msCleanup is not being called regularly.

 

By looking into the code the issue should probably be dedicated to the global font cache approach implemented in mapserver 7. Since freetype keeps the font files open, we get the number of fonts used in the mapfile multiplied with the thread pool size which may exceed the number of stdio open file handle limit as per https://msdn.microsoft.com/en-us/library/kdfaxaay(v=vs.140).aspx

 

With the former approach (ie. 6.x) the font cache was implemented in the renderers and the cache cleanup was happening on the renderer cleanup in msFreeOutputFormat/msFreeMap.

 

Do you have any suggestions about a proper workaround to this?

 

I might think that we should either bind the font cache to each map objects (like earlier) or use a single common font cache for the entire process. I would probably prefer the first one as soon as we want to clean up every resources what a rendering session allocates.

 

Thanks,

 

Tamas

 

 

 

 


_______________________________________________
mapserver-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/mapserver-dev
Reply | Threaded
Open this post in threaded view
|

Re: Mapserver causes too many open files error

Tamas Szekeres
Hi Steve,

Sorry for the confusion those commits were basically for testing the changes. 
I've now uploaded the commits I would prefer to see in mapserver/master:



With regards to the FastCGI configuration I think that should also work without the USE_THREAD setting which is not affected with the changes. But if we don't set these environment variables, we would get back the original behavior.

Best regards,

Tamas




2017-12-05 21:40 GMT+01:00 Lime, Steve D (MNIT) <[hidden email]>:

Seem relatively minor. Couple of questions:

 

·         Since changes for item 1 would apply more generally, is there any downside for FastCGI or should there be a 3rd state where you have threads AND are using Windows (or is that what you mean by being switchable).

·         Why the symbol change from USE_THREAD to USE_THREAD_NOTSET?

 

Steve

 

From: Tamas Szekeres [mailto:[hidden email]]
Sent: Friday, December 01, 2017 9:51 AM


To: Lime, Steve D (MNIT) <[hidden email]>
Cc: [hidden email]
Subject: Re: [mapserver-dev] Mapserver causes too many open files error

 

This change demonstrates what we need in #1

 

 

For #2 this is the example:

 

 

Both of them should be switchable which is not yet added to this code example.

 

Best regards,

 

Tamas

 

 

 

2017-12-01 15:48 GMT+01:00 Lime, Steve D (MNIT) <[hidden email]>:

What does #1 entail? The other change is Windows specific so seems ok – I don’t know what the right value should be of course.

 

From: Tamas Szekeres [mailto:[hidden email]]
Sent: Friday, December 01, 2017 1:53 AM
To: Lime, Steve D (MNIT) <[hidden email]>
Cc: [hidden email]
Subject: Re: [mapserver-dev] Mapserver causes too many open files error

 

We did some investigation in this regard and the following 2 changes could solve this specific problem on Windows and ASP.NET.

 

1. Adding the ability to use a single cache instead of a per-thread cache. 

 

2. Increase the number of open stdio file handle limit for Windows. Default is 512 the upper limit is 2048 accoding to this documentation: https://msdn.microsoft.com/en-us/library/6e3b887c.aspx  

 

 

These settings should probably be controlled with compile options or environment variables.

 

Any objections to apply such changes?

 

Best regards,

 

Tamas

 

 

 

 

 

 

2017-11-27 16:42 GMT+01:00 Lime, Steve D (MNIT) <[hidden email]>:

The former is certainly simpler. How expensive is setting up and clearing out the cache? Probably should look at Thomas’ RFC on the topic.

 

From: mapserver-dev [mailto:[hidden email]] On Behalf Of Tamas Szekeres
Sent: Sunday, November 26, 2017 2:01 PM
To: [hidden email]
Subject: [mapserver-dev] Mapserver causes too many open files error

 

Hi Devs,

 

We experienced an issue with mapserver 7.x (comparing to 6.x) that it causes too many open file handles in a multi-threaded environment. Actually the process is using the ASP.NET backend where the web requests are served through the ASP.NET worker process by using a thread pool, and the process should be considered as a long running process, where msCleanup is not being called regularly.

 

By looking into the code the issue should probably be dedicated to the global font cache approach implemented in mapserver 7. Since freetype keeps the font files open, we get the number of fonts used in the mapfile multiplied with the thread pool size which may exceed the number of stdio open file handle limit as per https://msdn.microsoft.com/en-us/library/kdfaxaay(v=vs.140).aspx

 

With the former approach (ie. 6.x) the font cache was implemented in the renderers and the cache cleanup was happening on the renderer cleanup in msFreeOutputFormat/msFreeMap.

 

Do you have any suggestions about a proper workaround to this?

 

I might think that we should either bind the font cache to each map objects (like earlier) or use a single common font cache for the entire process. I would probably prefer the first one as soon as we want to clean up every resources what a rendering session allocates.

 

Thanks,

 

Tamas

 

 

 

 



_______________________________________________
mapserver-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/mapserver-dev
Reply | Threaded
Open this post in threaded view
|

Re: Mapserver causes too many open files error

Lime, Steve D (MNIT)

I’m ok with the changes… +1  Anyone else?

 

From: Tamas Szekeres [mailto:[hidden email]]
Sent: Sunday, December 10, 2017 12:06 PM
To: Lime, Steve D (MNIT) <[hidden email]>
Cc: [hidden email]
Subject: Re: [mapserver-dev] Mapserver causes too many open files error

 

Hi Steve,

 

Sorry for the confusion those commits were basically for testing the changes. 

I've now uploaded the commits I would prefer to see in mapserver/master:

 

 

 

With regards to the FastCGI configuration I think that should also work without the USE_THREAD setting which is not affected with the changes. But if we don't set these environment variables, we would get back the original behavior.

 

Best regards,

 

Tamas

 

 

 

 

2017-12-05 21:40 GMT+01:00 Lime, Steve D (MNIT) <[hidden email]>:

Seem relatively minor. Couple of questions:

 

·         Since changes for item 1 would apply more generally, is there any downside for FastCGI or should there be a 3rd state where you have threads AND are using Windows (or is that what you mean by being switchable).

·         Why the symbol change from USE_THREAD to USE_THREAD_NOTSET?

 

Steve

 

From: Tamas Szekeres [mailto:[hidden email]]
Sent: Friday, December 01, 2017 9:51 AM


To: Lime, Steve D (MNIT) <[hidden email]>
Cc: [hidden email]
Subject: Re: [mapserver-dev] Mapserver causes too many open files error

 

This change demonstrates what we need in #1

 

 

For #2 this is the example:

 

 

Both of them should be switchable which is not yet added to this code example.

 

Best regards,

 

Tamas

 

 

 

2017-12-01 15:48 GMT+01:00 Lime, Steve D (MNIT) <[hidden email]>:

What does #1 entail? The other change is Windows specific so seems ok – I don’t know what the right value should be of course.

 

From: Tamas Szekeres [mailto:[hidden email]]
Sent: Friday, December 01, 2017 1:53 AM
To: Lime, Steve D (MNIT) <[hidden email]>
Cc: [hidden email]
Subject: Re: [mapserver-dev] Mapserver causes too many open files error

 

We did some investigation in this regard and the following 2 changes could solve this specific problem on Windows and ASP.NET.

 

1. Adding the ability to use a single cache instead of a per-thread cache. 

 

2. Increase the number of open stdio file handle limit for Windows. Default is 512 the upper limit is 2048 accoding to this documentation: https://msdn.microsoft.com/en-us/library/6e3b887c.aspx  

 

 

These settings should probably be controlled with compile options or environment variables.

 

Any objections to apply such changes?

 

Best regards,

 

Tamas

 

 

 

 

 

 

2017-11-27 16:42 GMT+01:00 Lime, Steve D (MNIT) <[hidden email]>:

The former is certainly simpler. How expensive is setting up and clearing out the cache? Probably should look at Thomas’ RFC on the topic.

 

From: mapserver-dev [mailto:[hidden email]] On Behalf Of Tamas Szekeres
Sent: Sunday, November 26, 2017 2:01 PM
To: [hidden email]
Subject: [mapserver-dev] Mapserver causes too many open files error

 

Hi Devs,

 

We experienced an issue with mapserver 7.x (comparing to 6.x) that it causes too many open file handles in a multi-threaded environment. Actually the process is using the ASP.NET backend where the web requests are served through the ASP.NET worker process by using a thread pool, and the process should be considered as a long running process, where msCleanup is not being called regularly.

 

By looking into the code the issue should probably be dedicated to the global font cache approach implemented in mapserver 7. Since freetype keeps the font files open, we get the number of fonts used in the mapfile multiplied with the thread pool size which may exceed the number of stdio open file handle limit as per https://msdn.microsoft.com/en-us/library/kdfaxaay(v=vs.140).aspx

 

With the former approach (ie. 6.x) the font cache was implemented in the renderers and the cache cleanup was happening on the renderer cleanup in msFreeOutputFormat/msFreeMap.

 

Do you have any suggestions about a proper workaround to this?

 

I might think that we should either bind the font cache to each map objects (like earlier) or use a single common font cache for the entire process. I would probably prefer the first one as soon as we want to clean up every resources what a rendering session allocates.

 

Thanks,

 

Tamas

 

 

 

 

 


_______________________________________________
mapserver-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/mapserver-dev
Reply | Threaded
Open this post in threaded view
|

Re: Mapserver causes too many open files error

jmckenna
Administrator
+1 here.  Thanks Tamas.



-jeff



On 2017-12-11 5:49 PM, Lime, Steve D (MNIT) wrote:

> I’m ok with the changes… +1  Anyone else?
>
> *From:*Tamas Szekeres [mailto:[hidden email]]
> *Sent:* Sunday, December 10, 2017 12:06 PM
> *To:* Lime, Steve D (MNIT) <[hidden email]>
> *Cc:* [hidden email]
> *Subject:* Re: [mapserver-dev] Mapserver causes too many open files error
>
> Hi Steve,
>
> Sorry for the confusion those commits were basically for testing the
> changes.
>
> I've now uploaded the commits I would prefer to see in mapserver/master:
>
> https://github.com/szekerest/mapserver/commit/def90bfed47abaaba7ec12f49eb218494a39cbdc
>
> https://github.com/szekerest/mapserver/commit/b74fd63f6838f1cf20cee7f1951e8cbac8da443e
>
> With regards to the FastCGI configuration I think that should also work
> without the USE_THREAD setting which is not affected with the changes.
> But if we don't set these environment variables, we would get back the
> original behavior.
>
> Best regards,
>
> Tamas
>
> 2017-12-05 21:40 GMT+01:00 Lime, Steve D (MNIT) <[hidden email]
> <mailto:[hidden email]>>:
>
>     Seem relatively minor. Couple of questions:
>
>     ·Since changes for item 1 would apply more generally, is there any
>     downside for FastCGI or should there be a 3^rd state where you have
>     threads AND are using Windows (or is that what you mean by being
>     switchable).
>
>     ·Why the symbol change from USE_THREAD to USE_THREAD_NOTSET?
>
>     Steve
>
>     *From:*Tamas Szekeres [mailto:[hidden email]
>     <mailto:[hidden email]>]
>     *Sent:* Friday, December 01, 2017 9:51 AM
>
>
>     *To:* Lime, Steve D (MNIT) <[hidden email]
>     <mailto:[hidden email]>>
>     *Cc:* [hidden email]
>     <mailto:[hidden email]>
>     *Subject:* Re: [mapserver-dev] Mapserver causes too many open files
>     error
>
>     This change demonstrates what we need in #1
>
>     https://github.com/szekerest/mapserver/commit/87cacc6a37e802bc3af16656d45966c05deb4d62
>
>     For #2 this is the example:
>
>     https://github.com/szekerest/mapserver/commit/e16069972b43f837b47d0f736c2df4d5c0c7a475
>
>     Both of them should be switchable which is not yet added to this
>     code example.
>
>     Best regards,
>
>     Tamas
>
>     2017-12-01 15:48 GMT+01:00 Lime, Steve D (MNIT)
>     <[hidden email] <mailto:[hidden email]>>:
>
>         What does #1 entail? The other change is Windows specific so
>         seems ok – I don’t know what the right value should be of course.
>
>         *From:*Tamas Szekeres [mailto:[hidden email]
>         <mailto:[hidden email]>]
>         *Sent:* Friday, December 01, 2017 1:53 AM
>         *To:* Lime, Steve D (MNIT) <[hidden email]
>         <mailto:[hidden email]>>
>         *Cc:* [hidden email]
>         <mailto:[hidden email]>
>         *Subject:* Re: [mapserver-dev] Mapserver causes too many open
>         files error
>
>         We did some investigation in this regard and the following 2
>         changes could solve this specific problem on Windows and ASP.NET
>         <http://ASP.NET>.
>
>         1. Adding the ability to use a single cache instead of a
>         per-thread cache.
>
>         2. Increase the number of open stdio file handle limit for
>         Windows. Default is 512 the upper limit is 2048 accoding to this
>         documentation:
>         https://msdn.microsoft.com/en-us/library/6e3b887c.aspx
>
>         These settings should probably be controlled with compile
>         options or environment variables.
>
>         Any objections to apply such changes?
>
>         Best regards,
>
>         Tamas
>
>         2017-11-27 16:42 GMT+01:00 Lime, Steve D (MNIT)
>         <[hidden email] <mailto:[hidden email]>>:
>
>             The former is certainly simpler. How expensive is setting up
>             and clearing out the cache? Probably should look at Thomas’
>             RFC on the topic.
>
>             *From:*mapserver-dev
>             [mailto:[hidden email]
>             <mailto:[hidden email]>] *On Behalf
>             Of *Tamas Szekeres
>             *Sent:* Sunday, November 26, 2017 2:01 PM
>             *To:* [hidden email]
>             <mailto:[hidden email]>
>             *Subject:* [mapserver-dev] Mapserver causes too many open
>             files error
>
>             Hi Devs,
>
>             We experienced an issue with mapserver 7.x (comparing to
>             6.x) that it causes too many open file handles in a
>             multi-threaded environment. Actually the process is using
>             the ASP.NET <http://ASP.NET> backend where the web requests
>             are served through the ASP.NET <http://ASP.NET> worker
>             process by using a thread pool, and the process should be
>             considered as a long running process, where msCleanup is not
>             being called regularly.
>
>             By looking into the code the issue should probably be
>             dedicated to the global font cache approach implemented in
>             mapserver 7. Since freetype keeps the font files open, we
>             get the number of fonts used in the mapfile multiplied with
>             the thread pool size which may exceed the number of stdio
>             open file handle limit as per
>             https://msdn.microsoft.com/en-us/library/kdfaxaay(v=vs.140).aspx
>
>             With the former approach (ie. 6.x) the font cache was
>             implemented in the renderers and the cache cleanup was
>             happening on the renderer cleanup
>             in msFreeOutputFormat/msFreeMap.
>
>             Do you have any suggestions about a proper workaround to this?
>
>             I might think that we should either bind the font cache to
>             each map objects (like earlier) or use a single common font
>             cache for the entire process. I would probably prefer the
>             first one as soon as we want to clean up every resources
>             what a rendering session allocates.
>
>             Thanks,
>
>             Tamas
>
>
_______________________________________________
mapserver-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/mapserver-dev
Reply | Threaded
Open this post in threaded view
|

Re: Mapserver causes too many open files error

wicksell
+1

We had the same problem with too many open files when running a
multi-threaded .NET application using MapScript.

We tried the patches provided by Tamas but started to get a lot of errors
similar to
msGetGlyphByIndex(): General error message. unable to load glyph 36d for
font "arial"

We solved this by adding locks to msGetGlyphByIndex() similar to those used
in msGetGlyphOutline(). I guess there should be locking in msGetGlyphIndex()
too.

/Krister





--
Sent from: http://osgeo-org.1560.x6.nabble.com/Mapserver-Dev-f4252683.html
_______________________________________________
mapserver-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/mapserver-dev