Is MapServer Thread-safe?

classic Classic list List threaded Threaded
26 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Is MapServer Thread-safe?

adrian kruk

After reading article  'Is MapServer Thread-safe?' in FAQ I have doubts.

My question is what means that some component is unsafe, I don't know architecture of mapserver and I don't need to know.

So listed unsafe components telling me nothing except my imagination what these components are doing.

Answer for the question need also some examples of code (e.g. some workarounds in C# with using mapscript_csharp)

I have this kind of situation:

 
I 'm created ASP.NET application and I'm using mapscript_csharp dll.
Each of browser clients will have its own map object instance stored in session cache. So each client request is creating new thread on IIS server.

So, new client must wait for its map object if mapscript is just creating map object for other client (other thread), even if server has many processors? Becouse of Map config file loading its global parser and it is locked. I Am wrong?

The same situation after some time. Two clients (two threads on server) requests for image calling map.draw(). This is unsafe situation becouse of 'some static data in Imagemap output'?




Regards,
Adrian Kruk
Reply | Threaded
Open this post in threaded view
|

Re: Is MapServer Thread-safe?

Tamas Szekeres
2008/1/21, adrian kruk <[hidden email]>:

>
>
> After reading article  'Is MapServer Thread-safe?' in FAQ I have doubts.
>
> My question is what means that some component is unsafe, I don't know
> architecture of mapserver and I don't need to know.
>
> So listed unsafe components telling me nothing except my imagination what
> these components are doing.
>

Hi,

Actually there are many places inside the code may use global
resources (eg. variables) that should be protected from the
simultaneous access from multiple threads. This may apply as well to
any of the dependent libraries (dll-s) that mapscript actually use.
If a dll is loaded by the application (by the ASP.NET host process in
this case), then these resources are shared among the threads within
the same host process.
Many of the shared reasouce accesses are protected by locks inside the
mapserver core if the USE_THEAD option is enabled during the
compilation. However there are a few mapserver components and
dependent libraries which may continue to use the global resources
without protection. Those components were declared as thread unsafe.

> Answer for the question need also some examples of code (e.g. some
> workarounds in C# with using mapscript_csharp)
>

The problem is not related to the C# interface, so you cannot control
the access to these resources from the c# interface. However you can
minimize the multi-thread access to the mapscript classes as much as
you can. For example you can resonstruct the map object in every
subsequent requests, or create a clone of the object stored in a
session variable at the beginning of each request.

In some cases we could possibly restructure the application for
creating different processes for the map generation (1 worker
thread/process accesses the mapscript objects) and set up a
communication protocol between the ASP.NET application and the process
pool.

> I have this kind of situation:
>
>
> I 'm created ASP.NET application and I'm using mapscript_csharp dll.
> Each of browser clients will have its own map object instance stored in
> session cache. So each client request is creating new thread on IIS server.
>
> So, new client must wait for its map object if mapscript is just creating
> map object for other client (other thread), even if server has many
> processors? Becouse of Map config file loading its global parser and it is
> locked. I Am wrong?
>

Yes, and it may affect the overall performance. However these cricital
sections may be quite limited in duration and the possibility that the
threads wait each other might not be significant in many cases.

> The same situation after some time. Two clients (two threads on server)
> requests for image calling map.draw(). This is unsafe situation becouse of
> 'some static data in Imagemap output'?
>

Using the unsafe portions of the code should be avoided. Accessing
unprotected resources by different threads may result in access
violation issues.


The most reasonable solution would be to reimplement the critical code
segments so as to allow multithread access, definitely. MS RFC 15 have
been addressed this goal
http://mapserver.gis.umn.edu/development/rfc/ms-rfc-15

However it would require much more effors to realize than expected before.

Best regards,

Tamas
Reply | Threaded
Open this post in threaded view
|

Re: Is MapServer Thread-safe?

adrian kruk
In reply to this post by adrian kruk
In my opinion this bug should be done as soon as possible. Maybe core
developers should consider to plan new release only for doing mapserver
multithreaded.
These is big limitation.

What about running MapServer as WMS server, is there the same problem with
performance with many clients running simultanously?
Reply | Threaded
Open this post in threaded view
|

Re: Is MapServer Thread-safe?

tbonfort
patches graciously accepted ;)

On Jan 22, 2008 9:22 AM, Adrian Kruk <[hidden email]> wrote:
> In my opinion this bug should be done as soon as possible. Maybe core
> developers should consider to plan new release only for doing mapserver
> multithreaded.
> These is big limitation.
>
> What about running MapServer as WMS server, is there the same problem with
> performance with many clients running simultanously?
>
Reply | Threaded
Open this post in threaded view
|

Re: Is MapServer Thread-safe?

Umberto Nicoletti
or funding... ;-)

On Jan 22, 2008 9:29 AM, thomas bonfort <[hidden email]> wrote:

> patches graciously accepted ;)
>
>
> On Jan 22, 2008 9:22 AM, Adrian Kruk <[hidden email]> wrote:
> > In my opinion this bug should be done as soon as possible. Maybe core
> > developers should consider to plan new release only for doing mapserver
> > multithreaded.
> > These is big limitation.
> >
> > What about running MapServer as WMS server, is there the same problem with
> > performance with many clients running simultanously?
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Is MapServer Thread-safe?

Tamas Szekeres
In reply to this post by adrian kruk
2008/1/22, Adrian Kruk <[hidden email]>:
> In my opinion this bug should be done as soon as possible. Maybe core
> developers should consider to plan new release only for doing mapserver
> multithreaded.
> These is big limitation.
>

The most fundamental components can be used safely right now that are
protected with locks. However a higher level of multithreading support
would be desired, but there's many things should be done for this
purpose that we cannot do out of contract like:

- Reviewing all the portions of the mapserver and the dependent libraries code
- Alter the method how the common resources are accessed (negotiate
the desired changes with the component owners)
- Create test applications that can cover most parts of the
mapserver/mapscript code.


> What about running MapServer as WMS server, is there the same problem with
> performance with many clients running simultanously?
>

I this case you probably run mapserver as a CGI or FCGI application,
that doesn't suffer from this issue since only one request is handled
at a given time within the same process.

Best regards,

Tamas
Reply | Threaded
Open this post in threaded view
|

Re: Is MapServer Thread-safe?

adrian kruk
In reply to this post by adrian kruk
>The most fundamental components can be used safely right now that are
>protected with locks.

Safely yes, but with lower performance.
Function msAcquireLock is called 50 times. Maybe it is possible to try to
less this number one by one, or better starting from ones which are called
the most often.

Can help, but as understand RFC15 has not started yet.

>- Reviewing all the portions of the mapserver and the dependent libraries code

Is it means that mapserver could has code which you aren't sure that is
stable in multithreading calls (or you are sure that is unstable and not
locked)?

>- Create test applications that can cover most parts of the
>mapserver/mapscript code.
I can create some simpl test application to test mapscript multithreading in
VS2005 or Mono.



Regards,
Adrian
Reply | Threaded
Open this post in threaded view
|

Re: Is MapServer Thread-safe?

Tamas Szekeres
2008/1/22, Adrian Kruk <[hidden email]>:
> >The most fundamental components can be used safely right now that are
> >protected with locks.
>
> Safely yes, but with lower performance.
> Function msAcquireLock is called 50 times. Maybe it is possible to try to
> less this number one by one, or better starting from ones which are called
> the most often.
>

I think the thread safety and the performance are not exactly the same
in requirements. If we are targeting the performance to be increased,
then we should have correct measures where are the bottlenecks to be
eliminated. I guess eliminating a few fine grained locks wouldn't give
a big increment in performance in an achitecture with 1 or 2
processors.

Some of the issues would possibly require an alternative solution. For
example to remove the lock around the parser we probably have to
reimplement the parser functionality and eliminate the global
variables around there. However we might want to consider an
alternative solution like supporting binary serialization and
deserialization of the mapObj state for example.

> Can help, but as understand RFC15 has not started yet.
>

yes it hasn't. And the statements might be out of date.

> >- Reviewing all the portions of the mapserver and the dependent libraries code
>
> Is it means that mapserver could has code which you aren't sure that is
> stable in multithreading calls (or you are sure that is unstable and not
> locked)?
>

We're tending to summarize these in the thread safety faq. However
it's slightly outdated and I'm not totally sure about the actual state
of the components enumerated there.

> >- Create test applications that can cover most parts of the
> >mapserver/mapscript code.
> I can create some simpl test application to test mapscript multithreading in
> VS2005 or Mono.
>

That would be cool. Currently the java bindings have a thread test
solution. We should have such tests at the C# part as well.

Best regards,

Tamas
Reply | Threaded
Open this post in threaded view
|

Re: Is MapServer Thread-safe?

Ed McNierney-4
In reply to this post by adrian kruk
Adrian -

Why do you think this is a "big limitation"?  There are many MapServer users, myself included, running MapServer as a CGI application - including as a WMS server - with "many clients running simultaneously".  What kind of equipment are you using and what kind of user load do you expect?  It sounds like you have concluded that a standard CGI WMS application of MapServer is simply unsuitable for you, and I haven't seen information that explains why you think that.

     - Ed

Ed McNierney
Chief Mapmaker
Demand Media / TopoZone.com
73 Princeton Street, Suite 305
North Chelmsford, MA  01863
Phone: 978-251-4242, Fax: 978-251-1396
[hidden email]

-----Original Message-----
From: UMN MapServer Developers List [mailto:[hidden email]] On Behalf Of Adrian Kruk
Sent: Tuesday, January 22, 2008 3:22 AM
To: [hidden email]
Subject: Re: [UMN_MAPSERVER-DEV] Is MapServer Thread-safe?

In my opinion this bug should be done as soon as possible. Maybe core
developers should consider to plan new release only for doing mapserver
multithreaded.
These is big limitation.

What about running MapServer as WMS server, is there the same problem with
performance with many clients running simultanously?
Reply | Threaded
Open this post in threaded view
|

Re: Is MapServer Thread-safe?

Umberto Nicoletti
In reply to this post by Tamas Szekeres
>
> We're tending to summarize these in the thread safety faq. However
> it's slightly outdated and I'm not totally sure about the actual state
> of the components enumerated there.
>

I update that FAQ entry whenever I find out that a new component has
been made thread safe (and I have some spare time ;-)).
Anyway, I think it pretty accurately describes the current state,
except maybe for the new and not yet released SQLServer 2008 plugin.

Umberto
Reply | Threaded
Open this post in threaded view
|

Re: Is MapServer Thread-safe?

Dan Little
In reply to this post by adrian kruk
(steps out on a limb, presents neck to chopping block)

After writing the below email: This probably sounds (via email) much more negative and fire-starting than it is intended.  I've tried to edit it to not sound that way, so please take the following statements with a grain of salt, and I understand my general conclusion is, "Threads are nice, processes are probably better (for some reasons), and it's probably a better idea for Mapserver to work on additional features and bug fixes before thread safety."

Frankly, thread-safety-ness seems like a nice goal and all, but really it can convolute the code and can create for poor performance.

Very general:
http://badtux.org/home/eric/editorial/threads.php

Good academic discussion of thread programming problems:
http://www.eecs.berkeley.edu/Pubs/TechRpts/2006/EECS-2006-1.pdf

Also, I do not see thread-safety-ness serving the "mass appeal" of Mapserver users and developers.  Here are a (somewhat) limited set of Mapserver applications that I've found (in moderate-to-high demand environments):

1) WFS/WMS servers, Mapserver Image serving, and Imagemap generation.  From my experience these are the actions that the vast majority of Mapservers perform on the most regular basis.  Mapserver is instantiated as a CGI script, reads the mapfile, processes the CGI request and returns an appropriate result based on the request, after the result is returned Mapserver ends.   You might be saying, "Yes, but this creates the ever-so-scary heavyweight process." To which I say the following:
    a) There isn't a risk for as many memory leaks with a heavyweight process.  Since the heavy weight process can typically  be run in a sequential fashion there is no need for complicated semaphores and generally there is no need to write code that will try to grab the same part of memory simultaneously.
    b) Heavy weight processes can use more memory, yes, but given the Mapserver binary's relative size, even with every option under-the-sun compiled into the binary, a system with 1Gb of RAM and a Pentium IV can still adequately serve nearly 300 users without a significant bottle neck (assuming all other parts of the system are adequate for those demands, e.g., the network to which the machine is connected). The baseless claim is made from the experiences we had at the City of Saint Paul serving nearly that many users off of a relatively pathetic single-CPU mid-range Dell-server.
    c) "The operating system is better at process management than you are."  This is something I've head to learn the HARD WAY!   It's also something I've spent a lot of time coding that I learned I should not have spent coding because entire teams of developers had already done a much better job than I was doing.  Having to consider resource management a top actual application development and trying to do any form of optimization can leave any developer chasing their tail.   And even once that code is "perfected" there is still a fight with the operating system over threads.   Thread clean-up is a very difficult thing for the operating system to perform (especially Windows, which really only "emulates" POSIX thread management).  The OS has no deterministic way of knowing when the code is done with the thread and most will keep the thread around for a while after it thinks it's done (even if the code tells the OS to free the thread) because it doesn't want to upset the execution of the code.   When a process calls exits or reaches the end of the code block the OS can rightly say those pages of memory can be freed.
    d) "But the data will be unloaded from memory and my files are BIG."  Not true on any modern operating system.   Operating systems manage disk cache for a reason.  Every modern OS, all the UNIX/Linux derivatives and NT4+, implements a very good method for caching frequently used files.  Even if the file is large, it will be kept in memory if it is being called frequently enough. 
    In conclusion, these users would not see a major benefit from people slaving hours away at thread-safety, when in fact, greater optimization would "help" them the most.

2) Mapscripters!  Mapscript is fantastic, it is a very well documented API into Mapserver, which is a coalescing of a number of very useful libraries.  I'm quite sure a lot of folks here have spent some time working with Mapscript and a number of them in high-demand environments.  I under stand the .Net family of languages break this rule, but most of the "Mapscript languages" really do not require thread safety to run efficiently or to handle a great number of users.  Arguably most of the scripting languages in a CGI environment perform the following steps:  load the interpreter (php/python/perl/etc.), the interpreter parses the script, loads/links the appropriate modules/libraries, executes the code, cleans up the processes.  The vast majority of these are done using a "heavy weight" process (fork()'d!). 


----- Original Message ----
From: Umberto Nicoletti <[hidden email]>
To: [hidden email]
Sent: Tuesday, January 22, 2008 7:34:56 AM
Subject: Re: [UMN_MAPSERVER-DEV] Is MapServer Thread-safe?

>
> We're tending to summarize these in the thread safety faq. However
> it's slightly outdated and I'm not totally sure about the actual state
> of the components enumerated there.
>

I update that FAQ entry whenever I find out that a new component has
been made thread safe (and I have some spare time ;-)).
Anyway, I think it pretty accurately describes the current state,
except maybe for the new and not yet released SQLServer 2008 plugin.

Umberto



Never miss a thing. Make Yahoo your homepage.
Reply | Threaded
Open this post in threaded view
|

Re: Is MapServer Thread-safe?

adrian kruk
In reply to this post by adrian kruk
>Why do you think this is a "big limitation"?  There are many MapServer
users, >myself included, running MapServer as a CGI application - including
as a WMS >server - with "many clients running simultaneously".  

What does mean "many" for you?

>What kind of equipment are you using and what kind of user load do you
expect?  >It sounds like you have concluded that a standard CGI WMS
application of >MapServer is simply unsuitable for you, and I haven't seen
information that >explains why you think that.

WMS is ok, but I need C# API to create any gis application (e.g. with
adding/modyfing spatial features, with requested by bussiness client
webservice interface, etc).
 
The problem is with scalability with using mapscript. Changing server from 1
processor architecture to 4 performance will increase performance but I
think that not much in really big workload.
Reply | Threaded
Open this post in threaded view
|

Re: Is MapServer Thread-safe?

Dan Little
In reply to this post by adrian kruk
Adrian,

I don't mean this suggestion to upset the faithful, but it sounds like you might need a transactional solution better served with MapGuide OS or GeoServer.

2cents,

-Duck

----- Original Message ----
From: Adrian Kruk <[hidden email]>
To: [hidden email]
Sent: Tuesday, January 22, 2008 8:36:38 AM
Subject: Re: [UMN_MAPSERVER-DEV] Is MapServer Thread-safe?

>Why do you think this is a "big limitation"?  There are many MapServer
users, >myself included, running MapServer as a CGI application - including
as a WMS >server - with "many clients running simultaneously". 

What does mean "many" for you?

>What kind of equipment are you using and what kind of user load do you
expect?  >It sounds like you have concluded that a standard CGI WMS
application of >MapServer is simply unsuitable for you, and I haven't seen
information that >explains why you think that.

WMS is ok, but I need C# API to create any gis application (e.g. with
adding/modyfing spatial features, with requested by bussiness client
webservice interface, etc).

The problem is with scalability with using mapscript. Changing server from 1
processor architecture to 4 performance will increase performance but I
think that not much in really big workload.



Be a better friend, newshound, and know-it-all with Yahoo! Mobile. Try it now.
Reply | Threaded
Open this post in threaded view
|

Re: Is MapServer Thread-safe?

adrian kruk

I don't mean this suggestion to upset the faithful, but it sounds like you might need a transactional solution better served with MapGuide OS or GeoServer.

Transactional is not as important for me as .NET API. 
MapScript has cool C# API. MapGuide could be good, I will find out. Thanks for advice


Best regards,
Adrian
Reply | Threaded
Open this post in threaded view
|

Re: Is MapServer Thread-safe?

Ed McNierney-4
In reply to this post by adrian kruk
Adrian -

Well, what really matters is what "many" means to YOU and your application; my MapServer applications are probably quite different from yours, so if I have 2,000 simultaneous users that doesn't really tell you anything.  If you can describe it in as much detail as possible, then it will make it much easier for us to give you advice.  There are quite a few serious production implementation of both MapServer CGI and MapScript applications.  And, as Dan correctly points out, a different tool might be even better for you.

     - Ed

Ed McNierney
Chief Mapmaker
Demand Media / TopoZone.com
73 Princeton Street, Suite 305
North Chelmsford, MA  01863
[hidden email]
Phone: +1 (978) 251-4242
Fax: +1 (978) 251-1396

-----Original Message-----
From: UMN MapServer Developers List [mailto:[hidden email]] On Behalf Of Adrian Kruk
Sent: Tuesday, January 22, 2008 9:37 AM
To: [hidden email]
Subject: Re: [UMN_MAPSERVER-DEV] Is MapServer Thread-safe?

>Why do you think this is a "big limitation"?  There are many MapServer
users, >myself included, running MapServer as a CGI application - including
as a WMS >server - with "many clients running simultaneously".  

What does mean "many" for you?

>What kind of equipment are you using and what kind of user load do you
expect?  >It sounds like you have concluded that a standard CGI WMS
application of >MapServer is simply unsuitable for you, and I haven't seen
information that >explains why you think that.

WMS is ok, but I need C# API to create any gis application (e.g. with
adding/modyfing spatial features, with requested by bussiness client
webservice interface, etc).
 
The problem is with scalability with using mapscript. Changing server from 1
processor architecture to 4 performance will increase performance but I
think that not much in really big workload.
Reply | Threaded
Open this post in threaded view
|

Re: Is MapServer Thread-safe?

adrian kruk
Perphaps some results from mapscript performance tests would be helpful for me.
Here is some details as you wish:

Environment:
QA Server:  2x3GHz Pentium (PRD: 4 - 8 processors)
Application Server: IIS 5.0/6.0
Memory: 3GB
OS: WinXP or 2003 Server
Technology: .NET/C#
Application type: webservice

Data:
Map of Poland, shapefiles, above layers:
roads ( 600k polylines, categorized by attribute in table),
railroads (2K polylines)
towns: (60K points, 3K polygons)
some adminitration districts (regions, subregions, etc, summary: 10Kpolygons in 3 layers)
forest, parks,etc (30K polygons)
lakes and rivers ( 21K polylines, 7K polygons)

Additional layers:
Labels of almost each layer in good quality (antialiasing) - visible depends on scale.
Some data from my business client (could be e.g. 100K polygons in 3 layers)

Example application:
Performance requirement: 1000 users sessions/hour.
Users session definition:
Session time: few random (avg. 10) minutes with doing actions zooming, changing coverage,  on/off layers, etc.
Between actions user waits for avg 7 secs.

WebService provide below functionality:
each user has its own map in session (will consider map pooling), data filtering (by atributes)
High quaility images like "maps.google.com " but one image with 800x600 resolution

WebService methods:
bytes[] GetImage(sessionID) - getting image

SetZoom(sessionID, int zoom)
SetRectangle(.....
SetCenter
SetMapSize  (default 800x600)
SetLayerVisibility(sessionID, layerid, bool visibility)
PutSymbol( x,y, symbolID)
GetLegend
GetLayerAtributes
DrawLayerFiltered(sessionID, query) - drawing in memory, image with new layer will be returned by GetImage
...
GetZoom,
GetCenter
etc.....


Can mapscript do it (quality and performance)?
How to store data: shapefiles or postgis? Maybe use 2 or 3 harddiscs if it will increase performance?
What about connection pooling to postgis, mapserver has one connection per process or more?

Please send be some advice, I will be very graceful.


Best Regards,
Adrian


2008/1/22, Ed McNierney <[hidden email]>:
Adrian -

Well, what really matters is what "many" means to YOU and your application; my MapServer applications are probably quite different from yours, so if I have 2,000 simultaneous users that doesn't really tell you anything.  If you can describe it in as much detail as possible, then it will make it much easier for us to give you advice.  There are quite a few serious production implementation of both MapServer CGI and MapScript applications.  And, as Dan correctly points out, a different tool might be even better for you.

     - Ed

Ed McNierney
Chief Mapmaker
Demand Media / TopoZone.com
73 Princeton Street, Suite 305
North Chelmsford, MA 01863
[hidden email]
Phone: +1 (978) 251-4242
Fax: +1 (978) 251-1396

-----Original Message-----
From: UMN MapServer Developers List [mailto:[hidden email]] On Behalf Of Adrian Kruk
Sent: Tuesday, January 22, 2008 9:37 AM
To: [hidden email]
Subject: Re: [UMN_MAPSERVER-DEV] Is MapServer Thread-safe?

>Why do you think this is a "big limitation"?  There are many MapServer
users, >myself included, running MapServer as a CGI application - including
as a WMS >server - with "many clients running simultaneously".

What does mean "many" for you?

>What kind of equipment are you using and what kind of user load do you
expect?  >It sounds like you have concluded that a standard CGI WMS
application of >MapServer is simply unsuitable for you, and I haven't seen
information that >explains why you think that.

WMS is ok, but I need C# API to create any gis application ( e.g. with
adding/modyfing spatial features, with requested by bussiness client
webservice interface, etc).

The problem is with scalability with using mapscript. Changing server from 1
processor architecture to 4 performance will increase performance but I
think that not much in really big workload.



--
Pozdrawiam,
Adrian Kruk
Reply | Threaded
Open this post in threaded view
|

Re: Is MapServer Thread-safe?

Ed McNierney-4

Adrian –

 

My first reaction is to wonder why you feel you need MapScript (which leads to the whole issue of thread safety in the first place).  I would be inclined to build your application as I have done others, using the .NET/C# environment to process Web requests and then compute an appropriate WMS request to render the map image required.  You would use MapServer in simple CGI/WMS mode, sending HTTP requests for images which you then retrieve in your C# code and embed in an output page of your design.

 

Your user activity seems quite high; an average 10-minute session with an average 7-second wait between map requests would mean the average visit requested 85 map views.  That’s a very high number!  Just as a data point, users on my TopoZone site average about 30 seconds per map view (and no, that’s not because the maps are slow to load <g>).  Users tend to navigate quickly, panning and zooming and selecting layers to get something they want – and then they actually want to take a moment to look at it.

 

Shapefiles, properly organized, are almost certainly the fastest storage option in terms of retrieval.  However, I don’t know how often those data sets change.  If your data changes with any frequency, you will need to either automate the data-organization steps to update the shapefiles, or store your data in a database.  Don’t just put static, simple data into a database if you’re not going to otherwise use the database’s capabilities.

 

-          Ed

 

Ed McNierney

Chief Mapmaker

Demand Media / TopoZone.com

73 Princeton Street, Suite 305

North Chelmsford, MA  01863

Phone: 978-251-4242, Fax: 978-251-1396

[hidden email]

 

From: UMN MapServer Developers List [mailto:[hidden email]] On Behalf Of adrian kruk
Sent: Tuesday, January 22, 2008 11:04 AM
To: [hidden email]
Subject: Re: [UMN_MAPSERVER-DEV] Is MapServer Thread-safe?

 

Perphaps some results from mapscript performance tests would be helpful for me.
Here is some details as you wish:

Environment:
QA Server:  2x3GHz Pentium (PRD: 4 - 8 processors)
Application Server: IIS 5.0/6.0
Memory: 3GB
OS: WinXP or 2003 Server
Technology: .NET/C#
Application type: webservice

Data:
Map of Poland, shapefiles, above layers:
roads ( 600k polylines, categorized by attribute in table),
railroads (2K polylines)
towns: (60K points, 3K polygons)
some adminitration districts (regions, subregions, etc, summary: 10Kpolygons in 3 layers)
forest, parks,etc (30K polygons)
lakes and rivers ( 21K polylines, 7K polygons)

Additional layers:
Labels of almost each layer in good quality (antialiasing) - visible depends on scale.
Some data from my business client (could be e.g. 100K polygons in 3 layers)

Example application:
Performance requirement: 1000 users sessions/hour.
Users session definition:
Session time: few random (avg. 10) minutes with doing actions zooming, changing coverage,  on/off layers, etc.
Between actions user waits for avg 7 secs.

WebService provide below functionality:
each user has its own map in session (will consider map pooling), data filtering (by atributes)
High quaility images like "maps.google.com " but one image with 800x600 resolution

WebService methods:
bytes[] GetImage(sessionID) - getting image

SetZoom(sessionID, int zoom)
SetRectangle(.....
SetCenter
SetMapSize  (default 800x600)
SetLayerVisibility(sessionID, layerid, bool visibility)
PutSymbol( x,y, symbolID)
GetLegend
GetLayerAtributes
DrawLayerFiltered(sessionID, query) - drawing in memory, image with new layer will be returned by GetImage
...
GetZoom,
GetCenter
etc.....


Can mapscript do it (quality and performance)?
How to store data: shapefiles or postgis? Maybe use 2 or 3 harddiscs if it will increase performance?
What about connection pooling to postgis, mapserver has one connection per process or more?

Please send be some advice, I will be very graceful.


Best Regards,
Adrian

2008/1/22, Ed McNierney <[hidden email]>:

Adrian -

Well, what really matters is what "many" means to YOU and your application; my MapServer applications are probably quite different from yours, so if I have 2,000 simultaneous users that doesn't really tell you anything.  If you can describe it in as much detail as possible, then it will make it much easier for us to give you advice.  There are quite a few serious production implementation of both MapServer CGI and MapScript applications.  And, as Dan correctly points out, a different tool might be even better for you.

     - Ed

Ed McNierney
Chief Mapmaker
Demand Media / TopoZone.com
73 Princeton Street, Suite 305
North Chelmsford, MA 01863
[hidden email]
Phone: +1 (978) 251-4242
Fax: +1 (978) 251-1396

-----Original Message-----
From: UMN MapServer Developers List [mailto:[hidden email]] On Behalf Of Adrian Kruk
Sent: Tuesday, January 22, 2008 9:37 AM
To: [hidden email]
Subject: Re: [UMN_MAPSERVER-DEV] Is MapServer Thread-safe?

>Why do you think this is a "big limitation"?  There are many MapServer
users, >myself included, running MapServer as a CGI application - including
as a WMS >server - with "many clients running simultaneously".

What does mean "many" for you?

>What kind of equipment are you using and what kind of user load do you
expect?  >It sounds like you have concluded that a standard CGI WMS
application of >MapServer is simply unsuitable for you, and I haven't seen
information that >explains why you think that.

WMS is ok, but I need C# API to create any gis application ( e.g. with
adding/modyfing spatial features, with requested by bussiness client
webservice interface, etc).

The problem is with scalability with using mapscript. Changing server from 1
processor architecture to 4 performance will increase performance but I
think that not much in really big workload.




--
Pozdrawiam,
Adrian Kruk

Reply | Threaded
Open this post in threaded view
|

Re: Is MapServer Thread-safe?

adrian kruk
My first reaction is to wonder why you feel you need MapScript (which leads to the whole issue of thread safety in the first place).  I would be inclined to build your application as I have done others, using the .NET/C# environment to process Web requests and then compute an appropriate WMS request to render the map image required.  You would use MapServer in simple CGI/WMS mode, sending HTTP requests for images which you then retrieve in your C# code and embed in an output page of your design.

Ok I will give an example of functionality. MapScript has map object, shape object, layer object, etc. With using these, it is simple to run query in one layer, get results as shapes and then run query on different layer, create new layer, then get image, i can store map object in application cache in very simple way, etc, etc.
It is possible in CGI/WMS? If it is tell me how to do it.





Reply | Threaded
Open this post in threaded view
|

Re: MapServer Application Design Questions

Ed McNierney-4

Adrian –

 

If you are running a .NET/C# application, that application can – using ServerXMLHttpRequest calls – make as many requests to a MapServer CGI as you need to produce a single output page.

 

I am not suggesting that that’s the best design for your application.  But you started with a very specific question about thread safety and we’re trying to back up to the broader question of application design (hence the edit I made to the Subject line), because it seems like that’s what you are trying to accomplish.  If you’d like assistance with that, then you’ll need to really describe your application in detail.  Your last email with lists of layers, etc. was a good description of your data (although you didn’t talk about updates and/or update rates).  But you didn’t, for example, describe the situation below.

 

There are many users here with expertise in many different sorts of MapServer applications.  But we can’t help if we only have little bits of information, and if you only send detailed, specific requests for “how to do it”.  If you can take the time to really describe your application then you will both be in the best position to get advice from users on this list and we will be able to best use our time by providing information that’s really useful!

 

-          Ed

 

P.S. It also really helps to include the previous message history in your replies, so we can all understand the context, and so users who might now be reading this message (because they didn’t know much about thread safety but know a lot about application design) will know what’s going on – thanks.

 

Ed McNierney

Chief Mapmaker

Demand Media / TopoZone.com

73 Princeton Street, Suite 305

North Chelmsford, MA  01863

Phone: 978-251-4242, Fax: 978-251-1396

[hidden email]

 

From: UMN MapServer Developers List [mailto:[hidden email]] On Behalf Of adrian kruk
Sent: Wednesday, January 23, 2008 3:07 AM
To: [hidden email]
Subject: Re: [UMN_MAPSERVER-DEV] Is MapServer Thread-safe?

 

My first reaction is to wonder why you feel you need MapScript (which leads to the whole issue of thread safety in the first place).  I would be inclined to build your application as I have done others, using the .NET/C# environment to process Web requests and then compute an appropriate WMS request to render the map image required.  You would use MapServer in simple CGI/WMS mode, sending HTTP requests for images which you then retrieve in your C# code and embed in an output page of your design.


Ok I will give an example of functionality. MapScript has map object, shape object, layer object, etc. With using these, it is simple to run query in one layer, get results as shapes and then run query on different layer, create new layer, then get image, i can store map object in application cache in very simple way, etc, etc.
It is possible in CGI/WMS? If it is tell me how to do it.



 

Reply | Threaded
Open this post in threaded view
|

Re: MapServer Application Design Questions

adrian kruk

I am not suggesting that that's the best design for your application.  But you started with a very specific question about thread safety and we're trying to back up to the broader question of application design (hence the edit I made to the Subject line), because it seems like that's what you are trying to accomplish. 

I was asking about thread safety becouse architecure based on mapscript_dll give me all functionality what I need now. This architecure is the best for functionality, but bad for multithreading (stability and performance).

If you'd like assistance with that, then you'll need to really describe your application in detail. 

Here is some description of simple example of talking protocol between client and server - to easier understand. It is not all functionality. Forget about updates, I don't need this now,  maybe in future. Application will be created in stateful mode.


Many clients are connecting to webservice(my IIS application).
Each clients has its own map session so it means that has mapobject on server side in session cache, on which he can set parameters like zoom, center point, bounding rectangle, layers visibility,etc via webservice methods.
Webservice recognises proper mapobject becouse clients sends sessionId in each method.
e.g. SetZoom(long sessionId, float zoom).

After setting this parameters on his own map object(which is in webservice memory),  client requests for image by executing GetImage webservice method. After that image will be sended to the webservice client via SOAP from webservice app.
After a while client want to see all towns with first letter "W" in name and marked as different color. So client executes proper method on webservice. WebService is creating new layer (filtered from towns) and put this layer to proper mapobject. After that client execute GetImage method to see those "w*" towns in all country (with also other layers which visibility he had set before).








12