MapScript multithreading performance

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

MapScript multithreading performance

adrian kruk
Hi,
I did some multithreading performance tests using C# mapscript (We've discussed about safety and performance before)

I have run 200 threads running simultanously and performing actions like: create mapObject, zoom, pan, query, draw, etc
Testing application environment:
-4processors server (Intel Xeon E5320)
-Win2003EE+SP3
-4GM RAM
-MapScript 5.1-DEV from http://www.coordinatesolutions.com/download/ASPNET20Sample_50.zip
-1GB data in shapefiles on RAID

I expected performance problems with parsing mapfile (static variables, locks). But there were no problems with it.
During parsing mapfile by 200 threads processors was used in 98-100%. So it is perfect result.
But, I find problem probably with draw method. Each thread was executing methods on its own map object like:
drawLegend, drawScalebar, queryByAttributes, zoomPoint, setExtent, Draw.
During running test application, and executing those methods, processors was used in 25-35% ONLY (25% - one processor).
Its very poor result. I guess that problem may be related with using locks in drawing process{ imageObj img = map.draw() }, which takes much more time than other mapObject methods.

How to improve performance? It is hard to fix it?

Regards,
Adrian Kruk,
infovidematrix.pl

_______________________________________________
mapserver-dev mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/mapserver-dev
Reply | Threaded
Open this post in threaded view
|

Re: MapScript multithreading performance

Steve Lime
That's the expected bottleneck point.  The locks are around pretty specific
pieces of code, mostly related to expression handling if I'm not mistaken. So
I wonder if moving to a threadsafe lexer/parser would help. Both bison and
flex support that. We'd have to carry the static variables associated with those
as a piece of the mapObj. I've not looked at doing that very closely but it's
probably relatively straight forward.

Steve

>>> "adrian kruk" <[hidden email]> 02/11/08 6:15 AM >>>
Hi,
I did some multithreading performance tests using C# mapscript (We've
discussed about safety and performance before)

I have run 200 threads running simultanously and performing actions like:
create mapObject, zoom, pan, query, draw, etc
Testing application environment:
-4processors server (Intel Xeon E5320)
-Win2003EE+SP3
-4GM RAM
-MapScript 5.1-DEV from
http://www.coordinatesolutions.com/download/ASPNET20Sample_50.zip
-1GB data in shapefiles on RAID

I expected performance problems with parsing mapfile (static variables,
locks). But there were no problems with it.
During parsing mapfile by 200 threads processors was used in 98-100%. So it
is perfect result.
But, I find problem probably with draw method. Each thread was executing
methods on its own map object like:
drawLegend, drawScalebar, queryByAttributes, zoomPoint, setExtent, Draw.
During running test application, and executing those methods, processors was
used in 25-35% ONLY (25% - one processor).
Its very poor result. I guess that problem may be related with using locks
in drawing process{ imageObj img = map.draw() }, which takes much more time
than other mapObject methods.

How to improve performance? It is hard to fix it?

Regards,
Adrian Kruk,
infovidematrix.pl

_______________________________________________
mapserver-dev mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/mapserver-dev
Reply | Threaded
Open this post in threaded view
|

Re: MapScript multithreading performance

Tamas Szekeres
In reply to this post by adrian kruk
Adrian,

I'm not aware of much locking around the drawing code of it's own.
This might be another issue I think. Would you share your mapfile and
the C# code you test with?
How do you detect which part of the code causes the problem and which not?

I expect your problem reported might be related to some operations
requiring heavy resource access (eg. file loads during epsg lookups).
Have you monitored your disk access that might also be optimized?

I'm not sure if the processor usage is the perfect measure of the
efficiency of the application either.


Best regards,

Tamas




2008/2/11, adrian kruk <[hidden email]>:

> Hi,
> I did some multithreading performance tests using C# mapscript (We've
> discussed about safety and performance before)
>
> I have run 200 threads running simultanously and performing actions like:
> create mapObject, zoom, pan, query, draw, etc
>  Testing application environment:
> -4processors server (Intel Xeon E5320)
> -Win2003EE+SP3
> -4GM RAM
> -MapScript 5.1-DEV from
> http://www.coordinatesolutions.com/download/ASPNET20Sample_50.zip
>  -1GB data in shapefiles on RAID
>
> I expected performance problems with parsing mapfile (static variables,
> locks). But there were no problems with it.
> During parsing mapfile by 200 threads processors was used in 98-100%. So it
> is perfect result.
>  But, I find problem probably with draw method. Each thread was executing
> methods on its own map object like:
> drawLegend, drawScalebar, queryByAttributes, zoomPoint, setExtent, Draw.
> During running test application, and executing those methods, processors was
> used in 25-35% ONLY (25% - one processor).
>  Its very poor result. I guess that problem may be related with using locks
> in drawing process{ imageObj img = map.draw() }, which takes much more time
> than other mapObject methods.
>
> How to improve performance? It is hard to fix it?
>
> Regards,
> Adrian Kruk,
> infovidematrix.pl
>
> _______________________________________________
> mapserver-dev mailing list
> [hidden email]
> http://lists.osgeo.org/mailman/listinfo/mapserver-dev
>
>
_______________________________________________
mapserver-dev mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/mapserver-dev
Reply | Threaded
Open this post in threaded view
|

Re: MapScript multithreading performance

Frank Warmerdam
Tamas Szekeres wrote:

> Adrian,
>
> I'm not aware of much locking around the drawing code of it's own.
> This might be another issue I think. Would you share your mapfile and
> the C# code you test with?
> How do you detect which part of the code causes the problem and which not?
>
> I expect your problem reported might be related to some operations
> requiring heavy resource access (eg. file loads during epsg lookups).
> Have you monitored your disk access that might also be optimized?
>
> I'm not sure if the processor usage is the perfect measure of the
> efficiency of the application either.

Adrian,

I agree with Tamas - we don't have enough information to give an informed
answer.

I will note that DrawMap() has to access the feature data, and so is quite
subject to feature access lock contention.  For instance, if you are using
CONNECTIONTYPE OGR and MapServer 5 there is a big lock around OGR.  That means
only one drawing thread at a time can be in OGR fetching data which is often
the dominant time consumer.

So, providing details of your datasource would be helpful.  But really,
you need a way of interrupting your threads to see where they are blocked
in some fashion to get a better sense of what is the hold up.

I will say, that I'm unaware of any locks around the rendering work itself,
at least with the GD vector rendering code.  So for actual rendering intensive
applications, multi-threading should give good cpu utilization.

Best regards,
--
---------------------------------------+--------------------------------------
I set the clouds in motion - turn up   | Frank Warmerdam, [hidden email]
light and sound - activate the windows | http://pobox.com/~warmerdam
and watch the world go round - Rush    | President OSGeo, http://osgeo.org

_______________________________________________
mapserver-dev mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/mapserver-dev
Reply | Threaded
Open this post in threaded view
|

Re: MapScript multithreading performance

adrian kruk
>Would you share your mapfile and the C# code you test with?
Yes,  here is my mapfile.

>Have you monitored your disk access that might also be optimized?
Yes it could be a reason. I was not monitored a disk. Thank you Tamas for advice. How can I optimize access to shapefiles?
Maybe Linux will be faster for reading shapefiles (NTFS is slow)? What about PostgreSQL/PostGIS on different machine (instead of shapefiles on the same machine as mapscript application), will it makes better overall performance of mapscript application?


>I'm not sure if the processor usage is the perfect measure of the efficiency of the application either.
I have never mention that processor usage is a measure of application efficiency.
I mean that 30% usage obviously is to low. If I will bought 8 processors machine I wouldn't like to find out that usage will be 15-20%.


Regards
Adrian


_______________________________________________
mapserver-dev mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/mapserver-dev

mapapolskiPG.map (33K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Fwd: MapScript multithreading performance

adrian kruk
Second mapfile without projections and without postgis connection couse the same low usage.
TestApp01.png - taskmgr screenshot- starting test tapplication, 100%usage is when I'm creating threads and mapObjects, after that usage is decreasing


--
Regards,
AK

_______________________________________________
mapserver-dev mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/mapserver-dev

mapapolski.map (30K) Download Attachment
TestApp01.png (26K) Download Attachment