[UMN_MAPSERVER-USERS] conditional/regular expressions are slow

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

[UMN_MAPSERVER-USERS] conditional/regular expressions are slow

Douglass Davis
I have an application which allows me to have about 5 different zoom levels.

 From the lowest zoom level, you can see the biggest cities only.  Az
you zoom in, can see the biggest highways, then small highways, then
small streets.

The way I do this is in my mapfile I have:

  LAYER
     NAME big_cities
...
     CLASS
        NAME big_city_class
       EXPRESSION ('[POP_2000]'>80000 and '[STATE]'='NC')
...

I am only interested in North Carolina by the way.

Also, the highways are numbered A1.* and A2.* for the big highways A3.*
for the smaller highways etc.  So, What I do is use regular expressions
like:

      EXPRESSION /A[12].*/
      EXPRESSION /A[3].*/

This appears to be slowing things down.  What's the best way to handle this?

Three ideas I have for speeding this up are:
1. Export the shapefile to PostGIS where it can be indexed correctly
2. Separate the shapefile into multiple shapefiles somehow.  One file
would have big cities, the next file would have medium size cities and
big cities etc.
3. Somehow add more indexing to the shapefile.

Neither of which I know how to do.

Does any one have any idea of how to do any of these things?  I would
prefer using some kind of free tool... Even a command line one would be
fine, as I do not have a big budget.


--
http://www.douglassdavis.com
Reply | Threaded
Open this post in threaded view
|

Re: [UMN_MAPSERVER-USERS] conditional/regular expressions are slow

Dylan Keon-3
On 08/05/2005 05:07 PM, Douglass Davis wrote:
> Also, the highways are numbered A1.* and A2.* for the big highways A3.*
> for the smaller highways etc.  So, What I do is use regular expressions
> like:
>
>      EXPRESSION /A[12].*/
>      EXPRESSION /A[3].*/
>
> This appears to be slowing things down.  What's the best way to handle
> this?

I haven't tested this, but I wonder if /^A[12]/ and /^A3/ would be any
faster.

> Three ideas I have for speeding this up are:
> 1. Export the shapefile to PostGIS where it can be indexed correctly
> 2. Separate the shapefile into multiple shapefiles somehow.  One file
> would have big cities, the next file would have medium size cities and
> big cities etc.
> 3. Somehow add more indexing to the shapefile.

Try creating spatial indexes on your shapefiles if you haven't done so
already.  Use the shptree utility that is packaged with MapServer.

--Dylan
Reply | Threaded
Open this post in threaded view
|

Re: [UMN_MAPSERVER-USERS] conditional/regular expressions are slow

Paul Ramsey-2
Douglas,

Sounds like you are trying to draw *just* the big highways at certain  
scales.  Using the feature code as a filter.  This is not mapserver  
being slow, this is mapserver processing a huge amount of data.  You  
need to think about how mapserver has to answer your request (use  
this shape file and show me the big highways).  Since it has no  
attribute indexes, it has to open the file and for every shape in the  
current view (it does have spatial indexes) it reads the shape and  
checks the feature code.  It probably examines 100 ordinary road  
shapes for every 1 highway shape it draws for you.  Not efficient at  
all.

The answer is your open 2 below.  Pre-process your data so that  
mapserver can do the bare minimum of work for each rendering pass.  
There is no point in having mapserver read 100000 features that it is  
not even going to draw.

Paul


On 5-Aug-05, at 5:48 PM, Dylan Keon wrote:

> On 08/05/2005 05:07 PM, Douglass Davis wrote:
>
>> Also, the highways are numbered A1.* and A2.* for the big highways  
>> A3.* for the smaller highways etc.  So, What I do is use regular  
>> expressions like:
>>      EXPRESSION /A[12].*/
>>      EXPRESSION /A[3].*/
>> This appears to be slowing things down.  What's the best way to  
>> handle this?
>>
>
> I haven't tested this, but I wonder if /^A[12]/ and /^A3/ would be  
> any faster.
>
>
>> Three ideas I have for speeding this up are:
>> 1. Export the shapefile to PostGIS where it can be indexed correctly
>> 2. Separate the shapefile into multiple shapefiles somehow.  One  
>> file would have big cities, the next file would have medium size  
>> cities and big cities etc.
>> 3. Somehow add more indexing to the shapefile.
>>
>
> Try creating spatial indexes on your shapefiles if you haven't done  
> so already.  Use the shptree utility that is packaged with MapServer.
>
> --Dylan
>
Reply | Threaded
Open this post in threaded view
|

Re: [UMN_MAPSERVER-USERS] conditional/regular expressions are slow

Douglass Davis
Paul Ramsey wrote:

>
> The answer is your open 2 below.  Pre-process your data so that  
> mapserver can do the bare minimum of work for each rendering pass.  
> There is no point in having mapserver read 100000 features that it is  
> not even going to draw.
>
> Paul
>

thanks.  Does any one know of any free tools that can separate a
shapefile into multiple shapefiles using queries/regular
expressions/attributes...?
Reply | Threaded
Open this post in threaded view
|

Re: [UMN_MAPSERVER-USERS] conditional/regular expressions are slow

Bart van den Eijnden (OSGIS)
OGR has tools for this. Check ogr2ogr.

http://www.gdal.org/ogr/ogr_utilities.html

Best regards,
Bart

On Sat, 06 Aug 2005 17:43:33 +0200, Douglass Davis  
<[hidden email]> wrote:

> Paul Ramsey wrote:
>
>>
>> The answer is your open 2 below.  Pre-process your data so that
>> mapserver can do the bare minimum of work for each rendering pass.
>> There is no point in having mapserver read 100000 features that it is
>> not even going to draw.
>>
>> Paul
>>
>
> thanks.  Does any one know of any free tools that can separate a
> shapefile into multiple shapefiles using queries/regular
> expressions/attributes...?
>



--
Using Opera's revolutionary e-mail client: http://www.opera.com/mail/