Map file with 35,000+ Lines... management issue

classic Classic list List threaded Threaded
20 messages Options
Reply | Threaded
Open this post in threaded view
|

Map file with 35,000+ Lines... management issue

ritesh ambastha
Dear Readers,

I have developed a map file with more than 35,000 Lines. Its size will grow by double/triple in next few months. Now, I am trying to tune my map file by removing unwanted lines. Still, I am bit confused about its maintenance.

Please throw some lights over writing map files by following best practices.

Thanks in advance.

Regards,
Ritz
Reply | Threaded
Open this post in threaded view
|

Re: Map file with 35,000+ Lines... management issue

richf
riteshambastha wrote
I have developed a map file with more than 35,000 Lines.

Please throw some lights over writing map files by following best practices.
Have you considered more than one map file, depending on the circumstances, each one a bit smaller, rather than one huge map file?

It's hard to say how appropriate that may or may not be without knowing more about your circumstances, but that's what I have.

- Rich
Reply | Threaded
Open this post in threaded view
|

Re: Map file with 35,000+ Lines... management issue

ritesh ambastha
Thanks Rich,

Yeah, even I am trying to solve this issue by separating the huge map file with smaller chunks of many map files. Am bit confused if it will lead to performance issues.

Warm Regards,
Ritz

 
rich.fromm wrote
riteshambastha wrote
I have developed a map file with more than 35,000 Lines.

Please throw some lights over writing map files by following best practices.
Have you considered more than one map file, depending on the circumstances, each one a bit smaller, rather than one huge map file?

It's hard to say how appropriate that may or may not be without knowing more about your circumstances, but that's what I have.

- Rich
Reply | Threaded
Open this post in threaded view
|

Re: Map file with 35,000+ Lines... management issue

Fawcett, David (MNIT)
Everything is a performance issue, the question is, 'Is the performance
acceptable to you?'...

You have built a pretty massive map file.  I would look at stripping it
down to only what is needed for a particular application.  

For questions about how to improve performance, you may want to look at:
http://mapserver.gis.umn.edu/docs/howto/mapfiletuning

David.

-----Original Message-----
From: UMN MapServer Users List [mailto:[hidden email]] On
Behalf Of riteshambastha
Sent: Wednesday, January 23, 2008 12:59 PM
To: [hidden email]
Subject: Re: [UMN_MAPSERVER-USERS] Map file with 35,000+ Lines...
management issue


Thanks Rich,

Yeah, even I am trying to solve this issue by separating the huge map
file with smaller chunks of many map files. Am bit confused if it will
lead to performance issues.

Warm Regards,
Ritz

 

rich.fromm wrote:

>
>
> riteshambastha wrote:
>>
>> I have developed a map file with more than 35,000 Lines.
>>
>> Please throw some lights over writing map files by following best
>> practices.
>>
>
> Have you considered more than one map file, depending on the
> circumstances, each one a bit smaller, rather than one huge map file?
>
> It's hard to say how appropriate that may or may not be without
> knowing more about your circumstances, but that's what I have.
>
> - Rich
>
>

--
View this message in context:
http://www.nabble.com/Map-file-with-35%2C000%2B-Lines...-management-issu
e-tp15048892p15048905.html
Sent from the Mapserver - User mailing list archive at Nabble.com.
Reply | Threaded
Open this post in threaded view
|

Re: Map file with 35,000+ Lines... management issue

Espen Isaksen
Regarding this, is it possible to "include" a map file inside another map file?

Espen

On 23/01/2008, Fawcett, David <[hidden email]> wrote:

> Everything is a performance issue, the question is, 'Is the performance
> acceptable to you?'...
>
> You have built a pretty massive map file.  I would look at stripping it
> down to only what is needed for a particular application.
>
> For questions about how to improve performance, you may want to look at:
> http://mapserver.gis.umn.edu/docs/howto/mapfiletuning
>
> David.
>
> -----Original Message-----
> From: UMN MapServer Users List [mailto:[hidden email]] On
> Behalf Of riteshambastha
> Sent: Wednesday, January 23, 2008 12:59 PM
> To: [hidden email]
> Subject: Re: [UMN_MAPSERVER-USERS] Map file with 35,000+ Lines...
> management issue
>
>
> Thanks Rich,
>
> Yeah, even I am trying to solve this issue by separating the huge map
> file with smaller chunks of many map files. Am bit confused if it will
> lead to performance issues.
>
> Warm Regards,
> Ritz
>
>
>
> rich.fromm wrote:
> >
> >
> > riteshambastha wrote:
> >>
> >> I have developed a map file with more than 35,000 Lines.
> >>
> >> Please throw some lights over writing map files by following best
> >> practices.
> >>
> >
> > Have you considered more than one map file, depending on the
> > circumstances, each one a bit smaller, rather than one huge map file?
> >
> > It's hard to say how appropriate that may or may not be without
> > knowing more about your circumstances, but that's what I have.
> >
> > - Rich
> >
> >
>
> --
> View this message in context:
> http://www.nabble.com/Map-file-with-35%2C000%2B-Lines...-management-issu
> e-tp15048892p15048905.html
> Sent from the Mapserver - User mailing list archive at Nabble.com.
>
Reply | Threaded
Open this post in threaded view
|

RE : [UMN_MAPSERVER-USERS] Map file with 35,000+ Lines... management issue

Léveillé, James
As simple as

  #INCLUDE
  INCLUDE                 "other_map_file.map"



______________________________________________________
JAMES LÉVEILLÉ
Service des systèmes de Mission
Direction des technologies de l'information
Ministère des Transports du Québec
5833, boul. Pierre-Bertrand, 2ième étage
Québec (Québec) G2K 1K7
Téléphone:   (418) 380-2005 poste 227
Télécopieur: (418) 644-6653
[hidden email]

 


-----Message d'origine-----
De : UMN MapServer Users List [mailto:[hidden email]] De la
part de Espen Isaksen
Envoyé : 23 janvier 2008 14:26
À : [hidden email]
Objet : Re: [UMN_MAPSERVER-USERS] Map file with 35,000+ Lines... management
issue


Regarding this, is it possible to "include" a map file inside another map
file?

Espen

On 23/01/2008, Fawcett, David <[hidden email]> wrote:

> Everything is a performance issue, the question is, 'Is the
> performance acceptable to you?'...
>
> You have built a pretty massive map file.  I would look at stripping
> it down to only what is needed for a particular application.
>
> For questions about how to improve performance, you may want to look
> at: http://mapserver.gis.umn.edu/docs/howto/mapfiletuning
>
> David.
>
> -----Original Message-----
> From: UMN MapServer Users List [mailto:[hidden email]]
> On Behalf Of riteshambastha
> Sent: Wednesday, January 23, 2008 12:59 PM
> To: [hidden email]
> Subject: Re: [UMN_MAPSERVER-USERS] Map file with 35,000+ Lines...
> management issue
>
>
> Thanks Rich,
>
> Yeah, even I am trying to solve this issue by separating the huge map
> file with smaller chunks of many map files. Am bit confused if it will
> lead to performance issues.
>
> Warm Regards,
> Ritz
>
>
>
> rich.fromm wrote:
> >
> >
> > riteshambastha wrote:
> >>
> >> I have developed a map file with more than 35,000 Lines.
> >>
> >> Please throw some lights over writing map files by following best
> >> practices.
> >>
> >
> > Have you considered more than one map file, depending on the
> > circumstances, each one a bit smaller, rather than one huge map
> > file?
> >
> > It's hard to say how appropriate that may or may not be without
> > knowing more about your circumstances, but that's what I have.
> >
> > - Rich
> >
> >
>
> --
> View this message in context:
> http://www.nabble.com/Map-file-with-35%2C000%2B-Lines...-management-is
> su
> e-tp15048892p15048905.html
> Sent from the Mapserver - User mailing list archive at Nabble.com.
>
Reply | Threaded
Open this post in threaded view
|

Re: Map file with 35,000+ Lines... management issue

Kralidis,Tom [Ontario]
In reply to this post by Espen Isaksen
 

> -----Original Message-----
> From: UMN MapServer Users List
> [mailto:[hidden email]] On Behalf Of Espen Isaksen
> Sent: 23 January, 2008 2:26 PM
> To: [hidden email]
> Subject: Re: [UMN_MAPSERVER-USERS] Map file with 35,000+
> Lines... management issue
>
> Regarding this, is it possible to "include" a map file inside
> another map file?
>

One can do just this with the INCLUDE directive.  See
http://mapserver.gis.umn.edu/docs/reference/mapfile/Include

..Tom

 

> Espen
>
> On 23/01/2008, Fawcett, David <[hidden email]> wrote:
> > Everything is a performance issue, the question is, 'Is the
> > performance acceptable to you?'...
> >
> > You have built a pretty massive map file.  I would look at
> stripping
> > it down to only what is needed for a particular application.
> >
> > For questions about how to improve performance, you may
> want to look at:
> > http://mapserver.gis.umn.edu/docs/howto/mapfiletuning
> >
> > David.
> >
> > -----Original Message-----
> > From: UMN MapServer Users List
> [mailto:[hidden email]]
> > On Behalf Of riteshambastha
> > Sent: Wednesday, January 23, 2008 12:59 PM
> > To: [hidden email]
> > Subject: Re: [UMN_MAPSERVER-USERS] Map file with 35,000+ Lines...
> > management issue
> >
> >
> > Thanks Rich,
> >
> > Yeah, even I am trying to solve this issue by separating
> the huge map
> > file with smaller chunks of many map files. Am bit confused
> if it will
> > lead to performance issues.
> >
> > Warm Regards,
> > Ritz
> >
> >
> >
> > rich.fromm wrote:
> > >
> > >
> > > riteshambastha wrote:
> > >>
> > >> I have developed a map file with more than 35,000 Lines.
> > >>
> > >> Please throw some lights over writing map files by
> following best
> > >> practices.
> > >>
> > >
> > > Have you considered more than one map file, depending on the
> > > circumstances, each one a bit smaller, rather than one
> huge map file?
> > >
> > > It's hard to say how appropriate that may or may not be without
> > > knowing more about your circumstances, but that's what I have.
> > >
> > > - Rich
> > >
> > >
> >
> > --
> > View this message in context:
> >
> http://www.nabble.com/Map-file-with-35%2C000%2B-Lines...-management-is
> > su
> > e-tp15048892p15048905.html
> > Sent from the Mapserver - User mailing list archive at Nabble.com.
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Map file with 35,000+ Lines... management issue

Espen Isaksen
Thanks! Don't know how I missed that one...

Espen

On 23/01/2008, Kralidis,Tom [Burlington] <[hidden email]> wrote:

>
>
>
> > -----Original Message-----
> > From: UMN MapServer Users List
> > [mailto:[hidden email]] On Behalf Of Espen Isaksen
> > Sent: 23 January, 2008 2:26 PM
> > To: [hidden email]
> > Subject: Re: [UMN_MAPSERVER-USERS] Map file with 35,000+
> > Lines... management issue
> >
> > Regarding this, is it possible to "include" a map file inside
> > another map file?
> >
>
> One can do just this with the INCLUDE directive.  See
> http://mapserver.gis.umn.edu/docs/reference/mapfile/Include
>
> ..Tom
>
>
> > Espen
> >
> > On 23/01/2008, Fawcett, David <[hidden email]> wrote:
> > > Everything is a performance issue, the question is, 'Is the
> > > performance acceptable to you?'...
> > >
> > > You have built a pretty massive map file.  I would look at
> > stripping
> > > it down to only what is needed for a particular application.
> > >
> > > For questions about how to improve performance, you may
> > want to look at:
> > > http://mapserver.gis.umn.edu/docs/howto/mapfiletuning
> > >
> > > David.
> > >
> > > -----Original Message-----
> > > From: UMN MapServer Users List
> > [mailto:[hidden email]]
> > > On Behalf Of riteshambastha
> > > Sent: Wednesday, January 23, 2008 12:59 PM
> > > To: [hidden email]
> > > Subject: Re: [UMN_MAPSERVER-USERS] Map file with 35,000+ Lines...
> > > management issue
> > >
> > >
> > > Thanks Rich,
> > >
> > > Yeah, even I am trying to solve this issue by separating
> > the huge map
> > > file with smaller chunks of many map files. Am bit confused
> > if it will
> > > lead to performance issues.
> > >
> > > Warm Regards,
> > > Ritz
> > >
> > >
> > >
> > > rich.fromm wrote:
> > > >
> > > >
> > > > riteshambastha wrote:
> > > >>
> > > >> I have developed a map file with more than 35,000 Lines.
> > > >>
> > > >> Please throw some lights over writing map files by
> > following best
> > > >> practices.
> > > >>
> > > >
> > > > Have you considered more than one map file, depending on the
> > > > circumstances, each one a bit smaller, rather than one
> > huge map file?
> > > >
> > > > It's hard to say how appropriate that may or may not be without
> > > > knowing more about your circumstances, but that's what I have.
> > > >
> > > > - Rich
> > > >
> > > >
> > >
> > > --
> > > View this message in context:
> > >
> > http://www.nabble.com/Map-file-with-35%2C000%2B-Lines...-management-is
> > > su
> > > e-tp15048892p15048905.html
> > > Sent from the Mapserver - User mailing list archive at Nabble.com.
> > >
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Map file with 35,000+ Lines... management issue

Ed McNierney-4
However, that technique is a management solution, NOT a performance solution!

        - Ed

Ed McNierney
Chief Mapmaker
Demand Media / TopoZone.com
73 Princeton Street, Suite 305
North Chelmsford, MA  01863
[hidden email]
Phone: +1 (978) 251-4242
Fax: +1 (978) 251-1396

-----Original Message-----
From: UMN MapServer Users List [mailto:[hidden email]] On Behalf Of Espen Isaksen
Sent: Wednesday, January 23, 2008 2:31 PM
To: [hidden email]
Subject: Re: [UMN_MAPSERVER-USERS] Map file with 35,000+ Lines... management issue

Thanks! Don't know how I missed that one...

Espen

On 23/01/2008, Kralidis,Tom [Burlington] <[hidden email]> wrote:

>
>
>
> > -----Original Message-----
> > From: UMN MapServer Users List
> > [mailto:[hidden email]] On Behalf Of Espen Isaksen
> > Sent: 23 January, 2008 2:26 PM
> > To: [hidden email]
> > Subject: Re: [UMN_MAPSERVER-USERS] Map file with 35,000+
> > Lines... management issue
> >
> > Regarding this, is it possible to "include" a map file inside
> > another map file?
> >
>
> One can do just this with the INCLUDE directive.  See
> http://mapserver.gis.umn.edu/docs/reference/mapfile/Include
>
> ..Tom
>
>
> > Espen
> >
> > On 23/01/2008, Fawcett, David <[hidden email]> wrote:
> > > Everything is a performance issue, the question is, 'Is the
> > > performance acceptable to you?'...
> > >
> > > You have built a pretty massive map file.  I would look at
> > stripping
> > > it down to only what is needed for a particular application.
> > >
> > > For questions about how to improve performance, you may
> > want to look at:
> > > http://mapserver.gis.umn.edu/docs/howto/mapfiletuning
> > >
> > > David.
> > >
> > > -----Original Message-----
> > > From: UMN MapServer Users List
> > [mailto:[hidden email]]
> > > On Behalf Of riteshambastha
> > > Sent: Wednesday, January 23, 2008 12:59 PM
> > > To: [hidden email]
> > > Subject: Re: [UMN_MAPSERVER-USERS] Map file with 35,000+ Lines...
> > > management issue
> > >
> > >
> > > Thanks Rich,
> > >
> > > Yeah, even I am trying to solve this issue by separating
> > the huge map
> > > file with smaller chunks of many map files. Am bit confused
> > if it will
> > > lead to performance issues.
> > >
> > > Warm Regards,
> > > Ritz
> > >
> > >
> > >
> > > rich.fromm wrote:
> > > >
> > > >
> > > > riteshambastha wrote:
> > > >>
> > > >> I have developed a map file with more than 35,000 Lines.
> > > >>
> > > >> Please throw some lights over writing map files by
> > following best
> > > >> practices.
> > > >>
> > > >
> > > > Have you considered more than one map file, depending on the
> > > > circumstances, each one a bit smaller, rather than one
> > huge map file?
> > > >
> > > > It's hard to say how appropriate that may or may not be without
> > > > knowing more about your circumstances, but that's what I have.
> > > >
> > > > - Rich
> > > >
> > > >
> > >
> > > --
> > > View this message in context:
> > >
> > http://www.nabble.com/Map-file-with-35%2C000%2B-Lines...-management-is
> > > su
> > > e-tp15048892p15048905.html
> > > Sent from the Mapserver - User mailing list archive at Nabble.com.
> > >
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Map file with 35,000+ Lines... management issue

blammo
In reply to this post by ritesh ambastha
Ritz,
 
Whew, 35k lines, that big.  How many layers is that anyway?   The Googlish mapfile I just did only has 1100 lines in it, and that's mostly for readability.  I could probably get it down to half that size if I tried.
 
Don't know what I can contribute as a "Best Practice", because I feel that in most cases, that form follows function, if you need a capability, you build it.  Anyway, here are some of my thoughts.
 
These same sorts of performance questions crossed my mind too.  The Googlish mapfile I've been working on has 72 separate STYLE definitions for example.  Mostly ranged around threshholding of certain styles.  I can see that adding in the Water bodies, Railroad, Parks, and such, is really going to make this thing big.  I may just do those as separate MapFiles though since GeoMoose handles things like these separations very nicely.
 
These are some of the primary reasons that contributed to the way we've built GeoMoose as a client and why it runs against MapServer CGI, so that it can abstract the layer calls in this fashion.  We're running 135+ layers internally at the moment, and they all have their own MAPFILE and are all called separately from the client.  It has made life much easier with regard to MapFile creation and maintenance, since each data custodian handles their respective MAPFILE.  The performance issues are minimized well since even the data intensive layers are not too bad from a performance standpoint.
 
But even my Googlish looking mapfile got prettty big (in my opinion) for simply displaying centerlines of streets.   I've learned quite a bit from these exercises about these types of questions.  While I have yet to attack the performance side of things, I anticpate that I'll need to segregate the data out at differing thresholds in order to gain some performance boots.  We're all about doing dynamic requests here since many of our datasets change very frequently, in some cases, down to the minute.  I may look into tiling at some point in the future, but it will still be only for some of the layers, there will still be a need to have this dynamic request structure in place.
 
bobb
 
 
 
Bob Basques
GIS Systems Developer

Moose Powered GISMO
 
Powered by
 


>>> riteshambastha <[hidden email]> wrote:
Dear Readers,

I have developed a map file with more than 35,000 Lines. Its size will grow
by double/triple in next few months. Now, I am trying to tune my map file by
removing unwanted lines. Still, I am bit confused about its maintenance.

Please throw some lights over writing map files by following best practices.

Thanks in advance.

Regards,
Ritz
--
View this message in context: http://www.nabble.com/Map-file-with-35,000+-Lines...-management-issue-tp15048892p15048892.html
Sent from the Mapserver - User mailing list archive at Nabble.com.
Reply | Threaded
Open this post in threaded view
|

Re: Map file with 35,000+ Lines... management issue

blammo
In reply to this post by Kralidis,Tom [Ontario]
Espen,
 
What if any are the performance hits with this approach?  Is the Mapfile assembled before any sort of reading if it is attempted, and it's simply the opening of the additional files that affect the performance?
 
I do have a system like this in place, but it's for some rather simple looking cartography and it works well with a request handler (Not GeoMoose) that assembles the INCLUDE chunk upon request to build the MAPFILE on the fly for the MapSERVER CGI request, this is real borderline DataBase stuff too, as it's really getting close to the point of putting it into a DB instead of keeping in FLAT files.
 
This applies to my previous message too about the abundance of layers in our applications.  It would be far easier to build a Webified version of a MAPFILE builder, that still maintained the option(s) of letting each data custodian handle their own cartography for example.
 
This would certainly be a way of removing the some of the MAPFILE complexities from the average data custodians.  It's something we have been thinking about doing as a module for GeoMoose for example.  Just haven't got it all quite figured out completely yet.   There are some other similar systems for this too that I need to review first as well.  But in the interest of getting the MAPFILE smaller and more manageable . . . .  :c)
 
bobb
 
 
 
 
 
Bob Basques
GIS Systems Developer

Moose Powered GISMO
 
Powered by
 


>>> Espen Isaksen <[hidden email]> wrote:
Thanks! Don't know how I missed that one...

Espen

On 23/01/2008, Kralidis,Tom [Burlington] <[hidden email]> wrote:

>
>
>
> > -----Original Message-----
> > From: UMN MapServer Users List
> > [mailto:[hidden email]] On Behalf Of Espen Isaksen
> > Sent: 23 January, 2008 2:26 PM
> > To: [hidden email]
> > Subject: Re: [UMN_MAPSERVER-USERS] Map file with 35,000+
> > Lines... management issue
> >
> > Regarding this, is it possible to "include" a map file inside
> > another map file?
> >
>
> One can do just this with the INCLUDE directive.  See
> http://mapserver.gis.umn.edu/docs/reference/mapfile/Include
>
> ..Tom
>
>
> > Espen
> >
> > On 23/01/2008, Fawcett, David <[hidden email]> wrote:
> > > Everything is a performance issue, the question is, 'Is the
> > > performance acceptable to you?'...
> > >
> > > You have built a pretty massive map file.  I would look at
> > stripping
> > > it down to only what is needed for a particular application.
> > >
> > > For questions about how to improve performance, you may
> > want to look at:
> > > http://mapserver.gis.umn.edu/docs/howto/mapfiletuning
> > >
> > > David.
> > >
> > > -----Original Message-----
> > > From: UMN MapServer Users List
> > [mailto:[hidden email]]
> > > On Behalf Of riteshambastha
> > > Sent: Wednesday, January 23, 2008 12:59 PM
> > > To: [hidden email]
> > > Subject: Re: [UMN_MAPSERVER-USERS] Map file with 35,000+ Lines...
> > > management issue
> > >
> > >
> > > Thanks Rich,
> > >
> > > Yeah, even I am trying to solve this issue by separating
> > the huge map
> > > file with smaller chunks of many map files. Am bit confused
> > if it will
> > > lead to performance issues.
> > >
> > > Warm Regards,
> > > Ritz
> > >
> > >
> > >
> > > rich.fromm wrote:
> > > >
> > > >
> > > > riteshambastha wrote:
> > > >>
> > > >> I have developed a map file with more than 35,000 Lines.
> > > >>
> > > >> Please throw some lights over writing map files by
> > following best
> > > >> practices.
> > > >>
> > > >
> > > > Have you considered more than one map file, depending on the
> > > > circumstances, each one a bit smaller, rather than one
> > huge map file?
> > > >
> > > > It's hard to say how appropriate that may or may not be without
> > > > knowing more about your circumstances, but that's what I have.
> > > >
> > > > - Rich
> > > >
> > > >
> > >
> > > --
> > > View this message in context:
> > >
> > http://www.nabble.com/Map-file-with-35,000+-Lines...-management-is
> > > su
> > > e-tp15048892p15048905html
> > > Sent from the Mapserver - User mailing list archive at Nabble.com.
> > >
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Map file with 35,000+ Lines... management issue

ritesh ambastha
In reply to this post by blammo
Thanks Bob,

The map file consists of 658 Layers.
It runs with openlayers and postgis.

Now, am trying to sort out the best way for solving this issue.
Your reply helped me to view at the problems+solutions in broad spectrum..

Warm Regards,
Ritesh

On Jan 24, 2008 1:04 AM, Bob Basques <[hidden email]> wrote:

>
>
> Ritz,
>
> Whew, 35k lines, that big.  How many layers is that anyway?   The Googlish
> mapfile I just did only has 1100 lines in it, and that's mostly for
> readability.  I could probably get it down to half that size if I tried.
>
> Don't know what I can contribute as a "Best Practice", because I feel that
> in most cases, that form follows function, if you need a capability, you
> build it.  Anyway, here are some of my thoughts.
>
> These same sorts of performance questions crossed my mind too.  The Googlish
> mapfile I've been working on has 72 separate STYLE definitions for example.
> Mostly ranged around threshholding of certain styles.  I can see that adding
> in the Water bodies, Railroad, Parks, and such, is really going to make this
> thing big.  I may just do those as separate MapFiles though since GeoMoose
> handles things like these separations very nicely.
>
> These are some of the primary reasons that contributed to the way we've
> built GeoMoose as a client and why it runs against MapServer CGI, so that it
> can abstract the layer calls in this fashion.  We're running 135+ layers
> internally at the moment, and they all have their own MAPFILE and are all
> called separately from the client.  It has made life much easier with regard
> to MapFile creation and maintenance, since each data custodian handles their
> respective MAPFILE.  The performance issues are minimized well since even
> the data intensive layers are not too bad from a performance standpoint.
>
> But even my Googlish looking mapfile got prettty big (in my opinion) for
> simply displaying centerlines of streets.   I've learned quite a bit from
> these exercises about these types of questions.  While I have yet to attack
> the performance side of things, I anticpate that I'll need to segregate the
> data out at differing thresholds in order to gain some performance boots.
> We're all about doing dynamic requests here since many of our datasets
> change very frequently, in some cases, down to the minute.  I may look into
> tiling at some point in the future, but it will still be only for some of
> the layers, there will still be a need to have this dynamic request
> structure in place.
>
> bobb
>
>
>
>
>
>
> Bob Basques
> GIS Systems Developer
> City of Saint Paul, MN
>
>
> GISmo
> Powered by
> GeoMOOSE
>
>
> >>> riteshambastha <[hidden email]> wrote:
>
> Dear Readers,
>
> I have developed a map file with more than 35,000 Lines. Its size will grow
> by double/triple in next few months. Now, I am trying to tune my map file by
> removing unwanted lines. Still, I am bit confused about its maintenance.
>
>
> Please throw some lights over writing map files by following best practices.
>
> Thanks in advance.
>
> Regards,
> Ritz
> --
> View this message in context:
> http://www.nabble.com/Map-file-with-35,000+-Lines...-management-issue-tp15048892p15048892.html
>
> Sent from the Mapserver - User mailing list archive at Nabble.com.
>



--
Ritesh Ambastha,

Project Manager
Mobiance Technologies,
Bangalore

+91-80-41264755
Reply | Threaded
Open this post in threaded view
|

Re: Map file with 35,000+ Lines... management issue

Ed McNierney-4
Ritesh -

Obviously you cannot possibly need 658 layers to respond to a single map request.  The first thing to do is figure out how to split up this one map file into multiple map files and then use some front-end logic to select the appropriate map file to serve a particular request.  You may find the INCLUDE directive helpful, but you should simply begin by eliminating the unnecessary.  The theoretical ideal would be that a single MapServer request would specify a map file that contained nothing more than the symbols, fonts, and layers required to serve that single request - nothing more.

        - Ed

Ed McNierney
Chief Mapmaker
Demand Media / TopoZone.com
73 Princeton Street, Suite 305
North Chelmsford, MA  01863
[hidden email]
Phone: +1 (978) 251-4242
Fax: +1 (978) 251-1396

-----Original Message-----
From: UMN MapServer Users List [mailto:[hidden email]] On Behalf Of ritesh ambastha
Sent: Wednesday, January 23, 2008 2:44 PM
To: [hidden email]
Subject: Re: [UMN_MAPSERVER-USERS] Map file with 35,000+ Lines... management issue

Thanks Bob,

The map file consists of 658 Layers.
It runs with openlayers and postgis.

Now, am trying to sort out the best way for solving this issue.
Your reply helped me to view at the problems+solutions in broad spectrum..

Warm Regards,
Ritesh

On Jan 24, 2008 1:04 AM, Bob Basques <[hidden email]> wrote:

>
>
> Ritz,
>
> Whew, 35k lines, that big.  How many layers is that anyway?   The Googlish
> mapfile I just did only has 1100 lines in it, and that's mostly for
> readability.  I could probably get it down to half that size if I tried.
>
> Don't know what I can contribute as a "Best Practice", because I feel that
> in most cases, that form follows function, if you need a capability, you
> build it.  Anyway, here are some of my thoughts.
>
> These same sorts of performance questions crossed my mind too.  The Googlish
> mapfile I've been working on has 72 separate STYLE definitions for example.
> Mostly ranged around threshholding of certain styles.  I can see that adding
> in the Water bodies, Railroad, Parks, and such, is really going to make this
> thing big.  I may just do those as separate MapFiles though since GeoMoose
> handles things like these separations very nicely.
>
> These are some of the primary reasons that contributed to the way we've
> built GeoMoose as a client and why it runs against MapServer CGI, so that it
> can abstract the layer calls in this fashion.  We're running 135+ layers
> internally at the moment, and they all have their own MAPFILE and are all
> called separately from the client.  It has made life much easier with regard
> to MapFile creation and maintenance, since each data custodian handles their
> respective MAPFILE.  The performance issues are minimized well since even
> the data intensive layers are not too bad from a performance standpoint.
>
> But even my Googlish looking mapfile got prettty big (in my opinion) for
> simply displaying centerlines of streets.   I've learned quite a bit from
> these exercises about these types of questions.  While I have yet to attack
> the performance side of things, I anticpate that I'll need to segregate the
> data out at differing thresholds in order to gain some performance boots.
> We're all about doing dynamic requests here since many of our datasets
> change very frequently, in some cases, down to the minute.  I may look into
> tiling at some point in the future, but it will still be only for some of
> the layers, there will still be a need to have this dynamic request
> structure in place.
>
> bobb
>
>
>
>
>
>
> Bob Basques
> GIS Systems Developer
> City of Saint Paul, MN
>
>
> GISmo
> Powered by
> GeoMOOSE
>
>
> >>> riteshambastha <[hidden email]> wrote:
>
> Dear Readers,
>
> I have developed a map file with more than 35,000 Lines. Its size will grow
> by double/triple in next few months. Now, I am trying to tune my map file by
> removing unwanted lines. Still, I am bit confused about its maintenance.
>
>
> Please throw some lights over writing map files by following best practices.
>
> Thanks in advance.
>
> Regards,
> Ritz
> --
> View this message in context:
> http://www.nabble.com/Map-file-with-35,000+-Lines...-management-issue-tp15048892p15048892.html
>
> Sent from the Mapserver - User mailing list archive at Nabble.com.
>



--
Ritesh Ambastha,

Project Manager
Mobiance Technologies,
Bangalore

+91-80-41264755
Reply | Threaded
Open this post in threaded view
|

Re: Map file with 35,000+ Lines... management issue

blammo
In reply to this post by ritesh ambastha
Ed,
 
While I do have 72 STYLE def's, I only have 11 layers defined in my Googlish mapfile, that's 11 layer definitions for essentially the same data type, Street centerlines.  That would equate to approximately 60 real layers being queried, granted, the googlish mapfile I've been working on is somewhat complex, but I would venture that most layers could be as complex with similar cartographic needs.
 
A couple of the layers i have are for Legends and such for additional output control, and if the control were not a big factor, they could be removed altogether if needed.   But it does point to the need in some circles (like those of us who fancy ourselves as some sort of fancy "Cartographer" ;c) to be able to add in our flourishes to the outputs that do indeed require defining more than one layer.
 
Some instances will require these mapfiles to get very large, and the standards bar is being risen very fast IMHO, and is getting harder and harder to keep up with some of the average user expectations for readability as well as printabilty of online maps.  This is a high priority item with me, mostly because I don't want to be stuck doing cartography day in and day out.   I'm hoping that there will be some discussion to come out of this Thread related to developing some different methods for applying styles that may end up being easier to maintain as well as easier to teach to others.
 
bobb
 
 
 
 
 
Bob Basques
GIS Systems Developer

Moose Powered GISMO
 
Powered by
 


>>> Ed McNierney <[hidden email]> wrote:
Ritesh -

Obviously you cannot possibly need 658 layers to respond to a single map =
request.  The first thing to do is figure out how to split up this one =
map file into multiple map files and then use some front-end logic to =
select the appropriate map file to serve a particular request.  You may =
find the INCLUDE directive helpful, but you should simply begin by =
eliminating the unnecessary.  The theoretical ideal would be that a =
single MapServer request would specify a map file that contained nothing =
more than the symbols, fonts, and layers required to serve that single =
request - nothing more.

- Ed

Ed McNierney
Chief Mapmaker
Demand Media / TopoZone.com
73 Princeton Street, Suite 305
North Chelmsford, MA=A0 01863
[hidden email]
Phone: +1 (978) 251-4242
Fax: +1 (978) 251-1396

-----Original Message-----
From: UMN MapServer Users List [mailto:[hidden email]] On =
Behalf Of ritesh ambastha
Sent: Wednesday, January 23, 2008 2:44 PM
To: [hidden email]
Subject: Re: [UMN_MAPSERVER-USERS] Map file with 35,000+ Lines... =
management issue

Thanks Bob,

The map file consists of 658 Layers.
It runs with openlayers and postgis.

Now, am trying to sort out the best way for solving this issue.
Your reply helped me to view at the problems+solutions in broad =
spectrum..

Warm Regards,
Ritesh

On Jan 24, 2008 1:04 AM, Bob Basques <[hidden email]> =
wrote:
>
>
> Ritz,
>
> Whew, 35k lines, that big.  How many layers is that anyway?   The =
Googlish
> mapfile I just did only has 1100 lines in it, and that's mostly for
> readability.  I could probably get it down to half that size if I =
tried.
>
> Don't know what I can contribute as a "Best Practice", because I feel =
that
> in most cases, that form follows function, if you need a capability, =
you
> build it.  Anyway, here are some of my thoughts.
>
> These same sorts of performance questions crossed my mind too.  The =
Googlish
> mapfile I've been working on has 72 separate STYLE definitions for =
example.
> Mostly ranged around threshholding of certain styles.  I can see that =
adding
> in the Water bodies, Railroad, Parks, and such, is really going to =
make this
> thing big.  I may just do those as separate MapFiles though since =
GeoMoose
> handles things like these separations very nicely.
>
> These are some of the primary reasons that contributed to the way =
we've
> built GeoMoose as a client and why it runs against MapServer CGI, so =
that it
> can abstract the layer calls in this fashion.  We're running 135+ =
layers
> internally at the moment, and they all have their own MAPFILE and are =
all
> called separately from the client.  It has made life much easier with =
regard
> to MapFile creation and maintenance, since each data custodian handles =
their
> respective MAPFILE.  The performance issues are minimized well since =
even
> the data intensive layers are not too bad from a performance =
standpoint.
>
> But even my Googlish looking mapfile got prettty big (in my opinion) =
for
> simply displaying centerlines of streets.   I've learned quite a bit =
from
> these exercises about these types of questions.  While I have yet to =
attack
> the performance side of things, I anticpate that I'll need to =
segregate the
> data out at differing thresholds in order to gain some performance =
boots.
> We're all about doing dynamic requests here since many of our datasets
> change very frequently, in some cases, down to the minute.  I may look =
into
> tiling at some point in the future, but it will still be only for some =
of

> the layers, there will still be a need to have this dynamic request
> structure in place.
>
> bobb
>
>
>
>
>
>
> Bob Basques
> GIS Systems Developer
> City of Saint Paul, MN
>
>
> GISmo
> Powered by
> GeoMOOSE
>
>
> >>> riteshambastha <[hidden email]> wrote:
>
> Dear Readers,
>
> I have developed a map file with more than 35,000 Lines. Its size will =
grow
> by double/triple in next few months. Now, I am trying to tune my map =
file by
> removing unwanted lines. Still, I am bit confused about its =
maintenance.
>
>
> Please throw some lights over writing map files by following best =
practices.
>
> Thanks in advance.
>
> Regards,
> Ritz
> --
> View this message in context:
> =
http://www.nabble.com/Map-file-with-35,000+-Lines...-management-issue-tp1=
5048892p15048892.html
>
> Sent from the Mapserver - User mailing list archive at Nabble.com.
>



--=20
Ritesh Ambastha,

Project Manager
Mobiance Technologies,
Bangalore

+91-80-41264755
Reply | Threaded
Open this post in threaded view
|

Re: Map file with 35,000+ Lines... management issue

Stephen Woodbridge
In reply to this post by ritesh ambastha
Ok, I need to ask the obvious question, WHY? do you feel you need 658
layers. Is this because you have lots of shapefiles? and most of the
layer definitions are the same except for the data source?

For Tiger data I have 33000 shapefiles, but I only have about 20+-
layers. Are you using tileindexes? Do you know what they are? Just
trying to diagnose your situation a little better so we can help.

-Steve W

ritesh ambastha wrote:

> Thanks Bob,
>
> The map file consists of 658 Layers.
> It runs with openlayers and postgis.
>
> Now, am trying to sort out the best way for solving this issue.
> Your reply helped me to view at the problems+solutions in broad spectrum..
>
> Warm Regards,
> Ritesh
>
> On Jan 24, 2008 1:04 AM, Bob Basques <[hidden email]> wrote:
>>
>> Ritz,
>>
>> Whew, 35k lines, that big.  How many layers is that anyway?   The Googlish
>> mapfile I just did only has 1100 lines in it, and that's mostly for
>> readability.  I could probably get it down to half that size if I tried.
>>
>> Don't know what I can contribute as a "Best Practice", because I feel that
>> in most cases, that form follows function, if you need a capability, you
>> build it.  Anyway, here are some of my thoughts.
>>
>> These same sorts of performance questions crossed my mind too.  The Googlish
>> mapfile I've been working on has 72 separate STYLE definitions for example.
>> Mostly ranged around threshholding of certain styles.  I can see that adding
>> in the Water bodies, Railroad, Parks, and such, is really going to make this
>> thing big.  I may just do those as separate MapFiles though since GeoMoose
>> handles things like these separations very nicely.
>>
>> These are some of the primary reasons that contributed to the way we've
>> built GeoMoose as a client and why it runs against MapServer CGI, so that it
>> can abstract the layer calls in this fashion.  We're running 135+ layers
>> internally at the moment, and they all have their own MAPFILE and are all
>> called separately from the client.  It has made life much easier with regard
>> to MapFile creation and maintenance, since each data custodian handles their
>> respective MAPFILE.  The performance issues are minimized well since even
>> the data intensive layers are not too bad from a performance standpoint.
>>
>> But even my Googlish looking mapfile got prettty big (in my opinion) for
>> simply displaying centerlines of streets.   I've learned quite a bit from
>> these exercises about these types of questions.  While I have yet to attack
>> the performance side of things, I anticpate that I'll need to segregate the
>> data out at differing thresholds in order to gain some performance boots.
>> We're all about doing dynamic requests here since many of our datasets
>> change very frequently, in some cases, down to the minute.  I may look into
>> tiling at some point in the future, but it will still be only for some of
>> the layers, there will still be a need to have this dynamic request
>> structure in place.
>>
>> bobb
>>
>>
>>
>>
>>
>>
>> Bob Basques
>> GIS Systems Developer
>> City of Saint Paul, MN
>>
>>
>> GISmo
>> Powered by
>> GeoMOOSE
>>
>>
>>>>> riteshambastha <[hidden email]> wrote:
>> Dear Readers,
>>
>> I have developed a map file with more than 35,000 Lines. Its size will grow
>> by double/triple in next few months. Now, I am trying to tune my map file by
>> removing unwanted lines. Still, I am bit confused about its maintenance.
>>
>>
>> Please throw some lights over writing map files by following best practices.
>>
>> Thanks in advance.
>>
>> Regards,
>> Ritz
>> --
>> View this message in context:
>> http://www.nabble.com/Map-file-with-35,000+-Lines...-management-issue-tp15048892p15048892.html
>>
>> Sent from the Mapserver - User mailing list archive at Nabble.com.
>>
>
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Map file with 35,000+ Lines... management issue

ritesh ambastha
In reply to this post by Espen Isaksen
Dear Espen,

you can use 'include' for including multiple map files.
Make one master map file, write extents etc there.. eg:

MAP
 NAME 'include'
 EXTENT 0 0 500 500
 SIZE 250 250

 INCLUDE "c:/ms4w/apps/partners/trialPost.map"
 INCLUDE "c:/ms4w/apps/partners/bangPost.map"
END


Then, in the included map files, just add the LAYER parts.

Thats all !

Regard,
Ritz

Espen Isaksen wrote
Thanks! Don't know how I missed that one...

Espen

On 23/01/2008, Kralidis,Tom [Burlington] <Tom.Kralidis@ec.gc.ca> wrote:
>
>
>
> > -----Original Message-----
> > From: UMN MapServer Users List
> > [mailto:MAPSERVER-USERS@LISTS.UMN.EDU] On Behalf Of Espen Isaksen
> > Sent: 23 January, 2008 2:26 PM
> > To: MAPSERVER-USERS@LISTS.UMN.EDU
> > Subject: Re: [UMN_MAPSERVER-USERS] Map file with 35,000+
> > Lines... management issue
> >
> > Regarding this, is it possible to "include" a map file inside
> > another map file?
> >
>
> One can do just this with the INCLUDE directive.  See
> http://mapserver.gis.umn.edu/docs/reference/mapfile/Include
>
> ..Tom
>
>
> > Espen
> >
> > On 23/01/2008, Fawcett, David <David.Fawcett@state.mn.us> wrote:
> > > Everything is a performance issue, the question is, 'Is the
> > > performance acceptable to you?'...
> > >
> > > You have built a pretty massive map file.  I would look at
> > stripping
> > > it down to only what is needed for a particular application.
> > >
> > > For questions about how to improve performance, you may
> > want to look at:
> > > http://mapserver.gis.umn.edu/docs/howto/mapfiletuning
> > >
> > > David.
> > >
> > > -----Original Message-----
> > > From: UMN MapServer Users List
> > [mailto:MAPSERVER-USERS@LISTS.UMN.EDU]
> > > On Behalf Of riteshambastha
> > > Sent: Wednesday, January 23, 2008 12:59 PM
> > > To: MAPSERVER-USERS@LISTS.UMN.EDU
> > > Subject: Re: [UMN_MAPSERVER-USERS] Map file with 35,000+ Lines...
> > > management issue
> > >
> > >
> > > Thanks Rich,
> > >
> > > Yeah, even I am trying to solve this issue by separating
> > the huge map
> > > file with smaller chunks of many map files. Am bit confused
> > if it will
> > > lead to performance issues.
> > >
> > > Warm Regards,
> > > Ritz
> > >
> > >
> > >
> > > rich.fromm wrote:
> > > >
> > > >
> > > > riteshambastha wrote:
> > > >>
> > > >> I have developed a map file with more than 35,000 Lines.
> > > >>
> > > >> Please throw some lights over writing map files by
> > following best
> > > >> practices.
> > > >>
> > > >
> > > > Have you considered more than one map file, depending on the
> > > > circumstances, each one a bit smaller, rather than one
> > huge map file?
> > > >
> > > > It's hard to say how appropriate that may or may not be without
> > > > knowing more about your circumstances, but that's what I have.
> > > >
> > > > - Rich
> > > >
> > > >
> > >
> > > --
> > > View this message in context:
> > >
> > http://www.nabble.com/Map-file-with-35%2C000%2B-Lines...-management-is
> > > su
> > > e-tp15048892p15048905.html
> > > Sent from the Mapserver - User mailing list archive at Nabble.com.
> > >
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Map file with 35,000+ Lines... management issue

ritesh ambastha
In reply to this post by Stephen Woodbridge
Dear Stephen,

The number of layers exceeded much because I am including each individual shapefile multiple times for different Maxcale/Minscale values.  So, a shapefile is now called by 3-4 layers, each layer having different Maxscale/Minscale values.

I hope I made my point clear.

Regards,
Ritz

Stephen Woodbridge wrote
Ok, I need to ask the obvious question, WHY? do you feel you need 658
layers. Is this because you have lots of shapefiles? and most of the
layer definitions are the same except for the data source?

For Tiger data I have 33000 shapefiles, but I only have about 20+-
layers. Are you using tileindexes? Do you know what they are? Just
trying to diagnose your situation a little better so we can help.

-Steve W

ritesh ambastha wrote:
> Thanks Bob,
>
> The map file consists of 658 Layers.
> It runs with openlayers and postgis.
>
> Now, am trying to sort out the best way for solving this issue.
> Your reply helped me to view at the problems+solutions in broad spectrum..
>
> Warm Regards,
> Ritesh
>
> On Jan 24, 2008 1:04 AM, Bob Basques <Bob.Basques@ci.stpaul.mn.us> wrote:
>>
>> Ritz,
>>
>> Whew, 35k lines, that big.  How many layers is that anyway?   The Googlish
>> mapfile I just did only has 1100 lines in it, and that's mostly for
>> readability.  I could probably get it down to half that size if I tried.
>>
>> Don't know what I can contribute as a "Best Practice", because I feel that
>> in most cases, that form follows function, if you need a capability, you
>> build it.  Anyway, here are some of my thoughts.
>>
>> These same sorts of performance questions crossed my mind too.  The Googlish
>> mapfile I've been working on has 72 separate STYLE definitions for example.
>> Mostly ranged around threshholding of certain styles.  I can see that adding
>> in the Water bodies, Railroad, Parks, and such, is really going to make this
>> thing big.  I may just do those as separate MapFiles though since GeoMoose
>> handles things like these separations very nicely.
>>
>> These are some of the primary reasons that contributed to the way we've
>> built GeoMoose as a client and why it runs against MapServer CGI, so that it
>> can abstract the layer calls in this fashion.  We're running 135+ layers
>> internally at the moment, and they all have their own MAPFILE and are all
>> called separately from the client.  It has made life much easier with regard
>> to MapFile creation and maintenance, since each data custodian handles their
>> respective MAPFILE.  The performance issues are minimized well since even
>> the data intensive layers are not too bad from a performance standpoint.
>>
>> But even my Googlish looking mapfile got prettty big (in my opinion) for
>> simply displaying centerlines of streets.   I've learned quite a bit from
>> these exercises about these types of questions.  While I have yet to attack
>> the performance side of things, I anticpate that I'll need to segregate the
>> data out at differing thresholds in order to gain some performance boots.
>> We're all about doing dynamic requests here since many of our datasets
>> change very frequently, in some cases, down to the minute.  I may look into
>> tiling at some point in the future, but it will still be only for some of
>> the layers, there will still be a need to have this dynamic request
>> structure in place.
>>
>> bobb
>>
>>
>>
>>
>>
>>
>> Bob Basques
>> GIS Systems Developer
>> City of Saint Paul, MN
>>
>>
>> GISmo
>> Powered by
>> GeoMOOSE
>>
>>
>>>>> riteshambastha <ritesh.linux@GMAIL.COM> wrote:
>> Dear Readers,
>>
>> I have developed a map file with more than 35,000 Lines. Its size will grow
>> by double/triple in next few months. Now, I am trying to tune my map file by
>> removing unwanted lines. Still, I am bit confused about its maintenance.
>>
>>
>> Please throw some lights over writing map files by following best practices.
>>
>> Thanks in advance.
>>
>> Regards,
>> Ritz
>> --
>> View this message in context:
>> http://www.nabble.com/Map-file-with-35,000+-Lines...-management-issue-tp15048892p15048892.html
>>
>> Sent from the Mapserver - User mailing list archive at Nabble.com.
>>
>
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Map file with 35,000+ Lines... management issue

Stephen Woodbridge
OK, I think there is a simple solution for you, You need to create
tileindexes. tileindexes are a way to group a collection of shapefiles
that have the same content like roads into a single layer. Say you have
a file structure like:

data/city1/roads.shp
data/city2/roads.shp
data/city3/roads.shp
etc

And if you have a layer for each city/roads.shp you can combine all
these into a single layer like this:

find data -name roads.shp -print > roads-idx.in
tile4ms roads-idx.in roads-idx.shp
shptree roads-idx.shp

And in your mapfile create a layer like:

LAYER
   ...
   TILEINDEX "roads-idx.shp"
   # instead of a DATA statement
   ...
END

Now all these shapefiles in the tileindex will be considered in a very
efficient manner.

-Steve

riteshambastha wrote:

> Dear Stephen,
>
> The number of layers exceeded much because I am including each individual
> shapefile multiple times for different Maxcale/Minscale values.  So, a
> shapefile is now called by 3-4 layers, each layer having different
> Maxscale/Minscale values.
>
> I hope I made my point clear.
>
> Regards,
> Ritz
>
>
> Stephen Woodbridge wrote:
>> Ok, I need to ask the obvious question, WHY? do you feel you need 658
>> layers. Is this because you have lots of shapefiles? and most of the
>> layer definitions are the same except for the data source?
>>
>> For Tiger data I have 33000 shapefiles, but I only have about 20+-
>> layers. Are you using tileindexes? Do you know what they are? Just
>> trying to diagnose your situation a little better so we can help.
>>
>> -Steve W
>>
>> ritesh ambastha wrote:
>>> Thanks Bob,
>>>
>>> The map file consists of 658 Layers.
>>> It runs with openlayers and postgis.
>>>
>>> Now, am trying to sort out the best way for solving this issue.
>>> Your reply helped me to view at the problems+solutions in broad
>>> spectrum..
>>>
>>> Warm Regards,
>>> Ritesh
>>>
>>> On Jan 24, 2008 1:04 AM, Bob Basques <[hidden email]> wrote:
>>>> Ritz,
>>>>
>>>> Whew, 35k lines, that big.  How many layers is that anyway?   The
>>>> Googlish
>>>> mapfile I just did only has 1100 lines in it, and that's mostly for
>>>> readability.  I could probably get it down to half that size if I tried.
>>>>
>>>> Don't know what I can contribute as a "Best Practice", because I feel
>>>> that
>>>> in most cases, that form follows function, if you need a capability, you
>>>> build it.  Anyway, here are some of my thoughts.
>>>>
>>>> These same sorts of performance questions crossed my mind too.  The
>>>> Googlish
>>>> mapfile I've been working on has 72 separate STYLE definitions for
>>>> example.
>>>> Mostly ranged around threshholding of certain styles.  I can see that
>>>> adding
>>>> in the Water bodies, Railroad, Parks, and such, is really going to make
>>>> this
>>>> thing big.  I may just do those as separate MapFiles though since
>>>> GeoMoose
>>>> handles things like these separations very nicely.
>>>>
>>>> These are some of the primary reasons that contributed to the way we've
>>>> built GeoMoose as a client and why it runs against MapServer CGI, so
>>>> that it
>>>> can abstract the layer calls in this fashion.  We're running 135+ layers
>>>> internally at the moment, and they all have their own MAPFILE and are
>>>> all
>>>> called separately from the client.  It has made life much easier with
>>>> regard
>>>> to MapFile creation and maintenance, since each data custodian handles
>>>> their
>>>> respective MAPFILE.  The performance issues are minimized well since
>>>> even
>>>> the data intensive layers are not too bad from a performance standpoint.
>>>>
>>>> But even my Googlish looking mapfile got prettty big (in my opinion) for
>>>> simply displaying centerlines of streets.   I've learned quite a bit
>>>> from
>>>> these exercises about these types of questions.  While I have yet to
>>>> attack
>>>> the performance side of things, I anticpate that I'll need to segregate
>>>> the
>>>> data out at differing thresholds in order to gain some performance
>>>> boots.
>>>> We're all about doing dynamic requests here since many of our datasets
>>>> change very frequently, in some cases, down to the minute.  I may look
>>>> into
>>>> tiling at some point in the future, but it will still be only for some
>>>> of
>>>> the layers, there will still be a need to have this dynamic request
>>>> structure in place.
>>>>
>>>> bobb
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Bob Basques
>>>> GIS Systems Developer
>>>> City of Saint Paul, MN
>>>>
>>>>
>>>> GISmo
>>>> Powered by
>>>> GeoMOOSE
>>>>
>>>>
>>>>>>> riteshambastha <[hidden email]> wrote:
>>>> Dear Readers,
>>>>
>>>> I have developed a map file with more than 35,000 Lines. Its size will
>>>> grow
>>>> by double/triple in next few months. Now, I am trying to tune my map
>>>> file by
>>>> removing unwanted lines. Still, I am bit confused about its maintenance.
>>>>
>>>>
>>>> Please throw some lights over writing map files by following best
>>>> practices.
>>>>
>>>> Thanks in advance.
>>>>
>>>> Regards,
>>>> Ritz
>>>> --
>>>> View this message in context:
>>>> http://www.nabble.com/Map-file-with-35,000+-Lines...-management-issue-tp15048892p15048892.html
>>>>
>>>> Sent from the Mapserver - User mailing list archive at Nabble.com.
>>>>
>>>
>>>
>>
>
Reply | Threaded
Open this post in threaded view
|

Re: Map file with 35,000+ Lines... management issue

ritesh ambastha
In reply to this post by ritesh ambastha
Yeah, you pointed out a very valid point.
As open layers handled my map file in an efficient manner, I didn't overloaded myself with implementation of Tiling.

But, there is one more serious thought. Lets try to understand this problem:

Example:

Layer 1 : [shapefile] Roads of a country.

The attributes of this road layer changes w.r.t. zoom-levels. For instance, at higher zoom level roads seems thinner, and at lower it seems broader.
So, for these zoom-levels, I used Maxscale/Minscale values and developed multiple layers for this Layer.

Layer 2: [shapefile] Roads of a state

The above case is true with this layer too.

I hope this could magnify my problem.

Regards,
Ritz

On Jan 24, 2008 8:55 PM, Fawcett, David <[hidden email]> wrote:
You mention creating individual layers for each shapefile.  So, does
this mean that if you have a shapefile of road data for each state you
are creating a MapServer layer for each shapefile?

If this is the case, you can definitely reduce the number of layers (and
likely increase performance) by using a tileindex.

A tileindex is a polygon dataset, usually a shapefile, with a polygon
that defines the boundaries of each individual dataset.  In other words,
you would use a utility to create a new shapefile with polygons that
define the boundaries of each of your state road shapefiles.  In the
attribute table of your tileindex, there is a column that tells
MapServer where to find the actual shapefile represented by a particular
feature.

You then create just one layer using the tileindex as the data source.
Take a look at:  http://mapserver.gis.umn.edu/docs/howto/tileindex if
you haven't already.

David.

-----Original Message-----
From: UMN MapServer Users List [mailto:[hidden email]] On
Behalf Of riteshambastha
Sent: Thursday, January 24, 2008 9:16 AM
To: [hidden email]
Subject: Re: [UMN_MAPSERVER-USERS] Map file with 35,000+ Lines...
management issue


Dear Stephen,

The number of layers exceeded much because I am including each
individual shapefile multiple times for different Maxcale/Minscale
values.  So, a shapefile is now called by 3-4 layers, each layer having
different Maxscale/Minscale values.

I hope I made my point clear.

Regards,
Ritz


Stephen Woodbridge wrote:

>
> Ok, I need to ask the obvious question, WHY? do you feel you need 658
> layers. Is this because you have lots of shapefiles? and most of the
> layer definitions are the same except for the data source?
>
> For Tiger data I have 33000 shapefiles, but I only have about 20+-
> layers. Are you using tileindexes? Do you know what they are? Just
> trying to diagnose your situation a little better so we can help.
>
> -Steve W
>
> ritesh ambastha wrote:

>> Thanks Bob,
>>
>> The map file consists of 658 Layers.
>> It runs with openlayers and postgis.
>>
>> Now, am trying to sort out the best way for solving this issue. Your
>> reply helped me to view at the problems+solutions in broad spectrum..
>>
>> Warm Regards,
>> Ritesh
>>
>> On Jan 24, 2008 1:04 AM, Bob Basques <[hidden email]>
>> wrote:
>>>
>>> Ritz,
>>>
>>> Whew, 35k lines, that big.  How many layers is that anyway?   The
>>> Googlish
>>> mapfile I just did only has 1100 lines in it, and that's mostly for
>>> readability.  I could probably get it down to half that size if I
>>> tried.
>>>
>>> Don't know what I can contribute as a "Best Practice", because I
>>> feel that in most cases, that form follows function, if you need a
>>> capability, you build it.  Anyway, here are some of my thoughts.
>>>
>>> These same sorts of performance questions crossed my mind too.  The
>>> Googlish mapfile I've been working on has 72 separate STYLE
>>> definitions for example.
>>> Mostly ranged around threshholding of certain styles.  I can see
that
>>> adding
>>> in the Water bodies, Railroad, Parks, and such, is really going to
make
>>> this
>>> thing big.  I may just do those as separate MapFiles though since
>>> GeoMoose
>>> handles things like these separations very nicely.
>>>
>>> These are some of the primary reasons that contributed to the way
>>> we've built GeoMoose as a client and why it runs against MapServer
>>> CGI, so that it can abstract the layer calls in this fashion.  We're

>>> running 135+ layers internally at the moment, and they all have
>>> their own MAPFILE and are all
>>> called separately from the client.  It has made life much easier
with
>>> regard
>>> to MapFile creation and maintenance, since each data custodian
handles
>>> their
>>> respective MAPFILE.  The performance issues are minimized well since
>>> even
>>> the data intensive layers are not too bad from a performance
standpoint.
>>>
>>> But even my Googlish looking mapfile got prettty big (in my opinion)
for
>>> simply displaying centerlines of streets.   I've learned quite a bit
>>> from
>>> these exercises about these types of questions.  While I have yet to

>>> attack the performance side of things, I anticpate that I'll need to

>>> segregate the
>>> data out at differing thresholds in order to gain some performance
>>> boots.
>>> We're all about doing dynamic requests here since many of our
datasets
>>> change very frequently, in some cases, down to the minute.  I may
look
>>> into
>>> tiling at some point in the future, but it will still be only for
some

>>> of
>>> the layers, there will still be a need to have this dynamic request
>>> structure in place.
>>>
>>> bobb
>>>
>>>
>>>
>>>
>>>
>>>
>>> Bob Basques
>>> GIS Systems Developer
>>> City of Saint Paul, MN
>>>
>>>
>>> GISmo
>>> Powered by
>>> GeoMOOSE
>>>
>>>
>>>>>> riteshambastha < [hidden email]> wrote:
>>> Dear Readers,
>>>
>>> I have developed a map file with more than 35,000 Lines. Its size
>>> will grow by double/triple in next few months. Now, I am trying to
>>> tune my map file by
>>> removing unwanted lines. Still, I am bit confused about its
maintenance.

>>>
>>>
>>> Please throw some lights over writing map files by following best
>>> practices.
>>>
>>> Thanks in advance.
>>>
>>> Regards,
>>> Ritz
>>> --
>>> View this message in context:
>>> http://www.nabble.com/Map-file-with-35,000+-Lines...-management-issu
>>> e-tp15048892p15048892.html
>>>
>>> Sent from the Mapserver - User mailing list archive at Nabble.com.
>>>
>>
>>
>>
>
>

--
View this message in context:
e-tp15048892p15065742.html
Sent from the Mapserver - User mailing list archive at Nabble.com.



--
Ritesh Ambastha,

Project Manager
Mobiance Technologies,
Bangalore

+91-80-41264755
Reply | Threaded
Open this post in threaded view
|

Re: Map file with 35,000+ Lines... management issue

Fawcett, David (MNIT)
Message
I would suggest a tileindex to display all of the individual shapefiles that make up your state_roads layer in one layer.  Same thing for your country_roads layer if it is split up in to smaller data files. 
 
You may even want to put  MINSCALE/MAXSCALE   values in place so your state_roads doesn't display when zoomed out too far. 
 
Then, for symbology purposes, you can create multiple classes for each layer, each with their own MINSCALE/MAXSCALE values so you can style your roads differently based on how far you are zoomed into.
 
David.
-----Original Message-----
From: UMN MapServer Users List [mailto:[hidden email]] On Behalf Of ritesh ambastha
Sent: Thursday, January 24, 2008 9:39 AM
To: [hidden email]
Subject: Re: [UMN_MAPSERVER-USERS] Map file with 35,000+ Lines... management issue

Yeah, you pointed out a very valid point.
As open layers handled my map file in an efficient manner, I didn't overloaded myself with implementation of Tiling.

But, there is one more serious thought. Lets try to understand this problem:

Example:

Layer 1 : [shapefile] Roads of a country.

The attributes of this road layer changes w.r.t. zoom-levels. For instance, at higher zoom level roads seems thinner, and at lower it seems broader.
So, for these zoom-levels, I used Maxscale/Minscale values and developed multiple layers for this Layer.

Layer 2: [shapefile] Roads of a state

The above case is true with this layer too.

I hope this could magnify my problem.

Regards,
Ritz

On Jan 24, 2008 8:55 PM, Fawcett, David <[hidden email]> wrote:
You mention creating individual layers for each shapefile.  So, does
this mean that if you have a shapefile of road data for each state you
are creating a MapServer layer for each shapefile?

If this is the case, you can definitely reduce the number of layers (and
likely increase performance) by using a tileindex.

A tileindex is a polygon dataset, usually a shapefile, with a polygon
that defines the boundaries of each individual dataset.  In other words,
you would use a utility to create a new shapefile with polygons that
define the boundaries of each of your state road shapefiles.  In the
attribute table of your tileindex, there is a column that tells
MapServer where to find the actual shapefile represented by a particular
feature.

You then create just one layer using the tileindex as the data source.
Take a look at:  http://mapserver.gis.umn.edu/docs/howto/tileindex if
you haven't already.

David.

-----Original Message-----
From: UMN MapServer Users List [mailto:[hidden email]] On
Behalf Of riteshambastha
Sent: Thursday, January 24, 2008 9:16 AM
To: [hidden email]
Subject: Re: [UMN_MAPSERVER-USERS] Map file with 35,000+ Lines...
management issue


Dear Stephen,

The number of layers exceeded much because I am including each
individual shapefile multiple times for different Maxcale/Minscale
values.  So, a shapefile is now called by 3-4 layers, each layer having
different Maxscale/Minscale values.

I hope I made my point clear.

Regards,
Ritz


Stephen Woodbridge wrote:

>
> Ok, I need to ask the obvious question, WHY? do you feel you need 658
> layers. Is this because you have lots of shapefiles? and most of the
> layer definitions are the same except for the data source?
>
> For Tiger data I have 33000 shapefiles, but I only have about 20+-
> layers. Are you using tileindexes? Do you know what they are? Just
> trying to diagnose your situation a little better so we can help.
>
> -Steve W
>

> ritesh ambastha wrote:
>> Thanks Bob,
>>
>> The map file consists of 658 Layers.
>> It runs with openlayers and postgis.
>>
>> Now, am trying to sort out the best way for solving this issue. Your
>> reply helped me to view at the problems+solutions in broad spectrum..
>>
>> Warm Regards,
>> Ritesh
>>
>> On Jan 24, 2008 1:04 AM, Bob Basques <[hidden email]>
>> wrote:
>>>
>>> Ritz,
>>>
>>> Whew, 35k lines, that big.  How many layers is that anyway?   The
>>> Googlish
>>> mapfile I just did only has 1100 lines in it, and that's mostly for
>>> readability.  I could probably get it down to half that size if I
>>> tried.
>>>
>>> Don't know what I can contribute as a "Best Practice", because I
>>> feel that in most cases, that form follows function, if you need a
>>> capability, you build it.  Anyway, here are some of my thoughts.
>>>
>>> These same sorts of performance questions crossed my mind too.  The
>>> Googlish mapfile I've been working on has 72 separate STYLE
>>> definitions for example.
>>> Mostly ranged around threshholding of certain styles.  I can see
that
>>> adding
>>> in the Water bodies, Railroad, Parks, and such, is really going to
make
>>> this
>>> thing big.  I may just do those as separate MapFiles though since
>>> GeoMoose
>>> handles things like these separations very nicely.
>>>
>>> These are some of the primary reasons that contributed to the way
>>> we've built GeoMoose as a client and why it runs against MapServer
>>> CGI, so that it can abstract the layer calls in this fashion.  We're

>>> running 135+ layers internally at the moment, and they all have
>>> their own MAPFILE and are all
>>> called separately from the client.  It has made life much easier
with
>>> regard
>>> to MapFile creation and maintenance, since each data custodian
handles
>>> their
>>> respective MAPFILE.  The performance issues are minimized well since
>>> even
>>> the data intensive layers are not too bad from a performance
standpoint.
>>>
>>> But even my Googlish looking mapfile got prettty big (in my opinion)
for
>>> simply displaying centerlines of streets.   I've learned quite a bit
>>> from
>>> these exercises about these types of questions.  While I have yet to

>>> attack the performance side of things, I anticpate that I'll need to

>>> segregate the
>>> data out at differing thresholds in order to gain some performance
>>> boots.
>>> We're all about doing dynamic requests here since many of our
datasets
>>> change very frequently, in some cases, down to the minute.  I may
look
>>> into
>>> tiling at some point in the future, but it will still be only for
some

>>> of
>>> the layers, there will still be a need to have this dynamic request
>>> structure in place.
>>>
>>> bobb
>>>
>>>
>>>
>>>
>>>
>>>
>>> Bob Basques
>>> GIS Systems Developer
>>> City of Saint Paul, MN
>>>
>>>
>>> GISmo
>>> Powered by
>>> GeoMOOSE
>>>
>>>
>>>>>> riteshambastha < [hidden email]> wrote:
>>> Dear Readers,
>>>
>>> I have developed a map file with more than 35,000 Lines. Its size
>>> will grow by double/triple in next few months. Now, I am trying to
>>> tune my map file by
>>> removing unwanted lines. Still, I am bit confused about its
maintenance.

>>>
>>>
>>> Please throw some lights over writing map files by following best
>>> practices.
>>>
>>> Thanks in advance.
>>>
>>> Regards,
>>> Ritz
>>> --
>>> View this message in context:
>>> http://www.nabble.com/Map-file-with-35,000+-Lines...-management-issu
>>> e-tp15048892p15048892.html
>>>
>>> Sent from the Mapserver - User mailing list archive at Nabble.com.
>>>
>>
>>
>>
>
>

--
View this message in context:
e-tp15048892p15065742.html
Sent from the Mapserver - User mailing list archive at Nabble.com.



--
Ritesh Ambastha,

Project Manager
Mobiance Technologies,
Bangalore

+91-80-41264755