Questions on MapServer architecture and attribute conversions

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Questions on MapServer architecture and attribute conversions

Seth G-2
Hi devs,

I've created a pull request to add a __geo_interface__ to the Python MapScript PointObj, LineObj, and ShapeObjs.
This allows Python scripts to easily share MapScript objects with other Python geospatial libraries such as
Shapely, QGIS, and ArcPy. It also completes a nice cycle as Sean Gillies was a core developer of the MapScript Python bindings
and created __geo_interface__.  This only affects the Python MapScript bindings - do I need to create a RFC or ask for a vote to merge this for a 7.4 release?

I also have a few questions on the internal architecture of MapServer classes. Could other devs confirm (or correct) the following assumptions?

- the shapeObj has no link back to a layerObj - they are completely independent with no way to get a reference to a layer from a shape
- all shapeObj attribute values are stored internally as strings

To get attribute names for a shapeObj I had to get retrieve them  from the relevant layerObj and store these in a new property on the shapeObj in Python MapScript. I can then output something like the following:

    "properties": {
        "guid": "954BADBF-2891-48ED-A132-AED39C60E4C9",
        "featureid": "203529",
        "classid": "12"
    }

It would be nice to be able to convert the property values to their relevant types e.g. integers and floats. Looking through the
MapServer source they seem to only be converted to types using METADATA blocks and "gml_FIELD_NAME_type" "datatype" and in certain cases e.g. in mapogroutput.c
Are these conversions all handled differently for different data inputs? Is there an easy way to reuse these conversions in MapScript?

Thanks for any information on the above.
Regards,

Seth

--
web:http://geographika.co.uk
twitter: @geographika
_______________________________________________
mapserver-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/mapserver-dev
Reply | Threaded
Open this post in threaded view
|

Re: Questions on MapServer architecture and attribute conversions

Even Rouault-2
Seth,

>
> I've created a pull request to add a __geo_interface__ to the Python
> MapScript PointObj, LineObj, and ShapeObjs. This allows Python scripts to
> easily share MapScript objects with other Python geospatial libraries such
> as Shapely, QGIS, and ArcPy. It also completes a nice cycle as Sean Gillies
> was a core developer of the MapScript Python bindings and created
> __geo_interface__.  This only affects the Python MapScript bindings - do I
> need to create a RFC or ask for a vote to merge this for a 7.4 release?
>
> I also have a few questions on the internal architecture of MapServer
> classes. Could other devs confirm (or correct) the following assumptions?
>
> - the shapeObj has no link back to a layerObj - they are completely
> independent with no way to get a reference to a layer from a shape - all
> shapeObj attribute values are stored internally as strings

Yes that's my understanding too.

>
> To get attribute names for a shapeObj I had to get retrieve them  from the
> relevant layerObj and store these in a new property on the shapeObj in
> Python MapScript. I can then output something like the following:
>
>     "properties": {
>         "guid": "954BADBF-2891-48ED-A132-AED39C60E4C9",
>         "featureid": "203529",
>         "classid": "12"
>     }
>
> It would be nice to be able to convert the property values to their relevant
> types e.g. integers and floats. Looking through the MapServer source they
> seem to only be converted to types using METADATA blocks and
> "gml_FIELD_NAME_type" "datatype" and in certain cases e.g. in
> mapogroutput.c Are these conversions all handled differently for different
> data inputs? Is there an easy way to reuse these conversions in MapScript?

Yes this lack of proper typing on layer attributes is a bit inconvenient and
causes lot of adhoc coding in various areas of the code base: ogroutput, OGC
filtering, WFS, GML output, MVT output, etc.
The fundamental issue is that layerObj::items is just an array of field names.

For brainstorming... A potential solution could take inspiration from what is
done in GDAL where:
OGRLayer (~ equivalent of layerObj) -> contains a OGRFeatureDefn
OGRFeature (~ equivalent of shapeObj) -> points to a OGRFeatureDefn, and has
an array of field values

A OGRFeatureDefn contains (among other things) an array of OGRFieldDefn
A OGRFieldDefn is a set of attribute: field name, field type, etc...

Or if not doing a majorhaul, have indeed a pointer from shapeObj back to
layerObj (which interesting lifetime consideration, but I'm not sure MapScript
really shines on maintaining object ownership), and use information returned
by msGMLGetItems().


Even


--
Spatialys - Geospatial professional services
http://www.spatialys.com
_______________________________________________
mapserver-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/mapserver-dev
Reply | Threaded
Open this post in threaded view
|

Re: Questions on MapServer architecture and attribute conversions

Lime, Steve D (MNIT)
Couple of questions here...

On your pull request. I don't think there's a need for a RFC given the scope of the change. it's not breaking anything and seems like a nice addition - I certainly defer to your expertise as the Python MapScript maintainer. I guess it's never a bad idea to alert the -dev list and take your email as that notice - so +1 from me.

On your assumptions, both are correct.

On item types, it's funny this hasn't come up more over the years. Type conversions seem to happen either for output using the metadata referenced or in the expression handling based on context. You're correct the that MapScript, most of the time you'd get the column/item names from layer but not always since you can work with shapefiles directly and independently of a layerObj or mapObj.

Each layer has an attribute has something called iteminfo and it holds the index value associated with corresponding value in the items array. So you'd look up the item someone wanted for labeling purposes (or whatever) and would know what value to grab from a shapes values array. Those item index values are stored all over the place (e.g. labelitem and labelitemindex). I think with a scheme like Even describes items/iteminto could be replaced by one property (items) that would equate to an array of field definitions and then specific things like a labelitem becomes a struct consisting of a string and a pointer to one of the items. For some reason I thought Dan made a run at that at one point but I might be mistaken.

-----Original Message-----
From: mapserver-dev [mailto:[hidden email]] On Behalf Of Even Rouault
Sent: Thursday, March 14, 2019 11:39 AM
To: [hidden email]
Subject: Re: [mapserver-dev] Questions on MapServer architecture and attribute conversions

Seth,

>
> I've created a pull request to add a __geo_interface__ to the Python
> MapScript PointObj, LineObj, and ShapeObjs. This allows Python scripts to
> easily share MapScript objects with other Python geospatial libraries such
> as Shapely, QGIS, and ArcPy. It also completes a nice cycle as Sean Gillies
> was a core developer of the MapScript Python bindings and created
> __geo_interface__.  This only affects the Python MapScript bindings - do I
> need to create a RFC or ask for a vote to merge this for a 7.4 release?
>
> I also have a few questions on the internal architecture of MapServer
> classes. Could other devs confirm (or correct) the following assumptions?
>
> - the shapeObj has no link back to a layerObj - they are completely
> independent with no way to get a reference to a layer from a shape - all
> shapeObj attribute values are stored internally as strings

Yes that's my understanding too.

>
> To get attribute names for a shapeObj I had to get retrieve them  from the
> relevant layerObj and store these in a new property on the shapeObj in
> Python MapScript. I can then output something like the following:
>
>     "properties": {
>         "guid": "954BADBF-2891-48ED-A132-AED39C60E4C9",
>         "featureid": "203529",
>         "classid": "12"
>     }
>
> It would be nice to be able to convert the property values to their relevant
> types e.g. integers and floats. Looking through the MapServer source they
> seem to only be converted to types using METADATA blocks and
> "gml_FIELD_NAME_type" "datatype" and in certain cases e.g. in
> mapogroutput.c Are these conversions all handled differently for different
> data inputs? Is there an easy way to reuse these conversions in MapScript?

Yes this lack of proper typing on layer attributes is a bit inconvenient and
causes lot of adhoc coding in various areas of the code base: ogroutput, OGC
filtering, WFS, GML output, MVT output, etc.
The fundamental issue is that layerObj::items is just an array of field names.

For brainstorming... A potential solution could take inspiration from what is
done in GDAL where:
OGRLayer (~ equivalent of layerObj) -> contains a OGRFeatureDefn
OGRFeature (~ equivalent of shapeObj) -> points to a OGRFeatureDefn, and has
an array of field values

A OGRFeatureDefn contains (among other things) an array of OGRFieldDefn
A OGRFieldDefn is a set of attribute: field name, field type, etc...

Or if not doing a majorhaul, have indeed a pointer from shapeObj back to
layerObj (which interesting lifetime consideration, but I'm not sure MapScript
really shines on maintaining object ownership), and use information returned
by msGMLGetItems().


Even


--
Spatialys - Geospatial professional services
https://gcc01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.spatialys.com&data=02%7C01%7Csteve.lime%40state.mn.us%7Cbc8c4af6794f4e52259608d6a89c923d%7Ceb14b04624c445198f26b89c2159828c%7C0%7C0%7C636881787755412437&sdata=IPFoO8gWTM72w7ROYMPbYcLwGyWSEPkGga0rllSx%2FhI%3D&reserved=0
_______________________________________________
mapserver-dev mailing list
[hidden email]
https://gcc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.osgeo.org%2Fmailman%2Flistinfo%2Fmapserver-dev&data=02%7C01%7Csteve.lime%40state.mn.us%7Cbc8c4af6794f4e52259608d6a89c923d%7Ceb14b04624c445198f26b89c2159828c%7C0%7C0%7C636881787755412437&sdata=hYFnDNHUyhJF1VRBQXLYXKwaBjCD4q3G9QL%2B346FOZw%3D&reserved=0
_______________________________________________
mapserver-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/mapserver-dev
Reply | Threaded
Open this post in threaded view
|

Re: Questions on MapServer architecture and attribute conversions

Seth G-2
Hi Steve, Even,

Thanks for your replies. Thinking back I  think I've run into the issue of strings being used for int fields from a database, and CLASS expressions.

I may make an attempt at the following based on both of your comments:

- add gmlItemObj (and also gmlItemListObj) as classes to MapScript
- allow msGMLGetItems to be called on a layerObj to populate this list
- allow this list to be associated/passed to a method on a shapeObj, and use the GML types to set the output attribute types
- if there are no field definitions supplied then attribute names will simply be indexes, and all values strings

Seth
--
web:http://geographika.co.uk
twitter: @geographika

On Thu, Mar 14, 2019, at 10:14 PM, Lime, Steve D (MNIT) wrote:

> Couple of questions here...
>
> On your pull request. I don't think there's a need for a RFC given the
> scope of the change. it's not breaking anything and seems like a nice
> addition - I certainly defer to your expertise as the Python MapScript
> maintainer. I guess it's never a bad idea to alert the -dev list and
> take your email as that notice - so +1 from me.
>
> On your assumptions, both are correct.
>
> On item types, it's funny this hasn't come up more over the years. Type
> conversions seem to happen either for output using the metadata
> referenced or in the expression handling based on context. You're
> correct the that MapScript, most of the time you'd get the column/item
> names from layer but not always since you can work with shapefiles
> directly and independently of a layerObj or mapObj.
>
> Each layer has an attribute has something called iteminfo and it holds
> the index value associated with corresponding value in the items array.
> So you'd look up the item someone wanted for labeling purposes (or
> whatever) and would know what value to grab from a shapes values array.
> Those item index values are stored all over the place (e.g. labelitem
> and labelitemindex). I think with a scheme like Even describes
> items/iteminto could be replaced by one property (items) that would
> equate to an array of field definitions and then specific things like a
> labelitem becomes a struct consisting of a string and a pointer to one
> of the items. For some reason I thought Dan made a run at that at one
> point but I might be mistaken.
>
> -----Original Message-----
> From: mapserver-dev [mailto:[hidden email]] On
> Behalf Of Even Rouault
> Sent: Thursday, March 14, 2019 11:39 AM
> To: [hidden email]
> Subject: Re: [mapserver-dev] Questions on MapServer architecture and
> attribute conversions
>
> Seth,
>
> >
> > I've created a pull request to add a __geo_interface__ to the Python
> > MapScript PointObj, LineObj, and ShapeObjs. This allows Python scripts to
> > easily share MapScript objects with other Python geospatial libraries such
> > as Shapely, QGIS, and ArcPy. It also completes a nice cycle as Sean Gillies
> > was a core developer of the MapScript Python bindings and created
> > __geo_interface__.  This only affects the Python MapScript bindings - do I
> > need to create a RFC or ask for a vote to merge this for a 7.4 release?
> >
> > I also have a few questions on the internal architecture of MapServer
> > classes. Could other devs confirm (or correct) the following assumptions?
> >
> > - the shapeObj has no link back to a layerObj - they are completely
> > independent with no way to get a reference to a layer from a shape - all
> > shapeObj attribute values are stored internally as strings
>
> Yes that's my understanding too.
>
> >
> > To get attribute names for a shapeObj I had to get retrieve them  from the
> > relevant layerObj and store these in a new property on the shapeObj in
> > Python MapScript. I can then output something like the following:
> >
> >     "properties": {
> >         "guid": "954BADBF-2891-48ED-A132-AED39C60E4C9",
> >         "featureid": "203529",
> >         "classid": "12"
> >     }
> >
> > It would be nice to be able to convert the property values to their relevant
> > types e.g. integers and floats. Looking through the MapServer source they
> > seem to only be converted to types using METADATA blocks and
> > "gml_FIELD_NAME_type" "datatype" and in certain cases e.g. in
> > mapogroutput.c Are these conversions all handled differently for different
> > data inputs? Is there an easy way to reuse these conversions in MapScript?
>
> Yes this lack of proper typing on layer attributes is a bit inconvenient and
> causes lot of adhoc coding in various areas of the code base: ogroutput, OGC
> filtering, WFS, GML output, MVT output, etc.
> The fundamental issue is that layerObj::items is just an array of field names.
>
> For brainstorming... A potential solution could take inspiration from what is
> done in GDAL where:
> OGRLayer (~ equivalent of layerObj) -> contains a OGRFeatureDefn
> OGRFeature (~ equivalent of shapeObj) -> points to a OGRFeatureDefn, and has
> an array of field values
>
> A OGRFeatureDefn contains (among other things) an array of OGRFieldDefn
> A OGRFieldDefn is a set of attribute: field name, field type, etc...
>
> Or if not doing a majorhaul, have indeed a pointer from shapeObj back to
> layerObj (which interesting lifetime consideration, but I'm not sure MapScript
> really shines on maintaining object ownership), and use information returned
> by msGMLGetItems().
>
>
> Even
>
>
> --
> Spatialys - Geospatial professional services
> https://gcc01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.spatialys.com&data=02%7C01%7Csteve.lime%40state.mn.us%7Cbc8c4af6794f4e52259608d6a89c923d%7Ceb14b04624c445198f26b89c2159828c%7C0%7C0%7C636881787755412437&sdata=IPFoO8gWTM72w7ROYMPbYcLwGyWSEPkGga0rllSx%2FhI%3D&reserved=0
> _______________________________________________
> mapserver-dev mailing list
> [hidden email]
> https://gcc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.osgeo.org%2Fmailman%2Flistinfo%2Fmapserver-dev&data=02%7C01%7Csteve.lime%40state.mn.us%7Cbc8c4af6794f4e52259608d6a89c923d%7Ceb14b04624c445198f26b89c2159828c%7C0%7C0%7C636881787755412437&sdata=hYFnDNHUyhJF1VRBQXLYXKwaBjCD4q3G9QL%2B346FOZw%3D&reserved=0
> _______________________________________________
> mapserver-dev mailing list
> [hidden email]
> https://lists.osgeo.org/mailman/listinfo/mapserver-dev
_______________________________________________
mapserver-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/mapserver-dev
Reply | Threaded
Open this post in threaded view
|

Re: Questions on MapServer architecture and attribute conversions

Lime, Steve D (MNIT)
I wish we could make it less GMLish and more generic. For example, it feels odd in MVT code to be relying on what feels like format-specific metadata. The various service specs can use a generic “ows” or service specific metadata namespaces. Could do the same for output... “out” as the generic namespace with “gml”, “mvt”, “ogr”, “kml” as specific namespaces.

gmlItemObj and gmlItemListObj could also be renamed more generically and the latter could become iteminfo as part of a layer. A cast function for type conversations would also be helpful for more standard use.

Just thinking out loud - I know that’s way more than you want to get into,

—Steve


From: mapserver-dev <[hidden email]> on behalf of Seth G <[hidden email]>
Sent: Friday, March 15, 2019 6:58:41 PM
To: [hidden email]
Subject: Re: [mapserver-dev] Questions on MapServer architecture and attribute conversions
 
Hi Steve, Even,

Thanks for your replies. Thinking back I  think I've run into the issue of strings being used for int fields from a database, and CLASS expressions.

I may make an attempt at the following based on both of your comments:

- add gmlItemObj (and also gmlItemListObj) as classes to MapScript
- allow msGMLGetItems to be called on a layerObj to populate this list
- allow this list to be associated/passed to a method on a shapeObj, and use the GML types to set the output attribute types
- if there are no field definitions supplied then attribute names will simply be indexes, and all values strings

Seth
--
web:http://geographika.co.uk
twitter: @geographika

On Thu, Mar 14, 2019, at 10:14 PM, Lime, Steve D (MNIT) wrote:
> Couple of questions here...
>
> On your pull request. I don't think there's a need for a RFC given the
> scope of the change. it's not breaking anything and seems like a nice
> addition - I certainly defer to your expertise as the Python MapScript
> maintainer. I guess it's never a bad idea to alert the -dev list and
> take your email as that notice - so +1 from me.
>
> On your assumptions, both are correct.
>
> On item types, it's funny this hasn't come up more over the years. Type
> conversions seem to happen either for output using the metadata
> referenced or in the expression handling based on context. You're
> correct the that MapScript, most of the time you'd get the column/item
> names from layer but not always since you can work with shapefiles
> directly and independently of a layerObj or mapObj.
>
> Each layer has an attribute has something called iteminfo and it holds
> the index value associated with corresponding value in the items array.
> So you'd look up the item someone wanted for labeling purposes (or
> whatever) and would know what value to grab from a shapes values array.
> Those item index values are stored all over the place (e.g. labelitem
> and labelitemindex). I think with a scheme like Even describes
> items/iteminto could be replaced by one property (items) that would
> equate to an array of field definitions and then specific things like a
> labelitem becomes a struct consisting of a string and a pointer to one
> of the items. For some reason I thought Dan made a run at that at one
> point but I might be mistaken.
>
> -----Original Message-----
> From: mapserver-dev [[hidden email]] On
> Behalf Of Even Rouault
> Sent: Thursday, March 14, 2019 11:39 AM
> To: [hidden email]
> Subject: Re: [mapserver-dev] Questions on MapServer architecture and
> attribute conversions
>
> Seth,
>
> >
> > I've created a pull request to add a __geo_interface__ to the Python
> > MapScript PointObj, LineObj, and ShapeObjs. This allows Python scripts to
> > easily share MapScript objects with other Python geospatial libraries such
> > as Shapely, QGIS, and ArcPy. It also completes a nice cycle as Sean Gillies
> > was a core developer of the MapScript Python bindings and created
> > __geo_interface__.  This only affects the Python MapScript bindings - do I
> > need to create a RFC or ask for a vote to merge this for a 7.4 release?
> >
> > I also have a few questions on the internal architecture of MapServer
> > classes. Could other devs confirm (or correct) the following assumptions?
> >
> > - the shapeObj has no link back to a layerObj - they are completely
> > independent with no way to get a reference to a layer from a shape - all
> > shapeObj attribute values are stored internally as strings
>
> Yes that's my understanding too.
>
> >
> > To get attribute names for a shapeObj I had to get retrieve them  from the
> > relevant layerObj and store these in a new property on the shapeObj in
> > Python MapScript. I can then output something like the following:
> >
> >     "properties": {
> >         "guid": "954BADBF-2891-48ED-A132-AED39C60E4C9",
> >         "featureid": "203529",
> >         "classid": "12"
> >     }
> >
> > It would be nice to be able to convert the property values to their relevant
> > types e.g. integers and floats. Looking through the MapServer source they
> > seem to only be converted to types using METADATA blocks and
> > "gml_FIELD_NAME_type" "datatype" and in certain cases e.g. in
> > mapogroutput.c Are these conversions all handled differently for different
> > data inputs? Is there an easy way to reuse these conversions in MapScript?
>
> Yes this lack of proper typing on layer attributes is a bit inconvenient and
> causes lot of adhoc coding in various areas of the code base: ogroutput, OGC
> filtering, WFS, GML output, MVT output, etc.
> The fundamental issue is that layerObj::items is just an array of field names.
>
> For brainstorming... A potential solution could take inspiration from what is
> done in GDAL where:
> OGRLayer (~ equivalent of layerObj) -> contains a OGRFeatureDefn
> OGRFeature (~ equivalent of shapeObj) -> points to a OGRFeatureDefn, and has
> an array of field values
>
> A OGRFeatureDefn contains (among other things) an array of OGRFieldDefn
> A OGRFieldDefn is a set of attribute: field name, field type, etc...
>
> Or if not doing a majorhaul, have indeed a pointer from shapeObj back to
> layerObj (which interesting lifetime consideration, but I'm not sure MapScript
> really shines on maintaining object ownership), and use information returned
> by msGMLGetItems().
>
>
> Even
>
>
> --
> Spatialys - Geospatial professional services
> https://gcc01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.spatialys.com&amp;data=02%7C01%7Csteve.lime%40state.mn.us%7C8af807e4be164b6cb28608d6a9a229ad%7Ceb14b04624c445198f26b89c2159828c%7C0%7C0%7C636882911288492978&amp;sdata=gq%2FwxERwIC7hIsUULQuBdz5NuWGPSjEEejynXTrU%2BWc%3D&amp;reserved=0
> _______________________________________________
> mapserver-dev mailing list
> [hidden email]
> https://gcc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.osgeo.org%2Fmailman%2Flistinfo%2Fmapserver-dev&amp;data=02%7C01%7Csteve.lime%40state.mn.us%7C8af807e4be164b6cb28608d6a9a229ad%7Ceb14b04624c445198f26b89c2159828c%7C0%7C0%7C636882911288492978&amp;sdata=K6js4DgRQJ4giCNpYqJ3oROAb7HQg3W6xHJaAbXgUxI%3D&amp;reserved=0
> _______________________________________________
> mapserver-dev mailing list
> [hidden email]
> https://gcc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.osgeo.org%2Fmailman%2Flistinfo%2Fmapserver-dev&amp;data=02%7C01%7Csteve.lime%40state.mn.us%7C8af807e4be164b6cb28608d6a9a229ad%7Ceb14b04624c445198f26b89c2159828c%7C0%7C1%7C636882911288502991&amp;sdata=48CiR%2B8XMq030GchIzqmpAdse4sgV1b9Fvjk8Jzt%2Fq8%3D&amp;reserved=0
_______________________________________________
mapserver-dev mailing list
[hidden email]
https://gcc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.osgeo.org%2Fmailman%2Flistinfo%2Fmapserver-dev&amp;data=02%7C01%7Csteve.lime%40state.mn.us%7C8af807e4be164b6cb28608d6a9a229ad%7Ceb14b04624c445198f26b89c2159828c%7C0%7C1%7C636882911288502991&amp;sdata=48CiR%2B8XMq030GchIzqmpAdse4sgV1b9Fvjk8Jzt%2Fq8%3D&amp;reserved=0

_______________________________________________
mapserver-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/mapserver-dev
Reply | Threaded
Open this post in threaded view
|

Re: Questions on MapServer architecture and attribute conversions

Seth G-2
Hi,

I implemented a simple addition to MapScript to retrieve the gmlItemObj for a layer item (field) and added to the pull request.
This currently works for the Python bindings but is throwing an error for C# I'll look into.

To use gmlItemObj I had to include mapows.h in mapscript.i. When I initially did this there were 28 errors similar to below:

mapscriptPYTHON_wrap.obj : error LNK2019: unresolved external symbol msWMSDispatch referenced in function _wrap_msWMSDispatch

The relevant commit is at: https://github.com/mapserver/mapserver/pull/5765/commits/0260296eecfb8b9bad979fc20ba5f9034ca37ffb (Ill see what happened to the whitespace in this commit and revert with just the relevant changes).

I simply added the macro MS_DLL_EXPORT to export all these functions and was able to build and test the MapScript bindings.
All the other header files included in SWIG have #ifndef SWIG guards everywhere. Is anyone able to explain to me the purpose of these and provide examples of where these should be added to mapows.h?

Seth

--
web:http://geographika.co.uk
twitter: @geographika


On Sat, Mar 16, 2019, at 11:58 AM, Lime, Steve D (MNIT) wrote:
I wish we could make it less GMLish and more generic. For example, it feels odd in MVT code to be relying on what feels like format-specific metadata. The various service specs can use a generic “ows” or service specific metadata namespaces. Could do the same for output... “out” as the generic namespace with “gml”, “mvt”, “ogr”, “kml” as specific namespaces.

gmlItemObj and gmlItemListObj could also be renamed more generically and the latter could become iteminfo as part of a layer. A cast function for type conversations would also be helpful for more standard use.

Just thinking out loud - I know that’s way more than you want to get into,

—Steve



From: mapserver-dev <[hidden email]> on behalf of Seth G <[hidden email]>
Sent: Friday, March 15, 2019 6:58:41 PM
To: [hidden email]
Subject: Re: [mapserver-dev] Questions on MapServer architecture and attribute conversions
 
Hi Steve, Even,

Thanks for your replies. Thinking back I  think I've run into the issue of strings being used for int fields from a database, and CLASS expressions.

I may make an attempt at the following based on both of your comments:

- add gmlItemObj (and also gmlItemListObj) as classes to MapScript
- allow msGMLGetItems to be called on a layerObj to populate this list
- allow this list to be associated/passed to a method on a shapeObj, and use the GML types to set the output attribute types
- if there are no field definitions supplied then attribute names will simply be indexes, and all values strings

Seth
--
web:http://geographika.co.uk
twitter: @geographika

On Thu, Mar 14, 2019, at 10:14 PM, Lime, Steve D (MNIT) wrote:
> Couple of questions here...
>
> On your pull request. I don't think there's a need for a RFC given the
> scope of the change. it's not breaking anything and seems like a nice
> addition - I certainly defer to your expertise as the Python MapScript
> maintainer. I guess it's never a bad idea to alert the -dev list and
> take your email as that notice - so +1 from me.
>
> On your assumptions, both are correct.
>
> On item types, it's funny this hasn't come up more over the years. Type
> conversions seem to happen either for output using the metadata
> referenced or in the expression handling based on context. You're
> correct the that MapScript, most of the time you'd get the column/item
> names from layer but not always since you can work with shapefiles
> directly and independently of a layerObj or mapObj.
>
> Each layer has an attribute has something called iteminfo and it holds
> the index value associated with corresponding value in the items array.
> So you'd look up the item someone wanted for labeling purposes (or
> whatever) and would know what value to grab from a shapes values array.
> Those item index values are stored all over the place (e.g. labelitem
> and labelitemindex). I think with a scheme like Even describes
> items/iteminto could be replaced by one property (items) that would
> equate to an array of field definitions and then specific things like a
> labelitem becomes a struct consisting of a string and a pointer to one
> of the items. For some reason I thought Dan made a run at that at one
> point but I might be mistaken.
>
> -----Original Message-----
> From: mapserver-dev [[hidden email]] On
> Behalf Of Even Rouault
> Sent: Thursday, March 14, 2019 11:39 AM
> Subject: Re: [mapserver-dev] Questions on MapServer architecture and
> attribute conversions
>
> Seth,
>
> >
> > I've created a pull request to add a __geo_interface__ to the Python
> > MapScript PointObj, LineObj, and ShapeObjs. This allows Python scripts to
> > easily share MapScript objects with other Python geospatial libraries such
> > as Shapely, QGIS, and ArcPy. It also completes a nice cycle as Sean Gillies
> > was a core developer of the MapScript Python bindings and created
> > __geo_interface__.  This only affects the Python MapScript bindings - do I
> > need to create a RFC or ask for a vote to merge this for a 7.4 release?
> >
> > I also have a few questions on the internal architecture of MapServer
> > classes. Could other devs confirm (or correct) the following assumptions?
> >
> > - the shapeObj has no link back to a layerObj - they are completely
> > independent with no way to get a reference to a layer from a shape - all
> > shapeObj attribute values are stored internally as strings
>
> Yes that's my understanding too.
>
> >
> > To get attribute names for a shapeObj I had to get retrieve them  from the
> > relevant layerObj and store these in a new property on the shapeObj in
> > Python MapScript. I can then output something like the following:
> >
> >     "properties": {
> >         "guid": "954BADBF-2891-48ED-A132-AED39C60E4C9",
> >         "featureid": "203529",
> >         "classid": "12"
> >     }
> >
> > It would be nice to be able to convert the property values to their relevant
> > types e.g. integers and floats. Looking through the MapServer source they
> > seem to only be converted to types using METADATA blocks and
> > "gml_FIELD_NAME_type" "datatype" and in certain cases e.g. in
> > mapogroutput.c Are these conversions all handled differently for different
> > data inputs? Is there an easy way to reuse these conversions in MapScript?
>
> Yes this lack of proper typing on layer attributes is a bit inconvenient and
> causes lot of adhoc coding in various areas of the code base: ogroutput, OGC
> filtering, WFS, GML output, MVT output, etc.
> The fundamental issue is that layerObj::items is just an array of field names.
>
> For brainstorming... A potential solution could take inspiration from what is
> done in GDAL where:
> OGRLayer (~ equivalent of layerObj) -> contains a OGRFeatureDefn
> OGRFeature (~ equivalent of shapeObj) -> points to a OGRFeatureDefn, and has
> an array of field values
>
> A OGRFeatureDefn contains (among other things) an array of OGRFieldDefn
> A OGRFieldDefn is a set of attribute: field name, field type, etc...
>
> Or if not doing a majorhaul, have indeed a pointer from shapeObj back to
> layerObj (which interesting lifetime consideration, but I'm not sure MapScript
> really shines on maintaining object ownership), and use information returned
> by msGMLGetItems().
>
>
> Even
>
>
> --
> Spatialys - Geospatial professional services
> _______________________________________________
> mapserver-dev mailing list
> _______________________________________________
> mapserver-dev mailing list
_______________________________________________
mapserver-dev mailing list


_______________________________________________
mapserver-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/mapserver-dev
Reply | Threaded
Open this post in threaded view
|

Re: Questions on MapServer architecture and attribute conversions

Seth G-2
Hello again,

After reading through various header files again and the SWIG docs I think I understand better how it fits together.

#ifndef SWIG..#endif guards are added throughout the header file so when it is included via %include "../../mapows.h" only the selected functions are exposed.

I'm not sure if there is a standard way of adding the guards - it seems to be added for individual structs and groups of functions throughout the header file so I tried to fit with this convention - it does make for a large diff though to review.

Now only gmlItemObj is exposed (as an immutable object), and gmlItemListObj (although this will only be available when USE_WMS_SVR and USE_WFS_SVR are defined.

Exposing only the relevant parts of the header file fixes the C# issue.

Seth


--
web:http://geographika.co.uk
twitter: @geographika


On Sun, Mar 17, 2019, at 12:48 AM, Seth G wrote:
Hi,

I implemented a simple addition to MapScript to retrieve the gmlItemObj for a layer item (field) and added to the pull request.
This currently works for the Python bindings but is throwing an error for C# I'll look into.

To use gmlItemObj I had to include mapows.h in mapscript.i. When I initially did this there were 28 errors similar to below:

mapscriptPYTHON_wrap.obj : error LNK2019: unresolved external symbol msWMSDispatch referenced in function _wrap_msWMSDispatch

The relevant commit is at: https://github.com/mapserver/mapserver/pull/5765/commits/0260296eecfb8b9bad979fc20ba5f9034ca37ffb (Ill see what happened to the whitespace in this commit and revert with just the relevant changes).

I simply added the macro MS_DLL_EXPORT to export all these functions and was able to build and test the MapScript bindings.
All the other header files included in SWIG have #ifndef SWIG guards everywhere. Is anyone able to explain to me the purpose of these and provide examples of where these should be added to mapows.h?

Seth

--
web:http://geographika.co.uk
twitter: @geographika


On Sat, Mar 16, 2019, at 11:58 AM, Lime, Steve D (MNIT) wrote:
I wish we could make it less GMLish and more generic. For example, it feels odd in MVT code to be relying on what feels like format-specific metadata. The various service specs can use a generic “ows” or service specific metadata namespaces. Could do the same for output... “out” as the generic namespace with “gml”, “mvt”, “ogr”, “kml” as specific namespaces.

gmlItemObj and gmlItemListObj could also be renamed more generically and the latter could become iteminfo as part of a layer. A cast function for type conversations would also be helpful for more standard use.

Just thinking out loud - I know that’s way more than you want to get into,

—Steve



From: mapserver-dev <[hidden email]> on behalf of Seth G <[hidden email]>
Sent: Friday, March 15, 2019 6:58:41 PM
To: [hidden email]
Subject: Re: [mapserver-dev] Questions on MapServer architecture and attribute conversions
 
Hi Steve, Even,

Thanks for your replies. Thinking back I  think I've run into the issue of strings being used for int fields from a database, and CLASS expressions.

I may make an attempt at the following based on both of your comments:

- add gmlItemObj (and also gmlItemListObj) as classes to MapScript
- allow msGMLGetItems to be called on a layerObj to populate this list
- allow this list to be associated/passed to a method on a shapeObj, and use the GML types to set the output attribute types
- if there are no field definitions supplied then attribute names will simply be indexes, and all values strings

Seth
--
web:http://geographika.co.uk
twitter: @geographika

On Thu, Mar 14, 2019, at 10:14 PM, Lime, Steve D (MNIT) wrote:
> Couple of questions here...
>
> On your pull request. I don't think there's a need for a RFC given the
> scope of the change. it's not breaking anything and seems like a nice
> addition - I certainly defer to your expertise as the Python MapScript
> maintainer. I guess it's never a bad idea to alert the -dev list and
> take your email as that notice - so +1 from me.
>
> On your assumptions, both are correct.
>
> On item types, it's funny this hasn't come up more over the years. Type
> conversions seem to happen either for output using the metadata
> referenced or in the expression handling based on context. You're
> correct the that MapScript, most of the time you'd get the column/item
> names from layer but not always since you can work with shapefiles
> directly and independently of a layerObj or mapObj.
>
> Each layer has an attribute has something called iteminfo and it holds
> the index value associated with corresponding value in the items array.
> So you'd look up the item someone wanted for labeling purposes (or
> whatever) and would know what value to grab from a shapes values array.
> Those item index values are stored all over the place (e.g. labelitem
> and labelitemindex). I think with a scheme like Even describes
> items/iteminto could be replaced by one property (items) that would
> equate to an array of field definitions and then specific things like a
> labelitem becomes a struct consisting of a string and a pointer to one
> of the items. For some reason I thought Dan made a run at that at one
> point but I might be mistaken.
>
> -----Original Message-----
> From: mapserver-dev [[hidden email]] On
> Behalf Of Even Rouault
> Sent: Thursday, March 14, 2019 11:39 AM
> Subject: Re: [mapserver-dev] Questions on MapServer architecture and
> attribute conversions
>
> Seth,
>
> >
> > I've created a pull request to add a __geo_interface__ to the Python
> > MapScript PointObj, LineObj, and ShapeObjs. This allows Python scripts to
> > easily share MapScript objects with other Python geospatial libraries such
> > as Shapely, QGIS, and ArcPy. It also completes a nice cycle as Sean Gillies
> > was a core developer of the MapScript Python bindings and created
> > __geo_interface__.  This only affects the Python MapScript bindings - do I
> > need to create a RFC or ask for a vote to merge this for a 7.4 release?
> >
> > I also have a few questions on the internal architecture of MapServer
> > classes. Could other devs confirm (or correct) the following assumptions?
> >
> > - the shapeObj has no link back to a layerObj - they are completely
> > independent with no way to get a reference to a layer from a shape - all
> > shapeObj attribute values are stored internally as strings
>
> Yes that's my understanding too.
>
> >
> > To get attribute names for a shapeObj I had to get retrieve them  from the
> > relevant layerObj and store these in a new property on the shapeObj in
> > Python MapScript. I can then output something like the following:
> >
> >     "properties": {
> >         "guid": "954BADBF-2891-48ED-A132-AED39C60E4C9",
> >         "featureid": "203529",
> >         "classid": "12"
> >     }
> >
> > It would be nice to be able to convert the property values to their relevant
> > types e.g. integers and floats. Looking through the MapServer source they
> > seem to only be converted to types using METADATA blocks and
> > "gml_FIELD_NAME_type" "datatype" and in certain cases e.g. in
> > mapogroutput.c Are these conversions all handled differently for different
> > data inputs? Is there an easy way to reuse these conversions in MapScript?
>
> Yes this lack of proper typing on layer attributes is a bit inconvenient and
> causes lot of adhoc coding in various areas of the code base: ogroutput, OGC
> filtering, WFS, GML output, MVT output, etc.
> The fundamental issue is that layerObj::items is just an array of field names.
>
> For brainstorming... A potential solution could take inspiration from what is
> done in GDAL where:
> OGRLayer (~ equivalent of layerObj) -> contains a OGRFeatureDefn
> OGRFeature (~ equivalent of shapeObj) -> points to a OGRFeatureDefn, and has
> an array of field values
>
> A OGRFeatureDefn contains (among other things) an array of OGRFieldDefn
> A OGRFieldDefn is a set of attribute: field name, field type, etc...
>
> Or if not doing a majorhaul, have indeed a pointer from shapeObj back to
> layerObj (which interesting lifetime consideration, but I'm not sure MapScript
> really shines on maintaining object ownership), and use information returned
> by msGMLGetItems().
>
>
> Even
>
>
> --
> Spatialys - Geospatial professional services
> _______________________________________________
> mapserver-dev mailing list
> _______________________________________________
> mapserver-dev mailing list
_______________________________________________
mapserver-dev mailing list

_______________________________________________
mapserver-dev mailing list
https://lists.osgeo.org/mailman/listinfo/mapserver-dev


_______________________________________________
mapserver-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/mapserver-dev