Regression test issue: NULLs in shapefiles...

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Regression test issue: NULLs in shapefiles...

Lime, Steve D (MNIT)

Hi all: I’m trying to debug a failing CITE test (wfs_200_cite_filter_like.xml) for RFC 91 that uses PropertyIsLike. In master that resolves to “item like ‘%’”  for OGR and other databases. For a shapefile it resolves to “item ~ ‘.*’”. In the test shapefile data there are a bunch of null values for the item being checked. The test in question uses the OGR driver. Here’s a quick summary:

 

Master:

-          Using OGR driver gets expression WHERE island_e like “%” and returns 3 features, ignoring nulls

-          Using shapefile driver gets expression “[island_e]” ~ “.*” and returns all features

 

RFC 91:

-          Using OGR driver gets expression  “[island_e]” ~ “.*” and returns all features

-          Using shapefile driver gets expression “[island_e]” ~ “.*” and returns all features

 

This change is because RFC 91 translation runs through the MapServer expression syntax and the OGR driver can handle both cases. However, that’s not the problem since you shouldn’t see the difference in master either – a straight shapefile fails. I’m guessing that way down in the shapefile driver nulls in the DBF are handled as empty string which technically match the “0 or more character” pattern. Any ideas on how to deal with something like this? Changing the regex to “.+” makes the test pass but I don’t think that’s the correct solution.

 

Steve

 

StEVE LIME  | DATA & Applications MANAGER

MN.IT Services @ MnDNR

651-259-5473 (w)  |  651-297-4946 (f) |  [hidden email]

Information Technology for Minnesota Government   |   mn.gov/oet

 

 


_______________________________________________
mapserver-dev mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/mapserver-dev
Reply | Threaded
Open this post in threaded view
|

Re: Regression test issue: NULLs in shapefiles...

Even Rouault-2
Le lundi 25 août 2014 17:42:26, Lime, Steve D (MNIT) a écrit :

> Hi all: I'm trying to debug a failing CITE test
> (wfs_200_cite_filter_like.xml) for RFC 91 that uses PropertyIsLike. In
> master that resolves to "item like '%'"  for OGR and other databases. For
> a shapefile it resolves to "item ~ '.*'". In the test shapefile data there
> are a bunch of null values for the item being checked. The test in
> question uses the OGR driver. Here's a quick summary:
>
> Master:
>
> -          Using OGR driver gets expression WHERE island_e like "%" and
> returns 3 features, ignoring nulls
>
> -          Using shapefile driver gets expression "[island_e]" ~ ".*" and
> returns all features
>
> RFC 91:
>
> -          Using OGR driver gets expression  "[island_e]" ~ ".*" and
> returns all features
>
> -          Using shapefile driver gets expression "[island_e]" ~ ".*" and
> returns all features
>
> This change is because RFC 91 translation runs through the MapServer
> expression syntax and the OGR driver can handle both cases. However,
> that's not the problem since you shouldn't see the difference in master
> either - a straight shapefile fails. I'm guessing that way down in the
> shapefile driver nulls in the DBF are handled as empty string which
> technically match the "0 or more character" pattern. Any ideas on how to
> deal with something like this?

In OGR, empty strings in .DBF are consider as NULL values, and not empty
strings. I'm not sure this is a universal choice, because DBF has no proper
way of specifying the NULL concept.
And I don't think there's this concept of NULL currently in MapServer
attributes (the lack of typing is also quite limiting sometimes). The start
would be to change msDBFReadAttribute() to return NULL if DBFIsValueNULL().
But that would likely cause many problems in the rest of the code base....
Or a more restricted change when a LIKE is evaluated to consider that empty
string is NULL and then evaluate it to FALSE.

> Changing the regex to ".+" makes the test
> pass but I don't think that's the correct solution.
>
> Steve
>
> StEVE LIME  | DATA & Applications MANAGER
> MN.IT Services @ MnDNR
> 651-259-5473 (w)  |  651-297-4946 (f) |
> [hidden email]<mailto:[hidden email]>
>
> [cid:image003.jpg@01CFC051.43750460]<http://www.mn.gov/oet>
>
> Information Technology for Minnesota Government   |  
> mn.gov/oet<http://www.mn.gov/oet>

--
Spatialys - Geospatial professional services
http://www.spatialys.com
_______________________________________________
mapserver-dev mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/mapserver-dev
Reply | Threaded
Open this post in threaded view
|

Re: Regression test issue: NULLs in shapefiles...

Lime, Steve D (MNIT)
I think your last suggestion is a good one - it's a pretty localized change to msEvalExpression() for regex expressions and the logical expression parser (mapparser.y).

Steve

-----Original Message-----
From: Even Rouault [mailto:[hidden email]]
Sent: Monday, August 25, 2014 1:01 PM
To: [hidden email]
Cc: Lime, Steve D (MNIT)
Subject: Re: [mapserver-dev] Regression test issue: NULLs in shapefiles...

Le lundi 25 août 2014 17:42:26, Lime, Steve D (MNIT) a écrit :

> Hi all: I'm trying to debug a failing CITE test
> (wfs_200_cite_filter_like.xml) for RFC 91 that uses PropertyIsLike. In
> master that resolves to "item like '%'"  for OGR and other databases.
> For a shapefile it resolves to "item ~ '.*'". In the test shapefile
> data there are a bunch of null values for the item being checked. The
> test in question uses the OGR driver. Here's a quick summary:
>
> Master:
>
> -          Using OGR driver gets expression WHERE island_e like "%" and
> returns 3 features, ignoring nulls
>
> -          Using shapefile driver gets expression "[island_e]" ~ ".*" and
> returns all features
>
> RFC 91:
>
> -          Using OGR driver gets expression  "[island_e]" ~ ".*" and
> returns all features
>
> -          Using shapefile driver gets expression "[island_e]" ~ ".*" and
> returns all features
>
> This change is because RFC 91 translation runs through the MapServer
> expression syntax and the OGR driver can handle both cases. However,
> that's not the problem since you shouldn't see the difference in
> master either - a straight shapefile fails. I'm guessing that way down
> in the shapefile driver nulls in the DBF are handled as empty string
> which technically match the "0 or more character" pattern. Any ideas
> on how to deal with something like this?

In OGR, empty strings in .DBF are consider as NULL values, and not empty strings. I'm not sure this is a universal choice, because DBF has no proper way of specifying the NULL concept.
And I don't think there's this concept of NULL currently in MapServer attributes (the lack of typing is also quite limiting sometimes). The start would be to change msDBFReadAttribute() to return NULL if DBFIsValueNULL().
But that would likely cause many problems in the rest of the code base....
Or a more restricted change when a LIKE is evaluated to consider that empty string is NULL and then evaluate it to FALSE.

> Changing the regex to ".+" makes the test pass but I don't think
> that's the correct solution.
>
> Steve
>
> StEVE LIME  | DATA & Applications MANAGER MN.IT Services @ MnDNR
> 651-259-5473 (w)  |  651-297-4946 (f) |
> [hidden email]<mailto:[hidden email]>
>
> [cid:image003.jpg@01CFC051.43750460]<http://www.mn.gov/oet>
>
> Information Technology for Minnesota Government   |  
> mn.gov/oet<http://www.mn.gov/oet>

--
Spatialys - Geospatial professional services http://www.spatialys.com
_______________________________________________
mapserver-dev mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/mapserver-dev
Reply | Threaded
Open this post in threaded view
|

Re: Regression test issue: NULLs in shapefiles...

Lime, Steve D (MNIT)
Just a FYI that this was included as part of the RFC 91 change set - it only affects regex comparisons. Probably something to make note of in the release notes.

Steve

-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of Lime, Steve D (MNIT)
Sent: Monday, August 25, 2014 2:25 PM
To: Even Rouault; [hidden email]
Subject: Re: [mapserver-dev] Regression test issue: NULLs in shapefiles...

I think your last suggestion is a good one - it's a pretty localized change to msEvalExpression() for regex expressions and the logical expression parser (mapparser.y).

Steve

-----Original Message-----
From: Even Rouault [mailto:[hidden email]]
Sent: Monday, August 25, 2014 1:01 PM
To: [hidden email]
Cc: Lime, Steve D (MNIT)
Subject: Re: [mapserver-dev] Regression test issue: NULLs in shapefiles...

Le lundi 25 août 2014 17:42:26, Lime, Steve D (MNIT) a écrit :

> Hi all: I'm trying to debug a failing CITE test
> (wfs_200_cite_filter_like.xml) for RFC 91 that uses PropertyIsLike. In
> master that resolves to "item like '%'"  for OGR and other databases.
> For a shapefile it resolves to "item ~ '.*'". In the test shapefile
> data there are a bunch of null values for the item being checked. The
> test in question uses the OGR driver. Here's a quick summary:
>
> Master:
>
> -          Using OGR driver gets expression WHERE island_e like "%" and
> returns 3 features, ignoring nulls
>
> -          Using shapefile driver gets expression "[island_e]" ~ ".*" and
> returns all features
>
> RFC 91:
>
> -          Using OGR driver gets expression  "[island_e]" ~ ".*" and
> returns all features
>
> -          Using shapefile driver gets expression "[island_e]" ~ ".*" and
> returns all features
>
> This change is because RFC 91 translation runs through the MapServer
> expression syntax and the OGR driver can handle both cases. However,
> that's not the problem since you shouldn't see the difference in
> master either - a straight shapefile fails. I'm guessing that way down
> in the shapefile driver nulls in the DBF are handled as empty string
> which technically match the "0 or more character" pattern. Any ideas
> on how to deal with something like this?

In OGR, empty strings in .DBF are consider as NULL values, and not empty strings. I'm not sure this is a universal choice, because DBF has no proper way of specifying the NULL concept.
And I don't think there's this concept of NULL currently in MapServer attributes (the lack of typing is also quite limiting sometimes). The start would be to change msDBFReadAttribute() to return NULL if DBFIsValueNULL().
But that would likely cause many problems in the rest of the code base....
Or a more restricted change when a LIKE is evaluated to consider that empty string is NULL and then evaluate it to FALSE.

> Changing the regex to ".+" makes the test pass but I don't think
> that's the correct solution.
>
> Steve
>
> StEVE LIME  | DATA & Applications MANAGER MN.IT Services @ MnDNR
> 651-259-5473 (w)  |  651-297-4946 (f) |
> [hidden email]<mailto:[hidden email]>
>
> [cid:image003.jpg@01CFC051.43750460]<http://www.mn.gov/oet>
>
> Information Technology for Minnesota Government   |  
> mn.gov/oet<http://www.mn.gov/oet>

--
Spatialys - Geospatial professional services http://www.spatialys.com _______________________________________________
mapserver-dev mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/mapserver-dev
_______________________________________________
mapserver-dev mailing list
[hidden email]
http://lists.osgeo.org/mailman/listinfo/mapserver-dev