WFS GeoJSON Output Missing numberMatched

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

WFS GeoJSON Output Missing numberMatched

Seth G-2
Hi all,

I'm currently working on a front-end grid that requests features via WFS 2.0 and loads them using OpenLayers.
OpenLayers 4 doesn't support parsing of GML3 used by WFS 2.0, but I can return features as GeoJSON by setting the output format as follows:

    OUTPUTFORMAT
        NAME "geojson"
        DRIVER "OGR/GEOJSON"
        MIMETYPE "application/json; subtype=geojson"
        FORMATOPTION "FORM=SIMPLE"
        FORMATOPTION "STORAGE=memory"
    END

Then setting WEB > METADATA > "wfs_getfeature_formatlist" "gml,geojson,application/json"

WFS requests  are then made with URL parameters such as &service=WFS&version=2.0.0&request=GetFeature&typeName=MyLayer&outputFormat=application%2Fjson&startIndex=0&count=20

Everything works fine including paging, but in order to calculate how many "pages" of data I need to get the numberMatched property.
This can be turned on for GML3 using WEB > METADATA > "wfs_compute_number_matched" "true" (as noted at https://mapserver.org/fr/ogc/wfs_server.html#web-object-metadata).

If I then request GML with outputFormat=gml3 the GML response includes numberMatched="6607" (the number of features in the layer).

As GML3 features can't be rendered in OpenLayers I'd like to get this property added to the GeoJSON output. This appears to be implemented in GeoServer.
I was wondering if this is something that can be configured somewhere in the Mapfile perhaps as an option for the OGR/GEOJSON driver (https://gdal.org/drivers/vector/geojson.html).
The timeStamp, numberReturned are also missing from the GeoJSON output.

Any ideas or hints appreciated,

Seth


--
web:http://geographika.co.uk
twitter: @geographika
_______________________________________________
mapserver-users mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/mapserver-users
Reply | Threaded
Open this post in threaded view
|

Re: WFS GeoJSON Output Missing numberMatched

Even Rouault-2
On mercredi 5 juin 2019 23:43:22 CEST Seth G wrote:

> Hi all,
>
> I'm currently working on a front-end grid that requests features via WFS 2.0
> and loads them using OpenLayers. OpenLayers 4 doesn't support parsing of
> GML3 used by WFS 2.0, but I can return features as GeoJSON by setting the
> output format as follows:
>
>     OUTPUTFORMAT
>         NAME "geojson"
>         DRIVER "OGR/GEOJSON"
>         MIMETYPE "application/json; subtype=geojson"
>         FORMATOPTION "FORM=SIMPLE"
>         FORMATOPTION "STORAGE=memory"
>     END
>
> Then setting WEB > METADATA > "wfs_getfeature_formatlist"
> "gml,geojson,application/json"
>
> WFS requests  are then made with URL parameters such as
> &service=WFS&version=2.0.0&request=GetFeature&typeName=MyLayer&outputFormat
> =application%2Fjson&startIndex=0&count=20
>
> Everything works fine including paging, but in order to calculate how many
> "pages" of data I need to get the numberMatched property. This can be
> turned on for GML3 using WEB > METADATA > "wfs_compute_number_matched"
> "true" (as noted at
> https://mapserver.org/fr/ogc/wfs_server.html#web-object-metadata).
>
> If I then request GML with outputFormat=gml3 the GML response includes
> numberMatched="6607" (the number of features in the layer).
>
> As GML3 features can't be rendered in OpenLayers I'd like to get this
> property added to the GeoJSON output. This appears to be implemented in
> GeoServer. I was wondering if this is something that can be configured
> somewhere in the Mapfile perhaps as an option for the OGR/GEOJSON driver
> (https://gdal.org/drivers/vector/geojson.html). The timeStamp,
> numberReturned are also missing from the GeoJSON output.
>
> Any ideas or hints appreciated,

If your need is just to compute the number of patches, to get the
numberMatched, you could issue a GetFeature request with resultType=hits and
the default GML3 outputFormat. But I guess that's not an option given your
architecture.

If you absolutely need that in the GeoJSON output itself, you'll need to add
that specific behaviour in MapServer.
If you provide the NATIVE_MEDIA_TYPE=application/vnd.geo+json and
NATIVE_DATA={"numberMatched":XXXX} layer creation options to
OGR_DS_CreateLayer(), the OGR GeoJSON driver should include them in the output
(with a recent enough GDAL version)


$ ogr2ogr -f geojson /vsistdout/ poly.shp \
   -lco NATIVE_MEDIA_TYPE=application/vnd.geo+json \
   -lco 'NATIVE_DATA={"numberMatched":1234}'
{
"type": "FeatureCollection",
"numberMatched": 1234,
"name": "poly",
"features": [
[snip]


Even

--
Spatialys - Geospatial professional services
http://www.spatialys.com
_______________________________________________
mapserver-users mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/mapserver-users
Reply | Threaded
Open this post in threaded view
|

Re: WFS GeoJSON Output Missing numberMatched

Seth G-2
Thanks Even for both suggestions.
Using resultType=hits and caching the result for the duration of the session could be an option for read-only layers (and would likely speed things up as setting  "wfs_compute_number_matched" is not recommended for performance reasons). Editable layers would require two requests on every request, which might be ok depending on speeds.

The second approach is probably more robust, and works fine in a Mapfile by setting these layer creation options (although obviously the numberMatched in hard-coded):

    OUTPUTFORMAT
        NAME "application/json"
        DRIVER "OGR/GEOJSON"
        MIMETYPE "application/json"
        FORMATOPTION "FORM=SIMPLE"
        FORMATOPTION "STORAGE=memory" # filesystem and memory both work (only use one)
        FORMATOPTION "LCO:NATIVE_MEDIA_TYPE=application/vnd.geo+json"
        FORMATOPTION "LCO:NATIVE_DATA={'numberMatched':6607}"
    END

It looks like mapogroutput.c would need to be modified to support this approach.
Thanks again for the options,

Seth

--
web:http://geographika.co.uk
twitter: @geographika

On Thu, Jun 6, 2019, at 12:06 AM, Even Rouault wrote:

> On mercredi 5 juin 2019 23:43:22 CEST Seth G wrote:
> > Hi all,
> >
> > I'm currently working on a front-end grid that requests features via WFS 2.0
> > and loads them using OpenLayers. OpenLayers 4 doesn't support parsing of
> > GML3 used by WFS 2.0, but I can return features as GeoJSON by setting the
> > output format as follows:
> >
> >     OUTPUTFORMAT
> >         NAME "geojson"
> >         DRIVER "OGR/GEOJSON"
> >         MIMETYPE "application/json; subtype=geojson"
> >         FORMATOPTION "FORM=SIMPLE"
> >         FORMATOPTION "STORAGE=memory"
> >     END
> >
> > Then setting WEB > METADATA > "wfs_getfeature_formatlist"
> > "gml,geojson,application/json"
> >
> > WFS requests  are then made with URL parameters such as
> > &service=WFS&version=2.0.0&request=GetFeature&typeName=MyLayer&outputFormat
> > =application%2Fjson&startIndex=0&count=20
> >
> > Everything works fine including paging, but in order to calculate how many
> > "pages" of data I need to get the numberMatched property. This can be
> > turned on for GML3 using WEB > METADATA > "wfs_compute_number_matched"
> > "true" (as noted at
> > https://mapserver.org/fr/ogc/wfs_server.html#web-object-metadata).
> >
> > If I then request GML with outputFormat=gml3 the GML response includes
> > numberMatched="6607" (the number of features in the layer).
> >
> > As GML3 features can't be rendered in OpenLayers I'd like to get this
> > property added to the GeoJSON output. This appears to be implemented in
> > GeoServer. I was wondering if this is something that can be configured
> > somewhere in the Mapfile perhaps as an option for the OGR/GEOJSON driver
> > (https://gdal.org/drivers/vector/geojson.html). The timeStamp,
> > numberReturned are also missing from the GeoJSON output.
> >
> > Any ideas or hints appreciated,
>
> If your need is just to compute the number of patches, to get the
> numberMatched, you could issue a GetFeature request with resultType=hits and
> the default GML3 outputFormat. But I guess that's not an option given your
> architecture.
>
> If you absolutely need that in the GeoJSON output itself, you'll need to add
> that specific behaviour in MapServer.
> If you provide the NATIVE_MEDIA_TYPE=application/vnd.geo+json and
> NATIVE_DATA={"numberMatched":XXXX} layer creation options to
> OGR_DS_CreateLayer(), the OGR GeoJSON driver should include them in the output
> (with a recent enough GDAL version)
>
>
> $ ogr2ogr -f geojson /vsistdout/ poly.shp \
>    -lco NATIVE_MEDIA_TYPE=application/vnd.geo+json \
>    -lco 'NATIVE_DATA={"numberMatched":1234}'
> {
> "type": "FeatureCollection",
> "numberMatched": 1234,
> "name": "poly",
> "features": [
> [snip]
>
>
> Even
>
> --
> Spatialys - Geospatial professional services
> http://www.spatialys.com
>
_______________________________________________
mapserver-users mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/mapserver-users
Reply | Threaded
Open this post in threaded view
|

Re: WFS GeoJSON Output Missing numberMatched

Julian
In reply to this post by Seth G-2
Hi,

Maybe this is a stupid idea, but could you not simply keep asking until you get a page with zero features?

Regards,
Julian

-----Oprindelig meddelelse-----
Date: Wed, 05 Jun 2019 23:43:22 +0200
From: "Seth G" <[hidden email]>
To: [hidden email]
Subject: [mapserver-users] WFS GeoJSON Output Missing numberMatched
Message-ID: <[hidden email]>
Content-Type: text/plain

Hi all,

I'm currently working on a front-end grid that requests features via WFS 2.0 and loads them using OpenLayers.
OpenLayers 4 doesn't support parsing of GML3 used by WFS 2.0, but I can return features as GeoJSON by setting the output format as follows:

    OUTPUTFORMAT
        NAME "geojson"
        DRIVER "OGR/GEOJSON"
        MIMETYPE "application/json; subtype=geojson"
        FORMATOPTION "FORM=SIMPLE"
        FORMATOPTION "STORAGE=memory"
    END

Then setting WEB > METADATA > "wfs_getfeature_formatlist" "gml,geojson,application/json"

WFS requests  are then made with URL parameters such as &service=WFS&version=2.0.0&request=GetFeature&typeName=MyLayer&outputFormat=application%2Fjson&startIndex=0&count=20

Everything works fine including paging, but in order to calculate how many "pages" of data I need to get the numberMatched property.
This can be turned on for GML3 using WEB > METADATA > "wfs_compute_number_matched" "true" (as noted at https://mapserver.org/fr/ogc/wfs_server.html#web-object-metadata).

If I then request GML with outputFormat=gml3 the GML response includes numberMatched="6607" (the number of features in the layer).

As GML3 features can't be rendered in OpenLayers I'd like to get this property added to the GeoJSON output. This appears to be implemented in GeoServer.
I was wondering if this is something that can be configured somewhere in the Mapfile perhaps as an option for the OGR/GEOJSON driver (https://gdal.org/drivers/vector/geojson.html).
The timeStamp, numberReturned are also missing from the GeoJSON output.

Any ideas or hints appreciated,

Seth


--
web:http://geographika.co.uk
twitter: @geographika


------------------------------

Message: 3
Date: Thu, 06 Jun 2019 00:39:03 +0200
From: "Seth G" <[hidden email]>
To: "Even Rouault" <[hidden email]>,
        [hidden email]
Subject: Re: [mapserver-users] WFS GeoJSON Output Missing
        numberMatched
Message-ID: <[hidden email]>
Content-Type: text/plain

Thanks Even for both suggestions.
Using resultType=hits and caching the result for the duration of the session could be an option for read-only layers (and would likely speed things up as setting  "wfs_compute_number_matched" is not recommended for performance reasons). Editable layers would require two requests on every request, which might be ok depending on speeds.

The second approach is probably more robust, and works fine in a Mapfile by setting these layer creation options (although obviously the numberMatched in hard-coded):

    OUTPUTFORMAT
        NAME "application/json"
        DRIVER "OGR/GEOJSON"
        MIMETYPE "application/json"
        FORMATOPTION "FORM=SIMPLE"
        FORMATOPTION "STORAGE=memory" # filesystem and memory both work (only use one)
        FORMATOPTION "LCO:NATIVE_MEDIA_TYPE=application/vnd.geo+json"
        FORMATOPTION "LCO:NATIVE_DATA={'numberMatched':6607}"
    END

It looks like mapogroutput.c would need to be modified to support this approach.
Thanks again for the options,

Seth

--
web:http://geographika.co.uk
twitter: @geographika


_______________________________________________
mapserver-users mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/mapserver-users
Reply | Threaded
Open this post in threaded view
|

Re: WFS GeoJSON Output Missing numberMatched

Seth G-2
The grid has a "paging" toolbar at the bottom so it is the user that pages through the data - it also includes text such as "Page 3/56" but to calculate this it needs to know how many records in the whole dataset.
It coul d be an option to have an "infinite scrolling" grid which loads records on demand when scrolling down without needed to know the total count.

Seth

--
web:http://geographika.co.uk
twitter: @geographika

On Fri, Jun 7, 2019, at 2:35 PM, Julian Hollingbery wrote:

> Hi,
>
> Maybe this is a stupid idea, but could you not simply keep asking until
> you get a page with zero features?
>
> Regards,
> Julian
>
> -----Oprindelig meddelelse-----
> Date: Wed, 05 Jun 2019 23:43:22 +0200
> From: "Seth G" <[hidden email]>
> To: [hidden email]
> Subject: [mapserver-users] WFS GeoJSON Output Missing numberMatched
> Message-ID: <[hidden email]>
> Content-Type: text/plain
>
> Hi all,
>
> I'm currently working on a front-end grid that requests features via
> WFS 2.0 and loads them using OpenLayers.
> OpenLayers 4 doesn't support parsing of GML3 used by WFS 2.0, but I can
> return features as GeoJSON by setting the output format as follows:
>
>     OUTPUTFORMAT
>         NAME "geojson"
>         DRIVER "OGR/GEOJSON"
>         MIMETYPE "application/json; subtype=geojson"
>         FORMATOPTION "FORM=SIMPLE"
>         FORMATOPTION "STORAGE=memory"
>     END
>
> Then setting WEB > METADATA > "wfs_getfeature_formatlist"
> "gml,geojson,application/json"
>
> WFS requests  are then made with URL parameters such as
> &service=WFS&version=2.0.0&request=GetFeature&typeName=MyLayer&outputFormat=application%2Fjson&startIndex=0&count=20
>
> Everything works fine including paging, but in order to calculate how
> many "pages" of data I need to get the numberMatched property.
> This can be turned on for GML3 using WEB > METADATA >
> "wfs_compute_number_matched" "true" (as noted at
> https://mapserver.org/fr/ogc/wfs_server.html#web-object-metadata).
>
> If I then request GML with outputFormat=gml3 the GML response includes
> numberMatched="6607" (the number of features in the layer).
>
> As GML3 features can't be rendered in OpenLayers I'd like to get this
> property added to the GeoJSON output. This appears to be implemented in
> GeoServer.
> I was wondering if this is something that can be configured somewhere
> in the Mapfile perhaps as an option for the OGR/GEOJSON driver
> (https://gdal.org/drivers/vector/geojson.html).
> The timeStamp, numberReturned are also missing from the GeoJSON output.
>
> Any ideas or hints appreciated,
>
> Seth
>
>
> --
> web:http://geographika.co.uk
> twitter: @geographika
>
>
> ------------------------------
>
> Message: 3
> Date: Thu, 06 Jun 2019 00:39:03 +0200
> From: "Seth G" <[hidden email]>
> To: "Even Rouault" <[hidden email]>,
> [hidden email]
> Subject: Re: [mapserver-users] WFS GeoJSON Output Missing
> numberMatched
> Message-ID: <[hidden email]>
> Content-Type: text/plain
>
> Thanks Even for both suggestions.
> Using resultType=hits and caching the result for the duration of the
> session could be an option for read-only layers (and would likely speed
> things up as setting  "wfs_compute_number_matched" is not recommended
> for performance reasons). Editable layers would require two requests on
> every request, which might be ok depending on speeds.
>
> The second approach is probably more robust, and works fine in a
> Mapfile by setting these layer creation options (although obviously the
> numberMatched in hard-coded):
>
>     OUTPUTFORMAT
>         NAME "application/json"
>         DRIVER "OGR/GEOJSON"
>         MIMETYPE "application/json"
>         FORMATOPTION "FORM=SIMPLE"
>         FORMATOPTION "STORAGE=memory" # filesystem and memory both work
> (only use one)
>         FORMATOPTION "LCO:NATIVE_MEDIA_TYPE=application/vnd.geo+json"
>         FORMATOPTION "LCO:NATIVE_DATA={'numberMatched':6607}"
>     END
>
> It looks like mapogroutput.c would need to be modified to support this
> approach.
> Thanks again for the options,
>
> Seth
>
> --
> web:http://geographika.co.uk
> twitter: @geographika
>
>
> _______________________________________________
> mapserver-users mailing list
> [hidden email]
> https://lists.osgeo.org/mailman/listinfo/mapserver-users
_______________________________________________
mapserver-users mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/mapserver-users