Re: [GeoNode-users] GeoNode - PyCSW integration issues

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: [GeoNode-users] GeoNode - PyCSW integration issues

Cor Melse
Dear Alessio,

One should never use the Dublin Core output from a CSW, but instead use the OGC- CWS ISO19139 output!
There the right information for geo datasets (services) is provided.

Following the notes by Stephen Richard is therefore the only right thing to do.

as an alternative for pyCSW Geonetwork is also a great tool to use (and OSS)

Cheers, Cor

2017-07-26 11:10 GMT+02:00 Alessio Fabiani <[hidden email]>:
Dear all,
I want to highlight some important issues about the current integration between GeoNode and PyCSW.

Other than a non-trivial security issue (basically the CSW exposes metadata for all Layers no matter if they are accessible or not by the user), one more, I would say, big issue is that currently is not possible to recognize the Layer "real name" from the metadata documents.

Let me provide an example. Here below a snippet of a DublinCore XML produced by PyCSW

...
<csw:Record>
    <dc:identifier>941a8d7e-7159-11e7-b283-002215079c40</dc:identifier>
    <dc:title>san_andres_y_providencia_highway</dc:title>
    <dc:type>dataset</dc:type>
    <dc:subject/>
    ...
    <dct:references scheme="OGC:WMS">http://localhost:8080/geoserver/geonode/wms</dct:references>
    <dct:references scheme="OGC:WFS">http://localhost:8080/geoserver/geonode/wfs</dct:references>
...


Seems to be good right? Wrong! There some important things missing here:

1. It is not possible to recognize the layer name at all; The element "title" is mapped to the resource title, meaning that whenever I change it on the metadata into a human readable fashion, I lose the layer name forever

2. Layer qualified name is missing; if the name had a namespace, like on GeoServer, it is not possible to know which is at all

3. WMS and WFS links cannot be used at all, since the layer name cannot be retrieved anywhere

Proposal

Following the notes written by Stephen Richard here https://github.com/geopython/pycsw/wiki/Geonode-notes 

the proposal here would be to add more references

OGC:WMS-1.1.1-http-get-capabilities
OGC:WMS-1.3.0-http-get-capabilities
OGC:WMS-1.1.1-http-get-map
OGC:WMS-1.3.0-http-get-map
OGC:WMS-1.1.1-http-get-featureinfo
OGC:WMS-1.3.0-http-get-featureinfo

OGC:WFS-1.0.0-http-get-capabilities
OGC:WFS-1.0.0-http-describe-feature
OGC:WFS-1.0.0-http-get-feature

OGC:WCS-1.1.0-http-get-capabilities
OGC:WCS-1.1.0-http-describe-coverage
OGC:WCS-1.1.0-http-get-coverage
...

in order to let CSW being able to send correct links to the clients.

If there are no objections, I will create a PR accordingly related to the following ISSUE


Please comment on this.


Regards,

Alessio Fabiani

==
GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information.
==

Ing. Alessio Fabiani

@alfa7691
Founder/Technical Lead


GeoSolutions S.A.S.
Via di Montramito 3/A
55054  Massarosa (LU)
Italy
phone: <a href="tel:+39%200584%20962313" value="+390584962313" target="_blank">+39 0584 962313
fax:     <a href="tel:+39%200584%20166%200272" value="+3905841660272" target="_blank">+39 0584 1660272
mob:   <a href="tel:+39%20331%20623%203686" value="+393316233686" target="_blank">+39 331 6233686

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

-------------------------------------------------------

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility  for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.


_______________________________________________
geonode-users mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/geonode-users



_______________________________________________
geonode-devel mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/geonode-devel
Reply | Threaded
Open this post in threaded view
|

Re: [GeoNode-users] GeoNode - PyCSW integration issues

Tom Kralidis
Hi Alessio: thanks for the info.  Comments interleaved:

On Wed, 26 Jul 2017, Alessio Fabiani wrote:

> Date: Wed, 26 Jul 2017 11:10:36 +0200
> From: Alessio Fabiani <[hidden email]>
> To: geonode-devel <[hidden email]>,
>     geonode-users <[hidden email]>
> Subject: [GeoNode-users] GeoNode - PyCSW integration issues
>
> Dear all,
> I want to highlight some important issues about the current integration
> between GeoNode and PyCSW.
>
> Other than a non-trivial security issue (basically the CSW exposes metadata
> for all Layers no matter if they are accessible or not by the user),

This is in my queue to fix (half way there).

> one
> more, I would say, big issue is that currently is not possible to recognize
> the Layer "real name" from the metadata documents.
>
> Let me provide an example. Here below a snippet of a DublinCore XML
> produced by PyCSW
>
> ...
> <csw:Record>
>    <dc:identifier>941a8d7e-7159-11e7-b283-002215079c40</dc:identifier>
>    <dc:title>san_andres_y_providencia_highway</dc:title>
>    <dc:type>dataset</dc:type>
>    <dc:subject/>
>    ...
>    <dct:references scheme="WWW:DOWNLOAD-1.0-http--download">
> http://localhost:8080/geoserver/wms?request=GetLegendGraphic&amp;format=image/png&amp;WIDTH=20&amp;HEIGHT=20&amp;LAYER=geonode:san_andres_y_providencia_highway&amp;legend_options=fontAntiAliasing:true;fontSize:12;forceLabels:on
> </dct:references>
>    <dct:references scheme="OGC:WMS">
> http://localhost:8080/geoserver/geonode/wms</dct:references>
>    <dct:references scheme="OGC:WFS">
> http://localhost:8080/geoserver/geonode/wfs</dct:references>
> ...
>
> (full XML here https://pastebin.com/Wu0vm79B)
>
> Seems to be good right? *Wrong!* There some important things missing here:
>
> 1. It is not possible to recognize the layer name at all; The element
> "title" is mapped to the resource title, meaning that whenever I change it
> on the metadata into a human readable fashion, I lose the layer name forever
>

You should use ISO instead?  See below comment.

> 2. Layer qualified name is missing; if the name had a namespace, like on
> GeoServer, it is not possible to know which is at all
>
> 3. WMS and WFS links cannot be used at all, since the layer name cannot be
> retrieved anywhere
>

Anyone who wants to do tight coupling to OWS resources via CSW search
workflows needs to use the ISO output, which provides the bare URL
as well as the given Layer/FeatureType/Coverage.

Otherwise, the Dublin Core approach allows for discovery followed by
the client doing a full round trip (GetCapabilities, etc.) to be able
to find the layer again.  This is not optimal, which is exactly why
the CSW ISO Application Profile was created to help tight couplling
workflows.

If there are issues in the ISO output then let's tackle those?



> *Proposal*
>
> Following the notes written by Stephen Richard here
> https://github.com/geopython/pycsw/wiki/Geonode-notes
>
> the proposal here would be to add more references
>
> OGC:WMS-1.1.1-http-get-capabilities
> OGC:WMS-1.3.0-http-get-capabilities
> OGC:WMS-1.1.1-http-get-map
> OGC:WMS-1.3.0-http-get-map
> OGC:WMS-1.1.1-http-get-featureinfo
> OGC:WMS-1.3.0-http-get-featureinfo
>
> OGC:WFS-1.0.0-http-get-capabilities
> OGC:WFS-1.0.0-http-describe-feature
> OGC:WFS-1.0.0-http-get-feature
>
> OGC:WCS-1.1.0-http-get-capabilities
> OGC:WCS-1.1.0-http-describe-coverage
> OGC:WCS-1.1.0-http-get-coverage
> ...

Note that we promoted this into a canonical lookup at:

https://github.com/OSGeo/Cat-Interop/blob/master/LinkPropertyLookupTable.csv


>
> in order to let CSW being able to send correct links to the clients.
>
> If there are no objections, I will create a PR accordingly related to the
> following ISSUE
>
> https://github.com/GeoNode/geonode/issues/2720#issuecomment-262829872
>
> Please comment on this.
>
>
> Regards,
>
> Alessio Fabiani
>
> ==
> GeoServer Professional Services from the experts! Visit http://goo.gl/it488V
> for more information.
> ==
>
> Ing. Alessio Fabiani
>
> @alfa7691
> Founder/Technical Lead
>
>
> GeoSolutions S.A.S.
> Via di Montramito 3/A
> 55054  Massarosa (LU)
> Italy
> phone: +39 0584 962313
> fax:     +39 0584 1660272
> mob:   +39 331 6233686
>
> http://www.geo-solutions.it
> http://twitter.com/geosolutions_it
>
> -------------------------------------------------------
>
> AVVERTENZE AI SENSI DEL D.Lgs. 196/2003
>
> Le informazioni contenute in questo messaggio di posta elettronica e/o
> nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
> loro utilizzo è consentito esclusivamente al destinatario del messaggio,
> per le finalità indicate nel messaggio stesso. Qualora riceviate questo
> messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
> darcene notizia via e-mail e di procedere alla distruzione del messaggio
> stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
> divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
> utilizzarlo per finalità diverse, costituisce comportamento contrario ai
> principi dettati dal D.Lgs. 196/2003.
>
> The information in this message and/or attachments, is intended solely for
> the attention and use of the named addressee(s) and may be confidential or
> proprietary in nature or covered by the provisions of privacy act
> (Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection
> Code).Any use not in accord with its purpose, any disclosure, reproduction,
> copying, distribution, or either dissemination, either whole or partial, is
> strictly forbidden except previous formal approval of the named
> addressee(s). If you are not the intended recipient, please contact
> immediately the sender by telephone, fax or e-mail and delete the
> information in this message that has been received in error. The sender
> does not give any warranty or accept liability as the content, accuracy or
> completeness of sent messages and accepts no responsibility  for changes
> made after they were sent or for other risks which arise as a result of
> e-mail transmission, viruses, etc.
>
_______________________________________________
geonode-devel mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/geonode-devel
Reply | Threaded
Open this post in threaded view
|

Re: [GeoNode-users] GeoNode - PyCSW integration issues

Alessio Fabiani-2
Hi Tom,
agree, the DublinCore was just a simplified example.

Currently on ISO output there's no layer/feature/coverage name at all, but only the title.

Regards,

Alessio Fabiani

==
GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information.
==

Ing. Alessio Fabiani

@alfa7691
Founder/Technical Lead


GeoSolutions S.A.S.
Via di Montramito 3/A
55054  Massarosa (LU)
Italy
phone: +39 0584 962313
fax:     +39 0584 1660272
mob:   +39 331 6233686

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

-------------------------------------------------------

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility  for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.


On Thu, Jul 27, 2017 at 1:39 PM, Tom Kralidis <[hidden email]> wrote:
Hi Alessio: thanks for the info.  Comments interleaved:

On Wed, 26 Jul 2017, Alessio Fabiani wrote:

Date: Wed, 26 Jul 2017 11:10:36 +0200
From: Alessio Fabiani <[hidden email]>
To: geonode-devel <[hidden email]>,
    geonode-users <[hidden email]>
Subject: [GeoNode-users] GeoNode - PyCSW integration issues

Dear all,
I want to highlight some important issues about the current integration
between GeoNode and PyCSW.

Other than a non-trivial security issue (basically the CSW exposes metadata
for all Layers no matter if they are accessible or not by the user),

This is in my queue to fix (half way there).

one
more, I would say, big issue is that currently is not possible to recognize
the Layer "real name" from the metadata documents.

Let me provide an example. Here below a snippet of a DublinCore XML
produced by PyCSW

...
<csw:Record>
   <dc:identifier>941a8d7e-7159-11e7-b283-002215079c40</dc:identifier>
   <dc:title>san_andres_y_providencia_highway</dc:title>
   <dc:type>dataset</dc:type>
   <dc:subject/>
   ...
   <dct:references scheme="WWW:DOWNLOAD-1.0-http--download">
http://localhost:8080/geoserver/wms?request=GetLegendGraphic&amp;format=image/png&amp;WIDTH=20&amp;HEIGHT=20&amp;LAYER=geonode:san_andres_y_providencia_highway&amp;legend_options=fontAntiAliasing:true;fontSize:12;forceLabels:on
</dct:references>
   <dct:references scheme="OGC:WMS">
http://localhost:8080/geoserver/geonode/wms</dct:references>
   <dct:references scheme="OGC:WFS">
http://localhost:8080/geoserver/geonode/wfs</dct:references>
...

(full XML here https://pastebin.com/Wu0vm79B)

Seems to be good right? *Wrong!* There some important things missing here:

1. It is not possible to recognize the layer name at all; The element
"title" is mapped to the resource title, meaning that whenever I change it
on the metadata into a human readable fashion, I lose the layer name forever


You should use ISO instead?  See below comment.

2. Layer qualified name is missing; if the name had a namespace, like on
GeoServer, it is not possible to know which is at all

3. WMS and WFS links cannot be used at all, since the layer name cannot be
retrieved anywhere


Anyone who wants to do tight coupling to OWS resources via CSW search
workflows needs to use the ISO output, which provides the bare URL
as well as the given Layer/FeatureType/Coverage.

Otherwise, the Dublin Core approach allows for discovery followed by
the client doing a full round trip (GetCapabilities, etc.) to be able
to find the layer again.  This is not optimal, which is exactly why
the CSW ISO Application Profile was created to help tight couplling
workflows.

If there are issues in the ISO output then let's tackle those?



*Proposal*

Following the notes written by Stephen Richard here
https://github.com/geopython/pycsw/wiki/Geonode-notes

the proposal here would be to add more references

OGC:WMS-1.1.1-http-get-capabilities
OGC:WMS-1.3.0-http-get-capabilities
OGC:WMS-1.1.1-http-get-map
OGC:WMS-1.3.0-http-get-map
OGC:WMS-1.1.1-http-get-featureinfo
OGC:WMS-1.3.0-http-get-featureinfo

OGC:WFS-1.0.0-http-get-capabilities
OGC:WFS-1.0.0-http-describe-feature
OGC:WFS-1.0.0-http-get-feature

OGC:WCS-1.1.0-http-get-capabilities
OGC:WCS-1.1.0-http-describe-coverage
OGC:WCS-1.1.0-http-get-coverage
...

Note that we promoted this into a canonical lookup at:

https://github.com/OSGeo/Cat-Interop/blob/master/LinkPropertyLookupTable.csv




in order to let CSW being able to send correct links to the clients.

If there are no objections, I will create a PR accordingly related to the
following ISSUE

https://github.com/GeoNode/geonode/issues/2720#issuecomment-262829872

Please comment on this.


Regards,

Alessio Fabiani

==
GeoServer Professional Services from the experts! Visit http://goo.gl/it488V
for more information.
==

Ing. Alessio Fabiani

@alfa7691
Founder/Technical Lead


GeoSolutions S.A.S.
Via di Montramito 3/A
55054  Massarosa (LU)
Italy
phone: <a href="tel:%2B39%200584%20962313" value="+390584962313" target="_blank">+39 0584 962313
fax:     <a href="tel:%2B39%200584%201660272" value="+3905841660272" target="_blank">+39 0584 1660272
mob:   <a href="tel:%2B39%20331%206233686" value="+393316233686" target="_blank">+39 331 6233686

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

-------------------------------------------------------

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o
nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il
loro utilizzo è consentito esclusivamente al destinatario del messaggio,
per le finalità indicate nel messaggio stesso. Qualora riceviate questo
messaggio senza esserne il destinatario, Vi preghiamo cortesemente di
darcene notizia via e-mail e di procedere alla distruzione del messaggio
stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso,
divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od
utilizzarlo per finalità diverse, costituisce comportamento contrario ai
principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for
the attention and use of the named addressee(s) and may be confidential or
proprietary in nature or covered by the provisions of privacy act
(Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection
Code).Any use not in accord with its purpose, any disclosure, reproduction,
copying, distribution, or either dissemination, either whole or partial, is
strictly forbidden except previous formal approval of the named
addressee(s). If you are not the intended recipient, please contact
immediately the sender by telephone, fax or e-mail and delete the
information in this message that has been received in error. The sender
does not give any warranty or accept liability as the content, accuracy or
completeness of sent messages and accepts no responsibility  for changes
made after they were sent or for other risks which arise as a result of
e-mail transmission, viruses, etc.



_______________________________________________
geonode-devel mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/geonode-devel