OAI ListRecords temporal search

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

OAI ListRecords temporal search

Kralidis,Tom [Ontario]
Using 2.6.4 via the OAI provider, a simple request like:

http://HOST/geonetwork/srv/en/oaipmh?verb=ListRecords&metadataPrefix=iso
19139

yields results as expected.

However, using the 'from' parameter:

http://HOST/geonetwork/srv/en/oaipmh?verb=ListRecords&from=1999-10-10&me
tadataPrefix=iso19139

...yields no results, however all our records were inserted yesterday
(with gmd:dateStamp of 2013-11-08T05:38:05).

Clients use the 'from' parameter in support of periodic synchronization.

Any idea what's going on here?  What is OAI.ListRecords.from bound to
FYI we are bound to 2.6.4 and providing via OAI.

Thanks

..Tom

------------------------------------------------------------------------------
November Webinars for C, C++, Fortran Developers
Accelerate application performance with scalable programming models. Explore
techniques for threading, error checking, porting, and tuning. Get the most
from the latest Intel processors and coprocessors. See abstracts and register
http://pubads.g.doubleclick.net/gampad/clk?id=60136231&iu=/4140/ostg.clktrk
_______________________________________________
GeoNetwork-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geonetwork-users
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork
Reply | Threaded
Open this post in threaded view
|

Re: OAI ListRecords temporal search

kater2
It does seem to be acting on the metadata date (creation/revision), but isn't handling it properly.  
Eg it is treating the 'from=[date]' search as a "match [date]" search  (not that OAI has such a thing!)

TEST*:
When I tried the following, it only retrieved records with a metadata date of exactly 2010-08-26 (whereas the catalogue definitely had records that had been revised in 2012).

http://[hostname]/geonetwork/srv/eng/oaipmh?verb=ListRecords&from=2010-08-26&until=2013-10-31&metadataPrefix=iso19139

(*Using an older version of GN, so it might be different in 2.10?)
Reply | Threaded
Open this post in threaded view
|

Re: OAI ListRecords temporal search

Jose Garcia
Hi

At least in 2.6.X, seem the from parameter in the query in used to query
the lucene fields tempExtentBegin and tempExtentEnd. It's not using the
gmd:dateStamp.

These fields use the information from the following metadata elements (you
can check further in the file index-fields.xsl):


   - gmd:identificationInfo/gmd:MD_DataIdentification/gmd:date/gmd:CI_Date


   -
   gmd:identificationInfo/gmd:MD_DataIdentification/gmd:extent/gmd:EX_Extent/gmd:temporalElement/gmd:EX_TemporalExtent/gmd:extent/gml:TimePeriod


At least in 2.6.X, seem not only matching the exact date. This is part of
the lucene query created, using an interval:

 +(tempExtentBegin:[1999-10-10 TO *] tempExtentEnd:[1999-10-10 TO *])


I'm not an expert in OAI, if the metadata fields queried are not according
to the OAI specification, please open a bug in GitHub to be fixed in next
version.

Regards,
Jose García



On Tue, Nov 12, 2013 at 6:37 AM, kater2 <[hidden email]> wrote:

> It does seem to be acting on the metadata date (creation/revision), but
> isn't
> handling it properly.
> Eg it is treating the 'from=[date]' search as a "match [date]" search  (not
> that OAI has such a thing!)
>
> TEST*:
> When I tried the following, it only retrieved records with a metadata date
> of exactly 2010-08-26 (whereas the catalogue definitely had records that
> had
> been revised in 2012).
>
> http://
> [hostname]/geonetwork/srv/eng/oaipmh?verb=ListRecords&from=2010-08-26&until=2013-10-31&metadataPrefix=iso19139
>
> (*Using an older version of GN, so it might be different in 2.10?)
>
>
>
> --
> View this message in context:
> http://osgeo-org.1560.x6.nabble.com/OAI-ListRecords-temporal-search-tp5088176p5088667.html
> Sent from the GeoNetwork users mailing list archive at Nabble.com.
>
>
> ------------------------------------------------------------------------------
> November Webinars for C, C++, Fortran Developers
> Accelerate application performance with scalable programming models.
> Explore
> techniques for threading, error checking, porting, and tuning. Get the most
> from the latest Intel processors and coprocessors. See abstracts and
> register
> http://pubads.g.doubleclick.net/gampad/clk?id=60136231&iu=/4140/ostg.clktrk
> _______________________________________________
> GeoNetwork-users mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/geonetwork-users
> GeoNetwork OpenSource is maintained at
> http://sourceforge.net/projects/geonetwork
>



--

* GeoCat Bridge for ArcGIS allows instant publishing of data and metadata
on GeoServer and GeoNetwork. Visit http://geocat.net
<http://geocat.net/> for details.  _________________________Jose
GarcíaGeoCat bvVeenderweg 13 6721 WD BennekomThe
Netherlandshttp://GeoCat.net <http://geocat.net/> *
------------------------------------------------------------------------------
November Webinars for C, C++, Fortran Developers
Accelerate application performance with scalable programming models. Explore
techniques for threading, error checking, porting, and tuning. Get the most
from the latest Intel processors and coprocessors. See abstracts and register
http://pubads.g.doubleclick.net/gampad/clk?id=60136231&iu=/4140/ostg.clktrk
_______________________________________________
GeoNetwork-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geonetwork-users
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork
Reply | Threaded
Open this post in threaded view
|

Re: OAI ListRecords temporal search

Kralidis,Tom [Ontario]
Hi: thanks for the info.  After further investigation, per below, our metadata (WMO Core Metadata Profile 1.3) uses GML 3.2, which GN 2.6.4 does not recognize, which is why temporal searches were always empty.

Having said this, there is a bug in system configuration settings which does not allow me to set OAI Datesearch to "modification date" (keeps reverting to "temporal extent").  Fixing this manually under the hood fixes this.

Does latest stable GN support GML 3.2?

Thanks

..Tom


> -----Original Message-----
> From: Jose Garcia [mailto:[hidden email]]
> Sent: 2013-11-12 02:37
> To: kater2
> Cc: [hidden email]
> Subject: Re: [GeoNetwork-users] OAI ListRecords temporal search
>
> Hi
>
> At least in 2.6.X, seem the from parameter in the query in used to query the
> lucene fields tempExtentBegin and tempExtentEnd. It's not using the
> gmd:dateStamp.
>
> These fields use the information from the following metadata elements (you
> can check further in the file index-fields.xsl):
>
>
>    -
> gmd:identificationInfo/gmd:MD_DataIdentification/gmd:date/gmd:CI_Date
>
>
>    -
>
> gmd:identificationInfo/gmd:MD_DataIdentification/gmd:extent/gmd:EX_Ext
> ent/gmd:temporalElement/gmd:EX_TemporalExtent/gmd:extent/gml:Time
> Period
>
>
> At least in 2.6.X, seem not only matching the exact date. This is part of the
> lucene query created, using an interval:
>
>  +(tempExtentBegin:[1999-10-10 TO *] tempExtentEnd:[1999-10-10 TO *])
>
>
> I'm not an expert in OAI, if the metadata fields queried are not according to
> the OAI specification, please open a bug in GitHub to be fixed in next version.
>
> Regards,
> Jose García
>
>
>
> On Tue, Nov 12, 2013 at 6:37 AM, kater2 <[hidden email]> wrote:
>
> > It does seem to be acting on the metadata date (creation/revision),
> > but isn't handling it properly.
> > Eg it is treating the 'from=[date]' search as a "match [date]" search
> > (not that OAI has such a thing!)
> >
> > TEST*:
> > When I tried the following, it only retrieved records with a metadata
> > date of exactly 2010-08-26 (whereas the catalogue definitely had
> > records that had been revised in 2012).
> >
> > http://
> > [hostname]/geonetwork/srv/eng/oaipmh?verb=ListRecords&from=2010-
> 08-26&
> > until=2013-10-31&metadataPrefix=iso19139
> >
> > (*Using an older version of GN, so it might be different in 2.10?)
> >
> >
> >
> > --
> > View this message in context:
> > http://osgeo-org.1560.x6.nabble.com/OAI-ListRecords-temporal-search-tp
> > 5088176p5088667.html Sent from the GeoNetwork users mailing list
> > archive at Nabble.com.
> >
> >
> > ----------------------------------------------------------------------
> > -------- November Webinars for C, C++, Fortran Developers Accelerate
> > application performance with scalable programming models.
> > Explore
> > techniques for threading, error checking, porting, and tuning. Get the
> > most from the latest Intel processors and coprocessors. See abstracts
> > and register
> > http://pubads.g.doubleclick.net/gampad/clk?id=60136231&iu=/4140/ostg.c
> > lktrk _______________________________________________
> > GeoNetwork-users mailing list
> > [hidden email]
> > https://lists.sourceforge.net/lists/listinfo/geonetwork-users
> > GeoNetwork OpenSource is maintained at
> > http://sourceforge.net/projects/geonetwork
> >
>
>
>
> --
>
> * GeoCat Bridge for ArcGIS allows instant publishing of data and metadata on
> GeoServer and GeoNetwork. Visit http://geocat.net <http://geocat.net/>
> for details.  _________________________Jose GarcíaGeoCat
> bvVeenderweg 13 6721 WD BennekomThe Netherlandshttp://GeoCat.net
> <http://geocat.net/> *
> ------------------------------------------------------------------------------
> November Webinars for C, C++, Fortran Developers Accelerate application
> performance with scalable programming models. Explore techniques for
> threading, error checking, porting, and tuning. Get the most from the latest
> Intel processors and coprocessors. See abstracts and register
> http://pubads.g.doubleclick.net/gampad/clk?id=60136231&iu=/4140/ostg.clk
> trk
> _______________________________________________
> GeoNetwork-users mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/geonetwork-users
> GeoNetwork OpenSource is maintained at
> http://sourceforge.net/projects/geonetwork
------------------------------------------------------------------------------
November Webinars for C, C++, Fortran Developers
Accelerate application performance with scalable programming models. Explore
techniques for threading, error checking, porting, and tuning. Get the most
from the latest Intel processors and coprocessors. See abstracts and register
http://pubads.g.doubleclick.net/gampad/clk?id=60136231&iu=/4140/ostg.clktrk
_______________________________________________
GeoNetwork-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geonetwork-users
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork
Reply | Threaded
Open this post in threaded view
|

Re: OAI ListRecords temporal search

Kralidis,Tom [Ontario]
A few people have asked me to elaborate, so I'll do so on list:

1./
- system configuration update bug:  when updating System Configuration in the UI from "temporal extent" (which is the default value GN deploys with) to "modification date", clicking "save", then restarting GN (as the docs say to do), the same setting is reverted back to "temporal extent".  Even trying directly with the XML Services API yields the same result.  The brute force fix is to shutdown GN, and update the value in your DB with:

> update settings set value='2' where name='mdmode';

Alternatively, one can change the appropriate initial/default value in the db setup file https://github.com/geonetwork/core-geonetwork/blob/2.6.4/web/src/main/webapp/WEB-INF/classes/setup/sql/data/data-db-default.sql#L8150

Having said this, I think "modification date" should be the default value (mdmode=2) given the nature of OAI-PMH and harvesting use cases seem to be (AFAICT) centred around synchronization as opposed to dataset temporal extent querying

2./

WMO 1.3/GML3.2: starting with GML 3.2, the GML namespace is version specific.  I'm guessing GN is scanning for {http://www.opengis.net/gml}TimePeriod type thing (as the namespace prefix resolves to the namespace URI), so for the OAI datesearch by temporal extent use case, this gets ignored.  I'm not sure if this applies to all indexing of the metadata in GN.


> -----Original Message-----
> From: Kralidis,Tom [Ontario] [mailto:[hidden email]]
> Sent: 2013-11-12 06:56
> To: Jose Garcia; kater2
> Cc: [hidden email]
> Subject: Re: [GeoNetwork-users] OAI ListRecords temporal search
>
> Hi: thanks for the info.  After further investigation, per below, our metadata
> (WMO Core Metadata Profile 1.3) uses GML 3.2, which GN 2.6.4 does not
> recognize, which is why temporal searches were always empty.
>
> Having said this, there is a bug in system configuration settings which does
> not allow me to set OAI Datesearch to "modification date" (keeps reverting
> to "temporal extent").  Fixing this manually under the hood fixes this.
>
> Does latest stable GN support GML 3.2?
>
> Thanks
>
> ..Tom
>
>
> > -----Original Message-----
> > From: Jose Garcia [mailto:[hidden email]]
> > Sent: 2013-11-12 02:37
> > To: kater2
> > Cc: [hidden email]
> > Subject: Re: [GeoNetwork-users] OAI ListRecords temporal search
> >
> > Hi
> >
> > At least in 2.6.X, seem the from parameter in the query in used to
> > query the lucene fields tempExtentBegin and tempExtentEnd. It's not
> > using the gmd:dateStamp.
> >
> > These fields use the information from the following metadata elements
> > (you can check further in the file index-fields.xsl):
> >
> >
> >    -
> >
> gmd:identificationInfo/gmd:MD_DataIdentification/gmd:date/gmd:CI_Date
> >
> >
> >    -
> >
> >
> gmd:identificationInfo/gmd:MD_DataIdentification/gmd:extent/gmd:EX_Ext
> >
> ent/gmd:temporalElement/gmd:EX_TemporalExtent/gmd:extent/gml:Time
> > Period
> >
> >
> > At least in 2.6.X, seem not only matching the exact date. This is part
> > of the lucene query created, using an interval:
> >
> >  +(tempExtentBegin:[1999-10-10 TO *] tempExtentEnd:[1999-10-10 TO *])
> >
> >
> > I'm not an expert in OAI, if the metadata fields queried are not
> > according to the OAI specification, please open a bug in GitHub to be fixed
> in next version.
> >
> > Regards,
> > Jose García
> >
> >
> >
> > On Tue, Nov 12, 2013 at 6:37 AM, kater2 <[hidden email]> wrote:
> >
> > > It does seem to be acting on the metadata date (creation/revision),
> > > but isn't handling it properly.
> > > Eg it is treating the 'from=[date]' search as a "match [date]"
> > > search (not that OAI has such a thing!)
> > >
> > > TEST*:
> > > When I tried the following, it only retrieved records with a
> > > metadata date of exactly 2010-08-26 (whereas the catalogue
> > > definitely had records that had been revised in 2012).
> > >
> > > http://
> > >
> [hostname]/geonetwork/srv/eng/oaipmh?verb=ListRecords&from=2010-
> > 08-26&
> > > until=2013-10-31&metadataPrefix=iso19139
> > >
> > > (*Using an older version of GN, so it might be different in 2.10?)
> > >
> > >
> > >
> > > --
> > > View this message in context:
> > > http://osgeo-org.1560.x6.nabble.com/OAI-ListRecords-temporal-search-
> > > tp 5088176p5088667.html Sent from the GeoNetwork users mailing list
> > > archive at Nabble.com.
> > >
> > >
> > > --------------------------------------------------------------------
> > > --
> > > -------- November Webinars for C, C++, Fortran Developers Accelerate
> > > application performance with scalable programming models.
> > > Explore
> > > techniques for threading, error checking, porting, and tuning. Get
> > > the most from the latest Intel processors and coprocessors. See
> > > abstracts and register
> > >
> http://pubads.g.doubleclick.net/gampad/clk?id=60136231&iu=/4140/ostg
> > > .c lktrk _______________________________________________
> > > GeoNetwork-users mailing list
> > > [hidden email]
> > > https://lists.sourceforge.net/lists/listinfo/geonetwork-users
> > > GeoNetwork OpenSource is maintained at
> > > http://sourceforge.net/projects/geonetwork
> > >
> >
> >
> >
> > --
> >
> > * GeoCat Bridge for ArcGIS allows instant publishing of data and
> > metadata on GeoServer and GeoNetwork. Visit http://geocat.net
> > <http://geocat.net/> for details.  _________________________Jose
> > GarcíaGeoCat bvVeenderweg 13 6721 WD BennekomThe
> > Netherlandshttp://GeoCat.net <http://geocat.net/> *
> > ----------------------------------------------------------------------
> > -------- November Webinars for C, C++, Fortran Developers Accelerate
> > application performance with scalable programming models. Explore
> > techniques for threading, error checking, porting, and tuning. Get the
> > most from the latest Intel processors and coprocessors. See abstracts
> > and register
> > http://pubads.g.doubleclick.net/gampad/clk?id=60136231&iu=/4140/ostg.c
> > lk
> > trk
> > _______________________________________________
> > GeoNetwork-users mailing list
> > [hidden email]
> > https://lists.sourceforge.net/lists/listinfo/geonetwork-users
> > GeoNetwork OpenSource is maintained at
> > http://sourceforge.net/projects/geonetwork
> ------------------------------------------------------------------------------
> November Webinars for C, C++, Fortran Developers Accelerate application
> performance with scalable programming models. Explore techniques for
> threading, error checking, porting, and tuning. Get the most from the latest
> Intel processors and coprocessors. See abstracts and register
> http://pubads.g.doubleclick.net/gampad/clk?id=60136231&iu=/4140/ostg.clk
> trk
> _______________________________________________
> GeoNetwork-users mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/geonetwork-users
> GeoNetwork OpenSource is maintained at
> http://sourceforge.net/projects/geonetwork
------------------------------------------------------------------------------
DreamFactory - Open Source REST & JSON Services for HTML5 & Native Apps
OAuth, Users, Roles, SQL, NoSQL, BLOB Storage and External API Access
Free app hosting. Or install the open source package on any LAMP server.
Sign up and see examples for AngularJS, jQuery, Sencha Touch and Native!
http://pubads.g.doubleclick.net/gampad/clk?id=63469471&iu=/4140/ostg.clktrk
_______________________________________________
GeoNetwork-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geonetwork-users
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork