Quantcast

OAI/PMH Harvesting

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

OAI/PMH Harvesting

hoddih
Hello

I'm trying to harvest using oai-pmh protocol with no results.

I'm using this url for the harvesting and it gives a result, but Geonetwork isn't receiving anything.


I always get java error in the log fle:

2016-11-08 14:06:24,075 WARN  [x] - Raised exception while getting metadata file : java.lang.NullPointerException
2016-11-08 14:06:24,075 WARN  [x] - Check with your administrator the error.
2016-11-08 14:06:24,075 ERROR [x] -
java.lang.NullPointerException
        at org.fao.oaipmh.responses.GetRecordResponse.build(GetRecordResponse.java:85)
        at org.fao.oaipmh.responses.GetRecordResponse.<init>(GetRecordResponse.java:47)
        at org.fao.oaipmh.requests.GetRecordRequest.execute(GetRecordRequest.java:84)
        at org.fao.geonet.kernel.harvest.harvester.oaipmh.Harvester.retrieveMetadata(Harvester.java:372)
        at org.fao.geonet.kernel.harvest.harvester.oaipmh.Harvester.addMetadata(Harvester.java:316)
        at org.fao.geonet.kernel.harvest.harvester.oaipmh.Harvester.align(Harvester.java:284)
        at org.fao.geonet.kernel.harvest.harvester.oaipmh.Harvester.harvest(Harvester.java:173)
        at org.fao.geonet.kernel.harvest.harvester.oaipmh.OaiPmhHarvester.doHarvest(OaiPmhHarvester.java:154)
        at org.fao.geonet.kernel.harvest.harvester.AbstractHarvester$HarvestWithIndexProcessor.process(AbstractHarvester.java:497)
        at org.fao.geonet.kernel.harvest.harvester.AbstractHarvester.harvest(AbstractHarvester.java:566)
        at org.fao.geonet.kernel.harvest.harvester.HarvesterJob.execute(HarvesterJob.java:28)
        at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
        at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557)
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: OAI/PMH Harvesting

hoddih
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: OAI/PMH Harvesting

Mendes de Jesus, Jorge
Hi

It should be ok with the URL:

http://api.gbif.org/v1/oai-pmh/registry


I have been using GN for OAI harvesting without any problems

Jorge


On 11-11-16 14:14, hoddih wrote:

> Hi
> please anyone
>
>
> I'm wondering if it is possible to use this URL for harvesting:
> http://api.gbif.org/v1/oai-pmh/registry?verb=ListIdentifiers&set=installation:b494d108-871b-4dfa-bffe-c7949c7400e4&metadataPrefix=oai_dc
>
> <http://osgeo-org.1560.x6.nabble.com/file/n5295335/harvest2.jpg>
>
>
>
> --
> View this message in context: http://osgeo-org.1560.x6.nabble.com/OAI-PMH-Harvesting-tp5295006p5295335.html
> Sent from the GeoNetwork users mailing list archive at Nabble.com.
>
> ------------------------------------------------------------------------------
> Developer Access Program for Intel Xeon Phi Processors
> Access to Intel Xeon Phi processor-based developer platforms.
> With one year of Intel Parallel Studio XE.
> Training and support from Colfax.
> Order your platform today. http://sdm.link/xeonphi
> _______________________________________________
> GeoNetwork-users mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/geonetwork-users
> GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork

--
ISRIC - World Soil Information
Post: PO box 353, 6700 AJ, Wageningen,
The Netherlands Visiting Address: Droevendaalsesteeg 3, 6708 PB
Wageningen (Bdg. 101),
Office: C.013
Office Phone: +31 (0) 317 4 83715
Mobile Phone: +31 (0) 613 9 06950
Skype: jorge.de.jesus (Jorge Samuel Mendes de Jesus)
OpenPGPKey: 0xA3D0065A


------------------------------------------------------------------------------
Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today. http://sdm.link/xeonphi
_______________________________________________
GeoNetwork-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geonetwork-users
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: OAI/PMH Harvesting

hoddih
Thanks Jorge

It runs normally but I always get the messages that that I showed in the first post in this thread.
 
  total: 62
 

 unretrievable: 62



And in the log file is some Java NullPointerException!  Any clue?

p.s.  My hair is starting be little gray  
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: OAI/PMH Harvesting

Jose Garcia
Hi

I've done a quick test using the same set as you are using with a 3.2.x
branch setup and get the 62 records harvested properly.

I see 62 records harvested. Which version are you using?

Regards,
Jose García


On Fri, Nov 11, 2016 at 4:15 PM, Jose Garcia <[hidden email]> wrote:

> Hi
>
> I've done a quick test using the same set as you are using with a 3.2.x
> branch setup and get the 62 records harvested properly.
>
> [image: Inline image 1]
>
> [image: Inline image 2]
>
> Which version are you using?
>
> Regards,
> Jose García
>
> On Fri, Nov 11, 2016 at 3:50 PM, hoddih <[hidden email]> wrote:
>
>> Thanks Jorge
>>
>> It runs normally but I always get the messages that that I showed in the
>> first post in this thread.
>>
>>   total: 62
>>  *
>>  unretrievable: 62
>> *
>>
>> And in the log file is some Java NullPointerException!  Any clue?
>>
>> p.s.  My hair is starting be little gray
>>
>>
>>
>> --
>> View this message in context: http://osgeo-org.1560.x6.nabbl
>> e.com/OAI-PMH-Harvesting-tp5295006p5295359.html
>> Sent from the GeoNetwork users mailing list archive at Nabble.com.
>>
>> ------------------------------------------------------------
>> ------------------
>> Developer Access Program for Intel Xeon Phi Processors
>> Access to Intel Xeon Phi processor-based developer platforms.
>> With one year of Intel Parallel Studio XE.
>> Training and support from Colfax.
>> Order your platform today. http://sdm.link/xeonphi
>> _______________________________________________
>> GeoNetwork-users mailing list
>> [hidden email]
>> https://lists.sourceforge.net/lists/listinfo/geonetwork-users
>> GeoNetwork OpenSource is maintained at http://sourceforge.net/project
>> s/geonetwork
>>
>
>
>
> --
>
>
>
>
>
>
>
>
>
>
>
>
>
> *Vriendelijke groeten / Kind regards,Jose García
> <http://www.geocat.net/>Veenderweg 136721 WD BennekomThe NetherlandsT: +31
> (0)318 416664 <+31318416664> <https://www.facebook.com/geocatbv>
> <https://twitter.com/geocat_bv>
> <https://plus.google.com/u/1/+GeocatNetbv/posts>Please consider the
> environment before printing this email.*
>


--













*Vriendelijke groeten / Kind regards,Jose García
<http://www.geocat.net/>Veenderweg 136721 WD BennekomThe NetherlandsT: +31
(0)318 416664 <+31318416664> <https://www.facebook.com/geocatbv>
<https://twitter.com/geocat_bv>
<https://plus.google.com/u/1/+GeocatNetbv/posts>Please consider the
environment before printing this email.*

------------------------------------------------------------------------------

_______________________________________________
GeoNetwork-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geonetwork-users
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork

harvester-config.png (101K) Download Attachment
result.png (28K) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: OAI/PMH Harvesting

hoddih
Thanks a millions Jose

It works now.

I was doing it in a wrong way.  I putted the"set" and "prefix" in the URL
not in the "search Filter"

You guys are great

On Fri, Nov 11, 2016 at 3:17 PM, Jose Garcia <[hidden email]> wrote:

> Hi
>
> I've done a quick test using the same set as you are using with a 3.2.x
> branch setup and get the 62 records harvested properly.
>
> I see 62 records harvested. Which version are you using?
>
> Regards,
> Jose García
>
>
> On Fri, Nov 11, 2016 at 4:15 PM, Jose Garcia <[hidden email]>
> wrote:
>
>> Hi
>>
>> I've done a quick test using the same set as you are using with a 3.2.x
>> branch setup and get the 62 records harvested properly.
>>
>> [image: Inline image 1]
>>
>> [image: Inline image 2]
>>
>> Which version are you using?
>>
>> Regards,
>> Jose García
>>
>> On Fri, Nov 11, 2016 at 3:50 PM, hoddih <[hidden email]> wrote:
>>
>>> Thanks Jorge
>>>
>>> It runs normally but I always get the messages that that I showed in the
>>> first post in this thread.
>>>
>>>   total: 62
>>>  *
>>>  unretrievable: 62
>>> *
>>>
>>> And in the log file is some Java NullPointerException!  Any clue?
>>>
>>> p.s.  My hair is starting be little gray
>>>
>>>
>>>
>>> --
>>> View this message in context: http://osgeo-org.1560.x6.nabbl
>>> e.com/OAI-PMH-Harvesting-tp5295006p5295359.html
>>> Sent from the GeoNetwork users mailing list archive at Nabble.com.
>>>
>>> ------------------------------------------------------------
>>> ------------------
>>> Developer Access Program for Intel Xeon Phi Processors
>>> Access to Intel Xeon Phi processor-based developer platforms.
>>> With one year of Intel Parallel Studio XE.
>>> Training and support from Colfax.
>>> Order your platform today. http://sdm.link/xeonphi
>>> _______________________________________________
>>> GeoNetwork-users mailing list
>>> [hidden email]
>>> https://lists.sourceforge.net/lists/listinfo/geonetwork-users
>>> GeoNetwork OpenSource is maintained at http://sourceforge.net/project
>>> s/geonetwork
>>>
>>
>>
>>
>> --
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> *Vriendelijke groeten / Kind regards,Jose García
>> <http://www.geocat.net/>Veenderweg 136721 WD BennekomThe NetherlandsT: +31
>> (0)318 416664 <+31318416664> <https://www.facebook.com/geocatbv>
>> <https://twitter.com/geocat_bv>
>> <https://plus.google.com/u/1/+GeocatNetbv/posts>Please consider the
>> environment before printing this email.*
>>
>
>
>
> --
>
>
>
>
>
>
>
>
>
>
>
>
>
> *Vriendelijke groeten / Kind regards,Jose García
> <http://www.geocat.net/>Veenderweg 136721 WD BennekomThe NetherlandsT: +31
> (0)318 416664 <+31318416664> <https://www.facebook.com/geocatbv>
> <https://twitter.com/geocat_bv>
> <https://plus.google.com/u/1/+GeocatNetbv/posts>Please consider the
> environment before printing this email.*
>


--
Hólmgrímur Helgason (Hóddi)
CAFF - Data Manager
[hidden email]
+354 462 3357

------------------------------------------------------------------------------

_______________________________________________
GeoNetwork-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geonetwork-users
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork

result.png (28K) Download Attachment
harvester-config.png (101K) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: OAI/PMH Harvesting

Mendes de Jesus, Jorge
In reply to this post by hoddih
Hi

I am not certain about GN log verbose (can't remember how to increase
verbose of logs), but I think the geonetwork log contains the requests
done during harvest. It is very likely that you have a URL list of the
request made to the OAI, copy a few requests and see what the OAI
returns or does, maybe there is something misconfigured

I think NullPointerException happens because you are not getting any
content. Looking at your last email,  have you configured GN to use:

 
set  --> installation:b494d108-871b-4dfa-bffe-c7949c7400e4
metadataPrefix --> oai_dc


Also, after setting the URL on the GN do you see available sets ???
(manual list of sets)
http://api.gbif.org/v1/oai-pmh/registry?verb=ListSets

and see the metadata options
http://api.gbif.org/v1/oai-pmh/registry?verb=ListMetadataFormats

Cheers
Jorge

On 11-11-16 15:50, hoddih wrote:

> Thanks Jorge
>
> It runs normally but I always get the messages that that I showed in the
> first post in this thread.
>  
>   total: 62
>  *
>  unretrievable: 62
> *
>
> And in the log file is some Java NullPointerException!  Any clue?
>
> p.s.  My hair is starting be little gray  
>
>
>
> --
> View this message in context: http://osgeo-org.1560.x6.nabble.com/OAI-PMH-Harvesting-tp5295006p5295359.html
> Sent from the GeoNetwork users mailing list archive at Nabble.com.
>
> ------------------------------------------------------------------------------
> Developer Access Program for Intel Xeon Phi Processors
> Access to Intel Xeon Phi processor-based developer platforms.
> With one year of Intel Parallel Studio XE.
> Training and support from Colfax.
> Order your platform today. http://sdm.link/xeonphi
> _______________________________________________
> GeoNetwork-users mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/geonetwork-users
> GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork

--
ISRIC - World Soil Information
Post: PO box 353, 6700 AJ, Wageningen,
The Netherlands Visiting Address: Droevendaalsesteeg 3, 6708 PB
Wageningen (Bdg. 101),
Office: C.013
Office Phone: +31 (0) 317 4 83715
Mobile Phone: +31 (0) 613 9 06950
Skype: jorge.de.jesus (Jorge Samuel Mendes de Jesus)
OpenPGPKey: 0xA3D0065A


------------------------------------------------------------------------------
Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today. http://sdm.link/xeonphi
_______________________________________________
GeoNetwork-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geonetwork-users
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: OAI/PMH Harvesting

hoddih
In reply to this post by hoddih
Thanks a millions

It works now.

I was doing it in a wrong way.  I putted the"set" and "prefix" in the URL not in the "search Filter"

You guys are great
Loading...