Update metadata record through records API using Python

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Update metadata record through records API using Python

Rein van t Veer
Hi list,

First of all, thanks for having me. I'm having trouble updating a metadata
record through the REST API and don't know where to start. I've posted an
issue at
https://gis.stackexchange.com/questions/371749/automated-updates-for-fields-in-a-metadata-record
but my understanding now is that I'll be probably better off asking in this
mailing list.

I've tried re-creating the solution from
https://sourceforge.net/p/geonetwork/mailman/message/36703820/ but it
doesn't work; there's no 'XSRF-TOKEN' in the cookie jar after the first
post request. I've tried doing a first dummy call that gets me this token,
but then I strike out with a 500 error message:

```python
def gn_api_xml_push(xml_file_path: str) -> str:
    gn_base_url = 'https://my_geonetwork <https://data.geodan.io>'
    gn_api_url = gn_base_url + '/api/0.1/records'

    # set up the session
    session = requests.Session()
    session.auth = ('user', 'pass')

    # Log in
    login_response = session.get(gn_base_url)
    assert login_response.status_code == 200, f'Error authenticating with
server {gn_base_url}: response {login_response.status_code}'
    session.post(gn_api_url)  # Do a dummy post to get XSRF-TOKEN

    # Do the record posting itself
    headers = {
        'Cache-Control': 'no-cache',
        'Content-Type': 'application/x-www-form-urlencoded',
        'X-XSRF-TOKEN': session.cookies.get_dict()['XSRF-TOKEN'],
        'Accept': "application/json"
    }
    response = session.post(gn_api_url,
                            files= {'file': open(xml_file_path, 'rb')} ,
                            data= {'rejectIfInvalid': True} ,
                            headers=headers,
                            cookies=session.cookies.get_dict())
```

I'm also unsure on whether this will do it for us - will this update an
existing record with the contents of the XML? The XML I used by the way is
just a copy of the existing record - will this work? Or should I do a put
call to displace the contents, or a delete call to drop the record first?

Maybe the answer is somewhere in the archives of this mailing list, but
there's so much content to go through...

Kind regards and many thanks,
Rein van 't Veer
Geospatial Data Scientist

_______________________________________________
GeoNetwork-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geonetwork-users
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork
Reply | Threaded
Open this post in threaded view
|

Re: Update metadata record through records API using Python

Jo Cook
Hi Rein,

We do quite a bit of work manipulating metadata in geonetwork with
python. Here's some scripts that might help:
https://github.com/AstunTechnology/geonetwork-utility-scripts

These are not completely up to date but should get you started with the basics.

All the best

Jo

On Thu, Sep 10, 2020 at 3:00 PM Rein van t Veer <[hidden email]> wrote:

>
> Hi list,
>
> First of all, thanks for having me. I'm having trouble updating a metadata
> record through the REST API and don't know where to start. I've posted an
> issue at
> https://gis.stackexchange.com/questions/371749/automated-updates-for-fields-in-a-metadata-record
> but my understanding now is that I'll be probably better off asking in this
> mailing list.
>
> I've tried re-creating the solution from
> https://sourceforge.net/p/geonetwork/mailman/message/36703820/ but it
> doesn't work; there's no 'XSRF-TOKEN' in the cookie jar after the first
> post request. I've tried doing a first dummy call that gets me this token,
> but then I strike out with a 500 error message:
>
> ```python
> def gn_api_xml_push(xml_file_path: str) -> str:
>     gn_base_url = 'https://my_geonetwork <https://data.geodan.io>'
>     gn_api_url = gn_base_url + '/api/0.1/records'
>
>     # set up the session
>     session = requests.Session()
>     session.auth = ('user', 'pass')
>
>     # Log in
>     login_response = session.get(gn_base_url)
>     assert login_response.status_code == 200, f'Error authenticating with
> server {gn_base_url}: response {login_response.status_code}'
>     session.post(gn_api_url)  # Do a dummy post to get XSRF-TOKEN
>
>     # Do the record posting itself
>     headers = {
>         'Cache-Control': 'no-cache',
>         'Content-Type': 'application/x-www-form-urlencoded',
>         'X-XSRF-TOKEN': session.cookies.get_dict()['XSRF-TOKEN'],
>         'Accept': "application/json"
>     }
>     response = session.post(gn_api_url,
>                             files= {'file': open(xml_file_path, 'rb')} ,
>                             data= {'rejectIfInvalid': True} ,
>                             headers=headers,
>                             cookies=session.cookies.get_dict())
> ```
>
> I'm also unsure on whether this will do it for us - will this update an
> existing record with the contents of the XML? The XML I used by the way is
> just a copy of the existing record - will this work? Or should I do a put
> call to displace the contents, or a delete call to drop the record first?
>
> Maybe the answer is somewhere in the archives of this mailing list, but
> there's so much content to go through...
>
> Kind regards and many thanks,
> Rein van 't Veer
> Geospatial Data Scientist
>
> _______________________________________________
> GeoNetwork-users mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/geonetwork-users
> GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork



--
Jo Cook
t:+44 7930 524 155/twitter:@archaeogeek
Please note that currently I do not work on Friday afternoons. For
urgent responses at that time, please visit
support.astuntechnology.com or phone our office on 01372 744009

--
-- 
*Sign up to our mailing list
<https://astuntechnology.com/company/#email-updates> for updates on news,
products, conferences, events and training*
*
*

Astun Technology Ltd,
Epsom Square Centre, 6-7 The Derby Square, Epsom, Surrey, KT19 8AG, UK 

t:+44 1372 744 009 w: astuntechnology.com <http://astuntechnology.com/
twitter:@astuntech <https://twitter.com/astuntech>



iShare - enterprise
geographic intelligence platform <https://astuntechnology.com/ishare/>

GeoServer, PostGIS and QGIS training
<https://astuntechnology.com/training-courses/>
Helpdesk and customer
portal
<https://astuntech.atlassian.net/wiki/spaces/ISHAREHELP/pages/364970043/Astun+Technology+Support+Portal>





Company registration no. 5410695. Registered in England and Wales.
Registered office: 120 Manor Green Road, Epsom, Surrey, KT19 8LN VAT no.
864201149.


_______________________________________________
GeoNetwork-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geonetwork-users
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork
Reply | Threaded
Open this post in threaded view
|

Re: Update metadata record through records API using Python

Jo Cook
Hi Rein,

Yes, the examples I posted do use an additional xsl but I do have some
working code for just updating the metadata using the API- I will find
it for you (they are not currently in a public repository so it might
take a day or two).

Jo

On Mon, Oct 12, 2020 at 12:33 PM Rein van t Veer
<[hidden email]> wrote:

>
> Hi Jo, all,
>
> I tried applying your suggestion but I have no way of uploading a workflow xslt to the server, it's all managed. I found this thread: http://osgeo-org.1560.x6.nabble.com/API-HttpMediaTypeNotAcceptableException-td5386516.html on posting XML records to the {portal}/api/0.1/records endpoint but I got to about the same stage as Marko did - either the multipart/form-data header states that you need to supply a multipart boundary, or the application/x-www-form-urlencoded Content-Type header gives me a 400 that "A file MUST be provided". Unfortunately, the mail thread ends without a clear conclusion on what the complete syntax in cURL should be.
>
> I also tried the editor API. I'm able to fetch a html form using a simple get call, but when I try to post the unchanged form back to the API it gives me a 400 response with a runtime_exception and a NullPointerException in the body text. I assumed that the post body should contain the record data - it's far too much data to feed into the request parameters as the API docs suggest.
>
> Are there any working examples available (preferably in Python, cURL, or JS but I guess any automation language would do) on how to post changes to either {portal}/api/0.1/records or {portal}/api/0.1/records/{metadata_uuid}/editor? The workflow approach seems a little convoluted to me for a few simple value updates and I can't upload workload specifications to my machine anyway... Or is there another way?
>
> Kind regards,
> Rein
>
> Op vr 11 sep. 2020 om 10:14 schreef Rein van t Veer <[hidden email]>:
>>
>> Thanks Jo that was quick! I'll definitely be looking into your code. It looks like it's organized a little differently than using the records endpoint, but if this is the way to do it, we certainly will.
>> I like the helper function at https://github.com/AstunTechnology/geonetwork-utility-scripts/blob/master/email_update.py#L51 ;)
>>
>> Op do 10 sep. 2020 om 17:21 schreef Jo Cook <[hidden email]>:
>>>
>>> Hi Rein,
>>>
>>> We do quite a bit of work manipulating metadata in geonetwork with
>>> python. Here's some scripts that might help:
>>> https://github.com/AstunTechnology/geonetwork-utility-scripts
>>>
>>> These are not completely up to date but should get you started with the basics.
>>>
>>> All the best
>>>
>>> Jo
>>>
>>> On Thu, Sep 10, 2020 at 3:00 PM Rein van t Veer <[hidden email]> wrote:
>>> >
>>> > Hi list,
>>> >
>>> > First of all, thanks for having me. I'm having trouble updating a metadata
>>> > record through the REST API and don't know where to start. I've posted an
>>> > issue at
>>> > https://gis.stackexchange.com/questions/371749/automated-updates-for-fields-in-a-metadata-record
>>> > but my understanding now is that I'll be probably better off asking in this
>>> > mailing list.
>>> >
>>> > I've tried re-creating the solution from
>>> > https://sourceforge.net/p/geonetwork/mailman/message/36703820/ but it
>>> > doesn't work; there's no 'XSRF-TOKEN' in the cookie jar after the first
>>> > post request. I've tried doing a first dummy call that gets me this token,
>>> > but then I strike out with a 500 error message:
>>> >
>>> > ```python
>>> > def gn_api_xml_push(xml_file_path: str) -> str:
>>> >     gn_base_url = 'https://my_geonetwork <https://data.geodan.io>'
>>> >     gn_api_url = gn_base_url + '/api/0.1/records'
>>> >
>>> >     # set up the session
>>> >     session = requests.Session()
>>> >     session.auth = ('user', 'pass')
>>> >
>>> >     # Log in
>>> >     login_response = session.get(gn_base_url)
>>> >     assert login_response.status_code == 200, f'Error authenticating with
>>> > server {gn_base_url}: response {login_response.status_code}'
>>> >     session.post(gn_api_url)  # Do a dummy post to get XSRF-TOKEN
>>> >
>>> >     # Do the record posting itself
>>> >     headers = {
>>> >         'Cache-Control': 'no-cache',
>>> >         'Content-Type': 'application/x-www-form-urlencoded',
>>> >         'X-XSRF-TOKEN': session.cookies.get_dict()['XSRF-TOKEN'],
>>> >         'Accept': "application/json"
>>> >     }
>>> >     response = session.post(gn_api_url,
>>> >                             files= {'file': open(xml_file_path, 'rb')} ,
>>> >                             data= {'rejectIfInvalid': True} ,
>>> >                             headers=headers,
>>> >                             cookies=session.cookies.get_dict())
>>> > ```
>>> >
>>> > I'm also unsure on whether this will do it for us - will this update an
>>> > existing record with the contents of the XML? The XML I used by the way is
>>> > just a copy of the existing record - will this work? Or should I do a put
>>> > call to displace the contents, or a delete call to drop the record first?
>>> >
>>> > Maybe the answer is somewhere in the archives of this mailing list, but
>>> > there's so much content to go through...
>>> >
>>> > Kind regards and many thanks,
>>> > Rein van 't Veer
>>> > Geospatial Data Scientist
>>> >
>>> > _______________________________________________
>>> > GeoNetwork-users mailing list
>>> > [hidden email]
>>> > https://lists.sourceforge.net/lists/listinfo/geonetwork-users
>>> > GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork
>>>
>>>
>>>
>>> --
>>> Jo Cook
>>> t:+44 7930 524 155/twitter:@archaeogeek
>>> Please note that currently I do not work on Friday afternoons. For
>>> urgent responses at that time, please visit
>>> support.astuntechnology.com or phone our office on 01372 744009
>>>
>>> --
>>> --
>>> *Sign up to our mailing list
>>> <https://astuntechnology.com/company/#email-updates> for updates on news,
>>> products, conferences, events and training*
>>> *
>>> *
>>>
>>> Astun Technology Ltd,
>>> Epsom Square Centre, 6-7 The Derby Square, Epsom, Surrey, KT19 8AG, UK
>>>
>>> t:+44 1372 744 009 w: astuntechnology.com <http://astuntechnology.com/>
>>> twitter:@astuntech <https://twitter.com/astuntech>
>>>
>>>
>>>
>>> iShare - enterprise
>>> geographic intelligence platform <https://astuntechnology.com/ishare/>
>>>
>>> GeoServer, PostGIS and QGIS training
>>> <https://astuntechnology.com/training-courses/>
>>> Helpdesk and customer
>>> portal
>>> <https://astuntech.atlassian.net/wiki/spaces/ISHAREHELP/pages/364970043/Astun+Technology+Support+Portal>
>>>
>>>
>>>
>>>
>>>
>>> Company registration no. 5410695. Registered in England and Wales.
>>> Registered office: 120 Manor Green Road, Epsom, Surrey, KT19 8LN VAT no.
>>> 864201149.
>
>
>
> --
> Met vriendelijke groet,
>
> Rein van 't Veer
> Geospatial Data Scientist
>


--
Jo Cook
t:+44 7930 524 155/twitter:@archaeogeek
Please note that currently I do not work on Friday afternoons. For
urgent responses at that time, please visit
support.astuntechnology.com or phone our office on 01372 744009

--
-- 
*Sign up to our mailing list
<https://astuntechnology.com/company/#email-updates> for updates on news,
products, conferences, events and training*
*
*

Astun Technology Ltd,
Epsom Square Centre, 6-7 The Derby Square, Epsom, Surrey, KT19 8AG, UK 
t:+44 1372 744 009 w: astuntechnology.com <http://astuntechnology.com/
twitter:@astuntech <https://twitter.com/astuntech>



iShare - enterprise
geographic intelligence platform <https://astuntechnology.com/ishare/>
GeoServer, PostGIS and QGIS training
<https://astuntechnology.com/training-courses/>
Helpdesk and customer
portal
<https://astuntech.atlassian.net/wiki/spaces/ISHAREHELP/pages/364970043/Astun+Technology+Support+Portal>




Company registration no. 5410695. Registered in England and Wales.
Registered office: 120 Manor Green Road, Epsom, Surrey, KT19 8LN VAT no.
864201149.


_______________________________________________
GeoNetwork-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geonetwork-users
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork