Re: GeoNetwork API calls to /records

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: GeoNetwork API calls to /records

Kim Mortimer
Hi all,

I'm working on a Python API call to upload records from a fancy metadata submission form. I'm using the api/0.1/records post method, but I can't seem to get it to work. It always gives me a 403 error, which if I understand the Swagger UI doc correctly, indicates I don't have sufficient access. But this is a fresh GeoNetwork server, I've opened a session with the default admin username and password, and I can do an empty GET and receive a HTTP 200 response (and changing login details causes that to change to a 401 error).

I've tried adding auth details into the post command, but that didn't change anything.

Any thoughts or ideas?

Code below
------
# -*- coding: utf-8 -*-
"""
Accesses the GeoNetwork REST API to upload an xml metadata record.
Uses the Requests package for convenience
Made in Spyder Editor
Kim Mortimer, June 2019
"""

import requests
from xml.dom import minidom

def gn_api_xml_push(XMLFilePath):
    GN_BaseURL = 'http://[! YOUR GEONETWORK HERE]/geonetwork/' #removed local IP address
    GN_API_URL = GN_BaseURL + 'api/0.1/records'
    session = requests.Session()
    session.auth=('admin', 'admin') #Your username/password here
    loginResponse = session.get(GN_API_URL)
    OpenedFile = open(XMLFilePath, 'r')
    XMLDoc = minidom.parse(OpenedFile) #turns a file object into an XML document
    OpenedFile.close()
    parameters = {
            'file': [XMLDoc], #Unsure if this is the Array format needed, but the error is the same even if I pass an empty array in
            'rejectIfInvalid': True
            }
    requestResponse = session.post(GN_API_URL, data=parameters)
    if requestResponse.status_code == 201:
        responseDict = requestResponse.json()
        if responseDict['numberOfRecordsNotFound'] > 0:
            return {'response': "not found"}
        elif responseDict['numberOfNullRecords'] > 0:
            return {'response': "empty"}
        elif responseDict['numberofRecordsWithErrors'] > 0:
            errorString = ''
            for error in responseDict['errors']:
                errorString = errorString + error['message'] + '\n'
            metadataErrorString = ''
            for value in responseDict['metadataErrors'].values():
                metadataErrorString = metadataErrorString + value[0]['message'] + '\n'
            return {'response':"Errors in metadata record",
                     'errors': errorString,
                     'metadataErrors': metadataErrorString
                     }
        elif responseDict['numberOfRecordsProcessed'] > 0:
            return {'response': 'processed'}
        else:
            return {'response': 'Unknown response - HTTP 201 success'}
    elif requestResponse.status_code == 403:
        return {'response': 'HTTP 403 - insufficient access',
                'loginResponse': loginResponse}
    else:
        return {'response': 'Unknown',
                'statuscode': requestResponse.status_code,
                'loginResponse': loginResponse}
------

PS: The Swagger UI "try it out" feature for this request also doesn't seem to work... I just used a known working XML document but the response is 400 -
{
  "message": "MultipartException",
  "code": "unsatisfied_request_parameter",
  "description": "The current request is not a multipart request"
}
Feels weird to me, but might just mean that the Swagger UI is out of date?

Thanks in advance,
Kim

MERIDIAN on blue circle containing many numbers, with an orange wave pulse to the right.
Kim Mortimer
Data Manager
MERIDIAN - Marine Environmental Research Infrastructure for Data Integration and Application Network
Institute for Big Data Analytics, Faculty of Computer Sciences, Dalhousie University
p: + 1 902 494 1812 m: +1 902 880 1863
a: 6050 University Ave, Halifax, NS, B3H 4R2, Canada
w: https://meridian.cs.dal.ca e: [hidden email]


_______________________________________________
GeoNetwork-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geonetwork-devel
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork
Reply | Threaded
Open this post in threaded view
|

Re: GeoNetwork API calls to /records

delawen
Hi,

Did you check the CSRF token too?



El mar., 25 jun. 2019 17:54, Kim Mortimer <[hidden email]> escribió:
Hi all,

I'm working on a Python API call to upload records from a fancy metadata submission form. I'm using the api/0.1/records post method, but I can't seem to get it to work. It always gives me a 403 error, which if I understand the Swagger UI doc correctly, indicates I don't have sufficient access. But this is a fresh GeoNetwork server, I've opened a session with the default admin username and password, and I can do an empty GET and receive a HTTP 200 response (and changing login details causes that to change to a 401 error).

I've tried adding auth details into the post command, but that didn't change anything.

Any thoughts or ideas?

Code below
------
# -*- coding: utf-8 -*-
"""
Accesses the GeoNetwork REST API to upload an xml metadata record.
Uses the Requests package for convenience
Made in Spyder Editor
Kim Mortimer, June 2019
"""

import requests
from xml.dom import minidom

def gn_api_xml_push(XMLFilePath):
    GN_BaseURL = 'http://[! YOUR GEONETWORK HERE]/geonetwork/' #removed local IP address
    GN_API_URL = GN_BaseURL + 'api/0.1/records'
    session = requests.Session()
    session.auth=('admin', 'admin') #Your username/password here
    loginResponse = session.get(GN_API_URL)
    OpenedFile = open(XMLFilePath, 'r')
    XMLDoc = minidom.parse(OpenedFile) #turns a file object into an XML document
    OpenedFile.close()
    parameters = {
            'file': [XMLDoc], #Unsure if this is the Array format needed, but the error is the same even if I pass an empty array in
            'rejectIfInvalid': True
            }
    requestResponse = session.post(GN_API_URL, data=parameters)
    if requestResponse.status_code == 201:
        responseDict = requestResponse.json()
        if responseDict['numberOfRecordsNotFound'] > 0:
            return {'response': "not found"}
        elif responseDict['numberOfNullRecords'] > 0:
            return {'response': "empty"}
        elif responseDict['numberofRecordsWithErrors'] > 0:
            errorString = ''
            for error in responseDict['errors']:
                errorString = errorString + error['message'] + '\n'
            metadataErrorString = ''
            for value in responseDict['metadataErrors'].values():
                metadataErrorString = metadataErrorString + value[0]['message'] + '\n'
            return {'response':"Errors in metadata record",
                     'errors': errorString,
                     'metadataErrors': metadataErrorString
                     }
        elif responseDict['numberOfRecordsProcessed'] > 0:
            return {'response': 'processed'}
        else:
            return {'response': 'Unknown response - HTTP 201 success'}
    elif requestResponse.status_code == 403:
        return {'response': 'HTTP 403 - insufficient access',
                'loginResponse': loginResponse}
    else:
        return {'response': 'Unknown',
                'statuscode': requestResponse.status_code,
                'loginResponse': loginResponse}
------

PS: The Swagger UI "try it out" feature for this request also doesn't seem to work... I just used a known working XML document but the response is 400 -
{
  "message": "MultipartException",
  "code": "unsatisfied_request_parameter",
  "description": "The current request is not a multipart request"
}
Feels weird to me, but might just mean that the Swagger UI is out of date?

Thanks in advance,
Kim

MERIDIAN on blue circle containing many numbers, with an orange wave pulse to the right.
Kim Mortimer
Data Manager
MERIDIAN - Marine Environmental Research Infrastructure for Data Integration and Application Network
Institute for Big Data Analytics, Faculty of Computer Sciences, Dalhousie University
p: + 1 902 494 1812 m: +1 902 880 1863
a: 6050 University Ave, Halifax, NS, B3H 4R2, Canada
w: https://meridian.cs.dal.ca e: [hidden email]
_______________________________________________
GeoNetwork-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geonetwork-devel
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork


_______________________________________________
GeoNetwork-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geonetwork-devel
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork
Reply | Threaded
Open this post in threaded view
|

Re: GeoNetwork API calls to /records

Kim Mortimer
Hi Maria,

Thanks for the tip! That was what I was missing, and that has gotten me into the POST processing now.

My new issue is that I can't figure out what format is wanted for the 'file' parameter. It feels like everything I try gives me an HTTP 400 error "Could not find acceptable representation", which implies to me I'm sending the wrong 'thing'.

From the SwaggerUI doc - I can see that it wants "array [file] (formData)"... but I've seen many examples of FormData and it's unclear whether that's meaningful (other attributes say "query", for example, but query is irrelevant to how they're constructed/filled)

Since I'm only working with a single file right now, in Python I can just wrap whatever 'file' is in a list. But it's not clear what 'file' is and how 'formData' plays a part. Some of the things I've tried...
A Python dictionary
A single string
The file path
The opened file object (using the built in open function)
The opened file object then read using default python (probably a bitwise string?)
A Python document created via DOM
A Python ElementTree
A Python Element (both of these are part of Python's built in XML parsing)
JSON

But none of these seemed to work. From the manual https://geonetwork-opensource.org/manuals/trunk/eng/users/api/the-geonetwork-api.html I thought a string inside the array would work, but no such luck. I'm probably misinterpreting the example on that page, though.

Kim


From: María Arias de Reyna <[hidden email]>
Sent: 25 June 2019 13:04
To: Kim Mortimer
Cc: Devel [hidden email]
Subject: Re: [GeoNetwork-devel] GeoNetwork API calls to /records
 
Hi,

Did you check the CSRF token too?



El mar., 25 jun. 2019 17:54, Kim Mortimer <[hidden email]> escribió:
Hi all,

I'm working on a Python API call to upload records from a fancy metadata submission form. I'm using the api/0.1/records post method, but I can't seem to get it to work. It always gives me a 403 error, which if I understand the Swagger UI doc correctly, indicates I don't have sufficient access. But this is a fresh GeoNetwork server, I've opened a session with the default admin username and password, and I can do an empty GET and receive a HTTP 200 response (and changing login details causes that to change to a 401 error).

I've tried adding auth details into the post command, but that didn't change anything.

Any thoughts or ideas?

Code below
------
# -*- coding: utf-8 -*-
"""
Accesses the GeoNetwork REST API to upload an xml metadata record.
Uses the Requests package for convenience
Made in Spyder Editor
Kim Mortimer, June 2019
"""

import requests
from xml.dom import minidom

def gn_api_xml_push(XMLFilePath):
    GN_BaseURL = 'http://[! YOUR GEONETWORK HERE]/geonetwork/' #removed local IP address
    GN_API_URL = GN_BaseURL + 'api/0.1/records'
    session = requests.Session()
    session.auth=('admin', 'admin') #Your username/password here
    loginResponse = session.get(GN_API_URL)
    OpenedFile = open(XMLFilePath, 'r')
    XMLDoc = minidom.parse(OpenedFile) #turns a file object into an XML document
    OpenedFile.close()
    parameters = {
            'file': [XMLDoc], #Unsure if this is the Array format needed, but the error is the same even if I pass an empty array in
            'rejectIfInvalid': True
            }
    requestResponse = session.post(GN_API_URL, data=parameters)
    if requestResponse.status_code == 201:
        responseDict = requestResponse.json()
        if responseDict['numberOfRecordsNotFound'] > 0:
            return {'response': "not found"}
        elif responseDict['numberOfNullRecords'] > 0:
            return {'response': "empty"}
        elif responseDict['numberofRecordsWithErrors'] > 0:
            errorString = ''
            for error in responseDict['errors']:
                errorString = errorString + error['message'] + '\n'
            metadataErrorString = ''
            for value in responseDict['metadataErrors'].values():
                metadataErrorString = metadataErrorString + value[0]['message'] + '\n'
            return {'response':"Errors in metadata record",
                     'errors': errorString,
                     'metadataErrors': metadataErrorString
                     }
        elif responseDict['numberOfRecordsProcessed'] > 0:
            return {'response': 'processed'}
        else:
            return {'response': 'Unknown response - HTTP 201 success'}
    elif requestResponse.status_code == 403:
        return {'response': 'HTTP 403 - insufficient access',
                'loginResponse': loginResponse}
    else:
        return {'response': 'Unknown',
                'statuscode': requestResponse.status_code,
                'loginResponse': loginResponse}
------

PS: The Swagger UI "try it out" feature for this request also doesn't seem to work... I just used a known working XML document but the response is 400 -
{
  "message": "MultipartException",
  "code": "unsatisfied_request_parameter",
  "description": "The current request is not a multipart request"
}
Feels weird to me, but might just mean that the Swagger UI is out of date?

Thanks in advance,
Kim

MERIDIAN on blue circle containing many numbers, with an orange wave pulse to the right.
Kim Mortimer
Data Manager
MERIDIAN - Marine Environmental Research Infrastructure for Data Integration and Application Network
Institute for Big Data Analytics, Faculty of Computer Sciences, Dalhousie University
p: + 1 902 494 1812 m: +1 902 880 1863
a: 6050 University Ave, Halifax, NS, B3H 4R2, Canada
w: https://meridian.cs.dal.ca e: [hidden email]
_______________________________________________
GeoNetwork-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geonetwork-devel
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork


_______________________________________________
GeoNetwork-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geonetwork-devel
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork
Reply | Threaded
Open this post in threaded view
|

Re: GeoNetwork API calls to /records

Juan Luis Rodríguez Ponce


On Tue, Jun 25, 2019 at 8:34 PM Kim Mortimer <[hidden email]> wrote:
Hi Maria,

Thanks for the tip! That was what I was missing, and that has gotten me into the POST processing now.

My new issue is that I can't figure out what format is wanted for the 'file' parameter. It feels like everything I try gives me an HTTP 400 error "Could not find acceptable representation", which implies to me I'm sending the wrong 'thing'.

From the SwaggerUI doc - I can see that it wants "array [file] (formData)"... but I've seen many examples of FormData and it's unclear whether that's meaningful (other attributes say "query", for example, but query is irrelevant to how they're constructed/filled)

Since I'm only working with a single file right now, in Python I can just wrap whatever 'file' is in a list. But it's not clear what 'file' is and how 'formData' plays a part. Some of the things I've tried...
A Python dictionary
A single string
The file path
The opened file object (using the built in open function)
The opened file object then read using default python (probably a bitwise string?)
A Python document created via DOM
A Python ElementTree
A Python Element (both of these are part of Python's built in XML parsing)
JSON

But none of these seemed to work. From the manual https://geonetwork-opensource.org/manuals/trunk/eng/users/api/the-geonetwork-api.html I thought a string inside the array would work, but no such luck. I'm probably misinterpreting the example on that page, though.


 
--
Vriendelijke groeten / Kind regards,

Juan Luis Rodríguez.


Veenderweg 13
6721 WD Bennekom
The Netherlands
T: <a href="tel:+31318416664" style="color:rgb(17,85,204);font-family:Helvetica,Arial,sans-serif" target="_blank">+31 (0)318 416664

Please consider the environment before printing this email.


_______________________________________________
GeoNetwork-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geonetwork-devel
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork
Reply | Threaded
Open this post in threaded view
|

Re: GeoNetwork API calls to /records

Juan Luis Rodríguez Ponce


On Tue, Jun 25, 2019 at 11:33 PM Juan Luis Rodríguez Ponce <[hidden email]> wrote:


On Tue, Jun 25, 2019 at 8:34 PM Kim Mortimer <[hidden email]> wrote:
Hi Maria,

Thanks for the tip! That was what I was missing, and that has gotten me into the POST processing now.

My new issue is that I can't figure out what format is wanted for the 'file' parameter. It feels like everything I try gives me an HTTP 400 error "Could not find acceptable representation", which implies to me I'm sending the wrong 'thing'.

From the SwaggerUI doc - I can see that it wants "array [file] (formData)"... but I've seen many examples of FormData and it's unclear whether that's meaningful (other attributes say "query", for example, but query is irrelevant to how they're constructed/filled)

Since I'm only working with a single file right now, in Python I can just wrap whatever 'file' is in a list. But it's not clear what 'file' is and how 'formData' plays a part. Some of the things I've tried...
A Python dictionary
A single string
The file path
The opened file object (using the built in open function)
The opened file object then read using default python (probably a bitwise string?)
A Python document created via DOM
A Python ElementTree
A Python Element (both of these are part of Python's built in XML parsing)
JSON

But none of these seemed to work. From the manual https://geonetwork-opensource.org/manuals/trunk/eng/users/api/the-geonetwork-api.html I thought a string inside the array would work, but no such luck. I'm probably misinterpreting the example on that page, though.


I guess the main problem is that you need to send a multipart/form-data request. You can check some examples here:

--
Vriendelijke groeten / Kind regards,

Juan Luis Rodríguez.


Veenderweg 13
6721 WD Bennekom
The Netherlands
T: <a href="tel:+31318416664" style="color:rgb(17,85,204);font-family:Helvetica,Arial,sans-serif" target="_blank">+31 (0)318 416664

Please consider the environment before printing this email.


_______________________________________________
GeoNetwork-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geonetwork-devel
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork
Reply | Threaded
Open this post in threaded view
|

Re: GeoNetwork API calls to /records

Kim Mortimer
Hi Juan Luis Rodriguez,

Thanks for the suggestions. That old mailing list thread ended up having most of the answers in it.
My working code looks like this...
------
def gn_api_xml_push(XMLFilePath):
    GN_BaseURL = 'YOUR GEONETWORK URL HERE'
    GN_API_URL = GN_BaseURL + 'srv/api/0.1/records'
    session = requests.Session()    
    session.auth=('admin', 'admin')    
    loginResponse = session.get(GN_BaseURL)
    session.post(GN_BaseURL)
    fileBundle = {'file': open(XMLFilePath, 'rb')}
    queryBundle = {
            'rejectIfInvalid': True            
            }
    headerBundle = {
            "X-XSRF-TOKEN": session.cookies.get_dict()['XSRF-TOKEN'],
            "Accept": "application/json"
            }
    requestResponse = session.post(GN_API_URL, files=fileBundle, data=queryBundle, headers=headerBundle, cookies=session.cookies.get_dict())
    responseDict = requestResponse.json()
    #Response handling below, removed for brevity
------

After some testing with a known invalid XML file (I put text in a numerical field), I ended up with one last question...

When 'rejectIfInvalid' is true, an invalid record generates a HTTP code 400 with details in the response description. But the code 201 model contains room for reporting "numberOfRecordsWithErrors" and error strings - so I was surprised that wasn't used instead. When are the 'errors' and 'metadataErrors' parts of the 201 response used?

Thanks,

Kim

From: Juan Luis Rodríguez Ponce <[hidden email]>
Sent: 25 June 2019 18:40
To: Kim Mortimer
Cc: María Arias de Reyna; Devel [hidden email]
Subject: Re: [GeoNetwork-devel] GeoNetwork API calls to /records
 


On Tue, Jun 25, 2019 at 11:33 PM Juan Luis Rodríguez Ponce <[hidden email]> wrote:


On Tue, Jun 25, 2019 at 8:34 PM Kim Mortimer <[hidden email]> wrote:
Hi Maria,

Thanks for the tip! That was what I was missing, and that has gotten me into the POST processing now.

My new issue is that I can't figure out what format is wanted for the 'file' parameter. It feels like everything I try gives me an HTTP 400 error "Could not find acceptable representation", which implies to me I'm sending the wrong 'thing'.

From the SwaggerUI doc - I can see that it wants "array [file] (formData)"... but I've seen many examples of FormData and it's unclear whether that's meaningful (other attributes say "query", for example, but query is irrelevant to how they're constructed/filled)

Since I'm only working with a single file right now, in Python I can just wrap whatever 'file' is in a list. But it's not clear what 'file' is and how 'formData' plays a part. Some of the things I've tried...
A Python dictionary
A single string
The file path
The opened file object (using the built in open function)
The opened file object then read using default python (probably a bitwise string?)
A Python document created via DOM
A Python ElementTree
A Python Element (both of these are part of Python's built in XML parsing)
JSON

But none of these seemed to work. From the manual https://geonetwork-opensource.org/manuals/trunk/eng/users/api/the-geonetwork-api.html I thought a string inside the array would work, but no such luck. I'm probably misinterpreting the example on that page, though.


I guess the main problem is that you need to send a multipart/form-data request. You can check some examples here:

--
Vriendelijke groeten / Kind regards,

Juan Luis Rodríguez.


Veenderweg 13
6721 WD Bennekom
The Netherlands
T: <a href="tel:&#43;31318416664" target="_blank" style="color:rgb(17,85,204); font-family:Helvetica,Arial,sans-serif">+31 (0)318 416664

Please consider the environment before printing this email.


_______________________________________________
GeoNetwork-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geonetwork-devel
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork