Use extra_columns in pycsw-admin setup_db

Next Topic
 
classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Use extra_columns in pycsw-admin setup_db

Thomas Mistelbauer

Dear pycsw developers,

I am currently setting up a pycsw server for the Earth Observation Data Centre (https://pycsw.eodc.eu/), where we mainly host Copernicus Sentinel data.

So far, I managed to set up the server using the default metadata profiles, however, in the next step, we need to add additional metadata elements (e.g. cloud-coverage), that are not covered by both the standard profiles and the database table columns.

I understand that I have to create a custom profile, however I don't really know what is the best way to extend the database table. I see that it is possible pass 'extra_columns' in setup_db, however, this argument is not used in the admin.setup_db in pycsw-admin.py. If I run setup_py in admin.py "manually", I lose the comfort of the config file. I know that this is no big deal, however, I would find it very convenient if it would be possible to set 'extra_columns' int the repository section of the config file, so that these columns are automatically created when pycsw-admin -c setup_db is executed.

Is there any reason why this is not the case already, or am I missing something completely, and there is a better way to implement additional columns? If not, then I would be happy to work on that :-)

Thank you for your help and time! :-)

Best regards,
Thomas​


_______________________________________________
pycsw-devel mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/pycsw-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Use extra_columns in pycsw-admin setup_db

Tom Kralidis
Hi Thomas:

On Fri, Jun 9, 2017 at 3:29 AM, Thomas Mistelbauer
<[hidden email]> wrote:
> Dear pycsw developers,
>
> I am currently setting up a pycsw server for the Earth Observation Data
> Centre (https://pycsw.eodc.eu/), where we mainly host Copernicus Sentinel
> data.
>

Great to see this! Please feel free to add this deployment to our Live
Deployments
wiki (https://github.com/geopython/pycsw/wiki/Live-Deployments) which powers
our map at http://pycsw.org/community

> So far, I managed to set up the server using the default metadata profiles,
> however, in the next step, we need to add additional metadata elements (e.g.
> cloud-coverage), that are not covered by both the standard profiles and the
> database table columns.
>
> I understand that I have to create a custom profile, however I don't really
> know what is the best way to extend the database table. I see that it is
> possible pass 'extra_columns' in setup_db, however, this argument is not
> used in the admin.setup_db in pycsw-admin.py. If I run setup_py in admin.py
> "manually", I lose the comfort of the config file. I know that this is no
> big deal, however, I would find it very convenient if it would be possible
> to set 'extra_columns' int the repository section of the config file, so
> that these columns are automatically created when pycsw-admin -c setup_db is
> executed.
>

Very good point; setup_db's extra_columns argument was implemented
primarily for use by composable workflows (see how CKAN uses it in
https://github.com/ckan/ckanext-spatial/blob/master/bin/ckan_pycsw.py#L17)


> Is there any reason why this is not the case already, or am I missing
> something completely, and there is a better way to implement additional
> columns? If not, then I would be happy to work on that :-)
>

Can you open a GitHub issue for this enhancement? I think having
configurable columns in the pycsw config is a great idea and user
friendly.  Thanks for this feedback!

..Tom


> Thank you for your help and time! :-)
>
> Best regards,
> Thomas
>
>
> _______________________________________________
> pycsw-devel mailing list
> [hidden email]
> https://lists.osgeo.org/mailman/listinfo/pycsw-devel
_______________________________________________
pycsw-devel mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/pycsw-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Use extra_columns in pycsw-admin setup_db

Angelos Tzotsos
Hi Thomas,

I just opened a ticket for the OGC OpenSearch EO extension, that would
probably bring into pycsw all the extra fields that you require.

https://github.com/geopython/pycsw/issues/538

Regards,
Angelos

On 06/15/2017 04:26 AM, Tom Kralidis wrote:

> Hi Thomas:
>
> On Fri, Jun 9, 2017 at 3:29 AM, Thomas Mistelbauer
> <[hidden email]> wrote:
>> Dear pycsw developers,
>>
>> I am currently setting up a pycsw server for the Earth Observation Data
>> Centre (https://pycsw.eodc.eu/), where we mainly host Copernicus Sentinel
>> data.
>>
> Great to see this! Please feel free to add this deployment to our Live
> Deployments
> wiki (https://github.com/geopython/pycsw/wiki/Live-Deployments) which powers
> our map at http://pycsw.org/community
>
>> So far, I managed to set up the server using the default metadata profiles,
>> however, in the next step, we need to add additional metadata elements (e.g.
>> cloud-coverage), that are not covered by both the standard profiles and the
>> database table columns.
>>
>> I understand that I have to create a custom profile, however I don't really
>> know what is the best way to extend the database table. I see that it is
>> possible pass 'extra_columns' in setup_db, however, this argument is not
>> used in the admin.setup_db in pycsw-admin.py. If I run setup_py in admin.py
>> "manually", I lose the comfort of the config file. I know that this is no
>> big deal, however, I would find it very convenient if it would be possible
>> to set 'extra_columns' int the repository section of the config file, so
>> that these columns are automatically created when pycsw-admin -c setup_db is
>> executed.
>>
> Very good point; setup_db's extra_columns argument was implemented
> primarily for use by composable workflows (see how CKAN uses it in
> https://github.com/ckan/ckanext-spatial/blob/master/bin/ckan_pycsw.py#L17)
>
>
>> Is there any reason why this is not the case already, or am I missing
>> something completely, and there is a better way to implement additional
>> columns? If not, then I would be happy to work on that :-)
>>
> Can you open a GitHub issue for this enhancement? I think having
> configurable columns in the pycsw config is a great idea and user
> friendly.  Thanks for this feedback!
>
> ..Tom
>
>
>> Thank you for your help and time! :-)
>>
>> Best regards,
>> Thomas
>>
>>
>> _______________________________________________
>> pycsw-devel mailing list
>> [hidden email]
>> https://lists.osgeo.org/mailman/listinfo/pycsw-devel
> _______________________________________________
> pycsw-devel mailing list
> [hidden email]
> https://lists.osgeo.org/mailman/listinfo/pycsw-devel


--
Angelos Tzotsos, PhD
Charter Member
Open Source Geospatial Foundation
http://users.ntua.gr/tzotsos

_______________________________________________
pycsw-devel mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/pycsw-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Use extra_columns in pycsw-admin setup_db

Thomas Mistelbauer
In reply to this post by Tom Kralidis
Hi Tom,

On Wed, 2017-06-14 at 21:26 -0400, Tom Kralidis wrote:

> Hi Thomas:
>
> On Fri, Jun 9, 2017 at 3:29 AM, Thomas Mistelbauer
> <[hidden email]> wrote:
> > Dear pycsw developers,
> >
> > I am currently setting up a pycsw server for the Earth Observation
> > Data
> > Centre (https://pycsw.eodc.eu/), where we mainly host Copernicus
> > Sentinel
> > data.
> >
>
> Great to see this! Please feel free to add this deployment to our
> Live
> Deployments
> wiki (https://github.com/geopython/pycsw/wiki/Live-Deployments) which
> powers
> our map at http://pycsw.org/community

Did so :-)

>
> > So far, I managed to set up the server using the default metadata
> > profiles,
> > however, in the next step, we need to add additional metadata
> > elements (e.g.
> > cloud-coverage), that are not covered by both the standard profiles
> > and the
> > database table columns.
> >
> > I understand that I have to create a custom profile, however I
> > don't really
> > know what is the best way to extend the database table. I see that
> > it is
> > possible pass 'extra_columns' in setup_db, however, this argument
> > is not
> > used in the admin.setup_db in pycsw-admin.py. If I run setup_py in
> > admin.py
> > "manually", I lose the comfort of the config file. I know that this
> > is no
> > big deal, however, I would find it very convenient if it would be
> > possible
> > to set 'extra_columns' int the repository section of the config
> > file, so
> > that these columns are automatically created when pycsw-admin -c
> > setup_db is
> > executed.
> >
>
> Very good point; setup_db's extra_columns argument was implemented
> primarily for use by composable workflows (see how CKAN uses it in
> https://github.com/ckan/ckanext-
> spatial/blob/master/bin/ckan_pycsw.py#L17)

Thank you for this tip, I'll have a closer look on that!

>
>
> > Is there any reason why this is not the case already, or am I
> > missing
> > something completely, and there is a better way to implement
> > additional
> > columns? If not, then I would be happy to work on that :-)
> >
>
> Can you open a GitHub issue for this enhancement? I think having
> configurable columns in the pycsw config is a great idea and user
> friendly.  Thanks for this feedback!

I will do so! Meanwhile, I implemented this option in a very quick and
dirty way in our fork (https://github.com/eodcgmbh/pycsw/)

What I'm still struggeling with, is how to make this columns queryable.

Thanks for your response!

Best regards,
Thomas
_______________________________________________
pycsw-devel mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/pycsw-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Use extra_columns in pycsw-admin setup_db

Thomas Mistelbauer
In reply to this post by Angelos Tzotsos
Hi Angelos,

this looks indeed very promising for us! Thank you for this tip!

Best regards,
Thomas

On Sun, 2017-06-18 at 14:44 +0300, Angelos Tzotsos wrote:

> Hi Thomas,
>
> I just opened a ticket for the OGC OpenSearch EO extension, that
> would 
> probably bring into pycsw all the extra fields that you require.
>
> https://github.com/geopython/pycsw/issues/538
>
> Regards,
> Angelos
>
> On 06/15/2017 04:26 AM, Tom Kralidis wrote:
> > Hi Thomas:
> >
> > On Fri, Jun 9, 2017 at 3:29 AM, Thomas Mistelbauer
> > <[hidden email]> wrote:
> > > Dear pycsw developers,
> > >
> > > I am currently setting up a pycsw server for the Earth
> > > Observation Data
> > > Centre (https://pycsw.eodc.eu/), where we mainly host Copernicus
> > > Sentinel
> > > data.
> > >
> >
> > Great to see this! Please feel free to add this deployment to our
> > Live
> > Deployments
> > wiki (https://github.com/geopython/pycsw/wiki/Live-Deployments)
> > which powers
> > our map at http://pycsw.org/community
> >
> > > So far, I managed to set up the server using the default metadata
> > > profiles,
> > > however, in the next step, we need to add additional metadata
> > > elements (e.g.
> > > cloud-coverage), that are not covered by both the standard
> > > profiles and the
> > > database table columns.
> > >
> > > I understand that I have to create a custom profile, however I
> > > don't really
> > > know what is the best way to extend the database table. I see
> > > that it is
> > > possible pass 'extra_columns' in setup_db, however, this argument
> > > is not
> > > used in the admin.setup_db in pycsw-admin.py. If I run setup_py
> > > in admin.py
> > > "manually", I lose the comfort of the config file. I know that
> > > this is no
> > > big deal, however, I would find it very convenient if it would be
> > > possible
> > > to set 'extra_columns' int the repository section of the config
> > > file, so
> > > that these columns are automatically created when pycsw-admin -c
> > > setup_db is
> > > executed.
> > >
> >
> > Very good point; setup_db's extra_columns argument was implemented
> > primarily for use by composable workflows (see how CKAN uses it in
> > https://github.com/ckan/ckanext-spatial/blob/master/bin/ckan_pycsw.
> > py#L17)
> >
> >
> > > Is there any reason why this is not the case already, or am I
> > > missing
> > > something completely, and there is a better way to implement
> > > additional
> > > columns? If not, then I would be happy to work on that :-)
> > >
> >
> > Can you open a GitHub issue for this enhancement? I think having
> > configurable columns in the pycsw config is a great idea and user
> > friendly.  Thanks for this feedback!
> >
> > ..Tom
> >
> >
> > > Thank you for your help and time! :-)
> > >
> > > Best regards,
> > > Thomas
> > >
> > >
> > > _______________________________________________
> > > pycsw-devel mailing list
> > > [hidden email]
> > > https://lists.osgeo.org/mailman/listinfo/pycsw-devel
> >
> > _______________________________________________
> > pycsw-devel mailing list
> > [hidden email]
> > https://lists.osgeo.org/mailman/listinfo/pycsw-devel
>
>
_______________________________________________
pycsw-devel mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/pycsw-devel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Use extra_columns in pycsw-admin setup_db

Tom Kralidis
In reply to this post by Thomas Mistelbauer
On Wed, Jun 21, 2017 at 7:11 AM, Thomas Mistelbauer
<[hidden email]> wrote:

> Hi Tom,
>
> On Wed, 2017-06-14 at 21:26 -0400, Tom Kralidis wrote:
>> Hi Thomas:
>>
>> On Fri, Jun 9, 2017 at 3:29 AM, Thomas Mistelbauer
>> <[hidden email]> wrote:
>> > Dear pycsw developers,
>> >
>> > I am currently setting up a pycsw server for the Earth Observation
>> > Data
>> > Centre (https://pycsw.eodc.eu/), where we mainly host Copernicus
>> > Sentinel
>> > data.
>> >
>>
>> Great to see this! Please feel free to add this deployment to our
>> Live
>> Deployments
>> wiki (https://github.com/geopython/pycsw/wiki/Live-Deployments) which
>> powers
>> our map at http://pycsw.org/community
>
> Did so :-)
>
>>
>> > So far, I managed to set up the server using the default metadata
>> > profiles,
>> > however, in the next step, we need to add additional metadata
>> > elements (e.g.
>> > cloud-coverage), that are not covered by both the standard profiles
>> > and the
>> > database table columns.
>> >
>> > I understand that I have to create a custom profile, however I
>> > don't really
>> > know what is the best way to extend the database table. I see that
>> > it is
>> > possible pass 'extra_columns' in setup_db, however, this argument
>> > is not
>> > used in the admin.setup_db in pycsw-admin.py. If I run setup_py in
>> > admin.py
>> > "manually", I lose the comfort of the config file. I know that this
>> > is no
>> > big deal, however, I would find it very convenient if it would be
>> > possible
>> > to set 'extra_columns' int the repository section of the config
>> > file, so
>> > that these columns are automatically created when pycsw-admin -c
>> > setup_db is
>> > executed.
>> >
>>
>> Very good point; setup_db's extra_columns argument was implemented
>> primarily for use by composable workflows (see how CKAN uses it in
>> https://github.com/ckan/ckanext-
>> spatial/blob/master/bin/ckan_pycsw.py#L17)
>
> Thank you for this tip, I'll have a closer look on that!
>>
>>
>> > Is there any reason why this is not the case already, or am I
>> > missing
>> > something completely, and there is a better way to implement
>> > additional
>> > columns? If not, then I would be happy to work on that :-)
>> >
>>
>> Can you open a GitHub issue for this enhancement? I think having
>> configurable columns in the pycsw config is a great idea and user
>> friendly.  Thanks for this feedback!
>
> I will do so! Meanwhile, I implemented this option in a very quick and
> dirty way in our fork (https://github.com/eodcgmbh/pycsw/)
>
> What I'm still struggeling with, is how to make this columns queryable.
>

Here's a minimal example to query based on a `cloud_cover` column:
https://gist.github.com/tomkralidis/ddf5ea8b08ae90df592863cf065bd35f

Having said this, it's a good time to update our profile plugin support to be
able to live outside the pycsw source code.  I'll work on this one (please
file a ticket? Thanks!).

..Tom

> Thanks for your response!
>
> Best regards,
> Thomas
_______________________________________________
pycsw-devel mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/pycsw-devel
Loading...