OSGeo.org › OSGeo Software Projects › GeoNetwork opensource › GeoNetwork users

Geonetwork and S3

_‹ Previous Topic Next Topic _›

Classic

List

Threaded

4 messages Options

Andreas Böhme

Geonetwork and S3

Hi,

I am trying to understand how I can integrate S3 in my running instance
of GN3.10.2 I am reading the docs and I need some insight. First of all
I would like to mention that some of the links mentioned here:
https://geonetwork-opensource.org/manuals/trunk/en/install-guide/customizing-data-directory.html#using-a-s3-object-storage
are not working anymore. Maybe that is a small issus to be corrected.
Then I need to understand where I have to change my
"config-spring-geonetwork.xml"-file regarding S3. When I compare the
file in my instance to the one that is on github, there are quite some
changes. So can I just add whatever I need to configure about S3 in this
file, or do I need to clone the repo and change it there and use maven
and so on (which I dont have so many knowledge on how to do). Any help
is much appreciated.

Kind regards
Andreas

_______________________________________________
GeoNetwork-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geonetwork-users
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork

smime.p7s (7K) Download Attachment

Andreas Böhme

Re: Geonetwork and S3

Is there anybody who can point me in some direction?
Thanks
Andreas

Am 22.03.2021 um 11:57 schrieb Andreas Böhme:

> Hi,
>
> I am trying to understand how I can integrate S3 in my running
> instance of GN3.10.2 I am reading the docs and I need some insight.
> First of all I would like to mention that some of the links mentioned
> here:
> https://geonetwork-opensource.org/manuals/trunk/en/install-guide/customizing-data-directory.html#using-a-s3-object-storage
> are not working anymore. Maybe that is a small issus to be corrected.
> Then I need to understand where I have to change my
> "config-spring-geonetwork.xml"-file regarding S3. When I compare the
> file in my instance to the one that is on github, there are quite some
> changes. So can I just add whatever I need to configure about S3 in
> this file, or do I need to clone the repo and change it there and use
> maven and so on (which I dont have so many knowledge on how to do).
> Any help is much appreciated.
>
> Kind regards
> Andreas
>
>
>
>
> _______________________________________________
> GeoNetwork-users mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/geonetwork-users
> GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork

--
--------------------------------------------------------------------------------------------

Andreas Böhme
Techniker/Technician
Landschaftsökologie/Computational Landscape Ecology

Helmholtz-Zentrum für Umweltforschung GmbH - UFZ
Helmholtz Centre for Environmental Research GmbH - UFZ
Permoserstraße 15, 04318 Leipzig, Germany
Phone +49 341 235 1058
[hidden email], www.ufz.de

Sitz der Gesellschaft/Registered Office: Leipzig
Registergericht/Registration Office: Amtsgericht Leipzig
Handelsregister Nr./Trade Register Nr.: B 4703
Vorsitzender des Aufsichtsrats/Chairman of the Supervisory Board: N.N.
Wissenschaftlicher Geschäftsführer/Scientific Managing Director:
Prof. Dr. Georg Teutsch
Administrative Geschäftsführerin/Administrative Managing Director:
Dr. Sabine König

Vermeiden Sie unnötige Ausdrucke./Think before printing.

Informationen aus der Wissenschaft (z.B. Veranstaltungshinweise, Projektangaben, Publikationen,
Bücher)

--------------------------------------------------------------------------------------------

_______________________________________________
GeoNetwork-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geonetwork-users
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork

smime.p7s (7K) Download Attachment

Jose Garcia

Re: Geonetwork and S3

Hi Andreas

The documentation points to the file in master, that is quite different
indeed:
https://github.com/geonetwork/core-geonetwork/blob/master/core/src/main/resources/config-spring-geonetwork.xml,
it should be updated.

Ian added CMIS support also and refactored the configuration. The file for
3.10.x it's this one, that matches the documentation:

https://github.com/geonetwork/core-geonetwork/blob/2d9d7472fd4045fc2a9719d8a41ca616a7210678/core/src/main/resources/config-spring-geonetwork.xml#L203-L219

You need to change that file and rebuild the core jar package (I think with
the refactor in master branch, that is no longer required and can be
changed directly in a file in the web application, but it's not the case
for Geonetwork 3.10.x.

I can't help much more about the issues in S3, but if you can send the log
files with the error or open a ticket in GitHub, hopefully the developers
of this module can provide support.

Regards,
JOse García

On Tue, Mar 30, 2021 at 11:10 AM Andreas Böhme <[hidden email]>
wrote:

> Is there anybody who can point me in some direction?
> Thanks
> Andreas
>
> Am 22.03.2021 um 11:57 schrieb Andreas Böhme:
> > Hi,
> >
> > I am trying to understand how I can integrate S3 in my running
> > instance of GN3.10.2 I am reading the docs and I need some insight.
> > First of all I would like to mention that some of the links mentioned
> > here:
> >
> https://geonetwork-opensource.org/manuals/trunk/en/install-guide/customizing-data-directory.html#using-a-s3-object-storage
> > are not working anymore. Maybe that is a small issus to be corrected.
> > Then I need to understand where I have to change my
> > "config-spring-geonetwork.xml"-file regarding S3. When I compare the
> > file in my instance to the one that is on github, there are quite some
> > changes. So can I just add whatever I need to configure about S3 in
> > this file, or do I need to clone the repo and change it there and use
> > maven and so on (which I dont have so many knowledge on how to do).
> > Any help is much appreciated.
> >
> > Kind regards
> > Andreas
> >
> >
> >
> >
> > _______________________________________________
> > GeoNetwork-users mailing list
> > [hidden email]
> > https://lists.sourceforge.net/lists/listinfo/geonetwork-users
> > GeoNetwork OpenSource is maintained at
> http://sourceforge.net/projects/geonetwork
>
> --
>
> --------------------------------------------------------------------------------------------
>
> Andreas Böhme
> Techniker/Technician
> Landschaftsökologie/Computational Landscape Ecology
>
> Helmholtz-Zentrum für Umweltforschung GmbH - UFZ
> Helmholtz Centre for Environmental Research GmbH - UFZ
> Permoserstraße 15, 04318 Leipzig, Germany
> Phone +49 341 235 1058
> [hidden email], www.ufz.de
>
> Sitz der Gesellschaft/Registered Office: Leipzig
> Registergericht/Registration Office: Amtsgericht Leipzig
> Handelsregister Nr./Trade Register Nr.: B 4703
> Vorsitzender des Aufsichtsrats/Chairman of the Supervisory Board: N.N.
> Wissenschaftlicher Geschäftsführer/Scientific Managing Director:
> Prof. Dr. Georg Teutsch
> Administrative Geschäftsführerin/Administrative Managing Director:
> Dr. Sabine König
>
> Vermeiden Sie unnötige Ausdrucke./Think before printing.
>
> Informationen aus der Wissenschaft (z.B. Veranstaltungshinweise,
> Projektangaben, Publikationen,
> Bücher)
>
>
> --------------------------------------------------------------------------------------------
>
> _______________________________________________
> GeoNetwork-users mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/geonetwork-users
> GeoNetwork OpenSource is maintained at
> http://sourceforge.net/projects/geonetwork
>

ianwallen

Re: Geonetwork and S3

Andreas,

As Jose indicated, I did make some modifications to the storage layout when implementing CMIS and documentation may need to be updated. In regards to S3, I mostly updated the code so that you can pass an environment variables or via properties so that it is possible to connect to different filesystem storages (default, s3, cmis, jcloud) without having to do a custom build.
i.e.
geonetwork.store.type can be supplied using the -Dgeonetwork.store.type or via env as GEONETWORK_STORE_TYPE If not supplied it will default to "default" store type which is the local file system.
Option include
default - Default file system store
s3 - AWS S3 storage (see config-store/config-s3.xml for more details)
cmis - CMIS storage (see config-store/config-cmis.xml for more details)
https://github.com/geonetwork/core-geonetwork/blob/5ff61938639b3ea33298dd6aafdd18025d6c2982/core/src/main/resources/config-spring-geonetwork.xml#L193-L205
Of course, you can modify the file manually if desired and perform a build.

I also did add a jcloud implementation which is a generic driver that will allow all kinds of cloud storages including S3
https://github.com/geonetwork/core-geonetwork/pull/4570
The PR has instructions on how to test with Azure blob. It should be similar for S3, unfortunately, I don't have access to S3 storage to do testing.
I have tested this with Azure blobs and it works fine. I don't believe many in the community have access to cloud storage such as S3 or Azure Blob so it makes it difficult to get these approved!

I'm not sure if there is interest in adding this feature? The PR has been there for a year now and I have been updating the code to work with the latest changes but if there is no interest, I'm not sure how long I will keep this updated?

________________________________
From: Jose Garcia <[hidden email]>
Sent: March 30, 2021 6:39 AM
To: Andreas Böhme <[hidden email]>
Cc: [hidden email] <[hidden email]>
Subject: Re: [GeoNetwork-users] Geonetwork and S3

Hi Andreas

The documentation points to the file in master, that is quite different
indeed:
https://github.com/geonetwork/core-geonetwork/blob/master/core/src/main/resources/config-spring-geonetwork.xml,
it should be updated.

Ian added CMIS support also and refactored the configuration. The file for
3.10.x it's this one, that matches the documentation:

https://github.com/geonetwork/core-geonetwork/blob/2d9d7472fd4045fc2a9719d8a41ca616a7210678/core/src/main/resources/config-spring-geonetwork.xml#L203-L219

You need to change that file and rebuild the core jar package (I think with
the refactor in master branch, that is no longer required and can be
changed directly in a file in the web application, but it's not the case
for Geonetwork 3.10.x.

I can't help much more about the issues in S3, but if you can send the log
files with the error or open a ticket in GitHub, hopefully the developers
of this module can provide support.

Regards,
JOse García

On Tue, Mar 30, 2021 at 11:10 AM Andreas Böhme <[hidden email]>
wrote:

--

*Vriendelijke groeten / Kind regards,Jose García
<http://www.geocat.net/>Veenderweg 136721 WD BennekomThe NetherlandsT: +31
(0)318 416664 <+31318416664>Please consider the environment before printing
this email.*

_______________________________________________
GeoNetwork-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geonetwork-users
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork

_______________________________________________
GeoNetwork-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geonetwork-users
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork