General question about recording metadata on feature counts

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

General question about recording metadata on feature counts

Jo Cook
Hi All,

I'm working on a GeoNetwork implementation where the client is keen to
record additional metadata on the count of different features in the
dataset. Not a count of different geometries, but totals of different
features based on some attribute. The metadata will be stored in Gemini 2.3
format, so ISO19139-based, but I've been looking in both the Feature
Catalog and Data Quality standards to see if I can find an existing
metadata element for storing this information and I'm not getting anywhere.
Does anyone have an example of this being done in practice? If there's even
an informal standard for recording this sort of information then I'd be
keen to use that rather than making something up.

Thanks

Jo



--
*Jo Cook*
t:+44 7930 524 155/twitter:@archaeogeek
Please note that currently I do not work on Friday afternoons. For urgent
responses at that time, please visit support.astuntechnology.com or phone
our office on 01372 744009

--
-- 
*Sign up to our mailing list
<https://astuntechnology.com/company/#email-updates> for updates on news,
products, conferences, events and training*
*
*

Astun Technology Ltd, 120
Manor Green Road, Epsom, Surrey, KT19 8LN, UK 
t:+44 1372 744 009 w: 
astuntechnology.com <http://astuntechnology.com/> twitter:@astuntech
<https://twitter.com/astuntech>



iShare - enterprise geographic
intelligence platform <https://astuntechnology.com/ishare/>
GeoServer,
PostGIS and QGIS training <https://astuntechnology.com/training-courses/>
Helpdesk and customer portal
<https://astuntech.atlassian.net/wiki/spaces/ISHAREHELP/pages/364970043/Astun+Technology+Support+Portal>




Company registration no. 5410695. Registered in England and Wales.
Registered office: 120 Manor Green Road, Epsom, Surrey, KT19 8LN VAT no.
864201149.

_______________________________________________
GeoNetwork-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geonetwork-users
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork
Reply | Threaded
Open this post in threaded view
|

Download large files

Viktor Stroh
Hi guys,
we are currently trying to solve the problems with uploading and
downloading large data.
Upload size could be set upwards: either as suggested in earlier
versions in the source code under services in the
config-spring-geonetwork.xml file
or with new parameter api.params.maxUploadSize in config.properties in
the newer versions.
Now we have the problem to download larger files via geonetwork. When
clicking on a link to a 7.3 GB file under Download and links, an error
message comes up immediately:
<code>runtime_exception</code>.
<description>
Handler dispatch failed; nested exception is java.lang.OutOfMemoryError:
Required array size too large
</description>
<message>NestedServletException</message>.
The memory parameters are currently:
CATALINA_OPTS="-Xms5g -Xmx12g -XX:MaxDirectMemorySize=8g
-Dgeonetwork.dir=/var/lib/gn_data"
Tomcat8 can only be reached locally and uses NGinx Proxy for external
communication via HTTPS protocol. The server.xml looks like this:
<Connector port="8080" protocol="HTTP/1.1"
                connectionTimeout="20000"
                address="127.0.0.1"
                redirectPort="443" />
the nginx configuration for the domain contains the section:
location / {
             proxy_pass <a href="http://127.0.0.1:8080;">http://127.0.0.1:8080;
             ...
             }
Do we have a configuration error? Is all data routed through Tomcat8 and
cannot be sent through an external web server?
We want a portal that provides the metadata as well as the actual data
and takes care of user rights, and we find the geonetwork software good
in this regard.

best regards

Viktor



_______________________________________________
GeoNetwork-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geonetwork-users
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork
Reply | Threaded
Open this post in threaded view
|

Re: Download large files

ianwallen
Viktor,

Geonetwork uses byte arrays to store the data before streaming it to the user.

https://github.com/geonetwork/core-geonetwork/blob/182575ffe37282b18e1911f1a80011e8cfb451df/services/src/main/java/org/fao/geonet/api/records/attachments/AttachmentsApi.java#L261

So I believe the limit should be around 2GB


________________________________
From: Viktor Stroh <[hidden email]>
Sent: February 24, 2021 11:26 AM
To: [hidden email] <[hidden email]>
Subject: [GeoNetwork-users] Download large files

Hi guys,
we are currently trying to solve the problems with uploading and
downloading large data.
Upload size could be set upwards: either as suggested in earlier
versions in the source code under services in the
config-spring-geonetwork.xml file
or with new parameter api.params.maxUploadSize in config.properties in
the newer versions.
Now we have the problem to download larger files via geonetwork. When
clicking on a link to a 7.3 GB file under Download and links, an error
message comes up immediately:
<code>runtime_exception</code>.
<description>
Handler dispatch failed; nested exception is java.lang.OutOfMemoryError:
Required array size too large
</description>
<message>NestedServletException</message>.
The memory parameters are currently:
CATALINA_OPTS="-Xms5g -Xmx12g -XX:MaxDirectMemorySize=8g
-Dgeonetwork.dir=/var/lib/gn_data"
Tomcat8 can only be reached locally and uses NGinx Proxy for external
communication via HTTPS protocol. The server.xml looks like this:
<Connector port="8080" protocol="HTTP/1.1"
                connectionTimeout="20000"
                address="127.0.0.1"
                redirectPort="443" />
the nginx configuration for the domain contains the section:
location / {
             proxy_pass <a href="http://127.0.0.1:8080;">http://127.0.0.1:8080;
             ...
             }
Do we have a configuration error? Is all data routed through Tomcat8 and
cannot be sent through an external web server?
We want a portal that provides the metadata as well as the actual data
and takes care of user rights, and we find the geonetwork software good
in this regard.

best regards

Viktor



_______________________________________________
GeoNetwork-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geonetwork-users
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork

_______________________________________________
GeoNetwork-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geonetwork-users
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork
Reply | Threaded
Open this post in threaded view
|

Re: Download large files

Andreas Böhme
Hi,
so that means, that it is not possible to provide download for files
bigger than 2 GB? Or is this just for the bulitin datastore?
Regards
Andreas

Am 25.02.2021 um 01:36 schrieb Ian Allen:

> Viktor,
>
> Geonetwork uses byte arrays to store the data before streaming it to the user.
>
> https://github.com/geonetwork/core-geonetwork/blob/182575ffe37282b18e1911f1a80011e8cfb451df/services/src/main/java/org/fao/geonet/api/records/attachments/AttachmentsApi.java#L261
>
> So I believe the limit should be around 2GB
>
>
> ________________________________
> From: Viktor Stroh <[hidden email]>
> Sent: February 24, 2021 11:26 AM
> To: [hidden email] <[hidden email]>
> Subject: [GeoNetwork-users] Download large files
>
> Hi guys,
> we are currently trying to solve the problems with uploading and
> downloading large data.
> Upload size could be set upwards: either as suggested in earlier
> versions in the source code under services in the
> config-spring-geonetwork.xml file
> or with new parameter api.params.maxUploadSize in config.properties in
> the newer versions.
> Now we have the problem to download larger files via geonetwork. When
> clicking on a link to a 7.3 GB file under Download and links, an error
> message comes up immediately:
> <code>runtime_exception</code>.
> <description>
> Handler dispatch failed; nested exception is java.lang.OutOfMemoryError:
> Required array size too large
> </description>
> <message>NestedServletException</message>.
> The memory parameters are currently:
> CATALINA_OPTS="-Xms5g -Xmx12g -XX:MaxDirectMemorySize=8g
> -Dgeonetwork.dir=/var/lib/gn_data"
> Tomcat8 can only be reached locally and uses NGinx Proxy for external
> communication via HTTPS protocol. The server.xml looks like this:
> <Connector port="8080" protocol="HTTP/1.1"
>                  connectionTimeout="20000"
>                  address="127.0.0.1"
>                  redirectPort="443" />
> the nginx configuration for the domain contains the section:
> location / {
>               proxy_pass <a href="http://127.0.0.1:8080;">http://127.0.0.1:8080;
>               ...
>               }
> Do we have a configuration error? Is all data routed through Tomcat8 and
> cannot be sent through an external web server?
> We want a portal that provides the metadata as well as the actual data
> and takes care of user rights, and we find the geonetwork software good
> in this regard.
>
> best regards
>
> Viktor
>
>
>
> _______________________________________________
> GeoNetwork-users mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/geonetwork-users
> GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork
>
> _______________________________________________
> GeoNetwork-users mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/geonetwork-users
> GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork
--
--------------------------------------------------------------------------------------------
                                               
Andreas Böhme
Techniker/Technician
Landschaftsökologie/Computational Landscape Ecology
                               
Helmholtz-Zentrum für Umweltforschung GmbH - UFZ
Helmholtz Centre for Environmental Research GmbH - UFZ
Permoserstraße 15, 04318 Leipzig, Germany
Phone +49 341 235 1058
[hidden email], www.ufz.de

Sitz der Gesellschaft/Registered Office: Leipzig
Registergericht/Registration Office: Amtsgericht Leipzig
Handelsregister Nr./Trade Register Nr.: B 4703
Vorsitzender des Aufsichtsrats/Chairman of the Supervisory Board: N.N.
Wissenschaftlicher Geschäftsführer/Scientific Managing Director:
Prof. Dr. Georg Teutsch
Administrative Geschäftsführerin/Administrative Managing Director:
Dr. Sabine König

Vermeiden Sie unnötige Ausdrucke./Think before printing.

Informationen aus der Wissenschaft (z.B. Veranstaltungshinweise, Projektangaben, Publikationen,
Bücher)

--------------------------------------------------------------------------------------------




_______________________________________________
GeoNetwork-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geonetwork-users
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork

smime.p7s (7K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Download large files

Francois Prunayre
Hi, also if you have a lot of files to distribute for download it may be
easier to rely on a data store where you can easily push large datasets to
publish the files and which may provide different access protocols.

For example, EEA provides datasets in various format using different
protocols (HTTP, Webdav, CIFS, FTPS depending on you network access) and
owncloud allow to download data easily eg.
https://sdi.eea.europa.eu/catalogue/srv/eng/catalog.search#/metadata/4b3a3319-4db3-4a33-b18d-2ba55b3fe2ce


Using the GeoNetwork datastore requires to ZIP format based on multiple
formats like shapefile and requires uploading all files to the internal
store.

Francois


Le jeu. 25 févr. 2021 à 11:49, Andreas Böhme <[hidden email]> a
écrit :

> Hi,
> so that means, that it is not possible to provide download for files
> bigger than 2 GB? Or is this just for the bulitin datastore?
> Regards
> Andreas
>
> Am 25.02.2021 um 01:36 schrieb Ian Allen:
> > Viktor,
> >
> > Geonetwork uses byte arrays to store the data before streaming it to the
> user.
> >
> >
> https://github.com/geonetwork/core-geonetwork/blob/182575ffe37282b18e1911f1a80011e8cfb451df/services/src/main/java/org/fao/geonet/api/records/attachments/AttachmentsApi.java#L261
> >
> > So I believe the limit should be around 2GB
> >
> >
> > ________________________________
> > From: Viktor Stroh <[hidden email]>
> > Sent: February 24, 2021 11:26 AM
> > To: [hidden email] <
> [hidden email]>
> > Subject: [GeoNetwork-users] Download large files
> >
> > Hi guys,
> > we are currently trying to solve the problems with uploading and
> > downloading large data.
> > Upload size could be set upwards: either as suggested in earlier
> > versions in the source code under services in the
> > config-spring-geonetwork.xml file
> > or with new parameter api.params.maxUploadSize in config.properties in
> > the newer versions.
> > Now we have the problem to download larger files via geonetwork. When
> > clicking on a link to a 7.3 GB file under Download and links, an error
> > message comes up immediately:
> > <code>runtime_exception</code>.
> > <description>
> > Handler dispatch failed; nested exception is java.lang.OutOfMemoryError:
> > Required array size too large
> > </description>
> > <message>NestedServletException</message>.
> > The memory parameters are currently:
> > CATALINA_OPTS="-Xms5g -Xmx12g -XX:MaxDirectMemorySize=8g
> > -Dgeonetwork.dir=/var/lib/gn_data"
> > Tomcat8 can only be reached locally and uses NGinx Proxy for external
> > communication via HTTPS protocol. The server.xml looks like this:
> > <Connector port="8080" protocol="HTTP/1.1"
> >                  connectionTimeout="20000"
> >                  address="127.0.0.1"
> >                  redirectPort="443" />
> > the nginx configuration for the domain contains the section:
> > location / {
> >               proxy_pass <a href="http://127.0.0.1:8080;">http://127.0.0.1:8080;
> >               ...
> >               }
> > Do we have a configuration error? Is all data routed through Tomcat8 and
> > cannot be sent through an external web server?
> > We want a portal that provides the metadata as well as the actual data
> > and takes care of user rights, and we find the geonetwork software good
> > in this regard.
> >
> > best regards
> >
> > Viktor
> >
> >
> >
> > _______________________________________________
> > GeoNetwork-users mailing list
> > [hidden email]
> > https://lists.sourceforge.net/lists/listinfo/geonetwork-users
> > GeoNetwork OpenSource is maintained at
> http://sourceforge.net/projects/geonetwork
> >
> > _______________________________________________
> > GeoNetwork-users mailing list
> > [hidden email]
> > https://lists.sourceforge.net/lists/listinfo/geonetwork-users
> > GeoNetwork OpenSource is maintained at
> http://sourceforge.net/projects/geonetwork
>
> --
>
> --------------------------------------------------------------------------------------------
>
> Andreas Böhme
> Techniker/Technician
> Landschaftsökologie/Computational Landscape Ecology
>
> Helmholtz-Zentrum für Umweltforschung GmbH - UFZ
> Helmholtz Centre for Environmental Research GmbH - UFZ
> Permoserstraße 15, 04318 Leipzig, Germany
> Phone +49 341 235 1058
> [hidden email], www.ufz.de
>
> Sitz der Gesellschaft/Registered Office: Leipzig
> Registergericht/Registration Office: Amtsgericht Leipzig
> Handelsregister Nr./Trade Register Nr.: B 4703
> Vorsitzender des Aufsichtsrats/Chairman of the Supervisory Board: N.N.
> Wissenschaftlicher Geschäftsführer/Scientific Managing Director:
> Prof. Dr. Georg Teutsch
> Administrative Geschäftsführerin/Administrative Managing Director:
> Dr. Sabine König
>
> Vermeiden Sie unnötige Ausdrucke./Think before printing.
>
> Informationen aus der Wissenschaft (z.B. Veranstaltungshinweise,
> Projektangaben, Publikationen,
> Bücher)
>
>
> --------------------------------------------------------------------------------------------
>
>
> _______________________________________________
> GeoNetwork-users mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/geonetwork-users
> GeoNetwork OpenSource is maintained at
> http://sourceforge.net/projects/geonetwork
>

_______________________________________________
GeoNetwork-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geonetwork-users
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork
Reply | Threaded
Open this post in threaded view
|

Re: Download large files

Viktor Stroh
In reply to this post by ianwallen
Hi Ian,
thank you. Good to know, that explains a lot.

> Viktor,
>
> Geonetwork uses byte arrays to store the data before streaming it to
> the user.
>
> https://github.com/geonetwork/core-geonetwork/blob/182575ffe37282b18e1911f1a80011e8cfb451df/services/src/main/java/org/fao/geonet/api/records/attachments/AttachmentsApi.java#L261
>
> So I believe the limit should be around 2GB


_______________________________________________
GeoNetwork-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geonetwork-users
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork
Reply | Threaded
Open this post in threaded view
|

Re: Download large files

Juan Luis Rodríguez Ponce
I've created this issue [1]  about this topic. It seems quite easy to fix
it and stream the result instead of copying the file content in memory.

[1] https://github.com/geonetwork/core-geonetwork/issues/5449

On Thu, Feb 25, 2021 at 3:36 PM Viktor Stroh <[hidden email]> wrote:

> Hi Ian,
> thank you. Good to know, that explains a lot.
>
> > Viktor,
> >
> > Geonetwork uses byte arrays to store the data before streaming it to
> > the user.
> >
> >
> https://github.com/geonetwork/core-geonetwork/blob/182575ffe37282b18e1911f1a80011e8cfb451df/services/src/main/java/org/fao/geonet/api/records/attachments/AttachmentsApi.java#L261
> >
> > So I believe the limit should be around 2GB
>
>
> _______________________________________________
> GeoNetwork-users mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/geonetwork-users
> GeoNetwork OpenSource is maintained at
> http://sourceforge.net/projects/geonetwork
>


--










*Vriendelijke groeten / Kind regards,Juan Luis Rodríguez.
<http://www.geocat.net/>Veenderweg 136721 WD BennekomThe NetherlandsT: +31
(0)318 416664 <+31318416664>Please consider the environment before printing
this email.*

_______________________________________________
GeoNetwork-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/geonetwork-users
GeoNetwork OpenSource is maintained at http://sourceforge.net/projects/geonetwork