ImagePyramid "sorry too many clients"

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

ImagePyramid "sorry too many clients"

Arnaud L.
Hi list,
I have an imagepyramid made of of 16 imagemosaic layers.
The imagemosaic indexes are stored in a postgis database.
I've had connection exhaustion problems (i.e. "sorry, too many clients"
from the PostgreSQL backend).

My guess is that even though the imagepyramid has its own
datastore.properties file, each imagemosaic layer uses the "max\
connections" parameter from its own datastore.properties file. So for 16
imagemosaic layers with the default max connections setting (10), we
could go as high as 160 idle open connections.

Is this assumption correct ? If I am right, then the only way to keep
these connections at a reasonable level would be to set the max
connections per imagemosaic to something like 4 or 5.

Thanks for your advices on this !


Cheers !

--
Arnaud


_______________________________________________
Geoserver-users mailing list

Please make sure you read the following two resources before posting to this list:
- Earning your support instead of buying it, but Ian Turton: http://www.ianturton.com/talks/foss4g.html#/
- The GeoServer user list posting guidelines: http://geoserver.org/comm/userlist-guidelines.html

If you want to request a feature or an improvement, also see this: https://github.com/geoserver/geoserver/wiki/Successfully-requesting-and-integrating-new-features-and-improvements-in-GeoServer


[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-users
Reply | Threaded
Open this post in threaded view
|

Re: ImagePyramid "sorry too many clients"

geowolf
Hi Arnaud,
for that case you need to "step up" your installation and have everything use the
same JNDI connection pool (same connection pool for all stores), see also:


Cheers
Andrea


On Fri, Feb 1, 2019 at 10:01 AM Arnaud L. <[hidden email]> wrote:
Hi list,
I have an imagepyramid made of of 16 imagemosaic layers.
The imagemosaic indexes are stored in a postgis database.
I've had connection exhaustion problems (i.e. "sorry, too many clients"
from the PostgreSQL backend).

My guess is that even though the imagepyramid has its own
datastore.properties file, each imagemosaic layer uses the "max\
connections" parameter from its own datastore.properties file. So for 16
imagemosaic layers with the default max connections setting (10), we
could go as high as 160 idle open connections.

Is this assumption correct ? If I am right, then the only way to keep
these connections at a reasonable level would be to set the max
connections per imagemosaic to something like 4 or 5.

Thanks for your advices on this !


Cheers !

--
Arnaud


_______________________________________________
Geoserver-users mailing list

Please make sure you read the following two resources before posting to this list:
- Earning your support instead of buying it, but Ian Turton: http://www.ianturton.com/talks/foss4g.html#/
- The GeoServer user list posting guidelines: http://geoserver.org/comm/userlist-guidelines.html

If you want to request a feature or an improvement, also see this: https://github.com/geoserver/geoserver/wiki/Successfully-requesting-and-integrating-new-features-and-improvements-in-GeoServer


[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-users


--

Regards, Andrea Aime == GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it ------------------------------------------------------- Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia. This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.



_______________________________________________
Geoserver-users mailing list

Please make sure you read the following two resources before posting to this list:
- Earning your support instead of buying it, but Ian Turton: http://www.ianturton.com/talks/foss4g.html#/
- The GeoServer user list posting guidelines: http://geoserver.org/comm/userlist-guidelines.html

If you want to request a feature or an improvement, also see this: https://github.com/geoserver/geoserver/wiki/Successfully-requesting-and-integrating-new-features-and-improvements-in-GeoServer


[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-users
Reply | Threaded
Open this post in threaded view
|

Re: ImagePyramid "sorry too many clients"

Arnaud L.
Hi Andrea

Le 05/02/2019 à 12:14, Andrea Aime a écrit :
> for that case you need to "step up" your installation and have
> everything use the
> same JNDI connection pool (same connection pool for all stores), see also:
> https://docs.geoserver.org/stable/en/user/tutorials/tomcat-jndi/tomcat-jndi.html

That looks like a good option.
For the time being I was thinking about configuring a pgBouncer service
(i.e. connection pooling middleware), which I guess would achieve the
same result.

But I'll look into JNDI first if that can spare me an additional
software to maintain. Thanks for the suggestion !

Cheers
--
Arnaud


_______________________________________________
Geoserver-users mailing list

Please make sure you read the following two resources before posting to this list:
- Earning your support instead of buying it, but Ian Turton: http://www.ianturton.com/talks/foss4g.html#/
- The GeoServer user list posting guidelines: http://geoserver.org/comm/userlist-guidelines.html

If you want to request a feature or an improvement, also see this: https://github.com/geoserver/geoserver/wiki/Successfully-requesting-and-integrating-new-features-and-improvements-in-GeoServer


[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-users
Reply | Threaded
Open this post in threaded view
|

Re: ImagePyramid "sorry too many clients"

Arnaud L.
In reply to this post by geowolf
Le 05/02/2019 à 12:14, Andrea Aime a écrit :
> for that case you need to "step up" your installation and have
> everything use the
> same JNDI connection pool (same connection pool for all stores), see also:
>
> https://docs.geoserver.org/stable/en/user/tutorials/tomcat-jndi/tomcat-jndi.html

JNDI works like a charm ! Thanks !

Just a note, I don't know whether this is a bug or something missing in
the documentation, or maybe even something I have overlooked :
the "schema" parameter in the Datastore Configuration page does not seem
to have any effect.
I've set it to the name of my postgresql's database schema, but the log
was flooded with geotools.jdbc trying to read the primary keys of all
the tables in my database (and not in my schema alone). I had not
checked "expose primary keys".
More problematic, geoserver that the relation containing my mosaic index
did not exist.

In the datastore.properties of each mosaic (and of my imagepyramid) I
added "schema=myschema", and everything worked perfectly afterwards.
Is the the expected behaviour ? Should I propose a quick patch in the
jndi configuration doc ?

Also for my datastore.properties file, I had followed the guide from the
"geoserver training" section of your website :
https://geoserver.geo-solutions.it/edu/en/multidim/imagemosaic/mosaic_datastore.html#using-a-jndi-connection-pool
which did not mention this "schema" parameter, so mabe I could add this
somewhere ?

Cheers
--
Arnaud


_______________________________________________
Geoserver-users mailing list

Please make sure you read the following two resources before posting to this list:
- Earning your support instead of buying it, but Ian Turton: http://www.ianturton.com/talks/foss4g.html#/
- The GeoServer user list posting guidelines: http://geoserver.org/comm/userlist-guidelines.html

If you want to request a feature or an improvement, also see this: https://github.com/geoserver/geoserver/wiki/Successfully-requesting-and-integrating-new-features-and-improvements-in-GeoServer


[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-users
Reply | Threaded
Open this post in threaded view
|

Re: ImagePyramid "sorry too many clients"

geowolf
On Tue, Feb 5, 2019 at 2:12 PM Arnaud L. <[hidden email]> wrote:
Just a note, I don't know whether this is a bug or something missing in
the documentation, or maybe even something I have overlooked :
the "schema" parameter in the Datastore Configuration page does not seem
to have any effect.

Hum... the schema in the datastore configuration should affect only the feature
types served by the datastore itself.
Are you expecting it to affect the mosaics somehow? Did you setup any connection
between the datastore config and the mosaic?
 
In the datastore.properties of each mosaic (and of my imagepyramid) I
added "schema=myschema", and everything worked perfectly afterwards.
Is the the expected behaviour ? Should I propose a quick patch in the
jndi configuration doc ?

I think it's the expected behavior, but without seeing if there is any connection
between the mosaic and the store page in the UI I cannot comment further
 
Also for my datastore.properties file, I had followed the guide from the
"geoserver training" section of your website :
https://geoserver.geo-solutions.it/edu/en/multidim/imagemosaic/mosaic_datastore.html#using-a-jndi-connection-pool
which did not mention this "schema" parameter, so mabe I could add this
somewhere ?

I'm afraid not, while the generated output of the training material is under a creative commons license,
the source code of it is in a private repo. But it's a good suggestion, I've opened a ticket to remind
us a fix the next time we work on the material.

Cheers
Andrea

==

GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it ------------------------------------------------------- Con riferimento alla normativa sul trattamento dei dati personali (Reg. UE 2016/679 - Regolamento generale sulla protezione dei dati “GDPR”), si precisa che ogni circostanza inerente alla presente email (il suo contenuto, gli eventuali allegati, etc.) è un dato la cui conoscenza è riservata al/i solo/i destinatario/i indicati dallo scrivente. Se il messaggio Le è giunto per errore, è tenuta/o a cancellarlo, ogni altra operazione è illecita. Le sarei comunque grato se potesse darmene notizia. This email is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. We remind that - as provided by European Regulation 2016/679 “GDPR” - copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this email by mistake, please notify us immediately by telephone or e-mail.



_______________________________________________
Geoserver-users mailing list

Please make sure you read the following two resources before posting to this list:
- Earning your support instead of buying it, but Ian Turton: http://www.ianturton.com/talks/foss4g.html#/
- The GeoServer user list posting guidelines: http://geoserver.org/comm/userlist-guidelines.html

If you want to request a feature or an improvement, also see this: https://github.com/geoserver/geoserver/wiki/Successfully-requesting-and-integrating-new-features-and-improvements-in-GeoServer


[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-users
Reply | Threaded
Open this post in threaded view
|

Re: ImagePyramid "sorry too many clients"

Arnaud L.
Hi Andrea

Le 07/02/2019 à 10:09, Andrea Aime a écrit :
> Hum... the schema in the datastore configuration should affect only the
> feature types served by the datastore itself.
> Are you expecting it to affect the mosaics somehow?

No, sorry, I got lost in the process. Now I believe I understand.
For some reason I thought that the JNDI datastore would be used by the
imagemosaic.
But the imagemosaic directely references the JNDI connection, not the
JNDI datastore. Makes sense.


> Did you setup any connection between the datastore config and the mosaic?

No, that's what I thought was happening when I set up the
jndiReferenceName in the datastore.properties file to be the same as the
jndiReferenceName in the JNDI datastore configuration. Now I understand
that both refernce the jndiReferenceName defined in the context.xml of
my tomcat server. Totally logical, but as you can see I'm not very
knowledgeable in that matter.


> I'm afraid not, while the generated output of the training material is under a creative commons license, the source code of it is in a private repo. But it's a good suggestion, I've opened a ticket to remind us a fix the next time we work on the material.

OK, thanks for your help Andrea.

Cheers !

--
Arnaud


_______________________________________________
Geoserver-users mailing list

Please make sure you read the following two resources before posting to this list:
- Earning your support instead of buying it, but Ian Turton: http://www.ianturton.com/talks/foss4g.html#/
- The GeoServer user list posting guidelines: http://geoserver.org/comm/userlist-guidelines.html

If you want to request a feature or an improvement, also see this: https://github.com/geoserver/geoserver/wiki/Successfully-requesting-and-integrating-new-features-and-improvements-in-GeoServer


[hidden email]
https://lists.sourceforge.net/lists/listinfo/geoserver-users