Readers/iterators open -> Java Heap issues

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Readers/iterators open -> Java Heap issues

Ben Jerrim
This post has NOT been accepted by the mailing list yet.
Hi all, we have a 2.11 goeserver backend running on tomcat 8.5 using the oracle db connector to serve most of our data from 11g. We see our memory usage climb pretty quickly (a minute or two) to consume basically all of it and are seeing the following in the logs a lot.

2017-08-16 01:42:10,461 WARN [geotools.jdbc] - There is code leaving feature readers/iterators open, this is leaking statements and connections!
2017-08-16 01:42:13,691 ERROR [geoserver.ows] -java.lang.OutOfMemoryError: Java heap space


Currently the Catalina Options are set as:
-Xms2048m -Xmx5120m \
-Djava.awt.headless=true \
-XX:+UseParNewGC \
-XX:+CMSClassUnloadingEnabled \
-XX:SoftRefLRUPolicyMSPerMB=36000 \
-verbose:gc \
-XX:+PrintTenuringDistribution \
-XX:+HeapDumpOnOutOfMemoryError


We have 4-6 geoservers load balanced via aws elastic beanstalk each on an m4.large. Request volumes are around 1k/minute. We seed gwc for quite a number of layers but are still having stability issues which i think come from the memory errors. We use control flow to manage request volumes and lab tests always look positive, yet in the wild we still see instances failing.

Any ideas on how i can improve the stability and start to understand what's killing the instances?
how do i troubleshoot the jdbc issues?


Reply | Threaded
Open this post in threaded view
|

Re: Readers/iterators open -> Java Heap issues

Ben Jerrim
This post has NOT been accepted by the mailing list yet.
Played with the procedure cache setting today and found that if i disable it (-1) things seem to improve a lot. More testing required but it looks promising.

Analysis of the heap dump pointed me in this direction.


Anyone experienced this?
Reply | Threaded
Open this post in threaded view
|

Re: Readers/iterators open -> Java Heap issues

Ben Jerrim
This post has NOT been accepted by the mailing list yet.
just coming back to close the loop on this and confirm that disabling the Oracle connection database cache solved my stability issues. This change has been in production for 4 days now and we've not had a single memory issue. phew!