I'm running GeoServer 2.1.0 with Tomcat 6 and Java 1.6.0_21-b06 on a CentOS 5.5 server running in the Amazon cloud, and I have eight ImageMosaic layers configured in GeoServer. From time to time, Tomcat crashes, and I see numerous "too many files open" exceptions in Tomcat's catalina.out log file, despite minimal traffic. The eight raster layers contain less than 10,000 tiles in total; ulimit is set to 1024; and cat /proc/sys/fs/file-max returns 762,674. The server has 180 GB of free disk space and 8 GB of RAM.
I've tried increasing ulimit to unlimited, but that causes Tomcat to take 10 minutes to start (not ideal in a development environment). Oddly enough, if I then reset ulimit to 1024, Tomcat still takes 10 minutes to start, even if I reboot the server. I've tried listing the open files (lsof | wc -l) during minimal traffic, but I never see more than 2,500 open files.
(I should mention that I'm getting the same issue with GeoServer 2.1 RC1 and Tomcat 5 on CentOS 5.5.)
At this point, I'm thoroughly puzzled why I'm getting this exception. Can anyone suggest a solution?