Jan Schulz

Our Domino Mailserver crashed or better became unresponsive with the server task at 100%CPU. The log showed during that time:

• Unable to access notes.ini. Determine what application or hardware fault is preventing access. Previous cache values used.
• Error updating local ID file: Could not open the ID file

Also, there were a few "Too many open Files" Error in DDM (mostly fulltext index files, sometimes also databases) which I at first didn't relate to this. It started after the 8.0.2 update in fall and got worse after that. A restart of the domino "fixed" the problem, but after the xmas holidays we had to do this every day :-(

The Server was a ubuntu linux (latest LTS-release). The first level IBM Support couldn't help and the second level Support didn't want to support us as we are running an unsuported OS :-(

I turns out, that the problem was a too low "ulimit -n" of 1024. This limits the open files of processes and can be set via PAM configuration files. Our domino server process has about 700 open DBs + other files in normal mode:

# lsof |grep " lotus " |grep "^server" |wc -l
1135

I suspect that the IO changes in 8.0.x and also that we had to add a few new databases during the start of term (we are a student organsation and we get new member during that time) were the reason why we never saw that before and also didn't see this on a different domino on the same Server, but without so many mailfiles.

The changes to fix it were:
/etc/security/limits.conf
+ lotus            hard    nofile          4096
+ lotus            soft    nofile          4096
also add this line in /etc/pam.d/su:
+ session    required   pam_limits.so

The last change is because we use the rcdomino startscript, which uses su to start the domino server.

After that: no more hangs...

Anyway: I still suspect a bug, as the domino should not hang without an error if that problem occures. Lets see if the report in our PMR gets a reaction :-)

Another question: what is this limit on RedHat or Suse systems? We are not really big, so othere should have run into this much earlier if the limit wouldn't be higher... Or it really is an ubuntu problem...