Microsoft Windows 2008 64-bit servers sometimes have significantly increased CPU usage and I/O degradation caused by the Operating System (OS) when Lotus Domino opens or backs up large numbers of databases.
Working with Microsoft the following was ascertained:
After Domino opens many NSF files in quick succession during a backup, the Virtual Address Space of the OS system cache may be completely used up (for example, 1TB of data may be used in this OS cache). Successive calls into the OS cache manager to get memory from the OS system cache then results in mapping/unmapping of views from the system cache. These mapping/unmapping operations takes a lot of CPU time and as a result shows as high OS CPU usage. In addition, because the large OS system cache may now reside on the disk (the RAM is not large enough to hold the OS system cache) this results in significant I/O on the system.
Microsoft Windows 2008
Microsoft Windows 2008 R2
Diagnosing the problem
Programs and data can reside in the OS system cache even after they have been closed down. The OS looks in the system cache when opening a program or file. But this cache can result in performance problems with Domino. For performance reasons, Domino opens databases with a RANDOM flag (FILE_FLAG_RANDOM_ACCESS ). In Windows 2008 64-bit, this flag causes file blocks that are read to stay in the cache until the file is closed. Because Domino keeps files open in the Database Cache (dbcache) for performance reasons, it takes quite a long time until the cache is released. During a backup of a larger Domino server, the file-system cache can grow dramatically and cause performance issues.
Here is a general guide to check if this problem is happening:
- Check if the OS system cache is increasingly being used in line with CPU and I/O degradation, for example, you can use Windows perfmon or Windows task manager to see cache usage:
If there is a CPU impact (this will not be experienced in all cases) you would see the "% CPU Usage" for the "System" process to be high when using Windows Perfmon:
Resolving the problem
This issue is now fixed in Lotus Notes and Domino 8.5.2 FP3 and 8.5.3.
Fix details: SPR# KBRN8AKKA9
Refer to the Upgrade Central site for details on upgrading Notes/Domino.
This issue is caused by a change Microsoft made to the Windows 2008 file system cache. The fix in SPR# KBRN8AKKA9 disables the way in which Domino opens files (not using the RANDOM flag) and; therefore, preventing the OS cache from growing too large.