FileNet Image Services ToolKit: Windows Shared Memory Conflict Errors (fnc_shmat failed err=487)
The SysV library maintains shared memory for FileNet Image Services ToolKit (ISTK) processes. Shared memory is allocated in segments. Image Services stores pointers in shared memory and this practice requires that each SHM segment be mapped at the same virtual memory address in each FileNet Image Services process.
Prior to 4.1.1 FP11 and 4.1.2 FP8, SysV mapped SHM segments at a given starting address then continued to map new segments at higher addresses until it encountered an address that was already in use (not free). The starting address was configurable, so users could examine a memory map dump of the process to determine the best place to start SHM (the largest area of free memory).
The start address is set as a hex value in the Windows Registry KEY:
If this KEY is not set, then a default address 0x45000000 is used. However, over time new DLL files (not part of Image Services) were linked in to FileNet Image Services processes and these occupied memory locations that conflicted with FileNet Image Services ToolKit SHM. Also, shared memory usage increased. These changes made it increasingly difficult to find a single free area of memory for SHM, causing FileNet Image Services ToolKit to crash. Sometimes a good starting address was found, only to have it no longer work due to more changes in the memory address environment.
As of 4.1.1 FP11 and 4.1.2 FP8 changes were made to the SysV method of determining shared memory addresses on Windows. The new procedure no longer requires all FileNet Image Services ToolKit SHM addresses to be located in one large free area of memory. Instead, all free addresses sufficiently large to contain a SHM segment are used. This allows for a greater number of segments and also provides more flexibility in mapping the segments.
The new SysV procedure examines the virtual memory map of every FileNet Image Services ToolKit process at FileNet Image Services ToolKit startup. A list of addresses that are free in every process is created and updated while FileNet Image Services ToolKit is up. When FileNet Image Services ToolKit is shut down, the new free address list is compared with the addresses used during the current cycle. If the list is different, then the new free addresses are saved in a file. When FileNet Image Services ToolKit is brought back up, this file is read and the new addresses are used. In this manner, changes in the address environment are automatically detected and are incorporated in the next cycle. If the list is different, then the new free addresses are saved in the file:
NOTE: The value of WAL_ROOT varies based on the ISTK installation directory.
When the FileNet Image Services ToolKit software is started, this file is read and the new addresses are used. Thus changes in the address environment are automatically detected and are incorporated in the next cycle. The addr_list file is created and maintained automatically by the FileNet Image Services ToolKit software.
NOTE: The current addresses in use for shared memory are displayed with the command: wal_ipc -A
The operation of the new shared memory address code is configurable through the use of five new trigger files.
NOTE: The following trigger files are NOT created automatically. They must be created manually if they are needed. The trigger files should ONLY be created at the direction of IBM if the installation of the Fix Pack does not resolve the original issue.
1. (WAL_ROOT)\NO_ADDR_LIST - If this file is created, then starting with the next recycle of FileNet Image Services SysV will NOT use the addr_list file and instead use the older address method explained above. This trigger file is useful if the new addr_list method encounters some problem.
Note: The software only checks for the existence of the trigger file NO_ADDR_LIST. If the file is created, it should be an empty file with no contents. If it does have any contents, the contents are ignored by the software.
2. (WAL_ROOT)\KEEP_ADDR_LIST- If this file is created, then SysV will stop creating new addr_list files when the need arises. Instead, it will always use the current addr_list file. This is useful if a system has a volatile memory layout thus causing frequent changes in the addr_list file. These frequent changes are avoided by creating the trigger file.
Note: The software only checks for the existence of the trigger file KEEP_ADDR_LIST. If the file is created, it should be an empty file with no contents. If it does have any contents, the contents are ignored by the software.
3. (WAL_ROOT)\REGEN_ADDR_LIST - By default the old addr_list file values are used as a base for the new software recycle. So once an addr_list file is created, the automatic operation of the software only removes address from the file, it does not add addresses . Over time new SHM conflicts may occur, and these conflicts remove addresses from the addr_list file. It may be desirable to continuously regenerate the base list of addresses for each recycle (rather than use the previous addr_list addresses). This could be advantageous in a volatile environment where big changes in addressing occur frequently. To regenerate the addr_list from scratch during each software recycle, create the REGEN_ADDR_LIST trigger. Note that a new addr_list file will automatically be created from scratch during the next recycle if the previous addr_list file is removed or renamed.
Note: The software only checks for the existence of the trigger file REGEN_ADDR_LIST. If the file is created, it should be an empty file with no contents. If it does have any contents, the contents are ignored by the software.
4. (WAL_ROOT)\LOW_ADDR - If this file is created, then SysV reads the file contents to obtain the HEX address of the lowest virtual memory address to avoid. On some systems the lower memory addresses may be volatile, causing the addr_list file to constantly change. By default SysV does not use address 0x04000000 or lower. The trigger file can be used to change this lower boundary.
For example, to set the lowest address to avoid at 0x11000000:
echo 11000000 > \fnsw_loc\sd\1\LOW_ADDR
5) \fnsw_loc\sd\1\HIGH_ADDR - If this file is created, then SysV reads the file contents to obtain the HEX address of the highest virtual memory address to avoid. On some systems the higher memory addresses may be volatile, causing the addr_list file to constantly change. By default SysV does not use address 0x70000000 or higher. The trigger file can be used to change this upper boundary.
For example, to set the highest address to avoid at 0x60000000:
echo 60000000 > \fnsw_loc\sd\1\HIGH_ADDR
Note: The status of SysV trigger files can be checked with the command:
wal_ipc -T | more
The trigger files can be created or removed by using the same command without the "pipe":
This command displays each trigger file and prompts you whether to create or remove it.
When shared memory conflicts occur errors are logged in the FileNet Image Services event log. Below is an example of the messages logged.
2010/04/09 13:20:51.063 202,0,24 <fnsw> HPII_start (4596.7584.0 0x11f4.1da0) ... [CRITICAL]
fnc_shmat failed for key=0x464a0000 from_catch_segv=0 err=487
2010/04/09 13:20:51.125 202,6,5 <fnsw> HPII_start (4596.7584.0 0x11f4.1da0) ... [INFO]
fn_NT_VMMap: saving virtual memory map in fnsw\client\logs directory.
2010/04/09 13:20:51.266 202,6,5 <fnsw> HPII_start (4596.7584.0 0x11f4.1da0) ... [INFO]
The Windows Registry may be updated to change the starting SysV shared memory address to the largest free area in memory (at address 0x40000000). A new registry edit script was created with the name: d:\fnsw\client\logs\shm_s_4596-7584.reg.txt To change the SysV shared memory address execute this script after completely shutting down all ISTK applications the script first remove the '.txt' extension then double-click the script from Windows explorer.
2010/04/09 13:20:51.281 202,0,2005 <fnsw> HPII_start (4596.7584.0 0x11f4.1da0) ... [CRITICAL]
SysV: Error 487 mapping file view. Process Aborting...
This is most likely due to a shared memory conflict.
2010/04/09 13:20:51.297 202,0,2005 <fnsw> HPII_start (4596.7584.0 0x11f4.1da0) ... [CRITICAL]
fnc_abort: The program encountered an irrecoverable error and aborted.
Resolving the problem
To resolve Windows shared memory conflicts, apply FileNet Image Services ToolKit 4.1.1 FP11. The Interim Fix for APAR PJ37757 can be applied for the FileNet Image Services ToolKit 4.1.2 release. The Interim Fix is included in FileNet Image Services ToolKit 4.1.2 FP8 scheduled for release in August 2010.
After applying FileNet Image Services ToolKit 4.1.1 FP11 or FileNet Image Services ToolKit 4.1.2 FP8, the ISTK application must be restarted at least two to three times so that all of the bad addresses are located and the contents of the addr_list file is stabilized.
The improvements replace the previous resolution which was to create a Windows Registry key named StartShmAddress and enter the starting Image Services shared memory address. It is not necessary to remove the key if it has already been implemented, as the key is now ignored.
More support for:
FileNet Image Services
Image Services ToolKit
Software version: 4.1.1, 4.1.2
Operating system(s): Windows
Software edition: All Editions
Reference #: 1447182
Modified date: 2010-12-09