FileNet Image Services: Troubleshooting Shared Memory Issues

Technote (FAQ)


Question

How do I configured shared memory for FileNet Image Services ?

Answer

How to Configure Shared Memory for Image Services

Shared Memory

Image Services software makes extensive use of shared memory (SHM). This is memory that is accessible to all Image Services processes. SHM is used to share data among Image Services processes – one process writes data to SHM and another process reads the data. The ipc_tool utility is used to monitor Image Services SHM.

Shared Memory Segments


    Shared memory is allocated in units called segments. The total shared memory used for Image Services is determined by two factors: the number and size of the segments. Both of these can be configured. Shared memory segments are “attached” to processes, so that the data saved in these segments can be “shared” across multiple processes.
Image Services Shared Memory Segment Types
    Image Services uses shared memory for the following:
    1. Process Log Segment (contains information on Image Services operation).
    2. Circular Buffer Debug Log Segment (optional segment only used when some kinds of debugging are enabled – by default this segment is not created).
    3. Address Manager Segment (contains list of addresses used by type #4 - the shared memory data segments).
    4. Shared Memory Data Segment (contain Image Services working data).
      Types 1 through 3 each use one single small segment and do not require special attention.

    The remainder of this document deals with type #4 – the data segment. Image Services allocates at least one data segment on the system. The Image Services subsystems (such as MKF, DOC, CSM, etc) store their working data in data segments.

    To display information on SHM segment usage run: ipc_tool –A
      The following is an example. The output can vary depending upon the hardware platform and release.
      tusker(fnsw)/fnsw/local/logs/elogs> ipc_tool -A

      Shared memory usage for Image Services:

      ipc_tool VERSION 4.1.1.1

      Segment size: 33554432 bytes (32 MB)

      Max shared memory configured for Image Services: 129 segments (4128 MB)

      Max shared memory allowed by the system: 51 segments (1632 MB)

      Address Manager segment:

      Address Shm id Creator
      0 0x00800000 6 Shared addrmgr

      Shared memory data segments:

      Address Shm id Creator Used Free
      1 0x02c00000 16777267 FileNet server 33525472 0
      2 0x05000000 16777268 FileNet server 33550592 0
      3 0x07400000 16777269 FileNet server 33550640 0
      4 0x09800000 16777270 FileNet server 20000440 13546912
      5 0x0bc00000 16777271 FileNet server 0 33554384
      6 0x0e000000 16777272 FileNet server 0 33554384
      7 0x10400000 16777273 FileNet server 0 33554384
      8 0x12800000 16777274 FileNet server 0 33554384
      9 0x14c00000 16777275 FileNet server 0 33554384
      10 0x17000000 16777276 FileNet server 0 33554384
      11 0x19400000 16777277 FileNet server 0 33554384
      12 0x1b800000 16777278 FileNet server 0 33554384
      13 0x1dc00000 16777279 FileNet server 0 33554384
      14 0x20000000 16777280 FileNet server 0 33554384

    The size of a data segment, the number of allocated segments, and the amount of free memory in the segments is shown.

    To summarize SHM usage by Image Services subsystem run: ipc_tool –t
      Abstract Total Avg Block Total Bytes
      -------- ----- --------- -----------
      ARM 3 5270.67 15812
      ASH 1 56.00 56
      BES 3 881.33 2644
      BESl 1 132.00 132
      . . .
      WQS 1 36.00 36
      WQSl 2 524.00 1048
      WRT 1 20.00 20

    The output determines which subsystems are using the most shared memory.

    To display detailed SHM subsystem usage run: ipc_tool –a

    Shared memory usage is broken down into the individual chunks used by the various subsystems.

Maximum Number of Shared Memory Data Segments

    The maximum number of shared memory data segments that can be used by Image Services is determined by a hard-coded value in the IS software and by limitations imposed by the operating system.

    AIX


      The AIX operating system architecture limits the number of shared memory segments that can be attached to a single process. This limitation is 11 segments. One of these segments is used for the process log, and another for the address manager, leaving 9 segments remaining. Other software libraries (such as a database) that link to Image Services may also create their own segments, so the segment count on AIX can become critical.

      Since the segment count on AIX is limited, the best way to allocate SHM is to optimize the segment size. See the next section on configuring segment size.

    Solaris, HP, and Windows

      These operating systems do not impose a practical limit on the number of segments. However, Image Services uses an internal table for shared memory – this table has a maximum value of 128 segments. If shared memory usage grows to approach this 128 limit, then the segment size can be increased thus lowering the need for so many segments (see the next section on configuring segment size).
Dynamic Allocation of Shared Memory Data Segments

    When necessary, Image Services automatically allocates a new SHM data segment. This dynamic segment allocation occurs when system activity increases thus requiring more shared memory. However, there is a certain amount of overhead encountered in the software when segments are dynamically allocated. At first none of the IS processes have attached to the new segment, and this causes problems when one of these processes tries to access the unattached segment. A fault is generated and handled by Image Services. The fault handler attaches the process to the new segment and the process resumes operation. On UNIX systems this can cause performance issues if there are a lot of new segments to attach. And on Windows systems some third-party software (e.g. JAVA) linked to IS processes can interfere with the fault handling thus preventing the process from attaching to the segment and causing the process to abort.

    NUM_SEG File

      The NUM_SEG file is provided to help with dynamic segment allocation problems.
        On UNIX: /fnsw/local/sd/1/NUM_SEG

        On Windows: \fnsw_loc\sd\1\NUM_SEG
      When Image Services is first installed, the NUM_SEG file does not yet exist. It is automatically created when IS allocates the first SHM data segment and it is updated whenever a new data segment is allocated. Image Services stores the number of data segments currently in use inside NUM_SEG, so the contents of this file represents the greatest number of SHM data segments that have been in use on the system.

      When Image Services is shut down all the SHM data segments are removed. New (empty) segments are created when Image Services restarts. At this time the NUM_SEG file is read to determine the number of data segments to allocate. These segments are created before any of the IS processes startup. This allows each process to immediately attach to all SHM data segments as the process starts, therefore avoiding the overhead and potential problems encountered by the fault-generated attach mechanism described previously.

    SHM_MB File

      The SHM_MB file is a counterpart to NUM_SEG. It is also created and maintained by Image Services, and represents the maximum allocated amount of shared memory data in MB. If a new segment is created, then the value in SHM_MB is increased by the segment size. This is done at the same time when NUM_SEG is updated.
        On UNIX: /fnsw/local/sd/1/SHM_MB

        On Windows: \fnsw_loc\sd\1\SHM_MB

      The SHM_MB file is only used as a safety check during the startup of FileNet Image Services to catch situations where either the segment size or number of segments has changed without making a corresponding adjustment to the other value. For example, if the segment size is doubled but NUM_SEG remains the same, then Image Services may have problems trying to allocate the total shared memory. The amount of shared memory allocated is limited to be less than or equal to SHM_MB.

    Configuring NUM_SEG

      As described above, the NUM_SEG file is automatically maintained by Image Services, and its main purpose is to pre-allocate the total amount of SHM necessary for Image Services. The allocated shared memory is divided into used and free bytes. It is recommended to periodically check how much of the allocated SHM is used vs. free. There should be sufficient free memory for SHM to expand if necessary without creating another new segment. If the allocated SHM is close to 100% used, then the NUM_SEG file should be manually incremented to increase free SHM.

      To see the current Image Services shared memory usage, run: ipc_tool –A (be sure to use an upper-case “A”).

      Examine the output under “Shared memory data segments:” - this section provides information on the total number of allocated segments. The “Used” and “Free” columns show the number of bytes of in-use and available shared memory for each allocated segment.

      If the combined “Free” value for all segments is small relative to the total allocated SHM, then NUM_SEG should be increased.

      There are no “hard” rules to determine whether NUM_SEG should be changed. If the segment size is small (e.g. 16 MB), then there should be no problem increasing NUM_SEG by one or two segments. But if the segment size is large (e.g. 256 MB), then increasing NUM_SEG will cause a lot more shared memory to be allocated, and this should only be done if there is very little Free memory available.

      WARNING: Setting NUM_SEG too high can prevent Image Services from starting up.

      To change NUM_SEG on UNIX systems perform the following steps:
        1. Shutdown Image Services: initfnsw –y stop
        2. Remove all IS resources and stop processes: killfnsw –ADy
        3. Remove the SHM_MB file: rm /fnsw/local/sd/1/SHM_MB
        4. Edit /fnsw/local/sd/1/NUM_SEG to increase its value
        5. Restart Image Services: initfnsw start

      To change NUM_SEG on Windows systems perform the following steps:
        1. Shutdown Image Services: initfnsw –y stop
        2. Remove all IS resources and stop processes: killfnsw –Dy
        3. Remove the SHM_MB file: del \fnsw_loc\sd\1\SHM_MB
        4. Edit \fnsw_loc\sd\1\NUM_SEG to increase its value
        5. Restart IS Control Service: net start IMSService
        6. If Image Services is not configured for auto-start: initfnsw start
      After restarting Image Services run “ipc_tool –A” to confirm that the new NUM_SEG value has taken effect.

      Note: If difficulties are encountered during this procedure, it is always possible to start over “from scratch” by removing both NUM_SEG and SHM_MB then initiating an Image Services recycle. This will reset both files to their minimum required values and allow them to increment automatically as Image Services allocates shared memory when needed.
Shared Memory Data Segment Size

    The size of a segment is configurable, and should be optimized to improve system performance. The optimal value for a segment size varies depending on the operating system.

    Setting segment size using fn_edit


      The fn_edit tool is used to configure the size of a shared memory segment. This is done “indirectly” by setting the value in the “Max Shared Memory” tab. This number represents the anticipated maximum shared memory needed for Image Services (in MB) but does not actually cause this amount of shared memory to be created. For example, setting “Max Shared Memory” to 2048 MB will not force Image Services to allocate this amount of SHM. Instead, this setting adjusts the shared memory data segment size to the minimum value needed in order to be able to achieve the desired max shared memory given the maximum allowed number of segments for the system.

      The maximum number of segments is divided into the Max Shared Memory value to calculate the segment size. The result is checked that it is not less than the 16 MB minimum size and that it does not exceed the maximum allowed (varies by operating system). The result is also adjusted to be a multiple of 4.

      For example, on an AIX system the maximum number of segments for Image Services is 10. Setting Max Shared Memory to 1024 MB yields a segment size 1024/10 = 102 MB. This is adjusted to 104 to be a multiple of 4.

      On Solaris, HP, or Windows platforms the maximum number of segments is 128. Setting Max Shared Memory to 1024 MB yields a segment size 1024/128 = 8 MB. Since this is lower than the minimum 16 MB, it is rejected and the minimum value used instead.

      The calculated segment size (in MB) is saved in the following file:
        UNIX: /fnsw/local/sd/1/FNSHMSEGSZ

        Windows: \fnsw_loc\sd\1\FNSHMSEGSZ

      The size is stored as an ASCII string, so the file contents can be easily examined. However, if the contents are changed, the fn_build program will restore the calculated value based on the fn_edit setting during the next recycle.

      WARNING: The NUM_SEG and SHM_MB files must either be removed or edited to compensate for the new segment size. For example, if the segment size is doubled then NUM_SEG must be half its previous value so that the total SHM is the same. If in doubt about the value for NUM_SEG, remove the file – it will be recreated by Image Services.
      The SHM_MB file must also be removed.

      For example, to set fn_edit Max Shared Memory on UNIX systems:
        1. Shutdown Image Services: initfnsw –y stop
        2. Remove all IS resources and stop processes: killfnsw –ADy
        3. Remove the SHM_MB file: rm /fnsw/local/sd/1/SHM_MB
        4. Manually edit or remove /fnsw/local/sd/1/NUM_SEG
        5. Run fn_edit to change Max Shared Memory
        6. Run fn_build -a
        7. Restart Image Services: initfnsw start

      For example, to set fn_edit Max Shared Memory on Windows systems:
        1. Shutdown Image Services: initfnsw –y stop
        2. Remove all IS resources and stop processes: killfnsw –Dy
        3. Remove the SHM_MB file: del \fnsw_loc\sd\1\SHM_MB
        4. Manually edit or remove \fnsw_loc\sd\1\NUM_SEG
        5. Run fn_edit to change Max Shared Memory
        6. Run fn_build -a
        7. Restart IS Control Service: net start IMSService
        8. If Image Services is not configured for auto-start: initfnsw start

      Verify the change by checking ipc_tool –A

    Setting segment size directly

      Sometimes it is desirable to directly set the segment size rather than “guess” at the value calculated by changing the fn_edit Max Shared Memory field. This is accomplished by creating a new file that contains the segment size:
        UNIX: /fnsw/local/sd/1/FNSHMSEGSZ2

        Windows: \fnsw_loc\sd\1\FNSHMSEGSZ2
      If this file exists, then its contents are read and used to set the segment size and the FNSHMSEGSZ file is ignored.

      WARNING: The NUM_SEG and SHM_MB files must either be removed or edited to compensate for the new segment size. For example, if the segment size is doubled then NUM_SEG must be half its previous value so that the total SHM is the same. If in doubt about the value for NUM_SEG, remove the file – it will be recreated by Image Services.
      The SHM_MB file must also be removed.

      Note: When setting the segment size, make sure the value is a multiple of 4.

      For example, to set the segment size to 32 MB on UNIX systems:
        1. Shutdown Image Services: initfnsw –y stop
        2. Remove all IS resources and stop processes: killfnsw –ADy
        3. Remove the SHM_MB file: rm /fnsw/local/sd/1/SHM_MB
        4. Manually edit or remove /fnsw/local/sd/1/NUM_SEG
        5. Set new segment size: echo 32 > /fnsw/local/sd/1/FNSHMSEGSZ2
        6. Restart Image Services: initfnsw start

      For example, to set the segment size to 32 MB on Windows systems:
        1. Shutdown Image Services: initfnsw –y stop
        2. Remove all IS resources and stop processes: killfnsw –Dy
        3. Remove the SHM_MB file: del \fnsw_loc\sd\1\SHM_MB
        4. Set new segment size: echo 32 > \fnsw_loc\sd\1\FNSHMSEGSZ2
        5. Manually edit or remove \fnsw_loc\sd\1\NUM_SEG
        6. Restart IS Control Service: net start IMSService
        7. If Image Services is not configured for auto-start: initfnsw start

      Verify the change by examining the output of ipc_tool –A

    Recommended segment sizes

      The minimum size for a data segment is 16 MB. The maximum allowed size varies depending on the operating system:
        Operating System Maximum SHM Segment Size
        AIX 256 MB
        Windows Varies with system
        Solaris 22 MB
        HP Varies with system
      The optimal segment size also varies based on the operating system:
          Operating System
        Recommended SHM Segment Size
        AIX 256 MB
        Windows 16 MB
        Solaris 22 MB
        HP 256 MB
      AIX divides memory into 256 MB partitions. Each shared memory segment occupies one partition. Memory usage is optimized if a segment is set to the same size as the memory partitions – 256 MB.

      Shared memory conflicts can occur on Windows systems. These conflicts occur because other third-party libraries that are linked to Image Services processes can occupy the same virtual memory addresses that are used for SHM. If segments are kept small – 16 MB – then they can more easily be scattered throughout the virtual memory address space and thus help avoid the conflicts. See the special Windows Shared Memory Addressing section below for more information.

      Testing on Solaris systems has revealed a correlation between the SHM segment size and the amount of local memory that can be allocated by a process. This testing indicated that the optimal balance between local and shared memory occurs when a shared memory segment is set to 22 MB.
        Note: 22 MB is not a multiple of 4, but it does work properly on the Solaris platform.

      Testing on HP systems has shown improved performance when segments are made large, such as 256 MB.
Windows Shared Memory Addressing

    The remainder of this document applies only to Windows systems.

    The allocation of shared memory on Windows systems is more complicated than on UNIX, and requires additional configuration. Image Services was designed to store virtual memory addresses within SHM data. For example, this allows placing linked lists in SHM. However, storing addresses in SHM forces a restriction: all processes must map (attach) a particular SHM data segment at the same virtual memory address. For example, if SHM segment #1 is mapped to virtual address 0x45000000 in one process, then it must be mapped at this same address in all other Image Services processes. This creates a problem because an address that is available for SHM in one process may not be available in another process – it might be used (occupied) by a third-party DLL, or for a thread stack, etc.

    The addresses that are available for Image Services SHM vary between processes and systems, and can change over time. For example, installing a new Windows update may include a new DLL file whose base address conflicts with one or more addresses previously used for SHM. This requires changing the Image Services memory layout to avoid the conflict. Shared memory allocation is very dynamic, and the mechanism to handle the addresses must be flexible.

    The errors resulting from a shared memory conflict can vary, but the following example is typical of the problem:
      2010/04/09 13:20:51.063 202,0,24 <fnsw> vwtool (4596.7584.0 0x11f4.1da0) ... [CRITICAL]
      fnc_shmat failed for key=0x464a0000 from_catch_segv=0 err=487

      2010/04/09 13:20:51.281 202,0,2005 <fnsw> vwtool (4596.7584.0 0x11f4.1da0) ... [CRITICAL]
      SysV: Error 487 mapping file view. Process Aborting...
      This is most likely due to a shared memory conflict.

    Image Services Shared Memory Addresses on Windows

      Image Services checks each of its processes to build a list of virtual addresses that can be used for shared memory. This list is stored in a file:
        \fnsw_loc\sd\1\addr_list

      The addresses are saved as ASCII text, so the file can be examined. The file is automatically created and updated by Image Services.

      Note: The addr_list feature was introduced in releases 4.1.1 FP 11 and 4.1.2 FP 8.

      The addr_list file is initially created by the first Image Services process that starts up after a new install. That process scans its virtual memory map for free areas large enough to contain a shared memory segment. The addresses for these areas are saved for use by other Image Services processes. All Image Services processes periodically check the list against their own memory usage to determine if each address is free in that process. If an address is not free, then it is removed from the list. The new, revised address list is saved in the addr_list file during a recycle (the previous version of the file is renamed so it is not lost).

      This method has the advantage of creating an address list that is uniquely tailored to each individual system and continually updated when necessary. However, when the addr_list file is first created the list may not yet be stable because it has not yet been modified by all Image Services processes to remove the in-use addresses.

      IMPORTANT: The system should be recycled multiple times after a new Image Services install or update. This will help stabilize the list of addresses used for shared memory.

    Minimizing shared memory conflicts on Windows

      As discussed previously, the allocation and use of memory on Windows systems is very dynamic and changes over time. It is possible that a new shared memory conflict will occur and this will cause Image Services to fail.

      When Image Services processes start up, they attach to all existing SHM segments as part of the process initialization before they begin doing any real work. Once the segments are all attached, the addresses used for the attaches are no longer free – these addresses are now “locked in” and no conflict can occur with them. It is thus advantageous to perform all SHM mapping as soon as possible at process startup.

      However, addresses that are not “locked in” may still get used for something else, such as loading a DLL or allocating a new thread stack, etc. If a new SHM data segment is created dynamically because of an increased need for SHM, then the address used to map the segment may not be available in all processes and a conflict will occur. This conflict is more serious, because it can happen in a process that is already doing work and cause the process to abort.

      The chance for this kind of conflict can be reduced by configuring SHM to have enough free space to handle an increase in performance without needing to dynamically allocate a new segment. Refer back to the section Configuring NUM_SEG for instructions on how to “pad” shared memory to have enough free bytes that a new segment will not be allocated (e.g. maintain at least two empty segments). Shared memory usage should be periodically monitored and adjusted to provide this ability to expand.

Rate this page:

(0 users)Average rating

Add comments

Document information


More support for:

FileNet Image Services
Image Services

Software version:

4.1.1, 4.1.2

Operating system(s):

AIX, HP Itanium, HP-UX, Solaris, Windows

Software edition:

All Editions

Reference #:

1458889

Modified date:

2011-01-04

Translate my page

Machine Translation

Content navigation