A JVM heap out-of-memory error can be caused by a single operation on the server that fetches too many objects into one MBO set. Such an error can cause all users to be disconnected from the server.
With the 126.96.36.199 base services fix pack, you can configure system properties in the System Properties application to detect this situation and stop the problematic operation before it crashes the server.
You can use the fetch stop limit properties to set an upper limit on the number of objects that can be fetched from the database and constructed on the server into a single set. You can set different limits for different types of objects.
When the upper limit is reached, an exception is thrown so that the process is stopped. The error message or the log trace explains that the fetch limit was reached and the operation was stopped. The user can then either reduce the number of objects by using filters or otherwise change the operation to prevent excessive fetching of objects from the database.
The fetch limit stop is enabled by default with a limit of 5000 for all objects.
If the fetch limit is reached and the exception is thrown during a user interface operation, the error message is typically displayed to the user. Depending on how the outer logic is implemented, it is possible that the error message is wrapped by another message or not displayed. However, the error is always logged on the server in all cases.
Summary of fetch stop limit properties
|mxe.db.fetchStopLimitEnabled||Enables or disables checking of the fetch stop limit.||0 to disable, 1 to enable. Default is 1.|
|mxe.db.fetchResultStopLimit||The fetch stop limit used when checking of the fetch stop limit is enabled. The limit applies to all objects for which a specific fetch stop limit property is not specified.||-1 means no limit. 5000 is the default.|
|mxe.db.fetchResultStopLimit.OBJECTNAME||The fetch stop limit for the specified object. If this property exists, its value is used as the fetch stop limit for this object only. To disable the fetch stop limit for a specific object, create this property and sets its value to -1.||Any integer value.
-1 means no fetch stop limit for this object.
|mxe.db.fetchStopExclusion||A comma separated list of object names. If an object name is in the list, the fetch stop limit check is disabled for the object. If an object name is in the list and the same object is specified in an mxe.db.fetchResultStopLimit.OBJECTNAME property, the exclusion overrides the other property.||Comma separated object name list, as they appear in the MAXOBJECT table.|
Changes to these property values take effect immediately.
If a mxe.db.fetchResultStopLimit. OBJECTNAME property does not exist, you can a add the property. If the property exists and you add the property, you receive the following error message:
BMXAA4129E - Record already exists for Property Name=mxe.db.fetchResultStopLimit.<OBJECTNAME>
In this case, locate the existing property and modify the property value.
Configuring the fetch stop limit
The overall limit and the limit for specific objects needs to suit your typical use of the system. The purpose is to stop only operations whose large number of fetched objects can cause JVM out-of-memory errors.
You can use the property mxe.db.fetchResultLogLimit to determine the typical or largest fetch count for every object. The log file contains a stack trace every time a multiple of the fetchLogResultLimit is reached. For example, if you set the property to 1000, the stack trace is logged at the 1000th record loaded into memory, the 2000th record, and so on.
If you are not sure if the default fetch limit will be high enough, set the mxe.db.fetchResultLogLimit property to 5000 or the value you plan to use. This will cause warning messages to be printed in the Maximo log when any fetch more than 5000 is detected. The text in the log is similar to the following:
17 Sep 2009 13:33:38:172 [INFO] BMXAA6702I - -------FetchResultLogLimit logging
17 Sep 2009 13:33:38:172 [INFO] BMXAA6703I - User Name : WILSON
17 Sep 2009 13:33:38:172 [INFO] BMXAA6705I - Query : select * from someobject order by description
17 Sep 2009 13:33:38:172 [INFO] BMXAA6706I - MboSet Name : SOMEOBJECT Reference
= SomeObjectSet[UnicastServerRef [liveRef: [endpoint:[188.8.131.52:2451](local),objID:]]]
17 Sep 2009 13:33:38:172 [INFO] BMXAA6707I - MboSet Size : 50000
17 Sep 2009 13:33:38:172 [INFO] BMXAA6708I - Fetch count so far : 5000
17 Sep 2009 13:33:38:172 [INFO] psdi.util.MXSystemException: BMXAA6709I - Printing StackTrace: java.lang.Exception
17 Sep 2009 13:33:38:172 [INFO] BMXAA6710I - -------FetchResultLogLimit logging
Monitor your log files and catch any of these warnings. Your log files are in the location you specify in the Logging application. When you see a number that exceeds your planned limit, you need to identify if they are legitimate operations or if the operations need to be prevented. This helps you to determine what your fetch stop limit should be.
If you have a cron job that fetches more objects than you would accept, you can set a different fetch limit for the cron task server. All of the related properties can be set globally or specific to a server.
If you have customizations in your system, avoid fetching all objects in one large set. Explore if discardable MBOs is an option. If you cannot avoid that, make sure the limit set for the object is sufficient for the operation to occur.
If a customization requires a high fetch number, but you do not want other user operations to be capped by this high number, you can call the following method on the MboSet for which the large fetch needs to occur: MboSet.setLogLargFetchResultDisabled(true)
What to do when the fetch limit stop error occurs
Large fetch counts are associated with two typical scenarios:
- The user applies a particular action to a large result set. The result set is retrieved without sufficient filtering. This could occur when a user inadvertently invokes a list page action or when a legitimate operation, due to its data distribution, exceeds the preset server capacity.
- An operation embedded in a user action retrieves a large set because of how the data is distributed. The user does not have direct control of the data retrieval and cannot adjust the operation by filtering the records.
In either scenario, the user needs to analyze the data and identify the operation that causes the error. If the error cannot be corrected by filtering the data or by splitting the target of the action into smaller batches, the administrator needs to be informed and some of the following system adjustments need to be made:
- Increase the upper limit for the particular object if it is safe to use a higher limit.
- Correct the data to avoid processing of such a large data set.
- Implement the use of discardable MBO sets to prevent all the objects being fetched into memory. A discardable MBO set is a read-only MBO set for which only forward fetching is allowed.
- Move the operation into a cron task and schedule the cron task to run at a low-demand time or on another server.
- Modify the process to alleviate the need to fetch such a large number of objects in memory.
|Systems and Asset Management||Maximo Asset Management Essentials|
|Systems and Asset Management||Tivoli Asset Management for IT|
|Systems and Asset Management||Tivoli Change and Configuration Management Database|
|Systems and Asset Management||Tivoli Service Request Manager|