Question & Answer
Question
What information should be collected when I experience a problem with the IBM FileNet P8 Process Engine 5.0?
Answer
Must Gather for FileNet Process Engine 5.0
Collecting data before calling IBM support will help you understand the problem and save time analyzing the data.
Collecting data early, before opening the PMR, helps IBM® Support quickly determine if:
- Symptoms match known problems (rediscovery).
- There is a non-defect problem that can be identified and resolved.
- There is a defect that identifies a workaround to reduce severity.
Locating the root cause of a problem can speed development of a code fix.
Note that there are many significant differences between the new PE 5.x software and the older PE 4.x software. This technote describes how to collect information for the PE 5.0 release. The Must Gather for PE 4.5.1 and older releases is here:
http://www.ibm.com/support/docview.wss?uid=swg21327304.
This document describes the information that should be collected for Process Engine problems. You can collect this information "by hand," using the information described here. Or, you can use the automated IBM FileNet Process Engine 5.0 Data Collector. Using the Process Engine 5.0 ISA Data Collector is significantly easier than gathering the information by hand.
The use of the automated IBM FileNet Process Engine 5.0 Data Collector is highly encouraged. If you use the IBM FileNet Process Engine 5.0 data collector, you would only have to be concerned about the steps described under the "Collecting Information Specific to the Problem" heading. For example, if you can reproduce the problem, you might enable the appropriate traces and replicate the problem and then run the automated data collector. The data collector would then gather those traces along with all of the other basic information described below.
The IBM FileNet Process Engine 5.0 Data Collector is described here:
http://www.ibm.com/support/docview.wss?uid=swg21454222.
Gathering General Information
This information is needed, but is generally added to the PMR or the defect:
- How often does this problem occur?
- Can the problem be reproduced?
- How many users are affected by this problem?
- What is the business impact of this problem?
- Is there a workaround?
Gathering Operating System Versions, Settings and Logs
All of the information in this section is collected automatically if you use the IBM FileNet Process Engine 5.0 Data Collector.
These commands collect:
- OS Version
- OS Environment Variables
- OS Kernel parameters
- OS running processes
- OS networking information
- OS locale information
- OS log files
Run the following commands on the PE Server and collect the output:
AIX Servers
Command | Collects |
uname -a or oslevel -s | OS Version |
/usr/sbin/instfix -ivq | OS Patches or Maintenance level |
env | OS Environment Variables |
errpt -a | OS Configuration & Error Reporting |
/usr/sbin/lsattr -E -l sys0 | OS Kernel parameters |
ps -ef | currently running processes |
netstat -an | socket information |
no -a | grep udp_ephemeral | ephemeral port info |
locale locale -a | locale information |
/var/adm/syslog/syslog.log | include system log files |
ls -RalF of the ProcessEngine directory | directory listing of Process Engine files |
HP Servers
Command | Collects |
uname -a | OS Version |
/usr/sbin/swlist -l product | OS Patches or Maintenance level |
env | OS Environment Variables |
/usr/sbin/dmesg | OS Configuration & Error Reporting |
/usr/sbin/sysdef | OS Kernel parameters |
ps -ef | currently running processes |
netstat -an | socket information |
ndd /dev/udp udp_smallest_anon_port ndd /dev/udp udp_largest_anon_port | ephemeral port information |
locale locale -a | locale information |
/var/adm/syslog/syslog.log | include system log files |
ls -RalF of the ProcessEngine directory | directory listing of Process Engine files |
Solaris Servers
Command | Collects |
uname -a | OS Version |
/usr/bin/showrev -a | OS Patches or Maintenance level |
env | OS Environment Variables |
/usr/sbin/prtdiag -v | OS Configuration & Error Reporting |
/usr/sbin/sysdef -D /usr/bin/getconf -a | OS Kernel parameters |
ps -ef | currently running processes |
netstat -an | socket information |
/usr/sbin/ndd /dev/udp udp_smallest_anon_port /usr/sbin/ndd /dev/udp udp_largest_anon_port /usr/contrib/bin/nettune -l | ephemeral port information |
locale locale -a | locale information |
/var/adm/messages & /var/log/syslog | include system log files |
ls -RalF of the ProcessEngine directory | directory listing of Process Engine files |
Windows Servers
Command | Collects |
systeminfo | OS Version, Patches and Maintenance level |
tasklist | currently running processes |
WMIC /OUTPUT:C:\ProcessList.txt PROCESS get Caption,Processid,Usermodetime,Commandline | current running processes including the command line |
set | OS Environment Variables |
Windows 2003: cscript C:\windows\system32\eventquery.vbs /L application /R 1000 /V cscript C:\windows\system32\eventquery.vbs /L system /R 1000 /V Windows 2008: wevtutil epl System Systemlog.evtx wevtutil epl Application Applicationlog.evtx | OS Event Logs |
netstat -an | socket information |
chcp | display code page |
dir /s of the ProcessEngine directory | directory listing of Process Engine files |
the following registry keys from SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\ MaxUserPort TcpMaxConnectTransmissions TcpMaxConnectRetransmissions | networking registry settings |
Linux Servers
Command | Collects |
uname -a | OS Version |
/usr/bin/showrev -a | OS Patches or Maintenance level |
env | OS Environment Variables |
/usr/sbin/prtdiag -v | OS Configuration & Error Reporting |
/usr/sbin/sysdef -D /usr/bin/getconf -a | OS Kernel parameters |
cat /proc/sys/net/ipv4/ip_local_port_range | ephemeral port information |
/usr/bin/getconf -a | configuration info |
/bin/dmesg | messages |
ps -ef | currently running processes |
netstat -an | socket information |
locale locale -a | locale information |
/var/log/messages.* | include system log files |
ls -RalF of the ProcessEngine directory | directory listing of Process Engine files |
Collecting PE Server Version and Configuration Information
All of the information in this section is collected automatically if you use the IBM FileNet Process Engine 5.0 Data Collector.
For the following commands, it is assumed that you're collecting information for the Default instance of the Process Engine. If you have multiple instances of the PE on the same server, you will need to specify the particular instance in place of the Default instance. That is, you would replace the word Default with the name of the desired PE virtual server in the commands below.
PE's CE Connection Information:
PEInit Default -sc
PE Server Version and Build information:
java -classpath pesvr.jar filenet.jpe.general.JPEVersion
java -classpath pe.jar filenet.vw.toolkit.utils.VWBuildStamp
You may need to fully specify the path to java and you may need to fully specify the path to pe.jar and pesvr.jar.
Run vwtool and collect the output from these vwtool commands:
Specify the appropriate Process Engine virtual server name when you run vwtool. If you are using the Default server, you do not need to specify the virtual server name when running vwtool. Otherwise run vwtool <server name> in the command prompt window. Then set the region appropriately, using vwtool's region command and run these commands, collecting the resulting output:
threaddump
configdetails
schemastatus
sysconfig b
appspace
dump f c s
config
queueconfig *
rosterconfig *
logconfig *
threaddump
dump s
regionConfig
dbviews
views
loadstatus
count *
threaddump
Note that the threaddump command appears three times, spread out between the other commands. This gives us three separate threaddumps separated by 30 or so seconds.
PE Database information
Database Type and Version: For example, Oracle 11g 11.1.0.6 or SQL Server 2008 or DB2 LUW 9.5 FP2. Note that vwtool's config command (collected above) will include this information in its output.
Is the database on the same server as the PE or on a remote database server?
Include the OS type and version information for database server if the database is remote.
PE's configured connection to the Content Engine
In a web browser on the PE Server, go to your CE Server's basic info pages:
http://ceserver:9080/wsi/FNCEWS40MTOM
In the above URLs, replace "ceserver:9080" with the name and port for your CE Server.
Include the returned HTML pages.
If you don't have a web browser on the PE Server, you can do this in a command prompt to get an HTML page:
java -classpath pe.jar;pe3pt.jar filenet.vw.server.ConfigUtils /url http://ceserver:9080/wsi/FNCEWS40MTOM/
NOTE: You may need to fully specify the path to java and to the pe.jar and pe3pt.jar files
NOTE: If your PE Server is running on Unix or Linux, you will need to separate the jar file names with a colon rather than a semi-colon.
PE Configuration
Include the returned HTML page for these URLs as seen in a browser on the PE Server:
http://localhost:32776/IOR/ping?farm=true&user=Admin&pw=xxxhttp://localhost:32776/IOR/ping?farm=true&acminfo=1&user=Admin&pw=xxx
http://localhost:32776/IOR/FileNet.PE.vworbbroker
You must substitute a valid PE administrator user and password for the example user name (Admin) and password (xxx) in the two IOR/ping URLs.
You must specify the an appropriate region number to replace the "1" in the "acminfo=" clause in the second URL.
NOTE: If you have changed the IOR port from 32776, you must use that new port in the above URLs.
If you don't have a web browser on your PE Server, you can use the same filenet.vw.server.ConfigUtils utility to acquire the HTML pages (as described above).
Process Engine Logs and Traces
Include the appropriate number of Process Engine log and trace files. There are two folders from which logs and traces should be collected. Both folders are under the ProcessEngine install directory. The first folder is .../ProcessEngine/data/logs/ and the second folder is .../ProcessEngine/data/<pe server>/logs/ folder. For example, on Windows, these might be
c:\Program Files\IBM\FileNet\ProcessEngine\data\logs\ and
c:\Program Files\IBM\FileNet\ProcessEngine\data\pesvr.default\logs\.
Zip up an appropriate number of logs and traces that are in those folders.
Collecting Information Specific to the Problem
The following sections describe extra information that can be collected for particular types of problems.
Farm Issues - Load Balancer Configuration
If the PE is being used in a farm configuration, read through the PE 5.0 Farm Troubleshooter:
http://www.ibm.com/support/docview.wss?uid=swg21570520
Then, include the following information:
- Brand and model of load balancer
- Load Balancer settings and configuration information
- The results of what was found when working through the Farm Troubleshooter document.
- The "hosts" file from each PE Servers in the farm.
The data collector does not gather this information automatically.
Tracing In General
The data collector does not automatically enable or create any tracing. But it will collect any traces that have already been created at the time the data collector is run.
If you will be collecting traces, it is easiest to generate the traces before running the data collector so that the traces are automatically included as part of the data collection.
If you have collected any traces, be sure to state the date and time when the problem has occurred.
Screenshots
The data collector does not automatically take any screenshots. It may be appropriate to include screenshots of errors or other symptoms that will help us understand the problem.
PE Server Fault
If the problem is reproducible, it can be invaluable to enable the appropriate PE traces and re-created the problem. For example, in vwtool, you can use the trace command to enable server tracing. The exact traces you enable would vary, depending upon the type of server problem. And after enabling the traces, reproduce the problem.
Include the trace output file from a period where the problem has occurred. Be sure to state the date and time when the problem has occurred.
Include any dumps or cores that may be produced.
Reproducible Problem
If the problem is reproducible, it may be helpful to turn on "ORB RPC", "ORB RPC INPUT", and "Database Access" tracing. Then, reproduce the problem and include trace output. Be sure to state the date and time when the problem has occurred.
PE Server Hung or Performance Problem
If the server is hung or there is a performance issue, run vwtool on the PE Server (or on each server in the farm) and issue a "threaddump" command 3 times, separated by 30 second intervals. Include the thread dump files.
Also note the CPU load when the server was hung. Was the load at zero (indicating everything was blocked)? Or was it at some other number, indicating the server was possibly looping?
If the server is hung and the CPU load is not at zero, turn on "ORB RPC", "ORB RPC Input", and "Database access" tracing, and include the trace output. Be sure to state the date and time when the problem has occurred.
Security / Connection to CE / Logon / Group Membership issues
Enable the appropriate PE traces using vwtool's trace command. For security issues, it is generally good to enable the tracing for "Security calls" and "Environment cache".
Then, reproduce the problem and include the trace output. Be sure to state the date and time when the problem has occurred.
PE API Issues
Enable the appropriate java trace options in the fnlog4j.properties file.
Put that fnlog4j.properties file in the appropriate JVM’s LIB folder. Note that the appropriate JVM may be within an application server.
See the ...\ProcessEngine\data\fnlog4j.properties.sample file for an example fnlog4j.properties file.
Replicate the problem and submit the resulting java trace file. Be sure to state the date and time when the problem has occurred.
Was this topic helpful?
Document Information
Modified date:
17 June 2018
UID
swg21446755