IC64192: TIVOLI STORAGE MANAGER SERVER INSTANCE BECOMES UNRESPONSIVE WHEN SECOND INSTANCE STARTED WITH ENVIRONMENT OF FIRST INSTANCE.

IBM Tivoli Storage Manager V6.1 Fix Pack 3 (6.1.3) Server Downloads

APAR status

• Closed as program error.

Error description

• When a second Tivoli Storage Manager server instance is started
  using the environment of an already running first instance, the
  first instance becomes unresponsive or may crash. When this
  happens, the already running server instance may not accept
  anymore connections and will stop doing any work.
  Whether the Tivoli Storage Manager server instance becomes
  unresponsive or crashes depends on the work being done by the
  instance at the time the second instance is started.
  
  Customer/L2 Diagnostics:
  Below is an example of conditions leading to this issue.
  
  1. First instance is started. For example :
  
  [root@mylinux ~]# cd /home/tsminst1
  [root@mylinux ~]# source /home/tsminst1/sqllib/db2profile
  [root@mylinux tsminst1]# /opt/tivoli/tsm/server/bin/dsmserv
  
  2. Second instance is started using same profile as first one.
     For example :
  
  [root@mylinux ~]# cd /home/tsminst2
  [root@mylinux ~]# source /home/tsminst1/sqllib/db2profile
  [root@mylinux tsminst2]# /opt/tivoli/tsm/server/bin/dsmserv
  
  3. The second instance correctly fails to start with the
     following error :
      ANR0227S Incorrect database opened.  Server cannot start.
     This is the expected result.
  
  4. Following this, the first instance becomes unresponsive or
     crashes. This is the unexpected result.
  
  In the case where the TSM server crashes, the following messages
  are logged in the server activity log :
  
  ANR0162W Supplemental database diagnostic information:
   -1:40003:-1224 ([IBM][CLI Driver] SQL1224N  The database
   manager is not able to accept new requests, has
    terminated all requests in progress, or has terminated
   your particular request due to an error or a force
   interrupt.  SQLSTATE=55032).
  ANR0130E dbieval.c(825): Server LOG space exhausted.
  
  The call stack of the core file produced by the server crash
  shows :
  
  #1  0x000000365c431ae0 in abort () from /lib64/libc.so.6
  #2  0x0000000000c63506 in PsAbortServer ()
  #3  0x0000000000c5dfe5 in pkAbort ()
  #4  0x0000000000787b9c in RdbHangMonitorThread ()
  #5  0x0000000000c5df0b in StartThread ()
  #6  0x000000365d0062e7 in start_thread () from
        /lib64/libpthread.so.0
  #7  0x000000365c4ce3bd in clone () from /lib64/libc.so.6
  
  In the case where the TSM server becomes unresponsive, the
  following messages are logged at the console of the first
  instance running in the foreground :
  
  ANR0130E dbieval.c(825): Server LOG space exhausted.
  ANR0162W Supplemental database diagnostic information:
   -1:40003:-1224
  ([IBM][CLI Driver] SQL1224N  The database manager is not able to
  accept new requests, has terminated all requests in progress, or
  has terminated your particular request due to an error or a
  force interrupt.  SQLSTATE=55032).
  
  Other error messages also logged, but not limited to, are :
  
  ANR0106E cscmdsch.c(546): Unexpected error 5 fetching row in
   table "Schedule.Pending".
  ANR9999D_4073936223 CsCmdSchedulerThread(cscmdsch.c:308)
   Thread<nn0>: Invalid recovery criteria (4) in the central
   scheduler - the task is terminating.
  ANR9999D Thread<nn> issued message 9999 from:
  ANR9999D Thread<nn>  0x0000000000bee072 OutDiagToCons+0x0x142
  ANR9999D Thread<nn>  0x0000000000bf0bf4 outDiagfExt+0x0x194
  ANR9999D Thread<nn>  0x00000000006f1ea4
                                     CsCmdSchedulerThread+0x0x174
  ANR9999D Thread<nn>  0x0000000000c5df0b StartThread+0x0x7b
  ANR9999D Thread<nn>  0x000000365d0062e7 *UNKNOWN*
  ANR9999D Thread<nn>  0x000000365c4ce3bd *UNKNOWN*
  
  ANR2010W Server is out of LOG space in adding entries to the
   Activity Log. Console messages will not be logged until LOG
   space is available.
  ANR0171I dbieval.c(812): Error detected on 0:8, database in
   evaluation mode.
  ANR0101E bfdedup.c(8038): Error 2157 opening table
   "BF.Dereferenced.Chunks".
  ANR0152I Database manager successfully started.
  ANR9999D_0645605689 tbOpenX(tbtbl.c:4248) Thread<5>:
   Failure participating on transaction.
  ANR9999D Thread<nn> issued message 9999 from:
  ANR9999D Thread<nn>  0x0000000000bee072 OutDiagToCons+0x0x142
  ANR9999D Thread<nn>  0x0000000000bf0bf4 outDiagfExt+0x0x194
  ANR9999D Thread<nn>  0x000000000071ad89 tbOpenX+0x0x699
  ANR9999D Thread<nn>  0x0000000000561629
                                      AdmActivityLogThread+0x0x2d9
  ANR9999D Thread<nn>  0x0000000000c5df0b StartThread+0x0x7b
  ANR9999D Thread<nn>  0x000000365d0062e7 *UNKNOWN*
  ANR9999D Thread<nn>  0x000000365c4ce3bd *UNKNOWN*
  ANR0101E admactlg.c(2836): Error 16 opening table
    "Activity.Log".
  ANR2009E Activity log process has stopped - database error.
  ANR2006E Activity log process was not started, the default
   output stream cannot be opened.
  DbiEvalSQLOutcome: Handle SQL_NULL_HANDLE.
  DbiEvalSQLOutcome: SQLGetDiagRec returned -2.
  DbiEvalSQLOutcome: SQL returned -1.
  ANR0101E bfdedup.c(7965): Error 11 opening table
   "BF.Dereferenced.Chunks".
  ANR0171I dbieval.c(812): Error detected on 0:4, database in
   evaluation mode.
  
  TSM Versions Affected:
  TSM 6.1 servers on all supported platforms.
  
  Additional Keywords:
  zz61 multiple servers same machine
  

Local fix

• Start the TSM server instance with the correct environment.
  

Problem summary

• ****************************************************************
  * USERS AFFECTED: All Tivoli Storage Manager server V6 users.  *
  ****************************************************************
  * PROBLEM DESCRIPTION: See error description.                  *
  ****************************************************************
  * RECOMMENDATION: Apply fixing level when available. This      *
  *                 problem is fixed in level 6.1.3.             *
  ****************************************************************
  *
  

Problem conclusion

• This problem was fixed.
  
  Affected platforms:  AIX, HP-UX, Sun Solaris, and Linux.
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  TSM SERVER

• Fixed component ID

  5698ISMSV

Applicable component levels

• R61A PSY

     UP

• R61H PSY

     UP

• R61L PSY

     UP

• R61S PSY

     UP