EJBDeploy command Exceptions on WebSphere Application Server and Eclipse OSGI cache purge issues

Troubleshooting

Problem

Attempts to install an EAR on IBM AIX WebSphere Application Server ND v6.0.2.31 using the admin console with Deploy EJB box checked results in this error: java.lang.RuntimeException: Application "com.ibm.etools.ejbdeploy.batch_extension" could not be found in the registry

Symptom

Installation of an EAR in the admin console with Deploy EJB box checked result in errors.

Same problem using wsejbdeploy.
Other similar errors were encountered using the ejbdeploy command directly.

Example 1, EJBDeploy Error trace for above Admin Console issue:

ADMA5018I: The EJBDeploy command is running on enterprise archive (EAR) file /usr/WebSphere/AppServer/profiles/DSWFVTDmgr/wstemp/415291458/upload/Quo teSchedulerEAR_FVT.ear. Starting workbench. An unexpected exception was thrown. Halting execution. An unexpected exception was thrown. Halting execution. Shutting down workbench. Error executing deployment: java.lang.RuntimeException. Error is Application "com.ibm.etools.ejbdeploy.batch_extension" could not be found in the registry. The applications available are: .. java.lang.RuntimeException: Application "com.ibm.etools.ejbdeploy.batch_extension" could not be found in the registry. The applications available are: . at org.eclipse.core.internal.runtime.PlatformActivator$1.run(PlatformActiva

Example 2. Other EJBDeploy Error:

************ Start Display Current Environment ************
WebSphere Platform 7.0.0.3 [ND 7.0.0.3 cf030911.09] ...

[10/29/09 13:21:54:650 CDT] 0000017b DeployEJBTask E ADMA0086E: [EJBDeploy] An unexpected exception was thrown. Halting execution. [10/29/09 13:21:54:654 CDT] 0000017b DeployEJBTask I ADMA0158I:
[EJBDeploy] Shutting down workbench.
[10/29/09 13:21:57:031 CDT] 0000017b DeployEJBTask I ADMA0158I:
[EJBDeploy] Error executing deployment: java.lang.NoClassDefFoundError.
Error is com.ibm.ws.ast.st.core.WASRuntimeLocator.

[10/29/09 13:21:57:035 CDT] 0000017b DeployEJBTask I ADMA0158I:
[EJBDeploy] java.lang.NoClassDefFoundError:
com.ibm.ws.ast.st.core.WASRuntimeLocator
[10/29/09 13:21:57:039 CDT] 0000017b DeployEJBTask I ADMA0158I:
[EJBDeploy] at
com.ibm.etools.ejbdeploy.batch.plugin.BatchExtension.getRuntimeTargetJar
s(BatchExtension.java:2383)
-------------

Example 3. Other EJBDeploy Error:

Root error:
[wsejbdeploy] Working Directory does not exist. Creating.
[wsejbdeploy] Exception in thread "main" java.lang.NoClassDefFoundError:
com.ibm.etools.ejbdeploy.EJBDeploy
[wsejbdeploy] Caused by: java.lang.ClassNotFoundException:
com.ibm.etools.ejbdeploy.EJBDeploy
....
[wsejbdeploy] at
java.net.URLClassLoader.findClass(URLClassLoader.java:419)
[wsejbdeploy] at java.lang.ClassLoader.loadClass(ClassLoader.java:643)
[wsejbdeploy] at
sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:345)
[wsejbdeploy] at java.lang.ClassLoader.loadClass(ClassLoader.java:609)
[wsejbdeploy] Could not find the main class:
com.ibm.etools.ejbdeploy.EJBDeploy. Program will exit.
[wsejbdeploy] Java Result: 1

Example 4. Other EJBDeploy Error:

Error executing deployment: org.eclipse.core.runtime.CoreException. Error is Plug-in com.ibm.etools.ejbdeploy was unable to load class com.ibm.etools.ejbdeploy.batch.plugin.BatchEx
tension..
org.eclipse.core.runtime.CoreException: Plug-in com.ibm.etools.ejbdeploy was unable to load class com.ibm.etools.ejbdeploy.batch.plugin.BatchExtension. at org.eclipse.core.internal.registry.osgi.RegistryStrategyOSGI.throwException(RegistryStrategyOSGI.java:165) at org.eclipse.core.internal.registry.osgi.RegistryStrategyOSGI.createExecutableExtension(RegistryStrategyOSGI.java:149) at org.eclipse.core.internal.registry.ExtensionRegistry.createExecutableExtension(ExtensionRegistry.java:759) at .. ....
org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.runApplication(EclipseAppLauncher.java:92)
at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start(EclipseAppLauncher.java:68)
at ...
at com.ibm.etools.ejbdeploy.batch.impl.BatchDeploy.execute(BatchDeploy.java:106)
at com.ibm.etools.ejbdeploy.EJBDeploy.execute(EJBDeploy.java:107)
at

Example 5. Other EJBDeploy Error:

Error message:
[exec] [wsadmin] An unexpected exception was thrown. Halting
execution.
[exec] [wsadmin] Shutting down workbench.
[exec] [wsadmin] Error executing deployment: java.lang.
IllegalStateException. Error is Platform not running.
[exec] [wsadmin] java.lang.IllegalStateException: Platform not
running
[exec] [wsadmin] at org.eclipse.core.runtime.adaptor.EclipseStarter.
run(EclipseStarter.java:374)
[exec] [wsadmin] at sun.reflect.NativeMethodAccessorImpl.invoke0
(Native Method)
[exec] [wsadmin] at sun.reflect.NativeMethodAccessorImpl.invoke
(NativeMethodAccessorImpl.java:48)
[exec] [wsadmin] at sun.reflect.DelegatingMethodAccessorImpl.invoke
(DelegatingMethodAccessorImpl.java:25) [exec] [wsadmin] at java.
lang.reflect.Method.invoke(Method.java:600)
[exec] [wsadmin] at com.ibm.etools.ejbdeploy.batch.impl.
BootLoaderLoader.run(BootLoaderLoader.java:494)
[exec] [wsadmin] at com.ibm.etools.ejbdeploy.batch.impl.BatchDeploy.
execute(BatchDeploy.java:114)
[exec] [wsadmin] at com.ibm.etools.ejbdeploy.EJBDeploy.execute
(EJBDeploy.java:107) ...
[exec] [wsadmin] ADMA5007I: The EJBDeploy program completed on file
C:\Users\us325660\AppData\Local\Temp\app2473060703210803978.ear
[exec] [wsadmin] ADMA0063E: An error occurred during Enterprise
JavaBeans (EJB) deployment. Exception: com.ibm.websphere.management.
exception.AdminException: ADMA0062E: The EJBDeploy program issued the
following error messages: [An unexpected exception was thrown. Halting
execution.]
[exec] [wsadmin] ADMA5011I: The cleanup of the temp directory for
application deployment.ear is complete.
[exec] [wsadmin] ADMA5069E: The installation of application
deployment.ear failed. ...

Example 6. Other EJBDeploy Error:

Error message in SystemOut.log:
[...] 000001e2 DeployEJBTask I ADMA0158I: [EJBDeploy] Caused by: java.io.EOFException
[...] 000001e2 DeployEJBTask I ADMA0158I: [EJBDeploy]   at java.io.DataInputStream.readByte(DataInputStream.java:278)
[...] 000001e2 DeployEJBTask I ADMA0158I: [EJBDeploy]   at org.eclipse.osgi.container.ModuleDatabase$Persistence.readString(ModuleDatabase.java:1517)
[...] 000001e2 DeployEJBTask I ADMA0158I: [EJBDeploy]   at org.eclipse.osgi.container.ModuleDatabase$Persistence.readMap(ModuleDatabase.java:1339)

Cause

EJBDeploy is a headless Eclipse application in WebSphere Application Server v6.x/7.x/8.0.x, and the above problems are often specific to the Eclipse OSGI cache it uses. EJBDeploy in in WebSphere Application Server v6.x/7.x is based on Eclipse 3.2.x , while in v8.0.x it is based on Eclipse 3.6.x.

During the install of WebSphere Application Server or application of a FixPack update, the EJBDeploy application may not have been correctly installed or refreshed. In particular a WebSphere Application Server FixPack install has succeeded but has an OSGi configuration cache that was not purged/cleared as expected. This may especially be the case in a clustered WebSphere Application Server Network Deployment (ND) environment or other complex install configurations of the server such as clusters.

Note: The issue of a corrupted Eclipse OSGI cache, should not be confused with other unrelated defects:

- APAR PM21491: "NullPointerException in org.eclipse.jem.internal.adapters.jdom.JDOMSearchHelper.needsToResolveName"
- APAR PM40815: Error "java.util.ConcurrentModificationException" occurs during ejbdeploy command.

They involve an error that is intermittent and is often associated with a ConcurrentModificationException (CME), in addition to a NullPointerException (NPE). This involves the eclipse Java Eclipse Modeling Framework (EMF/JEM)). It can be an issue in multi-core CPU machines or with multiple users or scripts initiating several or simultaneous ejbdeploy(s). If you encounter this issue, contact IBM Support for Rational Application Developer.

Environment

AIX WebSphere Application Server ND v6.0.2.31 upgraded from 6.0.2.19

Resolving The Problem

Before pursuing the EJBDeploy OSGI Cache solutions in this technote:

Check you have the latest IBM Java SDK update for your WebSphere Application Server version as per technote 1573915: EJBDeploy failed to start with java.lang.NoClassDefFoundError, especially so if there is a NoClassDefFoundError as soon as ejbdeploy starts up.
You may still observe the "..ejbdeploy.batch_extension could not be found" and the "..IllegalStateException:Platform not running" error messages after clearing the cache. In UNIX or Linux systems the file system ulimit must be large enough (such as ulimit -n 8192), otherwise it may cause problems with the file system based cache. Check the ulimit in advance, and if there is still a problem after clearing the cache increase the ulimit and retest. See also this ejbdeploy ulimit related technote with ulimit examples: 1383342: EJBDeploymentException and ADMA0086E is thrown by WebSphere Process Server (WPS) when deploying SCA modules.

This technote is applicable to all Operating Systems where WebSphere Application Server 6.x/7.x/8..x
are run and ejbdeploy is used.

FIRST - Quick (possible) workaround:

Note, however, that the recommendation is to apply the SECOND solution first.

This workaround has worked well for some clients when encountering ejbdeploy problems immediately after initial installation of the server. Otherwise a re-install of WebSphere Application Server 6.x/7.x/8.0.x might be the best and quickest solution if re-install is an option in this case.

The goal is to have a good $USER_INSTALL_ROOT/deploytool directory that is not corrupted or incomplete.

Additionally being at the latest WebSphere Application Server v6.x/7.x/8.0.x FixPack with the most recent EJBDeploy may help, as there are ongoing improvements in this area.

Replacing $USER_INSTALL_ROOT/deploytool directory from the working WebSphere Application Server v6.x/7.x/8.0.x environment:

Try another similar O/S machine with the same version of WebSphere Application Server that is able to complete the EJBDeploy on the same EAR. If that works, proceed to step 2.
Then try copying {WAS_HOME}\deploytool\itp directory from a working machine to the bad machine (backup first). Copying the $USER_INSTALL_ROOT/deploytool directory from the working WebSphere Application Server 6.x environment fixed the issue in deploying the EARs with EJBs.

HOWEVER, while has been known to fix the issue, deleting the ../deploytool/itp/configuration/ directory as described next would have most likely done the same as the ../configuration directory is regenerated when an ejb deploy is done. Its also more useful if another good machine does not exist

SECOND - Recommended EJBDeploy solution:

1. In the reported scenarios, this has often been analyzed to be an Eclipse OSGI cache issues, in that EJBDeploy runs as an Eclipse based application. In which case it is always suggested to remove a few predefined known caching directories as follows (some of these may not exist).

To determine the location of the EJB Deploy configuration directory, run the ejbdeploy command with the -log option like this:

On Windows:
> cd "C:\Program Files\IBM\WebSphere\AppServer\bin" > ejbdeploy.bat {input_EAR} {working_DIR} {output_EAR} -log

On Linux:
> cd /opt/IBM/WebSphere/AppServer/bin > ./ejbdeploy.sh {input_EAR} { working_DIR} {output_EAR} -log

In the commands, {input_EAR} is the input EAR file, {output_EAR} is the output EAR file, and {working_DIR} is the temporary working directory. The above {WAS_HOME install dir}/bin are typical defaults and may not apply to your installation location.

Similar ejbdeploy commands can be issued on any other platforms on which IBM WebSphere Application Server v6.x/7.x/8.0.x is supported.

The output of the ejbdeploy command, with -log option, will show the location of the EJB Deploy configuration directory. As an example, on a Unix/Linux platform with WebSphere Application Server Network Deployment (ND) installed in /usr/WebSphere/AppServer, you may have two locations as per this UNIX example:

/usr/WebSphere/AppServer/deploytool/itp/configuration
/usr/WebSphere/AppServer/profiles/Dmgr01/ejbdeploy/configuration/

Note: in an ND cluster there is only an ejbdeploy folder in the Dmgr Profile

Next you will typically go do that directory or directories and delete all files except for the config.ini file (apart from WebSphere Application Server 6.0.2.x) as described in the next set of steps. The next set of steps (a) and (b) have more details and examples on UNIX and Linux and Windows platforms, and possible exceptions and especially so if using the Admin Console .
After that, re-run your ejbdeploy command, as per 2. (below) and it should no longer have the error.

1. Only if a UNIX/Linux box, delete the profile ejbdeploy configuration directories:
  
  <default profile or dmgr profile>/ejbdeploy/configuration/
  
  and "possibly" these if they exist at the root of the UNIX file system:
  
  /ejbdeploy/configuration/ ~/.eclipse/configuration/
  
  Note - Administrative (admin) Console ejbdeploy: Ideally, those directories at the root for the UNIX file system would never be used though. The preferred folder for EJBDeploy folder would be the eclipse: ../configuration directory in a server profile, which is what the cmd-line ejbdeploy uses. If you look at the cmd line: ../deploytool/itp/ejbdeploy.sh, you will see a system property that specifies the directory to use:
  
  -Dejbdeploy.user.install.root=$USER_INSTALL_ROOT/ejbdeploy
  
  That directory cannot be set by means of the admin console, so sometimes it is not straightforward to determine which directory is being used.
  The .eclipse folder is a default that Eclipse itself has configured, but since it is a default, you probably will not find it anywhere explicitly in the ini files or in any registry settings. Eclipse uses that folder when it does not have write access to the main copy of Eclipse, and also in shared configurations where multiple users are sharing the same machine. It can be in a UNIXx/Linux /.eclipse if a root user or in some user's $HOME/.eclipse. This folder can also end up being created in an ant build using the WebSphere wsejbdeploy ant task configured to output to a location to which it does not have write access.
  Also, Check to see if the directory $WAS_HOME/deploytool/itp is writeable by the user launching ejbdeploy.sh. If the directory cannot be made writeable for the user launching EJBDeploy, then you can work around this problem by changing the
  -Dejbdeploy.user.install.root=$USER_INSTALL_ROOT
  to
  -Dejbdeploy.user.install.root=<some temporary directory that is writeable>
  in the $WAS_HOME/deploytool/itp/ejbdeploy.sh file.
2. Regardless of operating system, delete the server ejbdeploy /deploytool/itp/configuration directory
  $WAS_HOME/deploytool/itp/configuration/
  as follows:
  - in WebSphere Application Server 6.0.2.x it is safe to just delete the directory contents
  - in WebSphere Application Server 6.1.0.x (and 7.0.0.x) you need to delete all files in the configuration directory, except the "config.ini" file
  
  The ../itp/configuration directory is created the first time you run ejbdeploy and is recreated when you run ejbdeploy if it was deleted.
  
  Notes:
  - On a UNIX/Linux box both the affected profile and server configuration directories mentioned need to be deleted/cleaned out as described
  - Windows uses $WAS_HOME/deploytool/itp/configuration/ by default, but on UNIX/Linux sometimes the WebSphere Application Server install is read-only, so ejbdeploy will find and use a location that is writable which is typically your profile directory)
  - Earlier WebSphere Application Server 6.x version Information:
    Originally WebSphere Application Server 6.0.2.x/v6.1.0.x did not have a $WAS_HOME/deploytool/scripts folder.
    For v6.0.2.x it was not introduced until FixPack 6.0.2.31+.
    For v6.1.0.x it was not introduced until Fixpack 6.1.0.15+.
    So it is advisable to be at the latest server FixPack.

2. Re-run your ejbdeploy command: ejbdeploy.{bat|sh} or corresponding wsadmin cmd, and it should no longer have the error. If using the Admin Console try to install EAR with ejbDeploy checkbox checked.

3. If purging those directories doesn't fix then problem, then you need to zip up copy of the entire "deploytools" directory to send to IBM Rational Application Developer Support.

If the above does not help and this was after an initial of the server, re-install should be considered and may be quicker in obtaining a resolution

THIRD - When is the EJBDeploy Eclipse OSGI cache normally cleared in WebSphere Application Server v6.x/7.x/8.0.x?

During install/uninstall of WebSphere Application Server 6.x/7.x/8.0.x FixPack (FP) , a WebSphere Application Server config action is run which executes an ejbdeploy-clear-cache script. These are the ejbdeploy clear cache scripts:

Windows:
.\deploytool\scripts\install\ejbdeploy-clear-cache.bat
Linux and UNIX:
.\deploytool\scripts\install\ejbdeploy-clear-cache.sh
z/OS:
.\deploytool\scripts\install\ejbdeploy-clear-cache

The script is run during install/uninstall of a FP on WebSphere Application Server, as a part of updating ejbdeploy.

Note: It needs to be run every time ejbdeploy is updated, even if getting a test fix patch from IBM Rational Application Developer Support.

A user/administrator on WebSphere Application Server can run the script anytime if a post install/update ejbdeploy cache problem is suspected.

After running the script, the ejbdeploy will takes a little longer to run the next time it is executed for the first time. This is just like after manually adding new plugins or updates to Rational Application Developer v7.5/Eclipse 3.4.x/P2 and restart Rational Application Developer v7.5. In the case of the older Rational Application Developer v7.0/Eclipse 3.2 we restart Rational Application Developer using -clean.
When Rational Application Developer is updated using Installation manager, it is also slow restarting the first time. As of WebSphere Application Server v7.0.0.x, the ejbdeploy application is based on Eclipse 3.2.x. The script is similar to the Eclipse 3.2.x -clean option, but more of a direct brute force function designed for ejbdeploy on WebSphere Application Server v6.x/7.x. It also applies to WebSphere Application Server v8.0.x, in which the ejbdeploy application is based on Eclipse 3.6.x , but does not use Equinox/p2

Note: If the script does not help, it may still be necessary to clear certain directories manually in Linux/Unix as per the previous instructions under "SECOND - Recommended EJBDeploy solution:"

Sometimes after the update of a WebSphere Application Server server, it appears as if clearing of the Eclipse OSGI cache did not occur and resulted in the problems reported in this technote, while other updated servers are fine. It is important to understand there is no explicit actions that triggers the action to clear the Eclipse OSGI cache. The ejbdeploy clear cache script is always run as part of a WebSphere Application Server update or install

It's possible a failure may occur when the clear cache script is run because of some environmental O/S reason or the server update/install may have had an error that negatively affected the script. The WebSphere Application Server update/install logs should show any errors issued directly by the ejbdeploy cache clear script. In some cases there may not be an obvious error issued. Its a good practice to review the install/update logs and system logs after an install or update for anything that may suggest a problem with any part of the update or install, even if the WebSphere Application Server install or update appears to have completed successfully.

FOURTH - Always clear the cache by default

If there is no clear cause identified by IBM Rational Application Developer support, and clear or delete of the ejbdeploy eclipse/OSGI cache has been done or if it has to be done on a frequent basis, a better workaround than running the clear cache script to clean up the configuration folders (as in THIRD above), would be to modify/edit the ejbdeploy command/script to have it do a cache clean very time:

The the ejbdeploy command/script is located:

<WAS>/deploytool/itp/ejbdeploy.bat (or .sh)

Add -Dosgi.clean to the Java command of {ejbdeploy.{bat | sh} as per the following examples:

1. Example of edited UNIX/Linux ejbdeploy.sh on WebSphere Application Server v6.x/7.x ND

> cd /usr/local/opt/was/was70/deploytool/itp
> vi ejbdeploy.sh

//------edited ejbdeploy.sh excerpt"
..
$JAVA_CMD \
-Xbootclasspath/a:$ejbd_bootpath \
-Xms256m -Xmx256m \
-Dws.ext.dirs=$WAS_HOME/eclipse/plugins/j2ee.javax_1.4.0:$WAS_HOME/eclipse/plugins/com.ibm.ws.runtime.eclipse_1.0.0:$WAS_EXT_DIRS \
-Dwebsphere.lib.dir=$WAS_HOME/lib \
-Dwas.install.root=$WAS_HOME \
-Ditp.loc=$ITP_LOC \
-Dorg.osgi.framework.bootdelegation=* \
-Dejbdeploy.user.install.root=$USER_INSTALL_ROOT/ejbdeploy \
$USER_INSTALL_PROP \
-Dcom.ibm.sse.model.structuredbuilder="off" \
-Dosgi.clean="true" \
-cp $ejbd_cp \
..
//----------

Notes:

-Dosgi.clean="true" \ was added after -Dcom.ibm.sse.model.structuredbuilder="off" \
You need to add a "\" when you add a new line.
There should be no [space] character or other hidden character after the "\"

2. Example of edited Windows ejbdeploy.bat

//------edited ejbdeploy.bat excerpt"
..
"%JAVA_HOME%\bin\java" -Ditp.loc="%ITP_LOC%" %EJBDEPLOY_JVM_ARGS% -Dorg.osgi.framework.bootdelegation=* -Dwas.install.root="%WAS_HOME%" -Dwebsphere.lib.dir="%WAS_HOME%\lib" -Dws.ext.dirs="%WAS_HOME%\eclipse\plugins\j2ee.javax_1.4.0";"%WAS_HOME%\eclipse\plugins\com.ibm.ws.runtime.eclipse_1.0.0";"%WAS_EXT_DIRS%" -Dcom.ibm.sse.model.structuredbuilder="off" -Dosgi.clean="true" -cp "%ejbd_cp%" -Xbootclasspath/a:"%bootpath%" -Xj9 -Xquickstart -Xverify:none -Xms256M -Xmx256M %WAS_DEBUG% %JDB_DEBUG% com.ibm.etools.ejbdeploy.EJBDeploy %*
goto end
..
/-------

This is with the understanding, it can add several seconds to ejbdeploy.bat starting up.

This fourth solution also applies to WebSphere Application Server v8.0.x, in which EJBDeploy is based on Eclipse 3.6.x, since it is still running in non-Equinox/p2 mode, similar to EJBDeploy on Eclipse 3.2.x in WebSphere Application Server v6.x/7.x .

Conclusion: If none of the proposed solutions help to the problem, contact IBM Support for Rational Application Developer. You will be asked to zip up a copy of the entire 'deploytools' directory referred to in "SECOND - Recommended EJBDeploy solution:" (above) and send it.

Related Information

EJB Deploy Error: Unable to acquire application service

[{"Product":{"code":"SSRTLW","label":"Rational Application Developer for WebSphere Software"},"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Component":"EJB Development","Platform":[{"code":"PF002","label":"AIX"},{"code":"PF016","label":"Linux"},{"code":"PF033","label":"Windows"},{"code":"PF010","label":"HP-UX"},{"code":"PF027","label":"Solaris"}],"Version":"7.0;7.0.0.1;7.0.0.2;7.0.0.3;7.0.0.4;7.0.0.5;7.0.0.6;7.0.0.7;7.0.0.8;7.5;7.5.1;7.5.2;7.5.3;7.5.4;7.5.5;7.5.5.1;7.5.5.2;7.5.5.3;8.0;8.0.1;8.0.2;8.0.3","Edition":"","Line of Business":{"code":"LOB45","label":"Automation"}}]

Tips

EJBDeploy command Exceptions on WebSphere Application Server and Eclipse OSGI cache purge issues

Troubleshooting

Problem

Symptom

Cause

Environment

Resolving The Problem

Related Information

Was this topic helpful?

Document Information

UID

Share your feedback

Need support?