Go away legendaries!

Few days ago I found a performance issue in a SOA architecture.

JEE Architecture (simplified)

Browser ----(Comet)----> WEB ----(CXF/SOAP)----> SOA ----> SGBD

A Real-Time (RT) like implementation was required to deliver notifications to connected front-end users. Because of the SOAP backend server, RT is not possible between WEB and SOAP instance.
The current implementation was the following : one thread is created on session creation event. This thread located in WEB instance periodically performs a SOAP request. This thread is attached to user's session lifecycle.

I suggested to move to another implementation : one thread to manage all notifications of all users.

Performance Issue ?

Does this new implementation delivers better regarding all performance facets? CPU, memory and response time? Will the CPU usage be best for several small tasks?


Let's talking the JMX results with different use cases, but all with this settings :
TT=5s (think time)
duration=270s, (total time, includes the loading)
1 of 2 shared core for WEB and SOA (so only 50% CPU available)
SOA and WEB on the same instance (so no network connections)
SGBD is MySQL on the LAN, ping 19ms

Old implementation

First, with existing implementation, 100 VU

WEB (lighweight) side result, 100VU

SOA side results, 100VU

All available CPU is used, SOA thread pool reach 54 concurrent threads, GC ran very many times.

Next scenario, same implementation, 200 VU

WEB (lighweight) side result, 200VU

SOA side results, 200VU

Again, all available CPU is used, SOA thread pool reach 70 concurrent threads, GC ran as fast as possible when memory limit 256M was reached. I addition, we see an additional 50% memory usage on client side dued to threads management.

Next scenario, same implementation, 200 VU, plenty of memory

As the GC ran ran too many times, I increased the SOA memory to an unreachable limit.
Client side

SOA side
In this implementation, GC was not the root cause of CPU overhead.

New implementation

For the new implementation, in order to cut off this post, I will show only the 200VU results.
WEB (lighweight) side result, 200VU

SOA side results, 200VU
That's it, barely 1% CPU used both sides, only 2 GC ran on SOA, and it's initial thread pool (30) has not been increased.


In addition, old implementation results could be worst if loopback interface was not used because of HTTP network connections.
Hoping this showcase will definitely chase these kind of IT urban legends.


How to run a Spring 3+JPA in JBoss 6 ?

To integrate to JBoss 6 a Spring application tested in Tomcat, we face to some JEE and CL issues. We won't discuss about solutions synchronizing the JEE Server or application dependencies. If you accept that, then you can find a solution here.

The JEE issues

All JEE APIs are brought by the JEE server (Servlet - as Tomcat does -, Java Persistence, JAXB, ...) : they must be removed from the application WAR/EAR.
This could be performed by a profile in your Maven build.

The main JPA issue is the file "persistence.xml" watched by JBoss. We have to rename it to "persistence-spring.xml". This way JBoss (and other JEE servers) will not detect JPA API usage in your application. We agree ... , this is a huge fault regarding an expected JEE Application using JPA and so, expecting a JPA implementation to be provided by the JEE Server with a total abstraction. Anyway I've never seen any application does not use some Hibernate or Toplink hints/properties/references in the "persistence.xml".

Also, to accomplish this, we have to update Spring PU reference from the EntityManagerFactory :

<bean id="emf-pu" 

As we'll see in the CL tuning, the libraries brought by JBoss are not hidden; there is just a preference to the Application's ones. So when any library does a Class.forName( X ), if X is not present in the application's CL, but in the JEE Server's CL, this library will be able to load X, and to cause some unexpected behaviors since this does not happen in Tomcat environment (or in another JEE servers). This is the case of Hibernate trying to locate dynamically some Hibernate-Validator classes brought by JBoss. To avoid this, you'll have to add these properties in your old "persistence.xml" :

<property name="hibernate.validator.apply_to_ddl" value="false" />
<property name="hibernate.validator.autoregister_listeners" value="false"/>

A similar issue for JAXB with a JDK 6u4+ with <2.2.1 (as in JBoss 6.0.0) requires -Dcom.sun.xml.bind.v2.bytecode.ClassTailor.noOptimize=true in System properties/JAVA_OPTS/...

ClassLoading issues

File "/WEB-INF/jboss-classloading.xml" needs to be added :

<classloading xmlns="urn:jboss:classloading:1.0"

A JBoss 5+'s specific problem is brought by the new VFS protocol causes to all class/file scanners to return an empty list. Spring ORM uses this process to discover @Entity classes, JAR containing the file "persistence.xml" has an URL like vfs:/usr/.../myEnties.jar/. As the VFS protocol in not shared to application's CL, we have to switch to our specific scanner. Add this property to "persistence.xml" :

<property name="hibernate.ejb.resource_scanner" value="com.company.ResourceScanner"/>

The corresponding implementation :

public class ResourceScanner extends NativeScanner {

 public Set<Class<?>> getClassesInJar(final URL jarToScan,
   final Set<Class<? extends Annotation>> annotationsToLookFor) {
  return super.getClassesInJar(patchUrl(jarToScan), annotationsToLookFor);

 public Set<NamedInputStream> getFilesInJar(final URL jarToScan, final Set<String> filePatterns) {
  return super.getFilesInJar(patchUrl(jarToScan), filePatterns);

 public Set<Package> getPackagesInJar(final URL jarToScan,
   final Set<Class<? extends Annotation>> annotationsToLookFor) {
  return super.getPackagesInJar(patchUrl(jarToScan), annotationsToLookFor);

 public String getUnqualifiedJarName(final URL jarToScan) {
  return super.getUnqualifiedJarName(patchUrl(jarToScan));

  * Patch the VFS URL to a FILE protocol URL.
  * @param url
  *            original URL.
  * @return either the original, either the corresponding FILE protocol of given VFS URL.
 protected URL patchUrl(final URL url) {
  if (url.getProtocol().equals("vfs")) {
   try {
    return new File(url.getFile()).toURI().toURL();
   } catch (final MalformedURLException e) {
    return url;
  return url;

Take care, VFS is not always a simple replacement of protocol "FILE". This tip assumes there is no VFS configuration in JBoss (mapping, etc.).
Also there is the jboss-vfs maven artifact that can help to handle more seriously this file system.

Note : don't forget the standard JBoss configuration files "jboss-web.xml" (context-root, resource-ref, ...) and "*-ds.xml" in your EAR or deplyoment location.