We're seeing some strange behavior on our servers where threads are locking up on deployment of our code. Taking a look at the stack traces in the thread dumps when this issue occurs points to a couple different culprits, none of which seem to originate in our codebase.
The most common culprit usually ends up in this stack trace:
java.lang.Thread.State: BLOCKED (on object monitor)
at sun.misc.Unsafe.defineClass(Native Method)
at sun.reflect.ClassDefiner.defineClass(ClassDefiner.java:45)
at sun.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:381)
at java.security.AccessController.doPrivileged(Native Method)
at sun.reflect.MethodAccessorGenerator.generate(MethodAccessorGenerator.java:377)
at sun.reflect.MethodAccessorGenerator.generateMethod(MethodAccessorGenerator.java:59)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:28)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at javax.el.BeanELResolver.getValue(BeanELResolver.java:62)
at javax.el.CompositeELResolver.getValue(CompositeELResolver.java:53)
at org.apache.el.parser.AstValue.getValue(AstValue.java:97)
at org.apache.el.parser.AstEqual.getValue(AstEqual.java:21)
at org.apache.el.parser.AstChoice.getValue(AstChoice.java:27)
at org.apache.el.ValueExpressionImpl.getValue(ValueExpressionImpl.java:186)
at org.apache.sling.scripting.jsp.jasper.runtime.PageContextImpl.proprietaryEvaluate(PageCon textImpl.java:975)
The other most common culprit usually ends in this:
java.lang.Thread.State: BLOCKED (on object monitor)
at org.apache.felix.framework.BundleWiringImpl$BundleClassLoader.loadClass(BundleWiringImpl. java:1873) - waiting to lock <0x000000076ce21a60> (a org.apache.felix.framework.BundleWiringImpl$BundleClassLoaderJava5)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
at org.apache.felix.framework.BundleWiringImpl.getClassByDelegation(BundleWiringImpl.java:13 56)
at org.apache.felix.framework.BundleWiringImpl.searchDynamicImports(BundleWiringImpl.java:15 97)
at org.apache.felix.framework.BundleWiringImpl.findClassOrResourceByDelegation(BundleWiringI mpl.java:1478)
at org.apache.felix.framework.BundleWiringImpl.access$400(BundleWiringImpl.java:75)
at org.apache.felix.framework.BundleWiringImpl$BundleClassLoader.loadClass(BundleWiringImpl. java:1882)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:169)
Other less common places where blocked threads end up seem to be mostly random and I'm not totally sure they have any hand in the cause of this issue.
Snooping around the web gives very few promising results. The only hints I've gotten are that maybe there are too many bundles starting up at the same time, but we don't have an atypical number of bundles starting up on our CQ instance. Also, I think it may be an OSGi or an Apache Sling issue.
Has anyone experienced this issue before?