Solr and Jetty deployment issue

Question

I am having trouble with getting solr + jetty to work. I am following all instructions to the letter from - http://wiki.apache.org/solr/SolrJetty. It works like a good. But when I restart jetty multiple times, after 3/4 such restarts it starts hanging. Admin pages just don't load and my app fails to acquire a connection with solr. I also created a work folder - /opt/solr/work. I am also setting tmpdir to a new path in /etc/default/jetty. I can confirm the tmpdir is set to the new path from admin dashboard, under args. So it's mostly not an issue with purging of tmp files by OS.

What might I be missing? Should I be rather looking at my code and see if I am not committing correctly?

My configs - Solr 4.0.0 and jetty from example. Ubuntu 12.04 with Open JDK 7.

Edit:

I am running Jetty 8, bundled with Solr example on a Ubuntu 12.04 machine. When I use start.jar and server does not come up properly, while shutting down jetty throws ThreadPoolException - Failing to stop threads.

Here is a dump of stack trace:

2012-12-27 23:00:15.084:WARN:oejut.QueuedThreadPool:1 threads could not be stopped
2012-12-27 23:00:15.084:INFO:oejut.QueuedThreadPool:Couldn't stop Thread[qtp766488133-16,5,main]
2012-12-27 23:00:15.085:INFO:oejut.QueuedThreadPool: at sun.misc.Unsafe.park(Native Method)
2012-12-27 23:00:15.085:INFO:oejut.QueuedThreadPool: at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
2012-12-27 23:00:15.085:INFO:oejut.QueuedThreadPool: at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2081)
2012-12-27 23:00:15.085:INFO:oejut.QueuedThreadPool: at java.util.concurrent.ThreadPoolExecutor.awaitTermination(ThreadPoolExecutor.java:1425)
2012-12-27 23:00:15.086:INFO:oejut.QueuedThreadPool: at java.util.concurrent.Executors$DelegatedExecutorService.awaitTermination(Executors.java:636)
2012-12-27 23:00:15.086:INFO:oejut.QueuedThreadPool: at org.apache.solr.core.SolrCore.close(SolrCore.java:835)
2012-12-27 23:00:15.086:INFO:oejut.QueuedThreadPool: at org.apache.solr.handler.admin.CoreAdminHandler.getCoreStatus(CoreAdminHandler.java:865)

can you edit and add in the more from that ThreadPoolException? — jesse mcconnell
Added a stacktrace. Let me know if you need more information. — sbidwai
ok, think that exception is a red herring, likely a request blocked on something internal to solr and not getting cleaned up on close fully. — jesse mcconnell
So it is definitely a solr side issue and not a config issue like tmpdir or work directory setting? — sbidwai

jesse mcconnell jesse mcconnell · Accepted Answer · 2012-12-27T17:38:34

Since the Exception doesn't point out anything interesting, I would recommend that when it is hanging that you now take a thread dump of the whole server and analyze that for what is blocking correct startup. These can take a bit of trial and error to sort out what is what but should let you clue into what is holding things up. Note that the threads that start with qtp in the thread dump are rarely the issue, they are for processing requests so while there are likely a number of them there that is not an indication they are an issue. Common issues are things like database pooling that is 'await' on a pooled resource and things like that.

Solr and Jetty deployment issue

2 Answers