Play Framework 1.2.7 Heroku update crashing

Question

I pushed an update to my app on Heroku, and I now receive the following error with the app crashing. If you look at the error, you will see it has to do with S3Blobs.java. However, my last update didn't affect this file, or in fact any of the Java code. The last update was to html, javascript and css files.

Is anyone familiar with this error, and what could have caused it to crop up seemingly out of nowhere on Heroku. Locally the app runs great.

If I do a rollback to a previous version of the app on Heroku, app still runs (once again rollback doesn't affect S3Blobs.java).

Could the way Heroku is compiling my Java code have changed?

2014-09-27T22:24:13.874161+00:00 app[web.2]: ~ _ _

2014-09-27T22:24:13.874166+00:00 app[web.2]: ~ _ __ | | __ _ _ | | 2014-09-27T22:24:13.874167+00:00 app[web.2]: ~ | ' | |/ ' | || || 2014-09-27T22:24:13.874189+00:00 app[web.2]: ~ | /||\|\ () 2014-09-27T22:24:13.874190+00:00 app[web.2]: ~ || |/

2014-09-27T22:24:13.874192+00:00 app[web.2]: ~ 2014-09-27T22:24:13.874193+00:00 app[web.2]: ~ play! 1.2.7, http://www.playframework.org 2014-09-27T22:24:13.874195+00:00 app[web.2]: ~ framework ID is prod 2014-09-27T22:24:13.874196+00:00 app[web.2]: ~ 2014-09-27T22:24:13.874197+00:00 app[web.2]: ~ Ctrl+C to stop 2014-09-27T22:24:13.874199+00:00 app[web.2]: ~ 2014-09-27T22:24:13.623029+00:00 app[web.1]: 22:24:13,622 INFO ~ Connected to jdbc:postgresql://ec2-23-23-147-250.compute-1.amazonaws.com:5582/d7p28cauibod8d?ssl=true&sslfactory=org.postgresql.ssl.NonValidatingFactory 2014-09-27T22:24:14.543346+00:00 heroku[web.2]: Process exited with status 1 2014-09-27T22:24:14.553771+00:00 heroku[web.2]: State changed from starting to crashed 2014-09-27T22:24:14.807111+00:00 app[web.1]: 22:24:14,806 INFO ~ Application 'tfp_v2_fix' is now started ! 2014-09-27T22:24:14.956075+00:00 app[web.1]: 2014-09-27T22:24:14.956085+00:00 app[web.1]: Starting S3Blobs 2014-09-27T22:24:14.956088+00:00 app[web.1]: 2014-09-27T22:24:16Z app[postgres.12036]: [OLIVE] could not receive data from client: Connection reset by peer 2014-09-27T22:24:16Z app[postgres.12035]: [OLIVE] could not receive data from client: Connection reset by peer 2014-09-27T22:24:16Z app[postgres.12034]: [OLIVE] could not receive data from client: Connection reset by peer 2014-09-27T22:24:16.122974+00:00 app[web.1]: Sep 27, 2014 22:24:16 +0000 [3 5] com.newrelic.agent.Agent INFO: New Relic Agent has shutdown 2014-09-27T22:24:16.116801+00:00 app[web.1]: 22:24:16,115 ERROR ~ 2014-09-27T22:24:16.116807+00:00 app[web.1]: 2014-09-27T22:24:16.116810+00:00 app[web.1]: @6jk9a374i 2014-09-27T22:24:16.116811+00:00 app[web.1]: Error during job execution (s3.S3Blobs) 2014-09-27T22:24:16.116813+00:00 app[web.1]: 2014-09-27T22:24:16.116815+00:00 app[web.1]: Execution exception (In /app/s3/S3Blobs.java around line 32) 2014-09-27T22:24:16.116817+00:00 app[web.1]: IllegalStateException occured : Unsupported cookie spec: default 2014-09-27T22:24:16.116818+00:00 app[web.1]: 2014-09-27T22:24:16.116820+00:00 app[web.1]: play.exceptions.JavaExecutionException: Unsupported cookie spec: default 2014-09-27T22:24:16.116821+00:00 app[web.1]: at play.jobs.Job.call(Job.java:155) 2014-09-27T22:24:16.116823+00:00 app[web.1]: at Invocation.Job(Play!) 2014-09-27T22:24:16.116824+00:00 app[web.1]: Caused by: java.lang.IllegalStateException: Unsupported cookie spec: default 2014-09-27T22:24:16.116826+00:00 app[web.1]: at org.apache.http.cookie.CookieSpecRegistry.getCookieSpec(CookieSpecRegistry.java:110) 2014-09-27T22:24:16.116828+00:00 app[web.1]: at org.apache.http.cookie.CookieSpecRegistry$1.create(CookieSpecRegistry.java:163) 2014-09-27T22:24:16.116829+00:00 app[web.1]: at org.apache.http.client.protocol.RequestAddCookies.process(RequestAddCookies.java:157) 2014-09-27T22:24:16.116831+00:00 app[web.1]: at org.apache.http.protocol.ImmutableHttpProcessor.process(ImmutableHttpProcessor.java:132) 2014-09-27T22:24:16.116833+00:00 app[web.1]: at org.apache.http.protocol.HttpRequestExecutor.preProcess(HttpRequestExecutor.java:166) 2014-09-27T22:24:16.116834+00:00 app[web.1]: at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:485) 2014-09-27T22:24:16.116835+00:00 app[web.1]: at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:878) 2014-09-27T22:24:16.116837+00:00 app[web.1]: at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:84) 2014-09-27T22:24:16.116838+00:00 app[web.1]: at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:109) 2014-09-27T22:24:16.116839+00:00 app[web.1]: at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57) 2014-09-27T22:24:16.116840+00:00 app[web.1]: at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:268) 2014-09-27T22:24:16.799309+00:00 heroku[web.1]: Process exited with status 1 2014-09-27T22:24:16.809162+00:00 heroku[web.1]: State changed from starting to crashed 2014-09-27T22:24:16.116842+00:00 app[web.1]: at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:170) 2014-09-27T22:24:16.116843+00:00 app[web.1]: at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:2597) 2014-09-27T22:24:16.116845+00:00 app[web.1]: at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:2569) 2014-09-27T22:24:16.116846+00:00 app[web.1]: at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:415) 2014-09-27T22:24:16.116847+00:00 app[web.1]: at com.amazonaws.services.s3.AmazonS3Client.doesBucketExist(AmazonS3Client.java:723) 2014-09-27T22:24:16.116849+00:00 app[web.1]: at s3.S3Blobs.doJob(S3Blobs.java:32) 2014-09-27T22:24:16.116850+00:00 app[web.1]: at play.jobs.Job.doJobWithResult(Job.java:50) 2014-09-27T22:24:16.116851+00:00 app[web.1]: at play.jobs.Job.call(Job.java:146) 2014-09-27T22:24:16.116853+00:00 app[web.1]: ... 1 more 2014-09-27T22:24:16.118780+00:00 app[web.1]: Exception in thread "main" play.exceptions.JavaExecutionException: Unsupported cookie spec: default 2014-09-27T22:24:16.118817+00:00 app[web.1]: at play.jobs.Job.call(Job.java:155) 2014-09-27T22:24:16.118846+00:00 app[web.1]: at Invocation.Job(Play!) 2014-09-27T22:24:16.118882+00:00 app[web.1]: Caused by: java.lang.IllegalStateException: Unsupported cookie spec: default 2014-09-27T22:24:16.118912+00:00 app[web.1]: at org.apache.http.cookie.CookieSpecRegistry.getCookieSpec(CookieSpecRegistry.java:110) 2014-09-27T22:24:16.118942+00:00 app[web.1]: at org.apache.http.cookie.CookieSpecRegistry$1.create(CookieSpecRegistry.java:163) 2014-09-27T22:24:16.118973+00:00 app[web.1]: at org.apache.http.client.protocol.RequestAddCookies.process(RequestAddCookies.java:157) 2014-09-27T22:24:16.118998+00:00 app[web.1]: at org.apache.http.protocol.ImmutableHttpProcessor.process(ImmutableHttpProcessor.java:132) 2014-09-27T22:24:16.119030+00:00 app[web.1]: at org.apache.http.protocol.HttpRequestExecutor.preProcess(HttpRequestExecutor.java:166) 2014-09-27T22:24:16.119060+00:00 app[web.1]: at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:485) 2014-09-27T22:24:16.119091+00:00 app[web.1]: at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:878) 2014-09-27T22:24:16.119112+00:00 app[web.1]: at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:84) 2014-09-27T22:24:16.119145+00:00 app[web.1]: at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:109) 2014-09-27T22:24:16.119174+00:00 app[web.1]: at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57) 2014-09-27T22:24:16.119206+00:00 app[web.1]: at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:268) 2014-09-27T22:24:16.119227+00:00 app[web.1]: at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:170) 2014-09-27T22:24:16.119258+00:00 app[web.1]: at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:2597) 2014-09-27T22:24:16.119288+00:00 app[web.1]: at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:2569) 2014-09-27T22:24:16.119317+00:00 app[web.1]: at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:415) 2014-09-27T22:24:16.119342+00:00 app[web.1]: at com.amazonaws.services.s3.AmazonS3Client.doesBucketExist(AmazonS3Client.java:723) 2014-09-27T22:24:16.119374+00:00 app[web.1]: at s3.S3Blobs.doJob(S3Blobs.java:32) 2014-09-27T22:24:16.119404+00:00 app[web.1]: at play.jobs.Job.doJobWithResult(Job.java:50) 2014-09-27T22:24:16+00:00 app[heroku-postgres]: source=HEROKUPOSTGRESQL_OLIVE sample#current_transaction=1953 sample#db_size=6842552bytes sample#tables=1 sample#active-connections=5 sample#waiting-connections=0 sample#index-cache-hit-rate=0.97895 sample#table-cache-hit-rate=0.99444 sample#load-avg-1m=0.535 sample#load-avg-5m=0.375 sample#load-avg-15m=0.285 sample#read-iops=0 sample#write-iops=20.195 sample#memory-total=15405616kB sample#memory-free=243784kB sample#memory-cached=14421060kB sample#memory-postgres=253500kB 2014-09-27T22:24:16.119434+00:00 app[web.1]: at play.jobs.Job.call(Job.java:146) 2014-09-27T22:24:16.119459+00:00 app[web.1]: ... 1 more 2014-09-27T22:24:16.121017+00:00 app[web.1]: Sep 27, 2014 22:24:16 +0000 [3 5] com.newrelic.agent.Agent INFO: JVM is shutting down

Answer 1

Heroku syncs your dependencies.yml file everytime you push your app to them. One of my dependencies ended up being out of date, and heroku automatically grabbed a newer version of the file, which in turn broke the Amazon dependency. Updating my Amazon to the latest version ended up fixing the problem.

I hadn't touched my dependenies.yml file for weeks, and also hadn't run "play deps --sync" for weeks, so didn't think to look there on my local machine.

Answer 2

Same problem here ("IllegalStateException occured : Unsupported cookie spec: default"), not even on Play framework nor Heroku but in a classic Java web app (Mifos open source microfinance platform), and solved for us by an upgrade of the rest-assured test framework from v1.7.2 to latest v2.3.3 (and probably some of its implicit transitive dependencies which may be the root cause more so than rest-assured itself, perhaps the change from org.apache.httpcomponents httpclient from v4.2.3 to currently latest v4.3.5) - clearly some version mismatch is the root cause of this one.

Answer 3

org.apache.httpcomponents httpclient seems to be the root cause for me.

Anything after version 4.3.5 will cause this error in libraries that use it (and aren't ready for it), so check which version you have.

Answer 4

Apache httpclient 4.3.6 works fine, use this version.