AWS CodeDeploy: Java application is not up despite being successful deployment

Question

I am struggling with a strange problem. I am trying to deploy a tomcat based java servlet application on ec2 using AWS CodeBuild and CodeDeploy.

After several tutorials and videos, I successfully set it up and it is indeed deploying application on my ec2, but my service is not responding on desired URLs. I tried to find processes on port 8080 but none available.

my BuildSpec.yml looks like below

version: 0.2
env:
  variables:
    JAVA_HOME: "/usr/lib/jvm/java-8-openjdk-amd64"
phases:
  install:
    commands:
      - echo installing maven...
      - apt-get update -y
      - apt-get install -y maven
  build:
    commands:
      - echo building Mudoku-webapp
      - mvn install
artifacts:
  files:
    - target/*.war
    - target/dependency/webapp-runner.jar
    - scripts/*.sh
    - appspec.yml
  discard-paths: yes
cache:
  paths:
    - '/root/.m2/**/*'

My appspec.yml looks like below:

version: 0.0
os: linux
files:
  - source: /
    destination: /home/ubuntu
hooks:
  AfterInstall:
    - location: fix_previleges.sh
      timeout: 300
      runas: root
  ApplicationStop:
    - location: stop_server.sh
      timeout: 300
      runas: root
  ApplicationStart:
    - location: start_server.sh
      timeout: 300
      runas: root

scripts that are executing during deployment are

fix_previleges.sh:

#!/bin/bash

echo "Setting executable previliges for all the scripts and required jars"
chmod +x /home/ubuntu/*.jar
chmod +x /home/ubuntu/*.sh
echo "Done setting the previliges for needed scripts and jars"

start_server.sh:

#!/bin/bash

echo "deploying Mudoku-webapp.war"
java -jar webapp-runner.jar *.war </dev/null &>/dev/null &
echo "done deploying Mudoku-webapp.war"

stop_server.sh:

#!/bin/bash

echo "Killing process on port 8080"
kill -9 $(lsof -t -i:8080)
echo "Done killing the process on port 8080"

Deployment logs are looking as expected

[2020-04-25 15:48:42.336] [d-OB5YU0VQ2]LifecycleEvent - ApplicationStop
[2020-04-25 15:48:42.336] [d-OB5YU0VQ2]Script - stop_server.sh
[2020-04-25 15:48:42.389] [d-OB5YU0VQ2][stdout]Killing process on port 8080
[2020-04-25 15:48:42.398] [d-OB5YU0VQ2][stderr]kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec]
[2020-04-25 15:48:42.398] [d-OB5YU0VQ2][stdout]Done killing the process on port 8080
[2020-04-25 15:48:46.679] [d-OB5YU0VQ2]LifecycleEvent - AfterInstall
[2020-04-25 15:48:46.680] [d-OB5YU0VQ2]Script - fix_previleges.sh
[2020-04-25 15:48:46.736] [d-OB5YU0VQ2][stdout]Setting executable previliges for all the scripts and required jars
[2020-04-25 15:48:46.738] [d-OB5YU0VQ2][stdout]Done setting the previliges for needed scripts and jars
[2020-04-25 15:48:47.723] [d-OB5YU0VQ2]LifecycleEvent - ApplicationStart
[2020-04-25 15:48:47.723] [d-OB5YU0VQ2]Script - start_server.sh
[2020-04-25 15:48:47.775] [d-OB5YU0VQ2][stdout]deploying Mudoku-webapp.war
[2020-04-25 15:48:47.776] [d-OB5YU0VQ2][stdout]done deploying Mudoku-webapp.war

Error in kill was expected as there was no process running on port 8080 but I don't think it impacts further execution.

The last two statements in the log clearly imply that start_server.sh have been successfully executed.

But to no benefit, the result look like this

Interestingly, AWS CodePipeline copied all the desired build artifacts are copied as expected.

If I run start_server.sh on my ec2 instance manually using command

ubuntu@ip-172-31-27-194:~$ ls
Mudoku-webapp.war  appspec.yml  fix_previleges.sh  install  start_server.sh  stop_server.sh  webapp-runner.jar
ubuntu@ip-172-31-27-194:~$ ./start_server.sh 
deploying Mudoku-webapp.war
done deploying Mudoku-webapp.war
ubuntu@ip-172-31-27-194:~$

It works fine and service become available

Any answer and hint is highly appreciated as I already spent two days to debug this to no gain.

shariqmaws shariqmaws · Accepted Answer · 2020-04-29T10:42:18

Clearly your application is not started by CodeDeploy. The fact that you see two lines:

"deploying Mudoku-webapp.war"

&

"done deploying Mudoku-webapp.war"

means nothing, the interesting output is from:

java -jar webapp-runner.jar *.war </dev/null &>/dev/null &

which is redirected to null. Does your application need an interactive terminal to work? Can you redirect the output of java execution to a file and inspect that for any clues.

AWS CodeDeploy: Java application is not up despite being successful deployment

1 Answers