Java application as a systemd service

Question

I'm trying to run a java application as a system service in CentOS7. The jar should be run with a specific user: appuser. I have a shell script that runs the jar with the following command. The whole script is much bigger because it also handles stop, restart and status, but this is the start part:

servicename="myservice"
user="appuser"
pid_file="/var/run/$servicename.pid"
get_pid_from_file() {
  cat "$pid_file"
}
get_pids() {
  (ps -ef | grep myjar | grep $user | grep -v grep | awk '{print $2}')
}

is_running() {
  ! [ -z "`get_pids`" ] || ([ -f "$pid_file" ] && ps `get_pid_from_file` > /dev/null 2>&1)
}
case "$1" in
  start)
    if is_running; then
      echo "Already started"
    else
      case "$2" in
        *)
    su -s /bin/sh $user -c "cd /app/myworkingdir ; java -jar myjar.jar >> /var/log/systemout.log 2>> /var/log/systemerr.log" &
    pid=`ps -ef | grep myjar | grep $user | grep -v grep | awk '{print $2}' | tail -1`
    echo $pid
    echo $pid > $pid_file

When I run the script from the command line, it will start the jar and write the pidfile. I use the tail in the command to get the PID, because I actually have 3 processes: the su, the /bin/sh and the actual java -jar command.

Now I also have a systemctl script in /etc/systemd/system/multi-user.target.wants/myservice.service, which looks like this:

[Unit]
Description=myservice
After=syslog.target
After=network.target

[Service]
Type=simple

WorkingDirectory=/app/myworkingdir/run

PIDFile=/var/run/myservice.pid
ExecStart=/app/myworkingdir/run/myscript.sh start
ExecStop=/app/myworkingdir/run/myscript.sh stop

User=appuser
Group=appgrp

Restart=always
RestartSec=30
StartLimitInterval=60
StartLimitBurst=5
TimeoutStartSec=30

LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

The problem is that the systemctl start myservice.service will not start the service correctly.

I see this as I run journeyctl -xe:

May 21 13:03:23 myserver.com systemd[1]: Starting my service...
-- Subject: Unit myservice.service has begun start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit myservice.service has begun starting up.
May 21 13:03:23 myserver.com polkitd[619]: Unregistered Authentication Agent for unix-process:19329:16
May 21 13:03:23 myserver.com myscript.sh[19335]: Already started
May 21 13:03:24 myserver.com myscript.sh[19345]: Stopping myscript.sh..

And here is logging that is in /var/log/messages:

[root@myserver run]# systemctl start myservice.service
May 21 13:34:00 myserver systemd: myservice.service holdoff time over, scheduling restart.
May 21 13:34:00 myserver systemd: Started myservice.
May 21 13:34:00 myserver systemd: Starting myservice...
[root@myserver run]# May 21 13:34:00 ctor-app52 myscript.sh: Already started
May 21 13:34:00 myserver denver.sh: Stopping myscript.sh..
May 21 13:34:31 myserver systemd: myservice.service holdoff time over, scheduling restart.
May 21 13:34:31 myserver systemd: Started myservice.
May 21 13:34:31 myserver systemd: Starting myservice...
May 21 13:34:31 myserver myscript.sh: Already started
May 21 13:34:31 myserver myscript.sh: Stopping myscript.sh..
May 21 13:35:01 myserver systemd: Started Session 122559 of user root.
May 21 13:35:01 myserver systemd: Starting Session 122559 of user root.
May 21 13:35:01 myserver su: (to appuser) root on none
May 21 13:35:01 myserver systemd: LPdenver.service holdoff time over, scheduling restart.
May 21 13:35:01 myserver systemd: Started myservice.
May 21 13:35:01 myserver systemd: Starting myservice...
May 21 13:35:01 myserver myscript.sh: Already started
May 21 13:35:01 myserver myscript.sh: Stopping myscript.sh..
May 21 13:35:31 myserver systemd: myservice.service holdoff time over, scheduling restart.
May 21 13:35:31 myserver systemd: Started myservice.
May 21 13:35:31 myserver systemd: Starting myservice...
May 21 13:35:31 myserver myscript.sh: Already started
May 21 13:35:31 myserver myscript.sh: Stopping myscript.sh..
May 21 13:36:01 myserver systemd: myservice.service holdoff time over, scheduling restart.
May 21 13:36:01 myserver systemd: Started myservice.
May 21 13:36:01 myserver systemd: Starting myservice...
May 21 13:36:01 myserver myscript.sh: Already started
May 21 13:36:02 myserver myscript.sh: Stopping myscript.sh..

What am I doing wrong?

Answer 1

You are specifying a user in your .service file, hence you should not need to do any of the su magic in your script.

Replace this:

su -s /bin/sh $user -c "cd /app/myworkingdir ; java -jar myjar.jar >> /var/log/systemout.log 2>> /var/log/systemerr.log"

with this:

cd /app/myworkingdir
java -jar myjar.jar >> /var/log/systemout.log 2>> /var/log/systemerr.log"

Besides, I see two further issues in your script:

• You are grepping the process table to find the PID for your service. This may fail if there is another running process with a command line that contains the same characters (the system may end up picking the wrong process to terminate). echo $!>$pid_file right after launching java is a safer way of accomplishing this.

• The service type is simple, yet your script forks it as a separate process and then returns. This will confuse systemd, and your service will fail to start.

Both of these might be easy to solve. I assume you need the PID only to stop the service, and doing so is as simple as sending it a SIGINT. In that case you can take advantage of the fact that, in the absence of ExecStop, systemd will simply send a SIGINT to the service process to stop it.

• Drop the ampersand after the java call, and prefix it with exec instead. That way, the java process becomes will be treated as the daemon process by systemd.

• Drop the PID magic after the java call altogether.

• In the .service file, remove the ExecStop entry.

• Instead, also under [Service], add SuccessExitStatus=143. (When terminated with a signal, the JVM will exit with a nonzero exit status, namely 128 plus the signal. By default, systemd treats an exit status of 143 as a failure; this entry will tell systemd that 143 stands for graceful exit.)

You could take this one step further and drop the script altogether:

• Specify the java command line as ExecStart
• Pass the full path to the java binary (as reported by which java)
• Drop the redirections: by default, systemd will redirect stdout and stderr to the journal. (This can be configured in the .service file as well.)