bash script runs in terminal, but doesn't auto-start with systemd

Question

I'm trying to run a bash script on startup as a systemd service, I'm doing this on a Raspberry Pi 4 with Raspbian Buster Lite.

I'm able to execute the bash script if I run it manually ./hls.sh and I am able to also run the service if I do sudo service tv start but the tv.service does not appear to be able to execute the bash script hls.sh on start. I did give permissions chmod 777 for both the service and the bash file as well.

Any help here would be appreciated, I've been trying to figure this out on and off for a month now.

Edit: Using the suggestions by Carl, I modified the files. However, it still doesn't work. I noticed also that when Type=oneshot you can't do a Restart=always, and if I do a restorecon -r there's an error that says the command isn't found. Per Carl's suggestion I put everything in /opt I decided not to use the temp file suggestion because I need to have the ffmpeg output go to the same place every time (my understanding is that the other way would randomly generate a folder?).

Edit2: Issue was related to this systemd issue: https://unix.stackexchange.com/questions/209832/debian-systemd-network-online-target-not-working the workaround was simply to do RestartSec=5s under [Service]

Bash Script [Before]

#!/bin/bash
/usr/bin/ffmpeg -reconnect 1 -reconnect_at_eof 1 -reconnect_streamed 1 -reconnect_delay_max 2 -y -nostdin \
-hide_banner -loglevel fatal \
-i http://10.0.0.11:9981/stream/channelnumber/2 \
-vcodec copy -acodec copy -scodec copy -g 60 \
-fflags +genpts -user_agent HLS_delayer \
-metadata service_provider="TimeShift" \
-metadata service_name="TV 1" \
-f hls -hls_flags delete_segments \
-hls_time 60 \
-hls_list_size 480 \
-hls_wrap 481 \
-hls_segment_filename /home/pi/hls/1_%03d.ts /home/pi/hls/1_hls.m3u8

Bash Script [After]

#!/usr/bin/env bash
set -e -o pipefail # return exit code of command with non-zero exit code and stop executing
echo "The solution works!"
/usr/bin/ffmpeg -reconnect 1 -reconnect_at_eof 1 -reconnect_streamed 1 -reconnect_delay_max 2 -y -nostdin \
-hide_banner -loglevel fatal \
-i http://10.0.0.11:9981/stream/channelnumber/2 \
-vcodec copy -acodec copy -scodec copy -g 60 \
-fflags +genpts -user_agent HLS_delayer \
-metadata service_provider="TimeShift" \
-metadata service_name="JSTV 1" \
-f hls -hls_flags delete_segments \
-hls_time 60 \
-hls_list_size 480 \
-hls_wrap 481 \
-hls_segment_filename /opt/hls/tmp/1_%03d.ts /opt/hls/tmp/1_hls.m3u8

Service [Before]

[Unit]
Description=Timeshift TV
After=tvheadend.service
PartOf=tvheadend.service
Restart=always

[Service]
ExecStartPre=/bin/mkdir -p /home/pi/hls/
ExecStart=/home/pi/hls.sh 103 &
ExecStop=/bin/rm -rf /home/pi/hls

[Install]
WantedBy=default.target

Service [After]

[Unit]
Description=Timeshift TV
After=tvheadend.service
PartOf=tvheadend.service

[Service]
WorkingDirectory=/opt/hls
User=nobody
Type=oneshot
ExecStartPre=+/bin/mkdir -p -m777 /opt/hls/tmp
ExecStart=/opt/hls/hls.sh
ExecStopPost=+/bin/rm -rf /opt/hls/tmp
#Restart=always

[Install]
WantedBy=multi-user.target

After a reboot, checking the status via systemctl status [Before]

● tv.service - Timeshift TV
   Loaded: loaded (/etc/systemd/system/tv.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Mon 2020-05-18 05:36:40 BST; 45s ago
  Process: 465 ExecStartPre=/bin/mkdir -p -m777 /home/pi/hls/ (code=exited, status=0/SUCCESS)
  Process: 472 ExecStart=/home/pi/hls.sh 103 & (code=exited, status=1/FAILURE)
 Main PID: 472 (code=exited, status=1/FAILURE)

May 18 05:36:39 tv3 systemd[1]: Starting Timeshift TV...
May 18 05:36:39 tv3 systemd[1]: Started Timeshift TV.
May 18 05:36:40 tv3 systemd[1]: tv.service: Main process exited, code=exited, status=1/FAILURE
May 18 05:36:40 tv3 systemd[1]: tv.service: Failed with result 'exit-code'.

After a reboot, checking the status via systemctl status [After]

● tv.service - Timeshift tv
   Loaded: loaded (/etc/systemd/system/tv.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Mon 2020-05-18 20:13:39 BST; 15min ago
  Process: 464 ExecStartPre=/bin/mkdir -p -m777 /opt/hls/tmp (code=exited, status=0/SUCCESS)
  Process: 471 ExecStart=/opt/hls/hls.sh (code=exited, status=1/FAILURE)
  Process: 484 ExecStopPost=/bin/rm -rf /opt/hls/tmp (code=exited, status=0/SUCCESS)
 Main PID: 471 (code=exited, status=1/FAILURE)

May 18 20:13:38 tv3 systemd[1]: Starting Timeshift tv...
May 18 20:13:38 tv3 hls.sh[471]: The solution works!
May 18 20:13:39 tv3 systemd[1]: tv.service: Main process exited, code=exited, status=1/FAILURE
May 18 20:13:39 tv3 systemd[1]: tv.service: Failed with result 'exit-code'.
May 18 20:13:39 tv3 systemd[1]: Failed to start Timeshift tv.

[After] starting the process manually still works

● tv2.service - Timeshift tv
   Loaded: loaded (/etc/systemd/system/tv2.service; enabled; vendor preset: enabled)
   Active: activating (start) since Mon 2020-05-18 20:53:13 BST; 17s ago
  Process: 865 ExecStartPre=/bin/mkdir -p -m777 /opt/hls/tmp (code=exited, status=0/SUCCESS)
 Main PID: 866 (bash)
    Tasks: 2 (limit: 4915)
   Memory: 15.3M
   CGroup: /system.slice/tv2.service
           ├─866 bash /opt/hls/hls.sh
           └─867 /usr/bin/ffmpeg -reconnect 1 -reconnect_at_eof 1 -reconnect_streamed 1 -reconnect_delay_max 2 -y -nostdin -hide_banner -logle

May 18 20:53:13 tv3 systemd[1]: Starting Timeshift tv...
May 18 20:53:13 tv3 hls.sh[866]: The solution works!

pi@tv3:/etc/systemd/system $ cd /opt/hls
pi@tv3:/opt/hls $ ls
hls.sh  tmp
pi@tv3:/opt/hls $ cd tmp
pi@tv3:/opt/hls/tmp $ ls
1_000.ts

Answer 1

There are a few issues with this service file. I'll address them below and then show the final solution.

1. Avoid putting system-level services under the /home directory. While it may work, it is not advised. There are many out-of-scope issues that could occur by doing that. Instead, consider placing the scripts under /opt, /srv, or /usr/local.
   • When moving the files around in a big way like this, make sure to run restorecon -r on that directory after moving it. This ensures SELinux contexts are reset to their correct values, rather than mixing the new context with the old context. This can help solve permission denied errors when SELinux is in enforcing mode.
2. The Restart= directive goes under [Service] not [Unit], so that should be moved there or be removed entirely. It's probably innocuous since systemd is sometimes forgiving, but the systemd.service is the one that defines that directive, not systemd.unit.
3. Change ExecStart= to not have & at the end. Just leave it be and run it as a normal script. Systemd will handle the rest, if properly configured.
4. Change ExecStop= to ExecStopPost= instead. This will allow systemd to do its normal stop behavior, which is to send a SIGTERM to the script when being asked to stop the service. After the stop has been completed (either gracefully or forcefully), the ExecStopPost= will run. Since it appears you're using it to cleanup after the script ran, that should be changed. If left as is, systemd will remove the directory then wait for the script to exit, which the script will never do since it wasn't told to, so systemd will wait the timeout period before asking the kernel to SIGKILL the script. Definitely not ideal.
5. A few extra directives under [Service] would solve some of the assumptions about how this service should behave.
   • WorkingDirectory= — so that it is defined rather than assumed. If left as default, it becomes the home directory of the User, which if also is left default, would be /root. Instead, set it to the location of the script or to /tmp if PrivateTmp=true.
   • User= — so that the script doesn't run as root which is the default user. I suggest nobody or whichever user you wish to have managing the ffmpeg execution. This prevents giving the script root privileges when it runs, which ffmpeg shouldn't need.
   • Type= — this is arguably the most important, since the script is being ran as a one-off and will then exit after succeeding; it doesn't daemonize to the background and continue running as the default simple service would. This should be set to oneshot instead so that systemd knows that; it will treat the script exiting (with a status code of 0) as a successful run of the service, and will move on.
   • PrivateTmp= — In most cases, this should be enabled. Only in rare circumstances should it be disabled. The default behavior is disabled. When enabled, anything the script tries to store in /tmp will actually be transparently stored in a namespaced directory called /tmp/systemd-private-xxxx-servicename.service-xxxxx/, preventing any conflicts with other files in /tmp. If you use this, I would change your ffmpeg to write to /tmp instead of managing a directory next to the script, enabling you to remove ExecStartPre= and ExecStopPost= altogether.
6. Make sure the ExecStartPre= and ExecStopPost= have a + prepended, that way they run as root instead of the defined User= which might not be root; this may or may not be necessary depending on your expectations with the directory. See systemd.service Table 1. Special executable prefixes for more info.
7. Change WantedBy=default.target to multi-user.target; it is more likely that this service is supposed to be ran for that target, rather than whatever target happens to get booted as default.
8. You probably shouldn't make the script chmod 777, I would go with a modest chmod 755 instead. It should only need the read and execute bits (+rx).

So without further adieu, this is the final solution:

Directory Structure: ls -lZ /opt/sfo-61862759

total 8
-rwxr-xr-x. 1 carl users unconfined_u:object_r:usr_t:s0  58 May 18 01:44 sfo-61862759.bash
-rw-r--r--. 1 carl users unconfined_u:object_r:usr_t:s0 316 May 18 01:53 sfo-61862759.service

Shell Script: /opt/sfo-61862759/sfo-61862759.bash

#!/usr/bin/env bash
set -e -o pipefail # return exit code of command with non-zero exit code and stop executing
echo "The solution works!"
# put ffmpeg stuff here and remove the echo if you choose

Service File: /etc/systemd/system/sfo-61862759.service

[Unit]
Description=Stackoverflow Question #61862759 Solution

[Service]
WorkingDirectory=/tmp
User=nobody
Type=oneshot
#ExecStartPre=+/bin/mkdir -p /opt/sfo-61862759/temp
ExecStart=/opt/sfo-61862759/sfo-61862759.bash
#ExecStopPost=+/bin/rm -rf /opt/sfo-61862759/temp
PrivateTmp=true

[Install]
WantedBy=multi-user.target

Systemd Status Output:

01:53 [email protected]:/opt/sfo-61862759$ sudo systemctl start sfo-61862759.service
01:54 [email protected]:/opt/sfo-61862759$ sudo systemctl status sfo-61862759.service -n 4
● sfo-61862759.service - Stackoverflow Question #61862759 Solution
     Loaded: loaded (/opt/sfo-61862759/sfo-61862759.service; linked; vendor preset: disabled)
     Active: inactive (dead)

May 18 01:54:02 utility.localdomain systemd[1]: Starting Stackoverflow Question #61862759 Solution...
May 18 01:54:03 utility.localdomain sfo-61862759.bash[459746]: The solution works!
May 18 01:54:03 utility.localdomain systemd[1]: sfo-61862759.service: Succeeded.
May 18 01:54:03 utility.localdomain systemd[1]: Finished Stackoverflow Question #61862759 Solution.