Taming systemd units
People love to dunk on systemd, but a unit file is the most honest contract you can hand a server: here is the process, here is when to start it, here is what to do when it dies. Stop nohup-ing things in a screen session. Write the unit.
My baseline for a long-running service looks like this:
[Service]
ExecStart=/opt/app/run
Restart=on-failure
RestartSec=5
NoNewPrivileges=true
ProtectSystem=strict
The two lines that earn their keep are Restart=on-failure and
ProtectSystem=strict. The first means I stop getting 3 a.m. pages for
things that just needed a kick. The second means a compromised process can't casually
scribble over /usr.
Then systemctl enable --now app, check journalctl -u app -f,
and go to bed. That's the whole religion.