How to run an always-on agent on Linux: the core steps

The ten steps

1. Create a user that exists only to run the agent. No password, no login shell, a home under /srv. I never let an agent run as root, and never as me.
2. Give it a virtual environment and the program. A venv the user owns, with the agent inside it. For the test, the agent is a twelve-line Python loop that writes a heartbeat file every ten seconds.
3. Put the secrets in a file only root can read. An EnvironmentFile with the API key and configuration. The unit file references it and never contains a secret.
4. Write the service unit. Restart=always, RestartSec=5, MemoryMax, and the sandboxing options that cost nothing: NoNewPrivileges, ProtectSystem=strict, ProtectHome, PrivateTmp, with the agent's folder listed as writable.
5. Write a timer for anything that runs on a clock. A oneshot service plus a timer with OnCalendar and Persistent=true, so a run missed during downtime happens as soon as the machine is back.
6. Verify the units before you enable them. systemd-analyze verify on every unit file. It catches typos that would otherwise fail silently at boot.
7. Enable and start, then read the journal. systemctl enable --now for the service and the timer; journalctl -u is where your agent's output goes from now on.
8. Kill it on purpose and watch it come back. A SIGKILL to the main process, then five seconds later the service is active with a new PID. If this does not happen, nothing else matters.
9. Use flock if you must use cron. Cron has no idea whether the last run finished. A lock makes an overlapping run refuse to start instead of doubling up.
10. Watch the heartbeat. A tiny script that fails when the heartbeat is older than a minute, run on its own timer, with its failure wired to whatever reaches you: email, a chat message, a push notification.

1. The user, the venv and a tiny agent

Terminal, as rootuseradd --system --create-home --home-dir /srv/agent --shell /usr/sbin/nologin agent
id agent
# uid=997(agent) gid=997(agent) groups=997(agent)
python3 -m venv /srv/agent/venv
chown -R agent:agent /srv/agent

For the test I wanted an agent with no dependencies and one observable behaviour, so it writes a timestamp to a heartbeat file every ten seconds and prints a line. Your real agent goes in the same place and gets started the same way; the point of the stand-in is that every step after this one is about systemd, not about the agent.

/srv/agent/agent.pyimport os, time, datetime
name = os.environ.get("AGENT_NAME", "unnamed")
while True:
    stamp = datetime.datetime.now(datetime.timezone.utc).isoformat(timespec="seconds")
    with open("/srv/agent/heartbeat", "w") as fh:
        fh.write(stamp + "\n")
    print(f"{name} alive at {stamp}", flush=True)
    time.sleep(10)

2. Secrets outside the unit

The unit file ends up in /etc/systemd/system, readable by every user on the box, and it shows up in systemctl cat. It must never contain a key. The EnvironmentFile pattern keeps the secrets in a file that only root can read; systemd reads it as root before dropping to the agent user, so the agent sees the variables and nothing else sees the file.

Terminal, as rootprintf 'AGENT_NAME=demo\nOPENAI_API_KEY=sk-not-a-real-key\n' > /etc/agent.env
chmod 600 /etc/agent.env && chown root:root /etc/agent.env
ls -l /etc/agent.env
# -rw------- 1 root root 49 Oct 20 19:38 /etc/agent.env

3. The service unit, line by line

/etc/systemd/system/agent.service[Unit]
Description=Always-on agent (demo)
After=network-online.target
Wants=network-online.target

[Service]
User=agent
Group=agent
WorkingDirectory=/srv/agent
EnvironmentFile=/etc/agent.env
ExecStart=/srv/agent/venv/bin/python /srv/agent/agent.py
Restart=always
RestartSec=5
MemoryMax=512M
NoNewPrivileges=yes
ProtectSystem=strict
ProtectHome=yes
ReadWritePaths=/srv/agent
PrivateTmp=yes

[Install]
WantedBy=multi-user.target

I will take the lines I actually care about, because the rest is boilerplate.

• After and Wants network-online.target. Start only once the network is up. On my Mac the lack of this exact guarantee lost me a day's post; systemd gives it to you for free, with the caveat that online means whatever your network manager says it means.
• Restart=always, RestartSec=5. Come back after any exit, five seconds later. Not on-failure: an agent that exits cleanly because it thinks it is finished should also come back, because an always-on agent is never finished.
• MemoryMax=512M. A hard ceiling. A leaking agent gets killed and restarted rather than taking the box down. The status output below shows the limit applied.
• NoNewPrivileges, ProtectSystem=strict, ProtectHome, PrivateTmp. The agent cannot escalate, cannot write anywhere except what you list in ReadWritePaths, cannot see home directories, and has its own /tmp. These four lines are the cheapest security you will ever buy.
• EnvironmentFile. Secrets come from the root-owned file above.

One default worth knowing: systemd's start rate limit. Out of the box, a unit that restarts more than five times in ten seconds is put into a failed state and left there. With RestartSec=5 you will not hit it, but if you ever lower RestartSec to one second for a crashy agent, add StartLimitIntervalSec=0 to the unit's [Unit] section or you will discover the limit at 3am.

4. A timer for the clock-driven jobs

Most real agents have both shapes of work: a loop that runs all the time, and jobs that run at a time. The morning brief is the classic. For those I use a oneshot service plus a timer, and the single most important line is Persistent=true, which is the catch-up property I keep going on about in this series: if the machine was off at 07:00, the job runs as soon as it is back, instead of being skipped until tomorrow.

/etc/systemd/system/agent-brief.service[Unit]
Description=Morning brief (one run)

[Service]
Type=oneshot
User=agent
EnvironmentFile=/etc/agent.env
ExecStart=/srv/agent/venv/bin/python /srv/agent/brief.py

/etc/systemd/system/agent-brief.timer[Unit]
Description=Run the morning brief at 07:00

[Timer]
OnCalendar=*-*-* 07:00:00
Persistent=true
RandomizedDelaySec=120

[Install]
WantedBy=timers.target

RandomizedDelaySec is a courtesy to whatever API you call at 07:00, and to yourself if you ever run ten of these on one box.

5. Verify, enable, start, read

Terminal, as rootsystemd-analyze verify /etc/systemd/system/agent.service \
    /etc/systemd/system/agent-brief.service /etc/systemd/system/agent-brief.timer
# (no output means no complaints)
systemctl daemon-reload
systemctl enable --now agent.service agent-brief.timer
# Created symlink /etc/systemd/system/multi-user.target.wants/agent.service -> /etc/systemd/system/agent.service.
# Created symlink /etc/systemd/system/timers.target.wants/agent-brief.timer -> /etc/systemd/system/agent-brief.timer.

What I saw three seconds later$ systemctl is-active agent.service
active
$ systemctl status agent.service --no-pager | head -8
* agent.service - Always-on agent (demo)
     Loaded: loaded (/etc/systemd/system/agent.service; enabled; preset: enabled)
     Active: active (running) since Tue 2026-10-20 19:38:54 UTC; 3s ago
   Main PID: 168 (python)
      Tasks: 1 (limit: 9396)
     Memory: 2.8M (max: 512.0M available: 509.1M peak: 3.0M)
        CPU: 28ms
$ journalctl -u agent.service -n 2 --no-pager
Oct 20 19:38:54 51889e260f7e systemd[1]: Started agent.service - Always-on agent (demo).
Oct 20 19:38:54 51889e260f7e python[168]: demo alive at 2026-10-20T19:38:54+00:00

That Memory: 2.8M (max: 512.0M ...) line is the MemoryMax in force. The heartbeat file held the same timestamp as the journal line. And the timer showed its next run, with the randomised delay already applied to the 07:00 slot:

Terminal$ systemctl list-timers agent-brief.timer --no-pager
NEXT                        LEFT LAST PASSED UNIT              ACTIVATES
Wed 2026-10-21 07:00:33 UTC  11h -         - agent-brief.timer agent-brief.service

6. Kill it and watch it come back

This is the test I would never skip, because it is the whole reason to use a service manager instead of a terminal with nohup in it. I sent SIGKILL to the agent's process and waited seven seconds.

Terminal$ kill -9 168
$ sleep 7; systemctl is-active agent.service
active
$ systemctl show -p MainPID --value agent.service
180
$ journalctl -u agent.service --no-pager | grep -i "exited\|restart"
Oct 20 19:38:57 51889e260f7e systemd[1]: agent.service: Main process exited, code=killed, status=9/KILL
Oct 20 19:39:02 51889e260f7e systemd[1]: agent.service: Scheduled restart job, restart counter is at 1.

Killed at 19:38:57, restarted at 19:39:02, five seconds later as configured, new PID, restart counter at one. The heartbeat file picked up on the next loop. If your agent does not do this, you have a script, not a service.

7. If you must use cron, use flock

I prefer timers, and the first post explains why a cron job on its own is not an agent. But cron is everywhere and sometimes it is the pragmatic choice. The one thing cron will not do for you is notice that the previous run is still going. flock -n on a lock file makes the second run refuse instead of overlap, which is what you want for anything that posts, sends or writes.

Terminal$ flock -n /tmp/agent-brief.lock /srv/agent/venv/bin/python -c 'print("one run, lock held")'
one run, lock held
$ flock -n /tmp/agent-brief.lock sleep 3 &
$ flock -n /tmp/agent-brief.lock echo "second run" || echo "second run refused: lock held"
second run refused: lock held

crontab -e, as the agent user# m h dom mon dow command
0 7 * * * flock -n /tmp/agent-brief.lock /srv/agent/venv/bin/python /srv/agent/brief.py >> /srv/agent/brief.log 2>&1

Note what cron still lacks even with the lock: a missed 07:00 is gone, there is no journal, and the environment is nearly empty, the same trap launchd sets on a Mac. The timer gives you all three, which is why it is step five and cron is step nine.

8. The watchdog

An always-on agent that dies quietly is worse than no agent, because you have stopped looking. The heartbeat file the demo agent writes is the hook. The watchdog is a four-line shell script that exits non-zero when the heartbeat is older than a minute; put it on its own timer every few minutes, and make its failure reach you. On my own site, the equivalent job emails me only when an invariant breaks, and the absence of that email is a signal I have learned to trust because I built the job to fail loudly.

/usr/local/bin/agent-watchdog#!/bin/sh
age=$(( $(date +%s) - $(stat -c %Y /srv/agent/heartbeat) ))
if [ "$age" -gt 60 ]; then echo "heartbeat is ${age}s old"; exit 1; fi
echo "heartbeat ok (${age}s)"

Terminalchmod +x /usr/local/bin/agent-watchdog
/usr/local/bin/agent-watchdog
# heartbeat ok (2s)

Wire the alert in the same unit with an OnFailure= line pointing at a service that sends the message, or simply have the watchdog script post to a chat webhook when it fails. The mechanism matters less than the rule: silence must become a notification, not a comfort.

What this test did not cover

A container with systemd is a faithful model of units, restarts, timers and permissions. It is not a faithful model of three things, and the distro pages pick them up where it matters.

• Boot. The container was started, not booted; I did not test that the service comes up after a real reboot, although enable is exactly the mechanism that makes it do so. Reboot your box once and check.
• Real networks. network-online.target in a container means little. On a VPS it means what NetworkManager or systemd-networkd says, and on a laptop-grade Wi-Fi link it can still lie. Keep a resolve-and-wait loop in the agent itself.
• Mandatory access control. No SELinux here. On Fedora it is on by default and it will stop a service in /srv without telling you why; that is the first thing the Fedora page covers.

Everything else, the user, the venv, the EnvironmentFile, the restart, the MemoryMax, the timer with Persistent, flock and the watchdog, ran exactly as shown, and I deleted the container when I was done. The cloud post gets you a box to do this on; the Windows post gets you here through WSL2; the permissions post is the argument for the user and the identity. The whole series is on one page, and if you would rather see an always-on agent than host one, the one I run for social video is here.

Sources

1. freedesktop.org, systemd.service: service unit configuration (accessed October 20, 2026)
2. freedesktop.org, systemd.timer: timer unit configuration (accessed October 20, 2026)
3. freedesktop.org, systemd.exec: execution environment configuration (sandboxing and EnvironmentFile) (accessed October 20, 2026)
4. freedesktop.org, systemd.unit: StartLimitIntervalSec and StartLimitBurst (accessed October 20, 2026)

Written by Lera Leonteva. Outputs are from a fresh Ubuntu 24.04 container running systemd 255, captured on the access date.