Running an always-on agent on Ubuntu

The error that stops most people on day one

Since 23.04, Ubuntu ships Python with a marker file that makes pip install refuse to touch the system interpreter. This is PEP 668, and it is correct: apt owns the system Python and pip writing into it is how servers end up with two versions of a library and a broken apt upgrade. But the first time you see it on a fresh box you think something is wrong with the machine. Here is the exact message from my 24.04 container:

Ubuntu 24.04.5, fresh container$ pip install requests
error: externally-managed-environment

× This environment is externally managed
╰─> To install Python packages system-wide, try apt install
    python3-xyz, where xyz is the package you are trying to
    install.

    If you wish to install a non-Debian-packaged Python package,
    create a virtual environment using python3 -m venv path/to/venv.
    Then use path/to/venv/bin/python and path/to/venv/bin/pip. Make
    sure you have python3-full installed.

$ ls /usr/lib/python3*/EXTERNALLY-MANAGED
/usr/lib/python3.12/EXTERNALLY-MANAGED

The fix is the one the message suggests, and it is the step the core page already tells you to do: a virtual environment owned by the agent user. Do not reach for --break-system-packages, which does what it says, and do not delete the marker file, which a future apt upgrade will put back. The agent lives in its own venv, system Python stays apt's.

Terminalsudo apt install -y python3-venv
sudo python3 -m venv /opt/agent/venv
sudo /opt/agent/venv/bin/pip install requests
/opt/agent/venv/bin/python -c "import requests; print(requests.__version__)"
# 2.34.2, inside the venv, no flags, nothing touched system-wide

One Ubuntu-specific gotcha inside the gotcha: python3-venv is a separate package on Ubuntu and is not installed by default on server images. If python3 -m venv tells you ensurepip is missing, that is why. And note the version jump between the two LTS releases: 24.04 ships Python 3.12.3, 26.04.1 ships 3.14.4, and the marker file moves with it. An agent whose dependencies pin an older Python is another argument for the venv, or for the deadsnakes PPA if you truly need a specific interpreter.

apt, not snap, for anything that holds a secret

Ubuntu pushes snaps hard, and for desktop apps I do not mind. For a daemon that holds API tokens I do, for three reasons I have run into. Snaps auto-refresh on their own schedule, so the version of your agent's runtime can change under it at 3am. Snaps are confined, which means the paths your agent can read and write are not the paths you think, and the failure mode is a permission error with no obvious cause. And a snap's data lives under ~/snap/, where your backup script is probably not looking.

So on an agent host I install the runtime from apt (python3, python3-venv, git) or from the project's own installer into /opt, and I would remove snapd entirely on a server I control, which is one command and one reboot. Note that the official ubuntu container images do not include snapd at all, which is itself a hint about what Canonical thinks a minimal Ubuntu runtime needs.

ufw is the firewall, and the agent needs no inbound ports

The core page says deny inbound by default. On Ubuntu the tool is ufw, a front end over nftables that is installed on server images and off by default. An always-on agent of the kind this series builds makes outbound calls and receives nothing, so the only inbound rule you need is SSH, and if you reach the box over Tailscale, not even that from the public interface.

Terminalsudo apt install -y ufw
sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow OpenSSH          # or: sudo ufw allow from 100.64.0.0/10 to any port 22  (Tailscale only)
sudo ufw enable
sudo ufw status verbose

Honesty about my test: inside a plain container ufw cannot talk to the kernel's firewall and prints problem running iptables... Permission denied. In the systemd-booted privileged container the commands ran and ufw reported itself active with the right defaults, below, but a container's network namespace is not a server's and I would not call that a test of the firewall. The commands are the documented ones and I have run them on real Ubuntu servers before; I did not run them on one today.

Ubuntu 26.04.1, privileged container$ sudo ufw status verbose
Status: active
Logging: on (low)
Default: deny (incoming), allow (outgoing), deny (routed)
New profiles: skip

AppArmor is on, and you will mostly never notice

Ubuntu ships AppArmor enabled with profiles for a set of system services. Your own agent, started by systemd from /opt/agent, has no profile and runs unconfined, so it will not fight you. Where AppArmor does bite is when your agent calls a confined program: the one I have seen most is a confined tool refusing to read a file under /opt with a DENIED line in the journal and nothing on the terminal. The check is sudo aa-status, which lists loaded profiles and which are in enforce mode.

This is the piece I could not test today: AppArmor is a kernel feature and the container reports apparmor not present. I can tell you that the apparmor-utils package installs and that 24.04 ships profiles in /etc/apparmor.d, and I can tell you to grep the journal for apparmor="DENIED" the first time something fails silently. I cannot show you a denial from a fresh install, because I did not have one.

unattended-upgrades is on by default, and it is right

This is the Ubuntu behaviour that matters most for an always-on host, and the one people turn off when they should not. On a server install, security updates apply themselves daily. On my fresh 24.04 container the package was present and the two switches were on:

Ubuntu 24.04.5$ cat /etc/apt/apt.conf.d/20auto-upgrades
APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Unattended-Upgrade "1";

$ grep -n 'Automatic-Reboot "' /etc/apt/apt.conf.d/50unattended-upgrades
94://Unattended-Upgrade::Automatic-Reboot "false";

$ sudo unattended-upgrade --dry-run -d 2>&1 | grep -E "Allowed origins|No packages"
Allowed origins are: o=Ubuntu,a=noble, o=Ubuntu,a=noble-security, o=UbuntuESMApps,a=noble-apps-security, o=UbuntuESM,a=noble-infra-security
No packages found that can be upgraded unattended and no pending auto-removals

Note the allowed origins: only the release itself and the security pockets, so unattended upgrades will not move you to a new Python minor version or surprise you with a new systemd. Keep it on. An agent host that nobody logs into for three months needs to patch itself more than a laptop does.

The catch is line 94: automatic reboot is off. So a kernel update lands, the machine keeps running the old kernel, and a file appears at /var/run/reboot-required that nothing reads. My rule is that the agent's own heartbeat message, the one that proves it is alive, includes this check, so I learn about a pending reboot from the agent itself rather than from a security scanner weeks later:

Terminal# a one-line check the agent's own heartbeat can include
test -f /var/run/reboot-required && echo "reboot pending: $(tr '\n' ' ' < /var/run/reboot-required.pkgs)" || echo "no reboot pending"

If you would rather let the box reboot itself, set Automatic-Reboot "true" and Automatic-Reboot-Time "04:00" in the same file. Then your service needs Restart=always and WantedBy=multi-user.target, which the core page's unit already has, so the agent comes back without you. I tested exactly that return-after-death below.

The service, enabled and killed on 26.04

The unit is the core page's unit; nothing Ubuntu-specific in it, which is the point. I installed it in the systemd-booted 26.04 container, enabled it, killed the process with SIGKILL and watched systemd bring it back five seconds later.

/etc/systemd/system/agent.service[Unit]
Description=Always-on agent
After=network-online.target
Wants=network-online.target

[Service]
User=agent
WorkingDirectory=/opt/agent
EnvironmentFile=/etc/agent/env
ExecStart=/opt/agent/venv/bin/python /opt/agent/agent.py
Restart=always
RestartSec=5
NoNewPrivileges=yes
ProtectSystem=strict
ProtectHome=yes
PrivateTmp=yes
ReadWritePaths=/opt/agent/state

[Install]
WantedBy=multi-user.target

Ubuntu 26.04.1, systemd in a privileged container$ grep PRETTY_NAME /etc/os-release
PRETTY_NAME="Ubuntu 26.04.1 LTS"
$ python3 --version && ls /usr/lib/python3*/EXTERNALLY-MANAGED
Python 3.14.4
/usr/lib/python3.14/EXTERNALLY-MANAGED

$ sudo systemd-analyze verify /etc/systemd/system/agent.service && echo verify OK
verify OK
$ sudo systemctl daemon-reload && sudo systemctl enable --now agent.service
Created symlink '/etc/systemd/system/multi-user.target.wants/agent.service' → '/etc/systemd/system/agent.service'.
$ systemctl is-active agent.service; systemctl show agent.service -p NRestarts
active
NRestarts=0
$ journalctl -u agent.service -n 3 -o short-iso --no-pager
2026-10-20T20:49:28+00:00 8cda6a7bcdc8 systemd[1]: Started agent.service - Always-on agent.
2026-10-20T20:49:28+00:00 8cda6a7bcdc8 python[182]: agent started
2026-10-20T20:49:28+00:00 8cda6a7bcdc8 python[182]: tick

$ sudo systemctl kill -s KILL agent.service; sleep 7
$ systemctl is-active agent.service; systemctl show agent.service -p NRestarts
active
NRestarts=1

$ ls -l /etc/agent/env
-rw------- 1 agent agent 20 Oct 20 20:49 /etc/agent/env

Two things I always check in that output. NRestarts went from 0 to 1, which is systemd telling me the restart policy did its job. And the secrets file is readable by the agent user and nobody else; ProtectSystem=strict then makes everything except /opt/agent/state read-only to the process, so a bug in the agent cannot overwrite its own code or wander into /etc.

Which Ubuntu, and for how long

Use an LTS. 24.04 (Noble Numbat) is supported to 2029 and is what most VPS providers still offer first; 26.04 (Resolute Raccoon) shipped in April 2026 and its first point release, 26.04.1, is what my container reported, which is also when Canonical starts offering the upgrade from the previous LTS. The interim releases are supported for nine months and are the wrong choice for a machine whose whole job is to not need attention. Check which one you are on with grep PRETTY_NAME /etc/os-release; on a server install, set Prompt=lts in /etc/update-manager/release-upgrades so the upgrader never offers you an interim release. That file was absent in my container, because the minimal image leaves out update-manager-core, which is also why do-release-upgrade does not exist there.

That is the whole Ubuntu delta. If you are on Debian the story is older packages and backports; on Fedora it is SELinux doing what AppArmor mostly does not; both have their own pages in the series. And if you came here because you want an agent that posts video for you rather than one you run yourself, that is the thing I actually make.

Sources

1. Python Enhancement Proposals, PEP 668: Marking Python base environments as externally managed (accessed October 20, 2026)
2. Ubuntu documentation, Automatic updates (unattended-upgrades) (accessed October 20, 2026)
3. Ubuntu documentation, Firewall (ufw) (accessed October 20, 2026)
4. Ubuntu documentation, AppArmor (accessed October 20, 2026)
5. Ubuntu Wiki, Releases (accessed October 20, 2026)

Written by Lera Leonteva. Tested on 20 October 2026 in fresh ubuntu:24.04 and ubuntu:26.04 containers; the 26.04 one was booted with systemd. Kernel-side features (AppArmor, the firewall) could not be exercised in a container and are marked as such.