systemd Services skill for Codex

A good unit starts after what it actually needs, runs as an unprivileged user with a sandbox that systemd-analyze security scores as OK or better, restarts on failure without looping forever, stops gracefully within a known timeout, and logs to the journal with useful identifiers. Timers replace cron because they log, catch up after downtime, and can be tested on demand.

Inspect first

1. systemctl --version (features like ProtectSystem=strict, LoadCredential= and DynamicUser= need reasonably recent systemd; check before using).
2. systemctl cat <unit> shows the unit plus drop-ins; systemctl status <unit> and journalctl -u <unit> -b --no-pager | tail -50 show current state.
3. Know what the binary does: listens on which ports, writes where, needs which env vars, handles SIGTERM or not, forks or stays in the foreground.
4. Read AGENTS.md or the app README for the production start command.

Service unit template

# /etc/systemd/system/api.service
[Unit]
Description=Acme API
Documentation=https://git.example.com/acme/api
Wants=network-online.target
After=network-online.target postgresql.service
StartLimitIntervalSec=300
StartLimitBurst=5

[Service]
Type=exec
User=api
Group=api
WorkingDirectory=/opt/api
EnvironmentFile=/etc/api/api.env
ExecStart=/opt/api/bin/api --port 8080
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure
RestartSec=5s
TimeoutStopSec=30s
KillMode=mixed

StateDirectory=api
LogsDirectory=api
RuntimeDirectory=api

# Hardening
NoNewPrivileges=yes
ProtectSystem=strict
ProtectHome=yes
PrivateTmp=yes
PrivateDevices=yes
ProtectKernelTunables=yes
ProtectKernelModules=yes
ProtectControlGroups=yes
ProtectClock=yes
ProtectHostname=yes
RestrictNamespaces=yes
RestrictRealtime=yes
RestrictSUIDSGID=yes
LockPersonality=yes
MemoryDenyWriteExecute=yes
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX
SystemCallArchitectures=native
SystemCallFilter=@system-service
SystemCallFilter=~@privileged @resources
CapabilityBoundingSet=
UMask=0027

[Install]
WantedBy=multi-user.target

• StartLimit* belong in [Unit]; in [Service] they are ignored with a warning.
• ProtectSystem=strict makes the filesystem read-only; StateDirectory= and friends create writable /var/lib/api, /var/log/api, /run/api owned by the service user. Add ReadWritePaths= only for other locations.
• MemoryDenyWriteExecute=yes breaks JIT runtimes (Node.js, JVM, .NET, LuaJIT, PyPy). Drop it for those.
• Binding ports below 1024 as a non-root user: AmbientCapabilities=CAP_NET_BIND_SERVICE plus CapabilityBoundingSet=CAP_NET_BIND_SERVICE. Better: listen on 8080 behind a proxy.

Type

App behaviour	Type
Stays in foreground (most modern apps)	exec (fails start if the binary cannot execute; simple does not)
Calls sd_notify(READY=1) when ready	notify (dependents wait for real readiness)
Forks and exits parent (legacy daemon)	forking + PIDFile=; prefer a foreground flag instead
Runs to completion (migration, backup)	oneshot, often with RemainAfterExit=yes for setup tasks

Users and secrets

• Create a system user: useradd --system --home /opt/api --shell /usr/sbin/nologin api. Or use DynamicUser=yes for stateless services; it implies several protections and allocates a transient UID (use StateDirectory= for persistent files).
• EnvironmentFile= files must be 0600 root:root or 0640 root:api; environment is visible via /proc/<pid>/environ to the same user and in systemctl show.
• Prefer LoadCredential=db_password:/etc/api/creds/db_password; the app reads $CREDENTIALS_DIRECTORY/db_password, which only that service can access.
• Never put secrets in Environment= lines inside the unit; unit files are world-readable.

Timers instead of cron

# /etc/systemd/system/db-backup.service
[Unit]
Description=Nightly database backup

[Service]
Type=oneshot
User=backup
ExecStart=/usr/local/bin/db-backup.sh
Nice=10
IOSchedulingClass=idle

# /etc/systemd/system/db-backup.timer
[Unit]
Description=Run db-backup nightly

[Timer]
OnCalendar=*-*-* 02:30:00
RandomizedDelaySec=15m
Persistent=true

[Install]
WantedBy=timers.target

systemd-analyze calendar '*-*-* 02:30:00'      # check next elapse
sudo systemctl enable --now db-backup.timer
systemctl list-timers db-backup.timer
sudo systemctl start db-backup.service         # run once now to test

Persistent=true runs a missed job after the machine was off. Add OnFailure=notify-failure@%n.service to the service to alert on failures.

Overrides and lifecycle

sudo systemctl edit api            # creates /etc/systemd/system/api.service.d/override.conf
sudo systemctl daemon-reload       # after editing unit files directly
sudo systemctl enable --now api
sudo systemctl restart api

• Edit vendor units only through drop-ins; package upgrades overwrite /usr/lib/systemd/system.
• To clear a list setting in a drop-in, assign it empty first (ExecStart= then the new ExecStart=...).
• User services (no root): ~/.config/systemd/user/, systemctl --user, and loginctl enable-linger <user> so they run without a login session.
• Starting, stopping, restarting or enabling services on a production host interrupts or changes what runs; show the commands and get the user’s confirmation first.

journald

journalctl -u api -f                          # follow
journalctl -u api --since "1 hour ago" -p warning
journalctl -u api -b -1                       # previous boot
journalctl -u api -o json-pretty -n 5         # structured fields
journalctl --disk-usage

Set retention in /etc/systemd/journald.conf (SystemMaxUse=2G, MaxRetentionSec=1month) and ensure /var/log/journal exists for persistence across reboots.

Debugging

Symptom	Check
start request repeated too quickly	Crash loop hit StartLimitBurst; read the first failure in the journal, then systemctl reset-failed api
status=203/EXEC	Wrong path, missing exec bit, or wrong interpreter line
status=217/USER	User in User= does not exist
status=226/NAMESPACE	A path in ReadWritePaths= or similar does not exist
Works manually, fails as a service	Different env, working dir, or sandbox; add one hardening option at a time
Stop takes 90 s	App ignores SIGTERM; fix the handler or set KillSignal=

Verification

systemd-analyze verify /etc/systemd/system/api.service
systemd-analyze security api.service          # target exposure 4.0 or lower ("OK")
systemctl is-active api && curl -fsS localhost:8080/healthz
sudo systemctl kill -s SIGKILL api && sleep 6 && systemctl is-active api   # restarts (staging only)

Report the unit path, user, exposure score before and after, writable paths, and any hardening option deliberately left off with the reason.