Linux Server Hardening skill for Codex

Done well means: SSH accepts keys only from named users, the firewall exposes only what the server is for, security updates install themselves, every listening port is accounted for, and nobody (including the agent) got locked out along the way.

Lock-out safety first

Hardening SSH and firewall rules over SSH is how people lose servers. Before any change:

1. Confirm out-of-band access exists: the provider’s web console or serial console, and that the user knows how to reach it. If none exists, say so and slow down.
2. Keep the current SSH session open the whole time. Test every change from a second, new session before closing the first.
3. Arm a rollback for firewall changes:

# Debian/Ubuntu: needs the 'at' package; cancel with atrm <job> once verified
echo 'ufw disable' | sudo at now + 10 minutes
# RHEL family: firewalld can apply runtime-only rules first; --permanent only after testing

4. Validate configs before reloading: sudo sshd -t must print nothing.

These steps change system security settings on the user’s server: describe each change and get confirmation before applying it, one area at a time.

Inventory before changing

cat /etc/os-release
sudo ss -tulpn                                     # every listening socket and its process
systemctl list-units --type=service --state=running
sudo sshd -T | grep -Ei 'permitrootlogin|passwordauthentication|kbdinteractive|pubkey|allowusers|port'
ls /etc/ssh/sshd_config.d/
getent group sudo wheel; awk -F: '$3>=1000 {print $1, $7}' /etc/passwd
sudo ufw status verbose 2>/dev/null || sudo firewall-cmd --list-all
last -n 20; sudo journalctl -u ssh -u sshd --since "24 hours ago" | grep -c 'Failed'

If the server is managed by Ansible, cloud-init, or another config tool, make changes there instead of by hand, or they will be reverted. Check AGENTS.md and the repo for playbooks.

Users and sudo

sudo adduser deploy                      # Debian/Ubuntu (useradd -m on RHEL)
sudo usermod -aG sudo deploy             # 'wheel' on RHEL family
sudo install -d -m 700 -o deploy -g deploy /home/deploy/.ssh
sudo install -m 600 -o deploy -g deploy /dev/stdin /home/deploy/.ssh/authorized_keys <<< "ssh-ed25519 AAAA... alice@laptop"

• One account per human; no shared logins. Service accounts get /usr/sbin/nologin.
• Ask the user for their public key. Never generate a key pair on the server for them and never handle private keys.
• Prefer ed25519 keys. Remove stale keys from every authorized_keys, including root’s.

SSH

sshd uses the first value it reads for most keywords, and the Include /etc/ssh/sshd_config.d/*.conf line sits at the top of the stock Debian/Ubuntu config. Drop-ins are read in lexical order, so a cloud image’s 50-cloud-init.conf setting PasswordAuthentication yes beats a later 99-hardening.conf. Name yours to sort first:

# /etc/ssh/sshd_config.d/00-hardening.conf
PermitRootLogin no
PasswordAuthentication no
KbdInteractiveAuthentication no
PubkeyAuthentication yes
AuthenticationMethods publickey
AllowGroups ssh-users
MaxAuthTries 3
LoginGraceTime 30
X11Forwarding no
AllowAgentForwarding no
ClientAliveInterval 300
ClientAliveCountMax 2

sudo groupadd -f ssh-users && sudo usermod -aG ssh-users deploy
sudo sshd -t && sudo systemctl reload ssh    # unit is 'sshd' on RHEL family
sudo sshd -T | grep -Ei 'passwordauthentication|permitrootlogin|allowgroups'   # effective values
# From a NEW terminal: ssh deploy@host  -> must work; ssh root@host -> must fail

• Changing the port is noise reduction, not security. On Ubuntu 22.10 and later, sshd is socket-activated: a new Port needs sudo systemctl daemon-reload && sudo systemctl restart ssh.socket, and the firewall must allow the new port before the switch. On RHEL with SELinux, semanage port -a -t ssh_port_t -p tcp <port> too.
• If the provider console requires a password, keep a strong password on the sudo user for console login while SSH stays key-only.

Firewall

# Ubuntu/Debian with ufw
sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw limit OpenSSH            # allow + rate-limit 22/tcp
sudo ufw allow 80,443/tcp
sudo ufw enable && sudo ufw status numbered

# RHEL family with firewalld
sudo firewall-cmd --permanent --add-service=ssh --add-service=http --add-service=https
sudo firewall-cmd --reload && sudo firewall-cmd --list-all

Restrict admin ports by source when the user has a fixed IP or VPN: sudo ufw allow from 198.51.100.0/24 to any port 22 proto tcp.

Docker bypasses ufw and firewalld. Ports published with -p 5432:5432 are opened in Docker’s own iptables chains and are reachable from the internet even if ufw denies them. Bind internal services to loopback (-p 127.0.0.1:5432:5432 or Compose "127.0.0.1:5432:5432"), keep databases on an internal Docker network without published ports, or add rules to the DOCKER-USER chain. Verify from outside the server with nc -vz <public-ip> 5432 from another machine.

Cloud security groups are a second layer, not a replacement; keep both aligned.

Automatic security updates

# Debian/Ubuntu
sudo apt install unattended-upgrades needrestart
sudo dpkg-reconfigure -plow unattended-upgrades
sudo unattended-upgrade --dry-run --debug | tail -20

# RHEL family
sudo dnf install dnf-automatic
sudo sed -i 's/^apply_updates.*/apply_updates = yes/' /etc/dnf/automatic.conf
sudo systemctl enable --now dnf-automatic.timer

• Decide reboots explicitly: Unattended-Upgrade::Automatic-Reboot "true"; with a Automatic-Reboot-Time in a quiet window, or a monitored “reboot required” alert (/var/run/reboot-required). Kernel fixes do nothing until reboot.
• Security updates only by default; full upgrades stay a planned change.

fail2ban

With key-only SSH, brute force cannot succeed; fail2ban mainly cuts log noise and slows scanners. It matters more for services with passwords (web logins, mail).

# /etc/fail2ban/jail.local
[DEFAULT]
bantime = 1h
findtime = 10m
maxretry = 5
backend = systemd
ignoreip = 127.0.0.1/8 ::1 198.51.100.10

[sshd]
enabled = true

sudo systemctl enable --now fail2ban
sudo fail2ban-client status sshd

backend = systemd is required on distros where sshd logs only to journald (no /var/log/auth.log). Put the user’s own IP in ignoreip.

Remaining baseline

• Remove or disable what the server does not need: sudo systemctl disable --now <unit> for each unexpected entry in ss -tulpn. Databases listen on 127.0.0.1 unless remote access is required.
• Time sync on (timedatectl shows System clock synchronized: yes); TLS, TOTP and logs depend on it.
• Logs persist across reboots (Storage=persistent in journald.conf) and ship off-box if the server matters; an attacker with root can edit local logs.
• Run app services as dedicated users with systemd sandboxing (see $systemd-services).
• Backups exist and a restore was tested; hardening does not help after rm -rf.

Audit

sudo apt install lynis    # or dnf install lynis (EPEL)
sudo lynis audit system --quick | tee lynis.txt
grep -E 'Hardening index|Suggestion' -A1 lynis.txt | head -60

Treat Lynis as a checklist, not a score to maximise. Fix warnings, read suggestions, and record why skipped items do not apply.

Verification

• From a new session: key login as the sudo user works; root login and password login are refused (ssh -o PubkeyAuthentication=no deploy@host must fail).
• From another machine: only intended ports are open (nmap -Pn -p- <ip> if the user approves scanning their own host, else nc -vz per port).
• sudo unattended-upgrade --dry-run or systemctl status dnf-automatic.timer is healthy.
• fail2ban-client status sshd shows the jail active.
• The rollback at job is cancelled (atq empty) only after all checks pass.

Report template

**Host:** web-1 (Ubuntu 24.04, Hetzner), console access confirmed
| Area | Before | After |
|---|---|---|
| SSH | root + password allowed | keys only, group ssh-users, root denied |
| Firewall | inactive | ufw deny-in; 22 (limit), 80, 443 |
| Exposed ports | 22, 80, 443, 5432 (Docker) | 22, 80, 443; Postgres bound to 127.0.0.1 |
| Updates | manual | unattended security updates, reboot Sun 04:00 |
**Verified from second session and external host.** Lynis index 62 -> 78.
**Open items:** off-box log shipping; restore test for nightly backups