Docker Networking skill for Codex

Done well means: the failing hop is identified with evidence (DNS, routing, listening address, or firewall) before anything is changed, containers talk to each other by service name on user-defined networks, and only the ports meant for the outside world are published, bound to the right interface.

The five facts that explain most failures

1. localhost inside a container is that container. To reach another container use its service or container name; to reach the host use host.docker.internal.
2. Name resolution between containers works only on user-defined networks (Compose creates <project>_default automatically). The legacy default bridge network has no DNS for container names.
3. A server must listen on 0.0.0.0 (or ::) inside the container. An app bound to 127.0.0.1 is unreachable from outside its own container even with -p.
4. EXPOSE documents a port; only -p / ports: publishes it on the host. Containers on the same network reach each other on container ports with no publishing at all.
5. Published ports bypass ufw and firewalld on Linux and bind to all host interfaces unless an IP is given. -p 5432:5432 puts the database on the internet.

Diagnose in layers

docker ps --format 'table {{.Names}}\t{{.Ports}}\t{{.Networks}}'
docker network ls
docker network inspect myapp_default --format '{{range .Containers}}{{.Name}} {{.IPv4Address}}{{"\n"}}{{end}}'
docker inspect api --format '{{json .NetworkSettings.Networks}}' | python3 -m json.tool
docker compose ps; docker compose config | grep -A5 -E 'networks:|ports:'

Then test from inside the network with the same view the app has:

docker exec api getent hosts db                      # DNS (works in most glibc images)
docker exec api sh -c 'nc -vz db 5432'               # TCP reachability, if nc exists
docker exec db ss -ltnp                              # what the server listens on (if ss exists)
# Minimal images lack tools: attach a toolbox to the container's network namespace
docker run --rm -it --network container:api nicolaka/netshoot \
  sh -c 'dig db; nc -vz db 5432; curl -sv http://worker:8080/health'

Pulling nicolaka/netshoot needs network access (ask in a restricted Codex sandbox); busybox has nslookup, wget and nc if that is all that is available.

Result	Meaning	Fix
getent hosts db returns nothing	Different networks, default bridge, or wrong name	Put both on one user-defined network; use the Compose service name
Resolves, connection refused	Server listening on 127.0.0.1, wrong port, or not started yet	Bind to 0.0.0.0; use the container port, not the published one; add a healthcheck
Resolves, timeout	Firewall/internal network/NetworkPolicy-like rules, or wrong IP after restart	Check iptables -S DOCKER-USER, network internal flag; never hard-code container IPs
Works from container, not from host via localhost:8080	Port not published, or published on another IP	docker port api; check ports:
Works on host, not from another machine	Bound to 127.0.0.1, host firewall, or cloud security group	Intended? If not, publish on the right interface and open the cloud firewall
Container cannot reach the internet	DNS from host’s /etc/resolv.conf is a local stub, or MTU	See DNS and MTU sections

Compose patterns

services:
  api:
    build: .
    ports:
      - "127.0.0.1:8080:8080"     # host-only during development
    environment:
      DATABASE_URL: postgres://app:app@db:5432/app   # service name, container port
    depends_on:
      db: { condition: service_healthy }
    networks: [frontend, backend]
  db:
    image: postgres:17
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U app"]
      interval: 5s
      retries: 10
    networks: [backend]            # no ports: not reachable from the host or internet
  proxy:
    image: caddy:2
    ports: ["80:80", "443:443"]
    networks: [frontend]
networks:
  frontend: {}
  backend:
    internal: true                 # no outbound internet for anything only on this network

• depends_on without condition: service_healthy only orders container start; the app still races the database. Retry connections in the app as well.
• Services in different Compose projects do not share networks. Create one with docker network create shared and declare it external: true in both files.
• Container IPs change on recreate. Always address by name.
• Network aliases (networks: { backend: { aliases: [postgres] } }) give a service extra DNS names, useful when migrating.

Reaching the host

Platform	How
Docker Desktop (macOS, Windows)	host.docker.internal works out of the box
Linux Engine	Add extra_hosts: ["host.docker.internal:host-gateway"] (or --add-host=host.docker.internal:host-gateway)

The host service must listen on an address the bridge can reach (not only 127.0.0.1) on Linux; the bridge gateway (often 172.17.0.1) is the source. Check with ss -ltnp on the host.

network_mode: host shares the host network stack: no port mapping, no container DNS names, and it behaves differently on Docker Desktop than on Linux. Use it for performance-sensitive or multicast workloads on Linux only, deliberately.

DNS problems

• Containers on user-defined networks use the embedded resolver at 127.0.0.11, which forwards to the host’s upstream servers. If the host uses a local stub (systemd-resolved at 127.0.0.53), Docker substitutes the real upstreams, or falls back to public DNS when it cannot find any; corporate split-horizon names then fail inside containers.
• Fix per service with dns: in Compose, or globally with "dns": ["10.0.0.2"] in /etc/docker/daemon.json. Editing the daemon config and restarting Docker affects every container on the host: confirm with the user first.
• Alpine (musl) resolves differently from glibc for search domains and some TCP fallback cases; test with the actual image, not a Debian debug container.

Subnet clashes and MTU

Docker allocates 172.17.0.0/16 and further 172.x / 192.168.x ranges for networks. If a VPN, office LAN or cloud VPC uses the same range, traffic to those destinations goes into the Docker bridge instead (symptom: VPN hosts unreachable while Docker runs).

{
  "bip": "10.200.0.1/24",
  "default-address-pools": [{ "base": "10.201.0.0/16", "size": 24 }]
}

Apply in /etc/docker/daemon.json (Docker Desktop: Settings > Docker Engine), then restart the daemon and recreate networks (docker compose down && docker compose up -d). Confirm the chosen ranges with whoever owns the network; restarting Docker stops containers unless live-restore is enabled.

MTU: if small requests work but large responses or TLS handshakes hang (common inside VPNs, some clouds, nested virtualization, CI runners), the overlay or tunnel MTU is lower than the bridge’s 1500. Set the network MTU:

networks:
  default:
    driver_opts:
      com.docker.network.driver.mtu: "1400"

Test with ping -M do -s 1372 <host> from netshoot: it fails if the path MTU is smaller.

Exposure audit

docker ps --format '{{.Names}}\t{{.Ports}}' | grep -E '0\.0\.0\.0|:::'
sudo iptables -S DOCKER-USER

Every 0.0.0.0: binding is reachable from every network the host is on, firewall rules notwithstanding. Databases, caches, admin UIs and metrics endpoints should show 127.0.0.1: or no binding. Confirm from another machine with nc -vz <host-ip> <port>. Host-level firewall work belongs to $linux-server-hardening.

Verification

• From inside the client container (or netshoot in its namespace): DNS resolves the target name, TCP connects, and the application request succeeds.
• docker ps shows only intended published ports, each bound to the intended interface.
• After daemon changes, docker network inspect shows the new subnet and the VPN or LAN route works with Docker running.
• Recreate the stack (docker compose up -d --force-recreate) and repeat: fixes that depend on a container IP or start order show up here.

Report template

**Symptom:** api -> db "connection refused" after moving db to its own compose file
**Evidence:** `getent hosts db` empty in api; api on myapp_default, db on dbstack_default
**Fix:** shared external network `backend`; DATABASE_URL uses db:5432; db ports removed
**Exposure:** postgres no longer published (was 0.0.0.0:5432)
**Checks:** netshoot in api namespace: dig db OK, nc -vz db 5432 OK; app /health 200 after recreate

For Compose development workflow beyond networking (profiles, watch, healthchecks), see $docker-compose-dev.