Linux Networking for Sysadmins and DevOps Engineers - Software and Systems Engineering

Networking problems are responsible for a disproportionate share of outages and escalations. In over two decades of systems work I have found that most incidents boil down to one of three root causes: a routing or firewall rule that is wrong, a DNS entry that is stale or missing, or a service that is not actually listening on the expected address and port. The tools and workflows on this page address all three — methodically, without guessing.

Network Diagnostics

Connectivity Testing
DNS Lookups

# Basic reachability
ping -c 4 8.8.8.8                   # 4 packets to Google DNS
ping -c 4 -I eth0 10.0.0.1          # bind to specific interface
ping6 -c 4 2001:4860:4860::8888     # IPv6

# Trace path to destination
traceroute 8.8.8.8                  # UDP probes by default
traceroute -T -p 443 google.com     # TCP to port 443 (bypasses ICMP blocks)
traceroute -I 8.8.8.8               # ICMP (like Windows tracert)

# mtr — combines ping + traceroute, live stats
mtr --report --report-cycles 20 8.8.8.8   # 20-cycle report mode
mtr -n --tcp --port 443 api.example.com   # TCP, no DNS resolution

# curl — test HTTP/S endpoints
curl -v https://api.example.com/health              # verbose output
curl -o /dev/null -s -w "%{http_code}\n" http://... # just the status code
curl -I https://example.com                         # HEAD request (headers only)
curl --max-time 5 -sf http://localhost:8080/health  # timeout + silent + fail on 4xx/5xx
curl -k https://self-signed.internal/api            # ignore cert errors
curl --resolve api.example.com:443:10.0.0.5 https://api.example.com/  # override DNS

# wget — alternative for downloading
wget -qO- http://checkip.amazonaws.com             # get external IP
wget --spider https://example.com                  # check URL without downloading

# dig — the definitive DNS tool
dig google.com                       # A record lookup
dig google.com AAAA                  # IPv6 address
dig google.com MX                    # mail exchangers
dig google.com NS                    # authoritative nameservers
dig google.com TXT                   # TXT records (SPF, DKIM, etc.)
dig -x 8.8.8.8                       # reverse lookup (PTR)
dig @1.1.1.1 google.com              # query specific resolver (Cloudflare)
dig +short google.com                # minimal output (just IPs)
dig +trace google.com                # full delegation path from root
dig +nocmd +noall +answer google.com # clean answer-only output

# nslookup — simpler, available everywhere
nslookup google.com
nslookup google.com 8.8.8.8         # against specific server
nslookup -type=MX example.com

# host — quick lookups
host google.com
host -t MX gmail.com
host 8.8.8.8                        # reverse lookup

# Check which DNS server is being used
cat /etc/resolv.conf
resolvectl status                   # systemd-resolved

Interface Management with `ip`

The legacy ifconfig / route / netstat tools from net-tools are obsolete. Use ip and ss from the iproute2 suite on any modern Linux system.

ip addr show                         # all interfaces
ip addr show eth0                    # specific interface
ip addr show dev eth0                # same, explicit
ip -4 addr show                      # IPv4 only
ip -6 addr show                      # IPv6 only
ip -br addr show                     # brief one-line-per-interface format

# Add/remove address
sudo ip addr add 192.168.1.10/24 dev eth0
sudo ip addr del 192.168.1.10/24 dev eth0

Firewall Management

firewalld (RHEL/CentOS/Fedora)
ufw (Ubuntu/Debian)
iptables (universal)
nftables (modern)

# Status
firewall-cmd --state
firewall-cmd --list-all               # active zone rules
firewall-cmd --list-all-zones         # all zones

# Allow a service (permanent = survives reboot)
firewall-cmd --permanent --add-service=http
firewall-cmd --permanent --add-service=https
firewall-cmd --reload                 # apply permanent changes

# Allow a specific port
firewall-cmd --permanent --add-port=8080/tcp
firewall-cmd --permanent --remove-port=8080/tcp

# Allow a source IP
firewall-cmd --permanent --add-rich-rule='rule family="ipv4" source address="10.0.0.0/8" accept'

# Port forwarding
firewall-cmd --permanent --add-forward-port=port=80:proto=tcp:toport=8080

# Reload and verify
firewall-cmd --reload
firewall-cmd --list-all

# Status
ufw status verbose
ufw status numbered         # show rule numbers for deletion

# Enable / disable
ufw enable
ufw disable

# Allow / deny rules
ufw allow 22/tcp            # SSH
ufw allow 80/tcp            # HTTP
ufw allow 443               # HTTPS (tcp implied)
ufw allow from 10.0.0.0/8 to any port 5432   # PostgreSQL from internal

ufw deny 23/tcp             # block telnet
ufw delete 3                # delete rule by number

# Reset all rules
ufw reset

# List all rules with line numbers
iptables -L -n -v --line-numbers
iptables -L INPUT -n -v --line-numbers

# Allow established connections (stateful)
iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT

# Allow SSH
iptables -A INPUT -p tcp --dport 22 -j ACCEPT

# Allow HTTP/S
iptables -A INPUT -p tcp -m multiport --dports 80,443 -j ACCEPT

# Block an IP
iptables -A INPUT -s 203.0.113.45 -j DROP

# Drop everything else (default deny — put this LAST)
iptables -A INPUT -j DROP

# Delete a rule by number
iptables -L INPUT --line-numbers
iptables -D INPUT 5

# Save rules (persist across reboots)
# RHEL/CentOS:
service iptables save
# Debian/Ubuntu:
iptables-save > /etc/iptables/rules.v4

# List rules
nft list ruleset

# Add a simple rule (allow SSH)
nft add rule inet filter input tcp dport 22 accept

# Example minimal ruleset in /etc/nftables.conf
cat > /etc/nftables.conf <<'EOF'
#!/usr/sbin/nft -f
flush ruleset

table inet filter {
    chain input {
        type filter hook input priority 0; policy drop;
        iifname lo accept
        ct state established,related accept
        tcp dport {22, 80, 443} accept
        icmp type echo-request accept
    }
    chain forward {
        type filter hook forward priority 0; policy drop;
    }
    chain output {
        type filter hook output priority 0; policy accept;
    }
}
EOF

systemctl enable --now nftables

SSH Power-User Tips

# Generate ED25519 key (preferred over RSA since ~2014)
ssh-keygen -t ed25519 -C "valeriy@hostname-$(date +%Y%m%d)" -f ~/.ssh/id_ed25519

# Copy public key to remote server
ssh-copy-id -i ~/.ssh/id_ed25519.pub user@remote-host

# Manual alternative (when ssh-copy-id is unavailable)
cat ~/.ssh/id_ed25519.pub | ssh user@remote-host \
    "mkdir -p ~/.ssh && cat >> ~/.ssh/authorized_keys && chmod 600 ~/.ssh/authorized_keys"

# Start ssh-agent and add key
eval "$(ssh-agent -s)"
ssh-add ~/.ssh/id_ed25519
ssh-add -l     # list loaded keys

Add ControlMaster auto, ControlPath ~/.ssh/cm-%r@%h:%p, and ControlPersist 10m to your ~/.ssh/config to multiplex SSH connections. After the first login, subsequent connections to the same host are near-instant with no re-authentication.

Common Port Reference

Port	Protocol	Service
22	TCP	SSH
25	TCP	SMTP
53	TCP/UDP	DNS
80	TCP	HTTP
110	TCP	POP3
143	TCP	IMAP
443	TCP	HTTPS
587	TCP	SMTP (submission)
3306	TCP	MySQL / MariaDB
5432	TCP	PostgreSQL
5672	TCP	RabbitMQ AMQP
6379	TCP	Redis
8080	TCP	HTTP alt / app servers
8443	TCP	HTTPS alt
9090	TCP	Prometheus
9200	TCP	Elasticsearch HTTP
27017	TCP	MongoDB

DNS Troubleshooting Steps

Check the local resolver configuration

cat /etc/resolv.conf
# Look for: nameserver, search, domain directives

resolvectl status           # systemd-resolved details
resolvectl query google.com # query through systemd-resolved

Verify basic name resolution

# Does the host resolve at all?
dig +short google.com

# If that fails, try a public resolver directly
dig @8.8.8.8 +short google.com
dig @1.1.1.1 +short google.com

# If direct resolvers work but /etc/resolv.conf does not,
# the problem is local resolver configuration or caching

Check for DNS caching issues

# Flush systemd-resolved cache
resolvectl flush-caches
systemd-resolve --flush-caches

# Flush nscd cache (if running)
nscd -i hosts

# Verify TTL on the record (low TTL = propagating change)
dig +nocmd +noall +answer +ttl google.com

Trace the full delegation chain

# Follow the resolution from root servers down
dig +trace example.com

# Check all authoritative nameservers agree
for ns in $(dig +short NS example.com); do
    echo "=== ${ns} ==="
    dig @"${ns}" example.com A +short
done

Diagnose split-horizon / internal DNS

# Compare internal vs external resolution
dig @10.0.0.53 internal-app.company.com    # internal DNS
dig @8.8.8.8   internal-app.company.com    # external DNS

# Check search domain is set correctly
cat /etc/resolv.conf | grep search

# Test with FQDN (trailing dot forces full lookup)
dig internal-app.company.com.              # FQDN

Inspect /etc/hosts for overrides

grep -v '^#' /etc/hosts | grep -v '^$'
# Entries here override DNS — a common source of surprises
getent hosts hostname   # shows effective resolution order

Quick Network Diagnostics Checklist

Cannot reach external host — checklist

# 1. Check interface is up and has an IP
ip -br addr show

# 2. Check default route exists
ip route show default

# 3. Ping the default gateway
GATEWAY=$(ip route show default | awk '/default/{print $3}')
ping -c 3 "${GATEWAY}"

# 4. Ping a public IP (bypasses DNS)
ping -c 3 8.8.8.8

# 5. Test DNS resolution
dig +short google.com

# 6. Test HTTP connectivity
curl -sv --max-time 5 https://google.com 2>&1 | head -30

# 7. Check local firewall
iptables -L OUTPUT -n -v
firewall-cmd --list-all   # if using firewalld

Service port not accessible — checklist

# 1. Is the service listening locally?
ss -tlnp | grep ':8080'

# 2. Is it bound to 0.0.0.0 or only 127.0.0.1?
ss -tlnp | grep ':8080'
# 127.0.0.1 means not reachable from outside

# 3. Test local loopback
curl -v http://127.0.0.1:8080/health

# 4. Test from the server's external IP
curl -v http://$(curl -s checkip.amazonaws.com):8080/health

# 5. Check firewall
iptables -L INPUT -n -v | grep 8080

# 6. Test from a remote host
nc -zv remote-host 8080
telnet remote-host 8080

Linux Essentials

Core commands including ss, ip, and file system tools.

Troubleshooting

Systematic workflows when networking is part of a larger incident.

Kubernetes

Kubernetes networking — Services, Ingress, and network policies.

AWS

VPC, security groups, and Route 53 in the cloud context.

​Network Diagnostics

​Interface Management with ip

​Firewall Management

​SSH Power-User Tips

​Common Port Reference

​DNS Troubleshooting Steps

​Quick Network Diagnostics Checklist

​Related Pages

Linux Essentials

Troubleshooting

Kubernetes

AWS

Network Diagnostics

Interface Management with `ip`

Firewall Management

SSH Power-User Tips

Common Port Reference

DNS Troubleshooting Steps

Quick Network Diagnostics Checklist

Related Pages