Chapter 7

Troubleshooting SSH

Permission denied, host key changed, debug -vvv.

Learning objectives

  • Diagnose SSH failures using verbose mode and logs
  • Fix common issues: keys, permissions, firewall, DNS
  • Follow a systematic checklist for Workshop Co. outages

Start with verbose output

ssh -vvv workshop-db

Three vs show handshake details: which keys offered, KEX algorithms, auth success/failure. Read from the bottom up for the first fatal error.

Common failures and fixes

SymptomLikely causeFix
Connection refusedsshd down or wrong portsystemctl status ssh on server; check Port in config
Connection timed outFirewall, wrong IP, host offlineping/traceroute; verify router forward; Proxmox console
Permission denied (publickey)Key not in authorized_keys, wrong userCheck ~/.ssh/authorized_keys, file perms, AllowUsers
Too many authentication failuresSSH offering too many keysIdentitiesOnly yes + correct IdentityFile
Host key verification failedServer rebuilt, IP reusedRemove old line from ~/.ssh/known_hosts after verifying legitimate change
Bad owner or permissionsLoose perms on ~/.sshchmod 700 ~/.ssh; chmod 600 authorized_keys

Worked example — Saturday morning outage

Marcus cannot reach workshop-web before class. His checklist:

1

ping bastion.workshopco.ca — up?

2

ssh -vvv bastion — auth OK?

3

ssh workshop-web — fails at jump?

4

Proxmox console: VM 110 running? IP 10.20.0.10?

5

On VM: sudo journalctl -u ssh -n 50

Root cause example: VM disk full → sshd cannot write logs → auth weirdness. Fix: expand disk, journalctl --vacuum-size=200M, restart ssh.

Server-side log locations

# Debian/Ubuntu
sudo journalctl -u ssh -f
sudo tail -f /var/log/auth.log

# Check who is listening
sudo ss -tlnp | grep :22

Client-side checks

# Confirm key loaded
ssh-add -l

# Test specific key
ssh -i ~/.ssh/workshopco_ed25519 -vvv deploy@10.20.0.10

# Compare host key fingerprint (out-of-band verify)
ssh-keyscan bastion.workshopco.ca
MITM warning

If host key changes unexpectedly, verify with Proxmox console or phone call before deleting known_hosts entry — could be attack or legitimate rebuild. Workshop Co. documents expected fingerprints in internal wiki.

Try it yourself

Match scenario to first diagnostic command:

  1. Auth worked yesterday; today "Permission denied (publickey)" on db only
  2. Nothing reachable at bastion.workshopco.ca
  3. Bastion works; jump to web times out
Answers
  1. Check db VM authorized_keys, recent deploy that reset home dir, AllowUsers
  2. ISP outage, router down, DNS failure — dig bastion.workshopco.ca, ping gateway
  3. Web VM powered off or wrong IP — Proxmox console, ip a on guest

Check your understanding

  1. Does ssh -vvv show your private key contents?
  2. Why fix ~/.ssh permissions to 700?
Answers
  1. No — it shows key paths and auth attempts, never private key material.
  2. sshd rejects keys if group/world can read ~/.ssh or authorized_keys — OpenSSH strict mode.