Troubleshooting SSH
Permission denied, host key changed, debug -vvv.
Learning objectives
- Diagnose SSH failures using verbose mode and logs
- Fix common issues: keys, permissions, firewall, DNS
- Follow a systematic checklist for Workshop Co. outages
Start with verbose output
ssh -vvv workshop-db
Three vs show handshake details: which keys offered, KEX algorithms, auth success/failure. Read from the bottom up for the first fatal error.
Common failures and fixes
| Symptom | Likely cause | Fix |
|---|---|---|
Connection refused | sshd down or wrong port | systemctl status ssh on server; check Port in config |
Connection timed out | Firewall, wrong IP, host offline | ping/traceroute; verify router forward; Proxmox console |
Permission denied (publickey) | Key not in authorized_keys, wrong user | Check ~/.ssh/authorized_keys, file perms, AllowUsers |
Too many authentication failures | SSH offering too many keys | IdentitiesOnly yes + correct IdentityFile |
Host key verification failed | Server rebuilt, IP reused | Remove old line from ~/.ssh/known_hosts after verifying legitimate change |
Bad owner or permissions | Loose perms on ~/.ssh | chmod 700 ~/.ssh; chmod 600 authorized_keys |
Worked example — Saturday morning outage
Marcus cannot reach workshop-web before class. His checklist:
ping bastion.workshopco.ca — up?
ssh -vvv bastion — auth OK?
ssh workshop-web — fails at jump?
Proxmox console: VM 110 running? IP 10.20.0.10?
On VM: sudo journalctl -u ssh -n 50
Root cause example: VM disk full → sshd cannot write logs → auth weirdness. Fix: expand disk, journalctl --vacuum-size=200M, restart ssh.
Server-side log locations
# Debian/Ubuntu
sudo journalctl -u ssh -f
sudo tail -f /var/log/auth.log
# Check who is listening
sudo ss -tlnp | grep :22
Client-side checks
# Confirm key loaded
ssh-add -l
# Test specific key
ssh -i ~/.ssh/workshopco_ed25519 -vvv deploy@10.20.0.10
# Compare host key fingerprint (out-of-band verify)
ssh-keyscan bastion.workshopco.ca
If host key changes unexpectedly, verify with Proxmox console or phone call before deleting known_hosts entry — could be attack or legitimate rebuild. Workshop Co. documents expected fingerprints in internal wiki.
Try it yourself
Match scenario to first diagnostic command:
- Auth worked yesterday; today "Permission denied (publickey)" on db only
- Nothing reachable at bastion.workshopco.ca
- Bastion works; jump to web times out
Answers
- Check db VM
authorized_keys, recent deploy that reset home dir,AllowUsers - ISP outage, router down, DNS failure —
dig bastion.workshopco.ca, ping gateway - Web VM powered off or wrong IP — Proxmox console,
ip aon guest
Check your understanding
- Does
ssh -vvvshow your private key contents? - Why fix
~/.sshpermissions to 700?
Answers
- No — it shows key paths and auth attempts, never private key material.
- sshd rejects keys if group/world can read
~/.sshorauthorized_keys— OpenSSH strict mode.