SSH Key Authentication Setup for a Small Linux Server

Put a fresh Linux server on a public IP address and check the auth log an hour later. You will usually find a few hundred failed password attempts from addresses you have never heard of, all guessing at root, admin and ubuntu. Passwords are the weak point, and SSH key authentication removes them from the picture entirely.

This guide walks through SSH key authentication setup on a small server from start to finish: creating the key, copying it across, testing it safely, and only then switching off password logins. The whole job takes about fifteen minutes, and the order of the steps matters more than any single command.

Why SSH Key Authentication Beats Passwords

A password is a secret you send to the server every time you log in. If it is short, reused or leaked from another site, a bot can guess it. A key pair works differently. The private key stays on your laptop, the server only holds the public half, and login is a cryptographic challenge that never sends the secret over the wire.

In practice that means three things for a small team or a home lab:

• Brute force guessing becomes pointless, because there is no password field left to attack.
• Each person gets their own key, so you can remove one laptop without changing anything for everyone else.
• Scripts and deploy jobs can log in without a password stored in plain text in a config file.

Generate an Ed25519 Key Pair on Your Machine

Use Ed25519 unless you have an old system that cannot read it. The keys are short, fast and considered strong by current standards. RSA still works, but if you choose it, use at least 3072 bits.

On Linux, macOS or Windows with OpenSSH installed, run:

ssh-keygen -t ed25519 -C "marisa-laptop-2026"

The comment is only a label, but a clear one saves confusion later when a server has six keys on it and you need to know which belongs to the laptop that was stolen. Accept the default path, which is ~/.ssh/id_ed25519, and set a passphrase. A passphrase protects the key if someone copies the file off your disk. With an agent running, you type it once per session, not once per login.

Copy the Public Key to the Server

The easiest route is ssh-copy-id, which logs in with your password one last time and appends the public key to the right file with the right permissions:

ssh-copy-id -i ~/.ssh/id_ed25519.pub [email protected]

On Windows, where that tool is often missing, you can do it by hand. Print the contents of id_ed25519.pub, log in to the server, and paste that single line into ~/.ssh/authorized_keys. Then fix the permissions, because OpenSSH silently ignores keys in files that other users can write to:

• chmod 700 ~/.ssh
• chmod 600 ~/.ssh/authorized_keys

Wrong permissions are the number one reason a key setup "does not work" while every command looks correct.

Test the Key Before You Change Anything

This is the step people skip, and it is the one that locks them out. Open a new terminal, keep your existing session open, and connect again:

ssh -i ~/.ssh/id_ed25519 [email protected]

If the server asks for your key passphrase rather than the account password, the key is working. If it still asks for the account password, run the same command with -v and look for the line that says which keys were offered and whether the server accepted one. Do not move on until a key login succeeds in a fresh window.

Disable Password Logins in sshd_config

Now you can close the door. Edit /etc/ssh/sshd_config, or better, create a small drop-in file such as /etc/ssh/sshd_config.d/10-keys-only.conf so package upgrades do not overwrite your changes. Set these lines:

Check the syntax with sudo sshd -t. No output means no errors. Then reload with sudo systemctl reload ssh on Debian and Ubuntu, or sshd on most other distributions. Your open session survives a reload, which is your safety net if something went wrong.

Finally, try a password login on purpose from another terminal with ssh -o PubkeyAuthentication=no [email protected]. It should fail with "Permission denied (publickey)". That message is the proof that passwords are truly off.

Manage SSH Keys Across a Small Team

A single key is easy. Five people and four servers gets messy fast unless you keep a simple record. A plain table in your team wiki is enough: key comment, owner, date added, and which servers hold it.

A few habits keep the list honest:

1. One key per person per device. Never share a private key between colleagues.
2. When someone leaves or loses a laptop, delete their line from every authorized_keys file the same day.
3. Review the list every quarter and remove keys nobody can identify.
4. Give automation its own key with a from="10.0.0.5" restriction so it only works from the deploy host.

If you are also opening the machine to the wider internet for the first time, work through the checklist in what to secure before exposing a home server to the internet, because keys only cover the SSH door.

Back Up Your Private Key Without Weakening It

Keys-only login has one real downside: lose the key and you lose access. Plan for that before it happens. Keep a second key on a different device, such as a desktop or a hardware security key, and add it to the server now while you still can.

Store an encrypted copy of the private key in your password manager as a secure attachment. Keep console access through your hosting panel working too, since most providers offer a browser console that bypasses SSH entirely. The same thinking applies to other logins, and the steps in recovering two-factor access after losing a phone follow the same principle of having a second route ready before you need it.

FAQ

Is SSH key authentication more secure than a strong password?

Yes, because the private key is never sent to the server and cannot be guessed by a bot the way a password can. A strong password helps, but it still travels with every login and can be reused or phished.

Should I set a passphrase on my SSH key?

Yes, set one for any key used by a person. It protects the key if the file is copied from your laptop, and an SSH agent means you only type it once per session.

What happens if I lose my only SSH private key?

You will be locked out of SSH until you add a new public key through another route, usually the hosting provider's web console or a rescue mode. That is why you should add a second backup key on day one.

The Short Version

Generate an Ed25519 key with a passphrase, copy the public half across, prove it works in a fresh terminal, and only then turn off passwords. Add a backup key the same day and keep a small record of who holds what. For more hardening ideas, browse the rest of the security guides on this site.