How to Check a Small Server for Signs of Compromise
The email from the hosting provider arrived at 06:12 on a Tuesday and said that outbound traffic from my address had been reported scanning port 22 across a range in another country.

The email from the hosting provider arrived at 06:12 on a Tuesday and said that outbound traffic from my address had been reported scanning port 22 across a range in another country. The server in question ran one small web application and a game server for about nine people. I had not logged into it in five weeks, and the load graph showed it pinned near full for the last three days.
That is the ordinary shape of a compromise on a small box. Not a dramatic ransom note, just somebody using your bandwidth and your reputation for something cheap. This walkthrough is the triage I now run, in the order I run it, on a single Ubuntu or Debian host with no monitoring agent installed. It takes about twenty minutes and it tells you whether to keep investigating or start rebuilding.
Before you type anything
Three decisions come first, and getting them wrong destroys the evidence you need.
Do not reboot. A lot of what you are looking for lives in running processes and open sockets, and a restart clears it while usually leaving whatever restarts the malware in place. Do not run a package update either, because that can overwrite timestamps you are about to read.
Take a snapshot if your provider offers one. It costs a few cents a day, it takes under a minute, and it gives you something to examine later without the clock running. If the machine is doing active harm, such as sending spam or scanning other hosts, snapshot it and then detach the public network interface from the provider panel rather than shutting the box down.
Lastly, assume your own session might be watched. If you have reason to think an attacker has root, do not paste other credentials into that shell and do not use it to log into anything else.
The twenty minute triage
Run these in order. Each one takes seconds, and the output of the earlier ones tells you what to look for in the later ones.
- Check who is logged in now and who has been: w for current sessions, then last for the history. An IP you do not recognise next to a username you do is the clearest single signal there is.
- Look at failed and successful authentication: journalctl -u ssh since the date in question, or read /var/log/auth.log. Thousands of failures are normal background noise. One success from an unfamiliar address after those failures is not.
- List listening sockets and the processes behind them: ss -tulpn. You should be able to name every line. A process listening on a high random port, or anything bound to an address you did not configure, goes on the list to investigate.
- Find what is eating the machine: top sorted by processor, then ps auxf to see the parent of anything suspicious. Malware on small servers is usually a mining process or a scanner, and it rarely bothers to hide its load.
- Check scheduled jobs for every user, not just root: crontab -l as each account, then read /etc/cron.d and /etc/cron.daily. Persistence on cheap compromises lives here more often than anywhere else.
- List systemd units that are enabled and recently created: systemctl list-unit-files filtered to enabled, and ls -lt /etc/systemd/system to see what was written last. A unit file with a timestamp from the week the trouble started is the answer most of the time.
- Read the authorized keys files for every account with a shell, including root. An extra public key is a quiet, durable back door that survives a password change.
- Check for new accounts and changed privileges: the tail of /etc/passwd, the sudo group in /etc/group, and anything new in /etc/sudoers.d.
- Compare installed binaries against the package database with debsums -c on Debian or Ubuntu. It reports files whose checksum no longer matches the package that shipped them.
- Look at the shell history for the accounts you found, then at the file modification times in /tmp, /dev/shm and /var/tmp. Attackers stage there because those paths are writable, and they often forget to clean up.
Nine times out of ten on a small server, steps one, five and six are enough. The pattern is a weak password or a reused key, a cron entry or a systemd unit for persistence, and a process doing something that earns somebody a small amount of money.
Reading what you found
Separate your findings into two piles: evidence of entry and evidence of persistence. Entry tells you what to fix so it does not happen again. Persistence tells you how much of the system you can still trust.
If entry was an SSH password and persistence was one cron line, that is a shallow compromise and a known quantity. If debsums reports modified system binaries, or you find a kernel module you did not install, or the logs have gaps where they should be continuous, treat the whole machine as untrustworthy. Log gaps matter more than people expect. A missing hour in auth.log is not a glitch.
Write down the earliest timestamp you can tie to the intruder. Everything created after that moment is suspect, including backups, which is the detail that catches most people out.
Why cleaning it up is the wrong instinct
The common advice is to install a rootkit scanner, remove what it finds, change the passwords and carry on. I think that is wrong for almost every small server, and I say that as somebody who did it twice before learning better.
Rootkit scanners compare against known signatures. On a commodity compromise they find the obvious miner and miss anything bespoke, and a clean report is read as proof of safety when it only means the scanner has no signature for what is there. You cannot prove absence this way. You can only fail to find something.
A rebuild, on the other hand, is a known good starting point. On a small box with configuration you can describe in a page of notes, a fresh install takes between one and three hours. That is less time than most people spend arguing with a scanner, and at the end of it you actually know what is running. The only honest reason to clean in place is that you cannot rebuild quickly, and the fix for that is to make rebuilding quick before you need it.
What I do keep from the old machine is data, never binaries or configuration files I have not read line by line. Database dumps, uploaded files and the content directory come across. Anything executable stays behind.
Rebuilding so this is boring next time
Build the replacement before you delete the original, so you can compare the two. Start with key only SSH access and no password authentication at all, following the SSH key authentication setup, and keep the administrative account separate from any service account.
Then put three things in place on day one rather than day four hundred. A firewall that denies by default, covered in the UFW firewall setup. Automatic security patching, so a published fix lands without you remembering. And backups that run on a schedule and get restored once on purpose, which is the part people skip, described in restic backups for a self hosted server.
Keep one plain text file in your password manager listing every service on the box, the port it uses and the command that installed it. It is the difference between a three hour rebuild and a three day one, and it is the single most useful thing I have ever written down.
Frequently asked questions
Should I tell my hosting provider?
Yes, and reply to their abuse notice rather than ignoring it. Providers are generally patient with somebody who answers and explains the plan, and considerably less patient with silence, which tends to end in a suspended account.
Can I trust my backups after a compromise?
Only the ones from before the earliest timestamp you tied to the intruder. Restore data from those, verify by hand, and never restore configuration or binaries from a window you cannot vouch for.
How did they get in if I had a strong password?
Usually not the password. A reused credential from somewhere else, an exposed key, or an unpatched service listening on the internet are more common on small servers. Checking which services were reachable is often more informative than checking password strength.
What should I monitor so I notice sooner?
Outbound traffic volume, processor load, and failed authentication counts are enough for a single host, with alerts going to an address you read. More practical account and server hardening sits in our security section.
More in Guides
Guides
How to Trace Game Server Latency With mtr and iperf3
Four people on the voice call told me the server was lagging.
Guides
Vaultwarden Setup to Self Host Your Own Password Manager
You want a password manager you control, but the official Bitwarden server wants several containers and a few gigabytes of RAM...
Guides
DKIM Key Rotation Without Breaking Your Outgoing Email
Your DKIM key was generated years ago, probably by someone who no longer works with you, and nobody knows if it is 1024 or 2048...