How to Recover a Hacked Server

What Do I Need?

• Any Dedicated or Virtual Server
• CentOS
• Putty

What is a Compromised Server?

Usually, you find out your server got hacked because your datacenter or provider has network-restricted you. Or maybe you find out because of failed services, or defaced websites showing malware. It’s a scary situation as you don’t know what’s wrong and the immediate reaction is to panic. What’s most annoying is that it always seems to happen at the worst time. If you’re hosting multiple websites and platforms for external clients you could find that it’s the insecure practices of your clients that have brought this calamitous compromise of your systems. When dealing with this situation there are a number of potential ways of fixing it.

1. Work Out the Problem
   1. It’s of paramount importance that you find out what’s being hacked or compromised. This will give you a number of questions:
      • Is it an inbound attack?
      • Is it an outbound attack?
      • Is it malware, defaced website, or altered/stolen data?
      • How much access did the hackers/bad actors gain into your system?
      • Backdoor/script running a user directory?
      • Root-level intrusion compromise?
      • Did your web host or datacenter limit your network connections or bandwidth?
      • Do you have a mitigation plan for restoring your critical client sites?
   2. In some cases, if your servers and systems have been badly compromised, it’s often quicker to rebuild your server from an image. This saves time in trying to fully locate the source of the compromise and repair all the damaged services and config files in the server. Personally, this is not a recommended course of action, as it means you never learn the source of the compromise.
   3. A lot of people get lazy and try to resolve ‘simple’ hacks by restoring a previous or older backup. This can work if data hasn’t changed much. However, just restoring a backup can also lead to you just restoring the seeds of the initial compromise, especially as statistically most compromises are only detected long after the actual original compromise injection.

2. Recover from Inbound Attacks
   1. An inbound attack is when external machines/servers attack your server. Some of these attacks actually try to gain entry into your server but others only seek to disrupt services by overwhelming your server or network. Irrespective of their intentions, all of these attacks consume your server resources, rendering it unable to load your website for real website visitors.
   2. Different kinds of inbound attacks:
      • Brute force – multiple attempts in rapid succession at guessing your admin passwords and gaining entry through your login pages or other connection protocols; for example, XML-RPC for WordPress, Joomla, or Drupal.
      • Flood attacks – denial of service (DOS), the even stronger distributed denial of service (DDOS), or SYNFLOOD attacks. These specifically target the server on different ports and protocols, requesting many open connections, beyond the server’s limit. It’s the equivalent of mass-calling someone’s phone-line to make it unavailable for other callers.
   3. Detecting Inbound Attacks
      • Server running low – this is an obvious sign you might be getting hacked; especially if you haven’t changed anything else on the site and traffic is still the same.
      • Check for high server (CPU) load – Anything at or above the number of CPU cores you have is considered high; for example, a load of ‘5’ when you only have 4 cores. High CPU usually means an attack at the network level, bombarding services.

        grep processor /proc/cpuinfo ¦ wc -l
        

      • Check for high memory usage – High swap messages in your control panel at random intervals are also an obvious indicator. Sometimes the attack will be erratic and sporadic and you’ll just have to scan those log files. High memory usually means an attack at the software level, bombarding PHP scripts.

        cat /proc/meminfo or top
        

Conclusion

When dealing with a server or systems compromise it’s important to remain as calm and patient as possible. It’s best to take your time and try to unravel the mystery. Despite the fact that a compromise is bad, it’s a great opportunity to learn what the bad actor’s attack vector was and work out new strategies for their rectification and protection. Be sure to keep an eye out for more how-to guides about hardening and fixing your servers; coming soon.

• Want to avoid problems of shared hosting? Click here and know about the best dedicated servers hosting.