This is an overview to how one would detect if a system you happened to be working on was rooted or had malware etc. Obviously such a guide cannot be complete or cover all such cases but a general overview is provided here. A typical scenario might be taking over machines for a new customer some of which are linux that you have not used before. Doing a general audit and checking for malicious items is a good idea in such cases as often servers are neglected and not patched for long periods of time.
First option when dealing with a potentially suspect system is to enable logging on your firewall for that IP. This can assist greatly as the logging happens off the box, so t does not tip of any intruders that you are monitoring it and it also means that you dont have to trust the box you are checking. It also gives you an opportunity to see what type of traffic is being sent or received and assess the traffic for suspect/suspicious activity. This should be done first as the logging takes time to build a picture. If you need to go further you can often capture packets to a pcap file (wireshark etc) and inspect what is happening.
If the machine is a vm which is common these days you can make a copy of it and audit that which is excellent for not modifying the original too much if required.
Another good place to check is the login logs. One cool command is the command “lastlog” which shows the output of all users and the last time they logged in, this can be super useful if you notice a strange user listed and a recent login time. Similar to this the “last” command will list the last logins by time for the box in question and “last - a” will show ip's if relevant.
In debian you can also intstall rkhunter which scans for rootkits (apt-get install rkhunter) and then modify the config file (sudo nano /etc/rkhunter.conf). In this file make sure UPDATE_MIRRORS= is set to a value of 1 and that WEB_CMD=“/bin/false” is commented out with a # (search as its quite far in the config). MIRRORS_MODE should also be set to a value of 0.
Once you have changed these 3 items you can confirm that the config is valid with “sudo rkhunter -C”. The changes we made allow it to update. Then run an update with “sudo rkhunter –update” and a scan with “sudo rkhunter –check”.
You can google how to interpret the results of the scan and lookup on their documentation as they keep it up to date.
Another tool is clamav although it has limited use. You can install it with “apt-get install clamav clamtk-gnome” but it might not be super useful. Again just look into it, although it primarily detects windows viruses.
Another tool is maldet which you can install, however its not currently in the debian repos. It is best to check how to install it from the github page as its maintained by the author: https://github.com/rfxn/linux-malware-detect
Dont forget to check open ports which is documented here: Install A Simple Software Firewall
You can also lookup how to use yara, and there are signatures here (https://github.com/Yara-Rules/rules) but personally I have not had to use yara and cant comment much on it. You can research it as a tool, I believe its widly used especially in the kali community.
And last thing to note, as always, if anything malicious is ever found, then its best to totally wipe and start again with a new system rather than fix or attempt to fix a rooted box. Hopefully some of these overviews help you out as a basic starting place :)
Further reading once the basics are no longer helpful: