Linux Penguin

Top Tools for Every System Administrator's Toolbox

As anyone who has worked in the Open Source World for some time, I have stocked my toolbox with a large number of common programs and utilities that help get the job done. All major distributions have each of the tools I am going to talk about either installed by default during setup or available via their core repositories post installation.

These tools will provide us the information and metrics to not only see what is going on with our system but will help with the analysis necessary when performance or stability becomes an issue. We are going to look at the ‘Top’ tools for finding disk, CPU, network and/or other system bottlenecks as well as some that just answer some basic questions. By no means is this initial list anywhere near comprehensive. I intend to follow up this article with several others introducing other tools and eventually diving in deeper with the more complex applications.

Top of The List: ‘top’

I wish that I could have been more original for the number one tool, but alas, there is a reason it is called ‘top’ beyond the obvious function. This one simple command provides you with a live view of everything going on on your system now and in the recent past. By default, we will see the processes that are using the most CPU updated every five seconds:

As you can see, we quickly get a number of useful bits about what is going on at this moment on this system. We can see that this system has been running just under 40 minutes, load is currently less than 1.0, the system has 4gb of memory and equal swap space and the top process consumers are XWindows and KDE with the ‘plasma-desktop’ compositing. I will pause here to point you at a better version of top, ‘htop’. This provides us with a little more information more clearly laid out:

Now here, we clearly see that this system also has 2 CPUs with clear representation of how much each CPU/Core is being used along with memory and swap space. Additionally (although cut out of this screen shot), we have access to a number of Function Key shortcuts for rearranging display of processes, stopping a process or filtering them. Although this is all available via ‘top’ as well, it is a bit cleaner and more user friendly here.

 Who Goes There: ‘w’

I think this little gem gets overlooked by even the most experienced System Administrators. Simply put, this command will display information about the users currently on the machine as well as their processes. You can look at a single user or everyone at once:

What Am I Waiting For: ‘iostat’

This command is great for a snapshot view of what is going on in your system’s IO subsystems (disk, CPU and network fileshares). It will report if the system is idle or waiting on something and then list the read/write behavior of all your partitions (Ubuntu users will note that some versions will require the installation of the ‘sysstat’ package from the default repository to get this application). This will help indicate other areas on the system that need further investigation:

In this case, the system in question is effectively idle with a single partition that is not doing much at all (my Ubuntu VM for testing). What this clearly tells me is that I do not have any IO wait going on and any performance issues are likely not related to that subsystem.

Is Anyone Listening: ‘nmap’

Although an entire article could be dedicated to ‘nmap’ (and many have), it is nonetheless an easy tool to glean useful information from quickly. This will scan your local (or, with appropriate permission from the owner, a remote) system and tell you all kinds of useful network information about it. Locally, during troubleshooting performance or application functions, it can tell you if your system is indeed listening on the ports you expect it to. Output of a full TCP scan of your local system will look something like this:

As you can see, on this basic system, we have running MySQL (port 3306), Apache or another application listening over HTTP (port 80) and Internet Printing (IPP on port 631). This information is not only useful to verify an application has opened the appropriate port, but if I were hardening this system for internet access, I would obviously take steps to close off port 631 since there are any number of exploits available that might expose that system to attack.

Final Thoughts

This is a small sampling of some basic tools or more complex tools used to obtain basic information. Each of them have more complex use cases that can be discovered on the command line or the man pages they come with. In the future, we will add more tools to our belt as well as learn how to use these tools to discover more in depth information about our systems. Please comment below and let us know what you use and how you use it, it might make it into one of our future articles.

Terrence T. Cox

A veteran of twenty years in Information Technology in a variety of roles. He has worked in development, security and infrastructure well before they merged into what we now call DevOps. He provides training in Linux, VMWare, DevOps (Ansible, Jenkins, etc) as well as containers and AWS topics.

One thought on “Top Tools for Every System Administrator's Toolbox

Leave a Reply

Your email address will not be published. Required fields are marked *