Secure Container Host Operating System
This is the first course of a four-course learning path related to securing containers. This course is created with the goal of teaching how to prepare and harden the operating system so it is secured as much as possible before we actually deploy containers. We will go over various configurations and see how we can use the operating system’s security mechanisms to best protect and harden our system.
In the first part of the course, we provide knowledge regarding firewalld and SELinux. firewalld is a Linux tool used for managing iptables. We need a firewall so we can control what kind of traffic can be let through and what kind of traffic will be denied. We will go over the firewalld installation and overall setup. Here we teach several basic firewalld commands and how to utilize them regarding zones, ports, services, and other related system aspects. We will also talk about traffic control where we learn how we can allow traffic from one IP to a port, how to allow traffic from a list of IPs (whitelisting), and how to create a list of IPs we do not want to have access (blacklisting), etc. After firewalld, we will move on to SELinux. SELinux is a tool which allows fine control over access to files, controls, processes, or other things on the system. It is very effective and used by practically all Red Hat-based distributions. In this section, we will start off by talking about SElinux states, the SELinux context, and context adjustments. We will then go over some useful commands that enable us to list restricted ports and protocols, talk about booleans, port labels, SELinux modules, and logs. Later on in the course, we will also have a section dedicated to firewalld and SElinux automated scripts.
When we learn the basics of firewalld and SELinux, we will move on to learn about server access and authentication configuration. At this point, we are going to play around with the access to the system. In addition to doing some base configuration by just changing the default port we use to connect via an SSH service port, we will also perform changes such as adding extra layers of authentication and creating a jump point server which is similar to a VPN with SSH. We will show how to set up different authentication methods to work simultaneously: the standard key-based authentication, password-based authentication, plus a third layer of authentication where we integrate Google Authenticator. This will need our phone in order to log in to the server. This significantly improves our security since it is highly unlikely someone has access to our key, our phone, and our password. We will also get acquainted with jump points. We show what jump points are and what they are used for. A jump point is a place where we connect. It allows us to connect to the rest of our infrastructure. They can greatly improve the security of our front-facing and infrastructure servers.
Furthermore, we also talk about seccomp (Secure Computing Mode). This is a very important tool that we will use along with containers and properly imposing some additional limitations. We use it to restrict system calls. Basically, it participates in jailing a process and limiting what can be done from the process itself. It gives us the ability to dictate what the process cannot do. After dealing with seccomp, we will move on to a lengthy section regarding logs, where we will learn some very useful commands to help navigate through very large log files. We will also learn about notification systems. In the last section of the course, we will talk about vulnerability scans and reports.
About the Author
Here I give a few words about myself.
About the Course
The video deals with the course itself and what we will cover in the course.
How to Get Help
Need help? Get stuck? Something doesn't work? I am here. Feel free to ask me anything!
Here we share what should be known before starting this course.
Text Editor Vim Basics (Optional)
Here we cover the basics of the VIM text editor.
Job Market (Optional)
What is the job market? How much can one expect to make? Where can we use this knowledge? Where do we find jobs? These are some of the questions this video attempts to answer.
firewalld and SELinux
firewalld Part 1 - Installation, Zones, Interfaces
In this introductory video, we will be talking about firewalls, understand why we need them, and how they are used. We need firewalls so that we can control what kind of traffic is let through and what kind of traffic will be denied. This is an important thing that needs to be done on any server when it is started up. There are many firewalls that can be used, and in this course, we will be using
firewalld. First, we will install
firewalld and learn some basic
firewalld commands that allow us to get the state, list available zones, list zones used by network interfaces, and get default zone.
firewalld Part 2 - Ports and Services
So far, we have seen how we can get the configuration settings of a particular zone and see what is allowed and what is not, and the important part was that the SSH service was allowed through. Ports can also be opened and closed manually, and that is what we will also learn in the video. If we want the changes to be permanent and stay after a restart is done, then
--permanent must be added to the end of the command. We will also talk about firewall services and go over the commands needed in order to add a firewall service, remove a firewall service, and get the default zone.
firewalld Part 3 - Ping Block
This time we will learn how we can block a ping. It is not always a good idea to block a ping because some of your services could end up using a ping, and you can bring them down by disabling that to happen. However, if, in some cases, you can afford to disable a ping when you know your services are not using it, you can choose to do it. First, we will check if we can ping the server or not. After that, we will block the ICMP by editing the
sysctl.conf file and uncommenting one line in it. This is just one of the ways this can be done; there are more. After editing the file, we just apply the changes.
firewalld Part 4 - ipset, blacklist, whitelist
In this video, we will talk about traffic control. It is important to be able to allow traffic from one IP to one single port and to allow a list of IPs to access the server regardless of the port. Because when a port is open, that means anybody can attempt to access that port. In some cases, you want people to access a port if it is a web server in question, but sometimes you do not want others to attempt to access a port, and you want to limit that access in some way. In the video, we will learn how we can allow traffic from one IP to a port, how to allow traffic from a list of IPs (whitelisting) and how to create a list of IPs you do not want to allow (blacklisting).
firewalld Part 5 - ipset, blacklist, redirect
Now that we have created our blacklist and added an IP address to it, we need to state that we do not want the firewall to allow any traffic that is coming from any address on the blacklist. This can be done by using one command and performing a reload. We can then check to verify that traffic will not be allowed anymore. In general, it is not efficient to add a one-by-one IP address to a blacklist. Instead, we can create a file with a list of IP addresses and then add them all together at once. We will also learn how we can remove entries from a file from a blacklist, as well as how we can forward traffic from one port to another.
firewalld Part 6 - Lockdown, Panic
In this video, we will learn how we can forward traffic from one server to another. This is useful in scenarios where someone is trying to perform a brute force attack because you can redirect that IP address from which the attack is coming from your server to a different server. We also have a lockdown mode that prevents any other applications to perform any changes to the firewall, which adds another layer of security. We also have panic mode and recovery that will basically disable access to the server to yourself as well, and there will be no incoming and outgoing traffic to the server.
SELinux Part 1 - SELinux States
Here we are going to start talking about SELinux, which will be the topic of discussion in the next several videos. SELinux is a tool that allows fine-grain control over access to files, controls, processes, or other things on the system. It is very effective and used by practically all Red Hat-based distributions. You cannot fully secure your system without fundamentally knowing the security mechanisms that the operating system is using. We will talk about why SELinux is a good tool to use and go over the three SELinux states: Enforcing, Permissive, and Disabled. We will see how we can set SELinux to these states and switch between them.
SELinux Part 2 - Move, Copy, Create
In this video, we will continue talking about SELinux context and see how that relates to moving, copying, and creating files. We will see how SElinux behaves when these actions are performed. This is very strict and very well defined. Moved files will retain their current context, while copied files might or might not retain their current context. Created files will inherit the context for the location where they are created, so these are some things that should be watched out for. In this video, we will test out these behaviors directly through an example.
SELinux Part 3 - SELinux Context
Continuing on the topic of SELinux, we start talking about SELinux contexts. Contexts are written for Linux files, and the context of a file basically defines who can access that file, who can read the data in it, what the process can do (if it is a process file), what kind of access something has, etc. Everything on the file system has a specific context tied to it. In the video, we will talk about three different contexts — User, Role, and Type — in more detail. We will list out the contexts on a file system to see how they look. Every file and folder kind has a predefined context, whether they exist or not. These contexts are assigned to them when they are created.
SELinux Part 4 - Context Adjustments
Like we mentioned before, context can be changed and adjusted if needed for various reasons. In this video, we will learn how we can do that through an example and learn several new commands. We will create a file that will have a certain context assigned to it with certain access permissions. Then, we will attempt to change this and see what will happen. We will also learn how we can remove a file context. Changes that are made to files are recorded if you want to check and see what modifications have been made.
SELinux Part 5 - Booleans
SELinux also enables us to make temporary changes that are not saved upon system reboot. We mentioned in the previous videos that contexts can be listed out for files, but they can also be listed out for processes as well. In this video, we will mainly talk about Booleans. They also enable us to determine certain rules of a context and are used to define if something can or cannot be done. We will take a look at some of these Booleans, see how we can set them to temporary or permanent changes, and how we can check if the default SELinux context is set.
SELinux Part 6 - Port Labels
The topic of this video is Port Labels. Even though SELinux is not a firewall, it has the ability to allow or deny communication on a specific port to a process. In this video, we will go over some useful commands that will enable us to list restricted ports and protocols, to add a port or protocol to a label, and to remove a port or protocol as well. You will learn through a practical example where we will attempt to change the Apache listening port and configure SELinux to allow listening through another port.
SELinux Part 7 - SELinux Domains
This time we will talk about SELinux domains. In general, a mistake that people often make at first is that they set everything to permissive mode on SELinux, which causes various problems. This does not need to be done because it is pretty simple to figure out what specific domain can be put into a permissive mode, and if you can figure out in which domain the process runs, then you just put that domain in permissive mode. In this video, we will take a look at how we can configure these domains.
SELinux Part 8 - SELinux Modules
Today we will talk about SELinux modules. Modules can be tied to logs, and you can generate policy modules based on the log files. We will go over several essential commands tied to SELinux modules. We will go over the commands for listing out, loading, and disabling SELinux modules. We will also see how we can generate a policy module based on the log entries.
SELinux Part 9 - SELinux Logs
This time we will talk about SELinux logs. Firstly, we will install a few tools and packages that are needed for log analysis, and then we will restart
auditd after the install. After that, we will go over to the
ausearch, where we will go over a few examples to see how we can search through the log files. When that is done, our goal is to look for process-related actions and look for failed login attempts.
Server Access and Authentication Configuration
Initial SSH Configuration
SSH Configuration Part 1 - No root Login, Change Default Port, Adapt Firewall and SELinux Rules
We are going to play around with the access to the system. In addition to doing some base configuration by just changing the default port at which you connect via SSH service port. We will also perform changes such as adding extra layers of authentication and creating a jump point server, which is similar to a VPN with SSH. First, we will disable all root login, after which we will change the SSH port from 22 to 61613. After that, we will open port 61613 with
firewalld. Then, have SELinux allow port 61613 for SSH, after which we must restart SSH. Lastly, we will log out and log back in via SSH on port 61613.
SSH Configuration Part 2 - Ports and firewalld
Continuing on where we left off from the last video, here we will first close port 22 if it is open, after which we will reconfigure
firewalld service SSH to run on port 61613 and add the service. It is then important to close port 61613 via firewall. With that, we can say that we have partially configured SSH as a means of logging in and communicating with the server for the purposes of maintenance and control.
Multi-Step SSH Verification
Configure a 3-Step Verification for SSH Part 1 - Key-Based and Key Generation
In this video, we will set up public-key authentication, and after we set it up, we will combine it with passwords, so that both the key and passwords are required. First, we need to generate a private-public key pair on your client with which you will connect to the host. Then we will need to copy the public key over to the server. When everything is set, we will attempt to login to the server using the key we created.
Configure a 3-Step Verification for SSH Part 2 - Adding Password-Based Authentication
Continuing on, we are now going to add a password-based login on top of the key-based login. Technically, the password-based login already exists, but SSH is currently configured in a way where it will choose whether it will use key-based authentication or password authentication. What we are going to do is configure it so that it asks for both. First, we will edit the
sshd_config file and change the
PasswordAuthentication parameter and add an
AuthenticationMethods line. After that, we will confirm that it works by testing it out.
Configure a 3-Step Verification for SSH Part 3 - Google Authenticator
In this video, we will proceed to create a third layer of authentication. We will integrate google authenticator where we will need our phone as well to log in to the server. This significantly improves our security since it is highly unlikely that someone has access to your key, your phone, and your password simultaneously. This kind of configuration is usually done on a jump point, which we will talk about in the upcoming videos. First, we will configure a repository, after which we will install Google Authenticator. The next step is to initialize it and lastly configure SSH to enable all three methods at once.
SSH Jump Point
SSH Jump Point Part 1 - Characteristics
In this introductory video, you will get acquainted with jump points. You will learn what jump points are and what they are used for. A jump point is a place where you connect that allows you to connect to the rest of your infrastructure. They can significantly improve security to your front-facing and infrastructure servers. Here we will define several characteristics for the jump point we will be creating. We want to create a 3-step verification process and allow traffic only from a given range of IP addresses to port 61613. We want SSH to run on port 61613, and we want SELinux to enable port 61613 for SSH. We want to disable connections on port 22 with
SSH Jump Point Part 2 - Container Server Configuration
Here we will proceed with creating our custom jump point. On the container server, we need to create such a configuration that it only accepts connections via SSH from a particular jump point. We need a firewall rule that allows traffic on port 61613 only from the jump point IP address. We need SSH to work on port 61613, and it won’t allow for root login and password-based authentication. First, we will install
firewalld, after which we will configure SSH. We will also need to generate a key which we will connect to our infrastructure server. If everything is configured correctly, we should be able to connect successfully with our jump point key.
SSH Jump Point Part 3 - Container Server Configuration
Continuing on with our jump point configuration, now we will allow IP addresses only from a specific place to go to the container server. The jump point server has one public IP address but it also has a private IP address and you generally do not know which one will be used. So you want to add both the private and public IP address to your infrastructure server (which is one of our container servers) just in case. We will also add a few rich rules. All that is then left to do is reload to the rules are applied.
SSH Jump Point Part 4 - SSH Tunnel, SOCKS5
Now that we have set up our jump point, let's see how we can utilize it. The first thing we can do is log in to our jump point from our client machine. Then from there, we can jump to our container server. In this video, we will take a look at a command that will help us configure our proxy system on our local machine and see how a web browser would use this proxy. The command will allow us to connect to the server via SSH via the jump point, but it will also allow us to be able to connect to other services that might be running on the server.
SSH Jump Point Part 5 - SOCKS5 Proxy
In this video, we will see how we can configure the webserver to accept connections on one port from one IP address and on another port from all IP addresses. We will configure the webserver to only allow connections from the jump point to go to the backend. First, we will connect to our container server and install HTTPD. After that, we will open up the
httpd.conf file, add our configuration code, then restart HTTPD. If there are no warnings or errors, all is good. After that, we will add our firewall rules. When everything is set, all that's left is to verify if everything is configured correctly.
Logs and Notifications
Seccomp - Secure Computing
Seccomp Part 1 - Introduction
In this video, you will be introduced to Seccomp (Secure Computing Mode) and get an idea of what the following videos will be about. This is a very important tool that we will be using along with containers and properly imposing some additional limitations on them. We use it to restrict system calls. Basically, it participates in jailing a process and limiting what can be done from the process itself. It gives us the ability to dictate what the process cannot do. It can forbid system calls to be made, define which system calls can or can not be made, etc.
Seccomp Part 2 - Check Seccomp Status from within a Program
Now that you have learned what seccomp is, here we will talk about seccomp’s two different modes. There are two modes: strict mode and filter mode. Filter mode is more advanced, while strict mode is more limited. We go through a practical and simple C++ example where the code checks if seccomp is available on a system or not and prints out a message based on the status. The code additionally checks to see what Seccomp modes are available on a system as well. We will also see how we can check the mode of seccomp for a process using a terminal command.
Seccomp Part 3 - Strace and Syscalls
If you want to be able to control what system calls a process can make, then there needs to be a way of knowing what calls a particular process makes. This can be done using strace. Strace allows us to find out which calls a process has made after we run that process with strace. It essentially detects what calls are made during that run and lists them out.
Seccomp Part 4 - Seccomp in Code Restrictions
This time we will take a look at the incode limitations of seccomp and see how it looks in the code itself; our goal is to write a program that requires
write access to a set of files or a file. Then, write a program that will make a series of those system calls needed to create a file, then read from it, change permissions on a file, etc. This program will be written with implemented seccomp limitations and security measures. When all the code is written, we will need to run tests to verify that the limitations and security measures are valid and functional.
Seccomp Part 5 - Seccomp in Code Restrictions
Now that we have written all the necessary code, we will see how it all works in practice. We will compile our program and take a look at the output. After that, we will play around with the code a bit and change some things in order to see what difference that will make when we recompile the program. We will change some default policies, play around with whitelists, blacklists, etc. We will also run the program with strace in order to get more information about the system calls.
Seccomp Part 6 - Seccomp in Code Restrictions Troubleshooting
In this video, we will start working through our third example, where we will need to write a program and a corresponding
systemd file. After that, we will implement seccomp limitations. Our first task will be to write a program that requires
write access to a set of files. Then, we will write a program that makes a series of system calls, after which we will create a corresponding
systemd service file for the program in order to daemonize it. Furthermore, we will implement seccomp restrictions to the program in the service file. Lastly, we will apply the changes and run tests to verify that the limitations and security measures are valid and functional.
Seccomp Part 7 - Seccomp in Code Restrictions Command-Line Arguments
Here we will talk about a certain feature that is common in programs that have seccomp integrated in their source code. This feature allows you to pass a specific argument at the command line that determines the mode of seccomp at the start of the program. That way, you can choose if you want a particular mode and whether or not you want seccomp to function at all. You cannot remove seccomp from the source code, so this is a ] useful feature because, in some cases, you might need to disable seccomp, and there really is not any other way to do that, then have a specially implemented option. We will go over a practical example and see what this looks like.
Seccomp Part 8 - Seccomp systemd Restrictions Troubleshooting
In order to use seccomp with
systemd, we will first need to create a service file for our program. The program file will need this service file, and then it can be daemonized and run in the background and be handled by
systemd. We will first create our service file and change its permissions. Then we will populate our file with three necessary parts: Unit, Service, and Install.
We must not forget to reload
systemctl before starting the service since we just added a new file. After checking the status of our service, we will encounter some problems that we will attempt to resolve by using some of the commands we have learned so far and changes some parts of our program.
Seccomp Part 9 - Seccomp systemd Restrictions Troubleshooting
We will need to conduct some modifications to our service file to make our program fully functional and make it work to its fullest capacity. One of the first things that we will add is the user in whose context this application runs. We will also add a working directory for the program and set restart to
always. After that, we will check the status and see if everything is working properly. In general, when creating a
systemd file, it is important to consider all requirements of the program, such as: Does it require operations within a certain directory? Does it require a network? Or certain dependencies on the system?
systemd files are created, the program can run while encountering several problems. This is why we have gone through a troubleshooting process of common problems because you may very likely encounter several errors when doing this.
Seccomp Part 10 - Seccomp in Code Restrictions
Now that everything is set, we can now start applying the seccomp filters to harden our program. We will completely remove the whitelist and seccomp filters from our program. We will actually move the whitelists and the rules to our
systemd file, where we will need to define them and make sure that all the program needs, in terms of system calls, is listed there. We modify the
SystemCallFilter that takes a space-separated list of system call names in our
systemd file. Here we will also encounter some problems that we will troubleshoot along the way.
Logs Part 1 - tail
Before we get into logs, in this video, we will go over a few commands that will really make our lives easier because log files tend to be extremely long. So it is necessary to have some way of processing these files easier, extracting data you need from them better, and search through them to find what you need easier. The command we will talk about in this video is Tail. Tail gives you different options related to printing out lines from a log file such as printing a specific number of lines or start printing from a specific line, etc. It also enables you to monitor a file live and precede data by file name.
Logs Part 2 - cat
Continuing on with our commands, in this video, we talk about the
cat gives you numerous options, such as outputting the contents of a single file or multiple files if needed. You can also use it to create files, redirect outputs to and from files in different ways, as well as display line numbers of a file. In this video, you will see in better detail how to perform these actions.
Logs Part 3 - Log Files
In this video, we will take a look and see where Linux stores its log files by default, what kind of log files are located there, and what they are used for. First, we will open up the messages log file, in which general messages and system information can be found. It is considered a kind of global logging file that contains information that does not really have a place in other log files. We will also take a look at the
mysql log, and
yum log, and filter out some information to see what we can find in these logs.
Logs Part 4 - auditd Start, Stop, Restart, Reload, Status, Rules, and General Information
Logs Part 5 - Audit Configuration, Log Configuration, systemd Service File
Continuing on, this time, we will take a look at three relevant files related to audit. One of them governs the process itself; this is the
auditd.conf configuration file. Another one is used to write the rules of what will be logged and what will be monitored; this is the
audit.rules file. The last file is the
audit.log log file where the results are written. We will go a bit more into detail about what kind of information that is in these files and how they are written.
Logs Part 6 - File System Rules
In the last video, we talked about the relevant files we have related to audit. Now, you will learn how to implement a rule for a certain file. First, we will create a file of an arbitrary name that we will monitor. The process is the same for any kind of file, be it a log file, system file, etc. We can implement a rule by editing the
audit.rules file we mentioned in the last video, but we can also implement a rule at runtime, which is what will be demonstrated in this video. After we implement the rule, we will again take a look at the created log files upon these changes to see what information was recorded and what can be found there.
Logs Part 7 - System Call Rules
In this video, we will be doing some system call examples with rules. First, we will create the rule at runtime as such, where we will configure the action to happen always. The filter will be an exit, which means that the event will be logged on
exit process. We will also set it to log the
chmod and open system call, and we will set it to monitor all users with the user IDs up to 2000. Lastly, we will set a key. When the rule is configured, we will then start the monitoring process and check out our log files. To make our rule permanent, we will need to edit the
audit.rules file and paste our rule setup there.
Logs Part 8 - Audit Config Troubleshooting
Continuing on from where we left off in the previous video, here we will need to do one more thing in order for our rule to be permanently implemented. In addition to editing the
audit.rules file, we also need to move the
auditd.service from its current location which is
/etc/systemd/system. If this is not done, everything you have configured in the rule will be lost at reboot.
Logs Part 9 - journalctl
In this video, we will talk about the
journalctl command and see what kind of things we can do with it and what it is helpful for.
journalctl collects data from whatever source is available. Since the data we have will be in binary form, we either need to know exactly how it was written to read and manipulate it, or we can use
journalctl can be used to display timestamps in UTC, to display entries since the last reboot, filter by PID, and various other things that will be covered in the video.
Logs Part 10 - journalctl
Continuing with our
journalctl command, here we will go over some more options that this command provides us. We will see how we can get kernel messages, filter by priority (we have seven priority levels), modify output format, get the number of entries to show, actively follow logs, see how much space the logs are using and how to remove old logs and downsize log data size.
Notifications Part 1 - General Talk
In this section, we talk about notification systems. This video goes through a general talk about the subject. Notification systems are often not given as much value as other security aspects. However, having a well-configured monitoring and notification system can be very useful and important when it comes to security. Notifications can be set to notify us for various things such as sending an email if a user logs in, messaging if there is high RAM or CPU usage, messaging if a service has crashed, etc. We go over the parts that make up a notification system such as monitoring, triggers, and notifications.
Notifications Part 2 - Email Notifications Cover Password
Now that we've said a few words about notifications, let's try and set one up ourselves. Here we create a notification that will go off every time a user logs into the server. First, we install the prerequisites and then code our script in Python 3.6. If everything is done correctly, the notifications should be sent via email when we log in to the server.
Notifications Part 3 - SMS Notifications, AWS, LAMBDA
In this video, we learn how to set up notifications to arrive via SMS on our phones. To achieve this, we use the Amazon Web Services (AWS). In order to do this, we create a lambda function in Python 3.6. We also add a trigger for an API call and set up all necessary parameters. In the configure features, we select SMS and voice. Then we set up the origination and destination numbers, along with some other configuration settings.
Notifications Part 4 - SMS Notifications, AWS, LAMBDA
After adding the phone numbers, we need to add a policy to allow this to go through. When the policy is set and we test out the notification, we should receive a notification that the HTTP service is down if everything is okay.
Notifications Part 5 - SMS Notifications, AWS, REST API CALL
In this video, we take what we did in AWS and make calls from the terminal to the API we created. We will write a Python script where we specify the checks we want to perform and the conditions that need to be satisfied to perform them. We will make a set where we check if the output of
systemctl is active or not. If it is not active, we want to send a request to the API that will call the lambda function which starts the process of sending the SMS notification.
Notifications Part 6 - SMS Notifications, AWS, REST API CALL
Continuing on from the last video, we implement a few fixes to what we've already created. When we apply the fixes, we will check and see what changes have happened. Finally, we wrap up what we've learned about notifications in this section.
Scripting, Task Automation, and Vulnerability Scans
Vulnerability Scans and Reports
Nmap Vulnerability Scans Part 1
In this video, we perform a simple scan of our server that does a sweep for known vulnerabilities and generates a good report. With automated scans, we protect ourselves from most potential attackers. This is done by eliminating all the known vulnerabilities from our system by doing things such as updating the system, etc. The script that we write needs to conduct scans of the infrastructure, generate vulnerability scan reports, and send reports back to the admin.
Nmap Vulnerability Scans Part 2
In this video, we see how to put a scan into a script, send ourselves an email, and schedule the script to run once on a weekly basis. We pack the scan results into an email and send it.
Let's review what we covered in the course and briefley summarize it.
Take this course and learn a new skill today.
Transform your learning with our all access plan.Start 7-Day Free Trial