Skip to main content

Troubleshooting kernel issues

Hands-On Lab

 

Photo of Michael Christian

Michael Christian

Course Development Director in Content

Length

01:30:00

Difficulty

Advanced

In this lesson, you will enable kernel dumps and give cloud_user the ability to run a SystemTap module.

What are Hands-On Labs?

Hands-On Labs are scenario-based learning environments where learners can practice without consequences. Don't compromise a system or waste money on expensive downloads. Practice real-world skills without the real-world risk, no assembly required.

Troubleshooting kernel issues

Introduction

In this lesson, you will enable kernel dumps and give cloud_user the ability to run a SystemTap module.

Configure the system to perform a kernel dump upon a crash, using no more than 128 MB for the crash kernel.

Permit cloud_user to run /root/example.stp as a module stap_example.

Solution

Start by logging in to the lab server using the credentials provided on the hands-on lab page:

ssh cloud_user@PUBLIC_IP_ADDRESS

Become the root user:

sudo su -

Enable kernel dumps

Note: The hypervisor used by EC2 prevents kdump from being engaged. Forcing a kernel crash will simply prevent you from logging back in for 3-4 miniutes. No crash will be present in /var/crash.


  1. Start and enable the kdump service:

    systemctl start kdump && systemctl enable kdump
  2. You'll notice kdump doesn't start. Let's check the status of the kdump service:

    systemctl status kdump
  3. Here, we see the error "Failed to start Crash recovery kernel arming". To see the memory settings for crashkernel, let's run the following command:

    cat /boot/grub2/grub.cfg | grep crash
  4. Install necessary packages:

    yum --enablerepo=rhui-REGION-rhel-server-releases-debug install -y kexec-tools crash kernel-debug kernel-debuginfo-$(uname -r)
  5. Modify /etc/default/grub to set crash memory to 128M.

    vim /etc/default/grub

    Change:

    GRUB_CMDLINE_LINUX="console=ttyS0,115200n8 console=tty0 net.ifnames=0 crashkernel=auto"

    To:

    GRUB_CMDLINE_LINUX="console=ttyS0,115200n8 console=tty0 net.ifnames=0 crashkernel=128M"

    Save and close the file:

    :wq
  6. Install the new Grub config:

    grub2-mkconfig -o /boot/grub2/grub.cfg

Permit cloud_user to run the example.stp via SystemTap

  1. Install SystemTap:

    yum install -y systemtap kernel-devel-$(uname -r)
  2. Run the example script: /root/example.stp

    stap /root/example.stp
  3. Compile the script as a module:

    stap -p 4 -v -m stap_example example.stp
  4. Create a systemtap directory for the module:

    mkdir /lib/modules/$(uname -r)/systemtap
  5. Move the module into the new directory:

    mv stap_example.ko /lib/modules/`uname -r`/systemtap/
  6. Run the module:

    staprun stap_example
  7. Add cloud_user to the stapusr group:

    usermod -aG stapusr cloud_user
  8. Verify cloud_user can run the module:

    runuser -l cloud_user -c 'staprun stap_example'

Conclusion

Congratulations, you've completed this hands-on lab!