How to Troubleshoot Boot Up Issues with CentOS 8

Hosting Expert 2 min read

May 22, 2024 11:37 AM

What Do I Need?

A Dedicated or VPS Linux Server
CentOS

The Problem

Your server has broken and just wonâ€™t appear to do what you need it to do. As a sysadmin you should always be looking for the root cause. So you need to do the analysis so as to avoid such failures in the future. In this how-to guide, weâ€™ll look at the basic troubleshooting of booting issues. This is a very high-level overview and things can get pretty deep depending on the booting issue.

Inspecting Logs
1. Looking at the logs of a previously failed boot can be super useful. If the system journals are persistent across reboots, which they should be, you can use the journalctl tool to inspect those pesky logs.
2. Remember that by default, the system journals are kept in the /run/log/journal directory, which means the journals are cleared when the system reboots. To store journals in the /var/log/journal directory, which persists across reboots, set the Storage parameter to persistent in /etc/systemd/journald.conf.
3. Using the following command you can access your journald.conf file and edit as required, ensuring to press ctr+x when finished to save and exit.
```
sudo nano /etc/systemd/journald.conf
```
1. To inspect the logs of a previous boot, use the -b option of journalctl. Without any arguments, the -b option only displays messages since the last boot. With a negative number as an argument, it displays the logs of previous boots. Use the following command:
```
sudo journalctl -b -1 -p err
```

Repair a systemctl Boot Issue

Enabling the Early Debug Shell
- By enabling the debug-shell service with systemctl, enabling debug-shell.service, the system spawns a root shell on TTY9 (ctrl+alt+f9) early during the boot sequence. This shell is automatically logged in as root so that administrators can debug the system while the operating system is still booting.
- Donâ€™t forget to disable the debug-shell.service when youâ€™re done debugging because otherwise, it leaves an unauthenticated root shell open to anyone with local console access.

Using the Emergency and Rescue Targets
1. By adding the following or alternative to the kernel command line from the boot loader, the system spawns into a rescue or emergency shell instead of starting normally. Both of these shells require the root password.
```
systemd.unit=rescue.target
systemd.unit=emergency.target
```
1. The emergency target keeps the root file system mounted read-only, while the rescue target waits for the sysinit.target to complete so that more of the system is correctly initialized; for example, the logging services or the file systems. The root user at this point can not make changes to /etc/fstab until the drive is remounted in a read/write state by using the following command:
```
mount -o remount,rw /
```
1. Administrators and sysops can use these shells to fix pretty much any issue that prevents the system from booting normally; for example, a dependency loop between services, or an incorrect entry in /etc/fstab. Existing these shells continues the regular boot process.

Identifying Stuck Jobs
1. During the startup process, systemd spawns a number of jobs. If some of these jobs cannot complete or unexpectedly fail, they block other important jobs from running. To inspect the current job list, administrators can use the following command:
```
systemctl list-jobs
```
1. What youâ€™re hoping for is something a little like above, however, if not, no drama. Any jobs listed as running must complete before the jobs listed as waiting can continue.

Conclusion

There are lots more useful commands to explore so be sure to keep an eye out for more incoming how-to guides on maintaining and monitoring your Linux servers.