Tips on probing log files after your Linux server crashes

When your Linux server crashes, one place to look is in /var/log/kern.log for clues as to what went wrong.

This file contains a lot of output that can be difficult to sift through. One way to navigate this log file, is by noting the first line of the log file.

The first line in my log looks like this:

May 29 09:18:24 lab kernel: [    0.000000] Linux version 4.15.0-1039-aws (buildd@lgw01-amd64-034) (gcc version 7.3.0 (Ubuntu 7.3.0-16ubuntu3)) #41-Ubuntu SMP Wed May 8 10:43:54 UTC 2019 (Ubuntu 4.15.0-1039.41-aw
s 4.15.18)

The log file starts off with a line about the Linux version. You can use this as a sort of anchor point for navigating through the log file. Now you know what line to search for in order to see where the server has stopped and started. When analyzing your log file, you can search for all occurrences of this line to understand where in the log file your server had restarted.

If you use the command less /var/log/kern.log it will open the log file in less, and you can navigate the output more easily.

When you type a forward slash in less, you can then enter some search terms, such as Linux version, and then you can use n and p to go to the next or previous occurrences of your search term.

Try searching for Linux version in the output, this allows you to navigate to the points within your log file at which the server has started. From these particular points in the log files, you can then look further up the log file to see if anything sticks out as to what may have caused your server to crash.