Have you ever imagined running a company and you have no record of transactions or activities that happens in the company?. You will ruin the entire state of the company when issues arise, and you have no record to fall back.
In Linux, different files reside in a directory which handles various logs in the Linux systems. This, in turn, helps the system administrator to have a document to fall back to enable them to troubleshoot the system when issues arise.
By convention, the /var/log directory is where these logs are persistently stored. In Linux, the syslog messages are managed by two services. The rsyslog service and systemd-journald service.
Now let's look at how these two services are related and how they work. The systemd-journald provides better management of log messages. This daemon collects log messages at the start of system boot, including the error messages and writes these messages to a structured journal which by default, is not persistent after the system reboots. In turn, these messages are forwarded by the systemd-journald to rsyslog for further processing. The rsyslog will then categorize these messages base on priority and severity and writes them to files where it will remain permanently.
I guess you now understand the relationship between both services. They work together for better management of your log messages.
In this article, you will learn how to monitor log files and how to use logger command to log files.
UNDERSTANDING PRIORITY AND FACILITY OF A LOG MESSAGE
Every log messages are categorized based on its priority (The severity of the message) and facility (The type of message).
They are various facility-level supported by Linux. The facility is one of the following keywords: lpr, mail, mark, news, syslog, authpriv, auth, cron, daemon, kern user, uucp, local 0 through local 7.
I will be explaining a few:
auth: Used for security events
authpriv: Used for access control related messages
cron: Used for cron jobs
daemon: Used by process and daemons
kern: Used for kernel-related messages
The priority levels in Linux are in increasing order of urgency. They include;
Debug, Info, Notice, Warning, err, crit, alert, emerg.
Below are the eight priorities and how they are standardized and ranked
0 emerg: The system is unusable
1 alert: Action must be taken immediately
2 crit: The system is in critical condition
3 err: Non-critical error condition
4 warning: Warning condition
5 notice: Normal but significant event
6 info: Informational message
7 debug: Debugging messages
The rsyslog uses the priority and severity level to determine where a message should be logged, which is configured in the /etc/rsyslog.conf file
Now, let's view the /etc/rsyslog.conf file to have an overview of how messages are logged in Linux
The rsyslog service maintains log files in the /var/log directory which hold files where different log messages are categorized and stored. For example, email log messages are stored in /var/log/maillog
They are various log files that reside in the /var/log directory;
/var/log/messages: This file stores most syslog messages except messages that are related to the system debug, authentication or email processing
/var/log/secure: The file holds error-related messages and authentication and security-related messages.
/var/log/maillog: It holds mail server related messages
/var/log/cron: This file holds cron jobs messages
/var/log/boot.log: This file holds messages related to system startup
HOW LOG FILES ARE ROTATED
Have you ever imagined junked of messages packed up in your system?. It occupies lots of system memory. Even when files are saved persistently in Linux systems, they also are not stored forever. A time will come when those messages will be wiped out from the system to preserve the system memory.
To save you the stress of having to go to your /var/log directory to be deleting old log messages, Linux decided to integrate its Operating system with the logrotate utility. The logrotate utility is designed to ease administrative task on Linux systems that generate large log files. It allows automatic rotation, compression and mailing of log files. Rotating of files could be handled daily, weekly or monthly depending on your configuration.
Every file rotated is renamed with the date which it was rotated. The cron job runs the logrotate utility every day to see if any file needs to be rotated. For the example, the /var/log/messages file will become /var/log/messages 20190823 after rotation
HOW TO ANALYZE SYSLOG ENTRY
The rsyslog service writes the system logs starting with the oldest message at the top to the newest message which resides at the bottom of the file. All messages that are logged are recorded in a standard format as shown below
With the indication to give a better view of the explanation. The first sector indicates the time stamp at which the log entry was recorded. The second sector shows the hostname from which the log message was sent. The third sector indicates the process that sent the log messages and the fourth sector is the actual message sent.
HOW TO MONITOR LOG FILES
To monitor log files as they are written to the system, you use the tail command with the –f option. The tail command with the –f option gives a realtime overview of the messages as they are logged in. By default, it displays the last 10 lines of the log messages. You can also use the cat command, the head or less command, to view your log messages. Using these commands implies running the commands every time to see newly logged messages. So, it is convenient to use the tail command with the –f option
You can see the command prompt waiting to display to the terminal newly logged in messages
HOW TO SEND THE SYSLOG MESSAGE TO THE RSYSLOG SERVICE
You can also use the logger command to send a message to the rsyslog service. By default, it sends the message to the facility user with the severity notice (user.notice) unless specified otherwise with the –p option.
Follow the command below. I’m going to open two terminals. In one terminal I will run the tail command with the –f option to see my logged messages immediately they are logged, and in the other terminal I will run the logger command to send the message to the rsyslog service
To log a message, you can use the command logger [message] or specify the –p option to log to a specified facility level. As seen below;
logger –p [facility.priority] [message]
I just logged a message to the rsyslog, and this is the output on another terminal.
FIVE CRITICAL LOG FILES YOU SHOULD BE MONITORING AS A SYSTEM ADMINISTRATOR
As a system administrator, listing and analyzing all the log files in your system could be tasking. As there are lots of log messages in the system, it could take days to analyze all of them when issues arise.
This is why I have researched to add to this article some few log files that you could be monitoring for the security of your server. Meanwhile, this is not a comprehensive list of all the vital log files but this a tip of what you should be monitoring.
Here, I will be introducing some crucial files you need to be monitoring for better security of your Linux system.
It provides authentication events that are logged in by any user in the system and also tracks their activities
It provides the number of failed and successful attempts in the system. So it can be useful when detecting possible hacker attempts
It also tracks sudo logins, SSH logins and other system daemon error messages
you should analyze this log file to investigate issues related to improper shutdown or booting failures. Can also be used to determine the duration the system has been down which is caused by an unexpected failure
If you are having issues with some hardware in your systems or some hardware are not getting detected, you can fall back to this file to detect the issue. It is also useful for dedicated customers.
It contains information about failing log in regarding attempted security breaches as regard to username and password attack. It can be used to detect any security breaches in your system
This file track the installation of your software. You can check this file to see whether a package was installed correctly or not. It helps you troubleshoot issues related to software installation.
In a situation where your server is not behaving normal as usual, the file will help you track down the particular software that is causing the issue since the recent install.
In summary, monitoring all log files is hard, but there are readily available open-source tools that can help you monitor your log files in real-time such as Nagios log server. It is worth investing in getting a monitoring tool as monitoring log servers manually can be really hard
Log files are managed by the rsyslog service, which is stored in the /var/log directory. The directory holds a variety of log files which are separated according to their type.
Read through the man logger, man logrotate, man rsyslog.conf to get better information on log entries and the system log architecture