Configure logs
In the last chapter Update Linux server we updated the operating system.
In this chapter we will look at where the log files are stored and how we can read them. Admittedly, this is not the most exciting chapter and the minor reconfiguration we will do will not bring any security or performance optimization. You could just skip this chapter and your server would run just as well. However, every administrator should know about log files. If something doesn't work, they are the first place to go for troubleshooting.
The log file, which logs logins and logouts, is additionally written to by background processes. This makes it confusing and difficult to read. Therefore, we will make a setting to record the "uninteresting" events separately. This way the entries are neatly separated, which can be analyzed more easily.
Log files
The central location for log files is /var/log.
__$ sudo ls -l /var/log
Output:
total 11264
-rw-r--r-- 1 root root 31252 Jan 4 22:20 alternatives.log
drwxr-xr-x 2 root root 4096 Jan 4 22:19 apt
-rw-r----- 1 syslog adm 1294710 Jan 4 22:23 auth.log
-rw-r--r-- 1 root root 104003 Jan 3 2021 bootstrap.log
-rw-rw---- 1 root utmp 1100928 Jan 4 20:40 btmp
-rw-r--r-- 1 syslog adm 474428 Jan 4 22:28 cloud-init.log
-rw-r----- 1 root adm 23633 Jan 4 22:28 cloud-init-output.log
drwxr-xr-x 2 root root 4096 Jan 3 2021 dist-upgrade
-rw-r--r-- 1 root adm 61927 Jan 4 22:28 dmesg
-rw-r--r-- 1 root adm 61624 Jan 4 22:17 dmesg.0
-rw-r--r-- 1 root adm 16365 Jan 4 15:02 dmesg.1.gz
-rw-r--r-- 1 root adm 15996 Jan 4 15:00 dmesg.2.gz
-rw-r--r-- 1 root adm 16027 Jan 4 19:35 dmesg.3.gz
-rw-r--r-- 1 root adm 16092 Jan 4 19:34 dmesg.4.gz
-rw-r--r-- 1 root root 497519 Jan 4 22:21 dpkg.log
-rw-r----- 1 root adm 20285 Jan 4 10:53 fail2ban.log
-rw-r--r-- 1 root root 32064 Jan 4 21:31 faillog
drwxr-xr-x 3 root root 4096 Jan 4 12:09 installer
drwxr-sr-x+ 3 root systemd-journal 4096 Jan 4 12:14 journal
-rw-r----- 1 syslog adm 2322573 Jan 4 22:23 kern.log
drwxr-xr-x 2 landscape landscape 4096 Jan 4 12:18 landscape
-rw-rw-r-- 1 root utmp 292584 Jan 4 22:23 lastlog
-rw-r----- 1 syslog adm 2627 Jan 4 22:12 mail.err
-rw-r----- 1 syslog adm 303319 Jan 4 22:23 mail.log
drwx------ 2 root root 4096 Jan 3 2021 private
-rw-r----- 1 syslog adm 1547818 Jan 4 22:23 syslog
-rw-r----- 1 syslog adm 1683517 Jan 4 00:00 syslog.1
-rw------- 1 root root 0 Jan 3 2021 ubuntu-advantage.log
-rw------- 1 root root 157 Jan 4 22:20 ubuntu-advantage-timer.log
-rw-r----- 1 syslog adm 1723687 Jan 4 22:23 ufw.log
drwxr-x--- 2 root adm 4096 Jan 4 12:14 unattended-upgrades
-rw-rw-r-- 1 root utmp 83328 Jan 4 22:23 wtmp
Log files are typically divided into four categories:
- Application
- Event
- Service
- System
The SSH service logs to the file /var/log/auth.log. Let's have a look in there:
__$ sudo less /var/log/auth.log
At the beginning of each line is the date and at the end log message. As you may have noticed, there are also login events that cannot be from you. These contain the phrase pam_unix(cron:session) and come from internal processes. But more about that later.
For example, we can filter the output with grep and a search word. The next command shows all lines where the phrase user tom is mentioned:
__$ sudo less /var/log/auth.log | grep "user tom"
Another example with the phrase cron:session:
__$ sudo less /var/log/auth.log | grep "cron:session"
Output the latest events and read them live
With tail a certain number of the last lines can be output. For example the last 10:
__$ tail -10 /var/log/auth.log
Log files can be read live. This helps to not have to constantly close and reopen the log file. New entries or changes are output with the parameter -f:
__$ tail -f /var/log/auth.log
The key combination CTRL+c terminates the output.
At the moment, hardly anything is happening. This will be more interesting later when we observe the logs from the server service.
Logrotate
After a while, a lot of data ends up in the logs. The files are bloated and consume corresponding hard disk space. To counteract this, the log files are emptied at certain intervals, but of course copied and archived beforehand. The archives get the original file name and additionally a sequential number. This process is called "logrotate". Old log archives are then eventually deleted. Each installed package or service that logs can store its own configuration for this at the logrotade.d service and design the process individually.
Check version of logrotade:
__$ sudo logrotate --version
The configuration files are located in the folder of the same name /var/logrotate.d. There we will create another Logrotate configuration, for the SSH service.
__$ sudo ls -l /etc/logrotate.d
Excursus Cron
To make the next section more understandable, here is a brief explanation of crons. A cron daemon is a service that runs in the background and triggers processes at specified times. Also called a "cron job." Hourly, daily and weekly intervals are configured by default. Each user can have their own cron table. The cronjobs from the root user are in the /etc/crontab file. Let's have a quick look:
__$ sudo cat /etc/crontab
For me, the crontab table looks like this:
# /etc/crontab: system-wide crontab
# Unlike any other crontab you don't have to run the `crontab'
# command to install the new version when you edit this file
# and files in /etc/cron.d. These files also have username fields,
# that none of the other crontabs do.
SHELL=/bin/sh
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
# Example of job definition:
# .---------------- minute (0 - 59)
# | .------------- hour (0 - 23)
# | | .---------- day of month (1 - 31)
# | | | .------- month (1 - 12) OR jan,feb,mar,apr ...
# | | | | .---- day of week (0 - 6) (Sunday=0 or 7) OR sun,mon,tue,wed,thu,fri,sat
# | | | | |
# * * * * * user-name command to be executed
17 * * * * root cd / && run-parts --report /etc/cron.hourly
25 6 * * * root test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily )
47 6 * * 7 root test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.weekly )
52 6 1 * * root test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.monthly )
#
You probably have other numbers, because they are set at the time of the Linux installation. The lower lines or the non-commented lines represent a table. With the first columns, the interval can be set and at the end are the commands that are then triggered. In my case: the hourly cronjob at the 17th minute (around the clock) with the command cd / && run-parts --report /etc/cron.hourly. Any executable in the etc/cron.hourly directory will be run with run-parts. Let's take a quick look in the folder:
__$ sudo ls -l /etc/cron.hourly
There are probably no executable files at the moment.
The situation is different for daily crons:
__$ sudo ls -l /etc/cron.daily
Redirect cron sessions
It happens that applications launched at cronjobs authenticate themselves to the system via the PAM interface. They generate the cron:session log entries mentioned above. They are actually uninteresting as an entry and just make the /var/log/auth.log messy. Therefore, we will redirect them to another file.
Let's first create a log file for the cron session events with touch:
__$ sudo touch /var/log/pam-cron-session.log
We change the file permissions with chmod to 0640:
__$ sudo chmod 0640 /var/log/pam-cron-session.log
Since we are the owner of the file, there would only be problems if background processes without permission wanted to write in there. As new owner we appoint the system user syslog from the group adm. We change owner and group with chown:
__$ sudo chown syslog:adm /var/log/pam-cron-session.log
To catch said log events, we place a configuration file at the rysylog.d service. This service evaluates events and writes log messages to appropriate log files. Let's create a new file (/etc/rsyslog.d/30-cron.conf) with nano:
__$ sudo nano /etc/rsyslog.d/30-cron.conf
And write in the following:
Excerpt from /etc/rsyslog.d/30-cron.conf
# Cronjob-Spam redirect
:msg, contains, "pam_unix(cron:session):" -/var/log/pam-cron-session.log
& stop
Save and close (CTRL+s, CTRL+x).
The configuration says: if pam_unix(cron:session): is in a log message (:msg, contains), then write it to the file /var/log/pam-cron-session.log. The hyphen -/ first caches or collects the events before writing them. This reduces the write accesses on the hard disk.
Let's optimize this further with a logrotate. As discussed above we create a configuration file for the logrotate.d service:
__$ sudo nano /etc/logrotate.d/pam-cron-session
And write in the following:
/etc/rsyslog.d/30-cron.conf
/var/log/pam-cron-session
{
rotate 4
weekly
missingok
notifempty
compress
delaycompress
sharedscripts
postrotate
invoke-rc.d rsyslog rotate > /dev/null
endscript
}
Save and close.
This will rotate weekly (weekly) and no more than 4 archives will be created. The fifth archive, which is the oldest, is deleted.
All that remains is to restart the rsyslog.d service to apply the new configuration:
__$ sudo systemctl restart rsyslog.service