UNIX / Linux command-line interface.

Upgrading Red Hat Enterprise Linux / CentOS

RHEL/CentOS 5–6–7 Migration: Logging

Linux logging has gone through enormous changes with the move to systemd and the associated journal daemon. Rsyslog is still included, and very likely will be needed in a large enterprise for its ability to collect logs on a central Syslog server. Meanwhile the journalctl command is a very useful but entirely new and unfamiliar tool.

Previous: Booting

The previous page described the changes in booting along the RHEL/CentOS 5–6–7 migration path.

3 — Logging

The logging mechanism has changed twice, although Red Hat hasn't changed what it saves to which file in /var/log for a long time. Most interesting things go into messages, authentication events that might contain sensitive information go into auth.log, and subsystems like printing and mail have their own files.

RHEL 5 Syslog
RHEL 6 Rsyslog
RHEL 7 journalctl plus Rsyslog

See my page on Linux booting and process control for the details on this. Journalctl is less complex than Rsyslog, if you consider Rsyslog capabilities like TLS-protected remote logging and message content matching and rewriting. But its use of database files and its own tool before you apply grep and sed and awk certainly gives it a different feel.

I was accustomed to the rsyslog.conf manual page saying "of the specified priority and higher" to mean worse or more important. But the journalctl manual page uses "lower" to mean worse or more important (in the section about -p to specify a priority). However, it also uses "higher" to mean mean "more important"! (in the section about highlighting and colors in the output) I have decided I should just say "worse" as higher and lower are now ambiguous!

RHEL 7 includes many subsystems moving toward a "file plus directory" rather than "just one file" configuration, and Rsyslog is the first example we've come to involving a change when moving from RHEL 6 to 7.

Rsyslog in RHEL 6:
Start Rsyslog daemon according to /etc/sysconfig/rsyslog,
configure it with the file /etc/rsyslog.conf

Rsyslog in RHEL 7:
Start Rsyslog daemon according to /etc/sysconfig/rsyslog,
configure it with the file /etc/rsyslog.conf and then apply all files /etc/rsyslog.d/*

Similarly, logrotate is configured first by /etc/logrotate.conf and then all of /etc/logrotate.d/*.

This is more along the lines of errata, but I have noticed that the rsyslog service does not successfully restart or even receive a HUP signal to re-read its configuration file. Notice that it does not have cron, secure, or or maillog open after the restart or HUP. After changes in rsyslog.conf or rsyslog.d/* it seems that you must reboot to get the right data going to the right files. I don't know why it works on the first time only.

[root@example ~]# lsof /var/log/* | egrep 'COMMAND|rsyslogd'
COMMAND   PID USER   FD   TYPE DEVICE SIZE/OFF   NODE NAME
rsyslogd  854 root    4w   REG    8,4   786764 533365 /var/log/messages
rsyslogd  854 root    6w   REG    8,4     3952    153 /var/log/cron
rsyslogd  854 root    7w   REG    8,4    15835 533366 /var/log/secure
rsyslogd  854 root    8w   REG    8,4     3230 533367 /var/log/maillog
[root@example ~]# systemctl restart rsyslog
[root@example ~]# lsof /var/log/* | egrep 'COMMAND|rsyslogd'
COMMAND    PID USER   FD   TYPE DEVICE SIZE/OFF   NODE NAME
rsyslogd  3576 root    6w   REG    8,4   787293 533365 /var/log/messages
[root@example ~]# reboot

 ...

[root@example ~]# lsof /var/log/* | egrep 'COMMAND|rsyslogd'
COMMAND   PID USER   FD   TYPE DEVICE SIZE/OFF   NODE NAME
rsyslogd  847 root    4w   REG    8,4   996534 533365 /var/log/messages
rsyslogd  847 root    6w   REG    8,4     9996    153 /var/log/cron
rsyslogd  847 root    7w   REG    8,4    23397 533366 /var/log/secure
rsyslogd  847 root    8w   REG    8,4     3398 533367 /var/log/maillog
[root@example ~]# pkill -HUP rsyslogd
[root@example ~]# lsof /var/log/* | egrep 'COMMAND|rsyslogd'
COMMAND   PID USER   FD   TYPE DEVICE SIZE/OFF   NODE NAME
rsyslogd  847 root    4w   REG    8,4   996745 533365 /var/log/messages
[root@example ~]# reboot

The next page describes the changes in user and group management.

Next: Users and Groups