Table of Contents
Introduction
Systemd Journal Logging system made life easier while troubleshooting something in a production environment. Today in our guide, let us see how to use the journalctl command to verify logging information.
Read more logging guides:
Persistently saving journal logs
By default, the Journal logs will not be saved persistently. To save the journal logs we need to do a few configurations. Let’s start to configure the same.
create the directory to store the journal logs
# mkdir -p /var/log/journal/
Change the permission and ownership as follows
# chmod 2755 /var/log/journal/ # chown root:systemd-journal /var/log/journal
Edit the journal configuration and change the parameter “Storage=auto” to “Storage=persistent” or use sed to search and replace.
# vim /etc/systemd/journald.conf # sed -i 's/#Storage=auto/Storage=persistent/g' /etc/systemd/journald.conf
To make the changes restart the journal service.
# systemctl restart systemd-journald.service # systemctl status systemd-journald.service
Finally, verify the status of the service. By following, let’s start on how to use systemd journal logs with examples.
Similar to tail
To list end of journal log and follow mode (f) same like tail -f
# journalctl -ef
Listing latest at top
To list the newest entries at first.
# journalctl -r
Checking for specific service
To list only the logging information about sshd.service unit.
# journalctl -u httpd.service
Checking for the specific Systemd unit
To list all the logging information about sshd.service system unit.
# journalctl _SYSTEMD_UNIT=httpd.service
Listing with priority in Systemd Journal Logging
Listing logs with priority, In our example we are listing level 0 and level 3 logs. We have 8 levels of priority in the journal logging system.
# journalctl -p emerg..err
Find the list of priority level
| S:NO: | Priority Level | Priority No |
| 1. | emerg | 0 |
| 2. | alert | 1 |
| 3. | crit | 2 |
| 4. | err | 3 |
| 5. | warning | 4 |
| 6. | notice | 5 |
| 7. | info | 6 |
| 8. | debug | 7 |
Looking for SELinux context
In some case, if you are creating an interface file manually and facing some permission related issue or facing any challenge in bringing up the interface it’s good to look for SELinux context related to network as shown below.
# journalctl _SELINUX_CONTEXT=system_u:system_r:NetworkManager_t:s0
The output for the above command will too long.
Printing for specific UID
List the logs for any specific user ID, to sort the logs only for specific user this will be handy.
# journalctl _UID=998
Checking for an executable file
List all executable files, in our example, one of new account has been created
# journalctl _EXE=/usr/sbin/useradd
Output for your reference
[root@webserver ~]# journalctl _EXE=/usr/sbin/useradd -- Logs begin at Thu 2019-12-26 20:28:51 GST, end at Tue 2019-12-31 21:15:34 GST. -- Dec 31 21:15:34 webserver.linuxsysadmins.local useradd[13700]: new group: name=testuser1, GID=1002 Dec 31 21:15:34 webserver.linuxsysadmins.local useradd[13700]: new user: name=testuser1, UID=1002, GID=1002, home=/home/testuser1, shell=/bin/bash [root@webserver ~]#
Troubleshooting Kernel Devices
In our example, we are listing for attached SCSI disks.
# journalctl _KERNEL_DEVICE=+scsi:0:0:0:0
Output for reference
[root@servera ~]# journalctl _KERNEL_DEVICE=+scsi:0:0:0:0 -- Logs begin at Thu 2019-12-26 20:28:51 GST, end at Tue 2019-12-31 21:20:01 GST. -- Dec 26 20:28:51 webserver.linuxsysadmins.local kernel: sd 0:0:0:0: [sda] 41943040 512-byte logical blocks: (21.4 GB/20.0 GiB) Dec 26 20:28:51 webserver.linuxsysadmins.local kernel: sd 0:0:0:0: [sda] Write Protect is off Dec 26 20:28:51 webserver.linuxsysadmins.local kernel: sd 0:0:0:0: [sda] Attached SCSI disk
Depends on the number of disks attached in server the output will be too long.
Show for a specific boot
To list the last system boot information.
# journalctl -b -1 # journalctl _BOOT_ID=-1
Listing for a specific range
To list journal logs for a range of period.
# journalctl --since "2017-07-25 13:00:00" --until "2017-09-03 15:00:00"
Print the dmesg
To list the kernel dmesg logs from the current boot time.
# jouralctl -k
Printing logs using PID
In case, if you need to print the journal logs for a specific PID we can use below. In our example, we are looking for one of cron service’s PID.
[root@webserver ~]# journalctl _PID=12532 -- Logs begin at Thu 2019-12-26 20:28:51 GST, end at Tue 2019-12-31 20:30:01 GST. -- Dec 27 23:10:01 webserver.linuxsysadmins.local CROND[12532]: (root) CMD (/usr/lib64/sa/sa1 1 1) [root@webserver ~]#
Looking for specific machine log
This help to look for a specific machines log using _MACHINE_ID
# journalctl _MACHINE_ID=a73f9d838205494593574145673376d1
Format the journal output
Print the output in any format, for instance, we are printing in JSON-pretty format.
# journalctl _SYSTEMD_UNIT=sshd.service -o json-pretty # journalctl _SYSTEMD_UNIT=sshd.service -o short-iso
These are the supported formats.
short, short-iso, short-precise, short-montonic, verbose, json, export, json-pretty, json-sse and cat
Printing with more verbose
A short description about any issue won’t help us to find the root cause, to get more information it’s better to print with more verbosity.
# journalctl _SYSTEMD_UNIT=sshd.service -o json-sse -o verbose
Just by adding -o verbose after any options it will give you more information.
Disk usage of Journal logs
Show total disk usage of all journal files
# jouralctl --disk-usage
Verifying journal consistency
To verify the journal file consistency
# journalctl --verify
Listing message catalogue
To list the entries in the message catalogue
# journalctl --dump-catalog
One or more messages could not be forwarded to the Syslog service
running side-by-side with journald. This usually indicates that the
syslog implementation has not been able to keep up with the speed of
messages queued.
Learning more about Journal
Read more about journalctl command run below man page.
# man journalctl
Conclusion
Systemd Journal logging system is one of the easiest ways to read logs on any RHEL based Linux operating systems. Hope this guide helps you to walk through troubleshooting. Subscribe to our newsletter and stay tuned for more guides.