Journald logs: ship them to Dynatrace with OneAgent
You open a recent Linux node to look for an error and there is no file to read: everything goes through systemd. The journald logs stay on the machine in a binary format, and they vanish with it if nobody ships them elsewhere. This article starts from that operational decision: who collects the journal, under which rule, and how you prove the lines really reach Dynatrace. The scope is Dynatrace…
This article explores the process of collecting journald logs from Linux nodes and forwarding them to Dynatrace with the OneAgent log module. After opening a Linux node and discovering that errors do not appear in the system logs due to journald handling everything through systemd, the author delves into three approaches for collecting journald logs: third-party agents, syslog forwarding, and the OneAgent log module.
The focus here is on the OneAgent log module, which reads the journal through the systemd API without requiring manual tailing of files.
The article explains that the OneAgent log module opens the libsystemd.so library and acts as a journald client, reading entries one by one and remembering its position. It converts each entry into attributes, such as the MESSAGE field becoming the log content, the UNIT field becoming the Journald.unit attribute, and the PRIORITY field determining the severity. The author emphasizes the importance of mapping the PRIORITY field accurately, as alerts should cover priorities 0 to 3, not just 3.
To activate journald log collection, the article outlines prerequisites such as having OneAgent version 1.307 or later on a Linux host with systemd and the necessary permissions to modify tenant settings. Two settings need to be enabled: the Journald log detector in the log module feature flags, which is on by default for new tenants, and the built-in "Ingest Journald logs" rule.
If the rule needs to target specific units, the article provides instructions on how to create a rule with appropriate matchers for the desired units.
The author emphasizes the importance of checking that the logs are appearing correctly in Dynatrace, providing step-by-step instructions on how to verify the logs. It also discusses the benefits of using one rule per unit instead of a global rule, potentially reducing log volume and improving ingestion efficiency. Finally, the article suggests enabling the detector and the rule together, recording them in the runbook, and using one rule per unit once the initial test is conclusive.
Rewritten by urgent.news; may contain errors — check the original.