Large log records in Dynatrace: limits, native support and best practices
A ticket lands: « the partner call is cut off halfway through in the logs ». You open the record and it simply stops. The payload crossed a limit somewhere. Large log records in Dynatrace raise exactly this operational question: how big can a record be, and on which path? This article starts from that decision (keep, split or shrink a record) and checks every figure against the official…
Large log records in Dynatrace can present operational challenges. The default limits dictate a maximum entry body of 10 MB for the Grail path, but only 512 kB for logs processed by the Classic pipeline. API requests are limited to 10 MB at the endpoint, while a batch exceeding 16 MB after preprocessing will result in an HTTP 413 error. Additionally, 50,000 attributes per log are permitted, with each attribute capped at 32 kB and nested objects limited to five levels deep.
After processing within OpenPipeline, the record size limit expands to 16 MB. It's crucial to note that "10 MB" is not a universal promise, as different pipelines and paths have distinct limits. For instance, records exceeding 600,000 bytes will be rejected on the Classic pipeline but can still be accommodated on the Grail path, which allows for up to 10 MB.
To test the handling of large logs, create an API token with log ingest scope, build a JSON file with a substantial content field, and send it using curl to the ingest API. Check the status code to determine if the entire record was accepted. If not, consult the documentation to identify the specific limit that was breached.
When dealing with large logs, it's essential to consider who will read them, what sensitive data they may contain, and which pipeline is most suitable for their processing. If the full content of a log is not necessary for all users, extract only the essential fields, such as error codes or order IDs, and retain the rest for investigative purposes. Additionally, ensure that any sensitive information is masked appropriately within the log using masking techniques in OpenPipeline.
Lastly, route these logs to a dedicated pipeline tailored to their specific needs. Monitor two critical processing limits: the size of extracted attributes (capped at 4,096 characters) and the character limit for DQL processor scripts (8,192 characters). By measuring the actual size of your largest events before making any changes, confirming the pipeline type for each source, and avoiding the practice of using logs to store files, you can effectively manage large log records in Dynatrace while adhering to best practices and the platform's limitations.
Rewritten by urgent.news; may contain errors — check the original.