Fixing Loki Regex Pipeline Stage Failures on Nginx Logs

When Loki’s regex pipeline stage fails on Nginx logs, it can cause zero results for certain label queries and empty panels in Grafana Explore. This issue often arises when a regex is copied from a tutorial that assumes a standard Nginx log format, but the actual log format differs. The regex may compile, but it will fail to match the real log lines due to inconsistencies.

Key problems include mismatches in field order, extra fields, quoting issues, and the use of Go/RE2-specific regex features like greediness and named capture groups. Symptoms include empty field panels, no errors in logs, and later spikes in ingester memory usage and rate limit errors due to cardinality explosion. The root cause is typically a regex that doesn’t accurately reflect the actual log format.

To fix the issue, first align the regex with the actual Nginx log_format defined in /etc/nginx/nginx.conf. Use Promtail’s dry-run mode with a real log sample to validate the regex. Next, ensure the pipeline stages are ordered correctly, with labels added after the regex stage so that the extracted data can be referenced. Set the timestamp using the real $time_local field from the log format to ensure accurate time-range correlation.

Finally, move high-cardinality fields like remote_addr and user_agent to the structured_metadata stage instead of labels to prevent cardinality explosion and keep your ingesters from OOMing. This structured approach ensures that your Loki deployment remains stable and efficient.