Urgent.News

What's breaking now, across hundreds of outlets.

Tech

Presentation: Wrangling Telemetry at Scale: A Guide to Self-Hosted Observability

Colin Douch shares battle-tested strategies for building and scaling self-hosted observability stacks. He explains how to effectively correlate metrics, structured logs, and distributed traces using open-source tooling like Prometheus, VictoriaMetrics, and Jaeger, while navigating the engineering trade-offs of avoiding SaaS vendor lock-in. By Colin Douch

Colin Douch, a Site Reliability Engineer at DuckDuckGo, recently presented on the topic of "Wrangling Telemetry at Scale: A Guide to Self-Hosted Observability" at QCon London. In his presentation, Douch shared battle-tested strategies for building and scaling self-hosted observability stacks, focusing on the effective correlation of metrics, structured logs, and distributed traces using open-source tooling such as Prometheus, VictoriaMetrics, and Jaeger.

Douch explained that while there are several escape hatches to avoid complexity, such as rewriting systems or ignoring them, these methods are often ineffective in the face of large-scale, distributed architectures. Instead, he advocated for turning to observability when faced with complex systems that cannot be easily rewritten or outsourced.

This approach, however, often leads to the "complexity demon" rearing its head, causing engineers to question whether they should run their own observability stack or continue using managed services like Datadog or Splunk.

Douch emphasized that while there may be times when self-hosting observability systems is necessary, it is generally advisable to stick with a provider to avoid the exponential increase in costs and the potential loss of features as one outgrows their current solution. He also highlighted instances where engineers may need to build custom features or address auditor questions, which can be challenging when relying on SaaS providers.

In his presentation, Douch discussed his extensive experience building self-hosted observability systems, including his work at Cloudflare and Xero, as well as his involvement in the mining sector. He shared insights on how these experiences have shaped his approach to building, maintaining, and monitoring complex observability systems.

Rewritten by urgent.news; may contain errors — check the original.

Read the original at infoq.com →

Report a problem with this story

More in Tech

Stop Pasting Production JWTs into Random Websites: The Security Guide to JSON Web Tokens

How many times this week have you opened a random online "JWT Debugger", pasted an Authorization: Bearer eyJhbGci... token from your production network tab, and hit enter?

  • Developers often paste production JWTs into random debuggers, exposing sensitive user data
  • JWTs are signed, not encrypted; use JWE for confidentiality
  • Three critical vulnerabilities: alg: none attack, asymmetric key confusion, and ignoring expiration

How to integrate a legal secretary with artificial intelligence into WordPress

Conversational artificial intelligence is opening up new possibilities for developing more interactive and useful websites.

  • Digital legal secretary integrates with WordPress for legal inquiries
  • Conversational AI interprets messages, generates responses within limits
  • Privacy compliance with GDPR and Spanish data protection regulations

MCP tool cost: 20 job Actors on Apify, 2.1K to 32.7K tokens each

Here is what an agent pays before it asks a job-search tool anything. I pinned 20 job-search Actors from the Apify Store, one at a time, on the hosted Apify MCP server and read what tools/list sends…

  • Twenty Apify MCP Actors used for job search cost analysis
  • Token counts range from 2,139 to 32,737 per tool definition
  • Author's input schema is 406 tokens, median is 3,829

Which Microcontroller to Choose for Matter and Thread (ESP32-C6, H2, and nRF52840)

Originally published on iot.gripe. Matter is the standard that aims for the same device to work with Apple Home, Google Home, Alexa and Home Assistant.

  • ESP32-C6 is best for Matter due to Wi-Fi and Thread compatibility and Espressif support
  • ESP32-H2 or nRF52840 alternatives for battery-powered devices using only Thread
  • Matter is an application layer standard, not a radio, traveling over Wi-Fi, Thread, and Internet

More from Wednesday 28 October →