{
  "id": 20610371,
  "title": "S3 vs. ScyllaDB for tiny telemetry: What Red Canary learned the hard way",
  "url": "https://urgent.news/2026/11/18/s3-vs-scylladb-for-tiny-telemetry-what-red-canary-learned-the-hard-way",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-11-18T15:00:00.000Z",
  "source": {
    "name": "The New Stack",
    "slug": "the-new-stack",
    "url": "https://thenewstack.io/scylladb-s3-tiny-telemetry/"
  },
  "original_language": "en",
  "account": "Red Canary faced a significant challenge in managing the cost of transmitting small amounts of telemetry data from their Linux Endpoint Detection and Response (EDR) sensors, which are installed on hundreds of thousands of endpoints. Using Amazon S3 to store this data proved to be expensive, as S3 charges a flat fee per PUT operation, which occurs for every payload upload. To address this issue, Red Canary experimented with buffering small data streams and writing them to S3 in larger chunks, but this solution introduced new problems such as strict one-megabyte-per-shard limits, delayed data freshness, and difficulties in debugging individual endpoint records.\n\nAfter exploring various options, Red Canary ultimately decided to switch their data storage from Amazon S3 and PostgreSQL to ScyllaDB Cloud, a write-optimized database. Staff software engineer Andres Medina presented their journey at the Monster Scale Summit, highlighting the benefits they gained by moving to ScyllaDB. The key advantages included reduced latency and lower costs.\n\nReplacing PostgreSQL with ScyllaDB was relatively straightforward, as each sensor event simply appends a single row, and queries for active endpoints in the web portal always operate in a time-series manner. However, they encountered challenges with tombstones and uneven data volumes. To resolve these issues, they implemented a short TTL on the table and optimized tombstone garbage collection, which significantly improved query latency.\n\nAdditionally, Red Canary had to address the challenge of varying data volume across customers, ranging from a few dozen endpoints to hundreds of thousands. To prevent hot partitions in ScyllaDB, they introduced a bucketing strategy, dividing each customer's data into smaller numbered \"buckets.\" These buckets are then distributed across shards, allowing the system to query all buckets in parallel for faster responses. This approach effectively addressed the scalability issues they faced with their existing database solution.",
  "summary": "At Red Canary, storing large amounts of small data is a big deal. Their Linux Endpoint Detection and Response (EDR) The post S3 vs. ScyllaDB for tiny telemetry: What Red Canary learned the hard way appeared first on The New Stack .",
  "key_points": [],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}