Urgent.News

What's breaking now, across thousands of outlets.

Tech

ClickHouse vs DuckDB: Embedded Analytical Processing vs Distributed Server Scale

ClickHouse vs DuckDB: Embedded Analytical Processing vs Distributed Server Scale These two engines solve the same mathematical problem — vectorized columnar scan over compressed data — at opposite ends of the deployment spectrum. DuckDB is an in-process library: no daemon, no ports, no network, data lives in a single file you can rsync . ClickHouse is a distributed server: a cluster coordinator,…

ClickHouse and DuckDB are two engines that solve the same problem of vectorized columnar scan over compressed data, but at different ends of the deployment spectrum. DuckDB is an in-process library that doesn't require a daemon, ports, or a network. The data lives in a single file that can be rsynced. ClickHouse, on the other hand, is a distributed server with a cluster coordinator, replicated MergeTree storage, and a query planner that distributes work to shards over gRPC.

When picking the wrong engine, you could either double your compute bill (running ClickHouse for a 50GB problem) or face a painful migration when your 400GB dataset outgrows a laptop (trying to scale DuckDB beyond its design envelope). The executive summary and decision matrix compares the two engines on various metrics:

* Warm-up time for the first query: DuckDB loads in about 40ms, while ClickHouse takes between 3 and 8 seconds (server boot + metadata load).

* Scan throughput: DuckDB can handle 1.1-1.4 GB/s/thread, totaling 18 GB/s aggregate with 32 cores. ClickHouse can manage 2.0-2.5 GB/s/thread, totaling 40 GB/s aggregate with 32 cores due to better SIMD and prefetch.

* Single-file max practical size: DuckDB's maximum is around 1-2 TB due to single-writer contention and checkpoint stalls. ClickHouse has no practical size limit, having tested sharded parts for effectively unlimited amounts of data.

* Concurrent writers: DuckDB allows one writer per process with a read-write lock. ClickHouse can handle dozens of writers per shard, with asynchronous part merges.

* Memory ceiling: DuckDB's memory limits are host RAM, causing out-of-core spilling and slowdown when triggered. ClickHouse's memory ceiling is per-query max_memory_usage limits, with spills to /tmp or disk if necessary.

* Ingest rate: ClickHouse can ingest between 2-5 million rows per second with batch INSERTs, and up to 8-15 million rows per second with async insert using 32 cores and small parts.

Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.

Read the original at dev.to →

More in Tech

How I stopped getting paged at 3am for raw SSH logs (and built a host agent for Linux)

If you run your own Linux boxes, you already know what /var/log/auth.log looks like five minutes after spinning up a fresh VPS. Nobody knows the IP. You haven't pointed a domain at it yet.

  • Vigilhold monitors Linux servers for security events without opening inbound ports.
  • Agent records and blocks malicious activities while maintaining tamper-evident audit logs.

Scoping a first version: What to build first, and what to leave out

Originally published on ayurchenko.xyz . A first version is scoped well when it proves one thing to one kind of user, and leaves out everything that does not help prove it.

  • Single claim validated by one specific user type
  • Essential features: end-to-end job and trust-building elements
  • First version excludes non-contributing features

More from Monday 16 November →