{
  "id": 675249,
  "title": "S3 Metadata: Indexing Billions of Objects Without Falling Over",
  "url": "https://urgent.news/2026/08/12/s3-metadata-indexing-billions-of-objects-without-falling-over",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-08-12T15:29:51.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/ethan-carter/s3-metadata-indexing-billions-of-objects-without-falling-over-15ob"
  },
  "original_language": "en",
  "account": "Exceeding Metadata Limits in Large-Scale Object Storage Systems\n\nStorage systems handle two types of data when you upload an object: the actual file content and its metadata. Metadata includes details like file name, size, type, and timestamps. Every object uploaded adds roughly 200-800 bytes of metadata overhead, stored separately from the file itself.\n\nAs storage systems scale to billions of objects, listing these objects becomes increasingly slow. Here's a breakdown of the performance issues:\n\n- Less than 10,000 objects: Listing takes about 100ms\n- 1 million objects: Queries take 1-5 seconds depending on the prefix\n- 100 million objects: Latency can reach 10-60+ seconds or even timeouts\n- Over 1 billion objects: Naive listing calls become unreliable\n\nThe bottleneck lies in the metadata index. Different storage systems manage metadata differently:\n\n- AWS S3: Proprietary distributed index; LIST throttling kicks in at extreme scale\n- MinIO: Metadata in xl.meta alongside data, with in-cluster caching and background scanning; scales well with selective prefixes\n- Ceph RGW RADOS: Per-entry metadata scales reasonably but LISTs metadata linearly\n- RustFS: Embedded RocksDB + custom sharded index; built for scale with subsecond LIST on 100M+ objects\n- SeaweedFS: Filer + LevelDB/SQL backend; performance depends on the chosen database backend\n\nStrategies to handle high object counts:\n\n1. Partition your namespace: Instead of putting all data in one bucket, create separate buckets for different types of data. This isolates the metadata index for each type of data, so listing one type doesn't affect the others.\n\n2. Use structured prefixes: Within a single bucket, organize keys using a hierarchical prefix. Instead of a flat layout, use a structure like \"year/month/day/hour/object.json\". This narrows down the metadata scan range.\n\n3. Avoid full bucket scans: Don't use list_objects_v2 without prefixes on large buckets. This scans the entire metadata index. Instead, use targeted prefixes with pagination.\n\n4. Maintain an external metadata index: For systems with over 100 million objects, consider maintaining a separate metadata database alongside the object storage. Create a schema with columns like key, bucket, size, last_modified, content_type, tags, and a timestamp. Indexes on bucket and key help with common queries. Sync the database on every object operation and query metadata directly from the database.\n\n5. Choose storage built for scale: If you know you'll need to grow beyond 100 million objects, look for storage systems specifically built for that scale. Ask vendors about their p99 ListObjectsV2 latency on large buckets with selective prefixes, and whether metadata runs in-process or relies on external databases.",
  "summary": "S3 Metadata: Indexing Billions of Objects Without Falling Over If you operate S3-compatible storage at scale, slow ListObjectsV2 requests will eventually find you. Stay under your metadata threshold, and listings return fast. Cross it, and you'll watch timeouts, Slow Down errors, and after-hours alerts pile up. Every storage backend hits this wall at a different object count. Let's walk through…",
  "key_points": [],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}