Urgent.News

What's breaking now, across thousands of outlets.

Tech

The BIGINT Problem Explained

What Happens When Your Database Reaches 1 TB? How a growing events platform like Matoo helps us understand database limits, performance challenges, and the need to plan for scale. It Starts with a Simple Question Picture a Saturday morning, when thousands of people across Europe open Matoo , discover local events, and make plans for the weekend. Behind every event listing, every RSVP, and every…

When Matoo's database reaches 1 TB, it signals the need to plan for scale, despite the seemingly large number. The primary key uses BIGINT(20), which supports a vast range of positive values—up to 9.2 quintillion—making exhaustion through normal use unlikely. However, the concern shifts from the number of records to how quickly the application can retrieve the needed information as the database grows.

Indexes help locate data efficiently but add storage overhead. For instance, searching for events in Luxembourg, checking friends' attendance, and filtering by date or location could become slow without proper indexing. Meanwhile, MySQL's InnoDB storage engine reserves a significant portion of disk space for indexes and other structures, making 1 TB of data potentially more demanding than it appears.

To prepare Matoo for this scale, gradual adjustments are suggested rather than overhauling the entire system. Optimizing indexes to match real application queries can improve performance. Archiving older data into separate storage can prevent unnecessary space consumption, while partitioning large tables by date or other relevant keys allows the database to focus on relevant data sets, enhancing query speed.

Read replicas can distribute read-heavy tasks across multiple servers, easing the load on the primary database without complicating write operations. However, as databases grow, more advanced strategies like sharding—splitting data across multiple databases—might become necessary, though this introduces additional complexity. The key takeaway is that while BIGINT can support immense data growth, thoughtful database design ensures that scale remains fast, affordable, and manageable for platforms like Matoo.

Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.

Read the original at dev.to →

More in Tech

Brain Frizz, Part 1: Blocking My Day

Brain Frizz, Part 1: Blocking My Day I review a lot of pull requests (PRs) now. Some of that is the job. As a staff engineer, part of my work is helping other people get unstuck, and on a team where…

  • Engineer experiences "brain frizz" during workday
  • Divides day into fixed blocks for specific tasks
  • Tracks brain frizz levels twice daily during experiment

Una estrategia de backup NoSQL necesita automatización y ensayos de recuperación

Autora: Adriana Rafaela Laos Gonzales Equipo: Antony Piero Solorzano Zegarra y Adriana Rafaela Laos Gonzales. Una estrategia de respaldo debe responder qué información protege, cuánto podría perderse…

  • Effective NoSQL backup strategy requires automation and recovery testing
  • Backup copy of Firestore-based task app retained for at least 24 hours, daily execution
  • Backup process involves temporary token, document querying, encryption, and artifact storage

One clip-path doubled my browser cat's CPU cost in Chrome's GPU process

I made Tabloaf , a pixel cat for Chrome and Firefox. It walks across whatever page you have open, sits on pictures and headings, and chases a ball of yarn.

  • Chrome's GPU process increased from 15-17% to around 7% due to clip-path feature
  • Firefox's page process dropped from 19-20% to approximately 2% after optimizations
  • Optimized Tabloaf maintained consistent 17.5 ms frame time with no frame exceeding 20 ms

pdfjs-dist standard_fonts Missing in Next.js standalone

The short version pdfjs-dist's standard_fonts/ is read in a way Next.js file tracing cannot follow, so it can go missing from an output: "standalone" build. The effect of the omission is quiet.

  • pdfjs-dist missing standardfonts in Next.js standalone build
  • Causes text extraction from PDFs to break without exception
  • Explicitly state standardfonts in Next.js config to resolve

More from Thursday 19 November →