--- title: "Is Your Data Really Big, or Just Badly Modeled?" description: "Most big-data problems are really bad modeling or a missing index; before moving to a distributed system, ask whether the data is actually big" url: https://sade.dev/en/journal/big-data-syndrome/ lang: en author: "Muhammet Şafak" published: 2026-09-19 section: Journal tags: ["database","scaling","architecture","opinion"] --- # Is Your Data Really Big, or Just Badly Modeled? > Most big-data problems are really bad modeling or a missing index; before moving to a distributed system, ask whether the data is actually big A sentence came up in a meeting: "Our data has grown so much, we need to move to a distributed database now." Then we looked at the actual size: 12 GB. For PostgreSQL on a modern server, 12 GB isn't big — it's small. "Big data" is said most often about data that isn't big at all. And that sentence isn't a diagnosis; more often it's an excuse. ## "Big" is a feeling, not a measurement The sentence "our data is big" almost never rests on a number. It rests on a sense of slowness, a fear, a conference talk. Yet a modern PostgreSQL comfortably carries hundreds of GB on an ordinary server, and low TB if it's set up carefully. Whether data is big isn't felt; it's measured. A "big" verdict reached without measuring is the first step toward the wrong architecture. ## What most "big data" problems actually are The real cause of the slowness mistaken for big data is almost always one of three: - **Missing index.** A query doing a sequential scan makes a 5 GB table feel like 500 GB. The problem isn't the size, it's the scanned size. - **Bad modeling.** Unnormalized tables, giant JSON blobs embedded in every row, an append-heavy table that was never partitioned. The data isn't "big," it's badly placed. - **Wrong access pattern.** N+1 queries, pulling the whole row when one column would do, queries inside a loop. The application uses small data in a big way. All three are solved by **discipline**, not by size — and none of the three requires a distributed database. ## Measuring the real size To answer the "is it big" question, ask concrete questions: - How much space does the table actually take? Look with `pg_total_relation_size` — not by guessing. - Does the frequently accessed data (working set) fit in RAM? Has the cache hit ratio from `pg_stat_database` dropped below 99%? - What's the growth rate? 1 GB per month and 100 GB per month are entirely different decisions. - Which query is slow, and **why**? What does the `EXPLAIN ANALYZE` output say? If the answers to these questions reveal a real breaking point, what to do next — index, read replica, partitioning — is in [the breaking points of data-intensive systems](/en/systems/data-intensive-systems-breaking-points). This piece is about the measurement that comes before those steps. ## The distributed-system tax Moving to a distributed system before the data is genuinely big is the most expensive wrong turn. Because you pay the costs **up front** — with manual sharding or distributed NoSQL, losing the `JOIN` across tables and eventual consistency; even with distributed SQL, twice the operational load and distributed debugging — and nothing gets solved in return. A real tax for a problem that doesn't exist. The rule from [is PostgreSQL enough for everything](/en/journal/is-postgresql-enough-for-everything) applies here too: most teams use a small fraction of PostgreSQL's `jsonb`, partitioning, BRIN, and indexing capabilities and then conclude it "isn't enough." ## If it really is big Let's be honest: real big data exists. Throughput that sustainably exceeds the write capacity of a single large machine, or a genuinely multi-TB hot working set — these are real limits. A system that gets there has also reached a scale that can finance that complexity; at that point a distributed architecture isn't a debt, it's an earned cost. The point isn't to be against distributed systems; it's to avoid moving to them before they're truly needed — that is, before measuring. --- When you hear "our data is big," the only thing to ask for is a number. If there's no number, what you have isn't big data, it's unmeasured data. Real size is measured; felt size is, most of the time, a missing index.