A single page of an admin panel was firing 1,400 queries. The cause wasn’t visible on screen: inside a loop, the innocent-looking $order->customer->name call fired a separate query for every order. The ORM hadn’t actually hidden anything — it had only let the developer stop thinking about which query was running.
Is the ORM making us lazy, or protecting us? The answer is both — and the team decides which one dominates.
\nThe ORM has real upsides, and they shouldn’t be dismissed:
\nUser::create($data) is shorter and clearer than a hand-written INSERT.The vast majority of projects are CRUD at their core, and for that work the ORM is the right default. I have no objection to that.
\nThe danger isn’t in what the ORM does, but in what it hides. The ORM keeps SQL out of sight; and when the developer can’t see the SQL, they stop thinking about which query is running.
\nThe result is familiar: the N+1 from the opening of this piece; pulling a whole row into memory when a single column would do; forgetting a where and unknowingly loading an entire table. The ORM doesn’t “do” any of these; they’re all done by the developer who never looks at what the ORM produces.
In the data-intensive systems piece, the first row of the false-breaking-points table is exactly this: what looks like “the database is slow” is, most of the time, the application firing 200 queries.
\nThe ORM shines at transactional CRUD: create a record, update it, read it with its relationships. Where it struggles is just as clear:
\nGROUP BY.UPDATE instead of walking millions of rows one model at a time.For this kind of work, insisting on the ORM produces code that is both slower and less readable. Plain SQL is both faster and clearer here. Picking the right indexes is part of this work too — the index management note covers that side.
\nThe two aren’t rivals. The right use is roughly this:
\nThe mark of seniority isn’t knowing the ORM; it’s knowing where it ends. A team that forces the ORM onto everything turns it into a performance trap; a team that never uses the ORM and writes everything by hand gives up safety and readability for nothing.
\nA common mistake: “plain SQL = unsafe.” What provides safety isn’t the ORM; it’s parameter binding. A plain query written with bound parameters is as closed to injection as the ORM:
\nDB::select('SELECT id, total FROM orders WHERE status = ?', ['pending']);What’s dangerous isn’t plain SQL; it’s embedding user input into the query through string concatenation — and you can do that inside the ORM’s whereRaw too. The habit protects you, not the tool.
The ORM can be a laziness machine or a safety net — the difference is made by how the team looks at it. Use it, but never stop seeing what it produces.
\nA good developer writes with the ORM; they also know when to put it down.
", "content_text": "A single page of an admin panel was firing 1,400 queries. The cause wasn't visible on screen: inside a loop, the innocent-looking `$order->customer->name` call fired a separate query for every order. The ORM hadn't actually hidden anything — it had only let the developer stop thinking about which query was running.\n\nIs the ORM making us lazy, or protecting us? The answer is both — and the team decides which one dominates.\n\n## What an ORM buys you\n\nThe ORM has real upsides, and they shouldn't be dismissed:\n\n- **Safety.** Queries are parameterized by default; under ordinary use, the SQL injection surface is closed.\n- **Readability.** For CRUD operations, `User::create($data)` is shorter and clearer than a hand-written `INSERT`.\n- **Less repetition.** Relationships, migrations, model events — all in one place.\n\nThe vast majority of projects are CRUD at their core, and for that work the ORM is the right default. I have no objection to that.\n\n## Where the ORM makes you lazy\n\nThe danger isn't in what the ORM does, but in what it **hides**. The ORM keeps SQL out of sight; and when the developer can't see the SQL, they stop thinking about which query is running.\n\nThe result is familiar: the N+1 from the opening of this piece; pulling a whole row into memory when a single column would do; forgetting a `where` and unknowingly loading an entire table. The ORM doesn't \"do\" any of these; they're all done by the developer who never looks at what the ORM produces.\n\nIn the [data-intensive systems](/en/systems/data-intensive-systems-breaking-points) piece, the first row of the false-breaking-points table is exactly this: what looks like \"the database is slow\" is, most of the time, the application firing 200 queries.\n\n## Where the ORM ends\n\nThe ORM shines at transactional CRUD: create a record, update it, read it with its relationships. Where it struggles is just as clear:\n\n- Multi-table, multi-step reports.\n- Aggregations, window functions, complex conditions on top of `GROUP BY`.\n- Bulk operations — a single `UPDATE` instead of walking millions of rows one model at a time.\n\nFor this kind of work, insisting on the ORM produces code that is both slower and less readable. Plain SQL is both faster and clearer here. Picking the right indexes is part of this work too — the [index management](/en/notes/index-management) note covers that side.\n\n## The call: ORM as default, SQL as a tool\n\nThe two aren't rivals. The right use is roughly this:\n\n- 90% of the work — CRUD, simple lists — with the ORM. Fast, safe, readable.\n- The remaining 10% — heavy reporting, aggregation, bulk operations — with the query builder or plain SQL.\n\nThe mark of seniority isn't knowing the ORM; it's knowing **where it ends**. A team that forces the ORM onto everything turns it into a performance trap; a team that never uses the ORM and writes everything by hand gives up safety and readability for nothing.\n\n## Both can be written safely\n\nA common mistake: \"plain SQL = unsafe.\" What provides safety isn't the ORM; it's **parameter binding**. A plain query written with bound parameters is as closed to injection as the ORM:\n\n```php\nDB::select('SELECT id, total FROM orders WHERE status = ?', ['pending']);\n```\n\nWhat's dangerous isn't plain SQL; it's embedding user input into the query through string concatenation — and you can do that inside the ORM's `whereRaw` too. The habit protects you, not the tool.\n\n---\n\nThe ORM can be a laziness machine or a safety net — the difference is made by how the team looks at it. Use it, but never stop seeing what it produces.\n\nA good developer writes with the ORM; they also know when to put it down.", "date_published": "2026-09-12T00:00:00.000Z", "tags": [ "database", "orm", "sql", "opinion", "Journal" ], "authors": [ { "name": "Muhammet Şafak", "url": "https://www.muhammetsafak.com.tr" } ] }, { "id": "https://sade.dev/en/notes/index-management/", "url": "https://sade.dev/en/notes/index-management/", "title": "Indexes: Too Few Slow Reads, Too Many Kill Writes", "summary": "An index is not free read speed; it is read speed paid for with writes. Every INSERT writes an entry into every index on the table, and an UPDATE does the same the moment it touches an indexed column. A HOT update and a partial index are the exceptions, not the default. So add an index because a real query plan asked for it, and drop one because pg_stat_user_indexes shows idx_scan = 0.", "content_html": "I’ve seen two opposite teams. One had put no indexes on the table at all — every query a sequential scan, every list page taking seconds. The other had done the exact opposite: “just in case,” it had created an index on every column, and now every INSERT crawled along.
Both are two ends of the same fallacy: thinking an index is a free source of speed.
\nAn index speeds up reads, because instead of scanning the whole table the database looks at an ordered structure. But that ordered structure doesn’t stay current on its own: an INSERT writes an entry into every index on the table, and an UPDATE does the same as soon as it touches an indexed column. There are exceptions — a HOT update that changes no indexed column leaves the indexes alone, a partial index is skipped for rows outside its WHERE clause, and a DELETE leaves its index entries behind for VACUUM to clean up later — but they are exceptions, not the default.
So every index is a trade: you buy read speed with write cost. Inserting a single row into a table with five indexes means updating six structures at once. An index isn’t “free read speed,” it’s “read speed paid for with writes.”
\nFind a missing index from the query plan, not from a hunch:
\nEXPLAIN ANALYZE\nSELECT * FROM orders WHERE status = 'pending';If you see a Seq Scan on a large table in the output and the query returns a small fraction of the rows, an index is probably missing. Scan the pg_stat_user_tables table for large tables with a high seq_scan count — those are your candidates.
Add the index for a real, slow query. An index added “in case we need it later” is the database-layer version of speculative generality.
\nIn the other direction, indexes that are never used make every write more expensive for nothing. Find those by measurement too:
\nSELECT indexrelname, idx_scan\nFROM pg_stat_user_indexes\nWHERE idx_scan = 0;An index with idx_scan = 0 means it’s speeding up no reads at all but adding cost to every write. (Keep the indexes for primary keys and unique constraints, of course.) This is the first cleanup listed at the write breaking point of data-intensive systems.
An index isn’t just “present or absent”; choosing the right type and shape matters:
\n(a, b) index serves equality on a + a range query on b; a lookup on b alone can still use the index, but it doesn’t narrow the portion that gets scanned — before PostgreSQL 18 the whole index is scanned, and from 18 on the skip scan optimization narrows it down. Put the column filtered by equality first, the one filtered by range last.CREATE INDEX ... WHERE status = 'active'. A smaller index, cheaper maintenance.INCLUDE you can add frequently read columns to the index and keep the database from going to the table at all.jsonb and full text; small, cheap BRIN for purely ordered, append-heavy data.An index on (a) is redundant if you already have an (a, b) index — the composite index serves queries starting with a too. Clean up overlaps like these periodically; each one is a silent write tax.
One rule: don’t add an index by guesswork, and don’t drop one by guesswork either. An index is added because a real query plan asks for it; an index is dropped because the statistics show no one is using it. The balance between too few and too many is struck with EXPLAIN ANALYZE and pg_stat_user_indexes, not with gut feeling.
Index management starts with shedding the belief that “more indexes is better.” Every index is a read gain and a write cost; good management is keeping the two in balance by measuring them.
\nA missing index slows the query; too many slow the whole table. Measurement guards against both.
", "content_text": "I've seen two opposite teams. One had put no indexes on the table at all — every query a sequential scan, every list page taking seconds. The other had done the exact opposite: \"just in case,\" it had created an index on every column, and now every `INSERT` crawled along.\n\nBoth are two ends of the same fallacy: thinking an index is a free source of speed.\n\n## An index is not free speed\n\nAn index speeds up reads, because instead of scanning the whole table the database looks at an ordered structure. But that ordered structure doesn't stay current on its own: an `INSERT` writes an entry into **every index** on the table, and an `UPDATE` does the same as soon as it touches an indexed column. There are exceptions — a HOT update that changes no indexed column leaves the indexes alone, a partial index is skipped for rows outside its `WHERE` clause, and a `DELETE` leaves its index entries behind for `VACUUM` to clean up later — but they are exceptions, not the default.\n\nSo every index is a trade: you buy read speed with write cost. Inserting a single row into a table with five indexes means updating six structures at once. An index isn't \"free read speed,\" it's \"read speed paid for with writes.\"\n\n## A missing index: measure, don't guess\n\nFind a missing index from the query plan, not from a hunch:\n\n```sql\nEXPLAIN ANALYZE\nSELECT * FROM orders WHERE status = 'pending';\n```\n\nIf you see a `Seq Scan` on a large table in the output and the query returns a small fraction of the rows, an index is probably missing. Scan the `pg_stat_user_tables` table for large tables with a high `seq_scan` count — those are your candidates.\n\nAdd the index for a real, slow query. An index added \"in case we need it later\" is the database-layer version of [speculative generality](/en/journal/the-cost-of-just-in-case-code).\n\n## Too many indexes: find the unused ones\n\nIn the other direction, indexes that are never used make every write more expensive for nothing. Find those by measurement too:\n\n```sql\nSELECT indexrelname, idx_scan\nFROM pg_stat_user_indexes\nWHERE idx_scan = 0;\n```\n\nAn index with `idx_scan = 0` means it's speeding up no reads at all but adding cost to every write. (Keep the indexes for primary keys and unique constraints, of course.) This is the first cleanup listed at the write breaking point of [data-intensive systems](/en/systems/data-intensive-systems-breaking-points).\n\n## How do you choose the right index?\n\nAn index isn't just \"present or absent\"; choosing the right type and shape matters:\n\n- **Column order in a composite index.** An `(a, b)` index serves equality on `a` + a range query on `b`; a lookup on `b` alone can still use the index, but it doesn't narrow the portion that gets scanned — before PostgreSQL 18 the whole index is scanned, and from 18 on the skip scan optimization narrows it down. Put the column filtered by equality first, the one filtered by range last.\n- **Partial index.** If the query always looks at the same subset, limit the index to that subset too: `CREATE INDEX ... WHERE status = 'active'`. A smaller index, cheaper maintenance.\n- **Covering index.** With `INCLUDE` you can add frequently read columns to the index and keep the database from going to the table at all.\n- **Non-B-tree types.** B-tree for equality/ordering; GIN for `jsonb` and full text; small, cheap BRIN for purely ordered, append-heavy data.\n\n## Duplicate and overlapping indexes\n\nAn index on `(a)` is redundant if you already have an `(a, b)` index — the composite index serves queries starting with `a` too. Clean up overlaps like these periodically; each one is a silent write tax.\n\n## Balance is set with the query plan\n\nOne rule: don't add an index by guesswork, and don't drop one by guesswork either. An index is added because a real query plan asks for it; an index is dropped because the statistics show no one is using it. The balance between too few and too many is struck with `EXPLAIN ANALYZE` and `pg_stat_user_indexes`, not with gut feeling.\n\n---\n\nIndex management starts with shedding the belief that \"more indexes is better.\" Every index is a read gain and a write cost; good management is keeping the two in balance by measuring them.\n\nA missing index slows the query; too many slow the whole table. Measurement guards against both.", "date_published": "2026-09-05T00:00:00.000Z", "tags": [ "postgresql", "database", "performance", "indexing", "Note" ], "authors": [ { "name": "Muhammet Şafak", "url": "https://www.muhammetsafak.com.tr" } ] }, { "id": "https://sade.dev/en/notes/zero-downtime-database-migrations/", "url": "https://sade.dev/en/notes/zero-downtime-database-migrations/", "title": "Changing Schema in Production With Zero Downtime", "summary": "A schema change is not what is dangerous; doing it in one step is. Only operations taking an ACCESS EXCLUSIVE lock queue up reads as well as writes, and on a fifty-million-row table that queue is the outage. Break the change into expand-contract steps: add the constraint NOT VALID and run VALIDATE separately, build indexes CONCURRENTLY, and give the migration session a short lock_timeout.", "content_html": "A deploy ran a single ALTER TABLE on a table with 50 million rows and locked it for four minutes. For four minutes, every request that touched that table waited; the site was effectively down. The migration itself was correct — the problem was doing it in one step.
The dangerous thing is not the schema change. The dangerous thing is making a schema change all at once, without thinking about backward compatibility.
\nNot every schema change costs the same. In modern PostgreSQL, adding a column with a constant DEFAULT is a metadata operation — it’s fast. The real danger is in operations that lock the table for a long time:
CREATE INDEX — without CONCURRENTLY, it closes the table to writes.ALTER COLUMN ... TYPE changes that rewrite the table.NOT NULL, CHECK, or foreign key additions that scan the whole table.These operations take a strong lock; the ones that take it in ACCESS EXCLUSIVE mode — a table rewrite, or adding NOT NULL/CHECK — queue up every query touching the table, reads included. The rest block writes only. If the table is large, the queue grows.
The solution is not to avoid migrations; it’s to break every dangerous migration into small steps, each of which is safe and backward-compatible on its own. This is called the expand-contract pattern.
\nSuppose you want to rename a column. A single-step RENAME COLUMN instantly breaks running code that reads the old column. Instead, three deploys:
Each step works with both the code from the previous release and the new code. At no moment is the running code incompatible with the schema.
\nThe zero-downtime versions of common changes:
\nA new NOT NULL column. Adding NOT NULL in one step scans the table. Split it:
-- 1. Add it as nullable first\nALTER TABLE orders ADD COLUMN status text;\n\n-- 2. Backfill existing rows in batches (on the application side)\n\n-- 3. Add the constraint NOT VALID first, then validate in a separate step\nALTER TABLE orders ADD CONSTRAINT orders_status_not_null\n CHECK (status IS NOT NULL) NOT VALID;\nALTER TABLE orders VALIDATE CONSTRAINT orders_status_not_null;A constraint added with NOT VALID applies immediately to new rows but does not scan existing ones; VALIDATE then scans the table with only a SHARE UPDATE EXCLUSIVE lock — it doesn’t block writes.
An index. Always CONCURRENTLY:
CREATE INDEX CONCURRENTLY idx_orders_status ON orders (status);A foreign key. The same two-step pattern: first ADD CONSTRAINT ... NOT VALID, then VALIDATE CONSTRAINT.
lock_timeout. While a migration waits for a lock behind a long-running query, it blocks everything behind it too. To prevent this, give the migration session a short lock_timeout — if the lock can’t be taken immediately, let the migration fail instead of waiting, and retry it yourself.
The essence of expand-contract is one rule: at every intermediate step, both the old code still running and the new code must be able to work with the current schema.
\nThe database schema is what I called a “one-way door” in the cost of just-in-case code — rolling it back is expensive. So design the change not as one big, irreversible step, but as small steps that can each be rolled back individually.
\nZero-downtime migration is not a tool but a discipline: breaking every schema change into steps small enough that the running code never notices.
\nThe dangerous thing is not the change itself, but doing it in one breath.
", "content_text": "A deploy ran a single `ALTER TABLE` on a table with 50 million rows and locked it for four minutes. For four minutes, every request that touched that table waited; the site was effectively down. The migration itself was correct — the problem was doing it in one step.\n\nThe dangerous thing is not the schema change. The dangerous thing is making a schema change all at once, without thinking about backward compatibility.\n\n## Who takes the lock?\n\nNot every schema change costs the same. In modern PostgreSQL, adding a column with a constant `DEFAULT` is a metadata operation — it's fast. The real danger is in operations that lock the table for a long time:\n\n- `CREATE INDEX` — without `CONCURRENTLY`, it closes the table to writes.\n- `ALTER COLUMN ... TYPE` changes that rewrite the table.\n- `NOT NULL`, `CHECK`, or foreign key additions that scan the whole table.\n\nThese operations take a strong lock; the ones that take it in `ACCESS EXCLUSIVE` mode — a table rewrite, or adding `NOT NULL`/`CHECK` — queue up every query touching the table, reads included. The rest block writes only. If the table is large, the queue grows.\n\n## The dangerous part is the single step\n\nThe solution is not to avoid migrations; it's to break every dangerous migration into small steps, each of which is safe and backward-compatible on its own. This is called the **expand-contract** pattern.\n\nSuppose you want to rename a column. A single-step `RENAME COLUMN` instantly breaks running code that reads the old column. Instead, three deploys:\n\n1. **Expand.** Add the new column. Have the code write to both old and new, still reading from old.\n2. **Migrate.** Move the old data into the new column in batches. Now have the code read from new.\n3. **Contract.** Drop the old column.\n\nEach step works with both the code from the previous release and the new code. At no moment is the running code incompatible with the schema.\n\n## Safe recipes\n\nThe zero-downtime versions of common changes:\n\n**A new `NOT NULL` column.** Adding `NOT NULL` in one step scans the table. Split it:\n\n```sql\n-- 1. Add it as nullable first\nALTER TABLE orders ADD COLUMN status text;\n\n-- 2. Backfill existing rows in batches (on the application side)\n\n-- 3. Add the constraint NOT VALID first, then validate in a separate step\nALTER TABLE orders ADD CONSTRAINT orders_status_not_null\n CHECK (status IS NOT NULL) NOT VALID;\nALTER TABLE orders VALIDATE CONSTRAINT orders_status_not_null;\n```\n\nA constraint added with `NOT VALID` applies immediately to new rows but does not scan existing ones; `VALIDATE` then scans the table with only a `SHARE UPDATE EXCLUSIVE` lock — it doesn't block writes.\n\n**An index.** Always `CONCURRENTLY`:\n\n```sql\nCREATE INDEX CONCURRENTLY idx_orders_status ON orders (status);\n```\n\n**A foreign key.** The same two-step pattern: first `ADD CONSTRAINT ... NOT VALID`, then `VALIDATE CONSTRAINT`.\n\n**`lock_timeout`.** While a migration waits for a lock behind a long-running query, it blocks everything behind it too. To prevent this, give the migration session a short `lock_timeout` — if the lock can't be taken immediately, let the migration fail instead of waiting, and retry it yourself.\n\n## Code and schema must be compatible together\n\nThe essence of expand-contract is one rule: at every intermediate step, both the **old code still running** and the **new code** must be able to work with the current schema.\n\nThe database schema is what I called a \"one-way door\" in [the cost of just-in-case code](/en/journal/the-cost-of-just-in-case-code) — rolling it back is expensive. So design the change not as one big, irreversible step, but as small steps that can each be rolled back individually.\n\n---\n\nZero-downtime migration is not a tool but a discipline: breaking every schema change into steps small enough that the running code never notices.\n\nThe dangerous thing is not the change itself, but doing it in one breath.", "date_published": "2026-08-29T00:00:00.000Z", "tags": [ "postgresql", "database", "migrations", "production", "Note" ], "authors": [ { "name": "Muhammet Şafak", "url": "https://www.muhammetsafak.com.tr" } ] }, { "id": "https://sade.dev/en/notes/read-write-splitting/", "url": "https://sade.dev/en/notes/read-write-splitting/", "title": "Read/Write Splitting: Separating Read and Write Load", "summary": "A read replica scales reads and adds nothing to write capacity, so the trade is not free: the price is replication lag. Laravel's sticky option repairs read-after-write only inside a single request; across two requests the user still sees the profile they just updated as stale. Do index discipline first, because a replica is often an expensive way to buy what one CREATE INDEX would have solved.", "content_html": "The primary database’s CPU was constantly maxed out. The interesting part: the write rate was low. Almost all of the load was reads — report pages, listing endpoints, search. A single primary was trying to carry a pile of reads that never needed it in the first place.
\nSeparating read and write load — read/write splitting — is the known fix for this picture. But applied at the wrong time, or without awareness of the right traps, it brings more problems than it solves.
\nHow read-heavy the traffic is depends on the workload: in measurements of the standard OLTP benchmarks, TPC-E runs 90.69% reads while TPC-C stays at 65.71% — so measure your own ratio instead of assuming it. Every order is written once but read dozens of times: in the list, in the detail view, in a report, on a dashboard. This asymmetry is what makes read/write splitting appealing — because the side you need to scale is obvious.
\nStop before adding a replica. A full primary doesn’t always mean “out of capacity.” Often a single missing index makes the primary look many times busier than it is.
\nAdding a replica — a new server, replication setup, lag monitoring — can amount to expensively buying your way out of a problem a single CREATE INDEX would have solved. The order in the breaking points of data-intensive systems is clear: index discipline first, then replicas. Don’t skip that order.
Measure your queries with EXPLAIN ANALYZE. If the primary is genuinely saturating under correctly indexed queries — that’s when you reach for a replica.
Let’s be clear: a read replica adds nothing to your write capacity. The same writes are replayed on every replica. A replica solves a read-load problem; if you have a write-load problem, a replica is the wrong tool.
\nLaravel supports read/write connection splitting natively:
\n// config/database.php\n'pgsql' => [\n 'driver' => 'pgsql',\n 'read' => ['host' => ['10.0.0.2']], // replica\n 'write' => ['host' => ['10.0.0.1']], // primary\n 'sticky' => true,\n // ...shared settings\n],SELECTs go to the replica, INSERT/UPDATE/DELETEs go to the primary. The replica gets its own connection pool — separate from the primary’s; if you use pgBouncer, they are two distinct pools.
The replica trails the primary by a few milliseconds — a few seconds under load. This delay is the real cost of read/write splitting, and its name is the read-after-write problem.
\nsticky => true partly addresses it: if you wrote within a request, the subsequent reads in that same request go to the primary. But sticky only works within the boundary of a single request.
Outside that boundary it’s still open: a user updates their profile (request 1, written to the primary), moves to the next page (request 2, read from the replica), and sees their old profile, not yet replayed on the replica. The user sees their own data as stale. This looks like a bug, but it’s actually a tradeoff the architecture accepts — one that has to be accepted deliberately.
\nSetting up read/write splitting requires every read to answer one question: can this query read stale data?
\nThis classification is now part of the architecture, and it needs to be documented. A new developer must have somewhere to look for “where should this query read from” — otherwise the classification quietly rots.
\nRead/write splitting is the right move when these three conditions hold together:
\nIf any one of these is missing, deferring the replica is cheaper.
\nRead/write splitting is a cheaper scaling move than growing vertically — but it isn’t free. The price is replication lag, and the cost of ignoring it is paid by showing users their own data as stale.
\nBefore splitting reads, know which reads can tolerate staleness.
", "content_text": "The primary database's CPU was constantly maxed out. The interesting part: the write rate was low. Almost all of the load was reads — report pages, listing endpoints, search. A single primary was trying to carry a pile of reads that never needed it in the first place.\n\nSeparating read and write load — read/write splitting — is the known fix for this picture. But applied at the wrong time, or without awareness of the right traps, it brings more problems than it solves.\n\n## Most load is read-heavy\n\nHow read-heavy the traffic is depends on the workload: in measurements of the standard OLTP benchmarks, TPC-E runs 90.69% reads while TPC-C stays at 65.71% — so measure your own ratio instead of assuming it. Every order is written once but read dozens of times: in the list, in the detail view, in a report, on a dashboard. This asymmetry is what makes read/write splitting appealing — because the side you need to scale is obvious.\n\n## First: is this really a capacity problem?\n\nStop before adding a replica. A full primary doesn't always mean \"out of capacity.\" Often a single missing index makes the primary look many times busier than it is.\n\nAdding a replica — a new server, replication setup, lag monitoring — can amount to expensively buying your way out of a problem a single `CREATE INDEX` would have solved. The order in [the breaking points of data-intensive systems](/en/systems/data-intensive-systems-breaking-points) is clear: index discipline first, then replicas. Don't skip that order.\n\nMeasure your queries with `EXPLAIN ANALYZE`. If the primary is genuinely saturating under correctly indexed queries — that's when you reach for a replica.\n\n## A replica scales reads, not writes\n\nLet's be clear: a read replica adds **nothing** to your write capacity. The same writes are replayed on every replica. A replica solves a read-load problem; if you have a write-load problem, a replica is the wrong tool.\n\n## Setting it up in Laravel\n\nLaravel supports read/write connection splitting natively:\n\n```php\n// config/database.php\n'pgsql' => [\n 'driver' => 'pgsql',\n 'read' => ['host' => ['10.0.0.2']], // replica\n 'write' => ['host' => ['10.0.0.1']], // primary\n 'sticky' => true,\n // ...shared settings\n],\n```\n\n`SELECT`s go to the replica, `INSERT/UPDATE/DELETE`s go to the primary. The replica gets its own connection pool — separate from the primary's; if you use [pgBouncer](/en/notes/pgbouncer-auth-query), they are two distinct pools.\n\n## Replication lag: the real bill\n\nThe replica trails the primary by a few milliseconds — a few seconds under load. This delay is the real cost of read/write splitting, and its name is the **read-after-write** problem.\n\n`sticky => true` partly addresses it: if you wrote within a request, the subsequent reads in that same request go to the primary. But `sticky` only works within the boundary of **a single request**.\n\nOutside that boundary it's still open: a user updates their profile (request 1, written to the primary), moves to the next page (request 2, read from the replica), and sees their old profile, not yet replayed on the replica. The user sees their own data as stale. This looks like a bug, but it's actually a tradeoff the architecture accepts — one that has to be accepted deliberately.\n\n## The query classification discipline\n\nSetting up read/write splitting requires every read to answer one question: **can this query read stale data?**\n\n- **Can read from a replica:** lists, reports, search results, dashboards. A few seconds of delay is irrelevant.\n- **Must read from the primary:** account balance, stock count, authorization checks, any read a write decision depends on.\n\nThis classification is now part of the architecture, and it needs to be documented. A new developer must have somewhere to look for \"where should this query read from\" — otherwise the classification quietly rots.\n\n## When do you actually need it?\n\nRead/write splitting is the right move when these three conditions hold together:\n\n1. Index discipline is complete; queries are correctly indexed and the primary is still saturating.\n2. The load is measurably read-heavy.\n3. There's the discipline to do and document the stale-read classification.\n\nIf any one of these is missing, deferring the replica is cheaper.\n\n---\n\nRead/write splitting is a cheaper scaling move than growing vertically — but it isn't free. The price is replication lag, and the cost of ignoring it is paid by showing users their own data as stale.\n\nBefore splitting reads, know which reads can tolerate staleness.", "date_published": "2026-08-22T00:00:00.000Z", "tags": [ "postgresql", "database", "scaling", "performance", "Note" ], "authors": [ { "name": "Muhammet Şafak", "url": "https://www.muhammetsafak.com.tr" } ] }, { "id": "https://sade.dev/en/notes/the-right-cache-strategy/", "url": "https://sade.dev/en/notes/the-right-cache-strategy/", "title": "Before You Reach for Redis: The Right Cache Strategy", "summary": "A cache is never a faster database; it is speed bought with accepted staleness, so the first question is not technical: how stale may this data be? TTL is the boring default, because it does not require knowing every write path and it does not break silently when one is forgotten. Explicit invalidation is exact only if you catch every path. And a cache reduces read load, never write load.", "content_html": "A team added a Redis cache because “the site is slow.” A week later the support queue filled up: “I updated the price but the old one still shows.” The cache had fixed the slowness — and quietly created a new problem.
\nAdding a cache is not a free performance win. It trades freshness for speed; and if you didn’t make that trade deliberately, the bill comes back as a bug.
\nThe moment you cache a value, you’ve accepted this: someone, for a while, may see stale data. A cache is never a “faster database”; it’s “speed in exchange for accepted staleness.”
\nSo the first question isn’t technical: how stale can this data be? Data whose answer is “not at all” — an account balance, a stock count — should be thought through twice before it’s cached.
\nBefore you turn on Redis:
\nCREATE INDEX would solve is moving the problem, not solving it — this is exactly the wrong breaking point in data-intensive systems.Cache invalidation gets cited as one of the two hard problems in software — and that’s no exaggeration. Putting the cache in is easy; clearing it at the right moment is hard.
\nBecause a cached value must be cleared on every write path that affects it. A product price changes not only from the “edit product” screen; it also changes from a bulk price update, a discount job, an admin script. Finding all of those paths and clearing the cache — that’s the hard part. Miss one, and you get the support ticket from the top of this post.
\nIn practice there are two roads:
\nMy default is TTL. It’s boring, predictable, and it doesn’t break silently because of a forgotten write path. I move to explicit invalidation only when staleness is genuinely unacceptable and the number of write paths is limited and known. Most of the time the two are used together: a short TTL as a safety net, explicit invalidation for speed.
\nA common mistake: trying to rescue a system under write load by adding a cache. A cache reduces read load; it has no effect on write load — if anything, invalidation itself is extra write work. If your problem is on the write side, a cache is the wrong tool.
\nA cache is not the cure for a slow system; it’s the relief of a measured read bottleneck, bought with a deliberate consistency concession. A cache added without an invalidation plan takes back more as debt than it speeds up.
\nBefore you add a cache, ask: how old can this data be — and who’s going to clear it?
", "content_text": "A team added a Redis cache because \"the site is slow.\" A week later the support queue filled up: \"I updated the price but the old one still shows.\" The cache had fixed the slowness — and quietly created a new problem.\n\nAdding a cache is not a free performance win. It trades freshness for speed; and if you didn't make that trade deliberately, the bill comes back as a bug.\n\n## A cache is a consistency concession\n\nThe moment you cache a value, you've accepted this: someone, for a while, may see stale data. A cache is never a \"faster database\"; it's \"speed in exchange for accepted staleness.\"\n\nSo the first question isn't technical: **how stale can this data be?** Data whose answer is \"not at all\" — an account balance, a stock count — should be thought through twice before it's cached.\n\n## Questions to ask first\n\nBefore you turn on Redis:\n\n- **Is this really a read bottleneck?** \"The site is slow\" is a hypothesis, not a measurement. Adding a cache without seeing the source of the slowness is covering up something you don't understand.\n- **Is the slowness actually a missing index?** A cache papers over a bad query but doesn't fix it. Hiding behind a cache what a `CREATE INDEX` would solve is moving the problem, not solving it — this is exactly the wrong breaking point in [data-intensive systems](/en/systems/data-intensive-systems-breaking-points).\n- **How stale can this data stay?** The answer determines which strategy you pick.\n\n## Why is invalidation hard?\n\nCache invalidation gets cited as one of the two hard problems in software — and that's no exaggeration. Putting the cache in is easy; **clearing** it at the right moment is hard.\n\nBecause a cached value must be cleared on every write path that affects it. A product price changes not only from the \"edit product\" screen; it also changes from a bulk price update, a discount job, an admin script. Finding **all** of those paths and clearing the cache — that's the hard part. Miss one, and you get the support ticket from the top of this post.\n\n## Two strategies\n\nIn practice there are two roads:\n\n- **TTL-based.** You assign the value a lifetime — 60 seconds, 5 minutes. When it expires, the cache refreshes itself. Simple, sturdy, and it doesn't require knowing every write path. In return: you accept staleness up to the TTL.\n- **Explicit invalidation.** You delete the cache by hand when the data changes. Exact and fresh — but only if you catch every write path completely.\n\nMy default is TTL. It's boring, predictable, and it doesn't break silently because of a forgotten write path. I move to explicit invalidation only when staleness is genuinely unacceptable and the number of write paths is limited and known. Most of the time the two are used together: a short TTL as a safety net, explicit invalidation for speed.\n\n## A cache doesn't reduce writes\n\nA common mistake: trying to rescue a system under write load by adding a cache. A cache reduces **read** load; it has no effect on write load — if anything, invalidation itself is extra write work. If your problem is on the write side, a cache is the wrong tool.\n\n## Practical patterns\n\n- **Cache-aside.** Read: check the cache first, and on a miss fetch the value from the database and write it to the cache. The most common and most understandable pattern.\n- **Stampede protection.** The instant a popular key's TTL expires, hundreds of requests hit the database at once. On a cache miss, use a short lock so only one request goes to the database.\n- **Key discipline.** Namespace your cache keys — the pattern in the [shared Redis namespace isolation](/en/notes/shared-redis-namespace-isolation) note applies to cache keys too.\n\n---\n\nA cache is not the cure for a slow system; it's the relief of a measured read bottleneck, bought with a deliberate consistency concession. A cache added without an invalidation plan takes back more as debt than it speeds up.\n\nBefore you add a cache, ask: how old can this data be — and who's going to clear it?", "date_published": "2026-08-15T00:00:00.000Z", "tags": [ "caching", "redis", "performance", "production", "Note" ], "authors": [ { "name": "Muhammet Şafak", "url": "https://www.muhammetsafak.com.tr" } ] }, { "id": "https://sade.dev/en/journal/the-nosql-trap/", "url": "https://sade.dev/en/journal/the-nosql-trap/", "title": "The NoSQL Trap: Starting Because \"It Has No Schema\"", "summary": "There is no such thing as schemaless data, only the question of where the shape is defined and who enforces it. Choosing a document database for the comfort of not wanting a schema does not delete the schema; it moves it into every read, every write and the team's heads, where nothing enforces it. NoSQL earns its place on horizontal write scale or genuinely heterogeneous documents, not on schema avoidance.", "content_html": "I once saw a project that had picked MongoDB on the logic of “no schema, we’ll move fast” — two years later. Every read was defensive: data?.user?.address?.city ?? null. The reason was simple — no two documents had exactly the same shape. The schema hadn’t disappeared; it had just left the database and scattered across every bit of code that reads a document.
There is no such thing as schemaless data. Data always has a shape: a user’s email, the line items of an order, the date on an invoice. The question is not whether there is a shape; it is where that shape is defined.
\nThere are two options. Either the schema lives in the database and is enforced by the database, or it lives in the application code and everyone is assumed to comply with it. Saying “schemaless” means choosing the second — not deleting the schema, but moving it somewhere it isn’t enforced.
\nWhen the database stops enforcing the schema, that work doesn’t vanish; it scatters to these places:
\nNOT NULL, type check, and foreign key that the database does in a single line all turn into hand-written checks.A schema is not bureaucracy; it is validation the database does for free on your behalf. Refusing it doesn’t erase the bill; it just charges it to someone else.
\nProjects that start with the comfort of “schemaless” sooner or later hit these:
\nALTER TABLE does in the relational world becomes, here, a script that walks millions of documents one by one and has to be batched.These are invisible in a small project; they blow up at exactly the moment the project grows, the team changes, and no one can answer “why is that field sometimes missing?”
\nThis is not an anti-NoSQL piece. NoSQL has real and justified uses:
\nThe trap isn’t NoSQL; it’s choosing NoSQL not for one of these real strengths but just because “I don’t want a schema.” Right tool, wrong reason.
\nIf the need for “let some fields be flexible” is real, you don’t have to sacrifice all your integrity for it. PostgreSQL’s jsonb column gives you a flexible document field within the integrity of a relational table — as I touched on in is PostgreSQL enough for everything. The fields that are fixed stay as schema, and the part that is genuinely variable lives in jsonb. The best of both, and in a single system.
A schema is not a burden; it is a shield. When you remove it from the database it doesn’t disappear — it just leaves the place where it protected you and moves to a place where it can’t.
\nThere is no such thing as “schemaless”; there is only the question of “who enforces the schema.”
", "content_text": "I once saw a project that had picked MongoDB on the logic of \"no schema, we'll move fast\" — two years later. Every read was defensive: `data?.user?.address?.city ?? null`. The reason was simple — no two documents had exactly the same shape. The schema hadn't disappeared; it had just left the database and scattered across every bit of code that reads a document.\n\n## \"Schemaless\" is a misnomer\n\nThere is no such thing as schemaless data. Data always has a shape: a user's email, the line items of an order, the date on an invoice. The question is not whether there is a shape; it is **where that shape is defined**.\n\nThere are two options. Either the schema lives in the database and is enforced by the database, or it lives in the application code and everyone is assumed to comply with it. Saying \"schemaless\" means choosing the second — not deleting the schema, but moving it somewhere it isn't enforced.\n\n## Where does the schema move?\n\nWhen the database stops enforcing the schema, that work doesn't vanish; it scatters to these places:\n\n- **To every read.** The code can no longer assume the incoming document has the expected fields; it checks every field defensively.\n- **To every write.** There is no guarantee you wrote the right shape; there is only hope.\n- **To scattered validation code.** The `NOT NULL`, type check, and foreign key that the database does in a single line all turn into hand-written checks.\n- **To the team's heads.** \"That field is sometimes a string, sometimes an array\" becomes undocumented tribal knowledge.\n\nA schema is not bureaucracy; it is validation the database does for free on your behalf. Refusing it doesn't erase the bill; it just charges it to someone else.\n\n## The crises you hit\n\nProjects that start with the comfort of \"schemaless\" sooner or later hit these:\n\n- **Data drift.** Over time, ten different versions of a document live in the same collection. A record written in 2023 doesn't have the same shape as one written in 2026, and no migration ever enforced it.\n- **No referential integrity.** An order pointing to a user that doesn't exist isn't blocked by the database. Orphan records pile up silently.\n- **Migration becomes an application job.** What a single `ALTER TABLE` does in the relational world becomes, here, a script that walks millions of documents one by one and has to be batched.\n\nThese are invisible in a small project; they blow up at exactly the moment the project grows, the team changes, and no one can answer \"why is that field sometimes missing?\"\n\n## Where NoSQL really is the right answer\n\nThis is not an anti-NoSQL piece. NoSQL has real and justified uses:\n\n- Workloads that exceed the write capacity of a single primary and genuinely need horizontal scale — the hardest breaking point in [data-intensive systems](/en/systems/data-intensive-systems-breaking-points).\n- Documents that are genuinely heterogeneous by nature.\n- Specific access patterns — pure key-value, wide-column, graph.\n\nThe trap isn't NoSQL; it's choosing NoSQL not for one of these real strengths but just because \"I don't want a schema.\" Right tool, wrong reason.\n\n## If you want flexibility, PostgreSQL already gives it\n\nIf the need for \"let some fields be flexible\" is real, you don't have to sacrifice all your integrity for it. PostgreSQL's `jsonb` column gives you a flexible document field within the integrity of a relational table — as I touched on in [is PostgreSQL enough for everything](/en/journal/is-postgresql-enough-for-everything). The fields that are fixed stay as schema, and the part that is genuinely variable lives in `jsonb`. The best of both, and in a single system.\n\n---\n\nA schema is not a burden; it is a shield. When you remove it from the database it doesn't disappear — it just leaves the place where it protected you and moves to a place where it can't.\n\nThere is no such thing as \"schemaless\"; there is only the question of \"who enforces the schema.\"", "date_published": "2026-08-08T00:00:00.000Z", "tags": [ "nosql", "database", "architecture", "opinion", "Journal" ], "authors": [ { "name": "Muhammet Şafak", "url": "https://www.muhammetsafak.com.tr" } ] }, { "id": "https://sade.dev/en/journal/is-postgresql-enough-for-everything/", "url": "https://sade.dev/en/journal/is-postgresql-enough-for-everything/", "title": "Is PostgreSQL Enough for Everything?", "summary": "Most of the list that puts Elasticsearch, MongoDB and RabbitMQ next to PostgreSQL is carried by a single PostgreSQL for a long time: tsvector with a GIN index for search, jsonb for documents, SELECT FOR UPDATE SKIP LOCKED for a moderate-volume queue. The limits are honest and real, but a second data system should arrive on a measured limit, not on a hunch or a blog post.", "content_html": "An architecture plan, drawn up before there was a single user, carried this list: PostgreSQL for durable data, Elasticsearch for search, Redis for cache, MongoDB for flexible documents, RabbitMQ for queueing. Five separate data systems — each with its own backups, monitoring, version upgrades, and distinct failure mode.
\nMost of that list, for a long time, a single PostgreSQL carries on its own. The question isn’t “can PostgreSQL do this”; it’s “did you really buy the operational burden of these five systems”.
\nTreating PostgreSQL as just a table-row store uses a small fraction of what it can do. Most of the needs that prompt a separate system are already inside it:
\ntsvector, tsquery, and a GIN index. It covers most apps’ “search products” and “search posts” needs, including stemming, weighting, and ranking.jsonb column, paired with a GIN index, is a queryable document store. You can keep the fields that want schema flexibility right inside the relational table.SELECT ... FOR UPDATE SKIP LOCKED and LISTEN/NOTIFY run a moderate-volume job queue without a separate broker. Laravel’s database queue driver uses the SKIP LOCKED half of it — its workers poll the jobs table at intervals rather than waiting on a LISTEN/NOTIFY signal.These features aren’t “present but unused”; they’re mature capabilities that run reliably in production.
\nEvery new data system isn’t just a box; it’s a maintenance commitment. What you gain by staying on one system:
\nThis is the data-layer version of the innovation-token logic from the boring architecture piece: every new system is a token, and tokens are limited.
\nPostgreSQL isn’t enough for everything — it has honest limits, and ignoring them is also a mistake:
\nWhen you hit one of these limits, bringing in an extra system is the right call. Bringing it in before you hit it is just a guess.
\nThe rule is plain: in a new project, make PostgreSQL do everything PostgreSQL can do. Let it be the first stop for search, queue, and cache-like needs. A separate system should arrive only when a measured limit is crossed — not from a hunch, a blog post, or résumé anxiety.
\nMost teams use maybe 20% of PostgreSQL, then say “it’s not enough”. What’s not enough is usually not PostgreSQL, but the effort to get to know it.
\nPostgreSQL isn’t enough for everything — but it’s enough for far more than you think. Before you bring in a second data system, ask whether you’ve truly reached the end of the first.
\nPostgreSQL first; let the rest follow from measurement.
", "content_text": "An architecture plan, drawn up before there was a single user, carried this list: PostgreSQL for durable data, Elasticsearch for search, Redis for cache, MongoDB for flexible documents, RabbitMQ for queueing. Five separate data systems — each with its own backups, monitoring, version upgrades, and distinct failure mode.\n\nMost of that list, for a long time, a single PostgreSQL carries on its own. The question isn't \"can PostgreSQL do this\"; it's \"did you really buy the operational burden of these five systems\".\n\n## PostgreSQL's little-known breadth\n\nTreating PostgreSQL as just a table-row store uses a small fraction of what it can do. Most of the needs that prompt a separate system are already inside it:\n\n- **Full-text search.** A real search built on `tsvector`, `tsquery`, and a GIN index. It covers most apps' \"search products\" and \"search posts\" needs, including stemming, weighting, and ranking.\n- **Document storage.** A `jsonb` column, paired with a GIN index, is a queryable document store. You can keep the fields that want schema flexibility right inside the relational table.\n- **Queues.** `SELECT ... FOR UPDATE SKIP LOCKED` and `LISTEN/NOTIFY` run a moderate-volume job queue without a separate broker. Laravel's `database` queue driver uses the `SKIP LOCKED` half of it — its workers poll the jobs table at intervals rather than waiting on a `LISTEN/NOTIFY` signal.\n- **Analytical queries.** Window functions, CTEs, materialized views — serious reporting without reaching for a separate analytical database.\n- **Geospatial data.** Location queries via the PostGIS extension.\n\nThese features aren't \"present but unused\"; they're mature capabilities that run reliably in production.\n\n## The quiet payoff of a single system\n\nEvery new data system isn't just a box; it's a maintenance commitment. What you gain by staying on one system:\n\n- **One backup line.** One backup/restore procedure, one recovery drill.\n- **One monitoring target.** One set of metrics to learn, one alerting setup to build.\n- **Cross-consistency.** This is the most important one. If your search index lives in the same database as your data, there **can be no drift** between them — both update in the same transaction. A separate Elasticsearch will eventually fall out of sync with the data, and fixing that becomes its own line item.\n- **One mental model.** The team learns the quirks of one system, not five.\n\nThis is the data-layer version of the innovation-token logic from the [boring architecture](/en/journal/why-boring-architecture) piece: every new system is a token, and tokens are limited.\n\n## Where does the limit begin?\n\nPostgreSQL isn't enough for everything — it has honest limits, and ignoring them is also a mistake:\n\n- **Search.** When typo tolerance, advanced relevance tuning, faceted search, and multilingual analysis are needed at serious scale, a dedicated search engine genuinely pays off.\n- **Queues.** When you need very high throughput, complex routing, or multi-consumer fan-out, a real message broker is the right tool. A PostgreSQL queue is comfortable at moderate volume, not at the extremes.\n- **Write throughput.** Sustained writes beyond a single primary's fsync capacity — this is the hardest breaking point in [data-intensive systems](/en/systems/data-intensive-systems-breaking-points), and a real limit.\n\nWhen you hit one of these limits, bringing in an extra system is the right call. Bringing it in before you hit it is just a guess.\n\n## The decision: PostgreSQL first, then measure\n\nThe rule is plain: in a new project, make PostgreSQL do everything PostgreSQL can do. Let it be the first stop for search, queue, and cache-like needs. A separate system should arrive only when a **measured** limit is crossed — not from a hunch, a blog post, or résumé anxiety.\n\nMost teams use maybe 20% of PostgreSQL, then say \"it's not enough\". What's not enough is usually not PostgreSQL, but the effort to get to know it.\n\n---\n\nPostgreSQL isn't enough for everything — but it's enough for far more than you think. Before you bring in a second data system, ask whether you've truly reached the end of the first.\n\nPostgreSQL first; let the rest follow from measurement.", "date_published": "2026-08-01T00:00:00.000Z", "tags": [ "postgresql", "database", "architecture", "opinion", "Journal" ], "authors": [ { "name": "Muhammet Şafak", "url": "https://www.muhammetsafak.com.tr" } ] }, { "id": "https://sade.dev/en/journal/signals-of-an-over-complex-system/", "url": "https://sade.dev/en/journal/signals-of-an-over-complex-system/", "title": "Signals That a System Has Grown Too Complex", "summary": "Complexity does not arrive in one decision; it accumulates in defensible steps, which is why simplification starts from named signals rather than a feeling. Eight concrete signals, then a four-step subtraction: dead code, single-implementation interfaces, premature indirection, then clear boundaries. The target is not the least code but the least unnecessary code — essential complexity cannot be deleted.", "content_html": "At the end of her first week, an engineer who had just joined the team asked: “Why does adding a single field take two days?” The question was innocent; the diagnosis was clear. Nobody had sat down one day and decided to make that system complex. The complexity had simply accumulated.
\nThat is the heart of it: complexity doesn’t arrive in a single decision, it builds up unnoticed. To undo it, you first have to be able to see it.
\nNo complex system decides to be complex. Each step looks reasonable on its own: “let’s open an interface for now”, “let’s put a flag on this case”, “one more layer in between”. Each is small, each is defensible.
\nTheir sum is not defensible. Complexity is not an event but an accumulation — and an accumulation stays invisible until it is named. That is why simplification cannot start from a feeling; it starts from concrete signals.
\nThe red flags, not open to debate, that tell you a system is more complex than it needs to be:
\nIf two or three of these signals are present in a system, the debate isn’t “is it complex” but “where do we start”.
\nSimplification is not a rewrite; most of the time it’s a subtraction. In order:
\nAfter each step, stop and measure. Simplification can also go too far; the goal is not “the least code” but “the least unnecessary code”.
\nAn important distinction: there are two kinds of complexity. In Fred Brooks’s terms — essential and accidental.
\nEssential complexity comes from the domain itself. A tax calculation is complex because tax law is complex; you can’t delete that, you can only model it honestly. Accidental complexity is what you add: the unnecessary layer, the premature abstraction, the forgotten flag.
\nThe aim of simplification is to clear away the accidental — not the essential. Making a system “too simple” and ignoring the domain’s real complexity is also a mistake; that complexity doesn’t get deleted, it just moves to the wrong place — usually into the calling code.
\nGood architecture is exactly as complex as the domain is; no more.
\nComplexity accumulates unnoticed; that is why knowing by heart the signals that make it noticeable is an engineering skill. If you can’t name the signal, you can’t start simplifying either.
\nKeeping a system simple takes more effort than building it simple — and that effort is worth it.
", "content_text": "At the end of her first week, an engineer who had just joined the team asked: \"Why does adding a single field take two days?\" The question was innocent; the diagnosis was clear. Nobody had sat down one day and decided to make that system complex. The complexity had simply accumulated.\n\nThat is the heart of it: complexity doesn't arrive in a single decision, it builds up unnoticed. To undo it, you first have to be able to **see** it.\n\n## How does complexity accumulate?\n\nNo complex system decides to be complex. Each step looks reasonable on its own: \"let's open an interface for now\", \"let's put a flag on this case\", \"one more layer in between\". Each is small, each is defensible.\n\nTheir sum is not defensible. Complexity is not an event but an accumulation — and an accumulation stays invisible until it is named. That is why simplification cannot start from a feeling; it starts from concrete signals.\n\n## The concrete signals\n\nThe red flags, not open to debate, that tell you a system is more complex than it needs to be:\n\n- **You open a lot of files to understand a single feature.** If seeing where a request goes takes six files and four levels of indirection, the problem isn't you, it's the structure.\n- **A one-line change touches a dozen files.** To add one field: migration, model, DTO, mapper, interface, factory, test fixture... The cost of a change is not proportional to the size of the change.\n- **Interfaces with a single implementation.** Abstractions opened \"just in case\" whose second implementation never arrived. This is the residue [speculative generality](/en/journal/the-cost-of-just-in-case-code) leaves in a codebase.\n- **Onboarding takes weeks.** In a mature, lean system a new engineer is productive in the first days. If it takes weeks, the code isn't explaining itself.\n- **Nobody can draw the system on a whiteboard.** An architecture that can't be described as \"this box connects to that one\" is an architecture that isn't understood.\n- **The test setup is longer than the test itself.** The ten mocks you need to stand up to test one behavior are a confession that the behavior depends on far too much.\n- **\"Don't touch that, it'll break.\"** If a region of code has been declared untouchable, that region is already broken — it just hasn't blown up yet.\n- **Config, flags, and options nobody uses.** Every option that was opened and forgotten is a branch that everyone reading the code has to account for.\n\nIf two or three of these signals are present in a system, the debate isn't \"is it complex\" but \"where do we start\".\n\n## Where do you start simplifying?\n\nSimplification is not a rewrite; most of the time it's a **subtraction**. In order:\n\n1. **Delete dead code.** Unused flags, methods never called, branches never reached. The lowest-risk, highest-return step — deleted code has no bugs.\n2. **Collapse single-implementation interfaces.** Use the concrete class directly until the abstraction earns a real second use. When a seam is genuinely needed — [once the need is proven](/en/journal/the-cost-of-just-in-case-code) — it comes back in a few hours of refactoring.\n3. **Inline premature indirection.** Remove the layers that are called from a single place and only pass an argument on to another function.\n4. **Clarify boundaries.** Gather the remaining complexity behind clear boundaries, like in a [modular monolith](/en/journal/why-i-start-with-a-modular-monolith) — so the next accumulation shows up early.\n\nAfter each step, stop and measure. Simplification can also go too far; the goal is not \"the least code\" but \"the least unnecessary code\".\n\n## Complexity isn't always bad\n\nAn important distinction: there are two kinds of complexity. In Fred Brooks's terms — **essential** and **accidental**.\n\nEssential complexity comes from the domain itself. A tax calculation is complex because tax law is complex; you can't delete that, you can only model it honestly. Accidental complexity is what you add: the unnecessary layer, the premature abstraction, the forgotten flag.\n\nThe aim of simplification is to clear away the accidental — not the essential. Making a system \"too simple\" and ignoring the domain's real complexity is also a mistake; that complexity doesn't get deleted, it just moves to the wrong place — usually into the calling code.\n\nGood architecture is exactly as complex as the domain is; no more.\n\n---\n\nComplexity accumulates unnoticed; that is why knowing by heart the signals that make it noticeable is an engineering skill. If you can't name the signal, you can't start simplifying either.\n\nKeeping a system simple takes more effort than building it simple — and that effort is worth it.", "date_published": "2026-07-25T00:00:00.000Z", "tags": [ "architecture", "simplicity", "decisions", "opinion", "Journal" ], "authors": [ { "name": "Muhammet Şafak", "url": "https://www.muhammetsafak.com.tr" } ] }, { "id": "https://sade.dev/en/notes/circuit-breaker-external-service-integration/", "url": "https://sade.dev/en/notes/circuit-breaker-external-service-integration/", "title": "Circuit Breakers for External Service Integrations", "summary": "A slow external service takes an application down faster than a crashed one: the call holds the process, the pool fills, and unrelated endpoints start timing out. Three defenses, in order — a short explicit timeout on every call, retry only with exponential backoff and jitter, and a circuit breaker that stops calling at all. Its value is decided by the fallback you serve while the circuit is open.", "content_html": "An SMS provider slowed down — it didn’t even crash, its response time just climbed to 30 seconds. Within half an hour the entire application became unresponsive; even endpoints with nothing to do with SMS were timing out. One slow external service had dragged the whole system down with it.
\nAn external service will fail sooner or later. That’s not the question. The question is: will your system go down with it?
\nThe logic is simple and merciless. A slow external call locks up the worker that makes it — or the PHP-FPM process. If the call takes 30 seconds, that process can’t look at any other request for 30 seconds.
\nRequests pile up, the process pool fills. Once the pool is full, new requests — including ones with nothing to do with SMS — wait too. Your healthy endpoints die because of one sick dependency. This is called cascading failure, and it almost always starts with a slowdown.
\nThe first and cheapest defense is to give every external call an explicit and short timeout. Default timeouts — 30 seconds or more in most HTTP clients — are unacceptable for production.
\n$response = Http::timeout(5) // total time\n ->connectTimeout(2) // connection establishment time\n ->get('https://sms-provider.example/send');The rule from the Laravel queue post applies here too: not the default 30 seconds, but 5. Don’t leave to chance how long a process waits on a dependency — that number should be a decision.
\nRetry makes sense for transient errors, but blind retry does harm. Immediately retrying a failing service piles more load onto it — this is called a retry storm; it knocks the service back down before it can fully recover.
\nIf you retry, two things are mandatory: exponential backoff (increasing wait on each attempt) and jitter (random offset — so all clients don’t retry at the same moment).
\nHttp::retry(3, function (int $attempt) {\n // exponential backoff (200, 400, 800 ms) plus jitter\n return 200 * (2 ** ($attempt - 1)) + random_int(0, 100);\n}, function ($exception) {\n return $exception instanceof ConnectionException;\n})->timeout(5)->get($url);Keep the retry count low. If there’s still an error after three attempts, the problem isn’t transient; continuing to retry only ties up your own process for longer.
\nA timeout caps a single call; a circuit breaker decides to stop making calls at all. Its logic comes from an electrical circuit breaker, and it has three states:
\nclosed; if not, back to open.The gain: when the external service is sick, you don’t wait on it. Even a 5-second timeout is 5 seconds per process; in the open state that drops to 0. Fail-fast beats slow-fail.
// Conceptual flow — leave the threshold, counter, and timer to a mature package.\nif ($breaker->isOpen('sms')) {\n return $this->queueForLater($message); // circuit open: no attempt at all\n}\n\ntry {\n $response = Http::timeout(5)->post($smsUrl, $payload);\n $breaker->recordSuccess('sms');\n} catch (\\Throwable $e) {\n $breaker->recordFailure('sms'); // circuit opens if the threshold is exceeded\n return $this->queueForLater($message);\n}Instead of writing this logic from scratch, use a mature package — getting the state counting, the threshold, and the timer right is subtler than it looks.
\nA circuit breaker’s value shows up in what you do in the open state. “Return an error” is the weakest option. Better ones:
Which fallback is right depends on the business — but “no fallback” is not an answer.
\nNot every external call wants a circuit breaker. A timeout is always mandatory. Retry, only if the operation is idempotent. A circuit breaker mainly adds value when the call is frequent and the service is on the user’s path — for an integration called a few times a day, a timeout and a reasonable retry are usually enough.
\nAn external service failing isn’t a question of “if,” it’s a question of “when.” Going down with it, though, is a design choice — and a choice you can change.
\nDon’t leave your system as fragile as your weakest dependency.
", "content_text": "An SMS provider slowed down — it didn't even crash, its response time just climbed to 30 seconds. Within half an hour the entire application became unresponsive; even endpoints with nothing to do with SMS were timing out. One slow external service had dragged the whole system down with it.\n\nAn external service will fail sooner or later. That's not the question. The question is: will your system go down with it?\n\n## How does cascading failure happen?\n\nThe logic is simple and merciless. A slow external call locks up the worker that makes it — or the PHP-FPM process. If the call takes 30 seconds, that process can't look at any other request for 30 seconds.\n\nRequests pile up, the process pool fills. Once the pool is full, new requests — including ones with nothing to do with SMS — wait too. Your healthy endpoints die because of one sick dependency. This is called **cascading failure**, and it almost always starts with a slowdown.\n\n## First defense: timeout\n\nThe first and cheapest defense is to give every external call an **explicit and short** timeout. Default timeouts — 30 seconds or more in most HTTP clients — are unacceptable for production.\n\n```php\n$response = Http::timeout(5) // total time\n ->connectTimeout(2) // connection establishment time\n ->get('https://sms-provider.example/send');\n```\n\nThe rule from the [Laravel queue post](/en/notes/laravel-queue-production-slowdown) applies here too: not the default 30 seconds, but 5. Don't leave to chance how long a process waits on a dependency — that number should be a decision.\n\n## Second defense: retry — but carefully\n\nRetry makes sense for transient errors, but blind retry does harm. Immediately retrying a failing service piles more load onto it — this is called a **retry storm**; it knocks the service back down before it can fully recover.\n\nIf you retry, two things are mandatory: exponential backoff (increasing wait on each attempt) and jitter (random offset — so all clients don't retry at the same moment).\n\n```php\nHttp::retry(3, function (int $attempt) {\n // exponential backoff (200, 400, 800 ms) plus jitter\n return 200 * (2 ** ($attempt - 1)) + random_int(0, 100);\n}, function ($exception) {\n return $exception instanceof ConnectionException;\n})->timeout(5)->get($url);\n```\n\nKeep the retry count low. If there's still an error after three attempts, the problem isn't transient; continuing to retry only ties up your own process for longer.\n\n## Third defense: circuit breaker\n\nA timeout caps a single call; a circuit breaker decides to **stop making calls** at all. Its logic comes from an electrical circuit breaker, and it has three states:\n\n- **Closed** — everything's normal, calls pass through. Failures are counted.\n- **Open** — most of the last N calls failed; the circuit opens. The external service is no longer hit at all, and the call returns an error **instantly**.\n- **Half-open** — after a while the circuit allows a single trial call. If it succeeds, it returns to `closed`; if not, back to `open`.\n\nThe gain: when the external service is sick, you don't wait on it. Even a 5-second timeout is 5 seconds per process; in the `open` state that drops to 0. Fail-fast beats slow-fail.\n\n```php\n// Conceptual flow — leave the threshold, counter, and timer to a mature package.\nif ($breaker->isOpen('sms')) {\n return $this->queueForLater($message); // circuit open: no attempt at all\n}\n\ntry {\n $response = Http::timeout(5)->post($smsUrl, $payload);\n $breaker->recordSuccess('sms');\n} catch (\\Throwable $e) {\n $breaker->recordFailure('sms'); // circuit opens if the threshold is exceeded\n return $this->queueForLater($message);\n}\n```\n\nInstead of writing this logic from scratch, use a mature package — getting the state counting, the threshold, and the timer right is subtler than it looks.\n\n## Fallback: what do you do when the circuit is open?\n\nA circuit breaker's value shows up in what you do in the `open` state. \"Return an error\" is the weakest option. Better ones:\n\n- **Queue it, send it later.** If the SMS doesn't have to go out immediately, drop it on a [queue](/en/notes/sync-vs-async) to be processed once the circuit closes.\n- **Serve stale but valid data.** If an exchange-rate service is down, showing the last known rate beats showing none.\n- **Degrade the feature gracefully.** A \"Recommendations can't load right now\" message beats crashing the whole page.\n\nWhich fallback is right depends on the business — but \"no fallback\" is not an answer.\n\n## When don't you need all this?\n\nNot every external call wants a circuit breaker. A timeout is **always** mandatory. Retry, only if the operation is idempotent. A circuit breaker mainly adds value when the call is frequent and the service is on the user's path — for an integration called a few times a day, a timeout and a reasonable retry are usually enough.\n\n---\n\nAn external service failing isn't a question of \"if,\" it's a question of \"when.\" Going down with it, though, is a design choice — and a choice you can change.\n\nDon't leave your system as fragile as your weakest dependency.", "date_published": "2026-07-18T00:00:00.000Z", "tags": [ "resilience", "architecture", "api", "production", "Note" ], "authors": [ { "name": "Muhammet Şafak", "url": "https://www.muhammetsafak.com.tr" } ] }, { "id": "https://sade.dev/en/notes/serverless-migration-decision/", "url": "https://sade.dev/en/notes/serverless-migration-decision/", "title": "The Serverless Decision: Cold-Start and Vendor Lock-in", "summary": "Serverless is not an architectural fashion, it is a tool with a workload profile: variable, sparse, bursty. Steady traffic turns pay-per-use into pay-per-second, and on a low-traffic path a cold start is the rule rather than the exception, so the user pays it on nearly every request. The bond is not the runtime but the glue around it — event sources, IAM, gateway — which makes leaving a rewrite, not a migration.", "content_html": "A team built its new service on AWS Lambda with a “serverless first” decision. Yet the traffic was steady and predictable — roughly the same all day. At the end of the month the bill came in at several times what a single VPS doing the same work would cost; on top of that, p95 latency was worse.
\nServerless isn’t the wrong technology. It was the right technology applied to the wrong workload.
\nFaaS (Function as a Service) has real upsides:
\nThese three genuinely pay off on the right workload. The trick is diagnosing that workload correctly.
\nIf a function hasn’t been called for a while, the provider brings it up from scratch: the runtime loads, dependencies initialize, the code runs. This cold-start adds anywhere from under 100 ms to seconds of latency compared to an already-running process.
\nThe key distinction: for a low-traffic endpoint, cold-start is the rule, not the exception. On a function that gets a few calls an hour, nearly every request is cold. “Pay by scale” sounds appealing, but at near-zero traffic what you pay is the latency the user sees every single time. On a latency-sensitive user path, that’s a direct UX cost.
\nLock-in isn’t just the runtime; the real bond is in the glue around it. Over time a serverless application also couples to the provider’s event sources, identity/authorization model (IAM), managed queue, API gateway, and deployment tooling.
\nMoving a single function is easy. But in a real system, “getting off serverless” is usually not a migration, it’s a rewrite. This ties directly into the innovation-token logic from the boring architecture piece: what looks like a single decision is actually a bond that’s expensive to undo.
\nServerless shines where the load is variable and sparse:
\nOn these workloads, scale-to-zero is real money saved, and cold-start is usually irrelevant — nobody cares that a nightly report starts 800 ms late.
\nThe reverse is just as clear:
\nFor a workload with this profile, a boring VPS — like the setup in the multiple projects on a single VPS piece — is both cheaper and more predictable. The cost of a steady load matches a steady server.
\nOn the Laravel side, Laravel Vapor makes it possible to run the application on Lambda, and it works well. But that it works doesn’t mean it’s the right decision: with Vapor you also inherit cold-start and the AWS bond.
\nThe decision is, again, the workload. For a Laravel app with bursty, sparse, or unpredictable traffic, Vapor makes sense. For a continuous, latency-sensitive app, a classic server is still the plainer, cheaper answer.
\nThis isn’t a binary choice. A boring server carrying the continuous load + handing only the sparse/bursty work (webhooks, the nightly report) to serverless is often the most honest split. Put every workload where its own cost profile belongs.
\nServerless isn’t an architectural fashion, it’s a tool for a specific workload profile. If you have that profile, it pays off; if you don’t, it’s paid back as cold-start and lock-in.
\nMeasure your load first; the tool comes after.
", "content_text": "A team built its new service on AWS Lambda with a \"serverless first\" decision. Yet the traffic was steady and predictable — roughly the same all day. At the end of the month the bill came in at several times what a single VPS doing the same work would cost; on top of that, p95 latency was worse.\n\nServerless isn't the wrong technology. It was the right technology applied to the wrong workload.\n\n## What serverless buys you\n\nFaaS (Function as a Service) has real upsides:\n\n- **No server management.** Patching, scaling, capacity planning all sit with the provider.\n- **Scale to zero.** No requests means nothing running and nothing paid for.\n- **Pay per use.** You don't pay for idle capacity.\n\nThese three genuinely pay off on the right workload. The trick is diagnosing that workload correctly.\n\n## The real cost of cold-start\n\nIf a function hasn't been called for a while, the provider brings it up from scratch: the runtime loads, dependencies initialize, the code runs. This **cold-start** adds anywhere from under 100 ms to seconds of latency compared to an already-running process.\n\nThe key distinction: for a low-traffic endpoint, cold-start is **the rule, not the exception**. On a function that gets a few calls an hour, nearly every request is cold. \"Pay by scale\" sounds appealing, but at near-zero traffic what you pay is the latency the user sees every single time. On a latency-sensitive user path, that's a direct UX cost.\n\n## Vendor lock-in\n\nLock-in isn't just the runtime; the real bond is in the glue around it. Over time a serverless application also couples to the provider's event sources, identity/authorization model (IAM), managed queue, API gateway, and deployment tooling.\n\nMoving a single function is easy. But in a real system, \"getting off serverless\" is usually not a migration, it's a rewrite. This ties directly into the innovation-token logic from the [boring architecture](/en/journal/why-boring-architecture) piece: what looks like a single decision is actually a bond that's expensive to undo.\n\n## Which workloads win?\n\nServerless shines where the load is **variable and sparse**:\n\n- Scheduled jobs — nightly reports, cron-like tasks.\n- Webhook receivers — uncertain when and how often the calls will arrive.\n- Low-volume internal tools — panels used a few times a day.\n- Bursty batch jobs — once an hour, briefly high parallelism.\n\nOn these workloads, scale-to-zero is real money saved, and cold-start is usually irrelevant — nobody cares that a nightly report starts 800 ms late.\n\n## Which workloads lose?\n\nThe reverse is just as clear:\n\n- **Steady, continuous traffic.** On an always-running API, \"pay per use\" turns into \"pay every second\" — and that's usually more expensive than a continuously running server.\n- **Latency-sensitive paths.** Cold-start is unacceptable on a request the user is waiting on.\n- **Long-running jobs.** FaaS has execution-time limits; long jobs either get split up or don't fit.\n\nFor a workload with this profile, a boring VPS — like the setup in the [multiple projects on a single VPS](/en/systems/single-vps-multi-project-architecture) piece — is both cheaper and more predictable. The cost of a steady load matches a steady server.\n\n## Laravel and serverless\n\nOn the Laravel side, Laravel Vapor makes it possible to run the application on Lambda, and it works well. But that it works doesn't mean it's the right decision: with Vapor you also inherit cold-start and the AWS bond.\n\nThe decision is, again, the workload. For a Laravel app with bursty, sparse, or unpredictable traffic, Vapor makes sense. For a continuous, latency-sensitive app, a classic server is still the plainer, cheaper answer.\n\n## Mix them\n\nThis isn't a binary choice. A boring server carrying the continuous load + handing only the sparse/bursty work (webhooks, the nightly report) to serverless is often the most honest split. Put every workload where its own cost profile belongs.\n\n---\n\nServerless isn't an architectural fashion, it's a tool for a specific workload profile. If you have that profile, it pays off; if you don't, it's paid back as cold-start and lock-in.\n\nMeasure your load first; the tool comes after.", "date_published": "2026-07-11T00:00:00.000Z", "tags": [ "serverless", "architecture", "cloud", "Note" ], "authors": [ { "name": "Muhammet Şafak", "url": "https://www.muhammetsafak.com.tr" } ] }, { "id": "https://sade.dev/en/notes/bff-backend-for-frontend/", "url": "https://sade.dev/en/notes/bff-backend-for-frontend/", "title": "BFF Pattern: A Separate API Layer for Mobile and Web", "summary": "Bending one general-purpose API to fit every client is paid for in over-fetching, chatty flows, conditional fields and a versioning knot that locks web and mobile into one contract. A BFF cancels that bill with a thin adaptation layer per client — aggregation and response shaping, never business rules. What justifies the layer is the distance between clients, not their number.", "content_html": "I once watched a mobile app fire nine separate requests just to render a single home screen. The API had actually been designed for web; mobile was collecting the data the way web sliced it, piece by piece, and stitching it back together on screen. Nine round-trips per launch — under mobile network conditions, no less.
\nThat was the bill for the “one API fits everyone” assumption. The BFF — Backend for Frontend — exists precisely to cancel that bill.
\nA BFF is a thin API layer tailored to one specific client type. Web gets its own BFF, mobile gets its own. Each one calls the same underlying services and domain, but shapes the response to fit its own client’s screen.
\nThe real business logic doesn’t live in the BFF. The BFF is only an adaptation layer: it aggregates, trims unneeded fields, and gathers everything a client needs for one screen into a single response.
\nA single general-purpose API can’t fully satisfy two clients over time:
\nif mobile then ... else .... A single endpoint tries to carry the requirements of two clients at once and does a mediocre job for both.Each of these costs looks small on its own; their sum is an architectural problem.
\nA BFF is justified when client needs genuinely diverge:
\nWhen these differences exist, giving each client its own BFF is cheaper than constantly bending one API to fit both.
\nIf you have a single client, a BFF is more layer than solution — just an extra hop.
\nIf you have two clients but both want almost the same data in almost the same shape, it’s still unnecessary. What justifies a BFF isn’t the number of clients but the difference between them. No difference, no separate layer — otherwise it’s speculative generality in the shape of an API layer.
\nA common confusion: adding a BFF doesn’t drop you into distributed architecture. A BFF is a presentation/aggregation layer, not a domain service. Putting business rules inside it means copying the rule once per client.
\nThe BFF’s only kinship with microservices is this: both ask “is this worth a separate deployable?” The answer is usually “no” — the measured-signal threshold from the decision to move to microservices applies here too.
\n“BFF” doesn’t have to mean standing up a separate server. Within the same monolith:
\nroutes/web-api.php, routes/mobile-api.php — each with its own controllers and its own response shape.The real idea isn’t a box, it’s a boundary: collect the client-specific adaptation in one place and keep the domain clean of it. Move to a separate deployable only when a measured signal forces it.
\nThe BFF is the antidote to the “every client makes do with the same API” stubbornness — but only when clients genuinely diverge. No difference, no layer.
\nWhat adds the layer isn’t the number of clients, it’s the distance between them.
", "content_text": "I once watched a mobile app fire nine separate requests just to render a single home screen. The API had actually been designed for web; mobile was collecting the data the way web sliced it, piece by piece, and stitching it back together on screen. Nine round-trips per launch — under mobile network conditions, no less.\n\nThat was the bill for the \"one API fits everyone\" assumption. The BFF — Backend for Frontend — exists precisely to cancel that bill.\n\n## What a BFF is\n\nA BFF is a thin API layer tailored to one specific client type. Web gets its own BFF, mobile gets its own. Each one calls the same underlying services and domain, but shapes the response to fit its own client's screen.\n\nThe real business logic doesn't live in the BFF. The BFF is only an **adaptation layer**: it aggregates, trims unneeded fields, and gathers everything a client needs for one screen into a single response.\n\n## The hidden cost of bending one API to fit everyone\n\nA single general-purpose API can't fully satisfy two clients over time:\n\n- **Over-fetching.** Mobile downloads a 40-field user object and uses 4 of them. The rest is wasted bandwidth and battery.\n- **Chatty flow.** The \"fetch the list first, then fetch details for each row\" flow that web finds perfectly normal turns into nine round-trips on a mobile network.\n- **Conditional fields.** The response starts branching with `if mobile then ... else ...`. A single endpoint tries to carry the requirements of two clients at once and does a mediocre job for both.\n- **Versioning knot.** A change needed for web risks breaking mobile's published version. Two clients get locked into one contract.\n\nEach of these costs looks small on its own; their sum is an architectural problem.\n\n## When is a BFF warranted?\n\nA BFF is justified when client needs **genuinely** diverge:\n\n- Mobile and web need distinctly different screens and different data shapes.\n- Mobile network and battery constraints make round-trip count and payload size a first-class concern.\n- The clients ship at different cadences — the mobile release is stuck in store review while web deploys every day; a single contract locks the two together.\n\nWhen these differences exist, giving each client its own BFF is cheaper than constantly bending one API to fit both.\n\n## When is a BFF unnecessary?\n\nIf you have a single client, a BFF is more layer than solution — just an extra hop.\n\nIf you have two clients but both want almost the same data in almost the same shape, it's still unnecessary. What justifies a BFF isn't the number of clients but the **difference between** them. No difference, no separate layer — otherwise it's [speculative generality](/en/journal/the-cost-of-just-in-case-code) in the shape of an API layer.\n\n## A BFF is not a microservice\n\nA common confusion: adding a BFF doesn't drop you into distributed architecture. A BFF is a presentation/aggregation layer, not a domain service. Putting business rules inside it means copying the rule once per client.\n\nThe BFF's only kinship with microservices is this: both ask \"is this worth a separate deployable?\" The answer is usually \"no\" — the measured-signal threshold from [the decision to move to microservices](/en/journal/when-i-move-to-microservices) applies here too.\n\n## Start light\n\n\"BFF\" doesn't have to mean standing up a separate server. Within the same monolith:\n\n- One route namespace per client — `routes/web-api.php`, `routes/mobile-api.php` — each with its own controllers and its own response shape.\n- Or a single GraphQL layer: each client queries exactly what it needs, and over-fetching resolves itself.\n\nThe real idea isn't a box, it's a boundary: collect the client-specific adaptation in one place and keep the domain clean of it. Move to a separate deployable only when [a measured signal](/en/journal/when-i-move-to-microservices) forces it.\n\n---\n\nThe BFF is the antidote to the \"every client makes do with the same API\" stubbornness — but only when clients genuinely diverge. No difference, no layer.\n\nWhat adds the layer isn't the number of clients, it's the distance between them.", "date_published": "2026-07-04T00:00:00.000Z", "tags": [ "architecture", "api", "bff", "Note" ], "authors": [ { "name": "Muhammet Şafak", "url": "https://www.muhammetsafak.com.tr" } ] }, { "id": "https://sade.dev/en/notes/sync-vs-async/", "url": "https://sade.dev/en/notes/sync-vs-async/", "title": "Synchronous or Asynchronous? HTTP or the Queue", "summary": "Synchronous or asynchronous is not a performance question, it is a question of who is waiting. If the user has to see the result, the job stays inside the request; if not, it goes to a queue and the request answers 202 Accepted. Failure sharpens the same line: a synchronous failure is an error screen, an asynchronous one is a retry — which is why an unstable third party never belongs on the synchronous path.", "content_html": "A signup form took eight seconds. Inside the request, in order: the user got saved, a welcome email went out, a PDF was generated, a record was pushed to a CRM. For all eight seconds the user stared at a spinner — when the only thing they cared about was whether their account had been created.
\nShould a job run inside the HTTP request, or go to a queue? The line is clearer than it looks.
\nOne question decides most of it: does the user have to see the result of this job?
\nIn the form above, the user is only waiting for the answer to “was my account created?” The email, the PDF, the CRM — none of their results need to show up on that screen. All three were on the wrong side.
\nJobs whose result the user sees directly stay inside the request:
\nPutting these on a queue means telling the user “check back later” — and that is usually a bad experience.
\nEvery job whose result the user doesn’t need right then is asynchronous:
\nFor these, the right response is 202 Accepted: “I got your request, I’m working on it.” The user doesn’t wait, and the system does the work at its own pace.
Every endpoint should have a p95 budget — say 300 ms. If a single step eats that budget on its own and the user doesn’t have to see that step’s result, the step goes to a queue.
\nA synchronous chain is only as fast as its slowest link. If PDF generation takes 4 seconds, user creation takes 4 seconds too — even though the PDF has nothing to do with the response.
\nThe real issue isn’t speed, it’s failure. Synchronous and asynchronous handle failure in completely different ways:
\nFrom this comes a single rule: an unstable third party should never sit on the synchronous path. If that CRM call is inside the request, your signup form goes down every time the CRM does. If the call is queued, the retry completes on its own once the CRM comes back. The third party’s uptime stops being your uptime.
\nOnce a job goes to a queue, leave the user a way to follow up:
\nPOST /exports → 202 Accepted { "id": "exp_1a2b", "status": "processing" }\nGET /exports/exp_1a2b → 200 OK { "status": "done", "url": "..." }The user isn’t waiting on the result, but sometimes they get curious. A status endpoint, or a notification sent once it’s ready, closes that gap.
\nSome jobs sit between the two: the user wants to see the result, but the job is also slow. There’s a third path here — start the job synchronously, return the first meaningful result immediately, hand the rest off to a queue. Create the order record instantly (synchronous), generate the invoice in the background (asynchronous).
\nDraw the line per job; “everything to a queue” is as wrong as “everything synchronous.”
\nSynchronous or asynchronous isn’t a performance question, it’s a “who’s waiting” question. No job the user isn’t waiting on should make their request wait.
\nTie up the request only with the work whose answer they actually want.
", "content_text": "A signup form took eight seconds. Inside the request, in order: the user got saved, a welcome email went out, a PDF was generated, a record was pushed to a CRM. For all eight seconds the user stared at a spinner — when the only thing they cared about was whether their account had been created.\n\nShould a job run inside the HTTP request, or go to a queue? The line is clearer than it looks.\n\n## The decision line: is the user waiting on the result?\n\nOne question decides most of it: **does the user have to see the result of this job?**\n\n- If the answer is \"yes,\" the job stays synchronous — it finishes inside the request, before the response returns.\n- If the answer is \"no,\" the job goes to a queue — the request says \"accepted\" and returns.\n\nIn the form above, the user is only waiting for the answer to \"was my account created?\" The email, the PDF, the CRM — none of their results need to show up on that screen. All three were on the wrong side.\n\n## Which jobs have to stay synchronous?\n\nJobs whose result the user sees directly stay inside the request:\n\n- Input validation — the error must come back immediately.\n- Creating the actual resource — user creation, order creation.\n- The **result of a payment authorization** — did the card go through or not? The user has to know right then.\n- Data the user will see in the next step.\n\nPutting these on a queue means telling the user \"check back later\" — and that is usually a bad experience.\n\n## Which jobs should go to a queue?\n\nEvery job whose result the user doesn't need right then is asynchronous:\n\n- Email and notifications.\n- PDF and report generation.\n- Webhook delivery.\n- Third-party synchronization — CRM, analytics, search index.\n- Image processing, exports.\n\nFor these, the right response is `202 Accepted`: \"I got your request, I'm working on it.\" The user doesn't wait, and the system does the work at its own pace.\n\n## Response time sharpens the line\n\nEvery endpoint should have a p95 budget — say 300 ms. If a single step eats that budget on its own and the user doesn't have to see that step's result, the step goes to a queue.\n\nA synchronous chain is only as fast as its slowest link. If PDF generation takes 4 seconds, user creation takes 4 seconds too — even though the PDF has nothing to do with the response.\n\n## Fault tolerance: the real distinction\n\nThe real issue isn't speed, it's failure. Synchronous and asynchronous handle failure in completely different ways:\n\n- **Synchronous failure** = the user sees an error screen. The work is lost.\n- **Asynchronous failure** = the job is retried, drops to a DLQ if it has to, and the [queue](/en/notes/laravel-queue-production-slowdown) processes it later.\n\nFrom this comes a single rule: **an unstable third party should never sit on the synchronous path.** If that CRM call is inside the request, your signup form goes down every time the CRM does. If the call is queued, the retry completes on its own once the CRM comes back. The third party's uptime stops being your uptime.\n\n## The 202-and-status-poll pattern\n\nOnce a job goes to a queue, leave the user a way to follow up:\n\n```\nPOST /exports → 202 Accepted { \"id\": \"exp_1a2b\", \"status\": \"processing\" }\nGET /exports/exp_1a2b → 200 OK { \"status\": \"done\", \"url\": \"...\" }\n```\n\nThe user isn't waiting on the result, but sometimes they get curious. A status endpoint, or a notification sent once it's ready, closes that gap.\n\n## Where does the line dissolve?\n\nSome jobs sit between the two: the user wants to see the result, but the job is also slow. There's a third path here — start the job synchronously, return the first meaningful result immediately, hand the rest off to a queue. Create the order record instantly (synchronous), generate the invoice in the background (asynchronous).\n\nDraw the line per job; \"everything to a queue\" is as wrong as \"everything synchronous.\"\n\n---\n\nSynchronous or asynchronous isn't a performance question, it's a \"who's waiting\" question. No job the user isn't waiting on should make their request wait.\n\nTie up the request only with the work whose answer they actually want.", "date_published": "2026-06-27T00:00:00.000Z", "tags": [ "architecture", "queue", "async", "api", "Note" ], "authors": [ { "name": "Muhammet Şafak", "url": "https://www.muhammetsafak.com.tr" } ] }, { "id": "https://sade.dev/en/systems/transactional-outbox-dual-write-and-idempotent-consumption/", "url": "https://sade.dev/en/systems/transactional-outbox-dual-write-and-idempotent-consumption/", "title": "Transactional Outbox: Dual-Write, Idempotent Consumers", "summary": "An outbox collapses dual-write into a single write but gives no free guarantee — it gives at-least-once. Polling vs CDC, multi-relay with SKIP LOCKED, ordering, pruning processed_messages, the exactly-once illusion, and field-level encryption.", "content_html": "The symptom was small: one or two invoices a day sat in the database but the consumer never saw them. The user said “I issued it,” and there was no trace on the other side. Always at night, always during a deploy or a network blip. Once I laid the picture out, the cause was clear — the INSERT into the database and the publish to the queue were two separate systems, with no guarantee between them.
This post is the systems side of the transactional outbox pattern that closed that gap, the at-least-once repeats it opened in return, and finally encrypting the personal data that flows through the event at the field level. I wrote the decision-chain side as a log on muhammetsafak.com.tr; here I collect the decisions you have to make for the pattern to be durable in production.
\nThe crux in one sentence: your local database transaction doesn’t cover RabbitMQ. The classic code is dangerous because it works most of the time:
\nDB::transaction(function () use ($invoice) {\n $invoice->save(); // 1) write to MySQL\n $this->publishToRabbit($invoice); // 2) publish to the queue\n});There are two distinct failure modes:
\nsave() succeeds, publish drops on a network error → the invoice is in the DB, the event isn’t. Lost event.publish succeeds, then the transaction rollbacks for another reason → the event went out, no counterpart in the DB. Phantom event.DB::transaction doesn’t save you, because commit/rollback only wraps the MySQL side; the broker isn’t part of that transaction. The practical way to hold two systems in one atomic step is to reduce the write to a single system.
The idea is plain: instead of pushing the message straight to the queue, write it as a row into an outbox table inside the same transaction. The invoice and the event either both exist or both don’t, in the same commit — dual-write collapses into a single write.
CREATE TABLE outbox (\n id BINARY(16) NOT NULL, -- event id = idempotency key\n aggregate_id BIGINT NOT NULL, -- ordering and partition key\n topic VARCHAR(120) NOT NULL,\n payload JSON NOT NULL,\n status ENUM('pending','published') NOT NULL DEFAULT 'pending',\n created_at DATETIME(6) NOT NULL,\n published_at DATETIME(6) NULL,\n PRIMARY KEY (id),\n KEY idx_dispatch (status, created_at) -- the relay scan goes through this index\n);The write is no longer the business code’s concern:
\nDB::transaction(function () use ($invoice) {\n $invoice->save();\n Outbox::write('invoice.issued', $invoice->id, $this->payload($invoice));\n});A separate relay does the queue push: it reads the pending rows, publishes them to RabbitMQ, and stamps them published on success. The critical fact here: the relay can fall into the “I published but crashed before stamping it published” state. So while the outbox solves dual-write, it gives you no free guarantee — it gives you at-least-once. The message isn’t lost, but it can repeat. Everything else arranges itself around this fact.
The relay can see pending rows two ways.
Polling — the relay scans the table periodically:
\nSELECT id, topic, payload\nFROM outbox\nWHERE status = 'pending'\nORDER BY created_at\nLIMIT 100;Simple, works on every database, low operational overhead. The cost is two things: latency up to the scan interval (a 1s poll = ~1s of queue latency) and an idling query. Without the idx_dispatch index this scan gets expensive as the table grows; even with it, you trade poll frequency against DB load. At small-to-medium volume, polling is the right answer — tune it frequent enough to keep latency acceptable, sparse enough not to hammer the DB (in practice 200ms–1s).
CDC (change data capture) — the relay listens to the database’s WAL/binlog rather than the table (Debezium is the typical tool). Every INSERT into outbox turns into an event almost instantly; polling latency and idle scanning disappear. The cost is operational: binlog access, a connector process, one more moving part. I turn CDC on when latency genuinely matters (sub-second) or when volume strains polling; otherwise I count polling’s simplicity as an advantage.
\n\nRule: your latency budget and your operational budget conflict. CDC buys you latency and charges you an infrastructure piece in return. Start with polling; switch to CDC when there’s a measured reason.
\n
SKIP LOCKEDA single relay is a bottleneck and a single point of failure. Running multiple relays over the same table creates a new risk: two of them grab the same row and publish the same message twice. The naive fix is to lock the table — which kills the parallelism.
\nThe right tool is SELECT ... FOR UPDATE SKIP LOCKED. Each relay claims a batch of rows by locking them; instead of queuing behind rows another relay has locked, it skips them:
BEGIN;\n SELECT id, topic, payload\n FROM outbox\n WHERE status = 'pending'\n ORDER BY created_at\n LIMIT 100\n FOR UPDATE SKIP LOCKED; -- skip rows another relay holds, don't wait\n\n -- publish this batch, then:\n UPDATE outbox SET status = 'published', published_at = NOW(6)\n WHERE id IN (...);\nCOMMIT;Without SKIP LOCKED, the relays queue for the same rows and the parallelism effectively runs serially. With it, each relay pulls a disjoint set and all of them move at once. PostgreSQL and MySQL 8+ support it.
A subtlety: if the relay crashes between publish and UPDATE ... published, the row stays pending and gets republished. That’s acceptable — we’re already at at-least-once; the fix is on the consumer side, below. The dangerous order is the reverse: stamping published first and then publishing. A crash there produces a lost event — exactly what we ran from. So the order is fixed: publish first, stamp second.
“Let events arrive in the order they were written” is an intuitive expectation, but global ordering is expensive and usually unnecessary. What you actually need is per-aggregate order: the same invoice’s created event should arrive before its updated; the order of two different invoices relative to each other is nobody’s concern.
You get this by partitioning on aggregate_id: events with the same aggregate_id go to the same queue partition (or, in RabbitMQ, the same queue via a consistent-hash exchange), and a single consumer processes that partition in order. Different aggregates flow in parallel.
outbox (in created_at order)\n │ hash(aggregate_id) % N\n ├── partition 0 ──► consumer-0 (events for aggregates A,D — ordered)\n ├── partition 1 ──► consumer-1 (events for aggregate B — ordered)\n └── partition 2 ──► consumer-2 (events for aggregate C — ordered)Two things still break ordering, and the design has to be ready for them: multiple relays under SKIP LOCKED can publish rows in a different order, and at-least-once repeats can cut in. So rather than leaning on a global order, you write the consumer to be resilient to out-of-order and duplicated events. The practical shield: put a monotonic version/sequence on the event and drop the stale one (lower version) at the consumer. If you genuinely need strict global ordering you drop to a single partition + single consumer — which then ties throughput to that one consumer; most systems don’t want to pay that.
Because the relay can republish, you have to assume every event arrives at least once, sometimes more. The fix isn’t to make the event unique — it’s to set up the consumer so that processing the same event twice has the same effect as once. Write the id of every processed event into a dedupe table, and do the work inside the same transaction:
DB::transaction(function () use ($event) {\n $inserted = DB::table('processed_messages')->insertOrIgnore([\n 'message_id' => $event['id'],\n 'processed_at' => now(),\n ]);\n\n if ($inserted === 0) {\n return; // it's a repeat; produce no side effects\n }\n\n $this->applyBusinessEffect($event); // the real work — exactly once\n});The unique constraint on processed_messages.message_id is the heart of it: a second arrival of the same id is silently dropped by insertOrIgnore. Putting the business effect in the same transaction is mandatory — otherwise the “I processed it but crashed before stamping” gap opens. Where the dedupe key comes from, and designing a dedupe that survives a crash, is a topic of its own; I covered it in a separate note.
If the side effect is an external system (an e-invoice integrator, a payment API), dedupe alone isn’t enough — because the side effect is outside the transaction. There you have to carry the key to the external service: pass the same id as an idempotency key to the integrator, or ask “does this already exist?” before sending. I worked through that boundary in detail in the JIT reservation post, where sealing the number before the external call turns recovery into a lookup.
processed_messages can’t grow foreverThe dedupe table accumulates a row per event; if you don’t prune it, it becomes a performance problem of its own. The key observation: you don’t need to keep an event id forever — only as long as the window in which a repeat could arrive. Relay retries and broker redeliveries are on the order of hours, not days.
In practice I pick a fixed retention window (say 7 days — comfortably above the longest possible redelivery) and prune older rows:
\nDELETE FROM processed_messages\nWHERE processed_at < NOW() - INTERVAL 7 DAY\nLIMIT 10000; -- small batches at a time, keeping replica lag and locks in checkIf message_id is a UUID the table and its index grow; pruning keeps that bounded. When choosing the window, frame the rule backwards: how late could the latest redelivery arrive? Make a safe upper bound of that your retention. Set the window too short and a late repeat misses the dedupe, so the side effect runs a second time.
In a distributed system “exactly-once delivery” is tempting but can’t be guaranteed end to end. The reason is the same dual-write, at the broker boundary: if the consumer did the work but crashed before getting its ack to the broker, the broker redelivers the message — because it never saw the ack. “I processed it” and “I reported that I processed it” are two separate steps, and a crash can fall between them. This is the practical face of the Two Generals problem.
So the goal isn’t exactly-once delivery, it’s exactly-once effect. And you get that by combining two cheap guarantees:
\nat-least-once delivery + idempotent consumer = effectively-onceThat is, you stop trying to deliver exactly once and instead make the repeat harmless. Under systems that market “exactly-once,” this is usually exactly what’s there: at-least-once plus a dedupe layer. Look for the guarantee in the effect, not in the delivery.
\nOnce the outbox settles in, a new surface appears: the events’ payload carries personal data — customer name, tax ID, address — and that payload now sits in a durable table (outbox) and passes through the broker. If I leave the data as-is, I’m copying GDPR/KVKK-scoped fields, unencrypted, into more than one place.
I didn’t want to encrypt the whole payload — I needed fields like topic, aggregate_id, and invoice_id in the clear for relay routing and observability. The need is field-level encryption: only the sensitive fields encrypted, the rest in the clear. I marked the sensitive fields in the schema with x-gdpr-sensitive and encrypted only those at serialization time:
$schema = [\n 'invoice_id' => ['x-gdpr-sensitive' => false],\n 'customer_name' => ['x-gdpr-sensitive' => true],\n 'tax_id' => ['x-gdpr-sensitive' => true],\n 'total' => ['x-gdpr-sensitive' => false],\n];\n\n$payload = collect($raw)->map(fn ($value, $field) =>\n ($schema[$field]['x-gdpr-sensitive'] ?? false)\n ? Crypt::encryptString((string) $value) // only the sensitive field is encrypted\n : $value\n)->all();That the schema declares the sensitive field is no accident — combine it with schema validation at the edge of the queue and both questions (“which field is personal data, and is it the right type?”) get answered at the edge. The real engineering decision isn’t the encryption itself, it’s the three questions around it:
\nKey management and rotation. Laravel’s Crypt uses AES-256 keyed by APP_KEY. Bind yourself to a single key and rotation becomes a nightmare: the new key can’t open outbox rows written with the old one. The fix is to store a key version (key id) alongside the encrypted value and keep a keyring that recognizes several keys at once — new writes use the current key, old values decrypt with their own version. In Laravel, key + previous_keys in config/app.php is exactly this; decryption tries the older keys in turn. That’s why you can rotate without migrating data.
Searching an encrypted field. Crypt::encryptString produces different ciphertext on every call (a random IV), which is correct for security but makes WHERE tax_id = ? impossible. If you need equality search you add a blind index: keep a deterministic HMAC of the searchable field in a separate column and query that. The encrypted column carries confidentiality, the blind index carries searchability; they’re separate columns because they do two separate jobs.
Losing the key is losing the data. Field encryption turns APP_KEY into an availability dependency: lose the key and the encrypted fields are permanent garbage. Key backup and access control are as much a part of the design as the encryption itself.
Outbox + idempotent consumption + field encryption isn’t cheap: it brings a table, a relay process (or a CDC connector), a dedupe table and its pruning, plus key management. If you don’t know what you’re buying in return, don’t pay for it. I don’t build it when:
\npublish is enough. You build the outbox because loss matters.The need for the pattern starts when three conditions arise together: event loss is unacceptable, production/consumption is parallel, and the data goes to a separate system (a broker). If all three aren’t present, there’s a plainer solution.
\nThe three pieces come down to one discipline: write the data to a single system, suppress repeats at the consumer, seal the sensitive field before it leaves. The outbox makes trust in the queue unnecessary, idempotent consumption makes trust in delivery unnecessary, and field encryption makes trust in the payload unnecessary — each moves a guarantee out of the code and into the design.
", "content_text": "The symptom was small: one or two invoices a day sat in the database but the consumer never saw them. The user said \"I issued it,\" and there was no trace on the other side. Always at night, always during a deploy or a network blip. Once I laid the picture out, the cause was clear — the `INSERT` into the database and the `publish` to the queue were two separate systems, with no guarantee between them.\n\nThis post is the systems side of the **transactional outbox** pattern that closed that gap, the **at-least-once** repeats it opened in return, and finally encrypting the personal data that flows through the event at the **field level**. I wrote the decision-chain side as [a log on muhammetsafak.com.tr](https://muhammetsafak.com.tr/en/blog/from-dual-write-to-outbox-idempotent-consumption-and-field-encryption); here I collect the decisions you have to make for the pattern to be durable in production.\n\n## Dual-write: you can't stretch a transaction across two systems\n\nThe crux in one sentence: your local database transaction doesn't cover RabbitMQ. The classic code is dangerous *because it works most of the time*:\n\n```php\nDB::transaction(function () use ($invoice) {\n $invoice->save(); // 1) write to MySQL\n $this->publishToRabbit($invoice); // 2) publish to the queue\n});\n```\n\nThere are two distinct failure modes:\n\n- `save()` succeeds, `publish` drops on a network error → the invoice is in the DB, the event isn't. **Lost event.**\n- `publish` succeeds, then the transaction `rollback`s for another reason → the event went out, no counterpart in the DB. **Phantom event.**\n\n`DB::transaction` doesn't save you, because commit/rollback only wraps the MySQL side; the broker isn't part of that transaction. The practical way to hold two systems in one atomic step is to reduce the write to **a single system**.\n\n## Outbox: write to one system first, let a separate process publish\n\nThe idea is plain: instead of pushing the message straight to the queue, write it as a row into an `outbox` table **inside the same transaction**. The invoice and the event either both exist or both don't, in the same commit — dual-write collapses into a single write.\n\n```sql\nCREATE TABLE outbox (\n id BINARY(16) NOT NULL, -- event id = idempotency key\n aggregate_id BIGINT NOT NULL, -- ordering and partition key\n topic VARCHAR(120) NOT NULL,\n payload JSON NOT NULL,\n status ENUM('pending','published') NOT NULL DEFAULT 'pending',\n created_at DATETIME(6) NOT NULL,\n published_at DATETIME(6) NULL,\n PRIMARY KEY (id),\n KEY idx_dispatch (status, created_at) -- the relay scan goes through this index\n);\n```\n\nThe write is no longer the business code's concern:\n\n```php\nDB::transaction(function () use ($invoice) {\n $invoice->save();\n Outbox::write('invoice.issued', $invoice->id, $this->payload($invoice));\n});\n```\n\nA separate **relay** does the queue push: it reads the `pending` rows, `publish`es them to RabbitMQ, and stamps them `published` on success. The critical fact here: the relay can fall into the \"I published but crashed before stamping it `published`\" state. So while the outbox solves dual-write, it gives you **no free guarantee** — it gives you **at-least-once**. The message isn't lost, but it can repeat. Everything else arranges itself around this fact.\n\n## How do you feed the relay: polling or CDC?\n\nThe relay can see `pending` rows two ways.\n\n**Polling** — the relay scans the table periodically:\n\n```sql\nSELECT id, topic, payload\nFROM outbox\nWHERE status = 'pending'\nORDER BY created_at\nLIMIT 100;\n```\n\nSimple, works on every database, low operational overhead. The cost is two things: **latency** up to the scan interval (a 1s poll = ~1s of queue latency) and an idling query. Without the `idx_dispatch` index this scan gets expensive as the table grows; even with it, you trade poll frequency against DB load. At small-to-medium volume, polling is the **right** answer — tune it frequent enough to keep latency acceptable, sparse enough not to hammer the DB (in practice 200ms–1s).\n\n**CDC** (change data capture) — the relay listens to the database's **WAL/binlog** rather than the table (Debezium is the typical tool). Every `INSERT` into `outbox` turns into an event almost instantly; polling latency and idle scanning disappear. The cost is operational: binlog access, a connector process, one more moving part. I turn CDC on when latency genuinely matters (sub-second) or when volume strains polling; otherwise I count polling's simplicity as an advantage.\n\n> Rule: your latency budget and your operational budget conflict. CDC buys you latency and charges you an infrastructure piece in return. Start with polling; switch to CDC when there's a measured reason.\n\n## Multiple relays: collision-free dispatch with `SKIP LOCKED`\n\nA single relay is a bottleneck and a single point of failure. Running multiple relays over the same table creates a new risk: two of them grab the same row and publish the same message twice. The naive fix is to lock the table — which kills the parallelism.\n\nThe right tool is `SELECT ... FOR UPDATE SKIP LOCKED`. Each relay claims a batch of rows **by locking them**; instead of queuing behind rows another relay has locked, it **skips** them:\n\n```sql\nBEGIN;\n SELECT id, topic, payload\n FROM outbox\n WHERE status = 'pending'\n ORDER BY created_at\n LIMIT 100\n FOR UPDATE SKIP LOCKED; -- skip rows another relay holds, don't wait\n\n -- publish this batch, then:\n UPDATE outbox SET status = 'published', published_at = NOW(6)\n WHERE id IN (...);\nCOMMIT;\n```\n\nWithout `SKIP LOCKED`, the relays queue for the same rows and the parallelism effectively runs serially. With it, each relay pulls a disjoint set and all of them move at once. PostgreSQL and MySQL 8+ support it.\n\nA subtlety: if the relay crashes between `publish` and `UPDATE ... published`, the row stays `pending` and gets **republished**. That's acceptable — we're already at at-least-once; the fix is on the consumer side, below. The dangerous order is the reverse: stamping `published` first and then `publish`ing. A crash there produces a **lost event** — exactly what we ran from. So the order is **fixed**: `publish` first, stamp second.\n\n## Ordering: what you can guarantee, what you can't\n\n\"Let events arrive in the order they were written\" is an intuitive expectation, but global ordering is expensive and usually unnecessary. What you actually need is **per-aggregate** order: the same invoice's `created` event should arrive before its `updated`; the order of two different invoices relative to each other is nobody's concern.\n\nYou get this by partitioning on `aggregate_id`: events with the same `aggregate_id` go to the same queue partition (or, in RabbitMQ, the same queue via a consistent-hash exchange), and a **single consumer** processes that partition in order. Different aggregates flow in parallel.\n\n```\noutbox (in created_at order)\n │ hash(aggregate_id) % N\n ├── partition 0 ──► consumer-0 (events for aggregates A,D — ordered)\n ├── partition 1 ──► consumer-1 (events for aggregate B — ordered)\n └── partition 2 ──► consumer-2 (events for aggregate C — ordered)\n```\n\nTwo things still break ordering, and the design has to be ready for them: multiple relays under `SKIP LOCKED` can publish rows in a different order, and at-least-once repeats can cut in. So rather than leaning on a global order, you write the consumer to be resilient to **out-of-order and duplicated** events. The practical shield: put a monotonic `version`/`sequence` on the event and drop the stale one (lower version) at the consumer. If you genuinely need strict global ordering you drop to a single partition + single consumer — which then ties throughput to that one consumer; most systems don't want to pay that.\n\n## Once you have at-least-once: the consumer must be idempotent\n\nBecause the relay can republish, you have to assume every event arrives at least once, sometimes more. The fix isn't to make the event unique — it's to set up the **consumer** so that processing the same event twice has the same effect as once. Write the `id` of every processed event into a dedupe table, and do the work **inside the same transaction**:\n\n```php\nDB::transaction(function () use ($event) {\n $inserted = DB::table('processed_messages')->insertOrIgnore([\n 'message_id' => $event['id'],\n 'processed_at' => now(),\n ]);\n\n if ($inserted === 0) {\n return; // it's a repeat; produce no side effects\n }\n\n $this->applyBusinessEffect($event); // the real work — exactly once\n});\n```\n\nThe **unique** constraint on `processed_messages.message_id` is the heart of it: a second arrival of the same `id` is silently dropped by `insertOrIgnore`. Putting the business effect in the same transaction is mandatory — otherwise the \"I processed it but crashed before stamping\" gap opens. Where the dedupe key comes from, and designing a dedupe that survives a crash, is a topic of its own; I covered it in [a separate note](/en/notes/idempotency-duplicate-delivery).\n\nIf the side effect is an external system (an e-invoice integrator, a payment API), dedupe alone isn't enough — because the side effect is outside the transaction. There you have to **carry the key to the external service**: pass the same `id` as an idempotency key to the integrator, or ask \"does this already exist?\" before sending. I worked through that boundary in detail in [the JIT reservation post](/en/systems/race-conditions-and-gaps-in-sequential-numbering), where sealing the number before the external call turns recovery into a lookup.\n\n## `processed_messages` can't grow forever\n\nThe dedupe table accumulates a row per event; if you don't prune it, it becomes a performance problem of its own. The key observation: you don't need to keep an event `id` forever — only as long as the **window in which a repeat could arrive**. Relay retries and broker redeliveries are on the order of hours, not days.\n\nIn practice I pick a fixed **retention window** (say 7 days — comfortably above the longest possible redelivery) and prune older rows:\n\n```sql\nDELETE FROM processed_messages\nWHERE processed_at < NOW() - INTERVAL 7 DAY\nLIMIT 10000; -- small batches at a time, keeping replica lag and locks in check\n```\n\nIf `message_id` is a UUID the table and its index grow; pruning keeps that bounded. When choosing the window, frame the rule backwards: *how late could the latest redelivery arrive?* Make a safe upper bound of that your retention. Set the window too short and a late repeat misses the dedupe, so the side effect runs a second time.\n\n## Why exactly-once is usually an illusion\n\nIn a distributed system \"exactly-once delivery\" is tempting but can't be guaranteed end to end. The reason is the same dual-write, at the broker boundary: if the consumer did the work but crashed before getting its `ack` to the broker, the broker redelivers the message — because it never saw the `ack`. \"I processed it\" and \"I reported that I processed it\" are two separate steps, and a crash can fall between them. This is the practical face of the Two Generals problem.\n\nSo the goal isn't exactly-once **delivery**, it's exactly-once **effect**. And you get that by combining two cheap guarantees:\n\n```\nat-least-once delivery + idempotent consumer = effectively-once\n```\n\nThat is, you stop trying to deliver exactly once and instead make the **repeat harmless**. Under systems that market \"exactly-once,\" this is usually exactly what's there: at-least-once plus a dedupe layer. Look for the guarantee in the effect, not in the delivery.\n\n## Personal data flowing through the event: field-level encryption\n\nOnce the outbox settles in, a new surface appears: the events' payload carries personal data — customer name, tax ID, address — and that payload now sits in a **durable** table (`outbox`) and passes through the broker. If I leave the data as-is, I'm copying GDPR/KVKK-scoped fields, unencrypted, into more than one place.\n\nI didn't want to encrypt the whole payload — I needed fields like `topic`, `aggregate_id`, and `invoice_id` in the clear for relay routing and observability. The need is **field-level** encryption: only the sensitive fields encrypted, the rest in the clear. I marked the sensitive fields in the schema with `x-gdpr-sensitive` and encrypted only those at serialization time:\n\n```php\n$schema = [\n 'invoice_id' => ['x-gdpr-sensitive' => false],\n 'customer_name' => ['x-gdpr-sensitive' => true],\n 'tax_id' => ['x-gdpr-sensitive' => true],\n 'total' => ['x-gdpr-sensitive' => false],\n];\n\n$payload = collect($raw)->map(fn ($value, $field) =>\n ($schema[$field]['x-gdpr-sensitive'] ?? false)\n ? Crypt::encryptString((string) $value) // only the sensitive field is encrypted\n : $value\n)->all();\n```\n\nThat the schema declares the sensitive field is no accident — combine it with schema validation at the edge of the queue and both questions (\"which field is personal data, and is it the right type?\") get answered [at the edge](/en/notes/validate-schema-at-the-edge). The real engineering decision isn't the encryption itself, it's the three questions around it:\n\n**Key management and rotation.** Laravel's `Crypt` uses AES-256 keyed by `APP_KEY`. Bind yourself to a single key and rotation becomes a nightmare: the new key can't open outbox rows written with the old one. The fix is to store a **key version** (key id) alongside the encrypted value and keep a keyring that recognizes several keys at once — new writes use the current key, old values decrypt with their own version. In Laravel, `key` + `previous_keys` in `config/app.php` is exactly this; decryption tries the older keys in turn. That's why you can rotate without migrating data.\n\n**Searching an encrypted field.** `Crypt::encryptString` produces different ciphertext on every call (a random IV), which is correct for security but makes `WHERE tax_id = ?` impossible. If you need equality search you add a **blind index**: keep a deterministic HMAC of the searchable field in a separate column and query that. The encrypted column carries confidentiality, the blind index carries searchability; they're separate columns because they do two separate jobs.\n\n**Losing the key is losing the data.** Field encryption turns `APP_KEY` into an **availability** dependency: lose the key and the encrypted fields are permanent garbage. Key backup and access control are as much a part of the design as the encryption itself.\n\n## When do you not build this pattern?\n\nOutbox + idempotent consumption + field encryption isn't cheap: it brings a table, a relay process (or a CDC connector), a dedupe table and its pruning, plus key management. If you don't know what you're buying in return, don't pay for it. I don't build it when:\n\n- **Event loss is tolerable.** If what you publish is a best-effort notification (cache invalidation, a \"new content available\" signal) and losing one or two doesn't matter, a direct `publish` is enough. You build the outbox because loss matters.\n- **The write and the publish are in the same system.** If your target is the same database (there's no separate broker), there's no dual-write; you don't need an outbox.\n- **Volume fits a single consumer and ordering is critical.** At low volume a single-consumer queue gives you both order and uniqueness simply; building the multi-relay-and-partition machinery is solving a problem you don't have.\n\nThe need for the pattern starts when three conditions arise **together**: event loss is unacceptable, production/consumption is parallel, and the data goes to a separate system (a broker). If all three aren't present, there's a plainer solution.\n\n---\n\nThe three pieces come down to one discipline: **write the data to a single system, suppress repeats at the consumer, seal the sensitive field before it leaves.** The outbox makes trust in the queue unnecessary, idempotent consumption makes trust in delivery unnecessary, and field encryption makes trust in the payload unnecessary — each moves a guarantee out of the code and into the design.", "date_published": "2026-06-22T00:00:00.000Z", "tags": [ "architecture", "messaging", "idempotency", "reliability", "security", "System" ], "authors": [ { "name": "Muhammet Şafak", "url": "https://www.muhammetsafak.com.tr" } ] }, { "id": "https://sade.dev/en/notes/when-event-driven-architecture/", "url": "https://sade.dev/en/notes/when-event-driven-architecture/", "title": "Event-Driven Architecture: What It Solves and Hides", "summary": "Events buy loose coupling by letting the producer stay unaware of its consumers, and the same move makes the flow invisible: no compiler verifies an event contract the way it verifies a method call. Two costs are paid up front or not at all — versioned, schema-validated payloads, and a correlation id carried into every log and every downstream event. A single-listener event is a method call in disguise.", "content_html": "I once watched a team where no one could give a straight answer to “what happens when an order is created?” In the codebase, eight separate listeners were subscribed to the OrderCreated event; you couldn’t read from any single place which one ran in which order, under which condition. The flow didn’t live in the code — it lived in people’s heads, and incompletely at that.
Event-driven architecture loosens coupling. This post is a reminder that something else loosens along with it: causality.
\nCode that publishes an event doesn’t know who listens to it. OrderCreated is published; billing, notifications, and analytics each listen to it separately. The order code is unaware of all three.
The benefit is real: adding a new consumer happens without touching the producer. Independent development, independent deployment, fan-out. Parts that don’t know each other.
\nThe flip side of the same coin: as coupling loosens, the flow becomes invisible.
\nIn a direct method call, the flow is the code — you go to the definition and read on. Not so with an event. To see what happens when OrderCreated is published, you have to find every listener by hand. The compiler won’t help you: a compiler verifies the contract of a method call, but no one verifies the contract of an event.
The line from the modular monolith post applies here too — like a network contract, an event contract sits in the compiler’s blind spot.
\nBefore moving to an event-driven architecture, you pay two costs up front.
\nSchema discipline. An event is a contract. When the payload of OrderCreated changes, the eight places listening to it won’t know — not until it breaks in production. Events must be versioned, fields must be added in a backward-compatible way, payloads must be validated against a schema. Without this discipline, loose coupling means “silent breakage.”
Traceability. To be able to see a request travel through the system, every event must carry a correlation ID:
\nevent(new OrderCreated(\n orderId: $order->id,\n correlationId: request()->header('X-Correlation-Id') ?? (string) Str::uuid(),\n));If this ID isn’t carried into the logs and into subsequent events, then in a multi-listener flow there is no answer to “why did this job run?” Tracing isn’t an optional add-on to event-driven; it’s a prerequisite.
\nLaravel’s event()/listener mechanism, when it runs synchronously, is really an organized method call — same process, same transaction, same stack trace. This is safe and traceable; use it freely.
The difficulties event-driven architecture brings start when an event crosses a boundary: into a queue, into a message broker, into another process. The moment you add ShouldQueue to a listener and send the event to RabbitMQ, the schema and tracing debt kicks in. Don’t conflate the two: using in-process listeners does not move you into “event-driven architecture.”
Actually carrying an event across a boundary is justified in these cases:
\nWithout these, a direct service call is both more readable and safer. An event with a single listener is just a method call in disguise — and a harder one to trace, at that.
\nEvent-driven architecture loosens coupling; in doing so, it loosens causality too. If you want the first, pay the bill for the second — schema and tracing — up front.
\nLoose coupling isn’t free; you pay for it with invisible flow.
", "content_text": "I once watched a team where no one could give a straight answer to \"what happens when an order is created?\" In the codebase, eight separate listeners were subscribed to the `OrderCreated` event; you couldn't read from any single place which one ran in which order, under which condition. The flow didn't live in the code — it lived in people's heads, and incompletely at that.\n\nEvent-driven architecture loosens coupling. This post is a reminder that something else loosens along with it: causality.\n\n## What do events solve?\n\nCode that publishes an event doesn't know who listens to it. `OrderCreated` is published; billing, notifications, and analytics each listen to it separately. The order code is unaware of all three.\n\nThe benefit is real: adding a new consumer happens without touching the producer. Independent development, independent deployment, fan-out. Parts that don't know each other.\n\n## What do events hide?\n\nThe flip side of the same coin: as coupling loosens, **the flow becomes invisible**.\n\nIn a direct method call, the flow *is* the code — you go to the definition and read on. Not so with an event. To see what happens when `OrderCreated` is published, you have to find every listener by hand. The compiler won't help you: a compiler verifies the contract of a method call, but no one verifies the contract of an event.\n\nThe line from [the modular monolith post](/en/journal/why-i-start-with-a-modular-monolith) applies here too — like a network contract, an event contract sits in the compiler's blind spot.\n\n## Don't choose it before paying: two prerequisites\n\nBefore moving to an event-driven architecture, you pay two costs up front.\n\n**Schema discipline.** An event is a contract. When the payload of `OrderCreated` changes, the eight places listening to it won't know — not until it breaks in production. Events must be versioned, fields must be added in a backward-compatible way, payloads must be validated against a schema. Without this discipline, loose coupling means \"silent breakage.\"\n\n**Traceability.** To be able to see a request travel through the system, every event must carry a correlation ID:\n\n```php\nevent(new OrderCreated(\n orderId: $order->id,\n correlationId: request()->header('X-Correlation-Id') ?? (string) Str::uuid(),\n));\n```\n\nIf this ID isn't carried into the logs and into subsequent events, then in a multi-listener flow there is no answer to \"why did this job run?\" Tracing isn't an optional add-on to event-driven; it's a prerequisite.\n\n## In Laravel: in-process events ≠ event-driven architecture\n\nLaravel's `event()`/listener mechanism, when it runs synchronously, is really an organized method call — same process, same transaction, same stack trace. This is safe and traceable; use it freely.\n\nThe difficulties event-driven architecture brings start when an event crosses a **boundary**: into a queue, into a message broker, into another process. The moment you add `ShouldQueue` to a listener and send the event to RabbitMQ, the schema and tracing debt kicks in. Don't conflate the two: using in-process listeners does not move you into \"event-driven architecture.\"\n\n## When should you move to event-driven?\n\nActually carrying an event across a boundary is justified in these cases:\n\n- **There's a real fan-out.** A single event is listened to by many independent consumers that shouldn't know about each other.\n- **The consumer must be asynchronous.** The work shouldn't keep the producer waiting for a response; \"accepted\" is enough.\n- **The consumers must scale or be distributed independently.** This also brings us to [the microservices decision](/en/journal/when-i-move-to-microservices) — the same measured-signal threshold.\n\nWithout these, a direct service call is both more readable and safer. An event with a single listener is just a method call in disguise — and a harder one to trace, at that.\n\n---\n\nEvent-driven architecture loosens coupling; in doing so, it loosens causality too. If you want the first, pay the bill for the second — schema and tracing — up front.\n\nLoose coupling isn't free; you pay for it with invisible flow.", "date_published": "2026-06-20T00:00:00.000Z", "tags": [ "architecture", "event-driven", "messaging", "Note" ], "authors": [ { "name": "Muhammet Şafak", "url": "https://www.muhammetsafak.com.tr" } ] }, { "id": "https://sade.dev/en/systems/gpu-finops-with-ebpf/", "url": "https://sade.dev/en/systems/gpu-finops-with-ebpf/", "title": "GPU FinOps with eBPF", "summary": "You are billed for a whole GPU and you use 3% of it — and your monitoring cannot tell you whose 3% it was. eBPF attributes GPU work to a pod by watching the syscall boundary; DCGM measures how well the silicon ran. They answer different questions, and a real chargeback needs both.", "content_html": "A GPU node costs more per hour than a rack of CPU machines, and the question finance asks about it is the same question they ask about everything: who used this, and how much? For CPU, memory, and disk you can answer that — cgroups account it per process, cAdvisor reports it per pod, Prometheus rolls it up per team. For the GPU, the honest answer in most clusters is that nobody knows. You are paying for the most expensive resource in the fleet and you cannot attribute a dollar of it.
\nThis is not a tooling gap you forgot to fill. It is structural: the GPU is outside the accounting plane the rest of your stack relies on. This piece is about why, and about using eBPF to get attribution back — along with a hard line about what eBPF can and cannot see, because the most expensive mistake here is believing one tool answers a question it physically cannot.
\nThree separate blindnesses stack up.
\ncgroups don’t account the GPU. A cgroup is a kernel construct that accounts and limits CPU, memory, block I/O, and PIDs. The GPU is none of those. Compute and framebuffer (VRAM) are managed by the NVIDIA kernel driver and the userspace CUDA runtime, and are opaque to cgroup accounting. There is a device cgroup controller, but it only gates access to the device nodes (/dev/nvidia0 and friends) — allow or deny. It does not account a single second of GPU time or a megabyte of VRAM. So the entire cAdvisor → Prometheus pipeline that gives you per-pod CPU and memory has, by construction, nothing to say about the GPU.
nvidia.com/gpu is an allocation count, not a utilization signal. Kubernetes learns about GPUs through the NVIDIA device plugin, which registers them to the kubelet as the extended resource nvidia.com/gpu. Pods request it as an integer — whole GPUs. The scheduler matches requested count to available count. That is the only thing the allocation layer knows. Requesting a GPU is not using a GPU. A pod can hold a whole device — and be billed for the whole device — while driving it at 3%. The allocation number you could charge back against is precisely the number that tells you nothing about consumption.
Even nvidia-smi’s utilization number lies to you. Reach for nvidia-smi and the utilization.gpu field looks like salvation. It is not what you think. NVML defines it as the percent of time over the sample period during which one or more kernels was executing. It measures temporal presence — was the GPU busy at all — not how much of it was busy. A single-thread kernel occupying one SM out of dozens can report close to 100%. Microsoft has reported under 10% compute utilization during the memory-bound decode phase of serving an 8B-parameter model on A100s — while a naive reading of “utilization” would call those GPUs full. If you charge back on utilization.gpu, you are billing on a number that says “busy” when the silicon is nearly idle.
So: the allocation layer knows count-not-use, cgroups know nothing, and the one easy percentage is measuring the wrong thing. That is the gap.
\nBefore any tooling, separate the two questions, because conflating them is the core error — the same shape as confusing the problem a tool solves with the problem at hand.
\nThese map onto two different measurement planes, and the rule that organizes everything below is: eBPF answers attribution; the GPU’s own counters answer efficiency. Neither substitutes for the other.
\neBPF runs in the kernel and attaches to kprobes, uprobes, tracepoints, and syscalls. There is no GPU-utilization tracepoint to read — but there are two surfaces where GPU work is requested, and both are visible from the kernel:
\nioctl() to /dev/nvidiactl and /dev/nvidia0…N. A kprobe on the driver’s entry points (nvidia_unlocked_ioctl, nvidia_open) sees that traffic. (The closed driver historically exposes only one tracepoint, nvidia:nvidia_dev_xid, for hardware error events; everything else is kprobed.)libcuda.so / libcudart.so traces the API itself: cuLaunchKernel, cuMemAlloc, cuMemcpyHtoD, cuStreamSynchronize, and friends. Pairing an entry uprobe with a return uretprobe measures each call’s duration.The reason this is attribution is that at every hook point eBPF reads the calling PID/TGID and cgroup natively — bpf_get_current_pid_tgid and the cgroup id are right there. That gives you the chain PID → cgroup → pod → namespace → team with no application changes and no cooperation from NVIDIA. Per pod you get: kernel-launch counts, launch dimensions, VRAM allocation sizes, memcpy volume and direction, and call timing. That is a real, defensible chargeback signal derived entirely from the kernel side of the boundary.
The ecosystem here is young but real. The primitives work today — there are working write-ups and tutorials uprobing the CUDA libraries and kprobing the nvidia ioctl path, and bpftime explores tying eBPF logic to GPU events. Tetragon (Cilium) ships generic process_uprobe, process_kprobe, and ioctl tracing with Kubernetes pod identity, so it can be pointed at the CUDA symbols or the ioctl path — but understand that this is a TracingPolicy you author, not a shipped “GPU FinOps” feature. There is no dominant turnkey eBPF chargeback product yet. Anyone selling you “drop-in eBPF GPU FinOps” is selling further than the ecosystem currently reaches.
This is the honest limit, and it is not a maturity problem that will be fixed in a release — it is physics of where the data lives.
\neBPF sees the control plane: API calls, ioctls, allocation sizes, launch counts, and submit-to-complete timing where you can trace it. It does not see inside the GPU. It cannot read SM occupancy, tensor-core utilization, or achieved memory bandwidth, because those are hardware performance counters that live on the device and are exposed only through NVML / DCGM / CUPTI. eBPF has no path to them. It can tell you a pod launched ten thousand kernels and allocated 40 GB of VRAM; it cannot tell you whether those kernels saturated the SMs or left them 90% idle.
\nThere is a second, subtler limit. Even at the ioctl boundary eBPF can hook, the payloads are largely opaque. The driver’s command structures (the NV_ESC_* Resource Manager API) are complex and effectively proprietary. You can see that an ioctl happened — its command number, the calling PID, the timing — but decoding the semantic content of an arbitrary RM payload is impractical and fragile. You get the fact of the work and its attribution, not a free reading of its meaning.
A note that closes a tempting door: NVIDIA’s open kernel modules (open-gpu-kernel-modules, Turing and newer) open the kernel interface layer — the module init, the ioctl entry points, the NV_ESC_* command surface. That genuinely helps you understand what to hook. But the GPU’s brain stays closed: on Turing+ much of the management runs on on-GPU GSP firmware, and that firmware — along with the user-mode driver components the modules require — is still shipped closed. Opening the kernel modules does not expose hardware performance counters to eBPF. The on-device limit is unchanged. If someone claims the open modules let eBPF read SM occupancy, they are wrong on current evidence.
Because eBPF cannot see the silicon, the efficiency half of the answer comes from userspace, and the standard tool is NVIDIA DCGM (Data Center GPU Manager) with dcgm-exporter for Prometheus. DCGM reads the GPU’s hardware counters through the driver and exposes the fields eBPF can’t reach:
\nDCGM_FI_PROF_SM_ACTIVE — fraction of time at least one warp was active on a multiprocessor, averaged across all of them.DCGM_FI_PROF_SM_OCCUPANCY — fraction of resident warps relative to the maximum supported: true occupancy.DCGM_FI_PROF_PIPE_TENSOR_ACTIVE — fraction of cycles the tensor pipe was active.DCGM_FI_PROF_DRAM_ACTIVE — a memory-bandwidth proxy.DCGM_FI_DEV_FB_USED — framebuffer (VRAM) actually in use.These are the numbers that tell you whether a team’s expensive allocation was earning its keep. DCGM attributes them to pods through the kubelet Pod Resources API (a gRPC service at /var/lib/kubelet/pod-resources), which maps each GPU UUID to the pod that holds it.
But DCGM’s attribution has a hard edge that is exactly where eBPF earns its place: DCGM produces device-level metrics, so it cannot disambiguate consumers sharing one physical GPU. Under time-slicing or MPS (Multi-Process Service), all pods sharing a GPU receive identical, duplicated device-level values — including DCGM_FI_DEV_FB_USED. NVIDIA’s own exporter documents that it does not associate metrics to containers when time-slicing is enabled; with --kubernetes-virtual-gpus=true, every sharing pod mirrors the whole physical GPU’s state. Under MIG, attribution shifts to the GPU-instance level, not arbitrary pods. So precisely in the shared-GPU case — the case that exists because whole-GPU allocation wastes money — DCGM cannot tell you who consumed what. eBPF’s PID-level tracing can. The two tools are complementary at exactly the seam where each is weakest.
(For the deepest efficiency profiling there is CUPTI, the CUDA Profiling Tools Interface, which can read counters DCGM doesn’t surface — but metric-replay profiling like Nsight Compute re-runs kernels multiple times and carries heavy overhead. It is a profiling-session tool, not always-on per-tenant telemetry. DCGM is the lower-overhead, sampling-based continuous path; CUPTI is the heavy, deep one.)
\n| Plane | Tool | Sees | Cannot see |
|---|---|---|---|
| Kernel (control) | eBPF | PID→pod attribution, launch counts, alloc sizes, memcpy volume, call timing — including per-pod under time-slicing | on-device SM/tensor occupancy, memory bandwidth, ioctl payload internals |
| Userspace (device) | DCGM | true SM occupancy, tensor activity, DRAM activity, VRAM used — from hardware counters | per-pod attribution under shared GPU (time-slicing/MPS): all sharers get identical values |
eBPF is low-overhead, needs no app changes, needs no vendor cooperation, and attributes natively by PID/cgroup — it is the right tool for who and how much. DCGM reads the silicon — it is the right tool for how well. The costs are honest too: uprobes carry real overhead and the CUDA symbols are versioned (cuMemAlloc@CUDA_11.0 vs @CUDA_12.0), so probes need dynamic symbol resolution and break across driver upgrades; CUDA API calls can fire ten-thousand-plus times a second, so handlers must stay cheap or they slow the very workload they measure. DCGM is userspace polling with its own sampling overhead and a hardware constraint that only certain counter groups can be read together.
The wrong turn is picking one tool and asking it the other tool’s question — billing teams on DCGM’s device-level numbers and silently overcharging everyone who shares a GPU, or trusting eBPF’s launch counts as a proxy for efficiency and concluding a busy-looking workload was well-utilized. The order that actually works:
\n1. eBPF: attribute GPU work to pod/team (who, how much — the bill)\n2. DCGM: measure on-device efficiency (how well — was it justified)\n3. Join them on pod identity (the team's spend AND its efficiency)Attribution first, because that is the question finance actually asked and the one your existing stack cannot answer at all. Efficiency second, because it turns the bill into a decision — a team holding an expensive allocation at 4% occupancy is paying a just-in-case capacity tax you can now see, where before the GPU was a flat line item nobody could open. Joined on pod identity, you finally have what you have for every other resource: spend, attributed, with the efficiency context to act on it. The reason it took two planes is the same reason it was invisible — the silicon never lived in the accounting plane, and no single probe spans the boundary.
\nSee also: the DCGM exporter docs cover the pod-attribution path and the DCGM feature overview defines the profiling fields, and the eBPF primitives this leans on are documented at ebpf.io and in Tetragon.
", "content_text": "A GPU node costs more per hour than a rack of CPU machines, and the question finance asks about it is the same question they ask about everything: **who used this, and how much?** For CPU, memory, and disk you can answer that — cgroups account it per process, cAdvisor reports it per pod, Prometheus rolls it up per team. For the GPU, the honest answer in most clusters is that nobody knows. You are paying for the most expensive resource in the fleet and you cannot attribute a dollar of it.\n\nThis is not a tooling gap you forgot to fill. It is structural: the GPU is **outside the accounting plane the rest of your stack relies on.** This piece is about why, and about using eBPF to get attribution back — along with a hard line about what eBPF can and cannot see, because the most expensive mistake here is believing one tool answers a question it physically cannot.\n\n## Why is the GPU invisible to your existing metrics?\n\nThree separate blindnesses stack up.\n\n**cgroups don't account the GPU.** A cgroup is a kernel construct that accounts and limits CPU, memory, block I/O, and PIDs. The GPU is none of those. Compute and framebuffer (VRAM) are managed by the NVIDIA kernel driver and the userspace CUDA runtime, and are opaque to cgroup accounting. There is a `device` cgroup controller, but it only *gates access* to the device nodes (`/dev/nvidia0` and friends) — allow or deny. It does not account a single second of GPU time or a megabyte of VRAM. So the entire cAdvisor → Prometheus pipeline that gives you per-pod CPU and memory has, by construction, **nothing to say about the GPU.**\n\n**`nvidia.com/gpu` is an allocation count, not a utilization signal.** Kubernetes learns about GPUs through the NVIDIA device plugin, which registers them to the kubelet as the extended resource `nvidia.com/gpu`. Pods request it as an integer — whole GPUs. The scheduler matches requested *count* to available *count*. That is the only thing the allocation layer knows. **Requesting a GPU is not using a GPU.** A pod can hold a whole device — and be billed for the whole device — while driving it at 3%. The allocation number you could charge back against is precisely the number that tells you nothing about consumption.\n\n**Even `nvidia-smi`'s utilization number lies to you.** Reach for `nvidia-smi` and the `utilization.gpu` field looks like salvation. It is not what you think. NVML defines it as *the percent of time over the sample period during which one or more kernels was executing.* It measures **temporal presence — was the GPU busy at all — not how much of it was busy.** A single-thread kernel occupying one SM out of dozens can report close to 100%. Microsoft has reported under 10% compute utilization during the memory-bound decode phase of serving an 8B-parameter model on A100s — while a naive reading of \"utilization\" would call those GPUs full. If you charge back on `utilization.gpu`, you are billing on a number that says \"busy\" when the silicon is nearly idle.\n\nSo: the allocation layer knows count-not-use, cgroups know nothing, and the one easy percentage is measuring the wrong thing. That is the gap.\n\n## Two questions, and why they need different tools\n\nBefore any tooling, separate the two questions, because conflating them is the core error — the same shape as [confusing the problem a tool solves with the problem at hand](/en/systems/data-intensive-systems-breaking-points).\n\n- **Attribution — *who* did GPU work, and *how much*?** This is the FinOps question. It needs per-pod, per-team accounting of GPU activity: which workload launched kernels, allocated VRAM, moved data, and for how long. This is what chargeback runs on.\n- **Efficiency — *how well* was the silicon used?** Were the SMs actually occupied, were the tensor cores active, was memory bandwidth the bottleneck? This is the performance question. It tells you whether a team's spend was *justified*, but it cannot tell you *whose* spend it was.\n\nThese map onto two different measurement planes, and the rule that organizes everything below is: **eBPF answers attribution; the GPU's own counters answer efficiency. Neither substitutes for the other.**\n\n## What eBPF can see: the control plane\n\neBPF runs in the kernel and attaches to kprobes, uprobes, tracepoints, and syscalls. There is no GPU-utilization tracepoint to read — but there are two surfaces where GPU *work is requested*, and both are visible from the kernel:\n\n- **The driver ioctl boundary.** Every piece of GPU work — kernel submission, memory allocation, synchronization — ultimately becomes an `ioctl()` to `/dev/nvidiactl` and `/dev/nvidia0…N`. A kprobe on the driver's entry points (`nvidia_unlocked_ioctl`, `nvidia_open`) sees that traffic. (The closed driver historically exposes only one tracepoint, `nvidia:nvidia_dev_xid`, for hardware error events; everything else is kprobed.)\n- **The CUDA library boundary.** A uprobe on `libcuda.so` / `libcudart.so` traces the API itself: `cuLaunchKernel`, `cuMemAlloc`, `cuMemcpyHtoD`, `cuStreamSynchronize`, and friends. Pairing an entry uprobe with a return uretprobe measures each call's duration.\n\nThe reason this is *attribution* is that at every hook point eBPF reads the calling PID/TGID and cgroup natively — `bpf_get_current_pid_tgid` and the cgroup id are right there. That gives you the chain **PID → cgroup → pod → namespace → team** with no application changes and no cooperation from NVIDIA. Per pod you get: kernel-launch counts, launch dimensions, VRAM allocation sizes, memcpy volume and direction, and call timing. That is a real, defensible chargeback signal derived entirely from the kernel side of the boundary.\n\nThe ecosystem here is young but real. The primitives work today — there are working write-ups and tutorials uprobing the CUDA libraries and kprobing the nvidia ioctl path, and `bpftime` explores tying eBPF logic to GPU events. Tetragon (Cilium) ships generic `process_uprobe`, `process_kprobe`, and ioctl tracing with Kubernetes pod identity, so it *can* be pointed at the CUDA symbols or the ioctl path — but understand that this is a `TracingPolicy` you author, not a shipped \"GPU FinOps\" feature. There is no dominant turnkey eBPF chargeback product yet. Anyone selling you \"drop-in eBPF GPU FinOps\" is selling further than the ecosystem currently reaches.\n\n## What eBPF cannot see: the silicon\n\nThis is the honest limit, and it is not a maturity problem that will be fixed in a release — it is physics of where the data lives.\n\neBPF sees the **control plane**: API calls, ioctls, allocation sizes, launch counts, and submit-to-complete timing where you can trace it. It does **not** see inside the GPU. It cannot read SM occupancy, tensor-core utilization, or achieved memory bandwidth, because those are hardware performance counters that live *on the device* and are exposed only through NVML / DCGM / CUPTI. eBPF has no path to them. It can tell you a pod launched ten thousand kernels and allocated 40 GB of VRAM; it cannot tell you whether those kernels saturated the SMs or left them 90% idle.\n\nThere is a second, subtler limit. Even at the ioctl boundary eBPF can hook, the *payloads* are largely opaque. The driver's command structures (the `NV_ESC_*` Resource Manager API) are complex and effectively proprietary. You can see *that* an ioctl happened — its command number, the calling PID, the timing — but decoding the semantic content of an arbitrary RM payload is impractical and fragile. You get the fact of the work and its attribution, not a free reading of its meaning.\n\nA note that closes a tempting door: NVIDIA's open kernel modules (`open-gpu-kernel-modules`, Turing and newer) open the **kernel interface layer** — the module init, the ioctl entry points, the `NV_ESC_*` command surface. That genuinely helps you understand *what* to hook. But the GPU's brain stays closed: on Turing+ much of the management runs on on-GPU GSP firmware, and that firmware — along with the user-mode driver components the modules require — is still shipped closed. Opening the kernel modules **does not** expose hardware performance counters to eBPF. The on-device limit is unchanged. If someone claims the open modules let eBPF read SM occupancy, they are wrong on current evidence.\n\n## The userspace path you still need: DCGM\n\nBecause eBPF cannot see the silicon, the efficiency half of the answer comes from userspace, and the standard tool is NVIDIA **DCGM** (Data Center GPU Manager) with **dcgm-exporter** for Prometheus. DCGM reads the GPU's hardware counters through the driver and exposes the fields eBPF can't reach:\n\n- `DCGM_FI_PROF_SM_ACTIVE` — fraction of time at least one warp was active on a multiprocessor, averaged across all of them.\n- `DCGM_FI_PROF_SM_OCCUPANCY` — fraction of resident warps relative to the maximum supported: *true* occupancy.\n- `DCGM_FI_PROF_PIPE_TENSOR_ACTIVE` — fraction of cycles the tensor pipe was active.\n- `DCGM_FI_PROF_DRAM_ACTIVE` — a memory-bandwidth proxy.\n- `DCGM_FI_DEV_FB_USED` — framebuffer (VRAM) actually in use.\n\nThese are the numbers that tell you whether a team's expensive allocation was earning its keep. DCGM attributes them to pods through the kubelet **Pod Resources API** (a gRPC service at `/var/lib/kubelet/pod-resources`), which maps each GPU UUID to the pod that holds it.\n\nBut DCGM's attribution has a hard edge that is exactly where eBPF earns its place: **DCGM produces device-level metrics, so it cannot disambiguate consumers sharing one physical GPU.** Under time-slicing or MPS (Multi-Process Service), all pods sharing a GPU receive *identical, duplicated* device-level values — including `DCGM_FI_DEV_FB_USED`. NVIDIA's own exporter documents that it does not associate metrics to containers when time-slicing is enabled; with `--kubernetes-virtual-gpus=true`, every sharing pod mirrors the whole physical GPU's state. Under MIG, attribution shifts to the GPU-instance level, not arbitrary pods. So precisely in the shared-GPU case — the case that exists *because* whole-GPU allocation wastes money — DCGM cannot tell you who consumed what. eBPF's PID-level tracing can. The two tools are complementary at exactly the seam where each is weakest.\n\n(For the deepest efficiency profiling there is **CUPTI**, the CUDA Profiling Tools Interface, which can read counters DCGM doesn't surface — but metric-replay profiling like Nsight Compute re-runs kernels multiple times and carries heavy overhead. It is a profiling-session tool, not always-on per-tenant telemetry. DCGM is the lower-overhead, sampling-based continuous path; CUPTI is the heavy, deep one.)\n\n## The kernel-vs-userspace tradeoff, stated plainly\n\n| Plane | Tool | Sees | Cannot see |\n| --- | --- | --- | --- |\n| Kernel (control) | eBPF | PID→pod attribution, launch counts, alloc sizes, memcpy volume, call timing — *including* per-pod under time-slicing | on-device SM/tensor occupancy, memory bandwidth, ioctl payload internals |\n| Userspace (device) | DCGM | true SM occupancy, tensor activity, DRAM activity, VRAM used — from hardware counters | per-pod attribution under shared GPU (time-slicing/MPS): all sharers get identical values |\n\neBPF is low-overhead, needs no app changes, needs no vendor cooperation, and attributes natively by PID/cgroup — it is the right tool for *who and how much*. DCGM reads the silicon — it is the right tool for *how well*. The costs are honest too: uprobes carry real overhead and the CUDA symbols are **versioned** (`cuMemAlloc@CUDA_11.0` vs `@CUDA_12.0`), so probes need dynamic symbol resolution and break across driver upgrades; CUDA API calls can fire ten-thousand-plus times a second, so handlers must stay cheap or they slow the very workload they measure. DCGM is userspace polling with its own sampling overhead and a hardware constraint that only certain counter groups can be read together.\n\n## Why you need both, in order\n\nThe wrong turn is picking one tool and asking it the other tool's question — billing teams on DCGM's device-level numbers and silently overcharging everyone who shares a GPU, or trusting eBPF's launch counts as a proxy for efficiency and concluding a busy-looking workload was well-utilized. The order that actually works:\n\n```\n1. eBPF: attribute GPU work to pod/team (who, how much — the bill)\n2. DCGM: measure on-device efficiency (how well — was it justified)\n3. Join them on pod identity (the team's spend AND its efficiency)\n```\n\nAttribution first, because that is the question finance actually asked and the one your existing stack cannot answer at all. Efficiency second, because it turns the bill into a decision — a team holding an expensive allocation at 4% occupancy is paying a [just-in-case capacity tax](/en/journal/the-cost-of-just-in-case-code) you can now *see*, where before the GPU was a flat line item nobody could open. Joined on pod identity, you finally have what you have for every other resource: spend, attributed, with the efficiency context to act on it. The reason it took two planes is the same reason it was invisible — the silicon never lived in the accounting plane, and no single probe spans the boundary.\n\n---\n\n*See also:* the [DCGM exporter docs](https://docs.nvidia.com/datacenter/dcgm/latest/installation/install-dcgm-exporter.html) cover the pod-attribution path and the [DCGM feature overview](https://docs.nvidia.com/datacenter/dcgm/latest/user-guide/feature-overview.html) defines the profiling fields, and the eBPF primitives this leans on are documented at [ebpf.io](https://ebpf.io/what-is-ebpf/) and in [Tetragon](https://tetragon.io/).", "date_published": "2026-06-20T00:00:00.000Z", "tags": [ "ebpf", "gpu", "finops", "observability", "kubernetes", "System" ], "authors": [ { "name": "Muhammet Şafak", "url": "https://www.muhammetsafak.com.tr" } ] }, { "id": "https://sade.dev/en/systems/zero-static-authority-multi-cluster-gitops/", "url": "https://sade.dev/en/systems/zero-static-authority-multi-cluster-gitops/", "title": "Zero Static Authority in Multi-Cluster GitOps", "summary": "A stored kubeconfig is a long-lived credential pointed at your whole fleet — the exact thing an attacker wants. Replace it with a workload identity that expires in an hour and is never written to disk. Here is the rule, the order, and the new control plane you take on in return.", "content_html": "A GitOps controller — Argo CD, Flux — sits in one cluster and reconciles many. To reach the others it needs to authenticate to each remote kube-apiserver. The default way to do that is to store a credential: a bearer token, a client certificate, a kubeconfig, kept as a Kubernetes Secret in the control cluster. In Argo CD these are the objects labelled argocd.argoproj.io/secret-type: cluster.
Stop and look at what that Secret is. It is a long-lived credential, usually with broad rights, pointing at your entire fleet, sitting at rest in one place. It does not rotate on its own. It is in your backups. It is the single most valuable thing an attacker can find in your control cluster, because owning it is owning every cluster it can reach. The whole discipline of boring architecture is about not keeping liabilities you don’t have to — and a stored fleet-wide token is a liability you keep purely because issuing a fresh, short-lived one at the moment of use seemed like more work.
This piece is about doing that more work, in order. The target is zero static authority: no long-lived credential that authenticates the controller to a remote cluster is ever written down. The identity the controller presents is minted on demand, expires in about an hour, and lives only in memory. The mechanism is SPIFFE for the identity model and SPIRE for issuing it.
\nBefore any of it, this is not a tool for three clusters you could count on one hand. SPIRE is a control plane you operate, not a library you import — it is exactly the kind of capability you should be slow to take on, the way modular monolith beats microservices until you’ve earned the split. It earns its weight only when you genuinely run a fleet — many clusters, many teams, an audit requirement that a stolen Secret would fail. If a single shared cluster covers you, the cheapest secure credential is the one you never stand up the machinery to issue. The rest of this assumes you have actually arrived at the fleet.
State it before the mechanics: the credential that crosses a cluster boundary must be short-lived and never stored. Everything below is in service of that one sentence. If at the end you still have a long-lived token written to disk somewhere on the authentication path, you have built complexity without buying the property you came for — the same wrong turn as adding a read replica to dodge a missing index.
\n“Short-lived and never stored” has a precise meaning here. The identity is an X.509 certificate whose default lifetime is one hour — or a JWT, whose default is five minutes — fetched by the workload from a local socket with no token of its own to present, and rotated automatically at half its life. There is no secret to steal that is worth stealing tomorrow.
\nSPIFFE is a spec; SPIRE is the reference implementation that issues the things the spec describes. Four nouns carry the whole design.
\nA SPIFFE ID is a URI: spiffe://<trust-domain>/<path>, e.g. spiffe://prod.example.org/ns/argocd/sa/application-controller. The authority part is the trust domain — one logical root of trust, one CA. The path identifies the workload. That is the name a workload proves it holds.
An SVID (SPIFFE Verifiable Identity Document) is the proof. Two forms:
\nsub is the SPIFFE ID and whose aud you scope to the intended verifier. You present it as a bearer token.The lifetimes are the point. In SPIRE’s server config the defaults are default_x509_svid_ttl: 1h and default_jwt_svid_ttl: 5m, and the SVIDs are signed by a CA whose own ca_ttl defaults to 24h. These are short on purpose: a leaked SVID is worthless within the hour, which is exactly why it is safe to use it where you used to store a token.
The thing that makes “never stored” possible is the Workload API, served by a SPIRE agent over a local Unix domain socket (the documented default is /tmp/spire-agent/public/api.sock; many deployments mount it at /run/spire/sockets/...). The workload connects to that socket and asks for its SVID. It presents no credential to do so. The agent identifies the caller by inspecting the calling process through the kernel — its PID, and from that its container, namespace, service account — and performs workload attestation against a set of selectors. Identity is established by what the process verifiably is, not by what secret it holds. That inversion is the entire reason there is nothing to store.
Underneath, SPIRE is two components and two attestations:
\nUpstreamAuthority plugin — a decision you make once and live with.k8s_psat, which validates a projected service-account token through the Kubernetes TokenReview API. No shared secret is planted on the node. Then it does workload attestation for each local process before handing it an SVID.A registration entry is the server-side record that ties a SPIFFE ID to a parent (the node/agent) and a set of selectors (namespace, service account, image). It is the policy: this workload, attested this way, gets this identity. Managing the lifecycle of these entries is real work, and it is where most of the ongoing operational cost lives.
\nOne trust domain is one root of trust. A multi-cluster fleet usually means multiple trust domains — one per cluster, or per region — and now a workload in domain A must be able to validate an SVID minted in domain B. That is federation, and its unit is the trust bundle: the public CA certs and JWT signing keys (a JWKS) that let you verify a domain’s SVIDs.
\nEach domain stands up a bundle endpoint — a URL serving its current bundle, the SPIFFE analogue of OIDC’s jwks_uri — and each peer polls it to stay current. Two profiles, and the difference is exactly the bootstrap question:
https_web — the endpoint is fronted by a Web-PKI certificate from a public CA. The peer validates it with the ordinary public trust store, so no initial bundle has to be exchanged out of band. Trust bootstraps off the existing Web PKI.https_spiffe — the endpoint authenticates with its own X.509-SVID. To talk to it the first time, the peer must already hold that domain’s initial bundle. So the very first bundle has to arrive out of band, after which the latest fetched bundle is used going forward.On a registration entry, federatesWith lists the foreign trust domains a workload is allowed to authenticate against; the agent then delivers those foreign bundles to the workload over the Workload API. Bundles refresh on the spiffe_refresh_hint (commonly around five minutes), and you publish a new signing key several refresh cycles before you start using it, so federated peers have already learned it when the rotation lands.
Note the honest shape of “zero static authority”: federation relocates the bootstrap trust decision, it does not erase it. With https_spiffe you ship one initial bundle by hand; with https_web you lean on the public CA system. Either way the root of trust still comes from somewhere — what you’ve eliminated is the long-lived, broadly-scoped, per-cluster credential, not the one-time trust anchor. That is the right trade, but call it what it is.
Here is where intent meets the Kubernetes API, and where it is easy to be wrong. There are two mechanisms, and only one is clean today.
\nThe JWT-SVID-as-OIDC path (the one that works). Kubernetes can be told to trust an external OIDC issuer. SPIRE ships an OIDC Discovery Provider that serves /.well-known/openid-configuration and a JWKS backed by SPIRE’s JWT-SVID signing keys. You point the remote (spoke) cluster’s apiserver at that provider as an OIDC issuer. The controller fetches a fresh JWT-SVID from its local Workload API and presents it as a bearer token; the apiserver validates it against SPIRE’s JWKS and maps it to a subject. No Secret, no stored token — a new JWT per request, expiring in minutes.
This is not a thought experiment. Red Hat ships SPIFFE/SPIRE support as the Zero Trust Workload Identity Manager operator starting in OpenShift 4.20, and documents exactly this wiring as a how-to guide rather than a shipped OpenShift GitOps feature: Argo CD uses a client-go ExecCredential plugin (apiVersion: client.authentication.k8s.io/v1beta1) via execProviderConfig, which on each call reads the SPIFFE socket (SPIFFE_ENDPOINT_SOCKET), requests a JWT-SVID with the audience the spoke apiserver expects (SPIFFE_JWT_AUDIENCE), and hands it over. The credential is manufactured at the moment of use and discarded.
The mTLS-with-X.509-SVID path (the trap). The instinct is to do straight mTLS: the controller presents its X.509-SVID, the apiserver trusts the SPIFFE bundle as a client CA, done. It is not done. Kubernetes client-cert auth derives the username from the certificate’s Subject CN and groups from the Subject O — and the SPIFFE ID lives in the URI SAN, which apiserver client-cert auth does not read. A raw SPIFFE X.509-SVID therefore does not map to a Kubernetes user. To use mTLS you insert a proxy — Ghostunnel or Envoy — in front of the apiserver: it terminates SPIFFE mTLS, validates the URI-SAN SPIFFE ID, and forwards with an identity the apiserver does understand. Ghostunnel consumes the Workload API directly for its own rotating SVIDs. This works, but it is a moving part in front of every apiserver, and the OIDC path avoids it.
\nOn maturity, be honest. Upstream Argo CD has no native SPIFFE auth; it works through the generic ExecCredential plugin pattern, and the most mature productization is OpenShift’s. Flux can be GitOps-managed to deploy SPIRE, and its OCI registry auth can consume a JWT-SVID, but a first-class “authenticate Flux to a remote cluster via SPIFFE instead of a kubeconfig” feature is not documented — it rides the same OIDC/exec-credential/proxy plumbing. Treat native remote-cluster SPIFFE auth as a pattern you assemble, not a checkbox you enable.
This is the part the architecture diagrams omit. You have removed a stored credential; in return you have made an identity service a hard dependency on the critical path, and short-lived things fail in ways long-lived things didn’t.
\navailability_target knob (must be ≥ 24h if set) makes it rotate early to bank headroom for graceful downtime. But the failure mode is real and new: identity-plane down means fleet auth down.The steps are not a menu; they are a sequence, and skipping it is how teams get the cost without the benefit.
\n1. Stand up SPIRE: server + agents, node attestation (foundation)\n2. Issue workload SVIDs to the controller via Workload API\n3. Make the spoke apiserver trust SPIRE (OIDC path first)\n4. Federate trust domains across clusters\n5. Delete the stored kubeconfig Secrets (the payoff)Step 5 is the whole point, and it is only safe once 1–4 actually work. The common failure is to keep the old Secret “as a fallback” — which means the long-lived fleet-wide credential is still at rest, still in your backups, still the thing an attacker takes, and you are now also running SPIRE. You have paid for the control plane and kept the liability. Either the stored credential is gone or you have not done this; there is no half-credit.
The reverse error is reaching for SPIRE before the order can pay off — standing up a federated identity plane for a couple of clusters whose Secret would be fine. That is just-in-case complexity wearing a security badge: machinery built for a blast radius you don’t yet have. The machinery is justified by the fleet, the audit requirement, and the blast radius of a stolen token. Below that line, the cheapest secure credential really is the boring one you never have to issue.
Where this approach genuinely ends is clear: a fleet large enough that a static per-cluster credential is an unacceptable blast radius, with the operational maturity to run a stateful, HA, on-the-critical-path identity service and keep its clocks in sync. A team that has arrived there already pays for that maturity elsewhere — and at that point zero static authority is not gold-plating, it is the credential model the blast radius was always demanding. Not before.
\nSee also: the SPIFFE and SPIRE docs define the identity model and the server/agent components, and the SPIFFE Federation spec specifies the bundle-endpoint profiles this design relies on.
", "content_text": "A GitOps controller — Argo CD, Flux — sits in one cluster and reconciles many. To reach the others it needs to authenticate to each remote `kube-apiserver`. The default way to do that is to store a credential: a bearer token, a client certificate, a kubeconfig, kept as a Kubernetes `Secret` in the control cluster. In Argo CD these are the objects labelled `argocd.argoproj.io/secret-type: cluster`.\n\nStop and look at what that `Secret` is. It is a **long-lived credential, usually with broad rights, pointing at your entire fleet, sitting at rest in one place.** It does not rotate on its own. It is in your backups. It is the single most valuable thing an attacker can find in your control cluster, because owning it is owning every cluster it can reach. The whole discipline of [boring architecture](/en/journal/why-boring-architecture) is about not keeping liabilities you don't have to — and a stored fleet-wide token is a liability you keep purely because issuing a fresh, short-lived one at the moment of use seemed like more work.\n\nThis piece is about doing that more work, in order. The target is **zero static authority**: no long-lived credential that authenticates the controller to a remote cluster is ever written down. The identity the controller presents is minted on demand, expires in about an hour, and lives only in memory. The mechanism is SPIFFE for the identity model and SPIRE for issuing it.\n\nBefore any of it, this is not a tool for three clusters you could count on one hand. SPIRE is a control plane you operate, not a library you import — it is exactly the kind of capability you should be slow to take on, the way [modular monolith beats microservices until you've earned the split](/en/journal/why-i-start-with-a-modular-monolith). It earns its weight only when you genuinely run a fleet — many clusters, many teams, an audit requirement that a stolen `Secret` would fail. If a single shared cluster covers you, the cheapest secure credential is the one you never stand up the machinery to issue. The rest of this assumes you have actually arrived at the fleet.\n\n## The one rule up front\n\nState it before the mechanics: **the credential that crosses a cluster boundary must be short-lived and never stored.** Everything below is in service of that one sentence. If at the end you still have a long-lived token written to disk somewhere on the authentication path, you have built complexity without buying the property you came for — the same wrong turn as adding a read replica to dodge a missing index.\n\n\"Short-lived and never stored\" has a precise meaning here. The identity is an X.509 certificate whose default lifetime is **one hour** — or a JWT, whose default is five minutes — fetched by the workload from a local socket with **no token of its own to present**, and rotated automatically at half its life. There is no secret to steal that is worth stealing tomorrow.\n\n## What SPIFFE and SPIRE actually are\n\nSPIFFE is a spec; SPIRE is the reference implementation that issues the things the spec describes. Four nouns carry the whole design.\n\nA **SPIFFE ID** is a URI: `spiffe://A downstream API was down for twenty minutes. Four thousand messages exhausted their retries and landed on the dead-letter queue. The API is back now, the messages are still good, and the obvious move is to put them back where they came from. So you reach for the broker console and start moving messages from orders.dlq to orders.
Stop there. A dead-letter queue is a holding area, not a graveyard — but moving messages out of it is a loaded operation, and “just move them back” is how you turn a recovered outage into a second incident.
\nIt re-poisons the queue. Your DLQ rarely holds only the outage’s victims. Mixed in are messages that died for a real reason — a malformed payload, a bug, an unroutable URN — and those will fail again the instant you replay them. If your replay just shovels everything back, the genuinely broken messages fail, hit the DLQ again, and you replay them again. A loop that costs CPU, fills logs, and buries the messages that would actually succeed.
\nIt re-fires side effects. A message that dead-lettered after three attempts may have partially run each time. The charge went through but the confirmation email threw; the message failed, retried, dead-lettered. Replay it naively and the handler runs start to finish again — a second charge, on top of the email it now finally sends. The queue moved a message; your customer got billed twice.
\nBoth traps come from treating replay as “move bytes back.” It isn’t. It’s “reset a message and reprocess it, safely” — and each of those words is a safety catch.
\nA dead-lettered message carries an extra block describing how it died — the reason, the original queue, the attempt count. To replay it you have to reset it: drop that block, and set the attempt counter back to zero so it gets a fresh retry budget instead of immediately re-exhausting it.
\nBut reset is not the same as new. Preserve everything that identifies the message — its id, its payload, and above all its correlation id (trace_id). Two reasons:
trace_id and the replay shows up in the same trace as the original failure.Even reset and idempotent, a replay is a write to a live system. Earn confidence in stages:
\nThe order matters more than any single step. Replaying straight to production because the messages “look fine” is the same confidence that moved them on the broker console in paragraph one.
\nThe outage’s victims share a signature — a window of time, a failure reason, a URN. Replay that set, not the whole queue. Selecting by reason is the difference between “replay the 4,000 that failed on a timeout” and “replay the 4,000 timeouts plus the 30 genuinely poison messages that will just come back.” The poison messages aren’t your problem to replay; they’re your problem to fix, and they should stay on the DLQ where you can see them.
Here is the honest hard part. Reset clears the envelope; idempotency dedupes exact replays; sandboxing protects production while you test. But a deliberate replay into production — the real one, the one you actually want — runs the handler, and the handler does what it does: it charges, it emails, it calls the third party. Idempotency stops a duplicate; it does not stop the intended reprocess from doing its job, side effects included.
\nThe clean fix is to let a handler know it is running a replay so it can skip the external effects that already happened — re-run the database write, but don’t re-send the email. The catch is where to put that flag: a frozen message envelope has nowhere to add it. The answer is the same one distributed tracing uses for the same constraint — carry it out of band, as a transport header riding alongside the message, one that the runtime surfaces to the handler. A replay-bypass marker the handler can check, set only on replayed messages, costing nothing on the normal path. It’s the most involved piece, and the right one to build last — after reset, dry-run, sandbox, and select have made replay safe enough to be routine.
A dead-letter queue is the most useful thing in your system on the worst day of the quarter. Treat replaying it like what it is — reprocessing live traffic — and it stays useful. Treat it like moving bytes, and the DLQ gets its sequel.
\nSee also: the BabelQueue redrive-and-replay spec standardizes this reset-and-reprocess shape, and the dlq-redrive example is a runnable version of it.
A message is produced by a PHP service, lands on a queue, is consumed by a Go worker, which publishes a follow-up that a Python service handles. Four hops, three languages, one or more brokers in between. When that flow is slow, or one message in ten thousand dies, the question is always the same: where did this message actually go, and what happened at each step?
\nThe honest answer, for most polyglot queue systems, is that nobody knows. You have a correlation id in the logs — if you remembered to log it in all three languages — and you have the patience to grep across three log streams on different hosts. What you do not have is a picture. You cannot see “produced by PHP in 2ms → sat in Redis for 40ms → processed by Go in 210ms, retried twice, dead-lettered” as one connected thing.
That picture is a distributed trace, and OpenTelemetry is the standard way to draw it. This piece is about adding it to a message standard whose wire format is frozen and whose cores carry zero dependencies — two constraints that, together, make the obvious approach illegal and force a more interesting one.
\nLet me state it before anything else: do not add a field to the envelope.
\nThe instinct, when you want distributed tracing across a message bus, is to carry a W3C traceparent — the standard 55-character string that encodes a trace id, a span id, and flags. HTTP does exactly this in a header. The instinct is correct for HTTP and wrong here, because the envelope is a frozen contract. Every SDK in every language emits the byte-identical shape job, trace_id, data, meta, attempts. Adding a traceparent field — even an optional one — changes that shape, which means a version bump, which means coordinating a wire change across six language implementations and every broker binding. For a feature that is supposed to be optional observability, that is an absurd price.
So the rule is the constraint: solve tracing without touching the wire. Everything below follows from taking that seriously.
\nHere is the thing the envelope already gives you for free. The trace_id field is a correlation id — a UUID, minted at produce time and forwarded unchanged across every hop. It is already on the wire, already propagated, already the one value that ties the whole flow together.
Now look at what OpenTelemetry uses to tie a trace together: a TraceID. In the spec, a TraceID is exactly 16 bytes.
\nA UUID is exactly 16 bytes.
\nThat is the whole trick. A trace_id UUID maps one-to-one onto an OTel TraceID — strip the hyphens, read the 32 hex characters as the 16-byte id, done. (A trace_id that is not a UUID — say it came from a non-BabelQueue producer — gets hashed to 16 bytes with SHA-256, deterministically.) Every hop that shares a trace_id therefore derives the same OTel TraceID, with no agreement protocol and no new field. The correlation id you were already carrying is the distributed trace.
So the design writes itself:
\nprocess <urn>, but force that span into the trace derived from the message’s trace_id. Tag it with the messaging conventions — messaging.system, messaging.destination.name, messaging.message.id, and messaging.message.conversation_id set to the trace_id itself — then run the handler, and record any exception as the span’s error status. The runtime’s retry / dead-letter behaviour is untouched; the span just observes it.publish <urn> span, take its trace id, format it back into a UUID, and stamp that into the message’s trace_id as you build the envelope. The downstream consumer, deriving its TraceID from that same trace_id, lands in the same trace.Wire a TracerProvider for Jaeger, Tempo, Honeycomb or Datadog and the flow shows up as one waterfall, across all three languages, with per-hop timing and the retry/error markers in place. Don’t wire one, and nothing changes — it is entirely opt-in.
There is one detail worth being precise about, because it is where the design is both clever and limited.
\nTo start a span inside a specific trace in OpenTelemetry, you give it a remote parent — a span context carrying the trace id you want. But a span context is only valid if it has both a trace id and a span id. We have the trace id (from trace_id); we do not have the upstream span’s id, because we deliberately did not propagate one. So we synthesize a deterministic, non-zero span id by hashing the trace_id. The parent is valid, the consumer span lands in the right trace — but its parent points at a span that never existed. A phantom.
This is the honest limit of the design as it stands above, and it is worth stating plainly. Cross-hop spans all share one trace — you can see every step of a message’s life grouped together, timed, with errors marked. What this alone does not give you is exact parent-child linkage between hops: the consumer’s span is not wired as the child of the producer’s span, because the producer’s real span id was never carried across the wire. Within a single process the hierarchy is correct; across the queue it is flat under one trace.
\nGetting true cross-hop parent-child back means propagating a span id, which means a traceparent — and since the envelope is frozen, that traceparent has to ride out of band, as a transport header alongside whichever slot each broker binding already uses for the trace_id: a bq-trace-id attribute on SQS, Pulsar and Kafka, the native correlation id on RabbitMQ, Azure Service Bus and ActiveMQ Artemis, and on Redis — which carries no per-message metadata at all — nothing but the envelope body. That is a real feature, but it is a different scale of work: it touches every transport binding in every SDK. So it was left as a deliberate phase two — and that phase closed two days after this piece went up: ADR-0028 shipped traceparent transport-header propagation as v0.2 across all six SDK cores on 21 June 2026. A delivered traceparent now upgrades the consumer span to a true child of the producer span, and where it cannot be carried — PHP’s Kafka, Pulsar and STOMP producers are the remaining gap — propagation degrades cleanly back to the v0.1 trace_id correlation described above. The ordering was the point: the phase-one design delivers correlation, per-hop timing and error/retry visibility — the 90% — with zero wire change and a few hundred lines per language, and it stays the floor that v0.2 sits on top of. Shipping the 90% first and the last 10% behind a bigger lift was the right order, the same way you do the cheap scaling steps before the expensive ones.
The constraint that the core stays dependency-free has a consequence: wherever the OpenTelemetry code has to import the OTel API, it cannot live in the core. So in five of the six languages it lives beside the core, reached only when you opt in — and “beside the core, optional” is spelled differently in every ecosystem. (.NET is the exception, for a reason we will get to.) The semantics are identical in all six; the packaging is where each language shows its personality:
\nbabelqueue-go/otel, its own go.mod), exactly like the transport submodules. The core module never sees the OTel dependency.[otel] extra. pip install babelqueue[otel] pulls opentelemetry-api; the module imports it, so it is only importable when you asked for it. A TraceID here is a 128-bit int, not bytes — same value, different shape.@babelqueue/core/otel, with @opentelemetry/api as an optional peer dependency. Critically, the tracing code is not re-exported from the package root: if it were, importing @babelqueue/core would eagerly load the OTel import and break for anyone who didn’t install the peer. The subpath keeps the main entry truly dependency-free.optional Maven dependency on opentelemetry-api. Optional dependencies are not transitive, so a consumer who never touches the tracing classes never pulls OTel onto their classpath.System.Diagnostics.ActivitySource, which lives in the base class library — and it is exactly what OpenTelemetry .NET is built on. So the .NET module needs no dependency at all — which is why it is the one that sits inside the core (BabelQueue.Core / Telemetry) rather than beside it: it emits Activity objects, and the consumer’s OTel pipeline collects them by calling AddSource("BabelQueue"). The core stays zero-dep not by isolating the dependency but by not having one.suggest plus a dev requirement, mirroring the existing optional helpers. The tracing namespace is there; open-telemetry/api is only needed if you use it.Six idioms, one rule held in all of them: using observability is a choice the consumer makes, never a tax the core charges.
\nStrip away the mechanics and the payoff is small to describe and large to have. In your existing tracing backend, a message that used to be a needle in three log haystacks becomes a single waterfall: which service produced it, how long it waited on the broker, how long each consumer took, whether it was retried, whether it was dead-lettered — across languages and brokers you never had to make agree on anything but a UUID they were already passing around.
\nAnd the cost of that, on the wire, is nothing. The envelope that shipped before tracing existed and the envelope that ships with it are byte-for-byte identical. The trace was hiding in the trace_id the whole time; all the work was in noticing that a UUID and a TraceID are the same sixteen bytes.
See also: the BabelQueue observability spec writes this design down as a standard, and the SDKs that implement it — across PHP, Go, Python, Node, Java and .NET — live in the BabelQueue ecosystem.
", "content_text": "A message is produced by a PHP service, lands on a queue, is consumed by a Go worker, which publishes a follow-up that a Python service handles. Four hops, three languages, one or more brokers in between. When that flow is slow, or one message in ten thousand dies, the question is always the same: **where did this message actually go, and what happened at each step?**\n\nThe honest answer, for most polyglot queue systems, is that nobody knows. You have a correlation id in the logs — if you remembered to log it in all three languages — and you have the patience to `grep` across three log streams on different hosts. What you do not have is a picture. You cannot see \"produced by PHP in 2ms → sat in Redis for 40ms → processed by Go in 210ms, retried twice, dead-lettered\" as one connected thing.\n\nThat picture is a **distributed trace**, and OpenTelemetry is the standard way to draw it. This piece is about adding it to a message standard whose wire format is **frozen** and whose cores carry **zero dependencies** — two constraints that, together, make the obvious approach illegal and force a more interesting one.\n\n## The one rule up front\n\nLet me state it before anything else: **do not add a field to the envelope.**\n\nThe instinct, when you want distributed tracing across a message bus, is to carry a W3C `traceparent` — the standard 55-character string that encodes a trace id, a span id, and flags. HTTP does exactly this in a header. The instinct is correct for HTTP and wrong here, because the envelope is a frozen contract. Every SDK in every language emits the byte-identical shape `job`, `trace_id`, `data`, `meta`, `attempts`. Adding a `traceparent` field — even an optional one — changes that shape, which means a version bump, which means coordinating a wire change across six language implementations and every broker binding. For a feature that is supposed to be *optional observability*, that is an absurd price.\n\nSo the rule is the constraint: solve tracing **without touching the wire**. Everything below follows from taking that seriously.\n\n## The insight nobody uses\n\nHere is the thing the envelope already gives you for free. The `trace_id` field is a correlation id — a UUID, minted at produce time and [forwarded unchanged across every hop](/en/systems/data-intensive-systems-breaking-points). It is already on the wire, already propagated, already the one value that ties the whole flow together.\n\nNow look at what OpenTelemetry uses to tie a trace together: a **TraceID**. In the spec, a TraceID is exactly **16 bytes**.\n\nA UUID is exactly 16 bytes.\n\nThat is the whole trick. A `trace_id` UUID maps one-to-one onto an OTel TraceID — strip the hyphens, read the 32 hex characters as the 16-byte id, done. (A `trace_id` that is *not* a UUID — say it came from a non-BabelQueue producer — gets hashed to 16 bytes with SHA-256, deterministically.) Every hop that shares a `trace_id` therefore derives the **same** OTel TraceID, with no agreement protocol and no new field. The correlation id you were already carrying *is* the distributed trace.\n\nSo the design writes itself:\n\n- On the **consumer** side, wrap the handler. Before it runs, start a span named `processA worker started throwing KeyError in production. The cause wasn’t its code — an upstream service had shipped a change that dropped a field from the event’s payload, and the queue had carried the malformed message across without a word. The bug wasn’t the bad message. It was that nothing checked it at the boundary.
A request body hits a typed handler; a framework rejects it at the door if it’s the wrong shape. A queue gives you none of that. The payload is opaque JSON — the broker moves bytes and never looks inside. Whatever the producer put in, the consumer gets, valid or not. The type safety you have inside one service evaporates the moment the data crosses the wire.
\nSo if you want a guarantee about a message’s shape, you have to add it yourself — at the edge, where data enters and where it leaves.
\nThere are two edges, and they catch different failures:
\nA subtlety worth knowing: most queue runtimes have no “reject immediately” hook, so a consumer-side rejection usually retries before it dead-letters — and invalid data never becomes valid on retry. That’s wasted work, which is exactly why producer-side is where the value is. Consumer-side is the seatbelt, not the steering.
\nThe thing you validate against is a schema per message type — keyed by the event’s identity, not bolted onto the transport envelope. Keep it in a registry that versions independently, and the same schema you evolve carefully without breaking consumers becomes the schema you enforce at runtime. One contract, checked at change-time (does this edit break anyone?) and at runtime (does this message obey it?).
\nThis is the structural cousin of making a duplicate delivery a no-op: both accept that you don’t control what arrives, and put the guarantee in your own boundary instead of trusting the sender.
\nA single service’s internal queue — one producer, one consumer, deployed together — doesn’t need this. The type system already spans both ends; edge validation is ceremony. It starts paying the moment a second, independently-deployed producer or consumer exists — a different team, a different language, a different release cadence. That’s also the moment the untyped payload quietly becomes your most fragile contract.
\nSee also: the BabelQueue schema-validation spec defines the per-URN schema and where it’s enforced, and the babelqueue-registry holds those schemas — its bqschema tool is what runs the check.
A new engineer asked me why the domain layer was importing the database package — “doesn’t an ADR (architecture decision record) say it shouldn’t?” It did. It had said so for two years. And for most of those two years, somewhere, the rule had been quietly broken: a helper here, a “just this once” import there, each one reasonable in its own pull request. The decision was still true on the wiki. It had stopped being true in the code months ago, and nobody noticed, because nothing was watching.
\nA decision you write down but can’t enforce isn’t a rule. It’s a wish with good formatting.
\nNobody decides to violate the architecture. It erodes one defensible commit at a time: a synchronous call added because the async path was slower to write, an import that crosses a boundary because the function you needed happened to live on the other side. Each diff looks fine on its own. The damage is cumulative and structural — invisible at the scale of a single change, obvious only when you step back and find the layering gone.
\nThat’s exactly the scale a human review operates at, and exactly the scale it misses. Review works best on the architecture of a change, but a reviewer sees one PR, not the thousandth import that finally dissolved the boundary. You can’t ask a person to hold the whole dependency graph in their head on every commit. The wiki page can’t either; it’s prose, and prose doesn’t fail a build.
\nThe fix isn’t a better-written ADR. It’s moving the enforceable part of the decision out of prose and into something that runs.
\n“The domain must not depend on infrastructure” is not, at heart, an opinion — it’s a constraint on the import graph: files under domain/ may not import db/. That’s a rule a tool can evaluate on every commit, deterministically, and fail the build when it’s broken — before the violating import reaches main, not two years later. The cost of catching it moves from “an archaeology session when something finally breaks” to “a red check on the PR that introduced it.”
This is the same move as versioning a schema instead of mutating it: a rule everyone agrees on is worth nothing until something mechanical holds the line, because human discipline doesn’t scale to every commit across every contributor.
\nThis doesn’t kill the ADR — it splits it. An architecture decision has two parts: the rationale (why we chose hexagonal layering, what we traded away, when we’d revisit it) and the constraint (domain imports nothing; the API talks to the DB only through a repository). The rationale belongs in prose — it’s context a tool can’t hold and a human needs. The constraint belongs in code, where it can be checked.
\nKeep writing the why down. Stop trusting prose to enforce the what.
\nIf your whole system is two layers and a team that fits around one table, the boundaries live in everyone’s head, and a CI gate for them is ceremony. The moment a third contributor joins, or the codebase outgrows what one person can hold, the unenforced boundary is already eroding — you just won’t see it until the new engineer asks why domain imports the database.
\nAn architecture is not what you decided; it’s what your code currently does. The only decisions that survive contact with a growing codebase are the ones something checks on every commit. Write the rationale for humans — and hand the boundary to a machine that never gets tired of enforcing it.
\nSee also: archlint is the tool I built for exactly this — a deterministic import-boundary linter (Go, TypeScript and Python) that reads your layer rules and fails the build on a violation, with a packaged CI Action.
", "content_text": "A new engineer asked me why the domain layer was importing the database package — \"doesn't an ADR (architecture decision record) say it shouldn't?\" It did. It had said so for two years. And for most of those two years, somewhere, the rule had been quietly broken: a helper here, a \"just this once\" import there, each one reasonable in its own pull request. The decision was still true on the wiki. It had stopped being true in the code months ago, and nobody noticed, because nothing was watching.\n\nA decision you write down but can't enforce isn't a rule. It's a wish with good formatting.\n\n## The diagram and the code drift apart in small, reasonable steps\n\nNobody decides to violate the architecture. It erodes one defensible commit at a time: a synchronous call added because the async path was slower to write, an import that crosses a boundary because the function you needed happened to live on the other side. Each diff looks fine on its own. The damage is **cumulative and structural** — invisible at the scale of a single change, obvious only when you step back and find the layering gone.\n\nThat's exactly the scale a human review operates at, and exactly the scale it misses. Review works best on the architecture of a change, but a reviewer sees one PR, not the thousandth import that finally dissolved the boundary. You can't ask a person to hold the whole dependency graph in their head on every commit. The wiki page can't either; it's prose, and prose doesn't fail a build.\n\n## Make the constraint executable\n\nThe fix isn't a better-written ADR. It's moving the *enforceable part* of the decision out of prose and into something that runs.\n\n\"The domain must not depend on infrastructure\" is not, at heart, an opinion — it's a constraint on the import graph: files under `domain/` may not import `db/`. That's a rule a tool can evaluate on every commit, deterministically, and fail the build when it's broken — before the violating import reaches `main`, not two years later. The cost of catching it moves from \"an archaeology session when something finally breaks\" to \"a red check on the PR that introduced it.\"\n\nThis is the same move as [versioning a schema instead of mutating it](/en/journal/schema-evolution-without-breaking): a rule everyone *agrees* on is worth nothing until something mechanical holds the line, because human discipline doesn't scale to every commit across every contributor.\n\n## What stays in the document, and what becomes code\n\nThis doesn't kill the ADR — it splits it. An architecture decision has two parts: the **rationale** (why we chose hexagonal layering, what we traded away, when we'd revisit it) and the **constraint** (domain imports nothing; the API talks to the DB only through a repository). The rationale belongs in prose — it's context a tool can't hold and a human needs. The constraint belongs in code, where it can be checked.\n\nKeep writing the *why* down. Stop trusting prose to enforce the *what*.\n\n## When does this stop mattering?\n\nIf your whole system is two layers and a team that fits around one table, the boundaries live in everyone's head, and a CI gate for them is ceremony. The moment a third contributor joins, or the codebase outgrows what one person can hold, the unenforced boundary is *already* eroding — you just won't see it until the new engineer asks why domain imports the database.\n\n---\n\nAn architecture is not what you decided; it's what your code currently does. The only decisions that survive contact with a growing codebase are the ones something checks on every commit. Write the rationale for humans — and hand the boundary to a machine that never gets tired of enforcing it.\n\n---\n\n*See also:* [archlint](https://github.com/muhammetsafak/archlint) is the tool I built for exactly this — a deterministic import-boundary linter (Go, TypeScript and Python) that reads your layer rules and fails the build on a violation, with a packaged CI Action.", "date_published": "2026-06-18T00:00:00.000Z", "tags": [ "architecture", "governance", "opinion", "Journal" ], "authors": [ { "name": "Muhammet Şafak", "url": "https://www.muhammetsafak.com.tr" } ] }, { "id": "https://sade.dev/en/notes/idempotency-duplicate-delivery/", "url": "https://sade.dev/en/notes/idempotency-duplicate-delivery/", "title": "Idempotency: When the Same Message Arrives Twice", "summary": "At-least-once is a guarantee to design around, not a bug to remove. The key has to be minted by the sender and never derived from the payload alone, and the gap between doing the work and recording the key decides the shape: record it in the same transaction when you own the effect, keep a seen-set when you do not. Concurrent duplicates need a unique constraint, not a check.", "content_html": "A customer got charged twice for one order. Nothing ran twice on purpose: the payment worker charged the card, then crashed before it could ack the message — so the broker, doing exactly what it promised, delivered the message again.
\nThe duplicate wasn’t the bug. The handler that wasn’t built for a duplicate was.
\nAlmost every queue — and every retried HTTP call — is at-least-once. A handler MAY see the same message more than once: after a crash, a release, a redelivery, or a client that timed out and retried. Exactly-once across a network is impractical without coordination the broker can’t give you cheaply, so few brokers offer it — and those that do, like Google Cloud Pub/Sub, fence it in with narrow conditions: a single region, pull subscriptions only.
\nThat reframes the problem. The redelivery isn’t a failure to eliminate; it’s a guarantee to design around. (It’s the same trade the queue makes when it retries a failed job instead of losing it.)
\nAn operation is idempotent if applying it many times has the same effect as applying it once — f(f(x)) = f(x).
paid” is idempotent.The whole job is turning the second kind of operation into the first.
\nYou need a stable identity for this operation, and it must be decided by the sender, not the receiver:
\nIdempotency-Key header — the Stripe model.Two rules keep the key honest:
\nThe handler does one check before the work: has this key been processed? If yes, skip and ack. If no, do the work, then record the key. That’s the entire pattern — the weight is in where the record lives.
\nThe hard part is the gap between doing the work and recording the key. Two shapes, and the right one depends on the side effect:
\nUPSERT, a “set to paid”.So the side effect chooses for you: a row you own → transactional; a third-party call you can’t enlist in your transaction (email, a charge) → seen-set, and make the downstream idempotent too by forwarding your key as its Idempotency-Key.
Two deliveries of the same key racing before either records it slip through a “check then write” — that sequence isn’t atomic. A unique constraint on the key column is: the second insert becomes a conflict you catch and treat as “already handled.” This is the same shape as the gaps a race opens in sequential numbering — the database, not the application, is where uniqueness is actually enforced.
\nIf the handler is already idempotent — a pure UPSERT keyed by a stable id, a “set field to value” — you may need nothing at all. Reach for a key plus dedupe when the operation has a non-idempotent side effect (money, email, an external POST) or accumulates (increment, append). Spend the complexity only where a second run does real damage.
At-least-once isn’t the part you fix; it’s the part you accept. Build the handler so the second delivery is a no-op, and redelivery stops being an incident and goes back to being what it is — the broker keeping its promise.
\nSee also: the BabelQueue idempotency spec pins down where the key comes from and how dedupe behaves, and the idempotency-payments example is exactly this double-charge case made runnable.
A team I watched started shipping roughly twice the code once they leaned on agents. Six weeks later they were shipping slower. The pull-request queue had tripled, CI was red more often than green, and two incidents traced back to changes nobody had really read. Output went up; throughput went down.
\nNothing about that is mysterious. They sped up one station on the line and called it a win.
\nA delivery pipeline is a line of stations: write → review → integrate → verify → release. Its throughput is set by the slowest station, not the fastest. Make writing 3× faster and, if review and verification stay where they were, you haven’t tripled throughput — you’ve just moved the constraint one station downstream and grown a pile of work-in-progress in front of it.
\nThat pile isn’t free. It’s unreviewed PRs aging into merge conflicts, half-integrated branches, a test suite that grew faster than anyone can keep green. The bottleneck didn’t disappear when code got cheap. It relocated — and got harder to see, because it stopped looking like “typing code” and started looking like “waiting.”
\nData from the 2024 and 2025 DORA (DevOps Research and Assessment) reports lands on the same point from the other side: AI is an amplifier, not a shortcut. Teams with strong delivery practices get more out of it; teams without them ship their dysfunction faster. Both reports saw AI adoption correlate with lower stability — because the process, not the typing, was always the real constraint.
\nAn amplifier turns up whatever you feed it. Feed it a clean signal and it’s louder and clearer; feed it noise and it’s just louder noise. AI doesn’t decide which one you’ve got. Your downstream does.
\nIt moved to the parts that were always the expensive ones, and are now drowning:
\nNone of these is a typing problem. All of them are the bill for typing faster.
\nSo the engineering edge in this era isn’t “the AI wrote it.” It’s keeping the cost of trusting what was written low enough that the new bottleneck doesn’t choke. That work is unglamorous and it’s where I’d spend the time AI gives back:
\nThis is the same lesson as “vibe coding lowers the cost of a first version, not the cost of being wrong” — read at the level of the whole pipeline instead of a single change.
\nIf your real constraint genuinely is code production — a true greenfield, a solo prototype, a spike where there’s nothing downstream to protect yet — then go fast and don’t build process for a system that doesn’t exist. The shift only bites once there’s something to keep honest: a team, a queue of reviews, a production others depend on.
\nKnowing which world you’re in is the actual judgment call. Most teams shipping twice the code are no longer in the first one.
\nAI didn’t make engineering cheaper; it made one part of it cheaper. The discipline is to spend the time you saved on the part that just became the bottleneck — not to pour the saved time back into producing even more of what’s already piling up.
", "content_text": "A team I watched started shipping roughly twice the code once they leaned on agents. Six weeks later they were shipping *slower*. The pull-request queue had tripled, CI was red more often than green, and two incidents traced back to changes nobody had really read. Output went up; throughput went down.\n\nNothing about that is mysterious. They sped up one station on the line and called it a win.\n\n## You can't speed up one station and call it throughput\n\nA delivery pipeline is a line of stations: write → review → integrate → verify → release. Its throughput is set by the **slowest** station, not the fastest. Make writing 3× faster and, if review and verification stay where they were, you haven't tripled throughput — you've just moved the constraint one station downstream and grown a pile of work-in-progress in front of it.\n\nThat pile isn't free. It's unreviewed PRs aging into merge conflicts, half-integrated branches, a test suite that grew faster than anyone can keep green. The bottleneck didn't disappear when code got cheap. It relocated — and got harder to see, because it stopped looking like \"typing code\" and started looking like \"waiting.\"\n\n## DORA already has the word for this: amplifier\n\nData from the [2024](https://dora.dev/research/2024/dora-report/) and [2025](https://dora.dev/dora-report-2025/) DORA (DevOps Research and Assessment) reports lands on the same point from the other side: AI is an **amplifier, not a shortcut**. Teams with strong delivery practices get more out of it; teams without them ship their dysfunction faster. Both reports saw AI adoption *correlate with lower stability* — because the process, not the typing, was always the real constraint.\n\nAn amplifier turns up whatever you feed it. Feed it a clean signal and it's louder and clearer; feed it noise and it's just louder noise. AI doesn't decide which one you've got. Your downstream does.\n\n## Where the bottleneck actually went\n\nIt moved to the parts that were always the expensive ones, and are now drowning:\n\n- **Review.** Generation produces PRs faster than humans can give them real attention — so review degrades into rubber-stamping, which is how unread changes reach production.\n- **Verification.** Tests are now nearly free to write, which is exactly why suites fill with [flaky, timing-dependent ones](/en/journal/flaky-tests-and-determinism) that erode trust faster than they build it.\n- **Integration.** More code, written faster by more hands, means more coupling and more ways for [the same message or request to arrive twice](/en/notes/idempotency-duplicate-delivery) and corrupt state.\n\nNone of these is a typing problem. All of them are the bill for typing faster.\n\n## The durable investment is the boring downstream\n\nSo the engineering edge in this era isn't \"the AI wrote it.\" It's keeping **the cost of trusting what was written** low enough that the new bottleneck doesn't choke. That work is unglamorous and it's where I'd spend the time AI gives back:\n\n- Make verification **deterministic**, so a green check means something and review can lean on it.\n- Make integration **idempotent and resilient**, so faster, messier change doesn't turn redelivery into a double-charge.\n- Make review about architecture, not commas — hand the mechanical parts to tools so the human attention you can't scale goes to the decisions that matter.\n- Catch the obvious failure at the **commit boundary**, not three stations later in CI where it costs a hundred times more.\n\nThis is the same lesson as [\"vibe coding lowers the cost of a first version, not the cost of being wrong\"](/en/journal/can-you-ship-vibe-coded-to-production) — read at the level of the whole pipeline instead of a single change.\n\n## When does this not apply?\n\nIf your real constraint genuinely *is* code production — a true greenfield, a solo prototype, a spike where there's nothing downstream to protect yet — then go fast and don't build process for a system that doesn't exist. The shift only bites once there's something to keep honest: a team, a queue of reviews, a production others depend on.\n\nKnowing which world you're in is the actual judgment call. Most teams shipping twice the code are no longer in the first one.\n\n---\n\nAI didn't make engineering cheaper; it made *one part* of it cheaper. The discipline is to spend the time you saved on the part that just became the bottleneck — not to pour the saved time back into producing even more of what's already piling up.", "date_published": "2026-06-17T00:00:00.000Z", "tags": [ "ai-workflow", "productivity", "opinion", "Journal" ], "authors": [ { "name": "Muhammet Şafak", "url": "https://www.muhammetsafak.com.tr" } ] }, { "id": "https://sade.dev/en/journal/flaky-tests-and-determinism/", "url": "https://sade.dev/en/journal/flaky-tests-and-determinism/", "title": "Flaky Tests: Retrying Isn't a Fix", "summary": "Retrying, quarantine and a ticket do not make a flaky test deterministic; they hide the symptom after CI has already broken. Most flakiness comes from a small set of known anti-patterns — a fixed sleep, unseeded randomness, a wall-clock assertion, a structural selector — and every one of them is visible statically in the diff the moment the test is written. That line is the cheapest place to catch it.", "content_html": "A pull request crossed my review: half the diff was tests, and a model had written them. They passed locally. In CI they failed once, passed on a re-run, then failed again two days later under load. Nobody had touched the code.
\nThat test wasn’t wrong. It was flaky — and flaky is its own kind of broken.
\nWriting a test used to cost something, so you wrote fewer and thought harder about each. A model writes them in seconds, from the code’s current shape alone. It can’t see the concurrency, the resource limits, or the real timing of an async call — so it reaches for the patterns that look right and are quietly non-deterministic:
\ntime.Sleep(2 * time.Second), await page.waitForTimeout(2000), cy.wait(2000).Math.random(), rand.Intn(100), a faker with no fixed seed.nth-child, an auto-generated class name — that breaks the moment the markup shifts.Each looks reasonable in isolation. At scale they’re a flakiness factory.
\nA single test with a 0.5% chance of a spurious failure is invisible. Put 800 of those in a suite and the chance that some test fails on a clean run is 1 − 0.995^800 ≈ 98%. The failures aren’t independent of each other either: CI runs on shared, throttled hardware where the async test that finishes in 50 ms on your laptop times out under load.
\nAs the model keeps adding tests, that count climbs and the suite’s odds of a false red march toward certainty. The pass you care about drowns in noise you don’t.
\nThe reflex is to absorb the noise: auto-retry the job, mark the test flaky, quarantine it, open a ticket. Every one of these is reactive — it acts after the test has already broken CI, usually more than once.
\nNone of it makes the test deterministic. It hides the symptom and bills you for the privilege.
\nA flaky test rarely breaks for a subtle reason. It breaks because of a small set of well-known anti-patterns — and those are visible in the diff, before the test ever runs. A fixed sleep, an unseeded RNG, a wall-clock assertion, a structural selector: each one is detectable statically and deterministically, at the moment the test is written.
\nI run that check at the commit boundary — a static pass over the added lines of changed test files, flagging the known anti-patterns. Same diff, same result, no model, no network. It won’t catch a clever race, and it isn’t meant to. The point is narrower and worth a lot: stop the obvious flake from ever reaching CI, where catching it costs a hundred times more and a human’s afternoon.
\nDetection is the backstop. The real fix is to write the test deterministically in the first place:
\nEventually/waitFor. Wait on a fact, not a clock.now() and Math.random() mid-test.A static gate is precision-first: it catches the obvious anti-patterns, not a race condition buried in your own code. Keep its rules conservative — a noisy gate gets ignored, and an ignored gate is worse than none. It will miss the clever flake; don’t let it guess.
\nAnd none of this substitutes for the harder read: a test that’s flaky because the system under test is non-deterministic isn’t telling you about the test. It’s telling you about the system.
\nA green CI you have to re-run to trust isn’t green. The cheapest place to kill a flaky test is the line where it’s written — before it ever costs you a build.
\nSee also: the commit-boundary check described here is what I shipped in CommitBrief — its flaky-test detector statically catches fixed sleeps and unseeded randomness in changed test files, deterministically, before the model.
", "content_text": "A pull request crossed my review: half the diff was tests, and a model had written them. They passed locally. In CI they failed once, passed on a re-run, then failed again two days later under load. Nobody had touched the code.\n\nThat test wasn't wrong. It was **flaky** — and flaky is its own kind of broken.\n\n## AI didn't invent flakiness — it mass-produced it\n\nWriting a test used to cost something, so you wrote fewer and thought harder about each. A model writes them in seconds, from the code's current shape alone. It can't see the concurrency, the resource limits, or the real timing of an async call — so it reaches for the patterns that *look* right and are quietly non-deterministic:\n\n- A fixed sleep to \"wait\" for async work: `time.Sleep(2 * time.Second)`, `await page.waitForTimeout(2000)`, `cy.wait(2000)`.\n- An unseeded random source whose value differs every run: `Math.random()`, `rand.Intn(100)`, a `faker` with no fixed seed.\n- A brittle selector pinned to DOM structure — `nth-child`, an auto-generated class name — that breaks the moment the markup shifts.\n- A mock for everything, so the test verifies an interaction, not a result — the same empty green I wrote about in chasing coverage with tests that do nothing.\n\nEach looks reasonable in isolation. At scale they're a flakiness factory.\n\n## The suite gets *less* reliable as it grows\n\nA single test with a 0.5% chance of a spurious failure is invisible. Put 800 of those in a suite and the chance that *some* test fails on a clean run is 1 − 0.995^800 ≈ 98%. The failures aren't independent of each other either: CI runs on shared, throttled hardware where the async test that finishes in 50 ms on your laptop times out under load.\n\nAs the model keeps adding tests, that count climbs and the suite's odds of a false red march toward certainty. The pass you care about drowns in noise you don't.\n\n## Retrying is a treadmill, not a cure\n\nThe reflex is to absorb the noise: auto-retry the job, mark the test flaky, quarantine it, open a ticket. Every one of these is **reactive** — it acts after the test has already broken CI, usually more than once.\n\n- Retries burn real CI minutes and stretch the feedback loop; you pay, twice, to learn nothing new.\n- Quarantine grows a silent pile of disabled tests. That pile is debt, and it compounds — each skipped test is a check nobody runs anymore.\n- The ticket ages into backlog noise and quietly stops mattering.\n\nNone of it makes the test deterministic. It hides the symptom and bills you for the privilege.\n\n## Catch the anti-pattern, not the failure\n\nA flaky test rarely breaks for a subtle reason. It breaks because of a small set of well-known anti-patterns — and those are **visible in the diff, before the test ever runs.** A fixed sleep, an unseeded RNG, a wall-clock assertion, a structural selector: each one is detectable statically and deterministically, at the moment the test is written.\n\nI run that check at the commit boundary — a static pass over the added lines of changed test files, flagging the known anti-patterns. Same diff, same result, no model, no network. It won't catch a clever race, and it isn't meant to. The point is narrower and worth a lot: stop the *obvious* flake from ever reaching CI, where catching it costs a hundred times more and a human's afternoon.\n\n## Design the test so there's nothing to detect\n\nDetection is the backstop. The real fix is to write the test deterministically in the first place:\n\n- Replace fixed waits with **condition-based** waiting — poll until the state you expect, or use the framework's `Eventually`/`waitFor`. Wait on a fact, not a clock.\n- **Seed** every random source, and inject a deterministic clock and IDs instead of reading `now()` and `Math.random()` mid-test.\n- Select by **role or test id**, never by DOM position.\n- Mock the boundary, not the **logic** — and keep at least one real integration path, or the suite verifies a world that doesn't exist.\n\n## When does this stop being the right answer?\n\nA static gate is precision-first: it catches the obvious anti-patterns, not a race condition buried in your own code. Keep its rules conservative — a noisy gate gets ignored, and an ignored gate is worse than none. It will miss the clever flake; don't let it guess.\n\nAnd none of this substitutes for the harder read: a test that's flaky because the *system under test* is non-deterministic isn't telling you about the test. It's telling you about the system.\n\n---\n\nA green CI you have to re-run to trust isn't green. The cheapest place to kill a flaky test is the line where it's written — before it ever costs you a build.\n\n---\n\n*See also:* the commit-boundary check described here is what I shipped in [CommitBrief](https://commitbrief.com) — its [flaky-test detector](https://github.com/CommitBrief/commitbrief) statically catches fixed sleeps and unseeded randomness in changed test files, deterministically, before the model.", "date_published": "2026-06-17T00:00:00.000Z", "tags": [ "testing", "ci", "ai", "quality", "Journal" ], "authors": [ { "name": "Muhammet Şafak", "url": "https://www.muhammetsafak.com.tr" } ] }, { "id": "https://sade.dev/en/journal/schema-evolution-without-breaking/", "url": "https://sade.dev/en/journal/schema-evolution-without-breaking/", "title": "Schema Evolution Without Breaking the Contract", "summary": "You cannot change a cross-service contract atomically, so the only safe changes are the ones where old and new coexist: loosening is safe, tightening is not. Adding an optional field or dropping a required constraint keeps old data valid; adding a required field, renaming, retyping or closing additionalProperties does not. When a change is breaking, mint a new identity and upgrade consumers before producers.", "content_html": "Someone added a required customer_id field to an event and deployed the producer. Half the consumers hadn’t shipped the change yet. For twenty minutes, every message that older consumers had handled fine for a year started failing validation — until the rollout caught up. No code was “wrong.” A contract was changed out from under the people bound by it.
A message’s data shape is an API. The fact that it’s JSON on a queue instead of a response body doesn’t make it less of a contract — it makes it a harder one, because you can’t change both sides at once.
An HTTP API has one server you control. A published message has many consumers, deployed independently, on their own schedules — and with at-least-once delivery, in-flight messages produced under the old shape are still arriving while you roll out the new one. There is no moment where “everyone is on the new schema.” So the only safe changes are the ones where old and new can coexist.
\nThat single constraint gives you the whole rule set.
\nA change is backward-compatible if data valid under the old schema is still valid under the new one — so a consumer that upgraded early still accepts messages a not-yet-upgraded producer emits.
\nrequired constraint (make a field optional) — safe. Old data still satisfies the looser rule.enum, relax a minimum — safe.And the ones that bite:
\nenum value, raise a minimum, close additionalProperties — breaking.The asymmetry is the point: loosening is safe, tightening is not. A consumer can tolerate data that’s more permissive than it expected far more easily than data that’s missing something it now demands.
\nThe instinct on a breaking change is to “just update the schema.” Don’t. Mutating the shape behind an existing identity is exactly what broke the rollout above. Instead, mint a new identity — a new message URN (urn:babel:orders:created.v2), a new topic, a new event name — and run both in parallel:
v1; you publish v2 alongside it.v2 on their own schedules.v1 has no consumers left, retire it.The rule underneath every step: consumers upgrade before producers. Never emit a version no deployed consumer understands. (This is the same shape as designing the handler so a duplicate delivery is a no-op — you make the change safe to apply in any order, because you don’t control the order.)
\nNone of this is judgment you want to re-derive in a code review at 5 p.m. on a Friday. The compatibility rules above are deterministic — given the old and new schema, a tool can tell you “additive, ship it” or “breaking, mint a new version” with no opinion involved. So I gate it: a check at the boundary that diffs the two schemas and fails the build on a breaking change, before it reaches anyone downstream. Catching it there costs a comment on a PR; catching it in production costs a rollout window like the one above.
\nThis is the boring-infrastructure end of the same thesis as “the bottleneck moved downstream”: the schema is cheap to change and expensive to change wrongly, so you spend a little tooling to keep the second cost off the table.
\nIf a message has exactly one producer and one consumer that deploy together — a single service’s internal queue — the contract isn’t really distributed, and you can change both sides at once. Then the ceremony is overhead. The rules earn their keep the moment a second, independently deployed consumer exists. Most events that outlive their first month reach that point.
\nA schema isn’t a struct you own; it’s a promise other services planned around. Evolve it the way you’d evolve any promise you can’t unmake — additively, or under a new name. Never by quietly redefining the old one.
\nSee also: the BabelQueue schema-validation spec writes these compatibility rules down, and the babelqueue-registry is where per-URN schemas live — its bqschema tool (and packaged Action) is the boundary check that fails the build on a breaking change.
“It worked locally” has a new relative: “it worked in the demo.” A product described a week ago and stood up in minutes, with its first users already on board — then the first real load arrives and it slows to a crawl, the first curious user sees someone else’s invoice, the first schema change blows up in a migration. The person who built it is surprised: the code was working.
\nIt was. But what vibe coding lowered was the cost of the first version, not the cost of being wrong. That cost is deferred, not erased — and deferred cost charges interest.
\nLet’s not strawman it. Shipping a first version has never been cheaper: a CRUD screen, a REST endpoint, a form validation — describe it, watch it work. For a prototype, a throwaway, an internal tool, a learning project, it’s a brilliant accelerator.
\nThere’s exactly one thing it doesn’t make cheaper: the price of being wrong about the data model, the security boundary, and the failure mode. If those three are wrong in production, the bill comes not at build time but much later — usually at 2 a.m.
\nFour patterns I see again and again in systems shipped with vibe coding. None of them are the “code won’t compile” kind; they all live in the gap between intent and reality.
\n1. A data model that can’t migrate. A NOT NULL column added to a populated table, with no DEFAULT. Fine on an empty table, a failure on the production one. Deeper still: a schema with the wrong relationships makes a trivial feature impossible six months later. Fixing the data model afterward isn’t a refactor — it’s a data-migration risk, done at the most expensive moment, with real data.
2. Security that never accounted for adversarial input. Object IDs that pass the identity check but not the ownership check (IDOR), user input reaching a query, secrets hardcoded into the source. The “standard” solution the agent produced doesn’t assume the input could be hostile — because the question it was asked was “make this work,” not “make this withstand an attack.”
\n3. An access pattern that ignores scale. A query inside a loop (N+1), SELECT *, an unpaginated list, a synchronous call where it should be async. Invisible on a ten-row table; it wrecks your p95 at a hundred thousand.
4. A system you can’t observe or roll back. No logs, no metrics, no rollback plan. When it breaks, you can’t see why — which is exactly what you’ll need most in a vibe-coded system.
\nThe common denominator: the tool is most dangerous where it looks safest. It writes common patterns solidly; it produces a “standard” solution without seeing the problem’s unique dimension. A standard solution is the answer to a problem you didn’t ask.
\nPicture the same agent in three different hands. The difference isn’t who has it type faster; it’s the quality of the question asked and the review performed.
\nThis is a budget statement, not an ideology: a senior is the cheapest insurance against deferred cost. Building the right data model up front is an afternoon; fixing the wrong one on a live system is a quarter. “We’ll fix it once we make money” is absurd for this reason — the product that makes money is also the one that ties your hands, the one whose migration is riskiest. A senior is cheaper before, not after.
\nLike any recommendation, this one has an expiry — and I won’t make a recommendation without stating its limit.
\nVibe coding is the right tool for a prototype, a throwaway demo, a hobby, an internal tool with no real data or users, learning. There, the cost of being wrong is low, so deferring it is rational.
\nThe equation flips the moment one of three things comes to the table: real users, real data, real money (or a real uptime commitment). Past that threshold, deferred cost starts charging its interest, and you need someone holding the wheel.
\nProducing with AI agents is possible, and genuinely fast. But the agent changed who writes the code, not who’s responsible for it. Vibe coding made the first version cheaper, not being wrong — and that cost depends on whose hands shipped it to production.
", "content_text": "\"It worked locally\" has a new relative: \"it worked in the demo.\" A product described a week ago and stood up in minutes, with its first users already on board — then the first real load arrives and it slows to a crawl, the first curious user sees someone else's invoice, the first schema change blows up in a migration. The person who built it is surprised: the code was working.\n\nIt was. But what vibe coding lowered was the cost of the **first version**, not the cost of being wrong. That cost is deferred, not erased — and deferred cost charges interest.\n\n## What vibe coding genuinely makes cheaper\n\nLet's not strawman it. Shipping a first version has never been cheaper: a CRUD screen, a REST endpoint, a form validation — describe it, watch it work. For a prototype, a throwaway, an internal tool, a learning project, it's a brilliant accelerator.\n\nThere's exactly one thing it doesn't make cheaper: the price of being **wrong** about the data model, the security boundary, and the failure mode. If those three are wrong in production, the bill comes not at build time but much later — usually at 2 a.m.\n\n## The bill shows up in production, not at build time\n\nFour patterns I see again and again in systems shipped with vibe coding. None of them are the \"code won't compile\" kind; they all live in the **gap between intent and reality**.\n\n**1. A data model that can't migrate.** A `NOT NULL` column added to a populated table, with no `DEFAULT`. Fine on an empty table, a failure on the production one. Deeper still: a schema with the wrong relationships makes a trivial feature impossible six months later. Fixing the data model afterward isn't a refactor — it's a data-migration risk, done at the most expensive moment, with real data.\n\n**2. Security that never accounted for adversarial input.** Object IDs that pass the identity check but not the ownership check (IDOR), user input reaching a query, secrets hardcoded into the source. The \"standard\" solution the agent produced doesn't assume the input could be hostile — because the question it was asked was \"make this work,\" not \"make this withstand an attack.\"\n\n**3. An access pattern that ignores scale.** A query inside a loop (N+1), `SELECT *`, an unpaginated list, a synchronous call where it should be async. Invisible on a ten-row table; it wrecks your p95 at a hundred thousand.\n\n**4. A system you can't observe or roll back.** No logs, no metrics, no rollback plan. When it breaks, you can't see *why* — which is exactly what you'll need most in a vibe-coded system.\n\nThe common denominator: the tool is most dangerous where it looks safest. It writes common patterns solidly; it produces a \"standard\" solution without seeing the problem's unique dimension. **A standard solution is the answer to a problem you didn't ask.**\n\n## What sets output quality: not typing speed, but review\n\nPicture the same agent in three different hands. The difference isn't who has it type faster; it's **the quality of the question asked and the review performed.**\n\n- **Little to no knowledge.** Can't frame the question, can't read the output critically. The \"standard solution\" ships to production as-is, together with the four patterns above. Can't yet feel the difference between \"it works\" and \"it works correctly.\"\n- **Low-to-mid knowledge.** Frames some questions right, catches part of the output. The ceiling is at the system's boundaries: where it breaks under load, whether this abstraction is right, what change the data model tolerates. Result: good pieces, a fragile whole.\n- **Senior + systems architecture.** The agent is a **multiplier**, not a substitute. Have it plan → approve → write step by step → review like a junior's code → don't merge a line you can't defend. The concrete form of that discipline is its own post: [My AI-Assisted Engineering Workflow](/en/journal/ai-assisted-engineering-workflow). In these hands, speed rises and quality holds — because the reasoning stays in a human.\n\nThis is a budget statement, not an ideology: a senior is the **cheapest insurance** against deferred cost. Building the right data model up front is an afternoon; fixing the wrong one on a live system is a quarter. \"We'll fix it once we make money\" is absurd for this reason — the product that makes money is also the one that ties your hands, the one whose migration is riskiest. A senior is cheaper before, not after.\n\n## When is vibe coding the right answer?\n\nLike any recommendation, this one has an expiry — and I won't make a recommendation without stating its limit.\n\nVibe coding is the **right** tool for a prototype, a throwaway demo, a hobby, an internal tool with no real data or users, learning. There, the cost of being wrong is low, so deferring it is rational.\n\nThe equation flips the moment one of three things comes to the table: **real users, real data, real money** (or a real uptime commitment). Past that threshold, deferred cost starts charging its interest, and you need someone holding the wheel.\n\n---\n\nProducing with AI agents is possible, and genuinely fast. But the agent changed who *writes* the code, not who's *responsible* for it. **Vibe coding made the first version cheaper, not being wrong** — and that cost depends on whose hands shipped it to production.", "date_published": "2026-06-13T00:00:00.000Z", "tags": [ "ai-workflow", "vibe-coding", "production", "architecture", "opinion", "Journal" ], "authors": [ { "name": "Muhammet Şafak", "url": "https://www.muhammetsafak.com.tr" } ] } ] }