You fire a hundred concurrent requests at the same API key. The usage counter says eighty-seven. No errors in the logs, no failed requests, nobody complaining. Thirteen increments are just missing.
That failure has hit me three times, in projects that don't share a single line of code, always for the same reason: the counter was written as if only one process were writing.
The pattern: read, add and write in three steps
The code that causes it never looks dangerous. In a transcription API, authenticating a key did two things: check it wasn't revoked and update useCount and lastUsedAt. Read the row, add one in memory, write the result.
With one user at a time it works. With batch workers hammering the same key, it doesn't: between the read and the write another request comes in, reads the same value and writes the same result. Two uses, one increment. Measured with k6 at 10, 50 and 100 virtual users, the loss was between 8 and 14 percent.
The fix fits in one statement:
UPDATE api_key
SET use_count = use_count + 1,
last_used_at = now()
WHERE id = $1 AND revoked_at IS NULL
RETURNING use_count;
What matters isn't just the use_count + 1, which is the obvious part. It's that revoked_at IS NULL sits inside the same UPDATE. Before, the revocation check and the increment were two separate round trips, so there was a window where someone revoked the key right after it was validated and the use got counted anyway. With the filter inside, either the row meets the condition and gets incremented, or nothing comes back and you throw a 401. No intermediate state to manage.
After the change, nine runs of the concurrency test (three levels times three repetitions), zero losses.
The opposite case: charging twice for the same thing
The second case was the other way round. In a semantic search product, each search consumes one quota unit and stores a row with what it cost: embeddings model, tokens, rerank model, input and output tokens.
There was deduplication: repeat the same search and the reservation returned duplicate, reused the original row and didn't charge quota again.
The problem is that the deduplicated search did run. It called embeddings again, called the rerank again, and the provider billed it again. That cost had nowhere to go: the original row already carried its own, and overwriting it would have erased the first attempt's spend. So it vanished. The admin panel said searches cost less than they cost.
The temptation was to insert another row in the same table with a different status. Bad idea, for two concrete reasons:
- Every quota query counts rows without filtering. An extra row there would charge quota nobody had consumed.
- The duplicate lookup doesn't filter by status either. It takes the most recent row with that hash. The new row would become the anchor for the next deduplication — a row that was never a quota unit.
The solution was a separate table, unbilled_search_spend, with the same usage columns and a reference to the reservation the duplicate resolved against. That reference is for auditing, with ON DELETE set null, not a quota reference. The cost shows up in reports, but it doesn't enter the cost per billed search, which is the number used to set prices.
The third: two balances read separately
The third was an image generation SaaS. I added daily credits on top of subscription credits and generation stopped working for anyone who only had the daily ones. The endpoint checked profile.subCredits < estimatedCost and returned a 402, ignoring dailyCredits entirely. The credits existed, they were rendered in the sidebar, and the server said there weren't enough.
That isn't a concurrency bug, it's splitting a counter in two and forgetting one. But the root is the same: the balance stopped being a number and became the result of an operation spread across the code, done by hand in every place.
const totalAvailableCredits = (profile?.dailyCredits || 0) + (profile?.subCredits || 0);
That line appears once in the check and once when computing the remaining balance returned to the client. Two places where you have to remember to add. The day a third credit type shows up, it'll be three.
And the one that isn't a bug, but an unwritten policy
The fourth case is different but belongs to the same family. The invoice.paid webhook stomped the balance on every renewal:
balanceCents: pricing.balanceCents, // stomped the previous balance
Everything unspent from the previous month evaporated on renewal day. Across forty users on a plan with 1499 cents of balance, the median loss was 310 cents, and the 90th percentile hit 890. Almost 60 percent of the plan thrown away in some cases.
What makes it interesting is that nobody had decided it. It wasn't the policy, it was the consequence of writing an assignment instead of an addition. Now it accumulates with a three-month cap:
const nextBalance = Math.min(
current.balanceCents + pricing.balanceCents,
pricing.balanceCents * 3
);
The cap isn't cosmetic: without it, an account with sporadic usage piles up twelve months of balance and then spends it in one spike, which is exactly what you don't want when upstream you pay an AI provider per use.
The rule
Any counter that represents money gets written with three things from day one. One: the increment and its conditions live in a single statement, SET x = x + 1 WHERE ..., never a read-modify-write. Two: if there's deduplication, the real cost has somewhere to land even when quota isn't charged. Three: the policy on what's left over at renewal is written down somewhere, even in a comment, because if it isn't, the assignment operator writes it for you.
None of the three costs time when done at the start. All three cost an incident when done later.