So the last phase of the project was a test suite that kills the app. Not a mocked crash or a thrown exception. A real SIGKILL, so no finally block runs, no shutdown hook runs, and Postgres is left holding an open transaction from a client that no longer exists.

The app runs as a child process

The tests build the app and start dist/src/main.js as a child process on a free port, against the test database and Redis. Killing it is process.kill(pid, 'SIGKILL'). Restarting it is starting it again.

One snag: the expiry and reconciliation workers run every ten minutes, hard-coded, and a test can't wait ten minutes. The child is started with a preload, -r fast-timers.cjs, that swaps any setInterval of 600000 ms for 300 ms. The app code doesn't know.

Stopping at an exact statement

Killing a process at a random moment mostly tests nothing, because most moments are between requests. The interesting kills are inside a transaction, after some writes and before the commit. I didn't want to add if (process.env.CRASH_HERE) lines to money code to get there.

Postgres already has a way to make a transaction wait at a known point: a row lock. Approving a payment updates the store's account row last, right before commit. So the test opens its own connection and locks that row first.

await pg.query('BEGIN');
await pg.query(
  'SELECT 1 FROM "Account" WHERE id = $1 FOR UPDATE', [storeAccountId]);

// fire the approve request, then wait until the app is stuck on our lock
await waitFor(() => pg.query(
  "SELECT 1 FROM pg_stat_activity WHERE wait_event_type = 'Lock'"));

app.kill();               // the claim, debit, ledger row and outbox row are written
await pg.query('ROLLBACK');

When pg_stat_activity shows the app waiting on a lock, it has already claimed the payment, debited the user and written the ledger and outbox rows, all uncommitted. Then it dies. The same trick works for top-ups, by locking the funding account, and for the expiry worker, by locking one of the payments it's about to expire.

There's one more wait after the kill. Postgres only notices a client is gone when it next reads from its socket, so for a moment the dead app's transaction still looks alive. The harness waits until no backend is idle in transaction or waiting on a lock before it checks anything.

The webhook case

The hardest crash for an outbox is after the HTTP request reaches the store and before the worker records that it did. The test webhook receiver handles that by killing the app the moment an event arrives. After restart, the event is still PENDING with a lease that hasn't expired. When the 60 second lease runs out, a worker picks it up again and sends it with the same event id, which is how the store is meant to recognise the duplicate.

The chaos loop

The last test kills the app eight times at random while four users and two stores create, approve and top up, payments expire after two seconds, and the receiver answers 10% of webhooks with a 500. After the last restart everything is allowed to drain, then the test checks:

• Every create, approve and top-up the app acknowledged before dying is in the database.
• Every top-up that got no answer, retried with the same idempotency key, applied exactly once.
• Every event id reached the receiver at least once.
• The reconciliation job reports zero drift, and the ledger and payment invariants hold row by row.

What it found

Nothing in src had to change, which was the result I was hoping for and not the one I expected. It found two smaller things. The production start script pointed at the wrong file, because the build output nests under dist/src. And a crash between the PIN attempt counter's INCR and its DEL burns one of the user's five attempts until the window expires. I left that one in and wrote it down. It costs a user one guess.

The suite takes about two and a half minutes. It's the slowest thing in the repo and the test I trust most.

My first version did the expiry in batches of 100 with Prisma's limit option, and it read like exactly that rule:

await tx.payment.updateMany({
  where: { status: 'PENDING', expiresAt: { lte: now } },
  data: { status: 'EXPIRED' },
  limit: 100,
});

It failed when an approval landed at the same moment the worker ran. The balances were fine. A constraint fired, which meant something had tried to write a state that shouldn't exist.

What Prisma sends

Postgres has no LIMIT on UPDATE, so Prisma moves the conditions into a subquery:

UPDATE "Payment" SET status = 'EXPIRED'
WHERE id IN (
  SELECT id FROM "Payment"
  WHERE status = 'PENDING' AND "expiresAt" <= $1
  LIMIT 100
);

That's fine until a row is locked. Say an approval has claimed payment P, setting it to SUCCESS, and hasn't committed yet. The worker's subquery runs on its snapshot, sees P as pending, and puts it in the list. The outer UPDATE then waits on P's row lock.

When the approval commits, Postgres under READ COMMITTED re-checks the waiting update's WHERE clause against the new version of the row. The update's own WHERE is only id IN (...), and P's id is still in that list. The status condition lived in the subquery, and the subquery isn't re-run. So the worker overwrote a committed SUCCESS with EXPIRED.

What stopped it

Two constraints I'd added for other reasons. A CHECK on Payment says userId is set if and only if the status is SUCCESS, so an expired payment with a payer is illegal. And the outbox table has one event per payment, so the worker's payment.expired row collided with the approval's payment.succeeded row. Either one was enough to roll the batch back. The money was never wrong, but only because the schema refused to store the wrong answer.

The fix

Pick candidates with a read, then put the full state condition in the update's own WHERE, and take every side effect from the rows the update returned:

const due = await tx.payment.findMany({
  where: { status: 'PENDING', expiresAt: { lte: now } },
  select: { id: true },
  take: 100,
});

const expired = await tx.payment.updateManyAndReturn({
  where: { id: { in: due.map((p) => p.id) }, status: 'PENDING' },
  data: { status: 'EXPIRED' },
});

await tx.webhookEvent.createMany({
  data: expired.map((p) => ({ paymentId: p.id })),
});

Now when the update waits on P's lock and re-checks, status = 'PENDING' is part of what it re-checks. P has moved on, so it drops out, and no event is written for it because events come from expired, not from due.

I wrote the rule into the project notes so I'd stop relearning it: the condition that makes a write safe has to be in that write's own WHERE, and anything that depends on the write uses the rows it returned, never an earlier lookup. Later, for manual webhook replay, I turned on Postgres statement logging to check what Prisma generated for a relation filter before trusting it. It compiles to an EXISTS inside the update's own WHERE, which is safe. limit is the one to watch.

How an order moves

Stock lives in Redis. When an order comes in, a Lua script checks the stock key and decrements it in one step, so two buyers can't both see the last unit:

local stock = tonumber(redis.call("get", KEYS[1]) or 0)
local want = tonumber(ARGV[1])
if stock >= want then
  return redis.call("decrby", KEYS[1], want)
else
  return -1
end

If that succeeds, the order is written to Postgres as PENDING, a job goes onto a BullMQ payment queue, and the buyer gets a response in milliseconds. The payment worker picks the job up and calls the mock provider, which takes one to four seconds and fails 20% of the time on purpose. A failed charge puts the stock back.

Jobs can get lost. The process can die mid-charge, or a job can sit in the queue too long. So a cleanup worker runs every ten seconds, finds orders stuck for more than a minute, settles them, and returns their stock.

The bug

Both workers finished an order the same way: write the final status by order id. Nothing checked what the status was before.

Picture an order whose payment job is still waiting in a slow queue when it passes the one minute mark. Cleanup marks it FAILED and puts the stock back. Then the payment job finally runs, charges the card, and writes PAID. The buyer has paid for a unit that has also gone back on sale. Run it the other way round and stock gets restored twice. Either way the count is wrong.

A status as the lock

I didn't want a second system to coordinate the two, like a Redis lock per order. That's another thing to keep in sync, and another thing to lose. The order row already has a status. I added one value to it: PROCESSING, meaning a worker has claimed this order.

Every worker now claims before it acts, with a conditional update, and gives up if the claim didn't match:

const { count } = await prisma.order.updateMany({
  where: { id: order.id, status: 'PENDING' },
  data: { status: 'PROCESSING' },
});
if (count === 0) return; // cleanup already took it

const result = await payments.chargeCard(/* ... */);

await prisma.order.updateMany({
  where: { id: order.id, status: 'PROCESSING' },
  data: { status: result },
});

Cleanup does the same in reverse. It only fails orders that are still PENDING, or PROCESSING and older than the cutoff, and it restores stock only for the rows its update actually returned. One line in the cleanup worker still carries the comment // <- guard was missing next to the status: 'PROCESSING' condition on the PAID branch, because that's where it went wrong.

Making the race happen

A race you only reason about isn't fixed. scripts/order-race-test.js starves the payment queue on purpose so jobs sit past cleanup's one minute window while still queued, which forces both workers onto the same orders.

In the run I kept, 400 buyers went after 400 units. Cleanup swept 127 orders while their jobs were still queued. You can spot those afterwards because they're FAILED with no payment row ever created. The payment worker reached all 127 later, lost the claim on every one, and did nothing. Final count: 192 units left and 208 paid, which is 400.

The same idea came back in my next project, a payments ledger, where every money movement is a conditional update with a row count checked after it. If the update matched zero rows, someone else got there first, and the only correct move is to stop.

Before calling it done I ran a QA pass over every admin page against the demo seed data. The seed has a reference file with the expected value of every headline figure, worked out by hand, so each number on screen could be checked against something independent.

The tax arithmetic

Most of the risk was here. Each billing cycle stores its PO amount, the income and services tax rates, and how much of each tax the client withholds. Everything else is derived:

servicesTax         = po * servicesRate
incomeTax           = (po + servicesTax) * incomeRate
taxPaidByClient     = incomeTax * incomeWithheld
                    + servicesTax * servicesWithheld
companyTaxLiability = (incomeTax + servicesTax) - taxPaidByClient
actualAmount        = po - taxPaidByClient

These are Mongoose virtuals on the cycle model. One catch: virtuals don't exist in aggregation results. Any figure that comes out of an aggregation pipeline has to re-derive the formula from the raw fields, so the code does that and the project notes say never to trust a virtual from a pipeline.

All of that checked out. Collection rates, tax liability, vendor balances, payroll and inventory matched the reference to the rupee. One thing didn't.

The mismatch

The dashboard showed one project's estimated profit as 2.71 crore. Its own detail page said 2.66 crore. The gap was 480,000, which was exactly that project's material spend.

The detail page computed profit as the amount received, minus contract costs, minus the company's tax liability, minus materials. The dashboard's version had been written first and never got the materials term. Two projects with no material spend matched on both screens, which is why it hadn't been noticed.

Worse, the dashboard had two copies. The per-project health cards used one, and the summary tile at the top derived its own total with the same missing term. Fixing only the first would have left the top tile disagreeing with the cards below it.

The fix

I added the missing term to the per-project calculation, then deleted the summary's copy entirely. The summary now adds up the per-project figures instead of computing its own:

const estimatedProfit = projects.reduce(
  (sum, p) => sum + p.estimatedProfit, 0);

The top tile dropped from 9.42 crore to 9.30 crore, a difference of exactly the company's total material spend. No frontend change was needed, because the dashboard components only render what the API sends.

The pass also turned up two things that looked like bugs and weren't. The vendor list's Given and Remaining columns don't add up to Total, on purpose: Given counts contracts and materials, Remaining counts only what's left on contracts. And an inflow tile shows 0% change when the previous period had no revenue, rather than infinity. The first was confirmed with the client and the second is documented in the code, so both stayed.

The rule I took away is short. A number that appears on two screens comes from one function. If two places compute it, they'll drift, and the first person to notice will be the one reading the dashboard.

The obvious way to build that is a nightly job that adds interest to every investment. I didn't want one. The backend had to be safe to run on serverless hosting, where there's no process sitting around to run a job, and a job that silently fails one night leaves every balance wrong the next morning.

Two numbers per investment

Each investment stores a base amount and the time it was last settled. Nothing else. Its value right now is compound growth over the time since then:

function calculateInvestmentValue(baseCoins, dailyReturn, lastSettledAt) {
  const days = (Date.now() - lastSettledAt) / 86_400_000;
  const value = baseCoins * (1 + dailyReturn / 100) ** days;
  return Math.max(0, Math.round(value * 100) / 100);
}

days is fractional, so growth accrues continuously rather than jumping at midnight. The result is floored at zero, since a market with a negative return shouldn't be able to take a child's investment below nothing. Every controller that needs a value calls this one function.

Settling

The trick that keeps it simple is settling. When a child adds more coins to a market they're already in, or takes some out, the controller first computes the current value, then folds the change into it and resets the clock:

const current = calculateInvestmentValue(inv.baseCoins, market.dailyReturn, inv.lastSettledAt);
inv.baseCoins = current + deposit;   // or current - withdrawal
inv.lastSettledAt = new Date();

All the growth so far becomes part of the new base, and growth from here on starts from the new time. That's why one entry per market per child is enough, with no history inside the wallet.

The "grew this week" figure on the wallet screen uses the same function, evaluated at now and at seven days ago. For an investment younger than a week, it's simply the growth since it was made.

What it costs

The market's rate isn't snapshotted. If an admin changes a market's daily return, the new rate applies back to each investment's last settle time. For a teaching app with admin-set rates that was an acceptable trade, and it's written down in the docs so nobody is surprised by it. A real brokerage would need rate history.

History lives in a separate, append-only transaction log: every deposit, withdrawal and refund, indexed by user and date. Each entry copies the market's name at the time it was written. When an admin deletes a market, every child's investment in it is valued, refunded to their main balance and logged as a refund, and their history still reads correctly because it never depended on the market existing.

The same lesson showed up in the allowance feature later. A withdrawal stores the coin-to-dollar ratio that applied when it was made, so a parent changing the ratio tomorrow doesn't change what yesterday's withdrawal was worth. If a number can change later, store the value you used, not a pointer to it.

RedAuth uses two tokens, both in httpOnly cookies. The access token lives 15 minutes and is checked by signature, so a request doesn't need a database query to know who's calling. The refresh token lives seven days and is the only thing that can get you a new access token. That split is standard. What matters is what happens to the refresh token after it's used.

A refresh token works once

Every refresh token carries a tokenId in its payload, and there's a database record with that id holding a SHA-256 hash of the token, the device it belongs to and its expiry. The id in the payload makes the lookup a primary-key read instead of a hash search.

When a client trades in a refresh token, the server deletes that record and issues a new pair with a new id. The old token still has a valid signature for another week, but its record is gone.

Now say there are two copies: the real one on the user's laptop and a stolen one somewhere else. Whichever is used first wins and gets rotated. When the second arrives, the server sees a token that verifies but has no record. There's no innocent explanation for that, and RedAuth doesn't try to work out which side is the attacker:

if (!record) {
  if (strict) {
    // A valid token with no record: it was already rotated. Someone has a copy.
    const sessions = await prisma.token.findMany({
      where: { userId: decoded.id, type: "REFRESH_TOKEN" },
    });
    await prisma.token.deleteMany({ where: { userId: decoded.id } });
    await Promise.all(sessions.map((s) =>
      redis.set(`revoked-${s.id}`, "true", "EX", 60 * 30)));
  }
  throw new AppError(401, "Session revoked or expired. Please login again.");
}

Every session that user has, on every device, is deleted. The user signs in again everywhere. The attacker's copy is dead, and so is anything they rotated it into.

Why access tokens die too

Deleting refresh records doesn't stop an access token that's already out there. It would stay valid for up to 15 minutes. So revocation also writes revoked-<tokenId> to Redis for each session, and the auth middleware checks that key on every request. The key expires after 30 minutes, which is longer than any access token lives, so it never needs cleaning up. Logout uses the same key, which is how a single-device logout takes effect immediately.

That one Redis read per request is the price of instant revocation. I think it's the right price. A purely stateless access token means "logged out" really means "logged out in up to 15 minutes".

One session per device

Each device gets one refresh record, enforced by a unique index on user, device and token type. Signing in again on the same device replaces its record instead of piling up new ones. The user can list their sessions, with device name and last activity, and end any of them from their account.

The reuse rule has a cost. Two tabs on the same device refreshing at nearly the same moment can look like a replay: the first rotates the token, the second arrives with the old one. RedAuth treats that as theft and signs the user out. I chose that over trying to tell the cases apart, but a client that refreshes from several tabs should share one refresh in flight.

The other parts

• Rate limits are a sliding window in a Redis sorted set, keyed by IP, so they hold across several server instances: 100 requests a minute overall, 10 per five minutes on auth routes, 3 per five minutes on anything that sends email.
• Emails, like the alert sent on each new login, go through a BullMQ queue with retries and backoff, so nobody waits on an SMTP server to sign in.
• Login, forgot-password and resend-verification do the same work whether or not the account exists, so response timing doesn't reveal which emails are registered.
• Only hashes are stored: bcrypt for passwords, SHA-256 for tokens. A leaked database dump has nothing in it you can replay.

None of this is my invention. Refresh token rotation with reuse detection is in the OAuth security best practice, RFC 9700. Building it myself was the fastest way to understand why each part is there.

If I had to keep one rule from this project for the next auth system I build, it's the first one: a refresh token is good exactly once.

There's no texture made per garment or per fabric. The photograph of the fabric that's already in the catalogue is the texture.

Swapping the material

The garments are GLB models loaded with React Three Fiber and drei. When a fabric image is ready, the scene is cloned and every mesh in it gets the same new material:

const tex = texture.clone();
tex.wrapS = tex.wrapT = THREE.RepeatWrapping;
tex.repeat.set(textureScale[0], textureScale[1]);

const material = new THREE.MeshStandardMaterial({
  map: tex,
  roughness: 1.0,   // cloth doesn't shine
  metalness: 0.0,
  color: new THREE.Color(1.15, 1.15, 1.15),
});

const scene = gltf.scene.clone();
scene.traverse((child) => {
  if (child.isMesh) child.material = material;
});

Roughness at 1 and metalness at 0 make the surface fully matte, which is what woven cotton looks like. The colour of 1.15 is a brightness boost: a texture's colour is multiplied by the material's, so a value just over 1 lifts the photo slightly.

The repeat is per garment

A swatch photo covers a few centimetres of cloth. Stretched once across a whole jacket it looks like a giant blurry print. RepeatWrapping tiles it instead, and the repeat count decides how big the weave reads.

Every model was authored at a different scale with different UV layouts, so there's no single right number. The shirt and trousers use a repeat of 3.6. The jacket needed 5, and one of my commits is literally "Fixed texture scale on jacket". The models themselves needed different scale factors too, from 0.003 for the jacket to 2.5 for the shirt, because they came from different sources.

That's also why the texture is cloned before the repeat is set. repeat lives on the texture, not the material, so sharing one texture between two garments would make them fight over its tiling.

Keeping it fast

Buyers click through colours quickly, and every colour is a different photo. A module-level cache keyed by URL keeps each texture loaded once for the page's lifetime, and textures are disposed when they're evicted so the GPU memory comes back. Three's own cache is switched on for the model files. Mipmaps are generated so the weave doesn't shimmer when the garment is rotated away from the camera.

Each effect that loads a texture also tracks whether it's still current, so a slow image that arrives after the buyer has picked another colour is dropped instead of painting the wrong fabric.

It isn't physically accurate. There's no drape simulation, and a heavy twill looks as soft as a voile. It answers the question buyers actually had, which was how the pattern looks on a garment, and it does it with photos the client was already taking.