Problem
Every crypto balance writer in FireblocksLedgerService (deposit, withdrawal, refund, transfer, conversion in/out, manual adjustment) computed balance_after from an unlocked read of user_crypto_balances and then upserted the absolute value. Two concurrent writers on the same row lost an update.
Example: a user has 100 USDC and converts 30, while a 50 USDC deposit webhook arrives at the same moment. The deposit reads the stale 100 and writes 150, but the correct balance is 120. The ledger entries are still complete, so the cache silently drifts from the ledger until reconciliation catches it.
Fireblocks webhooks are only serialised per transaction. Two deposits, or a deposit and a withdrawal for the same user, can therefore run in parallel without any user action.
Fix
-
calculateBalancesFromCachereads throughlockUserBalanceForUpdate(SELECT ... FOR UPDATE), andmanageris now required. This single change covers every writer. -
recordTransferlocks the sender and receiver rows up front, inpostLendingJournal's order: control rows first, then user rows by id. This avoids an A→B / B→A deadlock and matches the order used by the conversion provider, ramp reversal and lending. -
approveManualAdjustmentnow locks the HOUSE row before the user row, following the same global order.
Tests
-
New
fireblocks-ledger.concurrency.postgres.spec.ts, run on real Postgres over two connections:- a deposit blocked behind an open locking transaction (checked with
pg_blocking_pids) - two concurrent deposits
- opposite-direction transfers with no
40P01
- a deposit blocked behind an open locking transaction (checked with
-
Mutation checks:
- Removing the lock turns the spec red (expected 190, got 140).
- Removing the up-front transfer locks turns it red with
deadlock detected. - Swapping the manual-adjustment order back turns its new unit test red.
-
Results:
-
tscis clean, andyarn jest src/modules/fireblockspasses (180 tests). - The fireblocks-ledger DB specs pass.
- On MR !88 (merged) plus this branch, 34/34 conversion and ledger suites pass, including 16 simultaneous mixed-direction converts with no deadlock.
-
-
Already failing on develop (not caused by this branch):
- The ramp-order and lending postgres specs fail at seed with
CK_user_crypto_balances_account_code, identically on the base commit. -
admin-lending.risk.spec.tsfails with a Nest DI error and is unrelated to this diff.
- The ramp-order and lending postgres specs fail at seed with
Opus review gate: PASS after 2 rounds.
Follow-ups (not in this MR)
- The internal user-to-user transfer checks the sender's balance outside the transaction and without a lock (
fireblocks.service.ts~2857). Two concurrent transfers can therefore push the sender negative. -
syncUserBalanceis dead code and still does an unlocked read with an absolute write. - Lock-order inversion on USD accounts between Convert and card deposit/transfer. It is rare, can cause a 500 on the deposit side, and never misposts money.