Merge order: stacked on !104 (merged) (
fix/withdraw-reconcile-stuck). Merge !104 (merged) first; GitLab then retargets this MR todevelop.
Summary
Crypto withdrawals no longer call Fireblocks inside the HTTP request. The request reserves the funds and returns PENDING_SUBMISSION. A BullMQ worker then submits to Fireblocks at a controlled rate, so a burst of withdrawals does not hit the Fireblocks rate limit.
Backend
-
Queue
queue.fireblocks_withdrawal: limiter 5 jobs/s, concurrency 1, 5 attempts with exponential backoff from 10 s,jobId= withdrawal row id (a replayed request never enqueues twice). -
Request path: Phase A (lock balance, debit, insert row) is unchanged. Instead of submitting, it enqueues and returns 200
PENDING_SUBMISSION. If the enqueue fails, it still returns 200 and the reconciliation cron picks the row up. -
Requests without a
requestId(old app builds) still submit synchronously, because their idempotency key cannot be rebuilt for a retry. - 429 and 409 from Fireblocks are transient: retried, never treated as a rejection, never refunded. Only a definite 4xx rejection refunds, exactly once (as in !104 (merged)).
-
Worker and cron are serialised per row with a Redis lock (
fireblocks:withdrawal-submit:<id>, 180 s). Inside the lock the row is re-read, and the submit is skipped (SUPERSEDED) if another path already linked it. -
Reconciler grace 5 → 15 min. Its attempt bump and escalation are conditional updates on rows that are still unlinked and
PENDING_SUBMISSION, so they cannot undo a link made by the worker. - The worker refuses jobs older than the 20 h idempotency window and leaves them to the cron's escalation (prevents a second transfer after a long outage).
-
FireblocksServicetakes the queue and Redis as new constructor arguments; specs updated.
Behaviour notes
- A definite Fireblocks rejection (e.g. a bad address) now reaches the user as 200
PENDING_SUBMISSIONfollowed by an async FAILED + refund, instead of an immediate 400. The app already polls the status and shows FAILED. - On the legacy no-
requestIdpath, a 429/409 now parks the row and the cron later escalates it to manual review, where before it was refunded. - Needs Redis/BullMQ running with the API. BullMQ already runs for send-mail, so this is not a new dependency.
Known, not fixed
- A 409 is logged with the
PROVIDER_429label (cosmetic). - The escalation log prints the batch copy's attempt count and error, which can be stale (cosmetic).
- Open question for Fireblocks support: if Fireblocks ever stored a 429 response under the idempotency key, retries would replay it for 24 h until escalation.
Test plan
-
BE unit (fireblocks + passcode coverage): 267 passed -
BE DB specs on moanapay_test(fireblocks): 255 passed -
yarn buildclean, eslint clean on changed files -
Every new assertion mutation-checked -
Review gate passed after 3 rounds (per-row lock, 409 as ambiguous, window guard and conditional reconciler updates came out of it)