Skip to content
GitLab
Projects Groups Topics Snippets
  • /
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
    • Contribute to GitLab
  • Register
  • Sign in
  • A Ahad Mobile
  • Project information
    • Project information
    • Activity
    • Labels
    • Members
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributor statistics
    • Graph
    • Compare revisions
  • Issues 0
    • Issues 0
    • List
    • Boards
    • Service Desk
    • Milestones
  • Merge requests 0
    • Merge requests 0
  • CI/CD
    • CI/CD
    • Pipelines
    • Jobs
    • Schedules
  • Deployments
    • Deployments
    • Environments
    • Releases
  • Packages and registries
    • Packages and registries
    • Package Registry
    • Terraform modules
  • Monitor
    • Monitor
    • Incidents
  • Analytics
    • Analytics
    • Value stream
    • CI/CD
    • Repository
  • Wiki
    • Wiki
  • Snippets
    • Snippets
  • Activity
  • Graph
  • Create a new issue
  • Jobs
  • Commits
  • Issue Boards
Collapse sidebar
  • holatech
  • Ahad Mobile
  • Merge requests
  • !105

feat(withdraw): submit crypto withdrawals through a rate-limited queue

  • Review changes

  • Download
  • Patches
  • Plain diff
Merged Khanh Pham requested to merge feat/withdraw-queue into develop Oct 07, 2026
  • Overview 0
  • Commits 4
  • Pipelines 2
  • Changes 15

Merge order: stacked on !104 (merged) (fix/withdraw-reconcile-stuck). Merge !104 (merged) first; GitLab then retargets this MR to develop.

Summary

Crypto withdrawals no longer call Fireblocks inside the HTTP request. The request reserves the funds and returns PENDING_SUBMISSION. A BullMQ worker then submits to Fireblocks at a controlled rate, so a burst of withdrawals does not hit the Fireblocks rate limit.

Backend

  • Queue queue.fireblocks_withdrawal: limiter 5 jobs/s, concurrency 1, 5 attempts with exponential backoff from 10 s, jobId = withdrawal row id (a replayed request never enqueues twice).
  • Request path: Phase A (lock balance, debit, insert row) is unchanged. Instead of submitting, it enqueues and returns 200 PENDING_SUBMISSION. If the enqueue fails, it still returns 200 and the reconciliation cron picks the row up.
  • Requests without a requestId (old app builds) still submit synchronously, because their idempotency key cannot be rebuilt for a retry.
  • 429 and 409 from Fireblocks are transient: retried, never treated as a rejection, never refunded. Only a definite 4xx rejection refunds, exactly once (as in !104 (merged)).
  • Worker and cron are serialised per row with a Redis lock (fireblocks:withdrawal-submit:<id>, 180 s). Inside the lock the row is re-read, and the submit is skipped (SUPERSEDED) if another path already linked it.
  • Reconciler grace 5 → 15 min. Its attempt bump and escalation are conditional updates on rows that are still unlinked and PENDING_SUBMISSION, so they cannot undo a link made by the worker.
  • The worker refuses jobs older than the 20 h idempotency window and leaves them to the cron's escalation (prevents a second transfer after a long outage).
  • FireblocksService takes the queue and Redis as new constructor arguments; specs updated.

Behaviour notes

  • A definite Fireblocks rejection (e.g. a bad address) now reaches the user as 200 PENDING_SUBMISSION followed by an async FAILED + refund, instead of an immediate 400. The app already polls the status and shows FAILED.
  • On the legacy no-requestId path, a 429/409 now parks the row and the cron later escalates it to manual review, where before it was refunded.
  • Needs Redis/BullMQ running with the API. BullMQ already runs for send-mail, so this is not a new dependency.

Known, not fixed

  • A 409 is logged with the PROVIDER_429 label (cosmetic).
  • The escalation log prints the batch copy's attempt count and error, which can be stale (cosmetic).
  • Open question for Fireblocks support: if Fireblocks ever stored a 429 response under the idempotency key, retries would replay it for 24 h until escalation.

Test plan

  • BE unit (fireblocks + passcode coverage): 267 passed
  • BE DB specs on moanapay_test (fireblocks): 255 passed
  • yarn build clean, eslint clean on changed files
  • Every new assertion mutation-checked
  • Review gate passed after 3 rounds (per-row lock, 409 as ambiguous, window guard and conditional reconciler updates came out of it)

🤖 Generated with Claude Code

Edited Oct 08, 2026 by Khanh Pham
Assignee
Assign to
Reviewers
Request review from
Time tracking
Source branch: feat/withdraw-queue