Lee ChanyangBackend Engineer

Language

← Back to design approaches

Process Evidence · 06

Verify behavior after failure through operational and recovery criteria

I retained failure points as states visible to operators and distinguished duplicate execution, partial failure, and reprocessing criteria.

I designed rerun and recovery criteria using idempotency keys, row locks, the Outbox, batch cursors, FAILED/DEAD states, and operational documents.

How I worked

Actual Working Approach

  1. Input

    I examined implementation code, database constraints, failure logs, batch cursors, queue states, and operational reprocessing conditions together.

  2. Questions to ask

    I checked whether the system could recover to the same result after duplicate execution, partial failure, deployment interruption, or external-service failure.

  3. Output

    I documented idempotency keys, row locks, the Outbox, cursors, FAILED/DEAD states, reprocessing SQL, queue-backlog diagnostics, and criteria for recovering long-stalled states.

Source evidence

Classifying operational evidence by failure states, reprocessing units, and direct contribution scope

For the operations and recovery card, I gathered material showing the state left at a stopping point, the unit that can be rerun, and whether the implementation was within my direct contribution scope, rather than focusing on normal success.

Recovery reflections
I linked reflections on Outbox/NotificationDelivery recovery, Transactional Outbox two-stage retries, recovery from database-finalization failure after gateway authorization, self-rescheduling settlement batches, and on-chain cursor batches.
Runbooks
Operational documents specifying procedures for DEAD records, queue backlogs, long-stalled ENQUEUED/SENDING states, Notification DEAD, and retries of transitions to PENDING serve as recovery criteria.
Code evidence
I cross-checked Outbox retry/DEAD repositories, BullMQ publishing and consumption, settlement-batch execution, rescheduling and stalled-state restoration, and subscription-renewal internal-finalization retry handlers as implementation evidence.
Exclusion criteria
Routine/Daily records serve only as work-tracking evidence. Claims of actual operational recovery are used only where operational documents, code, and reflections are all available.
Detailed work example

Verifying rerun capability by preserving stopping points in state and logs

After implementation, I checked not only normal success but also which states operators could inspect and what they could rerun after duplicate execution, partial failure, queue backlogs, or batch interruptions.

Verifying that the implementation is operableMermaid

Rendering the diagram.

Where This Led to Implementation

  • Points Accounting & Rewards System

    In Outbox · Queue incident documentation, I organized DEAD records, queue backlogs, long-stalled ENQUEUED/SENDING states, and Notification DEAD into distinct symptoms, diagnostics, and recovery procedures.

    View Outbox and notification recovery boundaries
  • On-Chain Activity Rewards & Statistics Data Pipeline

    I separated on-chain statistics into search windows, processing chunks, and statistics-job cursor updates, creating a batch flow that can resume after interruption.

    View cursor-based batch processing
Input
I examined implementation code, database constraints, failure logs, batch cursors, queue states, and operational reprocessing conditions together.
Failure scenarios
I checked which records serve as the recovery baseline after duplicate execution, partial failure, deployment interruption, external-service failure, queue backlogs, or long-stalled states.
Recovery
Rather than reprocessing everything, I separated recovery paths by failure scope, such as type-specific DEAD recovery, internal-finalization retries, and cursor-based restarts.
Output
I documented idempotency keys, row locks, the Outbox, cursors, FAILED/DEAD states, reprocessing SQL, queue-backlog diagnostics, and criteria for recovering long-stalled states.
Verification criteria

Checking operational recovery together with direct contribution scope

I do not claim operability from pattern names alone. I document which states inform which reprocessing actions, together with the scope of my direct implementation work.

Where This Led to Implementation

  • Points Accounting & Rewards System

    The project details show Outbox, notification, and settlement reruns and their operational recovery boundaries.

    View settlement and recovery design
  • Large-Scale Rewards & Statistics Pipeline

    Core domain and batch-processing implementations can be examined within the publicly shareable scope of my direct work.

    View direct contribution evidence
Recovery evidence
I documented distinct recovery paths for Outbox DEAD, long-stalled ENQUEUED/SENDING states, queue backlogs, payment internal-finalization retries, and batch-cursor resumption.
Direct contribution
For recent payment and points projects, I show the scope of my sole backend responsibility. For the on-chain case, I show only the publicly shareable scope of domain implementation, batch processing, and administrator workflows.
Portfolio links
Related implementation examples link directly to the relevant recovery, batch, or infrastructure section, rather than to a generic project overview.