Process Evidence · 06
Verify behavior after failure through operational and recovery criteria
I retained failure points as states visible to operators and distinguished duplicate execution, partial failure, and reprocessing criteria.
I designed rerun and recovery criteria using idempotency keys, row locks, the Outbox, batch cursors, FAILED/DEAD states, and operational documents.
How I worked
Actual Working Approach
- Input
I examined implementation code, database constraints, failure logs, batch cursors, queue states, and operational reprocessing conditions together.
- Questions to ask
I checked whether the system could recover to the same result after duplicate execution, partial failure, deployment interruption, or external-service failure.
- Output
I documented idempotency keys, row locks, the Outbox, cursors, FAILED/DEAD states, reprocessing SQL, queue-backlog diagnostics, and criteria for recovering long-stalled states.
Classifying operational evidence by failure states, reprocessing units, and direct contribution scope
For the operations and recovery card, I gathered material showing the state left at a stopping point, the unit that can be rerun, and whether the implementation was within my direct contribution scope, rather than focusing on normal success.
- Recovery reflections
- I linked reflections on Outbox/NotificationDelivery recovery, Transactional Outbox two-stage retries, recovery from database-finalization failure after gateway authorization, self-rescheduling settlement batches, and on-chain cursor batches.
- Runbooks
- Operational documents specifying procedures for DEAD records, queue backlogs, long-stalled ENQUEUED/SENDING states, Notification DEAD, and retries of transitions to PENDING serve as recovery criteria.
- Code evidence
- I cross-checked Outbox retry/DEAD repositories, BullMQ publishing and consumption, settlement-batch execution, rescheduling and stalled-state restoration, and subscription-renewal internal-finalization retry handlers as implementation evidence.
- Exclusion criteria
- Routine/Daily records serve only as work-tracking evidence. Claims of actual operational recovery are used only where operational documents, code, and reflections are all available.
Verifying rerun capability by preserving stopping points in state and logs
After implementation, I checked not only normal success but also which states operators could inspect and what they could rerun after duplicate execution, partial failure, queue backlogs, or batch interruptions.
Rendering the diagram.
Where This Led to Implementation
- Points Accounting & Rewards System
In Outbox · Queue incident documentation, I organized DEAD records, queue backlogs, long-stalled ENQUEUED/SENDING states, and Notification DEAD into distinct symptoms, diagnostics, and recovery procedures.
View Outbox and notification recovery boundaries - On-Chain Activity Rewards & Statistics Data Pipeline
I separated on-chain statistics into search windows, processing chunks, and statistics-job cursor updates, creating a batch flow that can resume after interruption.
View cursor-based batch processing
- Input
- I examined implementation code, database constraints, failure logs, batch cursors, queue states, and operational reprocessing conditions together.
- Failure scenarios
- I checked which records serve as the recovery baseline after duplicate execution, partial failure, deployment interruption, external-service failure, queue backlogs, or long-stalled states.
- Recovery
- Rather than reprocessing everything, I separated recovery paths by failure scope, such as type-specific DEAD recovery, internal-finalization retries, and cursor-based restarts.
- Output
- I documented idempotency keys, row locks, the Outbox, cursors, FAILED/DEAD states, reprocessing SQL, queue-backlog diagnostics, and criteria for recovering long-stalled states.
Checking operational recovery together with direct contribution scope
I do not claim operability from pattern names alone. I document which states inform which reprocessing actions, together with the scope of my direct implementation work.
Where This Led to Implementation
- Points Accounting & Rewards System
The project details show Outbox, notification, and settlement reruns and their operational recovery boundaries.
View settlement and recovery design - Large-Scale Rewards & Statistics Pipeline
Core domain and batch-processing implementations can be examined within the publicly shareable scope of my direct work.
View direct contribution evidence
- Recovery evidence
- I documented distinct recovery paths for Outbox DEAD, long-stalled ENQUEUED/SENDING states, queue backlogs, payment internal-finalization retries, and batch-cursor resumption.
- Direct contribution
- For recent payment and points projects, I show the scope of my sole backend responsibility. For the on-chain case, I show only the publicly shareable scope of domain implementation, batch processing, and administrator workflows.
- Portfolio links
- Related implementation examples link directly to the relevant recovery, batch, or infrastructure section, rather than to a generic project overview.