Jelou - Degradation in Conversation Processing, Routing, and Automated Workflows – Detalles del incidente

RCA-140826

1. Incident Summary

During the afternoon of August 14, a partial degradation occurred in scheduled task processing, with impact observed around 16:06 (Ecuador Time / ECT), causing delays in dependent features—primarily ticket management and scheduled message delivery. Following an initial partial recovery, the service remained under observation until a subsequent surge in request volume required scaling up the available capacity of the database supporting this processing pipeline. The capacity increase was completed at 17:45, enabling progressive recovery and subsequent normalization of task execution.

Timeline (High Level):

Incident Window: 16:06–18:06 ECT, with partial recovery during incident management and full normalization at 18:06.

2. Impact

The component responsible for scheduled task execution experienced latency during the incident window, impacting dependent features, primarily ticket management and scheduled message delivery. Unrelated platform functionalities continued operating normally. Impact was limited strictly to customers whose operations required the execution of these specific tasks during the incident window.

3. Detection

The incident was detected via internal monitoring signals alongside incoming customer reports regarding delays in scheduled message delivery and ticket management operations. These signals facilitated rapid impact assessment and guided the technical diagnostic effort.

4. Incident Response

Upon detection, engineering identified that the spike in request volume had exceeded the provisioned capacity of the supporting database, leading to task queuing and elevated processing latency. As a remediation measure, available database capacity was expanded, allowing the accumulated backlog to drain and progressively restoring normal processing throughput. By 18:06, standard service performance had fully resumed.

5. Root Cause

The root cause was insufficient database capacity to sustain an unanticipated, significant traffic surge within the scheduled task processing layer. The sudden spike acted as the primary trigger, exhausting provisioned capacity and causing backlog buildup with increased execution latency. No application bugs or logic defects were identified in connection with this event.

6. Resolution

The immediate remediation was scaling up the provisioned database capacity, enabling the system to drain queued tasks and progressively restore processing throughput. Following mitigation, telemetry verified that processing latency returned to baseline metrics, and the incident was marked as resolved.

7. Preventive Actions