RCA-140826

1. Incident Summary

During the afternoon of August 14, a partial degradation occurred in scheduled task processing, with impact observed around 16:06 (Ecuador Time / ECT), causing delays in dependent features—primarily ticket management and scheduled message delivery. Following an initial partial recovery, the service remained under observation until a subsequent surge in request volume required scaling up the available capacity of the database supporting this processing pipeline. The capacity increase was completed at 17:45, enabling progressive recovery and subsequent normalization of task execution.

Timeline (High Level):

Incident Window: 16:06–18:06 ECT, with partial recovery during incident management and full normalization at 18:06.

2. Impact

The component responsible for scheduled task execution experienced latency during the incident window, impacting dependent features, primarily ticket management and scheduled message delivery. Unrelated platform functionalities continued operating normally. Impact was limited strictly to customers whose operations required the execution of these specific tasks during the incident window.

3. Detection

The incident was detected via internal monitoring signals alongside incoming customer reports regarding delays in scheduled message delivery and ticket management operations. These signals facilitated rapid impact assessment and guided the technical diagnostic effort.

4. Incident Response

Upon detection, engineering identified that the spike in request volume had exceeded the provisioned capacity of the supporting database, leading to task queuing and elevated processing latency. As a remediation measure, available database capacity was expanded, allowing the accumulated backlog to drain and progressively restoring normal processing throughput. By 18:06, standard service performance had fully resumed.

5. Root Cause

The root cause was insufficient database capacity to sustain an unanticipated, significant traffic surge within the scheduled task processing layer. The sudden spike acted as the primary trigger, exhausting provisioned capacity and causing backlog buildup with increased execution latency. No application bugs or logic defects were identified in connection with this event.

6. Resolution

The immediate remediation was scaling up the provisioned database capacity, enabling the system to drain queued tasks and progressively restore processing throughput. Following mitigation, telemetry verified that processing latency returned to baseline metrics, and the incident was marked as resolved.

7. Preventive Actions

A fix has been implemented by the provider and we are currently monitoring the results to ensure full service stability is restored.

We will continue to provide updates as the situation progresses. Thank you for your patience.

Incident Summary

On July 14, 2026, an intermittent disruption affected the biometric validation services. During the incident window, a portion of validation requests failed to start or complete, while others were processed successfully. The incident impacted a subset of customers and end users attempting to use the service during this period.

The root cause was identified as an internal processing lock, in which multiple requests became blocked while waiting on shared resources, preventing normal operation. The service was fully restored the same day after the lock was released and request processing was stabilized.

Impact

The biometric validation services experienced intermittent failures for approximately 4 hours and 35 minutes. During this period, a portion of validation requests could not be initiated or completed, while others continued to be processed normally.

The incident affected only a subset of validation attempts made during the incident window. Its severity was considered significant due to the operational impact on biometric validation processes.

Detection

The incident was detected through platform monitoring and confirmed by customer reports indicating intermittent service failures.

Detection occurred almost immediately after the impact began (approximately 12:40 p.m. UTC-5), resulting in a minimal detection gap. Most of the incident duration was dedicated to diagnosing the root cause and implementing the mitigation.

Response

Once the incident was detected, the Engineering team initiated an investigation and identified an internal lock in the request processing flow, where multiple requests became blocked while waiting on shared resources, preventing part of the validation workload from being processed.

As an immediate mitigation, the processing lock was released and request processing was stabilized, gradually restoring normal service operation. Continuous monitoring was then performed until full service recovery was confirmed.

Root Cause

The root cause was an internal lock in the request processing mechanism. During operation, multiple requests became blocked while waiting on shared resources, creating a condition that prevented part of the system from continuing to process biometric validations.

The observed symptom was the inability to start or complete some biometric validations. The triggering event was the request processing lock, while the root cause was related to the way shared resources were managed during concurrent request processing, allowing this blocking condition to occur.

Resolution

As an immediate corrective action, the processing lock was released and service operation was stabilized, restoring the availability of the biometric validation service.

Recovery was verified through continuous monitoring until normal platform operation was confirmed.

As permanent corrective actions, improvements were implemented in memory management and critical resource handling, along with optimizations to the biometric validation services, to prevent similar conditions from impacting service availability in the future.

Preventive Measures

We are continuing to make adjustments to stabilize the service and are closely monitoring its performance. Our team remains actively working to restore normal operations. We will share further updates through this channel as they become available.

We are applying a fix for this incident.

Incident Summary
On June 30, 2026, between approximately 16:30 and 17:00 (GMT-5), a temporary intermittency of approximately 30 minutes occurred in one of the platform's highest-demand databases. Under high traffic volume, certain costly query operations executed by long-running sessions progressively consumed all available connections until they were exhausted, causing transaction rejections and isolated processing delays. The root cause was identified and resolved, and the service was fully restored. As a preventive measure, automatic controls were reinforced on the highest-traffic databases.

Impact
During the incident window, some bots experienced intermittency and response delays across messaging channels (WhatsApp, Facebook, Instagram, and web), and in isolated cases, certain interactions were not completed — corresponding to transactions that were rejected due to the temporary connection saturation. The impact was transient and limited to the event window; no data loss occurred and the service was fully restored following the fix.

Detection
The incident was automatically detected by our infrastructure monitoring systems, which immediately notified the technical team upon identifying anomalous behavior in the affected database, enabling a timely response.

Response
Once the incident was detected, the technical team investigated the state of the affected database and identified a set of long-running sessions executing costly query operations that were holding and saturating the available connections, impacting transaction processing. As an immediate mitigation action, those sessions were terminated, releasing the connections and restoring normal transaction processing. In parallel, as a stabilization measure, the cache layer was reinforced to reduce load on the affected database and data access for the involved operations was optimized, ensuring the service remained stable under the traffic volume.

Root Cause
The root cause of the incident was connection saturation in the affected database. Under high traffic volume, a set of costly query operations executed by long-running sessions held connections for longer than usual and progressively consumed them until the available limit was exhausted; as a result, new transactions were intermittently rejected. The trigger was the sustained traffic volume toward that database; no deployment or code change was identified as associated with the start of the incident.

Resolution
The immediate action was to identify and terminate the sessions running those costly operations that were keeping connections saturated, which released them and normalized the service within minutes. As a permanent fix, the processing of these query operations was optimized to reduce their cost and duration under high-traffic scenarios, alongside a configuration review currently under evaluation. Additionally, automatic controls on the highest-traffic databases were reinforced, incorporating specific thresholds and alerts for connection usage and early detection of this type of saturation, enabling the pattern to be anticipated and addressed before it impacts the service. Recovery was confirmed through continuous monitoring of connection and resource usage metrics, which remained stable following resolution.

Preventive Measures

• The processing of these query operations in the affected database was optimized, reducing their cost and duration under high traffic.

• Automatic controls were reinforced with specific thresholds and alerts on connection usage, to detect and respond more swiftly to traffic spikes, accelerating detection and escalation for this type of saturation.

• Additional database configuration adjustments are currently under evaluation.