Incident Summary

On July 14, 2026, an intermittent disruption affected the biometric validation services. During the incident window, a portion of validation requests failed to start or complete, while others were processed successfully. The incident impacted a subset of customers and end users attempting to use the service during this period.

The root cause was identified as an internal processing lock, in which multiple requests became blocked while waiting on shared resources, preventing normal operation. The service was fully restored the same day after the lock was released and request processing was stabilized.

Impact

The biometric validation services experienced intermittent failures for approximately 4 hours and 35 minutes. During this period, a portion of validation requests could not be initiated or completed, while others continued to be processed normally.

The incident affected only a subset of validation attempts made during the incident window. Its severity was considered significant due to the operational impact on biometric validation processes.

Detection

The incident was detected through platform monitoring and confirmed by customer reports indicating intermittent service failures.

Detection occurred almost immediately after the impact began (approximately 12:40 p.m. UTC-5), resulting in a minimal detection gap. Most of the incident duration was dedicated to diagnosing the root cause and implementing the mitigation.

Response

Once the incident was detected, the Engineering team initiated an investigation and identified an internal lock in the request processing flow, where multiple requests became blocked while waiting on shared resources, preventing part of the validation workload from being processed.

As an immediate mitigation, the processing lock was released and request processing was stabilized, gradually restoring normal service operation. Continuous monitoring was then performed until full service recovery was confirmed.

Root Cause

The root cause was an internal lock in the request processing mechanism. During operation, multiple requests became blocked while waiting on shared resources, creating a condition that prevented part of the system from continuing to process biometric validations.

The observed symptom was the inability to start or complete some biometric validations. The triggering event was the request processing lock, while the root cause was related to the way shared resources were managed during concurrent request processing, allowing this blocking condition to occur.

Resolution

As an immediate corrective action, the processing lock was released and service operation was stabilized, restoring the availability of the biometric validation service.

Recovery was verified through continuous monitoring until normal platform operation was confirmed.

As permanent corrective actions, improvements were implemented in memory management and critical resource handling, along with optimizations to the biometric validation services, to prevent similar conditions from impacting service availability in the future.

Preventive Measures

We are continuing to make adjustments to stabilize the service and are closely monitoring its performance. Our team remains actively working to restore normal operations. We will share further updates through this channel as they become available.

We are applying a fix for this incident.

Incident Summary
On June 30, 2026, between approximately 16:30 and 17:00 (GMT-5), a temporary intermittency of approximately 30 minutes occurred in one of the platform's highest-demand databases. Under high traffic volume, certain costly query operations executed by long-running sessions progressively consumed all available connections until they were exhausted, causing transaction rejections and isolated processing delays. The root cause was identified and resolved, and the service was fully restored. As a preventive measure, automatic controls were reinforced on the highest-traffic databases.

Impact
During the incident window, some bots experienced intermittency and response delays across messaging channels (WhatsApp, Facebook, Instagram, and web), and in isolated cases, certain interactions were not completed — corresponding to transactions that were rejected due to the temporary connection saturation. The impact was transient and limited to the event window; no data loss occurred and the service was fully restored following the fix.

Detection
The incident was automatically detected by our infrastructure monitoring systems, which immediately notified the technical team upon identifying anomalous behavior in the affected database, enabling a timely response.

Response
Once the incident was detected, the technical team investigated the state of the affected database and identified a set of long-running sessions executing costly query operations that were holding and saturating the available connections, impacting transaction processing. As an immediate mitigation action, those sessions were terminated, releasing the connections and restoring normal transaction processing. In parallel, as a stabilization measure, the cache layer was reinforced to reduce load on the affected database and data access for the involved operations was optimized, ensuring the service remained stable under the traffic volume.

Root Cause
The root cause of the incident was connection saturation in the affected database. Under high traffic volume, a set of costly query operations executed by long-running sessions held connections for longer than usual and progressively consumed them until the available limit was exhausted; as a result, new transactions were intermittently rejected. The trigger was the sustained traffic volume toward that database; no deployment or code change was identified as associated with the start of the incident.

Resolution
The immediate action was to identify and terminate the sessions running those costly operations that were keeping connections saturated, which released them and normalized the service within minutes. As a permanent fix, the processing of these query operations was optimized to reduce their cost and duration under high-traffic scenarios, alongside a configuration review currently under evaluation. Additionally, automatic controls on the highest-traffic databases were reinforced, incorporating specific thresholds and alerts for connection usage and early detection of this type of saturation, enabling the pattern to be anticipated and addressed before it impacts the service. Recovery was confirmed through continuous monitoring of connection and resource usage metrics, which remained stable following resolution.

Preventive Measures

• The processing of these query operations in the affected database was optimized, reducing their cost and duration under high traffic.

• Automatic controls were reinforced with specific thresholds and alerts on connection usage, to detect and respond more swiftly to traffic spikes, accelerating detection and escalation for this type of saturation.

• Additional database configuration adjustments are currently under evaluation.

Incident Summary

An overload was recorded in the bot system that limited its ability to process incoming messages in a timely manner. As support resources became saturated, the service experienced excessive wait times, preventing end users from receiving automated responses during the event.

Impact

The impact was limited to the interruption of service flows that depend on real-time message processing and user responses. Static or independent flows continued to run normally.

Detection

The incident was identified through automatic performance alerts and high latency in bot responses, in addition to reports from the support team and customers experiencing failures in digital signature flows. This confirmed service degradation in the processing of interactive messages.

Response

Once the issue was identified, the engineering team diagnosed the root cause and deployed a fix in under 30 minutes from escalation, increasing the performance and capacity of the resources supporting the bot to normalize the service.

Root Cause

The incident was triggered by a mass event caused by the simultaneous sending of campaigns by two customers. This generated a sudden and concentrated spike in activity volume that exceeded the response speed of the automatic scaling mechanisms, preventing the system from reacting in time to contain the load and causing temporary resource saturation.

Resolution

As a definitive resolution, scaling controls were adjusted and the base performance capacity of the bot infrastructure was increased. This adjustment ensures the system has the necessary headroom to absorb mass events and sudden activity spikes without degrading service.

Incident Summary

On May 19, 2026, during an automated hardening and infrastructure controls strengthening process, a temporary disruption occurred in some platform services. The intervention began at approximately 11:10 (UTC-5); the infrastructure team applied the corresponding fix at around 11:30, conducted behavior monitoring until 11:50, and confirmed full service stabilization at approximately 12:07.

Impact

The total period between the application of the change and full service stabilization was approximately one hour. During that interval, certain services related to workflows and integrations experienced intermittency and availability errors for approximately 30 minutes, while the team applied the fix and validated recovery. The impact was limited to temporary service availability. There was no data loss. There was no security compromise or unauthorized access.

Detection

The behavior was flagged by the service Health Status controls, which notify via Slack and email upon detection of disruptions, allowing the infrastructure team to immediately begin reviewing the event.

Response

Once the behavior was identified, the infrastructure team analyzed the affected services and components, determined the origin of the event, applied the fix at 11:30, carried out validations and post-fix monitoring until 11:50, and confirmed full platform stabilization at 12:07.

Root Cause

During an automated hardening and infrastructure controls strengthening process, an internal connectivity dependency between platform services reacted in an unexpected way. The dependency had a lower-than-optimal level of detail in the configuration inventory, which caused a temporary degradation in communication between components and, consequently, the transient unavailability of the affected services during the stabilization period.

Resolution

As part of the continuous improvement of controls already operated by the engineering team, the following mechanisms are being reinforced: deepening the validation controls applied before and after each infrastructure change, incorporating more thorough connectivity tests into the standard intervention protocol; expanding the scope of monitoring and early alerting mechanisms already in use, with more agile escalation to on-call teams to further shorten detection and response times; strengthening the internal connectivity dependency inventory maintained by the team, expanding its level of detail so that each future change is systematically cross-referenced against it prior to implementation; and reinforcing the incremental change protocol already in place, strengthening the practice of creating and validating new specific configurations before retiring previous ones. These measures enhance the operational maturity of the platform, reduce the risk of recurrence, and consolidate the resilience with which the team already manages infrastructure maintenance and strengthening activities. We sincerely apologize for any inconvenience caused and thank you for your understanding.

We are working on a fix for this incident.