Affected
Major outage from 5:00 PM to 7:45 PM, Degraded performance from 7:45 PM to 8:39 PM, Operational from 8:39 PM to 9:37 PM, Major outage from 9:37 PM to 10:35 PM
Major outage from 5:00 PM to 7:45 PM, Degraded performance from 7:45 PM to 8:39 PM, Operational from 8:39 PM to 9:37 PM, Major outage from 9:37 PM to 10:35 PM
- PostmortemUTCPostmortemUTC
Incident Summary
On July 14, 2026, an intermittent disruption affected the biometric validation services. During the incident window, a portion of validation requests failed to start or complete, while others were processed successfully. The incident impacted a subset of customers and end users attempting to use the service during this period.
The root cause was identified as an internal processing lock, in which multiple requests became blocked while waiting on shared resources, preventing normal operation. The service was fully restored the same day after the lock was released and request processing was stabilized.
Impact
The biometric validation services experienced intermittent failures for approximately 4 hours and 35 minutes. During this period, a portion of validation requests could not be initiated or completed, while others continued to be processed normally.
The incident affected only a subset of validation attempts made during the incident window. Its severity was considered significant due to the operational impact on biometric validation processes.
Detection
The incident was detected through platform monitoring and confirmed by customer reports indicating intermittent service failures.
Detection occurred almost immediately after the impact began (approximately 12:40 p.m. UTC-5), resulting in a minimal detection gap. Most of the incident duration was dedicated to diagnosing the root cause and implementing the mitigation.
Response
Once the incident was detected, the Engineering team initiated an investigation and identified an internal lock in the request processing flow, where multiple requests became blocked while waiting on shared resources, preventing part of the validation workload from being processed.
As an immediate mitigation, the processing lock was released and request processing was stabilized, gradually restoring normal service operation. Continuous monitoring was then performed until full service recovery was confirmed.
Root Cause
The root cause was an internal lock in the request processing mechanism. During operation, multiple requests became blocked while waiting on shared resources, creating a condition that prevented part of the system from continuing to process biometric validations.
The observed symptom was the inability to start or complete some biometric validations. The triggering event was the request processing lock, while the root cause was related to the way shared resources were managed during concurrent request processing, allowing this blocking condition to occur.
Resolution
As an immediate corrective action, the processing lock was released and service operation was stabilized, restoring the availability of the biometric validation service.
Recovery was verified through continuous monitoring until normal platform operation was confirmed.
As permanent corrective actions, improvements were implemented in memory management and critical resource handling, along with optimizations to the biometric validation services, to prevent similar conditions from impacting service availability in the future.
Preventive Measures
#
Action
Type
Owner
Status
1
Implement improvements to memory management and critical resource handling to prevent request processing locks.
Code
Engineering Team
Completed
2
Optimize the biometric validation services to improve stability and operational performance.
Code
Engineering Team
Completed
3
Strengthen monitoring and alerting to enable early detection of request processing lock conditions.
Observability
Engineering Team
In Progress
- ResolvedUTCResolvedUTCThis incident has been resolved.
- UpdateUTCUpdateUTC
We are continuing to make adjustments to stabilize the service and are closely monitoring its performance. Our team remains actively working to restore normal operations. We will share further updates through this channel as they become available.
- MonitoringUTCMonitoringUTCWe implemented a fix and are currently monitoring the result.
- IdentifiedUTCIdentifiedUTC
We are applying a fix for this incident.
- InvestigatingUTCInvestigatingUTC
We are currently experiencing intermittent issues with our biometric agents, which may affect the processing of biometric validations. As a result, some users may experience occasional issues during the validation process.
Our team is actively investigating the situation and working to restore normal service as soon as possible. We appreciate your patience and understanding.