Jelou - Performance Degradation in Biometric Agents – Incident details

All systems operational

Performance Degradation in Biometric Agents

Resolved
Major outage
Started 1 day agoLasted about 6 hours

Affected

API

Major outage from 5:00 PM to 7:45 PM, Degraded performance from 7:45 PM to 8:39 PM, Operational from 8:39 PM to 9:37 PM, Major outage from 9:37 PM to 10:35 PM

API de biometría

Major outage from 5:00 PM to 7:45 PM, Degraded performance from 7:45 PM to 8:39 PM, Operational from 8:39 PM to 9:37 PM, Major outage from 9:37 PM to 10:35 PM

Updates
  • Postmortem
    UTC
    Postmortem

    Incident Summary

    On July 14, 2026, an intermittent disruption affected the biometric validation services. During the incident window, a portion of validation requests failed to start or complete, while others were processed successfully. The incident impacted a subset of customers and end users attempting to use the service during this period.

    The root cause was identified as an internal processing lock, in which multiple requests became blocked while waiting on shared resources, preventing normal operation. The service was fully restored the same day after the lock was released and request processing was stabilized.

    Impact

    The biometric validation services experienced intermittent failures for approximately 4 hours and 35 minutes. During this period, a portion of validation requests could not be initiated or completed, while others continued to be processed normally.

    The incident affected only a subset of validation attempts made during the incident window. Its severity was considered significant due to the operational impact on biometric validation processes.

    Detection

    The incident was detected through platform monitoring and confirmed by customer reports indicating intermittent service failures.

    Detection occurred almost immediately after the impact began (approximately 12:40 p.m. UTC-5), resulting in a minimal detection gap. Most of the incident duration was dedicated to diagnosing the root cause and implementing the mitigation.

    Response

    Once the incident was detected, the Engineering team initiated an investigation and identified an internal lock in the request processing flow, where multiple requests became blocked while waiting on shared resources, preventing part of the validation workload from being processed.

    As an immediate mitigation, the processing lock was released and request processing was stabilized, gradually restoring normal service operation. Continuous monitoring was then performed until full service recovery was confirmed.

    Root Cause

    The root cause was an internal lock in the request processing mechanism. During operation, multiple requests became blocked while waiting on shared resources, creating a condition that prevented part of the system from continuing to process biometric validations.

    The observed symptom was the inability to start or complete some biometric validations. The triggering event was the request processing lock, while the root cause was related to the way shared resources were managed during concurrent request processing, allowing this blocking condition to occur.

    Resolution

    As an immediate corrective action, the processing lock was released and service operation was stabilized, restoring the availability of the biometric validation service.

    Recovery was verified through continuous monitoring until normal platform operation was confirmed.

    As permanent corrective actions, improvements were implemented in memory management and critical resource handling, along with optimizations to the biometric validation services, to prevent similar conditions from impacting service availability in the future.

    Preventive Measures

    #

    Action

    Type

    Owner

    Status

    1

    Implement improvements to memory management and critical resource handling to prevent request processing locks.

    Code

    Engineering Team

    Completed

    2

    Optimize the biometric validation services to improve stability and operational performance.

    Code

    Engineering Team

    Completed

    3

    Strengthen monitoring and alerting to enable early detection of request processing lock conditions.

    Observability

    Engineering Team

    In Progress

  • Resolved
    UTC
    Resolved
    This incident has been resolved.
  • Update
    UTC
    Update

    We are continuing to make adjustments to stabilize the service and are closely monitoring its performance. Our team remains actively working to restore normal operations. We will share further updates through this channel as they become available.

  • Monitoring
    UTC
    Monitoring
    We implemented a fix and are currently monitoring the result.
  • Identified
    UTC
    Identified

    We are applying a fix for this incident.

  • Investigating
    UTC
    Investigating

    We are currently experiencing intermittent issues with our biometric agents, which may affect the processing of biometric validations. As a result, some users may experience occasional issues during the validation process.

    Our team is actively investigating the situation and working to restore normal service as soon as possible. We appreciate your patience and understanding.