Jelou - Performance Degradation in Biometric Agents – Detalles del incidente

Sistemas funcionando con normalidad

Performance Degradation in Biometric Agents

Resuelto
Interrupción mayor
Iniciado el hace 1 díaDuró alrededor de 6 horas

Afectado

API

Interrupción mayor de 5:00 PM a 7:45 PM, Rendimiento degradado de 7:45 PM a 8:39 PM, En funcionamiento de 8:39 PM a 9:37 PM, Interrupción mayor de 9:37 PM a 10:35 PM

API de biometría

Interrupción mayor de 5:00 PM a 7:45 PM, Rendimiento degradado de 7:45 PM a 8:39 PM, En funcionamiento de 8:39 PM a 9:37 PM, Interrupción mayor de 9:37 PM a 10:35 PM

Actualizaciones
  • Después de la muerte
    UTC
    Después de la muerte

    Incident Summary

    On July 14, 2026, an intermittent disruption affected the biometric validation services. During the incident window, a portion of validation requests failed to start or complete, while others were processed successfully. The incident impacted a subset of customers and end users attempting to use the service during this period.

    The root cause was identified as an internal processing lock, in which multiple requests became blocked while waiting on shared resources, preventing normal operation. The service was fully restored the same day after the lock was released and request processing was stabilized.

    Impact

    The biometric validation services experienced intermittent failures for approximately 4 hours and 35 minutes. During this period, a portion of validation requests could not be initiated or completed, while others continued to be processed normally.

    The incident affected only a subset of validation attempts made during the incident window. Its severity was considered significant due to the operational impact on biometric validation processes.

    Detection

    The incident was detected through platform monitoring and confirmed by customer reports indicating intermittent service failures.

    Detection occurred almost immediately after the impact began (approximately 12:40 p.m. UTC-5), resulting in a minimal detection gap. Most of the incident duration was dedicated to diagnosing the root cause and implementing the mitigation.

    Response

    Once the incident was detected, the Engineering team initiated an investigation and identified an internal lock in the request processing flow, where multiple requests became blocked while waiting on shared resources, preventing part of the validation workload from being processed.

    As an immediate mitigation, the processing lock was released and request processing was stabilized, gradually restoring normal service operation. Continuous monitoring was then performed until full service recovery was confirmed.

    Root Cause

    The root cause was an internal lock in the request processing mechanism. During operation, multiple requests became blocked while waiting on shared resources, creating a condition that prevented part of the system from continuing to process biometric validations.

    The observed symptom was the inability to start or complete some biometric validations. The triggering event was the request processing lock, while the root cause was related to the way shared resources were managed during concurrent request processing, allowing this blocking condition to occur.

    Resolution

    As an immediate corrective action, the processing lock was released and service operation was stabilized, restoring the availability of the biometric validation service.

    Recovery was verified through continuous monitoring until normal platform operation was confirmed.

    As permanent corrective actions, improvements were implemented in memory management and critical resource handling, along with optimizations to the biometric validation services, to prevent similar conditions from impacting service availability in the future.

    Preventive Measures

    #

    Action

    Type

    Owner

    Status

    1

    Implement improvements to memory management and critical resource handling to prevent request processing locks.

    Code

    Engineering Team

    Completed

    2

    Optimize the biometric validation services to improve stability and operational performance.

    Code

    Engineering Team

    Completed

    3

    Strengthen monitoring and alerting to enable early detection of request processing lock conditions.

    Observability

    Engineering Team

    In Progress

  • Resuelto
    UTC
    Resuelto
    This incident has been resolved.
  • Actualizar
    UTC
    Actualizar

    We are continuing to make adjustments to stabilize the service and are closely monitoring its performance. Our team remains actively working to restore normal operations. We will share further updates through this channel as they become available.

  • Supervisando
    UTC
    Supervisando
    We implemented a fix and are currently monitoring the result.
  • Identificado
    UTC
    Identificado

    We are applying a fix for this incident.

  • Investigando
    UTC
    Investigando

    We are currently experiencing intermittent issues with our biometric agents, which may affect the processing of biometric validations. As a result, some users may experience occasional issues during the validation process.

    Our team is actively investigating the situation and working to restore normal service as soon as possible. We appreciate your patience and understanding.