NASCAR - Reports of high volume of heartbeat messages on ERDP – Incident details

One or more systems are currently offline

Reports of high volume of heartbeat messages on ERDP

Resolved
Major outage
Started about 2 months agoLasted 1 day

Affected

Event Racing Data Platform (ERDP) - Production

Degraded performance from 10:09 PM to 10:11 PM, Operational from 10:09 PM to 7:34 PM, Degraded performance from 10:11 PM to 2:26 AM, Operational from 2:26 AM to 4:55 AM

Timing Data (Loop)

Degraded performance from 10:09 PM to 2:26 AM, Operational from 2:26 AM to 4:55 AM

Pit Road Data

Operational from 10:09 PM to 7:34 PM, Degraded performance from 7:34 PM to 2:26 AM, Operational from 2:26 AM to 4:55 AM

Updates
  • Resolved
    UTC
    Resolved

    The data flow appeared normal for the NOAPS race. We are marking this issue as resolved, but will re-evaluate before the Cup race 07/05

  • Update
    UTC
    Update

    No longer seeing duplicates. Will continue monitoring.

  • Monitoring
    UTC
    Monitoring
    We implemented a fix and are currently monitoring the result.
  • Update
    UTC
    Update

    Engineers are continuing to troubleshoot the issue to identify a potential fix.

  • Investigating
    UTC
    Investigating

    Engineers are continuing to troubleshoot the issue to identify a potential fix. Affected ERDP data includes timing data and potentially pit road data. Duplicate messages may be sent inconsistently until a fix can be implemented.

  • Identified
    UTC
    Identified

    Engineers are working on a potential fix.

  • Investigating
    UTC
    Investigating

    We are still investigating, we will post another update in around 45 minutes.

  • Identified
    UTC
    Identified

    We still do not have a solution to the duplicate messages. We have been investigating consistently and will continue to look into this issue. As of right now, Pit Road data will also be marked as degraded since we are seeing duplicate pitroad messages as well.

  • Update
    UTC
    Update

    As of 07/04 we are still investigating this and working on a solution/workaround

  • Update
    UTC
    Update

    This is also likely to cause increased latency on ERDP and connected systems

  • Investigating
    UTC
    Investigating
    We are currently investigating this incident.