NASCAR - Reports of high volume of heartbeat messages on ERDP – Incident details

All systems operational

Show current status

Reports of high volume of heartbeat messages on ERDP

ResolvedMajor outage30 hours 46 minutes

Affected

Event Racing Data Platform (ERDP) - Production

Degraded performance from 10:09 PM to 10:11 PM, Operational from 10:09 PM to 7:34 PM, Degraded performance from 10:11 PM to 2:26 AM, Operational from 2:26 AM to 4:55 AM

Timing Data (Loop)

Degraded performance from 10:09 PM to 2:26 AM, Operational from 2:26 AM to 4:55 AM

Pit Road Data

Operational from 10:09 PM to 7:34 PM, Degraded performance from 7:34 PM to 2:26 AM, Operational from 2:26 AM to 4:55 AM

  • ResolvedUTC

    The data flow appeared normal for the NOAPS race. We are marking this issue as resolved, but will re-evaluate before the Cup race 07/05

  • UpdateUTC

    No longer seeing duplicates. Will continue monitoring.

  • MonitoringUTC
    We implemented a fix and are currently monitoring the result.
  • UpdateUTC

    Engineers are continuing to troubleshoot the issue to identify a potential fix.

  • InvestigatingUTC

    Engineers are continuing to troubleshoot the issue to identify a potential fix. Affected ERDP data includes timing data and potentially pit road data. Duplicate messages may be sent inconsistently until a fix can be implemented.

  • IdentifiedUTC

    Engineers are working on a potential fix.

  • InvestigatingUTC

    We are still investigating, we will post another update in around 45 minutes.

  • IdentifiedUTC

    We still do not have a solution to the duplicate messages. We have been investigating consistently and will continue to look into this issue. As of right now, Pit Road data will also be marked as degraded since we are seeing duplicate pitroad messages as well.

  • UpdateUTC

    As of 07/04 we are still investigating this and working on a solution/workaround

  • UpdateUTC

    This is also likely to cause increased latency on ERDP and connected systems

  • InvestigatingUTC
    We are currently investigating this incident.