SiteSpect Initiating Bypass

Incident Report for SiteSpect Cloud

Postmortem

SiteSpect Cloud Incident RCA: 25 March 2020

At 16:07 UTC, SiteSpect Engineering was alerted to an issue involving our DNS service. At approximately 16:40 UTC, SiteSpect decided to initiate a cloud wide bypass to mitigate any potential service degradation.

At 18:00 UTC, SiteSpect implemented a solution in our staging environments. At approximately 18:30 UTC, the solution was implemented in production and SiteSpect service was restored for all SiteSpect Cloud customers.

The investigation uncovered that the third party service used by DNS "bind" software (dlv.isc.org) had inadvertently removed their own DNS A record, which prevented the SiteSpect platform from resolving origin hostnames. Besides SiteSpect, this impacted many other organizations worldwide. SiteSpect implemented a workaround to resolve the issue specifically for SiteSpect Cloud. The SiteSpect solution was implemented prior to ISC issuing a correction for the issue around 21:30 UTC.

Despite the issue being fixed by ISC, SiteSpect has decided to discontinue use of the affected feature set. This will not affect the service level or how Customers are able to use the platform.

Posted Mar 25, 2020 - 21:15 UTC

Resolved

SiteSpect has implemented the solution. Service was restored at approximately 18:30 UTC.

Posted Mar 25, 2020 - 19:18 UTC

Identified

The SiteSpect Engineering team has identified the issue and are currently working on a solution. We will continue to keep you updated.

Posted Mar 25, 2020 - 18:07 UTC

Investigating

At approximately 16:15 UTC, the SiteSpect Engineering team was alerted to a potential issue and promptly started an investigation. In effort to prevent any service degradation, we are initiating bypass for all clients. We will update the status page as our investigation progresses.

Posted Mar 25, 2020 - 16:39 UTC

This incident affected: SiteSpect Cloud.