SiteSpect Cloud Incident RCA: 25 March 2020
At 16:07 UTC, SiteSpect Engineering was alerted to an issue involving our DNS service. At approximately 16:40 UTC, SiteSpect decided to initiate a cloud wide bypass to mitigate any potential service degradation.
At 18:00 UTC, SiteSpect implemented a solution in our staging environments. At approximately 18:30 UTC, the solution was implemented in production and SiteSpect service was restored for all SiteSpect Cloud customers.
The investigation uncovered that the third party service used by DNS "bind" software (dlv.isc.org) had inadvertently removed their own DNS A record, which prevented the SiteSpect platform from resolving origin hostnames. Besides SiteSpect, this impacted many other organizations worldwide. SiteSpect implemented a workaround to resolve the issue specifically for SiteSpect Cloud. The SiteSpect solution was implemented prior to ISC issuing a correction for the issue around 21:30 UTC.
Despite the issue being fixed by ISC, SiteSpect has decided to discontinue use of the affected feature set. This will not affect the service level or how Customers are able to use the platform.