StarRez Cloud Status - Service Disruption - Core Applications

Service Disruption - Core Applications - Australia East

Incident Report for StarRez Cloud

Postmortem

StarRez Root Cause Analysis
Australia East Outage - 7th Nov 2022

Summary
On the 7th Nov 2022, a subset of customers within the Australia East region experienced downtime to a number of core services (StarRez Web, PortalX and Schedule Service)
This lead to a downtime event of ~30min for customer applications whilst erroneous infrastructure was removed from service.

Root Cause
The root cause was determined to be an underlying node that was experiencing 2 known bugs that StarRez engineers are currently working with our upstream vendor to resolve.
These bugs impact DNS resolution and network access within the application, preventing access to the backend database.

Resolution
The problematic node was removed from service and customer applications move to healthy infrastructure within the same cluster.

StarRez engineers will continue to work with our upstream vendor to resolve this ongoing bug within the platform.

Posted Nov 07, 2022 - 08:41 UTC

Resolved

This incident has been resolved.

Posted Nov 07, 2022 - 02:54 UTC

Monitoring

All sites are now online.
Engineers will be monitoring this region closely and will close this incident out once stability has been confirmed.

-Next update as warranted by a change of events.

StarRez Team

Posted Nov 06, 2022 - 23:42 UTC

Identified

The underlying infrastructure causing this outage has been found.
Engineers are currently removing this infrastructure from the platform with sites now starting to come back online.

-Next update expected within 60 minutes, or as warranted by a change of events.

Apologies for any inconvenience,
StarRez Team

Posted Nov 06, 2022 - 23:01 UTC

Investigating

Customers in some regions are experiencing a service disruption with Core Applications within the Australia East.
-Engineers are actively working to remediate the issue.
-Next update expected within 60 minutes, or as warranted by a change of events.

Apologies for any inconvenience,
StarRez Team

Posted Nov 06, 2022 - 22:55 UTC