Incident Report

Summary

From Friday, 2 June 2023 to Saturday, 3 June 2023, some Google Workspace Administrators experienced issues performing List, Create, Update, or Delete operations for users and organizational units (OU) when using the Admin Console or Directory APIs. The total outage duration was 16 hours and 32 minutes. We have conducted an internal investigation and are taking steps to improve our service.

Root Cause

The Google Workspace Admin Console and Directory APIs use a backend service to authorize requests to create/update/delete/read user and OU resources. A workflow synchronizes user and OU resources to that backend authorization service.

On Friday, 2 June 2023 at 12:25 PDT, engineers rolled out a configuration change that inadvertently caused the workflow to stop synchronizing resources to the authorization service. Because the authorization service was not aware of the new resources (those created during the outage), it was unable to authorize requests and returned a 403 Unauthorized error. The incorrect configuration went undetected before being deployed to production because the automatic pre-submit checks had been suppressed because engineers incorrectly believed the change was safe.

Remediation and Prevention

Google engineers were alerted to the outage via a support case on 2 June 2023 at 19:33 PDT and immediately started an investigation. Engineers identified that the roll out of the configuration change aligned with the start of 403 Unauthorized errors and initiated a rollback on 3 June 2023 at 00:14 PDT. The rollback completed at 00:57 PDT, mitigating the issue for new users and OU resources created after that time. Engineers performed a backfill to synchronize the impacted users and OU resources that were created after the start of the issue, fully mitigating the issue at 04:57 PDT.

Google is committed preventing a repeat of this issue in the future and is completing the following actions:

• We will be updating the pre-submit checks for workflow configuration changes to better identify potential issues before rolling out to production.
• We will be adding a requirement to have a designated reviewer approve workflow configuration changes to ensure that automated presubmit checks are being run before rolling out to production.

Detailed Description of Impact

From 2 June 2023 at 12:25 PDT to 3 June 2023 at 04:57, Google Workspace Administrators may have experienced 403 Unauthorized errors when attempting to perform actions for user and OU resources via Admin Console or Directory APIs. This issue affected Create, Fetch, Update, Delete and other related operations for user and OU resources created between 2 June 2023 at 12:25 PDT and 3 June 2023 at 00:57 PDT.

Mini Incident Report

We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Workspace Support using help article https://support.google.com/a/answer/1047213.

(All Times US/Pacific)

Incident Start: 2 June 2023 at 12:25

Incident End: 3 June 2023 at 04:57

Duration: 16 hours, 32 minutes

Affected Services and Features:

Google Workspace Admin Console

Regions/Zones: Global

Description:

Some Google Workspace Administrators experienced issues performing List, Create, Update, or Delete operations for users and organizational units when using the Admin Console or Directory APIs. From preliminary analysis, the root cause of the issue was a configuration change that prevented resources (new or changes to existing resources) from synchronizing with a backend system used for authorizing requests. This resulted in 403 errors for attempted changes to these resources. The total outage duration was 16 hours and 32 minutes. Google will complete a full IR in the following days that will provide a detailed root cause.

Customer Impact:

Some Google Workspace Administrators experienced 403 Forbidden errors when performing List, Create, Update, or Delete operations for users and organizational units via the Admin Console or Directory APIs. The issue only affected resources created after Friday, 2 June 2023 at 12:25.

Additional details:

Engineers rolled back the configuration change to prevent additional resources from being impacted on Saturday, 3 June 2023 at 00:45. At 04:57, engineers fully mitigated impact to all resources by backfilling their data to the backend system.

Google engineering has released a fix and we believe that the issue with Admin Console was partially resolved on 2023-06-03, at 00:40:00 US/ Pacific.

As per the current status, the Google Workspace Administrators will be able to perform List, Create, Update, or Delete operations for any OrgUnits/ Users created after 2023-06-03, 00:40:00 US/ Pacific.

The operations on OrgUnits/ Users created during the impact window (from 2023-06-02, 12:37:00 to 2023-06-03, 00:40:00 US/ Pacific) may still experience the issue.

Our engineering team continues the investigation and the effort to fully resolve the issue is still in progress.

We do not have an ETA for full resolution at this point.

We will provide an update by Saturday, 2023-06-03 12:00 US/Pacific with current details.

Google engineering team has released a fix and we believe that the issue with Admin Console was partially resolved on June 3, 2023, at 07:40: AM UTC.

As per the current status, the Google Workspace Administrators will be able to perform List, Create, Update, or Delete operations for any OrgUnits/ Users created after June 3, 2023, 07:40: AM UTC.

The operations on OrgUnits/ Users created during the impact window (from June 2, 2023, 07:37 PM to June 3, 2023, 07:40: AM UTC) may still experience the issue.

Our engineering team continues the investigation and the effort to fully resolve the issue is still under progress. We do not have an ETA for full resolution at this point.

We will provide an update by Saturday, June 3, 2023 04:00 PM UTC with current details.

We're aware of a problem with Admin Console affecting a subset of Google Workspace Administrators. Some Google Workspace Administrators may experience issues when performing List, Create, Update, or Delete operations for OrgUnits using the Admin Console or Directory APIs. Attempting to navigate to the admin.google.com/ac/users/<user_id> path may result in an Error 403 Forbidden.

We will provide an update by Jun 3, 2023, 12:30 PM UTC detailing when we expect to resolve the problem. Please note that this resolution time is an estimate and may change.