Canada backend jobs processing system backlogs

Incident Report for Collabspace

Resolved

This incident has been resolved.
Posted Nov 14, 2023 - 18:18 PST

Update

We estimate the backlogs will be clear within the next 24 hours.
Posted Nov 13, 2023 - 18:50 PST

Update

We're continuing to monitor as the backlogs are processed.
Posted Nov 11, 2023 - 18:16 PST

Monitoring

Around 2023-11-09 at 6:00 PM PST, a backend job processing system began experiencing significant delays in processing jobs that ingest and index content due to a failure of a database component that is used to manage these jobs. The system is operational, however there are significant delays in processing the backlog of jobs that was present and has accumulated since the start of this event.

We are working to process the backlogs. It may take several hours for changes made in content sources to appear in Collabspace during this time. We do not yet have estimates for how long it will take to process the backlog.
Posted Nov 10, 2023 - 09:24 PST
This incident affected: Canada Region (Indexing, SharePoint Online/OneDrive Connector, Exchange Online Connector, Fileshare Connector).