Penn Computing
Computing Menu Computing A-Z
Computing Home Information Systems & Computing Penn

Microsoft's Incident Report for PennO365

Following is a data extract from the PennO365 administrative console of incident reports sent by Microsoft within the last 30 days. We pull out incidents that are relevant to the current PennO365 service offerings and we refresh this data every 15 minutes. The last extract was taken on April 6, 2020 04:16.

--------------------------------------------------------------------------------------------------
Title Auto-Signature formatting issue in Outlook on the web
ID EX208783
Service Exchange Online
Feature E-Mail and calendar access
Status Service degradation
Impact Users may notice their auto-signature being formatted incorrectly when replying to or
Severity Sev2
Start time 2020-04-06T03:35:55Z
End time
Last update 2020-04-06T05:00:48.977Z
More info This issue impacts Outlook on the web only; no other protocols are affected.
Root cause A recent service update unintentionally caused a formatting code issue in Outlook on the web.

EX208783 Details

Status: We're investigating a potential issue and checking for impact to your organization. We'll provide an update within 30 minutes.; PublishedTime=2020-04-06T03:49:03.8

Status: We're analyzing system logs to determine the source of the issue.
Next update by: Monday, April 6, 2020, at 6:00 AMUTC; PublishedTime=2020-04-06T04:15:09.027Z

Status: We identified that a recent service update caused a code regression in Outlook on the web. We're working on a fix to remediate the impact.
Next update by: Monday, April 6, 2020, at 7:00 PMUTC; PublishedTime=2020-04-06T04:55:34.913Z

--------------------------------------------------------------------------------------------------
Title Admins can't access the admin center in the modern experience
ID SP208445
Service SharePoint Online
Feature SharePoint Features
Status Post-incident report published
Impact Admins may have been unable to access the SharePoint Online admin center in the modern
Severity Sev2
Start time 2020-04-02T21:56:00Z
End time 2020-04-03T01:31:00Z
Last update 2020-04-04T09:40:06.257Z
More info While we were focused on remediation, admins may have been able to access the admin center in the classic experience as well as PowerShell.
Root cause

SP208445 Details

Status: We're analyzing system logs to help identify the source of the issue.
Next update by: Friday, April 3, 2020, at 1:30 AMUTC; PublishedTime=2020-04-02T23:34:57Z

Status: Our investigation indicates that there may be a recent service update that could be causing inadvertent impact to admin center access. We're reviewing the update to determine if this is the source of the issue, how impact is occurring from the update, and what our options are for mitigating impact.
Next update by: Friday, April 3, 2020, at 3:30 AMUTC; PublishedTime=2020-04-03T01:25:44.107Z

Final status: We've confirmed that the recent service update intended to improve the client experience inadvertently caused impact when attempting to access the admin center in the modern experience. We've reverted this update and verified that the issue is resolved.
Next update by: Friday, April 3, 2020, at 3:30 AMUTC; PublishedTime=2020-04-03T01:25:44.107Z

--------------------------------------------------------------------------------------------------
Title Admin usage reports processing delays for Teams/Skype starting March 31
ID MO208476
Service Office 365 Portal
Feature Administration
Status Service degradation
Impact Admins may notice that usage reports for Teams and Skype for Business do not contain
Severity Sev2
Start time 2020-03-31T07:00:00Z
End time
Last update 2020-04-06T02:01:59.86Z
More info As of Friday, April 3, 2020, usage reports for Microsoft Teams are now up-to-date.
Root cause An unexpected increase in traffic is causing usage report delivery delays.

MO208476 Details

Status: Due to a large unexpected increase in data traffic, the data pipeline from Teams and Skype for Business is delayed.
Next update by: Monday, April 6, 2020, at 3:00 AMUTC; PublishedTime=2020-04-03T03:18:53.437Z

Status: We've reprocessed the data for Microsoft Teams and confirmed that reports are now up-to-date. We're continuing to monitor the service as the Skype for Business reports are processed.
Next update by: Monday, April 6, 2020, at 3:00 AMUTC; PublishedTime=2020-04-03T16:33:11.583Z

Status: We've confirmed that reports are up to date for the vast majority of the affected infrastructure. Users in the North American region may notice that there are some lingering delays for reports for Exchange Online, OneDrive for Business, SharePoint Online, Skype for Business, and Microsoft Teams. We're continuing to monitor the remaining services to ensure the reports are processed successfully.
Next update by: Tuesday, April 7, 2020, at 4:00 AMUTC; PublishedTime=2020-04-06T02:01:59.86Z

--------------------------------------------------------------------------------------------------
Title Users may be unable to access sites and content
ID SP208189
Service SharePoint Online
Feature SharePoint Features
Status Post-incident report published
Impact Users may have seen intermittent delays and errors accessing SharePoint Online sites and
Severity Sev2
Start time 2020-03-30T21:00:00Z
End time 2020-03-31T13:12:00Z
Last update 2020-04-02T16:44:09.63Z
More info Users may have been inadvertently throttled when accessing sites and content. Users who experienced this error may have seen latency, timeout errors, or have been re-directed to a page notifying them of the throttle.
Root cause

SP208189 Details

Status: We identified that there is an issue causing access problems in SharePoint Online and OneDrive for Business. We're analyzing system logs to help identify the source of the problem.
Next update by: Tuesday, March 31, 2020, at 6:00 AMUTC; PublishedTime=2020-03-31T04:48:27.963Z

Status: We've identified that users may be incorrectly throttled when accessing SharePoint Online sites, and are working to identify the potential cause of this throttling to formulate a mitigation strategy.
Next update by: Tuesday, March 31, 2020, at 7:00 AMUTC; PublishedTime=2020-03-31T05:49:53.977Z

Status: We've confirmed that a new throttling rule intended to optimize service availability is incorrectly acting against users, resulting in errors when accessing the service. We're working to deactivate this rule to mitigate impact.
Next update by: Tuesday, March 31, 2020, at 8:00 AMUTC; PublishedTime=2020-03-31T07:04:33.543Z

Status: We're continuing to deactivate the rule throughout the affected environment. Once the rule takes effect, we're monitoring the service to ensure full recovery. Users should see immediate relief once the fix reaches their environment.
Next update by: Tuesday, March 31, 2020, at 9:00 AM UTC; PublishedTime=2020-03-31T07:56:34.787Z

Status: We suspect that the timer jobs may be locked, preventing automation from applying the new configuration rule. We're checking timer jobs to unlock them. In parallel, we're confirming that the configuration rule has applied correctly elsewhere in the environment.
Next update by: Tuesday, March 31, 2020, at 11:00 AM UTC; PublishedTime=2020-03-31T09:52:39.63Z

Status: We've confirmed that the timer jobs were processing correctly but slowly. We're refreshing the throttle overrides which will restore service to the affected users. Additionally, we're deploying a fix to permanently address the slow-processing timer jobs.
Next update by: Tuesday, March 31, 2020, at 1:00 PM UTC; PublishedTime=2020-03-31T10:59:46.717Z

Status: We've encountered errors with timer jobs when applying the configuration change, meaning that some parts of the affected environment are unable to pick up the updated configuration. Users hosted there will not see recovery until the updated configuration is applied.
Next update by: Tuesday, March 31, 2020, at 10:00 AM UTC; PublishedTime=2020-03-31T08:56:21.853Z

Status: We've applied the throttle overrides and the fix has completed deployment. We're monitoring the environment to ensure the mitigations have applied successfully.
Next update by: Tuesday, March 31, 2020, at 3:00 PM UTC; PublishedTime=2020-03-31T12:24:15.733Z

Final status: After a period of monitoring, we can confirm that service has been restored.
Next update by: Tuesday, March 31, 2020, at 3:00 PM UTC; PublishedTime=2020-03-31T12:24:15.733Z

--------------------------------------------------------------------------------------------------
Title Users may be unable to access sites and content
ID SP208189
Service SharePoint Online
Feature SharePoint Features
Status Post-incident report published
Impact Users may have seen intermittent delays and errors accessing SharePoint Online sites and
Severity Sev2
Start time 2020-03-30T21:00:00Z
End time 2020-03-31T13:12:00Z
Last update 2020-04-02T16:44:09.63Z
More info Users may have been inadvertently throttled when accessing sites and content. Users who experienced this error may have seen latency, timeout errors, or have been re-directed to a page notifying them of the throttle.
Root cause

SP208189 Details

Status: We identified that there is an issue causing access problems in SharePoint Online and OneDrive for Business. We're analyzing system logs to help identify the source of the problem.
Next update by: Tuesday, March 31, 2020, at 6:00 AMUTC; PublishedTime=2020-03-31T04:48:27.963Z

Status: We've identified that users may be incorrectly throttled when accessing SharePoint Online sites, and are working to identify the potential cause of this throttling to formulate a mitigation strategy.
Next update by: Tuesday, March 31, 2020, at 7:00 AMUTC; PublishedTime=2020-03-31T05:49:53.977Z

Status: We've confirmed that a new throttling rule intended to optimize service availability is incorrectly acting against users, resulting in errors when accessing the service. We're working to deactivate this rule to mitigate impact.
Next update by: Tuesday, March 31, 2020, at 8:00 AMUTC; PublishedTime=2020-03-31T07:04:33.543Z

Status: We're continuing to deactivate the rule throughout the affected environment. Once the rule takes effect, we're monitoring the service to ensure full recovery. Users should see immediate relief once the fix reaches their environment.
Next update by: Tuesday, March 31, 2020, at 9:00 AM UTC; PublishedTime=2020-03-31T07:56:34.787Z

Status: We suspect that the timer jobs may be locked, preventing automation from applying the new configuration rule. We're checking timer jobs to unlock them. In parallel, we're confirming that the configuration rule has applied correctly elsewhere in the environment.
Next update by: Tuesday, March 31, 2020, at 11:00 AM UTC; PublishedTime=2020-03-31T09:52:39.63Z

Status: We've confirmed that the timer jobs were processing correctly but slowly. We're refreshing the throttle overrides which will restore service to the affected users. Additionally, we're deploying a fix to permanently address the slow-processing timer jobs.
Next update by: Tuesday, March 31, 2020, at 1:00 PM UTC; PublishedTime=2020-03-31T10:59:46.717Z

Status: We've encountered errors with timer jobs when applying the configuration change, meaning that some parts of the affected environment are unable to pick up the updated configuration. Users hosted there will not see recovery until the updated configuration is applied.
Next update by: Tuesday, March 31, 2020, at 10:00 AM UTC; PublishedTime=2020-03-31T08:56:21.853Z

Status: We've applied the throttle overrides and the fix has completed deployment. We're monitoring the environment to ensure the mitigations have applied successfully.
Next update by: Tuesday, March 31, 2020, at 3:00 PM UTC; PublishedTime=2020-03-31T12:24:15.733Z

Final status: After a period of monitoring, we can confirm that service has been restored.
Next update by: Tuesday, March 31, 2020, at 3:00 PM UTC; PublishedTime=2020-03-31T12:24:15.733Z

--------------------------------------------------------------------------------------------------
Title Call data may be delayed
ID TM208056
Service Microsoft Teams
Feature Teams Components
Status Service restored
Impact Users may see delayed call data in Microsoft Teams.
Severity Sev2
Start time 2020-03-30T04:15:00Z
End time 2020-03-30T06:45:00Z
Last update 2020-03-30T07:16:47.91Z
More info The issue was affecting users from receiving data after 4:15 AM UTC on Monday, March 30, 2020. The issue was specifically affecting Call Analytics, Call Quality Dashboard, and eDiscovery.
Root cause Recent service maintenance to a portion of infrastructure responsible for fresh call data caused it to perform below the expected service threshold.

TM208056 Details

Status: We're investigating a potential issue and checking for impact to your organization. We'll provide an update within 30 minutes.; PublishedTime=2020-03-30T05:44:55.013

Status: We identified that a portion of infrastructure responsible for providing fresh call data is performing below the expected service threshold. We're analyzing system logs to help determine our next troubleshooting steps.
Next update by: Monday, March 30, 2020, at 8:00 AMUTC; PublishedTime=2020-03-30T06:11:31.507Z

Final status: We identified that recent service maintenance work was causing higher than anticipated load on the affected infrastructure. We restarted and rerouted traffic to facilitate the processing of requests and remediate impact.
Next update by: Monday, March 30, 2020, at 8:00 AMUTC; PublishedTime=2020-03-30T06:11:31.507Z

--------------------------------------------------------------------------------------------------
Title Call data may be delayed
ID TM208056
Service Microsoft Teams
Feature Teams Components
Status Service restored
Impact Users may see delayed call data in Microsoft Teams.
Severity Sev2
Start time 2020-03-30T04:15:00Z
End time 2020-03-30T06:45:00Z
Last update 2020-03-30T07:16:47.91Z
More info The issue was affecting users from receiving data after 4:15 AM UTC on Monday, March 30, 2020. The issue was specifically affecting Call Analytics, Call Quality Dashboard, and eDiscovery.
Root cause Recent service maintenance to a portion of infrastructure responsible for fresh call data caused it to perform below the expected service threshold.

TM208056 Details

Status: We're investigating a potential issue and checking for impact to your organization. We'll provide an update within 30 minutes.; PublishedTime=2020-03-30T05:44:55.013

Status: We identified that a portion of infrastructure responsible for providing fresh call data is performing below the expected service threshold. We're analyzing system logs to help determine our next troubleshooting steps.
Next update by: Monday, March 30, 2020, at 8:00 AMUTC; PublishedTime=2020-03-30T06:11:31.507Z

Final status: We identified that recent service maintenance work was causing higher than anticipated load on the affected infrastructure. We restarted and rerouted traffic to facilitate the processing of requests and remediate impact.
Next update by: Monday, March 30, 2020, at 8:00 AMUTC; PublishedTime=2020-03-30T06:11:31.507Z

--------------------------------------------------------------------------------------------------
Title Call data may be delayed
ID TM208056
Service Microsoft Teams
Feature Teams Components
Status Service restored
Impact Users may see delayed call data in Microsoft Teams.
Severity Sev2
Start time 2020-03-30T04:15:00Z
End time 2020-03-30T06:45:00Z
Last update 2020-03-30T07:16:47.91Z
More info The issue was affecting users from receiving data after 4:15 AM UTC on Monday, March 30, 2020. The issue was specifically affecting Call Analytics, Call Quality Dashboard, and eDiscovery.
Root cause Recent service maintenance to a portion of infrastructure responsible for fresh call data caused it to perform below the expected service threshold.

TM208056 Details

Status: We're investigating a potential issue and checking for impact to your organization. We'll provide an update within 30 minutes.; PublishedTime=2020-03-30T05:44:55.013

Status: We identified that a portion of infrastructure responsible for providing fresh call data is performing below the expected service threshold. We're analyzing system logs to help determine our next troubleshooting steps.
Next update by: Monday, March 30, 2020, at 8:00 AMUTC; PublishedTime=2020-03-30T06:11:31.507Z

Final status: We identified that recent service maintenance work was causing higher than anticipated load on the affected infrastructure. We restarted and rerouted traffic to facilitate the processing of requests and remediate impact.
Next update by: Monday, March 30, 2020, at 8:00 AMUTC; PublishedTime=2020-03-30T06:11:31.507Z

--------------------------------------------------------------------------------------------------
Title Some users are unable to record meetings in Microsoft Teams
ID TM207986
Service Microsoft Teams
Feature Teams Components
Status Service restored
Impact Users may have been unable to start recording meetings within Teams.
Severity Sev2
Start time 2020-03-28T07:00:00Z
End time 2020-03-28T09:48:00Z
Last update 2020-03-28T14:57:39.593Z
More info
Root cause A section of infrastructure which facilitates the recording on Teams meetings became unresponsive.

TM207986 Details

Status: We're analyzing call controller logs to determine the next troubleshooting steps.
Next update by: Saturday, March 28, 2020, at 11:00 AM UTC; PublishedTime=2020-03-28T08:54:14.203Z

Status: We're investigating a potential issue and checking for impact to your organization. We'll provide an update within 30 minutes.; PublishedTime=2020-03-28T08:36:45.293Next update by: Saturday, March 28, 2020, at 11:00 AM UTC; PublishedTime=2020-03-28T08:54:14.203Z

Final status: We identified that a section of infrastructure which facilitates the recording of meetings within Microsoft Teams became unresponsive. We restarted the affected infrastructure which resolved the issue.
Next update by: Saturday, March 28, 2020, at 11:00 AM UTC; PublishedTime=2020-03-28T08:54:14.203Z

--------------------------------------------------------------------------------------------------
Title Expired download link for meeting recordings
ID TM207955
Service Microsoft Teams
Feature Teams Components
Status Service restored
Impact Users attempting to view meeting recordings may see an expired download link within the
Severity Sev2
Start time 2020-03-27T14:00:00Z
End time 2020-03-27T18:00:00Z
Last update 2020-03-27T23:13:56.823Z
More info Users who uploaded meeting recordings could still watch and share the recording, but would have to follow an alternative means to find the recordings, since the meeting chat would show an expired download link rather than a link to the Stream video.
Root cause

TM207955 Details

Status: Our internal monitoring indicates that a number of Microsoft Teams meeting recordings are producing expired download links for meeting recordings. We're investigating the data and logs associated with this process to determine the source of the issue.
Next update by: Saturday, March 28, 2020, at 12:30 AM UTC; PublishedTime=2020-03-27T22:50:58.947Z

Final status: While we were in the process of analyzing system logs to formulate a remediation plan, system monitoring indicated that the service had returned to normal health. We've confirmed that the problem is no longer occurring, and we'll continue to monitor the service to ensure that the problem does not happen again.
Next update by: Saturday, March 28, 2020, at 12:30 AM UTC; PublishedTime=2020-03-27T22:50:58.947Z

--------------------------------------------------------------------------------------------------
Title Expired download link for meeting recordings
ID TM207903
Service Microsoft Teams
Feature Teams Components
Status Service restored
Impact Users attempting to view meeting recordings may see an expired download link within the
Severity Sev2
Start time 2020-03-26T16:30:00Z
End time 2020-03-27T01:47:31.237Z
Last update 2020-03-27T01:47:31.237Z
More info Users who uploaded meeting recordings can still watch and share the recording, but will have to follow an alternative means to find the recordings, since the meeting chat will show an expiring download link rather than a link to the Stream video.
Root cause A number of Microsoft Teams meeting recordings were taking longer than expected when processing through our infrastructure, resulting in expiring download links.

TM207903 Details

Final status: Our internal monitoring indicated that a number of Microsoft Teams meeting recordings were taking longer than expected when processing through our infrastructure, resulting in expiring download links. We've rebalanced traffic to alternate infrastructure to resolve the issue and verified through our monitoring that meeting recordings aren't producing expired links.


--------------------------------------------------------------------------------------------------
Title Any user in the Asia region may have been unable to create forms
ID FM207912
Service Microsoft Forms
Feature Form functionality
Status Service restored
Impact Users may have been unable to create forms.
Severity Sev2
Start time 2020-03-26T06:00:00Z
End time 2020-03-26T07:15:00Z
Last update 2020-03-27T06:04:40.957Z
More info
Root cause A section of infrastructure responsible for handling user requests and traffic was experiencing performance issues.

FM207912 Details

Final status: We've determined that a section of infrastructure responsible for handling user requests and traffic was experiencing performance issues. We've redirected and optimized traffic and confirmed via monitoring that impact is mitigated.


--------------------------------------------------------------------------------------------------
Title Affected users can't initiate or join Skype Meeting Broadcasts
ID LY207759
Service Skype for Business
Feature Audio and Video
Status Service restored
Impact Users may be intermittently unable to initiate or join Skype Meeting Broadcasts.
Severity Sev2
Start time 2020-03-25T20:28:44Z
End time 2020-03-25T22:10:00Z
Last update 2020-03-26T11:46:16.407Z
More info Users may have seen the error message:
Root cause An unexpected increase in traffic caused the affected infrastructure to become imbalanced and process traffic inefficiently, resulting in impact.

LY207759 Details

Status: We're investigating a potential issue and checking for impact to your organization. We'll provide an update within 30 minutes.; PublishedTime=2020-03-25T20:29:02.947

Status: We're reviewing server telemetry to identify the underlying cause.
Next update by: Wednesday, March 25, 2020, at 10:30 PMUTC; PublishedTime=2020-03-25T20:52:05.587Z

Status: We've identified an unexpected increase in traffic may have caused the affected infrastructure to become imbalanced and process traffic inefficiently, resulting in impact. We're working to rebalance load from the affected infrastructure to provide relief and confirm the suspected cause.
Next update by: Thursday, March 26, 2020, at 3:00 AMUTC; PublishedTime=2020-03-25T22:24:00.713Z

Final status: Our investigation confirmed that an unexpected increase in traffic caused the affected infrastructure to become imbalanced and process traffic inefficiently, resulting in impact. We've rebalanced load from the affected infrastructure and confirmed that our actions have successfully mitigated impact. To ensure that meetings created before Wednesday March 25 2020 at 10:10 PM UTC are not impacted, we recommend recreating the meeting invite.
Next update by: Thursday, March 26, 2020, at 3:00 AMUTC; PublishedTime=2020-03-25T22:24:00.713Z

--------------------------------------------------------------------------------------------------
Title A subset of users in US East can't place calls to users and see delays creating meetings
ID LY207775
Service Skype for Business
Feature Online Meetings
Status Service restored
Impact Users may have been unable to place calls to other users and see delays when creating
Severity Sev2
Start time 2020-03-25T19:54:00Z
End time 2020-03-25T21:55:35.54Z
Last update 2020-03-25T21:55:35.54Z
More info
Root cause A portion of infrastructure that facilitates meetings and individual calls wasn't processing traffic as expected.

LY207775 Details

Final status: We identified a potential issue where users may have been unable to place calls to other users and see delays when creating Skype for Business meetings. We determined that a portion of infrastructure that facilitates meetings and individual calls wasn't processing traffic as expected. We've connected the affected users to alternate infrastructure to remediate impact.


--------------------------------------------------------------------------------------------------
Title All users were unable to view content in the Excel web access web part
ID SP207734
Service SharePoint Online
Feature SharePoint Features
Status Service restored
Impact Users were unable to view content in the Excel web access web part.
Severity Sev2
Start time 2020-03-24T13:00:00Z
End time 2020-03-26T09:55:00Z
Last update 2020-03-26T10:27:22.373Z
More info - Impact was limited to Classic SharePoint Online and should not impact Modern SharePoint Online
Root cause A recent standard service update to the environment introduced a code regression, resulting in impact.

SP207734 Details

Status: We're investigating a potential issue and checking for impact to your organization. We'll provide an update within 30 minutes.


Status: We're reviewing available telemetry data to determine our next troubleshooting steps.
Next update by: Wednesday, March 25, 2020, at 8:30 PMUTC; PublishedTime=2020-03-25T18:13:14.133Z

Status: We've identified that a recent standard service update to the environment introduced a code regression, resulting in impact. We've developed a fix and are preparing to initiate deployment, which we anticipate will take eight to ten hours to complete.
Next update by: Thursday, March 26, 2020, at 7:00 AMUTC; PublishedTime=2020-03-25T19:24:32.97Z

Status: We've begun deployment of the fix to the affected environments, and are monitoring the process to ensure the fix progresses properly.
Next update by: Thursday, March 26, 2020, at 11:00 AMUTC; PublishedTime=2020-03-26T05:17:09.463Z

Final status: We've successfully deployed a code regression fix to the affected environments, and after monitoring can confirm the issue has been successfully mitigated.
Next update by: Thursday, March 26, 2020, at 11:00 AMUTC; PublishedTime=2020-03-26T05:17:09.463Z

--------------------------------------------------------------------------------------------------
Title Intermittent enrollment and device check in failures
ID IT207427
Service Microsoft Intune
Feature Microsoft Intune
Status Service restored
Impact Users may have intermittently experienced failures with enrollments and device check ins.
Severity Sev2
Start time 2020-03-23T22:00:00Z
End time 2020-03-24T05:00:00Z
Last update 2020-03-24T05:47:55.353Z
More info For example, enrollments and device check ins may be impacted
Root cause A code regression within a portion of SQL infrastructure is causing impact.

IT207427 Details

Status: We've determined that a code regression within a portion of SQL infrastructure is causing impact. We're continuing to restart the impacted services to mitigate the issue.
Next update by: Tuesday, March 24, 2020, at 7:00 AM UTC; PublishedTime=2020-03-24T02:52:23.503Z

Status: We're restarting impacted services to remediate the issue.
Next update by: Tuesday, March 24, 2020, at 3:00 AMUTC; PublishedTime=2020-03-24T01:16:15.107Z

Status: We're investigating a potential issue and checking for impact to your organization. We'll provide an update within 30 minutes.; PublishedTime=2020-03-24T01:04:32.297Next update by: Tuesday, March 24, 2020, at 3:00 AMUTC; PublishedTime=2020-03-24T01:16:15.107Z

Final status: We've finished restarting the impacted services and confirmed via monitoring that impact is mitigated.
Next update by: Tuesday, March 24, 2020, at 3:00 AMUTC; PublishedTime=2020-03-24T01:16:15.107Z

--------------------------------------------------------------------------------------------------
Title All users intermittently unable to register for MFA or Self Service Password Reset
ID IS207401
Service Identity Service
Feature Sign-In
Status Service restored
Impact Users may have been intermittently unable to registrar for Multi-Factor Authentication
Severity Sev2
Start time 2020-03-23T18:00:00Z
End time 2020-03-23T20:26:08.697Z
Last update 2020-03-23T20:26:08.697Z
More info Additionally, external users were unable to accept guest invitations.
Root cause A recent update to improve service request processing encountered a network configuration issue which resulted in impact.

IS207401 Details

Final status: We've determined that a recent update to improve service request processing encountered a network configuration issue which resulted in impact. We've reverted the update in the affected environment and confirmed that the problem is resolved.


--------------------------------------------------------------------------------------------------
Title Provisioning delays for multiple scenarios
ID SP207374
Service SharePoint Online
Feature Provisioning
Status Service degradation
Impact Users and admins may experience delays when performing provisioning actions.
Severity Sev1
Start time 2020-03-23T16:32:12Z
End time
Last update 2020-04-02T16:22:20.15Z
More info Scenarios impacted by provisioning delays include, but are not limited to; - Newly created tenants - Newly added users to existing tenants - Newly created sites - Users being added to security groups may not be reflected immediately - Users being granted admin access via Azure Privileged Identity Management (PIM) may not be reflected - The Microsoft Teams files tab appearing for a SharePoint Online or OneDrive for Business sites - Assigning licenses to new or existing users
Root cause A large increase in provisioning requests has resulted in queued requests developing for internal data gathering directory services.

SP207374 Details

Status: We're reviewing provisioning diagnostics to determine the next troubleshooting steps.
Next update by: Monday, March 23, 2020, at 6:30 PM UTC; PublishedTime=2020-03-23T16:39:25.797Z

Status: A large increase in provisioning requests has resulted in queued requests developing for internal data gathering directory services. We're reviewing potential options to speed up the processing of the provisioning requests queue to remediate impact.
Next update by: Tuesday, March 24, 2020, at 5:30 PM UTC; PublishedTime=2020-03-23T17:27:13.183Z

Status: We're developing multiple workstreams to speed up the processing of provisioning requests. These workstreams include; - Developing code changes to enable the service to separate requests between provisioning and general data ingestion recrawls. - Changing how the service read/writes objects so a new provisioning pipeline can be created. - Load-balance the queues by adding additional synchronization streams.
Additionally, we're monitoring the provisioning request queues whilst they process.

Next update by: Thursday, March 26, 2020, at 5:30 PM UTC; PublishedTime=2020-03-24T15:55:19.927Z

Status: The provisioning components are continuing to process requests; however, due to the number of incoming requests, the backlog of queued requests has not been significantly reduced. We're continuing development with the previously mentioned workstreams to produce code changes to expedite the provisioning request process as a whole.
Next update by: Thursday, April 2, 2020, at 5:30 PM UTC; PublishedTime=2020-03-26T16:02:50.82Z

Status: We've implemented load-balancing configurations to prevent requests for new tenant creations being directed onto infrastructure components with higher queues, in an effort to reduce provisioning times. Additionally, we're continuing the development and testing of the code changes for the aforementioned work streams. These deployments will be scheduled over the coming weeks due to the amount of changes needed to be processed safely to prevent any further impact.
Next update by: Tuesday, April 7, 2020, at 5:30 PM UTC; PublishedTime=2020-04-02T16:22:20.15Z

--------------------------------------------------------------------------------------------------
Title All admins can't enable DKIM for custom domains
ID EX207156
Service Exchange Online
Feature Management and Provisioning
Status Service restored
Impact Admins can't enable DomainKeys Identified Mail (DKIM) for custom domains.
Severity Sev2
Start time 2020-03-20T14:36:56Z
End time 2020-03-21T04:00:00Z
Last update 2020-03-21T04:52:48.923Z
More info
Root cause A portion of Azure infrastructure that handles authentication requests encountered a communication problem after a configuration change.

EX207156 Details

Status: We're analyzing client logs to determine next troubleshooting steps.
Next update by: Friday, March 20, 2020, at 5:30 PMUTC ; PublishedTime=2020-03-20T15:04:20.03Z

Status: After further analysis, we determined that the previously mentioned code change was not causing impact. Instead, we determined the issue was due to a portion of Azure infrastructure that handles authentication requests encountering a communication problem after a configuration change. We're reverting the offending change to mitigate impact.
Next update by: Saturday, March 21, 2020, at 12:00 AMUTC ; PublishedTime=2020-03-20T19:58:46.6Z

Status: Our analysis of the client logs determined a recent code change intended to improve Domain Name System (DNS) record validation inadvertently resulted in the impact. We're reverting the offending change to mitigate impact.
Next update by: Friday, March 20, 2020, at 10:00 PMUTC; PublishedTime=2020-03-20T16:48:04.59Z

Status: We're investigating a potential issue and checking for impact to your organization. We'll provide an update within 30 minutes.; PublishedTime=2020-03-20T14:42:26.143Next update by: Friday, March 20, 2020, at 10:00 PMUTC; PublishedTime=2020-03-20T16:48:04.59Z

Status: We've validated the change that will be rolling out to revert the configuration update that resulted in impact. The rollout has now begun and we expect it to complete within the next eight hours.
Next update by: Saturday, March 21, 2020, at 6:30 AMUTC; PublishedTime=2020-03-20T22:28:50.367Z

Final status: The rollout has been completed and we've confirmed that it has successfully mitigated impact.
Next update by: Saturday, March 21, 2020, at 6:30 AMUTC; PublishedTime=2020-03-20T22:28:50.367Z

--------------------------------------------------------------------------------------------------
Title All admins seeing stale usage report information for O365 active user count and Teams
ID MO207509
Service Office 365 Portal
Feature Administration
Status Service restored
Impact Admin users may have received usage reports that may have not been up-to-date.
Severity Sev2
Start time 2020-03-20T00:00:00Z
End time 2020-03-25T12:20:00Z
Last update 2020-03-25T13:24:17.117Z
More info The following reports may have not been up-to-date: - Microsoft Teams device usage - Microsoft Teams user activity - Office 365 active users
Root cause An increase in traffic caused the pipeline that handles reporting data to perform below acceptable thresholds, resulting in data that may have not been up-to-date.

MO207509 Details

Status: We've identified that impact related to MO206524 persists. We're gathering fresh diagnostic data to help determine the next troubleshooting steps.
Next update by: Tuesday, March 24, 2020, at 2:00 PM UTC ; PublishedTime=2020-03-24T11:49:33.773Z

Status: We're performing data management optimization to improve the processing of reports in an attempt to mitigate impact, while we determine the source of the issue.
Next update by: Tuesday, March 24, 2020, at 4:00 PM UTC; PublishedTime=2020-03-24T14:00:53.107Z

Status: We've determined that an increase in traffic has cause the pipeline that handles reporting data to become degraded, resulting in data that may not be up-to-date. We're continuing to perform data management optimizations to improve the processing of reports in an attempt to mitigate impact.
Next update by: Tuesday, March 24, 2020, at 5:30 PMUTC; PublishedTime=2020-03-24T15:39:58.583Z

Status: Our initial data management optimizations didn't produce the expected relief. We've discovered a few options that we're narrowing down to determine which will produce the best result to remediate impact.
Next update by: Tuesday, March 24, 2020, at 7:30 PMUTC; PublishedTime=2020-03-24T17:16:50.613Z

Status: We're continuing to review all available mitigation options to determine the optimal action, which we expect will take some time to complete.
Next update by: Wednesday, March 25, 2020, at 9:00 AMUTC; PublishedTime=2020-03-24T18:36:51.773Z

Status: We've increased the report processing efficiency for specific requests, and confirmed that the Teams usage report within the Teams admin center is now showing data. We're continuing our efforts to mitigate the issue for the O365 admin center data.
Next update by: Wednesday, March 25, 2020, at 11:00 AM UTC ; PublishedTime=2020-03-25T09:00:10.917Z

Status: We're continuing to increase report processing efficiency, and expect our next stage of mitigation to complete with four hours. Once completed, we expect usage and activity reports in the O365 admin center data to display data up to the 23rd of March.
Next update by: Wednesday, March 25, 2020, at 3:00 PM UTC; PublishedTime=2020-03-25T10:59:38.76Z

Final status: We've completed our mitigation steps and confirmed that all data up to March 23rd is now viewable, and the issue is resolved.
Next update by: Wednesday, March 25, 2020, at 3:00 PM UTC; PublishedTime=2020-03-25T10:59:38.76Z

--------------------------------------------------------------------------------------------------
Title North America users can't sign in, send messages or reply to channel threads
ID TM207043
Service Microsoft Teams
Feature Teams Components
Status Service restored
Impact Users may have been unable to perform specific actions in Microsoft Teams.
Severity Sev2
Start time 2020-03-19T20:34:00Z
End time 2020-03-19T21:20:11.07Z
Last update 2020-03-19T21:20:11.07Z
More info Users may have also been unable to: -Sign in -Create new teams or channels -Send messages -Reply to channel threads
Root cause A portion of networking infrastructure responsible for processing request queues was performing below expected thresholds.

TM207043 Details

Final status: We've identified a portion of networking infrastructure responsible for processing request queues which was performing below expected thresholds. We've rebalanced traffic across the affected infrastructure and confirmed that impact is mitigated.


--------------------------------------------------------------------------------------------------
Title All admins may be unable to address access requests for classic SharePoint Online sites
ID SP207180
Service SharePoint Online
Feature SharePoint Features
Status Post-incident report published
Impact Admins were unable to approve or deny access requests to classic sites in the SharePoint
Severity Sev2
Start time 2020-03-19T17:25:00Z
End time 2020-03-27T00:00:00Z
Last update 2020-04-03T18:27:14.48Z
More info The approve and deny buttons weren't populating.
Root cause A recent deployment resulted in the accept/decline button not populating for admins within the SPO admin center.

SP207180 Details

Status: We've identified a recent deployment for SharePoint Online that has caused the accept/ decline button to not populate for admins. We're reviewing the impacting deployment to identify a root cause and determine our next troubleshooting steps.
Next update by: Saturday, March 21, 2020, at 1:30 AMUTC; PublishedTime=2020-03-20T20:12:19.977Z

Status: We're analyzing system logs to determine why the approve/deny option isn't populating.
Next update by: Friday, March 20, 2020, at 8:30 PMUTC; PublishedTime=2020-03-20T18:59:12.567Z

Status: We've validated a code fix for this issue and expect deployment to complete by Monday, March 30, 2020.
Next update by: Monday, March 30, 2020, at 9:00 PMUTC; PublishedTime=2020-03-20T21:50:23.537Z

Status: We've confirmed that the fix has reached 78 percent of the environment. We expect this process will complete within the next 48 hours.
Next update by: Saturday, March 28, 2020, at 4:00 AMUTC; PublishedTime=2020-03-27T01:15:27.733Z

Final status: We've confirmed that the fix has been completely deployed to the affected environment. Our internal testing and telemetry indicate that the issue has been mitigated.
Next update by: Saturday, March 28, 2020, at 4:00 AMUTC; PublishedTime=2020-03-27T01:15:27.733Z

--------------------------------------------------------------------------------------------------
Title PSTN users in North America can't join meetings by dialing in or Call me at feature
ID LY207018
Service Skype for Business
Feature Audio and Video
Status Service restored
Impact PSTN users may be unable to join Skype for Business meetings when dialing in or using
Severity Sev2
Start time 2020-03-19T17:17:31Z
End time 2020-03-19T17:34:00Z
Last update 2020-03-19T19:16:43.213Z
More info Additionally, affected users may have been unable to communicate with mobile and desktop users and would have seen failures with in-call operations.
Root cause A portion of infrastructure that handles authentication requests encountered a communication problem, resulting in the impact.

LY207018 Details

Status: We're investigating a potential issue and checking for impact to your organization. We'll provide an update within 30 minutes.; PublishedTime=2020-03-19T17:17:52.707

Status: We suspect that a portion of infrastructure that handles authentication requests encountered a communication problem, resulting in the impact. We're restarting the affected infrastructure to resolve the communication problem and mitigate impact.
Next update by: Thursday, March 19, 2020, at 7:30 PMUTC; PublishedTime=2020-03-19T17:41:17.807Z

Final status: We successfully restarted the affected infrastructure to correct the communication problem. After a period of monitoring service telemetry, we've confirmed that this issue is resolved.
Next update by: Thursday, March 19, 2020, at 7:30 PMUTC; PublishedTime=2020-03-19T17:41:17.807Z

--------------------------------------------------------------------------------------------------
Title Calendar, meeting and live event functionality
ID TM206882
Service Microsoft Teams
Feature Teams Components
Status Service restored
Impact Users may have experienced decreased functionality in calendars, meetings and live event
Severity Sev2
Start time 2020-03-18T21:15:00Z
End time 2020-03-18T22:33:16Z
Last update 2020-03-18T22:46:02.12Z
More info
Root cause

TM206882 Details

Final status: We've identified a portion of Microsoft Teams infrastructure that hosts user content was performing below expected thresholds, resulting in impact. Our automated recovery system repaired this problem, and we confirmed that service was restored after monitoring the environment.


--------------------------------------------------------------------------------------------------
Title Email delays
ID EX206878
Service Exchange Online
Feature E-Mail timely delivery
Status Service restored
Impact Users may have experienced delays when sending and receiving email messages.
Severity Sev2
Start time 2020-03-17T16:00:00Z
End time 2020-03-18T21:30:00Z
Last update 2020-03-18T22:49:46Z
More info
Root cause A power outage on a portion of infrastructure that facilitates email delivery introduced an authentication problem, resulting in email delays.

EX206878 Details

Status: We've identified an authentication problem within the service that facilitates email delivery, which is causing delays when sending and receiving mail. We're discussing the possibility for the affected message delivery service to use a different authentication method as a potential solution to remediate the issue.
Next update by: Wednesday, March 18, 2020, at 10:30 PMUTC; PublishedTime=2020-03-18T20:36:47.58Z

Final status: We further clarified the cause to be a power outage that introduced the authentication problem. After power was restored, we deployed a fix to implement a backup authentication method in the event the primary method is unsuccessful. Following deployment, we're now seeing email queues decrease significantly, which confirms that mail flow has been restored.
Next update by: Wednesday, March 18, 2020, at 10:30 PMUTC; PublishedTime=2020-03-18T20:36:47.58Z

--------------------------------------------------------------------------------------------------
Title All admins receiving errors within the Security and Compliance Center (SCC)
ID EX207367
Service Exchange Online
Feature Management and Provisioning
Status Service restored
Impact Admins may have experienced error messages when attempting to access Advanced Threat
Severity Sev2
Start time 2020-03-17T07:00:00Z
End time 2020-04-01T23:13:00Z
Last update 2020-04-01T23:28:29.593Z
More info
Root cause A recent update contained a code issue causing authentication to fail, resulting in impact.

EX207367 Details

Status: We're investigating a potential issue and checking for impact to your organization. We'll provide an update within 30 minutes.; PublishedTime=2020-03-23T15:23:51.487

Status: We're working with representatives of your organization to gather Fiddler network traces of the issue in progress to identify the underlying cause.
Next update by: Monday, March 23, 2020, at 6:00 PMUTC; PublishedTime=2020-03-23T15:52:53.677Z

Status: We've identified a code issue in a recent service update that is preventing the authentication of ATP reports. We're developing a fix which we expect to begin deployment on Tuesday, March 24, 2020.
Next update by: Tuesday, March 24, 2020, at 8:00 PMUTC; PublishedTime=2020-03-23T17:05:36.147Z

Status: We're in the process of scheduling deployment of the fix. After we've assessed the deployment progress for a period of time, we'll provide an estimated timeline to completion.
Next update by: Wednesday, March 25, 2020, at 8:00 PMUTC; PublishedTime=2020-03-24T17:37:34.377Z

Status: We've confirmed that the fix has started to deploy to the affected environment. After we've assessed the deployment progress further, we'll provide an estimated timeline for completion.
Next update by: Thursday, March 26, 2020, at 1:30 AMUTC; PublishedTime=2020-03-25T17:33:57.84Z

Status: An unexpected error resulted in us having to restart our deployment of the fix. We are monitoring the deployment to ensure that it progresses normally.
Next update by: Friday, March 27, 2020, at 1:30 AMUTC ; PublishedTime=2020-03-26T00:15:44.947Z

Status: We're continuing to monitor the fix as it's progressing and admins may experience remediation as the fix continues to saturate to the affected environments.
Next update by: Saturday, March 28, 2020, at 1:30 AMUTC ; PublishedTime=2020-03-26T23:53:04.123Z

Status: The fix has completed saturation in multiple, but not all, affected environments, and some users may begin to see relief. We're continuing to monitor the progress of the fix to ensure all environments are mitigated.
Next update by: Tuesday, March 31, 2020, at 1:30 AMUTC; PublishedTime=2020-03-27T23:35:15.917Z

Status: The fix is continuing to saturate throughout the final affected environment, and has reached 60 percent saturation. Users in this environment will begin to see relief as it continues, and users in the previously saturated environments have confirmed mitigation.
Next update by: Wednesday, April 1, 2020, at 1:30 AMUTC; PublishedTime=2020-03-30T22:50:16.177Z

Status: The fix is 70 percent saturated in the remaining environment. We'll continue to monitor its progression to ensure mitigation for all users.
Next update by: Thursday, April 2, 2020, at 1:30 AMUTC; PublishedTime=2020-03-31T23:38:08.88Z

Final status: We've confirmed that the deployment of the fix has completed and impact is mitigated. Some users may need to retry after their first attempt but will see no further impact.
Next update by: Thursday, April 2, 2020, at 1:30 AMUTC; PublishedTime=2020-03-31T23:38:08.88Z

--------------------------------------------------------------------------------------------------
Title Delayed activation reports
ID MO206967
Service Office 365 Portal
Feature Administration
Status Service restored
Impact Admins may notice delays in Office 365 activation reports starting from March 17, 2020.
Severity Sev2
Start time 2020-03-17T00:00:00Z
End time 2020-03-20T22:00:00Z
Last update 2020-03-23T07:12:29.973Z
More info
Root cause An access folder error prevented the timely delivery of the activation reports.

MO206967 Details

Status: We've identified that activation reports past March 17, 2020 are delayed as a result of an access folder data error. We're investigating the access error to determine our next troubleshooting steps.
Next update by: Friday, March 20, 2020, at 7:30 AM UTC; PublishedTime=2020-03-19T07:19:39.52Z

Status: We've resolved the folder data error and we're working to reprocess the activation reports to ensure that they're successfully sent. Based on our current estimates, we suspect that this will complete by Monday, March 23, 2020.
Next update by: Monday, March 23, 2020, at 7:30 AMUTC; PublishedTime=2020-03-20T06:31:36.203Z

Final status: We've finished reprocessing the activation reports to mitigate impact.
Next update by: Monday, March 23, 2020, at 7:30 AMUTC; PublishedTime=2020-03-20T06:31:36.203Z

--------------------------------------------------------------------------------------------------
Title Some meeting recordings are not capturing screensharing
ID TM207532
Service Microsoft Teams
Feature Teams Components
Status Service restored
Impact Some meeting recordings were not capturing screensharing or other videos.
Severity Sev2
Start time 2020-03-16T18:00:00Z
End time 2020-03-28T20:44:00Z
Last update 2020-03-29T00:36:48.507Z
More info
Root cause A decoding issue within the video pipeline resulted in the record meetings feature to not work as expected.

TM207532 Details

Status: We're analyzing system logs and running various tests to reproduce and isolate the source of the issue.
Next update by: Tuesday, March 24, 2020, at 6:00 PMUTC; PublishedTime=2020-03-24T16:36:42.53Z

Status: We've collected Microsoft Teams debug, media, and desktop logs captured during active reproduction of the impact to continue our investigation into the source of this issue.
Next update by: Tuesday, March 24, 2020, at 8:00 PMUTC; PublishedTime=2020-03-24T17:59:29.537Z

Status: We suspect that a communication error between Teams video-based sharing infrastructure and the process to distribute those requests may be contributing to impact. We're updating our logs to investigate this potential issue in depth.
Next update by: Tuesday, March 24, 2020, at 10:00 PMUTC; PublishedTime=2020-03-24T19:54:15.65Z

Status: We're attempting to reproduce the issue locally to further our investigation. Additionally we're reviewing processing usage on the video-based sharing infrastructure to verify that the process was operating within threshold during peak business hours.
Next update by: Wednesday, March 25, 2020, at 12:00 AMUTC; PublishedTime=2020-03-24T21:46:23.927Z

Status: We're continuing to attempt to reproduce the issue in a controlled environment. Additionally we're reviewing processing usage on the video-based sharing infrastructure to determine potential impact mitigation options.
Next update by: Wednesday, March 25, 2020, at 5:00 PMUTC ; PublishedTime=2020-03-25T00:08:08.823Z

Status: We've reproduced the issue internally and are reviewing media stream logs to isolate the cause and develop mitigation strategies.
Next update by: Wednesday, March 25, 2020, at 9:00 PMUTC; PublishedTime=2020-03-25T16:28:24.633Z

Status: We've identified that an unexpectedly large influx of traffic to a portion of Microsoft Teams infrastructure resulted in a degradation to the recording meetings feature, resulting in impact to capturing screensharing. We're working to rebalance traffic throughout the affected environment to mitigate impact.
Next update by: Thursday, March 26, 2020, at 1:00 AMUTC; PublishedTime=2020-03-25T19:44:25.583Z

Status: We're continuing our efforts to rebalance traffic. Once the fix is developed, we'll deploy it to the affected environment.
Next update by: Thursday, March 26, 2020, at 6:30 PMUTC; PublishedTime=2020-03-25T23:37:34.253Z

Status: We've verified an issue with the video decoding pipeline that was exposed due to the larger than normal influx of traffic, which is resulting in the impact to the record meetings feature and screensharing. We're continuing to rebalance traffic on the affected infrastructure to restore service.
Next update by: Friday, March 27, 2020, at 7:30 PMUTC; PublishedTime=2020-03-26T17:14:46.137Z

Status: We're continuing to monitor as the fix deployment progresses and are reviewing telemetry to develop a timeline for mitigation. Additionally, users should notice relief as the deployment progresses.
Next update by: Saturday, March 28, 2020, at 3:00 AMUTC; PublishedTime=2020-03-27T18:06:54.097Z

Status: Due to the complicated nature of meeting recordings, the fix is taking longer than expected to safely deploy to the affected infrastructure. We're continuing to monitor the progress of the deployment and will provide an update on the status on Sunday, March 29, 2020, at 2:00 AM UTC.
Next update by: Sunday, March 29, 2020, at 2:00 AMUTC; PublishedTime=2020-03-28T00:36:35.597Z

Final status: We've confirmed that the fix has been completely deployed the affected environment and our testing and telemetry indicated that the issue has been mitigated.
Next update by: Sunday, March 29, 2020, at 2:00 AMUTC; PublishedTime=2020-03-28T00:36:35.597Z

Status: We identified a decoding issue within the video pipeline, which resulted in the record meetings feature to not work as expected. We've developed a fix and tested it successfully. This will now be deployed to the affected infrastructure to mitigate impact.
Next update by: Friday, March 27, 2020, at 7:30 PM UTC; PublishedTime=2020-03-27T07:44:56.177Z

--------------------------------------------------------------------------------------------------
Title Admins seeing delays in data to CQD and Call Analytics
ID LY206932
Service Skype for Business
Feature Management and Provisioning
Status Service restored
Impact Admins may have been seeing delays up to 50 hours in data to their Call Quality
Severity Sev2
Start time 2020-03-16T16:00:00Z
End time 2020-03-20T23:00:00Z
Last update 2020-03-25T00:27:07.263Z
More info
Root cause An increase in unprecedented traffic levels degraded the pipeline that supplied data to the CQD, as well as all Skype Data Platform services, resulting in up to 50 hours of delayed data.

LY206932 Details

Status: We've determined that an increase in traffic degraded the pipeline that supplies data to the CQD, resulting delayed data. We're optimizing the components responsible for transmitting this data to increase performance and remediate impact. While we're taking these actions, admins should see the delays reduced and the missing data populated throughout the next few days.
Next update by: Friday, March 20, 2020, at 7:00 PMUTC; PublishedTime=2020-03-19T01:41:11.497Z

Status: We're continuing to optimize the pipeline facilitating the Call Quality Dashboard and see improvements in the service. We continue to expect this issue to be fully resolved by Friday, March 20, 2020 at 11:00 PM UTC.
Next update by: Saturday, March 21, 2020, at 12:00 AMUTC; PublishedTime=2020-03-20T17:41:55.323Z

Status: We've completed our optimization efforts and we're seeing continuous improvements to healthy service performance levels. To ensure that service functions as expected during the business week, we're opting to monitor the service health into the next business week to ensure that service is fully restored. We expect to have our results by the end of Monday, March 23, 2020.
Next update by: Tuesday, March 24, 2020, at 12:00 AMUTC; PublishedTime=2020-03-20T23:43:32.667Z

Status: We're seeing drastic improvements to the service performance levels, but we're going to continue our monitoring process longer than expected to ensure that service is fully restored.
Next update by: Wednesday, March 25, 2020, at 1:00 AMUTC; PublishedTime=2020-03-23T23:57:27.963Z

Final status: We've confirmed through our monitoring process that service is now fully restored after upgrading our resources and components to scale for the high level of traffic volume.
Next update by: Wednesday, March 25, 2020, at 1:00 AMUTC; PublishedTime=2020-03-23T23:57:27.963Z

--------------------------------------------------------------------------------------------------
Title Admins seeing delays in data to CQD and Call Analytics
ID LY206932
Service Skype for Business
Feature Management and Provisioning
Status Service restored
Impact Admins may have been seeing delays up to 50 hours in data to their Call Quality
Severity Sev2
Start time 2020-03-16T16:00:00Z
End time 2020-03-20T23:00:00Z
Last update 2020-03-25T00:27:07.263Z
More info
Root cause An increase in unprecedented traffic levels degraded the pipeline that supplied data to the CQD, as well as all Skype Data Platform services, resulting in up to 50 hours of delayed data.

LY206932 Details

Status: We've determined that an increase in traffic degraded the pipeline that supplies data to the CQD, resulting delayed data. We're optimizing the components responsible for transmitting this data to increase performance and remediate impact. While we're taking these actions, admins should see the delays reduced and the missing data populated throughout the next few days.
Next update by: Friday, March 20, 2020, at 7:00 PMUTC; PublishedTime=2020-03-19T01:41:11.497Z

Status: We're continuing to optimize the pipeline facilitating the Call Quality Dashboard and see improvements in the service. We continue to expect this issue to be fully resolved by Friday, March 20, 2020 at 11:00 PM UTC.
Next update by: Saturday, March 21, 2020, at 12:00 AMUTC; PublishedTime=2020-03-20T17:41:55.323Z

Status: We've completed our optimization efforts and we're seeing continuous improvements to healthy service performance levels. To ensure that service functions as expected during the business week, we're opting to monitor the service health into the next business week to ensure that service is fully restored. We expect to have our results by the end of Monday, March 23, 2020.
Next update by: Tuesday, March 24, 2020, at 12:00 AMUTC; PublishedTime=2020-03-20T23:43:32.667Z

Status: We're seeing drastic improvements to the service performance levels, but we're going to continue our monitoring process longer than expected to ensure that service is fully restored.
Next update by: Wednesday, March 25, 2020, at 1:00 AMUTC; PublishedTime=2020-03-23T23:57:27.963Z

Final status: We've confirmed through our monitoring process that service is now fully restored after upgrading our resources and components to scale for the high level of traffic volume.
Next update by: Wednesday, March 25, 2020, at 1:00 AMUTC; PublishedTime=2020-03-23T23:57:27.963Z

--------------------------------------------------------------------------------------------------
Title Limited Teams functionality
ID TM206559
Service Microsoft Teams
Feature Teams Components
Status Post-incident report published
Impact Users may have experienced limited functionality within the Microsoft Teams service.
Severity Sev2
Start time 2020-03-16T14:03:00Z
End time 2020-03-17T11:45:00Z
Last update 2020-03-25T19:52:54.25Z
More info
Root cause

TM206559 Details

Status: We're investigating a potential issue with Microsoft Teams. We'll provide an update within 30 minutes.; PublishedTime=2020-03-16T15:11:32.127

Status: We're reviewing system telemetry to isolate the source of the issue.
Next update by: Monday, March 16, 2020, at 5:30 PM UTC; PublishedTime=2020-03-16T15:30:24.157Z

Status: We're continuing to analyze system telemetry to determine the next steps.
Next update by: Monday, March 16, 2020, at 6:30 PM UTC; PublishedTime=2020-03-16T17:23:00.233Z

Final status: As part of our investigation regarding an issue impacting users served through our European data centers, we've expanded communications to our customers more broadly under TM206556. We've determined that the impact for this event was related to incident post TM206556 and was specific to customers hosted within Europe; however, this issue could have impacted users in other regions if their tenant is hosted within Europe. We're closing this post and will be keeping TM206556 active as we continue to monitor the environment to confirm that our resolution steps have fully mitigated impact.
Next update by: Monday, March 16, 2020, at 6:30 PM UTC; PublishedTime=2020-03-16T17:23:00.233Z

--------------------------------------------------------------------------------------------------
Title Delayed reports
ID MO206963
Service Office 365 Portal
Feature Administration
Status Service restored
Impact Admins may have noticed that user activity reports were not up to date.
Severity Sev2
Start time 2020-03-16T00:00:00Z
End time 2020-03-20T07:00:00Z
Last update 2020-03-23T06:29:33.99Z
More info
Root cause An issue within the upstream source data for some usage activity reports was causing the data delay, resulting in impact.

MO206963 Details

Status: We've resolved the upstream data delay issue and we're working to reprocess the remaining user reports to ensure that they're successfully sent. Based on current estimates, we suspect that this will be done by Monday, March 23, 2020.
Next update by: Monday, March 23, 2020, at 7:30 AMUTC; PublishedTime=2020-03-20T06:37:00.743Z

Status: We've identified an upstream data delay causing admins to experience user activity report delays. We're reviewing the source data in order to understand the reason for the data delay and resolve the issue.
Next update by: Friday, March 20, 2020, at 7:30 AMUTC; PublishedTime=2020-03-19T06:36:29.513Z

Final status: We finished reprocessing the remaining user reports to mitigate the impact.
Next update by: Friday, March 20, 2020, at 7:30 AMUTC; PublishedTime=2020-03-19T06:36:29.513Z

--------------------------------------------------------------------------------------------------
Title Can't log into some services
ID MO206505
Service Office 365 Portal
Feature Portal
Status Service restored
Impact Users may have been unable to access some Microsoft 365 services.
Severity Sev2
Start time 2020-03-15T20:30:00Z
End time 2020-03-15T21:51:00Z
Last update 2020-03-16T00:49:38.613Z
More info Users may have experienced impact when accessing Exchange Online, Skype for Business, Microsoft Teams, Intune, Power BI, and potentially other services.
Root cause An upstream power outage resulted in impact for multiple Microsoft 365 services.

MO206505 Details

Status: We're investigating a potential issue and checking for impact to your organization. We'll provide an update within 30 minutes. ; PublishedTime=2020-03-15T21:48:20.03

Status: We're investigating a power outage within one of our datacenters and have routed traffic to alternate infrastructure to mitigate impact.
Next update by: Monday, March 16, 2020, at 12:30 AM UTC; PublishedTime=2020-03-15T22:07:50.473Z

Final status: We've confirmed that service traffic was successfully routed to alternate infrastructure to expedite mitigation. Additionally, we've taken the necessary actions to restore power for the affected datacenter, enabling us to revert traffic back to its primary pathway.
Next update by: Monday, March 16, 2020, at 12:30 AM UTC; PublishedTime=2020-03-15T22:07:50.473Z

--------------------------------------------------------------------------------------------------
Title Delayed usage reports
ID MO206800
Service Office 365 Portal
Feature Administration
Status Service restored
Impact Admins may have noticed delays in usage reports.
Severity Sev2
Start time 2020-03-15T00:00:00Z
End time 2020-03-21T20:30:00Z
Last update 2020-03-23T07:33:37.357Z
More info The Email activity and email app usage reports were delayed past March 16, 2020.
Root cause An issue within the upstream source data for some usage activity reports was causing the data delay, resulting in impact.

MO206800 Details

Status: We've identified an upstream data delay causing admins to experience report delays. We're reviewing the source data in order to understand the reason for the data delay and resolve the issue.
Next update by: Thursday, March 19, 2020, at 7:30 AMUTC; PublishedTime=2020-03-18T06:02:50.733Z

Status: We've mitigated the upstream data delay causing admins to experience report delays, which has made reports up to March 16, 2020 available in the admin center. We are playing back the remaining delayed notifications to remediate impact.
Next update by: Friday, March 20, 2020, at 7:30 AMUTC; PublishedTime=2020-03-19T06:37:21.64Z

Status: We're continuing to play back the delayed notifications to fully remediate the issue. Based on our current estimates, we believe that admins should be able to see reports up to March 18, 2020 by our next update. We're monitoring the progress of this to ensure that the notifications are caught up without issue.
Next update by: Monday, March 23, 2020, at 7:30 AMUTC; PublishedTime=2020-03-20T06:27:54.11Z

Final status: We've completed the process of playing back the delayed notifications and we've confirmed that the reports have caught up.
Next update by: Monday, March 23, 2020, at 7:30 AMUTC; PublishedTime=2020-03-20T06:27:54.11Z

--------------------------------------------------------------------------------------------------
Title All users may be unable to play Skype Meeting Broadcast recordings
ID LY208401
Service Skype for Business
Feature Audio and Video
Status Service restored
Impact Users may have been unable to play Skype Meeting Broadcast recordings.
Severity Sev2
Start time 2020-03-14T22:49:00Z
End time 2020-04-03T07:00:00Z
Last update 2020-04-03T16:49:19.54Z
More info As an alternative method, users could download the affected recordings for playback.
Root cause A recent update to the URL facilitating streaming of Skype Meeting Broadcast recordings caused the service to become unusable.

LY208401 Details

Status: We're investigating a potential issue and checking for impact to your organization. We'll provide an update within 30 minutes. ; PublishedTime=2020-04-02T16:41:01.137

Status: We're analyzing system logs to verify the possible root cause and determine our next troubleshooting steps.
Next update by: Thursday, April 2, 2020, at 6:30 PMUTC; PublishedTime=2020-04-02T17:00:53.737Z

Status: We've identified a recent update to the URL facilitating the streaming of Skype Meeting Broadcast recordings which has caused the service to become unavailable. We're working to apply a code fix to the affected service and restore streaming functionality.
Next update by: Friday, April 3, 2020, at 12:30 AMUTC; PublishedTime=2020-04-02T18:02:55.287Z

Status: We've developed an elaborate code fix that we're currently deploying to our test environment. Once internal testing is complete, we'll deploy the fix to the affected environment and expect this to fully mitigate the underlying issue.
Next update by: Friday, April 3, 2020, at 6:00 AMUTC ; PublishedTime=2020-04-03T00:25:27.377Z

Status: We're continuing to monitor the fix as it's being deployed to the test environment to ensure it resolves the underlying issue.
Next update by: Friday, April 3, 2020, at 6:00 PMUTC ; PublishedTime=2020-04-03T04:56:33.127Z

Final status: We've completed the fix deployment in both test and production environments and have confirmed that the issue is resolved.
Next update by: Friday, April 3, 2020, at 6:00 PMUTC ; PublishedTime=2020-04-03T04:56:33.127Z

--------------------------------------------------------------------------------------------------
Title Stale admin usage reports information
ID MO206524
Service Office 365 Portal
Feature Administration
Status Service restored
Impact Admin users may have received reports with stale information.
Severity Sev2
Start time 2020-03-14T00:00:00Z
End time 2020-03-17T12:00:00Z
Last update 2020-03-18T02:22:34.337Z
More info The following reports may not have been up to date: Microsoft Teams device usage Microsoft Teams user activity Office 365 active users
Root cause There was an upstream source data delay, resulting in impact to reports for Microsoft Teams device usage, Microsoft Teams user activity, and Office 365 active users.

MO206524 Details

Status: We've determined there is an upstream source data delay, resulting in impact to reports for Microsoft Teams device usage, Microsoft Teams user activity, and Office 365 active users. We're analyzing scripts to determine the best way to mitigate the issue.
Next update by: Wednesday, March 18, 2020, at 3:00 AM UTC; PublishedTime=2020-03-16T02:50:34.35Z

Final status: We've resolved the upstream source data delay by backfilling historical data and confirmed via telemetry that impact is mitigated.
Next update by: Wednesday, March 18, 2020, at 3:00 AM UTC; PublishedTime=2020-03-16T02:50:34.35Z

--------------------------------------------------------------------------------------------------
Title Can't add guest users
ID TM206387
Service Microsoft Teams
Feature Teams Components
Status Post-incident report published
Impact Users may have been unable to add guest users to teams.
Severity Sev2
Start time 2020-03-13T16:02:01Z
End time 2020-03-14T22:00:16Z
Last update 2020-03-20T14:34:21.44Z
More info User's access to external file services and sharing files may have been restricted.
Root cause During a recent deployment, a configuration setting was prematurely disabled before the deployment completed, resulting in requests reading the value of some settings as "off" rather than "on".

TM206387 Details

Status: We're investigating a potential issue and checking for impact to your organization. We'll provide an update within 30 minutes.; PublishedTime=2020-03-13T16:02:57.217

Status: We're analyzing client logs to determine the source of the issue.
Next update by: Friday, March 13, 2020, at 6:30 PMUTC; PublishedTime=2020-03-13T16:21:31.047Z

Status: We're reviewing the Microsoft Teams code facilitating the ability to add guest users. Additionally, we've requested a new reproduction and Fiddler trace of the issue as the previously provided logs were not taken at the same time and were inconclusive.
Next update by: Friday, March 13, 2020, at 8:30 PMUTC; PublishedTime=2020-03-13T18:07:00.57Z

Status: We're reviewing the additional logs provided by your representatives to determine the source of this issue.
Next update by: Friday, March 13, 2020, at 10:30 PMUTC; PublishedTime=2020-03-13T20:09:45.097Z

Status: We're continuing to review the Microsoft Teams code in conjunction with the updated logs provided by your representatives to determine where the discrepancy in the service is occurring, as the logs are showing conflicting information.
Next update by: Saturday, March 14, 2020, at 1:30 AMUTC; PublishedTime=2020-03-13T21:17:19.593Z

Status: We've successfully reproduced the issue and are investigating the logs to identify the root cause and determine out next troubleshooting steps.
Next update by: Saturday, March 14, 2020, at 4:30 AMUTC ; PublishedTime=2020-03-14T00:06:32.983Z

Status: Our investigation into the issue has been delayed and we're working to repair this item so that we can resume our analysis into the logs to determine our next troubleshooting steps.
Next update by: Saturday, March 14, 2020, at 7:30 AMUTC ; PublishedTime=2020-03-14T04:21:24.757Z

Status: Our investigation is focused on the people picker component and settings flags to identify the underlying root cause.
Next update by: Saturday, March 14, 2020, at 11:30 AM UTC; PublishedTime=2020-03-14T07:14:16.527Z

Status: We're continuing to analyze the people picker component and settings flags to identify the underlying root cause and next steps to remediate impact.
Next update by: Saturday, March 14, 2020, at 9:00 PM UTC; PublishedTime=2020-03-14T09:07:03.533Z

Status: Additional reviews of the people picker component returned inconclusive; however, we've identified that a recent deployment caused the issue and are currently in the process of reverting the change to mitigate the impact.
Next update by: Saturday, March 14, 2020, at 10:00 PM UTC; PublishedTime=2020-03-14T20:54:30.77Z

Final status: We've successfully reverted the recent deployment, and verified that service has been restored.
Next update by: Saturday, March 14, 2020, at 10:00 PM UTC; PublishedTime=2020-03-14T20:54:30.77Z

--------------------------------------------------------------------------------------------------
Title Some channels may not be displayed in the list of channels in Teams clients
ID TM208143
Service Microsoft Teams
Feature Teams Components
Status Service restored
Impact Users of specific Microsoft Teams groups may be unable to view some channels within a
Severity Sev2
Start time 2020-03-13T16:00:00Z
End time 2020-04-05T04:00:00Z
Last update 2020-04-05T20:16:14.777Z
More info Impacted customers were directed to contact our support resources and provide the impacted team ID in order to expedite our mitigation steps. To get this ID, you could use the "Get link to team" option accessible by clicking "..." next to the team name.
Root cause A code issue within Microsoft Teams is causing some channels to not be viewable by a subset of users.

TM208143 Details

Status: We're continuing to analyze client logging and recently collected audit log data. In parallel, we're investigating a group of recent service updates to determine if a recent update could be adversely affecting the service.
Next update by: Wednesday, April 1, 2020, at 1:00 AMUTC; PublishedTime=2020-03-31T15:53:04.767Z

Status: We're continuing our investigation by reviewing newly acquired data and audit logs to further help us understand the underlying issue.
Next update by: Tuesday, March 31, 2020, at 5:00 PMUTC; PublishedTime=2020-03-31T03:09:53.943Z

Status: We're investigating the issue holistically by reviewing data and similar incidents to have a better understanding of the underlying issue.
Next update by: Tuesday, March 31, 2020, at 4:00 AMUTC; PublishedTime=2020-03-31T01:55:05.4Z

Status: We're awaiting the requested audit logs from some of the impacted users. Additionally, we will work with them to gather the channel thread IDs for the missing channels to further our investigation.
Next update by: Tuesday, March 31, 2020, at 2:00 AM UTC; PublishedTime=2020-03-30T23:53:18.167Z

Status: While we check channel configuration information, we've requested your organization provide audit logs for review to determine the next steps in our investigation.
Next update by: Tuesday, March 31, 2020, at 12:00 AM UTC; PublishedTime=2020-03-30T21:53:46.797Z

Status: We're investigating a potential issue where users are unable to see channels for certain groups in Microsoft Teams. We'll provide an update within 30 minutes.; PublishedTime=2020-03-30T21:26:53.53Next update by: Tuesday, March 31, 2020, at 12:00 AM UTC; PublishedTime=2020-03-30T21:53:46.797Z

Status: We've determined that a code issue within Microsoft Teams is preventing users from viewing some channels within Microsoft Teams. We have disabled the associated change and are working on mitigating impact for the affected channels. Please reach out to our support resources and provide the impacted teamID in order to expedite our mitigation steps.
Next update by: Wednesday, April 1, 2020, at 1:00 AMUTC; PublishedTime=2020-03-31T20:39:26.707Z

Status: We're developing a fix for this issue which is preparing to undergo validation. In parallel, we're running an additional script to identify potentially impacted channels for further repair. Please reach out to our support resources and provide the impacted team ID in order to expedite our mitigation steps. To get this ID, you can use the "Get link to team" option accessible by clicking "..." next to the team name.
Next update by: Thursday, April 2, 2020, at 1:00 AMUTC; PublishedTime=2020-03-31T23:35:41.953Z

Status: We've prepared a potential remediation to identify and restore visibility to the affected Teams channels within user channel lists. We're testing this on a portion of affected Teams channels to confirm that this successfully restores visibility of the affected channels.
Next update by: Thursday, April 2, 2020, at 1:00 AM UTC; PublishedTime=2020-04-01T03:03:18.397Z

Status: We've validated the fix and have confirmed visibility has been restored to many channels that were reported as being affected through Tuesday, March 31, 2020. We're running a crawl to proactively identify and fix additional channels that may be affected by the event. We anticipate this process to complete by the end of this week.
Next update by: Saturday, April 4, 2020, at 1:00 AM UTC; PublishedTime=2020-04-02T00:16:23.713Z

Status: We've confirmed that the majority of the missing channels have been fixed. We're continuing to identify the remaining affected channels and will have these fixed by Sunday, April 5, 2020.
Next update by: Monday, April 6, 2020, at 1:00 AMUTC; PublishedTime=2020-04-03T23:47:59.113Z

Final status: We've confirmed that the remaining missing channels have been fixed and that the incident is resolved.
Next update by: Monday, April 6, 2020, at 1:00 AMUTC; PublishedTime=2020-04-03T23:47:59.113Z

--------------------------------------------------------------------------------------------------
Title Poor audio and video quality
ID LY206705
Service Skype for Business
Feature Audio and Video
Status Service degradation
Impact Users may experience poor audio and video quality when using Skype for Business.
Severity Sev2
Start time 2020-03-12T22:12:00Z
End time
Last update 2020-04-03T20:36:13.35Z
More info Skype Meeting Broadcasts are unaffected by this issue.
Root cause Network congestion on a portion of infrastructure responsible for the quality of audio and video functionality in Skype for Business is unexpectedly causing service impact.

LY206705 Details

Status: We're analyzing diagnostic data to determine the source of the issue.
Next update by: Tuesday, March 17, 2020, at 4:30 PM UTC; PublishedTime=2020-03-17T14:28:38.02Z

Status: We're reviewing network traces and Call Quality Dashboard (CQD) reports to determine if there is a network issue contributing to the problem.
Next update by: Tuesday, March 17, 2020, at 6:30 PM UTC; PublishedTime=2020-03-17T16:27:04.417Z

Status: We've identified network congestion that is contributing to the audio and video quality issues in Skype for Business. We're preparing to implement a configuration change to reprioritize network traffic. Once applied, it could take up to eight hours to propagate throughout the environment.
Next update by: Wednesday, March 18, 2020, at 6:00 PM UTC; PublishedTime=2020-03-17T17:34:34.817Z

Status: We've applied the configuration change to the affected infrastructure and are attempting to review monitoring information to confirm the issue has been resolved.
Next update by: Wednesday, March 18, 2020, at 8:00 PMUTC; PublishedTime=2020-03-18T17:41:09.91Z

Status: We're optimizing infrastructure that facilitates audio and video usage, and are monitoring the service as this propagates throughout the affected infrastructure.
Next update by: Thursday, March 19, 2020, at 10:00 PM UTC; PublishedTime=2020-03-19T10:39:20.86Z

Status: We're reviewing additional infrastructure facilitating audio and video usage to verify if there is further optimization needed for additional network traffic.
Next update by: Thursday, March 19, 2020, at 10:00 PMUTC; PublishedTime=2020-03-18T19:38:59.913Z

Status: We're redirecting audio and video usage traffic via alternate infrastructure once European business hours have completed, to minimize any further disruption to the service.
Next update by: Thursday, March 19, 2020, at 10:00 PM UTC; PublishedTime=2020-03-19T14:03:59.563Z

Status: We've begun redirecting audio and video usage traffic to alternate infrastructure for European traffic as their business hours have completed. We'll begin doing the same for North America once their business hours have completed, to minimize any further disruption to the service.
Next update by: Friday, March 20, 2020, at 8:00 PMUTC ; PublishedTime=2020-03-19T20:58:19.15Z

Status: We've confirmed that our redirecting of audio and video usage traffic has been successful in reducing the frequency of the problem, although some users are reporting continued intermittent issues. We'll continue monitoring for problems over the next three days to determine if additional actions are necessary.
Next update by: Monday, March 23, 2020, at 8:30 PMUTC; PublishedTime=2020-03-20T19:44:23.793Z

Status: Our monitoring has shown improvement for some affected users. We are continuing to optimize bandwidth by adjusting limits to alleviate traffic and continue to monitor until healthy.
Next update by: Wednesday, March 25, 2020, at 8:30 PMUTC ; PublishedTime=2020-03-23T20:29:23.023Z

Status: While we continue to optimize bandwidth, we're also optimizing the affected infrastructure so that it can operate more efficiently and stabilize the service. This is expected to take some time to complete; however we're monitoring traffic and service health to prevent additional impact.
Next update by: Friday, March 27, 2020, at 9:00 PM UTC; PublishedTime=2020-03-25T19:43:22.117Z

Status: We're continuing to perform enhancements to the infrastructure facilitating audio and video services in parallel with our optimizations to restore service quality.
Next update by: Tuesday, March 31, 2020, at 9:00 PMUTC; PublishedTime=2020-03-27T19:18:29.973Z

Status: While the process of enhancing the impacted infrastructure is ongoing, we're seeing improved telemetry for most affected environments. We're continuing to optimize infrastructure in the impacted environment to resolve this issue for all impacted users.
Next update by: Wednesday, April 1, 2020, at 9:00 PMUTC; PublishedTime=2020-03-31T20:56:34.9Z

Status: An unexpected delay is slowing our ability to enhance the impacted infrastructure. We're still seeing improved telemetry for most affected environments due to our continuing effort to optimize the impacted infrastructure and resolve user impact.
Next update by: Thursday, April 2, 2020, at 9:00 PMUTC; PublishedTime=2020-04-01T19:48:25.77Z

Status: We're continuing to experience a delay that is hindering our ability to complete our enhancement to a subsection of impacted infrastructure. Following our initial optimizations to the impacted infrastructure, we're continuing to see improved telemetry for a majority of the affected environments. We're expecting to have a more accurate ETA for the completion of our infrastructure enhancements by Friday, April 3, 2020, at 9:00 PM UTC.
Next update by: Friday, April 3, 2020, at 9:00 PMUTC ; PublishedTime=2020-04-02T20:43:56.73Z

Status: Our enhancement to a subsection or impacted infrastructure is taking longer than expected as we expect to have it completed by next week. We are also working on an exception process that will allow us to on-board customers faster to the updated infrastructure (when possible) and expect to publish that process by Monday, April 6th, at 9:00pm UTC.
Next update by: Monday, April 6, 2020, at 7:00 PMUTC ; PublishedTime=2020-04-03T20:36:13.35Z

--------------------------------------------------------------------------------------------------
Title Problem with commerce purchases
ID MO206343
Service Office 365 Portal
Feature Purchase and Billing
Status Service restored
Impact Users may have experienced intermittent errors when attempting to make payments or
Severity Sev2
Start time 2020-03-12T22:00:00Z
End time 2020-03-12T22:45:00Z
Last update 2020-03-13T00:08:36.29Z
More info
Root cause A recent configuration change caused unintended impact.

MO206343 Details

Status: We identified a configuration issue that is causing some requests to fail. We're analyzing system logs to help identify the source of the issue and determine our next troubleshooting steps.
Next update by: Friday, March 13, 2020, at 1:00 AMUTC; PublishedTime=2020-03-12T23:17:11.053Z

Final status: We fixed the configuration issue and confirmed through our monitoring that the impact was remediated.
Next update by: Friday, March 13, 2020, at 1:00 AMUTC; PublishedTime=2020-03-12T23:17:11.053Z

--------------------------------------------------------------------------------------------------
Title Delays and errors processing videos in Microsoft Stream
ID MO206397
Service Office 365 Portal
Feature Portal
Status Service restored
Impact Users may have experienced longer than normal video processing times and upload errors
Severity Sev2
Start time 2020-03-12T15:02:00Z
End time 2020-03-23T16:00:00Z
Last update 2020-03-24T19:40:57.59Z
More info Admins could have checked the data center your tenant was tied to from your Stream website: Microsoft Stream > ' > About Microsoft Stream > Your data was stored in'
Root cause A code issue within Azure Media Services (AMS) caused batch calls to hit the quota limit, resulting in periodic delays in video processing and a small percentage of failures that would have automatically retried.

MO206397 Details

Status: We've identified a coding issue within Azure Media Services (AMS) that is causing batch calls to hit their quota limit, resulting in periodic delays in video processing and a small percentage of failures. We're working to identify a fix to mitigate the impact.
Next update by: Friday, March 13, 2020, at 10:30 PMUTC; PublishedTime=2020-03-13T20:10:01.063Z

Status: We're currently deleting old jobs to free up quota to mitigate the delays while we continue to investigate a fix for the coding issue that is causing the errors.
Next update by: Monday, March 16, 2020, at 6:00 PM UTC; PublishedTime=2020-03-13T22:29:28.343Z

Status: The AMS code issue has been remediated, and we're seeing significant improvement with upload latency and failures. For videos that remain in the queues, we have initiated a reprocessing of the queues to ensure they are uploaded as expected.
Next update by: Wednesday, March 18, 2020, at 9:00 PM UTC; PublishedTime=2020-03-16T16:54:52.803Z

Status: Reprocessing of videos in the queues is continuing and progressing as expected so far. Due to the amount of videos built up in queue, we expect processing to take over a week's time, after which impact will be fully mitigated.
Next update by: Wednesday, March 25, 2020, at 9:00 PMUTC; PublishedTime=2020-03-18T19:09:24.74Z

Final status: After monitoring the queues, we've validated that the affected videos have been successfully reprocessed.
Next update by: Wednesday, March 25, 2020, at 9:00 PMUTC; PublishedTime=2020-03-18T19:09:24.74Z

--------------------------------------------------------------------------------------------------
Title Assignment issues
ID TM206299
Service Microsoft Teams
Feature Teams Components
Status Service restored
Impact Users were experiencing delays or getting errors while accessing Assignments.
Severity Sev2
Start time 2020-03-12T08:00:00Z
End time 2020-03-12T19:35:00Z
Last update 2020-03-13T16:45:32.7Z
More info
Root cause An unexpected increase in user traffic was causing high latency and errors when attempting to access the Assignments app in Teams.

TM206299 Details

Status: We're analyzing diagnostic data to determine the source of the issue.
Next update by: Thursday, March 12, 2020, at 2:00 PM UTC; PublishedTime=2020-03-12T12:19:52.74Z

Status: We've identified that an unexpected increase in user requests caused impact. We're connecting the service to additional healthy infrastructure to re-distribute load and optimize performance.
Next update by: Thursday, March 12, 2020, at 6:00 PM UTC; PublishedTime=2020-03-12T13:53:56.76Z

Status: We're continuing to add additional resources to optimize the affected infrastructure while re-routing some traffic to alternate infrastructure.
Next update by: Thursday, March 12, 2020, at 8:00 PMUTC; PublishedTime=2020-03-12T17:30:38.437Z

Final status: We've completed adding additional resources to the affected infrastructure and confirmed that this, in parallel with redirecting some of the impacted traffic, has resolved the issue via monitoring.
Next update by: Thursday, March 12, 2020, at 8:00 PMUTC; PublishedTime=2020-03-12T17:30:38.437Z

Status: We're investigating a potential issue and checking for impact to your organization. We'll provide an update within 30 minutes.; PublishedTime=2020-03-12T11:58:22.407Next update by: Thursday, March 12, 2020, at 8:00 PMUTC; PublishedTime=2020-03-12T17:30:38.437Z

--------------------------------------------------------------------------------------------------
Title Autocomplete and @mention not working
ID YA206308
Service Yammer Enterprise
Feature Yammer Components
Status Service restored
Impact Users were unable to use the autocomplete and @mention function in certain scenarios.
Severity Sev2
Start time 2020-03-12T07:58:00Z
End time 2020-03-12T15:02:00Z
Last update 2020-03-12T16:56:29.38Z
More info Autocomplete and @mention may have failed for the below scenarios:
Root cause A portion of infrastructure that facilitates autocomplete and search functionality was performing below expected thresholds.

YA206308 Details

Status: We're investigating a potential issue and checking for impact to your organization. We'll provide an update within 30 minutes.; PublishedTime=2020-03-12T14:30:38.967

Status: We've determined that a portion of infrastructure that facilitates autocomplete and search functionality is performing below acceptable thresholds due to an unusually high number of requests. We're balancing affected traffic throughout the environment to mitigate impact.
Next update by: Thursday, March 12, 2020, at 5:00 PMUTC; PublishedTime=2020-03-12T15:04:52.867Z

Final status: We've completed traffic rebalancing throughout the affected environment and confirmed that the issue is resolved.
Next update by: Thursday, March 12, 2020, at 5:00 PMUTC; PublishedTime=2020-03-12T15:04:52.867Z

--------------------------------------------------------------------------------------------------
Title Unable to access Planner
ID PL206214
Service Planner
Feature Planner
Status Service restored
Impact Users may have been unable to access the Planner service.
Severity Sev2
Start time 2020-03-11T01:30:00Z
End time 2020-03-11T15:20:00Z
Last update 2020-03-11T17:17:32.567Z
More info This issue would have affected access to Planner resources using any protocol, such as Microsoft Teams or other applications.
Root cause A portion of Planner infrastructure was experiencing issues processing traffic, which resulted in problems accessing the service.

PL206214 Details

Status: We've received reports of an issue affecting the Planner service. We'll provide an update within the next 30 minutes.; PublishedTime=2020-03-11T15:04:40.537

Status: We've identified a portion of Planner infrastructure that is experiencing issues processing traffic, which is resulting in problems accessing the service. We've restarted the affected infrastructure and are performing tests to ensure that impact has been mitigated.
Next update by: Wednesday, March 11, 2020, at 5:30 PM UTC; PublishedTime=2020-03-11T15:24:42.66Z

Final status: We've successfully completed our testing and have confirmed with affected users that the issue has been resolved.
Next update by: Wednesday, March 11, 2020, at 5:30 PM UTC; PublishedTime=2020-03-11T15:24:42.66Z

--------------------------------------------------------------------------------------------------
Title Can't mention
ID TM206129
Service Microsoft Teams
Feature Teams Components
Status Service restored
Impact Users may have been unable to mention recently added teams members.
Severity Sev2
Start time 2020-03-10T18:00:00Z
End time 2020-03-10T21:03:00Z
Last update 2020-03-10T21:18:55.29Z
More info Additionally, mentions may have still completed for recently removed team members.
Root cause A portion of system infrastructure that facilitates mention requests wasn't processing requests as efficiently as expected.

TM206129 Details

Status: We've identified a possible issue in a portion of system infrastructure that facilitates mention requests, resulting in mention requests processing less efficiently than expected. We're working to rebalance load to resolve the issue.
Next update by: Tuesday, March 10, 2020, at 10:30 PMUTC; PublishedTime=2020-03-10T20:30:10.467Z

Status: We're investigating a potential issue and checking for impact to your organization. We'll provide an update within 30 minutes.; PublishedTime=2020-03-10T20:08:01.09Next update by: Tuesday, March 10, 2020, at 10:30 PMUTC; PublishedTime=2020-03-10T20:30:10.467Z

Final status: We've completed our rebalancing efforts and verified via telemetry that this issue has been resolved.
Next update by: Tuesday, March 10, 2020, at 10:30 PMUTC; PublishedTime=2020-03-10T20:30:10.467Z

--------------------------------------------------------------------------------------------------
Title Unknown user in Teams
ID TM206109
Service Microsoft Teams
Feature Teams Components
Status Service restored
Impact Users may see an unknown user appear within teams in the Microsoft Teams service.
Severity Sev2
Start time 2020-03-09T23:00:00Z
End time 2020-03-10T17:00:00Z
Last update 2020-03-11T22:06:47.75Z
More info An unknown user wasn't being added to team channels. The problem was cosmetic in nature.
Root cause A recent code change was causing the application that was responsible for team creation to arbitrarily appear within channels as an unknown user.

TM206109 Details

Status: We've requested diagnostic data from your organization, to help us determine the root cause of impact.
Next update by: Tuesday, March 10, 2020, at 4:30 PM UTC ; PublishedTime=2020-03-10T14:42:53.093Z

Status: We've discovered that a recent code change is causing the application that is responsible for team creation to arbitrarily appear within channels as an unknown user. We're exploring options to remediate impact.
Next update by: Tuesday, March 10, 2020, at 11:00 PMUTC; PublishedTime=2020-03-10T16:34:05.277Z

Status: We've rolled back the code change to prevent further instances of this issue from occurring and we're exploring options to remove the team creation application from team membership.
Next update by: Tuesday, March 10, 2020, at 11:00 PMUTC; PublishedTime=2020-03-10T18:51:27.497Z

Status: Reverting the code change is expected to remediate impact in the next 24 hours when the automated team creation process triggers and removes the unknown user from the team.
Next update by: Wednesday, March 11, 2020, at 11:00 PMUTC; PublishedTime=2020-03-10T22:26:13.54Z

Final status: We've completed reverting the code change and confirmed that impact has been mitigated.
Next update by: Wednesday, March 11, 2020, at 11:00 PMUTC; PublishedTime=2020-03-10T22:26:13.54Z

--------------------------------------------------------------------------------------------------
Title Can't make PSTN calls
ID TM205962
Service Microsoft Teams
Feature Teams Components
Status Service restored
Impact Users may have been unable to make outbound Public Switched Telephone Network (PSTN)
Severity Sev2
Start time 2020-03-09T13:14:00Z
End time 2020-03-09T13:16:00Z
Last update 2020-03-09T14:33:01.977Z
More info
Root cause

TM205962 Details

Status: We're investigating a potential issue with the Microsoft Teams service. We'll provide an update within 30 minutes.; PublishedTime=2020-03-09T14:07:06.727

Final status: While we were in the process of reviewing diagnostic data to isolate the source of the problem, system monitoring indicated that the service had returned to normal health. We've confirmed that the problem is no longer occurring, and we'll continue to monitor the service to ensure that the problem does not happen again.


--------------------------------------------------------------------------------------------------
Title Slowness and timeouts
ID SP205965
Service SharePoint Online
Feature SharePoint Features
Status Service restored
Impact Users may have experienced slowness or timeouts when attempting to access SharePoint
Severity Sev2
Start time 2020-03-09T13:00:00Z
End time 2020-03-09T16:30:00Z
Last update 2020-03-09T17:04:21.647Z
More info
Root cause An unusually high amount of file sync traffic has resulted in a portion of SharePoint Online infrastructure performing below expected thresholds.

SP205965 Details

Status: We're reviewing service telemetry to isolate the source of the issue. Additionally, we're implementing targeted throttling on specific request to prevent further impact.
Next update by: Monday, March 9, 2020, at 4:30 PM UTC; PublishedTime=2020-03-09T14:43:15.617Z

Status: We've determined that an unusually high amount of file sync traffic has resulted in a portion of SharePoint Online infrastructure performing below expected thresholds. We've applied sync traffic flow management measures to a subset of files in the affected environment to mitigate impact. We're monitoring the effectiveness of this action to ensure that is resolves the issue.
Next update by: Monday, March 9, 2020, at 6:30 PMUTC; PublishedTime=2020-03-09T16:25:35.793Z

Final status: We've confirmed that service health and availability have returned to normal levels following our sync traffic flow management measures to resolve the issue.
Next update by: Monday, March 9, 2020, at 6:30 PMUTC; PublishedTime=2020-03-09T16:25:35.793Z

--------------------------------------------------------------------------------------------------
Title Slowness and timeouts
ID SP205965
Service SharePoint Online
Feature SharePoint Features
Status Service restored
Impact Users may have experienced slowness or timeouts when attempting to access SharePoint
Severity Sev2
Start time 2020-03-09T13:00:00Z
End time 2020-03-09T16:30:00Z
Last update 2020-03-09T17:04:21.647Z
More info
Root cause An unusually high amount of file sync traffic has resulted in a portion of SharePoint Online infrastructure performing below expected thresholds.

SP205965 Details

Status: We're reviewing service telemetry to isolate the source of the issue. Additionally, we're implementing targeted throttling on specific request to prevent further impact.
Next update by: Monday, March 9, 2020, at 4:30 PM UTC; PublishedTime=2020-03-09T14:43:15.617Z

Status: We've determined that an unusually high amount of file sync traffic has resulted in a portion of SharePoint Online infrastructure performing below expected thresholds. We've applied sync traffic flow management measures to a subset of files in the affected environment to mitigate impact. We're monitoring the effectiveness of this action to ensure that is resolves the issue.
Next update by: Monday, March 9, 2020, at 6:30 PMUTC; PublishedTime=2020-03-09T16:25:35.793Z

Final status: We've confirmed that service health and availability have returned to normal levels following our sync traffic flow management measures to resolve the issue.
Next update by: Monday, March 9, 2020, at 6:30 PMUTC; PublishedTime=2020-03-09T16:25:35.793Z

--------------------------------------------------------------------------------------------------
Title Windows device import issue
ID IT205818
Service Microsoft Intune
Feature Microsoft Intune
Status Service restored
Impact Admins may have been unable to import new autopilot devices.
Severity Sev2
Start time 2020-03-07T06:29:00Z
End time 2020-03-07T20:00:00Z
Last update 2020-03-07T20:16:18.05Z
More info
Root cause Infrastructure that handles Device Directory Service (DDS) authentication encountered a communication problem that resulted in impact.

IT205818 Details

Status: We're investigating a potential issue and checking for impact to your organization. We'll provide an update within 30 minutes.; PublishedTime=2020-03-07T14:17:13.853

Status: We're analysing diagnostic data to identify the next troubleshooting steps.
Next update by: Saturday, March 7, 2020, at 5:00 PM UTC; PublishedTime=2020-03-07T14:40:28.927Z

Status: We're continuing to review available diagnostic data, such as service logs, to isolate the cause of impact and determine optimal troubleshooting actions.
Next update by: Saturday, March 7, 2020, at 7:00 PMUTC; PublishedTime=2020-03-07T16:43:43.947Z

Status: We've identified that infrastructure that handles Device Directory Service (DDS) authentication encountered a communication problem that resulted in impact. We're deploying a fix to mitigate the communication problem and expect it to be fully saturated in the affected infrastructure within the next hour.
Next update by: Saturday, March 7, 2020, at 9:00 PMUTC; PublishedTime=2020-03-07T18:48:43.497Z

Final status: We've deployed the fix in full to the affected environment and we've confirmed that it has successfully resolved the issue.
Next update by: Saturday, March 7, 2020, at 9:00 PMUTC; PublishedTime=2020-03-07T18:48:43.497Z

--------------------------------------------------------------------------------------------------
Title Some users are unable to create Microsoft Teams channels within specific teams
ID TM207796
Service Microsoft Teams
Feature Teams Components
Status Service restored
Impact A small number of users may have been unable to create Microsoft Teams channels within
Severity Sev2
Start time 2020-03-04T21:00:00Z
End time 2020-03-29T14:00:00Z
Last update 2020-03-30T17:40:18.93Z
More info
Root cause A code issue is causing a specific cache to become stale, resulting in the channel creation in Microsoft Teams not working as designed.

TM207796 Details

Final status: The fix has completed deployment to all impacted environments and we've confirmed with users that the issue is resolved.


Status: We've begun deployment of the fix which has saturated approximately five percent of the affected environment. We expect that the deployment should complete on Tuesday, March 31, 2020. We'll monitor the progress of the fix saturation to ensure it remains on schedule.
Next update by: Monday, March 30, 2020, at 6:00 PMUTC; PublishedTime=2020-03-27T16:19:17.243Z

Status: We're continuing to validate the fix in an internal testing environment to confirm it does not cause any additional impact. Once validated, we'll begin deploying it to the affected infrastructure to remediate impact.
Next update by: Friday, March 27, 2020, at 9:00 PM UTC; PublishedTime=2020-03-26T15:01:42.673Z

Status: We've investigated this issue by reviewing numerous logs, and have determined a hotfix code change is needed to restore the functionality of creating Microsoft Teams channels. We've developed a fix and are in the deployment process to our test environment to ensure that the issue is fully mitigated prior to rolling this out to the full production environment. Once we confirm the fix is successful, we'll start deployment to the affected environments to full mitigate the issue.
Next update by: Saturday, March 28, 2020, at 2:00 AMUTC; PublishedTime=2020-03-26T02:32:32.323Z

--------------------------------------------------------------------------------------------------
Title Dial pad issues in Teams
ID TM205715
Service Microsoft Teams
Feature Teams Components
Status Service restored
Impact Users may have been unable to access the dial pad or perform specific actions using the
Severity Sev2
Start time 2020-03-04T00:00:00Z
End time 2020-03-11T10:20:00Z
Last update 2020-03-11T23:45:18.903Z
More info Actions that may be unable to be performed include, making calls, accepting calls, forwarding calls, or other scenarios that have been set by admins.
Root cause A configuration problem has caused a data sync issue to the Business Voice Directory (BVD), resulting in impact to dial pad appearance and functionality.

TM205715 Details

Status: We've determined that a configuration problem has caused a data sync issue to the Business Voice Directory (BVD), resulting in impact to dial pad appearance and functionality. We've updated the configuration which is currently propagating throughout the affected environment. We expect that this process should take up to four hours. Once complete, processing the additional backlog changes may take up to 12 hours.
Next update by: Saturday, March 7, 2020, at 12:30 AMUTC; PublishedTime=2020-03-06T19:56:53.48Z

Status: We're analyzing provided client logs and reviewing affected users' configurations to determine the source of the issue.
Next update by: Friday, March 6, 2020, at 8:00 PMUTC; PublishedTime=2020-03-06T18:04:20.757Z

Status: The configuration update has been completed and any updates to dial pad should now complete as expected. We're replaying updates made since Friday, March 6, 2020, at 12:00 AM UTC so that currently affected user issues will be mitigated. While we expect this process may last through Monday, March 9, 2020, we're reviewing possible ways to accelerate these changes.
Next update by: Saturday, March 7, 2020, at 4:00 AMUTC; PublishedTime=2020-03-06T21:33:59.547Z

Status: The replay of updates made since Friday March 6, 2020 at 12:00 AM UTC is progressing as expected. At the rate it's progressing, we expect the replays to complete by Sunday, March 8, 2020. We'll continue monitoring its progress and provide an update.
Next update by: Sunday, March 8, 2020, at 3:30 AMUTC ; PublishedTime=2020-03-07T02:38:47.707Z

Status: Replaying updates in North America is taking longer than originally anticipated. We now expect that the replays in this region will complete within the next 24 hours.
Next update by: Wednesday, March 11, 2020, at 10:00 PMUTC; PublishedTime=2020-03-10T20:33:23.77Z

Status: We've finished replaying updates for the Europe, Middle East and Africa regions. We're continuing to replay updates in North America, and expect the issue to be resolved for all users within the next seven hours.
Next update by: Tuesday, March 10, 2020, at 10:00 PM UTC; PublishedTime=2020-03-10T14:28:35.073Z

Status: The replay of updates continues to progress, albeit at a slower rate than expected. At the rate it's currently progressing, we expect the replays to complete by Monday, March 9, 2020.
Next update by: Monday, March 9, 2020, at 8:00 AMUTC; PublishedTime=2020-03-08T01:52:02.473Z

Final status: We've confirmed that replays in North America have completed and that impact has been fully remediated.
Next update by: Monday, March 9, 2020, at 8:00 AMUTC; PublishedTime=2020-03-08T01:52:02.473Z

Status: We've finished replaying updates for the Asia Pacific region. We're continuing to replay updates in the remaining regions, and expect the process to complete by Tuesday, March 10, 2020.
Next update by: Tuesday, March 10, 2020, at 4:00 PM UTC; PublishedTime=2020-03-09T07:53:17.707Z

--------------------------------------------------------------------------------------------------
Title Can't sign in to Outlook on the web
ID EX206326
Service Exchange Online
Feature Sign-in
Status Service restored
Impact Users may have experienced intermittent access issues when signing in from Outlook on
Severity Sev2
Start time 2020-02-25T08:00:00Z
End time 2020-03-17T16:45:00Z
Last update 2020-03-17T17:20:14.487Z
More info This was an incremental update to provide the latest status on the issue.
Root cause A recent deployment inadvertently introduced a configuration issue affecting the infrastructure that hosts the sign-in process for Outlook on the web.

EX206326 Details

Status: We've determined that this issue is being caused by a recent deployment that inadvertently introduced a configuration issue affecting the infrastructure hosting the sign in process for Outlook on the web, which resulted in impact. We've halted deployment of the affected build and developed a fix for the configuration problem that's being deployed to the affected environment.
Next update by: Friday, March 13, 2020, at 1:00 AMUTC; PublishedTime=2020-03-12T18:19:54.187Z

Status: Our initial deployment of the fix was unsuccessful, and we have begun a new deployment of the fix to the affected environments to mitigate impact.
Next update by: Friday, March 13, 2020, at 7:00 PMUTC; PublishedTime=2020-03-12T23:28:00.167Z

Status: The fix deployment has saturated 10 percent of the affected environment. We're monitoring this process and will provide an accurate deployment timeline when one becomes available.
Next update by: Saturday, March 14, 2020, at 8:00 PMUTC; PublishedTime=2020-03-13T17:43:23.51Z

Status: We've confirmed that the fix is successfully propagating in the affected environment and deployment saturation has reached 75 percent. We're continuing to monitor the progress of the fix and currently suspect the fix will be fully saturated in the next 24 hours.
Next update by: Sunday, March 15, 2020, at 8:00 PMUTC; PublishedTime=2020-03-14T16:06:54.747Z

Status: The fix has saturated approximately 90 percent of the infrastructure impacted by this event. For complete details on this event, please reference the post from 11:06 PM UTC on March 14, 2020.
Next update by: Monday, March 16, 2020, at 1:00 AM UTC ; PublishedTime=2020-03-15T17:37:08.76Next update by: Monday, March 16, 2020, at 1:00 AM UTC ; PublishedTime=2020-03-15T17:37:08.76Z

Status: The fix is continuing to saturate throughout the affected infrastructure. For complete details on this event, please reference the post from 11:06 PM UTC on March 14, 2020.
Next update by: Tuesday, March 17, 2020, at 1:00 AMUTC; PublishedTime=2020-03-16T00:32:36.407Next update by: Tuesday, March 17, 2020, at 1:00 AMUTC; PublishedTime=2020-03-16T00:32:36.407Z

Status: The fix has reached the vast majority of the affected infrastructure. We're monitoring to ensure the fix saturates the remaining infrastructure. For complete details on this event, please reference the post from 11:06 PM UTC on March 14, 2020.
Next update by: Thursday, March 19, 2020, at 1:00 AMUTC; PublishedTime=2020-03-16T23:36:55.207Next update by: Thursday, March 19, 2020, at 1:00 AMUTC; PublishedTime=2020-03-16T23:36:55.207Z

Final status: The fix has been completely rolled out to the affected infrastructure and we've confirmed via telemetry that the issue is resolved.
Next update by: Thursday, March 19, 2020, at 1:00 AMUTC; PublishedTime=2020-03-16T23:36:55.207Z

--------------------------------------------------------------------------------------------------
Title Android automatic line of business app updates failing
ID IT206230
Service Microsoft Intune
Feature Microsoft Intune
Status Service restored
Impact Users were experiencing failures of automatic line of business app updates on Samsung
Severity Sev2
Start time 2020-02-24T22:20:00Z
End time 2020-03-14T13:00:00Z
Last update 2020-03-15T18:05:30.753Z
More info This only affected devices that were enrolled using device administrator.
Root cause A recent update to the Android OS has blocked some core activities, resulting in impact.

IT206230 Details

Status: We've isolated a recent update to the Android OS that has blocked some core activities to Samsung devices, resulting in impact. We're developing a fix to mitigate the issue.
Next update by: Wednesday, March 11, 2020, at 11:00 PMUTC; PublishedTime=2020-03-11T18:04:07.233Z

Status: We've completed developing the fix and are working with internal and external resources to validate the fix prior to deployment.
Next update by: Thursday, March 12, 2020, at 2:30 AMUTC; PublishedTime=2020-03-11T22:02:39.53Z

Status: We're continuing to validate the fix internally before we deploy it. We're also reaching out to some affected users to assist with validation. Furthermore, our investigation has determined that this issue may have begun occurring when Samsung devices started using Android 10 and above.
Next update by: Thursday, March 12, 2020, at 6:30 PMUTC; PublishedTime=2020-03-12T02:00:27.233Z

Status: Validation of the fix is taking longer than anticipated due to an unexpected delay of a necessary update to some affected users' Company Portals. This update has become available and the validation process is proceeding.
Next update by: Thursday, March 12, 2020, at 9:30 PMUTC; PublishedTime=2020-03-12T17:12:10.253Z

Status: We've confirmed with your representatives that the fix was successful in resolving the issue. We're making the fix available to a small subset of impacted users through Google Play to collect telemetry for additional validation before making the fix available to all impacted users.
Next update by: Friday, March 13, 2020, at 3:00 PMUTC; PublishedTime=2020-03-12T21:03:59.573Z

Status: We've completed our additional validations of the collected telemetry of a set of impacted users who have received the fix, and we're deploying the fix to all impacted users.
Next update by: Saturday, March 14, 2020, at 7:00 PM UTC; PublishedTime=2020-03-13T13:27:32.517Z

Status: We've confirmed that the fix has saturated to 100 percent of the affected environment and we're working to ensure this fix successfully mitigated the issue in full.
Next update by: Sunday, March 15, 2020, at 7:00 PMUTC; PublishedTime=2020-03-14T17:31:45.82Z

Final status: Following a period of monitoring the environment, we've confirmed that the fix available through Google Play Store has remediated impact.
Next update by: Sunday, March 15, 2020, at 7:00 PMUTC; PublishedTime=2020-03-14T17:31:45.82Z

--------------------------------------------------------------------------------------------------
Title Can't search in Outlook on the web light
ID EX205600
Service Exchange Online
Feature E-Mail and calendar access
Status Service restored
Impact Users were receiving an error message when attempting to search within Outlook on the
Severity Sev2
Start time 2020-01-28T00:00:00Z
End time 2020-03-12T17:00:00Z
Last update 2020-03-13T01:12:05.257Z
More info Users were receiving the following error message: "Your request couldn't be completed."
Root cause A recent code change was causing the browser session cookie to become unreadable during Outlook on the web light searches.

EX205600 Details

Status: We're investigating a potential issue where users are receiving an error message when attempting to search within Outlook on the web light. We'll provide an update within 30 minutes.; PublishedTime=2020-03-05T22:57:34.443

Status: We're reviewing infrastructure logs to identify any inconsistencies that may determine a possible source of the issue.
Next update by: Friday, March 6, 2020, at 2:00 AM UTC; PublishedTime=2020-03-05T23:29:08.11Z

Status: We're deploying a potential fix and we're working to confirm whether this fix will resolve impact experienced by users.
Next update by: Friday, March 6, 2020, at 8:00 PMUTC ; PublishedTime=2020-03-06T00:37:45.867Z

Status: Deployment of the fix is underway. We expect that it will take about nine days for the deployment to fully saturate the impacted environment and resolve this issue for all impacted users.
Next update by: Monday, March 9, 2020, at 8:00 PMUTC; PublishedTime=2020-03-06T18:32:45.877Z

Status: Deployment of the fix is at 86 percent. We expect that it will take another three to four days for the deployment to fully saturate the impacted environment and resolve this issue for all impacted users.
Next update by: Thursday, March 12, 2020, at 8:00 PMUTC; PublishedTime=2020-03-09T19:51:01.417Z

Final status: We've verified that the fix has fully saturated throughout the affected environment, and we've confirmed that the fix has successfully resolved the issue.
Next update by: Thursday, March 12, 2020, at 8:00 PMUTC; PublishedTime=2020-03-09T19:51:01.417Z

--------------------------------------------------------------------------------------------------
Title New and changed query rules used within search queries are slow to take effect
ID SP203888
Service SharePoint Online
Feature Search and Delve
Status Service restored
Impact Users may have not seen best bets or promoted content for up to 24 hours after query
Severity Sev2
Start time 2020-01-26T15:07:00Z
End time 2020-03-20T12:00:00Z
Last update 2020-03-20T12:13:49.027Z
More info Previously defined query rules worked as expected during the 24 hour propagation delay. There was no effect on freshness of the document.
Root cause The best bets and promoted content weren't being collected from caching infrastructure correctly, resulting in impact.

SP203888 Details

Status: Our initial investigation indicates that our query processors are failing to update the results in a timely manner. We're working to implement additional logging capabilities within our service to better understand how the issue is occurring so that we can formulate an action plan for full remediation.
Next update by: Friday, February 14, 2020, at 9:00 PMUTC; PublishedTime=2020-02-14T04:39:04.287Z

Status: We've implemented additional logging to identify why query rule changes are not being propagated quickly. Additionally, we're implementing a further workaround to reduce the refresh time from 24 hours to 1 hour until the root cause is identified and resolved.
Next update by: Friday, February 14, 2020, at 10:00 PM UTC; PublishedTime=2020-02-14T13:34:42.563Z

Status: We've developed a fix to reduce the refresh time to one hour. The fix is being tested and validated before it can be deployed to the impacted environment, which we estimate will occur by midweek, next week.
Next update by: Monday, February 17, 2020, at 2:00 PMUTC; PublishedTime=2020-02-14T20:36:44.82Z

Status: As part of the deployment process, the fix is going through an internal test environment for further validation. Once this validation has completed, we'll begin deploying it to the affected components. The deployment to the affected infrastructure is expected to begin on Wednesday, February 19, 2020.
Next update by: Wednesday, February 19, 2020, at 2:00 PM UTC; PublishedTime=2020-02-17T12:28:41.55Z

Status: Due to an unforeseen issue caused by a misconfiguration with generating new updates, we've been unable to deploy the fix to the internal test environment. The misconfiguration with generating new updates has been corrected and the update will begin deploying to the internal test environment on Thursday, February 20, 2020. Once this update has been deployed, we'll validate the fix before deploying it to the affected infrastructure.
Next update by: Friday, February 21, 2020, at 2:00 PM UTC; PublishedTime=2020-02-19T12:12:39.22Z

Status: We've validated that the short term fix, which reduces the refresh time to one hour, provides the intended relief within our test environment. We're preparing to deploy this fix to the affected infrastructure and expect that the deployment will take approximately one week to complete. Users may begin to experience relief as the fix propagates throughout the environment. Additionally, we're continuing to investigate the root cause and establish a fix to remediate impact fully.
Next update by: Wednesday, February 26, 2020, at 2:00 PM UTC; PublishedTime=2020-02-21T12:23:56.657Z

Status: The short-term fix, which reduces the refresh time to one hour, is 40% complete. This is expected to complete on Monday, March 2, 2020. We're continuing multiple work-streams to understand the root cause and formulate a remediation plan.
Next update by: Monday, March 2, 2020, at 2:00 PM UTC; PublishedTime=2020-02-26T12:16:14.177Z

Status: The short-term fix is now complete. We're continuing to work on the longer-term fix to address the root cause.
Next update by: Friday, March 6, 2020, at 2:00 PM UTC; PublishedTime=2020-03-02T12:16:19.177Z

Status: We've identified that the best bets and promoted content weren't being collected from caching infrastructure correctly, resulting in impact. We've developed a long-term fix to correct the service gathering the data from the caching components. We're testing the fix internally to validate that it does not cause any additional impact. We expect to complete validations by Monday, March 9, 2020 and then we will be deploying the fix to the affected systems.
Next update by: Monday, March 9, 2020, at 2:00 PM UTC; PublishedTime=2020-03-03T12:26:56.447Z

Status: We've validated the long-term fix and have begun deploying it to the affected components. We expect the deployment to complete in approximately two weeks. Users will begin to experience relief as the fix propagates throughout the environment.
Next update by: Thursday, March 12, 2020, at 2:00 PM UTC; PublishedTime=2020-03-09T12:13:35.55Z

Status: The deployment has completed to 69 percent of the affected infrastructure. We're continuing to monitor the deployment as it propagates throughout the environment.
Next update by: Monday, March 16, 2020, at 2:00 PM UTC; PublishedTime=2020-03-12T12:01:09.887Z

Status: The deployment has now reached 84% percent and is progressing as expected.
Next update by: Friday, March 20, 2020, at 2:00 PM UTC; PublishedTime=2020-03-16T12:10:55.903Z

Final status: The deployment has completed to the affected environment and we've confirmed that the issue is resolved.
Next update by: Friday, March 20, 2020, at 2:00 PM UTC; PublishedTime=2020-03-16T12:10:55.903Z

--------------------------------------------------------------------------------------------------
Title Chinese, Japanese, and Korean characters replaced with '''
ID EX203925
Service Exchange Online
Feature E-Mail and calendar access
Status Post-incident report published
Impact Users may have noticed that some Chinese, Japanese, and Korean characters were replaced
Severity Sev2
Start time 2019-11-25T06:41:00Z
End time 2020-03-17T21:12:00Z
Last update 2020-03-25T18:00:17.833Z
More info
Root cause An incompatibility issue occurred due to an update to the encoding provider as part of an anti-spam feature deployment, which resulted in an issue with some character encoding for double-byte characters.

EX203925 Details

Status: After further investigation, we've identified that platform dependent Japanese characters are also affected by this issue. We're continuing our investigation into the recent code changes that may have introduced impact.
Next update by: Thursday, February 27, 2020, at 8:00 PM UTC; PublishedTime=2020-02-20T07:59:00.83Z

Status: Our analysis thus far has been inconclusive to further isolating the source of the issue. Our investigation into the recent code changes that may have introduced impact continues as we narrow our scope of analysis so we can determine our next troubleshooting steps.
Next update by: Thursday, February 27, 2020, at 8:00 PMUTC; PublishedTime=2020-02-20T02:06:03.883Z

Status: We're investigating a potential issue and checking for impact to your organization. We'll provide an update within 30 minutes.; PublishedTime=2020-02-14T17:58:40.813Next update by: Thursday, February 27, 2020, at 8:00 PMUTC; PublishedTime=2020-02-20T02:06:03.883Z

Status: We're continuing to investigate further to isolate the particular change that may have introduce the code issue and address the underlying root cause of the issue.
Next update by: Thursday, February 20, 2020, at 3:00 AMUTC; PublishedTime=2020-02-19T19:36:51.803Z

Status: Our analysis of logs from an internal reproduction of this issue has identified that a recently introduced code problem has caused the impact. We're investigating further to isolate the particular change that may have introduced the code issue as well as identify mitigation actions.
Next update by: Wednesday, February 19, 2020, at 8:00 PMUTC; PublishedTime=2020-02-18T19:08:12.807Z

Status: We've successfully reproduced the issue in our internal testing environment. Due to the nature of the issue, an extended analysis of the logs from the internal reproduction is necessary to determine our next steps towards mitigation.
Next update by: Tuesday, February 18, 2020, at 8:00 PMUTC; PublishedTime=2020-02-15T02:09:13.02Z

Status: We're creating an internal reproduction of the issue to help us better understand the cause of impact and form a remediation strategy.
Next update by: Saturday, February 15, 2020, at 2:30 AMUTC; PublishedTime=2020-02-14T23:27:03.147Z

Status: We're continuing to analyze client logs and debugging to determine next troubleshooting steps.
Next update by: Saturday, February 15, 2020, at 12:30 AMUTC; PublishedTime=2020-02-14T20:11:12.973Z

Status: We're analyzing client logs to determine the source of the issue.
Next update by: Friday, February 14, 2020, at 8:30 PMUTC; PublishedTime=2020-02-14T18:33:38.48Z

Status: We're reviewing the source code to investigate a potential HTML issue that may be contributing to impact.
Next update by: Thursday, February 27, 2020, at 11:00 PMUTC; PublishedTime=2020-02-27T18:54:47.353Z

Status: We're continuing our review of the source code to investigate a potential HTML issue, as thus far we've been unable to identify a potential root cause.
Next update by: Friday, February 28, 2020, at 2:30 AMUTC; PublishedTime=2020-02-27T21:14:30.91Z

Status: The issue has proven difficult to reproduce which is causing the investigation into the HTML code to take an extended amount of time. We're continuing our analysis to pinpoint where the issue is occurring and our next troubleshooting steps.
Next update by: Friday, February 28, 2020, at 7:30 PMUTC; PublishedTime=2020-02-28T02:27:02.62Z

Status: Our review of the HTML code hasn't shown anything conclusive. We're working to reproduce the issue in our test environment under the same criteria in which the problem occurs for affected users so that we can further our troubleshooting efforts.
Next update by: Saturday, February 29, 2020, at 2:00 AMUTC ; PublishedTime=2020-02-28T19:17:03.09Z

Status: We've been able to reproduce the issue in our internal testing environment, but haven't been able to narrow down the source of impact through our internal testing. We're currently producing a code change to capture logging of impacted messages to gain additional data to continue our investigation.
Next update by: Tuesday, March 3, 2020, at 12:30 AMUTC ; PublishedTime=2020-02-29T00:47:21.52Z

Status: Our additional logging capabilities has been successful in capturing necessary details to further troubleshoot the issue. We've identified a potential string encoding issue that may be contributing to the impact and are investigating further to confirm.
Next update by: Wednesday, March 4, 2020, at 12:30 AMUTC; PublishedTime=2020-03-02T22:39:36.83Z

Status: We're continuing our investigation into the string encoding issue that is potentially the underlying cause of impact.
Next update by: Thursday, March 5, 2020, at 12:30 AMUTC; PublishedTime=2020-03-03T22:52:03.927Z

Status: We've determined that an incompatibility issue during code initialization for an anti-spam function is likely resulting in impact. We're updating the incompatible code and verifying that the fix works correctly in our internal test environment prior to a broader release.
Next update by: Friday, March 6, 2020, at 3:00 AMUTC; PublishedTime=2020-03-04T23:32:11.987Z

Status: We've updated the code and deployed the fix to our test environment, but the fix hasn't remediated impact as expected. We're working on developing a new fix to deploy to our test environment that will remove the dependencies on the problem and may resolve the underlining issue.
Next update by: Tuesday, March 10, 2020, at 3:00 AMUTC ; PublishedTime=2020-03-06T02:47:41.957Z

Status: We've developed a new fix that we expect to resolve the underlying issue. We're currently deploying it to our test environment and are looking into options for having it deploy at a faster rate.
Next update by: Wednesday, March 11, 2020, at 2:00 AMUTC ; PublishedTime=2020-03-10T01:24:26.243Z

Status: We've completed development of the expedited build which is starting to deploy to the affected environment. We expect that this process will take up to one week to complete. We'll monitor the deployment and provide a better estimate to full saturation when available.
Next update by: Saturday, March 14, 2020, at 2:00 AMUTC; PublishedTime=2020-03-12T18:02:12.43Z

Status: We're monitoring the continued deployment of the fix while, in parallel, working on developing a new build which could possibly expedite deployment. We'll provide an updated ETA for saturation once available.
Next update by: Friday, March 13, 2020, at 2:00 AMUTC; PublishedTime=2020-03-12T00:53:48.17Z

Status: We're monitoring the fix as we start deploying to the affected environments. We're exploring options to expedite deployment of the fix and will provide an updated ETA for saturation once available.
Next update by: Thursday, March 12, 2020, at 2:00 AMUTC; PublishedTime=2020-03-11T00:46:38.793Z

Status: We've initiated deployment of the fix to the affected environment and it is progressing as expected.
Next update by: Saturday, March 14, 2020, at 9:00 PM UTC; PublishedTime=2020-03-13T18:24:39.993Z

Status: The deployment has reached 53 percent saturation within the affected environment and continues to progress as expected. We're continuing to monitor this activity to ensure it completes successfully.
Next update by: Sunday, March 15, 2020, at 9:00 PM UTC; PublishedTime=2020-03-14T19:27:22.577Z

Status: The deployment is ongoing and the affected environment has reached 78 percent saturation. This activity is progressing at an optimal rate and we'll continue to monitor and provide updates as they become available.
Next update by: Monday, March 16, 2020, at 9:00 PM UTC; PublishedTime=2020-03-15T19:21:20.26Z

Status: The deployment continues to progress throughout the affected environment at an optimal rate, and we anticipate it will complete within the next three days.
Next update by: Wednesday, March 18, 2020, at 9:00 PM UTC; PublishedTime=2020-03-16T21:00:35.78Z

Final status: The deployment of the fix was completed on Tuesday, March 17, 2020. Users can now expect to see characters correctly displayed that were previously shown as '''.
Next update by: Wednesday, March 18, 2020, at 9:00 PM UTC; PublishedTime=2020-03-16T21:00:35.78Z

top

Information Systems and Computing
University of Pennsylvania
Comments & Questions


University of Pennsylvania Penn Computing University of Pennsylvania Information Systems & Computing (ISC)
Information Systems and Computing, University of Pennsylvania