Penn Computing
Computing Menu Computing A-Z
Computing Home Information Systems & Computing Penn

Microsoft's Incident Report for PennO365

Following is a data extract from the PennO365 administrative console of incident reports sent by Microsoft within the last 30 days. We pull out incidents that are relevant to the current PennO365 service offerings and we refresh this data every 15 minutes. The last extract was taken on May 18, 2021 21:30.

--------------------------------------------------------------------------------------------------
Title Users may see inaccurate presence information for other users in Microsoft Teams
ID TM256805
Service Microsoft Teams
Feature Teams Components
Status Service restored
Impact Users may have seen inaccurate presence information for other users in Microsoft Teams.
Severity Sev2
Start time 2021-05-17T21:10:00Z
End time 2021-05-17T22:57:00Z
Last update 2021-05-17T23:50:56.517Z
More info This issue impacted the Microsoft Teams desktop and web clients. Final status: We've completed correcting service configurations and confirmed via monitoring that service availability has recovered, mitigating impact. Scope of impact: Any user routed through the impacted infrastructure may have been affected. Start time: Monday, May 17, 2021, at 9:10 PM'UTC End time: Monday, May 17, 2021, at 10:57 PM'UTC Root cause: A portion of service database infrastructure that facilitates presence services for Microsoft Teams became misconfigured, resulting in inaccurate presence information. Next steps: - We're reviewing the affected database infrastructure to better understand how the misconfiguration was introduced, and to explore potential options to more quickly resolve and prevent similar impact in the future. This is the final update for the event.; PublishedTime=2021-05-17T23:40:17.01Z
Root cause A portion of service database infrastructure that facilitates presence services for Microsoft Teams became misconfigured, resulting in inaccurate presence information. Next steps: - We're reviewing the affected database infrastructure to better understand how the misconfiguration was introduced, and to explore potential options to more quickly resolve and prevent similar impact in the future. This is the final update for the event.; PublishedTime=2021-05-17T23:40:17.01Z

TM256805 Details

Status: We're investigating a potential issue and checking for impact to your organization. We'll provide an update within 30 minutes.

Status: We're reviewing service-side telemetry to pinpoint the cause of the issue. Next update by: Monday, May 17, 2021, at 11:30 PMUTC; PublishedTime=2021-05-17T22:18:03.05Z

Status: Our investigation determined that a portion of service database infrastructure that facilitates presence services for Microsoft Teams became misconfigured, resulting in inaccurate presence information. We're working to correct the service configuration to mitigate impact. Next update by: Tuesday, May 18, 2021, at 1:30 AMUTC; PublishedTime=2021-05-17T23:24:16.113Z

Final status: We've completed correcting service configurations and confirmed via monitoring that service availability has recovered, mitigating impact. Next update by: Tuesday, May 18, 2021, at 1:30 AMUTC; PublishedTime=2021-05-17T23:24:16.113Z

--------------------------------------------------------------------------------------------------
Title Some admins may have experienced delays or timeout errors when attempting to access the Microsoft 365 Admin Center
ID MO256732
Service Microsoft 365 suite
Feature Portal
Status Service restored
Impact Admins may have experienced delays or timeout errors when attempting to access the Microsoft 365 Admin Center.
Severity Sev2
Start time 2021-05-17T09:46:00Z
End time 2021-05-17T12:15:00Z
Last update 2021-05-17T13:46:14.183Z
More info
Root cause A section of networking infrastructure was processing an unexpectedly high number of requests, causing delays. Next steps: - We're continuing to analyze the affected networking infrastructure to identify the exact root cause of the issue. - We're reviewing our code for improved performance and potential automated recovery options to reduce or avoid similar impact in the future. This is the final update for the event.; PublishedTime=2021-05-17T13:46:14.183Z

MO256732 Details

Status: We're investigating network connectivity and relevant diagnostic data to understand the root cause and establish a fix. We'll provide an update within 30 minutes.

Status: We're continuing to analyze network connectivity and other relevant diagnostic data to isolate the source of the issue and develop a mitigation plan. Next update by: Monday, May 17, 2021, at 1:00 PM UTC; PublishedTime=2021-05-17T11:44:39.617Z

Status: We've identified that a section of networking infrastructure is attempting to process a higher than expected number of requests, causing delays. We're analyzing request traffic to determine why this is occurring. Next update by: Monday, May 17, 2021, at 1:30 PM UTC; PublishedTime=2021-05-17T12:12:26.497Z

Status: We've rerouted request traffic to alternate infrastructure while we perform targeted restarts on the affected devices. Once complete, we'll monitor the service to confirm recovery. Next update by: Monday, May 17, 2021, at 3:00 PM UTC; PublishedTime=2021-05-17T13:19:23.057Z

Final status: We've confirmed that rerouting the affected request traffic to alternate infrastructure has mitigated the issue. Next update by: Monday, May 17, 2021, at 3:00 PM UTC; PublishedTime=2021-05-17T13:19:23.057Z

--------------------------------------------------------------------------------------------------
Title Admins' attempts to access the Service Health Dashboard (SHD) may intermittently fail
ID MO256452
Service Microsoft 365 suite
Feature Portal
Status Service restored
Impact Admins' attempts to access the SHD may have intermittently failed.
Severity Sev2
Start time 2021-05-14T14:30:00Z
End time 2021-05-14T19:00:00Z
Last update 2021-05-14T21:49:41.847Z
More info
Root cause A portion of infrastructure that facilitates SHD access wasn't processing traffic as efficiently as expected, resulting in SHD access failures. Next steps: - We're reviewing our automated request rebalancing processes as well as determining why the impacted infrastructure was processing requests inefficiently to prevent similar events in the future. This is the final update for the event.; PublishedTime=2021-05-14T21:49:41.847Z

MO256452 Details

Status: We've investigated a problem with the SHD in which admins may have experienced intermittent failures when attempting to access it, and we've confirmed that service is now restored. We'll provide additional information in a closure summary within the 'History' section of the 'Service health' dashboard shortly.; PublishedTime=2021-05-14T21:27:59.683

Final status: Our investigation determined that a portion of infrastructure that facilitates SHD access was performing below acceptable thresholds, causing admins to experience intermittent failures when attempting to access the SHD. We routed impacted traffic to alternate infrastructure to resolve this issue and confirmed via service telemetry that mitigation has been achieved.

--------------------------------------------------------------------------------------------------
Title Users in Europe can't sign in to the Yammer classic web experience
ID YA256206
Service Yammer Enterprise
Feature Yammer Components
Status Service restored
Impact Users were unable to sign in to the Yammer classic web experience.
Severity Sev2
Start time 2021-05-13T13:00:00Z
End time 2021-05-13T20:22:00Z
Last update 2021-05-13T20:48:58.613Z
More info Users of the Yammer classic web experience that were already signed in would not have been affected. Additionally, Users of the modern Yammer experience and Yammer mobile apps were unaffected. Final status: We've identified an unexpected network routing misconfiguration which caused localized infrastructure authentication failures, leading to impact for users located in Europe. We've resolved the routing configuration issue and restarted the portion of infrastructure which processes user authentication requests, which has successfully mitigated the impact. Scope of impact: Any user in Europe starting a fresh login session may have experienced this impact. Start time: Thursday, May 13, 2021, at 1:00 PM UTC End time: Thursday, May 13, 2021, at 8:22 PM UTC Root cause: An unexpected network routing misconfiguration caused localized infrastructure authentication failures, leading to impact for users located in Europe. Next steps: - We're continuing our analysis of the historical telemetry and networking logs correlated with the affected infrastructure to better understand the cause of the routing misconfiguration and to help prevent this problem from happening again. This is the final update for the event.; PublishedTime=2021-05-13T20:45:48.357Z
Root cause An unexpected network routing misconfiguration caused localized infrastructure authentication failures, leading to impact for users located in Europe. Next steps: - We're continuing our analysis of the historical telemetry and networking logs correlated with the affected infrastructure to better understand the cause of the routing misconfiguration and to help prevent this problem from happening again. This is the final update for the event.; PublishedTime=2021-05-13T20:45:48.357Z

YA256206 Details

Status: We're investigating a potential issue and checking for impact to your organization. We'll provide an update within 30 minutes.; PublishedTime=2021-05-13T18:59:41.667

Status: We're reviewing service telemetry and networking logs to isolate the source of the issue and determine our mitigation strategy. Next update by: Thursday, May 13, 2021, at 8:30 PMUTC; PublishedTime=2021-05-13T19:06:29.823Z

Status: Our analysis of the service telemetry and networking logs is ongoing. In parallel, we're reviewing any recent service changes which may be leading to impact. Next update by: Thursday, May 13, 2021, at 10:30 PMUTC; PublishedTime=2021-05-13T19:28:07.41Z

Final status: We've identified an unexpected network routing misconfiguration which caused localized infrastructure authentication failures, leading to impact for users located in Europe. We've resolved the routing configuration issue and restarted the portion of infrastructure which processes user authentication requests, which has successfully mitigated the impact. Next update by: Thursday, May 13, 2021, at 10:30 PMUTC; PublishedTime=2021-05-13T19:28:07.41Z

--------------------------------------------------------------------------------------------------
Title Users on Windows 7/8 Service Pack 1 with TLS 1.0/1.1 can't connect to OneDrive for Business and receive error 0x8004de40
ID OD255960
Service OneDrive for Business
Feature OneDrive for Business
Status Investigation suspended
Impact Users are unable to connect to OneDrive for Business and receive error code 0x8004de40.
Severity Sev2
Start time 2021-05-12T19:41:00Z
End time 2021-05-14T23:20:37Z
Last update 2021-05-14T23:20:45.773Z
More info For additional information, including a mechanism to prevent the impact from persisting, review MC240160 for details related to the TLS 1.2 enforcement specific to OneDrive for Business connectivity. Final status: As this event is related to our ongoing efforts to enforce TLS 1.2 use and not a service issue, this communication will be closed. However, the Message Center communication MC240160 will remain available to review for details on how to mitigate this specific impact associated with our continuous service improvement. Scope of impact: The implementation of TLS 1.2 enforcement may affect OneDrive for Business connectivity for customers using TLS 1.0 or 1.1 with users on Windows 7 or 8 Service Pack 1. This is the final update for the event.; PublishedTime=2021-05-14T23:20:45.773Z
Root cause

OD255960 Details

Status: We've received reports from users that they're unable to connect to OneDrive for Business and are presented the error code 0x8004de40. Our review of the reports and our service telemetry has confirmed that this is a result of our enforcement of TLS 1.2, which we previously communicated through MC240160 and https://docs.microsoft. com/en-us/microsoft-365/compliance/prepare-tls-1.2-in-office-365'view=o365-worldwide. For this specific error, users that continue to use TLS 1.0 and 1.1 can work around the impact associated with our enforcement of TLS 1.2 as documented in our online documentation: https://docs.microsoft.com/en-us/sharepoint/troubleshoot/administration/au thentication-errors-windows7. Next update by: Saturday, May 15, 2021, at 12:00 AM UTC; PublishedTime=2021-05-12T19:45:54.993Z

Final status: As this event is related to our ongoing efforts to enforce TLS 1.2 use and not a service issue, this communication will be closed. However, the Message Center communication MC240160 will remain available to review for details on how to mitigate this specific impact associated with our continuous service improvement. Next update by: Saturday, May 15, 2021, at 12:00 AM UTC; PublishedTime=2021-05-12T19:45:54.993Z

--------------------------------------------------------------------------------------------------
Title Users on Windows 7/8 Service Pack 1 with TLS 1.0/1.1 can't connect to OneDrive for Business and receive error 0x8004de40
ID SP255959
Service SharePoint Online
Feature SharePoint Features
Status Investigation suspended
Impact Users are unable to connect to OneDrive for Business and receive error code 0x8004de40.
Severity Sev2
Start time 2021-05-12T19:40:58Z
End time 2021-05-14T23:20:02Z
Last update 2021-05-14T23:20:20.293Z
More info For additional information, including a mechanism to prevent the impact from persisting, review MC240160 for details related to the TLS 1.2 enforcement specific to OneDrive for Business connectivity. Final status: As this event is related to our ongoing efforts to enforce TLS 1.2 use and not a service issue, this communication will be closed. However, the Message Center communication MC240160 will remain available to review for details on how to mitigate this specific impact associated with our continuous service improvement. Scope of impact: The implementation of TLS 1.2 enforcement may affect OneDrive for Business connectivity for customers using TLS 1.0 or 1.1 with users on Windows 7 or 8 Service Pack 1. This is the final update for the event.; PublishedTime=2021-05-14T23:20:09.993Z
Root cause

SP255959 Details

Status: We've received reports from users that they're unable to connect to OneDrive for Business and are presented the error code 0x8004de40. Our review of the reports and our service telemetry has confirmed that this is a result of our enforcement of TLS 1.2, which we previously communicated through MC240160 and https://docs.microsoft. com/en-us/microsoft-365/compliance/prepare-tls-1.2-in-office-365'view=o365-worldwide. For this specific error, users that continue to use TLS 1.0 and 1.1 can work around the impact associated with our enforcement of TLS 1.2 as documented in our online documentation: https://docs.microsoft.com/en-us/sharepoint/troubleshoot/administration/au thentication-errors-windows7. Next update by: Saturday, May 15, 2021, at 12:00 AM UTC; PublishedTime=2021-05-12T19:45:53.053Z

Final status: As this event is related to our ongoing efforts to enforce TLS 1.2 use and not a service issue, this communication will be closed. However, the Message Center communication MC240160 will remain available to review for details on how to mitigate this specific impact associated with our continuous service improvement. Next update by: Saturday, May 15, 2021, at 12:00 AM UTC; PublishedTime=2021-05-12T19:45:53.053Z

--------------------------------------------------------------------------------------------------
Title Issue affecting viewing email content in Outlook
ID EX255650
Service Exchange Online
Feature E-Mail and calendar access
Status Post-incident report published
Impact Users may have been unable to view email message content within Outlook.
Severity Sev2
Start time 2021-05-11T18:24:00Z
End time 2021-05-12T02:00:00Z
Last update 2021-05-18T21:18:21.717Z
More info Impact was specific to the Outlook client and users with access to other protocols, such as Outlook on the web or the Outlook mobile app, could view message content in those platforms as a potential workaround while our fix was applied. Final status: We've confirmed that the fix successfully resolved the issue, though some users will be required to restart their email client for the change to take effect. In some circumstances, a second restart may be necessary. Scope of impact: This issue could have affected any user attempting to view an email message in the Outlook client. Start time: Tuesday, May 11, 2021, at 6:24 PM UTC End time: Wednesday, May 12, 2021, at 2:00 AM UTC Root Cause: While mitigating a separate problem in Microsoft Word, a configuration rollout was deployed for specific build version 16.0.13929.xxxxx. However, due to a deployment error there was a misconfiguration within the parameters that determine the amount of content displayed via the Outlook client and Web Layout View used by Microsoft Word. Next steps: - For a more comprehensive list of next steps and actions, please refer to the Post Incident Review document.; PublishedTime=2021-05-12T04:02:30.09Z
Root cause A recent change to systems that facilitate text display management for content within the Outlook client caused impact. Next update by: Wednesday, May 12, 2021, at 4:00 AM UTC; PublishedTime=2021-05-12T03:04:24.523Z

EX255650 Details

Status: We're investigating a potential issue affecting users' ability to view email message in Outlook. We'll provide an update in 30 minutes.

Status: We're gathering and analyzing data in an effort to isolate the cause of impact. Initial reports indicate that impact is specific to the Outlook client and users with access to other protocols, such as Outlook on the web or the Outlook mobile app, can view message content in those platforms as a potential workaround. Next update by: Tuesday, May 11, 2021, at 9:30 PM UTC; PublishedTime=2021-05-11T20:28:15.037Z

Status: We're analyzing recent changes to the desktop client and are working to narrow down which versions of the Outlook desktop client may be affected. This is a quick update designed to give the latest information on this issue. For full details on this event, please reference the post made on May 11, 2021 at 8:28 PM UTC.; PublishedTime=2021-05-11T20:47:03.257Next update by: Tuesday, May 11, 2021, at 9:30 PM UTC; PublishedTime=2021-05-11T20:28:15.037Z

Status: We're continuing to analyze recent updates to the environment and the associated code as we work to isolate the cause of impact. Our efforts are focused on determining the most expedient means of remediating this issue. Next update by: Tuesday, May 11, 2021, at 10:30 PMUTC; PublishedTime=2021-05-11T21:19:31.25Z

Status: We've identified the likely source of impact and are testing a potential solution. If successful, we'll apply this mitigation to all affected environments to restore service. This is a quick update designed to give the latest information on this issue. For full details on this event, please reference the post made on May 11, 2021 at 9:19 PM UTC. ; PublishedTime=2021-05-11T21:47:53.367Next update by: Tuesday, May 11, 2021, at 10:30 PMUTC; PublishedTime=2021-05-11T21:19:31.25Z

Status: We've identified the underlying cause of impact and are applying a fix. This fix will reach all affected users incrementally over the course of the next four-to-five hours. Once users receive the fix, they will need to restart their email client to apply the fix. In some circumstances, users may need to restart their client a second time for the changes to take effect. We expect to complete this process and restore service for all affected users by May 12, 2021, at 3:00 AM UTC. We encourage affected users who are able to do so to leverage the workarounds described above in the "More info" section of this post while we complete the process of fixing this problem. Next update by: Wednesday, May 12, 2021, at 3:00 AM UTC; PublishedTime=2021-05-11T22:24:52.067Z

Status: We've confirmed that the fix has been applied to all affected environments. Users who continue to experience impact are encouraged to restart their email client. In some circumstances, users may need to restart their client a second time for the changes to take effect. We'll continue to monitor service health to ensure the issue is resolved. Next update by: Wednesday, May 12, 2021, at 4:00 AM UTC; PublishedTime=2021-05-12T03:04:24.523Z

Final status: We've confirmed that the fix successfully resolved the issue, though some users will be required to restart their email client for the change to take effect. In some circumstances, a second restart may be necessary. Next update by: Wednesday, May 12, 2021, at 4:00 AM UTC; PublishedTime=2021-05-12T03:04:24.523Z

--------------------------------------------------------------------------------------------------
Title Users may notice a delay in updates for profiles with PowerSchool or OneRoster data sources
ID OE256213
Service School Data Sync
Feature School Data Sync
Status Service restored
Impact Users may have noticed a delay in updates for profiles with PowerSchool or OneRoster data sources.
Severity Sev2
Start time 2021-05-11T17:48:00Z
End time 2021-05-18T22:30:00Z
Last update 2021-05-18T23:50:49.42Z
More info
Root cause A recent configuration change within a portion of infrastructure that facilitates authentication inadvertently resulted in impact. Next steps: - To help prevent similar impact in the future, we're further reviewing why the configuration change caused impact and how we can improve our validation procedures to catch such issues before implementation. This is the final update for the event.; PublishedTime=2021-05-18T23:50:49.42Z

OE256213 Details

Status: We've identified that a recent configuration change within a portion of infrastructure that facilitates authentication inadvertently resulted in impact. We've corrected and updated the impacting change and confirmed that requests are processing as expected, however due to the delays, a backlog has formed. We expect all updates to affected profiles be processed by 9:00 AM UTC on Friday, May 14, 2021. Next update by: Friday, May 14, 2021, at 8:00 PMUTC; PublishedTime=2021-05-13T19:28:26.7Z

Status: We're continuing to monitor the backlog of requests as they process, and anticipate impact will be fully mitigated by our next scheduled update. Next update by: Monday, May 17, 2021, at 9:30 PMUTC; PublishedTime=2021-05-14T18:44:58.677Z

Status: Monitoring the backlog of requests is in its final stages and we anticipate impact will be fully mitigated by our next scheduled update. Next update by: Tuesday, May 18, 2021, at 7:30 PMUTC; PublishedTime=2021-05-17T20:43:58.027Z

Status: The backlog of requests is taking longer to process than previously estimated, but we've confirmed the requests are processing successfully. We now anticipate resolution by our next update at 1:30 AM UTC on Wednesday, May 19, 2021. Next update by: Wednesday, May 19, 2021, at 1:30 AMUTC; PublishedTime=2021-05-18T18:05:46.483Z

Final status: We've finished processing the backlog of requests and confirmed via telemetry that this action, in conjunction with our previous fix for the impacting change, has mitigated the impact. Next update by: Wednesday, May 19, 2021, at 1:30 AMUTC; PublishedTime=2021-05-18T18:05:46.483Z

--------------------------------------------------------------------------------------------------
Title Users leveraging Microsoft Teams Meeting Room devices can't join meetings, make calls, or sign in to Microsoft Teams
ID TM256447
Service Microsoft Teams
Feature Teams Components
Status Service degradation
Impact Users leveraging Microsoft Teams Meeting Room devices can't join meetings, make calls, or sign in to Microsoft Teams.
Severity Sev2
Start time 2021-05-11T15:00:00Z
End time
Last update 2021-05-17T20:41:20.44Z
More info Microsoft Teams Meeting Room devices leveraging OS version 18363.1556 may be affected. We recommend restarting affected devices to mitigate the problem. More information about the affecting OS build can be found here: May 11, 2021'KB5003169 (OS Build 18363.1556) (microsoft.com) Current status: We've confirmed that affected users can restart the devices manually, through PowerShell, or the Microsoft Teams Admin center, to remediate impact. We're analyzing a group of affected users along with Unified Logging Service (ULS) data to formulate a mitigation strategy. Scope of impact: Your organization is affected by this event, and any Microsoft Teams Meeting Room device updated to version 18363.1556 may be affected. Start time: Tuesday, May 11, 2021, at 3:00 PM UTC Root cause: OS update 18363.1556 is inadvertently preventing some Microsoft Teams Meeting Room devices from accessing the Microsoft Teams service, which prevents users from joining meetings, making calls, or signing in to the Microsoft Teams service from affected devices. Next update by: Wednesday, May 19, 2021, at 7:00 PM UTC; PublishedTime=2021-05-17T20:41:20.44Z
Root cause OS update 18363.1556 is inadvertently preventing some Microsoft Teams Meeting Room devices from accessing the Microsoft Teams service, which prevents users from joining meetings, making calls, or signing in to the Microsoft Teams service from affected devices. Next update by: Wednesday, May 19, 2021, at 7:00 PM UTC; PublishedTime=2021-05-17T20:41:20.44Z

TM256447 Details

Status: We've determined that a recent OS update (18363.1556) is inadvertently preventing some Microsoft Teams Meeting Room devices from accessing the Microsoft Teams service, and resulting in various impact scenarios. Based off our analysis, we suspect that rebooting the affected devices manually, through PowerShell, or the Microsoft Teams Admin center, mitigates the problem. We're reviewing available logs and testing internally to determine if additional remediation steps are required. Next update by: Monday, May 17, 2021, at 10:00 PM UTC; PublishedTime=2021-05-14T20:23:46.07Z

Status: We've confirmed that affected users can restart the devices manually, through PowerShell, or the Microsoft Teams Admin center, to remediate impact. We're analyzing a group of affected users along with Unified Logging Service (ULS) data to formulate a mitigation strategy. Next update by: Wednesday, May 19, 2021, at 7:00 PM UTC; PublishedTime=2021-05-17T20:41:20.44Z

--------------------------------------------------------------------------------------------------
Title Alert emails from Identity protection may be delayed
ID IS255520
Service Identity Service
Feature Administration
Status Service restored
Impact Users may notice delays receiving alerts emails from Identity protection.
Severity Sev2
Start time 2021-05-10T16:00:00Z
End time 2021-05-11T01:05:00Z
Last update 2021-05-11T01:17:35.453Z
More info
Root cause A portion of infrastructure facilitating Identity protection alert emails was unexpectedly performing below the expected service threshold. Next steps: -We're continuing to analyze the source of the initial impact to better understand what caused the issue and to help develop ways to prevent similar issues in the future. This is the final update for the event.; PublishedTime=2021-05-11T01:17:35.453Z

IS255520 Details

Status: We're reviewing system logs to help identify the source of the delays. Next update by: Tuesday, May 11, 2021, at 2:30 AMUTC; PublishedTime=2021-05-11T00:40:37.907Z

Final status: After further review, we identified that a portion of infrastructure facilitating alert emails was performing below the expected service threshold. We restarted portions of the affected infrastructure and monitored the environment to ensure that service was restored. Next update by: Tuesday, May 11, 2021, at 2:30 AMUTC; PublishedTime=2021-05-11T00:40:37.907Z

--------------------------------------------------------------------------------------------------
Title Users notice that some icon links in Outlook on the web for mail, calendar, To Do, and other services aren't working
ID EX256425
Service Exchange Online
Feature E-Mail and calendar access
Status Service restored
Impact Users may have noticed that some icon links in Outlook on the web for mail, calendar and other services weren't working.
Severity Sev2
Start time 2021-05-10T15:00:00Z
End time 2021-05-16T20:20:00Z
Last update 2021-05-18T00:28:10.153Z
More info When clicking on the navigation icon links on the bottom portion of the browser window for other services, no action occurred. Users may have noticed that the first attempt worked, but subsequent navigation between services didn't work. Users could've accessed the services using the navigation icons in the App Launcher on the upper-left portion of the page. Other options included using the Outlook desktop client or Outlook mobile app to access the services. Final status: The deployment of the additional targeted code fix has completed, and we've confirmed with some impacted users that the issue is resolved. Scope of impact: This issue impacted any Outlook on the web users who were served through the portions of infrastructure which have received the updated configuration. Start time: Monday, May 10, 2021, at 3:00 PM UTC End time: Sunday, May 16, 2021, at 8:20 PM UTC Root cause: A recent configuration change corresponding with the Outlook on the web UI incidentally impacted the functionality of the navigation icon links at the bottom of the page, leading to impact. Next steps: - We're reviewing the recent configuration change to better understand what caused the functionality impact, and to assess how the problem wasn't identified during our update testing procedures. This is the final update for the event.; PublishedTime=2021-05-18T00:28:10.153Z
Root cause A recent configuration change corresponding with the Outlook on the web UI incidentally impacted the functionality of the navigation icon links at the bottom of the page, leading to impact. Next steps: - We're reviewing the recent configuration change to better understand what caused the functionality impact, and to assess how the problem wasn't identified during our update testing procedures. This is the final update for the event.; PublishedTime=2021-05-18T00:28:10.153Z

EX256425 Details

Status: We're investigating a potential issue and checking for impact to your organization. We'll provide an update within 30 minutes.

Status: We're reviewing whether any recent UI changes or updates may have impacted the functionality of the navigation icons links at the bottom of the page. Next update by: Friday, May 14, 2021, at 8:30 PMUTC; PublishedTime=2021-05-14T19:04:56.04Z

Status: We've identified a recent configuration change corresponding with the Outlook on the web UI has incidentally impacted the functionality of the navigation icon links at the bottom of the page, leading to impact. We're developing a targeted fix to repair the issue, which we expect will begin deployment prior to the next scheduled update, at which point we'll provide an estimated time to mitigation. Next update by: Saturday, May 15, 2021, at 2:30 AM UTC; PublishedTime=2021-05-14T20:25:57.907Z

Status: We've completed development of our targeted fix to repair the issue and have initiated deployment. We expect the fix will be fully deployed by our next scheduled update. Next update by: Saturday, May 15, 2021, at 7:00 PM UTC; PublishedTime=2021-05-15T00:56:27.55Z

Status: The deployment of the fix is ongoing, however it's progressing more slowly than previously expected. We'll provide an updated completion time by the next scheduled update. Next update by: Saturday, May 15, 2021, at 9:00 PM UTC; PublishedTime=2021-05-15T18:11:37.557Z

Status: The initial code fix has completed saturating all affected environments as of Saturday, May 15, 2021, at 6:17 PM UTC, and we've validated with some affected users that this has repaired the majority of impacted scenarios. However, some users may continue to experience navigation issues via the icon links when attempting to navigate from the "People" module. We're applying an additional targeted code fix to repair this functionality as well, thereby fully mitigating the impact, which we expect will complete by the next scheduled update. Next update by: Tuesday, May 18, 2021, at 1:30 AM UTC; PublishedTime=2021-05-15T20:01:53.94Z

Final status: The deployment of the additional targeted code fix has completed, and we've confirmed with some impacted users that the issue is resolved. Next update by: Tuesday, May 18, 2021, at 1:30 AM UTC; PublishedTime=2021-05-15T20:01:53.94Z

--------------------------------------------------------------------------------------------------
Title Users may experience poor audio and video quality during meetings when using the Microsoft Teams service
ID TM255448
Service Microsoft Teams
Feature Teams Components
Status Service restored
Impact Users may have experienced poor audio and video quality during meetings when using the Microsoft Teams service.
Severity Sev2
Start time 2021-05-10T11:40:00Z
End time 2021-05-10T17:24:00Z
Last update 2021-05-10T19:14:35.49Z
More info Direct calling was unaffected by this outage. Final status: We've completed our rerouting around the impacted infrastructure and confirmed mitigation via telemetry. Scope of impact: Impact was specific to a subset of users hosted in North America. Start time: Monday, May 10, 2021, at 11:40 AM'UTC End time: Monday, May 10, 2021, at 5:24 PM'UTC Root cause: A specific subset of infrastructure that facilitates audio and video with Microsoft Teams meetings began operating in a sub-optimal state, resulting in impact. Next steps: - We're reviewing our code for improved call quality performance and potential automated rerouting processes to reduce or avoid similar impact in the future. This is the final update for the event.; PublishedTime=2021-05-10T19:14:35.49Z
Root cause A specific subset of infrastructure that facilitates audio and video with Microsoft Teams meetings began operating in a sub-optimal state, resulting in impact. Next steps: - We're reviewing our code for improved call quality performance and potential automated rerouting processes to reduce or avoid similar impact in the future. This is the final update for the event.; PublishedTime=2021-05-10T19:14:35.49Z

TM255448 Details

Status: We're gathering diagnostic data for analysis. Next update by: Monday, May 10, 2021, at 4:30 PM UTC; PublishedTime=2021-05-10T14:17:04.013Z

Status: We've determined that a specific subset of infrastructure that facilitates audio and video with Microsoft Teams meetings began operating in a sub-optimal state, resulting in impact. We've removed the affected subset of infrastructure to re-route the affected traffic through a healthy infrastructure and confirmed that some users are experiencing impact mitigation. We're continuing to monitor the affected environment as the service continues to recover to ensure that the issue is resolved for all affected users. Next update by: Monday, May 10, 2021, at 8:30 PMUTC; PublishedTime=2021-05-10T15:38:14.033Z

Final status: We've completed our rerouting around the impacted infrastructure and confirmed mitigation via telemetry. Next update by: Monday, May 10, 2021, at 8:30 PMUTC; PublishedTime=2021-05-10T15:38:14.033Z

--------------------------------------------------------------------------------------------------
Title Users in Brazil can't start call recording in Microsoft Teams
ID TM255452
Service Microsoft Teams
Feature Teams Components
Status Service restored
Impact Users may have experienced an error message when attempting to initiate call recording within Microsoft Teams.
Severity Sev2
Start time 2021-05-10T10:10:00Z
End time 2021-05-10T14:45:00Z
Last update 2021-05-10T16:53:41.6Z
More info
Root cause A portion of infrastructure that facilitates call recording in the Brazil region wasn't processing requests as efficiently as expected, resulting in impact. Next steps: - We're reviewing our code for improved call recording performance and potential automated rebalancing actions to reduce or avoid similar impact in the future. This is the final update for the event.; PublishedTime=2021-05-10T16:51:34.09Z

TM255452 Details

Status: We're investigating a potential issue and checking for impact to your organization. We'll provide an update within 30 minutes.; PublishedTime=2021-05-10T14:54:15.23

Status: We've identified that a portion of infrastructure that facilitates call recording in the Brazil region isn't processing requests as efficiently as expected, resulting in impact. We're rebalancing load in the region to mitigate impact. Next update by: Monday, May 10, 2021, at 5:00 PMUTC; PublishedTime=2021-05-10T15:16:30.657Z

Final status: We've completed our rebalancing efforts and monitored the environments to confirm mitigation via telemetry. Next update by: Monday, May 10, 2021, at 5:00 PMUTC; PublishedTime=2021-05-10T15:16:30.657Z

--------------------------------------------------------------------------------------------------
Title Some users may experience Office app crashes upon app launch or opening files on iOS devices
ID OP255548
Service Microsoft 365 Apps
Feature Office Client issues
Status Service restored
Impact Users may have experienced Office app crashes upon app launch or when opening files on iOS devices.
Severity Sev2
Start time 2021-05-10T04:00:00Z
End time 2021-05-11T19:00:00Z
Last update 2021-05-11T19:52:59.28Z
More info Impact was limited to some users that had release 2.49 of iOS Office apps including Office Mobile, Word, Excel, and PowerPoint. While we were focused on remediation, users may have followed these steps to mitigate impact: -Go to "System Settings" -Select the Office app that was experiencing the issue -Click on "Reset [name of selected app]" -Enable "Clear all [documents, workbooks, presentations]". Final status: We've deployed version 2.49.1 of the apps to the iOS App Store to remediate impact. Affected users without automatic updates can download the app update from the iOS App Store to remediate impact. Scope of impact: Impact was specific to iOS Office apps that attempted to open documents that were still open during the last session. The apps were continuing to do this, causing a crash loop. Start time: Monday, May 10, 2021, at 4:00 AM UTC End time: Tuesday, May 11, 2021, at 7:00 PM UTC Root cause: A recent feature update contained a code regression, which may have caused Office app crashes on iOS devices upon app launch or when opening documents. Next steps: - We're reviewing our update procedures and the associated code regression to better identify similar issues during our development and testing cycles. This is the final update for the event.; PublishedTime=2021-05-11T19:49:49.95Z
Root cause A recent feature update contained a code regression, which may have caused Office app crashes on iOS devices upon app launch or when opening documents. Next steps: - We're reviewing our update procedures and the associated code regression to better identify similar issues during our development and testing cycles. This is the final update for the event.; PublishedTime=2021-05-11T19:49:49.95Z

OP255548 Details

Status: We've identified that a recent feature update containing a code regression was deployed resulting in impact. We have reverted the update, which should mitigate impact for the majority of users. While we are in the process of releasing version 2.49.1 of the apps to the iOS App Store to fully remediate impact, users can manually reset their Office app preferences to resolve the issue. Next update by: Tuesday, May 11, 2021, at 6:00 PM UTC; PublishedTime=2021-05-11T02:25:23.417Z

Status: We're continuing our efforts to expedite the release of version 2.49.1 of the apps to the iOS App Store for full remediation of the issue. Additionally, users can manually reset their Office app preferences as outlined in the More info section to mitigate impact. Next update by: Tuesday, May 11, 2021, at 9:00 PM UTC; PublishedTime=2021-05-11T16:33:19.657Z

Final status: We've deployed version 2.49.1 of the apps to the iOS App Store to remediate impact. Affected users without automatic updates can download the app update from the iOS App Store to remediate impact. Next update by: Tuesday, May 11, 2021, at 9:00 PM UTC; PublishedTime=2021-05-11T16:33:19.657Z

--------------------------------------------------------------------------------------------------
Title Admins and users may see multiple issues related to email and Microsoft Defender products.
ID MO255463
Service Microsoft 365 suite
Feature Portal
Status Service restored
Impact Admins and users may have seen multiple impacts related to email and Microsoft Defender products.
Severity Sev2
Start time 2021-05-10T00:00:00Z
End time 2021-05-18T08:30:00Z
Last update 2021-05-18T17:19:16.647Z
More info Users and admins may have experienced multiple issues from this incident: - Users may have been unable to receive emails that contained URLs from legitimate domains. - Users may have noticed legitimate messages were getting quarantined or sent to junk. - Users were unable to access links within emails that were incorrectly identified to be risky. - Users may have become blocked from sending emails if their messages were incorrectly detected as spam or phish. - Admins may have experienced delays with Automated Investigations in Microsoft Defender for Office 365 (MDO) before the service was disabled to alleviate impact. - Admins would have seen alert reporting latency within the Microsoft Defender 365 portal before the service was disabled to alleviate impact. This was due to the alert ingestion service working through an unexpectedly large queue of erroneous alerts coming from the incorrectly blocked URLs. - After all services were re-enabled, admins may have seen stale false positive alerts within Microsoft Defender 365 portal. Microsoft provided instructions to manually remove these alerts from the portal in the 'More info' section of service incident MO255463 on Thursday, May 13, 2021. Admins may have also experienced ancillary impact scenarios related to this event: - An error caused the Quarantine Log queue to process message state changes for emails released from quarantine slower than expected. As a result, admins may have seen previously released emails still listed as quarantined in the Microsoft Defender 365 portal quarantine UI. - On Thursday, May 13, 2021, we identified that a portion of emails that were still showing as quarantined were originating from a subset of URLs that were not properly unblocked and released during the initial event. Admins may have seen additional delays for these messages to be de-listed from quarantine status in the Microsoft Defender 365 portal quarantine UI. Final status: We've completed processing the change backlog and confirmed that newly released messages are no longer appearing as quarantined within the Microsoft Defender 365 portal quarantine UI. Our continued monitoring of the service environment has shown no further impact from this event. Scope of impact: Any user or admin could have experienced impact from this event. Start time: Monday, May 10, 2021, at 12:00 AM UTC End time: Tuesday, May 18, 2021, at 8:30 AM UTC Preliminary root cause: An incorrect change in one of the components of Exchange Online Protection's detection system caused multiple URL domains to be erroneously flagged as malicious, some of which were related to popular internet sites. As a result, inbound emails that contained URLs from these legitimate domains were inadvertently blocked in Exchange Online Protection (EOP), which caused emails from the affected domains to be quarantined and/or sent to junk and flagged as spam or phishing attempts. Next steps: - We're working to understand the factors that led to the incorrect change being deployed. We'll review the change management process and monitoring systems to identify gaps and repair items to prevent similar issues in future. - We're working to identify the factors that prevented the recently blocked URLs from being discovered. We'll publish a post-incident report within five business days.; PublishedTime=2021-05-18T17:19:16.647Z
Root cause A recent configuration change inadvertently caused emails that contained URLs from a number legitimate domains to be blocked in Exchange Online Protection (EOP), and emails from the affected domains to be quarantined or sent to junk and flagged as spam or phishing attempts. Next update by: Tuesday, May 18, 2021, at 6:00 PM UTC; PublishedTime=2021-05-17T21:39:32.433Z

MO255463 Details

Status: We've identified that an inadvertent block was placed on valid and healthy domains, resulting in the multiple impacts reported. We've ran commands to unblock the URLs, which will mitigate impact for all new email messages and links within email messages. We're working to release the email messages that were quarantined, as well as investigating options to restore the health of the Microsoft Defender portal. Next update by: Monday, May 10, 2021, at 6:30 PM UTC; PublishedTime=2021-05-10T17:00:36.783Z

Status: We've confirmed that newly sent emails and links within emails are working as expected. We've reprocessed emails for the most heavily impacted domains and we're monitoring email queues to ensure they deliver as expected. Once those emails email queues have drained, we'll continue to reprocess the emails on the remaining domains. Additionally, we've restored the latency issues with the Microsoft Defender portal. Next update by: Monday, May 10, 2021, at 8:30 PM UTC; PublishedTime=2021-05-10T18:28:06.793Z

Status: We've completed reprocessing emails that were incorrectly quarantined during the impact window. We've identified a number of users that remain unable to send new messages and are working to unblock them; though, we've confirmed that no new users will be incorrectly blocked. Furthermore, we're investigating the most expedient means of optimizing Threat Explorer ingestion and resolving the ingestion delays. Next update by: Monday, May 10, 2021, at 10:30 PM UTC; PublishedTime=2021-05-10T20:37:51.463Z

Status: We've successfully unblocked any known impacted user. We're currently testing a fix in our internal environment to confirm if it'll properly resolve the Threat Ingestion issue. Next update by: Tuesday, May 11, 2021, at 7:00 PM UTC; PublishedTime=2021-05-11T00:06:54.46Z

Status: We are working on cancelling investigations that resulted from the erroneous alerts and working on reducing latency of rest of the automated investigations and system. We expect our testing and deployment process to take an extended period of time. Next update by: Tuesday, May 11, 2021, at 7:00 PM UTC ; PublishedTime=2021-05-11T11:57:12.233Z

Status: We've finished cancelling the erroneous investigations within Microsoft Defender for Office 365. We anticipate that the remaining legitimate investigations will be processed with reduced latency, and we'll continue to monitor the service to ensure full recovery. We're also investigating reports that some emails may still show as quarantined. We're working to confirm the source of this issue; though, our initial findings indicate that this is a User Interface (UI) issue related to the MDO latency, and will recover in tandem. Next update by: Wednesday, May 12, 2021, at 1:00 AM UTC; PublishedTime=2021-05-11T19:07:04.53Z

Status: The remaining MDO investigations are largely completed and we're actively monitoring the service to ensure full recovery. We're continuing our investigation into reports of emails still showing as quarantined and expect that these messages will reflect accurately in the admin portal once the remainder of the data queue is processed. Next update by: Wednesday, May 12, 2021, at 5:00 AM UTC; PublishedTime=2021-05-11T21:34:23.72Z

Status: We've mitigated the issue causing delays for Automated Investigations and Threat Explorer. In parallel, Alert policies and investigations related to the false positive event and investigations will remain disabled for MDO while we fully resolve the issue. Next update by: Wednesday, May 12, 2021, at 11:00 PM UTC ; PublishedTime=2021-05-12T04:41:50.927Z

Status: All remaining data logging queues have cleared and admins should not be seeing any delays in Threat Explorer. Any emails previously blocked are now getting delivered and have been released from quarantine. Further, admins and users should no longer see any impact for this event; however, some admins may see active alerts related to the false positive events that fired during the impact window which flowed in to the Defender Suite. We're developing a list of steps to help admins clear these stale alerts from their environments, and expect to provide more information in tomorrow's update. Next update by: Thursday, May 13, 2021, at 11:00 PM UTC; PublishedTime=2021-05-12T18:53:40.263Z

Status: We've begun reenabling Microsoft Defender for Office 365 (MDO) Investigations for the previously affected alert policies, which we'll gradually bring to full functionality throughout the remainder of this week. We've also re-enabled alert ingestion for the Microsoft Defender 365 suite and will monitor the service for any additional queuing while MDO Investigations are reintroduced. Additionally, we've identified a quarantine log issue which may still cause some emails to be shown as quarantined despite being previously released. We're examining options to resolve this issue; though, this is a purely cosmetic problem that does not impact functionality. Next update by: Friday, May 14, 2021, at 11:00 PM UTC; PublishedTime=2021-05-13T22:37:07.73Z

Status: We're still working to re-enable MDO Investigations across the impacted infrastructure. This process is taking longer than expected and we now anticipate that it will finish by early next week. Additionally, we identified a batch of messages related to a subset of URL domains that were not correctly remediated during Monday's incident. We're releasing the affected messages, which will finish by the end of today. Furthermore, we've initiated deployment of a fix to ensure the quarantine UI shows the correct message state for messages that were previously released but still listed as quarantined. We expect this issue to be fully remediated by the end of this weekend. Next update by: Monday, May 17, 2021, at 11:00 PM UTC; PublishedTime=2021-05-14T20:35:53.87Z

Status: We've reenabled MDO investigations and alert ingestion for Microsoft Defender 365, and confirmed from active monitoring that all prior impacted services are now working as expected. Additionally, we've completed deployment of the fix and confirmed that all previously released messages are no longer listed as quarantined. However, released emails from the additional URLs that were erroneously blocked are still being processed and may be listed under the incorrect state in the Quarantine UI. We're processing the backlog of these changes to ensure those messages no longer appear as quarantined and we expect to finish this action by the end of today. Next update by: Tuesday, May 18, 2021, at 6:00 PM UTC; PublishedTime=2021-05-17T21:39:32.433Z

Final status: We've completed processing the change backlog and confirmed that newly released messages are no longer appearing as quarantined within the Microsoft Defender 365 portal quarantine UI. Our continued monitoring of the service environment has shown no further impact from this event. Next update by: Tuesday, May 18, 2021, at 6:00 PM UTC; PublishedTime=2021-05-17T21:39:32.433Z

--------------------------------------------------------------------------------------------------
Title Users unable to use the Dynamic View feature of the Microsoft Teams desktop app during meetings
ID TM255111
Service Microsoft Teams
Feature Teams Components
Status Service degradation
Impact Users are unable to use the Dynamic View feature of the Microsoft Teams desktop app during meetings.
Severity Sev2
Start time 2021-05-08T00:00:00Z
End time
Last update 2021-05-18T19:47:41.913Z
More info The Dynamic View feature was enabled on Monday, May 3, 2021. The feature was temporarily disabled as of Saturday, May 8, 2021, at 12:00 AM UTC. Current status: We've completed validation and have started deploying our fix to correct the latent issues associated with the Dynamic View feature. The fix should be fully deployed by Thursday, May 20, 2021. Once the fix has been fully deployed, we'll begin progressively re-enabling the Dynamic View feature. We'll provide a more precise timeline for when the feature will be fully available for use by our next scheduled update, though we anticipate that it'll be fully available during the week of May 28, 2021. Scope of impact: All users will notice that the Dynamic View feature is no longer available. Start time: Saturday, May 8, 2021, at 12:00 AM UTC Root cause: Dynamic View feature of the Microsoft Teams desktop app introduced latent issues that run the risk of causing adverse service impact. We've temporarily disabled the Dynamic View feature while we address the issues. Next update by: Monday, May 24, 2021, at 9:00 PM UTC; PublishedTime=2021-05-18T19:47:41.913Z
Root cause Dynamic View feature of the Microsoft Teams desktop app introduced latent issues that run the risk of causing adverse service impact. We've temporarily disabled the Dynamic View feature while we address the issues. Next update by: Monday, May 24, 2021, at 9:00 PM UTC; PublishedTime=2021-05-18T19:47:41.913Z

TM255111 Details

Status: We've temporarily disabled the Dynamic View feature of the Microsoft Teams desktop app so that we can address latent problems attributed to the feature. Once the latent problems have been addressed, Dynamic View will be re-enabled. We estimate that it will take up to a week to address the problems and re-enable the feature, and we're aiming to provide a more precise estimate by our next scheduled update. Next update by: Tuesday, May 11, 2021, at 8:40 PMUTC; PublishedTime=2021-05-08T01:38:01.313Z

Status: The Dynamic View feature of the Microsoft Teams desktop app exposed latent issues that run the risk of causing adverse service impact. We've temporarily disabled the Dynamic View feature while we address the issues. Next update by: Thursday, May 13, 2021, at 11:30 PMUTC; PublishedTime=2021-05-11T18:41:08.027Z

Status: We've developed and validated a fix which is scheduled to start deploying on Friday, May 14, 2021. We expect that the deployment will complete by our next scheduled update to resolve the issue. Next update by: Tuesday, May 18, 2021, at 9:00 PMUTC; PublishedTime=2021-05-13T22:32:43.507Z

Status: We've completed validation and have started deploying our fix to correct the latent issues associated with the Dynamic View feature. The fix should be fully deployed by Thursday, May 20, 2021. Once the fix has been fully deployed, we'll begin progressively re-enabling the Dynamic View feature. We'll provide a more precise timeline for when the feature will be fully available for use by our next scheduled update, though we anticipate that it'll be fully available during the week of May 28, 2021. Next update by: Monday, May 24, 2021, at 9:00 PM UTC; PublishedTime=2021-05-18T19:47:41.913Z

--------------------------------------------------------------------------------------------------
Title Users getting Non-Delivery Reports (NDRs) when sending messages via any Exchange Online connection method
ID EX254972
Service Exchange Online
Feature E-Mail timely delivery
Status Service restored
Impact Users received NDRs when sending messages from new domains recently delisted via https://sender.office.com/.
Severity Sev2
Start time 2021-05-07T02:00:00Z
End time 2021-05-08T11:30:00Z
Last update 2021-05-08T14:53:58.82Z
More info
Root cause A service update changed the method Exchange Online Protection serializes email messages, causing a contention with how they were being deserialized after being delisted, resulting in impact. Next steps: - We're reviewing why the impact was missed during our update testing and validation processes to prevent similar impact in the future. This is the final update for the event.; PublishedTime=2021-05-08T14:53:58.82Z

EX254972 Details

Status: We're investigating a potential issue and checking for impact to your organization. We'll provide an update within 30 minutes.

Status: We've determined that the IP address of the Proofpoint service is blocked within our service. We're in the process of unblocking the IP address to mitigate impact. Next update by: Friday, May 7, 2021, at 7:00 AMUTC; PublishedTime=2021-05-07T05:47:41.557Z

Status: Due to an unexpected issue, the process to unblock the affected IP addresses is taking longer than expected. We're continuing to run through the necessary steps. Next update by: Friday, May 7, 2021, at 9:00 AMUTC; PublishedTime=2021-05-07T06:38:31.33Z

Status: We're continuing to unblock the affected IP addresses and we're monitoring the fix until impact has been fully mitigated. Next update by: Friday, May 7, 2021, at 11:00 AM UTC; PublishedTime=2021-05-07T08:31:02.887Z

Status: We're experiencing problems unblocking the affected IP addresses due to a back-end data issue. We're addressing the underlying issue, and once fixed, we'll continue with unblocking the affected IP addresses, to mitigate impact. Next update by: Friday, May 7, 2021, at 2:00 PM UTC; PublishedTime=2021-05-07T10:56:46.817Z

Status: We're continuing to investigate and fix the underlying back-end data issue, which is preventing our automated bulk delisting processes. Whilst we address this, we're manually delisting addresses; however, this may take a longer period of time to complete. Next update by: Friday, May 7, 2021, at 5:00 PM UTC; PublishedTime=2021-05-07T14:00:25.683Z

Status: We've identified that a recent service update changed the method Exchange Online Protection serializes messages causing a contention with how they're deserialized after being delisted, resulting in impact. We're working to determine the fastest method to mitigate impact. Next update by: Friday, May 7, 2021, at 7:00 PMUTC; PublishedTime=2021-05-07T16:42:02.697Z

Status: We're developing a fix to resolve the serialization issue and looking into reverting the change that caused impact to determine which will mitigate impact more rapidly. We expect to have a timeline to mitigation by our next scheduled update. Next update by: Saturday, May 8, 2021, at 12:00 AMUTC; PublishedTime=2021-05-07T18:53:29.123Z

Status: The fix is still under development and will be deployed once validation is completed. We expect deployment to begin by 3:00 AM UTC on Saturday, May 8, 2021. Next update by: Saturday, May 8, 2021, at 5:00 AMUTC; PublishedTime=2021-05-07T22:18:14.63Z

Status: We've validated the fix and have started deploying it to the affected environments. We expect the deployment to be complete and the issue fully mitigated by our next communications update. Next update by: Saturday, May 8, 2021, at 5:00 PMUTC; PublishedTime=2021-05-08T03:42:49.427Z

Final status: The deployment of the fix has completed and we've performed tests to confirm that it successfully resolved the issue. Next update by: Saturday, May 8, 2021, at 5:00 PMUTC; PublishedTime=2021-05-08T03:42:49.427Z

--------------------------------------------------------------------------------------------------
Title Some users are unable to reply to email via Outlook on the web
ID EX255041
Service Exchange Online
Feature E-Mail timely delivery
Status Service restored
Impact Users were unable to reply to email via Outlook on the web.
Severity Sev2
Start time 2021-05-06T17:00:00Z
End time 2021-05-10T15:00:00Z
Last update 2021-05-11T19:44:10.113Z
More info Users attempting to send a reply may have received an error message stating "This message can't be sent right now. Please try again later. ChangeKey is required for this operation." As a workaround, users were able to utilize the Outlook desktop client or Outlook mobile to send replies. Additionally, if users refreshed the browser page they may have experienced temporary relief. Final status: We've confirmed that reverting the change has mitigated impact with impacted users. Scope of impact: This issue affected some users attempting to reply to email via Outlook on the web. Start time: Thursday, May 6, 2021, at 5:00 PM'UTC End time: Monday, May 10, 2021, at 3:00 PM'UTC Root cause: A recent service update changed the way ChangeKeys are generated and introduced a contention with Outlook on the web, resulting in impact. Next steps: - We're reviewing our ChangeKey update procedures to better identify similar issues and prevent similar impact in future updates. This is the final update for the event.; PublishedTime=2021-05-11T19:42:39.373Z
Root cause A recent service update changed the way ChangeKeys are generated and introduced a contention with Outlook on the web, resulting in impact. Next steps: - We're reviewing our ChangeKey update procedures to better identify similar issues and prevent similar impact in future updates. This is the final update for the event.; PublishedTime=2021-05-11T19:42:39.373Z

EX255041 Details

Status: We're investigating a potential issue and checking for impact to your organization. We'll provide an update within 30 minutes.

Status: We're reviewing support case details, including screenshots of impact, Fiddler trace logs and Problem Steps Recorder (PSR) logs as we formulate our next troubleshooting steps. Next update by: Friday, May 7, 2021, at 7:00 PMUTC; PublishedTime=2021-05-07T17:42:19.65Z

Status: We've identified 404 "Not Found" errors in our Fiddler trace log review, and are investigating the source of these errors to identify a potential root cause. Next update by: Friday, May 7, 2021, at 9:00 PMUTC; PublishedTime=2021-05-07T18:46:34.247Z

Status: We're working to gather additional logging information from an underlying infrastructure supporting the Outlook on the web service to identify any potential performance issues or additional errors which may be contributing to impact. Next update by: Friday, May 7, 2021, at 11:00 PMUTC; PublishedTime=2021-05-07T20:52:06.46Z

Status: Our log review has been inconclusive thus far. We're reviewing change logs to determine if a recent change may have resulted in impact, while in parallel, we're working to deduce steps to reproduce impact to assist in further debugging the issue. Next update by: Saturday, May 8, 2021, at 1:00 AMUTC; PublishedTime=2021-05-07T22:55:50.873Z

Status: We've confirmed that incoming email is expected to contain a property value referred to as ChangeKey, which the replying email references when responding. Our log review determined that incoming email does include this ChangeKey, but the response messages are not populating with the identical ChangeKey value as expected, potentially due to an issue occurring within a specific component which populates the ChangeKey value. We're continuing our debugging and log review to better understand this operation and confirm any potential issues which may be affecting this component and contributing to impact. Next update by: Monday, May 10, 2021, at 11:00 PMUTC; PublishedTime=2021-05-08T00:32:01.86Z

Status: We've confirmed that a recent service update contained a change in ChangeKey generation resulting in a contention with Outlook on the web, causing impact. We've disabled this change across all impacted infrastructure and are reaching out to impacted users to confirm this has mitigated impact. Next update by: Tuesday, May 11, 2021, at 11:00 PMUTC; PublishedTime=2021-05-10T21:32:26.227Z

Final status: We've confirmed that reverting the change has mitigated impact with impacted users. Next update by: Tuesday, May 11, 2021, at 11:00 PMUTC; PublishedTime=2021-05-10T21:32:26.227Z

--------------------------------------------------------------------------------------------------
Title Users may be unable to create or edit flows within Power Automate in Microsoft 365
ID MF254212
Service Microsoft Power Automate in Microsoft 365
Feature Service and web access issues
Status Post-incident report published
Impact Users may have been unable to create or edit flows within Power Automate in Microsoft 365.
Severity Sev2
Start time 2021-05-04T09:27:00Z
End time 2021-05-04T15:45:00Z
Last update 2021-05-11T13:02:08.127Z
More info
Root cause A recent change to introduce a feature intended to increase application launch reliability by pre-calculating dependency information, led to an issue with auto-scaling systems. Previously within Power Apps, auto-scaling systems have been triggered in accordance to Central Processing Unit (CPU) utilization. The new feature resulted in an increased memory footprint and the auto-scale rules were not configured to detect the new resource pattern. Usually, auto-scaling systems increase the number of available processing components based on load spikes. As memory usage increased on processing components, Redis cache components started to perform below expected thresholds. The automated recovery system (Circuit breaker) detected the issue with Redis and activated, causing requests to use alternate code paths to gather dependency information. This further compounded service load on already impacted processing components. Additionally, an error within a recent monitoring update prevented the specific monitor from performing as expected and not show memory resource exhaustion issues. This lead to prolonged impact as we needed to troubleshoot the issue and find the error conditions through alternate sources. Next steps: - For a more comprehensive list of next steps and actions, please refer to the Post Incident Review document. We'll publish a post-incident report within five business days.; PublishedTime=2021-05-04T16:46:45.997Z

MF254212 Details

Status: We've identified a portion of the infrastructure responsible for creating or editing flows is performing below acceptable thresholds. We're gathering additional telemetry to isolate the root cause and determine the next troubleshooting steps. Next update by: Tuesday, May 4, 2021, at 3:30 PM UTC; PublishedTime=2021-05-04T13:13:59.953Z

Status: We've identified timeout errors for requests going to a section of infrastructure responsible for caching. Whilst we continue to investigate, we're performing targeted restarts and rerouting traffic to alternate infrastructure in an attempt to provide relief. Next update by: Tuesday, May 4, 2021, at 3:30 PM UTC; PublishedTime=2021-05-04T14:30:57.023Z

Status: We've completed targeted restarts and re-balanced traffic, however, there has not been any significant improvement. We're performing optimizations on the caching infrastructure as a potential mitigation. In parallel, we're reviewing options to increase the connection timeout thresholds to improve performance. Next update by: Tuesday, May 4, 2021, at 5:00 PM UTC; PublishedTime=2021-05-04T15:20:59.343Z

Status: We're seeing improvements and are continuing to monitor the environment.; PublishedTime=2021-05-04T16:13:02.187Next update by: Tuesday, May 4, 2021, at 5:00 PM UTC; PublishedTime=2021-05-04T15:20:59.343Z

Final status: We've performed optimizations on both caching and service processing infrastructure, and we restarted role components to flush the affected caches. After a period of monitoring, telemetry indicated the service recovered, and we've confirmed that impact has been remediated. Next update by: Tuesday, May 4, 2021, at 5:00 PM UTC; PublishedTime=2021-05-04T15:20:59.343Z

--------------------------------------------------------------------------------------------------
Title Users may be unable to create or edit flows within Power Automate in Microsoft 365
ID MF254212
Service Microsoft Power Automate in Microsoft 365
Feature Service and web access issues
Status Post-incident report published
Impact Users may have been unable to create or edit flows within Power Automate in Microsoft 365.
Severity Sev2
Start time 2021-05-04T09:27:00Z
End time 2021-05-04T15:45:00Z
Last update 2021-05-11T13:02:08.127Z
More info
Root cause A recent change to introduce a feature intended to increase application launch reliability by pre-calculating dependency information, led to an issue with auto-scaling systems. Previously within Power Apps, auto-scaling systems have been triggered in accordance to Central Processing Unit (CPU) utilization. The new feature resulted in an increased memory footprint and the auto-scale rules were not configured to detect the new resource pattern. Usually, auto-scaling systems increase the number of available processing components based on load spikes. As memory usage increased on processing components, Redis cache components started to perform below expected thresholds. The automated recovery system (Circuit breaker) detected the issue with Redis and activated, causing requests to use alternate code paths to gather dependency information. This further compounded service load on already impacted processing components. Additionally, an error within a recent monitoring update prevented the specific monitor from performing as expected and not show memory resource exhaustion issues. This lead to prolonged impact as we needed to troubleshoot the issue and find the error conditions through alternate sources. Next steps: - For a more comprehensive list of next steps and actions, please refer to the Post Incident Review document. We'll publish a post-incident report within five business days.; PublishedTime=2021-05-04T16:46:45.997Z

MF254212 Details

Status: We've identified a portion of the infrastructure responsible for creating or editing flows is performing below acceptable thresholds. We're gathering additional telemetry to isolate the root cause and determine the next troubleshooting steps. Next update by: Tuesday, May 4, 2021, at 3:30 PM UTC; PublishedTime=2021-05-04T13:13:59.953Z

Status: We've identified timeout errors for requests going to a section of infrastructure responsible for caching. Whilst we continue to investigate, we're performing targeted restarts and rerouting traffic to alternate infrastructure in an attempt to provide relief. Next update by: Tuesday, May 4, 2021, at 3:30 PM UTC; PublishedTime=2021-05-04T14:30:57.023Z

Status: We've completed targeted restarts and re-balanced traffic, however, there has not been any significant improvement. We're performing optimizations on the caching infrastructure as a potential mitigation. In parallel, we're reviewing options to increase the connection timeout thresholds to improve performance. Next update by: Tuesday, May 4, 2021, at 5:00 PM UTC; PublishedTime=2021-05-04T15:20:59.343Z

Status: We're seeing improvements and are continuing to monitor the environment.; PublishedTime=2021-05-04T16:13:02.187Next update by: Tuesday, May 4, 2021, at 5:00 PM UTC; PublishedTime=2021-05-04T15:20:59.343Z

Final status: We've performed optimizations on both caching and service processing infrastructure, and we restarted role components to flush the affected caches. After a period of monitoring, telemetry indicated the service recovered, and we've confirmed that impact has been remediated. Next update by: Tuesday, May 4, 2021, at 5:00 PM UTC; PublishedTime=2021-05-04T15:20:59.343Z

--------------------------------------------------------------------------------------------------
Title Some users may not be receiving email notifications from Cloud App Security
ID CS254208
Service Cloud App Security
Feature Microsoft Cloud App Security
Status Service restored
Impact Users may have been unable to receive email notifications from Cloud App Security.
Severity Sev2
Start time 2021-04-30T11:08:00Z
End time 2021-05-04T15:45:00Z
Last update 2021-05-04T16:49:09.587Z
More info This issue impacted Microsoft Cloud App Security (MCAS) and Cloud App Security Discovery (CASD) functionality. Final status: Our analysis of the diagnostic data and system logs determined that an authentication component misconfiguration within the affected infrastructure was preventing users from receiving email notifications from Cloud App Security. We've updated this configuration and restarted the affected infrastructure. We've confirmed via our system monitoring services that the issue is resolved and users can receive email notification again. Scope of impact: Impact was specific to users who were served through the affected infrastructure. Start time: Friday, April 30, 2021, at 11:08 AM'UTC End time: Tuesday, May 4, 2021, at 3:45 PM'UTC Root cause: An authentication component misconfiguration within the affected infrastructure was preventing users from receiving email notifications from Cloud App Security. Next steps: - We're investigating how the authentication component became misconfigured so that we can prevent similar issues in the future. This is the final update for the event.; PublishedTime=2021-05-04T16:49:09.587Z
Root cause An authentication component misconfiguration within the affected infrastructure was preventing users from receiving email notifications from Cloud App Security. Next steps: - We're investigating how the authentication component became misconfigured so that we can prevent similar issues in the future. This is the final update for the event.; PublishedTime=2021-05-04T16:49:09.587Z

CS254208 Details

Status: We're reviewing and analyzing diagnostic data and system logs to identify the root cause of impact and determine a mitigation plan. Next update by: Tuesday, May 4, 2021, at 5:00 PM UTC; PublishedTime=2021-05-04T12:31:06.153Z

Final status: Our analysis of the diagnostic data and system logs determined that an authentication component misconfiguration within the affected infrastructure was preventing users from receiving email notifications from Cloud App Security. We've updated this configuration and restarted the affected infrastructure. We've confirmed via our system monitoring services that the issue is resolved and users can receive email notification again. Next update by: Tuesday, May 4, 2021, at 5:00 PM UTC; PublishedTime=2021-05-04T12:31:06.153Z

--------------------------------------------------------------------------------------------------
Title Users may have experienced delays sending email messages via any Exchange Online connection method
ID EX253410
Service Exchange Online
Feature E-Mail timely delivery
Status Service restored
Impact Users may have experienced delays of over an hour when sending email messages via any Exchange Online connection method.
Severity Sev2
Start time 2021-04-29T09:16:00Z
End time 2021-04-30T06:51:00Z
Last update 2021-04-30T10:19:17.723Z
More info
Root cause A recent update which contained listings and de-listings were not properly deployed, as a result of a corrupted rule that had recently deployed to the affected environment. Next steps: - We're reviewing our update procedures to better identify similar issues during our development and testing cycles. This is the final update for the event.; PublishedTime=2021-04-30T10:19:17.723Z

EX253410 Details

Status: We're investigating a potential issue and checking for impact to your organization. We'll provide an update within 30 minutes.; PublishedTime=2021-04-29T21:38:58.69

Status: We initially identified that a message sending threshold was exceeded, leading to impact; however, after attempting to process messages that were affected by this, mailflow remains blocked. We're reviewing system logs to isolate the underlying additional reasons that this problem is occurring. Next update by: Thursday, April 29, 2021, at 11:30 PMUTC; PublishedTime=2021-04-29T22:01:19.65Z

Status: We're continuing to review system logs and newly acquired Non-Delivery Reports (NDRs) to assist with determining why mail flow remains blocked for all users. Next update by: Friday, April 30, 2021, at 4:30 AMUTC; PublishedTime=2021-04-29T23:18:34.427Z

Status: We've determined that a recent update which contained listings and de-listings weren't properly deploying due to a corrupted rule which had recently saturated to the affected environment. We've developed a code fix which is currently deploying to mitigate impact for all users, and we're monitoring its progress as it saturates to the impacted environment. Next update by: Friday, April 30, 2021, at 3:00 AMUTC; PublishedTime=2021-04-30T00:42:35.953Z

Status: Our deployment of the code fix completed but we've received feedback from affected users that impact persists. We're reviewing available data to determine why the fix didn't resolve this issue and what we need to change or develop to mitigate impact. Next update by: Friday, April 30, 2021, at 5:00 AMUTC; PublishedTime=2021-04-30T02:47:45.307Z

Status: We've discovered that there was a misconfiguration in the deployment containing the fix that we deployed earlier. We're currently working to correct the misconfiguration within the deployment, after that we will deploy again and that should mitigate impact. Next update by: Friday, April 30, 2021, at 7:00 AMUTC; PublishedTime=2021-04-30T04:52:13.213Z

Status: We've found the corrupted rules and are currently working to fix them. Once the rules are fixed, we will build a new deployment to remediate the problem. Next update by: Friday, April 30, 2021, at 10:30 AMUTC; PublishedTime=2021-04-30T06:23:36.133Z

Final status: We've fixed the corrupted rules and have successfully deployed the new build, which has remediated impact. Next update by: Friday, April 30, 2021, at 10:30 AMUTC; PublishedTime=2021-04-30T06:23:36.133Z

--------------------------------------------------------------------------------------------------
Title Multiple functionality issues within the Teams Admin Portal
ID TM253297
Service Microsoft Teams
Feature Teams Components
Status Service restored
Impact Users may have experienced issues for multiple functionalities within the Teams Admin Portal.
Severity Sev2
Start time 2021-04-29T06:50:00Z
End time 2021-04-29T10:50:00Z
Last update 2021-04-29T11:34:53.077Z
More info Users may have experienced issues with the following functionalities: - Unable to view or makes changes to Auto Attendant. - Unable to view or make changes to call queues. - Experience delays with the opt-in and opt-out functionality. Final status: We finished rerouting user request traffic, along with reverting the impacting deployment. We've confirmed that the issue has been resolved after a period of monitoring. Scope of impact: Impact affected any admin attempting to perform the functionalities listed in the 'More Info' section. Start time: Thursday, April 29, 2021, at 6:50 AM UTC End time: Thursday, April 29, 2021, at 10:50 AM UTC Root cause: A recent service deployment which was part of regularly scheduled maintenance, inadvertently caused impact. Next steps: - We're reviewing our standard service update procedures to avoid similar impact in the future. This is the final update for the event.; PublishedTime=2021-04-29T11:34:53.077Z
Root cause A recent service deployment which was part of regularly scheduled maintenance, inadvertently caused impact. Next steps: - We're reviewing our standard service update procedures to avoid similar impact in the future. This is the final update for the event.; PublishedTime=2021-04-29T11:34:53.077Z

TM253297 Details

Status: We're analyzing diagnostic data to determine the root cause. Next update by: Thursday, April 29, 2021, at 10:30 AM UTC; PublishedTime=2021-04-29T08:37:50.753Z

Status: We've identified that a recent service deployment, which was part of regularly scheduled maintenance, has inadvertently caused impact. We're rerouting user request traffic as a mitigation step while we develop a plan to fully resolve the issue. Next update by: Thursday, April 29, 2021, at 12:30 PM UTC ; PublishedTime=2021-04-29T10:27:56.447Z

Final status: We finished rerouting user request traffic, along with reverting the impacting deployment. We've confirmed that the issue has been resolved after a period of monitoring. Next update by: Thursday, April 29, 2021, at 12:30 PM UTC ; PublishedTime=2021-04-29T10:27:56.447Z

--------------------------------------------------------------------------------------------------
Title Users intermittently can't install Volume Purchase Program (VPP) apps or Apple Books on Apple devices
ID IT253164
Service Microsoft Intune
Feature Microsoft Intune
Status Service restored
Impact Users were intermittently unable to install VPP apps or Apple Books on Apple devices via any installation method.
Severity Sev2
Start time 2021-04-28T18:22:00Z
End time 2021-04-28T20:30:00Z
Last update 2021-04-28T21:04:48.25Z
More info
Root cause A portion of third-party infrastructure that facilitates VPP and Apple Books installation wasn't processing requests as efficiently as expected, resulting in impact. Next steps: - We're reviewing ways to reduce or avoid impact when our third-party partners experience similar issues in the future. This is the final update for the event.; PublishedTime=2021-04-28T21:04:48.25Z

IT253164 Details

Status: Our telemetry has indicated an increase in failures of requests to a portion of third-party infrastructure that facilitates VPP apps and Apple Books installation requests. We're working with the third-party to mitigate impact and are monitoring as system health is restored. Next update by: Wednesday, April 28, 2021, at 9:00 PMUTC; PublishedTime=2021-04-28T20:02:36.03Z

Final status: We've confirmed our third-party partner has taken action to resolve the issue, and our telemetry indicates that service has been restored. Next update by: Wednesday, April 28, 2021, at 9:00 PMUTC; PublishedTime=2021-04-28T20:02:36.03Z

--------------------------------------------------------------------------------------------------
Title Admins intermittently can't access user data via the Microsoft Teams admin center
ID TM253162
Service Microsoft Teams
Feature Teams Components
Status Service restored
Impact Admins intermittently couldn't access user data via the Microsoft Teams admin center.
Severity Sev2
Start time 2021-04-28T17:45:00Z
End time 2021-05-10T05:30:00Z
Last update 2021-05-11T14:50:25.69Z
More info Admins may have see errors such as: "We can't load all of the users right now", "We can't get the activity log", and "We can't get any policies right now" when accessing user data. Admins may have been able to access user data as expected after clearing their browser cache, signing out of the browser user profile and restarting their browser, or by using InPrivate window browsing. Final status: We've completed deployment of the fix and have confirmed with some affected admins that impact has been fully remediated. Scope of impact: Any admin accessing user data via the Microsoft Teams admin center may have been impacted. Start time: Wednesday, April 28, 2021, at 5:45 PM UTC End time: Monday, May 10, 2021, at 5:30 AM UTC Root cause: A migration to an alternate authentication library within the Microsoft-managed environment caused an authentication configuration issue that resulted in errors when trying to access user data in the Microsoft Teams admin center. Next steps: - We're working to better understand why the migration to an alternate authentication library caused an authentication configuration issue to help prevent similar problems in the future. This is the final update for the event.; PublishedTime=2021-05-10T17:49:57.703Z
Root cause A migration to an alternate authentication library within the Microsoft-managed environment caused an authentication configuration issue that resulted in errors when trying to access user data in the Microsoft Teams admin center. Next steps: - We're working to better understand why the migration to an alternate authentication library caused an authentication configuration issue to help prevent similar problems in the future. This is the final update for the event.; PublishedTime=2021-05-10T17:49:57.703Z

TM253162 Details

Status: We're analyzing browser credential tokens to better understand the root cause of the issue. Next update by: Wednesday, April 28, 2021, at 9:00 PMUTC; PublishedTime=2021-04-28T20:01:37.27Z

Status: We've determined through our analysis of the browser credential tokens that this issue only impacts the Edge and Internet Explorer browsers . We're continuing our analysis to understand why it is only impacting specific browsers to identify the root cause and determine our next troubleshooting steps. Next update by: Wednesday, April 28, 2021, at 11:30 PMUTC; PublishedTime=2021-04-28T20:58:10.27Z

Status: Our analysis of the impacting specific browsers determined that a recent migration caused an authentication misconfiguration responsible for access to the Teams admin center, which results in errors for admins. We're in the process of reviewing mitigation options to safely resolve the issue. Next update by: Thursday, April 29, 2021, at 9:00 AMUTC; PublishedTime=2021-04-28T22:56:50.323Z

Status: We've identified a potential fix that we're in the process of testing, to validate it's effectiveness before moving to deploy across the affected environment. Next update by: Monday, May 3, 2021, at 7:00 PM UTC; PublishedTime=2021-04-29T07:38:19.587Z

Status: We're in the final stages of testing the fix to confirm it will be effective. Once testing is complete, we'll deploy it across all impacted environments. Next update by: Tuesday, May 4, 2021, at 7:00 PM UTC; PublishedTime=2021-05-03T07:31:01.613Z

Status: We've completed testing the fix and have begun its deployment to the affected environment. We anticipate that the fix will be fully deployed and impact remediated by Monday, May 10, 2021. Next update by: Monday, May 10, 2021, at 7:00 PM UTC; PublishedTime=2021-05-04T18:26:56.803Z

Final status: We've completed deployment of the fix and have confirmed with some affected admins that impact has been fully remediated. Next update by: Monday, May 10, 2021, at 7:00 PM UTC; PublishedTime=2021-05-04T18:26:56.803Z

--------------------------------------------------------------------------------------------------
Title Can't access Teams
ID TM252802
Service Microsoft Teams
Feature Teams Components
Status Post-incident report published
Impact Users may have been unable to access Microsoft Teams through both the web and desktop client.
Severity Sev1
Start time 2021-04-27T09:58:00Z
End time 2021-04-27T12:05:00Z
Last update 2021-05-04T18:08:08.717Z
More info Users able to access the Microsoft Teams may have experienced degraded performance with multiple features. These included but were not limited to: - Can't send or receive messages - Can't join meetings - Some channels may have not appeared - Can't join channels - Can't join chats Users still experiencing impact were encouraged to restart their client to expedite recovery. Final status: We've monitored the environment and confirmed that the service is performing as expected. If a user is still experiencing impact, they need to restart their clients to recover. Some users may see 'Int' text on top of their profile pictures and a restart of the client should also remove this new text. Scope of impact: Any user could have been affected by this issue. Start time: Tuesday, April 27, 2021, at 9:58 AM UTC End time: Tuesday, April 27, 2021, at 12:05 PM UTC Root cause: Microsoft Teams uses a pipeline mechanism to push client configurations to a central repository which manages all Microsoft Team client configurations. This pipeline facilitates change management and is involved in all configuration changes implemented to Microsoft Teams. We implemented a change to split the client configuration files into two types of settings; one for features and the other for content policies. This was implemented to ensure that changes to one settings type could not inadvertently override settings within the other type, if the file used was not the latest version. The change exposed a previously unknown bug within the pipeline mechanism that would result in merging spilt configuration files when it was unable to handle a missing split config file correctly. This merging of configuration files resulted in specific feature parameters being set to incorrect values or be removed. As the feature services did not expect these values, the services were unable to process the requests and failed. Next steps: - For a more comprehensive list of next steps and actions, please refer to the Post Incident Review document. We'll publish a post-incident report within five business days.; PublishedTime=2021-04-27T13:00:41.88Z
Root cause Microsoft Teams uses a pipeline mechanism to push client configurations to a central repository which manages all Microsoft Team client configurations. This pipeline facilitates change management and is involved in all configuration changes implemented to Microsoft Teams. We implemented a change to split the client configuration files into two types of settings; one for features and the other for content policies. This was implemented to ensure that changes to one settings type could not inadvertently override settings within the other type, if the file used was not the latest version. The change exposed a previously unknown bug within the pipeline mechanism that would result in merging spilt configuration files when it was unable to handle a missing split config file correctly. This merging of configuration files resulted in specific feature parameters being set to incorrect values or be removed. As the feature services did not expect these values, the services were unable to process the requests and failed. Next steps: - For a more comprehensive list of next steps and actions, please refer to the Post Incident Review document. We'll publish a post-incident report within five business days.; PublishedTime=2021-04-27T13:00:41.88Z

TM252802 Details

Status: We're investigating a potential issue and checking for impact to your organization. We'll provide an update within 30 minutes.; PublishedTime=2021-04-27T10:26:47.53

Status: We're analyzing system telemetry to isolate the source for the issue. In parallel, we're reviewing recent changes made to the service as a potential root cause. Next update by: Tuesday, April 27, 2021, at 12:30 PM UTC; PublishedTime=2021-04-27T10:47:42.577Z

Status: We're investigating telemetry from the Packet Delivery Service (PDS) and, in parallel, the Chat service to identify the root cause of the impact.; PublishedTime=2021-04-27T11:11:34.467Next update by: Tuesday, April 27, 2021, at 12:30 PM UTC; PublishedTime=2021-04-27T10:47:42.577Z

Status: We've identified errors within the chat service and are investigating these further to identify the source of the issue. Next update by: Tuesday, April 27, 2021, at 1:00 PM UTC; PublishedTime=2021-04-27T11:39:34.197Z

Status: We've identified a root cause of this issue and have pushed out a configuration change. We're monitoring service health as it recovers.; PublishedTime=2021-04-27T12:07:21.96Next update by: Tuesday, April 27, 2021, at 1:00 PM UTC; PublishedTime=2021-04-27T11:39:34.197Z

Status: We've identified the root cause as a recent configuration change. We've reverted the change and are seeing signs of recovery. Users still experiencing impact are encouraged to restart their client to expedite recovery. Next update by: Tuesday, April 27, 2021, at 2:00 PM UTC; PublishedTime=2021-04-27T12:14:04.42Z

Final status: We've monitored the environment and confirmed that the service is performing as expected. If a user is still experiencing impact, they need to restart their clients to recover. Some users may see 'Int' text on top of their profile pictures and a restart of the client should also remove this new text. Next update by: Tuesday, April 27, 2021, at 2:00 PM UTC; PublishedTime=2021-04-27T12:14:04.42Z

--------------------------------------------------------------------------------------------------
Title Admins may receive a "Session Expired" error when using the Intune Portal
ID IT253221
Service Microsoft Intune
Feature Microsoft Intune
Status Service restored
Impact Admins may have received a "Session Expired" error when using the Intune Portal.
Severity Sev2
Start time 2021-04-27T00:00:00Z
End time 2021-04-29T05:00:00Z
Last update 2021-04-29T05:58:16.3Z
More info Admins that received the "Session Expired" error would have been automatically signed out of the service. Admins could then sign back in to service without issue. Final status: As part of our efforts to prepare the fix, we found that recycling the authentication components from the affected regions has caused service availability to return to healthy levels, effectively mitigating impact. We'll continue to deploy the fix as a precaution; however, our internal monitoring shows that admins can now access the Intune portal as expected and service is restored. Scope of impact: Any admin attempting to use or access the Intune Portal would have been affected by this issue. Start time: Tuesday, April 27, 2021, at 12:00 AM UTC End time: Thursday, April 29, 2021, at 5:00 AM UTC Root cause: A recent auto-rotation of authentication components into service caused a misconfiguration in our infrastructure. Next steps: - We're continuing our review of our automated rotation systems for the impacted authentication component to better understand why the misconfiguration occurred and how to prevent it moving forward. - We're deploying our fix to prevent future instances of this issue from reoccurring. This is the final update for the event.; PublishedTime=2021-04-29T05:58:07.657Z
Root cause A recent auto-rotation of authentication components into service caused a misconfiguration in our infrastructure. Next steps: - We're continuing our review of our automated rotation systems for the impacted authentication component to better understand why the misconfiguration occurred and how to prevent it moving forward. - We're deploying our fix to prevent future instances of this issue from reoccurring. This is the final update for the event.; PublishedTime=2021-04-29T05:58:07.657Z

IT253221 Details

Status: We're investigating a potential issue and checking for impact to your organization. We'll provide an update within 30 minutes.; PublishedTime=2021-04-29T01:06:50.057

Status: Our review of the Intune Portal logs indicates that the requests aren't making it to our service. We're currently reviewing the Azure Portal logs to better understand impact. Next update by: Thursday, April 29, 2021, at 2:30 AMUTC; PublishedTime=2021-04-29T01:31:27.953Z

Status: We've determine that some of our authentication components that were auto-rotated into service are causing a misconfiguration in our infrastructure. We're developing a fix to correct the misconfiguration and subsequently mitigate impact. Next update by: Thursday, April 29, 2021, at 4:30 AMUTC; PublishedTime=2021-04-29T02:29:19.22Z

Status: We're currently pushing out a deployment to all affected regions to stage them for the deployment of the fix. Once we've staged all of the affected regions, we will be pushing out the fix for impact. Next update by: Thursday, April 29, 2021, at 6:30 AMUTC; PublishedTime=2021-04-29T03:28:24.03Z

Status: We're continuing to prepare affected regions for the fix deployment, which system monitoring indicates is helping to restore service availability, although we expect the fix will be needed to completely mitigate impact. We'll provide an updated deployment timeline for the fix when available. Next update by: Thursday, April 29, 2021, at 6:00 PMUTC; PublishedTime=2021-04-29T05:26:51.213Z

Final status: As part of our efforts to prepare the fix, we found that recycling the authentication components from the affected regions has caused service availability to return to healthy levels, effectively mitigating impact. We'll continue to deploy the fix as a precaution; however, our internal monitoring shows that admins can now access the Intune portal as expected and service is restored. Next update by: Thursday, April 29, 2021, at 6:00 PMUTC; PublishedTime=2021-04-29T05:26:51.213Z

--------------------------------------------------------------------------------------------------
Title Users may be unable to post messages
ID YA252648
Service Yammer Enterprise
Feature Yammer Components
Status Service restored
Impact Users may have been unable to post messages
Severity Sev2
Start time 2021-04-26T20:51:00Z
End time 2021-04-26T21:57:00Z
Last update 2021-04-26T22:32:55.193Z
More info
Root cause A subsection of SQL database infrastructure dropped below our manageable service performance thresholds, resulting in impact. Next steps: - We're analyzing performance data and trends on the affected systems to determine why services fell below performance thresholds, and to help prevent this problem from happening again. This is the final update for the event.; PublishedTime=2021-04-26T22:21:45.11Z

YA252648 Details

Status: We've identified an issue that is impacting the Microsoft Yammer service which is preventing users from posting messages. We're analyzing system logs to determine the source for this event. Next update by: Monday, April 26, 2021, at 11:30 PMUTC; PublishedTime=2021-04-26T22:05:22.39Z

Final status: We've determined that a subsection of SQL database infrastructure dropped below our manageable service performance thresholds, resulting in impact. We've rerouted traffic to an alternate subsection of SQL databases to remediate impact. Next update by: Monday, April 26, 2021, at 11:30 PMUTC; PublishedTime=2021-04-26T22:05:22.39Z

--------------------------------------------------------------------------------------------------
Title Admins seeing SharePoint Online storage quota changes being reverted after two days
ID SP252302
Service SharePoint Online
Feature SharePoint Features
Status Service degradation
Impact Admins' SharePoint Online storage quota changes are reverting after two days.
Severity Sev2
Start time 2021-04-23T18:29:29Z
End time
Last update 2021-05-18T00:20:48.023Z
More info Admins may receive email alerts or see banner alerts advising of low storage. Sites may enter 'read-only' mode if quota limits are exceeded. If you need immediate relief for specific sites, please reach out to your support representative for additional details for a temporary solution while we continue to develop a long-term fix. Current status: We're continuing our efforts of developing a long-term solution. Due to the complexity of the issue, we'll ensure to provide an updated timeline of completion when available. Scope of impact: This issue could potentially affect any of your admins who make changes to their SharePoint Online storage quota. Preliminary root cause: A back-end process is unexpectedly causing SharePoint Online storage quota changes to be reverted. Next update by: Tuesday, May 25, 2021, at 1:00 AMUTC; PublishedTime=2021-05-18T00:20:48.023Z
Root cause A back-end process is unexpectedly causing SharePoint Online storage quota changes to be reverted. Next update by: Tuesday, May 11, 2021, at 1:00 AM UTC; PublishedTime=2021-05-03T23:29:42.043Z

SP252302 Details

Status: We're investigating a potential issue and checking for impact to your organization. We'll provide an update within 30 minutes.; PublishedTime=2021-04-23T18:31:17.497

Status: We're reviewing system telemetry and logs in conjunction with support case details to identify the root cause of impact. Next update by: Friday, April 23, 2021, at 8:00 PMUTC; PublishedTime=2021-04-23T18:49:38.433Z

Status: We're expanding our investigation to include the code that facilitates automated quota changes. Next update by: Saturday, April 24, 2021, at 12:00 AMUTC; PublishedTime=2021-04-23T19:27:41.063Z

Status: Our investigation into the code that facilitates automated quota changes has been inconclusive so far. We're continuing that investigation while we work to investigate specific examples of quota changes from affected users' sites. Next update by: Sunday, April 25, 2021, at 1:00 AMUTC; PublishedTime=2021-04-23T23:53:20.803Z

Status: Our review of the quota changes from affected sites has helped narrow our investigative path but hasn't introduced a definitive source of impact. We're analyzing Unified Logging Service (ULS) logs and gathered additional data on the affected SharePoint Online sites for further review. We expect to have an action plan ready by our next scheduled update. Next update by: Tuesday, April 27, 2021, at 1:00 AMUTC; PublishedTime=2021-04-24T23:39:54.37Z

Status: We were able to correlate the issue based on our analysis of the Unified Logging Service (ULS) logs. We determined that a back-end process is unexpectedly causing the quota changes to be reverted. We're developing a fix to prevent the quota changes from being reverted. Next update by: Tuesday, May 4, 2021, at 1:00 AMUTC; PublishedTime=2021-04-27T00:28:13.953Z

Status: We're continuing to work on a long-term solution to resolve the issue. We do expect that the testing and deployment process may require several weeks to complete and recommend users reach out to support if more immediate assistance is required. Next update by: Tuesday, May 11, 2021, at 1:00 AM UTC; PublishedTime=2021-05-03T23:29:42.043Z

Status: The process to develop a long-term solution is continuing as expected. We anticipate that we'll have an estimated timeline for completion by our next scheduled update. Next update by: Tuesday, May 18, 2021, at 1:00 AMUTC; PublishedTime=2021-05-11T00:08:17.3Z

Status: We're continuing our efforts of developing a long-term solution. Due to the complexity of the issue, we'll ensure to provide an updated timeline of completion when available. Next update by: Tuesday, May 25, 2021, at 1:00 AMUTC; PublishedTime=2021-05-18T00:20:48.023Z

--------------------------------------------------------------------------------------------------
Title Some users experience an error when attempting to launch apps with a Microsoft 365 or SharePoint Online connector
ID MM252194
Service Power Apps in Microsoft 365
Feature Service and web access issues
Status Service restored
Impact Users may have experienced an error when attempting to launch apps with a Microsoft 365 or SharePoint Online connector.
Severity Sev2
Start time 2021-04-22T20:48:00Z
End time 2021-04-22T21:35:00Z
Last update 2021-04-23T01:25:23.673Z
More info Users may have intermittently received a '520 internal error' error message. Final status: We've determined that a portion of infrastructure responsible for data calls to SharePoint Online and Microsoft 365 wasn't performing as efficiently as expected, which resulted in impact. We've rerouted traffic to alternate infrastructure and confirmed via telemetry that service is restored. Scope of impact: Your organization was affected by this event, and some users attempting to launch apps with a Microsoft 365 or SharePoint Online connector may have been impacted. Start time: Thursday, April 22, 2021, at 8:48 PM UTC End time: Thursday, April 22, 2021, at 9:35 PM UTC Root cause: A portion of infrastructure responsible for data calls to SharePoint Online and Microsoft 365 wasn't performing as efficiently as expected, which caused some users to intermittently experience a '520 internal error' error message when attempting to launch apps with a Microsoft 365 or SharePoint Online connector. Next steps: - To help prevent similar impact in the future, we're further reviewing the underlying cause of the performance issue. This is the final update for the event.; PublishedTime=2021-04-23T01:25:23.673Z
Root cause A portion of infrastructure responsible for data calls to SharePoint Online and Microsoft 365 wasn't performing as efficiently as expected, which caused some users to intermittently experience a '520 internal error' error message when attempting to launch apps with a Microsoft 365 or SharePoint Online connector. Next steps: - To help prevent similar impact in the future, we're further reviewing the underlying cause of the performance issue. This is the final update for the event.; PublishedTime=2021-04-23T01:25:23.673Z

MM252194 Details

Status: We've investigated a problem with the Microsoft Power Apps service in which users may have experienced a '520 internal error' error message intermittently when attempting to launch apps with a Microsoft 365 or SharePoint Online connector, and we've confirmed that service is now restored. We'll provide additional information in a closure summary within the 'History' section of the 'Service health' dashboard shortly.; PublishedTime=2021-04-23T00:51:43.54

Final status: We've determined that a portion of infrastructure responsible for data calls to SharePoint Online and Microsoft 365 wasn't performing as efficiently as expected, which resulted in impact. We've rerouted traffic to alternate infrastructure and confirmed via telemetry that service is restored.

--------------------------------------------------------------------------------------------------
Title Some users are unable to send or receive email messages within Exchange Online
ID EX252124
Service Exchange Online
Feature E-Mail timely delivery
Status Post-incident report published
Impact Users may have experienced delays or been unable to send or receive email messages within the Exchange Online service.
Severity Sev2
Start time 2021-04-22T13:00:00Z
End time 2021-04-22T16:30:00Z
Last update 2021-05-12T00:11:32.807Z
More info Some users may have received the error message '451 4.3.2 Temporary server error. Please try again later ATTR1' if they attempted to email an impacted organization. Final status: We've monitored the environment and confirmed that the queued emails have processed as expected. Scope of impact: Impact was specific to organizations hosted from infrastructure located in North America. However, if a user not hosted on the affected infrastructure attempted to send an email to a user within an impacted organization, they may have been unable to send the email. Start time: Thursday, April 22, 2021, at 1:00 PM UTC End time: Thursday, April 22, 2021, at 4:30 PM UTC Root cause: Our preliminary investigation determined the impact was caused by a load-balancing configuration issue. This behavior was triggered by a specific type of call which occurred when inbound mail was received. Under normal circumstances this type of call would be routed to the Data Center cache, but in this instance, the call was also being sent directly to the database, causing a conflict, and resulting in a spike of resource utilization. Next steps: - We're analyzing performance data and trends on the affected systems to help prevent this problem from happening again. The Post-Incident Report (PIR) was published on Thursday, April 29, 2021.; PublishedTime=2021-04-22T17:09:27.183Z
Root cause Our preliminary investigation determined the impact was caused by a load-balancing configuration issue. This behavior was triggered by a specific type of call which occurred when inbound mail was received. Under normal circumstances this type of call would be routed to the Data Center cache, but in this instance, the call was also being sent directly to the database, causing a conflict, and resulting in a spike of resource utilization. Next steps: - We're analyzing performance data and trends on the affected systems to help prevent this problem from happening again. The Post-Incident Report (PIR) was published on Thursday, April 29, 2021.; PublishedTime=2021-04-22T17:09:27.183Z

EX252124 Details

Status: We're reviewing service monitoring telemetry to isolate the source of the issue and establish a fix. Next update by: Thursday, April 22, 2021, at 4:00 PM UTC; PublishedTime=2021-04-22T14:31:36.7Z

Status: Our preliminary investigation has determined impact is being caused by a load balancing configuration issue. This occurs when a call to validate inbound mail is received. This call is being duplicated, creating a spike in utilization. We're working to restore the underlying database infrastructure to remediate impact. Emails were queued for delivery and the service will automatically retry deliveries once the underlying database infrastructure has recovered. Users will not need to resend the impacted emails. Next update by: Thursday, April 22, 2021, at 5:30 PM UTC; PublishedTime=2021-04-22T15:40:04.073Z

Status: Our database infrastructure has recovered and we've confirmed that new mail will be delivered as expected going forward; however, a queue of messages built up during this event and is now being processed. We estimate that all mail affected by this event will be delivered within one to two hours. This is a quick update intended to provide the latest information on this event. For full details on this event, please reference the full post made on April 22, 2021, at 3:40 PM UTC.; PublishedTime=2021-04-22T16:03:46.74Next update by: Thursday, April 22, 2021, at 5:30 PM UTC; PublishedTime=2021-04-22T15:40:04.073Z

Status: Users will not need to resend the impacted emails. Emails were queued for delivery and the service will automatically retry deliveries once the underlying database infrastructure has recovered. This process is underway now. This is a quick update intended to provide the latest information on this event. For full details on this event, please reference the full post made on April 22, 2021, at 3:40 PM UTC.; PublishedTime=2021-04-22T16:12:27.383Next update by: Thursday, April 22, 2021, at 5:30 PM UTC; PublishedTime=2021-04-22T15:40:04.073Z

Status: We've recovered the underlying database components and confirmed that new emails are processing as expected without delays. However, during the incident, a queue of email messages built up. The service is automatically retrying these emails and we expect recovery within the next hour. We're monitoring the service to confirm these email queues are processing correctly. Organizations will see recovery at different times depending on their volume of queued emails. Next update by: Thursday, April 22, 2021, at 6:00 PM UTC; PublishedTime=2021-04-22T16:17:00.76Z

Status: Please be advised that while mail is being delivered as expected, some users may observe delivery latency as our systems process messages that were queued during this event. This is a quick update intended to provide the latest information on this event. For full details on this event, please reference the full post made on April 22, 2021, at 4:17 PM UTC.; PublishedTime=2021-04-22T17:00:50.17Next update by: Thursday, April 22, 2021, at 6:00 PM UTC; PublishedTime=2021-04-22T16:17:00.76Z

Final status: We've monitored the environment and confirmed that the queued emails have processed as expected. Next update by: Thursday, April 22, 2021, at 6:00 PM UTC; PublishedTime=2021-04-22T16:17:00.76Z

--------------------------------------------------------------------------------------------------
Title Users sending messages in Microsoft Teams are intermittently seeing the messages appear out of order
ID TM254853
Service Microsoft Teams
Feature Teams Components
Status Service degradation
Impact Users that are sending messages in Microsoft Teams are intermittently seeing the messages appearing out of order.
Severity Sev2
Start time 2021-04-21T03:37:00Z
End time
Last update 2021-05-17T21:13:41.26Z
More info Exiting and then returning to the chat or thread with the problematic messages may provide relief from this issue. Current status: Validation of the fix is complete and we've initiated an incremental deployment, which is already providing relief to some affected users. We expect to deploy the final portion of the fix by Monday, May 21, 2021, and we'll provide an estimated timeline to full mitigation at our next scheduled update. Scope of impact: Your organization is affected by this event, and any user may notice that Microsoft Teams messages in threads and chats are intermittently appearing out of order. Start time: Wednesday, April 21, 2021, at 3:37 AM'UTC Root cause: A recent change intended to fix message timestamp flickering in Microsoft Teams unexpectedly resulted in impact. Next update by: Monday, May 24, 2021, at 10:30 PMUTC; PublishedTime=2021-05-17T21:13:41.26Z
Root cause A recent change intended to fix message timestamp flickering in Microsoft Teams unexpectedly resulted in impact. Next update by: Monday, May 24, 2021, at 10:30 PMUTC; PublishedTime=2021-05-17T21:13:41.26Z

TM254853 Details

Status: We're investigating a potential issue and checking for impact to your organization. We'll provide an update within 30 minutes.

Status: We're comparing screenshots of impacted messages and analyzing Microsoft Teams diagnostics logs provided by affected users to determine the source of this problem. Next update by: Thursday, May 6, 2021, at 9:30 PMUTC; PublishedTime=2021-05-06T20:08:07.7Z

Status: We've identified that a recent change intended to fix message timestamp flickering in Microsoft Teams unexpectedly resulted in impact. We've developed and are validating a fix for this problem to ensure it will resolve impact before applying it to the affected environments. Next update by: Monday, May 10, 2021, at 10:30 PMUTC; PublishedTime=2021-05-06T21:21:41.693Z

Status: We're continuing to internally test and validate the fix. We're expecting for internal validation to have completed and to be able to provide an estimate for the fix deployment by Monday, May 17, 2021, at 10:30 PM UTC. Next update by: Monday, May 17, 2021, at 10:30 PMUTC; PublishedTime=2021-05-10T21:03:29.2Z

Status: Validation of the fix is complete and we've initiated an incremental deployment, which is already providing relief to some affected users. We expect to deploy the final portion of the fix by Monday, May 21, 2021, and we'll provide an estimated timeline to full mitigation at our next scheduled update. Next update by: Monday, May 24, 2021, at 10:30 PMUTC; PublishedTime=2021-05-17T21:13:41.26Z

--------------------------------------------------------------------------------------------------
Title Users are unable to launch Outlook mobile for iOS due to crashes on sign-in
ID EX251697
Service Exchange Online
Feature Sign-in
Status Service restored
Impact Users were unable to launch Outlook mobile for iOS due to crashes on sign-in.
Severity Sev2
Start time 2021-04-20T08:00:00Z
End time 2021-04-20T23:30:00Z
Last update 2021-04-21T01:27:34.35Z
More info This issue was specific to Outlook Mobile for iOS version 4.2115.0 installed on Microsoft Intune-managed devices. Users may have been able to restore Outlook Mobile for iOS functionality by re-downloading the app directly from the App Store. Final status: We've released the updated build 4.2115.2, which has been validated and confirmed to resolve the underlying issue. We suspect it may take up to 24 hours for the build to be available for all users to download via the App Store or the Company Portal app. We recommend users to periodically check the App Store or the Company Portal app until the new build 4.2115.2 is available for download. Scope of impact: Users who installed Outlook Mobile for iOS on their Microsoft Intune-managed devices were impacted. Start time: Tuesday, April 20, 2021, at 8:00 AM UTC End time: Tuesday, April 20, 2021, at 11:30 PM UTC Root cause: A recent Outlook Mobile for iOS update resulted in app crashes, preventing access to the app. Next steps: - We're reviewing the recent Outlook Mobile for iOS update to better understand why it resulted in app crashes, and to assess how the problem wasn't identified during our update testing procedures. This is the final update for the event.; PublishedTime=2021-04-21T01:22:56.583Z
Root cause A recent Outlook Mobile for iOS update resulted in app crashes, preventing access to the app. Next steps: - We're reviewing the recent Outlook Mobile for iOS update to better understand why it resulted in app crashes, and to assess how the problem wasn't identified during our update testing procedures. This is the final update for the event.; PublishedTime=2021-04-21T01:22:56.583Z

EX251697 Details

Status: We're gathering diagnostic data for analysis. Next update by: Tuesday, April 20, 2021, at 4:00 PM UTC; PublishedTime=2021-04-20T13:57:38.33Z

Status: We're reviewing case details from users who have reported impact to better isolate the underlying cause of the issue and determine our next troubleshooting steps. Next update by: Tuesday, April 20, 2021, at 8:00 PMUTC; PublishedTime=2021-04-20T15:58:08.983Z

Status: We're reviewing recent changes made to the service, specifically with a new iOS application update, to determine the next steps. Next update by: Tuesday, April 20, 2021, at 7:00 PM UTC; PublishedTime=2021-04-20T16:07:38.69Z

Status: We've determined that a recent update introduced a code issue causing Outlook Mobile for iOS to become inaccessible. We're developing an Outlook Mobile for iOS update to resolve the issue. Next update by: Tuesday, April 20, 2021, at 10:00 PMUTC; PublishedTime=2021-04-20T18:58:14.607Z

Status: We've completed developing a newer build of the application expected to resolve the issue and it's currently undergoing review and validation prior to release. Next update by: Wednesday, April 21, 2021, at 2:00 AMUTC; PublishedTime=2021-04-20T21:47:45.087Z

Status: We've determined that this issue impacts all Outlook Mobile for iOS installs for Microsoft Intune-managed devices, not just VPP installed apps. The problem was introduced in the most recent version of Outlook Mobile for iOS, version 4.2115.0. The updated build, 4.2115.2, has been approved and we'll provide a better timeline on availability in our next scheduled update. Next update by: Wednesday, April 21, 2021, at 2:30 AMUTC; PublishedTime=2021-04-20T23:12:25.807Z

Final status: We've released the updated build 4.2115.2, which has been validated and confirmed to resolve the underlying issue. We suspect it may take up to 24 hours for the build to be available for all users to download via the App Store or the Company Portal app. We recommend users to periodically check the App Store or the Company Portal app until the new build 4.2115.2 is available for download. Next update by: Wednesday, April 21, 2021, at 2:30 AMUTC; PublishedTime=2021-04-20T23:12:25.807Z

--------------------------------------------------------------------------------------------------
Title Users are unable to launch Outlook mobile for iOS due to crashes on sign-in
ID EX251697
Service Exchange Online
Feature Sign-in
Status Service restored
Impact Users were unable to launch Outlook mobile for iOS due to crashes on sign-in.
Severity Sev2
Start time 2021-04-20T08:00:00Z
End time 2021-04-20T23:30:00Z
Last update 2021-04-21T01:27:34.35Z
More info This issue was specific to Outlook Mobile for iOS version 4.2115.0 installed on Microsoft Intune-managed devices. Users may have been able to restore Outlook Mobile for iOS functionality by re-downloading the app directly from the App Store. Final status: We've released the updated build 4.2115.2, which has been validated and confirmed to resolve the underlying issue. We suspect it may take up to 24 hours for the build to be available for all users to download via the App Store or the Company Portal app. We recommend users to periodically check the App Store or the Company Portal app until the new build 4.2115.2 is available for download. Scope of impact: Users who installed Outlook Mobile for iOS on their Microsoft Intune-managed devices were impacted. Start time: Tuesday, April 20, 2021, at 8:00 AM UTC End time: Tuesday, April 20, 2021, at 11:30 PM UTC Root cause: A recent Outlook Mobile for iOS update resulted in app crashes, preventing access to the app. Next steps: - We're reviewing the recent Outlook Mobile for iOS update to better understand why it resulted in app crashes, and to assess how the problem wasn't identified during our update testing procedures. This is the final update for the event.; PublishedTime=2021-04-21T01:22:56.583Z
Root cause A recent Outlook Mobile for iOS update resulted in app crashes, preventing access to the app. Next steps: - We're reviewing the recent Outlook Mobile for iOS update to better understand why it resulted in app crashes, and to assess how the problem wasn't identified during our update testing procedures. This is the final update for the event.; PublishedTime=2021-04-21T01:22:56.583Z

EX251697 Details

Status: We're gathering diagnostic data for analysis. Next update by: Tuesday, April 20, 2021, at 4:00 PM UTC; PublishedTime=2021-04-20T13:57:38.33Z

Status: We're reviewing case details from users who have reported impact to better isolate the underlying cause of the issue and determine our next troubleshooting steps. Next update by: Tuesday, April 20, 2021, at 8:00 PMUTC; PublishedTime=2021-04-20T15:58:08.983Z

Status: We're reviewing recent changes made to the service, specifically with a new iOS application update, to determine the next steps. Next update by: Tuesday, April 20, 2021, at 7:00 PM UTC; PublishedTime=2021-04-20T16:07:38.69Z

Status: We've determined that a recent update introduced a code issue causing Outlook Mobile for iOS to become inaccessible. We're developing an Outlook Mobile for iOS update to resolve the issue. Next update by: Tuesday, April 20, 2021, at 10:00 PMUTC; PublishedTime=2021-04-20T18:58:14.607Z

Status: We've completed developing a newer build of the application expected to resolve the issue and it's currently undergoing review and validation prior to release. Next update by: Wednesday, April 21, 2021, at 2:00 AMUTC; PublishedTime=2021-04-20T21:47:45.087Z

Status: We've determined that this issue impacts all Outlook Mobile for iOS installs for Microsoft Intune-managed devices, not just VPP installed apps. The problem was introduced in the most recent version of Outlook Mobile for iOS, version 4.2115.0. The updated build, 4.2115.2, has been approved and we'll provide a better timeline on availability in our next scheduled update. Next update by: Wednesday, April 21, 2021, at 2:30 AMUTC; PublishedTime=2021-04-20T23:12:25.807Z

Final status: We've released the updated build 4.2115.2, which has been validated and confirmed to resolve the underlying issue. We suspect it may take up to 24 hours for the build to be available for all users to download via the App Store or the Company Portal app. We recommend users to periodically check the App Store or the Company Portal app until the new build 4.2115.2 is available for download. Next update by: Wednesday, April 21, 2021, at 2:30 AMUTC; PublishedTime=2021-04-20T23:12:25.807Z

--------------------------------------------------------------------------------------------------
Title Some users can't load their communities and feeds
ID YA253431
Service Yammer Enterprise
Feature Yammer Components
Status Service restored
Impact Users may have been unable to load their communities and feeds.
Severity Sev2
Start time 2021-04-19T23:04:00Z
End time 2021-04-30T00:30:00Z
Last update 2021-04-30T00:41:16.583Z
More info
Root cause A portion of infrastructure that facilitates loading Yammer communities and feeds wasn't performing as efficiently as expected, resulting in impact. Next steps: - To help prevent similar impact in the future, we're further reviewing the underlying cause of the performance issue. This is the final update for the event.; PublishedTime=2021-04-30T00:41:16.583Z

YA253431 Details

Status: We're investigating a potential issue and checking for impact to your organization. We'll provide an update within 30 minutes.

Status: We've determined a portion of infrastructure that facilitates loading Yammer communities and feeds isn't performing as efficiently as expected, resulting in impact. We've restarted the affected portion of service infrastructure to mitigate impact and we're monitoring telemetry to ensure service is restored. Next update by: Friday, April 30, 2021, at 1:00 AMUTC; PublishedTime=2021-04-30T00:01:45.677Z

Final status: We've finished monitoring telemetry and confirmed that service is restored. Next update by: Friday, April 30, 2021, at 1:00 AMUTC; PublishedTime=2021-04-30T00:01:45.677Z

--------------------------------------------------------------------------------------------------
Title Admins may notice that all usage reports data is delayed since April 22, 2021 within the Microsoft 365 admin center
ID MO251848
Service Microsoft 365 suite
Feature Administration
Status Service restored
Impact Admins may have noticed all usage report data was delayed since April 22, 2021 within the Microsoft 365 admin center.
Severity Sev2
Start time 2021-04-19T00:00:00Z
End time 2021-05-01T00:00:00Z
Last update 2021-05-01T16:15:14.71Z
More info This issue previously impacted usage report data starting from April 18, 2021, but that impact was resolved and isolated to April 22, 2021 and beyond with our communication update on Sunday, April 25, 2021. Final status: After continued monitoring, we've confirmed that all usage report data is now up to date in the Microsoft 365 admin center and the issue is resolved. Scope of impact: This issue affected any of your admins attempting to view usage reports data after April 22, 2021. Start time: Monday, April 19, 2021, at 12:00 AM'UTC End time: Saturday, May 1, 2021, at 12:00 AM'UTC Root cause: A portion of upstream data source service infrastructure was performing below acceptable thresholds, resulting in impact. Next steps: - We're reviewing the affected service infrastructure to better understand why it wasn't operating as expected and for improved performance opportunities. This is the final update for the event.; PublishedTime=2021-05-01T16:15:14.71Z
Root cause A portion of upstream data source service infrastructure was performing below acceptable thresholds, resulting in impact. Next steps: - We're reviewing the affected service infrastructure to better understand why it wasn't operating as expected and for improved performance opportunities. This is the final update for the event.; PublishedTime=2021-05-01T16:15:14.71Z

MO251848 Details

Status: Our monitoring systems raised an issue where admins will notice that any usage report data is delayed since April 18, 2021. We've determined that a portion of our upstream data source service infrastructure is performing below acceptable thresholds, resulting for impact. We've resolved the data source issue and are working to restore affected reports to ensure admins can access all reports as expected. Next update by: Thursday, April 22, 2021, at 5:30 AMUTC; PublishedTime=2021-04-21T03:46:17.447Z

Status: We've successfully restored most of the affected reports and are continuing to restore the remaining portion to fully resolve this issue. Next update by: Friday, April 23, 2021, at 5:30 AMUTC; PublishedTime=2021-04-22T04:31:21.863Z

Status: Most of the affected usage reports have been restored and we're continuing to restore the residual affected reports. We anticipate the remaining delays will be resolved by our next scheduled update. Next update by: Monday, April 26, 2021, at 5:30 AM UTC; PublishedTime=2021-04-23T05:28:31.587Z

Status: A separate upstream data source issue has resulted in returning delays for all previously restored reports starting from April 22, 2021. We're analyzing the available data from the data source and the delivery pipeline to determine why impact has resurfaced. Next update by: Monday, April 26, 2021, at 5:30 AM UTC; PublishedTime=2021-04-25T02:31:45.267Z

Status: We've fixed the pipeline, but impact persists. We're further analyzing available diagnostic data from the upstream data source and delivery pipeline to identify any additional mitigation actions that may be necessary to allow the affected usage reports to be restored. Next update by: Tuesday, April 27, 2021, at 9:00 AMUTC; PublishedTime=2021-04-26T05:28:03.797Z

Status: We identified that a code bug issue within upstream was delaying remediation. We've fixed this issue and the pipeline is running again. We're monitoring the environment over the next 24 hours to ensure full remediation. Next update by: Wednesday, April 28, 2021, at 9:00 AM UTC; PublishedTime=2021-04-27T08:58:02.99Z

Status: We're processing the backlog of usage report data. We're continuing to monitor the environment as the usage report data continues to run until full remediation. Next update by: Thursday, April 29, 2021, at 9:00 AM UTC; PublishedTime=2021-04-28T07:42:29.58Z

Status: We continue to assess the affected environment as the usage report data is continuing to run until full mitigation. Next update by: Friday, April 30, 2021, at 9:00 AM UTC; PublishedTime=2021-04-29T07:54:52.053Z

Status: We've confirmed that most reports are now updated with one remaining. We'll continue to monitor the usage report data as it progressing towards it's final steps. Next update by: Monday, May 3, 2021, at 1:30 PM UTC; PublishedTime=2021-04-30T07:52:52.573Z

Final status: After continued monitoring, we've confirmed that all usage report data is now up to date in the Microsoft 365 admin center and the issue is resolved. Next update by: Monday, May 3, 2021, at 1:30 PM UTC; PublishedTime=2021-04-30T07:52:52.573Z

--------------------------------------------------------------------------------------------------
Title Admins were unable to view up-to-date reports in the Microsoft 365 admin center
ID MO251134
Service Microsoft 365 suite
Feature Administration
Status Service restored
Impact Admins may have been unable to view up-to-date reports from the Microsoft 365 admin center.
Severity Sev2
Start time 2021-04-16T00:00:00Z
End time 2021-04-20T09:22:00Z
Last update 2021-04-20T09:33:44.57Z
More info The problem affected several reports, including the Microsoft Teams usage reports and Microsoft 365 activation reports. The reports haven't updated since Tuesday, April 13, 2021. Final status: Following additional monitoring, we've confirmed that the Forms reports have populated and the issue is now resolved. Scope of impact: The problem may have impact any admin attempting to view reports from the Microsoft 365 admin center. Start time: Friday, April 16, 2021, at 12:00 AM UTC End time: Tuesday, April 20, 2021, at 9:22 AM UTC Root cause: Processing delays in our report generation pipeline were causing impact. Next steps: - We're reviewing our report generation and pipeline procedures to find ways to prevent this problem from happening again. This is the final update for the event.; PublishedTime=2021-04-20T09:33:44.57Z
Root cause Processing delays in our report generation pipeline were causing impact. Next steps: - We're reviewing our report generation and pipeline procedures to find ways to prevent this problem from happening again. This is the final update for the event.; PublishedTime=2021-04-20T09:33:44.57Z

MO251134 Details

Status: We've determined that an upstream relay is causing a delay in updating the reports. We're working to restore the reports and expect the process to be complete within the next 18 hours. Next update by: Monday, April 19, 2021, at 5:00 AM UTC; PublishedTime=2021-04-18T10:57:03.38Z

Status: It's taking longer than initially expected to restore the reports. Most of the reports are now up-to-date; however, Exchange, Yammer, and Forms reports are still delayed. We're monitoring as the remaining reports are restored, which we anticipate will complete by our next scheduled update. Next update by: Tuesday, April 20, 2021, at 4:00 AMUTC; PublishedTime=2021-04-19T03:49:12.653Z

Status: The majority of reports are up to date except for Forms, which is taking longer than expected. We're continuing to monitor to ensure the final report is up to date. Next update by: Wednesday, April 21, 2021, at 4:00 AMUTC; PublishedTime=2021-04-20T02:23:21.623Z

Final status: Following additional monitoring, we've confirmed that the Forms reports have populated and the issue is now resolved. Next update by: Wednesday, April 21, 2021, at 4:00 AMUTC; PublishedTime=2021-04-20T02:23:21.623Z

--------------------------------------------------------------------------------------------------
Title Users experience failures when adding a DocuSign connection or running a flow with a DocuSign connection
ID MF250962
Service Microsoft Power Automate in Microsoft 365
Feature Service and web access issues
Status Service restored
Impact Users experience failures when adding a DocuSign connection or running a flow with a DocuSign connection.
Severity
Start time 2021-04-15T19:30:00Z
End time 2021-04-16T23:00:00Z
Last update 2021-04-17T00:40:39.023Z
More info
Root cause A recent deployment contained a misconfiguration change, causing impact to the DocuSign connections infrastructure. Next steps: - We're reviewing this misconfiguration to understand how it occurred and help prevent similar impact from reoccurring in the future. This is the final update for the event.; PublishedTime=2021-04-17T00:40:39.023Z

MF250962 Details

Status: We've received reports of failures impacting flows with DocuSign connections. We've identified a misconfiguration within a portion of the infrastructure responsible for DocuSign connections to flows is causing this impact. We are reviewing additional service telemetry to formulate a mitigation plan. Next update by: Saturday, April 17, 2021, at 12:00 AMUTC; PublishedTime=2021-04-16T23:07:39.957Z

Status: We identified a misconfiguration on a portion of the infrastructure responsible for DocuSign connections. We've applied configuration changes which we've redeployed to the impacted portion of the infrastructure and are monitoring telemetry to confirm that the issue is resolved. Next update by: Saturday, April 17, 2021, at 1:30 AMUTC; PublishedTime=2021-04-16T23:21:34.277Z

Final status: We've verified after monitoring internal telemetry that DocuSign connections are now completing as expected. Next update by: Saturday, April 17, 2021, at 1:30 AMUTC; PublishedTime=2021-04-16T23:21:34.277Z

--------------------------------------------------------------------------------------------------
Title Users experience failures when adding a DocuSign connection or running a flow with a DocuSign connection
ID MF250962
Service Microsoft Power Automate in Microsoft 365
Feature Service and web access issues
Status Service restored
Impact Users experience failures when adding a DocuSign connection or running a flow with a DocuSign connection.
Severity
Start time 2021-04-15T19:30:00Z
End time 2021-04-16T23:00:00Z
Last update 2021-04-17T00:40:39.023Z
More info
Root cause A recent deployment contained a misconfiguration change, causing impact to the DocuSign connections infrastructure. Next steps: - We're reviewing this misconfiguration to understand how it occurred and help prevent similar impact from reoccurring in the future. This is the final update for the event.; PublishedTime=2021-04-17T00:40:39.023Z

MF250962 Details

Status: We've received reports of failures impacting flows with DocuSign connections. We've identified a misconfiguration within a portion of the infrastructure responsible for DocuSign connections to flows is causing this impact. We are reviewing additional service telemetry to formulate a mitigation plan. Next update by: Saturday, April 17, 2021, at 12:00 AMUTC; PublishedTime=2021-04-16T23:07:39.957Z

Status: We identified a misconfiguration on a portion of the infrastructure responsible for DocuSign connections. We've applied configuration changes which we've redeployed to the impacted portion of the infrastructure and are monitoring telemetry to confirm that the issue is resolved. Next update by: Saturday, April 17, 2021, at 1:30 AMUTC; PublishedTime=2021-04-16T23:21:34.277Z

Final status: We've verified after monitoring internal telemetry that DocuSign connections are now completing as expected. Next update by: Saturday, April 17, 2021, at 1:30 AMUTC; PublishedTime=2021-04-16T23:21:34.277Z

--------------------------------------------------------------------------------------------------
Title Users on Outlook on the web are noticing that signatures in email messages aren't showing up
ID EX252201
Service Exchange Online
Feature E-Mail and calendar access
Status Service restored
Impact Users on Outlook on the web were noticing that signatures in email messages weren't showing up as expected.
Severity Sev2
Start time 2021-04-15T01:10:00Z
End time 2021-04-28T05:00:00Z
Last update 2021-04-28T18:29:59.833Z
More info Users with email signatures that were already attached noticed that the signature wasn't appearing when creating new emails. Users may have also noticed that options "Automatically include my signature on new messages that I compose" and "Automatically include my signature on messages I forward or reply to" were appearing as unchecked after refreshing. Final status: We've confirmed that the fix has successfully completed deployment throughout the affected environment and we've verified with affected users that this action has alleviated the problem. Scope of impact: Users with existing email signatures or users attempting to add new email signatures using Outlook on the web were experiencing this issue. Start time: Thursday, April 15, 2021, at 1:10 AM UTC End time: Wednesday, April 28, 2021, at 5:00 AM UTC Root cause: A recent Outlook on the web update introduced a code issue, resulting in the impact. Next steps: - We're reviewing our update validation and testing processes to better identify similar code issues impacting email signatures to prevent similar problems from happening again in the future. This is the final update for the event.; PublishedTime=2021-04-28T18:29:46.67Z
Root cause A recent Outlook on the web update introduced a code issue, resulting in the impact. Next steps: - We're reviewing our update validation and testing processes to better identify similar code issues impacting email signatures to prevent similar problems from happening again in the future. This is the final update for the event.; PublishedTime=2021-04-28T18:29:46.67Z

EX252201 Details

Status: We're investigating a potential issue and checking for impact to your organization. We'll provide an update within 30 minutes.

Status: We're reviewing available system data to determine possible causes of this issue. Next update by: Friday, April 23, 2021, at 2:30 AMUTC; PublishedTime=2021-04-23T01:28:14.693Z

Status: We're reviewing network traces provided by affected users to investigate a possible server related issue that may be causing save requests for signatures to fail. Next update by: Friday, April 23, 2021, at 4:30 AMUTC; PublishedTime=2021-04-23T02:15:55.27Z

Status: As we continue our review of provided network traces, we're reaching out to affected users to gather additional traces with more focused system data to assist our investigation into possible causes of impact, including the potential server issue. Next update by: Friday, April 23, 2021, at 6:30 PMUTC; PublishedTime=2021-04-23T03:05:53.81Z

Status: We've gathered additional network trace logs and our initial analysis suggests there may be a server error occurring along the save process, preventing signatures from loading completely and displaying properly in email messages. We're continuing to examine Fiddler traces to better determine why this is happening and how we can best mitigate the issue. Next update by: Friday, April 23, 2021, at 8:30 PMUTC; PublishedTime=2021-04-23T17:44:41Z

Status: Our analysis of Fiddler traces indicates that the request action to retrieve and add signatures to the message may not be completing as expected. We're continuing our investigation to confirm the cause or the problem and develop a mitigation strategy. Next update by: Friday, April 23, 2021, at 10:30 PMUTC; PublishedTime=2021-04-23T20:30:13.263Z

Status: Our investigation has determined that the server logs we're analyzing are incomplete and subsequently unable to provide enough information to conclusively isolate the root cause. Moving forward, we're attempting to reproduce the issue so we can utilize fresh diagnostic data in determining the underlying cause and develop an effective mitigation plan. Next update by: Saturday, April 24, 2021, at 12:30 AMUTC; PublishedTime=2021-04-23T22:18:41.283Z

Status: We're reaching out to affected users to grab additional network trace details to assist in our analysis while we continue our attempts to reproduce the issue in our internal test environment. Next update by: Saturday, April 24, 2021, at 2:30 AMUTC; PublishedTime=2021-04-24T00:20:24.117Z

Status: The provided network trace logs were inconclusive as there weren't any identifiable errors seen. We're reaching out to additional users to provide additional logging for further review. Next update by: Monday, April 26, 2021, at 8:00 PMUTC; PublishedTime=2021-04-24T01:45:21.15Z

Status: We've identified a potential issue within the Outlook on the Web compose signature feature that's preventing successful signature requests, which may be contributing to the impact. We're analyzing the code path responsible for retrieving the signature data to isolate the further issue, and to formulate a mitigation plan. Next update by: Monday, April 26, 2021, at 10:00 PMUTC; PublishedTime=2021-04-26T19:58:42.26Z

Status: We've determined that a recent Outlook on the web update introduced a code issue, resulting in the impact. We've developed and are validating a fix which we expect to be ready for deployment in the next few hours and complete by our next scheduled update on Wednesday, April 28, 2021. Next update by: Wednesday, April 28, 2021, at 1:30 AM UTC; PublishedTime=2021-04-26T21:48:47.147Z

Status: We've successfully validated the fix, which took longer than anticipated. We've starting deploying it to the affected environment and expect the deployment to complete by Wednesday, April 28, 2021 at approximately 7:30 PM UTC. Next update by: Wednesday, April 28, 2021, at 7:30 PMUTC; PublishedTime=2021-04-28T00:26:24.39Z

Final status: We've confirmed that the fix has successfully completed deployment throughout the affected environment and we've verified with affected users that this action has alleviated the problem. Next update by: Wednesday, April 28, 2021, at 7:30 PMUTC; PublishedTime=2021-04-28T00:26:24.39Z

--------------------------------------------------------------------------------------------------
Title Users may be unable to perform searches within Word on the web documents
ID SP251691
Service SharePoint Online
Feature SharePoint Features
Status Service restored
Impact Users may have been unable to perform searches within Word on the web documents.
Severity Sev2
Start time 2021-04-14T22:30:00Z
End time 2021-04-20T18:15:00Z
Last update 2021-04-20T22:17:47.247Z
More info
Root cause An Office on the web update resulted in a build compatibility problem with the search-dependent UI service, resulting in impact. Next steps: - We're reviewing our update testing and verification processes to better identify similar build compatibility issues between dependent services and prevent future impact. This is the final update for the event.; PublishedTime=2021-04-20T22:17:47.247Z

SP251691 Details

Status: We're analyzing system telemetry to identify the next troubleshooting steps. Next update by: Tuesday, April 20, 2021, at 3:30 PM UTC; PublishedTime=2021-04-20T13:25:48.87Z

Status: We're continuing to analyze system telemetry as well as reproducing the issue within our environment to gather additional logs to determine the root cause of the issue. Next update by: Thursday, April 22, 2021, at 5:00 PMUTC; PublishedTime=2021-04-20T15:37:27.097Z

Final status: Our analysis of logs collected while reproducing the issue in our test environment determined that an Office on the web update resulted in a build compatibility problem with the search-dependent UI service, resulting in impact. We deployed a fix to the affected environment and confirmed within our testing environment that the issue is resolved. Next update by: Thursday, April 22, 2021, at 5:00 PMUTC; PublishedTime=2021-04-20T15:37:27.097Z

--------------------------------------------------------------------------------------------------
Title Users attempting to launch apps via SharePoint or Microsoft 365 connectors may have experienced failures
ID MM250639
Service Power Apps in Microsoft 365
Feature Service and web access issues
Status Service restored
Impact Users attempting to launch apps via SharePoint or Microsoft 365 connectors may have experienced failures.
Severity
Start time 2021-04-14T17:45:00Z
End time 2021-04-14T19:18:00Z
Last update 2021-04-14T23:23:09.44Z
More info Users may have seen a red banner with an error message stating "Something went wrong." Final status: We've confirmed that a portion of infrastructure facilitating the use of Microsoft Power Apps via SharePoint or Microsoft 365 connectors was experiencing abnormally high load, resulting in impact. We've rerouted traffic through alternate infrastructure and confirmed via telemetry that impact is mitigated. Scope of impact: Your organization was affected by this event, and all users may have experienced impact. Start time: Wednesday, April 14, 2021, at 5:45 PM'UTC End time: Wednesday, April 14, 2021, at 7:18 PM'UTC Root cause: A portion of infrastructure facilitating the use of Microsoft Power Apps via SharePoint or Microsoft 365 connectors was experiencing abnormally high load, resulting in impact. Next steps: - We're continuing to monitor the affected infrastructure to identify why it experienced impact to better prevent similar future occurrences. This is the final update for the event.; PublishedTime=2021-04-14T20:11:07.03Z
Root cause A portion of infrastructure facilitating the use of Microsoft Power Apps via SharePoint or Microsoft 365 connectors was experiencing abnormally high load, resulting in impact. Next steps: - We're continuing to monitor the affected infrastructure to identify why it experienced impact to better prevent similar future occurrences. This is the final update for the event.; PublishedTime=2021-04-14T20:11:07.03Z

MM250639 Details

Status: We've investigated a problem with the Microsoft Power Apps service in which users attempting to launch apps via SharePoint or Microsoft 365 connectors may have experienced failures, and we've confirmed that service is now restored. We'll provide additional information in a closure summary within the 'History' section of the 'Service health' dashboard shortly.

Final status: We've confirmed that a portion of infrastructure facilitating the use of Microsoft Power Apps via SharePoint or Microsoft 365 connectors was experiencing abnormally high load, resulting in impact. We've rerouted traffic through alternate infrastructure and confirmed via telemetry that impact is mitigated.

--------------------------------------------------------------------------------------------------
Title Microsoft Forms service inaccessible or delayed
ID FM250605
Service Microsoft Forms
Feature Service
Status Service restored
Impact Users may have been unable to access the Microsoft Forms service.
Severity
Start time 2021-04-14T12:00:00Z
End time 2021-04-14T13:30:00Z
Last update 2021-04-14T15:10:50.523Z
More info Those that could access the service may have experienced loading delays when attempting to navigate the service. Final status: We identified that an unexpected spike in user traffic caused access issues and navigation delays. We've run service optimization commands to improve service functionality and remediate impact. Scope of impact: This issue may have impacted any user attempting to access the service. Start time: Wednesday, April 14, 2021, at 12:00 PM UTC End time: Wednesday, April 14, 2021, at 1:30 PM UTC Root cause: An unexpected spike in user traffic caused access issues and navigation delays to the service. Next steps: - We're analyzing performance data and trends on the affected systems to help prevent this problem from happening again. This is the final update for the event.; PublishedTime=2021-04-14T13:59:33.24Z
Root cause An unexpected spike in user traffic caused access issues and navigation delays to the service. Next steps: - We're analyzing performance data and trends on the affected systems to help prevent this problem from happening again. This is the final update for the event.; PublishedTime=2021-04-14T13:59:33.24Z

FM250605 Details

Status: We're gathering diagnostic data for analysis. Next update by: Wednesday, April 14, 2021, at 2:00 PM UTC; PublishedTime=2021-04-14T13:02:31.61Z

Final status: We identified that an unexpected spike in user traffic caused access issues and navigation delays. We've run service optimization commands to improve service functionality and remediate impact. Next update by: Wednesday, April 14, 2021, at 2:00 PM UTC; PublishedTime=2021-04-14T13:02:31.61Z

--------------------------------------------------------------------------------------------------
Title Some users may have experienced errors or performance issues when using Microsoft Forms
ID FM250432
Service Microsoft Forms
Feature Service
Status Service restored
Impact Users may have experienced errors or performance issues within the Microsoft Forms service.
Severity
Start time 2021-04-13T11:55:00Z
End time 2021-04-13T13:10:00Z
Last update 2021-04-13T13:35:58.017Z
More info
Root cause A section of infrastructure, responsible for facilitating user requests to the service, was performing below acceptable performance thresholds. Next steps: - We're analyzing performance data and trends on the affected systems to help prevent this problem from happening again. This is the final update for the event.; PublishedTime=2021-04-13T13:32:13.533Z

FM250432 Details

Status: We're investigating a potential issue with the Microsoft Forms service. We'll provide an update within 30 minutes. ; PublishedTime=2021-04-13T13:03:30.447

Final status: We've identified that a section of infrastructure responsible for facilitating user requests to the service was performing below acceptable performance thresholds. We've ran commands to optimize traffic and performance on the affected infrastructure and can confirm after monitoring that impact has been mitigated.

--------------------------------------------------------------------------------------------------
Title Some admins may see an inaccurate "Domain expiring soon" message in the Exchange Admin Center (EAC)
ID MO250594
Service Microsoft 365 suite
Feature Administration
Status Service restored
Impact Admins may have seen an inaccurate "Domain expiring soon" message in the Exchange Admin Center.
Severity
Start time 2021-04-11T23:53:00Z
End time 2021-04-14T16:52:00Z
Last update 2021-04-14T17:34:19.73Z
More info Admins may have also received multiple emails with the same message. We've confirmed that there was no operational impact associated with these alerts and emails; these were being generated unintentionally and could have been ignored. Final status: We've verified that the change has replicated throughout environment and that impact has been mitigated. Scope of impact: This issue may have affected any tenant administrator viewing the EAC and through email. Start time: Sunday, April 11, 2021, at 11:53 PM UTC End time: Wednesday, April 14, 2021, at 4:52 PM UTC Root cause: The back-end database components used to query domain expiry dates contained incorrect data, which resulted in impact. Next steps: - We're reviewing the back-end database components and data sources to identify why the data inadvertently included incorrect domain expiry dates. - Once the root cause has been identified, we'll develop fixes to prevent the issue from reoccurring. This is the final update for the event.; PublishedTime=2021-04-14T17:34:19.73Z
Root cause The back-end database components used to query domain expiry dates contained incorrect data, which resulted in impact. Next steps: - We're reviewing the back-end database components and data sources to identify why the data inadvertently included incorrect domain expiry dates. - Once the root cause has been identified, we'll develop fixes to prevent the issue from reoccurring. This is the final update for the event.; PublishedTime=2021-04-14T17:34:19.73Z

MO250594 Details

Status: Early reports suggest that there is no impact associated with these notifications. The issue appears to be that these notification are being generated unintentionally. We're reviewing details provided in the support case to validate this. Next update by: Wednesday, April 14, 2021, at 1:00 PM UTC; PublishedTime=2021-04-14T10:58:15.407Z

Status: We've confirmed that there is no operational impact associated with these alerts and emails; these are being generated unintentionally and can be ignored. We're running a script to disable the underlying feature which is generating the alerts and emails to mitigate impact, whilst in parallel, we continue to investigate and address the underlying root cause. Next update by: Wednesday, April 14, 2021, at 3:00 PM UTC; PublishedTime=2021-04-14T12:51:00.797Z

Status: We've disabled the User Interface (UI) experience which was showing the domain expiry alert in the EAC. Additionally, we're reviewing options to disable the feature which generates the emails. Analysis into the root cause has identified that the back-end database, used to query domain expiry dates, has incorrect data and we continue to investigate the cause of this. Next update by: Wednesday, April 14, 2021, at 5:00 PM UTC; PublishedTime=2021-04-14T14:54:46.16Z

Status: We've applied a configuration change to disable the feature which generates the email. We expect this change will mitigate impact, and that it will replicate throughout the environment within the next hour. Next update by: Wednesday, April 14, 2021, at 7:00 PM UTC; PublishedTime=2021-04-14T16:47:49.17Z

Final status: We've verified that the change has replicated throughout environment and that impact has been mitigated. Next update by: Wednesday, April 14, 2021, at 7:00 PM UTC; PublishedTime=2021-04-14T16:47:49.17Z

--------------------------------------------------------------------------------------------------
Title Some users experiencing intermittent Zero-hour auto purge (ZAP) failures in their Exchange Online mailboxes
ID EX249720
Service Exchange Online
Feature Backend Data Processing
Status Service restored
Impact Users may have experienced intermittent ZAP failures in their Exchange Online mailboxes.
Severity
Start time 2021-04-09T09:30:00Z
End time 2021-04-09T18:00:00Z
Last update 2021-04-09T18:38:39.113Z
More info Affected users may have experienced this impact in the form of an increased presence of spam messages in their mailboxes. Final status: Our investigation determined a recent directory configuration change within a portion of the antispam infrastructure created a version compatibility issue between directory and ZAP, which caused ZAP to stop functioning as expected. We've applied an additional configuration change to resolve the issue, and our telemetry confirms that ZAP functionality has been restored. Scope of impact: Some users served through the affected infrastructure may have experienced intermittent impact. Start time: Friday, April 9, 2021, at 9:30 AM UTC End time: Friday, April 9, 2021, at 6:00 PM UTC Root cause: A recent directory configuration change within a portion of the antispam infrastructure created a version compatibility issue between directory and ZAP, which caused ZAP to stop functioning as expected. Next steps: - We're reviewing our directory configuration change procedures to better understand why this particular change caused impact, and to find ways to prevent this problem from happening again. This is the final update for the event.; PublishedTime=2021-04-09T18:32:00.283Z
Root cause A recent directory configuration change within a portion of the antispam infrastructure created a version compatibility issue between directory and ZAP, which caused ZAP to stop functioning as expected. Next steps: - We're reviewing our directory configuration change procedures to better understand why this particular change caused impact, and to find ways to prevent this problem from happening again. This is the final update for the event.; PublishedTime=2021-04-09T18:32:00.283Z

EX249720 Details

Status: We're reviewing the service telemetry corresponding with the ZAP feature in the Exchange Online service to isolate the source of the issue and develop a mitigation plan. Next update by: Friday, April 9, 2021, at 7:30 PMUTC; PublishedTime=2021-04-09T17:06:47.047Z

Final status: Our investigation determined a recent directory configuration change within a portion of the antispam infrastructure created a version compatibility issue between directory and ZAP, which caused ZAP to stop functioning as expected. We've applied an additional configuration change to resolve the issue, and our telemetry confirms that ZAP functionality has been restored. Next update by: Friday, April 9, 2021, at 7:30 PMUTC; PublishedTime=2021-04-09T17:06:47.047Z

--------------------------------------------------------------------------------------------------
Title Unable to access multiple Microsoft 365 services.
ID MO249573
Service Microsoft 365 suite
Feature Portal
Status Post-incident report published
Impact Users were unable to access multiple Microsoft 365 services such as Teams, SharePoint Online, Exchange, or Skype.
Severity Sev2
Start time 2021-04-08T18:40:00Z
End time 2021-04-08T18:59:00Z
Last update 2021-04-19T17:33:36.687Z
More info During the impact period, customers may have experienced intermittent issues connecting to Microsoft 365 services. Final status: We determined that a network infrastructure issue occurred impacting the flow of traffic into and out of Argentina and surrounding areas. Our automated recovery systems re-routed traffic to alternate infrastructure and after a period of monitoring, we've confirmed service availability has been restored. Scope of impact: Impact was specific to user traffic into and out of Argentina and the surrounding region. Start time: Thursday, April 8, 2021, at 6:40 PM UTC End time: Thursday, April 8, 2021, at 6:59 PM UTC Root cause: A physical network infrastructure issue prevented traffic from processing as expected. Next steps: - For a more comprehensive list of next steps and actions, please refer to the Post Incident Review document. We're providing this post incident report due to recent network-related issues which happened within Argentina and the surrounding region. This document aims to provide further context on the impact users may have experienced, the root causes, and the steps Microsoft is taking to prevent these issues from happening again.; PublishedTime=2021-04-08T19:52:35.647Z
Root cause A physical network infrastructure issue prevented traffic from processing as expected. Next steps: - For a more comprehensive list of next steps and actions, please refer to the Post Incident Review document. We're providing this post incident report due to recent network-related issues which happened within Argentina and the surrounding region. This document aims to provide further context on the impact users may have experienced, the root causes, and the steps Microsoft is taking to prevent these issues from happening again.; PublishedTime=2021-04-08T19:52:35.647Z

MO249573 Details

Status: We're investigating a potential issue and checking for impact to your organization. We'll provide an update within 30 minutes. Next update by: Thursday, April 8, 2021, at 7:45 PM UTC; PublishedTime=2021-04-08T19:09:58.68Z

Status: We've identified some service availability issues with a portion of infrastructure located in Argentina. A portion of networking infrastructure experienced a problem and our automated recovery systems remediated impact. We're working to confirm service availability has been restored. Next update by: Thursday, April 8, 2021, at 9:00 PM UTC.; PublishedTime=2021-04-08T19:35:58.457Z

Final status: We determined that a network infrastructure issue occurred impacting the flow of traffic into and out of Argentina and surrounding areas. Our automated recovery systems re-routed traffic to alternate infrastructure and after a period of monitoring, we've confirmed service availability has been restored. Next update by: Thursday, April 8, 2021, at 9:00 PM UTC.; PublishedTime=2021-04-08T19:35:58.457Z

--------------------------------------------------------------------------------------------------
Title Users can't change Microsoft Teams meeting participants' roles via the web client
ID TM249540
Service Microsoft Teams
Feature Teams Components
Status False positive
Impact Users may have been unable to change Microsoft Teams meeting participants' roles via the web client.
Severity
Start time 2021-04-08T13:43:00Z
End time 2021-04-08T14:45:00Z
Last update 2021-04-08T15:34:59.083Z
More info Users were unable to make participants attendees or presenters. Final status: We determined that a portion of infrastructure responsible for the affected functionality stopped operating as intended, resulting in impact. While we were in the process of analyzing system logs to determine a remediation plan, system monitoring indicated that the service had returned to normal health. We've confirmed that the problem is no longer occurring, and we'll continue to monitor the service to ensure that the problem does not happen again. Scope of impact: Your organization was affected by this event, and any user may have been impacted. Start time: Thursday, April 8, 2021, at 1:43 PM'UTC End time: Thursday, April 8, 2021, at 2:45 PM'UTC Root cause: A portion of infrastructure responsible for the affected functionality stopped operating as intended, resulting in impact. Next steps: We're further reviewing what caused the Microsoft Teams meeting participants roles to stop working to help prevent this problem from happening again. This is the final update for the event.; PublishedTime=2021-04-08T15:34:59.083Z
Root cause A portion of infrastructure responsible for the affected functionality stopped operating as intended, resulting in impact. Next steps: We're further reviewing what caused the Microsoft Teams meeting participants roles to stop working to help prevent this problem from happening again. This is the final update for the event.; PublishedTime=2021-04-08T15:34:59.083Z

TM249540 Details

Status: We're investigating a potential issue and checking for impact to your organization. We'll provide an update within 30 minutes.; PublishedTime=2021-04-08T15:06:50.513

Final status: We determined that a portion of infrastructure responsible for the affected functionality stopped operating as intended, resulting in impact. While we were in the process of analyzing system logs to determine a remediation plan, system monitoring indicated that the service had returned to normal health. We've confirmed that the problem is no longer occurring, and we'll continue to monitor the service to ensure that the problem does not happen again.

--------------------------------------------------------------------------------------------------
Title Admins unable to export or purge mailboxes due to unexpected limitations in the Security and Compliance Center (SCC)
ID EX251482
Service Exchange Online
Feature E-Mail and calendar access
Status Service restored
Impact Admins were unable to export or purge mailboxes due to unexpected limitations in the SCC.
Severity Sev2
Start time 2021-04-08T00:00:00Z
End time 2021-04-19T22:06:38.79Z
Last update 2021-04-19T22:06:38.79Z
More info Admins received an error containing diagnostic information when attempting to perform searches containing 100,000 mailboxes or more, or performing purges on 50,000 mailboxes or more. Final status: We've investigated a problem with the eDiscovery service in which admins were unable to perform eDiscovery searches of 100,000 mailboxes or more, or purges containing 50,000 mailboxes or more. We've determined that this was caused by a feature update that was incorrectly deployed. We've disabled this feature update and confirmed that these size limitations have been removed to mitigate the issue. Scope of impact: Any admin attempting to perform eDiscovery searches of 100,000 mailboxes or more, or purges containing 50,000 mailboxes or more may have experienced impact. Start time: Thursday, April 8, 2021, at 12:00 AM UTC End time: Saturday, April 17, 2021, at 2:30 AM UTC Root cause: An incorrectly deployed feature update caused a problem with the eDiscovery service in which admins were unable to perform eDiscovery searches of 100,000 mailboxes or more, or purges containing 50,000 mailboxes or more. Next steps: - We're reviewing our deployment and provisioning procedures to help prevent similar eDiscovery search issues in the future. This is the final update for the event.; PublishedTime=2021-04-19T22:06:38.79Z
Root cause An incorrectly deployed feature update caused a problem with the eDiscovery service in which admins were unable to perform eDiscovery searches of 100,000 mailboxes or more, or purges containing 50,000 mailboxes or more. Next steps: - We're reviewing our deployment and provisioning procedures to help prevent similar eDiscovery search issues in the future. This is the final update for the event.; PublishedTime=2021-04-19T22:06:38.79Z

EX251482 Details

Final status: We've investigated a problem with the eDiscovery service in which admins were unable to perform eDiscovery searches of 100,000 mailboxes or more, or purges containing 50,000 mailboxes or more. We've determined that this was caused by a feature update that was incorrectly deployed. We've disabled this feature update and confirmed that these size limitations have been removed to mitigate the issue.

--------------------------------------------------------------------------------------------------
Title Admins unable to export or purge mailboxes due to unexpected limitations in the Security and Compliance Center (SCC)
ID EX251482
Service Exchange Online
Feature E-Mail and calendar access
Status Service restored
Impact Admins were unable to export or purge mailboxes due to unexpected limitations in the SCC.
Severity Sev2
Start time 2021-04-08T00:00:00Z
End time 2021-04-19T22:06:38.79Z
Last update 2021-04-19T22:06:38.79Z
More info Admins received an error containing diagnostic information when attempting to perform searches containing 100,000 mailboxes or more, or performing purges on 50,000 mailboxes or more. Final status: We've investigated a problem with the eDiscovery service in which admins were unable to perform eDiscovery searches of 100,000 mailboxes or more, or purges containing 50,000 mailboxes or more. We've determined that this was caused by a feature update that was incorrectly deployed. We've disabled this feature update and confirmed that these size limitations have been removed to mitigate the issue. Scope of impact: Any admin attempting to perform eDiscovery searches of 100,000 mailboxes or more, or purges containing 50,000 mailboxes or more may have experienced impact. Start time: Thursday, April 8, 2021, at 12:00 AM UTC End time: Saturday, April 17, 2021, at 2:30 AM UTC Root cause: An incorrectly deployed feature update caused a problem with the eDiscovery service in which admins were unable to perform eDiscovery searches of 100,000 mailboxes or more, or purges containing 50,000 mailboxes or more. Next steps: - We're reviewing our deployment and provisioning procedures to help prevent similar eDiscovery search issues in the future. This is the final update for the event.; PublishedTime=2021-04-19T22:06:38.79Z
Root cause An incorrectly deployed feature update caused a problem with the eDiscovery service in which admins were unable to perform eDiscovery searches of 100,000 mailboxes or more, or purges containing 50,000 mailboxes or more. Next steps: - We're reviewing our deployment and provisioning procedures to help prevent similar eDiscovery search issues in the future. This is the final update for the event.; PublishedTime=2021-04-19T22:06:38.79Z

EX251482 Details

Final status: We've investigated a problem with the eDiscovery service in which admins were unable to perform eDiscovery searches of 100,000 mailboxes or more, or purges containing 50,000 mailboxes or more. We've determined that this was caused by a feature update that was incorrectly deployed. We've disabled this feature update and confirmed that these size limitations have been removed to mitigate the issue.

--------------------------------------------------------------------------------------------------
Title Admins are unable to migrate mailboxes from Exchange 2010 to Exchange Online
ID EX250992
Service Exchange Online
Feature Management and Provisioning
Status Service restored
Impact Admins were unable to migrate mailboxes from Exchange 2010 to Exchange Online.
Severity Sev2
Start time 2021-04-07T09:17:00Z
End time 2021-04-28T19:00:00Z
Last update 2021-04-28T21:12:08.65Z
More info Only migrations from Exchange 2010 and earlier to build 4042 of Exchange Online would have experienced impact. Final status: We've confirmed that the fix has completed deploying to all impacted infrastructure, and we've validated mitigation with affected admins. Scope of impact: Any admin migrating users from the Exchange 2010 environment or earlier to Exchange Online, build 4042, experienced impact. Start time: Wednesday, April 7, 2021, at 9:17 AM'UTC End time: Wednesday, April 28, 2021, at 7:00 PM'UTC Root cause: A recent code change in Exchange Online had caused an incompatibility with the Exchange 2010 infrastructure, resulting in an authentication issue that disrupted admins' ability to migrate mailboxes from Exchange 2010 to Exchange Online, build 4042. Next steps: - We're continuing to develop compatibility testing for cross-generation migrations to isolate and resolve similar issues in the future. This is the final update for the event.; PublishedTime=2021-04-28T21:12:08.65Z
Root cause A recent code change in Exchange Online had caused an incompatibility with the Exchange 2010 infrastructure, resulting in an authentication issue that disrupted admins' ability to migrate mailboxes from Exchange 2010 to Exchange Online, build 4042. Next steps: - We're continuing to develop compatibility testing for cross-generation migrations to isolate and resolve similar issues in the future. This is the final update for the event.; PublishedTime=2021-04-28T21:12:08.65Z

EX250992 Details

Status: We've investigated an issue where admins are unable to migrate mailboxes to Exchange Online and our initial findings indicate that a recent code change to the service may have inadvertently introduced an authentication issue. We're working to validate that hypothesis and explore potential mitigation actions. Next update by: Monday, April 19, 2021, at 8:30 PMUTC; PublishedTime=2021-04-17T03:03:37.26Z

Status: We've confirmed that a recent code change in Exchange Online has caused an incompatibility with the Exchange 2010 infrastructure, resulting in an authentication issue that disrupts admins' ability to migrate mailboxes from Exchange 2010 to Exchange Online, build 4042. We've created and are deploying a code fix that will remediate impact. We're monitoring the deployment process and we'll provide a timeline for completion as soon as one becomes available. Next update by: Wednesday, April 21, 2021, at 9:00 PMUTC; PublishedTime=2021-04-19T19:28:40.413Z

Status: The fix has reached 40 percent saturation and we're continuing to monitor as the fix deploys to the impacted infrastructure. Admins will see relief as the fix progresses. Next update by: Friday, April 23, 2021, at 9:00 PMUTC; PublishedTime=2021-04-21T20:14:39.007Z

Status: Our telemetry indicates the fix has reached approximately 60 percent saturation of the impacted infrastructure. We expect to have mitigation by the next scheduled update. Next update by: Monday, April 26, 2021, at 9:00 PMUTC; PublishedTime=2021-04-23T20:08:53.893Z

Status: Our telemetry indicates the fix has now reached approximately 86 percent saturation of the impacted infrastructure. We're continuing to monitor the deployment progress, which we are now expecting to complete by Wednesday, April 28, 2021, by 9:00 PM UTC. Next update by: Wednesday, April 28, 2021, at 9:00 PMUTC; PublishedTime=2021-04-26T18:23:15.887Z

Final status: We've confirmed that the fix has completed deploying to all impacted infrastructure, and we've validated mitigation with affected admins. Next update by: Wednesday, April 28, 2021, at 9:00 PMUTC; PublishedTime=2021-04-26T18:23:15.887Z

--------------------------------------------------------------------------------------------------
Title Some users had intermittent failures and latency in multiple device services and scenarios within Microsoft Intune
ID IT249320
Service Microsoft Intune
Feature Microsoft Intune
Status Service restored
Impact Users may experience intermittent failures and latency in multiple device services and scenarios in Microsoft Intune.
Severity
Start time 2021-04-07T00:40:00Z
End time 2021-04-08T05:55:00Z
Last update 2021-04-08T07:40:31.98Z
More info Impacted scenarios included device enrollment, policy check, and compliance, among other high-use device services and scenarios. Final status: We have verified the deployment has saturated across the affected environment successfully and the issue is now resolved. Scope of impact: Some users served through the affected infrastructure may have experienced intermittent impact. Start time: Wednesday, April 7, 2021, at 12:40 AM UTC End time: Thursday, April 8, 2021, at 5:55 AM UTC Root cause: A section of load balancing infrastructure was experiencing a data processing issue, which was revealed as a result of a partially completed standard service update that was being exacerbated by a peak in user traffic, leading to impact. Next steps: - We're analyzing performance data and trends on the affected systems to help prevent this problem from happening again. This is the final update for the event.; PublishedTime=2021-04-08T07:40:09.903Z
Root cause A section of load balancing infrastructure was experiencing a data processing issue, which was revealed as a result of a partially completed standard service update that was being exacerbated by a peak in user traffic, leading to impact. Next steps: - We're analyzing performance data and trends on the affected systems to help prevent this problem from happening again. This is the final update for the event.; PublishedTime=2021-04-08T07:40:09.903Z

IT249320 Details

Status: We're investigating a potential issue and checking for impact. We'll provide an update within 30 minutes.; PublishedTime=2021-04-07T15:49:59.413

Status: We're analyzing the service telemetry and recently deployed changes correlated with the infrastructure which supports the affected device services and scenarios to help us isolate the source of the issue, better understand the scope of the impact, and identify our next steps. Next update by: Wednesday, April 7, 2021, at 5:30 PMUTC; PublishedTime=2021-04-07T16:12:55.697Z

Status: Our ongoing analysis of the service telemetry isolated a data processing issue within a portion of load balancing infrastructure, which was revealed as a result of a partially completed standard service update that is being exacerbated by a peak in user traffic. This data processing issue is leading to intermittent drops in service availability and latency in the impacted device services and scenarios as automated recovery processes continually attempt to rebalance the load. We're deploying additional load balancing infrastructure to the affected environment, which will provide relief as we continue to investigate the underlying cause of this issue so that we can prevent further impact. Next update by: Wednesday, April 7, 2021, at 7:30 PM UTC; PublishedTime=2021-04-07T17:23:00.94Z

Status: Our deployment of additional load balancing infrastructure is providing relief, and we're actively monitoring the service telemetry to determine whether or not any additional mitigation steps are required to ensure all instances of data processing issues are resolved. Next update by: Wednesday, April 7, 2021, at 9:30 PMUTC; PublishedTime=2021-04-07T19:10:10.657Z

Status: We've determined that a configuration change corresponding with the automated recovery mechanisms for the traffic management infrastructure is required to ensure all remaining instances of data processing issues are resolved. We expect this additional step will fully mitigate the impact, and we'll continue to monitor the environment through its completion. Next update by: Wednesday, April 7, 2021, at 11:30 PMUTC; PublishedTime=2021-04-07T21:10:17.73Z

Status: Our telemetry indicates the user impact is now mitigated; however, we'll continue to monitor the environment through the completion of the service update to ensure the impact doesn't reoccur. Next update by: Thursday, April 8, 2021, at 1:30 AMUTC; PublishedTime=2021-04-07T22:22:04.687Z

Status: We've confirmed with telemetry that the issue is resolved and users are no longer experiencing impact. Additionally, we're deploying another configuration change to the traffic management infrastructure. Though we don't expect this deployment to cause any additional user impact, we're continuing to monitor telemetry until the deployment is complete to ensure the issue doesn't resurface. Next update by: Thursday, April 8, 2021, at 5:30 AMUTC; PublishedTime=2021-04-07T23:58:33.58Z

Status: We're continuing to monitor as the deployment progresses, and we expect it to complete by our next communications update. Additionally, telemetry has proven that there has been no additional impact with this deployment; however, we're continuing to monitor to ensure impact doesn't reoccur. Next update by: Thursday, April 8, 2021, at 9:00 AMUTC; PublishedTime=2021-04-08T03:47:10.267Z

Final status: We have verified the deployment has saturated across the affected environment successfully and the issue is now resolved. Next update by: Thursday, April 8, 2021, at 9:00 AMUTC; PublishedTime=2021-04-08T03:47:10.267Z

--------------------------------------------------------------------------------------------------
Title Admins may see Advanced Threat Protection (ATP) email attachment scans aren't occurring prior to delivery to users
ID EX249554
Service Exchange Online
Feature Networking Issues
Status Service restored
Impact Admins may have seen ATP email attachment scans weren't occurring prior to delivery to users.
Severity
Start time 2021-04-07T00:27:00Z
End time 2021-04-08T18:48:00Z
Last update 2021-04-08T19:45:06.023Z
More info Malware detection functionality remained unaffected during this event. Final status: Our investigation determined that a configuration change caused some email attachment scans to be bypassed by ATP, resulting in impact. We've since reverted the configuration and have verified through monitoring that mitigation has been achieved. Scope of impact: Any admin may have experienced impact. Start time: Wednesday, April 7, 2021, at 12:27 AM'UTC End time: Thursday, April 8, 2021, at 6:48 PM'UTC Root cause: A configuration change caused some email attachment scans to be bypassed by ATP, resulting in impact. Next steps: - We're reviewing our configuration change procedures to ensure future changes don't cause similar problems. This is the final update for the event.; PublishedTime=2021-04-08T19:45:06.023Z
Root cause A configuration change caused some email attachment scans to be bypassed by ATP, resulting in impact. Next steps: - We're reviewing our configuration change procedures to ensure future changes don't cause similar problems. This is the final update for the event.; PublishedTime=2021-04-08T19:45:06.023Z

EX249554 Details

Status: We're investigating a potential issue and checking for impact to your organization. We'll provide an update within 30 minutes.

Status: We've determined a recent deployment introduced a code change which may have potentially contributed to impact. Moving forward, we're analyzing message trace logs and service telemetry to better understand why this happened as we determine our next troubleshooting steps. Next update by: Thursday, April 8, 2021, at 6:30 PM UTC; PublishedTime=2021-04-08T17:17:15.243Z

Status: We've completed analyzing service telemetry from recent deployments and have determined that the aforementioned code change did not contribute to impact. We're continuing to analyze message trace logs to identify why ATP is not scanning email attachments as expected. Next update by: Thursday, April 8, 2021, at 8:30 PMUTC; PublishedTime=2021-04-08T18:27:47.46Z

Final status: Our investigation determined that a configuration change caused some email attachment scans to be bypassed by ATP, resulting in impact. We've since reverted the configuration and have verified through monitoring that mitigation has been achieved. Next update by: Thursday, April 8, 2021, at 8:30 PMUTC; PublishedTime=2021-04-08T18:27:47.46Z

--------------------------------------------------------------------------------------------------
Title Users of the modern experience are unable to share OneDrive for Business content with external users
ID OD250334
Service OneDrive for Business
Feature OneDrive for Business
Status Service restored
Impact Users of the modern experience were unable to share OneDrive for Business content with external users.
Severity
Start time 2021-04-05T16:00:00Z
End time 2021-04-14T04:00:00Z
Last update 2021-04-14T16:35:22.98Z
More info Users were experiencing this issue when attempting to share OneDrive for Business content with external users from OneDrive Web, the OneDrive sync client, or File Explorer. We recommended users who could switch to the classic experience on OneDrive Web (via browser) use that as an alternative for sharing OneDrive for Business content with approved external users. Details on how to switch to the classic experience could have been found at this page: ht tps://support.microsoft.com/en-us/office/switch-the-default-experience-for-lists-and-libr aries-from-new-or-classic-66dac24b-4177-4775-bf50-3d267318caa9 Final status: We've confirmed with affected users that this issue is resolved. Scope of impact: This issue may have potentially affected any user of the modern experience that attempted to share OneDrive for Business content with external users. Start time: Monday, April 5, 2021, at 4:00 PM UTC End time: Wednesday, April 14, 2021, at 4:00 AM UTC Root cause: A recent service update designed to improve sharing with external users inadvertently introduced a code issue which was preventing users of the OneDrive for Business modern experience from sharing content with external users. Next steps: - We're reviewing our update and validation procedures to better understand why the code issue was missed and to prevent issues like this from occurring in the future. This is the final update for the event.; PublishedTime=2021-04-14T16:35:22.98Z
Root cause A recent service update designed to improve sharing with external users inadvertently introduced a code issue which was preventing users of the OneDrive for Business modern experience from sharing content with external users. Next steps: - We're reviewing our update and validation procedures to better understand why the code issue was missed and to prevent issues like this from occurring in the future. This is the final update for the event.; PublishedTime=2021-04-14T16:35:22.98Z

OD250334 Details

Status: We've identified a code issue that was introduced by a recent service update, which is preventing users from sharing OneDrive for Business content with external users. We're developing a code fix that's designed to correct the code issue and remediate impact. We're aiming to provide a timeline for the fix deployment and remediation of impact during our next scheduled update. Next update by: Tuesday, April 13, 2021, at 6:00 PMUTC; PublishedTime=2021-04-12T22:45:53.957Z

Status: We're finalizing the development of our code fix which is designed to correct the code issue and remediate impact. We're planning for the deployment of our fix to begin within the next seven hours, and we'll provide an estimated fix deployment timeline once it has been confirmed. Next update by: Wednesday, April 14, 2021, at 8:00 PM UTC; PublishedTime=2021-04-13T17:05:41.633Z

Status: Development of the code fix is taking slightly longer than expected to complete. We anticipate that it will take approximately two hours to finalize the fix, at which point we'll begin our deployment procedures to the affected environment. Next update by: Wednesday, April 14, 2021, at 4:00 AM UTC; PublishedTime=2021-04-14T00:11:17.52Z

Status: We've finalized the fix and deployment to the affected environments is underway. We anticipate the deployments will complete with an hour. Next update by: Wednesday, April 14, 2021, at 5:00 AM UTC; PublishedTime=2021-04-14T03:58:10.743Z

Status: We've confirmed the deployment has completed to the affected environments and our internal testing shows that the issue is no longer occurring. We're reaching out to customers to verify that the issue is mitigated. Next update by: Wednesday, April 14, 2021, at 11:00 AM UTC; PublishedTime=2021-04-14T04:42:50.34Z

Status: We're continuing to reach out to customers to verify that the issue is mitigated. Next update by: Wednesday, April 14, 2021, at 6:00 PM UTC; PublishedTime=2021-04-14T10:46:04.39Z

Final status: We've confirmed with affected users that this issue is resolved. Next update by: Wednesday, April 14, 2021, at 6:00 PM UTC; PublishedTime=2021-04-14T10:46:04.39Z

--------------------------------------------------------------------------------------------------
Title Users of the modern experience are unable to share OneDrive for Business content with external users
ID OD250334
Service OneDrive for Business
Feature OneDrive for Business
Status Service restored
Impact Users of the modern experience were unable to share OneDrive for Business content with external users.
Severity
Start time 2021-04-05T16:00:00Z
End time 2021-04-14T04:00:00Z
Last update 2021-04-14T16:35:22.98Z
More info Users were experiencing this issue when attempting to share OneDrive for Business content with external users from OneDrive Web, the OneDrive sync client, or File Explorer. We recommended users who could switch to the classic experience on OneDrive Web (via browser) use that as an alternative for sharing OneDrive for Business content with approved external users. Details on how to switch to the classic experience could have been found at this page: ht tps://support.microsoft.com/en-us/office/switch-the-default-experience-for-lists-and-libr aries-from-new-or-classic-66dac24b-4177-4775-bf50-3d267318caa9 Final status: We've confirmed with affected users that this issue is resolved. Scope of impact: This issue may have potentially affected any user of the modern experience that attempted to share OneDrive for Business content with external users. Start time: Monday, April 5, 2021, at 4:00 PM UTC End time: Wednesday, April 14, 2021, at 4:00 AM UTC Root cause: A recent service update designed to improve sharing with external users inadvertently introduced a code issue which was preventing users of the OneDrive for Business modern experience from sharing content with external users. Next steps: - We're reviewing our update and validation procedures to better understand why the code issue was missed and to prevent issues like this from occurring in the future. This is the final update for the event.; PublishedTime=2021-04-14T16:35:22.98Z
Root cause A recent service update designed to improve sharing with external users inadvertently introduced a code issue which was preventing users of the OneDrive for Business modern experience from sharing content with external users. Next steps: - We're reviewing our update and validation procedures to better understand why the code issue was missed and to prevent issues like this from occurring in the future. This is the final update for the event.; PublishedTime=2021-04-14T16:35:22.98Z

OD250334 Details

Status: We've identified a code issue that was introduced by a recent service update, which is preventing users from sharing OneDrive for Business content with external users. We're developing a code fix that's designed to correct the code issue and remediate impact. We're aiming to provide a timeline for the fix deployment and remediation of impact during our next scheduled update. Next update by: Tuesday, April 13, 2021, at 6:00 PMUTC; PublishedTime=2021-04-12T22:45:53.957Z

Status: We're finalizing the development of our code fix which is designed to correct the code issue and remediate impact. We're planning for the deployment of our fix to begin within the next seven hours, and we'll provide an estimated fix deployment timeline once it has been confirmed. Next update by: Wednesday, April 14, 2021, at 8:00 PM UTC; PublishedTime=2021-04-13T17:05:41.633Z

Status: Development of the code fix is taking slightly longer than expected to complete. We anticipate that it will take approximately two hours to finalize the fix, at which point we'll begin our deployment procedures to the affected environment. Next update by: Wednesday, April 14, 2021, at 4:00 AM UTC; PublishedTime=2021-04-14T00:11:17.52Z

Status: We've finalized the fix and deployment to the affected environments is underway. We anticipate the deployments will complete with an hour. Next update by: Wednesday, April 14, 2021, at 5:00 AM UTC; PublishedTime=2021-04-14T03:58:10.743Z

Status: We've confirmed the deployment has completed to the affected environments and our internal testing shows that the issue is no longer occurring. We're reaching out to customers to verify that the issue is mitigated. Next update by: Wednesday, April 14, 2021, at 11:00 AM UTC; PublishedTime=2021-04-14T04:42:50.34Z

Status: We're continuing to reach out to customers to verify that the issue is mitigated. Next update by: Wednesday, April 14, 2021, at 6:00 PM UTC; PublishedTime=2021-04-14T10:46:04.39Z

Final status: We've confirmed with affected users that this issue is resolved. Next update by: Wednesday, April 14, 2021, at 6:00 PM UTC; PublishedTime=2021-04-14T10:46:04.39Z

--------------------------------------------------------------------------------------------------
Title Admins may experience delays for usage reports data
ID MO249061
Service Microsoft 365 suite
Feature Administration
Status Service restored
Impact Admins may have experienced delays for usage reports data.
Severity
Start time 2021-04-05T00:00:00Z
End time 2021-04-06T15:14:00Z
Last update 2021-04-06T20:50:21.773Z
More info All usage reports were impacted, and usage reports for April 2, 2021 and onward may have been delayed. Final status: We've confirmed after monitoring the service that the delayed data has successfully completed processing and the service is now restored. Scope of impact: Any admin attempting to view usage report data may have been impacted. Start time: Monday, April 5, 2021, at 12:00 AM'UTC End time: Tuesday, April 6, 2021, at 3:14 PM'UTC Root cause: An issue in our processing pipeline was causing delays in usage reports data. Next steps: - We're analyzing the processing pipeline components and data flow patterns to better understand why this issue occurred and to prevent similar usage report data availability problems from happening again in the future. This is the final update for the event.; PublishedTime=2021-04-06T20:50:21.773Z
Root cause An issue in our processing pipeline was causing delays in usage reports data. Next steps: - We're analyzing the processing pipeline components and data flow patterns to better understand why this issue occurred and to prevent similar usage report data availability problems from happening again in the future. This is the final update for the event.; PublishedTime=2021-04-06T20:50:21.773Z

MO249061 Details

Status: We're investigating a potential issue and checking for impact to your organization. We'll provide an update within 30 minutes.

Status: We've determined that an issue in our processing pipeline caused delays in usage reports data. We've developed and deployed a fix to resolve the pipeline issue. We're monitoring as the delayed data finishes processing, and we expect service will be restored by the end of Wednesday, April 7, 2021. Next update by: Wednesday, April 7, 2021, at 10:00 AMUTC; PublishedTime=2021-04-06T04:41:04.317Z

Final status: We've confirmed after monitoring the service that the delayed data has successfully completed processing and the service is now restored. Next update by: Wednesday, April 7, 2021, at 10:00 AMUTC; PublishedTime=2021-04-06T04:41:04.317Z

--------------------------------------------------------------------------------------------------
Title Admins can't update certain device policies
ID IT251543
Service Microsoft Intune
Feature Microsoft Intune
Status Service restored
Impact Admins may have been unable to update certain device policies.
Severity Sev2
Start time 2021-04-04T03:00:00Z
End time 2021-04-20T18:16:00Z
Last update 2021-04-20T19:50:27.88Z
More info This issue impacted certain policies that were created or edited in Intune service release 2103. The policies are: -Endpoint Security policies targeted to Configuration Manager -Settings Catalog device configurations Final status: We confirmed that the deployment completed and we monitored service telemetry to verify that the issue has been resolved. Scope of impact: This issue could have potentially affected any admin who created or edited specific policies during Intune service release 2103. Start time: Sunday, April 4, 2021, at 3:00 AM'UTC End time: Tuesday, April 20, 2021, at 6:16 PM'UTC Root cause: A coding issue with device policies created during Intune service release 2103 was preventing updates to policies in service release 2104. Next steps: - We're reviewing the code issue in question to identify how it was introduced to better prevent similar future impact. This is the final update for the event.; PublishedTime=2021-04-20T19:50:27.88Z
Root cause A coding issue with device policies created during Intune service release 2103 was preventing updates to policies in service release 2104. Next steps: - We're reviewing the code issue in question to identify how it was introduced to better prevent similar future impact. This is the final update for the event.; PublishedTime=2021-04-20T19:50:27.88Z

IT251543 Details

Status: We're investigating a potential issue and checking for impact to your organization. We'll provide an update within 30 minutes.; PublishedTime=2021-04-20T02:42:33.827

Status: We identified a potential fix for the issue and are testing internally to validate it remediates impact. Next update by: Tuesday, April 20, 2021, at 4:30 AM UTC; PublishedTime=2021-04-20T03:11:09.28Z

Status: We've validated the fix internally and are continuing with our deployment process to the impacted infrastructure. We'll be monitoring the deployment and validating during each stage to avoid complications. Next update by: Tuesday, April 20, 2021, at 6:30 PM UTC; PublishedTime=2021-04-20T04:30:08.693Z

Status: We're continuing to monitor the fix as it deploys through the impacted environments. We expect the deployment to complete by the next scheduled update. Next update by: Tuesday, April 20, 2021, at 9:30 PMUTC; PublishedTime=2021-04-20T16:46:18.793Z

Final status: We confirmed that the deployment completed and we monitored service telemetry to verify that the issue has been resolved. Next update by: Tuesday, April 20, 2021, at 9:30 PMUTC; PublishedTime=2021-04-20T16:46:18.793Z

--------------------------------------------------------------------------------------------------
Title Admins can't submit items older than seven days in the Submissions portal in the Security & Compliance Center (SCC)
ID EX248435
Service Exchange Online
Feature Management and Provisioning
Status Extended recovery
Impact Admins may be unable to submit items older than seven days in the Submissions portal in the SCC.
Severity Sev2
Start time 2021-04-02T19:00:00Z
End time
Last update 2021-05-07T23:31:03.103Z
More info For more information regarding the Submissions portal, please review the following link: https://docs.microsoft.com/en-us/microsoft-365/security/office-365-securi ty/admin-submission Admins can still send items for scanning if they're within the seven day limit. Current status: The validation process of the fix is ongoing and taking longer than expected. Due to the thorough testing process to ensure it fully mitigates the issue, we anticipate the deployment to start by our next communications update. Once the deployment has started, we'll be able to provide an estimated timeline of completion. Scope of impact: Any admin attempting to submit items older than seven days to the Submissions portal may encounter this issue. Start time: Friday, April 2, 2021, at 7:00 PM'UTC Root cause: Recent service improvements required us to revert some updates made previously that enabled admins to submit items to the Submissions portal if they were older than seven days old but did not exceed 30 days. Next update by: Saturday, May 22, 2021, at 1:00 AMUTC; PublishedTime=2021-05-07T23:31:03.103Z
Root cause Recent service improvements required us to revert some updates made previously that enabled admins to submit items to the Submissions portal if they were older than seven days old but did not exceed 30 days. Next update by: Saturday, May 22, 2021, at 1:00 AMUTC; PublishedTime=2021-05-07T23:31:03.103Z

EX248435 Details

Status: Due to ongoing improvements to our service, we've determined that reverting previous updates that allowed admins to submit items for scanning beyond the seven day limit in the Submissions portal were needed, subsequently resulting in impact. We're working to develop a fix so that admins can submit items until the previous 30 day limit. Next update by: Saturday, April 10, 2021, at 1:00 AMUTC; PublishedTime=2021-04-03T01:52:04.297Z

Status: The development and testing of the fix for this problem is expected to last at least two more weeks due to it's complexity and dependencies. Once the fix is validated, we'll be able to provide a timeline for mitigation. Next update by: Saturday, April 24, 2021, at 1:00 AMUTC; PublishedTime=2021-04-10T00:11:29.367Z

Status: Testing and validation of our fix is ongoing, and while progress has been positive, due to the complexity of the issue, we're continuing to ensure the fix successfully mitigates impact with no additional side effects. We're anticipating deployment of the fix to begin by early May 2021, and will provide an updated timeline to completion with our next update. Next update by: Saturday, May 8, 2021, at 1:00 AMUTC; PublishedTime=2021-04-23T23:29:47.207Z

Status: The validation process of the fix is ongoing and taking longer than expected. Due to the thorough testing process to ensure it fully mitigates the issue, we anticipate the deployment to start by our next communications update. Once the deployment has started, we'll be able to provide an estimated timeline of completion. Next update by: Saturday, May 22, 2021, at 1:00 AMUTC; PublishedTime=2021-05-07T23:31:03.103Z

--------------------------------------------------------------------------------------------------
Title Admins can't utilize the Attack Simulator feature in Microsoft Defender for Office 365
ID EX249060
Service Exchange Online
Feature Management and Provisioning
Status Service restored
Impact Admins may have been unable to use the Attack Simulator feature in Microsoft Defender for Office 365.
Severity
Start time 2021-04-02T02:25:00Z
End time 2021-04-06T14:30:00Z
Last update 2021-04-06T17:35:09.813Z
More info
Root cause A recent service update contained an issue that resulted in impact. Next steps: - We're reviewing the affecting update as well as our testing and validation processes to identify how to avoid problems like this in future deployments. This is the final update for the event.; PublishedTime=2021-04-06T17:35:09.813Z

EX249060 Details

Status: We're investigating a potential issue and checking for impact to your organization. We'll provide an update within 30 minutes.; PublishedTime=2021-04-06T04:14:46.387

Status: Our investigation into an issue where users can't utilize the Attack Simulator feature in Microsoft Defender for Office 365 indicates that a recent service update is responsible for impact. We're working to revert that update to restore the service to expected functionality. Next update by: Tuesday, April 6, 2021, at 5:30 AMUTC; PublishedTime=2021-04-06T04:32:12.823Z

Status: We've initiated reversion of the impacting service update. Once reversion completes, we'll conduct testing to ensure the service has returned to normal functionality. Next update by: Tuesday, April 6, 2021, at 7:30 PMUTC; PublishedTime=2021-04-06T05:08:34.457Z

Final status: We've completed reverting of the impacting service update, and confirmed via telemetry that impact is mitigated. Next update by: Tuesday, April 6, 2021, at 7:30 PMUTC; PublishedTime=2021-04-06T05:08:34.457Z

--------------------------------------------------------------------------------------------------
Title DNS issue affecting multiple Microsoft Cloud services
ID CR248164
Service Dynamics 365 Apps
Feature Organization access
Status Post-incident report published
Impact Users may have been unable to access multiple Microsoft Cloud services and features.
Severity
Start time 2021-04-01T21:21:00Z
End time 2021-04-01T22:30:00Z
Last update 2021-04-05T18:22:18.227Z
More info
Root cause

CR248164 Details

Status: MessageText=Title: DNS issue affecting multiple Microsoft Cloud services User Impact: Users may have been unable to access multiple Microsoft Cloud services and features. More Info: The following services may have experienced impact during the outage window. Microsoft Dynamics 365, Microsoft Finance and Operations, Microsoft Power Apps, Microsoft Power Automate, Microsoft Power Virtual Agents Summary of Impact: Between approximately 21:21 UTC and 22:30 UTC on 01 Apr 2021, customers may have experienced intermittent issues accessing multiple Microsoft 365 services including Azure, Microsoft Dynamics 365 and Microsoft Power Platform services. Root Cause: Azure DNS servers experienced an anomalous surge in DNS queries from across the globe targeting a set of domains hosted on Azure. Normally, Azure's layers of caches and traffic shaping would mitigate this surge. In this incident, one specific sequence of events exposed a code defect in our DNS service that reduced the efficiency of our DNS Edge caches. As our DNS service became overloaded, DNS clients began frequent retries of their requests which added workload to the DNS service. Since client retries are considered legitimate DNS traffic, this traffic was not dropped by our volumetric spike mitigation systems. This increase in traffic led to decreased availability of our DNS service. Next Steps: Additional details can be found here: https://status.azure.com/en-us/status/history/ under Tracking ID GVY5-TZZ A Post Incident Report will be published within five business days.; PublishedTime=2021-04-02T00:39:46.933

--------------------------------------------------------------------------------------------------
Title DNS issue affecting multiple Microsoft Cloud services
ID CR248164
Service Dynamics 365 Apps
Feature Organization access
Status Post-incident report published
Impact Users may have been unable to access multiple Microsoft Cloud services and features.
Severity
Start time 2021-04-01T21:21:00Z
End time 2021-04-01T22:30:00Z
Last update 2021-04-05T18:22:18.227Z
More info
Root cause

CR248164 Details

Status: MessageText=Title: DNS issue affecting multiple Microsoft Cloud services User Impact: Users may have been unable to access multiple Microsoft Cloud services and features. More Info: The following services may have experienced impact during the outage window. Microsoft Dynamics 365, Microsoft Finance and Operations, Microsoft Power Apps, Microsoft Power Automate, Microsoft Power Virtual Agents Summary of Impact: Between approximately 21:21 UTC and 22:30 UTC on 01 Apr 2021, customers may have experienced intermittent issues accessing multiple Microsoft 365 services including Azure, Microsoft Dynamics 365 and Microsoft Power Platform services. Root Cause: Azure DNS servers experienced an anomalous surge in DNS queries from across the globe targeting a set of domains hosted on Azure. Normally, Azure's layers of caches and traffic shaping would mitigate this surge. In this incident, one specific sequence of events exposed a code defect in our DNS service that reduced the efficiency of our DNS Edge caches. As our DNS service became overloaded, DNS clients began frequent retries of their requests which added workload to the DNS service. Since client retries are considered legitimate DNS traffic, this traffic was not dropped by our volumetric spike mitigation systems. This increase in traffic led to decreased availability of our DNS service. Next Steps: Additional details can be found here: https://status.azure.com/en-us/status/history/ under Tracking ID GVY5-TZZ A Post Incident Report will be published within five business days.; PublishedTime=2021-04-02T00:39:46.933

--------------------------------------------------------------------------------------------------
Title Some users in South America may have been unable to access multiple Microsoft 365 services
ID MO248087
Service Microsoft 365 suite
Feature Portal
Status Service restored
Impact Users in South America may have been unable to access Microsoft 365 services.
Severity
Start time 2021-04-01T13:19:00Z
End time 2021-04-01T14:07:00Z
Last update 2021-04-01T15:11:55.437Z
More info Users that could access services may have experienced degraded performance. Final status: We've identified that a section of our networking infrastructure had unexpectedly become unresponsive, causing connection requests to fail. We've redirected user requests to alternate infrastructure, which after a period of monitoring service health, we've confirmed has mitigated impacted. Scope of impact: Impact was specific to users who were served by the affected infrastructure in Argentina. Start time: Thursday, April 1, 2021, at 1:19 PM UTC End time: Thursday, April 1, 2021, at 2:07 PM UTC Root cause: A section of our networking infrastructure had unexpectedly become unresponsive, causing connection requests to fail. Next steps: -We're analyzing the affected infrastructure to prevent similar problems in the future. This is the final update for the event.; PublishedTime=2021-04-01T15:11:55.437Z
Root cause A section of our networking infrastructure had unexpectedly become unresponsive, causing connection requests to fail. Next steps: -We're analyzing the affected infrastructure to prevent similar problems in the future. This is the final update for the event.; PublishedTime=2021-04-01T15:11:55.437Z

MO248087 Details

Status: We're investigating reports of a potential issue with the Microsoft 365 service. We'll provide an update within 30 minutes.; PublishedTime=2021-04-01T13:46:59.007

Status: We're reviewing service monitoring telemetry to isolate the source of the issue. Next update by: Thursday, April 1, 2021, at 3:30 PM UTC; PublishedTime=2021-04-01T14:14:30.287Z

Final status: We've identified that a section of our networking infrastructure had unexpectedly become unresponsive, causing connection requests to fail. We've redirected user requests to alternate infrastructure, which after a period of monitoring service health, we've confirmed has mitigated impacted. Next update by: Thursday, April 1, 2021, at 3:30 PM UTC; PublishedTime=2021-04-01T14:14:30.287Z

--------------------------------------------------------------------------------------------------
Title UK users may be unable to access multiple Microsoft 365 services
ID MO248076
Service Microsoft 365 suite
Feature Portal
Status Service restored
Impact Users in the UK may have been unable to access Microsoft 365 services.
Severity
Start time 2021-04-01T10:46:00Z
End time 2021-04-01T11:01:00Z
Last update 2021-04-01T12:14:18.31Z
More info Users that could access services may have experienced degraded performance. Services that may have been impacted include, but were not limited to: Microsoft Teams, Yammer, SharePoint Online and OneDrive for Business. Final status: We've analyzed service telemetry and confirmed that the Microsoft-Managed environment is performing as expected. We believe the issue occurred outside of our environment. Additionally, reports of impact have now stopped. Scope of impact: Reports of impact were limited to users located within the United Kingdom. Start time: Thursday, April 1, 2021, at 10:46 AM UTC End time: Thursday, April 1, 2021, at 11:01 AM UTC This is the final update for the event.; PublishedTime=2021-04-01T11:57:51.743Z
Root cause

MO248076 Details

Status: We're reviewing service telemetry to identity the source of the problem. Next update by: Thursday, April 1, 2021, at 2:00 PM UTC; PublishedTime=2021-04-01T11:23:43.833Z

Final status: We've analyzed service telemetry and confirmed that the Microsoft-Managed environment is performing as expected. We believe the issue occurred outside of our environment. Additionally, reports of impact have now stopped. Next update by: Thursday, April 1, 2021, at 2:00 PM UTC; PublishedTime=2021-04-01T11:23:43.833Z

--------------------------------------------------------------------------------------------------
Title Global admins can't add or modify custom app launcher tiles from the Microsoft 365 admin center
ID MO250626
Service Microsoft 365 suite
Feature Portal
Status Service restored
Impact Global admins can't add or modify custom app launcher tiles from the Microsoft 365 admin center.
Severity
Start time 2021-04-01T00:00:00Z
End time 2021-04-16T22:00:00Z
Last update 2021-04-16T22:53:51.823Z
More info Admins would have seen a grayed out Save button that prevents them from adding or modifying custom app launcher tiles. Final status: We've successfully deployed the fix and confirmed via telemetry that this resolved. Scope of impact: Any global admin may have experienced impact. Start time: Thursday, April 1, 2021, at 12:00 AM'UTC End time: Friday, April 16, 2021, at 10:00 PM'UTC Root cause: A recent service update contained a code regression, causing impact. Next steps: - We're reviewing our service update and validation procedures to determine why this issue wasn't caught prior to being deployed and prevent issues like this from occurring in the future. This is the final update for the event.; PublishedTime=2021-04-16T22:52:02.413Z
Root cause A recent service update contained a code regression, causing impact. Next steps: - We're reviewing our service update and validation procedures to determine why this issue wasn't caught prior to being deployed and prevent issues like this from occurring in the future. This is the final update for the event.; PublishedTime=2021-04-16T22:52:02.413Z

MO250626 Details

Status: We're investigating a potential issue and checking for impact to your organization. We'll provide an update within 30 minutes.; PublishedTime=2021-04-14T18:00:11.563

Status: We're reviewing system logs to isolate the origin of this issue and determine our next steps. Next update by: Thursday, April 15, 2021, at 1:00 AMUTC; PublishedTime=2021-04-14T18:25:29.457Z

Status: We've determined that a recent service update intended to improve a recent TypeScript update contained a code regression, causing impact. We're developing a fix which we anticipate will resolve the issue. Next update by: Friday, April 16, 2021, at 1:00 AMUTC; PublishedTime=2021-04-15T00:24:38.53Z

Status: We've completed developing the fix and are finalizing the deployment scheduling, after which we'll be able to provide a timeline to mitigation. Next update by: Saturday, April 17, 2021, at 1:30 AMUTC; PublishedTime=2021-04-15T23:59:32.013Z

Final status: We've successfully deployed the fix and confirmed via telemetry that this resolved. Next update by: Saturday, April 17, 2021, at 1:30 AMUTC; PublishedTime=2021-04-15T23:59:32.013Z

--------------------------------------------------------------------------------------------------
Title Approximately .03 percent of Public Switched Telephone Network (PSTN) outbound calls may suddenly drop for users
ID TM254360
Service Microsoft Teams
Feature Teams Components
Status Service degradation
Impact Users are intermittently experiencing PSTN outbound calls dropping after ten or more minutes.
Severity Sev2
Start time 2021-03-31T22:00:00Z
End time
Last update 2021-05-13T20:57:13.713Z
More info It's highly unlikely that the same user will repeatedly run into this issue. Users who experience the sudden drop should be able to place another outbound PSTN call without issue, or can continue to retry with eventual success. Current status: Our extended internal validation of the fix to correct the code regression continues to progress. We're expecting for our validation to be complete by Friday, May 21, 2021, at which time, we'll provide an estimate for the deployment and mitigation. Scope of impact: This issue affects approximately .03 percent of total outbound PSTN calls that last ten minutes or longer. Start time: Wednesday, March 31, 2021, at 10:00 PM'UTC Root cause: A recent service update intended to fix an unrelated issue inadvertently introduced a code regression, resulting in the current impact. Next update by: Friday, May 21, 2021, at 10:30 PMUTC; PublishedTime=2021-05-13T20:57:13.713Z
Root cause A recent service update intended to fix an unrelated issue inadvertently introduced a code regression, resulting in the current impact. Next update by: Friday, May 21, 2021, at 10:30 PMUTC; PublishedTime=2021-05-13T20:57:13.713Z

TM254360 Details

Status: We're investigating a potential issue and checking for impact to your organization. We'll provide an update within 30 minutes.

Status: We're reviewing a recent update which we suspect is contributing to the impact. In parallel, we're reviewing options to revert the update or develop a fix if it is confirmed to be the cause of this issue. Next update by: Wednesday, May 5, 2021, at 1:00 AMUTC; PublishedTime=2021-05-04T23:46:38.817Z

Status: We're currently attempting to reproduce the error in our test environment to better understand impact and to find the root cause. Next update by: Wednesday, May 5, 2021, at 5:00 AMUTC; PublishedTime=2021-05-05T00:33:17.2Z

Status: Our investigation indicates that a recent service update intended to fix an unrelated issue inadvertently introduced a code regression, resulting in the current impact. We're working to develop a fix for this issue, and we'll provide a deployment timeline at our next scheduled update time. Next update by: Thursday, May 6, 2021, at 10:30 PMUTC; PublishedTime=2021-05-05T03:36:32.737Z

Status: We're continuing our efforts to develop a fix, but given the breadth of impact, the fix is undergoing extended validation to ensure it resolves the issue without adversely impacting other aspects of the service. We expect to have an update on the validation and deployment timelines by the next scheduled update. Next update by: Thursday, May 13, 2021, at 10:30 PMUTC; PublishedTime=2021-05-06T21:10:26.53Z

Status: Our extended internal validation of the fix to correct the code regression continues to progress. We're expecting for our validation to be complete by Friday, May 21, 2021, at which time, we'll provide an estimate for the deployment and mitigation. Next update by: Friday, May 21, 2021, at 10:30 PMUTC; PublishedTime=2021-05-13T20:57:13.713Z

--------------------------------------------------------------------------------------------------
Title Organizers and producers can't see the download option for live event recordings
ID TM250131
Service Microsoft Teams
Feature Teams Components
Status Extended recovery
Impact Organizers and producers can't see the "Recording" option to download a live event recording.
Severity Sev2
Start time 2021-03-31T18:00:00Z
End time
Last update 2021-05-17T19:55:37.4Z
More info The download option isn't visible under "Live event resources". While we're focused on fixing the download option, users can click the live event attendee link to view the recording. Live events initiated after 1:45 PM UTC on Monday, April 12, 2021 are not impacted. Current status: We're continuing to re-encode the backups of the remaining live event recordings and are monitoring the re-encoding progress. Currently, progress stands at approximately 98 percent complete and we anticipate that the remaining two percent of the re-encoding jobs will be complete by the end of this week. Scope of impact: Impact is specific to organizers and producers trying to download live event recordings. Start time: Wednesday, March 31, 2021, at 6:00 PM'UTC Root cause: An API configuration change within a recent deployment, which was part of regularly scheduled maintenance, inadvertently caused impact. Next update by: Friday, May 21, 2021, at 9:00 PMUTC; PublishedTime=2021-05-17T19:55:37.4Z
Root cause An API configuration change within a recent deployment, which was part of regularly scheduled maintenance, inadvertently caused impact. Next update by: Friday, May 21, 2021, at 9:00 PMUTC; PublishedTime=2021-05-17T19:55:37.4Z

TM250131 Details

Status: We're investigating a potential issue and checking for impact to your organization. We'll provide an update within 30 minutes.

Status: We're analyzing impact information provided by your representatives to determine the source of the issue. Next update by: Monday, April 12, 2021, at 5:30 AMUTC; PublishedTime=2021-04-12T04:25:22.157Z

Status: We're analyzing service logs to isolate the problem. Next update by: Monday, April 12, 2021, at 7:30 AMUTC; PublishedTime=2021-04-12T05:25:53.517Z

Status: We're continuing to analyze service logs, specifically focusing our attention on reviewing the code for recording data to identify if the issue is coming from there. Next update by: Monday, April 12, 2021, at 8:30 PM UTC; PublishedTime=2021-04-12T07:15:53.94Z

Status: We identified that a deployment, which was part of regularly scheduled maintenance, has inadvertently caused impact. We've rolled back this deployment and confirmed that the issue no longer occurs for new or recent meetings. We're developing a plan to restore the recordings for older events to fully remediate impact. Next update by: Monday, April 12, 2021, at 10:00 PM UTC ; PublishedTime=2021-04-12T14:23:23.307Z

Status: Further review of the impacting update has identified the issue is caused by an API mismatch between Microsoft Teams infrastructure and Azure Media Services. While the API configuration has been corrected, we're continuing our investigation into potential post-processing of these files to restore the downloads for live events recorded prior to 1:45 PM UTC on Monday, April 12, 2021. Next update by: Tuesday, April 13, 2021, at 10:00 PMUTC; PublishedTime=2021-04-12T20:55:05.607Z

Status: We've identified that impacted recordings were properly encoded, but were stamped incorrectly by the service, making them unavailable for download. We're reviewing options to either correct how these recordings are stamped, or re-encoding affected recordings from prior to mitigate impact. Next update by: Wednesday, April 14, 2021, at 10:00 PMUTC; PublishedTime=2021-04-13T20:31:30.197Z

Status: We've started the re-encoding process for the impacted recordings which is expected to require an extended period of time to complete. In parallel, we're continuing to confirm if we're able to correct how the affected recordings were stamped which would greatly expedite the remediation of impact. Next update by: Friday, April 16, 2021, at 10:00 PMUTC; PublishedTime=2021-04-14T20:36:07.267Z

Status: We've confirmed that the re-encoding process for the impacted recordings is processing at expected rates and it has reached 40 percent completion. This process may require an extended period of time to complete in full and we're monitoring the service to ensure it completes successfully. Additionally, we've determined that correcting how the affected recordings were stamped as an expedited remediation method isn't viable. Next update by: Monday, April 19, 2021, at 6:30 PMUTC; PublishedTime=2021-04-16T20:45:01.137Z

Status: The re-encoding process continues to progress at expected rates and has reached 70 percent completion. As this process completes, organizers and producers will begin to see live event recordings available for download. We'll continue monitoring the service to ensure it completes successfully and provide an estimated time for completion as soon as one becomes available. Next update by: Monday, April 26, 2021, at 6:30 PMUTC; PublishedTime=2021-04-19T17:01:08.447Z

Status: Our telemetry indicates that the re-encoding process has completed for 92% of recordings, and that the vast majority can now be downloaded. Though we don't have a precise timeline for completion, we expect all recordings to be downloadable by next week, Monday, May 5, 2021. Next update by: Monday, May 3, 2021, at 6:30 PMUTC; PublishedTime=2021-04-26T17:15:54.61Z

Status: Our most recent telemetry indicates that the re-encoding process has now completed for 95% of the recordings, and that the vast majority of recordings can now be downloaded. We're continuing to monitor the re-encoding process and are expecting for the remaining recordings to be re-encoded and for impact to be resolved by Thursday, May 6, 2021, at 11:00 PM UTC. Next update by: Thursday, May 6, 2021, at 11:00 PM UTC; PublishedTime=2021-05-03T17:34:17.877Z

Status: Our re-encoding process hasn't completed as expected, although it has progressed to 95.2 percent. We're continuing to monitor the progress of re-encoding and will provide an updated estimate timeline for resolution with our next update. Next update by: Monday, May 10, 2021, at 9:00 PMUTC; PublishedTime=2021-05-06T19:20:50.493Z

Status: We've determined that all the live event recordings have backups which will also need to be re-encoded, prolonging the process. We're continuing to monitor the progress of re-encoding, which is approximately 95 percent complete, and may be able to provide an updated estimate timeline for resolution with our next update. Next update by: Monday, May 17, 2021, at 9:00 PMUTC; PublishedTime=2021-05-10T20:16:49.76Z

Status: We're continuing to re-encode the backups of the remaining live event recordings and are monitoring the re-encoding progress. Currently, progress stands at approximately 98 percent complete and we anticipate that the remaining two percent of the re-encoding jobs will be complete by the end of this week. Next update by: Friday, May 21, 2021, at 9:00 PMUTC; PublishedTime=2021-05-17T19:55:37.4Z

--------------------------------------------------------------------------------------------------
Title Admins may be unable to make updates to Autopilot profile assignments and may see incorrect win32 app install reporting
ID IT250579
Service Microsoft Intune
Feature Microsoft Intune
Status Service restored
Impact Admins were unable to make updates to Autopilot profile assignments and saw incorrect win32 app install reporting.
Severity
Start time 2021-03-31T00:00:00Z
End time 2021-04-16T22:00:00Z
Last update 2021-04-16T22:42:56.08Z
More info Autopilot profiles assignments may have updated incorrectly when admins: - Altered the security groups memberships of Autopilot devices. - Altered nesting groups containing Autopilot devices. - Added new Intune Mobile Device Management (MDM) app, profile or policy assignments to a security group containing Autopilot devices that did not have any assignments before, or deleted the last assignment from a group containing Autopilot devices. Admins would have seen the win32 app install reporting issues reflected in the Microsoft Endpoint Manager (MEM) admin center. Final status: We've deployed the fix and confirmed that it restores assignments for previously impacted devices. Our previous fix prevented impact to further devices. Scope of impact: Impact was specific to admins hosted on the affected infrastructure. Start time: Wednesday, March 31, 2021, at 12:00 AM UTC End time: Friday, April 16, 2021, at 10:00 PM UTC Root cause: A recent service update introduced an unexpected regression within the code that supports Autopilot profile assignments and win32 app reporting. Next steps: - We're reviewing our service update procedures to assess why impact wasn't identified prior to deployment and to better identify similar code issues during our development and testing cycles. This is the final update for the event.; PublishedTime=2021-04-16T22:38:33.083Z
Root cause A recent service update introduced an unexpected regression within the code that supports Autopilot profile assignments and win32 app reporting. Next steps: - We're reviewing our service update procedures to assess why impact wasn't identified prior to deployment and to better identify similar code issues during our development and testing cycles. This is the final update for the event.; PublishedTime=2021-04-16T22:38:33.083Z

IT250579 Details

Status: We've determined a recent deployment introduced an unexpected regression within Autopilot profile assignments. We've developed a fix and are deploying it to the affected infrastructure to mitigate this impact. Next update by: Wednesday, April 14, 2021, at 10:30 AM UTC; PublishedTime=2021-04-14T08:07:13.8Z

Status: We're monitoring the progress of our fix as it continues to deploy across the affected environments to ensure it provides the expected relief. Next update by: Wednesday, April 14, 2021, at 6:00 PM UTC; PublishedTime=2021-04-14T09:36:39.603Z

Status: The deployment of the fix is progressing as expected and we're continuing to monitor as it saturates throughout the affected environment. Once complete, additional validation is needed to ensure that the issue is fully resolved. Next update by: Wednesday, April 14, 2021, at 9:00 PMUTC; PublishedTime=2021-04-14T17:00:47.097Z

Status: Our continued investigation into the nature of the regression has identified impact to win32 app install reporting in addition to the previously identified impact to Autopilot profile assignments. We're actively monitoring the deployment of the fix for this issue as it continues to progress through our change management processes. Subsequent validation will be needed upon completion to ensure that the issue is fully resolved. Next update by: Thursday, April 15, 2021, at 2:00 AM UTC; PublishedTime=2021-04-14T19:45:09.417Z

Status: Our fix deployment to revert the problematic changes is progressing as expected; however, we've determined that a script is also needed to resolve issues caused by the service update. We're working on this script in parallel and expect to provide a timeline for completion in our next scheduled update. Next update by: Thursday, April 15, 2021, at 5:00 AM UTC; PublishedTime=2021-04-15T01:16:15.553Z

Status: Our fix deployment to revert the problematic changes is continuing to progress as expected. We've finished developing the mitigation script to restore assignments and reporting, which is undergoing validation before being run on the affected environment. Next update by: Thursday, April 15, 2021, at 6:00 PM UTC; PublishedTime=2021-04-15T04:54:10.093Z

Status: Our fix deployment to revert the problematic changes has completed. We're continuing the mitigation steps to restore assignments for previously impacted device, which is still undergoing validation before being implemented on the affected environment. Next update by: Friday, April 16, 2021, at 6:00 PM UTC; PublishedTime=2021-04-15T17:12:13.727Z

Status: Due to the complexity of this issue, our efforts to validate an optimal fix to restore assignments for previously impacted devices is still ongoing. We're ensuring this fix will alleviate impact in full before being implemented in the affected environment. Next update by: Saturday, April 17, 2021, at 2:00 AM UTC; PublishedTime=2021-04-16T17:23:00.08Z

Final status: We've deployed the fix and confirmed that it restores assignments for previously impacted devices. Our previous fix prevented impact to further devices. Next update by: Saturday, April 17, 2021, at 2:00 AM UTC; PublishedTime=2021-04-16T17:23:00.08Z

--------------------------------------------------------------------------------------------------
Title Admins may be unable to make updates to Autopilot profile assignments and may see incorrect win32 app install reporting
ID IT250579
Service Microsoft Intune
Feature Microsoft Intune
Status Service restored
Impact Admins were unable to make updates to Autopilot profile assignments and saw incorrect win32 app install reporting.
Severity
Start time 2021-03-31T00:00:00Z
End time 2021-04-16T22:00:00Z
Last update 2021-04-16T22:42:56.08Z
More info Autopilot profiles assignments may have updated incorrectly when admins: - Altered the security groups memberships of Autopilot devices. - Altered nesting groups containing Autopilot devices. - Added new Intune Mobile Device Management (MDM) app, profile or policy assignments to a security group containing Autopilot devices that did not have any assignments before, or deleted the last assignment from a group containing Autopilot devices. Admins would have seen the win32 app install reporting issues reflected in the Microsoft Endpoint Manager (MEM) admin center. Final status: We've deployed the fix and confirmed that it restores assignments for previously impacted devices. Our previous fix prevented impact to further devices. Scope of impact: Impact was specific to admins hosted on the affected infrastructure. Start time: Wednesday, March 31, 2021, at 12:00 AM UTC End time: Friday, April 16, 2021, at 10:00 PM UTC Root cause: A recent service update introduced an unexpected regression within the code that supports Autopilot profile assignments and win32 app reporting. Next steps: - We're reviewing our service update procedures to assess why impact wasn't identified prior to deployment and to better identify similar code issues during our development and testing cycles. This is the final update for the event.; PublishedTime=2021-04-16T22:38:33.083Z
Root cause A recent service update introduced an unexpected regression within the code that supports Autopilot profile assignments and win32 app reporting. Next steps: - We're reviewing our service update procedures to assess why impact wasn't identified prior to deployment and to better identify similar code issues during our development and testing cycles. This is the final update for the event.; PublishedTime=2021-04-16T22:38:33.083Z

IT250579 Details

Status: We've determined a recent deployment introduced an unexpected regression within Autopilot profile assignments. We've developed a fix and are deploying it to the affected infrastructure to mitigate this impact. Next update by: Wednesday, April 14, 2021, at 10:30 AM UTC; PublishedTime=2021-04-14T08:07:13.8Z

Status: We're monitoring the progress of our fix as it continues to deploy across the affected environments to ensure it provides the expected relief. Next update by: Wednesday, April 14, 2021, at 6:00 PM UTC; PublishedTime=2021-04-14T09:36:39.603Z

Status: The deployment of the fix is progressing as expected and we're continuing to monitor as it saturates throughout the affected environment. Once complete, additional validation is needed to ensure that the issue is fully resolved. Next update by: Wednesday, April 14, 2021, at 9:00 PMUTC; PublishedTime=2021-04-14T17:00:47.097Z

Status: Our continued investigation into the nature of the regression has identified impact to win32 app install reporting in addition to the previously identified impact to Autopilot profile assignments. We're actively monitoring the deployment of the fix for this issue as it continues to progress through our change management processes. Subsequent validation will be needed upon completion to ensure that the issue is fully resolved. Next update by: Thursday, April 15, 2021, at 2:00 AM UTC; PublishedTime=2021-04-14T19:45:09.417Z

Status: Our fix deployment to revert the problematic changes is progressing as expected; however, we've determined that a script is also needed to resolve issues caused by the service update. We're working on this script in parallel and expect to provide a timeline for completion in our next scheduled update. Next update by: Thursday, April 15, 2021, at 5:00 AM UTC; PublishedTime=2021-04-15T01:16:15.553Z

Status: Our fix deployment to revert the problematic changes is continuing to progress as expected. We've finished developing the mitigation script to restore assignments and reporting, which is undergoing validation before being run on the affected environment. Next update by: Thursday, April 15, 2021, at 6:00 PM UTC; PublishedTime=2021-04-15T04:54:10.093Z

Status: Our fix deployment to revert the problematic changes has completed. We're continuing the mitigation steps to restore assignments for previously impacted device, which is still undergoing validation before being implemented on the affected environment. Next update by: Friday, April 16, 2021, at 6:00 PM UTC; PublishedTime=2021-04-15T17:12:13.727Z

Status: Due to the complexity of this issue, our efforts to validate an optimal fix to restore assignments for previously impacted devices is still ongoing. We're ensuring this fix will alleviate impact in full before being implemented in the affected environment. Next update by: Saturday, April 17, 2021, at 2:00 AM UTC; PublishedTime=2021-04-16T17:23:00.08Z

Final status: We've deployed the fix and confirmed that it restores assignments for previously impacted devices. Our previous fix prevented impact to further devices. Next update by: Saturday, April 17, 2021, at 2:00 AM UTC; PublishedTime=2021-04-16T17:23:00.08Z

--------------------------------------------------------------------------------------------------
Title Admins may be unable to make updates to Autopilot profile assignments and may see incorrect win32 app install reporting
ID IT250579
Service Microsoft Intune
Feature Microsoft Intune
Status Service restored
Impact Admins were unable to make updates to Autopilot profile assignments and saw incorrect win32 app install reporting.
Severity
Start time 2021-03-31T00:00:00Z
End time 2021-04-16T22:00:00Z
Last update 2021-04-16T22:42:56.08Z
More info Autopilot profiles assignments may have updated incorrectly when admins: - Altered the security groups memberships of Autopilot devices. - Altered nesting groups containing Autopilot devices. - Added new Intune Mobile Device Management (MDM) app, profile or policy assignments to a security group containing Autopilot devices that did not have any assignments before, or deleted the last assignment from a group containing Autopilot devices. Admins would have seen the win32 app install reporting issues reflected in the Microsoft Endpoint Manager (MEM) admin center. Final status: We've deployed the fix and confirmed that it restores assignments for previously impacted devices. Our previous fix prevented impact to further devices. Scope of impact: Impact was specific to admins hosted on the affected infrastructure. Start time: Wednesday, March 31, 2021, at 12:00 AM UTC End time: Friday, April 16, 2021, at 10:00 PM UTC Root cause: A recent service update introduced an unexpected regression within the code that supports Autopilot profile assignments and win32 app reporting. Next steps: - We're reviewing our service update procedures to assess why impact wasn't identified prior to deployment and to better identify similar code issues during our development and testing cycles. This is the final update for the event.; PublishedTime=2021-04-16T22:38:33.083Z
Root cause A recent service update introduced an unexpected regression within the code that supports Autopilot profile assignments and win32 app reporting. Next steps: - We're reviewing our service update procedures to assess why impact wasn't identified prior to deployment and to better identify similar code issues during our development and testing cycles. This is the final update for the event.; PublishedTime=2021-04-16T22:38:33.083Z

IT250579 Details

Status: We've determined a recent deployment introduced an unexpected regression within Autopilot profile assignments. We've developed a fix and are deploying it to the affected infrastructure to mitigate this impact. Next update by: Wednesday, April 14, 2021, at 10:30 AM UTC; PublishedTime=2021-04-14T08:07:13.8Z

Status: We're monitoring the progress of our fix as it continues to deploy across the affected environments to ensure it provides the expected relief. Next update by: Wednesday, April 14, 2021, at 6:00 PM UTC; PublishedTime=2021-04-14T09:36:39.603Z

Status: The deployment of the fix is progressing as expected and we're continuing to monitor as it saturates throughout the affected environment. Once complete, additional validation is needed to ensure that the issue is fully resolved. Next update by: Wednesday, April 14, 2021, at 9:00 PMUTC; PublishedTime=2021-04-14T17:00:47.097Z

Status: Our continued investigation into the nature of the regression has identified impact to win32 app install reporting in addition to the previously identified impact to Autopilot profile assignments. We're actively monitoring the deployment of the fix for this issue as it continues to progress through our change management processes. Subsequent validation will be needed upon completion to ensure that the issue is fully resolved. Next update by: Thursday, April 15, 2021, at 2:00 AM UTC; PublishedTime=2021-04-14T19:45:09.417Z

Status: Our fix deployment to revert the problematic changes is progressing as expected; however, we've determined that a script is also needed to resolve issues caused by the service update. We're working on this script in parallel and expect to provide a timeline for completion in our next scheduled update. Next update by: Thursday, April 15, 2021, at 5:00 AM UTC; PublishedTime=2021-04-15T01:16:15.553Z

Status: Our fix deployment to revert the problematic changes is continuing to progress as expected. We've finished developing the mitigation script to restore assignments and reporting, which is undergoing validation before being run on the affected environment. Next update by: Thursday, April 15, 2021, at 6:00 PM UTC; PublishedTime=2021-04-15T04:54:10.093Z

Status: Our fix deployment to revert the problematic changes has completed. We're continuing the mitigation steps to restore assignments for previously impacted device, which is still undergoing validation before being implemented on the affected environment. Next update by: Friday, April 16, 2021, at 6:00 PM UTC; PublishedTime=2021-04-15T17:12:13.727Z

Status: Due to the complexity of this issue, our efforts to validate an optimal fix to restore assignments for previously impacted devices is still ongoing. We're ensuring this fix will alleviate impact in full before being implemented in the affected environment. Next update by: Saturday, April 17, 2021, at 2:00 AM UTC; PublishedTime=2021-04-16T17:23:00.08Z

Final status: We've deployed the fix and confirmed that it restores assignments for previously impacted devices. Our previous fix prevented impact to further devices. Next update by: Saturday, April 17, 2021, at 2:00 AM UTC; PublishedTime=2021-04-16T17:23:00.08Z

--------------------------------------------------------------------------------------------------
Title Some users are unable to access their voicemail via Microsoft Teams
ID TM252025
Service Microsoft Teams
Feature Teams Components
Status Service restored
Impact Users were unable to access their voicemail via all Microsoft Teams connection methods.
Severity Sev2
Start time 2021-03-30T18:13:00Z
End time 2021-04-30T16:00:00Z
Last update 2021-04-30T18:23:01.523Z
More info Affected users would've encountered an "Oops" error page while trying to access their voicemail. During impact, users could leverage any of the Outlook connection methods to access their voicemail as a potential alternative solution. Final status: The fix has completed saturating the affected environment. We've reviewed Microsoft Teams client-side service telemetry and confirmed the impact is now mitigated. Scope of impact: Your organization was affected by this event, and impact was limited to some users trying to access their voicemail in Microsoft Teams. Start time: Tuesday, March 30, 2021, at 6:13 PM'UTC End time: Friday, April 30, 2021, at 4:00 PM'UTC Root cause: A weekly update deployment inadvertently introduced a code change that caused voicemail to become inaccessible to some users in Microsoft Teams. Next steps: - We're reviewing our ongoing update procedures to more quickly identify and more seamlessly repair similar issues during our development and testing cycles to avoid impact in the future. This is the final update for the event.; PublishedTime=2021-04-30T17:06:54.537Z
Root cause A weekly update deployment inadvertently introduced a code change that caused voicemail to become inaccessible to some users in Microsoft Teams. Next steps: - We're reviewing our ongoing update procedures to more quickly identify and more seamlessly repair similar issues during our development and testing cycles to avoid impact in the future. This is the final update for the event.; PublishedTime=2021-04-30T17:06:54.537Z

TM252025 Details

Status: We're investigating a potential issue and checking for impact to your organization. We'll provide an update within 30 minutes.

Status: We've identified that a source folder that facilitates voicemail storage may be inaccessible for the impacted users. We're reviewing telemetry related to this folder to confirm this while also reviewing ways to restore access to it. Next update by: Thursday, April 22, 2021, at 2:30 AMUTC; PublishedTime=2021-04-22T00:48:27.32Z

Status: We're continuing to review telemetry related to the voicemail folder to determine the most efficient way to restore full access. While our analysis continues to fully remediate this issue, users can leverage any of the Outlook connection methods to access their voicemail as a potential alternative solution. Next update by: Thursday, April 22, 2021, at 4:30 AMUTC; PublishedTime=2021-04-22T02:24:30.07Z

Status: We suspect a previous deployed build could be causing the voicemail folder to become inaccessible, resulting in some users being unable to access their voicemail. We're reviewing the code of the suspected build to confirm our theory and formulate a mitigation plan. Next update by: Thursday, April 22, 2021, at 6:30 AMUTC; PublishedTime=2021-04-22T04:28:57.12Z

Status: We're continuing to review coding to assist with determining the root cause and to formulate a mitigation plan. Next update by: Thursday, April 22, 2021, at 7:00 PMUTC; PublishedTime=2021-04-22T06:08:06.477Z

Status: Our analysis of coding suggests a recent deployment may have introduced a code change that's causing voicemail to become inaccessible to some users. To confirm this, we're internally testing a new build, which will further assist in our isolation of the underlying cause of the issue and viable mitigation steps moving forward. Next update by: Friday, April 23, 2021, at 7:00 PMUTC; PublishedTime=2021-04-22T18:45:26.24Z

Status: We've determined that a recent weekly update deployment inadvertently introduced a code change that's causing voicemail to become inaccessible to some users. We've developed a fix and initiated the deployment to the affected environment, which is currently about 13 percent deployed. We expect that the fix should be complete by early next week, and we'll have a better idea on it's completion by our next update on Sunday, April 25, 2021. Next update by: Sunday, April 25, 2021, at 7:00 PMUTC; PublishedTime=2021-04-23T18:43:45.373Z

Status: After reviewing telemetry, we've confirmed that the fix is now 93 percent saturated and anticipate it will complete by our next scheduled update. Next update by: Tuesday, April 27, 2021, at 8:00 PMUTC; PublishedTime=2021-04-25T17:51:55.663Z

Status: Upon additional telemetry analysis, we've determined that the deployment will require another couple of days to complete. The fix has now saturated approximately 97 percent of the affected environment, and we're expecting for it to complete and for impact to be resolved by our next scheduled update. Next update by: Friday, April 30, 2021, at 6:00 PMUTC; PublishedTime=2021-04-27T18:37:10.473Z

Final status: The fix has completed saturating the affected environment. We've reviewed Microsoft Teams client-side service telemetry and confirmed the impact is now mitigated. Next update by: Friday, April 30, 2021, at 6:00 PMUTC; PublishedTime=2021-04-27T18:37:10.473Z

--------------------------------------------------------------------------------------------------
Title Users may experience failures installing requested or Volume Purchase Program (VPP) assigned apps for iOS and Mac
ID IT247609
Service Microsoft Intune
Feature Microsoft Intune
Status Service restored
Impact Users may have experienced failures when installing requested or VPP assigned applications for iOS and Mac devices.
Severity
Start time 2021-03-29T16:00:00Z
End time 2021-03-29T18:35:00Z
Last update 2021-03-29T20:17:36.287Z
More info
Root cause An issue affecting Apple services resulted in impact to application installations on iOS and Mac devices. Next steps: - We're working with Apple to gather additional details regarding impact and their remediation efforts. This is the final update for the event.; PublishedTime=2021-03-29T20:17:36.287Z

IT247609 Details

Status: We're investigating a potential issue and checking for impact to your organization. We'll provide an update within 30 minutes.

Status: Our review of service health telemetry information indicates this issue has self-mitigated. We're monitoring the service to confirm errors don't reoccur and will provide additional mitigation details in our next update. Next update by: Monday, March 29, 2021, at 9:00 PMUTC; PublishedTime=2021-03-29T19:32:11.727Z

Final status: We've confirmed that an issue affecting Apple services resulted in impact to application installations on iOS and Mac devices within the Microsoft Intune service. We've confirmed with Apple that they've resolved the issue, and that impact is mitigated. Next update by: Monday, March 29, 2021, at 9:00 PMUTC; PublishedTime=2021-03-29T19:32:11.727Z

--------------------------------------------------------------------------------------------------
Title Users may intermittently experience sign in issues and apps flashing while connected to a spotty network on Android
ID MO254584
Service Microsoft 365 suite
Feature Portal
Status Service restored
Impact Users may have intermittently experienced sign in issues and apps flashing while connected to a spotty network.
Severity Sev2
Start time 2021-03-25T19:00:00Z
End time 2021-05-12T21:19:00Z
Last update 2021-05-13T02:16:19.43Z
More info Multiple Microsoft 365 mobile apps may have experienced bright white flashing, including Microsoft Teams, Microsoft Power Apps, and Outlook. Additionally, users may have been unable to sign into Microsoft Power Apps and Microsoft Teams. If already signed in, users may have experienced functionality problems within Microsoft Power Apps and Microsoft Teams, such as being unable to open apps within Microsoft Power Apps. Once the fix was released, users should have received the update automatically; however, users could also have opted to force the update. For users that opted to the force-update: The apps would have been published on the Google Play Public Beta channels, so users could have opted in to the public Beta of Microsoft Authenticator and the Intune Company Portal and then checked for version number 6.2104.2785. Final status: We determined that the deployment has completed and the issue has been mitigated. Scope of impact: A subset of users using a Microsoft 365 app on an Android device with Microsoft Authenticator (version 6.2104.2785) and the Intune Company Portal (version 5.0.5134.0) while also connected to a network experiencing issues may have been impacted. Start time: Thursday, March 25, 2021, at 7:00 PM UTC End time: Wednesday, May 12, 2021, at 9:19 PM UTC Root cause: Microsoft Authenticator (version 6.2104.2785) and the Intune Company Portal (version 5.0.5134.0) were inadvertently causing Microsoft 365 apps on Android devices to experience flashing, sign-in problems, and app functionality issues if users were also connected to a spotty network. Next steps: - We're reviewing version our update procedures to better identify similar issues during our development and testing cycles. This is the final update for the event.; PublishedTime=2021-05-13T02:14:05.02Z
Root cause Microsoft Authenticator (version 6.2104.2785) and the Intune Company Portal (version 5.0.5134.0) were inadvertently causing Microsoft 365 apps on Android devices to experience flashing, sign-in problems, and app functionality issues if users were also connected to a spotty network. Next steps: - We're reviewing version our update procedures to better identify similar issues during our development and testing cycles. This is the final update for the event.; PublishedTime=2021-05-13T02:14:05.02Z

MO254584 Details

Status: We've identified that an additional version combination is resulting in persistent impact associated with MO252668 persists for a subset of users after the initial fix was deployed. We've developed an additional fix to address the remaining impacted version combination and expect that it will complete deployment by Monday, May 10, 2021. Next update by: Tuesday, May 11, 2021, at 1:30 AM UTC; PublishedTime=2021-05-05T20:03:34.197Z

Status: Due do the complexity of the additional fix, the deployment is taking longer than expected and is currently 20 percent complete. We expect the deployment to be fully rolled out and available to the Google Play store by Wednesday, May 12, 2021, at approximately 1:30 AM UTC. Next update by: Wednesday, May 12, 2021, at 1:30 AM UTC; PublishedTime=2021-05-10T21:42:11.023Z

Status: We've confirmed that our deployments to correct the issue within the Authenticator app have completed, and users are able to access this version via app stores at this time. We're continuing to deploy a parallel fix to address the issue with the Company Portal and fully mitigate impact for all users, and are monitoring its progress as it continues to saturate throughout the affected environment. Next update by: Thursday, May 13, 2021, at 1:30 AM UTC; PublishedTime=2021-05-11T23:59:15.067Z

Status: We're continuing to monitor the deployment of the fix to address the issue with the Company Portal, which is taking longer than anticipated. Next update by: Thursday, May 13, 2021, at 4:00 AM UTC; PublishedTime=2021-05-13T01:29:10.333Z

Final status: We determined that the deployment has completed and the issue has been mitigated. Next update by: Thursday, May 13, 2021, at 4:00 AM UTC; PublishedTime=2021-05-13T01:29:10.333Z

--------------------------------------------------------------------------------------------------
Title Users may intermittently experience sign in issues and apps flashing while connected to a spotty network on Android
ID MO254584
Service Microsoft 365 suite
Feature Portal
Status Service restored
Impact Users may have intermittently experienced sign in issues and apps flashing while connected to a spotty network.
Severity Sev2
Start time 2021-03-25T19:00:00Z
End time 2021-05-12T21:19:00Z
Last update 2021-05-13T02:16:19.43Z
More info Multiple Microsoft 365 mobile apps may have experienced bright white flashing, including Microsoft Teams, Microsoft Power Apps, and Outlook. Additionally, users may have been unable to sign into Microsoft Power Apps and Microsoft Teams. If already signed in, users may have experienced functionality problems within Microsoft Power Apps and Microsoft Teams, such as being unable to open apps within Microsoft Power Apps. Once the fix was released, users should have received the update automatically; however, users could also have opted to force the update. For users that opted to the force-update: The apps would have been published on the Google Play Public Beta channels, so users could have opted in to the public Beta of Microsoft Authenticator and the Intune Company Portal and then checked for version number 6.2104.2785. Final status: We determined that the deployment has completed and the issue has been mitigated. Scope of impact: A subset of users using a Microsoft 365 app on an Android device with Microsoft Authenticator (version 6.2104.2785) and the Intune Company Portal (version 5.0.5134.0) while also connected to a network experiencing issues may have been impacted. Start time: Thursday, March 25, 2021, at 7:00 PM UTC End time: Wednesday, May 12, 2021, at 9:19 PM UTC Root cause: Microsoft Authenticator (version 6.2104.2785) and the Intune Company Portal (version 5.0.5134.0) were inadvertently causing Microsoft 365 apps on Android devices to experience flashing, sign-in problems, and app functionality issues if users were also connected to a spotty network. Next steps: - We're reviewing version our update procedures to better identify similar issues during our development and testing cycles. This is the final update for the event.; PublishedTime=2021-05-13T02:14:05.02Z
Root cause Microsoft Authenticator (version 6.2104.2785) and the Intune Company Portal (version 5.0.5134.0) were inadvertently causing Microsoft 365 apps on Android devices to experience flashing, sign-in problems, and app functionality issues if users were also connected to a spotty network. Next steps: - We're reviewing version our update procedures to better identify similar issues during our development and testing cycles. This is the final update for the event.; PublishedTime=2021-05-13T02:14:05.02Z

MO254584 Details

Status: We've identified that an additional version combination is resulting in persistent impact associated with MO252668 persists for a subset of users after the initial fix was deployed. We've developed an additional fix to address the remaining impacted version combination and expect that it will complete deployment by Monday, May 10, 2021. Next update by: Tuesday, May 11, 2021, at 1:30 AM UTC; PublishedTime=2021-05-05T20:03:34.197Z

Status: Due do the complexity of the additional fix, the deployment is taking longer than expected and is currently 20 percent complete. We expect the deployment to be fully rolled out and available to the Google Play store by Wednesday, May 12, 2021, at approximately 1:30 AM UTC. Next update by: Wednesday, May 12, 2021, at 1:30 AM UTC; PublishedTime=2021-05-10T21:42:11.023Z

Status: We've confirmed that our deployments to correct the issue within the Authenticator app have completed, and users are able to access this version via app stores at this time. We're continuing to deploy a parallel fix to address the issue with the Company Portal and fully mitigate impact for all users, and are monitoring its progress as it continues to saturate throughout the affected environment. Next update by: Thursday, May 13, 2021, at 1:30 AM UTC; PublishedTime=2021-05-11T23:59:15.067Z

Status: We're continuing to monitor the deployment of the fix to address the issue with the Company Portal, which is taking longer than anticipated. Next update by: Thursday, May 13, 2021, at 4:00 AM UTC; PublishedTime=2021-05-13T01:29:10.333Z

Final status: We determined that the deployment has completed and the issue has been mitigated. Next update by: Thursday, May 13, 2021, at 4:00 AM UTC; PublishedTime=2021-05-13T01:29:10.333Z

--------------------------------------------------------------------------------------------------
Title Admins receiving Data Loss Prevention (DLP) alerts for users outside of rule scope
ID EX247940
Service Exchange Online
Feature Management and Provisioning
Status Service restored
Impact Admins were receiving DLP alerts for users outside of the scope of DLP rules.
Severity
Start time 2021-03-25T12:00:00Z
End time 2021-04-12T18:00:00Z
Last update 2021-04-13T05:15:05Z
More info As a temporary workaround, admins could have created a new policy with SharePoint Online and OneDrive for Business as the workload, used the parameter 'OneDriveSharedByMemberOf', and disabled the existing policy. Final status: We confirmed that the fix deployment completed. In order to fully remediate impact for your organization, you may need to edit your organization's rules for the fix to fully apply. Scope of impact: Impact was specific to admins who receive DLP alerts. Start time: Thursday, March 25, 2021, at 12:00 PM'UTC End time: Monday, April 12, 2021, at 6:00 PM'UTC Root cause: A recent feature deployment to enable DLP for distribution groups contained a code issue, resulting in impact. Next steps: -We're reviewing our feature updates procedures to help develop ways to identify similar issues prior to their deployment in the future. This is the final update for the event.; PublishedTime=2021-04-13T05:15:05Z
Root cause A recent feature deployment to enable DLP for distribution groups contained a code issue, resulting in impact. Next steps: -We're reviewing our feature updates procedures to help develop ways to identify similar issues prior to their deployment in the future. This is the final update for the event.; PublishedTime=2021-04-13T05:15:05Z

EX247940 Details

Status: We're investigating a potential issue and checking for impact to your organization. We'll provide an update within 30 minutes.; PublishedTime=2021-03-31T15:42:42.59

Status: We're analyzing support case data to better understand the circumstance in which the problem occurs and to identify our next troubleshooting steps. Next update by: Wednesday, March 31, 2021, at 5:30 PMUTC; PublishedTime=2021-03-31T16:07:14.947Z

Status: Our analysis of the support case data hasn't shown a conclusive cause of the problem; however, our analysis continues. Next update by: Wednesday, March 31, 2021, at 7:30 PMUTC; PublishedTime=2021-03-31T17:24:38.93Z

Status: We're continuing to analyze the support case data in parallel with the policy evaluation logs that were extracted from the data provided by your organization to determine our next trouble shooting steps. Next update by: Thursday, April 1, 2021, at 6:00 PMUTC; PublishedTime=2021-03-31T18:44:20.443Z

Status: Our analysis of the policy evaluation logs determined that a recent feature deployment to enable DLP for distribution groups contained a code issue, resulting in impact. We've provided your representatives with a temporary workaround, which is also included in the above More info, while we are in the process of developing a code fix. Once completed, we'll provide an estimated deployment timeline. Next update by: Friday, April 2, 2021, at 6:30 AMUTC; PublishedTime=2021-04-01T17:59:47.777Z

Status: The process to develop and validate the code fix is taking longer than expected. We expect to have a estimated deployment time by our next scheduled update. Next update by: Saturday, April 3, 2021, at 6:30 AMUTC; PublishedTime=2021-04-02T06:01:18.387Z

Status: We've completed the development of the fix and are validating that it works without causing any further regressions. We expect to begin deployment early next week and anticipate full saturation by approximately April 16, 2021. We'll provide an update early next week to ensure the deployment begins as expected. Next update by: Tuesday, April 6, 2021, at 6:30 AMUTC; PublishedTime=2021-04-03T04:48:01.363Z

Status: The deployment of the fix is underway and we have validated it in our internal environment. We'll continue to monitor the deployment progress as it saturates the remaining affected infrastructure. We anticipate full saturation by approximately April 16, 2021. Next update by: Friday, April 9, 2021, at 6:30 AMUTC; PublishedTime=2021-04-06T04:55:06.307Z

Status: We've confirmed that the fix deployment is approximately 80% saturated throughout the affected environment and we still expect the fix to be completed by April 16, 2021; however, at the current rate of progress, it may complete sooner than expected. Next update by: Tuesday, April 13, 2021, at 6:30 AMUTC; PublishedTime=2021-04-09T04:53:48.973Z

Final status: We confirmed that the fix deployment completed. In order to fully remediate impact for your organization, you may need to edit your organization's rules for the fix to fully apply. Next update by: Tuesday, April 13, 2021, at 6:30 AMUTC; PublishedTime=2021-04-09T04:53:48.973Z

--------------------------------------------------------------------------------------------------
Title Users may be unable to view invoices, view/manage or purchase new subscriptions
ID MO246276
Service Microsoft 365 suite
Feature Portal
Status Service restored
Impact Users may have been unable to view invoices, view/manage existing subscriptions or purchase new subscriptions.
Severity
Start time 2021-03-25T00:30:00Z
End time 2021-03-25T02:13:00Z
Last update 2021-03-25T02:30:15.02Z
More info
Root cause A portion of infrastructure was performing below expected standards, resulting in users being unable to view or manage existing subscriptions or purchase new subscriptions. Next steps: - We're reviewing the impacted portion of infrastructure to better understand why the issue took place and how we can prevent any similar issues from reoccurring. This is the final update for the event.; PublishedTime=2021-03-25T02:27:23.56Z

MO246276 Details

Status: We're investigating a potential issue and checking for impact to your organization. We'll provide an update within 30 minutes.

Status: We've determined that a portion of infrastructure is performing below expected standards, resulting in issues with viewing and managing existing subscriptions or purchasing new subscriptions. We're rerouting traffic to alternate infrastructure to mitigate impact and once complete, we'll monitor service health to ensure service is fully restored. Next update by: Thursday, March 25, 2021, at 3:00 AMUTC; PublishedTime=2021-03-25T02:06:11.147Z

Final status: We've completed rerouting the traffic to alternate infrastructure and confirmed with internal monitoring that the service is fully restored. Next update by: Thursday, March 25, 2021, at 3:00 AMUTC; PublishedTime=2021-03-25T02:06:11.147Z

--------------------------------------------------------------------------------------------------
Title Admins are unable to view active sites in the SharePoint Online admin center
ID SP247748
Service SharePoint Online
Feature Tenant Admin
Status Service restored
Impact Admins were unable to view active sites in the SharePoint Online admin center.
Severity
Start time 2021-03-24T22:00:00Z
End time 2021-03-30T16:30:00Z
Last update 2021-03-30T16:49:14.93Z
More info
Root cause A recent service update to improve access reliability had prevented admins from viewing active sites in the SharePoint Online admin center. Next steps: - We're reviewing our update procedures to better understand why impact was produced and to help identify similar issues during our development and testing cycles. This is the final update for the event.; PublishedTime=2021-03-30T16:49:14.93Z

SP247748 Details

Status: We're reviewing trace logs and the details provided in the support case, to determine the next troubleshooting steps. Next update by: Tuesday, March 30, 2021, at 4:30 PM UTC; PublishedTime=2021-03-30T14:38:28.7Z

Status: We're continuing to investigate trace log data. In parallel, we're reviewing recent service updates to determine if recently deployed changes unexpectedly caused this problem. Next update by: Tuesday, March 30, 2021, at 6:30 PMUTC; PublishedTime=2021-03-30T15:53:27.027Z

Final status: We've determined that a recent service update that was designed to improve SharePoint Online access reliability had resulted in impact to admins attempting to view sites within the SharePoint Online admin center. We've reverted the offending update, and confirmed with previously affected admins that it has successfully remediated impact. Next update by: Tuesday, March 30, 2021, at 6:30 PMUTC; PublishedTime=2021-03-30T15:53:27.027Z

--------------------------------------------------------------------------------------------------
Title Users are unable to open Information Rights Management (IRM) messages via any Exchange Online connection method
ID EX250788
Service Exchange Online
Feature E-Mail and calendar access
Status Service restored
Impact Users may have been unable to open IRM messages via any Exchange Online connection method.
Severity Sev2
Start time 2021-03-24T15:00:00Z
End time 2021-04-22T15:45:00Z
Last update 2021-04-23T21:48:39.04Z
More info Admins could have set the attribute "AzureRMSLicensingEnabled" to 'True' as a workaround for this issue while impact was occurring. Final status: We've confirmed that the fix has completely saturated throughout the affected environment and that impact has been remediated. Scope of impact: This issue may have potentially affected any users attempting to open IRM encrypted email. Start time: Wednesday, March 24, 2021, at 3:00 PM UTC End time: Thursday, April 22, 2021, at 3:45 PM'UTC Root cause: A recent code update intended to secure IRM messaging contained a code regression that resulted in impact. Next steps: - We're reviewing our update procedures to better understand why impact to IRM messages wasn't identified prior to deployment and to avoid similar issues in the future. This is the final update for the event.; PublishedTime=2021-04-23T19:26:53.933Z
Root cause A recent code update intended to secure IRM messaging contained a code regression that resulted in impact. Next steps: - We're reviewing our update procedures to better understand why impact to IRM messages wasn't identified prior to deployment and to avoid similar issues in the future. This is the final update for the event.; PublishedTime=2021-04-23T19:26:53.933Z

EX250788 Details

Status: We're investigating a potential issue and checking for impact to your organization. We'll provide an update within 30 minutes.

Status: We're analyzing screenshots of message samples to better determine why users are unable to open IRM messages. We've also requested your organization reproduce the issue and provide us with the resulting data so we can better identify the underlying cause of the issue and develop our next troubleshooting steps. Next update by: Thursday, April 15, 2021, at 10:00 PMUTC; PublishedTime=2021-04-15T20:36:30.837Z

Status: We're continuing to analyze diagnostic data from your reproductions of the issue in order better isolate the root cause and determine a mitigation plan. Next update by: Friday, April 16, 2021, at 12:00 AMUTC; PublishedTime=2021-04-15T21:55:33.71Z

Status: We've determined that a recent code update intended to secure IRM messaging contained a code regression that resulted in this impact. We're developing a fix for this issue; however, due to the complexity of the code structure and the scope of the impact, this process is expected to take approximately one week to complete. Next update by: Monday, April 19, 2021, at 10:00 PMUTC; PublishedTime=2021-04-15T23:36:20.82Z

Status: We've completed development and validation of the fix, and deployment to the affected infrastructure has been initiated. Next update by: Wednesday, April 21, 2021, at 10:00 PMUTC; PublishedTime=2021-04-19T21:06:22.633Z

Status: We're continuing to deploy the fix to the affected infrastructure and now expect deployment to complete by Friday, April 23, 2021. We'll continue our monitoring to ensure the fix deploys as expected and completely remediates impact. Next update by: Friday, April 23, 2021, at 9:00 PMUTC; PublishedTime=2021-04-21T20:54:16.02Z

Final status: We've confirmed that the fix has completely saturated throughout the affected environment and that impact has been remediated. Next update by: Friday, April 23, 2021, at 9:00 PMUTC; PublishedTime=2021-04-21T20:54:16.02Z

--------------------------------------------------------------------------------------------------
Title Some admins may be unable to access the Exchange Online admin center
ID EX246396
Service Exchange Online
Feature E-Mail and calendar access
Status Service restored
Impact Admins may have been unable to access the Exchange Online admin center.
Severity
Start time 2021-03-24T11:30:00Z
End time 2021-03-25T23:45:00Z
Last update 2021-03-26T04:39:25.063Z
More info Admins trying to access the admin center may have seen the error message "User Auth Token Null". Final status: We've confirmed via telemetry that reverting our expansion has succeeded in mitigating impact. Scope of impact: Impact was specific to users who were served through the affected infrastructure. Start time: Wednesday, March 24, 2021, at 11:30 AM'UTC End time: Thursday, March 25, 2021, at 11:45 PM'UTC Root cause: Recently added regional infrastructure may have contributed to some admins being unable to access the Exchange Online admin center. Next steps: - We're reviewing the recently added regional infrastructure to understand why it caused this impact and help prevent similar problems when re-expanding in the future. This is the final update for the event.; PublishedTime=2021-03-26T04:15:54.937Z
Root cause Recently added regional infrastructure may have contributed to some admins being unable to access the Exchange Online admin center. Next steps: - We're reviewing the recently added regional infrastructure to understand why it caused this impact and help prevent similar problems when re-expanding in the future. This is the final update for the event.; PublishedTime=2021-03-26T04:15:54.937Z

EX246396 Details

Status: We're reviewing system logs to determine why these errors are occurring and help develop the next troubleshooting steps. Next update by: Thursday, March 25, 2021, at 4:30 PMUTC; PublishedTime=2021-03-25T15:08:35.1Z

Status: Our review of system logs indicates that expanding our service to include additional regional infrastructure may have contributed to some admins being unable to access the Exchange Online admin center. Moving forward, we've reverted our expansion, and we expect alleviation of impact to begin within the next 12 hours. In parallel, we're continuing to investigate viable methods for successful re-expansion to these new sectors. Next update by: Friday, March 26, 2021, at 5:00 AMUTC; PublishedTime=2021-03-25T16:30:44.39Z

Final status: We've confirmed via telemetry that reverting our expansion has succeeded in mitigating impact. Next update by: Friday, March 26, 2021, at 5:00 AMUTC; PublishedTime=2021-03-25T16:30:44.39Z

--------------------------------------------------------------------------------------------------
Title Users may have found that PowerPoint content was not being recorded in Microsoft Teams meetings
ID TM246126
Service Microsoft Teams
Feature Teams Components
Status Service restored
Impact Users may have found that PowerPoint content being shared within Microsoft Teams meetings was not being recorded.
Severity
Start time 2021-03-22T19:00:00Z
End time 2021-03-24T12:00:00Z
Last update 2021-03-24T12:25:20.767Z
More info
Root cause Impact was caused by an authentication issue introduced in a regularly scheduled update to our content sharing service. Next steps: - We're reviewing our update procedures to better identify similar issues during our development and testing cycles. This is the final update for the event.; PublishedTime=2021-03-24T12:25:20.767Z

TM246126 Details

Status: We're analyzing system diagnostics to determine why this specific content is not included within Microsoft Teams meeting recordings. Next update by: Wednesday, March 24, 2021, at 12:30 PM UTC; PublishedTime=2021-03-24T10:35:36.58Z

Final status: We've identified that impact was caused by an authentication issue, introduced in a regularly scheduled update to our content sharing service. We've reverted the update, and confirmed using our internal reproduction of the issue that impact has been mitigated. Next update by: Wednesday, March 24, 2021, at 12:30 PM UTC; PublishedTime=2021-03-24T10:35:36.58Z

--------------------------------------------------------------------------------------------------
Title Users are unable to access their favorited public folders from Outlook on the web
ID EX246391
Service Exchange Online
Feature E-Mail and calendar access
Status Service restored
Impact Users were unable to access their favorited public folders from Outlook on the web.
Severity
Start time 2021-03-22T13:05:00Z
End time 2021-03-25T22:50:00Z
Last update 2021-03-26T21:53:45.24Z
More info
Root cause A code issue contained within a recent service update was preventing users from accessing their favorited public folders from Outlook on the web. Next steps: -We're reviewing our service update procedures to help develop ways for identifying similar problems prior to their deployment in the future. This is the final update for the event.; PublishedTime=2021-03-26T00:29:19.597Z

EX246391 Details

Status: We're reviewing support case details from affected users to determine the next troubleshooting steps. Next update by: Thursday, March 25, 2021, at 4:00 PM UTC; PublishedTime=2021-03-25T14:08:48.437Z

Status: We've reviewed support case information and analyzed public folder mailboxes from which we've determined the issue likely lies within Outlook on the web. We're now investigating Outlook on the web system logs to isolate the root cause and determine next steps. Next update by: Thursday, March 25, 2021, at 6:00 PMUTC; PublishedTime=2021-03-25T15:51:08.957Z

Status: We've identified that a recently deployed update contained a code issue that's causing the problem. We're performing a code override to disable the offending code and remediate impact. The process is expected to take a few hours to complete. Next update by: Thursday, March 25, 2021, at 10:00 PMUTC; PublishedTime=2021-03-25T18:08:08.333Z

Status: We're continuing to monitor the deployment of the code override, and we see that it should take a few more hours to complete. Next update by: Friday, March 26, 2021, at 1:30 AMUTC; PublishedTime=2021-03-25T21:15:00.56Z

Final status: We confirmed that the deployment of the code override completed successfully and that the impact is remediated. Next update by: Friday, March 26, 2021, at 1:30 AMUTC; PublishedTime=2021-03-25T21:15:00.56Z

--------------------------------------------------------------------------------------------------
Title Admins experiencing permissions issues when creating new custom roles within Role Based Access Control (RBAC)
ID IT249381
Service Microsoft Intune
Feature Microsoft Intune
Status Service restored
Impact Admins may have noticed that permissions were incorrectly set when creating custom roles within Microsoft Intune's RBAC.
Severity
Start time 2021-03-22T13:01:00Z
End time 2021-04-08T15:20:00Z
Last update 2021-04-08T16:33:23.443Z
More info Impact was specific to new custom roles created through the Microsoft Endpoint Manager admin center. Final status: We've confirmed that the deployment of the fix has completed saturating throughout the remaining environments and after monitoring the service, we've verified that the fix has successfully resolved the problem. Scope of impact: Any admin creating new custom roles through the Microsoft Endpoint Manager admin center may have been impacted. Start time: Monday, March 22, 2021, at 1:01 PM'UTC End time: Thursday, April 8, 2021, at 3:20 PM'UTC Root cause: A recent update to the Microsoft Endpoint Manager admin center contained a code issue which resulted in misconfigured permissions for newly created custom roles. Next steps: - We're investigating why the code issue was missed during our update development and validation processes to prevent this problem from happening again in the future. This is the final update for the event.; PublishedTime=2021-04-08T16:27:02.533Z
Root cause A recent update to the Microsoft Endpoint Manager admin center contained a code issue which resulted in misconfigured permissions for newly created custom roles. Next steps: - We're investigating why the code issue was missed during our update development and validation processes to prevent this problem from happening again in the future. This is the final update for the event.; PublishedTime=2021-04-08T16:27:02.533Z

IT249381 Details

Status: We're investigating a potential issue and checking for impact to your organization. We'll provide an update within 30 minutes.; PublishedTime=2021-04-07T22:18:04.957

Status: We're reviewing recent service updates which may be contributing to impact. Next update by: Thursday, April 8, 2021, at 12:00 AMUTC; PublishedTime=2021-04-07T22:32:06.257Z

Status: We've confirmed that a code issue in the most recent service build update that was published caused the impact. We're developing a fix and we'll provide an estimated deployment timeline once available. Next update by: Thursday, April 8, 2021, at 2:00 AMUTC; PublishedTime=2021-04-07T23:23:04.16Z

Status: We're continuing to develop a fix for the code issue and expect to have this update deployed by Thursday, April 8, 2021, at 8:00 AM UTC. Next update by: Thursday, April 8, 2021, at 4:00 AMUTC; PublishedTime=2021-04-07T23:59:26.69Z

Status: We've completed development of our fix for the code issue and are beginning deployment. We now expect that the fix will saturate affected service environments by Thursday, April 8, 2021 at 4:00 PM UTC. Next update by: Thursday, April 8, 2021, at 10:00 AMUTC; PublishedTime=2021-04-08T02:32:52.323Z

Status: The deployment is progressing as expected and we can confirm that most users will already start to be experiencing relief. We'll continue to monitor the fix as it fully saturates across the remaining impacted environments. Next update by: Thursday, April 8, 2021, at 6:00 PM UTC ; PublishedTime=2021-04-08T08:06:05.983Z

Final status: We've confirmed that the deployment of the fix has completed saturating throughout the remaining environments and after monitoring the service, we've verified that the fix has successfully resolved the problem. Next update by: Thursday, April 8, 2021, at 6:00 PM UTC ; PublishedTime=2021-04-08T08:06:05.983Z

--------------------------------------------------------------------------------------------------
Title Admins can't create or edit policy sets that include device enrollment policies in the Microsoft Endpoint Manager (MEM)
ID IT246205
Service Microsoft Intune
Feature Microsoft Intune
Status Service restored
Impact Admins may have been unable to create or edit policy sets that include device enrollment policies in the MEM.
Severity
Start time 2021-03-22T10:35:00Z
End time 2021-03-25T02:00:00Z
Last update 2021-03-25T03:03:32.697Z
More info In order to get the most update to date experience, users may need to refresh their page or open a new tab in their browser. Final status: We confirmed that the deployment completed to all the affected environments and validated that the issue was fixed. Scope of impact: Impact was specific to admins attempting to create or edit policy sets that include device enrollment policies in the MEM. Start time: Monday, March 22, 2021, at 10:35 AM'UTC End time: Thursday, March 25, 2021, at 2:00 AM'UTC Root cause: An offending code regression tied to creating or editing policy sets that include device enrollment policies, and contained within a recent standard service update, was not fully repaired prior to deployment, leading to impact. Next steps: -We're analyzing our code check procedures for standard service updates to better identify similar issues prior to their deployment in the future. This is the final update for the event.; PublishedTime=2021-03-25T03:03:32.697Z
Root cause An offending code regression tied to creating or editing policy sets that include device enrollment policies, and contained within a recent standard service update, was not fully repaired prior to deployment, leading to impact. Next steps: -We're analyzing our code check procedures for standard service updates to better identify similar issues prior to their deployment in the future. This is the final update for the event.; PublishedTime=2021-03-25T03:03:32.697Z

IT246205 Details

Status: We're investigating a potential issue and checking for impact to your organization. We'll provide an update within 30 minutes.; PublishedTime=2021-03-24T20:23:09.623

Status: Our internal monitoring identified an issue preventing admins from creating or editing policy sets that include device enrollment policies in the MEM. Our investigation into recent standard service updates identified an offending code regression tied to this functionality which was not fully repaired prior to deployment. We've since repaired the associated code and are deploying it to all impacted environments. We're investigating potential ways to expedite this deployment to more quickly mitigate the impact to admins. Next update by: Wednesday, March 24, 2021, at 10:00 PMUTC; PublishedTime=2021-03-24T20:44:37.197Z

Status: We've confirmed the deployment of the fix is progressing as quickly as possible given our safe deployment practices, and admins will begin to experience mitigation as it saturates the affected environments. We'll provide regular updates through the deployment process. Next update by: Thursday, March 25, 2021, at 12:00 AMUTC; PublishedTime=2021-03-24T21:16:01.143Z

Status: The deployment of the fix is progressing as expected, and has completed saturating the Asia Pacific region as of Wednesday, March 24, 2021, at 9:45 PM UTC. Admins in this region can expect that the impact is now mitigated. The deployment to the remaining environments is ongoing, and we'll continue to provide regular updates throughout the process. Next update by: Thursday, March 25, 2021, at 2:30 AMUTC; PublishedTime=2021-03-24T22:53:56.7Z

Status: We've validated that the deployment in the Asia Pacific regions and the Europe, Middle East and Africa regions have completed. We're continuing the deployment process to the remaining impacted environments and will monitor the progress. Admins in the remaining affected environments may notice remediation as the fix continues to saturate. Next update by: Thursday, March 25, 2021, at 4:30 AMUTC; PublishedTime=2021-03-25T01:13:01.38Z

Final status: We confirmed that the deployment completed to all the affected environments and validated that the issue was fixed. Next update by: Thursday, March 25, 2021, at 4:30 AMUTC; PublishedTime=2021-03-25T01:13:01.38Z

--------------------------------------------------------------------------------------------------
Title Admins can't edit permissions for public folders within the Exchange Online admin center
ID EX247338
Service Exchange Online
Feature Management and Provisioning
Status Service restored
Impact Admins were unable to edit permissions for public folders within the Exchange Online admin center.
Severity
Start time 2021-03-20T07:00:00Z
End time 2021-03-30T20:43:00Z
Last update 2021-03-31T00:36:32.667Z
More info Admins were able to use PowerShell commands to edit permissions for public folders. For further details use the link below: https://docs.microsoft.com/en-us/exchange/troubleshoot/public-folders/can%E2%80%99t-apply -permissions-public-folder-subfolders Final status: Following the deployment of our fix, we've confirmed that impact has been resolved. Scope of impact: Impact was specific to admins attempting to edit permissions for public folders within the Exchange Online admin center. Start time: Saturday, March 20, 2021, at 7:00 AM'UTC End time: Tuesday, March 30, 2021, at 8:43 PM'UTC Root cause: A recent service update to improve latency optics is preventing admins from editing permissions for public folders within the Exchange Online admin center. This is the final update for the event.; PublishedTime=2021-03-30T21:20:49.427Z
Root cause A recent service update to improve latency optics is preventing admins from editing permissions for public folders within the Exchange Online admin center. This is the final update for the event.; PublishedTime=2021-03-30T21:20:49.427Z

EX247338 Details

Status: We're reviewing details provided in the support case to determine the next steps needed to resolve this issue. Next update by: Sunday, March 28, 2021, at 12:30 PM UTC; PublishedTime=2021-03-28T09:48:38.947Z

Status: We're continuing to review and analyze support case details along with server logs to identify the root cause of impact and develop a mitigation plan. Next update by: Friday, April 2, 2021, at 10:30 PM UTC; PublishedTime=2021-03-28T11:09:21.847Z

Status: Following our server log and support case review, we've identified a recent service update which is preventing admins from editing permissions for public folders within the Exchange Online admin center. We've deployed a fix, designed to revert the offending service update and we're confirming that it has remediated impact. Next update by: Friday, April 2, 2021, at 11:00 AMUTC; PublishedTime=2021-03-30T20:48:08.47Z

Final status: Following the deployment of our fix, we've confirmed that impact has been resolved. Next update by: Friday, April 2, 2021, at 11:00 AMUTC; PublishedTime=2021-03-30T20:48:08.47Z

--------------------------------------------------------------------------------------------------
Title Some users' presence status will appear as offline even though they are online
ID TM245382
Service Microsoft Teams
Feature Teams Components
Status Service restored
Impact Users' presence status may have appeared as offline even though they were online.
Severity
Start time 2021-03-19T21:00:00Z
End time 2021-03-20T02:25:00Z
Last update 2021-03-20T03:18:36.297Z
More info Affected users would have still been able to view other users' presence status. Final status: We've determined that the optimization method were successful and affected users' presence is now accurate. Scope of impact: Impact was specific to a small number of users who were served through the affected infrastructure. Start time: Friday, March 19, 2021, at 9:00 PM'UTC End time: Saturday, March 20, 2021, at 2:25 AM'UTC Root cause: A small portion of the Unified Presence infrastructure was performing below acceptable thresholds, resulting in presence lookup issues. Next steps: - We're investigating the affected infrastructure to determine why the performance was degraded and find was to prevent this from happening in the future. - We're reviewing our monitoring services to look for ways to reduce detection time and more quickly restore service. This is the final update for the event.; PublishedTime=2021-03-20T03:11:17.753Z
Root cause A small portion of the Unified Presence infrastructure was performing below acceptable thresholds, resulting in presence lookup issues. Next steps: - We're investigating the affected infrastructure to determine why the performance was degraded and find was to prevent this from happening in the future. - We're reviewing our monitoring services to look for ways to reduce detection time and more quickly restore service. This is the final update for the event.; PublishedTime=2021-03-20T03:11:17.753Z

TM245382 Details

Status: We're investigating a potential issue and checking for impact to your organization. We'll provide an update within 30 minutes.

Status: We've determined that a small portion of the Unified Presence infrastructure is performing below acceptable thresholds, resulting in presence lookup issues. We've applied optimization methods to improve infrastructure performance and subsequently mitigate impact. We expect impact to be mitigated for affected users with one hour. Next update by: Saturday, March 20, 2021, at 4:00 AMUTC; PublishedTime=2021-03-20T02:55:35.207Z

Final status: We've determined that the optimization method were successful and affected users' presence is now accurate. Next update by: Saturday, March 20, 2021, at 4:00 AMUTC; PublishedTime=2021-03-20T02:55:35.207Z

--------------------------------------------------------------------------------------------------
Title Some users can't open protected emails and documents
ID MO245049
Service Microsoft 365 suite
Feature Portal
Status Service restored
Impact Users may have been unable to open protected emails and documents in Microsoft 365 services.
Severity
Start time 2021-03-19T02:56:08Z
End time 2021-03-19T03:00:32Z
Last update 2021-03-19T03:14:48.097Z
More info
Root cause The service responsible for facilitating BYOK configurations was unexpectedly interrupted. Next steps: -We're analyzing the responsible service to help identify what triggered the interruptions and to help develop ways to prevent similar occurrences in the future. This is the final update for the event.; PublishedTime=2021-03-19T03:00:40.113Z

MO245049 Details

Final status: Our investigation determined that the service responsible for facilitating Bring Your Own Key (BYOK) configurations was temporarily interrupted. We rerouted service connections to alternate infrastructure to remediate the impact.

--------------------------------------------------------------------------------------------------
Title Some users may be unable to use multiple features within the Microsoft 365 service
ID MO244957
Service Microsoft 365 suite
Feature Portal
Status Post-incident report published
Impact Users may have been unable to use multiple features within the Microsoft 365 service.
Severity
Start time 2021-03-18T07:46:00Z
End time 2021-03-18T22:16:00Z
Last update 2021-03-25T19:24:37.95Z
More info This issue was an amalgamation of EX244931, EX244952, TM244945 and SP244939. This event impacted the following services: Exchange Online - Users may have been unable to use some features within the Exchange Online service. Microsoft Teams - Users may have been unable to search recent messages in the search bar or see suggestions as expected when searching names within Microsoft Teams. Additionally, users may have been unable to create or modify teams if "Enable MIP Label" flag was enabled. SharePoint Online - Users may have been unable to search within SharePoint Online. To Do: Users may have experienced access and sync issues with To Do. Final status: We completed additional restarts on service infrastructure located within Germany and after extensive monitoring, we've validated that service has been restored. Scope of impact: This issue could have affected any user located in EMEA if their request was routed through the affected segmented regional infrastructure. Reports indicated that users located in UK, France, Germany and Holland were disproportionately impacted. Start time: Thursday, March 18, 2021, at 7:46 AM UTC End time: Thursday, March 18, 2021, at 10:16 PM UTC Root cause: A load-balancing issue affecting shared infrastructure, responsible for cross-service search functionality and access to specific dependent features resulted in impact. Multiple Client Access Front End (CAFE) groups within the environment were not receiving any requests even though they were in a provisioned and active state. We've identified that a latent code bug within an internal service was exposed under a rare condition. This internal service gathers usage metrics, and then uploads aggregate utilization data to the overall load-balancing processing mechanism. Once the mechanism fails to receive a certain number of data uploads, the specific type of routing system being used at the time stops routing requests to that CAFE group believing it was unhealthy. This resulted in other CAFE groups within the environment receiving significantly increased volume of requests, causing impact. Furthermore, automated recovery features identified the impact occurring on some CAF' components, due to the higher volume of requests, and started to remove these from active service, allowing the components time to recover. However, in this rare scenario, this reduced the number of components available to handle requests, routing all user requests to the remaining active components. This created a cycle and exacerbated impact. Next steps: - For a more comprehensive list of next steps and actions, please refer to the Post Incident Review document. We'll publish a post-incident report within five business days.; PublishedTime=2021-03-18T22:21:48.007Z
Root cause A load-balancing issue affecting shared infrastructure, responsible for cross-service search functionality and access to specific dependent features resulted in impact. Multiple Client Access Front End (CAFE) groups within the environment were not receiving any requests even though they were in a provisioned and active state. We've identified that a latent code bug within an internal service was exposed under a rare condition. This internal service gathers usage metrics, and then uploads aggregate utilization data to the overall load-balancing processing mechanism. Once the mechanism fails to receive a certain number of data uploads, the specific type of routing system being used at the time stops routing requests to that CAFE group believing it was unhealthy. This resulted in other CAFE groups within the environment receiving significantly increased volume of requests, causing impact. Furthermore, automated recovery features identified the impact occurring on some CAF' components, due to the higher volume of requests, and started to remove these from active service, allowing the components time to recover. However, in this rare scenario, this reduced the number of components available to handle requests, routing all user requests to the remaining active components. This created a cycle and exacerbated impact. Next steps: - For a more comprehensive list of next steps and actions, please refer to the Post Incident Review document. We'll publish a post-incident report within five business days.; PublishedTime=2021-03-18T22:21:48.007Z

MO244957 Details

Status: We've been investigating a load-balancing issue within the infrastructure responsible for Exchange Online access and search functionality. We're rerouting traffic to alternate, healthy infrastructure to mitigate impact. In parallel, we're restarting affected service components to bring them online and help with the load balancing operations. Next update by: Thursday, March 18, 2021, at 4:30 PM UTC; PublishedTime=2021-03-18T15:20:50.907Z

Status: We're continuing our impact mitigation actions by restarting impacted service components once traffic rerouting efforts are complete. Our initial tests with these actions indicates service improvement and we're expanding this this fix to the remaining area of impact. Next update by: Thursday, March 18, 2021, at 5:30 PMUTC; PublishedTime=2021-03-18T16:39:22.717Z

Status: Our monitoring shows improved service reliability within the affected regions as we identify and restart affected service components. Users may begin to see reduced impact while we continue our efforts to fix the problem. Next update by: Thursday, March 18, 2021, at 6:30 PM UTC; PublishedTime=2021-03-18T17:34:05.12Z

Status: We've confirmed that service reliability is improving as we target and restart the affected service components. We expect that users in the impacted region will continue to see improved Microsoft 365 service functionality while we expand these efforts across the region. Next update by: Thursday, March 18, 2021, at 8:30 PM UTC; PublishedTime=2021-03-18T18:29:02.94Z

Status: We've identified that the majority of users in the affected regions will no longer experience this issue; however, we've identified that users in Germany are still affected by the event. We're examining the impacted infrastructure in the region to identify the cause of continued impact. Next update by: Thursday, March 18, 2021, at 10:30 PM UTC; PublishedTime=2021-03-18T20:40:00.147Z

Final status: We completed additional restarts on service infrastructure located within Germany and after extensive monitoring, we've validated that service has been restored. Next update by: Thursday, March 18, 2021, at 10:30 PM UTC; PublishedTime=2021-03-18T20:40:00.147Z

--------------------------------------------------------------------------------------------------
Title Users are unable to use the People Picker function in InfoPath forms within SharePoint Online
ID SP245941
Service SharePoint Online
Feature SharePoint Features
Status Service restored
Impact Users were unable to use the People Picker function in InfoPath forms within SharePoint Online.
Severity
Start time 2021-03-17T05:04:00Z
End time 2021-03-31T09:18:00Z
Last update 2021-03-31T17:05:47.437Z
More info
Root cause A recent service update intended to improve data processing for People Picker functionality was preventing users from using the People Picker function in InfoPath forms within SharePoint Online. Next steps: - We're reviewing our update deployment and validation procedures to determine why impact to People Picker functionality within InfoPath forms wasn't identified prior to deployment and to prevent issues like this from occurring in the future. This is the final update for the event.; PublishedTime=2021-03-31T17:05:47.437Z

SP245941 Details

Status: We're analyzing system logs to determine the source of the issue. Next update by: Tuesday, March 23, 2021, at 2:30 PM UTC; PublishedTime=2021-03-23T12:16:45.227Z

Status: We've identified that a portion of code, introduced in a recent deployment, has unexpectedly resulted in impact. We've developed a fix to revert these changes, and are performing tests within our internal testing environment before deploying it to the affected environment. Next update by: Wednesday, March 24, 2021, at 6:30 PM UTC; PublishedTime=2021-03-23T13:46:29.5Z

Status: We've developed and internally validated our fix which is designed to revert the recent offending service update and remediate impact. We've deployed our fix and are monitoring as it saturates the affected environment. We're expecting the ability to provide an estimate for the competition of the fix deployment and remediation of impact during our next scheduled update. Next update by: Thursday, March 25, 2021, at 6:30 PMUTC; PublishedTime=2021-03-24T17:02:57.467Z

Status: Due to the nature of the fix, it's deploying more slowly than expected. We're continuing to monitor as it progresses through the affected environments to determine when the deployment is likely to complete. Next update by: Friday, March 26, 2021, at 6:30 PMUTC; PublishedTime=2021-03-25T17:25:17.687Z

Status: As the deployment progresses, we've confirmed that the fix has resolved the issue for some affected users, and we expect for it to complete and the issue be fully resolved for all affected users by our next update. Next update by: Monday, March 29, 2021, at 6:30 PMUTC; PublishedTime=2021-03-26T16:54:56.46Z

Status: The deployment of our fix to restore users' ability to use the People Picker function in InfoPath within SharePoint Online is taking longer than expected. Moving forward, we're continuing to monitor the deployment process to ensure the fix propagates successfully. Next update by: Monday, March 29, 2021, at 8:30 PMUTC; PublishedTime=2021-03-29T17:00:39.603Z

Status: The fix has successfully propagated for the majority of impacted users, though it appears a portion of infrastructure may still be experiencing some impact due to the fix not yet being fully deployed. As we continue to monitor the deployment progress, we're investigating ways to better expedite our deployment of the fix to the remaining impacted environment. Next update by: Tuesday, March 30, 2021, at 11:00 PMUTC; PublishedTime=2021-03-29T19:49:51.523Z

Status: Our deployment of the fix continues, and we anticipate the final affected portions of infrastructure will receive the fix by our next update. We're monitoring the fix deployment progress and are continuing to review options to expedite the fix should delays continue. Next update by: Wednesday, March 31, 2021, at 11:00 PMUTC; PublishedTime=2021-03-30T21:22:34.703Z

Final status: The deployment of the fix is complete, and we've confirmed with affected users that this issue is resolved. Next update by: Wednesday, March 31, 2021, at 11:00 PMUTC; PublishedTime=2021-03-30T21:22:34.703Z

--------------------------------------------------------------------------------------------------
Title Users can't access public folder calendars in Outlook on the web
ID EX247565
Service Exchange Online
Feature E-Mail and calendar access
Status Service restored
Impact Users were unable to access public folder calendars in Outlook on the web.
Severity
Start time 2021-03-16T14:00:00Z
End time 2021-03-31T03:20:00Z
Last update 2021-03-31T04:12:49.937Z
More info While we were focused on remediation, users that could access the Outlook desktop client could have used this connection method to access the affected public folder calendars. Final status: We've validated the fix and have deployed it to the affected infrastructure. Our internal testing shows that users can now access public folders and the issue has been mitigated. Scope of impact: This problem may have impacted any of your users that were accessing public folder calendars in Outlook on the web. Start time: Tuesday, March 16, 2021, at 2:00 PM'UTC End time: Wednesday, March 31, 2021, at 3:20 AM'UTC Root cause: An Outlook on the web update inadvertently introduced a code regression, which was preventing users from accessing public folder calendars in Outlook on the web. Next steps: - We're reviewing our Outlook on the web update procedures to better identify similar issues during our development and testing cycles. This is the final update for the event.; PublishedTime=2021-03-31T04:12:49.937Z
Root cause An Outlook on the web update inadvertently introduced a code regression, which was preventing users from accessing public folder calendars in Outlook on the web. Next steps: - We're reviewing our Outlook on the web update procedures to better identify similar issues during our development and testing cycles. This is the final update for the event.; PublishedTime=2021-03-31T04:12:49.937Z

EX247565 Details

Status: We're investigating a potential issue and checking for impact to your organization. We'll provide an update within 60 minutes.; PublishedTime=2021-03-29T15:54:45.993

Status: We're analyzing system logs and a group of affected public folder calendars as we determine the source for this event. Next update by: Monday, March 29, 2021, at 6:30 PMUTC; PublishedTime=2021-03-29T16:36:51.323Z

Status: We've internally reproduced the issue and are reviewing the associated source code. In parallel, we're reviewing recent service updates which may have introduced this issue, as we determine our next steps for remediating impact. Next update by: Monday, March 29, 2021, at 8:30 PMUTC; PublishedTime=2021-03-29T17:59:54.55Z

Status: Our review of the reproduction of the problem and associated source code has focused our investigation on a possible mailbox migration issue that could be affecting access to public folder calendars in Outlook on the web. We're further investigating recent mailbox migrations while we continue to review recent service updates that may have introduced the problem. Next update by: Monday, March 29, 2021, at 10:30 PMUTC; PublishedTime=2021-03-29T19:56:22.18Z

Status: After confirming that a mailbox migration issue had not occurred, we've identified a caching issue between Outlook on the web and a subsection of Active Directory (AD) service infrastructure. We're attempting to internally confirm our findings before applying the needed remediation action to the affected service infrastructure. Next update by: Tuesday, March 30, 2021, at 12:30 AMUTC; PublishedTime=2021-03-29T22:04:09.68Z

Status: Through further review, we've determined that an Outlook on the web update inadvertently introduced a code regression, which is preventing users from accessing public folder calendars in Outlook on the web. We're in the process of isolating the code regression to identify a mitigation plan. Next update by: Tuesday, March 30, 2021, at 2:00 AMUTC; PublishedTime=2021-03-30T00:26:30.42Z

Status: We've identified the code regression and we're in the process of developing a code fix to mitigate impact. Next update by: Tuesday, March 30, 2021, at 6:00 AM UTC; PublishedTime=2021-03-30T01:52:30.46Z

Status: We've developed a code fix and have started deploying it to the affected environment. Next update by: Tuesday, March 30, 2021, at 5:30 PMUTC; PublishedTime=2021-03-30T04:20:16.063Z

Status: The deployment of the code fix completed, however, we've determined it didn't fully resolve the issue. We're investigating why the code fix was not successful as well as the remaining failures to determine our next steps. Next update by: Tuesday, March 30, 2021, at 8:00 PMUTC; PublishedTime=2021-03-30T16:22:24.727Z

Status: We've developed a secondary code fix to to address the remaining impact that wasn't mitigated by our original solution. We suspect that validation and testing of this fix will complete by our next update, at which point we'll also provide an estimated deployment timeline. Next update by: Tuesday, March 30, 2021, at 10:30 PMUTC; PublishedTime=2021-03-30T18:28:14.387Z

Status: We're running additional tests to validate the secondary fix before deploying it to the affected environments. We suspect that the fix should be ready to deploy within the next three hours, at which point we'll also provide an estimated deployment timeline. Next update by: Wednesday, March 31, 2021, at 1:30 AMUTC; PublishedTime=2021-03-30T21:24:28.253Z

Status: The fix validation process is taking longer than anticipated. We expect to complete validation and begin the fix deployment within the next five hours. Next update by: Wednesday, March 31, 2021, at 6:00 AMUTC; PublishedTime=2021-03-31T01:16:29.7Z

Final status: We've validated the fix and have deployed it to the affected infrastructure. Our internal testing shows that users can now access public folders and the issue has been mitigated. Next update by: Wednesday, March 31, 2021, at 6:00 AMUTC; PublishedTime=2021-03-31T01:16:29.7Z

--------------------------------------------------------------------------------------------------
Title Admins can't change OneDrive for Business quota settings via the Microsoft 365 admin center
ID SP245711
Service SharePoint Online
Feature Tenant Admin
Status Post-incident report published
Impact Admins were seeing errors when trying to change OneDrive for Business quota settings via the Microsoft 365 admin center.
Severity
Start time 2021-03-16T11:07:00Z
End time 2021-03-31T18:46:00Z
Last update 2021-04-07T18:10:16.373Z
More info Admins may have seen the following error, "We couldn't save your changes. Please try again." As an alternative method, admins could have changed quotas using PowerShell. For more information on this process, please go to: https://docs.microsoft.com/en-us/powershell/mo dule/sharepoint-online/set-sposite'view=sharepoint-ps Final status: We deployed a fix to the affected environment and have validated that service has been restored. Scope of impact: All admins attempting to change OneDrive for Business quota settings through the admin center would have experienced impact. Start time: Tuesday, March 16, 2021, at 11:07 AM UTC End time: Wednesday, March 31, 2021, at 6:46 PM UTC Root cause: A code change, designed to streamline quota management by addressing an existing misconfiguration, unexpectedly introduced a code regression. This caused the system to generate an exception, which resulted in errors when admins tried to change quota settings. Next Steps: - For a more comprehensive list of next steps and actions, please refer to the Post Incident Review document. We'll publish a post-incident report within five business days.; PublishedTime=2021-03-31T21:06:34.737Z
Root cause A code change, designed to streamline quota management by addressing an existing misconfiguration, unexpectedly introduced a code regression. This caused the system to generate an exception, which resulted in errors when admins tried to change quota settings. Next Steps: - For a more comprehensive list of next steps and actions, please refer to the Post Incident Review document. We'll publish a post-incident report within five business days.; PublishedTime=2021-03-31T21:06:34.737Z

SP245711 Details

Status: We're reviewing trace logs provided by affected users to determine the root cause of the issue. Next update by: Monday, March 22, 2021, at 11:30 AM UTC; PublishedTime=2021-03-22T09:47:10.193Z

Status: We're continuing to analyze the provided trace logs along with our internal reproduction of the issue to help isolate the source of impact and develop a mitigation plan. Next update by: Monday, March 22, 2021, at 1:30 PM UTC; PublishedTime=2021-03-22T11:05:15.313Z

Status: Our investigation of the trace logs and internal reproduction of the issue has proved inconclusive. We're gathering additional details from log reviews to determine the next troubleshooting steps. Next update by: Monday, March 22, 2021, at 3:30 PM UTC; PublishedTime=2021-03-22T13:11:54.83Z

Status: We're continuing our work to reproduce this issue internally as well as gather Unified Logging Service (ULS) logs to assist in the investigation. Next update by: Monday, March 22, 2021, at 6:00 PMUTC; PublishedTime=2021-03-22T15:13:22.92Z

Status: Our review of ULS logs indicates a potential authentication issue, preventing an API that facilitates OneDrive configuration changes. We're expanding our review of ULS logs to correlate with system logs to narrow the investigation and confirm the root cause. Next update by: Monday, March 22, 2021, at 9:00 PMUTC; PublishedTime=2021-03-22T17:08:33.3Z

Status: The ULS and system logs have confirmed that when performing the quota settings changes via PowerShell the actions are performed with a different set of permissions than when performed via the admin center. We're continuing our investigation into how the permissions were changed and are developing a fix to mitigate impact. We expect to have a timeline for this fix by the next scheduled update. Next update by: Tuesday, March 23, 2021, at 7:30 PMUTC; PublishedTime=2021-03-22T20:11:38.09Z

Status: Due to the complexity of the issue, we're continuing to investigate how the permissions were changed and test and validate a potential fix. We're unable to provide a fix timeline with this update but will provide one as soon it becomes available. Next update by: Tuesday, March 23, 2021, at 10:30 PMUTC; PublishedTime=2021-03-23T18:01:30.453Z

Status: We've determined that a recent deployment inadvertently introduced a code misconfiguration, resulting in impact. We've developed a fix which is scheduled for deployment soon and expected to complete by our next scheduled meeting. Next update by: Friday, March 26, 2021, at 7:30 PMUTC; PublishedTime=2021-03-23T22:09:18.297Z

Status: We've initiated deployment of our fix, and are monitoring its progress as it saturates throughout the affected infrastructure. We'll provide an updated timeline for completion of the deployment with our next update. Next update by: Monday, March 29, 2021, at 8:00 PMUTC; PublishedTime=2021-03-26T18:12:26.817Z

Status: We're reviewing the ongoing fix deployment progress and checking with specific affected users who've received the fix to validate whether the issue persists, which will help us determine our next steps. Next update by: Monday, March 29, 2021, at 10:00 PMUTC; PublishedTime=2021-03-29T19:53:22.26Z

Status: We've confirmed that our fix to correct the affecting code misconfiguration was unsuccessful in resolving the problem. We've isolated an error we suspect may have preventing the fix from being successful, and we're reviewing our options to develop and re-deploy a code fix to mitigate the impact as soon as possible. Next update by: Tuesday, March 30, 2021, at 12:00 AM UTC; PublishedTime=2021-03-29T21:57:37.63Z

Status: We're continuing to determine if our proposed mitigation strategy will work to properly resolve the issue. If successful, deployment of the fix will begin within the next 24 hours. Next update by: Tuesday, March 30, 2021, at 2:00 AMUTC; PublishedTime=2021-03-29T23:46:59.193Z

Status: We're working to test our proposed mitigation strategy for resolving the issue internally. Once we've confirmed it is successful, we'll begin deployment of the fix to the affected environment. Next update by: Tuesday, March 30, 2021, at 4:00 AMUTC; PublishedTime=2021-03-30T01:55:40.027Z

Status: Validating our solution is taking longer than anticipated. In an effort to provide more expedient relief, we're deploying a more robust service build that has the potential to mitigate impact. Next update by: Tuesday, March 30, 2021, at 7:00 PMUTC; PublishedTime=2021-03-30T03:54:28.223Z

Status: Our internal validation of the fix is ongoing. As validation continues, we're reviewing our options to deploy the fix to the affected environment as efficiently as possible. We'll provide an ETA for the deployment as soon as it's available. Next update by: Tuesday, March 30, 2021, at 10:00 PMUTC; PublishedTime=2021-03-30T18:26:16.847Z

Status: We're continuing our validations to confirm our fix mitigates impact as expected. We expect to begin deploying the fix today, March 30, 2021, and anticipate completion by our next update, although the timeline is subject to change. Next update by: Thursday, April 1, 2021, at 8:00 PMUTC; PublishedTime=2021-03-30T20:40:37.113Z

Final status: We deployed a fix to the affected environment and have validated that service has been restored. Next update by: Thursday, April 1, 2021, at 8:00 PMUTC; PublishedTime=2021-03-30T20:40:37.113Z

--------------------------------------------------------------------------------------------------
Title Admins can't change OneDrive for Business quota settings via the Microsoft 365 admin center
ID SP245711
Service SharePoint Online
Feature Tenant Admin
Status Post-incident report published
Impact Admins were seeing errors when trying to change OneDrive for Business quota settings via the Microsoft 365 admin center.
Severity
Start time 2021-03-16T11:07:00Z
End time 2021-03-31T18:46:00Z
Last update 2021-04-07T18:10:16.373Z
More info Admins may have seen the following error, "We couldn't save your changes. Please try again." As an alternative method, admins could have changed quotas using PowerShell. For more information on this process, please go to: https://docs.microsoft.com/en-us/powershell/mo dule/sharepoint-online/set-sposite'view=sharepoint-ps Final status: We deployed a fix to the affected environment and have validated that service has been restored. Scope of impact: All admins attempting to change OneDrive for Business quota settings through the admin center would have experienced impact. Start time: Tuesday, March 16, 2021, at 11:07 AM UTC End time: Wednesday, March 31, 2021, at 6:46 PM UTC Root cause: A code change, designed to streamline quota management by addressing an existing misconfiguration, unexpectedly introduced a code regression. This caused the system to generate an exception, which resulted in errors when admins tried to change quota settings. Next Steps: - For a more comprehensive list of next steps and actions, please refer to the Post Incident Review document. We'll publish a post-incident report within five business days.; PublishedTime=2021-03-31T21:06:34.737Z
Root cause A code change, designed to streamline quota management by addressing an existing misconfiguration, unexpectedly introduced a code regression. This caused the system to generate an exception, which resulted in errors when admins tried to change quota settings. Next Steps: - For a more comprehensive list of next steps and actions, please refer to the Post Incident Review document. We'll publish a post-incident report within five business days.; PublishedTime=2021-03-31T21:06:34.737Z

SP245711 Details

Status: We're reviewing trace logs provided by affected users to determine the root cause of the issue. Next update by: Monday, March 22, 2021, at 11:30 AM UTC; PublishedTime=2021-03-22T09:47:10.193Z

Status: We're continuing to analyze the provided trace logs along with our internal reproduction of the issue to help isolate the source of impact and develop a mitigation plan. Next update by: Monday, March 22, 2021, at 1:30 PM UTC; PublishedTime=2021-03-22T11:05:15.313Z

Status: Our investigation of the trace logs and internal reproduction of the issue has proved inconclusive. We're gathering additional details from log reviews to determine the next troubleshooting steps. Next update by: Monday, March 22, 2021, at 3:30 PM UTC; PublishedTime=2021-03-22T13:11:54.83Z

Status: We're continuing our work to reproduce this issue internally as well as gather Unified Logging Service (ULS) logs to assist in the investigation. Next update by: Monday, March 22, 2021, at 6:00 PMUTC; PublishedTime=2021-03-22T15:13:22.92Z

Status: Our review of ULS logs indicates a potential authentication issue, preventing an API that facilitates OneDrive configuration changes. We're expanding our review of ULS logs to correlate with system logs to narrow the investigation and confirm the root cause. Next update by: Monday, March 22, 2021, at 9:00 PMUTC; PublishedTime=2021-03-22T17:08:33.3Z

Status: The ULS and system logs have confirmed that when performing the quota settings changes via PowerShell the actions are performed with a different set of permissions than when performed via the admin center. We're continuing our investigation into how the permissions were changed and are developing a fix to mitigate impact. We expect to have a timeline for this fix by the next scheduled update. Next update by: Tuesday, March 23, 2021, at 7:30 PMUTC; PublishedTime=2021-03-22T20:11:38.09Z

Status: Due to the complexity of the issue, we're continuing to investigate how the permissions were changed and test and validate a potential fix. We're unable to provide a fix timeline with this update but will provide one as soon it becomes available. Next update by: Tuesday, March 23, 2021, at 10:30 PMUTC; PublishedTime=2021-03-23T18:01:30.453Z

Status: We've determined that a recent deployment inadvertently introduced a code misconfiguration, resulting in impact. We've developed a fix which is scheduled for deployment soon and expected to complete by our next scheduled meeting. Next update by: Friday, March 26, 2021, at 7:30 PMUTC; PublishedTime=2021-03-23T22:09:18.297Z

Status: We've initiated deployment of our fix, and are monitoring its progress as it saturates throughout the affected infrastructure. We'll provide an updated timeline for completion of the deployment with our next update. Next update by: Monday, March 29, 2021, at 8:00 PMUTC; PublishedTime=2021-03-26T18:12:26.817Z

Status: We're reviewing the ongoing fix deployment progress and checking with specific affected users who've received the fix to validate whether the issue persists, which will help us determine our next steps. Next update by: Monday, March 29, 2021, at 10:00 PMUTC; PublishedTime=2021-03-29T19:53:22.26Z

Status: We've confirmed that our fix to correct the affecting code misconfiguration was unsuccessful in resolving the problem. We've isolated an error we suspect may have preventing the fix from being successful, and we're reviewing our options to develop and re-deploy a code fix to mitigate the impact as soon as possible. Next update by: Tuesday, March 30, 2021, at 12:00 AM UTC; PublishedTime=2021-03-29T21:57:37.63Z

Status: We're continuing to determine if our proposed mitigation strategy will work to properly resolve the issue. If successful, deployment of the fix will begin within the next 24 hours. Next update by: Tuesday, March 30, 2021, at 2:00 AMUTC; PublishedTime=2021-03-29T23:46:59.193Z

Status: We're working to test our proposed mitigation strategy for resolving the issue internally. Once we've confirmed it is successful, we'll begin deployment of the fix to the affected environment. Next update by: Tuesday, March 30, 2021, at 4:00 AMUTC; PublishedTime=2021-03-30T01:55:40.027Z

Status: Validating our solution is taking longer than anticipated. In an effort to provide more expedient relief, we're deploying a more robust service build that has the potential to mitigate impact. Next update by: Tuesday, March 30, 2021, at 7:00 PMUTC; PublishedTime=2021-03-30T03:54:28.223Z

Status: Our internal validation of the fix is ongoing. As validation continues, we're reviewing our options to deploy the fix to the affected environment as efficiently as possible. We'll provide an ETA for the deployment as soon as it's available. Next update by: Tuesday, March 30, 2021, at 10:00 PMUTC; PublishedTime=2021-03-30T18:26:16.847Z

Status: We're continuing our validations to confirm our fix mitigates impact as expected. We expect to begin deploying the fix today, March 30, 2021, and anticipate completion by our next update, although the timeline is subject to change. Next update by: Thursday, April 1, 2021, at 8:00 PMUTC; PublishedTime=2021-03-30T20:40:37.113Z

Final status: We deployed a fix to the affected environment and have validated that service has been restored. Next update by: Thursday, April 1, 2021, at 8:00 PMUTC; PublishedTime=2021-03-30T20:40:37.113Z

--------------------------------------------------------------------------------------------------
Title Microsoft Teams reports have been delayed since March 14, 2021
ID MO244918
Service Microsoft 365 suite
Feature Administration
Status Service restored
Impact Admins may have noticed that the Microsoft Teams report had not updated since March 14, 2021.
Severity
Start time 2021-03-16T00:00:00Z
End time 2021-03-21T10:00:00Z
Last update 2021-03-22T06:17:32.187Z
More info
Root cause An issue affecting underlying Azure Active Directory (AAD) infrastructure resulted in a delay of upstream data source of Microsoft Teams. Next steps: - To help prevent similar impact in the future, we're further determining the underlying cause of the issue that affected Azure Active Directory infrastructure. This is the final update for the event.; PublishedTime=2021-03-22T06:17:32.187Z

MO244918 Details

Status: Our monitoring systems alerted us of an issue in which the upstream data source of Microsoft Teams reports is delayed, resulting in reports delayed since March 14, 2021. We're waiting for the data to automatically sync, and we expect to be able to provide an accurate timeline of mitigation during our next communications update. Next update by: Monday, March 22, 2021, at 8:00 AMUTC; PublishedTime=2021-03-18T04:03:08.027Z

Final status: The data has finished automatically syncing and we've confirmed via telemetry that service is restored. Next update by: Monday, March 22, 2021, at 8:00 AMUTC; PublishedTime=2021-03-18T04:03:08.027Z

--------------------------------------------------------------------------------------------------
Title Continuing Service Recovery from MO244568
ID IT244611
Service Microsoft Intune
Feature Microsoft Intune
Status Service restored
Impact Admins may have experienced intermittent impact for various Intune features.
Severity
Start time 2021-03-15T19:15:00Z
End time 2021-03-16T21:27:00Z
Last update 2021-03-16T22:06:54.623Z
More info - Admins may have notice enrollment failures as devices retry enrollment - Applications may have not all be shown in the UX or users may have see 401 errors - Compliance checks may have not be successful - Exporting reports may have fail - Admins may have notice their tenant's devices were not visible within the Intune admin center Final status: We've confirmed that the re-deployment is complete and we've verified internally that the issue has been resolved. Scope of impact: This issue may have affected any admins. Start time: Monday, March 15, 2021, at 7:15 PM'UTC End time: Tuesday, March 16, 2021, at 9:27 PM'UTC Root cause: Preliminary information on the underlying root cause is subject to change and may be found here: https://status.azure.com/en-us/status/history/ Next steps: - We're reviewing our monitoring and alerting processes to more quickly identify Azure Active Directory (AAD) service dependent functionality problems in the future. This is the final update for the event.; PublishedTime=2021-03-16T22:06:54.623Z
Root cause Preliminary information on the underlying root cause is subject to change and may be found here: https://status.azure.com/en-us/status/history/ Next steps: - We're reviewing our monitoring and alerting processes to more quickly identify Azure Active Directory (AAD) service dependent functionality problems in the future. This is the final update for the event.; PublishedTime=2021-03-16T22:06:54.623Z

IT244611 Details

Status: We're working on restoring functionality for end-to-end service delivery. Additional details can be found here: MO244568 in their Microsoft 365 admin portal or https://status.office365.com/. Next update by: Tuesday, March 16, 2021, at 5:00 PMUTC; PublishedTime=2021-03-16T00:36:56.837Z

Status: We've determined that a service re-deployment will properly mitigate impact related to MO244568. We expect this process to be fully completed by Tuesday, March 16, 2021 at 9:00 PM UTC, your service instance may be updated prior. Additional details surrounding the root cause can be found here: MO244568 in their Microsoft 365 admin portal or https://status.office365.com/. Next update by: Tuesday, March 16, 2021, at 5:00 PMUTC; PublishedTime=2021-03-16T05:40:37.883Z

Status: We're continuing with the re-deployment to mitigate impact related to MO244568. We expect this process to be fully completed by Tuesday, March 16, 2021 at 9:00 PM UTC, your service instance may be updated prior. Additional details surrounding the root cause can be found here: MO244568 in their Microsoft 365 admin portal or https://status.office365.com/. Next update by: Tuesday, March 16, 2021, at 9:00 PM UTC; PublishedTime=2021-03-16T14:38:46.277Z

Status: Our re-deployment to mitigate impact related to MO244568 is taking longer than initially expected. We anticipate the final stages of deployment and the validation process will be fully completed by Wednesday, March 17, 2021 at 12:00 AM UTC. Additional details surrounding the root cause can be found within MO244568 in the Microsoft 365 admin center. Next update by: Wednesday, March 17, 2021, at 12:00 AMUTC; PublishedTime=2021-03-16T19:23:50.057Z

Final status: We've confirmed that the re-deployment is complete and we've verified internally that the issue has been resolved. Next update by: Wednesday, March 17, 2021, at 12:00 AMUTC; PublishedTime=2021-03-16T19:23:50.057Z

--------------------------------------------------------------------------------------------------
Title Continuing Service Recovery from MO244568
ID IT244611
Service Microsoft Intune
Feature Microsoft Intune
Status Service restored
Impact Admins may have experienced intermittent impact for various Intune features.
Severity
Start time 2021-03-15T19:15:00Z
End time 2021-03-16T21:27:00Z
Last update 2021-03-16T22:06:54.623Z
More info - Admins may have notice enrollment failures as devices retry enrollment - Applications may have not all be shown in the UX or users may have see 401 errors - Compliance checks may have not be successful - Exporting reports may have fail - Admins may have notice their tenant's devices were not visible within the Intune admin center Final status: We've confirmed that the re-deployment is complete and we've verified internally that the issue has been resolved. Scope of impact: This issue may have affected any admins. Start time: Monday, March 15, 2021, at 7:15 PM'UTC End time: Tuesday, March 16, 2021, at 9:27 PM'UTC Root cause: Preliminary information on the underlying root cause is subject to change and may be found here: https://status.azure.com/en-us/status/history/ Next steps: - We're reviewing our monitoring and alerting processes to more quickly identify Azure Active Directory (AAD) service dependent functionality problems in the future. This is the final update for the event.; PublishedTime=2021-03-16T22:06:54.623Z
Root cause Preliminary information on the underlying root cause is subject to change and may be found here: https://status.azure.com/en-us/status/history/ Next steps: - We're reviewing our monitoring and alerting processes to more quickly identify Azure Active Directory (AAD) service dependent functionality problems in the future. This is the final update for the event.; PublishedTime=2021-03-16T22:06:54.623Z

IT244611 Details

Status: We're working on restoring functionality for end-to-end service delivery. Additional details can be found here: MO244568 in their Microsoft 365 admin portal or https://status.office365.com/. Next update by: Tuesday, March 16, 2021, at 5:00 PMUTC; PublishedTime=2021-03-16T00:36:56.837Z

Status: We've determined that a service re-deployment will properly mitigate impact related to MO244568. We expect this process to be fully completed by Tuesday, March 16, 2021 at 9:00 PM UTC, your service instance may be updated prior. Additional details surrounding the root cause can be found here: MO244568 in their Microsoft 365 admin portal or https://status.office365.com/. Next update by: Tuesday, March 16, 2021, at 5:00 PMUTC; PublishedTime=2021-03-16T05:40:37.883Z

Status: We're continuing with the re-deployment to mitigate impact related to MO244568. We expect this process to be fully completed by Tuesday, March 16, 2021 at 9:00 PM UTC, your service instance may be updated prior. Additional details surrounding the root cause can be found here: MO244568 in their Microsoft 365 admin portal or https://status.office365.com/. Next update by: Tuesday, March 16, 2021, at 9:00 PM UTC; PublishedTime=2021-03-16T14:38:46.277Z

Status: Our re-deployment to mitigate impact related to MO244568 is taking longer than initially expected. We anticipate the final stages of deployment and the validation process will be fully completed by Wednesday, March 17, 2021 at 12:00 AM UTC. Additional details surrounding the root cause can be found within MO244568 in the Microsoft 365 admin center. Next update by: Wednesday, March 17, 2021, at 12:00 AMUTC; PublishedTime=2021-03-16T19:23:50.057Z

Final status: We've confirmed that the re-deployment is complete and we've verified internally that the issue has been resolved. Next update by: Wednesday, March 17, 2021, at 12:00 AMUTC; PublishedTime=2021-03-16T19:23:50.057Z

--------------------------------------------------------------------------------------------------
Title Continuing Service Recovery from MO244568
ID IT244611
Service Microsoft Intune
Feature Microsoft Intune
Status Service restored
Impact Admins may have experienced intermittent impact for various Intune features.
Severity
Start time 2021-03-15T19:15:00Z
End time 2021-03-16T21:27:00Z
Last update 2021-03-16T22:06:54.623Z
More info - Admins may have notice enrollment failures as devices retry enrollment - Applications may have not all be shown in the UX or users may have see 401 errors - Compliance checks may have not be successful - Exporting reports may have fail - Admins may have notice their tenant's devices were not visible within the Intune admin center Final status: We've confirmed that the re-deployment is complete and we've verified internally that the issue has been resolved. Scope of impact: This issue may have affected any admins. Start time: Monday, March 15, 2021, at 7:15 PM'UTC End time: Tuesday, March 16, 2021, at 9:27 PM'UTC Root cause: Preliminary information on the underlying root cause is subject to change and may be found here: https://status.azure.com/en-us/status/history/ Next steps: - We're reviewing our monitoring and alerting processes to more quickly identify Azure Active Directory (AAD) service dependent functionality problems in the future. This is the final update for the event.; PublishedTime=2021-03-16T22:06:54.623Z
Root cause Preliminary information on the underlying root cause is subject to change and may be found here: https://status.azure.com/en-us/status/history/ Next steps: - We're reviewing our monitoring and alerting processes to more quickly identify Azure Active Directory (AAD) service dependent functionality problems in the future. This is the final update for the event.; PublishedTime=2021-03-16T22:06:54.623Z

IT244611 Details

Status: We're working on restoring functionality for end-to-end service delivery. Additional details can be found here: MO244568 in their Microsoft 365 admin portal or https://status.office365.com/. Next update by: Tuesday, March 16, 2021, at 5:00 PMUTC; PublishedTime=2021-03-16T00:36:56.837Z

Status: We've determined that a service re-deployment will properly mitigate impact related to MO244568. We expect this process to be fully completed by Tuesday, March 16, 2021 at 9:00 PM UTC, your service instance may be updated prior. Additional details surrounding the root cause can be found here: MO244568 in their Microsoft 365 admin portal or https://status.office365.com/. Next update by: Tuesday, March 16, 2021, at 5:00 PMUTC; PublishedTime=2021-03-16T05:40:37.883Z

Status: We're continuing with the re-deployment to mitigate impact related to MO244568. We expect this process to be fully completed by Tuesday, March 16, 2021 at 9:00 PM UTC, your service instance may be updated prior. Additional details surrounding the root cause can be found here: MO244568 in their Microsoft 365 admin portal or https://status.office365.com/. Next update by: Tuesday, March 16, 2021, at 9:00 PM UTC; PublishedTime=2021-03-16T14:38:46.277Z

Status: Our re-deployment to mitigate impact related to MO244568 is taking longer than initially expected. We anticipate the final stages of deployment and the validation process will be fully completed by Wednesday, March 17, 2021 at 12:00 AM UTC. Additional details surrounding the root cause can be found within MO244568 in the Microsoft 365 admin center. Next update by: Wednesday, March 17, 2021, at 12:00 AMUTC; PublishedTime=2021-03-16T19:23:50.057Z

Final status: We've confirmed that the re-deployment is complete and we've verified internally that the issue has been resolved. Next update by: Wednesday, March 17, 2021, at 12:00 AMUTC; PublishedTime=2021-03-16T19:23:50.057Z

--------------------------------------------------------------------------------------------------
Title Continuing Service Recovery from MO244568
ID IT244611
Service Microsoft Intune
Feature Microsoft Intune
Status Service restored
Impact Admins may have experienced intermittent impact for various Intune features.
Severity
Start time 2021-03-15T19:15:00Z
End time 2021-03-16T21:27:00Z
Last update 2021-03-16T22:06:54.623Z
More info - Admins may have notice enrollment failures as devices retry enrollment - Applications may have not all be shown in the UX or users may have see 401 errors - Compliance checks may have not be successful - Exporting reports may have fail - Admins may have notice their tenant's devices were not visible within the Intune admin center Final status: We've confirmed that the re-deployment is complete and we've verified internally that the issue has been resolved. Scope of impact: This issue may have affected any admins. Start time: Monday, March 15, 2021, at 7:15 PM'UTC End time: Tuesday, March 16, 2021, at 9:27 PM'UTC Root cause: Preliminary information on the underlying root cause is subject to change and may be found here: https://status.azure.com/en-us/status/history/ Next steps: - We're reviewing our monitoring and alerting processes to more quickly identify Azure Active Directory (AAD) service dependent functionality problems in the future. This is the final update for the event.; PublishedTime=2021-03-16T22:06:54.623Z
Root cause Preliminary information on the underlying root cause is subject to change and may be found here: https://status.azure.com/en-us/status/history/ Next steps: - We're reviewing our monitoring and alerting processes to more quickly identify Azure Active Directory (AAD) service dependent functionality problems in the future. This is the final update for the event.; PublishedTime=2021-03-16T22:06:54.623Z

IT244611 Details

Status: We're working on restoring functionality for end-to-end service delivery. Additional details can be found here: MO244568 in their Microsoft 365 admin portal or https://status.office365.com/. Next update by: Tuesday, March 16, 2021, at 5:00 PMUTC; PublishedTime=2021-03-16T00:36:56.837Z

Status: We've determined that a service re-deployment will properly mitigate impact related to MO244568. We expect this process to be fully completed by Tuesday, March 16, 2021 at 9:00 PM UTC, your service instance may be updated prior. Additional details surrounding the root cause can be found here: MO244568 in their Microsoft 365 admin portal or https://status.office365.com/. Next update by: Tuesday, March 16, 2021, at 5:00 PMUTC; PublishedTime=2021-03-16T05:40:37.883Z

Status: We're continuing with the re-deployment to mitigate impact related to MO244568. We expect this process to be fully completed by Tuesday, March 16, 2021 at 9:00 PM UTC, your service instance may be updated prior. Additional details surrounding the root cause can be found here: MO244568 in their Microsoft 365 admin portal or https://status.office365.com/. Next update by: Tuesday, March 16, 2021, at 9:00 PM UTC; PublishedTime=2021-03-16T14:38:46.277Z

Status: Our re-deployment to mitigate impact related to MO244568 is taking longer than initially expected. We anticipate the final stages of deployment and the validation process will be fully completed by Wednesday, March 17, 2021 at 12:00 AM UTC. Additional details surrounding the root cause can be found within MO244568 in the Microsoft 365 admin center. Next update by: Wednesday, March 17, 2021, at 12:00 AMUTC; PublishedTime=2021-03-16T19:23:50.057Z

Final status: We've confirmed that the re-deployment is complete and we've verified internally that the issue has been resolved. Next update by: Wednesday, March 17, 2021, at 12:00 AMUTC; PublishedTime=2021-03-16T19:23:50.057Z

--------------------------------------------------------------------------------------------------
Title Continuing Service Recovery from MO244568
ID IT244611
Service Microsoft Intune
Feature Microsoft Intune
Status Service restored
Impact Admins may have experienced intermittent impact for various Intune features.
Severity
Start time 2021-03-15T19:15:00Z
End time 2021-03-16T21:27:00Z
Last update 2021-03-16T22:06:54.623Z
More info - Admins may have notice enrollment failures as devices retry enrollment - Applications may have not all be shown in the UX or users may have see 401 errors - Compliance checks may have not be successful - Exporting reports may have fail - Admins may have notice their tenant's devices were not visible within the Intune admin center Final status: We've confirmed that the re-deployment is complete and we've verified internally that the issue has been resolved. Scope of impact: This issue may have affected any admins. Start time: Monday, March 15, 2021, at 7:15 PM'UTC End time: Tuesday, March 16, 2021, at 9:27 PM'UTC Root cause: Preliminary information on the underlying root cause is subject to change and may be found here: https://status.azure.com/en-us/status/history/ Next steps: - We're reviewing our monitoring and alerting processes to more quickly identify Azure Active Directory (AAD) service dependent functionality problems in the future. This is the final update for the event.; PublishedTime=2021-03-16T22:06:54.623Z
Root cause Preliminary information on the underlying root cause is subject to change and may be found here: https://status.azure.com/en-us/status/history/ Next steps: - We're reviewing our monitoring and alerting processes to more quickly identify Azure Active Directory (AAD) service dependent functionality problems in the future. This is the final update for the event.; PublishedTime=2021-03-16T22:06:54.623Z

IT244611 Details

Status: We're working on restoring functionality for end-to-end service delivery. Additional details can be found here: MO244568 in their Microsoft 365 admin portal or https://status.office365.com/. Next update by: Tuesday, March 16, 2021, at 5:00 PMUTC; PublishedTime=2021-03-16T00:36:56.837Z

Status: We've determined that a service re-deployment will properly mitigate impact related to MO244568. We expect this process to be fully completed by Tuesday, March 16, 2021 at 9:00 PM UTC, your service instance may be updated prior. Additional details surrounding the root cause can be found here: MO244568 in their Microsoft 365 admin portal or https://status.office365.com/. Next update by: Tuesday, March 16, 2021, at 5:00 PMUTC; PublishedTime=2021-03-16T05:40:37.883Z

Status: We're continuing with the re-deployment to mitigate impact related to MO244568. We expect this process to be fully completed by Tuesday, March 16, 2021 at 9:00 PM UTC, your service instance may be updated prior. Additional details surrounding the root cause can be found here: MO244568 in their Microsoft 365 admin portal or https://status.office365.com/. Next update by: Tuesday, March 16, 2021, at 9:00 PM UTC; PublishedTime=2021-03-16T14:38:46.277Z

Status: Our re-deployment to mitigate impact related to MO244568 is taking longer than initially expected. We anticipate the final stages of deployment and the validation process will be fully completed by Wednesday, March 17, 2021 at 12:00 AM UTC. Additional details surrounding the root cause can be found within MO244568 in the Microsoft 365 admin center. Next update by: Wednesday, March 17, 2021, at 12:00 AMUTC; PublishedTime=2021-03-16T19:23:50.057Z

Final status: We've confirmed that the re-deployment is complete and we've verified internally that the issue has been resolved. Next update by: Wednesday, March 17, 2021, at 12:00 AMUTC; PublishedTime=2021-03-16T19:23:50.057Z

--------------------------------------------------------------------------------------------------
Title Users may be unable to access multiple Microsoft 365 services
ID MO244568
Service Microsoft 365 suite
Feature Portal
Status Post-incident report published
Impact Users may have been unable to access multiple Microsoft 365, Azure, and Dynamics 365 services.
Severity
Start time 2021-03-15T19:00:00Z
End time 2021-03-16T09:25:00Z
Last update 2021-03-22T23:23:38.267Z
More info Any service that leverages Azure Active Directory (Azure AD) may have been affected. While we're focused on remediating all residual impact as soon as possible, some users may have continue to experience the following: Intune: Admins and users may have intermittently experience the following scenarios: -Admins may have notice enrollment failures as devices retry enrollment -Applications may have not all be shown in the UX or users may have see 401 errors -Compliance checks may have not be successful -Exporting reports may have fail -Admins may notice their tenant's device were not visible within the Intune admin portal Microsoft Managed Desktop: Admins may have been unable to perform actions within the Microsoft Managed Desktop admin portal. In addition, this issue may have impact end users' ability to interact with new Microsoft Managed Desktop devices. Outlook.com: Recovered. Office Licensing: Recovered. Microsoft Teams: Recovered. Exchange Online: Recovered. Microsoft Forms: Recovered. SharePoint Online/OneDrive for Business: Recovered. Yammer: Recovered. Admin portal: Recovered. Office Web Applications: Recovered. OneNote: Recovered. Microsoft Stream: Recovered. PowerBI: Recovered. Final status: We've completed mitigation for the affected Microsoft 365 services, and can confirm that most of these services have recovered. However, Intune and Microsoft Managed Desktop remain impacted. We've published IT244611 and MG244657 to follow these events. Scope of impact: This issue could affect any user. Start time: Monday, March 15, 2021, at 7:00 PM UTC End time: Tuesday, March 16, 2021, at 9:25 AM UTC Root cause: Information on the underlying root cause is subject to change and may be found here: https://status.azure.com/en-us/status/history/ under Tracking ID LN01-P8Z. Additionally, it is provided within the PIR document attached to this incident. Next steps: - For a more comprehensive list of next steps and actions, please refer to the Post Incident Review document. We'll publish a post-incident report within five business days.; PublishedTime=2021-03-16T10:18:12.227Z
Root cause Information on the underlying root cause is subject to change and may be found here: https://status.azure.com/en-us/status/history/ under Tracking ID LN01-P8Z. Additionally, it is provided within the PIR document attached to this incident. Next steps: - For a more comprehensive list of next steps and actions, please refer to the Post Incident Review document. We'll publish a post-incident report within five business days.; PublishedTime=2021-03-16T10:18:12.227Z

MO244568 Details

Status: We're investigating a potential issue and checking for impact to your organization. We'll provide an update within 30 minutes.; PublishedTime=2021-03-15T19:39:14.463

Status: We've identified the underlying cause of the problem and deployed an update to resolve the issue. The update has finished its deployment to all impacted regions. Microsoft 365 services continue the process of recovery and are showing decreasing error rates in telemetry. We'll continue to monitor service health as availability is restored. Next update by: Monday, March 15, 2021, at 11:00 PM UTC; PublishedTime=2021-03-15T22:12:16.887Z

Status: While service health has improved and many users are no longer experiencing problems, we're taking further steps to identify and remediate isolated residual impact for all Microsoft 365 services including Teams, Exchange services, SharePoint Online, Intune, Outlook.com, and the admin portal. In parallel, we're continuing our monitoring actions while the service returns to healthy levels. Next update by: Tuesday, March 16, 2021, at 12:30 AM UTC; PublishedTime=2021-03-15T23:00:24.76Z

Status: We're continuing to take actions to mitigate the remaining impact for all Microsoft 365 services. We've confirmed that users will see impact resolution as our remediation actions are completed. Next update by: Tuesday, March 16, 2021, at 1:30 AMUTC; PublishedTime=2021-03-16T00:06:30.737Z

Status: Our actions to remediate impact across all of our Microsoft 365 services are ongoing. We're continuing to receive positive feedback that some Microsoft 365 services are fully recovered. Next update by: Tuesday, March 16, 2021, at 2:30 AM UTC; PublishedTime=2021-03-16T01:20:15.997Z

Status: We're continuing to see multiple Microsoft 365 services return to healthy availability. We're addressing the delays in resolving the residual impact that is present for a subset of the impacted Microsoft 365 services. Service specific updates will continue to be provided under the "More Info" section. Next update by: Tuesday, March 16, 2021, at 4:30 AM UTC; PublishedTime=2021-03-16T02:30:20.617Z

Status: We're working on various mitigation streams for the services that are experiencing residual impact. Service specific updates will continue to be provided under the "More Info" section. Next update by: Tuesday, March 16, 2021, at 6:30 AM UTC; PublishedTime=2021-03-16T04:29:02.99Z

Status: We've initiated a mitigation strategy for the impacted Intune services. In parallel, we're still pursuing multiple mitigation strategies to restore services to the other Microsoft 365 services. Service specific updates will continue to be provided under the "More Info" section. Next update by: Tuesday, March 16, 2021, at 8:30 AM UTC; PublishedTime=2021-03-16T06:26:59.103Z

Status: We've made progress with the mitigation strategies for the affected Microsoft 365 services, and expect users to see recovery while these deploy. We'll provide service specific updates in the "More Info" section. Next update by: Tuesday, March 16, 2021, at 10:30 AM UTC; PublishedTime=2021-03-16T08:20:02.237Z

Final status: We've completed mitigation for the affected Microsoft 365 services, and can confirm that most of these services have recovered. However, Intune and Microsoft Managed Desktop remain impacted. We've published IT244611 and MG244657 to follow these events. Next update by: Tuesday, March 16, 2021, at 10:30 AM UTC; PublishedTime=2021-03-16T08:20:02.237Z

--------------------------------------------------------------------------------------------------
Title Users may be unable to access multiple Microsoft 365 services
ID MO244568
Service Microsoft 365 suite
Feature Portal
Status Post-incident report published
Impact Users may have been unable to access multiple Microsoft 365, Azure, and Dynamics 365 services.
Severity
Start time 2021-03-15T19:00:00Z
End time 2021-03-16T09:25:00Z
Last update 2021-03-22T23:23:38.267Z
More info Any service that leverages Azure Active Directory (Azure AD) may have been affected. While we're focused on remediating all residual impact as soon as possible, some users may have continue to experience the following: Intune: Admins and users may have intermittently experience the following scenarios: -Admins may have notice enrollment failures as devices retry enrollment -Applications may have not all be shown in the UX or users may have see 401 errors -Compliance checks may have not be successful -Exporting reports may have fail -Admins may notice their tenant's device were not visible within the Intune admin portal Microsoft Managed Desktop: Admins may have been unable to perform actions within the Microsoft Managed Desktop admin portal. In addition, this issue may have impact end users' ability to interact with new Microsoft Managed Desktop devices. Outlook.com: Recovered. Office Licensing: Recovered. Microsoft Teams: Recovered. Exchange Online: Recovered. Microsoft Forms: Recovered. SharePoint Online/OneDrive for Business: Recovered. Yammer: Recovered. Admin portal: Recovered. Office Web Applications: Recovered. OneNote: Recovered. Microsoft Stream: Recovered. PowerBI: Recovered. Final status: We've completed mitigation for the affected Microsoft 365 services, and can confirm that most of these services have recovered. However, Intune and Microsoft Managed Desktop remain impacted. We've published IT244611 and MG244657 to follow these events. Scope of impact: This issue could affect any user. Start time: Monday, March 15, 2021, at 7:00 PM UTC End time: Tuesday, March 16, 2021, at 9:25 AM UTC Root cause: Information on the underlying root cause is subject to change and may be found here: https://status.azure.com/en-us/status/history/ under Tracking ID LN01-P8Z. Additionally, it is provided within the PIR document attached to this incident. Next steps: - For a more comprehensive list of next steps and actions, please refer to the Post Incident Review document. We'll publish a post-incident report within five business days.; PublishedTime=2021-03-16T10:18:12.227Z
Root cause Information on the underlying root cause is subject to change and may be found here: https://status.azure.com/en-us/status/history/ under Tracking ID LN01-P8Z. Additionally, it is provided within the PIR document attached to this incident. Next steps: - For a more comprehensive list of next steps and actions, please refer to the Post Incident Review document. We'll publish a post-incident report within five business days.; PublishedTime=2021-03-16T10:18:12.227Z

MO244568 Details

Status: We're investigating a potential issue and checking for impact to your organization. We'll provide an update within 30 minutes.; PublishedTime=2021-03-15T19:39:14.463

Status: We've identified the underlying cause of the problem and deployed an update to resolve the issue. The update has finished its deployment to all impacted regions. Microsoft 365 services continue the process of recovery and are showing decreasing error rates in telemetry. We'll continue to monitor service health as availability is restored. Next update by: Monday, March 15, 2021, at 11:00 PM UTC; PublishedTime=2021-03-15T22:12:16.887Z

Status: While service health has improved and many users are no longer experiencing problems, we're taking further steps to identify and remediate isolated residual impact for all Microsoft 365 services including Teams, Exchange services, SharePoint Online, Intune, Outlook.com, and the admin portal. In parallel, we're continuing our monitoring actions while the service returns to healthy levels. Next update by: Tuesday, March 16, 2021, at 12:30 AM UTC; PublishedTime=2021-03-15T23:00:24.76Z

Status: We're continuing to take actions to mitigate the remaining impact for all Microsoft 365 services. We've confirmed that users will see impact resolution as our remediation actions are completed. Next update by: Tuesday, March 16, 2021, at 1:30 AMUTC; PublishedTime=2021-03-16T00:06:30.737Z

Status: Our actions to remediate impact across all of our Microsoft 365 services are ongoing. We're continuing to receive positive feedback that some Microsoft 365 services are fully recovered. Next update by: Tuesday, March 16, 2021, at 2:30 AM UTC; PublishedTime=2021-03-16T01:20:15.997Z

Status: We're continuing to see multiple Microsoft 365 services return to healthy availability. We're addressing the delays in resolving the residual impact that is present for a subset of the impacted Microsoft 365 services. Service specific updates will continue to be provided under the "More Info" section. Next update by: Tuesday, March 16, 2021, at 4:30 AM UTC; PublishedTime=2021-03-16T02:30:20.617Z

Status: We're working on various mitigation streams for the services that are experiencing residual impact. Service specific updates will continue to be provided under the "More Info" section. Next update by: Tuesday, March 16, 2021, at 6:30 AM UTC; PublishedTime=2021-03-16T04:29:02.99Z

Status: We've initiated a mitigation strategy for the impacted Intune services. In parallel, we're still pursuing multiple mitigation strategies to restore services to the other Microsoft 365 services. Service specific updates will continue to be provided under the "More Info" section. Next update by: Tuesday, March 16, 2021, at 8:30 AM UTC; PublishedTime=2021-03-16T06:26:59.103Z

Status: We've made progress with the mitigation strategies for the affected Microsoft 365 services, and expect users to see recovery while these deploy. We'll provide service specific updates in the "More Info" section. Next update by: Tuesday, March 16, 2021, at 10:30 AM UTC; PublishedTime=2021-03-16T08:20:02.237Z

Final status: We've completed mitigation for the affected Microsoft 365 services, and can confirm that most of these services have recovered. However, Intune and Microsoft Managed Desktop remain impacted. We've published IT244611 and MG244657 to follow these events. Next update by: Tuesday, March 16, 2021, at 10:30 AM UTC; PublishedTime=2021-03-16T08:20:02.237Z

--------------------------------------------------------------------------------------------------
Title Admins intermittently can't migrate or create new mailboxes in Exchange Online
ID EX244857
Service Exchange Online
Feature Management and Provisioning
Status Service restored
Impact Admins may have been intermittently unable to migrate or create new mailboxes in the Exchange Online service.
Severity
Start time 2021-03-14T00:00:00Z
End time 2021-03-28T22:00:00Z
Last update 2021-03-29T19:25:38.873Z
More info Affected admins may have see an error indicating "The socket connection was aborted." Final status: We've confirmed that the deployment of the fix is complete, and we've confirm via telemetry and through our internal testing that the issue has been resolved. Scope of impact: Admins attempting to migrate or create new mailboxes in Exchange Online were intermittently affected by this issue. Start time: Sunday, March 14, 2021, at 12:00 AM'UTC End time: Sunday, March 28, 2021, at 10:00 PM'UTC Root cause: A code regression within a recent service update was causing traffic processing issues correlated with mailbox administration. Next steps: We're investigating how the code regression was missed during testing and validation of the recent service update to prevent this problem from happening again in the future. This is the final update for the event.; PublishedTime=2021-03-29T19:25:38.873Z
Root cause A code regression within a recent service update was causing traffic processing issues correlated with mailbox administration. Next steps: We're investigating how the code regression was missed during testing and validation of the recent service update to prevent this problem from happening again in the future. This is the final update for the event.; PublishedTime=2021-03-29T19:25:38.873Z

EX244857 Details

Status: We're investigating a potential issue and checking for impact to your organization. We'll provide an update within 30 minutes.; PublishedTime=2021-03-17T18:02:59.913

Status: We're reviewing the diagnostic information provided by the representatives of impacted admins to attempt to isolate the source of the issue and determine our next troubleshooting steps. Next update by: Wednesday, March 17, 2021, at 7:30 PMUTC; PublishedTime=2021-03-17T18:10:04.387Z

Status: Our analysis of the diagnostic information provided by impacted admins is ongoing. In parallel, we're analyzing the correlated service telemetry to isolate the source of the issue and develop an approach to mitigation. Next update by: Wednesday, March 17, 2021, at 9:30 PMUTC; PublishedTime=2021-03-17T19:08:18.917Z

Status: Our analysis of the correlated service telemetry has so far not identified any definitive issues; however, it remains ongoing. We're focused on isolating any changes within recent service updates which may potentially be causing intermittent impact to this functionality. Next update by: Thursday, March 18, 2021, at 2:00 AMUTC; PublishedTime=2021-03-17T20:43:52.317Z

Status: We're continuing our analysis and review of service telemetry and our investigation into any recent service changes to determine causes of impact. In parallel, we're working to complete restarts of affected service infrastructure to provide relief for admins. Next update by: Thursday, March 18, 2021, at 4:00 AMUTC; PublishedTime=2021-03-18T01:50:01.377Z

Status: We've confirmed via service monitoring that restarting affected service infrastructure is resolving impact. We're working to complete restarts in all affected environments to return the service to normal health while we continue our investigation into the cause of impact. Next update by: Thursday, March 18, 2021, at 6:00 AMUTC; PublishedTime=2021-03-18T03:49:43.547Z

Status: We're continuing to restart all affected service infrastructure to provide relief to admins. In parallel, we're exploring potential long term fixes to ensure the issue doesn't reoccur. Next update by: Thursday, March 18, 2021, at 6:00 PMUTC; PublishedTime=2021-03-18T05:06:49.833Z

Status: Our efforts to quickly remediate the issue by restarting all affected service infrastructure are ongoing. Our continued investigation has also revealed that the impact associated with this issue is likely tied to a code regression within a recent service update which is causing traffic processing issues correlated with mailbox administration. We've developed a targeted fix for this issue, which we expect will begin deployment by the next scheduled update. This fix is in addition to the aforementioned short-term mitigations steps, and will help to prevent continued impact. Next update by: Friday, March 19, 2021, at 6:00 PMUTC; PublishedTime=2021-03-18T16:23:05.413Z

Status: Our short-term mitigation efforts of restarting all affected service infrastructure is ongoing. Additionally, we've started to deploy the permanent fix and it is currently at 50 percent saturation. As the fix progresses, users may experience alleviation and we suspect the fix will complete within the next 24 hours. Next update by: Saturday, March 20, 2021, at 8:00 PMUTC; PublishedTime=2021-03-19T16:47:41.4Z

Status: Our short-term mitigation efforts of restarting all affected service infrastructure continues. In parallel, the saturation of the long-term fix is currently at 81 percent. This fix is deploying more slowly than initially anticipated, though we expect it will complete by the next scheduled update. The impact to admins will be gradually alleviated as the fix continues to saturate the affected environments. Next update by: Monday, March 22, 2021, at 9:00 PMUTC; PublishedTime=2021-03-20T18:13:42.867Z

Status: After rolling out the initial fix, we've identified an additional issue and have developed a secondary fix, which will fully mitigate impact to your organization. This secondary fix will be deploying gradually within the next 24 hours, which should be completely saturated within 48-72 hours. While we are taking a variety of steps to minimize the scope of this issue, impact may persist until both fixes are completely saturated. Next update by: Tuesday, March 23, 2021, at 9:00 PMUTC; PublishedTime=2021-03-22T20:52:08.057Z

Status: The deployment of the secondary fix is progressing slower than originally expected, and we now anticipate it may take until Monday, March 29, 2021 to complete. We're continuing to monitor the progress to ensure it progresses without issue and will provided updated estimates to completion as they become available. Next update by: Friday, March 26, 2021, at 9:00 PMUTC; PublishedTime=2021-03-23T19:49:18.99Z

Status: Progress of the secondary fix continues as expected and we still anticipate completion by Monday, March 29, 2021. We're continuing to monitor the fix deployment to ensure it progresses without issue. Next update by: Monday, March 29, 2021, at 9:00 PMUTC; PublishedTime=2021-03-26T19:18:14.657Z

Final status: We've confirmed that the deployment of the fix is complete, and we've confirm via telemetry and through our internal testing that the issue has been resolved. Next update by: Monday, March 29, 2021, at 9:00 PMUTC; PublishedTime=2021-03-26T19:18:14.657Z

--------------------------------------------------------------------------------------------------
Title Users don't see expected location suggestions in "Search for a room or location" in Calendar events
ID EX245650
Service Exchange Online
Feature E-Mail and calendar access
Status Service restored
Impact Users didn't see expected location suggestions in "Search for a room or location" in events via Outlook on the web.
Severity
Start time 2021-03-12T03:00:00Z
End time 2021-03-22T06:00:00Z
Last update 2021-03-22T17:22:45.513Z
More info Alternatively, affected users could have clicked "Search for a room or location" or "Browse with Room Finder" and then searched for and selected the room in Room Finder. Final status: We've confirmed with affected users that the fix deployment has resolved the impact. Scope of impact: This issue affected any Outlook on the web users attempting to get location suggestions in Calendar events. Start time: Friday, March 12, 2021, at 3:00 AM UTC End time: Monday, March 22, 2021, at 6:00 AM UTC Root cause: A configuration change was inadvertently causing location suggestions to not show as expected. Next steps: - We're reviewing our configuration implementation procedures to better identify similar issues during our development and testing cycles. This is the final update for the event.; PublishedTime=2021-03-22T17:22:45.513Z
Root cause A configuration change was inadvertently causing location suggestions to not show as expected. Next steps: - We're reviewing our configuration implementation procedures to better identify similar issues during our development and testing cycles. This is the final update for the event.; PublishedTime=2021-03-22T17:22:45.513Z

EX245650 Details

Status: We've received a report indicating that impact associated with EX245066 persists. We've resumed our investigation of the issue.

Status: We're analyzing network trace logs provided by users reporting the persisting impact to determine the source of the issue. Next update by: Monday, March 22, 2021, at 1:00 AMUTC; PublishedTime=2021-03-21T23:54:45.62Z

Status: We're comparing network trace logs showing and not showing impact to isolate the problem. Next update by: Monday, March 22, 2021, at 2:30 AMUTC; PublishedTime=2021-03-22T00:54:20.8Z

Status: We're continuing our analysis of the provided network trace logs to identify the source of impact. Next update by: Monday, March 22, 2021, at 4:00 AMUTC; PublishedTime=2021-03-22T02:29:57.443Z

Status: We've identified a change that could potentially be contributing to impact. We're reviewing the code and troubleshooting to identify a mitigation. Next update by: Monday, March 22, 2021, at 5:30 AMUTC; PublishedTime=2021-03-22T04:00:52.393Z

Status: We've determined that a configuration change is inadvertently causing location suggestions to not show as expected. We've developed a fix to resolve the remaining impact and we're validating it before deploying it to the affected environment. Next update by: Monday, March 22, 2021, at 7:30 PMUTC; PublishedTime=2021-03-22T05:29:31.61Z

Status: We've finished validating the fix and started deploying it to the affected environment. We anticipate the fix will finish saturating by our next scheduled update. Next update by: Monday, March 22, 2021, at 7:30 PMUTC; PublishedTime=2021-03-22T05:56:54.433Z

Status: The fix has finished saturating throughout the affected environment. We're reaching out to affected users to confirm mitigation. Next update by: Monday, March 22, 2021, at 7:30 PMUTC; PublishedTime=2021-03-22T06:33:10.937Z

Final status: We've confirmed with affected users that the fix deployment has resolved the impact. Next update by: Monday, March 22, 2021, at 7:30 PMUTC; PublishedTime=2021-03-22T06:33:10.937Z

--------------------------------------------------------------------------------------------------
Title Users don't see expected location suggestions in "Search for a room or location" in Calendar events
ID EX245650
Service Exchange Online
Feature E-Mail and calendar access
Status Service restored
Impact Users didn't see expected location suggestions in "Search for a room or location" in events via Outlook on the web.
Severity
Start time 2021-03-12T03:00:00Z
End time 2021-03-22T06:00:00Z
Last update 2021-03-22T17:22:45.513Z
More info Alternatively, affected users could have clicked "Search for a room or location" or "Browse with Room Finder" and then searched for and selected the room in Room Finder. Final status: We've confirmed with affected users that the fix deployment has resolved the impact. Scope of impact: This issue affected any Outlook on the web users attempting to get location suggestions in Calendar events. Start time: Friday, March 12, 2021, at 3:00 AM UTC End time: Monday, March 22, 2021, at 6:00 AM UTC Root cause: A configuration change was inadvertently causing location suggestions to not show as expected. Next steps: - We're reviewing our configuration implementation procedures to better identify similar issues during our development and testing cycles. This is the final update for the event.; PublishedTime=2021-03-22T17:22:45.513Z
Root cause A configuration change was inadvertently causing location suggestions to not show as expected. Next steps: - We're reviewing our configuration implementation procedures to better identify similar issues during our development and testing cycles. This is the final update for the event.; PublishedTime=2021-03-22T17:22:45.513Z

EX245650 Details

Status: We've received a report indicating that impact associated with EX245066 persists. We've resumed our investigation of the issue.

Status: We're analyzing network trace logs provided by users reporting the persisting impact to determine the source of the issue. Next update by: Monday, March 22, 2021, at 1:00 AMUTC; PublishedTime=2021-03-21T23:54:45.62Z

Status: We're comparing network trace logs showing and not showing impact to isolate the problem. Next update by: Monday, March 22, 2021, at 2:30 AMUTC; PublishedTime=2021-03-22T00:54:20.8Z

Status: We're continuing our analysis of the provided network trace logs to identify the source of impact. Next update by: Monday, March 22, 2021, at 4:00 AMUTC; PublishedTime=2021-03-22T02:29:57.443Z

Status: We've identified a change that could potentially be contributing to impact. We're reviewing the code and troubleshooting to identify a mitigation. Next update by: Monday, March 22, 2021, at 5:30 AMUTC; PublishedTime=2021-03-22T04:00:52.393Z

Status: We've determined that a configuration change is inadvertently causing location suggestions to not show as expected. We've developed a fix to resolve the remaining impact and we're validating it before deploying it to the affected environment. Next update by: Monday, March 22, 2021, at 7:30 PMUTC; PublishedTime=2021-03-22T05:29:31.61Z

Status: We've finished validating the fix and started deploying it to the affected environment. We anticipate the fix will finish saturating by our next scheduled update. Next update by: Monday, March 22, 2021, at 7:30 PMUTC; PublishedTime=2021-03-22T05:56:54.433Z

Status: The fix has finished saturating throughout the affected environment. We're reaching out to affected users to confirm mitigation. Next update by: Monday, March 22, 2021, at 7:30 PMUTC; PublishedTime=2021-03-22T06:33:10.937Z

Final status: We've confirmed with affected users that the fix deployment has resolved the impact. Next update by: Monday, March 22, 2021, at 7:30 PMUTC; PublishedTime=2021-03-22T06:33:10.937Z

--------------------------------------------------------------------------------------------------
Title Unable to access Microsoft Teams live event usage report
ID TM244196
Service Microsoft Teams
Feature Teams Components
Status Service restored
Impact Admins may have been unable to expand on the details of the Teams live event usage report.
Severity
Start time 2021-03-11T21:30:00Z
End time 2021-03-16T12:20:00Z
Last update 2021-03-16T14:59:43.453Z
More info The report dashboard would have been available via the Analytics & Reports page; however, when an admin attempted to expand the details of a report, they would have received the following error: "The page you were looking for does not exist." There was no impact to any other reports within the Analytics & Reports page. Final status: We identified an issue with the service responsible for generating Teams live event usage reports and deployed a fix to mitigate impact. Scope of impact: Any admin attempting to expand on the details of the Teams live event usage report were impacted by this event. Start time: Thursday, March 11, 2021, at 9:30 PM UTC End time: Tuesday, March 16, 2021, at 12:20 PM UTC Root cause: The service responsible for generating Teams live event usage reports had become degraded, resulting in impact. Next steps: - We're analyzing performance data and trends on the affected systems to help prevent this problem from happening again. This is the final update for the event.; PublishedTime=2021-03-16T14:59:37.72Z
Root cause The service responsible for generating Teams live event usage reports had become degraded, resulting in impact. Next steps: - We're analyzing performance data and trends on the affected systems to help prevent this problem from happening again. This is the final update for the event.; PublishedTime=2021-03-16T14:59:37.72Z

TM244196 Details

Status: We've identified an issue in which admins are unable to expand on the details of the Teams live event usage reports. We're reviewing diagnostic data associated with the reporting mechanism to understand why the reports are not rendering and are working to determine a solution to remediate impact. Next update by: Wednesday, March 17, 2021, at 10:00 PM UTC; PublishedTime=2021-03-11T22:28:35.033Z

Final status: We identified an issue with the service responsible for generating Teams live event usage reports and deployed a fix to mitigate impact. Next update by: Wednesday, March 17, 2021, at 10:00 PM UTC; PublishedTime=2021-03-11T22:28:35.033Z

--------------------------------------------------------------------------------------------------
Title Admins changing single device settings on Microsoft Teams Rooms for Windows devices resulted in device reset
ID TM244613
Service Microsoft Teams
Feature Teams Components
Status Service restored
Impact Admins changing single device settings on Microsoft Teams Rooms for Windows devices resulted in device reset.
Severity
Start time 2021-03-11T06:30:00Z
End time 2021-03-16T07:50:00Z
Last update 2021-03-16T08:45:41.957Z
More info Impact was specific to Microsoft Teams Rooms for Windows devices with single device settings changed via the Teams admin center. Changes made via Extensible Markup Language (XML) configurations were not affected. Impact was limited to the following admin roles: - Global admin - Teams service admin - Skype for Business service admin - Teams device admin Admins could avoid impact by not applying single device settings until after the fix was finished deploying. Final status: The fix has been deployed successfully to the affected service environments and we've confirmed that the issue is resolved after a period of monitoring. Scope of impact: This issue may have affected any admin attempting to change single device settings for Microsoft Teams Rooms for Windows devices, via the Teams admin center. Start time: Thursday, March 11, 2021, at 6:30 AM UTC End time: Tuesday, March 16, 2021, at 7:50 AM UTC Root cause: A recent service update which contained changes to facilitate future feature enabling, unexpectedly caused impact. Next steps: - We're reviewing our update procedures to better identify similar issues during our development and testing cycles. This is the final update for the event.; PublishedTime=2021-03-16T08:45:41.957Z
Root cause A recent service update which contained changes to facilitate future feature enabling, unexpectedly caused impact. Next steps: - We're reviewing our update procedures to better identify similar issues during our development and testing cycles. This is the final update for the event.; PublishedTime=2021-03-16T08:45:41.957Z

TM244613 Details

Status: We're investigating a potential issue and checking for impact to your organization. We'll provide an update within 30 minutes.

Status: We've confirmed that a recent service update which contained changes to facilitate future feature enabling resulted in impact. We're developing a fix to mitigate impact which is undergoing testing and validation as we prepare for deployment. Next update by: Tuesday, March 16, 2021, at 3:00 AMUTC; PublishedTime=2021-03-16T01:00:07.937Z

Status: We're continuing to validate the fix before deploying it to the affected environment. We anticipate the fix will be fully deployed within the next 12 hours. Next update by: Tuesday, March 16, 2021, at 6:00 AMUTC; PublishedTime=2021-03-16T02:17:01.887Z

Status: We've completed fix validation and begun deployment to the affected service environments. We expect the fix to complete in the next three hours. Next update by: Tuesday, March 16, 2021, at 9:00 AM UTC; PublishedTime=2021-03-16T04:40:52.01Z

Final status: The fix has been deployed successfully to the affected service environments and we've confirmed that the issue is resolved after a period of monitoring. Next update by: Tuesday, March 16, 2021, at 9:00 AM UTC; PublishedTime=2021-03-16T04:40:52.01Z

--------------------------------------------------------------------------------------------------
Title Admins see inaccurate or delayed policy reporting in the Intune admin center
ID IT244284
Service Microsoft Intune
Feature Microsoft Intune
Status Service restored
Impact Admins may have seen inaccurate or delayed policy reporting in the Intune admin center.
Severity
Start time 2021-03-11T00:35:00Z
End time 2021-03-17T02:40:00Z
Last update 2021-03-17T03:04:35.81Z
More info The issue appeared to be cosmetic in nature as policy, profile and enrollment updates were being processed as expected. Final status: We've confirmed that the deployment of the fix is now complete and our telemetry shows the issue has been mitigated. Scope of impact: The problem potentially affected all admins attempting to view policy reporting in the admin center. Start time: Thursday, March 11, 2021, at 12:35 AM'UTC End time: Wednesday, March 17, 2021, at 2:40 AM'UTC Root cause: The infrastructure responsible for processing policy reports was performing in a sub-optimal state, resulting in impact. Next steps: - We're investigating the affected infrastructure to better understand how it got into this state in order to prevent this from happening again. - We're reviewing our monitoring services to look for ways to better detect service degradation issues. This is the final update for the event.; PublishedTime=2021-03-17T03:04:35.81Z
Root cause The infrastructure responsible for processing policy reports was performing in a sub-optimal state, resulting in impact. Next steps: - We're investigating the affected infrastructure to better understand how it got into this state in order to prevent this from happening again. - We're reviewing our monitoring services to look for ways to better detect service degradation issues. This is the final update for the event.; PublishedTime=2021-03-17T03:04:35.81Z

IT244284 Details

Status: We're investigating a potential issue and checking for impact to your organization. We'll provide an update within 30 minutes.

Status: We're currently analyzing service telemetry to identify processing trends, isolate a root cause, and develop next troubleshooting steps. Next update by: Friday, March 12, 2021, at 9:00 PMUTC; PublishedTime=2021-03-12T19:28:48.02Z

Status: We've discovered via telemetry that the issue is potentially resulting due to processing issues with a large number of queued requests. We're analyzing additional service logs to confirm the root cause and develop subsequent mitigation steps. Next update by: Friday, March 12, 2021, at 11:00 PMUTC; PublishedTime=2021-03-12T20:52:20.063Z

Status: Our analysis thus far has been inconclusive in determining the cause for the processing issues with the queued requests. Moving forward, we're reviewing additional service logs to determine our next steps in alleviating impact. Next update by: Saturday, March 13, 2021, at 1:00 AMUTC; PublishedTime=2021-03-12T22:59:02.443Z

Status: We've determined that the infrastructure responsible for processing policy reports is performing in a sub-optimal state, resulting in impact. We're implementing infrastructure optimizations, which we anticipate will resolve the issue. Next update by: Saturday, March 13, 2021, at 7:00 AMUTC; PublishedTime=2021-03-13T00:23:16Z

Status: We're continuing to implement infrastructure optimizations as we closely monitor to ensure no further issues occur. Next update by: Saturday, March 13, 2021, at 10:00 AMUTC; PublishedTime=2021-03-13T05:20:02.457Z

Status: We've completed our first infrastructure optimization and are seeing the backlog slowly reducing. We've started to implement optimizations on all remaining infrastructure and we'll continue to monitor to ensure no further issues occur. Next update by: Saturday, March 13, 2021, at 10:00 PM UTC ; PublishedTime=2021-03-13T08:24:28.077Z

Status: We've confirmed that the backlog has successfully completed draining on the initial portion of infrastructure we optimized, resolving impact for users hosted there. We're continuing to monitor the optimizations as they are being implemented in the remaining infrastructure to ensure no further issues occur. Next update by: Tuesday, March 16, 2021, at 12:00 AMUTC; PublishedTime=2021-03-13T21:17:58.483Z

Status: We've observed continuous success in the implemented optimizations and are going to monitor further in order to ensure full service restoration before deploying the optimizations to the remaining infrastructure. Next update by: Tuesday, March 16, 2021, at 5:00 PMUTC; PublishedTime=2021-03-15T22:41:45.193Z

Status: After continued monitoring, we're proceeding with deploying the optimizations to the remaining infrastructure to fully resolve the issue. Next update by: Tuesday, March 16, 2021, at 11:30 PMUTC; PublishedTime=2021-03-16T15:59:00.957Z

Status: We've confirmed via telemetry that the optimizations have nearly completed deployment to the entire impacted infrastructure. As the deployment concludes, we're continuing to monitor the environment to ensure impact is mitigated. Next update by: Wednesday, March 17, 2021, at 1:30 AMUTC; PublishedTime=2021-03-16T21:46:09.813Z

Status: We're actively monitoring the environment while deployment continues to ensure impact is fully mitigated. Next update by: Wednesday, March 17, 2021, at 2:00 AMUTC; PublishedTime=2021-03-16T23:56:45.017Z

Status: Deployment is taking longer than expected due to incoming traffic. We're continuing to monitor the environment while the queue continues to drain to ensure impact is fully mitigated. Next update by: Wednesday, March 17, 2021, at 5:30 AMUTC; PublishedTime=2021-03-17T01:14:53.74Z

Final status: We've confirmed that the deployment of the fix is now complete and our telemetry shows the issue has been mitigated. Next update by: Wednesday, March 17, 2021, at 5:30 AMUTC; PublishedTime=2021-03-17T01:14:53.74Z

--------------------------------------------------------------------------------------------------
Title Some users in Brazil may be unable to join Microsoft Teams meetings or calls
ID TM244031
Service Microsoft Teams
Feature Teams Components
Status Post-incident report published
Impact Users may have been unable to join Microsoft Teams meetings, calls and live events.
Severity
Start time 2021-03-10T19:00:00Z
End time 2021-03-10T19:15:00Z
Last update 2021-03-30T18:22:19.58Z
More info
Root cause An external networking issue with a local ISP affected Teams users in Brazil, resulting in call drops for existing calls, meetings, and live events. This caused an increase in traffic when those affected users retried their connections. This increase occurred in-tandem with an additional spike in top-of-the-hour traffic and caused unexpectedly high resource utilization on the regional Teams calling infrastructure. As a result, users may have been dropped from existing connections or been unable to join new calls, meetings, and live events. Next steps: - For a more comprehensive list of next steps and actions, please refer to the Post Incident Review document. This is the final update for the event.; PublishedTime=2021-03-10T19:50:30.573Z

TM244031 Details

Status: We're investigating a potential issue and checking for impact to your organization. We'll provide an update within 30 minutes.

Status: We've identified components that facilitate call joining functionality are performing below acceptable thresholds, resulting in impact. We're optimizing performance parameters on the affected infrastructure and rerouting some user connections to alternate components to remediate impact. Next update by: Wednesday, March 10, 2021, at 9:00 PM UTC; PublishedTime=2021-03-10T19:35:32.123Z

Final status: We've completed the rerouting actions and after a period of monitoring, we've confirmed that impact has been remediated. Next update by: Wednesday, March 10, 2021, at 9:00 PM UTC; PublishedTime=2021-03-10T19:35:32.123Z

--------------------------------------------------------------------------------------------------
Title Some users in Turkey may have experienced issues accessing several Microsoft 365 services
ID MO243992
Service Microsoft 365 suite
Feature Portal
Status Service restored
Impact Users may have experienced issues accessing several Microsoft 365 services.
Severity
Start time 2021-03-10T11:28:00Z
End time 2021-03-10T12:52:11.01Z
Last update 2021-03-10T12:52:11.01Z
More info
Root cause

MO243992 Details

Final status: After a period of monitoring, we can confirm that Microsoft 365 services were operating as expected. However, customers routed through a third party internet service provider would likely have seen access issues. The third party took corrective action by rerouting user connections which resolved the issue.

--------------------------------------------------------------------------------------------------
Title Content creation failures
ID CR243947
Service Dynamics 365 Apps
Feature Components/Features
Status Service restored
Impact Users may have been unable to upload content to the image gallery control within your organization.
Severity
Start time 2021-03-09T18:21:00Z
End time 2021-03-10T02:10:00Z
Last update 2021-03-10T02:19:51.973Z
More info
Root cause

CR243947 Details

--------------------------------------------------------------------------------------------------
Title Some users can't access multiple Microsoft 365 services in specific South American countries
ID MO243853
Service Microsoft 365 suite
Feature Portal
Status Post-incident report published
Impact Users may have been unable to access multiple Microsoft 365 services.
Severity
Start time 2021-03-09T17:04:00Z
End time 2021-03-09T17:19:00Z
Last update 2021-03-25T16:37:47.333Z
More info
Root cause Some network components, used between local Internet Service Providers (ISPs) and the Microsoft 365 service, started to perform below acceptable thresholds, resulting in impact. Two third-party Internet Service Providers (ISP) were experiencing connectivity issues. It took approximately 15 minutes for automated recovery systems to re-route traffic via Argentina instead. Due to the convergence on some network routing components, these network components were suddenly isolated from the network, and it can sometimes take longer to withdraw our prefixes to local peers. As the network components continued to advertise our address space to peers longer than expected, this ultimately resulting in the traffic being dropped and impact. Next steps: - For a more comprehensive list of next steps and actions, please refer to the Post Incident Review document. This is the final update for the event.; PublishedTime=2021-03-09T18:19:00.723Z

MO243853 Details

Status: We're investigating a potential issue with accessing multiple Microsoft 365 services. We'll provide an update within 30 minutes.

Status: We've identified an issue with a section of network components used between local Internet Service Providers (ISP) and the Microsoft 365 service. By design, our automated recovery features have routed connections to alternate components and our telemetry indicates that impact has been mitigated. We're monitoring the environment to confirm that the issue has been fully resolved. Next update by: Tuesday, March 9, 2021, at 8:00 PM UTC; PublishedTime=2021-03-09T17:52:14Z

Final status: After a period of monitoring, we've confirmed that impact has been remediated. Next update by: Tuesday, March 9, 2021, at 8:00 PM UTC; PublishedTime=2021-03-09T17:52:14Z

--------------------------------------------------------------------------------------------------
Title Users experiencing intermittent delays when accessing Exchange Online via all connection methods
ID EX243828
Service Exchange Online
Feature E-Mail and calendar access
Status Service restored
Impact Users may have experienced intermittent delays when accessing Exchange Online via all connection methods.
Severity
Start time 2021-03-09T16:03:00Z
End time 2021-03-09T17:13:00Z
Last update 2021-03-09T18:00:29.917Z
More info
Root cause

EX243828 Details

Status: We're analyzing system logs to determine the source of the issue. Next update by: Tuesday, March 9, 2021, at 6:30 PMUTC; PublishedTime=2021-03-09T17:14:32.413Z

Final status: While we were in the process of analyzing system logs to isolate the source of the problem, system monitoring indicated that the service had returned to normal health. We've confirmed that the problem is no longer occurring, and we'll continue to monitor the service to ensure that the problem does not happen again. Next update by: Tuesday, March 9, 2021, at 6:30 PMUTC; PublishedTime=2021-03-09T17:14:32.413Z

--------------------------------------------------------------------------------------------------
Title Users may be intermittently unable to enroll or check in devices
ID IT243832
Service Microsoft Intune
Feature Microsoft Intune
Status Service restored
Impact Users may have been intermittently experiencing delays or failures when enrolling or checking in devices.
Severity
Start time 2021-03-09T06:00:00Z
End time 2021-03-10T18:30:00Z
Last update 2021-03-11T07:01:27.713Z
More info
Root cause A portion of SQL database infrastructure was performing below acceptable performance levels, resulting in impact. Next steps: - We're reviewing the affected SQL database service infrastructure to better understand how impact began. This is the final update for the event.; PublishedTime=2021-03-11T05:26:40.687Z

IT243832 Details

Status: We've identified an issue that's intermittently preventing enrollment of new devices and device check-in. We're analyzing system logs to better understand the nature of impact while also determining the source for this event. Next update by: Tuesday, March 9, 2021, at 7:00 PMUTC; PublishedTime=2021-03-09T17:22:52.547Z

Status: Following our system log analysis, we've focused our investigation to a potential replication issue affecting a subsection of SQL database infrastructure. We're confirming the performance availability levels for the subsection of SQL database infrastructure to determine our next steps for remediating impact. Next update by: Tuesday, March 9, 2021, at 9:00 PMUTC; PublishedTime=2021-03-09T18:29:31.107Z

Status: Rather than a replication issue, we've determined that a group of SQL database infrastructure has fallen below our manageable service performance thresholds, resulting in impact to enrollment and device check in. We're exploring our available options for restoring the affected SQL databases and remediating impact. Next update by: Tuesday, March 9, 2021, at 11:00 PMUTC; PublishedTime=2021-03-09T20:26:09.037Z

Status: We've further determined that the affected SQL databases experienced an indexing issue, which lowered performance levels causing delays and failures when enrolling or checking in devices. We're re-indexing the affected SQL databases to remediate impact. Next update by: Wednesday, March 10, 2021, at 1:00 AMUTC; PublishedTime=2021-03-09T22:28:18.083Z

Status: We're continuing to re-index the affected SQL databases to remediate impact. Next update by: Wednesday, March 10, 2021, at 3:00 AMUTC; PublishedTime=2021-03-10T00:52:37.86Z

Status: The re-indexing is taking longer than expected to complete. We're continuing to monitor the progress. In parallel, we're investigating alternative routes that may expedite the mitigation process. Next update by: Wednesday, March 10, 2021, at 5:00 AMUTC; PublishedTime=2021-03-10T02:54:03.04Z

Status: While the re-indexing continues to progress, we're increasing the database storage size and upgrading the database instance to aid in the performance. Next update by: Wednesday, March 10, 2021, at 7:00 AMUTC; PublishedTime=2021-03-10T05:08:55.073Z

Status: We've paused the database re-indexing procedure and are focusing on increasing the database storage size and upgrading the database instance stock keeping unit (SKU). We expect this process to complete by our next scheduled update. Next update by: Wednesday, March 10, 2021, at 9:00 AMUTC; PublishedTime=2021-03-10T06:52:47.097Z

Status: We've completed the increase of the database storage size and the database instance stock keeping unit, but this hasn't provided the expected relief. We're reviewing system telemetry to identify our next troubleshooting steps and to develop a mitigation plan. Next update by: Wednesday, March 10, 2021, at 11:00 AM UTC; PublishedTime=2021-03-10T08:56:46.69Z

Status: We've identified that a section of SQL infrastructure, responsible for managing the enrollment and check in of deceives, was performing below acceptable performance thresholds. We've rerouted traffic to alternate infrastructure, and are continuing to monitor system performance, to ensure the issue does not re-occur. Next update by: Wednesday, March 10, 2021, at 5:00 PM UTC; PublishedTime=2021-03-10T10:32:26.507Z

Status: After monitoring the service, we've determined that rerouting traffic was unsuccessful in mitigating impact. We're continuing to analyze service telemetry to determine our next steps, which has led us to identify a potential indexing issue that may be contributing to impact. As we continue to investigate, we're conducting an index rebuild and infrastructure optimizations to potentially mitigate impact. Next update by: Wednesday, March 10, 2021, at 7:00 PMUTC; PublishedTime=2021-03-10T16:49:07.857Z

Status: While conducting an index rebuild and infrastructure optimizations, we noticed the device check-in health again began showing decreased performance. We're currently monitoring service logs to determine why this happened, while simultaneously re-attempting the index rebuild to potentially mitigate initial impact. Next update by: Wednesday, March 10, 2021, at 9:30 PMUTC; PublishedTime=2021-03-10T18:54:29.743Z

Status: Though we've determined via telemetry that device check-in health has returned to normal, our attempt at re-running the index rebuild is not executing as expected. We're planning to once again attempt running the new index rebuild within the next five hours. Next update by: Thursday, March 11, 2021, at 3:00 AMUTC; PublishedTime=2021-03-10T21:27:41.437Z

Status: Our new index build has encountered unexpected errors. We're working to investigate and resolve those errors to ensure the build is able to complete as expected. Additionally, we're exploring alternate actions to mitigate impact as quickly as possible. Next update by: Thursday, March 11, 2021, at 5:00 AMUTC; PublishedTime=2021-03-11T02:55:44.44Z

Status: We've increased service infrastructure resource thresholds to allow our index build to complete without issue. We're reviewing monitoring to confirm that the service is healthy. Next update by: Thursday, March 11, 2021, at 7:00 AMUTC; PublishedTime=2021-03-11T04:55:52.147Z

Final status: After further evaluation into available data, we've confirmed that device enrollments and check ins have been remediated since Wednesday, March 10, 2021 at 6:30 PM UTC, and that updates since then have been communicated to outline our preventative efforts. Since increasing thresholds for specific services and contributions from other preventative service actions, we've validated that our changes will prevent reoccurrences and the issue is fully resolved. Next update by: Thursday, March 11, 2021, at 7:00 AMUTC; PublishedTime=2021-03-11T04:55:52.147Z

--------------------------------------------------------------------------------------------------
Title Users can't use the Request files feature in OneDrive for Business
ID OD246639
Service OneDrive for Business
Feature OneDrive for Business
Status Service restored
Impact Users were unable to use the Request files feature in OneDrive for Business.
Severity
Start time 2021-03-08T14:15:00Z
End time 2021-03-26T21:30:00Z
Last update 2021-03-26T21:45:46.433Z
More info
Root cause A recent feature update had introduced a code issue that disrupted the Request files dialog box, causing it to close before the link could be accessed. Next steps: - We're continuing to review our feature update procedures to ensure future updates don't cause similar issues. This is the final update for the event.; PublishedTime=2021-03-26T21:45:46.433Z

OD246639 Details

Status: We're analyzing network trace logs to determine the source of the issue. Next update by: Friday, March 26, 2021, at 4:00 PMUTC; PublishedTime=2021-03-26T15:01:36.993Z

Status: We've determined that a recent feature update has introduced a code issue that disrupts the Request files dialog box, causing it to close before the link can be accessed. We're disabling the feature update to remediate impact. Next update by: Friday, March 26, 2021, at 8:00 PMUTC; PublishedTime=2021-03-26T15:46:15.37Z

Status: We're continuing to disable the recent feature update which disrupted the Request files dialog box and resulted in impact. In parallel, we're monitoring the process to ensure it propagates as expected. Next update by: Friday, March 26, 2021, at 10:00 PMUTC; PublishedTime=2021-03-26T19:52:17.987Z

Final status: We've successfully disabled the recent feature update which initially disrupted the Request files dialog, and have confirmed mitigation with some affected users. Next update by: Friday, March 26, 2021, at 10:00 PMUTC; PublishedTime=2021-03-26T19:52:17.987Z

--------------------------------------------------------------------------------------------------
Title Some users were unable to receive call or message notifications within the Skype for Business service
ID LY243580
Service Skype for Business
Feature All Features
Status Service restored
Impact Users couldn't receive call or message notifications within the Skype for Business service.
Severity
Start time 2021-03-08T07:47:00Z
End time 2021-03-08T09:07:00Z
Last update 2021-03-08T09:45:48.827Z
More info
Root cause A dependent service that's responsible for handling WNS was performing below acceptable thresholds, resulting in impact. Next steps: - We're analyzing performance data and trends on the affected systems to help prevent this problem from happening again. This is the final update for the event.; PublishedTime=2021-03-08T09:45:48.827Z

LY243580 Details

Status: We're reviewing service telemetry to determine the next troubleshooting steps. Next update by: Monday, March 8, 2021, at 11:00 AM UTC; PublishedTime=2021-03-08T08:33:39.267Z

Final status: We determined a dependent service, responsible for handling Windows Notification Services (WNS) was performing below acceptable thresholds. We've resolved the issue with the dependent service and verified that impact has been mitigated. Next update by: Monday, March 8, 2021, at 11:00 AM UTC; PublishedTime=2021-03-08T08:33:39.267Z

--------------------------------------------------------------------------------------------------
Title Admins seeing an increased count in the meeting created and sent metric in Exchange Online email activity report
ID MO247735
Service Microsoft 365 suite
Feature Administration
Status Service degradation
Impact Admins are seeing an increased count in the meeting created and sent metric in Exchange Online email activity report.
Severity Sev2
Start time 2021-03-07T00:00:00Z
End time
Last update 2021-04-29T09:47:42.823Z
More info
Root cause

MO247735 Details

Status: We're analyzing system telemetry to determine the underlying cause of the issue. Next update by: Tuesday, March 30, 2021, at 1:30 PM UTC; PublishedTime=2021-03-30T10:20:12.887Z

Status: We're analyzing logging logic to help identify the underlying root cause and to formulate a remediation plan. Next update by: Tuesday, April 6, 2021, at 1:30 PM UTC; PublishedTime=2021-03-30T13:26:42.843Z

Status: We're continuing to review logging logic, in addition to gathering additional diagnostics from our upstream data sources, to help us better understand the new logging behavior and develop a mitigation strategy. Next update by: Tuesday, April 13, 2021, at 11:30 AM UTC; PublishedTime=2021-04-06T12:07:07.583Z

Status: We've identified that there are several signals from upstream data sources that are being included within the metrics. We're validating that the new signals are suitable for this reporting before we determine the next mitigation steps. Next update by: Thursday, April 15, 2021, at 11:00 AM UTC; PublishedTime=2021-04-13T11:23:03.797Z

Status: We're continuing to validate that the new signals are suitable for reporting and are validating that switching to these signals will not cause any further impact. Next update by: Thursday, April 29, 2021, at 11:00 AM UTC; PublishedTime=2021-04-15T10:37:39.227Z

Status: We're in the process of implementing and testing new signals. Once verified, we'll begin to implement them in place of current signals. In parallel, we're engaging with some affected admins to collect additional information we'll use to further improve the accuracy of these new signals. Next update by: Wednesday, May 19, 2021, at 11:00 AM UTC; PublishedTime=2021-04-29T09:47:42.823Z

--------------------------------------------------------------------------------------------------
Title Users may only be able to set the language setting to English in Outlook on the web
ID EX243748
Service Exchange Online
Feature E-Mail and calendar access
Status Service restored
Impact Users may have only been able to set the language setting to English in Outlook on the web.
Severity
Start time 2021-03-05T19:00:00Z
End time 2021-03-13T17:31:00Z
Last update 2021-03-16T05:50:54.473Z
More info
Root cause A code update inadvertently introduced a regression, which prevented users from selecting a language other than English in Outlook on the web. Next steps: - To help prevent similar impact in the future, we're further determining why the introduce a regression and how we can improve our validation procedures to catch such issues before deployment. This is the final update for the event.; PublishedTime=2021-03-15T23:58:11.403Z

EX243748 Details

Status: We're investigating a potential issue and checking for impact to your organization. We'll provide an update within 30 minutes.

Status: We've determined that a code update inadvertently introduced a regression, which is preventing users from selecting a language other than English in Outlook on the web. We've disabled the update to prevent further impact and we're troubleshooting to determine a mitigation plan. Next update by: Tuesday, March 9, 2021, at 7:30 PMUTC; PublishedTime=2021-03-09T03:02:50.057Z

Status: After isolating and disabling the offending code set to prevent further impact, we're now deploying a fixed build to resolve impact for affected users. We expect for the fixed build to complete deployment and reach all affected users by Thursday, March 11, 2021. Next update by: Thursday, March 11, 2021, at 7:30 PMUTC; PublishedTime=2021-03-09T17:50:59.49Z

Status: Deployment of the fix is taking longer than we expected; however, we've confirmed that the fix has reached most affected users. We'll continue to monitor the ongoing deployment as we look for ways to expedite the deployment to resolve the issue for all affected users. Next update by: Friday, March 12, 2021, at 7:30 PMUTC; PublishedTime=2021-03-11T18:51:33.187Z

Status: Nearly all infrastructure that had received the problematic code update have been fixed. We're monitoring the remaining few infrastructure deployments and expect that they'll complete by our next scheduled update. Next update by: Monday, March 15, 2021, at 7:00 PMUTC; PublishedTime=2021-03-12T18:33:52.17Z

Status: Our code fix has successfully been applied to all affected infrastructure. We're confirming with previously affected users that impact has been remediated. Next update by: Tuesday, March 16, 2021, at 1:30 AMUTC; PublishedTime=2021-03-15T17:14:01.773Z

Final status: We've confirmed with some users that the fix has restored service. Next update by: Tuesday, March 16, 2021, at 1:30 AMUTC; PublishedTime=2021-03-15T17:14:01.773Z

--------------------------------------------------------------------------------------------------
Title Admins are unable to manage scope tag assignments in the Microsoft Endpoint Manager admin center
ID IT243641
Service Microsoft Intune
Feature Microsoft Intune
Status Service restored
Impact Admins were unable to manage scope tag assignments in the Microsoft Endpoint Manager admin center.
Severity
Start time 2021-03-05T17:09:00Z
End time 2021-03-08T22:45:00Z
Last update 2021-03-08T23:18:04.327Z
More info Admins were unable to manage Original Equipment Manufacturer (OEM) application profiles or policy objects. Final status: We've completed deployment of the fix to the affected environment and confirmed via our internal validations that the issue has been resolved. Scope of impact: Impact was specific to admins served through the affected infrastructure. Start time: Friday, March 5, 2021, at 5:09 PM'UTC End time: Monday, March 8, 2021, at 10:45 PM'UTC Root cause: A recent deployment inadvertently introduced a code regression, resulting in impact. Next steps: - We're investigating how the code regression was missed during testing and validation to prevent this problem from happening again in the future. This is the final update for the event.; PublishedTime=2021-03-08T23:16:41.473Z
Root cause A recent deployment inadvertently introduced a code regression, resulting in impact. Next steps: - We're investigating how the code regression was missed during testing and validation to prevent this problem from happening again in the future. This is the final update for the event.; PublishedTime=2021-03-08T23:16:41.473Z

IT243641 Details

Status: We've determined that a recent deployment inadvertently introduced a code regression, resulting in impact. We're in the process of testing and validating a fix before we deploy to the affected environment. Next update by: Monday, March 8, 2021, at 9:00 PMUTC; PublishedTime=2021-03-08T18:55:44.667Z

Status: We've completed testing and validation of the fix and have deployed it to our testing environment for final validation that it will address the problem without introducing additional impact. Once completed, we will begin the deployment to the affected environment. Next update by: Monday, March 8, 2021, at 11:00 PMUTC; PublishedTime=2021-03-08T20:52:52.427Z

Final status: We've completed deployment of the fix to the affected environment and confirmed via our internal validations that the issue has been resolved. Next update by: Monday, March 8, 2021, at 11:00 PMUTC; PublishedTime=2021-03-08T20:52:52.427Z

--------------------------------------------------------------------------------------------------
Title Users can't edit the "choice" field in lists in SharePoint Online
ID SP243954
Service SharePoint Online
Feature SharePoint Features
Status Service restored
Impact Users were unable to edit the "choice" field in lists in SharePoint Online.
Severity
Start time 2021-03-05T07:00:00Z
End time 2021-03-11T02:28:00Z
Last update 2021-03-11T03:31:16.547Z
More info As alternatives, users may have been able to edit lists using the Classic view. They may have also been able to use the Edit all options from the Info pane/Details pane. Additionally, users accessing SharePoint Online on Internet Explorer may have seen the "choice" field appear stuck on the page, and not able to close it. Final status: We've reverted the recent service update that was causing the issue with editing the "choice" field and confirmed through internal testing that the issue is resolved. Scope of impact: This issue may have impacted any user editing the "choice" field in lists in the modern experience. Start time: Friday, March 5, 2021, at 7:00 AM UTC End time: Thursday, March 11, 2021, at 2:28 AM UTC Root cause: A recent service update was unexpectedly causing issues with editing the "choice" field in lists in SharePoint Online. Next steps: -We're reviewing the recent service update that caused the issue with editing the "choice" field, to gain a better understanding for why impact happened and to prevent any similar issues from reoccurring. This is the final update for the event.; PublishedTime=2021-03-11T03:28:52.017Z
Root cause A recent service update was unexpectedly causing issues with editing the "choice" field in lists in SharePoint Online. Next steps: -We're reviewing the recent service update that caused the issue with editing the "choice" field, to gain a better understanding for why impact happened and to prevent any similar issues from reoccurring. This is the final update for the event.; PublishedTime=2021-03-11T03:28:52.017Z

SP243954 Details

Status: We're investigating a potential issue and checking for impact to your organization. We'll provide an update within 30 minutes.; PublishedTime=2021-03-10T03:10:32.19

Status: We identified that a recent service update is unexpectedly causing issues with editing the "choice" field in lists. We're developing potential fixes to remediate the impact. Next update by: Thursday, March 11, 2021, at 4:30 AMUTC; PublishedTime=2021-03-10T03:39:39.25Z

Final status: We've reverted the recent service update that was causing the issue with editing the "choice" field and confirmed through internal testing that the issue is resolved. Next update by: Thursday, March 11, 2021, at 4:30 AMUTC; PublishedTime=2021-03-10T03:39:39.25Z

--------------------------------------------------------------------------------------------------
Title Users may experience various latency or sync issues with Microsoft Intune service
ID IT243064
Service Microsoft Intune
Feature Microsoft Intune
Status Service restored
Impact Users may have experienced various latency or sync issues with Microsoft Intune service.
Severity
Start time 2021-03-05T05:13:00Z
End time 2021-03-05T05:35:31.077Z
Last update 2021-03-05T05:51:42.137Z
More info Intune users may have experienced the following issues: -Newly subscribed accounts may have not provisioned to Intune. -New users may have not been able to login or enroll new devices. -New users may have been unable to sync their Azure Active Directory (AAD) data to Intune. -Existing users may have experienced latency in changes from AAD to Intune. Final status: We've determined that a recently deployed service update resulted in a synchronization issue in the latest build, resulting in impact. We've reverted the service update and confirmed with internal monitoring that the issue is resolved. Scope of impact: This issue affected users hosted on the impacted infrastructures. Start time: Friday, March 5, 2021, at 4:02 AM UTC End time: Friday, March 5, 2021, at 5:08 AM UTC Root cause: A recently deployed service update resulted in a synchronization issue in the latest build, resulting in impact. Next steps: -We're reviewing the service update which resulted in a synchronization issue to gain a better understanding of why impact occurred, and to prevent any similar issues in the future. This is the final update for the event.; PublishedTime=2021-03-05T05:35:31.077Z
Root cause A recently deployed service update resulted in a synchronization issue in the latest build, resulting in impact. Next steps: -We're reviewing the service update which resulted in a synchronization issue to gain a better understanding of why impact occurred, and to prevent any similar issues in the future. This is the final update for the event.; PublishedTime=2021-03-05T05:35:31.077Z

IT243064 Details

Final status: We've determined that a recently deployed service update resulted in a synchronization issue in the latest build, resulting in impact. We've reverted the service update and confirmed with internal monitoring that the issue is resolved.

--------------------------------------------------------------------------------------------------
Title Users may not receive the correct set of applications during the Enrollment Status Page for Autopilot provisioning
ID IT246674
Service Microsoft Intune
Feature Microsoft Intune
Status Service restored
Impact Users weren't receiving the correct set of applications during the Enrollment Status Page for Autopilot provisioning.
Severity
Start time 2021-03-04T03:45:00Z
End time 2021-03-30T21:58:00Z
Last update 2021-03-30T22:46:41.367Z
More info This issue only caused impact when certain conditions were met: - Previously enrolled devices were reset and reused. - Applications were targeted to devices. - Windows Autopilot was used for provisioning. Additionally, users who only received a subset of their expected apps could have resolved the issue by re-enrolling their devices after 48 hours. For users that received additional apps, the issue could have been resolved by deleting the apps. Final status: The fix has saturated to the remaining affected service environments and internal monitoring indicates that impact has been remediated. Scope of impact: Impact was specific to users with a previously enrolled Windows PC that uses Microsoft Intune with Autopilot to deliver applications. Start time: Thursday, March 4, 2021, at 3:45 AM UTC End time: Tuesday, March 30, 2021, at 9:58 PM UTC Root cause: A service update inadvertently introduced a race condition for the Autopilot component, which may have caused the service to identify and install the incorrect applications on initial setup. Next steps: - We're reviewing our update procedures to better identify similar application targeting issues during our development and testing cycles. This is the final update for the event.; PublishedTime=2021-03-30T22:46:41.367Z
Root cause A service update inadvertently introduced a race condition for the Autopilot component, which may have caused the service to identify and install the incorrect applications on initial setup. Next steps: - We're reviewing our update procedures to better identify similar application targeting issues during our development and testing cycles. This is the final update for the event.; PublishedTime=2021-03-30T22:46:41.367Z

IT246674 Details

Status: We're investigating a potential issue and checking for impact to your organization. We'll provide an update within 30 minutes.; PublishedTime=2021-03-26T19:20:49.27

Status: We've determined that the Autopilot component which assigns applications for installation based on configured group policies is incorrectly selecting older group records, resulting in a subset of applications installing on initial setup instead of the expected amount. We're reviewing options for potential fixes to mitigate impact. Next update by: Friday, March 26, 2021, at 9:00 PMUTC; PublishedTime=2021-03-26T19:45:40.017Z

Status: We've confirmed that for some users who only received a subset of their expected apps, the issue will self-resolve after 48 hours. We continue to investigate mitigation option for users who receive additional applications on installation. Next update by: Monday, March 29, 2021, at 10:00 PM UTC; PublishedTime=2021-03-26T20:56:14.557Z

Status: After further investigation, we've determined that users who receive a subset of the expected apps need to re-enroll their devices after 48 hours to resolve the issue. In parallel, we've determined that a code fix will be needed to completely remediate impact for all users. We're developing and internally validating our fix, designed to correct the group targeting issue, and remediate impact. Next update by: Tuesday, March 30, 2021, at 2:00 AM UTC; PublishedTime=2021-03-29T20:48:05.033Z

Status: We've completed development and validation for our fix for this issue, and we're beginning deployment to affected service environments. We'll provide an updated deployment timeline at our next scheduled update. Next update by: Tuesday, March 30, 2021, at 6:00 PMUTC; PublishedTime=2021-03-30T00:33:27.45Z

Status: We're continuing to deploy the fix to affected service environments. We expect that the issue will be fully resolved by Tuesday, March 30, 2021, at 11:00 PM UTC. Next update by: Wednesday, March 31, 2021, at 12:00 AM UTC; PublishedTime=2021-03-30T16:56:21.713Z

Final status: The fix has saturated to the remaining affected service environments and internal monitoring indicates that impact has been remediated. Next update by: Wednesday, March 31, 2021, at 12:00 AM UTC; PublishedTime=2021-03-30T16:56:21.713Z

--------------------------------------------------------------------------------------------------
Title Users may be unable to access their mailbox in the Exchange Online service via any connection method
ID EX242563
Service Exchange Online
Feature E-Mail and calendar access
Status Service restored
Impact Users may have been unable to access their mailbox in the Exchange Online service via any connection method.
Severity
Start time 2021-03-02T22:50:00Z
End time 2021-03-02T23:42:00Z
Last update 2021-03-03T00:19:02.83Z
More info
Root cause

EX242563 Details

Status: We're reviewing service health telemetry to identify the origin of impact and formulate our next troubleshooting steps. Next update by: Wednesday, March 3, 2021, at 1:00 AMUTC; PublishedTime=2021-03-02T23:40:38.433Z

Final status: We've identified a portion of infrastructure which was performing below expected thresholds, resulting in impact. Our automated recovery system repaired this problem, and we confirmed that service was restored after monitoring the environment. Next update by: Wednesday, March 3, 2021, at 1:00 AMUTC; PublishedTime=2021-03-02T23:40:38.433Z

--------------------------------------------------------------------------------------------------
Title Users may be unable to sign in to the Microsoft Teams mobile app on iOS devices
ID TM242633
Service Microsoft Teams
Feature Teams Components
Status Service restored
Impact Users may have been unable to sign in to the Microsoft Teams mobile app on iOS devices.
Severity
Start time 2021-03-02T22:00:00Z
End time 2021-03-12T20:06:00Z
Last update 2021-03-12T20:35:36.057Z
More info Affected users could have avoided this issue by removing their user account in the Microsoft Authenticator app to access the service. Alternatively, users who have access to the web app or desktop client could have leveraged these connection methods as well. Final status: We've confirmed that the fix has completed successfully in the remaining environments and has resolved the issue. Users who do not have auto-update enabled can go to the iOS App Store to download the fix and alleviate impact. Scope of impact: Any user attempting to sign in to the Microsoft Teams mobile app on iOS devices may have been impacted. Start time: Tuesday, March 2, 2021, at 10:00 PM UTC End time: Friday, March 12, 2021, at 8:06 PM UTC Root cause: A regression within a recent update for the iOS Microsoft Authenticator app was preventing users from signing in to the Microsoft Teams mobile app on iOS devices. Next steps: - We're investigating why the regression wasn't detected during our update development and validation processes to prevent this problem from happening again in the future. This is the final update for the event.; PublishedTime=2021-03-12T20:35:36.057Z
Root cause A regression within a recent update for the iOS Microsoft Authenticator app was preventing users from signing in to the Microsoft Teams mobile app on iOS devices. Next steps: - We're investigating why the regression wasn't detected during our update development and validation processes to prevent this problem from happening again in the future. This is the final update for the event.; PublishedTime=2021-03-12T20:35:36.057Z

TM242633 Details

Status: We're investigating a potential issue and checking for impact to your organization. We'll provide an update within 30 minutes.

Status: We're analyzing service logs to determine the source of the issue. Next update by: Wednesday, March 3, 2021, at 4:00 AMUTC; PublishedTime=2021-03-03T02:43:11.667Z

Status: We've determined that a regression within a recent update for the iOS Microsoft Authenticator app is preventing users from signing in to the Microsoft Teams mobile app on iOS devices. We've halted the update to prevent further impact and we're developing a fix to restore service. Once the update has been fixed, it will be available in the App Store and users should automatically receive this update. Next update by: Thursday, March 4, 2021, at 3:00 AMUTC; PublishedTime=2021-03-03T03:36:01.673Z

Status: We've completed validation of the fix; however, additional verification is needed in correlating current failure rates associated with this problem before we submit the update to Apple for approval to the App Store. Next update by: Friday, March 5, 2021, at 3:00 AMUTC; PublishedTime=2021-03-04T01:40:46.05Z

Status: We've verified our fix and submitted the update for approval in the Apple App Store. We expect the update to complete over the weekend and we'll monitor the service as it saturates iOS devices to ensure the issue is resolved. Next update by: Tuesday, March 9, 2021, at 3:00 AMUTC; PublishedTime=2021-03-05T01:20:38.573Z

Status: We've started deploying the fix, and expect it to saturate the remaining affected environments by Friday, March 12 2021, at approximately 3:00 AM UTC. Next update by: Friday, March 12, 2021, at 3:00 AMUTC; PublishedTime=2021-03-09T01:23:33.653Z

Status: The fix is taking longer than expected to saturate in the remaining environments. We expect that impact will be remediated by our next communications update. Next update by: Friday, March 12, 2021, at 10:00 PMUTC; PublishedTime=2021-03-12T01:18:23.383Z

Final status: We've confirmed that the fix has completed successfully in the remaining environments and has resolved the issue. Users who do not have auto-update enabled can go to the iOS App Store to download the fix and alleviate impact. Next update by: Friday, March 12, 2021, at 10:00 PMUTC; PublishedTime=2021-03-12T01:18:23.383Z

--------------------------------------------------------------------------------------------------
Title Some users may be unable to access some Microsoft 365 services in Brazil
ID MO242464
Service Microsoft 365 suite
Feature Portal
Status Service restored
Impact Users may have been unable to access some Microsoft 365 services in Brazil.
Severity
Start time 2021-03-02T13:25:00Z
End time 2021-03-02T14:25:00Z
Last update 2021-03-02T17:34:37.043Z
More info Impacted services included but were not limited to: - The Microsoft 365 admin center - Microsoft Teams - Microsoft Planner Final status: While we were in the process of analyzing system logs to formulate a remediation plan, our system monitoring indicated that the service had returned to normal health. We've confirmed that the problem is no longer occurring, and we'll continue to monitor the service to ensure that the problem does not happen again. Scope of impact: Impact was specific to users located in Brazil. Start time: Tuesday, March 2, 2021, at 1:25 PM'UTC End time: Tuesday, March 2, 2021, at 2:25 PM'UTC This is the final update for the event.; PublishedTime=2021-03-02T16:22:11.483Z
Root cause

MO242464 Details

Status: We're investigating a potential issue with the Microsoft 365 service. We'll provide an update within 30 minutes.; PublishedTime=2021-03-02T14:33:03.54

Status: Our telemetry indicates that the issue is no longer occurring. We're reviewing service telemetry and connectivity logs to help isolate the root cause of the problem to prevent further occurrence. Next update by: Tuesday, March 2, 2021, at 4:30 PM UTC; PublishedTime=2021-03-02T15:00:20.107Z

Final status: While we were in the process of analyzing system logs to formulate a remediation plan, our system monitoring indicated that the service had returned to normal health. We've confirmed that the problem is no longer occurring, and we'll continue to monitor the service to ensure that the problem does not happen again. Next update by: Tuesday, March 2, 2021, at 4:30 PM UTC; PublishedTime=2021-03-02T15:00:20.107Z

--------------------------------------------------------------------------------------------------
Title Users may be unable to save edits to some recently created tasks for existing projects in Project for the web
ID PW248114
Service Project for the web
Feature General
Status Service restored
Impact Users may have been unable to save edits to some recently created tasks for existing projects in Project for the web.
Severity Sev2
Start time 2021-03-02T05:00:00Z
End time 2021-04-23T01:10:00Z
Last update 2021-04-26T18:06:51.377Z
More info
Root cause A recent change intended to provide updated reporting metrics resulted in impact. Next steps: - We're reviewing our update procedures to better identify similar issues during our development and testing cycles. This is the final update for the event.; PublishedTime=2021-04-26T18:06:51.377Z

PW248114 Details

Status: We've identified an issue affecting projects with some recently created tasks where users are unable to save edits to tasks. We've determined that a recent change intended to provide updated reporting metrics resulted in impact. We've developed a code fix to prevent future projects from being affected which is currently in deployment. We anticipate it will take approximately 48 hours to fully deploy. In parallel, we're working on a script to identify and correct existing projects affected by impact. Next update by: Friday, April 2, 2021, at 9:00 PMUTC; PublishedTime=2021-04-01T18:08:45.113Z

Status: We've completed the deployment of the code fix to prevent future projects from being affected. We expect that development of the script to identify and correct the existing impacted projects will take up to two weeks. Next update by: Friday, April 9, 2021, at 9:00 PMUTC; PublishedTime=2021-04-02T19:26:42.207Z

Status: We're in the final validation stages of the script development to repair the impacted projects. We expect the script to initiate within the next 24 hours, and complete by the next scheduled update. Next update by: Friday, April 16, 2021, at 9:00 PMUTC; PublishedTime=2021-04-09T19:34:16.25Z

Status: We're continuing our final testing and validation in our internal environment, and confirmed we're seeing success restoring projects. We anticipate final validations will complete and deployment of the fix to the affected environment will initiate by our next update. Next update by: Wednesday, April 21, 2021, at 10:00 PM UTC; PublishedTime=2021-04-16T19:39:34.403Z

Status: We've successfully validated the fix and deployed it to all impacted environments to prevent edit saving issues in newly created projects and tasks. We're now correcting projects and tasks with identified edit saving problems that were created during the impact period. We expect to have this complete by our next scheduled update. Next update by: Monday, April 26, 2021, at 9:00 PMUTC; PublishedTime=2021-04-21T21:21:56.933Z

Final status: After completing our correction of projects and tasks that were created during the impact period, we've confirmed that our fix deployment has completed, and impact has been resolved. Next update by: Monday, April 26, 2021, at 9:00 PMUTC; PublishedTime=2021-04-21T21:21:56.933Z

--------------------------------------------------------------------------------------------------
Title Users receive a blank page when attempting to restore file versions and open list attachments in SharePoint Online
ID SP244198
Service SharePoint Online
Feature SharePoint Features
Status Post-incident report published
Impact Users received a blank page when attempting to restore previous file versions and open list attachments.
Severity
Start time 2021-03-02T02:27:00Z
End time 2021-03-16T14:30:00Z
Last update 2021-03-23T18:19:32.653Z
More info Opening list attachments was only affected in the classic experience. Users were able to use the modern experience can switch to avoid impact. As a workaround, users could have right clicked on the file version and then selected the "Open link in new window" option to restore the file version in the Office client. Final status: The deployment of the fix has completed and we've validated that service has been restored. Scope of impact: Any user attempting to restore file versions and open list attachments in the classic experience may have been affected by this issue. Start time: Tuesday, March 2, 2021, at 2:27 AM UTC End time: Tuesday, March 16, 2021, at 2:30 PM UTC Root cause: Fluid files are a new SharePoint file type that uses the Microsoft Fluid Framework to enable development of hyper-fast collaboration experiences for end users. This file type is scheduled for customer release soon, and a recent code change was implemented in preparation for this release which was designed to enable viewing Fluid file versions in app-specific experiences in a new tab. However, the update inadvertently activated a section of code that the service was not yet properly configured to utilize, causing users to be shown a blank page in a new tab when attempting to view previous file versions and open list file attachments. Next steps: - For a more comprehensive list of next steps and actions, please refer to the Post Incident Review document. We'll publish a post-incident report within five business days.; PublishedTime=2021-03-16T17:12:21.06Z
Root cause Fluid files are a new SharePoint file type that uses the Microsoft Fluid Framework to enable development of hyper-fast collaboration experiences for end users. This file type is scheduled for customer release soon, and a recent code change was implemented in preparation for this release which was designed to enable viewing Fluid file versions in app-specific experiences in a new tab. However, the update inadvertently activated a section of code that the service was not yet properly configured to utilize, causing users to be shown a blank page in a new tab when attempting to view previous file versions and open list file attachments. Next steps: - For a more comprehensive list of next steps and actions, please refer to the Post Incident Review document. We'll publish a post-incident report within five business days.; PublishedTime=2021-03-16T17:12:21.06Z

SP244198 Details

Status: We're analyzing network trace logs to determine the source of the issue. Next update by: Friday, March 12, 2021, at 12:30 AMUTC; PublishedTime=2021-03-11T23:21:08.093Z

Status: We've determined that a recent code change is inadvertently causing users to receive a blank page when attempting to restore previous file versions and open list attachments. We're analyzing the code change to determine a mitigation plan. Next update by: Friday, March 12, 2021, at 2:00 AMUTC; PublishedTime=2021-03-12T00:30:02.953Z

Status: We've developed a fix to remediate the code change impact and we're in the process of deploying it to the affected environment. Next update by: Friday, March 12, 2021, at 7:00 PMUTC; PublishedTime=2021-03-12T02:02:20.013Z

Status: We're continuing to monitor the fix validation and deployment as it progresses. We expect to have a mitigation timeline by the next scheduled update. Next update by: Friday, March 12, 2021, at 11:00 PMUTC; PublishedTime=2021-03-12T17:40:00.283Z

Status: The deployment of the fix is progressing as expected and we anticipate that impact will be fully remediated by Monday, March 15, 2021. Next update by: Monday, March 15, 2021, at 7:00 PM UTC; PublishedTime=2021-03-12T21:52:36.697Z

Status: The deployment of the fix has reached approximately 98.2 percent saturation. For complete details on this event, please reference the post from 9:52 PM on Friday, March 12, 2021. Next update by: Tuesday, March 16, 2021, at 7:00 PM UTC; PublishedTime=2021-03-15T17:33:02.383Next update by: Tuesday, March 16, 2021, at 7:00 PM UTC; PublishedTime=2021-03-15T17:33:02.383Z

Final status: The deployment of the fix has completed and we've validated that service has been restored. Next update by: Tuesday, March 16, 2021, at 7:00 PM UTC; PublishedTime=2021-03-15T17:33:02.383Z

--------------------------------------------------------------------------------------------------
Title Users accessing Microsoft Outlook app on iOS devices may see unexpected user display photos
ID EX242218
Service Exchange Online
Feature Sign-in
Status Service restored
Impact Users accessing the Microsoft Outlook app on iOS devices may have seen unexpected user display photos.
Severity
Start time 2021-03-01T03:50:00Z
End time 2021-03-08T22:14:00Z
Last update 2021-03-09T00:52:28.277Z
More info
Root cause A code regression in the latest version of the Microsoft Outlook app for iOS resulted in impact. Next steps: - To help prevent similar impact in the future, we're further determining the underlying cause of the code regression and how we can improve our validation procedures to catch such issues before release. This is the final update for the event.; PublishedTime=2021-03-09T00:52:28.277Z

EX242218 Details

Status: We're analyzing Outlook mobile data to determine our next steps. Next update by: Monday, March 1, 2021, at 8:30 PMUTC; PublishedTime=2021-03-01T19:19:10.887Z

Status: We're analyzing network trace logs to determine the root cause and create a strategy that remediates impact. Next update by: Monday, March 1, 2021, at 10:30 PMUTC; PublishedTime=2021-03-01T20:19:58.71Z

Status: We've determined that a code bug in the latest version of the Microsoft Outlook app for iOS has resulted in impact. We're creating a fix that will be submitted to Apple for review before being available on the Apple Store. We expect the fix to be created and ready for review by Apple at the end of Tuesday, March 2, 2021. We'll keep you updated on when the fix will reach the Apple Store for download. Next update by: Wednesday, March 3, 2021, at 2:30 AMUTC; PublishedTime=2021-03-01T22:36:11.523Z

Status: We've submitted the fix to Apple for review. Once it's been approved and introduced to the App Store, we'll update users on whether any additional actions are needed to expedite mitigation. Next update by: Wednesday, March 3, 2021, at 8:00 PMUTC; PublishedTime=2021-03-03T01:06:31.567Z

Status: Our fix is under review by Apple as they work to validate it before introducing the fix in the App Store. Once the fix is available, users who have automatic updates enabled will automatically get the fix and users who don't have automatic updates enabled can manually download the fix in the App Store to mitigate this issue. Next update by: Thursday, March 4, 2021, at 2:00 AMUTC; PublishedTime=2021-03-03T18:23:04.487Z

Status: We're revising the app update per suggestions from Apple. We'll continue to provide updates on our progress and resubmission for approval to the App Store. Next update by: Thursday, March 4, 2021, at 8:00 PMUTC; PublishedTime=2021-03-04T00:51:07.14Z

Status: We're continuing our revisions to the app and working with Apple to have the update available via the App Store as soon as possible. Next update by: Friday, March 5, 2021, at 2:00 AMUTC; PublishedTime=2021-03-04T17:59:34.893Z

Status: Our revisions to the app are taking longer than expected. We're unable to provide an estimate for when the app will be resubmitted for App Store approval at this time. We expect to provide a better idea of the timeline at our next scheduled update. Next update by: Saturday, March 6, 2021, at 2:00 AMUTC; PublishedTime=2021-03-05T00:57:58.137Z

Status: We've completed our revisions to the app and have submitted it for App Store review. Once we receive approval from Apple, we'll perform our final validation tests and deploying the app to the App Store. Next update by: Monday, March 8, 2021, at 8:00 PMUTC; PublishedTime=2021-03-05T17:29:26.74Z

Status: Our previous app revisions weren't approved by Apple. We've implemented required changes and have submitted the new build to Apple for review. Next update by: Tuesday, March 9, 2021, at 9:30 PMUTC; PublishedTime=2021-03-08T18:38:55.087Z

Final status: Apple has approved our revised app update. The update is available in the App Store and we've confirmed with affected users that service is restored. Next update by: Tuesday, March 9, 2021, at 9:30 PMUTC; PublishedTime=2021-03-08T18:38:55.087Z

--------------------------------------------------------------------------------------------------
Title Site admins are unable to save sites as templates within the SharePoint Online service
ID SP246317
Service SharePoint Online
Feature Tenant Admin
Status Service restored
Impact Site admins were unable to save sites as templates within the SharePoint Online service.
Severity
Start time 2021-03-01T01:48:00Z
End time 2021-04-06T14:30:00Z
Last update 2021-04-06T16:05:12Z
More info Admins would have seen the solution within the solution gallery after attempting to save a site as a template; however, these solutions would not have been functional. Final status: We've confirmed with affected users that our code fix has remediated impact. Scope of impact: This issue would have affected any site admins that were attempting to save sites as templates within the SharePoint Online service. Start time: Monday, March 1, 2021, at 1:48 AM'UTC End time: Tuesday, April 6, 2021, at 2:30 PM'UTC Root cause: A recent service update containing multiple feature improvements had introduced a code issue, which prevented site admins from saving sites as templates within the SharePoint Online service. Next steps: - We're reviewing our update procedures to better identify similar code issues during our development and testing cycles. This is the final update for the event.; PublishedTime=2021-04-06T15:10:01.617Z
Root cause A recent service update containing multiple feature improvements had introduced a code issue, which prevented site admins from saving sites as templates within the SharePoint Online service. Next steps: - We're reviewing our update procedures to better identify similar code issues during our development and testing cycles. This is the final update for the event.; PublishedTime=2021-04-06T15:10:01.617Z

SP246317 Details

Status: We're reviewing service telemetry to determine the next troubleshooting steps. Next update by: Thursday, March 25, 2021, at 1:00 PM UTC; PublishedTime=2021-03-25T10:40:19.63Z

Status: We're reviewing recent changes made to the affected environment in addition to analyzing correlationID logs to identify the source of impact. Next update by: Thursday, March 25, 2021, at 3:00 PM UTC; PublishedTime=2021-03-25T12:43:14.367Z

Status: Our review of recent changes to the affected environment was inconclusive. Based on the information from the correlationID logs, we've successfully reproduced the problem in our testing environment and are actively debugging the issue to isolate the underlying cause of impact. Next update by: Thursday, March 25, 2021, at 5:00 PM UTC; PublishedTime=2021-03-25T14:19:01.263Z

Status: We've identified a recent service update that has introduced a code issue, which is preventing site admins from saving sites as templates with the SharePoint Online service. We're exploring the quickest method for remediating impact by investigating whether reverting the update will be possible or a code fix will need to be developed. Next update by: Friday, March 26, 2021, at 12:00 PMUTC; PublishedTime=2021-03-25T15:36:59.6Z

Status: We've developed a code fix which we are currently validating before deploying it to the impacted infrastructure. Next update by: Monday, March 29, 2021, at 2:30 PM UTC; PublishedTime=2021-03-26T11:41:47.84Z

Status: We've validated the code fix and are preparing the fix to be deployed, which we expect to start in the next 72 hours. Next update by: Wednesday, March 31, 2021, at 12:00 PM UTC; PublishedTime=2021-03-29T13:02:43.177Z

Status: We've successfully deployed our fix to all the affected infrastructure. The progress of the fix's saturation is approximately 70% worldwide. We anticipate that saturation will be complete within the next 24 hours. During this time, users may start to experience relief. We'll continue to monitor this deployment as it saturates globally. Next update by: Thursday, April 1, 2021, at 12:00 PM UTC; PublishedTime=2021-03-31T10:36:09.933Z

Status: The fix is approximately 90% deployed and we're continuing to monitor it's progress as it approaches it's final stages of saturation. Next update by: Friday, April 2, 2021, at 4:30 PM UTC; PublishedTime=2021-04-01T10:55:55.32Z

Status: The fix remains over 90 percent deployed and we expect it to complete over the weekend. Once fully saturated, we'll contact affected users to confirm impact has been remediated. Next update by: Monday, April 5, 2021, at 4:00 PMUTC; PublishedTime=2021-04-02T15:10:10.02Z

Status: The fix has completely saturated the affected environment, and we're waiting for confirmation from affected users that impact has been remediated. Next update by: Tuesday, April 6, 2021, at 4:00 PMUTC; PublishedTime=2021-04-05T15:01:53.007Z

Final status: We've confirmed with affected users that our code fix has remediated impact. Next update by: Tuesday, April 6, 2021, at 4:00 PMUTC; PublishedTime=2021-04-05T15:01:53.007Z

--------------------------------------------------------------------------------------------------
Title Admins assigning Exchange Online licenses may notice Simple Mail Transfer Protocol (SMTP) addresses becoming unavailable
ID EX245679
Service Exchange Online
Feature Management and Provisioning
Status Service restored
Impact Admins assigning Exchange Online licenses may have noticed that SMTP addresses weren't appearing as expected.
Severity
Start time 2021-03-01T00:00:00Z
End time 2021-03-23T17:00:00Z
Last update 2021-03-26T03:53:05.557Z
More info
Root cause A recent feature update intended to facilitate SMTP address provisioning inadvertently resulted in impact. Next steps: - We're reviewing the impacting feature update to better understand why the issue took place and how we can prevent any similar issues from reoccurring. This is the final update for the event.; PublishedTime=2021-03-26T03:53:05.557Z

EX245679 Details

Status: We're investigating a potential issue and checking for impact to your organization. We'll provide an update within 30 minutes.; PublishedTime=2021-03-22T04:31:04.407

Status: We're reviewing system logs to help identify the source of the issue and determine our next steps toward remediation. Next update by: Monday, March 22, 2021, at 6:00 AMUTC; PublishedTime=2021-03-22T04:50:20.123Z

Status: We're looking into a potential syncing issue that may be causing the SMTP addresses to be dropped. Next update by: Monday, March 22, 2021, at 8:00 AMUTC; PublishedTime=2021-03-22T05:58:52.717Z

Status: As we continue to investigate the potential sync issue, we're investigating a possible issue with the Azure Active Directory (AAD) in parallel to identify the underlying cause of impact. Next update by: Monday, March 22, 2021, at 10:00 AM UTC; PublishedTime=2021-03-22T07:37:34.537Z

Status: We've identified recent changes to some Exchange Online infrastructure that may be causing impact. We're reviewing the nature of these changes to isolate the root cause of the issue and prepare appropriate mitigation steps. Next update by: Monday, March 22, 2021, at 12:00 PM UTC; PublishedTime=2021-03-22T09:49:48.027Z

Status: As we continue to investigate the recent change made to the affected infrastructure, we're reviewing mitigation options with responsible subject matter experts. Next update by: Monday, March 22, 2021, at 2:00 PM UTC; PublishedTime=2021-03-22T11:38:30.557Z

Status: We're analyzing change logs to understand the nature of the issue while we also review options to mitigate the issue without causing unintended impact elsewhere. Next update by: Monday, March 22, 2021, at 7:00 PM UTC; PublishedTime=2021-03-22T13:47:56.077Z

Status: Our review of the change logs has proven inconclusive thus far, however we've identified a recently enabled feature that we suspect may be responsible for impact. We're reviewing options to potentially disable this feature to confirm our findings and potentially remediate impact, while in parallel, we're continuing to investigate the underlying root cause. Next update by: Tuesday, March 23, 2021, at 4:00 AMUTC; PublishedTime=2021-03-22T18:52:50.153Z

Status: Our investigation indicates that a recent feature update intended to facilitate SMTP address provisioning inadvertently resulted in impact. We're working to revert that update to resolve this issue. Next update by: Tuesday, March 23, 2021, at 6:00 AMUTC; PublishedTime=2021-03-23T03:01:09.363Z

Status: Our work to revert the impacting feature update is progressing as expected, which we anticipate to complete in a few hours. Next update by: Tuesday, March 23, 2021, at 9:00 AMUTC; PublishedTime=2021-03-23T03:48:42.613Z

Status: We've encountered unexpected issues reverting the impacting feature update. We're working to resolve those issues to ensure the reversion completes as quickly as possible. Next update by: Wednesday, March 24, 2021, at 5:00 AMUTC; PublishedTime=2021-03-23T05:41:40.43Z

Status: We've successfully reverted the impacting feature update and we're reaching out to affected admins to confirm that service is restored. Next update by: Thursday, March 25, 2021, at 5:00 AMUTC; PublishedTime=2021-03-24T03:43:32.29Z

Status: We're continuing to reach out to some affected admins to confirm that the issue is fully resolved. Next update by: Friday, March 26, 2021, at 5:00 AMUTC; PublishedTime=2021-03-25T03:52:41.153Z

Final status: We've confirmed with some affected users that the issue is fully resolved. Next update by: Friday, March 26, 2021, at 5:00 AMUTC; PublishedTime=2021-03-25T03:52:41.153Z

--------------------------------------------------------------------------------------------------
Title Users in Europe can't access Yammer
ID YA241857
Service Yammer Enterprise
Feature Yammer Components
Status Service restored
Impact Users may have been unable to access Yammer.
Severity
Start time 2021-02-26T21:41:00Z
End time 2021-02-26T23:12:55.2Z
Last update 2021-02-26T23:41:10.58Z
More info
Root cause An unexpected traffic processing issue affected a portion of the infrastructure which supports the Yammer service, causing impact. Next steps: - We're reviewing our code for improved automated recovery mechanisms to reduce or avoid impact due to similar traffic processing issues in the future. This is the final update for the event.; PublishedTime=2021-02-26T23:12:55.2Z

YA241857 Details

Final status: We identified an unexpected traffic processing issue affecting a portion of the infrastructure which supports the Yammer service, leading to impact. We've rerouted user traffic to an alternate portion of infrastructure, which has resolved the issue and mitigated the impact to users.

--------------------------------------------------------------------------------------------------
Title Some users in Japan may notice that services and custom apps that leverage the Azure App Service may experience issues
ID MO241785
Service Microsoft 365 suite
Feature Portal
Status Post-incident report published
Impact Users may notice that services and custom apps that leverage the Azure App Service may experience varying issues.
Severity
Start time 2021-02-26T03:29:00Z
End time 2021-02-26T10:02:00Z
Last update 2021-03-05T18:41:19.73Z
More info The following impact scenarios have been identified: - Exchange Online users may have been unable to sign in. - Users may have been unable to access the Microsoft 365 Planner service. - Some Teams users may have had problems sending and receiving media such as images, audio and video messages shared in chats. - Users may have been unable to open Bookings in Web. - OneNote for the web and the desktop client were experiencing slowness and sync failures. Final status: Our mitigation actions have completed successfully and we have verified that the issue is now resolved. Scope of impact: This issue only affected admins and users that were routed through the Japan infrastructure. Start time: Friday, February 26, 2021, at 3:29 AM UTC End time: Friday, February 26, 2021, at 10:02 AM UTC Root cause: There were contributing factors that led to customer impact. Firstly, we had an active deployment in progress on a single storage scale unit. Our safe deployment process normally reserves some resources within a scale unit so that deployments can take place. In addition to this space being reserved for the deployment, some components in the scale unit entered an unhealthy state and so they were removed from use from the scale unit. The final factor was that resource demand on the scale unit was unusually high. In this case, our resource balancing automation was not able to keep up and spread the load to other scale units. A combination of all these factors resulted in a high utilization of this scale unit causing it to be heavily throttled to prevent failure. This resulted in a loss of availability for customers and Azure services attempting to utilize Storage resources within the impacted storage scale unit. Next steps: - For a more comprehensive list of next steps and actions, please refer to the Post Incident Review document. We'll publish a post-incident report within five business days.; PublishedTime=2021-02-26T10:26:11.573Z
Root cause There were contributing factors that led to customer impact. Firstly, we had an active deployment in progress on a single storage scale unit. Our safe deployment process normally reserves some resources within a scale unit so that deployments can take place. In addition to this space being reserved for the deployment, some components in the scale unit entered an unhealthy state and so they were removed from use from the scale unit. The final factor was that resource demand on the scale unit was unusually high. In this case, our resource balancing automation was not able to keep up and spread the load to other scale units. A combination of all these factors resulted in a high utilization of this scale unit causing it to be heavily throttled to prevent failure. This resulted in a loss of availability for customers and Azure services attempting to utilize Storage resources within the impacted storage scale unit. Next steps: - For a more comprehensive list of next steps and actions, please refer to the Post Incident Review document. We'll publish a post-incident report within five business days.; PublishedTime=2021-02-26T10:26:11.573Z

MO241785 Details

Status: We've received an automated alert notifying us of performance issues occurring in a subsection of service infrastructure. We're analyzing relevant data and logs to determine the source of the issue and additional impact scenarios. Next update by: Friday, February 26, 2021, at 6:00 AMUTC; PublishedTime=2021-02-26T04:39:39.36Z

Status: Our investigation indicates that a portion of service infrastructure that hosts multiple Microsoft 365 service features is performing below expected standards, causing impact. We're continuing to analyze service logs and performance metrics to validate this theory and determine our best course of mitigation. Next update by: Friday, February 26, 2021, at 6:30 AMUTC; PublishedTime=2021-02-26T05:25:27.347Z

Status: We've identified some migration actions may be contributing to the impact. We're expediting those migrations to alleviate resource for the impacted infrastructure. Next update by: Friday, February 26, 2021, at 8:00 AMUTC; PublishedTime=2021-02-26T06:23:43.917Z

Status: Internal service health monitoring indicates that service is recovering as the migrations continue to progress. In addition, we've received initial reports that users are seeing progressive relief throughout various services. We're continuing to expedite the migrations to mitigate impact. Next update by: Friday, February 26, 2021, at 9:00 AMUTC; PublishedTime=2021-02-26T07:54:54.037Z

Status: Our monitoring indicates that service is continuing to recover while we're completing mitigation actions to ensure full recovery. Next update by: Friday, February 26, 2021, at 11:00 AM UTC; PublishedTime=2021-02-26T09:00:57.09Z

Final status: Our mitigation actions have completed successfully and we have verified that the issue is now resolved. Next update by: Friday, February 26, 2021, at 11:00 AM UTC; PublishedTime=2021-02-26T09:00:57.09Z

--------------------------------------------------------------------------------------------------
Title Users can't open Microsoft Bookings on the web through some URLs
ID MO242513
Service Microsoft 365 suite
Feature Portal
Status Service restored
Impact Users were unable to open Microsoft Bookings on the web through some URLs.
Severity
Start time 2021-02-26T02:10:00Z
End time 2021-03-03T04:00:00Z
Last update 2021-03-04T05:54:24.293Z
More info Users were alternatively able to access Microsoft Bookings via the following methods: - Signing into Outlook on the web via https://outlook.office.com/owa/ and then accessing Bookings - Navigating to https://bookings.office.com Final status: We've determined that a code regression was the cause of the issue. We've developed and deployed the fix and have confirmed via telemetry and your representatives that the issue is now resolved. Scope of impact: This issue could have potentially affected any of your users if they were attempting to open Microsoft Bookings on the web through some URLs, including https://outlook.office.com/bookings. Start time: Friday, February 26, 2021, at 2:10 AM'UTC End time: Wednesday, March 3, 2021, at 4:00 AM'UTC Root cause: A code regression occurred in a recent update, resulting in impact. Next steps: - We're reviewing the portion of regressed code to determine why the regression occurred to help prevent similar problems in the future. This is the final update for the event.; PublishedTime=2021-03-04T05:54:24.293Z
Root cause A code regression occurred in a recent update, resulting in impact. Next steps: - We're reviewing the portion of regressed code to determine why the regression occurred to help prevent similar problems in the future. This is the final update for the event.; PublishedTime=2021-03-04T05:54:24.293Z

MO242513 Details

Status: We've received reports indicating that users are unable to open Bookings on the web through some URLs. We suspect that this is a routing issue involving some Microsoft Bookings and Outlook on the web URLs but are still working to confirm the cause. In parallel, we're developing a fix to appropriately route users, which we expect to will begin deployment the week of Monday, March 8, 2021. We'll provide additional root cause details as well as a deployment timeline at our next scheduled update. Next update by: Friday, March 5, 2021, at 10:00 AMUTC; PublishedTime=2021-03-02T20:37:43.527Z

Final status: We've determined that a code regression was the cause of the issue. We've developed and deployed the fix and have confirmed via telemetry and your representatives that the issue is now resolved. Next update by: Friday, March 5, 2021, at 10:00 AMUTC; PublishedTime=2021-03-02T20:37:43.527Z

--------------------------------------------------------------------------------------------------
Title Some users may not have seen question text when creating polls with Microsoft Forms
ID FM244248
Service Microsoft Forms
Feature Form functionality
Status Service restored
Impact Users may not have seen question text when creating polls with Microsoft Forms.
Severity
Start time 2021-02-26T02:02:00Z
End time 2021-03-12T10:50:00Z
Last update 2021-03-15T15:07:35.207Z
More info
Root cause A standard update, designed to improve the overall performance of the service contained a code regression that caused impact. Next steps: - We're reviewing our update procedures to better identify similar issues during our development and testing cycles. This is the final update for the event.; PublishedTime=2021-03-12T11:11:25.78Z

FM244248 Details

Status: We've identified a recent service update included a code regression causing question text to appear blank. We're developing a fix before testing and deploying to the affected infrastructure. Next update by: Friday, March 12, 2021, at 11:30 AM UTC; PublishedTime=2021-03-12T08:17:22.373Z

Final status: We completed the roll out of the code fix to the affected environment and after testing, have confirmed the issue has been mitigated. Next update by: Friday, March 12, 2021, at 11:30 AM UTC; PublishedTime=2021-03-12T08:17:22.373Z

--------------------------------------------------------------------------------------------------
Title Users can't join conference calls through Public Switched Telephone Network (PSTN) via Skype for Business
ID LY241735
Service Skype for Business
Feature Audio and Video
Status Service restored
Impact Users may have been unable to join conference calls through PSTN via Skype for Business.
Severity
Start time 2021-02-25T19:00:00Z
End time 2021-02-25T23:37:08.717Z
Last update 2021-02-25T23:37:08.717Z
More info
Root cause

LY241735 Details

Final status: Our automated recovery system repaired this problem, and we confirmed that service was restored after monitoring the environment.

--------------------------------------------------------------------------------------------------
Title Users can't join conference calls through Public Switched Telephone Network (PSTN) via Skype for Business
ID LY241735
Service Skype for Business
Feature Audio and Video
Status Service restored
Impact Users may have been unable to join conference calls through PSTN via Skype for Business.
Severity
Start time 2021-02-25T19:00:00Z
End time 2021-02-25T23:37:08.717Z
Last update 2021-02-25T23:37:08.717Z
More info
Root cause

LY241735 Details

Final status: Our automated recovery system repaired this problem, and we confirmed that service was restored after monitoring the environment.

--------------------------------------------------------------------------------------------------
Title Some users may be unable to sign in to the Microsoft Stream service
ID SV241643
Service Microsoft Stream
Feature Live Events
Status Post-incident report published
Impact Users may have been unable to sign in to the Microsoft Stream service.
Severity
Start time 2021-02-25T07:14:00Z
End time 2021-02-25T23:30:00Z
Last update 2021-03-04T23:58:09.607Z
More info Some users were unable to upload a Teams meeting recording to Stream. Additionally, users may have been unable to initiate the playback of Stream media in Teams. Final status: We've confirmed that the service availability remains at expected levels via our extended monitoring. Scope of impact: Impact was specific to users who were hosted out of the affected infrastructure located in the Europe, Middle East, and Africa regions. Start time: Thursday, February 25, 2021, at 7:14 AM'UTC End time: Thursday, February 25, 2021, at 11:30 PM'UTC Root cause: We've determined that some back-end services responsible for processing Teams external encoder live events and Microsoft Stream video upload and playback were not operating as efficiently as expected. There were three different problems: 1) A bug in Microsoft Teams where the client would make an API call to Microsoft Stream service at the beginning of every meeting a user joined. This contributed to approximately 40% higher API traffic. 2) A spike in user requests to join Microsoft Teams meetings caused locking and blocking on some Microsoft Stream database components. This caused a spike in API requests. The requests eventually timed out or failed and were subsequently retried. This 'retry storm' prevented the service from processing the requests in a timely manner. 3) A third service was retrying at a constant rate over the course of multiple hours. The incoming request rate for these retries were higher than Stream typically sees and introduced previously unseen contention on a different database table. Next steps: - For a more comprehensive list of next steps and actions, please refer to the Post Incident Review document. We'll publish a post-incident report within five business days.; PublishedTime=2021-02-26T03:48:44.237Z
Root cause We've determined that some back-end services responsible for processing Teams external encoder live events and Microsoft Stream video upload and playback were not operating as efficiently as expected. There were three different problems: 1) A bug in Microsoft Teams where the client would make an API call to Microsoft Stream service at the beginning of every meeting a user joined. This contributed to approximately 40% higher API traffic. 2) A spike in user requests to join Microsoft Teams meetings caused locking and blocking on some Microsoft Stream database components. This caused a spike in API requests. The requests eventually timed out or failed and were subsequently retried. This 'retry storm' prevented the service from processing the requests in a timely manner. 3) A third service was retrying at a constant rate over the course of multiple hours. The incoming request rate for these retries were higher than Stream typically sees and introduced previously unseen contention on a different database table. Next steps: - For a more comprehensive list of next steps and actions, please refer to the Post Incident Review document. We'll publish a post-incident report within five business days.; PublishedTime=2021-02-26T03:48:44.237Z

SV241643 Details

Status: We're analyzing diagnostic data to determine the source of the issue. Next update by: Thursday, February 25, 2021, at 12:00 PM UTC; PublishedTime=2021-02-25T10:01:16.603Z

Status: We've identified an unexpected spike in user traffic was causing a database request threshold to be reached. We're analyzing database logs to isolate why this isolate why this is occurring and develop a mitigation plan. Next update by: Thursday, February 25, 2021, at 1:00 PM UTC; PublishedTime=2021-02-25T10:53:29.767Z

Status: We've identified that the threshold was being reached as our automated systems were not resolving excess connections within an acceptable time span. We're redirecting user traffic to alternate infrastructure to mitigate impact. Once completed, we'll restart the affected systems to resolve the issue with connections not being closed. Next update by: Thursday, February 25, 2021, at 3:00 PM UTC; PublishedTime=2021-02-25T12:41:30.183Z

Status: Migrating users and restarting the affected infrastructure was not successful in mitigating impact. We're attempting to update the lock on the connections to reduce the amount of queries queued and to remediate impact. Next update by: Thursday, February 25, 2021, at 5:00 PM UTC; PublishedTime=2021-02-25T14:51:20.227Z

Status: We're continuing our work to implement a configuration change to reduce the connection queues. In parallel, we're evaluating additional mitigation strategies to provide relief within the environment. Next update by: Thursday, February 25, 2021, at 6:00 PM UTC; PublishedTime=2021-02-25T16:32:12.7Z

Status: We're implementing the configuration change and are restarting some services to process the connection queues. Next update by: Thursday, February 25, 2021, at 7:00 PM UTC; PublishedTime=2021-02-25T17:49:05.447Z

Status: We've restarted approximately 66 percent of the service components and we continue to monitor the environment whilst we restart the remaining infrastructure. Next update by: Thursday, February 25, 2021, at 9:00 PM UTC; PublishedTime=2021-02-25T18:59:46.817Z

Status: We've completed restarting the service components and have not observed any relief within the environment. We understand the significant impact this may have on your organization and are working to determine the optimal solution for remediation at our highest priority. Next update by: Thursday, February 25, 2021, at 10:00 PM UTC; PublishedTime=2021-02-25T20:21:33.19Z

Status: We identified several back-end services that were overloaded and have since scaled them back. We've observed significant relief within the environment and confirmed that user impact has been remediated. We're entering an extended period of monitoring to ensure service availability remains at a healthy level prior to and during the start of business hours in the affected region. Next update by: Friday, February 26, 2021, at 4:00 AM UTC; PublishedTime=2021-02-25T22:08:06.663Z

Final status: We've confirmed that the service availability remains at expected levels via our extended monitoring. Next update by: Friday, February 26, 2021, at 4:00 AM UTC; PublishedTime=2021-02-25T22:08:06.663Z

--------------------------------------------------------------------------------------------------
Title Admins can't download deprecation reports within the Service Trust Portal
ID MO246336
Service Microsoft 365 suite
Feature Administration
Status Service restored
Impact Admins were unable to download any deprecation reports on the Service Trust Portal.
Severity Sev2
Start time 2021-02-25T07:00:00Z
End time 2021-05-04T03:17:00Z
Last update 2021-05-04T04:23:07.097Z
More info
Root cause A code issue associated with report and data generation was preventing admins from downloading deprecation reports within the Service Trust Portal. Next steps: -We're reviewing our code change process to help find ways to identify similar issues in the future. This is the final update for the event.; PublishedTime=2021-05-04T04:23:07.097Z

MO246336 Details

Status: We're analyzing system logs to determine the source of the issue. Next update by: Thursday, March 25, 2021, at 2:30 PM UTC; PublishedTime=2021-03-25T13:04:07.237Z

Status: We're reviewing extensive diagnostic data to determine our next troubleshooting steps. Next update by: Friday, March 26, 2021, at 10:00 AM UTC; PublishedTime=2021-03-25T14:04:39.517Z

Status: Analysis of current diagnostic data has so far proven inconclusive. We're gathering fresh service telemetry logs to help us with our investigation in determining the source of the issue. Next update by: Friday, March 26, 2021, at 7:00 PM UTC; PublishedTime=2021-03-26T09:59:32.85Z

Status: We're analyzing fresh service telemetry logs to better understand the root cause as well as attempting to reproduce the issue in our test environment to identify potential code issues. Next update by: Saturday, March 27, 2021, at 12:00 AMUTC; PublishedTime=2021-03-26T18:01:23.68Z

Status: We've reproduced the issue in our test environment and are continuing to analyze the telemetry data to better understand the root cause andidentify potential code issues. Next update by: Saturday, March 27, 2021, at 7:00 PMUTC; PublishedTime=2021-03-26T22:34:45.09Z

Status: We've identified that the data that is used to generate the reports may have changed format, and we've developed a potential code fix that will allow the service to handle the unexpected format. We're performing tests to ensure the fix mitigates impact while in parallel, we're reviewing recent changes that may have caused the format change and impact to occur. Next update by: Monday, March 29, 2021, at 10:30 PMUTC; PublishedTime=2021-03-27T17:53:12.027Z

Status: We've deployed the first aspect of our code fix to the affected environment and are monitoring its progression. We're expecting to be able to confirm a timeline for the fix deployment during our next scheduled update. The second part of the code fix, which will fully remediate the problem, is still under development. Next update by: Tuesday, March 30, 2021, at 10:00 PMUTC; PublishedTime=2021-03-29T18:06:36.55Z

Status: We've determined that the first aspect of our code fix had experienced a problem and our internal validation process continues to confirm that it will be acceptable for deployment. The second aspect of our code fix is taking longer than expected to develop which is preventing us from providing an update for its deployment at this time. We'll provide an update on the completion of both aspects of our remediation plan once available. Next update by: Wednesday, March 31, 2021, at 10:00 PMUTC; PublishedTime=2021-03-30T20:35:12.053Z

Status: The processes of validating the first aspect of our code fix, and developing the second, are relying on some additional dependencies, further delaying their completion. We'll provide a estimated deployment timeline once both aspects of code fix have completed development and validation. Next update by: Thursday, April 1, 2021, at 11:00 PMUTC; PublishedTime=2021-03-31T22:00:12.263Z

Status: Both aspects of the fix are still in progress due to the complexity this solution. Once both code fixes are fully validated, we'll provide an estimated deployment timeline. Next update by: Friday, April 2, 2021, at 11:00 PMUTC; PublishedTime=2021-04-01T22:41:04.66Z

Status: One aspect of the fix has been validated. Once the other aspect of the code fix is fully validated, we'll provide an estimated deployment timeline. Next update by: Monday, April 5, 2021, at 11:30 PMUTC; PublishedTime=2021-04-02T22:34:18.973Z

Status: We're continuing our efforts to validate all aspects of the fix prior to deployment. We expect to have a mitigation timeline by our next scheduled update. Next update by: Tuesday, April 6, 2021, at 11:30 PMUTC; PublishedTime=2021-04-05T21:58:19.997Z

Status: We're continuing to validate a portion of the fix in our testing environment, which is taking more time than expected. As the validation process is ongoing, we're reviewing our options to deploy the fix to the affected environment as efficiently as possible. Next update by: Wednesday, April 7, 2021, at 10:00 PMUTC; PublishedTime=2021-04-06T22:30:01.123Z

Status: We've identified an issue with the fix that is being addressed before validation can complete. As the validation process is ongoing, we're continuing to review our options to deploy the fix to the affected environment as efficiently as possible. Next update by: Thursday, April 8, 2021, at 7:00 PMUTC; PublishedTime=2021-04-07T21:22:11.36Z

Status: We're continuing to monitor our validation procedures as they progress through our test environments and expect the fix to complete deployment approximately the week of Monday, May 3, 2021. Next update by: Monday, April 12, 2021, at 7:00 PMUTC; PublishedTime=2021-04-08T18:41:27.917Z

Status: Deployment of the fix to our test environments is nearing completion and is progressing as expected. We still expect the fix to complete deployment approximately the week of Monday, May 3, 2021, and we're reviewing our options to potentially expedite this process. Next update by: Wednesday, April 14, 2021, at 5:00 AMUTC; PublishedTime=2021-04-12T18:06:10.24Z

Status: Deployment of the fix to our test environments completed successfully. We've begun deployment of the fix to the remaining affected environment and estimate that it will be done by the week of Monday, May 3, 2021. Next update by: Friday, April 16, 2021, at 5:00 AMUTC; PublishedTime=2021-04-14T03:26:05.017Z

Status: Deployment of the fix to our test environments completed successfully. We've begun deployment of the fix to the remaining affected environment and estimate that it will be done by the week of Monday, May 3, 2021. Next update by: Friday, April 16, 2021, at 5:00 AMUTC; PublishedTime=2021-04-14T03:26:39.753Z

Status: The deployment of the fix is at approximately 92 percent and is progressing as expected. We anticipate that at the current rate of deployment, the fix should be 100 percent saturated by Tuesday, April 20, 2021. Next update by: Tuesday, April 20, 2021, at 5:00 AMUTC; PublishedTime=2021-04-16T03:17:42.51Z

Status: The deployment of the fix is approximately 93 percent complete. At the current deployment rate we expect it to complete by our initial estimated date which is by the week of Monday, May 3, 2021. Next update by: Thursday, April 22, 2021, at 5:00 AMUTC; PublishedTime=2021-04-20T03:51:11.627Z

Status: The progression of the fix is deploying as expected and is approximately 97 percent complete. We expect the remaining three percent to complete by May 3, 2021. Next update by: Wednesday, April 28, 2021, at 5:00 AMUTC; PublishedTime=2021-04-21T03:21:21.61Z

Status: The fix is still at 97% saturation due to unexpected delays. We still expect the saturation to reach 100% by May 3, 2021. Next update by: Thursday, April 29, 2021, at 5:00 AMUTC; PublishedTime=2021-04-28T03:38:42.42Z

Status: The deployment is continuing to progress as expected, although the fix remains 97 percent saturated. The remaining percent of affected infrastructure is taking longer to saturate as a precaution to avoid unnecessary risks to overall service stability. This process is expected to take an extended period of time, and we expect the fix to be fully saturated by May 4, 2021. Next update by: Tuesday, May 4, 2021, at 5:00 AMUTC; PublishedTime=2021-04-29T00:46:23.737Z

Final status: We verified that the fix deployment completed and that the impact is remediated. Next update by: Tuesday, May 4, 2021, at 5:00 AMUTC; PublishedTime=2021-04-29T00:46:23.737Z

--------------------------------------------------------------------------------------------------
Title Admins may notice incomplete results when executing eDiscovery searches
ID MO254624
Service Microsoft 365 suite
Feature Administration
Status Restoring service
Impact Admins may notice that some eDiscovery queries will return incomplete results.
Severity Sev2
Start time 2021-02-25T00:00:00Z
End time
Last update 2021-05-05T22:29:51.597Z
More info
Root cause

MO254624 Details

--------------------------------------------------------------------------------------------------
Title Admins may notice incomplete results when executing eDiscovery searches
ID MO254624
Service Microsoft 365 suite
Feature Administration
Status Restoring service
Impact Admins may notice that some eDiscovery queries will return incomplete results.
Severity Sev2
Start time 2021-02-25T00:00:00Z
End time
Last update 2021-05-05T22:29:51.597Z
More info
Root cause

MO254624 Details

--------------------------------------------------------------------------------------------------
Title Some users may have been unable to connect to the Exchange Online service when using a REST connection method
ID EX241419
Service Exchange Online
Feature Sign-in
Status Service restored
Impact Users can't connect to the Exchange Online service when using a Representational State Transfer (REST) connection method
Severity
Start time 2021-02-24T11:27:00Z
End time 2021-02-24T11:44:00Z
Last update 2021-02-24T12:24:22.663Z
More info
Root cause A portion of infrastructure, responsible for handling REST connection requests, was performing below acceptable performance thresholds. Next steps: - We're analyzing performance data and trends on the affected systems to help prevent this problem from happening again. This is the final update for the event.; PublishedTime=2021-02-24T12:24:10.74Z

EX241419 Details

Status: We've identified a portion of infrastructure, responsible for handling REST connection requests, is performing below acceptable performance thresholds. We're running commands to redirect traffic to alternate infrastructure. Next update by: Wednesday, February 24, 2021, at 2:00 PM UTC; PublishedTime=2021-02-24T12:09:32.797Z

Final status: We've successfully run our commands in addition to rebooting the affected infrastructure. After a period of monitoring, we've confirmed that the issue is now resolved. Next update by: Wednesday, February 24, 2021, at 2:00 PM UTC; PublishedTime=2021-02-24T12:09:32.797Z

--------------------------------------------------------------------------------------------------
Title Users may experience general latency or errors when attempting to access OneDrive for Business content
ID OD241374
Service OneDrive for Business
Feature OneDrive for Business
Status Service restored
Impact Users may have noticed general latency or errors when attempting to access OneDrive for Business content.
Severity
Start time 2021-02-24T04:00:00Z
End time 2021-02-24T05:29:00Z
Last update 2021-02-24T05:48:49.473Z
More info
Root cause Due to inclement weather around one of our North American region data centers, we had proactively rerouted user requests to alternate systems to ensure optimum service availability. This is the final update for the event.; PublishedTime=2021-02-24T05:48:49.473Z

OD241374 Details

Status: Due to inclement weather around one of our North American Region data centers, we had proactively rerouted user requests to alternate systems to ensure optimum service availability. We've confirmed that the weather conditions have improved and are now working to route user requests back to the data center to balance service requests between our data centers and restore optimal service balance. Next update by: Wednesday, February 24, 2021, at 6:00 AMUTC; PublishedTime=2021-02-24T04:00:07.457Z

Final status: We've confirmed that the process to route user requests back to the data center is complete and service is restored. Next update by: Wednesday, February 24, 2021, at 6:00 AMUTC; PublishedTime=2021-02-24T04:00:07.457Z

--------------------------------------------------------------------------------------------------
Title Users may experience general latency or errors when attempting to access OneDrive for Business content
ID OD241374
Service OneDrive for Business
Feature OneDrive for Business
Status Service restored
Impact Users may have noticed general latency or errors when attempting to access OneDrive for Business content.
Severity
Start time 2021-02-24T04:00:00Z
End time 2021-02-24T05:29:00Z
Last update 2021-02-24T05:48:49.473Z
More info
Root cause Due to inclement weather around one of our North American region data centers, we had proactively rerouted user requests to alternate systems to ensure optimum service availability. This is the final update for the event.; PublishedTime=2021-02-24T05:48:49.473Z

OD241374 Details

Status: Due to inclement weather around one of our North American Region data centers, we had proactively rerouted user requests to alternate systems to ensure optimum service availability. We've confirmed that the weather conditions have improved and are now working to route user requests back to the data center to balance service requests between our data centers and restore optimal service balance. Next update by: Wednesday, February 24, 2021, at 6:00 AMUTC; PublishedTime=2021-02-24T04:00:07.457Z

Final status: We've confirmed that the process to route user requests back to the data center is complete and service is restored. Next update by: Wednesday, February 24, 2021, at 6:00 AMUTC; PublishedTime=2021-02-24T04:00:07.457Z

--------------------------------------------------------------------------------------------------
Title Some admins may have found a blank audit search page within the Security and Compliance Center
ID EX241424
Service Exchange Online
Feature Networking Issues
Status Service restored
Impact Admins may have found a blank audit search page within the Security and Compliance Center.
Severity
Start time 2021-02-22T18:20:00Z
End time 2021-02-24T15:25:00Z
Last update 2021-02-24T15:31:49.743Z
More info
Root cause A code change in a recent deployment caused unexpected impact to audit search. Next steps: - We're reviewing our update procedures to better identify similar issues during our development and testing cycles. This is the final update for the event.; PublishedTime=2021-02-24T15:31:49.743Z

EX241424 Details

Status: We're reviewing service telemetry to determine the next troubleshooting steps. Next update by: Wednesday, February 24, 2021, at 4:00 PM UTC; PublishedTime=2021-02-24T14:09:51.483Z

Status: We've determined that a code change in a recent deployment has caused unexpected impact to audit search. We're reviewing actions to revert the deployment, to mitigate impact. Next update by: Wednesday, February 24, 2021, at 5:00 PM UTC; PublishedTime=2021-02-24T15:00:48.993Z

Final status: We've successfully reverted the deployment, and after conducting tests, have verified that the issue is resolved. Next update by: Wednesday, February 24, 2021, at 5:00 PM UTC; PublishedTime=2021-02-24T15:00:48.993Z

--------------------------------------------------------------------------------------------------
Title Admins may see inaccurate data in some reports in the Microsoft 365 admin center
ID MO240984
Service Microsoft 365 suite
Feature Administration
Status Service restored
Impact Admins may have seen inaccurate data in some reports in the Microsoft 365 admin center.
Severity
Start time 2021-02-22T03:00:00Z
End time 2021-02-26T00:00:00Z
Last update 2021-03-01T05:10:56.497Z
More info The following report data may have been inaccurate: -SharePoint Online -OneDrive for Business -Office 365 Group Final status: We confirmed that backfill completed and that the reports have been updated. Scope of impact: This issue affected admins attempting to review the reports listed in the More info section within the Microsoft 365 admin center. Start time: Monday, February 22, 2021, at 3:00 AM'UTC End time: Friday, February 26, 2021, at 12:00 AM'UTC Root cause: An issue with report data collection logic was causing inaccurate data in the Microsoft 365 admin center. Next steps: -We're continuing our analysis of the report data collection logic to help develop ways to prevent similar issues in the future. This is the final update for the event.; PublishedTime=2021-03-01T05:10:56.497Z
Root cause An issue with report data collection logic was causing inaccurate data in the Microsoft 365 admin center. Next steps: -We're continuing our analysis of the report data collection logic to help develop ways to prevent similar issues in the future. This is the final update for the event.; PublishedTime=2021-03-01T05:10:56.497Z

MO240984 Details

Status: We identified an issue that is causing some report data to not reflect information as intended. We've implemented a change to the report data collection logic and are performing a backfill to update reports with the accurate data sets. Next update by: Monday, March 1, 2021, at 7:00 AMUTC; PublishedTime=2021-02-22T05:41:13.277Z

Final status: We confirmed that backfill completed and that the reports have been updated. Next update by: Monday, March 1, 2021, at 7:00 AMUTC; PublishedTime=2021-02-22T05:41:13.277Z

--------------------------------------------------------------------------------------------------
Title Some users may be unable to load the Apps Health page for their Productivity Score
ID MO241008
Service Microsoft 365 suite
Feature Portal
Status Service restored
Impact Users may have been unable to load the Apps Health page for their Productivity Score.
Severity
Start time 2021-02-20T07:00:00Z
End time 2021-02-26T04:00:00Z
Last update 2021-02-26T05:59:26.397Z
More info
Root cause The upstream data source stopped sending information in the data stream from February 10, 2021 to February 16, 2021. Next steps: - We're investigating the upstream data source to better understand the issue and find ways to prevent this from happening in the future. This is the final update for the event.; PublishedTime=2021-02-26T05:59:26.397Z

MO241008 Details

Status: We've identified an issue with the Apps Health source, resulting in the specific Productivity Score page not loading. We're analyzing the upstream data source to develop a mitigation plan. Next update by: Wednesday, February 24, 2021, at 8:00 AM UTC; PublishedTime=2021-02-22T08:02:20.127Z

Status: We're continuing to analyze the upstream data source to determine the underlying cause of the issue and develop a mitigation plan. Next update by: Friday, February 26, 2021, at 8:00 AMUTC; PublishedTime=2021-02-24T06:24:53.61Z

Final status: We've determined that the upstream data source stopped sending information in the data stream from February 10, 2021 to February 16, 2021. We're retrieved the data and backfilled it to the service restoring the Apps Health page functionality. Next update by: Friday, February 26, 2021, at 8:00 AMUTC; PublishedTime=2021-02-24T06:24:53.61Z

--------------------------------------------------------------------------------------------------
Title Users may be unable to access mailboxes in Outlook on the web
ID EX240591
Service Exchange Online
Feature E-Mail and calendar access
Status Service restored
Impact Users may have been unable to access mailboxes in Outlook on the web.
Severity
Start time 2021-02-19T06:00:00Z
End time 2021-02-20T00:30:00Z
Last update 2021-02-20T01:22:31.873Z
More info Users may have seen an error stating "HTTP Error 500.37 - ANCM Failed to Start Within Startup Time Limit". The Outlook desktop client and Outlook Mobile were unaffected by this issue and users were able to access their mail using these connection methods. Final status: We've confirmed that a recent configuration change caused intermittent issues when accessing through Outlook on the web, which resulted in impact. We've restarted the underlying service components facilitating the service to accept the new configuration and confirmed that impact is mitigated. Scope of impact: Any user attempting to access their Exchange Online mailbox through Outlook on the web may have experienced impact. Start time: Friday, February 19, 2021, at 6:00 AM'UTC End time: Saturday, February 20, 2021, at 12:30 AM'UTC Root cause: A recent configuration change caused intermittent issues when accessing through Outlook on the web, which resulted in impact. Next steps: - We're reviewing the affecting change to identify how impact was missed during testing to prevent similar future impact. This is the final update for the event.; PublishedTime=2021-02-20T01:22:31.873Z
Root cause A recent configuration change caused intermittent issues when accessing through Outlook on the web, which resulted in impact. Next steps: - We're reviewing the affecting change to identify how impact was missed during testing to prevent similar future impact. This is the final update for the event.; PublishedTime=2021-02-20T01:22:31.873Z

EX240591 Details

Status: We're analyzing internal data to determine the source of the issue. Next update by: Friday, February 19, 2021, at 11:30 PMUTC; PublishedTime=2021-02-19T22:09:42.237Z

Status: We're reviewing system and server logs to identify the source of the issue and create a strategy to remediate impact. Next update by: Saturday, February 20, 2021, at 1:30 AMUTC; PublishedTime=2021-02-19T23:27:13.057Z

Final status: We've confirmed that a recent configuration change caused intermittent issues when accessing through Outlook on the web, which resulted in impact. We've restarted the underlying service components facilitating the service to accept the new configuration and confirmed that impact is mitigated. Next update by: Saturday, February 20, 2021, at 1:30 AMUTC; PublishedTime=2021-02-19T23:27:13.057Z

--------------------------------------------------------------------------------------------------
Title Some Admins with delegated permissions may be unable to view configuration details for devices via Microsoft Intune
ID IT241414
Service Microsoft Intune
Feature Microsoft Intune
Status Service restored
Impact Admins with delegated permissions may have been unable to view configuration details for devices via Microsoft Intune.
Severity
Start time 2021-02-17T18:00:00Z
End time 2021-02-25T02:50:00Z
Last update 2021-02-25T03:07:42.887Z
More info
Root cause A recent update, designed to add the settings catalogue feature to Microsoft Intune, unexpectedly caused impact to occur. Next steps: - We're reviewing our update procedures to better identify similar issues during our development and testing cycles. This is the final update for the event.; PublishedTime=2021-02-25T03:07:42.887Z

IT241414 Details

Status: We're reviewing system telemetry to identify the source of the issue. Next update by: Wednesday, February 24, 2021, at 1:30 PM UTC; PublishedTime=2021-02-24T11:28:24.657Z

Status: We've identified that a recent update, designed to add the settings catalogue feature to Microsoft Intune, unexpectedly caused impact to occur. We're developing a fix which will be tested before being deployed to the affected infrastructure. Next update by: Thursday, February 25, 2021, at 1:30 AM UTC; PublishedTime=2021-02-24T13:20:10.683Z

Status: We've validated the fix in our test environment and have begun deployment to the affected infrastructure. We expect the fix to be fully deployed by our next scheduled update. Next update by: Thursday, February 25, 2021, at 4:00 AMUTC; PublishedTime=2021-02-24T23:40:42.667Z

Status: Our fix deployment is taking longer than expected to complete. We're working to ensure it completes as fast as possible, which we estimate to be within the next few hours. Next update by: Thursday, February 25, 2021, at 5:00 AMUTC; PublishedTime=2021-02-25T02:09:58.337Z

Final status: We've confirmed that the deployment is now complete and our telemetry shows that the issue is mitigated. Next update by: Thursday, February 25, 2021, at 5:00 AMUTC; PublishedTime=2021-02-25T02:09:58.337Z

--------------------------------------------------------------------------------------------------
Title Some users may experience delays with chat messages or issues joining Teams live events
ID TM240228
Service Microsoft Teams
Feature Teams Components
Status Post-incident report published
Impact Users may have experienced delays in sending and receiving chat messages or been unable to join live events.
Severity
Start time 2021-02-17T12:40:00Z
End time 2021-02-18T00:45:00Z
Last update 2021-02-25T22:44:06.77Z
More info This is an incremental update to provide the latest status on the issue. Current status: We're continuing to monitor the service re-routing process and will provide a more accurate ETA as soon as it becomes available. While the majority of impact has been mitigated, a subset of users may continue to experience residual impact until the service re-routing is finished. For complete details on this event, please reference the post from Wednesday, February 17, 2021, at 11:00 PM UTC. Next update by: Thursday, February 18, 2021, at 1:00 AM UTC; PublishedTime=2021-02-17T23:59:48.84Z
Root cause A higher than expected surge in connection attempts to a small section of our North Central infrastructure caused throughput and latency issues, resulting in some users receiving 4xx and 5xx errors when attempting to use some Microsoft Teams features. Next steps: - For a more comprehensive list of next steps and actions, please refer to the Post Incident Review document. We'll publish a post-incident report within five business days.; PublishedTime=2021-02-18T01:22:24.863Z

TM240228 Details

Status: We're reviewing service telemetry to determine the next troubleshooting steps. Next update by: Wednesday, February 17, 2021, at 4:30 PM UTC; PublishedTime=2021-02-17T14:33:53.273Z

Status: We've identified errors for some requests attempting to execute database queries and we're investigating these further to isolate the source of the issue. Next update by: Wednesday, February 17, 2021, at 5:00 PM UTC; PublishedTime=2021-02-17T15:51:41.86Z

Status: We've identified a potential networking issue that may be causing the problem. We're working to confirm our findings while simultaneously exploring the optimal solution to remediate impact. Next update by: Wednesday, February 17, 2021, at 6:00 PM UTC; PublishedTime=2021-02-17T16:57:44.19Z

Status: We're rerouting some services to alternate infrastructure in order to alleviate some strain within the affected infrastructure. In parallel, we're continuing to evaluate long-term solutions to restore service. Next update by: Wednesday, February 17, 2021, at 7:00 PM UTC; PublishedTime=2021-02-17T17:51:18.993Z

Status: We're analyzing network trace logs to better understand the underlying source of the issue and to determine our next mitigation steps. Next update by: Wednesday, February 17, 2021, at 8:00 PM UTC; PublishedTime=2021-02-17T18:55:59.983Z

Status: We're continuing to route some services to alternate infrastructure in order to provide some relief while we continue to develop a plan for remediation. Next update by: Wednesday, February 17, 2021, at 9:00 PM UTC; PublishedTime=2021-02-17T20:00:33.7Z

Status: We've completed our work to re-route some services to alternate infrastructure and have confirmed that the impact to chat notification and message delivery has been mitigated. We're still working to develop a solution to mitigate impact related to creating chats and the ability to join Team live events. Next update by: Wednesday, February 17, 2021, at 10:00 PM UTC; PublishedTime=2021-02-17T21:03:11.627Z

Status: We've routed the services responsible for chat creation and Teams live events to alternate infrastructure and anticipate that this impact will be mitigated within one hour. Next update by: Wednesday, February 17, 2021, at 11:00 PM UTC; PublishedTime=2021-02-17T22:04:22.04Z

Status: The change to route services to alternate infrastructure is taking longer than anticipated, and may cause some minor residual impact before it reaches completion. We're continuing to monitor the service re-routing process and expect it to be completed in the next hour. Next update by: Thursday, February 18, 2021, at 12:00 AM UTC; PublishedTime=2021-02-17T22:57:22.437Z

Status: We're continuing to monitor the service re-routing process and will provide a more accurate ETA as soon as it becomes available. While the majority of impact has been mitigated, a subset of users may continue to experience residual impact until the service re-routing is finished. For complete details on this event, please reference the post from Wednesday, February 17, 2021, at 11:00 PM UTC. Next update by: Thursday, February 18, 2021, at 1:00 AM UTC; PublishedTime=2021-02-17T23:59:48.84Next update by: Thursday, February 18, 2021, at 1:00 AM UTC; PublishedTime=2021-02-17T23:59:48.84Z

Final status: We've completed our work to re-route the impacted network traffic to alternate infrastructure and confirmed via extensive monitoring that the issue has been resolved. Next update by: Thursday, February 18, 2021, at 1:00 AM UTC; PublishedTime=2021-02-17T23:59:48.84Z

--------------------------------------------------------------------------------------------------
Title A small number of users in Australia couldn't sign in to some Microsoft 365 services
ID MO240222
Service Microsoft 365 suite
Feature Portal
Status Service restored
Impact Users may have been unable to sign in to some Microsoft 365 services.
Severity
Start time 2021-02-17T03:00:00Z
End time 2021-02-17T10:43:00Z
Last update 2021-02-17T14:04:15.59Z
More info Impacted users may have seen the following error message: "AADSTS50000: There was an error issuing a token or an issue with our sign-in service." Subsequent retries may have been successful. Final status: We identified that a recent update, designed to improve the routing of internal request traffic, contained a configuration issue. We've reverted the configuration change, and confirmed that the issue was resolved after monitoring the affected environment. Scope of impact: Impact was limited to a small sections of users located in Australia. Start time: Wednesday, February 17, 2021, at 3:00 AM UTC End time: Wednesday, February 17, 2021, at 10:43 AM UTC Root cause: A recent update designed to improve the routing of internal request traffic contained a configuration issue, resulting in impact. Next steps: - We're reviewing our update procedures to better identify similar issues during our development and testing cycles. This is the final update for the event.; PublishedTime=2021-02-17T12:45:10.383Z
Root cause A recent update designed to improve the routing of internal request traffic contained a configuration issue, resulting in impact. Next steps: - We're reviewing our update procedures to better identify similar issues during our development and testing cycles. This is the final update for the event.; PublishedTime=2021-02-17T12:45:10.383Z

MO240222 Details

Final status: We identified that a recent update, designed to improve the routing of internal request traffic, contained a configuration issue. We've reverted the configuration change, and confirmed that the issue was resolved after monitoring the affected environment.

--------------------------------------------------------------------------------------------------
Title Users may be intermittently unable to view their own camera feed during Microsoft Teams meetings
ID TM241746
Service Microsoft Teams
Feature Teams Components
Status Service restored
Impact Users may have been intermittently unable to view their own camera feed during Microsoft Teams meetings.
Severity Sev2
Start time 2021-02-16T22:12:00Z
End time 2021-05-04T19:23:00Z
Last update 2021-05-05T19:17:23.95Z
More info Other users in the meeting were able to see the impacted user's camera feed. This issue was specific to the standard gallery mode. Affected users could use either the large gallery mode or together mode to prevent this issue. Final status: We've determined that the fix has reached 100 percent saturation and confirmed that the issue is fully resolved. Scope of impact: Your organization was affected by this event, and this problem may have intermittently impacted any user using standard gallery mode in Microsoft Teams meetings. Start time: Tuesday, February 16, 2021, at 10:12 PM'UTC End time: Tuesday, May 4, 2021, at 7:00 PM'UTC Root cause: A code issue was causing a problem with user camera feeds during meetings when using the standard gallery mode. Next steps: - We're reviewing this portion of impacted code to understand why it was causing impact to prevent similar impact in the future. This is the final update for the event.; PublishedTime=2021-05-05T18:47:20.64Z
Root cause A code issue was causing a problem with user camera feeds during meetings when using the standard gallery mode. Next steps: - We're reviewing this portion of impacted code to understand why it was causing impact to prevent similar impact in the future. This is the final update for the event.; PublishedTime=2021-05-05T18:47:20.64Z

TM241746 Details

Status: We've received reports of an issue in which users may be intermittently unable to view their own camera feed during Microsoft Teams meetings identified. We've identified that a code issue is causing this impact and we're deploying a code fix for this problem. We'll provide a more detailed timeline regarding the deployment completion by Friday, March 5, 2021, at 5:00 PM UTC. Next update by: Friday, March 5, 2021, at 5:00 PMUTC; PublishedTime=2021-02-26T00:04:24.793Z

Status: We're validating the fix for this issue in our internal testing environments before beginning deployment to the affected environments. Next update by: Friday, March 12, 2021, at 5:00 PM UTC; PublishedTime=2021-03-05T15:27:15.09Z

Status: We've confirmed that the fix is deploying across our testing environments as expected, and we're continuing to monitor it's progress, before deployment to the impacted environments. Next update by: Friday, March 19, 2021, at 5:00 PM UTC; PublishedTime=2021-03-12T15:31:28.88Z

Status: The fix continues to deploy across our testing environments as expected, and we're continuing to monitor it's progress, before deployment to the impacted environments. Next update by: Friday, March 26, 2021, at 5:00 PM UTC; PublishedTime=2021-03-19T14:46:40.8Z

Status: As deployment across our testing environments continues without issue, we continue to monitor the overall progress and anticipate the fix will reach live environments by April 19th, 2021. Next update by: Friday, April 2, 2021, at 5:00 PM UTC; PublishedTime=2021-03-26T12:43:15.893Z

Status: We'll continue to monitor the progress of the fix through our testing environments over the weekend, and expect that we'll know if the solution is ready to be deployed to the affected environment early next week. We still expect for the the deployment to complete by Monday, April 19, 2021. Next update by: Tuesday, April 6, 2021, at 10:00 AMUTC; PublishedTime=2021-04-02T17:00:30.097Z

Status: The deployment of the fix through our testing environments is taking longer than expected. We'll continue to monitor the status of the deployment until completion, where once we've validated the fix works as expected, we'll start to deploy it to the affected environment. Next update by: Tuesday, April 13, 2021, at 10:00 AM UTC; PublishedTime=2021-04-06T09:43:00.607Z

Status: We've identified during testing that the fix has not had the expected result. We're reviewing system logs to identify why our fix was unsuccessful, and to help develop a new mitigation plan. Next update by: Friday, April 16, 2021, at 10:00 AM UTC; PublishedTime=2021-04-13T09:30:05.88Z

Status: We're continuing to review system logs to identify the underlying root cause. Next update by: Friday, April 23, 2021, at 10:00 AM UTC; PublishedTime=2021-04-16T09:56:11.69Z

Status: We've identified a code issue caused video streaming to fail unexpectedly, resulting in impact and have developed a fix to resolve this, which is currently being deployed. As the fix is deployed, users may begin to see relief. Additionally, we're monitoring the overall progress of the deployment, to ensure that the fix fully saturates across the affected environment. Next update by: Friday, April 30, 2021, at 12:00 PM UTC; PublishedTime=2021-04-23T08:55:02.38Z

Status: We're continuing to monitor the progress of the deployment as it progresses across impacted environments. Next update by: Friday, April 30, 2021, at 7:00 PM UTC ; PublishedTime=2021-04-30T11:14:08.57Z

Status: The fix has reached approximately 40 percent saturation throughout the affected infrastructure and we expect it to complete by the next scheduled update. Next update by: Wednesday, May 5, 2021, at 7:00 PMUTC; PublishedTime=2021-04-30T18:15:37.677Z

Final status: We've determined that the fix has reached 100 percent saturation and confirmed that the issue is fully resolved. Next update by: Wednesday, May 5, 2021, at 7:00 PMUTC; PublishedTime=2021-04-30T18:15:37.677Z

--------------------------------------------------------------------------------------------------
Title Admins may see delays in reporting data in the Security and Compliance Center (SCC)
ID EX240291
Service Exchange Online
Feature E-Mail timely delivery
Status Service restored
Impact Admins may have seen delays in reporting data in the SCC and on-demand and scheduled reports.
Severity
Start time 2021-02-16T18:00:00Z
End time 2021-02-19T18:15:00Z
Last update 2021-02-19T18:45:18.737Z
More info This issue affected all reports, including but not limited to Security, Threat, Compliance, and Campaign reports. Final status: We've confirmed that the backlog processing has completed. We've verified through service health monitoring that balancing requests across additional infrastructure and processing backlogged data has successfully resolved the issue. Scope of impact: Any admin attempting to see reporting data from the SCC or on-demand and scheduled reports may have experienced impact. Start time: Tuesday, February 16, 2021, at 6:00 PM'UTC End time: Friday, February 19, 2021, at 6:15 PM'UTC Root cause: Infrastructure responsible for processing data for the impacted reports wasn't handling the requests as expected, resulting in impact. Next steps: - We're investigating why the infrastructure responsible for processing data for the impacted reports wasn't handling the requests as expected to prevent this problem from happening again in the future. This is the final update for the event.; PublishedTime=2021-02-19T18:45:18.737Z
Root cause Infrastructure responsible for processing data for the impacted reports wasn't handling the requests as expected, resulting in impact. Next steps: - We're investigating why the infrastructure responsible for processing data for the impacted reports wasn't handling the requests as expected to prevent this problem from happening again in the future. This is the final update for the event.; PublishedTime=2021-02-19T18:45:18.737Z

EX240291 Details

Status: We've determined that there is an issue in our reporting aggregation pipeline causing delays in the generation of reporting data in the SCC as well as with on-demand and scheduled reports. We're investigating available data to determine the root cause and create a strategy that remediates impact. Next update by: Wednesday, February 17, 2021, at 9:30 PMUTC; PublishedTime=2021-02-17T20:44:01.987Z

Status: We've begun adding additional report processing infrastructure to expedite handling of the backlog in our reporting aggregation pipeline to remediate impact. Next update by: Wednesday, February 17, 2021, at 11:30 PMUTC; PublishedTime=2021-02-17T21:30:03.82Z

Status: While the additional report processing infrastructure did expedite the handling of the backlog, impact was not remediated. We're investigating available telemetry data to determine the root cause and create a strategy that remediates impact. Next update by: Thursday, February 18, 2021, at 8:00 PMUTC; PublishedTime=2021-02-17T23:27:23.077Z

Status: We've determined that infrastructure responsible for processing data for the impacted reports isn't handling the requests as expected, resulting in impact. We've verified that continuing to balance the requests across additional report processing infrastructure is the best method to expedite the handling of the backlog. We'll monitor the backlog processing until it's complete while also investigating additional optimizations for timely processing of affected reports. Next update by: Friday, February 19, 2021, at 8:00 PMUTC; PublishedTime=2021-02-18T19:41:19.723Z

Final status: We've confirmed that the backlog processing has completed. We've verified through service health monitoring that balancing requests across additional infrastructure and processing backlogged data has successfully resolved the issue. Next update by: Friday, February 19, 2021, at 8:00 PMUTC; PublishedTime=2021-02-18T19:41:19.723Z

--------------------------------------------------------------------------------------------------
Title Admins can't access Mobile Device Management (MDM) for Microsoft 365 or Microsoft Intune for Office (MIFO) devices pages
ID MD241715
Service Mobile Device Management for Office 365
Feature Mobile Device Management
Status Service restored
Impact Admins were intermittently unable to directly access the MDM or MIFO devices pages within the admin center.
Severity
Start time 2021-02-15T14:42:00Z
End time 2021-02-27T00:40:00Z
Last update 2021-02-27T00:50:06.933Z
More info While we worked to mitigate impact, affected admins were able to load a different page in the admin center first and then change the URL to point to "https://portal.office.com/adminportal/home#/MifoDevices", or admins could have clicked on the Device Management link in the admin center, then navigate to the home icon on the left menu and use the browser back button to go back to the Device Management page to get it to load correctly. Final status: We've confirmed that the deployment of the fix has completed and our internal testing shows that the issue has been mitigated. Scope of impact: Any admin attempting to directly access the MDM or MIFO devices pages within the admin center may have been impacted. Start time: Monday, February 15, 2021, at 2:42 PM UTC End time: Saturday, February 27, 2021, at 12:40 AM UTC Root cause: A recent update to improve the page rendering performance of the admin center resulted in impact. Next steps: - We're reviewing our update procedures to better identify similar issues during our development and testing cycles. This is the final update for the event.; PublishedTime=2021-02-27T00:50:06.933Z
Root cause A recent update to improve the page rendering performance of the admin center resulted in impact. Next steps: - We're reviewing our update procedures to better identify similar issues during our development and testing cycles. This is the final update for the event.; PublishedTime=2021-02-27T00:50:06.933Z

MD241715 Details

Status: We're investigating a potential issue and checking for impact to your organization. We'll provide an update within 30 minutes.; PublishedTime=2021-02-25T22:09:23.8

Status: We're reproducing the issue in our internal testing environment to gather additional diagnostic data to help us isolate the source of the issue. Next update by: Friday, February 26, 2021, at 12:00 AMUTC; PublishedTime=2021-02-25T22:34:15.413Z

Status: We're analyzing the diagnostic data gathered during our internal reproduction of the issue, including server and client logs, as we work to identify any recent service changes which may be causing impact. Next update by: Friday, February 26, 2021, at 2:00 AMUTC; PublishedTime=2021-02-25T23:22:57.53Z

Status: We've determined that a recent update to improve the page rendering performance of the admin center has resulted in impact. We're developing a fix to restore page functionality and we expect the development, testing and deployment to be complete by our next scheduled update. Next update by: Friday, February 26, 2021, at 10:00 PMUTC; PublishedTime=2021-02-26T01:01:01.127Z

Status: We've developed the fix and have started to deploy it to the affected environment. We suspect it will complete by approximately 11:00 PM UTC on Friday, February 26, 2021, and we're monitoring the deployment to ensure it completes as expected. Next update by: Saturday, February 27, 2021, at 1:00 AM UTC; PublishedTime=2021-02-26T20:32:57.047Z

Final status: We've confirmed that the deployment of the fix has completed and our internal testing shows that the issue has been mitigated. Next update by: Saturday, February 27, 2021, at 1:00 AM UTC; PublishedTime=2021-02-26T20:32:57.047Z

--------------------------------------------------------------------------------------------------
Title Some users may have experienced impact to certain Microsoft Teams features
ID TM239744
Service Microsoft Teams
Feature Teams Components
Status Service restored
Impact Users may have experienced impact to some Microsoft Teams features.
Severity
Start time 2021-02-15T12:25:00Z
End time 2021-02-15T13:22:18.163Z
Last update 2021-02-15T13:53:21.73Z
More info Affected scenarios included: -Users were unable to see the 'blocked' icon for users they blocked. -Users may have been unable to see an updated priority contact list. Final status: While we were in the process of reviewing service monitoring telemetry to isolate the source of the issue, we identified that the service had returned to normal health. We've confirmed that the issue is no longer occurring, and we'll continue to monitor the service to ensure that the problem does not happen again. Scope of impact: Your organization was affected by this event, and may have impacted any user. Start time: Monday, February 15, 2021, at 12:25 PM UTC End time: Monday, February 15, 2021, at 12:55 PM UTC This is the final update for the event.; PublishedTime=2021-02-15T13:22:18.163Z
Root cause

TM239744 Details

Final status: While we were in the process of reviewing service monitoring telemetry to isolate the source of the issue, we identified that the service had returned to normal health. We've confirmed that the issue is no longer occurring, and we'll continue to monitor the service to ensure that the problem does not happen again.

--------------------------------------------------------------------------------------------------
Title Some users may have been experiencing issues within Microsoft Teams chats
ID TM239745
Service Microsoft Teams
Feature Teams Components
Status Service restored
Impact Users may have been experiencing issues within Microsoft Teams chats.
Severity
Start time 2021-02-15T12:20:00Z
End time 2021-02-15T13:00:00Z
Last update 2021-02-15T14:43:57.743Z
More info More info: Users may have been unable to: -Create group or private chats. -Remove or add users to chats. -Start calls from existing chats. -Search for external users in the search bar. Final status: We've identified that a recent deployment resulted in an unexpected change, causing impact. We've reverted the deployment and, after monitoring the environment, confirmed that this resolved the issue. Scope of impact: This issue may have impacted any user. Start time: Monday, February 15, 2021, at 12:20 PM UTC End time: Monday, February 15, 2021, at 1:00 PM UTC Root cause: A recent deployment resulted in an unexpected change, causing impact. Next steps: - We're reviewing our deployment and provisioning procedures to help prevent similar problems in the future. This is the final update for the event.; PublishedTime=2021-02-15T14:10:43.727Z
Root cause A recent deployment resulted in an unexpected change, causing impact. Next steps: - We're reviewing our deployment and provisioning procedures to help prevent similar problems in the future. This is the final update for the event.; PublishedTime=2021-02-15T14:10:43.727Z

TM239745 Details

Status: We're analyzing diagnostic data to determine the source of the issue. Next update by: Monday, February 15, 2021, at 3:30 PM UTC; PublishedTime=2021-02-15T13:34:41.8Z

Final status: We've identified that a recent deployment resulted in an unexpected change, causing impact. We've reverted the deployment and, after monitoring the environment, confirmed that this resolved the issue. Next update by: Monday, February 15, 2021, at 3:30 PM UTC; PublishedTime=2021-02-15T13:34:41.8Z

--------------------------------------------------------------------------------------------------
Title Users attending a meeting may experience latency or may be dropped from the meeting without manual action
ID TM239516
Service Microsoft Teams
Feature Teams Components
Status Service restored
Impact Users who were attending a meeting may have experienced latency or may have suddenly dropped from the meeting.
Severity
Start time 2021-02-14T05:00:00Z
End time 2021-02-14T06:52:18.08Z
Last update 2021-02-14T07:05:42.607Z
More info Some meetings may have inadvertently removed all users from the voice call, causing the meeting to end. Final status: Our automated service health monitoring systems alerted us to an issue with service issue where users may have experienced latency during meetings, were dropped from meetings, or had their meetings remove all users. While we were in the process of analyzing system telemetry to formulate a remediation plan, system monitoring indicated that the service had returned to normal health. We've confirmed that the problem is no longer occurring, and we'll continue to monitor the service to ensure that the problem does not happen again. Scope of impact: Users who were in a meeting may have experienced may have suddenly dropped from the meeting or experienced latency during meetings. Start time: Sunday, February 14, 2021, at 5:00 AM'UTC End time: Sunday, February 14, 2021, at 5:45 AM'UTC This is the final update for the event.; PublishedTime=2021-02-14T06:52:18.08Z
Root cause

TM239516 Details

Final status: Our automated service health monitoring systems alerted us to an issue with service issue where users may have experienced latency during meetings, were dropped from meetings, or had their meetings remove all users. While we were in the process of analyzing system telemetry to formulate a remediation plan, system monitoring indicated that the service had returned to normal health. We've confirmed that the problem is no longer occurring, and we'll continue to monitor the service to ensure that the problem does not happen again.

--------------------------------------------------------------------------------------------------
Title Offline mode data synchronization failures
ID CR239129
Service Dynamics 365 Apps
Feature Components/Features
Status Service restored
Impact Users may have been unable to synchronize data in offline mode.
Severity
Start time 2021-02-12T00:34:00Z
End time 2021-02-12T05:53:00Z
Last update 2021-02-12T06:04:42.537Z
More info
Root cause

CR239129 Details

--------------------------------------------------------------------------------------------------
Title Users may experience sync issues with new pages or sections when using OneNote on specific connection methods
ID MO239124
Service Microsoft 365 suite
Feature Portal
Status Service restored
Impact Users may have experienced sync issues with new pages or sections when using OneNote on specific connection methods.
Severity
Start time 2021-02-12T00:20:00Z
End time 2021-02-12T04:30:00Z
Last update 2021-02-12T05:05:04.417Z
More info Impact was specific to the 64 bit version of the desktop client, Android, iOS, MacOS, and Universal Windows Platform (UWP) apps. Users who have access to the OneNote web app or the 32 bit version of the desktop client may have leveraged these connection methods as alternative access routes. Final status: We've confirmed that the section of Azure infrastructure is now performing as expected, and we've confirmed via system monitoring that impact has been remediated. Scope of impact: This issue only affected users who were routed through the West US infrastructure. Start time: Friday, February 12, 2021, at 12:20 AM'UTC End time: Friday, February 12, 2021, at 4:30 AM'UTC Root cause: A section of Azure infrastructure was performing below expected thresholds, resulting in impact. Next steps: - We're reviewing how our service interacts with the impacted Azure infrastructure to better understand how impact began and to explore potential options to prevent similar impact in the future. This is the final update for the event.; PublishedTime=2021-02-12T05:04:55.187Z
Root cause A section of Azure infrastructure was performing below expected thresholds, resulting in impact. Next steps: - We're reviewing how our service interacts with the impacted Azure infrastructure to better understand how impact began and to explore potential options to prevent similar impact in the future. This is the final update for the event.; PublishedTime=2021-02-12T05:04:55.187Z

MO239124 Details

Status: Our automated service health monitoring alerted us to availability issues occurring within a subsection of infrastructure responsible for new page and section syncs. Our investigation indicates that a section of Azure infrastructure is performing below expected thresholds, resulting in impact. We're working to resolve the Azure infrastructure problem to remediate impact. Next update by: Friday, February 12, 2021, at 5:00 AMUTC; PublishedTime=2021-02-12T03:38:20.163Z

Status: We're continuing our work to resolve the Azure infrastructure problem to resolve this issue. Next update by: Friday, February 12, 2021, at 7:00 AMUTC; PublishedTime=2021-02-12T04:48:39.44Z

Final status: We've confirmed that the section of Azure infrastructure is now performing as expected, and we've confirmed via system monitoring that impact has been remediated. Next update by: Friday, February 12, 2021, at 7:00 AMUTC; PublishedTime=2021-02-12T04:48:39.44Z

--------------------------------------------------------------------------------------------------
Title Power Platform capacity resource failures
ID CR239114
Service Dynamics 365 Apps
Feature Other
Status Service restored
Impact Admins may have been unable to make changes to capacity for their environments in the Power Platform admin center.
Severity
Start time 2021-02-12T00:16:00Z
End time 2021-02-12T02:00:00Z
Last update 2021-02-12T03:10:38.747Z
More info
Root cause

CR239114 Details

--------------------------------------------------------------------------------------------------
Title Users may experience errors in Microsoft Teams when attempting to use any feature
ID TM239098
Service Microsoft Teams
Feature Teams Components
Status Post-incident report published
Impact Users may have been unable to access or utilize various Microsoft Teams features.
Severity
Start time 2021-02-12T00:13:00Z
End time 2021-02-12T04:20:00Z
Last update 2021-02-19T11:05:30.703Z
More info Users may have experienced one or more of the following scenarios: ' Users launching Microsoft Teams for the first time on a browser may have seen an 'oops' page ' Users may have failed to load channels/chat ' Users may have seen inaccurate pill counts on their iOS and Android devices ' Users may have been unable to see their pinned channels ' Users may have been unable to create @ mention tags ' Users may have been unable to search for users ' Users may have been unable to send/receive messages ' Users may have been unable to join, schedule, or own meetings ' Users may have seen calendaring issues ' Users may have been unable to host live events ' Voice applications, like call queues and auto attendants may have been affected Final status: Our mitigation efforts have been successful and we've confirmed recovery has remained stable after extensive monitoring. Scope of impact: Users attempting to access or utilize Microsoft Teams features may have been impacted if their tenant was hosted in the North American region or if their network traffic flowed through North America. Start time: Friday, February 12, 2021, at 12:13 AM UTC End time: Friday, February 12, 2021, at 4:20 AM UTC Root cause: On February 11, 10:04 AM UTC (approximately thirteen hours before the incident impact), a Cosmos DB deployment was completed in a specific portion of the North American regional infrastructure using safe deployment practices; unfortunately, it introduced a code regression that triggered several hours later, resulting in impact. A failure condition in the configuration store occurred within a small section of regional segmented components. The front-end service, which is responsible for request routing of user traffic, should handle this; however, due to the code regression, the front-end service failed to address the condition and crashed. Additionally, front-end services for other segmented components within the region also made requests to the impacted front-end services to obtain configurations. These requests would have timed out because of unavailability, triggering the same unhandled failure condition and resulting in a crash. This cascading effect impacted most Cosmos DB front-end services within a specific portion of North American regional infrastructure. Next steps: - For a more comprehensive list of next steps and actions, please refer to the Post Incident Review document. We'll publish a post-incident report within five business days.; PublishedTime=2021-02-12T06:00:34.943Z
Root cause On February 11, 10:04 AM UTC (approximately thirteen hours before the incident impact), a Cosmos DB deployment was completed in a specific portion of the North American regional infrastructure using safe deployment practices; unfortunately, it introduced a code regression that triggered several hours later, resulting in impact. A failure condition in the configuration store occurred within a small section of regional segmented components. The front-end service, which is responsible for request routing of user traffic, should handle this; however, due to the code regression, the front-end service failed to address the condition and crashed. Additionally, front-end services for other segmented components within the region also made requests to the impacted front-end services to obtain configurations. These requests would have timed out because of unavailability, triggering the same unhandled failure condition and resulting in a crash. This cascading effect impacted most Cosmos DB front-end services within a specific portion of North American regional infrastructure. Next steps: - For a more comprehensive list of next steps and actions, please refer to the Post Incident Review document. We'll publish a post-incident report within five business days.; PublishedTime=2021-02-12T06:00:34.943Z

TM239098 Details

Status: We're investigating a potential issue and checking for impact to your organization. We'll provide an update within 30 minutes.; PublishedTime=2021-02-12T01:17:52.44

Status: We're analyzing available data to help identify cause of impact and determine troubleshooting method. Next update by: Friday, February 12, 2021, at 2:30 AMUTC; PublishedTime=2021-02-12T01:48:58.777Z

Status: We're continuing to narrow down the cause of impact and suspect this may be related to a portion of our networking infrastructure. We've determined this issue primarily affects users if they're are hosted or have traffic routed through North America. Next update by: Friday, February 12, 2021, at 4:00 AM UTC; PublishedTime=2021-02-12T02:19:36.587Z

Status: We've confirmed the issue is caused by a portion of our networking infrastructure performing below expected thresholds. We're seeing recovery for a majority of scenarios; however, we're continuing to monitor the service health and exploring mitigation options to ensure complete recovery. Next update by: Friday, February 12, 2021, at 6:00 AM UTC; PublishedTime=2021-02-12T04:30:16.98Z

Final status: Our mitigation efforts have been successful and we've confirmed recovery has remained stable after extensive monitoring. Next update by: Friday, February 12, 2021, at 6:00 AM UTC; PublishedTime=2021-02-12T04:30:16.98Z

--------------------------------------------------------------------------------------------------
Title Users are unable to search content or view files in the Files tab in Microsoft Teams
ID TM242990
Service Microsoft Teams
Feature Teams Components
Status False positive
Impact Users are unable to search content or view files in the Files tab in Microsoft Teams.
Severity Sev2
Start time 2021-02-10T11:02:00Z
End time
Last update 2021-05-12T20:10:58.533Z
More info
Root cause An API facilitating Microsoft Teams search began performing below expected thresholds due to performance issues on components that facilitate the affected API, resulting in impact. Next update by: Wednesday, May 19, 2021, at 9:00 PMUTC; PublishedTime=2021-05-12T20:10:58.533Z

TM242990 Details

Status: We've identified that an API facilitating Microsoft Teams search began performing below expected thresholds due to performance issues on components that facilitate the affected API, resulting in impact. We're working to improve performance of the API and affected components to remediate impact. We suspect that the issue may take several weeks to resolve in full, and we'll work to provide a more precise estimated time to remediation by our next scheduled update. Next update by: Thursday, March 11, 2021, at 9:00 PMUTC; PublishedTime=2021-03-04T20:07:23.113Z

Status: We're continuing to deploy additional processing infrastructure and optimizing traffic flow in the environment to improve API and affected component performance. We anticipate completion of these efforts by March 25, 2021. In parallel, we're investigating further actions that could expedite this process. Next update by: Thursday, March 18, 2021, at 9:00 PMUTC; PublishedTime=2021-03-11T19:38:49.867Z

Status: The processing infrastructure deployment and optimization actions didn't have the desired effect. While we're continuing to work on identifying other methods to remediate impact, we're in the process of procuring additional infrastructure to allow us to further scale up available processing capabilities to remediate the impact. Next update by: Thursday, March 25, 2021, at 9:00 PMUTC; PublishedTime=2021-03-18T20:42:08.767Z

Status: We understand the importance of being able to search for content and view files in Microsoft Teams. While this is taking longer than desired to fully remediate, we're actively working to resolve the issue as quickly as possible. We've procured and are configuring additional infrastructure to allow us to further scale up available processing capabilities and remediate impact. Thank you for your continued patience as our remediation process continues. Next update by: Thursday, April 1, 2021, at 9:00 PMUTC; PublishedTime=2021-03-25T19:33:15.367Z

Status: We're continuing our efforts to configure additional infrastructure that will allow us to further scale up available processing capabilities and remediate impact. At this time, we anticipate these efforts will be completed and impact remediated during the week of Monday, April 12, 2021. This estimate is subject to change and we'll provide a more concrete timeline to resolution when available. Next update by: Wednesday, April 7, 2021, at 9:00 PMUTC; PublishedTime=2021-04-01T19:17:34.273Z

Status: Our configuration of additional infrastructure is proceeding as expected as we're currently at the provisioning stage of remediation. We still anticipate mitigation during the week of April 12, 2021, and possibly by Wednesday April 14 specifically. We'll confirm the latest estimate for resolution with our next update. Next update by: Wednesday, April 14, 2021, at 9:00 PMUTC; PublishedTime=2021-04-07T19:23:36.517Z

Status: Our configuration of additional infrastructure is progressing slower than we originally expected; however, users may see relief as it progresses. We now anticipate it may take until the week of April 19th, 2021, for the issue to be fully resolved. Next update by: Wednesday, April 21, 2021, at 9:00 PMUTC; PublishedTime=2021-04-14T20:05:34.907Z

Status: We're in the final stages of configuring the additional infrastructure we procured to scale up available processing capabilities and remediate impact. Our efforts are completing without issue; however, due to the complexity of the work this process may take longer than last communicated. As we work to complete our efforts and remediate impact, we're also aiming to provide a more accurate estimate for full remediation by our next scheduled update. Next update by: Wednesday, April 28, 2021, at 9:00 PMUTC; PublishedTime=2021-04-21T20:10:54.823Z

Status: We've confirmed that our efforts to complete the final stages for configuring the additional infrastructure we procured and scale up available processing capabilities is successfully in progress. We're aiming for this to complete and the issue be resolved by Wednesday, May 5, 2021, but given the complexity of the work this is subject to change and we will provide an updated timeline for remediation when available. Next update by: Wednesday, May 5, 2021, at 9:00 PMUTC; PublishedTime=2021-04-28T20:39:10.547Z

Status: The process of configuring the additional infrastructure we procured and scaling up available processing capabilities is still in progress and taking more time than anticipated. We now expect for the process to complete to resolve this issue by Wednesday, May 12, 2021. As previously mentioned, given the complexity of the work this is subject to change and we will provide an updated timeline for remediation when available. Next update by: Wednesday, May 12, 2021, at 9:00 PMUTC; PublishedTime=2021-05-05T19:46:35.953Z

Status: Our estimated timelines to resolution have evolved as the work to finalize the procurement of additional infrastructure and prepare it to take active traffic progresses. The work is complex and takes time to ensure the infrastructure will be successful in handling in traffic and cause no adverse impact. Our work continues, but we are now targeting a completion date of Wednesday, May 19, 2021. Next update by: Wednesday, May 19, 2021, at 9:00 PMUTC; PublishedTime=2021-05-12T20:10:58.533Z

--------------------------------------------------------------------------------------------------
Title Users experiencing delays receiving email messages from external senders via all Exchange Online connection methods
ID EX238762
Service Exchange Online
Feature E-Mail timely delivery
Status Post-incident report published
Impact Users may have experienced delays receiving email messages from external senders via all connection methods.
Severity
Start time 2021-02-09T20:04:00Z
End time 2021-02-09T20:31:00Z
Last update 2021-03-05T19:46:44.63Z
More info Our telemetry suggests that inbound email delivery was delayed by a few minutes. Users may have noticed the problem from all connection methods including the Outlook client, Outlook on the web, and Exchange ActiveSync (EAS). The problem impacted inbound email from external senders. Mailflow for internal email messages wasn't affected. Final status: While we were in the process of analyzing system logs to isolate the source of the problem, system monitoring indicated that the service had returned to normal health. We've confirmed that the problem is no longer occurring, and we'll continue to monitor the service to ensure that the problem does not happen again. Scope of impact: Most users may have experienced delays receiving email messages. Start time: Tuesday, February 9, 2021, at 8:04 PM'UTC End time: Tuesday, February 9, 2021, at 8:31 PM'UTC This is the final update for the event.; PublishedTime=2021-02-09T21:31:57.343Z
Root cause

EX238762 Details

Status: We're investigating a potential issue and checking for impact to your organization. We'll provide an update within 30 minutes.; PublishedTime=2021-02-09T20:55:28.393

Final status: While we were in the process of analyzing system logs to isolate the source of the problem, system monitoring indicated that the service had returned to normal health. We've confirmed that the problem is no longer occurring, and we'll continue to monitor the service to ensure that the problem does not happen again.

--------------------------------------------------------------------------------------------------
Title Users can't create Skype Meeting Broadcast events and watch broadcasts scheduled for the year 2021
ID LY238775
Service Skype for Business
Feature Audio and Video
Status Service restored
Impact Users may have been unable to create Skype Meeting Broadcast events and watch broadcasts scheduled for the year 2021.
Severity
Start time 2021-02-09T02:49:00Z
End time 2021-02-10T02:53:00Z
Last update 2021-02-10T03:26:55.477Z
More info
Root cause A recent deployment inadvertently introduced an authentication misconfiguration to the affected environment, resulting in impact. Next steps: - We're reviewing the recent deployment to better understand how impact occurred and to explore potential actions to prevent similar impact from recurring. This is the final update for the event.; PublishedTime=2021-02-10T03:26:44.263Z

LY238775 Details

Status: We're investigating a potential issue and checking for impact to your organization. We'll provide an update within 30 minutes.; PublishedTime=2021-02-09T22:27:23.027

Status: We determined that a recent deployment inadvertently introduced an authentication misconfiguration to the affected environment, resulting in impact. We're in the process of deploying a fix to the affected environment to resolve the issue. Next update by: Wednesday, February 10, 2021, at 12:30 AMUTC; PublishedTime=2021-02-09T22:45:14.767Z

Status: We've encountered internal issues that are preventing our fix from deploying as expected. We're working to resolve those issues to ensure the fix finishes deployment as quickly as possible. Next update by: Wednesday, February 10, 2021, at 2:30 AMUTC; PublishedTime=2021-02-10T00:23:12.487Z

Status: We're continuing to resolve internal issues that are preventing our fix deployment from progressing as expected. Next update by: Wednesday, February 10, 2021, at 4:30 AMUTC; PublishedTime=2021-02-10T02:15:59.327Z

Final status: We've completed fix deployment to all affected environments and confirmed via internal testing that impact has been remediated. Next update by: Wednesday, February 10, 2021, at 4:30 AMUTC; PublishedTime=2021-02-10T02:15:59.327Z

--------------------------------------------------------------------------------------------------
Title Users may be unable to access the Yammer service
ID YA238626
Service Yammer Enterprise
Feature Yammer Components
Status Post-incident report published
Impact Users may have been unable to access the Yammer service.
Severity
Start time 2021-02-09T00:05:00Z
End time 2021-02-09T01:10:00Z
Last update 2021-02-16T20:12:07.247Z
More info Users may have received the following error: "{"type":"Service", "error": "Not enough healthy backends available"
Root cause As part of improving a core background feature, we made a configuration change to our network load-balancing components, intended to optimize routing and depreciate some legacy service infrastructure. The configuration update inadvertently missed a parameter. This resulted in the network load-balancing components marking all traffic routing to that core background feature as unhealthy, resulting in impact. Next steps: - For a more comprehensive list of next steps and actions, please refer to the Post Incident Review document. We'll publish a post-incident report within five business days.; PublishedTime=2021-02-09T01:50:37.997Z

YA238626 Details

Status:''''''''". Current status: We're investigating a potential issue and checking for impact to your organization. We'll provide an update within 30 minutes.

Status:''''''''". Current status: We're analyzing service logs to determine the source of the issue. Next update by: Tuesday, February 9, 2021, at 2:30 AMUTC; PublishedTime=2021-02-09T01:04:53.613Z

Final status:". Final status: We've determined that a recent configuration change to a portion of networking infrastructure inadvertently prevented users from accessing the Yammer service. We've reverted the change and confirmed via telemetry that service is restored. Next update by: Tuesday, February 9, 2021, at 2:30 AMUTC; PublishedTime=2021-02-09T01:04:53.613Z

--------------------------------------------------------------------------------------------------
Title Admins may not see up to date data for any report in the Microsoft 365 admin center
ID MO239120
Service Microsoft 365 suite
Feature Administration
Status Service restored
Impact Admins attempting to view any report in the admin center may not have seen up to date data.
Severity
Start time 2021-02-09T00:00:00Z
End time 2021-02-22T15:00:00Z
Last update 2021-02-23T03:39:13.047Z
More info The following reports were delayed starting from February 15, 2021: - OneDrive for Business activity and site usage - SharePoint Online activity and site usage The following reports were delayed starting from February 15, 2021, but have been resolved: - Email activity - Any Yammer report - Email app usage - Group activity - License activation Forms and Forms Reports were delayed starting on February 14, 2021 and have since been remediated. Final status: We've resolved an additional authentication issue that was responsible for residual reports delays, and we've confirmed that all reports are now up to date. Scope of impact: This issue affected admins attempting to access the reports listed in the More info section within the Microsoft 365 admin center. Start time: Tuesday, February 9, 2021, at 12:00 AM UTC End time: Monday, February 22, 2021, at 3:00 PM UTC Root cause: A portion of infrastructure that facilitates reports in the Microsoft 365 admin portal was performing below acceptable thresholds, resulting in impact. Next steps: - We're reviewing our service infrastructure to better understand how authentication and partner service infrastructure configurations caused impact, and to explore potential actions to prevent similar impact in the future. This is the final update for the event.; PublishedTime=2021-02-23T03:38:38.647Z
Root cause A portion of infrastructure that facilitates reports in the Microsoft 365 admin portal was performing below acceptable thresholds, resulting in impact. Next steps: - We're reviewing our service infrastructure to better understand how authentication and partner service infrastructure configurations caused impact, and to explore potential actions to prevent similar impact in the future. This is the final update for the event.; PublishedTime=2021-02-23T03:38:38.647Z

MO239120 Details

Status: We've detected an issue where admins may not see recent License, Group, Site Usage and OneDrive reports in the Microsoft 365 admin portal. We're currently investigating the impacted infrastructure to identify the root cause so that we can determine the best mitigation method. Next update by: Saturday, February 13, 2021, at 4:00 AMUTC; PublishedTime=2021-02-12T03:18:51.597Z

Status: We've determined that a portion of infrastructure that facilitates reports in the Microsoft 365 admin portal is performing below acceptable thresholds, resulting in impact. We've confirmed that admins are able to view all of the Group reports as expected, and we're working to resolve this infrastructure issue to ensure License, Site Usage, and OneDrive reports are generated as quickly as possible. Next update by: Sunday, February 14, 2021, at 5:00 AMUTC; PublishedTime=2021-02-13T03:20:42.827Z

Status: During our mitigation efforts to resolve the remaining reports, we found that there are additional delays to Teams reports as well starting on Sunday, February 14, 2021. While we're continuing to address the data delays for the License and Site Usage reports, we're also investigating the associated logs in relation to the Teams reporting infrastructure to better understand what's causing impact and how to best remediate impact. Next update by: Monday, February 15, 2021, at 5:00 AMUTC; PublishedTime=2021-02-14T04:03:46.787Z

Status: We've confirmed that the Teams Usage reports are now up to date. We're continuing our efforts to allocate resources to process the remaining License and Site Usage reports to get them up to date. Next update by: Tuesday, February 16, 2021, at 5:00 AMUTC; PublishedTime=2021-02-15T03:29:25.12Z

Status: Our monitoring indicates that License reports are now up to date. We're working to allocate the needed resources to process the remaining Site Usage reports and get them up to date. Next update by: Wednesday, February 17, 2021, at 5:00 AMUTC; PublishedTime=2021-02-16T03:58:35.113Z

Status: We've made a configuration change to improve the data source and expect the Site Usage reports to be refreshed through February 13, 2021 within 24 hours. Next update by: Thursday, February 18, 2021, at 5:00 AMUTC; PublishedTime=2021-02-17T03:20:40.793Z

Status: We've confirmed that Site Usage reports through February 14, 2021 are now refreshed and reports for February 15, 2021 are currently in the process of backfilling. Admins may see reports refresh over the next 24 hours as the data continues to catch up in our processing pipelines. In parallel, we're reviewing the newly impacted reports to determine if additional actions are needed. Next update by: Friday, February 19, 2021, at 5:00 AM UTC; PublishedTime=2021-02-18T02:09:53.31Z

Status: Our review of the newly impacted reports indicates that there's an authentication issue occurring within our data reporting service that's causing latency when fetching said data for the admin center. We're reviewing the authentication path and associated logs to better understand why the issue is occurring and how to most efficiently mitigate it. Next update by: Saturday, February 20, 2021, at 5:00 AM UTC; PublishedTime=2021-02-19T03:47:47.08Z

Status: We've reviewed the authentication path and associated logs and confirmed that the best method of mitigation was to update the configuration of partner service infrastructure to allow the service to properly access the needed information again. We've made the necessary adjustments and confirmed that impact has been resolved for all reports except for OneDrive for Business, SharePoint Online, groups activity, and License activity. Additional analysis shows that there's a secondary issue preventing these reports from updating properly and we're investigating the service infrastructure to identify the root cause so we can fully restore the remaining reports. Next update by: Monday, February 22, 2021, at 5:00 AM UTC; PublishedTime=2021-02-20T04:08:31.263Z

Status: We can see in our monitoring that all reports except for the SharePoint Online and OneDrive for Business site usage are now up to date. We're continuing our analysis of the service infrastructure to identify what is preventing the reports of updating as expected. Next update by: Tuesday, February 23, 2021, at 5:00 AM UTC; PublishedTime=2021-02-22T02:13:38.943Z

Final status: We've resolved an additional authentication issue that was responsible for residual reports delays, and we've confirmed that all reports are now up to date. Next update by: Tuesday, February 23, 2021, at 5:00 AM UTC; PublishedTime=2021-02-22T02:13:38.943Z

--------------------------------------------------------------------------------------------------
Title Admins can't provision or sync Azure Active Directory (Azure AD) changes between on-premises and Exchange Online
ID EX239037
Service Exchange Online
Feature Management and Provisioning
Status Service restored
Impact Admins may have been unable to provision or sync Azure AD changes between on-premises and Exchange Online environments.
Severity
Start time 2021-02-08T16:00:00Z
End time 2021-02-11T23:22:00Z
Last update 2021-02-12T04:30:26.547Z
More info
Root cause A recent service update setting change prevented admins from provisioning or syncing Azure AD changes between on-premises and Exchange Online environments. Next steps: - To help prevent similar impact in the future, we're further determining why the change affected admins and how we can improve our validation procedures to catch such issues before deployment. This is the final update for the event.; PublishedTime=2021-02-12T04:30:26.547Z

EX239037 Details

Status: We're investigating a potential issue and checking for impact to your organization. We'll provide an update within 30 minutes.; PublishedTime=2021-02-11T17:08:17.257

Status: We've identified a recent service update that has introduced a setting change that is impacting admins' ability to provision or sync Azure AD changes between on-premises and Exchange Online environments. We're exploring our available options for updating the setting change to remediate impact. Next update by: Thursday, February 11, 2021, at 7:30 PMUTC; PublishedTime=2021-02-11T17:35:23.947Z

Status: We've determined that a fix had previously been developed and began deployment, which is expected to address this issue; however, the deployment has yet to complete. We're monitoring the fix deployment, which is currently 99 percent complete, and we expect it will finish and resolve impact by Friday, February 12, 2021, at approximately 1:00 AM UTC Next update by: Friday, February 12, 2021, at 1:00 AMUTC; PublishedTime=2021-02-11T18:17:20.65Z

Status: While monitoring our fix deployment, we've determined that the solution required a change to ensure the problem doesn't reoccur when future service updates are deployed. We've updated the fix and now expect that it will be full deployed by Friday, February 12, 2021, at approximately 6:00 AM UTC. Next update by: Friday, February 12, 2021, at 6:00 AMUTC; PublishedTime=2021-02-11T19:32:56.67Z

Final status: We've finished deploying the fix and confirmed via telemetry that service is restored. Next update by: Friday, February 12, 2021, at 6:00 AMUTC; PublishedTime=2021-02-11T19:32:56.67Z

--------------------------------------------------------------------------------------------------
Title Some users may have been experiencing issues when saving files within the SharePoint Online service
ID SP238518
Service SharePoint Online
Feature SharePoint Features
Status Service restored
Impact Users users may have experienced issues when saving files within the SharePoint Online service.
Severity
Start time 2021-02-08T04:50:00Z
End time 2021-02-08T09:50:00Z
Last update 2021-02-08T10:38:13.173Z
More info
Root cause A recent flight change that was made to resolve a different issue, was causing save requests to SharePoint Online to timeout, causing impact. Next steps: - We're reviewing our configuring procedures to find ways to prevent this problem from happening again. This is the final update for the event.; PublishedTime=2021-02-08T10:24:12.033Z

SP238518 Details

Status: We're analyzing diagnostic data to determine the source of the issue. Next update by: Monday, February 8, 2021, at 11:00 AM UTC; PublishedTime=2021-02-08T08:49:11.82Z

Final status: We've determined that a recent flight change that was made to resolve a different issue, was causing save requests to SharePoint Online to timeout, causing impact. Our auto-recovery system has repaired this problem and we've confirmed, after monitoring the environment, that this has resolved the issue. Next update by: Monday, February 8, 2021, at 11:00 AM UTC; PublishedTime=2021-02-08T08:49:11.82Z

--------------------------------------------------------------------------------------------------
Title Admins may have been unable to view up to date usage report data in Microsoft 365 admin center
ID MO238490
Service Microsoft 365 suite
Feature Administration
Status Service restored
Impact Admins may have been unable to view up to date usage report data in the Microsoft 365 admin center.
Severity
Start time 2021-02-08T00:00:00Z
End time 2021-02-10T09:00:00Z
Last update 2021-02-10T11:19:39.55Z
More info Some examples of impacted usage reports include: -Email -Group Final status: We've confirmed that the usage reports are caught up, and that impact is mitigated. Scope of impact: This issue could have potentially affected any of your admins attempting to view usage report data in the Microsoft 365 admin center if they were routed through the affected infrastructure. Start time: Monday, February 8, 2021, at 12:00 AM UTC End time: Wednesday, February 10, 2021, at 9:00 AM UTC Root cause: An upstream data pipeline issue was causing Microsoft 365 report delays. Next steps: - We're analyzing performance data and trends on the affected systems to help prevent this problem from happening again. This is the final update for the event.; PublishedTime=2021-02-10T11:19:39.55Z
Root cause An upstream data pipeline issue was causing Microsoft 365 report delays. Next steps: - We're analyzing performance data and trends on the affected systems to help prevent this problem from happening again. This is the final update for the event.; PublishedTime=2021-02-10T11:19:39.55Z

MO238490 Details

Status: An issue with our data pipeline for Microsoft 365 report data is causing unintended delays with generating usage reports. We're working to determine the best method to address the issue and remediate the impact. Next update by: Tuesday, February 9, 2021, at 4:00 AMUTC; PublishedTime=2021-02-08T04:23:52.673Z

Status: We can see that OneDrive for Business and Site Usage report data is up to date. We're continuing our efforts to allocate more resources to get the remaining affected usage reports caught up. Next update by: Wednesday, February 10, 2021, at 4:00 AM UTC; PublishedTime=2021-02-09T02:51:40.17Z

Status: Our monitoring shows that data is continuing to process for email and group usage reports. We'll continue to allocate resources as needed and monitor to ensure that the usage reports are caught up. Next update by: Wednesday, February 10, 2021, at 12:00 PMUTC; PublishedTime=2021-02-10T02:33:34.097Z

Final status: We've confirmed that the usage reports are caught up, and that impact is mitigated. Next update by: Wednesday, February 10, 2021, at 12:00 PMUTC; PublishedTime=2021-02-10T02:33:34.097Z

--------------------------------------------------------------------------------------------------
Title Users accessing Microsoft Intune via Microsoft Graph workloads may experience timeouts and failures
ID IT238316
Service Microsoft Intune
Feature Microsoft Intune
Status Service restored
Impact Users attempting to access Microsoft Intune via Microsoft Graph workloads may have noticed timeouts and failures.
Severity
Start time 2021-02-07T00:45:00Z
End time 2021-02-07T08:00:00Z
Last update 2021-02-07T18:49:46.5Z
More info This issue impacted both connections via programmatic Microsoft Graph calls and calls made through the Microsoft Endpoint Manager Console. Final status: We've confirmed that our fix deployment has completed and that impact has been remediated. Scope of impact: Users attempting to access Microsoft Intune through programmatic Microsoft Graph workflows and calls made using the Microsoft Endpoint Manager Console may have experienced failures. Start time: Sunday, February 7, 2021, at 12:45 AM'UTC End time: Sunday, February 7, 2021, at 8:00 AM'UTC Root cause: A subsection of SQL database infrastructure dropped below our manageable service performance thresholds, resulting in impact. Next steps: - We're analyzing performance data and trends on the affected systems to determine why the SQL databases dropped below our manageable service performance thresholds and to help prevent this problem from happening again. This is the final update for the event.; PublishedTime=2021-02-07T18:49:46.5Z
Root cause A subsection of SQL database infrastructure dropped below our manageable service performance thresholds, resulting in impact. Next steps: - We're analyzing performance data and trends on the affected systems to determine why the SQL databases dropped below our manageable service performance thresholds and to help prevent this problem from happening again. This is the final update for the event.; PublishedTime=2021-02-07T18:49:46.5Z

IT238316 Details

Status: We're investigating a potential issue and checking for impact to your organization. We'll provide an update within 30 minutes.; PublishedTime=2021-02-07T04:30:02.05

Status: Our automated health monitoring systems alerted us to access issues through Microsoft Graph workloads and Microsoft Endpoint Manager Console. During our initial investigation into the available data, our service telemetry indicated that the service had recovered without manual actions. We're continuing to monitor the impacted environment to ensure that the issue does not reoccur. Next update by: Sunday, February 7, 2021, at 6:00 AMUTC; PublishedTime=2021-02-07T05:00:01.54Z

Status: Additional investigation confirmed that the issue is caused by a recent update to the service's SQL infrastructure. We reverted this change to mitigate impact, but further monitoring confirms that we're still seeing intermittent drops in connection reliability. We're continuing our analysis to identify the root cause of this residual impact. Next update by: Sunday, February 7, 2021, at 8:30 AMUTC; PublishedTime=2021-02-07T05:57:47.337Z

Status: We've confirmed that the residual impact is being caused by a subsection of our infrastructure not performing as expected. We're applying a fix and restarting the affected infrastructure to mitigate impact. Next update by: Sunday, February 7, 2021, at 6:00 PM UTC; PublishedTime=2021-02-07T08:04:34.517Z

Status: The deployment of our fix is ongoing and we're continuing to monitor the environment while the fix is applied. We're expecting for the deployment to have completed and for impact to be remediated by Sunday, February 7, 2021, at 8:30 PM UTC. Next update by: Sunday, February 7, 2021, at 8:30 PMUTC; PublishedTime=2021-02-07T16:29:57.817Z

Final status: We've confirmed that our fix deployment has completed and that impact has been remediated. Next update by: Sunday, February 7, 2021, at 8:30 PMUTC; PublishedTime=2021-02-07T16:29:57.817Z

--------------------------------------------------------------------------------------------------
Title Admins may experience intermittent latency receiving reports in the Security & Compliance Center (SCC)
ID EX238239
Service Exchange Online
Feature E-Mail and calendar access
Status Service restored
Impact Admins may have experienced intermittent latency receiving reports in the SCC.
Severity
Start time 2021-02-05T20:00:00Z
End time 2021-02-07T00:30:00Z
Last update 2021-02-07T01:09:47.57Z
More info
Root cause Some of our services responsible for handling data reporting had an unexpected increase in requests, resulting in intermittent delays. Next steps: - We'll continue to monitor performance metrics into the upcoming business week to ensure that service continues to operate at expected levels. - We're evaluating our internal data processing services to formulate a long term solution to prevent impact from reoccurring. This is the final update for the event.; PublishedTime=2021-02-07T01:09:47.57Z

EX238239 Details

Status: We're investigating a potential issue and checking for impact to your organization. We'll provide an update within 30 minutes.

Status: We're reviewing recent changes to isolate the source of the issue while in parallel, we're working to validate and test a potential fix. Next update by: Saturday, February 6, 2021, at 11:30 PMUTC; PublishedTime=2021-02-06T22:09:52.603Z

Status: We're continuing to review recent changes to isolate the source of the issue and test a potential fix. Next update by: Sunday, February 7, 2021, at 1:30 AMUTC; PublishedTime=2021-02-06T23:09:24.533Z

Final status: Our investigation determined that some of our services responsible for handling data reporting had an unexpected increased in requests, resulting in intermittent delays. We've temporarily increased capacity for the affected services to assist with processing these requests and confirmed through internal monitoring that SCC reports now generate in a timely manner. We'll monitor service health into the upcoming business week to ensure that service continues to operate at expected levels. Next update by: Sunday, February 7, 2021, at 1:30 AMUTC; PublishedTime=2021-02-06T23:09:24.533Z

--------------------------------------------------------------------------------------------------
Title Admins intermittently can't perform actions within the Microsoft Endpoint Manager admin console
ID IT237956
Service Microsoft Intune
Feature Microsoft Intune
Status Service restored
Impact Admins may have experienced timeouts or errors when performing actions in the Microsoft Endpoint Manager admin console.
Severity
Start time 2021-02-05T19:20:00Z
End time 2021-02-05T20:00:00Z
Last update 2021-02-05T21:05:28.84Z
More info Admins may have also experience intermittent errors and timeouts with any Graph API calls to Microsoft Intune. Final status: We've completed restarting the affected SQL infrastructure and confirmed mitigation via telemetry. Scope of impact: Any admin serviced through the affected infrastructure may have been impacted. Start time: Friday, February 5, 2021, at 7:20 PM'UTC End time: Friday, February 5, 2021, at 8:00 PM'UTC Root cause: A portion of database infrastructure experienced a SQL contention issue, resulting in impact. Next steps: - We're reviewing our SQL code for improved performance and potential automated recovery options to reduce or prevent future occurrences. This is the final update for the event.; PublishedTime=2021-02-05T21:03:24.577Z
Root cause A portion of database infrastructure experienced a SQL contention issue, resulting in impact. Next steps: - We're reviewing our SQL code for improved performance and potential automated recovery options to reduce or prevent future occurrences. This is the final update for the event.; PublishedTime=2021-02-05T21:03:24.577Z

IT237956 Details

Status: We've identified a portion database of infrastructure that experienced a SQL contention issue, resulting in impact. We've restarted the affected infrastructure and are monitoring as system health improves. Next update by: Friday, February 5, 2021, at 10:00 PMUTC; PublishedTime=2021-02-05T20:42:08.077Z

Final status: We've completed restarting the affected SQL infrastructure and confirmed mitigation via telemetry. Next update by: Friday, February 5, 2021, at 10:00 PMUTC; PublishedTime=2021-02-05T20:42:08.077Z

--------------------------------------------------------------------------------------------------
Title Some users may see incoming external email with attachments unexpectedly flagged as malware in Exchange Online
ID EX237827
Service Exchange Online
Feature E-Mail timely delivery
Status Service restored
Impact Users may have seen incoming external email with attachments unexpectedly flagged as malware in Exchange Online.
Severity
Start time 2021-02-04T21:59:00Z
End time 2021-02-04T23:00:00Z
Last update 2021-02-05T04:54:04.877Z
More info
Root cause A recent logic change inadvertently caused incoming external email messages with attachments to go to quarantine. Next steps: - To help prevent similar impact in the future, we're further investigating why the logic change caused impact and how we can improve our validation procedures to catch these issues before applying such changes. This is the final update for the event.; PublishedTime=2021-02-05T04:43:54.997Z

EX237827 Details

Status: We're investigating a potential issue and checking for impact to your organization. We'll provide an update within 30 minutes.

Status: We're reviewing message headers along with examples of impacted messages to determine our next troubleshooting steps. Next update by: Friday, February 5, 2021, at 4:00 AMUTC; PublishedTime=2021-02-05T01:30:49.393Z

Status: We've determined that a recent logic change is inadvertently causing incoming external email messages with attachments to go to quarantine. We're reprocessing the affected email messages that shouldn't be in quarantine to release them from quarantine. Next update by: Friday, February 5, 2021, at 5:30 AMUTC; PublishedTime=2021-02-05T03:59:11.02Z

Final status: We've confirmed that reprocessing the affected email messages is successfully releasing them from quarantine and new external email messages sent aren't being improperly flagged, mitigating impact. Next update by: Friday, February 5, 2021, at 5:30 AMUTC; PublishedTime=2021-02-05T03:59:11.02Z

--------------------------------------------------------------------------------------------------
Title Users may experience errors and timeouts when calling Microsoft Graph-based workloads via Intune
ID IT237744
Service Microsoft Intune
Feature Microsoft Intune
Status Service restored
Impact Users may have experienced errors and timeouts when calling Microsoft Graph-based workloads via Intune.
Severity
Start time 2021-02-04T18:15:00Z
End time 2021-02-04T19:00:00Z
Last update 2021-02-04T19:55:07.013Z
More info Users may have also experienced errors and timeouts when calling the Microsoft Endpoint Manager admin console. Final status: We've confirmed via monitoring that restarting the affected infrastructure to restore the service has successfully resolved the issue. Scope of impact: Impact may have occurred for all users attempting to call Microsoft Graph-based workloads via Intune and the Microsoft Endpoint Manager admin console. Start time: Thursday, February 4, 2021, at 6:15 PM'UTC End time: Thursday, February 4, 2021, at 7:00 PM'UTC Root cause: A portion of SQL infrastructure was performing below acceptable thresholds, resulting in impact. Next steps: - We're analyzing performance data and trends on the affected SQL infrastructure to better understand why it was degraded and help prevent this problem from happening again. This is the final update for the event.; PublishedTime=2021-02-04T19:55:07.013Z
Root cause A portion of SQL infrastructure was performing below acceptable thresholds, resulting in impact. Next steps: - We're analyzing performance data and trends on the affected SQL infrastructure to better understand why it was degraded and help prevent this problem from happening again. This is the final update for the event.; PublishedTime=2021-02-04T19:55:07.013Z

IT237744 Details

Status: We've identified that a portion of SQL infrastructure is performing below acceptable thresholds, resulting in impact. We're restoring the impacted infrastructure to potentially alleviate the issue and we're monitoring the service to ensure that the issue resolves successfully. Next update by: Thursday, February 4, 2021, at 8:30 PMUTC; PublishedTime=2021-02-04T19:13:09.613Z

Final status: We've confirmed via monitoring that restarting the affected infrastructure to restore the service has successfully resolved the issue. Next update by: Thursday, February 4, 2021, at 8:30 PMUTC; PublishedTime=2021-02-04T19:13:09.613Z

--------------------------------------------------------------------------------------------------
Title Users couldn't join Microsoft Teams meetings
ID TM237733
Service Microsoft Teams
Feature Teams Components
Status Post-incident report published
Impact Users may have been unable to join Microsoft Teams meetings and Live Events.
Severity
Start time 2021-02-04T16:00:00Z
End time 2021-02-04T17:20:00Z
Last update 2021-02-11T20:22:08.167Z
More info Users may have been unable to join a meeting with the Teams Desktop client, Teams web app, Teams mobile clients, or via phone dial in. Users would have seen the message "Sorry, we couldn't connect you." when attempting to connect to meetings or calls. Additionally, this event may have impacted users' ability to join Teams Live Events. Final status: We've determined that a drop in API availability resulted in an issue in which users may have been unable to join Teams meetings or Live Events. We rebalanced user traffic load and have validated that service has been restored. Scope of impact: Any user hosted on the affected infrastructure would have been potentially impacted by this incident. Start time: Thursday, February 4, 2021, at 4:00 PM UTC End time: Thursday, February 4, 2021, at 5:20 PM UTC Root cause: An incorrect load-balancing configuration caused a higher than expected weighting of user requests to be routed to a specific section of North American processing infrastructure. During the period of peak business hours, the section of infrastructure which received the increased amount of service traffic began to perform below acceptable thresholds, resulting in impact. Next steps: - For a more comprehensive list of next steps and actions, please refer to the Post Incident Review document. We'll publish a post-incident report within five business days.; PublishedTime=2021-02-04T17:54:26.103Z
Root cause An incorrect load-balancing configuration caused a higher than expected weighting of user requests to be routed to a specific section of North American processing infrastructure. During the period of peak business hours, the section of infrastructure which received the increased amount of service traffic began to perform below acceptable thresholds, resulting in impact. Next steps: - For a more comprehensive list of next steps and actions, please refer to the Post Incident Review document. We'll publish a post-incident report within five business days.; PublishedTime=2021-02-04T17:54:26.103Z

TM237733 Details

Status: We're investigating a potential issue and checking for impact to your organization. We'll provide an update within 30 minutes.; PublishedTime=2021-02-04T16:26:07.803

Status: We've identified a drop in availability for services that facilitate meeting join functionality. We're working to determine the next steps to remediate the impact. Next update by: Thursday, February 4, 2021, at 6:00 PM UTC; PublishedTime=2021-02-04T16:51:22.297Z

Final status: We've determined that a drop in API availability resulted in an issue in which users may have been unable to join Teams meetings or Live Events. We rebalanced user traffic load and have validated that service has been restored. Next update by: Thursday, February 4, 2021, at 6:00 PM UTC; PublishedTime=2021-02-04T16:51:22.297Z

--------------------------------------------------------------------------------------------------
Title Admins may experience failures when performing search queries for telephone numbers in the Microsoft Teams admin center
ID TM237660
Service Microsoft Teams
Feature Teams Components
Status Service restored
Impact Admins may have experienced failures when performing search queries for telephone numbers.
Severity
Start time 2021-02-03T23:30:00Z
End time 2021-02-04T02:50:00Z
Last update 2021-02-04T03:00:13.73Z
More info
Root cause A recent change caused a corruption event on the Telephone Geography Service, resulting in impact. Next steps: -We're reviewing our change implementation procedures to find better ways to prevent similar events in the future. This is the final update for the event.; PublishedTime=2021-02-04T03:00:13.73Z

TM237660 Details

Status: We're investigating a potential issue and checking for impact to your organization. We'll provide an update within 30 minutes.

Status: We've identified that a recent update to the Telephone Geography Service, which the affected search relies on, has resulted in impact. We're working to revert the change to mitigate impact for all users. Next update by: Thursday, February 4, 2021, at 4:00 AMUTC; PublishedTime=2021-02-04T01:36:24.127Z

Final status: We reverted the change and monitored the environment through service health monitoring to verify that service was restored. Next update by: Thursday, February 4, 2021, at 4:00 AMUTC; PublishedTime=2021-02-04T01:36:24.127Z

--------------------------------------------------------------------------------------------------
Title Users will be unable to create Join a Team codes, or join a team using the Join by Code feature
ID TM237659
Service Microsoft Teams
Feature Teams Components
Status Service restored
Impact Users attempting to create a Join a Team code, or join a team using a Join a Team code, would have experienced failures.
Severity
Start time 2021-02-03T23:24:00Z
End time 2021-02-06T03:20:00Z
Last update 2021-02-07T23:42:44.78Z
More info The ability of team owners and admins to add members to teams was available via other standard methods: - Users were able to be added to teams directly by an owner. - Admins could have utilized the Teams admin center to add users. - The Microsoft Graph API was available for admins to add users. - Owners were able to add users using the M365 Group functionality. How to add a member to a team: https://support.microsoft.com/en-us/office/add-members-to-a-team-in-teams-aff2249d-b456-4 bc3-81e7-52327b6b38e9 Final status: We've completed deployment of the fix to the impacted systems and confirmed after extensive monitoring that the service is working as expected. Furthermore, the Join by Code feature has been re-enabled and the issue is resolved. Scope of impact: Any user attempting to join a team using the Join by Code feature would have experienced impact. Start time: Wednesday, February 3, 2021, at 11:24 PM UTC End time: Saturday, February 6, 2021, at 3:20 AM UTC Preliminary root cause: We experienced a problem with a portion of our infrastructure responsible for the Join by Code functionality, which prompted us to temporarily disable the feature while we addressed the issue. Next steps: - We are working on adding additional monitoring signals to alert us more quickly, prevent similar issues from happening in the future, and help improve our mitigation timelines. This is the final update for the event.; PublishedTime=2021-02-06T16:30:34.94Z
Root cause

TM237659 Details

Status: We've identified that an issue within a portion of our service infrastructure requires us to temporarily disable the Join by Code feature. We're reviewing the impacted infrastructure to determine the best mitigation method. Next update by: Friday, February 5, 2021, at 2:00 AMUTC; PublishedTime=2021-02-04T00:09:23.513Z

Status: We're continuing to review the impacted infrastructure to gain an understanding of the underlying root cause, and to formulate a mitigation plan. Next update by: Saturday, February 6, 2021, at 12:00 AMUTC; PublishedTime=2021-02-05T00:36:09.033Z

Status: We're working to deploy a fix for the affected infrastructure in the next 24 hours. Once the rollout is finished we'll complete post-deployment validations and testing to ensure the service is functioning as intended before re-enabling the Join by Code functionality. Next update by: Sunday, February 7, 2021, at 12:00 AM UTC; PublishedTime=2021-02-05T23:41:30.357Z

Final status: We've completed deployment of the fix to the impacted systems and confirmed after extensive monitoring that the service is working as expected. Furthermore, the Join by Code feature has been re-enabled and the issue is resolved. Next update by: Sunday, February 7, 2021, at 12:00 AM UTC; PublishedTime=2021-02-05T23:41:30.357Z

--------------------------------------------------------------------------------------------------
Title Users may be experiencing delays exceeding ten minutes or failures when receiving external email messages
ID EX237654
Service Exchange Online
Feature E-Mail timely delivery
Status Post-incident report published
Impact Users may have experienced delays exceeding ten minutes or failures when receiving external email messages.
Severity
Start time 2021-02-03T21:20:00Z
End time 2021-02-04T02:10:00Z
Last update 2021-02-11T20:27:32.813Z
More info Some users may have received the error "4.3.2 Temporary server error. Please try again later", indicating that the message was not received. Subsequent attempts may have continued to be unsuccessful. This was expected while we worked to retry queued messages. Additionally, third-party email filtering and management services may have been experiencing more significant delays or failures as external senders retry to send us messages they had queued. Final status: We've confirmed after extended monitoring that re-routing to alternate infrastructure was successful in mitigating impact to users. Email messages impacted by this issue should now be delivered as expected. Scope of impact: This issue could have potentially affected any of your users if they were routed through the affected infrastructure. Start time: Wednesday, February 3, 2021, at 9:20 PM UTC End time: Thursday, February 4, 2021, at 2:10 AM UTC Root cause: Due to an issue with the freshness of data our front-end components restarted, triggering a clearing of the cache, which subsequently increased query rates into the directory database components. This prevented caches from filling up with the required data, resulting in email delays and delivery failures Next steps: - For a more comprehensive list of next steps and actions, please refer to the Post Incident Review document. We'll publish a post-incident report within five business days.; PublishedTime=2021-02-04T02:25:41.673Z
Root cause Due to an issue with the freshness of data our front-end components restarted, triggering a clearing of the cache, which subsequently increased query rates into the directory database components. This prevented caches from filling up with the required data, resulting in email delays and delivery failures Next steps: - For a more comprehensive list of next steps and actions, please refer to the Post Incident Review document. We'll publish a post-incident report within five business days.; PublishedTime=2021-02-04T02:25:41.673Z

EX237654 Details

Status: We're reviewing email transport logs to determine the underlying cause of the issue. Next update by: Thursday, February 4, 2021, at 12:30 AMUTC; PublishedTime=2021-02-03T22:29:56.797Z

Status: We're reviewing a recent deployment that may be the source of the issue. Further investigation has identified that a subset of degraded infrastructure is unable to process requests, which is resulting in a message backlog and causing delivery delays. Next update by: Thursday, February 4, 2021, at 12:00 AM UTC; PublishedTime=2021-02-03T23:18:09.527Z

Status: We're in the process of re-routing traffic from the affected database infrastructure to alternate systems to improve service health and provide user relief. In parallel, we're continuing to review recent changes and database telemetry in order to isolate and address the source of the issue. Next update by: Thursday, February 4, 2021, at 1:00 AM UTC; PublishedTime=2021-02-03T23:55:43.63Z

Status: We're continuing to manually re-route traffic from the affected database infrastructure to alternate systems and are seeing signs of recovery. Additionally, we're working determine the source of degradation. Next update by: Thursday, February 4, 2021, at 2:30 AMUTC; PublishedTime=2021-02-04T00:59:30.93Z

Final status: We've confirmed after extended monitoring that re-routing to alternate infrastructure was successful in mitigating impact to users. Email messages impacted by this issue should now be delivered as expected. Next update by: Thursday, February 4, 2021, at 2:30 AMUTC; PublishedTime=2021-02-04T00:59:30.93Z

--------------------------------------------------------------------------------------------------
Title Bot authoring functionality degradation
ID CR237620
Service Dynamics 365 Apps
Feature Organization access
Status Service restored
Impact Users may have been experiencing errors and latency when attempting to author bots.
Severity
Start time 2021-02-03T18:20:10Z
End time 2021-02-04T05:53:00Z
Last update 2021-02-04T06:45:43.533Z
More info
Root cause

CR237620 Details

--------------------------------------------------------------------------------------------------
Title Users may be unable to use the '@' function to add attendees when scheduling a meeting on Microsoft Teams
ID TM241048
Service Microsoft Teams
Feature Teams Components
Status Service restored
Impact Users may have been unable to use the '@' function to add attendees when scheduling a meeting on Microsoft Teams.
Severity Sev2
Start time 2021-02-03T17:00:00Z
End time 2021-05-07T04:49:00Z
Last update 2021-05-07T05:02:38.603Z
More info As an alternative method, users were able to utilize this functionality using the Microsoft Teams web app. Final status: The deployment has completed and we've verified internally that impact is remediated. Scope of impact: Any user attempting to perform this action on a Mac OS device may have been impacted. Start time: Wednesday, February 3, 2021, at 5:00 PM'UTC End time: Friday, May 7, 2021, at 4:49 AM'UTC Root cause: A code regression prevented the use of special characters within Microsoft Teams, which prevented users from adding attendees to meetings. Next steps: -We're reviewing our update and testing procedures to help better identify similar code regressions prior to their deployment. This is the final update for the event.; PublishedTime=2021-05-07T05:02:38.603Z
Root cause A code regression prevented the use of special characters within Microsoft Teams, which prevented users from adding attendees to meetings. Next steps: -We're reviewing our update and testing procedures to help better identify similar code regressions prior to their deployment. This is the final update for the event.; PublishedTime=2021-05-07T05:02:38.603Z

TM241048 Details

Status: We're reviewing details provided in the support case to determine the next steps needed to resolve this issue. Next update by: Monday, February 22, 2021, at 3:30 PM UTC; PublishedTime=2021-02-22T13:29:54.153Z

Status: We've identified that there was a code regression which was preventing the use of special characters within Microsoft Teams and therefore, not allowing users to add attendees to meetings. We've developed and deployed a fix for this issue and estimate that it will reach all the affected infrastructure within 4 weeks. Next update by: Friday, March 12, 2021, at 7:00 AM UTC; PublishedTime=2021-02-22T15:17:31.683Z

Status: The code fix is continuing to progress as expected. We anticipate it will finish saturating throughout the affected environment by Friday, March 19, 2021. Next update by: Monday, March 8, 2021, at 7:00 AMUTC; PublishedTime=2021-03-01T05:50:31.523Z

Status: We're monitoring the fix as it continues deploying on schedule. We still expect it to finish saturating throughout the affected environment by Friday, March 19, 2021. Next update by: Friday, March 19, 2021, at 7:00 AMUTC; PublishedTime=2021-03-08T05:30:22.46Z

Status: The fix deployment is taking longer than expected to reach the affected environments. We're continuing to monitor the fix as it deploys at a monitored pace to ensure no further issues occur. Additionally, we recommend users to use the Microsoft Teams web app to avoid this issue while scheduling meetings until the deployment is complete. Next update by: Friday, March 26, 2021, at 7:00 AMUTC; PublishedTime=2021-03-19T06:11:44.887Z

Status: The fix is continuing to run a little slower than expected. We're continuing to monitor as the fix is safely implemented through our change management process. Next update by: Friday, April 2, 2021, at 7:00 AMUTC; PublishedTime=2021-03-26T05:10:32.167Z

Status: The fix is continuing to saturate throughout the affected environment and users may notice remediation as it continues to progress. We're monitoring the progress of the event to ensure it doesn't encounter any unexpected issues. Next update by: Friday, April 9, 2021, at 7:00 AMUTC; PublishedTime=2021-04-02T05:53:16.673Z

Status: The fix is nearing it's final stages of deployment, and based on our current estimates, we expect the deployment of this fix will be complete by Saturday, April 24, 2021. We'll provide an update by April 16, 2021 to ensure the deployment is still on track for the provided resolution time. Next update by: Friday, April 16, 2021, at 7:00 AMUTC; PublishedTime=2021-04-09T06:02:06.98Z

Status: We've confirmed that deployment remains successfully on track to be complete by Saturday, April 24, 2021. Next update by: Saturday, April 24, 2021, at 7:00 AMUTC; PublishedTime=2021-04-16T06:23:15.75Z

Status: The deployment of the fix to the remaining impacted infrastructure has been unexpectedly postponed to avoid causing inadvertent impact. We're preparing for the remaining portion to resume deploying by Monday, April 26, 2021 and we'll confirm that deployment has begun to provide a more accurate timeline to full remediation. Next update by: Tuesday, April 27, 2021, at 6:30 AMUTC; PublishedTime=2021-04-24T05:57:45.58Z

Status: The deployment of the fix to the remaining infrastructure is taking longer than expected. We're continuing to prepare for the fix to resume deployment which we expect may occur by our next communications update. Next update by: Friday, April 30, 2021, at 6:30 AMUTC; PublishedTime=2021-04-27T04:52:41.613Z

Status: The deployment of the fix is nearly complete. We expect the deployment to be done within the next four days. Next update by: Tuesday, May 4, 2021, at 6:30 AMUTC; PublishedTime=2021-04-30T04:53:38.303Z

Status: We've encountered unexpected issues deploying the fix to the remaining affected service infrastructure. We now expect the deployment to complete by the end of this week. Next update by: Friday, May 7, 2021, at 6:30 AMUTC; PublishedTime=2021-05-04T04:54:02.477Z

Final status: The deployment has completed and we've verified internally that impact is remediated. Next update by: Friday, May 7, 2021, at 6:30 AMUTC; PublishedTime=2021-05-04T04:54:02.477Z

--------------------------------------------------------------------------------------------------
Title Admins can't create or modify apps in the Microsoft Intune admin center
ID IT237615
Service Microsoft Intune
Feature Microsoft Intune
Status Service restored
Impact Admins may have been unable to create or modify apps in the Microsoft Intune admin center.
Severity
Start time 2021-02-03T14:45:00Z
End time 2021-02-03T18:00:00Z
Last update 2021-02-03T19:21:45.743Z
More info Users may have also been unable to see the latest policy reporting. Final status: We've determined that a configuration change to components utilized for service redundancy caused the primary components to become unbalanced and process traffic inefficiently. We've implemented processing optimizations on the primary components to mitigate impact, and validated impact remediation. Scope of impact: All admins may have experienced impact when attempting to create or modify apps in the Microsoft Intune admin center. Start time: Wednesday, February 3, 2021, at 2:45 PM'UTC End time: Wednesday, February 3, 2021, at 6:00 PM'UTC Root cause: A configuration change to components utilized for service redundancy caused the primary components to become unbalanced and process traffic inefficiently, resulting in impact. Next steps: - We're working to resolve the issue on the redundancy components to help ensure that the problem doesn't reoccur. This is the final update for the event.; PublishedTime=2021-02-03T19:21:45.743Z
Root cause A configuration change to components utilized for service redundancy caused the primary components to become unbalanced and process traffic inefficiently, resulting in impact. Next steps: - We're working to resolve the issue on the redundancy components to help ensure that the problem doesn't reoccur. This is the final update for the event.; PublishedTime=2021-02-03T19:21:45.743Z

IT237615 Details

Status: We're analyzing internal telemetry data to determine the source of the issue. Next update by: Wednesday, February 3, 2021, at 7:00 PMUTC; PublishedTime=2021-02-03T17:59:30.507Z

Final status: We've determined that a configuration change to components utilized for service redundancy caused the primary components to become unbalanced and process traffic inefficiently. We've implemented processing optimizations on the primary components to mitigate impact, and validated impact remediation. Next update by: Wednesday, February 3, 2021, at 7:00 PMUTC; PublishedTime=2021-02-03T17:59:30.507Z

--------------------------------------------------------------------------------------------------
Title Users may have been unable to access the Yammer service via Glammer
ID YA237538
Service Yammer Enterprise
Feature Yammer Components
Status Service restored
Impact Users may have been unable to access the Yammer service via Glammer.
Severity
Start time 2021-02-03T00:16:00Z
End time 2021-02-03T01:36:06.497Z
Last update 2021-02-03T01:36:06.497Z
More info
Root cause A recent change to provide general service optimizations introduced a misconfiguration which resulted in an outage to the Glammer service. Next steps: - We're reviewing the affecting change to determine how impact was missed during testing and validation to better prevent similar future occurrences. This is the final update for the event.; PublishedTime=2021-02-03T01:36:06.497Z

YA237538 Details

Final status: We've confirmed that a recent change to provide general service optimizations introduced a misconfiguration which resulted in an outage to the Glammer service. We've reverted the affecting change and confirmed that impact is mitigated.

--------------------------------------------------------------------------------------------------
Title Some admins experiencing intermittent errors when attempting to upload or update line-of-business (LOB) apps
ID IT237636
Service Microsoft Intune
Feature Microsoft Intune
Status Service restored
Impact Admins may experience intermittent errors when attempting to upload or update LOB apps in the Intune admin console.
Severity
Start time 2021-02-01T19:00:00Z
End time 2021-02-03T22:30:00Z
Last update 2021-02-04T00:12:52.337Z
More info
Root cause Changes within a recent build deployment were unexpectedly affecting the library which facilitates LOB app functionality in the Intune service, resulting in impact. Next steps: - We're reviewing the affected build to better understand how impact began and to explore potential actions to prevent similar impact in the future. This is the final update for the event.; PublishedTime=2021-02-04T00:12:52.337Z

IT237636 Details

Status: We're investigating a potential issue and checking for impact to your organization. We'll provide an update within 30 minutes.; PublishedTime=2021-02-03T20:20:31.033

Status: We're investigating changes within a recent deployment which is potentially causing impact to the library which facilitates the LOB apps functionality within the Intune service. We'll subsequently determine whether we can revert these changes as an approach to mitigation. Next update by: Wednesday, February 3, 2021, at 11:00 PMUTC; PublishedTime=2021-02-03T20:42:43.977Z

Status: We've determined that changes deployed within a recent build are causing the problem. We're redeploying a previous build that doesn't contain the impacting changes. Affected admins will begin to experience relief as the build deploys incrementally through the affected environments. We expect to be able to provide an estimated time to completion by the next scheduled update. Please note that once the process of reverting to the previous build completes, affected admins may need to clear their browser cache to fully remediate impact. Next update by: Thursday, February 4, 2021, at 1:00 AMUTC; PublishedTime=2021-02-03T22:29:29.28Z

Final status: We've successfully deployed a build not containing the impacting changes to all affected environments. Any admins still noticing impact will need to clear their browser cache to experience full resolution. Next update by: Thursday, February 4, 2021, at 1:00 AMUTC; PublishedTime=2021-02-03T22:29:29.28Z

--------------------------------------------------------------------------------------------------
Title Delays when accessing Microsoft 365 services and features
ID MO237339
Service Microsoft 365 suite
Feature Portal
Status Post-incident report published
Impact Users may have been experiencing delays when attempting to access Microsoft 365 services.
Severity
Start time 2021-02-01T15:40:00Z
End time 2021-02-01T18:54:00Z
Last update 2021-02-08T18:10:56.547Z
More info Reports indicate that a majority of the impact related to this event was specific to the Outlook client. However, some users reported issues with Microsoft Teams and Planner. Final status: We've monitored the environment for an extended period of time and have validated that service has been restored. We took additional measures to prevent further impact by re-routing traffic to alternate infrastructure. Scope of impact: Any user located within or served from North America may have been affected by this event. Start time: Monday, February 1, 2021, at 3:40 PM UTC End time: Monday, February 1, 2021, at 7:00 PM UTC Root cause: A configuration change made in a peering exchange's routing components limited the amount of bandwidth available at the peering point for a subset of third-party service providers, resulted in excessive packet drops at the peering point for these third-party providers, causing impact to customers accessing the service through these providers. Next steps: - For a more comprehensive list of next steps and actions, please refer to the Post Incident Review document. We'll publish a post-incident report within five business days.; PublishedTime=2021-02-01T21:57:56.257Z
Root cause A configuration change made in a peering exchange's routing components limited the amount of bandwidth available at the peering point for a subset of third-party service providers, resulted in excessive packet drops at the peering point for these third-party providers, causing impact to customers accessing the service through these providers. Next steps: - For a more comprehensive list of next steps and actions, please refer to the Post Incident Review document. We'll publish a post-incident report within five business days.; PublishedTime=2021-02-01T21:57:56.257Z

MO237339 Details

Status: We're investigating a potential issue and checking for impact to your organization. We'll provide an update within 30 minutes.; PublishedTime=2021-02-01T17:15:48.137

Status: We're reviewing diagnostic data to better understand the problem and to determine our next troubleshooting steps. Next update by: Monday, February 1, 2021, at 7:00 PM UTC; PublishedTime=2021-02-01T17:36:30.327Z

Status: We're reviewing log data gathered from affected users in order to identify the cause of the problem and to develop a plan to mitigate impact. Next update by: Monday, February 1, 2021, at 8:00 PM UTC; PublishedTime=2021-02-01T18:51:53.74Z

Status: We're investigating authentication errors within the client logs to understand why the error occurs. In parallel, we're working with a subset of affected users to capture additional data to assist with the investigation. Next update by: Monday, February 1, 2021, at 10:00 PM UTC; PublishedTime=2021-02-01T19:57:30.867Z

Status: Our monitoring indicates that the majority of the impact related to this event has been mitigated as of Monday, February 1, 2021, at 7:00 PM UTC. Our investigation has identified that a networking problem is responsible for the impact from this event, though we are continuing our work to isolate the underlying root cause. Next update by: Tuesday, February 2, 2021, at 2:00 AM UTC; PublishedTime=2021-02-01T21:08:15.977Z

Final status: We've monitored the environment for an extended period of time and have validated that service has been restored. We took additional measures to prevent further impact by re-routing traffic to alternate infrastructure. Next update by: Tuesday, February 2, 2021, at 2:00 AM UTC; PublishedTime=2021-02-01T21:08:15.977Z

--------------------------------------------------------------------------------------------------
Title Some users may be unable to collaborate in chats for newly created meetings within Microsoft Teams
ID TM237264
Service Microsoft Teams
Feature Teams Components
Status False positive
Impact Users may have been unable to collaborate in chats for newly created meetings within Microsoft Teams.
Severity
Start time 2021-01-29T20:00:00Z
End time 2021-02-01T20:15:00Z
Last update 2021-02-01T22:03:17.263Z
More info As a workaround, users could have created new meetings and edit them using Outlook on the web. Final status: We've completed reverting the affecting build, and have confirmed that impact is mitigated. Scope of impact: This issue affected internal Microsoft users only. Start time: Friday, January 29, 2021, at 8:00 PM'UTC End time: Monday, February 1, 2021, at 8:15 PM'UTC Root cause: A recently deployed build introduced a code error, resulting in impact. Next steps: - We're reviewing the affecting build to determine how impact was missed during testing and validation to better prevent similar future occurrences. This is the final update for the event.; PublishedTime=2021-02-01T21:36:10.22Z
Root cause A recently deployed build introduced a code error, resulting in impact. Next steps: - We're reviewing the affecting build to determine how impact was missed during testing and validation to better prevent similar future occurrences. This is the final update for the event.; PublishedTime=2021-02-01T21:36:10.22Z

TM237264 Details

Status: We're analyzing diagnostic data to determine the source of the issue. Next update by: Monday, February 1, 2021, at 10:00 AM UTC; PublishedTime=2021-02-01T08:00:06.533Z

Status: We've determined that a recently deployed build is causing impact. We're analyzing Application Programming Interface (API) logs to isolate the specific problem with the build and help develop a mitigation plan. Next update by: Monday, February 1, 2021, at 5:00 PM UTC; PublishedTime=2021-02-01T09:11:41.233Z

Status: We've identified a code error within the recently deployed build. We're reverting the build to mitigate impact, and expect it to complete within the next six hours. Next update by: Monday, February 1, 2021, at 10:00 PM UTC; PublishedTime=2021-02-01T15:52:57.807Z

Final status: We've completed reverting the affecting build, and have confirmed that impact is mitigated. Next update by: Monday, February 1, 2021, at 10:00 PM UTC; PublishedTime=2021-02-01T15:52:57.807Z

--------------------------------------------------------------------------------------------------
Title Admins may notice the following reports are not updated to show usage data after Jan 25, 2021 in Office 365 admin center
ID MO236561
Service Microsoft 365 suite
Feature Administration
Status Service restored
Impact Admins noticed the following reports weren't updated to show usage data after Jan 25, 2021 in Office 365 admin center.
Severity
Start time 2021-01-29T00:00:00Z
End time 2021-01-31T18:00:00Z
Last update 2021-02-01T03:18:17.983Z
More info Admins may have noticed the following reports weren't updated: Office 365 Active users report Office 365 groups activity report Yammer user activity report Yammer device usage report Yammer groups activity report Final status: We've finished deploying the fix to the affected infrastructure and confirmed via telemetry that service is restored. Scope of impact: Impact was specific to admins who were served through the infrastructure. Start time: Friday, January 29, 2021, at 12:00 AM'UTC End time: Sunday, January 31, 2021, at 6:00 PM'UTC Root cause: An issue from the upstream data pipeline caused a delay in reports being uploaded. Next steps: - To help prevent similar impact in the future, we're further investigating the underlying cause of the upstream data pipeline issue. This is the final update for the event.; PublishedTime=2021-02-01T03:18:17.983Z
Root cause An issue from the upstream data pipeline caused a delay in reports being uploaded. Next steps: - To help prevent similar impact in the future, we're further investigating the underlying cause of the upstream data pipeline issue. This is the final update for the event.; PublishedTime=2021-02-01T03:18:17.983Z

MO236561 Details

Status: We've identified that there is an issue from the upstream data pipeline, which is causing a delay in reports being uploaded. We're developing a fix for testing before deploying to the affected infrastructure. Next update by: Tuesday, February 2, 2021, at 4:00 AM UTC; PublishedTime=2021-01-29T09:37:08Z

Final status: We've finished deploying the fix to the affected infrastructure and confirmed via telemetry that service is restored. Next update by: Tuesday, February 2, 2021, at 4:00 AM UTC; PublishedTime=2021-01-29T09:37:08Z

--------------------------------------------------------------------------------------------------
Title Admins may notice the following reports are not updated to show usage data after Jan 25, 2021 in Office 365 admin center
ID MO236561
Service Microsoft 365 suite
Feature Administration
Status Service restored
Impact Admins noticed the following reports weren't updated to show usage data after Jan 25, 2021 in Office 365 admin center.
Severity
Start time 2021-01-29T00:00:00Z
End time 2021-01-31T18:00:00Z
Last update 2021-02-01T03:18:17.983Z
More info Admins may have noticed the following reports weren't updated: Office 365 Active users report Office 365 groups activity report Yammer user activity report Yammer device usage report Yammer groups activity report Final status: We've finished deploying the fix to the affected infrastructure and confirmed via telemetry that service is restored. Scope of impact: Impact was specific to admins who were served through the infrastructure. Start time: Friday, January 29, 2021, at 12:00 AM'UTC End time: Sunday, January 31, 2021, at 6:00 PM'UTC Root cause: An issue from the upstream data pipeline caused a delay in reports being uploaded. Next steps: - To help prevent similar impact in the future, we're further investigating the underlying cause of the upstream data pipeline issue. This is the final update for the event.; PublishedTime=2021-02-01T03:18:17.983Z
Root cause An issue from the upstream data pipeline caused a delay in reports being uploaded. Next steps: - To help prevent similar impact in the future, we're further investigating the underlying cause of the upstream data pipeline issue. This is the final update for the event.; PublishedTime=2021-02-01T03:18:17.983Z

MO236561 Details

Status: We've identified that there is an issue from the upstream data pipeline, which is causing a delay in reports being uploaded. We're developing a fix for testing before deploying to the affected infrastructure. Next update by: Tuesday, February 2, 2021, at 4:00 AM UTC; PublishedTime=2021-01-29T09:37:08Z

Final status: We've finished deploying the fix to the affected infrastructure and confirmed via telemetry that service is restored. Next update by: Tuesday, February 2, 2021, at 4:00 AM UTC; PublishedTime=2021-01-29T09:37:08Z

--------------------------------------------------------------------------------------------------
Title Users may be experiencing Forms becoming blocked and unavailable within Microsoft Forms
ID FM236458
Service Microsoft Forms
Feature Service
Status Service restored
Impact Users may have experienced Forms becoming blocked within the Microsoft Forms service.
Severity
Start time 2021-01-28T05:11:00Z
End time 2021-01-28T19:16:07.73Z
Last update 2021-01-28T19:16:07.73Z
More info
Root cause A recent service update introduced a configuration issue which resulted in impact. Next steps: - We're reviewing our update procedures to determine why the configuration issue occurred and to better identify similar issues during our development and testing cycles. This is the final update for the event.; PublishedTime=2021-01-28T19:16:07.73Z

FM236458 Details

Final status: We've identified a recent service update that was affecting your organization and preventing users from submitting Forms within the Microsoft Forms service. We've reverted the offending service update, which will prevent new Forms from becoming blocked. To unblock previous affected Forms, Forms owners will need to update the affected Form with a new title, which will remediate impact.

--------------------------------------------------------------------------------------------------
Title Users are unable to see new messages being displayed in Outlook on the web while using Internet Explorer 11 (IE11)
ID EX237403
Service Exchange Online
Feature E-Mail timely delivery
Status Service restored
Impact Users were unable to see new messages being displayed in Outlook on the web while using IE11.
Severity
Start time 2021-01-28T04:15:00Z
End time 2021-02-02T21:00:00Z
Last update 2021-02-02T23:34:14.397Z
More info This issue was specifically occurring to some users who were accessing Outlook on the web using IE11. As an alternate method, users were encouraged to use a different web browser to access new messages being displayed. As an additional workaround, users may have experienced remediation if they moved between folders or manually refreshed the page. Final status: We've verified that the fix has completed deploying to all affected infrastructure and confirmed mitigation with affected users. Scope of impact: Any user accessing Outlook on the web via IE11 may have been intermittently impacted. Start time: Thursday, January 28, 2021, at 4:15 AM'UTC End time: Tuesday, February 2, 2021, at 9:00 PM'UTC Root cause: A recent service update had exposed an incompatibility between Exchange Online and the IE11 browser. Next steps: - We're reviewing our service update procedures to better identify similar issues with IE11 during our development and testing cycles. This is the final update for the event.; PublishedTime=2021-02-02T23:34:14.397Z
Root cause A recent service update had exposed an incompatibility between Exchange Online and the IE11 browser. Next steps: - We're reviewing our service update procedures to better identify similar issues with IE11 during our development and testing cycles. This is the final update for the event.; PublishedTime=2021-02-02T23:34:14.397Z

EX237403 Details

Status: We're reviewing internal telemetry to gain a better understanding of why impacts occurring. Next update by: Tuesday, February 2, 2021, at 10:00 AMUTC; PublishedTime=2021-02-02T07:48:10.663Z

Status: We're continuing to review system telemetry and recent updates to identify the source of the issue and develop a mitigation plan. Next update by: Tuesday, February 2, 2021, at 8:00 PM UTC; PublishedTime=2021-02-02T09:51:17.423Z

Status: Following our system telemetry review, we've identified a recent service update that has exposed an incompatibility between certain functionality within the Exchange Online service and the affected Internet Explorer version. We've developed and are deploying a fix, and expect to be able to provide an estimate for the remediation of impact by our next scheduled update. Next update by: Wednesday, February 3, 2021, at 1:00 AMUTC; PublishedTime=2021-02-02T18:50:15.243Z

Final status: We've verified that the fix has completed deploying to all affected infrastructure and confirmed mitigation with affected users. Next update by: Wednesday, February 3, 2021, at 1:00 AMUTC; PublishedTime=2021-02-02T18:50:15.243Z

--------------------------------------------------------------------------------------------------
Title Users are unable to publish using the contextual publish button in the SharePoint Online modern experience
ID SP241278
Service SharePoint Online
Feature SharePoint Features
Status Post-incident report published
Impact Users were unable to publish using the contextual publish button in the SharePoint Online modern experience.
Severity
Start time 2021-01-27T19:00:00Z
End time 2021-02-23T22:00:00Z
Last update 2021-03-02T13:53:46.917Z
More info While the publish button was unavailable, users could have used the following workarounds: - Users were able to use SharePoint Online classic experience - Users could publish using the 'Republish' button on the top navigation bar Final status: We deployed the fix to the affected environment and have validated that service has been restored. Scope of impact: Impact was specific to users using the SharePoint Online modern experience. Start time: Wednesday, January 27, 2021, at 7:00 PM UTC End time: Tuesday, February 23, 2021, at 10:00 PM UTC Root cause: An update intended to decrease the page loading time in SharePoint Online contained a code bug that mistakenly removed the publish button functionality. Next steps: - For a more comprehensive list of next steps and actions, please refer to the Post Incident Review document. We'll publish a post-incident report within five business days.; PublishedTime=2021-02-23T23:05:20.937Z
Root cause An update intended to decrease the page loading time in SharePoint Online contained a code bug that mistakenly removed the publish button functionality. Next steps: - For a more comprehensive list of next steps and actions, please refer to the Post Incident Review document. We'll publish a post-incident report within five business days.; PublishedTime=2021-02-23T23:05:20.937Z

SP241278 Details

Status: We're reviewing system telemetry to identify the source of the issue. Next update by: Tuesday, February 23, 2021, at 5:30 PM UTC; PublishedTime=2021-02-23T15:25:39.65Z

Status: We've determined that an update intended to decrease the page loading speed in SharePoint Online contained a code bug that mistakenly removes the publish button functionality. We're creating a fix to return the publish button's functionality and remediate impact. Next update by: Tuesday, February 23, 2021, at 11:30 PMUTC; PublishedTime=2021-02-23T17:32:22.913Z

Final status: We deployed the fix to the affected environment and have validated that service has been restored. Next update by: Tuesday, February 23, 2021, at 11:30 PMUTC; PublishedTime=2021-02-23T17:32:22.913Z

--------------------------------------------------------------------------------------------------
Title Some admins intermittently can't run PowerShell cmdlets in Exchange Online
ID EX238740
Service Exchange Online
Feature Management and Provisioning
Status Service restored
Impact Admins may have been intermittently unable to run PowerShell cmdlets within the Exchange Online service.
Severity
Start time 2021-01-27T14:28:00Z
End time 2021-02-15T16:48:00Z
Last update 2021-02-16T17:34:29.073Z
More info Affected admins would intermittently encounter domain controller overloaded error messages and were able to start a new PowerShell session as a workaround until mitigation was achieved. Final status: Following our adjustment to the code logic for a subsection of affected domain controllers, we've confirmed that impact has been resolved. Scope of impact: A small portion of admins who were served through the affected infrastructure may have experienced intermittent impact. Start time: Wednesday, January 27, 2021, at 2:28 PM'UTC End time: Monday, February 15, 2021, at 4:48 PM'UTC Root cause: Processing inefficiencies as well as code problems were preventing a subset of domain controllers from processing retry requests as expected, which intermittently prevented admins from running PowerShell cmdlets. Next steps: - We're reviewing our domain controller processing procedures and code for improved performance and to find ways to prevent this problem from happening again. This is the final update for the event.; PublishedTime=2021-02-16T17:34:29.073Z
Root cause Processing inefficiencies as well as code problems were preventing a subset of domain controllers from processing retry requests as expected, which intermittently prevented admins from running PowerShell cmdlets. Next steps: - We're reviewing our domain controller processing procedures and code for improved performance and to find ways to prevent this problem from happening again. This is the final update for the event.; PublishedTime=2021-02-16T17:34:29.073Z

EX238740 Details

Status: We're investigating a potential issue and checking for impact to your organization. We'll provide an update within 30 minutes.

Status: We're reviewing the available diagnostic data and system telemetry to isolate the source of the issue and determine our next troubleshooting steps. Next update by: Tuesday, February 9, 2021, at 6:30 PMUTC; PublishedTime=2021-02-09T17:13:14.533Z

Status: We're continuing our analysis of the system telemetry corresponding with the affected Domain Controllers to isolate the cause of the errors and develop a mitigation strategy. Next update by: Tuesday, February 9, 2021, at 8:30 PMUTC; PublishedTime=2021-02-09T18:06:49.137Z

Status: Our analysis of the Domain Controller telemetry is ongoing in an effort to identify the underlying cause of the errors. In parallel, we're rerouting a portion of the traffic to an alternate infrastructure to potentially rebalance the load and provide some relief as we work to identify the most effective mitigation strategy. Next update by: Tuesday, February 9, 2021, at 10:30 PMUTC; PublishedTime=2021-02-09T20:23:37.3Z

Status: Our analysis of the Domain Controller telemetry has identified an issue with the cmdlet infrastructure which is preventing requests from completing as expected, leading to impact. We're developing a code fix for this issue and will provide an estimated time to completion when available. Next update by: Thursday, February 11, 2021, at 8:00 PMUTC; PublishedTime=2021-02-09T22:10:34.36Z

Status: We're still in the process of developing the code fix for this issue and will provide and an estimated time of completion at our next scheduled update. Next update by: Thursday, February 11, 2021, at 10:00 PMUTC; PublishedTime=2021-02-11T18:56:01.927Z

Status: We've determined that our earlier communicated code fix will not be an acceptable method for remediating impact. Due to the complexity of this issue, our remediation plan will require additional time to complete. We'll provide an estimate for the remediation of impact once we've confirmed a viable solution. Next update by: Friday, February 12, 2021, at 7:30 PMUTC; PublishedTime=2021-02-11T21:26:09.023Z

Status: After further investigation, we've determined that processing inefficiencies as well as code problems are preventing the affected domain controllers from processing retry requests as expected and causing the problem. We're looking into methods to improve the code logic of the domain controllers and mitigate impact. Next update by: Tuesday, February 16, 2021, at 7:00 PMUTC; PublishedTime=2021-02-12T18:55:29.737Z

Final status: Following our adjustment to the code logic for a subsection of affected domain controllers, we've confirmed that impact has been resolved. Next update by: Tuesday, February 16, 2021, at 7:00 PMUTC; PublishedTime=2021-02-12T18:55:29.737Z

--------------------------------------------------------------------------------------------------
Title Admins are unable to import data to a review set or complete Analytics jobs in Advanced eDiscovery
ID EX236144
Service Exchange Online
Feature Management and Provisioning
Status Service restored
Impact Admins may have seen data imports to a review set or Analytics jobs stuck at 0 percent in Advanced eDiscovery.
Severity
Start time 2021-01-25T21:30:00Z
End time 2021-01-26T21:24:48.04Z
Last update 2021-01-26T22:01:14.687Z
More info When retying jobs, admins may need to select modern attachments for the jobs to run successfully. Final status: We confirmed that maintenance to clear a deprecated portion of service had inadvertently removed a storage account that some Advanced eDiscovery functionality was reliant upon, resulting in impact. We've redirected the affected requests through an actively available storage account and confirmed via telemetry that this issue is resolved. Scope of impact: Any admin may have experienced impact. Start time: Monday, January 25, 2021, at 9:30 PM UTC End time: Tuesday, January 26, 2021, at 7:30 PM UTC Root cause: - Maintenance to clear a deprecated portion of service had inadvertently removed a storage account that some Advanced eDiscovery functionality was reliant upon, resulting in impact. Next steps: - We're reviewing our service deprecation cleanup and removal process to prevent issues like this from occurring in the future. This is the final update for the event.; PublishedTime=2021-01-26T21:24:48.04Z
Root cause - Maintenance to clear a deprecated portion of service had inadvertently removed a storage account that some Advanced eDiscovery functionality was reliant upon, resulting in impact. Next steps: - We're reviewing our service deprecation cleanup and removal process to prevent issues like this from occurring in the future. This is the final update for the event.; PublishedTime=2021-01-26T21:24:48.04Z

EX236144 Details

Final status: We confirmed that maintenance to clear a deprecated portion of service had inadvertently removed a storage account that some Advanced eDiscovery functionality was reliant upon, resulting in impact. We've redirected the affected requests through an actively available storage account and confirmed via telemetry that this issue is resolved.

--------------------------------------------------------------------------------------------------
Title Multi-geo admins can't view details in Mailbox Usage tab in Exchange Control Panel (ECP)
ID EX236297
Service Exchange Online
Feature Management and Provisioning
Status Service restored
Impact Admins may have been unable to view details in the Mailbox Usage tab of the ECP.
Severity
Start time 2021-01-25T13:38:00Z
End time 2021-03-29T15:30:00Z
Last update 2021-03-29T15:50:24.577Z
More info If accessing the mailbox usage of a user in a region that was different from the signed-in admin, please check whether the IsEncrypted attribute was 'True' by using the PowerShell command: $mbx = Get-MailboxStatistics -Identity $mbx.IsEncrypted If this value was set, the workaround was to execute the Get-MailboxStatistics command in a remote PowerShell session with the User Principal Name (UPN) of the user as the identity. Final status: Following the deployment of our code fix, we've confirmed that impact has been remediated. Scope of impact: Any multi-geo admin attempting to use the Mailbox Usage tab of the ECP may have experienced impact. Start time: Monday, January 25, 2021, at 1:38 PM UTC End time: Monday, March 29, 2021, at 3:30 PM UTC Root cause: There was an issue with deserialization of data encryption details when querying mailbox usage data of encrypted mailboxes from other regions, resulting in multi-geo admins being unable to view details in the Mailbox Usage tab of the ECP. Next steps: - We're reviewing our code for improved performance and potential automated recovery options for reducing similar impact in the future. This is the final update for the event.; PublishedTime=2021-03-29T15:50:24.577Z
Root cause There was an issue with deserialization of data encryption details when querying mailbox usage data of encrypted mailboxes from other regions, resulting in multi-geo admins being unable to view details in the Mailbox Usage tab of the ECP. Next steps: - We're reviewing our code for improved performance and potential automated recovery options for reducing similar impact in the future. This is the final update for the event.; PublishedTime=2021-03-29T15:50:24.577Z

EX236297 Details

Status: We're investigating a potential issue and checking for impact to your organization. We'll provide an update within 30 minutes.

Status: We're currently reviewing system logs to pinpoint the source of impact and develop our next troubleshooting steps. Next update by: Wednesday, January 27, 2021, at 7:30 PMUTC; PublishedTime=2021-01-27T17:35:24.433Z

Status: We're reviewing session capture logs recently provided by affected users to further identify the underlying reason this problem is happening. Next update by: Wednesday, January 27, 2021, at 9:30 PMUTC; PublishedTime=2021-01-27T18:55:11.613Z

Status: We've reached out to affected users in an effort to generate mailbox usage content via PowerShell, which failed. We're reviewing session capture logs to determine our next steps. Next update by: Wednesday, January 27, 2021, at 11:30 PMUTC; PublishedTime=2021-01-27T21:09:04.66Z

Status: We suspect that a cross-geographical communication issue is occurring within the Exchange Online environment when admins attempt to view mailbox usage statistics from the ECP. We're working to isolate what's causing the communication problem so that we can identify remediation steps. Next update by: Thursday, January 28, 2021, at 1:00 AMUTC; PublishedTime=2021-01-27T23:21:44.913Z

Status: We're reviewing error logs associated with admin ECP request failures for the usage data to confirm the suspected root cause of this issue, and then evaluate mitigation options. Next update by: Thursday, January 28, 2021, at 4:00 PMUTC; PublishedTime=2021-01-28T00:18:12.597Z

Status: We require additional data from affected users, such as network trace logs and sample user logs to help us narrow down the root cause of impact. We're working with a subset of them to get this information. Next update by: Thursday, January 28, 2021, at 6:00 PM UTC; PublishedTime=2021-01-28T15:21:03.073Z

Status: We've determined that there is an issue with deserialization of data encryption details when querying mailbox usage data of encrypted mailboxes from other regions, resulting in multi-geo admins being unable to view details in the Mailbox Usage tab of the ECP. We're deploying a code fix that we expect will fully saturate in one to two months, remediating impact. Next update by: Thursday, February 4, 2021, at 6:00 PM UTC; PublishedTime=2021-01-28T18:09:38.487Z

Status: We've started to deploy the code fix to the affected environments and currently suspect that it will fully saturate by approximately Wednesday, March 31, 2021. We're continuing to monitor the fix as it deploys to ensure it completes successfully, and will provide updated estimates to completion as we have them. Next update by: Thursday, February 18, 2021, at 6:00 PM UTC; PublishedTime=2021-02-04T16:15:35.327Z

Status: We're continuing to monitor the code fix to ensure it saturates throughout the affected environment successfully. Based on current deployment progress, we suspect that the fix will likely complete before our targeted date of Wednesday, March 31, 2021; however, we're assessing further before providing an updated estimate to completion. Next update by: Thursday, February 25, 2021, at 6:00 PM UTC; PublishedTime=2021-02-18T16:43:47.537Z

Status: We're continuing to monitor the code fix to ensure it saturates throughout the affected environment. Based on current telemetry, we expect that the fix will likely complete early next week. Next update by: Monday, March 1, 2021, at 7:00 PMUTC; PublishedTime=2021-02-25T02:25:56.563Z

Status: The fix is taking longer than expected, and is currently 76 percent complete. We now expect the deployment to complete by the next scheduled update. Next update by: Monday, March 15, 2021, at 7:00 PM UTC; PublishedTime=2021-03-01T17:02:15.133Z

Status: We determined that our initial fix wasn't functioning to de-serialize all of the affected data encryption details when querying mailbox usage data of encrypted mailboxes from other regions in certain circumstances. We've adjusted our code fix and restarted the deployment, which has reached our initial testing environment successfully. Deployment of the updated fix is being initiated worldwide, and we expect for the fix to reach all affected users by the end of the month, on Wednesday, March 31, 2021. Next update by: Monday, March 29, 2021, at 7:00 PM UTC; PublishedTime=2021-03-15T16:59:53.38Z

Final status: Following the deployment of our code fix, we've confirmed that impact has been remediated. Next update by: Monday, March 29, 2021, at 7:00 PM UTC; PublishedTime=2021-03-15T16:59:53.38Z

--------------------------------------------------------------------------------------------------
Title Some users that utilize the same username with work/school and personal Microsoft account can't launch Teams desktop app
ID TM236316
Service Microsoft Teams
Feature Teams Components
Status Post-incident report published
Impact Users may have been unable to launch the Microsoft Teams desktop client.
Severity
Start time 2021-01-25T10:12:00Z
End time 2021-02-03T18:00:00Z
Last update 2021-02-10T23:24:44.36Z
More info Users may have see a message that states, "Loading Microsoft Teams"; however, the client did not load. As a workaround, users had the option of mitigating impact by navigating to the teams.microsoft.com/downloads page, uninstalling the client, and redownloading the updated version. Additionally, users with the same email address for their work and personal account could have renamed their personal account by following the instructions described here: https://go.microsoft.com/fwlink/p/'LinkID=733247 Final status: We successfully implemented a service-side change that will automatically update affected users' current version of Teams and mitigate impact upon restarting the application. Furthermore, users in a bad state that had their Teams client running in the background have been automatically updated to the new version, restoring service functionality. The new Teams desktop client build remains in a testing state will be ready for deployment within the next few days. As this issue has been resolved for all affected users, further deployment efforts will be managed separately from this event. Scope of impact: Impact was specific to users that utilized the same username with work/school accounts and their personal Microsoft account. Start time: Monday, January 25, 2021, at 10:12 AM UTC End time: Wednesday, February 3, 2021, at 6:00 PM UTC Root cause: A recent update caused a misconfiguration within the infrastructure responsible for identifying user account type(s). This resulted in authentication failures for a subset of users that have both AAD and MSA associated to the same account and prevented the client from launching successfully. Next steps: - For a more comprehensive list of next steps and actions, please refer to the Post Incident Review document. We'll publish a post-incident report within five business days.; PublishedTime=2021-02-03T22:07:31.84Z
Root cause A recent update caused a misconfiguration within the infrastructure responsible for identifying user account type(s). This resulted in authentication failures for a subset of users that have both AAD and MSA associated to the same account and prevented the client from launching successfully. Next steps: - For a more comprehensive list of next steps and actions, please refer to the Post Incident Review document. We'll publish a post-incident report within five business days.; PublishedTime=2021-02-03T22:07:31.84Z

TM236316 Details

Status: We're investigating a potential issue and checking for impact to your organization. We'll provide an update within 30 minutes.; PublishedTime=2021-01-27T19:11:44.347

Status: We're reviewing system logs to isolate the origin of this issue. Next update by: Wednesday, January 27, 2021, at 9:00 PMUTC; PublishedTime=2021-01-27T19:27:59.757Z

Status: We've determined that a recently deployed update inadvertently introduced a code bug that causes authentication issues for a small subset of users, resulting in impact. We're reverting the update to mitigate impact. Next update by: Wednesday, January 27, 2021, at 11:00 PMUTC; PublishedTime=2021-01-27T20:42:23.077Z

Status: We're continuing the process of reverting the offending update to mitigate impact. Next update by: Thursday, January 28, 2021, at 1:00 AMUTC; PublishedTime=2021-01-27T22:56:29.95Z

Status: Our efforts to revert the affecting update continue, however, to safely mitigate impact and prevent additional issues, we're deploying our fix to revert the affected update in phases. We anticipate users may see relief as early as February 1, 2021. Next update by: Monday, February 1, 2021, at 7:00 PM UTC; PublishedTime=2021-01-28T01:04:10.9Z

Status: The deployment is continuing to progress as expected. In parallel, we're reviewing additional strategies to potentially expedite the deployment process. Next update by: Thursday, January 28, 2021, at 8:00 PM UTC; PublishedTime=2021-01-28T17:22:34.383Z

Status: We've developed additional mitigation plans to expedite user relief, and are in the process of determining and validating which strategy will be most effective. Next update by: Friday, January 29, 2021, at 12:00 AM UTC; PublishedTime=2021-01-28T19:41:45.71Z

Status: At this time, additional testing and validation is required to ensure the fix fully mitigates the issue, while also ensuring it doesn't introduce additional impact. Next update by: Friday, January 29, 2021, at 8:00 PM UTC; PublishedTime=2021-01-28T23:27:13.957Z

Status: Our short-term mitigation and long-term remediation efforts are progressing as intended. To prevent further impact from occurring in new downloads of the Teams desktop client, we've updated the Teams download page with a prior build to prevent further recurrence of this issue for any future downloads. As a longer-term fix, we are currently testing and validating the effectiveness of an updated build. This process is progressing as expected and we estimate that the new build will be ready for deployment early next week. In parallel, we're continuing to develop and test an additional fix that will update users' current version, as well as automatically remediate the issue for any user that is in an impacted state. While this work is underway, users with the same email address for their work and personal accounts have the option of mitigating impact by navigating to the teams.microsoft.com/downloads page, uninstalling the client, and redownloading the updated version. Next update by: Monday, February 1, 2021, at 11:00 PM UTC; PublishedTime=2021-01-29T19:52:10.187Z

Status: We've successfully deployed several short-term fixes to the Teams service that prevent users from encountering the issue described in this incident. In parallel, our long-term fix is progressing as intended and we estimate that the new Teams desktop client build will be ready for deployment in the next few days. Next update by: Tuesday, February 2, 2021, at 11:30 PM UTC; PublishedTime=2021-02-01T23:03:27.943Z

Status: Our long-term fixes are continuing to be tested and validated. We estimate that the new Teams desktop client build will be ready for deployment within the next few days. In tandem with this deployment, we plan to implement a service side change that will automatically update users' current version, remediating the issue for any user that is in an impacted state. Next update by: Wednesday, February 3, 2021, at 11:30 PM UTC; PublishedTime=2021-02-02T17:54:50.9Z

Final status: We successfully implemented a service-side change that will automatically update affected users' current version of Teams and mitigate impact upon restarting the application. Furthermore, users in a bad state that had their Teams client running in the background have been automatically updated to the new version, restoring service functionality. The new Teams desktop client build remains in a testing state will be ready for deployment within the next few days. As this issue has been resolved for all affected users, further deployment efforts will be managed separately from this event. Next update by: Wednesday, February 3, 2021, at 11:30 PM UTC; PublishedTime=2021-02-02T17:54:50.9Z

--------------------------------------------------------------------------------------------------
Title Some admins may be unable to create new shared mailboxes within the Exchange Online service
ID EX235961
Service Exchange Online
Feature Management and Provisioning
Status Service restored
Impact Admins may have been unable to create new shared mailboxes within the Exchange Online service.
Severity
Start time 2021-01-25T01:32:00Z
End time 2021-01-25T14:35:00Z
Last update 2021-01-26T04:01:51.257Z
More info Admins may have also experienced issues in managing or creating distribution groups. Users that were recently migrated to the Exchange Online service may have been unable to access their mailboxes. Final status: We've confirmed that a manual sync of directory and service data has completed, and, with the flight successfully reverted, admins will no longer see impact. Scope of impact: Impact was specific to admins who were served through the affected infrastructure, as well as recently migrated users. Start time: Monday, January 25, 2021, at 1:32 AM'UTC End time: Monday, January 25, 2021, at 2:35 PM'UTC Root cause: A recent configuration change hadn't saturated the affected environment before the deployment of the function to use the new configuration change had saturated, resulting in a misconfiguration that lead to impact. Next steps: - We're reviewing our validation and deployment procedures to better prevent similar impact from occurring. This is the final update for the event.; PublishedTime=2021-01-26T04:01:51.257Z
Root cause A recent configuration change hadn't saturated the affected environment before the deployment of the function to use the new configuration change had saturated, resulting in a misconfiguration that lead to impact. Next steps: - We're reviewing our validation and deployment procedures to better prevent similar impact from occurring. This is the final update for the event.; PublishedTime=2021-01-26T04:01:51.257Z

EX235961 Details

Status: We're investigating a potential issue with the Exchange Online service. We'll provide an update within 30 minutes. ; PublishedTime=2021-01-25T12:05:05.663

Status: We've determined that a recent configuration change has not fully saturated to the affected environment as expected. We're reverting this change to mitigate impact. Next update by: Monday, January 25, 2021, at 7:00 PM UTC; PublishedTime=2021-01-25T12:33:41.667Z

Status: We're monitoring the progress of the flight as it's being reverted. We expect it to be completed within the next 6 hours. Next update by: Monday, January 25, 2021, at 7:00 PM UTC; PublishedTime=2021-01-25T12:56:17.837Z

Status: The flight has been successfully reverted, and we've confirmed with some of impacted admins that this has resolved the issue after a manual sync of the data between the directory and the service had been completed. Our investigation further determined that this sync is required to fully mitigate the impact and that it will complete automatically within eight hours. Affected admins will experience relief as the sync completes incrementally, and we expect to be able to confirm mitigation for all admins by the next scheduled update. Additionally, admins that require an immediate mitigation can contact support to request that we perform a manual sync. Next update by: Tuesday, January 26, 2021, at 5:30 AM UTC; PublishedTime=2021-01-25T17:02:40.907Z

Final status: We've confirmed that a manual sync of directory and service data has completed, and, with the flight successfully reverted, admins will no longer see impact. Next update by: Tuesday, January 26, 2021, at 5:30 AM UTC; PublishedTime=2021-01-25T17:02:40.907Z

--------------------------------------------------------------------------------------------------
Title Users can't access SharePoint Online documents in Microsoft Teams
ID TM238743
Service Microsoft Teams
Feature Teams Components
Status Service restored
Impact Users were experiencing error messages when attempting to open documents within their SharePoint tab in Microsoft Teams.
Severity
Start time 2021-01-25T00:22:00Z
End time 2021-02-10T19:40:00Z
Last update 2021-02-10T20:19:12.453Z
More info Affected users may have seen the following error messages: "An unknown error occurred" or "This item might not exist or is no longer available" Final status: Following the deployment of our fix and testing with previously affected users, we've confirmed that impact has been remediated. Scope of impact: Any user attempting to open items through the SharePoint Online document library tab in Microsoft Teams may have been impacted. Start time: Monday, January 25, 2021, at 12:22 AM'UTC End time: Wednesday, February 10, 2021, at 7:40 PM'UTC Root cause: A recent service update contained a code issue, resulting in impact. Next steps: - We're reviewing our update procedures to better understand the introduced code issue and determine why it was not identified during our development and testing cycles. This is the final update for the event.; PublishedTime=2021-02-10T20:19:12.453Z
Root cause A recent service update contained a code issue, resulting in impact. Next steps: - We're reviewing our update procedures to better understand the introduced code issue and determine why it was not identified during our development and testing cycles. This is the final update for the event.; PublishedTime=2021-02-10T20:19:12.453Z

TM238743 Details

Status: We're investigating a potential issue and checking for impact to your organization. We'll provide an update within 30 minutes.; PublishedTime=2021-02-09T17:43:21.12

Status: We're reviewing networking traces provided by affected users to isolate the underlying cause of impact. Next update by: Tuesday, February 9, 2021, at 7:30 PMUTC; PublishedTime=2021-02-09T18:01:48.637Z

Status: Our review of network traces, in addition to Unified Logging Service (ULS) logs, has confirmed that a recent service update contained a code issue, resulting in impact. We've developed and validated a fix which is currently progressing under our standard deployment procedures, and we're investigating potential methods to expedite the fix. Next update by: Wednesday, February 10, 2021, at 7:30 PMUTC; PublishedTime=2021-02-09T19:04:08.003Z

Status: We're continuing to deploy the fix, which we expect will complete and remediate impact on Wednesday, February 10, 2021, at 8:00 PM UTC. Next update by: Wednesday, February 10, 2021, at 10:00 PMUTC; PublishedTime=2021-02-10T18:56:42.09Z

Final status: Following the deployment of our fix and testing with previously affected users, we've confirmed that impact has been remediated. Next update by: Wednesday, February 10, 2021, at 10:00 PMUTC; PublishedTime=2021-02-10T18:56:42.09Z

--------------------------------------------------------------------------------------------------
Title Users seeing Microsoft Confidential footer appearing unexpectedly when searching modern lists and libraries
ID SP237746
Service SharePoint Online
Feature Search and Delve
Status Service restored
Impact Users may have seen a Microsoft Confidential footer appearing unexpectedly when searching modern lists and libraries.
Severity
Start time 2021-01-24T08:00:00Z
End time 2021-02-09T02:45:00Z
Last update 2021-02-09T03:17:06.457Z
More info
Root cause A recent update intended to provide performance optimizations contained a code regression that was causing conflicts with searching in modern lists and libraries. Next steps: -We're reviewing code regression to gain a better understanding of why impact happened, and to prevent any similar issues from reoccurring. This is the final update for the event.; PublishedTime=2021-02-09T03:17:06.457Z

SP237746 Details

Status: We're investigating a potential issue and checking for impact to your organization. We'll provide an update within 30 minutes.

Status: We're reviewing the available diagnostic data to determine our next troubleshooting steps. We're also attempting to reproduce the issue internally to help us better understand the nature of the impact. Next update by: Thursday, February 4, 2021, at 9:00 PMUTC; PublishedTime=2021-02-04T19:52:21.943Z

Status: We're currently analyzing diagnostic logs and Fiddler network traces gathered from our internal reproduction of the issue. We're also investigating any recent service changes to help us isolate the potential source of this issue. Next update by: Thursday, February 4, 2021, at 11:00 PMUTC; PublishedTime=2021-02-04T20:44:24.7Z

Status: We're continuing to analyze the available diagnostic logs and service telemetry as well as any recent service changes which may be causing the unexpected footer to appear. Next update by: Friday, February 5, 2021, at 1:00 AMUTC; PublishedTime=2021-02-04T22:37:37.85Z

Status: We've identified a code regression within a recent service update that is causing conflicts when searching modern lists and libraries. We're validating a code fix in order to mitigate impact to your organization. Next update by: Friday, February 5, 2021, at 7:30 PMUTC; PublishedTime=2021-02-05T00:58:16.733Z

Status: We're continuing our development of a code fix, while in parallel looking into potentially reverting the impacting update to determine which would mitigate impact earlier. Next update by: Saturday, February 6, 2021, at 2:30 AMUTC; PublishedTime=2021-02-05T19:06:44.427Z

Status: We've concluded that reverting the affecting update will not be the fastest way to mitigate impact, and are continuing our development of a fix. We expect to be able to provide an ETA for deployment with our next update. Next update by: Tuesday, February 9, 2021, at 2:30 AMUTC; PublishedTime=2021-02-06T00:55:11.013Z

Status: We've completed development of our fix, which has undergone validation and testing. We're preparing to deploy the fix, and expect it to saturate the affected environment and the issue to be resolved by our next communications update. Next update by: Tuesday, February 9, 2021, at 4:30 AMUTC; PublishedTime=2021-02-09T02:22:54.643Z

Final status: We've deployed the fix to the affected environment and confirmed that the issue is resolved. Next update by: Tuesday, February 9, 2021, at 4:30 AMUTC; PublishedTime=2021-02-09T02:22:54.643Z

--------------------------------------------------------------------------------------------------
Title Users experiencing delays of up to an hour exporting Microsoft Teams meeting recordings to SharePoint and OneDrive
ID TM235434
Service Microsoft Teams
Feature Teams Components
Status Service restored
Impact Users may have experienced delays of an hour exporting Microsoft Teams meeting recordings to SharePoint and OneDrive.
Severity
Start time 2021-01-22T09:00:00Z
End time 2021-01-22T18:00:00Z
Last update 2021-01-22T18:47:01.403Z
More info This affected users that had meeting recordings enabled to export to SharePoint Online and OneDrive for Business only. Users who had meeting recordings set to upload to Microsoft Stream were not impacted. Final status: We've completed our optimizations on the affected infrastructure and confirmed via our telemetry that the issue has been resolved. Scope of impact: Impact was specific to a subset of users who were served through the affected infrastructure. Start time: Friday, January 22, 2021, at 9:00 AM'UTC End time: Friday, January 22, 2021, at 6:00 PM'UTC Root cause: A portion of infrastructure responsible for processing Microsoft Teams recording exports was processing traffic inefficiently, resulting in delays exporting recordings to SharePoint Online and OneDrive for Business. Next steps: - We're analyzing performance data and trends on the affected systems to better understand why traffic was processing inefficiently so that we can help prevent this problem from happening again. This is the final update for the event.; PublishedTime=2021-01-22T18:47:01.403Z
Root cause A portion of infrastructure responsible for processing Microsoft Teams recording exports was processing traffic inefficiently, resulting in delays exporting recordings to SharePoint Online and OneDrive for Business. Next steps: - We're analyzing performance data and trends on the affected systems to better understand why traffic was processing inefficiently so that we can help prevent this problem from happening again. This is the final update for the event.; PublishedTime=2021-01-22T18:47:01.403Z

TM235434 Details

Status: We're investigating a potential issue and checking for impact to your organization. We'll provide an update within 30 minutes.; PublishedTime=2021-01-22T17:36:21.61

Status: Our investigation determined that a portion of infrastructure responsible for processing Microsoft Teams recording exports is processing traffic inefficiently, resulting in delays exporting recordings. We're implementing traffic processing optimizations on the affected infrastructure to remediate impact. Next update by: Friday, January 22, 2021, at 7:30 PMUTC; PublishedTime=2021-01-22T18:03:53.803Z

Final status: We've completed our optimizations on the affected infrastructure and confirmed via our telemetry that the issue has been resolved. Next update by: Friday, January 22, 2021, at 7:30 PMUTC; PublishedTime=2021-01-22T18:03:53.803Z

--------------------------------------------------------------------------------------------------
Title Users can't purchase services in the Microsoft 365 admin center as the page renders blank
ID MO235448
Service Microsoft 365 suite
Feature Portal
Status Service restored
Impact Users may have been unable to purchase services in the Microsoft 365 admin center as the page renders blank.
Severity
Start time 2021-01-21T23:25:00Z
End time 2021-01-22T18:50:00Z
Last update 2021-01-22T19:24:29.28Z
More info Users see impact when attempting to purchase services within the Billing section of the Microsoft 365 admin center. Final status: We've determined that a recent update to IP restrictions inadvertently resulted in impact to the Microsoft 365 admin center. We've reverted the affecting update, and confirmed that the admin center page now renders as expected, mitigating impact. Scope of impact: Users attempting to access the Billing section of the Microsoft 365 admin center to purchase services may have experienced impact. Start time: Thursday, January 21, 2021, at 11:25 PM'UTC End time: Friday, January 22, 2021, at 6:50 PM'UTC Root cause: A recent update to IP restrictions inadvertently resulted in impact to the Microsoft 365 admin center. Next steps: - We're reviewing the affecting update to identify how impact was missed during testing and development to better prevent similar future impact. This is the final update for the event.; PublishedTime=2021-01-22T19:23:51.87Z
Root cause A recent update to IP restrictions inadvertently resulted in impact to the Microsoft 365 admin center. Next steps: - We're reviewing the affecting update to identify how impact was missed during testing and development to better prevent similar future impact. This is the final update for the event.; PublishedTime=2021-01-22T19:23:51.87Z

MO235448 Details

Status: We're investigating a potential issue and checking for impact to your organization. We'll provide an update within 60 minutes.; PublishedTime=2021-01-22T18:31:30.15

Final status: We've determined that a recent update to IP restrictions inadvertently resulted in impact to the Microsoft 365 admin center. We've reverted the affecting update, and confirmed that the admin center page now renders as expected, mitigating impact.

--------------------------------------------------------------------------------------------------
Title Users may intermittently be unable to create or update tasks in the Microsoft To Do app
ID MO236616
Service Microsoft 365 suite
Feature Portal
Status Service restored
Impact Users may have been unable to create or update tasks in the Microsoft To Do app intermittently.
Severity
Start time 2021-01-21T21:00:00Z
End time 2021-01-30T00:00:00Z
Last update 2021-01-30T06:21:27.523Z
More info When the problem occurred, it lasted around five minutes. Affected users could have attempted to update or create tasks five minutes after being affected. Final status: We've confirmed that a code fix was the best mitigation method for this situation. A fix has been developed and deployed to the affected service infrastructure, and we've confirmed via service telemetry that it has successfully mitigated impact. Scope of impact: Your organization was affected by this event, and some users may have been unable to create or update tasks in the Microsoft To Do app. Start time: Thursday, January 21, 2021, at 9:00 PM'UTC End time: Saturday, January 30, 2021, at 12:00 AM'UTC Root cause: A recent change made to Microsoft Graph API authorization components introduced a code issue that unexpectedly resulted in impact to the Microsoft To Do app. Next steps: - We're reviewing our service update and quality assurance procedures to identify how this issue was missed so we can prevent similar impact in the future. This is the final update for the event.; PublishedTime=2021-01-30T06:11:05.087Z
Root cause A recent change made to Microsoft Graph API authorization components introduced a code issue that unexpectedly resulted in impact to the Microsoft To Do app. Next steps: - We're reviewing our service update and quality assurance procedures to identify how this issue was missed so we can prevent similar impact in the future. This is the final update for the event.; PublishedTime=2021-01-30T06:11:05.087Z

MO236616 Details

Status: We've identified that a recent change made to Microsoft Graph API authorization components introduced a code issue that unexpectedly resulted in impact. We're assessing options to mitigate impact. Next update by: Friday, January 29, 2021, at 7:00 PMUTC; PublishedTime=2021-01-29T17:36:56.98Z

Status: We're continuing to run internal tests to assess and validate an optimal solution to mitigate impact. Next update by: Saturday, January 30, 2021, at 8:00 AMUTC; PublishedTime=2021-01-29T18:05:32.523Z

Final status: We've confirmed that a code fix was the best mitigation method for this situation. A fix has been developed and deployed to the affected service infrastructure, and we've confirmed via service telemetry that it has successfully mitigated impact. Next update by: Saturday, January 30, 2021, at 8:00 AMUTC; PublishedTime=2021-01-29T18:05:32.523Z

--------------------------------------------------------------------------------------------------
Title Users may encounter some inbound Public Switched Telephone Network (PSTN) calls intermittently failing
ID TM235108
Service Microsoft Teams
Feature Teams Components
Status Service restored
Impact Users may have been unable to receive inbound PSTN calls via the Microsoft Teams service.
Severity
Start time 2021-01-19T15:15:00Z
End time 2021-01-20T00:40:00Z
Last update 2021-01-20T02:04:36.943Z
More info Some inbound calls to assigned users weren't ringing at all, while other calls were ringing once and then dropping. 100 percent of calls made into call queues were ringing once and then dropping. Final status: After further analysis, we identified a configuration change responsible for call assignments that was inadvertently reverted. We re-implemented the necessary configuration change to remediate the impact. Scope of impact: This issue could've potentially affected any of your users receiving inbound PSTN calls intermittently if they were routed through the affected infrastructure. Start time: Tuesday, January 19, 2021, at 3:15 PM'UTC End time: Wednesday, January 20, 2021, at 12:40 AM'UTC Root cause: A configuration change responsible for call assignments was unexpectedly reverted. Next steps: -We're looking into what caused the configuration change to initially revert which will help us to avoid similar situations in the future. This is the final update for the event.; PublishedTime=2021-01-20T01:37:14.01Z
Root cause A configuration change responsible for call assignments was unexpectedly reverted. Next steps: -We're looking into what caused the configuration change to initially revert which will help us to avoid similar situations in the future. This is the final update for the event.; PublishedTime=2021-01-20T01:37:14.01Z

TM235108 Details

Status: We're investigating a potential issue and checking for impact to your organization. We'll provide an update within 30 minutes. ; PublishedTime=2021-01-19T20:04:13.177

Status: We're reviewing logs from affected calls to isolate the origin of this issue. Next update by: Tuesday, January 19, 2021, at 10:00 PMUTC; PublishedTime=2021-01-19T20:27:26.13Z

Status: We're continuing to review call logs to investigate a specific gateway error we suspect is contributing to impact. Next update by: Wednesday, January 20, 2021, at 12:00 AMUTC; PublishedTime=2021-01-19T21:58:14.503Z

Status: We're continuing our analysis of the gateway error seen in the call logs for more information to determine our next troubleshooting steps. Next update by: Wednesday, January 20, 2021, at 2:00 AMUTC; PublishedTime=2021-01-19T23:51:55.68Z

Final status: After further analysis, we identified a configuration change responsible for call assignments that was inadvertently reverted. We re-implemented the necessary configuration change to remediate the impact. Next update by: Wednesday, January 20, 2021, at 2:00 AMUTC; PublishedTime=2021-01-19T23:51:55.68Z

--------------------------------------------------------------------------------------------------
Title Admins in Eastern United States experiencing latency when accessing and navigating within the Microsoft 365 admin center
ID MO235102
Service Microsoft 365 suite
Feature Portal
Status Service restored
Impact Admins may have experienced latency when accessing and navigating within the Microsoft 365 admin center.
Severity
Start time 2021-01-19T13:00:00Z
End time 2021-01-19T21:04:00Z
Last update 2021-01-19T22:18:42.423Z
More info
Root cause A portion of our traffic management system was performing below acceptable thresholds, causing the infrastructure which serves the Microsoft 365 admin center to become imbalanced and lead to impact. Next steps: - We're reviewing our code for improved performance of the affected traffic management systems, and are seeking to identify potential automated recovery options to reduce or avoid similar impact in the future. This is the final update for the event.; PublishedTime=2021-01-19T22:12:29.72Z

MO235102 Details

Status: We're investigating a potential issue and checking for impact to your organization. We'll provide an update within 60 minutes.; PublishedTime=2021-01-19T19:22:24.66

Status: Although our internal monitoring currently indicates the service is healthy, we're reviewing the available telemetry to isolate any potential issues and develop mitigation steps accordingly. Next update by: Tuesday, January 19, 2021, at 9:30 PMUTC; PublishedTime=2021-01-19T20:02:56.13Z

Status: We've determined a portion of our traffic management system is performing below acceptable thresholds, causing the infrastructure which serves the Microsoft 365 admin center to become imbalanced, leading to impact. We're in the process of rerouting traffic to an alternate traffic management system, which we anticipate will rebalance traffic to the affected infrastructure and mitigate the impact to admins. We'll provide an estimated time to completion when available. Next update by: Tuesday, January 19, 2021, at 10:30 PMUTC; PublishedTime=2021-01-19T20:34:56.8Z

Final status: We've successfully rerouted the traffic to an alternate traffic management system, which has rebalanced the traffic and mitigated the impact to admins. Next update by: Tuesday, January 19, 2021, at 10:30 PMUTC; PublishedTime=2021-01-19T20:34:56.8Z

--------------------------------------------------------------------------------------------------
Title Some users can't subscribe to other users' presence and will see stale or unknown presence in Microsoft Teams
ID TM235087
Service Microsoft Teams
Feature Teams Components
Status Service restored
Impact Users may have been unable to subscribe to other users' presence and may have seen stale presence in Microsoft Teams.
Severity
Start time 2021-01-19T12:35:00Z
End time 2021-01-19T16:38:38Z
Last update 2021-01-19T17:39:43.89Z
More info
Root cause Redis cache issues in a subset of infrastructure had degraded both the ability to subscribe to other users' presence and see the presence of others in Microsoft Teams. Next steps: - We're analyzing performance data and trends on the affected systems to help prevent similar Redis cache problems from happening again. This is the final update for the event.; PublishedTime=2021-01-19T16:38:38.187Z

TM235087 Details

Final status: We've identified a degradation of the Redis cache and have restarted the affected infrastructure, remediating impact.

--------------------------------------------------------------------------------------------------
Title Some admins are unable to download email in the Threat Explorer within the Security & Compliance Center (SCC)
ID EX237173
Service Exchange Online
Feature Networking Issues
Status Service restored
Impact Admins were unable to download email in the Microsoft Defender for Office 365 Threat Explorer within the SCC.
Severity
Start time 2021-01-19T00:00:00Z
End time 2021-02-08T13:15:00Z
Last update 2021-02-09T06:20:41.71Z
More info This issue was only affecting admins attempting to download email in the Microsoft Defender for Office 365 Threat Explorer via the most recent versions of Microsoft Edge and Google Chrome browsers. Final status: We developed and deployed a fix to the affected environment to remediate the impact. Scope of impact: This issue may have potentially affected any of your admins that have a preview role enabled and were attempting to download email messages in the Microsoft Defender for Office 365 Threat Explorer within the Security & Compliance Center. Start time: Tuesday, January 19, 2021, at 12:00 AM'UTC End time: Monday, February 8, 2021, at 1:15 PM'UTC Root cause: A set of recent web browser updates were preventing admins from downloading email in the Microsoft Defender for Office 365 Threat Explorer within the SCC. Next steps: -We're reviewing how the web browser updates interact with Microsoft Defender for Office 365 Threat Explorer to help better understand ways to avoid similar issues in the future. This is the final update for the event.; PublishedTime=2021-02-09T06:20:41.71Z
Root cause A set of recent web browser updates were preventing admins from downloading email in the Microsoft Defender for Office 365 Threat Explorer within the SCC. Next steps: -We're reviewing how the web browser updates interact with Microsoft Defender for Office 365 Threat Explorer to help better understand ways to avoid similar issues in the future. This is the final update for the event.; PublishedTime=2021-02-09T06:20:41.71Z

EX237173 Details

Status: We're reviewing service monitoring telemetry to isolate the source of the issue and establish a fix. Next update by: Sunday, January 31, 2021, at 6:00 PMUTC; PublishedTime=2021-01-31T16:04:10.613Z

Status: Following our service telemetry analysis, we've determined that a recent browser update is preventing admins from downloading email in the Microsoft Defender for Office 365 Thread Explorer within the SCC from the most recent versions of the Microsoft Edge and Google Chrome browsers. We're developing a fix designed to remediate impact, and we're expecting the ability to provide an estimate for the fix completing deployment and remediating impact by our next scheduled update. Next update by: Tuesday, February 2, 2021, at 6:00 AM UTC; PublishedTime=2021-01-31T17:00:13.407Z

Status: We've developed a fix to remediate impact and are in the process of validating it for deployment to the affected environment. We estimate that the fix will saturate the affected environment in about 48 hours. Next update by: Thursday, February 4, 2021, at 6:00 AMUTC; PublishedTime=2021-02-02T04:18:36.18Z

Status: We're continuing to validate the fix and anticipate it will start deploying to the affected environment by our next communication update. Next update by: Saturday, February 6, 2021, at 6:00 AMUTC; PublishedTime=2021-02-04T04:48:37.603Z

Status: We've successfully validated the fix, but have encountered an issue during our pre-deployment process that halted progress. We're working to resolve the issue so we can begin deployment to the affected environment. Next update by: Tuesday, February 9, 2021, at 7:30 AMUTC; PublishedTime=2021-02-06T04:35:40.707Z

Final status: We developed and deployed a fix to the affected environment to remediate the impact. Next update by: Tuesday, February 9, 2021, at 7:30 AMUTC; PublishedTime=2021-02-06T04:35:40.707Z

--------------------------------------------------------------------------------------------------
Title Some admins may experience delays provisioning Microsoft Office licenses
ID MO234982
Service Microsoft 365 suite
Feature Administration
Status Service restored
Impact Admins may have experienced delays provisioning Microsoft Office licenses.
Severity
Start time 2021-01-18T13:05:00Z
End time 2021-01-18T20:40:00Z
Last update 2021-01-18T23:12:03.213Z
More info
Root cause A portion of infrastructure became degraded and wasn't efficiently processing provisioning requests for Microsoft Office. Next steps: - We're analyzing performance data and trends on the affected Microsoft Office provisioning systems to better assess what caused the issue so that we can help prevent this problem from happening again. This is the final update for the event.; PublishedTime=2021-01-18T22:43:37.293Z

MO234982 Details

Status: We've identified a portion of infrastructure that has become degraded and isn't efficiently processing provisioning requests for Microsoft Office. We've resolved the problematic portion of infrastructure and are monitoring as the backlog of requests are processed to confirm that these actions have remediated impact. Next update by: Monday, January 18, 2021, at 11:00 PMUTC; PublishedTime=2021-01-18T21:03:28.677Z

Final status: We've confirmed through monitoring that the backlog of requests has been processed, remediating impact. Next update by: Monday, January 18, 2021, at 11:00 PMUTC; PublishedTime=2021-01-18T21:03:28.677Z

--------------------------------------------------------------------------------------------------
Title Some users may find applications that leverage EWS are unable to access mailboxes within Exchange Online
ID EX234941
Service Exchange Online
Feature Sign-in
Status Service restored
Impact Users may find applications that leverage EWS were unable to access mailboxes within Exchange Online.
Severity
Start time 2021-01-18T08:45:00Z
End time 2021-01-18T18:40:00Z
Last update 2021-01-18T21:02:20.17Z
More info
Root cause Recently enabled feature updates adversely impacted applications that relied on EWS for Exchange Online mailbox access. Next steps: - We're reviewing our feature update procedures to better identify similar issues impacting Exchange Online access through applications that rely on EWS during our development and testing cycles. This is the final update for the event.; PublishedTime=2021-01-18T21:02:20.17Z

EX234941 Details

Status: We're reviewing service telemetry to determine the next troubleshooting steps. Next update by: Monday, January 18, 2021, at 5:30 PM UTC; PublishedTime=2021-01-18T15:44:07.223Z

Status: We've identified recently enabled feature updates that may be the cause of this issue and are disabling them as a possible means of remediating impact. Next update by: Monday, January 18, 2021, at 7:00 PMUTC; PublishedTime=2021-01-18T16:43:09.597Z

Status: We're continuing to roll back these feature updates as a possible means of resolving the issue. It may take up to two hours to complete this process and determine if impact was remediated. Next update by: Monday, January 18, 2021, at 9:30 PMUTC; PublishedTime=2021-01-18T17:55:40.547Z

Final status: We've completed the rollback of the problematic feature update and confirmed with affected users, as well as through internal telemetry, that the issue is resolved. Next update by: Monday, January 18, 2021, at 9:30 PMUTC; PublishedTime=2021-01-18T17:55:40.547Z

--------------------------------------------------------------------------------------------------
Title Users couldn't open apps for the first time in Internet Explorer
ID MM238681
Service Power Apps in Microsoft 365
Feature Service and web access issues
Status Service restored
Impact Users may have been unable to open apps for the first time in Internet Explorer and may have seen a 404 error.
Severity
Start time 2021-01-16T01:01:00Z
End time 2021-02-09T09:45:00Z
Last update 2021-02-09T10:42:38.167Z
More info As a workaround, users would need to delete their personal connections to the app in addition to using another browser. Final status: The fix has been successfully deployed to all affected infrastructure, and, after monitoring, we have confirmed the issue is now resolved. Scope of impact: Any user attempting to open an app for the first time in Internet Explorer may have experienced this issue. Start time: Saturday, January 16, 2021, at 1:01 AM UTC End time: Tuesday, February 9, 2021, at 9:45 AM UTC Root cause: A recent service update inadvertently introduced a code regression that caused app connection infrastructure to perform below acceptable performance thresholds. Next steps: - We're reviewing our update procedures to better identify similar issues during our development and testing cycles. This is the final update for the event.; PublishedTime=2021-02-09T10:41:21.297Z
Root cause A recent service update inadvertently introduced a code regression that caused app connection infrastructure to perform below acceptable performance thresholds. Next steps: - We're reviewing our update procedures to better identify similar issues during our development and testing cycles. This is the final update for the event.; PublishedTime=2021-02-09T10:41:21.297Z

MM238681 Details

Status: We've investigated an issue with the Power Apps service and determined that a recent service update inadvertently introduced a code regression that caused app connection infrastructure to perform below acceptable thresholds. We're working to develop and validate a fix for the issue. Next update by: Tuesday, February 9, 2021, at 7:30 AMUTC; PublishedTime=2021-02-09T06:13:41.423Z

Status: We're continuing our work to validate the fix for this issue in our internal testing environments before beginning deployment to affected environments. Next update by: Tuesday, February 9, 2021, at 9:30 AMUTC; PublishedTime=2021-02-09T07:13:43.687Z

Status: We've successfully validated our fix and moved to deployment across all affected infrastructure. We're monitoring the progress of the fix to confirm mitigation of impact. Next update by: Tuesday, February 9, 2021, at 11:30 AM UTC; PublishedTime=2021-02-09T08:52:42.313Z

Final status: The fix has been successfully deployed to all affected infrastructure, and, after monitoring, we have confirmed the issue is now resolved. Next update by: Tuesday, February 9, 2021, at 11:30 AM UTC; PublishedTime=2021-02-09T08:52:42.313Z

--------------------------------------------------------------------------------------------------
Title Users may experience delays of up to two hours when receiving Yammer messages
ID YA234576
Service Yammer Enterprise
Feature Yammer Components
Status Service restored
Impact Users may have experienced delays of up to two hours when receiving Yammer messages.
Severity
Start time 2021-01-15T20:05:00Z
End time 2021-01-15T22:03:00Z
Last update 2021-01-15T22:28:48.057Z
More info Users may have also experienced intermittent failures when loading the Yammer feed. Final status: We've confirmed that the code fix has fully saturated in the affected environments and after monitoring the service, we've validated that the fix successfully resolved the problem. Scope of impact: Impact may have occurred for all users attempting to receive Yammer messages and when loading the Yammer feed. Start time: Friday, January 15, 2021, at 8:05 PM'UTC End time: Friday, January 15, 2021, at 10:03 PM'UTC Root cause: A recent change intended to update Yammer feed features unexpectedly resulted in impact. Next steps: - We're investigating why the recent change intended to update Yammer feed features resulted in impact to prevent this problem from happening again in the future. This is the final update for the event.; PublishedTime=2021-01-15T22:28:48.057Z
Root cause A recent change intended to update Yammer feed features unexpectedly resulted in impact. Next steps: - We're investigating why the recent change intended to update Yammer feed features resulted in impact to prevent this problem from happening again in the future. This is the final update for the event.; PublishedTime=2021-01-15T22:28:48.057Z

YA234576 Details

Status: We've identified that a recent change intended to update Yammer feed features unexpectedly resulted in impact. We've developed a code fix and we've started to deploy this fix to the affected environments. Additionally, we're monitoring this fix to ensure it resolves the impact in full. Next update by: Friday, January 15, 2021, at 11:00 PMUTC; PublishedTime=2021-01-15T22:03:35.753Z

Final status: We've confirmed that the code fix has fully saturated in the affected environments and after monitoring the service, we've validated that the fix successfully resolved the problem. Next update by: Friday, January 15, 2021, at 11:00 PMUTC; PublishedTime=2021-01-15T22:03:35.753Z

--------------------------------------------------------------------------------------------------
Title Users in South America can't connect to Microsoft Teams via all connections
ID TM234578
Service Microsoft Teams
Feature Teams Components
Status Service restored
Impact Users in South America may have been unable to connect to Microsoft Teams via all connections.
Severity
Start time 2021-01-15T17:30:00Z
End time 2021-01-15T20:15:00Z
Last update 2021-01-27T15:25:12.17Z
More info
Root cause A portion of network infrastructure wasn't processing traffic as expected, leading to impact. Next steps: - We're analyzing performance data and trends on the affected infrastructure to better understand why it stopped processing traffic as expected so that we can prevent similar issues in the future. This is the final update for the event.; PublishedTime=2021-01-15T22:06:52.62Z

TM234578 Details

Final status: We've investigated an issue in which some users in South America couldn't access Microsoft Teams and determined that a portion of network infrastructure wasn't processing traffic as expected, leading to impact. We've prevented the affected infrastructure from processing traffic and rerouted traffic to alternate infrastructure to mitigate the problem.

--------------------------------------------------------------------------------------------------
Title Some users may have been unable to create approvals within Microsoft Teams
ID TM234355
Service Microsoft Teams
Feature Teams Components
Status Service restored
Impact Users may have been unable to create approvals within Microsoft Teams.
Severity
Start time 2021-01-14T09:15:00Z
End time 2021-01-14T10:23:31.893Z
Last update 2021-01-14T10:32:36.863Z
More info
Root cause

TM234355 Details

Final status: While we were in the process of analyzing service telemetry to isolate the source of the problem, system monitoring indicated that the service had returned to normal health. We've confirmed that the problem is no longer occurring, and we'll continue to monitor the service to ensure that the problem does not happen again.

--------------------------------------------------------------------------------------------------
Title Users see calls from call queues and auto attendants failing
ID LY234459
Service Skype for Business
Feature Audio and Video
Status Service restored
Impact Users may have experienced calls from calls queues and auto attendants failing.
Severity
Start time 2021-01-14T01:00:00Z
End time 2021-01-15T03:28:06.95Z
Last update 2021-01-15T03:29:42.54Z
More info
Root cause Recent service infrastructure changes inadvertently resulted in call routing failures for calls from call queues and auto attendants. Next steps: - We're reviewing our recent infrastructure changes to better understand how impact started and to prevent similar impact in the future. This is the final update for the event.; PublishedTime=2021-01-15T03:28:06.95Z

LY234459 Details

Final status: We've investigated an issue in which call queue and auto attendant functionality wasn't working as expected. We've reverted the changes to resolve the issue and confirmed via service monitoring that this has mitigated impact.

--------------------------------------------------------------------------------------------------
Title Users see calls from call queues and auto attendants failing
ID LY234459
Service Skype for Business
Feature Audio and Video
Status Service restored
Impact Users may have experienced calls from calls queues and auto attendants failing.
Severity
Start time 2021-01-14T01:00:00Z
End time 2021-01-15T03:28:06.95Z
Last update 2021-01-15T03:29:42.54Z
More info
Root cause Recent service infrastructure changes inadvertently resulted in call routing failures for calls from call queues and auto attendants. Next steps: - We're reviewing our recent infrastructure changes to better understand how impact started and to prevent similar impact in the future. This is the final update for the event.; PublishedTime=2021-01-15T03:28:06.95Z

LY234459 Details

Final status: We've investigated an issue in which call queue and auto attendant functionality wasn't working as expected. We've reverted the changes to resolve the issue and confirmed via service monitoring that this has mitigated impact.

--------------------------------------------------------------------------------------------------
Title Users with Samsung devices running Android version 7.0 or lower are unable to receive new or updated polices
ID IT246676
Service Microsoft Intune
Feature Microsoft Intune
Status Service restored
Impact Users with Samsung devices running Android version 7.0 or lower were unable to receive new or updated polices.
Severity
Start time 2021-01-12T18:18:00Z
End time 2021-03-31T19:34:00Z
Last update 2021-03-31T20:30:10.267Z
More info This affected Samsung devices running Android 7.0 or lower that were enrolled using Device Administrator before Tuesday, January 12, 2021, and that leveraged the most recent version of the Intune Company Portal app. Final status: The fix has rolled out to the Google Play Store and has become accessible to users. Please update the Intune Company Portal app to version 5.0.5103.0 or newer. Users need to click on a notification that will require them to accept the KNOX privacy notice. If you do not have this notification, please sync your device using the steps documented here: https://docs.microsoft.com/en-us/mem/intune/user-help/sync-device-androi d#:~:text=Sync%20device.%201%20Sign%20in%20to%20the%20Company,the%20timestamp%20of%20the% 20last%20successful%20sync. Scope of impact: Impact was specific to users that met the impact criteria outlined in the 'More info' section of this communication. Start time: Tuesday, January 12, 2021, at 6:18 PM UTC End time: Wednesday, March 31, 2021, at 7:34 PM UTC Root cause: The Intune service wasn't properly recognizing the affected device licenses as accepted. Next steps: - We're further reviewing what caused the device licenses to not be accepted to prevent similar problems for Samsung devices in the future. This is the final update for the event.; PublishedTime=2021-03-31T20:11:28.96Z
Root cause The Intune service wasn't properly recognizing the affected device licenses as accepted. Next steps: - We're further reviewing what caused the device licenses to not be accepted to prevent similar problems for Samsung devices in the future. This is the final update for the event.; PublishedTime=2021-03-31T20:11:28.96Z

IT246676 Details

Status: We've identified that the Intune service isn't properly recognizing the affected device licenses as accepted, which is causing the problem. We're exploring impact remediation options. Next update by: Friday, March 26, 2021, at 9:30 PMUTC; PublishedTime=2021-03-26T19:32:31.753Z

Status: We're currently in the process of testing and validating of a fix to the affected environment. Once completed, we'll provide a deployment timeline for mitigation. Next update by: Saturday, March 27, 2021, at 1:00 AMUTC; PublishedTime=2021-03-26T21:24:54.083Z

Status: We're nearing completion of testing and validating the fix. We expect to confirm it's effectiveness at our next scheduled update as well as to provide a deployment timeline for mitigation. Next update by: Saturday, March 27, 2021, at 3:30 AMUTC; PublishedTime=2021-03-27T00:14:03.903Z

Status: We've discovered an issue during testing which we are now in the process of debugging. Once the issue is resolved, we'll initiate our deployment procedures and provide a deployment timeline for mitigation as soon as possible. Next update by: Saturday, March 27, 2021, at 4:00 AMUTC; PublishedTime=2021-03-27T01:56:24.89Z

Status: We've fixed the issue found in the fix and are currently validating it again prior to initiating our deployment procedures to ensure the update remediates the policy issue without inadvertent impact. Next update by: Saturday, March 27, 2021, at 9:00 AMUTC; PublishedTime=2021-03-27T03:45:48.243Z

Status: We've completed validating the fix and have started our deployment process to the Google Play Store as an update to the Intune Company Portal app. The fix is in the Company Portal version 5.0.5103.0. Devices experiencing this issue must install at least that version to remediate the issue, and then tap the notification on the device to accept the KNOX privacy notice. The roll out process to the Google Play Store occurs in stages. We expect it to be fully published to the Play Store by Wednesday, March 31, 2021. As the fix propagates through the Google Play Store, some devices may see the fixed version of Company Portal. The fix should be available in the Beta channel in Google Play within 4 to 8 hours. Customers may optionally join the Beta channel in Google Play. Follow these instructions to opt in to the Company Portal Beta: https://support.google.com/googleplay/answer/7003180'hl=en Next update by: Saturday, March 27, 2021, at 9:00 PM UTC; PublishedTime=2021-03-27T07:37:50.913Z

Status: As previously communicated, the roll out process to the Google Play Store occurs in stages, and we've confirmed that it's progressing as expected and is on track to be available by Wednesday, March 31, 2021. The fix will be in the Intune Company Portal app version 5.0.5103.0. Next update by: Sunday, March 28, 2021, at 9:00 PMUTC; PublishedTime=2021-03-27T19:40:25.357Z

Status: Our staged roll out to the Google Play Store continues to progress as expected and remains on schedule to be available by Wednesday, March 31, 2021. The fix will be in the Intune Company Portal app version 5.0.5103.0. Next update by: Monday, March 29, 2021, at 9:30 PMUTC; PublishedTime=2021-03-28T19:15:34.107Z

Status: The staged roll out to the Google Play Store is progressing as expected, and we'll continue to monitor as it saturates throughout the impacted environment. We continue to expect for it to complete by Wednesday March 31, 2021. Next update by: Wednesday, March 31, 2021, at 9:30 PMUTC; PublishedTime=2021-03-29T20:44:13.037Z

Final status: The fix has rolled out to the Google Play Store and has become accessible to users. Please update the Intune Company Portal app to version 5.0.5103.0 or newer. Users need to click on a notification that will require them to accept the KNOX privacy notice. If you do not have this notification, please sync your device using the steps documented here: https://docs.microsoft.com/en-us/mem/intune/user-help/sync-device-androi d#:~:text=Sync%20device.%201%20Sign%20in%20to%20the%20Company,the%20timestamp%20of%20the% 20last%20successful%20sync. Next update by: Wednesday, March 31, 2021, at 9:30 PMUTC; PublishedTime=2021-03-29T20:44:13.037Z

--------------------------------------------------------------------------------------------------
Title Issues accessing Microsoft 365 services
ID MO234041
Service Microsoft 365 suite
Feature Portal
Status Post-incident report published
Impact Users may have experienced access issues when using one or more Microsoft 365 services.
Severity
Start time 2021-01-12T15:51:00Z
End time 2021-01-12T16:40:00Z
Last update 2021-01-19T18:51:22.017Z
More info This issue was primarily affecting SharePoint Online and Microsoft Teams users in Europe; though, users may have also experienced similar issues with Exchange Online, OneDrive for Business, and Yammer. We've identified that the issue occurred in a datacenter located in London, and thus the majority of impact was for users located in the United Kingdom. Additionally, users in other European countries such as France and Spain may have been impacted if their connections were routed through the affected components in the London datacenter. Final status: We've confirmed that processing infrastructure that handles user access requests for Microsoft 365 services in the European region experienced failures, resulting in impact. While we were in the process of analyzing system logs to identify the cause of these failures, our systems redirected requests to alternate, healthy infrastructure and our monitoring confirms that the service has returned to normal health. We'll continue to monitor the service to ensure that the problem does not happen again. Scope of impact: Impact was primarily to users located in the United Kingdom; however, users in other regions who were routed through the affected United Kingdom-based infrastructure may have also been impacted. Start time: Tuesday, January 12, 2021, at 3:51 PM UTC End time: Tuesday, January 12, 2021, at 4:40 PM UTC Root cause: A spike in Teams EDU traffic routed through our UK data center, double that of the previous month's peak, resulted in a peering link between Microsoft edge network and one Internet Service Provider (ISP) within the region to become congested and experience packet drops. These failures affected traffic for both new access requests and service usage for users already signed in prior to the impact period. Next steps: - For a more comprehensive list of next steps and actions, please refer to the Post Incident Review document. We'll publish a post-incident report within five business days.; PublishedTime=2021-01-12T17:38:00.64Z
Root cause A spike in Teams EDU traffic routed through our UK data center, double that of the previous month's peak, resulted in a peering link between Microsoft edge network and one Internet Service Provider (ISP) within the region to become congested and experience packet drops. These failures affected traffic for both new access requests and service usage for users already signed in prior to the impact period. Next steps: - For a more comprehensive list of next steps and actions, please refer to the Post Incident Review document. We'll publish a post-incident report within five business days.; PublishedTime=2021-01-12T17:38:00.64Z

MO234041 Details

Status: We've identified that impact reported in SP234036 and TM234038 are part of the same incident. We're working to identify the cause of the problem; though, our monitoring indicates that services are recovering. Next update by: Tuesday, January 12, 2021, at 6:00 PMUTC; PublishedTime=2021-01-12T16:57:47.933Z

Final status: We've confirmed that processing infrastructure that handles user access requests for Microsoft 365 services in the European region experienced failures, resulting in impact. While we were in the process of analyzing system logs to identify the cause of these failures, our systems redirected requests to alternate, healthy infrastructure and our monitoring confirms that the service has returned to normal health. We'll continue to monitor the service to ensure that the problem does not happen again. Next update by: Tuesday, January 12, 2021, at 6:00 PMUTC; PublishedTime=2021-01-12T16:57:47.933Z

--------------------------------------------------------------------------------------------------
Title Can't send messages or load chats/channels
ID TM233996
Service Microsoft Teams
Feature Teams Components
Status Post-incident report published
Impact Users may have been unable to send messages or load chats and channels.
Severity
Start time 2021-01-12T09:00:00Z
End time 2021-01-12T09:44:00Z
Last update 2021-01-19T19:12:37.38Z
More info
Root cause Some front-end components were performing below optimal performance thresholds. Next steps: - For a more comprehensive list of next steps and actions, please refer to the Post Incident Review document. We'll publish a post-incident report within five business days.; PublishedTime=2021-01-12T10:21:01.097Z

TM233996 Details

Status: We're investigating potential a potential issue with the Microsoft Teams service in Europe. We'll provide an update within 30 minutes.; PublishedTime=2021-01-12T09:25:30.073

Status: We're analyzing diagnostic reports to understand the cause of the issue and establish a fix. Next update by: Tuesday, January 12, 2021, at 11:00 AM UTC; PublishedTime=2021-01-12T09:54:55.117Z

Final status: We determined that some front-end components were performing below optimal performance thresholds. We removed the affected components from active service to remediate the issue. Next update by: Tuesday, January 12, 2021, at 11:00 AM UTC; PublishedTime=2021-01-12T09:54:55.117Z

--------------------------------------------------------------------------------------------------
Title Users may have been unable to open any Microsoft PowerApps Canvas apps using the Internet Explorer web browser
ID MM234188
Service Power Apps in Microsoft 365
Feature Service and web access issues
Status Service restored
Impact Users may have been unable to open any Microsoft PowerApps Canvas apps using the Internet Explorer web browser.
Severity
Start time 2021-01-12T00:08:00Z
End time 2021-01-13T08:08:00Z
Last update 2021-01-13T08:25:47.1Z
More info
Root cause A code compatibility issue with the most recent release of Microsoft PowerApps, was resulting in an issue opening Canvas apps. Next steps: - We're reviewing our standard service update procedures to avoid similar impact in the future. This is the final update for the event.; PublishedTime=2021-01-13T08:25:47.1Z

MM234188 Details

Status: We've determined that there is a code compatibility issue with the most recent release of Microsoft PowerApps. We're reverting the affected PowerApps version to mitigate impact. Next update by: Wednesday, January 13, 2021, at 9:00 AMUTC; PublishedTime=2021-01-13T07:15:48.607Z

Final status: We identified a recent update, designed to improve the overall performance of the PowerApps service, contained a code compatibility issue. We rolled back the build and after testing and monitoring, confirmed impact had been mitigated. Next update by: Wednesday, January 13, 2021, at 9:00 AMUTC; PublishedTime=2021-01-13T07:15:48.607Z

--------------------------------------------------------------------------------------------------
Title Admins may experience up to 25 minute delays in updating shift details in Microsoft Teams
ID TM233837
Service Microsoft Teams
Feature Teams Components
Status Service restored
Impact Admins may have experienced a delay of up to 25 minutes when updating users' shift details within Microsoft Teams.
Severity
Start time 2021-01-11T13:25:00Z
End time 2021-01-11T19:00:00Z
Last update 2021-01-11T19:41:58.557Z
More info
Root cause A subset of infrastructure that facilitates shift update processing wasn't performing as efficiently as expected, resulting in impact. Next steps: - We're reviewing our code for improved shift update performance and potential automated recovery options to reduce or avoid similar impact in the future. This is the final update for the event.; PublishedTime=2021-01-11T19:41:58.557Z

TM233837 Details

Status: We're investigating a potential issue and checking for impact to your organization. We'll provide an update within 30 minutes.; PublishedTime=2021-01-11T16:08:57.713

Status: We've identified a portion of infrastructure that processes shift updates prior to sending to users' devices isn't performing as efficiently as expected. We're reviewing system telemetry to isolate the underlying cause. Next update by: Monday, January 11, 2021, at 6:00 PMUTC; PublishedTime=2021-01-11T16:31:49.01Z

Status: We're working to rebalance load from the affected infrastructure to mitigate impact while continuing our review of system telemetry to determine the root cause. Next update by: Monday, January 11, 2021, at 8:00 PMUTC; PublishedTime=2021-01-11T17:47:37.033Z

Final status: We've completed rebalancing operations and confirmed mitigation via telemetry. Next update by: Monday, January 11, 2021, at 8:00 PMUTC; PublishedTime=2021-01-11T17:47:37.033Z

--------------------------------------------------------------------------------------------------
Title Users can't assign SharePoint Online site permissions to SharePoint Online groups
ID SP235979
Service SharePoint Online
Feature SharePoint Features
Status Service restored
Impact Users may have experienced errors when attempting to assign site permissions to SharePoint Online groups.
Severity
Start time 2021-01-11T12:00:00Z
End time 2021-01-25T20:00:00Z
Last update 2021-01-25T20:42:45.873Z
More info Users may have seen the following error: "Sharing failed: Couldn't resolve user" Users were able to add individual users' permissions to sites. Additionally, impact was only experienced if external sharing was enabled. Final status: We've identified that a recent service update intended to improve permission assignment performance contained a configuration change that resulted in impact. We've corrected this change and confirmed mitigation with affected users. Scope of impact: Any user editing site permissions may have been impacted. Start time: Monday, January 11, 2021, at 12:00 PM'UTC End time: Monday, January 25, 2021, at 8:00 PM'UTC Root cause: A recent service update intended to improve permission assignment performance contained a configuration change that resulted in impact. Next steps: - We're reviewing our service update configuration procedures to better identify similar issues during our development and testing cycles. This is the final update for the event.; PublishedTime=2021-01-25T20:42:45.873Z
Root cause A recent service update intended to improve permission assignment performance contained a configuration change that resulted in impact. Next steps: - We're reviewing our service update configuration procedures to better identify similar issues during our development and testing cycles. This is the final update for the event.; PublishedTime=2021-01-25T20:42:45.873Z

SP235979 Details

Status: We're investigating a potential issue and checking for impact to your organization. We'll provide an update within 30 minutes.; PublishedTime=2021-01-25T17:21:56.197

Status: We're reviewing network traces provided by impacted users to isolate the underlying cause of impact. Next update by: Monday, January 25, 2021, at 7:00 PMUTC; PublishedTime=2021-01-25T17:43:32.567Z

Status: Our review of network traces has found that the People Picker is making calls to an unrelated service. We're expanding our investigation to include Unified Logging Service (ULS) logs to determine the root cause. Next update by: Monday, January 25, 2021, at 9:00 PMUTC; PublishedTime=2021-01-25T18:47:39.167Z

Final status: We've identified that a recent service update intended to improve permission assignment performance contained a configuration change that resulted in impact. We've corrected this change and confirmed mitigation with affected users. Next update by: Monday, January 25, 2021, at 9:00 PMUTC; PublishedTime=2021-01-25T18:47:39.167Z

--------------------------------------------------------------------------------------------------
Title Users may have been unable to access the Microsoft Teams service
ID TM233796
Service Microsoft Teams
Feature Teams Components
Status False positive
Impact Users may have been unable to access the Microsoft Teams service.
Severity
Start time 2021-01-11T09:00:00Z
End time 2021-01-11T12:35:00Z
Last update 2021-01-11T14:56:43.623Z
More info Initial reports indicate this may be specific to users based in the United Kingdom and Northern Europe. Current status: We're investigating a potential access issue and checking for impact to your organization. We'll provide an update within 30 minutes.; PublishedTime=2021-01-11T09:28:18.6Z
Root cause A spike in user requests unexpectedly caused some infrastructure, responsible for handling Teams access, to perform below acceptable thresholds. Next steps: - We're analyzing performance data and trends on the affected systems to help prevent further occurrence of this issue. This is the final update for the event.; PublishedTime=2021-01-11T14:56:43.623Z

TM233796 Details

Status: We're investigating a potential access issue and checking for impact to your organization. We'll provide an update within 30 minutes.; PublishedTime=2021-01-11T09:28:18.6

Status: We're investigating service telemetry to determine the underlying cause of the issue. Next update by: Monday, January 11, 2021, at 12:00 PM UTC; PublishedTime=2021-01-11T09:56:11.233Z

Status: As we continue to investigate service telemetry to determine the underlying cause of the issue, we're also reviewing diagnostic data for signs of any network related issues. Next update by: Monday, January 11, 2021, at 2:00 PM UTC; PublishedTime=2021-01-11T11:48:05.28Z

Status: We've identified that a spike in user requests has unexpectedly caused some infrastructure, responsible for handling Teams access requests, to perform below acceptable thresholds. Our automated recovery services have run commands to redirect and optimize traffic. We're monitoring service health to ensure these commands have mitigate impact as expected. Next update by: Monday, January 11, 2021, at 4:00 PM UTC; PublishedTime=2021-01-11T13:58:57.227Z

Final status: After additional monitoring, we have confirmed the issue is now resolved and will continue to monitor service health and trends to ensure it does not happen again. Next update by: Monday, January 11, 2021, at 4:00 PM UTC; PublishedTime=2021-01-11T13:58:57.227Z

--------------------------------------------------------------------------------------------------
Title Users may experience intermittent errors when performing various actions in Exchange Online mailboxes on legal hold
ID EX234238
Service Exchange Online
Feature E-Mail and calendar access
Status Service restored
Impact Users experienced intermittent errors when performing various actions in Exchange Online mailboxes on legal hold.
Severity
Start time 2021-01-08T10:36:00Z
End time 2021-01-15T04:00:00Z
Last update 2021-01-15T17:57:37.72Z
More info Impact could have occurred for any user or shared mailbox under legal hold during any write operation including, but not limited to, reply all, forwarding email, applying categories to or flagging email, and moving email between mailbox folders. Affected users may have seen the following error: "The operation failed. The messaging interfaces have returned an unknown error. If the problem persists, restart Outlook." Additionally, some users may have noticed that their Outlook desktop client crashed, froze, or restarted after attempting these operations. While we were focused on mitigation, affected users that could use Outlook on the web may have been able to consistently perform these actions through this connection method as a workaround. Final status: We've confirmed that the fix has completed deployment to all affected environments. We confirmed with users and verified through internal telemetry that the issue is resolved. Scope of impact: Any user attempting to perform actions within a mailbox under a legal hold may have been intermittently impacted. Start time: Friday, January 8, 2021, at 10:36 AM UTC End time: Friday, January 15, 2021, at 4:00 AM UTC Root cause: A recent configuration update intended to resolve a retention issue inadvertently prevented certain message properties from being set correctly, resulting in impact. Next steps: - We're reviewing our configuration update procedures to better identify similar Exchange Online mailbox function issues during our development and testing cycles. This is the final update for the event.; PublishedTime=2021-01-15T17:57:37.72Z
Root cause A recent configuration update intended to resolve a retention issue inadvertently prevented certain message properties from being set correctly, resulting in impact. Next steps: - We're reviewing our configuration update procedures to better identify similar Exchange Online mailbox function issues during our development and testing cycles. This is the final update for the event.; PublishedTime=2021-01-15T17:57:37.72Z

EX234238 Details

Status: We're investigating a potential issue and checking for impact to your organization. We'll provide an update within 30 minutes.; PublishedTime=2021-01-13T21:12:25.567

Status: We're reviewing network traces provided by affected users to isolate the underlying cause. Next update by: Wednesday, January 13, 2021, at 11:00 PMUTC; PublishedTime=2021-01-13T21:38:11.99Z

Status: Our review of network traces has shown that a recent configuration update is preventing certain mail properties from updating as expected. We've developed a fix and are working to validate it prior to deployment. Next update by: Thursday, January 14, 2021, at 2:00 AMUTC; PublishedTime=2021-01-13T22:44:30.227Z

Status: We're continuing our work to validate the fix, and expect validation to complete on Friday, January 15, 2020 at which point the fix will be deployed. Next update by: Friday, January 15, 2021, at 10:00 PM UTC; PublishedTime=2021-01-13T23:21:22.497Z

Status: During the validation of our code fix to address the issue, we've determined that the more expedited method for remediating impact will be reverting the offending service update. The service update is being reverted, and we're expecting for the process to complete by our next scheduled update. Next update by: Friday, January 15, 2021, at 3:00 AM UTC; PublishedTime=2021-01-14T21:44:48.55Z

Status: We've completed reverting the offending service update for some environments, and are reaching out to affected users in those environments to confirm that impact has been mitigated for them. In parallel, we're continuing our efforts to revert the offending service update in the remaining impacted. Next update by: Friday, January 15, 2021, at 8:00 PM UTC; PublishedTime=2021-01-15T01:45:01.833Z

Final status: We've confirmed that the fix has completed deployment to all affected environments. We confirmed with users and verified through internal telemetry that the issue is resolved. Next update by: Friday, January 15, 2021, at 8:00 PM UTC; PublishedTime=2021-01-15T01:45:01.833Z

--------------------------------------------------------------------------------------------------
Title Users selecting Request files for folders in OneDrive for Business aren't prompted to select which files are requested
ID SP234055
Service SharePoint Online
Feature SharePoint Features
Status Service restored
Impact Users may have been unable to use Request files for folders in OneDrive for Business.
Severity
Start time 2021-01-08T00:00:00Z
End time 2021-01-13T04:18:00Z
Last update 2021-01-13T17:37:51.167Z
More info
Root cause A recent client-side code change introduced a resource configuration issue which resulted in impact. Next steps: - We're reviewing our client-side code change procedures when they are associated with resource configurations to find ways to prevent this problem from happening again. This is the final update for the event.; PublishedTime=2021-01-13T17:37:51.167Z

SP234055 Details

Status: We're investigating a potential issue and checking for impact to your organization. We'll provide an update within 60 minutes.; PublishedTime=2021-01-12T18:38:41.783

Status: We're reviewing available diagnostic data to isolate the source of impact and identify troubleshooting actions. Next update by: Tuesday, January 12, 2021, at 9:00 PMUTC; PublishedTime=2021-01-12T19:30:30.693Z

Status: We're reviewing error logs and recent changes made to the affected infrastructure to isolate the source of this issue. Next update by: Wednesday, January 13, 2021, at 12:30 AMUTC; PublishedTime=2021-01-12T20:42:53.257Z

Status: We've identified a recent client-side code change introduced a resource configuration issue which is leading to impact. We're developing a targeted fix for this issue, which we'll internally validate before deploying it to the affected environment. We'll provide an estimated deployment timeline by the next scheduled update. Next update by: Wednesday, January 13, 2021, at 2:30 AMUTC; PublishedTime=2021-01-12T22:57:10.547Z

Status: The development process for our fix continues, as we're engaged in the validation and testing phase to ensure the fix works as expected without creating secondary impact. Once this phase is completed, we'll establish a timeline for the deployment of the fix for full mitigation. Next update by: Wednesday, January 13, 2021, at 8:00 PMUTC; PublishedTime=2021-01-13T00:48:12.707Z

Final status: We've fully developed and deployed the fix, and we confirmed internally and with affected users that impact has been remediated. Next update by: Wednesday, January 13, 2021, at 8:00 PMUTC; PublishedTime=2021-01-13T00:48:12.707Z

--------------------------------------------------------------------------------------------------
Title Users selecting Request files for folders in OneDrive for Business aren't prompted to select which files are requested
ID SP234055
Service SharePoint Online
Feature SharePoint Features
Status Service restored
Impact Users may have been unable to use Request files for folders in OneDrive for Business.
Severity
Start time 2021-01-08T00:00:00Z
End time 2021-01-13T04:18:00Z
Last update 2021-01-13T17:37:51.167Z
More info
Root cause A recent client-side code change introduced a resource configuration issue which resulted in impact. Next steps: - We're reviewing our client-side code change procedures when they are associated with resource configurations to find ways to prevent this problem from happening again. This is the final update for the event.; PublishedTime=2021-01-13T17:37:51.167Z

SP234055 Details

Status: We're investigating a potential issue and checking for impact to your organization. We'll provide an update within 60 minutes.; PublishedTime=2021-01-12T18:38:41.783

Status: We're reviewing available diagnostic data to isolate the source of impact and identify troubleshooting actions. Next update by: Tuesday, January 12, 2021, at 9:00 PMUTC; PublishedTime=2021-01-12T19:30:30.693Z

Status: We're reviewing error logs and recent changes made to the affected infrastructure to isolate the source of this issue. Next update by: Wednesday, January 13, 2021, at 12:30 AMUTC; PublishedTime=2021-01-12T20:42:53.257Z

Status: We've identified a recent client-side code change introduced a resource configuration issue which is leading to impact. We're developing a targeted fix for this issue, which we'll internally validate before deploying it to the affected environment. We'll provide an estimated deployment timeline by the next scheduled update. Next update by: Wednesday, January 13, 2021, at 2:30 AMUTC; PublishedTime=2021-01-12T22:57:10.547Z

Status: The development process for our fix continues, as we're engaged in the validation and testing phase to ensure the fix works as expected without creating secondary impact. Once this phase is completed, we'll establish a timeline for the deployment of the fix for full mitigation. Next update by: Wednesday, January 13, 2021, at 8:00 PMUTC; PublishedTime=2021-01-13T00:48:12.707Z

Final status: We've fully developed and deployed the fix, and we confirmed internally and with affected users that impact has been remediated. Next update by: Wednesday, January 13, 2021, at 8:00 PMUTC; PublishedTime=2021-01-13T00:48:12.707Z

--------------------------------------------------------------------------------------------------
Title Admins unable to create or edit apps and could see stale data in policy and apps reports
ID IT233199
Service Microsoft Intune
Feature Microsoft Intune
Status Service restored
Impact Admins were unable to create or edit apps within the Microsoft Endpoint Manager admin center.
Severity
Start time 2021-01-07T18:10:00Z
End time 2021-01-07T21:53:00Z
Last update 2021-01-07T22:40:12.15Z
More info Admins may have also observed stale data for policy and app reports, and may have been unable to conduct Apple Volume Purchase Program (VPP) token syncs within the Microsoft Endpoint Manager admin center. Final status: After our rerouting of traffic to an alternate subsection of SQL database infrastructure we've confirmed that impact has been resolved. Scope of impact: This issue may have potentially affected any of your admins attempting to create or edit apps, see up-to-date policy and app data, or sync Apple VPP tokens within the Microsoft Endpoint Manager admin center. Start time: Thursday, January 7, 2021, at 6:10 PM UTC End time: Thursday, January 7, 2021, at 9:53 PM UTC Root cause: A subsection of SQL database infrastructure fell below our manageable service performance thresholds, resulting in impact to admins within the Microsoft Endpoint Manager admin center. Next steps: - We're analyzing performance data and trends on the affected systems to determine why the SQL databases dropped below our performance thresholds, and to help prevent this problem from happening again. This is the final update for the event.; PublishedTime=2021-01-07T22:40:12.15Z
Root cause A subsection of SQL database infrastructure fell below our manageable service performance thresholds, resulting in impact to admins within the Microsoft Endpoint Manager admin center. Next steps: - We're analyzing performance data and trends on the affected systems to determine why the SQL databases dropped below our performance thresholds, and to help prevent this problem from happening again. This is the final update for the event.; PublishedTime=2021-01-07T22:40:12.15Z

IT233199 Details

Status: We're investigating a potential issue and checking for impact to your organization. We'll provide an update within 30 minutes.; PublishedTime=2021-01-07T20:13:14.153

Status: We're analyzing system logs for a subsection of SQL database infrastructure as we work to determine the source for this event. Next update by: Thursday, January 7, 2021, at 10:30 PMUTC; PublishedTime=2021-01-07T20:30:45.16Z

Status: Following our system log review, we've identified a subsection of SQL database infrastructure that has fallen below our acceptable service performance thresholds, resulting in impact. We're rerouting traffic to an alternate subsection of SQL database infrastructure which is expected to remediate impact. Next update by: Friday, January 8, 2021, at 1:00 AMUTC; PublishedTime=2021-01-07T22:03:39.507Z

Final status: After our rerouting of traffic to an alternate subsection of SQL database infrastructure we've confirmed that impact has been resolved. Next update by: Friday, January 8, 2021, at 1:00 AMUTC; PublishedTime=2021-01-07T22:03:39.507Z

--------------------------------------------------------------------------------------------------
Title Some users may have been unable to connect to multiple Microsoft 365 services
ID MO233139
Service Microsoft 365 suite
Feature Portal
Status Service restored
Impact Users located in Denmark may have been unable to connect to one or more Microsoft 365 services.
Severity
Start time 2021-01-07T09:00:00Z
End time 2021-01-07T13:00:00Z
Last update 2021-01-07T13:57:06.483Z
More info
Root cause A networking tool was improperly balancing user connections in the affected region. Next steps: - We're analyzing performance data and trends on the networking tool to avoid similar impact in the future. This is the final update for the event.; PublishedTime=2021-01-07T13:57:06.483Z

MO233139 Details

Status: We're investigating a potential issue with the Microsoft 365 service in Denmark. We'll provide an update within 30 minutes.; PublishedTime=2021-01-07T11:21:35.127

Status: We're reviewing service monitoring telemetry to try and isolate the source of the issue and determine effective troubleshooting steps. Next update by: Thursday, January 7, 2021, at 2:00 PM UTC; PublishedTime=2021-01-07T11:50:13.767Z

Final status: We identified that a networking tool was incorrectly balancing user connections in the affected region. We've manually adjusted the load balancing and confirmed, after monitoring the impacted environment, that the issue has been resolved. Next update by: Thursday, January 7, 2021, at 2:00 PM UTC; PublishedTime=2021-01-07T11:50:13.767Z

--------------------------------------------------------------------------------------------------
Title Small number of users intermittently unable to access Microsoft Teams from the web app and desktop client
ID TM233064
Service Microsoft Teams
Feature Teams Components
Status Service restored
Impact Users may have intermittently been unable to access Microsoft Teams from the web app and desktop client.
Severity
Start time 2021-01-06T16:10:00Z
End time 2021-01-06T17:20:00Z
Last update 2021-01-06T18:58:22.407Z
More info Users already signed in to the Microsoft Teams web app or desktop client should not have been affected. Final status: We identified a portion of Content Delivery Network (CDN) infrastructure that was not performing as expected and resulting in impact. While we were in the process of analyzing CDN logs to understand why the infrastructure wasn't operating as expected and develop mitigation steps, system monitoring indicated that the service had returned to normal health. We've confirmed that the problem is no longer occurring, and we'll continue to monitor the service to ensure that the problem does not happen again. Users may need to sign out and sign back in to fully remediate the impact. Scope of impact: Impact was specific to a small number of users in North America and Europe that were attempting to access Microsoft Teams from the web app and desktop client. Start time: Wednesday, January 6, 2021, at 4:10 PM'UTC End time: Wednesday, January 6, 2021, at 5:20 PM'UTC Root cause: A portion of CDN infrastructure was not working as intended, resulting in impact. This is the final update for the event.; PublishedTime=2021-01-06T18:58:22.407Z
Root cause A portion of CDN infrastructure was not working as intended, resulting in impact. This is the final update for the event.; PublishedTime=2021-01-06T18:58:22.407Z

TM233064 Details

Status: We're reviewing available diagnostic data, such as Content Delivery Network (CDN) logs, to isolate the source of impact and identify troubleshooting actions. Next update by: Wednesday, January 6, 2021, at 7:00 PMUTC; PublishedTime=2021-01-06T18:00:59.037Z

Final status: We identified a portion of Content Delivery Network (CDN) infrastructure that was not performing as expected and resulting in impact. While we were in the process of analyzing CDN logs to understand why the infrastructure wasn't operating as expected and develop mitigation steps, system monitoring indicated that the service had returned to normal health. We've confirmed that the problem is no longer occurring, and we'll continue to monitor the service to ensure that the problem does not happen again. Users may need to sign out and sign back in to fully remediate the impact. Next update by: Wednesday, January 6, 2021, at 7:00 PMUTC; PublishedTime=2021-01-06T18:00:59.037Z

--------------------------------------------------------------------------------------------------
Title Users may see inaccurate owner information in imported solution flows in a disabled or deactivated state
ID MF234260
Service Microsoft Power Automate in Microsoft 365
Feature Service and web access issues
Status Service restored
Impact Users may have seen inaccurate owner information in imported solution flows in a disabled or deactivated state.
Severity
Start time 2021-01-06T01:10:00Z
End time 2021-01-16T00:30:00Z
Last update 2021-01-16T02:14:50.367Z
More info The owner information may have not been displayed or may have been inaccurate in the flow details page. Final status: We've confirmed that our fix deployment has reached the entirety of the affected environment, mitigating impact. Scope of impact: Users attempting to view owner information in imported solution flows in a disabled or deactivated state may have been impacted. Start time: Wednesday, January 6, 2021, at 1:10 AM'UTC End time: Saturday, January 16, 2021, at 12:30 AM'UTC Root cause: A service update inadvertently contained a code regression, which caused imported solution flows in a disabled or deactivated state to not display owner information or display inaccurate owner information. Next steps: - We're reviewing the recent service update to better understand how impact occurred and to prevent similar impact in the future. This is the final update for the event.; PublishedTime=2021-01-16T02:14:37.58Z
Root cause A service update inadvertently contained a code regression, which caused imported solution flows in a disabled or deactivated state to not display owner information or display inaccurate owner information. Next steps: - We're reviewing the recent service update to better understand how impact occurred and to prevent similar impact in the future. This is the final update for the event.; PublishedTime=2021-01-16T02:14:37.58Z

MF234260 Details