Penn Computing
Computing Menu Computing A-Z
Computing Home Information Systems & Computing Penn

Microsoft's Incident Report for PennO365

Following is a data extract from the PennO365 administrative console of incident reports sent by Microsoft within the last 30 days. We pull out incidents that are relevant to the current PennO365 service offerings and we refresh this data every 15 minutes. The last extract was taken on February 22, 2020 10:30.

--------------------------------------------------------------------------------------------------
Title Can't see file previews
ID YA202430
Service Yammer Enterprise
Feature Yammer Components
Status Service restored
Impact Users may have encountered a blank image when attempting to preview a file in Yammer.
Severity Sev2
Start time 2020-01-28T06:05:00Z
End time 2020-01-28T10:49:00Z
Last update 2020-01-28T11:32:39.063Z
More info
Root cause A recent code change, to enable an access control feature, unexpectedly caused impact.

YA202430 Details

Status: We're analyzing diagnostic data to determine the source of the issue.
Next update by: Tuesday, January 28, 2020, at 12:30 PM UTC; PublishedTime=2020-01-28T10:14:41.157Z

Final status: We identified that a recent code change, to enable an access control feature, unexpectedly caused impact. We reverted the change, and confirmed that the service recovered after monitoring the environment.
Next update by: Tuesday, January 28, 2020, at 12:30 PM UTC; PublishedTime=2020-01-28T10:14:41.157Z

--------------------------------------------------------------------------------------------------
Title Delay of less than 10 minutes when posting a message
ID YA201984
Service Yammer Enterprise
Feature Yammer Components
Status Service restored
Impact Users may have experienced a delay of less than 10 minutes when posting a message.
Severity Sev2
Start time 2020-01-23T02:50:00Z
End time 2020-01-23T04:05:00Z
Last update 2020-01-23T04:39:43.25Z
More info
Root cause A portion of infrastructure which facilitates message posting wasn't performing as efficiently as expected.

YA201984 Details

Status: We're investigating a potential issue and checking for impact to your organization. We'll provide an update within 30 minutes.; PublishedTime=2020-01-23T04:06:59.047

Final status: We've determined that a portion of infrastructure which facilitates message posting wasn't performing as efficiently as expected. We removed the affected component from service and routed traffic through alternate infrastructure to mitigate impact.


--------------------------------------------------------------------------------------------------
Title Can't view emojis and reactions
ID TM204302
Service Microsoft Teams
Feature Teams Components
Status Service restored
Impact Users may have been unable to select emojis or reactions in the Microsoft Teams desktop
Severity Sev2
Start time 2020-02-20T22:30:53Z
End time 2020-02-21T09:58:00Z
Last update 2020-02-21T10:41:14.093Z
More info The emoji and reactions dialog box may have rendered but was not populating with the expected content.
Root cause

TM204302 Details

Status: We're investigating a potential issue and checking for impact to your organization. We'll provide an update within 30 minutes. ; PublishedTime=2020-02-20T22:32:34.65

Status: We're requiring additional Fiddler trace and client logs to effectively troubleshoot this issue. Our support engineers will reach out to you with detailed steps on how to collect this data. Once received, we'll analyze the logs and traces to isolate the source of the issue.
Next update by: Friday, February 21, 2020, at 8:00 PMUTC; PublishedTime=2020-02-20T23:00:55.383Z

Status: We're working to gather additional logging data and reviewing recent service update content to identify the cause of this problem.
Next update by: Friday, February 21, 2020, at 10:00 AM UTC; PublishedTime=2020-02-21T08:18:35.933Z

Status: We've determined that the environment was not configured correctly to accept a new update. We've reconfigured the majority of the environment and users should experience relief once they restart their client.
Next update by: Friday, February 21, 2020, at 12:00 PM UTC; PublishedTime=2020-02-21T10:02:28.57Z

Final status: We've reconfigured all of the affected environment and users should experience relief once they restart their client.
Next update by: Friday, February 21, 2020, at 12:00 PM UTC; PublishedTime=2020-02-21T10:02:28.57Z

--------------------------------------------------------------------------------------------------
Title Can't update presence status
ID TM203694
Service Microsoft Teams
Feature Teams Components
Status Service restored
Impact Users may have been unable to update their presence status within Microsoft Teams.
Severity Sev2
Start time 2020-02-12T13:50:14Z
End time 2020-02-12T14:10:54.883Z
Last update 2020-02-12T14:33:37.987Z
More info
Root cause A section of infrastructure, responsible for maintaining user presence data, was not performing as expected.

TM203694 Details

Final status: We identified that a section of infrastructure, responsible for maintaining user presence data, was performing below acceptable thresholds. We've rerouted user connections to alternate infrastructure, and confirmed that the issue was resolved after monitoring the environment.


--------------------------------------------------------------------------------------------------
Title Can't leverage PSTN; create or join live events, meetings, and group calls
ID TM203663
Service Microsoft Teams
Feature Teams Components
Status False positive
Impact Users may have been unable to leverage Public Switched Telephone Network (PSTN) features.
Severity Sev2
Start time 2020-02-12T01:55:00Z
End time 2020-02-12T02:30:00Z
Last update 2020-02-12T04:41:35.467Z
More info Additionally, users also may have been unable to create or join live events, meetings, and group calls.
Root cause An abnormal increase of service requests to our service, resulted in our infrastructure performing in an inefficient state.

TM203663 Details

Status: We're investigating a potential issue and checking for impact to your organization. We'll provide an update within 30 minutes.; PublishedTime=2020-02-12T02:51:43.14

Status: We determined there's an abnormal increase of service requests to our service, resulting in our infrastructure performing in an inefficient state. We've rerouted traffic to alternative infrastructure to mitigate impact and we're monitoring service health to verify that the issue is remediated.
Next update by: Wednesday, February 12, 2020, at 5:30 AMUTC; PublishedTime=2020-02-12T03:33:01.54Z

Final status: We successfully rerouted traffic to the alternate infrastructure, and have confirmed that the issue is mitigated and service is restored.
Next update by: Wednesday, February 12, 2020, at 5:30 AMUTC; PublishedTime=2020-02-12T03:33:01.54Z

--------------------------------------------------------------------------------------------------
Title Email delays to Teams channels
ID TM203281
Service Microsoft Teams
Feature Teams Components
Status Service restored
Impact Users may have noticed channel update delays for emails directed to Microsoft Teams
Severity Sev2
Start time 2020-02-02T12:00:00Z
End time 2020-02-07T02:41:00Z
Last update 2020-02-07T04:41:13.04Z
More info
Root cause A feature update is inadvertently causing processing capability issues on a portion of mailbox infrastructure, resulting in the delays.

TM203281 Details

Final status: We've confirmed that the feature update is disabled and impact is mitigated.


Status: We're investigating a potential issue and checking for impact to your organization. We'll provide an update within 30 minutes. ; PublishedTime=2020-02-06T18:39:10.617

Status: We've determined that an issue with a newly deployed feature to the Microsoft Teams service is resulting a in an lack of processing capability on mailbox infrastructure and resulting in impact. We've developed a fix for the issue and are following validation procedures before deploying the fix to the affected environment.
Next update by: Thursday, February 6, 2020, at 9:00 PMUTC; PublishedTime=2020-02-06T19:08:07.273Z

Status: We've redirected traffic away from the affected infrastructure to mitigate impact and are monitoring to confirm that the issue no longer persists.
Next update by: Thursday, February 6, 2020, at 11:00 PMUTC; PublishedTime=2020-02-06T20:46:07.143Z

Status: We've confirmed that redirecting traffic away from the affected infrastructure did not yield the expected results. We've developed a new fix that we expect will fully mitigate this issue and suspect that it will be deployed within the next three hours.
Next update by: Friday, February 7, 2020, at 2:30 AMUTC; PublishedTime=2020-02-06T22:52:24.7Z

Status: We've deployed the fix to disable the feature update. We're monitoring its progression throughout the affected infrastructure to ensure the delays are mitigated.
Next update by: Friday, February 7, 2020, at 8:00 AMUTC; PublishedTime=2020-02-07T02:19:45.63Z

--------------------------------------------------------------------------------------------------
Title Can't see new Call Analytics
ID TM203190
Service Microsoft Teams
Feature Teams Components
Status Service restored
Impact Admins may have been unable to see recent Call Analytics data via the Microsoft Teams
Severity Sev2
Start time 2020-02-05T00:00:00Z
End time 2020-02-06T08:32:00Z
Last update 2020-02-06T09:17:17.42Z
More info This issue is affecting data from calls made after Wednesday, February 5, 2020, at 12:00 AM UTC.
Root cause A recent update to improve monitoring capacity inadvertently introduced a code issue that resulted in impact.

TM203190 Details

Status: We've determined that a recent update to improve monitoring capacity inadvertently introduced a code issue that resulted in impact. We're reverting the update to mitigate impact, and we expect the process will complete on Wednesday, February 5, 2020, by 11:00 PM UTC.
Next update by: Wednesday, February 5, 2020, at 11:00 PMUTC; PublishedTime=2020-02-05T19:13:53.55Z

Status: We've reverted the affecting update, and are now processing the Call Analytics data that was previously unavailable so admins will be able to view it as expected. We'll provide an updated estimated completion time for this process once available.
Next update by: Thursday, February 6, 2020, at 1:00 AMUTC; PublishedTime=2020-02-05T21:10:56.737Z

Status: We're continuing processing of the affected data, and estimate that processing will complete in four hours.
Next update by: Thursday, February 6, 2020, at 3:30 AMUTC; PublishedTime=2020-02-05T21:42:51.883Z

Status: An unexpected volume of incoming calls has delayed the processing of affected data. We're continuing to process the affected data to mitigate impact, and users hosted through infrastructure where processing has completed may begin to see relief.
Next update by: Thursday, February 6, 2020, at 7:30 AMUTC ; PublishedTime=2020-02-06T02:04:04.463Z

Status: We've determined that call volume has dropped back to expected levels. We're continuing to process the affected data to mitigate impact, and users hosted through infrastructure where processing has completed may begin to see relief.
Next update by: Thursday, February 6, 2020, at 12:00 PMUTC; PublishedTime=2020-02-06T06:13:35.593Z

Final status: We've finished processing the remaining data, and confirmed that the service has recovered after monitoring the environment.
Next update by: Thursday, February 6, 2020, at 12:00 PMUTC; PublishedTime=2020-02-06T06:13:35.593Z

Status: We're investigating a potential issue and checking for impact to your organization. We'll provide an update within 30 minutes.; PublishedTime=2020-02-05T18:55:46.91Next update by: Thursday, February 6, 2020, at 12:00 PMUTC; PublishedTime=2020-02-06T06:13:35.593Z

--------------------------------------------------------------------------------------------------
Title Can't access Microsoft Teams
ID TM202916
Service Microsoft Teams
Feature Teams Components
Status Service restored
Impact Users may have been unable to access Microsoft Teams.
Severity Sev2
Start time 2020-02-03T13:44:00Z
End time 2020-02-03T20:00:00Z
Last update 2020-02-10T18:53:17.863Z
More info Reports of impact to other Teams features such as chat history visibility and Auto Attendant functionality were related to this authentication issue.
Root cause An authentication certificate expired, which impacted any user with an expired authentication token, or any user trying to login and authenticate from a logged-out state.

TM202916 Details

Status: We're investigating an issue where users may be unable to access Microsoft Teams. We're reviewing systems data to determine the cause of the issue.
Next update by: Monday, February 3, 2020, at 3:30 PM UTC; PublishedTime=2020-02-03T14:13:18.277Z

Status: We've determined that an authentication certificate has expired causing users who have logged out and those that are still logged in to have issue using the service. We're developing a fix to apply a new authentication certificate to the service which will remediate impact.
Next update by: Monday, February 3, 2020, at 4:30 PM UTC; PublishedTime=2020-02-03T15:03:33.383Z

Status: We've deployed the fix to roll out a new authentication certificate, which has completed. We're monitoring the service, however, affected users should start to see recovery in the next hour.
Next update by: Monday, February 3, 2020, at 5:30 PM UTC; PublishedTime=2020-02-03T15:58:02.103Z

Status: We've deployed the new authentication certificate to mitigate impact. We're currently performing targeted component restarts across our infrastructure to ensure the fix is applied to all affected customer environments. Additionally, we're investigating reports of impact to other Teams features such as chat history visibility and Auto Attendant functionality that may arise as our fix propagates.
Next update by: Monday, February 3, 2020, at 7:00 PM UTC; PublishedTime=2020-02-03T17:31:11.837Z

Status: We've deployed the new authentication certificate to mitigate impact and continue to perform targeted component restarts across our infrastructure to ensure the fix is applied to all affected customer environments.
Next update by: Monday, February 3, 2020, at 8:00 PM UTC; PublishedTime=2020-02-03T19:02:26.173Z

Status: Our telemetry continues to indicate the service has largely recovered; however we're seeing residual impact in some specific regions. To fully remediate the issue, we're moving select traffic to alternative infrastructure and continuing to perform targeted component restarts. We've previously deployed the new authentication certificate to mitigate impact.
Next update by: Monday, February 3, 2020, at 10:00 PM UTC; PublishedTime=2020-02-03T20:06:25.32Z

Final status: Our telemetry indicates the service largely recovered around 4:30 PM UTC after the fix to an authentication certificate was deployed across the global environment. Following this remediation action, we identified and resolved some residual impact for users hosted in North America and Northern Europe regions, by performing targeted component restarts and redirecting a portion of traffic.
Next update by: Monday, February 3, 2020, at 10:00 PM UTC; PublishedTime=2020-02-03T20:06:25.32Z

--------------------------------------------------------------------------------------------------
Title Delayed notifications, activity emails and invite emails
ID TM202536
Service Microsoft Teams
Feature Teams Components
Status Service restored
Impact Users may be seeing delays of up to three hours in getting Teams notifications, activity
Severity Sev2
Start time 2020-01-29T15:00:00Z
End time 2020-01-30T00:03:00Z
Last update 2020-01-30T00:12:31.583Z
More info
Root cause A portion of infrastructure that facilitates the timely processing of Teams app and client notifications, as well as activity and team invite emails, wasn't processing requests as expected.

TM202536 Details

Status: We've determined that a portion of infrastructure that facilitates the timely processing of Teams app and client notifications, as well as activity and team invite emails, isn't processing requests as expected. We're rebalancing resources in the affected environment to mitigate impact.
Start time: Wednesday, January 29, 2020, at 3:00 PM'UTC

Root cause: A portion of infrastructure that facilitates the timely processing of Teams app and client notifications, as well as activity and team invite emails, isn't processing requests as expected.

Next update by: Wednesday, January 29, 2020, at 6:00 PMUTC; PublishedTime=2020-01-29T15:58:21.387Next update by: Wednesday, January 29, 2020, at 6:00 PMUTC; PublishedTime=2020-01-29T15:58:21.387Z

Status: Our traffic rebalancing efforts are continuing. In addition, we've restarted the Domain Controllers (DC) responsible for directing the affected requests; however, notification and email queues still remain.
Next update by: Wednesday, January 29, 2020, at 8:00 PMUTC; PublishedTime=2020-01-29T17:53:02.53Z

Status: The process of rebalancing resources is taking longer than expected. We've added additional resources to a subset of the impacted environment and are monitoring to confirm that it effectively processes the queued items.
Next update by: Wednesday, January 29, 2020, at 10:00 PMUTC; PublishedTime=2020-01-29T20:14:15.09Z

Status: We're continuing to monitor as the queued items undergo processing. We anticipate all queues fully processing within two hours.
Next update by: Thursday, January 30, 2020, at 12:00 AMUTC; PublishedTime=2020-01-29T21:47:07.303Z

Status: We've confirmed that the notifications queue has completed processing, while email queues continue. The email queues are taking longer than anticipated, but we've performed additional optimizations and the remaining queues are processing faster now. We anticipate resolution within two hours.
Next update by: Thursday, January 30, 2020, at 2:00 AMUTC; PublishedTime=2020-01-29T23:42:56.32Z

Final status: We've successfully processed the remaining email queues and confirmed via telemetry that impact is mitigated.
Next update by: Thursday, January 30, 2020, at 2:00 AMUTC; PublishedTime=2020-01-29T23:42:56.32Z

--------------------------------------------------------------------------------------------------
Title Message and email notification delays
ID TM202444
Service Microsoft Teams
Feature Teams Components
Status Service restored
Impact Users may have experienced delays of up to 45 minutes when receiving message
Severity Sev2
Start time 2020-01-27T14:00:00Z
End time 2020-01-29T00:08:00Z
Last update 2020-01-29T00:17:31.893Z
More info Additionally, invite and reporting emails may have been delayed up to four hours.
Root cause A recent feature update resulted in insufficient resource allocation causing the impact.

TM202444 Details

Status: We're analyzing Chat Service (CS) logs to help determine the source of impact.
Next update by: Tuesday, January 28, 2020, at 5:30 PM UTC; PublishedTime=2020-01-28T15:25:10.803Z

Status: We've identified a Remote Dictionary Server (redis) caching issue caused by insufficient resource allocation during a scheduled deployment that wasn't expected to cause impact. We're rebalancing traffic to alleviate impact.
Next update by: Tuesday, January 28, 2020, at 7:30 PMUTC; PublishedTime=2020-01-28T17:25:46.747Z

Status: We've completed rebalancing traffic across the environment, which has mitigated the Teams notification delay. We're reviewing system telemetry to identify the best mitigation actions to resolve the email delays.
Next update by: Tuesday, January 28, 2020, at 9:30 PMUTC; PublishedTime=2020-01-28T19:10:06.853Z

Status: We've determined that that the resource allocation also impacted the infrastructure that facilitates email notifications and are rebalancing that traffic to fully resolve the problem.
Next update by: Tuesday, January 28, 2020, at 11:30 PMUTC; PublishedTime=2020-01-28T21:26:45.287Z

Status: We've completed additional traffic rebalancing and are monitoring as the remaining emails are processed.
Next update by: Wednesday, January 29, 2020, at 1:30 AMUTC; PublishedTime=2020-01-28T23:21:22.137Z

Final status: We've confirmed via monitoring that all queues have processed, and impact is completely mitigated.
Next update by: Wednesday, January 29, 2020, at 1:30 AMUTC; PublishedTime=2020-01-28T23:21:22.137Z

--------------------------------------------------------------------------------------------------
Title Can't find user to mention in channels
ID TM202387
Service Microsoft Teams
Feature Teams Components
Status Service restored
Impact Users may intermittently not appear in search when attempting to mention in channels.
Severity Sev2
Start time 2020-01-25T15:30:00Z
End time 2020-01-27T21:54:00Z
Last update 2020-01-27T22:09:58.113Z
More info
Root cause A recent build deployment inadvertently introduced a regression to a portion of user search index infrastructure.

TM202387 Details

Status: We're aware of an issue impacting the Microsoft Teams service. We've collected multiple logs and confirmed after review that a recent build deployment inadvertently introduced a regression to a portion of user search index infrastructure, which resulted in impact. We're working to reroute traffic to alternate infrastructure while we repair the affected machines.
Next update by: Monday, January 27, 2020, at 10:30 PMUTC; PublishedTime=2020-01-27T21:28:11.413Z

Final status: We've completed our efforts to reroute traffic to alternate infrastructure, and have confirmed that impact is now mitigated for all users.
Next update by: Monday, January 27, 2020, at 10:30 PMUTC; PublishedTime=2020-01-27T21:28:11.413Z

--------------------------------------------------------------------------------------------------
Title Message/notification delays and stale presence data
ID TM202160
Service Microsoft Teams
Feature Teams Components
Status False positive
Impact Users may notice messages and notifications delayed up to 15 minutes and stale presence
Severity Sev2
Start time 2020-01-24T21:58:01Z
End time 2020-01-24T22:52:49Z
Last update 2020-01-24T23:34:45.473Z
More info
Root cause

TM202160 Details

Status: We're investigating a potential issue and checking for impact to your organization. We'll provide an update within 30 minutes.; PublishedTime=2020-01-24T21:58:27.29

Status: We're reviewing system telemetry to identify the cause.
Next update by: Friday, January 24, 2020, at 11:30 PMUTC; PublishedTime=2020-01-24T22:24:43.987Z

Final status: This service incident number is a duplicate of an existing issue.; PublishedTime=2020-01-24T23:34:12.67Next update by: Friday, January 24, 2020, at 11:30 PMUTC; PublishedTime=2020-01-24T22:24:43.987Z

--------------------------------------------------------------------------------------------------
Title Delays receiving message notifications
ID TM202088
Service Microsoft Teams
Feature Teams Components
Status Service restored
Impact Users may have experienced a 10-60 second delay receiving message notifications from
Severity Sev2
Start time 2020-01-23T23:00:00Z
End time 2020-01-24T01:10:00Z
Last update 2020-01-24T01:59:08.283Z
More info
Root cause A subset of infrastructure wasn't communicating with the necessary components to effectively process notification requests as intended.

TM202088 Details

Status: Our monitoring has alerted us of a subset of infrastructure performing below acceptable thresholds, resulting in delays. We're working to remove the impacted infrastructure from service and reroute impacted traffic to healthy infrastructure.
Next update by: Friday, January 24, 2020, at 3:00 AMUTC; PublishedTime=2020-01-24T01:06:07.427Z

Final status: We've successfully removed the impacted infrastructure from service and monitoring confirmed notifications are successfully being sent.
Next update by: Friday, January 24, 2020, at 3:00 AMUTC; PublishedTime=2020-01-24T01:06:07.427Z

--------------------------------------------------------------------------------------------------
Title Can't sign in to Teams
ID TM202028
Service Microsoft Teams
Feature Teams Components
Status Post-incident report published
Impact Users may have been unable to sign in to the Microsoft Teams service.
Severity Sev1
Start time 2020-01-23T13:13:00Z
End time 2020-01-23T17:30:00Z
Last update 2020-02-02T10:40:47.077Z
More info Additionally, users who were already signed in may have experienced message delays when receiving chat messages, and users joining meetings via link could also be impacted.
Root cause

TM202028 Details

Status: We're reviewing failure alerts to help us determine the cause of impact.
Next update by: Thursday, January 23, 2020, at 4:30 PM UTC ; PublishedTime=2020-01-23T14:17:28.223Z

Status: We've identified a Domain Naming Service (DNS) issue within one of our datacenters, which caused the chat service to fail. We've initiated a failover to an alternate infrastructure and are monitoring the service while it recovers.
Next update by: Thursday, January 23, 2020, at 4:30 PM UTC; PublishedTime=2020-01-23T15:06:48.457Z

Status: We've completed the failover to alternate components and this has mitigated some impact. However, we've identified that some client-side telemetry is showing errors and we're reviewing systems diagnostics to isolate any additional impact.
Next update by: Thursday, January 23, 2020, at 6:00 PM UTC; PublishedTime=2020-01-23T16:27:42.117Z

Final status: We've identified that some networking components, which route traffic between data centres, was experiencing impact. We've removed the impacted networking components, which rerouted traffic through redundant systems, and we've monitored the environment to confirm that the issue is resolved. Additionally, we've received confirmation from some users who previously reported impact that the issue has been remediated.
Next update by: Thursday, January 23, 2020, at 6:00 PM UTC; PublishedTime=2020-01-23T16:27:42.117Z

--------------------------------------------------------------------------------------------------
Title New and changed query rules used within search queries are slow to take effect
ID SP203888
Service SharePoint Online
Feature Search and Delve
Status Service degradation
Impact Users may not see best bets or promoted content for up to 24 hours after a query rules
Severity Sev2
Start time 2020-02-14T04:34:34Z
End time
Last update 2020-02-21T12:23:56.657Z
More info Previously defined query rules do work as expected during the 24 hour propagation delay. There is no effect on freshness of document.
Root cause

SP203888 Details

Status: Our initial investigation indicates that our query processors are failing to update the results in a timely manner. We're working to implement additional logging capabilities within our service to better understand how the issue is occurring so that we can formulate an action plan for full remediation.
Next update by: Friday, February 14, 2020, at 9:00 PMUTC; PublishedTime=2020-02-14T04:39:04.287Z

Status: We've implemented additional logging to identify why query rule changes are not being propagated quickly. Additionally, we're implementing a further workaround to reduce the refresh time from 24 hours to 1 hour until the root cause is identified and resolved.
Next update by: Friday, February 14, 2020, at 10:00 PM UTC; PublishedTime=2020-02-14T13:34:42.563Z

Status: We've developed a fix to reduce the refresh time to one hour. The fix is being tested and validated before it can be deployed to the impacted environment, which we estimate will occur by midweek, next week.
Next update by: Monday, February 17, 2020, at 2:00 PMUTC; PublishedTime=2020-02-14T20:36:44.82Z

Status: As part of the deployment process, the fix is going through an internal test environment for further validation. Once this validation has completed, we'll begin deploying it to the affected components. The deployment to the affected infrastructure is expected to begin on Wednesday, February 19, 2020.
Next update by: Wednesday, February 19, 2020, at 2:00 PM UTC; PublishedTime=2020-02-17T12:28:41.55Z

Status: Due to an unforeseen issue caused by a misconfiguration with generating new updates, we've been unable to deploy the fix to the internal test environment. The misconfiguration with generating new updates has been corrected and the update will begin deploying to the internal test environment on Thursday, February 20, 2020. Once this update has been deployed, we'll validate the fix before deploying it to the affected infrastructure.
Next update by: Friday, February 21, 2020, at 2:00 PM UTC; PublishedTime=2020-02-19T12:12:39.22Z

Status: We've validated that the short term fix, which reduces the refresh time to one hour, provides the intended relief within our test environment. We're preparing to deploy this fix to the affected infrastructure and expect that the deployment will take approximately one week to complete. Users may begin to experience relief as the fix propagates throughout the environment. Additionally, we're continuing to investigate the root cause and establish a fix to remediate impact fully.
Next update by: Wednesday, February 26, 2020, at 2:00 PM UTC; PublishedTime=2020-02-21T12:23:56.657Z

--------------------------------------------------------------------------------------------------
Title Can't change SharePoint Online communication site themes
ID SP203268
Service SharePoint Online
Feature SharePoint Features
Status Service restored
Impact Users may have been unable to change the theme on newly created communication sites in
Severity Sev2
Start time 2020-01-27T21:46:00Z
End time 2020-02-07T03:23:00Z
Last update 2020-02-07T04:40:51.877Z
More info
Root cause A recent service update introduced a code issue that was preventing users from changing the theme on newly created communications sites in SharePoint Online.

SP203268 Details

Status: We're analyzing diagnostic data to determine the source of the issue.
Next update by: Thursday, February 6, 2020, at 5:00 PM UTC; PublishedTime=2020-02-06T14:52:25.353Z

Status: Following our analysis into diagnostic data, we've identified a recent service update that has introduced a code issue which is preventing changes to the theme for newly created communication sites in SharePoint Online. We're developing a fix that will revert the update and remediate impact.
Next update by: Thursday, February 6, 2020, at 8:00 PMUTC ; PublishedTime=2020-02-06T16:30:54.253Z

Status: We're preparing to deploy the fix to the affected environment, which is expected to complete within two hours.
Next update by: Thursday, February 6, 2020, at 11:30 PM UTC; PublishedTime=2020-02-06T18:33:25.6Z

Status: The deployment of the solution was delayed due to unforeseen technical issues. We've corrected these issues and are initiating our deployment of the fix. We'll provide a timeline for completion of the deployment with our next update.
Next update by: Friday, February 7, 2020, at 2:00 AMUTC; PublishedTime=2020-02-06T21:49:15.163Z

Status: We have completed validation and testing of our fix for the issue in preparation for deployment. Once the fix has completed saturating, we'll monitor telemetry for the affected environment to confirm that change requests are processing as expected.
Next update by: Friday, February 7, 2020, at 8:00 PMUTC ; PublishedTime=2020-02-07T00:40:15.123Z

Final status: We validated the fix and confirmed that the deployment completed saturating the affected environment. We confirmed via our internal monitoring systems that the impact was remediated.
Next update by: Friday, February 7, 2020, at 8:00 PMUTC ; PublishedTime=2020-02-07T00:40:15.123Z

--------------------------------------------------------------------------------------------------
Title Unable to render SharePoint Online Classic UI pages
ID SP202628
Service SharePoint Online
Feature SharePoint Features
Status Post-incident report published
Impact Users may have been unable to render SharePoint Online Classic UI pages.
Severity Sev1
Start time 2020-01-30T14:01:00Z
End time 2020-01-30T17:46:00Z
Last update 2020-02-06T17:59:36.317Z
More info Some users may have seen blank pages when trying to load SharePoint Online Classic UI. Additionally, users may have been unable to render Project Online pages as well.
Root cause A configuration error by our third-party CDN provider resulted in impact to SharePoint Online and Project Online Classic UI page rendering.

SP202628 Details

Status: We're investigating a potential issue with rendering SharePoint Online pages. We'll provide an update within 30 minutes.; PublishedTime=2020-01-30T14:18:53.727

Status: We're reviewing diagnostic data to isolate the source of the issue.
Next update by: Thursday, January 30, 2020, at 4:00 PM UTC; PublishedTime=2020-01-30T14:34:28.893Z

Status: We've reproduced the issue and have gathered network trace logs for analysis.
Next update by: Thursday, January 30, 2020, at 4:30 PM UTC; PublishedTime=2020-01-30T15:06:27.843Z

Status: We've identified a potential issue with a third-party Content Delivery Network (CDN) provider. We're working with the CDN provider to investigate the issue.
Next update by: Thursday, January 30, 2020, at 5:00 PM UTC; PublishedTime=2020-01-30T15:39:51.747Z

Status: We've re-routed traffic away from our third-party Content Delivery Network (CDN) provider to our internal CDN service, while our third-party CDN works to deploy a permanent fix for the issue. Additionally, we've halted the current SPO deployment to avoid further cache invalidation issues and prevent the spread of impact. We're reaching out to reporting users to confirm if the issue is resolved.
Next update by: Thursday, January 30, 2020, at 6:00 PM UTC; PublishedTime=2020-01-30T16:42:50.897Z

Final status: We identified a configuration issue within the third-party Content Delivery Network (CDN) provider that caused impact to the service. Our actions to re-route the impacted traffic was successful in mitigating impact and we've confirmed that the issue is no longer occurring with multiple users. Engineers will continue to monitor the service and work with our third-party CDN to ensure mitigation.
Next update by: Thursday, January 30, 2020, at 6:00 PM UTC; PublishedTime=2020-01-30T16:42:50.897Z

--------------------------------------------------------------------------------------------------
Title No search results on Office.com
ID SP201994
Service SharePoint Online
Feature Search and Delve
Status Service restored
Impact Users may not have received results when searching on Office.com.
Severity Sev2
Start time 2020-01-23T00:30:00Z
End time 2020-01-23T08:35:00Z
Last update 2020-01-23T09:13:53.997Z
More info
Root cause An updated intended to improve search performance contained a code regression, inadvertently causing impact.

SP201994 Details

Status: We're investigating a potential issue and checking for impact to your organization. We'll provide an update within 30 minutes. ; PublishedTime=2020-01-23T06:18:08.79

Status: We are investigating a recent update to Office.com to try to determine if it is the source of the impact and determine the next troubleshooting steps.
Next update by: Thursday, January 23, 2020, at 9:30 AMUTC; PublishedTime=2020-01-23T06:45:29.547Z

Final status: We've confirmed that the update to Office.com, which was intended to improve performance for native search users, contained a code regression. We've reverted this update and confirmed the issue is resolved after monitoring the environment.
Next update by: Thursday, January 23, 2020, at 9:30 AMUTC; PublishedTime=2020-01-23T06:45:29.547Z

--------------------------------------------------------------------------------------------------
Title Can't create Planner tab in Microsoft Teams
ID PL203237
Service Planner
Feature Planner
Status Service restored
Impact Users may be unable to create Planner tabs in Microsoft Teams.
Severity Sev2
Start time 2020-02-05T23:40:00Z
End time 2020-02-06T11:11:00Z
Last update 2020-02-06T11:41:10.913Z
More info
Root cause A recent code change to the Planner service caused a mismatch between the client and server side, resulting in impact.

PL203237 Details

Status: We've determined that a recent code change to the Planner service caused a mismatch between the client and server side, resulting in impact. We've developed a fix and validated a fix in our test environment. The fix has begun deployment to the affetced infrastructure.
Next update by: Thursday, February 6, 2020, at 9:00 AMUTC; PublishedTime=2020-02-06T07:20:30.97Z

Status: We're monitoring the progress of the fix as it propagates throughout the impacted infrastructure.
Next update by: Thursday, February 6, 2020, at 1:00 PM UTC; PublishedTime=2020-02-06T08:58:25.663Z

Final status: We've confirmed that the fix has completed and monitored the service to confirm resolution.
Next update by: Thursday, February 6, 2020, at 1:00 PM UTC; PublishedTime=2020-02-06T08:58:25.663Z

--------------------------------------------------------------------------------------------------
Title Delays syncing data
ID OE203544
Service School Data Sync
Feature School Data Sync
Status Service restored
Impact Users may have experienced delays when syncing data into Azure Active Directory (AAD).
Severity Sev2
Start time 2020-01-22T19:15:00Z
End time 2020-02-13T19:30:00Z
Last update 2020-02-13T19:38:56.487Z
More info To prevent further sync delays, we recommended users refrain from pressing the "Reset Sync" button.
Root cause A recent update to improve logging functionality introduced a code issue that was resulting in sync delays.

OE203544 Details

Status: We're aware of an issue affecting users' ability to sync data from their on-premises environment to the AAD. We're reviewing server logs to determine potential sources of impact.
Next update by: Tuesday, February 11, 2020, at 3:00 AM UTC; PublishedTime=2020-02-10T22:23:24.25Z

Status: We're analyzing recent AAD code changes to determine if any of them may be causing the issue.
Next update by: Tuesday, February 11, 2020, at 8:30 PMUTC; PublishedTime=2020-02-11T01:42:27.057Z

Status: On further review, we've determined that users are syncing data into AAD; however, this sync process is severely delayed. We've determined that a recent update to improve logging functionality introduced a code issue that is resulting in the sync delays. We're evaluating methods for improving the log data flow that's impacting the sync to mitigate impact.
Next update by: Wednesday, February 12, 2020, at 8:30 PMUTC; PublishedTime=2020-02-11T20:51:02.903Z

Status: We've developed a code fix, which we're testing in our internal environment. Once validated, we'll confirm when it will begin deployment to the affected environment.
Next update by: Thursday, February 13, 2020, at 3:00 AM UTC; PublishedTime=2020-02-12T20:00:36.41Z

Status: We've finished validating the code fix, so now we're in the process of deploying it to the affected environment.
Next update by: Thursday, February 13, 2020, at 9:30 PM UTC; PublishedTime=2020-02-13T01:13:27.77Z

Final status: We deployed the fix to the affected environment, and after additional manual optimizations, we've confirmed via telemetry that impact is mitigated.
Next update by: Thursday, February 13, 2020, at 9:30 PM UTC; PublishedTime=2020-02-13T01:13:27.77Z

--------------------------------------------------------------------------------------------------
Title Unable to render OneDrive for Business Classic UI pages
ID OD202633
Service OneDrive for Business
Feature OneDrive for Business
Status Post-incident report published
Impact Users may have been unable to render OneDrive for Business Classic UI pages.
Severity Sev2
Start time 2020-01-30T14:01:00Z
End time 2020-01-30T17:46:00Z
Last update 2020-02-06T17:59:50.513Z
More info Some users may have seen blank pages when trying to load OneDrive for Business Classic pages. Additionally, users may have been unable to render Project Online pages.
Root cause A configuration error by our third-party CDN provider resulted in impact to OneDrive for Business page rendering.

OD202633 Details

Status: We've identified a potential issue with a third-party Content Delivery Network (CDN) provider. We're working with the CDN provider to investigate the issue.
Next update by: Thursday, January 30, 2020, at 5:00 PM UTC; PublishedTime=2020-01-30T15:42:51.157Z

Status: We've re-routed traffic away from our third-party Content Delivery Network (CDN) provider to our internal CDN service, while our third-party CDN works to deploy a permanent fix for the issue. Additionally, we've halted the current OneDrive deployment to avoid further cache invalidation issues and prevent the spread of impact. We're reaching out to reporting users to confirm if the issue is resolved.
Next update by: Thursday, January 30, 2020, at 6:00 PM UTC; PublishedTime=2020-01-30T16:50:25.03Z

Final status: We identified configuration issue within the third-party Content Delivery Network (CDN) provider caused impact to the service. Our actions to re-route the impacted traffic was successful in mitigating impact and we've confirmed that the issue is no longer occurring with multiple users. Engineers will continue to monitor the service and work with our third-party CDN to ensure mitigation.
Next update by: Thursday, January 30, 2020, at 6:00 PM UTC; PublishedTime=2020-01-30T16:50:25.03Z

--------------------------------------------------------------------------------------------------
Title Issues with Microsoft 365 services
ID MO204473
Service Office 365 Portal
Feature Portal
Status False positive
Impact Users and admins may experience various issues when using a number of the Microsoft 365
Severity Sev2
Start time 2020-02-22T01:21:50Z
End time 2020-02-22T01:24:54Z
Last update 2020-02-22T01:25:14.613Z
More info SharePoint Online and OneDrive for Business: Users are unable to access SharePoint Online sites or OneDrive for Business content.
Root cause A recent change made to an Azure networking component has inadvertently caused impact to service.

MO204473 Details

Status: Our service health monitoring indicates that a number of our Azure infrastructure are performing in a suboptimal state, resultng in impact. Our investigation has determined that a recent change made to an Azure networking component has inadvertently caused impact to service. We're reverting the change to mitigate impact.
Next update by: Saturday, February 22, 2020, at 3:00 AM UTC; PublishedTime=2020-02-22T01:24:12.653Z

--------------------------------------------------------------------------------------------------
Title Issues with Microsoft 365 services
ID MO204472
Service Office 365 Portal
Feature Portal
Status Service restored
Impact Users and admins may have experienced various issues when using a number of the
Severity Sev2
Start time 2020-02-21T21:47:00Z
End time 2020-02-22T00:56:00Z
Last update 2020-02-22T02:54:04.47Z
More info SharePoint Online and OneDrive for Business: Users were unable to access SharePoint Online sites or OneDrive for Business content.
Root cause A recent change made to an Azure networking component has inadvertently caused impact to service.

MO204472 Details

Status: Our service health monitoring indicates that a number of our Azure infrastructure are performing in a suboptimal state, resultng in impact. Our investigation has determined that a recent change made to an Azure networking component has inadvertently caused impact to service. We're reverting the change to mitigate impact.
Next update by: Saturday, February 22, 2020, at 3:00 AM UTC; PublishedTime=2020-02-22T01:22:50.817Z

Final status: We've reverted the Azure networking change and confirmed through our service health monitoring that the issue is mitigated.
Next update by: Saturday, February 22, 2020, at 3:00 AM UTC; PublishedTime=2020-02-22T01:22:50.817Z

--------------------------------------------------------------------------------------------------
Title Can't pull Office 365 Auditing records
ID MO204370
Service Office 365 Portal
Feature Administration
Status Service restored
Impact Admins may have been unable to pull Office 365 Auditing records from the Office 365
Severity Sev2
Start time 2020-02-21T12:30:00Z
End time 2020-02-21T18:30:00Z
Last update 2020-02-21T19:54:48.323Z
More info
Root cause An unexpected spike in traffic caused a subset of web service infrastructure to handle requests inefficiently.

MO204370 Details

Status: We've identified that an unexpected spike in traffic caused a section of web service infrastructure to handle requests inefficiently. We've optimized traffic management parameters on the affected systems and the service has recovered. We're monitoring the environment to confirm there is no residual impact.
Next update by: Friday, February 21, 2020, at 3:00 PM UTC; PublishedTime=2020-02-21T13:12:40Z

Status: We're continuing to monitor the environment to confirm impact has been remediated, after optimizing traffic management parameters.
Next update by: Friday, February 21, 2020, at 5:00 PM UTC ; PublishedTime=2020-02-21T14:56:33.58Z

Status: While optimizing the traffic management parameters, we've discovered a code issue that causes the affected infrastructure to stop processing traffic when service load gets high. We're rebalancing service traffic to alleviate system pressure and remediate impact while we further investigate the underlying code problem.
Next update by: Friday, February 21, 2020, at 7:00 PMUTC; PublishedTime=2020-02-21T17:04:09.843Z

Final status: We've rebalanced the service traffic in the affected environment and impact has been remediated.
Next update by: Friday, February 21, 2020, at 7:00 PMUTC; PublishedTime=2020-02-21T17:04:09.843Z

--------------------------------------------------------------------------------------------------
Title Search suggestions issue
ID MO204364
Service Office 365 Portal
Feature Portal
Status Service degradation
Impact Users may be intermittently unable to see search suggestions from the Search bar.
Severity Sev2
Start time 2020-02-21T05:30:00Z
End time
Last update 2020-02-22T07:03:28.003Z
More info Impact is seen in, but not limited to, the following services: - Microsoft Teams - SharePoint Online - Outlook client and Outlook on the web - Microsoft 365 admin center
Root cause A recent search feature update was incompatible with other components, causing impact.

MO204364 Details

Status: We've identified that a recent update intended to improve service functionality, has inadvertently caused impact. We're reverting this update to remediate impact. Additionally, we're reviewing the contents of the update to identify the root cause.
Next update by: Friday, February 21, 2020, at 7:30 PM UTC ; PublishedTime=2020-02-21T11:40:03.163Z

Status: After further review of the offending update, we determined that the search feature was updated at a different rate from other components, resulting in impact. We're continuing our rollback and monitoring as it progresses. We expect the rollback to complete within the next 12 hours.
Next update by: Saturday, February 22, 2020, at 8:00 AMUTC; PublishedTime=2020-02-21T18:23:51.677Z

Status: Our monitoring shows that the rollback has completed to approximately 97 percent of the affected infrastructure. We're continuing to monitor the remaining infrastructure to ensure the issue is fully remediated.
Next update by: Saturday, February 22, 2020, at 6:00 PMUTC ; PublishedTime=2020-02-22T07:03:28.003Z

--------------------------------------------------------------------------------------------------
Title Can't add or edit payment methods or complete purchases in the Microsoft 365 admin center
ID MO204296
Service Office 365 Portal
Feature Purchase and Billing
Status Service restored
Impact Admins may have been unable to add or edit payment methods or complete purchases within
Severity Sev2
Start time 2020-02-20T19:55:00Z
End time 2020-02-20T22:10:00Z
Last update 2020-02-20T22:37:21.707Z
More info
Root cause A subset of billing infrastructure experienced a configuration issue which dropped it below service performance thresholds, preventing admins from adding or editing payment methods or completing purchases within the Microsoft 365 admin center.

MO204296 Details

Status: We're investigating a potential issue and checking for impact to your organization. We'll provide an update within 30 minutes. ; PublishedTime=2020-02-20T21:17:03.007

Status: We've identified a subset of billing infrastructure that has fallen below our service performance thresholds, which is preventing the editing of payment methods and the ability to complete purchases for admins within the Microsoft 365 admin center. We're rerouting traffic to alternate infrastructure, which is expected to remediate impact.
Next update by: Thursday, February 20, 2020, at 11:30 PMUTC ; PublishedTime=2020-02-20T21:41:18.163Z

Final status: Following the rerouting of traffic to alternate infrastructure, we've confirmed that impact has been resolved.
Next update by: Thursday, February 20, 2020, at 11:30 PMUTC ; PublishedTime=2020-02-20T21:41:18.163Z

--------------------------------------------------------------------------------------------------
Title Can't access Microsoft 365 services
ID MO204281
Service Office 365 Portal
Feature Portal
Status Service restored
Impact Users may have been unable to access the Microsoft 365 portal or Microsoft 365 services.
Severity Sev2
Start time 2020-02-20T16:17:53Z
End time 2020-02-20T19:15:00Z
Last update 2020-02-20T20:09:02.587Z
More info
Root cause We observed a substantial increase in user requests within a portion of our traffic management infrastructure, which resulted in some intermittent access issues.

MO204281 Details

Status: Our telemetry indicates the service is recovering without intervention, and we're continuing to monitor the stability of the network until the issue is resolved. We're analyzing system logs to determine the root cause.
Next update by: Thursday, February 20, 2020, at 6:30 PM UTC; PublishedTime=2020-02-20T16:49:39.84Z

Status: We've identified that a subset of traffic management infrastructure responsible for handling user access requests was performing below acceptable thresholds. Our automated recovery systems restarted the affected infrastructure and these actions have recovered some of the service. However, we're receiving reports that some users are still experiencing impact. We're reviewing system telemetry to determine our next troubleshooting steps.
Next update by: Thursday, February 20, 2020, at 8:30 PM UTC; PublishedTime=2020-02-20T18:23:51.593Z

Final status: We identified that a subset of traffic management infrastructure responsible for handling user access requests was performing below acceptable thresholds. Our automated recovery systems restarted the affected infrastructure and these actions recovered the service. Some residual impact was observed as traffic was rerouted through load-balancers responsible for maintaining healthy traffic levels; however, our telemetry indicates the the service has now fully recovered.
Next update by: Thursday, February 20, 2020, at 8:30 PM UTC; PublishedTime=2020-02-20T18:23:51.593Z

--------------------------------------------------------------------------------------------------
Title Unable to access Microsoft 365 services
ID MO203839
Service Office 365 Portal
Feature Portal
Status Service restored
Impact Users may have been unable to access Microsoft 365 services.
Severity Sev2
Start time 2020-02-13T14:50:00Z
End time 2020-02-13T16:45:00Z
Last update 2020-02-13T18:36:59.223Z
More info Users may have been unable to access the following services: - Microsoft Teams - SharePoint Online.
Root cause A section of infrastructure, responsible for processing access requests, was handling user requests inefficiently.

MO203839 Details

Status: We're investigating a potential issue and checking for impact to your organization. We'll provide an update within 30 minutes.; PublishedTime=2020-02-13T15:30:10.427

Status: We're investigating system logs and recent changes within the affected infrastructure to identify the underlying cause and remediate impact.
Next update by: Thursday, February 13, 2020, at 5:00 PM UTC; PublishedTime=2020-02-13T15:49:03.54Z

Status: We've identified a potential issue with a section of infrastructure responsible for processing access requests. We're analyzing service logs from these components to isolate the source of the problem. In parallel, we're reviewing options to route user connections to alternate components to remediate impact.
Next update by: Thursday, February 13, 2020, at 7:00 PM UTC; PublishedTime=2020-02-13T16:45:06.203Z

Final status: We've reconnected the service to alternate systems and confirmed that impact has been remediated after monitoring the environment.
Next update by: Thursday, February 13, 2020, at 7:00 PM UTC; PublishedTime=2020-02-13T16:45:06.203Z

--------------------------------------------------------------------------------------------------
Title Delays in receiving audit logs and compliance center data
ID MO203678
Service Office 365 Portal
Feature Administration
Status Service restored
Impact Admins may see delays when attempting to receive audit logs and compliance center data.
Severity Sev2
Start time 2020-02-12T03:00:00Z
End time 2020-02-12T08:30:00Z
Last update 2020-02-12T09:54:46.94Z
More info
Root cause After resolving a recent delay in processing Exchange Online data, the responsible infrastructure inadvertently began to perform in an inefficient state

MO203678 Details

Status: We've determined that after resolving a recent delay for processing Exchange Online data, subsequent requests for other services began processing on our infrastructure at a higher than expected rate, resulting in performance issues. We're working to reroute traffic to healthy infrastructure to expedite the process and mitigate impact.
Next update by: Wednesday, February 12, 2020, at 10:00 AMUTC; PublishedTime=2020-02-12T08:00:47.947Z

Final status: We've finished rerouting traffic, and confirmed after monitoring that the issue has been resolved.
Next update by: Wednesday, February 12, 2020, at 10:00 AMUTC; PublishedTime=2020-02-12T08:00:47.947Z

--------------------------------------------------------------------------------------------------
Title Microsoft 365 reports data may be inaccurate
ID MO203671
Service Office 365 Portal
Feature Administration
Status Service restored
Impact Admins may have noticed that Microsoft 365 reports from January 10, 2020 to January 23,
Severity Sev2
Start time 2020-01-10T00:46:00Z
End time 2020-02-19T02:28:00Z
Last update 2020-02-19T02:53:59.337Z
More info The affected reports were as follows:
Root cause A recent configuration change in Azure Active Directory (AAD) resulted in a data sync issue.

MO203671 Details

Status: Our telemetry indicates that impact associated with MO200511 persists. We've deployed an additional code fix which has updated the snapshot and we're in the process of backfilling the additional data. Once the data backfill is complete, we'll re-sync the reports so they reflect accurate information.
Next update by: Thursday, February 13, 2020, at 2:00 AMUTC; PublishedTime=2020-02-12T04:40:30.757Z

Status: We're continuing to back-fill the remaining data before we re-sync the reports so they reflect the accurate information and mitigate the issue.
Next update by: Thursday, February 13, 2020, at 7:00 PMUTC ; PublishedTime=2020-02-13T00:16:43.257Z

Status: We've completed the back-fill of the remaining data. We're reaching out to some customers that reported the problem to confirm mitigation.
Next update by: Friday, February 14, 2020, at 7:00 PMUTC ; PublishedTime=2020-02-13T17:23:53.903Z

Status: We've determined that an issue occurred during the filling of the backlogged data that's resulting in impact persisting for some affected users. We're analyzing the backlog to isolate the problem and determine our next steps.
Next update by: Monday, February 17, 2020, at 4:00 AMUTC; PublishedTime=2020-02-14T17:28:39.557Z

Status: We've resolved the problem that occurred during the backfill. We're verifying the data while the backfill continues to progress.
Next update by: Tuesday, February 18, 2020, at 4:00 AMUTC; PublishedTime=2020-02-17T02:52:33.547Z

Status: The backfill is continuing to progress, and we expect it to complete by the end of the day. Once completed, we'll verify the data again for accuracy and then re-sync the reports to reflect that data.
Next update by: Wednesday, February 19, 2020, at 4:00 AMUTC; PublishedTime=2020-02-18T02:55:24.407Z

Final status: We've finished backfilling the data and re-syncing the reports. We've verified that the affected reports now show the accurate data.
Next update by: Wednesday, February 19, 2020, at 4:00 AMUTC; PublishedTime=2020-02-18T02:55:24.407Z

--------------------------------------------------------------------------------------------------
Title Can't access multiple services
ID MO203172
Service Office 365 Portal
Feature Portal
Status Service restored
Impact Users may have been unable to access multiple Microsoft 365 services.
Severity Sev2
Start time 2020-02-05T12:47:00Z
End time 2020-02-05T15:41:00Z
Last update 2020-02-05T16:58:00.897Z
More info Additionally, users who were able to access services may have experienced latency.
Root cause A third-party networking fiber provider experienced a network disruption, resulting in impact.

MO203172 Details

Status: We're investigating a potential access and latency issue with multiple Microsoft 365 services. We'll provide an update within 30 minutes.


Status: We're investigating a potential network interruption as the root cause.
Next update by: Wednesday, February 5, 2020, at 5:30 PM UTC; PublishedTime=2020-02-05T15:40:16.073Z

Final status: We've identified that a third-party networking fiber provider experienced a network interruption. We've rerouted user connections to alternate networking components and confirmed that impact has been remediated after monitoring the environment.
Next update by: Wednesday, February 5, 2020, at 5:30 PM UTC; PublishedTime=2020-02-05T15:40:16.073Z

--------------------------------------------------------------------------------------------------
Title Can't access Delve
ID MO203106
Service Office 365 Portal
Feature Portal
Status Service restored
Impact Users are unable to access the Delve application
Severity Sev2
Start time 2020-02-05T01:56:41Z
End time 2020-02-05T02:06:00Z
Last update 2020-02-05T04:51:26.573Z
More info Impact was limited to the Delve web application, the Delve mobile application was unaffected.
Root cause A recent change that was made to the Delve web app ID caused users to be unable to access the service using the Delve web app.

MO203106 Details

Status: We're investigating a potential issue and checking for impact to your organization. We'll provide an update within 30 minutes.; PublishedTime=2020-02-05T01:57:13.597

Status: We are currently investigating a recent authorization change to see if it is the source of the issue and determine our next troubleshooting steps.
Next update by: Wednesday, February 5, 2020, at 4:30 AMUTC; PublishedTime=2020-02-05T02:24:14.263Z

Final status: We have completed our investigation of the recent change and found that this change altered the app ID, which created impact. We have rolled back this change and confirmed that impact has been mitigated.
Next update by: Wednesday, February 5, 2020, at 4:30 AMUTC; PublishedTime=2020-02-05T02:24:14.263Z

--------------------------------------------------------------------------------------------------
Title Microsoft 365 usage reports are delayed
ID MO202839
Service Office 365 Portal
Feature Portal
Status Service restored
Impact Admins may have been unable to see any Microsoft 365 usage report starting from January
Severity Sev2
Start time 2020-02-01T00:00:00Z
End time 2020-02-04T00:00:00Z
Last update 2020-02-04T03:28:44.947Z
More info
Root cause A source data stream responsible for usage reports was experiencing a processing delay.

MO202839 Details

Status: Our initial investigation indicates that the source data stream responsible for the reports may be processing in an inefficient state. We're analyzing the associated performance data, data sources, and pipeline responsible for processing the reports to determine the source of the issue.
Next update by: Tuesday, February 4, 2020, at 5:00 AMUTC; PublishedTime=2020-02-02T03:15:31.467Z

Final status: We've determined that a source data stream responsible for usage reports was experiencing a processing delay. We've fixed that delay and confirmed the usage reports are viewable.
Next update by: Tuesday, February 4, 2020, at 5:00 AMUTC; PublishedTime=2020-02-02T03:15:31.467Z

--------------------------------------------------------------------------------------------------
Title Can't access multiple services
ID MO202741
Service Office 365 Portal
Feature Portal
Status Investigation suspended
Impact Users may be unable to access multiple Microsoft 365 services.
Severity Sev2
Start time 2020-01-31T15:11:00Z
End time 2020-01-31T17:52:35Z
Last update 2020-01-31T17:52:38.287Z
More info
Root cause A third-party internet service provider is experiencing issues which is resulting in connectivity issues to Microsoft 365 services.

MO202741 Details

Status: We're investigating a potential issue with accessing multiple Microsoft 365 services. We'll provide an update within 30 minutes.


Status: We're reviewing service availability metrics to identify the source of the issue.
Next update by: Friday, January 31, 2020, at 5:30 PM UTC; PublishedTime=2020-01-31T15:55:25.677Z

Status: We've identified an outage with a local Internet Service Provider (ISP) in the United Kingdom. Our investigation reveals no issues are occurring within the Microsoft 365 service infrastructure at this time, we're continuing to monitor service health and work with the impacted ISP to mitigate impact.
Next update by: Friday, January 31, 2020, at 7:30 PMUTC; PublishedTime=2020-01-31T17:15:33.24Z

Final status: We've confirmed that our service remains stable and the third-party ISP will continue to investigate the underlying cause of the issue.
Next update by: Friday, January 31, 2020, at 7:30 PMUTC; PublishedTime=2020-01-31T17:15:33.24Z

--------------------------------------------------------------------------------------------------
Title Title: Can't upload new Microsoft Stream videos
ID MO202653
Service Office 365 Portal
Feature Portal
Status Service restored
Impact User impact: Users may be unable to upload new Microsoft Stream videos.
Severity Sev2
Start time 2020-01-30T20:10:37Z
End time 2020-01-30T23:26:13Z
Last update 2020-01-30T23:27:04.91Z
More info
Root cause A recent deployment was causing an unexpected influx of traffic to a portion of SQL infrastructure and resulted in a degradation to Azure Media Services, which supports Microsoft Stream videos.

MO202653 Details

Status: We identified that a recent deployment was causing an unexpected influx of traffic to a portion of SQL infrastructure and resulted in a degradation to Azure Media Services, which supports Microsoft Stream videos. We've reverted the offending deployment to mitigate impact, and we're now monitoring the queue as the backlog of videos to be uploaded processes.
Next update by: Thursday, January 30, 2020, at 11:00 PM UTC; PublishedTime=2020-01-30T20:12:36.12Z

Status: We've confirmed that the backlog of videos is successfully being uploaded and we're continuing to monitor the queue to ensure it completes successfully.
Next update by: Friday, January 31, 2020, at 12:30 AM UTC; PublishedTime=2020-01-30T21:11:27.593Z

Status: We've confirmed through monitoring that recovery is proceeding at a steady rate and newly uploaded videos are being processed; however, it is still not processing at optimal speed. Additionally, transcript and closed captioning generation may be delayed during this time, and we're continuing to monitor the service to ensure successful mitigation.
Next update by: Friday, January 31, 2020, at 3:00 AM UTC; PublishedTime=2020-01-30T22:44:04.04Z

Final status: We've confirmed that the recovery process has completed and that the service is now operating at optimal health.
Next update by: Friday, January 31, 2020, at 3:00 AM UTC; PublishedTime=2020-01-30T22:44:04.04Z

--------------------------------------------------------------------------------------------------
Title Multiple degraded Microsoft 365 services
ID MO202169
Service Office 365 Portal
Feature Portal
Status Service restored
Impact Users may have experienced degraded service for Microsoft Teams, SharePoint Online,
Severity Sev2
Start time 2020-01-24T21:15:00Z
End time 2020-01-24T22:21:00Z
Last update 2020-01-24T23:13:11.373Z
More info Within Microsoft Teams, users may have been unable to create chats, teams or channels, send/receive messages, update presence or receive notifications.
Root cause A portion of networking infrastructure that facilitates Azure services routing became degraded, which resulted service impact.

MO202169 Details

Status: We've identified a networking issue and are working to restore connections.
Next update by: Friday, January 24, 2020, at 11:30 PMUTC; PublishedTime=2020-01-24T22:46:18.683Z

Final status: A portion of networking infrastructure that facilitates Azure services routing became degraded. We've restored the network connections and verified via telemetry that impact has been mitigated.
Next update by: Friday, January 24, 2020, at 11:30 PMUTC; PublishedTime=2020-01-24T22:46:18.683Z

--------------------------------------------------------------------------------------------------
Title Microsoft 365 reports data may be inaccurate
ID MO200511
Service Office 365 Portal
Feature Administration
Status Service restored
Impact Admins may have noticed that Microsoft 365 reports from January 10, 2020 to January 23,
Severity Sev2
Start time 2020-01-10T00:46:00Z
End time 2020-02-08T00:45:51Z
Last update 2020-02-08T00:47:06.387Z
More info The affected reports were as follows:
Root cause A recent configuration change in Azure Active Directory (AAD) resulted in a data sync issue.

MO200511 Details

Status: We've determined that a recent configuration change in Azure Active Directory (AAD) resulted in a data sync issue. We've corrected the configuration change, which has fixed the sync issue. We're resetting the "watermark" on all sync instances in AAD to the day prior to the original configuration change and re-syncing the data. Once the data in AAD is synced, the data will be back-filled for all the affected reports.
Next update by: Saturday, January 18, 2020, at 3:00 AM UTC; PublishedTime=2020-01-17T07:16:15.083Z

Status: We're continuing to build a script to reset the "watermark" on all sync instances in Azure Active Directory (AAD). Once we've completed building and have validated the script will reset the "watermark", we'll deploy it to all impacted infrastructure to re-sync the affected reports.
Next update by: Tuesday, January 21, 2020, at 4:00 AM UTC; PublishedTime=2020-01-18T02:31:25.38Z

Status: We're continuing to develop the script to reset the "watermark" on all sync instances in Azure Active Directory (AAD).
Next update by: Thursday, January 23, 2020, at 4:00 AMUTC; PublishedTime=2020-01-21T03:51:00.23Z

Status: We're currently waiting for the remaining partitions to be caught up in the syncing properties. Once completed, we will deploy the fix to the affected environments.
Next update by: Sunday, January 26, 2020, at 4:00 AMUTC; PublishedTime=2020-01-23T03:02:54.73Z

Status: We've confirmed that all but one of the partitions have completed the sync process. The final partition is currently 93 percent complete. Once completed, we'll deploy the fix to the affected environments.
Next update by: Monday, January 27, 2020, at 4:00 AM UTC; PublishedTime=2020-01-26T02:53:37.577Z

Status: We've finished developing and validating the script. We're in the process of deploying that fix to the affected environments so that the Microsoft 365 reports from January 10, 2020 to January 15, 2020 will be accurate.
Next update by: Tuesday, January 28, 2020, at 2:30 AM UTC; PublishedTime=2020-01-27T03:19:14.21Z

Status: In the process of deploying the fix, we determined a subset of data necessary to generate the reports is unavailable. We're working to determine the best way to address the source data issue.
Next update by: Wednesday, January 29, 2020, at 2:30 AM UTC; PublishedTime=2020-01-28T02:12:44.157Z

Status: We're backfilling historical data so that the affected reports can be accurately synced. In parallel, we're assessing the progress of the backfill to determine an estimated time to resolution.
Next update by: Friday, January 31, 2020, at 2:30 AM UTC; PublishedTime=2020-01-29T02:30:18.647Z

Status: Our current monitors of the backfill indicate that the reports are continuing to sync. We're continuing to evaluate the progress of the backfill to ensure that all affected reports are processed as expected.
Next update by: Wednesday, February 5, 2020, at 2:30 AM UTC; PublishedTime=2020-01-31T01:21:09.43Z

Status: We've confirmed that three out of four processes have completed syncing for the reports. We're starting the remaining data backfill, which we expect will remediate the issue.
Next update by: Saturday, February 8, 2020, at 2:30 AMUTC ; PublishedTime=2020-02-04T00:57:20.44Z

Final status: We've confirmed that all of the backfills are now complete and admins are able to see all accurate data.
Next update by: Saturday, February 8, 2020, at 2:30 AMUTC ; PublishedTime=2020-02-04T00:57:20.44Z

--------------------------------------------------------------------------------------------------
Title Canvas components failing within Power Apps
ID MM204224
Service Power Apps in Microsoft 365
Feature Service and web access issues
Status Service restored
Impact Users may have been unable to use existing canvas components within the latest version
Severity Sev2
Start time 2020-02-11T06:00:00Z
End time 2020-02-20T10:30:00Z
Last update 2020-02-20T10:42:10.923Z
More info
Root cause A recent routine service update caused a code misconfiguration, resulting in impact.

MM204224 Details

Status: We identified through telemetry review that the code misconfiguration stems from an infrastructure change within a recent Power Apps deployment. We've developed a fix and are working to validate this before deploying to impacted environments.
Next update by: Thursday, February 20, 2020, at 3:00 AM UTC; PublishedTime=2020-02-19T22:16:26.697Z

Status: We've encountered an error in the process of validating and preparing the fix. We're working to resolve this error to continue our validation process to deploy the fix to the impacted environment.
Next update by: Thursday, February 20, 2020, at 7:00 AMUTC; PublishedTime=2020-02-20T01:16:42.803Z

Status: We've resolved the error and validated that the fix remediates impact. We're deploying the fix and expect it to fully saturate the impacted environment within an hour.
Next update by: Thursday, February 20, 2020, at 7:30 AMUTC ; PublishedTime=2020-02-20T05:23:42.25Z

Status: We've encountered an additional error deploying the fix to the impacted environment. We're actively investigating the error to formulate a resolution and move forward with deployment.
Next update by: Thursday, February 20, 2020, at 11:00 AMUTC ; PublishedTime=2020-02-20T06:58:26.947Z

Final status: We've successfully deployed the fix to the impacted environment and, following observation, confirmed restoration of service health.
Next update by: Thursday, February 20, 2020, at 11:00 AMUTC ; PublishedTime=2020-02-20T06:58:26.947Z

--------------------------------------------------------------------------------------------------
Title Can't create flows from templates
ID MF202481
Service Microsoft Power Automate in Microsoft 365
Feature Service and web access issues
Status Service restored
Impact Users may have been unable to create new flows when using a template.
Severity Sev2
Start time 2020-01-28T23:00:10Z
End time 2020-01-29T05:53:00Z
Last update 2020-01-29T06:13:59.66Z
More info
Root cause A recent service update inadvertently caused impact with creating flows from templates.

MF202481 Details

Status: We're investigating a potential issue and checking for impact to your organization. We'll provide an update within 30 minutes.; PublishedTime=2020-01-28T23:09:19.723

Status: We've determined that a recent service update deployment inadvertently resulted in impact. We're investigating to determine the last healthy build and will be rolling back the affected update to mitigate impact.
Next update by: Wednesday, January 29, 2020, at 4:00 AMUTC; PublishedTime=2020-01-28T23:34:45.633Z

Status: After further analysis, we determined that reverting to a previous build would not resolve the problem. We're now working on a code fix for the issue to remediate impact.
Next update by: Wednesday, January 29, 2020, at 8:30 PM UTC; PublishedTime=2020-01-29T03:33:08.633Z

Final status: We developed a code fix and deployed it to the affected environment to remediate the impact.
Next update by: Wednesday, January 29, 2020, at 8:30 PM UTC; PublishedTime=2020-01-29T03:33:08.633Z

--------------------------------------------------------------------------------------------------
Title Samsung 7.0 app crashes
ID IT204282
Service Microsoft Intune
Feature Microsoft Intune
Status Service restored
Impact Users with Android work profiles on Samsung 7.0 devices may have experienced crashes in
Severity Sev2
Start time 2020-02-07T11:30:00Z
End time 2020-02-21T00:00:00Z
Last update 2020-02-21T17:31:48.74Z
More info
Root cause A code issue introduced in a recent update unexpectedly caused impact to app functionality for Android work profiles on Samsung 7.0 devices.

IT204282 Details

Status: We're investigating a potential issue and checking for impact to your organization. We'll provide an update within 30 minutes.; PublishedTime=2020-02-20T16:36:37.707

Status: We're working with Google to identify if a possible code change in the Android service environment has caused a degradation in work profiles for Samsung 7.0 devices serviced through Microsoft Intune to determine our next trouble shooting steps.
Next update by: Thursday, February 20, 2020, at 10:30 PMUTC; PublishedTime=2020-02-20T16:58:43.41Z

Status: We received confirmation from Google that a code issue introduced in a recent update unexpectedly caused impact. Google is currently deploying a code fix, which they estimate will complete within the next four to six hours. Additionally, we've validated the fix as successful in our internal environment.
Next update by: Friday, February 21, 2020, at 7:00 PM UTC; PublishedTime=2020-02-20T21:12:42.817Z

Final status: We've confirmed with Google that the code fix has been deployed and we've validated the fix in our internal environment. Users may need to update their Google Chrome app to see full remediation.
Next update by: Friday, February 21, 2020, at 7:00 PM UTC; PublishedTime=2020-02-20T21:12:42.817Z

--------------------------------------------------------------------------------------------------
Title Can't enroll iOS devices using the Apple Configurator
ID IT203715
Service Microsoft Intune
Feature Microsoft Intune
Status Service restored
Impact Users may be unable to enroll iOS devices using Apple Configurator profiles.
Severity Sev2
Start time 2020-02-12T18:33:24Z
End time 2020-02-13T05:00:00Z
Last update 2020-02-13T05:18:42.637Z
More info Users who attempted to enroll profiles using the 'enroll without user affinity' feature of the Apple Configurator would have seen the following error: 'Profile was invalid or missing required values'.
Root cause A recent update caused issues with the user interface controls which caused problems enrolling iOS devices with Apple Configurator profiles.

IT203715 Details

Status: We're investigating a potential issue and checking for impact to your organization. We'll provide an update within 30 minutes.; PublishedTime=2020-02-12T18:33:49.857

Status: We're analyzing system logs and working to internally reproduce the issue to determine the source of the problem.
Next update by: Wednesday, February 12, 2020, at 9:00 PMUTC; PublishedTime=2020-02-12T18:53:17.703Z

Status: As our system log analysis progresses, we've investigated a subset of certificate assignments and confirmed that additional enrollment methods are not impacted by this issue. In parallel, we've investigated a subset of recent service updates and confirmed that they have not contributed to this event. We've successfully reproduced the issue in our internal testing environment and are analyzing the results and the certificate trust verification process to identify our next steps for resolving this issue.
Next update by: Thursday, February 13, 2020, at 12:30 AMUTC ; PublishedTime=2020-02-12T20:45:33.8Z

Status: We analyzed the results of our internal testing and have identified a potential fix for the issue. We're working to develop the fix and apply it to a testing environment for validation.
Next update by: Thursday, February 13, 2020, at 3:30 AMUTC; PublishedTime=2020-02-13T00:20:29.263Z

Status: We've completed development of the fix and initial testing indicates that the issue is no longer reproducible We're continuing our validation of the fix and will begin deployment to the affected environment as soon as validation is completed.
Next update by: Thursday, February 13, 2020, at 4:30 AMUTC ; PublishedTime=2020-02-13T02:24:20.833Z

Status: We've completed validation of the fix and confirmed that it mitigates impact. We're deploying the fix to the affected environments.
Next update by: Thursday, February 13, 2020, at 7:30 AMUTC ; PublishedTime=2020-02-13T03:54:46.43Z

Final status: We completed deployment of the fix to the affected environments, and confirmed that it has mitigated impact.
Next update by: Thursday, February 13, 2020, at 7:30 AMUTC ; PublishedTime=2020-02-13T03:54:46.43Z

--------------------------------------------------------------------------------------------------
Title Can't add or renew Automated Device Enrollment or Device Enrollment Program (DEP) tokens
ID IT203207
Service Microsoft Intune
Feature Microsoft Intune
Status Service restored
Impact Admins may have been unable to add or renew Automated Device Enrollment or DEP tokens.
Severity Sev2
Start time 2020-02-03T21:00:00Z
End time 2020-02-06T02:24:00Z
Last update 2020-02-06T03:16:40.273Z
More info Admins may have seen the following error when adding or renewing tokens:
Root cause A feature update meant to improve Microsoft Graph data was inadvertently causing the issue.

IT203207 Details

Status: We're currently analyzing system telemetry to isolate the underlying cause.
Next update by: Thursday, February 6, 2020, at 1:00 AMUTC; PublishedTime=2020-02-05T23:15:09.653Z

Status: We're evaluating affected accounts to confirm if impact is intermittent or constant, which will allow us to more accurately troubleshoot the issue.
Next update by: Thursday, February 6, 2020, at 3:00 AM UTC; PublishedTime=2020-02-06T00:58:57.537Z

Status: We've determined that a feature update meant to improve Microsoft Graph data is inadvertently causing the issue. We've disabled the update and are validating that impact is mitigated.
Next update by: Thursday, February 6, 2020, at 5:00 AMUTC; PublishedTime=2020-02-06T02:43:37.103Z

Final status: We've confirmed via telemetry and our test account that impact is mitigated.
Next update by: Thursday, February 6, 2020, at 5:00 AMUTC; PublishedTime=2020-02-06T02:43:37.103Z

--------------------------------------------------------------------------------------------------
Title Managed Windows devices showing as "non-compliant"
ID IT203034
Service Microsoft Intune
Feature Microsoft Intune
Status Service restored
Impact Admins and users may see a subset of their Windows-managed devices showing as
Severity Sev2
Start time 2020-02-03T20:00:00Z
End time 2020-02-12T08:00:00Z
Last update 2020-02-12T21:17:20.567Z
More info Users may have been able to mitigate the issue by restarting the impacted Windows 10 device, which would stop the update install, and turn Windows Defender back on.
Root cause A recent Windows Defender update failed to complete, causing the Windows Defender service to remain in a "off" state. This resulted in devices being non-compliant.

IT203034 Details

Status: We've identified a recent code change in Windows Defender antimalware version KB4052623 that's causing devices to be evaluated as non-compliant, and we're working to develop a fix.
Next update by: Wednesday, February 5, 2020, at 4:00 AMUTC; PublishedTime=2020-02-04T19:25:41.873Z

Status: We've halted the deployment of the Windows Defender update within and are continuing our work to develop a solution.
Next update by: Wednesday, February 5, 2020, at 4:00 AMUTC; PublishedTime=2020-02-04T22:26:36.05Z

Status: We're continuing our work to develop a fix for the code issue in Windows Defender to mitigate impact.
Next update by: Wednesday, February 5, 2020, at 7:00 PMUTC; PublishedTime=2020-02-05T02:48:43.167Z

Status: The fix in Windows Defender is in the final stages of development. Once development is complete, the fix will then be tested and validated before being approved for deployment.
Next update by: Thursday, February 6, 2020, at 12:30 AMUTC; PublishedTime=2020-02-05T17:40:02.037Z

Status: The creation of the fix to mitigate this issue continues, and we'll provide additional updates on testing and deployment timeline once available.
Next update by: Thursday, February 6, 2020, at 6:30 PMUTC; PublishedTime=2020-02-05T22:44:04.47Z

Status: The primary mitigation for this issue has been determined, and guidance is being provided via the KB4052623 article located here: https://support.microsoft.com/e n-us/help/4052623/update-for-windows-defender-antimalware-platform
This article will help your organization determine if a device needs to be rebooted; admins are encouraged to reference the article in determining which devices may be impacted and the proper steps to mitigate the impact.

Next update by: Wednesday, February 12, 2020, at 6:30 PM UTC; PublishedTime=2020-02-06T18:16:04.643Z

Status: We're validating a fix that should be deployed to the affected infrastructure sometime during the week of February 17, 2020.
Next update by: Wednesday, February 19, 2020, at 6:30 PMUTC; PublishedTime=2020-02-12T18:01:07.483Z

Final status: We've verified via internal testing that restarting the affected infrastructure has mitigated impact by putting the Windows Defender service in a protected state until a long term fix can be deployed.
Next update by: Wednesday, February 19, 2020, at 6:30 PMUTC; PublishedTime=2020-02-12T18:01:07.483Z

--------------------------------------------------------------------------------------------------
Title Device Restriction policy setting reports may be inaccurate
ID IT202581
Service Microsoft Intune
Feature Microsoft Intune
Status Service restored
Impact Admins may have received inaccurate data from their Device Restriction policy setting
Severity Sev2
Start time 2019-12-16T12:00:00Z
End time 2020-01-31T19:45:00Z
Last update 2020-01-31T21:57:51.967Z
More info Policy settings delivered to devices were applying as expected. This issue only impacted reporting. The inaccuracy may have occurred in the "Type of system scan to perform" setting in Device Restrictions. The policy may have appeared to fail even when it successfully applied.
Root cause A recent update introduced a code regression in the affected environment, which was causing policy setting reports to return inaccurate results.

IT202581 Details

Status: We're investigating a potential issue and checking for impact to your organization. We'll provide an update within 30 minutes.


Status: We've determined that a recent update has introduced a code regression in the affected environment, which is causing the policy setting reports to return inaccurate results. We're validating a potential fix that we expect will remediate impact.
Next update by: Thursday, January 30, 2020, at 1:30 AMUTC; PublishedTime=2020-01-29T23:29:29.23Z

Status: During validation of the fix, we determined that the potential fix won't resolve the inaccuracy. We're continuing our investigation to produce a fix to remediate the issue.
Next update by: Thursday, January 30, 2020, at 7:00 AMUTC; PublishedTime=2020-01-30T01:22:54.13Z

Status: We suspect there may be a reporting issue where multiple values are being reported for a single value field, causing the inaccurate data reports. We're continuing our investigation to validate this finding and determine our next steps.
Next update by: Thursday, January 30, 2020, at 7:00 PMUTC ; PublishedTime=2020-01-30T05:49:57.903Z

Status: We've developed a fix which is undergoing internal validation prior to deployment. We're also continuing to investigate if any other values are impacted.
Next update by: Thursday, January 30, 2020, at 10:00 PMUTC; PublishedTime=2020-01-30T17:31:29Z

Status: We've validated the fix internally and have begun strategically deploying it to specific environments to ensure a safe roll out. We'll provide an update to our deployment timeline as soon as one becomes available.
Next update by: Friday, January 31, 2020, at 12:30 AMUTC; PublishedTime=2020-01-30T20:36:53.417Z

Status: We've begun deploying the fix to all regions and expect to be completed by Friday, January 31, 2020, at 3:00 PM UTC.
Next update by: Friday, January 31, 2020, at 7:00 PMUTC ; PublishedTime=2020-01-30T23:33:23.307Z

Status: We've discovered an unexpected issue that has delayed our deployment timeline. This issue has been resolved and deployment has continued. We expect the completed deployment to remediate impact by end of day Friday, January 31, 2020.
Next update by: Friday, January 31, 2020, at 8:00 PMUTC; PublishedTime=2020-01-31T16:45:18.547Z

Status: We're continuing our deployment of the fix that we expect will remediate impact by Friday, January 31, 2020, at 8:00 PM UTC.
Next update by: Friday, January 31, 2020, at 10:00 PMUTC; PublishedTime=2020-01-31T18:10:42.197Z

Status: The fix has completely saturated the affected environment and we're seeking confirmation that this has remediated impact from a subset of affected users.
Next update by: Friday, January 31, 2020, at 11:30 PMUTC; PublishedTime=2020-01-31T20:05:03.51Z

Final status: We've confirmed with a subset of affected users that the Device Restriction policy setting reports are now calculating data correctly. Users will have to wait for their device to check-in before Device Restriction policy settings reports are calculated correctly. Admins can force a check-in and recalculation by modifying the title of the Device Restriction policy and point protection policy, then saving these changes.
Next update by: Friday, January 31, 2020, at 11:30 PMUTC; PublishedTime=2020-01-31T20:05:03.51Z

--------------------------------------------------------------------------------------------------
Title App deployment and reporting delays for Android Enterprise
ID IT202579
Service Microsoft Intune
Feature Microsoft Intune
Status Service restored
Impact Users saw a delay in the app deployments, and admins experienced a delay in work profile
Severity Sev2
Start time 2020-01-27T20:50:00Z
End time 2020-01-30T04:43:00Z
Last update 2020-01-30T15:37:59.707Z
More info
Root cause A recent update to an API in the Android Enterprise service, which the Intune service is reliant on, was causing impact.

IT202579 Details

Status: We're investigating a potential issue and checking for impact to your organization. We'll provide an update within 30 minutes.; PublishedTime=2020-01-29T23:00:32.973

Status: We're reviewing system telemetry to isolate the underlying cause.
Next update by: Thursday, January 30, 2020, at 1:00 AMUTC; PublishedTime=2020-01-29T23:25:04.523Z

Status: We identified this to be an issue with a dependent Application Programming Interface (API) in the Android Enterprise service. We're working with the team responsible for the API to address the issue. In addition, we're working on a fix that will remediate the impact for customers who have less than 100 apps installed.
Next update by: Thursday, January 30, 2020, at 5:00 AMUTC; PublishedTime=2020-01-29T23:56:43.597Z

Status: The team responsible for the Application Programming Interface (API) in the Android Enterprise service has confirmed that they have begun the steps of deploying their fix. We expect the delays to improve as the fix saturates the affected environment.
Next update by: Thursday, January 30, 2020, at 5:00 PMUTC; PublishedTime=2020-01-30T03:32:05.303Z

Final status: The fix has been fully deployed and we've confirmed the issue is resolved after monitoring the environment.
Next update by: Thursday, January 30, 2020, at 5:00 PMUTC; PublishedTime=2020-01-30T03:32:05.303Z

--------------------------------------------------------------------------------------------------
Title Can't view audit events that occurred after January 16th, 2020
ID IT202102
Service Microsoft Intune
Feature Microsoft Intune
Status Service restored
Impact Admins may have been unable to view audit events that occurred after January 16th, 2020.
Severity Sev2
Start time 2020-01-16T22:00:00Z
End time 2020-01-29T11:00:00Z
Last update 2020-01-29T17:22:56.47Z
More info No data loss was observed, all data was successfully stored and retrieved from the server.
Root cause A recent Auditing service update to improve retry logic unexpectedly caused impact.

IT202102 Details

Status: We're analyzing system logs to determine the source of the issue and facilitate viewing of the delayed audit events.
Next update by: Friday, January 24, 2020, at 5:30 AMUTC; PublishedTime=2020-01-24T03:18:09.987Z

Status: We're troubleshooting within a portion of the affected infrastructure to determine what's delaying the audit event logs from being viewed by admin users.
Next update by: Friday, January 24, 2020, at 10:00 AMUTC; PublishedTime=2020-01-24T05:28:53.36Z

Status: We've determined that a recent Auditing service update to improve retry logic, unexpectedly caused impact. We're reverting the update to a section of affected infrastructure to confirm that this mitigates the issue. Once confirmed, we will continue to revert the update to the rest of the affected infrastructure.
Next update by: Friday, January 24, 2020, at 7:00 PM UTC; PublishedTime=2020-01-24T09:31:34.213Z

Status: We've confirmed that reverting the update is effective in remediating impact. The update is now being reverted in the remainder of the affected environment and should complete on Friday, January 24, 2020, by 11:00 PM UTC.
Next update by: Friday, January 24, 2020, at 9:00 PMUTC; PublishedTime=2020-01-24T17:25:47.183Z

Status: The offending update is still being reverted and we continue to expect it to complete on Friday, January 24, 2020, by 11:00 PM UTC. We'll monitor the progress and provide updates as it completes.
Next update by: Friday, January 24, 2020, at 11:30 PMUTC ; PublishedTime=2020-01-24T19:54:35.847Z

Status: Reverting the update is taking longer than we expected to complete. We now expect it to complete on Saturday, January 25, 2020, by 1:00 AM UTC. We're continuing to monitor its progress.
Next update by: Saturday, January 25, 2020, at 2:00 AMUTC ; PublishedTime=2020-01-24T23:24:38.437Z

Status: We've successfully reverted the update and the data deployment is complete. We're replenishing the remaining audit logs which will result in full mitigation.
Next update by: Monday, January 27, 2020, at 7:00 PMUTC ; PublishedTime=2020-01-25T01:33:54.927Z

Status: After successfully reverting the update and completing the data deployment, we've begun replenishing the remaining audit logs. Upon further investigation, about half of the customers impacted by this do not need data back filling. Approximately 30 percent of the data that does need to be back filled has completed, and we expect the remaining data back fill to complete in about two days.
Next update by: Tuesday, January 28, 2020, at 7:00 PM UTC; PublishedTime=2020-01-27T17:32:23.423Z

Status: Our telemetry indicates that backfilling of the necessary data is 75 percent complete. We expect the data backfilling process to complete on Wednesday, January 29, 2020.
Next update by: Wednesday, January 29, 2020, at 7:00 PMUTC; PublishedTime=2020-01-28T17:22:51.417Z

Final status: We've confirmed that the backfilling process has completed and that all audit event logs are now accessible.
Next update by: Wednesday, January 29, 2020, at 7:00 PMUTC; PublishedTime=2020-01-28T17:22:51.417Z

--------------------------------------------------------------------------------------------------
Title Can't login to Microsoft 365 services or resources
ID IS204465
Service Identity Service
Feature Sign-In
Status Service restored
Impact Users may have been unable to login to or access Microsoft 365 services or resources.
Severity Sev1
Start time 2020-02-21T20:20:00Z
End time 2020-02-22T00:29:30.937Z
Last update 2020-02-22T00:29:30.937Z
More info This event affected a subset of users in the Western European and Southeast Asian regions.
Root cause A networking issue in the Identity Services resulted in login and access issues.

IS204465 Details

Final status: We determined that a networking issue in the Identity Services resulted in users being unable to access Microsoft 365 resources. We've rerouted affected traffic to an alternate infrastructure and our telemetry indicates that the issue has been mitigated.


--------------------------------------------------------------------------------------------------
Title Delays when triggering Power Automate flows
ID FM203685
Service Microsoft Forms
Feature Integration
Status Service restored
Impact Users may have experienced delays of several hours when triggering Power Automate flows.
Severity Sev2
Start time 2020-02-10T00:00:00Z
End time 2020-02-12T12:13:57.937Z
Last update 2020-02-12T12:28:54.677Z
More info
Root cause A specific workflow was generating a substantial amount of messages, resulting in excessive queues.

FM203685 Details

Final status: We identified that a specific workflow was generating a substantial amount of messages, resulting in queues to process them. We developed and deployed a fix, which prevented the workflow from generating more messages. Additionally, we ran optimization commands to clear the remaining queues, which we confirmed resolved the issue.


--------------------------------------------------------------------------------------------------
Title Can't access to Audit Management API service
ID EX203702
Service Exchange Online
Feature E-Mail and calendar access
Status Service restored
Impact Users may have been unable to access the Audit Management API service.
Severity Sev2
Start time 2020-02-12T11:00:00Z
End time 2020-02-12T22:56:00Z
Last update 2020-02-12T23:49:46.8Z
More info
Root cause A backlog of Exchange Audit logs was causing Audit Management API timeouts.

EX203702 Details

Status: We've discovered that an unexpected increase in traffic through the Audit Management API is causing the service to perform below acceptable thresholds. We've performed optimization actions on the affected subset of infrastructure and are monitoring the service to confirm mitigation.
Next update by: Wednesday, February 12, 2020, at 7:30 PMUTC; PublishedTime=2020-02-12T17:03:14.747Z

Status: Our optimizing actions on the affected subset of infrastructure are progressing, though taking longer than expected to remediate impact. We're monitoring the situation to ensure remediation as quickly as possible.
Next update by: Thursday, February 13, 2020, at 12:00 AMUTC; PublishedTime=2020-02-12T19:04:28.273Z

Final status: Our optimizing actions have completed, and we've confirmed via monitoring that the impacting backlog has fully drained, mitigating impact.
Next update by: Thursday, February 13, 2020, at 12:00 AMUTC; PublishedTime=2020-02-12T19:04:28.273Z

--------------------------------------------------------------------------------------------------
Title Can't search in light version of Outlook on the web
ID EX203674
Service Exchange Online
Feature E-Mail and calendar access
Status Service restored
Impact Users may have been unable to perform searches in the light version of Outlook on the
Severity Sev2
Start time 2020-01-25T12:00:00Z
End time 2020-02-19T01:35:00Z
Last update 2020-02-19T03:23:11.493Z
More info As a workaround, users could search within the Outlook desktop client or the standard version of Outlook on the web.
Root cause A recent security update was inadvertently applied to the light version of Outlook on the web, causing an issue with cookie creation.

EX203674 Details

Status: We've received reports that the impact associated with EX202502 persists. We've resumed our investigation of the issue and are currently analyzing logs to determine why impact is continuing.
Next update by: Wednesday, February 12, 2020, at 7:30 AMUTC; PublishedTime=2020-02-12T05:40:42.63Z

Status: We've determined that the fix that was previously applied to resolve the cookie creation issue wasn't successful in mitigating impact. We're investigating why the fix was unsuccessful so that we can develop another mitigation plan.
Next update by: Wednesday, February 12, 2020, at 8:00 PMUTC; PublishedTime=2020-02-12T07:22:43.677Z

Status: After further review, we identified that around two percent of the affected environment hadn't yet received the fix deployment for this impact as early as February 10, 2020. We've confirmed that the fix has now saturated 99 percent of all affected infrastructure and should reach full saturation in the next 24 hours. We're reaching out to users that have reported this issue to confirm that the problem has been mitigated.
Next update by: Thursday, February 13, 2020, at 8:00 PMUTC; PublishedTime=2020-02-12T19:25:58.723Z

Status: We've identified that the fix hasn't yielded the expected results in resolving the problem fully and we're working to investigate why it hasn't mitigated impact for all affected users. In parallel, we're formulating an updated mitigation plan to resolve this problem.
Next update by: Thursday, February 13, 2020, at 10:00 PM UTC; PublishedTime=2020-02-13T19:59:17.813Z

Status: We're continuing our investigation intowhy the fix didn't work a expected in order toidentify an optimal mitigation action for resolution.
Next update by: Friday, February 14, 2020, at 8:00 PMUTC; PublishedTime=2020-02-13T21:59:43.653Z

Status: We've identified that the initial fix didn't bypass the default cookie values as expected, resulting in persistent search problems. We've made the necessary changes to the fix to bypass the default cookie values and we're verifying to ensure it fully resolves the issue.
Next update by: Saturday, February 15, 2020, at 3:00 AMUTC; PublishedTime=2020-02-14T18:59:32.397Z

Status: We've validated the fix and deployment was initiated on February 14, 2020. We expect the deployment to be fully saturated by Monday, February 17, 2020.
Next update by: Tuesday, February 18, 2020, at 3:00 AMUTC; PublishedTime=2020-02-15T01:33:06.743Z

Status: We've identified that the fix is taking longer than expected to complete deployment. So far, we can confirm that the fix has successfully saturated two thirds of the affected environments.
Next update by: Wednesday, February 19, 2020, at 3:00 AMUTC ; PublishedTime=2020-02-18T02:25:12.637Z

Final status: The fix has successfully deployed to all the impacted environments and we've reached out to impacted users to confirm they're able to perform searches again.
Next update by: Wednesday, February 19, 2020, at 3:00 AMUTC ; PublishedTime=2020-02-18T02:25:12.637Z

--------------------------------------------------------------------------------------------------
Title Can't view complete email subject
ID EX203373
Service Exchange Online
Feature E-Mail timely delivery
Status Service restored
Impact Users may have been unable to view the complete subject line for emails within the
Severity Sev2
Start time 2020-01-30T00:00:00Z
End time 2020-02-11T16:00:00Z
Last update 2020-02-11T21:54:46.397Z
More info The issue only occurs in the Outlook client. Users could have seen the full subject line in Outlook on the web or their mobile devices.
Root cause A recent update for the Outlook client introduced a code regression that prevents users from seeing the full content of the subject line within the Outlook client.

EX203373 Details

Status: We've identified a recent update for the Outlook client that has introduced a code regression, causing the email subject line to be partially hidden. We've developed a code fix which is undergoing validations and testing in preparation, and suspect that the solution will be fully deployed by Tuesday, February 11, 2020.
Next update by: Tuesday, February 11, 2020, at 11:00 PMUTC; PublishedTime=2020-02-07T20:17:00.857Z

Final status: We've confirmed that the code fix is available to anyone via the Backstage view. Users can manually update to receive the fix via Backstage by navigating through File > Office Account > Update Options and clicking Update Now to fix the issue.
Next update by: Tuesday, February 11, 2020, at 11:00 PMUTC; PublishedTime=2020-02-07T20:17:00.857Z

--------------------------------------------------------------------------------------------------
Title Inbox won't refresh in Outlook on the web
ID EX202989
Service Exchange Online
Feature E-Mail timely delivery
Status Post-incident report published
Impact User inboxes may not have seen new email until refreshing or after changing folders when
Severity Sev2
Start time 2020-01-30T05:10:00Z
End time 2020-02-12T02:00:00Z
Last update 2020-02-20T18:39:45.517Z
More info Users may notice this issue after signing in to Outlook on the web for approximately 30 minutes from any browser.
Root cause A change within Office portal to switch from outlook.office365.com to outlook.office.com caused an increase in channel notification disconnections for Outlook on the web.

EX202989 Details

Status: We're investigating a potential issue and checking for impact to your organization. We'll provide an update within 30 minutes.; PublishedTime=2020-02-04T03:08:23.217

Status: We're reviewing client logs of affected users to help determine the source of the issue.
Next update by: Tuesday, February 4, 2020, at 5:00 AM UTC; PublishedTime=2020-02-04T03:34:29.913Z

Status: We're reproducing the issue and gathering additional logs to help isolate the cause of the problem.
Next update by: Tuesday, February 4, 2020, at 6:00 AM UTC; PublishedTime=2020-02-04T04:42:50.293Z

Status: We're analyzing networking logs to gain a better understanding of how the requests are routed and where the issue is happening during the routing process.
Next update by: Tuesday, February 4, 2020, at 7:00 AMUTC; PublishedTime=2020-02-04T05:53:30.657Z

Status: We identified a recent change that modified how connection requests are directed to Outlook on the web. We're looking into making a settings change that will allow connection requests to call to Outlook on the web via alternate infrastructure.
Next update by: Tuesday, February 4, 2020, at 8:00 AMUTC; PublishedTime=2020-02-04T06:48:59.227Z

Status: We've confirmed that the settings change deployment has finished. Affected users will need to refresh their Outlook on the web browser page to enable these settings, which will cause the client to automatically reconnect to the server at five-minute intervals. Once this initial refresh is performed, users having this problem will see new email notifications within a maximum of five minutes after the message arrival. We're additionally working to enable enhanced telemetry logging within the service to gather information on the root cause. We expect this process to be completed within a few days.
Next update by: Thursday, February 13, 2020, at 3:00 AMUTC; PublishedTime=2020-02-12T03:01:06.823Z

Final status: We've confirmed via server telemetry that the corrective measures are an effective long-term solution to mitigate the problematic Outlook on the web behavior reported by users. We're beginning an extended period of monitoring and analysis to identify and fully address the underlying cause of this problem. Should we identify additional or resurfaced impact due to this problem, we will resume active investigation.
Next update by: Thursday, February 13, 2020, at 3:00 AMUTC; PublishedTime=2020-02-12T03:01:06.823Z

Status: We've determined that using https://outlook.office365.com/mail does not successfully mitigate the impact. The HTTP Archive (HAR) traces we've collected do not contain valid captures with a confirmed reproduction of the issue. We're investigating recent notification changes to explore other avenues of mitigation, the option of implementing more explicit logging, and reviewing retry logic to further troubleshoot the issue.
Next update by: Friday, February 7, 2020, at 2:00 AMUTC; PublishedTime=2020-02-06T07:56:32.277Z

Status: We're working to enable additional logging in a controlled portion of the affected environment to better understand how the requests are failing within our service. We're also working to implement a potential mitigation strategy within our test environment to determine if it will alleviate impact. In parallel, we're continuing to investigate our recent notification changes to determine if there may be an alternative means of providing relief for users.
Next update by: Saturday, February 8, 2020, at 2:00 AMUTC; PublishedTime=2020-02-07T01:12:19.313Z

Status: We've completed testing a solution to mitigate impact for affected users. Once implemented, this fix should allow users to receive new mail notifications without performing a page refresh. Furthermore, we're working to enable expanded logging in a portion of the affected environment to provide additional data on the root of the problem. We've initiated deployment of this change throughout the affected environment, which we anticipate will complete by Tuesday, February 11, 2020.
Next update by: Tuesday, February 11, 2020, at 9:00 PMUTC; PublishedTime=2020-02-08T00:50:13.033Z

Status: We've completed our staged rollout to approximately ten percent of the impacted infrastructure and confirmed that our solution is successful in mitigating impact. We're beginning deployment of the mitigation and expanded logging to the rest of the affected environment and expect this to finish within the next six hours.
Next update by: Wednesday, February 12, 2020, at 3:00 AMUTC; PublishedTime=2020-02-11T19:19:43.223Z

Status: We've made the settings change to allow connection requests to call to Outlook on the web via alternate infrastructure. We're testing whether these action steps have mitigated impact.
Next update by: Tuesday, February 4, 2020, at 9:30 AMUTC; PublishedTime=2020-02-04T07:47:12.9Z

Status: After a period of testing, we've been unable to reproduce the problem. We're monitoring the service to ensure that we've recovered the service fully.
Next update by: Wednesday, February 5, 2020, at 2:00 AM UTC; PublishedTime=2020-02-04T08:52:22.24Z

Status: We've received reports that impact persists after our attempted mitigation. We're reviewing network traces provided by affected users to help identify the next troubleshooting steps.
Next update by: Wednesday, February 5, 2020, at 6:00 AMUTC; PublishedTime=2020-02-05T00:53:12.383Z

Status: We've reviewed the network traces provided and the results were inconclusive. We've requested that some users provide specific HTTP Archive (HAR) traces to assist our investigation.
Next update by: Wednesday, February 5, 2020, at 9:00 AMUTC; PublishedTime=2020-02-05T05:12:33.363Z

Status: We're continuing to gather HTTP Archive (HAR) traces from affected users to help isolate the cause of the problem. Additionally, we've identified that signing out of Outlook on the web and then signing back in will resolve any cache issues and enable the routing change that was previously enabled, which should mitigate impact.
Next update by: Thursday, February 6, 2020, at 3:00 AMUTC; PublishedTime=2020-02-05T07:37:15.047Z

Status: We've gathered additional HTTP Archive (HAR) trace data showing reproductions of the issue and are reviewing the data to better understand the source of the problem.
Next update by: Thursday, February 6, 2020, at 8:30 AMUTC; PublishedTime=2020-02-06T02:38:59.247Z

--------------------------------------------------------------------------------------------------
Title Can't access archive mailbox
ID EX202959
Service Exchange Online
Feature E-Mail and calendar access
Status Service restored
Impact Users' archive mailboxes may have unexpectedly appeared as empty.
Severity Sev2
Start time 2019-12-17T09:20:00Z
End time 2020-02-12T12:00:00Z
Last update 2020-02-19T12:24:32.523Z
More info This was a continuation of EX199646.
Root cause A recent update to streamline the process of enabling and disabling mailboxes caused issues with archive mailboxes.

EX202959 Details

Status: We've reverted the change, which has stopped the issue from re-occurring, and we're continuing to gather data to manually reconnect the archive mailboxes for impacted users.
Next update by: Tuesday, February 4, 2020, at 6:00 AMUTC; PublishedTime=2020-02-03T23:39:00.22Z

Status: We've deployed a restoration script in order to reconnect the archive mailboxes to a subset number of affected users. We're reviewing diagnostic data to identify the remaining users who may still be impacted and who have not received the deployment yet.
Next update by: Thursday, February 6, 2020, at 6:30 AMUTC ; PublishedTime=2020-02-04T04:56:34.51Z

Status: We're continuing to run the script to mitigate affected users. In parallel, we're working to identify additional users that need to have their archives reconnected.
Next update by: Friday, February 7, 2020, at 6:30 AMUTC; PublishedTime=2020-02-06T05:04:16.743Z

Status: We've completed reconnecting archives for known impacted users. We're continuing to identify any other users that are affected so we can run the script for those users.
Next update by: Friday, February 7, 2020, at 12:30 PMUTC ; PublishedTime=2020-02-07T05:34:49.5Z

Status: We've completed reconnecting archives for known impacted users. We're working to identify additional users that may be affected by this issue so we can remediate impact by running the script to reconnect archives.
Next update by: Monday, February 10, 2020, at 12:30 PM UTC; PublishedTime=2020-02-07T10:59:02.92Z

Status: We've completed running the script, and 90% of the affected archives have been reconnected. We're monitoring the environment while the remaining archives receive the fix.
Next update by: Tuesday, February 11, 2020, at 12:30 PM UTC; PublishedTime=2020-02-10T10:44:35.79Z

Status: Although the vast majority of mailbox archives have been fixed, there remains a small number of affected archives that we are in the process of fixing.
Next update by: Thursday, February 13, 2020, at 12:30 PM UTC; PublishedTime=2020-02-11T11:05:00.47Z

Status: We've finished reconnecting the remaining archives that we identified were affected. We're investigating a small number of error messages have been generated since completion to confirm if the issue has been successfully resolved.
Next update by: Tuesday, February 18, 2020, at 12:30 PM UTC; PublishedTime=2020-02-13T11:16:49.423Z

Status: We've determined that 160 mailboxes did not reconnect at the first attempt. We expect these mailboxes to be processed and reconnected within the next 24 hours. Additionally, we've identified 48 mailboxes with inconsistent user ID's and are reviewing configuration data to isolate the cause of impact.
Next update by: Wednesday, February 19, 2020, at 12:30 PM UTC; PublishedTime=2020-02-18T11:55:38.877Z

Final status: We've confirmed that that impact has been resolved for the majority of customers. Those customers that are still affected by this issue will receive further updates through EX204189.
Next update by: Wednesday, February 19, 2020, at 12:30 PM UTC; PublishedTime=2020-02-18T11:55:38.877Z

--------------------------------------------------------------------------------------------------
Title Can't open encrypted emails or use one time passwords
ID EX202693
Service Exchange Online
Feature E-Mail and calendar access
Status Service restored
Impact Users may have been unable to open encrypted emails or use one time passwords.
Severity Sev1
Start time 2020-01-27T20:00:00Z
End time 2020-02-01T10:00:00Z
Last update 2020-02-02T03:43:52.87Z
More info Users may have gotten an error that said "We were unable to display this page, please wait five minutes and try again." when attempting to access encrypted emails.
Root cause A recent service update renewed certificates for some servers, but not all servers received the new certificates.

EX202693 Details

Status: We're investigating a potential issue and checking for impact to your organization. We'll provide an update within 30 minutes. ; PublishedTime=2020-01-31T03:21:23.643

Status: We are reviewing logs provided by your organization to determine a root cause and identify our next troubleshooting steps.
Next update by: Friday, January 31, 2020, at 7:00 AMUTC ; PublishedTime=2020-01-31T04:27:59.393Z

Status: After reviewing the logs we found that the certificates on some servers were changed in a recent service update, but some servers did not receive the updated certificates, causing impact. We are redeploying the new certificates to the servers that didn't receive them to mitigate impact.
Next update by: Saturday, February 1, 2020, at 6:00 AMUTC; PublishedTime=2020-01-31T05:49:57.543Z

Status: Our telemetry indicates that the new certificates have been redeployed to approximately 70 percent of the affected servers. Some customers will begin to see resolution as the certificates continue to be deployed throughout the affected infrastructure.
Next update by: Sunday, February 2, 2020, at 6:00 AMUTC; PublishedTime=2020-02-01T04:19:58.377Z

Final status: We've confirmed that the new certificate has been deployed to all affected servers and our telemetry indicates that the issue is resolved.
Next update by: Sunday, February 2, 2020, at 6:00 AMUTC; PublishedTime=2020-02-01T04:19:58.377Z

--------------------------------------------------------------------------------------------------
Title Can't view/download Security and Compliance Center (SCC) message traces
ID EX202641
Service Exchange Online
Feature Management and Provisioning
Status Post-incident report published
Impact Admins may have been unable to view or download offline message traces older than 10
Severity Sev2
Start time 2020-01-12T23:43:00Z
End time 2020-01-31T05:00:00Z
Last update 2020-02-20T18:39:13.997Z
More info While we were focused on remediation, admins were able to obtain message traces through PowerShell as an alternative method. Additionally, admins may have been able to mitigate the impact by clearing their browser cache and cookies or by opening the download link in private browsing.
Root cause A change in the service caused cookies to be validated incorrectly, resulting in impact to viewing or downloading message traces from the SCC.

EX202641 Details

Status: An issue is occurring where two cookies from client browsers are missing during requests to view or download message traces from the SCC, which results in this error. We're collecting and reviewing network trace data from working and non-working scenarios to isolate where the error is occurring and determine how to resolve the event.
Next update by: Thursday, January 30, 2020, at 10:30 PM UTC ; PublishedTime=2020-01-30T17:42:10.81Z

Status: We're continuing to capture and review the associated network trace data in order to better understand the issue. In parallel, we're working to reproduce the problem in multiple browsers to determine whether the impact is limited to specific scenarios.
Next update by: Friday, January 31, 2020, at 8:30 PM UTC; PublishedTime=2020-01-30T22:48:39.977Z

Status: MessageText=Title: Can't view/download Security and Compliance Center (SCC) message traces
User Impact: Admins may be unable to view or download offline message traces older than 10 days from the SCC.

More info:While we're focused on remediation, admins can obtain message traces through PowerShell as an alternative method. Additionally, admins may be able to mitigate the impact by clearing their browser cache and cookies or by opening the download link in private browsing.

This issue was previously communicated under EX200226, and this new incident has been opened to address continued reports from impacted customers.

Currentstatus:We've developed, validated, and deployed a fix that will address the cookie issue to the impacted environment. We're reaching out to your representatives to confirm that the issue has been resolved.

Next update by: Friday, January 31, 2020, at 8:30 PM UTC; PublishedTime=2020-01-30T22:48:39.977Z

Final status: We've confirmed via internal attempts to reproduce the issue and user reports that the issue has been successfully resolved.
Next update by: Friday, January 31, 2020, at 8:30 PM UTC; PublishedTime=2020-01-30T22:48:39.977Z

--------------------------------------------------------------------------------------------------
Title Can't view shared calendars in Outlook on the web
ID EX202603
Service Exchange Online
Feature E-Mail and calendar access
Status Service restored
Impact Delegate users may have been unable to view shared calendars in Outlook on the web.
Severity Sev2
Start time 2020-01-06T08:00:00Z
End time 2020-02-10T09:31:40Z
Last update 2020-02-10T09:33:38.473Z
More info As an alternate access method, delegate users could use the Outlook desktop client to view shared calendars.
Root cause A recent change to the environment, which inadvertently affected calls to Outlook on the web from a component used to retrieve data for calendar information, resulted in impact.

EX202603 Details

Status: We're investigating a potential issue and checking for impact to your organization. We'll provide an update within 30 minutes.


Status: We're analyzing information provided by your organization to determine the source of the issue. In parallel, we're gathering system logs for further analysis.
Next update by: Thursday, January 30, 2020, at 5:30 AMUTC; PublishedTime=2020-01-30T03:44:18.833Z

Status: We've analyzed network trace logs and found a 404 error. We're continuing to analyze the provided support case information, as well as Outlook on the web server logs, to identify what's causing the error.
Next update by: Thursday, January 30, 2020, at 9:00 PMUTC; PublishedTime=2020-01-30T05:01:11.427Z

Status: Our log review has proven inconclusive, and we've been unable to reproduce impact in our internal testing environments. We're currently reviewing the code involved in this process and are looking to see if any recent changes have been made. In parallel, we're working to reproduce impact on a test account in the production environment to gather additional information about impact.
Next update by: Thursday, January 30, 2020, at 11:00 PMUTC; PublishedTime=2020-01-30T20:55:30.943Z

Status: Our code review was inconclusive, as we found no issues in the code nor any recent changes which may have resulted in impact. However, we've successfully reproduced the impact via testing, and we're collecting additional logs from our reproduction for further debugging.
Next update by: Friday, January 31, 2020, at 1:00 AMUTC; PublishedTime=2020-01-30T22:50:03.69Z

Status: We've identified a recent change to the environment which inadvertently affected calls to Outlook on the web from a component used to retrieve data for calendar information, resulting in impact. We're developing a code fix to repair this connection and mitigate impact.
Next update by: Friday, January 31, 2020, at 6:00 PMUTC; PublishedTime=2020-01-31T00:57:05.06Z

Status: We've finished development of the code fix and we're working to verify this fix before deploying it to the entirety of the affected infrastructure. We suspect the fix will start deploying within the next ten hours and that it will take two days to fully saturate.
Next update by: Saturday, February 1, 2020, at 4:00 AMUTC; PublishedTime=2020-01-31T16:18:06.263Z

Status: We've successfully tested and validated the fix and have initiated deployment. We anticipate the fix saturating all affected environments by Monday, February 3, 2020.
Next update by: Monday, February 3, 2020, at 10:00 AMUTC; PublishedTime=2020-02-01T01:40:57.743Z

Status: We've confirmed the fix has been successfully deployed. Saturation of the fix is taking longer than expected, and is now anticipated to be completed by Monday, February 10, 2020.
Next update by: Monday, February 10, 2020, at 10:00 AM UTC; PublishedTime=2020-02-03T08:19:06.583Z

Final status: We've confirmed the deployed fix has now fully saturated, and the issue has been resolved.
Next update by: Monday, February 10, 2020, at 10:00 AM UTC; PublishedTime=2020-02-03T08:19:06.583Z

--------------------------------------------------------------------------------------------------
Title Can't use Attach files button in Outlook on the web
ID EX202386
Service Exchange Online
Feature E-Mail and calendar access
Status Service restored
Impact Users may have been unable to attach files to emails using the Attach button in Outlook
Severity Sev2
Start time 2020-01-21T17:00:00Z
End time 2020-01-28T17:40:00Z
Last update 2020-01-28T20:43:55.107Z
More info While we were focused on remediation, users could have attached files by dragging and dropping the file into the email.
Root cause A recent update to Outlook on the web intended to improve attachment suggestions was inadvertently causing a scripting error when users had incorrect attachment suggestions from the suggestion service, resulting in the inability to attach files using the Attach files button.

EX202386 Details

Status: We're investigating a potential issue and checking for impact to your organization. We'll provide an update within 30 minutes.; PublishedTime=2020-01-27T21:12:29.693

Status: We've determined that a recent update to Outlook on the web intended to improve attachment suggestions is inadvertently causing a scripting error when users have incorrect attachment suggestions from the suggestion service, resulting in the inability to attach files using the Attach files button. We've developed a fix for this problem which is currently being validated to ensure it resolves the issue before being deployed to the affected environment.
Next update by: Tuesday, January 28, 2020, at 1:00 AMUTC; PublishedTime=2020-01-27T21:42:43.147Z

Status: We validated the fix and have initiated the deployment process. We'll monitor the deployment to ensure the impact is remediated.
Next update by: Tuesday, January 28, 2020, at 7:00 PMUTC; PublishedTime=2020-01-28T00:52:43.073Z

Status: We've confirmed that the fix has been fully deployed and have verified with some affected users that the issue is resolved; however, we're awaiting for additional user confirmation to ensure that the problem is fully mitigated.
Next update by: Tuesday, January 28, 2020, at 9:00 PMUTC; PublishedTime=2020-01-28T18:57:03.623Z

Final status: We've confirmed with additional users that the issue is no longer occurring following the fix deployment.
Next update by: Tuesday, January 28, 2020, at 9:00 PMUTC; PublishedTime=2020-01-28T18:57:03.623Z

--------------------------------------------------------------------------------------------------
Title Can't update meetings
ID EX202170
Service Exchange Online
Feature E-Mail and calendar access
Status Service restored
Impact Users may have intermittently been unable to update calendar meetings and resource
Severity Sev2
Start time 2020-01-03T08:00:00Z
End time 2020-02-01T01:00:00Z
Last update 2020-02-03T21:49:19.253Z
More info Users may have been able to attempt to make updates, but the changes would not take effect, thus users did not receive meeting update emails.
Root cause A miscategorization of meeting update requests intermittently prevented users from updating calendar meetings and resource booking.

EX202170 Details

Status: We've identified that meeting update requests are being miscategorized. We suspect that the problem may correlated to a recent service update. We're working to confirm our findings, and pending confirmation, anticipate that a potential fix for this issue will be verified and ready for deployment by Monday, January 27, 2020.
Next update by: Monday, January 27, 2020, at 12:00 AMUTC; PublishedTime=2020-01-24T22:55:36.98Z

Status: We've developed the fix designed to correct the categorization problem affecting the meeting update requests and deployed it to our internal testing environment. We're testing the deployment to confirm that it will be a viable solution for remediating impact. In parallel, we're continuing to investigate our recent service updates to confirm the source of this event and prevent it from reoccurring in the future.
Next update by: Tuesday, January 28, 2020, at 1:00 AMUTC ; PublishedTime=2020-01-26T23:29:22.313Z

Status: We've completed deployment and testing of the fix in our internal testing environment and confirmed this fix mitigates the issue. We're preparing the fix for deployment to the impacted infrastructure and will provide a timeline of the deployment schedule once the fix has been deployed.
Next update by: Tuesday, January 28, 2020, at 8:00 PMUTC ; PublishedTime=2020-01-27T23:52:05.627Z

Status: We've begun the deployment of the fix to the affected environments. We expect to provide an updated timeline of the deployment schedule tomorrow, Wednesday, January 29, 2020, as the fix continues to propagate across the affected infrastructure.
Next update by: Wednesday, January 29, 2020, at 8:00 PM UTC; PublishedTime=2020-01-28T18:46:36.147Z

Status: The fix continues to deploy into the affected environments, and we expect the deployment to fully propagate across the impacted infrastructure by end-of-day on Friday, January 31, 2020.
Next update by: Saturday, February 1, 2020, at 12:00 AMUTC; PublishedTime=2020-01-29T18:51:38.187Z

Status: The fix continues its deployment, and has nearly saturated all affected environments. We're continuing to monitor and expect it to complete by Monday, February 3, 2020.
Next update by: Monday, February 3, 2020, at 11:00 PMUTC; PublishedTime=2020-01-31T23:30:38.453Z

Final status: After monitoring the deployment, we've confirmed that the fix has saturated throughout the affected environment and that the issue is resolved.
Next update by: Monday, February 3, 2020, at 11:00 PMUTC; PublishedTime=2020-01-31T23:30:38.453Z

--------------------------------------------------------------------------------------------------
Title Can't access email
ID EX202161
Service Exchange Online
Feature E-Mail and calendar access
Status Service restored
Impact Users may have been intermittently unable to access their email through Exchange Online.
Severity Sev2
Start time 2020-01-24T21:10:00Z
End time 2020-01-24T22:25:00Z
Last update 2020-01-25T17:18:11.873Z
More info This may have affected all protocols including Outlook on the web, the Outlook client, and Exchange ActiveSync.
Root cause A subset of Exchange Online infrastructure was attempting an auto-optimization of traffic, but failed, resulting in impact

EX202161 Details

Final status: We've identified that a subset of Exchange Online infrastructure was attempting an auto-optimization of traffic, but failed, resulting in impact. We've restarted the impacted infrastructure and have confirmed that this action restored service.


--------------------------------------------------------------------------------------------------
Title Can't sign in to Outlook on the web
ID EX201977
Service Exchange Online
Feature E-Mail and calendar access
Status Service restored
Impact Users may have been unable to sign in to Outlook on the web.
Severity Sev1
Start time 2020-01-23T01:45:00Z
End time 2020-01-23T10:49:00Z
Last update 2020-01-23T11:35:38.903Z
More info This issue only impacted users who did not have Outlook on the web cache. Users who were already signed in may not have experienced impact.
Root cause A configuration update for the Hypertext Transfer Protocol (HTTP) proxy contained a code error.

EX201977 Details

Status: We're investigating a potential issue and checking for impact to your organization. We'll provide an update within 30 minutes.


Status: We've determined that a configuration update for the Hypertext Transfer Protocol (HTTP) proxy contained a code error. We've developed and validated a fix for the code error and are deploying it to the affected infrastructure.
Next update by: Thursday, January 23, 2020, at 5:00 AMUTC; PublishedTime=2020-01-23T03:01:40.493Z

Status: The deployment is progress as expected and is approximately 95 percent complete.
Next update by: Thursday, January 23, 2020, at 7:00 AMUTC; PublishedTime=2020-01-23T04:55:33.367Z

Status: The deployment is taking longer than expected. We're investigating ways to expedite the process and we're continuing monitoring the deployment status.
Next update by: Thursday, January 23, 2020, at 9:00 AMUTC; PublishedTime=2020-01-23T06:55:11.707Z

Status: We're in the final stages of deployment and can confirm that users will start to experience relief as it progresses across the environment.
Next update by: Thursday, January 23, 2020, at 11:00 AM UTC ; PublishedTime=2020-01-23T08:52:58.783Z

Final status: The deployment has been fully applied and we've confirmed that the issue is resolved after monitoring the service.
Next update by: Thursday, January 23, 2020, at 11:00 AM UTC ; PublishedTime=2020-01-23T08:52:58.783Z

top

Information Systems and Computing
University of Pennsylvania
Comments & Questions


University of Pennsylvania Penn Computing University of Pennsylvania Information Systems & Computing (ISC)
Information Systems and Computing, University of Pennsylvania