Penn Computing
Computing Menu Computing A-Z
Computing Home Information Systems & Computing Penn

Microsoft's Incident Report for PennO365

Following is a data extract from the PennO365 administrative console of incident reports sent by Microsoft within the last 30 days. We pull out incidents that are relevant to the current PennO365 service offerings and we refresh this data every 15 minutes. The last extract was taken on October 21, 2019 13:15.

--------------------------------------------------------------------------------------------------
Title TM191518
Service Affected Microsoft Teams
Status Service restored
Time 9/25/2019 3:51:00 AM UTC
LastUpdated 9/28/2019 6:04:38 AM UTC
Message Title: Azure Active Directory (AAD) sync delays

TM191518 Details

==========
User Impact: Users may see delayed AAD sync for changes to user objects within Microsoft Teams.

More info: The problem affects AAD sync for newly added Teams users and changes to existing user objects within Microsoft Teams. This behavior will manifest in stale @ mention lists.

Current status: We've identified provisioning that recently took place when reviewing the system logs that we suspect contributed to impact. We're continuing to examine the AAD service to determine our next troubleshooting steps.

Scope of impact: Your organization is affected by this event, and any of your users may be impacted.

Start time: Wednesday, September 25, 2019, at 3:51 AM UTC

Next update by: Thursday, September 26, 2019, at 9:30 PM UTC, Title: Azure Active Directory (AAD) sync delays

==========
User Impact: Users may see delayed AAD sync for changes to user objects within Microsoft Teams.

More info: The problem affects AAD sync for newly added Teams users and changes to existing user objects within Microsoft Teams. This behavior will manifest in stale @ mention lists.

Current status: We've confirmed that while the configuration change has been enabled, there is currently a backlog of AAD sync changes. We're monitoring the backlog and anticipate it will be fully processed within the next four to five hours.

Scope of impact: Your organization is affected by this event, and any of your users may be impacted.

Start time: Wednesday, September 25, 2019, at 3:51 AM'UTC

Root cause: A subset of AAD infrastructure fell below service performance thresholds which resulted in delayed AAD sync for changes to user objects within the Microsoft Teams service.

Next update by: Saturday, September 28, 2019, at 10:00 AMUTC, Title: Azure Active Directory (AAD) sync delays

==========
User Impact: Users may have seen delayed AAD sync for changes to user objects within Microsoft Teams.

More info: The problem affected AAD sync for newly added Teams users and changes to existing user objects within Microsoft Teams. This behavior would have manifested in stale @ mention lists.

Final status: We've confirmed that the backlog of AAD sync changes has been fully processed and this issue is now resolved.

Scope of impact: Your organization was affected by this event, and any of your users may have been impacted.

Start time: Wednesday, September 25, 2019, at 3:51 AM'UTC

End time: Saturday, September 28, 2019, at 3:55 AM'UTC

Root cause: A subset of AAD infrastructure fell below service performance thresholds which resulted in delayed AAD sync for changes to user objects within the Microsoft Teams service.

Next steps: - We're reviewing data associated with the affected infrastructure to determine to determine the cause of the performance issue and steps which can be taken to prevent this from occurring in the future.

This is the final update for the event., Title: Azure Active Directory (AAD) sync delays

==========
User Impact: Users may see delayed AAD sync for changes to user objects within Microsoft Teams.

More info: The problem affects AAD sync for newly added Teams users and changes to existing user objects within Microsoft Teams. This behavior will manifest in stale @ mention lists.

Current status: We've determined that a subset of AAD infrastructure has dropped below service performance thresholds, creating a backlog of service requests and delaying AAD sync for changes to user objects within the Microsoft Teams service. We're conducting a configuration change which is designed to remediate impact. We're expecting the configuration change to be fully enabled across all environments by Saturday, September 28, 2019, which is expected to remediate impact.

Scope of impact: Your organization is affected by this event, and any of your users may be impacted.

Start time: Wednesday, September 25, 2019, at 3:51 AM UTC

Root cause: A subset of AAD infrastructure fell below service performance thresholds which resulted in delayed AAD sync for changes to user objects within the Microsoft Teams service.

Next update by: Saturday, September 28, 2019, at 6:00 AM UTC, Title: Azure Active Directory (AAD) sync delays

==========
User Impact: Users may see delayed AAD sync for changes to user objects within Microsoft Teams.

More info: The problem affects AAD sync for newly added Teams users and changes to existing user objects within Microsoft Teams. This behavior will manifest in stale @ mention lists.

Current status: We've identified an issue that is causing AAD sync delays for newly-created user objects or changes to existing user objects within Microsoft Teams. We're examining the problem to identify the best method to mitigate impact.

Scope of impact: Your organization is affected by this event, and any of your users may be impacted.

Start time: Wednesday, September 25, 2019, at 3:51 AM UTC

Next update by: Wednesday, September 25, 2019, at 9:30 PM UTC}
--------------------------------------------------------------------------------------------------
Title EX190647
Service Affected Exchange Online
Status Investigating
Time 9/9/2019 10:29:00 AM UTC
LastUpdated 10/8/2019 7:17:09 PM UTC
Message Title: Retention policy not working for Skype for Business messages in inbox

EX190647 Details

==========
User Impact: Skype for Business messages are not routing to the correct folder as dictated by the retention policy.

More info: Users are seeing the Skype for Business messages in their "Conversation History" folder in Outlook and Outlook on the web longer than their retention policy dictates.

While we work on a long-term solution, we recommend users apply the following manual mitigation:

Please modify any currently assigned retention policies that have tags for the Conversation History folder. Modifying the retention policy will initiate a full crawl which ensures that every mailbox will have the proper tag restamped on the Conversation History folder. The full crawl can take up to seven days to complete.

Current status: We've completed our development of a long-term solution which is undergoing testing in an internal environment. Due to the complexity of the solution, we expect that the testing and deployment process may require several weeks to complete.

Scope of impact: This issue may potentially affect any of your users who utilize the affected retention policies.

Start time: Monday, September 9, 2019, at 10:29 AM UTC

Estimated time to resolve: We anticipate that the testing and deployment process may require between two and four weeks to complete.

Root cause: A code defect is causing impact to the functionality of retention policies that have tags for the Conversation History folder.

Next update by: Tuesday, October 8, 2019, at 8:00 PM UTC, Title: Retention policy not working for Skype for Business messages in inbox

==========
User Impact: Skype for Business messages are not routing to the correct folder as dictated by the retention policy.

More info: Users are seeing the Skype for Business messages in their "Conversation History" folder in Outlook and Outlook on the web longer than their retention policy dictates.

While we work on a long-term solution, we recommend users apply the following manual mitigation:

Please modify any currently assigned retention policies that have tags for the Conversation History folder. Modifying the retention policy will initiate a full crawl which ensures that every mailbox will have the proper tag restamped on the Conversation History folder. The full crawl can take up to seven days to complete.

Current status: We're continuing our development of a long-term solution and we'll provide an estimated timeline for deployment as soon as one is available.

Scope of impact: This issue may potentially affect any of your users who utilize the affected retention policies.

Start time: Monday, September 9, 2019, at 10:29 AM UTC

Root cause: A code defect is causing impact to the functionality of retention policies that have tags for the Conversation History folder.

Next update by: Tuesday, September 24, 2019, at 8:00 PM UTC, Title: Retention policy not working for Skype for Business messages in inbox

==========
User Impact: Skype for Business messages are not routing to the correct folder as dictated by the retention policy.

More info: Users are seeing the Skype for Business messages in their "Conversation History" folder in Outlook and Outlook on the web longer than their retention policy dictates.

While we work on a long-term solution, we recommend users apply the following manual mitigation:

Please modify any currently assigned retention policies that have tags for the Conversation History folder. Modifying the retention policy will initiate a full crawl which ensures that every mailbox will have the proper tag restamped on the Conversation History folder. The full crawl can take up to seven days to complete.

Current status: We've completed testing of the fix and have initiated deployment throughout the affected infrastructure. We estimate that our solution may require two to three weeks to fully propagate.

Scope of impact: This issue may potentially affect any of your users who utilize the affected retention policies.

Start time: Monday, September 9, 2019, at 10:29 AM UTC

Estimated time to resolve: We anticipate the fix to be fully deployed within the next two to three weeks.

Root cause: A code defect is causing impact to the functionality of retention policies that have tags for the Conversation History folder.

Next update by: Tuesday, October 22, 2019, at 8:00 PM UTC, Title: Email messages for certain retention policy aren't sent to correct folder

==========
User Impact: Users are seeing email messages that meet the requirements of a specific retention policy in an incorrect inbox folder.

More info: User are seeing the affected email messages in their "Conversation History" folder in Outlook and Outlook on the web.

Current status: We're checking a reproduction of the issue for some affected users to determine how the issue occurs and our next steps in the troubleshooting process.

Scope of impact: This issue may potentially affect any of your users who utilize the affected retention policy.

Next update by: Monday, September 16, 2019, at 6:30 PM UTC, Title: Email messages for certain retention policy aren't sent to correct folder

==========
User Impact: Users are seeing email messages that meet the requirements of a specific retention policy in an incorrect inbox folder.

More info: User are seeing the affected email messages in their "Conversation History" folder in Outlook and Outlook on the web.

Current status: We've requested sample email messages from an affected customer to ensure the messages are being marked correctly in relation to the affected retention policy.

Scope of impact: This issue may potentially affect any of your users who utilize the affected retention policy.

Next update by: Monday, September 16, 2019, at 8:30 PM UTC, Title: Retention policy not working for Skype for Business messages in inbox

==========
User Impact: Users are seeing their Skype for Business messages not routing to the correct inbox folder per the retention policy.

More info: Users are seeing the Skype for Business messages in their "Conversation History" folder in Outlook and Outlook on the web.

Current status: We've determined that a code defect is causing impact to the functionality of the affected retention policy. While we work on developing a permanent fix, we've recommend that the affected customers modify the existing retention policy. Once complete, it will initiate a full crawl will be initiated that will take approximately one week to complete. We're also working to determine how the code issue was introduced.

Scope of impact: This issue may potentially affect any of your users who utilize the affected retention policy.

Start time: Monday, September 9, 2019, at 10:29 AM UTC

Root cause: A code defect is causing impact to the functionality of the affected retention policy.

Next update by: Tuesday, September 17, 2019, at 4:00 PM UTC, Title: Retention policy not working for Skype for Business messages in inbox

==========
User Impact: Users are seeing their Skype for Business messages not routing to the correct folder per the retention policy.

More info: Users are seeing the Skype for Business messages in their "Conversation History" folder in Outlook and Outlook on the web longer than their retention policy dictates.

To mitigate this issue, please modify any retention policies currently assigned that have tags for the Conversation History folder. Modifying the retention policy will initiate a full crawl which ensures that every mailbox will have the proper tag re-stamped on the Conversation History folder. The full crawl can take up to seven days to complete.

Current status: We're continuing to work to develop a permanent fix and investigate the source of the code defect. While we work to develop a long-term solution, we've provided a workaround for admins to follow in the "more info" section of this message.

Scope of impact: This issue may potentially affect any of your users who utilize the affected retention policy.

Start time: Monday, September 9, 2019, at 10:29 AM UTC

Root cause: A code defect is causing impact to the functionality of the affected retention policy.

Next update by: Tuesday, September 17, 2019, at 7:00 PM UTC, Title: Email messages for certain retention policy aren't sent to correct folder

==========
User Impact: Users are seeing email messages that meet the requirements of a specific retention policy in an incorrect inbox folder.

Current status: We're investigating a potential issue where users are seeing email messages that meet the requirements of a specific retention policy in an incorrect inbox folder. We'll provide an update within 30 minutes. , Title: Retention policy not working for Skype for Business messages in inbox

==========
User Impact: Skype for Business messages are not routing to the correct folder as dictated by the retention policy.

More info: Users are seeing the Skype for Business messages in their "Conversation History" folder in Outlook and Outlook on the web longer than their retention policy dictates.

While we work on a long-term solution, we recommend users apply the following manual mitigation:

Please modify any currently assigned retention policies that have tags for the Conversation History folder. Modifying the retention policy will initiate a full crawl which ensures that every mailbox will have the proper tag restamped on the Conversation History folder. The full crawl can take up to seven days to complete.

Current status: We're working to develop the long-term solution which is expected to take several days to complete. While we're focused on delivering the solution as soon as possible, we recommend admins utilize the provided manual mitigation as a workaround.

Scope of impact: This issue may potentially affect any of your users who utilize the affected retention policies.

Start time: Monday, September 9, 2019, at 10:29 AM UTC

Root cause: A code defect is causing impact to the functionality of retention policies that have tags for the Conversation History folder.

Next update by: Friday, September 20, 2019, at 8:00 PM UTC}
--------------------------------------------------------------------------------------------------
Title MO191513
Service Affected Office 365 Portal
Status Service restored
Time 8/30/2019 7:00:00 PM UTC
LastUpdated 10/18/2019 3:32:25 PM UTC
Message Title: Problems accessing Microsoft Search analytics

MO191513 Details

==========
User Impact: Admins may not be able to access any new analytics from The Analytics page within the Microsoft Search Settings.

More info: Data since August 30, 2019, may not be available for access.

Current status: The fix continues to saturate in the affected infrastructure. We maintain our efforts to ensure the process continues successfully. We're also working to identify an estimated time for service restoration.

Scope of impact: This issue may potentially affect any admin attempting to access the Analytics page within Microsoft Search Settings.

Start time: Friday, August 30, 2019, at 7:00 PM UTC

Root cause: A subset of infrastructure was consuming additional memory, causing our automated recovery system to restart the infrastructure too frequently.

Next update by: Thursday, September 26, 2019, at 10:00 PM UTC, Title: Problems accessing Microsoft Search analytics

==========
User Impact: Admins may not be able to access any new analytics from The Analytics page within the Microsoft Search Settings.

More info: Data since August 30, 2019, may not be available for access.

Current status: We're continuing to process data through the analytics portal and expect that this data will be up-to-date by Sunday, October 13, 2019. In parallel, we're looking into further possibilities to expedite this process.

Scope of impact: This issue may potentially affect any admin attempting to access the Analytics page within Microsoft Search Settings.

Start time: Friday, August 30, 2019, at 7:00 PM'UTC

Estimated time to resolve: We expect to see impact mitigation by Sunday, October 13, 2019.

Root cause: A subset of infrastructure was consuming additional memory, causing our automated recovery system to restart the infrastructure too frequently.

Next update by: Thursday, October 10, 2019, at 9:00 PMUTC, Title: Problems accessing Microsoft Search analytics

==========
User Impact: Admins may not be able to access any new analytics from The Analytics page within the Microsoft Search Settings.

More info: Data since August 30, 2019, may not be available for access.

Current status: The fix is has been deployed, and we're actively monitoring the fix as it saturates to the affected infrastructure to ensure it progresses successfully.

Scope of impact: This issue may potentially affect any admin attempting to access the Analytics page within Microsoft Search Settings.

Start time: Friday, August 30, 2019, at 7:00 PM UTC

Root cause: A subset of infrastructure was consuming additional memory, causing our automated recovery system to restart the infrastructure too frequently.

Next update by: Thursday, September 26, 2019, at 5:00 PM UTC, Title: Problems accessing Microsoft Search analytics

==========
User Impact: Admins may not be able to access any new analytics from The Analytics page within the Microsoft Search Settings.

More info: Data since August 30, 2019, may not be available for access.

Current status: We've experienced a couple of delays in deployment and processing, but have noted an uptick in speed and expect processing to complete by the evening on Wednesday, October 16, 2019.

Scope of impact: This issue may potentially affect any admin attempting to access the Analytics page within Microsoft Search Settings.

Start time: Friday, August 30, 2019, at 7:00 PM'UTC

Estimated time to resolve: We expect to see impact mitigation by Wednesday, October 16, 2019.

Root cause: A subset of infrastructure was consuming additional memory, causing our automated recovery system to restart the infrastructure too frequently.

Next update by: Wednesday, October 16, 2019, at 9:00 PMUTC, Title: Problems accessing Microsoft Search analytics

==========
User Impact: Admins may not be able to access any new analytics from The Analytics page within the Microsoft Search Settings.

More info: Data since August 30, 2019, may not be available for access.

Current status: The fix has saturated the affected infrastructure and the underlying problem has been fixed. Given the excessive amount of data that the analytics portal will be processing, it may take over two weeks to see full impact mitigation. We'll be monitoring situation and providing optimization updates to speed up this process, if possible.

Scope of impact: This issue may potentially affect any admin attempting to access the Analytics page within Microsoft Search Settings.

Start time: Friday, August 30, 2019, at 7:00 PM UTC

Estimated time to resolve: We expect to see impact mitigation by Sunday, October 13th, 2019.

Root cause: A subset of infrastructure was consuming additional memory, causing our automated recovery system to restart the infrastructure too frequently.

Next update by: Thursday, October 3, 2019, at 9:00 PM UTC, Title: Can't access new Microsoft Search analytics

==========
User Impact: Admins may have been unable to access any new analytics from The Analytics page within the Microsoft Search Settings.

More info: From October 15 and later may have not been available. Previously, data since August 30, 2019, wasn't accessible.

Final status: We've confirmed that data is now up-to-date and the issue has been resolved.

Scope of impact: This issue may have potentially affected any admin attempting to access the Analytics page within Microsoft Search Settings.

Start time: Friday, August 30, 2019, at 7:00 PM'UTC

End time: Friday, October 18, 2019, at 2:30 PM'UTC

Root cause: A subset of infrastructure was consuming additional memory, causing our automated recovery system to restart the infrastructure too frequently.

Next steps: -We're reviewing the infrastructure facilitating Microsoft Search analytics to prevent additional memory consumption from triggering automated restarts too often in the future.

This is the final update for the event., Title: Problems accessing Microsoft Search analytics

==========
User Impact: Admins may not be able to access any new analytics from The Analytics page within the Microsoft Search Settings.

More info: Data since August 30, 2019 may not be available for access.

Current status: We've determined that a subset of infrastructure that handles the processing of data was taking additional memory, causing our automated recovery system to restart the infrastructure, resulting in data not being processed completely. To mitigate impact, we're applying a fix to minimize the automated restarts to the infrastructure. Once the fix has been applied, we'll re-process all of the affected data.

Scope of impact: This issue may potentially affect any admin attempting to access the Analytics page within Microsoft Search Settings.

Start time: Friday, August 30, 2019, at 7:00 PM UTC

Root cause: A subset of infrastructure was consuming additional memory, causing our automated recovery system to restart the infrastructure too frequently.

Next update by: Wednesday, September 25, 2019, at 5:00 PM UTC, Title: Problems accessing Microsoft Search analytics

==========
User Impact: Admins may not be able to access any new analytics from The Analytics page within the Microsoft Search Settings.

More info: Data since August 30, 2019, may not be available for access.

Current status: We've confirmed that progress is continuing as expected, and data up to October 10, 2019 should now be available. We're continuing to monitor progress as it nears completion, which is anticipated to be within the next twenty four hours.

Scope of impact: This issue may potentially affect any admin attempting to access the Analytics page within Microsoft Search Settings.

Start time: Friday, August 30, 2019, at 7:00 PM'UTC

Estimated time to resolve: We expect to see impact mitigation by Thursday, October 17, 2019.

Root cause: A subset of infrastructure was consuming additional memory, causing our automated recovery system to restart the infrastructure too frequently.

Next update by: Thursday, October 17, 2019, at 9:00 PMUTC, Title: Problems accessing Microsoft Search analytics

==========
User Impact: Admins may not be able to access any new analytics from The Analytics page within the Microsoft Search Settings.

More info: Data since August 30, 2019, may not be available for access.

Current status: Data continues to reprocess and based on our current progress of the data, we expect that the issue will be fully resolved by Monday, October 14, 2019.

Scope of impact: This issue may potentially affect any admin attempting to access the Analytics page within Microsoft Search Settings.

Start time: Friday, August 30, 2019, at 7:00 PM'UTC

Estimated time to resolve: We expect to see impact mitigation by Monday, October 14, 2019.

Root cause: A subset of infrastructure was consuming additional memory, causing our automated recovery system to restart the infrastructure too frequently.

Next update by: Monday, October 14, 2019, at 9:00 PMUTC, Title: Can't access new Microsoft Search analytics

==========
User Impact: Admins may not be able to access any new analytics from The Analytics page within the Microsoft Search Settings.

More info: Data from October 15 and later may not be available. Previously, data since August 30, 2019, wasn't accessible.

Current status: We've verified that user information is current through October 14, 2019, and we expect the results to be fully caught up in approximately 24 hours.

Scope of impact: This issue may potentially affect any admin attempting to access the Analytics page within Microsoft Search Settings.

Start time: Friday, August 30, 2019, at 7:00 PM'UTC

Estimated time to resolve: We expect to see full impact mitigation by the end of Friday, October 18, 2019.

Root cause: A subset of infrastructure was consuming additional memory, causing our automated recovery system to restart the infrastructure too frequently.

Next update by: Friday, October 18, 2019, at 6:00 PM'UTC}
top

Information Systems and Computing
University of Pennsylvania
Comments & Questions


University of Pennsylvania Penn Computing University of Pennsylvania Information Systems & Computing (ISC)
Information Systems and Computing, University of Pennsylvania