How To Use Microsoft 365 Productivity Score To Drive Digital Transformation

With the recent public release of the Microsoft 365 Productivity Score. It has never been easier to assess your organization’s adoption of Microsoft collaboration tools, identify areas for improvement and plan to help users get the most out of the tools available.

The productivity score is a great baselining tool in the same vein as the Secure Score and Compliance Score. Giving a numerical value based on a wide set of statistics, along with suggestions for improvement.

Enable the Productivity Score

The first step to understand how Microsoft 365 tools are used in your organization is to enable the tool. This can be done from the Microsoft 365 Admin Center by navigating to ‘Reports’ -> ‘Productivity Score’. From here, we can enable the tool, this might take up to 24 hours to finish, then we can start using it.

Once the Productivity Score dashboard is enabled, it will show up in this section of the Admin Center.

Score Breakdown

The overall Productivity Score is broken down into two overall ‘buckets’. The total score is the sum of the ‘People’ score and the ‘Technology’ score. You can track your score against the benchmark set by similar organizations.

This provides an ‘at a glance’ metric of how the Microsoft 365 toolset is being used in our organization. Along with the current score, we also see how our score has changed over time, this gives us insight into the usage trend in the organization.

People Score

The ‘People’ Score helps us track how our users are leveraging the tools available to them across the various classifications (‘Communication’, ‘Meetings’, ‘Content Collaboration’, ‘Teamwork’ and ‘Mobility’).

We can dive into one of these classifications to see the metrics that make up our score in a particular area. In ‘Communication’ for example, we can see the different methods our users are using to communicate.

For each metric, we can view content such as videos and articles to help promote the particular method of communication. This gives us access to some prepackaged user training and communications which can help maximizes the utilization.

We can also see a per-user breakdown of active communication methods, allow us to identify areas of the business that need the most help.

Using the resources in the ‘People’ score, we can easily get a view of how the tools are being used, and by who, and guide our adoption and change management efforts efficiently. In the above examples we can see that our efforts should focus on promoting and providing materials around Yammer much more than Teams chat. We can also see a subset of our users are actively using ‘@ mentions’ however some users aren’t, despite active Teams chat usage.

Technology Score

In our ‘Technology’ score, we can see how the Technical aspects may be affecting our user productivity. In Technology, we see Endpoint Analytics from Microsoft Endpoint Manager (this needs to be enabled), The impact of Network Connectivity and the health of our Microsoft 365 Apps.

In the Microsoft 365 Apps health section for example, we can see the versions of the Office apps our users and connecting with and the associated update channels.

In this example, we can see quite number of our users have unsupported versions of the Microsoft 365 Apps suite and the update channel is configured to ‘Semi-Annual’. This will

As with the ‘User’ score, we can dive into some useful resources around how to manage and improve this baseline, which will in turn increase our total score.

Summary

As with the Secure Score and Compliance Score, I do recommend that these metrics are taken with a pinch of salt and context is considered before circulating the score. For instance if corporate structure or policy blocks a large group of our users from using Teams, our score there will always be lower. If licensing models or technical restrictions dictate we are using Office 2019 in our organization, that will obviously affect our score.

The Productivity Score is a great addition to the toolset, providing some quick insights and prepackaged metrics that are easily consumed at C-Level in our organization. Overall it is nice to have this information readily available but please don’t rely on any of the Microsoft ‘Scores’ to accurately depict the nuances of your organization.

PSA: The importance of disabling legacy authentication in Microsoft 365

This topic seems talked to death nowadays. Almost everyone has come across the strong recommendation to disable legacy authentication in their Microsoft 365/Azure AD tenancy. If you haven’t done this yet then the clock is ticking down to the day Microsoft disable the functionality automatically – October 2020 for new tenants at the time of writing. 2021 for existing.

Something that I find is not as widespread as the recommendation to disable, is the reason behind disabling it. Other than stating it is ‘unsecure’ and vulnerable to password spray and brute force type attacks, there are not a whole lot of real world examples of the issues with legacy auth for Microsoft 365 readily available. A lot of the high level messaging relates to Modern Authentication allowing us to use Conditional Access and MFA but not many examples of why.

To understand, let’s take a typical Azure AD Conditional Access Policy which enforces Muti-Factor Authentication on all Android Devices:

The above policy should enforce MFA on Android devices connecting to our Office 365 service. Now when we log in as this user from an Android device and app/protocol that support Modern Authentication, we can see Conditional Access applies and our user is prompted for MFA.

We can check the Policy Details to see the Conditional Access criteria too.

Great, our user is protected by our Conditional Access policy, we’re secure….right? We can sleep soundly knowing we have the extra protection needed…

Well, now lets try sign in with a third party app using POP3 and Legacy Authentication.

Looks like our user got in without issue and Conditional Access checked the sign-in attempt successfully, however when we look at the details of the policy applied, we see a problem…

Our Conditional Access policy didn’t apply! Looking at the Conditional Access Policy Details we can see that the Device Platform condition wasn’t met.

When we look at the sign in logs we see why.

We don’t have any of the additional details we would generally see with a sign-in using Modern Authentication. As this detail is missing (In this case, Conditional Access is unaware we are coming from an Android device), our Conditional Access policy doesn’t know it should apply and the user bypasses our MFA requirement.

This is just one of many real world examples of where Legacy Authentication creates gaps in our security policy. By now, hopefully almost every tenancy is disabling or working to disable Legacy Authentication. As it stands Microsoft will enforce this change in the very near future so there is still time for anyone who hasn’t taken this step to prepare for that date.

Hopefully this post helps to put the associated risk into real world context. For information on how to disable Legacy Authentication, check out the Microsoft Documentation.

Office 365 ATP Preset Security Policies

Office 365 ATP is a fantastic tool for protecting Office 365 users from threats such as spam, phishing and malicious content. A lot of the time, as consultants, the initial set up of these policies is pretty similar from customer to customer with some potential change on granular items such as thresholds and actions.

Often a customer will look for a “best practice” implementation so they can assess without going into each configuration item and making a decision based on past experience with other products or gut feeling. “Should we send suspected Phishing mails to junk or quarantine? What about high-confidence Phishing?”

There is some guidance available from Microsoft for the initial set up in the documentation and the Office 365 ATP Recommended Configuration Analyzer (ORCA) is great for assessing gaps from this perspective but still requires knowledge of the configuration and monitoring of any new features released or changes to recommendations.

The new Office 365 ATP Preset Security Policies option allows for a more ‘hands off’ approach of accepting best practices for ATP configuration. The policies allow for a selection of “Standard” and “Strict” Protection of users and can be assigned to separate user groups. This allows organizations who want to be protected, but aren’t too concerned with understanding the “nuts and bolts” of it all a nice option to deploy best practice protection to users rapidly.

While this is probably not a good option for a large enterprise looking to replace something like Mimecast, for SME’s the Presets should save time and effort in deploying some extremely powerful features to protect users.

The individual configuration items in these policies can be found in the Recommended settings for EOP and Office 365 ATP security documentation.