By Len Lipkin

Part 1 | Part 2

I argued in part 1 and 2 of this series that traditional metrics like number of clicks are ill-suited to measure usability of product enhancements.  Well, I lied a bit.  In fact, these metrics can be suitable as long as they are taken in the proper context, which very often they are not.

Roger Cauvin, on his blog and in the comments to part 1 of this post asserts that we measure usability along multiple parameters.  I am re-framing his thoughts a bit, but they are in the spirit of his original ideas.

• Inputs
  • Profile. A typical user’s prerequisite skills and experience.
• Metrics
  • Effort. A limit on the number of user gestures (keystrokes, mouse clicks, etc.) it should take to accomplish a functional task.
  • Duration. A limit on the amount of time that it will take a typical user to perform the functional task.
    • Engagement duration – the amount of time the user is actively attending to the task.
    • Total duration – the total amount of time it takes for the task to complete.

I think that this is a great way for PMs to craft usability requirements for a feature set.  Cauvin rightly contextualizes the metrics themselves, adding user profile (a.k.a., persona) to the mix.  He also wisely expands the “time to complete task” metric.  I offer kudos to Cauvin for these great ideas.

My problem with the metrics that I disparaged in parts 1 and 2 is that they are somewhat arbitrary by themselves taken out of context.  Limits on duration or number of mouse clicks or gestures are great, but you are likely to be challenged by developers upstream, or users downstream.  You certainly wouldn’t want the feature to pass the metrics you establish and then have customers complain about a feature being hard to use, right?  To address this, additional inputs could be defined to help clarify the metrics in question, and at least one additional metric should be considered.

My solution to this would be to extend Cauvin’s profile input (skills + experience) and add additional inputs or attributes to the profile vis-à-vis the feature you are requesting.  Some possible additional inputs I would imagine are:

• Feature Importance to Role-(Critical, High, Low)
• Role Priority–If defining metrics for multiple roles, which role is most critical to the success of this feature?  Theoretically, there could be multiple parallel usability metrics on the same feature for different roles.
• Frequency of Use: Several times a day, daily, weekly, monthly, annual, etc.

The additional inputs will help you determine appropriate targets metrics for feature that you propose.  The target metrics for a feature should be determined by who uses the feature and how they use it.  Less important?  Less frequently used?  Profile is a strong user?  No need to reduce the number of clicks as compared to a very important feature that is used frequently by a less experienced user.

I don’t consider myself to be a guru on this subject, even after doing lots of research for this series of posts.  I am not convinced that I have the right inputs or the right metrics, but I am convinced that metrics taken in the absence of these empirical inputs lead to bad UI.  UI is not just number of clicks.  My question to you all—what other inputs and metrics do you see as having value?

By Len Lipkin

In part 1, I discussed some ill-conceived metrics for UI, specifically number of clicks and time to complete task.  I know that I promised some legitimate metrics in part 2, but that will have to wait for part 3 so that I can respond to some of the comments from part 1.

Some pointed out that you should be tracking things like customer satisfaction.  I agree completely, but it’s not exactly what I was getting at.  Customer satisfaction is not a UI-specific metric.  It is hard to discern the influence of UI versus customer support versus product stability or any other number of factors when using that metric.  You could create a customer satisfaction measure that includes a special usability component, but I would argue that that is not what I was getting at either, for two reasons.  One, you don’t necessarily have good data on the individual customers who complete your customer satisfaction survey as far as their experience level or which specific features that they find usable or un-usable.  You could create a UI customer satisfaction questionnaire or something and gather information about who is responding and ask them about every feature in your system, but that runs the risk of turning into a 45-minute exercise that will at best, yield unreliable results, and at worst anger your customers.

Along the same lines, “number of support calls for the feature” has the same problem, though possibly not as severely.  It is hard to completely tie support calls directly to usability.  If your support team can classify calls (explanatory, bug, or customer-specific) this metric could provide a lot of value.  Still, this measure is only useful after a feature is released into the wild and doesn’t account for the profile of the user reporting the problem very well.

So basically these are valuable metrics but not for my needs.  What I am exploring is the possibility of a feature-by-feature measurement that can help a PM assess a development team or UI Designers, not measuring the product overall.  While there is great value in measuring the product overall, it is not the focus of my thesis here.

Product Management blogger (and coincidental commenter on part 1) Roger Cauvin proposed some good UI metrics that are where I was headed.  I will discuss and refine his ideas in part 3.

By Len Lipkin

I see (and have participated in) a lot of talk on Twitter among Product Managers about the responsibility of Product Management for UI design.  Nobody seems to advocate that Product Managers should directly design user interfaces, but being product managers, we obviously want some degree of control.  After all, the success of the product depends in part on UI, whether it be in snazzy product demos for buyers or ease of use for users.

“Make the feature usable” is not a valid requirement because it is inherently subjective.  What is usable for a developer is only usable to the customer if your customer happens to be a developer.  The only real way for a Product Manager to define UI requirements is to define metrics for successful UI.  Unfortunately, metrics are problematic because many of the metrics that are used are poorly conceived.  In particular the most commonly used “number of clicks” metric is insane.  It’s great when something is “one click away”—it’s fun to say and fun to demo, but does a customer really want or need it?  Consider what would happen if your product took “one click away” to its inevitable conclusion—an unusable product.

So what are some alternative metrics?  A common one I have heard is “time to complete task.”  This one makes a bit more sense in that it incorporates more than clicks to determine how “usable” something is.  Unfortunately, it is hard to measure, and the results will likely vary widely depending on whether you are measuring novice or experienced users.

So what are some legitimate metics?  Read part 2 (and then part 3).