Quantifying Qualitative Analysis by Leisa Reichelt


At #mtpcon San Francisco, Leisa Reichelt, head of analysis and insights at Atlassian, took on our present strategy to person analysis and the way we take an evidence-based strategy to doing utterly the flawed factor.

5 years in the past, the business was centered on simply getting individuals to do any person analysis. Product managers weren’t often speaking to prospects in any respect. Lately, the drive for firms to be “buyer obsessed” has led to a brand new period the place everybody is speaking to prospects. Designers, product managers, and researchers all assume that doing utilizing analysis is a main operate of their job. And whereas analysis is greatest when accomplished as a group, how a lot time are we spending to develop our experience in analysis in order that we do it effectively?

Whereas we needs to be on the market doing analysis, we’re letting dangerous analysis move for proof, and we make selections on knowledge we shouldn’t be utilizing for this goal. We mess up analysis by attempting to quantify issues and make them appear extra scientific than they are surely.

Perspective and Reality

Taking a look at solely elements of an image can provide a warped perspective, and we will see issues that seem significant however which aren’t. Now we have to verify we’re getting a sufficiently big image in analysis to make an excellent choice. For those who see the shadow of a duck, it could possibly look very totally different from the duck itself. This occurs in analysis as effectively. We’re getting a slender view of knowledge and are making selections primarily based on the shadow. The shadow is actual, however the duck is extra actual, and now we have to broaden our context to get the total fact.

Frequent Analysis Errors

Buyer Interviews

Individuals could make up an opinion about something, they usually’ll accomplish that if requested. Jakob Nielsen

Buyer interviews are a typical technique of person analysis that groups mess up. The solutions you get in interviews rely on the questions you ask, and by asking the flawed questions, you may get actually enthusiastic about knowledge that doesn’t matter in any respect. We ask feature-based questions resembling “What’s the factor you do most frequently in Jira?” and get slender solutions. We needs to be asking questions like, “What are you doing most frequently while you come to make use of Jira?” With broad context, you be taught concerning the “why”, hear gaps and alternatives that the product could also be lacking, and may finally dig deeper to probe into the options themselves.
Leisa Reichelt speaks at mtpcon
To do higher buyer interviews:

  • Begin with huge context: don’t begin with questions on your product
  • Be user-centered, not product-centered: you need to have the ability to use the identical script for a competitor’s product
  • Spend money on evaluation: get transcripts of the interviews, take the time to do affinity diagrams, and actually get the total worth from the analysis

Usability Testing

We make selections primarily based on usability testing on a regular basis. A well-liked technique that has emerged is “Unmoderated Distant Usability Testing.” With such a testing, you utilize software program that lets you construct assessments, put up them, and let individuals full usability assessments from anyplace on their very own. It lets you get a variety of testing accomplished with out ever speaking to a person instantly, but it surely comes with a variety of pitfalls.
Leisa Reichelt talks about unmoderated usability testing at mtpcon
With in-person usability assessments, an skilled researcher is ready to discern the distinction in what a person says and what they do. They’re capable of see physique language that signifies when a person is annoyed, or see when a person solely accomplished a job by randomly clicking round a display. However with unmoderated usability testing, groups usually merely obtain knowledge slightly than really watch the movies of the assessments; this stuff get changed into a easy metric of “job accomplished”, and quotes and scores are pulled with out the broader context.

To do higher usability testing:

  • Make assessments job not feature-based: take a look at what individuals would actually be doing, not the function you want they’d use
  • Watch the movies in actual time: don’t simply obtain the information from unmoderated usability testing
  • Look past job completion: it’s not OK that individuals finally get to the appropriate consequence – the standard of the expertise alongside the way in which issues


Religion in knowledge grows in relation to your to your distance from the gathering of it. Scott Berkun

Product managers love surveys. The additional away you’re from the purpose of assortment of knowledge, the extra you belief it. However deceptive questions and an absence of context can present knowledge that tells a really totally different story from the reality. Leisa gave an instance from Atlassian Summit, the place a person mentioned in a dot-voting survey that they’d nice work-life stability. When she requested them about their reply, they mentioned “I can do business from home, so although I work an excessive amount of, I’ve flexibility.” The context offers a unique perspective from the information level. Simply because you may put it in a graph, doesn’t make it science. We have to be far more vital about what knowledge we belief.
Leisa Reichelt shows a bad user survey at mtpcon
To do higher surveys:

  • Use qualitative methods earlier than quantitative: do interviews or contextual enquiry earlier than you design a survey
  • Triple take a look at your survey: ensure individuals perceive questions persistently and are capable of reply precisely
  • Use people to analyse phrases: don’t depend on automated textual content evaluation. Learn the suggestions.

Cease Quantifying Qualitative Analysis

Why can we take these analysis strategies and use them incorrectly? Leisa thinks the principle motive is that we don’t have time. But when we declare to be “buyer obsessed”, why are we attempting to field analysis and empathy into the smallest period of time doable? Take the time to do it proper.

Leisa leaves us with the metric we should always really be aiming for. Slightly than finishing dangerous person analysis, simply ask: “Within the final six weeks, have you ever personally noticed a buyer, utilizing your product?” If we’re going to make higher merchandise, the reality of our prospects’ expertise is what we’d like. And fact can’t be automated or graphed. It takes time, and it takes people.

The put up Quantifying Qualitative Analysis by Leisa Reichelt appeared first on Thoughts the Product.


Supply hyperlink


Please enter your comment!
Please enter your name here