[Music] we ready we're ready man all right let's do it hey everybody and welcome to product happy hour where you can go to happy hour with your favorite product people in your sweatpants we are product folks here to share what we've learned often the hard way over great drinks why happy hour why not it's the best way to get the inside scoop from grizzle vets with the scars to prove it thanks for giving us a listen the best ways you can help us keep this party going is to donate at producthappyhour.com support that's so mouthful and subscribe on youtube spotify apple podcasts or wherever you get your podcasts with me as always this ira joe hall hey era hi yep at phh we share a zero bs zero fluff approach um to learning how to become a seasoned pm basically we're sharing the war stories you only hear at happy hour with your industry friends and i've got my drink i've got my product friend there he is and i'm ready to roll firefighter all right today we're covering qualitative research quantitative research and the relationship between the two trippy episode this week y'all lots of counter-intuitive stuff in here but first day in the life of a product manager era how was your day in the famous words of ice cube today was a good day we're recording on a weekend and this weekend i turned off my slack notifications and dude i even turned off my text notifications wow and yeah i spent the better part of the weekend and today like hanging out with close friends and family doing some birthday things very nice happy birthday thank you very much we uh cruised around on a speedboat i live in san francisco so we took it out into the bay and we stopped at a little marina diner for those of you bay area natives you might have heard of sam's but they have they have this cool thing called sales side service where you can like roll up in your boat i guess swim up paddle up in your boat that's probably where appropriate yeah but yeah it was cool they'll like feed you food on the boat and yeah this was so good if this weekend was like a product it's getting five stars on the app store dude nice are you were you like a roman emperor like a senator just sitting there they just like you lay down and they just kind of like drop food in your mouth so that pretty works that's how it's imagined in my mind like a big vegan burger just like can you [Laughter] okay yeah yeah just a little bit yeah but uh naturally all weekends must come to an end and it's time to mull over product learnings with a cocktail couldn't agree more perfect segue into our next segment what's that drink era what are we drinking today um i brought this bottle but i swear i'm not drinking right out of it yeah sure that was yesterday [Laughter] yeah so okay today i'm because it's like 12 o'clock i'm drinking a mimosa because it's like prime brunch time on the west coast and isn't it funny how like oj and champagne is like totally acceptable alcoholic morning beverages but if i showed up to this podcast with a fifth a fireball you'd be like what the hell are you doing just fireball the breakfast of champions it does seem a touch more aggressive than shipping you're right okay what bevy do you have in that phh pint class all right today i have a dallas blonde made by deep ellum brewing in my hometown of dallas texas this sucker has got a little bit of kick which is really great oh yeah and i've got it here yeah it's awesome i've got it here in my product happy hour beer mug all right like happy hour we've got our great drinks let's dive in to cool product things where we talk about a cool product thing that represents a key product concept era it's your turn this week what do you have for us okay i'm so excited to tell you about this my cool product thing is tonal prs so a little bit of background my germ aversion which i am so ocd but yeah i hit like all-time highs during the pandemic as you might imagine so i made an investment in um an at-home weight lifting system called tonal just to keep my body from turning into like a squishy blob while the gyms were closed audrey always sees like in the background of my zoom but um yeah it's not a tv it's actually digital weights and it's kind of new to the gym market it's like less than five years old and if you're having trouble envisioning it it's basically a compact cable machine like you might see at the gym with a flat screen attached to it and you can basically do any lift like i'm talking deadlifts bicep curls chest flies on this thing and it'll digitally record your exercise and the weight and the volume of your sets isn't that cool that's awesome i mean so cool that's great yeah that's i want it i know it sounds good in theory but it's also really cool in practice because the next time you attempt the same lift it bumps up your weight for you and if you're able to complete the set at that higher weight it rewards you and that's what a tonal pr is or tonal personal record it um makes a cute little sound i would uh intim i would do it right now but it's probably not going to be good and we're not going to get sponsored by tony you can try i'll just oh you could try it i'll just edit it out it's like boo boo i'm keeping it i'm definitely keeping that i don't know why i made my eyes like that for those watching the video [Laughter] okay it gives you a visual on the flat screen it's basically like a full full screen notification that lets you know you hit a personal record for that lift so why is this a cool product thing well basically eliminated like nine million steps of friction the first being i don't have to remember what my the last like weight was and set or rep style that i did or volume for that exercise and second it has intelligence required to know like what the next weight up should be for me like i am mostly squishy blob so it's not like 20 pounds higher you know for miss joe hall it's not doing that because you know i mean i don't know about you maybe i tend to like i'm ten i tend to like underestimate how much weight i can actually lift so this actually like digitally kind of like encourages me to start you know getting more and more weight under my belt and it has all my exercises and all my weight and all my volume stored so essentially it's a hundred percent personalizing the pace and weight that i aim for next isn't it just i'm like okay okay like i'm never going like what's better than this what more could i ask for oh i could ask for a hard body and not have to do any of this work but we're not there yet okay one last working on it yeah so last but not least the tonal personal record alert is positive reinforcement it quietly and seamlessly ups your weight for you and if you perform it you know like i said it provides the tone but also like now my brain is like wired to celebrate when i hear that noise and so is my fiance like we're both working from home right so we're both using this expensive behemoth we bought and sometimes i'll hear him working out and if i'm like within earshot and i hear the pr tone i'm like oh my god good job babe yeah so it's just like dripping with like good habit loop design right yeah i really love how it leans into its strengths right it's a digital system and it's taking full advantage of that aspect with data-driven personalized features knowing you hit a personal record is definitely going to make the product sticky and make it hard to go back to lifting analog weights like why would you do that it does all this stuff for you it actually reminds me of um peloton implemented this thing where you can lock your resistance and it'll automatically change your resistance for you oh my god so cool like why would i ever go back to a normal spin bike so i totally get it this sounds that is amazing yeah yeah i'm like my squat rack at my like dirty smelly you know slightly sweat glazed gym equipment is never gonna like give me a celebration when i hit a pr it's gonna i'm never going back probably not and if it did you'd freak out you'd be like what is that well i'm just gonna like try to punch it and break my hand definitely true all right well we've got quite the show for everybody today um just a quick breakdown of today's show pms need a quan a combination of both quantitative data and qualitative data to make healthy bets on how to shape their product that's pretty common knowledge but what we're going to cover is how to avoid making rookie mistakes when interpreting these inputs by examining the relationship between quant and qual or quantitative or qualitative first we'll define the concepts of quantum qual with real life examples from our experience second we share how to avoid common pitfalls by teaching you how to respect the relationship between quant and qual and last but certainly not least we will give you an insightful cheat sheet aptly named what could have saved us pain if we knew this earlier in our careers this one's worth seeing tuned for it has some solid takeaways that will undoubtedly save you from wasting dev cycles because you'll have had the benefit of learning from your product friends that's us here at happy hour let's do it let's do this so good let's go let's go let's go boom [Music] and the fact that to know that they're over there thinking like how can we make this experience even better knowing that it's a digital product in your house it's just like that just that's you know in a lot of ways that's just like great marketing it really just kind of makes you want to go out and buy one that's why i'm like maybe we should get tonal to sponsor some of this stuff so we can get some free shipping i mean hit us up because tonal is like such a game changer and we talk about game changing products here so if you guys want to be continued plugged you'll keep you'll send us a discount code at least that's right and we can do something like crazy thing be like hey if you're a product manager and you want to be at your best at your game like you know buy total get a total watch like 10 episodes from now i'm totally gonna be reading that ad okay art imitates live people i i gotta tell you like it's so they have pms at tonal and they are so fire because the software updates the features just get better and not worse yeah like now there's like digital trainers and more programs that's yeah okay and they tell you what muscle group is fatigued based on your last workout they're like don't try your quads again you tore those up last time let's do arms that's fine yeah man i know all right cool i'll be spending my whole week in miami on on the tonal website thanks you're welcome okay well let's go in and talk a little bit about these concepts um so what is quantitative data well let me tell you quantitative data is basically numbers about your product that includes like dashboards that show metrics data from queries you might run basically anything numeric or data related that represents user behavior key business metrics or trends like one of the things i define i think about as quant is um basically anything that you know you see no user face you hear no user voice but it's just the calculated effect of a group of people doing things on your site so yeah should i give you some examples let's do it i'll give some examples okay here's some real life examples of quantitative data from my experience in the e-commerce space and just to note about this like maybe you're like a starting out or you're at a smaller product shop and you hear this list and you're like well i don't have that data available don't worry because pms are actually also responsible for driving research roadmaps so if you think that the data that you hear about here is valuable but it's not available then you know your task at hand is simply to you know ask your closest research partner or data scientist if it's something that could be collected and naturally you know we always like want more than the data we actually have so it's not like it's not rare that it's not available and also most of the data isn't really ready to interpret so this can still be useful in helping you kind of tee up the collection of this data and knowing what to ask for before is just as useful as learning how to interpret the data so this will probably still save you time if it's not there right aj yeah incredible skill that's actually one of the things to learn is is really you know what to ask for is um so yeah that's that's such a great point that's that's awesome yeah okay cool so in ecommerce our our key metric usually involves sales of x it could be anything typical quant data that i've sourced in my last couple of rolls it really has to do with sales by demographic so it doesn't matter if you're sending selling like dog leashes or water bottles it doesn't matter this is the very bottom of the sales funnel and this is pretty typical data that you'll look at when you're trying to evaluate that product's performance so here's a happy hour version of this advice demographic isn't simply revenue by country or mobile versus desktop sales you actually need to look a little deeper to unlock insightful quant so for example one of my search roles we plotted sales over different slices of time and different demographics and we got some game changing insights we found that like four to nine pm is peak media consumption for the over 18 demographic in english-speaking countries i know that sounds very specific and you might need a lot of information to come up with that insight but if you have it i mean knowing the peaks and pits of sales cycles and the demographic that maps to it can expose super cool products of opportunity like this information totally changed how i targeted and ran my a b test like i i might examine the weekend data separately or only bucket users for my feature who like fit this like what i call opportunity demographic aj do you have any other interesting examples of quant like that yeah totally just building on this one um one example that comes to mind is a different marketplace i worked on well for context there was two different marketplaces um okay and in one of them the sales cycle um in total would take weeks you know people would evaluate the different options they had over a week or weeks long time frame uh and often they were doing it with other people you know you would get together you talk about the products that are available and discuss them and look at the different features and stuff and so it would happen over a week's long time frame but then this other marketplace people were making decisions like same day on the spot it was much more of like an impulse purchase they would sign up and then you know they weren't really discussing or researching that much most people like 80 90 plus people were making decisions same day um and so you can imagine if you're in a marketplace where people are making decisions over weeks long that's a very different experience that you probably want to have than for people that are trying to make a decision like right away um yeah it's like a fundamentally different type of experience and for that snap decision making you probably want to make sure data is available immediately you know you probably don't want to be um messing around screwing around and and and not showing people stuff that they need access to right away so so that's kind of one quantitative example uh that i think of um and another one is uh you know sales and sales cycles is a good one another quantitative example that we talk about a lot is engagement data collected from site and device behavior so clicks views revisits these are top of the sales funnel and can be really powerful but a game changing quant analysis i did was to look at engagement and conversion drivers what behaviors map to successful conversion essentially oh my gosh so good and so instead of just like looking at the raw behavior counts you're like looking at like patterns of behavior that result in sales did i get that right that's right exactly so the background here was that we did a study where we looked at our engagement and conversion data through a few specific lenses frequency of use conversion rate at those frequency rates and volume at those rates and through that analysis we found some really interesting behaviors for example users when they are deeper in the conversion funnel on things like cart or checkout are surprised ready to check out and you have a few shots in making that happen um so you want to focus on on getting people to essentially purchase uh in that part of the experience set instead of getting them out to something else um that's that's a really critical part does that make sense here oh yeah totally so okay this sounds um i don't say complex but it sounds like there's probably a few different like steps needed to like extract that level of information do you think you need like a data scientist to be able to do this or could you just like do this without a math degree if you just had more time yeah that's a great question i think you can get to high level insights on your own if you have sql skills and your site is instrumented but there are also some commercially available tools that you can use to interpret uh quantitative data i really like uh mixed mammal and amplitude have you heard of this oh my gosh those are way more self-serve than like issuing like a sql statement on like some massive data lake um i remember like when i started at um a new role it's really hard to like get all the permissions get all the tools figure out what tables have what you know like historically any company i've worked on my career like documentation is fair but is it up to date no is there a how-to on it no like on your own so like i get a lot of value out of these tools where they have like you know companies have completely outsourced the representation and the access to tables and done the instrumentation of stringing up like events uh digital events that are happening on your digital experience to like interpretation tools so yeah i really like that you can take like clicks and views and dwell time which is really important for search and funnel that into a really cool ui yeah exactly i mean you you know data science is extremely extremely useful and this is actually a good topic for the podcast for very complicated analysis like what i was talking about earlier but you know things like mix panel and amplitude are really great self-service products uh where you can basically search and investigate data slices of user behavior on your site or app to get a quantitative view for x segment time period uh or cohort um right on your own you know and it really saves you some time while your data scientists can do a lot of the really really more complicated stuff right and like the more bespoke stuff like i swear we're not sponsored by mixpanel or amplitude or tonal at this point guys i swear for real they're just really legit product experiences they're so good but audrey can you take us into examples of cool and interesting qualitative data yeah for sure so uh before we do that let's define qualitative data okay qualitative data yeah yeah i'm sorry very important didn't mean talk over you that's hot that's how important it is i was like let's define it okay so qualitative data is basically what users actually tell you like what hero is saying with their face and actual audio oftentimes uh usually in user research settings this can also happen in one-on-one consumer conversations or direct written user feedback it's user contributed data that is more descriptive of the qualities or characteristics of whatever you're studying it might look like a user study where you are asking for example we don't work at uber but for example uber drivers how their experience was with navigation software it could be asking a vacation home renter to rate and describe the accuracy of amenities things like that yeah yeah totally it's like you know the the stuff that you like would hear if you asked a friend like oh what did you think about that they were like oh well that experience sucked um it's that kind of data so it's more the voice of the customer as opposed to the actions that they took which are like facts right it's their opinion about it um yeah 100 uh so some useful quantitative data that i've gathered is um listening to interviews of both sides of a marketplace i was working on both the the consumers that are coming to to buy things on our marketplace and the suppliers or sellers of those things on the other side of the marketplace you could really hear the frustrations that our suppliers and our consumers were having with various things like the stagnation of features on the mark on the marketplace this can help us inform a decision to instrument various parts of the consumer experience with click and view events so we could learn more about the behaviors related to the negative or positive sentiment shared in the survey or instrumenting supply side features for suppliers to better understand where they're getting hung up in the process of getting their inventory onto the platform and and really trying to understand that better so you can so you can better marry the two which we're going to talk about a little later yeah yeah totally and because you can't like call up user bob and be like bobby tell me what was it like yeah um there's there's definitely some tools in place to actually allow you to gather this qualitative data um i've used a lot of bad ones but why don't you tell us if you've used any good ones yeah and and i will say before we hop into that uh the the tooling and stuff one of the things that is interesting about qualitative is you have to do it very well you know you want to make sure and we're going to talk about this a little bit more later um but you know you can talk to bob but you have to know how to you have to learn how to talk to bob you know you can't oh so you can't just be like bobby bobby you know there's a practice to that there's there's a science to that um and doing that well um and what we're gonna talk about here in a bit but um you know you can do that but there are lots of other different types of qualitative data tools that help control for those types of things so naturally yeah there are lots of different mechanics to collect the different types but my current favorite is full story you can actually watch your user's digital experience and it classifies things like range clicks i found uh super super useful uh it increasingly uh it interestingly gives you um the quant on the qual you know you can you can really watch and extract the voice of the enraged customer um and it's it's motivating because you can really see the frustration which is um which is really great but there's lots of ways of getting it full story is awesome i've totally like mouse clicked i also hear my partner like slam his mouse when his training software isn't working it's cool like full story you can like you can literally see the session so cool yeah also a little creepy because we're like watching your mouse movements guys but i think we force you to sign off on that before we do it yeah and it's really uh you know being able to see a lot of those nuances is so helpful i mean we're not doing it just you know i think product managers around the world hopefully they're not like oh my god you're telling everybody your secrets it's not really a secret but you know uh it's not intended to spy on everybody or anything like that it's really just to understand like what everybody's doing within the context of of using your product you know people when they're using their products are are in system one shout out to the first pod uh it's worth it that's right yeah and most people when they're in system one are doing things out of habit or are not completely half uh all the way thinking about what they're doing so really watching them in their environment is really really helpful so it's not we're not trying to be creepy it's all anonymized um yeah we're trying to make it better like imagine if the dmv watched you annoyingly go to six different counters before you could get your license reviewed like they would probably design a better dmv like that's what pms are doing when they're watching they're they're trying to make it better um and hopefully hopefully not worse worse but okay yeah so would you say aj that uh quant and qual are equally as important or is one better than the other what should like a new pm focus on yeah totally uh i think they're both important to use together and the relationship between them is super interesting let's get into that shall we okay let's go all right so my hope for this segment is that every pm and future pm walks away able to use quant and qual together effectively does that sound like a good goal uh that is a very good goal and this is um my third mimosa of this episode quantitatively i may still be sober enough to finish this podcast but maybe we'll get the whole story once the qual is available in the comments i don't know i don't know maybe maybe we'll see yeah leave us a comment on youtube or rating on spotify if you thought uh this part of the episode was particularly good but the episode overall yeah let us know yeah let us know okay so um here's my first takeaway when you perform a qualitative user study you need to know what cohort uh the people that you're talking to are in and to really understand how to apply the insight you know demographics contacts cohort gen z versus millennial you know you really want to know um what these cohorts are because they can really kind of shape you know the uh the information and what you should extract out of the information and i'll use uh a really generic example uh so bon boss is a socks website um which i bought stuff on bone boss of you no i i'm wearing them right now i would show you but i am not that flexible yet well if i was wearing socks i uh i would i would show you i would try to show you mine and probably hurt myself um but you know i think what what um so like let's say you know we're designing bobas and and you're talking to um you know different users in your user study maybe there's uh just for the sake of example let's let's say we talked to four users and one user was like um a millennial buying socks for his toddler that's probably me um and let's say another one is um like a millennial mom buying socks for a soccer team you know they have socks for for the soccer team in their seat in the in those colors maybe you have somebody that's more like a senior citizen looking for comfortable socks to help them make sure they don't fall um and maybe it's like um an eight-year-old kid browsing the site and then eventually their parents are gonna purchase them something for example um okay super different needs super different users super different needs super different users super different user feedback you can imagine you're probably going to get a lot different feedback from an eight-year-old uh than you would from a 70 year old i know it sounds like a little bit not a common example but it could happen you know um no it those are like there are products that have that age range or at least the journey starts with an eight-year-old and ends with an 80-year-old purchase like that is very common right yeah for sure so you know you might get like like you know spoken feedback from them um that's that's very different um so somebody in their 70s probably wants a lot of accessibility features to make the site easier to use and and they're probably really concerned about very specific features around accessibility versus the eight-year-old is probably more interested in like the sesame street socks which they have i have some um that my son and i share uh and you know is probably interested in like oh this is so cool they have that rather than you know things like accessibility features so you want to marry a lot of what you hear with a lot of the data that you see and where people are struggling so you know you might see that different age core cohort is struggling with a certain part of the site that has accessibility problems but the eight-year-old is struggling more with the top of funnel experience because they're not finding the the brand that they're they're interested in in particular okay um so you know that's why it's really important to understand like who you're talking to which cohorts they fall into and how to apply that does that make sense yeah i mean like you don't want to take the eight-year-old's feedback and change a whole site based on you know uh you know their input because that's gonna marginalize some of your other users i think like if i'm honest it's really hard to figure out who your users are from just looking at quant or just looking at qual so i really like that you highlighted that you need a little bit of both and conversely you know not to make the opposite argument argument but more or less a complementary argument many of the blind spots of quantitative data like the numbers are addressed by qualitative data the the customer's voice so basically you should like to like continue on with our alcohol analogy you should have like the equivalent of beer goggles or mimosa goggles depending on what you're drinking today yeah uh when you look at god um don't do that for other situations guys just just this yeah well more clearly what i mean is don't take quant at face value instead get a little more color from the customer voice before you make a time intensive or cost intensive product decision like there is no team that wants to be dragged down based on your like little tiny sliver of research into like a multi-sprint project right like oh jay i know you've experienced this ah two too many yeah i'm halfway down the beer yeah it's too many times okay so here here's a real life example where you can like learn from my pain okay we were trying to understand how to improve the search experience for rare sneakers so we looked at the health of a query like nike air max on this big ecommerce site that shall not be named uh we had some uh supply but um the top results weren't getting clicked okay like what the hell based on the quant the top five results had like a really low click rate and if you've worked in search or know about search like that's a negative signal okay we spent all this money writing a ranking model and no one is clicking on the on the top it's like not good um so you know what we do is when we have a negative conversion signal we retrain our model or we go and investigate new features these can take sprints man like this is like not like it's not simple so yeah we we could have done this but um yeah it would have taken us down the wrong path because we had a feedback survey on the sneaker listing page so this is double-sided marketplace where individuals can list sneakers and you know i really don't pay attention to this survey too much because it's for literally every product and it's kind of generic and i don't really look at it every day it's just too much like it'd be like reading a novel on everything from like you know q-tips to like ford trucks those are like those are categories on this website so um yeah i ended up wondering like why aren't we're doing this big marketing campaign for rare sneakers like the yeezys and all the new nikes and all the retro jordans like why are people not clicking on the on these results and i looked at the survey because i don't know i i had some sense back then yeah and our sellers were filling out the listing feedback survey and there was a comment in there about how the latest nike air max wasn't actually even available yet like meaning nike had not even released it and what we were showing on our site was actually a bunch of counterfeits so the sellers in this category i know i was like mind blown emoji where is that reaction yeah the sellers were like super eager to have search down rank these counterfeits which was uh the exact opposite of what i was about to do yeah i was going to like try and figure out how to get them more visibility because they didn't want counterfeits in their category because when they try and sell the real thing at the right price point um they wouldn't be able to compete so re-ranking a bunch of counterfeits fits would actually you know really like isn't going to help the sale of uh like this really important sneaker to sneaker heads the seller the buyers who were coming they were like oh these are counterfeits that's why i'm not going to buy them but the quant data is like oh these are really popular sneakers they're showing in position one we're getting a ton of views and no clicks like but nowhere in the quant is it gonna say that these are counterfeits and not attractive to buy right so super interesting super interesting i it's probably such a gut punch to hear that but also probably shared it probably saved future era a lot of pain oh my gosh and you would have been like why is this happening and just would have made a bad situation worse um yeah and i learned what trust and safety is i was like oh i gotta report these listings so they can get taken down so the algorithm can actually match people to stuff that's good you know yeah i and sometimes when you listen to quantitative sometimes they validate something that uh that can apply to everybody and do very well with everyone like the counterfeit example nobody likes kind of like who wants to buy counterfeits right no um right that's a that's a really great example that's of something that would hit all demographics something that i've seen is is um uh you know a case where you're being able to take a lot of the content that people people want and bringing that forward and pulling that forward that's always you know something that does really well with lots and lots of different demographics and i've seen examples where you'd run the quant 10 out of 10 users say like this thing is awesome and then you run the uh i'm sorry you run the qual and 10 out of 10 say that this is awesome but then you run the quant a b test and it just blows out the water now you have a good sense of like why these things work yes um oh when they line up yeah oh so satisfying so satisfying and it is so good it only it only happens just so everybody knows some it happens sometimes because thanks to system one and system two again shouting out to episode one um what people say and do in research studies versus what they do on the site is is very different often um so yeah having these two things dimensions and learning how to marry them together is really really important for that reason uh yes totally super super interesting stuff okay let's share that insightful cheat sheet on how listeners can save themselves from misinterpreting the qual and the quant sound good oh yes yes i wish i had this earlier in my career so all upms out there you're welcome okay cool all right let's just bare let's just barrel right into it okay number one question framing matters i talked about this a little earlier um but you really want to understand the questions you're asking customers um usually you know uh you might have to do this yourself if you're like a smaller shop um or maybe you're starting on a product that's new and you're not getting uh tons of resourcing just yet so you might have to do this yourself but oftentimes you know you'll be working with if you're in a bigger shop you're you're working with a user researcher um or somebody in in in product marketing that can kind of help you craft these surveys and so you want to make sure that you're um not baking in assumptions into your qualitative survey so you want to always pick apart your your questions and make sure that the your assumptions are not leaking out uh when you're when you're talking to users uh you also wonder so guilty of this i do i you know i i think i might have said this in an earlier pod i've definitely done a lot of this i actually you know blasphemy on my part i used to uh just be like i don't really get why we have to craft these survey questions and like really be diligent about it now i know because this stuff yeah we learned yeah so uh and i did a lot of that earlier on in my career so everybody that's listening here that's uh that's earlier on don't do that you know trying to get some help uh do some research you know talk to talk to some great folks in your company that can help you with that same with bias you know what bias are you introducing in your questions uh you know you can really easily introduce bias in your questions so so make sure you're working with somebody even somebody that can just hear you out another pm and and uh call you on some of this stuff to make sure you change it um and uh trying to get research support that could help you construct a survey that gets at the info you need to make a product decision like you know often if you've got a great user research team use them it's worth taking the time uh to work with them and be more diligent about that it's totally 100 worth it you don't want to run with the wrong wrong insights you know i mean yeah yeah i've totally done the sentence construction and baked in an a bias like there's one that's common the positivity bias where i'm like how good do you think this page is scale of 1 to 100 billion [Laughter] yeah you shouldn't write those alone um and also if you do not have a research team there are tons of resources on the internet um that can help you remove bias just from your conversation or interview style like you can apply that if you are forced to construct a survey on your own and this is like really like as i said in your best interest um the cleaner the data you get the more interpretable it is and if you've interpreted well because you've constructed it well you can actually action on those and not have to run a full dev cycle based on kind of crappy questions that then gave you crappy answers and they gave you a crappy product yes i gotta fix it a little higher in the funnel yeah okay so uh here's number two okay literal interpretation of the summary isn't good enough you learn isn't that a good one you learn more when you watch and observe people in the wild so yeah this is this is not to aj because he uses full story and um i do now but it's you you're actually looking at the mouse and tap behavior um real life in the wild versus one of these constructed prototypes where you set up like a fake version of your site and then you ask questions and then they do the behavior as they you ask the questions you don't actually see natural behavior when you do that and that will just screw you out of learning how people really interact with your feature so like you get to an a b test and you should instrument it not just with metrics but also with tools like full story like ajay was talking about because i always prefer this option when it when it's possible over the prototyping questionnaire because you get a combo of actual a b test quant with a qual um and there's no weird like paid participants who are using your new feature and then are asking questions like i always find them to be overly positive i don't know it's because they're getting paid aj like have you found that i have and they don't often times it's not even just that they're getting paid i i think certainly that that has a a role to play in it but also they just you know users want to make a good impression you know like they and you're they're talking to often a company that they use and and they like so you know they don't want to that's a good point you know they don't want to just come out and be like hey you guys suck you know sometimes they do yeah um sometimes it definitely happens um and that's how you build a thick skin but then sometimes it's just like they just uh don't want you to feel bad um right but then that's where things like full story or things like the quantitative data really helps keep a lot of that honest for sure that's that's why we need both right okay that's so good so yeah those can be riddled with bias yeah totally so good all right so number three don't underestimate the power of cohorts we talked about it earlier uh what is a cohort a cohort is basically just a group of people um and it's the groupings of people cohorts are just groupings of people um and this can happen on different dimensions it can happen on device type it can happen on age it can happen on country um and the reason this is important is because often like we talked about uh sorry i burped in the mic uh that's the beer so we're drinking that we're at that part of the happy hour okay like this is just what happens [Laughter] [Music] okay so um okay why is this important so oftentimes when and this is important in both contexts like we were talking about earlier qualitatively you're going to hear things from users in different cohorts that apply differently in different cohorts um so you know different ages different countries something you might hear in japan or germany or france is going to be might be unique to that country relative to other countries but okay you also are going to see this in quantitative data oftentimes what happens is you might see an overall change for example in your quantitative metrics maybe things went up and to the right and you're like yeah we killed it everywhere but in reality uh what you're seeing is one specific country having like a gang buster quarter that's pulling all of your numbers up but the other countries are oh my god neutral you know and this can happen overall a b match a b test metrics uh you know cohorts you really need them to really understand really what you're looking at uh you reacted pretty strongly you ever experienced that sort of stuff uh dude i don't even know of course of course i am i have been tricked so much by so much quant so often um i almost feel like it doesn't matter how much experience you have or like how much training you have like you will learn by fire because a b testing is king but it's also like a dark horse you're like oh my future my feature was amazing look at this left and then you do the ad hoc analysis and it's like oh no it was just all indonesia yeah i know uh or like some very small like some segment um which i hadn't intended but and that's what an a b test is you're trying to learn if your hypothesis is right but you need to interpret a little bit deeper than just like face value or key metrics and that's where cohorts come into play 100 100 i love it all right bring us home mira okay so last but not least always ask yourself how trustworthy is your data now i told you like i want you to learn from my pain my hot tip is it doesn't matter how amazing your instrumentation is how many cool tools you use but if you don't know the source of the data and the limitations of that data even if it's quant it's like do not interpret it do not try and make judgments based on data that you cannot trust and this seems like oh like duh i wouldn't do that but i have learned that if you have some metrics that you really care about invest and partner with your engineers to go and make sure that monitoring is set up for those metrics because you set it up once and you're like oh ha i'm invented i'm instrumented look at my beautiful dashboard but let's say an event drops or someone makes a code change like you're not looking at that every day um you may be looking at it when you make decisions which might be like you know on a quarterly pace or like by sprint like every two sprints make sure that you've set up monitoring for the metrics that matter most so that if you do have a drop in data like for example you're only collecting 80 of clicks but then interpreting it as if it's 100 of that cohort um you'll have a monitor in place that actually says like okay this these are all green so this is all reliable so yeah have your team take the time to set up monitoring so that you know your quant is reliable and thus interpretable 100 it's like i can feel all of the data scientists watching this pod or listening to the spot just being like thank you like this is probably the hardest part of quant i think is uh it's not always just interpreting the results it's making sure that the data is quality data and that everything is firing the way it's supposed to fire um man critical critical uh part of the cheat sheet here and then it's appropriate it's last one hopefully it's the one you remember the most for sure all right episode five in the can oh we did it we did it halfway sober all right well uh thank you thank you for uh joining us for product happy hour if you enjoyed happy hour today please support us by donating on our podcast website producthappyhour.com support there are monthly one dollar five dollar and ten dollar options we're trying it out donations help us keep this sucker going without too many annoying ads so thank you in advance for your support you can also support the show by following the show on youtube spotify or wherever you get your podcasts please also rate the show on spotify apple podcast or wherever you get the your podcasts follow us on instagram or tick tock for clips at product producthappyhr and please share with your friends and spread the word the more people at the bar the merrier thank you so much for listening to the show and we'll see you next time cheers cheers [Music] you
2022-08-16