Manage your CEO: how to avoid opinion-based product prioritization (Apps track - Playtime EMEA 2017)

Okay so the next session is about. How to manage your CEO so let me ask you how, many people, hands, up if you're. An employee, and, not. The CEO. Excellent. So, hopefully. You'll. Be able to take away, some. Tangible, lessons, and tips. That you can take to rein, those crazy people in. Who. Can empathize with. This, meme hands, up. Yeah. Knowing. How. To handle, the prioritization, of, what. To build in your product when. You have so many competing, opinions. Stakeholders. Self-interest. Not. To mention. Important. People like, your CEO or, CFO s very, strong views, maybe. On the, type of color of green for example, so. How do you avoid opinion. Based product, prioritization. Well. This. Was an interest, of mine so a few months ago I interviewed. About. 15, to 20 of our top developers, I. Interviewed. Google engineers Google. Product managers directors. Of growth and. Like you know tried to tease out of them how, did they approach this problem and. You. Know what struck me was there, were two, key, characteristics. The. First is that. Successful, developers. Rely. On data, to. Make a call on what, to experiment. On first and, what to build and, the. Second, really important, factor was. That they had imbued, this. Culture. Of data and. Sharing. Of information throughout. Their organization. So. To add color and context, to this discussion. In. A few minutes I'll be inviting on stage both one tap and. 2017. App of, the year winner memorize. To. Share with us their experience, is on raining in their crazy CEOs, I. Know. One of them is there laughing he's not that crazy so. Let's get, started with with prioritization, and just a note, the. Models and suggestions. That we make here today are, just some of the best practices that you, know you, can use so they're, not the only ones you can use but. Hopefully you'll be able to take away some key nuggets of gold. Right. So. Just, a few seconds ago many of you had put up your hands, to. Say that prioritization. Was hard. But. It can be made easier with data, thank, you for reconfirming. So. Over. The last month I've, worked with the one tap team to. Test, out a model, that Google. Used internally, for, deciding, how to prioritize, what. We build, we. Use this model on a, low YouTube. Gmail. And, we. Actually have an internal, growth, team, who. Gets seconded, out to our different products so. I harnessed that team and. We worked with one tap. Now. Again this is not the only way to do it bear. With me but. If it gives you a different perspective this. Is a good thing now. Before, I introduce John, on stage let, me just give you a brief overview, of this Google model. Okay. So, as you can see it's. Pretty simple it, looks pretty simple. It's. Made up of four components, the. Northstar metric. User. Flows a, growth. Model and a. Spreadsheet because. Let's. Face it, what. Database, model would be complete without a spreadsheet. Now. The, model serves to. Help. You uncover opportunities. And more. Importantly, it serves as a common communication tool for your whole company to, talk about growth. Products, and prioritization. So. To kick off and go deeper on this the, first one I mentioned was the North Star metric, now, listening, to Antony rebo's talk about. Behavioral. Design it's, struck me that we were talking about the same thing. Which. Is it's that key, engagement. User metric, now. The way that we define, a Northstar, metric is that. It encapsulates, everything. That, you do in your app and it, captures. All the value that you're providing to a user. So. As an example if. Anyone here is staying at Airbnb this, week it. Would be nights booked, for. Whatsapp it would be messages, sent. So. Again the purpose is to create a shared goal across all of your different divisions, in your company, so they're all going, for the same goal, the. Next. Element is mapping the user flows. As. Anyone here maps their user flows a. Few. Of you good okay so this might be familiar to you the. Way that we approach it is that. Essentially. We define user flows as a visual representation, of your user journeys, throughout your app across. Acquisitions. Engagement. Conversion, or retention and that. Closed from, sources. Of acquisition. Critical. User journeys, you. Drill down on some important aspects. Perhaps. Your engagement. Activities. And, your. Retention activities. Now. I, can. See you taking photos madly the, next bits gonna really mess it up take. Your photo now, once. You've mapped this out and I. Quite like the color coding it helps me visualize it, better the. Next thing you want to do is mess. It up with some lines so. You stick in how. All of these components, relate. To one another. The. Idea being that this will then show you where. Some of the activities, are not supporting.

The, Rest of the app and help. You identify opportunities, to, improve now, that's all well and good but I mentioned, this was all about data didn't know so. What with what's missing here and which. Will really rely on your having a very robust analytics, system is to. Put, in the proportion, of users who were successfully, going from Step a to step B, which. Will then allow you to see where the weak points are in your flow for. Example there might be some. Issues, with return from notifications. And some. Issues with notification, permission you, can draw a conclusion there. So. This, mapping, of user flows really. Helps everyone in your business talk about the product in the same way and understand, how their activities, impact the. Overall that, north start metric. Once. You've worked out where your area for improvement is I mentioned. Notifications. Was something in this example, then. Next step is to drill down on what are the key. Key. Levers, that. Will impact that, metric so in the case of notifications, for example we had average. Users opted in we, had average number of notifications sent, and for. Example the click-through rate of those notifications this, is not rocket science but just drawing it out helps, everyone, to be clear on what the potential. Levers. You have are now. I mentioned. Just, need. To have a spreadsheet in here somewhere so, once you've mapped out that subsection, transfer. It to a spreadsheet link. It all up together and you can run scenario, analysis, to, compare your different competing ideas to see which one will, have the biggest impact on your, Northstar, metric. Now. To. Tell us more about their. Experience, because, as I said the, Northstar metric, seems pretty simple, but. I think we know there's a lot more complex, thinking. That that goes into it I'd like to welcome on stage John. Butterfield a head of growth of one tap. So. One of the things we did when we first started, making, this product is focusing, on analytics. And. We track everything user does within the app and this. Is something that we are really proud of. But. We'll see kind of how that one unraveled. While. Working with Townsend. So. The first thing we did as Tamsin showed you earlier was we created this user flow this. Thing up here this, is about the 10th iteration of, it the first one had, thousands, and thousands, of different. Things in it and. It. Was just a cluster thing. I won't, say the word and. We don't. It down to this and we. Actually worked with the, head of growth at Google on this one as well and the, thing he really gave us as a key takeaway is don't.

Include Everything, from the app just, the key moments, and the key journeys and the. First thing this did for us actually was show, that we have some problems and this, at the end that weird mustardy call it bit where. It's essentially like an exit point the, user doesn't flirt the, user journey doesn't come all the way back around so. This was the first useful part of, building. Building. This model. So. The next thing was about the North Star metric and, now this is an interesting one because for. Us it's pretty simple, so our mission as a company as. You can see is to make being being, self-employed easier, than being employed and. So. If we want to make this easier, the best thing we can do is. Quite. Simple. There. We go ace collects, transactions. The, more transactions, we collects for a user the more we can provide value, and it's all about providing that value so. Transactions. Per user, is and is a really easy one for us but then we break it down per product so. With one tap receipts the. Obvious thing here is the. Transaction, in that is, a receipt, receipt per user. Now. The tricky, part here and the thing that really throws off was, what's. More important. Users. Adding a lot of receipts or a lot of users just adding a few receipts, so. Again you've got to think about the value and the value is them, you, know if they, give us more data we can then help. Them with their taxes, and do everything. For. Them essentially, so it. Is all about receipts, per user and the more receipts the better. So. Then we put it into our growth model as you saw is like this really long document of all things the. Whole equation of, the growth model so I, didn't put all that in because, it's just too much and it's really boring. But. When we first wrote that out it is about 16, pages of this document, and he was crazy. So. Again. We took the idea of just dumbing it down and making it something that anybody can understand. And. The headline here was receipts. Added, per user and it's broken down by the Mau. Divided. By, the. Receipt submissions, and then it's the receipt, submissions, as you can see defined. By the different entry points and new. And returning users. So. What, was interesting in here as I said we're really proud of the analytics, and the tracking that we put in with it within the app. But. While we started working on this we suddenly realized hang on we're not tracking this this and this and these are all things that provide. Value and make it easier for us to, deliver. Our Northstar metric so, suddenly it made us realized we've got to go back and update, our analytics, and things like that, and. With. This growth model one of the most interesting things it did, was. That it made it easier for us to explain, to. The, CEO, who I won't, bash in this session today because it's that right there I Nick. Its building as well actually which is even worse so. It, made, it so he could see why the the mail comes from and then the CFO could see worth all, the revenue comes from and products who's, also sitting there could see what. What. Essentially would work next to what, levers to pull to. Get the most amount of value. So. Then we put it into this calculator. This, growth, model spreadsheet. This. Is just the the first part of it but, again you can see here with the starting values essentially. What happens over ten days, and. This is just the high level it goes further into detail as you go into it but again was boring to show you nobody really wants to see that, but. The best thing is it's all it's a predictor, and, this, an, in this calculator, the spreadsheet we actually renamed, it as a calculator that's why I keep calling you that because, to be honest nobody really cares about spreadsheets especially, myself. Calculator. Because, it helps us predict what we can do, so. Absolutely. While bringing in the equation. From the previous one I'm gonna go back that one so, once we had this equation and we put it into the spreadsheet, we. Suddenly started questioning where, the value was as well so there was a lot of things we put into the original model that, suddenly just made no sense, so, we really it really gave us a good lesson on how what. We should be tracking and why so. It was really useful. All. Right and then the next part these what the model actually gave us so. The first thing is it, gave us clarity. About. What, it would cross all departments, about what it is we're doing so.

Like The CEO Nick no. Longer had SIL I asked about why the mal was coming from and how we could increase it clear. As day in this model and. Then products, didn't have to like necessarily ask what, are we gonna do next what's priority. Is all based on this and, the. CFO, knew, that by increasing revenue, we, increase retention so they could see from the growth model as well essentially. What is happening next. Although. The CFO, dis just still sometimes go, very Jerry Maguire and show. Me the money but that's, yeah. It's not too bad. And then the resource is the, big thing we realized, is that because, we're a seat up and we do a lot with finances. We. Should be putting a lot of priority, into accountants. Right and. That's something we've not really done, too much of in the past so it's a huge wake up call that is a huge lever there that, if we put more effort into that we. Could essentially get lots more users, a lot easier, and help them keep retained, now. That also helped with hires, so. One of the problems you have that. We certainly have is going. Trigger-happy on glass door and just hiring the wrong people at the wrong time. So. This really helps with knowing who you should hire purpura. Kind of problem you have at the time and what levers you want to pull so, for us it, was partner channels so that was really good. And. That's it. That's. It Thank You Tamsin oh. Wait. On I think mine's working, thank you John that was fantastic. Hopefully. That give you some gave you some really good insights into I, guess one model, or one way of working oh thank you yes what was I gonna click with my, fingers. One, one way of working to, help, you prioritize and. Bring the company together, now. One, thing you may have picked up on apart. For the fact that their CEOs in the audience was. That the model really helped them, help, John to communicate, why. They were doing certain things with. Product. Marketing. CEO. And all, the other teams. So. Let me ask you a question how. Many people, in the audience today. Do. Prioritization. Or, experimentation. By them just by themselves without working, with anyone else. Okay. A few people how. Many of you do. This in a team. Just. One team a lot more okay, if. You're not doing it across your company you. Can do better in, a very positive way so. Top. Developers, I've spoken to ensure that there's cross-functional. Representation. When. They do this kind of prioritization, so that everyone's, on board and aligned, towards, that common goal whether. You call it the North Star metric or anything. Else you like to choose and. Culture. Is critical, to this you, need to have an acceptance of risk and, return, and, to.

Take, Us through their approach. More. On the riskier side I'd like to invite on the stage, Christina, knodel's head, of product for memorize. Thank, you. Good. Afternoon everyone, I'm so. Happy to be here and I. Know, that we are running out of time shorten, everyone is quite. Excited, to get to the longer networking, break I think. My talk is going to be brief I have to apologize, I should be doing this in German, working. In a language learning gap but English. Is not my first language so maybe this is an excuse for this time. Memorize. Is a, favorite, language learning app for about 25. Million people, it's. Very, vibrant. And lifelike, way to become. Confident. In using a foreign language in. Any country, or any, situation, people. Experience, and we. Have a record number of over. 200. Language pairs that, people. Can learn, and hopefully. Eventually speak. And. The. Next. Few minutes I, will be sharing some, some. Insights of how we do product, prioritization, and, also manage our CEO. Memorize. Has. Had a very unique journey, so far. So. We've. Been super lucky, with having. One of the most creative and experimental. CEOs. Out there. It, has its upsides which, means that it's. A great inspiration and, it, injects, a lot of creative, thinking. And and, experimentalism. To the rest of the team the. Downside. Is that it, kind, of sets, the slightly different challenge, for for, product people and product team to. Manage the whole thing and manage all these creative ideas coming, out of the brains like that, and, as. An, example. About. Two years ago I. Got a text. Message from Ed our CEO, on Sunday morning, saying. That good. Morning Christina, I'm. In Scotland, I bought, a double-decker. Bus. I'm going, to find a repairman who is going to fix it a. Driver, and, the film crew and, the. Bus is going to tour, the. Rest of the Europe filming. Videos. Of native speakers, saying, phrases, and sentences in, all these different languages that are spoken, in these European countries.

So. It. Was quite. A unique experience, to get a message like that and what. Happened was that we, we had to sort of come up ways how to handle. These kind of creative, ideas and and, very. Often slightly, crazy ideas, and. How to turn them into successful features, for, our product, so. What happened was we had. To. Come up with, a way how, to filter. And how to assess. And measure such crazy, ideas that not. Just the CEOs, but also everyone, in the team can come up with and. We. Kind of felt like spreadsheets. And, and different, models and, frameworks. And. Systems and processes are maybe too harsh for, filtering. Out some, genius, pieces, of. Creative. Glimpses and, we. Ended up having and, coming up with six. Criteria for, how to filter, out useful. Ideas out of like generic nonsense, and, the six criteria. That. We came up with was every, single idea has to be iterable which. Means that we. Should start small we, should, take. Every feature as an MVP, and, then iterate if it is successful. We, think that the the, idea has to have an immediate impact so as, soon as we roll out the feature and and launch it for our users we do want to see some kind of a fail behavior, change or or, some. Change in metrics. It, has to have a lasting, effect, which. Means that, one-trick. Ponies do not quite. Work. For us we. Do want to build features that, have, an impact on on language learners for an extended period of time not just for the first five minutes or or the first day. Everything. Has to be merged obviously. And. It. Has to have a power of localization, which I think for a language learning app is maybe a unique but user. Base for. Almost any up in the out there is very global, so it's. Good to benefit from from, making sure that, it works in every single language for every culture, for. Right-to-left. And left-to-right languages. And. Finally. Obviously it all has to fit into a coherent. Product so even, the craziest ideas, just have to fit in somewhere and make sense. And. We. Went, even deeper. So. In, order to keep this kind of creative. And experimental, mindset in the company, and an in every single team, we. Divided, our product, development. Lifecycle. Into, three stages, the. First one being the discovery, stage where we. Prototype, we sketch, we, design withdraw we, use the dust the. Idea we. Build, something, that we can sort, of get random people from the street to test it out in. The. Memorize. Members. Case, the. Discovery, phase included. Sending, the bus to a mini trip to Oxford, to fill in the videos in English first to see whether like this makes, any sense or not before. Touring the rest of the Europe and. After. Sort of doing, the dry runs and and testing these ideas out. We. Define it in. A fairly detailed spec, with with. All the functionality. Described.

Everything. All the sketches all, the designs thrown out we. Do involve, data team in this, stage. Of the process quite heavily because they, we, have to make sure that we we, built the feature for, the right user for. The. Right use case we we, track and implement. The the, data points in a correct way so it's super important, that that. The. Data people are involved in the process. Once. Everything. Is ready and set up. The. Development, teams take over. And actually start building, the features, because. Everyone, has been, quite. Nicely involved, in in the discovery, and in the definition, phase it's. Fairly easy for the developers to understand, why we're building things and and how exactly they need to be built. From. The product perspective a super. Key thing to keep in mind is that even with the most sophisticated systems. And and, processes, in place it, even happens for the most experienced. Product managers, that you sometimes. Forget to switch on the feature and experiment, when it goes, out so. That the users will actually see it what you've built, and. Once. Everything. Is nicely set up and, the new version of the of. The app, is out and now it's time to take a photo of this as well. It's. The. It's. The joyful. Moment of watching. Her when and how the results come in. We. Are organized, as well so it's not like we're all over, the place with crazy ideas we have a standard set of metrics, standard. Car graphs for every single experiment, a B test feature, that goes out and. Sort, of the day after the release it's like refreshing this Bailey page, all the time which. Is the sort of reasonable, and sensible way to, to. See how the feature, and the new idea has performed, in. Our. Team's. Case it's, also the joy and fun of seeing. The results coming in and and, learning, whether we build something super successful that people enjoy or or whether, we need to go back to the whiteboard and and sort of sketch and start, thinking again so, I. Think your vision is quite popular in Germany so it's almost like watching the results come in votes coming in and and then seeing like whether we've on, a winning case a losing case so, hopefully. This has given you a few ideas how, to do, product. Prioritization, without spreadsheets, as well and and also. Manage, the crazy ideas, coming from your CEO thank, you.

2018-02-20 10:27

Other news