Understanding the technologies that power “AI”—a product-owner’s guide

Understanding the technologies that power “AI”—a product-owner’s guide

Show Video

Hi and thanks for having me. My name  is Lindsay Silver. I'm the Head of   Platform Technology at Conde Nast. I'm really  excited to be here today to talk to you about   Data and AI and product development and to kind  of go through how we think about it at Conde Nast.  So, first, a bit of context, Conde Nast is better  known as the home of brands like Vogue and GQ   and the New Yorker, Allure and Architectural  Digest. We're a 100-year-old magazine company,  

that over the past 20 years has had  to go through a transition from being   a company that produces written content and  photography around the world to one that creates   dozens of digital products. So, by the numbers,  we're about 72 market brands around the world   that would be Vogue in France or GQ in Russia.  We have about 185 web mobile and OTT products;   those include things like the New Yorker crossword  app or self-magazines website and web properties.  

Across that network of products, we have about 350  million monthly active users, about 450 million   to 500 million in annual digital revenue.  And between five and 6,000 editorial users.   So, let's talk a little bit about how we build  products at Conde Nast because I think it's   important regardless of what the features are to  sort of discuss how you think about products in an   ecosystem like ours. So, obviously, if you think  about our audiences at the top of this chart,   we have audiences of all kinds. We have audiences  in specific markets with specific interests and   specific utility is in products that we need to  service. What we find about those products and   those audiences is that the products they use  have to be different but the tools, the APIs   and technologies, the data and infrastructure  often don't. And a lot of times when we think   about feature development, we're thinking about  how do we build a tool, an API, or a technology   that services the biggest number of end products.  And so, we've built this model that's sort of two  

dimensional where we have our portfolio of  consumer products across the top. We have   teams that support those individually. And then  as we go down the stack, we've built different   tools and teams and organizations focused on  different levels here. So, we have teams within   what we call our global platforms that are  responsible for things like content management   or add systems we also have teams lower in the  stack that are responsible for data architecture   and how we manage our APIs and technologies. And  what we've found is we generally have a smaller  

number of teams per organization, as you go  down the stack. And so, our platform teams   basically the same size that's the four  levels below are basically the same size   as our single consumer product team in terms of  the number of people we need to actually develop.   Okay. So, let's talk about developing AI driven  features or data-driven features. At the core  

of any data-driven feature is a feedback loop  actually at the core of almost anything that   we do, and it involves thinking whether it's  artificial thinking or human thinking is a   feedback loop but assume that for any feature that  you're building that's AI driven or data-driven,   you're talking about a feedback loop and I think  that's a really important thing that people   don't sort of boil these down to but basically,  what we're talking about is some sort of action   or some sort of feature that is on the page that  a user sees, some sort of signal that comes out of   that feature that synthesized by a machine  that then leads to a decision about how to   change that feature or to improve the experience.  And then the circle, the cycle starts over again   there's a test. There's some sort of conversion  measured that synthesized by the computer again or   by some sort of model. And then a decision is made  and then actions happens. Now this is of course,  

very general. I mean, this is how you live  your life, probably. You're constantly   trying out new foods, deciding if you like  them applying the tastes and to your historical   knowledge of the foods you've eaten, making a  decision about whether you'll eat them again   and then maybe eating them or  not eating them next time. That's   almost universal in terms of thinking. What's  interesting about building features that   are AI driven is that you just have to think  about it in terms of how a computer thinks or   how to build these things in a way that allow  computers to think about them. So, starting with  

that action systems like your multivariate testing  platform that you might use. Your ad servers,   all of those personalization engines are built  so that they can actually serve features in this   way. So, we use a multivariate testing platform  called Wasabi that was out open sourced by Intuit.   We've changed, we modified it a little bit to  allow us to do tests that are self-optimizing   but it effectively serves a lot of our feature  variation. So, if we want to test two button   colors, if we want to test different recirculation  algorithms, systems to decide how to rank content,   we can use Wasabi to do that. Once we deployed a  feature with Wasabi, the next step is for a sort  

of a set of systems to synthesize the information  that comes out of that. We synthesize data and   we'll talk about in a sec and we get into the  domains that we synthesize data about our content.   So, we extract things like keywords and entities  and topics. We synthesize data about our users,   so that might be building models of propensity  or likelihood to subscribe and we synthesize data   about our advertisements and our monetization  options. So, that might be whether we should show,   so let's say a house ad or a marketing ad or  a commercial ad to a specific user at specific   point. All of this data is held that we store  in various data systems. And we're doing this  

in real time as articles are published or as users  come to our sites. When we need to make a decision   then that usually is when you're visiting our site  and we're trying to decide what to do with you or   what to show you, we pass all of that data into  some sort of predictive model. And this is when   you hear people talk about ML. A lot of times  they're talking about the models or the systems  

usually there's an API on top of them that make  predictions based on data that you pass into them.   And so, in the example that I've sort of given  around recirculation or getting people to click   on pieces of content that send them around your  website we would use a predictive model that tries   to guess what content you're most likely to click  on and a lot of different models that we have   and we've run a few dozen of these models at the  same time, depending on the type of user you are.   We will pass data about the context, about what  article you're viewing, we'll pass data to those   models about what we know about you historically.  Do you usually click on articles about kittens   or about sports or about fashion and then we'll  pass data about the advertisement or monetization   options we have. So, are we in Q4 and are  we trying to do a lot of commerce or are we   trying to push for a lot of top of funnel brand  lift type campaigns with some of our partners.   All of that data goes into these predictive models  and those send back a set of articles or a ranking   for those pieces of content. We pass that  back to our systems our multivariate testing  

systems or our personalization engines.  And those actually render the page based on   what you've said. That happens again and  again, over and over again, millions of times   per hour for Conde Nast and we're actually making  these decisions over and over again. And what's   cool about that and why really, I fell in love  with this type of feature is that if they're set   up correctly, every time this happens every  time that you go through this feedback loop,   you're actually improving the system just a  little bit. And so, when we deploy a new feature,   we have zero data and signal about how it  will perform but as these things happen,   we're actually building a better and better system  and historically this was done kind of manually   when you hear people talk about retraining  models or pulling down and redeploying models   that's speaking manually about the  improvement of that feedback loop.  

More recently, a lot of our models, a lot  of the models we use for recirculation and   for advertising and for defining user  propensity are auto updating. And so,   that means that they can that every time someone  views something or in a lot of cases every certain   time intervals, let's say every hour, the models  are retrained and that means that this improvement   is happening behind the scenes. The models are  getting better and better and better without us   actually doing any work ourselves. And that's  that I think is the key to this type of model   in the context of companies like Facebook or  Google or a lot of the big media companies now   is that these models are designed in such a way  that they add value to the company incrementally   without much additional human interaction, you  kind of have these cooks in the kitchen sort of   adjusting things and making sure that the levers  are sort of pulled the right way but you don't   actually have incremental development going on  in the same way that you would with a completely   human driven model or sorry, feature. This is kind  of the core of AI driven models. Now this doesn't   go into the technicalities of building these too  much. I can tell you, basically, we use a system,   we model these in specific domains which I'll talk  about. So, the really interesting thing happens  

with feedback loops when you turn them on their  side and I think this is the core of the reason   that AI driven features have taken such focus over  the last few years that, when you have built a   feedback loop like this just like when you build a  really high performing team developing features in   general. Every time you go around this feedback  loop, you get slightly better at what you're   doing. And so, if you take the example of our  Vogue recirculation models every time we go from   act, to synthesize, to decide, to act. We know a  little bit more about our audience and our users,   and that's not because our teams know more, it's  not because people have become more seasoned in   their jobs. It's because as we have a stronger  signal and more data within our models or, or   that we have trained our models with. And  so, we're constantly retraining our models  

and that. Adds this incremental value that makes  them better and better at their jobs. And if you   look at Google or Facebook, you'll see they've  built their whole businesses on this model of   incremental improvement. That's automated because  of the way that their ML systems are set up   and it's really cool. It's actually the core. It's  sort of what the root of my curiosity about these  

features and the reason that we've driven to build  several or so many of them within Conde Nast.   Okay. So, the second thing to know about AI  driven feature development is that it relies on   an understanding of your domain and just like  the feedback loops from before understanding   your domains is something that's obviously  really important to developing any feature   but when you're building an AI driven feature,  knowing what your domains are is important,   because you need to translate those into things  that the computer or the servers that are   making your recommendations, understand. So, the  way we look at this at Conde Nast is in terms   of the three major domains that we have to deal  with and the names sometimes fluctuate when we're   discussing things but roughly, they translate  to our content as our first domain. Our content   represented by a content identifier or an  image identifier of some sort and with traits,   things like who wrote the content what the  content is about, what keywords are important   to the content. All of these are traits of a piece  of content and when we look at our content model   from a data standpoint, there are hundreds and  hundreds of traits in for each piece of content.  

Our second domain is obviously our user base  and for almost any business users will be one of   the domains that you need to understand are the  areas that data that you need to understand our   users are usually represented by some sort of ID  whether it's a session ID for anonymous users or   an email or a hashed email address for users  who we have seen. And that's important because   obviously we need to be able to link an individual  user to individual pieces of content they've seen   or propensities that they have a propensity to  subscribe or a propensity to view additional   content but there are dozens of other traits that  we have. We derive a lot of first party traits   about our users. What types of content they're  interested in, how often they come to our sites  

and we use those when we're building AI driven  features. It's really common to have a model   that takes an associate, a piece of content with a  user who might want to read that type of content.   The third domain we have, we call it sometimes we  call it monetization. Sometimes we call it our ads   business. Here, I'm calling it our experiences.  Those are the actual context, the actual   applications that were represented that tie  together content with our users. And they are   important because they stand alone if they were  just pieces of content, we wouldn't need this but   a lot of times we'll show a piece of content to  a specific user across their mobile device or a   mobile version of our sites, across our desktop  versions of our sites possibly within a mobile   application. Sometimes even with our videos within  other media like our set top box applications and  

other places. And I apologize, there's that low  flying airplane above me right now. Basically, our   first goal and something we tell any the product  manager who's in the space is thinking about these   types of features is they need to understand  these domains that they're working with.   Not every feature needs to understand or needs  you to understand all three of these domains,   but you definitely need to understand any  domains that are related to your feature.  

Good example here is a feature that we released  fairly recently that took an auto recommended   editorial tags to our editors. So, in that case  we needed to understand the individual editor,   what tags they'd used in the past, what types  of content they'd written. And then on the   other hand, we needed to understand the pieces of  content that they were potentially writing. So,  

we had to have our NLP systems extracting  keywords and understanding that content.   It's important for the folks, any of those  features to understand both of those areas   and what, how they might relate to each other.  You can come up with a hypothesis that that   brings me to step two is which is how these  things relate. So, it's obviously extremely   important for anyone building a feature  that does recommendation to understand   what they're recommending, what basis on you  might recommend something and a good example   here is in something that happens a lot is that we  can come up with false causation. Food contents,  

a great one for this. So, when we started  doing a lot of food content recommendation,   we really tried to look at what a user might've  looked up before, what ingredients they wanted,   all kinds of really deep information about  our users to build our recommendation model.   On the content side, we needed to understand  what ingredients were in our content,   all kinds of things. And we came up with a pretty  complex model that tried to do personalized recipe   recommendation. What we found over time after  testing a few variations was there were actually   other contexts they're way more important to  this then user's past behavior. The biggest one   was time of day and day of the year. So, when we  started looking at what content worked the best,  

we found is specific content, specific recipes  work way better different at different times   of the year. And that was actually what we  consider an experience or a contextual attribute   time and also geography. A lot of times we  consider those contextual. So, we actually   started building models for recipe content that  took into account day and time as one of their   attributes and what times and days that content  had performed well in the past. And that actually   increased our click through rates substantially  on those types of recommendation models.   What's interesting here and you can obviously  take this as many degrees as you want. But I think   that step three is the interesting one, which is  asking yourself when you're building a feature,   how that feature relates to all of your other,  there are domains that are involved. And so,  

a good way to look at this is if you  start with your subject in this case,   let's say, it's your experience, how does this  specific experience relate to a specific user   with specific content is a question we ask. If  we create an inline link serving module, which   we do a lot, we do automatic hot linking within  our sites how does that apply when the content   is about a certain a subject and is being applied  to a different audience group. And that actually   is the foundation for a lot of the hypothesis for  features that we build. We rarely say something is   going to be so powerful that it will affect all  users in all content or all contexts but we do   focus down on multiple as many user groups as  we can and as many content types or as many   content types with as many users and in looking  at as many experiences that kind of gives that   ubiquity that we talked about in earlier when  we were talking about how to build a platform.  

Obviously, the broader and maybe a general  concept in AI as sort of generalized AI that's   a computer's ability to handle a variety of  situations and the Holy grail for machine learning   engineers that somewhere like Google is to build  systems that solve really general problems. The   same goes here that the wider the context,  the more problems you can see for the model   the better you are. So, we obviously strive for  personalization models. Let's say for our pages   that are as broad as possible. But usually those  are actually much more segmented. Usually, we'll   find a model that works really well for a set  of people in a specific context and that's okay.   But you need to understand deeply what you can  do with each of these how users are identified,   how contents identified, how you know where you  are in an experience, and then what data does   the computer have to work with related to that  context. Cool. So, I mentioned these, this'll be  

a quick one. I think it's important to specify the  key ingredients for building an AI driven feature.   So, as I mentioned, a couple of times, everything  we have in any domain has an identifier and it's   extremely important when you're building anything  that related to data that you can somehow identify   that domain really uniquely. So, a user always  needs to be identified uniquely a piece of content   needs to be identified. And then what we call the  context or the page of a website or the screen of   an application or the specific spot even within  a page needs to be identifiable and that allows   you to attach traits and information that to  those and so in our case, our traits as I said   are in the hundreds for our users in the tens to  hundreds for our piece of content and definitely   in the tens at least for our experiences, our  advertisements. At the end of the last slide   are the relationships. So, every piece of content  has a relationship to a user who's looked at it.   They've either scrolled or they haven't scrolled,  they've bounced or they've clicked to another page   those are things that relate to those help you  understand the user's relationship with a piece   of content and allow you to build a feature off  of that. With these things and thinking in terms  

of these domains and in terms of those feedback  loops, you can build a whole myriad of features   and next I want to just take you into kind of  what these AI driven features might look like   at Conde Nast. So, let's talk about some of the  features that Conde has built a little bit more.   So, the first is personalized recirculation.  So, about three years ago, we started doing   personalized recirculation. That meant looking at  things we knew about our user domains and about   the context. So, at the time of day, as I said was  one of the aspects but things like what platform   you were on, where you were viewing the content  and using those to actually personalize our   recommendations for recirculation. When we talk  about recirculation, we include things like hot   linking and I think one of the more interesting  applications of this was a system that we built   to extract tease from our content like name's  Tina Fey and Amy Poehler in this case. And then  

actually look for content within our site that was  most relevant to that user about Tina Fey or Amy   Poehler, and actually auto link that content  back. And so, at the end of the day, the user   when they clicked on this piece of content would  be redirected to another article about Tina Fey   that then also took into account things that they  knew and we saw some really interesting things   happen with some folks where content actually  was linked that was related to something else   they'd been reading. So, if they were interested  in the MET Gala, on Vogue, you might see this Tina   Fey article actually linked to an article about  Tina Fey at the MET Gala. Now, obviously there   are a lot of parameters to this. You don't want  to link to an extremely old piece of content. You   don't want to link to a piece of content where  Tina Fey, whose name might be mentioned a lot   but which is about someone else or something else.  And so, when you're building these models, there's  

a lot to think about. And that's why this deep  understanding of your domains is really important.   The other thing that we think about when we create  features like this is how do you validate them?   So, how do you watch and make sure that that  feedback loop is actually closing and improving   every time. And with this feature over time,  we saw it improve a fair amount. Although what   we found was that its kind of plateaued after a  little while you couldn't get these just straight   personalization features with a single model or a  universal model to work any better than, about 10%   better than when they started. So, what we did  and what applies to a lot of the features that   we've done now is that we actually treat them as  families of models. And so, when we try a model   that might be what we would call a personalization  model, we may have five or six variations of that   model and we're running at the same time  using what's called a Multi-armed Bandit.  

Those models in parallel, we're testing them  against different cohorts of our audience. So,   we're actually doing experiments with multiple  models at the same time and for some people,   the most popular article about Tina  Fey might always be the one that wins   for other people, we may test a contextual model  that really targets what they've read before   and at different times those will  work for different audiences. And so,   that layered approach is really important  to making this type of system successful.   Another area that we've really honed in on  our honed in terms of this is our experience   optimization. So, we know that different users  and different contexts or different experiences   respond to placements of content in different  ways. And so, there was a whole study maybe five  

years ago and the human driven study that found  that slide ups from the bottom of pages actually   had a pretty significant positive impact on users.  What we found over time was that depending on the   situation or the time or the type of device you're  on, actually those are detrimental. They increase   bounce rates or exit rates. And so, we've built  a system or a set of systems that allow us to   run those things experiments in parallel,  and then using that same Multi-armed Bandit   or auto optimization algorithms to actually change  where these things are placed based on the person,   based on the device. And a lot of times those are  sort of black box. We put in a lot of information   about the context. Again, what a user is using  to view the content, what content is on the page   what time of day it is, what we know about  the user from the past and the models will   offer will spit out versions or responses to that  placement and without actually giving us a full   explanation. And so, we've over time, we've sort  of honed these models and improve them but they  

constantly are getting better on their own. So,  as we retrain us, we get stronger and stronger   signal. We're seeing increases in a clip through  rates and response to these types of units.   Third area that Conde Nast has had a lot of impact  are sort of AI driven features that had a lot of   impact in are our pushed content. So, in addition  to changing the order of content on our emails   and our notifications, we're actually using AI  now to make decisions about when we send email,   what ads are served in that email. And in some  cases, even whether we're sending email or not.   We're timing out users or we're decaying the  number of emails that we send to users over time   based on their likelihood to open those. And also  based on things like what content we know they've  

enjoyed in the past. So, a lot of times and in  some of our bigger newsletters now, we're making   decisions based on the content and the newsletter  what audience will get it and those are all   dynamic. If you've never clicked on an article  about politics, let's say, and we have a heavy   newsletter about politics. We'll actually adjust  downward, your likelihood of getting that email  

and it's still probabilistic and we still include  audience in there that we think is low likelihood   of opening, mainly to test and to validate those  models and every time we do that and they improve,   we've got something that helps our AI driven  feature improve. So, we're always thinking in   terms of how to build those features in a way that  gives us that signal back and improves the signal.   So, the next feature is less of an explicitly  AI driven feature and more of a feature that   allows us to power more complex AIs  within our applications and that's when   of advanced interaction. So, then the labels a  little bit abstract but the concept is basically   that when you're building features that take  data about your users or about their context   and then synthesize it and make decisions,  the more the proprietary, the information   is a more specific to your experiences, the  better and so with brands. Especially brands   that are kind of like. Specific in their  outlook things like Golf Digest or Brides,   we need to get information that's specific to  those users. And so, over the years, we've done  

a lot to actually get information about people  that isn't necessarily that general information,   their age or the time of day or things that just  the real basics. Golf Digest is one where we got   a lot of information early on that I thought  was really cool. We were able to find people's   favorite club types the courses that they enjoyed,  specific shots that were difficult to them.   And then what we could actually surface content  and surface information for them. That actually   was really relevant to their level of play and  their interests and that's it. I know this was   pretty high level and I'm happy to delve into more  detail when in the question and answers section   or feel free to reach out to me by email or on  LinkedIn. I look forward to hearing from you. Bye.

2023-10-02 03:11

Show Video

Other news