Get. To be back on this stage and I hope that you all have some energy left from two fantastic days of lots of content, and content, is what we gonna smooth, Li gonna be talking. About as well, because I would like to talk to, you about what we have done to, bring your content, out more, in Google search and make it shine. Cool. All right so I, think. One. Of the things that I have, learned over the last one, and a half years ish of talking. With you all and also bringing my perspective, as a web developer to the Google search team is that. Sometimes. We. Underestimate the. Power of Google search and what, it does to our website, because fundamentally why do we build websites not. Just for I mean sure, okay I yeah, we also build websites because we are kinda like excited, about the stuff that we can build like you know you've capabilities. And all that stuff that we brought to the web platform they're exciting, but. Fundamentally, we build our websites, for users, and all our our, work towards performance, and user experience and. Safety, and privacy and, all that are basically, mirroring, that as well but fundamentally. We want users to find our content, because the web apps that we create the content that we put on the web we think is very very valuable. Hopefully. You think that your content is valuable if, not we. Should have a chat I believe but okay let's assume that you think your content is really good and you have done the, necessary steps, to make it good how. Much does Google search give you how much does it help you well it turns out by a recent. Study that that, was. Done that. Approximately. 53% of, the content, that websites are CSS re the traffic that web sites are seeing are, actually, coming through organic, search what. Is organic search that's your you, know day to day search engine, of choice whatever, it is and that can be Google search for instance so what we are doing is we, are trying to bring free traffic to you to, bring your, content to the users to make sure that your content can actually, shine and prosper, on the web and that you your, company your business can actually be successful, as well so. That's like one thing one way to look at search and I think. For us developers, it's an important, venue to understand where the users coming from who we. Hope to be on the site and how can we make sure that they are seeing the content that they need to see because it's our content, after all and today. I'd like to look back at what we have done this year, to. Make Google search better for you so we are making things better without you having to do the work isn't that amazing like, free, good stuff it's, like free cookie I don't, know it's not as good as a cookie though I, still.
Think Though like cookie, is the entire thing with like baking cookies is a weird one anyway it's like I think like you have good cookie, dough and you put it in the oven and it comes out that's good anyways, might, just be me so, one, of the biggest things that has happened in. This year is actually probably our announcement, at Google i/o that Google bot is now evergreen, right. That's like a quantum leap we have been using a crawler. Or like a rendering, engine that was a little behind and we have jumped, forward to, making that evergreen, to keep up with Google Chrome and actually, give you all the features that Google brings to the web platform also, in rendering, so that we can index the content that relies on these new technologies. And possibilities. And capabilities of, the web platform but, what does that really mean well, first. Things first, this, is like a huge leap and has been basically worked on by, many teams many people a lot of the work is now also visible in other venues like puppeteer, for instance, is a side effect from what we have done in Google search as well and it brings over a thousand, new features, es6. We, have proper web component, version one support, we have serve as well no we don't have we, have a bunch of new stuff that, we didn't have before you can now rely on that to not be an SEO problem, to not get in the way of your content actually being accessible, to Google search and actually being visible in Google search we, have also made sure that, we. Are not just doing like this one update one time but. Within a, relatively. Short timeframe of every, new stable release that comes to, Chrome we're actually updating, our Google bot rendering engine as well so that what you see in your chrome is also. What you see in Google search and not, just that we, also made sure that the testing tools we have out there the rich results, test the EM tests the mobile-friendly, test, this URL inspection, tool within search console all these tools they, are also up-to-date. With whatever version, of Chrome we are using when Googlebot renders, your pages so, you can actually verify, that, things are working in. Google search and. The. Nice thing is that with this new change we are all like still catching. Up to what this means when when Bing and now our sorry Microsoft. Announced that edge, is now using chromium as well as the rendering engine we, were really really excited to see that Bing has also went, forward, and went like ok yes we are also running. With, an evergreen Bing, bot and we want to render JavaScript, using a state-of-the-art. Rendering. As well so, this means that now across. The industry in search engines we are seeing signals, of actually getting up to speed with the modern web in search engine, indexing. And that's really really exciting to me. Also. Basically. Google. Search got a lot more powerful because we are using the browser that you are probably using as well when, we are rendering your pages and making sure that, we work and continue to work on making sure that what, we see when we index your content to show it to use this on search later that, we are seeing what you're seeing as well right. By. Them there's. One thing where we still have some work to do. This. Is the user agent that you see in your server logs if we are accessing. Your content, with, Googlebot, and, the. Kenai might spot that something's, off here, right. And, I get asked this question like but you said it's an evergreen Googlebot, but then I look at my server logs and I see this. What's. Happening here the simple answer to that is that we, are trying, to maximum, to, maximize, the compatibility, with between google search and web sites and we, can't just make changes like this we have to test things very carefully and we make sure that we are supporting, everyone's, content, in Google search so we had to basically be, very very careful moving, forward, with the user agent but, we have announced, multiple times now that this is about to change we are about to update the Google, rendering and sorry the user agent in Google rendering so. Please please please if you, having, some sort of code that is, doing, you a sniffing, for Googlebot do. Not match to the exact string that you saw earlier match, basically very, roughly, about Googlebot if you, want to know if it's a real Googlebot, or not that is perfectly, possible we have documentation, on that you, can do a reverse DNS lookup from the IP that we tell you we're coming from if that resolves, towards Google IP addresses, then you know it's an actual Googlebot because the user agent string does not really give you that benefit, so, please make, sure that if you have workarounds.
For Googlebot, and others that you are not to strictly, matching, the user agent in the future going forward because. This change is hopefully happening in December and that's, what we are aiming for if. You want to learn more you can go to webmasters, Google, box comm, and learn more about this change as well cool. Another. Thing that I oftentimes, get asked and it's, a very very fair question is, what, about JavaScript. Being so much slower in indexing, and rendering in, Google search than non JavaScript, websites is that still a problem. Well. Last. Year Tom, and I we're on this stage and telling. You well you know it can take up to a week we, are very sorry for this forget. This okay. Because, the, new numbers look a lot better, so we actually went over the numbers and found that. It turns out that at median, the, time we, spent between crawling, and actually having the render these results, is on. Median, it's five seconds. Right. If, you keen, eyes you might be like a but what's like the 90th. Percentile, and to which I can say don't worry about that that's within minutes so. You don't have to worry about week-long delays or something or anything like that however be. Prepared that, you might still hear, that and the reason for that is the, way that the entire. Pipeline really works so the pipeline that Googlebot runs through is actually, quite a complicated one this is a gross. Oversimplification, it's. Much much more complex and that a bunch of stuff might happen at the same time a few, steps might run multiple times it is more complicated than this however most. Of it is transparent, to you because it's a like. Implementation. Detail that you shouldn't be worried about and you don't have to be worried about but, effectively, that means that the, only moments, when you actually see, if something's happening, are, actually, when, we cross when, we make an HTTP, request to your service we leave a mark in your server, logs right you see aha Googlebot, is fetching, this request. But. You don't really see when we are processing because, there's not much actionable, things that happen in processing, it's just like us dealing. With the responds, and like with the robots.txt, and all that kind of stuff happens, and. Crawling. And in processing, you, don't really see that rendering. Itself, you don't really see that because we have looks like working, through the resources, executing. Your JavaScript, producing, content bringing that back to processing, there's, not much you can see well, you can see though is at the very end of this entire process you see when the content, shows up in search results right, so. There's a whole lot of stuff happening, in between these two points that you can actually measure, and, see, there's. Like canonicalization. Which, means like removing duplicates, there's like error handling what happens if there's a network timeout, what happens if your server was down for a moment these kind of things we handle for you you don't have to worry about them but, that means that if we are only coming around crawling.
Every Now and then and then like certain things happen on our site that you don't have to worry about it might take a while until you actually see things in the index but that doesn't mean that JavaScript, held them back, generally. Speaking as we saw we are really really fast in rendering, so, that's not really an issue that you need to be worried about anymore. Okay. So now I have. Basically like address the two biggest questions that I kept getting since I owe and we announced the Evergreen Googlebot, the user update is gonna change and rendering. Delays are not really, a big problem that you have to deal with you can be pretty relaxed, about using, javascript in your in. Your pages for SEO reasons, but. Now let's let's take it take a step back and see like what. Is actually the complexity, of this entire thing why did it took so long and what's like what's that what's involved in making a, renderer. Or how do I make a rendering, engine how hard can this be just, give me ten minutes and I'll have like a renderer right so. What if we take like puppeteer, for instance which is effectively, a programmatically. Controlled, chromium, instance which is what our rendering, is as well right and we, just build our own rendering, engine okay cool or about renderer, for search purposes, so. I think like the easiest thing when we use puppeteer, is to just get started you just open. A new page and your puppeteer, instance and you set the viewport. Except. What do you set your viewport, to. Mobile. Did my dimensions, okay but what about mobile dimensions, and what if the. Content is like very, long and this. Is this is already a little weird okay let's let's just assume we somehow figured that one out we, set our viewport dimensions, easygoing whatever next, step is we just have to fetch the actual page right. So we make a network request we get all the the HTML. In that loads all the images and all the JavaScript, so we have to load these resources, as well and. We. Wait until the page is done. But. What does done mean. Do. We wait until the Dom content loaded do, we wait for page load do, we wait for when the CPU doesn't do anything, anymore but. What if the CPU keeps, like cycling, through JavaScript, because there's like some sort of periodic, thing that happens what if it takes too long what if there's an infinite loop in your website does that mean that this this instance, is now completely, like. You know blocked forever, do, we wait five seconds. Ten seconds, if I, this. Is this is what puppet dev what render Tron does render Tron just goes like until we do not have anything on the network anymore or a certain threshold is over like, then we're just gonna kill, this but. This is the simplification, we are having a lot of work to do to make sure that we actually wait until your website is ready and do not fall in the trap of waiting, too long either, right, so that's a problem and last but not least okay now we have actually like gotten this website to render what.
Do We do next well we need. To like figure out what is the content here so can we just like, pull. The HTML. From. The entire document like this but. What about shadow DOM, hmm. That's. A tricky one so how do I deal with shadow DOM being there because I might not get shadow Dom through this because the shadow DOM border is gonna hide that shadow Dom content ah. This. Is tricky you see like the, point here is that, it. Isn't, as straightforward as, you think to actually build a renderer. Our. Teams have worked years, to make this as perfect. As possible and we are continuously, working on improving and we're continuously testing and monitoring what happens. In search to make sure that your website, is rendered properly, there. Are things you can do to help us out here make. Your website fast make your websites reliable, you, know make sure that they are not, taking too much CPU resources they, are not like killing, the network and because these are effects that users will also feel right. So. We are working to make these complexities go, away but you should not worry about these complexities this, is the nice thing about it it's an API that is relatively easy and straightforward to deal with you just make good websites for the user the rest is my problem, it's. Nice isn't it it's like it's not my problems Martin. How. Can I ever sleep again I, don't, know we'll see and, I mean this has been quite a journey I mean this sounds quite not large, and big about the Evergreen Googlebot, turned. Search. Result. Indexing. From websites, that showed up when Googlebot, was accessing them like this, into. Like getting the proper content, so. We are finally ready to use whatever you, are throwing at us as long as it is reasonable for the users and the, web in total and. I think this is a very important, moment so these are complexities that we are taking away from you and you can focus on building better things for the user and I think that's the key here, and we. Are pretty happy that this is not just like us being like what. I've done. It's actually being seen in the industry especially, also from SEO so. Bartosh. From. 1ly which is an technical, SEO agency. Published. Like a case study and an article of the JavaScript, SEO state in 2019. And, he. Compared, that to 2018. And said like this is an amazing, improvement. And we have been doing so, much better in these terms and I think I like this specific, quote where he said like if you look at the National Geographic they're, a very good example, a hundred percent of their JavaScript content. Is indexed. Quickly, so. That makes us very very happy and this should be your signal, to bring this information back to your SEO experts, and SEO advisors, and tell, them like look javascript, is an important, part of the web and modern. Web applications can, be successful. In google search that is absolutely, possible. Right. Now I would like to quickly walk through a few things and a few patterns that I'd like to like quickly, discuss because I think they're great patterns to use and they, are also very very helpful for indexing. In Google search or rendering. In Google search one is lazy loading, I think we're all very happy, that we finally have the native ladies lazy loading api's for images and iframes, these, work in Googlebot as Googlebot. Is evergreen we are supporting, these attributes. And they are not hurting, you so if you are not using them yet you should definitely, consider using them it's such a low, hanging fruit to, do if you need more complex, lazy loading or lazy loading on elements, or content, that does not, fall. Into the two categories where late native lazy loading exists, I highly, recommend checking out the intersection observer, because, who would have thought this is also now supported, in Google bot and we are testing this quite frequently as well so. This allows you to have like more, in. Our custom, lazy loading if you need to as. I said web components, are supported, but as I also said when I said like let's build a naive renderer. Using, a puppeteer, shadow. DOM is an interesting thing right so like I got questions about like how, does that how, yeah.
Is, It supported, does it work and we actually index content inside shadow Dom's and the, answer is yes yes. We do like this is absolutely perfect you find now, we have a shadow Dom content in this case like this indexed, yes, it is what. About this and then we use some mechanism, to get whatever is in the light Dom in this case can you render me we pull this into the shadow Dom and that gets flattened, into the tree and then we can index both of this so all of the content here will be successfully, indexed, in Google search, so that, means, we. Have finally, full proper, version 1 web component, support in Google search I. Cried. A little when we had like that moment like Oh finally. That's. A beautiful moment, another, thing that I'd like to just bring up because that's something that doesn't just like magically, we fix it because we think it's not broken is, if, you have images, images are a fantastic, way of engaging with your audience and making sure that, you surface, your content if you have your original image. Content, you should put that an image search because it's a fantastic venue for, people to discover your articles, around these images, as well if. You want us to index your images totally. Cool like we got you the HTML image element is there and he'll do you know it's perfectly, fine this. Is what we do we search for image, elements, in your. Content, and then we index them unless you prevent us from doing so by setting the normal index or something. That's. Not. Great. We're. Not gonna index that. Don't. Okay. Sweet. Speaking, of images we, are also basically. Working, towards making image search more useful for users we. Have a bunch of different venues of doing so so, we are having better query and understanding. As well as we are trying to be more useful for like, figuring out the context, of the image in the article, so we are featuring, images differently, and soon. You will be able to use search console to also tell us if you want to, use. Structured data to give us high resolution, images. These. High resolution are sorry I. Shouldn't. Have liked songs of that many songs yesterday, anyway. Stage. Right. So. If you want your high resolution, images, - not hurt performance you, can tell us in. Structure data in the future that, you have a higher resolution version and we can surface this in other places, like Google, homes or, Google Assistant which, brings more traffic to these images if you're lucky so we, highly recommend checking out the articles that are coming up on the Google block in the near future and also, using search console, to see if there's like new stuff coming. Speaking. Of structured data we. Have more structured data to make sure that your content really can shine we have how-to markups, you have QA pages, and. All that kind of lovely stuff, if you want to check that out, I highly, recommend using, that link to find what, kind of verticals, we support we've support all sorts of versions we support events like this for instance the website has structured data so, that we can highlight that an assistant as well as in the, search results, like here we. Have recipes we have articles we, have books, we have movie, movies. I think and TV series and all that kind, of stuff and this this gallery is a fantastic, place to check out what's. Happening there and definitely, it gets you like this is very very much an eye-catcher right you get more screen and state in google search which means your content, is more likely to catch the eye of the user recipes. Are one example. Sweet. Last but not least I said we have made sure that our tools are always up to date with the actual rendering engine so when you are using. Mobile-friendly. Test which results test all of these tests, you, are actually rendering, through the real Googlebot rendering, process and. This updates as Googlebot updates, now so we are no longer having like a discrepancy, between what we do in the testing tools and what we're doing the actual in. The actual Googlebot so, we. Also made, sure that we gave you a better flow of working. With things in Google search console the new Google search console allows, you to monitor how your entire site is doing is in terms of like how is it performing, in search but, also are there pages that are having problems, is there structure data that is broken and, also like all sorts of other things you.
Can Easily debug, these things within Google, search console and use like other tools on top of it as well to. Make sure that you are on top of what's happening on your page and you can use the validate fix flow where you can signal, to us as Google search. I think, we fix this and then we Oleg double-check if that is true if it is fixed we let you know and also, re index the page you. Can validate if everything that you have been doing is right and then last but not least the, performance, report also helps you figure out where you can do better I can grow your, search traffic. So, besides debugging. Any sort of issues. That you might have on your site we have a few new really cool things so this is the URL inspection, tool that has been there for a while but it's it's worthwhile I like checking it out. Tells. You how, Google search was accessing your page the last time what, was the result can we index this content or not why, can we not index, it if we can't index it what can I do about it and there's also the validate, fix flow in the screen but. Also. We. Basically give you JavaScript. Debugging capabilities in. Search console, too so you get the stack trace you get the error messages, that we, are seeing when we are rendering, your website so you no longer have to just like guess you. Can actually run this and see what, we are stumbling over if we are stumbling over some JavaScript, problems, a very. Nice one is the speed report that launched last week so, we. Are now having this report, that gives you an idea of based, on crux based on the chrome user experience, report from real user data where. Our pages that we might want to optimize so it takes them into three buckets it, tells you these pages this is a sample, of pages that are fast these, are pages that are okay moderately, good and then these pages are slow and you can work with, these pages and basically integrate with other tools like lighthouse, or PageSpeed. Insights to figure out where you need to spend a little bit of time to make them faster. Imagine. You have a website with like a million, pages. Do. You want to like run lighthouse on oven-like, okay, cool I'll be gone for a week I'm light, housing, again lighthouse, is a fantastic, tool once you know where these problems, are the, speed report is your way your lens into that you see where, can I spend some time to make my website significantly. Faster what are the low-hanging fruit where can I really where should I be drilling in and digging, in to make sure my website is fast and great for users so we highly recommend you check out this report. As it gives you a life life wonderful, sight light site-wide, overview of what's happening. Cool. As, you can see it, lets you drill into the different kinds of issues and you can also then like see specifically, what the pages need I see, people taking pictures that's good we. Also have a fantastic, new document, what new it's like not that new but it's new to many people here probably as the, troubleshooter guide if you have JavaScript, problems we have a guide that tries to identify the individual, problems and then guides you through potential, solutions, to them this. Guide is relatively new and we have worked with the community to make sure that we are covering the most frequent. Problems but. We would love to hear your feedback on this if you see problems that were not tackled, in the report then or in the document and please let us know we are happily extending, the document, with your feedback as well, sweet. Last, but not least I like you to like take a few things away and also to bring this back to the SEO experts, that you're working with if you are an SEO, expert in the group in the crowd here whoo, then. Please. Also bring this back to your developers, who might not be watching this stream I think. One. Of the things that shocked, me the most over the last one. And a half years was that oftentimes, SEO. And developers are not working hand-in-hand together because, I think we are very powerful allies, if we work together and they, can help you not deal, with all the monitoring, effort, and making sure that your content strategy is right they're really good at that but.
You Have developers, you can help them understand, better what does these what. Do what do these performance, metrics mean what does the slow performance mean, on a specific page what does the soft 404 really do and basically. Help them prioritize the, things and if we work together we'll. Be able to bring better content, to all our users much, much more efficiently, and we, will be able to unlock, these like additional. Users that might come over organic traffic, if we like take all the boxes so, we can be working together we, should be allies we should not be working against, each other ok so. Last but not least I'd like to sum it up by saying like Google search is here, and we'll be here to. Help you and your users to find each other we, want to bring relevant content, to the users that you care about. You might not even have met yet and was. That I like. To say thank you very much for your time enjoy the rest of the evening and bye, bye. You.
2019-11-15