Saying goodbye to full stack tests with task analysis | Mark Winteringham | #SeConfLondon
Hi. Everyone I've never come under stage to music before that, makes me very very happy I also. Can't see a lot of people because the lights are in the way so. I feel very much like a rock star which was what I was always supposed to be, anyway. Yes. So thank, you for all dragging, yourself away from the fish and chips to come and hear me talk about, the. Automation breakup I don't know why they keep let, me come to speak here at sea comp because I spend. Most of my time trying to convince you all to stop using selenium-webdriver But Here I am again so. I'm, going to be exploring. Two different, questions. The. First theis is that how, do we know that the, checks, that we've created are, valuable. And useful to us and then. The second question that saw comes that sort of stems from that is. If. Our checks are not valuable, how. Do we go about, identifying. Valuable. Checks. What. I'm going to do is, I'm going to share some some ideas, some. Techniques show, their code maybe, even get it running who knows. But. Essentially, all of this comes out of a. Mindset. A namespace, that myself, and Richard Bradshaw, who speak it's morning. Called. Automation, in testing and we. Believe that in, automation and testing we should use testing. Automation to support our, testing, so. Part. Of this is this, analysis. Of our checks to see whether or not they're, of use to us. So. Yeah first question how do we know if. A check is valuable, I. Thought. The best way to kind. Of demonstrate, this, and, also I struggle. Sometimes with talking, about full stack tests, and to end tests, they kind of mean different things to different people so. What, I'm going to do is I'm actually going to demonstrate, what. I see, is the sort of common pattern. Of. Tests. That will checks that are written in, webdriver. So. My. Check is going to focus on a, specific risk it's. Going to focus on the, fact that user can log into an application. This. Risk is quite sort of abstract, quite high level. But. Will sort of dig into that a little bit D in more detail little, bit later yeah. I'm, basically. Written something that, focuses, on whether. Or not a user can log into an application now, I'm. Working, with. An. Application, that I've built called. Restful, Booker platform. That's. A sort of free to use open source project, where you can practice test, automation against, there's. A version of it on automation in testing, online but. I'm cheating, and I'm gonna use mine locally, because I don't trust conference, wi-fi's, sorry. Okay. So let's, talk very briefly about what this does so, to, start off with, we're. Using webdriver. To navigate. To our home page. Once. We've got to the home page it's gonna click on an admin, link it's, gonna find it click on it which, takes me to a login page and, then, I'm gonna fill in the username and, password click Submit, and then. I'm doing just like a final, assertion. Here, just to say do, I see the room listing, page, which. Means that I've logged in now. For. Those of you who can see the threat sleeps, please. Don't, panic, they're just here so you can see the script running. This. Is not something I would normally do, but. Let's give that a run so we can see what it is they're actually trying to test I think across this works the demo gods, are good to me. Cool. So it loads up the home page clicks. On the admin link. Fills. The details in checks. That the room, listing, pages there and it passes lovely, fantastic. So. As I say this is a common. Pattern common. Approach I, see. To how. A. Lot, of those do test, automation or. Automated, checking. Now. I want, to take. A step back and I actually want to sort of ask that question is this, cheque valuable, is. It, something that I want to be running regularly, and to. Help me determine that in. Automation in testing we've created this, series of characteristics, like quality characteristics. Ways, in which we can judge, whether. Or not a cheque is of value so. The. First characteristic. Is targeted. So. I want to ask myself is this cheque, targeted, so. Bring it back to that risk the, user can lock in that. Risk. Is actually quite abstract, it's quite high level it's, very much focused on the flow of, being able to log in it's. Not necessarily. Targeted, at specific things that occur inside the product, so. Say for example. During. That user login flow there's. Stuff going on in the back end but. I'm actually driving everything through the browser level so. It's not necessarily, the most targeted, thing because if, I'm doing everything on the UI layer and something happens in the backend I'm. Not necessarily gonna get the best feedback. And. Also talk about that more in a minute. So. The next thing is is it reliable. So. We want our checks to, be non. Flaky, we don't want them to be flaky at all and we want them to be deterministic, until. The product changes. That's. The value there that's you know if a product changes, and that check fails that's.
Sort Of triggering us to sort of explore. The system to find out more so. If I end up in a situation where my check fails and I roll my eyes and, I go oh for, god sake the checks failed for the millionth time I better go and fix it only, spend hour and a half trying to fix it to go oh wait. No, the products. Broken, the, check was fine it's the products broken or if. I use that awful. Plugin for Jenkins that makes me run the, test three times and, then pass on the third time if the other two failed I'm, sorry if you've got that in your tool set you need to remove it now. But. That, sort of stuff states to me I don't trust my checks. They are not reliable, to me if I don't trust them they don't have a lot of value because they, should be there to help me trigger other testing, activities, and. Then. We have informative. Is this, check that I created informative, so let's go back to that issue with the not. With a check not being very targeted, so. I'm. Running my run. In that check it's. Interacting. With the login form it clicks, submit and. Back. Ends not working for some reason now, I could tell you for a fact because, I developed, that application, that, I did not develop it very well, so. When you click that login link and the backend failed back-end fails nothing. Happens you just get stuck with the login form that's. Not necessarily, telling me anything about what's. Going on in the backend so. It's. Not very informative so. We want to mean making sure that we give. Our robots. Or automate checks of voice we want them to, give. Us as much detail as possible so, that we can you, know get started, with our other activities so. Whilst. It might not be targeted I could still think about like throwing some logs in there some screenshots that sort of stuff but, not, massively informative. Then. Is it maintainable. And. This is this tends, to be the topic that we talk a lot about in the industry about making, sure that our code is not brittle, that we use. Things like dry principal, you, know name our methods and our functions, well and, the. Fact is that it's clear it's readable it's legible and. I. Would say that. The. Code that I've created I've, sort of followed this or typical. Page objects, pattern. Half. And half I'd say I think it is maintainable. But. It could certainly be improved upon and, then. There's finally the speedy. Now. Speedy is an interesting one because again.
When We talk about speed we tend to talk about it in the context of how fast to our checks run you. Know should we paralyze, them so we throw more resources at, them but. Speed really, should factor in the whole cycle. So Richard and I like to say a failed check is an invitation to explore, and. The. Check that fails is the starter that exploration, is telling, me that something's not right in that part of the product and I need to go and look over there now. Imagine. That scenario in, which the login forms just stuck, there it. May take me an hour, two. Hours, to. Actually, to. Actually, find. Where. The. Problem lies in, the product that's, just two hours of finding the problem before even trying to solve what the issue is so. When we factor that aspect, in the speed is actually quite slow they, don't have if they don't have very good feedback that, I can't react very quickly to them, so. This makes up for an, acronym called trims. We. Were gonna go with Stremme but, we thought that was a bit aggressive and it might you know, encourage. A few consultants, to come in with weed whackers or something like that so a little. Gardening joke there for you, yeah. So trims targeted, reliable, informative maintainable. And speedy so it's. Not very targeted it's. Kind. Of reliable, the system's not changing, in this demonstration, but. It is dealing with a complex system with lots of different things that move and change if. I'm only focused, on a specific part, of the application but I have to turn everything else on those. Are all things, that can contribute, to the. Failing of my check is. It informative nope. Is, it, maintainable, I'm gonna go with yes but I'm sure you probably will all end up looking at my code at some point and have some things to say and it. Is it's speedy it's, not, so. This, pattern, that we see typically, that's creative, with our automated checks. That. We try and do everything all, in one go this sort Big Bang approach full stack check test.
Everything At the same time it's. Not necessarily. Going to give you the value that you're expecting, to get. So. If it's not necessarily, the right thing. For us how, do we go about implementing. Valuable. Checks how do we identify there. Up the opportunities, for them and then, how do we go back implementing, them. Well this is where task analysis, comes in, so. Task, analysis. Richard. Put me on to task analysis, and from, some work by a chap called Rob Sabourin, who, talks, about the task analysis within, the context. Of testing, in, general but we sort of we, realized, that an. Activity that we were doing. By. Ourselves was a sort of a form of task analysis, so. I'm going to demonstrate it's like a model. Of how. We might approach this typically. Once you've done this a few times you start to internalize it in your head because, you start looking at how the system works, and throughout. The layers and throughout the different components, but. For. This demonstration. We're going to show you a visual I'm going to show you a visualization, of how it works, so. What is task analysis, it's, essentially, it's the taking, and, act a tile. Script sort of abstract task and breaking, it down to its different, parts so for example, we could look at the sort of process, of making a cup of tea and breaking. It down into its different parts so we've got the section where we're focusing, on. Boiling. The water, we've. Got the section where we put, the tea bag into the mug and if you're wrong, wrong. You put sugar into it you do not put sugar into tea this, should be your main takeaway, of this talk do, not put sugar into your tea. But. Yeah we can see we can break down the different actions, that we do to make a cup of tea, we. Also could, go even further we can get quite granular, with this we could look at the point in which we turn, on the kettle what. Different apps what, different tasks, are occurring inside that, kettle, so. There's always this balance of trying. To hit, the right level of granularity, and. You. Know you don't want to go to too much detail sometimes, because, you'll, end up with reams and reams and reams of different tasks, and which, become, hard to process but. You don't want to come so abstract, that you end up back in that original position, that you're in. Also. They. Add the milk after, after. The hot waters gone in so that, argument settles, as well. Again. The. Important things to take away is how to make a cup of tea. Okay. So like I said we. Created a model to try and sort of represent, how this task analysis works, in our head so, rather than making a cup of tea we're thinking about how our products, work we. Can look at something like that successful. Login flow like. What did that user do what actually, occurred in the system, and. We could start mapping it out so.
That Could affect the login flow and that, would be something, like the, login form is not rendered correctly, so. All I care, about is that when. I'm presented. That mocking form as a user that I can see in the browser. It's. All correct, and it. Works in a way that allows me to sort of in. Release look visually appealing. So. From. There I'm, thinking about visual check, and. The reason why I'm thinking about visual check is you'll notice now I've added in two, new icons we got robots, and a little magnifying, glass. So. When, we automate, we are replacing, that person, or that thing that is interacting. With that part of the system with. Some sort of robot so. When we use Liam, webdriver we're typically. Removing. The user out of that and getting getting, that tool to interact with the broke browser instead so. We put this little robot in there because that's impersonating. The user, created. A new actor. So. That, robot is going to do. The absolute sort, of bare minimum, here it's just gonna open. Up the login page to trigger off that build login, page component. And then. That's. Going to give. Us a rendered, form view in the browser and, then. The little magnifying glass is at the top there is because because I'm focused, on how the user would interact with the system I probably. Want to assert on that level as well so. A user has eyes they're, going to look at it so, this is where something like visual testing or visual checking can come in. So. Oh. And. One other thing is well I should probably add before I start showing you the toys is that. Notice, how all those other points are grayed out and, that's. Because I'm focused. On one specific area, of the application I don't necessarily need. The rest of the application up and running I don't necessarily need the backend up and running I could. You know talk, about things like mocking in dependency, but, for now all i care about is is that this, specific area is tested and the rest of it we can deal with at a later date and we'll sort of cover that as we start to build all these different checks up so. I'm just, focused, on. Spoilers. I'm. Just. Focused, on how, the. Page renders, so. Let's look our second example. For. This I'm using applitools. Because. It's something I've used commonly. In the passed and I'm. Not trying to endorse any specific tools and stuff these are just things that I feel comfortable with and the. Point is is that. The. The. Point at the point on which I was trying to build that check the area the risk that I'm focused on helps dictate which, tools I want to use so. I chose applitools. The. Webdriver script or the webdriver code Slean web trader code there is just that line I just need to open that page because, the application, is all up and running so, I just focus on that page and then the rest of it is applitools. Doing, it's, definitely. Doing magic, to. Do some screenshot. Comparisons. Okay. So let's give that a run. Here. We go. So. We're still working in that space where we're, interacting. With the browser but. I'm not necessarily manipulating. Lots of form fields, or, adding. In any data or doing any state. Setup or anything all I'm doing here is loading. That page up and doing, a screenshot comparison, so, we can see that it's failed that's. Good I was hoping for that this, is all going very well. Now. If we go back to my, dashboard in applitools we can see that we have a unresolved. Test. And. In here, did. He did he. Little. Highlight buttons because it might be quite hard to see out the back see, if I can zoom in a little bit but we can see these little marks here that, show me that there. Are spin some changes, between the different, visual. Comparisons. So. That. Doesn't necessarily tell. Me how the API is behaving, it doesn't tell me, whether. Or not the front-end is talking to the backend potentially. Depends which API you're interacting with I'm just, focused. Specifically. On how, the form is rendered. So. That brings us to our next one so. This time we're going to go down, a layer we're going to go down into the backend so. Now I'm focus on a risk that the, off API, process. These requests, properly, and sends, the right, months back.
Within That method is felt and I can react to that. And. What we've done here is we've we've sort of. Douve. Divin. Never. Never know we know what the right. Word is for that we went lower. Into. The application, we went further into the backend but we can apply the same sort of rules to. Front-end. As well so. As I said we're using react, react. Is a highly, testable, application. Or code. Base api, sdk but. It's highly testable, so as, we saw demonstrated in the accessibility. Talk that we saw this morning if something, you saw, that we. Can actually isolate these. React, components. We can render them headless lee and we, can then. View, the output of that as if it was HTML, and make some decisions here so. If, I'd have had time I would have loved to put that accessibility, example, in here about how it would work in in the in the wider strategy, but. For now my risk is that I just want to check the lock in hTML, is correct, so. When we were looking at it on the render level we had complained, around not just the HTML but the styling as well whereas, this time I'm just like is the semantic, structure of the HTML correct. So. This is a unit check but this is something that's done in the UI layer. So. For. Our final check. Let's, move over to here. Although. I've got it down to two lines of code I think I would, be taking, the mick if I got it down to one line of code I think you'd probably all chase, me out of here with pitchforks, and flames, or something like that but, all I'm doing here is. Admittedly. Some, setup, is in a different different. File. That's being imported, in but, essentially what I'm doing is I'm using a tool called enzyme, and gest is my runner I'm. Using, enzyme, so I import, my reactor, component whose enzyme, to load the component, in and load. It up well. It loads in into. A small little Dom which, triggers, that whole react process, to turn into HTML, that. HTML is dumped into my login component. And then, I'm using an assertion here where I'm saying expect, the. HTML. To, match a previously, accepted, snapshot, that HTML, so it's similar, to what we saw in applitools, but this time we're doing it on a. In. Terms of sort of some code or some you. Know some string data, so. Let's give that final, one a run, I don't know how well we're gonna be able to see this because it's a dark screen oh. Thank. You thank you lights will make it all much, more intimate okay, so there. We go that's our final check there so. Now we've seen it. Checking. That the HTML is correct and just to, sort of add, a bit to that because that's just basically. A, screen. Rendering. Some staff if, we go to snapshots. We, can actually see the HTML here, on the right-hand, side that's, all being captured. Okay. We've taken, one. Big. Check and we've broken it out into two into, four small ones and, some. Of you might have that what I like to call this sort of software tester, spidey-sense. The. Fact is is that if, we're focusing on specific, areas we're ignoring, other areas, and we. Need to be wary of that, and. That's, good that's good to have that spidey sense because that's telling you that there's potential, checks that could be, identified. In creating other parts of the application that are still targeted, on specific, areas, but. What we can do is we can actually combine all of these things to create a collage, of different checks, and we. Can actually build. Like an create a build pipeline, or. You. Know just a batch script or a bash script that just runs these different, checks in a specific order. And. What this does is this can create a chain of trust, so. If I'm to run my API, check to make sure that the request and the, response is. Behaving. Correctly that's. Not necessarily, the most targeted. Thing if something. Fails within, the service layer but. If I've run a previous, or I executed, a previous, run of unit checks that tell me that the service layer is behaving properly.