Web Application Technologies and Django (Course 1)
hello and welcome to django for everybody one of the things that i like to do at the beginning of each class is sort of say why why should you take this class why did i make this class and in particular why django there are some questions like oh why not flask and i'll get to that so i just want to take you back in time a little bit three years ago i was a happy professor i was teaching we were teaching python for our beginning users in our data mining and then i thought it was really important to learn a second language and so i taught php for our web dev i like php it teaches you like how the web really works i taught sql and i love that and taught about all the good stuff and then we talked javascript jquery and json as well and uh and i had been building that class literally for five years i had been building the graders for it the stuff and and one of my metrics as a teacher is how far do my students get at the end of class and so i look at 15 weeks and what was the last assignment and then that to me is the way i judge whether or not i've done well or not and over the years i try to get the students farther and farther and i adjust the first part of the class to be more efficient so that i can be more effective at the end and so i can put more useful stuff because there's so much in the beginning classes that is just like oh the mechanics of it and you got to learn the mechanics and then you can apply those mechanics but you just can't start off and say here cut and paste these four thousand lines of code it does you no good whatsoever so three years ago i had a very well developed php class and in no not too many words i was told that they didn't want me to teach php for web dev anymore because everybody wanted to do python and a lot of people were actually doing flask in some of the more advanced classes like our mobile application classes so i was told to switch to python sort of told to switch to flask i i've done ruby on rails i've done php which is pretty raw i've done cake php which is a framework i've done a lot of programming with and without frameworks and i just another teacher taught a class using flask and i didn't like how that class turned out i looked at the end of what they were doing at the very end and i thought whoa that's not very much stuff but i started with an open mind and so i actually had a whole year when i was teaching php for one whole year they let me teach it for the last year that's also when i developed web applications for everybody and that's also become a mooc as well which i love and i think a lot of people use it they use it it's really a great mooc and i'm still convinced that learning more than one language is a good idea so you might want to take web applications for everybody after you take django for everybody so i would go everywhere i would go i would ask everyone that i would meet do you use flask or django i mean luckily five years earlier there was like 25 choices in python but by three years ago it really narrowed down to flask and django there's other cool ones like sanic that are specialized but flask and django are kind of like the big ones that everybody not web to pi there were so many of them that i'm like oh man which one am i going to pick and that's the problem with php frameworks is there's so many of them although symphony and php is kind of winning so i would ask people django or flask and people would give me very passionate responses they would say django is the greatest thing ever and flask is absolute trash i would go on to the next person they would say oh django is trash and flask is awesome you know fifty percent of the time they would love one thing and hate the other one it's a it was amazing i really got no trend whatsoever and so about halfway through that i'm like i'm getting no data i mean i'm getting i'm getting half the people love one thing and hate the other thing and so i changed the question i stopped asking whether it was django or flask because they were saying what do you like i was really asking what do you like better not what should i teach and so i said look i can only teach one django and flask and which would you teach first and then which would you teach second what order and then universally everybody just had to accept the fact that django was way better way way better to teach in the as the first class um and so and so i i felt i got really strong data on that so i picked django i wanted to use django and in retrospect it's brilliant right i've been working on this django class for two two and a half years now and each semester i make it better each semester i improve the slides and i do that thing where i try to like compress the beginning and get efficiently done with the beginning part so that i can do better stuff at the end and i'm really pleased and so what you're seeing in this django for everybody's mooc is that two years of careful engineering of the class and i'm really proud you can go to projects.djfree.com and see some of the things that the students have built i'm amazed at how much they can actually build on their own and literally couldn't have done it with php and couldn't have done it with flask django is the only way that students would be able to produce complete functioning websites with a real user interface at the end of a 15 weeks and so that means that to me made django a great success i'll close with i just got done looking at the most recent python community programmer community survey and um i was surprised to see that the number one use that they had of python was web applications and the number two use that they had was data applications which like floored me because i know that python is great for data applications and it's just super dominant part of the reason that it worked out that way is it wasn't like web was twice as much the python is used for so many things and so you know that there was a lot of things on the list and then web just edged out uh data analysis by a little tiny bit you can be critical of python web development and my answer is if it's not perfect it's going to be perfect because people like to write in python and they don't want to write in rust or elixir or scala those are harder to learn and honestly whatever the shortcoming of python web development um it's going to get fixed so welcome to django for everybody i really enjoyed and i've been looking forward to the moment where this be was mature enough and well enough developed to become a mooc and so here we are and i'm glad to make your acquaintance cheers hello and welcome to the first course in the django for everybody specialization the purpose of this course is to in fact make it so that the only prerequisite you need are python skills like you might have taken the python for everybody specialization and finish that and then you can come here and and so i don't want to put a bunch of prerequisites like html and css and sql and so what i've done is i've just covered all the foundational knowledge that sort of uh supports the rest of the specialization in this class we're going to use python anywhere in this first class we're going to install django we're going to use it for some html css and sql assignments and so the idea is to get everybody on the same page when we uh when we finish this course so that we can sort of at that point start digging into learning how python and django work together hello and welcome to our lecture on dynamic web content uh it may seem kind of weird to be talking about building web applications but and to when we start out talking about build building web applications that we're going to talk about protocols and stuff but in a way when you're building a web application if you don't understand the protocol and you just sort of say i changed this a little bit of code and hit refresh and some magic happened um you really are cheating yourself in terms of becoming a really good web application developer because ultimately these protocols are what enable web applications and as you go forward and use frameworks you don't need to know all these things in detail but when you debug your applications you absolutely do and i don't know if you've um had your firefox console or looked at things and watched there's so much complexity going on and and when you're using a web application then hopefully everything works and that complexity is hidden from you so if you have uh some interest in this and want to go even lower to sort of more than network protocols not just the http protocols i've written a free book on networking called introduction to networking how the internet works that you are welcome it's you can buy a print if you want but it's also completely free like every book that i write these and so if we take a look at uh what we're going to really be talking about in this lecture and in this web application course is the technologies the different technologies that are used to in effect produce a web page right so we have a web page over here uh you go to url dub dub data i mean data.pr.pr4.org page1.htm and you know here comes a page and this is like a super simple page but there are so many technologies like how browsers work with javascript and jquery and css and eventually vue and things like that that's a whole set of front-end technologies and then there's sort of back-end technologies like django or flask and databases and we're gonna we're gonna learn in this my focus in this class is less about how to make it all look beautiful although that's important um and more about how to uh deal with the back end so what we're gonna see is these technologies that define the look and feel and the interactivity of your web pages but also then how data and html cs and javascript are served up by the remote server and so the internet here is in the middle the browser is sitting on your laptop or your phone or whatever and it has all these technologies built into it it makes requests and then the server comes back with responses and so the simplest version of this protocol is that you click on a tag and when you click on that tag you get a new web page um now if you watch your debugger console you will see that sometimes it's 120 different request response cycles used to construct the page but other than but in in a sense they're all the same now as we move to things like http 2 there are ways to pull down multiple documents in a single connection but we'll stick with the old http one way of thinking about it in the simplest form the browser asks for a document and then retrieves the document and then shows the document and that is the basic request response cycle so if you look at how this works that browser is an application running inside your computer on your laptop or your phone that is a piece of code and in that it's got a connection to the screen or the keyboard or the mouse or whatever and it's watching for events meaning that you're going to go and you're going to click somewhere and then this application is going to respond to the events so on a web page in the simplest case you got stuff that you can click on and stuff that you can can't and you go find your way through whatever to click on part of that web page and that click is intercepted by the application whether it's chrome or firefox or safari running on your piece of equipment and then that's that application then opens what's called a network socket a socket across the network to a web server and sends a request and that request as we'll see in a bit is a specially formatted request it has a get command and then it has the url that it wants to get from the web server and then the web server does a bunch of work inside itself a bunch of work inside itself maybe reading disks files and running programs generating what it is that you want and then it sends the response back on that same socket connection so there's like this socket connection through which all of this stuff is viewed and so the html is one that kind of the basic thing that comes back as we go further we'll see that it's not just html it comes back but other things as well come back and so when the browser receives that html then it says oh i know what html is i've got this header one tag here let me let me change the color of this thing so you can see that see it a little better yellow is way better okay so when the browser um receives this page it it sort of sees it and it looks at it and then it looks at all the html and so part of this course is to understand the syntax of html um that itself is a you know how to build good html and css and how to make it look beautiful but ultimately something in that html plus it's css plus its javascript leads to a visual display of the page and then that's the thing that you see and so this is the request response cycle so up next we're going to talk about what these network sockets are or the things that fundamentally underlie this request response cycle so network sockets are like phone calls for computers and literally in the 1960s when they were designing the concept for network software that is the concept that uh they came up with and that is that you know in a sense you know we make a call we talk on the call and hang the phone up and as as computers we're going to share data and some computers were going to have data and other computers were going to want data they thought oh the way to do this is not necessarily have a permanent connection to that data because there's going to be so many computers and so many uh so many sources of data and so many computers that want to consume those data and of course in 1960 they had no imagination that there would be billions they just thought there would be ten thousand and they thought that was a lot but they came up with a protocol that basically said when you you you don't have continuous permanent access to data you actually make a phone call you know where you the data is at you kind of say you dial up you grab the stuff you get it back and then you let the connection go and so it means that with billions of computers each talking to each other the network isn't so complex and so there's this connect talk disconnect connect talk disconnect and it's the way the phone system in our world um scales to all the people and it's the way the internet scales to all the people that want to use that internet so this is made inside of software using an abstraction or library called sockets and sockets really are computer phone calls you know where you're going to call you can start the call wait for them to answer once they answer there is a two-way communication it's kind of like a file except that you can simultaneously read from the file and write from the file although it's important for the two cooperating pieces of software to know who's going to read first and who's going to write first because it's like if you know when you pick the phone up after it rings we all know to say you know hello um and that's that's kind of a protocol who talks first as a protocol like or and then the person goes like who is this and then eventually conversations happen pretty naturally you know somebody talks listens someone else talks listens and that works but there's there's complicated protocols and you can think of this not necessarily as the browser on your on your laptop or phone as talking to data on the server it actually is talking to another program so even when it's just retrieving a file it's talking to an application on that server and that application is reading the file and sending it out and we'll see that eventually how we have to do that when we're doing things like uploading pictures we upload pictures and then we have to actually serve them back out we can't just say oh the pictures are there you can grab them but sometimes not everyone can see every picture and so part of the software application decides if you are allowed to see this picture i will send it if you're not allowed to see the picture i will not send it and so that's where it's a piece of software that you're talking even if you're looking at a picture sometimes that software is really simple and you barely notice it but there is software in the loop it's not like you just go straight to data it is really two pieces of computer software talking to each other and we use the internet as that intermediary to allow that conversation to happen so if you read my book about the tcp you'll learn this in super great detail but every computer has an ip address it is a number there are two kinds of addresses there's ipv4 and ipv6 ipv4 version 4 is the one that's got four dotted numbers like you know 142.16.42.14 but within each of those they have what's called a tcp port number and because this whole internet thing is applications talking to applications we need to have a good way to know which application we're talking to and that's where tcp port numbers come in and you can think of this as like a phone extension uh you know so you have a phone number and then you have you know extension 1436 and so tcp port numbers are like that so we can not have a specific address for every application but an application that has an address and a port within that address or a phone number and an extension within that phone number and so you can look these things up port 25 is used for email service server communication in the old days we use this thing called telnet we'll talk about that on port 23. and so the the
client is on the right hand side and it might be it might be a computer where you've got a keyboard and it's you or it might be a server that's got some mail that's being forwarded from somewhere and they all talk and they connect to an ip address in this case 74208 28177 is the ip address of this computer and then port 25 is the extension where one computer sends email to another computer and each of the arrows in between them is a separate protocol so if you're talking to an email server you expect to talk an email protocol and if you're talking a login server your login protocol that's kind of the simplest one or a web server web server can be on port 80 if it's unsecure preferred of course is https which is on port 443 if you have a mail application like say thunderbird that's reading gmail it talks to ports 109 or 110 or a couple of other ones that are ways for a mailbox application to talk a protocol to a server-based mailbox to retrieve the new messages or delete a message or view a message or whatever so there are these ports on servers that have protocols and then we can build software on our clients that talk these protocols the one we're going to be playing with the most in development mode is port 80 and the production mode port 443 for the http for you know web documents basically in web application documents so up next we're going to take a quick look at the hypertext transfer protocol i also want to call it hypertext transport protocol http and that is the protocol that we talked to port 80 or port 443 hypertext transfer protocol is the protocol that browsers use to talk to servers now we've used it for all kinds of other things because it turns out that it's a super simple and super elegant protocol this goes to a long time ago this protocol was invented in 1990 which is like going on 30 years ago yeah going on 30 years ago now or more and there were many protocols you would have one piece of software that and their server software and client software you had to match them up and then you talk to the right port with this server software and one of the inventions of tim berners-lee and robert caillou at cern was that we came up with this notion of instead of lots of pieces of software we would have one piece of software called web browser and it would be multi-protocol and so we have this thing called the url or uniform resource locator and we just sling these around we just use them but these things this was a itself an amazing innovation in 1990 because in the old days it'd be like use this piece of software and talk to this address so the url captured three really important very separate things the first thing it did is it allowed for multiple protocols now the one that you mostly have seen is http or https but that's a protocol ftp colon or male 2 colon might be the other ones that you've seen and then a host and then a document and that host is a domain name which is a nice symbolic way to to get a server address and that that eventually resolves to you know an ip address like 141 206 14 22 or something and then document within that server and that slash page1.htm is the document in the server and i'm just saying before 1990 there were many protocols and we did many things but after 1990 this one protocol http found its way into so many awesome solutions that it's by far other than email the the dominant protocol that we use on the internet it like i said it was invented at cern by tim berners-lee and robert caillou to retrieve documents and images and and as soon as they built it because it was all the other protocols that came to most of the other protocols were kind of complex because we're computer scientists and we make complex things and we love complex things and we're very very good at complex things um but this was you know when the web was being built tim and robert were not like we're going to make the greatest thing ever we're just going to make a simple thing so they made a simple thing but it turned out to be the greatest thing ever because different engineers would be like oh wow i can use that a little different way and it was the grounding of it was super simple and the basic concept was you connect to a server you figure out where it is you send a single command with a bit of extra data and then you get back something a document it could be an image it could be data it could be html and it was really amazing right and so the sockets the underlying sockets are the things that make the phone call http is what we do once this phone call has been established so one of the things that made the internet so successful starting in the 1960s was a radical openness a very we're all around the same campfire feeling uh very friendly respectful uh that a lot still allowed criticism there wasn't like one super genius that designed this or one company that designed this this was a collaboration and the collaboration was around a set of open standards and meetings to help build these standards and they were called rfcs they're called request for comments and there was a group that built these called the internet engineering task force the ietf and the request for comments is a fun notion we're looking here at a september 1981 document which is 40 years old about now and it's called a request for comments and the idea is as engineers we might have a design that you know everyone thinks is the greatest thing ever but we should always question no design is perfect no design is beyond question no design is beyond commentary and so even when they're 40 years old and used worldwide and dang amazing we still say there is room for improvement and so this talks about um i think this one's ip and there's an ipv6 there's another one if you look for uh this rfc that we're looking at here there'd be ipv6 and you'd start reading it and it's pretty dense reading but as an engineer it gives you all the details to build a router or build a piece of software or build whatever and these are 100 100 percent public documents and this the idea was is that no company because literally the companies that were around in 1960 if they had said burroughs or sperry owns the internet or digital equipment owns the internet they're all out of business because they didn't make good choices but the market was able to make good choices and because the the commons were these open standards and open specifications new companies like sun microsystems and new operating systems like linux could come in and even windows could come in and participate as full participants because they just read these specs and all of a sudden they're interoperating with everybody else this is fundamental and foundational to how the internet works and i'm not saying you're supposed to go read them but it's just fascinating to understand that the entire technological infrastructure upon which the internet is built is free license free royalty free and you could build a brand new gadget and you could hook it to the internet by reading these specifications you don't have to pay a developer's license or nothing it's dang it's cool sorry i'm just a little bit too excited about that so let's take a look at one of these in particular now this this is probably supplanted but we'll look at rfc 26 16. they go up in numbers and so um this is http hypertext transport transfer protocol and if you wanted to build a web browser or a web server you could read this document go ahead read the document and if you read down far enough you'll say oh this is how you make a request blah blah blah it's got a header and it's got like a carriage return line feed and then a message body and you got to do this and this is what it looks like this is how it works right and so at some point you're deep in this document and you're finding out and this page right here of this document how your browser is supposed to format it and this tells both the client and the server what the rules of the protocol are now it's a lot easier to just for me to tell you so you connect to the server on a socket usually port 80 or port 443 and then the client is supposed to make the first sound it makes first request it sends a line with a carriage return line feed at the end that has the characters there's other ones but get g e t space a url space and then a protocol and right now i'm using this hdb 1.0 because i can do it by hand right and uh and so you ask for a document and then you can optionally send some header information uh things like what language this browser person at this browser prefers you know what are the capabilities of this browser what version browser we have so there's a series of things you say give me this document and here's something about the browser that's requesting the document including information like cookies that are being said now so this in case you want to know who is doing this request because you've logged somebody in and set a cookie to indicate that they're logged in that's all sent on this get request but we'll not send the headers there's actually incoming headers and then outgoing headers we'll see these when we start looking at uh you know firefox's debugging console and so this is an entire interaction that you can do on your laptop now i'm not going to do it for you but i would note that the program telnet that this uses has been removed from most computers it makes me sad they think of it as a security hole because it doesn't use encrypted connections like port 80 is generally is not an encryption cryptic connection but um ultimately you can install go go search on your favorite search engine and find out how to install telnet but if you get in a command line whether it's a windows command line or a mac command line or linux command line and you get telnet properly installed you can type the following command telnet data pr4e.org 80. then hit enter and at that point so telnet originally was used to log
into computers but if the server we're talking to in the protocol that we're talking to is simple enough and in this case http 1 is simple enough for us to be able to do this we're actually talking to the server because i remember i said that these are applications they're not you're not your browser is not talking to files your browser is talking to an application on a server and the server might give you back a file so it's really talking to a piece of software and we'll see this in more detail in the next section and so what will happen is because the http protocol requires that the client speak first we have to type now if you can get this working and type wrong things and you know type couple enters new lines um you will see that you're talking to a piece of software and it will say you have violated my protocol you're not talking to a thing that has any online help or user interface or anything like that you're talking to a server and your job is to request some data using the proper format because you read the specifications about the proper format right so in this particular one we just it's easier to cut and paste this because some of these servers if you don't type fast enough they'll be like you're not a software you're a human quit playing with me like you talk to facebook or something and you you can talk to facebook and see what happens on port 80 um and it will time you out really fast so it's good to have a cut and paste buffer so you paste this get command in and the other thing that's important is you got to put an extra blank line and this extra blank line is to indicate that you're not going to send any of those headers if we were going to send things like what language we would prefer or what formats we would prefer as a browser you know the browser has certain languages and other configurations and login information and that can get sent up to the web server along with the request for the document and we would be able to type them in here but we're not going to do that in this very simple we're just going to say enter which means no more headers if you were putting in headers you type header header header header and then enter on a blank line and then the server would know that our request was finished but we can just ask for a page and then what the server does is it sends us back two pieces of information first it sends us headers the first header is http 1.1 200 okay that's actually a status line now you'd have to go read the documentation on how what that works but for example 200 means that you got a document another one that you might have seen is 404 it might say 404 not found so if you go to a web page and it doesn't exist something will save 404 in your browser usually unless it says here's a search box go find it but you can there's a status and so you might the thing that you're typing page1.htm may or may not be on this server if it is then you'll get a 200 okay and the data or you get a 404 not found and then we get some response data some header response and this header response looks pretty much like the same format of the data we would have sent into the browser if we were sending things like that in there it here it's what the date is what the server we're using last modified and one of the important things is what kind of content are you about to see content type in this case is tech slash html which means the thing coming next and then there's a blank line and the blank line is our indication as we're reading this data that we've finished the header which is really metadata about the page we're retrieving and the actual page itself and so the rest of this page is the actual data and we know the format of it because we are told before the page starts this could be image slash png this could be xml application xml and it could be json it could be like anything and then the browser is supposed to and if you this was an image it would be like garble garble garble garble garble garble right it would be all garbled stuff that we can't really see but the browser knows what a png or jpeg looks like and it shows it to you and it basically uses this text html to tell you what's going on and so this then your browser has read all this information it has both the metadata and the data about the retrieve page and then its job is to sort of produce a pretty rendered version of the page and show it to you and that is the request reset cycle again you don't have to do this you can just sort of believe that it's done here but if you want to install talent on your system it's not a bad thing to install telnet it's kind of a classic fun thing it's far less useful than it was in the early days because we used to use telnet to test everything now we've got like browser developer consoles that are way easier than uh than than telnet but um it's kind of a cool thing feel free to do this and you'll be like i know what's really going on and um so i'm a big fan of using the console of using the terminal of using text-based interfaces i think they're actually way more accessible i think it's awesome and and in a way all these fancy graphical user interfaces uh distract us from the simplicity of what's going on inside of computers because you think well where's the button and the answer is oh there's probably some command inside this computer that does what the button does so i have a former student of mine who uh actually wrote the the scene and i think maybe it was made matrix two i think this was matrix two uh yeah matrix reloaded and um this scene where trinity is breaking into the power grid and she is using the console and it turned out that the way this scene was written is it was written exactly as a hacker whether it's a hacker with bad intent or good intent because there are hackers with good intent who are trying to break into your system so they can tell you for you pay them to break into your system so they can tell your vulnerabilities but you can take a look at the analysis of this little scene um and just this sort of is just one of the cool things about me teaching people about how to use the command line and increasingly me teaching people about how to use linux i really think that it's okay to know your mac and windows command line but really increasingly the world in the server world and we're starting to work on web servers in this class and you might as well start learning some linux because other that's an important skill just writing some java code or ruby code or django code that's a skill but knowing how to start the server debug your application find log files etc in the command line is really important in real production systems because we're going to play in some simplified environments to make it a little easier but those simplified environments go away once you go into real production in a real job and so i'm really into teaching you linux command line etc etc so up next we're going to show you in python in effect as few lines as i can possibly show you how a browser works how browser sends the http protocol how a server reacts to the http protocol and returns documents so that's what's coming up next so now we're going to build a very simple browser and server in python okay so import a library import socket and that just like you know import math or import whatever that pulls in a library into python sockets are built into python it's very cool um and it's very very simple so we're sockets kind of function like files but they're files that we can send data to and receive data to but when you're using a file in python you've got to first open it and so the opening of a file of a socket is a really a two-step process first you create the socket and that socket sort of lives in your computer as a as an endpoint that's not yet finished it's it's the end point that you're going to send and receive data to inside your computer but then we have to actually make the phone call so so socket dot socket and don't worry too much about this syntax we don't change it socket.socket basically says make a phone that's what we're making and then my sock.connect
that second line is dial the phone and we're dialing the phone to a to a domain name data.pr4e.org and a port in that domain name so that is the call now if we made a mistake right if we had we were dialing and not talking to the right server and or that server didn't exist this connect would blow up meaning we'd make a phone call a data phone call to a server that doesn't exist that line was is what's going to blow up but as you continue down to the next line if you continue down to the next line if it gets that far now you put try and accepts around this if there was but i don't want to keep this really short so i'm not putting in all the stuff all the error checking so connect might blow up socket.socket won't blow up because it's like unless you don't have networking at all in your computer but it's unlikely these days um but my sock connect is like dial the phone dial the data phone to this endpoint and it includes the port the port 80 that we talked about before and now we need to send a command and in this command it's exactly what we talked about before it's exactly what we sent by telnet it's a get followed by a space followed by our url followed by a space followed by a protocol http slash 1.0 and then we hit the enter now it turns out in network world you actually hit two things it's the
slash r slash n is a return and a new line um and you have to hit it twice the enter at the beginning the first line and then remember we had to put that blank line in because to say no headers now we would put in between here we put all our headers in there if we were going to do that but we're not going to do that and then you have to use encode and encode means that the data sent across the internet is generally sent in what's called utf-8 inside of python the data is in unicode so that you can put literally any character in a python string and then preparing it to send it out across the internet we kind of encode it into the more dense uh uti or you know more compressed utf-8 format rather than we don't send strings across the internet in unicode in general and it's very rare but it would you'd have to control both ends we want to send them in utf-8 and so the string get blah blah that's in the unicode string in python and dot encode says convert this into a utf-8 string so command cmd is a variable that has a utf-8 string in it and then we say send now remember i said that you can simultaneously send and receive on these sockets but the protocol tells us whether the first thing we're supposed to do is listen or talk and we're the browser in this case and so we're supposed to talk first so we send the request out and that's just like you typed it in telnet because in effect you're scripting telnet telnet we liked telnet in the old days because it was kind of like a socket i could connect to any host any port and type things and see if it's broken or not um and what and the protocol says once the server has received the first line any headers and then the blank line then it's to return it so then the next part of this is we're going to have a little loop so the protocol tells us we are supposed to receive data until the sockets closed and at the bottom of that telnet you saw that it sent a bunch of lines like four i think and then it closed the that closed it that's what's going on here we are retrieving data and then this receive this actually waits if the server is coming slower the network is slow this piece of python will be sit sit sit sit oh got a little bit got a little bit up to 512 characters it will it will wait until it's gotten 512 or it's that it's got it's done and it's got 100 or so but the receive is a blocking waiting operation in python says give me up to the next 512 characters now usually the network's so fast that the python doesn't even wait because this operating system already has it but we get it and if the we get uh no data whatsoever that is our indication that the network connection the socket has been hung up on closed by the remote server and away you go so that's why we are going to break if the length of the date is less than one and if not we're going to uh simply print out the data but we're decoding it and that's because the data that comes in is in utf-8 but the print statement wants to print uh unicode and so decode says here's a variable that has utf-8 in it convert it to unicode for printing inside of python so the because this data is externalized to us we encode it before we send it and we decode it when we receive it before we use it because python inside itself doesn't use utf-8 python inside itself uses unicode which is awesome and a whole separate conversation i'm just kind of bringing that up to remind you how important it is because we're not going to write much of this in python really and all this stuff is taken care of for us in web servers when we get to that point but it's just a good time to review python's strings and unicode and encode and decode because it's really important so if you run this and you've got it right it's it runs uh pretty simply let's get the color back get back to yellow here so if you run it in your system python socket one dot py um it will all the work is done up here before you see your first output and then this all this output that you see is just this loop running and what you see is really the exact headers that are coming from this server and then you see a blank line and then you see the actual page that's coming out and you don't see it but at the end here the connection gets closed and you break out and you're done and then we we close our end of the socket the socket the far end was already closed that's how we know to stop reading data um and then we hang ourself up so that our local system does not end up with a bunch of half half hung up phone calls and so that's basically how we talk to a socket in python now that was literally 10 years ago how i would debug applications i would like telnet or write a little thing like that and dump all the headers and that got really tiring really boring really tiring and so uh browser developer mode is going to be your friend in a way browser developer mode i'm going to have a separate video on this browser developer mode lets you look at the sent headers the received headers the receive data the sent data what url it went to and it's super awesome especially because a lot of web pages have like 40 to 150 request response cycles so you don't want to bug them all by hand you just let the browser do its thing and say what was that second one and what went wrong in the second one i spent a lot of time when i'm debugging web applications uh using the developer mode and that's why i have a special little video that gets you familiar with it i tell you by the time you've done web stuff for a little while you're going to know your browser developer mode so next we're going to talk about writing code in the server to respond to these requests so we build a browser in python it's time to build a web server in python so remember that what we've got here is we've got you know the browser is an application and we've we've made this a python application now that sends this get request so we we have effectively like i said built the world's simplest browser and that sends a get request the server does something and then it sends the response back and now we're going to kind of change this right we're going to assume the browser exists and it's like okay because we're going to be the server folks right so now we're going to look in greater detail at what goes on in the server and so here is a very very simple web server it's a few more lines than the uh than the uh the simplest web browser because we have to put a little bit of error checking in with some tries and accepts okay so let's let's walk through what this code does okay um let me let me change the color again to black if you're from microsoft and you can tell them to make a key to change the color on the on the scribbler i'd greatly appreciate it oh and then i went down a page too okay so we're going to pull in some more stuff from this from the socket we're going to pull in some stuff from the socket right here and um so we're going to make a function called create server that we're going to call right down here and it's going to print out how you're supposed to access it and then start the server now the whole idea of the server is the server is woke up to wait for incoming connections so the server already exists so when you start talking to a web server that server's in that computer already the server software is already running registering its interest and incoming request so that's what we do so when this python program starts it's going to sit there and wait in an infinite loop for incoming requests so the first thing we do is we're going to make a socket this looks very much like the time we made it's an end point it's not actually making the phone call we're remember i said this socket call is making the phone so we're saying we're going to make a phone and it's up to us later to decide if we're going to make a phone call or receive phone calls and then this next thing is like to connect except this is i am willing on port 90 to receive the phone calls now it turns out there's only one program on port 90 that can receive phone calls so this might blow up you might try to do this i mean port 9000 and if you run this twice and you can do it in two windows on your computer the second one will blow up and say you can't have port 9000 because another piece of software has it but as long as there's only one running away you go and that's part of this whole try accepting because it's if you run this twice it's gonna blow up the second time because you cannot receive phone calls on this server on port 9000 with two applications one application gets the phone calls that's what socket listen says the five on there says um dear operating system uh if i'm busy handling one phone call uh you can hold on to four more and cue them and then i'll come back and get them for you and that you're asking the operating system to cue incoming calls don't just say you're busy shut down if you didn't say this listen five if you're busy writing the data for phone call one and phone call two came in and you weren't ready for it right that instant it would just deny the phone call so that's what listen says is dear operating system hold hold those temporarily i'll get back to you which is how it works and then what happens is we go into this accept now this accept is i'm at the phone i've registered what my number is and what my extension is and i'm ready to pick the phone up let me know so this accept is blocking it stops and it just sits there and and it can sit there forever you know just literally forever if nobody calls it nothing happens the next line never exit never runs until you blow it up or the server goes down so that's accept is a blocking and the reason we do this except well we have to establish the phone call first right so then this next line runs only when a phone call is received so that means that we on the browser side we already connected you know we made the phone call now we haven't seen any data yet right and so this is just now at this point somewhere out there there's a piece of client software that has done a socket.connect and we have done accept and our accept has succeeded in the server the connect has succeeded and we are ready to talk so the phone call has been made now the question is who's going to say hello and this is where the http protocol solves our problem the server knows that the client must speak first so it just does a receive now you'll notice the receive and the send are the same function calls because it's a two-way thing the whatever the browser is sending the server is receiving and whatever the server sends the browser receives and they can do it simultaneously if they can figure it out but usually you kind of like say it's your turn to listen and my turn to send and then i'll listen and usually they kind of go back and forth in this there's only one step you the server listens gets the get request sends the data back and um so then we're going to uh read some data and we're going to get 5000 characters we're get the whole thing remember it's one line it has a get request in it it has the optional parameters in it and so we're going to split it now again we've received it as utf-8 we've got it decoded for utf-16 um i mean not utf-16 unicode inside of um python and then we split it based on new lines because the get request is on one line then header header header header header and then the blank line to tell us that we're into the headers and we're only going to take in this we're just going to look at that first line we're not going to do anything with it most most servers actually like look at the line to figure out what document to send it's a very simple server it sends the same document no matter what the url is all we do with the url is we print to it print it out to prove that we got it and then we construct a response and in this response again go back to rfc 260 2616 and it will tell you what this response is supposed to look like it sends back a 200 okay remember this is all the stuff we saw probably i cut and pasted it from a working thing we tell it that we're sending it back in uh content type text html the character set utf-8 remember i was telling you that a blank line and then some html html body hello world body and i'm throwing in these slash r slash ends which is the network's version of new lines and then you will notice that i encode it before i send it because it's unicode inside of python and it needs to be sent as utf-8 because the phone when you're in a socket the thing that's going across the socket almost always is supposed to be utf-8 so we encode it and and then because the protocol says so as soon as we send the data we close the connection now remember that the client had to close the connection too so we close it it's half closed and then the client gets all of its data gets the indication that it closed knows it's finished all the data and then the client shuts its side down and so there are two hang-ups it's kind of like a phone call the person hangs up and then click you hang up now there you can't really keep talking but you do want to have both sides hang up and that shutdown is what's going on and then the rest of this is all try except for various things so we clean up our socket so we don't have to restart our server when our software blows up and so the rest of it's pretty simple and then so this while one the first browser will talk and then it'll send some data and then go back and wait for the next phone call and that's what's going on the next phone call comes in it doesn't get sees the get request it sends the data back then goes and asks waits for another phone call so this is an infinite loop that sits and waits for incoming phone calls and it you know it answers the same thing so it's like you call this phone number and it goes and you go i would like a pizza and it says hello world and then click and then you call it up and say i would like a car and it says hello world click and then you say i would like to register for this class and it says hello world click so this is not a very flexible server you will build far more flexible servers but it's the one i could build and fit on a single page in a slide deck so to run this if you uh you know you download this code and it's tiny bit of code and you run it so you go somewhere maybe your laptop maybe python anywhere i don't know if it works on python it probably doesn't work on python anymore because you can actually talk to ports but let's just run this on your laptop and you you start the server up somewhere get a hold of my sample code or download it from here server.py and it's telling you this is just output from the server and this is the moment at which it's waiting right it's waiting the server could wait for hours at this point but i just give you this in a print statement so you can copy that and put it into your browser so you paste localhost 9000 into your browser and then the server is printing that it got a get request for the slash document from this browser that's what the browser send i mean we're talking from firefox browser to the little web server that we just built now there's another a second get request that you don't even know because you didn't ask for it and that's because the browser is built in to ask for a url called favicon.ixco so that it can make an icon and that icon ends up in the tab or wherever in your browser so that's like an icon for the website so you didn't ask the browser to do this but it's a thing browsers do and you'll see when you're looking your debug logs when you're building stuff that it's a favico and so this this happens like blink link and now it's waiting again right and you can hit refresh or you can go start a different browser and you would see these get requests every get request you're going to see uh in your browser and your server so it's uh so it's working and it works really well and you're welcome to go ahead and play with it and try it all out so now we're going to do is build a simple web client and that'll talk to our server so the other web client we made talk to data.pr4y.org this one is going to talk to yourself so it's this it almost looks identical right we're going to make a phone call we're making a phone we're going to connect to localhost 12701 is the ipv4 connection what's called loopback right to the same host we're talking to port 9000 because that's where we've got our web server running that little web server we just wrote and then we're going to send a get request a valid gut request to this thing http one zero and then two new lines and we're going to encode it into utf-8 before we send it then we send it and then we just have a loop to print it all out and then when the socket gets closed we hit a break and then we close our end of the socket so the server clicks the phone and then we're know we're done with our data but we've already printed it out and then we hang up the phone on our end to kind of clean everything up all the way in between everything else so this is a very simple web client and so now you can run the web server right just like we did before and instead of talking in a browser we run this client and this client gets a header of http one okay it's content type html it gets a new line and then the body and we're done now i mean it's not a browser but it is client that's talking http and so now you'll notice that this server sort of sits and waits for incoming calls you could run this over and over on the client run the client over and over again right um and then you would see more requests but in general the server is destined to just sit and wait for hours potentially until a client uh connects to it and then at some point you'll just abort this server and then if you ctrl c or blow this up somehow this will fail the client will fail and it will fail it will fail in this connect code so if you want to play with it try to run this client without running the server and you'll see that the connect which is the part of actually making the phone call this will continue to work which is make me a phone on my current computer and then connect is make a call to the remote computer that will blow up and then if you wanted to play with this you could put some try and accepts around it say remote computer is not responding on port 9000 that's if you blew up you could put a try and accept around that but i'm not doing it because it don't fit on my slides so here is an even simpler web client that we can use and you again you can get this code from my sample code um we don't talk this we there's a a higher level thing that so the the previous client talked to sockets so i could show you the low level but because we spend so much time talking about to urls we then have a url lib so i'm just going to use url lib i'm gonna and this could be a local host or it could be whatever i'm just going to make a four-line url call and so i just do a url open of that same url now we're operating at the http level because url the concept of url is not a socket concept a concept of url's http concept and we just get an effect what looks like a file handle to us and we loop through it and print it all out so we run our server and the server sort of says this and waits and then we run our client it basically hits our server and we can see in the server window you got to do these two things in two windows right you got to do these things in two separate windows so in the server window you'll it'll sit and wait and then in the client window when it runs you'll see the client will make the request to the server or blow up with the server not running and the server will see the get request and then the data will come back to the client so you sort of do this in in two things and you can run the client over and over and over again and you'll see every time you run the client the uh the server sees it responds and away you go and again our server always says hello world no matter what which is delightful and classic it's a classic so you actually can um watch when you're running django there will be a way for you to see when you're running django locally and you start the server up this managed three don't you we'll do this later manage pi dot run server and then you talk to it and you will see all the get requests and so django when you run it local you can debug a lot of what's going on so you see the you can see like the favicon that it's asking for and that is you know five or six http requests to produce that little django webpage and so that that will be something that you will uh learn how to do and so so that sums up what we're doing in this uh you know at the beginning you look at it very simply pretty soon you'll just be looking at the developer console and it'll all make sense to you i want you to be able to know that you can dig as deep as you want to and understand all of these protocols that are going back and forth okay cheers hello and welcome to another walkthrough for django for everybody in this walkthrough we're going to talk about the browser debugger console um and so this is something that in the old days was only in firefox but eventually everybody built it in and used to have to build a plug-in and um and it shows up at different places and so in firefox it's here under tools it turns out that you can kind of go to any of these things because these are the you know they're just different tabs here in chrome we do view developer javascript console and then you see a similar kind of thing here right so so i'm going to do it in uh in in firefox here and so you can see things like you can see the document object model you can see the javascript console this little error hardly matters it's because it's such a simple page you can actually debug javascript code you can watch the network you can play with css etc etc etc performs cookies are in here so there's a lot of things that you can do um i'm going to start and show you right now we're at the very beginning class i'm just going to show you some of the most basic things here i'm going to clear this out um and then refresh the page and so what you see here is you see each request response cycle right and so i i came up here as if i typed this let me clear it again so you see i'm going to type this i'm going to press enter and that tells the browser to go retrieve this page so it connects to the server www.chuck.com
and gets the document page1.htm this favicon.ico that's basically getting this little favicon right here that's the little thing that shows up in the in the tool in the tab bar and so websites can have a file and later we will actually make a favicon but what i want t
2021-09-28 12:56
