AI for Developers: Pair-Programming with ChatGPT to Ship Faster
What if we could run an entire software project with the help of AI from the requirements to the actual coding? So what I'm trying to do in this video is discuss with the chpt and gemini. This is going to be split into video and uh there will be part one where I will use chat GPT and part two where I'm going to use Gemini. For both of them I paid the subscription. So I have CHBT plus and Gemini advanced. So we're going to see how they can help us during the entire software development life cycle and how we can do a better job in a faster way. If you're new, hi. I'm a software
developer and I help software developer with productivity and ADHDX. If that's you, hit subscribe so you won't miss the part two. So in this video, what I want to do is building an application is a back-end application and I want to try to leverage AI in all the areas of the SDLC. So from the beginning where we capture the requirements to creating documentation, creating the stories and the actual implementation. So and today we're going to use specifically chat GPT plus. So let's start with the simple prompt. I'm going to select as a model
the 03 which here says that is used for advanced reasoning. So what we want to do um I'm going to start with this and we may probably end up so this is not scripted so I don't know what is going to happen in this video. What I'm going to do I'm going to start with the um series of questions and brainstorming with AI and I have a clear idea of what I want to do. So we're going to do the
requirements then the design of the technical implementation the documentation story refinement and in the end we are also going to try to get some estimation and the actual code for the implementation. So let's see how we can start this. The project I want to do is to build a backend application for file uploads. So it's going to use as a stack AWS S3. We're going to use Java as our programming language. The other
thing I would like to have is security scanning. So every time a file is uploaded, I want to have some sort of security scanning being applied and then the files should be marked or moved into another bucket so we can safely access them. Okay friends, this is Ando from the future. I just finished a good part
of the project. Probably I was very optimistic at the beginning. I wanted to build everything including the code. So
what I've done so far is I managed to create the entire requirements that are needed and the help that I had is massive. So I didn't even start the code but the information that I have the documentation that I generated is already extremely useful and in a real scenario could save weeks of time just this activity because just keep watching because it's quite interesting. Okay, so let's start. I'm going to start with um
I want to design a solution for file uploads. The files are uploaded by our web users to our website. I want to use S3 with resigned URLs. So I don't want to give too much deter. So what
we're going to do here, we're going to ask for um support in creating a requirements document. So I will continue saying help me in defining the requirements for the S3. There's an error here. Help me in defining the requirements uh for this project. Ask as many questions you want. Make sure this
is a conversation. So, please ask questions one at the time. And the other thing I want to give uh this AI a little bit of context. You are a software developer. You are a senior software
developer and we work in the same team. We want to design a solution for file uploads. The files are uploaded by our um web users. I want we want to use we want to use S3 represent help me in defining the requirements for this project. Ask as many question as you
want. Make sure this is a conversation. So please ask questions one at a time. Uh I think this is good enough. Let's see where we go with this. What kind of files do users need to upload and what's the maximum file size we should support for each? Okay, that's a good question. Um files should be max 100 meg. And the
type of files we expect are text, uh, PDF, um, XLS, and images. Yeah, that's good enough for now. Do users have to be authenticated logged in before they can request the presign URL? Should each uploaded object be linked to a specific user or resource in your system? So, let's skip the authentication requirement. What I'm thinking here is that this is a backend service that is invoked by the website back end. So it's not directly exposed
to users. It will be exposed only to our backend service. So so this means the backend service we're creating won't be accessible directly from outside but will only be accessible by the back end of the website. So it's only internally accessible. I'm going to say um no the service we want not starting. No. So do users have to be authenticated? No, users um will not need to be authenticated. Actually we can simplify
this entire thing by saying users don't need to be authenticated. Um but the other question which is more important should each uploaded object be linked to a specific user or resource in your system? I think for this one user not need to be authenticated directly with our service uh but we need to know their I'm going to simplify this by saying their ID once a file is uploaded how long do we need to keep it in S3 forever 90 days until the users deletes it and are there any automatic archival deletion rules we should enforce another good ction. So, we want to keep the files for 90 days. I'm curious to see if it's going to ask question about security and the vulnerability scanning and other things that we want to do after a file is uploaded. Oh, and user back office
stuff and an automated job etc. needs to download and view it and should we issue time limited pre-sign get your rares for that access or do you have another delivery mechanism in mind I love this because that's exactly what I was thinking so what we want to do um we want to have an automated job will pick the files and we'll do it by using the pre-signed URL the users um that upload the file won't be able to download it. Is this relevant? Let's have this information. Let's see how now
the way the jug will pick the files and we'll do it by using the presign URL. The the get pres URL. The users that upload the files won't be able to download it. Okay, let's see what happens now. Do we need to run any
server validation on malware? Wow, fantastic. That's what I was looking for. And I didn't specify in the original prompt and it it's asking the right question. So do we need to run any server side validation or malware virus scanning on each file after it lands on S3 before the automated job picks up? Yes, we need to make sure the files are safe before we access them. Okay. So now we want to do this. What I would
like to do ideally is to once we know that the file is clean, I want to move that file somewhere else where we can access it. So it can be on the same bucket with a different name or can be on a different bucket. I I don't want to hint anything. Let's see what happens. Roughly how many files do you expect users to upload each day and that big burst? So we can size the S3 bucket policy, scanning workflow and any Q lambda concurrency limits correctly. Uh, let's say pick person. Let's say 100
files. I'm going to be generic. We have 10,000 active users on our site. Uh, we don't know exactly the usage at peak burst for S3. I don't understand the question. I'm not entirely sure, but
maybe that I don't know about S3. But we can size the S3 bucket policies. Let's see. How long should each present URL
remain valid after we hand it to the browser? just a few minutes or something longer to accommodate slower mobile connection. So I want this to be a parameter that is sent by the actual user. This is a configurable parameter and this is a configurable parameter and will be passed by the caller of the service but has a limit of 60 minutes.
Okay, let's see how do you want to associate each object in S3 with the user who uploaded it. Embed the URL ID in the object key. Touch it as an S3 object tag meta field. Record it in the
database row or some combination of this. I want to user ID should be in the object key and in the metadata. The reason is because if it's in the object key, we're going to make sure users cannot upload files on uh override files from other users. And the other thing is
because if is in the metadata, we can validate that as part of the parameters that we send as well. So let's see do we have any encryption? Okay, I don't want to use at the moment any encryption at rest. This is just a parameter that we can set in our S3 configuration. So for simplicity, I'm just going to say no, we will use the AWS default encryption.
After the virus scan completes, how should we flag that an object is safe and ready for your automated job? That's the question we were expecting. So move or copy it into a separate clean bucket prefix / prefix. add an S3 object tag or metadata flag publish a message to an SQSSNS topic the job listen to or some other mechanism I'm going to make this part very tricky I would like to have a separate bucket and also the SQSSNS that is optional so so that we can have both mode we can have a synchronous and then a synchronous way of checking I want to use a clean bucket and when the new file without without specifying the mechanism I want to use a clean bucket and optionally SQS/SNS simple no more description so this part is already quite interesting because what we are seeing is that we are brainstorming and we are being treated I I feel I'm talking to a colleague and I also feel that all the questions are very relevant so we are it's already helping hug usually helping us to define the requirements for this so question 11 do we need to guarantee that every uploads gets a unique object ID so nothing is ever overwritten so it's quite smart from this point of view. If so, how should we generate those keys? Uids time stamp plus user ID or another scheme. So, I wanted to use time stamp plus user ID and time stamp plus user ID. That's so far I'm really impressed with the with the help I'm getting. So, I feel I can this part is
very very useful because you it's distraction free as well. So, I can stick to this. Do we already have a preferred antivirus malware scanning solution in AWS an existing lambda clam AV pattern an AWS marketplace product or something managed by security team? Do we need to design that part from scratch? I think AWS provides already um a way to scan the bucket but probably charge GPT doesn't know that. I think
AWS provides a I think as pro scanning mechanis but I don't know how that works. Um, I would like to use native native tools from AWS if possible. I think I can enable deep search. I'm going to try with deep search. I don't know what is going to happen. Can you search the web to see what we can use? That's quite interesting. Just to clarify, you would
like me to search the native tools or services that can perform virus and malware scanning in S3 uploaded files. Could you please confirm if you're open to solution that involve involve as marketplace product officially recommended by WS are you just thinking to looking for services fully managed just services fully managed by AWS. So the more I use it the more I feel that I I tried to do something similar few months ago and the level that we are today it's it's incredible. I was dreaming few months ago about this level of support and now it's here. So in few months we're going to see really amazing things. Okay. So we got our result and it it got blocked on my on my machine but then I checked from my phone and I noticed that it completed in 7 minutes.
I've been waiting for more than half an hour but so here we go. We have AWS m services for malware scanning in S3. So now what I want to discuss here this this is quite important because we saved lots of time in this moment because if it took 7 minutes and it checked 19 sources what means is that what would have been the alternative if we were just doing this as in our team. So we we had to stop the meeting because we didn't know what was the solution what we could use. We had to put down the the
notes for starting a spike time box it for a sprint. So we are going to operate work on this in next sprint. So we are already saving if if you're running a two week sprint we're already saving two weeks with just this. So now this may be good quality bad quality but you can see that there is a massive advantage in using these technologies to support you not to replace you. So that's that's the
key of this video and this series of video that I want to make overview scanning files in S3 for viruses or malware is crucial to protect data downstream. Okay, all this uh fluff um key AWS man services include Amazon Guard Duty with the new S3 malware protection feature for content scanning. That's exactly what we want. Content discovery for sensitive data. We don't
care about this. AWS security app for centralized security finding. Okay, what we want is Amazon Guard Duty uh malware protection for S3 mid of 2024. It includes malware. So you are also
learning as part of this how it works. you specify uh which buckets or specific prefixes to protect and guard duty will listen to the S3 event when okay it automatically triggers findings if guard duty main service is enabled okay so I'm not going to read the entire thing but what I want to do we can use this as a documentation for keeping track of our decision so why we went with this decision and very much uh we want to proceed with AWS s card duty for S3. Okay, so thank you very much. We want to proceed with the WS card duty for S3. Let's see what is the Awesome. That's a solid choice. Card will fit
really well with what you described. Do you want to scan all files in the bucket where uploads happen or should we restrict guard duty to only specific prefixes folder or certain buckets? That's interesting. Um, we want to scan only the files that are uploaded by web users. Okay. Will file uploaded by web users always go into a specific bucket or a specific prefix inside bucket so that we can easily target it for scanning? If so, could you describe the structure you have in mind? So, I'm going to say yes, but I don't have any structure structure in mind. Let's
see. No problem. We can define a clear structure together. Would you prefer that each user uploads go into their own folder like uploads user ID time stamp or just have all uploads go into shared folder like upload slash? I think I like the first one. So I'm going to copy this. I'm going to say I like the folder structure upload user ID time stone.
Nice choice. That will make things much cleaner and easier to manage later. When generating the preigned put URL for uploads, should we enforce file type validation? Yes, up front when creating the URL or will we just validate file types after the file is uploaded? I would like to enforce it. I would like to enforce it up front. Good call. Better to stop bad files early. Do you
want to validate mine types only or would you also validate the file extension during the presign URL generation? Both. Okay. So these are all details that will be very useful during the implementation. So capturing them is hugely helpful. Got it. Will enforce both mime type and file extension when generating the pre-signed URL. If the
validation fails, for example, wrong file type, should we return a detailed error message to the caller like unsupported file type or just a generic upload not allowed message? Okay, I just go with unsupported file type. Unsupported file type. Copy paste. Send. Perfect. We return a clear unsupported file file type message when validation fails. Do you want the service that generates pre-signed URLs to be publicly accessible caller from your front end app or should be protected behind some sort of gateway API key or IM role? I'm going to say that is um it's behind a gateway. I'm saying this because I don't
want to have any authentication mechanism implemented into the service codebase, but I want to match that externally with things like this in a in a Kubernetes cluster. It's behind the gateway and will be accessible only by trusted clients. Great. Having it behind the gateway for trusted clients sounds secure and clean. Should we log every time a presenter URL is generated including details like user ID name time stamp prerequest requested IP if available for audit troubleshooting purposes? Yes. Perfect. We'll make sure
to log all important details when a pre-signed URL is generated. Do you want the presign URL service to check if the user is allowed to upload at that moment? For example, enforcing grade limits like max number of uploads per hour or quarter. Uh I don't want to over complicate this. No, I Okay, let's say let's keep it simple. No. All right. No upload rate limits or quot needed.
Should we allow overriting if a user uploads a file with the same time stamp or should we prevent overwriting and force each upload to be truly unique? I want to avoid overwriting. Got it. We'll make sure each upload is a truly unique key to avoid any overrites. In case the upload fails, for example, the user never actually uploads after getting the present. Do we want do we need any
cleanup mechanism for unused object or expired presign URLs or is it fine to just let S3 life cycle rule clean up after 90 days? So this is I think incorrect because my understanding of pre-signed URLs is that it doesn't create any object. It's only a signature for the URL that will get validated that will expire automatically. So there is not much to do in this case and is true only if an upload fails midway through.
So yeah it's partially true. Do we need any cleanup mechanism for unused object? So we we rely on the life cycle rule. We rely on the life cycle rule for cleanup. Do we want the version object in S3? Turn on versioning in case somehow duplicate keys happen by mistake or is that unnecessary since we are enforcing uniqueness up front? Let's say is unnecessary.
Is sorry stop asking questions now. Cool. No versioning needed. Last one for now. Okay, cool. Sometimes I feel that it listens. Do you want to configure any notification like an SNS topic or even bridge event when a file is successfully scanned and mark clean by guard duty so downstream system can start proceeding it immediately. Do you want to configure
any notification? Let's say yes. Use an SNS topic. Uh so now what we want to do I want to use the SNS topic but this is optional and requested by the service caller getting the pre-signed URL when requesting requesting the pre-signed URL. So essentially what we're doing
here we're saying I want to get the event the notification on SNS but because we made it optional we only want that notification to happen in the moment that the requesttor specifies that the option is true. So only in that case we're going to get the event sent on the SNS topic. Awesome. So if the caller provides an SNS topic when requesting the present URL we'll hook it up for postcard notification. Otherwise
we skip it. I can now draft the full list of requirements based on everything we discussed. If you want would you like me to summarize it into a nice document format? Yes please. Okay. So now yes please. So now we should get our
document format. File upload service requirement document purpose. That's very cool. So now this one is already going to be very useful because it's capturing all the information of what we discussed failing pretty neat eight points is not overly verbose which is something that I like and all the information captured here. So now what I want to do how we're going to proceed with this because I don't want to make a video that is too long. So what we're
going to do now we have the requirements. Now what I'm going to do I'm going to request the list of stories that we need to do. So generate a list of stories that I can use. I'm also going to copy this into here. So let me go here. So I'm going to add all the
documents generated into a public GitHub repo. So I'm going to call this uh chatgpt plus and we're going to do requirements. Uh actually we're going to call it MD. So and then I'm going to paste everything in markdown. So it's nice. Here is the requirement based on everything we discussed. Remove this.
Okay. So the other thing I wanted was the architecture diagram. So before the surgery the list of stories. I'm going to ask to proceed with the architecture diagram. Create an architecture diagram. And I'm going to
ask to use mermaidjs. Uh do it using mermaid js. Okay. I don't know how this is going to look. We can see at the end. So I'm going to add this. Create a new file design. So this is dot. How do I include the mermaid in briefly describe the architecture and also add a sequence diagram? Is that a sequence diagram? Yeah, probably is diagram. Okay, so it replaced the entire
document. I'm going to copy paste and ignore the previous one. I think this one is very similar. Oh, there sequence diagram here. Interesting. Copy paste everything and boom.
Okay. Now generate a list of stories this work to be completed from coding to production. J the list of stories for this work to be completed from coding to production. Um including validation of uh that's it. So J the list of stories for this work to be completed from code to production. Let's
see what happens. User stories of file upload presigned URL story one. As a developer, I want to build a service that generates pre-signed input errors after validating story file upload flow. As a web user, I want to Okay, I was expecting a little bit more effort on this. So, probably is my fault because
I'm rushing it. So, I think I will skip this part. I'm going to commit this. I'm going to start to see the design. So, adding design from CH GP. Cool. Let's see what do we see on our repo and how that looks like. Unable to render. Okay,
there is a little bit of mistake here. And here we have our sequence diagram. Let me see what's the issue with this file. This related to uploads user ID and the user slash par line nine. Okay, line nine. Let's have a look. I think is one, two,
three, four, five, six, seven, eight, nine. I think it's here. So what we're going to do, we're going to remove the slashes as I suspect are causing issues. So let me see. No, probably the brackets. Yeah, very likely it's the brackets. Okay, we have a design API gateway present URL log validate file and type user valid generate present put URL return unsupported file type malware scan tag object as clean tag object as malicious move to quarantine bucket and SNS topic provided yes no publish success no action okay that that's quite clear okay so as I already said I was expecting a little bit more in terms of uh ability of completing everything but I think I was underestimating the amount of effort required with the with the chat with the actual AI. So, we I'm I'm
pretty happy with the results, by the way, because we managed to get a really clean idea what we want. And this will save literally weeks of development because most of the time what happens when we rush this part, we have lots of unanswered questions. Each question will result in a follow-up meeting and at least half an hour of a meeting for for each of the questions. So I think with the the number of question we got from chat GPT were 25. So we got 25 questions from chat GPT and some of them we didn't think about that. So I didn't have them
in mind and I was surprised to get those questions. So I was pleasantly surprised to get those questions. So I think this is the way I like to use AI not to just say write the code. So we you can also try to do a very similar thing. try to explain in a short sentence the code that you want to write. You will
generate the code but then you don't know what are the requirements. Now that we have these requirements, we can be very specific to how we want to structure the code, how we want to call the different components of the service and so on and so forth. So, and that's exactly how I think we should use AI to become 10x engineers. I think the amount of time we can save is massive. The other thing is that I'm going to share the link to the repo with all the files that have been generated so you can review at your own time. So, I didn't go
through every single thing at the moment, but I think it's a starting point. I don't expect the AI to do my job, and I don't want the AI to to do my job, otherwise I will be replaced. So, don't use it as a replacement, but use it as someone that can support you. So, I think the value we got from from this is already massive. And in a follow-up
video, what I'm going to do, I'm going to compare this exact same part with Gemini, and then we're going to focus on the implementation in uh in other videos. Hope you like the video. Let me know what you think down in the comments. Press the like button if you like the video and don't forget to subscribe so you will get notified when the new episodes will come out. In the
meantime, see you and ciao. [Music]
2025-05-17 10:19
