Build Anything With Voice AI Tool Calling (Vapi & n8n – Step by Step Guide/w Troubleshooting)

watch as I create a voice agent that books my calendar automatically while I chat with Elon Musk hello this is Elon Musk from SpaceX how can I assist you today my voice agent checks realtime Google Calendar availability and makes bookings instantly you'll learn how to build this time-saving automation using the new fappy dashboard and Tool calling UI I'll walk you through connecting naden backend automation to create a complete handsfree booking system stop juggling scheduling apps and wasting time on manual calendar management because this voice agent does it all for you hello this is Elon mus from SpaceX how can I assist you today hey I was looking to make an appointment please for tomorrow at 2 p.m. is that possible um sure thing I just need a bit more info what's your full name and what's the purpose of the appointment uh my full name is Ali Muhammad and I want to talk about potential Partnerships with you guys is that possible just to confirm you want to book the appointment for tomorrow at 2m right yes that's correct give me one second to make the booking for [Music] you awesome Ally your appointment is all set for tomorrow at 2 p.m. can't wait for you to chat about those Partnerships so I want to hop into vapy real quick in case anyone doesn't know how to do it all you have to do is just type vapy inside of the Google uh search bar and we can go to the first one vy. a and this is basically a voice AI platform for developers and it's really cool they've done a full revamp of the entire thing so I want to go over it in this video but first thing we can do here is open dashboard we'll probably go over the rest of these features later but go into the dashboard for now and it'll probably prompt you so it didn't prompt me but it will probably prompt you to log in all you have to do is just log in either with your Google or GitHub account if you want to use that but once you do that you'll be brought to this page you probably won't have these statistics you'll be probably sent to a blank slate but that's all good basically just get familiar with this overview it's a really damn cool dashboard that they've created and I think the first thing we want to run through is there's a few different fields here so we're going to click on the build field and we want to click on assistants and today we're going to be building out an assistant using tool call functionality based on vy's new I guess software so let's get into it I've already created this agent from before but we're going to start from a blank template and we're going to be be creating an appointment setup so I'm going to call this one Nina and we are going to create the assistant awesome so it's given us a predefined prompt so you are a voice assistant from a dental office located on BL the hours are 8 a.m. to 5:00 p.m. and you're closed on Sundays and there's a few task details that they ask for they ask for the full name the purpose of the appointment request their preferred date and time for the appointment and then confirm all details with the caller including the date and in time of the appointment so this is actually great and then rest sure to be kind funny and Wily keep all your respons is short and simple honestly we'll probably stick to this prompt but one thing I really want to try out is this new vappy AI so this allows you to edit the prompt in real time within the vappy platform so you don't have to go out into an external llm to actually modify the prompt so we can actually try this out so I want to change this from Mary's Dental to I guess we'll just say neur of AI in Dental sure we are open from 900 a.m. to 5:00 p.m. every single day even

weekends and we can press generate and it should go through and actually modify the prompt for us which it has so this is really cool it'll basically highlight in blue all the new things that it's added it'll highlight in red all the things that it's removed and you can basically tweak around with it ask for follow-up refinements if you want to before actually making the change so let's quickly go through this they change it to 9 and then 5:00 p.m. every day including weekends style use kind kind and witty tone keep response is simple keep responses brief chatting with a real conversation avoid rambling awesome and then you can basically apply the changes ask questions about the business and book appointments if the caller wants to book an appointment gather necessary information ask for their name purpose requested preferred date and time excellent we're going to come back to this in a little bit because we're going to be adding an external tool to actually enable this function calling but one thing I want to bring your attention to is the providers so if you're completely new to this there's a lot of different llm providers that you can have uh basically for The Voice model in this case we're going to go with open AI just cuz they're really good and I really like 40 mini just cuz it's extremely fast and it's extremely cost-effective and if you prompt your voice agent well enough it pretty much does the same thing files so this is an external knowledge base if you want to implement that in this case we're not going to implement a knowledge base just because this is a simple appointment setter temperature so because I guess we use kind and funny Woody tone we can obviously keep this temperature quite High basically temperature is used to control the randomness of the output so basically how sporadic or spontaneous or guess colorful do you want your voice AI to be pretty much lower temperature equals The Voice AI is going to be more monotone higher temperature means it's just going to act like a 3-year-old saying God knows what we're going to put this on 0.7 and then the max tokens so this is like Max tokens per interaction not the entire thing and if you want your conversations to be quick sharp and stop the AI from Rambling I would highly recommend just putting this to 250 detect emotion can play around with this I honestly haven't had that much luck with it and I guess for this purpose we don't really need it to be honest so after that is all done we can click publish awesome and we can see some statistics up here so as of right now this will basically cost us 11 cents per minute and it's broken down by each of the different categories and then of course the latency so because we're using a web application so we're talking to the agent through a website it's going to have a significantly lower latency than if we were to basically call it so let's go into the transcriber and the transcriber basically allows us to configure the transcription settings for the assistant so basically when we speak to the assistant how well is it going to be able to take that data and put it inside of the transcript which is really important if you're trying to grab name phone number email Etc so I love deep G I think deep G is awesome for this and they just released their Nova 3 model and we're going to be testing it out in this video I think it'll be really cool language you can choose whatever language you want which is going to Goan for English and then there's key terms so key terms are used can separate them by spaces but these are specific terms that you really want to capture in your transcript this could be anything like phone numbers emails sort of thing in this case we're going to leave it blank click publish voice so the voice you can kind of play around with whatever voice you want I think 11 Labs is still one of the best voice providers out there and we're going to basically stick with the default we can actually use Elon Musk um I don't that we'll use Elon Musk for this the so this is is basically the voice model how quick it's going to respond how the quality of the speech as well so in this case I want want to go with here probably probably flash V2 for that decreased latency and because this is a web application I'm not too worried about the quality background sound we're going to go with the office it's always nice to have a background sound um just in case there is a little bit of pause between you speaking and the voice agents reply it's not just like Dead Space a little bit of background noise always helps BR punctuation boundaries so these are punctuations that are considered valid boundaries or delimiters and this helps decide decide the chunks that are sent to the voice provider for The Voice generation as the llm tokens are streaming it basically if there are certain words that need certain pronunciation you can kind of up inside of here um using the limits but I'm not going to go into it that much stability we're going to go down to 0.4 and then Clarity I like doing 0.7 speed um so agent is terribly slow you can increase it but keep in mind I'm pretty sure the latency goes up oh no it doesn't okay awesome you basically increase the speed if you want I'm going to keep it as default style exaggeration uh I'm going to keep that as default default and then user speaker boost I'm going to keep that as default for now nothing crazy functions we're going to come to functions a bit we're going to go to advanced settings so advanced settings there are some privacy settings that you can enable disable hipper compliance so I guess if you're working in medical field probably want to turn this on um you actually purchase a Hipp of subscription for billing to actually enable this PCI complaints so you can basically turn that on as well audio recording I highly recommend just keeping audio recording on it's great for when you want to go back into the call and basically analyze it video recording um turn off you don't really need video recording so this is important start speaking plan so this is the plan for when the assistant should start talking and this is really important so basically wait second this is how long the assistant Waits before speaking now honestly this really depends on your latency you in in theory it sounds great to put this on zero but what happens sometimes is if your latency is super low like minine is the voice agent will start speaking before you stop speaking and it can get super annoying when the latency between you and the voice agent is like a milliseconds and it just doesn't make for a natural conversation so for that reason I like putting it on 0.4 smart endp pointing so we can enable for more accurate speech endpoint detection um this actually pretty good I would turn this on and we can either let vappy do that or we can use live kit for this live kit is only available in English but I think for this video I want to show you all the settings so we'll probably go through that if you don't want to use Smart end pointing you can obviously configure this as you like so the minimum seconds to wait after transcription ending with punctuation so for example if you're speaking a sentence and the transcription was able to identify that there's a full stop at the end of the sentence that you've stopped speaking it'll basically wait oh what did I just click oh that's not good it'll basically wait 0.1 seconds before

speaking on no punctuation seconds so let's say that like you're think of something mid sentence and you're kind of like uh that sort of thing this is good to have a long wait time because the AI will basically be able to determine that hey this person is mids senten Let Me Wait 1.5 seconds before he speaks again in case thinking um I'm not going to increase this want I'm going to keep this as 1.5 and we should be happy on number settings so this is basically the amount of time the agent should wait after you say numbers so this could be important for like email number that sort of thing and then voicemail detection so voicemail detection is pretty cool this has been updated so this basically configures how the assistant detects voicemail and there's a few different options you can do twilio Google open a uh in this case we're not doing voicemail so I'm just going to leave it as blank the stop speaking plan so this is the plan for when the assistant should stop talking and uh yeah basically you you can configure this and what this basically means is that once we speak three words so if the assistant is talking and we want to interrupt it once we speak three words uh the assistant will just stop talking voice seconds so this is the seconds the customer has to speak before the assistant stops talking 2 seconds is pretty good and then back off seconds so this is the seconds to wait before the assistant will start talking again after being interrupted Ed and I probably just put this down to zero because obviously we have our wait seconds up here and those two add up silence timeout so basically how long to wait before a call is automatically ended due to inactivity I like these default settings to be honest 30 seconds is great like if if you haven't responded to the call in like 30 probably safe to assume it's over and then the max call duration obviously you can decrease this as you want we're going to decrease it to about 300 seconds so like what is that 5 minutes keypad input settings we're not going to be worrying about that and then then messaging is basically this is basically post call data extraction so if you want to basically put in a server URL whether this could be a web hook um up to we're not going to worry about that uh for the video we don't really need to do that but basically this is great for if you want to put in a web hook URL and then at the end of the call this data during the call will get transferred to that web hook and then you can kind up that as you please so once that is done we'll basically go to analysis um this is basically if you want to do that data extraction inside of vapy this video we're not going to be doing that cuz we're going to be utilizing nadn functions so tools tools a way to enable your voice Bots to perform certain actions and tasks during the call add your tools from the tools Library which is right here so that your voice agents can connect to either make.com or go high Lev workflows you can also have custom tools with your own backend this is important so before we get into tools I want to go through these real quick so the pre predefined tools that are already built into vapy that are very common use cases so we have prebuilt functions for common use cases so you can enable them and configure them Below have the enable end call function so this will allow the assistant to end the call from its side and you always want to turn this on this is like a brainer right make sure your voice agent can end the call because if the other person if the person on the other side doesn't end the call you're going to be burning through a lot of money pretty much so let's go through these workflows workflows we going to cover in a separate video just because I think it's really cool and it's completely different phone numbers obviously you can attach a phone number to your voice agent if you want to call it through a landline uh I'm not going to show you mine in this video but you can do that tools so this is what we want to do so vapy has pretty much redone their tools UI and there's a there's seven different types of tools you can make you can make a function tool which is what we're going to be doing in this video and if we just go through it basically function tool you can give your tool a name and in this case we're going to be calling it book appointment or appointment book then the description for this is trigger whenever the user and I actually want to I'll save this real quick I want to go back to the assistant and double check grab this and we want to basically just put this inside of here and we want to modify this trigger when the user has provided their full name asked for the purpose of the appointment request requested their preferred date and time of the appointment confirmed all the details with the caller okay so probably want to just do this one trigger whenever the user has confirmed all details with the caller including the date time appointment as well their name cool so what this basically means is this tool will trigger whenever this criteria is satisfied so should be pretty self-explanatory we're going to save it server settings so this is basically the server you URL or the end point that this data is going to get transferred to now we can actually hop into nadn and build out the workflow for this agent and I want to start with a clean slate we're going to start with obviously our web hook and in this case we're going to be do using a test web hook make sure you put this to post request and it's a post request because we're sending data to a server posting data to a server hence why we do a post request and then the response you want to change the response from immediately to use respond to web hook node and I'll show you why in a second basically we'll just leave that as is for now and then we want to basically go into advanced Ai and want to create an AI agent and this is not going to be connected to a chat trigger node which is going to Define it below obviously we haven't called this web hook so we can't send any test data yet but we can populate the fields that we need to pretty much so this is the prompt for the agent and before we fill in the prompt I I want to show you what we're going to do so before I decide to add any tools or chat models we want to go through and add a web hook response node so this is basically responding to the web hook and this we're just going to respond with Json and I'm going to show you why in a second but we going to leave that as is chat model I honestly just really depends on your preference but you want something that's lightweight fast and able to compute extremely well uh for the video we we just go with for mini just cuz it's super efficient actually I use for Mini in like all my videos let's try something else let's try Gemini chat model and I want to try their new Pro flashlight 2.0 we'll try this so the tools tools are going be very important we want two specific tools we want to go into Google Calendar we're going to be using Google calendar for this video and we're going to be doing two tools so this tool is basically get availability we're going to leave that as is and then we're going to create another tool which is basically going through and book appointment awesome so logically what's meant to happen here is data is going to be sent to this web Hoke which will contain the transcript and it will basically say hey I want to make a booking for this time on this date and this AI agent is going to first of all check the availability for that specific time the user has requested if that time is available it is going to book the appointment if it is not available it's going to skip this tool request and basically send back to the server or the voice agent hey sorry that time is already occupied would you like to try another date awesome so let's populate these fields we're going to first do the get availability and obviously if you don't know how to set up your credentials there's plenty of tutorials online it's very quick and easy but once you do that you want to go into resource you want to just put this is get availability so we actually want to put calendar for our resource and then the operation is just availability make sure to choose your calendar which I have the start is time so basically we want to make this an expression and we don't want now we want to basically create our own expression and there's a really cool method called from AI which takes in two parameters in this case which is basically the key so that is the title of the I guess um parameter in the description if you don't know what that means I'll quickly show you an example so for the start time we can literally just call this start time so that is our key and then for our description so that the AI understands what start time even is the start time the user has requested for the booking so now ai will basically determine that the parameter to put into start time will be the start time that the user has requested for the booking and yeah it should basically be able to do that for us and then end time is basically the same thing so we can go through and just copy paste and change some Fields so we can call this end time so this is basically the uh we're going to change this actually so assume the end time is 30 minutes after the start time awesome that is how you basically use the from AI node and then we probably want to put in time zone as well so our time zone is Australia Brisbane obviously change that accordingly and also you want to go into settings here and you basically want to change your time zone to Australia Brisbane so I'm going to do just that make sure everything is time synced so that is the get availability and now we're going to configure the book appointment so for this one we basically want to do an event and we want to create cuz we're creating a booking select our calendar and the start time and end time is basically the same thing as this so we'll basically just copy and paste right here and we'll even do this one as well awesome use default reminders you can keep that on if you want and I think that is pretty much everything we want to configure obviously you can add like a field as well and then you can also add a summary field this is basically going to describe in the meeting like what the meeting is even about which is actually probably important so we probably want to do that because we do ask for the purpose of the meeting so I think description is in my opinion going to be the purpose of meeting so the purpose of the meeting slappointment so we're basically going to do that and then I think summary is basically just like um 30 minutes uh voice AI booking that is I could have these flipped around the wrong way we'll see when we actually test it but now you can see the red halo has basically disappeared from these tools which means that we've configured them accordingly all that's left is the AI agent and we kind of need data from this web hook before we can even configure the AI agent so let's go back into vapy and now we can kind of add we can add the field for the server setting so let's save this real quick and we can go into this web hook and you can basically grab the URL so there's two URLs there's a test URL and then there's production URL test URL is well like it said is great for testing and then production URL is when you want to go mainstream with this so in this case we're going to be testing it so we're going to put in the test URL and you can also put in a secret token if you want to be extra secure in this case I'm not going to do it I'm going to get rid of this afterwards anyway you can also add custom headers as well to include in your API request we're not going to do that for this video cuz it's just a simple tool call but yeah and then messages is actually quite useful so there's different message types that we can basically have the AI save when it's making this tool call so at the beginning of the tool call we can tell them that give me give me 1 second to make the booking for you awesome and then once the request is completed we can basically say oh actually no we want to leave that blank we just want request start and I'll show you why in a second so awesome that is basically our tool configured and what's really cool about this is we don't even need to use the voice agent to test this tool before we push it to the voice agent and what I mean by that is vapy has included this test field here and this is really cool CU we can basically just send data directly without having to talk to the voice agent ourselves so the request body I'm pretty sure if I think about this correctly how we want to send the data first of all I want to probably save this um we'll just add that so that we don't get an error if we press test and then if we send a request error in workflow error fetching Google Gemini interesting generate content instructions are the instructions let's do that real quick awesome so let's try testing it again send request awesome so we just sent a tool request and obviously this has nothing in it but we got some data back which is all that we need so what we can do with this data is a few things first of all we need to actually this data is kind of useless I can't lie to you I think what we need is data is actually kind of useless I'm not going to lie to you what I might do is actually populate this with transcript so obviously this takes the content in Json so we're going to call this transcript and basically make a booking for 2 p.m. tomorrow so if I test this it should hopefully give me back something decently use and did that do anything I think Json position does this actually work failed to fetch is that because it's cuz this hasn't done anything res sending awesome so that should have sent data across properly and we can basically see here body the transcript so we're going to put this inside of here and hopefully this can work for now but basically we can just say transcript is this and then the system message is quite important so the system mes is basically going to describe number one read through oh basically what we can do is provide a roll and give me one second I'm going to quickly whip up a prompt and get back to you awesome so I've gone ahead and written up a prompt so basically you are an experienced AI appointment setter So based on the user transcript the task so use the get availability tool to first check that the time the user has requested for the booking is available number two if the time that the user has requested is available then proceed to use the book appointment tool to confirm and create the book looking for that time slot then I want it to return the specific output gr I've scheduled your appointment with oi at insert users requested date he looks forward to speaking with you soon and then number three is basically if the time is not available then it'll basically return a specific output I want it to return the specific output for a certain reason and it's very important that if you are using your llms and you want them to return a specific predefined output that you go into your AI model and you turn the sampling temperature to zero this will basically get rid of all the variation that the voice that the llm could produce and make sure that this predefined message that you want to get the agent to produce is actually correct so what we can do here is basically respond to web hook so this is very important and if we go into the fappy documentation what I really want to show you is vappy tool calling and what's really cool about this is not custom tools I'm pretty sure it's one second let me find it so vapy has a custom format for Tool calling and this is basically basically a server response format so providing results and context so this is how the payload in the web hook response should be structured in order to actually uh get a response from The Voice agent so results is basically what we're going to be inputting into this respond to web hook the tool call ID so that is literally just this this is our tool call ID and then the result is basically that output I was talking about earlier so let's actually go ahead and do this we can copy this and Chuck it into responder web hook can be a Json field so tool call ID we can actually just set this manually as this we can hardcode it the results is basically going to be the output from this AI agent and it's really cool is we can basically take it and do this and and hopefully it should work so that is basically our structured respond to web hook and basically what's going to happen is web Hook is going to send transcription data AI agent is going to read that transcript and determine the user wants to create a booking at 2 p.m.

tomorrow it'll first check availability and if it's available it'll book appointment and then it'll basically create that output of like great your booking has been created for this time all looks forward to speaking with you soon and then we'll basically that data will get transferred to the respond to web pooc node which is going to cause the voice agent to basically speak this output it's really cool we're going to test it out in real time and see what happens so we can go back into vapy and I think that's pretty much everything we wanted to do and we can click save here and we can go back to our assistant and we can go back into the functions Tab and then the tool we can add the tool this is very important I've had situations where I was stuck and I realized that I just didn't enable the tool so once that's done we can click publish and now fingers crossed we can actually speak with this AI agent and hopefully get a booking in so let's do that right now hello this is Mary from Mary's Dental how can I assist you today I really I need to change it this is Elon Musk this is Elon Musk from SpaceX let's actually just do that I think that'll be funny let's actually just do that you are a voice assistant for Space X um and okay sweet talk to Elon Musk and hopefully get a booking it hello this is Elon Musk from SpaceX how can I assist you today hey I was looking to make a booking please is that possible um hey there what we're looking for just give me the scoop uh can I make booking please um absolutely I can help that what's your full name uh my full name Ali Muhammad nice to meet you Ali what's the purpose of your appointment uh yeah I really do sit down with you guys and kind of talk about um potential partnership potential partnership sounds uh intriguing when would you like to chat what date and time work for you um let's do tomorrow at 2 p.m. got it so we got Ali Muhammad for a potential partnership chat tomorrow at 2 p.m. just to confirm everything is that right yes that is correct give me one second to make the booking for you awesome your appointment is all set for tomorrow okay beautiful that is what we expected to happen because this AI agent basically that that tool call we kind of did here kind of useless I'm not going to lie so we need to change this because this field is undefined and we need to go down into this web hook and this basically captured all that transcript information that we had with Elon Musk and what we can do is we can go down to I think which one is it so these are the messages that we had with him and this is so stupid but what we can do here is we can find the actual transcript itself I'm just trying to see if the transcript is available I don't think it is okay that's all good basically we can do here is the transcript we can grab the what was it it was the information there's so much data in here okay awesome this messages AI format and this basically contains everything that we need for the actual agent so this is basically containing all the information about the transcript and what we can do very briefly is just put that into here now it is returning object object object object which greatest thing in the world um so we probably want to go with messages and see if that works that also returns object object object great okay H effect object why does object that is the weirdest thing ever maybe if I test this it'll probably do something no it's still object object object that is um all right give me one I'm going to fix this and I'll get back to you okay awesome so basically what happening is we were just sending a bunch of object data to the um server and what I've done here is I've basically added a split out node so we're splitting out all the information from this artifact so from here sorry from basically Returns the message zero message one message two and then those produce individual items so I basically created an aggregate function which takes all those individual items and bundles it into one item and you can see the output on the side here is basically the role of the agent and then the transcript so that should in theory work now and we can actually make a c and see what happens so let's do it Mr Elon Musk couldn't get tool ID does not exist oh yeah sorry wrong one here we go let's test it out hello this is Elon Musk from SpaceX how can I assist you today hey can I make a booking please um it sounds like you're trying to make an appointment what's up uh yeah can I make an appointment 2 p.m. just to go over a potential

partnership sure thing um I just need a couple more what's your full name uh my full name is Ali Muhammad Ali so just to confirm you want to meet tomorrow at 2 p.m. to discuss a potential partnership right yes that's correct give me one second to make the booking for you uhoh it seems there was a hiccup okay beautiful that is exactly what we so check the syntax of the Json response and make sure it is valid interesting so there seems to be an issue with the actual Json response and that's not great at all and we probably need to fix that so if we go back into vappy um I guess what is the issue tool call ID okay awesome so I made a small ER in my judgment this is not the tool call ID we need to change that that is not the tool call ID the tool call ID basically comes from this web hook and if we scroll down a little bit this is the tool call ID and Tool call list so if we just change that to this should test it and then the response is still invalid is that because of how it's structured no structure is good um oh wait hold on I think I might see the issue did we format this correctly we did don't know why the respond to web Hook is invalid Json unless I'd need to change the data no response code 200 yeah I don't know okay let me this and come back to you and hopefully we can get this working okay I must have just messed up some formatting but we got it working now and that's awesome so great I've scheduled your appointment with Ali at this time he looks forward to speaking with you soon we actually want to change that so we want to go back into notes and make sure that when you output the users requested date for your final response make sure it is a userfriendly format not ISO 8601 format so for example what we can do is basically put in great we can do this we can do great schedule your appointment for Ali at uh let's say like 3 p.m. he looks speaking to soon awesome so that is all done and one thing I want to do is we need to go back and actually prompt this better and this could honestly be the model so I'm going to change this model to 40 mini and fingers crossed it should actually book the appointment so it should get availability and book the appointment so that should have done that and fingers crossed okay made a voice AI booking for a.m. that is a little bit of a problem

so let's figure out what they did wrong so we basically wanted to do tomorrow that was available and then the booking agent start time the user requested for the booking awesome so this seemed to have done it at um a different time so we're going to miss in Utz plus 10 uh Australia brisman time Z in we're going to copy this and basically put it down into here uh did I do that right no do this I'm going to do that awesome okay so let's test this again and what cool about n end is we can basically keep testing this and make sure that the output works properly before we call the voice agent so now now it actually works but now a 12 which is not two unless I yeah it's not two which is not so this was basically 2:00 that is correct um I might actually changes for the booking convert to UTC + 10 Australia SL Brisbane time zone awesome so we're going to grab this and we're going to put that down here as well do that and then let's test this again and fingers crossed should work properly go back into here I will delete this and see what happens so go back into here awesome okay now it actually converts 2 p.m. we've specified the actual time zone and form properly okay so all that is done so delete this and make the call to the voice agent unlucky flash 2.0 but CH GPT is superior let's go back into Nina and I really should change this just Elon mask at this point Elon Musk hello this okay we got to call him again because I changed the name hello this is Elon Musk from SpaceX how can I assist you today hey I was looking to make a booking please for tomorrow um it sounds like you're Sear ing for something could you give bit more detail what are you looking for um I'd like to make a booking please for tomorrow at 2 p.m. is that possible um sure thing I just need a few more details what's your full name and what's the purpose of the appointment uh yeah my full name is Ali Muhammad and the best appointment is to really just sort out some sort of partnership with you guys got it Ally so just to confirm you want to book an appointment for a partnership discussion tomorrow at 2 p.m. right yes that's correct give me one second to make the booking for you awesome um glad you think so if you need anything else just let me know have a stellar day okay perfect thank you Elon Musk awesome so looks like thatly and we can see here that the booking was made for 2: p.m. tomorrow and uh yeah 30 minute voice AI booking and it doesn't seem like there any notes for this booking so that's all good now what we can do is test and see what happens if I try to make another booking at that time and um see if it'll basically tell me hey that time is not available so Elon let's call you again my friend and hello this is Elon mus from SpaceX how can I assist you today hey I was looking to make a booking for tomorrow at 2 p.m. is that possible um it seems

like you got cut off there what were you do uh I was looking to make a booking for tomorrow at 2 is that possible um sure thing but first I need a bit more info can you tell me your full name and the purpose of the appointment uh yeah sure so my full name is Ali Muhammad and if this appointment is to really just sort out some sort of partnership with you guys got it Ali Muhammad so partnership chat at 2 p.m. tomorrow just to confirm that's for tomorrow at 2 p.m. right yes that's correct give me one second to make the booking for you all set your appointment is booked for tomorrow at 2: p.m. okay awesome so it looks like it messed up a little bit there all good and we can kind of go back into here and figure out okay so I specify something very important here with this and basically something I want to specify is use the get availability tool to first check the time the use it as requested for the booking and check if it's available that's weird Json is not letting me copy I want to copy this Json and basically tell the llm what to search for uh there why can I not do that that is terrible I can copy this Json I want to basically grab this copy and I want to put it inside of here you know what that I I don't like it but it it'll probably do for now we can probably format it bit better uh honestly if I literally just do this and if I go into here and if I click control no it does the same thing basically this is what the output of the get availability tool looks like if the available available field is false then there is no time slot at the user requested time if it is true then there is a availability okay this is really important we're basically showing the AI agent what the output of the get availability tool looks like so it has some way better understanding of hey is this time available if it's not don't even bother making the booking and now hopefully it should not make booking uh okay did still make a booking which is a bit of a um to be honest so let's actually see what happened here so get availability that is true um bit strange cuz that's that's 12:00 p.m. so that's not I think I need to specify I need to specify in here we are in Australia SL Brisbane time zone and I think if I do that it should hopefully get availability and yeah awesome now it doesn't book at the appointment cuz it actually oh my God that is so tedious but I'm glad we finally figured that out okay so let's test this agent so we still have our booking for tomorrow at 2 p.m. let's test it and see what happens

if I try and make that booking again hello this is Elon Musk from SpaceX how can I assist you today hey I was looking to make a booking for tomorrow please at 2 p.m. is that possible um it seems like you might have gotten cut off there what were you looking for um I'd like to make a actually no I'm going to change this model to Turbo V2 it is a little longer but I think the quality would just make for a much more pleasant experience so we're going to call this and hopefully should work hello this is Elon mus from SpaceX how can I assist you today hey can I make a booking please for tomorrow at 2 p.m. um sure thing I just need a bit more info what's your full name yeah my name is Ali muhamed awesome thanks Ally um what's the purpose appointment okay one thing I want to do real again is go through the settings uh not the voice settings sorry I want to through advanced and I want to do this wait seconds to zero now that our delay is a bit less so let's do that and then try calling Elon hello this is Elon must from SpaceX how can I assist you you today hey can I make a booking for tomorrow at 2 p.m. please um hey there what's up how can I help you uh can I make a for 2 p.m. tomorrow

sure thing just need a bit more info what's your full name and what's the purpose of your appointment yeah so my full name is Ali Muhammad and I want to make an appointment tomorrow basically talk about Partnerships awesome Ellie so to confirm you want an appointment tomorrow at 2: p.m. to chat about Partnerships right yes that's correct give me one second to make the booking for you uhoh looks like I hit a little snag there but no worries your appointment is set for tomorrow at 2:00 p.m. to discuss okay this is not great um yeah this is this is the reality of basically disappointment set it just doesn't work um so that time it didn't even uh a booking so that is um I think I need to go back and reformat this prompt so I'm going to quickly do that and then cut back to the video okay so I have gone ahead and reformatted this prompt entirely to basically include thorough oh not task the task is the same rle is the same we've added some specifics we've added context which is basically the actual structured format of the get availability tool we've added some examples as well which could honestly be huge and so now in theory if I test this with our previous one so this was basically uh what appointment was this so this is to book an appointment for tomorrow at 2 p.m.

obviously we already have one so in theory they shouldn't use the book appointment tool let's see what happens okay didn't even use the get availability tool um actually didn't even yeah can I for name and purpose the appointment did we even we did uh that's weird so can I use this okay I'm changing this to and then let's see what happens so do um I think even o03 mini how long is o03 mini um cuz it could honestly just be a context window uh issue okay in 03 is long so we're not going to use 03 for that one back into here I want to use 40 make sure we grab the most upto-date 40 which I think is this one so let's try that and then hopefully logically it didn't even do anything again it looks like you're looking for help on taking the appointment I specialize in managing appointment if you'd like me to tell me B needs I can guide you um okay I don't know what's going on there cuz in theory it should use the tool core and this is the transcript so I don't know why it's struggling to hold that um hm okay what if I do containing all the information about the user's name and booing time okay let's try that and uh it should okay it just doesn't it just doesn't call the G ability I don't know why it does that maybe if I put down in the always use the get availability tool if I literally just do that it should in theory it doesn't is that because the I think the output it seems you provide me the task with a different context have I hold on let me reformat this what is going on I am so confused what is going on it seems you're referring to SpaceX however I'm currently set up to assist in Booking appointments for Ali in the Australia bris time zone SCH on appointment with him feel free to let me know your preferred time awesome so I think if I literally just do this Theory it was literally just the role literally just messed up what that's actually crazy so the role was basically confusing the AI agent thinking that it was an experienced AI appointment setter when really we working for SpaceX that is actually crazy wow so basically it's gone ahead and checked that 2 p.m. is not available and would you like to choose a different time awesome that is the craziest debugging on my life but hopefully now we should be able to test this whole workflow and full practice so let's see what happens let's first try making a booking at 2 p.m. hello this is Elon mus from SpaceX how can I assist you today hey can I make a booking for tomorrow at 2 p.m. please um there what's up how can I help you today can I make a booking for 2 p.m. tomorrow please um repeat that I didn't quite catch it what are you trying to say uh I make a booking for 2: p.m.

tomorrow please sure thing um can I get your full name first yep sir my full name is Ali Muhammad awesome thanks Ali now what's the purpose of your appointment I want to talk about potential Partnerships if that's possible got it potential Partnerships at 2: p.m. tomorrow just just to confirm that's tomorrow right yes that's correct great so just to recap your name is Ali Muhammad you're booking an appointment to discuss potential Partnerships and it's set for 2 p.m. tomorrow sound good yes that sounds good give me one second to make the booking for you all right now it's not even checking the get availability tool okay this literally okay I think the issue is I'm providing the voice AI prompt on top of the system prompt so we need to create some sort of logic that basically removes this initial message and and only shows the transcript unfortunat I really hate how vappy doesn't actually have like a specific field for the transcript um in that case let me quickly do some more formatting and basically get rid of that first initial voice prompt message cuz I think that's confusing the AI agent um so yeah I'll basically do that okay so I've made some crazy improvements because vappy can't figure it out themselves basically I've taken this raw data from the payload and I basically actually let me show you an example I did so I've taken this raw data from the web hook and I basically used code and by the way I just used um to help me generate this code but I basically Ed code to go through the messages field so if we go into here I basically used it to go through this messages field and always remove the system message cuz this system message identifies the voice AI prompt and that was confusing with the AI agent prompt so what happens is it basically formats the output of the actual transcript not the role of the actual voice AI agent so now in theory when I test this AI agent what it'll do is it'll check the availability the availability is true so then it'll book an appointment and then what's going to happen is it's going to create an output saying I I've scheduled your appointment with Ali for tomorrow he looks forward to speaking with you and yeah was an absolute doozy this video has G on for a very long period of time but you can kind of like the troubleshooting process that I go through and yeah something as simple as this can actually be quite a bit of a headache uh if you're not prepared so yeah further Ado I think this should be the final time we're testing it so let's freaking do it first I'm going to delete this cuz I want to test the booking capability and give it a try okay awesome so I'm going to make a phone call to Elon Musk just cuz it's easier hello this is Elon Musk from SpaceX how can I assist you today hey I was looking to make an appointment please for tomorrow at 2 p.m. is that possible um sure thing I just need a bit more info what's your full name and what's the purpose of the appointment uh my full name is Ali Muhammad and I want to talk about potential Partnerships with you guys is that possible got it Allie so you're looking to chat about some potential Partnerships just to confirm you want to book the appointment for tomorrow at 2 p.m. right yes that's correct give me one second to make the booking for you awesome Ally your appointment is all set for tomorrow at 2 p.m can't wait for you to chat about those Partnerships you to El musk thank you have a great day you too goodbye awesome so that was us creating the booking and now let's call him again to make the same booking but see what happens hello this is Elon mus from SpaceX how can I assist you today hey Elon I was looking to make a booking for 2: p.m. tomorrow is that

possible um I can help with that but first I need a bit more info what's your full name and what's the purpose of your appointment uh yeah my full name is Ali Muhammad and I want to talk about potential Partnerships if that's possible got it Ali so you want to chat about Partnerships at 2 pm. tomorrow just to confirm that's tomorrow at 2 p.m. right uh yes that's correct give me one second to make the booking for you oops looks like 2 p.m. tomorrow is already booked how about a different time what works for you uh no that's all good thank you okay oh awesome it works now so that was the the core issue that we were facing was that the voice agent prompt was basically getting mixed with the AI agent prompt and that was causing it to get confused but now it finally works and honestly I can probably bring this down to 40 mini uh to be honest and I'll probably execute the same thing and yeah does the exact same thing so that is the video thank you so much for watching if you've made it this far this has been a very long video of just me troubleshooting but I hope you enjoyed it as always the template to the voice agent and of course this is going to be down at the link to description but yeah I'm Ali mobad thank you so much for watching and I will see you specifically you in the next one catch you later

2025-03-20 01:11

Other news