"Build Your Own Copilot" By Sam Nasr (NIS Technologies)
Welcome to Sci-Fi DevCon. My name is Sam Nasser coming to you live from the Imperial Start Destroyer Bridge or as I affectionately call it, my home office in Cleveland, Ohio. So, let's get started. But before we do, may the
fourth be with you. Had to say it. So, this session is build your own co-pilot. And a little bit about me, you can find me on Twitter, Sam Nasser. I've been a software developer since 1995. and I'm a senior software engineer and trainer for NIS Technologies. I've been certified with a
few certifications from Microsoft and two of my biggest joys are running the Cleveland C user group as well as the Azure Cleveland user group. I'm also a technical author and I am a seventime Microsoft MVP. Little bit about the Cleveland user groups that I run. We meet every month.
The meetings are free of charge and open to the public and we cover a variety of topics related to either Azure or C respectively. You can find meeting information posted at meetup.com at the links available on the slide. So without further ado, let's dive into today's presentation and then and let's start with asking what is co-pilot? So co-pilot is Microsoft's generative AI service. It allows for conversational chat using an interface that essentially gives me the ability to interact with it using natural spoken language similar to how I would speak to a friend or a colleague. It's built on Open AI GPT4 LLM or large language model. It was
launched in February of 2023 and it utilizes what's called the premium model, meaning it's available free of charge, but there is a premium for uh upgraded services. I think it's safe to say that as developers, we're all familiar with GitHub Copilot. However, if you haven't seen it before, essentially, once it's installed in Visual Studio or Visual Studio Code, it allows you a pane where you can enter in specific text asking it to explain code, a code segment, be able to debug a code segment, or explain something to you, or better yet, write the code for you. As we see in the this slide, essentially stating in a comment section that we wanted to generate a function that retrieves uh specific data and it's able to go ahead and execute that code and be able to provide it for us where we can then hit the tab button and it's included. Since then, Co-Pilot has gained more notoriety and it was set up as its own separate website similar to Bing or Google where now we no longer have to enter in key phrases, but rather a full question asking it exactly what we want. Um, from that point on, C-Pilot started to spread and infiltrate every product within the Microsoft realm. And
as we see here, it's now available not only in in Visual Studio and Visual Studio Code for developers, but it's also available in Office. It's available in operating systems, database products, and a variety of other products from Microsoft. The advantage is we no longer have to go uh hunt and and search for specific uh answers to our questions. We can go into C-pilot for that specific product and ask it a direct question and get a direct response back.
So now the question is, if we have C-Pilot available, why would we want to customize it? Well, essentially, you're going to be tailoring it to your own custom data. So this allows for a deeper control over the user experience. Meaning that as a customer, I no longer have to search the web to find out how much an item costs on your website or to see where I can find it on your website, but rather using C-Pilot, I can ask it a direct question about the product itself, what colors it's available in, the price, etc. And so it provides for a better user experience as opposed to the old search engines where you type in a keyphrase, you get a ton of information back, and you kind of have to sift through it. It is now Q&A, direct answer
and response. So, it focuses the responses based on the custom instructions on how I set it up. Meaning, I could tell it to be very elaborate or I could tell it to be very brief. I can tell it to have a comical attitude or be more direct and straightforward. It's essentially an instruction as if I was giving it to an assistant, which exactly what Copilot is. It's an assistant. In addition, I can plug additional skills with actions so that as it retrieves an answer for the user, I can ask it, for example, to go out and research an API uh to get a better price on the item or a more recent price uh or look up uh inventory. So, it can do all
those things with additional plugins. Now, as far as tools, since GitHub is a widespread product and it's available in a variety of different flavors, likewise, there are a variety of tools that can be used to build my own co-pilot. So, if I'm utilizing uh GitHub Copilot, I would need to utilize github.com to be able to build a GitHub
co-pilot. uh if I'm using it for Visual Studio Code, I can use the AI teams library for Office 365 and be able to build a co-pilot for Office 365 there. Later on, Microsoft then developed a portal called Copilot Studio, which is a web-based uh application uh or website, I should say, that allows me to be able to build my own custom co-pilot later. that another product was developed called Azure AI Studio which was then renamed Azure AI foundry. This is also another web-based tool and it allows for building my own co-pilot. So now the question is what is
Azure AI foundry? So the AI foundry is a unified platform for developing, testing, deploying and essentially it's one-stop shopping for generative AI applications i.e. building my own co-pilot. It is web- based as I mentioned and it uses pre-built and customizable models. So I have some models that I can that are already pre-built that I can start with that and then expand on them and customize them to my own data. It allows me to import my own data
which then allows for a custom user experience. This is available free of charge right from the get-go. However, your price will depend on the resources that you're utilizing. Meaning, am I storing a lot of my own customer data or custom data, I should say. Am I
utilizing a lot of search services? It all depends on what other things I'm using. However, the actual use of AI foundry initially is free of charge. So I mentioned that there are base models that are available and that number is approximately 1,900 and that number is constantly growing and evolving as more models are being added into the foundry. Once I get into the
foundry, I can then search and sort by a variety of different things. So here I see that these 1900 models, they are collected and curated from 19 different collections. And we see collections on the on the side such as Azure AI, Azure OpenAI, Microsoft, Nvidia, uh Facebook, a variety of different curators that have published models that are all available in the AI foundry. I can also
select on the specific task that I wanted to do. Meaning, do I want to do text classification, text generation, uh do I want to do image classification? All these inference tasks are available to me where I can filter on what exactly that I'm looking for. Likewise, there's also uh 21 different license types that are available. So, it depends on what I'm willing to pay for and what I'm willing to work with. So, what are the prerequisites? There is the Azure subscription. That's necessary because
again everything starts with Azure. Then I would need the open AI access to be able to access the models. Once that's all set up, then I need to be able to set up a hub and a project and then deploy an Azure Open AI uh service. I can then follow the AI
foundry playground quickart. And playgrounds are essentially sandboxes where just as the name suggests, I can play around and experiment with my model to see if I'm getting the results that I'm expecting. I will also need an Azure AI search service that will sift through the customer or excuse me that will sift through the custom data that I provided. So then it can retrieve the proper responses back to the questions asked. And last but not least, my actual data that I will be providing. So for the demonstration, essentially we're going to go through several steps. Some of them will kind of
we'll fast forward through them. Uh but essentially the steps to be able to create your own co-pilot are as such. First, I need to create a hub and a project in my Azure AI foundry. I need to connect to the Azure AI or Open AI service.
I then need to create a deployment. So I can choose from two of the different deployment types. Uh whether it's serverless or through a VM. Once that's set up, I can now open it in the playground and be able to experiment with it. Then I can add in my own data, add a
connection to the resource group, and then be able to deploy it where I can then call it through a web call or through an API. uh excuse me, where I can then see it on a web page that I deployed or be able to call it through an API. So, as I always say, the best way to explain things is to actually demonstrate it. So, as I mentioned, we need to be able to create a uh a project and a hub. And so instead of spending time and and watching the uh what I call the the silver donut repeatedly, uh what I did was I created some screenshots that I'll share with you. And so the first thing
that we need to do within the AI foundry is that we need to create a project. And so here are some screenshots. As I create the project, uh you'll be asked to give it a name and then also create a new hub to go along with it. So we're going to call ours demo project one. And then we're going to specify all the specifics for the hub, meaning what subscription do we want to utilize in our Azure portal. Uh what resource group name, the location, etc., etc. Notice
that here I'm skipping connecting to the Azure AI search service at first, but then later on I will connect to it. And like with everything in Azure, we review and then click create a project. and we'll spend some time as that is being created. But thanks to the miracle of screenshots, we're able to fast forward through it. After everything is said and done, I now have my project and my hub already created for me and I'm able to get off to a running start. Now, the next thing that I would need to do is to be able to uh have a deployment. So, I'm going to select the
model type of GPT35 Turbo 16K because it does the job. It's free of charge and it works with the licensing type. Once I select that, then I will uh I can set up my own safety messages and we'll talk about those later. Next is adding in a data source. So this requires creating a search service as I had mentioned. And so from within the chat playground I'll collect excuse me I will click add a new data source as we see here and then I'll specify the file name where it allows me to upload a file and then I will connect it to a search service because now that I have my own custom data I need to be able to allow the co-pilot to search through it using a search service and be able to retrieve those answers and then some additional configuration where I will add in vector search to this uh search source and then review and finish. And now we have everything all set up for us. So it will take a minute to ingest
that data as we see here in the lower uh lefthand corner. Uh it's approximately three steps. So instead of sitting and waiting, we have screen captions that kind of expedited this for us. Then afterwards, I can select to
deploy it to a web app. And we'll talk about this in more detail later, but essentially I'm specifying the name of the web app, the subscription, and the resource group of where I want to deploy it. And the location, and of course, the pricing plan. So to sum it all up, I created my project, my hub. I added in my um my own
custom data and I created a a search service all through um Azure AI foundry that got me to this point. So what does it look like now that I have all that set up? Once I get into the foundry here, I see all the various projects and the hubs that were created. The one that I want to go to is the demo project one and demo hub one. So we'll click on that and then you'll notice that on the left side I have playgrounds that are available. So here it's giving me all
the names of the resources, the endpoints, my API keys and this all ties to the Azure portal which as we see here we have demo hub one and demo project one all associated within the resource group RG-Samnasser_AI. So essentially everything everything ties back to the Azure portal but I am doing all my customization over in the Azure AI foundry. So now clicking on playground I'll select the chat playground. Here's the model that I had previously chosen for deployment the GPT35 Turbo. And this is the model instruction that allows me to be able to customize the model of how I wanted it to ask. So by
default, it's stating that you are an AI assistant that helps people find information. Sounds well and good. Now I can say I can ask it a question like how many states are in the USA? And I need to spell that correctly. and it comes back with a very direct straightforward answer. Now to spice things up a little bit, I can say uh I can give it my own custom message. And just for demonstration purposes, I'm going to ask it, pardon me, I'm going to erase this model instruction and I'm going to paste. You are a comical and light-hearted AI assistant that helps people find information, but you become very rude when the same question is asked twice.
And I'm doing this strictly for demonstration purposes to show you that it can have its own personality, if you will. It is not a a straight robot question answer, but rather it is more of an assistant. So, it's the face of your own uh organization, your own company. How do you want it to act and respond to users? So, now that I pasted in a new instruction, I'm going to click on apply changes. I'll select continue and then I will ask the same question again. So notice that I had clicked
apply changes after I asked the first question. Uh how many states are there? Oh sorry I cut and pasted the wrong thing. All right. So, I will ask it how many states are there in the USA? And it comes back and now it has a little bit of an attitude because I asked it the same question twice. I literally just told you that the United States has 50 states. Pay attention next time. Kind of rude, but that's what I
had asked for. All right. Now I can also modify it and I can remove this portion of rudeness and say provide a detailed answer in five sentences or less. I'll click apply changes, continue. And then I'll paste in the same question. And there it came back with a
more verbose answer. Definitely not rude. Uh, a little bit light-hearted, but as you see, I can customize the instructions for my assistant and make it act the way I want it to act. So, this is it. This is how it is right out of the gate. there is no uh
data customization, but now we can step it up a notch and add in our own data. And as you see, there's a tab here for adding your own data. And because I already imported this previously, and I already have the search service set up, I'll simply select a product info file. And what this looks like, we'll open up what that file looks like. Um, oh, sorry. This is information about the file. Let's backtrack. Okay,
so this is a product info file and let me show you what it looks like uh from my desktop. So I'll open up the folder where it's contained and we'll be able to step through it in more detail. So basically this is a markdown file that I imported about product information from a um fictitious company uh called that manufactures um outdoor products more specifically the trail walker hiking boots. So as we see here we have the the trail walker hiking shoes. The price is
$110. It's under category hiking footwear. And again, this is nothing but a markdown file that has all the product information just listed in plain English. Um, I have some additional
specs regarding the color, the dimensions of it or the sizes available, toe protection, a variety of different information, right? And you'll notice that it is not in any specific format, but rather it's a markdown file all in human readable English as opposed to C or JSON. So now that this file has been imported and ready for use can ask it a question like how much are the trail walker boots? And now we'll retrieve an answer stating the trail walker hiking shoes are priced at $110. it will show me a reference of where it had this of where it retrieved this information from. And that is in this markdown file that I received. And now if I click on that again, all this is within the the the the foundry. There's a side pane that opens up that shows me the exact file and there I see where it obtained that information from. Okay.
So now I'm able to actually be able to utilize my own uh custom data for my own organization allow customers to be able to enter in questions and we can expand on that by saying by asking what colors are available. So let's say that even I even though I read the spec sheet I still missed what colors are available and it comes back and saying they're available in black. And then I can ask questions such as sizes available. And you notice it wasn't a fully formed proper English question, but rather it was sizes available question mark similar to how I would ask a colleague or a friend of mine, right? But it understood that that that's what I'm after. And then it replied back with the response that I have sizes 7 through 12 available. And again, it shows me the name of the file where I retrieved that from. So I can import multiple files, be
able to have it search on it, but then as I'm testing it here, I want to know if that's the right answer or not and where it retrieved it from. And so it it gives me a reference where that file was retrieved. So let's say that everything is fine and dandy. I'm very happy with how everything is working out here and I'm ready to deploy it. Now I can select
deployment and I can select it as a new web app. I can zip it up as a dozip file to be able to share it in a teams app or I can create a new co-pilot and co-pilot studio from the data that I retrieved here. So I'll select to deploy it as a web app. And again here's where it's walking us through. I can either update an existing web application or I can create a new web application and like with everything else I need to select a subscription and the location, the pricing, etc.,
etc. Since I already have it deployed, I won't bother uh deploying it again. But once it's deployed, it is then accessible through the website where I have a web interface similar to this where I deployed it as a uh a website within Azure. And now I can ask it the same question. It'll take a moment to generate the answer and it should respond back with where it found it as well as the actual cost of it. Okay. So now this is a website that I can customize to fit in with my own um organizational website and be able to show the user the ability to type in a question and be able to retrieve an answer.
So gone are the days where they enter in key phrases in a search engine and having to sift through piles of data to be able to get the information they want or having to go through multiple clicks. It is now question answer done. Much better user experience uh for our own uh customers. Now let's take it a step further. So, we talked about how to set
up the uh co-pilot, be able to utilize it and and within the playground, be able to test it, evaluate the results, and now I can likewise I can also deploy it from the the playground. But let's go into more detail as far as what the AI foundry is and how the hub and the project work together. So the hub is essentially a grouping of projects that allows me to be able to share security settings, data connections, artifacts, etc. The project is for creating,
testing, and deploying the API. More specifically, the hub acts as a primary top level for Azure resource for the AI foundry. It gives me the ability to centralize the governance resources and connectivity and as a v and as well as a variety of other resources that can be contained within that hub and shared among all the projects. So every project that I create in that hub will inherit the same security and resource access that were specified in the hub. The advantage of this is that enables developers to create projects and access shared resources without having to go and request admin access. So basically I set up the hub, I
set up the resources that go along with it and then all the projects within it inherit or have access to these resources. So both the hub and the projects are created in the Azure AI foundry and they can be created in the Azure portal by utilizing templates. So as I pointed out earlier everything that I created in the hub and the project points back and is created in the Azure portal within that resource group that I initially specified. So within the hub resources when we talk about security configuration that includes public network access, virtual networking, uh manage key encryption, privileged access, all things that I want all my projects to be able to have access to. In addition, I can provide
connections and these are named and authenticated references to Azure and non-asure resources alike. as things like data storage and it uses a connection as a means of making an external resource available to a group of developers. In addition, I can provide the compute and quota allocations for that hub. So that way there is no rampant uh increase in cost or usage. And then AI services can be
utilized using access keys to to endpoints for pre-built AI models. and they're all managed on the hub scope. So all within the the hub and then I can enforce policy in Azure on the hub scope and it applies to all the projects underneath it. So as I manage everything in the hub naturally everything good or bad whether I'm adding or removing will affect all the projects within that hub. So it's also utilizes the uh Azure dependent resources. So anything that I
that the hub is dependent on will be uh included and managed within that hub and apply to all the projects. So what does this architecture look like? Essentially uh I have one hub and as we see here one hub has zero or many uh projects. Uh over on the right hand side uh the hub will contain the the data uploaded the artifact storage um the compute capabilities and quotas where as we see in the project it contains the the data sets and the models for each individual project and essentially these are isolated data containers that are contained within the hub. So each project is an isolated
container and its project scoped connections as well. Likewise, I can also connect to the hub by utilizing AI services by utilizing the uh uh model deployments. So, let's explore the the foundry a little bit more and take a look at some additional features. So, as we see within the AI foundry, we have a variety of things available to us. In the left pane, we
already talked about the the playgrounds and we started by going to the chat playground. However, there is the speech playground. There are agents playground, images playground where I can generate images using the the dolly models, uh speech playground which gives me the ability to utilize a variety of different services such as speech to text, text to speech, um speech translation, a variety of different things all available. So I have a variety of different playgrounds all within the foundry. In addition, if I click on the model catalog, this is where I see all the models that are available to me. And as we see here, as of today, there are 1928 models. So this number is constantly
evolving as new models are being added into the foundry. I can search for a specific model or a specific collection of models. So here I just clicked for Microsoft to see what's available for Microsoft. I can also filter down by the inference task where let's say I'm looking for speech recognition and there is one model that fits uh that criteria that it's both from Microsoft and does speech recognition. There's also a variety of other different filters that I can filter on for fine-tuning different license types, etc., etc. And let me go ahead and clear
this. Let's do clear all. In addition, I can select a specific model and be able to view its um specific data. or metadata I should say. Um I can see
if I have any existing deployments of that and I have none at this point. Let me go back to the model catalog. Here I can select various models and then I can compare the results to see how well they do. And
here it's showing me graphs of the quality index. Um, in addition, I can add in different models that I want to to search on and to compare. Excuse me. So, we'll add in 01 for chat completion. And so, this provides one-stop shopping for being able to view what the models are about. Um, be able to filter on
various models. Um, now going back into our project, pardon me. So, we left off where we're doing some testing here and in the playground. Now another thing that I could do um is I can select to uh import or export a model setup that I have and again this would be uh a JSON file or a prompty file where I can import it for and pick up where someone had left off i.e. a colleague or a uh a working partner. Likewise I can export everything that I have done here and export it either as a JSON file or as a prompty file.
In addition, if I need to see once this is deployed, I need to have a sample code of how is it that I'm going to access the model deployment. I can click on view code and it provides me a sample. This one is in Python, but I can select from a variety of different languages, i.e. C. And I have a code snippet here. Pardon me. I have a code snippet here of
what I would need to utilize to be able to access that model. So, it gives me a good start. Another thing I wanted to address is the uh safety system messages. So, if we look under uh system can add data here and there are uh various parameters that we can add in here as well. So this is regarding um the past messages whether they're going to be included in the question meaning the the context of what I'm referring to. So,
for example, when I asked what colors are available, this question by itself doesn't make much sense. But in the context of the trail walker boots that I asked about previously, it makes sense to the model. Right? Obviously, I'm asking about the colors that are available for the trail walker. So, this
included the previous messages uh that I sent in or the context In addition, I can specify how how strict I want it to be, meaning which documents uh or uh what is the most accurate document that I wanted to select from in the response. So, I can set that in my data content under advanced settings. And then in addition uh as we see here there's an add section where I can add in safety system messages and we touched on this earlier but here I can protect it since this is going to be open to my customer base and and the public at large. I want to be able to protect my
application and my model from certain things. For example, I want to be able to avoid any harmful content. So, if I am a uh outdoor sporting goods store, I don't want to display pictures of weapons. Um, use your imagination what that could be in a sporting goods store. So, I'm going to avoid harmful content from being displayed. I can avoid ungrounded
content, meaning that if I didn't train it on specific uh questions and answers, I don't want it to respond um outside of that. I'm going to avoid any copyright infringements. So, generally speaking, outside of this example, I can't say summarize um Midnight Summer's Dream for me, for example. Uh that would be a copyright infringement. Uh I can also select to avoid jailbreaks and manipulation. Meaning that if I set the
model to let me cancel out of this for a second. Even though I specified here in the model provide a detailed answer in five sentences or less. A jailbreak is when someone asks the model to do something that is outside of what is specified in the instructions here. So
that is what avoiding jailbreak is. I want it to stick to the instructions that I gave it. So that is what the safety content messages are all about. In addition, there are some chat features that are available, meaning verbal chat. So I can ask it a question. Uh and likewise, I can get it to respond back by enabling speechto text and text to speech. I can select the voice tone
of how I want it to respond, whether it's a male or a female. And more specifically, I think it will specify nationality and the type of English. So, some of these characters that are specified here, Nancy, Nova, Phoebe, in some cases, they have specific accents, uh, Australian, Irish, what have you, uh, to respond back to the user. So what this allows me to do is to be able to submit a question verbally and then likewise be able to get a response back verbally all tied to the data that I had provided it. Now keep in
mind there is a cost that can be incurred because now I'm utilizing additional services for speech to text and text to speech. And as we see here at the bottom, it needs to be checked that the user acknowledges that spoken chat will incur usage to my subscription. So keep that in mind and we'll go ahead and close out of that. So in a nutshell, this is what the
Azure AI foundry is about. Uh one of the reasons that I love it is because it is one-stop shopping for everything. I can do my testing. I can do my deployments. I can choose different models. Um I can compare the various models and then I can do uh all the content safety messages and then deploy it either to a web app uh or deploy it as a do.zip and be able to take a look at some sample code by clicking the view code button to see some sample code that I can use to be able to access that deployment.
All right. So, with that, let's get back into our PowerPoint. So, some gotchas that I wanted to mention. Um, you need to
utilize the directory associated with the subscription, meaning the the directory that's specified in your Azure portal. That needs to be associated with the subscription that you're using for the uh the foundry. your subscription needs to be below your quota limit in order to be able to deploy a new model. So if you're at the quota limit uh or exceeded it, then obviously you won't be able to deploy a new model. In addition, the user role must
be either Azure AI developer, contributor or owner on the hub. Lastly, whenever you make changes as we had done in the foundry, meaning that I updated uh the system messages and or the instructions rather of how I want the model to behave, I specified or I clicked apply changes afterwards. In addition, whenever I'm deploying to a web app, you will see the message pop up. Your web app will be configured in the Azure Active AI directory. And then there's a message saying here that it could be upwards of 10 minutes. So be patient but know that
it is happening in the back end and will be deployed. And so with that I usually like to ask what is it that we've learned here today? We covered uh what co-pilot is in general, more specifically how we can create our own co-pilot, the steps that are needed, and it can all be done within the Azure AI foundry. We had a general uh tour of the Azure AI foundry. We saw where the
models are located. How we can actually sort and uh pick our own specific model based on inference task or where it's curated from whether it's Microsoft, Nvidia or one of the other publishers. Uh we saw how we can test it, get the results back, how we can import our own data and essentially have a uh a portal that our customers could use to be able to ask questions and get answers back. some resources for the Azure AI foundry architecture. Uh Microsoft has some great writeups on that. In addition,
there is a quick start that's available for creating a project and using the chat playground in the foundry. There's also a link to deploy an enterprise chat web application and an AI foundry FAQ or frequently asked questions. All these links will be made available. In addition, if you're interested in this as a course in your organization or you're interested in having a proof of concept to build your own co-pilot for your organization, NIS Technologies is more than willing to help either in consulting or in training. Uh we'd be more than happy to help you. I myself am a consultant and a trainer and we'd be happy to show you how to get co-pilot off the ground in your organization.
So, as we wrap up this meeting, I always like to stress this doesn't end the conversation. Let's keep in touch. My email snasser nist techchnologies.com. If I can answer any questions for you regarding this presentation, or if you're looking for additional training, get a co-pilot off the ground in your organ in your own organization, I'd be more than happy to help. You can also find all my socials on linktree/samnasser. And lastly, if we're not connected on LinkedIn, it's my favorite platform for a good reason. Let's connect there. Uh would love to
hear from you. And uh with that, thank you all for attending. Appreciate your time and I certainly hope it was helpful for you. Hope you enjoy the rest of
Sci-Fi Defcon. Bye everyone.
2025-05-07 15:38
