What's up everybody? Welcome back. This is Automate It episode number oh i gotta go two hands thirteen. I am Jon Levesque, i'm your host and as always i am joined by the awesome crew of Kent, Pranav and Apostolis. Welcome guys how you doing great john how about you doing good doing good. Kent how are you up in cold Canada uh not too bad not too bad i see you got a fancy new studio behind you that looks pretty cool yeah yeah i got an upgrade in digs we're no longer in the office i'm now in the studio and pranav how you doing buddy i'm doing well in the rainy seattle show i know i wish i wish we would get some snow or some sun or something else i'm tired of gray and wet i can sell you some snow that's not a problem i mean a cooler of snow is not enough Kent i need like i needed to come from the sky you know i appreciate the thought though i wish the borders were open i'd come hang out up there and and also you know what real quick let's say a big happy new year this is our first episode back after the new year happy new year to everybody glad to have you back with us yes okay so on to the business at hand today Apostolis is going to show us some great stuff for so many of you the out of the box analytics you get in flow and power automate are just not enough we hear all the times how do i get deeper how do i find out more information and so apostles today is going to actually walk us through how we can do just that how we can dig into the common data service find the real set of statistics where all that information lives and decide what we want to display for ourselves so apostles i'm going to go ahead and hand things over to you my friend and the the floor is yours yeah thanks john so the topic of today is really how as john said how we can go deeper on the flow analytics uh now that we have uh you know the to distinguish also between cloud flows and desktop flows there are of course different reasons why you would like to dig deeper on specifics uh of uh past flow execution or maybe even on real time so in order to uh enable that um we leveraged uh something which is called microsoft databurst this is the underlying foundation of course of the power platform but also uh driving the whole dynamics 365 business solution stack so we will be leveraging uh the microsoft dataverse api system together with power bi and the power ultimate service to get a little bit deeper when it comes to the flow analytics so for that let me actually jump into the deck so the agenda will be we first of all the key drivers why why we did that dashboard how it looks like how it has been built some really important findings around that and then looking also into options of real-time monitoring and of course q a so before we start what i just said the underlying foundation of that is of course uh leveraging uh the power platform the breadth of the power platform um to to you know do this low code no code kind of extension and even reporting to derive those insights we are looking for so in our scenario from the left-hand side we will be using power bi the business analytics tool power automate of course for the flow execution and the Microsoft dataverse underlying foundation which is uh much more than only a data store right it's really uh it is a hyperscale database as a service if you would like so if you're looking here what this is comprised of i mean this is as you can see here on the right hand side Microsoft database first of all it runs on azure and it's much more than only you know azure sql database so it is really comprised of many many components here making it super scalable highly secure and of course hyper uh hyperdense and stuff right so and if you're looking at the internals from a feature perspective of dataverse as i said it comes with a lot of built-in services like security logic data storage integration and then if you're looking into the you know the foundational topics like authentication authorization but also auditing so you know who who's accessing those components when and uh you know have the full traceability of how many executions of a specific activities on a table has been done and so on then on the right hand side you see many connectivity options reporting you know connectivity to azure data lake and so on so forth so and the api this is what we have leveraged in the scenario to derive to connect to those endpoints which are you know whenever you execute a power automate desktop flow uh the telemetry the execution lock is then of course transferred to our cloud service and from there we can access uh that data through the data verse api connectivity and this is of course enabling many many scenarios across of course the power platform and even beyond that when it comes to integration and to surrounding of of the data uh sitting in database okay so why this dashboard first of all i was bored during the lockdown uh i mean yeah you know it's it's it's it's easy how it's really amazing how easy you can get started with those uh components once you you know a little bit the workings of the power platform it's so easy to extend that knowledge to different uh tool sets like powering bi in this case or even smaller power ups which then surface information uh or even allow you to to uh to you know enter information which will then be part of the overall flow so it's easy to up skill on that and uh you will see this uh in in the real-time dashboard at the end as well so it's focusing on uh desktop flow uh insights what i said before not only uh when an automation has been executed but how long it took what actions have been called uh what actions in in that automation have failed uh which ones uh you know took longer on the previous week for instance or previous execution some statistical process control you can do on these these components and these statistics so that was really uh to get deeper insights on on what's going on from an execution perspective um showcasing of course the api endpoint consumption what i just said so data versus a gift that keeps giving all these built-in hyperscale uh you know architecture security by design record level security so you know whenever you are you know accessing the end point it's respecting also the record level security for your user uh which has been applied audit management and so on then it allows for reactive governance and control like we have with the coe starter kit where you can really uh control execution on action level if you would like to we see this later in in the chart and then you know proactive uh reach out instead of blocking the whole service just proactively then monitoring and reaching out to folks and ask okay what were the use cases why they have used certain actions just to understand a little bit more about the demand is within the organization and also help drive of course the citizen development ship identification of api based operations and also pro code actions we see that in a second what that means what i said before statistical reporting so you can really compare flow runs with previous florence and then you know judge if this is considered an outlier or if there's a pattern you can see okay where you need maybe to look at the hardware or the vms and and so forth it is easy to deploy this was also you know one of the design principles many organizations they they don't have uh you know the full breadth of the power platform components enabled today or deployed things like the coe starter kit which is really a massive governance framework so if you just get started with the you know power automated desktop and the power domain service in general and you don't want to have uh the cre starter kit deployed in your organization but you would like to get also in the same time of course those insights from from the power automate service this is where you can really get going with this starter template and the last point what i just mentioned is really for organizations that haven't uh you know enabled the breadth of the platform yet so all you need is of course uh access to the database uh backend sufficient privileges is very important because otherwise you won't see uh things uh power bi desktop and off you're going okay any questions uh john team no it's all very clear to me so i guess one question just around uh when you mentioned the right level of access uh so what is required in order to to be able to retrieve this data so first of all you have to be uh you know flows are assigned to specific owners so whenever you you author a flow you are the owner that of that flow and then you can share that flow of course as co-owners with others so this is very important that you have to have uh dataverse environment access first of all so if you don't have access to that environment you would not see anything of course and then it goes really further down that you have object access you have rbox on the cds or in the database and tables that you can say okay on a flow session or in this and that workflow entity i would like to restrict that access to that so that's the idea behind yeah and whatever you are accessing you cannot even restrict it so let's say a flow has not been shared with you and also the execution has not been shared with you that normal regular user which is not really having any administrative control won't see that right so you don't get more access by accessing those power bi report through the dataverse api system because that's the essence what i said before it is secure by design so i guess just to follow up so if i was a regular maker that's building say api flows or cloudflows and desktop flows i would be able to see my information but if i say was an administrator i would be able to see um more information because i've been granted that administrator role is that a is that accurate then that's correct and you would see i will demo this at the end as well so what a normal user would see and what the administrator sees and that's i think one of the essential parts why i said you know dataverse is the central piece of our whole power platform and it is so important but that always front and center because all these you know uh scalability and security configurations and the auditability of that system this is just uh god given for us right as soon as you have a license for a power automate desktop or for power domain you can leverage all these technologies which are out of the box you don't have to worry about them right it's always whether you access the this data through the api through the ui or other means it's always respecting the api and the security layer behind it so that's that's very powerful of course yeah i like that it's god-given rights as soon as yeah that's awesome that's great all right so let's have a look uh at the at the demo let's do it demo time so first of all i'm opening here power sorry power bi and which is leveraging what i said before the Microsoft dataverse api to get all this information here so we're starting here and these are this it's on purpose renault you know it's a starter template so this is meant to be customizable like the rest of the series starter kit you can really customize this to your needs and here's the data model of that and here you can see a graphical representation of that uh you can really uh change uh you know add new relationships to that you can add new measures new column calculated members and so on so once you have started that as a template you are asked to provide two things one the dataverse environment url and the flow session history url in poweradmin portal then you have to authenticate with a user that has proper privileges and then you are print it's loading the data and it's presenting it in in this starter dashboard here okay so from here what we see is some filtering section here on top where you can flow status you know failed or succeeded run mode attended unattended or executed from the power automate desktop console then some basic statistics here on top how many makers do i have in my environment how many desktop flows how many legacy ui flows well which is using you know still the windows recorder this we come in a second to that so how many overall flow runs i had uh in that time frame here attended unattended runs console runs how many bots uh how many canceled flows failed and so on and this one is uh something when you have a super large power automated desktop runs which are running and many loops or a large loop within those uh it is producing a lot of course log contacts and this to identify those large log flow runs this is where you can see them here in this overview then you have a couple of uh lists here top three makers top ten bot hosts those are the hosts where the power automate desktop have been run on through the on-premise data gateway uh you know total hours processed and so on uh then the top actions in power automate desktop which have been used along the line you can see of course the right to excel has been used 23 25 000 times what was the average uh processing time on it so just a couple of uh you know basic statistics uh you can get all basic counts you can get out of whatever the flow portal is really producing here from an action perspective so if i click here on one of these flows and i look here at the processing history so all these actions which are registered here they are compiled in this power bi dashboard rights so from here as i said you see action subflow start duration and status but when we are looking later on in the action details you see of course many more things let's go to the desktop flow overview so this is also another important view which gives you on the desktop uh flow name level right here it gives you really a quick start how many times it has been run failed importantly how many action counts uh have been included in those flow runs uh is it using proscript we come to this in a second how many uh you know action calls have been done with pro script or for proscript and what is the total run time on minutes for that specific automation then we have here an indicator whether this is a legacy a ui flow or not uh just to give you an idea okay which of those might be potential targets to migrate to the new power ultimate desktop now what is very important here to note is uh this this feature here and that's this is where we start to really dig deeper on on those flow runs as you can see here as well in the overview statistics so i have here an indicator which says okay how many flows are using uh those pro pro scripting components in my flow run history and this will give you a good indication you know how many of these have been uh including things like run a vbscript run powershell run python scripts or open a command session so on so these are usually indicators where developers rpa developers if they're citizen developers or not when they leverage those you know more advanced components in the power automated desktop then you might want to dig a little bit deeper without opening each and every uh flow individually you would like to have really 30 000 feet you okay how many of my desktop flows are leveraging technologies which might reach out to other web servers do some you know uh powershell scripting which is of course very uh very sophisticated things you can do and potentially also uh dangerous things you can do with those components so to get an overview of those this is where these kpi here comes from that was the idea behind it right and we can drill down into these in a second so overview starts desktop flows what i said and now the desktop flow monitor the desktop flow monitor is nothing different than what we see in the power automator portal and also the layout is aligned to that so if you look here back uh the status is what you see here and start and stop and duration and so on so here conveniently we are showing also if it has been a failure what the error message was right and from here as well you have two links you can go to the actual uh power automate desktop flow so if i click on that so it's opening the power automate portal and directly goes to that position of the flow xor to the to the flow run history of that particular record and here i can again review whatever has happened and the error message of uh that thing so fail to obtain output for custom object here for instance right and here i can see the screenshot as well and what has happened exactly so if i minimize this or i can go even if this automation has been triggered from a cloud flow i can also go here to the para flow and then i have a look okay what happened here in that if there was a failure or not or white has been cancelled and so on so it's opening again on the other screen that's handy to give a view that looks aligned to the portal you know something familiar so it's like oh this makes sense to me i've seen this before exactly especially uh john in case of cloud flows you know that we have this option you can see each and every component in the cloud flow so we can really debug that specific uh action uh which was causing maybe the error or even you would like to look at the input and outputs of those actions individual individually just to make sure okay you understand what what happened in in those environments okay totally so from here on what's important as i said uh in the cloud flow history you will see if you go back to the history records you will see here a note which is saying on the cloud flows of course not on the desktop flows so you will see a note here on top of the run history which says 28 days run history so anything you know beyond that is cut off because of gdpr and privacy requirements and so on and to indicate you know before you before you really click on one of those icons it will give you an indicator graphical integra indicator with a flag okay there is uh that process execution or that flow execution is older let's say than uh 28 days okay so here then you see then the indicator of those flows okay so let's go up here and look at the next component from here on and that's also important right if i would like to review of course my flows and i would like to look at more details on specific errors what is nice here what you can also not do from a portal side is that you go for instance here you say okay on letter specific automation which succeeded in this case i would like to right click if you would like to drill through that record this is really the execution of all flow history right you can now go into details like to the run details so which actions have been included as part of that execution you see here 24 actions the average duration of each action was 1.85 seconds and what was the max duration uh of a specific execution here okay so in here you can see even again the flow itself so it will open again the portal or you can also look at to okay what information drilled down further what information uh has been used or what actions have been used uh in in those flows if for instance a script has been used then you would see you see that here or if web service calls have been done through that then you will see that in this experience so let's have a look at uh actually scripts i think those will be used here okay oh quite a lot of scripts as you can see and this is now i think one of the key reasons this has been actually built this gives coe folks a center of excellent folks really a view into uh the situation across environments or in this particular case of course for that environment but for all uh power to make desktop flows and their respective actions so if i go back here as i said here you will see a subflow the action name which always is reflecting here in power automate desktop so those are the subflows and those are then the specific actions what we see here is exactly the detail level and then the duration in the status but going one step further if you i would like to have a look here for instance if we go back to our power to my desktop flow and i look at one of the uh really pro code actions like you know calling web service so if i double click on that invoke web service action uh here i see okay i have a streaming url and here i have the request body and the request body is actually at this json object so i can i would have to go into each individual flows and then review that stuff and do my auditing and so on but i can uh by accessing the dataverse backend as we said before i can conveniently conveniently do this directly from this dashboard experience if i go here to scripts and i drill through here we'll see okay we had a write to command session action called which was calling a host name then there was a python script executed from dos command we had a run application action which was then calling python and then obviously some parameters uh powershell scripts here vbscript some javascript or even python scripts verb as part of this specific flow here right this gives a really a huge visibility over the whole uh command in action history if you would like to look at that on a general you know on a higher level so for all flows then there's also the script action monitor which allows you to filter that by a specific desktop row or even an action name if you're looking for only bb script for instance or run vbscript you can filter that here of course and then it will show here below all actions which involve vbscript in here for instance this is a an action uh which is calling some sap automation and then you can review okay are they doing something specific something complex here or is it using what transaction code is it using and just you know then maybe to proactive approach again those folks say okay we might have some standard interfaces for that or maybe an api so just to to drive that discussions and have that visibility what do you think folks i think that's a lot of scripting that's amazing though that you can dig into those individual steps and and show every piece of information that's passing through that desktop flow it's pretty awesome indeed i mean that's you know the kind of visibility uh the organizations or you know larger enterprises at least look for when they try to enable the service for the whole organization and having a reactive monitoring kind in place which gives them really uh a lot of flexibility when it comes to auditing and reviewing of those processes okay so this is from an auditing perspective so also seeing and you can really define i've just filtered here so if you open that thing here i've just selected which of those actions i consider to be pro code if you want to debug two through you know with other ones if you want to see okay how many times a write the exit command have been called you just i can filter that and then they will show up in the respective uh lines here so this is as i said completely flexible and up to the consumer to adjust to their needs so the next level will be okay now i know what actions are you are being used how about looking at uh you know process variability and also execution variability so that you know you see early signs if things go wrong or some of the processes run really longer than expected or they are failing consistently or consistently over a period of time so for that purpose you can hear filter then for instance if you have a particular one let's say desktop flow process vendor invoices and then you can limit also the amount of uh so the time frame you would like to look at let's say then you have a week's overview of the data you would like to have a look at and this is what the if you know the the chart would look like with the information you have filtered now here from here on you can then say again right click and look at that particular day the executable execution of that particular day by drilling through to the details and here you come again to that view which is filtered now for that specific uh process cluster on that day all the execution and here you see immediately okay there was something wrong here as well as well i would like to look at that look at the scripts and then see okay there was some some run application happening only some system in for who am i and so on you can read it and you review and look at the error so from here on as i said before you have always a link back to the power automate portal to look at the specifics for that specific execution and here you see if i scroll down i can click on that one there was a run subflow and this has failed actually the real action which has failed is further below here and this is the problem so this gives you really you know uh a link always between your custom reporting you can always play back to what the reality is of course in real time on those power automate desktop and power portal statistics okay and from here again if you would like of course you can then drill through again to web services and so on back in the performance chart as i said this is the historical view you could get and then you see of course here always the duration and the this this bar here next to it always reflects the previous days uh or the previous periods uh execution count all right and this is then the distribution and the change over time you see there as an indicator of course this one is maybe not a stable process it might have you know different inputs and outputs so that's why the processes run more often or longer but if you have really stable uh process with a stable variability of an input and output this will give you really good insight on on what's going on then the next level if you are sufficient in you know analyzing those those points then you can let's filter here to succeed only you can also look at a more statistical approach on that data right where you see here let's start from the left hand side you have here upper and lower control limits and then you know a central point and this is then controlled how many standard deviations your process is allowed to deviate from from the mean right so if i want to control my process with uh three standard deviations or sigma's i can open that a little bit okay i can say this is still now under control my process or if i want to uh drive a little bit more narrow uh execution and you know tighter execution i can even go to one or two uh sigma level below and this would then show okay i have an outlier on these on these dates these three points here and then you can look and dig again deeper on on this side okay and here's just the violin chart which shows you the distribution on the different hosts you are having where where those particular flaws are run on make sense yeah the level of intelligence here is intense my goodness i think each time you click down i'm like how can it get deeper my gosh yeah i mean you know and that's really not rocket science uh john right so this is normal and that's the power of power bi and actually of the dax language so uh this dashboard has been pre-loaded so i've done a couple of these measures as they are called which are really giving you the opportunity to look at data from yesterday for instance previous quarter month over month changes which we will see in a second so those have been defined and you can see here the code you can change this but it's also a good template to learn maybe those technologies and how we can use that you know without any coding or statistical uh backgrounds because powerbi and the dax language give you a lot of uh things out of the block where you can create for instance new quick measures which have a lot of this intelligence building so you don't have to think uh or to learn the language really from scratch right where i see okay average per category time intelligence or uh you know month over month changes quote up or quarter changes and so on so that's all built in again in power bi uh so there's tons of videos of course available in documentation on how to do those but that chart is meant to show you okay how you could address those specific reporting needs uh if you wish to okay so this is then the view you see on the on the portal when you have a flow execution right so header flow execution these are the actions and then durations and the stables so if we look now at this action performance day over day or month over month this will give you a different intelligence now again on action level but it will tell you also for that specific time frame uh what is here for instance the previous execution length for instance duration for vbscript for instance and what was the execution of the previous day and then it calculates the day over day changes right and you can look at this on action level and of course time period level or by host by action and so on so forth and then the same goes for month over month changes where you see okay for specific actions let's say you always having you know an update update script which does some uh month end uh transfer of data and you know always okay it's the same amount for instance stable amount of employees and a stable amount of data which should be transferred you can see okay performance of that specific action which is then calling maybe a web service on on the other end and then you can look at that from uh from from a really uh month over month shade day over uh day change and and so on okay and then the other three those were just the drill through uh reports you have seen uh before so this is this is all good and nice so imagine uh you have a power automate portal for our analytics where you can see not only of course the actions as you might have noticed we have here a filter which is also hostname which is actually the on-premise data gateway or a cluster and the same information we have here of course also the portal right if you go here to our data and then on the gateway side we see a list of on-premise data gateways or actually bought bot hosts and for targets so single on-premise data gateways or here we have two clusters one is a windows 10 cluster and this is a windows server cluster right so if i click on that then i see i have two participating clusters in that gateway and here i see again per club per a machine or per cluster the previous execution of my pad flows okay and here i did the three different sessions so that's that's good and that's nice uh the problem is let me start one of that of our cloud flows which is triggering so that cloud flow you will see in a second is triggering in parallel three uh power augmented desktop flows which are executed on a specific cluster this server cluster i've just shown you right it's executed on that cluster and because i have set here that flag run on all gateways in the cluster it makes a choice depending on load balancing where to run uh the the bots okay so these are the three uh uh bots i'm having here or the same bot sorry but about the three executions of that so i have three accounts here which is then accessing those machines uh accordingly so when i start now that flow and this has been newly added in December so they're really monitoring a cloud execution on on gateway level so if we go now here let this start okay so this has started as you can see here so if you go now to the gateway overview i go here to monitor and then desktop flow cues what i will be seeing here is then first of all again my gateway cluster gateway a gateway cluster how many gateways are part of that cluster and how many if i go to the live updates here how many of uh those clusters or gateways are currently executing flows and here we have two desktop flows which are cued and one which is currently running so if i click on that you will see really again the live update uh depending on priority uh if we go back here these three one has called with high priority high priority normal priority and this is of course then reflected here in this queue as well the problem now with that is that that's good and nice but you don't see uh okay which machine has been executed on you don't see what is the anticipated duration of that specific flow that happens here and what is the current context let's say you have you know a month and closing a power automate desktop which has really long long running reconciliation process and you have a loop with maybe a thousand records and so on but it's really running maybe two three hours it would be very convenient to have a live view or at least uh uh you know snapshots of the current execution state so i'm in the 15th loop for instance and i'm currently working with that category of data so in order to enable this again power automate sorry power bi has been uh used together with power automated desktop to define a so-called real-time streaming data set so what i've done here in power bi in the service i have created a real-time streaming data set which has these you can freely define them you can add and create and you know just as your data schema requires i've just defined a couple of properties here on the real-time streaming data set which allows you to collect information and save that either so if this plug is on then the data is stored on the power bi service if this historic flag is not on then the data is only cached for an hour and then it's gone so this is of course uh you know for purposes like iot monitoring and so on where you don't have to store maybe eventually data but you would like to detect real-time anomalies and so on so that's what i've done very simple i defined a couple of properties here and then i uh this is generating automatically for you an api you can post the data against right so here is the body of the data if you're using a c url or powershell you can do that accordingly okay so then what i've done on powerautomate on that specific so you would do this live monitoring of course only for specific boards you don't want to have that for bots which are running five seconds or 10 seconds or really not critical also from a sequence perspective but if you would like to do that then you could send uh those you know monitoring uh events or those status events to the power bi service to the streaming data set through an uh in invocation of a web service here on power admin desktop and that's what i'm doing here so i'm collecting cpu i'm collecting memory i'm collecting couple of telemetry which i think for that specifically for that specific flow uh would be valuable so this pro actually is a demo flow which takes an excel sheet with a list of countries and writes this into another excel spreadsheet right so this is a list of countries here very simple and what i've decided here for that demo flow that i would like the current context whenever it's looping through that i'm always sending to my power bi streaming data set the current country so the current role in my excel i'm i'm i'm working with okay so in order to show you that um you can define a so-called dashboard power bi dashboard which can host live connectivity to those services sorry to those service execution as you can see it has just changed because i have submitted a couple of new flows right so if i'm running those again here let me resubmit those okay so this will take a second until they warm up but for that we have our nice view here you see one is running one is uh queued up and the one the other one is the next two to the right so if i go back to my uh power bi uh live dashboard here as the execution happens those records will will come in and then you see here the last action which has been called is right to actually you see now it's in austria it's running now and you see if i want to look at the at the the specific row i can see here also the progress so it's live updating as i said every couple of seconds you can define this you see the memory consumption and also here the cpu consumption on that and you see the expected loop count of that process should be 196 loops because that's the amount of countries i have and i'm currently at position 80. so you know you can judge okay it will be finished maybe in two minutes or it might be finished in 10 hours right and this level of view where you see of course also the cpu utilization uh a free memory or whatever telemetry you would like to to get from your power to make a desktop host you can derive that here and then you can filter by uh action by host again and so on okay that's the and then you see of course here the development over those machines i can look at the specific machines and so on and from here again if i would like to know okay what exactly that specific flow was i can go here again to the flow history and this is as it is executing showing we then of course at which stage and position we are in the cloud flow sequence okay so this is rounding up actually uh john the uh you know real-time processing with more uh you know a a history processing of uh the analytical uh data i love it okay so you see it's still executing i have two buds to be finished and this is what you can see here directly from this dashboard as well okay any questions no questions i'm i'm overwhelmed that's a lot of information yeah and as i said no no c-sharp no python code needed to to do these things it's really configuration and the access to the cds of course back-end that's cool yeah pranav you've been quiet what do you have to say i am mesmerized in real time i know that awesome this sort of fills up a huge uh sort of gap that we had from a customer perspective you know we have the out-of-the-box analytics but that's sort of meant for a general purpose audience and gives you a sort of one lens into your desktop and cloud flows but as you start to write more and more these business critical automation on the platform and your the level of sensitivity that you want tends to be a lot more higher you want more more real in real time both insights and auditing so this gives you a nice sweet spot layering on top of the out-of-the-box analytics all the data as possible saying we're still in dataworks so it's the same source that we're connecting to it's not that we're copying the data over to some other data source and now we have this data fragmentation you know problem that you're running into it is just a different view over the same data set you know without having any c-sharp python or data science skills so this is like beauty that's why i was like uh totally mesmerized it changed real time it's amazing love it yeah it's really great i think uh you know i'm gonna ask the question that i know everyone watching wants me to ask and they're gonna ask how do i get my hands on this and so for those watching who have been mesmerized as well uh apostles how do how do others do this yeah it's a super question i think yeah you know the whole reason what i said before is to give this extra level of control to flow and desktop flow makers but also the ce folks and citizen developers whoever would have access eventually to that back-end data and that to that level of data can of course make use of those things and in order to give that beauty to everybody that's why it is called also starter kit and we are planning to announce this soon and uh yeah provide this downloadable component as a power bi template all you need what i said before is access to cds backend with sufficient privileges and then having those two urls and off you go this is for the desktop flow analytics in order to build the real-time dashboard right this is a different dashboard as you can see here and i have i don't have it open that's a different one because real-time data sets streaming data sets cannot reside in the same power bi desktop file as the other ones that's why there are two files but i'm planning i can not say any dates i'm planning to do because it's quite simple to do i'm planning to do a blog post on how to replicate that on your own right this real time that well because it just takes what i said before a schema you just these two actions here to provide something to that power bi streaming data set and then on the power bi service to define that schemas with this field you have seen and then you can monitor whatever you like uh from an execution perspective yeah but they will be very very soon to come yeah okay all right so you heard it here take a look uh i'll update this video when it releases so we're gonna put this video out to show a preview of what's to come but keep an eye on the description as one place i'll go ahead and update it with the links when it releases also keep an eye on the power automate blog i assume that's where the blog post will release right Apostolis yup perfect okay so keep an eye on that as well if you don't want to keep coming back to the video uh the news will launch there so you heard it it's coming soon you'll be able to download a template where you can plug in your information and get some of this goodness yourself and that to me is the most exciting part about this whole thing perfect okay so some final words uh john on because the deck has been called also behind the scenes or backstage um so leveraging database this is how i've realized this whole thing we have said that there are three core entities you would like to look into workflows flow sessions and file attachments uh flow workflows uh this is where the desktop flow uh are stored flow sessions is the execution history and in a file attachment there's the action context of those flow session executions okay then i've used the web.contents for those who know power bi because it allows you to handle
different http statuses but also allow you to include of course the odata annotations to get from database not only the ids but also their respective names for that then in order to do an automatic or schedule an automatic refresh on the power bi service you have to use parameters with relative paths and so on to to allow for that right it showcases also some medium medium to to advanced data processing commands in the m query language so if we go back to our dashboard and we go here to to edit the query just as an example many of these things uh you know have been introduced just to showcase of course so if i go here to the actions and i look at the advanced editor there are some things in there like if i'm create if i'm getting the action context that includes sensitive information like for instance a client id or client secret or anything so i can really obfuscate that information in the transformation process and this is really showcased here with a couple of those script commands you see here so that's why it's also a good showcase on how to do maybe power query and m transformations then this is also very important data privacy fundamentals so you cannot just say i'm ignoring the privacy models because this is of course a security risk but on top of that if you mix it up let you have for instance for your connections set the privacy on one data source to private and the other one for instance organizational you will not be able to uh publish so you will not be able to automatically refresh that thing again on the power bi service and there are some in terms of where i said it's a good showcase to to do that yeah you know and also to yeah to add to it that's a very good point also because in your flows also for folks who've been using sensitive text as input output both for desktop flow and cloud flows you know all that data is not logged if you mark them as sensitive to those that data does not get logged in in dataverse it does not show up in those dashboards as well so do sort of take a look at the uh the privacy fundamentals don't take no there are a lot of controls available both from a maker's perspective and from a cv perspective on it yeah absolutely true so whatever we see in the portal you see also here of course eventually right so some things like what i just described with client id and client secret and so on so those are things if you define them on the portal and they're of course not hidden then you would see them as well and this is where i'm doing some minor checks on those but you have of course to go into detail to your data and then make your according your judgments whether something is considered be sensitive or not and then filter that out and so on so this is not a fundamentals course on on security but uh just to show you that there are means where you can restrict that on on the power query and processing level already yeah so then there's uh the dax measures so the measures i've shown you before so month over month changes and so on and it leverages of course once it is deployed to the cloud also the os awards to authentication um yeah findings for those who are very well versed in dataverse we have uh introduced a concept which is called tds endpoint which allows you to connect to a database environment uh in to a read-only sql instance if you would like right and this is of course very important for for reporting so you're doing offloading all your reporting load from the production uh environments and from production execution and so on and its respects again uh security roads and so on so forth now the current implementation of tds endpoints doesn't expose the flow session entities that's why i had to use the web contents route uh instead of the dataverse or common data service standard connector we have there we are planning of course to support this in the future but there's currently no no eta for that uh okay so it might be that the cloud flow history retention differs from desktop flow history i think this has been rectified already and for the large flow run log history if you receive a 413 error this is usually if you have super super massive large uh legacy uh action flow runs uh which has been of course resolved now in in the current version but this is for the history you're having there if you receive that then you know and the dashboard the first page will surface uh those errors in the action log here so it won't fail they will just tell you okay we have five of them well i think the the important part here is there is a lot of power and you will be able to get your hands on it pretty soon uh as we talked about before keep an eye on the blog keep an eye on the description of this video i will update it when this comes out and you can get your hands on it in the meantime if you have questions for the team if you want to pick apostles brain about it a little bit go ahead and leave us a comment or a question down below we're happy to come back and answer those as best we can all right this has been it for episode number 13. you guys know what to do go ahead and click like and get subscribed so that you don't miss another video and that's it from us we'll see you in the next one you
2021-02-18