Data Hub X-Ray: What's Inside
Hi welcome to the data hub x-ray session. My name is john scalomero, and i'm here with my colleague georg, i'm the arc manager of architecture, and governance, here in product marketing at mendix. And georg, it's good to have you here why don't you go ahead and introduce yourself. Yeah sure. Uh i'm georg i'm a product manager for data hub here in rotterdam. So today we're going to go and dig in a little deeper, around external, entities we're going to talk about what they are and how they work, we're going to show you how they. Work within the studio pro environment. And then lastly we're going to talk about performance, and security, applications, of external entities. So gerard can you talk to me a little bit about what the goals were or the challenges were that we were trying to solve with external entities. Yeah absolutely. So one of the key, problems with integrations. Is that it simply takes too much time. Um, we've solved a lot of these challenges with low code and specifically, with mendix. So a lot of a big of a chunk, is already taken out but, still. You have a lot of, time consumed, in a project, it's usually around 30 percent. And, one of the reasons is that. Integrations. By themselves, are very complex. You have to uh. Um. Make sure that you standardize, across. Your, organization, to make sure. That, you can reuse, uh integrations, as much as possible. But at the same time you have to also make sure that you build secure, integration, they need to be performant. And especially, robust in an enterprise, context. To be able to do that you also require a very very thorough understanding. Of the technology, that you're using, so the combination, of, a lot of research, up front. Understanding, the architecture. Finding the right people to have that conversation, with. And then putting this all together, in the context, context of a project. Consumes, way too much time, so. What we're trying to do is make all of that as easy as possible, for any kind of developer, in the mendix, scope. Yeah that makes a lot of sense gary so but how do external entities solve these problems. Before we started, working, on data hub and external entities we set ourselves a couple of goals. We just, didn't want to repeat what's been done in the past. We wanted to make sure that we get the most out of the experience, when working with integration, so. One of, the goals we've set of sales is that, when we build integrations.
Or When, our, users build integrations. It should be done as quickly as possible. And not only the professional, developer, usually there's a handoff, even with manyx developers sometimes, to, an architect, or what we refer to tech tom. That just extends, the time frame to building, uh to build an application, so you create a an organizational, dependency. So we want to. Increase, the range of developers. That build and use data through integrations. Also we wanted to make sure that when we look at. Making this easier we look, out for standards. There's a lot of, integration, technologies, out there but, if we want to make this as easy as possible. We need to find something that is reusable, in the market and also known in the market. And, at the same time, the, consumption, of that external data should be simple and consistent. So, building, a lot of bells and whistles, to it does not really help. And also following, the, general approach of mendix. Making sure that we abstract away the technology, as much as possible, but at the same time, we want to keep your options open because what we don't want is to box you in on something. We don't want to trivialize, the integrations, work so, we will, make sure that, whatever is. Makes sense. We will. Follow the principles, of mendix, and help you, model faster. And give you the tools, um that you need. Okay, okay so, so with that being said. How did we solve that. So. The, approach we've taken is to look out uh look at in the market for uh, for standards, and uh we, we we looked at. Obviously what we want first and uh. I said, before we wanted to find a standard. A standard that helps us achieve all the goals and, the one that came closest. Uh is oh data. Odata. Open data protocol, is widely adopted. Um, it follows. The most common, technical. Integration. Capability, which is based on rest. But extends, it at the same time. You can almost think of odata being a query language. Similar to sql. So, you not only get the robustness. And, um. The maturity. Of rest as integration, technology. But also the capabilities. That you need already included, in that standard. So. These, additional capabilities. Could, be actions. Um similar to get methods for example it could be change sets or callbacks, aggregations, a lot of these. Typical capabilities, that you end up building in your application. Are already included in the standard. And the best of all odata comes with a metadata, model so that means that, when you use odata. Similar to soap for example. It describes. What it entails, the domain model but also its capabilities. So by, looking at the metadata, model, it's, fairly. Easy. And straightforward, to interpret. What, the integration, is about, and that all led us to using odata. For our, data external. Entities. Okay that's great key arc can you show me some examples of how odata works. Yeah absolutely. I uh i pulled together, a quick list of some of the, most common examples, um.
When. We, register. A service. In in data hub, we analyze, the metadata, model. So we know exactly what is included. In that specific, endpoint. And, that, metadata. File plus the additional metadata, from the catalog, is then provided. To. The, development, environments. So they can understand. What. The, order the service provides. We've also built a passing mechanism. That translates, the odata. Specification. Into, the mendix, modeling language. So instead of, i said earlier fiddling with the urls. You, get the same experience. That you have with local entities, on persistent, entities, you get a graphical, visual model. Of your data. And, all the capabilities. That you would use in your application, for example. The filtering. It's basically, a search what you need behind a search. That is, embedded. In. The external, entities, so you don't really use them. Specifically. Or. Directly. It's more that, we, made sure that the. Development, experience, that you know, of, in mendix already, is replicated, to external data as well. So the runtime, interprets. All of these requirements, and requests, for you, and the modeling, experience, stays the same. Okay georg that sounds pretty good but but can you kind of tell me. It sounds like. There's a lot of scope, around oh data there's a lot of. Parts to the standard, that it's pretty robust. It covers, the metadata, covers querying, and things like that. Can you talk to me a little bit about what the modeling, experience is going to look like yes, i think there's uh. Something, uh even better than talking about it. I can show you okay, um, awesome, so, let's take a look at how that actually works, so. We can take a look at modeling, experience. For external entities, but also. Look under the hood a bit so, let's get to the demo, what we have here is the studio pro. With a new pane the data hub pane which allows us, to search directly in the data hub. Datahub, will come with a, set of sample data sets. We've. Created. An hr. Service with employees, departments and offices. So anyone, can. Explore, and start modeling, with data hub. To be able to get there i just, have to search for, sample. And. The result will come up. The data hub search will make sure that you find data as easy as possible. And studio, pro will pick up the search result and make, makes it as easy as possible. To model with that external, data, so i can expand that service see what is available to me i find the employees. And just simply. Drag and drop them over. On my canvas. You see from, the look and feel, it is almost the same as, a, local. Entity, or, a non-persistent. Entity. The color is a bit different, it's purple, and it has an indicator.
What Type of application, is providing that information. Other than that. By dragging and dropping we get a bit of additional information, already. With. The external entities. Because we need to know how to connect, it comes with. The link to the environment, that provides the information. But also. With an opportunity, to. Configure, it further, so that screen will allow you to set security, information, for authentication. We will come to security and performance later, so, i'm going to move on from this for now, and focus a bit more on the modeling experience. As i said, it looks the same. Apart from the color, but you can almost do the same, as with a local entity so if i'm looking at the data model, i can remove. Information. Let's say i don't want to contact. My employees through email anymore because whatsapp, is the new. Type of communication. I can just remove. Well you have to go with the times. So what you can do is um. Access the, the complete, uh data model. Similarly, to any local um. Or non-persistent, entity. No problem at all. You also, are able to find, the. Hidden or deleted. Attributes, again, so it's not really. The same, 100. In terms of experience, but it is adopted, to the odata. Service, and metadata, behind the scenes. The same goes for associations. We already know from the search results, there is not only employees but also departments, and offices. So if i want to include that of course i can drag and drop the office and department, there, but i can also access that information, through, the entity. Document. And the window, right away so if i want to extend. My integration, to not only pull in employee information but also departmental, information, i just simply, check it. Accept it, and, the data model is built, for me. The interesting thing is that not only that is the same. Almost anything that comes after that, is also the same so let's, say we want to build a really really really simple application, just for the sake of this demonstration.
That Shows me, those. Those employees. What i need to do is. Build a data grid with that information. So i can do that from the home, home page. Drag and drop a data grid onto it it's empty. My connector. Already, shows me. The entities, i've modeled, i've pulled in from datahub. And, similar, as any persistent, or non-persistent, entities, i can drag and drop them, onto. My data grid. Same look and feel, still get asked the same question. Do i automatically. Automatically, fit the contents of the data grid yes i will, saves me some time. And i do get the usual error message that. I need to have, a page, for the details. Now i can. Build one myself, but the nice thing about mendix is there is a. Nice feature. That just for testing, or for even, wider, purposes, i can create, pages, directly from my data model, i can do that with external, entities, as well. So for my employees i'm going to create. Both an overview. And a detail page. Same, experience. If i go back to my home page. I can associate. So if i go to my. Home page again, with my data grid. I can, select an on click page, and just simply select the one i just created. Error gone and now we can run it so. Modeling, experience so far, pretty much the same i would say, it's interesting how easy it is, and how it combines, the external data use with what we're already able to do with the mendix and low code so that's really nice. Very good so let's take a look at the application. I'm going to start it and, to make sure that we can follow, what's happening. I'm also going to bring up the console. I'm going to trace. Those. Odata, calls, being made to. The sample data sets, so, let me expand this one. And what we need to do to see those is, to set appropriate, log levels. Um if it's unfamiliar, to you you could basically. Set the right lock levels on, almost anything that's happening, behind the scenes, what we want to do is make sure that whenever all data is consumed. Um, we're going to, trace, what's going on, that will help us see. See the calls that being made, from the external entities, so let's clear up everything, and then. Go to the application. And uh, see what's happening. So. The mendix application. Will then query, the appropriate, amount of, records from the external system, not all of them it's a million, just to show that we can also handle a lot of data which is no problem with odata. And that call, that is being determined, by the page, not by the external entities or the data model but the page so the application. Is then translated. Into, a specific. Odata, request. And this can be seen here. And this is. Very similar to the examples, we've seen before.
So You have the url. You have the endpoint. And. Obviously, also, the entity, which is employees. And everything that comes after the question mark is then, the. Typical. Notation. Of, the rest for, odata, interface. So we know we want the top, 20. And it's ordered by id. Why is it ordered by id i haven't specified, anything the only thing we know is, we want. The top 20 out of everything. So if we want to, see how that behavior, changes, and we can go back to the application, say, just. I don't know sort by first name. So i, change the application, behavior. And, the next call is being made which is fairly similar, without. With one, difference. I. Change the order by. To first name and then id. That. Action. Was not specified, anywhere in my model. By me at least, it's already. Defined, by the external entities, that you can do it, by just. Building a page. Actually you don't even have to do anything because, building a page with mendix, already incorporates, that and it did before, and now we can do the same with external data. It doesn't stop there we spoke a bit about the capabilities. So i can do the same with anything else that you saw on that page. Another thing that. Odata allows you to do is paginate, and we've built that in external, entities as well, so i can simply go ahead and skip to the next, 20, employees. If i want to understand what's going on. I get another request. And that request, basically, says, give me the top 20 still because i want to see top, 20 in my on my page but i'm skipping the first 20.. Standard, in odata. Implemented, in external entities, there's nothing you have to implement yourself. It's already part of the modeling, experience, so if the page, that you design, is supposed to paginate, the rest is taken care of you already know that from. From, persistent, entities. It's the same experience, also for, external. External entities, and external, data now. You can even take that a step further. By. Filtering. For information. So, i have my million, employees. That i want to search through and let's see if john has a namesake, in that database, so if i search for john. It comes back with, actually, fairly, a lot, because. It is not a fixed touch. Um, but what happens in the background, is that exactly, that. Query is being built automatically. So if you look at the. Search, string. That's being appended. You can find it here in the odata. Query. Filter, substring. John. All of that, again driven through, the, modeling experience. Modeling experience, when building a page. It's not technical, at all you don't have to add anything to the odata specification, or the integration. Directly, it's been taken care of for you. So that is basically, how we, handle odata with external, entities. It's all about the modeling experience, and making sure that we abstract, all. The technology, away where it makes sense. As i said security, and performance. We do want to leave you in control we'll handle that a bit later and then we can speak about the concepts too. So there's one more thing i wanted to show, as promised. Often cases you want to, build applications, with external, data and, local data so persistent. Information. We will allow you to do that obviously but we also wanted to make it a nice experience, while modeling. While still making sure, that, you stay in control of the architecture. As, soon as you relate. Local data with external, entities. Uh external, data you create a dependency. And. You don't want to end up with an application, that caches, everything. Just to make sure you have you can handle those, relationships, or associations. So the way it's done, with mendix is if we go back to our domain model. I can simply add a, local entity here so let's say i want to add. Time cards to my employees. With a specific. Start, date. No problem. I have my, my. Persisted, local entity now, and what i can do is, relate, the two together. As, they were local entities, so modeling experience. Is not different. Now. As i said. Behind the scenes. Different things will happen because, if i now relate a time card to an employee. I still need to keep track of that employee. But i also don't want to cache all the time, otherwise i create a. Frankenstein, a monster application, nobody wants that so what happens behind the scenes is if i run this, again. The changes, are applied to the application. And. Mendix makes sure that. The domain, model is. Is reflected, also, in the database, model. And, i can show you real quick how the caching. Mechanism, work and speak about the performance, aspect. A bit later, so. In case. You know it when, the application. Is, using the, built-in. Database. You can always access. That database, straight from studio pro, i'm going to take a look at the database, model this will be the same regardless, of what database you will use.
It's More about the structure. What we'll see, is. In. The viewer. The entire, tables. That are behind the mendix application. And. What we have here is, the time card, table. This time card table is the one that we would expect because it's local data, we need to keep it. Somewhere, and this is this is the place. But, we will also see. An employees. Table. But if you look at the data structure it does not, show, all the attributes, that we've seen in the model, we only keep the id. To make sure that when we access it again or relate, information, later, that we have something to track it with. So one of the requirements, obviously, is that, if you have an odata interface, that you want to connect you through external entities, you have to have an, identifiable. Id. So if, we want to take a look. How this works, fairly straightforward. Let's, see how many we already have. In. Our database. So i can query for the count. And it's 138.. If i now go ahead. And maybe just paginate, to the end. Also no problem. And, run this query again. It's 140. 58.. So the 20 were added. Now. Not going to go into details because we didn't really relate. Data yet don't want to end up building the entire application, but, the data structure is still there for us, if we choose. To. Put timecard, data, in and related to the employee. So here you can see the corresponding, table that's already prepared. With the timecard, id and the employee, id, so at some point later. If you decide to associate, it to in your application. Logic. The data. Model is already prepared for that and from that moment on external entities will also take care of the caching. And the performance, related aspect of it. Very good so this is what i wanted to show you. I have to say gay arc that's fantastic, i really like the part where you showed the sword again filtering and paging, uh having built that myself several times, it's really nice to see how quickly that came together. Um, security, is usually a really important, aspect, to how we build our software. Um so can you talk a little bit about, how security, works. Uh in data hub and with external entities. Absolutely absolutely, so. One important aspect, before. We dive into the security, aspect. Of external entities. Security. As always, relies on the publishing, end, so, the goal of external entities is not to take over security. Um, or replace, it in the publishing, app, it's just making sure that whatever security, model is chosen. Can be supported. So for. External, entities. The security. Options. Are, similar to other integrations. So you can choose. How to integrate. Whether with a using a password combination. Uh, token. Um. Or, any form of identity, federation, that's supported. You will find those control, options, directly, at, the service that you're consuming. Um. To a point where you can, create your own header information, or consume your own header information to connect. So that's a very important aspect that. Whatever you chose. As a security model on the publishing, end, can be consumed. Through external entities, at the same time. So there is no extra model that needs to be built it's incorporated. But it doesn't lock you in at the same time. It allows you to follow, the publishing, end so to speak. Okay so that's really interesting. If i could say it a different way what you're really saying is that, the downstream, systems, that own the data, also own the security, model over that data is that correct. That's the case absolutely. Awesome, okay so so that's going to make our lives a lot easier because, it will distribute, security. Uh distribute the security models but still make sure that everything's consistent. Um, but, what about the cases, where, the security model isn't quite that clear or clean. Yeah that's a very good point, um in the best case, scenario. You federate, your identity, through the downstream, systems. So if you log into a system a and system a consumes from system b. That identity, is clear throughout the chain so you would, not. Gain access to more data, than. You would, by. Accessing. Application, b directly. That's not always the case and, those, those cases, can be tricky but, bendix built access control, on any entity so, if you've modeled with with mendix.
The Persistent. Entities allow you to be very specific, on access control, and we've. Um we've added that, capability, also to external, entities, so even if you cannot provide, the full. Granular. Access control, in your source system through identity, federation, you can still, lock it down on, row level, for example you can say this user has specific, access on uh to that specific, external entity, so. Even. In your. Consuming, app, we want to make sure that applications, can build in the most secure way. So that's great that really goes back to what we were saying earlier about our goal to make sure that we never paint our customers or ourselves. Into a box when we're building this. So what about performance. Very good topic. We've just seen. An application, being built uh with, a service that provides a million record and, was, fairly quick. I have to say. Um the beauty about odata, is and i mentioned in the beginning it can almost be seen as sql on top of rest and rest itself is very performant. And audit itself does not really add. Much to it usually you get a bit of an extra through a query language, on top, without it it's it's you can't really, feel and see it so, the, page by page, comparison. Um. Is. Not really, visible. Um what's also interesting, is that um. When you talk about large, volumes, of data, as mentioned in the demo you don't want to end up. Replicating. All that information. That would defeat the whole purpose. You also don't. Want to. Be in a situation, where you lose, the relationship, so we needed to find the right balance in the middle and, what we've shown in a demo is basically the leeway. To a very performance system where we. Where we reuse. The mendix. Caching system. So we treat external. Entities, or the objects. That come through external. External entities, also as mendix objects. So. Based on the page, usage. And. The usage in general. We know what's being used or not, so when we cache information. More than the id that we store in the database. We keep it around as long as necessary. And only query the information. Again. When the caching system, thinks. It has been a while. Could have. Got rid of it earlier ago, if it's being used. Very often, then we keep it around, obviously, and that is very similar. Very consistent, to the behavior they would see with mendix applications, in general. The modeling, experience is great. So thank you so much for showing that but also, i really appreciated. Touching on security, and performance, because i know those are things that that, developers, of all stripes get beat up on all the time, and it's nice to show that. They're going to be enabled and empowered to successfully, navigate, that through. So that being said thank you garrick again. And thank you all for joining our medics world 2020, session here on data hub x-ray. There are three other sessions. For you to attend during mendix world here, we have an overview, session about democratizing. Data. We have a how-to, session. About how to use data hub in your day-to-day work. Also, please feel free to reach out to us via the medics world, chat slack channels. For these sessions. And. Mendix.com. Data dash hub. For getting more information, and letting us know that you're interested. Thanks again. Bye. You.
2020-10-07 15:15
