Real-Time 3D HMI for the High-End Yacht Industry | Unreal Fest 2024
Welcome, everybody. Today, we are looking at how we use the Unreal Engine to develop something interesting for the HMI point of view. Today we are me as a 3D supervisor for the 3D department of Applied. That is our company. We also have Lorenzo, that is actually my boss and the supervisor for all the department about 3D and videos in Applied. We have Davide, who will be happy to answer all the technician questions because he is a team leader for our department of vertical people that develop in Unreal.
And Igor that will help me because he is one of the people that actually developed the system that we are going to see. And just a quick introduction of who we are and what we do. We are a digital company intent on helping our customers to digitalize their flows through pipelines and innovating their world. We are mostly centered on industrial application on the manufacturing and the packaging, but we are also seeing something in very more creative applications.
For our main business unit, we are an interactive one where we apply the most creative part of our activities to produce innovative applications for our customers. This is our industrial innovation part that is more focused on more technical products for enhancing the technological advancement of our customers and two very big business unit dedicated to classical enterprises, application and infrastructure, IT services and everything. We are about 300 people divided in the various application. 20 people in all for Realtime 3D specialists.
So actually, vertical, we use Unreal Engine in various kind of products. Then the most interesting part, we partnership with the Wider Yacht. Wider is a company of yachting from Italy near where also is our headquarters. They build various kind of luxury yachts, from the small one to the very, very large one.
This one that we see today is about 30 meters long, and it is a very interesting boat and we will see why. Approaching the yachting industry, we find out a lot of interesting similarities with our markets, likely the fact that they use very similar technology, similar to mechanical machinery, same kind of PLC, the same kind of protocols and everything. So we were already ready to produce something for them. The very interesting thing is that we found out a very ready for innovation field because, as you can see, this is actually a boat from Wider Yachts from a couple of years ago.
You can see that they already have a very compelling and visually stunning interiors, but they still are using a very old style, two-dimensional HMIs and that will be very interesting to bring some 3D point of view, some new functionality to them. Obviously, there were some kind of challenges to this new field and we will see them one by one. We had to design and imagine a new kind of HMI for them that have to be incredibly visually stunning because it's a very high-end luxury field.
But also it has to maintain an extremely effective use because captains of the ships need to use the ships to drive them, to go into ports, into the world, and it has to function very, very strongly. Being on board of a yacht, obviously you have some power consumption and footprint limitation. You can't go there with giant servers, with giant video cards. You have to be similar to automotive, but with some liberty, with some flexibility more. We had also to manage a lot of data because all the HMI that you saw in the image previously, every system on board the yacht have its own HMIs made by the people that made every subsystem of the boat.
We wanted, and our partner wanted also to try to create an image that cover most of the subsystem of the boat possible. So we had to try and connect every PLC we find out and try to collect all the data and then sending them instructions and everything, and it was a quite challenging. And lastly, the boat is on the sea, so you have an environmental challenge to face the weather, the salt air, and a lot of problems to have. And in the end, we produced this solution.
We have called it the Pilot project. We're distributing the word under the Future Marine branding and actually is not only in a 3D view, but is already on board of the first yacht. There are little video with one of our colleagues that is trying and a photo from the Venice boat show of a couple of weeks ago. But we will see more in detail. To show you how we did it, we will start with the design of the system from a UX/UI point of view, then how we develop the system with the help of Igor, and how we deploy the system to the real world.
So the main issue was to build a very visually stunning but also very large and complex system of panels, details, data, and everything, more than 60 pages of data that we use and more than 600 variables that we manage in and out. We developed the system in two main components. The main part, the most interesting is the a couple of 3D views of the boat and the main system with a collection of the most interesting data that the captain can have much under control freely, and then a very large system of two-dimensional view pages with some overview for every system and a lot of detail pages for assisting. These are the three main view for the 3D visualization. We have the classical homepage with the collection of the most important data and one electric and one for the fluids.
And this boat is very particular because it's actually a fully electrical boat. It's like an electrical vehicle, so they have a couple of electrical motors to propel the boat around. They have on the top of the boat a collection of solar panel to charge the batteries, very big collection of batteries, so they wanted to have an overview all for the electrical part because it's very complex and a lot of system that do something. And then a classical view for the main part for the fluids with the various kind of waters and the diesel, because, obviously, it's an electrical boat, but you have to have on board a couple of diesel generator for when there is no sun or when you want to do prolonged trips. And there's a video of how we interact with the 3D part.
You can see that also the 3D is connected to the various system of the boat. So you can have an interactive, real-time visualization of, for example, the levels of the various tanks for the water on the upper side and the various diesel and the gray water underneath, and the classical 3D view for having that impactful visual representation of the boat. These are a couple examples of the overview pages for the two-dimensional part of the interface. This one is actually a visualization of all the electrical system of the boat from the batteries, the consumption for the hotel part of the boat. Hotel part is likely the rooms for the guests, the galley, the classical system, and the spaces of the boat.
You have the generators and the propulsion systems and also a classic view where you can choose in which mode the boat is, economy or cruise or full power for having maximum performance of the yacht. Here are some example of-- those are pages with a lot of data freely available to the captain, but not very exciting to view. Very classic.
We took all the old system that they are doing and try to give them a new visualization, more exciting. And thanks to Unreal, we were possibility to enable our designer people to be autonomously to autonomous, to build all these pages and scale up with other system and everything, because they don't need to have access to developer teams. They just have to design their elements and use our templates to build up more pages for new boats with a different system, different values, different variables around and everything. And lastly, we also had to manage the various alarms and events that they both will rise up from a classical list log style to a more interesting way to show the captain what is the part of the boat that is having problems, to enable them to identify the problem quicker, and obviously also to have a visually interesting view of the system. Here is a little video, full screen, to see how we can interact with this HMI.
Our designer, thanks to Unreal, could have much time to design a lot of little functionalities to show more details to the captain, like when a system is on or off, where the various flow are going, if the batteries are charging or discharging. It's actually very similar to a hybrid car. Sorry. This recording was from our screen, so this data is not the real data of the of the yacht. And now we'll have a look more in detail about how we build the system.
And Igor is the best person to show it to you because he's the one on the team that helped the most to develop this system. Hi, everyone. I'm Igor. I'm a software developer with Applied and I was part of the team that actually connected Unreal to the boat. The main problem was right at the beginning. Like, how do we get unreal engine to talk to properly talk to a boat, both in receiving data, gathering the data from the boat, and also sending commands because the normal HMIs have modalities to change the mode, the cruise control, or even precise parameters of how the motors function, like the speed.
We divided the challenges that we had into three main categories. These were not the only challenges that we found, but they were the most interesting to present. We divided it into connections, so how we talk to the boat; security, how we ensure that our system doesn't break down while it's in the middle of the sea; and performances, like how do we work with the existing boat systems as seamlessly as possible to not use up resources that could be needed for the practical purposes of the boat. Starting with the connections, the project is divided into three main areas. There is the Unreal application which handles all the visualization of what we have done; the Python application, which handles all the connections with the boat systems; and the actual boat systems. For who isn't of the field, basically, all the bot functionalities are handled with onboard computers that are mostly called PLCs, that are really low performance computers, but that handle all the logic of everything.
So a light switch being turned on, that's the PLC that does the gives the power to the light bulb. How did we manage to talk with the boat? Basically, we developed a multi-threaded system with the Python application that uses the modbus protocol to interact with the boat. For each PLC we did a separate thread that keeps getting updates from it and sends the right information to it. In Unreal, we developed the application using a mix of blueprints and the C++ blueprints, mostly for the widgets and for the visualization logics, like the changing of the cameras. And we use the C++ to handle the data structures and the connections with the Python application, which was done using the CPM. We have two different connections for from Unreal to the to our Python server, one to receive data from the server.
That is the least critical because the refresh rate is very high. So even if we miss a packet every once in a while, nothing happens. And the other one, which is the most critical, which is the right information.
Like, if the captain changes a variable, they will want the variable to be actually changed. So we can't afford to have errors on that part. So these distinct connections talk with Python and Python has basically two main threads, the first thread that handles the gathering of the information from all the other threads that talk to the PLCs and another thread that is just to handle the right data because that was very important. As we said, we talked with the PLCs using modbus, which is a proprietary protocol of this kind of system.
And we worked alongside the PLC producers. In our case, there was two main computers on board, one to handle all the electrical stuff and one to handle all the fluids. So there were two main partners that actually did all the rigging of the boat and we interacted with them to get the memory addresses or where do we send the data to actually make the boat do stuff. That was an overview of how we talk to the boat. Here you can see the blueprints that we used.
We have used our extensive workflows that we developed over the years that were built upon year by year. So we have a series of internal tools that we use to make the workflow smoother. And we developed the custom code in C++ to actually handle the connection and data logic. Going back to the second challenge was the security. So as we said before, how do we ensure that nothing goes wrong while the user is in the middle of the sea? We basically made the application weatherproof. What do I mean by that is that every system can fail and it itself will restart without impacting any other system.
So for whatever reason, the connection to a PLC is gone, automatically it will retry to restart it. And if it can't, it will signal to the user that there is nothing wrong on our end, it's actually the bot that is that has some problem. To ensure that for whatever reason, even if the computer that we are running our application on will still run, even in the case of malfunction, we have two computers on board, one which is always the main system and the other which is a backup system.
We talk with [INAUDIBLE], the client, and the PLC manufacturers to understand a series of use cases in which it will be useful to automatically switch from one system to the other so that from the user experience it's seamless. Like, something breaks, we have a loading screen and then we change to the other computer, which is now working because it understands that the other malfunctioned, and now it can talk to the PLCs. Other security features was all the handling of the exceptions that can happen because of the improper programming of the PLCs. So if someone doesn't follow the protocol as written, our application doesn't crash.
So even if something random happens once in a while, it will just signal it to the user and they will know what to do with it. Another important thing is that everything is disconnected from the internet, so there is no way that someone would be able to connect and disrupt whatever interaction we have on the PLC. We ensure that we are not an effective attack vector for someone to hijack the boat. And that was basically it for the security.
As before, I forgot to turn the slide. These are the visualization that the user sees once something goes wrong with our system. Normally, if-- we worked with the PLC manufacturers also to display the alarms that are normally displayed on the HMIs and we displayed them as well on our system. So we have the alarms of the boat. We integrated our alarms alongside the boat's alarms so that they are seamless between one and the other, and it will just display to the user. If something goes wrong, we will signal it to the user.
And if something goes really wrong, it will just say, here, press the button. Now you can try to restart everything, which has not happened yet. Lastly, we talk about the issues that we had with performance, and more than the issues, the choices that we made to make the system as smooth as possible. Basically, it all started with the choice of the computer.
What do we put on the boat that will handle Unreal? We decided to go with a nautical-grade computer from one of our partners that is running on an integrated graphics card. Why did we use an integrated graphics card? Because there is a huge problem with cooling. The computers are in the same room with all the other electrical stuff.
Even with air conditioning, it gets quite hot. So there is a lot of work done for the computer to keep it as cool as possible and we decided to not put any more load on it with a huge graphics card. Other than that, we worked alongside the PLC manufacturers to understand how to be as less impactful as possible on the PLCs because, as we said before, those are critical to the system to actually work.
So if we take up all its resources just for data visualization, then it wouldn't be quite good. So we worked alongside them and we discovered that with how this computer works, it's better to batch as many calls as possible so that you have few very big intensive calls rather than a lot of very small calls. So that's it. And lastly, for the performance, we really enjoy working with Unreal because it enables us to do the work once to make it available on basically all of the platforms. So our 3D models and artists just did one version of the boat as optimized as possible and with the various parameters.
And if we decide, like we decided to run it on a less powerful machine, we just have a less interesting visualization of the boat, but it's the same application. So we will be able to see we use the same assets to do a lot of stuff. So the client had to, to pay for the big work of 3D modeling once and we were able to enjoy a lot of different products. And that was it from my end.
I leave it to Mario to say how it went. Thank you. After developing the system, there were the deployment phase. Obviously, there is nothing more to see than our colleagues with the screwdriver installing computers and everything. So there are a collection of pictures of our teammates when they were testing the system and installing the various PC underneath the cockpit.
And when we were testing, you can see the industrial PCs where the system is running and how big these kind of both are in the real world. They are very, very, very big. And in the end, these are a little video showing you the system out in the wild. [VIDEO PLAYBACK] [MUSIC PLAYING] - [NON-ENGLISH SPEECH] [END PLAY BACK] OK. As you saw in the video and one of the main point for which we chose to work with Unreal, apart from it working with Unreal from many years ago, it was the fact that everything we build for the HMI system with a very, very low effort, as Igor said, could be ported to very different system. Like, we developed two other apps for PCs and iPads with various different applications, but covering all their needs for our yachting clients, like the configuration application where people were already both the yacht and go around the yacht and select every little features, every little materials, and everything for their boat and make it their own.
Actually, we have a recorded video with you can see more precisely. You can select a very interesting part of the of the yacht. It's actually a very simple configuration. I believe everyone have seen once on a time for a other parts. You can choose an option like if you have a BBQ on the deck of your boat. Who doesn't want a BBQ? And then a couple of other options.
If you want to learn more about this application, obviously we have a demo in our iPads and you can ask our colleagues. And the other application is more about the captain and the technician people aboard the yacht to help them to visualize all the systems and subsystems of the boat and a visual representation. But also we can see, if you look at the video, you can have a multiple kind of visualization, and just one of the classic exteriors. And the most recently one is right now.
Because having all the exteriors and interiors of the boat and the various systems and plans, we can keep it all together and enable the people to go around the boat, walking freely, and use an X-ray vision of the rooms and see where all the system will go around in the boat, where you can reach for it, where is every subsystem that are actually behind the walls. So without this kind of representation, technician people maybe doesn't know where to have to go to have access to various parts of the plans. And this one is one of the two engine rooms. And you can see a lot of systems. And actually, you can click on the main component of the systems and have access to a dedicated view for documentation, documental parts, video instruction, video tutorials for every part of the boat, enabling the owner of the boat to have a much more useful experience using it and maintaining it. Also, the owner can upload his own files during the lifecycle of the boat to have a collection of every documentation and the materials they collect and maybe pass the system to the new owner when you sell the used boat.
OK. Voila. We have said it all. Thank you for being there to see our system. And whatever questions you have, please, we are here to answer.
[APPLAUSE] Thank you.
2024-07-31 00:20
