I will also turn on the transcription and it is for accessibility reasons later of course. Alright, I think the recording is started. Everybody welcome to the Frontend Lightning talks. We're here today again for yet another great session. Yet another you think? Yes, we often do this. I will show you that in a bit. Some housekeeping rules, again session will be recorded. Please mute yourself when presented are presenting is lovely to hear their voices loud and clear so we can enjoy the show. And of course, if you have questions, put them in the chat. We can answer them afterwards or in the end of the Lightning talks you can turn off your mic, turn on your microphone and ask them in person. So that's always great as well to have a little discussion going on. What I just mentioned is that we often do this and what we what I mean with that is there is not our first rodeo. We have some big front and pens with different subjects. This year different speakers but the same awesomeness. In general we are trying to do on every two months, and as you can see the 9th of February. This was a past event with JavaScript and frameworks. Last was in April about CSS, UX and accessibility.
But now we are in for the hybrid and native development of course, but beware nothing is final. But in August I can already tell we have GraphQL and Serverless of course. So at that meeting will be announced very soon as well. So if you also want to present the session, of course you can always contact the meet up organizers. That's always nice to have. This way we have external and internal speakers talking about internal speakers. We're from so steep it's night company. It's part of Capgemini. We love technology. We love our community. That's why we do this. I like this as well. I like the coziness of the small sessions and just in general, technical knowledge that we learn here so you can sit back, sit back, relax, and enjoy rapid drink, sit in the sun. But still learn a lot. We do a lot of different things for a lot of different people and a lot of different customers, but we do it all together so we make technology. Work is very important for us. But fair to do, let's continue with the program. That's what we're all here for. This is the first introduction to welcome session. The announcements will do that a bit. Then we will start with native versus hybrid, which is done by immune. After that we will talk about about how Philips is adding AI to native apps using tensor flow. That will be done by Martin. If this. Then in the end we have a special announcement, so that will be really cool. It will be a collaboration with. I will not talk about it anymore because it's in yet. And completely afterwards, if you still want to discuss something or ask a question in person, we will stay here and chat for a bit. Make connection cracking joke or library like it's a freewheel session, so that's always good.
But let's look at the percentage for today. First up, we have a meal In Sync and then we have more time to freeze these two heroes of service. The are presenting the content and to be honest I am very interested in what they're going to present. So without further ado, I think Samuel the floor is yours. You can share your screen and show us a good show. Alright, let's give it a shot. You have any questions you can put them, of course in the chat we can do that afterwards.
OK, check question. Can everybody see my screen? Yes, me, oh, you're doing fine. OK yeah well never trust you finish the longer I work in. I think the less I trust software so. Good question. I don't know if that's a good thing or not. OK, so let's start my name is Samuel.
Come. Yeah, I mean uh, mainly an iOS developer it socially I work for some clients working on their native iOS apps. I have a lot of experience with all kinds of other platforms and things. My career started as a back end developer and we're all kinds of sites detours. I ended up as a mobile developer when the iPhone came to the markets. I joined Sogeti in 2017 after working at a small agency for a few years. Well, as I said, my work is mainly focused on iOS. But today we're not going to talk about that. Today we are going to talk about what if you don't want to do that, but if you want to make an app which you don't want to be limited to one platform and one technology. So we're going to talk about the differences between native and hybrid apps. He is not to explain all technologies, not to explain all the consequences, but to give you enough to make you make an informed decision whenever you need to choose a technology for an app for yourself or for client or whatever.
Uh, I have to leave at 8:00 so I won't be there for the discussion afterwards. Probably if you have a question an it's not a long winded, very complicated question then please just interrupt me. I will ask a question to give a shout whenever something happens on the chat that requires my attention because I cannot really read it. Uh, so well, let's get started. So you want to have your content available for many peoples. Then you basically have two choices. Nowadays you can build a website or you can build an app. Websites are mostly used for general content about your company. Things like that an extra more personal.
So for instance, if you're an insurer, your app would probably contain all the product steps. The person has and things like that. They can arrange things for themselves and the website will be more your what do we call it? Fishing took action, I forgot the English words for that. Your business cards. Let's call it that. If you want to make an app, you have two platforms that you have to make it available for. Nowadays you have iOS and you have Android. There used to be a third in the form of Windows Phone, but that's no longer there. I think it's a bit of a shame that it disappeared, but it makes our life a lot easier because now we only have two platforms to keep him accounts. There are more mobile platforms, but those are so small that typically, yeah, you don't really have to do anything for that. It's less than 1% for some Linux foundation. Things like that.
Technologies. They are quite diverse for web. You mainly use JavaScript or TypeScript which you compile to JavaScript. You can use CSS and HTML an that's why you can make pages and you build a beautiful website on the back end for your website. You can also write that in JavaScript or TypeScript or something similar to that. So when you know those three technologies and one or more web frameworks, now you're golden. You can make. Typically anything and presented anywhere. It's it's really wonderful for apps, things are a bit different because you have to know different technologies and different languages.
And it's a different way of working. So I guess you have to learn either Objective C or SWIFT or C++. For Android you have to know Java or if you want to make a modern app you have to know Kotlin. Or you can also do it in C++. Uh, development tools are different for Android you have Android studio. For iOS you have explode. An it it takes a bit of time to learn to properly build a native app for such a platform.
But yeah, well, those things are called native apps. Uh, they have a bit of well pluses and minuses for native apps, performance is always great because you're building it as close to the operating system of the platform vendor, which is Google or Apple. You have typically less weird books. The platform tools themselves are quite well ironed out and. The software that you make, the frameworks that you build them on are typically well testers. They are used by everyone, including the people who use. Uh, who make the hybrid frameworks which will get we will get to in a bit. I think the resulting app of roll this top is a bit opinionated, so I will you may disagree with me there. I see my namesake already smiling with a big smile so. Uh.
I think typically because you use the native tools, the apps that you make are easier to use. It's easy to follow the platform guidelines. It's easy to add accessibility to them. It's easy to integrate with all kinds of fancy operating system specific stuff. For iOS, things like augmented reality are quite impressive if you use them correctly and you have better integration with unique hardware stuff like camera mode, switch and switch, front camera back, camera wide angle cameras, LIDAR sensors and biometrics, and all kinds of things. Things that you can do a with web technology sometimes, but it's it's more extensive if you use the native integration for that, the app state you built are mostly more resource efficient. So for the user that's a benefit that there are battery lasts longer and you can have more apps in memory at the same time. So the operating system will not kill apps as often because they are based on the amount of memory that they use. Of course this has a dark side also. Because everything you make native is not platform independent. That means that you have to make one app for iOS and one app for Android. And if you want to have a website with the same functionality, you also have to make that. And that makes it quite an expensive exercise, and that's one of the things that a lot of clients have a bit of difficulty with.
Uh, together with the latest, the last point, and that is that it can be hard to find people with the right skills if you have an Android developer, you cannot typically ask them to make your iOS app also, because they typically specialize in one of the two platforms or they have a strong preference for one of them. So things that we actually heard from our clients is yes, I want an app, but I don't want it to be too expensive. I already have a bunch of developers which I spend a lot of money on Wednesday. Make my app.
Uh, do I have to have developers with different skills? Maybe you do, maybe you don't. And the last argument is mostly I already have a website. Why can't we use that? Well, it turns out it in a lot of cases. You can, and that's the first kind of hybrid app where you use web technology. So you take your HTML, your CSS and whatever. And your JavaScript of course, because you want things to be dynamic. And you package them into an app you have, well, 3 popular platforms for that. There are more of them, but those three are the most famous ones. I think the oldest of them is Cordova. It was originally started by Adobe and it's now part of Apache's software stack. You have a more modern version of it, that's the little blue icon on top of that, it's called capacitor. It does the same thing. It's even largely compatible with it.
But it's it's a much newer code base. It's a lot cleaner. It uses some more modern technologies and you have ionic, which is based on the other two an which brings the tool kit with a native look and feel. So your app will start to look like an app instead of website that style differently. For developers, it's wonderful because you can use JavaScript and CSS and HTML just like you used to, and most web developers can make quite a decent app with this technology stack. Of course it has also pluses and minuses. A big advantage is it's cross platform. You can write your code once and it will run on iOS and Android, so a lot of clients think. Well, that's beautiful. Now I can say 50% on my cost. And thereby they forget that there's a lot of overhead in project management and meetings and everything, so it's not a factor. Two different difference, but it is cheaper than building native apps.
You can use existing skills your Web developers can make your app. And if your app is not too difficult, if it's something that you could build as a website, you can perfectly package it this way and make an app. For development, I always feel a bit limited with these platforms, but if your app is simple enough that you don't have native integrations, you can develop it in your browser, and that's quite nice.
Because the development tools for Google Chrome and Safari are quite OK nowadays, so it's a nice way of developing and it's really quick to iterate that way. And there's lots of third party support. You have all the JavaScript toolkits and everything available, so yeah, you can save a lot of time on developing repetitive stuff or things that are part of your framework. This is founded just. Yeah, it's a website in disguise, so you can do things that you can do on the web. You can do some negative things if there's a bridge for it, but you can't expect to do everything, or you will have to build native code anyway to make a bridge too. Uh, to expose that to the web browser that's in your your runtime basically so Cordova. For instance, it's embeds a web view an it has a bridge outside of that where JavaScript can communicate with native components.
Uh, tooling as I said, I think it's not great once you have to get out of your browser. Native stuff is hard to test. You have to typically dive into the native tooling anyway. But the way you get there is a bit difficult. And integrating with device specific functionality can be quite easy if somebody has already built it for you, because then you can just import it via NPM or it can be incredibly hard if you have to build bridge functionality yourself. It's really annoying and I've seen both cases where sometimes fenders give a really wonderful SDK for Cordova or capacitor. And sometimes they say well, now we don't support this, but here's our native SDK and good luck with it. So depending on your use case, it can be wonderful or can be horrible. Yeah. It's still a website in disguise, and if you don't do your styling correctly it will look like a website for a lot of apps. That's OK because a lot of apps don't look like app anyway, but if you don't want to have that, there are two kids that make use of native user interface components. So.
The way this slide is built up is on the right side is the web and things that are on the right side armor web related things that are on the left side armor native and you see that slowly with all the toolkits we crawl towards the native side. You also see that the flexibility and the way that things are the pluses and the minuses of the toolkits. Become different. So if you want to have your Web developers that know JavaScript build your app that you want to have a native user interface. Well here I have put the logos of tree toolkits that can do that. One is appcelerator that's the triangle with the A in it. It's a commercial product, it's not well supported.
It's pretty awful and I would not recommend using it, but well, it's there. The big square with the N is Nativescript native script is nicer. It's somewhat a community product product. It's tries to use native components, but it also has its own user interface toolkits, which works a bit nicer because that's toolkit is platform independent. And the atom thing on the top is react native, and that's actually the best of the three because it uses react as a framework which most web developers will be familiar with nowadays.
Uh. And below that for rendering you can use native user interface components. It also comes with a small set of components that are platform independent, which you can make platform dependent. Use interfaces and the only platform dependent code that you will have typically. It's a user interface code, a little bit, and you have a bit of well, native integrations. Maybe same thing goes as far the web frameworks. If you have difficult integrations, it may not be the best choice. If you have a vendor that supplies in SDK, it's wonderful because you can do things really quickly. Uh, well, something it up. Cross platform web developer should know JavaScript so they can make things you can. Also you use TypeScript if you like that.
It will look and feel familiar for users because the user interface is native and if you use React Native community support is is quite OK. There's a lot of stuff available and a lot of stuff that you don't have to build yourself so you can build on top of that. Performance is a lot better than a website, but it's still JavaScript which is in not native thing and native user interface that need to communicate with each other and that takes a little bit of performance away. Tooling is a difficult thing with React native. They typically just drop you in the in the native development environments and good luck with it. Integration with device specific functionality. Well, if you have a vendor that delivers it, it's wonderful. If it's not there, then you have to build a brick yourself and it's a bit difficult maybe. Performance can be a thing, depending on the kind of app that you are building. If your insurance company and you just show a list of somebody's insurance is, it's going to be wonderful. If you're trying to make a game or something, it could be different. Why is this approach slower? Well, if you have native code, then the developer write something in Java or Kotlin or Swift.
And they compile the code into the app and the compiled code is converted to machine language and that runs natively directly on the CPU and the device. So there are no translation layers it can communicate with the operating system directly. Now, britches know difficult things and it just gives optimal performance. It's the shortest route you can have to watch your hardware. For JavaScript code, that's not really the case because JavaScript, even though you don't necessarily run it inside a web browser, it's always abridged off separate runtime component that the operating system supplies, but it Shields it from the outside world. You can talk to the outside world again for a bridge. The picture on the right side.
Is the previous approach that react native took to accomplishing this? It's, uh, outdated, because nowadays react native has done a lot of performance optimizations to make this nicer, but it made the picture way to complicate it, and this is something that's the same for most of the web based tool kits, except that you don't have JavaScript itself, web browser and a bridge in between. And it's also for native script and accelerate or it's the same approach. Uh. And, well, all these communication and every layer that you add it. That's a bit of slowdown. Moving closer to native.
We still want the performance to be OK. Uhm, what I found there are two quite common toolkits for this. Uh, one is similar in its by Microsoft. You can make your appusing.net code so you can use C sharp. Uh, it's quite nice. It has an extensive UI toolkit of itself. The other one is flutter. It's by Google. It also has a nice native UI toolkit, except that it's fixed the rendering because it just draws something that looks sort of like androids and sort of looks like iOS, but.
It gives an OK result, but the language that you have to programming is called rest, so it's a dialect of something that you have to learn. So what we see is the closer we move towards native. The more you have to learn to be able to use these toolkits or the further it goes away from the technology that you may know from building websites.
Both of them are cross platform look and feel familiar, compiles to native code, both of them, so performance is actually quite OK. Uh, integration with device specific functionality is OK. Both those platforms make it easier to bridge code to native really native stuff in a platform independent way, so that's OK. It's supported by Google and Microsoft accepted as far as I know, Microsoft for all their mother and apps. They use electron for desktop apps, so they are no longer using summer in themselves, but it's still a supported product. But the same thing that goes for native apps is that you have to have developers that know the languages that you have to use, and C sharp developers and rested fellers are kind of hard to find. C Sharp developers are popular in the financial business so they get a lot of money there and they don't go to an app developer. You need to learn the platforms you need to learn the libraries that they used to make platform independent user interfaces and integrating with third party software. Again, it can be really hard or it can be really easy. What I notice is that a lot of vendors don't make SDK is for example in or flutter, but they do supply something for React native or for Cordova or Ionic. So you're actually for third party stuff. Better off on the right side, closer to web.
Then to watch the left side where you're close to native but not quite. And the last, of course, is a native app perfect performance. You always have the latest technologies. Right now Apple is holding their be feleppa conference, the worldwide developer conference, and they have announced a lot of nice new stuff. And since I'm a native app developer, I can download the better versions of the software and can use this all today. I don't have to wait for my platform to get updated. Tooling is awesome. Uh, Xcode and Android studio. They they work really nice. Debugging tools are powerful and nice and you have all kinds of analytics tools and everything around it. You can do automated user interface testing and automated unit testing. It's all built into those tools so you don't have to grab a whole bunch of external stuff to get things to work. It's it's all there, you just get it for free.
Third party support. It's it's the best ever be 'cause the native platforms are used a lot and every Fender if they don't want to make a cross platform SDK for one of the hybrid platforms. They just make a native one and they let you build the build your own britches basically. But you have to build 2 apps, one for Android, one for iOS and you have to of course know specific languages and toolkits to get things to work. So can be a disadvantage depending on the people you have and the amount of money you want to spend on on getting the right people or hiring them from socially. Of course, that's always an option.
What we see is that what we call hybrid app. It's not 1 technology, it's a whole bunch of technologies and they're all called hybrids. Depending on what you want to do, you may want to choose one that's closer to native or one that's closer to web. In practice, for most apps.
Uh, we lean towards the Cordova and React Native products. Actually because those are well supported, they are popular for Delphi, helps you to get started easier with your website and react. Native is nice because it's based on react and you can get a lot of experience developers for that. So not one hybrid app, it's a range of technologies. So if we summarize this, this is the sheet that you can use to make your decision. In the end, we want great performance and great user experience. Do you have something difficult if you want to do that, you can do only natively. Go native and then yeah, it's expensive to develop, but that's the only way that you can go. Uh, for all other cases, go hybrids because it's cheaper to develop. You can build cross platform stuff. You only have to build it 1 1/2 times, so I love it's for most apps, more than sufficient and you have a wide range of technologies to choose from, so there's always something that you can use. Basically, this advantage you always a little bit outdated. You're either limited to what you can do with a web browser, or you're limited to the things that the defenders of your toolkits exposed to you. If there's a new version of iOS, it takes awhile for these platforms to get updated. Same goes for Android.
And you have things like support, which can be tricky. React native is made by Facebook, and it's not the most reliable Fender to build your whole business on top of. A filter is by Google, which also has a track record of dropping things if they don't find them interesting. Anymore. Thing with React native is that it's also a community effort so. If Facebook decides to drop it, it will probably live on for flutter. That remains to be seen. And that's it for now, so questions. Thank you me. Oh, very good presentation. I said give it up forum. You live in a cool applause analytical and I will clap as well. That's nice, well done. I'm you. I didn't see any questions in the chat, but I personally, really liked to see the landscape of the native and hybrid development. I didn't know much about it, so that's really cool to see and a personal win for myself. Native apps don't use a lot of battery, at least lesson hybrid apps. And it's important for my mobile phone.
Sounds great to hear. Are there some other questions? Maybe they want to speak out loud if they want? That's also possible. I see, Umm, you opinionated native developer yet. Yeah I have a question. If I if I may. Yeah, go ahead.
Do you always use Xcode or do you use a something else to developing like echoed? Uh, it's a client. I work now. 90% of the team uses Xcode because it's, well, the first thing that gets recommended when you use it. We have one developer that uses app code and he quite likes it because it's based on the same toolkit or in the same foundation that Android studio is built on. So those like refactoring and code completion and everything. They are a lot better than an expert on on that thing. On the other hand, it's not always as up to date as it should be, especially at the moment when new versions of SWIFT I released. It takes awhile for code completion to catch up because. Yeah. Yeah, they have to build build it in. I guess they have to make a database of completions or whatever so it can be a little bit annoying if you're always running cutting edge. Otherwise it's quite a good alternative to extract if you want that, yes. Yeah. OK, thank you. Nice, actually. There's not a question in the chat from Sammy. She can speak out loud. Sorry I don't know.
If not, then I will question it. Sure, it's a he. Cool. My question is. I I've been thinking of making the transition to become a iOS developer because I also feel that making native apps can, well. You can get more performance out of here. You have your app, but how much iOS developers in demand and that it's 'cause most of the clients have been talking with. The hugely want me to make a hybrid app or a TWA. So which opinion? Is it a good thing to do? Yeah.
Unfortunately, a lot of large companies are purely based on on money and they want to make an app as cheap as possible and they don't really care that much about user experience or the end result for the user as long as they have an app that does what it does, it's it's OK for them, and that's fine. There are some other companies. For instance, I've done the app Forex New York, which is a travel guides. For people who want to go to New York, it's really nice. Single screen app everything is met based and they explicitly asked for a native app because they wanted to have perfect animations. Perfect interaction with the app, really fast performance, fast searching, fast grouping of items and everything. So there's still, there's enough demand for native developers also.
Uh, if you're interested in getting to know more of them, I would recommend joining cocoaheads. Cocoaheads is a meter especially for iOS developers in the Netherlands. And a large group of iOS developers. Come there every now and then. If Corona is over, they will probably start hosting live meetups again and they used to have 1 meter every month at a different company. So you get to see something of that company. You get to see something of what they do, and you get to. Two other iOS developers, and sometimes it gets interesting leads and job opportunities, and you can learn a lot from them. Also because a lot of them are quite experienced. It's an interesting thing to go to.
Uh. Thank you Kate. Can you put it in the chat like progress? Is that with a C or mean after right? Yeah, I should put it in here. That's great stuff. Thank you for your answer. Yes, good question as well. Last question goes to Lawrence.
Yeah, sorry had to put on Mike on. Mr beginning so sorry if you already told something about this, but you said when you use ionic and capacitor, your app will still look like a website so. So which features or characteristics make an app look and feel like an app? Yeah, that's a bit of a. Difficult question. We're at my at a previous client that I worked for. We sort of held a contest where we would take a bunch of apps and it will try to guess if it's a native or hybrid app and if someone want technology it's built. You can download the APK for Android and you can pull them apart and you can see what technology they use.
And The thing is that what gives you the way typically are built in animations. The way the the user interface components look and feel. Mainly feel are different for for ionic apps, even though ionic those. Uh, there's quite a lot to make it look and feel native if you just have a static screenshot, you're unlikely to be able to tell the difference. But once you start using it it feels a bit slower. It it's interacts a little bit different and that that gives you the way. If you use Cordova.
It's basically just an HTML runtime with a wrapper around it, so you have to do all your styling yourself and depending on how good you are at styling for mobile, it may or may look more or less like a website. And I've seen a Cordova app stuff where basically just somebody's website and they packaged it in the app and that's it. That's a bunch of HTML files and that's it. On the other hand, I think uh untapped the beer tasting app is also based on the technology, and it's quite impressive what they've done with it, and you would be easily fooled to think that that's a native app because it's done really well.
So it it also depends on your skill level. And how far you want to go with it? Nice, hopefully that answer your question, Lawrence. Great stuff. Alright, uh, thank you. I'm you again, uh, I think it's time for next one and it will be more time. With this we will talk about how Philips is adding AI to native apps using Tensorflow multi are Are you ready for it? Because we are. Awesome. Yes, I'm ready. I will stop sharing my screen will second. Nice, take your time.
Screen. Uh. So. A question to you is do you see my screen and K we do you hear me right? Yes. You are perfectly loud and clear and we can see your screen.
So, uh, yeah, welcome, I'm holding a presentation about the feed lips MCC app. Later on I will tell you about where the MC stands for. Uh, first of all it was not working that my name is Martina Chris. I'm a software engineer for 33 years and eight days active for show steer the moment first June. Three years ago I was hired and at the moment I'm a full stack developer. Also do mobile and iOS. Uh, because the first two years for shosti I couldn't find the mobile app to work on or mobile mobile software job, so I developed full STACK app for national Lambda.
And ah yeah, the last year now I'm working for Mobile CEO a we do yeah simple apps for for a lot of customers. One of the customers just started Phillips. Then of course I'm doing that for now. Uh, other small customers we have. Uhm? So the first thing that's what I'm not, I'm not a data scientist. I'm also not a data engineer or a data analyst and I've little to know little to no knowledge what I've learned. I've learned mostly on this, uh, this job. So questions about data I really can't answer, but you can always try, of course. First of all I I I'm part of a team for a Philips MC app.
Uh. Hello. Uh, other team members are deeper. Massage and Colin. Those are the data scientists and data engineers behind object detection models and the other models we use in our app. When will Tina? I'm not not in this light. For shame is also helping on the development side of the iOS app we're making. Yeah, then what is Phillips and she up? I'm she stands for Mother child care. The meaning the meaning of the app is to create a small working prototype to predict volumes in a bottle. Taking a picture with your camera. In the film, the user, should I receive the results quickly and is able to provide feedback on my prediction.
Uh, ideas that this technology can later be used in the existing. A smart app that is already there for the assembles the you can see a picture on the left side of the image that is a smart bottle that can measure the amount of milk in the bottle. The sensor it's quite expensive, ultralight checked and the idea is that this app can also be used with normal bottles from Phillips to measure the volume inside even later container for breast breast pump milking. Uh, I can't show you that later, but not in this light because it's at the moment a prototype and I don't know if I'm allowed to share that fully with you guys, but I can maybe show picture later.
The idea of the app is that, uh, sorry, did the scope of the project itself contains of? Three models, uh, classification of the bottle type. This classification list object detection model we call. Generally it also creates a bounding box to crop the image or the the second step, the regression model. The regression model is based on. Based on a CNN, a regression network. The regression model itself outputs a volume and that can be shown to the user and then the other model is a segmentation model and that is the second model. Use of the second technique we use in this app to create a.
Uh, predict predict the volume for the user. So two pipelines are available. I will show the pipelines in more detail later. The app itself is a simple iOS app with inference on the device itself, so the picture from the camera is used. It also has a user feedback loop. The pictures it sells are uploaded in S3 bucket on Amazon and the images can be downloaded by the data science guys and used to retrain and update the models. Uh.
And the the idea the idea eventually for this app is to to give insights of how this technique works. An advice for the future for the for the Philips team, and maybe implement this in the existing app. Uh, like I said, there are two bottle types for testing on the FM portal 125. There's the bottle in the center of the picture, and a WBB, and that is the container I will show you later, maybe in the film when I get when I'm getting the demo. Uh, for the moment we're developing the app only for iOS, but the techniques were using should also work for Android and all the results are also recreatable on Android that is native. Of course not in that app.
Next screen, yeah, so there are two possible approaches. In the moment we're using, we have one step prediction on two step approach. The ones that prediction uses the object detection model to detect first the bottle in the image and then crops it an the grouped image will be imported as input it as a input in the regression model. The regression model itself then outputs a number that is multiplied with the maximum volume in the bottle. And that is the output of volume to the user and have a two step approach. The two step approach is on segmentation model, so the app.
Today's segmentation more scope the bottle and that mask is used to separate milk. Part of the bottle from the bottle part of the bottle and counting the pixels that is milk. Divided by the pixel status only, the bottle times the maximum capacity show show what? Yeah, what amount of milk you got in a bottle. Uh, the ocean challenges challenges with. With both approaches, the training for aggression is lot harder because you need a lot more data and pictures to go. Yeah, to get a good reflection of the amounts of milk in the bottle. So for every milliliter, or actually actually actually.
Every two of MLS, as we're training it, you need milk in the bottle and you need to, yeah, pictures of that to train the regression model and for the segmentation model you only pictures of the bottle with milk in it. The only downside of it is that the model itself will be quite a bit larger. Because the the generating masks is more challenging. Come on then.
Decided to purchase. I will also show the app later on in the demo, but the idea is that you got two inputs for the object detection. Uh, we processed image before that by resizing it to the input of the object detection model for us that this turn the pixels by two more pixels and we formatted to a flat bitmap so it can be used in the mobile itself, but in not in JPEG or other formats. And then we put it after that. Regression model the regression model itself has also some pre processing the processing done on the. The progression is that image needs to be padded with black and then cropped of course to the bounding boxes topic detection found.
And after that the output of that image is multiplied by the amount of maximum volume of the bottle and that gives you the total amount. Uh, for the segmentation classification or segmentation model, we just bet and resized image to the input of the model and then we get the, uh, the same kind of picture as output. Uh, and then we have to, uh, calculate for each pixel. If it is a bottle or milk and a separate separate those two and do the sum and check the amount of volume like that. You can see again the two types of prediction in a more visual way, so we take this also the picture of the container. We take a picture of the user, takes a picture. We pre process that by resizing it and putting it in a model object. Detection first creates a bounding box of that and put it in the CNN regression model and output of that is shown to the user. If you follow the segmentation classification you see that the segmentation creates a.
More, uh, uh. Uh, abounding, uh? Yeah, so segmentation of it and then shows A and we then calculate of course motor prediction is. Uh. Wow, OK.
Uh, you can see some pictures. We took the last days and then training at and you can see the differences in but it is looking at that and in the story in a regression model you can see here that. Maybe take a picture of the bottle and when it is cropped or resized to the the the bottle itself that the darker blue parts are the most important things in the picture of things and here it is looking at mostly at the top of the milk. As you can see. Uh, so actually it's not looking at the milk itself. The error is sales zero. We still don't know why, but it is looking at the empty void above it. So things that emptiness is more important than the milk itself and you can see of course, that sometimes the. Russian goes haywire and now it detects the kind of shadows inside the bottle and also shadowy over here is. Clearly show in this, uh, sort of important.
Choose to to predict it. Then you can see the amount needed. This is quite off. Uh, with the container? That's also the case. In this one you can clearly see that the blue part is the milk itself, so that goes OK. So sometimes the regression model kind of works on the container, but sometimes it goes haywire and hear. It clearly sees that unless most important, so the milk is kind of ignored probably and amount is way off. For the segmentation model, use a segmentation mask. You can see that is the shape of the bottle. This is the output we processed into an image, so we pulled all the image for all the pixels that were not important to black and all the pixels that were light blue is milk and all the pixels that are dark blue is the bottle itself and you can see. The shape of the bottle and it back, but also the amount of volume is calculated.
By getting the the pixels divided by the milk. Also following. Now I've got a small demo.
I prepared to app if it's possible. Yeah, you can all see the screen on my phone, right? Yes, we can see the screen. This is the field of some shape. I will move my mouse. It is still a proof of concept, of course, and it's entirely was only for now.
Also buy some Phillips guys to test it and to upload pictures to the bucket for training. Uh, I mean I start. I got the camera now. First of all crates you, I'll try it. But then of course the bottle I prepared a bottle with some milk in it.
For now I'm using a just normal cow milk breast milk is I talked more yellow and is not used in this training at the moment, but I can of course be used then yellowing milk of course. And other people also used formula, but that kind of gives the same result. It's maybe a little thicker, but milk is milk at the moment. And if I see happy I got an option to regression of segmentation.
For now I select regression. I can take a picture of it. The processing only takes a few seconds. Now the predicted phone was 96 ML. It's a little on the low side, so then can put in the real volume. It's 80 milliliters because I measured it before.
It also gives me the details in raw format so you can see the image taken. The pre processed image so we squash it too and rectangle 300 by 300. And you can see the input of the tensor expects. Uh, 1,000,000 uh 80,000 bytes. So if he see the picture for three on the battery on that? And then multiply that by three, because RGB and multiply that by 4 because we use float 32 values, because this number as input. Uh, and then the the object detection is run on that. You can see the output of the opiate action. The bounding box itself is quite nice around the bottle, so the prediction of the image of the prediction of the bottle itself was probably off. The cause of the regression model.
Uh, we can also see the. Soft object detection. You can see we do. We're doing 10 uh. But detection zone it or the model is doing 10 detections on it and only the best one is drawn at the bottom. But the best one has the confidence of .94, so over 90% confidence. That is the the event natural milk bottle like a shell 2.
Uh, and then the. Picture itself is cropped to the bounding box embedded with black and users input for the regression model and regression model then gives me 4 bytes. One flattery to value as output and it does not apply it by 125. And that gives me a prediction in this time of 80. Sorry 69 but it was off so I put in 80 and then I can apply the results. Under that allows me to make another guess. So now you segmentation should see and look what the showing for that. Uh, there's still little low segmentation. Is the model we drop this week because segmentation gave gave us a lot more better results because segmentation is really dependent on the angle of the of the meal plan on the film.
If the angle of the phone is a little too shallow, then the the milk is also shown from top, so you can see right now. Uh, but sometimes the segmentation mask it draws. This is really weird and not not predictable and not usable. Uh. Trump also put in 80. And the blood results. And I'm as lost if the detection really goes wrong, then it shows you a question mark. You can still put in the value it it really was. And if you uploaded, then the phone itself will get feedback about how to hold and keep the phone.
Uh, so it wants me to hold more stable and more upright. And then the chip got better for the user. As you can see this a little better now. Yeah, yeah, that was the demo and my presentation. Are there questions let me switch back, yeah? Thank you more time. Thank you, well, let's give it up with some applause and medical. I guess it's always nice to see. Yeah, great. I really liked your demo to be honest.
Yeah, I love to see the live demo ING. Firstly, I have not talked about milk dead long in my life, so that's pretty cool. No, there are some questions in the chat, so that's cool. Firstly, we'll start with me letting me. Do you want to speak about a lot of talk about or do you want me to do it? Yeah. Uh, you know it's fine. Let me get my. You can go.
There were two questions. I had an one sort of answered itself in your demo and that was I can imagine people using this in the kitchen are white background should problem, but it it really depends on how well it is able to see the bottle I guess. Yes. And the other one is. This is an iOS app. Do you use any of Apple's machine learning stuff? Is it optimized to run on? Uh.
No, uh, no. I I didn't really talk about it, but at the moment we using tensor flow Lite. The models itself got created in tensor flow. Because a little yeah, send the most self 30 of light also turns for low light and we use the flight's library to yeah to to use them to interpret them. As such OK. Right, hopefully? my. I was wondering if it would be with all the machine learning stuff Apple puts in their hardware nowadays if it was possible to do this in real time.
Yeah. It it probably is possible, but the flight gives us more control over the training because the training is kind of hard to do on the CPU, but I heard is that the data are guys as they have. Server somewhere with GPS on it of in it and can run multiple training sessions at once using multiple GPU's. It gives us more leeway to to try and to retrain the models after they got updated with new pictures. I'm between what I know of a. Uh, the mark or Cornell? What about Apple users? Is that the the training? Of course is done in Xcode or tool by Xcode and the process of detecting something is quite of this quite.
A country with an under Jeter. And you can't really see what's happening in the in between steps of the training and what is happening with the data and and how the pictures interpreted in the in model. Child or Mel Gibson's just more leeway in more? Knowledge about how it's all done. Yeah, yeah. I'm struggling to get my dog back.
No problem my time. I see a lot more questions. We will answer them in a few minutes. Let's do the ending at just right now and then we can have a chat with more tiny all night long if you want to of course. But first, let us continue with the program, if that's OK. If you more time as well. Yeah. Nice. Alright, so today where are we right now after this great session we have a lot of knowledge and a general view of native versus hybrid and now of course the story of more time. We have a special announcement to make from the front lighting talks an it's about next media as well. Next meeting will be graph QL, cloud and serverless, so that will be really cool, so please join there if you want to present. You can also do that contacting meet up organizers and you will see some communication of us.
And contact details of course, but even cooler. The special announcements will be that will be joining up with Romania for a one time off in September about modeling AB script and frameworks, so it'll be a special collaboration to Infomania so City we will do again the same formula 3 short Lightning talks. And then we will talk about yeah script, also, framework, architecture shared. It will be really cool. So keep in lines. Follow or boasts an please look at us. The meter page of course.
Uh, and in the end that is very important. We want to do this or not a lot more often we want to gather feedback as well. What are we doing right? But are we doing wrong? Do you like it? And in general, what do you want to see? It's also very important because we're doing it for everybody to learn an F on a bit. So there will be a feedback form in the end. Uh, will be your five question. You can fill it in if you want. We are really happy with that. If you want to fill it in. Let's see any leftover questions from a time already saw some points and then I will stop the recording as well so we can slide into the great fun as well and see you at the next Lightning talks. Thank you. Oh and let's go through the questions for my time. Let's see more time. There were a lot of questions from a thief as well. What is the benefit of this technology in everyday life? Yeah. Do we honest, and that is also kind of why, but what I'm struggling with is. I really don't know if the if the benefit because the bottle I just took pictures from has marks on the site that tells you how much ML is inside.
There already exist a bottle, of course. It's a little expensive, but that can see exactly what kind of volume is inside and what I've heard is that in hospitals they even don't look at the scale, they just put it on real scale, be cause milk and water don't really differ from each other when it comes about wait 2 liters. Even in the small amount, you can mostly say that one milliliter is the same as one milligram. Uh, so I I really don't see a benefit to this, but maybe it can use in other ways, I don't know. Alright, hopefully that answers your questions here. I think if you want a product can always be used. Oh yeah, for maybe 2/2 actually stipulate this, it is proof of concept to see if they can do it and if it's possible and. Max.
Maybe they got. Yeah and I deal with it afterwards. Nice, I also have a question Mark John uh where where the picture go when it's taken wrongly or when you showed it with your hands will also go in the same as three stores that will be kind of weird I guess or. Yes. No, the the prediction week. Maybe that's something I can show.
Sure, share screen again. But wait. This one, yes. And I got Xcode open. We we upload everything to the same as the bucket. But uh, what we upload is quite extensive, so we got the prediction model and the prediction model. Can you tell you the prediction type? So if it is segmentation regression that has the raw input image in base base 64 encoded Jason JPEG, the practice volume, the users real for him. So even if the prediction fails, it still allows the prediction with the real volume and the raw.
Uh, and then the the segmentation regression will pick detection results and those results are the raw image that is processed the input tensor as in raw base 64, data encoded data arrays and array output. Shelby can check what is going wrong and. Also check that against the trained model that this on the computer, so we still apply that to the same back up, but then the result. Uh, of course, has no predicted volume, but only as a real volume.
Alright, that answers my question correctly. Cool nice to see. I think there's from Edward also another question or you dare to question and ask it by yourself, or shall I do it? No reaction within 3 seconds and I'll do it just. Fine.
Alright, uh, question from Edward, what changes would you have to do to run it on Android? I think there will be a lot of changes, but maybe I'm wrong. Yeah. The the the app itself is, so the app itself is written natively in SWIFT for iOS, using few controllers. But if we both make for Android, we still need to remake the full app, but the the idea is that and that is also a reason why we not went for iOS for Xcode. Uh, sorry for for Quorum L but for tensor flow is that the tensor flow part of that would be kind of the same. There still need to be. Yeah, there still needs to be made a prediction object, but it is possible to do it on Android. That's the real reason. So actually everything has to be rebuilt, kind of, but the models can be reused and the type of. Uh, data has to be the same for the guys to to use, but it is recreate all this way on Android at least. That's very cool.
And last question is from Sammy, not related to the presentations. But what are you both excited about that came out of the WW DC 2021? Yeah, I'm trying to think so, yeah, but I I didn't have the time to look at it yet and I quickly looked at the presentation, but maybe the coming days and I have more time to look at the results and the most. Most interesting thing, I think is is async await in the new swift? Uh update. So I'm kind of kind of excited to look at it in the coming days. It's very good to hear. An yeah, except excitement. I guess that's always good. Yeah, yeah.
Next time I think this is all the questions for you more time. Thank you everybody for the questions as well. I just sent out a feedback form if you would like to fill that in. That will be great. Takes less than two minutes and will really help us and you in the end as well so we can make it better and maybe a bit of appropriate subjects next time. We have graph QL and servers. Of course there's also question from admin admin. Any special wishes so please fill that in. We can act on that. As well. And then I'll stop the recording and I would like to thank you all for being here and also asking the great questions and hopefully see you next time for the next meet up. We will stay around and chat for a bit, and if you want to enjoy the reader, that's also fine. Thank you all. Alright.
Recording will be stopped. So it's important. Uh, she somebody else or hands up or his hands up. Yes, this is Lydia hello.
Welcome. Yeah, this is my first time to join this program with the meet up and I had no clue about what you're talking about, but it is good to join. In there I would like to know if I can get any support from your side to learn programming or website development. If I keep get this opportunity because now I become passionate about their website development and I had no clue where to start or something like that and I'm seeing videos but. I I don't think it will.
Help me to go further. So in case you have a program too. For beginners. Great question. I see admin smiling a lot. Maybe wants to answer that. Yeah, I think with that we with this program we we try to have sessions on different levels, right? So especially this was more focused, especially on on the native part. And A and a bit on the hybrid. So we had some some data on that. At times we've all also done presentations, for example on a specific framework for getting started with Angular. I remember for one evening.
Uh, or picking up any framework and just getting started with it. So we did organize those sessions as well and it kind of depends on the evening I guess. So next time, for example, we're going to do a on graph QL and there is a cloud and serverless, and that's also why I wanted to do what I was wondering about is what would people be interested in? I know quite a bit about cloud and serverless an on that topic. So there is a lot I can talk about and I can do a starter session. For example, if people are interested or there is also opportunity to dive into some specifics, it depends all on what people are interested in. Yeah, but.
Yeah. Yeah, because I'm asking this because now my pool is different. I mean what a resource, but I do different modelings and that I want to let this. I want to extract some inputs, some outputs from the model. So I want to re late with this. Doctor from the model and the using a programming language, so I'm planning to do this kind of thing maybe. That's why I'm asking. It's always good to just try something else. Learn by doing an get in touch, maybe with one of our in the team. I already saw advert putting me on the spot. The cell.
Yeah, yeah you're always welcome, yeah. So you can always just do one on. Yeah, you're always welcome exactly. Thank you so much. Yes, yeah. Nice. Good question Lydia. Or later. Thank you so much.
No, any more specific questions or. Some vibe. Well, it it. It's a shame that the my name I will be called namesake emails already left the meeting because I always like the discussions on the specific technologies where the yeah. Yeah, then they differ is not meant if, uh, discussions are. Yeah, where it's always one is good and the other one is bad and there's no nothing in between. Uh. Always so ping you made it about the thing. Yeah. The one thing I'm always wondering about if you're doing native development then. Yeah.
How are you doing server side stuff? Or do you even do server side stuff? Well, for the for this app I only had a nasty bucket, but at the moment I'm also creating app for myself at home to keep track of my book collection. Actually manga collection and the app itself. I create in SWIFT for iOS and my Mac book 'cause it just looks nice but the server side I do a job script now. Here's an host and they will be less. Hello uh mixed mixed set of languages and make set of techniques. For me at least.
I know it was like five years ago when I just joined so City I was in a meeting with a lot of harsh text from from the Java Community and I said like in five years time, Jaffas replaced by JavaScript and they are still reminding me of that. No. Today, so I I actually learned not to be too much. Not to worry too much about one technique, only their websites and pros and cons about every year. Different kind of technique. So that's also what I liked about presentation of a meal. It was not just one. Speaking. One message about hybrid versus native and one is good and one is bad, but it's always you have to pros and cons.
Yes. Yeah, once you have a strong opinion, you'll probably get paint on it as well. Five years later. And now they are recording session, so it's even more difficult. Even be 'cause I I I myself have a strong opinion on on mobile audit on native fishes.
Non native, but I see that in practice. It really doesn't make a difference in the starter app I made. For starter is summer in. And that's because it's cheaper to just develop for those two and then just want a cheap happen so I can. I can recommend them to do native, but of course they won't go for it, because why should I? They just want an app for sure. Well, enjoy this as iOS for the most cheaper plastic and get. And then I do also believe it's not only about price, sometimes it's also about UI consistency. For example, I will also work for grabber bank and they said, like yeah, you can talk about, we want really to have native look and feel, but they have their own brands styles and they want to have the same consistent UI in both their website, their Android app in our iOS app and if that's your preferred option then web is easier to get a consistent consistent UI. So I don't think it was. Yeah. Yeah.
Mainly due to cost that they are more into the whole hybrids. Yeah, you could also argue they are doing a bit of the hybrid native sweet spot because some parts of the applications are actually completely fully native. They don't use Cordova, they're just making a native app and parts of their yeah, web parts of the applications are just a webview only, so it's it's a bit in between. It's a hybrid hybrid native app. Yeah, I think that more and more to be honest. I still think that that at some point in the near future we're going to get rid of those screens and all the hand interfaces. Then we're going to go with the digital assistance and AR anyway, so then we won't have any use these type of. Wait, I don't say that to heart. How many years do you think 'cause?
I'd say five years I think. I'm saying that we should do. You should Start learning developing agents in in JavaScript because at some point all the front end developers will be put full time to building agent because we're going to need a lot of them. Don't say that. I could be could be could be. Put together I'm also laid off the conversational. So. It's so easy, so if we all all moved to conversational interfaces, I I, that's cool to me. Yeah, just be prepared.
I already got a back up plan. Their immune, nice guys. I'm out of work, are in it with it. Yeah, yeah. Yeah. It's best to be prepared. There there has been another mouth added to be to be fed so.
Yeah, that's true. True. So best be prepared. Exactly. Like make a more expensive once they go studying and stuff like that. I I heard so now it's so cheap.
Diapers are expensive. So. I bought a lot of them, so I have. I think for a few months too much, so if anyone needs a number 2 diaper collection, then. I have a lot. Your garage has been.
You have to park around the block because your garage is filled with diapers. Yeah, yeah. now beyond the Bureau Office Office. That's why this officer small now. Yeah. Yeah, I gotta go great evening. Alright, see you bye thanks for joining. I, I figure going to Greece or not. Uh-huh
yeah pretty soon here. Good evening. Hey guys. Alright. So more people with strong opinions. Good or bad? I actually saw some people with the the that we don't usually see at the at the normal front end meeting with, which is quite nice. So effort back again. Yeah.
Yeah. I think, uh, I don't know if what I'm seeing did. Did he join the other sessions as well? I don't actually remember. I think I haven't seen him for awhile. No. Not sure, not sure. I also see a new colleague. See me.
Join today. Time and. Ah. Yeah, that's gotta be me. Only one of them. Nice to meet you guys.
One of the new guys welcome. Thank you. I see your face still in the session as well. I think he's from the AI part, right? I'm not. Oh, we got more info coming. I yeah, I used to be mobile so I I really enjoy the same discussion being held even 12 years after we started it so. Cool so James. And I'm really happy that you did not recall BlackBerry next to the Windows Mobile plug.
Thanks. BlackBerry. Yes. But I know I'm I'm still wondering, like, uh, where are you on this? You are a mobile developer, right? Yes. And and timer. Or is he is frozen now? I'm frozen.
So I'm I'm a mobile developer, but it's in the last two years. I didn't do anything mobile an even now my time is mostly. Yeah. Yes. Used for uh, creating angular apps for another company and only a small part of it is used for creating a mobile for mobile survey. Sorry my connection got interrupted. So.
We saw that you were frozen and again Oh no. I don't know. You were gone. Ask you to do do do you coming out? And again. Sorry, go on again. No, I was actually wondering like of our new colleagues like your mobile developer, right? Are you on the native completely focus native? Are you on both sides? Which side are you on? I'm both I'm on both sides. I like them all equally, but if I have to choose.
At gunpoint, then it will be iOS. And then native were not native. For sure. Native, of course I did. Later, yeah. Diversional It's for me as well. I would also only choose iOS at gunpoint. I. So yeah.
Yeah, but it's you have to consider who's actually putting the gun towards you so. I think the. Probably iOS as well. Yeah. If it is, if it is Android, Android guy in the points are gonna be done of course. Rather not. I'm very smart. And what's your what's your preference? What is your preference? What that from you like the most? For me.
Yeah. I have to be careful what I say because yeah what I already told like I've to like all the techniques now for me personally as a developer. I really like the combination of capacitor with a front end framework. I mostly work with Angular so that's for me. The issue's choice is not my preference that I say I only do angler. I only do or not, but it happens that the clients that I work for using Angular Cordova kind of things and I think a pastor is to. Following up from Cordova, it's it's newer, it's cooler. It's better so. Yeah.
Feel me tips I go. I think the first meet up of this year when we were talking about JavaScript and frameworks are told about the app I created for iOS and Android and web about encryption app showed that that's my hobby projects. That's where I spent some time. If I have some spare time. But yeah, I like to. I like to hybrid native sweet spot kind of thing like you are building partly nativ
2021-06-20