A.I. Assisted 3D Asset Production At Scale I Franz Tschimben I FLOW

A.I. Assisted 3D Asset Production At Scale  I Franz Tschimben I FLOW

Show Video

[Music] Great super excited to be here today New York flow I love it I can just say it I'm From Italy so it's always great to come here to to New York AI assisted 3D asset production scale so early morning New York lots of words right but My goal for today is really to let you know what is possible right now how companies Brands retailer studios are using Technologies to create 3D assets at scale how they are then being used to create content I think the panel discussion was just the best introduction ever to what I'm gonna talk about how to do this at scale of course the challenges that we're still facing in doing so and how they can be overcome with artificial intelligence especially when we think about post-production as well and let's just dive right In And before getting started you know this is kind of what the speakers do kind of involve the audience so I want you as the experts you lead studios you know photography you know e-commerce you surely are able to tell in the next three pictures if this is a real photo or not so if you think this is a real photo raise your hands instinctively okay I see some hands go up you rock the real experts classical 2D photo yes or no so if you think the number one is actually a real photo taken in the studio raise your hand if you think number two is it then you keep your hands down okay great and then last but not least also video creation right is this actually a real video yes or no those who think it's a real video raise your hand it's tough to say I see some hands kind of going up or not I think you get where I'm going at all of this content that I just showed you is the derivation is basically a product of the 3D asset 3D asset you have it right here so I just took a screen shot or screen video capture of a 3D asset and I want to show you two properties that are key for what I'm explaining right so you have the shoe here it has to be reliable meaning you can put it in any kind of context put the lights virtually like in a studio that's the first aspect the second aspect you look at the geometry so this is basically the same shoe it's not a different color it just lets us look at the geometry from a technological level it needs to be super accurate right it needs to show every little stitch every little thing those are the two crucial aspects about 3D assets in my conversation going forward for e-commerce applications we have five applications you have the 3D asset by itself you look at websites or mobile applications like I don't know Adidas or you have Zara right they have already 3D content out there you can play around with it engage with it zoom in and so on just to get the product better the second application is you derive the 2D from the 3D in an automated scalable way so you have the classic 2D photo you can derive a contextual 2D photo from it right so you put it kind of any kind of environment that you want in this case a race track and it looks photorealistic you can derive the video from it we just saw it and I think Juliana mentioned it as well a great way to engage the consumers well from a customer experience perspective is to then also have the virtual try on right so this is just a capture from a phone applications now you might ask why am I seeing sneakers why are these sneakers mostly from Adidas they allowed me to share some of the content here today we're working with them for a long time already I think research innovator taking this on right away and actually they're losing 3D assets to really produce all types of content for their websites and everything that you see here has been automated also from a workflow perspective now nice right why should we care 3D has been around for 10 years it was hyped it was not hyped this will use a lot for gaming I read one of the question cut models right so the designers you draw your customers or in your company they create these assets right and they already exist like why not use them for instance so why should we care think over the last one and a half years and this is not about AI right now it's just we have been capable of integrating this right I show here a few examples of technology companies of Brands retailers and so on so this is already happening right now you can you know leverage 3D make it happen the organization it's not rocket science you can just do it there's also providers out there you just need an internal Tech Team who loves to do this who sees value in this you need top management buying you can make it happen right and on the other hand and I think this is crucial then when we talk about also artificial intelligence how our products economically viable all of a sudden and how you can integrate this in your workflow is the processing power has gone up right so I can train models so I can create content also in terms of 3D which is basically a distillation of two images you can do this fast and rather cheaply now the device performance has gone up and I don't need to tell you that you know we heard about data sets imagenet is one of them but the amount of data is available and also the open source tools that computer vision machine learning engineers can leverage to create products they're just out there and free to use basically so everybody can do it now what does this lead to right so we've seen companies doing this in the past as well Farfetch is actually I think a great example they have been integrating these heads on for hero products right so just a few now you can scale this right now to all of your products right away footwear is one example right but you can do toys tools anything you want basically with varying levels of difficulty the second aspect is as I mentioned before it has become economically available the tools are out there to be used in this case you see basically a 3D scanner you put a product inside hit the button the 3D asset comes out of it that's one way to do this and I think this is important from a customer perspective and they of course are able to differentiate between a really really good model and a really bad model and in the past we historically had bad models so why even put it out there if the customer doesn't want this now you cannot tell the difference anymore I did the initial exercise a bit just to you know of course make you raise the hand and engage you but also it shows that it's not easy to tell the difference anymore right and let's say I convinced you now that 3D assets yes they have a reason to exist you could potentially think about them or maybe you already do and you did some tests the question is how do I get them right so as I said before you have existing 3D Assets in your design organization these are the product designers for you know big corporations especially in footworks either in somewhere in Europe Italy or Germany or you have them in Portland they designed this and then they get produced somewhere in the world right but they exist now on the other hand you can also resort to something that's called 3D scanning and you create the 3D asset from an existing real product right they have advantages and disadvantages that will not go into detail about them and you know you can maybe look at them on the video recording after the FLOW event but basically and this was the question before and that's why I liked it the cat model basically the model was created by Human by a designer by 3D artist to then actually produce an item or the shoe in this case they missed some properties especially they are not reliable and sometimes it takes a lot of a lot of manual work to actually perfect that model in a way that the customer doesn't see the difference now that is not scalable by definition you can do this for a few here products but that's it so I would say the way to go right now maybe this changes in the future also with AI 3D scanning is the way to go now how can you scan right so you have your mobile phone there's 3D scanning apps you can maybe show you later how this is done you have a handheld device of course it's not the same size just for illustrative purposes I put it in this way you go around the object with the handheld scanner it produces the 3D model or maybe you want to really scale the production you have specialized equipment and they have varying degrees of let's say speed where you can get the models also of course cost right it varies from one minute per scan on the phone to maybe 10/15 minutes per scan on a dedicated equipment one thousand dollars per machine to several hundred thousand dollars so you can do you know many things but there is a way to get started maybe just to illustrate how such a scanning process works I have asked Mark he's a colleague of mine to explain this and he actually sent me a short video to kind of show this off so let's hear it from Mark not just because he's super cute but also because he's got the right dimensions and model of detail of what we're trying to replicate so here's how it works there's a step on the phone and that's about it all right the little detail on this is impressive you can kind of see all these fine elements of it like each individual strand of its hair and you can even see through these semi-transparent bow tie that Amazon it's pretty awesome yeah it's right right it's pretty awesome thank you Mark it was late last night he just sent me the video and said hey use it um Aleksandra thanks for integrating it so Mark talks about 3D scanning mainly meaning you know it's happening it was at the connect 2022 event of last year and his scan basically get teddy bear with the phone now there's a specific reason he did that and now I want to get a bit more practical especially you who love the details and you know lots about post-production and so on as well so we started teddy bear it's notoriously very difficult with all the hairs and fur it's a nightmare for 3D scanning basically and I would argue that the teddy bear that we saw that in 3D was created by hand and not surely just by walking around the item and also the transparency of the bow ties not obviously it's very very difficult to do although they have of course great engineers but to kind of distill this and come to a conclusion mobile phones and handheld devices great way to get started right especially organization I love also with some familiar faces in here that use these kind of easy to access low-cost ways of creating assets maybe putting them out on the website already do some a b testing and figure out if this could be something for you and it helps to kind of create a small team of experts within the company if we want to scale this I would argue specialized equipment is the way to go now let's look at the workflow right it's a bit small to read but sort of guide you through to it the 3D scanning process looks very similar to like a typical 2D e-commerce process you have the styling of the object the preparation in terms of shoes it means you fill it you know with paper you clean it a bit you make it look nice you do the scanning or you do in the same way the 2D photo so you have the scanning happens it is human operator involved doesn't need to be necessarily super highly skilled but it happens um then the 3D asset generation right is usually here happens on workstations with a bunch of Nvidia gpus automatically in the case of a dedicated system if the technology like let's say a phone app is not that advanced you might need a human 3D artist kind of put the model together and make it look great then you have the quality check this is super important right done of course by an expert especially to the experts who know lighting who know photography and so on this is a great way to also get them involved in the 3D scanning process then you have the post-production especially on very let's say reflective items or some items that pose also challenges from just the way they are and they have many holes and so on this is difficult to do and then you create the content from it so the 2D photo the 3D virtual try on the 2D contextual photo video that is all automated right so you don't need to do this by hand you have a script you integrate it and it's done it can go live right and where does AI come in right so this is now the introduction to the second part of my presentation AI really helps also on the post-production side of things typically this is work that is being outsourced right now so I will show you how AI is helping especially making this economically more viable and those are the results more scalable and ready to use in a quicker way so 3D quality as we said is key right so you have another example here of an Adidas shoe reliability in the same way as we saw before on the geometry side I will now go into detail here anymore and AI has basically in two ways so you have the repetitive post production tasks and I think you know this already from the 2D world or the video world and so on this is this is nothing new under the sun and then you have transparency and mirror-like reflections now we are a computer vision machine learning company and transparency is more like reflections is a horror to start with and that is where typically they say disruptive machine learning comes in right so this is where we talk about the nerves neural radiance fields maybe a key term that some of you know it's the latest and greatest everybody talks about this in the research Community yes generative AI is cool but really nerves is the is also not a key term that you should pay attention to and we can maybe discuss this a bit later and then you have classical musician machine learning maybe good old-fashioned machine learning as it was mentioned before you tell the algorithm to learn from a database and say repeat this what you know has been done before the key here is you have to have of course a good data set now let's look at the repetitive post-processing path so the paper stuff in coverage again this is a 2d photo from the 3D right you look inside you see paper in there it's not nice right I mean you could put it online but you don't want to put it online so what typically happens by hand you mask this out you just make it a bit more black the key here is you can still see in the case of shoes the sock liner because that's usually where the logo is and you know to make it appeal nice but that is happening by hand right this can be potentially automated with artificial intelligence you tell the algorithm basically learn from what has been done by the human and just repeat repeat it a b testing is crucial here then you have and some of you might notice when especially you get the samples from the factory straight from production you have sample printed inside and this is something that needs to be masked out as well this cannot go online and certainly not into any kind of imagery or applications that you create so again this needs to be masked out this is sort of the typical stuff that needs to be done transparency is more like reflection this is more disruptive you see an example here of a football shoe again 2D from a 3D this has not been shot in nutritional photo studio and you know in first glance okay looks nice I see all the details if you look at the stats they might be they might seem a bit off right in the next image you see how they should be like and this is lots of post-production work that needs to be done by experts and the same way of course if you look a bit more detailed way on the sole this is the before and the after right so this needs to be done and this to where in the future AI could come in as well mirror-like reflections is another big topic mirror like means you can almost see yourself in the object and again here you see the before and the after and this is lots of work by 3D artists the great thing is you do it once you have to 3D asset and you create any type of content you want from it as you can see here now to sum it up how does AI help of course automation force production is one and great to hear also from Pixelz that they are way ahead of their game as well in that part here quality of the scans goes up as we integrate more and more artificial intelligence into our pipeline as well time to market is crucial especially when we talk about 3D assets so in the photo side it's immediately available when you take per item 15 000 photos in about 10 minutes that that get compressed on Nvidia GPUs into 3D asset you might need to wait for a few hours so this is kind of what you need to deal with and in the long run of course costs go down talking about scale right so the title was also about 3D asset production at scale what can I do of course there's a learning curve and this is actual data that we took from you know some of our customers and um you can see that increase especially as your team starts using scanners for instance what can be achieved 800 to 1 250 per month per machine if you operate a machine about two shifts per day of eight hours and five days a week but there's more details to it of course you need to upskill your organization speaking about upskilling everybody says AI will automate the jobs away it's not quite the case I think teamwork is essential those competencies that exist in the two world in the video world in the studio they're absolutely needed they're crucial to make this a success and all those stakeholders that you see here need to come together in order to make this work at scale the results look like this right so desktop but also on mobile you can see the imager here that has already created from 3D on the left side this is I think from the World Cup in track and field you can just select the item look at it in 3D open it up in augmented reality and then try it on and feel it for yourself and this is basically been done for all the content right now the results in terms of numbers I think everybody's always looking at that so conversion rates are crucial up to 2x is what can be achieved of course depends on how you integrate it on the other hand add to basket rates and product interactions they go up and all leads to those higher conversion rates now looking in the future right a bit of an outlook the spatial Computing devices will come up the native content there is all completed 2D it will be 3D so nobody knows if this really takes off we will see if Apple has launched great products in the past will this also be a great product we don't know how the market accepts it but one thing is clear it's 3D it's not 2D if you want to have a really great experience there right and looking at the 3D scanning process and this is I think the natural revolution also of the talk that was here before the scanning aspect as well as then the model creation here AI will play a crucial role as well it is essential on how you create the data set so let's say you are convinced by what I tell you and you say okay from now on tomorrow we start 3D scanning thousands of objects that will also help you then in the future to leverage these assets in generative AI applications to design maybe new shoes to design new applications and so on in the future just like ChatGPT is done with open AI or let's say open AI has done with ChatGPT they train their algorithm some text mid-journey training their algorithms on images that were available in the net but high quality 3D data does not exist right I think Google put out a paper a year ago they scanned about 7 000 items in five six years right so that data set just does not exist and B2B you need specific applications for your company you can't just say oh let me take all the shoes in the world and create the next Samba sneaker it doesn't exist right so you need your own data set and 3D scanning helps you to create that in the highest quality possible which makes it usable in the future and yeah this is it I hope it was insightful and interesting we're working on this and if you're interested in knowing a bit more happy to discuss it later thank you let's get to the questions okay Gabrielle "Is there a way to create a good 3D scan using multiple images from a regular DSLR camera and dedicated software?" So it's possible certainly I mean our first prototype that actually we kissed it goodbye last week has been outdated we started with DSLR cameras and built the software from it so it's absolutely possible to use that and just a few images to then create a high quality model is impossible I would say this is where AI could come in the future to maybe automate this in a way that is you know the same quality that you saw here today otherwise I think really you need to resort to specialized equipment a few 10 15 20 images are not enough to do tens of thousands and we actually right now use machine vision cameras that just are used typically in production lines and that's what we use to to create the core data to then generate the 3D model so the answer is yes but you know it's possible yeah all right Anonymous "What is an average size of a final file how will websites need to change to support these potentially large files and load times?" yeah I think that's actually a great question so you have 250 megabytes of the let's say the core file right of course they cannot go in any kind of website but that is the master file right which we distilled already from gigabytes of data that we automatically captured and generated into this 3D file four websites I think the 3D model is about five megabyte but that can be automatically generated too of course with a trade-off between quality and so on but it's absolutely possible yeah so kind of the same balancing act with everything images video that's just it's absolutely different type of asset okay Anonymous "How does the cost of AI 3D create creation compare to traditional photography and post-production?" yeah I could tell you I mean the models that you saw here today sell for 85 dollars per model for the master file then of course the declination into several applications cost and depends what really you want to do so there's an additional cost for that comparing on post-production I would say it's it's similar actually we work with there's not many 3D post-production studios out there that have been doing this for 10 years on high quality models that just don't exist so they switched as well their organization prices being similar of course but right now we're still in a face where this is being adopted so prices naturally will be higher compared to traditional photography and also in lots of companies and organizations we talked to who look at 3D as kind of a solution to go forward to kind of engage the customer in a better way to create various types of contents almost automatically they usually have a holistic view on 3D assets and that's provided by the top management right they say okay this looks good we can use this let's get started on it if you do a comparison one-to-one photography versus 3D there's no chance 3D will ever make it to the website because it's just not economically viable we'll see okay Ali McLeod "Do you anticipate brands will move or will provide 3D assets to marketers or multi-vendor retailers since scalability is still challenging from a PDP perspective?" yeah so I think great question the answer is absolutely yes I mean the same way relationships have been handled in 2D and perhaps video as well where the brands just say okay the open minded brand say okay here you can have all the content if you're an important partner to us put this on I don't know Zalando or any kind of other platforms out there I'm still staying within let's say footwear fashion right now so I think this is possible the question here is about the quality right you as a retailer you need to ensure that you have comparable quality if one company does this with phone scans and the other company has a specialized equipment doing this at scale there's a quality difference and I guess a retailer should have in mind what kind of quality he wants to have out there and I think there's still major differences whereas photo except for maybe how you style it and how you place it and so on it quality there's a certain quality standard that is widely accepted by both sides right awesome awesome all right Edward Melvin "Can models interact with the 3D models to imitate traditional photo shoots?" I like the question can models interact with the 3D models and so can basically yeah interesting there's some video out there I think it can be done but it's a lot of it is done manually basically in let's call it post-production when you create one type of application interacting with actual models it's still not automated you can do it can you scan I mean and that's a human yeah to make this really look photorealistic still long I would say quite some way to go is the typical analogy also with AI it's really great to go about 80% of the way quick demo it's like wow how is this even possible then you look at the application at scale where really the last 20% matters and then it's a question of how costly is that last 20% so I think there's still some way to go so there's a reason also why you know we have been focusing on scanning assets of product versus scanning humans just because it's so difficult and billions and billions of dollars of research actually go in to this if you look at say if you participate in a major computer vision machine learning conference the Googles the Facebooks of the world they do major research with big big teams on this and there's still no product right so is a bit of a way to go but I mean Mark's putting Adidas on the teddy bear right absolutely yeah yeah I just want to be clear about it yeah all right Scott "What is the biggest current bottleneck in the 3D workflow and how has AI solved some of the previous bottlenecks?" yeah so I wouldn't even call it the bottleneck it exists right now this can be done at scale right now and the bottlenecks the majors major bottlenecks have been resolved I think the major bottleneck was in the first place how do you get in an automated way for all your products to actually affiliate assets that can be used and it's again economically viable so when it's not crazy thousands of dollars of cost per one single item now looking into the future certainly some aspects of post-production especially transparencies more like reflections depending on the range of products that you have that you want to scan if you wanna you know if you're a glass producer or I don't know you are Moet and you want to scan many many bottles of champagne maybe it's big big effort on the post-production side and not that feasible right now depends on the range of products that you want to digitize and lighting those bottles would be fun yeah Anonymous "How does the cost of AI 3D correct creation compared to traditional" that's already been answered Stacy Tyrell oh it's moving around yeah I think there was a quite maybe I take one right so how you can you handle metallic objects generally etc nerves the answer so maybe so you can handle metallic objects but it needs in post-production it's just a fact but again there you can get pretty long way with the right equipment our nerves neuro radiance fields the answer so right now nerves what they do basically just to explain it for everybody so you have a bunch of images maybe your traditional 10 images a neuro radiance field basically creates in still in a machine learning way the 3D model basically but you cannot write this plate so in order to display it you fake a collection of 2D images which gives the illusion of a 360 around from a set of images neural radiance fields many images are created but it's not the 3D asset yet so I think right now not in the future this is I would say it's possible yeah let me ask a question about the lighting because I love lighting right what's that lighting lighting yeah "So when you create that lighting and you said it's reliable and as it zoomed around I mean are you sticking to a specific light like you know I'm going to do a soft box lighter I'm going to do a nice punchy light or is the client able to just change the lighting themselves? it's like all you can do whatever you want with it you can put make some videos you can put the sand there or you can put soft light a hard light really just you can the the comparison I showed with the two leather shoes in the beginning was basically we took the virtual lighting that was one to one the actual lighting in the studio was a studio photo the other one was a photo basically taken from 3D asset yeah you can't tell the difference I mean the consumer can tell the difference then if I look at it really into detail we zoom in like crazy then there is still a difference there but there was a huge expectations have shifted right and 2D world it was like really really accurate now they say yeah maybe the 95% that was mentioned before it's good enough the end final customer will never know Franz thank you so much everyone give Franz a round of applause here [Applause] [Music]

2023-09-25 21:03

Show Video

Other news