NETINT Technologies about Understanding the Benefits and Limitations of FFmpeg Gstreamer and GPAC

Show video

Welcome to NETINT's Voices in video, where we  explore critical streaming-related topics with   the experts who create and implement  new technology-related solutions.   If you're watching and have questions, please  post them as a comment on whichever platform   you're watching on. We'll answer live  if time permits. Today's episode is all   about encoding and packaging, and we'll be  discussing the GPAC open-source technology   framework recently licensed by  Netflix for their live streams.   Specifically, I'll speak with Romain Boouqueau  on his birthday, and he's a main contributor   to GPAC and the founder and CEO of Motion Spell,  the licensing arm of GPAC. And the reason we're   talking with Romain today relates back to an  experience we had with FFmpeg here at NETINT  We started doing some testing with some of our  cards, and the cards performed well up to the   point when the FFmpeg operations became more  and more complex. We saw that the throughput   dropped significantly, even though the  capacity in the cards and capacity in   the CPU was fine. So we started researching  alternative solutions. We found GStreamer,  

and we also found GPAC. And Romain was kind  enough to join us today and talk about these   in many other applications of both programs and  just describing the technologies where they fit,   what they perform, and how to consider which ones  to use for your application. So let's start there.   Romain, thanks for joining us. Thank you, Jan.  So why don't we just jump into the FFmpeg  issue. What's going on with multi-threaded   performance with FFmpeg. What are  the symptoms, what are the causes,  

and how do we work around it? And take it a bite  at a time. Don't try and jump into all that.  So we're going to talk a lot about multi-threads,  multi-threading. So first, what is a thread? The   thread is a sequence of instructions. Things that  you want to do sequentially. You cannot do them  

in parallel. If you want to do things in parallel  because you have many cores on your computers, for   example, so your computer is able to compute in  parallel, then you need to have several threads.   And these threads most of the time are, not all  the time, but most of the time, are said a kernel   thread. It's an object that is understood by the  kernel of your operating system that's handled   by a component that is called the scheduler. So  the scheduler analyzes which threats are running,   which threads are eligible for computing, and it  handles all of that. More specifically on FFmpeg  

for your question. FFmpeg is a multi-threaded  application. And so there are several levels   where it's multi-threaded. There can be a cardiac  that can be multi-threaded when you execute. For   example x264 or x265 as a library inside FFmpeg,  they create their own thread, but also FFmpeg   creates its own thread. The application FFmpeg. So what was I experiencing when we saw a drop   in throughput with… Typically the operation was  large file in say 4K, and then a pretty complex   encoding ladder out, say five or six rungs. And  testing HEVC because it's 4K output and, you know,  

H.264 doesn't make sense. It's pretty simple  command line. We do it all the time. But what's   happening that causes throughput to draw? It's not an easy question.   So first, I think there there's been an effort  in FFmpeg to try to bring more threads at the   application level. So that's, what I was saying,  there were not so many threads. I think in FFmpeg   five, one of the big changes is that now when you  have multiple outputs, each of these outputs run   in a separate thread. Which means that, for  example, if you have two outputs connected to   two different servers, if there is a problem in  one of the outputs. The second one is not hung,  

so they can run in parallel. When it comes to  4K, it's a really huge question. I think there   are many factors that affect performances  when you're trying to reach the limits   of computers. There are threadings of  course, if you're not threaded enough,   or if you have too many threads, or so maybe  that's something that we may want to talk about,   then you are not going to  run at your maximum speed.  But there is also iOS, right? If you don't deal  with iOS correctly, then you can be limited by   your iOS. So the input output, and there is also  some memory issues for example. So we're going to   compare maybe with the other frameworks, but in  FFmpeg, the queues are really large. If you queue   a high number of decoded frames that are pretty  large by themselves, then it can take several   gigabytes of memory. So RAM is cheap, but if you  don't have enough RAM, then it's going to write to  

the disk and then it's going to be super slow. Interesting. How do you mitigate that? And if   you're working with a command string as  opposed to I guess more programmatically,   is there any way you can mitigate against that? So FFmpeg is two parts. I think for most of the   people. There are some libraries like  libevcodec format for the libevcodec   network protocols and a bunch of others.  And then there is the FFmpeg application.   There are other applications in FFmpeg. There is  FFplay, which is a simple player. There is FFprobe   to inspector content. And the efforts that was  talking about are on the Ffmpeg part, which means  

that other people were able to leverage some of  the benefits that the FFmpeg developers are trying   to implement, because they directly integrated  the FFmpeg libraries. So when you consider big   companies like Vimeo, YouTube, or other projects  like GPAC, we integrate the libraries and we   could investigate with some improvements on the  performances. FFm impact is really great for this,   because you can plug at many points for the iOS,  or for the memory allocators or for the threading,   and you can do your own stuff. Not everything  is solved on the FFm impact site, but as I said,  

they've been working on it. So exciting. Version six came out right in the middle of   these problems, and I basically just used the  same command lines, and tried on version six,   and I found no difference. So I had thought that  one of the goals with version six was better   multi-threaded performance. What happened? What  changed with between five and six, and why didn't   I see any performance difference between those  two versions with these type of encoding projects?  Okay. Yeah. That's a bit technical. In version  five, as I said, they have separated the different   outputs, so now you can run several outputs in  parallel, and if there was a problem on one,   then it's not going to create problems on the  other output. In version six, it's a continuation  

of this and they've been working on this for at  least a year. I could find comments related to   better threading in FFmpeg. And at the beginning  of the year, the FFmpeg developers at the first   demo open those conference said that the effort  would last for the whole year. So probably you   will need to test with FFMX seven next year. You talked a moment ago about too many cores.   Go into that and then maybe describe how you  should configure an encoding machine better   for 4K as compared to say 1080p type projects. May be many factors on this. If you put too many  

threads, the problem is that in your operating  system, in the scheduler that we talked about,   then the scheduler says it's called preemptive. It  allocates you with slices of time, typically tens   of milliseconds. And each tens of milliseconds is  going to look if there are some new eligible tasks   so that everybody can have some time to execute.  If you put too many threads, there are two kind   of problems. The first one is that you switch  all the time. It's called a context switch.   The price is pretty low, but each time you have  to save your registers, you have to save a bunch   of stuff and switch to the other threads. So  instead of spending time doing some computation,   you're spending time switching tasks.  And the second problem is that we're,  

most of the time, we're CPU bound, and so we rely  heavily on cashes. And when you change course,   sometimes you break the cash. And so again, you  have a performance penalty. But this is something   that is really, really difficult to measure. Can you give any advice at all? Let's stick with   software because that's the most generic case, but  if I'm encoding 4K videos, is there a benefit to   going to 128K or 128 cores? Or should I stick with  32?or does it matter? Or more jobs, less jobs?   What's your sense of the best machine for  high volume 4K transcoding? Again, VOD is … Yeah. I think the number of threads on your  computer should be a couple of times the number   of physical cores that are available on your CPU.  And if you make some graphics related to this,  

you're going to see that it scales almost nearly  up to a certain point. That being said, it also   depends on the application. As I mentioned, codex  can create in FFmpeg. FFmpeg, the application,   creates its own codex, but then the application  may create their own codex. You have codex   specific options, for example, for H.264, H.265,  or AVC, HEVC encoding, or any code where you can   try to tune. So most of the time the encode  themselves provide an automatic parameter so   that it tries to scale by itself. Because only the  application knows if it scales better with twice  

the number, twice the number of cores, for  the number of threads or just the number of   cores plus a certain number because they use  this for scheduling or whatever. And again,   as I said, it's not only CPU cores. If you saturate your bandwidth for the memory,   if you saturate your iOS for example, because  you're reading from the network, or you're reading   from a disc that is not fast enough because you  launched several tasks at the same time, then it   can be a problem. That's really, really, measuring  performances is something that is really not easy.   And sometimes with small changes like  going from FFmpeg the application,   to another framework that uses FFmpeg as  libraries, you can see big differences. Just kind of an aside, I recently did some  work analyzing the CPUs that were available   on AWS. So I looked at Intel versus Graviton  versus AMD. What's your sense? What I found   was that Graviton was best for H.264 and AMD was  best for H.265. Are those general conclusions you  

can reach, or do you have different thoughts as  to which CPUs do the best job with those codecs?  I can say that we use a lot and advocate for  Gravitons and equivalents, because there are   more power efficient. And also that's because the  FFmpeg impact developer is developing the cards   they've made. And they've made for years  a tremendous job in implementing all the   assembly acceleration that were needed. I mean I found that true with H.264,  

but H.265 was very bad with Graviton. And that  was including a very new version from multi-core   wear that actually improved performance  by 300% or 400%. So any comment there?  Very different projects, right? H.264 is really  community-based passionate people and H.265 is   maintained by aware. And even though there is  passion in companies, especially on video cards,  

people you know, can have really interesting  discussions with them. It's not the same,   right? H.264 is made by people that were  genius, I would say even crazy at some   point. So for me, totally different perspective. What about compiling different versions of FFmpeg   for the different CPUs? Do you need to take  it to that level, or is that not necessary?  I don't think that's necessary, but yeah, you  could do this. You know, always have problems   with operations, like operational problems. So you  can come into a situation where you need to have   several builds or using several versions even  of a specific tool. That's a possibility. Yes. 

So let's transition over to GPAC, and let  me kind of set it up. I mean, I know FFmpeg,   it's really good on the encoding side, a  little bit less good on the packaging side.   We used GStreamer as kind of a workaround for some  of the bottlenecks that I described a few minutes   ago that worked pretty well. And then I've known  you for years and known about GPAC for years, but   it really came to prominence when Netflix licensed  the technology for their live streams. Give me an   overview of just, really high level, about where  you see those three projects, including FFmpeg,   GStreamer and GPAC. What they  do, how they fit in together,  

what they're good at, what they're not so good at. FFmpeg. As I said, they are the libraries and   these people are implementing codecs protocols.  It's a really nice toolbox that is really,   really complete. There has been a huge effort  on it. Really successful project, I would   say critical infrastructure for the industry  today. And then there is the FFmpeg application   that people use. People like it because the  interface is quite stable. You can reuse a lot   of your common lines from years ago and they still  work, which is not the case at the library level   where you have to implement the API, which is the  interface at the coding level any changes from   time to time. So that's more work to go on the  libraries. You can get some of the benefits. For  

most people you can use the FFmpeg, common line  libraries. The GPAC, so we are also implementing   some standards and some streaming protocols. So we're really experts on this. Maxing, maxing   packaging, encrypting streaming, we are active at  standardization, so we have a lot of R&D. That's   a difference. For example, with FFmpeg and inside  GPAC we have our own codes and we integrated with  

other libraries and FFmpeg is a big one for us.  It's a big dependency, and we leverage a lot of   FFmpeg for our customers, our users in general.  GStreamer is more like an aggregator. There is no   real code being implemented in FFmpeg even though  it has changed over time. But its ambition is to   aggregate everything into this kind of what they  call elements, which are kind of modules that you   connect together in a big graph. So GStreamer is  really good at this. You mentioned that GStreamer  

could handle some of the bottlenecks  that you had with FM peg.  The framework is really more complex and for  example, we were talking about saturating the   memory. GStreamer has nice mechanisms for it,  pre allocates the buffers, it never allocates   on the fly, which is also something that  Ffmpeg or GPAC would do. But with GStreamer,   when you have a queue, a buffer, when you are high  in the buffer or low in the buffer, it's going   to emit a message. And so I was talking about the  scheduler. This is something that is important at   the GStreamer level, they can decide to execute  or to not execute some of their elements, just   because the buffer level is high, or not high, and  so you avoid such a rating for example your memory   usage or whatever. Because when a buffer is full  there is nobody that's able to push data to you  

for example. So that's a really a precise example  of where GStreamer may have better performances.  GPAC is also quite flexible, and we have all  kind of, I think, intelligent mechanisms to   try to handle performances in the best  way. But again, it's magic by default,   there is a lot of automation, and if you don't  like it you can disable it. And there is a lot   of flags to try to control your performances,  depending on the whole environment GPAC may   not be aware of. It's not like when you run these  tools, there is no something that is benchmarking   your system and then says, "Okay. The best flags  for you are this or this." It's just saying,   "How many CPU cores do I have? Or how much memory  do I have?" And it doesn't consider the fact that   you're running a browser at the same time, using  already a lot of memory or other considerations. 

Let's look at the individual, I guess components  of what I want from an encoder packager. I mean   it feels like, again, FFmpeg is very strong  on the encode side and it's very usable,   very accessible. Pretty weak on the packaging  side. And I guess FFmpeg is over here. I don't   know where GStreamer is on packaging. I know  that you're pretty sophisticated there. I know   there are projects like Bento4 that are primarily  packaging. So just trying and understand that, and   I think you've given us a good start. What can you  tell us about GStreamer's packaging capabilities?  As I said, they write little code, so basically  they're more of an aggregator. So they don't have  

a GPAC integration for example. And most of their  packaging relies on Ffmpeg, so mostly they have   the packaging capability of FFmpeg with a few  other components on web, et cetera, where they   wrote their own code. But otherwise I think  that Gstreamer has some particular markets.   It's more on the hardware side, which is why I'm  not surprised that a company like Net Inc would   try to use it. But when it comes to pure software  workflows, I think FFmpeg is way way more popular,   and it's been years that we've tried. So  it's really, GPAC is more about file-based   packaging with MP4Box. So MP4Box is the main  tool of GPAC, but then people had to encode   our content with FFmpeg, dump it and package it. I think it was okay a couple of years ago where  

people wouldn't trust open source software to  do some live, but now things have changed so   it's really difficult to be integrated as a third  party to FFmpeg. I hope that we can change that   in the near future. As near as possible for me,  because FFmpeg is super popular, but we decided   to do the reverse. We said, "Okay. We can build  something that we think is leveraging FFmpeg and  

which is better than FFmpeg." And so we have a new  application in GPAC, which is simply called GPAC,   like the FFmpeg application in FFmpeg that's  allows you to build complex workflows and   it's used for live by many people. We have  a lot of users on this. And as I said,   because we're doing some standardization and a  lot of R&D, in GPAC, not only it's a mix of a   lot of things it's a player, it's an analyzer,  it's a content generator. So you can encode,   you can package, you can stream, you can dump. We  embed an HTTP server, so it does a lot of things  Taking a step in a different direction  for a moment, what are you seeing on   the packaging side regarding utilization?  I mean, we've heard about CMF for years,   we know DASH, we know AHLS. What are you  seeing as trends in packaging utilization   among the people that you're working with? I said that we were doing standardization. We are  

advocating for ISO BMFF, so that's the MP4 file  format. And so because we are in standardization.   Because it's pretty active. Because they have  this open source software GPAC. When it comes to   open standardization impact systems and the impact  file format are really, really, really powerful.   ISO BMFF is everywhere. The QuickTime file format  on the prediction site, CMA, fragmented MP4, MP4,   everything that's related also to encryption with  common encryption DASH HLS, that's fragmented MP4,   CMA. For storage, there is MP4. I think there's  a huge consensus contribute to what happens in  

codecs right now, where the situation is pretty  fragmented. That's the same for input contribution   protocols where peoples to use RTMP, they use also  SRT, but they don't know exactly where to move.   When it comes to file formats, it's pretty stable. ISO BMFF is everywhere. And I believe that GPAC is  

the best tool. But again you can try. It's open  source. There is a lot of nice features I think   that people are not so aware of. If you take a  raw H.2645 for example, or H.265, a file that you   dump from your color, if you package it in MP4,  the MP4 is going to be smaller than your input,   because we eliminate all the redundancy and  in addition you can get some indexation of   the content. And you can have a lot of nice  features that come with packaging your content.  Well, how does that translate to HLS slash or  CMA? I mean does that mean DASH? It feels like   HLS is still the leader years after standards  based approaches became available. Are you   saying that's not true? I can't remember what  Bitmovin said in their developer survey. I haven't  

looked at that in a while. But they have  to output one of those packaging standards.   What are you seeing in terms of support? HLS has dominated for years. It's going to   continue to dominate. Okay.  Rather that's the truth, but it's not important,  right? The way you handle your manifest is just   really tiny part compared to what you are doing  with the rest of the data, which is the majority   of everything. And then there again, there is  a convergence on CMM, which is ISOBMFF based.   So I think we're pretty close now that there is a  convergence also on the encryption, because that   everything that was industry based was AES CTR and  Apple said, "Okay. But CBC and CBCS in particular   are where we want to head to." So there is  a consensus right now to go to CMA and for  

the encryption CBCS. So I think the convergence  is here, we are really near the point where you   would have only one media farmer to distribute. And that is ISOBMFF with different packaging. And   that's the whole promise of CMA,  is the manifest is such a small   portion of it really doesn't matter. It's messy because when you consider,   so I say ISO BMFF, which is maybe a technical term  for most of the audience, that's the base standard   that is directed at mpeg. And you can imagine  that CMA is a profile of it, so it's going to   restrict the toolbox. I would've liked it to be  restricted to a point where you say, "Okay. I   need to package this and exactly what's going to  be out." Like we have for beat accuracy, like we  

have for codecs. When you take an encoding stream  exactly where you are going to have as an output,   it's not that true anymore. There are a few  discussions on that, but basically that's what you   have. But still with CMA not restricted anymore. So there is a lot of freedom. And when there is  

freedom, of course interoperability is much more  difficult. But again, in GPAC, separate from the   domain GPAC repository, we have a repository  called Compliance Warden, and the Compliance   Warden is all about compliance on ISOBMFF and  maybe one day we'll get some sponsors. You have   the full ISO BMFF and CMA stack so that people  can actually have some better compliance on this.  So let's take a deeper look at GPAC. You said  it's open source, you said it's command line.  

Codex that it handles, pretty much the standard  H.264, HEVC, AV1. Any limitations there?  A lot more. You can mix everything. Old codecs.  yYou could DASH any product. You can DASH also   WebM, whatever. You can put special flags to be  interpretable with basically everything. That's   the chance. We have pretty big users, not only  Netflix. And it gives us wide, wide range of   application and internal probability. I think  that a lot of people, because we're doing R&D.   For example, VVC, we've had some integration with  VVC for three years. So basically when you want  

to package VVC, there is a high chance that you  come to GPAC, and you try GPAC. So it's not a real   strategy for us to try to attract users. But this  is something that is pretty effective compared to   other packages. You were mentioning Bento4. So  Bento4 has only one maintainer who hasn't worked   in packaging for six or seven years. So for me there is a lot of packaging  

projects that are really good used  by the industry. But there is a real,   real risk when you have only one maintainer and  the projects is not actively maintained anymore.   So that's why we were talking about that, there  are different strategies I think in open source.   And projects like FFmpeg or GPAC, they come from  a real pain point from users. Passionate people   who… That's my case, I needed this solution.  I started to contribute to GPAC and then I  

made a company because I'm passionate. I think  people can see, that I'm passionate about it. I   tried to find a way to make it a living. And  also with GPAC licensing, the commercial arm   of GPAC inside Motion Spell to allow other  people also to be able to live off their   contributions. FFmpeg they have the  same, right? They have FF labs now.  They had a booth at NAB. And for GStreamer  there were these two companies. One of them   Collabora. I think that's really important  to give the industry confidence that they  

are real people supporting the project and  understanding their needs. It's not a bunch   of teenager or kids in their garage, because  sometimes that's the feeling that you can have   if you go for example on the FFFA book tracker.  You know can have a wrong image but still they're   trying to build something for the industry. I mean, they're tremendously talented programmers.   What about, so looking back at GPAC, what  about HDR? I mean, Dolby support is huge,   HDR10+ is huge. What are your capabilities there? We support everything. We have a very generic  

support and the latest release that was last  December had a special focus on HDR. So we support   everything static, dynamic metadata, all kind of  metadata re-inject your raw metadata, you can fix   the metadata along the stream because sometimes  there is a lot of streams that are just broken.   Let's frame it this way. So we support everything.  And again, we have a really deep integration with   Ffmpegs internal that allows us to take the  raw matrix coefficient and the color space,   et cetera and to map it into something  that would be recognized as Dolby Vision,   or a HDR, HDR 10, or HDR10+, et cetera. Where does the Dolby licensing come in?   And I guess taking a step back, can I do Dolby  Vision with FFmpeg? I know I can do HDR10+,   but can I do Dolby with FFmpeg? And what about  the licensing? Is that just device decode side,   or is that on the encode side as well? I don't know. I know they support it on  

the encode side, they support the meta data. They  have some special side data that comes with the   packets of the data and they support it. That's  all I can say. I don't know more about this but   that's sufficient for GPAC to package it. You know, one of the things that's kind of   been interesting to me is the compare and  contrast for commercial software you can   get from a number of vendors. Harmonic comes to  mind. What are the pros and cons of open source,   versus buying a commercial version that's  going to have a pretty extensive UI,   it's going to have a lot of testing, it's going  to have people you can call and get support from   pretty easily? What's your sense of that decision? Well, I think there are several reasons why you   would want to do that. So I had the chance  to talk at NAB streaming summit with Netflix,  

and they explained that they do it  first because it's open standards.   They don't see how they could compete on  this. It's not their goal now. They are   pushing contributions there, they want to compete  on other things, not these open standards. And   I think Netflix made a demonstration when it  migrated to GPAC, they moved super fast. Like   you see evolving a newer generation of codecs.  They announced… And their base tier - they move to   live. I don't know any company able to move that  fast. So I'm not saying it's 100%, because you're  

using output tools or whatever. But clearly that's  points where packaging is crucial. How do you make   sure if you're doing interactive… How do you make  sure that you can stitch your account and do, you   know, any commercial tool that says, "Okay.  I can stitch and create scenarios on this"?  If you don't control exactly what a  packager does, you cannot do this.  

And again, when there is new cards, when there are  other things, using an open source tool, you know,   can leverage the work that has already been done  by the community. So I think that's important. But   that being said, Netflix is a pretty innovative  company so we have the chance to have them like   our sponsors. And also they pay us when they need  integration of new tools, new codex, or whatever.   For us it's really easy to work at  with such a company, and we understand   it's kind of really, it's easy customers. I mean, they've got a phenomenal engineering  

team and I guess certainly a lot of publishers  don't have the engineering resources, engineering   resources that Netflix does. But let's move to  Netflix. Tell us what it is you license to them,   and what it's being used for at Netflix. They gave a number. It's like, where was it?   It was a million. I don't remember the scale, but  that's millions of content every month that are   being packaged basically to package everything.  They mentioned that, as I said, on the prediction   side. They handle the content, or at least some of  the content, with GPAC, because they don't control  

a hundred percent of the formats that comes  in. Also previews for the journalists packaging   the content for their 200 million viewers. So my  understanding is that they package everything. But   again, when it comes to open stores, the magic is  that you don't know what your users do. We don't  

put telemetry in our tools, that's pure freedom. So I don't know what Netflix is actually working   on if they don't send me an email, or when we  work with customers, we have acceptance tests. So   that test basically that we both execute. So that  Motion Spell for GPAC would execute, and Netflix   would execute. And we agree that this shouldn't  be broken otherwise there would be a problem. We   send them an email or when they want to upgrade  their version of GPAC, they would have to execute   this list of tests to make sure, to give them  confidence. So that's the only way for us to know   how a customer actually uses GPAC. So is it for live? Is it for  

live NVOD, or you just don't know? I don't know. I know for VOD for sure   we were not able to talk about live for the NAB  presentation, but I hope that if there is some   GPAC involved we can also share things. They  have a really great technical blog like many   of the companies, and so it would be great if  they could communicate on it. But I don't know,   maybe this needs to match. I'm not into,  as I said, Netflix or any other customers  

internal. We're a pure technical vendor. So I mean that leads me to the whole open   source versus for fee. And, I mean, how does  that all work? It's open source software,   which means I can use it for anything I want. How  does it evolve that I end up paying you? Is that   something I have to do, or is that something I  may want do to get support, or how does that work?  That's something that you may want to do to get  some support. You may want to support us also.  

If you use us and you are successful, you may want  to support us so that the project is sustainable,   basically. But there are some more. Sometimes  the open source, the new LGL license is a bit   tricky for companies to deal with, you know. You  cannot forbid reverse engineering, and these type   of things. That, you know, look twice if you  want to accept such clauses. So we are in a  

position to do a license GPAC and so we can emit  our commercial licenses. But these licenses, it's   not looking specifically into details. What is  important for us is that people have a solution.   We're a building video platform and I'm pretty  sure I have all the statistics of course. The   majority of the packaging now is done with GPAC.  And so I'm pretty sure as I said, because we are   on open standards, I'm pretty sure that there's  not going to be a lot of competition on this.  I think it's going to stabilize. We have ISO  BMFF, which is a really good technical basis.  

And once the innovation slows down and I think  it's going to happen on where we are. I hope so.   There are still the questions. As I said to the  contribution protocols. We're not on it and there   is a real problem. There is also the problem  of what's going to be placed and impact TS.   Exactly. So now we went into HTTP delivery with  OTT and ISO BMFF is pretty good, but that's not   a streaming protocol. So there are still things  I think to figure out on the transport level,  

but that's the future. I think ISO BMFF is going  to stay. It's going to stay for 20 or 30 years   and after that, who knows? Give me the… Compare and contrast with   Bento4 because… It seems like you extended back  into FFmpeg for more of the transcoding operation,   but that primarily you're being used as a  packager. Is it Bento4 your most, I guess,   the most pressing competitor at this  point? Or the one that's most obvious?  I think a lot of people in the industry use  Bento4. Some of them realize that it's not   maintained anymore, and that's a technological  risk. But that's not the only risk that they face.   As I said, it's a problem. You go into  open source, and then the project is not   maintained anymore. You're happy with the  project but then there is a new video codec  

that appears who's going to implement this? If  you have nobody to coordinate the developments,   what happens? So I think again, that's what I  think a lot of people like GPAC, and they go to   see us because there is an active community. It's  an active project. We are always ahead of time.   Everything is easy, everything is  flexible. There is a commercial arm.   So I think that maybe the next move for us is also  to help customers to migrate from Bento4 to us.   Just because at some point there is no solution. There is another competitor, pretty big in the   same area, that's Shaka Packager. The two main  maintainers are not working on it. There is a   lot of people who came to sales and said, "What do  we need to do? It's a Google project. We need to  

de Googleize it, and then we need to maintain it.  And when there are new codecs, who's going to do   this?" Even for big companies, there is no inside  resources. Netflix is the proof of that. They had   their own packager and they decided to move to  open source, and then some of the companies who   went to the wrong, they couldn't know to an open  source project that is not maintained anymore.   Now what do they do? Either they  invest on something else like GPAC,   or they need to maintain the tool  that is not maintained anymore,   or create a community around it or whatever.  So I think it's also our responsibility  

to help people migrate easily to us. Okay. I should note I'm doing a talk at Mile High   next in a couple weeks and I looked at.. I mean  there are some responses to questions posed on   the Bento4 whatever the resource is for support.  I don't know if there's any new development. There   have been responses within the last two or three  weeks. So, I don't know what that means. I just   want to mention that, because I'm not saying it's  not better to switch over to your tool. I'm just   saying there are signs of life there that may  be different than what it is you're expressing. 

I think there are things that you can have  a look at how many open issues in a project,   how many pending pool requests. It gives you a  sense of is the project active. Are there people,   are there enough people to deal with the  deactivity of the project or whatever.   I think that for people comparing  opensource projects, most of the   time they're comparing the stars, which is  on GitHub when the projects are on GitHub,   which is a good indicator of the popularity  of project, but then look at the open issues.  I'm actually looking at GPAC now to integrate  into the same talk. What are the educational   resources that you have that's going to be  important both for new implementers and for   people switching over from Shaka and Bento4. How  developed are those capabilities on your website?  The website is pretty old. It's just it  dispatches users. That's all it does. We  

have a pretty good wiki that's hosted on GitHub,  so if you make searches on GitHub directly,   you can find a lot of things. And if you read with  me in the front page, we give a lot of pointers to   our test, our billboard, there is a oxygen for  developers. But yeah. Again, I think the wiki   is the best entry point for people. And then on  the common line we have some really good magic.   If you make GPAC minus H, and you put whatever  words, it's going to output everything that is   related to this word. And by reading the doc, we  learned that we have several level of verbosity   on the help. So as I said, we want to make  it magic. I think that for a lot of people,   common lines can feel painful, and the  learning curve is really, really important. 

What I would like is that in the future we have  a graphical interface. I think that would be   great for people to see what's happening, et  cetera. Because multimedia is a lot of magic.   You now have H.264, it's annex B, and then someone  has to take care of this. That's the same. ASE,   implicit signaling, ADTS, LATM. That's a  nightmare. When you don't look into the   details it can be a nightmare. And I think  people would educate themselves by having to  

some graphical interface showing to them what  is the actual work that's being done. And I   wouldn't do that only for users, I would do that  so that we can also keep talents, and make it less   complex. Talent retention is something that the  whole industry should look at because there is a   lot of gifted people leaving our industry after  a couple of years and we need more of this.  We have a question. Can you describe the  extent of your VVC support? And I guess vvc,   I know fronthoffer had an issue with getting their  VVC implementation into FFmpeg. Which version  

are you supporting and how does that work? Yeah, we're pretty up-to-date, because we   have a close integration with the people of  Open VVVC, which is an open source VVC decor.   They have some integration in FFmpeg and so we are  following the latest developments of the standard.   If that's not the case, just open an issue on our  bulk tracker, but we are pretty up-to-date. There  

is impact this week, and so we're going to make an  update if they are fixes to the spec or additions.  Another question on your inspector,  what can your inspector do and how   is it different from say media info? Yeah, okay, so I think Media Info is a   great inspector. You take media info, the name of  the file, you can go common line, you can go with   the graphical interface. It gives you immediately  an overview. But one thing that it doesn't do is   inspect the packets. Like deep inspection of  each of your packets, and GPAC does that. We  

have several levels of verbosity, and so it can  display general information and I would like,   as I told you that I want to improve that maybe  a graphical interface so that people can navigate   into the content, and then you can  go deep, deep, deep, deep up to the   point of every bit that's been parsed. Netflix had an anecdote about this that   GPAC was really, really picky and really deeping  going really deep in the bit streams and actually   it was reporting or they were able to report  errors on some standards where files were   already widespread. And then you have, what  do you need to do in this situation? Do you   need to modify the standard, or do you need  to ping the other implementation and say,   "Hey. You should modify what you are doing?" One last question. What is Motion Spell? I   think you mentioned that at the start and we  just got a question in that relates to that is   what are the licensing terms for GPAC and  FFmpeg? So what is Motion spell and what   are the licensing terms for GPAC and FFmpeg? So Motion Spell is the company that holds   the commercial arm of GPAC, and we're also  making products most coming from the GPAC   technology or related. So we have things that  are related now. We have a OTT live subtitle  

inserter for people having a live OTT stream, but  they need to add some titles on the top of it and   they don't know how to do it. And as I said, also  related to compliance, we're doing a lot of work   with the biggest companies in the industry. I  think that's pretty important. And regarding   the license, the license of FFmpeg and GPAC  are the same. That's LGPLv2.1+. And my chance   at Motion Pad is that I went to see every of the  copyright holder in GPAC and design an agreement   with me and I can put a commercial license, which  is basically your terms. We propose basic terms,   and you can modify them to suit your needs. Another question came in, what's the status   of MP4Box.js? Is that being maintained? It's not actively maintained, so now it's  

not true. We are, we're in the same situation as  Bento4, right? We're merging the pool request but   we're not actively developing it and there's  a reason for that. There is web assembly now   and cheap, you can build GPAC and run it in your  browser. So I think it's going to be deprecated   within a couple of years. That being said,  in the middle, I think MP4Box.js is really,   really nice tool that's really popular and  people use it coordinated with web card and   it can transform their content in  the browser. That's really amazing. 

Pleasure seeing good NAB, pleasure working  with you on this, and I'm sure we'll be in   touch. I'm going to need some help with the  presentation I talked about. I'll be in touch,   and thank you for participating today. Yeah. My pleasure. Thank you Jan. Thank   you everybody for attending, and if  you have questions, I'm always here.

2023-05-07

Show video