Event-based imaging - increased speed, less data
Hello and welcome to this IDS Vision Channel event. My name is Boris Duché, I'm a product manager at IDS, and I will hold today's presentation introducing a newcomer in our area scan camera portfolio, the uEye EVS, integrating the disruptive event-based sensor co -developed by the French company Prophesee and Sony. To introduce the topic, I will use this side -by-side video comparison. On the left, you can see Luca Verre, the CEO and founder of Prophesee , talking. This video is acquired with a conventional camera, integrating a CMOS frame-based sensor. This technology, that is used in most if not all of the cameras in the world, whatever the type of application, was initially developed for consumer use cases.
The purpose was to record a scene to be later displayed on a screen or via a video projector in a cinema. The image quality is good, the technology is mature, yet it is not optimal for machine vision applications. On the right, you can see a rendering of the data acquired with an event-based sensor. With this technology, only the pixels detecting a change in luminosity over time will be activated. That's basically movement.
The lips, the hands, the eyes of Luca will generate data, while the background will simply be ignored, thus generating a significantly lower amount of data for real-time machine vision applications. Let's overview the conceptual differences between a conventional frame -based sensor and the event-based technology. A frame-based sensor is the type you would typically find in a consumer device, a video surveillance system, in a car, or in a conventional machine vision application. It is actually the type of sensor that you will find in almost every imaging device that is produced today.
It consists in a matrix of pixels, each being sensitive to light for a given period of time, called the exposure time. On a monochrome sensor, each pixel would be encoded with a single value, corresponding to the light intensity. On this slide, we take a look at a color image where each pixel is encoded with three values, corresponding to the red, the green, and the blue channels, each being typically encoded in 8 bits, that is a value that will range from 0 to 255. A monitor, or TV, a video projector, will then adjust at the pixel level the intensity of light it emits to recreate the image. The first pixel that we will take a look at is green. Therefore, the green channel will have the higher value.
This other example is a brown pixel, so the red channel will have the higher value. In machine vision applications, the goal being not to display an image but to process an image, it is common practice to increase the so-called bit depth at the pixel level to not 8 but 10 or 12 bits per channel. That will increase the dynamic range. The values, instead of ranging from 0 to 255, will range from 0 to 1023 or 0 to 4095. It gives more flexibility when part of the scene is dark and other part of the scene is bright. So, that will generate also a significant amount of data to acquire, to transmit, and to process.
In 4k resolution, which is pretty common today, 12 bits per channel that would make 36 megabytes of data for a single image. What we will focus on today is not single image processing, it is video processing. Let's take a look how does a video camera works. Once again, this technology was developed for consumer applications initially.
To record and display scenes that are pleasant to the human eye. The concept is pretty similar, you acquire a sequence of still images fast enough that they can be later displayed at the same speed. Today, we consider 60 hertz, so 60 images per second, to be conventional.
For an uncompressed stream in 4k, 60 hertz, 12 bits per channel, that would make 2.1 gigabytes of data per second. For a real-time machine vision application, that would simply be too much.
And today, either engineers tend to don't scale the images to process the video stream real time, thus losing the advantage of resolution at the sensor level. Either the application runs offline, that is, first we record a video, then we process a video, and then we analyze the results. An event-based sensor is specifically designed for machine vision applications. In the example you see in the slide, a typical machine vision application would try to evaluate the speed of the car, or to analyze the direction it's going from, and the direction it's going to. The background information would be useless, and in fact, any pixel information that do not correspond to the car would be filtered out later in a conventional frame-based camera.
That would generate a massive amount of data that would be unnecessary, and yet has to be transmitted, has to be processed by the machine vision algorithm. So the output of an event-based sensor is not an image, as it is with a frame -based sensor. It is a continuous stream of information acquired at the pixel level when an event, that is a light intensity change, is detected. Each event will contain four information, that is the x-y coordinates of the pixel that was activated, the polarity to distinguish between a positive and a negative contrast, so positive contrast light intensity is going up, negative contrast light intensity is going down, and the fourth element would be a timestamp. On an event-based sensor, pixel activation is asynchronous. It can be activated independently from any other pixel in the sensor.
There is no concept when talking about event-based sensors about light intensity. Either a pixel is activated, either it is not. This is a binary information. The concept of exposure time that we are familiar to with frame-based sensors does not apply. On a conventional frame-based sensor, the photo response curve is said to be linear. If you double the amount of photons that reach a pixel, you will double the pixel value until it reaches a limit known as the full capacity.
With the event-based sensor, we do not have a linear photo response curve, we have a logarithmic photo response curve, which permits to achieve a much higher dynamic range that is 120 decibels. 120 decibels being an unmatched dynamic range for a conventional camera. On this animation, you can see side by side the concepts of a conventional frame-based imaging system on the left, and an event-based camera on the right. Obviously, the information that we are interested in, in this demo, is the point that is seeking out onto the sensor.
With the frame-based camera, all the pixels are activated, though useless, generating a tremendous amount of information to transmit and process. On the event-based sensor, only the data that corresponds to the point will create events. Not only the data generated will be lower in quantity, but also the pixels will be much more reactive, avoiding the limitation known as the Shannon frequency that you face with any discrete system, such as a frame-based camera. Also, the event-based sensor having no concept of exposure time, it will create no motion blur, making it a perfect match for any application with fast -moving scenes.
Now that we have a clear understanding of the conceptual differences between a conventional frame-based sensor and an event-based camera, one may ask if this device should be operated with a conventional camera software, such as IDS peak, that we offer with our uEye Vision cameras. How should the event stream be processed? Would a conventional image processing library, such as OpenCV or MVTeC Argon, be an option. The uEye EVS is to be operated via Prophesee MetaVision Intelligent Suite. Prophesee develops this unique technology. They also offer the software to interact with it. The documentation is available online for free and without the need to create a user account.
The SDK for software engineers willing to integrate the camera within their own software is divided as follows. You will find the open modules, which are used to communicate with the camera, to set a few sensor parameters, to acquire the event stream. IDS will provide you with a partner plugin for the camera to be Android-compatible, whether you are working under Windows or Linux operating systems. And the advanced modules will contain algorithms or advanced tools to process the event stream.
As a user, you may want to access the source code of these advanced modules so you can use them as a basis and adapt them to the needs of your application. Therefore, Prophesee offers two different options to work with their software. You will have the option to work with the latest version of the Prophesee MetaVision Intelligent Suite. The latest version, you will get access to the source code also, but you have to pay a fee. You have to get in touch with Prophesee sales and Prophesee will make you an offer and you have to pay a one-time fee only for development so that you can access not only the binaries but also the source code. You can also decide to work not with the latest version but with the version 4.6.2,
which is a stable release. And in that case, you will not have access to the source code, but you will have access to the binaries for free. It is important to note that the sales structure of Prophesee Metavision software does not include runtime fees. You pay if you want to have access to the source code of the latest version, but once you have developed your application, you can deploy your application free of charge.
You do not have to pay for a runtime license. Once installed the Prophesee Metavision Intelligent Suite and the IDS plugin, the first thing that you want to do after connecting a camera is to open the Prophesee Metavision Studio software. For IDS customers who used to work with our uEye cameras, that would be the equivalent of other IDS peak cockpit software that is commonly used to evaluate or to set up a camera. You will find in here a rendering of the event stream. Don't get confused.
The output of the camera is not an image. We create in here an artificial one for visualization purposes. The background on this image will appear typically right. The events with a positive contrast will appear in blue, and the events with a negative contrast will appear in black.
You will set in here a few parameters, sensor parameters, typically to get rid of any noise that you would get on the background. From a customer's perspective, it is your journey to work with the newcomer, the uEye EVS. You would first get in touch with IDS sales or IDS distributors in your area to get a camera.
For a first evaluation, you can work with the free version of the Prophesee Metavision SDK. Just be sure to install the corresponding version of the IDS plugin from the product page. And here you go. You can open Prophesee Metavision Studio and get started to evaluate the technology. Once you have evaluated the technology and before you start development, you have to take a decision.
Should you stay with the free version of the software, the version 4.6.2, or should you invest the latest for a fee? Obviously, we highly recommend that we go for the latest version. The fee charged by Prophesee is known to be reasonable. It is a one-time fee only for development, yet it is up to you.
Maybe you do not need to have access to the source code. Maybe the version 4.6.2 is enough for your application. Anyway, when it comes to deploying your application, whatever type of software you decided to use, version 4 .6.2 or the latest, version 5.1, you will not have to pay a runtime license per system you deploy. It will be on this hardware. When it comes to operating the uEye EVS, another question you may face is how to operate the camera together with another IDS camera.
The software is of course compatible. You would have to operate the uEye EVS via the Prophesee Metavision SDK and you would have to operate the uEye Vision camera via the IDS Peak SDK. You just have to install both software on the host computer and make sure to include the corresponding libraries in your code when you start to develop your own software. Both cameras can be synchronized if connected together via the GPIOs.
One would be configured as a master, the other as a slave. When you synchronize multiple conventional frame-based cameras together, you expect them to acquire frames at the same time. Here, when working with a frame-based camera together with an event-based camera, the concept is different because the event-based camera does not output a frame.
So for every image acquired with the uEye Vision camera, you will generate a special type of event, a marker if you want, that will be integrated into the event stream of the UIEVS. Now that we have seen the conceptual differences between a conventional frame-based camera and the UIEVS with event-based technology, we understand how to operate them. It would be interesting to have an overview of the potential applications that we can address.
Our first example would be real-time tracking. Here you can see particles splattering around during a welding application. They are incredibly bright and numerous and the event -based sensor will not only permit to distinguish them from the background thanks to its incredible dynamic range, but also to track them real-time. In general, whenever you want to track real-time a fast-moving object, whatever the light intensity of the object, when you want to analyze its deformation in real-time or predict its trajectory, the uEye EVS would be a perfect match. In this second example, we address a use case that was not candidate for a conventional frame-based camera, that is vibration monitoring. Whether in the car or in the factory, engines tend to vibrate as for parts they are connected to.
With event-based sensing, you can measure real-time these vibrations with limited data usage, with limited processing resources. Think about it for predictive maintenance. Usually, you would rely on the maintenance plan of the engine manufacturer. Let's say once a year, you would have to stop the production line so you can change the air filter. And for those of you who have experience working in a car during the weekends, quite often you open the hood, you take a look at the air filter and you say, okay, it's brand new, but anyway, you open the hood and you will change the air filter. When you do it on your car during the weekend, it's no big deal.
When you stop a production line to do the maintenance, the predictive maintenance, it is another story. Now with the uEye EVS, you can track any change in vibration of the engine. If it starts to slightly deviate from the normal, it's not predictive maintenance anymore.
You know there is something to do that the engine is starting to malfunction. Another example with this video rendering that is significantly slowed down, you will find this kind of applications where particles or small objects are falling. In the chemical industry, when you produce pills or tablets, that's very high speed. I personally also face similar use cases in the agriculture for grape sorting machines. You have parts falling at high speed and you need to count them to analyze the shape for quality insurance or for later sorting. Usually these use cases are addressed with a line scan camera.
These are known to be extremely fast, but also very expensive and pretty complex to set up. If you set up the acquisition speed of a line scan camera too fast or too slow, you will deform the parts. The lighting systems for line scan cameras are pretty hard to find, pretty hard to set up, and pretty expensive also. With F1-based sensing technology, that is actually a very straightforward application. You just place the camera in front of the scene, use any conventional light that you would use with a frame-based sensor, and you might also actually get rid of specific lights.
Because with frame-based imaging, you need to reproduce very precisely the contrast in the scene from one frame acquisition to another. While with event-based sensing, as an event, a pixel can be activated or not, it's a binary information, it is less subject to changes in light. Now with this new example, once again, we talk about real-time process monitoring. You can think of an inkjet printer and you want to count and gauge the droplets so you know when to stop production to clean up the printing heads. This is today often addressed with line scan cameras that we just discussed, pretty complex to set up and expensive.
When it comes to using the UI-EVS event -based sensing technology to address the application, the application is once again straightforward to solve. Another application example would be plume monitoring. Medical inhalers, spray printing, thin film deposition, you want to analyze the spray distribution to assess the proper pressure is used, the shape is adequate, or a cleaning of the head is necessary. This type of applications are typically addressed with very expensive high-speed cameras today.
This will generate a massive amount of data for offline processing. Once again, you record the video and then you process the video and you analyze the result. While with event-based sensing technology, you can run such quality controls online, live, in a real-time application. The amount of data to transmit and process is reasonable and IDS, as a camera manufacturer, provides the camera in a small, cost-effective, easy to integrate yet industrial form factor. And last but not least, an application that you can address with the uEye EVS camera is 3D measuring. It is common practice to use a conventional frame -based sensor together with a laser line projector to extract a 3D profile when a part is moving on a conveyor.
Yet, it is known to be a limited solution when it comes to reflective materials such as metal. Working with event-based sensors, the concept will be exactly the same, only we will benefit from the incredible dynamic range and speed of the event-based sensor. That is a cost-effective solution, robust, fast, and easy to deploy.
So that's it for this presentation. I will conclude introducing the complete camera lineup. IDS has made the event-based sensing technology affordable.
We bring to the market a complete product line, from board-level cameras to robust industrial cameras, small form factors, cost-effective. Yet, the sensor you will find inside the cameras is not a low-end version of the Prophesee sensor. It is the ISN, the IMX636, with high speed and low latency and high resolution. We developed this camera in close collaboration with Prophesee, and I will seize the opportunity of this event to say thank you to the teams. Thank you to Philippe Berger, who helped a lot to have this very close collaboration, so we know our product is fully operational together with Prophesee Metavision Intelligent Suite software. Even though the product is soon to be released, we want to insist that the products are very mature already.
As we developed these cameras closely with Prophesee, we worked with more than 100 beta customers who regularly provided feedback so that we can adapt the hardware and firmware of the cameras to make it perfectly ready to use for animation vision applications. Thank you so much!
2025-03-06 12:56
