How to use Xilinx OpenCV on RidgeRun V4L2 FPGA and FPGA ISP
Updated: Jul 10, 2020
The Xilinx OpenCV library is now supported by V4L2 FPGA and FPGA ISP projects. Thanks to the adapters from the V4L2 FPGA project, it’s now possible to extend the functionality of your accelerators. The supported Xilinx OpenCV libraries are:
Xilinx OpenCV 2018.2
Xilinx OpenCV 2018.3
For the list of compatible functions, please visit our developer wiki List of Xilinx OpenCV functions compatible with V4L2_FPGA
Similarly to the FPGA ISP development flow, it’s possible to get the same advantages of data flow processing and pipelining by using Xilinx OpenCV, since they are both built for Vivado HLS.
Describing a custom accelerator
The example accelerator is composed by the following modules:
Figure 1 - Example of a custom accelerator
The image representation in V4L2 FPGA is different from the Xilinx OpenCV Mat representation, since the data stream in V4L2 FPGA is fixed to 64-bits. To connect the streams to the Xilinx OpenCV module, a data type conversion is required. The V4L2 FPGA also facilitates a couple of interface functions to carry out this job:
utils::stream::rraxi2xfMat(stream_in, xfmat_out, width, height): to transform from the stream to the Xilinx OpenCV Mat.
utils::stream::xfMat2rraxi(xfmat_in, stream_out, width, height): to transform from the Xilinx OpenCV Mat to stream.
Taking into account the functions listed above, it is possible to code of the accelerator as follows:
The variables which interconnect the several modules are stream variables (FIFO-based variables). Also, note that the interfaces are just placed at the input and at the output of the pipeline. To interface to a FPGA ISP module, the same process is required to transform the data types.
About the throughput
As explained in our previous blog, RidgeRun FPGA-ISP to easily create a Image Signal Processing ISP on FPGA, the throughput of the whole custom accelerator will be limited by the slowest part of the pipeline.
According to the tests of this accelerator, on the PicoEVB at 4K:
Theoretical throughput: 48 fps
Experimental throughput: 40 fps (limited by PCIe 2.0 x1 bandwidth)
For more information, please refer to FPGA Image Signal Processing with Xilinx OpenCV and V4L2-FPGA.
After running the tests on an image of 2048x1536, the time of solution was 803910 clock cycles.
Figure 2 - Original picture (grayscaled in software)
Figure 3 - Output image from the FPGA
One of the important remarks about the output image is that the result is approximated by the Xilinx OpenCV, since the operations are done by using fixed-point arithmetic instead of floating-point to save resources. This can be seen by an error image (Software OpenCV - Xilinx OpenCV) in the following picture.
Figure 4 - Error of Xilinx OpenCV respect to the same pipeline in OpenCV performed on software
Current development and research
Support for memory-based algorithms, which requires to pass the image more than once.
Driver development for ZYNQ-based platforms.
Learn more in
Related blog from RidgeRun : RidgeRun FPGA-ISP to easily create a Image Signal Processing ISP on FPGA
Any Questions? : firstname.lastname@example.org
Visit our Main Website for the RidgeRun online store, for pricing information of the RidgeRun products and Professional Services. Please email email@example.com for technical questions. Contact details for sponsoring the RidgeRun GStreamer projects are available at Sponsor Projects page.