FPGA enters into the game of image processing when the application goes beyond the GPU domain, allowing the developers to connect peripherals directly to the FPGA and offering acceleration of complex algorithms.
Below are the some examples to illustrate the scope of FPGAs on image processing
Cameras connected directly to the FPGA on custom hardware.
Custom devices with sensors and actuators.
Artificial Intelligence on the edge.
FPGA Image Signal Processing (ISP) Library is a collection of accelerators and templated C++ modules to perform image processing on video streams on FPGA for Vivado High-Level Synthesis (Vivado HLS). With FPGA ISP, you can create a low-latency ISP applications which execute in a high-predictable way since it is purely based on hardware description.
The real power of FPGA ISP is the capability of interconnecting several ISP modules in cascade without having a penalty of executing each module individually in a sequential fashion. Thanks to the pipelining and data flow programming models available when describing ISPs with HLS.
Describing a custom accelerator
You can combine the modules listed above to create a high-throughput ISP pipeline, performing operations on the whole image or by single channels. Let’s go for an example.
Figure 1 - Example of a custom accelerator
To describe this accelerator, the code of the whole accelerator is as follows:
The variables which interconnect the several modules are stream variables (FIFO-based variables).
About the throughput
Concurrent execution of all modules is achieved by HLS DATAFLOW pragma, making the following modules work as soon as one datum is ready. This execution can be also seen as a pipelining approach.
Figure 2 - Concurrent execution model
Figure 2 shows how each stage of the custom accelerator executes with the time (clock cycles are illustrative). The result’s latency will be determined by the slowest stage of the pipeline. In this case, adding new stages to the pipeline won’t affect the performance if they don’t have a slower time than the current slowest part.
Another feature to highlight is the pixel-package model, where each module encapsulates pixels in 64-bit transfers. For 8-bit transmissions, 8 pixels are transferred per clock cycle, whereas for ARGB 32, the transfer rate is 2 pixels per clock cycle.
Current development and research
Support for memory-based algorithms, which requires to pass the image more than once.
Learn more about RidgeRun FPGA ISP project in our developers wiki.
Any Questions? : support@ridgerun.com
Contact Us
Visit our Main Website for the RidgeRun online store, for pricing information of the RidgeRun products and Professional Services. Please email support@ridgerun.com for technical questions. Contact details for sponsoring the RidgeRun GStreamer projects are available at Sponsor Projects page.