Difference between revisions of "DM36x Video Stabilization"

From RidgeRun Developer Connection
Jump to: navigation, search
(Created page with '=Overview= Video Stabilization (vs) is a video enhancement technique which aims to remove shaky motion from videos. We offer a video stabilization solution for dm36x that uses t...')
 
Line 1: Line 1:
=Overview=
+
= Overview =
Video Stabilization (vs) is a video enhancement technique which aims to remove shaky motion from videos.  We offer a video stabilization solution for dm36x that uses the TI's video stabilization library. This solution moves the  position of the upper-left corner of the frame in the opposite direction of the camera movement.
+
  
The TI's video stabilization library provides an algorithm for video stabilization. This algorithm  takes the data provided by the Boundary Signal Calculator module for the current and previous frame and returns the dimensions and coordinates of the upper-left corner of a sub-frame in the video frame. The changes in the corner coordinates compensates the camera movement.  
+
Video Stabilization (vs) is a video enhancement technique which aims to remove shaky motion from videos. We offer a video stabilization solution for dm36x that uses the TI's video stabilization library. This solution moves the position of the upper-left corner of the frame in the opposite direction of the camera movement.  
  
The Boundary Signal Calculator(BSC) is a hardware module part of the dm36x Image Pipe (Ipipe).  The  BSC generates vectors of sum of rows and columns pixels.  This hardware module is programmable to fit with the frame's properties.  This hardware is part of the image processing path for a raw CCD/CMOS data, so the video stabilization can only be used with sources that provide Bayer data.
+
The TI's video stabilization library provides an algorithm for video stabilization. This algorithm takes the data provided by the Boundary Signal Calculator module for the current and previous frame and returns the dimensions and coordinates of the upper-left corner of a sub-frame in the video frame. The changes in the corner coordinates compensates the camera movement.  
  
Video stabilization algorithm compensates the camera movement which is up to +/- 10% of the  
+
The Boundary Signal Calculator(BSC) is a hardware module part of the dm36x Image Pipe (Ipipe). The BSC generates vectors of sum of rows and columns pixels. This hardware module is programmable to fit with the frame's properties. This hardware is part of the image processing path for a raw CCD/CMOS data, so the video stabilization can only be used with sources that provide Bayer data.  
sub-frame width in horizontal direction and sub-frame height in vertical direction. Therefore, when we
+
use the video stabilization we capture  an image of  120% the desired output image's width and height.
+
  
=Implementation=
+
Video stabilization algorithm compensates the camera movement which is up to +/- 10% of the sub-frame width in horizontal direction and sub-frame height in vertical direction. Therefore, when we use the video stabilization we capture an image of 120% the desired output image's width and height.
  
RidgeRun video stabilization support is composed by three elements:
+
= Implementation =
*A kernel support for the BSC module.
+
*Ipiped functionality for BSC configuration.
+
*A gstreamer plugin for video stabilization.
+
  
==BSC kernel support==
+
RidgeRun video stabilization support is composed by three elements:
  
The kernel patch includes the BSC module in the previewer driver. That allows you to specify the desired configuration parameters for the BSC hardware module.
+
*A kernel support for the BSC module.  
 +
*Ipiped functionality for BSC configuration.
 +
*A gstreamer plugin for video stabilization.
  
If the BSC is configured and enabled before a video streaming is initialized, it will attach the BSC interrupt(IPIPE_INT_BSC). This interrupt is  triggered when boundary signal calculation is finished for each frame. Since BSC is a sub-module of the Ipipe it always operates on the input video frame. To keep the relation between the frame the BSC calculations, when the interrupt occurs the BSC data is copy to the end of the current frame.
+
== BSC kernel support ==
                       
+
==BSC in ipiped==
+
  
In order to apply video stabilization to a video, the BSC module must be configured first. The ipiped offers the capability of configure the BSC according to the image's width and height. It enables the BSC module and defines the parameters that uses the module maximum capacity and best fits the image dimensions.  
+
The kernel patch includes the BSC module in the previewer driver. That allows you to specify the desired configuration parameters for the BSC hardware module.  
  
Also, the corresponding ipipe-client command to access this functionality is provided: config-vs. e.g config-vs 768 576  
+
If the BSC is configured and enabled before a video streaming is initialized, it will attach the BSC interrupt(IPIPE_INT_BSC). This interrupt is triggered when boundary signal calculation is finished for each frame. Since BSC is a sub-module of the Ipipe it always operates on the input video frame. To keep the relation between the frame the BSC calculations, when the interrupt occurs the BSC data is copy to the end of the current frame.
 +
 
 +
== BSC in ipiped ==
 +
 
 +
In order to apply video stabilization to a video, the BSC module must be configured first. The ipiped offers the capability of configure the BSC according to the image's width and height. It enables the BSC module and defines the parameters that uses the module maximum capacity and best fits the image dimensions.
 +
 
 +
Also, the corresponding ipipe-client command to access this functionality is provided: config-vs. e.g config-vs 768 576  
 +
 
 +
This command is not build by default with the ipiped because it requires the kernel be patched to supported. If you applied the video_stabilize_support.patch to the kernel and want to use the command, you need to uncomment the lines:
  
This command is not build by default with the ipiped because it requires the kernel be patched to supported. If you applied the  video_stabilize_support.patch to the kernel and want to use the command, you need to uncomment the lines:
 
 
     ipiped_VALAFLAGS += --define=BSC_ENABLE
 
     ipiped_VALAFLAGS += --define=BSC_ENABLE
    ipipe_client_VALAFLAGS += --define=BSC_ENABLE
+
  ipipe_client_VALAFLAGS += --define=BSC_ENABLE
 +
 
 
in the Makefile.am and recompile the ipiped.  
 
in the Makefile.am and recompile the ipiped.  
  
==Video Stabilization plugin==
+
== Video Stabilization plugin ==
The gstreamer plugin for video stabilization is called dm365vs. dm365vs adds the capability of reduce the video shaking in a preview pipeline.  
+
 
 +
The gstreamer plugin for video stabilization is called dm365vs. dm365vs adds the capability of reduce the video shaking in a preview pipeline.  
  
The video stabilization needs to move a sub-frame in the video frame in the opposite direction to the camera movement. Therefore the dm365vs negotiates a transform of the image capabilities to reduce the input image. The plugin looks for an input image's width of 120% of the output image's width and for an input image's height of 120% of the output image's height. e.g an input frame of 768x576 results in an output frame of 640x480
+
The video stabilization needs to move a sub-frame in the video frame in the opposite direction to the camera movement. Therefore the dm365vs negotiates a transform of the image capabilities to reduce the input image. The plugin looks for an input image's width of 120% of the output image's width and for an input image's height of 120% of the output image's height. e.g an input frame of 768x576 results in an output frame of 640x480  
  
 
During the initialization this plugin configures the TI's vs algorithm with the parameters of the BSC and the input image's dimensions.  
 
During the initialization this plugin configures the TI's vs algorithm with the parameters of the BSC and the input image's dimensions.  
  
When each buffer arrives to the dm365vs, it inputs the BSC data attached to the buffer and the BSC data of the previous buffer in the TI's video stabilization algorithm. The VS algorithm returns the coordinates of the upper left corner of a sub-frame and size of a sub-frame in the video frame and the dm365vs adjust this results to the negotiated output image's size. The calculated position of the upper-left corner of the frame is pushed in the output dmai transport frame.
+
When each buffer arrives to the dm365vs, it inputs the BSC data attached to the buffer and the BSC data of the previous buffer in the TI's video stabilization algorithm. The VS algorithm returns the coordinates of the upper left corner of a sub-frame and size of a sub-frame in the video frame and the dm365vs adjust this results to the negotiated output image's size. The calculated position of the upper-left corner of the frame is pushed in the output dmai transport frame.  
  
=Usage=
+
= Usage =
  
First configure BSC module:
+
First configure BSC module and be sure to set the ipipe mode to continuous mode first:  
<pre style="background:#d6e4f1">
+
<pre style="background:#d6e4f1">ipiped&amp;
ipiped&
+
ipipe-client set-previewer-mode cont
 
ipipe-client config-vs 768 576  
 
ipipe-client config-vs 768 576  
</pre>
+
</pre>  
Then run the video stabilization with your video pipeline:
+
Then run the video stabilization with your video pipeline:  
<pre style="background:#d6e4f1">
+
<pre style="background:#d6e4f1"> gst-launch v4l2src chain-ipipe=false always-copy=false&nbsp;! video/x-raw-yuv,format=\(fourcc\)NV12, width=768, height=576&nbsp;! dm365vs&nbsp;!  TIDmaiVideoSink
gst-launch v4l2src chain-ipipe=false always-copy=false ! video/x-raw-yuv,format=\(fourcc\)NV12, width=768, height=576 ! dm365vs !  TIDmaiVideoSink
+
 
</pre>
 
</pre>

Revision as of 17:38, 20 March 2013

Overview

Video Stabilization (vs) is a video enhancement technique which aims to remove shaky motion from videos. We offer a video stabilization solution for dm36x that uses the TI's video stabilization library. This solution moves the position of the upper-left corner of the frame in the opposite direction of the camera movement.

The TI's video stabilization library provides an algorithm for video stabilization. This algorithm takes the data provided by the Boundary Signal Calculator module for the current and previous frame and returns the dimensions and coordinates of the upper-left corner of a sub-frame in the video frame. The changes in the corner coordinates compensates the camera movement.

The Boundary Signal Calculator(BSC) is a hardware module part of the dm36x Image Pipe (Ipipe). The BSC generates vectors of sum of rows and columns pixels. This hardware module is programmable to fit with the frame's properties. This hardware is part of the image processing path for a raw CCD/CMOS data, so the video stabilization can only be used with sources that provide Bayer data.

Video stabilization algorithm compensates the camera movement which is up to +/- 10% of the sub-frame width in horizontal direction and sub-frame height in vertical direction. Therefore, when we use the video stabilization we capture an image of 120% the desired output image's width and height.

Implementation

RidgeRun video stabilization support is composed by three elements:

  • A kernel support for the BSC module.
  • Ipiped functionality for BSC configuration.
  • A gstreamer plugin for video stabilization.

BSC kernel support

The kernel patch includes the BSC module in the previewer driver. That allows you to specify the desired configuration parameters for the BSC hardware module.

If the BSC is configured and enabled before a video streaming is initialized, it will attach the BSC interrupt(IPIPE_INT_BSC). This interrupt is triggered when boundary signal calculation is finished for each frame. Since BSC is a sub-module of the Ipipe it always operates on the input video frame. To keep the relation between the frame the BSC calculations, when the interrupt occurs the BSC data is copy to the end of the current frame.

BSC in ipiped

In order to apply video stabilization to a video, the BSC module must be configured first. The ipiped offers the capability of configure the BSC according to the image's width and height. It enables the BSC module and defines the parameters that uses the module maximum capacity and best fits the image dimensions.

Also, the corresponding ipipe-client command to access this functionality is provided: config-vs. e.g config-vs 768 576

This command is not build by default with the ipiped because it requires the kernel be patched to supported. If you applied the video_stabilize_support.patch to the kernel and want to use the command, you need to uncomment the lines:

   ipiped_VALAFLAGS += --define=BSC_ENABLE
  ipipe_client_VALAFLAGS += --define=BSC_ENABLE

in the Makefile.am and recompile the ipiped.

Video Stabilization plugin

The gstreamer plugin for video stabilization is called dm365vs. dm365vs adds the capability of reduce the video shaking in a preview pipeline.

The video stabilization needs to move a sub-frame in the video frame in the opposite direction to the camera movement. Therefore the dm365vs negotiates a transform of the image capabilities to reduce the input image. The plugin looks for an input image's width of 120% of the output image's width and for an input image's height of 120% of the output image's height. e.g an input frame of 768x576 results in an output frame of 640x480

During the initialization this plugin configures the TI's vs algorithm with the parameters of the BSC and the input image's dimensions.

When each buffer arrives to the dm365vs, it inputs the BSC data attached to the buffer and the BSC data of the previous buffer in the TI's video stabilization algorithm. The VS algorithm returns the coordinates of the upper left corner of a sub-frame and size of a sub-frame in the video frame and the dm365vs adjust this results to the negotiated output image's size. The calculated position of the upper-left corner of the frame is pushed in the output dmai transport frame.

Usage

First configure BSC module and be sure to set the ipipe mode to continuous mode first:

ipiped&
ipipe-client set-previewer-mode cont
ipipe-client config-vs 768 576 

Then run the video stabilization with your video pipeline:

 gst-launch v4l2src chain-ipipe=false always-copy=false ! video/x-raw-yuv,format=\(fourcc\)NV12, width=768, height=576 ! dm365vs !  TIDmaiVideoSink