Ov5640 fps issue - 1080p@30fps

pradeep · March 6, 2019, 1:36pm

Hi all,

I am trying to record video @30fps with ov5640. 1080p settings are correct(vfe receiving 30 frames per second) but the recorded video is only 23fps.
vfe_isr_wm_done is happening only 23 times. Do anyone have idea?
In case of 1080p preview, i am getting 30 frames per second(vfe_isr_wm_done is happening 30 times).
where could be the frame drop? I will post my findings, if i get any.

Thanks,
Pradeep

Loic · March 6, 2019, 1:49pm

Which kernel version? which command are you running?

pradeep · March 6, 2019, 4:55pm

Hi @Loic ,
I am using 4.14.53 kernel(release/qcomlt-4.14 branch).

I am using below pipeline for record:
sudo media-ctl -d /dev/media0 -l ‘“msm_csiphy0”:1->“msm_csid0”:0[1],“msm_csid0”:1->“msm_ispif0”:0[1],“msm_ispif0”:1->“msm_vfe0_pix”:0[1]’

sudo media-ctl -d /dev/media0 -V ‘“ov5640 3-0078”:0[fmt:UYVY2X8/1920x1080 field:none],“msm_csiphy0”:0[fmt:UYVY2X8/1920x1080 field:none],“msm_csid0”:0[fmt:UYVY2X8/1920x1080 field:none],“msm_ispif0”:0[fmt:UYVY2X8/1920x1080 field:none],“msm_vfe0_pix”:0[fmt:UYVY2X8/1920x1080 field:none],“msm_vfe0_pix”:1[fmt:UYVY1_5X8/1920x1080 field:none]’

gst-launch-1.0 -e v4l2src device=/dev/video3 ! video/x-raw,format=NV12,width=1920,height=1080,framerate=30/1 ! v4l2h264enc extra-controls=“controls,h264_profile=4,video_bitrate=10000000;” ! h264parse ! mp4mux ! filesink location=test.h264

I am using below pipeline for preview:

sudo media-ctl -d /dev/media0 -l ‘“msm_csiphy0”:1->“msm_csid0”:0[1],“msm_csid0”:1->“msm_ispif0”:0[1],“msm_ispif0”:1->“msm_vfe0_pix”:0[1]’

sudo media-ctl -d /dev/media0 -V ‘“ov5640 3-0078”:0[fmt:UYVY2X8/1920x1080 field:none],“msm_csiphy0”:0[fmt:UYVY2X8/1920x1080 field:none],“msm_csid0”:0[fmt:UYVY2X8/1920x1080 field:none],“msm_ispif0”:0[fmt:UYVY2X8/1920x1080 field:none],“msm_vfe0_pix”:0[fmt:UYVY2X8/1920x1080 field:none],“msm_vfe0_pix”:1[fmt:UYVY1_5X8/1920x1080 field:none]’

gst-launch-1.0 v4l2src device=/dev/video3 ! ‘video/x-raw,format=NV12,width=1920,height=1080’ ! glimagesink

Please let me know your observations.

Thanks,
Pradeep

Loic · March 6, 2019, 6:36pm

could you try to increase buffer count (add num-buffers=12 to v4l2src) and add a queue element between capture en encoding (! queue ! v4l2h264enc... )

pradeep · March 7, 2019, 4:56am

Hi @Loic,

I tried, it is not working.
videotestsrc 1080p@30fps and 720p@30fps works fine. Have you observed this issue?

Thanks,
Pradeep

Loic · March 8, 2019, 8:05am

OK, just in case can you try to use the raw interface instead of pix, you’ll need to configure pipeline to msm_vfe0_rdi0 and add a videoconvert element between UYVY8 camera output and v4l2enc (for NV12 convertion).

pradeep · March 8, 2019, 11:45am

Hi @Loic,

I tried with RDI as well. Same issue.
Adding queue with buffer size like below to encoder, improved fps rate from 23 to 24-26fps(varying). Issue may be at encoder side.

’queue max-size-bytes=204857600’ ! v4l2h264enc

Default: 10 MB, 204857600 =195MB

Thanks,
Pradeep

pradeep · March 12, 2019, 1:18pm

Hi @Loic,

Please let me know, have you observed this issue when you tested?

Thanks,
Pradeep

pradeep · March 18, 2019, 5:21am

Hi @Loic,

Issue is fixed by increasing the clock frequency from 75 to 76 Mhz. Below is the change. Now 1080p is recording @30fps.

diff --git a/drivers/media/platform/qcom/venus/core.c b/drivers/media/platform/qcom/venus/core.c
index 41eef37…5608970 100644
— a/drivers/media/platform/qcom/venus/core.c
+++ b/drivers/media/platform/qcom/venus/core.c
@@ -340,7 +340,7 @@ static __maybe_unused int venus_runtime_resume(struct device dev)
{ 1944000, 490000000 }, / 4k UHD @ 60 /
{ 972000, 320000000 }, / 4k UHD @ 30 /
{ 489600, 150000000 }, / 1080p @ 60 /
– { 244800, 75000000 }, / 1080p @ 30 /
++{ 244800, 76000000 }, / 1080p @ 30 */
};

Thanks,
Pradeep

Loic · March 18, 2019, 8:41am

thanks, good to know, adding @svarbanov to check if your change can be applied.

pradeep · March 18, 2019, 9:29am

For Info, Below are clock frequencies from android:
/* Encoders /
<972000 490000000 0x55555555>, / 4k UHD @ 30 /
<489600 320000000 0x55555555>, / 1080p @ 60 /
<244800 150000000 0x55555555>, / 1080p @ 30 /
<108000 75000000 0x55555555>, / 720p @ 30 */

/* Decoders /
<1944000 490000000 0xffffffff>, / 4k UHD @ 60 /
< 972000 320000000 0xffffffff>, / 4k UHD @ 30 /
< 489600 150000000 0xffffffff>, / 1080p @ 60 /
< 244800 75000000 0xffffffff>; / 1080p @ 30 */

Keeler · February 27, 2020, 11:32am

Hi Loic

@Loic Will media-ctl pipeline configuration work with any UVC camera?

many thanks

Loic · February 27, 2020, 1:24pm

What do you mean? Here we use media-ct to configure the internal camera subsystem and route video frame to one of the camss video device. For UVC camera, you don’t need this, the camera will directly exposed as a video device that you can use with gstreamer.

Keeler · February 28, 2020, 9:35am

Hi Loic,

Correct me if i am wrong. My understanding is that ov5640 module outputs YUV422 format. and media-ctl is used to connect ov5640 to qualcomm’s csi physical layer then to isp and vfe where performs a format conversion to convert yuv422 to nv12 so that the v4l2h264enc can accept the data from ov5640.

In dragonboard’s camera readme file it stated below:

============================================================================

PIX interface of VFE

Format conversion of the input data.
Supported input formats:
- YUYV/UYVY/YVYU/VYUY (packed YUV 4:2:2 - V4L2_PIX_FMT_YUYV / V4L2_PIX_FMT_UYVY / V4L2_PIX_FMT_YVYU / V4L2_PIX_FMT_VYUY).
  Supported output formats:
- NV12/NV21 (two plane YUV 4:2:0 - V4L2_PIX_FMT_NV12 / V4L2_PIX_FMT_NV21);
- NV16/NV61 (two plane YUV 4:2:2 - V4L2_PIX_FMT_NV16 / V4L2_PIX_FMT_NV61).
- (8x96 only) YUYV/UYVY/YVYU/VYUY (packed YUV 4:2:2 - V4L2_PIX_FMT_YUYV / V4L2_PIX_FMT_UYVY / V4L2_PIX_FMT_YVYU / V4L2_PIX_FMT_VYUY).
Scaling support. Configuration of the VFE Encoder Scale module for downscalling with ratio up to 16x.
Cropping support. Configuration of the VFE Encoder Crop module.

Format conversion

Pipeline configuration for the format conversion looks like this:

sudo media-ctl -d /dev/media0 -l '"msm_csiphy0":1->"msm_csid0":0[1],"msm_csid0":1->"msm_ispif0":0[1],"msm_ispif0":1->"msm_vfe0_pix":0[1]'

Format configuration for NV16/NV61 output:

sudo media-ctl -d /dev/media0 -V '"ov5645 4-003b":0[fmt:UYVY8_2X8/1280x960 field:none],"msm_csiphy0":0[fmt:UYVY8_2X8/1280x960 field:none],"msm_csid0":0[fmt:UYVY8_2X8/1280x960 field:none],"msm_ispif0":0[fmt:UYVY8_2X8/1280x960 field:none],"msm_vfe0_pix":0[fmt:UYVY8_2X8/1280x960 field:none],"msm_vfe0_pix":1[fmt:UYVY8_2X8/1280x960 field:none]'

Format configuration for NV12/NV21 output (the format on msm_vfe0_pix source pad - for NV12/NV21 must be UYVY8_1_5_X8 - must be set explicitly):

sudo media-ctl -d /dev/media0 -V '"ov5645 4-003b":0[fmt:UYVY8_2X8/1280x960 field:none],"msm_csiphy0":0[fmt:UYVY8_2X8/1280x960 field:none],"msm_csid0":0[fmt:UYVY8_2X8/1280x960 field:none],"msm_ispif0":0[fmt:UYVY8_2X8/1280x960 field:none],"msm_vfe0_pix":0[fmt:UYVY8_2X8/1280x960 field:none],"msm_vfe0_pix":1[fmt:UYVY8_1_5X8/1280x960 field:none]'

And similar Gstreamer pipeline for a JPEG picture:

gst-launch-1.0 v4l2src device=/dev/video3 num-buffers=1 ! 'video/x-raw,format=NV12,width=1280,height=960,framerate=30/1' ! jpegenc ! filesink location=image02.jpg

Scale

Format configuration for NV12/NV21 with downscale ratio 2x. The compose element on msm_vfe0_pix sink pad defines the output size from the scaler. Syntax is (left,top)/widthxheight and only width and height are valid as this is scaling only. Downscaling with up to 16x ratio is supported:

media-ctl -d /dev/media0 -V '"ov5645 4-003b":0[fmt:UYVY8_2X8/1280x960 field:none],"msm_csiphy0":0[fmt:UYVY8_2X8/1280x960 field:none],"msm_csid0":0[fmt:UYVY8_2X8/1280x960 field:none],"msm_ispif0":0[fmt:UYVY8_2X8/1280x960 field:none],"msm_vfe0_pix":0[fmt:UYVY8_2X8/1280x960 field:none compose:(0,0)/640x480],"msm_vfe0_pix":1[fmt:UYVY8_1_5X8/640x480 field:none]'

Crop

Format configuration for NV12/NV21 with crop of the bottom right corner. The crop element on msm_vfe0_pix source pad defines the cropped area. Syntax is (left,top)/widthxheight and all fields are valid:

media-ctl -d /dev/media0 -V '"ov5645 4-003b":0[fmt:UYVY8_2X8/1280x960 field:none],"msm_csiphy0":0[fmt:UYVY8_2X8/1280x960 field:none],"msm_csid0":0[fmt:UYVY8_2X8/1280x960 field:none],"msm_ispif0":0[fmt:UYVY8_2X8/1280x960 field:none],"msm_vfe0_pix":0[fmt:UYVY8_2X8/1280x960 field:none],"msm_vfe0_pix":1[fmt:UYVY8_1_5X8/320x240 field:none crop:(960,720)/320x240]'

Scale + crop

Format configuration for NV12/NV21 with downscale ratio 2x and crop of the center area. Scaling is done first and then cropping (scaler module is in front of the crop module in the hardware pipeline of the VFE):

media-ctl -d /dev/media0 -V '"ov5645 4-003b":0[fmt:UYVY8_2X8/1280x960 field:none],"msm_csiphy0":0[fmt:UYVY8_2X8/1280x960 field:none],"msm_csid0":0[fmt:UYVY8_2X8/1280x960 field:none],"msm_ispif0":0[fmt:UYVY8_2X8/1280x960 field:none],"msm_vfe0_pix":0[fmt:UYVY8_2X8/1280x960 field:none compose:(0,0)/640x480],"msm_vfe0_pix":1[fmt:UYVY8_1_5X8/320x240 field:none crop:(160,120)/320x240]'

============================================================================

What i have is a USB camera that outputs YUV422 i wish to use the msm_vfe_pix to convert the format to NV12. However, it doesnt seem to be working as expected.

So i wonder if it is possible to do something similar and specify VFE input format and output format through media-ctl.

Many thanks.

Loic · February 28, 2020, 11:34am

No AFAIK, it’s not possible (at least with current driver), the pipeline can only have CSI as input. You can not provide your own buffer from USB camera. YUV422 to NV12 colorspace convertion can be ‘easily’ done in software with e.g. gstreamer.

Keeler · February 28, 2020, 1:33pm

Thank you for the confirmation.

Currently i use videoconvert element in gstreamer but it uses a lot of cpu power and also I experience some frame lost. So i wanted to test if it is converted by hardware would there still be frame lost.

If i use glcolorconvert element with glupload and gldownload it didnt work. It seems the format is incompatible as the internal data error is shown.

Loic · February 28, 2020, 1:40pm

is YUV422 the only format your camera can output (v4l2-ctl -d /dev/video0 --list-formats)

Keeler · February 28, 2020, 2:54pm

Not the only format but the only uncompressed format. The other format is mjpg, which we do not want.

Loic · March 2, 2020, 10:24am

did you to run your pipeline with egl:

export GST_GL_PLATFORM=egl

or

GST_GL_PLATFORM=egl gst-launch...

Keeler · March 2, 2020, 11:28am

Yes, I tried with egl platform. It didnt work either.

Warning is:
basersrc gstbasesrc.c:3055:gst_base_src_loop: error Internal data stream error.

basesrc gstbasesrc.c:3055:gst_base_src_loop: error streaming stopped, reason not-negotiated (-4)
ERROR: from element /GstPipeline:pipeline0/GstV4l2Src:v4l2src0: Internal data stream error.
Addtional debug infor:
gstbasesrc.c (3055): gst_base_src_loop (): /GstPipeline:pipeline0/GstV4l2Src:v4l2src0:
streaming stopped, reason not-negotiated (-4)
…