Venus Encoder GStreamer demonstration pipeline incorrect?

[Rob_Gries]

Hello,

I’m working with the Venus encoder on the Snapdragon 410c and came across this post from the GStreamer mailing list –
http://gstreamer-devel.966125.n4.nabble.com/How-do-gstreamer-interfaces-with-H264-hardware-encoders-and-creates-videos-tp4686862p4686881.html

Basically the maintainer, Nicolas Dufresne states the following:

I strongly discourage using extra-controls to select the profiles.
Please report to 96board so they can fix their wiki. Instead, the
profile should be selected using a caps filter downstream the encoder.
Applies to profile and level. Note that neither Venus driver or
GStreamer makes any validation of the profile/level combination, so you
can easily produce invalid stream at the moment.

The current wiki hosted by 96Boards shows the encoder being configured with extra-controls here:
https://github.com/96boards/documentation/blob/master/consumer/dragonboard/dragonboard410c/guides/camera-module.md#video-record-pipeline

Is the published encoder pipeline incorrect? How do I determine whether or not I am getting what I request as far as profile/level id is valid? The video output by the encoder seems to decode on some platforms but not others (e.g. Android will decode a stream without issue, but iOS and it’s videotoolbox implementation will have trouble decoding video at times).

I’m going to attempt to modify my encoder pipeline to incorporate the downstream caps filter to allow the proper profile/level id selection on the encoder. I will report back with my findings, in the interim can somebody take a look at this and change the demonstration encoder pipeline as necessary?

[Loic]

Yes AFAIU there is no check from gstreamer on applied v4l2 controls, so you can not know if control has failed or not.For profile selection (at least), look like gstreamer can manage that with the caps filter, checks validity and then applies the control. I’ll test that as well. feel free to propose change to the documentation if you get results before me.

[Rob_Gries]

Hello Loic,

Thanks for responding to my post, I am seeing some interesting effects when negotiating profile/level through the caps filter.

My results are below:
Caps Filter: resolution=1280x720, stream-format=byte-stream, alignment=au, profile=constrained-baseline, level=3.1, framerate=30/1:
Pipeline: gst-launch-1.0 -v -e v4l2src device=/dev/video3 ! video/x-raw,format=NV12,width=1280,height=720,framerate=30/1 ! v4l2h264enc ! video/x-h264, stream-format=byte-stream, alignment=au, profile=constrained-baseline, level=9, width=1280, height=720, framerate=30/1 ! h264parse ! mp4mux ! filesink location=capsFilter.mp4
Result: Fails to negotiate
Log: https://gist.github.com/RobGries/0703499b133eb4dda63f30da8d68f265#file-caps-filter-profile-and-level-txt

Caps Filter: resolution=1280x720, stream-format=byte-stream, alignment=au, profile=constrained-baseline, framerate=30/1:
Pipeline: gst-launch-1.0 -v -e v4l2src device=/dev/video3 ! video/x-raw,format=NV12,width=1280,height=720,framerate=30/1 ! v4l2h264enc ! video/x-h264, stream-format=byte-stream, alignment=au, profile=constrained-baseline, width=1280, height=720, framerate=30/1 ! h264parse ! mp4mux ! filesink location=capsFilter.mp4
Result: Negotiates properly, but chooses an incorrect level id for the resolution provided (uses level 1, which is not valid for 720P/30)
Log: https://gist.github.com/RobGries/0703499b133eb4dda63f30da8d68f265#file-caps-filter-profile-and-no-level-txt

Caps Filter: resolution=1280x720, stream-format=byte-stream, alignment=au, profile=constrained-baseline, level=3.1, framerate=30/1 with extra-controls=extra-controls="controls,h264_level=9(3.1),h264_profile=1(constrained-baseline)
Pipeline: gst-launch-1.0 -v -e v4l2src device=/dev/video3 ! video/x-raw,format=NV12,width=1280,height=720,framerate=30/1 ! v4l2h264enc extra-controls=“controls,h264_level=9,h264_profile=1” ! video/x-h264, stream-format=byte-stream, alignment=au, profile=constrained-baseline, level=9, width=1280, height=720, framerate=30/1 ! h264parse ! mp4mux ! filesink location=capsFilter.mp4
gst-launch-1.0 -v -e v4l2src device=/dev/video3 ! video/x-raw,format=NV12,width=1280,height=720,framerate=30/1 ! v4l2h264enc extra-controls=“controls,h264_level=9,h264_profile=1” ! video/x-h264, stream-format=byte-stream, alignment=au, profile=constrained-baseline, level=9, width=1280, height=720, framerate=30/1 ! h264parse ! mp4mux ! filesink location=capsFilter.mp4
Result: Fails to negotiate
Log: https://gist.github.com/RobGries/0703499b133eb4dda63f30da8d68f265#file-caps-filter-profile-and-level-with-extra-controls-txt

Caps Filter: resolution=1280x720, stream-format=byte-stream, alignment=au, profile=constrained-baseline, framerate=30/1 with extra-controls=extra-controls="controls,h264_level=9(3.1)
Pipeline: gst-launch-1.0 -v -e v4l2src device=/dev/video3 ! video/x-raw,format=NV12,width=1280,height=720,framerate=30/1 ! v4l2h264enc extra-controls=“controls,h264_level=9” ! video/x-h264, stream-format=byte-stream, alignment=au, profile=constrained-baseline, width=1280, height=720, framerate=30/1 ! h264parse ! mp4mux ! filesink location=capsFilter.mp4
Result: Negotiates properly and uses the proper level parameter
Log: https://gist.github.com/RobGries/0703499b133eb4dda63f30da8d68f265#file-caps-filter-profile-no-level-extra-controls-level-txt

Key Takeaway: It seems like the downstream (of the encoder) caps-filter can properly negotiate profile, however it cannot negotiate level as when it is set it fails to negotiate the stream properly and leads to a failure to encode. I am able to work around this by setting the upstream “extra-controls” for level along with the other characteristics of the stream such as resolution, stream-format, alignment, and framerate. Note that when neither the extra-controls nor the caps-filter set the level, it defaults to level 1 which is invalid since the resolution exceeds it. For more information regarding levels/resolution consult this table on wikipedia here: Advanced Video Coding - Wikipedia. Also note that this is not an exhaustive list of codec parameters that are able to be configured. There are many more different configurations that can be used here, the difficult part is ascertaining whether or not the codec parameters are compatible with a wide range of decoders. Also as stated above in the gstreamer mailing list post, there are no checks that engage on either the v4l2 codec side nor the gstreamer pipeline side that will confirm the validity of the stream. It would be helpful to ascertain a codec-side configuration that attempts to strictly adhere to the H.264 specification as there is a high possibility that parameters supplied to the encoder can create situations where looser conforming decoders may decode a stream with bitstream errors, but stricter decoders will fail to decode.

[Rob_Gries]

@svarbanov and @Loic

I was able to catch Nicolas Dufresne from GStreamer on #gstreamer on freenode and shared what I have posted here. Nicolas had the following to say about the potential issue that can cause Venus to create non-conformant H.264 streams:

{ndufresne} rob_gries, in GStreamer I have assumed the driver was to handle this, but apparently not
{ndufresne} I have no extra logic to make sure the level matches the profile / resolution and bit_depth, but that could be added
{rob_gries} The driver just appears to default the encoder profile/level irrespective of resolution/framerate/bitrate at current. I am unsure if the maintainer of the Venus driver is planning to add that logic at some point, but in the latest 96Boards kernel baselines this is still not supported.
{ndufresne} rob_gries, I can ping Stanimir, or you can do that, just jump on #v4l he’s svarbanov
{ndufresne} but I think long term it would be nice to have this in GStreamer itself
{ndufresne} I think we never made any helper because x264 was doing it for us, and likely VAAPI does the same (this last one need double check)

Basically since neither Gstreamer nor the Venus V4L2 driver have a mechanism to check the conformance to the H.264 spec when creating an encoder session, we are able to create non-conforming streams that may be able to be played back on some devices, but not all. This makes it difficult to rely on Venus as a reliable hardware codec. This could be worked around if either the driver or GStreamer were to check if the encoder was creating a valid stream.

[Loic]

Hi @svarbanov, any plan here?

[svarbanov]

Hello, I haven’t thought that this is responsibility of the driver, and I’m still unsure. There are v4l2 standard controls to set the profile and level, and I think that those two are most important codec parameters which the user application must set even before framerate and bitrate are known. I’d think what can be done in the driver.