September 1, 2016 - Javier Martinez Canillas

Video Decoding with the Exynos Multi-Format Codec & GStreamer

Exynos SoCs have an IP block known as the Multi-Format Codec (MFC) that allows them to do hardware accelerated video encoding/decoding, and the mainline kernel has a s5p-mfc Video for Linux2 (V4L2) driver that supports the MFC. The s5p-mfc driver is a Memory-to-Memory (M2M) V4L2 driver, it’s called M2M because the kernel moves video buffers from an output queue to a capture queue. The user-space enqueues buffers into the output queue, then the kernel passes these buffers to the MFC where they are converted and put it in the capture queue so the user-space can dequeue them.

The GStreamer (gst) multimedia framework supports V4L2 M2M devices, but only for decoders the v4l2videodec element supports. Randy Li is working to also support M2M encoders in GStreamer (v4l2videoenc), but this hasn’t landed in upstream GStreamer yet.

This post will explain how to use GStreamer and the Linux mainline kernel to do hardware video decoding on an Exynos based machine.

Install a Mainline Kernel on an Exynos Machine

The first step is to install a mainline kernel in the Exynos board. We have been working lately to make the s5p-mfc driver more stable and to have everything enabled in by default, so the Linux kernel version used should be at least v4.8-rc1 and the defconfig used should be exynos_defconfig. I’ve made previous blog posts that explain how to cross compile and install a mainline kernel on different Exynos-based machines, including Exynos5 Odroid boards and Exynos5 Chromebooks, so refer to these to learn how to do this.

GStreamer Uninstalled Setup

The GStreamer package in most Linux distributions doesn’t come with support for the v4l2videodec gst element enabled, so GStreamer should be built from source to do hardware accelerated video decoding. Fortunately, GStreamer has the gst-uninstalled script to setup a development environment that allows the use of gst elements that were built from source and not installed in the system. There are many guides on how to use gst-uninstalled, including this great tutorial from Arun Raghavan.

When building the gst-plugins-good module, the –enable-v4l2-probe configure option must be enabled to allow GStreamer to probe v4l2 devices. This is needed because v4l2 device drivers can be probed in different orders, meaning the video device nodes can change. This option needs to be  enabled explicitly before building gst-plugins-good:

Find the Video Device Node for MFC Decoding

At this point, the s5p-mfc driver can be used to do hardware accelerated video decoding. You first need to know which video device nodes are for the s5p-mfc video decoder; gst v4l2videodec expects to use a gst element named v4l2videoNdec, where N is the video device node number.

First, get the video device nodes the s5p-mfc driver has registered; the v4l2-ctl command from the v4l-utils can be use for this:

There are two device nodes the driver has registered, v4l2-ctl can also be used to know more about these nodes:

The video device node for the decoder is /dev/video5 in this case, there should be a v4l2video5dec gst element that should be used.

Video Decoding

MFC supports a variety of video formats for decoding and produces decoded NV12/NV21 YUV frames. The v4l2-ctl tool can be used with the option –list-formats-out to list all supported formats.

Now that the video device node for the decoder and its corresponding v4l2videodec gst element are known, a supported video (e.g. H.264) can be decoded using GStreamer such as with this example pipeline:

Happy hacking!

Javier Martinez Canillas

About Javier Martinez Canillas

Javier was a Senior Linux Kernel Developer for the Samsung Open Source Group. He has contributed to different kernel subsystems with a focus on ARM and Exynos SoC support. Besides hacking, he enjoys spending as much time as possible with his wife Tami and their twins, running, reading and writing technical articles like "Kbuild: the Linux Kernel Build System", published by the Linux Journal.

Image Credits: Christophe Dumas -via Wikimedia Commons

Embedded Technology / GStreamer / Linux Linux / Linux Media / Video encoding /

Comments

  • clafar says:

    Interesting guide. What about encoding?

  • Madi says:

    hello
    great guide,thanks ,much appreciated!
    I am trying to follow this tutorial but is different to what is in this output!!
    $$ v4l2-ctl -d /dev/video5 –list-formats-out
    Index : 0
    Type : Video Output Multiplanar
    Pixel Format: ‘YM12’
    Name : 4:2:0 3 Planes Y/Cr/Cb
    Index : 1
    Type : Video Output Multiplanar
    Pixel Format: ‘YM21’
    Name : 4:2:0 3 Planes Y/Cr/Cb
    Index : 2
    Type : Video Output Multiplanar
    Pixel Format: ‘VM12’
    Name : 4:2:0 2 Planes 16*16 tiles
    Index : 3
    Type : Video Output Multiplanar
    Pixel Format: ‘TM12’
    Name : 4:2:0 2 Planes 64*32 tiles
    Index : 4
    Type : Video Output Multiplanar
    Pixel Format: ‘NM12’
    Name : 4:2:0 2 Planes
    Index : 5
    Type : Video Output Multiplanar
    Pixel Format: ‘NM21’
    Name : 4:2:0 2 Planes Y/CbCr
    —–
    my output not similar to your guide!!
    What should I do? , any help here?

    • Javier Martinez Canillas Javier Martinez Canillas says:

      Are you sure the /dev/video5 video device node is the MFC? What does v4l2-ctl –info -d /dev/video5 say?

      Also, what kernel version and board are you testing with?

      • Madi says:

        Thank you for your great answer!
        My output Terminal for this command is:
        $$>> v4l2-ctl –info -d /dev/video6
        Driver Info (not using libv4l2):
        Driver name : MFC
        Card type : decoder
        Bus info :
        Driver version: 1.0.0
        Capabilities : 0x04003003
        Video Capture
        Video Capture Multiplanar
        Video Output
        Video Output Multiplanar
        Streaming
        —————————————————————————-
        —————————————————————————–
        $$>> v4l2-ctl –info -d /dev/video7
        Driver Info (not using libv4l2):
        Driver name : MFC
        Card type : encoder
        Bus info :
        Driver version: 1.0.0
        Capabilities : 0x04003003
        Video Capture
        Video Capture Multiplanar
        Video Output
        Video Output Multiplanar
        Streaming

        Can you give me your image that I download It?
        Thank you very much!!

        • Javier Martinez Canillas Javier Martinez Canillas says:

          Hello Madi,

          Ok, so then you should use /dev/video6 and not /dev/video5 (the video device nodes number may change since this depends on the driver’s probe order and that is not deterministic).

          No, I don’t have an image to download it sorry. You can use any armhf rootfs and install mainline Linux and get a GStreamer uninstalled setup.

  • rezapop says:

    hi

    i use ubuntu image thats provided for odroid xu3 (can i do that?)
    i have a odroidxu4 and i am trying to show 4k video with gstreanmer but i have problems
    1- my video node is /dev/video6 and /dev/video7
    2- /dev/video6 is decoder and /dev/video7 is encoder
    3- driver name is MFC versin 1.0 instead of s5p-mfc
    4- the MFC driver dosent support h264
    can you help me?
    should i use specific kernel or device tree ?
    can you give me image that you used ?
    thanks

    • Javier Martinez Canillas Javier Martinez Canillas says:

      Hello rezapop,

      I’m not familiar with the default image for Odroid XU3 provided by HardKernel, but AFAICT it has a very old 3.10 kernel so the instructions here may not work.

      This post talks about the upstream Linux and gstreamer capabilities, so I don’t think it applies to the vendor tree provided with the Odroid XU3.

      • rezapop says:

        hi
        thanks i solve my problem with kernel
        now my problem is kmssink element of gstreamer i cant find it
        i use gst-inspect1.0 kmssink command and i fond no element
        what am i doing?

  • mbarr says:

    Hello,
    I’m running on the ODROID XU4 single board computer. When I am logged in graphically and run:
    gst-inspect-1.0 | grep “v4l2”
    I can see the hardware decoders and encoders. When I am logged in remotely via SSH, and not graphically the same command returns no devices. I can still see the devices with v4l2-ctl commands and are present in the /dev/ folder. Any thoughts would be appreciated. I would like to be able to run a stand alone application written with gstreamer.

    Thanks!

Leave a Reply to Javier Martinez Canillas Cancel reply

Your email address will not be published. Required fields are marked *

Comments Protected by WP-SpamShield Anti-Spam