August 25, 2015 - Mauro Carvalho Chehab
Working Out Complicated Problems at the Media Controller Workshop
This article contains a summary of the discussions that took place at the Media Controller Workshop in Espoo, Finland from July 29 – 31, 2015. A more detailed breakdown of these discussions can be found on Linux TV.
This is the first workshop dedicated to the Linux Media Controller. It follows a v4l summit that took place back in 2010 in Finland that established the current foundation for the media controller. This was aimed at properly satisfying the needs of reporting pipelines on the smartphone System on a Chip (SoC). The focus of this year’s workshop was to clarify the kernel→userspace interfaces and extend the Media Controller to be used on other subsystems that need to represent graphs like Linux Digital Video Broadcasting (DVB), Advanced Linux Sound Architecture (ALSA), and Industrial I/O (IIO). Samsung had a strong representation at this workshop, including Shuah Khan and Mauro Carvalho Chehab from the Open Source Group.
Media Controller API Discussions
We started the workshop off with discussions about important changes that are needed for the Media Controller API. We identified a few key areas that needed to be developed:
- Proper representation of kernel→userspace API interfaces
- The ability to provide dynamic pipeline changes
- A definition of the namespace for interfaces and entities
- Support additional properties
- Better representation of entities that belong to more than one type
It was initially proposed to use planes, as defined by ITU and IEEE to represent different layers of a data or media network (ITU-T G.800). This would provide control and data planes: a concept largely used in modern networks.
Instead, it was decided to add a new graph element type (Interface) to represent the elements that control the entities.
We came up with the following terminology:
- Entity – A functional unit, typically a hardware component or subunit in a hardware component
- Pad – Data Input/Output of an entity
- Interface – Control point that implements a userspace API
- Link – Connection between a source pad and a sink pad
It was also proposed to use the name association for the links between interface and entity, but there was no consensus about this idea. Our discussions picked up on the term “links” to also refer to the connection between an entity and an interface. We then picked up the term “interface links” when we needed to distinguish from “data links” (e. g. pad to pad links).
There were concerns that using a different type for “interface links” might lead to significant code duplication in the Kernel if a different graph type is used. This will be considered during the development progresses.
Media Controller API Working Proposal
At the end of the first day, we developed a working proposal to guide our immediate development efforts.
- Introduce struct media_interface as an object type alongside with entities in the Kernel and in the user space
- Start scratching patches, starting with V4L2 and proceeding to DVB
- Determine the level of code duplication with the Kernel.
- Target 4.3 at earliest, but more likely 4.4 for release of the first series of Media Controller changes.
The current API does a lousy job at properly defining entities that are coupled to interfaces. This prevented the enablement of the MC DVB support which was submitted in December, 2014. This needs to be fixed as soon as possible because it’s resulting in a major negative impact on using the MC. Since the MC API is currently in a broken state, no patches, drivers, or framework changes related to the Media Controller will be merged upstream until this is fixed.
The second day of the workshop started with a series of presentations in order to propose new ideas for discussion and to cover recent changes in this ecosystem.
Videobuf2 Redesign to Allow it to Support DVB – Representatives from Samsung offered a presentation via video conference about the scheduled changes that are planned in order to split the videobuf2 core from V4L2. This will allow the videobuf2 core to also be used for DVB.
DVB Pipelines – Samsung representatives via video conference presented about the pipelines. Essentially, the pipelines on real use-cases for embedded devices would look like:
tuner 1 -> -> demux -> Video
tuner 2 -> cam 1 -> demux -> Audio
tuner 3 -> cam 2 -> demux -> Data
ALSA and the Media Controller – This presentation covered ALSA, and how it and Dynamic Audio Power Management (DAPM) can be properly implemented using the Media Controller API. It finished by discussing an iterative version that is in the works, and how, ideally, there should be a single implementation of graph walking and power management.
Updates to ALSA and au0828 to share resources – Shuah Khan presented on changes to the Managed Media Controller API, Au8522, V4L2 Core, DVB Core, bridge driver, and ASLA in order to improve the sharing of hardware resources.
IIO – Industrial I/O and the Media Controller – This presentation focused on how IIO was created to support low speed analog to digital, and digital to analog data streams, but is struggling to keep up with modern, high speed devices. Increasing device complexity creates the need for topology information that can be used by applications; the media controller offers video processing pipelines that are very similar to the data processing pipelines needed for IIO.
Following the presentations, we had a short keysign party for the participants to cross-sign each other’s GPG keys.
The final day of the workshop was once again dedicated to discussion. We covered many topics throughout the day and if you want to learn more about these topics, please see the full report from the workshop.
In Brief, we covered the following:
- The possibility of adding a generic graph framework to the Kernel. This would require an extensive review of the Linux Kernel which would likely consume a large amount of time, so such proposal has been dropped for now.
- A topology for simple hybrid TV hardware (PC-consumer hardware)
- An agreement was archived about the changes needed at the Userspace API (UAPI) interface, which will be detailed later via the Linux Media Mailing List (firstname.lastname@example.org).
- Improvements to V4L2 and DVB applications to allow them to discover if they are capable streaming, or if some other interface is already using the device. This presents some significant challenges because of the general lack of precise hardware architecture information on data sheets. Due to current limitations, our only option is to decouple the interface links on the digital side when analog is in use and vice-versa
- The need to allow bidirectional pads for bidirectional connectors, like coax, that use time or frequency multiplexing.
- The use of properties to represent device-specific capabilities. This would help applications find the exact entity the application requires: something that is especially useful for device-specific applications.
- A new media.h file that combines a single memory area amended with an array of properties that refer to the memory area. This addresses problems associated with using an array of properties as an IOCTL argument which leads to an unfeasible allocation of memory for each key-value pair in the array.
Determining the Path Forward
All of the individuals from this event came away with a clear list of action items for the next few months. Workshops like these have been a great way for us to get together with other experts in the open source industry to work out common technical problems and determine a path forward that is beneficial for everyone. If you want to read a more detailed report of this workshop, check out the entry for it over on Linux TV.Image Credits: Derivative of work by Subcommandante
About Mauro Carvalho Chehab
Mauro is the upstream maintainer of the Linux kernel media and EDAC subsystems, and also a major contributor for the Reliability Availability and Serviceability (RAS) subsystems. Mauro also maintains Tizen on Yocto packages upstream. He works for the Samsung Open Source Group since 2013. Has worked during 5 years at the Red Hat RHEL Kernel team, having broad experience in telecommunications due to his work at some of the Brazilian largest wired and wireless carriers.
Image Credits: Derivative of work by Subcommandante