November 8, 2016 - Arun Raghavan
GStreamer Media Synchronization Made Easy
A lesser known, but particularly powerful feature of GStreamer is the ability to synchronize media playback across multiple devices with fairly good accuracy. However, with the way things stand right now, it requires some amount of fiddling and a reasonably thorough knowledge of GStreamer’s synchronization mechanisms. While we have had some excellent talks about these at previous GStreamer conferences, getting things to work is still a fair amount of effort for someone not well-versed with GStreamer.
As part of my work with the Samsung OSG, I’ve been working on addressing this problem by wrapping all the complexity associated with this in a library. The intention is for anyone who wants to implement synchronized streaming between devices on the same network to be able to do so with a few lines of code and basic know-how of writing applications based on GStreamer. I’ve started work on this already, and you can find the code in the creatively named gst-sync-server repo.
Design and API
Let’s make this easier by starting with a picture …
Let’s say you’re writing a simple application that uses two or more devices to play the same video stream in sync. The system would consist of two entities:
- A server – This is configures what needs to be played. It instantiates a
GstSyncServerobject on which it can set a URI to be played. There are other controls available, but I’ll get to those in a moment.
- A client – Each device runs a copy of the client and gets information from the server about what to play and what clock to use to synchronize playback. In practical terms, a
GstSyncClientobject is created and given a
playbinelement that has been configured appropriately (this usually involves setting at least the appropriate video sink that integrates with the UI).
That’s pretty much it. The application instantiates these two objects and starts them up; as long as the clients can access the media URI, the result is multiple streams that are magically synchronized on all devices.
Controlling the Streams
The keen observers among you might have noticed there is a control entity in the diagram above that deals with communicating information from the server to the clients over the network. While I’ve currently implemented a simple TCP protocol for this, my goal is to abstract the control transport interface so that it’s easy to drop in a custom transport (Websockets, REST API, etc.).
The actual sync information is merely a structure that’s marshaled into a JSON string and sent to the clients every time something happens. Once the application has media playing, the next thing is for the server to control playback. This can include
- changing the media that’s playing, including after the current media ends,
- pausing and resuming the media,
- seeking, and
- “trick modes” such as fast forward or reverse playback.
The first two of these work already, and seeking is on my short-term to-do list. Trick modes, as the name suggests, can be a bit more tricky so I’ll likely get to them after other things are done.
My hope is to see this library get used in a few other interesting use cases:
- Video walls -A number of displays that are stacked together to provide one giant display; each device effectively plays different rectangles from the same video.
- Multi-room audio – Play the same music across different speakers in a single room, multiple rooms, or even group sets of speakers and play different media on different groups.
- Media sharing – Play music or videos on your phone while your friends listen and watch at the same time (a silent disco app, anyone?).
At this point, an outline of the API is completed; I still need to create the transport abstraction, but that’s basically a matter of extracting the properties and signals that are included in the existing TCP transport. What I would like is to hear from you, my dear readers who are interested in this library, is the following. Does the API look like it would work for you? Does the transport mechanism I describe above cover what you might need? I’ve included example code that should make it easier to understand how this library is meant to be used.
Depending on the feedback I get, my next steps will be to implement the transport interface, refine the API a bit, fix a bunch of FIXMEs, and then see if this is something we can include in
gst-plugins-bad. Feel free to comment either on the Github repo, in the comment section of this post, or via the gstreamer-devel mailing list. Don’t forget to watch this blog for videos and measurements of how GStreamer synchronized fares in real life!
About Arun Raghavan
Arun contributes to the GStreamer multimedia framework and the PulseAudio multimedia server. He lives in Bangalore, India and generally lurks online as louiswu or Ford_Prefect.
Image Credits: Christophe Dumas -via Wikimedia Commons