Every streaming technology vendor promises high picture quality, low costs and flawless playback. But how do these stacks really perform, at the viewer’s end? We wanted to know, so we started measuring. Not in a lab, and not from a brochure, but on the streams as viewers actually receive them.

Why we started benchmarking

Since 1994, Jet-Stream has been known for innovating, pioneering and deep knowledge of core streaming technology. We have always had an open character, and that comes with requirements: a solution has to work with multiple encoders, transcoders, packagers and CDNs. That openness is a precondition for sovereignty. It lets customers assemble their own pipeline of services, instead of being locked into a fixed combination of, for example, an encoder and a packager from the same vendor. That kind of forced bundling is outdated, and it certainly has no place in a time when organisations want to regain their digital independence.

At the same time, customers are looking for solutions that are affordable, digitally sovereign and well engineered. Our solutions therefore need to do two things at once: run their own encoders and packagers and serve streams themselves, but also work smoothlywith other vendors. First, that requires our own encoders and packagers to be of high quality and able to hold their own against the market. Second, our solutions must work with third-party solutions without dragging you into complex integration projects.

Jet-Stream had already built several robust encoders, and recently added a robust packager. We were curious how they compare to the rest of the market. On top of that, we are developing two new platforms: JTS and StreamZilla. Both need to be future-proof, standards-compliant and sustainable, deliver high picture quality and stability, use as little storage and data traffic as possible, and play on every device and in every player.

Measured in the wild

So we did not just want to benchmark our own technology against the market, we also wanted to put the products in the market under the microscope. How do they really perform? Do they work with other components, are they based on and compliant with the standards, and do they follow industry best practices?

To find out, we analysed the streams as viewers actually receive them: those of multiple Dutch broadcasters and of multiple paid Dutch OTT services. We tested every stream against the official specifications and against established guidelines for good packaging. Our own stack simply took part as one of the participants, without preferential treatment. You can learn a surprising amount of detail about a streaming stack simply by analysing manifests and segments. Has your vendor ever done that for you?

What the benchmark showed about our own stack

Not to boast, but the benchmark confirmed that our encoding quality is very good and that the entire stack complies with the official standards. Perhaps even more important: the stack is open and easy to integrate. The components do not depend on each other, and the architecture is solid enough that both the encoder and the packager handle variations in the source well, meaning video and audio that arrive just slightly differently than expected. That is exactly where things tend to break in practice.

What we saw in the market

Not surprisingly, most encoders and packagers in the market performed well. You do notice, however, that some systems are getting on in years and still run on outdated encoding and packaging settings. Beyond that, we found a number of striking problems.

One of the most striking was a far too high bitrate. Customers of such a platform end up paying roughly double for storage and data traffic, while viewers get no better picture, but do run into buffering problems sooner. The provider appears to offer sharp prices for storage and CDN traffic, but you are paying double for a worse service.

We also found configuration and compliance errors in manifests: the table of contents that tells the player which quality levels are available and how they are structured. If that description does not match the actual stream, the player makes the wrong decisions. The result is buffering, rebuffering, or a player that stalls completely.

At one party we saw a non-standard and weak packager architecture that shifts dependencies and responsibilities onto the encoder. That works, until it doesn’t: it significantly increases the risk of audio and video sync issues, stuttering and dropped streams, and the customer ends up with a dependency, a lock-in. That is not the level you expect from a professional vendor.

One party even mixed up its bitrates. A viewer on a slower connection, who should switch down to a lower quality, is instead sent to a higher bitrate. The result is even more buffering: exactly the wrong direction.

Finally, we saw poorly optimised pre-packaging, where video is packaged in fixed formats in advance and stored that way. Today that may go unnoticed, but it will cause a difficult and costly migration problem in the future. That is why the professional market increasingly uses just-in-time packaging, where video is only packaged at the moment a viewer requests it. You can then tune your packaging with a single configuration change, without having to repackage your entire video library (which quickly amounts to hundreds of terabytes).

Knowledge of the entire chain makes the difference

At some parties, the high quality is easy to explain: good encoders, renowned packagers, and above all knowledge of the entire chain. At others, you can see that the vendor managed to get a working chain up and running, but lacks the deeper knowledge. The costs and consequences end up with the customer, directly or eventually, usually without them noticing. After all, a stream that plays is not necessarily a stream that is right. Have you ever validated and benchmarked your own streams?

How does your streaming stack perform?

Curious how your streaming stack performs? Perhaps we have already included it in our benchmark. If not, we would be happy to run a benchmark for you and give you concrete tips to improve your stack right away. Get in touch with us for a confidential conversation.

Go back to see all blogposts