Why EUSV and EUSB (end-user stream validation and benchmarking) matter to CTOs of streaming services.

IN SHORT

Dashboards full of player data (RUM, CMCD) measure how well a stream is delivered, not how good the stream itself is. A stream can show green while viewers see dropped frames, consume twice the data or hear audio that lags behind.

EUSV (end-user stream validation) fetches the stream as a viewer receives it and checks the manifest and segments against the official specifications: encoding, packaging, timing and architecture. You measure the cause, not the symptom.

EUSB (end-user stream benchmarking) puts those same measurements next to the actual streams of other providers, and can also score efficiency, sustainability, sovereignty and privacy.

For the CTO, this delivers reproducible facts: what viewers actually receive, hidden risks, costs no viewer ever reports, and a stronger position in the conversation with the vendor.

If you run a professional streaming service, you look at dashboards every day, full of graphs from your viewers’ players: startup time, buffering, errors, selected bitrate. When everything shows green, the service seems healthy, and you conclude that you have done your job well. But such a dashboard only measures how well the stream is delivered, not how good the stream itself is. And that is exactly where the problems arise that viewers notice. EUSV (end-user stream validation) and EUSB (end-user stream benchmarking) give the CTO the tools to measure and tune the encoder-packager chain based on facts.

Measuring at the end of the chain

Real user monitoring, RUM for short, measures what the player reports. That is valuable, but also a bit outdated: CMCD, Common Media Client Data as it is called, is a relatively new standard that lets players send their measurement data to the CDN with every request. Both, however, only measure the end of the chain. They see that a player is buffering, but not why. They see which bitrate a viewer receives, and whether video segments are technically being loaded, but not whether that stream delivers a good picture.

As a result, a stream can look perfectly fine on the dashboard while the viewer watches a picture with dropped frames, consumes twice the data or hears audio that lags behind. The player reports no error, after all: technically, everything plays. And when something does go wrong, the dashboard is quick to point at the wrong culprit: buffering looks like a CDN problem, while the cause may be in the manifest.

And then you take a look yourself on your smart TV. The picture looks fine, and the blinkenlights confirm it. But a modern TV is remarkably good at hiding errors: clever filters and upscaling algorithms polish away artefacts, and a forgiving parser and decoder swallow without complaint whatever does not comply with the standard. On an older TV, a cheap set-top box or a phone with a stricter player, things look very different, or the stream simply stops. Your own device is not a good reference for everyone else’s.

Looking inside the stream

EUSV fetches the stream exactly as a viewer receives it, and inspects it in depth: the manifest is checked line by line against the official specifications, and the segments are analysed down to the level of individual frames and timestamps. So you measure not the symptom, but the cause.

Encoding

Is the bitrate in proportion to the picture quality, or is content delivered at archive quality, without a hard upper limit? Is the encoder running on speed presets or a version that is years old? Are B-frames used, the frames that reference both forwards and backwards and therefore save a lot of bits, and on HD the more efficient High profile? Is the bitrate ladder logically structured, without overlapping levels? Are keyframes placed at a fixed rhythm on the segment boundaries, or does alignment between quality levels rely on chance? And is the audio not sent again with every quality level?

An excessive bitrate looks like good news on the dashboard. In reality you pay double for storage and traffic, the viewer sees no better picture, and buffers sooner on a fluctuating connection. Missing B-frames or a keyframe every second easily cost 10 to 30 percent extra data, precisely in sports and in calm scenes. And a picture that falls apart into noise during fast scenes, or that has had the life encoded out of it: the player plays it back neatly, the dashboard reports nothing, and the viewer drops off.

Packaging and manifest

Does the bandwidth in the manifest match reality? Are codecs, resolutions and profiles listed completely and correctly? Is a current protocol version being used? And for live: is the window long enough to absorb hiccups and allow viewers to rewind?

A manifest that states too low a bandwidth makes the player choose a level the connection cannot handle. A manifest that announces a simpler codec profile than the one actually inside lets an older device happily start, after which it freezes. A ladder in which a lower level demands more bandwidth than a higher one sends the viewer exactly the wrong way. And with pre-packaged video that has a timestamp in the segment names, every reprocessing run breaks the caches. Your dashboard simply stays green.

Timing

In this part of the chain, almost everything revolves around timing, and EUSV measures it explicitly. Do the timestamps run continuously, or are there gaps and jumps? Is the segment duration stated exactly in the manifest, or rounded, so that every segment introduces a few milliseconds of difference that accumulates? Do segment boundaries fall at the same moment across all quality levels? Do audio and video stay in sync, even after days of uninterrupted broadcasting? Viewers notice deviations as a stutter or a brief black screen when switching quality, audio that lags behind, or a live stream that jumps back.

Architecture

A stream also reveals a lot about the chain behind it. There are, after all, many ways to align source signal, encoding and packaging with each other. The common, robust approach: each component does what it is good at, expects no more from the others than the standard prescribes, and can cope with small deviations. Some solutions deviate from this: responsibilities are shifted, or problems are masked and stay invisible until things really go wrong one day. Such a chain works as long as everything goes your way, but is far more sensitive to a faltering source signal, a restart or some packet loss. If a packager has to compensate for a flawed architecture or poor alignment, frames get skipped or shown twice, and you see that stutter immediately, for example during a football match. A live stream can even stop, where a well-aligned chain with a robust architecture would have absorbed the problem.

The effect on the CDN

Architectural choices carry through all the way into the CDN. If all channels publish a new segment at exactly the same moment, all cache misses arrive at the same time too: a spike on the origin every few seconds. If the packager is just too late, the first request gets an error, and if the CDN remembers that for a while, negative caching as it is called, entire groups of viewers miss a segment. One CDN is more sensitive to this than another: many small edges without an intermediate layer fetch every cache miss from the origin themselves, while request coalescing and an origin shield protect the origin. Your RUM dashboard then blames the CDN, while the problem lies in the combination of packager and CDN. A CDN problem is therefore far from always a problem of the CDN itself. Often the truth lies somewhere in the middle, and that is exactly why deeper expertise and properly measuring the chain are essential.

Measuring across the entire chain

A stream passes through many hands: source signal, ingest, encoder, packager, origin, CDN and player. Each of those links can cause a problem, pass it on or mask it. Yet in practice we often see that people work with only a limited number of monitoring dashboards, which moreover often hide the underlying logs. Then you can see that something is going wrong, but not where. Assumptions sooner or later lead to wrong conclusions and wrong choices, and you, as the customer, are the one who pays the price.

You may therefore expect your vendor to log and measure at every handover, to actually use those measurements, and to apply EUSV as the final check. With EUSB, that difference becomes crystal clear: parties with the expertise to understand, measure and tune the entire chain, versus parties that stop at “the dashboard says it works”. If anything is measured at all, because there are examples of that too.

Benchmarking: your chain next to the market

EUSB goes one step further than EUSV: you put the same measurements next to the actual streams of other providers. Not what is in their brochures, but what their viewers receive. You see how differently parties handle the same standards, where your own chain falls behind, and where it demonstrably outperforms your competitor. By scoring each component, from demonstrably wrong to demonstrably optimised, the comparison becomes concrete. You validate your own chain now and then, and you benchmark the market regularly: that is where you see exactly who is tuning their stack, or has replaced it. The nice part is that this can be done completely independently, based on the streams themselves.

More than technology

Besides the technology, you can also benchmark a chain on non-technical matters.

Efficiency and sustainability. Every superfluous megabit costs storage, traffic and energy, again for every single viewer. An efficient chain is therefore also a greener chain. Virtually no provider reports on this: there is still a lot to be gained.

Digital sovereignty. Where do origin, CDN and analytics actually run, and under which jurisdiction? That can simply be verified through DNS and network traffic. Sovereignty shows up in the brochure more and more often. In the DNS records, somewhat less so. Which SEAL level from the European Cloud Sovereignty Framework actually fits your chain?

Privacy. Is viewing behaviour sent to third parties during playback? And what are CDNs and RUM parties allowed to do, and what do they actually do, with the access logs about your market reach and subscribers?

Accessibility and future-proofing. Are subtitles and audio tracks included correctly, as the European Accessibility Act requires for many services? And can you replace components independently, or will the next migration be painful?

What EUSV and EUSB deliver for you as a CTO

Measuring at the player is a fine thermometer, but a thermometer tells you that you have a fever, not why. EUSV and EUSB are the diagnosis, and deliver surprisingly hard and rich information:

You see what your viewers actually receive. Not a test environment or a brochure, but the manifests and segments that arrive at their players.

You get facts instead of opinions. Common, open measurement tools, checked against public specifications such as the HLS standard and Apple’s authoring guidelines. Every finding can be traced back to a concrete rule or value, and is therefore reproducible.

You quickly know where you stand. The structure of a stream follows directly from the configuration. A single good measurement therefore already gives a complete picture, and repeating it is only needed at delivery, after a change, or as a spot check.

You make hidden risks visible. A standard is the contract between components. A validation shows where your chain relies on a forgiving player or a specific combination of components, before that breaks on a different component, player or during a migration.

You see costs no viewer reports. Viewers rarely complain: they zap away. And you don’t spot an excessive bitrate in a complaint, but on your invoice.

You are in a stronger position with your vendor. With reproducible measurements and a comparison with the market, you discuss facts, not feelings.

Seven questions for your vendor

Want to know how your chain is doing? Then start with these seven questions for your vendor.

  1. What requirements does your packager place on the encoder’s signal, where are they documented, and which architecture does the packager follow?
  2. What happens to the stream across the entire chain when the source falters or packets are lost along the way?
  3. Can you replace the encoder and the packager independently of each other, or are you locked into a specific combination?
  4. Has your chain been validated on its actual output, checked against the official specifications, or only on what its own monitoring reports?
  5. Does your vendor measure at every handover in the chain, or only at the end?
  6. How does your bitrate relate to the picture quality, and how do comparable services perform?
  7. Where do encoding, origin, CDN and analytics run, and which jurisdiction are they subject to?

If you don’t get a clear answer to these questions, that is an answer too. Perhaps not the answer you want to hear, but your customers will ask you these questions sooner or later. Or they will drop off.

Have your stream validated

Curious what is actually inside your streams? Jet-Stream is happy to run an EUSV on your own streams, complemented by a regular EUSB in which we put your streams next to the market, and if desired also on efficiency, sustainability and sovereignty. You will immediately receive concrete recommendations for improvement. Get in touch with me for a confidential conversation.

Frequently asked questions

What is the difference between RUM, CMCD and EUSV?
RUM and CMCD measure what the viewer’s player reports: buffering, startup time, selected bitrate. EUSV fetches the stream itself as the viewer receives it and checks the manifest and segments against the official specifications. RUM and CMCD see the symptom, EUSV the cause.

What is EUSB?
End-user stream benchmarking: the same measurements as EUSV, but placed next to the actual streams of other providers, so you can see where your chain falls behind or leads the way. You can additionally score on efficiency, sustainability, sovereignty and privacy.

How often should you validate a stream?
The structure of a stream follows from the configuration, so a single good measurement already gives a complete picture. Repeating it is needed at delivery, after a change, or as a spot check. The market you benchmark regularly.

Go back to see all blogposts