“Can you decrease Elisa Viihde’s network traffic?” asked Matti Manninen. “You are consuming a large fraction of the total traffic in Elisa’s mobile networks.”

“We could stream at a lower resolution, or use a more efficient codec to get better compression,” I attempted to deflect.

“A more efficient codec sound great! Will you be ready before the summer?”

“Before the summer sounds optimistic. What makes it so urgent?”

“When people go to their summer cottages the networks in remote areas easily get congested. If we cannot decrease the amount of traffic, we will have to invest in more capacity.”

“I see. We’ll get to it then.” And so we did.

A look back

This conversation took place in the early spring of 2023. Now, two full years later, we can look back at three months of incident-free Viihde streaming of content encoded using High Efficiency Video Coding (HEVC) for the most-watched live channels in the service. This reduced Viihde-specific network traffic: the positive effects on network utilisation have allowed Elisa to avoid costs and improve the experience for customers in previously congested areas.

The design, planning, and execution of this work happened in the Viihde and Content Delivery teams in Consumer Services, part of Digital Services in Elisa’s Production unit. For us, the single most important outcome is that this major change of the streaming platform went largely unnoticed by customers. In this post, I hope to elucidate the importance of keeping the customer experience in focus and the efforts involved.

Pre-work is work

We used the first three months for a feasibility study. We selected HEVC as the new video codec based on its wide customer device support. Already at this stage we knew that there are customer devices that do not support playback of the highest available resolution in HEVC. We also had a rough idea of the investment and development we needed.

What we did not know was how to ensure comparable video quality as our customers would perceive it. Encoding in very high quality decreases the level of compression; customers would notice content encoded in low quality, especially if they enjoy Viihde on large enough TV sets.

Two identical TVs helped with quality comparisons

Planning is work, too

Seeing is believing. So, we got two large-enough TV sets and started watching (see picture). Setting this up was not trivial. It requires live transcoding to a HEVC adaptive bit ladder, a live origin, and a few test channels in our service. To enable user testing, we configured pairs of channels with identical content but different encoding configuration. For example, one channel used the legacy codec, Advanced Video Coding (AVC), in its production configuration. The other channel used HEVC configured with specific quality parameters.

With this setup, we conducted experiments designed in collaboration with Consumer Services Design, with help from Viet Ba Hirvola and Ari Salokannel. These are the results of conducting the experiment with the production AVC configuration, and the HEVC configuration that ended up in production (see figure). It demonstrates that our test subjects did not notice a drop in quality from AVC to HEVC, and in fact did not see a difference in quality between the two codecs.

We showed live content on one TV and asked, “Is the picture quality good?” After we got the answer, we uncovered the second TV and asked, “Is the picture quality better, the same, or worse?” Each dot on the diagram represents an answer. The participants were Elisians who are not experts in streaming technology.

During this phase we had a peculiar challenge. AVC- and HEVC-encoded video do look different, and you can learn to tell which one is which. Timo Hakkarainen found additional information on the topic. Armed with this new terminology, it became impossible not to know which codec was used to encode a particular video.

Suddenly, Sampo Uosukainen and Timo Hakkarainen started seeing mosquito noise (see picture). It was obvious, right? Or were we imagining it? Did it depend on the input signal? Was it stronger with a low-quality source signal such as footage from a camera mounted on a motorcycle? How about the faces in the crowd at a sports event? We looked and looked, and we scratched our heads and rubbed our eyes and argued about it for a while. “No one will ever notice this,” I’d attempt, “We aren’t even sure it’s real.” This line of reasoning did not convince anyone: “But what if a customer does notice and complains?!” Sampo exclaimed during one of our arguments.

Eventually, our colleague Kimmo Aarnio at Teleste noticed that we were inadvertently overloading the CPUs of the encoders. That was causing the visible mosquito noise in the video. It was real, and customers would have noticed and complained, and we would have had to backtrack a really long way to get it fixed.

Are we there yet?

We started planning in the summer of 2023. By the beginning of 2024, a lot of people were asking whether we would be done before the summer. By late spring 2024, we had made sufficient progress to start execution. The line between planning and execution is naturally fuzzy. We decided by asking a question:

Do we have the schedule and all the details for the change management tickets we need to deploy in order to stream HEVC encoded video to most customers on the most watched channels?

This is a stringent requirement that enabled us to explore, by doing, the details of each change we need to implement. By the end of the planning, we had mostly done all steps twice: once in our development environment, and once in our RC environment. We had a clear idea of what had to happen, in what order, how long it would take, and what the most obvious risks were. We had not yet done meticulous device testing, as we hadn’t had the time.

And so, we had a scare. By the end of May, after additional testing, we could no longer deny that most of the customer devices had a problem with the adaptive bitrate ladder we had chosen. An adaptive bitrate ladder is what allows the player to find a low resolution video stream when the download speed cannot support the megabits per second necessary for a 1080p (Full HD) signal. It works by offering multiple versions of the video, one at the highest resolution, along with several lower resolutions. We had cleverly planned to employ a hybrid bitrate ladder with AVC-encoded low resolutions, up to 720p (HD), and a HEVC-encoded 1080p step at the top. It worked, most of the time, and it was somewhat efficient with encoding capacity.

I was trying to rationalise the visible breakage evident from the device testing results. It was Leo Niemi who suggested that we backtrack on our hand-crafted hybrid AVC/HEVC ladder and instead use a full double ladder with all resolutions available in both codecs, to support all devices without complications. We did leave out the highest resolution in AVC, and so, the few customers who cannot watch HEVC in 1080p resolution will see 720p video. This regression in user experience for a small number of customers was necessary to enable other positive outcomes: reasonable additional investment in encoding capacity, a decrease in bandwidth consumption, and an improved experience for the vast majority of customers.

Afterword

This short recap left out a lot of details, specifically when things went smoothly. The encoding solutions provided by Teleste and AWS Elemental met all our special requirements. The Unified Streaming products in our origins and CDN worked flawlessly.

One valuable lesson for me is the importance of end-to-end testing even if the task feels impossibly big. Lauri Rytkönen and Simo Kauppinen provided invaluable help with testing, making the impossible doable.

In retrospect, the customers did not even notice, and the team seems content with a job well done. In the few moments of quiet panic, it helped to look at the outcomes of our decisions from the customer’s point of view:

Is it better this way?