Back
R&D
360° video
360° video, or immersive video, allows you to no longer just passively watch video content by following the director's choice of camera shots, but to fully immerse yourself in the heart of a place and choose where you watch. Readable from any type of terminal, it can be viewed on a computer as well as on a mobile phone: just move your phone to change the viewing angle transparently, as if you were filming yourself.
Equipped with a virtual reality headset like the modest cardboard Google Cardboard or the top-of-the-range Samsung Gear VR, you feel like you're in the scene as if you were there.
Limitations of current technologies
In 2016, we have increased the quality of our 360° content that can be viewed live at the user's premises to 4K resolution (4096x2048, which is more or less the same resolution as 4K / UHD screens), which has enabled us to achieve a certain level of reading comfort on mobile terminals: as the screen of the phones remains small, the eye does not perceive the number of pixels and their distortion. But reading the same 360° content on a large screen like a TV or in a more comfortable virtual reality headset like the Samsung Gear VR proved to be much more tiring for our eyes. And yet, we're talking about 4K resolution, the same as the beautiful images we sometimes see on high-end TVs - if you've never seen one, go to your favourite electronics store to see the latest TVs, it's worth the diversion! Let's try to understand why there is such a difference.
With a traditional non 360° video, the problem is trivial: the total number of pixels contained in the video (the resolution) corresponds exactly to the number of pixels you see on your screen. On an old non-HD TV screen like we all had just fifteen years ago, looking at an SD image (720x576) very closely would allow you, if you don't care to look at it, to count the pixels individually: 720 horizontal, 576 vertical. The same is true with HD (1920x1080) and UHD (3840x2160), even if the pixels are harder to see with the naked eye.
With 360 video, the calculation is quite different! In fact, the entire virtual sphere - in front of you, behind you, on your sides, below, above - is packaged in a traditional rectangular video format in a style that strongly resembles the photo panoramas of the 2000s.
Voici à quoi ressemble une vidéo 360 dans son intégralité :
We notice in passing that it is the same principle to represent the planet Earth, spherical, in a map of the world held on a rectangular sheet of paper: the mathematical foundations are in fact nothing new!
In this format, which is called equirectangular format or spherical format, we are well in the classic case seen above where the number of pixels contained in the video corresponds to the number of pixels that can be seen on the screen: here 4K maximum. Note that to see the above video in 4K, you will need a compatible screen and a high bandwidth.
But to fully enjoy the 360° video and not play the chameleon that sees behind his head, the video player must understand that it is a 360° video and reproject it correctly on the screen so that we see only a part of it. We can then walk around in the image and choose our point of view, but in return we only look at a small part of the available image! We actually enjoy about 1/8th of the resolution read by the video player, which is a very bad HD-Ready (1280x720) with the added bonus of pixel distortion. Viewing it on a TV screen 10 times larger than a phone or in a virtual reality headset a few centimeters from your eyes greatly exacerbates the defects then visible.
In order to obtain an image that could flatter the retina and reach the limits of the human eye in a virtual reality helmet, it is commonly estimated that an image of 16K (16384x8096) or even 32K (32768x16384) would be needed! This is of course unimaginable: the power needed would be gigantic and the bandwidth required would be enormous. All this to watch only one eighth of the image, what a waste...
Encoding in "tiles
At the Innovation and Prospective Department, while we were handing over the 360 4K video to the production teams of France Télévisions with Roland-Garros VR, we studied and tested ways to increase the resolution while remaining within the power and bit rate constraints acceptable to the general public, by joining forces with the know-how of companies recognized in the field such as Harmonic or Ateme and with the invaluable help of producers such as Bemersive or Digital Immersion.
One method consists in "pre-cutting" a 360° video into many pieces called "tiles", each one composing a small angle of view. The 360 video player will then dynamically download and decode only those tiles that fit into the field of view, depending on the user's orientation. All other tiles outside the field of view are not downloaded, saving valuable bandwidth and processor power.
If the user decides to turn his phone, changing the viewing angle, then the new tiles will be downloaded and the old ones ignored. Optimization techniques are then essential to avoid having a black or frozen screen for two to three seconds, as is usually the case with classic streaming technologies, for example when uploading a new video on france.tv or Youtube. One of them is to continuously download a very low definition version of the entire sphere to use it in the classic way if the user quickly turns his head, waiting for the high resolution tiles to be loaded.
During Roland Garros 2017, we were able to test the production of two 360° video shoots in 8K and 16K, which we then encoded and - after several days of calculations - displayed on Virtual Reality headsets and giant TV screens.
The other significant advantage of this optimization technique is that it allows a larger number of users to benefit from the 4K resolution by reducing the necessary bandwidth: by using the same process on a 4K 360° video, we can divide the rate necessary to view the video by almost 4 to reach approximately the rate of a classic Full HD video (1080p), and thus allow all users who previously did not have the bandwidth necessary to watch a "classic" 4K 360° video to benefit from the quality improvement all the same.
The more "technical" readers among you will be able to read the presentation in English of the results of GPAC's work, notably supported by the Institut Mines Télécom, on the subject through a Dash implementation completed by Ateme, or the presentation in English on Tiledmedia'swebsite of their technology completed by Harmonic.
And at France Télévisions?
Although this technique is promising, there are still today few or no commercial solutions for France Télévisions to deploy it industrially. Various players are working to make their products more reliable and to reduce the power required - at the time of writing we need powerful and recent telephones to take advantage of this - and could soon offer their technology to distributors such as France Télévisions, which, thanks to these players, could soon continue its experiments on a larger scale.
However, we will note the complexity of setting up such a method. Integrating support for classic 360 videos can be done quite simply without having to change the entire existing infrastructure - as we have seen: these are classic "rectangular" videos, so that encoders, storage spaces, content distribution networks (CDNs), decoders, etc., do not actually know that they are 360 video, and only the video player has to work to display the right part of the image.
On the other hand, implementing tiling techniques requires changing and rethinking a large part of the infrastructure, potentially resulting in much higher initial setup costs.
Finally, although this point has not been addressed in this article, 16K capture at least for live broadcasts still requires sets of cameras and servers with very high budgets, limiting for the moment the democratization of these technologies, before Moore's law takes effect over time.
