[Inter BEE 2016 TV] (NHK Science & Technology Research Laboratories) It's coming out of the screen! Augmented TV: an augmented experience; a proposal to use digital signage with vertical displays

　NHK Science & Technology Research Laboratories demonstrated Augmented TV, an image technology that connects real life with digital technology, at the INTER BEE IGNITION booth at InterBEE 2016.

　Using AR technology, you can watch television through your tablet's camera, while also experiencing the special effect that what you're watching on TV is jumping out of the screen. Smoothness was a major focus of the development of a technology to synchronize images and CG with high precision on a tablet.
　In addition, they developed a method to estimate the position and orientation of a television screen using a gyro sensor and image processing, making it possible to make it seem as though the characters move freely in front of the television screen.

- What happens when AR is introduced to TV? From "Expansion TV" to "Augmented TV"
　VR technology and content attracted attention in 2016, but after that is AR (Augmented Reality) or MR (Mixed Reality). AR means to augment reality. In the movie "Terminator", the scene where the Terminator exports the data on the object into view and displays it in a superimposed manner, or more recently "Pokemon GO", where Pokemon can be seen through the smartphone camera in specific locations.
　"Augmented TV" is an attempt to adopt such AR technology as a directive expression of television. For example, it is possible in the future that a TV character may jump out of a television screen.

- "Pop-Out TV" and "Metafiction"
Hiroyuki Kawakita of the Science & Technology Research Laboratories, which promotes research and development of this technology, talks about the mechanisms of Augmented TV.
　"In a word, it's 'pop-out TV'. I think that it expresses the true feelings of the on-screen characters better than before. Although there have been quite a lot of scenes featuring people popping out of the TV, the main example being the scene where Sadako from the movie "Ju-On" jumps out of the TV. The shocking part comes from the fact that you're watching from a distance, but suddenly it's close to you"
　This is similar to an actor who suddenly calls out the audience, or an author who appears in the novel to talk to the characters. This is called "metafiction", a production technique, but Augmented TV is meant to realize it using AR technology.

　As a concrete mechanism, a smartphone or tablet calculates the position and direction of the TV screen and synthesizes the CG to display television through the camera. However, in order to "jump out", movement must be synchronized with the broadcasted content. For example, when a character makes a movement that "jumps out", if the synthesized CG display is too early or too late, the feeling of "jumping out" is ruined. This is what is known as a "miss".

■ Timing adjustment using the system clock
　Kawakita explains the synchronization method as follows:
　"Both the TV and the smartphone use the 'system clock' for the timing. Since the timing uses the CPU, the heart of the computer, it is not affected by applications. The question of "how many seconds until it's time to jump out" is set at the beginning of the program based on the system clock, making it possible to match the timing perfectly"
　This synchronization technique is an NHK Science & Technology Research Laboratories original. A "dynamic, two-dimensional barcode" is displayed at the beginning of the program, which is used to match with the smartphone. By using the system clock, accurate timing is possible by synchronizing at the beginning. Furthermore, by setting up a frame in the TV, the smartphone can recognize the position of the television and use its built-in gyrosensor to derive the position needed to synthesize the CG.

- Wanting to connect BtoB
　This technology was already introduced at the "NHK Research Open", held in June of this year, but now Kawakita talks about the aim and the exhibition demo at InterBEE:
　"The NHK Research Open is meant to broadcast NHK's technology research to a wider audience, but InterBEE needs a presentation assuming BtoB deployment. When I heard about the project "INTER BEE IGNITION", I thought it was really good since it used AR technology"
　"At our booth, we demonstrated Augmented TV with a portrait display, which is supposed to be digital signage. We played an advertisement for loop content that lasts several minutes, and by holding a smartphone in front of it, you could download an application that let you play in the next cycle. Digital signage goes well AR applications as well as TV, I feel. It features patented technology, broadcasting, and other basics. As a way to make the best use of technology, I think broadening B2B partnerships is the way to go"

[NHK Science & Technology Research Laboratories]
Head Office Address
1-10-11, Kinuta, Setagaya-ku, Tokyo, 157-8510
URL
http://www.nhk.or.jp/strl/index.html