A Research Agenda for the Visual Augmentation of the Television Watching Experience

Today's smart TVs feature a variety of capabilities, such as high, 8K resolution displays, wide field of view, curved form factors that favor immersion and the perception of depth, and connectivity to the Internet and to other devices, such as smartphones and tablets. A few industry initiatives have recently started to connect Augmented Reality (AR) technology with television, such as Augmen.TV, while research projects such as IllumiRoom, RoomAlive, and Around-TV demonstrated the rich opportunities of using AR for video games and television that expand to an entire wall or room. However, very few prototypes, studies, and experimental data exist for Augmented Reality TV and, overall, there has been little exploration to inform the design of visual augmentation of the television watching experience for viewers.



To foster research and development in this direction, this web page outlines a proposal for a research agenda for the visual augmentation of television watching that is based on (1) Augmented/Mixed Reality and (2) smart wearables, such as smartglasses and head-mounted displays. While this agenda is work-in-progress, we release it via this dedicated web page hoping that it will be inspiring to the community and, consequently, foster critical and constructive discussions towards new application opportunities, devices, techniques, and tools to augment visually the television watching experience.

Agenda


The agenda currently consists in 9 items that cover users, devices, smart environments, content and ambient media, interaction techniques, viewers with visual impairments (low vision), tools, applications, as well as connections to other fields of research:
  1. Understanding user preferences
    One important aspect to inform design and development of interaction techniques, user interfaces, I/O devices, and applications is to understand what users want and need. Several research questions can be envisaged regarding the visual augmentation of the television watching experience: What are users' preferences for the visual augmentation of TV watching in terms of devices, digital content, and applications? Are there specific types of content or television genres for which visual augmentation is more desirable? Do various age groups, e.g., children, young adults, and elderly, have different preferences regarding the visual augmentation of their television watching experience?
  2. Devices to support visual augmentation for television
    Visual augmentation can be performed using a variety of AR devices, from smartphones and tablets to wearables, from smartglasses, such as Vuzix Blade to head-mounted displays, such as Magic Leap One, and to the holographic computer HoloLens. According to Azuma (2016), wide field-of-view, optical see-through near-eye displays in compact form factors, as well as interfaces to control such displays represent two out of the four key challenges to be overcome for viable AR platforms and applications to happen.
  3. From smart TVs to smart TV environments
    Beyond smart TVs, the entire room can serve as a canvas to display computer-generated graphics to augment television watching. From this perspective, mobile devices, wearables, the smart TV, and sensors and display technology from the physical environment constitute a "smart TV space," for which flexible software architecture is needed to connect heterogeneous I/O devices.
  4. Accelerating research from multimedia to smart, augmented, ambient media
    Delivering viewers with a visually-augmented TV experience will likely need new ways to represent and transfer digital content, accelerating developments towards ambient media and implementations of its principles, i.e., manifestation, morphing, intelligence, experience, and collaboration; see Lugmayr et al. (2009) for an introduction to semantic ambient media.
  5. Interaction techniques to operate visual augmentation
    The TV remote control is unlikely to scale effectively for viewers to operate AR and ambient media content that is 3-D, overlaying the physical environment, and interactive in real-time. Alternate input modalities, such as free-hand and whole-body gestures, wearables, on-body input, voice, and eye gaze input will need to be explored and evaluated in this context.
  6. Visual augmentation of television for viewers with visual impairments
    According to a 2018 survey from Comcast and the American Foundation for the Blind, adults with visual impairments tune-in to television as much as people without impairments. Visual augmentation is especially important for people with low vision (i.e., visual impairments that cannot be corrected with contact lenses, medication, or surgery) who, unlike blind people, do rely on their visual abilities for everyday activities, but experience challenges caused by vision disturbances.
  7. Tools to support visual augmentation for television
    Tools are extremely important to support development of practical applications, assist designers, and stimulate practitioner's creativity. Example of tools that we believe are very relevant for visual augmented TV regard the creation of digital content, prototyping user interfaces for Augmented Reality TV using mobile and wearable devices, as well as tools to assist the analysis of user performance with Augmented Reality television.
  8. New applications
    Examples of new applications for visually augmented TV include collaborative television watching with remote audiences, multi-platform TV, immersive TV shows, or on-demand content consumption. Also, new forms of television are likely to emerge supported by Augmented Reality technology.
  9. Connections to other fields of research
    Important knowledge needs to be brought in from other fields, such as Psychology (understanding users), Wearable Computing (new I/O devices), Human-Computer Interaction (UI design, evaluation methodologies), Augmented Reality (rendering techniques and tools for overlaying computer-generated content onto the physical world). Practical application of such knowledge in the context of TVX can lead to new, exciting experiences for viewers.

Community Feedback

Intended to start up a community effort, we hope that our initiative will stimulate exciting research and development at the boundaries of Augmented Reality, Wearable Computing, and Human Vision Augmentation with focus on the television watching experience. Our plan is to update this agenda as discussion develops in the interested community and in the measure that we receive proposals and suggestions for improvement. Therefore, we invite TVX researchers and practitioners to get involved, react, and contribute with comments, suggestions, and new ideas for the benefit of everyone.
Of course, you can decide to remain anonymous, if you wish, but we would be very happy to credit your contribution by listing your name on this web page.

Contributors

We are very happy to acknowledge your name on this web page as contributor to this research agenda. Please share your comments using the above form, get involved, react, discuss, and share this web page with people that you think might be interested. We thank everyone for their involvement. Together as a community, we can shape urgent, innovative, and exciting directions of research and development in Augmented Reality TV for the years to come!

Contact

For any comment or suggestion about this initiative or web page, please contact Prof. Radu-Daniel Vatavu.
The icons used on this page were adapted from original versions by Freepik from www.flaticon.com licensed under CC 3.0 BY.