Video content commentator

The idea of this demonstrator is to learn to provide annotations of film or television material that would be suitable for someone with visual deficiencies. A visually impaired user can identify people in a video based on their voices. However, impaired users currently miss a number of important visual aspects: the people who are present but not speaking, the positions of the characters, the setting (open landscape, indoors, street scene,...), their interactions with one another and with scene objects (e.g. getting into a car).

We thus investigated methods for automatically augmenting film or television video with textual annotations, taking as an example the TV series Buffy the Vampire Slayer. We developed methods for several different kinds of annotations:

• Character identification (University of Oxford) - for more details see the Oxford character identification page.
  
  
• Action detection (INRIA Grenoble).
  
  
• Location recognition (KU Leuven).
  
  
• Emotion recognition (KU Leuven).
  
  

Videos

Examples of the output can be seen in two automatically annotated demonstration videos:

• Demo of emotion recognition
• Integrated demonstration of the other annotations

---

Contact: Bill Triggs (coordinator), Bill.Triggs@imag.fr, phone +33 4 7651 4553
Laboratoire Jean Kuntzmann, 51 rue des Mathematiques, 38402 Saint Martin d'Heres, Grenoble, France