CLASS - Cognitive-Level Annotation using Latent Statistical Structure

CLASS was a 3.5 year targeted research project that ran from January 2006 to June 2009. It was funded by the European Union Information Society Technologies unit E5 – Cognition.

Scientific Goals

The main goal of CLASS was to develop a basic cognitive ability for use in intelligent content analysis: the automatic discovery of content categories and attributes from unstructured content streams. The demonstrators focused on object recognition and scene analysis in images and video with accompanying text streams. Autonomous learning made recognition more adaptive and allowed more general classes and much larger and more varied data sets to be handled.

Technically, the work combined latent structure models and semi-supervised learning methods from machine learning with advanced visual descriptors from computer vision and state-of-the-art text analysis techniques. Three levels of abstraction were studied: new individuals (specific people, objects, scenes, actions); new object classes and attributes; and hierarchical categories and relations between entities.

The basic research results were illustrated by three proof-of-concept demonstrators: an Image Interrogator that interactively answers simple user-defined queries about image content; a Video Commentator that automatically creates textual descriptions of the action and content of situation comedy videos for visually impaired users; and a News Digester that combines television news stories with captions from several sources to create a textual and visual digest of them. New functionality was added to each demonstrator every year, so there were three versions of each demonstrator.

Consortium

CLASS was an interdisciplinary project, combining eight leading European research teams in visual recognition, text understanding & summarization, and machine learning:

• The LEAR team, INRIA, Grenoble, France (visual recognition). CLASS demo page at INRIA.
• The Visual Geometry Group, Department of Engineering Science, University of Oxford (image & video analysis). CLASS pages at Oxford.
• The VISICS team, Katholieke Universiteit Leuven (video summarization). Local CLASS pages at VISICS.
• the ICRI-LIIR team, Katholieke Universiteit Leuven (text summarization). Local CLASS page at ICRI-LIIR.
• the Empirical Inference for Machine Learning and Perception department of the Max-Planck Institute for Biological Cybernetics, Tübingen (semi-supervised learning). CLASS pages at MPI.
• (Until 1 June 2006) the Intelligent Systems Group, Computer Science Department, Technical University of Darmstadt (latent structure models).
• (From 1 June 2006) the Complex Systems Computation Group, Department of Computer Science, University of Helsinki (latent structure models).
• (From 1 January 2007) The AI team, CNRS and Laboratoire Jean Kuntzmann, Grenoble, France (visual recognition and machine learning).

Demonstrators

The publicly available versions of the CLASS demonstrators can be found on the demonstrator page.

Publications

CLASS produced a large number of scientific publications. Most of these are available for download from the publications page.

A feature on CLASS research appeared in the June 2009 the ICT Results service, and Vittorio Ferrari gave a talk covering some of our work on visual attributes at the ICT CogSys 2010 workshop.

The proceedings of a workshop associated with CLASS - the IJCAI-09 Workshop on Cross-Media Information Access and Mining (CIAM 2009) are also available.

Tutorials and Talks

A few of the tutorials and talks produced during CLASS are available on the tutorials page.

Datasets and Software

Some of the datasets and software associated with CLASS are available on the datasets page.

Statistics

• CLASS (IST project 027978) was a 3.5 year 6^th Framework Specific Targeted Research Project running from 1 January 2006 to 30 June 2009
• Total effort: 38 person years
• Total budget: 2.9 M€ including 2.4 M€ of European Union support

---

Home page: http://class.inrialpes.fr. CLASS flyer in PDF and MSWord formats.

Contact: Bill Triggs (coordinator), Bill.Triggs@imag.fr, phone +33 4 7651 4553
Laboratoire Jean Kuntzmann, 51 rue des Mathematiques, 38402 Saint Martin d'Heres, Grenoble, France