Department for Automation, Biocybernetics and Robotics

PACO+ (Perception, Action, and Cognition through Learning of Object Action Complexes)

Project duration: 2006 - 2010
Project area: Laboratory of Humanoid and Cognitive Robotics
Project type: Research
Project funding: EU
Project leader: Aleš Ude
Andrej Gams
Andrej Kos
Bojan Nemec
Damir Omrčen

External collaborators:

  • University of Karlsruhe, Germany (Prof. R. Dillmann, Dr. T. Asfour)
  • Royal Institute of Technology (KTH), Stockholm, Sweden, (Prof. J. O. Eklundh, Dr. D. Kragic)
  • Bernstein Center for Computational Neuroscience, University of Göttingen, Germany (Prof. F. Wörgötter)
  • Aalborg University, Copenhagen, Denmark (Prof. V. Krüger)
  • Consejo Superior de Investigaciones Científicas (CSIC), Barcelona (Prof. C. Torras, Dr. J. Andrade‐Cetto)
  • Leiden University,Netherlands (B. Hommel)
  • University of Edinburgh, United Kingdom (Prof. M. Steedman)
  • University of Southern Denmark, Odense (Prof. N. Krüger)
  • University of Liège, Belgium (Prof. J. Piater)


PACO‐PLUS brings together an interdisciplinary research team to design and build cognitive robots capable of developing perceptual, behavioural and cognitive categories that can be used, communicated and shared with other humans and artificial agents. To demonstrate our approach we are building robot systems that will display increasingly advanced cognitive capabilities over the course of the programme.

Project description


Our approach rests on three foundational assumptions:

  • Objects and Actions are inseparably intertwined in cognitive processing; that is "Object‐Action Complexes" (OACs) are the building blocks of cognition.
  • Cognition is based on reflective learning, contextualizing and then reinterpreting OACs to learn more abstract OACs, through a grounded sensing and action cycle.
  • The core measure of effectiveness for all learned cognitive structures is: Do they increase situation reproducibility and/or reduce situational uncertainty in ways that allow the agent to achieve its goals?


Therefore PACO‐PLUS has the following main objectives:

  1. Define invariant, hierarchical, multi‐sensory (vision, haptic, and proprioceptive) representations of objects through actions performed on them and define the processes by which these representations can be derived, refined and learned.
  2. Define a representation and sensorimotor learning mechanisms for Object‐Action‐Complexes (OACs) that fundamentally address the inherent uncertainty of real world domains and provides for action recognition, event anticipation, and action understanding.
  3. Define methodologies for learning new OACs from existing OACs generalizing along axis given by motor capabilities, Gestalt statistics, guided and chance exploration, and goal directed search. Robotic experiments will be paired with psychological experiments.
  4. Demonstrate the machine learning of appropriate grammar fragments for European and non‐ European languages including English from exposure to strings paired with plan and action representations induced automatically for non‐linguistic purposeful action in the world.
  5. Define a theoretical framework for a conjoint measure for learning (and action) success to evaluate OACs and associated learned categories. Devise methods which allow changing strategies for learning using return maximization and/or contingency minimization as required for achieving differing goals.
  6. Design and implement a planning system capable of leveraging OACs, making critical use of the capability to learn new OACs and leverage the uncertainties associated with execution in a principled framework.
  7. Design and implement a complete cognitive system with oculomotor behaviours and sensorimotor primitives to enable learning, imitating and understanding actions of humans as well as exploration of unknown objects. The system should be able to acquire cognitive capabilities through continuous interaction and exploration.
  8. Implement all of these components on the PACO‐PLUS humanoid robotic platform and integrate them into a single cognitive architecture informed by neurophysiological and psychological findings on the mechanisms and knowledge structures that drive natural cognition as well as theory and practice from research in robotics, vision, machine learning, artificial intelligence, linguistics, and cognitive science.