LOCEN IN BRIEF
LOCEN history and overview.
LOCEN, a research group of ISTC-CNR, was founded in 2006 by Gianluca Baldassarre. Before its formation, various members of LOCEN worked in the field of Artificial Life (in particular with Domenico Parisi and Stefano Nolfi): hence the importance given to evolution, adaptation, the function of brain and cognition, embodiment/robots, the interaction of agents with the environment and internal body, and complex system. LOCEN has specialised its research in two directions: (a) Autonomous and Developmental Robotics: here the objective is to create autonomous robots capable of open-ended learning based on intrinsic motivations and goals; (b) Computational Embodied Neuroscience models: here the objective is to understand how the brain acquires and produces behaviour, and also how it can be affected by impairments and diseases. LOCEN is currently formed by 6 Researchers, 4 Research Fellows, and various changing master students and visitors.
LOCEN RESEARCH TOPICS
Robots and animals: autonomous learning and development
Robots and animals: motor control, actions
Robots and animals: extrinsic and intrinsic motivations
Robots and animals: goal-based learning
Robots and animals: hierarchical reinforcement learning
Robots and animals: attention and active vision
Animals and robots: habitual and goal-directed behaviour and learning
Animals and robots: classical and instrumental conditioning
Animals: brain structures underlying the above processes (amygdala, hippocampus, basal ganglia, cerebellum, associative cortex, prefrontal cortex, motor cortex, dopamine).
Humans: Parkinson's Disease, Alzheimer's Disease, Autism Spectrum Disorder, Post-Traumatic Stress Disorder, Substace and Behavioural Dependencies
LOCEN RESEARCH METHOD
LOCEN investigates cumulative open-ended learning using two methodological approaches: (a) Developmental and Autonomous Robotics, focussed on building useful robots; (b) Computational Embodied Neuroscience, focussed on understanding how brain learns and produces behaviour in animals.
The group uses these approaches to investigate two distinct but related sets of topics and problems.
The two methods and the two sets of topics are now illustrated in detail.
LOCEN: Advanced School in AI
LOCEN organises and runs an Advanced School in AI. The School is run yearly at ISTC (Via San Martino della Battaglia 44, Rome), starts in October, involves 4 months of courses (two 8-hour days per week) and a project of the student focussed on either a research problem or an application targeting a need of a company (the project starts with the courses and lasts for 8 months). The School is highly interdisciplinary and involves courses and topics on fundamentals of mathematics and programming, machine learning (with a particular focus on deep neural networks), computational modelling of brain and behaviours, computational social science and AI ethics/law. More detail information can be found here:
www.as-ai.org .
LOCEN AUTONOMOUS ROBOTICS
LOCEN autonomous robotics: method
The research agenda and the research bio-inspired approach of the group
The group has a research agenda that aims to build autonomous cumulative learning robots. This are robots that are capable of autonomously acquiring a number of skills in autonomous fashion, without human intervention. Although the group fully recognise the great importance of social mechanisms for the acquisition of complex behaviours like those acquired by primates (e.g., social enhancement, joint attention, imitation, language, teaching, etc.), the group focusses its research on individual learning processes so to have a higher impact in this study. In this respect, the ``holy grail'' of LOCEN is to arrive to build robots that, like children, are capable of acquiring an increasingly sophisticated repertoire of sensorimotor and cognitive skills, from simple to complex ones, in a cumulative fashion, without external intervention. For this reason, we follow the developmental robotics approach to build its robots.
In this respect, the group believes that fundamental breakthroughs in specific fields of robotics and machine learning will come from the study of biological systems. The reason is that organisms are the result of a run of the ``gigantic'' genetic algorithm represented by natural evolution: such algorithm has found solutions to some problems that engineering approaches will not beat for long. The idea is so to reverse-engineer brain and behaviour of real animals to propose radically new ideas to autonomous robotics and machine learning, that might synergies with those discovered with more traditional approaches.
LOCEN autonomous robotics: topics
The topics investigated by the group are all directed to support its ``mission'' of building autonomous, cumulative learning robots. We believe that four critical challenges need to be overcome to accomplish this goal:
Dynamic behaviour, compliant hardware and muscles
All that ultimately counts, for both robots and animals, is action. Action means a change of the sensorimotor body in space that in turn can change the environment (e.g., to manipulate objects) and the body-environment relation (e.g., to look at something, or to change position in space). Behaviour emerges from the dynamic interplay between the robot's action, that changes the body-environment relation, and the close-loop physical and perceptive feedback flow that results from this. Part of the extraordinary capabilities that animals display in these interactions rely upon the hardware of their body, in particular their compliant skeleto-muscular system. For this reason, one topic of interest of the group is the challengs and opportunities offered by robots endowed with a biomorphic compliant hardware (e.g., bio-materials, biologically-plausible structure, pneumatic actuators).
Active vision, proprioception and abstraction
The sensorimotor flow brings to robots and organisms a huge amount of information. The group focuses on proprioception, touch, and vision. Proprioception is used to perform dynamic motor behaviour. Vision is instead used to guide behaviour at a higher level, e.g. to visually guide reaching and grasping, or to decide with which object to interact. We see perception not as a passive source of information, but rather as an active process of search of relevant information and avoidance of non-useful information given the goal pursued by the robot/organism. At both the perceptive and motor level, behaviour needs abstractions. For example, when an object is seen it is useful to abstract a number of distinct information elements used by different downstream processes of the controller, e.g. its location, size, identity, etc. Active vision and abstraction processes are so an important investigation topic of the group. ouch are used as a fundamental element to implement sensorimotor skills at the low level.
Extrinsic and intrinsic motivations, and learning
As the group is interested in autonomous cumulative learning, he is profoundly interested on the mechanisms that can guide learning in an open-ended fashion without external intervention. Motivations are a key element for this as they can: (a) drive the robots to perform different behaviours in different conditions; (b) generate the learning signals needed to guide their learning. Two types of motivations are studied by the group. The first are extrinsic motivations, i.e. motivations related to the task assigned to the robot (what is the best reward function function?). The second are intrinsic motivations. These are a rather novel concept in autonomous robotics and the group is keenly investigating them. Intrinsic motivations drive behaviour and generate learning signals on the basis of the fact that the controller is indeed acquiring knowledge with a high rate, e.g. is forming good abstractions, is learning to predict well, or is acquiring the sensorimotor competence to accomplish a goal. Intrinsic motivations are fundamental for autonomous robots as allows them to acquired knowledge and skills in the absence of tasks from the user, so that when some tasks are assigned to them they can exploite the knowledge acquired autonomously to readily solve them. The pivot of all these processes is of course learning, so the group keenly investigates all forms of learning useful for robots, from associative to unsupervised learning, from supervised to reinforcement learning (the latter is central in several models of the group).
A video showing a humanoid robot (iCub) that explores a `mechatronic board' under the drive of intrinsic motivations, and autonomously discovers and learns that pressing some buttons turns on some lights. In a later stage, the robot exploits the acquired action-outcome contingencies to accomplish useful purposes. For more detailed videos see here and here: these videos are explained in the web-pages here and here; the system is explained in detail in the article here. Work carried out within the project IM-CLeVeR.
Hierarchical architectures
The cumulative acquisition of skills and cognitive capabilities requires that the newly acquired knowledge: (a) is stored without destroying previously acquired knowledge; (b) supports the acquisition of further knowledge. The first goal is achieved through hierarchical architectures that avoid that the acquired different pieces of knowledge interfere with each other. The second goal is accomplished by investigating how the acquired knowledge can be transferred to new tasks to be solved (e.g., as in transfer reinforcement learning). Because of these two goals, the group is keenly interested in developing hierarchical reinforcement learning architectures.
Video of a humanoid robot (iCub) learning to throw a ball to a target based on a hierarchical reinforcement learning system with sophisticated generalisation capabilities (generation of new dynamic movement primitives on the fly based on the similarity of the new goal with respect to previously acquired goals). This work was carried out in collaboration with Bruno Castro da Silva and Andrew Barto, from the University of Amherst Massachusetts.
LOCEN BRAIN AND BEHAVIOUR
LOCEN brain and behaviour: method
Computational Embodied Neuroscience
The group has developed an original research method to study brain and behaviour, rooted on system-level computational neuroscience and artificial life, named Computational Embodied Neuroscience (CEN).
Differently from other computational neuroscience approaches, CEN aims to understand the brain with a ``top-down approach'' starting from behaviour and function. The key idea behind this is that the brain evolved to allow animals to act so as to improve their survival and reproductive chances. So to fully understand how brain works, we need to understand not only its mechanisms (anatomy and physiology, the common focus of neuroscience) but also its functions, i.e. ``what it is for''. This is why the group is keenly interested in linking the knowledge produced not only by neuroscience but also by psychology and psychobiology.
From system-level to detailed models (multi-scale models)
Another consequence of the goal of CEN of studying whole systems capable of acting is the tendency to build system-level models, reproducing the macro-architecture of various cortical and sub-cortical brain areas that underlie the target behaviours. This is in fact needed if one wants to understand how a certain area of brain works not studying it in isolation, but how its inner mechanisms play a certain function within a whole system. Of course, often the aspect of the system which are not under focus are represented in an abstract fashion, but they are nevertheless there. After sedimenting knowledge at the system-level, the group usually starts to refine the micro-architecture and functioning of the various components of the model (usually focussing on a subset of them). This leads to have multi-scale models that encompass not only the system-level but also the meso-level (e.g., different nuclei of basal ganglia) and micro-level (e.g., the canonical architecture of cortex) of brain. For these overall purposes, our models are usually based on firing rate neurons and leaky neurons, and only recently on spiking neurons (integrate-and-fire neurons). Recently, the group has started to build probabilistic graphical models whose functioning is implmemented on the basis of Bayesian inferences. These models have the advantage of facilitating system-level modelling at a high (often functional) level before moving to more detailed neural models, and also of offering the possibility to directly evaluate the goodness of models against empirical data, for example from brain imaging. We are also interested in probabilistic interpretations of brain processing (from which our recent interest for spiking neurons).
Aiming to build cumulative models
The hallmark of science is cumulativity. Too often different computational models are build to interpret different experiments. Instead, CEN aims to build models that allow the explanation of an increasing number of specific brain and behavioural data. This allows the isolation of general principles, the integrated theoretical systematisations of whole classes of phenomena, and so to help psychology and neuroscience to overcome the polverisation of results and views that they often encounter for their need of focussing. Integrated theories and models allow the production of detailed hypotheses that fill in the knowledge gaps of psychology and neuroscience and produce specific empirical predictions that can be tested in new empirical experiments.
The importance of a close dialogue between models and empirical data
Differently from other computational approaches to the study of brain and behaviour, CEN stresses the importance of having a tight, continuous dialogue with specific empirical data from psychology and neuroscience. The idea is that the understanding of brain and behaviour should proceed as any good science, namely it should rest on two pillars: (a) the theoretical understanding of the investigated phenomena, based on formal computational models; (b) the empirical investigation of such phenomena to select the best theories, models, and predictions. In this respect, we see computational modelling as a powerful theoretical means that should guide empirical research on brain and behaviour. The ultimate proof that computational modelling of brain and behaviour has successfully accomplished its mission is its capacity to change the daily research of the empirical neuroscientist and psychologist, and to publish papers in top journals of neuroscience and psychology.
The importance of embodiment: sensorimotor and visceral
We believe that brain generates behaviour by dynamically interacting with the environment through sensors and actuators in a circular fashion (embodiment). Sensors furnish a rich, redundant and noisy amount of information to organisms. Actuators are in turn noisy, redundant, compliant. Before facing high-level cognition problems, brain has to solve the problems posed by such input/output information channels (and also exploit the opportunities they offer) . The resulting computations might be radically different from those that would stem from, say, a clean, symbolic type of input/output information. For these reasons, we think that good models of brain and behaviour should function in simulated or real robotic systems that have the same sensors and actuators as the investigated animals. This poses strong challenges to models, especially because neuroscientists and psychologists often overlook them and require us to focus ``on their problems'', concerning higher-level aspects of cognition. For this latter reason, the more biologically constrained models produced by the group so far often use localistic representations, abstract input/output information codes, simplified environments. However, the group is fully aware of the importance of scaling up models to more realistic input/output and environmental conditions, so we try our best to incorporate in the models critical elements of a true embodiement, e.g. the sensorimotor loop and the test with simulated/real robots. Aside this, the group gives also a lot of important to a second type of ``embodiment'', most of the times neglected but as important as ``sensorimotor embodiment''. This might be called ``visceral embodiment'' and refers to the key relation that the brain has with the visceral body and its homeostatic regulations. These regulations are at the basis of extrinsic motivations and the subjective value (biologically saliency) that organisms assign to objects and experiences.
The importance of learning
We believe that for a large part the brain structure is as it is because it has not only to express behaviour but also to learn it. For this reason, most of our models aim not only to reproduce target behaviours, and the neural machinery to do so, but also the learning processes that lead to its acquisition with experience, and hence the physiological processes underlying this. For this reason we are keenly interested in studying all forms of biologically plausible learning: Hebbian learning, differential Hebbian learning, competitive self-organised learning, and reinforcement learning, and goal-based learning.
LOCEN brain and behaviour: topics
Given the interest of the group in understanding how brain and behaviour supports cumulative learning in organisms, our research is focussed on the following topics:
Vision and active vision (superior colliculus, brain dorsal and ventral pathways, basal ganglia, cortex)
The bio-contrained models developed by the group focus on two types of perception: proprioception and vision. Often we use proprioception in our models to guide low-level behaviour, but we do not study it per se. We instead are very interesting in studying vision, and especially active vision. The reason is that vision is the primary information source for primates and has a paramount role in guiding action (via the brain dorsal pathway, involving parietal and premotor cortex) and to support higher-level cognition, such as decision making and planning (via the ventral pathway, involving inferotemporal cortex and prefrontal cortex). We study vision not as a passive source of information but rather as an active one. In particular, we are interested in studying how overt attention, and the high resolution of the fovea with respect to peripheral vision, can actively search and gain information in the environment, and ignore irrelevant one, on the basis of the animal's goals. We are thus interested in bottom-up attentional processes, that drive eye gaze on most informative parts of the environment, and in top-down attentional processes, that drive the eye to collect information based on goals; and we are of course interested in their rich interplay. We also believe that attention is pivotal for the rest of behaviour and indeed there is a strong coupling between vision and arm/hand manipulation actions, with attention representing a powerful guidance for controlling such actions via the selection of suitable inputs (``eyes lead, arms execute'').
Extrinsic motivations, Pavlovian and instrumental learning (amygdala, nucleus accumbens, dopamine)
Behaviour needs to be driven. Learning needs to be guided. Extrinsic motivations are expressed by parts of brain (amygdala, nucleus accumbens, dopaminergic and other neuromoduatory systems) at the interface between visceral body and cognitive processes.
We are interested in studying how extrinsic motivations drive behaviour, directing it to specific activities, or how they generate learning signals that guide learning processes. In this respect, we are very interested in investigating how areas such as the amygdala can perform Pavlovian associations that allow triggering important internal reactions (e.g., for the internal regulation of visceral body and the neuromodulation of brain) and external behavioural reactions (feeding, approaching, orienting) in correspondence to biologically salient stimuli (e.g., food, water, sex) or stimuli anticipating them (conditioned stimuli). Also, we are very interested in understanding instrumental behaviour, i.e. the processes that allow organisms to learn to trigger (learned, instrumental) behaviours, when particular conditions are present, if this leads to rewards (S-R behaviour).
Intrinsic motivations (superior colliculus, hippocampus, dopamine, noradrenaline)
Aside extrinsic motivations, we are interested in studying intrinsic motivations, i.e. the motivations at the core of the performance and acquisition of actions ``for their own sake'', i.e. not for the achievement of results that directly increases biological fitness (e.g., food or money). Intrinsic motivation systems have evolved as they drive exploration and learning in the absence of extrinsic outcomes, and have the function of leading animals to acquire knowledge and skills that will be readily usable in the future when such extrinsic outcomes become available.
For example, intrinsic motivations are maximally apparent in children at play: in the absence of homoeostatic drives, children engage in ludic behaviours driven by curiosity, novelty, surprise, and changes in the environment, in general all experiences that cause an improvement of their knowledge and skills. Neuroscience is unrevealing some of the brain mechanisms behind these processes, e.g. the capacity of hippocampus to cause dopamine production (learning signals) when a novel object is perceived, or the capacity of superior colliculus to produce phasic dopamine when the world change, or the capacity of frontal cortex to cause noradrenaline production when predictions are violated.
Hierachical sensorimotor brain, habitual behaviour (hierarchical cortex, multiple basal ganglia loops, cerebellum)
Organisms' cumulative learning of sensorimotor behaviour and higher-level cognition requires a hierarchical soft-modular brain architecture that links motor behaviour to perception at different levels of abstractions and coupling. This architecture is organised in at least three levels, investigated by the group with system-level models. (1) At the lowest level, the close-loop between somatosensory cortex and primary motor cortex (forming a loop with sensorimotor basal ganglia involving putamen/globus pallidum) and involving cerebelllum, implements the dynamic production of movements (e.g., a reach, a grasp); (2) the dorsal brain pathway, encompassing visual cortex, parietal cortex (encoding affordances), and premotor cortex (encoding repertoires of actions) (which forms a loop with sensorimotor basal ganglia involving putamen-caudatum/pallidum-subtantia nigra reticulata), implement the on-line control of action (e.g., to guide a reach to an object, or to shape the hand to grasp it); (3) the ventral brain pathway, encompassing the visual cortex, the inferotemporal cortex (for object recognition), and the prefrontal cortex (for working memory and multimodal sensory integration) (which forms a loop with associative basal ganglia involving caudatum/pallidum), implements high-level decision making and executive control of behaviour; (4) the cortex communicating with sub-cortical limbic structures, such as amygdala and hippocampus (which forms a loop with limbic basal ganglia involving nucleus accumbens), processes value and encodes biologically salient action-outcomes and goals (e.g., food). The group studies this complex hierarchical system by usually focussing on different parts of it but always keeping in mind the overall architecture.
Goals, goal-directed behaviour, decision making (amygdala, hippocampus, nucleus accumbens, prefrontal cortex)
Starting form sensorimotor behaviour, the group is now gaining knowledge and skills for the investigation of higher-level cognition, in particular in relation to goal-directed behaviour and decision making (levels 3 and 4 of the framework of the previous point). ``Goals'' are now becoming a critical concept for the group for the pivotal role they play in cognition and, in particular, in autonomous cumulative learning. A goal is an internal representation of a future world state that is activated internally and drives action (and learning) for its accomplishment. The current investigation of ``goal-directed behaviour'' and decision making, strongly supported by theory-driven model-free and models-based reinforcement learning, is one of the hottest fields of investigation of cognitive science. Goals also play a key role in autonomous cumulative learning: a critical function that intrinsic motivations play is the self-generation of goals. If you look with attention a little child at play, you will realise how her/his autonomous exploration and learning is strongly guided by an incessant autonomous setting, pursuit, accomplishment, monitoring, and switch of goals. Goal-based processes, greatly enhanced by the uniquely-developed human prefrontal cortex, focus attention, inform on success, drive learning, and organise action in ways that render autonomous learning ``explosive'', i.e. powerful, omni-directional and open-ended.
LOCEN INTERACTIVE DEVICES
Since 2015 the group started collaborating with some rehabilitation centers to realise interactive devices for therapeutic use. This collaboration led to the implementation of "+me", a working experimental prototype addressed to the therapy of Autism Spectrum Disorders. The device is currently in clinical trial. More info on
www.plusme.it
CONCLUDING REMARKS
The presence in the same group of different perspectives and methods (robotics, modelling, neuroscience and psychology) on the same topics give the group a number of advantages:
An approach to problems of autonomous robotics and cognitive-science that is profoundly interdisciplinary
The capacity to coordinate and play a key role in large robotics/cognitive-science projects involving different approaches, themes, and teams
An ``eagle-eye view'' on robotics and cognitive-science issues that allows the group to see phenomena, problems, and solutions overlooked by other more focussed, but possible near-sighted, views and approaches.
