Predictive Coding for Dynamic Vision: Development of Functional Hierarchy in a Multiple Spatio-Temporal Scales RNN Model

Authors: Minkyu Choi and Jun Tani

The current paper presents a novel recurrent neural network model, the predictive multiple spatio-temporal scales RNN (P-MSTRNN), which can generate as well as recognize dynamic visual patterns in the predictive coding framework. The model is characterized by multiple spatio-temporal scales imposed on neural unit dynamics through which an adequate spatio-temporal hierarchy develops via learning from exemplars. The model was evaluated by conducting an experiment of learning a set of whole body human movement patterns which was generated by following a hierarchically defined movement syntax. The analysis of the trained model clarifies what types of spatio-temporal hierarchy develop in dynamic neural activity as well as how robust generation and recognition of movement patterns can be achieved by using the error minimization principle. 
(Choi, M., & Tani, J. (2017). Predictive Coding for Dynamic Visual Processing: Development of Functional Hierarchy in a Multiple Spatio-Temporal Scales RNN Model. accepted in Neural Computation.)
 [PDF]     [VIDEO]   

Achieving “Synergy” in Cognitive Behavior of Humanoids via Deep Learning of Dynamic Visuo-Motor-Attentional Coordination

Authors: Jungsik Hwang, Minju Jung, Naveen Madapana, Jinhyung Kim, Minkyu Choi and Jun Tani

The current study examines how adequate coordination among different cognitive processes including visual recognition, attention switching, action preparation and generation can be developed via learning of robots by introducing a novel model, the Visuo-Motor Deep Dynamic Neural Network (VMDNN). The proposed model is built on coupling of a dynamic vision network, a motor generation network, and a higher level network allocated on top of these two. The simulation experiments using the iCub simulator were conducted for cognitive tasks including visual object manipulation responding to human gestures. The results showed that “synergetic” coordination can be developed via iterative learning through the whole network when spatio-temporal hierarchy and temporal one can be self-organized in the visual pathway and in the motor pathway, respectively, such that the higher level can manipulate them with abstraction. 
[VIDEO]    [PDF]   

Neuro-Robotics toward general Intelligence

This project aims to understand the essential brain mechanisms for the higher-order cognition by synthesizing through robotics experiments. We build so-called the large scale brain network (LSBN) model by utilizing available data for connectivity matrix among local regions in the human brain under collaboration with Prof. Daeshik Kim in KAIST. A challenge is to show that humanoid robots can adapt to a wide range of cognitive tasks by flexibly combining various cognitive resources developed in the LSBN. Here, the aim is not just to train robots to be good at a particular cognitive task, but to educate them to be good at various cognitive tasks simultaneously where general intelligence would be required. The project may start with focusing on the main cortical areas including the prefrontal cortex, the premotor cortex, the temporal cortex, the parietal cortex and the sensory peripheral regions. It is presumed that general intelligence would appear in adequate harmony among those brain regions under specific connectivity. The project will utilize iCub as a humanoid robot platform.

Social Cognitive Neuro-Robotics

This project aims to explore possible neuropsychological mechanisms for social cognition by conducting a set of experiments for robot-human as well as robot-robot interactions. The major research question is how human and robots can be cooperative each other in particular task by reading others' minds as well as by establishing joint attentions. Emergence of cooperative behaviors among individuals is not trivial phenomena because each individual possesses independent volition or free will. We speculate that cooperation could emerge by achieving a sort of coherence among neuronal activities of individuals by means of the mirror neuron mechanisms of reading others intentions. By conducting neuro-robotics experiments, we examine the underlying dynamic mechanisms accounting for spontaneous generation of cooperative behaviors as well as their breakdown. (This project is conducted collaboratively with Ogata and Sugano labs in Waseda University.) [PDF]     [VIDEO]

Exploring Nontrivial Dynamic Computation by Recurrent Neural Network (RNN)

Our group has investigated how nontrivial computation accounting for some aspects of the higher-order cognitive functions can be achieved in dynamics of RNN models. We have shown that a version of RNN model can function as a generative model which can both generate and recognize a set of temporal patterns via learning of the exemplar (see more here). We have also shown that multiple timescale dynamics characteristics introduced in RNN models can afford self-organization of functional hierarchy such that the fast dynamics part tends to form a set of primitive dynamics while the slow dynamics part does for macroscopic manipulation of those primitives (see more here). These results suggest that meaningful computation in terms of its compositionality and systematicity can be effectively developed in those RNN models. By extending these studies, the current project will examine how a particular RNN model can learn to extract stochastic structures hidden in observed continuous spatio-temporal patterns. We will investigate this problem by utilizing mathematical theory of chaos and symbolic dynamics. By challenging an essential mathematical question, deterministic dynamic systems (including chaos) versus stochastic systems for modeling and reconstructing stochastic complex signals, we attempt to show that there exists the middle between these two extremes. The model will be applied for recognition and regeneration of video as well as auditory streams with expectation of more fluid and contextual processing of the streams as compared to the conventional Bayesian approaches including Hidden Markov Model.

NeuroInteractive Arts Projects

This project explores novel experiences of human participants while directly interacting with emergent dynamics developed in artificial brains. The continuous interactions take place between the participants and the artificial brains in a multimodal manner by utilizing haptic devices including an active force-control joystick and a vibrotactile stimulation system as well as auditory and visual streams while the artificial brains learn and regenerate the sensory images experienced in the past. The project will examine what sorts of interactive dynamics can appear between the two sides while the participants mental states go back and forth between unconscious and conscious states. The goal of the project is to find out if this type of interactive dynamics could bring new types of phenomenological experiences for human participants.