FY2018 Annual Report

Neural Computation Unit
Professor Kenji Doya

Group Photo

 

Abstract

The Neural Computation Unit pursues the dual goals of developing robust and flexible learning algorithms and elucidating the brain’s mechanisms for robust and flexible learning. Our specific focus is on how the brain realizes reinforcement learning, in which an agent, biological or artificial, learns novel behaviors in uncertain environments by exploration and reward feedback. We combine top-down, computational approaches and bottom-up, neurobiological approaches to achieve these goals.

In FY2018, we continued research along three major externally funded projects: Kakenhi project on Artificial Intelligence and Brain Science, Brain/MINDS project, and Post-K Supercomputing project on Brain and Artificial Intelligence.

In the Kakenhi project on Artificial Intelligence and Brain Science, we wrote perspective papers on what we can learn from the brain for next-generation AI. We also published papers on a Bayesian decision making model of the effect of serotonergic activation and identification of depression subtypes by unspervised learning.

In Japan's major brain science project, Brain/MINDS, we implemented a new optimization framework for fiber tracking into the neural data analysis pipelines through close collaboration with researchers at RIKEN and Kyoto University. We also started an international collaboration with Prof. Gustavo Deco in analysis and modeling of diffusion and functional MRI data.

In the Post-K supercomputing project on Brain and Artificial Intellignece, we built biologically constrained spiking neural network model of the basal ganglia circuit and integrated that with the cerebellum and thalamo-cortical network models constructed by our collaboratos at UEC and RIKEN for whole-brain simulations.

1. Staff

Systems Neurobiology Group

  • Katsuhiko Miyazaki, Staff Scientist
  • Kayoko Miyazaki, JSPS Research Fellow
  • Kazumi Kasahara, JSPS Research Fellow
  • Yukako Yamane, JSPS Research Fellow
  • Yuzhe Li, Postdoctoral Scholar
  • Zeng Jiafu, Technician
  • Sergey Zobnin, OIST student
  • Masakazu Taira, OIST student
  • Miles Desforges, OIST student
  • Tomohiko Yoshizawa, Visiting Researcher

Dynamical Systems Group

  • Carlos Enrique Gutierrez, Postdoctoral Scholar
  • Hiromichi Tsukada, Postdoctoral Scholar
  • Hiroaki Hamada, OIST Student
  • Jessica Verena Schulze, OIST Student
  • Junichiro Yoshimoto, Visiting Researcher
  • Tomoki Tokuda, Visiting Researcher

Adaptive Systems Group

  • ​​Marie-Lou Barnaud, Postdoctoral Scholar
  • Christopher Buckley, Technician
  • Shoko Igarashi, OIST Student
  • Chris Reinke, OIST Student
  • Farzana Rahmen, OIST Student
  • Tadashi Kozuno, OIST Student
  • Paavo Parmas, OIST Student
  • Ho Ching Chiu, OIST Student

Research Unit Administrators

  • Emiko Asato
  • Kikuko Matsuo
  • Misuzu Saito

2. Collaborations

  • Prof. Gustavo Deco at Pompeu Fabra University for integration of structural and functional MRI data in Brain/MINDS project
  • Prof. Shin Ishii and his lab members at Kyoto University for neural tracer data analysis in the Brain/MINDS project
  • Prof. Benoit Girarad at Sorbonne University, Dr. Jun Igarashi at RIKEN, and Prof. Tadashi Yamazaki at University of Electro-Communications for buildin whole-brain spiking networks models in Post-K Supercomputing Project
  • Dr. Tomoki Tokuda and Prof. Junichiro Yoshimoto at Nara Institute of Science and Technology and Prof. Shigeto Yamawaki at Hiroshima Univeristy School of Medicine for identification of subtypes of depression.

3. Activities and Findings

3.1 Neurobiology Experiments [Systems Neurobiology Group]

3.1.1 Neural substrate of dynamic Bayesian inference [Kakenhi Project on AI and Brain Science]

We designed a new experimental paradigm to address how predictive information from actions and sensory information through environmental interaction are integrated across different layers of the  somatosensory and motor cortical circuits in mice. We developed a novel active level manipulation tasks for head-fixed mice and prepared for simultaneous cross-layer calcium imaging by a prism lens.

3.1.2 The role of serotonin in the regulation of patience [Kakenhi Project on AI and Brain Science] 

In order to clarify the role of serotonin in behaviors, we performed neural recording, microdialysis measurement, and optogenetic manipulation of serotonin neural activity from the dorsal raphe nucleus (DRN), the major source of serotonergic projection to the cortex and the basal ganglia. We had found that optogenetic stimulation of DRN serotonin neurons prolonged the time animals spent for waiting for reward (Miyazaki et al., 2016). We further found that serotonin’s effect on waiting was seen only when reward probability was high, and more effective when reward timing was uncertain. We constructed a Bayesian decision model and reproduced the experimental results (Miyazaki et al. 2018)

3.2. Neural Data Analysis and Modeling [Dynamical Systems Group]

3.2.1 Subtype identification of depression from multidimensional data [Kakenhi Project on AI and Brain Science] 

For data-driven identification of subtypes of depression, we had developed a novel multiple co-clustering algorith (Tokuda et al., 2017). We applyed this method to high-dimensional data including resting-state functional MRI data,blood markers, genetic polymorphism, and clinical questionnaires, obtained by our collaborators at the department of psychiatry, Hiroshima University School of Medicine. As a result, three subtypes among depression patients were identified; those with higher functional connectivities centered around the angular gyrus and stressful history in the childhood tended to be less responsive standard therapy by SSRI (Tokuda et al., 2018)

3.2.2 Analysis and modeling of marmoset brain data [Brain/MINDS Project]

We develop data-analysis pipelines and modeling methods to utilize the variety of marmoset brain data obtained in the Brain/MIDNS program. We have developed Bayesian synaptic connectivity inference algorithms, a framework for optimizing diffusion MRI-based fiber-tracking algorithm in reference to neural tracer data, and a framework to integrate the structural connectivity based on diffusion MRI data and the resting-state functional MRI data.

3.2.3 Large-scale spiking network models of the basal ganglia [Post-K Supercomputing Project on Brain and Artificial Intelligence]

We constructed an integrate-and-fire neural network model of the basal ganglia circuit based on the anatomical and physiological data and explored the mechanisms for action selection. We further connected the basal ganglia model with the cerebellar and thalamo-cortical network models constructed by our collaborators at UEC and RIKEN.

3.3 Robotics and Reinforcement Learning [Adaptive Systems Group]

3.3.1 Data-efficient reinforcement learning [Kakenhi Project on AI and Brain Science]

While the combination of reinforcement learning with deep neural networks has shown remarkable performance, the requirement of huge data for training is a major issue in robotic applications. We formulated "Conservative Value Iteration" algorithm and thoeretically proved its data efficiency under sampling noise and approximation errors (Kozuno et al., 2019). We also clarified the problems in previous model-based reinforcement learning algorithms (Parmas et al., 2018) and derived an efficient gradient computation method for general probabilistic graphic models (Parmas, 2018).

3.3.2 Development of smartphone robot platform [Kakenhi Project on AI and Brain Science]

We started to rebump our smartphone robot platform to make its electronics more stable and software compatible with the latest Android OSs. The robots will be utileze for a variety of research topics, such as benchmarks for data-efficient reinforcement learning, evolution of alternative reproductive strategies, and learning to cooperate with symbolic communication.

4. Publications

4.1 Journals

  1. Doya K, Matsuo Y (2019). Artificial intelligence and brain science: the present and the future. Brain and Nerve. (in press, in Japanese).
  2. Doya K, Taniguchi T (2019). Toward evolutionary and developmental intelligence. Current Opinion in Behavioral Sciences, 29, 91-96. http://doi.org/10.1016/j.cobeha.2019.04.006
  3. Elfwing S, Uchibe E, Doya K (2018). Sigmoid-weighted linear units for neural network function approximation in reinforcement learning. Neural Networks http://doi.org/10.1016/j.neunet.2017.12.012
  4. Kasahara K, Hoshino H, Furusawa Y, DaSalla CS, Honda M, Murata M (2018). Initial experience with a sensorimotor rhythm-based brain-computer interface in a Parkinson’s disease patient. Brain-Computer Interfaces. http://doi.org/10.1080/2326263X.2018.1440781
  5. Magrans de Abril I, Yoshimoto J, Doya K (2018). Connectivity inference from neural recording data: Challenges, mathematical bases and research directions. Neural Networks. http://doi.org/10.1016/j.neunet.2018.02.016
  6. Miyazaki K, Miyazaki KW, Yamanaka A, Tokuda T, Tanaka KF, Doya K (2018). Reward probability and timing uncertainty alter the effect of dorsal raphe serotonin neurons on patience. Nat Commun, 9, 2048. http://doi.org/10.1038/s41467-018-04496-y
  7. Tokuda T, Yoshimoto J, Shimizu Y, Okada G, Takamura M, Okamoto Y, Yamawaki S, Doya K (2018). Identification of depression subtypes and relevant brain regions using a data-driven approach. Sci Rep, 8, 14082. http://doi.org/10.1038/s41598-018-32521-z
  8. Yoshizawa T, Ito M, Doya K (2018). Reward-predictive neural activities in striatal striosome compartments. eNeuro, 5. http://doi.org/10.1523/ENEURO.0367-17.2018

4.2 Books and other one-time publications

  1. Reinke C (2018). The gamma-ensemble: adaptive reinforcement learning via modular discounting. PhD Thesis, Okinawa Institute of Science and Technology Graduate University. http://doi.org/10.15102/1394.00000369
  2. Schulze JV (2018). Spatial and modular regularization in effective connectivity inference from neural activity data. PhD thesis, Okinawa Institute of Science and Technology Graduate University.  http://doi.org/10.15102/1394.00000808
  3. Hamada H (2019). Serotonergic control of brain-wide dynamics. PhD Thesis, Okinawa Institute of Science and Technology Graduate University. http://doi.org/10.15102/1394.00000739
  4. 銅谷賢治 監訳 (2019). ディープラーニング革命. ニュートンプレス. (Supervisory translation of The Deep Learning Revolution by Terrence J. Sejnowski. MIT Press, 2018)

4.3 Oral and Poster Presentations

  1. Berthelon G, Liénard J, Doya K, Girard B.(2018). Dichotomous organization of the globus pallidus externa reproduces long pauses in a spiking model of the monkey basal ganglia. 8th International Symposium on Biology of Decision Making (SBDM2018) Paris, France.
  2. Doya K.(2018). Introductory talk: Building autonomous robots to understand what brains do. Satellite workshop of 8th International Symposium on Biology of Decision Making (SBDM2018) Sorbonne Université, Paris, France.
  3. Doya K (2018). How can the brain circuit modules connect each other as needed? , Bunkyo-ku, Tokyo, 3rd whole brain architecture initiative symposium.
  4. Doya K (2018). Robotic and AI approaches to behavioral disorders and brain functions. Nago, Okinawa, 22nd Meeting of Japan Society for Eating Disorders  
  5. Doya K (2018). Visualizing the neural circuits for imagination and intelligence. Okinawa Japan, 58th Annual Scientific Meeting of the Japanese Society of Nuclear Medicine.
  6. Doya K (2018). What can we learn from the brain for artificial intelligence. Yurakucho Asahi Hall, Tokyo, 26th Brain Century Promotion Conference Symposium.
  7. Doya K (2018). What can we further learn from the brain for AI and robotics? ICDL-Epirob2018. Waseda University, Tokyo.
  8. Doya K.(2018). Neural mechanisms of mental simulation. 18th China-Japan-Korea Joint Workshop on Neurobiology and Neuroinformatics (NBNI2018). Jeju Island, Korea.
  9. Doya K.(2018). Imaging the neural circuit for mental simulation. OIST Mini Symposium "Voltage Imaging Symposium". OIST.
  10. Doya K.(2018). Neural circuits for reinforcement learning and mental simulation NGP Summer School 2018 Neuroscience for Emotion & Motivation. Tohoku University, Sendai.
  11. Doya K.(2018). Computational models of reinforcement learning. Methods in Computational Neuroscience Course. Marine Biological Laboratory, Woods Hole, USA.
  12. Doya K.(2018). Imaging the neural circuit for mental simulation. 2018 CFC (KIST) fall seminar. Korea Institute of Science and Technology(KIST).
  13. Doya K.(2019). Reinforcement learning: Computational theory and neural mechanisms. IRCN Neuro-Inspired Computation Course. Sanjo Conference Hall, The University of Tokyo.
  14. Doya K.(2019). Neural circuit for mental simulation. COSYNE 2019 Plenary lecture. Lisbon, Portugal.
  15. Doya K.(2019). Patience and beyond. COSYNE 2019 Workshop -Advances and Convergences in 5HT Research-. Cascais, Portugal.
  16. Gutierrez CE.(2018). Scaling of multiple-receptor synaptic connection methods in NEST NEST conference 2018: A forum for users and developers. Norwegian University of Life Sciences (NMBU)  
  17. Gutierrez C, Lienard J, Girard B, Igarashi J, Doya K.(2018). Spiking neural network model of the basal ganglia with realistic topological organization. AINI2018 Advances in Neuroinformatics 2018. RIKEN Wako-shi, Saitama.
  18. Gutierrez CE, Skibbe H, Nakae K, Lienard J, Woodward A, Watakabe A, Tsukada H, Hata J, Okano H, Yamamori T, Ishii S, Doya k.(2019). Multi-objective parameter optimization of DWI-based global fiber tracking with neuronal tracer signal as a reference. International Symposium of Brain/MINDS ISBM2019. Ito Hall, The University of Tokyo.
  19. Kozuno T, Doya K.(2018). Theoretical analysis of Non-exact retrace algorithm. JNNS2018 28th Annual Conference of the Japanese Neural Network Society OIST, Onna-son Okinawa, Japan.
  20. Kozuno T, Uchibe E, Doya K.(2019). Theoretical analysis of efficiency and robustness of softmax and gap-increasing operators in reinforcement learning. 22nd International Conference on Artificial Intelligence and Statistics. Naha city, Okinawa, Japan.
  21. Liénard J, Girard B, Doya K.(2018). Action selection and reinforcement learning in a basal ganglia model. 8th International Symposium on Biology of Decision Making (SBDM2018) Paris, France.
  22. Liénard J, Girard B, Doya K.(2018). Examination of the roles of basal ganglia afferents in action selection and learning by spiking neuron models Neuroscience 2018 Society for Neuroscience. SanDiego, USA.
  23. Paramas P.(2018). Total stochastic gradient algorithms with application to model-based reinforcement learning. IBIS 2018 21st Information -based induction science workshop. Hokkaido University, Sapporo.
  24. Parmas P, Rasmussen CE, Peters J, Doya K.(2018). PIPPS: Flexible model-based policy search robust to the curse of chaos. ICML2018 Thirty-fifth International Conference on Machine Learning. Stockholm, Sweden.
  25. Parmas P.(2018). Total stochastic gradient algorithms and applications in reinforcement learning. NeurIPS2018 Thirty-second Conference on Neural Information Processing Systems. Montreal, Canada.
  26. Tsukada H, Hamada H, Nakae K, Ishii S, Hata J, Okano H, Doya K.(2018). Analysis of structure-function relationship using a whole-brain model based on the common marmoset MRI data. 28th Annual Conference of the Japanese Neural Network Society (JNNS2018). OIST Onna-son, Okinawa, Japan.
  27. Tsukada H, Hamada H, Gutierrez C, Nakae K, Skibbe H, Ishii S, Woodward A, Hata J, Okano H, Doya K.(2018). Analysis of local excitatory-inhibitory balance using whole-brain model based on the common marmoset MRI data. AINI2018 Advances in Neuroinformatics 2018. RIKEN Wako-shi, Saitama.
  28. Tsukada H, Hamada H, Gutierrez CE, Nakae K, Skibbe H, Ishii S, Woodward A, Hata J, Okano H, Doya K.(2019). Whole-brain modeling and dynamical analysis of structural and functional MRI data of the common marmoset. International Symposium of Brain/MINDS ISBM2019. Ito Hall, The University of Tokyo.
  29. Yamane Y, Ito J, Joana C, Fujita I, Tamura H, Maldonado P, Doya K, Grün S.(2018). Representation of fixated objects by multiple single unit activity in visual cortices of freely viewing macaque monkeys. 11th FENS Forum of Neuroscience. CityCube Berlin, Berlin, Germany  
  30. Yamane Y, Ito J, Joana C, Fujita I, Tamura H, Maldonado P, Doya K, Grün S (2018). Inferring fixated objects in free viewing from parallel neuronal spiking activities in macaque monkeys. Kobe Convention Center, Kobe, Hyogo, The 41st Annual Meeting of the Japan Neuroscience Society.
  31. Yamane Y, Ito J, Joana C, Fujita I, Tamura H, Maldonado P, Doya K, Grun S.(2018). Neuronal activity of macaque visual cortices during free viewing. 19th winter workshop mechanism of brain and mind. Rusutsu, Hokkaido, Japan.

5. Intellectual Property Rights and Other Specific Achievements

  1. Paavo Parmas (2018). Total Stochastic Gradient Method. US Patent Application 62/749,908.

  2. T Sasaki, E Uchibe, K Doya, H Anai, H Yanami, H Iwane (2019). Recording medium, reinforcement learning method, and reinforcement learning apparatus. US Patent Application 16/130,482.

  3. T Sasaki, E Uchibe, K Doya, H Anai, H Yanami, H Iwane (2019). Recording medium, policy improving method, and policy improving apparatus. US Patent Application 16/130,469.

6. Meetings and Events

6.1 Seminars

Critical brains for autonomous robots

  • Date:  April 24, 2018
  • Venue: OIST Campus Lab1
  • Speaker: Dr. J. Michael Herrmann (University of Edinburgh and Edinburgh Centre for Robotics)

Functional anatomy of mouse parietal cortex with wide field-of-view two-photon calcium imaging

  • Date:  July 30, 2018
  • Venue: OIST Campus Lab1
  • Speaker: Dr. Riichiro Hira (Smith lab at UNC Chapel Hill)

Compressed data structures

  • Date:  August 21, 2018
  • Venue: OIST Campus Lab1
  • Speaker: Dr. Jean-François Baffier ( JSPS-CNRS research fellow, Tokyo Institute of Technology)

Flow of knowledge in information networks

  • Date:  August 22, 2018
  • Venue: OIST Campus Lab1
  • Speaker: Dr. Jean-François Baffier ( JSPS-CNRS research fellow, Tokyo Institute of Technology)

Deep reinforcement learning by parallelizing reward and punishment using the MaxPain architecture

  • Date:  November 14, 2018
  • Venue: OIST Campus Lab1
  • Speaker: Dr. Jiexin Wang (ATR Computational Neuroscience Laboratories)

 Collective computation and learning in nonlinear networks

  • Date:  January 15, 2019
  • Venue: OIST Campus Lab1
  • Speaker: Prof. Jean Jacques Slotine (Massachusetts Institute of Technology)

Adaptive nonlinear control and learning dynamics with spiking networks

  • Date:  January 16, 2019
  • Venue: OIST Campus Lab1
  • Speaker: Prof. Jean Jacques Slotine (Massachusetts Institute of Technology)

An automated high-throughput image processing pipeline for the extraction of structural connectivity from marmoset brain images

  • Date:  January 23, 2019
  • Venue: OIST Campus Lab1
  • Speaker: Dr. Henrik Skibbe  (Kyoto University)

Sharp-wave ripple production and coordination in sleeping dragons

  • Date:  March 19, 2019
  • Venue: OIST Campus Lab1
  • Speaker: Dr. Hiroaki Norimoto (Max Planck Institute for Brain Research)

Network architecture underlying sparse neural activity characterized by structured higher-order interactions

  • Date:  March 26, 2019
  • Venue: OIST Campus Lab1
  • Speaker: Dr. Hideaki Shimazaki (Kyoto University)

6.2 Events

Mini Workshop on Robot Learning as a Validation Tool for Neuroscience & Artificial Intelligence in Joint Workshop on Neuro-Computing, Bioinformatics, Mathematical modeling & Machine Learning

  • Date: June 14, 2018
  • Venue: OIST Conference Center
  • Speakers:
    • Prof. Jun Tani（OIST）
    • Dr. Eiji Uchibe (ATR）
    • Ass.Prof.Tadashi Yamazaki（UEC）

The Joint Research Area MeetingScientific Research on Innovative Areas:“Artificial Intelligence and Brain Science” and “Adaptive Circuit Shift”

  • Date: May 9 - May 11, 2018
  • Venue: OIST Center Building

OIST x CiNet Workshop for Future Neuroscience and Technology

  • Date: October 17- October 19, 2018
  • Venue: OIST Campus Center Building & Okinawa Electoromagnetic Technology Center
  • Sponsors: OIST & Center for Information and Neural Networks (CiNet)

The 28th Annual Conference of the Japanese Neural Network Society (JNNS2018)

  • Date: October 24- October 27, 2018
  • Venue: OIST Conference Center
  • Sponsor: Japanese Neural Network Society
  • Co-Sponsors: OIST & Kakenhi Project on Artificial Intelligence and Brain Science
  • Key Note Lecturers:

The 5th Research Area Meeting "Artificial Intelligence and Brain Science"

  • Date: November 12 - Novemver 13, 2018
  • Venue: Advanced Telecommunications Research Institute International (ATR)
  • Sponsor: Kakenhi Project on Artificial Intelligence and Brain Science

Research Area Meeting "Evolinguistics"

Neural Computation Workshop

  • Date: March 15, 2019
  • Venue: OIST Seaside House
  • Speakers:
    • Makoto Ito (Progress Technologies Inc.)
    • Tomohiko Yoshizawa (Tamagawa University)
    • Yuzhe Li (OIST)
    • Tomoki Tokuda (NAIST)
    • Junichiro Yoshimoto (NAIST)
    • Yukako Yamane (OIST)
    • Stefan Elfwing (ATR)
    • Paavo Parmas (OIST)
    • Jiexin Wang (ATR)
    • Christopher Buckley (OIST)
    • Jun Igarashi (RIKEN)
    • Hiromichi Tsukada (OIST)
    • Carlos Enrique Gutierrez (OIST)
    • Kenji Doya (OIST)

7. Other

  • Prof. Kenji Doya was awared D. O. Hebb Award from the International Neural Network Society.