FY2021 Annual Report

Neural Computation Unit
Professor Kenji Doya

Abstract

The Neural Computation Unit pursues the dual goals of developing robust and flexible learning algorithms and elucidating the brain’s mechanisms for robust and flexible learning. Our specific focus is on how the brain realizes reinforcement learning, in which an agent, biological or artificial, learns novel behaviors in uncertain environments by exploration and reward feedback. We combine top-down, computational approaches and bottom-up, neurobiological approaches to achieve these goals.

In FY2021, we continued research along three major externally funded projects: Kakenhi project on Artificial Intelligence and Brain Science, Brain/MINDS project, and Fugaku Supercomputing project for whole-brain simulation.

With the extension of Kakenhi Project on Artificial Intelligence and Brain Science due to the COVID pandemic, we edited a Special Issue of Neural Networks including perspective papers based on the sessions at the Internaltional Sumposium on Artificial Intelligence and Brain Science 2020. We further started to organize the Internaltional Sumposium on Artificial Intelligence and Brain Science 2022 in OIST as a satellite event for Neuro2022 in Okinawa Convention Center.

In the Brain/MINDS project, we started to analyze calcium imaging data collected by Matsuzaki lab in the University of Tokyo and RIKEN Center for Brain Science and also started to develop 'OptiNiSt', an optical neuroimage analysis platform.

Under the Program for Promoting Researches on the Supercomputer Fugaku, we collaborated with researchers of EU Human Brain Project to accelerate the spking neural network simulator NESTs and its connection with the neurorobotics simulation platform.

1. Staff

Systems Neurobiology Group

  • Katsuhiko Miyazaki, Senior Staff Scientist
  • Kayoko Miyazaki, Senior Staff Scientist
  • Yukako Yamane, Staff Scientist
  • Yuzhe Li, Postdoctoral Scholar
  • Masakazu Taira, Junior Research Fellow
  • Sergey Zobnin, OIST Student
  • Miles Desforges, OIST Student
  • Tomohiko Yoshizawa, Visiting Researcher (Hokkaido University)
  • Kazumi Kasahara, Visiting Researcher (AIST)

Dynamical Systems Group

  • Razvan Gamanut, JSPS Research Fellow
  • Soheil Keshmiri, Staff Scientist
  • Florian Lalande, OIST Student
  • Shuhei Hara, OIST Student
  • Hideyuki Yshimura, OIST Student
  • Hiroaki Hamada, Visiting Researcher (ARAYA)
  • Hiromichi Tsukada, Visiting Researcher (Chubu University)
  • Carlos Enrique Gutierrez, Visiting Researcher (Softbank)

Adaptive Systems Group

  • Christopher Buckley, Technician
  • Qiong Huang, OIST Student
  • Ho Ching Chiu, OIST Student
  • Kristine Faith Roque, OIST Student
  • Farzana Rahman, Visiting Researcher (Indipendent University)

Research Unit Administrators

  • Emiko Asato
  • Kikuko Matsuo
  • Misuzu Saito

2. Collaborations

  • Prof. Mark Walton and Dr. Thomas Akam at the University of Oxford for analyzing model-free and model-based decision making in mice.
  • Prof. Masanori Matsuzaki and Dr. Teppei Ebina at the University of Tokyo and RIKEN Center for Brain Science for the analysis of the calcium imaging data of the marmoset brain under Brain/MINDS project.
  • Prof. Hans Ekkehard Plesser in HBP EXABRAINPREP Boucher Program for acceleration of NEST3 spiking neural network simulation on Fugaku.
  • Prof. Fabrice Morin at HBP Neurorobotics Platform for combined simulation of our whole-brain model by NEST3 and realistic musculoskeletal systems.
  • An industrial collaboratory research project on brian-like computation.
  • An industrial collaboratory research project on the analysis and modeling of health-related data from wearable devices.

3. Activities and Findings

3.1 Nuerobiology Experiments [Systems Neurobilogy Group]

3.1.1 Neural substrate of dynamic Bayesian inference [Kakenhi Project on AI and Brain Science]

We continued our experiments to clarify how predictive information from actions and sensory information through environmental interaction are integrated across different layers of the somatosensory and motor cortical circuits in mice. Our preliminary analysis of calcium imaging data through a prism lens implanted in the somatosensory cortex revealed context-dependent modulation of preparatory and sensory activities.

3.1.2 The role of serotonin in the regulation of patience [Kakenhi Project on AI and Brain Science] 

To test the hypothesis that serotonin regulated the balance between model-free and model-based decision making, we implemented a multi-step reinforcement learning task for mice and found that optogenetic inhibition of serotonin neurons in the dorsal raphe nucleus decreases the weight of model-based component.

3.2. Neural Data Analysis and Modeling [Dynamical Systems Group]

3.2.1 Analysis and modeling of marmoset brain data [Brain/MINDS Project]

We started to analyze calcium imaging data collected by Matsuzaki lab in the University of Tokyo and RIKEN Center for Brain Science. For a wide-field imaging data spanning from the parietal cortex to the premotor cortex, we extracted tens of spatially localized components by non-negative matrix factorization (NMF) and revealed causal relationships among them by convergence cross mapping (CCM).

We also started to develop 'OptiNiSt', an optical neuroimage analysis platform.

3.2.2 Spiking neural network model of the basal ganglia [Fugaku Project]

We developed SNNbuilder, a softward platform to register anatomical and physiological data and produce simulation codes in NEST. We developed a biologically realistic model of the basal ganglia circuit using SNNbuilder and tested its behavoir by simulation on Fugaku supercomputer.

3.3 Robotics and Reinforcement Learning [Adaptive Systems Group]

3.3.1 Data-efficient reinforcement learning [Kakenhi Project on AI and Brain Science]

Our paper of model-free imitation learning by forward and inverse reinforcement learning was published in Neural Networks. In collaboration with Tani unit, our paper on 'oracle guidance' reinforcement learning which utilizes retrospective knowledge was accepted in ICLR2022 conference.

3.3.2 Smartphone robot platform [Kakenhi Project on AI and Brain Science]

Our smartphone robots were presented in the AI Art and Aesthetics Exhibition in Nakagawa Village. We started desining new robot hardware with improved energy management.

Smartphone Robots

4. Publications

4.1 Journals

  1. Doya K (2021). Canonical cortical circuits and the duality of Bayesian inference and optimal control. Current Opinion in Behavioral Sciences, 41, 160-167. https://doi.org/10.1016/j.cobeha.2021.07.003

  2. Doya K, Ema A, Kitano H, Sakagami M, Russell S (2022). Social impact and governance of AI and neurotechnologies, Neural Networks. https://doi.org/10.1016/j.neunet.2022.05.012

  3. Feldotto B, Eppler JM, Jimenez-Romero C, Bignamini C, Gutierrez CE, Albanese U, Retamino E, Vorobev V, Zolfaghari V, Upton A, Sun Z, Yamaura H, Heidarinejad M, Klijn W, Morrison A, Cruz F, McMurtrie C, Knoll AC, Igarashi J, Yamazaki T, Doya K, Morin FO (2022). Deploying and optimizing embodied wimulations of large-scale spiking neural networks on HPC infrastructure. Frontiers in Neuroinformatics, 16. https://doi.org/10.3389/fninf.2022.884180

  4. Girard B, Lienard J, Gutierrez CE, Delord B, Doya K (2021). A biologically constrained spiking neural network model of the primate basal ganglia with overlapping pathways exhibits action selection. European Journal of Neuroscience, 53, 2254-2277. https://doi.org/10.1111/ejn.14869

  5. Ito J, Joana C, Yamane Y, Fujita I, Tamura H, Maldonado P, Grün S (2021). Latency shortening with enhanced sparseness and responsiveness in V1 during active visual sensing. Scientific Reports, 12, 6021. https://www.nature.com/articles/s41598-022-09405-4

  6. Miyazaki K, Miyazaki KW (2021). Increased serotonin prevents compulsion in addiction. Science, 373, 1197-1198. https://www.science.org/doi/full/10.1126/science.abl6285

  7. Taira M, Desforges M (2021). ZFPM1 necessary for development of serotonergic projections related to anxiety and contextual fear learning. Journal of Neuroscience, 41, 3945-3947. https://doi.org/10.1523/jneurosci.3125-20.2021

  8. Taniguchi T, Yamakawa H, Nagai T, Doya K, Sakagami M, Suzuki M, Nakamura T, Taniguchi A A (2022). Whole brain probabilistic generative model: Toward realizing cognitive architectures for developmental robots. Neural Networks, 150, 293-312. https://doi.org/10.1016/j.neunet.2022.02.026

  9. Uchibe E, Doya K (2021). Forward and inverse reinforcement learning sharing network weights and hyperparameters. Neural Networks, 144, 138-153. https://doi.org/10.1016/j.neunet.2021.08.017

  10. Zhang X, Liu Y, Xu X, Huang Q, Mao H, Carie A (2021). Structural relational inference actor-critic for multi-agent reinforcement learning. Neurocomputing, 459, 383-394. https://doi.org/10.1016/j.neucom.2021.07.014

4.2 Books and other one-time publications

     Nothing to report

4.3 Oral and Poster Presentations

Invited Lecture/Seminar

  1. Doya K (2021). What can we learn from the brain for future AI?  Brain2AI Workshop, International Conference on Learning Representations (ICLR 2021) Online 2021/5/8 https://iclr.cc/virtual/2021/workshop/2135

  2. Doya K (2021). Duality of control and inference and the cortical circuit architecture. Symposiuum on Biology of Decision Making (SBDM 2021) Paris (online) 2021/5/21 https://sbdm2021.sciencesconf.org

  3. Doya K (2021). The duality of control and inference as a clue for cracking the codes of frontal and sensory cortical architectures. CJK Symposium 17, Theoretical Brain, The 44th annual meeting of the Japan Neuroscience Society Online 2021/7/31 http://ut-neurochem.umin.jp/cjk1/

  4. Doya K (2021). The duality of inference and control as a key to understanding canonical cortical circuits. RIKEN-OIST symposium
    "Kinds of Minds-What is thinking? Online 2021/10/6 https://groups.oist.jp/odr/event/oist-riken-2nd-symposium

  5. Doya K (2021). Predicting animal behaviors and neural activities. The 34th Annual Meeting of the Japanese Society for Alternatives to Animal Experiments Online 2021/11/11 https://square.umin.ac.jp/jsaae34/

  6. Doya K (2021). What will smart-phone robots evolve for? The 38th Artificial Intelligence Art and Aesthetics Symposium. Nakagawa Village 2021/12/11 https://www.aibigeiken.com/exhibition2021/index.html

  7. Doya K (2021). Natural and artificial reinforcement learning. Deep Reinforcement Learning Workshop, NeurIPS2021 Online 2021/12/13 https://sites.google.com/view/deep-rl-workshop-neurips2021

  8. Doya K (2022). Neural circuits for mental simulation. 8th Annual Conference of Cognitive Science
    Amrita (Online) 2022/1/21 https://amrita.edu/event/accs8/

  9. Doya K (2022). Context-dependent interaction of dopamine and serotonin. COSYNE2022 Neuromodulation Workshop Lisboa Congress Center (Online) 2022/3/22 https://www.cosyne.org/

  10. Doya K (2022). Theory-based and data-driven brain science. Kick-off Symposium, Center for Mathematical Science and Artificial Intelligence Chubu University 2021/10/17 https://sites.google.com/view/cmsai-kick-off-symposium/home

  11. Doya K (2022). Canonical cortical circuits and the duality of inference and control. 43rd Annual Meeting, National Institute of Neuroscience Online 2022/3/23

  12. Doya K (2022). Embodied agents for survival, reproduction, and prediction. International Conference on Embodied Intelligence Online 2022/3/23 https://embodied-intelligence.org

Conference Oral/Poster presentations

  1. Lalande, Florian (2021). Shedding light on the exoplanet population with machine learning. AbGradCon2021 Online 2021/9/15 https://astrobiology.nasa.gov/news/virtual-abgradcon-2021-video-online/

  2. Taira, Masakazu (2021). The effect of photoinhibition of dorsal raphe serotonin neurons on stationary waiting and sustained motor actions for future rewards. The 44th annual meeting of the Japan Neuroscience Society Kobe Convention Center 2021/7/30 https://neuroscience2021.jnss.org/en/index.html

  3. Yuzhe Li, Kenji Doya (2021). Investigation of information flow and temporal-spatial organization of neurons across cortical layers from multi-depth two-photon imaging data JNNS2021 Online 2021/9/21 https://jnns2021.brainsci.net

  4. Y. Yamane, T. Ebina, A. Sasagawa, S. Terada, M. Uemura, K. Ohki, M. Matsuzaki, and K. Doya (2021). Signal causality analysis of marmoset wide-field calcium imaging. JNNS2021 Online 2021/9/21 https://jnns2021.brainsci.net

  5. Yukako Yamane,  Kenji Doya (2021). Neural mechanism of uncertainty judgement. Vision Society of Japan Online 2021/9/24 http://vision-lab.tp.chiba-u.jp/vsj2021summer/index.html

  6. Carlos Enrique Gutierrez, Jean Lienard, Benoît Girard, Hidetoshi Urakubo, Yuko Ishiwaka, and Kenji Doya (2022). Exploration of learning by dopamine D1 and D2 receptors by a spiking network model of the basal ganglia. COSYNE2022 Lisboa Congress Center  2022/3/18 https://www.cosyne.org/poster-session-2

  7. Taira M (2022). Serotonergic control of model-based decision making. COSYNE2022 Lisboa Congress Center (Online) 2022/3/19 https://www.cosyne.org/

5. Intellectual Property Rights and Other Specific Achievements

       Nothing to report

6. Meetings and Events

6.1 Neural Computation Workshop 2021

  • Date: December 4, 2021
  • Venue: OIST conference center (Auditorium)
  • Sponsor: Kakenhi Project on Artificial Intelligence and Brain Science

6.2 Mechanism of Brain and Mind The 21st Winter Workshop

  • Date: January 12, 2022
  • Venue: Online (Webinar)
  • Sponsor: Kakenhi Project on Artificial Intelligence and Brain Science

7. Other

Kenji Doya serves the President of The 45th Annual Meeting of the Japan Neuroscience Society (Neuro2022) hosted for the first time in Okinawa.