FY2020 Annual Report

Neural Computation Unit
Professor Kenji Doya

Abstract

The Neural Computation Unit pursues the dual goals of developing robust and flexible learning algorithms and elucidating the brain’s mechanisms for robust and flexible learning. Our specific focus is on how the brain realizes reinforcement learning, in which an agent, biological or artificial, learns novel behaviors in uncertain environments by exploration and reward feedback. We combine top-down, computational approaches and bottom-up, neurobiological approaches to achieve these goals.

In FY2020, we continued research along three major externally funded projects: Kakenhi project on Artificial Intelligence and Brain Science, Brain/MINDS project, and Fugaku Supercomputing project for whole-brain simulation.

To conclude our Kakenhi Project on Artificial Intelligence and Brain Science, we organized the Internaltional Sumposium on Artificial Intelligence and Brain Science with more than 1,800 online registrations. Tadashi Kozuno's collaborative work with researchers at Google and DeepMind Paris was selected for an oral presentation at NeurIPS 2020. 

In the Brain/MINDS project, our paper on optimization and validation of diffusion-MRI-based fiber tracking algorithms using neural tracer data as the reference was published in Scientific Reports.

Our Project on Human-scale Whole Brain Simulation with collaboratos at UEC, RIKEN and Kyoto U was accepted by the Program for Promoting Researches on the Supercomputer Fugaku.

1. Staff

Systems Neurobiology Group

  • Katsuhiko Miyazaki, Staff Scientist
  • Kayoko Miyazaki, JSPS Research Fellow
  • Yukako Yamane, Staff Scientist
  • Yuzhe Li, Postdoctoral Scholar
  • Sergey Zobnin, OIST Student
  • Masakazu Taira, OIST Student
  • Miles Desforges, OIST Student
  • Tomohiko Yoshizawa, Visiting Researcher (TMDU)
  • Kazumi Kasahara, Visiting Researcher (AIST)

Dynamical Systems Group

  • Carlos Enrique Gutierrez, Postdoctoral Scholar
  • Hiromichi Tsukada, Postdoctoral Scholar (Associate Professor, Chubu University from Sep. 2020)
  • Florian Lalande, OIST Student
  • Zhiguang Mu, OIST Student
  • Tomoki Tokuda, Visiting Researcher (NAIST, ATR)
  • Hiroaki Hamada, Visiting Researcher (ARAYA)

Adaptive Systems Group

  • Christopher Buckley, Technician
  • Shoko Ota, OIST Student
  • Qiong Huang, OIST Student
  • Farzana Rahman, OIST Student/Junior Research Fellow (Assistant Professor at Independent University, Bangladesh from Feb. 2021)
  • Tadashi Kozuno, OIST Student/Junior Research Fellow (Postdoctoral Fellow, University of Alberta from Jan. 2021)
  • Paavo Parmas, OIST Student/Junior Research Fellow (Assistant Professor, Kyoto University from Nov. 2020)
  • Ho Ching Chiu, OIST Student
  • Kristine Faith Roque, OIST Student

Research Unit Administrators

  • Emiko Asato
  • Kikuko Matsuo
  • Misuzu Saito

2. Collaborations

  • Prof. Fabrice Morin at HBP Neurorobotics Platform for combined simulation of our whole-brain model by NEST3 and realistic musculoskeletal systems.
  • Prof. Hans Ekkehard Plesser in HBP EXABRAINPREP Boucher Program for acceleration of NEST3 spiking neural network simulation on Fugaku.

3. Activities and Findings

3.1 Nuerobiology Experiments [Systems Neurobilogy Group]

3.1.1 Neural substrate of dynamic Bayesian inference [Kakenhi Project on AI and Brain Science]

We continued our experiments to clarify how predictive information from actions and sensory information through environmental interaction are integrated across different layers of the  somatosensory and motor cortical circuits in mice. We trained mice to perform active level manipulation tasks and succeeded simultaneous cross-layer calcium imaging by a prism lens.

3.1.2 The role of serotonin in the regulation of patience [Kakenhi Project on AI and Brain Science] 

To clarify the role of serotonin in behaviors, we performed optogenetic stimulation of serotonergic axon terminals in the orbitofrontal cortex (OFC), medial prefrontal cortex (mPFC) and the nucleus accumbens and found differential effects on promoting waiting for delayed rewards (Miyazaki et al., Science Advances 2020). The effects were explained by extending our Bayesian decision model (Miyazaki et al., Nature Communications, 2018) .

3.2. Neural Data Analysis and Modeling [Dynamical Systems Group]

3.2.1 Analysis and modeling of marmoset brain data [Brain/MINDS Project]

Our framework for optimization and validation of diffusion MRI-based fiber-tracking algorithms in reference to neural tracer data was published in Scientific Reports (Gutierrez et al. 2020) and the source code was made public in GitHub.

3.2.1 Spiking neural network model of the basal ganglia [Fugaku Project]

Our spiking neural network model of the primate basal ganglia circuit was published in European Journal of Neuroscience (Girard et al., 2020).

3.3 Robotics and Reinforcement Learning [Adaptive Systems Group]

3.3.1 Data-efficient reinforcement learning [Kakenhi Project on AI and Brain Science]

While the combination of reinforcement learning with deep neural networks has shown remarkable performance, the requirement of huge data for training is a major issue in robotic applications. Tadashi Kozuno's analysis of data efficiency of KL-regularized reinforcement learning was accepted for oral presentation at NeurIPS 2020.

3.3.2 Smartphone robot platform [Kakenhi Project on AI and Brain Science]

We re-designed the hardware and software of our smartphone-based robots and realized docking to charging station of survival and data transfer by QR code for software reproduction.

 

 

4. Publications

4.1 Journals

  1. Abe Y, Takata N, Sakai Y, Hamada HT, Hiraoka Y, Aida T, Tanaka K, Bihan DL, Doya K, Tanaka KF (2020). Diffusion functional MRI reveals global brain network functional abnormalities driven by targeted local activity in a neuropsychiatric disease mouse model. NeuroImage, 223, 117318. https://doi.org/10.1016/j.neuroimage.2020.117318
  2. Girard B, Lienard J, Gutierrez CE, Delord B, Doya K (2020). A biologically constrained spiking neural network model of the primate basal ganglia with overlapping pathways exhibits action selection. European Journal of Neuroscience. https://doi.org/10.1111/ejn.14869
  3. Gutierrez CE, Skibbe H, Nakae K, Tsukada H, Lienard J, Watakabe A, Hata J, Reisert M, Woodward A, Yamaguchi Y, Yamamori T, Okano H, Ishii S, Doya K (2020). Optimization and validation of diffusion MRI-based fiber tracking with neural tracer data as a reference. Scientific reports. http://doi.org/10.1038/s41598-020-78284-4
  4. Han D, Doya K, Tani J (2020). Self-organization of action hierarchy and compositionality by reinforcement learning with recurrent neural networks. Neural Networks, 129, 149-162. http://doi.org/10.1016/j.neunet.2020.06.002.
  5. Miyazaki K, Miyazaki KW, Sivori G, Yamanaka A, Tanaka KF, Doya K (2020). Serotonergic projections to the orbitofrontal and medial prefrontal cortices differentially modulate waiting for future rewards. Science Advances, 6, eabc7246. http://doi.org/10.1126/sciadv.abc7246
  6. Doya K, Miyazaki KW, Miyazaki K,  (2021). Serotonergic modulation of cognitive computations Current Opinion in Behavioral Sciences, 38, 116-123. https://doi.org/10.1016/j.cobeha.2021.02.003
  7. Takahashi H, Yamashita Y, Doya K (2020). AIと脳神経科学―精神神経疾患へのデータ駆動と理論駆動のアプローチ. Clinical Neuroscience, 38, 1358-1363.

4.2 Books and other one-time publications

Doya K (in press). Reinforcement learning. in Ron Sun (ed.), The Cambridge Handbook on Computational Cognitive Sciences. Cambridge University Press.

4.3 Oral and Poster Presentations

Invited Lecture/Seminar

  1. Doya K, Kano E (2021). The art of science and science of art. Venture Cafe Tokyo: STEAM for Innovation #1. Online.
  2. Doya K (2021). Neural circuit for mental simulation. China-Japan Expert Symposium on Brain Science. Online.
  3. Doya K (2021). ロボット作りから脳科学へ：探究の楽しみ方. 沖縄県立向陽高校（SSH指定校）年間プロジェクト“探求プレゼンテーション”基調講演.
  4. Doya K (2020). 人工知能は脳から何を学べば良いのか. 応用脳科学アカデミーアドバンスコース「脳とAI」第１回. 東京.
  5. Doya K (2020). AI and brain science. 2020 IEEE CIS Summer School on Emerging Research Trends in Computational Intelligence: Theory and Applications. Online.
  6. Doya K (2020). Communication and self-organization of intelligent agents. NOLTA 2020. Online.
  7. Doya K (2020). 脳内シミュレーションの神経機構. 東京大学医学部 機能生物学セミナー. Online.
  8. Doya K (2020). 人工知能と脳科学の融合と社会. 第8回神経法学研究会. Online.
  9. Doya K (2020). 脳とAIの接点から何を学びうるのか. 第5回全脳アーキテクチャシンポジウム. Online.
  10. Doya K (2020). Toward the society of AI agents: what should we learn from the brain and human society. International Symposium on Artificial Intelligence and Brain Science. Online.
  11. Doya K (2020). Neural circuits for mental simulation. NeuFo Monday Seminar, University of Geneva. Online.
  12. Doya K (2020). What can we further learn from the brain for artificial intelligence? Neurotheory Forum. Online.
  13. Doya K (2020). How to let robots learn, develop, communicate and evolve. Latin American Summer School on Cognitive Robotics (LACORO). Online.
  14. Doya K (2020). What can we further learn from the brain for artificial intelligence. Neuroscience2020. Online.
  15. Doya K (2020). Toward multi-scale brain data assimilation. CNS*2020 Workshop: Machine learning and mechanistic modeling for understanding brain in health and disease. Online.
  16. Doya K (2020). Neural implementation of reinforcement learning. Virtual Seminar at DeepMind Paris.
  17. Li Y, Doya K (2020). Extracting information flow across cortical layers from multi-depth two-photon imaging data. 第63回自動制御連合講演会-講演論文集, 365-367, 第63回自動制御連合講演会. Online, http://doi.org/10.11511/jacc.63.0_365

Conference Oral/Poster presentations

  1. Gutierrez CE, Sun Z, Yamaura H, Morteza H, Igarashi J, Yamazaki T, Doya K (2020). Simulation of resting-state neural activity in a loop circuit of the cerebral cortex, basal ganglia, cerebellum, and thalamus using NEST simulator. Online, The 30th Annual Conference of Japanese Neural Network Society (JNNS2020).
  2. Gutierrez C, Lienard J, Girard B, Doya K (2020). Reinforcement learning by a biologically constrained spiking neural model of the basal ganglia. Virtual, Taiwan 1st Asia-Pacific Computational and Cognitive Neuroscience (2020 AP-CCN) Conference 
  3. Gutierrez CE, Skibbe H, Tsukada H, Nakae K, Ishii S, Doya K (2020). Data-driven modeling of the basal ganglia by a pipeline for generation of spiking neural networks. Virtual Conference (Kobe, Japan) The 43rd Annual Meeting of the Japan Neuroscience Society (Neuroscience 2020).
  4. Hikishima-Kasahara K, Doya K (2020). Changes in the basal ganglia-thalamic functional connectivity induced by longitudinal motor training in mice Online, Neuroscience 2020.
  5. Shimizu Y, Yoshimoto J, Takamura M, Okada G, Matsumoto T, Fuchikami M, Okada S, Morinobu S, Okamoto Y, Yamawaki S, Doya K (2020). Maximum credibility voting (MCV): An integrative approach for accurate diagnosis of major depressive disorder from clinically readily available data. Online (NewZealand), APSIPA 2020 online.
  6. Sugiura I, Irei T, Doya K, Kurata K, Miyata R (2020). Effects of the neural activity in basal ganglia on the choice behavior in rats Online, The 30th Annual Conference of Japanese Neural Network Society (JNNS2020).
  7. Vieillard N, Kozuno T, Scherrer B, Pietquin O, Munos R, Geist M (2020). Leverage the average: an analysis of KL regularization in reinforcement learning. Neural Information Processing Systems Conference 2020 (NeurIPS 2020). Online.

5. Intellectual Property Rights and Other Specific Achievements

  1. Uchibe E, Doya K (2021). Inverse reinforcement learning by density ratio estimation-US Patent 10,896,382 E Uchibe, K Doya.
  2. Uchibe E, Doya K (2021). Direct inverse reinforcement learning with density ratio estimation. US Patent 10,896,383.

6. Meetings and Events

6.1 The 8th Research Area Meeting "Artificial Intelligence and Brain Science"

  • Date: June 15-19, 2020
  • Venue: Online (Zoom)
  • Sponsor: Kakenhi Project on Artificial Intelligence and Brain Science

6.2 International Symposium on Artificial Intelligence and Brain Science

  • Date: October 10-12, 2020
  • Venue: Online
  • Speakers:
    • Josh Tenenbaum (Massachusetts Institute of Technology)
    • Yann LeCun (Facebook AI Research & New York University)
    • Yutaka Matsuo (The University of Tokyo)
    • Doina Precup (McGill University)
    • David Silver (DeepMind)
    • Masashi Sugiyama (RIKEN Center for Advanced Intelligence Project
      The University of Tokyo International Research Center for Neurointelligence)
    • Ila Fiete (Massachusetts Institute of Technology)
    • Karl Friston (University College London)
    • Yukie Nagai (The University of Tokyo International Research Center for Neurointelligence)
    • Maneesh Sahani (Gatsby Computational Neuroscience Unit)
    • Tadahiro Taniguchi (Ritsumeikan University)
    • Matthew Botvinick (DeepMind)
    • Ryota Kanai (ARAYA Inc.)
    • Angela Langdon (Princeton University)
    • Hiroyuki Nakahara (RIKEN Center for Brain Science)
    • Xiao-Jing Wang (New York University)
    • James J. DiCarlo (Massachusetts Institute of Technology)
    • Yukiyasu Kamitani (Kyoto University & ATR)
    • Rosalyn Moran (Department of Neuroimaging, IOPPN, King’s College London)
    • Terrence Sejnowski (Salk Institute & Univesity of California San Diego)
    • Hidehiko Takahashi (Tokyo Medical and Dental University)
    • Anne Churchland (niversity of California, Los Angeles)
    • Kenji Doya (Okinawa Institute of Science and Technology)
    • Arisa Ema (The University of Tokyo Institute for Future Initiatives)
    • Hiroaki Kitano (Okinawa Institute of Science and Technology)
    • Stuart Russell (University of California, Berkeley)

6.3 The 9th Research Area Meeting "Artificial Intelligence and Brain Science"

  • Date: October 13-14, 2020
  • Venue: Online (Teams)
  • Sponsor: Kakenhi Project on Artificial Intelligence and Brain Science

6.4 The 10th Research Area Meeting "Artificial Intelligence and Brain Science"

  • Date: March 5-8, 2021
  • Venue: Online (Teams)
  • Sponsor: Kakenhi Project on Artificial Intelligence and Brain Science

7. Other

Yuzhe Li received the 2020 AP-CCN Poster Award for her presentation "Neuron hubs distributed differently in deep layers and superficial layers in different brain states" at the Asia-Pacific Computational and Cognitive Neuroscience (AP-CNN) conference.

Tadahi Kozuno's NeurIPS paper presentation was featrued in Nikkei X-Tech and Bungei Shunju.

Kenji Doya was elected as a board member of International Neural Network Society and appointed as the co-chair of the Data Standards and Sharing Working Group of International Brain Initiatives.