FY2022 Annual Report

Neural Computation Unit
Professor Kenji Doya

Abstract

The Neural Computation Unit pursues the dual goals of developing robust and flexible learning algorithms and elucidating the brain’s mechanisms for robust and flexible learning. Our specific focus is on how the brain realizes reinforcement learning, in which an agent, biological or artificial, learns novel behaviors in uncertain environments by exploration and reward feedback. We combine top-down, computational approaches and bottom-up, neurobiological approaches to achieve these goals.

In FY2022, we continued research along three major externally funded projects: Kakenhi project on Artificial Intelligence and Brain Science, Brain/MINDS project, and Fugaku Supercomputing project for whole-brain simulation.

With the extension of Kakenhi Project on Artificial Intelligence and Brain Science due to the COVID pandemic, we organized the Internaltional Sumposium on Artificial Intelligence and Brain Science 2022 in OIST as a satellite event for Neuro2022 in Okinawa Convention Center.

Our pproposal of a Transformative Research Project on A Unified Theory of Predction and Action was accepted for FY2022-2026.

In the Brain/MINDS project, we analyzed calcium imaging data collected by Matsuzaki lab in the University of Tokyo and RIKEN Center for Brain Science. We released OptiNiSt, the optical neuroimage studio, in GitHub and DockerHub.

Under the Program for Promoting Researches on the Supercomputer Fugaku, we collaborated with researchers of EU Human Brain Project to perform combined simulation of the whole-brain spking neural network and the body dynamics using the NEST simulator and the neurorobotics simulation platform.

1. Staff

Systems Neurobiology Group

  • Katsuhiko Miyazaki, Senior Staff Scientist
  • Kayoko Miyazaki, Senior Staff Scientist
  • Anupama Chaudhary, Technician
  • Hajime Yamanaka, Technician
  • Sergey Zobnin, OIST Student
  • Miles Desforges, OIST Student
  • Yuma Kajihara, OIST Student
  • Jianning Chen, OIST Student
  • Naohiro Yamauchi, OIST Student
  • Tomohiko Yoshizawa, Visiting Researcher (Hokkaido University)
  • Kazumi Kasahara, Visiting Researcher (AIST)

Dynamical Systems Group

  • Yukako Yamane, Staff Scientist
  • Soheil Keshmiri, Staff Scientist
  • Yuzhe Li, Postdoctoral Scholar
  • Razvan Gamanut, Research Fellow
  • Huriye Atilgan, Research Fellow
  • Florian Lalande, OIST Student
  • Shuhei Hara, OIST Student
  • Hideyuki Yoshimura, OIST Student
  • Hiroaki Hamada, Visiting Researcher (ARAYA)
  • Hiromichi Tsukada, Visiting Researcher (Chubu University)
  • Carlos Enrique Gutierrez, Visiting Researcher (Softbank)

Adaptive Systems Group

  • Christopher Buckley, Technician
  • Qiong Huang, OIST Student
  • Ho Ching Chiu, OIST Student
  • Kristine Faith Roque, OIST Student
  • Yuji Kanagawa, OIST Student
  • Tojoarisoa Rakotoaritina, OIST Student
  • Farzana Rahman, Visiting Researcher (Independent University)

Research Unit Administrators

  • Emiko Asato
  • Kikuko Matsuo
  • Misuzu Saito

2. Collaborations

  • Dr. Takuya Isomura at RIKEN CBS and other collaborators in the Unified Theory project.
  • Prof. Mark Walton and Dr. Thomas Akam at the University of Oxford for analyzing model-free and model-based decision making in mice.
  • Prof. Masanori Matsuzaki and Dr. Teppei Ebina at the University of Tokyo and RIKEN Center for Brain Science for the analysis of the calcium imaging data of the marmoset brain under Brain/MINDS project.
  • Prof. Hans Ekkehard Plesser for acceleration of NEST3 spiking neural network simulation on Fugaku.
  • Prof. Fabrice Morin at HBP Neurorobotics Platform for combined simulation of our whole-brain model by NEST3 and realistic musculoskeletal systems.
  • Our alumni Dr. Dongqi Han at Microsoft Shang Hai
  • An industrial collaboratory research project on brian-like computation.
  • An industrial collaboratory research project on the analysis and modeling of health-related data from wearable devices.

3. Activities and Findings

3.1 Nuerobiology Experiments [Systems Neurobilogy Group]

3.1.1 Neural substrate of dynamic Bayesian inference [Kakenhi Project on AI and Brain Science]

We continued our experiments to clarify how predictive information from actions and sensory information through environmental interaction are integrated across different layers of the somatosensory and motor cortical circuits in mice. Our analysis of calcium imaging data through a prism lens implanted in the somatosensory cortex revealed neural activities related to prior prediction of sensory signal and the error between the predicted and actual outcome.

We proposed a Transformative Research Project on A Unified Theory of Prediction and Action, partly based on the theory of the duality of inference and control. The proposal was accepted for FY2023-2027 with a young leadership of Dr. Takuya Isomura at RIKEN CBS.

3.1.2 The role of serotonin in the regulation of patience [Moonshot Goal 9] 

Under the newly funded Moonshot Program, we started extensive experiments on the roles of serotonin and dopamine in motor actions and patient waiting to obtain rewards and to avoid punishments. We set up new experimental tools for fiber-photometry and multi-color endoscopic imaging.

3.2. Neural Data Analysis and Modeling [Dynamical Systems Group]

3.2.1 Analysis and modeling of marmoset brain data [Brain/MINDS Project]

We analyze calcium imaging data collected by Matsuzaki lab in the University of Tokyo and RIKEN Center for Brain Science. For a wide-field imaging data spanning from the parietal cortex to the premotor cortex, we extracted tens of spatially localized components by non-negative matrix factorization (NMF) and revealed causal relationships among them by embedding entropy (EE).

We developed a new software OptiNiSt, the optical neuroimage studio, which allows intuitive construction of data processing pipeline by graphic user interface and large-scale processing on PC clusters. The open-source software is available from GitHub and DockerHub. 

3.2.2 Spiking neural network model of the basal ganglia [Fugaku Project]

We published the paper on SNNbuilder, a software platform to register anatomical and physiological data and produce simulation codes in NEST. We developed a biologically realistic model of the basal ganglia circuit using SNNbuilder and proved its usability by simulation on Fugaku supercomputer.

3.3 Robotics and Reinforcement Learning [Adaptive Systems Group]

3.3.1 Data-efficient reinforcement learning [Kakenhi Project on AI and Brain Science]

In collaboration with Dr. Dongqi Han formerly at Tani Unit, we completed a paper on how model-free and model-based learning and control can help each other.

3.3.2 Smartphone robot platform [Kakenhi Project on AI and Brain Science]

We designed a brand new electronics for the smartphone robot with improved energy management. Using the present generation of robots, we implemented survival and reproduction behaviors.

4. Publications

4.1 Journals

  1. Abekawa N, Doya K, Gomi H (2023). Body and visual instabilities functionally modulate implicit reaching corrections. iSience, 26. https://doi.org/10.1016/j.isci.2022.105751
  2. Takata Y, Yamanaka H, Nakagawa H, Takada M (2023). Morphological changes of large layer V pyramidal neurons in cortical motor-related areas after spinal cord injury in macaque monkeys. Scientific Reports. 13, 1, 82. https://doi.org/10.1038/s41598-022-26931-3
  3. Lalande F, Trani A.A (2022) Predicting the stability of hierarchical triple systems with convolutional neural networks. The Astrophysical Journal. 938, 18. https://doi.org/10.3847/1538-4357/ac8eab
  4. 塚田啓道, 銅谷賢治 (2022) 神経トレーサー、構造MRI、機能MRIデータの統合による全脳モデルシミュレーション. 生体の科学. 73, 5, 436-437.
  5. 浦久保秀俊, 渡我部昭哉, 中江健, 石井信, 銅谷賢治 (2022) コネクトーム : ミクロ・メゾ・マクロレベルの新展開. 解剖学雑誌. 97, 2, 41-44.
  6. Doya K, Friston K, Sugiyama M, Tenenbaum J (2022) Neural Networks special issue on Artificial Intelligence and Brain Science. Neural Networks. 155, 328-329. https://doi.org/10.1016/j.neunet.2022.08.018
  7. Gutierrez E. C, Skibbe H, Musset H, Doya K (2022) A Spiking Neural Network Builder for Systematic Data-to-Model Workflow. Frontiers. 16. https://doi.org/10.3389/fninf.2022.855765
  8. Kimura K, Kodama A, Yamane Y, Sakai K (2022) Figure-ground responsive fields of monkey V4 neurons estimated from natural image patches. Plos One. 17, 6 ,e0268650. https://doi.org/10.1371/journal.pone.0268650
  9. Doya K, Ema A, Kitano H, Sakagami M, Russell S (2022) Social impact and governance of AI and neurotechnologies.152, 542-554. https://doi.org/10.1016/j.neunet.2022.05.012
  10. Feldotto B, Eppler JM, Jimenez-Romero C, Bignamini C, Gutierrez CE, Albanese U, Retamino E, Vorobev V,  Zolfaghari V, Upton A, Sun Z, Yamaura H, Heidarinejad M, Klijn W, Morrison A, Cruz F, McMurtrie C, Knoll AC, Igarashi J, Yamazaki T, Doya K, Morin FO (2022) Deploying and Optimizing Embodied Simulations of Large-Scale Spiking Neural Networks on HPC Infrastructure. Frontiers. 16. https://doi.org/10.3389/fninf.2022.884180
  11. Ito J,  Joana C, Yamane Y, Fujita I, Tamura H, Maldonado P. E , Grün S (2022) Latency shortening with enhanced sparseness and responsiveness in V1 during active visual sensing. Scientific Reports. 12, 1, 6021. https://doi.org/10.1038/s41598-022-26931-3

4.2 Books and other one-time publications

  1. Doya K (2023). Reinforcement learning. Sun R (ed.): The Cambridge Handbook of Computational Cognitive Sciences, 350-370, Cambridge University Press. https://doi.org/10.1017/9781108755610.013
  2. Doya K (2023). Computational cognitive models of reinforcement learning. Sun R (ed.): The Cambridge Handbook of Computational Cognitive Sciences, 739 - 766, Cambridge University Press. https://doi.org/10.1017/9781108755610.026

4.3 Oral and Poster Presentations

Invited Lecture/Seminar

  1. 銅谷賢治 (2022). 「人工知能美学芸術展：美意識のハードプロブレム」に出展作品についての小講演. 第40回AI美芸研「人工知能美学芸術展：美意識のハードプロブレム」全体報告. 東京都, 日本.
  2. Doya K(2022). Serotonin and model-based decision making. the Reward and Decision-Making meeting. Lake Arrowhead, USA.
  3. 銅谷賢治 (2022). 人工知能と脳科学. OIST 10th anniversary. 東京都, 日本.
  4. 銅谷賢治 (2022). 脳データ共有への期待と課題：International Brain Initiativeの試み. 革新脳・国際脳合同シンポジウム. 東京都, 日本.
  5. 銅谷賢治 (2022). 人工知能は脳から何を学べば良いのか. 応用脳科学コンソーシアム. Online.
  6. 銅谷賢治 (2022). Data-Driven and Theory-Driven Approaches in Neuroscience. 2nd Taiwan Society for Neuroscience Meeting (TSfN). Online.
  7. 銅谷賢治 (2023). 人工知能と脳の作動原理. 第52回日本臨床神経生理学会. 京都府, 日本.
  8. Doya K (2022). What is takes to create a humanoid. Humanoids2022. Okinawa. Japan
  9. Doya K (2023). Neural Circuits for Reinforcement Learning and Mental Simulation. The Taiwan Society of Cognitive Neuroscience Meeting (TSCN). Taiwan.
  10. 銅谷賢治 (2023). 人工知能・脳科学・法. 第22回神経法学研究会.沖縄県, 日本.

Conference Oral/Poster presentations

  1. Han D, Kozuno T, Luo XC, Zhao Y, Doya K, Yang Y, Li D (2022). Variational oracle guiding for reinforcement learning. International Conference on Learning Representations (ICLR2022). Online.
  2. 宮崎勝彦, 宮崎佳代子, 銅谷賢治 (2022). セロトニンによる報酬待機行動の制御機構.日本科学振興協会　第1回総会・キックオフミーティング. 東京, 日本.
  3. Yamane Y, Li Y, Akiyama S, Saeki S, Kuwabara M, Hasui S, Kanai R, Gutierrez CE, Doya K (2022). Optical Neuroimage Studio (OptiNiSt): GUI-based, extensible, scalable framework for neural image data analysis. The 45th Annual Meeting of the Japan Neuroscience Society (Neuron 2022). Okinawa, Japan.
  4. Li Y, Doya K (2022). Dual Bayesian PCA for Factor Analysis on Calcium imaging data. The 45th Annual Meeting of the Japan Neuroscience Society (Neuron 2022). Okinawa, Japan.
  5. Yoshimura H, Doya K, Yamazaki T (2022) A New Formulation of Temporal Difference Error in Reinforcement Learning for Spiking Neural Networks. The 45th Annual Meeting of the Japan Neuroscience Society (Neuron 2022). Okinawa, Japan.
  6.  Desforges M, Flotho P, Kuhn B, Doya K (2022). Simultaneous recording of neuromodulator and calcium spatiotemporal activity reveals state dependent differences between dopaminergic, noradrenergic and serotonergic cortical activity. Neuroscience 2022 (SfN 2022) . San Diego.
  7. Lalande F (2022). Numerical data imputation: choose kNN over deep learning. SISAP 2022. Bologna, Italy.
  8. Miyazaki Katsuhiko (2023). Serotonin mechanism for regulating reward waiting behavior. The 100th Anniversary Annual Meeting of The Physiological Society of Japan. Kyoto, Japan.
  9. Keshmiri S (2023). Identification of the Phenotypic Markers of Human Heart-Rate (HR) Through the Study of Its Circadian Dynamics. The 22nd Winter Workshop on Mechanism of Brain and Mind. Rusutsu, Hokkaido.
  10. Desforges M, Flotho P, Kuhn B, Doya K (2023). Two-photon imaging of extracellular neuromodulator activity reveals spatiotemporal state dependent differences between dopaminergic, noradrenergic and serotonergic systems. The 22nd Winter Workshop on Mechanism of Brain and Mind. Rusutsu, Hokkaido.
  11. Yamane Y, Ebina T, Sasagawa A, Terada S, Uemura M, Ohki K, Matsuzaki M, Doya K (2023). Causality analysis by embedding entropy across cortical areas of marmoset during upper-limb movement tasks. The 22nd Winter Workshop on Mechanism of Brain and Mind. Rusutsu, Hokkaido.
  12. Li Y, Doya K  (2023). Neural connectivity among different layers changes at different brain states. The 22nd Winter Workshop on Mechanism of Brain and Mind. Rusutsu, Hokkaido.
  13. Tomonaga S  (2023). Canonical Correlation Analysis with Fused-Lasso for Identifying Common Causes of Heart Rate Variability and Phenotypic Information. The 22nd Winter Workshop on Mechanism of Brain and Mind. Rusutsu, Hokkaido.

5. Intellectual Property Rights and Other Specific Achievements

       Nothing to report

6. Meetings and Events

6.1 Seminars

Linking Complex Behaviour to High-dimensional Neural Representations

  • Date: Thursday, August 25, 2022
  • Venue: OIST Campus Lab1 
  • Speaker: Prof. N Alex Cayco Gajic (LNC2, École Normale Supérieure)

Updating internal models for reward-based sequential choice decisions in the rat cingulate-motor circuits

  • Date: September 28, 2022
  • Venue: OIST Campus Lab1 
  • Speaker: Dr. Daigo Takeuchi
                   (The University of Tokyo School of Medicine, Department of Physiology)

Autodiagnosis and the Dynamical Emergence Theory of Basic Consciousness

  • Date: December 13, 2022
  • Venue: OIST Campus Lab1 
  • Speaker:  Prof. Shimon Edelman (Department of Psychology Cornell University)

Conscience, Conditional Cooperation, and the Prospects of Surviving Capitalism

  • Date: December 15, 2022
  • Venue: OIST Campus Lab1 
  • Speaker:  Prof. Shimon Edelman (Department of Psychology Cornell University)

Stable adaptation and learning

  • Date: January 12, 2023
  • Venue: OIST Campus Lab1 
  • Speaker: Prof. Jean-Jacques Slotine (Massachusetts Institute of Technology)

Contraction analysis of convergence and synchronization

  • Date: January 16, 2023
  • Venue: OIST Campus Lab1 
  • Speaker: Prof. Jean-Jacques Slotine (Massachusetts Institute of Technology)

6.2 Events

International Symposium on Artificial Intelligence and Brain Science 2022

  • Date: July 4-5, 2022
  • Venue: OIST conference center (Auditorium) & Online
  • Speakers:
    • Maneesh Sahani (Gatsby Computational Neuroscience Unit)
    • Angela Langdon (National Institute of Mental Health / NIH)
    • Xiao-Jing Wang (New York University)
    • Terrence Sejnowski (Salk Institute & Univesity of California San Diego)
    • Tetsuya Ogata (Waseda University)
    • Tom Macpherson (Osaka University)
    • Jun Izawa (Tsukuba University)
    • Rieko Osu (Waseda University)
    • Jun Tani (Okinawa Institute of Science and Technology)
    • Masashi Sugiyama (RIKEN Center for Advanced Intelligence Project
      The University of Tokyo International Research Center for Neurointelligence)
    • Yutaka Matsuo (The University of Tokyo)
    • Aida Nematzadeh (Deepmind)
    • Ryota Kanai (ARAYA Inc.)
    • Patricia Churchland (Salk Institute University of California San Diego)
    • Hideaki Shimazaki (Hokkaido University)
    • Keisuke Suzuki (Hokkaido University)
    • Makiko Yamada (National Institutes for Quantum Science and Technology)
    • Tadahiro Taniguchi (Ritsumeikan University)
    • Misako Komatsu (Tokyo Institute of Technology)
    • Karl Friston (University College London)
    • Yuichi Yamashita (National Center of Neurology and Psychiatry)
    • Matthew Botvinick (DeepMind)

OIST / Humanoids 2022 joint workshop

  • Date: December 1, 2022
  • Venue: OIST conference center (Auditorium)
  • URL: https://www.humanoids2022.org/program/oisthumanoids2022-joint-workshop
  • Sponsor: Kakenhi Project on Artificial Intelligence and Brain Science

Neural Computation Workshop 2022

  • Date: December 17, 2022
  • Venue: OIST seaside house
  • URL: https://groups.oist.jp/ncu/event/neural-computation-workshop-2022
  • Sponsor: Kakenhi Project on Artificial Intelligence and Brain Science

Mechanism of Brain and Mind The 22 Winter Workshop

  • Date: January 1-7, 2023
  • Venue: Rusutsu Resort Hotel & Convention, Hokkaido, Japan and Online (Webinar)
  • URL: https://brainmind.jnns.org/en/eng-wt2023/
  • Sponsor: Kakenhi Project on Artificial Intelligence and Brain Science

7. Other

Katsuhiko Miyazaki was appointed as a Project Manager for the Moonshot Project Gaol 9: Realization of a mentally healthy and dynamic society by increasing peace of mind and vitality by 2050.

Kenji Doya served the President of The 45th Annual Meeting of the Japan Neuroscience Society (Neuro2022) held for the first time in Okinawa. He was appointed as the President of the Japanese Neural Network Society from March 2023.