FY2022 Annual Report

Neural Computation Unit
Professor Kenji Doya

Abstract

The Neural Computation Unit pursues the dual goals of developing robust and flexible learning algorithms and elucidating the brain’s mechanisms for robust and flexible learning. Our specific focus is on how the brain realizes reinforcement learning, in which an agent, biological or artificial, learns novel behaviors in uncertain environments by exploration and reward feedback. We combine top-down, computational approaches and bottom-up, neurobiological approaches to achieve these goals.

In FY2021, we continued research along three major externally funded projects: Kakenhi project on Artificial Intelligence and Brain Science, Brain/MINDS project, and Fugaku Supercomputing project for whole-brain simulation.

With the extension of Kakenhi Project on Artificial Intelligence and Brain Science due to the COVID pandemic, we edited a Special Issue of Neural Networks including perspective papers based on the sessions at the Internaltional Sumposium on Artificial Intelligence and Brain Science 2020. We further started to organize the Internaltional Sumposium on Artificial Intelligence and Brain Science 2022 in OIST as a satellite event for Neuro2022 in Okinawa Convention Center.

In the Brain/MINDS project, we started to analyze calcium imaging data collected by Matsuzaki lab in the University of Tokyo and RIKEN Center for Brain Science and also started to develop 'OptiNiSt', an optical neuroimage analysis platform.

Under the Program for Promoting Researches on the Supercomputer Fugaku, we collaborated with researchers of EU Human Brain Project to accelerate the spking neural network simulator NESTs and its connection with the neurorobotics simulation platform.

1. Staff

Systems Neurobiology Group

Katsuhiko Miyazaki, Senior Staff Scientist
Kayoko Miyazaki, Senior Staff Scientist
Yukako Yamane, Staff Scientist
Yuzhe Li, Postdoctoral Scholar
Anupama Chaudhary, Technician
Hajime Yamanaka, Technician
Sergey Zobnin, OIST Student
Miles Desforges, OIST Student
Tomohiko Yoshizawa, Visiting Researcher (Hokkaido University)
Kazumi Kasahara, Visiting Researcher (AIST)

Dynamical Systems Group

Razvan Gamanut, Research Fellow
Huriye Atilgan, Research Fellow
Soheil Keshmiri, Staff Scientist
Florian Lalande, OIST Student
Shuhei Hara, OIST Student
Hideyuki Yoshimura, OIST Student
Hiroaki Hamada, Visiting Researcher (ARAYA)
Hiromichi Tsukada, Visiting Researcher (Chubu University)
Carlos Enrique Gutierrez, Visiting Researcher (Softbank)

Adaptive Systems Group

Christopher Buckley, Technician
Qiong Huang, OIST Student
Ho Ching Chiu, OIST Student
Kristine Faith Roque, OIST Student
Farzana Rahman, Visiting Researcher (Independent University)

Research Unit Administrators

Emiko Asato
Kikuko Matsuo
Misuzu Saito

2. Collaborations

Prof. Mark Walton and Dr. Thomas Akam at the University of Oxford for analyzing model-free and model-based decision making in mice.
Prof. Masanori Matsuzaki and Dr. Teppei Ebina at the University of Tokyo and RIKEN Center for Brain Science for the analysis of the calcium imaging data of the marmoset brain under Brain/MINDS project.
Prof. Hans Ekkehard Plesser in HBP EXABRAINPREP Boucher Program for acceleration of NEST3 spiking neural network simulation on Fugaku.
Prof. Fabrice Morin at HBP Neurorobotics Platform for combined simulation of our whole-brain model by NEST3 and realistic musculoskeletal systems.
An industrial collaboratory research project on brian-like computation.
An industrial collaboratory research project on the analysis and modeling of health-related data from wearable devices.

3. Activities and Findings

3.1 Nuerobiology Experiments [Systems Neurobilogy Group]

3.1.1 Neural substrate of dynamic Bayesian inference [Kakenhi Project on AI and Brain Science]

We continued our experiments to clarify how predictive information from actions and sensory information through environmental interaction are integrated across different layers of the somatosensory and motor cortical circuits in mice. Our preliminary analysis of calcium imaging data through a prism lens implanted in the somatosensory cortex revealed context-dependent modulation of preparatory and sensory activities.

3.1.2 The role of serotonin in the regulation of patience [Kakenhi Project on AI and Brain Science]

To test the hypothesis that serotonin regulated the balance between model-free and model-based decision making, we implemented a multi-step reinforcement learning task for mice and found that optogenetic inhibition of serotonin neurons in the dorsal raphe nucleus decreases the weight of model-based component.

3.2. Neural Data Analysis and Modeling [Dynamical Systems Group]

3.2.1 Analysis and modeling of marmoset brain data [Brain/MINDS Project]

We started to analyze calcium imaging data collected by Matsuzaki lab in the University of Tokyo and RIKEN Center for Brain Science. For a wide-field imaging data spanning from the parietal cortex to the premotor cortex, we extracted tens of spatially localized components by non-negative matrix factorization (NMF) and revealed causal relationships among them by convergence cross mapping (CCM).

We also started to develop 'OptiNiSt', an optical neuroimage analysis platform.

3.2.2 Spiking neural network model of the basal ganglia [Fugaku Project]

We developed SNNbuilder, a softward platform to register anatomical and physiological data and produce simulation codes in NEST. We developed a biologically realistic model of the basal ganglia circuit using SNNbuilder and tested its behavoir by simulation on Fugaku supercomputer.

3.3 Robotics and Reinforcement Learning [Adaptive Systems Group]

3.3.1 Data-efficient reinforcement learning [Kakenhi Project on AI and Brain Science]

Our paper of model-free imitation learning by forward and inverse reinforcement learning was published in Neural Networks. In collaboration with Tani unit, our paper on 'oracle guidance' reinforcement learning which utilizes retrospective knowledge was accepted in ICLR2022 conference.

3.3.2 Smartphone robot platform [Kakenhi Project on AI and Brain Science]

Our smartphone robots were presented in the AI Art and Aesthetics Exhibition in Nakagawa Village. We started desining new robot hardware with improved energy management.

Smartphone Robots

4. Publications

4.1 Journals

Abekawa N, Doya K, Gomi H (2023). Body and visual instabilities functionally modulate implicit reaching corrections. iSience, 26. https://doi.org/10.1016/j.isci.2022.105751
Takata Y, Yamanaka H, Nakagawa H, Takada M (2023). Morphological changes of large layer V pyramidal neurons in cortical motor-related areas after spinal cord injury in macaque monkeys. Scientific Reports. 13, 1, 82. https://doi.org/10.1038/s41598-022-26931-3
Lalande F, Trani A.A (2022) Predicting the stability of hierarchical triple systems with convolutional neural networks. The Astrophysical Journal. 938, 18. https://doi.org/10.3847/1538-4357/ac8eab
塚田啓道, 銅谷賢治 (2022) 神経トレーサー、構造MRI、機能MRIデータの統合による全脳モデルシミュレーション. 生体の科学. 73, 5, 436-437.
浦久保秀俊, 渡我部昭哉, 中江健, 石井信, 銅谷賢治 (2022) コネクトーム : ミクロ・メゾ・マクロレベルの新展開. 解剖学雑誌. 97, 2, 41-44.
Doya K (Guest Editors), Friston K, Sugiyama M, Tenenbaum J (2022) Neural Networks special issue on Artificial Intelligence and Brain Science. Neural Networks. 155, 328-329. https://doi.org/10.1016/j.neunet.2022.08.018
Gutierrez E. C, Skibbe H, Musset H, Doya K (2022) A Spiking Neural Network Builder for Systematic Data-to-Model Workflow. Frontiers. 16. https://doi.org/10.3389/fninf.2022.855765
Kimura K, Kodama A, Yamane Y, Sakai K (2022) Figure-ground responsive fields of monkey V4 neurons estimated from natural image patches. Plos One. 17, 6 ,e0268650. https://doi.org/10.1371/journal.pone.0268650
Doya K, Ema A, Kitano H, Sakagami M, Russell S (2022) Social impact and governance of AI and neurotechnologies.152, 542-554. https://doi.org/10.1016/j.neunet.2022.05.012
Feldotto B, Eppler JM, Jimenez-Romero C, Bignamini C, Gutierrez CE, Albanese U, Retamino E, Vorobev V, Zolfaghari V, Upton A, Sun Z, Yamaura H, Heidarinejad M, Klijn W, Morrison A, Cruz F, McMurtrie C, Knoll AC, Igarashi J, Yamazaki T, Doya K, Morin FO (2022) Deploying and Optimizing Embodied Simulations of Large-Scale Spiking Neural Networks on HPC Infrastructure. Frontiers. 16. https://doi.org/10.3389/fninf.2022.884180
Ito J, Joana C, Yamane Y, Fujita I, Tamura H, Maldonado P. E , Grün S (2022) Latency shortening with enhanced sparseness and responsiveness in V1 during active visual sensing. Scientific Reports. 12, 1, 6021. https://doi.org/10.1038/s41598-022-26931-3

4.2 Books and other one-time publications

Nothing to report

4.3 Oral and Poster Presentations

Invited Lecture/Seminar

銅谷賢治 (2022). 「人工知能美学芸術展：美意識のハードプロブレム」に出展作品についての小講演. 第40回AI美芸研「人工知能美学芸術展：美意識のハードプロブレム」全体報告. 東京都, 日本.
Doya K(2022). Serotonin and model-based decision making. the Reward and Decision-Making meeting. Lake Arrowhead, USA.
銅谷賢治 (2022). 人工知能と脳科学. OIST 10th anniversary. 東京都, 日本.
銅谷賢治 (2022). 脳データ共有への期待と課題：International Brain Initiativeの試み. 革新脳・国際脳合同シンポジウム. 東京都, 日本.
銅谷賢治 (2022). 人工知能は脳から何を学べば良いのか. 応用脳科学コンソーシアム. Online.
銅谷賢治 (2022). Data-Driven and Theory-Driven Approaches in Neuroscience. 2nd Taiwan Society for Neuroscience Meeting. Online.
銅谷賢治 (2023). 人工知能と脳の作動原理. 第52回日本臨床神経生理学会. 京都府, 日本.
Doya K (2022). What is takes to create a humanoid. Humanoids2022. Okinawa. Japan
Doya K (2023). Neural Circuits for Reinforcement Learning and Mental Simulation. The Taiwan Society of Cognitive Neuroscience Meeting (TSCN). Taiwan.
銅谷賢治 (2023). 人工知能・脳科学・法. 第22回神経法学研究会.沖縄県, 日本.

Conference Oral/Poster presentations

Han D, Kozuno T, Luo XC, Zhao Y, Doya K, Yang Y, Li D (2022). Variational oracle guiding for reinforcement learning. International Conference on Learning Representations (ICLR2022). Online.
宮崎勝彦, 宮崎佳代子, 銅谷賢治 (2022). セロトニンによる報酬待機行動の制御機構.日本科学振興協会　第1回総会・キックオフミーティング. 東京, 日本.
Yamane Y, Li Y, Akiyama S, Saeki S, Kuwabara M, Hasui S, Kanai R, Gutierrez CE, Doya K (2022). Optical Neuroimage Studio (OptiNiSt): GUI-based, extensible, scalable framework for neural image data analysis. The 45th Annual Meeting of the Japan Neuroscience Society (Neuron 2022). Okinawa, Japan.
Li Y, Doya K (2022). Dual Bayesian PCA for Factor Analysis on Calcium imaging data. The 45th Annual Meeting of the Japan Neuroscience Society (Neuron 2022). Okinawa, Japan.
Yoshimura H, Doya K, Yamazaki T (2022) A New Formulation of Temporal Difference Error in Reinforcement Learning for Spiking Neural Networks. The 45th Annual Meeting of the Japan Neuroscience Society (Neuron 2022). Okinawa, Japan.
Desforges M, Flotho P, Kuhn B, Doya K (2022). Simultaneous recording of neuromodulator and calcium spatiotemporal activity reveals state dependent differences between dopaminergic, noradrenergic and serotonergic cortical activity. Neuroscience 2022 (SfN 2022) . San Diego.
Lalande F (2022). Numerical data imputation: choose kNN over deep learning. SISAP 2022. Bologna, Italy.
Miyazaki Katsuhiko (2023). Serotonin mechanism for regulating reward waiting behavior. The 100th Anniversary Annual Meeting of The Physiological Society of Japan. Kyoto, Japan.
Keshmiri S (2023). Identification of the Phenotypic Markers of Human Heart-Rate (HR) Through the Study of Its Circadian Dynamics. The 22nd Winter Workshop on Mechanism of Brain and Mind. Rusutsu, Hokkaido.
Desforges M, Flotho P, Kuhn B, Doya K (2023). Two-photon imaging of extracellular neuromodulator activity reveals spatiotemporal state dependent differences between dopaminergic, noradrenergic and serotonergic systems. The 22nd Winter Workshop on Mechanism of Brain and Mind. Rusutsu, Hokkaido.
Yamane Y, Ebina T, Sasagawa A, Terada S, Uemura M, Ohki K, Matsuzaki M, Doya K (2023). Causality analysis by embedding entropy across cortical areas of marmoset during upper-limb movement tasks. The 22nd Winter Workshop on Mechanism of Brain and Mind. Rusutsu, Hokkaido.
Li Y, Doya K (2023). Neural connectivity among different layers changes at different brain states. The 22nd Winter Workshop on Mechanism of Brain and Mind. Rusutsu, Hokkaido.
Tomonaga S (2023). Canonical Correlation Analysis with Fused-Lasso for Identifying Common Causes of Heart Rate Variability and Phenotypic Information. The 22nd Winter Workshop on Mechanism of Brain and Mind. Rusutsu, Hokkaido.

5. Intellectual Property Rights and Other Specific Achievements