FY2020 Annual Report

Embodied Cognitive Science Unit (ECSU)
Assistant Professor Tom FROESE

Abstract

The Embodied Cognitive Science Unit (ECSU) started its existence at OIST in September 2019. The financial year FY2020 (April 2020 - March 2021) was our first complete financial year. We continued to work on 4 key areas:

1. growing the unit by recruiting excellent postdocs and technicians, and by attracting outstanding internal and external students and visiting researchers;
2. equipping the unit with custom-built human-computer interfaces, as well as state-of-the-art physiological measurement equipment, robots, and computer workstations;
3. developing the experimental protocols for our new lines of groundbreaking research spanning individual, dyadic, and collective interaction dynamics;
4. initiating a new series of events that will attract the top talent of our field to visit OIST, the International Conference on Embodied Cognitive Science (ECogS).

1. The ECSU-Team

From April 2020 to the end of March 2021, these were members of the unit:

• Katja SANGATI, Postdoc
• Sebastien LERIQUE, Postdoc
• Maria GOHLKE, Technician
• Stephen ESTELLE, Technician
• Alexey YUDIN, Technician
• Federico SANGATI, Research Assistant
• Ivan SHPUROV, Ph.D. candidate
• Chen Lam LOH, Ph.D. Candidate
• Laura Alejandra MOJICA LOPEZ, Special Research Student
• Susana RAMIREZ VIZCAYA, Special Research Student
• Hend SAMNIYA, Ph.D. Rotation Student
• Balashwethan CHOCKALINGAM, Ph.D. Rotation Student
• Javier Andres TEJEDA MORA, Ph.D. Rotation Student
• David BIERBRAUER, Ph.D. Rotation Student
• Manisha BISWAS, Ph.D. Rotation Student
• Yi-Shan CHENG, Ph.D. Rotation Student
• Kensei KIKUCHI, Ph.D. Rotation Student
• Natalya WEBER, Ph.D. Rotation Student
• Alexander BARANSKI, Ph.D. Rotation Student
• Hideyuki YOSHIMURA, Gap Program Student
• Georgina Montserrat RESENDIZ BENHUMEA, Research Intern
• Kazuma TAKADA, Research Intern
• Brian MORRISSEY, Research Intern
• Lorena LOBO NAVAS, Visiting Researcher
• Manuel HERAS ESCRIBANO, Visiting Researcher
• Kaori YAMASHIRO, Research Unit Administrator

2. Collaborations

2.1 Sensitivity to social contingency in adolescence

• Description: The aim of this collaboration is to investigate adolescents' sensitivity to social contingency during real-time embodied social interaction, using the perceptual crossing paradigm. It is part of a more comprehensive longitudinal population study of adolescents in Belgium led by Prof. Inez MYIN-GERMEYS. The goal is to find interaction patterns that could serve as warning signals for the development of mental disorders.
• Type of collaboration: Joint research
• Key collaborators:
  • Prof. Inez MYIN-GERMEYS, Head of Contextual Psychiatry, KU Leuven, Belgium
  • Karlijn S.F.M. HERMANS, Contextual Psychiatry, KU Leuven, Belgium
  • Leonardo ZAPATA-FONSECA, Faculty of Medicine, UNAM, Mexico

2.2 Sensitivity to social contingency in schizophrenia

• Description: The aim of this collaboration is to investigate sensitivity to social contingency during real-time embodied social interaction of people with schizophrenia, using the perceptual crossing paradigm to record interaction patterns. We also draw on the latest trends in philosophy of mind to advance the theory, diagnosis, and treatment of mental health problems. More specifically, the goal is to better understand the reasons why people with schizophrenia find it challenging to engage with others in social interaction.
• Type of collaboration: Joint research
• Researchers:
  • Prof. Dr. Dr. Thomas FUCHS, Head of Department of General Psychiatry, University of Heidelberg, Germany
  • Leonardo ZAPATA-FONSECA, Faculty of Medicine, UNAM, Mexico

2.3 New approaches to the addicted mind

• Description: The aim of this collaboration is to better understand the nature of addiction by developing a theory of the self that includes habitual interaction with the world as one of its constitutive parts. By shifting the focus from alterations in brain chemistry to this more interactive perspective, we obtain a more inclusive category of addiction. To explore these insights, we are currently organizing an international roundtable on "The Addicted Self in the Age of Information Technology: Exploring the mind’s propensity for habitual and compulsive interactions", to be held at UBC.
• Type of collaboration: Joint research
• Researchers:
  • Dr. Christian SCHÜTZ, Department of Psychiatry, University of British Columbia, Canada
  • Susana RAMIREZ VIZCAYA, Institute of Philosophical Research, UNAM, Mexico

2.4 Investigating the neural dynamics of altered states of consciousness

• Description: The goal of this collaboration is to better understand the effects of psychedelic substances on the human brain by adopting a complex adaptive systems perspective, which is expected to provide novel insights into their therapeutic potential. A part of this research is conducted with animals and with human participants at the National Institute of Mental Health in Prague. It also includes field work to investigate the group-level inter-personal effects of ayahuasca ceremonies conducted in the Amazon. Due to travel restrictions we are also analyzing human EEG data with psilocybin from a clinical context.
• Type of collaboration: Joint research
• Researchers:
  • Dr. Tomáš PALENICEK, National Institute of Mental Health, Czech Republic

2.5 Experiences of Social Distancing during the COVID-19 Pandemic

• Description: This project investigates how the pandemic has affected, and continues to affect, how people think and feel about other people, society, and themselves. In order to do so, we have conducted an online survey in English, Japanese, and Spanish. The collected corpus consists of subjective reports from 2,543 participants, comprising over 500,000 words. We are analyzing it using both quantitative and qualitative methods, specifically by using natural language processing techniques so as to guide phenomenological interpretation. The novelty of this research is that we are thereby integrating methods from the social and computer sciences.
• Type of collaboration: Joint research
• Researchers:
  • Dr. Tom Froese, Okinawa Institute of Science and Technology Graduate University, Japan
  • Dr. Tomoari Mori, Okinawa Institute of Science and Technology Graduate University, Japan
  • Dr. Federico Sangati, Okinawa Institute of Science and Technology Graduate University, Japan
  • Prof. Matthew Ratcliffe, University of York, UK
  • Prof. Havi Carel, University of Bristol, UK
  • Dr. Alice Malpass, University of Bristol, UK
  • Prof. Matthew Broome, University of Birmingham, UK
  • Dr. Clara Humpston, University of Birmingham, UK

2.6 Gaze-based dyadic interaction

• Description: This project investigates the dynamics of mutual gaze using a dual eye-tracking setup. We are interested in exploring the interpersonal synchrony exhibited in eye activity in adult-adult and infant-caretaker dyadic situations. Our experimental setup enables us to investigate both direct gaze, and gaze mediated over a screen-based visual interface.
• Type of collaboration: Joint research
• Researchers:
  • Dr. Tom Froese, OIST
  • Maria Gohlke, OIST
  • Chen Lam Loh, OIST
  • Sho Tsuji, IRCN, University of Tokyo
  • Jessica Tan, IRCN, University of Tokyo
  • Tomoko Isomura, DCPS, Nagoya University

3. Activities and findings

Our research activities investigate the brain-body-environment basis of different kinds of mental processes at three important scales of analysis: the individual, the dyadic, and the collective. Apart from several small student-led projects, we have continued to work on three major lines of investigation:

3.1 Agency and perception

We want to understand the role of active movement in perceptual experience. Does our bodily engagement with the tools and objects around us make a difference to how we perceive them and the wider world? If agency does make a difference, how so?

To address these questions we make use of a specialized human-computer interaction interface, the sensory substitution interface called the Enactive Torch, as well as a robot arm manipulator from Tokyo Robotics. We will record data about participants' movements, experiences, as well as about their physiological activity.

In addition, we are breaking new conceptual ground by building on the latest developments in theoretical cognitive science and philosophy of mind, which leads us to embrace, rather than to reduce, the complexities of the human mind. It also provides us with a critical perspective towards the methods and assumptions of traditional cognitive science, thereby opening new routes of scientific innovation.

3.2 Sensitivity to social interaction

We want to understand the bodily basis of intersubjectivity, that is, of the social experience of being together with another person in direct interaction. What kind of bodily interaction patterns permit other people to show up as such in our perceptual experience? What is the neural, bodily, and interactive basis of our ability to share moments of personal experience with others?

To address these questions we make use of a specialized human-computer interaction interface, the minimal virtual reality platform called the Perceptual Crossing Paradigm. We will record data in a highly comprehensive manner using a sophisticated hyperscanning setup that will enable us to keep track of pairs of participants' movements, experiences, as well as their neural and physiological activity.

3.3 Dynamics of multi-agent systems

4. Publications

4.1 Journals

1. Carel H., Ratcliff M., Froese T. (2020). Reflecting on experiences of social distancing. The Lancet, 396(10244), 87-88
2. ​Dudysová D., ..., Froese T., Páleníček T., and Horáček J. (2020). The effects of daytime psilocybin administration on sleep: Implications for antidepressant action​. Frontiers in Pharmacology 11:602590. doi: 10.3389/fphar.2020.602590
3. Froese, T., Zapata-Fronseca, L., Leenen, I., and Fossion, R. (2020). The feeling is mutual: Clarity of haptics-mediated social perception is not associated with the recognition of the other, only with recognition of each other​. Frontiers in Human Neuroscience 14:560567. doi: 10.3389/fnhum.2020.560567
4. Froese T. and Ortiz-Garin G.U. (2020). Where is the action in perception? An exploratory study with a haptic sensory substitution device. Frontiers in Psychology, 11:809. doi: 10.3389/fpsyg.2020.00809
5. Heras-Escribano M. (2020). The evolutionary role of affordances: Ecological psychology, niche construction, and natural selection. Biology & Philosophy, 35, 30. doi: 10.1007/s10539-020-09747-1
6. Heras-Escribano M. (2020). Précis of the Philosophy of Affordances. Constructivist Foundations, 15(3), 199–213
7. Heras-Escribano M. (2020). Author’s response: Affordances as a basis for a post-cognitivist approach to the mind. Constructivist Foundations, 15(3), 231–237
8. Mojica L. (2021). Material playgrounds: Opening an ontology of active matter. Adaptive Behavior, in press
9. Mojica L. (2020). Reclaiming meaning, reclaiming normativity. Constructivist Foundations,, 15(3), 216–218.
10. Mojica L.​ (2020).Mentes en cuerpos de afecto: Nota crítica sobre Ecology of the Brain, de Thomas Fuchs. Crítica. Revista Hispanoamericana De Filosofía, 51(153), 83–97
11. Ramírez-Vizcaya S. (2021).A world-involving theory of agency: Review of Sensorimotor Life: An Enactive Proposal by Ezequiel Di Paolo, Thomas Buhrmann, and Xabier Barandiaran. Oxford University Press, Oxford, 2017. Adaptive Behavior, https://doi.org/10.1177/1059712320976676
12. Travieso D., Lobo L., de Paz C., Langelaar T.E., Ibáñez-Gijón J., and Jacobs D.M. (2020). Dynamic touch as common ground for enactivism and ecological psychology. Frontiers in Psychology, 11:1257. doi: 10.3389/fpsyg.2020.01257
13. Valencia A.L. and Froese T. (2020). What binds us? Inter-brain neural synchronization and its implications for theories of human consciousness. Neuroscience of Consciousness, 1. doi: 10.1093/nc/niaa010

4.2 Books and other publications

1. Dotov D. and Froese T. (2020). Dynamic interactive artificial intelligence: Sketches for a future AI based on human-machine interaction. In: J. Bongard, J. Lovato, L. Hebert-Dufrésne, R. Dasari, and L. Soros (Eds.),ALIFE 2020: The 2020 Conference on Artificial Life​ (pp. 139-145). Cambridge, MA: MIT Press
2. Froese, T. and Gallaga E.​ (in press). Measuring entanglement in material traces of ritualized interaction: Preferential attachment in a prehistoric petroglyph distribution​. Handbook of Material Religion. Routledge.
3. Morales A. and Froese T. (2020). The distribution of inhibitory neurons in the C. elegans connectome facilitates self-optimization of coordinated neural activity. In: 2020 IEEE Symposium Series on Computational Intelligence (SSCI), (pp. 1195-1201). Canberra, Australia: IEEE Press
4. Ramírez-Vizcaya S. and Froese T. (2020). Agents of habit: Refining the artificial life route to artificial intelligence. In: J. Bongard, J. Lovato, L. Hebert-Dufrésne, R. Dasari, and L. Soros (Eds.), ALIFE 2020: The 2020 Conference on Artificial Life​ (pp. 78-86). Cambridge, MA: MIT Press
5. Reséndiz-Benhumea G.M. and Froese T. (2020). Enhanced neural complexity is achieved by mutually coordinated embodied social interaction: A state-space analysis. In: Proceedings of the Second International Workshop on Agent-Based Modelling of Human Behaviour (2020 ABMHuB Workshop). Montreal, Canada
6. ​Reséndiz-Benhumea G.M., Sangati E., and Froese T. (2020). Levels of coupling in dyadic interaction: An analysis of neural and behavioral complexity. In: 2020 IEEE Symposium Series on Computational Intelligence (SSCI) (pp. 2250-2256). Canberra, Australia: IEEE Press
7. Sangati E. and Hofmann S.M. (2020). The role of co-representations in joint tracking. In: J. Bongard, J. Lovato, L. Hebert-Dufrésne, R. Dasari, and L. Soros (Eds.), ALIFE 2020: The 2020 Conference on Artificial Life​ (pp. 526-534), Cambridge, MA: MIT Press

5. Media

5.1 Public outreach

1. OIST Science Challenge 2021 (online streaming lab tour)
2. "Scientists probe the nature of the mind" (OIST News Center)
3. "Exploring the human experience of social distancing" (OIST News Center)
4. "Unraveling the evolutionary role of affordances" (Extended Evolutionary Synthesis Blog)

6. Meetings and events

6.1 Conference talks

1. Froese T. (2021). Irruption theory of consciousness. What's Next? Colloquium, Feb. 26-27, 2021, Cincinnati, OH
2. Morales A. (2020). The distribution of inhibitory neurons in the C. elegans connectome facilitates self-optimization of coordinated neural activity. IEEE Symposium on Artificial Life, Dec. 1-3, 2020, Canberra, Australia
3. Ramírez-Vizcaya S. (2020). Agents of habit: Refining the artificial life route to artificial intelligence. The 2020 Conference on Artificial Life​, July 13-18, 2020, Montreal, Canada
4. ​Reséndiz-Benhumea G.M. (2020). Levels of coupling in dyadic interaction: An analysis of neural and behavioral complexity. IEEE Symposium on Artificial Life, Dec. 1-3, 2020, Canberra, Australia
5. Sangati E. The role of co-representations in joint tracking. The 2020 Conference on Artificial Life​, July 13-18, 2020, Montreal, Canada

6.2 Other presentations

1. Froese T. (2020). Panelist of Theme 4: Symbiotic View of Life. Tokyo ALIFE 2020. Nov. 17, 2020, Tokyo, Japan
2. Mojica L. (2020). El problema de la normatividad en ciencias cognitivas: un problema de valores (The problem of normativity un cognitive science: a problem of value). Colombia: Ciclo de conferencias en filosofía y cognición. Feb., UniValle, Bogota, Colombia
3. Reséndiz-Benhumea G.M. (2020). Enhanced neural complexity is achieved by mutually coordinated embodied social interaction: A state-space analysis. The Second International Workshop on Agent-Based Modelling of Human Behaviour (2020 ABMHuB Workshop). July, Montreal, Canada
4. Reséndiz-Benhumea G.M. (2020). Modeling the interdependence of neural and social complexity. 2020 Emerging Researchers in Artificial Life Workshop (2020 ERA Workshop). July, Montreal, Canada
5. Sangati E., Reséndiz-Benhumea G.M., Sangati F., Yudin A., and Froese T. (2020). Maximizing transfer entropy in agent-based models. International Symposium on Artificial Intelligence and Brain Science 2020 (AIBS 2020). Tokyo, Japan.

6.3 Conferences and workshops

1. Workshop on "Dyadic Interactions in a Comparative Perspective"
   • Date: Dec. 10, 2020
   • Venue: OIST Seaside House
   • Speakers:
     • Sho TSUJI, IRCN, University of Tokyo
     • Katja SANGATI, ECSU, OIST
     • Georgina M. RESENDIZ-BENHUMEA, ECSU, OIST
     • Susana RAMIREZ-VIZCAYA, ECSU, OIST
     • Megumi KOBAYASHI, Aichi Developmental Disability Center
     • Monica BARBIR, IRCN, University of Tokyo
     • Jessica TAN, Western Sydney University
     • Jelena KATIC, NMCPU, OIST
     • Tomoko ISOMURA, DCPS, Nagoya University
     • Tom FROESE, ECSU, OIST

7. Teaching

1. Froese taught the course "Computational Methods".
2. Froese taught a guest lecture on the course "The ShanghAI lectures".

8. Awards

1. JSPS Topic-Setting Program, Oct. 2020 – March 2023, “Phenomenological analysis of corpus of subjective reports about COVID-19 pandemic based on natural language processing” Yen 14.6M
2. OIST KICKS Program, April 2020 – March 2021. Co-PI, Sho Tsuji, Uni of Tokyo. “Gaze-based minimal virtual reality paradigm for tracking developing sensitivity to dyadic interactions”. Yen 8M