FY 2021 Annual Report

Embodied Cognitive Science Unit (ECSU)
Assistant Professor Tom FROESE

Abstract

The unit’s primary mission is to do groundbreaking cognitive science by working out novel implications of the core premises of embodied cognition, especially of the enactive approach. Cognitive science is conceived as a broad constellation of disciplines, but with a particular emphasis on biology, neuroscience, psychology, anthropology, linguistics, and philosophy. The specific phenomenon to be investigated is (multi)agent-environment interaction. The implications to be worked out are primarily scientific, both theoretical and experimental.

1. Staff

Katja SANGATI, Postdoc
Sebastien LERIQUE, Postdoc
Jamila RODRIGUES, Postdoc
Mark JAMES, JSPS Postdoctoral Fellow
Andrés G. MEJÍA RAMÓN, Postdoc
Brian MORRISSEY, Technician
Stephen ESTELLE, Technician
Maria GOHLKE, Technician
Alexey YUDIN, Technician
Rai SATO, Technician
Federico SANGATI, Research Assistant
Ivan SHPUROV, OIST Doctoral Candidate
Chen Lam LOH, OIST Doctoral Candidate
Natalya WEBER, OIST Doctoral Candidate
Shannon HAYASHI, OIST Doctoral Candidate
Yi-Shan CHENG, OIST Doctoral Candidate
Morgan MONTOYA, Research Intern
Laura Alejandra MOJICA LOPEZ, Special Research Student
Susana RAMIREZ VIZCAYA, Special Research Student
Kaori YAMASHIRO, Research Unit Administrator

2. Collaborations

2.1 Experiences of Social Distancing during the COVID-19 Pandemic

Supported by the JSPS Grant “Experiences of Social Distancing during the COVID-19 Pandemic”
Type of collaboration: Joint research
Researchers:
- Dr. Tom Froese, Okinawa Institute of Science and Technology
- Dr. Jamila Rodrigues, Okinawa Institute of Science and Technology,
- Dr. Mark James, Okinawa Institute of Science and Technology,
- Dr. Federico Sangati, Okinawa Institute of Science and Technology,
- Prof. Matthew Ratcliffe, University of York,
- Prof. Havi Carel, University of Bristol,
- Dr. Alice Malpass, University of Bristol,
- Prof. Matthew Broome, University of Birmingham,
- Dr. Clara Humpston, University of Birmingham.

2.2 Real-Time Social Interaction System

Supported by the PoC Grant “Real-Time Social Interaction System“
Type of collaboration: Joint research
Researchers:
- Dr. Tom Froese, Okinawa Institute of Science and Technology,
- Dr. Sebastien Lerique, Okinawa Institute of Science and Technology,
- Dr. Masanori Isobe, Kyoto University Hospital
- Dr. Olaf Witkowski, Crosslabs.

2.3 Gaze-based minimal virtual reality paradigm for tracking developing sensitivity to dyadic interactions

Supported by KICKS Grant “Gaze-based minimal virtual reality paradigm for tracking developing sensitivity to dyadic interactions
Type of collaboration: Joint research
Researchers:
- Dr. Tom Froese, Okinawa Institute of Science and Technology,
- Dr. Tomoko Isomura, Nagoya University,
- Dr. Sho Tsuji, International Research Center for Neurointelligence of the University of Tokyo

3. Activities and Findings

3.1 Lab set-up

We set up our EEG and Physiology hyperscanning equipment and rooms, used it in a number of hyperscanning pilots, and in our first single-person physiology experiment. We are currently finishing a switch of rooms to enable better organisation of participant displays, interaction devices, and recording computers.

3.2 Links between Game Theory and Perceptual Crossing

Perceptual Crossing is a reference interaction set-up regularly used in the human interaction and cognitive science literature. We showed that the game that people are asked to participate in such a set-up can be described using Game Theoretic formalism. This opens the door to analyzing human interaction in this minimal set-up with standard game theory, identifying possible limits both in standard game theory and in the traditional interpretation of Perceptual Crossing results (that is, the enactive interpretation).

3.3 Documenting Emerging Forms of Digitally Mediated Social Normativity

With the widespread adoption of videoconferencing technologies, we are observing the emergence of new domains of social normativity and new skills of being together in such domains. Our effort is to document and conceptualise the development of these new ways of being together in digital spaces. This involves both phenomenological reflection and the analysis of a large data set collected under the conditions of imposed social restrictions during the COVID-19 pandemic. To help understand these emerging dynamics, we have begun developing the notion of digital tact. Digital tact is an embodied relation to oneself and others when in digitally mediated spaces that is sensitive both to the possibilities of those spaces and the others that occupy them in a way that helps meet the needs and goals of a given interaction.

3.4 Enacting Design

Through an enactive lens, design at its most general can be understood as a form of spatiotemporally extended adaptivity, whereby we regulate our interactions in the local-present to make certain forms of regulation easier or more probable in the future. The value of such understanding is that it supplies us with a vocabulary of change that applies as much to an individual agent as it does to a collective, such as an organisation. In enacting design we strive to understand not only the the extra-agential factors that enable or limit change in living system, but also the intra-agential factors too. This can help us appreciate, the individuality of any change effort, why some efforts towards change fail, and so on.

3.5 COVID-19, Mental Health and Well-being

One of the aspects of this COVID-19 Survey study is to focus on a particular kind of emotional impact of the pandemic, namely the phenomenology of the experience of moral injury in healthcare professionals. Drawing on Weber’s reflections in his lecture Politics as a Vocation and data from the Experiences of Social Distancing during the COVID-19 Pandemic Survey, we analyse responses from healthcare professionals which show the experiences of burnout, sense of frustration and impotence, and how these affect clinicians’ emotional state. We argue that this may relate to the ethical conflicts they experience when they are forced to make clinical decisions where there are no optimal outcomes, and how in turn that impacts on their own emotional state. We will further examine the notion of ‘burnout’ and the phenomenology of ‘moral injury’, and how for certain areas of clinical practice, for example psychiatry and mental health care, (where, for example, coercion is employed, or where service provision is routinely suboptimal), moral injury and the phenomenological emotional changes may be independent of the pandemic.

3.6 Agency and Perception

This project focuses on the role of active movement in perception. In the FY2021 research has proceeded on several fronts.

First, the main device used in this project - the Enactive Torch - has been subject to significant upgrade and development. We have redesigned both hardware and software to allow for higher precision and stability. For example, while our previous iteration was only able to produce square waves in its vibration output, current version produces amplified sine waves. This will allow us to specify more precisely the vibration frequencies delivered to experiment participants and the kind of mapping between distance perception and vibration output. We have also uncovered further areas in which the device needs to be improved (e.g., stimulation delays, a possibility to integrate the device with psychophysiological measurement system) and development in this direction will continue in the new year.

Second, we have initiated a line of research on the psychophysical profile of the Enactive Torch. Previous studies merely assumed that implemented mapping function between distance measurement and vibration output is well suited for the tasks of navigation or object recognition. We decided that a more solid investigation is required to determine how well people can actually perceive distance or size differences and whether some adjustment to the device is required to make the task easier. We have submitted a research plan for ethics committee approval and are currently setting up pilot studies to further refine our experimental design.

Third, the behavioral study on the effects of active perception of object sizes has entered the data collection phase. We have finished pilot testing and started actual experiments with human participants. This process will be completed within the next month and we will then proceed with data analysis and publication preparation.

Fourth, we began exploring the possibility of adding an EEG measurement to the behavioral study. The aim of this part of the project would be to uncover neural mechanisms of active perception by comparing brain activity in active and passive conditions. Since our current experimental design involves a lot of movement by the participant, which generates a lot of movement artifacts in the brain activity recording, we are trying out different modifications that will allow us to produce cleaner data.

3.7 Social Interaction in Different Modalities

This section centers around two closely related projects. The first project is “gaze interactions” focused on comparing eye movement patterns between face-to-face and screen-mediated social interaction. The second project is “video chat” focused on comparing brain activity in the same conditions. It was planned initially that the latter project would involve a speech-based task, i.e., a conversation between people in face-to-face interaction compared to interaction over video conferencing program. We have conducted extensive piloting of this setup and discovered that speech produces a lot of artifacts in brain activity recording and that eliminating these artifacts would require significant theoretical and methodological development. At the same time, the “gaze interactions” project moved closer to the need for investigating concurrent brain activity. As a result in the recent months we have combined the two projects together and are now designing an experiment on non-verbal interaction in which face-to-face and video-chat conditions will be compared with respect to both eye movements and brain activity.

3.8 Agent-based Modeling

Our simulations of multi-agent systems have centered on two related questions: How does interaction affect individual brain activity? Does joint brain activity exhibit signatures of synergy that goes beyond the individuals involved?

The first question was addressed with simulations in which agents are allowed to interact freely and are evolved to maximize their individual neural complexity. We have investigated whether the effects of interaction depend on the individuals’ brain size. That is, does social interaction allow agents with smaller brains to exhibit behavioral and neural complexity comparable to that of agents with bigger brains that are evolved individually? And does the interaction effect scale with an increase in brain size (bigger benefit with bigger brains)? Answers to these questions are presented in our Frontiers in Neurorobotics publication.

The second question was addressed by simulating agents in a joint action task and looking at possible formalizations of “inter-brain synergy”. Synergy is a term that refers to instances in which the whole is more than the sum of the parts. A number of measures have been proposed to capture this phenomenon. We have used one of the most common measures to compare the synergy between the brain activity of agents that were evolved to coordinate their behavior in tasks that require different levels of co-dependence. We found that regardless of the task the synergy was negative indicating that the agents performed different tasks individually and merely in a correlated manner. Whether this means that the tasks we tested were inadequate or the measure we used needs to be adjusted remains a question for further research.

3.9 Paleohydrology of the Teotihuacan Valley Project

Led by Dr. Mejía Ramón, this project focuses on the relationship between humans and the natural environment across the longue durée of history in the Teotihuacan Valley as mediated by water. Previous field seasons have employed drone and satellite-based remote sensing, geophysical prospection, and archaeological excavation to identify ancient water management features and water ritual sites. Additional art historical, epigraphic, computational, and ecological research has contextualized such findings within the broader context of Mesoamerican archaeology and the evolution of complex societies. This project has been operating for eight years, and is now in its first year at its new home at ECSU in OIST. Within its first six months based out of Japan, the project has already completed its first field excursion, having discovered a new Teotihuacan-period (A.D. 200-550) carved stone monument to the Mesoamerican Storm God. Currently, project activities include the analysis and curation of ceramic materials from previous excavations, survey for sites with evidence for Archaic (5000 - 1200 BC) and Early Formative (1200 - 800 BC) settlement throughout the Basin of Mexico, additional survey for mountaintop rainfall petitioning sites, and computational modeling to identify thermodynamic limits to human sociopolitical organization. We have also secured funding to procure an unmanned aerial system incorporating lidar and multispectral technologies. These will be employed to identify new archaeological sites beneath forest and grassland vegetation that are predicted by our research projects to contain information about early sociopolitical organization. This project involves collaborations with Dr. Nadia E. Johnson (Rhea Engineers and Consultants), Mtro. Christian L. John (University of California, Davis), Dr. Luis Barba (Universidad Nacional Autónoma de México), Mtro. Jorge E. Blancas Vázquez (Universidad Nacional Autónoma de México), Dr. Agustín Ortiz Butrón (Universidad Nacional Autónoma de México), and Dr. Tom Froese (OIST).

4. Publications

4.1 Journals

Froese T., Broome M., Carel H., Humpston C., Malpass A., Mori T., Ratcliffe M., Rodrigues J., Sangati F. (2021). The Pandemic Experience: A Corpus of Subjective Reports on Life During the First Wave of COVID-19 in the UK, Japan, and Mexico, Front. Public Health, 9:725506. https://doi.org/10.3389/fpubh.2021.725506
Lerique, S. (2022). Embodied Rationality Through Game Theoretic Glasses: An Empirical Point of Contact. Frontiers in Psychology, 13
Sangati, E., Slors, M., Müller, B. C., & van Rooij, I. (2021). Joint Simon effect in movement trajectories. PLoS ONE, 16(12), Article e0261735. doi: 10.1371/journal.pone.0261735
Reséndiz-Benhumea, G. M., Sangati, E., Sangati, F., Keshmiri, S., & Froese, T. (2021). Shrunken social brains? A minimal model of the role of social interaction in neural complexity. Frontiers in Neurorobotics, 15, 634085. doi: 10.3389/fnbot.2021.634085.

4. 2 Books and book chapters

N/A

4.3 Oral and Poster Presentations

Rodrigues J., Presented Special Seminar Lecture at, International Research Centre for Japanese Studies Kyoto, (June 2021)
Froese T., James M., Rodrigues J., Conference presentation at Loneliness: A Discussion in Philosophy and Psychology Conference (July 12^th -15^th 2021)
James M., Laying down a path to walk in: enaction, design, and multiscale interventions for change. Presented as part of the Enso Seminar Series, the online forum for conversations at the intersection of enaction and ecological psychology. The talk is available for viewing at http://www.ensoseminars.com/presentations/tag/design (November 11^th 2021)
Rodrigues J., "The Moral Roots of Quarantine: the East and the West" conference hosted by the University of Macau (December 16^th2021)
Sangati, E., Keshmiri, S., & Sangati, F. Evolution of Neural Complexity in Division of Labor Tasks. In ALIFE 2021: The 2021 Conference on Artificial Life (pp. 88). MIT Press. doi: 10.1162/isal_a_00422 (2021)
Lerique, S., Froese T., Partial acts as enablers of shared embodied rationality. CILC5 2021.
Mejía Ramón, Andrés G. (2022). The Mechanics of Landscape and Behavior: A Least-Cost Path Approach Grounded in Mechanical Physics. Paper presented at the 87th Annual Meeting of the Society for American Archaeology, 30 March - 3 April 2022, Chicago.
Keshmiri, S., Sangati, F. & Sangati, E. Information Integration in Division of Labor: Validation of Earlier Findings. In Proceedings of the Joint Symposium of AROB-ISBC-SWARM 2022 (pp. 24-29). International Society of Artificial Life and Robotics. (2022)

5. Intellectual Property Rights and Other Specific Achievements

JSPS Grant
Title: Experiences of Social Distancing during the COVID-19 Pandemic
Researchers:
- Dr. Tom Froese, Okinawa Institute of Science and Technology
- Dr. Jamila Rodrigues, Okinawa Institute of Science and Technology,
- Dr. Mark James, Okinawa Institute of Science and Technology,
- Dr. Federico Sangati, Okinawa Institute of Science and Technology,
- Prof. Matthew Ratcliffe, University of York,
- Prof. Havi Carel, University of Bristol,
- Dr. Alice Malpass, University of Bristol,
- Prof. Matthew Broome, University of Birmingham,
- Dr. Clara Humpston, University of Birmingham.

PoC Grant
Title: Real-Time Social Interaction System“
Researchers:
- Dr. Tom Froese, Okinawa Institute of Science and Technology,
- Dr. Sebastien Lerique, Okinawa Institute of Science and Technology,
- Dr. Masanori Isobe, Kyoto University Hospital
- Dr. Olaf Witkowski, Crosslabs.

KICKS Grant
Title: Gaze-based minimal virtual reality paradigm for tracking developing sensitivity to dyadic interactions
Researchers:
- Dr. Tom Froese, Okinawa Institute of Science and Technology,
- Dr. Tomoko Isomura, Nagoya University,
- Dr. Sho Tsuji, International Research Center for Neurointelligence of the University of Tokyo

Kakenhi Grant-in-Aid for Early-Career Scientists
Titile: The role of action in perception: a literature review and behavioral experiments
Period: FY2022-2024
Principal Investigator: Ekaterina Sangati

Kakenhi Grant-in-Aid for Early Career Scientists
Title: Interaction-based markers of mental illnesses based on sensorimotor interaction patterns: towards the development of early, non-invasive, and specific measures of the risk for mental illness
Period: FY2022-2023
Principal Investigator: Sébastien Lerique

Kakenhi Grant-in-Aid Early Career

6. Meetings and Events

Speaker	Date	Theme(s)	Title	Links
Tony Chemero	2021-09-27	Ecological psychology	Inhabiting Space	YouTube
Michelle Maiese	2021-11-08	Habits and racism	Mindshaping, Racist Habits, and White Ignorance	YouTube
Sune Vork Steffensen	2021-11-26	Psychopathology	Cognitive Probatonics: what cognitive ethnography can teach us about problem-solving	YouTube
Katsunori Miyahara	2021-11-26	Habits and self	Narrative self and habits	YouTube
Shaun Gallagher	2021-12-06	Enactivism in psychiatry	Enactive solutions to the integration problem in psychiatry	YouTube
Jelle Bruineberg and Regina Fabri	2021-12-10	HCI	Social connection in a digital world: Phubbing as extended mind-wandering	Zoom
Marlou Rasenberg	2021-12-10	Sensorimotor	The use of multimodal resources for joint meaning-making in conversational repair sequences	YouTube
Shir Atzil	2022-01-07	Society, evolution	Growing a social brain	Zoom
Shigeru Taguchi	2022-01-17	Consciousness	A twisted loop between inside and outside: Searching for an appropriate image of consciousness based on phenomenology and enactivism	YouTube
Sanneke de Haan	2022-01-28, 3:30pm (Fri)	Enactivism in psychiatry	Enactive Psychiatry, existential sense-making, and enactive ethics	YouTube
Cathy Macpherson	2022-01-28, 3:30pm (Fri)	Psychopathology	Exploring the association between mental health and interpersonal coordination.	YouTube
Mizuki Oka	2022-02-14, 9:30am (Mon)	Society	Exploring online social ecosystems through bio-inspired perspectives	Zoom
Artur Czeszumski	2022-03-28, 4:00pm (Mon)	Society	2nd person approach to studying cooperative social interactions	YouTube
Juan M. Loaiza		Performative arts	Music and Wellbeing in Ritual contexts: pushing the embodied cognitive science of social dynamics beyond WEIRD research	YouTube

7. Other

James M., Host of the monthly Connectomics podcast.
- On the Connectomics podcast, Mark speaks with practitioners and theorists with some connection to embodied cognitive science (ECS) about the intersection between ECS, culture, technology, and design. The Connectomics podcast is a monthly installation. Future episodes this year aim to focus on embodied cognitive science in Japan, with the hope of highlighting Japanese philosophers and scientists to an English speaking audience.
James M., Featured on the panel for Japan Society New York, guest on the podcast The Algo Podcast