FY2022 Annual Report

Embodied Cognitive Science Unit (ECSU)
Assistant Professor Tom FROESE

ECSU_2023_Jan
 

Abstract

The unit’s primary mission is to do groundbreaking cognitive science by working out novel implications of the core premises of embodied cognition, especially of the enactive approach. Cognitive science is conceived as a broad constellation of disciplines, but with a particular emphasis on biology, neuroscience, psychology, anthropology, linguistics, and philosophy. The specific phenomenon to be investigated is (multi)agent-environment interaction. The implications to be worked out are primarily scientific, both theoretical and experimental.

1. Staff

  • Katja SANGATI, Postdoc
  • Sebastien LERIQUE, Postdoc
  • Mark JAMES, JSPS Postdoctoral Fellow
  • Andrés G. MEJÍA RAMÓN, Postdoc
  • Finda Dwi PUTRI, Postdoc
  • Brian MORRISSEY, Technician
  • Stephen ESTELLE, Technician
  • Rai SATO, Technician
  • Leticia BERKATE, Technician
  • Federico SANGATI, Research Assistant
  • Natalia KOSHKINA, Research Assistant
  • Tae MORRISSEY, Research Assistant
  • Ivan SHPUROV, OIST Ph.D. Student
  • Chen Lam LOH, OIST Ph.D. Student
  • Natalya WEBER, OIST Ph.D. Student
  • Shannon HAYASHI, OIST Ph.D. Student
  • Yi-Shan CHENG, OIST Ph.D. Student
  • Kazuma Takada, OIST Doctoral Candidate
  • Georgii Karelin, OIST Doctoral Candidate
  • Tereze SEDLINSKA, OIST Rotation Student
  • Rui FUKUSHIMA, OIST Rotation Student
  • John SYKES, Research Intern
  • Francesca FOTIA, Research Intern
  • Morgan MONTOYA, Research Intern
  • Kathryn BODY, Research Intern
  • Mats-Helge STROMBERG, Research Intern
  • Samira TAVASSOLI, Visiting Student
  • Kaori YAMASHIRO, Research Unit Administrator
     

2. Collaborations

2.1 Experiences of Social Distancing during the COVID-19 Pandemic

  • Supported by the JSPS Grant “Experiences of Social Distancing during the COVID-19 Pandemic”
  • Type of collaboration: Joint research
  • Researchers:
    • Dr. Tom Froese, Okinawa Institute of Science and Technology
    • Dr. Jamila Rodrigues, Okinawa Institute of Science and Technology,
    • Dr. Mark James, Okinawa Institute of Science and Technology,
    • Dr. Federico Sangati, Okinawa Institute of Science and Technology,
    • Dr. Natalia Koshkina, Okinawa Institute of Science and Technology,
    • Prof. Matthew Ratcliffe, University of York, 
    • Prof. Havi Carel, University of Bristol,
    • Kathryn Body, University of Bristol,
    • Prof. Matthew Broome, University of Birmingham,
    • Dr. Clara Humpston, University of Birmingham.

2.2 Real-Time Social Interaction System

  • Supported by the PoC Grant “Real-Time Social Interaction System“
  • Type of collaboration: Joint research
  • Researchers: 
    • Dr. Tom Froese, Okinawa Institute of Science and Technology, 
    • Dr. Sebastien Lerique, Okinawa Institute of Science and Technology,
    • Dr. Finda Dwi Putri, Okinawa Institute of Science and Technology,
    • Dr. Masanori Isobe, Kyoto University Hospital
    • Dr. Olaf Witkowski, Crosslabs. 

2.3. Gaze-based minimal virtual reality paradigm for tracking developing sensitivity to dyadic interactions​

  • Supported by KICKS Grant “Gaze-based minimal virtual reality paradigm for tracking developing sensitivity to dyadic interactions
  • Type of collaboration: Joint research
  • Researchers: 
    • Dr. Tom Froese, Okinawa Institute of Science and Technology,
    • Dr. Tomoko Isomura, Nagoya University,
    • Dr. Sho Tsuji, International Research Center for Neurointelligence of the University of Tokyo

3. Activities and Findings

3.1 Exploring the Emotional Dysregulation of ICT Use: Evidence from First-Person Subjective Reports

Our study investigates the emotional dysregulation caused by ICTs (information and communication technologies) and presents compelling evidence based on first-person subjective reports collected during the "Experiences of Social Distancing during the COVID-19 Pandemic" project. We identified five distinct dimensions of emotional dysregulation related to ICTs: infrastructure, functional use, mindful design individual, mindful design social, and digital tact. Our findings underscore the need for a deeper understanding of the complex relationship between affective experiences and our socio-technical environment, as well as the importance of developing the necessary skills to engage with and regulate these experiences. The significance of our research lies in its ability to shed light on the importance of considering the affective dimensions of our socio-technical environment in future policy and decision-making.

3.2 Agency and Perception

This project focuses on the role of active movement in perception. In the FY2022 we continued our efforts on research equipment development and human subject experimentation. The sensory substitution device (Enactive Torch) has been redesigned to be integrated with psychophysiological measurement system (body physiology and EEG) in that data can be acquired continuously from movement, vibration output and physiology and synchronized via event triggers. We have also tested possible delays in the system and began work on minimizing them. In the new FY we plan to further improve the device hardware and software and publish a technical paper which will introduce it to the broader scientific community. More importantly, we have completed one behavioral study that compared active and passive perception of object sizes. We analyzed the data and submitted a publication based on this research, which is currently under review. We also started designing a new experiment on shape recognition in a similar setup and we plan to complete this study in the new FY.

3.3 Social Interaction in Different Modalities

We have designed a study on non-verbal interaction in which face-to-face and video-chat conditions would be compared with respect to both eye movements and brain activity. We spent considerable efforts in developing the experimental interface that gives us full control over the video connection. This way, any differences that will be found between two modes of interaction cannot be attributed only to a greater interaction delay in the video condition but can be tested more broadly. We have also conducted several pilot studies of this experiment and further optimized experimental design.

3.4 Perceptual Crossing Experiment (PCE)

The Perceptual Crossing paradigm allows the condition to study social interaction in a pair of individuals, virtually, with minimum sensory stimulation. In our Perceptual Crossing Experiment (PCE), two separately located users were each asked to navigate a cursor attached to a device, in which the cursor movement represents their virtual movement in a shared two-dimensional circle. In that shared virtual space, the users have to control their virtual objects to find each other, while they also encounter non-target objects such as shadows or static objects. When a user’s object touches a shadow or a static object, a one-sided vibration will be given to that user and when both user’s objects meet (i.e., Perceptual Crossing), vibration will be given to both users. The users have to press a button on their device when they feel their object meets the other user’s object. In this study, we apply EEG hyperscanning (simultaneous multi-brain EEG recording) during PCE to further explore the brain features during social interaction. From this setup, we expect to identify both behavioural parameters and brain activity features that represent social interaction, particularly when users find each other. Once identified, we expect to investigate the links between these parameters and any social interaction related mental health disorder.

3.5 Irruption Theory

The enactive approach has made some progress by bringing human experience closer to cognitive science, including by placing normativity at its core: all cognitive behavior is held to be a kind of motivated behavior. Yet appeals to system-level properties of the organism are ultimately guilty of a similar sleight-of-hand, whereby the efficacy of normativity is offloaded to the efficacy of something non-normative while assuming functional equivalency. Here an alternative theory is proposed by introducing the concept of irruption with the goal of operationalizing an agent’s motivational involvement in terms of their behavior’s material underdetermination. Irruptions set a lower bound on the unpredictability of (neuro)physiological processes and should therefore be measurable in terms of information-theoretic entropy. Counterintuitively, the arbitrariness introduced by irruptions does not stand in the way of effective behavior. Rather, as indicated by artificial life models of complex adaptive systems, burstiness of arbitrary changes can facilitate lifetime adaptivity. Moreover, Irruption Theory suggests that the increased neural entropy associated with action, cognition, and even consciousness, can be interpreted as a marker of elevated motivational involvement. The concept of irruption makes it intelligible how we can be agents  whose motivations can make an effective difference to our behavior, yet without being able (nor needing) to directly control our body’s neurophysiological processes.

 

4. Publications

4.1 Peer-reviewed Papers

  1. Lerique, S. (2022). Embodied rationality through game theoretic glasses: an empirical point of contact. Frontiers in Psychology, 13, 815691. doi: 10.3389/fpsyg.2022.815691

  2. Hermans, K. S., Kirtley, O. J., Kasanova, Z., Achterhof, R., Hagemann, N., Hiekkaranta, A. P., Lecei, A., Zapata-Fonseca, L., Lafit, G., Fossion, R., Froese, T. & Myin‐Germeys, I. (2022). Capacity for social contingency detection continues to develop across adolescence. Social Development, 31, 530– 548. doi: 10.1111/sode.12567

  3. Hermans, K., Kirtley, O., Kasanova, Z., Achterhof, R., Hagemann, N., Hiekkaranta, A. P., Lecei, A., Zapata-Fonseca, L., Lafit, G., Fossion, R., Froese, T. & Myin‐Germeys, I. (2022). Ecological and convergent validity of experimentally and dynamically assessed social capacity using the Perceptual Crossing Experiment in adolescence. Assessment. doi: 10.1177/10731911221083613

  4. Froese, T. (2022). Scientific observation is socio-materially augmented perception: Toward a participatory realism. Philosophies, 7(2), Article 37. doi: 10.3390/philosophies7020037

  5. Pescador Canales, C., & Mojica, L. (2022). Making us Autonomous: The Enactive Normativity of Morality. Topoi, 41, 257–274. doi: 10.1007/s11245-022-09795-0

  6. James M. M., Rodrigues J., Montoya M., Koshkina N., Sangati F., Sangati E., Ratcliffe M., Carel H. and Froese T.(2022) The Pandemic Experience Survey II: A Second Corpus of Subject Reports of Life Under Social Restrictions During COVID-19 in the UK, Japan, and Mexico. Frontiers in Public Health, 10, 913096. doi: 10.3389/fpubh.2022.913096

  7. Shimizu K. , Ienaga N. , Takada, K. , Sugimoto M. , Kasahara S. (2022). Human Latent Metrics: Perceptual and Cognitive Response Correlates to Distance in GAN Latent Space for Facial Images. In M. Lau,  A. Robb, M. Barnett-Cowan, A. Serrano, S. Malpica (Eds.), ACM Symposium on Applied Perception, Article 3 (pp. 1-10). Association for Computing Machinery. doi: 10.1145/3548814.3551460

  8. Weber, N., Koch, W., & Froese, T. (2022). Scaling up the self-optimization model by means of on-the-fly computation of weights. In 2022 IEEE Symposium Series on Computational Intelligence (SSCI), Singapore, Singapore (pp. 1276-1282). IEEE Press. doi: 10.1109/SSCI51031.2022.10022074

4.2 Other Publications

  1. Keshmiri, S., Sangati, F. & Sangati, E. (2022). Information Integration in Division of Labor: Validation of Earlier Findings. In Proceedings of the Joint Symposium of AROB-ISBC-SWARM 2022 (pp. 24-29). International Society of Artificial Life and Robotics. [pdf]

  2. Loh, C. L. & Froese, T. (2022). Prospects of inter-brain synchronization with a virtual agent: Preliminary considerations. In Proceedings of the Joint Symposium of AROB-ISBC-SWARM 2022 (pp. 237-242). International Society of Artificial Life and Robotics.  [pdf]

  3. Ramírez-Vizcaya, S. (2022). A world-involving theory of agency: review of Sensorimotor Life: An Enactive Proposal by Ezequiel Di Paolo, Thomas Buhrmann, and Xabier Barandiaran. Oxford University Press, Oxford, 2017. Adaptive Behavior, 30(2), 205-208. doi: 10.1177/1059712320976676

  4. Mojica, L. (2022). Material playgrounds: opening an ontology of active matter. Adaptive Behavior, 30(6), 535-536. doi: 10.1177/1059712320988178

  5. Froese, T. (2021). To understand the origin of life we must first understand the role of normativity. Biosemiotics, 14, 657-663. doi: 10.1007/s12304-021-09467-3

  6. Sangati, E. (2022) Turning language-science prospects into science. Constructivist Foundations, 18(1), 049-051. [link]

4.3 Selected Oral Presentations

  1. Froese, T., Pandemic disorders of consciousness, 28th East Asia Joint Symposium on Biomedical Research, Neuroscience, psychoscience and aging Section, October 27th, 2022, Online

  2. Froese T., Enaction as irruptions and body memory,  Face-body Studies Wrap-up Symposium, Body Schema, Arts, and Social Participation, November 12th, 2022, Tokyo, Japan, Kokugakuin University Shibuya Campus

  3. Froese T., Enactive perspectives on consciousness and psychological flexibility, EAPP Conference "The Emotions", September 22-24, 2022, Heidelberg, Germany

  4. Takada K., The Design and Scalability of Online Experiment Platform, Young Perceptionists' Seminar 2022 (YPS2022), August 21-22, 2022, Japan.

5. Grants

  • JSPS Grant 
  • Title: Experiences of Social Distancing during the COVID-19 Pandemic
  • Period: FY2021-2023
  • Researchers:
    • Dr. Tom Froese, Okinawa Institute of Science and Technology
    • Dr. Jamila Rodrigues, Okinawa Institute of Science and Technology,
    • Dr. Mark James, Okinawa Institute of Science and Technology,
    • Dr. Federico Sangati, Okinawa Institute of Science and Technology,
    • Dr. Natalia Koshkina, Okinawa Institute of Science and Technology,
    • Prof. Matthew Ratcliffe, University of York, 
    • Prof. Havi Carel, University of Bristol,
    • Prof. Matthew Broome, University of Birmingham,
    • Dr. Clara Humpston, University of Birmingham.

​

  • PoC Grant 
  • Title: Real-Time Social Interaction System“
  • Period: FY2022-2024
  • Researchers: 
    • Dr. Tom Froese, Okinawa Institute of Science and Technology, 
    • Dr. Sebastien Lerique, Okinawa Institute of Science and Technology,
    • Dr. Masanori Isobe, Kyoto University Hospital
    • Dr. Olaf Witkowski, Crosslabs. 

 

  • KICKS Grant
  • Title: Gaze-based minimal virtual reality paradigm for tracking developing sensitivity to dyadic interactions
  • Period: FY2022-2024
  • Researchers: 
    • Dr. Tom Froese, Okinawa Institute of Science and Technology,
    • Dr. Tomoko Isomura, Nagoya University,
    • Dr. Sho Tsuji, International Research Center for Neurointelligence of the University of Tokyo

 

  • Kakenhi Grant-in-Aid for Early-Career Scientists
  • Titile: The role of action in perception: a literature review and behavioral experiments
  • Period: FY2022-2024
  • Principal Investigator: Ekaterina Sangati

 

  • Kakenhi Grant-in-Aid for Early Career Scientists 
  • Title: Interaction-based markers of mental illnesses based on sensorimotor interaction patterns: towards the development of early, non-invasive, and specific measures of the risk for mental illness
  • Period: FY2022-2023
  • Principal Investigator: Sébastien Lerique

6. Meetings and Events

6.1 Workshop "Experiences of social distancing during the COVID-19 pandemic" ​

  • Date: March 17, 2023
  • Venue: OIST 
  • Speakers

    • Tom Froese,  Okinawa Institute of Science and Technology 

    • Mark James, Okinawa Institute of Science and Technology 

    • Kathryn Body, University of Bristol

    • Masami Yamaguchi, Chuo University

    • Simon Høffding, University of Southern Denmark

    • Katsunori Miyahara, Hokkaido University, Centre for Human Nature, Artificial Intelligence, and Neuroscience

    • Matthew Ratcliffe, University of York

    • Shogo Tanaka, Tokai University

    6.2 International Conference on Embodied Cognitive Science "Interaction Matters" ​

    • Date: November 7 - 11, 2022
    • Venue: OIST Seaside House
    • Speakers

      • Tom Froese,  Okinawa Institute of Science and Technology 

      • Jun Tani, Okinawa Institute of Science and Technology 

      • Mog Stapleton, Okinawa Institute of Science and Technology 

      • Shaun Gallagher, University of Memphis

      • Ivana Konvalinka, Technical University of Denmark

      • Charles Lenay, Universite de Technologie Compiegne

      • Sanneke de Haan, Erasmus University Rotterdam

      • Takashi Ikegami, University of Tokyo

      • Anna Ciaunica, University of Lisbon

      • Shogo Tanaka, Tokai University

      • Mizuki Oka, Tsukuba University

      • Joel Krueger, University of Exeter

      • Shunichi Kasahara, Sony Computer Science Laboratories

      • Michelle Maiese, Emmanuel College Boston

      • Tony Chemero, University of Cincinnati

    6.3 Workshop "Enactive perspectives on consciousness and psychological flexibility"

    • Date: September 23, 2022
    • Venue: Heidelberg, Psychiatric Clinic, Germany
    • Speakers:

      • Tom Froese,  Okinawa Institute of Science and Technology 

      • Mark James, Okinawa Institute of Science and Technology 

    6.4 Workshop "Re-conceiving enaction as the irruption of consciousness"

    • Date: June 30
    • Venue: University Clinic Heidelberg
    • Co-organizers: 
    • Speakers:

      • Tom Froese,  Okinawa Institute of Science and Technology 

    6.5 ECogS Seminars

    1. Tanaka, S., Intercorporeality mediated by online meeting software, 2022-04-04, YouTube

    2. Tani, J., Studies on cognitive neurorobotics by extending the framework of predictive coding and active inference, 2022-04-11, YouTube

    3. Bolis D., I interact therefore I am': Human becoming in and through social interaction, 2022-04-15, YouTube

    4. Ciaunica, A., The 'First Prior': from Co-Embodiment to Co-Homeostasis in Early Life, 2022-05-13, YouTube

    5. Loaiza, J.M., Music and Wellbeing in Ritual contexts: pushing the embodied cognitive science of social dynamics beyond WEIRD research, 2022-05-13, YouTube

    6. Kyselo, M., Enactive Cognition meets Feminism - A New Solution to the Mind-Body-Body Problem, 2022-05-23, Zoom

    7. Zebrowski, R. and McGraw, E.B., Participatory Sense-Making and Sociomorphing: The Role of Embodied Artificial Systems in Social Interaction, 2022-06-10, YouTube

    8. Suzuki K., Hallucination Machine and Computational Neurophenomenology, 2022-06-10, YouTube

    9. Dotov D., Drumming humans obey similar collective dynamics as swarming and flocking animals, 2022-06-10, YouTube

    10. Branca M.I., The role of emotions in binding us, 2022-07-29, YouTube

    11. Patil G. and Nalepka P., A generalized framework for modelling human joint actions using dynamical perceptual motor primitives, 2022-08-12, YouTube

    12. Hart Y., Social Motion - On social prediction and decision making, 2022-08-12, YouTube

    13. Miller M., Artificial empathy: an active inference approach to affective computing, 2022-08-19

    14. Lenay C., Enactive approach to the question of the possible, 2022-09-09, YouTube

    15. Auvray M., The influence of sensory deficits and social cognition on spatial perspective-taking, 2022-09-09, YouTube

    16. Krueger J., Communing with the dead online: chatbots, grief, and continuing bonds, 2022-10-07, YouTube

    17. Pini S., Stage Presence in Dance: A Cognitive Ecological Ethnographic Approach, 2022-10-17, YouTube

    18. Konvalinka I., The interpersonal factors that shape real-time social interaction dynamics, 2022-10-28, YouTube

    19. Miles L., Coupled – for better or worse? Exploring how coordination dynamics shape collective performance. 2022-10-28,YouTube

    20. Schütz C.G. Rat Park, Pubs, and Red Fish: trying to make sense, 2022-11-04, YouTube

    21. Kono T. Embodied Bases of Attention-Deficit/Hyperactivity Disorder (ADHD), 2022-11-04, YouTube

    22. Osler L., Objectification, shame, and the gaze of digital technology, 2022-11-21, YouTube