Jingyi Wang
Information


Expanding from the exploration of mental images in urban experiences in film, Build Your City, Choose Your Weapon (Good Old Days Part.2) structures the complex system of societal space and mental structure into a replicable experience. It intricately intertwines an apocalyptic narrative through film, EEG, emotion recognition, Landmark Detection AI, SMS, and immersive installations.
          By translating brain activity into a cityscape via emotion recognition, this dynamic digital artcraft, and the resulting experience propose an alternative perspective on cognitive mapping—a spatial yet spiritual approach to commemorating and examining collective memory.


Outcome
Immersive Installation/Experience Design, Rendered Video Demo(02:08), Interactive Experience Demo(03:21)
Category
Interactive Art   Experimental Film   World Building   Multimedia Animation   Computational Video Art    Computational Design
Type
Individual Project
Date
October 2023
Keywords
Social Complexity, Cognitive Mapping, Collective Memory, Interactive Narrative
Techniques
MUSE2, Python, Unreal Engine, Blender, Touch Designer, Google Vision AI, Twilio







Your City,2023
A Rendered Demo of one’s city generated through BCCW

Unreal Engine, Touch Designer


[DEMO] Build Your City, Choose Your Weapon, 2023
An In-Home Demo of the Interactive Experience
MUSE 2, Computer, SmartPhone











MENTAL STRUCTURE,
COGNITIVE MAPPING,
& COLLECTIVE MEMORY



The bedrock of collective human memory and consensus rests upon the ever-shifting tapestry of human nature. In the realms of computation and artificial intelligence, investigations into fuzzy cognitive mapping have frequently been harnessed to transmute the cognitive and semantic fabric of spatial understanding into the rigidity of mathematical or logical constructs.

These constructs, in turn, have been instrumental in guiding our terrestrial journeys and prognosticating our behavioral trajectories. Yet, I seek to suspend mathematical models and focus on exploring whether our subjective cognition of space holds the key information that defines contemporary humanity - our commonality.

Such commonality may exhibit certain characteristics (capriciousness), typically intertwined with the silent authentication of entities (collective memory); it may reflect the foundation upon which consensus can still be achieved in the development of technology. How does our perception and recollection of space reflect such capriciousness?

I embark on research propelled by the use of brainwaves and computation, with the aim of resurrecting the subtle subterranean process of recalling the perception of space and memory when prompted by symbolic stimuli in the human mind, along with its topological relationship to collective memory.





RETHINKING INTERACVTIE FILM



Based on my study of moving images capturing urban human complexities, I reflected on how technology and media influence our collective memories. Rooted in our shared experiences, visual media can reflect evolving memories over time. Inspired by Deleuze's 'time-image' theory on how movement subordinates itself to time, I aim to challenge traditional interactive film approaches by creating an interactive narrative. This narrative blends the infinite continuity of time with the inherent metaphors of interaction in nature(including body), preserving human's autonomous initiative in human-computer interactions.






APPROACH & TOOLS



PART I
PERSONAL MEMORY & INHERENT BODY MOVEMENT - EEG TO CITY VISUALIZATION PIPELINE





PART II
From Personal to Collective Memory - Body Movement and Mobility in Urbanism

CORE CONCEPT



During the research, it was discovered that the model, designed to decipher brain wave data and adjust city structures along with game engine camera parameters to influence city’s imagery, surprisingly yet appropriately aligns with Sigmund Freud's concept of condensation in dreams. This theory describes how distinct elements merge to form singular dream images (Freud, 1900).











PRE-EXPERIMETN I
NARRATIVE MEDITATION SESSION




To explore the neural dynamics linked to the recollection of urban environments, I initiated a preliminary study involving a guided meditation session lasting 20-30 minutes. This session was flanked by pre- and post-session assessments of resting-state brain activity, culminating in a subsequent interview.








PRE-EXPERIMENT II
A Walk In the City




Image of the City, Cognitive Mapping, and the Other

How to capture the spirit of a city in an image? How do we engage with strangers, and how do our perceptions relate to them and urban structures in motion?

A city street is equipped to handle strangers (Jane Jacobs, The Death and Life of Great American Cities, 1961). Inspired by the urban imagery in Ellen Poe's Man in the Crowd and Calle's Suite Vénitienne, I undertook a brief experiment in New York: I followed a stranger during a moment of positive emotion (high valence) and retraced the path on a different day while experiencing negative emotion (low valence).







CORE CONCEPT

Translating Brain Activity to  Mental Structure, and Hence City Configuration Through Emotions









1







Emotions reflect our immediate mental state. When memories are triggered, emotions meld current feelings with past recollections.

Building on Mehrabian and Russell’s VAD (Valence-Arousal-Dominance) model [1], I’ve devised a model to translate real-time EEG data into city configurations: Valence-Arousal determines city structure parameters, while Dominance influences camera settings within Unreal Engine and Touch Designer.







2























The Valence-Arousal classification was conducted using the commercial MUSE 2 EEG headband with four electrodes[2]. The pipeline, executed in Python, was integrated from Touch Designer into Unreal Engine to manage the city parameters' PCG graph.












3





Dominance defines control over emotions, distinguishing emotions with similar arousal and valence levels. Various factors influence an image's dominance perception. Due to MUSE 2's technical constraints, the model classifying dominance levels using the beta/alpha ratio from frontal lobe electrodes (FC7 & F8) can't be applied[3]. In this model, dominance relates to a subject's perceived control over the imagined city, which, if viewed as a photographed image, correlates to the camera's depth of field and lens size.



  1. J. Ruseel and A. Mehrabian, “Evidence for a three-factor theory of emotions” Journal of Research in Personality, Volume 11, Issue 3, 1977, Pages 273-294, ISSN 0092-6566, https://doi.org/10.1016/0092-6566(77)90037-X.
  2. S. Moontaha, et al. “Online Learning for Wearable EEG-Based Emotion Classification.” Sensors, vol. 23, no. 5, Feb. 2023, p. 2387. Crossref, https://doi.org/10.3390/s23052387.
  3. Y. Liu and O. Sourina, “EEG-based Dominance Level Recognition for Emotion-Enabled Interaction.” 2012 IEEE International Conference on Multimedia and Expo, Melbourne, VIC, Australia, 2012, pp. 1039-1044, doi: 10.1109/ICME.2012.20.




DEVELOPMENT PROCESS I

Visualizing Mental Structure Through Brain Activity














DEVELOPMENT PROCESS II

Code: Land Mark Detection + SMS




DEVELOPMENT PROCESS III

Immersive Experience Design
& Spatial Design



(1)The interactive-immersive installation and the GPS + Web Page components are under development.

(2)A demo has been conducted to test the entire system, transforming the public interaction into a personal experience by utilizing blink detection on the MUSE device to capture screenshots. The film itself has been showcased at multiple venues.