Immersive Installation/Experience Design, Rendered Video Demo(02:08), Interactive Experience Demo(03:21)

Category

Interactive Art   Experimental Film   World Building   Multimedia Animation   Computational Video Art    Computational Design

Type

Individual Project

Date

October 2023

Keywords

Social Complexity, Cognitive Mapping, Collective Memory, Interactive Narrative

Techniques

MUSE2, Python, Unreal Engine, Blender, Touch Designer, Google Vision AI, Twilio

MENTAL STRUCTURE,
COGNITIVE MAPPING,
& COLLECTIVE MEMORY

The bedrock of collective human memory and consensus rests upon the ever-shifting tapestry of human nature. In the realms of computation and artificial intelligence, investigations into fuzzy cognitive mapping have frequently been harnessed to transmute the cognitive and semantic fabric of spatial understanding into the rigidity of mathematical or logical constructs.

These constructs, in turn, have been instrumental in guiding our terrestrial journeys and prognosticating our behavioral trajectories. Yet, I seek to suspend mathematical models and focus on exploring whether our subjective cognition of space holds the key information that defines contemporary humanity - our commonality.

Such commonality may exhibit certain characteristics (capriciousness), typically intertwined with the silent authentication of entities (collective memory); it may reflect the foundation upon which consensus can still be achieved in the development of technology. How does our perception and recollection of space reflect such capriciousness?

I embark on research propelled by the use of brainwaves and computation, with the aim of resurrecting the subtle subterranean process of recalling the perception of space and memory when prompted by symbolic stimuli in the human mind, along with its topological relationship to collective memory.

RETHINKING INTERACVTIE FILM

Based on my study of moving images capturing urban human complexities, I reflected on how technology and media influence our collective memories. Rooted in our shared experiences, visual media can reflect evolving memories over time. Inspired by Deleuze's 'time-image' theory on how movement subordinates itself to time, I aim to challenge traditional interactive film approaches by creating an interactive narrative. This narrative blends the infinite continuity of time with the inherent metaphors of interaction in nature(including body), preserving human's autonomous initiative in human-computer interactions.

APPROACH & TOOLS

PART I
PERSONAL MEMORY & INHERENT BODY MOVEMENT - EEG TO CITY VISUALIZATION PIPELINE

PART II
From Personal to Collective Memory - Body Movement and Mobility in Urbanism

CORE CONCEPT



During the research, it was discovered that the model, designed to decipher brain wave data and adjust city structures along with game engine camera parameters to influence city’s imagery, surprisingly yet appropriately aligns with Sigmund Freud's concept of condensation in dreams. This theory describes how distinct elements merge to form singular dream images (Freud, 1900).

To explore the neural dynamics linked to the recollection of urban environments, I initiated a preliminary study involving a guided meditation session lasting 20-30 minutes. This session was flanked by pre- and post-session assessments of resting-state brain activity, culminating in a subsequent interview.

Image of the City, Cognitive Mapping, and the Other

How to capture the spirit of a city in an image? How do we engage with strangers, and how do our perceptions relate to them and urban structures in motion?

A city street is equipped to handle strangers (Jane Jacobs, The Death and Life of Great American Cities, 1961). Inspired by the urban imagery in Ellen Poe's Man in the Crowd and Calle's Suite Vénitienne, I undertook a brief experiment in New York: I followed a stranger during a moment of positive emotion (high valence) and retraced the path on a different day while experiencing negative emotion (low valence).

CORE CONCEPT

Translating Brain Activity to  Mental Structure, and Hence City Configuration Through Emotions

Emotions reflect our immediate mental state. When memories are triggered, emotions meld current feelings with past recollections.

Building on Mehrabian and Russell’s VAD (Valence-Arousal-Dominance) model [1], I’ve devised a model to translate real-time EEG data into city configurations: Valence-Arousal determines city structure parameters, while Dominance influences camera settings within Unreal Engine and Touch Designer.

The Valence-Arousal classification was conducted using the commercial MUSE 2 EEG headband with four electrodes[2]. The pipeline, executed in Python, was integrated from Touch Designer into Unreal Engine to manage the city parameters' PCG graph.

Dominance defines control over emotions, distinguishing emotions with similar arousal and valence levels. Various factors influence an image's dominance perception. Due to MUSE 2's technical constraints, the model classifying dominance levels using the beta/alpha ratio from frontal lobe electrodes (FC7 & F8) can't be applied[3]. In this model, dominance relates to a subject's perceived control over the imagined city, which, if viewed as a photographed image, correlates to the camera's depth of field and lens size.

DEVELOPMENT PROCESS I

Visualizing Mental Structure Through Brain Activity

DEVELOPMENT PROCESS II

Code: Land Mark Detection + SMS

DEVELOPMENT PROCESS III

Immersive Experience Design
& Spatial Design

(1)The interactive-immersive installation and the GPS + Web Page components are under development.

(2)A demo has been conducted to test the entire system, transforming the public interaction into a personal experience by utilizing blink detection on the MUSE device to capture screenshots. The film itself has been showcased at multiple venues.