Spontaneous Adventure

Narrative / Storytelling
Immersive Experience
Installation

Duration

Oct.2022 - May.2023

Tools

Unreal Engine
iClone
Character Creator
StableDiffusion
Motion Builder
OptiTrack
Madmapper
Topaz
FinalCut
Adobe Premiere
Ableton Live
Isadora

Description

Spontaneous Adventure, is an introspective mixed reality experience that uses 5-view projection technology and immersive audio to tell a story of how gender oppression affects my dreams.

“Some memories lurk deep inside our brains, uncanny but having a lasting impact on us. These stories may come from the past, the future, or the collective subconscious. They are fictive, yet undeniably real, vividly expressing our deepest fears and desires.”

Overview

In this project, I took on multiple roles as a director, scriptwriter, multimedia artist, and collaborated with dancers and composer. The experience consists of three stages, smoothly transitioning with a hypnosis fade-in and a meditation fade-out. The central narrative follows a girl who confronts recurring nightmares that blur the line between reality and dreams. With the guidance of her psychologist, she embarks on an adventure to discover the root cause of everything.

Context

Since childhood, dreams have been my best escapism utopia. I have captured these fleeting memories and attempted to translate them into various forms of media for seven years, including books, motion graphics, sculptures and so on.

The reason focus me to create this project, is one day I suddenly realized that there is an elephant in my dream. It looks like I am full of expectations and ambitions for everything on the outside. But I'm the only one who knows how many times I almost didn't survive from the murderer’s hands in my dreams. Thanks to the excellent human defence mechanism, when we encounter extreme pressure, the hippocampus automatically shuts down. Which means we can't remember that event, but the intense emotions will maintain by amygdala. No pain is not a good thing; you might bleed to death without realising it. That’s why when dreams can help us spot early-stage and inconspicuous mental injuries is truly helpful.

I want more people to know that nightmares are not poison; it’s antidotes. The dream is a way to communicate with yourself. Spontaneous Adventure is an ode to girls and marginalized individuals who endure countless challenges and dilemmas in their daily lives.

Details

This project presented significant challenges for me, particularly in conveying emotions and storytelling within a non-VR space and coordinating multiple channels. I faced complexities in composition, color, and narration. To overcome these challenges, I delved into research areas such as emotional color palettes, body expressions, nested metaphoric storytelling, and spatial music. I experimented with various spatial narrative approaches and worked closely with team members to create specific emotional atmospheres within the virtual environment.

To optimize my workflow, I utilized a range of tools and technologies. I used Daz, CharacterCreator, and Unreal Metahuman to create characters, and employed Optitrack motion capture, motionbuilder, iclone, live face, recorder, and Unreal Sequence for smooth and realistic character animations.

Given the fluid and rapidly changing nature of AI-generated video, which aligns well with the concept of dreams, I employed techniques like stablediffusion. This involved using the white model animations generated from Unreal Sequence and converting them into image sequences through PR. By combining ControlNet, Lora, and Prompt, I achieved a stable visual style and continuous, smooth AI-generated videos. Finally, I utilized Ae's deflicker to eliminate jitter and Topaz to enhance the image resolution.

For multi-channel collaboration, I employed Ndisplay, Switchboard, and Offworld Viewport Export to achieve surround projections in the virtual world, including perspective distortion. I used PR and Final Cut for the editing of all video materials.

The projection aspect involved the use of Isadora and MadMapper for projecting surrounding imagery and the central installation, respectively.

In terms of audio, we utilized 7.1.4 Dolby surround sound, combined with perspective, to create an immersive experience. Skillful use of subwoofers allowed the audience to feel pronounced vibrations, adding a physical sense of pressure.

Poster & Postcard

Presentation

Next Step

The project received positive feedback from 15 shows with around 200 attendees, who found the 10-minute duration insufficient and expressed anticipation for more content. Improvements were made based on audience feedback, enhancing the clarity of the narrator's voice and slowing down the meditation section for a more immersive experience. The goal is to secure a more permanent venue for "Spontaneous Adventure" and apply for festivals and grants while continually enhancing the project.

Credits

Director: Chenyan Yu
Composer: Emil Bergh
Dancer: XInyi Zhang, Chenyan Yu
Translation Proofreader: Ziwen Pang
Writer Mentor: Wei Wu
Senior Technician: Erik Bell
Audio Technician: Trip Hopper, Delong Lin

Writer: Chenyan Yu
Animation: Chenyan Yu
Unreal Engine developer: Chenyan Yu
Editor: Chenyan Yu
Graphic Designer: Chenyan Yu

Voice actor
Girl: Chenyan Yu
Psychologist: Emil Bergh

Supervisor: Mimi Yin, Gabe Barcia Colombo
Jonathan Turner, Todd Bryant

Chenyan Yu