To explore the possibility of bringing the sense of touch into VR, Roland and I attended the Reality Virtually Hackathon 2019, a three day hackathon that brings together interdisciplinary minds to create new experiences using VR, AR and MR at MIT Media Lab. How can we come up with an idea and form a team to build a proof-of-concept within three days?
In most VR games, we are sitting in a small space alone looking into a boundless world. What if we revert the dynamic, experience VR in a swimming pool, free from the constraint of gravity while we interact with the environment above us? How does it feel like being a small fish looking at the world outside of the fish bowl? With these questions in mind, Roland and I decided to create a performative VR sculpture that resembles life through the lens of a goldfish.
We pitched our idea to 300+ hackathon attendees and found three like-minded creatives to join our team: Huiyi (a Masters Candidate at ITP with expertise in interactive performance), Connor (a PHD Candidate at MIT focused on Lidar research) and Maxime (a game music developer from Berklee College of Music).
As a team, we finalized our storyboard and divided production based on our expertise and interest: Roland on creating and attaching fins to user’s hands with Leap Motion in Unity.
Maxime on sound design using Wwise. Connor on interaction design for the cat paw using Unity and HTC Vive.
Huiyi and I on developing the environment and visual effects in Unity. I also served as the point person for our team which I attended meetings with the hackathon organizing team, documented our process and oversaw our project archive.
Two players are involved. Player 1 is put into the perspective of a goldfish via a VR headset and a Leap motion that turns Player 1’s hands into fins. Player 1 explores the world he/she lives in contained by a fish tank. Player 2 acts as a cat holding a cat paw with tracker attached. Player 2 uses his/her paw to attack Player 1 where his/her claw will appear in Player 1’s point of view.
User Testing I
The virtual cat paw appeared to be effective due to users’ responses. Many found the approaching claw scary.
Besides using software, we also built a physical cat paw out of recycling materials we found at MIT’s junkyard and attached it to a tracker.
User Testing II
We showcased our project at Reality Virtually’s Public Expo on January 21, 2019.
Little kids loved our project!
Especially when given the cat paw to attack Player Goldfish.
Adults enjoyed our Goldfish experience too!
It was a competition for some users.
Some Goldfish Players liked to fight back.
While others enjoyed the haptic feedback offered by the cat paw.
We successfully completed our cat interaction with a swimming goldfish as a proof of concept. More importantly, we are very happy that we not only created a project that brought so much joy into people’s lives but also bonded with interdisciplinary minds who shared the same passion.
We would like to experiment how the sensations of our project can be enhanced by submerging in a swimming pool. We are in conversation with a fellow hacker we met during the Hackathon, Pierre Friquet, who have created an underwater VR headset.
We are also interested in how haptics can improve motor functions and hand accuracy of stroke and parkinson's patients and would like to work with institutions to extend our research.
Meanwhile, we envision Goldfish to be an interactive installation at an aquarium or a science museum.