top of page
A participatory, multi-sensory experience for museum visitors to develop a deeper understanding of the intertwined relationship between humans and the natural world.
2021
Entangled
CONTEXT
MPS/MDes IxD Studio
Carnegie Mellon University
Fall 2021 (6 Weeks)
TEAM
Devika Pillai
Laura Pedrosa
Lokesh Fulfagar
Shih-Hsueh Wang
Advised by Professors Kyuha Shim and Dina El-Zanafy
ROLES
Concept Ideation
Physical Computing
Projection Content Animating
3D Modeling
Physical Env. Setting
Video Editing
TOOLS
After Effects
Premiere Pro
Arduino IDE
Rhino
Overivew
Entangled is a proposition of exhibition with multisensorial experiences for the Carnegie Museum of Natural History visitors to develop a deeper understanding of the intertwined relationship between humans and the natural world. It consists of a series of interactive installations and devices, including immersive projection, objects recognition, motion capture, and tangible interaction with sound feedback, to create a consecutive learning process through spatial experience.
Concept Video
Overview
Design Principles
Design Principles -
Museums as a public space for critical dialogue.
-
Showcase diversity in experience by leveraging the physical space and user tools available at hand.
-
Design a gamified experiences that entice users to see relevance in the curated stories around the museum.
The hall of botany is the particular area for the Entangled exhibition, where is currently less popular than other showrooms with lively animal dioramas.
Plants constitute the foundation of ecosystems that breed diverse creatures. However, human impacts from daily activities have been accelerating the change in the natural world.
Problem Statement -
How might we design an interactive learning experience to help the museum transform passive spectators to active participants who make conscious choices in their daily lives?
Problem Statement
Design Solution -
1. Enhancing museum visiting experiences
+
-
Information browsing in Immersive Environments (AR)
+
-
Wayfinding using digital projection mapping
-
Experiential Learning using posture detection
+
-
Social interaction through a tangible interface
+
2. Three core experiences
Discover and Learn
Uncover the human impact on plants across ecosystems.
Interact and Understand
Get immersed in multi-sensorial activities inside the Hall of Botany.
Pledge and Reflect
Reflect on how changing your lifestyle can improve the planet.
Design Solution
Multisensorial Experience Design -
1. Discover and Learn
Technology: AR via mobile phone
Identify and learn about plants anywhere in the museum with details on the major threats they face from human impact.
Featured Plants
AR Tags
Information Cards
2. Interact and Understand
Technology: Posture Detection and Screen Display, Projection Mapping and Object Detection and RGB LED Strip
Interactive Wall
Serves as an entry point to the dioramas and helps you learn more about the plants in the Hall of Botany.
Choose Diorama
Transform to Plant
System Diagram
(Drawn by Lokesh Fulfagar)
Diorama Activity
Visitors play with the dioramas in a series of activities. They are rewarded with a visual feedback on the installation tree and a learning card about human impact.
1.
According to the ecosystems respectively displayed immersive projections in front of each diorama.
Projected Instructions
Open Play tab
Associated Game Card
Projection Triggered
2.
Following the hint to start the interactive game.
3.
Using phone camera through the museum app to detect projected objects and collect them.
4.
Completing the mission to receive a success card, including further details, leaving messages, and guidance for the next.
5.
LED strips are gradually lit up toward the central tree, and one leaf representing the player is bloomed amidst the canopy on the ceiling projection.
System Diagram
(Drawn by Lokesh Fulfagar)
3. Pledge and Reflect
Technology: Visual, Audible, and Tactile Interface
Pledge Wall
Visitors interact with the pledge wall and leave a voice note. They will also be prompted at the end of each diorama activity.
Leave messages at the end of the activity
Listen to the messages as a separate experience after interacting with the wall.
System Diagram
(Drawn by Lokesh Fulfagar)
Multisensorial Exp.
Holisic Visiting Exp.
Holistic Visiting Experience -
Museum Visitors Journey
Diagram made by Laura Pedrosa
Scan App & Onboarding
Discover & Learn
Enter the hall & Interact
Play & Learn
Receive Feedback
Make pledges & Reflect
Journey at the Hall of Botany
Design Process -
1. Exploratory Research
On-site Observation
In our first meeting, our team had come up with a common idea of building an actual interaction demo with an immersive experience as the final outcome. Therefore, in exploring specific areas within the museum as our site target, we were looking for somewhere applicable for projection mapping and tangible interaction. We summarized with two options, which are mammal zone and botany hall. The former includes almost the entire second floor of the museum that we might focus on designing the visiting experience of museum journey, which could be applied to other museums as well. On the other hand, the hall of botany is more specific that we could explore the depth of the interaction technology and make the design more condensed. Plus, considering the spatial characteristics of the hall of botany, such as lighting, dome-like ceiling, and showcases along a circular path, it is a fantastic room for creating immersive projection to enhance the visiting experience of the botany hall.
Mammal zone
Botany hall
Secondary Research
We first researched the pain point of the museum, which we found out that, compared to animal areas with lively diorama, there are fewer visitors at the botany hall, and people spend only a short time staying in the room. After we decided on the target area, we started to discuss the importance of the role the plants play, whether to the museum or the earth.
Human impact is the entry point of the concept for the exhibition we intended to design. However, instead of showing problems and solutions, we managed to lead visitors to learn different kinds of plants that grow in different regions with distinct ecosystems, confronting certain threats caused by human activities. For example, marine pollution affects coast environments, and air pollution accelerates the desertification of highland regions, etc. In terms of interaction design, even though we encouraged visitors to save the planet through their proactive action, we preferred not to directly control the environment, such as adjusting the temperature to see the before-and-after change. Instead, visitors should start saving nature with daily human activities like removing floating trash and growing more plants. By doing so, we expect the visitors can be resonated with the action and reflect on their everyday experience.
Team meeting
Early proposal for the botany hall
(Projection/Motion Capture/Scents/Sounds)
2. Defining Objective & Ideation
Experiential learning and social interaction were our main design goals that could be achieved through projection mapping since it prompted multiple visitors to interact, see, or hear the same displayed objects. Regarding the multiple visitors, we targeted on multi-generation visitors that kids could play the game with their parents, and parents introduced the idea to their kids. Given the concept, the exhibition did not include complicated knowledge, it emphasized the prompts that motivated people to make a change of our life.
On top of using projection mapping for each diorama, how to begin and to close the experience were two questions that we kept ideating. We had gone back and forth with ideas from both activity and technology aspects, combined with the spatial experience. For instance, I could add AR to display an animal walking into the hall of botany to guide visitors to enter the exhibition from the mammal zone. In the end, we integrated our divergent concepts and mainly used projection with other techniques in support. Thus, a ground-projected flower guided visitors to the interactive wall as a start, and a wall-projected pledge wall prompted people to record messages and reflect as a conclusion.
Discussing possible proposals
Using storyboard to explore the entire journey
(Made by Devika Pillai)
3. Prototyping for Live Demo
Since projection mapping relies on actual environment when performing, we made scaled, physical models to test the effect of the projection. Before we actually build a tree for demo, we studied the relationship between the tree and the space under the projection through the physical models. It successfully visualized the concept and it allowed others to review our design effectively.
Ceiling-mounted screen display research
(Model and animation made by Shih-Hsueh Wang)
Ground-projected shadow research
(Model made by Shih-Hsueh Wang)
Due to the tight schedule before the final demo, we could not handle the connection between the display/hide of projected objects and physical objects that are modals to control the projection via motion capture. Instead, we use object detection techniques through phone cameras by applying and revising the open codes from COCO SSD. Based on the technique, visitors could use their mobile phones to detect projected objects and send data to the computer to control the projection.
In terms of interactive wall, we used PoseNet to detect the posture and mapping it to the svg illustration using PoseNet animation.
Object detection test
(Made by Lokesh Fulfagar)
Body motion detection test
(Made by Lokesh Fulfagar)
4. Physical Environment Setting
In order to conduct the live demo, we need to build up the physical environment for the entire multisensorial experience. Therefore, we set up the room with the same order of visiting circulation: interacting with the interactive wall, playing with the immersive projection, receiving feedback from the lit-up tree, and listening to messages from the pledge wall.
Setting up structures for the diorama and projector
Testing the projection on both floor and wall
Building cardboard-made tree
Setting LED strip through Arduino
Checking animation for ceiling projection
Testing LED strip
5. Live Demo
In the final design week of the School of Design, we demonstrated our outcome to the publics. We turned the room into an immersive space that is as close to real botany hall as possible. In the live demo, we explored technologies that could aid to achieving the similar experience: Posture detection and Animation mapping (Interactive Wall), Projection mapping and Object detection (Diorama Activity), Physical computing via Arduino and RGB LED strip (Tree), Bare touch board and malleable conductive tape (Pledge Sound Wall).
Takeaways -
Reflection
We successfully built up an effective live demo for people to interact and engage in the multisensorial experience. On the other hand, there are also some parts that we could not cover within six weeks. Combining with the feedback we have received, a few points we are keeping in mind to answer in the future:
-
How can we make the diorama activity multi-person?
-
How could we have used tangible objects to allow for a mobile-free immersive experience?
-
What are some of the other diorama activities we can explore?
Learnings
Personally, I have a few points that I learned from this project regarding my role in creating projection content and setting up the immersive environment:
-
Content
-
When making projected content, whether using Processing, p5js, or After Effects, pay close attention to Scale (considering human body scale), Color (contrast between objects and the background), Motion (the rhythm of movement).
-
-
Installation
-
When setting up the environment, if having multiple projects, blending each projection area is a huge work; if only having one projector, to avoid the shadow of participants, we might need to place the projector in a non-central edge position. However, it causes a tilted shape of the projection that needs to be adjusted through projector setting and projected contents.
-
Due to the limited area of the room, we were not able to hang the projectors above, so we made a projector holder fixed on a ladder to project onto the floor. Yet, the projection area is constrained because of the short distance.
-
-
Learn from actual experience:
-
We didn't know the problems above until we set everything up, and kept trying to solve them one by one on the spot.
-
Projection mapping is a contextual communication process between virtuality and physical environments.
-
Takeaways
Design Process
bottom of page