top of page

A participatory, multi-sensory experience for museum visitors to develop a deeper understanding of the intertwined relationship between humans and the natural world.

2021

Entangled

CONTEXT

MPS/MDes IxD Studio
Carnegie Mellon University
Fall 2021 (6 Weeks)

TEAM

Devika Pillai
Laura Pedrosa
Lokesh Fulfagar
Shih-Hsueh Wang

Advised by Professors Kyuha Shim and Dina El-Zanafy

ROLES

Concept Ideation
Physical Computing
Projection Content Animating
3D Modeling
Physical Env. Setting
Video Editing

TOOLS

After Effects
Premiere Pro
Arduino IDE
Rhino

Overivew


Entangled is a proposition of exhibition with multisensorial experiences for the Carnegie Museum of Natural History visitors to develop a deeper understanding of the intertwined relationship between humans and the natural world. It consists of a series of interactive installations and devices, including immersive projection, objects recognition, motion capture, and tangible interaction with sound feedback, to create a consecutive learning process through spatial experience.



Concept Video



 


Overview
Design Principles

Design Principles -

Museums as a public space for critical dialogue.

  • Showcase diversity in experience by leveraging the physical space and user tools available at hand.

  • Design a gamified experiences that entice users to see relevance in the curated stories around the museum.

Anthropocene.gif

The hall of botany is the particular area for the Entangled exhibition, where is currently less popular than other showrooms with lively animal dioramas.

Plants constitute the foundation of ecosystems that breed diverse creatures. However, human impacts from daily activities have been accelerating the change in the natural world.

Problem Statement -

How might we design an interactive learning experience to help the museum transform passive spectators to active participants who make conscious choices in their daily lives?

Problem Statement

Design Solution -

1. Enhancing museum visiting experiences

+

  • Information browsing in Immersive Environments (AR)

+

  • Wayfinding using digital projection mapping

  • Experiential Learning using posture detection

+

  • Social interaction through a tangible interface

+

2. Three core experiences

Discover and Learn

Uncover the human impact on plants across ecosystems.

3 core experience.png
3 core reflection.png

Interact and Understand

Get immersed in multi-sensorial activities inside the Hall of Botany.

3 core experience.png
3 core reflection.png

Pledge and Reflect

Reflect on how changing your lifestyle can improve the planet.

3 core experience.png
3 core reflection.png
Design Solution

Multisensorial Experience Design -

1. Discover and Learn

Technology: AR via mobile phone

Identify and learn about plants anywhere in the museum with details on the major threats they face from human impact.

discover and learn.png

Featured Plants

AR Tags

Information Cards

2. Interact and Understand

Technology: Posture Detection and Screen Display, Projection Mapping and Object Detection and RGB LED Strip

Interactive Wall

Serves as an entry point to the dioramas and helps you learn more about the plants in the Hall of Botany.

interact and understand 01.png

Choose Diorama

moving flower.gif

Transform to Plant

System Diagram
(Drawn by Lokesh Fulfagar)

interactive wall system.png

Diorama Activity

Visitors play with the dioramas in a series of activities. They are rewarded with a visual feedback on the installation tree and a learning card about human impact.

diorama-w-beach.gif

​1.
According to the ecosystems respectively displayed immersive projections in front of each diorama.

projection instruction.png

Projected Instructions

Open Play tab

Associated Game Card

Projection Triggered

​2.
Following the hint to start the interactive game.

Entangled.gif

3.
Using phone camera through the museum app to detect projected objects and collect them.

success card.png

4.
Completing the mission to receive a success card, including further details, leaving messages, and guidance for the next.

tree.gif

​5.
LED strips are gradually lit up toward the central tree, and one leaf representing the player is bloomed amidst the canopy on the ceiling projection.

System Diagram
(Drawn by Lokesh Fulfagar)

Projection system.png

3. Pledge and Reflect

Technology: Visual, Audible, and Tactile Interface

Pledge Wall

Visitors interact with the pledge wall and leave a voice note. They will also be prompted at the end of each diorama activity.

Pledge wall entry point.png

Leave messages at the end of the activity

Listen to the messages as a separate experience after interacting with the wall.

System Diagram
(Drawn by Lokesh Fulfagar)

Pledge wall system.png
Multisensorial Exp.
Holisic Visiting Exp.

Holistic Visiting Experience -

Museum Visitors Journey

Journey Map.png

Diagram made by Laura Pedrosa

Scan App & Onboarding

1 Onboarding.jpg
1-2.jpg

Discover & Learn

2 Discover.jpg
2-2.jpg

Enter the hall & Interact

3 Plant wall.jpg
3-2.jpg

Play & Learn

4 Projection.jpg
4-2.jpg

Receive Feedback

5 Tree.jpg

Make pledges & Reflect

6-2.jpg
6 pledge.jpg

Journey at the Hall of Botany

220105_3D diagram.gif

Design Process -

1. Exploratory Research

On-site Observation

​In our first meeting, our team had come up with a common idea of building an actual interaction demo with an immersive experience as the final outcome. Therefore, in exploring specific areas within the museum as our site target, we were looking for somewhere applicable for projection mapping and tangible interaction. We summarized with two options, which are mammal zone and botany hall. The former includes almost the entire second floor of the museum that we might focus on designing the visiting experience of museum journey, which could be applied to other museums as well. On the other hand, the hall of botany is more specific that we could explore the depth of the interaction technology and make the design more condensed. Plus, considering the spatial characteristics of the hall of botany, such as lighting, dome-like ceiling, and showcases along a circular path, it is a fantastic room for creating immersive projection to enhance the visiting experience of the botany hall.

IMG_2602.JPEG

Mammal zone

IMG_2785.JPEG

Botany hall

Secondary Research

​We first researched the pain point of the museum, which we found out that, compared to animal areas with lively diorama, there are fewer visitors at the botany hall, and people spend only a short time staying in the room. After we decided on the target area, we started to discuss the importance of the role the plants play, whether to the museum or the earth. 

Human impact is the entry point of the concept for the exhibition we intended to design. However, instead of showing problems and solutions, we managed to lead visitors to learn different kinds of plants that grow in different regions with distinct ecosystems, confronting certain threats caused by human activities. For example, marine pollution affects coast environments, and air pollution accelerates the desertification of highland regions, etc. In terms of interaction design, even though we encouraged visitors to save the planet through their proactive action, we preferred not to directly control the environment, such as adjusting the temperature to see the before-and-after change. Instead, visitors should start saving nature with daily human activities like removing floating trash and growing more plants. By doing so, we expect the visitors can be resonated with the action and reflect on their everyday experience.

IMG_2797.HEIC

Team meeting

6C58A368-7284-4022-B99E-9C16D1BFBEC2_1_105_c.jpeg

Early proposal for the botany hall
(Projection/Motion Capture/Scents/Sounds)

2. Defining Objective & Ideation

Experiential learning and social interaction were our main design goals that could be achieved through projection mapping since it prompted multiple visitors to interact, see, or hear the same displayed objects. Regarding the multiple visitors, we targeted on multi-generation visitors that kids could play the game with their parents, and parents introduced the idea to their kids. Given the concept, the exhibition did not include complicated knowledge, it emphasized the prompts that motivated people to make a change of our life. 

On top of using projection mapping for each diorama, how to begin and to close the experience were two questions that we kept ideating. We had gone back and forth with ideas from both activity and technology aspects, combined with the spatial experience. For instance, I could add AR to display an animal walking into the hall of botany to guide visitors to enter the exhibition from the mammal zone. In the end, we integrated our divergent concepts and mainly used projection with other techniques in support. Thus, a ground-projected flower guided visitors to the interactive wall as a start, and a wall-projected pledge wall prompted people to record messages and reflect as a conclusion.
 

IMG_2811.jpeg

Discussing possible proposals

877E91AD-9063-4011-82A1-0DE17161AA85_1_105_c.jpeg

Using storyboard to explore the entire journey
(Made by Devika Pillai)

3. Prototyping for Live Demo

Since projection mapping relies on actual environment when performing, we made scaled, physical models to test the effect of the projection. Before we actually build a tree for demo, we studied the relationship between the tree and the space under the projection through the physical models. It successfully visualized the concept and it allowed others to review our design effectively.
 

Physical-model-w-screen.gif
IMG_2714.gif

​Ceiling-mounted screen display research
(Model and animation made by Shih-Hsueh Wang)

Ground-projected shadow research
(Model made by Shih-Hsueh Wang)

​Due to the tight schedule before the final demo, we could not handle the connection between the display/hide of projected objects and physical objects that are modals to control the projection via motion capture. Instead, we use object detection techniques through phone cameras by applying and revising the open codes from COCO SSD​. Based on the technique, visitors could use their mobile phones to detect projected objects and send data to the computer to control the projection.

In terms of interactive wall, we used PoseNet​ to detect the posture and mapping it to the svg illustration using PoseNet animation. 

IMG_2851.HEIC
IMG_2791.png

Object detection test
(Made by Lokesh Fulfagar)

Body motion detection test
(Made by Lokesh Fulfagar)

4. Physical Environment Setting

​In order to conduct the live demo, we need to build up the physical environment for the entire multisensorial experience. Therefore, we set up the room with the same order of visiting circulation: interacting with the interactive wall, playing with the immersive projection, receiving feedback from the lit-up tree, and listening to messages from the pledge wall.

IMG_3060.HEIC
IMG_5911.png

Setting up structures for the diorama and projector

Testing the projection on both floor and wall

IMG_2889.HEIC
IMG_3047.HEIC

Building cardboard-made tree

Setting LED strip through Arduino

IMG_3243.HEIC
IMG_3239.HEIC

Checking animation for ceiling projection

Testing LED strip

5. Live Demo

In the final design week of the School of Design, we demonstrated our outcome to the publics. We turned the room into an immersive space that is as close to real botany hall as possible. In the live demo, we explored technologies that could aid to achieving the similar experience: Posture detection and Animation mapping (Interactive Wall), Projection mapping and Object detection (Diorama Activity), Physical computing via Arduino and RGB LED strip (Tree), Bare touch board and malleable conductive tape (Pledge Sound Wall). 

IMG_3062.jpeg
IMG_3400.HEIC
IMG_3415.HEIC
IMG_3410.HEIC
IMG_3384.JPG

Takeaways -

Reflection

​We successfully built up an effective live demo for people to interact and engage in the multisensorial experience. On the other hand, there are also some parts that we could not cover within six weeks. Combining with the feedback we have received, a few points we are keeping in mind to answer in the future:

​

  • How can we make the diorama activity multi-person?

  • How could we have used tangible objects to allow for a mobile-free immersive experience?

  • What are some of the other diorama activities we can explore?

Learnings

Personally, I have a few points that I learned from this project regarding my role in creating projection content and setting up the immersive environment:

​

  • Content

    • When making projected content, whether using Processing, p5js, or After Effects, pay close attention to Scale (considering human body scale), Color (contrast between objects and the background), Motion (the rhythm of movement).​

​​

  • Installation

    • When setting up the environment, if having multiple projects, blending each projection area is a huge work; if only having one projector, to avoid the shadow of participants, we might need to place the projector in a non-central edge position. However, it causes a tilted shape of the projection that needs to be adjusted through projector setting and projected contents.

    • Due to the limited area of the room, we were not able to hang the projectors above, so we made a projector holder fixed on a ladder to project onto the floor. Yet, the projection area is constrained because of the short distance.

​​

  • Learn from actual experience:

    • We didn't know the problems above until we set everything up, and kept trying to solve them one by one on the spot.

    • Projection mapping is a contextual communication process between virtuality and physical environments.

Takeaways
Design Process
bottom of page