ConVRsation : An Exploration in Immersive Communication Design

In early december 2020 I participated in a particular type of hackathon : a designathon

Called Interhackt, organized by https://twitter.com/azlenelza & other designers, the objective during a week was to explore and flesh out new design concepts around the theme “Making & Understanding Together”

Interested in working on this theme through the prism of VR with Unity, I was lucky to find another Unity dev, https://twitter.com/lukasdonkers, to jam with.

In summary, this was a very rewarding VR project that ended in us winning the first prize in the hackathon among 30+ other interesting projects.

Personally I learned and progressed a lot, notably through:

• Organizing brainstorming & collaboration with the Miro whiteboard software
• Implementing a VR player in Unity using the XR Interactions Toolkit SDK with an Oculus Quest headset
• Learning the Normecore Networking SDK and implementing multiplayer code in C#
• Writing an in depth presentation for the project submission to communicate design ideas & vision

Here’s the full story of what we worked through during that week and what came out of it.

Initial brainstorming & tests

Through an initial brainstorming in Miro, we quickly decided on creating a collaborative VR Mind Mapping tool & to find a use case from there.

By creating and connecting nodes in a 3D space, you’d have great freedom over the layout and structure.

We found our specific graph use case by plugging in the Normcore networking solution and meeting in VR as friendly avatars.

https://normcore.io/documentation/assets/media/working-avatar.3d41d6dd.mp4

By simply having a conversation with embodied 3D characters, we realized that VR as a conversational medium was an excellent thread to follow, hence “ConVRsation”.

One of the most powerful aspects of interacting and collaborating in VR is the Sense of Presence.

Just being able to see where someone is looking, and seeing their body language and gestures, you can :

• Engage better with each other
• Communicate more fluidly & naturally
• See the focus of each participant & collaborate better

Making in-person conversations tangible

We then decided to explore how the flow & items of conversation could also be a physical presence between the two participants.

We imagined that throughout the conversations, nodes representing ideas & thoughts would emerge, connected through a thread.

They would have the ability to revisit that same conversation in the same way you would revisit a physical place.

You could even find ways to branch out and explore alternate paths from that conversation.

Tools for making the mapping process seamless

In VR, using a keyboard ends up being very cumbersome. For that reason, we decided to have all text input through voice transcription. We implemented Google Speech to Text which worked perfectly.

Next we decided to give to the users a palette of core sentence types :

• Subject: To set the focus of the conversation, working as a visual anchor in the scene.
• Idea: To communicate ideas in the most general sense
• Question: To prompt a response in the next connected node
• Pros: To show arguments in favor of a point
• Cons: To show opposition to a point

These basic building blocks can be used to categorize all elements of conversation. The overall result was easy to navigate at a glance through color coding.

To reduce friction in adding content, we made the nodes available at the waist level, so you could grab and place them easily. This quick process allowed users to maintain a pace close to normal spoken conversation.

Emergent behaviors

One of the most interesting ideas we got to explore, is how this setup impacts the attitude of the participants themselves.

Presenting these node types invites the speakers to become more intentional about what they say

• A question node can be placed front and center to become the focus of the conversation until it is resolved.
• Pros & cons nodes can be used to lay out arguments in situations of decision making for a simple overview.

Our favorite discovery was how natural and even fun it is to engage with this process. The entire setting brings in a sense of play to the conversation, which feels much more like an exploration, or a process of co-creation.

Here’s a conversation we recorded during testing. As we played with our prototype, satisfying structures emerged along the way, pulling us back into previous subjects.

Future ideas to expand on this concept

One of the first ideas is enabling more variety of elements. Resources like images, 3D models, video players and webviews could be used as starting points or support elements in the conversation

Another powerful idea would be to allow the participants to drop in diagram templates.

Those would be tools to nudge certain parts of the conversation into recognized frameworks for a variety of uses, such as:

• Debate structures
• Collaborative story building
• Mapping out descriptions, explanations, relationships

You could even imagine taking those spatial constructs out of VR to instantiate them in AR instead, putting them in your own living space.

Truly the possibilities seem endless. Expanding on this idea may well lead to the best playground for conversation one can offer.

I hope you enjoyed reading about our little designathon experience 😊

I intend to continue working on these types of XR use cases & experiences, feel free to follow my work at https://twitter.com/HuguesVincey 👍