Google's researchers have used a headset removal technology to enhance Mixed Reality, allowing the medium to not only convey a VR user's interaction with the virtual environment but also show their face in a natural fashion.

Virtual reality lets you experience amazing things - from exploring new worlds, to painting with trails of stars, to defending your fleet to save the world. However, if you're watching someone else use VR, it's hard to tell what's going on and what they're seeing. And if you're in VR with someone else, there aren't easy ways to see their facial expressions without an avatar representation.

Daydream Labs and Google Research teamed up to start exploring how to solve these problems. Using a combination of machine learning, 3D computer vision, and advanced rendering techniques, the researchers are now able to "remove" headsets and show a person's identity, focus and full face in mixed reality. Mixed reality is a way to convey what's happening inside and outside a virtual place in a two dimensional format.

Using a calibrated VR setup including a headset (like the HTC Vive), a green screen, and a video camera, combined with accurate tracking and segmentation, you can see the "real world" and the interactive virtual elements together.

The first step to removing the VR headset is to construct a dynamic 3D model of the person's face, capturing facial variations as they blink or look in different directions. This model allows us to mimic where the person is looking, even though it's hidden under the headset.

Next, the researchers use an HTC Vive, modified by SMI to include eye-tracking, to capture the person's eye-gaze from inside the headset. From there, they create the illusion of the person's face by aligning and blending the 3D face model with a camera's video stream. A translucent "scuba mask" look helps avoid an "uncanny valley" effect.

Finally, the researchers composite the person into the virtual world, which requires calibrating between the Vive tracking system and the external camera. The end result is a complete view of both the virtual world and the person in it, including their entire face and where they're looking.

Headset removal is poised to enhance communication and social interaction in VR itself with diverse applications like VR video conference meetings, multiplayer VR gaming, and exploration with friends and family.