Microsoft is now shipping the $3,500 HoloLens 2 to customers, packed with some new features.

Microsoft announced HoloLens 2 back in February, and it was a significant improvement over its predecessor, with a more comfortable fit and a much better field of view: Where the initial unit was limited by a small 16:9 FOV, HoloLens 2 improves the view by about 3X.

HoloLens 2 also includes eye- and hand-tracking systems, has no latency, and the Azure Spatial Anchors functionality that shares spatial maps with others.

A HoloLens 2 user can leave holograms at specific locations and then other users, with HoloLens headsets or AR-based apps on mobile devices, can find and interact with them as well.

The first time people don the new HoloLens 2 on their heads, the device automatically gets to know them: It measures everything from the precise shape of their hands to the exact distance between their eyes.

The artificial intelligence research and development that enabled those capabilities “was astonishingly complicated” but essential to making the experience of using the device “instinctual,” said Jamie Shotton, Microsoft's partner scientist who leads the HoloLens science team in Cambridge, United Kingdom.

“We want you to know how to use HoloLens without having to be taught how to use it,” he said. “We know how to interact with things in the real, physical world: We pick things up, we press buttons, we point to things. We aim, as far as possible, to translate that directly into mixed reality.”

The sensor-packed holographic computing headset uses AI to displace space and time, creating a mixed reality of people, places and things in order to facilitate one of our most basic human impulses: exchanging knowledge.

Microsoft Technical Fellow Alex Kipman said the headset defines the highest watermark of intelligent edge devices – AI-capable technologies that can collect and process data even without a reliable internet connection – and that can then share some or all of that data with the intelligent cloud when connected.

The marriage of the AI in HoloLens 2 with the AI capabilities of Azure, Microsoft’s cloud computing platform, allows heads-up, hands-on workers to learn skills that advance their careers, and makes it possible for people on opposite ends of the Earth who speak different languages to collaborate with a shared sense of physical presence.

To enable instinctual interactions with HoloLens 2, Microsoft developed, trained and deployed AI models onto the device that track people’s hand motions and eye gaze so that, for example, they can perceive a hologram floating in front of them and reach out to resize it or reposition it.

To build the hand tracking system, the team built a rig with a dome of cameras pointing inward that they used to record a diverse range of people’s hands. Then the team used offline cloud processing to build a 3D model capable of representing all human hand shapes and motions.

From this 3D model, the team was able to use computer graphics to render realistic, synthetic images of hands along with synthetic labels to make the model robust across a variety of hand shapes, poses and movements.

The team used this data to train a compact deep neural network, a type of AI algorithm, that fits on the HoloLens’s onboard processor and runs on every frame coming from the device’s depth sensor.

When a new customer puts on a HoloLens 2, the system uses this neural network to help fit a personalized 3D model to the customer’s hands, enabling the precise tracking required to allow instinctual interaction with holograms.

Micrcosoft's team took a similar approach to build and train the eye tracking model, paying close attention to what’s called interpupillary distance, or the distance between the centers of the pupils of the eyes, which varies across people and affects how a person sees near or distant objects.

The result is an eye tracking model that allows the HoloLens 2 to precisely display holograms in front of customers for interaction and manipulation with their hands and eyes.

The hand and eye tracking capabilities, along with other intelligent features such as simultaneous localization and mapping that’s necessary to make holograms appear pinned to the world as a person moves around, are embedded on the HoloLens 2 in the second generation custom chip called a holographic processing unit, or HPU 2.0.

Perception AI on the HoloLens 2 enables people to manipulate and interact with holograms without worrying about what’s called latency – typically the hundreds of milliseconds it takes for data to travel to the cloud, be processed and returned to the edge.

Privacy concerns are another reason to do AI calculations locally on a device; the iris scans that HoloLens 2 performs to authenticate customers are the type of personal data people may not want sent to the cloud.

For many other types of data, however, there’s an advantage to sending it to the cloud: Once there, the customer can take advantage of Azure AI and mixed reality services and combine the data from their device with data from throughout the ubiquitous computing fabric. That allows for more advanced computation or cognition.

A key advantage of intelligent cloud-powered holographic computing is the ability to share information with others who have a HoloLens or another device with similar capabilities, said Marc Pollefeys, the director of Microsoft’s Mixed Reality and AI Zurich Lab in Switzerland.

Pollefeys is leading a team that develops core computer vision algorithms for a mixed reality cloud service called Azure Spatial Anchors that allows holograms to persist, locked in the real world, for anyone with the appropriate level of access to view.

For example, spatial anchor technology allows a manager in a factory to place holograms next to equipment on an assembly line that contain vital, real-time operating and maintenance information that any credentialed worker with a mixed reality capable device can access.

“If I can only place information that I will see back on my device, it’s probably never worth placing holograms in the world, but if I can annotate the world and afterward anyone else in the company that has the right access can see all of the information, it is suddenly much more valuable,” Pollefeys said.

To create this capability, Pollefeys and his team developed AI computer vision algorithms that process data from sensors to extract 3D geometric information about the environment and piece it together in the cloud to create a digital twin, or map, of the area of interest.

HoloLens has always built up a 3D or spatial understanding of its environment to function. Azure Spatial Anchors creates, refines and shares these maps across devices, Pollefeys noted. That’s why the maps from individual devices are pieced together and stored in the cloud.

These maps get denser, more precise and robust over time as different mixed reality capable devices – HoloLenses as well as properly equipped phones, tablets and laptops – map their environment and share the data with the cloud.

For example, the map of the factory floor where the manager left holograms floating over pieces of equipment on the assembly line is steadily refined as more and more credentialed workers view the holograms with their devices.

This capability also enables scenarios such as a meeting between architects and clients to view and interact with a holographic 3D blueprint of a building, each of them with mixed reality capable devices looking at the blueprint from their own point of view as they sit around a table.

Azure contains pre-built services to write applications for these types of experiences on HoloLens and any other mixed reality device, including smartphones and tablets running the iOS and Android operating systems, noted White.