The second version of the Holographic Processing Unit (HPU) that powers Microsoft's Hololens will incorporate an AI coprocessor to natively and flexibly implement Deep Neural Networks (DNNs.)

The announcement was made on Sunday by Harry Shum, executive vice president of our Artificial Intelligence and Research Group at Microsoft, in a keynote speech at CVPR 2017.

Deep learning approaches have two well-known challenges: they require large amounts of labelled data for training, and they require a type of compute that is not amenable to current general purpose processor/memory architectures. Some companies have responded with architectures designed to address the particular type of massively parallel compute required for DNNs, including our own use of FPGAs, for example, but to date these approaches have primarily enhanced existing cloud computing fabrics.

HoloLens, Microsoft's mixed reality device, puts the battery on your head, in addition to the compute, the sensors, and the display. Any compute that needs to run locally for low-latency, which you need for things like hand-tracking, has to run off the same battery that powers everything else. So Microsoft is creating a custom silicon to do it.

HoloLens contains a custom multiprocessor called the Holographic Processing Unit, or HPU. It is responsible for processing the information coming from all of the on-board sensors, including Microsoft's custom time-of-flight depth sensor, head-tracking cameras, the inertial measurement unit (IMU), and the infrared camera. The HPU is part of what makes HoloLens.

The new HPU, currently under development, supports a wide variety of layer types, fully programmable. At CVPR 2017, Harry Shum showed an early spin of the second version of the HPU running live code implementing hand segmentation.

The AI coprocessor is designed to work in the next version of HoloLens, running continuously, off the HoloLens battery. Microsoft did not provide much detail, but said that the new HPU is just one example of the new capabilities the company is developing for HoloLens.

Microsoft has been working on its own chips for a few years now. It built a motion-tracking processor for its Xbox Kinect video-game system. More recently, in an effort to take on Google and Amazon.com in cloud services, the company used customizable chips known as field programmable gate arrays to unleash its AI prowess on real-world challenges. Microsoft buys the chips from Altera, a subsidiary of Intel, and adapts them for its own purposes using software.

Amazon also uses field programmable gate arrays and plans to use a new chip design called Volta for AI built by Nvidia. Meanwhile Google has built it's own AI semiconductors, called Tensor Processing Units, and is already letting customers use them.