MIT researchers have developed a model that recovers data lost from images and video.

The model could be used to recreate video from motion-blurred images, or from new types of cameras that capture a person’s movement around corners but only as vague one-dimensional lines. While more testing is needed, the researchers think this approach could someday could be used to convert 2D medical images into more informative — but more expensive — 3D body scans, which could benefit medical imaging in poorer nations.

The model is described in a paper that will be being presented at next week’s International Conference on Computer Vision, authored by Guha Balakrishnan, a postdoc in Computer Science and Artificial Intelligence Laboratory (CSAIL).

Captured visual data often collapses data of multiple dimensions of time and space into one or two dimensions, called “projections.” X-rays, for example, collapse three-dimensional data about anatomical structures into a flat image.

Likewise, “corner cameras,” recently invented at MIT, detect moving people around corners. These could be useful for, say, firefighters finding people in burning buildings. But the cameras aren’t exactly user-friendly. Currently they only produce projections that resemble blurry, squiggly lines, corresponding to a person’s trajectory and speed.

The researchers invented a “visual deprojection” model that uses a neural network to “learn” patterns that match low-dimensional projections to their original high-dimensional images and videos. Given new projections, the model uses what it’s learned to recreate all the original data from a projection.

In experiments, the model synthesized accurate video frames showing people walking, by extracting information from single, one-dimensional lines similar to those produced by corner cameras. The model also recovered video frames from single, motion-blurred projections of digits moving around a screen, from the popular Moving MNIST dataset.

The researchers built a general model, based on a convolutional neural network (CNN) — a machine-learning model that’s become a powerhouse for image-processing tasks — that captures clues about any lost dimension in averaged pixels.

In training, the researchers fed the CNN thousands of pairs of projections and their high-dimensional sources, called “signals.” The CNN learns pixel patterns in the projections that match those in the signals. Powering the CNN is a framework called a “variational autoencoder,” which evaluates how well the CNN outputs match its inputs across some statistical probability. From that, the model learns a “space” of all possible signals that could have produced a given projection. This creates, in essence, a type of blueprint for how to go from a projection to all possible matching signals.

When shown previously unseen projections, the model notes the pixel patterns and follows the blueprints to all possible signals that could have produced that projection. Then, it synthesizes new images that combine all data from the projection and all data from the signal. This recreates the high-dimensional signal.

For one experiment, the researchers collected a dataset of 35 videos of 30 people walking in a specified area. They collapsed all frames into projections that they used to train and test the model. From a hold-out set of six unseen projections, the model accurately recreated 24 frames of the person’s gait, down to the position of their legs and the person’s size as they walked toward or away from the camera. The model seems to learn, for instance, that pixels that get darker and wider with time likely correspond to a person walking closer to the camera.

“It’s almost like magic that we’re able to recover this detail,” Balakrishnan says.

The researchers are now collaborating with Cornell University colleagues to recover 3D anatomical information from 2D medical images, such as X-rays, with no added costs — which can enable more detailed medical imaging in poorer nations. Doctors mostly prefer 3D scans, such as those captured with CT scans, because they contain far more useful medical information. But CT scans are generally difficult and expensive to acquire.

“If we can convert X-rays to CT scans, that would be somewhat game-changing,” Balakrishnan says. “You could just take an X-ray and push it through our algorithm and see all the lost information.”