From Minority Report to Iron Man, the Genesis of Gesture Technology

Good sci-fi movies have a way of influencing, if not actually predicting, the future when it comes to technological innovation. Take, for example, gestural interactions with computers — they are the next step after today's virtual keyboards.

One of the most popular movie approximations of this tech is used by Precrime Police Chief John Anderton in the 2002 movie Minority Report. Anderton solves a future murder by shuffling through holographic data with special gloves and a burning intensity.

Played by Tom Cruise, Anderton gestures manfully to pull in gauzy images, toss aside useless information and zoom in on clues supplied by three precognitive human mutants, called "precogs."

Without argument, it is a great scene. In fact, it has entered the collective expectation many people have for what the near future holds, up there with flying cars and personal food replicators.

Indeed, many computer and information-technology designers no doubt took notes as movies continue to inspire innovative people-computer interactions.

In the above video, Georg Hackenberg demonstrates the 3D interaction prototype that he has developed for his master thesis at Fraunhofer FIT in Bonn, Germany. "The system uses a depth sensing camera, which is based on the time-of-flight principle and measures phase shifts of a reflected infrared signal," Hackenberg writes. "My contribution is the computer vision algorithm, the gesture recognition technique as well as the concept and implementation of the demo application."

This is all very technical way of saying, Hackenberg has created something of a real life Minority Report-like console. But Minority Report was hardly the film to depict gestural interaction technology.

Christopher Noessel, managing director of product and business-tool design firm Cooper, says that film makers have depicted gestural interfaces with computers at least as far back as 1951 in the classic The Day the Earth Stood Still.

In an ongoing project of his, Noessel searches for computer gestures on celluloid going back to the 1902 debut of La Voyage Dans La Lune. Asked to name three movies that are influential in the work of real-life gestural-interface designers, he cited:

• Minority Report
• Johnny Mnemonic
• The Iron Man movies

 

It is not clear if The Day the Earth Stood Still was the first movie to depict a gesture controlling a computer, but if it was, its director, Robert Wise, introduced one of the few that could cross over to the real world.

In that movie, an alien arrives on post-World War II Earth with a peace offering and an ultimatum (stop killing or we will kill you). Of course, Klaatu, the alien, is shot. When he gets back to his idling flying saucer, he waves his hand over some controls to start the ship running again.

It is a simple thing, says Noessel, and yet the wave has become part of the film industry's most often-used computing gestures.

"You see 'wave to start' used often," he says, for instance in Johnny Mnemonic. It's used to turn a faucet on. The same gesture will turn faucets on in your home today as well as dispense paper towels in public restrooms.

Far more complex gestures are used in Johnny Mnemonic as the title character, played by the appropriately non-verbal Keanu Reeves, hacks through cyberspace while wearing gloves left over from Rollerball.

Johnny Mnemonic is noteworthy in a gestural sense at least in part because it shows a future where digital data and access to it is a quasi-physical task.

The same is true of the most recent movies featuring Marvel Entertainment's Iron Man/Tony Stark character. Stark is a multibillionaire engineering genius who, in this incarnation, does most of his best work while manipulating holographic displays.

Working with gestures and a supercomputer that would teach the human emotion of shame to Anderton's relative calculator, Stark (Robert Downey Jr.) simulates robotic suits and crime scenes. It appears that series directors Shane Black and Jon Favreau have dreamed up a vast gesture vocabulary.

"Stark has a whole language (built mostly by Robert Downey's sense of what's cool, I think) for manipulating volumetric projections," Noessel says.

Surprisingly, he doesn't think gestures will supplant keystrokes. They will play a support role in computing, at best, because they are inherently limited compared to the deftness of speech and the exactness of typed communication.

Maybe, but there are some gestures that have been used for generations in the Bronx that say so much with just a single finger. Perhaps there's more research to be done.

Featured image: Tony Stark uses highly advanced gesture technology. Courtesy Marvel/Walt Disney Pictures