First-generation interpretation of a multi-touch interface

Multi-touch human interfaces have arrived, but so far they are like surprise visitors: we may be happy to see them, but we’re not ready to let them move-in for good. For those of us who are engineers and work every day to realize the dreams of future products, multi-touch offers an opportunity to change the world, in much the same way the first graphical user interfaces did. The race has just begun, but make no mistake, it will be won soon. Personally, I’m not convinced we have a winner yet, and if we don’t step up to make multi-touch a great technology, we will live with it being marginal for a long time.

What is multi-touch?

The world applauded Apple’s first-generation interpretation of a multi-touch interface in the iPhone 1.0. The new user interface worked, it offered appropriate functionality, and they didn’t forget to make phone-calling easy. Apple iPhone added gestures to our vocabulary, offering two-finger pinch to zoom-in, two-finger open to zoom-out, and two-finger rotate; all intuitive and useful on a device with a camera and small screen.

Not many people realize that Apple began their foray into multi-touch with the two-finger touchpad sensing implemented in their large touchpad notebooks that was utilized by the operating system and every application. Even before the iPhone, an Apple notebook user could scroll, both vertical and horizontal, using two fingers on the touchpad. Apple even made their stubborn single-button-touchpad choice look prescient by enabling right-mouse button functionality by putting two fingers on the touchpad and clicking the button (eliminating the annoying <CTRL>-click). Again, it’s hard to call these choices anything but useful and welcome. Just recently, multi-touch offerings or announcements from many different vendors highlight the “final-frontier” opportunity today for multi-touch user interface technology. Apple introduced their next-generation MacBooks on October 9, 2008 showing the familiar two-finger gestures of iPhone with new – and less intuitive – three- and four-finger gestures. Microsoft began shipping its multi-touch brain-child called Surface that comes with a hefty $5,000 to $15,000 price tag making it clearly aimed at money-making businesses (early adopters are AT&T and Harrah’s Entertainment). At the October 2008 Microsoft Professional Developer’s conference, Microsoft unveiled its upcoming Windows 7 and promised multi-touch support. The Microsoft idea of multi-touch as shown with Surface provides some gestures, like the 2-finger gestures to zoom and rotate, but seems more targeted at a multi-user experience. So the question remains, what is multi-touch? The impact of multi-touch will be as far-reaching as the computer mouse and the engineering community needs to step in to ensure that multi-touch fares better in the future than the graphics pad (remember the digitizing, pen-like computer input device that none of us has on our notebooks?)

Multi-touch technology today

In contrast, single-touch interfaces are most often based on the old “touchscreen” technologies most of us experienced on our cell phones and PDAs were resistive, where the stylus position yielded two voltages, one representing the “X-axis” position and the other representing the “Y-axis”. Single-touch interfaces result in the same X-Y data that a traditional track-pad and mouse provides today. For their multi-touch interfaces, Apple uses capacitive sensing and Microsoft Surface uses cameras. Capacitive sensing is also the technology typically used in single-touch trackpads in notebook computers. There are several module, screen and individual silicon vendors supporting multi-touch, so there are many options available on the market, making selection a potentially confusing process. Depending upon the technology, it is possible to simultaneously sense the position of all 10 fingers on a display (see the video here)Certainly the capability to support multi-touch interfaces in a myriad of ways is available. The rub is that there is no one standard way to use this multi-touch data in a computer application.

To see where multi-touch can go, let’s return to how the two easiest-to-review implementations of multi-touch interpret what the users of the future (that’s you and me) need. The Apple iPhone implements what is referred to as “Multi-Touch Gestures” where two fingers are sensed and their relative motions translated into a gesture that a program can react to (i.e., rotate, zoom, select and move). At a minimum, these gestures need to appear quickly in the lexicon of every laptop, but that means every program and operating systems needs to change to accommodate them. Beyond the 2-finger gestures, the picture blurs quickly. “Multi-Touch All Point” technology enables many simultaneous inputs from the touchscreen or touchpad. What to do with these inputs, now, is the question. There are huge opportunities here, but the current examples of Apple and Microsoft are not exploiting them to the fullest.

Apple has included three- and four-finger gestures on the new MacBook, but only Apple applications use these gestures (unlike the scrolling and left clicking two-finger gestures). Additionally, the thought process behind which functions should take three fingers and which four fingers appears to be somewhat arbitrary. What doesn’t make sense is the comeback of the <CTRL> button, being used as a modifier to the three-finger gestures, even though the touchpad is large and has the ability to sense lots of fingers. Also, the multi-touch equivalent of the mouse-click+<SHIFT> -drag to select seems to have disappeared. Anyone else have trouble completely adopting a Palm(TM) PDA because Grafitti(TM) never became second nature?

Intuitive gestures equal easy adoption

Likewise, Microsoft with Surface(TM) seems to be stumbling in the dark when it comes to using more than two-fingers. For instance, they have a virtual air-hockey game demo that uses one finger for each player to grab and move the controller (there is another version that uses a physical controller like a traditional air-hockey table, but that isn’t a multi-touch interface). When I play real air-hockey, I would have been stupid to use one finger, and the true-to-life nature of Surface does reflect this behavior as you can see the user’s controller slip-and-slide around as if he were using only one finger to control a large disk. There are other problems too, such as the controllers sometimes switch players when they get too close. This commentary is not intended to slam Apple’s MacBooks or Microsoft’s Surface but rather to highlight the fact that the multitouch field is wide open. The technical capabilities available today do offer a much more natural and intuitive user interface IF AND ONLY IF we as engineers harness the power and direct it to the greater good.

The software development kits and software development tools for Microsoft, Apple, and Linux all provide built-in, standard support for keyboards and X-Y pointing devices (mouse and track-pad buttons as well). Anyone today can target any operating system and as long as the “input” is translated into one or more keyboard keys and X-Y position, any application can use that input. For example, someone could develop, using standard offerings in any operating system, a 10′-by-10′ room as a trackpad replacement, where one runs around on the floor, jumps up and down, and throws one’s body against a wall to select and move icons around a PC screen. In the multi-touch future, what do we as a development community need to do to secure a similar level of freedom to develop input devices and the programs that interact with them? What do we need to demand as a standard set of provided capabilities so program developers do not have to worry about the input device and input device makers do not have to worry about the programs? While the ultimate answers are up to all of us, the answers being developed today could seriously affect our lives for the future.

The future of Multi-touch is in our hands

“Standards” for multi-touch interfaces are being developed whether we like it or not and now is the time to get involved, make noise, ad shape the future of multi-touch. Let me put forward some of my suggestions for a better multi-touch future. Read them, use them to come up with better ideas, and then get involved by either working with the standards-setters or by implementing a multi-touch device and putting it into people’s hands so we can mobilize more troops in this fight for a great multi-touch future.First, we need a few standard, intuitive gestures and second, we need a standard data interface to provide position data for up to 10 fingers. Standard gestures should cover the most common computer/information device operations, like scroll, zoom, select, move, and grab-and-move, as well as all the new functions multi-touch will enable. What we do not need are multiple company-patented sets of gestures. Rather, let the innovators patent better and smarter techniques of determining the gestures. Note that we do not need a long list of standard gestures, because if that is what we get, users will need to print out the list and paste it to the back of their devices, just like we all did with Grafitti on our Palm PDAs before we stopped using them. Also, gesture detection cannot be forced exclusively upon the operating system nor can it be forced exclusively upon the input device. The best standard solution would accommodate both for maximum flexibility. However, if a choice between operating system or input device is forced, the operating system must be allowed to win.

Do not stop with standard gestures (or the corollary: Do not define everything as a gesture). Define a data standard for multitouch input devices for tracking up to ten independent inputs. Realize that the use of this data for quite some time to come will be application-specific. That said, and over time the best behaviors (we hope) will be adopted into the operating systems, Why ten and not more? Most devices are primarily single-user or have multiple users doing simpler actions on a relatively small screen. Let the special-purpose large-format devices like Surface explore what to do with more than ten inputs, and when something proves itself useful, it can trickle down. The multi-touch train is leaving the station. For those who do not like everything they have seen so far, get involved and push the envelope further. Use multi-touch in new and interesting ways and let others see and hear about your successes and failures (especially the guys in Cupertino and Redmond). The industry can make multi-touch great, but only if we work together.