Robert Scoble takes a Qik video of a couple people playing with a Microsoft Surface demo app.
When I look at something like this I see another twist on social computing–no not the multi-player game type, nor the Facebook type. No, I mean more than one person directly interacting with a computer device at a time.
Although it would have been cool for at least once for Bill Gates to use a Tablet PC during one of his keynotes, he at least doesn’t fail to mention them. In his final keynote as full-time Microsoft chairman at TechEd, Gates talks of Tablets and education:
“I think of every student having a device that avoids the need for paper textbooks. The Tablet device will let them take notes, record audio, connect to the Internet. It’ll be superior in every way and yet it can’t be purely keyboard based. It has to have this touch and pen as well.”
In case you’re wondering, he didn’t say much more about Windows 7 other than referring to the multi-touch demoed last week at D. He did add:
“We’ve also got the pen capability, that we’re taking to a whole new level in terms of easy recognition and how that is implemented in the hardware.”
Not sure if he was referring to Windows 7 here what’s currently available in Vista. My guess is the latter, but it’s hard to tell.
I was glad he finally is mentioning vision in more detail. This has been a very lacking aspect of Microsoft’s interaction strategy if you ask me. Surface, TouchWall and the like are all beginning to use cameras, though there’s lots more than can and should be done. I’d like to see a whole new vision processing capabilities added to the .NET Framework.
CNet reports that Robbie Bach several times contemplated killing the Microsoft Surface project. It appears he wasn’t sure whether there was a business case behind the project. He has changed his mind though, in light of the overwhelming positive reactions the Surface computer has received.
Bill Gates is quite the advocate of Surface too. As CNet reports, last month he “said he wants to turn everything we touch into a computer: “It will be absolutely pervasive,” he said. “When I say everywhere, I mean the individual’s office, the home, the living room.”
I’m not sure about everything, but I agree that interactive touch should be in more places.
Robert Scoble interviews Andy Wilson, one of the researchers behind the Microsoft Surface computer. Watching now.
Andy shows the Office Labs document demo on a 30″ multi-touch display. The multi-touch is done using an IR laser and a single above-mounted camera.
In a blog post reflecting on whether Microsoft needs to reach further into the consumer markets, Mary Jo Foley ponders:
“The wall-size, vertical touch screen (a Microsoft Research-Office Labs joint venture known as “TouchWall”) — which Gates touted as the “whiteboard of the future” — looks a heck of a lot more compelling than the Surface consumer-focused prototypes I’ve seen.”
Come on? Microsoft Surface is too consumer oriented? No way. It’s supposedly in the $5,000+ range. That’s no consumer device. Besides being expensive, it’s also bulky. Yes, consumers are interested in multi-touch technology–Check out Seth Sandler’s blog to explore why. But this doesn’t make it consumer focused. From an engineering standpoint, if Surface were consumer oriented, it’d probably would be more desk or table-top like–more like a piece of artful furniture than rugged kiosk.
Now in terms of cost, it appears that TouchWall has the edge. But let’s be clear that there are people on the Net, just like me, that are building homebrew multi-touch “surfaces” for thousands less than Microsoft can offer their multi-touch box. That’s in part the difference between a public use box and one meant for someone’s home.
Also, people–outside of maybe touch enthusiasts like Lora and myself–aren’t going to be rushing to put 60″ rear-projected multi-touch screens in their rooms. They’ll take up too much space and restrict lighting too much.
Now, will there be enthusiasts that build their own. You bet. Just like those building their own multi-touch tables. But this makes neither consumer.
One last thing: Over the years I’ve seen technology time and time again come from the “consumer” side–meaning that individuals often adopt technology first. You can call these individuals consumers–even those working in large corporations. The key to success though of many of these advances, yes, has come from corporate purchases, who are often not as financially cautious over the mid-level dollar amounts. So being a product that consumers are interested in is a good thing–especially for new technology. To only focus on mature technologies for businesses or for high-priced products that only large corporations can justify is certain to cap market opportunities though. This is not a good path to take.
At the 2008 Government Leaders Forum Asia on May 9th Bill Gates shared some more of his enthusiasm for the Tablet PCs and more specifically Tablets in education.
“….I’ve got one last thing to show, and I previewed this earlier, and that’s related to the student Tablet. To me this is an important milestone, and Microsoft has been investing in this for a long time. We see lots of ways that we’re going to drive this into the mainstream. In fact, my own daughter goes to a school where she uses a Tablet PC, and it’s phenomenal to see how comfortable she is, how she learns better. She tries out her knowledge, she communicates with her teacher in a new way. It is completely digital. The Internet is there, the ability to create things is there.”
and on technology in schools:
“One of the things that I always share my enthusiasm for when I talk about the future of technology is the idea of students having a computer individually, and later today you’re going to hear from a teacher and a student who are experiencing that, and piloting what that’s like. Clearly that has to be a very robust machine that can last - you can drop it - it’s got to be inexpensive, it’s go to be powerful, but the hardware and software changes make it a question of when we can do that, not a question of if we can do that. Textbooks are on their way out. In some countries, that will happen in three or four years. In some it will be five or six, but I’ll be so bold as to say that over the next decade we’ll look at a textbook the same way we look at things like a paper-based encyclopedia today. And we look at it and we say, hey, it’s not rich enough, it’s not animated, it’s not inexpensive enough, it’s not flexible enough, it doesn’t give you the richness that the digital form of that can provide. And so these advances really make a huge difference.
One that I think is particularly interesting is that the way that we interact with these machines will change. We use the keyboard and the mouse today. On the mobile phone, we largely use the little keyboard, and it’s impressive how well people are able to use that small keyboard. But we should complement that with new approaches, and there’s a number of additional approaches that I refer to as natural interface. Speech input, where you talk and the system recognizes it. Pen, where you take a pen and you write ink, and it recognizes what you’ve written. Touch, where you can just point to things and move them around. A great example of this is that in the future the desk of a worker will be a touch-sensitive surface. The cost of that display and having the software that can see what you’re doing will be very, very low, and so you can take different documents, have them laid out there, point to one, expand it, have the sales data, or the survey data, or the quality data, or the calendar easily manipulable so you can navigate through it just by pointing at things, and then if you see something that surprises you, you can take your pen, write a little note, pick which colleague you think should take a look at that, click on that, and off it goes. And it’s very straightforward.
Likewise, your walls will be able to display information, because the cost of the screen will be very low, and they too will have a camera that can watch what you’re doing with the magic of software. And so your whiteboard, what was the chalkboard, will be intelligent. That will be true in the office, in the meeting room, in the classroom, at home. And so things like taking photos and organizing them, just very natural.
This is the kind of thing you’ve seen in science fiction movies, but in fact it’s now moving into the market, and moving into the market in very low cost. Our first product in this space is called Microsoft Surface. It’s a flat table. And just a few months ago that rolled out to the early customers and the response has been phenomenal that people love that natural capability of interacting.
Well now, with computer technology being so amazing, and so empowering to the individual, how do we get it so that we have broad benefit? Whenever we make scientific advances, we’re faced with this dilemma, and when we have new medicines, are they only for the rich? We go all the way back to look at reading, which of course in its early days only the very elites were literate, and it took hundreds of years before low cost printing, and government programs around libraries and schools got to the point where every country took a goal of saying that all the population deserves to the literate. And the world as a whole, most of the world, has done very, very well on that. Well now we have almost a new type of literacy, digital literacy, and making the computer accessible.”
(Emphasis is mine.)
As I read comments like this I jump inside with excitement. Who knows how long it’s going to take to achieve and whether it all comes out like we think it should, but is there any doubt that we’re going to figure out even more ways to make our lives more efficient, more connected, more enjoyable?
After a bit of trial and error I got the multi-touch box working! With this homebrew multitouch box I can place and slide multiple fingertips across the top of a box and the computer tracks my every move. The setup is shown in the image at right. It’s so cool. What a terrific home project. Yesterday, we had a family get-together and all had a blast trying it out.
You can build your own in about 30 minutes or so. All you need is a computer (running Windows XP or Vista (32-bit or 64-bit)), a USB camera, and well, a box. Seth Sandler provides a great how-to on his blog here.
I used a small, 7″x7″ box with the top flaps folded up and a sheet of white paper taped to the top. I also taped a Microsoft NLX-6000 webcam facing up to the bottom of the box. Any reasonably good webcam should work. After you have made the box, you’ll want to plug in the boxed webcam, download some sample apps and binaries that Seth points to on his blog, and run the demo_smoke.exe demo. That’s the best sample app to get you going.
When you launch the demo_smoke program a command window like this:
and a camera config dialog box will appear if all goes well.
All you need to do is click OK button in this window titled “Property Sheet Properties”, which I’ve called the camera config window.
If you don’t have a camera connected properly you’ll get the following error dialog box:
After a couple seconds you’ll then see a third window titled “Demo Smoke.” This is the app that you’re really interested in. It displays colorful smoke trails that follow your fingertips as you slide them across the top of the box.
If all goes well, you’ll see small white circles where your fingertips are and as you move your fingertips around you’ll see trails of “smoke”.
A couple tips:
* Although you don’t need to calibrate anything to get this app running, you’ll want to make sure that the camera’s field of view covers the paper on the top of the box. I checked this before I taped the paper to the top of the box by connecting up the camera to the computer first and checking to see what I could see. This is a good idea anyway, because it gives you a chance to make sure the camera is working at all.
* You’ll also need enough light outside of the box. The setup works because your fingertips occlude the light or at least cast a strong shadow on the paper. Normal daylight room lighting worked for me and at night I simply had to turn on the room lights. No special set up.
* For best finger tracking lift your palm up away from the paper. The extra shadow cast by your hand may confuse the tracking.
* Depending on how light tight your box is you may see extra smoke around the edges. If so, there’s probably light leaking in.
* So far I’ve tested this on a Vista 64-bit system (Mac Pro running Bootcamp), a Lenovo X61 Tablet PC running Vista, and a MacBookPro running Bootcamp and Vista. I’ve been unable to get it to work on my Toshiba M700 Tablet PC. I think it’s because the built-in camera is conflicting with the USB webcam in the app, but I haven’t dug any deeper to see if I can get it working (outside of disabling the built-in webcam, but that didn’t help).
What about the software? There is source code available which is being archived at Google. I pointed to it the other day. The code is primarily written in C++ and built off of Intel’s OpenCV (Open Computer Vision) library. I can see where one of goals of the coders was support multi-platform, however, it does add a bit of complexity that a large chunk of the market is not going to need. Don’t get me wrong, the approach isn’t bad, however, a little brittleness and install issues may avoided by let’s say supporting primarily Windows and the largest language platform choice for many .NET. So….I’m thinking through how to incrementally port the code. It doesn’t look hard. I could start at the beginning and write my own segmentation routines in CSharp or maybe wrap the OpenCV calls needed for multi-touch, or maybe wrap the high level finger-movement event notifications in the current TouchLib. My current thought is to start near the top and work my way down–but that’s just because I can keep a lot of things running in the meantime. I have to think through this a bit more and figure out how much time I really have to do this :-).
Anyway, if you don’t think all of this is that wild or practical, don’t forget this only takes a few minutes to build and costs next to nothing. This is no Microsoft Surface computer. But hey, if this doesn’t get you thinking about building a full-sized multi-touch system, nothing will. You can guess what we’re up to now!
Here are a couple multi-touch physics demos that you may want to check out if multi-touch tables are your thing:
http://rxsurface.blogspot.com/2008/05/introducing-multitouch-crayon-physics.html
and
http://www.flickr.com/photos/rxsurface/2351234376/
What’s interesting to see is how Google is enabling these Surface-like, multi-touch efforts through it’s student summer programming programs. There’s a whole TouchLib that Google is hosting here which was built in part during this past summer. If you want to experiment with the binaries you can download them here and build your own multi-touch panel using the video below as a guide and Seth Sandlers’ description on his blog.
There’s a whole community growing out of these efforts. Keep your eyes on YouTube if you have any doubts.
I downloaded the binaries, but haven’t been successful yet in getting them working. I imagine I have a configuration issue somewhere along the line. Maybe I’ll have some time this weekend to figure out what it is. Their implementation is classic open source, so you’ll see a little bit of this, a little bit of that, a half dozen libraries required, and sparse documentation to boot. This isn’t holding back eager people from following their recipes though. Great to see such excitement.
ChannelWebNetwork posts an article about Microsoft’s Surface computing initiative–well, more precisely, about Microsoft Surface.
There’s no broad Surface initiative from what I see. In fact, as the article points out, the Surface SDK is only for the eyes of its current Surface partners. There’s no mention if there’s ever going to be a plan to release the SDK to the general developer community. So if you’ve been thinking about how these touch technologies have broader value, it’s better to hold that thought.
Too bad. I think there’s an opportunity here to broaden the innovation circle. Touch has many applications. Not just for the Surface computer as it stands–or sits–now.
We have to look no further than Apple, with its iPhone and now MacBook Air to see how popular the capability can be. In fact, CES was full of touch and multi-touch surfaces large and small, vertical and horizontal. Dell even has been demoing some multi-touch technologies. It makes sense to me to view multi-touch in a much broader way.
Microsoft Surface has sure captured the attention of many with its slick, multi-touch experience. However, it’s by no means the only touch “surface” making the rounds.
This evening I saw a clever “touch,” virtual dining surface mentioned on the news. It’s part of an exhibit on display at the Detroit Institute of Arts that projects a meal on a table with which you can interact with. This Free Press article gives a good idea what the table is like.
It appears that the dining room table has a forward facing projector that displays a collection of dinner items which people can move about. I assume there’s a camera sitting beside the projector which is used to track people’s movements and determine how to translate the virtual dinner items. Pretty clever.
Actually, there’s a somewhat similar interactive art installation at the Paris Hotel in Las Vegas. On the way to and from the monorail from the Paris Hotel is a projected pool of water displayed on the walkway which you can interact with as you walk over it. With each footstep virtual ripples spread throughout the image.
There are some other art-oriented surfaces that you might find in your local museum too. One is called the DialogTable, which museum goers can use to interact with explanations of what’s on display at the museum.
The DialogTable website explains how the device works:
“A computer interface is projected onto the bottom surface at each end of the table. A camera on the ceiling analyzes your hand gestures. Like a mouse click, a grasping hand motion allows you to drag images. A digital shadow of your hand is projected onto the screen allowing you to gesture above the table. Unlike a touchscreen up to eight hands can be manipulating the same interface at the same time.”
This is somewhat similar to a Microsoft Surface computer although in the case of a Surface device the camera(s) are placed behind the screen. This avoids some occlusion problems–although if there are physical objects on the display (such as cell phones, papers, or what have you), there will be “touch deadspots” on the display nonetheless. With cameras “above” the display the cameras are at least able to see how the users might be interacting with the objects. I’m not sure if this is a big deal in most cases though.