Incremental Blogger

Gizmodo is pointing to some TouchLib-based, multi-touch demos running on a Mac.

Multi-touch is gaining steam as a platform for devs to experiment with.

Great to see.

At the Microsoft CEO Summit, Ian Sands and Chris Pratley (of OneNote fame) are showing off something called the TouchWall. It uses three lasers to detect and track finger positions that break the plane of the surface. A rear-projector projects the image onto the back of the wall.

This is a research project where they are experimenting with various models for displaying and manipulating content. In the demo video shown below they are showing an infinite canvas with multi-touch merged together.

Here’s a link to a video where Bill Gates is demoing TouchWall (26.4MB) himself at the CEO Summit.

All of this looks fantastic.

Here’s the thing though: Microsoft needs to start realizing that all these multi-touch approaches are not magical and that there’s a common set of functionality going on here. As you know I’m going ahead with building a multi-touch surface because I’m not going to wait any longer to experiment with this technology. Again, it’s not rocket science. Now to build a product, I 100% agree that there are tons of issues that would need to be worked out. But in terms of facilitating Windows developers to keep pushing the envelope, I’d like to see Microsoft make an effort to get some of this stuff out in the ecosystem. Maybe the Microsoft Research team can do this. I’d love to see it.

Anyway, here’s the big news along these lines: “TouchWall…is built on a standard version of Vista.” Yep, multi-touch is not rocket science. It’s incremental. Google is demonstrating this with TouchLib. And Microsoft is demonstrating this with TouchWall. And from what I can gather–though I’m not positive–Microsoft Surface shows us too. There’s no magic here.

Let’s get the ball rolling.

In his speech today, I think Bill Gates has the right idea about touch and surfaces: “We’re saying it will be absolutely pervasive. When I say everywhere, I mean the individual’s office, the home, the living room.”

I’d add your notebook, phone and more to his list too. It’s not just about location. It’s about natural and efficient experiences.

File this under the Ignorable-Post category: I’m glad I’m not going to be attending TechEd 2008–(it’s out of my price range)–because if I were I’d be spending too much time and effort trying to figure out how to take along the multi-touch system we’re building.

Tablet PCs are so much more portable. I’m spoiled.

djay software has released a multi-touch trackpad scratching and mixing app for Macs. Gotta love it.

At the 2008 Government Leaders Forum Asia on May 9th Bill Gates shared some more of his enthusiasm for the Tablet PCs and more specifically Tablets in education.

“….I’ve got one last thing to show, and I previewed this earlier, and that’s related to the student Tablet. To me this is an important milestone, and Microsoft has been investing in this for a long time. We see lots of ways that we’re going to drive this into the mainstream. In fact, my own daughter goes to a school where she uses a Tablet PC, and it’s phenomenal to see how comfortable she is, how she learns better. She tries out her knowledge, she communicates with her teacher in a new way. It is completely digital. The Internet is there, the ability to create things is there.”

and on technology in schools:

“One of the things that I always share my enthusiasm for when I talk about the future of technology is the idea of students having a computer individually, and later today you’re going to hear from a teacher and a student who are experiencing that, and piloting what that’s like. Clearly that has to be a very robust machine that can last - you can drop it - it’s got to be inexpensive, it’s go to be powerful, but the hardware and software changes make it a question of when we can do that, not a question of if we can do that. Textbooks are on their way out. In some countries, that will happen in three or four years. In some it will be five or six, but I’ll be so bold as to say that over the next decade we’ll look at a textbook the same way we look at things like a paper-based encyclopedia today. And we look at it and we say, hey, it’s not rich enough, it’s not animated, it’s not inexpensive enough, it’s not flexible enough, it doesn’t give you the richness that the digital form of that can provide. And so these advances really make a huge difference.

One that I think is particularly interesting is that the way that we interact with these machines will change. We use the keyboard and the mouse today. On the mobile phone, we largely use the little keyboard, and it’s impressive how well people are able to use that small keyboard. But we should complement that with new approaches, and there’s a number of additional approaches that I refer to as natural interface. Speech input, where you talk and the system recognizes it. Pen, where you take a pen and you write ink, and it recognizes what you’ve written. Touch, where you can just point to things and move them around. A great example of this is that in the future the desk of a worker will be a touch-sensitive surface. The cost of that display and having the software that can see what you’re doing will be very, very low, and so you can take different documents, have them laid out there, point to one, expand it, have the sales data, or the survey data, or the quality data, or the calendar easily manipulable so you can navigate through it just by pointing at things, and then if you see something that surprises you, you can take your pen, write a little note, pick which colleague you think should take a look at that, click on that, and off it goes. And it’s very straightforward.

Likewise, your walls will be able to display information, because the cost of the screen will be very low, and they too will have a camera that can watch what you’re doing with the magic of software. And so your whiteboard, what was the chalkboard, will be intelligent. That will be true in the office, in the meeting room, in the classroom, at home. And so things like taking photos and organizing them, just very natural.

This is the kind of thing you’ve seen in science fiction movies, but in fact it’s now moving into the market, and moving into the market in very low cost. Our first product in this space is called Microsoft Surface. It’s a flat table. And just a few months ago that rolled out to the early customers and the response has been phenomenal that people love that natural capability of interacting.

Well now, with computer technology being so amazing, and so empowering to the individual, how do we get it so that we have broad benefit? Whenever we make scientific advances, we’re faced with this dilemma, and when we have new medicines, are they only for the rich? We go all the way back to look at reading, which of course in its early days only the very elites were literate, and it took hundreds of years before low cost printing, and government programs around libraries and schools got to the point where every country took a goal of saying that all the population deserves to the literate. And the world as a whole, most of the world, has done very, very well on that. Well now we have almost a new type of literacy, digital literacy, and making the computer accessible.”

(Emphasis is mine.)

As I read comments like this I jump inside with excitement. Who knows how long it’s going to take to achieve and whether it all comes out like we think it should, but is there any doubt that we’re going to figure out even more ways to make our lives more efficient, more connected, more enjoyable?

In the last two weeks, since Seth Sandler uploaded his video showing how to make a multi-touch box, people have played it over 350,000 times. That’s 350,000 plays in 14 days–roughly 25,000 a day. At half this rate over the next few weeks I wouldn’t be surprised to see this video go over 500,000. This speaks volumes about the degree to which people are interested in new interaction technologies. Although the keyboard is going to be central to how they will be using computers for a long time, people want more.

The strong interest in Jeff Han’s multi-touch work, Microsoft’s Surface initiative, and homebrew multi-touch efforts like Seth’s all reflect this.

I had a blast building a multi-touch box and am now on the way to constructing a full-sized multi-touch system. Others on YouTube have been posting their successes too.

By the way, I agree with the Interactive Multimedia Technology blog: “Hint for high school teachers: This sort of thing would be a great project for an after-school technology club!”

Absolutely.

Here are a couple other recent homebrew multi-touch efforts, several of which follow Seth’s multi-touch box recipe:

Front projection touch test by SkeeeTouch

A projectionless, multi-touch box by jpcancio

Multi-touch box by PatukyHC

Barecelona multi-touch demo.

“A first-rev 720p multi-touch video wall by Obscura (Love the ink in the demo ). Here’s the same whiteboard being used to in a two-player Missle Command game. You might remember this large display demo from last summer.

Multi-touch paint using a box

Hhomebrew FTIR multi-touch surface with rear projection

Brad Hayes’ with rear-projection multi-touch

And here are a couple other people that built multi-touch boxes and posted their results.

This homebrew movement is growing fast. It reminds me of the days back in the late 70s with homebrew microcomputers.

OK, I changed my mind. This is my last and final post today about multi-touch today.

Are Apple users more enthusiastic about multi-touch?

After scanning through various blog posts on multi-touch, I’m begining to wonder if the Apple community is vocally more interested in the technology than lets say those from Microsoft. A quick check using Google’s Blog Search makes it look like this is so. This may be a byproduct of the fact that Apple was the first to demonstrate multi-touch in a mass-shipping product: The iPhone.

I think, however, that the Windows world is also keenly interested in multi-touch too–whether standing up at a multi-touch whiteboard/display, or sitting down at a Microsoft Surface computer, or potentially sharing and sifting through content on a slate Tablet PC.

So far, Google (because of TouchLib) and Apple (because of the iPhone SDK) are early leaders in multi-touch when it comes to giving developers multi-touch tools.

Is multi-touch more slate friendly?

Is it just a coincidence that Apple launched its multi-touch technology on essentially a slate–the iPhone–rather than one of its notebooks? I don’t think so. In fact, I think that multi-touch is more natural and more compelling on a slate-type device. (I’m defining a slate as anything here where there isn’t a predominant, keyboard built-in.)

What does this mean for Tablet PCs? It means that Tablets as they have evolved in the Microsoft sphere (as notebooks with handwriting and touch) are vulnerable to an attack by slates–if and only if slates prove to work well with multi-touch. Take multi-touch away and maybe the Tablet PC convertibles will be safe for awhile.

On the other hand, if multi-touch proves out to be as freeing as I think it might–especially for “slate” surfaces–it could mean a resurgence in slates. This might begin for small handhelds, such as MIDs, as well as large displays and then slowly encroach into the notebook space.

It also might mean that if Apple were to sell a slate (as many rumors have indicated) with multi-touch (supporting native rotating windows and the like, which are very multi-touch friendly), that they could ignite a firestorm of interest in “Tablets.” Nah.

What might a multi-touch slate mean for keyboards that are so important to us now? I’m not sure. Maybe we’ll see them working more side by side with convertibles. At this point I wouldn’t see multi-touch being the key, but rather the interconnectivity between the various computers.

One more multi-touch post. I promise. Just one more.

You know that multi-touch board that CNN has been using during their election returns shows? It comes from Perceptive Pixel, the company founded by Jeff Hans–a pioneer in multi-touch systems.

[Found via MacRumors]

High-school student, Bridger Maxwell, won last week with his OS-X based multi-touch computer in a regional science fair. His project was featured last week on Engadget.

On his blog he writes: “I also won first place in the engineering category and *drumroll* an all-expense-paid trip to Atlanta, Georgia to compete in the Intel International Science and Engineering Fair (ISEF)!”

Cool! Congrats!

After a bit of trial and error I got the multi-touch box working! With this homebrew multitouch box I can place and slide multiple fingertips across the top of a box and the computer tracks my every move. The setup is shown in the image at right. It’s so cool. What a terrific home project. Yesterday, we had a family get-together and all had a blast trying it out.

You can build your own in about 30 minutes or so. All you need is a computer (running Windows XP or Vista (32-bit or 64-bit)), a USB camera, and well, a box. Seth Sandler provides a great how-to on his blog here.

I used a small, 7″x7″ box with the top flaps folded up and a sheet of white paper taped to the top. I also taped a Microsoft NLX-6000 webcam facing up to the bottom of the box. Any reasonably good webcam should work. After you have made the box, you’ll want to plug in the boxed webcam, download some sample apps and binaries that Seth points to on his blog, and run the demo_smoke.exe demo. That’s the best sample app to get you going.

When you launch the demo_smoke program a command window like this:

and a camera config dialog box will appear if all goes well.

All you need to do is click OK button in this window titled “Property Sheet Properties”, which I’ve called the camera config window.

If you don’t have a camera connected properly you’ll get the following error dialog box:

After a couple seconds you’ll then see a third window titled “Demo Smoke.” This is the app that you’re really interested in. It displays colorful smoke trails that follow your fingertips as you slide them across the top of the box.

If all goes well, you’ll see small white circles where your fingertips are and as you move your fingertips around you’ll see trails of “smoke”.

A couple tips:

* Although you don’t need to calibrate anything to get this app running, you’ll want to make sure that the camera’s field of view covers the paper on the top of the box. I checked this before I taped the paper to the top of the box by connecting up the camera to the computer first and checking to see what I could see. This is a good idea anyway, because it gives you a chance to make sure the camera is working at all.

* You’ll also need enough light outside of the box. The setup works because your fingertips occlude the light or at least cast a strong shadow on the paper. Normal daylight room lighting worked for me and at night I simply had to turn on the room lights. No special set up.

* For best finger tracking lift your palm up away from the paper. The extra shadow cast by your hand may confuse the tracking.

* Depending on how light tight your box is you may see extra smoke around the edges. If so, there’s probably light leaking in.

* So far I’ve tested this on a Vista 64-bit system (Mac Pro running Bootcamp), a Lenovo X61 Tablet PC running Vista, and a MacBookPro running Bootcamp and Vista. I’ve been unable to get it to work on my Toshiba M700 Tablet PC. I think it’s because the built-in camera is conflicting with the USB webcam in the app, but I haven’t dug any deeper to see if I can get it working (outside of disabling the built-in webcam, but that didn’t help).

What about the software? There is source code available which is being archived at Google. I pointed to it the other day. The code is primarily written in C++ and built off of Intel’s OpenCV (Open Computer Vision) library. I can see where one of goals of the coders was support multi-platform, however, it does add a bit of complexity that a large chunk of the market is not going to need. Don’t get me wrong, the approach isn’t bad, however, a little brittleness and install issues may avoided by let’s say supporting primarily Windows and the largest language platform choice for many .NET. So….I’m thinking through how to incrementally port the code. It doesn’t look hard. I could start at the beginning and write my own segmentation routines in CSharp or maybe wrap the OpenCV calls needed for multi-touch, or maybe wrap the high level finger-movement event notifications in the current TouchLib. My current thought is to start near the top and work my way down–but that’s just because I can keep a lot of things running in the meantime. I have to think through this a bit more and figure out how much time I really have to do this :-).

Anyway, if you don’t think all of this is that wild or practical, don’t forget this only takes a few minutes to build and costs next to nothing. This is no Microsoft Surface computer. But hey, if this doesn’t get you thinking about building a full-sized multi-touch system, nothing will. You can guess what we’re up to now!