Incremental Blogger

I just uploaded a screencast that shows handwriting recognition being performed by Mac’s InkWell technology as well as that provided on Tablet PCs and in Vista. In this video, the demo is all done on an iMac within Firefox. The recognition on the Mac side simply uses InkWell. For the Microsoft side, I used the Silverlight-based SearchTIP I described the other day, which uses server side recognition.

Here’s the video:

The video is a little longer than 8 minutes. Enjoy.

I’ve always wanted to do a side-by-side comparison of InkWell and Microsoft’s handwriting recognition engines. You see so much banter on the Internet about Apple’s handwriting recognition being better or that Microsoft’s doesn’t perform up to expectations. I hope this screencast gives people a better idea of how the two technologies compare. Personally, I believe the Microsoft recognition performs better, but that’s me. Others may have differing opinions. No problem.

A couple things that are worthy of pointing out in the video:

* InkWell handles printed text fairly well and does OK with a little bit of cursive text mixed in. However, when all or most of the letters are cursive, it doesn’t do very well. The Microsoft recognizer wins hands down with cursive handwriting.
* The first generation recognizer from Microsoft didn’t recognize all caps text that well. The new recognizer does much, much better.
* The InkWell recognizer appears to leverage stroke order over spatial relationships between the strokes. The Microsoft recognizer does much better at taking spatial information into account which means you can write letters “out of order.”
* The Microsoft recognizer leverages a word dictionary which is easy enough to add words too, however, if the word is a name or something unusual and the words not in the dictionary, sometimes the recognizer doesn’t get the word right. I’ve found if I print the word carefully, the recognition goes up. Same with InkWell’s handling of proper names–although in practice I seem to be able to write names better with InkWell.
* The Silverlight app used in the screencast doesn’t capture the ink as fast as that captured on a Tablet PC, so the quality isn’t as great. This is particularly noticeable in Safari. Because the screencast was performed on a Mac, the InkWell ink looks a bit smoother, but on a Tablet PC you could say the same thing. Silverlight’s ink is good, but the Tablet PC’s ink is even better.
* Editing gestures are supported by both InkWell and Microsoft’s ink recognition, however, the Write Anywhere feature supports the gestures directly. On a Tablet PC there’s no longer a built-in Write Anywhere feature, although there is programmatic support for gestures as well as Flicks gestures (for window navigation) as well as a correction UI built into the Tablet Input Panel (TIP). Is Write Anywhere better than the TIP? It’s probably a matter of choice. The TIP definitely provides more editing control over the ink. InkWell’s Write Anywhere may seem more natural in a greater variety of apps.
* Both recognizers do well with simple math expressions, however, the Microsoft recognizer can recognize bullet lists, simple shapes, and more. Unfortunately, in this simple demo there isn’t a good way to illustrate this. I’ll have to see if I can come up with a better demo.

You can try out the handwriting recognition yourself–even if you don’t have a Tablet PC–by visiting http://www.TabletPCPost.com/Search. You do need a browser that is capable of running Silverlight, however.

I think I down-sampled the video too much so it’s kind of hard to see what’s going on. Fortunately, I wrote large most of the time. I’ll see if I can fix the video.

Update: Want to learn more about InkWell? Check out this blog post on SkinYourScreen.com about InkWell. It does a pretty good job of walking through InkWell’s capabilities.

Scott Guthrie has announced on his blog that Silverlight 1.0 has been released and that Novell will work with Microsoft to build a version of Silverlight that supports Linux. Vert good news for Silverlight.

Linux support is necessary–you can’t leave out these devices when thinking about developing web content–even if they are the smaller part of the market. My crystal ball is watching the low-end of the notebook market and wondering if Linux might have a growth spurt, considering the One Laptop Per Child (OLPC) XO, Intel’s Mobile Internet Devices (MIDs), devices like ASUS’s Eee, and so on. All of these devices might grow favor during the next phase of mobile adoption and if they do Silverlight needs to be there.

By the way, here’s the official announcement about Silverlight 1.0 from Microsoft. As always, there’s lots of talk about video, which is fine–Silverlight has some current leverage there–however, if Silverlight is going to become anything more than a Flash alternative, it’s the rest of Silverlight that I’m most interested in. And there’s plenty to watch for. I’m particularly interested in the next release of Silverlight—Silverlight 1.1. When it rolls out, it’s going to be even easier to develop, richer, higher performance Silverlight experiences since it supports the .Net runtimes. Currently, developers have to use Javascript, which varies in performance from browser to browser and even when it’s good it’s not great.

Aside: Whew, the new inkable SearchTIP written in Silverlight that I just posted about works with the latest build–at least on Vista in IE. I’ll do more testing in the morning. (I can’t wait to try it out on Linux. That’ll be a blast.)

Remember the ole ActiveX-based inkable search tip I wrote awhile back which enabled you to search Google using handwritten queries?

Well, now it has a cousin: a Silverlight-based version. You can test it out now if you’re running Windows or a Mac with a Silverlight capable browser by clicking here.

Like its predecessor, you can ink what you want to search for in a panel and then have your handwriting recognized and the query sent off to Google. In this case however, Silverlight is used rather than requiring a user to download a custom ActiveX control (although you’ll need the Silverlight component of course ). Much nicer in my book.

Also, the handwriting recognition is not done on the client side in this configuration–it’s all done on the server. So you don’t have to have a Tablet PC or Vista SKU to get it working. You can even use a Mac. The ink you write in the Silverlight panel is translated into an XML packet that is sent over to the server via an AJAX request, which in turn is processed by a web service which reconstructs the ink from the XML packet, recognizes the ink, and then sends the reco result back to the client. I’ll talk more about how this is done in another post. I’ll then walk through the code and how I had to configure the server and the like.

Note: this is just a test app I’m working on to experiment with ink reco and Silverlight. I may take it down or rearrange things later–in fact, you can rest assured that minimally the structure of the page is sure to change. I thought people might enjoy trying it out in the meantime.

Oh, a couple tips in case you try out the Silverlight SearchTIP. When you hand write something in the inkable panel you can use a mouse or a stylus. The best looking ink is achievable on a Tablet PC though. Once you write something, a recognized version of it will appear at the bottom-left of the ink panel. In addition, a gray circle with an “x” in it will appear to the right of the ink, which you can click on to erase the ink, if needed. If you’re using a stylus to write and the pen has an eraser tip, you can use it to erase the ink too. You can also you a strikeout gesture to erase ink, which consists of a single ink stroke from right to left drawn over the ink to delete. This gesture is useful for striking out a single word. Once you have your query all constructed, tap or click on the word “Search” located at the bottom-right of the ink panel. Your search results will appear in an iframe below the ink panel. That’s about it….for now. Have fun.

Julie Lerman is really progressing with her Silverlight drawing app.

Reminds me that I need to get my server fixed so I can get my Silverlight apps posted.

I’ve had blast doodling with this Flash-based “Jackson Pollock” painting program (http://www.jacksonpollock.org).

The program works fine with a mouse, however it’s definitely loads of fun with a Tablet PC or UMPC. My personal favorite is to paint using my fingers.

The page will start with a white canvas. All you need to do is start moving the cursor over the canvas. This will cause paint to “spill” from your cursor. Stay in one spot and the paint will accumulate into a big blob. Move the cursor around and a paint trail will thin out. To change paint colors tap on the canvas. Once you have your Pollock painting the way you want it, be sure to capture the screen using Alt-Print Screen or something like the Snipping Tool in Vista. (Be careful not to move the cursor or stylus over the canvas once you have the painting just right, otherwise you’ll spill more paint on your artwork! Kind of annoying, but makes sense.)

I’ve been dabbling with oil-on-canvas effects in Silverlight and this Flash-implemented Jackson Pollock painting program has me rethinking how simple the effects can be. I’d like to take a stab at creating my own Pollock painting app in Silverlight. Maybe someone has time to beat me to it? This would be a great open source project for CodeProject.com.

Ernie Booth has developed a plug-in for Lutz Roeder’s Reflector that takes a Silverlight URL. Nice.

This brings up the age old question: Is there a way I can block people from looking at my Silverlight code? Ernie’s advice is to either place the code behind a webservice or use a code obfuscator.

I tried uploading the Silverlight-based SearchTIP to our public server, but I ran into a snag. Seems that I need to upgrade the version of ASP.NET on the server. I’m not sure what that might impact, so I’m done for the night. Oh well, that’ll be a good weekend project.

Julie Lerman has posted an article on DevSource detailing her explorations with ink in Silverlight. Great read. Take the ink in Silverlight devloper tour here.

I have my first pass at a Silverlight-based SearchTIP working.

SearchTIP is a webpage that enables you to handwrite search queries in the browser. My previous incarnation of this tool was ActiveX based. Because it was written as an ActiveX control, the code was limited to Windows and IE (for the most part, although there was a workaround for Firefox), which isn’t ideal.

With Silverlight, the query page can now run in IE and Firefox on Windows and Firefox and Safari on the Mac. (Silverlight doesn’t support Linux.) You can use a Tablet PC, UMPC, standalone digitizing pad, a mouse, or other pointing device to write your text. The recognition is done via a server-side component, which currently I have running on my own development machine. I’ll have to see what I can do about deploying a public version of this on a server so that other developers can see how server-side recognition can be done.

I’ll see what I can do. Layne? Any ideas?

I’m trying to resize a Silverlight control on the fly. I’m using the code from this MSDN page.

Unfortunately in IE 7 I’m getting a size of zero for the Silverlight control’s actualwidth/actualHeight dimensions in the loaded event. This only happens in IE 7. In Firefox, I’m getting the correct size. Hmmm.

MSDN suggests the following code to set the original size to:

// Retrieve a reference to the control.
var control = sender.getHost();

// Do initial layout of the app based on initial size.
updateLayout(control.content.actualWidth, control.content.actualHeight);

Unfortunately, this gives a size of zero in IE in the loaded event.

To get around this I use:

var silverlightObj = document.getElementById(”silverlightObj”);
updateLayout(silverlightObj.clientWidth, silverlightObj.clientHeight);

Seems to work in IE and Firefox. I need to test in Safari.

ArsTechnica: “Mono project lead developer Miguel de Icaza says that the Mono development community plans to have an experimental Linux-based Silverlight browser plug-in ready for testing by the end of the year.”

The wrong developers are doing this.

Over the weekend I’ve been experimenting with a version of InkPlayer written with Silverlight.

Originally I wrote InkPlayer as a C++ app that generated Flash SWF files which showed animated ink drawings. I could fairly easily embed these Flash animations (one or more instances) within a blog post. I’d like to do something similar with Silverlight.

Currently in Silverlight I have a page that I can use to draw something. The app then displays the code I need to place in a blog post in order to play back the ink drawn. I’m able to create an instance of the playback code in Silverlight as linked to above.

That’s not too hard to do, although placing a Silverlight app in a blog post requires a script block, which not all blog services allow. Likewise, you have to upload a series of files which is kind of messy. It looks like it’s possible to use Microsoft’s new Silverlight hosting service to simplify the app packaging, however, I haven’t gotten any further than signing up for an account yet. If you’re interested, Tim Heuer has some instructions on how to use the new service to host your Silverlight app.

The part I’m still trying to figure out is how to host multiple instances of the same Silverlight app, or similarly generated ones, in the same post or html page. Minimally duplicate IDs, variable names, and the like need to be avoided, however, is it even practical what I want to do?

I began to wonder if the Silverlight hosting service helps to solve some of these issues or if it introduces some other concerns. As a simple test I tried to take Tim Heuer’s “Click” example and call duplicates of it in a page, however, I couldn’t even get one instance to work, so I tried a sample from Bryant. Unfortunately I’m getting a run-time error in his app. I tried hosting two instances of it anyway in a page and sure enough they conflict. I tried making IDs and so forth unique, but that doesn’t seem to get multiple instances working in a single page. I’ll probably need to create my own simple hosted Silverlight code to see what special techniques I need, if any, for hosting multiple instances of a Silverlight app.

I also need to go back and find out if Silverlight really needs a parent element name as well as a unique name for each instance when it’s used. Ideally embedding Silverlight content should be no more challenging than embedding let’s say a YouTube video.

The Mix07 folks wouldn’t let me into the conference to cover ink support in Silverlight (excuses, excuses), but that didn’t stop me. I booted up my UMPC, attached a webcam, and connected up to UStream.tv to record whatever I could–even if I could get no further than the conference hallways.

Here I caught up with Gavin Gear of Microsoft’s Silverlight team, who was kind enough to join me in the hallway, and show me a couple Silverlight applications that illustrated Silverlight’s new inking capabilities.

In the first five minutes of the video Gavin talks through the cool PhotoStack app that Microsoft was showing at the conference. For a simple demo app it had quite a few features including photo annotation, server-side ink recognition, ink searching, a full-screen slideshow mode, and numerous cool transition effects typical of XAML apps. I hope Microosft makes this code public soon. I’m sure there are lots of good ideas for developers inside.

From minutes five to eight, Gavin switches over to work on a blog post. Nothing much interesting here except that you can see we were pretty informal about the recording. He got a phone call during this time too, so for a couple minutes I turned off the sound, although you’ll get a pretty good picture of the floor and our legs in the meantime. Heh.

Then at about 8:15 Gavin starts up again with a demo this time of the Ink Tattoo Studio. It’s a clever simple app that illustrates how to overlay ink on photos in an entertaining way. Unfortunately, after a couple minutes the video hangs, but you’ll get the idea.

If you couldn’t quite make out what’s shown in the video embedded above, you may want to check out a video recorded by the Mix folks which is of much better quality showing Gavin demo Silverlight ink.

I’ve been tinkering with a tiny Silverlight app, and I kept running into the situation where the code would work in IE and Safari, but not Firefox. Turns out I was using some older sample code as a guide that had a problem.

In particular, when creating the silverlight object, I was using code like this:

<div id=”agControlHost”>
  <script type=”text/javascript”>
    Sys.Silverlight.createObjectEx({source:’page.xaml’,
    parentElement: agControlHost,
    id:’silverlightObj’,
    properties: { width:’300′,
    height:’200′,
    background:’#ffffffff’,
    isWindowless: ‘false’,
    framerate:’24′,
    version:’0.8′ },
    events: { onError:null,
    onLoad:null },
    context:null });
  </script>
</div>

Unfortunately, I kept getting an error that the parentElement agControlHost didn’t exist. It took me awhile to realize that I needed to use getElementById() instead of using the element’s name by itself to fix the problem. Here’s the correct way to create a silverlight object:

<div id=”agControlHost”>
  <script type=”text/javascript”>
    Sys.Silverlight.createObjectEx({source:’page.xaml’,
    parentElement: document.getElementById(’agControlHost’),
    id:’silverlightObj’,
    properties: { width:’300′,
    height:’200′,
    background:’#ffffffff’,
    isWindowless: ‘false’,
    framerate:’24′,
    version:’0.8′ },
    events: { onError:null,
    onLoad:null },
    context:null });
  </script>
</div>

Now my Silverlight experiment works just fine in IE, Firefox, and Safari.

Lots of people like the idea of the UMPC, until they see the price–some north of $1100. It’s unfortunate. There have been lots of reasons bantered about as to why the prices are so high. Most sound like uninspired excuses to me. The devices were designed with low-cost, off the shelf components in mind and somewhere, somehow that reasoning vanished. I don’t get it.

Anyway, now we hear that Asustek is going to be launching Classmate PCs (based on Intel’s design specs) in the $249-$549 range that are surprisingly similar to UMPCs in terms of their core specs: 900MHz M ULV Celeron, 7″ display (800×480) and 256MB of DDR-II memory, and WiFi. One difference is that the Classmate PC includes a keyboard and the Classmate PC uses a tiny 1GB flash drive.

Odd. So either that extra bit of memory in most UMPCs is worth about $500 or the fact that it doesn’t have a keyboard presses the price to an extreme $1000. Sounds fishy to me.

If Intel and the OEMs are able to bring to market computers in this range and the OEMs don’t rethink their positioning of UMPCs, the UMPCs are all but dead at least in the education market.

Now over the years, I’ve heard many low-ball prices quoted before new equipment comes to market. This may be one more case of this, so I’m not exactly holding my breath. We’ll have to see if these prices are maintained.

However, it’s clear to me that Microsoft’s Origami concept has some stiff competition and much of it is Intel inspired. Couple weeks back at Beijing IDF Intel announced several UMPC-like Linux-based devices (called Mobile Internet Devices–MID) and now Asustech is talking about cheaper, lower-cost UMPC-like, Intel spec’d laptops. At times like this, I fall back on my old addage, “Follow Intel. The market goes where Intel goes.” Is Intel behind the UMPC or have they moved on? Which side of the track is Microsoft going to be on now? Stand behind their UMPC design and nourish the UMPC hardware and software ecosystem or let Intel lead and switch to these other platforms? ThoughtFix has some thoughts on the MID vs UMPC competition that’s worth checking out too.

The reason I’m thinking about this is that Silverlight and similar technologies could have a huge impact here–for getting applications running on these inexpensive devices.