When seeing the early sketches of Windows 7 designers, like those shown here, I wonder: Do they use a Tablet PC to design with?

This blog post on the Windows 7 Engineering Blog doesn’t give us a hint as to what the designers might use. In part the article says:

“We captured our, many, ideas in very quick sketches that we shared via an internal website. Transferred from the whiteboards in our offices and hallways, these took less than 5 minutes to sketch each.”

My guess is that most of the sketches were done on paper and scanned in, but that’s just a guess. Maybe they use AutoDesk’s Sketchbook. That would be my first choice anyway.

This talk of early design sketches brings me back to the SketchFlow feature that the Expression Design team is working on. Would the Windows designers have used SketchFlow if it were available? Would they have made their sketches within Expression Blend?

Outside of these questions, I think it at least illustrates to me my original belief: That professional designers want much, much better sketching tools than those shown in Expression Design 3 for SketchFlow. Minimally, we’re talking at a level set by the former Alias Sketchbook.

Again, I can’t help but think of how much of a mistake it was for Microsoft’s acquisition team to pass on acquiring Sketchbook. Of course, that could go for Apple too. I think they missed out–especially considering their core artist-slash-designer audience.

Related posts:

1. SketchFlow screams Tablet PC, but is anyone listening?
2. Expression Web and Blend to join MSDN
3. Updated: SketchFlow introduces ink to the design process

At Mix09 Microsoft introduced SketchFlow as part of the next release of Expression Blend. It’s a quite impressive concept: Keep the design process as “sketchy” as possible in order to encourage people to evolve better designs rather than getting caught up in minor details. A programmer might sketch out a sequence of screens, for instance, and then connect them up in SketchFlow until the flow seems right. Then, and only then, is the design committed and a better looking UI created–in some cases with a simple change in a style. Here’s a screenshot of SketchFlow in use:

It’s a pretty good idea and one I’d like to try.

Currently, I work through my designs either in OneNote, Journal, or on paper and I’d like to try using Blend 3’s approach when it becomes available. (There is an Expression Blend 3 Preview available now, but it doesn’t include SketchFlow.)

But here’s the thing: Whereas SketchFlow would appear to be ideal for a Tablet PC owner like myself, I think the approach falls flat. In fact, I think the Expression Blend team needs to spend more time with heavy Tablet users to see what they are missing in their own product designs.

Rather than rant, I’m going to try to be constructive as possible and spell out what I’d like to see with Blend. Some of this is a repeat of my longer post yesterday. If you’re really interested, you can read it first and then come back. I’ll wait.

Here’s my main issue, a sketch flow tool needs to have a good story about where the sketches come from in the first place. Focusing on screen sequences and transitions is OK, but that’s in the middle of the design process and I don’t think is where there’s the main design challenge.

For large organizations or those that have very well-defined processes, I guess I can see SketchFlow’s value in itself, but it really leaves out the sketching part. Yes, I know you can draw from within Blend, but it’s not high-quality sketch friendly. And is there any question that a designer will elect to use high-quality sketching tools rather than a Paint-like set of tools? I don’t think so.

From an engineering standpoint, the problem is that this top down focus on process would make so much more sense if there was bottom up value that could be leveraged. In other words, there should be very good sketching support within Blend or as part of the story. First. There isn’t. Instead you’re supposed to use some other tool. According to the Mix09 session video, I guess you’re supposed to use paper and then scan it in.

Uhm, this is the year 2009 folks. We have mobile computers. We have Tablet PCs. We have connectivity. We can be digital.

Now there’s nothing saying that Blend has to support full sketching. That’s fine if it doesn’t. However, what’s really needed is a sketching tool. It’s clear to me that what’s missing is Sketchbook, now owned by AutoDesk. There are other illustrator-oriented tools, but this one trumps all others. To me, any demo of Blend should include drawing in Sketchbook and then copy/pasting the result into Blend. That’s what I’d shoot for anyway. If you’re part of the Blend team, reread that last sentence: It’s got to work with copy/paste. You need to be able to minimally use the snipping tool to grab a portion of something in Sketchbook and paste it into Blend. That doesn’t work now. It should. Right now there’s a model of storing an image to a file and using that. That’s not fast enough for quick and dirty work.

At this point I’ll repeat what I think a bunch of us have said in the past: Microsoft should have purchased Sketchbook. That was a fundamental blunder in the larger view of things when it comes to advocating for the Tablet PC platform. With Blend 3’s SketchFlow isn’t it obvious now how Sketchbook could have fit into the product suite? Sketchbook would have been used to create sketches, SketchFlow would have used them, and everyone would have been happy. Instead, I imagine there was this whole argument over Sketchbook implementing “ink” differently than Microsoft so it wasn’t a match. Think again. Blend and Design and WPF and Silverlight–all products growing in adoption–have a different take on “Microsoft ink” too. So that was a false argument and one that’s led to a whole in the Tablet story and I think the success of the Tablet ecosystem as a whole. It’s cost millions and millions in lost adoption of the Tablet form factor. Anyway, that’s enough time on my soap box. I’ll save that rant for another time. Back to SketchFlow.

Why would I demo SketchFlow with SketchBook? On a Tablet PC? Because it keeps the content digital. That’s number one. And secondly, it showcases the value of the Tablet PC in the broader story. That’s a Microsoft platform after all.

Now, as much as I love Sketchbook, I don’t use it. It costs too much money. Yeah, I’m cheap. Instead, I think through my ideas most often in OneNote or Journal. The sketching tools are much weaker here than Sketchbook, but they’re reasonable. I can easily edit, select multiple colors, move things around, and share my drawings with others via files or screenshots.

Given that Microsoft doesn’t own Sketchbook, from a Microsoft standpoint a reasonable alternative would be to demo SketchFlow using OneNote. I’d suggest leveraging it with its infinite pages and infinite document approach. No file worry. No saving issues. Just draw. That’s a good match for sketching. The issue is, getting OneNote drawings into Blend. That’s not currently supported either as ink or as ink converted into vectors within Blend. It’s clunky, but you’d have to use the snipping tool to capture a sketch, save it to a file, import the image file into Blend and then select and use it from Blend. Minimally, the team should support copy/paste from OneNote. That would make a much more compelling story than the image file approach.

Now with images pasted into Blend, there’s no editing, which is unfortunate, so really staying with vectors is the better approach if possible. With vectors, though, there’s an issue of rendering. Blend pencil “ink” is different than OneNote or Windows ink. I’ll leave it to the geniuses here to argue how to address this, but I’d say maybe a quick fix would be to add real ink to Blend and potentially Blend ink to OneNote. Word has run into these kinds of issues, so maybe someone has some fixes there.

You might wonder, what’s so bad about Blend’s pencil tool for drawing? Well, give it a go. It’s not the same fidelity as Tablet ink. But let’s say designers could get over them. Where Blend’s pencil tool fails, is in handwriting little notes. Try to write something quickly with a Tablet PC stylus. It doesn’t work well. Try dotting the letter i. You can’t. You’ll have to draw a little circle. You see, Blend is not handdrawing oriented. I’d say, if Blend detects a pen being used, it should switch input models and render high-quality ink. If a mouse is used, you get mousey looking ink.

Now if ink and image pasting is supported in Blend, then I see lots of other possibilities here with whiteboard apps, Journal, and so on. To me that doesn’t sound that difficult to implement.

And it should would give you the potential for a stunning demo. Hey, I could even see someone sketching on Surface or a large HP TouchSmart or whiteboard, using Mesh to store content to the cloud, and in near-real-time have the content show up available on another machine running Blend. OK, that’s probably too complicated. Give the demoer OneNote instead. Just make sure they’re drawing real-time in front of the crowd using a Tablet PC .

It’s not just Blend that I’m concerned about the Tablet experience with. There’s a SketchFlow viewer that supports annotation and given the Mix09 demo I’m wondering about its inking quality. It ought to be stellar on a Tablet PC. I’m not so sure if it is.

To sum all of this up, my concern with Blend is that it’s not a good enough Tablet citizen. From the outside it looks like it should be an amazing platform for Tablets. However, on closer inspection it’s best with a mouse and marginal with a Tablet PC. It should be screaming with a Tablet PC.

Update: At the end of this Mix09, titled “The Future Of Expression Blend,” Douglas Olson (General Manager of Expression) and Christian Schormann (Director, Program Management of Expression) answer the question, “What about the Tablet PC?” The response? Blend is not optimized for the Tablet PC and support for it is something they will consider in the future. They go on to say that you can still use the Tablet with Blend, but I’d add that from my experience Blend works much better with the mouse than the Tablet.

Now I’m kicking myself that I didn’t attend Mix09 than for no other reason than to track down Blend team members and make the case for Tablet support. I think they’re missing out. And the thing is I think they can take an incremental approach to getting to where they already should be.

Related posts:

1. Do Windows designers use Tablet PCs?
2. Updated: SketchFlow introduces ink to the design process
3. Expression Web and Blend to join MSDN

If Twitter is any gauge, a forthcoming feature of Expression Blend, called SketchFlow is one of the highlights of this week’s Mix09 conference.

In brief, SketchFlow is a paper-and-ink-like model applied to application design within Expression Blend–a tool used by WPF and Silverlight developers. The idea? Sketch out application concepts and sequences of your ideas in ink first before you commit them to the more straight-jacket designed developer apps. By staying in sketch mode, the theory goes, you’ll have more flexibility to try out different and hopefully better ideas. SketchFlow is not just about drawing states of an application though. You can animate the handcreated components too. Yes, the sketches become manipulatable objects. And when you have the design the way you want it, the goal is that it shouldn’t be too difficult for Blend to generate a pixel clean and aligned version of your design.

Like a lot of people, when I saw Microsoft demo SketchFlow during the Mix09 keynote (starts about 99 minutes into the presentation), I immediately jumped for the Blend 3 Preview download. Unfortunately, SketchFlow is not supported in it yet. We’ll have to wait for a later build.

In the meantime, we can study the Mix09 videos (more are to be posted later according to Arturo Toledo PM of Blend and Silverlight). I don’t quite know how SketchFlow works, but supposedly you’ll be able to draw you dialogs or whatever, sequence them, and then transition them into a final version of your app.

You’ll even be able to generate documentation from your SketchFlow sequence. Handwritten annotations are also supported. And playback is supported on Windows as well as the Mac.

As a Tablet enthusiast, the part that interests me the most right now is the drawing of the sketches. Supposedly you can do this from within Blend. So look closely at what Microsoft was showing at Mix. These sketches don’t appear to be using the classic Tablet PC ink. That’s quite notable.

However, I think we have a couple different “sketch” styles here going on here. The details aren’t clear though. Via Twitter people are saying that you can create sketches in Blend, which would be terrific, but I wonder if they are meaning a subset can be created in Blend and for others you have to create bitmaps in some 3rd party app. Not sure.

Anyway, I’ve been thinking for awhile that the definition of what ink is in Windows needs to be revisited. It’s not just a smooth stroke. It needs to have variability and the appearance of pencil on paper, or ink on paper, or oil on canvas and so on. WPF and Silverlight are beginning to take us in this direction in terms of viewing ink as a vector with some style applied to it. It appears as if the Blend team has gone this direction and is making the sketches look “realistic.” If so, this beats hands down any “Tablet” ink.

If this is the case, I’d like to see the Blend style ink migrate into WPF and Silverlight as inking styles. Maybe one day they can then be blended into the Windows platform, but hey, if they’re available to all WPF or Silverlight apps, I’d be quite satisfied.

Now another thing that appears to be going on here, is that within the Blend demo you can see the person creating standard controls with a handdrawn rendering style, but in actuallity they’re drawing the control just as they normally would with the mouse. The rendering is different; not the input approach. The person isn’t drawing the control freehand.

I hope that’s not what people are referring to when they say Blend 3 can be used to create these sketches. This would just give the viewer the impression that the screen is handdrawn, but it’s not. Now maybe that’s sufficient, but I don’t think so. People would quickly catch on and think of the wiggly-styled controls just like any regular control. I hope there’s more to it than drawing perturbed vectors with the mouse. I wonder.

Reading a post by Christian Schormann on SketchFlow this sentence seems to lead me to believe that there’s “sketching” and then there’s “sketching” when it comes to SketchFlow:

“SketchFlow is informal and quick, enabling you to sketch out plentiful ideas for dynamic interactions in a cost effective manner. SketchFlow also supports the evolution of your rough sketches into living and breathing prototypes that can be as real as you need them to be.”

If I read this correctly, you can create “sketches” quickly–I’m guessing using the mouse and keyboard rendering them the wiggly style and handwritten font (Buxton Sketch TTF). These aren’t sketches at all; they’re just rendered as them. And then there’s the “rough sketches” which I’m guessing are scanned paper images or something drawn in Photoshop or Illustrator. Again, I hope this distinction doesn’t live out.

Now where these “rough sketches” are created is a very interesting question to me. I see a huge opportunity here with even OneNote or Journal or even an add-in to Visual Studio (like I’ve worked on in the past) that supports some form of inking. Of course, AutoDesk’s Sketchbook would be the ultimate here with it’s “realistic” drawing instruments, but we’ll have to see. In fact, come to think of it, maybe an ArtRage style set of screen renderings would fit in here. Oil painted or crayon rendered screens anyone?

To me, Blend 3’s SketchFlow is a perfect place for the Tablet PC. As someone mentioned to me on Twitter, SketchFlow screams Tablet. I’m very excited to see how this plays out.

Update: A few more screenshots are on my keynote post from the other day.

Update #2: About the notion of drawing freehand sketches with the pen versus the mouse. I can appreciate the issue of freehand drawing and the need to clean it up (such as snapping coordinates and algning multiple objects in a column or row) as well as the need for recognition. However, that’s where the state of the art is and that’s where the magic is. The Tablet PC group has created a reasonable first attempt at shape recognition in their SDK and possibly this could be used. I see lots of context here so the recognizer would minimally need to be adjusted. However, for standard dialogs, I can see handdrawn creations being recognized in the background whether they are replaced with recognized symbols or not. And I think there’s this whole other question as to how much recognition you need to do anyway. There’s lots of room for creativity here. A programming model and access to ink (if there is any) are two requests I’d make.

Update #3: The Mix09 session on SketchFlow (Sketch Flow: From Concept to Production) is now online. Christian Schormann, Director of Program Management for Expression, gives the presentation.

Well, it looks like my original take on Blend 3 was right. Blend 3 does not support realistic looking sketching. It allows you to draw vectors with the pen tool just like Blend always has. For those of you familiar with Blend, these are not quick and dirty sketching tools like let’s say in AudoDesk’s Sketchbook. And they aren’t handwriting friendly pen strokes either unless you write very large.

So really, SketchFlow is not really that Tablet friendly. Mouse and keyboard still are king. That’s not a bad thing, but it’s worth pointing out.

In a nutshell, here’s a screenshot “handdrawn” in SketchFlow’s Blend 3:

I wonder if you’re thinking what I’m thinking: This looks like something I’d draw. Not very inspiring, but doable.

As a step up, Blend 3 also provides predefined handdrawn looking styles for all common controls. That way you can keep your screens looking less finished, which will help promote thinking in terms of the design itself than the appearance of the controls. These are called Wiggly styles:

Now here’s a SketchFlow screenshot more like what Microsoft showed in the keynotes:

Looks better doesn’t it? It looks more like what a designer would come up with. Unfortunately this is something that was not drawn in Blend. In fact, it wasn’t drawn with any tools. It was drawn on paper and scanned in.

Uhm, this is quite unfortunate.

I think Blend is missing a fundamental opportunity here: Sketching itself needs to be part of the electronic process. Either better tools need to be added to Blend or the Expression team needs a better 3rd party story. With Tablet PCs especially, but of course with any Wacom digitizing tablet, there should be a reasonable story for creating sketches.

Without this easy flow, there’s going to be more work unless most of the time is spent on transition editing. Maybe that’s the case. Also, there’s the issue of managing all of these assets during the design process. Imagine all the permutations.

Minimally, here’s what I’d like to see. For Blend 3, I’d like to see two tie ins for sketching, since my guess is there’s no feasible way to add realistic sketching to Blend 3. First, AutoDesk’s Sketchbook needs support. You should be able to copy and image from Sketchbook into Blend without creating any intermediate image file. Blend should paste the image copy from Sketchbook and create a local Blend asset for you. The copy/paste is a little extra work for the user, but not terrible. Currently, since Blend doesn’t support copy/paste like this a user has to save to an image file–probably with their other working sketches, then copy the file over to Blend’s project infrastructure.

Alternatively, there needs to be good OneNote and/or Journal support (I’ll leave MS Paint out of the picture for now, because I doubt most people would use it for designing.) I can see people using OneNote to sketch out their ideas and then after they’ve settled of a few of them copy/paste or import the content into Blend–again without having to copy/paste to an image file via the snipping tool or any direct means of copying. Note also that OneNote doesn’t have a file model like Word or Sketchbook or most other apps. You just draw and write, you don’t think about saving to a file. All of that is done for you in the background. So to use ink or an image out of OneNote requires creating an external image file (via the snipping tool possibly) and then importing the image into Blend. That’s not optimal, nor is it the natural way to work. As I see it, minimally Microsoft should make OneNote the sketch king and Blend the flow king.

I’d take this issue of copy/paste even further. Just being able to take a screenshot of an existing window let’s say and paste it into a page makes sense to me. Blend ought to be able to do this.

Beyond all this, what I’d like to see is Blend become a better Tablet PC tool. May I suggest to the Expression team that one of the things they do is switch to Tablet PCs for their notebooks. And by all means if you’re giving a presentation, think about using a Tablet.

Interaction with a mouse and keyboard still needs to be very good, but pen support needs to be better than it is. Small targets are already an issue in Blend. Now the sketching tools or at least sketching process needs to be revisited.

I applaud the Expression team for trying to improve the design process and like Bill Buxton said in his keynote, this is a step in the right direction.

Related posts:

1. SketchFlow screams Tablet PC, but is anyone listening?
2. Do Windows designers use Tablet PCs?
3. Thoughts on ink-enabled designing