Using Windows Touch Gestures with .NET (Part 1: Configuration)

The last two posts discussing Windows Touch gave a high level view of using the Windows Touch SDK in Windows 7 with the .NET Framework. In this post, I’d like to go into more detail on using Gestures. Remember from the first post – Gestures cannot be used when the RegisterTouchWindow() has been called.

Configuration – Where to Configure

In the Windows Touch SDK, Gesture messages are sent by default to all windows, without any kind of configuration settings. However, configuration is needed to fine-tune what gestures are sent and which ones are ignored. This is done via a p/invoke of the SetGestureConfig() function in user32.dll in one of two places: in the Load event or when the WM_GESTURENOTIFY message has been sent. Ideally, the Load event is the place to set the configuration, unless the configuration parameters will be changed multiple times.

Structures and P/Invoke Setup

When setting up the configuration for using the Windows Touch SDK with .NET, you’ll need to declare the structure needed to hold the configuration. You’ll also need to declare the external functions (p/invoke) for setting the configuration, along with the values needed for each configuration option.


The Configuration Structure (GESTURECONFIG) is pretty straightforward, with only three ints in the structure. It is good to note that more than one GESTURECONFIG structures can be sent, as this allows specific settings for each individual gesture.

The GESTURECONFIG can be declared as follows:

private struct GESTURECONFIG
{
public int id;
public int want;
public int block;
}

The id field determines which gesture for which you are setting the configuration. (See Here For Gestures And Their IDs). The Want and Block fields determine which features (specified by a flag) within a specific gesture are wanted or blocked. Most Gestures only have a feature flag of ID 0x0000001, as the gesture has only one feature (like Zoom). Pan has five gesture flags: 0x0000001 (All gestures), 0x00000002 (vertical pans with one finger), 0x00000004 (horizontal pans with one finger), 0x00000008 (Pan with a ‘gutter’), and 0x00000010 (Pan with Inertia).

So, a new GESTURECONFIG for activating just a vertical pan would be:


GESTURECONFIG config = new GESTURECONFIG(){
id = 0x00000004, // This is the ID for PAN
want = 0x00000002, // We want to have vertical PAN
block = 0x00000004 // We want to block the horizontal PAN
};


The p/invoke declaration is set up in the class you are using by doing a DllImport to the user32.dll. It will look like this:


[DllImport("user32")]
private static extern bool SetGestureConfig(IntPtr window, int reserved, int ids, ref GESTURECONFIG config, int size);

Remember to include the using / import statement for System.Runtime.InteropServices.

Setting the Configuration

At this point, we have set up all the prerequisite information for setting the Gesture Configuration. The SetGestureConfig() function takes in five different parameters:

  • hWnd – The Handle to the Window that we’re using to capture gestures.
  • Reserved – This is reserved for future use and needs to be set to 0.
  • IDs – This is how many GESTURECONFIG structures that are being in passed into SetGestureConfig().
  • config – An array of GESTURECONFIG structures containing the configuration information.
  • size – the size of the array passed in.

Once called, SetGestureConfig() return a zero if it is unsuccessful in setting the gesture configuration and a nonzero value if successful.

bool result = SetGestureConfig(this.Handle, 0, 1, ref config, Marshal.SizeOf(config));

Putting It All Together

If we were to set the PAN Gesture to only accept vertical gestures, we’d put the previous code snippets together into a single form. It would look something like this:


using System;
using System.Collections.Generic;
using System.ComponentModel;
using System.Data;
using System.Drawing;
using System.Linq;
using System.Text;
using System.Windows.Forms;


using System.Runtime.InteropServices;


namespace VerticalPanOnly
{
public partial class frmGestures : Form
{
private struct GESTURECONFIG
{
public int id;
public int want;
public int block;
}


[DllImport("user32")]
private static extern bool SetGestureConfig(IntPtr window, int reserved, int ids, ref GESTURECONFIG config, int size);

public frmGestures()
{
InitializeComponent();
}

private void frmGestures_Load(object sender, EventArgs e)
{
GESTURECONFIG config = new GESTURECONFIG(){
id = 0x00000004, // This is the ID for PAN
want = 0x00000002, // We want to have vertical PAN
block = 0x00000004 // We want to block the horizontal pan
};

int size = Marshal.SizeOf(config);


bool result = SetGestureConfig(this.Handle, 0, 1, ref config, Marshal.SizeOf(config));
}

}
}

In another post, I’ll demonstrate how to obtain gesture information from the Windows Touch API in .NET once a gesture has occurred.

Touch and Gestures

When programming using Windows Touch (The Windows 7 Touch API), it is important to determine the best method to use. Windows Touch has three main ways of implementing Touch functionality.

  1. Touch Input
  2. Gestures
  3. Legacy

Touch Input

Touch input allows for fine-grained control over the input digitizer. In order to do this, you’ll need to register the window you’re working with to the Windows Touch API via the RegisterTouchWindow() function, located in the user32.dll. If you’re using .NET, this will have to be done through a p-invoke. Once this is done, you’ll need to override the WndProc function and switch on a WM_TOUCH message being passed. From there, you’ll find that that you’ll be able to grab all sorts of information related to the touch in question.

Message FlagFunctionality
MoveDetermines if the touch is movement (as opposed to a touch-down event).
DownA new touch contact has been made.
UpThe touch contact has been broken.
In RangeAllows to determine if the touch message comes from a “hover”, meaning the user has their finger over the screen, instead of on it. Not all hardware supports this.
PrimaryDetermines if this is the first contact point.
PalmDetermines if this is the contact was made by the users palm.

This is in the TOUCHINPUT Structure. It also includes information such as X/Y coordinates, source ID, TouchPointID, Time Stamp, width/height of the actual touch contact, and a few other items.

Gestures

Gestures are the ‘default’ way of handling touch within the Windows Touch API. if you’ll note in the Touch Input section of this post, there is an explicit call to the API to register a window as Touch. This isn’t needed for gestures: WM_GESTURE is always sent to your window unless you call the RegisterTouchWindow() function. Implementing Gestures is done in a similar manner to Touch Input: override the WndProc function and branch on the WM_GESTURE message. From here, the Gesture ID will tell you what gesture is being performed, along with related information.

Gestures are defined as a specific action performed with one or more fingers on a touch screen. This can be something as simple as a single touch on the screen to a more complex task of rotating two fingers in opposite directions.

GestureGesture How-ToGesture IDIs Single-Touch?Is Multi-Touch?
PanTouch the screen with one or two fingers. Drag the fingers up or down.4YesYes
Press and TapTouch the screen with a finger and hold. While holding, touch the screen in a different spot with another finger.7NoYes
ZoomTouch the screen with two fingers and hold. Move the fingers away from each other, or bring them together.3NoYes
RotateTouch the screen with two fingers and rotate them.5NoYes
Two Finger TapTouch the screen briefly with two fingers and let go.6NoYes
FlicksTouch the screen and quickly drag up, down, left, or right. N/AYesYes

To capture a straight-forward single touch / tap to the screen, handle the Click event of your window.

Legacy

If you want only a the simplest form of Touch support, this method is, by far, the easiest to implement. No work at all is required to use the API. The downside of this, however, is that only four different gestures are supported. These four gestures map back to Windows messages that we’re all familiar with.

GestureGesture How-ToMapping
PanTouch the screen with one or two fingers. Drag the fingers up or down.Scroll up / down
Press and HoldTouch the screen with a finger and hold. Wait until a circle is drawn around your finger and then let go.Right Click
ZoomTouch the screen with two fingers and hold. Move the fingers away from each other, or bring them together.Mouse Wheel
TapTouch the screen briefly and let go.Left Click

In conclusion, Windows Touch offers flexibility for simple and complex development tasks. Windows Touch offers basic support for beginners and extended functionality for experts. By planning your application well, and implementing the appropriate touch strategy, you can greatly enhance your user experience and make your apps highly desirable.

Come learn about SketchFlow!

On Saturday, November 7, I'll be presenting an introduction to SketchFlow at the Central Maryland Association of .NET Professionals (CMAP) Code Camp in Columbia, Maryland.

A bit more info on my talk:
Microsoft's Expression Blend 3 has a baked-in tool called SketchFlow, targeted directly at prototyping your WPF and Silverlight applications. In this talk, I'll walk through building a prototype and exercising several SketchFlow features available out-of-the-box. I'll also show how easy it is to distribute a prototype and get back annotated feedback from your reviewers.

A bit more info about the code camp (from cmapcodecamp.org): The Code Camp will run from 8:30am - 5:30pm with 20-25 awesome sessions covering a wide range of database, software and portal development topics. It's totally free. No gimmicks. No sales pitches. Enjoy breakfast and lunch at no charge while you mingle with your peers. To register for this event, visit here.



For even more info, visit http://www.cmapcodecamp.org/.

Starting To Use The Windows 7 Touch API in .NET

I don’t know about you, but I’ve been really looking forward to the touch features built into Windows 7 (also known as Windows Touch). Now that we’ve hit general availability for Windows 7, I took a look at how I might be able to start using the Windows Touch in my .NET applications.


After a little review, it turns out that it is not as simple as you’d think. Windows Touch is not a .NET native, and a platform invoke (p-invoke) is needed to really reap the full benefits. I say full benefits, because Windows Touch sends some events to applications, without doing anything at all. It’s pretty much what you’d expect: a single touch sends a Click event to the form/control in question, holding a touch to the screen will send a RightClick to the form. But in order to do the really neat stuff, you need to use p-invoke.


A basic example of how to perform a platform invoke is creating a simple form which determines the touch capabilities of the client running your application. While this doesn’t use the touch API, it does provide a basic example of how access the user32.dll.


Start off by creating a form in .NET. Be sure to include an imports / using statement which accesses System.RunTime.InteropServices. Next, in the form you’ve created, be sure to add a DLL Import of user32.dll for the GetSystemMetrics() function. This is what we’ll use to determine what touch features are available. It will look something like this:


PInvoke


Next, we’ll need to create some constants to handle the value returned from the GetSystemMetrics function.


TouchConstants


These constants are AND-wise compared with returned value to see all of the capabilities available. This is needed, because a computer might well have an integrated pen and an external pen. The AND-wise compare looks like this.


CheckTouchCapabilities


In case you are wondering, the 94 submitted to the GetSystemMetrics() function is a ‘magic number’ that tells the function to bring back just the Windows Touch-related capabilities. You can reference the Windows API and see other kinds of metrics (How many buttons are on the mouse, whether or not the call is being performed in a Terminal Services environment, whether you’re running a “Starter” edition, etc..)


In conclusion, this is a good start for getting your feet wet in using platform invoke and to determine how to what touch capabilities your computer is capable of. In another post, I’ll describe how to register a touch window in .NET with p-invoke and how to get information from the Windows Touch API.

Our Code Stinks

When I say “our” I am of course referring to the editorial we. I did not mean to imply that your code stinks. How could I know that? I haven’t even seen it. If it does stink, it’s not for the reasons you think. The ultimate audience for your code is not another developer. It’s someone much more unpredictable: a user. The sooner you face up to the fact that your code stinks the easier the road ahead will be.

When a user declares that some piece of software stinks, the complaints normally fall into one of three categories:

  1. It is difficult to use
  2. It crashes
  3. Performance is poor

Notice what this list does not contain: lack of comments, bad framework choices, incorrect design patterns, lack of OO purity, poor variable naming conventions. You get the idea. All these things are important (and the subject of many good books) but users don’t care. Users want software that is easy to use, resilient, and impossibly fast (even when they try crazy stuff). There are people with fashion sense and trendy footwear who handle the interface design, so clearly if the software is difficult to use, it is through no fault of the developer. Since we’re all professionals, I am going to assume #2 is covered (mostly). That leaves us with #3: performance.  Oh, how i hate that word. Buy faster hardware and leave me be!

Performance, it turns out, is not so difficult (well maybe it still is but you can at least make your life easier). It starts with your test data. Your goal as the developer should be to find a data sample that is so mind bogglingly huge that no user could ever even conceive of it. Finding that data is no easy task. You have to consider what a “big” sample looks like right now, what it will look like in a year, and what it will look like in five years. Oh sure, you’ll be long gone in five years but trust me, these things, much like a bad penny, have a way of finding you. Test with that five year data and you won’t be sorry. If someone tells you they have no idea what the data will look like in five years or even what a “big” sample looks like, then stop. Stop developing and don’t start again until you figure that out. Do yourself a favor and find the biggest insurance policy, table, spreadsheet, XML file or image you can, then use it, daily.

The second item you’ll need is a baseline. How fast does your application perform with that crazy test data? Write that number down and keep it in a safe place. After you make a change, re-test and compare the results to your baseline. Did the code get slower? You know immediately what you did and stand a better chance of fixing it. If you can, get multiple baselines with the same data and average them just as a sanity check.

The third key to ensuring proper performance is profiling. You have to understand what your application is doing at run-time. How many times will that method be called? How often are files being read? How long does it take to find an item in that collection? These are all good questions and a good profiler can answer all of them. It’s nearly impossible to guess these answers, so don’t even try. The profiler will tell you exactly where to look and what to fix. Every time I’ve ever tried to guess the source of a performance bottleneck, I wasted time optimizing code that wasn't really the problem. A profiler will quickly help you isolate the bottleneck. There’s even one built into Visual Studio. Multi-tier applications can be complex to profile but you can always divide and conquer by profiling each tier separately.

Once you have isolated you’re bottleneck, optimize it. Optimizing code is an art unto itself. All I’ll say about this topic is that unknown loops are the enemy. Always know what’s going on inside of a loop and how many times you’ll be doing it.

To sum it all up:

Big Data –> Baseline –> Profile –> Isolate –> Optimize –> Repeat

Follow these simple steps and you may hear “this code stinks!” from a user again. No guarantees about the coder in the next cubicle.

Silverlight Firestarter Videos Available

About two weeks ago David Makogon made a post about what then was an upcoming day-long presentation about Silverlight 3. If you missed it the videos are now available on-demand through Silverlight streaming playlist. The playlist sections include the following.
  • Event Kickoff
  • Keynote - Scott Guthrie
  • Silverlight Overview - Tim Heuer
  • Expressions 3 Overview - Adam Kinney
  • Sketch Flow - Janete Perez
  • Silverlight Toolkit - Marcos Matos
  • .Net RIA Services - Brad Abrams
  • Silverlight UIs with XAML Powertoys - Karl Shifflett





XNA on the Zune HD

The Zune HD was released a little over a week ago and I received the unit that I had ordered. One of the first things I did (after loading my music) was to take a look at what had changed in the XNA framework for the ZuneHD. In case you are unfamiliar with it the XNA framework is a .Net based library created by Microsoft for making games. XNA Framework based applications can be run on a PC, on the Xbox 360, or on the Zune. It can either be used with Visual Studio 2008 or Visual Studio Express. With the release of the Zune HD Microsoft added two new classes: the Accelerometer and the TouchPanel. It seems that on every accelerometer equiped device that I've seen there's a digital level available for it. So in sticking with tradition I decided that I'd create one for my Zune HD.

Using Expressions Design I was able to make a few simple graphic assets, imported them into Visual Studio, and had the program up and running in a short period of time. After making the program and seeing that it worked I uploaded a video of it and later received a few request for the source code.





So I've uploaded the source code to the program to CodeProject.com along with a description of how the code works and how it was put together. If you are interested check it out.

Math Without a Floating Point Unit

The everyday development that I do targets desktop processors. The processors are more than capable of meeting the computational demands and performances requirements so its rare that I ever have to think about how long it takes to execute a specific mathimatical operation. But while that's the general case it isn't always the case.

When I was finishing grad school one of my projects dealt with machine vision and automatic classification of photographs. I had an understanding of the algorithms I needed to implement and how everything was going to fit together. During development of the individual components I used small datasets that were sufficient for letting me know that the components were working as designed. It wasn't until I had brought all of the components together that I gave the system full sized images to process (around 2 megapixel) . I knew that a full sized image would take longer to process, but processing of the image took around three hours! In reasearching the cause of the slowness I found that some of the functions I was trying to execute were not nativly supported by the machine's processor and were being emulated. Some of the math operations consumed 50 to 100 times more time than the native operations. There was a time constraint on the exeution time of this program since I had to demonstrate its execution in class during a presentation. To stay within those constraings the more expensive mathimatical operations were replaced with lookup tables and the program was changed from just being multithreaded to taking full advantage of multiprocessing.

In the past few weeks I was reminded of the experience after getting a couple of e-mails from other developers that were trying to figure out why their programs were having such poor performance. Both developers were creating programs that performed graphical processing and both were targetting Windows Mobile devices (which use ARMS processors). ARMS processors are available with a broad range of performance characteristics. On the lower end the processors only support integer operations, have no divide instruction, and typically run around 200 MHz. On the upper end the processors may have hardware implementation for floating point operations (including a divide instruction), built in 3D graphic accelerators, and run at up to 1 GHz. Both of the developers were testing their programs on devices that had no native floating point support and no divide instruction. So these operations were being emulated. That information alone was enough to answer their questions. But I decided to do a few measurements.

I dug up every ARMs based device running a Microsoft OS that I could find. In my possession I have several Windows Mobile devices from PocketPC 2002 devices to a newly released device that will soon have Widnows Mobile 6.5. The Microsoft Zune is also ARMS based. So I included it in my test. I've also got remote access to a few newly released devices. For the test I had each device perform a million additions, subtractions, multiplications, and divisions for both integer numbers and double-precision floating point numbers. For the Windows Mobile devices I did this using native (C-language) programs since the .Net Compact framework does not support floating point operations. For the Zunes I used .Net through the XNA framework. The Zune version of the .Net framework supports floating point operations. Because of the broad range of clock frequencies that these devices used for each device I used the time taken to perform a million additions as a base measurement. My findings were fairly consistent. In general the devices that had no floating point support took about 30-times longer to perform a division on a double-precision number than the integer addition. Devices that had floating point support took about 2-time longer to perform a double-precision division than addition.

The algorithms that both developers were implementing made heavy use of floating point operations. There had been some expectations made about the performance of the programs when run on a mobile device that had been transfered from experience with desktops. Desktop processors have a more complete set of hardware implemented math operations than mobile processors. Both developers also had the same device which had no floating point support. So it comes as no surprise why the algorithms were running so slow. So what could they do about it? There's no universally satisfying substitute for a capable hardware so the exact solution is going to depend on what one is trying to accomplish. For one developer an acceptable solution was to use a different algorithm that produced acceptable results and had a much lower computational demand. For the other developer compromises on the result were not as acceptable so the hardware requirements for his software were revised so that they identified the needed hardware.

The experience from school and the experience of the two developers underscore the importance of making sure that solution's hardware and implementation are in sync with each other.

Silverlight Firestarter online

Microsoft's Silverlight Firestarter is online now, streamed via Silverlight. Go here to connect.
The slides are available here.
Presenter pages:

Upcoming talk: Intro to SketchFlow

On October 14, I'll be presenting an intro to SketchFlow, the UI prototyping tool built into Microsoft Expression Blend 3.0. The talk will be at the monthly Rockville .NET User Group meeting in Rockville, MD. See http://www.rocknug.org/ for more info and directions.

Stick around afterward for Q&A, plenty of pizza, and an ASP.NET 4.0 talk by Kevin Jones, a local Microsoft MVP.