User Interface - Microsoft Research at its annual TechFest event this past week (Mar. 5-7, 2013, Microsoft Conference Center, Redmond, Washington, USA) revealed a number of innovations aimed at delivering more natural user interfaces and enabling improved applications of large format displays.
For example, SketchInsight is an interactive whiteboard system that aims to improve storytelling with data using real-time sketching. The SketchInsight application is illustrated in the video below from Microsoft Research.
Microsoft Research notes that SketchInsight takes advantage of advances in input technologies such as text recognition, gesture recognition, and pen-and-touch displays to create a “smart” whiteboard. When a team member starts to sketch something such as a chart or a diagram, SketchInsight pulls information from an existing data set, such as an Excel spreadsheet, determines what the goal is, and retrieves real data to complete the drawing.
Microsoft researcher Bongshin Lee points out that “People could start drawing parts of a visualization they are thinking of as a hint,” Lee says. “Then, the system automatically recognizes the elements of the drawing, consults relevant data sources, and infers and completes the target visualization.” The goal then is for a simple sketch that might have had little real data to become a powerful discussion or presentation tool with concrete, relevant data.
Researcher Lee goes on to say, “SketchInsight’s presentation aspect would be really useful for education. It could be used in lectures, where a lecturer could begin to sketch a complex diagram and the remaining part of the diagram could be completed with additional details pulled in from an outside source.” The research behind SketchInsight is described in a paper (PDF, 1.1 MB) entitled Understanding Pen and Touch Interaction for Data Exploration on Interactive Whiteboards, by Jagoda Walny and Sheelagh Carpendale of the University of Calgary, and Lee, Paul Johns, and Nathalie Henry Riche of Microsoft Research Redmond.
In another presentation (video below) entitled Toward Large-Display Experiences Microsoft researcher Michel Pahud demonstrates using a large display in conjunction with touch gestures, pens, and mobile phones as input devices. The researchers note that, “Large displays are becoming ubiquitous. Soon everyone potentially could have a large office display. This project addresses two important things in the context of an augmented office: 1) when the user is close to the large display, a new user experience designed for large displays, with commands appearing directly next to the finger in combination with a pen. 2) when the user is far from the large display: a model that shows that the phone can be used as a proxy for a large display, whether it is used as a remote mouse or keyboard for digital inclusion; an extension in the context of the current experience, such as a palette for a painting application; or as a device to initiate document sharing on a large display.”
A third Microsoft Research project at the 2013 TechFest entitled SandDance (video below) aims to bring a natural user interface to enable visualization of large data sets. The researchers state that “SandDance is a web-based visualization system that exploits 3D hardware acceleration to explore the relationships between hundreds of thousands of items. Arbitrary data tables can be loaded, and results can be filtered using facets and displayed using a variety of layouts. Natural-user-interaction techniques including multitouch and gesture interactions are supported.”
The above are just three of the many innovations described at the 2013 Microsoft Research TechFest. In addition to the technical demonstrations, a video of the TechFest opening keynote address by Rick Rashid (Microsoft Chief Research Office and head of Microsoft Research) is available here (1 hour). Rich Rashid’s keynote provides background on the history and objectives of the Microsoft TechFest event. The accomplishments of Microsoft Research as demonstrated by the 2013 TechFest event are a strong indication of Microsoft’s depth of capabilities and provide insight into possible future Microsoft product introductions. – Phil Wright
