Microsoft CEO Satya Nadella says Surface Neo devices based on Windows 10 X will have a stylus-enabled user interface. Judging from the leaks known so far, the operating system’s pen functionality doesn’t seem to be much innovative, primarily because the main app can interact with OneNote at split screen. The good news is that Microsoft Research has been delving into a more novel, interesting, natural user interface that combines stylus with touch.
(Screenshot via MSPU)
In a recent presentation, Microsoft Research introduced a way to add powerful awareness to tablets, enabling the operating system to better adapt to the way users are handheld devices and naturally suit their intents.
This perceptive technique allows you to perceive nuances in moving and fixing the use scene through posture and actively convert.
Examples include moving the handle, changing the inclination and orientation of the screen, pressing the screen with the palm of your hand while writing or drawing, and detecting which direction the hand is approaching.
Sensing Posture-Aware Pen Touch Interaction on Tablets (via)
To do this, Microsoft Research combines three sensing methods in this system, namely, the original capacitive touchscreen image, inertial motion, and the electric field sensor (grip and hand proximity detection) around the screen frame.
Ultimately, this set of UI enables stroke-first, such as the use of a stylus to “store” and retrieve pictures, physical widgets and floating menus for other clipboard items, and menus that always surround the tip of the pen, no matter where on the screen.
Details of this study have been presented at the ACM CHI 2019 Conference and have received an honorary award for their results.