1
Cover Sheet: The future of natural user
interfaces: a scenario based design
workshop
Jhilmil Jain
Primary Contact
User Experience Strategist
Jhilmil Jain
Microsoft
1310 Villa Street
Mountain View, CA 94041
[email protected]
Arnie Lund
Principal Director, User Experience
Microsoft
27133 SE 25th PL
Sammamish, WA 98075
[email protected]
Dennis Wixon
User Research Manager
Microsoft
One Microsoft Way
Redmond, WA 980652
[email protected]
Extended Abstract: The future of natural
user interfaces: a scenario based
design workshop
Jhilmil Jain
User Experience Strategist
Abstract
Microsoft
This workshop is a forum to advance an integrated
approach to multi-modal Natural User Interfaces. Up until
now the research and design of NUI interfaces for various
modalities (speech, touch, gesture) has proceeded
independently. We propose leveraging rich scenarios as a
framework to stimulate the exchange of knowledge about
the various modalities and how they might be fruitfully
combined, and identifying key areas of future research and
design that make the case for multi-modal NUIs. The goal
is to not only create a vision of synthetic applications of
NUI by connecting researchers but to also discuss ways to
make the vision a reality.
1310 Villa Street
Mountain View, CA 94041
[email protected]
Arnie Lund
Principal Director, User Experience
Microsoft
27133 SE 25th PL
Sammamish, WA 98075
[email protected]
Dennis Wixon
User Research Manager
Keywords
Microsoft
Natural user interface, speech, gesture, touch, augmented
reality, 3D, multimodal interface
One Microsoft Way
Redmond, WA 980652
[email protected]
ACM Classification Keywords
H5.m. Information interfaces and presentation (e.g., HCI):
Miscellaneous.
Introduction
Copyright is held by the author/owner(s).
CHI 2009, April 3 – April 9, 2009, Boston, MA, USA
ACM 978-1-60558-247-4/08/04.
Natural user interfaces are those that enable users to
interact with computers in the way we interact with the
world. When people refer to NUIs they are often talking
about interaction modes such as speech or touch, for
example. But if the focus is on combinations of input and
2
output that are experienced as natural, the collection of
natural user interfaces includes modes such as gesture and
body language, proximity and location, eye gaze and
expression, and biometrics on the input side, and the full
spectrum of audio and visual output, smell, tactile and
object location, and other experiences on the “output” side
(leveraging the full range human senses). A given
combination of type of input (e.g., touch) and output (e.g.,
visual) is a mode, and as we will use the term here multimodal experiences are those that refer to either a
combination of more than one output or more than one
input as part of the natural interaction. Human interaction
with the world is multi-modal, and rich multi-modal
interaction is part of what defines a natural experience.
For this workshop we are particularly interested in multimodal natural experiences that haven’t been studied or
that are just emerging. We are interested in filling in some
of the gaps in the taxonomy of possible combinations of
experiences and exploring the opportunities that result.”
Natural user interfaces aim to provide a seamless user
experience where the technology is invisible. Experience
and action in the natural world is integrated, in that is it
typically involves a combination of multiple modalities such
as voice recognition, gesture, touch, augmented reality
etc. While researchers have published breakthrough work
and designers have created innovative designs in each of
these modalities individually: speech [2, 8], gesture [4,5,
7], touch [3], 3D manipulation, augmented reality [1],
there has been little research work and design thinking
combining them. In other words, this work has been siloed.
We believe there are untapped opportunities in the
integration of multiple NUI modalities.
The theme of the CHI 2011 conference is “Connecting
people, cultures, technologies, experiences, and ideas”. We
believe the future of NUI is really about integrated
experience and to accomplish this goal there is a need for
the leaders in each of these areas to share their
understanding of each modality such that we can work in
concert and “connect” technologies and ideas to identify
the opportunities and challenges, and to stretch the limits
of creating natural user interfaces.
This one and a half day workshop will bring together
researchers with expertise in state-of-the-art design and
evaluation methods for creating speech, gestural, touch,
3D, augmented reality and other interfaces. Participants
will be encouraged well in advance to consider the
questions they have for designers and researchers outside
their modality and the lessons they could share from their
work. Our aim is to establish a forum to set an agenda for
the future of multi-modal NUIs.
Organizers’ background
Jhilmil Jain is a User Experience Strategist at Microsoft.
Prior to that Dr. Jain was a Senior Researcher at HP Labs.
She has several publications and patents in information
visualization, user research, multimodal interaction
modeling, personal information management systems, and
experimental evaluation. She has served as the program
chair for CHIMIT 2009; on the program committees of
various conferences such as CHI, HCII, and UPA; on the
editorial board for the International Journal of Handheld
Computing Research; on the review boards for two books
“Handheld Computing for Mobile Commerce” and “The
Psychology of Facebook”; and is currently serving a second
term as the UX community chair for CHI 2011. She is a
member of ACM, UPA, Phi Kappa Phi, and Upsilon Pi
Epsilon.
Arnie Lund is Principal Director, UX at Microsoft, and UX
Community Lead for Microsoft IT; and occasional instructor
3
at the University of Washington. Dr. Lund’s PhD is in
experimental psychology from Northwestern University. He
co-chaired the CHI conferences in 1998 and 2008. A Fellow
of the Human Factors Society (HFES), he has served on the
HFES Executive Committee, where he oversaw various US
standards activities (including the first US HCI standard).
Dr. Lund founded the Chicago chapter of HFES and has
been active in various Society positions. He was the
president of the Board of Certification for Professional
Ergonomists, and holds a CUXP certification. Arnie has
been a researcher and research manager at companies
such as Bell Labs, Ameritech, US West, Sapient, and
Microsoft, as well as delivering innovative products for the
companies. Dr. Lund has published widely and holds 20
patents.
Dennis Wixon
Dennis Wixon is currently Discipline Lead for Microsoft US
BPD. Prior to this role he managed user research teams at
Microsoft Surface, Microsoft Game Studies, and MSN/Home
Products. Before joining Microsoft, Dennis managed the
usability team at Digital Equipment Corporation, where a
number of important usability methods such as Usability
Engineering and Contextual Inquiry were developed.
Dennis has been an active member of the user-research
community for over 25 years. He co-chaired CHI 2002
served as Vice President for Conferences for ACM SIGCHI.
Dennis has co-authored over sixty articles, book chapters
and presentations on research methods and theory. He is
an adjunct Full Professor in the Human Centered Design
and Engineering Department at University of Washington
and co-edited with Dr. Judy Ramey the book Field Methods
Case Book for Software Design. He and Daniel Widgor are
completing a NUI manifesto titled Brave NUI World.
Dennis holds a Ph.D. in Social Psychology from Clark
University.
References
[1] Azuma, Ronald T. A Survey of Augmented Reality.
Presence: Teleoperators and Virtual Environments 6, 4
(August 1997), 355 - 385. Earlier version appeared in
Course Notes #9: Developing Advanced Virtual Reality
Applications, ACM SIGGRAPH (Los Angeles, CA, 6-11
August 1995), 20-1 to 20-38.
[2] Gong, Y. Speech recognition in noisy environments: a
survey. Speech Commun. 16, 3 (Apr. 1995), 261-291
[3] Hofmeester, K. and Wixon, D. Using metaphors to
create a natural user interface for microsoft surface. In
Proceedings of the 28th of the international Conference
Extended Abstracts on Human Factors in Computing
Systems (Atlanta, Georgia, USA, April 10 - 15, 2010). CHI
EA '10. ACM, New York, NY, 4629-4644.
[4] LaViola, J. J. A Survey of Hand Posture and Gesture
Recognition Techniques and Technology. Technical Report.
UMI Order Number: CS-99-11., Brown University.
[5] Oviatt, S., Cohen, P., Wu, L., Vergo, J., Duncan, L.,
Suhm, B., Bers, J., Holzman, T., Winograd, T., Landay, J.,
Larson, J., and Ferro, D. 2000. Designing the user interface
for multimodal speech and pen-based gesture applications:
state-of-the-art systems and future research directions.
Hum.-Comput. Interact. 15, 4 (Dec. 2000), 263-322.
[6] Watson, Richard. 'A Survey of Gesture Recognition
Techniques'. Trinity College Dublin, Department of
Computer Science, TCD-CS-1993-11, 1993, pp31
[7] Yankelovich, N., Levow, G., and Marx, M. Designing
SpeechActs: issues in speech user interfaces. In
Proceedings of the SIGCHI Conference on Human Factors
in Computing Systems (Denver, Colorado, United States,
May 07 - 11, 1995). I. R. Katz, R. Mack, L. Marks, M. B.
Rosson, and J. Nielsen, Eds. Conference on Human Factors
in Computing Systems. ACM Press/Addison-Wesley
Publishing
Co.,
New
York,
NY,
369-376
1
Proposal: The future of natural user
interfaces: a scenario based design
workshop
Jhilmil Jain
Abstract
User Experience Strategist
This workshop is a forum to advance an integrated
approach to multi-modal Natural User Interfaces. Up
until now the research and design of NUI interfaces for
various modalities (speech, touch, gesture) has
proceeded independently. We propose leveraging rich
scenarios as a framework to stimulate the exchange of
knowledge about the various modalities and how they
might be fruitfully combined, and identifying key areas
of future research and design that make the case for
multi-modal NUIs. The goal is to not only create a
vision of synthetic applications of NUI by connecting
researchers but to also discuss ways to make the vision
a reality.
Microsoft
1310 Villa Street
Mountain View, CA 94041
[email protected]
Arnie Lund
Principal Director, User Experience
Microsoft
27133 SE 25th PL
Sammamish, WA 98075
[email protected]
Dennis Wixon
User Research Manager
Keywords
Microsoft
Natural user interface, speech, gesture, touch,
augmented reality, 3D, multimodal interface
One Microsoft Way
Redmond, WA 980652
[email protected]
ACM Classification Keywords
H5.m. Information interfaces and presentation (e.g.,
HCI): Miscellaneous.
Introduction
Copyright is held by the author/owner(s).
CHI 2009, April 3 – April 9, 2009, Boston, MA, USA
ACM 978-1-60558-247-4/08/04.
Natural user interfaces aim to provide a seamless user
experience where the
technology is invisible.
Experience and action in the natural world is
integrated, in that is it typically involves a combination
2
of multiple modalities such as voice recognition,
gesture,
touch,
augmented
reality
etc.
While
researchers have published breakthrough work and
designers have created innovative designs in each of
these modalities individually: speech, gesture, touch,
3D manipulation, augmented reality, there has been
little research work and design thinking combining
them. In other words, this work has been siloed. We
believe there are untapped opportunities in the
integration of multiple NUI modalities.
The theme of the CHI 2011 conference is “Connecting
people, cultures, technologies, experiences, and ideas”.
We believe the future of NUI is really about integrated
experience and to accomplish this goal there is a need
for the leaders in each of these areas to share their
understanding of each modality such that we can work
in concert and “connect” technologies and ideas to
identify the opportunities and challenges, and to stretch
the limits of creating natural user interfaces.
This one and a half day workshop will bring together
researchers with expertise in state-of-the-art design
and evaluation methods for creating speech, gestural,
touch, 3D, augmented reality and other interfaces.
Participants will be encouraged well in advance to
consider the questions they have for designers and
researchers outside their modality and the lessons they
could share from their work. Our aim is to establish a
forum to set an agenda for the future of multi-modal
NUIs.
Detailed Plan for Conducting the Workshop
Preparation Before the Conference
The maximum of 20 participants in this 1.5 day
workshop will include HCI researchers from academia
and industry with expertise in state-of-the-art methods
for creating and evaluating natural user interfaces.
Workshop participants will be selected on the basis of a
3-5 page position paper. The position paper should
cover the following questions:
What are the 3 lessons you would want to share
from your work for people who have worked in
other modalities?
What are 3 questions you would want to know from
researchers and designers working in other
modalities?
What are some promising problems that would
require an integrate approach
As pre-work the selected candidates will be asked to
prepare two slides: one of lessons and one of
questions. A wiki will be established to share all
selected position papers and slides of the workshop and
facilitate and encourage in-depth discussion well ahead
of the time of the conference.
Workshop Agenda During the Conference
DAY 1:
[1] INTRODUCTIONS AND OBJECTIVES (30 MINUTES):
Introductions, outline of the workshop objectives,
verification/modification of the agenda.
[2] PRESENTATION OF CORE POINTS TO IDENTIFY
DISCUSSION ISSUES AND SCENARIOS (60 MINUTES):
Brief presentation of positions (5 minutes each, as
participants will already have read each other’s position
papers). These will have been grouped by topic in
advance.
3
[3] PRIORITIZATION OF DISCUSSION SCENARIOS (60
MINUTES): Outline of relevant scenarios raised in the
pre-conference discussion that are not included in the
extended abstract or the position papers. Discussion of
a potential model representing experience diversity and
how the scenarios fit. Rating of the top 5-7 scenarios
that the participants would like to use as a foundation,
and that capture a broad area of the experiences of
interest.
[4] SCENARIO BUILDING (REST OF THE DAY): Break-out
into small groups for focused scenario based design
exercises to develop proposed new products that
exploit integration. Ideally we will have 5 groups with 4
people each running in parallel. At least one check-in
will be held to drive the scenarios into areas that
represent the greatest diversity and richness.
“Scenarios are stories. They consist of a setting, or
situation state, one or more actors with personal
motivations, knowledge, and capabilities, and various
tools and objects that the actors encounter and
manipulate. The scenario describes a sequence of
actions and events that lead to an outcome. These
actions and events are related in a usage context that
includes the goals, plans, and reactions of the people
taking part in the episode1” The goal of this exercise is
to envision future use possibilities for each scenario by
describing how people would use NUI to accomplish
their tasks. We will provide paper, pencils, crayons etc.
for sketching and capturing the design concepts.
1
Rosson, M.B. and Carroll, J. M. Scenario-Based Design.
In J. Jacko & A. Sears (Eds.), The Human-Computer
Interaction
Handbook:
Fundamentals,
Evolving
Technologies and Emerging Applications. Lawrence
Erlbaum Associates, 2002, pp. 1032-1050
DAY 2:
[5] SCENARIO PRESENTATIONS (90 MINUTES): Each
group will present the outcomes of the small-group
discussions. Each group will have 15-20 mins.
[6]
PROMISE
AND
TECHNICAL
FEASIBILITY
DISCUSSION (90 MINUTES): Experts from each modality
will weigh in on the technical feasibility of each scenario
– what would be valuable? what are the obstacles
today? what needs further research or new technology?
[7] SUMMARY AND WRAP-UP (30 MINUTES): The
organizers will revisit the scenarios to summarize the
best practices and future research questions.
Follow-Up after the Workshop and Conference:
One of the goals of this workshop is to have the results
reach a wider audience, so the organizers plan the
following follow-up activities after the workshop:
•
•
•
A SIG session will be proposed to inform the wider
CHI 2011 audience of the results and involve them
in the discussion.
The organizers will update the wiki with the
workshop results and publicize it to a wider HCI
community. http://nui.wikispaces.com/
Submit a proposal to create a formal “featured
community” for CHI 2012. Based on the workshop
and SIG we will identify researchers in academia
and industry that have common interests to
systematically build a formal community of core
people using the SIG structure.
Organizers’ Backgrounds
Please see workshop extended abstract
1
CFP: The future of natural user
interfaces: a scenario based design
workshop
Jhilmil Jain
User Experience Strategist
Microsoft
1310 Villa Street
Mountain View, CA 94041
[email protected]
Arnie Lund
Principal Director, User Experience
Microsoft
27133 SE 25th PL
Sammamish, WA 98075
[email protected]
Dennis Wixon
User Research Manager
Microsoft
One Microsoft Way
Redmond, WA 980652
[email protected]
Copyright is held by the author/owner(s).
CHI 2009, April 3 – April 9, 2009, Boston, MA, USA
ACM 978-1-60558-247-4/08/04.
Call for Proposal
This workshop is a forum to advance an integrated
approach to multi-modal Natural User Interfaces. Up
until now the research and design of NUI interfaces for
various modalities (speech, touch, gesture, augmented
reality) has proceeded independently.
We propose
leveraging rich scenarios as a framework to stimulate
the exchange of knowledge about the various
modalities and how they might be fruitfully combined,
and identifying key areas of future research and design
that make the case for multi-modal NUIs. The goal is to
not only create a vision of synthetic applications of NUI
by connecting researchers but to also discuss ways to
make the vision a reality.
We invite practitioners and academics to submit a
3-5 page position paper (in CHI Extended Abstracts
format) plus a short bio by email to
[email protected] by January 4th 2011 in order to
participate in this one and a half day workshop.
Your position paper should attempt to answer the
following:
What are the 3 lessons you would want to
share from your work for people who have
worked in other modalities?
2
-
-
What are 3 questions you would want to know
from researchers and designers working in
other modalities?
What are some promising problems that would
require an integrate approach
Up-to-date information on the workshop will be
available at: http://nui.wikispaces.com/
Please note that workshop participants must register
for the workshop and for one or more days of the CHI
2011 conference. Workshop fees for participants in
2011 are estimated to be $xxxx for a one and a half
day workshop.
Important dates:
Submission deadline – 4th January 2011
Notification
- 15th February 2011
Workshop
- xxx May 2011