1. business and industrial
2. business operations
3. management
4. project management

The Wheelchair Project
An investigation on how to improve
an information sharing system for disabled people
JOHAN
WEDFELT
Master of Science Thesis
Stockholm, Sweden 2008
The Wheelchair Project
An investigation on how to improve
an information sharing system for disabled people
JOHAN
WEDFELT
Master’s Thesis in Human Computer Interaction (30 ECTS credits)
at the IT Program
Royal Institute of Technology year 2008
Supervisor at CSC was Fredrik Winberg
Examiner was Yngve Sundblad
TRITA-CSC-E 2008:100
ISRN-KTH/CSC/E--08/100--SE
ISSN-1653-5715
Royal Institute of Technology
School of Computer Science and Communication
KTH CSC
SE-100 44 Stockholm, Sweden
URL: www.csc.kth.se
Abstract
The wheelchair project
- An investigation on how to improve an
information sharing system for disabled people
This report investigates how to better correlate the needs of wheelchair users
and disabled people with an information sharing system being designed for
disabled people in Japan. By using two different test groups, two different web
based tests and one interview, some results for the future system development
process were retrieved. A problem that deserves mentioning is the low amount
of users taking part in the tests and the interview. Due to this low amount of
users, the results in this report are limited. Nevertheless, there are clear
indications that wheelchair users are interested in sharing information with each
other regarding things such as toilets, grocery stores etc, as well as getting
access to information about places like these. “Understanding how to
implement functions answering to these interests”, should be a subject that is
further researched during the future development process. Some ideas regarding
specific future functionality as well as some ideas on which methods that can be
used for the continuation of the project are also presented in this report.
Sammanfattning
Rullstolsprojektet
- En undersökning i hur det går att förbättra
ett informationsdelningssystem
för funktionshindrade
Den här rapporten utreder hur det bättre går att korrelera de behov
rullstolsburna och funktionshindrade har med ett informationsdelningssystem
som designas för funktionshindrade i Japan. Genom att använda två olika
testgrupper, två olika webbaserade tester och en intervju, samlades det in
resultat som kan komma att användas för den fortsatta framtida utvecklingen av
detta informationsdelningssystem. Det bör dock nämnas att ett problem som
dök upp i den här undersökningen, var det låga antalet användare som
medverkade i testerna, och under intervjun. P.g.a. detta låga antal användare, är
resultaten i denna rapport begränsade. Trots detta finns det starka indikationer
på att rullstolsanvändare vill dela information mellan varandra angående
specifika platser som t.ex. toaletter och matbutiker. Det finns också resultat som
visar att det är intressant att denna typ av information görs lättillgänglig för
användarna av systemet. “Hur det går att implementera funktioner som svarar
mot dessa intressen” är något som bör utredas mer allt eftersom projektet
fortskrider. Några idéer angående möjlig framtida funktionalitet och
tankegångar angående vilka metoder som kan användas för projektets fortsatta
utvecklingsprocess, presenteras också i denna rapport.
Acknowledgments
First of all I would like to thank professor Akira Aiba for providing me with the opportunity to
come to Japan and perform my master thesis at the Aiba-laboratory, and express a “ありがとう
ございます” for all the support provided throughout the project. Secondly I would like to thank
professors Hitoshi Kuwata and Katsumi Nitta for helping me contacting suitable user groups
similar to what the end users to the system may look like. I would also like to thank professor
Yutaka Nakai for making an effort in checking the questions that were used in Survey 1. To move
on I would like to thank Mr Hideki Sorakado for providing help with the translation in the
laboratory, and assisting me during the first meeting with the Heart Net Waffle group, Mr Hideki
Tounai for helping me test parts of System test 1, and the rest of the people residing in the Aibalaboratory during my stay in Japan, I really enjoyed it. みな！ありがとう　ございます。Also I
would like to thank members from the Heart Net Waffle organization for participating both in
System test 1 and in the interview session, as well as I would like to thank the member from the
Aoba Barrer Free Support 21 group taking part in System test 2. Finally I would like to thank my
supervisor Dr Fredrik Winberg for providing support through a number of phone calls throughout
the project as well as giving me input on how to improve this report once I came back to Sweden.
Table of contents
1 Project background...............................................................................................................................1
1.1 System history.................................................................................................................................1
1.2 Goal.................................................................................................................................................2
1.3 Why interesting?.............................................................................................................................2
1.4 Users and installations....................................................................................................................3
1.5 The system and how it is constructed.............................................................................................4
1.5.1 Hardware & Software............................................................................................................4
1.5.2 Interface & Functionality.......................................................................................................5
1.5.2.1 Altering information section......................................................................................6
1.5.2.2 Viewing information section....................................................................................15
2 Methodology.......................................................................................................................................25
2.1 Background of Methodology........................................................................................................25
2.2 About the data...............................................................................................................................26
2.2.1 Qualitative data....................................................................................................................26
2.2.1.1 Qualitative data concept...........................................................................................26
2.2.1.2 Why open questions?...............................................................................................26
2.2.1.3 Open questions setup...............................................................................................27
2.2.1.4 Open questions trouble and sources of error............................................................27
2.2.2 Quantitative data..................................................................................................................28
2.2.2.1 Time usage concept..................................................................................................28
2.2.2.2 Why time usage?......................................................................................................28
2.2.2.3 Time usage questions setup.....................................................................................28
2.2.2.4 Time usage trouble and sources of error..................................................................29
2.3 About the tests...............................................................................................................................30
2.3.1 Design choices for system test 1 and system test 2.............................................................30
2.4 System test 1.................................................................................................................................31
2.4.1 Background of system test 1................................................................................................31
2.4.2 Trouble, system test 1..........................................................................................................32
2.4.3 Technical information, system test 1...................................................................................32
2.4.4 Iterations, system test 1........................................................................................................33
2.4.5 How did system test 1 work?...............................................................................................33
2.5 Survey 1........................................................................................................................................36
2.5.1 Background of survey 1.......................................................................................................36
2.5.2 Open questions of survey 1..................................................................................................36
2.5.3 Multiple choice questions of survey 1.................................................................................37
2.5.4 Layout of survey 1...............................................................................................................37
2.6 Interview with Heart Net Waffle...................................................................................................38
2.6.1 Background of the interview...............................................................................................38
2.6.2 Brief summary of the interview...........................................................................................39
2.6.3 Strategy used during the interview......................................................................................40
2.7 System test 2.................................................................................................................................42
2.7.1 Background of system test 2................................................................................................42
2.7.2 Trouble, system test 2..........................................................................................................43
2.7.3 Technical information, system test 2...................................................................................44
2.7.4 Iterations, system test 2........................................................................................................44
2.7.5 How did system test 2 work?...............................................................................................45
2.8 Survey 2........................................................................................................................................52
2.8.1 Background of survey 2.......................................................................................................52
2.8.2 Open questions of survey 2..................................................................................................52
2.8.3 Multiple choice questions of survey 2.................................................................................53
2.8.4 Layout of survey 2...............................................................................................................54
3 Results.................................................................................................................................................55
3.1 Qualitative data.............................................................................................................................55
3.1.1 System test 1........................................................................................................................55
3.1.1.1 System test 1, results................................................................................................56
3.1.2 Results from the interview with Heart Net Waffle..............................................................56
3.1.2.1 Toilets......................................................................................................................57
3.1.2.2 Parking space and traffic jam...................................................................................58
3.1.2.3 Restaurants...............................................................................................................58
3.1.2.4 Elevators..................................................................................................................59
3.1.2.5 Daily shopping.........................................................................................................59
3.1.2.6 Gas stations..............................................................................................................60
3.1.2.7 Chat and email.........................................................................................................60
3.1.2.8 Static information.....................................................................................................60
3.1.2.9 More accessibility issues..........................................................................................61
3.1.2.10 Weather conditions................................................................................................61
3.1.3 System test 2........................................................................................................................63
3.1.3.1 System test 2, results................................................................................................63
3.2 Quantitative data...........................................................................................................................64
3.2.1 System test 1 & system test 2..............................................................................................65
3.2.2 Possible indications.............................................................................................................66
3.2.3 Analysis of time usage concept...........................................................................................66
4 Sources of error – participation...........................................................................................................68
4.1 The communication between involved parties..............................................................................68
4.2 The tests time-span........................................................................................................................68
4.3 A low amount of test-participants.................................................................................................68
4.4 Reasons and possible measures.....................................................................................................69
4.4.1 Reasons behind “the communication between involved parties”........................................69
4.4.2 Reasons behind “the tests time-span”..................................................................................69
4.4.3 Reasons behind “the low amount of test-participants”........................................................70
4.4.3.1 Uninterested users....................................................................................................70
4.4.3.2 Sickness or unavailability........................................................................................70
4.4.3.3 Low number of users to start with...........................................................................71
4.5 General effects on the results........................................................................................................71
5 Analysis & conclusion........................................................................................................................72
5.1 Methodologies for the future system development process..........................................................72
5.2 Needs and wants............................................................................................................................73
5.3 The interface.................................................................................................................................74
5.4 Design ideas..................................................................................................................................75
5.4.1 Tree structure.......................................................................................................................75
5.4.2 Post information function....................................................................................................77
5.4.3 Descriptive information input..............................................................................................78
5.4.4 E-chat function.....................................................................................................................80
5.4.5 Subscription function...........................................................................................................81
5.5 Some final words..........................................................................................................................82
List of references......................................................................................................................................83
Appendices...............................................................................................................................................85
Appendix A — Questions used in survey 1..........................................................................................85
Appendix B — Questions used in the interview with Heart Net Waffle..............................................87
Appendix C — Questions used in survey 2..........................................................................................88
Appendix D — Student's t-test..............................................................................................................91
Appendix E — Quantitative results displayed in tables........................................................................92
Glossary
In this report some words and abbreviations are used more or less frequently. To clarify to the
unexperienced reader, below is a brief explanation to some of these words and abbreviations.
GPS – Refers to the Global Positioning System technology.
HCI – Refers to the interdisciplinary subject of Human Computer Interaction.
HTML – Referrers to the latest common accepted standard in Hyper Text Markup Language that make
up for most of the code put up on the Internet.
JSP – Refers to the so called JavaServer Pages technology.
MySQL – Refers to a specific database management system.
PHP – Refers to a server sided scripting language.
SVG – Refers to the so called Scalable Vectors Graphics technology.
System image – Refers to the two systems used in system test 1 and system test 2. By testing these two
systems, users would get a mental image of the real system (the wheelchair project) and how it behaves.
The system image would therefore in a sense change somewhat from system test 1 to system test 2.
The wheelchair project – Refers to the actual information sharing system for wheelchair users that is
being developed by staff at Shibaura Institute of Technology and Tokyo Institute of Technology, and
sometimes even the system development process that goes with it.
Time usage – Refers to the amount of time a user is willing to spend or will spend on, using a system or a
certain function within the same system.
Web hotel – this terms refers to the server and software solutions used by companies to house web pages
on the Internet.
System test 1 – The first web-based test which did let users try the first system (the original system
image) consisting of slide shows.
System test 2 – The second web-based test which did let users try the prototype (the upgraded system
image).
Survey 1 – The part of system test 1 that made up the web-pages needed to answer questions about the
original system image and the users daily life.
Survey 2 – The part of system test 2 that made up the web-pages needed to answer questions about
the upgraded system image and the users daily life.
List of figures
Figure 1.1: Wheelchair prototype 2...........................................................................................................4
Figure 1.2: Screen-shot of the interface.....................................................................................................5
Figure 1.3:“Add information”-function, start............................................................................................7
Figure 1.4:“Add information”-function, the assistant................................................................................7
Figure 1.5:“Add information”-function, adding the information...............................................................8
Figure 1.6:“Add information”-function, adding id and submitting...........................................................9
Figure 1.7:“Add information”-function, new restaurant............................................................................9
Figure 1.8:“Add information”-function, displaying information and rating options...............................10
Figure 1.9:“Delete information”-function, start.......................................................................................11
Figure 1.10:“Delete information”-function, choose a place....................................................................12
Figure 1.11:“Delete information”-function, type id-number and submit................................................13
Figure 1.12:“Delete information”-function, end......................................................................................13
Figure 1.13:“Search path”-function, check the check-box......................................................................15
Figure 1.14:“Search path”-function, choose a starting-point...................................................................16
Figure 1.15: “Search path”-function, check the radio-button and choose an end point...........................17
Figure 1.16:“Search path”-function, press the “Search path”-button......................................................17
Figure 1.17:“Search path”-function, displaying the results.....................................................................18
Figure 1.18:“Search”-function, start........................................................................................................18
Figure 1.19:“Search”-function, input window.........................................................................................19
Figure 1.20:“Search and recommend”-function, start..............................................................................20
Figure 1.21:“Search and recommend”-function, put in coordinates and submit.....................................21
Figure 1.22:“Search and recommend”-function, results..........................................................................21
Figure 1.23:“Navigation A”-function, the mini-map...............................................................................22
Figure 1.24: “Navigation B”-function, the “keyboard”-layout................................................................23
Figure 1.25:“Zoom”-function, zooming out............................................................................................24
Figure 2.1: System test 1 - flow chart......................................................................................................34
Figure 2.2: Screen shot from slide-show, system test 1...........................................................................35
Figure 2.3: Interview session with the Heart Net Waffle group..............................................................40
Figure 2.4: Start page, system tests 1 & 2................................................................................................45
Figure 2.5: Information page, system test 2.............................................................................................46
Figure 2.6: System test 2 – flow chart.....................................................................................................47
Figure 2.7: Prototype, main window........................................................................................................48
Figure 2.8: Prototype, entering and posting information.........................................................................49
Figure 2.9: Prototype, “register/unregister”- function.............................................................................50
Figure 2.10: Prototype, “search”- function..............................................................................................51
Figure 2.11: Survey 2, questions..............................................................................................................54
Figure 5.1: Tree-structure, example 1......................................................................................................76
Figure 5.2: Tree-structure, example 2......................................................................................................76
Figure 5.3: “Information posting”-function, example 1...........................................................................77
Figure 5.4: “Information posting”-function, example 2...........................................................................78
Figure 5.5: Static information, example 1................................................................................................79
Figure 5.6: E-chat function, example 1....................................................................................................80
Figure 5.7: E-chat function, example 2....................................................................................................81
Figure 1: Student's t-test, unequal sample sizes.......................................................................................91
Figure 2: Standard deviation....................................................................................................................91
Figure 3: Definitions................................................................................................................................91
List of tables
Table 1: Demographic questions. Results................................................................................................92
Table 2: Interface's usability question. Results........................................................................................92
Table 3: Time usage questions. Results...................................................................................................93
Project background
1 Project background
This first chapter will describe the background of the project as well as shed some light on the
research topic and why it is interesting.
1.1 System history
In Japan, an information-sharing system aimed to assist disabled people and then mainly
wheelchair users is under development. This information-sharing system is the combined effort of
professors and various students from different laboratories at Shibaura Institute of Technology,
Japan and Tokyo Institute of Technology, Japan. In particular prof. Akira Aiba, Department of
Electronic Information Systems, College of Systems Engineering, Shibaura Institute of
Technology, prof. Hitoshi Kuwata, Department of Architecture & Environment Systems, College
of System Engineering, Shibaura Institute of Technology, prof. Masao Miyagi, Department of
Machinery & Control Systems, College of System Engineering, Shibaura Institute of Technology
and prof. Katsumi Nitta from Interdisciplinary Graduate School of Science and Engineering,
Tokyo Institute of Technology have been in charge of the development process.
The system has been under development since 2002. The goal of the system up until this project
started has mainly been to provide wheelchair users with information regarding different locations
and the static accessibility to those locations. The phrase “static accessibility” could in this case
mean specific information about the dimensions of a ramp or the dimensions of a sidewalk. This
means that a user would be able to get information about an area on the map and then be able to
judge how user friendly and accessible that area would be for the user to visit. The idea was to
create this functionality based on a clickable map in the user interface. So when the users would
navigate around the map they could get information by clicking on different places on the map.
To provide accurate maps that would be able to provide such information, tests and measurement
operations did commence. But since sites change over time due to unforeseen actions so too does
the accessibility to those locations. If information could be shared amongst users, knowledge about
changes to locations could be spread between users. So it came to be that the system that is being
developed, lets call it the wheelchair project, in essence would be built on a server based
technology, meaning that all users would act as clients to a server based solution. So all users
would connect to a server through their client software and then they would receive information
about changes to different locations from this server. As time went by the need for users
interacting with the server and other users became imminent. The idea of users posting
information on the system for other users to view, was approved and development of new different
functionality began. This new emerging functionality in the system as well as the system as a
whole is still under development though.
Two different physical models or prototypes of the system has been made. Besides a wheelchair,
1
Project background
the two different prototypes hold a small computer, a GPS and an indoor positioning system. The
first prototype did hold a large GPS and a fairly large laptop. The second and the latter developed
prototype, included smaller devices in general but used the same technology. The decision to
incorporate smaller devices into the second model could have been derived from the results of a
test. On the 11th of December 2004 one major test involving the GPS system was done in Kita
Urawa, Saitama prefecture, Japan. One of the things that could be concluded from this test was
that the wheelchair could be considered a hindrance to other people once out in the traffic, due to
the fairly large laptop platform attached to the front-side of the wheelchair.
Furthermore in April 2007 a new project continuing on the wheelchair project was scheduled to
begin. During this project, new functions and more functionality was to be incorporated. Since the
actual system has yet not had any testing towards real end-users and new functionality is about to
be incorporated into the project, my idea was to help this development process by testing the
project towards real end-users and try to find out how well the functions within the system
corresponds to those users' needs, and also try to understand what real end-users might want to see
implemented into the system.
1.2 Goal
The goal of this project was to find ideas for how the continued system development would satisfy
the end-users' needs (wheelchair users), and then mainly in terms of what functionality the system
has to offer these users. By trying to find the functional shortcomings this system has from a user
perspective and by considering the users' thoughts on how to improve the system, ideas for the
future system development would be presented. All of this would be done in a user-centered
process. The results of this endeavor are put forth in this report.
As will be discussed further down in the report, there was also a subgoal concerning the statistical
comparison between two web-based tests. This subgoal was setup to help reach the goal
mentioned above.
1.3 Why interesting?
There are a lot of disabled people around the world. According to WHO there are about 750
million people around the world1, which counts for approximately 11% of the world's population2,
that have disabilities in various forms. Also not many systems similar to this system have been
done before. In USA something similar is under construction, though the focus of that project is
more to cover areas indoors (Bhatia et al. 2006). Furthermore, as mentioned above, this system has
not yet been tested against groups of people that accurately resemble the end-users. Since user
testing is an important corner stone when designing systems with a high amount of usability
1Youthink-but do you know: Available at: http://youthink.worldbank.org/issues/disabilities/, [cited 2007-11-28]
2Worldometers -real time world statistics. Available at:http://www.worldometers.info/, [cited 2007-11-28]
2
Project background
according to the subject of HCI, doing user tests could prove to improve the system by providing
input on how to continue with the system development process. See for example Preece et al.
(2002) or Göransson and Gulliksen (2000) for further reference.
1.4 Users and installations
Users for the tests in this project were aimed to be disabled people, and then mainly wheelchair
users with some prior knowledge of using the Internet. This since a user that has never before
experienced a computer would probably start by first try to use a computer instead of letting her or
his decision making depend on a computerized system. If information had to be retrieved to be
able to make correct choices, an inexperienced user would probably contact an assistant before
trying to use this system. Also since the system itself does not provide movement support of any
kind and only provide information to the users, the test participants representing the end-users
were not meant to be, at least not at this time in the project, a paraplegic 3 or quadriplegic4 group of
users, or users with similar disabilities who did not have the possibility to get help from an
assistant.
So far the system requires that the users are able to give input by using a touch screen and a
keyboard, but since there are a lot of different users with a lot of different individual needs, the
hardware and platform used might be a subject to change as the system development process
continues.
In total there were two different groups of users that came to represent the end-user group during
the different system tests.
Thanks to professor Hitoshi Kuwata, Shibaura Institute of Technology the first group of people
was found and contacted. This first group of users was found through the organization Heart Net
Waffle5.
The second group of people were found and contacted through the help of professor Katsumi
Nitta, Tokyo Institute of Technology. This group of participants did all belong to the Aoba Barrer
Free 21 organization6.
3 Paraplegia. Available at: http://en.wikipedia.org/wiki/Paraplegia, [cited 2008-09-20].
4 Quadriplegia. Available at: http://en.wikipedia.org/wiki/Quadriplegia, [cited 2008-09-20].
5Heart Net Waffle homepage. Available at: http://www.h7.dion.ne.jp/~waffle/, [cited 2007-01-22].
6Aoba Barrer Free Support 21. Available at: http://www.abs21.com/, [cited 2007-04-09].
3
Project background
1.5 The system and how it is constructed
This section will describe a bit of the system by looking at the hardware and software aspects of
the system as well as parts of the interface design and the functionality of the system.
1.5.1 Hardware & Software
The latest system prototype does consist of a hand held computer constructed onto a wheelchair
(see Figure 1.1). This wheelchair has also been equipped with a GPS as well as an indoor
positioning system (as mentioned above).
If the users are outside and move around, information about the users' location, as well as the
users' direction is then transmitted via the GPS-system to a central server which in turn will let the
Figure 1.1: Wheelchair prototype 2
users know where they are. If the wheelchair resides indoors the idea is that an indoor positioning
system will be able to calculate and provide information to the system and thus enabling the
system to know the current position of the users.
4
Project background
JSP and HTML together with SVG makes up the programming language and the technology used
in the interface. During the period of the tests, the web server of the wheelchair project was an
Apache Tomcat 5.5.17 server and an SQL server was used as a database server.
1.5.2 Interface & Functionality
By opening the web browser in the hand-held computer on the wheelchair, the users were taken to
the user interface of the wheelchair project. This interface was displayed as a two dimensional map
of the user surroundings with a menu-bar on the right side of the screen in which a total of ten
different buttons and icons resided representing the different functions within the system (see
Figure 1.2). Also a login button was presented hovering in a top-bar, above the menu-bar. Finally,
there was a green arrow on the map that did represent the user and his or her position on the map.
Figure 1.2: Screen-shot of the interface
The functions could be divided into two different categories. One category being altering
information and the other one viewing information. In the altering information category there were
three different functions that could be performed. Here the functions were either connected to
adding information to the system, or deleting information from the system. In the other category,
the viewing information category, there were six different functions that in some sense had been
5
Project background
implemented into the system. Finally it is worth mentioning the assistant. The assistant being a
small “Merlin” (sorcerer) would comment and express boundaries for the users when needed. If
for example some actions had to be taken in a specific order to finish a task, Merlin would point
this out to the users by making a chat bubble with relevant text in it.
Also, what needs to be mentioned here is that the system as a whole, still was an Alpha-version
during the time for the tests. This means that the system was limited in its functionality and did not
yet work as one could imagine a finished system would do. Also the whole system was coded and
documented in Japanese, and this called for a translation of the different functions from Japanese
to English. These translations may not have been totally accurate and with the correct meaning
preserved from the Japanese language since this translation was done orally with the help of
comrades from the laboratory during a short period of time.
1.5.2.1 Altering information section
In the altering information section, there were as mentioned three functions:
●
●
●
Add information
Delete information
Change point of origin
The “Add information”-function would let a user add information to a specific place on the map.
For example a user could add information about a new restaurant to a spot on the map. Also worth
mentioning is that this information could be rated by other users. How was this done then? Let's
say a user wants to add a restaurant to the map, he or she would then firstly click the “Add
information”-button which is circled and numbered in Figure 1.3. Upon doing this, the assistant
Merlin would ask the users to choose were on the map they wanted to add this restaurant-spot as
seen in Figure 1.4.
6
Project background
Figure 1.3:“Add information”-function, start
Figure 1.4:“Add information”-function, the assistant
7
Project background
Figure 1.5:“Add information”-function, adding the information
After clicking on a spot on the map, the user would be presented with a new window in where
information about the new location could be posted. (see Figure 1.5). And after adding relevant
information including a location id-number of choice, by submitting the information (see Figure
1.6), the new restaurant would show up on the map as seen in Figure 1.7. If other users would like
to rate the information supplied, they could do so by clicking on a spot on the map. Clicking on the
spot would let them access that spot's information and there they could choose to agree, partly
agree or disagree with an assumption that proclaimed the information on the page as being good
and qualitative (see Figure 1.8).
8
Project background
Figure 1.6:“Add information”-function, adding id and submitting
Figure 1.7:“Add information”-function, new restaurant
9
Project background
Figure 1.8:“Add information”-function, displaying information and rating options
10
Project background
The “Delete information”-function would let the user do the opposite compared to the “Add
information”-function. If the restaurant just recently went out of business or closed for some other
reason, the user that posted the restaurant could delete the restaurant from the map. This would be
done by firstly clicking on the “Delete information”-button as shown in Figure 1.9. Then Merlin
would ask the user to choose an information spot to delete (see Figure 1.10). As shown in Figure
1.11. After clicking on the spot of choice, a new window would emerge with information about the
location and a text-field in where the id-number to the uploaded spot should be typed if the
information where to be removed. After submitting the id-number the information spot would be
removed from the map as displayed in Figure 1.12.
Figure 1.9:“Delete information”-function, start
11
Project background
Figure 1.10:“Delete information”-function, choose a place
12
Project background
Figure 1.11:“Delete information”-function, type id-number and submit
Figure 1.12:“Delete information”-function, end
13
Project background
Finally there was the function that lets users change their point of origin. When the system started,
all users would have a point of origin, and since different users would most likely have different
points of origin, this function would let the user specify his or her starting point. All of the
functions described above were tested in system test 1, except the function that let users change
their point of origin (“Change point of origin”). This since to much of the function had yet to be
implemented into the system, making it hard to properly simulate the function with a slide show.
14
Project background
1.5.2.2 Viewing information section
In this section there were three different search functions intended to let the users find the
information he or she would be searching for and three functions that intended to the let the user
navigate smoothly throughout the map. The three search functions were:
●
●
●
Search path
Search
Search and recommend
and the three functions used for navigating we call:
●
●
●
Navigation A
Navigation B
Zoom
The “Search path”-function would let the users decide a starting point and a finishing point on the
map and then gave the users an estimate on the distance and which route to take. During the
system test and translation process, I did not get this function to work properly since it’s not yet
fully implemented into the system, nevertheless this function and what it represents was tested in
System test 1. By firstly clicking the check-box (see Figure 1.13), the user could choose a starting
point on the map as displayed in Figure 1.14. After choosing a starting point the user could then
choose an end point by shifting the highlighted radio button as shown in Figure 1.15.
Figure 1.13:“Search path”-function, check the check-box
15
Project background
After pressing the “Search path”-button (see Figure 1.16), the results for the search would show up
in a new pop-up window as displayed in Figure 1.17.
Figure 1.14:“Search path”-function, choose a starting-point
16
Project background
Figure 1.15: “Search path”-function, check the radio-button and choose an end point
Figure 1.16:“Search path”-function, press the “Search path”-button
17
Project background
The users could search for places using the “Search”-function which would let users search for a
Figure 1.17:“Search path”-function, displaying the results
Figure 1.18:“Search”-function, start
18
Project background
Finally the system did hold a search function I named, the “Search and recommend”-function. This
function would let the users search for a specific type of place close to where they are located at.
The idea is that the user chooses a topic (see Figure 1.20), type his or her coordinates in the right
area, and press the “Search and recommend”-button as seen in Figure 1.21. This will generate a
list of possible results which will emerge in a new pop-up window as seen in Figure 1.22.
Figure 1.19:“Search”-function, input window
19
Project background
Figure 1.20:“Search and recommend”-function, start
20
Project background
Figure 1.21:“Search and recommend”-function, put in coordinates and submit
Figure 1.22:“Search and recommend”-function, results
21
Project background
As mentioned above there were three functions meant to ease the users navigation trough the
system.
The first two functions, “Navigation A” and “Navigation B”, were pure navigation functions. The
“Navigation A”-function would let the user browse the map by using a click-able mini-map in the
menu area. If the user clicked on an area in the mini-map the user would be taken to the
corresponding area on the map (see Figure 1.23).
Figure 1.23:“Navigation A”-function, the mini-map
22
Project background
“Navigation B” would help the user to browse the map by using a navigation system consisting of
click-able arrows allocated at the side-bar to the right of the map, similar to the layout of the arrow
button-group on a keyboard (see Figure 1.24).
Figure 1.24: “Navigation B”-function, the “keyboard”-layout
23
Project background
Finally there is a “Zoom”-function (see Figure 1.25). This function will let the user zoom in or out,
making it easier for the user to localize him or herself on the map. These three functions were
tested in system test 1.
Figure 1.25:“Zoom”-function, zooming out
24
Methodology
2 Methodology
This chapter will first discuss a bit of the background of the methodology used in this project and
then discuss the types of data that were collected to achieve the goal set up earlier in this report.
After this, the chapter will shift focus and go in more thoroughly in to the two different tests and
the interview that were used to gather data. Also since it was deemed important by the professors
and me to describe how the data was collected this will be done in two sections describing the
surveys.
2.1 Background of Methodology
Since the idea was to understand how to make the future system development process and in the
long run, the future system better correlated to the needs and desires of the users, this information
needed to be deduced from the users. Also since it was decided early on in the process that more
than one test were to be conducted, it was interesting to know if the information gathered from the
users were interpreted correctly between the tests. In other words, it was informative to see if users
found the system image more appealing after the ideas from a previous user test had been
incorporated into the (new) system image. So, when trying to achieve the main goal which was:
●
Gather information on how to improve the system by correlating the functions within the
system to the users needs
I also tried to compare the results from the two different tests. The idea was that this subgoal of
comparing how users viewed the two different system images (the original and the updated system
image), would give feedback on the methodology we used for the comparison of the tests as well
as, and more importantly, provide feedback on the idea of iterative collaboration with users. Since
the project had used a different approach involving less user-contact up until this master's project
started it was interesting to see if this comparison between two system images of the “same”
system, one with user input and thus iterative collaboration with users and one without, would
show a difference between how users perceived this “one” system.
It was decided that the data that were to be collected from the tests were to be of both quantitative
and qualitative nature. The qualitative data, the most important data, would provide information on
what needs, desires or ideas the users experienced when faced with a system image and the
functions represented within that system image. So in a sense the qualitative data would help us
reach the goal of this project. The quantitative data on the other hand, would instead make it
possible to compare the system images in investigating how the users’ attitude towards these
different system images varied between the tests, and thus fulfill the subgoal.
The idea was firstly to go with a total of two different tests. By conducting two web based tests,
where the second test would include ideas from the first test, qualitative data could be collected
and then in some sense checked in the second test by comparing the quantitative data from both
25
Methodology
tests. But due to somewhat insignificant results from the first web-based test, an interview was
also conducted to provide extra qualitative results that could be used to take forth ideas on how to
improve the wheelchair project so that it better correlates to the needs and wishes of future users.
2.2 About the data
As mentioned, this project did focus on collecting two types of data. Qualitative and quantitative
data.
2.2.1 Qualitative data
The retrieved qualitative data were either collected using open questions from the two surveys
used in the two different web-based tests, or by using questions and follow-up questions during the
interview session that was conducted with the Heart Net Waffle group in the end of February
2007.
2.2.1.1 Qualitative data concept
The idea with qualitative data as described in chapter 1, Bell (2000), is to get a deeper
understanding and insight in how things are from a user perspective or how users perceive their
world. In our case, this partly means how the users perceive the actual system, or more accurately
the system images presented for the users in the two tests (system test 1 and system test 2). Also it
means that by finding out users needs and wants we would better understand what they would like
to see incorporated into the future system.
One important aspect of qualitative data is that it is mostly based on users own experience.
Depending on what people has experienced in life, those experiences will usually affect how the
users look upon a situation or how a user reflects on an issue. This is also the case with the results
gathered from this research.
2.2.1.2 Why open questions?
A good way to gather qualitative data is to use open questions. Differing from multiple choice
questions in the fact that the users have more freedom to answer what they feel like, open
questions does also lessen the risk of prejudice effecting the results. By this I mean that controlled
answers must hold alternatives set up beforehand, and it is possible and probable that these
alternatives do not match what the users would like to express. Since the goal of gathering
qualitative data is, as described above, to get a deeper understanding and insight in how things are
from a user perspective or how users perceive their world, it is crucial that the way users express
26
Methodology
things about their world is not affected by the prejudices and thoughts of the test-authors. The
users should also have the possibility to actually describe their world, which is easier to do by
using open questions.
2.2.1.3 Open questions setup
Following the concept presented above in section 2.2.1.2, all of the open questions were designed
to provide the users with freedom when answering them. Which in this context meant that an
empty text-area or a listening (hopefully) unprejudiced ear was presented to the user with each
open question. During the design process of the open questions some of the things that were taken
into consideration were: Asking only one question at a time, not putting strain on the memory of
the users by asking them to recall something that happened a long time ago, not using leading or
offending questions or letting my valuation of a subject affect the design of the questions (Bell
2000, chap 8.). See appendices A, B and C to view the open questions used in this investigation.
2.2.1.4 Open questions trouble and sources of error
The main issues and challenges open questions have are in my opinion questions about validity
and reliability. Validity is a measure of how well a question is measuring or describing the things
we want it to measure, whilst reliability is a measure of to which extent an instrument or method
provides the same results at different times during otherwise similar circumstances (Bell 2000).
Therefore, how do we know that we are measuring the things we want to measure and how do we
know that the results are reliable (which in this case could mean, can these results be repeatable
during similar circumstances)?
Beginning in the wrong end, one major issue for the reliability of the results is: Do all users
interpret the questions similarly, because if not, it is likely that the results will be different each
time and thus not comparable. To lessen the risk of bias coming from misinterpretation, the open
questions did go through a number of iterations in where the communication with professors and
other people involved in the project was central. As will be mentioned in the sections about the
surveys, some actions deduced from this communication process included putting in better
explanatory texts as well as displaying options for the users to minimize the risk of bias in the
results. In the interview on the other hand follow up questions were used to minimize the risk of
bias coming from misinterpretation.
To lessen the doubts about validity in the research, the questions did go through a number of
iterations in where communication with people connected to the project was central, as mentioned
earlier. Furthermore interviewing and conducting tests only with disabled people, the target group
of this research, adds more validity to the research.
However, it should be mentioned, that even though precautions and measures were taken to lessen
the risk of bias in the results it is still possible that bias coming from misinterpretation did affect
the results to some extent. And since validity is correlated to the amount of reliability in the sense
that if a question isn't reliable it can't be valid, see Bell (2000), it should be considered that even
the validity in the research might be limited. As a final note, open questions do provide freedom to
27
Methodology
the users, but they also provide bias to the results since it may be hard to interpret the data
correctly. Since all users are different, so might the person be that will try to interpret the results.
Even though this error could be regarded as being static, since I interpreted all of the results, it still
exists.
2.2.2 Quantitative data
Discussing the quantitative data, and how to fulfill the sub-goal, there is the issue about
comparison. To be able to compare the quantitative data and still be on the track of improving the
system from a user-point of view, some models had to be used. In this project the statistical model
used for the comparison between the sample sets was the so called student's t-test7 whilst the model
used for gathering the data was something I call time usage.
2.2.2.1 Time usage concept
The idea for time usage is; by asking the users the amount of time they consider they want to
spend on using the system or the functions within the system, one could get a better understanding
of how well correlated the functions within the system and the system as a whole are to the needs
of the users. This is based out of the assumption that if a user wants to spend a lot of time with a
particular function, then that function is probably more correlated to the needs of that wheelchair
user than a function that a user do not want to spend a lot of time with.
2.2.2.2 Why time usage?
There are other models that will register and analyze the patterns users undergo when using
functions and clicking their way through a system, see for example Srivastava et al. (2000), but in
this project it was considered better to ask direct questions, rather than to analyze a clicking
behavior. Also, when analyzing a pattern it can be hard to distinguish which factors contribute to
the behavior. For instance, it can be hard to tell if the results mostly depend on the fact that the
user interface is poorly designed or if the function, is not interesting to the user.
There was also a time limit to the project and therefore the amount of work had to be considered.
By using a question-based method to deduce quantitative results, less code and administration had
to be done compared to if a remote pattern analysis method was chosen. Since the primary goal of
the project was to collect qualitative data, it was thought of time efficient to use a similar approach
when collecting the quantitative data that were to be used for comparison (and possibly to give
indications). So it came to be that the concept of time usage was invented, and a questionnaire that
held questions of both qualitative and quantitative nature was used.
2.2.2.3 Time usage questions setup
The time usage multiple choice questions each had five different alternatives answering to a fixed
7Student's t-test. Available at: http://en.wikipedia.org/wiki/Student's_t-test, [cited 2007-03-19].
28
Methodology
amount of time from which the users could choose from. By using alternatives with fixed amounts,
“Once a week”, “Every day” etc, it was easier to avoid errors based on differences between the
users. To exemplify, the sentence: “I would use the system much.” would most likely mean
different things for different users, in comparison to a sentence or an option involving a fixed
value as “Every day” or such.
For the five different alternatives: “A number of times every day”, “Every day”, “Three or four
times a week”, “Once a week” and “Once a month or less”, a five grade scale ranging from five to
one, was used enabling comparison with quantified values. The more time a user wanted to use a
system, the higher the number in the scale she got. See Appendix A to view the time usage
questions used in this project.
2.2.2.4 Time usage trouble and sources of error
Even though the amount of time a user would like to use a system (the amount of time usage),
could depend on several different things, such as the usability of the user interface and the fact that
users differ in aptitude and preferences, the decision was made to go with this method. The idea
was that with the usage of good questions, the users could to some extent be guided to distinguish
between not using a function because of its utility and not using a function because of its poor
layout design. In the tests, two questions that involved time usage were used. The second question
was meant to act as a check-question to the first question. This to try to minimize the errors of
users misinterpreting the questions and thus the concept of time usage. Also, since the usability of
a systems user-interface or design may affect how much time a user would be willing to spend on
using a system (as mentioned above), there was a question in each one of the tests, that did ask the
users for their opinion on the easiness of the system images presented for them. The idea was that
this would help to discover bias in the results.
However, it can not be said with certainty that users did not misinterpret the time-usage questions
or the question meant to check the easiness of the system. The questions had been rewritten
several times to lessen the risk of misinterpretation, but it cannot be neglected that users and
people are different and thus interpret differently.
Time usage is trying to convey qualitative subjective values into quantified measurable and
comparable values. Since this conversion is done by the users, it forces them to try to consider all
the qualitative aspects of a situation when estimating how much time they would be willing to
spend on task, and this could be hard.
The concept of time usage is based out of the assumption that how well a system answer to a user's
needs do stand in direct correlation to the amount of time this user would like to use the system,
and it is possible that this assumption, and thus the concept of time usage is not applicable for all
users since different users might use computers and systems differently depending on their specific
handicap or situation.
When possible it might be better to actually let users try a real working system and observe how
much time they would spend on using the system and the different functions within the system.
29
Methodology
But since letting users try a real working system or observing them directly were not eligible
options during this project, letting users do estimations through time usage became a possible
solution to the problem.
More criticism that could be given to the concept of time usage concern the subjective aspects.
One major issue is for example that the time usage concept implores users to do an estimation on
how much they think they would use a system, and it is not actually measuring how much a user
is, or will use a system. An estimation is likely to differ from the real case scenario.
Time usage as a concept might still be interesting though since it does make it possible to judge
approximately how much time a user would use a system and compare between users. Even if
people would be asked in a more qualitative way letting users explain more of their reasoning to
why they think they would use a system a certain amount of time, there would be the issue about
the interpretation and the comparison of the data.
2.3 About the tests
As mentioned earlier there were in total two web-based tests and one interview conducted. The
two web-based tests were each connected to a survey, and the questions used in the interview were
deduced from questions (that were to be) used in survey 2.
The first test, system test 1, did consist of showing modified screen-shots from the actual system to
the users. By using slide shows, a type of story-board was used to get the users to understand how
the original system and its functions worked. See chapter 3 in Göransson and Gulliksen (2000) for
further reference. In between the two tests, the interview was conducted with the same test-group
that did system test 1. The interview was done mainly to add more qualitative data to the analysis
and conclusion sections. The second test, system test 2, did consist of exposing the users to a
prototype. This prototype was mainly based on theories from another project as will be discussed
later in this report.
After discussing some design choices for system test 1 and 2, I will describe the methodology and
layout of the tests, their corresponding surveys and the interview in the chronological order they
were conducted and presented (see above).
2.3.1 Design choices for system test 1 and system test 2
My choice for going with a remote usability evaluation technique, and in fact an asynchronous
remote questionnaire technique does come from the idea that remote usability evaluations with
disabled people could be just as efficient as synchronous non-remote techniques when it comes to
comparing quantitative data, which was needed to make a comparison between system test 1 and
system test 2. For further reference, see the conclusion from the two tests put forth in Petri et al.
(2006). Even though its also argued that the richness of the qualitative data could have been higher
30
Methodology
if a non-remote technique would have been used, this way of action was deemed futile. This since
a non-remote technique would, at the time, have been too time-consuming and money-consuming
in terms of hiring an interpreter, finding suitable test-persons and finding time when both testpersons, interpreter and the author would have time to meet and perform the tests.
By letting the tests be a remotely asynchronous usability test, no press on how much critique the
users are able to express would be present. In other words there were no by-standing, interfering
system developers by the users side that would like to hear “good things” about their system. This
could however have occurred if an interpreter and interviewers would have been present on the
test site together with the users (Petri et al. 2006). furthermore it is argued in the same article that
disabled people often have special personalized equipment at home which will enable them to
perform the tests with less restraints compared to users shuttled to usability laboratories for
different usability testing. By making the two tests asynchronous, the idea was to get more control
over my time and not be dependent of other people. Finally, I considered it to be a good choice to
have equal or similar ways of testing the two system images presented in system test 1 and system
test 2 since it would then be easier to compare the results from the tests. Therefore the method of
collecting data did not change considerably as the project continued.
2.4 System test 1
This section will describe system test 1, by firstly go into a bit of the background of the test and
then talk a bit about the trouble that arose during its creation process. Some technical information
regarding the test will then be presented. After this the section will shift focus and discuss how the
test was created and what happened when the users did take the test.
2.4.1 Background of system test 1
This test was done on the world wide web. The test was constructed to be an asynchronous remote
usability test of the system image presented. The test itself did not test the actual system, since the
system is still under development and thus not still function properly as mentioned earlier in this
report, but instead the test showed a system simulation of the system and how it is supposed to
work when run at this stage at the development. If the actual system would have been tested by the
asynchronous remote usability testing method, instead of as now a prototype of the actual system,
it was believed that it would be hard for the users to get an understanding on how the functions
within the system and thus the system as a whole worked.
The test and the site supporting the actual test went through a series of usability tests, thus
primarily using students from the same laboratory with a similar background. It is argued in Krug
(2006, chap. 9), that even though people are different, people tend to act similar when presented
with a specific situation if they have some previous knowledge on how to surf the web that is.
These usability test were done to provide the actual test with better usability, making the system
31
Methodology
test 1 site easy to browse, allowing the users of the test to concentrate their energy on performing
the actual test and evaluating the simulated system. The idea was to save the goodwill of the users
for the actual system test and not waste it on surfing the site as described in Krug (2006, chap. 10).
There were nine functions that were simulated and tested. This since those nine functions were the
functions that so far had been implemented into the system and in some sense worked properly.
See section 1.5.2 for further reference. Each one of the nine functions were simulated with a slide
show that did show how the function behaved when used. Also to minimize the work load on the
users, only one scenario for each function were presented in the slide shows. This since according
to Krug (2006), users on the Internet only have a certain amount of energy to spend on task when
presented with one.
The test was originally written in English and then translated to Japanese by a firm called Nai 8. By
letting this firm translate the test and the survey, the text supplied to the Japanese wheelchair users
gained more quality. Furthermore by using the help of a professor, who had previously worked as
a survey-creator for a Japanese company, some cultural differences with the difference in the
semantic structure of the language, would vanish. The idea was that the professor would minimize
the cultural difference issue by controlling and in some cases rephrasing the questions before the
questions was sent to be translated.
2.4.2 Trouble, system test 1
I faced some problems when creating system test 1. Mostly there were technical problems ranging
from login-requirements, reliability with web-hotels and Asian character handling. During the
development process technical problems like these were solved. To ensure that all users could
access the test and to improve the test’s general accessibility, extra code was added to ensure that
old, new and different browsers could access the test. As discussed in chapter 4.2, it took a long
time before results were gathered from system test 1, and this did surely affect the continuation of
the project.
2.4.3 Technical information, system test 1
System test 1 was a web based application that could be found on the word wide web at the
location: http://www.kurumaisuproject.net during the period 2006-12-18 until 2007-01-27. In total
System test 1 came to be approximately 165 megabytes in size, including 137 images and 134
HTML and PHP files. The survey which was presented at the end of the test held 24 multiple
choice questions and 6 open questions.
8Nai. Available at: http://www.nai.co.jp, [cited 2007-04-11].
32
Methodology
2.4.4 Iterations, system test 1
The test went through a total series of four iterated versions during the system test 1 development
process.
When the first version were to be implemented there was yet no translation support, and it wasn't
known that the project would be giving a funding from the Japanese government. So then the plan
was to focus on mostly multiple choice questions, since then less translation from Japanese to
English would be necessary in general.
During the second iteration there was still no knowledge about the funding from the Japanese
government, but now JavaScript was embedded into the system which opened up for the
manipulation of how user surfed their way through the test. For example, now the users only
needed to put in their password once if compared to the previous version. However this
manipulation created usability problems which were discovered in the end of the second iteration
thanks to usability testing.
In the third iteration, the possibility of funding and translation support emerged. Here the
questionnaire changed to include more open questions and less multiple choice questions. This
choice was deduced from discussions, and the fact that translation support now was possible,
making it easier to get qualitative answers from users. Also an obligatory tutorial and a better
navigation system were added to the test to make the test understandable since the test itself didn’t
include a working prototype.
In the fourth iteration continuing usability testing led the way to further improve the tutorial and
the navigation system with less obligatory text, and more navigation options. Also the layout of
the survey was changed to minimize scrolling and unnecessary mouse movements.
2.4.5 How did system test 1 work?
Firstly the test-group were contacted through their organization Heart Net Waffle group. A
password and link to the system test site was supplied to the users and an invitation to do the test.
When the users entered the system site, the users were presented with a welcome screen and one
link to the actual system test. When the link was clicked upon, a new window would pop-up in
where the system test emerged. This system test window was generated through JavaScript to give
control over how the user navigated through the site. Since there weren't any back-buttons that
could be used in the pop-up window, custom made surf-buttons were made to make up for this
restraint. The idea of restraining users from hitting the back-button is not argued to be a good idea
in general web usability practices (Krug 2006), but by using practices from the same book, I tried
to make the user always have the possibility to navigate forward and backwards in the site. Also if
the users’ browsers didn't support JavaScript, another regular browser window would pop up
instead. As mentioned earlier the idea was to make the test accessible for all users regardless of
platform used.
33
Methodology
When presented with the system test window, the users needed to present a password to enter the
test. This feature did in somewhat provide a guarantee that the users did belong to the actual test
group.
Once the users entered the system test, they were presented with test information. This information
told the users important information about the test in a compressed form (see Figure 2.1). The
users also had the possibility to read more about the test through different links that were presented
Figure 2.1: System test 1 - flow chart
on this information-page. After reading the information about the test, the users could agree to
continue doing the test or quit the test if the users found the test methodology inappropriate, how
the gathered test-information would be used or something else with the test not satisfactory.
If the users agreed to continue with the test they would be taken to a short tutorial in where it was
explained how system test 1 was constructed. This was done by showing some pictures from the
slide shows as well as providing explanatory text to the pictures. When the users had finished the
tutorial they would be presented with the possibility to start the test. Also during the tutorial, users
would have the possibility to go back to former pages by pressing a button labeled “last page”.
This button's function was also explained in the tutorial.
34
Methodology
Figure 2.2: Screen shot from slide-show, system test 1
After starting the test, the users were taken to the main testing page, in where nine different
buttons were placed. Each one of these buttons were linked to a function simulation. These
function simulations where each made up of a slide show that expressed how the function would
behave on the actual system. The pictures in these slide shows where all taken from the actual
system and modified somewhat. The modifications to the pictures where mere highlights
consisting of red circles showing the actions taken in the slide (see Figure 2.2), or red arrows
pointing out important information shown in the slide.
It was taken into consideration that by highlighting certain elements in the screen shots, there was
a possibility that the system image itself was made easier to understand than the actual system. But
since the main goal of the research was to find out which functions within the system that should
be added or further developed, it was decided that a clear overview of the system and its functions
would not present a major problem. Instead it might help the users to better judge whether or not
the function is interesting or not, since focus could be put on judging the actual function instead of
how the function was implemented. I know that these two, the users interest in the function and
how the function is implemented, are correlated. Still this decision was made because the test did
not involve a moving working system, but instead only screen shots from a working system. When
the user had viewed and tried all of the different functions in the main page, she could click her
way to the survey. After answering the questions and pressing the button labeled “submit survey”
35
Methodology
the users where presented with a screen thanking them for their cooperation and patience.
2.5 Survey 1
This section will describe survey 1 by firstly focusing on the background of survey 1 and then look
a little on the idea of the open questions and the multiple choice questions that were present in the
survey. Finally there will be information about the layout of the survey.
2.5.1 Background of survey 1
To be correlated to the project goals, the goal of the survey was to catch and quantify the feelings
the participants of system test 1 had about the wheelchair project, and also to try to catch the users
wishes and needs for the future development of this project. To do this it was concluded that the
survey needed two different parts. One qualitative part, consisting of open questions, that would
try to find out more of the users needs and ideas for the iteration of the original system image (the
slide shows in system test 1) so that the upgraded system image (the prototype in system test 2)
would hold ideas better correlated to the ideas of the users. And one quantitative part, consisting of
multiple choice questions, that would try to quantify the users opinion about the original system
image through the concept of time usage, and thus make it easier to test the data against the
upgraded system image. For further reference on quantitative and qualitative data see for example
Bell (2000, chap. 1).
2.5.2 Open questions of survey 1
This part of the survey was meant to be the major part of the formative evaluation of the system. It
is hard to validate that the open questions presented in a survey will retrieve the information that is
wanted as discussed by Bell (2000, chap. 8).
One major problem with open questions (and multiple choice questions), as mentioned earlier, is
the possibility that the users misinterprets the questions, since all users have different
backgrounds. To solve this problem and minimizing the possibility of the users interpreting totally
different meanings, the questions did go through a number of iterations in where the focus was,
firstly trying to validate that the questions asked were most likely to be interpreted in a similar way
by all users, and secondly that the answers provided from these questions did provide information
on how to improve the system from a user point of view. During these iterations a number of
persons, ranging from professors to master students were involved with providing feedback and
suggestions on how to improve these open questions.
36
Methodology
2.5.3 Multiple choice questions of survey 1
The quantitative data was in a sense supposed to act for the summative part of the evaluation. So
in some ways the results would provide data for the development of system test 2 (in providing
indications), but mainly the test would provide data for comparison between system test 1 and
system test 29. Except for two questions, a five grad scale was used on all of the multiple choice
questions. Ranging from one to five, different alternatives were presented to the users.
Besides trying to get users' time usage rating for the system image in general, the idea was also to
test the nine different functions individually. This to see if the best rated functions from system
test 1 could be implemented in the prototype, which hopefully would improve the system and also
make it possible to, in a quantified way, compare the functions in the updated system image (the
prototype) with the functions from the original system image (the slide shows). Therefore the two
time usage questions were also transformed to work with the nine different functions presented in
the original system image. In total there were 18 time usage questions concerning the different
functions and two time usage questions concerning the overall system image presented for the test
subjects.
Besides the question regarding the usability of the user interface, there were also some multiple
choice questions concerning the demographic properties of users such as age and gender etc.
2.5.4 Layout of survey 1
The total amount of questions came to be 30 questions, including 24 multiple choice questions and
6 open questions. These questions were then divided into three parts; questions about the users,
questions about the system and questions about the functions within the system.
The questions were organized in a way to ease the task for the users by maintaining the users
goodwill as described in Krug (2006). This by having easily answered questions at the start, and
using a good mix of question throughout the survey. Finally, to minimize the scrolling work
required from the users, the eighteen time usage questions connected to the nine different
functions were divided into two rectangular structures. See Appendix A to see the questions used
in survey 1.
9Formative and summative evaluations. Available at:
http://jan.ucc.nau.edu/edtech/etc667/proposal/evaluation/summative_vs._formative.htm, [cited 2007-01-25].
37
Methodology
2.6 Interview with Heart Net Waffle
This section will describe the interview with the Heart Net Waffle group by firstly focus on the
background of the interview, secondly, give a brief summary of the interview, and thirdly, discuss
the strategy used at the interview.
2.6.1 Background of the interview
When the results from system test 1 arrived, a short meeting was held with some of the professors.
The overall results weren't to satisfactory and measures had to be taken to ensure that some
qualitative data were retrieved from the user test group. During this meeting, when reviewing and
discussing the results from system test 1 a new option became available. Namely to conduct an
interview with the test group from system test 1. Before this option was thought of as not possible
but as the time went by this option emerged.
There are a number of different methodologies one can use to research a topic. Methods and
theories ranging from inductive to deductive and more. When more than just one method is used to
perform research on a topic it is called triangulation. This is normally seen as a difficult task not
suitable for a number of different disciplines, but for HCI it is different. As discussed by Mackay
and Fayard (1997), by using triangulation, practitioners of HCI can efficiently secure the validity
in the results gathered from the different tests.
“Individual research strategies are necessarily limited and contain serious flaws. Yet what
is a weakness for individual research paradigms is actually a strength for HCI: if we can
understand the relationships among them, we can triangulate across our component
disciplines and improve the validity and value of our research.”
Mackay and Fayard (1997, p.232)
It was decided that an interview with the Heart Net Waffle group should be conducted since the
results and thus primarily meaning the qualitative results from system test 1, were not satisfactory.
The interview took place at the Saitama Toyopet10 store, in Saitama prefecture, Japan the 26th of
February 2007. Two persons from the Heart Net Waffle group did participate, and two professors
from Shibaura Institute of Technology acted as translators during the interview. This was done to
minimize the risk of the questions and answers from the session becoming misunderstood.
Since the results from system test 1 weren't satisfactory, the goal of the session was to get more
qualitative data from the test group. The results from this interview were then supposed to be a
part of the final analysis on how to make the system and its functionality more correlated to the
needs of Japanese wheelchair users. As mentioned earlier in this report, the original idea when
creating system test 2 was to use the results from system test 1 to make new functions, or better
10Saitama Toyopet homepage. Available at: http://www.saitama-toyopet.co.jp/, [cited 2007-02-28].
38
Methodology
correlated functions, for the prototype used in system test 2. The problems that arose were
shortage of time and not so giving qualitative results. Firstly since system test 1 was delayed, so
where the results from system test 1. So the development of system test 2 had to start prior to a
satisfactory amount of data had been collected from system test 1. Secondly the qualitative results
from system test 1 did not hold too much interesting information. To provide better input to the
parts of system test 2 that were not already developed, this interview session were thought of as
good additional input of information to the development process of system test 2 as well as a
chance to get extra input and ideas for the future development process of the wheelchair project.
The option of conducting an interview was not seriously considered until a later stage in the
development process. Therefore due to lack of valuable information, the main parts of system test
2 were built using a different approach. This will be discussed more thoroughly in section 2.7.
2.6.2 Brief summary of the interview
Firstly the users were contacted and informed about the interview and what it was going to be
about. After some consideration the users did agree to meet up for the interview. The plan was to
make the interview in two parts.
The first part was supposed to try the different functions within the system using a printed version
of system test 1. In other words; printed out versions of the slide shows that did expose nine of the
different functions in the wheelchair project. This to be able to judge which already implemented
functions would be interesting to the users. The idea was to let users rate the different functions
within the system according to the same five grade scale that was used in system test 1.
The second part of the interview was meant to gather more qualitative data from the users
regarding their daily life and thus by that try to figure out which needs the wheelchair project
could assist with.
During the actual interview only the second part of the interview was conducted. This due to users
requests and the fact that the users felt that they already had done the first part of the interview
(through system test 1). Prior to the interview, the participants of the interview were informed
about the ethic policy of the test. The questions used in the interview were based on a couple of
main questions deduced from the questions used in survey 1, but also during the interview, new
questions were formed from the input gotten from the users.
Generally speaking, the interview could be regarded as being more informal than formal in its
appearance, as discussed by Bell (2000, chap. 9). This since the interview mainly focused on
finding out users own ideas and needs (qualitative data) if users were presented with a system such
as the one being developed in the wheelchair project, and such data is highly subjective and thus
hard to deduce from theories or presumably before-hand picked questions alone. See Appendix B
for the interview plan.
39
Methodology
2.6.3 Strategy used during the interview
The idea was firstly to go with a contextual interview structure as described by Beyer and
Holtzblatt (1998, chap. 3) and then mainly by using a Master/Apprentice model for the interview
Figure 2.3: Interview session with the Heart Net Waffle group
and the rating part of the different functions within system test 1. But since the users interviewed
already felt that they had put in their opinion about how they saw the functions within the system
when doing system test 1, the idea was altered to follow the line of a conventional interview,
including the parts of: the conventional interview and the wrap-up.
The conventional interview being the main part of the actual interview in where the actual
interview takes place, did consist of an interaction between the user and the interviewer (see
Figure 2.3). It is worth mentioning that before the interview started, some green tea was enjoyed
and other topics than ones related to the interview were discussed to make the users as well as the
interviewers feel comfortable with the situation. Once the interview started the interviewer did use
a pen and a paper taking notes as the interview proceeded. It was important to verify that those
notes correlated to what had been said during the interview. This was done in the wrap-up.
The wrap-up, which followed, means summarizing the information gathered and interpreted
40
Methodology
during the session. In other words: the interpreted information was repeated to the users so that the
users could verify that the interviewer had interpreted the information correctly. This was done as
the interview proceeded and helped to lower the risk of misinterpretation of the data. Directly after
the interview these notes were rewritten into something the results presented further down in this
report would be based upon. This since then it was still easy to accurately recall what had been
said during the interview and thus minimized the risk for misinterpretation even more. It is
important to verify and document the data as described in Bell (2000, chap. 9).
The possibility of bias in the results was considered. To minimize the risk of bias affecting the
results, besides doing the wrap-up as described above, there were only one of the interviewers that
actually did the talking towards the interviewed. This, to lesser the risk of misinterpretation in both
directions. Also since I was keen on knowing what thoughts and ideas the interviewed had, I tried
to keep an open approach and not let prejudice control the questions, or the follow-up questions
that did occur during the interview. Still there is always a possibility that bias did affect the results
from the interview. This since, maybe the interviewed wanted to please the interviewers (us) in
some way, since we “were there” for them in a sense. And secondly since the development of the
prototype for system test 2 had started and, though I tried not to, unconsciously, maybe I wanted to
try the ideas of those newly developed functions towards the needs and wants of real users by
formulating the follow-up questions in a way to better suit that purpose. The fact that there were
only two participants in the interview did possibly also contribute to bias in the results.
What could have been done was to record the interview, since a recording could prove to be useful
when summarizing the interview results.
41
Methodology
2.7 System test 2
This section will discuss system test 2 by, just as the section about system test 1, firstly describe a
bit of the background of the test and some trouble that arose during the development process.
Thereafter this section will give away some technical information, discuss some about how the test
came to be and finally, what users saw and experienced when they entered the test.
2.7.1 Background of system test 2
What is worth mentioning before going deeper into the background, is that the ideas from the
interview with the Heart Net Waffle group showed that the development of system test 2 and the
theories the test was based upon were on “the right track”, even though the development of the
prototype had started before sufficient results had been collected from system test 1 or before the
interview session with the Heart Net Waffle group had been conducted. This correlation between
the ideas deducted from the interview session and the system development process of system test 2
lead me to continue with the project of finishing system test 2, even though the results from system
test 1 were not too satisfactory as described in chapter 3. This since the idea was still to be able to
see if incorporated thoughts from users would affect how users viewed the two different system
images. In the updated system image (the prototype) there were now implemented functions that
corresponded to input from real wheelchair users, so would it make a difference?
System test 2 differs from system test 1 in a couple of ways, but maybe the most conspicuous
difference is the fact that system test 2 exposed a prototype to the users whilst system test 1 did
consist of slide shows. As described in Göransson and Gulliksen (2000, chap. 3) there are different
types of prototyping methodologies that can be used when creating a prototype. Methodologies
focusing on depth or width or a combination of the two. The prototype that was constructed for
system test 2, were created using a mixed focus but with an overlay in depth focus. This because
the input that were used to decide which functions to implement in the prototype were well
described but only for a few functions. It was decided early on that a different set of users than the
set used in system test 1 were to be used in system test 2. This both because my professor saw this
method fitting and also because this would minimize the risk of specific users demands being met
which could result in a less accurate and giving result for the whole population. Even though there
was a possibility that the two user groups differed somewhat in terms of age mean value, this was
considered a trade off worth taking.
As with system test 1, I did choose to go with a remote usability evolution technique. This because
there was a time limit on the project, which means that starting over with a different method would
not have been time efficient in terms of using the knowledge gathered from system test 1, but also,
more importantly, the possibility to compare results from the two tests would have been
compromised. Since the two system tests had different structures with one test using slide shows
and the other focused on exposing a prototype it might already have been hard to compare the
results from these two tests, but it would probably have become even harder if a different
42
Methodology
methodology all over was used. The idea was to base the functions and the layout of the prototype
used in system test 2, on the results gathered from system test 1. But as will be discussed here
below, system test 2 came to be based mainly on the results from another test done in USA.
2.7.2 Trouble, system test 2
During the creation process of system test 2, some trouble did occur. Problems ranging from
technical issues to translation problems. But maybe the biggest issue that I faced when performing
this study and creating system test 2, was the fact the results from system test 1 were delayed and
not at all so fruitful. In fact, the results did not give the desired information required to build the
prototype. Since time was an issue and there was a time-line with a deadline to follow, something
had to be done. The solution was to go with qualitative input data from another project when
deciding which functions would be implemented into the prototype of system test 2. The two
systems are only somewhat correlated since the system developed in Bhatia et al. (2006),
VTAssist, so far had been focused on areas inside buildings. But the idea of letting users share
information between each other and the knowledge of, that to users, some places are more
important than others, connect the two projects among other things.
Among the technical problems were server incompatibilities, leading to extra time spent rewriting
program code so that the software, and system test 2 would work with the server side software.
One thing that was learned is that when attempting to write code that will require the support of
foreign languages and characters , it is a very good idea to have a very similar if not identical
testing ground compared to the platform or software one is using when launching the project.
From system test 1 it was learned that the pictures used in the slide shows contributed to long
downloading times. Hence maybe users entering system test 1 were affected by this when entering
their results. Since the two tests were different in structure, it was decided that minimizing the
need of pictures in system test 2, would make the system data faster to download for users. Then
the users would not have to wait an inconsiderable amount of time when trying to enter the test or
when doing the test.
During system test 1 it was understood that it is wise, when translating from English to Japanese,
to take help from a translator who understands the context. There is a possibility that the varying
results deduced from system test 1 were an indirect result of poor translation. To solve this
problem, my professor, who has a good knowledge in English and experience with computers,
took on the task of translating system test 2. Also this way the context and meanings I tried to get
across, could easily be checked and corrected since me and the professor did communicate on a
weekly basis.
43
Methodology
2.7.3 Technical information, system test 2
System test 2, a web-based application, did consist of totally 167 files and came to have a total size
of about 5 mega bytes. Totally there were about 12 different tables created in the database to keep
track of the traffic and data generated by the users and the system itself. As in system test 1,
MySQL technology was used in collaboration with HTML scripting, JavaScript and PHP
techniques. See for example Williams and Lane (2004) for further reference.
2.7.4 Iterations, system test 2
The system did go through approximately one system iteration before the test was put online.
Unfortunately not much user testing was conducted in the laboratory environment. This since the
presumed users that would test the prototype were laboratory comrades and at the time very busy
writing their master thesis reports. A lot of internal iterated testing was done to try to ensure that
all options that included actions from users in the prototype lead to some form of result. But the
only actual user testing the system did go through was a walk through with two professors. Worth
to mention (again) is that the results from system test 1 were delayed, which in turn delayed the
development process of system test 2 and minimized the time that could be spent on system
iterations.
In the first system design, slide shows were to be incorporated into the prototype. As insinuated in
2.5.3, the idea was to take the best rated functions from system test 1 and let those functions be
represented as incorporated slide shows in the prototype. When a user pushed the button that
represented the function found in the original system, the slide show for that function would
emerge in a new pop-up window. However the results from system test 1 were insignificant so the
idea of incorporating slide shows into the test was abandoned. Instead the only thing tested during
system test 2 was the prototype itself with the functions that so far had been implemented into it.
There was no tutorial in the actual test since it was considered to take up to much of the users’
time. The idea was that: instead of trying to read a manual on how the prototype works, it might be
faster to just try the prototype to get an understanding of how it works. As described in Krug
(2006, chap. 10) it is important to try to preserve the users good will. If a tutorial was to be put up,
and the users had to go through it before starting the test, this demand of reading could effect the
overall happiness and willingness to take the test. This since reading and understanding a tutorial
would take up an extra amount of time and may not even help the user to properly understand how
the actual prototype works. To implement a tutorial as an extra click-able option was considered
but due to time shortage this idea was taken out of the picture. Instead a paper based tutorial was
created and then sent out to the Aoba Barrer Free Support 21 group prior to test start.
44
Methodology
2.7.5 How did system test 2 work?
System test 2 did follow a similar procedure as its predecessor. Firstly the users were taken to the
main page that did show that out of the two system tests only system test 2 was online (see Figure
2.4). If then the users clicked on the appropriate link leading to system test 2, the users would be
Figure 2.4: Start page, system tests 1 & 2
taken to a login page. This login page did fill the same function as the login page in system test 1 –
namely to somewhat guarantee that the users came from the correct user test group. Upon entering
the correct password the users were then taken to an information page that in short did debrief the
user on info about the test.
45
Methodology
Information about the background of the test, ethic information, contact information, as well as
Figure 2.5: Information page, system test 2
some information about the prototype was written in that page (see Figure 2.5). Also if the users
felt the need for more information, five different links were provided at that page offering just that.
If these links were pressed new browser windows would pop-up exposing different information.
One link provided information about how the results from the test would be used and where it
would be published, and in another link the survey and the amount of questions were discussed.
Another link would let the users know more about the actual test and the prototype, and a third
link would give the users encouragement on how to think when doing the test. The final linked
page did display contact information to the author if the users felt compelled to know more about
system test 2. At the end of the information page, there was a link to the main testing page and a
link to end the test. It is important to inform the users about the test and what it is about, how the
results will be treated and the rights of the users as described in Bell (2000, chap 4.). So if the
users felt that they did not want to go through with the test after reading the information about the
test, they could finish the test before it had started as well as quitting any time they felt like
quitting. The users were also informed that they only needed to answer the questions they felt like
answering and that all results and users would be anonymous when used and presented in the final
report.
When arriving at the main testing page, the users had three clickable options. They could either
bring up the information page once again if the users felt that they had missed some information,
46
Methodology
Figure 2.6: System test 2 – flow chart
start the prototype or go to the survey to answer some questions (see Figure 2.6).
47
Methodology
Now if the prototype was started a new window would pop-up in where the prototype would be.
The prototype had a similar layout compared to the layout of the original system. In the top there
was a top-bar, to the right a control panel with buttons and to the left, a city map was taking up the
major part of the graphical interface (see Figure 2.7).
In the top-bar, a logo stating that the prototype was a part of system test 2, and two buttons were
found. In the old system, most of the functions could be divided into either searching the system
for information or submitting information to the system (see sections 1.5.2.1 and 1.5.2.2). So these
two buttons did represent those two categories of functions. In other words one button did give the
users access to functions that did search the system for information and the other button did give
the users access to functions that submitted information to the system. The functions were divided
Figure 2.7: Prototype, main window
into these fields mainly because of the fact that the control panel belonging to the original system
held more than ten different buttons or click-able choices for the users to choose from and I saw it
fitting to make the design more esthetical.
In the prototype top-bar there were the buttons “Registration/Modification - 登 録 ／ 修 正 ” and
“Search information – 探索情報”. By pressing these buttons the add-remove bar or the search-bar
would show up as a side-bar to the right of the city map.
48
Methodology
In the city map there were icons representing specific places on the map just as the original
system. Places as handicap toilets, restaurants, hospitals, food shops and elevators were
represented by these icons. If the icons were pressed a new window would pop-up with
information about that particular place. In this new window there would be an icon letting the user
know where he or she had arrived at, a description containing information about the place and
Figure 2.8: Prototype, entering and posting information
some buttons letting the users do various actions (see Figure 2.8). To the right of the description
there were two buttons connected to the “register/unregister”-function that will be discussed
shortly. But bluntly put, the user could choose to press these buttons letting him or her register or
unregister to information from that location. As told in Bhatia et al. (2006) users found it
interesting to be able to get continuing information from a specific place. So in system test 2, this
idea was implemented.
49
Methodology
Below the descriptive information there was room for a list of comments and a input-field,in the so
called “information-posting”-function. By using the input-field, users could post comments to the
specific location. The comments were ordered from top to bottom in a reversed chronological
order. Thus meaning that the latest post would end up in the top of the list with a time stamp on it
showing when it was posted. So lets say that a user went to a toilet and discovered that the toilet
was flooded, the user could log on to the system and post a comment on this specific toilet place,
enabling other users to know what has happened. By judging from the time stamp and in the order
which the comments appear it would be easy for the users to deduce if the information is relevant
or not.
Figure 2.9: Prototype, “register/unregister”- function
With the ”register/unregister”-function (see Figure 2.9) users could, by either entering a specific
information-spot (for example a toilet) and checking a “register to spot”-box, located at the right
side of that page, or by using the “add or remove subscription”-function located in the add/remove
control panel, put specific places under observation. By registering to a place, a link to that place
would show up in the lower right part of the system, making these sites easy to access for the
users. Lets say a wheelchair user works in a specific area of the town and need to know if a toilet
or an elevator he or she uses on a daily basis is accessible for the time being, the user can log in to
the system and get to the desired location fast by using the links in his or her hot-list. Since users
tend to help each other and are willing to post serious information when they are dependent on
similar information, as described in Bhatia et al. (2006), the information that users post can most
likely be taken to be accurate.
50
Methodology
The “search”-function (see Figure 2.10) does somewhat resemble the “Search and recommend”function that were found in the original system with the modification of not needing to put in the
user coordinates. Since the results from system test 1, were not satisfactory it was hard to tell how
appreciated the “Search and recommend”- function really was. This lead way to experimentation
through analysis of literature (Bhatia et al. 2006). Even though some minor alterations were done
Figure 2.10: Prototype, “search”- function
to the function if compared to the function implemented in the original system, the idea stayed
quite the same. The idea was to let users search for specific places on the map. Let's say a user
wanted to find a good lunch restaurant which was accessible for wheelchair users, the user could
then use the search function to get a list of interesting restaurants. In the prototype, the only way to
perform a search was to search for a kind of place like toilets or restaurants etc, but in the future
maybe other search-options could be implemented into the system. When accessing this function
from the search side-bar, a new pop-up window would appear and the user would be able to
choose from a drop-down list of different categories. And then when pressing the search button, all
alternatives that match the one that the user chose would appear as links just below the drop-down
list.
When the users had tested the prototype they could then go to the survey from the main testing
page. After answering the questions the users felt like answering they could then send the
information to a database by pressing the “submit survey”-button. By doing this, the users were
taken to a page that confirmed that the user had posted the information as well as thanked them for
51
Methodology
putting up with the test.
2.8 Survey 2
This section will follow a similar pattern to that of survey 1.
First, the background of survey 2 will be brought up. Then there will be some focus on the theory
of the open and multiple choice questions of the survey. Finally the section will end with some
information about the layout of the survey.
2.8.1 Background of survey 2
As mentioned the results from system test 1 were not so good in terms of providing valuable
information, and there were a number of different factors to consider as to why the results had
become the way they had. One factor could have been the amount of questions used in survey 1.
When reviewing the results it was concluded that there were simply put to many questions for the
users to answer and also that some of the questions were easy to misinterpret, even though
measures had been taken and methods used to try to minimize these sources of error as described
in chapter 2.2. During system test 2, the amount of questions were heavily reduced, in fact less
than half of the amount of questions that system test 1 had were kept and used in survey 2. Some
of the questions were kept and rewritten other questions were completely new. The alternatives to
most of the questions were rewritten to include more information with examples and explanations.
This was done to possibly make the users interpret the questions similarly to how I did. The
background information to the examples and explanations were taken from the results collected
during the interview with the Heart Net Waffle group. As discussed in Bell (2000, chap. 9), doing
pilot interviews can deduce alternatives or areas of interests that can be used for further studies or
analysis.
2.8.2 Open questions of survey 2
The goal of the open questions in survey 2 was the same as the open questions in survey 1; To
catch users needs and wishes for the future development of the wheelchair project. There were six
open questions that gave the users a chance to provide qualitative input to the results of system test
2. Just as survey 1, the qualitative questions tried to understand the daily life of users and how
users perceived the updated system image (the prototype). The idea of conduct and usage of theory
was the same as for the open questions of survey 1, but with the exceptions that, now explanatory
text was provided to the questions when needed, and also two questions now included multiple
choice alternatives as well as a text-area for input. This to make it easier for the users to answer the
questions.
52
Methodology
2.8.3 Multiple choice questions of survey 2
The quantitative data extracted from survey 2 were meant to be used as a comparison between the
quantitative data gathered from survey 1. To be able to compare efficiently, the questions and the
alternatives did not change much. The focus was still to find out how well the system image, in
this case the prototype, correlated to the users needs. To be consistent and to be able to compare, I
used the same questions and answering alternatives that I used in survey 1. The quantitative
questions only differed in one way. The difference was that there were not any multiple choice
questions (time usage questions) regarding the functions of the updated system image (the
prototype) included in survey 2. This due to insignificant results gathered from system test 1.
Therefore there were only two time usage questions in this survey, and of course the question
regarding the usability of the user interface.
53
Methodology
2.8.4 Layout of survey 2
The amount of questions had been shrunk to a total of 14 questions, including a total of six open
questions and eight multiple choice questions (see Figure 2.11).
Survey 2 was divided into two different parts; questions about the system and questions about the
user. These two parts of questions were mixed with each other in the survey. This to ease the task
Figure 2.11: Survey 2, questions
of answering survey 2 and possibly to get a better result than the result collected from survey 1.
To minimize scrolling and downloading times the questions were ordered in a similar way as the
questions in survey 1. See Appendix C for further reference.
54
Results
3 Results
In this chapter the qualitative and quantitative results gathered from system test 1 and system test 2
will be displayed. The qualitative results and information gathered from the Interview with the
Heart Net Waffle group will also be presented in this chapter. The qualitative data will be
presented first followed by the quantitative, and since the chronological order of the tests and
interview were: system test 1, interview and system test 2, I will present the results in that order.
But first, what needs to be mentioned here is that during the time for deciding the method of
investigation, the amount of possible participants for the tests from both the Heart Net Waffle
organization and the Aoba Barrer Free 21 organization, were thought to be housing at least 10-15
members capable of doing the tests each. There is a possibility that the exact amount of people
differed from that number at the time for the tests, but since contact was not established directly
towards the test-groups during research it was hard to estimate the exact amount of users that
could have been or was reached by the different tests and the interview. Even though there was a
short meeting with some members from the Heart Net Waffle organization just prior to system test
1 and the interview, it was hard to estimate the number of users belonging to the organization
capable of doing the test.
3.1 Qualitative data
In this section the report will present qualitative data obtained from the two different system tests
and the informative data collected from the interview with Heart Net Waffle group.
3.1.1 System test 1
The qualitative results from system test 1 were almost absent, and not particularly giving. In total
there were only two people from the Heart Net Waffle organization that did go through with the
test and answer most of the questions in survey 1. Even though there was more quantitative data
collected, in comparison to qualitative data input, the data was in general limited, and in fact, there
was only one of the users that did go through and answer the open questions (the basis for the
qualitative data). It is probable that the low amount of test-participants is what caused the data to
be poor. Three possible reasons for why the test came to have this low amount of participants are
discussed in chapter 4. As a side note, maybe the system tests should have been designed to better
keep track of the amount of users as well as how many questions they answered since this would
have helped to better understand how many of the users that entered the tests answered the
questions in the survey.
Even though it seemed quite clear that the user that had answered the questions, had interpreted
55
Results
them differently than the way I hoped for, some results were still achieved. In total the user
answered three open questions, which were questions 4, 5 and 6 (see Appendix A for further
reference). The user did also provide some extra input in the extra comments space.
3.1.1.1 System test 1, results
As a response to question 4, the user typed: “work, play and others”, which is hard to interpret,
and probably indicates that this particular question could have been expressed in a different
manner. This since the goal of that question was to get a deeper understanding of the daily life of
the users, which then hopefully could have been extracted into a need and in the long run a
function answering to that need.
When answering question 5, the question trying to find out time-consuming actions, the answers
were: “work”. I had hoped to get more descriptive answers, making it possible to single out
specific time consuming actions, such as maybe going to the toilet, looking for a food shop or
such, but these were the qualitative results gathered from system test 1.
In the space of question 6: “If you could have daily access to information, what kind of
information would ease your daily life?”, the user wrote; ”Barrier-free place and in the store of a
multipurpose restroom.” This could be interpreted as there were an interest in knowing about
where barrier-free places are located, and information about where handicap accessible restrooms
are situated and what kind of equipment they hold. However since the richness and amount of
qualitative data from system test 1 weren’t very satisfactory the validity of the interpretation could
be questioned.
Finally, in the extra comment space (question 14), the user did write a comment about the
hardware of the system. The user wanted to see this system become easy accessible from hardware
the users could bring with them. This could mean that the hardware, on which the software system
runs, is important. Besides this data, no qualitative data of importance were put in to survey 1.
3.1.2 Results from the interview with Heart Net Waffle
In total there were only two participants from the Heart New Waffle organization that were taking
part in the interview. A couple of possible reasons for this decline in users, or low amount of users,
could have been the communication process between involved parties (people involved in setting
up the interview) as well as the possibility of a low amount of users in the test-group to start with,
as discussed in chapter 4.
During the interview both users answered the questions and helped each other, filling in where the
other one left off. In total there were approximately 10-15 questions being threated during the
interview, including the follow-up questions that followed the main questions as described in
Appendix B, but at the same time it is really hard to estimate the exact amount of questions that
was asked during the session since the interview was more informal than formal in its nature as
56
Results
described in section 2.6.2.
One of the things that could be concluded from the interview was that users thought it was hard to
understand system test 1. Even though measures had been taken to make the test easy to
comprehend by using various methods as described in section 2.4, users had a difficulty of
understanding how the system worked.
Another thing that came up was that users wished for a different and maybe more dynamic layout
of the hardware when it was to be implemented. This since there are a variety of wheelchair users
with different handicaps. In other words some users might not be able to use a system that only
works on a specific platform. But this projects focus is not on the hardware, it is on the functions
within the system and how those correlate with the needs of wheelchair users. Therefore, I will
from now talk about the functions and the needs the participants had and describe some of the
results deduced from the interview.
Even though the newly discovered needs differed, most of them were connected in the fact that,
generally users were interested in information about the accessibility to places in different forms.
What more is that sometimes another wheelchair user might be the best suited person to check that
accessibility. Therefore the users were also interested in functions or ways of communicating with
other wheelchair users when discussing accessibility to different places.
The main issues addressed by the interviewed could be labeled into the categories:
●
●
●
●
●
●
●
●
●
Toilets
Parking space and traffic jam
Restaurants
Elevators
Daily shopping
Gas stations
Static information
More accessibility issues
Weather conditions
3.1.2.1 Toilets
I learned that finding an accessible toilet when exposed for a new area is one of the most crucial
things for a wheelchair user according to the participants of the interview. Also during this process
it is important to know what this particular toilet holds. Is the toilet equipped with a stretch board?
Does the toilet have alarm buttons placed low and high in case of an emergency? Are there
supporting armrests installed in the facility? Since all wheelchair users have different needs,
information like this is of utmost importance.
When asked the question if users would be interested in a function that would let users share
information about places like different toilets for example, all of the participants of the interview
said yes. To be able to subscribe to information from a specific location, like a for example a
toilet, was also considered to be an interesting function. Then users might help each other by
57
Results
posting information such as; “The toilet paper is out” or “The toilet is currently flooded, search for
a different toilet” to the system.
3.1.2.2 Parking space and traffic jam
All of the participants of the interview were using a car in their daily life, and finding a place to
park their car was considered difficult at times. This since normal parking spots do not offer the
extra amount of space between the parked cars that is required for a wheelchair user to be able to
get in and out of the car with the wheelchair. It was considered a good idea to have the option of
getting information on where to find different parking spots and if they are free. This could help
wheelchair users save time when they are commuting.
As a side note, one of the participants also mentioned an invention she had witnessed on a
wheelchair fare 2006. It was a remote control that would let wheelchair users lower an iron fence
from the wheelchair parking spot, thus stopping non-wheelchair users from using the wheelchair
users’ parking spot. Maybe there is a possibility to connect the parking spots in the city to this
system, letting the users see if the spot is available or not, and thus incorporating this remote
control- function into the system.
Finally users thought it would be interesting to come across information about if there is a traffic
jam or something else blocking roads in their vicinity. If the system was to be connected with a
central server providing traffic information, maybe this could be incorporated into the system in
some form.
3.1.2.3 Restaurants
Another issue that was raised during the interview was that of trying to find good accessible lunch
restaurants for wheelchair users. If the participants in the interview wanted to visit a new lunch
restaurant, some pieces of information was crucial to make the visit possible at all.
First of all they had to know if the chairs in the restaurant were movable, so that users’
wheelchairs could take the place of the chairs in the restaurant. Moreover they needed to know at
what floor the restaurant was situated. If the restaurant is not located at the bottom floor, is there
an elevator accessible for wheelchair users in the same building? Finally one important thing about
restaurants was the accessibility and closeness to toilets adapted to wheelchair users. If there
wasn't a toilet nearby that easily could be accessed, the restaurant was not considered a good
option. Usually this kind of information is hard to get across, so wheelchair users tend to go to the
places they know of instead of putting down the effort it takes of finding new restaurants.
Searching is also connected to a risk. If I was a wheelchair user, maybe some of my attempts of
finding a new restaurant would be fruitless and thus result in frustration and less willingness to
search for a new place to eat. By letting wheelchair users share information between each other,
maybe through a system like the one being developed, a network of good restaurants could be built
up. This network could let users use the experience of other users to let them know where good
58
Results
“new” restaurants are located. Furthermore, other values might be important. For example does the
staff omit a good feeling to the wheelchair users, making the users feel comfortable as guests at the
place? Since wheelchair users understand wheelchair users best, information such as this could
also be interesting to share between the users in the system according to one of the participants.
3.1.2.4 Elevators
The participants of the interview thought that it could ease their life to get some knowledge about
specific elevators before entering them. This since some elevators are fitted with carpets which
makes it hard to move in and out of the elevator for wheelchair users. Also many of the elevators
have poorly designed control panels, hindering wheelchair users from reaching the top floors since
the buttons on these control panels are placed to high, out of reach for the wheelchair users.
Furthermore many of these control panels also hold poorly designed close-buttons for the elevator
doors. With this I mean that these buttons tend to be placed out of reach for a normal wheelchair
user, and thus forcing the wheelchair user to wait the extra seconds before the doors in the elevator
are closed. This is obviously frustrating in the long run.
Some elevators are also too small to fit a wheelchair, and others can only fit just one wheelchair
and no one else (like for example an assistant). Even though sometimes annoying, elevators may
be crucial for the transportation of wheelchair users. Receiving the information that an elevator is
not functioning before starting out on an errand, may save wheelchair users a considerable amount
of time and frustration. So simply put, it would be interesting to know about elevators, their
accessibility for wheelchair users and their working-status. The possibility to share this
information between different wheelchair users with, for example a commentary system, was seen
as interesting by the participants. The possibility to share information between users in general,
was seen as something interesting.
3.1.2.5 Daily shopping
The participants of the interview said that they usually went to the same store on a daily basis.
This since it is easier to use shops that one knows about and is familiar with, but also because of
the fact that some uncertain elements are connected to the process of finding a new grocery store
to do errands in. For example, it is important that the distance between the shelves is large enough
to enable for passage with wheelchairs, so how to get this information? More questions could be
raised, for example: “How is the service for disabled people in that place?” and “What things do
the store hold that can be of use to wheelchair users?”
If some users have been to a place before, maybe they know the manager and know if a wheelchair
user can get help in that place or just know some other information that is interesting to other
wheelchair users. If this information could be shared between wheelchair users it could help
wheelchair users to get a greater variety in their food consumption habits.
59
Results
3.1.2.6 Gas stations
Some gas stations are manned and some aren't. If there is no service personal at the gas station the
option of stopping and filling up gas could be seen as not eligible for some wheelchair users.
Furthermore some gas stations provide service and help when washing cars, and some do not
provide this service. In other words wheelchair users are more or less dependent on the fact that
they can get help when serving their car. So information about if the gas station is manned and if a
customer can get help when washing his or her car is crucial information when making the choice
of which gas station to stop at.
3.1.2.7 Chat and email
Connect all users to a forum, a chat-room or through email! The interview showed that users
would find it interesting to be able to help each other through the system. For example if users had
questions about different topics, such as: where to find a descent restaurant that is accessible? or
other questions that other wheelchair users might have knowledge about, the participants of the
interview thought it would be interesting to be able to share that information between wheelchair
users. If all users of the system could be connected to a single forum, or if they would be able to
receive email from, or chat with other wheelchair users then this knowledge could be shared
among the different users.
3.1.2.8 Static information
Information about sites and how accessible those sites are to wheelchair users is interesting
information. For example do buildings have ramps that wheelchair users may use when entering
the building? And how about the interior of buildings? Sometimes there are stairs inside a building
but no ramp close by to let the wheelchair users pass. It is important to be able to get information
about sites like these according to the interviewed.
The participants of the interview were also asked if they thought it would be a good idea to let
different users share their experience from entering sites like those buildings described above. For
example one user would describe a ramp and what he or she thought was good or bad about the
ramp in a post for other users to view. But the participants thought it would be more important
with exact descriptive information about sites like this and if possible also a photograph or such to
let the wheelchair users themselves be able to decide if the location was accessible for them or not.
Let's take a ramp as an example. In case of a ramp, the information could be expressing
dimensions such as height and width of the ramp and if the ramp includes an armrest. This since
there are a lot of different wheelchair users and all with their own specific needs. Furthermore all
wheelchair users are experts on their own ability and what is required for them to be able to move
from one spot to another. So if given the correct information they themselves make the best
decision whether or not the spot is accessible to them.
60
Results
3.1.2.9 More accessibility issues
As mentioned earlier, the interviewed seemed to take interest in the accessibility of different
places, and as the interview progressed more and more things came up as described in the sections
above, but there were also a couple of things connected to the accessibility of places that were only
briefly discussed during the interview and did not get as much time and attention as the other
issues, but are nevertheless important. Some of these things are described below.
When doing errands in shops or in office buildings, some of the registers are placed to high for
wheelchair users, making conversation and service difficult. The interviewed thought it would be
interesting to know about what places hold adapted registers for disabled people. Providing this
type of information could be helpful to wheelchair users when they choose sites to do errands in.
When going on vacation, the interviewed thought it would be nice to know which ryokans(旅館)11,
accept wheelchair users and thus primarily wheelchairs into the rooms. This since during the
interview it was learned that the traditional tatami floor(畳)12, of these rooms tend to break easily
when the wheelchair is turned on top of the floor. Maybe the system could be regularly updated
with information such as this.
Some of the buses that is being used in Tokyo, do not currently have ramps installed at the doors.
This makes it hard for a wheelchair user that wants to use the bus, since it is hard to tell which bus
is easy for the user to get on. Getting information about buses and if they do have ramps could
ease the daily life of wheelchair users.
One of the participants of the interview said that when searching for information about where to go
on different errands, in general the course of action is to check for a place through the Internet,
then call to that place to check that the information posted on the Internet is accurate, and finally
go there. Some users are shy and might not want to make that phone call to check if the
information is accurate, and thus deciding not to go since they can not be certain if the information
posted on the Internet is accurate or not. If the system could be updated regularly, users would not
have to check if the information that the system holds is accurate or not, enabling more users to go
to the places they want to go to without exposing themselves too much to other people.
3.1.2.10 Weather conditions
At the end of the interview, it was concluded that weather conditions does not significantly affect
the way wheelchair users live their lives. This since even though some places may not be
accessible when the weather is bad, it is today hard to tell who takes which kind of precautions
concerning the weather and where those precautions are taken. For example if it snows it is
obviously more slippery outside and some ramps and buildings may not be accessible. But
nevertheless since it is hard to tell which people or companies that take measures with changing
11Ryokan. Available at: http://en.wikipedia.org/wiki/Ryokan_%28inn%29, [cited 2007-03-02].
12Tatami. Available at: http://en.wikipedia.org/wiki/Tatami, [cited 2007-03-02].
61
Results
weather and which people and companies that don't, wheelchair users tend not to to alter their
daily routines when the weather is changed. Although it was mentioned that if information about
where non-slippery places, extra outfitted ramps or such, were to be found, it would be interesting
to know where those ramps or such were located.
62
Results
3.1.3 System test 2
In total there were only one participant that did follow through and finish system test 2, and thus
trying the prototype. Since this low amount of users could, and probably will add bias to the
results, some possible reasons for this, “bigger than anticipated” decline in users, are, as with the
other test and the interview, discussed further in chapter 4. Also it needs to be mentioned that, by
judging from the demographic quantitative results, the participant in system test 2 did not consider
herself to be a “real” wheelchair user. This could affect the validity of the results and possibly adds
bias to both the qualitative and quantitative result. Still, being a member of the Aoba Barrer Free
21 organization, an organization that houses disabled people and then mainly people that could be
labeled as wheelchair users according to my professor, it is possible that the results from system
test 2 still could prove to be informative and somewhat qualitative for other disabled people such
as wheelchair users.
Go to Appendix C to see the questions providing the results discussed below.
3.1.3.1 System test 2, results
Firstly the user thought that the things that required the most effort in the users daily life business
(question 1) was to find a parking spot for the car. Also the same participant mentioned that one of
the most time consuming business that the user faced each day (question 2), was to go to the toilet.
Important information that the user wanted to have easy access to each day (question 3), was the
operation situation of trains. In other words the user wanted to get information that would help the
user to answer questions such as: “Is there any problems with the trains?” and “Is there a delay in
the system today?”
The participant thought that the function that was least useful or hard to understand (question 12),
was the search function that would let users search for different places on the map. There were not
more input written than just the functions name, so here is room to speculate why the user
answered the way he or she did.
The most interesting and useful function (question 11) according to the user was the ability to post
information to different spots and places. Also it was appreciated that posting this information was
done easily and individually. As discussed earlier in this report the prototype's “information
posting”-function did let the users post their name and their message on a location which would
show up at the top of the list with a time stamp on it in that page. The time stamp feature has also
been appreciated in other projects similar to the wheelchair project, see Bhatia et al. (2006).
For future interface layout, the user wanted to see the functions and how they were accessed more
like a tree structure (question 13). This can be compared to what's been discussed in Bhatia et al.
(2006). The user also thought it was hard to understand the map at first. The map in the prototype
63
Results
did not differ significantly to the map presented in the actual system. A big difference is of course
that there is a possibility to move around on the map in the actual system (the wheelchair project).
This is also described as a positive thing by the users that tested the system in Bhatia et al. (2006),
particularly the zooming function (which is also present in the wheelchair project). But
nevertheless the actual layout of the map were thought of as hard to understand. Investigating
more in how to improve the map representation, could prove to be a good choice for the future
system development process.
Some misunderstandings did seem to appear when it came to the use of the so called “unregister
from subscription list”- button (question 4), which was present in all of the windows representing
the different places in the prototype. If it is decided to keep the “register/unregister”-function,
more investigation and user testing could be done in the future to minimize future
misunderstandings. This would preferably be done by using a user centered development process.
Finally the one participant did express a wish for detailed accessible information regarding
different sites all around the city (question 4). In other words the user thought that providing
detailed information on degrees of inclination on ramps around the city, pavement heights, etc
would prove to be interesting for all wheelchair users.
3.2 Quantitative data
As mentioned earlier, to reach the subgoal, student's t-test was used when trying to analyze the
quantitative data (see Appendix D for further reference). This since the method did fit the need of
the investigation in the sense that a t-test can be adequate for comparing two relatively small sets
of samples. Student's t-test does assume the two different sample sets to have the same variance
which could be eligible in this research situation since both test-groups (sample groups) included
male and female wheelchair users in various ages. Since there were different amounts of samples
collected from the two tests, the correct type of Student's t-test was used. It is hard to state a
minimum number of samples required to make a statistically correct comparison between two sets
of samples when using a t-test since the results are affected by the difference between the samples
as well as the amount of people taking part in the test, but the idea of having 10-15 samples per set
would probably be sufficient with this particular method.
However, since the method of choice, student's t-test requires that the unbiased estimate of the
variance13 in each sample set is calculated, which in turn requires that at least two samples is found
within that set (and thus not making the variance become 0), the method could not provide any
valuable quantitative results. Maybe it could be argued if not a different statistical method of
choice could have been used, but since the amount of user were to low providing a total amount of
only three samples, it is questionable that even another method would have provided valuable
quantitative information to the research. It is hard to provide an exact number of minimum
samples required to statistically show a difference between two means because different methods
use different approaches, but a minimum-number that has been re-occurring in the discussions I
13Unbiased estimator.Available at: http://en.wikipedia.org/wiki/Unbiased_estimator, [cited 2007-11-27]
64
Results
came across is five14 which is larger than three.
3.2.1 System test 1 & system test 2
When looking at the quantitative data, the most outstanding thing, was the fact that there was (as
mentioned) too little quantitative data to make a statistically significant comparison between the
two tests. If more, samples had been gathered, it is possible that the quantitative data could have
been compared through the method of choice (student's t-test), but since the received sample size
from survey 1 and survey 2 were too small, the results were not satisfactory enough to make such
a comparison. The amount of samples (three) were just not enough to produce a meaningful result.
The low amount of participants in the tests, which resulted in the low amount of samples in the
tests, is discussed further in chapter 4
As discussed in Elementary Concepts in Statistics15 it is hard to prove any relation between
samples in different sets of data if the sample size is low. This since the relation and thus the
possibility to see a difference between the two sets of data, is dependent on the sample size and the
magnitude of the relation (the difference). Also, if the magnitude of the relation (the difference) is
large it doesn't matter statistically if the sample size is too small. Still, the magnitude of the
relation can be interesting in different aspects since it can give an indication qualitatively. So
therefore we take a look at the quantitative results gathered from system test 1 & 2.
As mentioned, there were two questions connected to the time usage concept discussed earlier in
the report, and one question connected to the system's general usability, or more correctly the userinterface's general usability. Multiple choice questions regarding the users were also present.
The question about the users involved users' age, gender and what type of user the users
considered themselves to be (e.g. wheelchair user, assistant etc). Of the total three participants
doing the two different system tests, two participated in the first system test. The two individuals
doing system test 1 did consist of one male (user 1) and one female (user 2), both were wheelchair
users and both belonged to the age-group of 21-30. The third test-subject (user 3), who
participated in the second system test, was a female user, considered herself to be the option
“other” when asked what type of user she was (as mentioned in 3.1.3), and belonged to the age
group “51 or over”.
The time-usage questions, did consist of two questions. The main question and a check-question as
mentioned. The main question: “How much would you use this system?” was in other words
followed by the check-question “How often do you think that this system would be helpful to you
in your everyday life?”. On the five grade scale presented for the test-subjects, user 1 chose the
alternative answering to “Once a week” on both questions. User 2 chose the alternative answering
to “Once a month or less” on both questions. User 3 did however chose the alternative “Three or
14Small numbers in chi-square and G-tests. http://udel.edu/~mcdonald/statsmall.html, [cited 2007-11-27]
15Elementary Concepts in Statistics. Available at: http://www.statsoft.com/textbook/esc.html,
[cited 2007-03-19].
65
Results
four times a week” when answering the main question, and the alternative “Once a week” on the
check-question. For further reference, see Appendix E.
All three test-subjects did also answer the interface's general usability question. Upon answering
the question:”Is it easy to understand how this system works?”, both individuals taking part in
system test 1 (User 1 & User 2) did choose the alternative that said:”Not really”. User 3
participating in system test 2 however did put in the alternative that said:”Easy”.
3.2.2 Possible indications
As mentioned, statistically significant results can not be deduced from the data discussed above,
however maybe some informative indications showing the differences between the two tests, and
how the test-subjects saw the different system images could be deduced to provide input to the
conclusion & analysis chapter.
According to the results from the interface's usability question, it seems like the upgraded system
image (the prototype) was easier to understand than the original system image (the idea the slideshows tried to get across) since the user that did system test 2 rated the easiness of the system
interface higher than both the users that did system test 1. And by looking at the questions
connected to the concept of time usage, it seems as if the upgraded system image was somewhat
more correlated to the needs of wheelchair users than the original system image since both the
main time usage question as well as the check-question were rated as high or higher in system test
2 if compared to system test 1. This could give an indication that the input from the literature
(Bhatia et al. 2006), as well as listening to the participants of the interview did provide some
improvements from a user-perspective as well as ideas on how to proceed with the wheelchair
project. However it needs to be stressed that these indications are connected to possible and
probable bias coming from both the validity of the concept (time usage), as well as bias coming
from the low amount of test-participants taking part in the different tests and at the interview.
As a side note, it is also possible that the fact that the system images differed from each other, in
one consisting of slide-shows and the other of a prototype, do add bias to the results since it might
be harder to understand a mere representation of a system (the slide-shows) than a working system
(the prototype).
3.2.3 Analysis of time usage concept
Analyzing the results and the concept of time usage, and it is hard to tell if the method really
works and that if it shows what it is supposed to show. This mainly since the amount of
participating test-subjects was low. It is probable that a better understanding and more feedback on
the concept of time usage could have been retrieved if more test-subjects would have been
involved in the tests. With more test-subjects, it might have been possible to distinguish patterns in
the results that could have made it easier to understand the impact of the different possible sources
of errors and troubles the concept of time usage faces (as discussed in 2.2.2.4). But since the
66
Results
amount of users was low it is hard to distinguish anything at all about the method besides that, in
this case, it did not produce any results that could be used to determine the validity of the method,
and nor did it produce a statistical comparison between the two tests.
What can be said from my point of view is that it generally was hard to transform something that
usually can be regarded as qualitative data into quantitative data. And by this I mean the process of
trying to understand the qualitative aspects and things that need to be considered from a user point
of view given a certain situation and at the same time finding a way of capturing these aspects
within a quantified measure. This process included parts such as: formulating questions that
wouldn't be misread or misunderstood, as well as minimizing the impact from other sources of
errors on the time usage measure (the impact of the easiness of the user-interface for example).
Therefore if anyone would like to continue developing this new method, I urge for further
experimentation to conclude whether or not the method is suitable and eligible for labeling how
well a system or its functions correlate to the different needs users have. However, this further
experimentation are out of the scope for this master's thesis.
67
Sources of error – participation
4 Sources of error – participation
As discussed earlier in this report, each one of the tests and surveys had their own specific
problems that might have had an affect on the results. But maybe the biggest problem that this
project faced, was the lack of people doing the actual tests. Since this “lack of people” affected the
results to the extent that the subgoal with this project failed (the statistical comparison between the
tests), as well as generally heightened the possibility of bias in the results, I see it fitting to
describe a bit more about some of the reasons behind this. And in my opinion there were mainly
three different reasons:
●
●
●
The communication between involved parties
The tests time-span
A low amount of test-participants
I will below address these three sources of error, address some reasons to these errors and possible
measures I could have taken to partly prevent them and then finally discuss these errors effects on
the results in general.
4.1 The communication between involved parties
Firstly, the communication between involved parties (the universities and the organizations) took a
long time since there were more people involved in the communication process than expected.
Also since this lead to the fact that there were more links to go through people-wise, the
information about the test conditions and goals could have been altered along the way. So
basically it took time to confirm that the tests actually had user groups that would try the tests, to
agree on what the tests were about and to regularly communicate around the tests. And even after
agreeing on what the test were to be about still there might have been some confusion about what
the tests really were meant to show.
4.2 The tests time-span
Secondly, the time-span between the moment the tests came online (specially system test 1) and
until the data came in from the tests, were too long and affected the evaluation and the general
project-process. As mentioned earlier, system test 1 was online for more than 5 weeks, just waiting
for data to come in.
4.3 A low amount of test-participants
Third, a low amount of test-participants for both tests, was probably one of the reasons behind the
68
Sources of error – participation
not so giving quantitative results, and if more people had taken the tests the qualitative results
could have become richer and possibly even more descriptive.
4.4 Reasons and possible measures
To better understand how these three sources of error could have been the cause of the varying
results I will now address them again more thoroughly and also try to say something about
measures I could have taken if I had known the outcome on the results.
4.4.1 Reasons behind “the communication between involved parties”
Talking about the communication between involved parties, and it is hard to “pin-point” any
particular reasons behind the waiting and the possible occurred confusion about the tests, but one
reason (besides the amount of people involved in the communication process) might have been the
lack of regular communication between the involved parties, which in this case means staff at the
university, contact-persons at the organizations as well as people from the user groups. This
“waiting time”, of course, acted as a “stopper” for the general continuation of the project, taking
time from other processes, which in turn affected the overall research and results.
The possible confusion about the tests and their goals could also have affected the quality of the
results. If it was known that the communication process would take the time it took, I could have
urged for a faster and more regular communication between the parties. This by setting up a
meeting or two with as many involved parties or people I possibly could get in contact with before
the first test (and in between tests). By doing this, maybe all involved parties would have gotten a
better understanding of what I wanted to accomplish with the tests, and more importantly, earlier
in the process. Setting up meetings and a more regular communication might have provided faster
feedback as well as fastened up the project and its processes to give room and space for processes
that now were somewhat shortened. This might even have encouraged involved parties to check
the status and the evolution of the project more often, which in turn would have made them more
engaged in the project.
4.4.2 Reasons behind “the tests time-span”
When discussing the time-span the tests were online and the problems that arose from this fact, it
should firstly be mentioned that during the time for system test 1, there was a holiday season
taking place (Christmas and new year) which surely did affect how much communication went
through to the user-group at the time. Furthermore since the project had different deadlines, for
instance, I had to return to Sweden after my project time was finished, there was no choice but to
move on even though the retrieved data weren't too satisfactory.
Regarding system test 1, it was somewhat unfortunate that the test came online as late as it did,
and even though the original idea was that users would have time to answer the questions during
69
Sources of error – participation
the holiday season, this didn't happen. Possibly due to miscalculation on users willingness to
answer questions during the holiday season, but probably also due to the problem with the
communication between involved parties as described above. In fact it wasn't clear if the users
doing system test 1 had gotten the information about the test start before the holiday season begun
at all! Since all processes are connected in a project, the results of waiting for the results from the
tests affected how much time and effort could be spent on other relevant processes within this
research-project. As mentioned I had to draw a line when a process had to be stopped to give room
for other processes, and that is also what I did. Still when looking back, maybe more reminders
could have been given to the test-groups, asking them to take the tests. Even though this is a tricky
issue, since being too “pushy” about something could irritate people and make them loose their
goodwill, maybe an extra email or two could have been sent out. So a bit of bad timing,
miscalculations and slow communication with involved parties might have prolonged the time the
tests were online and contributed to the lack of people doing system test 1.
4.4.3 Reasons behind “the low amount of test-participants”
The third reason, the low amount of test-participants could depend on either communication
difficulties when trying to reach the users, as discussed above in 4.1, resulting in a low amount of
participants due to users not being aware of the tests and when they were online, or on the
possibility that there was a low amount of users able to do the tests, or wanting to do the tests, in
the two different test-groups to start with. If the latter was the main issue (a low amount of users
able to do the tests or wanting to take the tests ), then this could be the result off three different
things.
●
●
●
Uninterested users
Sickness or unavailability
Low number of users to start with
4.4.3.1 Uninterested users
Firstly its a possibility that many of the users didn't want to take the tests and thus didn't. If this
was the case, then the organizations used to get to the users didn't really answer for the majority of
the users opinion regarding the tests and the project in general. By keeping a closer and more
regular contact with the organizations and by lobbying for the tests (with the help of the
organizations) this might have been solved. However during the project there were no indications
to if users were uninterested in taking the tests.
4.4.3.2 Sickness or unavailability
The second possibility states that many of the users were sick or unavailable during the time for
the tests. If users were temporarily unavailable during the time for the tests, this could have been
detected through better communication, and maybe I should have stressed for more meetings with
the parties involved, as discussed earlier. Maybe then the launch of the tests could have been better
70
Sources of error – participation
correlated to the availability of the users. Worth to mention though is that, during the entire
project, there was no indications on that the availability of the people in the test-groups changed.
So its probable that the availability of the users in the test-groups didn't change remarkably and
thus didn't affect the amount of participants doing the tests.
4.4.3.3 Low number of users to start with
Finally, if there was a low number of users in the test-groups able to do the tests to start with, then
getting information about this could have made me reconsider my choice of method to, maybe, try
to go with a more interviewed based research method from the start even though the difficulties I
would have faced with such an option, as mentioned earlier in the report, would have been many at
the time. During the entire project the amount of wheelchair users belonging to the different
organizations, which were actually capable of doing the tests, weren't discussed in depth.
Therefore I made assumptions on the amount of users capable of taking the tests based on the
number of members the organizations had. If it had been possible to keep a closer contact with the
organizations, maybe better estimations on the amount of people eligible for taking the tests could
have been made.
To conclude, if more contact had been possible with the user groups and all involved parties in
general, faster feedback on the methods and the layout of system test 1 and 2 could have been
retrieved and maybe better, and easier tests could have been constructed.
4.5 General effects on the results
The main effects that could be seen in the results due to the lack of participation is bias as well as
lack of richness in the data. The issue about bias is affecting both the qualitative and quantitative
investigation. The issue about richness is mostly affecting the qualitative data alone.
As discussed in 3.2.3, the quantitative investigation with the concept of time usage could not really
produce any fruitful or meaningful results mostly due to the fact that the participation was low.
Therefore as mentioned, the quantitative indications put forth in the result section should be taken
lightly.
Furthermore the richness of the qualitative data might have suffered due to the low participation in
the tests. If more people had participated, the qualitative data could possibly have been richer in
content and also more descriptive. Even though qualitative methods do not require as many testparticipants as quantitative methods to gather valuable data, it is possible that if more participants
had given their input to the matter, the data from the tests would have contributed to an even better
understanding of users' needs and wants, and by that indirectly the possibility to create ideas for a
even better system from a user-perspective. Also, the lack of participation in both tests and at the
interview did probably lead to some bias in the results since as discussed before, users are different
and it is hard to correctly generalize what needs and wants the majority of the population has from
a very limited amount of test-subjects.
71
Analysis & conclusion
5 Analysis & conclusion
As mentioned, the goal of this project was to find ideas for how the continued system development
would satisfy the end-users' needs (wheelchair users), and then mainly in terms of what
functionality the system has to offer towards the users.
To help reach that goal, this project also had a subgoal concerning the statistical comparison
between two web-based test.
What should be said first, is that the ideas brought forth by the investigation would probably be
richer in content, statistically more correct and even more descriptive, if more users would have
participated in the tests and at the interview session.
To summarize, the subgoal failed since it was not possible to show a statistically significant
difference or indifference between the two different system images presented in the two different
test, system test 1 and system test 2. This due to a small number of participants, which resulted in a
small sample size. As a side note, this was also the main reason to why the method used, the
experiment with checking time usage, could not be proven to be a valid or invalid method of
choice.
However, some qualitative ideas from the qualitative investigation of both system images exposed
in the two system tests were retrieved. Also, the interview session with the Heart Net Waffle
organization brought ideas that may be useful when continuing with the future system
development process.
These ideas and indications from the different tests and particularly from the interview session
might, even though the number of participants were low, still prove to be useful for the future
system development process and might help shape the wheelchair project to become a system
better correlated to the needs of wheelchair users in the future. These ideas and indications can be
divided into three different areas, with a final section describing some possible ideas for
implementation of these indications and ideas, as described below:
●
●
●
●
Methodologies for the future system development process
Needs and wants
The interface.
Design ideas
5.1 Methodologies for the future system development process
First of all it can be deduced from the results that finding suitable users representing the end users,
was hard. Since the subgoal of making a statistical comparison between the two system test mainly
failed because of this fact, the author recommend future developers to tend to their contacts and
72
Analysis & conclusion
nourish the relationship with them. In other words take care of the relation with the user groups
representing the end users since it could prove hard to find replacement user groups.
As shown by the results from the two web-based tests, it is hard to formulate open questions that
provide a high amount of validity and reliability, which is also discussed in Bell (2000). Also the
results show that with the limited amount of users participating in the test, it was hard to retrieve
qualitative information with the remote usability technique that was used. Also the results from the
interview session gave more qualitative and useful input than both of the two remotely governed
tests. Therefore to minimize the work load for the developers and heighten the possibility to
retrieve good qualitative results from users in the future on how the users view the system and
what the users would like to see implemented in the system in terms of functionality, interview
sessions with story-boards or prototyping could prove to be a good alternative in the near future of
the system development process (Göransson and Gulliksen 2000). This close collaboration with
the users should be continued during the whole development process to increase the correlation
between the users needs and wants and what the system delivers in terms of functionality and
general usability. To further emphasis this idea of collaborating with users, there are also some
small indications from the quantitative comparison between the tests showing, that implementing
ideas from users and listening to users could improve a system to be more correlated to the needs
of real wheelchair users. Though it could be mentioned again that these indications should be
taken lightly since the method of choice as well as the low amount of participants could have
contributed to bias in the results as discussed earlier in the report.
Even though the individual that did do system test 2, did not fit the description in being a
wheelchair user, disabled people might still be able to benefit from using the system (the software
part, which was the focus of this project) as well as provide valuable input that might in a sense,
also apply to other users of the system, such as “real” wheelchair users. Furthermore there is not
one conclusion in the analysis section based solely on the results from system test 2. Regarding a
usability issue in the interface, a conclusion deduced from system test 2 has some influence, but as
discussed in Krug (2006), users tend to act similarly in certain situations even though they are
different.
5.2 Needs and wants
Even though there are a lot of different types of wheelchair users with a lot of different needs,
some results were gathered during the two tests and interview session that might represent the
needs and wants that possible end users to this system (the wheelchair project) have.
What could be deduced from the interview and the qualitative results from system test 1 and
system test 2 was, among other things, that going to the toilet, finding information about toilets
and what services they provide as well as finding parking spots are all time consuming endeavors
for the users. Also finding new lunch restaurants is something that was considered hard, and
sharing information about interesting accessible restaurants between users was something that was
regarded as an interesting feature. More investigation on how to implement functionality to see to
these above mentioned needs could be conducted in the future system development process.
73
Analysis & conclusion
Generally speaking, receiving information about sites and there accessibility toward wheelchair
user is interesting. For example interest exists in finding out if a gas station is manned or if an
elevator can be considered to be accessible for wheelchair users. So in general knowing how
accessible different sites are for wheelchair users is something that could be presented as a need to
which a solution could be found.
Both results from the interview session and system test 2 show that giving access to static
information on different sites is an important service that all wheelchair users could benefit from.
This since wheelchair users themselves best know how accessible a site is if presented with the
sites' static information (height, width etc) as discussed in section 3.1.2. So the wheelchair
project's original plan of providing users with static information should not be abandoned.
Since the results from the interview show that the ability to share information between wheelchair
users was generally considered to be interesting for all the above mentioned needs and wants, this
should be further investigated when the system development process continues.
Finally, even though it seems as the user interface and the usage of words in the
“register/unregister”-function (from the prototype) could need some make over, it can be deduced
from the interview session that this type of function is something that the users find relevant. Also
the ability to register and unregister to important zones of interest has been proven to be interesting
in a previous similar study (Bhatia et al. 2006), therefore it is worth investigating this possible
function more in the future development process.
5.3 The interface
Since interface design is not the focus of this project, this part will be brief. First of all as
discussed in most HCI literature it can be hard to distinguish between a systems usability and the
usability of that systems interface. During the course of this project only a system image was
presented to the participants of the different tests, but still some observations could be done
regarding the usability of the user interface.
It was learned from system test 2 and the prototype that the map and how it is structured could be
subject to further research. Since system test 2 did represent the map similar to how the actual
system represents the map, it might be the actual representation of the map and how it is used that
is something that should be subject to further analysis.
Results from system test 2 as well results presented in Bhatia et al. (2006), show that using a treelike structure when presenting functions and information, might ease the task of navigating
through the system and the functions the system provides. Arranging functions, and the way they
are accessed, into a tree-like structure could be subject to further investigation when continuing
with the system development process.
74
Analysis & conclusion
5.4 Design ideas
Some design ideas based on the conclusion and analysis done in the sections above (sections 5.2
and 5.3) are presented here below .
5.4.1 Tree structure
As mentioned, it could prove to be a good idea to organize the buttons a user utilizes to navigate
through the different functions, in a tree-like structure. In the wheelchair project this idea could be
implemented by, for example letting the altering information section and viewing information
section with underlying functions represent two branches of the tree with underlying functions
represented as nodes, or leaves in the tree. When accessing a specific function in either of these
two categories, it will be easy for the user to orientate himself, and determine where he or she is.
General HCI principles should be considered when taking on the task of designing this treestructure. For example, if the underlying software structure of the system does not support a user
accessing a function (a leaf) when in particular state (say when in the deepest part of the altering
information section and want to reach the inner parts of the viewing information section), by
greying out the nodes or possibly not even show them, in correlation to Norman's fifth design
principle, “Exploit the power of constraints, both natural and artificial”, the user might have an
easier time when navigating through the system (Norman 2002).
In figures 5.1 and 5.2, I am showing a rough example of how the prototype might have looked like
if it was designed using a tree structure. To make the figures even more clear, Function block 1
could for example represent the “Add information”-section. The circles are present for mere
highlighting purposes.
75
Analysis & conclusion
Figure 5.1: Tree-structure, example 1
Figure 5.2: Tree-structure, example 2
76
Analysis & conclusion
5.4.2 Post information function.
As shown by the results from the interview, from the qualitative part of the results from system
test 2, as well as put forth by Bhatia et al. (2006), the ability to post information to different places
in a system like the wheelchair project could be interesting for the users. By incorporating the
possibility to comment on every information spot (restaurants, toilets etc), it would be easier to
keep sites updated as shown in Bhatia et al. (2006). More interaction between users in general
might then also be possible since users will be posting comments for other users. My suggestion is
to design a function similar to the “information posting”-function used in the prototype, since this
function did not only allow the users to first see the most recent posted message, but did also
provide the users with a time stamp making it easier to estimate the importance of the message, as
discussed earlier in the report. Figures 5.3 and 5.4, shows the prototype and an example of the
“information posting”-function.
Figure 5.3: “Information posting”-function, example 1
77
Analysis & conclusion
Figure 5.4: “Information posting”-function, example 2
5.4.3 Descriptive information input
The results from the interview and from the qualitative results from system test 2 show that,
having access to static information as well as information regarding different sites accessibility is
crucial to wheelchair users and seemingly to people with disabilities in general. My suggestion is
to incorporate static information regarding dimensions and such, pictures, as well as information
regarding personnel into the information-spots in the wheelchair project. This information
consisting of text and pictures could be embedded into the layout of the different locations on the
map.
By for example adding the dimensions of an elevator to an elevator-location, users of the system
can by themselves decide whether that elevator is accessible to them or not. If possible, pictures
can be used in combination to this static information. Seeing pictures of for example an elevator,
might ease the task of judging whether or not that elevator is accessible for a particular user.
By adding information about personnel, users can better understand if a specific location is
accessible to them or not. Receiving information about a restaurant having personnel trained to
handle people with disabilities or if a gas station is unmanned, is regarded as interesting as
discussed earlier in this report.
78
Analysis & conclusion
In figure 5.5, using the prototype as a basis, I show a rough example of how this static information
could look when implemented.
Figure 5.5: Static information, example 1
79
Analysis & conclusion
5.4.4 E-chat function
Results from the interview indicate that users would like to see a forum or an email & chatfunction integrated into the system, which would connect users of the wheelchair project. This
since other wheelchair users might have answers to certain questions a specific user might come to
face. Wheelchair users, or disabled people, understand the situation and the needs of other
wheelchair users and disabled people the best. Therefore this knowledge should be used. By
adding an email-function with a built in chat-function16, users could both reach each other
asynchronously and synchronously. Also, by adding the possibility of reaching a specific forum
from this new function, the possibility of discovering new users and their knowledge would
emerge for the users of the system. A suggestion would be to put in a new button connected to this
e-chat function that would always be reachable throughout the whole interface. Making a possible
contact and a solution to the users' problems, to be only a few clicks away.
In figures 5.6 and 5.7 I give a rough example of how this e-chat function might look like if
implemented.
Figure 5.6: E-chat function, example 1
16 See for example Gmail. Available at: http://www.gmail.com, [last visited 2007-11-28]
80
Analysis & conclusion
Figure 5.7: E-chat function, example 2
5.4.5 Subscription function
As mentioned, the interview and discussions in Bhatia et al. (2006) shows that users find it
interesting to subscribe to certain types of information. My suggestion would be to incorporate a
subscription-function similar to the “register/unregister”-function exposed in the prototype of
system test 2. By registering to a subscription a user can customize his or her profile, making it
faster for the user to access specific relevant information. If for example the daily routine of a user
involves going to a lunch restaurant, it would be relevant to know if the closest situated toilet with
respect to the restaurant is “up and running”. Instead of navigating through the system every day to
locate this particular toilet's information page, the user could choose to subscribe to the toilet by
adding the toilet's information page to the user's “hot-list” or list of subscriptions, providing faster
access to that particular piece of necessary information. As mentioned, the layout of this
subscription-function could resemble the “register/unregister”-function as shown in the prototype
where the list of subscriptions was accessible at all times. From my own point of view, it is
essential that such a subscription list will be accessible from practically everywhere in the layout
due to the main purpose of a hot-list is being to provide easy access to different types of
information.
The name of this subscription-function as well as the words used for the different attributes within
the function should be further examined through iterative collaboration with user-test groups
81
Analysis & conclusion
though, since results from system test 2 show that the name and words used in the prototype were
not fully understandable with the test subjects.
5.5 Some final words
Since all wheelchair users differ from each other in terms of their needs and personalities so might
also the hardware (and sometimes also the software) that assist them when using the system. Even
though it is hard to design optimal solutions for all users, using general HCI-principles and models
combined with important data from user tests, a solution more than acceptable for most users
could be constructed. Other studies could also be examined to provide further guidance on how to
improve the system for users with different needs, see for example Bhatia et al. (2006), Hwang
(2001, 2002) or Steriadis and Constantinou (2003). Still the value of user testing and keeping a
close collaboration with test subjects representing the end users should not be neglected when
continuing with the development of the wheelchair project.
82
List of references
List of references
Bell, J., 2000. Introduktion till forskningsmetodik, 3e upplagan, Lund:
Studentlitteratur AB, ISBN 91-44-01395-7.
Beyer, H., & Holtzblatt, K., 1998. Contextual design: defining customer-centered
systems, San Diego: Academic Press, ISBN 1-55860-411-1.
Bhatia, S. et al., 2006. 'VTAssist: a location-based feedback notification system for the disabled',
ACM Southeast Regional Conference, Proceedings of the 44th annual southeast regional
conference, SESSION: Agents, interactions and mobility II, pp. 512 - 517 , New York, ACM
Press, ISBN 1-59593-315-8.
Blom.G., 2005. Sannolikhetsteori och statistikteori med tillämpningar,
Lund: Studentlitteratur, ISBN 9144024428.
Göransson, B., & Gulliksen, J., 2000. Användarcentrerad systemutveckling,
version 1.0, Stockholm: KTH,TRITA-NA-D0005, CID-71, ISSN 1403-073X.
Hwang, F., 2002. 'A study of cursor trajectories of motion-impaired users', Conference on Human
Factors in Computing Systems, CHI '02 extended abstracts on Human factors in computing
systems, SESSION: Student Posters, pp. 842 – 843, New York, ACM Press, ISBN 1-58113-454-1.
Hwang, F. et al., 2001.'Perception and haptics: towards more accessible computers for motionimpaired users', ACM International Conference Proceeding Series; Vol. 15, Proceedings of the
2001 workshop on Perceptive user interfaces, SESSION: Paper session #3, pp. 1 – 9, New York,
ACM Press.
83
List of references
Krug, S., 2006. Don't make me think: a common sense approach to web
usability, 2nd edition, California: New Riders Publishing, ISBN 0-321-34475-8.
Norman, A.D., 2002. The design of everyday things (Paperback), Jackson: Basic Books, ISBN-10
0-465-06710-7
Mackay, W.E. & Fayard, AL., 1997. 'HCI, natural science and design: a framework for
triangulation across disciplines', Symposium on Designing Interactive Systems Proceedings of the
conference on Designing interactive systems: processes, practices, methods, and techniques, pp.
223 – 234, New York, ACM Press, ISBN 0-89791-863-0.
Palanque, P. & Bastide, R., 1996. 'Time modelling in Petri nets for the design of interaction
active', ACM SIGCHI Bulletin, Volume 28 , Issue 2 (April 1996), pp. 43 – 46,
New York, ACM Press, ISSN 0736-6906.
Petri, H. et al., 2006. 'Remote usability evaluations With disabled people',
Conference on Human Factors in Computing Systems, Proceedings of the
SIGCHI conference on Human Factors in computing systems, pp.
1133 – 1141, ISBN 1-59593-372-7.
Preece, J., Rogers, Y. & Sharp, H., 2002. Interaction design: beyond human computer
interaction , New York: John Wiley & Sons, Inc, ISBN 0-471-49278-7.
Spool, J.M. et al., 1999. Web site usability, a designers guide, San Francisco:
Morgan Kaufmann Publishers, Inc, ISBN1-55860-569-X.
Srivastava, J., Cooley, J., Deshpande, M. & Tan, P-N., 2000. 'Web usage mining: discovery and
applications of usage patterns from Web data', ACM SIGKDD Explorations Newsletter , Volume 1
, Issue 2 (January 2000), pp. 12 – 23, New York, ACM Press, ISSN 1931-0145.
Steriadis, C.E. & Constantinou, P., 2003. 'Designing human-computer interfaces for quadriplegic
people', ACM Transactions on Computer-Human Interaction (TOCHI), Volume 10, Issue 2 (June
2003), pp. 87 – 118, New York, ACM Press, ISSN 1073-0516.
Williams, H.E. & Lane, D., 2004. Web Database Applications with PHP &
MySQL, 2nd edition, Sebastopol: O'Reilly Media, ISBN 059-600-5431.
84
Appendices
Appendices
Appendix A — Questions used in survey 1
The questions in survey 1 were divided into three categories. First came questions about the user,
then questions about the simulated system (the slide shows) and then finally questions concerning
the functions within the simulated system. The abbreviation (mc) stands for multiple choice
alternative, and the abbreviation (text) was put there to indicate that the question held a text area
for the user to write input in.
Questions 9, 10, 12 and 13 were time usage questions. Questions 9 and 10 dealt with the system
image as a whole, whilst 12 and 13 were meant to get data from the nine different functions
presented in the system image.
Question 1: Who are you? (mc)
1. Assistant
2. Wheelchair user
3. Other
Question 2: Your gender? (mc)
1. Male
2. Female
3. Other
Question 3: Your age? (mc)
1.
2.
3.
4.
5.
51 or over
41-50
31-40
21-30
20 or under
Question 4: When you are out near your home, in your everyday life, what do you usually do?
(text)
Question 5: What daily business is the most time consuming for you? (text)
Question 6: If you could have daily access to information, what kind of information would ease
your daily life? (text)
Question 7: What is your opinion about sharing information between wheelchair users? (text)
85
Appendices
Question 8: Is it easy to understand how to use this system?(mc)
1.
2.
3.
4.
5.
Very easy
Easy
Quite easy
Not really
No, it was hard
Question 9: How often do you think that this system would be helpful to you in your everyday
life?(mc)
1.
2.
3.
4.
5.
A number of times every day
Every day
Three or four times a week
Once a week
Once a month or less
Question 10: How often would you use this system?(mc)
1.
2.
3.
4.
5.
A number of times every day
Every day
Three or four times a week
Once a week
Once a month or less
Question 11: What kind of new function would you like to see implemented into the system? (text)
Question 12: How often do you think that the functions listed below would be helpful to you in
your everyday life? (mc)
1.
2.
3.
4.
5.
A number of times every day
Every day
Three or four times a week
Once a week
Once a month or less
Question 13: How often would you use the functions listed below?(If you did not have to use it
in order for other functions to work? (mc)
1.
2.
3.
4.
5.
A number of times every day
Every day
Three or four times a week
Once a week
Once a month or less
Question 14: Do you have any additional comments about this system? (text)
86
Appendices
Appendix B — Questions used in the interview with Heart Net Waffle
Since the interview were more informal than formal in its nature, there weren't an insanely large
amount of predeclared questions asked during the interview. Still the interview situation needed a
plan or structure to follow. The questions used for this “structuring purpose”, were mainly taken
from the questions that were to be used in survey 2. To all of these below posted questions, followup questions deduced from the answers the interviewed provided, were asked to ensure that the
information retrieved were accurate but also, more importantly, to add qualitative value to the
information the users provided.
Question 1: When you are out of your home.
What specific things do you usually do on a daily basis that requires more effort?
Question 2: When you are out of your home.
What daily business of yours is time consuming for you?
Question 3: If you could have daily access to information about specific places in the city you live
in, what kind of information would ease your daily life? For example would you like to know if a
certain handicap toilet is out of toilet paper, or information about if an elevator at a train station is
functional?
Question 4: What is your opinion about sharing information between wheelchair users in an
information system such as this being developed?
Question 5: If you could design a new function to the system,what would that function do?
87
Appendices
Appendix C — Questions used in survey 2
The questions in survey 2 were divided into two different categories as described in the report.
Questions about the users and questions about the system. These two types of questions were
mixed with each other in the survey to minimize the risk of users getting tired of answering the
questions and maximizing the chance of retrieving valuable information as described in the report.
The abbreviation (mc) stands for multiple choice alternative, and the abbreviation (text) indicates
that the users were presented with a blank text area to provide input for the answer. In questions 1
and 2 the users were presented with both multiple choice alternatives as well as a text area so this
is displayed accordingly.
Question 1: When you are out of your home.
What specific things do you usually do on a daily basis that requires more effort?
1.
2.
3.
4.
Going to the toilet(mc)
Looking for parking spots(mc)
Searching for lunch restaurants(mc)
Other (text)
Question 2: When you are out of your home.
What daily single event business is the most time consuming for you?
1.
2.
3.
4.
Going to the toilet(mc)
Looking for parking spots(mc)
Searching for lunch restaurants(mc)
Other (text)
Question 3: If you could have daily access to information about specific places in the city you live
in, what kind of information would ease your daily life? For example would you like to know if a
certain handicap toilet is out of toilet paper, or information about if an elevator at a train station is
functional?
(text)
Question 4: What is your opinion about sharing information between wheelchair users in an
information system such as this being developed?
(text)
Question 5: Who are you? (mc)
1. Assistant
2. Wheelchair user
3. Other
Question 6: Your gender? (mc)
88
Appendices
1. Male
2. Female
3. Other
Question 7: Your age? (mc)
1.
2.
3.
4.
5.
51 or over
41-50
31-40
21-30
20 or under
Question 8: Is it easy to understand how to use this system?(mc)
1.
2.
3.
4.
5.
Very easy
Easy
Quite easy
Not really
No, it was hard
Question 9: How often do you think that this system would be helpful to you in your everyday
life?(mc)
1.
2.
3.
4.
5.
A number of times every day
Every day
Three or four times a week
Once a week
Once a month or less
Question 10: How often would you use this system?(mc)
1.
2.
3.
4.
5.
A number of times every day
Every day
Three or four times a week
Once a week
Once a month or less
Question 11: What was the most useful function, in the prototype, and why is that?
For example was it the ability to search for different sites on the map, like search for restaurants?
Was it the ability to subscribe to information? Or maybe you thought it was the ability to post
information to these different sites?(text)
Question 12: What was the least useful function, in the prototype, and why is that?
For example was it the ability to search for different sites on the map, like search for restaurants?
Was it the ability to subscribe to information? Or maybe you thought it was the ability to post
89
Appendices
information to these different sites?(text)
Question 13: If you could design a new function to the system,what would that function do?
(text)
Question 14: Do you have any additional comments about the system that was simulated?
(text)
90
Appendices
Appendix D — Student's t-test
Figures 117 and 218 are taken from the web. Figure 3 is based on the formulas presented in figures 1
and 2, and was made by me in March 2007.
Figure 1: Student's t-test, unequal sample sizes
Figure 2: Standard deviation
Df =n 1n 22
Df = Degrees of freedom
n 1=amount of samples ∈ group 1
n 2=amount of samples ∈ group 2
s12=squared sample of standard deviation for group 1
also known as an unbiased estimator of the variance , 2 for group 1
2
s 2=squared sample of standard deviation for group 2
2
also known as an unbiased estimator of the variance , for group 2
X 1=Mean value of samples ∈ group 1
X 2=Mean value of samples ∈ group 2
x=mean value of samples ∈ corresponding group
x i=ith x value ∈ corresponding group
t =tvalue
Figure 3: Definitions
17 http://en.wikipedia.org/wiki/Student%27s_t-test, [cited 2007-02-10]
18 http://en.wikipedia.org/wiki/Standard_deviation, [cited 2007-02-10]
91
Appendices
Appendix E — Quantitative results displayed in tables
This appendix shows all quantitative data gathered from System test 1 & System test 2 in tables.
All tables (1, 2, 3) were done by me in November 2007.
Demographic
questions.
Results
System test 1
System test 2
User 1
User 2
User 3
Question 1:
Who are you?
W
W
O
Question 2:
Your gender?
M
F
F
Question 3:
Your age?
2
2
5
Alternatives Question 1:
W = Wheelchair user
A = Assistant
O = Other
Alternatives Question 2:
M = Male
F = Female
O = Other
Alternatives Question 3:
5 = 51 or over
4 = 41-50
3 = 31-40
2 = 21-30
1 = 20 or under
Table 1: Demographic questions. Results
Interface's
usability question.
Results
Question:
Is it easy to understand how
to use this system?
System test 1
System test 2
User 1
User 2
User 3
2
2
4
Alternatives:
5 = Yes, it is very easy
4 = Yes, it is easy
3 = Quite easy
2 = Not really
1 = No, it was hard
Table 2: Interface's usability question. Results
92
Appendices
System test 1
Time usage questions.
Results
System test 2
User 1
User 2
User 3
Question 1:
How often would you use
this system?
2
1
3
Question 2:
How often do you think that
this system would be helpful
to you in your everyday
life?
2
1
2
Alternatives:
5 = A number of times every day
4 = Every day
3 = Three or four times a week
2 = Once a week
1 = Once a month or less
Table 3: Time usage questions. Results
93
TRITA-CSC-E 2008: 100
ISRN-KTH/CSC/E--08/100--SE
ISSN-1653-5715
www.kth.se