Designing a usability test - Revision history

Jjh at 18:28, 13 January 2008

2008-01-13T18:28:52Z

Jjh: /* Collecting test data */

2008-01-13T18:17:01Z

‎Collecting test data

Jjh: /* Collecting test data */

2008-01-13T18:15:24Z

‎Collecting test data

Jjh: /* NIACRO */

2008-01-13T17:39:21Z

‎NIACRO

Jjh at 17:38, 13 January 2008

2008-01-13T17:38:57Z

Jjh at 17:28, 13 January 2008

2008-01-13T17:28:53Z

Jjh: first stab

2007-09-21T17:15:15Z

first stab

New page

We set out to compare the usability of this map-based interface with the conventional paper
questionnaire produced by the Probation Board NI.47 Details of each stage can be found in
Appendix 11.2.

==Selecting the testers==
There is no point getting students to test the interface since what we wanted to know was
whether ex-offenders, some with low literacy levels, found the map-based interface more or less
usable than current consultation techniques. We needed to recruit ex-offenders to test the
system. The Northern Ireland Association for the Care and Resettlement of Offenders
(NIACRO) runs IT courses for ex-offenders and others. They have a small computer room in
which trainees can develop IT skills that may increase their employability. A number of
probationers take these courses. So NIACRO arranged for some of their trainees to test the
interface when they came in for their regular session on 13 April 2006.

==Selecting the tasks==
We designed a sequence of tasks that started with simple familiarization with the map-based
site, then got progressively more involved as the testers gained confidence, until they were
entering comments on the consultation topic. In brief they were:
· Explore the PBNI e-consultation website.
· Find your probation office or reporting centre on the map.
· Find the all offices or reporting centres you have been to, and add comments on each place.
· Find your new office and reporting centre, and comment on how it would affect you.
The control task was to complete as much of the PBNI questionnaire as they could in 15
minutes.

==Collecting test data==
Before the tests started, we asked testers to complete a questionnaire on their familiarity with
computers, the Internet and consultations. We set up Camtasia Studio on one PC. This software
produced a video of the screen, including the position of the mouse at any time, and what the
tester saw. The tester spoke aloud during the test, explaining what s/he was doing, and noting
any difficulties in using the interface. This was recorded on the same Camtasia video, using a
microphone plugged into the computer. In addition, the comments entered were automatically
stored on the server, so at the end of the session we copied the comments and deleted them from
the server. After completing the test tasks we asked the testers what they thought of the
interface, using the post-test questionnaire. We used a similar questionnaire to collect their
assessment of the control task, completing a paper questionnaire. Finally, we invited the testers,
in a group, to discuss their experiences in expressing their views through the computer map and
the paper questionnaire. We took notes of points raised.

@@ Line 1: / Line 1: @@
 We set out to compare the usability of this map-based interface with the conventional paper
-questionnaire produced by the [http://www.pbni.org.uk/ PBNI].
+questionnaire, [http://www.pbni.org.uk/annex-a-questionnaire.pdf viewable online], produced by the [http://www.pbni.org.uk/ PBNI].
 ==Selecting the testers==
-There is no point getting students to test the interface since what we wanted to know was whether ex-offenders, some with low literacy levels, found the map-based interface more or less usable than current consultation techniques.
+There is no point getting students to test the interface since what we wanted to know was whether ex-offenders, some with low literacy levels, found the map-based interface more or less usable than [http://www.pbni.org.uk/annex-a-questionnaire.pdf current consultation techniques].
 ====NIACRO====

@@ Line 31: / Line 31: @@
 After completing the test tasks we asked the testers what they thought of the interface, using the post-test questionnaire (See Figure ). We used a similar questionnaire to collect their assessment of the control task, completing a paper questionnaire.
-[[Image:|thumb|]]
+[[Image:PBNI_Postsurvey.png|thumb|Figure: Post-test questionnaire]]
 Finally, we invited the testers, in a group, to discuss their experiences in expressing their views through the computer map and the paper questionnaire. We took notes of points raised.  The questions asked in the group included:

@@ Line 29: / Line 29: @@
 The tester spoke aloud during the test, explaining what the user was doing, and noting any difficulties in using the interface.  This was recorded on the same Camtasia video, using a microphone plugged into the computer. In addition, the comments entered were automatically stored on the server, so at the end of the session we copied the comments and deleted them from the server.
-After completing the test tasks we asked the testers what they thought of the interface, using the post-test questionnaire. We used a similar questionnaire to collect their assessment of the control task, completing a paper questionnaire.
+After completing the test tasks we asked the testers what they thought of the interface, using the post-test questionnaire (See Figure ). We used a similar questionnaire to collect their assessment of the control task, completing a paper questionnaire.
-Finally, we invited the testers, in a group, to discuss their experiences in expressing their views through the computer map and the paper questionnaire. We took notes of points raised.
+[[Image:|thumb|]]

@@ Line 5: / Line 5: @@
 There is no point getting students to test the interface since what we wanted to know was whether ex-offenders, some with low literacy levels, found the map-based interface more or less usable than current consultation techniques.
-===NIACRO===
+====NIACRO====
 We needed to recruit ex-offenders to test the system. The [http://www.niacro.co.uk/ Northern Ireland Association for the Care and Resettlement of Offenders (NIACRO)] runs IT courses for ex-offenders and others. They have a small computer room in which trainees can develop IT skills that may increase their employability. A number of probationers take these courses. So [http://www.niacro.co.uk/ NIACRO] arranged for some of their trainees to test the interface when they came in for their regular session on 13 April 2006.

@@ Line 1: / Line 1: @@
 We set out to compare the usability of this map-based interface with the conventional paper
-questionnaire produced by the PBNI.
+questionnaire produced by the [http://www.pbni.org.uk/ PBNI].
 ==Selecting the testers==
-There is no point getting students to test the interface since what we wanted to know was
+There is no point getting students to test the interface since what we wanted to know was whether ex-offenders, some with low literacy levels, found the map-based interface more or less usable than current consultation techniques.
 whether ex-offenders, some with low literacy levels, found the map-based interface more or less
-usable than current consultation techniques. We needed to recruit ex-offenders to test the
+===NIACRO===
-system. The Northern Ireland Association for the Care and Resettlement of Offenders
+We needed to recruit ex-offenders to test the system. The [http://www.niacro.co.uk/ Northern Ireland Association for the Care and Resettlement of Offenders (NIACRO)] runs IT courses for ex-offenders and others. They have a small computer room in which trainees can develop IT skills that may increase their employability. A number of probationers take these courses. So [http://www.niacro.co.uk/ NIACRO] arranged for some of their trainees to test the interface when they came in for their regular session on 13 April 2006.
 ==Selecting the tasks==
 We designed a sequence of tasks that started with simple familiarization with the map-based
 site, then got progressively more involved as the testers gained confidence, until they were
-entering comments on the consultation topic. In brief they were:
+entering comments on the consultation topic.
-· Explore the PBNI e-consultation website.
-· Find your probation office or reporting centre on the map.
+In brief, the sequence of tasks were:
-· Find the all offices or reporting centres you have been to, and add comments on each place.
+* Explore the [http://www.pbni.org.uk/ PBNI] [http://www.pbni.org.uk/index/publications/consultation.htm e-consultation website].
-· Find your new office and reporting centre, and comment on how it would affect you.
+* Find your probation office or reporting centre on the map.
-The control task was to complete as much of the PBNI questionnaire as they could in 15
+* Find the all offices or reporting centres you have been to, and add comments on each place.
-minutes.
+* Find your new office and reporting centre, and comment on how it would affect you.
 ==Collecting test data==
 Before the tests started, we asked testers to complete a questionnaire on their familiarity with
-computers, the Internet and consultations. We set up Camtasia Studio on one PC. This software
+computers, the Internet and consultations.
 produced a video of the screen, including the position of the mouse at any time, and what the
-tester saw. The tester spoke aloud during the test, explaining what s/he was doing, and noting
+We set up [http://www.techsmith.com/camtasia.asp?CMP=KgoogleCStmhome Camtasia Studio], a computer activity recorder, on one PC. This software produced a video of the screen, including the position of the mouse at any time, and what the tester saw.
 any difficulties in using the interface. This was recorded on the same Camtasia video, using a
-microphone plugged into the computer. In addition, the comments entered were automatically
+The tester spoke aloud during the test, explaining what the user was doing, and noting any difficulties in using the interface.  This was recorded on the same Camtasia video, using a microphone plugged into the computer. In addition, the comments entered were automatically stored on the server, so at the end of the session we copied the comments and deleted them from the server.
 stored on the server, so at the end of the session we copied the comments and deleted them from
-the server. After completing the test tasks we asked the testers what they thought of the
+After completing the test tasks we asked the testers what they thought of the interface, using the post-test questionnaire. We used a similar questionnaire to collect their assessment of the control task, completing a paper questionnaire.
 interface, using the post-test questionnaire. We used a similar questionnaire to collect their
-assessment of the control task, completing a paper questionnaire. Finally, we invited the testers,
+Finally, we invited the testers, in a group, to discuss their experiences in expressing their views through the computer map and the paper questionnaire. We took notes of points raised.