Comparison of Questionnaire Based and User Model Based Usability Evaluation Methods

Meng Li*, Armagan Albayrak, Yu Zhang, Daan van Eijk, Zengyao Yang

*Corresponding author for this work

Research output: Chapter in Book/Conference proceedings/Edited volumeConference contributionScientificpeer-review

1 Citation (Scopus)
83 Downloads (Pure)


The usability now serves as a fundamental quality of a computational device, e.g. smartphone. Moreover, the smartphone has firmly embedded into our daily life as an indispensable part, so the context and style that user may interact with them are largely different from a decade ago. Nowadays, testing usability with end user has become a common sense. Thus, how valid a usability evaluation method could assess the ‘extent to which a product can be used by specified users’ (ISO 9241-11) to facilitate software design becomes an interesting question to explore. In this research, three usability evaluation methods are compared. Among these methods, IsoMetrics is a standard questionnaire aiming at offer usability data for summative and formative evaluation; SUMI aims to assess quality of software product from end users perspective; User Model Checklist is a method based on user’s cognition-motor chain in specific tasks. The coverage and amount of usability issues, user’s effort of evaluation and software developer’s feedback on evaluation result are compared under a simulated usability test on SMS function with a smartphone. The result indicate that User Model Checklist could cover 90.4% of the usability issues found by IsoMetrics and SUMI, while 26.3% usability issues found by User Model Checklist could not be covered by IsoMetrics and SUMI. Users put highest effort on accomplish IsoMetrics and lowest effort on User Model Checklist. Moreover, the feedbacks from the developers show that the User Model Checklist requires lower usability knowledge, offers clearer improvement points and supports detailed design better.

Original languageEnglish
Title of host publicationProceedings of the 20th Congress of the International Ergonomics Association (IEA 2018) - Volume VII
Subtitle of host publicationErgonomics in Design, Design for All, Activity Theories for Work Analysis and Design, Affective Design
EditorsSebastiano Bagnara, Riccardo Tartaglia, Sara Albolino, Thomas Alexander, Yushi Fujita
Place of PublicationCham
Number of pages18
ISBN (Electronic)978-3-319-96071-5
ISBN (Print)978-3-319-96070-8
Publication statusPublished - 2019
Event IEA 2018: 20th Congress of the International Ergonomics Association - Florence, Italy
Duration: 26 Aug 201830 Aug 2018
Conference number: 20

Publication series

NameAdvances in Intelligent Systems and Computing
ISSN (Print)2194-5357


Conference IEA 2018: 20th Congress of the International Ergonomics Association
Abbreviated title IEA 2018

Bibliographical note

Accepted author manuscript


  • IsoMetrics
  • SUMI
  • Usability evaluation comparison
  • User Model Checklist

Cite this