Comparison of Questionnaire Based and User Model Based Usability Evaluation Methods

The usability now serves as a fundamental quality of a computational device, e.g. smartphone. Moreover, the smartphone has firmly embedded into our daily life as an indispensable part, so the context and style that user may interact with them are largely different from a decade ago. Nowadays, testing usability with end user has become a common sense. Thus, how valid a usability evaluation method could assess the ‘extent to which a product can be used by specified users’ (ISO 9241-11) to facilitate software design becomes an interesting question to explore. In this research, three usability evaluation methods are compared. Among these methods, IsoMetrics is a standard questionnaire aiming at offer usability data for summative and formative evaluation; SUMI aims to assess quality of software product from end users perspective; User Model Checklist is a method based on user’s cognition-motor chain in specific tasks. The coverage and amount of usability issues, user’s effort of evaluation and software developer’s feedback on evaluation result are compared under a simulated usability test on SMS function with a smartphone. The result indicate that User Model Checklist could cover 90.4% of the usability issues found by IsoMetrics and SUMI, while 26.3% usability issues found by User Model Checklist could not be covered by IsoMetrics and SUMI. Users put highest effort on accomplish IsoMetrics and lowest effort on User Model Checklist. Moreover, the feedbacks from the developers show that the User Model Checklist requires lower usability knowledge, offers clearer improvement points and supports detailed design better.