Evaluating Intelligent Knowledge Systems: Experiences with a User-Adaptive Assistant Agent

Pauline M. Berry, Thierry Donneau-Golencer, Khang Duong, Melinda Gervasio, Bart Peintner, Neil Yorke-Smith

Research output: Contribution to journalArticleScientificpeer-review

4 Citations (Scopus)


This article examines experiences in evaluating a user-adaptive personal assistant agent designed to assist a busy knowledge worker in time management. We examine the managerial and technical challenges of designing adequate evaluation and the tension of collecting adequate data without a fully functional, deployed system. The CALO project was a seminal multi-institution effort to develop a personalized cognitive assistant. It included a significant attempt to rigorously quantify learning capability, which this article discusses for the first time, and ultimately the project led to multiple spin-outs including Siri. Retrospection on negative and positive experiences over the 6 years of the project underscores best practice in evaluating user-adaptive systems. Lessons for knowledge system evaluation include: the interests of multiple stakeholders, early consideration of evaluation and deployment, layered evaluation at system and component levels, characteristics of technology and domains that determine the appropriateness of controlled evaluations, implications of ‘in-the-wild’ versus variations of ‘in-the-lab’ evaluation, and the effect of technology-enabled functionality and its impact upon existing tools and work practices. In the conclusion, we discuss—through the lessons illustrated from this case study of intelligent knowledge system evaluation—how development and infusion of innovative technology must be supported by adequate evaluation of its efficacy.
Original languageEnglish
Pages (from-to)379-409
Number of pages31
JournalKnowledge and Information Systems
Issue number2
Publication statusPublished - Aug 2017
Externally publishedYes


  • Technology evaluation
  • Knowledge-based systems
  • Personal assistant agent
  • User-adaptive
  • Time management
  • CALO project

Cite this