Article details

Research area
Natural language & AI

. Poster presented at the ASIST’2007 Annual meeting


Hyekyung Kim and Lorna Peterson

Usability and effectiveness evaluation of a course-advising chat bot


This research will evaluate usability and effectiveness of a course-advising chat bot by comparing it with a text-based FAQs system in terms of functionality and interface. A chat bot is an interactive computer program that enables users to converse with a computer like a real person through natural language. Through natural language, the users can search information not only by keywords but with complete sentences with which they are familiar. Natural language facilitates the efficiency of expressions for novice or casual users of a system. In addition, chat bots can increase understanding of information by providing both text and voice responses.

For the test, a text-based FAQs system and a LIS course-advising chat bot will be created. The LIS course-advising chat bot, supporting new or current LIS (Library and Information Studies) students’ course-related questions, is created by using ‘Pandorabots’ and ‘Media Semantics Character Builder’. Unlike the FAQs system constructed with hyperlinks, the chat bot has an interactive interface to type questions and text as well as voice answering with an avatar.

This research will explore two aspects of the chat bot so that it could be used as an academic tool to address course-related questions. First, does the chat bot more effectively function than the text-based FAQs for academic advising? Second, does the interactive user interface of the chat bot positively affect the process of searching and understanding of information?

The usability test involves both qualitative and quantitative methods. The qualitative method will include the observation of participants during the task testing, analysis of performed tasks, log files and user feedback. Through three kinds of questionnaires, it will be collecting data for the following: Internet, chat program, and avatar experience, information-seeking attitudes, preference of the interface and functionality between two systems, usefulness, effectiveness, satisfaction, and the perception of the time to complete the tasks.

Read/download now