Murdoch University Research Repository

Welcome to the Murdoch University Research Repository

The Murdoch University Research Repository is an open access digital collection of research
created by Murdoch University staff, researchers and postgraduate students.

Learn more

A black-box approach for response quality evaluation of conversational agent systems

Goh, O.S., Ardil, C., Wong, W. and Fung, C.C.ORCID: 0000-0001-5182-3558 (2007) A black-box approach for response quality evaluation of conversational agent systems. International Journal of Computational Intelligence, 3 (3). pp. 195-203.

PDF - Published Version
Download (604kB)


The evaluation of conversational agents or chatterbots question answering systems is a major research area that needs much attention. Before the rise of domain-oriented conversational agents based on natural language understanding and reasoning, evaluation is never a problem as information retrieval-based metrics are readily available for use. However, when chatterbots began to become more domain specific, evaluation becomes a real issue. This is especially true when understanding and reasoning is required to cater for a wider variety of questions and at the same time to achieve high quality responses. This paper discusses the inappropriateness of the existing measures for response quality evaluation and the call for new standard measures and related considerations are brought forward. As a short-term solution for evaluating response quality of conversational agents, and to demonstrate the challenges in evaluating systems of different nature, this research proposes a blackbox approach using observation, classification scheme and a scoring mechanism to assess and rank three example systems, AnswerBus,START and AINI.

Item Type: Journal Article
Murdoch Affiliation: School of Information Technology
Publisher: World Academy of Science Engineering and Technology
Copyright: ©
Item Control Page Item Control Page


Downloads per month over past year