Catalog Home Page

A black-box approach for response quality evaluation of conversational agent systems

Goh, O.S., Ardil, C., Wong, W. and Fung, C.C. (2007) A black-box approach for response quality evaluation of conversational agent systems. International Journal of Computational Intelligence, 3 (3). pp. 195-203.

[img]
Preview
PDF - Published Version
Download (590kB) | Preview

    Abstract

    The evaluation of conversational agents or chatterbots question answering systems is a major research area that needs much attention. Before the rise of domain-oriented conversational agents based on natural language understanding and reasoning, evaluation is never a problem as information retrieval-based metrics are readily available for use. However, when chatterbots began to become more domain specific, evaluation becomes a real issue. This is especially true when understanding and reasoning is required to cater for a wider variety of questions and at the same time to achieve high quality responses. This paper discusses the inappropriateness of the existing measures for response quality evaluation and the call for new standard measures and related considerations are brought forward. As a short-term solution for evaluating response quality of conversational agents, and to demonstrate the challenges in evaluating systems of different nature, this research proposes a blackbox approach using observation, classification scheme and a scoring mechanism to assess and rank three example systems, AnswerBus,START and AINI.

    Publication Type: Journal Article
    Murdoch Affiliation: School of Information Technology
    Publisher: World Academy of Science Engineering and Technology
    Copyright: © www.waset.org
    URI: http://researchrepository.murdoch.edu.au/id/eprint/991
    Item Control Page

    Downloads

    Downloads per month over past year