Conversational AI 14-Evaluating Dialogue Systems

Evaluation is important for a number of reasons:

1. for developers it is important to determine whether the system performs as expected
2. for users it is important to determine whether the system meet s the user’s needs—whether it understands their utterances, whether in the case of task-oriented systems it helps them achieve their goals efficiently, and, in the case of non-task-oriented systems, whether it gives them an enjoyable experience and
3. for researchers it is important to establish whether the aims of the research have been met, for example, to validate a new research technique or to investigate whether the system shows improvement on various evaluation metrics against a baseline state-of-the-art system.