Check also my Research Projects and Data Page
Jokinen, K. (2009). Constructive Dialogue Modelling: Speech Interaction and Rational Agents. John Wiley.
Jokinen, K. and M. McTear (2009). Spoken Dialogue Systems. Synthesis Lectures on Human Language Technologies. Morgan and Claypool.
My background is in cooperative dialogue planning and spoken dialogue systems, which I developed in the Constructive Dialogue Model framework in my thesis in 1994. The research was inspired by Jens Allwood’s model of Ideal Cooperation and Rational Agency, which I developed and extended into the CDM model in the context of a practical dialogue system in the ESPRIT-II project PLUS (Pragmatic Language Understanding System). The main innovation of the research is to use rationality and cooperative principles to construct a shared dialogue model within which to resolve the underlying goal of the dialogue, and to implement certain communicative principles and obligations as filtering constraints on the planning of dialogue contributions.
I'm especially interested in applying machine learning and soft computing methods to design, develop and discover new possibilities for improving and widening dialogue processing. With the aim of building robust and natural user interfaces that take into account various input modalities, the research focuses on adaptive (agent-based) architectures, modelling interaction strategies, error situations, as well as features and requirements of the user. An important research topic is also to explore the new digital technology and its impact in society: to find possibilities it opens for communication, interaction and expression, and to evaluate dialogue systems and to understand the challenges they pose to human-computer interaction.
I have worked on most ares of Natural Language Processing (morphology, syntax, semantics, discourse), and I'm particulary interested in conversational models and dialogue management. My thesis dealt with cooperative communication from both theoretical side (Allwood's Ideal Cooperation) and practical side (building a response planner for the PLUS dialogue systems, ref to EU-project PLUS), and I developed the Constructive Dialogue Model (CDM) approach to dialogue management. The CDM has its view-point in response planning and co-operative communication: dialogue contributions are constructed as reactions to the previous contribution on the basis of local information, and different pragmatic constraints are used to filter out contextually inappropriate responses. I have also worked on modelling dialogue information (topics and speech acts) for a speech recogniser and a spoken dialogue system, and developed a Topic Model in which the relation between topic types and utterance words is measured by mutual information.
Current research issues
Due to technological and social development, the interaction with our environment has become more complex. I n order to design and build systems that would respond to challenges of the complex environment and to various needs of the users, it is important to provide tools, models, and concepts that would enable us to experiment with various types of complex systems, and to research and test processes underlying flexible human-computer interaction.
One of the main research aims is to take an active role in strategic planning and integration of matters relating to interactive media. My current projects concerns researching and producing applications for searching and presenting information to the users, building adaptive user models and rich interaction models in the application areas as varied as bus timetable system, email agents, and multimodal museum services to as wide a range of audiences as possible. By investigating what kind of options rapidly evolving multimedia technology can offer for relating and teaching visual arts in different multi-sensory ways, we can search for answers to such burning questions as which input and output modalities should be used for which task, which are the best combinations, and how to coordinate modalities appropriately for various purposes.
Related to this, the question of a suitable architecture arises. The development platform should allow flexible coordination of multimodal input and provide facilities for the integration of knowledge sources for various purposes. It is also necessary that the platform supports experimentation and testing with various modules concerning different types of input modalities and different, possibly alternative module implementations, and compatibility with current technical standards is also desirable. A hybrid system mixing rule-based approaches with machine learning algorithms may well provide the most interesting results.
Another research issue is how different modalities are used by or for users with different degrees of expertise and for users with special needs. Research on multimodal interaction will thus address more general topics of the principles of usability, accessibility and information design in network services (MUMIN network, MUMMI project).
Finally, a promising approach to modality integration is the use of different machine learning techniques and especially such techniques as neural networks. Previous research has shown excellent accuracy rates when applying these methods on natural language processing and dialogue management, and a challenge is to use such classification techniques as a basis for designing, testing and comparing various dialogue systems.
Finally, an important research topic is to explore the new digital technology and its impact in society: to find possibilities it opens for communication, and to evaluate dialogue systems in order to understand the challenges they pose to human-computer interaction. In this, my plan is to argue that the main principles of Ideal Cooperation, especially those related to cognitive and ethical consideration, are insightful both in forming evaluation standards and raising our understanding of the development and internal functioning of human-computer interaction.