建置與評估文字自動生成的情感對話系統

Development and Evaluation of Emotional Conversation System Based on Automated Text Generation

10.6120/JoEMLS.202011_57(3).0048.RS.CM

楊德倫(Te-Lun Yang)；曾元顯(Yuen-Hsien Tseng)

對話系統 ； 文字生成 ； 文意理解 ； 深度學習 ； 人工智慧 ； Conversational system ； Text generation ； Text understanding ； Deep learning ； Artificial intelligence

教育資料與圖書館學

57卷3期（2020 / 11 / 01）

355 - 378

繁體中文

本研究藉由2019年中文情緒對話生成（CECG）評比任務所提供約170萬則語料，運用深度學習GPT-2與BERT等技術與工具，實作了具備情感對話的系統，並以CECG提供的測試發文評估其成效。由三位人工判斷的結果，顯示本研究建置發展的系統，與2019年CECG評測最佳團隊的系統有類似的成效水準。而進一步的案例分析發現，對於訓練資料中較普遍的話題，GPT-2的語言建模技術，的確可以生成創新、有趣、完美的回應文句。本研究的主要貢獻為：㈠將情感融入發文字串中做為條件求機率，以便簡潔地依原方式訓練並使用GPT-2；㈡運用BERT來預測回應文句的連貫性以做為排序的依據。雖然這兩項技巧分別源自GPT與BERT的訓練機制，但本研究稍加修改應用於CECG的任務上，獲得了不錯的效果。

Based on the corpus provided by the 2019 Chinese Emotional Conversation Generation (CECG) evaluation task, an emotional conversation system is implemented in this paper using deep learning and other technologies such as GPT-2 and BERT. The effectiveness of the system is evaluated based on the test data and criteria provided by CECG. The results based on three human annotators show that the system has a similar effectiveness level with that of the best team participating in the 2019 CECG task. Further case studies reveal that the more post/reply pairs about a topic in the training data, the better the language model of GPT-2 to generate innovative, interesting, and perfect response sentences for that topic. The main contributions of this study are: 1. Integrating emotion into the post string as a condition for computing probability, so as to simply train GPT-2 and make GPT-2 predict in the original way; 2. Applying BERT to predict the coherence of response sentences as a basis for ranking. Although these two techniques are derived from the training mechanisms of GPT and BERT respectively, we have slightly modified them to fit the task of CECG and achieved good results.

1. Bengio, Y.,Ducharme, R.,Vincent, P.,Janvin, C.(2003).A neural probabilistic language model.Journal of Machine Learning Research,3,1137-1155.
2. Binsted, K.(1995).Using humour to make natural language interfaces more friendly.Workshop on AI, ALife and Entertainment, International Joint Conference on Artificial Intelligence,Montreal, Canada:
3. Binsted, K.,Bergen, B.,Coulson, S.,Nijholt, A.,Stock, O.,Strapparava, C.,Ritchie, G.,Manurung, R.,Pain, H.,Waller, A.,O’Mara, D.(2006).Computational humor.IEEE Intelligent Systems,21(2),59-69.
4. Brown, T. B.,Mann, B.,Ryder, N.,Subbiah, M.,Kaplan, J.,Dhariwal, P.,Neelakantan, A.,Shyam, P.,Sastry, G.,Askell, A.,Agarwal, S.,Herbert-Voss, A.,Krueger, G.,Henighan, T.,Child, R.,Ramesh, A.,Ziegler, D. M.,Wu, J.,Winter, C.,Amodei, D.(2020).,未出版
5. Cagan, T.,Frank, S. L.,Tsarfaty, R.(2017).Data-driven broad-coverage grammars for opinionated natural language generation (ONLG).Proceedings of the 55th annual meeting of the Association for Computational Linguistics (Vol. 1)
6. Devlin, J.,Chang, M.-W.,Lee, K.,Toutanova, K.(2018).,未出版
7. Du, Z. (2019). GPT2-Chinese: Tools for training GPT2 model in Chinese language. Retrieved January 12, 2020, from https://github.com/Morizeyao/GPT2-Chinese
8. Ghosh, S.,Chollet, M.,Laksana, E.,Morency, L.-P.,Scherer, S.(2017).Affect-LM: A neural language model for customizable affective text generation.Proceedings of the 55th annual meeting of the Association for Computational Linguistics (Vol. 1)
9. Hochreiter, S.,Schmidhuber, J.(1997).Long short-term memory.Neural Computation,9(8),1735-1780.
10. Hornik, K.(1991).Approximation capabilities of multilayer feedforward networks.Neural Networks,4(2),251-257.
11. Hu, Z.,Yang, Z.,Liang, X.,Salakhutdinov, R.,Xing, E. P.(2017).Toward controlled generation of text.Proceedings of the 34th international conference on machine learning (PMLR 70)
12. Krizhevsky, A.,Sutskever, I.,Hinton, G. E.(2012).ImageNet classification with deep convolutional neural networks.Proceedings of the 25th international conference on neural information processing systems
13. LeCun, Y.,Bengio, Y.,Hinton, G.(2015).Deep learning.Nature,521,436-444.
14. Mikolov, T.,Sutskever, I.,Chen, K.,Corrado, G. S.,Dean, J.(2013).Distributed representations of words and phrases and their compositionality.Proceedings of the 26th international conference on neural information processing systems (Vol. 2)
15. Minsky, M.,Papert, S. A.(1969).Perceptrons: An introduction to computational geometry.MIT Press.
16. Partala, T.,Surakka, V.(2004).The effects of affective interventions in human-computer interaction.Interacting with Computers,16(2),295-309.
17. Prendinger, H.,Ishizuka, M.(2005).The empathic companion: A character-based interface that addresses users’ affective states.Applied Artificial Intelligence: An International Journal,19(3-4),267-285.
18. Radford, A., Narasimhan, K., Salimans, T., & Sutskever, I. (2018). Improving language understanding by generative pre-training. https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf
19. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., & Sutskever, I. (2019). Language models are unsupervised multitask learners. https://d4mucfpksywv.cloudfront.net/better-language-models/language_models_are_unsupervised_multitask_learners.pdf
20. Rumelhart, D. E.,McClelland, J. L.(1986).Parallel distributed processing: Explorations in the microstructure of cognition: Vol. 1. Foundations.MIT Press.
21. Skowron, M.(2010).Affect listeners: Acquisition of affective states by means of conversational systems.Lecture notes in computer science: Vol. 5967. Development of multimodal interfaces: Active listening and synchrony
22. Sutskever, I.,Vinyals, O.,Le, Q. V.(2014).Sequence to sequence learning with neural networks.Proceedings of the 27th international conference on neural information processing systems (Vol. 2)
23. Vaswani, A.,Shazeer, N.,Parmar, N.,Uszkoreit, J.,Jones, L.,Gomez, A. N.,Kaiser, Ł.,Polosukhin, I.(2017).Attention is all you need.Proceedings of the 30th international conference on neural information processing systems
24. Zhang, Y.,Huang, M.(2019).Overview of the NTCIR-14 short text generation subtask: Emotion generation challenge.14th NTCIR Conference on Evaluation of Information Access Technologies,Tokyo Japan:
25. Zhou, H.,Huang, M.,Zhang, T.,Zhu, X.,Liu, B.(2018).Emotional chatting machine: Emotional conversation generation with internal and external memory.The thirty-second AAAI conference on artificial intelligence

1. 曾元顯(Yuen-Hsien Tseng);林郁綺(Yu-Chi Lin)(2021)。電腦生成的新聞有多真？－文字自動生成技術運用於經濟新聞的評估。圖書資訊學刊。19(1)。43-65。