SU Lixin1,2, GUO Jiafeng1,2, FAN Yixing1, LAN Yanyan1,2, CHENG Xueqi3
1. Key Laboratory of Network Data Science and Technology, Institute of Computing Technology, Chinese Academy of Sciences, Beijing 100190; 2. School of Computer and Control Engineering, University of Chinese Academy of Sciences, Beijing 100190; 3. Institute of Network Technology, Institute of Computing Technology (YANTAI), Chinese Academy of Sciences, Yantai 264005
Abstract In the existing extractive reading comprehension models, only the boundary of answers is utilized as the supervision signal and the labeling processed by human is ignored. Consequently, learned models are prone to learn the superficial features and the generalization performance is degraded. In this paper, a label-enhanced reading comprehension model is proposed to imitate human activity. The answer-bearing sentence, the content and the boundary of the answer are learned simultaneously. The answer-bearing sentence and the content of the answer can be derived from the boundary of the answer and these three types of labels are regarded as supervision signals. The model is trained by multitask learning. During prediction, the probabilities from three predictions are merged to determine the answer, and thus the generalization performance is improved. Experiments on SQuAD dataset demonstrate the effectiveness of LE-Reader model.
Fund:Supported by National Key Research and Development Program of China(No.2016QY02D0405), National Natural Science Foundation of China(No.61425016,61722211,61773362,61872338,61902381), Youth Innovation Promotion Association of CAS(No.20144310,2016102), Foundation and Frontier Research Key Program of Chongqing Science and Technology Commission(No.cstc2017jcyjBX0059), Taishan Scholars Program of Shandong Province(No.ts201511082)
Corresponding Authors:
GUO Jiafeng, Ph.D., professor. His research interests include data mining and information retrieval.
About author:: SU Lixin, Ph.D. candidate. His research interests include information retrieval and question answering; FAN Yixing, Ph.D., research assistant. His research interests include data mining and information retrieval; LAN Yanyan, Ph.D., associate professor. Her research interests include machine lear-ning and data mining; CHENG Xueqi, Ph.D., professor. His research interests include network science and social computing, web search and data mi-ning.
[1] HERMANN K M, KOCISKY T, GREFENSTETTE E, et al. Tea-ching Machines to Read and Comprehend[C/OL]. [2019-09-24]. https://arxiv.org/pdf/1506.03340.pdf. [2] RICHARDSON M, BURGES C J C, RENSHAW E. MCTest: A Challenge Dataset for the Open-Domain Machine Comprehension of Text // Proc of the Conference on Empirical Methods in Natural Language Processing. Stroudsburg, USA: ACL, 2013: 193-203. [3] LAI G K, XIE Q Z, LIU H X, et al. RACE: Large-Scale Reading Comprehension Dataset from Examinations // Proc of the Conference on Empirical Methods in Natural Language Processing. Stroudsburg, USA: ACL, 2017: 785-794. [4] TALMOR A, HERZIG J, LOURIE N, et al. CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge[C/OL]. [2019-09-24]. https://arxiv.org/pdf/1811.00937.pdf. [5] RAJPURKAR P, ZHANG J, LOPYREV K, et al. SQuAD: 100,000+ Questions for Machine Comprehension of Text // Proc of the Confe-rence on Empirical Methods in Natural Language Processing. Strouds-burg, USA: ACL, 2016: 2383-2392. [6] JOSHI M, CHOI E, WELD D S, et al. TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension // Proc of the 55th Annual Meeting of the Association for Computational Linguistics(Long Papers). Stroudsburg, USA: ACL, 2017: 1601-1611. [7] DUNN M, SAGUN L, HIGGINS M, et al. SearchQA: A New Q&A Dataset Augmented with Context from a Search Engine[C/OL]. [2019-09-24]. https://arxiv.org/pdf/1704.05179.pdf. [8] CHOI E, HE H, IYYER M, et al. QuAC: Question Answering in Context // Proc of the Conference on Empirical Methods in Natural Language Processing. Stroudsburg, USA: ACL, 2018: 2174-2184. [9] BAJAJ P, CAMPOS D, CRASWELL N, et al. MS MARCO: A Human Generated Machine Reading Comprehension Dataset[C/OL]. [2019-09-24]. https://arxiv.org/pdf/1611.09268.pdf. [10] HE W, LIU K, LIU J, et al. DuReader: A Chinese Machine Reading Comprehension Dataset from Real-World Applications[C/OL]. [2019-09-24]. https://arxiv.org/pdf/1711.05073.pdf. [11] SEO M, KEMBHAVI A, FARHADI A, et al. Bidirectional Attention Flow for Machine Comprehension[C/OL]. [2019-09-24]. https://arxiv.org/pdf/1611.01603.pdf. [12] HUANG H Y, ZHU C G, SHEN Y L, et al. FusionNet: Fusing via Fully-Aware Attention with Application to Machine Comprehension[C/OL]. [2019-09-24]. https://arxiv.org/pdf/1711.07341.pdf. [13] YU A W, DOHAN D, LUONG M T, et al. QANet: Combining Local Convolution with Global Self-attention for Reading Comprehension[C/OL]. [2019-09-24]. https://arxiv.org/pdf/1804.09541.pdf. [14] DEVLIN J, CHANG M W, LEE K, et al. Bert: Pre-training of Deep Bidirectional Transformers for Language Understanding[C/OL]. [2019-09-24]. https://arxiv.org/pdf/1810.04805.pdf. [15] VASWANI A, SHAZEER N, PARMAR N, et al. Attention Is All You Need[C/OL]. [2019-09-24]. https://arxiv.org/pdf/1706.03762.pdf. [16] WANG W H, YANG N, WEI F R, et al. Gated Self-matching Networks for Reading Comprehension and Question Answering // Proc of the 55th Annual Meeting of the Association for Computational Linguistics(Long Papers). Stroudsburg, USA: ACL, 2017: 189-198. [17] LIU X D, SHEN Y L, DUH K, et al. Stochastic Answer Networks for Machine Reading Comprehension[C/OL]. [2019-09-24]. ttps://arxiv.org/pdf/1712.03556.pdf. [18] RAJPURKAR P, JIA R B, LIANG P. Know What You Don't Know: Unanswerable Questions for SQuAD[C/OL]. [2019-09-24]. https://arxiv.org/pdf/1806.03822.pdf. [19] KINGMA D P, BA J L. ADAM: A Method for Stochastic Optimization[C/OL]. [2019-09-24]. https://arxiv.org/pdf/1412.6980.pdf. [20] HENDRYCKS D, GIMPEL K. Gaussian Error Linear Units(GELUs)[C/OL]. [2019-09-24]. https://arxiv.org/pdf/1606.08415.pdf. [21] SUGAWARA S, INUI K, SEKINE S, et al. What Makes Reading Comprehension Questions Easier? [C/OL]. [2019-09-24]. https://arxiv.org/pdf/1808.09384v1.pdf. [22] YATSKAR M. A Qualitative Comparison of COQA, Squad 2.0 and QUAC[C/OL]. [2019-09-24]. https://arxiv.org/pdf/1809.10735.pdf.
2020, Vol. 33 
