SU Lixin1,2, GUO Jiafeng1,2, FAN Yixing1, LAN Yanyan1,2, CHENG Xueqi3
1. Key Laboratory of Network Data Science and Technology, Institute of Computing Technology, Chinese Academy of Sciences, Beijing 100190; 2. School of Computer and Control Engineering, University of Chinese Academy of Sciences, Beijing 100190; 3. Institute of Network Technology, Institute of Computing Technology (YANTAI), Chinese Academy of Sciences, Yantai 264005
Abstract：In the existing extractive reading comprehension models, only the boundary of answers is utilized as the supervision signal and the labeling processed by human is ignored. Consequently, learned models are prone to learn the superficial features and the generalization performance is degraded. In this paper, a label-enhanced reading comprehension model is proposed to imitate human activity. The answer-bearing sentence, the content and the boundary of the answer are learned simultaneously. The answer-bearing sentence and the content of the answer can be derived from the boundary of the answer and these three types of labels are regarded as supervision signals. The model is trained by multitask learning. During prediction, the probabilities from three predictions are merged to determine the answer, and thus the generalization performance is improved. Experiments on SQuAD dataset demonstrate the effectiveness of LE-Reader model.
[1] HERMANN K M, KOCISKY T, GREFENSTETTE E, et al. Tea-ching Machines to Read and Comprehend[C/OL]. [2019-09-24]. https://arxiv.org/pdf/1506.03340.pdf. [2] RICHARDSON M, BURGES C J C, RENSHAW E. MCTest: A Challenge Dataset for the Open-Domain Machine Comprehension of Text // Proc of the Conference on Empirical Methods in Natural Language Processing. Stroudsburg, USA: ACL, 2013: 193-203. [3] LAI G K, XIE Q Z, LIU H X, et al. RACE: Large-Scale Reading Comprehension Dataset from Examinations // Proc of the Conference on Empirical Methods in Natural Language Processing. Stroudsburg, USA: ACL, 2017: 785-794. [4] TALMOR A, HERZIG J, LOURIE N, et al. CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge[C/OL]. [2019-09-24]. https://arxiv.org/pdf/1811.00937.pdf. [5] RAJPURKAR P, ZHANG J, LOPYREV K, et al. SQuAD: 100,000+ Questions for Machine Comprehension of Text // Proc of the Confe-rence on Empirical Methods in Natural Language Processing. Strouds-burg, USA: ACL, 2016: 2383-2392. [6] JOSHI M, CHOI E, WELD D S, et al. TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension // Proc of the 55th Annual Meeting of the Association for Computational Linguistics(Long Papers). Stroudsburg, USA: ACL, 2017: 1601-1611. [7] DUNN M, SAGUN L, HIGGINS M, et al. SearchQA: A New Q&A Dataset Augmented with Context from a Search Engine[C/OL]. [2019-09-24]. https://arxiv.org/pdf/1704.05179.pdf. [8] CHOI E, HE H, IYYER M, et al. QuAC: Question Answering in Context // Proc of the Conference on Empirical Methods in Natural Language Processing. Stroudsburg, USA: ACL, 2018: 2174-2184. [9] BAJAJ P, CAMPOS D, CRASWELL N, et al. MS MARCO: A Human Generated Machine Reading Comprehension Dataset[C/OL]. [2019-09-24]. https://arxiv.org/pdf/1611.09268.pdf. [10] HE W, LIU K, LIU J, et al. DuReader: A Chinese Machine Reading Comprehension Dataset from Real-World Applications[C/OL]. [2019-09-24]. https://arxiv.org/pdf/1711.05073.pdf. [11] SEO M, KEMBHAVI A, FARHADI A, et al. Bidirectional Attention Flow for Machine Comprehension[C/OL]. [2019-09-24]. https://arxiv.org/pdf/1611.01603.pdf. [12] HUANG H Y, ZHU C G, SHEN Y L, et al. FusionNet: Fusing via Fully-Aware Attention with Application to Machine Comprehension[C/OL]. [2019-09-24]. https://arxiv.org/pdf/1711.07341.pdf. [13] YU A W, DOHAN D, LUONG M T, et al. QANet: Combining Local Convolution with Global Self-attention for Reading Comprehension[C/OL]. [2019-09-24]. https://arxiv.org/pdf/1804.09541.pdf. [14] DEVLIN J, CHANG M W, LEE K, et al. Bert: Pre-training of Deep Bidirectional Transformers for Language Understanding[C/OL]. [2019-09-24]. https://arxiv.org/pdf/1810.04805.pdf. [15] VASWANI A, SHAZEER N, PARMAR N, et al. Attention Is All You Need[C/OL]. [2019-09-24]. https://arxiv.org/pdf/1706.03762.pdf. [16] WANG W H, YANG N, WEI F R, et al. Gated Self-matching Networks for Reading Comprehension and Question Answering // Proc of the 55th Annual Meeting of the Association for Computational Linguistics(Long Papers). Stroudsburg, USA: ACL, 2017: 189-198. [17] LIU X D, SHEN Y L, DUH K, et al. Stochastic Answer Networks for Machine Reading Comprehension[C/OL]. [2019-09-24]. ttps://arxiv.org/pdf/1712.03556.pdf. [18] RAJPURKAR P, JIA R B, LIANG P. Know What You Don't Know: Unanswerable Questions for SQuAD[C/OL]. [2019-09-24]. https://arxiv.org/pdf/1806.03822.pdf. [19] KINGMA D P, BA J L. ADAM: A Method for Stochastic Optimization[C/OL]. [2019-09-24]. https://arxiv.org/pdf/1412.6980.pdf. [20] HENDRYCKS D, GIMPEL K. Gaussian Error Linear Units(GELUs)[C/OL]. [2019-09-24]. https://arxiv.org/pdf/1606.08415.pdf. [21] SUGAWARA S, INUI K, SEKINE S, et al. What Makes Reading Comprehension Questions Easier? [C/OL]. [2019-09-24]. https://arxiv.org/pdf/1808.09384v1.pdf. [22] YATSKAR M. A Qualitative Comparison of COQA, Squad 2.0 and QUAC[C/OL]. [2019-09-24]. https://arxiv.org/pdf/1809.10735.pdf.
2020, Vol. 33 
