基于对抗强化学习的多跳知识推理

doi:10.16451/j.cnki.issn1003-6059.202501002

Abstract
Figure/Table
References
Related Citation (9)

Download: PDF (847 KB) HTML (1 KB)
Export: BibTeX | EndNote (RIS)

Abstract To address the issues of insufficient representation of complex relationships, data sparsity, and false paths in multi-hop reasoning models within existing knowledge graph question-answering systems, a multi-hop knowledge reasoning model based on adversarial reinforcement learning is proposed. First, high-order relation vectors are decomposed to parameterize and combine entity and relation features. An attention mechanism is introduced when neighboring nodes are aggregated to assign different weights, thereby enhancing the representation ability of complex relationships. Additionally, a knowledge graph embedding framework is designed to measure the credibility of <subject entity, question, answer entity> in the embedding space. Second, multi-dimensional information is integrated into the state representation of the reinforcement learning framework to enable the Agent to make reliable decisions despite data sparsity. The generator calculates the probability of candidate entities based on state information and generates answers, while the discriminator evaluates the reasonableness of the answers and the reasoning paths. The problem of false paths is alleviated by optimizing the feedback through soft rewards and path rewards, and adversarial training is utilized to alternately optimize the generator and the discriminator. Finally, the model is applied to a multi-hop question-answering system for cloud manufacturing product design knowledge to verify its effectiveness. Comparative experiments, ablation experiments and case studies verify the effectiveness of the proposed model.

Key words： Complex Relation Representation Multi-hop Reasoning Adversarial Reinforcement Learning False Path

Received: 04 November 2024

ZTFLH:

TP391.1

Fund:Central Government-Guided Local Science and Technology Development Fund Project(No.YDZJSX1A044), Open Fund of the Shanxi Key Laboratory of Intelligent Information Processing(No.CICIP2023001), Shanxi Provincial Graduate Practical Innovation Project(No.2024SJ320)

Corresponding Authors: GUO Yinzhang, Ph.D., professor. His research interests include crowd computing, cloud computing and deep learning.

About author:: CHENG Lingyun, Master student. Her research interests include cloud computing and cloud security, and knowledge graphs.LIU Qingfang, Master student. Her research interests include crowd computing, cloud computing and deep learning.

	Service

	E-mail this article
	Add to my bookshelf
	Add to citation manager
	E-mail Alert
	RSS
	Articles by authors
	CHENG Lingyun
	GUO Yinzhang
	LIU Qingfang

Cite this article:

CHENG Lingyun,GUO Yinzhang,LIU Qingfang. Multi-hop Knowledge Reasoning Based on Adversarial Reinforcement Learning[J]. Pattern Recognition and Artificial Intelligence, 2025, 38(1): 22-35.

URL:

http://manu46.magtech.com.cn/Jweb_prai/EN/10.16451/j.cnki.issn1003-6059.202501002 OR http://manu46.magtech.com.cn/Jweb_prai/EN/Y2025/V38/I1/22

[1] YE Z, KUMAR Y J, SING G O, et al. A Comprehensive Survey of Graph Neural Networks for Knowledge Graphs. IEEE Access, 2022, 10: 75729-75741.
[2] ZHU X, GAO W, LI T Y, et al. Event-Centric Hierarchical Hyperbolic Graph for Multi-hop Question Answering over Knowledge Gra-phs. Engineering Applications of Artificial Intelligence, 2024, 133(Part B). DOI: 10.1016/j.engappai.2024.107971.
[3] WANG Y P, NING B, JIANG S, et al. RiQ-KGC: Relation Instantiation Enhanced Quaternionic Attention for Complex-Relation Know-ledge Graph Completion. Applied Sciences, 2024, 14(8). DOI: 10.3390/app14083221.
[4] YI K X, WU J J, GAN C, et al. Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understanding//Proc of the 32nd International Conference on Neural Information Processing Systems. Cambridge, USA: MIT Press, 2018: 1039-1050
[5] SUN H T, BEDRAX-WEISS T, COHEN W W.PullNet: Open Domain Question Answering with Iterative Retrieval on Knowledge Bases and Text//Proc of the Conference on Empirical Methods in Na-tural Language Processing and the 9th International Joint Conference on Natural Language Processing. Stroudsburg, USA: ACL, 2019: 2380-2390.
[6] LIU B Y, YU H M, QI G D.GraftNet: Towards Domain Generalized Stereo Matching with a Broad-Spectrum and Task-Oriented Feature//Proc of the IEEE/CVF Conference on Computer Vision and Pa-ttern Recognition. Washington, USA: IEEE, 2022: 13002-13011.
[7] SAXENA A, TRIPATHI A, TALUKDAR P.Improving Multi-hop Question Answering over Knowledge Graphs Using Knowledge Base Embeddings//Proc of the 58th Annual Meeting of the Association for Computational Linguistics. Stroudsburg, USA: ACL, 2020: 4498-4507.
[8] YAO L, MAO C S, LUO Y.KG-BERT: BERT for Knowledge Gra-ph Completion[C/OL].[2024-10-21].https://arxiv.org/pdf/1909.03193.
[9] XIE X, LI Z B, WANG X H, et al. LambdaKG: A Library for Pre-Trained Language Model-Based Knowledge Graph Embeddings[C/OL].[2024-10-21]. https://arxiv.org/pdf/2210.00305.
[10] XIONG W H, HOANG T, WANG W Y.DeepPath: A Reinforcement Learning Method for Knowledge Graph Reasoning//Proc of the Conference on Empirical Methods in Natural Language Proc-essing. Stroudsburg, USA: ACL, 2017: 564-573.
[11] WANG Q, HAO Y S, CAO J.ADRL: An Attention-Based Deep Reinforcement Learning Framework for Knowledge Graph Reaso-ning. Knowledge-Based Systems, 2020, 197. DOI: 10.1016/j.knosys.2020.105910.
[12] EYSENBACH B, SALAKHUTDINOV R, LEVINE S.Search on the Replay Buffer: Bridging Planning and Reinforcement Learning//Proc of the 33rd International Conference on Neural Information Processing Systems. Cambridge, USA: MIT Press, 2019: 15246-15257.
[13] QIU Y Q, WANG Y Z, JIN X L, et al. Stepwise Reasoning for Multi-relation Question Answering over Knowledge Graph with Weak Supervision//Proc of the 13th International Conference on Web Search and Data Mining. New York, USA: ACM, 2020: 474-482.
[14] SAEBI M, KREIG S, ZHANG C X, et al. Heterogeneous Relational Reasoning in Knowledge Graphs with Reinforcement Lear-ning. Information Fusion, 2022, 88(12): 12-21.
[15] ZHANG Q X, WENG X Y, ZHOU G Y, et al. ARL: An Adaptive Reinforcement Learning Framework for Complex Question Answe-ring over Knowledge Base. Information Processing and Management, 2022, 59(3). DOI: 10.1016/j.ipm.2022.102933.
[16] ZHAN M, FAN J J, GUO J Y.Generative Adversarial Inverse Reinforcement Learning with Deep Deterministic Policy Gradient. IEEE Access, 2023, 11: 87732-87746.
[17] VIQUERAT J, DUVIGNEAU R, MELIGA P, et al. Policy-Based Optimization: Single-Step Policy Gradient Method Seen as an Evolution Strategy. Neural Computing and Application, 2023, 35(1): 449-467.
[18] SWAMINATHAN A, ZHANG H M, MAHATA D, et al. A Preliminary Exploration of GANs for Keyphrase Generation//Proc of the Conference on Empirical Methods in Natural Language Proce-ssing. Stroudsburg, USA: ACL, 2020: 8021-8030.
[19] HE G L, LI J Y, ZHAO W X, et al. Mining Implicit Entity Prefe-rence from User-Item Interaction Data for Knowledge Graph Completion via Adversarial Learning//Proc of the Web Conference. New York, USA: ACM, 2020: 740-751.
[20] ZHANG S Q, ZHANG N J, FAN S, et al. Knowledge Graph Re-commendation Model Based on Adversarial Training. Applied Sciences, 2022, 12(15). DOI: 10.3390/app12157434.
[21] SCHLICHTKRULL M, KIPF T N, BLOEM P, et al. Modeling Relational Data with Graph Convolutional Networks//Proc of the 15th European Semantic Web Conference. Berlin, Germany: Sprin-ger, 2018: 593-607.
[22] VASHISHTH S, SABYAL S, NITIN V, et al. Composition-Based Multi-Relational Graph Convolutional Networks[C/OL].[2024-10-21]. https://openreview.net/attachment?id=BylA_C4tPr&name=original_pdf.
[23] BALAZEVIC I, ALLEN C, HOSPEDALES T.TuckER: Tensor Factorization for Knowledge Graph Completion//Proc of the Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Proce-ssing. Stroudsburg, USA: ACL,2019: 5185-5194.
[24] WATKINS J C H C, DAYAN P. Technical Note: Q-Learning. Machine Learning, 1992, 8(3/4): 279-292.
[25] ZHOU M T, HUANG M L, ZHU X Y.An Interpretable Reasoning Network for Multi-relation Question Answering//Proc of the 27th International Conference on Computational Linguistics. Stroudsburg, USA: ACL, 2018: 2010-2022.
[26] 张元鸣,姬琦,徐雪松,等.基于知识图谱关系路径的多跳智能问答模型研究.电子学报, 2023, 51(11): 3092-3099.
(ZHANG Y M, JI Q, XU X S, et al. Knowledge Graph Relation Path Network for Multi-hop Intelligent Question Answering. Acta Electronica Sinica, 2023, 51(11): 3092-3099.)
[27] SUKHBAATAR S, SZLAM A, WESTON J, et al. End-to-End Memory Networks//Proc of the 29th International Conference on Neural Information Processing Systems. Cambridge, USA: MIT Press, 2015, II: 2440-2448.
[28] MILLER A, FISCH A, DODGE J, et al. Key-Value Memory Networks for Directly Reading Documents//Proc of the Conference on Empirical Methods in Natural Language Processing. Stroudsburg, USA: ACL,2016: 1400-1409.
[29] CUI H, PENG T, BAO T, et al. Stepwise Relation Prediction with Dynamic Reasoning Network for Multi-hop Knowledge Graph Question Answering. Applied Intelligence, 2023, 53(10): 12340-12354.
[30] HEO Y J, KIM E S, CHOI W S, et al. Hypergraph Transformer: Weakly-Supervised Multi-hop Reasoning for Knowledge-Based Vi-sual Question Answering//Proc of the 60th Annual Meeting of the Association for Computational Linguistics(Long Papers). Stroudsburg, USA: ACL, 2022: 373-390.
[31] WANG X, ZHAO S, CHENG B, et al. Explore Modeling Relation Information and Direction Information in KBQA. Neurocomputing, 2022, 471: 139-148.
[32] DAS R, DHULIAWALA S, ZAHEER M, et al. Go for a Walk and Arrive at the Answer: Reasoning over Paths in Knowledge Bases Using Reinforcement Learning[C/OL].[2024-10-21]. https://arxiv.org/pdf/1711.05851.
[33] LI C, ZHENG H T, SUN Y P, et al. Enhancing Multi-hop Know-ledge Graph Reasoning through Reward Shaping Techniques//Proc of the 4th International Conference on Machine Learning and Intelligent Systems Engineering. Washington, USA: IEEE, 2024. DOI: 10.1109/MLISE62164.2024.10674566.
[34] SHANG B, ZHAO Y L, LIU Y F, et al. Attention-Based Exploitation and Exploration Strategy for Multi-hop Knowledge Graph Reasoning. Information Sciences, 2024, 653. DOI: 10.1016/j.ins.2023.119787.
[35] TROUILLON T, WELBL J, RIEDEL S, et al. Complex Embe-ddings for Simple Link Prediction[C/OL].[2024-10-21]. https://arxiv.org/pdf/1412.6575v2.
[36] YANG B S, YIH W, HE X D, et al. Embedding Entities and Relations for Learning and Inference in Knowledge Bases[C/OL].[2024-10-21]. https://arxiv.org/pdf/1412.6575v2.
[37] DETTERS T, MINERVINI P, STENETORP P, et al. Convolutio-nal 2D Knowledge Graph Embeddings. Proceedings of the AAAI Conference on Artificial Intelligence, 2018, 32(1): 1811-1818.