Narrative-Driven Large Language Model for Temporal Knowledge Graph Prediction
CHEN Juan1,2, ZHAO Xinchao3, SUI Jingyan1,2, QI Lin4, TIAN Chen4, PANG Liang1,2, FANG Jinyun1
1. Prospective Research Laboratory, Institute of Computing Te-chnology, Chinese Academy of Sciences, Beijing 100190; 2. School of Computer Science and Technology, University of Chinese Academy of Sciences, Beijing 100049; 3. Landinn Center, Big Data Academy, Zhongke, Zhengzhou 450046; 4. School of Software Engineering, Beijing Jiaotong University, Beijing 100044
Abstract The temporal knowledge graph(TKG) is characterized by vast sparsity, and the long-tail distribution of entities leads to poor generalization in reasoning for out-of-distribution entities. Additionally, the low infrequency of historical interactions results in biased predictions for future events. Therefore, a narrative-driven large language model for TKG Prediction is proposed. The world knowledge and complex semantic reasoning capabilities of large language models are leveraged to enhance the understanding of out-of-distribution entities and the association of sparse interaction events. Firstly, a key event tree is selected based on the temporal and structural characteristics of TKG, and the most representative events are extracted through a historical event filtering strategy. Relevant historical information is summarized to reduce input data while the most important information is retained. Then, the large language model generator is fine-tuned to produce logically coherent "key event tree" narratives as unstructured input. During the generation process, special attention is paid to the causal relationships and temporal sequences of events to ensure the coherence and rationality of the generated stories. Finally, the large language model is utilized as a reasoner to infer the missing temporal entities. Experiments on three public datasets demonstrate that the proposed method effectively leverages the capabilities of large models to achieve more accurate temporal entity reasoning.
Corresponding Authors:
PANG Liang, Ph.D., assistant professor. His research interests include natural language processing and machine learning.
About author:: CHEN Juan, Ph.D.candidate. Her research interests include temporal knowledge graphs, large language models and natural language processing. ZHAO Xinchao, Master student. His research interests include natural language processing. SUI Jingyan, Ph.D.candidate. Her research interests include algorithm design, machine learning, deep learning and combinatorial optimization. QI Lin, Master student. Her research interests include graph neural networks and natural language processing.TIAN Chen, Master student. His research interests include graph neural networks and natural language processing. FANG Jinyun, Ph.D., professor. His research interests include massive spatial information processing technologies.
CHEN Juan,ZHAO Xinchao,SUI Jingyan等. Narrative-Driven Large Language Model for Temporal Knowledge Graph Prediction[J]. Pattern Recognition and Artificial Intelligence, 2024, 37(8): 715-728.
[1] JIN W, QU M, JIN X S, et al.Recurrent Event Network: Autoregressive Structure Inference over Temporal Knowledge Graphs // Proc of the Conference on Empirical Methods in Natural Language Processing. Stroudsburg, USA: ACL, 2020: 6669-6683. [2] HAN Z, DING Z F, MA Y P, et al. Temporal Knowledge Graph Forecasting with Neural ODE[C/OL].[2024-06-16]. https://arxiv.org/pdf/2101.05151v1. [3] WANG K Z, HAN C R, POON J.Re-Temp: Relation-Aware Temporal Representation Learning for Temporal Knowledge Graph Completion // Proc of the Conference on Empirical Methods in Natural Language Processing. Stroudsburg, USA: ACL, 2023: 258-269. [4] ZHAO Y J, LIU Y M, WAN Z H, et al. tHR-Net: A Hybrid Reasoning Framework for Temporal Knowledge Graph // Proc of the International Conference on Collaborative Computing: Networking, Applications and Worksharing. Berlin, Germany: Springer, 2023: 223-241. [5] RADFORD A, NARASIMHAN K, SALIMANS T, et al. et al. Improving Language Understanding by Generative Pre-Training[C/OL].[2024-06-16]. https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf. [6] BUBECK S, CHANDRASEKARAN V, ELDAN R, et al. Sparks of Artificial General Intelligence: Early Experiments with GPT-4[C/OL].[2024-06-16]. https://arxiv.org/pdf/2303.12712. [7] XU W J, LIU B, PENG M, et al. Pre-trained Language Model with Prompts for Temporal Knowledge Graph Completion // Proc of the Findings of the Association for Computational Linguistics. Stroudsburg, USA: ACL, 2023: 7790-7803. [8] LEE D H, AHRABIAN K, JIN W, et al. Temporal Knowledge Gra-ph Forecasting without Knowledge Using In-context Learning // Proc of the Conference on Empirical Methods in Natural Language Processing. Stroudsburg, USA: ACL, 2023: 544-557. [9] BELTAGY I, PETERS M E, COHAN A.Longformer: The Long-Document Transformer[C/OL]. [2024-06-16].https://arxiv.org/pdf/2004.05150v1. [10] PRESS O, SMITH N A, LEWIS M. Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation[C/OL]. [2024-06-16].https://arxiv.org/pdf/2108.12409. [11] RAFFEL C, SHAZEER N, ROBERTS A, et al. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transfor-mer. Journal of Machine Learning Research, 2020, 21(1): 5485-5551. [12] LEBLAY J, CHEKOL M W.Deriving Validity Time in Knowledge Graph // Proc of the Web Conference. New York, USA: ACM, 2018: 1771-1776. [13] LACROIX T, OBOZINSKI G, USUNIER N.Tensor Decompositions for Temporal Knowledge Base Completion[C/OL]. [2024-06-16].https://arxiv.org/pdf/2004.04926. [14] TRIVEDI R, DAI H J, WANG Y C, et al. Know-Evolve: Deep Temporal Reasoning for Dynamic Knowledge Graphs. Journal of Machine Learning Research, 2017, 70: 3462-3471. [15] TRIVEDI R, FARAJTABAR M, BISWAL P, et al. DyRep:Lear-ning Representations over Dynamic Graphs[C/OL].[2024-06-16]. https://openreview.net/pdf?id=HyePthR5KX. [16] ZHAO L, SONG Y J, ZHANG C, et al. T-GCN: A Temporal Graph Convolutional Network for Traffic Prediction. IEEE Transactions on Intelligent Transportation Systems, 2020, 21(9): 3848-3858. [17] LI Z X, JIN X L, LI W, et al. Temporal Knowledge Graph Reasoning Based on Evolutional Representation Learning // Proc of the 44th International ACM SIGIR Conference on Research and Deve-lopment in Information Retrieval. New York, USA: ACM, 2021: 408-417. [18] WU J P, CAO M, CHEUNG J C K, et al. TeMP: Temporal Me-ssage Passing for Temporal Knowledge Graph Completion // Proc of the Conference on Empirical Methods in Natural Language Proce-ssing. Stroudsburg, USA: ACL, 2020: 5730-5746. [19] ZHU C C, CHEN M H, FAN C J, et al. Learning from History: Modeling Temporal Knowledge Graphs with Sequential Copy-Gene-ration Networks. Proceedings of the AAAI conference on Artificial Intelligence, 2021, 35(5): 4732-4740. [20] YAO L, MAO C S, LUO Y.KG-BERT: BERT for Knowledge Graph Completion[C/OL]. [2024-06-16].https://arxiv.org/pdf/1909.03193. [21] PETRONI F, ROCKTÄSCHEL T, LEWIS P, et al. Language Mo-dels as Knowledge Bases? // Proc of the Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing. Stroudsburg, USA: ACL, 2019: 2463-2473. [22] LIAO R T, JIA X, LI Y Z, et al. GenTKG: Generative Forecasting on Temporal Knowledge Graph // Proc of the Findings of the Association for Computational Linguistics. Stroudsburg, USA: ACL, 2024: 4303-4317. [23] LUO R L, GU T L, LI H L, et al. Chain of History: Learning and Forecasting with LLMs for Temporal Knowledge Graph Completion[C/OL].[2024-06-16]. https://arxiv.org/pdf/2401.06072. [24] XIA Y W, WANG D, LIU Q, et al. Enhancing Temporal Know-ledge Graph Forecasting with Large Language Models via Chain-of-History Reasoning[C/OL].[2024-06-16]. https://arxiv.org/pdf/2402.14382v1. [25] HAN Z, CHEN P, MA Y P, et al. Explainable Subgraph Reaso-ning for Forecasting on Temporal Knowledge Graphs[C/OL].[2024-06-16]. https://arxiv.org/pdf/2012.15537. [26] LIU Y S, MA Y P, HILDEBRANDT M, et al. TLogic: Temporal Logical Rules for Explainable Link Forecasting on Temporal Know-ledge Graphs. Proceedings of the AAAI Conference on Artificial Intelligence, 2022, 36(4): 4120-4127. [27] HAN Z, DING Z F, MA Y P, et al. Learning Neural Ordinary Equations for Forecasting Future Links on Temporal Knowledge Graphs // Proc of the Conference on Empirical Methods in Natural Language Processing. Stroudsburg, USA: ACL, 2021: 8352-8364. [28] SUN H H, ZHONG J L, MA Y P, et al. TimeTraveler: Reinforcement Learning for Temporal Knowledge Graph Forecasting // Proc of the Conference on Empirical Methods in Natural Language Processing. Stroudsburg, USA: ACL, 2021: 8306-8319. [29] BLACK S, BIDERMAN S, HALLAHAN E, et al. GPT-NeoX-20B: An Open-Source Autoregressive Language Model // Proc of the Workshop on Challenges and Perspectives in Creating Large Language Models. Stroudsburg, USA: ACL, 2022: 95-136.
2024, Vol. 37 
