Binary Acceleration and Compression for Dense Vector Entity Retrieval Models
WANG Yuanzheng1,2, FAN Yixing1,2, CHEN Wei1,2, ZHANG Ruqing1,2, GUO Jiafeng1,2
1. Key Laboratory of Network Data Science and Technology, Institute of Computing Technology, Chinese Academy of Sciences, Beijing 100190; 2. School of Computer Science and Technology, University of Chinese Academy of Sciences, Beijing 101408
Abstract In entity retrieval tasks, dense vector entity retrieval models are utilized to efficiently filter candidate entities related to a query from a large-scale entity base.However, the existing dense vector retrieval models engender low real-time computation efficiency and large required storage space due to the high dimension of entity vectors. In this paper,it is found that these entity vectors contain a large amount of redundant information through experiments. Most entity vectors are distributed in non-overlapping quadrants and quadrants containing entities with similar semantics are also closer to each other.Thus, a binary entity retrieval method is proposed to compress entity vectors and accelerate similarity calculations.Specifically, the sign function is employed to binary-compress high-dimensional dense floating-point vectors, and Hamming distance is exploited to speed up the retrieval.The reason that the proposed method can guarantee the retrieval performance is theoretically analyzed.The correctness of the theory is verified through qualitative and quantitative analysis experiments, and a method for improving binary retrieval performance based on random dimension increase and rotation is provided.
Fund:National Natural Science Foundation of China(No.61902381,62006218), Youth Innovation Promotion Association Member Project of Chinese Academy of Sciences(No.20144310,2021100), Young Elite Scientist Sponsorship Program by China Association for Science and Technology(No.YESS20200121), Innovation Project of Institute of Computing Technology of Chinese Academy of Sciences(No.E261090)
Corresponding Authors:
FAN Yixing, Ph.D., associate professor. His research interests include data mining and information retrieval.
About author:: WANG Yuanzheng, Ph.D. candidate. His research interests include information retrieval and natural language processing.CHEN Wei, Ph.D., professor. Her research interests include machine learning.ZHANG Ruqing, Ph.D., assistant profe-ssor. Her research interests include natural language processing.GUO Jiafeng, Ph.D., professor. His research interests include data mining and information retrieval.
[1] HAN X P, SUN L, ZHAO J.Collective Entity Linking in Web Text: A Graph-Based Method // Proc of the 34th International ACM SIGIR Conference on Research and Development in Information Retrieval. New York, USA: ACM, 2011: 765-774. [2] XIONG C Y, LIU Z Z, CALLAN J, et al. JointSem: Combining Query Entity Linking and Entity Based Document Ranking // Proc of the ACM Conference on Information and Knowledge Management. New York, USA: ACM, 2017: 2391-2394. [3] NOORALAHZADEH F, ÓVRELID L.SIRIUS-LTG: An Entity Lin-king Approach to Fact Extraction and Verification // Proc of the 1st Workshop on Fact Extraction and VERification. Stroudsburg, USA: ACL, 2018: 119-123. [4] FERRUCCI D A.Introduction to "This Is Watson". IBM Journal of Research and Development, 2012, 56(3/4). DOI: 10.1147/JRD.2012.2184356. [5] CHEN D Q, FISCH A, WESTON J, et al. Reading Wikipedia to Answer Open-Domain Questions // Proc of the 55th Annual Meeting of the Association for Computational Linguistics. Stroudsburg, USA: ACL, 2017: 1870-1879. [6] LEWIS P, PEREZ E, PIKTUS A, et al.Retrieval-Augmented Ge-neration for Knowledge-Intensive NLP Tasks // Proc of the 34th International Conference on Neural Information Processing Systems. Cambridge, USA: MIT Press, 2020: 9459-9474. [7] ROLLER S, DINAN E, GOYAL N, et al. Recipes for Building an Open-Domain Chatbot // Proc of the 16th Conference of the European Chapter of the Association for Computational Linguistics(Main Volume). Stroudsburg, USA: ACL, 2021: 300-325. [8] XIONG C Y, CALLAN J, LIU T Y.Word-Entity Duet Representations for Document Ranking // Proc of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval. New York, USA: ACM, 2017: 763-772. [9] HASIBI F, BALOG K, BRATSBERG S E.Exploiting Entity Lin-king in Queries for Entity Retrieval // Proc of the ACM International Conference on the Theory of Information Retrieval. New York, USA: ACM, 2016: 209-218. [10] BALOG K, RAMAMPIARO H, TAKHIROV N, et al. Multi-step Classification Approaches to Cumulative Citation Recommendation // Proc of the 10th Conference on Open Research Areas in Information Retrieval. New York, USA: ACM, 2013: 121-128. [11] REINANDA R, MEIJ E, DE RIJKE M.Mining, Ranking and Recommending Entity Aspects // Proc of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval. New York, USA: ACM, 2015: 263-272. [12] YANG Y, IRSOY O, RAHMAN K S.Collective Entity Disambiguation with Structured Gradient Tree Boosting // Proc of the Confe-rence of the North American Chapter of the Association for Computational Linguistics(Human Language Technologies). Stroudsburg, USA: ACL, 2018: 777-786. [13] WU L, PETRONI F, JOSIFOSKI M, et al. Scalable Zero-Shot Entity Linking with Dense Entity Retrieval // Proc of the Conference on Empirical Methods in Natural Language Processing. Stroudsburg, USA: ACL, 2020: 6397-6407. [14] YAMADA I, ASAI A, SHINDO H, et al. LUKE: Deep Contextualized Entity Representations with Entity-Aware Self-Attention // Proc of the Conference on Empirical Methods in Natural Language Processing. Stroudsburg, USA: ACL, 2020: 6442-6454. [15] WANG Y Z, CHENG X Q, FAN Y X, et al. MGAD: Learning Descriptional Representation Distilled from Distributional Semantics for Unseen Entities // Proc of the 31st International Joint Confe-rence on Artificial Intelligence. San Francisco, USA: IJCAI, 2022: 4404-4410. [16] JÉGOU H, DOUZE M, SCHMID C. Product Quantization for Nearest Neighbor Search. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2011, 33(1): 117-128. [17] HOFFART J, YOSEF M A, BORDINO I, et al. Robust Disambi-guation of Named Entities in Text // Proc of the Conference on Empirical Methods in Natural Language Processing. Stroudsburg, USA: ACL, 2011: 782-792. [18] VAN HULST J M, HASIBI F, DERCKSEN K, et al. REL: An Entity Linker Standing on the Shoulders of Giants // Proc of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval. New York, USA: ACM, 2020: 2197-2200. [19] DELIN J, CHANG M W, LEE K, et al. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding // Proc of the Conference of the North American Chapter of the Association for Computational Linguistics(Human Language Technologies). Stroudsburg, USA: ACL, 2019: 4171-4186. [20] YAMADA I, ASAI A, HAJISHIRZI H.Efficient Passage Retrieval with Hashing for Open-Domain Question Answering // Proc of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing. Stroudsburg, USA: ACL, 2021: 979-986. [21] JEGOU H, DOUZE M, SCHMID C.Hamming Embedding and Weak Geometric Consistency for Large Scale Image Search // Proc of the European Conference on Computer Vision. Berlin, Ger-many: Springer, 2008: 304-317. [22] GONG Y C, LAZEBNIK S, GORDO A, et al. Iterative Quantization: A Procrustean Approach to Learning Binary Codes for Large-Scale Image Retrieval. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2013, 35(12): 2916-2929.
2023, Vol. 36 
