Semantic Search with AI Vector Search
DOI:
https://doi.org/10.63282/3050-9416.IJAIBDCMS-V5I2P114Keywords:
Semantic Search, AI, Vector Search, Embeddings, Natural Language Processing, Information Retrieval, BERT, FAISSAbstract
The concept of semantic search has become a ground breaking method of retrieving information that overcomes the weaknesses of the conventional system based on key words. By utilizing progressive developments in the field of artificial intelligence (AI) and vector-based embeddings, semantic search systems learn context, intent, and meanings of queries and documents. The current paper provides an in-depth research on neural search based on AI-driven vector search, the descriptions of used methodologies, experimental design, and performance analysis. The research points to the state-of-the-art embedding models, the techniques of the vectors indexing, and similarity that enhances the accuracy and relevance of the retrieval. The study puts a strain on the implementation of AI-based models like BERT, GPT, and Word2Vec to generate vectors representations with special focus on the effect of high-dimensional vectors in the process of semantic latencies. The paper also covers the architecture of similarity search at scale with the help of such vector database structures as FAISS, Annoy, and Milvus. When experimented over benchmark datasets, the submission can be seen to improve greatly both in terms of precision, recall and F1-score over traditional search methods based on keywords. The findings show that semantic vector search does not only improve relevance in retrieval but also promote complex query processing, which can be used in question-answering systems, recommendation systems, and enterprise search software. Moreover, the paper includes detailed methodology that includes data preprocessing, embedding generation, and indexing of vectors and query processing pipelines. Issues of computational overhead, dimensionality embedding, and latency in real-time search are also resolved and give information towards realistic implementation in large scale systems. The paper ends with the future directions, which include multi-modal embeddings, real-time vector updates, and knowledge graph integration that allow further better semantic understanding
References
1. Robertson, S., & Zaragoza, H. (2009). The probabilistic relevance framework: BM25 and beyond. Foundations and Trends® in Information Retrieval, 3(4), 333-389.
2. Tekale, K. M., & Rahul, N. (2022). AI and Predictive Analytics in Underwriting, 2022 Advancements in Machine Learning for Loss Prediction and Customer Segmentation. International Journal of Artificial Intelligence, Data Science, and Machine Learning, 3(1), 95-113. https://doi.org/10.63282/3050-9262.IJAIDSML-V3I1P111
3. Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013). Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781.
4. Pennington, J., Socher, R., & Manning, C. D. (2014, October). Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP) (pp. 1532-1543).
5. Devlin, J., Chang, M. W., Lee, K., & Toutanova, K. (2019, June). Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers) (pp. 4171-4186).
6. Tekale, K. M., Enjam, G. R., & Rahul, N. (2023). AI Risk Coverage: Designing New Products to Cover Liability from AI Model Failures or Biased Algorithmic Decisions. International Journal of AI, BigData, Computational and Management Studies, 4(1), 137-146. https://doi.org/10.63282/3050-9416.IJAIBDCMS-V4I1P114
7. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., ... & Polosukhin, I. (2017). Attention is all you need. Advances in neural information processing systems, 30.
8. Johnson, J., Douze, M., & Jégou, H. (2019). Billion-scale similarity search with GPUs. IEEE Transactions on Big Data, 7(3), 535-547.
9. Tekale, K. M., Enjam, G. R., & Rahul, N. (2023). AI Risk Coverage: Designing New Products to Cover Liability from AI Model Failures or Biased Algorithmic Decisions. International Journal of AI, BigData, Computational and Management Studies, 4(1), 137-146. https://doi.org/10.63282/3050-9416.IJAIBDCMS-V4I1P114
10. Thallam, N. S. T. (2020). Comparative Analysis of Data Warehousing Solutions: AWS Redshift vs. Snowflake vs. Google BigQuery. European Journal of Advances in Engineering and Technology, 7(12), 133-141.
11. Arpit Garg. (2022). Behavioral biometrics for IoT security: A machine learning framework for smart homes. Journal of Recent Trends in Computer Science and Engineering, 10(2), 71–92. https://doi.org/10.70589/JRTCSE.2022.2.7
12. Mohammed, M. T., & Rashid, O. F. (2023). Document retrieval using term term frequency inverse sentence frequency weighting scheme. Indones. J. Electr. Eng. Comput. Sci, 31(3), 1478.
13. Tekale, K. M., & Rahul, N. (2023). Blockchain and Smart Contracts in Claims Settlement. International Journal of Emerging Trends in Computer Science and Information Technology, 4(2), 121-130. https://doi.org/10.63282/3050-9246.IJETCSIT-V4I2P112
14. Zhang, X., Xia, M., Couturier, C., Zheng, G., Rajmohan, S., & Rühle, V. (2023). Hybrid retrieval-augmented generation for real-time composition assistance.
15. Wu, J., Gan, W., Chen, Z., Wan, S., & Yu, P. S. (2023, December). Multimodal large language models: A survey. In 2023 IEEE International Conference on Big Data (BigData) (pp. 2247-2256). IEEE.
16. Srivastava, A., Nalluri, M., Lata, T., Ramadas, G., Sreekanth, N., & Vanjari, H. B. (2023, December). Scaling AI-Driven Solutions for Semantic Search. In 2023 International conference on power energy, environment & intelligent control (PEEIC) (pp. 1581-1586). IEEE.
17. Kumar, R., Tripathi, R. C., & Singh, V. (2016). Keyword based search and its limitations in the patent document to secure the idea from its infringement. Procedia Computer Science, 78, 439-446.
18. Chen, Y., Wang, W., & Liu, Z. (2011, April). Keyword-based search and exploration on databases. In 2011 IEEE 27th International Conference on Data Engineering (pp. 1380-1383). IEEE.
19. Shi, Y., Zi, X., Shi, Z., Zhang, H., Wu, Q., & Xu, M. (2024). Eragent: Enhancing retrieval-augmented language models with improved accuracy, efficiency, and personalization. arXiv preprint arXiv:2405.06683.
20. Tekale, K. M., & Enjam, G. reddy. (2023). Advanced Telematics & Connected-Car Data. International Journal of Emerging Trends in Computer Science and Information Technology, 4(1), 124-132. https://doi.org/10.63282/3050-9246.IJETCSIT-V4I1P114
21. Sandeep Rangineni Latha Thamma reddi Sudheer Kumar Kothuru , Venkata Surendra Kumar, Anil Kumar Vadlamudi. Analysis on Data Engineering: Solving Data preparation tasks with ChatGPT to finish Data Preparation. Journal of Emerging Technologies and Innovative Research. 2023/12. (10)12, PP 11, https://www.jetir.org/view?paper=JETIR2312580
22. Fonseca, M. J., & Jorge, J. A. (2003, March). Indexing high-dimensional data for content-based retrieval in large databases. In Eighth International Conference on Database Systems for Advanced Applications, 2003.(DASFAA 2003). Proceedings. (pp. 267-274). IEEE.
23. Vector Search vs Semantic Search, tigerdata, 2024. Online. https://www.tigerdata.com/learn/vector-search-vs-semantic-search
24. Guo, J., Cai, Y., Fan, Y., Sun, F., Zhang, R., & Cheng, X. (2022). Semantic models for the first-stage retrieval: A comprehensive review. ACM Transactions on Information Systems (TOIS), 40(4), 1-42.
25. Bast, H., Buchhold, B., & Haussmann, E. (2016). Semantic search on text and knowledge bases. Foundations and Trends® in Information Retrieval, 10(2-3), 119-271.
26. Sehrawat, S. K. (2023). The role of artificial intelligence in ERP automation: state-of-the-art and future directions. Trans Latest Trends Artif Intell, 4(4).
27. Tekale , K. M. (2023). AI-Powered Claims Processing: Reducing Cycle Times and Improving Accuracy. International Journal of Artificial Intelligence, Data Science, and Machine Learning, 4(2), 113-123. https://doi.org/10.63282/3050-9262.IJAIDSML-V4I2P113
28. Sidorov, G., Gelbukh, A., Gómez-Adorno, H., & Pinto, D. (2014). Soft similarity and soft cosine measure: Similarity of features in vector space model. Computación y Sistemas, 18(3), 491-504.
29. Sarker, I. H. (2022). AI-based modeling: techniques, applications and research issues towards automation, intelligent and smart systems. SN computer science, 3(2), 158.
30. Tekale, K. M., & Rahul, N. (2023). Blockchain and Smart Contracts in Claims Settlement. International Journal of Emerging Trends in Computer Science and Information Technology, 4(2), 121-130. https://doi.org/10.63282/3050-9246.IJETCSIT-V4I2P112
31. Thallam, N. S. T. (2021). Privacy-Preserving Data Analytics in the Cloud: Leveraging Homomorphic Encryption for Big Data Security. Journal of Scientific and Engineering Research, 8(12), 331-337.
32. Drucker, H., Shahrary, B., & Gibbon, D. C. (2002). Support vector machines: relevance feedback and information retrieval. Information processing & management, 38(3), 305-323.
33. Settibathini, V. S., Kothuru, S. K., Vadlamudi, A. K., Thammreddi, L., & Rangineni, S. (2023). Strategic analysis review of data analytics with the help of artificial intelligence. International Journal of Advances in Engineering Research, 26, 1-10.
34. Garg, A. (2022). Unified Framework of Blockchain and AI for Business Intelligence in Modern Banking . International Journal of Emerging Research in Engineering and Technology, 3(4), 32-42. https://doi.org/10.63282/3050-922X.IJERET-V3I4P105
