QnA Chatbot with Mistral 7B and RAG method: Traffic Law Case Study

Muhammad Roiful Anam; Agus Subhan Akbar; Heru Saputro

doi:10.24843/LKJITI.2024.v15.i03.p06

Muhammad Roiful Anam Universitas Islam Nahdlatul Ulama Jepara
Agus Subhan Akbar
Heru Saputro

DOI: https://doi.org/10.24843/LKJITI.2024.v15.i03.p06

Abstract

Mistral 7B is a language model designed to achieve high efficiency and performance in handling Natural Language Processing (NLP). This research will evaluate the model's effectiveness in legal data processing using the Retrieval-Augmented Generation (RAG) method, focusing on road traffic and transportation law No 22/2009. The system was built using the LangChain framework, followed by fine-tuning the model and evaluated using BERTScore. Results showed that the fine-tuned Mistral 7B achieved an F1 score of 0.9151, higher than the version without fine-tuning (0.8804) and GPT-4 (0.8364). To improve accuracy, the model utilizes specific keywords that make it easier to find relevant data. Fine-tuning was shown to enhance precision, while the use of key elements in questions helped the model focus more on important information. The results are expected to support the development of artificial intelligence (AI) in Indonesia's legal system and provide practical guidance for applying AI technology in other areas of law.

Downloads

Download data is not yet available.

References

[1] M. N. Setiawan, R. S. Roring, Y. D. Atma, and H. Tetiawadi, “Studi Empiris Terhadap Asistensi Artificial Intelligence (AI) Dalam Rancang Bangun Aplikasi,” Digital Transformation Technology (Digitech), vol. 4, no. 1, pp. 364–373, Jun. 2024, doi: 10.47709/digitech.v4i1.4115.
[2] E. N. Ravizki and Lintang Yudhantaka, “Artificial Intelligence Sebagai Subjek Hukum: Tinjauan Konseptual dan Tantangan Pengaturan di Indonesia,” Notaire, vol. 5, no. 3, pp. 351–376, Oct. 2022, doi: 10.20473/ntr.v5i3.39063.
[3] A. Q. Jiang et al., “Mistral 7B,” Oct. 2023, [Online]. Available: https://arxiv.org/abs/2310.06825
[4] M. Salıcı and Ü. E. Ölçer, “Impact of Transformer-Based Models in NLP: An In-Depth Study on BERT and GPT,” in 2024 8th International Artificial Intelligence and Data Processing Symposium (IDAP), IEEE, Sep. 2024, pp. 1–6. doi: 10.1109/IDAP64064.2024.10710796.
[5] A. Salemi and H. Zamani, “Evaluating Retrieval Quality in Retrieval-Augmented Generation,” in Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, New York, NY, USA: ACM, Jul. 2024, pp. 2395–2400. doi: 10.1145/3626772.3657957.
[6] I. Pujiono, I. M. Agtyaputra, and Y. Ruldeviyani, “Implementing Retrieval-Augmented Generation And Vector Databases For Chatbots In Public Services Agencies Context,” JITK (Jurnal Ilmu Pengetah. dan Teknol. Komputer), vol. 10, no. 1, pp. 216–223, Aug. 2024, doi: 10.33480/jitk.v10i1.5572.
[7] K. M. Fitria, “Information Retrieval Performance in Text Generation using Knowledge from Generative Pre-trained Transformer (GPT-3),” Jambura Journal of Mathematics, vol. 5, no. 2, pp. 327–338, Aug. 2023, doi: 10.34312/jjom.v5i2.20574.
[8] A. T. Laksana, S. Sylviani, and A. Triska, “Studi Penerapan Konsep Vektor Dalam Permasalahan Penyisipan Kata-Kata Melalui Proses Normalisasi Vector Dan Transformasi Orthogonal,” Jurnal Lebesgue Jurnal Ilmiah Pendidikan Matematika, Matematika dan Statistik, vol. 5, no. 2, pp. 634–641, Aug. 2024, doi: 10.46306/lb.v5i2.493.
[9] F. Almeida and G. Xexéo, “Word Embeddings: A Survey,” Jan. 2019, doi: https://doi.org/10.48550/arXiv.1901.09069.
[10] K. Muludi, K. M. Fitria, J. Triloka, and S. -, “Retrieval-Augmented Generation Approach: Document Question Answering using Large Language Model,” International Journal of Advanced Computer Science and Applications(IJACSA), vol. 15, no. 3, 2024, doi: 10.14569/IJACSA.2024.0150379.
[11] S. Rahayu, N. S. Harahap, S. Agustian, and Pizaini, “Penerapan Teknologi LangChain pada Question Answering System Fikih Empat Madzhab,” MALCOM: Indonesian Journal of Machine Learning and Computer Science, vol. 4, no. 3, pp. 974–983, Jul. 2024, doi: https://doi.org/10.57152/malcom.v4i3.1397.
[12] G. I. E. Soen, Marlina, and Renny, “Implementasi Cloud Computing dengan Google Colaboratory pada Aplikasi Pengolah Data Zoom Participants,” JITU Journal of Informatic Technology and Communication, vol. 6, no. 1, pp. 24–30, Jun. 2022, doi: 10.36596/jitu.v6i1.781.
[13] Gradio, “Gradio: Build Machine Learning Web Apps in Python.” Accessed: Sep. 12, 2024. [Online]. Available: https://pypi.org/project/gradio
[14] C. S. R. Chan, C. Pethe, and S. Skiena, “Natural language processing versus rule-based text analysis: Comparing BERT score and readability indices to predict crowdfunding outcomes,” J. Bus. Ventur. Insights, vol. 16, p. e00276, Nov. 2021, doi: 10.1016/j.jbvi.2021.e00276.
[15] M. I. Syah, N. S. Harahap, Novriyanto, and S. Sanjaya, “PENERAPAN RETRIEVAL AUGEMENTED GENERATION MENGGUNAKAN LANGCHAIN DALAM PENGEMBANGAN SISTEM TANYA JAWAB HADIS BERBASIS WEB,” ZONAsi : Jurnal Sistem informasi, vol. 6, no. 2, pp. 370–379, May 2024, doi: 10.31849/zn.v6i2.19940.
[16] I. J. Unanue, J. Parnell, and M. Piccardi, “BERTTune: Fine-Tuning Neural Machine Translation with BERTScore,” Jun. 2021, [Online]. Available: https://arxiv.org/pdf/2106.02208
[17] A. T. U. B. Lubis, N. S. Harahap, S. Agustian, M. Irsyad, and I. Afrianty, “Question Answering System pada Chatbot Telegram Menggunakan Large Language Models (LLM) dan Langchain (Studi Kasus UU Kesehatan),” MALCOM: Indonesian Journal of Machine Learning and Computer Science, vol. 4, no. 3, pp. 955–964, May 2024, doi: 10.57152/malcom.v4i3.1378.
[18] F. Fajri, B. Tutuko, and S. Sukemi, “Membandingkan Nilai Akurasi BERT dan DistilBERT pada Dataset Twitter,” JUSIFO (Jurnal Sistem Informasi), vol. 8, no. 2, pp. 71–80, Dec. 2022, doi: 10.19109/jusifo.v8i2.13885.
[19] P. L. Romadloni, B. A. Kusuma, and W. M. Baihaqi, “KOMPARASI METODE PEMBELAJARAN MESIN UNTUK IMPLEMENTASI PENGAMBILAN KEPUTUSAN DALAM MENENTUKAN PROMOSI JABATAN KARYAWAN,” JATI : Jurnal Mahasiswa Teknik Informatika, vol. 6, no. 2, pp. 622–628, Sep. 2022, doi: 10.36040/jati.v6i2.5238.
[20] T. Zhang, V. Kishore, F. Wu, K. Q. Weinberger, and Y. Artzi, “BERTScore: Evaluating Text Generation with BERT,” Apr. 2019, [Online]. Available: http://arxiv.org/abs/1904.09675
[21] K. Pandya, “PEFT-MedAware: Large Language Model for Medical Awareness,” Nov. 2023, [Online]. Available: https://arxiv.org/abs/2311.10697
[22] W. Xia, C. Qin, and E. Hazan, “Chain of LoRA: Efficient Fine-tuning of Language Models via Residual Learning,” Jan. 2024, [Online]. Available: https://arxiv.org/abs/2401.04151
[23] R. Darman, “Peran ChatGPT Sebagai Artificial Intelligence Dalam Menyelesaikan Masalah Pertanahan dengan Metode Studi Kasus dan Black Box Testing,” Tunas Agraria, vol. 7, no. 1, pp. 18–46, Jan. 2024, doi: 10.31292/jta.v7i1.256.