Domain-Specific Summarization to Enhance LLM Effectiveness
DOI:
https://doi.org/10.65834/jdsi.12.32Keywords:
LoRA, fine-tuning, context length, petition text, prompt engineeringAbstract
Large Language Models (LLMs) have become increasingly prevalent in the field of natural language processing, particularly for text summarization tasks. Their ability to capture semantic relationships and generate coherent summaries has made them a powerful alternative to traditional extractive and abstractive summarization methods. However, the effective summarization of long and domain-specific texts, such as petitions, remains a challenging problem. This study explores the problem of petition text summarization by applying state-of-the-art LLMs and evaluating various training strategies on a custom-built dataset. The paper investigates different model training approaches, including Low-Rank Adaptation (LoRA) and prompt engineering techniques, to optimize summarization quality. Human-in-the-loop summarization methods are also incorporated to refine the models, ensuring high-quality outputs in real-world scenarios. Furthermore, the model’s performance is assessed through expert evaluation, which highlights areas for potential improvement in terms of both accuracy and coherence. The research further delves into methods for effectively summarizing long-form petition texts, exploring ways to preserve critical information while maintaining brevity and readability. The results demonstrate the feasibility of leveraging LLM-based summarization methods for complex, lengthy texts while providing insights into the challenges and opportunities within this evolving field.
References
Blei, D. M., Ng, A. Y., & Jordan, M. I. (2003). Latent Dirichlet allocation. Journal of Machine Learning Research, 3, 993–1022.
Chen, Y., Ma, Y., Mao, X., & Li, Q. (2019). Multi-task learning for abstractive and extractive summarization. Data Science and Engineering, 4(1), 14–23. https://doi.org/10.1007/s41019-019-0087-7
Dang, J., Singh, S., D’souza, D., Ahmadian, A., Salamanca, A., Smith, M., Peppin, A., Hong, S., Govindassamy, M., Zhao, T., Kublik, S., Amer, M., Aryabumi, V., Campos, J. A., Tan, Y.-C., Kocmi, T., Strub, F., Grinsztajn, N., Flet-Berliac, Y., ... Hooker, S. (2024). Aya Expanse: Combining research breakthroughs for a new multilingual frontier. arXiv. https://doi.org/10.48550/arXiv.2412.04261
Erkan, G., & Radev, D. R. (2004). LexRank: Graph-based lexical centrality as salience in text summarization. Journal of Artificial Intelligence Research, 22, 457–479. https://doi.org/10.1613/jair.1523
Guan, C., Chin, A., & Vahabi, P. (2024). Enhancing news summarization with eLearnFit through efficient in-context learning and efficient fine-tuning. arXiv. https://doi.org/10.48550/arXiv.2405.02710
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., & Chen, W. (2021). LoRA of LLMs. arXiv. https://doi.org/10.48550/arXiv.2106.09685
Huang, L., Cao, S., Parulian, N., Ji, H., & Wang, L. (2021). Efficient attentions for long document summarization. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (pp. 1419–1436). Association for Computational Linguistics. https://doi.org/10.18653/v1/2021.naacl-main.112
Liu, D., & Demberg, V. (2024). RST-LoRA: A discourse-aware LoRA for long document abstractive summarization. arXiv. https://doi.org/10.48550/arXiv.2405.00657
Liu, Y. (2019). Fine-tune BERT for extractive summarization. arXiv. https://doi.org/10.48550/arXiv.1903.10318
Meta AI. (2024). The LlaMA 3 herd of models. arXiv. https://doi.org/10.48550/arXiv.2407.21783
OpenAI. (2025). gpt-oss-120b & gpt-oss-20b model card. arXiv. https://doi.org/10.48550/arXiv.2508.10925
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P., Leike, J., & Lowe, R. (2022). Training language models to follow instructions with human feedback. arXiv. https://doi.org/10.48550/arXiv.2203.02155
Qin, H., Ma, X., Zheng, X., Li, X., Zhang, Y., Liu, S., Luo, J., Liu, X., & Magno, M. (2024). Accurate LoRA-finetuning quantization of LLMs via information retention. arXiv. https://doi.org/10.48550/arXiv.2402.05445
Rallapalli, S., Gallagher, S., Mellinger, A. O., Ratchford, J., Sinha, A., Brooks, T., & Brown, B. (2025). Fine-tuning LLMs for report summarization: Analysis on supervised and unsupervised data. arXiv. https://doi.org/10.48550/arXiv.2503.10676
See, A., Liu, P. J., & Manning, C. D. (2017). Get to the point: Summarization with pointer-generator networks. arXiv. https://doi.org/10.48550/arXiv.1704.04368
Xiao, W., & Carenini, G. (2019). Extractive summarization of long documents by combining global and local context. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) (pp. 3011-3021). https://doi.org/10.18653/v1/D19-1298
Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, C., Lv, C., Zheng, C., Liu, D., Zhou, F., Huang, F., Hu, F., Ge, H., Wei, H., Lin, H., Tang, J., ... Qiu, Z. (2025). Qwen3 technical report. arXiv. https://doi.org/10.48550/arXiv.2505.09388
Zhang, Y., Ni, A., Mao, Z., Wu, C.-H., Zhu, C., Deb, B., & Zhang, R. (2022). SummN: A multi-stage summarization framework for long input dialogues and documents. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 1592–1604). Association for Computational Linguistics. https://doi.org/10.18653/v1/2022.acl-long.112
Zhao, J., Wang, T., Abid, W., Angus, G., Garg, A., Kinnison, J., Sherstinsky, A., Molino, P., Addair, T., & Rishi, D. (2024). LoRA Land: 310 fine-tuned LLMs that rival GPT-4, a technical report. arXiv. https://doi.org/10.48550/arXiv.2405.00732
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2026 Journal of Defence and Security Industries: Strategy and Technology
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.