DEEP LEARNING-BASED MODIFIED TRANSFORMER MODEL FOR AUTOMATED NEWS ARTICLE SUMMARIZATION

B. Srinivas, Lavanya Bagadi, Naresh K. Darimireddy, P. Surya Prasad, Sivaji Satrupalli, Anil Kumar B.

DOI Number
https://doi.org/10.2298/FUEE2402261S
First page
261
Last page
276

Abstract


The amount of textual data on the internet is increasing enormously, so data summarization into text has become essential. As generating text summaries manually is an arduous task and humans are generally prone to make mistakes, deep learning techniques have evolved to overcome this problem. Modified transformer-based deep learning models with varying encoder-decoder and feed-forward network layers are proposed to develop an abstractive summary of the news articles. The proposed transformer model provides the advantage of parallelization with the help of multiple attention head layers to process long sentences, and hence, better text summarization performance is achieved. These models are trained on an ‘in-shorts’ dataset, and the proposed model is compared with the PEGASUS-CNNdaily-mail, BART-large-CNN, and DistilBART-CNN-12-6 models on the CNN/DailyMail dataset. The performance is evaluated in terms of the ROUGE score by comparing it with the existing Recurrent Neural Network (RNN) model. The suggested transformer model achieved a ROUGE score of 0.33, surpassing the RNN model score of 0.17. This innovative approach can be employed on extensive textual data to extract summaries or headlines.

Keywords

Natural Language Processing, Deep Learning, Abstractive summarization, Large Language Model, RNN, Transformer Model, News Article Summarization

Full Text:

PDF

References


M. Alfraheed, "An Approach for Features Matching between Bilateral Images of Stereo Vision System Applied for Automated Heterogeneous Platoon", Journal of Theoretical & Applied Information Technology, vol. 96, no.7, Apr. 2018.

N. Giarelis, M. Charalampos, and K. Nikos, "Abstractive vs. Extractive Summarization: An Experimental Review", Applied Sciences, vol. 13, no. 13, Jun 2023.

Y. Chen, M. Yun, M. Xudong, and L. Qing, "Multi-task learning for abstractive and extractive summarization", Data Science and Engineering, vol. 4, pp. 14–23, Mar 2019.

D. Suleiman, and A. Arafat, "Deep learning based abstractive text summarization: approaches, datasets, evaluation measures, and challenges", Mathematical problems in engineering, pp. 1–29, Aug 2020.

R. S. Shini, and K. VD. Ambeth, "Recurrent neural network based text summarization techniques by word sequence generation", In Proceedings of the 6th International Conference on Inventive Computation Technologies (ICICT), Jan. 2021, pp. 1224–1229.

W. Fang, J. TianXiao, J. Ke, Z. Feihong, D. Yewen, and S. Jack, "A method of automatic text summarization based on long short-term memory", International Journal of Computational Science and Engineering, vol. 22, no. 1, pp. 39–49, 2020.

T. Goyal, J. L. Junyi, and D. Greg, "News summarization and evaluation in the era of gpt-3." arXiv preprint arXiv: 2209.12356, Sep. 2022.

W. S. El-Kassas, R. S. Cherif, A. R. Ahmed, and K. M. Hoda, "Automatic text summarization: A comprehensive survey", Expert systems with applications, vol. 165, Mar 2021.

K. Ganesan, XZ. Cheng, and H. Jiawei, "Opinosis: A graph-based approach to abstractive summarization of highly redundant opinions." In Proceedings of the 23rd International Conference on Computational Linguistics (Coling 2010), 2010, pp. 340-348.

P.-E. Genest, and L. Guy, "Fully abstractive approach to guided summarization." In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics, 2012, vol. 2, pp. 354–358.

A. Khan, S. Naomie, F. Haleem, K. Murad, J. Bilal, A. Awais, A. Imran, and P. Anand, "Abstractive text summarization based on improved semantic graph approach", International Journal of Parallel Programming, vol. 46, pp. 992–1016, 2018.

T. Shi, K. Yaser, R. K. Naren, and K. R. Chandan, "Neural abstractive text summarization with sequence-to-sequence models", ACM Transactions on Data Science, vol. 2, no. 1, pp. 1–37, 2021.

X. Chen, G. Shen, T. Chongyang, S. Yan, Z. Dongyan, and Y. Rui, "Iterative document representation learning towards summarization with polishing." arXiv preprint arXiv: 1809.10324, 2018.

L. Dong, Y. Nan, W. Wenhui, W. Furu, L. Xiaodong, W. Yu, G. Jianfeng, Z. Ming, and H. Hsiao-Wuen, "Unified language model pre-training for natural language understanding and generation", Advances in neural information processing systems, vol. 32, 2019.

X. Zhang, and L. Mirella, "Sentence simplification with deep reinforcement learning", arXiv preprint arXiv: 1703.10931, 2017.

J. Lin, S. Xu, M. Shuming, and S. Qi, "Global encoding for abstractive summarization", arXiv preprint arXiv: 1805.0398, 2018.

R. Pasunuru, and B. Mohit, "Multi-reward reinforced summarization with saliency and entailment", arXiv preprint arXiv: 1804.06451, 2018.

N. Kalchbrenner, E. Lasse, S. Karen, van den O. Aaron, G. Alex, and K. Koray, "Neural machine translation in linear time", arXiv preprint arXiv: 1610.10099, 2016.

Van den O. Aaron, D. Sander, Z. Heiga, S Karen, V. Oriol, G. Alex, K. Nal, S. Andrew, and K. Koray, "Wavenet: A generative model for raw audio", arXiv preprint arXiv: 1609.03499, vol. 12, 2016.

A. Vaswani, S. Noam, P. Niki, U. Jakob, J. Llion, N. G. Aidan, K. Łukasz, and P. Illia, "Attention is all you need", Advances in neural information processing systems, vol. 30, 2017.

D. Bahdanau, C. Kyunghyun, and B. Yoshua, "Neural machine translation by jointly learning to align and translate", arXiv preprint arXiv: 1409.0473, 2014.

M. Lewis, L. Yinhan, G. Naman, G. Marjan, M. Abdelrahman, L. Omer, S. Ves, and Z. Luke, "Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension", arXiv preprint arXiv: 1910.13461, 2019.

https://www.kaggle.com/datasets/shashichander009/inshorts-news-data.

S. Abbes, B. A. Sarra, H. Rim, and C. Philippe, "Automatic text summarization using transformers", In Proceedings of the Knowledge Graphs and Semantic Web: Third Iberoamerican Conference and Second Indo-American Conference, November 2021, vol. 3, pp. 308–320.

S. Gupta, and K. G. Sanjai, "Abstractive summarization: An overview of the state of the art", Expert Systems with Applications, vol. 121, pp. 49–65, May 2019.

C.-Y. Lin, and F. J. Och, "Looking for a few good metrics: ROUGE and its evaluation", in NTCIR workshop, 2004.

T. Wolf, D. Lysandre, S. Victor, C. Julien, D. Clement, M. Anthony, C. Pierric et al, "Hugging face's transformers: State-of-the-art natural language processing", arXiv preprint arXiv:1910.03771, 2019.

C. Raffel, S. Noam, R. Adam, L. Katherine, N. Sharan, M. Michael, Z. Yanqi, L. Wei, and J. L. Peter, "Exploring the limits of transfer learning with a unified text-to-text transformer", Journal of machine learning research, vol. 21, no. 140, pp. 1–67, 2020.

F. Zhuang, Q. Zhiyuan, D. Keyu, X. Dongbo, Z. Yongchun, Z. Hengshu, X. Hui, and H. Qing, "A comprehensive survey on transfer learning", In Proceedings of the IEEE, 2020, vol. 109, no. 1, pp. 43–76.

B. Srinivas, and S. R. Gottapu, "Segmentation of multi-modal MRI brain tumor sub-regions using deep learning", Journal of Electrical Engineering & Technology, vol. 15, no. 4, pp.1899–1909, Jul 2020.

M. Allahyari, P. Seyedamin, A. Mehdi, S. Saeid, D. T. Elizabeth, B.G. Juan, and K. Krys, "Text summarization techniques: a brief survey", arXiv preprint arXiv: 1707.02268, 2017.

C. Khatri, S. Gyanit, and P. Nish, "Abstractive and extractive text summarization using document context vector and recurrent neural networks", arXiv preprint arXiv: 1807.08000, 2018.

M. Ranzato, C. Sumit, A. Michael, and Z. Wojciech, "Sequence level training with recurrent neural networks", arXiv preprint arXiv: 1511.06732, 2015.

https://www.kaggle.com/datasets/gowrishankarp/newspaper-text-summarization-cnn-dailymail.

Y. Yu, S. Xiaosheng, H. Changhua, and Z. Jianxun, "A review of recurrent neural networks: LSTM cells and network architectures", Neural computation, vol. 31, no. 7, pp. 1235–1270, Jul 2019.


Refbacks

  • There are currently no refbacks.


ISSN: 0353-3670 (Print)

ISSN: 2217-5997 (Online)

COBISS.SR-ID 12826626