Background

In last article, Automated Summarisation of PDFs with GPT and Python, we described how to use GPT-3 API to summarise articles from PDF. However, using GPT has some issues as listed below.

GPT Issues

• Length limitations: GPT models have a maximum output length, which can be a disadvantage when summarising long input texts, leading to a loss of accuracy.
• Repetition and redundancy: GPT models generates summaries word by word, without being able to “see” the entire summary as a whole. As a result, some phrases may be repeated or unnecessary information may be included.
• Difficulty with complex language: GPT models can struggle with understanding and summarising complex language, such as technical or scientific jargon.
• Dependence on training data: If the training data is not representative of the text to be summarised, or if the training data is biased in some way, the summaries produced by the model may be misleading.
• Data privacy concern: One potential big issue using GPT is that it is a cloud based model with data request going to OpenAI, individuals and organisations might not want their sensitive data stored on OpenAI servers.

To address the limitations of GPT, this article explores alternative libraries that can be run offline or on-premise, including Hugging Face Transformer and Scikit-Learn’s TfidfVectorizer.

Summarisation Types

Before we start there, just keep in mind there are two types of summarisations

• Extractive summarisation that refers to the process of selecting important sentences or phrases from a text.
• Abstractive summarisation involves generating new sentences that capture the essence of the original text.

The input article remains in an article published by UBS Wealth Management called “Let’s chat about ChatGPT”.

GPT

Generative Pre-trained Transformer is a multimodal large language model (LLM) created by OpenAI and currently the fourth in its GPT series.

The abstract below has been generated using GPT API with the prompt: ‘Summarise like an experienced consultant in 5 bullets:’

Generated Article Summary:

Summary Type: Abstractive

Pros:

• Intuitive and easy to use,
• Well constructed sentences,
• Using ChatGPT visual interface no coding required

Cons:

• Online cloud model (i.e. requires internet)
• Privacy status of data unknown
• Works good mainly for relatively short and non-technical texts

Hugging Face Transformer Library

The Transformer library, developed by Hugging Face, provides state-of-the-art models for various NLP tasks, including text summarisation. The library includes pre-trained models like BART, T5, and GPT-2, which can generate summaries of high quality. To generate a summary using the Transformer library, we can fine-tune the pre-trained models on a specific task using a large dataset of labeled examples.

Generated Article Summary:

Summary Type: Abstractive

Pros:

• Pre-trained model like ‘facebook/bart-large-cnn’ does an excellent job of translating regular (non-technical) text,
• Sentences are coherent and readable,
• This model can be enhanced in-house for custom technical text,
• The model is on-premise model taking care of data privacy issue

Cons:

• Being a neural network based model these models are difficult to interpret,
• Models like ‘facebook/bart-large-cnn’ have been trained to generate around 300-ish word summaries and tend to hallucinate for longer summarisations,
• The model is computationally expensive and require special hardware to run efficiently. This means unlike other mentioned models (TF-IDF, PyTexRank, spaCy) it better to run this model as a server with an application built on top

Scikit-Learn’s TfidfVectorizer (+ CosineSimilarity)

Scikit-learn is a popular machine learning library that provides a wide range of tools for various tasks, including text summarisation. One of its most popular tools for text summarisation is the TfidfVectorizer. The TfidfVectorizer is a simple yet powerful tool that computes the term frequency-inverse document frequency (TF-IDF) of each word in a document. The TF-IDF score measures the importance of a word in a document and is calculated by multiplying the term frequency (TF) of a word by the inverse document frequency (IDF) of the word. The TfidfVectorizer then generates a summary by selecting the most important sentences in the document based on their TF-IDF scores.

Generated Article Summary:

Summary Type: Extractive

Pros:

• Model is easy to implement and light weight
• Sentences being text from the article can be searched and do not suffer from hallucination,
• No pre-training or fine-tuning required, making it ideal for small-scale summarisation tasks

Cons:

• Text is not coherent or follow the flow of the text,
• Longer documents may require additional preprocessing or summarisation techniques to be effective, and may result in a summary that is too long or contains irrelevant information

spaCy

The “industrial strength” open-source library that can be used for wide variety of NLP tasks.

The abstract below has been generated by implementing Bag-of-Words (BoW) using spaCy. This model counts the frequency of unique words after excluding stop words (like “is, a, the,…”) and weighing these words by the number of words in the document.

Generated Article Summary:

PyTextRank

TextRank is a graph-based algorithm that computes the importance of sentences in a document based on the graph structure of the document. The algorithm constructs a graph where each node represents a sentence in the document, and the edges represent the similarity between sentences. The importance of a sentence is then computed based on its position in the graph and the importance of the sentences it is connected to.

Generated Article Summary:

Summary Type: Extractive

Pros:

• Model is easy to implement and light weight
• Like other extractive summarisation models it does not suffer from hallucination,
• No pre-training or fine-tuning required, making it ideal for small-scale summarisation tasks

Cons:

• Text is not coherent or follow the flow of the text,
• PyTextRank may have difficulty with complex or highly technical sentences,
• It can be difficult to interpret and explain, particularly for non-technical users