Four Effective Methods for Keyword Extraction from a Single Text using Python

Introduction

Keyword extraction is a fundamental task in natural language processing (NLP) that involves automatically identifying the most important or relevant words or phrases in a text. Extracting keywords can provide a quick summary of the main topics and themes of a document, which is useful for a variety of applications such as search engine optimization, document clustering and classification, topic modeling, and more.

In this article, we‘ll explore four of the most effective and easy-to-use methods for extracting keywords from a single text using Python: RAKE, YAKE, KeyBERT, and TextRank. For each method, we‘ll provide an overview of how it works and walk through a code example of applying it to a sample text. Let‘s get started!

Preparing the Text

Before we can extract keywords from our text, we need to do some basic preprocessing. This typically involves tasks like combining the title and body text (if separate), lowercasing, removing punctuation and stopwords, and tokenizing the text into individual words or phrases.

For our example, let‘s use the abstract from a scientific paper. We‘ll combine the title and abstract text as follows:

title = "VECTORIZATION OF TEXT USING DATA MINING METHODS"
abstract = "In the text mining tasks, textual representation should..."

text = title + " " + abstract

We could further preprocess the text by lowercasing, removing punctuation, etc. but for simplicity we‘ll skip that for now. With our text ready to go, let‘s take a look at the first keyword extraction method: RAKE.

Method 1: RAKE

RAKE, which stands for Rapid Automatic Keyword Extraction, is an unsupervised, domain-independent, and language-independent method for extracting keywords from individual documents. It works by analyzing the frequency of word appearance and its co-occurrence with other words in the text.

Here‘s how we can use the RAKE implementation from the rake-nltk library to extract keywords:

from rake_nltk import Rake

r = Rake()
r.extract_keywords_from_text(text)

keywords = r.get_ranked_phrases_with_scores()
print(keywords[:10])

This extracts the top 10 ranked keyword phrases along with their scores. In our case, the output includes relevant terms like "text mining tasks", "textual representation", and "data mining methods", which align well with the human-labeled keywords.

One advantage of RAKE is that it‘s very fast and easy to use. However, it may sometimes extract incomplete phrases or less meaningful words. This leads us to the next method, YAKE, which aims to improve on some of RAKE‘s shortcomings.

Method 2: YAKE

YAKE, or Yet Another Keyword Extractor, is an unsupervised approach that relies on statistical features extracted from single documents to identify the most relevant keywords. It does this by considering factors such as casing, word position, word frequency, and more.

To use YAKE in Python, we can leverage the yake library:

import yake

kw_extractor = yake.KeywordExtractor()
keywords = kw_extractor.extract_keywords(text)

for keyword, score in keywords[:10]:
    print(f"{keyword}: {score}")

This prints out the top 10 keywords along with their relevance scores. Lower scores indicate higher relevance. Some of the top keywords extracted include "text mining", "data mining methods", and "textual representation", which match closely with the gold standard labels.

YAKE tends to extract more meaningful and complete phrases compared to RAKE. It‘s also relatively fast and easy to use. However, it may sometimes assign lower scores to relevant but rare keywords.

Method 3: KeyBERT

KeyBERT is a keyword extraction technique that leverages BERT document embeddings and cosine similarity to find the most relevant keywords and phrases in a document. It first generates document-level embeddings using a pre-trained Sentence-BERT model, then creates candidate keywords/phrases and embeds them with the same model. Finally, it uses cosine similarity between the document and keyword embeddings to select the top N most similar keywords.

Here‘s how to use KeyBERT in Python:

from keybert import KeyBERT

model = KeyBERT(‘distilbert-base-nli-mean-tokens‘)
keywords = model.extract_keywords(text, keyphrase_ngram_range=(1, 3), stop_words=‘english‘, top_n=10)

print(keywords)

This extracts the top 10 keywords of length 1-3 tokens after removing stopwords. The resulting keywords for our example are quite good and include very relevant terms like "text vectorization methods", "data mining methods", and "text mining".

A major benefit of KeyBERT is that it captures more semantic and contextual information by using pre-trained BERT embeddings. This allows it to extract topically relevant terms that may not necessarily have high frequency in the text. However, KeyBERT is slower than methods like RAKE and YAKE due to the neural network computations.

Method 4: TextRank

TextRank is a graph-based ranking model for text processing that can be used for keyword and sentence extraction. It represents text as a graph, with words as nodes and co-occurrence frequency as edge weights, then runs the PageRank algorithm on the graph to score each word. The top-ranked words are chosen as keywords.

To apply TextRank, we‘ll use the summa library:

from summa import keywords

keywords = keywords.keywords(text, scores=True)
print(keywords[:10])

This extracts the top 10 keywords along with their relevance scores. The TextRank output includes terms like "vectorization", "concepts", and "data mining", which cover some of the main topics. However, it tends to extract more unigram keywords compared to the previous methods.

TextRank provides a different approach by considering the structure of the text as a graph. It‘s relatively simple and unsupervised like RAKE and YAKE. However, sometimes it may extract less meaningful individual words rather than complete phrases.

Comparison of Methods

To summarize and compare the results of the four keyword extraction methods, let‘s compile the top 5 keywords from each:

We can see that there is some overlap between the methods, with "text mining" and "data mining methods" appearing in several of the top 5 lists. However, we also observe differences, such as KeyBERT‘s extraction of longer phrases like "text vectorization methods" and YAKE‘s inclusion of "curse of dimensionality".

In general, RAKE and YAKE are very fast and easy to use, while KeyBERT provides more contextually relevant keywords due to its use of BERT embeddings. TextRank offers a novel graph-based approach, but tends to extract more individual words than meaningful phrases.

The choice of which method to use depends on your specific application and priorities. If speed is a concern, RAKE or YAKE may be good options. If you want to capture more semantic information, KeyBERT is a powerful choice. And if you‘re interested in exploring a graph-based approach, TextRank is worth considering. You may also choose to run multiple methods and combine or compare their outputs.

Conclusion

In this article, we introduced four effective methods for automatically extracting keywords from a single text using Python: RAKE, YAKE, KeyBERT, and TextRank. We provided an overview of each method and walked through code examples of how to apply them to a sample text.

Keyword extraction is a valuable tool for quickly summarizing the main topics and themes of a document. It can be used for a wide range of applications such as topic modeling, document retrieval, clustering, and classification. We encourage you to try out these methods on your own datasets and see which one works best for your use case.

There are many other keyword extraction techniques beyond the four covered here, so we encourage you to explore further. Some other methods to look into include TF-IDF, KPMiner, and SGRank. Keyword extraction is an active area of NLP research with new approaches continuously emerging.

References

• RAKE: https://github.com/csurfer/rake-nltk
• YAKE: https://github.com/LIAAD/yake
• KeyBERT: https://github.com/MaartenGr/KeyBERT
• TextRank: https://github.com/summanlp/textrank