Exploring News Topics with BERTopic and BERT Embeddings¶

In this notebook we dive into BERTopic, a cutting-edge topic modeling technique, to explore any text dataset to cluster documents into topics. In this notebook we will use the BBC News dataset to explore the semantic structure of the news articles.

BERTopic doesn't rely on traditional word co-occurrence counts like LDA — instead, it uses semantic vector representations generated by powerful language models like BERT. This makes it particularly good at capturing meaning, context, and similarity between documents.

---

🌍 Why BERT Makes This Possible¶

Pre-trained models like bert-base-uncased are trained on massive corpora and thus encode a rich, general understanding of language. This means that even if the model has never seen your exact data, it can still produce embeddings that:

• Cluster similar sentences or documents together,
• Reflect nuanced semantic distinctions (e.g., between “economy” and “finance”),
• Work well even without any additional training.

These embeddings capture not just the presence of words, but their contextual meaning.

🔧 However, in some cases we might want even better clustering. That's where fine-tuning comes in:

• If we fine-tune a BERT model on a sentence similarity task (like Semantic Textual Similarity Benchmark – STSb), the embeddings become even better at mapping semantically similar texts closer together.
• In this notebook, we'll use a version already optimized for that: all-MiniLM-L6-v2 from the SentenceTransformers library.

This lets us take pre-trained general knowledge, add task-specific learning, and apply it to unsupervised exploration like topic modeling — no labels needed!

---

📋 What You'll Learn¶

Section	Description
1. Quick Demo	Run BERTopic on BBC News using all-MiniLM-L6-v2 and generate cluster summaries with a LLM via LiteLLM.
2. Visualize Results	Use BERTopic's built-in visual tools to explore topic clusters and keywords.
3. Interactive Exploration	Dive deeper into specific clusters and inspect example articles.
4. Step-by-Step Breakdown	Understand the entire BERTopic pipeline: embeddings → dimensionality reduction → clustering → topic representation.

---

🧰 Libraries We'll Use¶

• BERTopic for topic modeling
• sentence-transformers for generating document embeddings
• datasets from Hugging Face to load and clean data
• LiteLLM for language model-powered summaries
• plotly and pandas for interactive data exploration

---

💾 Load the BBC News Dataset¶

We’ll start by loading the BBC News dataset using the Hugging Face datasets library.

This dataset contains short news articles from five topics: business, entertainment, politics, sport, and tech.

Each article is represented as a short paragraph of text, and labeled with its corresponding topic. While we won’t use the labels for unsupervised topic modeling, they are useful later for evaluation and visualization.

Let’s load the data and preview a few examples. We will concatenate the test

In [1]:

from datasets import load_dataset
import pandas as pd

# Load the dataset
train_dataset = load_dataset("SetFit/bbc-news", split="train")
test_dataset = load_dataset("SetFit/bbc-news", split="test")

# Convert to pandas DataFrame for easier handling
df = pd.concat([train_dataset.to_pandas(), test_dataset.to_pandas()]).reset_index(drop=True)
df = df.rename(columns={"text": "document", "label": "true_label"})

# Show basic info
print(f"Dataset size: {len(df)} documents")
df.head()

from datasets import load_dataset import pandas as pd # Load the dataset train_dataset = load_dataset("SetFit/bbc-news", split="train") test_dataset = load_dataset("SetFit/bbc-news", split="test") # Convert to pandas DataFrame for easier handling df = pd.concat([train_dataset.to_pandas(), test_dataset.to_pandas()]).reset_index(drop=True) df = df.rename(columns={"text": "document", "label": "true_label"}) # Show basic info print(f"Dataset size: {len(df)} documents") df.head()

Dataset size: 2225 documents

Out[1]:

	document	true_label	label_text
0	wales want rugby league training wales could f...	2	sport
1	china aviation seeks rescue deal scandal-hit j...	1	business
2	rock band u2 break ticket record u2 have smash...	3	entertainment
3	markets signal brazilian recovery the brazilia...	1	business
4	tough rules for ringtone sellers firms that fl...	0	tech

Quick Intro to BERTopic + LiteLLM¶

Now let’s briefly introduce the key tools we’ll use to extract and interpret topics:

---

🧠 What is BERTopic?¶

BERTopic is a modern topic modeling framework that:

• Uses pretrained embeddings (e.g., MiniLM, BERT) to compute document similarity in a meaningful way.
• Applies UMAP for dimensionality reduction and HDBSCAN for clustering documents.
• Can plug in LLMs to generate human-readable topic representations like titles and summaries.

This helps us group documents by theme without supervision, and make those clusters interpretable using LLM-generated descriptions.

---

💬 What is LiteLLM?¶

LiteLLM is a lightweight interface that lets us query LLM APIs (like OpenAI’s GPT-4o) with ease.

We’ll use it to:

• Generate a title and a summary for each topic.
• Customize the prompt for both — so the LLM knows exactly what to generate based on the documents and keywords in a cluster.

---

🧠 Our Prompting Strategy¶

We're using two distinct prompts to refine the topic representation, to make them human-readable.

1. Title Prompt

I have a topic that contains the following documents:
[DOCUMENTS]
The topic is described by the following keywords: [KEYWORDS]

Based on the above information, please provide a concise title for this topic

Format your response as:
title: <title>

2. Summary Prompt

I have a topic that contains the following documents:
[DOCUMENTS]
The topic is described by the following keywords: [KEYWORDS]

Based on the above information, please provide a two-sentence summary of the main points

Format your response as:
summary: <two sentences>

---

🔐 Setup Note¶

Make sure your OpenAI API key is available via:

• A .env file with OPENAI_API_KEY=sk-...
• Or set it directly in your Python environment.

Now let’s build our BERTopic model using these prompts for LLM-enhanced topic summarization!

In [2]:

import os
from bertopic import BERTopic
from bertopic.representation import LiteLLM

# Create a custom prompt for title and summary
summary_prompt = """
I have a topic that contains the following documents:
[DOCUMENTS]
The topic is described by the following keywords: [KEYWORDS]

Based on the above information, please provide a two-sentence summary of the main points

Format your response as:
summary: <two sentences>
"""

title_prompt = """
I have a topic that contains the following documents:
[DOCUMENTS]
The topic is described by the following keywords: [KEYWORDS]

Based on the above information, please provide a concise title for this topic

Format your response as:
title: <title>
"""

# Create your representation model with the custom prompt
representation_model = {"title": LiteLLM(
    model="gpt-4o-mini",
    prompt=title_prompt
), "summary": LiteLLM(
    model="gpt-4o-mini",
    prompt=summary_prompt
)}

# Create our BERTopic model
topic_model = BERTopic(representation_model=representation_model, verbose=True)

import os from bertopic import BERTopic from bertopic.representation import LiteLLM # Create a custom prompt for title and summary summary_prompt = """ I have a topic that contains the following documents: [DOCUMENTS] The topic is described by the following keywords: [KEYWORDS] Based on the above information, please provide a two-sentence summary of the main points Format your response as: summary: """ title_prompt = """ I have a topic that contains the following documents: [DOCUMENTS] The topic is described by the following keywords: [KEYWORDS] Based on the above information, please provide a concise title for this topic Format your response as: title: