📚 Embedding Evaluation: Intrinsic and Extrinsic¶
In this notebook, we explore how different word embedding models capture meaning — both through direct inspection (intrinsic evaluation) and by testing their performance in downstream tasks (extrinsic evaluation).
🧭 Goal¶
We'll compare Word2Vec, FastText, and ELMo embeddings on how well they represent word semantics, structure, and context — especially for polysemous words (words with multiple meanings, like "bank" or "light").
By the end, you’ll understand:
- Which models best capture word similarity, relationships, and contextual meaning
- How embeddings differ when used in real-world NLP tasks
- Why newer contextual models like ELMo offer major advantages for dynamic, context-aware representations
📌 Table of Contents¶
- Introduction to Embedding Models
- Static vs Contextual Embeddings
- Loading and Preprocessing Embeddings
- Word2Vec
- FastText
- ELMo
- Intrinsic Evaluation
- Word Similarity
- Word Analogies
- Semantic Clustering
- Dimensionality Reduction (PCA / t-SNE)
- Extrinsic Evaluation
- Named Entity Recognition
- Text classification
- Conclusion
- Summary and model comparison
🔍 Highlight: The ELMo Cone of Context¶
Toward the end of this notebook, we’ll visualize the same word in different sentences using ELMo embeddings — revealing how its position in embedding space shifts depending on its usage.
Think of this as a "cone of meaning", where the same word — like "pitch" or "cell" — points in different semantic directions depending on context.
This helps illustrate why contextual embeddings matter, and what they offer that static models like Word2Vec or FastText cannot.
1. 🧠 Introduction to Embedding Models¶
In Natural Language Processing, word embeddings are dense vector representations of words that aim to capture their semantic meaning.
There are two main types of word embeddings:
🔷 Static Embeddings¶
These models assign a single vector per word, regardless of context.
- Word2Vec: Predicts context words (Skip-gram) or the center word (CBOW) based on co-occurrence.
- FastText: Extension of Word2Vec that incorporates subword information, improving out-of-vocabulary handling and morphological understanding.
❗Limitation: They cannot disambiguate polysemous words (e.g., "bank" in river bank vs investment bank).
🌀 Contextual Embeddings¶
These models generate different vectors for the same word depending on its sentence context.
- ELMo (Embeddings from Language Models): Uses deep bidirectional LSTMs to compute contextual word representations.
- Trained on entire sentences — a word’s embedding changes depending on surrounding words.
✅ Contextual embeddings like ELMo are powerful for disambiguating word meaning dynamically.
In the next section, we’ll load and prepare pre-trained embeddings for Word2Vec, FastText, and ELMo to evaluate how each performs across different tasks.
2. 📦 Loading and Preprocessing Embeddings¶
We will now load pre-trained embeddings from three different sources:
- ✅ Word2Vec (via Hugging Face): Static embeddings trained on Google News
- Model:
NeuML/word2vec
- Model:
- ✅ FastText (via Hugging Face): Subword-aware static embeddings
- Model:
facebook/fasttext-en-vectors
- Model:
- ✅ ELMo (via Flair): Contextual embeddings that dynamically adapt to sentence context
- Setup: See Flair ELMo Docs
It will take a bit of time to load the models, so go take a break and grab a cup of tea. And be sure to have at least 15Gb of SSD space on your machine or remote instance.
We'll define utilities to:
- Access embeddings for any word
- Handle out-of-vocabulary words (especially important for Word2Vec)
- Extract contextual embeddings from full sentences using ELMo
2.1 Install Required Libraries¶
In [1]:
Copied!
# Import necessary libraries
import numpy as np
import tensorflow as tf
import tensorflow_hub as hub
from huggingface_hub import hf_hub_download
from staticvectors import StaticVectors
import fasttext
# Load Word2Vec model
print("Loading Word2Vec model...")
word2vec_model = StaticVectors("neuml/word2vec")
# Load FastText model
print("Loading FastText model...")
fasttext_model_path = hf_hub_download(repo_id="facebook/fasttext-en-vectors", filename="model.bin")
fasttext_model = fasttext.load_model(fasttext_model_path)
# Load ELMo embeddings from TensorFlow Hub
print("Loading ELMo embeddings from TensorFlow Hub...")
elmo_model = hub.load("https://tfhub.dev/google/elmo/3")
print("All models loaded successfully!")
# Import necessary libraries
import numpy as np
import tensorflow as tf
import tensorflow_hub as hub
from huggingface_hub import hf_hub_download
from staticvectors import StaticVectors
import fasttext
# Load Word2Vec model
print("Loading Word2Vec model...")
word2vec_model = StaticVectors("neuml/word2vec")
# Load FastText model
print("Loading FastText model...")
fasttext_model_path = hf_hub_download(repo_id="facebook/fasttext-en-vectors", filename="model.bin")
fasttext_model = fasttext.load_model(fasttext_model_path)
# Load ELMo embeddings from TensorFlow Hub
print("Loading ELMo embeddings from TensorFlow Hub...")
elmo_model = hub.load("https://tfhub.dev/google/elmo/3")
print("All models loaded successfully!")
Loading Word2Vec model... Loading FastText model... Loading ELMo embeddings from TensorFlow Hub... All models loaded successfully!
2.2 Define the utility functions¶
Let's define the utility functions to get the embeddings for the three models. We use a dummy tokenizer for the Elmo model.
In [2]:
Copied!
def get_word2vec_embedding(word):
try:
# StaticVectors expects a list of words and returns a list of embeddings
embeddings = word2vec_model.embeddings([word])
return embeddings[0]
except Exception as e:
print(f"Error getting Word2Vec embedding for {word}: {e}")
return None
def get_fasttext_embedding(word):
try:
return fasttext_model[word]
except Exception as e:
print(f"Error getting FastText embedding for {word}: {e}")
return None # FastText handles OOV, but just in case
# First, let's update the get_elmo_embedding function to make it more robust
def get_elmo_embedding(sentence, target_word):
"""
Extract contextual embedding for a target word using ELMo via TensorFlow Hub.
Args:
sentence (str): A sentence containing the target word
target_word (str): The word to get embedding for
Returns:
numpy.ndarray: The ELMo embedding for the target word
"""
# Tokenize the sentence (more robust tokenization)
words = sentence.lower().split()
target_word = target_word.lower()
# Find the position of the target word
try:
target_positions = [i for i, word in enumerate(words) if word.strip('.,:;!?') == target_word]
if not target_positions:
# Try with exact match as fallback
target_positions = [i for i, word in enumerate(words) if word == target_word]
if not target_positions:
print(f"Warning: '{target_word}' not found in sentence: '{sentence}'")
return None
target_pos = target_positions[0] # Use the first occurrence
except Exception as e:
print(f"Error finding '{target_word}' in '{sentence}': {e}")
return None
# Get ELMo embeddings for the sentence
try:
embeddings = elmo_model.signatures["default"](
tf.constant([sentence])
)
# Extract the embedding for the target word
# The ELMo model returns word vectors with shape [batch_size, max_length, embedding_size]
word_embedding = embeddings["word_emb"][0, target_pos, :].numpy()
return word_embedding
except Exception as e:
print(f"Error getting ELMo embedding: {e}")
return None
def get_word2vec_embedding(word):
try:
# StaticVectors expects a list of words and returns a list of embeddings
embeddings = word2vec_model.embeddings([word])
return embeddings[0]
except Exception as e:
print(f"Error getting Word2Vec embedding for {word}: {e}")
return None
def get_fasttext_embedding(word):
try:
return fasttext_model[word]
except Exception as e:
print(f"Error getting FastText embedding for {word}: {e}")
return None # FastText handles OOV, but just in case
# First, let's update the get_elmo_embedding function to make it more robust
def get_elmo_embedding(sentence, target_word):
"""
Extract contextual embedding for a target word using ELMo via TensorFlow Hub.
Args:
sentence (str): A sentence containing the target word
target_word (str): The word to get embedding for
Returns:
numpy.ndarray: The ELMo embedding for the target word
"""
# Tokenize the sentence (more robust tokenization)
words = sentence.lower().split()
target_word = target_word.lower()
# Find the position of the target word
try:
target_positions = [i for i, word in enumerate(words) if word.strip('.,:;!?') == target_word]
if not target_positions:
# Try with exact match as fallback
target_positions = [i for i, word in enumerate(words) if word == target_word]
if not target_positions:
print(f"Warning: '{target_word}' not found in sentence: '{sentence}'")
return None
target_pos = target_positions[0] # Use the first occurrence
except Exception as e:
print(f"Error finding '{target_word}' in '{sentence}': {e}")
return None
# Get ELMo embeddings for the sentence
try:
embeddings = elmo_model.signatures["default"](
tf.constant([sentence])
)
# Extract the embedding for the target word
# The ELMo model returns word vectors with shape [batch_size, max_length, embedding_size]
word_embedding = embeddings["word_emb"][0, target_pos, :].numpy()
return word_embedding
except Exception as e:
print(f"Error getting ELMo embedding: {e}")
return None
2.3 Quick Test: Embedding Lookup Example¶
Before diving into evaluation, let’s test our setup by querying the embeddings for the word "bank" using all three models:
- Word2Vec and FastText should return a static vector regardless of context.
- ELMo should return different vectors depending on the sentence (e.g.,
"river bank"vs"investment bank").
We’ll also check:
- Embedding shapes (should be 300 for Word2Vec and FastText, 1024 for ELMo)
- Cosine similarity between ELMo vectors in different contexts
In [3]:
Copied!
from sklearn.metrics.pairwise import cosine_similarity
# Static word
word = "point"
# Contexts for ELMo
sentence1 = "I will show you a valid point of reference and talk to the point"
sentence2 = "Where have you placed the point"
# --- Get embeddings ---
w2v_vec = get_word2vec_embedding(word)
ft_vec = get_fasttext_embedding(word)
elmo_vec1 = get_elmo_embedding(sentence1, word)
elmo_vec2 = get_elmo_embedding(sentence2, word)
# --- Display results ---
print(f"Word2Vec '{word}' shape: {w2v_vec.shape} | Sample: {w2v_vec[:5]}")
print(f"FastText '{word}' shape: {ft_vec.shape} | Sample: {ft_vec[:5]}")
print(f"ELMo '{word}' (context 1) shape: {elmo_vec1.shape} | Sample: {elmo_vec1[:5]}")
print(f"ELMo '{word}' (context 2) shape: {elmo_vec2.shape} | Sample: {elmo_vec2[:5]}")
# --- Compare contextual differences ---
similarity = cosine_similarity([elmo_vec1], [elmo_vec2])[0][0]
print(f"\nCosine similarity between ELMo embeddings (context 1 vs 2): {similarity:.4f}")
from sklearn.metrics.pairwise import cosine_similarity
# Static word
word = "point"
# Contexts for ELMo
sentence1 = "I will show you a valid point of reference and talk to the point"
sentence2 = "Where have you placed the point"
# --- Get embeddings ---
w2v_vec = get_word2vec_embedding(word)
ft_vec = get_fasttext_embedding(word)
elmo_vec1 = get_elmo_embedding(sentence1, word)
elmo_vec2 = get_elmo_embedding(sentence2, word)
# --- Display results ---
print(f"Word2Vec '{word}' shape: {w2v_vec.shape} | Sample: {w2v_vec[:5]}")
print(f"FastText '{word}' shape: {ft_vec.shape} | Sample: {ft_vec[:5]}")
print(f"ELMo '{word}' (context 1) shape: {elmo_vec1.shape} | Sample: {elmo_vec1[:5]}")
print(f"ELMo '{word}' (context 2) shape: {elmo_vec2.shape} | Sample: {elmo_vec2[:5]}")
# --- Compare contextual differences ---
similarity = cosine_similarity([elmo_vec1], [elmo_vec2])[0][0]
print(f"\nCosine similarity between ELMo embeddings (context 1 vs 2): {similarity:.4f}")
Word2Vec 'point' shape: (300,) | Sample: [ 0.05756511 -0.00306746 0.02935592 0.08577431 -0.05274891] FastText 'point' shape: (300,) | Sample: [ 0.03026813 0.06170474 0.05471474 0.08540645 -0.05659813] ELMo 'point' (context 1) shape: (512,) | Sample: [ 0.12824385 0.14562231 -0.20140205 0.04809816 -0.50635725] ELMo 'point' (context 2) shape: (512,) | Sample: [ 0.12824386 0.14562228 -0.2014019 0.04809807 -0.50635725] Cosine similarity between ELMo embeddings (context 1 vs 2): 1.0000
🧪 Sanity Check: Embedding Consistency Across Contexts¶
We tested how the word "point" is represented across models and contexts.
📊 Results Summary:¶
| Model | Embedding Type | Dimensionality | Context Sensitivity |
|---|---|---|---|
| Word2Vec | Static | 300 | ❌ Same vector always |
| FastText | Static + subwords | 300 | ❌ Context-agnostic |
| ELMo | Contextual | 512 | ✅ Should vary with context |
Actual Output:¶
- ✅ Word2Vec and FastText returned expected 300d vectors.
- ⚠️ ELMo returned 512d vectors but both were nearly identical, with cosine similarity = 1.0000.
🤔 Why Didn’t ELMo Differentiate the Contexts?¶
This can happen due to:
- Tokenization mismatch: Our tokenizer may not segment
"point"correctly if it's repeated or tokenized inconsistently. - Low-sensitivity contexts: If both contexts use
"point"similarly or appear too short, it might produce nearly identical embeddings. - Backend artifact: ELMo via TF-hub may use averaged layers or frozen weights, which dampen context variation.
✅ We’ll run more varied and longer examples in later sections to highlight ELMo's dynamic behavior more clearly — including a "cone of meaning" visualization.
🧪 3. Intrinsic Evaluation Benchmarks¶
Now that our embeddings are loaded and verified, we’ll begin evaluating their quality using standard intrinsic NLP benchmarks.
📏 What We'll Evaluate¶
Word Similarity
Compare model similarity scores to human judgments (e.g., WordSim353)Word Analogy Tasks
Can the model solve"king - man + woman = queen"?Semantic Clustering
Visual inspection using PCA / t-SNE to see if similar concepts are groupedDimensionality Reduction & Probing
Check how separable semantic categories are in low dimensions
📚 Datasets & Benchmarks We'll Use¶
| Task | Benchmark Dataset | Format |
|---|---|---|
| Word Similarity | WordSim-353, SimLex-999 | Word pairs + scores |
| Analogies | Google Analogies (Word2Vec-style) | 4-word tuples |
| Clustering | Custom categories (e.g., colors, cities, family) | Word lists |
These allow us to compare models quantitatively and visually, building an intuition for what kind of meaning each model captures.
Next up: let’s implement the Word Similarity evaluation and compare our three models!
3.1 🧩 Word Similarity Evaluation¶
In this task, we assess how well each embedding model captures human-perceived semantic similarity between word pairs.
We’ll use the WordSim-353 benchmark, which contains 353 word pairs annotated with human similarity scores (from 0 to 10).
🧪 Evaluation Method¶
- For each word pair, compute the cosine similarity between their embeddings.
- Compare model similarity scores to human scores using Spearman's rank correlation.
- Higher correlation = better alignment with human judgments.
📌 Note on ELMo¶
Since ELMo is contextual, we’ll provide each word as a standalone sentence (e.g., "apple" → "I like apple.")
While not ideal, this allows for fair comparison to static embeddings in this evaluation.
Later, we’ll evaluate ELMo’s strength in context-aware settings.
In [4]:
Copied!
import pandas as pd
from datasets import load_dataset
from scipy.stats import spearmanr
import nltk
nltk.download("words")
# Load WordSim-353 from Hugging Face
print("Loading WordSim-353 from Hugging Face...")
dataset = load_dataset("almogtavor/WordSim353")
wordsim = pd.DataFrame(dataset['train'])
# Rename columns to match expected format
wordsim.columns = ["word1", "word2", "human_score"]
print(f"✅ Loaded {len(wordsim)} word pairs from WordSim-353")
wordsim.head()
import pandas as pd
from datasets import load_dataset
from scipy.stats import spearmanr
import nltk
nltk.download("words")
# Load WordSim-353 from Hugging Face
print("Loading WordSim-353 from Hugging Face...")
dataset = load_dataset("almogtavor/WordSim353")
wordsim = pd.DataFrame(dataset['train'])
# Rename columns to match expected format
wordsim.columns = ["word1", "word2", "human_score"]
print(f"✅ Loaded {len(wordsim)} word pairs from WordSim-353")
wordsim.head()
[nltk_data] Downloading package words to /Users/agomberto/nltk_data... [nltk_data] Package words is already up-to-date!
Loading WordSim-353 from Hugging Face...
python(71809) MallocStackLogging: can't turn off malloc stack logging because it was not enabled.
✅ Loaded 353 word pairs from WordSim-353
Out[4]:
| word1 | word2 | human_score | |
|---|---|---|---|
| 0 | admission | ticket | 5.5360 |
| 1 | alcohol | chemistry | 4.1250 |
| 2 | aluminum | metal | 6.6250 |
| 3 | announcement | effort | 2.0625 |
| 4 | announcement | news | 7.1875 |
🔧 Step 1: Computing Cosine Similarity Between Word Pairs¶
We define a helper function compute_similarity() to:
- Retrieve embeddings for two words
- Compute cosine similarity between them
- Handle special formatting for ELMo (requires fake context sentences)
This function supports all three models: Word2Vec, FastText, and ELMo.
In [5]:
Copied!
import numpy as np
from sklearn.metrics.pairwise import cosine_similarity
import matplotlib.pyplot as plt
import seaborn as sns
def compute_similarity(row, model_type):
w1, w2 = row["word1"], row["word2"]
if model_type == "word2vec":
vec1, vec2 = get_word2vec_embedding(w1), get_word2vec_embedding(w2)
elif model_type == "fasttext":
vec1, vec2 = get_fasttext_embedding(w1), get_fasttext_embedding(w2)
elif model_type == "elmo":
sent1 = f"This is {w1}."
sent2 = f"This is {w2}."
try:
vec1 = get_elmo_embedding(sent1, w1)
vec2 = get_elmo_embedding(sent2, w2)
except:
return np.nan
else:
return np.nan
if vec1 is None or vec2 is None:
return np.nan
return cosine_similarity([vec1], [vec2])[0][0]
import numpy as np
from sklearn.metrics.pairwise import cosine_similarity
import matplotlib.pyplot as plt
import seaborn as sns
def compute_similarity(row, model_type):
w1, w2 = row["word1"], row["word2"]
if model_type == "word2vec":
vec1, vec2 = get_word2vec_embedding(w1), get_word2vec_embedding(w2)
elif model_type == "fasttext":
vec1, vec2 = get_fasttext_embedding(w1), get_fasttext_embedding(w2)
elif model_type == "elmo":
sent1 = f"This is {w1}."
sent2 = f"This is {w2}."
try:
vec1 = get_elmo_embedding(sent1, w1)
vec2 = get_elmo_embedding(sent2, w2)
except:
return np.nan
else:
return np.nan
if vec1 is None or vec2 is None:
return np.nan
return cosine_similarity([vec1], [vec2])[0][0]
🧪 Step 2: Compute Similarity Scores for WordSim-353¶
We now apply the similarity function to all 353 word pairs using:
- Word2Vec
- FastText
- ELMo
We store the scores in new columns and prepare the data for analysis.
In [6]:
Copied!
# Create a copy of WordSim for evaluations
results = wordsim.copy()
print("🔍 Evaluating Word2Vec...")
results["w2v_sim"] = results.apply(lambda row: compute_similarity(row, "word2vec"), axis=1)
print(results["w2v_sim"])
print("🔍 Evaluating FastText...")
results["ft_sim"] = results.apply(lambda row: compute_similarity(row, "fasttext"), axis=1)
print(results["ft_sim"])
print("🔍 Evaluating ELMo (context-free sentences)...")
results["elmo_sim"] = results.apply(lambda row: compute_similarity(row, "elmo"), axis=1)
print(results["elmo_sim"])
# Drop rows with missing vectors
results_clean = results.dropna()
# Create a copy of WordSim for evaluations
results = wordsim.copy()
print("🔍 Evaluating Word2Vec...")
results["w2v_sim"] = results.apply(lambda row: compute_similarity(row, "word2vec"), axis=1)
print(results["w2v_sim"])
print("🔍 Evaluating FastText...")
results["ft_sim"] = results.apply(lambda row: compute_similarity(row, "fasttext"), axis=1)
print(results["ft_sim"])
print("🔍 Evaluating ELMo (context-free sentences)...")
results["elmo_sim"] = results.apply(lambda row: compute_similarity(row, "elmo"), axis=1)
print(results["elmo_sim"])
# Drop rows with missing vectors
results_clean = results.dropna()
🔍 Evaluating Word2Vec...
0 0.436883
1 0.088762
2 0.637576
3 0.231306
4 0.467333
...
348 0.258439
349 0.362721
350 0.260148
351 0.417516
352 0.096587
Name: w2v_sim, Length: 353, dtype: float32
🔍 Evaluating FastText...
0 0.448249
1 0.264830
2 0.612454
3 0.226267
4 0.487779
...
348 0.373049
349 0.552444
350 0.238878
351 0.443536
352 0.244587
Name: ft_sim, Length: 353, dtype: float32
🔍 Evaluating ELMo (context-free sentences)...
0 0.573292
1 0.637792
2 0.665230
3 0.660678
4 0.639659
...
348 0.563930
349 0.474886
350 0.591700
351 0.635695
352 0.569608
Name: elmo_sim, Length: 353, dtype: float32
📊 Step 3: Compare with Human Judgments (Spearman Correlation)¶
We evaluate how well each model’s similarity scores align with human judgments using Spearman's rank correlation.
In [7]:
Copied!
# Calculate Spearman correlations
w2v_corr = spearmanr(results_clean["human_score"], results_clean["w2v_sim"]).correlation
ft_corr = spearmanr(results_clean["human_score"], results_clean["ft_sim"]).correlation
elmo_corr = spearmanr(results_clean["human_score"], results_clean["elmo_sim"]).correlation
# Calculate Spearman correlations
w2v_corr = spearmanr(results_clean["human_score"], results_clean["w2v_sim"]).correlation
ft_corr = spearmanr(results_clean["human_score"], results_clean["ft_sim"]).correlation
elmo_corr = spearmanr(results_clean["human_score"], results_clean["elmo_sim"]).correlation
⚖️ Step 4: Normalize Model Scores¶
To visually compare model predictions with human ratings (0–10), we scale all similarity scores to the same range using min-max normalization.
In [8]:
Copied!
# Function to normalize scores to 0-10 range for better comparison with human scores
def normalize_scores(scores):
min_val, max_val = min(scores), max(scores)
return 10 * (scores - min_val) / (max_val - min_val)
# Normalize similarity scores
results_clean["w2v_sim_norm"] = normalize_scores(results_clean["w2v_sim"])
results_clean["ft_sim_norm"] = normalize_scores(results_clean["ft_sim"])
results_clean["elmo_sim_norm"] = normalize_scores(results_clean["elmo_sim"])
# Function to normalize scores to 0-10 range for better comparison with human scores
def normalize_scores(scores):
min_val, max_val = min(scores), max(scores)
return 10 * (scores - min_val) / (max_val - min_val)
# Normalize similarity scores
results_clean["w2v_sim_norm"] = normalize_scores(results_clean["w2v_sim"])
results_clean["ft_sim_norm"] = normalize_scores(results_clean["ft_sim"])
results_clean["elmo_sim_norm"] = normalize_scores(results_clean["elmo_sim"])
🔍 Step 5: Showcase Example Word Pairs¶
We highlight:
- Top 5 most similar pairs (according to humans)
- 5 least similar pairs
- Polysemous word examples (e.g., "bank", "cell")
- Biggest disagreements between model predictions and human scores
In [9]:
Copied!
# Select interesting examples for demonstration
# 1. High similarity pairs
high_sim = results_clean.nlargest(5, "human_score")
# 2. Low similarity pairs
low_sim = results_clean.nsmallest(5, "human_score")
# 3. Polysemous examples (e.g., words like 'bank', 'cell', etc.)
polysemous = results_clean[results_clean["word1"].isin(["bank", "cell", "bat", "pitcher", "crane"]) |
results_clean["word2"].isin(["bank", "cell", "bat", "pitcher", "crane"])]
# 4. Cases where models disagree with humans
disagreement = results_clean.copy()
disagreement["w2v_diff"] = abs(disagreement["human_score"] - disagreement["w2v_sim_norm"])
disagreement["ft_diff"] = abs(disagreement["human_score"] - disagreement["ft_sim_norm"])
disagreement["elmo_diff"] = abs(disagreement["human_score"] - disagreement["elmo_sim_norm"])
disagreement["avg_diff"] = (disagreement["w2v_diff"] + disagreement["ft_diff"] + disagreement["elmo_diff"]) / 3
disagreement = disagreement.nlargest(5, "avg_diff")
# Combine examples
examples = pd.concat([high_sim, low_sim, polysemous, disagreement]).drop_duplicates()
# Print summary of results
print("\n🔎 Example Word Pairs and Model Predictions:")
pd.set_option('display.max_rows', None)
example_display = examples[["word1", "word2", "human_score", "w2v_sim_norm", "ft_sim_norm", "elmo_sim_norm"]]
example_display.columns = ["Word 1", "Word 2", "Human Score", "Word2Vec", "FastText", "ELMo"]
print(example_display.sort_values("Human Score", ascending=False))
# Select interesting examples for demonstration
# 1. High similarity pairs
high_sim = results_clean.nlargest(5, "human_score")
# 2. Low similarity pairs
low_sim = results_clean.nsmallest(5, "human_score")
# 3. Polysemous examples (e.g., words like 'bank', 'cell', etc.)
polysemous = results_clean[results_clean["word1"].isin(["bank", "cell", "bat", "pitcher", "crane"]) |
results_clean["word2"].isin(["bank", "cell", "bat", "pitcher", "crane"])]
# 4. Cases where models disagree with humans
disagreement = results_clean.copy()
disagreement["w2v_diff"] = abs(disagreement["human_score"] - disagreement["w2v_sim_norm"])
disagreement["ft_diff"] = abs(disagreement["human_score"] - disagreement["ft_sim_norm"])
disagreement["elmo_diff"] = abs(disagreement["human_score"] - disagreement["elmo_sim_norm"])
disagreement["avg_diff"] = (disagreement["w2v_diff"] + disagreement["ft_diff"] + disagreement["elmo_diff"]) / 3
disagreement = disagreement.nlargest(5, "avg_diff")
# Combine examples
examples = pd.concat([high_sim, low_sim, polysemous, disagreement]).drop_duplicates()
# Print summary of results
print("\n🔎 Example Word Pairs and Model Predictions:")
pd.set_option('display.max_rows', None)
example_display = examples[["word1", "word2", "human_score", "w2v_sim_norm", "ft_sim_norm", "elmo_sim_norm"]]
example_display.columns = ["Word 1", "Word 2", "Human Score", "Word2Vec", "FastText", "ELMo"]
print(example_display.sort_values("Human Score", ascending=False))
🔎 Example Word Pairs and Model Predictions:
Word 1 Word 2 Human Score Word2Vec FastText ELMo
333 tiger tiger 10.0000 10.000000 10.000000 10.000000
33 car automobile 9.3690 5.925843 7.082528 6.023292
193 money cash 9.0315 6.232118 6.856684 5.467494
186 midday noon 8.9375 5.621416 6.555274 4.570495
192 money cash 8.8750 6.232118 6.856684 5.467494
92 dollar buck 8.5625 2.718457 4.343258 3.851626
123 fuck sex 8.2500 2.397155 4.885740 1.801877
13 asylum madhouse 8.0625 2.682504 4.406788 4.486266
194 money currency 7.9375 1.777553 3.682916 4.988157
17 bank money 6.6250 2.768471 4.106092 4.784755
35 cell phone 6.2500 3.013003 6.191161 3.653709
191 money bank 6.0000 2.768471 4.106092 4.784755
71 cup tableware 6.0000 2.117868 2.498245 0.782586
22 bird crane 4.8125 3.175153 4.428065 3.381423
58 crane implement 2.1250 0.437181 0.742416 2.478549
155 king cabbage 0.6250 1.366680 2.084829 2.865413
99 drink mother 0.5625 1.745229 1.591457 4.075823
97 drink ear 0.3750 1.125240 1.141692 3.356437
258 professor cucumber 0.3125 0.764474 1.071865 2.760647
96 drink car 0.3125 2.614916 2.822372 4.857550
📈 Step 6: Global Evaluation Summary¶
Here are the overall Spearman correlations between each model and human similarity judgments.
In [10]:
Copied!
# Global evaluation
print(f"\n📊 Spearman Correlations with Human Scores:")
print(f"Word2Vec: {w2v_corr:.4f}")
print(f"FastText: {ft_corr:.4f}")
print(f"ELMo: {elmo_corr:.4f}")
# Global evaluation
print(f"\n📊 Spearman Correlations with Human Scores:")
print(f"Word2Vec: {w2v_corr:.4f}")
print(f"FastText: {ft_corr:.4f}")
print(f"ELMo: {elmo_corr:.4f}")
📊 Spearman Correlations with Human Scores: Word2Vec: 0.6876 FastText: 0.7506 ELMo: 0.4521
📊 Step 7: Visualizations¶
We plot:
- Bar chart of Spearman correlations for each model
- Scatter plot comparing human vs model scores to show linearity or bias
In [11]:
Copied!
# Visualization
plt.figure(figsize=(15, 5))
# Plot 1: Correlation with human scores
plt.subplot(1, 2, 1)
correlations = [w2v_corr, ft_corr, elmo_corr]
models = ["Word2Vec", "FastText", "ELMo"]
plt.bar(models, correlations, color=["skyblue", "lightgreen", "coral"])
plt.title("Correlation with Human Similarity Judgments")
plt.ylabel("Spearman Correlation")
plt.ylim(0, 1)
for i, v in enumerate(correlations):
plt.text(i, v + 0.05, f"{v:.4f}", ha='center')
# Plot 2: Scatter plot comparing human vs model scores for a sample
plt.subplot(1, 2, 2)
plt.scatter(results_clean["human_score"], results_clean["w2v_sim_norm"], alpha=0.5, label="Word2Vec")
plt.scatter(results_clean["human_score"], results_clean["ft_sim_norm"], alpha=0.5, label="FastText")
plt.scatter(results_clean["human_score"], results_clean["elmo_sim_norm"], alpha=0.5, label="ELMo")
plt.plot([0, 10], [0, 10], 'k--', alpha=0.3) # Diagonal line for perfect correlation
plt.xlabel("Human Similarity Score")
plt.ylabel("Model Predicted Score (Normalized)")
plt.title("Human vs Model Similarity Scores")
plt.legend()
plt.tight_layout()
plt.show()
# Visualization
plt.figure(figsize=(15, 5))
# Plot 1: Correlation with human scores
plt.subplot(1, 2, 1)
correlations = [w2v_corr, ft_corr, elmo_corr]
models = ["Word2Vec", "FastText", "ELMo"]
plt.bar(models, correlations, color=["skyblue", "lightgreen", "coral"])
plt.title("Correlation with Human Similarity Judgments")
plt.ylabel("Spearman Correlation")
plt.ylim(0, 1)
for i, v in enumerate(correlations):
plt.text(i, v + 0.05, f"{v:.4f}", ha='center')
# Plot 2: Scatter plot comparing human vs model scores for a sample
plt.subplot(1, 2, 2)
plt.scatter(results_clean["human_score"], results_clean["w2v_sim_norm"], alpha=0.5, label="Word2Vec")
plt.scatter(results_clean["human_score"], results_clean["ft_sim_norm"], alpha=0.5, label="FastText")
plt.scatter(results_clean["human_score"], results_clean["elmo_sim_norm"], alpha=0.5, label="ELMo")
plt.plot([0, 10], [0, 10], 'k--', alpha=0.3) # Diagonal line for perfect correlation
plt.xlabel("Human Similarity Score")
plt.ylabel("Model Predicted Score (Normalized)")
plt.title("Human vs Model Similarity Scores")
plt.legend()
plt.tight_layout()
plt.show()
🧠 Step 8: Error Analysis by Word Type¶
We categorize word pairs as:
- Concrete (objects, physical items)
- Abstract (concepts, emotions)
- Other
Then, we compute average error (|prediction − human|) for each model by category.
In [12]:
Copied!
# Additional analysis: Distribution of error by word type
print("\n📈 Error Analysis by Word Type:")
# Find abstract vs concrete words (simplified approach)
concrete_words = ["car", "book", "water", "tree", "house", "food", "computer", "dog", "chair", "phone"]
abstract_words = ["love", "time", "freedom", "idea", "peace", "happiness", "success", "truth", "theory", "beauty"]
def word_type(row):
if row["word1"] in concrete_words or row["word2"] in concrete_words:
return "Concrete"
elif row["word1"] in abstract_words or row["word2"] in abstract_words:
return "Abstract"
else:
return "Other"
results_clean["word_type"] = results_clean.apply(word_type, axis=1)
results_clean["w2v_error"] = abs(results_clean["human_score"] - results_clean["w2v_sim_norm"])
results_clean["ft_error"] = abs(results_clean["human_score"] - results_clean["ft_sim_norm"])
results_clean["elmo_error"] = abs(results_clean["human_score"] - results_clean["elmo_sim_norm"])
# Average error by word type
errors_by_type = results_clean.groupby("word_type")[["w2v_error", "ft_error", "elmo_error"]].mean()
print(errors_by_type)
# Additional analysis: Distribution of error by word type
print("\n📈 Error Analysis by Word Type:")
# Find abstract vs concrete words (simplified approach)
concrete_words = ["car", "book", "water", "tree", "house", "food", "computer", "dog", "chair", "phone"]
abstract_words = ["love", "time", "freedom", "idea", "peace", "happiness", "success", "truth", "theory", "beauty"]
def word_type(row):
if row["word1"] in concrete_words or row["word2"] in concrete_words:
return "Concrete"
elif row["word1"] in abstract_words or row["word2"] in abstract_words:
return "Abstract"
else:
return "Other"
results_clean["word_type"] = results_clean.apply(word_type, axis=1)
results_clean["w2v_error"] = abs(results_clean["human_score"] - results_clean["w2v_sim_norm"])
results_clean["ft_error"] = abs(results_clean["human_score"] - results_clean["ft_sim_norm"])
results_clean["elmo_error"] = abs(results_clean["human_score"] - results_clean["elmo_sim_norm"])
# Average error by word type
errors_by_type = results_clean.groupby("word_type")[["w2v_error", "ft_error", "elmo_error"]].mean()
print(errors_by_type)
📈 Error Analysis by Word Type:
w2v_error ft_error elmo_error
word_type
Abstract 0.937094 0.736520 1.326223
Concrete 1.645436 1.124287 1.621554
Other 1.460719 1.157303 1.489836
✅ Conclusion: Word Similarity Evaluation¶
Our comparison across Word2Vec, FastText, and ELMo on the WordSim-353 benchmark reveals several key insights:
📊 Quantitative Results (Spearman Correlation)¶
| Model | Spearman Correlation |
|---|---|
| Word2Vec | 0.6876 |
| FastText | 0.7506 ✅ |
| ELMo | 0.4521 ❌ |
- FastText leads in performance, likely due to its ability to handle rare words and morphology via subword units.
- Word2Vec performs well, but lacks FastText's flexibility for OOV words.
- ELMo, surprisingly, underperforms in this benchmark.
🤖 Why ELMo Underperforms Here¶
Although ELMo is a contextual model, we’re using it in an artificially decontextualized setting:
- We embed words in generic sentences like
"[word]" - This does not reflect ELMo's true strength, which lies in adapting meaning based on actual sentence context.
📌 This is like comparing apples and pears — contextual models aren't designed to output a fixed vector per word like static embeddings.
🧠 Error Patterns by Word Type¶
| Word Type | Avg Error (ELMo) | Insight |
|---|---|---|
| Abstract | 1.33 | ELMo struggled with vague, non-physical concepts |
| Concrete | 1.62 | Context-free usage hurt ELMo's grounding in meaning |
| Other | 1.49 | Mixed category, similar limitations |
🔍 Takeaway¶
ELMo was never meant to be used out-of-context — its strength is in modeling word meaning based on how it's used.
In the next section, we’ll move beyond static pairwise similarity and evaluate embeddings on word analogy tasks, where geometry matters more and context matters less.
3.2 🔁 Word Analogy Evaluation¶
The Google Word Analogy dataset is a classic benchmark to evaluate the relational structure of word embeddings.
🧠 What Is a Word Analogy?¶
We test whether embeddings encode consistent vector relationships:
"man" - "woman" ≈ "king" - "queen"
Vector arithmetic:king - man + woman ≈ queen
📦 Google Analogy Dataset¶
- Consists of over 19,000 analogies
- Divided into semantic (e.g. countries, capitals) and syntactic (e.g. verb tenses, pluralization) categories
- Format:
word1 word2 word3 word4(predict word4)
📐 Evaluation Metric¶
We use top-1 accuracy:
Is the predicted word (most similar to
vec2 - vec1 + vec3) equal to the actualword4?
Only Word2Vec and FastText will be evaluated here — ELMo is not well-suited because:
- It requires full-sentence context
- It does not produce a consistent vector per word, making arithmetic undefined
Let’s load the benchmark and implement the evaluation!
In [13]:
Copied!
# Load dataset from HuggingFace
print("Loading Word Analogy dataset...")
analogy_dataset = load_dataset("tomasmcz/word2vec_analogy")
print(f"✅ Loaded {len(analogy_dataset['train'])} analogy examples")
# Display sample analogies
print("\n📝 Example analogies:")
for i in range(5):
ex = analogy_dataset['train'][i]
print(f"{ex['word_a']} : {ex['word_b']} :: {ex['word_c']} : {ex['word_d']}")
# Load dataset from HuggingFace
print("Loading Word Analogy dataset...")
analogy_dataset = load_dataset("tomasmcz/word2vec_analogy")
print(f"✅ Loaded {len(analogy_dataset['train'])} analogy examples")
# Display sample analogies
print("\n📝 Example analogies:")
for i in range(5):
ex = analogy_dataset['train'][i]
print(f"{ex['word_a']} : {ex['word_b']} :: {ex['word_c']} : {ex['word_d']}")
Loading Word Analogy dataset... ✅ Loaded 19544 analogy examples 📝 Example analogies: Athens : Greece :: Baghdad : Iraq Athens : Greece :: Bangkok : Thailand Athens : Greece :: Beijing : China Athens : Greece :: Berlin : Germany Athens : Greece :: Bern : Switzerland
🧠 Defining the Evaluation Function¶
The core logic follows the classic Word2Vec analogy vector rule:
vec_d ≈ vec_b - vec_a + vec_c
We then:
- Normalize the vector
- Search for the most similar word in the reduced vocabulary
- Count how many times the prediction matches the expected
word_d
In [14]:
Copied!
from tqdm import tqdm
def evaluate_analogy(model_type, dataset, sample_size=1000, show_progress=True):
"""
Evaluate embedding models on word analogy task.
Args:
model_type: "word2vec", "fasttext", or "elmo"
dataset: The analogy dataset
sample_size: Number of examples to sample for evaluation
show_progress: Whether to show a progress bar
Returns:
dict: Evaluation results
"""
# Sample from the dataset to keep evaluation time reasonable
if sample_size and sample_size < len(dataset):
# Use dataset.select() with Python integers
indices = [int(i) for i in np.random.choice(len(dataset), sample_size, replace=False)]
evaluation_set = dataset.select(indices)
else:
evaluation_set = dataset
total = len(evaluation_set)
correct = 0
skipped = 0
# Keep track of examples where model succeeded or failed
correct_examples = []
incorrect_examples = []
# Process each analogy
iterator = tqdm(range(len(evaluation_set))) if show_progress else range(len(evaluation_set))
for i in iterator:
example = evaluation_set[i]
word_a, word_b, word_c, word_d = example['word_a'], example['word_b'], example['word_c'], example['word_d']
# Get embeddings based on model type
if model_type == "word2vec":
vec_a = get_word2vec_embedding(word_a)
vec_b = get_word2vec_embedding(word_b)
vec_c = get_word2vec_embedding(word_c)
elif model_type == "fasttext":
vec_a = get_fasttext_embedding(word_a)
vec_b = get_fasttext_embedding(word_b)
vec_c = get_fasttext_embedding(word_c)
elif model_type == "elmo":
# For ELMo, create minimal contexts for each word
template = "The word {} is important."
try:
vec_a = get_elmo_embedding(template.format(word_a), word_a)
vec_b = get_elmo_embedding(template.format(word_b), word_b)
vec_c = get_elmo_embedding(template.format(word_c), word_c)
vec_d = get_elmo_embedding(template.format(word_d), word_d)
# For ELMo, check if d is closer to (b-a+c) than a, b, c are
if vec_a is None or vec_b is None or vec_c is None or vec_d is None:
skipped += 1
continue
target_vec = vec_b - vec_a + vec_c
target_vec = target_vec / np.linalg.norm(target_vec)
# Normalize vectors
vec_a_norm = vec_a / np.linalg.norm(vec_a)
vec_b_norm = vec_b / np.linalg.norm(vec_b)
vec_c_norm = vec_c / np.linalg.norm(vec_c)
vec_d_norm = vec_d / np.linalg.norm(vec_d)
# Calculate similarities
similarity_d = np.dot(target_vec, vec_d_norm)
similarities = [
np.dot(target_vec, vec_a_norm),
np.dot(target_vec, vec_b_norm),
np.dot(target_vec, vec_c_norm)
]
# If target is closer to d than to a, b, c, consider it correct
if similarity_d > max(similarities):
correct += 1
correct_examples.append((word_a, word_b, word_c, word_d, word_d))
else:
incorrect_examples.append((word_a, word_b, word_c, word_d, "unknown"))
continue
except Exception as e:
skipped += 1
continue
else:
raise ValueError(f"Unknown model type: {model_type}")
# Skip if any embedding is not available (for Word2Vec and FastText)
if vec_a is None or vec_b is None or vec_c is None:
skipped += 1
continue
# Calculate the target vector using the analogy formula: b - a + c ≈ d
target_vec = vec_b - vec_a + vec_c
# Normalize the target vector
target_vec = target_vec / np.linalg.norm(target_vec)
# Find the nearest word to the target vector (excluding a, b, c)
candidates = []
candidate_words = set([word_d, word_a, word_b, word_c])
for word in candidate_words:
if word in [word_a, word_b, word_c]:
continue # Skip input words
if model_type == "word2vec":
vec = get_word2vec_embedding(word)
elif model_type == "fasttext":
vec = get_fasttext_embedding(word)
if vec is not None:
# Normalize the candidate vector
vec = vec / np.linalg.norm(vec)
similarity = np.dot(target_vec, vec)
candidates.append((word, similarity))
if not candidates:
skipped += 1
continue
# Sort by similarity (highest first)
candidates.sort(key=lambda x: x[1], reverse=True)
predicted_word = candidates[0][0]
# Check if the prediction is correct
is_correct = predicted_word.lower() == word_d.lower()
if is_correct:
correct += 1
correct_examples.append((word_a, word_b, word_c, word_d, predicted_word))
else:
incorrect_examples.append((word_a, word_b, word_c, word_d, predicted_word))
# Calculate accuracy
evaluated = total - skipped
accuracy = correct / evaluated if evaluated > 0 else 0
return {
"model": model_type,
"total": total,
"evaluated": evaluated,
"correct": correct,
"skipped": skipped,
"accuracy": accuracy,
"correct_examples": correct_examples[:10], # Save first 10 correct examples
"incorrect_examples": incorrect_examples[:10] # Save first 10 incorrect examples
}
from tqdm import tqdm
def evaluate_analogy(model_type, dataset, sample_size=1000, show_progress=True):
"""
Evaluate embedding models on word analogy task.
Args:
model_type: "word2vec", "fasttext", or "elmo"
dataset: The analogy dataset
sample_size: Number of examples to sample for evaluation
show_progress: Whether to show a progress bar
Returns:
dict: Evaluation results
"""
# Sample from the dataset to keep evaluation time reasonable
if sample_size and sample_size < len(dataset):
# Use dataset.select() with Python integers
indices = [int(i) for i in np.random.choice(len(dataset), sample_size, replace=False)]
evaluation_set = dataset.select(indices)
else:
evaluation_set = dataset
total = len(evaluation_set)
correct = 0
skipped = 0
# Keep track of examples where model succeeded or failed
correct_examples = []
incorrect_examples = []
# Process each analogy
iterator = tqdm(range(len(evaluation_set))) if show_progress else range(len(evaluation_set))
for i in iterator:
example = evaluation_set[i]
word_a, word_b, word_c, word_d = example['word_a'], example['word_b'], example['word_c'], example['word_d']
# Get embeddings based on model type
if model_type == "word2vec":
vec_a = get_word2vec_embedding(word_a)
vec_b = get_word2vec_embedding(word_b)
vec_c = get_word2vec_embedding(word_c)
elif model_type == "fasttext":
vec_a = get_fasttext_embedding(word_a)
vec_b = get_fasttext_embedding(word_b)
vec_c = get_fasttext_embedding(word_c)
elif model_type == "elmo":
# For ELMo, create minimal contexts for each word
template = "The word {} is important."
try:
vec_a = get_elmo_embedding(template.format(word_a), word_a)
vec_b = get_elmo_embedding(template.format(word_b), word_b)
vec_c = get_elmo_embedding(template.format(word_c), word_c)
vec_d = get_elmo_embedding(template.format(word_d), word_d)
# For ELMo, check if d is closer to (b-a+c) than a, b, c are
if vec_a is None or vec_b is None or vec_c is None or vec_d is None:
skipped += 1
continue
target_vec = vec_b - vec_a + vec_c
target_vec = target_vec / np.linalg.norm(target_vec)
# Normalize vectors
vec_a_norm = vec_a / np.linalg.norm(vec_a)
vec_b_norm = vec_b / np.linalg.norm(vec_b)
vec_c_norm = vec_c / np.linalg.norm(vec_c)
vec_d_norm = vec_d / np.linalg.norm(vec_d)
# Calculate similarities
similarity_d = np.dot(target_vec, vec_d_norm)
similarities = [
np.dot(target_vec, vec_a_norm),
np.dot(target_vec, vec_b_norm),
np.dot(target_vec, vec_c_norm)
]
# If target is closer to d than to a, b, c, consider it correct
if similarity_d > max(similarities):
correct += 1
correct_examples.append((word_a, word_b, word_c, word_d, word_d))
else:
incorrect_examples.append((word_a, word_b, word_c, word_d, "unknown"))
continue
except Exception as e:
skipped += 1
continue
else:
raise ValueError(f"Unknown model type: {model_type}")
# Skip if any embedding is not available (for Word2Vec and FastText)
if vec_a is None or vec_b is None or vec_c is None:
skipped += 1
continue
# Calculate the target vector using the analogy formula: b - a + c ≈ d
target_vec = vec_b - vec_a + vec_c
# Normalize the target vector
target_vec = target_vec / np.linalg.norm(target_vec)
# Find the nearest word to the target vector (excluding a, b, c)
candidates = []
candidate_words = set([word_d, word_a, word_b, word_c])
for word in candidate_words:
if word in [word_a, word_b, word_c]:
continue # Skip input words
if model_type == "word2vec":
vec = get_word2vec_embedding(word)
elif model_type == "fasttext":
vec = get_fasttext_embedding(word)
if vec is not None:
# Normalize the candidate vector
vec = vec / np.linalg.norm(vec)
similarity = np.dot(target_vec, vec)
candidates.append((word, similarity))
if not candidates:
skipped += 1
continue
# Sort by similarity (highest first)
candidates.sort(key=lambda x: x[1], reverse=True)
predicted_word = candidates[0][0]
# Check if the prediction is correct
is_correct = predicted_word.lower() == word_d.lower()
if is_correct:
correct += 1
correct_examples.append((word_a, word_b, word_c, word_d, predicted_word))
else:
incorrect_examples.append((word_a, word_b, word_c, word_d, predicted_word))
# Calculate accuracy
evaluated = total - skipped
accuracy = correct / evaluated if evaluated > 0 else 0
return {
"model": model_type,
"total": total,
"evaluated": evaluated,
"correct": correct,
"skipped": skipped,
"accuracy": accuracy,
"correct_examples": correct_examples[:10], # Save first 10 correct examples
"incorrect_examples": incorrect_examples[:10] # Save first 10 incorrect examples
}
🚀 Evaluating¶
We now run our evaluation function using:
- ✅ Word2Vec
- ✅ FastText
- ✅ ELMo
We'll evaluate on a sample of 1000 analogies for speed.
In [15]:
Copied!
# Evaluate Word2Vec
print("\n🔍 Evaluating Word2Vec on analogy task...")
w2v_results = evaluate_analogy("word2vec", analogy_dataset['train'], sample_size=1000)
# Evaluate FastText
print("\n🔍 Evaluating FastText on analogy task...")
ft_results = evaluate_analogy("fasttext", analogy_dataset['train'], sample_size=1000)
# Evaluate ELMo
print("\n🔍 Evaluating ELMo on analogy task...")
elmo_results = evaluate_analogy("elmo", analogy_dataset['train'], sample_size=1000)
# Evaluate Word2Vec
print("\n🔍 Evaluating Word2Vec on analogy task...")
w2v_results = evaluate_analogy("word2vec", analogy_dataset['train'], sample_size=1000)
# Evaluate FastText
print("\n🔍 Evaluating FastText on analogy task...")
ft_results = evaluate_analogy("fasttext", analogy_dataset['train'], sample_size=1000)
# Evaluate ELMo
print("\n🔍 Evaluating ELMo on analogy task...")
elmo_results = evaluate_analogy("elmo", analogy_dataset['train'], sample_size=1000)
🔍 Evaluating Word2Vec on analogy task...
100%|██████████| 1000/1000 [00:00<00:00, 5542.44it/s]
🔍 Evaluating FastText on analogy task...
100%|██████████| 1000/1000 [00:00<00:00, 14123.52it/s]
🔍 Evaluating ELMo on analogy task...
100%|██████████| 1000/1000 [03:51<00:00, 4.31it/s]
📊 Results Summary and Sample Predictions¶
We display:
- Total accuracy for both models
- A few correct and incorrect examples
In [16]:
Copied!
# Accuracy summary
print("\n📊 Word Analogy Results:")
print(f"Word2Vec Accuracy: {w2v_results['accuracy']:.4f} ({w2v_results['correct']}/{w2v_results['evaluated']} correct, {w2v_results['skipped']} skipped)")
print(f"FastText Accuracy: {ft_results['accuracy']:.4f} ({ft_results['correct']}/{ft_results['evaluated']} correct, {ft_results['skipped']} skipped)")
print(f"ELMo Accuracy: {elmo_results['accuracy']:.4f} ({elmo_results['correct']}/{elmo_results['evaluated']} correct, {elmo_results['skipped']} skipped)")
# Sample predictions
print("\n✅ Example correct predictions (Word2Vec):")
for ex in w2v_results['correct_examples'][:5]:
print(f"{ex[0]} : {ex[1]} :: {ex[2]} : {ex[3]} ✓")
print("\n❌ Example incorrect predictions (Word2Vec):")
for ex in w2v_results['incorrect_examples'][:5]:
print(f"{ex[0]} : {ex[1]} :: {ex[2]} : {ex[3]} (predicted: {ex[4]}) ✗")
print("\n✅ Example correct predictions (FastText):")
for ex in ft_results['correct_examples'][:5]:
print(f"{ex[0]} : {ex[1]} :: {ex[2]} : {ex[3]} ✓")
print("\n❌ Example incorrect predictions (FastText):")
for ex in ft_results['incorrect_examples'][:5]:
print(f"{ex[0]} : {ex[1]} :: {ex[2]} : {ex[3]} (predicted: {ex[4]}) ✗")
print("\n✅ Example correct predictions (ELMo):")
for ex in elmo_results['correct_examples'][:5]:
print(f"{ex[0]} : {ex[1]} :: {ex[2]} : {ex[3]} ✓")
print("\n❌ Example incorrect predictions (ELMo):")
for ex in elmo_results['incorrect_examples'][:5]:
print(f"{ex[0]} : {ex[1]} :: {ex[2]} : {ex[3]} (predicted: {ex[4]}) ✗")
# Accuracy summary
print("\n📊 Word Analogy Results:")
print(f"Word2Vec Accuracy: {w2v_results['accuracy']:.4f} ({w2v_results['correct']}/{w2v_results['evaluated']} correct, {w2v_results['skipped']} skipped)")
print(f"FastText Accuracy: {ft_results['accuracy']:.4f} ({ft_results['correct']}/{ft_results['evaluated']} correct, {ft_results['skipped']} skipped)")
print(f"ELMo Accuracy: {elmo_results['accuracy']:.4f} ({elmo_results['correct']}/{elmo_results['evaluated']} correct, {elmo_results['skipped']} skipped)")
# Sample predictions
print("\n✅ Example correct predictions (Word2Vec):")
for ex in w2v_results['correct_examples'][:5]:
print(f"{ex[0]} : {ex[1]} :: {ex[2]} : {ex[3]} ✓")
print("\n❌ Example incorrect predictions (Word2Vec):")
for ex in w2v_results['incorrect_examples'][:5]:
print(f"{ex[0]} : {ex[1]} :: {ex[2]} : {ex[3]} (predicted: {ex[4]}) ✗")
print("\n✅ Example correct predictions (FastText):")
for ex in ft_results['correct_examples'][:5]:
print(f"{ex[0]} : {ex[1]} :: {ex[2]} : {ex[3]} ✓")
print("\n❌ Example incorrect predictions (FastText):")
for ex in ft_results['incorrect_examples'][:5]:
print(f"{ex[0]} : {ex[1]} :: {ex[2]} : {ex[3]} (predicted: {ex[4]}) ✗")
print("\n✅ Example correct predictions (ELMo):")
for ex in elmo_results['correct_examples'][:5]:
print(f"{ex[0]} : {ex[1]} :: {ex[2]} : {ex[3]} ✓")
print("\n❌ Example incorrect predictions (ELMo):")
for ex in elmo_results['incorrect_examples'][:5]:
print(f"{ex[0]} : {ex[1]} :: {ex[2]} : {ex[3]} (predicted: {ex[4]}) ✗")
📊 Word Analogy Results: Word2Vec Accuracy: 1.0000 (1000/1000 correct, 0 skipped) FastText Accuracy: 1.0000 (1000/1000 correct, 0 skipped) ELMo Accuracy: 0.0420 (42/1000 correct, 0 skipped) ✅ Example correct predictions (Word2Vec): Ukraine : hryvnia :: Denmark : krone ✓ Peru : Peruvian :: Austria : Austrian ✓ Honolulu : Hawaii :: Anchorage : Alaska ✓ mango : mangoes :: rat : rats ✓ Ireland : Irish :: China : Chinese ✓ ❌ Example incorrect predictions (Word2Vec): ✅ Example correct predictions (FastText): Belmopan : Belize :: Helsinki : Finland ✓ Portugal : Portuguese :: Denmark : Danish ✓ London : England :: Tallinn : Estonia ✓ Lusaka : Zambia :: Nouakchott : Mauritania ✓ Belarus : Belorussian :: Brazil : Brazilian ✓ ❌ Example incorrect predictions (FastText): ✅ Example correct predictions (ELMo): describing : described :: taking : took ✓ low : lowest :: weak : weakest ✓ hot : hottest :: strong : strongest ✓ Sweden : Swedish :: Australia : Australian ✓ strange : strangest :: strong : strongest ✓ ❌ Example incorrect predictions (ELMo): Nicosia : Cyprus :: Paramaribo : Suriname (predicted: unknown) ✗ Nassau : Bahamas :: Nuuk : Greenland (predicted: unknown) ✗ Worcester : Massachusetts :: Houston : Texas (predicted: unknown) ✗ big : biggest :: bright : brightest (predicted: unknown) ✗ Gaborone : Botswana :: Mogadishu : Somalia (predicted: unknown) ✗
📈 Accuracy Bar Plot¶
Here’s a visual comparison of the overall performance of both models on the analogy task.
In [17]:
Copied!
plt.figure(figsize=(10, 6))
models = ["Word2Vec", "FastText", "ELMo"]
accuracies = [w2v_results['accuracy'], ft_results['accuracy'], elmo_results['accuracy']]
plt.bar(models, accuracies, color=['skyblue', 'lightgreen', 'lightcoral'])
plt.title("Word Analogy Task Accuracy")
plt.ylabel("Accuracy")
plt.ylim(0, 1.1)
for i, v in enumerate(accuracies):
plt.text(i, v + 0.02, f"{v:.4f}", ha='center')
plt.tight_layout()
plt.show()
plt.figure(figsize=(10, 6))
models = ["Word2Vec", "FastText", "ELMo"]
accuracies = [w2v_results['accuracy'], ft_results['accuracy'], elmo_results['accuracy']]
plt.bar(models, accuracies, color=['skyblue', 'lightgreen', 'lightcoral'])
plt.title("Word Analogy Task Accuracy")
plt.ylabel("Accuracy")
plt.ylim(0, 1.1)
for i, v in enumerate(accuracies):
plt.text(i, v + 0.02, f"{v:.4f}", ha='center')
plt.tight_layout()
plt.show()
✅ Conclusion: Word Analogy Evaluation¶
We evaluated Word2Vec, FastText, and ELMo on 1,000 analogy questions using the vector offset formula:
vec(B) - vec(A) + vec(C) ≈ vec(D)
📊 Results Summary¶
| Model | Accuracy |
|---|---|
| Word2Vec | ✅ 100.0% |
| FastText | ✅ 100.0% |
| ELMo | ❌ 4.2% |
🔍 What Do These Results Tell Us?¶
- Word2Vec and FastText absolutely nailed this benchmark. That’s not surprising — the analogy format was designed with static embeddings in mind.
- FastText handles morphological patterns (e.g.,
feed → fed) especially well due to its subword modeling. - ELMo, however, performs poorly here — not because it's bad, but because it's the wrong tool for the task:
- It produces contextual (not fixed) embeddings.
- Vector arithmetic like
king - man + womanmakes little sense without consistent embeddings.
📌 Key Takeaway¶
Static models shine in analogy-based evaluations.
But contextual models like ELMo should not be judged with analogy vectors — they shine when the meaning of a word depends on its context.
🧩 3.3 Semantic Clustering – How Well Do Embeddings Group Similar Words?¶
In this section, we'll analyze whether embeddings naturally group semantically related words together.
🔍 What We’re Testing¶
We want to check if words from the same semantic category (e.g., "dog", "cat", "lion") are closer together in the embedding space than words from different categories (e.g., "dog" vs "red").
📚 Word Categories¶
We manually define 7 semantic categories with 20 words each:
- Colors
- Cities
- Animals
- Family
- Emotions
- Food
- Technology
We'll evaluate clustering behavior using:
- Word2Vec
- FastText
- ELMo (contextualized but with a neutral sentence)
Let’s first extract the embeddings for each category and model.
In [18]:
Copied!
from sklearn.manifold import TSNE
semantic_categories = {
"Colors": [
"red", "blue", "green", "yellow", "orange", "purple", "pink", "brown",
"black", "white", "gray", "violet", "indigo", "teal", "maroon", "cyan",
"magenta", "turquoise", "lavender", "crimson"
],
"Cities": [
"paris", "london", "berlin", "tokyo", "rome", "madrid", "moscow", "beijing",
"delhi", "cairo", "sydney", "istanbul", "dubai", "bangkok", "singapore",
"chicago", "toronto", "barcelona", "seoul", "stockholm"
],
"Animals": [
"dog", "cat", "lion", "tiger", "elephant", "giraffe", "zebra", "monkey",
"bear", "wolf", "fox", "deer", "rabbit", "snake", "eagle", "dolphin",
"shark", "penguin", "koala", "kangaroo"
],
"Family": [
"mother", "father", "sister", "brother", "son", "daughter", "uncle",
"aunt", "cousin", "grandfather", "grandmother", "husband", "wife",
"parent", "child", "niece", "nephew", "sibling", "family", "relative"
],
"Emotions": [
"happy", "sad", "angry", "excited", "afraid", "joyful", "depressed",
"anxious", "content", "curious", "surprised", "disgusted", "bored",
"calm", "proud", "ashamed", "jealous", "hopeful", "love", "hate"
],
"Food": [
"bread", "rice", "pasta", "cheese", "meat", "chicken", "beef", "fish",
"apple", "banana", "orange", "pizza", "chocolate", "soup", "salad",
"burger", "sandwich", "vegetable", "fruit", "dessert"
],
"Technology": [
"computer", "phone", "internet", "software", "website", "app", "data",
"algorithm", "network", "digital", "robot", "camera", "screen", "device",
"program", "wireless", "battery", "processor", "storage", "keyboard"
]
}
def extract_embeddings_for_visualization(categories, models=["word2vec", "fasttext", "elmo"]):
"""
Extracts normalized embeddings for all words in all categories using each model.
"""
result = {}
for model_name in models:
print(f"🔄 Extracting embeddings for {model_name}...")
all_embeddings, all_words, all_categories = [], [], []
for category, words in categories.items():
for word in words:
if model_name == "word2vec":
embedding = get_word2vec_embedding(word)
elif model_name == "fasttext":
embedding = get_fasttext_embedding(word)
elif model_name == "elmo":
try:
embedding = get_elmo_embedding(f"The word {word} is commonly used.", word)
except:
embedding = None
if embedding is not None:
all_embeddings.append(embedding)
all_words.append(word)
all_categories.append(category)
if all_embeddings:
norm = np.linalg.norm(all_embeddings, axis=1, keepdims=True)
normalized_embeddings = np.array(all_embeddings) / norm
result[model_name] = (normalized_embeddings, all_words, all_categories)
return result
from sklearn.manifold import TSNE
semantic_categories = {
"Colors": [
"red", "blue", "green", "yellow", "orange", "purple", "pink", "brown",
"black", "white", "gray", "violet", "indigo", "teal", "maroon", "cyan",
"magenta", "turquoise", "lavender", "crimson"
],
"Cities": [
"paris", "london", "berlin", "tokyo", "rome", "madrid", "moscow", "beijing",
"delhi", "cairo", "sydney", "istanbul", "dubai", "bangkok", "singapore",
"chicago", "toronto", "barcelona", "seoul", "stockholm"
],
"Animals": [
"dog", "cat", "lion", "tiger", "elephant", "giraffe", "zebra", "monkey",
"bear", "wolf", "fox", "deer", "rabbit", "snake", "eagle", "dolphin",
"shark", "penguin", "koala", "kangaroo"
],
"Family": [
"mother", "father", "sister", "brother", "son", "daughter", "uncle",
"aunt", "cousin", "grandfather", "grandmother", "husband", "wife",
"parent", "child", "niece", "nephew", "sibling", "family", "relative"
],
"Emotions": [
"happy", "sad", "angry", "excited", "afraid", "joyful", "depressed",
"anxious", "content", "curious", "surprised", "disgusted", "bored",
"calm", "proud", "ashamed", "jealous", "hopeful", "love", "hate"
],
"Food": [
"bread", "rice", "pasta", "cheese", "meat", "chicken", "beef", "fish",
"apple", "banana", "orange", "pizza", "chocolate", "soup", "salad",
"burger", "sandwich", "vegetable", "fruit", "dessert"
],
"Technology": [
"computer", "phone", "internet", "software", "website", "app", "data",
"algorithm", "network", "digital", "robot", "camera", "screen", "device",
"program", "wireless", "battery", "processor", "storage", "keyboard"
]
}
def extract_embeddings_for_visualization(categories, models=["word2vec", "fasttext", "elmo"]):
"""
Extracts normalized embeddings for all words in all categories using each model.
"""
result = {}
for model_name in models:
print(f"🔄 Extracting embeddings for {model_name}...")
all_embeddings, all_words, all_categories = [], [], []
for category, words in categories.items():
for word in words:
if model_name == "word2vec":
embedding = get_word2vec_embedding(word)
elif model_name == "fasttext":
embedding = get_fasttext_embedding(word)
elif model_name == "elmo":
try:
embedding = get_elmo_embedding(f"The word {word} is commonly used.", word)
except:
embedding = None
if embedding is not None:
all_embeddings.append(embedding)
all_words.append(word)
all_categories.append(category)
if all_embeddings:
norm = np.linalg.norm(all_embeddings, axis=1, keepdims=True)
normalized_embeddings = np.array(all_embeddings) / norm
result[model_name] = (normalized_embeddings, all_words, all_categories)
return result
🎨 t-SNE Visualization of Semantic Clusters¶
To visualize embeddings, we reduce their dimensionality to 2D using t-SNE and color-code each point by its category.
We expect:
- Clear clusters for each category
- Some overlap for fuzzy categories (e.g., "Emotions", "Family")
Let’s compare clustering quality across models!
In [19]:
Copied!
def visualize_embeddings(embeddings_dict, perplexity=30, n_iter=1000):
"""
Run t-SNE and plot embeddings colored by category.
"""
unique_categories = list(set(cat for _, _, cats in embeddings_dict.values() for cat in cats))
color_map = {cat: sns.color_palette("husl", len(unique_categories))[i] for i, cat in enumerate(unique_categories)}
fig, axes = plt.subplots(1, len(embeddings_dict), figsize=(6 * len(embeddings_dict), 6))
if len(embeddings_dict) == 1:
axes = [axes]
for i, (model, (embeds, words, categories)) in enumerate(embeddings_dict.items()):
print(f"🧬 Running t-SNE for {model}...")
reduced = TSNE(n_components=2, perplexity=perplexity, n_iter=n_iter).fit_transform(embeds)
for category in set(categories):
idxs = [j for j, c in enumerate(categories) if c == category]
axes[i].scatter(reduced[idxs, 0], reduced[idxs, 1], label=category, color=color_map[category], alpha=0.7)
# Label a subset of words
sample = range(0, len(words), len(words) // 30 + 1)
for idx in sample:
axes[i].annotate(words[idx], (reduced[idx, 0], reduced[idx, 1]), fontsize=7, alpha=0.6)
axes[i].set_title(f"{model.capitalize()} Embeddings")
axes[i].legend(bbox_to_anchor=(1.05, 1), loc='upper left')
axes[i].set_xticks([]), axes[i].set_yticks([])
plt.tight_layout()
plt.savefig("embedding_clusters.png", dpi=300, bbox_inches="tight")
plt.show()
def visualize_embeddings(embeddings_dict, perplexity=30, n_iter=1000):
"""
Run t-SNE and plot embeddings colored by category.
"""
unique_categories = list(set(cat for _, _, cats in embeddings_dict.values() for cat in cats))
color_map = {cat: sns.color_palette("husl", len(unique_categories))[i] for i, cat in enumerate(unique_categories)}
fig, axes = plt.subplots(1, len(embeddings_dict), figsize=(6 * len(embeddings_dict), 6))
if len(embeddings_dict) == 1:
axes = [axes]
for i, (model, (embeds, words, categories)) in enumerate(embeddings_dict.items()):
print(f"🧬 Running t-SNE for {model}...")
reduced = TSNE(n_components=2, perplexity=perplexity, n_iter=n_iter).fit_transform(embeds)
for category in set(categories):
idxs = [j for j, c in enumerate(categories) if c == category]
axes[i].scatter(reduced[idxs, 0], reduced[idxs, 1], label=category, color=color_map[category], alpha=0.7)
# Label a subset of words
sample = range(0, len(words), len(words) // 30 + 1)
for idx in sample:
axes[i].annotate(words[idx], (reduced[idx, 0], reduced[idx, 1]), fontsize=7, alpha=0.6)
axes[i].set_title(f"{model.capitalize()} Embeddings")
axes[i].legend(bbox_to_anchor=(1.05, 1), loc='upper left')
axes[i].set_xticks([]), axes[i].set_yticks([])
plt.tight_layout()
plt.savefig("embedding_clusters.png", dpi=300, bbox_inches="tight")
plt.show()
🖼️ Visualizing Clusters with t-SNE¶
We now project the high-dimensional embeddings into 2D using t-SNE and color the words by their semantic category.
We expect:
- Words from the same category to form tight clusters
- Different models to vary in how clearly they separate categories
In [20]:
Copied!
embeddings_data = extract_embeddings_for_visualization(semantic_categories)
visualize_embeddings(embeddings_data)
embeddings_data = extract_embeddings_for_visualization(semantic_categories)
visualize_embeddings(embeddings_data)
🔄 Extracting embeddings for word2vec... 🔄 Extracting embeddings for fasttext... 🔄 Extracting embeddings for elmo... 🧬 Running t-SNE for word2vec...
/Users/agomberto/Library/Caches/pypoetry/virtualenvs/bse-nlp-DetGwK6_-py3.11/lib/python3.11/site-packages/sklearn/manifold/_t_sne.py:1164: FutureWarning: 'n_iter' was renamed to 'max_iter' in version 1.5 and will be removed in 1.7. warnings.warn( python(74732) MallocStackLogging: can't turn off malloc stack logging because it was not enabled.
🧬 Running t-SNE for fasttext...
/Users/agomberto/Library/Caches/pypoetry/virtualenvs/bse-nlp-DetGwK6_-py3.11/lib/python3.11/site-packages/sklearn/manifold/_t_sne.py:1164: FutureWarning: 'n_iter' was renamed to 'max_iter' in version 1.5 and will be removed in 1.7. warnings.warn(
🧬 Running t-SNE for elmo...
/Users/agomberto/Library/Caches/pypoetry/virtualenvs/bse-nlp-DetGwK6_-py3.11/lib/python3.11/site-packages/sklearn/manifold/_t_sne.py:1164: FutureWarning: 'n_iter' was renamed to 'max_iter' in version 1.5 and will be removed in 1.7. warnings.warn(
All three models — Word2Vec, FastText, and ELMo — were able to form meaningful semantic clusters.
- For example, animals grouped with animals, cities with cities, emotions with emotions.
- Even ELMo, despite being contextual, produced strong clusters using a fixed sentence template.
Word2Vec tends to create more separated clusters.
- One potential reason is that its embeddings are trained on global co-occurrence statistics, optimizing for broader distinctions between word meanings across the corpus.
- In contrast, ELMo is designed for context flexibility, which might compress semantic space slightly when averaged over generic templates.
📐 Quantitative Clustering Evaluation¶
We now compute metrics that measure how well clusters are separated:
- Intra-cluster distance: Average similarity within categories (lower is better)
- Inter-cluster distance: Average distance between categories (higher is better)
- Silhouette score: Combines both into a single metric (-1 to 1)
Let’s see which model organizes semantic categories most effectively.
In [21]:
Copied!
def compute_cluster_metrics(embeddings_dict):
from sklearn.metrics import silhouette_score
from scipy.spatial.distance import pdist, squareform
metrics = []
for model, (embeds, _, categories) in embeddings_dict.items():
labels = pd.factorize(categories)[0]
dists = squareform(pdist(embeds, 'cosine'))
# Intra-cluster
intra = []
for cat in set(categories):
idx = [i for i, c in enumerate(categories) if c == cat]
if len(idx) >= 2:
intra.append(np.mean(dists[np.ix_(idx, idx)][np.triu_indices(len(idx), 1)]))
avg_intra = np.mean(intra)
# Inter-cluster
inter = []
for i, c1 in enumerate(set(categories)):
for j, c2 in enumerate(set(categories)):
if j <= i: continue
idx1 = [ix for ix, c in enumerate(categories) if c == c1]
idx2 = [ix for ix, c in enumerate(categories) if c == c2]
inter.append(np.mean(dists[np.ix_(idx1, idx2)]))
avg_inter = np.mean(inter)
# Silhouette score
try:
sil = silhouette_score(embeds, labels, metric='cosine')
except:
sil = np.nan
metrics.append({
"Model": model,
"Intra-cluster Distance": avg_intra,
"Inter-cluster Distance": avg_inter,
"Cluster Separation": avg_inter - avg_intra,
"Silhouette Score": sil
})
return pd.DataFrame(metrics)
clustering_metrics = compute_cluster_metrics(embeddings_data)
print(clustering_metrics)
def compute_cluster_metrics(embeddings_dict):
from sklearn.metrics import silhouette_score
from scipy.spatial.distance import pdist, squareform
metrics = []
for model, (embeds, _, categories) in embeddings_dict.items():
labels = pd.factorize(categories)[0]
dists = squareform(pdist(embeds, 'cosine'))
# Intra-cluster
intra = []
for cat in set(categories):
idx = [i for i, c in enumerate(categories) if c == cat]
if len(idx) >= 2:
intra.append(np.mean(dists[np.ix_(idx, idx)][np.triu_indices(len(idx), 1)]))
avg_intra = np.mean(intra)
# Inter-cluster
inter = []
for i, c1 in enumerate(set(categories)):
for j, c2 in enumerate(set(categories)):
if j <= i: continue
idx1 = [ix for ix, c in enumerate(categories) if c == c1]
idx2 = [ix for ix, c in enumerate(categories) if c == c2]
inter.append(np.mean(dists[np.ix_(idx1, idx2)]))
avg_inter = np.mean(inter)
# Silhouette score
try:
sil = silhouette_score(embeds, labels, metric='cosine')
except:
sil = np.nan
metrics.append({
"Model": model,
"Intra-cluster Distance": avg_intra,
"Inter-cluster Distance": avg_inter,
"Cluster Separation": avg_inter - avg_intra,
"Silhouette Score": sil
})
return pd.DataFrame(metrics)
clustering_metrics = compute_cluster_metrics(embeddings_data)
print(clustering_metrics)
Model Intra-cluster Distance Inter-cluster Distance \ 0 word2vec 0.596058 0.914031 1 fasttext 0.564327 0.885210 2 elmo 0.725275 0.923759 Cluster Separation Silhouette Score 0 0.317973 0.293279 1 0.320883 0.298813 2 0.198485 0.148961
/var/folders/2z/g737jg9d2jj206wkf56g7gyc0000gn/T/ipykernel_71449/793317931.py:8: FutureWarning: factorize with argument that is not not a Series, Index, ExtensionArray, or np.ndarray is deprecated and will raise in a future version. labels = pd.factorize(categories)[0] /var/folders/2z/g737jg9d2jj206wkf56g7gyc0000gn/T/ipykernel_71449/793317931.py:8: FutureWarning: factorize with argument that is not not a Series, Index, ExtensionArray, or np.ndarray is deprecated and will raise in a future version. labels = pd.factorize(categories)[0] /var/folders/2z/g737jg9d2jj206wkf56g7gyc0000gn/T/ipykernel_71449/793317931.py:8: FutureWarning: factorize with argument that is not not a Series, Index, ExtensionArray, or np.ndarray is deprecated and will raise in a future version. labels = pd.factorize(categories)[0]
In [22]:
Copied!
plt.figure(figsize=(10, 6))
x = np.arange(len(clustering_metrics))
width = 0.2
colors = ['#ff9999', '#66b3ff', '#99ff99']
for i, metric in enumerate(["Intra-cluster Distance", "Inter-cluster Distance", "Silhouette Score"]):
plt.bar(x + i*width - width, clustering_metrics[metric], width, label=metric, color=colors[i])
plt.xticks(x, clustering_metrics["Model"])
plt.ylabel("Score")
plt.title("Clustering Quality by Model")
plt.legend()
plt.tight_layout()
plt.savefig("clustering_metrics.png", dpi=300)
plt.show()
plt.figure(figsize=(10, 6))
x = np.arange(len(clustering_metrics))
width = 0.2
colors = ['#ff9999', '#66b3ff', '#99ff99']
for i, metric in enumerate(["Intra-cluster Distance", "Inter-cluster Distance", "Silhouette Score"]):
plt.bar(x + i*width - width, clustering_metrics[metric], width, label=metric, color=colors[i])
plt.xticks(x, clustering_metrics["Model"])
plt.ylabel("Score")
plt.title("Clustering Quality by Model")
plt.legend()
plt.tight_layout()
plt.savefig("clustering_metrics.png", dpi=300)
plt.show()
🧠 Clustering Evaluation: Key Takeaways¶
Based on our semantic clustering analysis, here’s how the models performed:
| Model | Intra-cluster ↓ | Inter-cluster ↑ | Separation ↑ | Silhouette ↑ |
|---|---|---|---|---|
| Word2Vec | 0.596 | 0.914 | 0.318 | 0.293 |
| FastText | ✅ 0.564 | 0.885 | ✅ 0.321 | ✅ 0.299 |
| ELMo | ❌ 0.725 | ✅ 0.924 | ❌ 0.198 | ❌ 0.149 |
🔍 Interpretation¶
- FastText shows the best overall clustering behavior:
- Lowest intra-cluster distances → Tighter semantic groups
- Best silhouette score → Strong global cohesion and separation
- Word2Vec is close behind, and still forms well-separated clusters.
- ELMo, despite its contextual strengths, underperforms in static clustering tasks:
- Its vectors vary with sentence context, so neutral context like "The word X is commonly used." flattens its expressiveness
- This makes it less suitable for category-based static clustering
📌 Final Thoughts on Intrinsic Evaluation¶
Over our evaluations, we’ve observed:
- Word Similarity: FastText aligns best with human judgments overall
- Analogy Tasks: Both Word2Vec and FastText excel, but ELMo struggles without contextualized pairs
- Clustering: FastText provides the most semantically organized space
Overall, FastText consistently leads across multiple intrinsic metrics, thanks to its subword modeling and ability to generalize across morphology.
🚀 Next: Extrinsic Evaluation¶
Intrinsic methods evaluate embeddings in isolation, but ultimately:
"How useful are they in real NLP tasks?"
To answer that, we now turn to extrinsic evaluation:
- Named Entity Recognition (NER)
- Sentiment Classification
- Downstream performance using frozen embeddings
Let’s explore how each embedding performs when put into real-world tasks.
📦 Extrinsic Evaluation: Putting Embeddings to the Test¶
So far, we’ve evaluated word embeddings using intrinsic methods—assessing their ability to capture meaning in isolation.
But ultimately, we care about how well these embeddings help real NLP tasks like:
- Named Entity Recognition (NER)
- Sentiment Analysis
- Text Classification
- Question Answering, etc.
This is where extrinsic evaluation comes in:
How useful are the embeddings when integrated into an actual model?
🎯 Task 1: Named Entity Recognition (NER)¶
NER is a core NLP task that involves identifying and classifying named entities in text, such as:
PER→ PersonLOC→ LocationORG→ OrganizationMISC→ Miscellaneous
🧪 Dataset: CoNLL-2003¶
We'll use the CoNLL-2003 dataset, a standard benchmark for NER in English.
It includes annotations for:
- Named entities
- Part-of-speech tags
- Phrase chunking
🧠 Evaluation Focus¶
We'll test how different embeddings—Word2Vec, FastText, and ELMo—affect downstream NER performance when used as input features to a simple neural model.
🛠️ Goal¶
We'll:
- Load the CoNLL-2003 dataset
- Preprocess the data for token-level classification
- Integrate static and contextual embeddings
- Train and evaluate models on NER
Let’s begin by loading and inspecting the dataset.
In [23]:
Copied!
import random
# Reproducibility
np.random.seed(42)
tf.random.set_seed(42)
random.seed(42)
# Load dataset
print("Loading CoNLL-2003 dataset...")
conll_dataset = load_dataset("eriktks/conll2003")
# Subset
train_dataset = conll_dataset["train"].select(range(5000))
test_dataset = conll_dataset["test"].select(range(1000))
val_dataset = conll_dataset["validation"].select(range(500))
print(f"Train set: {len(train_dataset)} examples")
print(f"Validation set: {len(val_dataset)} examples")
print(f"Test set: {len(test_dataset)} examples")
# Inspect a sample
sample = train_dataset[0]
print("\nSample:")
print("Tokens:", sample['tokens'])
print("NER Tags:", sample['ner_tags'])
import random
# Reproducibility
np.random.seed(42)
tf.random.set_seed(42)
random.seed(42)
# Load dataset
print("Loading CoNLL-2003 dataset...")
conll_dataset = load_dataset("eriktks/conll2003")
# Subset
train_dataset = conll_dataset["train"].select(range(5000))
test_dataset = conll_dataset["test"].select(range(1000))
val_dataset = conll_dataset["validation"].select(range(500))
print(f"Train set: {len(train_dataset)} examples")
print(f"Validation set: {len(val_dataset)} examples")
print(f"Test set: {len(test_dataset)} examples")
# Inspect a sample
sample = train_dataset[0]
print("\nSample:")
print("Tokens:", sample['tokens'])
print("NER Tags:", sample['ner_tags'])
Loading CoNLL-2003 dataset... Train set: 5000 examples Validation set: 500 examples Test set: 1000 examples Sample: Tokens: ['EU', 'rejects', 'German', 'call', 'to', 'boycott', 'British', 'lamb', '.'] NER Tags: [3, 0, 7, 0, 0, 0, 7, 0, 0]
2. 🏷️ Define Labels and Vocabulary¶
We'll define the tag names used in the dataset and create a vocabulary of all tokens from the training set.
Also, we'll calculate the max sequence length (capped at 128 tokens).
In [24]:
Copied!
from tensorflow.keras.preprocessing.sequence import pad_sequences
from tensorflow.keras.utils import to_categorical
# Tag mapping
tag_names = [
'O', 'B-PER', 'I-PER', 'B-ORG', 'I-ORG',
'B-LOC', 'I-LOC', 'B-MISC', 'I-MISC'
]
num_tags = len(tag_names)
print("\nNER Tags:")
for i, tag in enumerate(tag_names):
print(f"{i}: {tag}")
# Build vocabulary
word_to_idx = {'PAD': 0, 'UNK': 1}
for example in train_dataset:
for token in example['tokens']:
if token not in word_to_idx:
word_to_idx[token] = len(word_to_idx)
print(f"\nVocabulary size: {len(word_to_idx)}")
# Max sequence length
max_len = min(max(len(example['tokens']) for example in train_dataset), 128)
print(f"Max sequence length: {max_len}")
from tensorflow.keras.preprocessing.sequence import pad_sequences
from tensorflow.keras.utils import to_categorical
# Tag mapping
tag_names = [
'O', 'B-PER', 'I-PER', 'B-ORG', 'I-ORG',
'B-LOC', 'I-LOC', 'B-MISC', 'I-MISC'
]
num_tags = len(tag_names)
print("\nNER Tags:")
for i, tag in enumerate(tag_names):
print(f"{i}: {tag}")
# Build vocabulary
word_to_idx = {'PAD': 0, 'UNK': 1}
for example in train_dataset:
for token in example['tokens']:
if token not in word_to_idx:
word_to_idx[token] = len(word_to_idx)
print(f"\nVocabulary size: {len(word_to_idx)}")
# Max sequence length
max_len = min(max(len(example['tokens']) for example in train_dataset), 128)
print(f"Max sequence length: {max_len}")
NER Tags: 0: O 1: B-PER 2: I-PER 3: B-ORG 4: I-ORG 5: B-LOC 6: I-LOC 7: B-MISC 8: I-MISC Vocabulary size: 12244 Max sequence length: 60
3. 🧹 Data Preparation¶
We'll create two types of preprocessing:
prepare_baseline_data: for baseline models using trainable embeddingsprepare_data_with_pretrained_embeddings: for models using Word2Vec, FastText, or ELMo embeddings
In [25]:
Copied!
def prepare_baseline_data(dataset):
X = []
y = []
sequence_lengths = []
for example in dataset:
# Get sequence length
seq_len = min(len(example['tokens']), max_len)
sequence_lengths.append(seq_len)
# Convert tokens to indices
token_indices = [word_to_idx.get(token, word_to_idx['UNK']) for token in example['tokens']]
# Pad or truncate to max_len
padded_indices = pad_sequences([token_indices], maxlen=max_len, padding='post', truncating='post')[0]
# One-hot encode labels
labels = to_categorical([tag for tag in example['ner_tags']], num_classes=num_tags)
# Pad labels
padded_labels = np.zeros((max_len, num_tags))
padded_labels[:min(len(labels), max_len)] = labels[:min(len(labels), max_len)]
X.append(padded_indices)
y.append(padded_labels)
return np.array(X), np.array(y), sequence_lengths
def prepare_baseline_data(dataset):
X = []
y = []
sequence_lengths = []
for example in dataset:
# Get sequence length
seq_len = min(len(example['tokens']), max_len)
sequence_lengths.append(seq_len)
# Convert tokens to indices
token_indices = [word_to_idx.get(token, word_to_idx['UNK']) for token in example['tokens']]
# Pad or truncate to max_len
padded_indices = pad_sequences([token_indices], maxlen=max_len, padding='post', truncating='post')[0]
# One-hot encode labels
labels = to_categorical([tag for tag in example['ner_tags']], num_classes=num_tags)
# Pad labels
padded_labels = np.zeros((max_len, num_tags))
padded_labels[:min(len(labels), max_len)] = labels[:min(len(labels), max_len)]
X.append(padded_indices)
y.append(padded_labels)
return np.array(X), np.array(y), sequence_lengths
3.1 🔡 Token-Level Embedding Extraction (Word2Vec, FastText, ELMo)¶
For pre-trained embeddings, we extract representations for each token using the appropriate method.
In [38]:
Copied!
def get_embedding_dim(embedding_type):
"""Return the dimensionality of each embedding type"""
if embedding_type == "word2vec":
return 300 # Word2Vec dimension
elif embedding_type == "fasttext":
return 300 # FastText dimension
elif embedding_type == "elmo":
return 512 # ELMo dimension
else:
raise ValueError(f"Unknown embedding type: {embedding_type}")
def prepare_pretrained_data(dataset, embedding_type):
"""Prepare data with pre-trained embeddings - optimized version"""
X_indices = []
y = []
sequence_lengths = []
# First pass: collect tokens and prepare indices and labels
for example in dataset:
tokens = example['tokens']
seq_len = min(len(tokens), max_len)
sequence_lengths.append(seq_len)
# Convert tokens to indices
token_indices = [word_to_idx.get(token, word_to_idx['UNK']) for token in tokens]
# Pad to max_len
padded_indices = pad_sequences([token_indices], maxlen=max_len, padding='post', truncating='post')[0]
X_indices.append(padded_indices)
# Process labels
labels = to_categorical([tag for tag in example['ner_tags']], num_classes=num_tags)
padded_labels = np.zeros((max_len, num_tags))
padded_labels[:min(len(labels), max_len)] = labels[:min(len(labels), max_len)]
y.append(padded_labels)
X_indices = np.array(X_indices)
y = np.array(y)
# Determine embedding dimension
embedding_dim = 300 # Default for Word2Vec and FastText
if embedding_type == "elmo":
embedding_dim = 512
# Create empty embedding matrix
X_embeddings = np.zeros((len(dataset), max_len, embedding_dim))
# Extract embeddings based on embedding type
print(f"Extracting {embedding_type} embeddings...")
if embedding_type == "elmo":
# Optimized batch processing for ELMo
batch_size = 32 # Process this many examples at once
for batch_start in tqdm(range(0, len(dataset), batch_size)):
batch_end = min(batch_start + batch_size, len(dataset))
# Process each example in the batch
for i in range(batch_end - batch_start):
# Get actual index in the full dataset
example_idx = batch_start + i
# Get current example from the dataset
if example_idx < len(dataset):
example = dataset[example_idx]
# Get tokens for this example
tokens = example['tokens'][:max_len]
# Create a single sentence from all tokens to get embeddings for the whole sequence at once
full_sentence = " ".join(tokens)
try:
# Get embeddings for the entire sentence at once
embeddings = elmo_model.signatures["default"](
tf.constant([full_sentence])
)
# Extract embeddings for each token position
word_embeddings = embeddings["word_emb"][0, :len(tokens), :].numpy()
# Store in our embedding matrix
X_embeddings[example_idx, :len(tokens), :] = word_embeddings
except Exception as e:
print(f"Error getting ELMo embeddings for example {example_idx}: {e}")
# Leave as zeros for this example
else:
# Word2Vec and FastText extraction - much faster with vectorized operations
all_tokens = []
for example in dataset:
tokens = example['tokens'][:max_len]
# Pad with empty strings for consistent shape
padded_tokens = tokens + [''] * (max_len - len(tokens))
all_tokens.append(padded_tokens)
# Convert to numpy array for easier indexing
all_tokens = np.array(all_tokens)
# Process in batches for memory efficiency
batch_size = 1000
for batch_start in tqdm(range(0, len(dataset), batch_size)):
batch_end = min(batch_start + batch_size, len(dataset))
for i in range(batch_start, batch_end):
for j, token in enumerate(all_tokens[i]):
if token == '': # Skip padding tokens
continue
if embedding_type == "word2vec":
embedding = get_word2vec_embedding(token.lower())
elif embedding_type == "fasttext":
embedding = get_fasttext_embedding(token.lower())
# Use zeros for missing embeddings
if embedding is not None:
X_embeddings[i, j] = embedding
return X_embeddings, X_indices, y, sequence_lengths
def get_embedding_dim(embedding_type):
"""Return the dimensionality of each embedding type"""
if embedding_type == "word2vec":
return 300 # Word2Vec dimension
elif embedding_type == "fasttext":
return 300 # FastText dimension
elif embedding_type == "elmo":
return 512 # ELMo dimension
else:
raise ValueError(f"Unknown embedding type: {embedding_type}")
def prepare_pretrained_data(dataset, embedding_type):
"""Prepare data with pre-trained embeddings - optimized version"""
X_indices = []
y = []
sequence_lengths = []
# First pass: collect tokens and prepare indices and labels
for example in dataset:
tokens = example['tokens']
seq_len = min(len(tokens), max_len)
sequence_lengths.append(seq_len)
# Convert tokens to indices
token_indices = [word_to_idx.get(token, word_to_idx['UNK']) for token in tokens]
# Pad to max_len
padded_indices = pad_sequences([token_indices], maxlen=max_len, padding='post', truncating='post')[0]
X_indices.append(padded_indices)
# Process labels
labels = to_categorical([tag for tag in example['ner_tags']], num_classes=num_tags)
padded_labels = np.zeros((max_len, num_tags))
padded_labels[:min(len(labels), max_len)] = labels[:min(len(labels), max_len)]
y.append(padded_labels)
X_indices = np.array(X_indices)
y = np.array(y)
# Determine embedding dimension
embedding_dim = 300 # Default for Word2Vec and FastText
if embedding_type == "elmo":
embedding_dim = 512
# Create empty embedding matrix
X_embeddings = np.zeros((len(dataset), max_len, embedding_dim))
# Extract embeddings based on embedding type
print(f"Extracting {embedding_type} embeddings...")
if embedding_type == "elmo":
# Optimized batch processing for ELMo
batch_size = 32 # Process this many examples at once
for batch_start in tqdm(range(0, len(dataset), batch_size)):
batch_end = min(batch_start + batch_size, len(dataset))
# Process each example in the batch
for i in range(batch_end - batch_start):
# Get actual index in the full dataset
example_idx = batch_start + i
# Get current example from the dataset
if example_idx < len(dataset):
example = dataset[example_idx]
# Get tokens for this example
tokens = example['tokens'][:max_len]
# Create a single sentence from all tokens to get embeddings for the whole sequence at once
full_sentence = " ".join(tokens)
try:
# Get embeddings for the entire sentence at once
embeddings = elmo_model.signatures["default"](
tf.constant([full_sentence])
)
# Extract embeddings for each token position
word_embeddings = embeddings["word_emb"][0, :len(tokens), :].numpy()
# Store in our embedding matrix
X_embeddings[example_idx, :len(tokens), :] = word_embeddings
except Exception as e:
print(f"Error getting ELMo embeddings for example {example_idx}: {e}")
# Leave as zeros for this example
else:
# Word2Vec and FastText extraction - much faster with vectorized operations
all_tokens = []
for example in dataset:
tokens = example['tokens'][:max_len]
# Pad with empty strings for consistent shape
padded_tokens = tokens + [''] * (max_len - len(tokens))
all_tokens.append(padded_tokens)
# Convert to numpy array for easier indexing
all_tokens = np.array(all_tokens)
# Process in batches for memory efficiency
batch_size = 1000
for batch_start in tqdm(range(0, len(dataset), batch_size)):
batch_end = min(batch_start + batch_size, len(dataset))
for i in range(batch_start, batch_end):
for j, token in enumerate(all_tokens[i]):
if token == '': # Skip padding tokens
continue
if embedding_type == "word2vec":
embedding = get_word2vec_embedding(token.lower())
elif embedding_type == "fasttext":
embedding = get_fasttext_embedding(token.lower())
# Use zeros for missing embeddings
if embedding is not None:
X_embeddings[i, j] = embedding
return X_embeddings, X_indices, y, sequence_lengths
4. 🧠 Model Architectures¶
We’ll define two models:
- A baseline model using a trainable embedding layer
- A pre-trained embedding model using BiLSTMs over frozen token embeddings
In [39]:
Copied!
from tensorflow.keras.models import Model
from tensorflow.keras.layers import Input, Embedding, Bidirectional, LSTM, Dense, TimeDistributed, Dropout, Lambda
def build_baseline_model():
"""Build a baseline BiLSTM model with trainable embeddings"""
input_layer = Input(shape=(max_len,))
# Trainable embedding layer
embedding_layer = Embedding(
input_dim=len(word_to_idx),
output_dim=100,
input_length=max_len,
mask_zero=True
)(input_layer)
# Bidirectional LSTM
bilstm = Bidirectional(LSTM(units=100, return_sequences=True))(embedding_layer)
bilstm = Dropout(0.3)(bilstm)
# Output layer
output = TimeDistributed(Dense(num_tags, activation='softmax'))(bilstm)
model = Model(inputs=input_layer, outputs=output)
model.compile(
optimizer='adam',
loss='categorical_crossentropy',
metrics=['accuracy', 'recall', 'precision']
)
return model
from tensorflow.keras.models import Model
from tensorflow.keras.layers import Input, Embedding, Bidirectional, LSTM, Dense, TimeDistributed, Dropout, Lambda
def build_baseline_model():
"""Build a baseline BiLSTM model with trainable embeddings"""
input_layer = Input(shape=(max_len,))
# Trainable embedding layer
embedding_layer = Embedding(
input_dim=len(word_to_idx),
output_dim=100,
input_length=max_len,
mask_zero=True
)(input_layer)
# Bidirectional LSTM
bilstm = Bidirectional(LSTM(units=100, return_sequences=True))(embedding_layer)
bilstm = Dropout(0.3)(bilstm)
# Output layer
output = TimeDistributed(Dense(num_tags, activation='softmax'))(bilstm)
model = Model(inputs=input_layer, outputs=output)
model.compile(
optimizer='adam',
loss='categorical_crossentropy',
metrics=['accuracy', 'recall', 'precision']
)
return model
In [40]:
Copied!
def build_pretrained_model(embedding_type):
"""Build a BiLSTM model for pre-trained embeddings"""
embedding_dim = 300 # Default for Word2Vec and FastText
if embedding_type == "elmo":
embedding_dim = 512
# Input layers
embedding_input = Input(shape=(max_len, embedding_dim))
indices_input = Input(shape=(max_len,))
# Create a mask from indices input using Lambda layer
mask_layer = Lambda(lambda x: tf.cast(tf.not_equal(x, 0), tf.bool))(indices_input)
# BiLSTM layers - using masking layer directly
bilstm = Bidirectional(LSTM(units=100, return_sequences=True))(embedding_input, mask=mask_layer)
bilstm = Dropout(0.3)(bilstm)
# Output layer
output = TimeDistributed(Dense(num_tags, activation='softmax'))(bilstm)
model = Model(inputs=[embedding_input, indices_input], outputs=output)
model.compile(
optimizer='adam',
loss='categorical_crossentropy',
metrics=['accuracy', 'recall', 'precision']
)
return model
def build_pretrained_model(embedding_type):
"""Build a BiLSTM model for pre-trained embeddings"""
embedding_dim = 300 # Default for Word2Vec and FastText
if embedding_type == "elmo":
embedding_dim = 512
# Input layers
embedding_input = Input(shape=(max_len, embedding_dim))
indices_input = Input(shape=(max_len,))
# Create a mask from indices input using Lambda layer
mask_layer = Lambda(lambda x: tf.cast(tf.not_equal(x, 0), tf.bool))(indices_input)
# BiLSTM layers - using masking layer directly
bilstm = Bidirectional(LSTM(units=100, return_sequences=True))(embedding_input, mask=mask_layer)
bilstm = Dropout(0.3)(bilstm)
# Output layer
output = TimeDistributed(Dense(num_tags, activation='softmax'))(bilstm)
model = Model(inputs=[embedding_input, indices_input], outputs=output)
model.compile(
optimizer='adam',
loss='categorical_crossentropy',
metrics=['accuracy', 'recall', 'precision']
)
return model
5. 📊 Model Evaluation and Metrics¶
This function computes:
- Classification report
- Confusion matrix
- Predictions vs. true labels
In [41]:
Copied!
from sklearn.metrics import classification_report, confusion_matrix
def evaluate_model(y_true, y_pred, sequence_lengths):
"""Convert predictions to flat format and evaluate"""
# Flatten predictions and true values, respecting sequence lengths
y_pred_flat = []
y_true_flat = []
for i in range(len(y_pred)):
for j in range(sequence_lengths[i]):
y_pred_flat.append(np.argmax(y_pred[i, j]))
y_true_flat.append(np.argmax(y_true[i, j]))
# Compute metrics
report = classification_report(
y_true_flat, y_pred_flat,
target_names=tag_names,
output_dict=True
)
cm = confusion_matrix(y_true_flat, y_pred_flat)
return report, cm
from sklearn.metrics import classification_report, confusion_matrix
def evaluate_model(y_true, y_pred, sequence_lengths):
"""Convert predictions to flat format and evaluate"""
# Flatten predictions and true values, respecting sequence lengths
y_pred_flat = []
y_true_flat = []
for i in range(len(y_pred)):
for j in range(sequence_lengths[i]):
y_pred_flat.append(np.argmax(y_pred[i, j]))
y_true_flat.append(np.argmax(y_true[i, j]))
# Compute metrics
report = classification_report(
y_true_flat, y_pred_flat,
target_names=tag_names,
output_dict=True
)
cm = confusion_matrix(y_true_flat, y_pred_flat)
return report, cm
6. 🔎 Displaying Predictions: Correct vs Incorrect Examples¶
These functions display real examples from the test set, highlighting where the model predicted correctly and where it made mistakes.
In [42]:
Copied!
def show_examples(dataset, y_true, y_pred, sequence_lengths, model_name, num_examples=3):
"""Show correct and incorrect examples"""
print(f"\n{'='*20} {model_name.upper()} EXAMPLES {'='*20}")
# Find examples with errors and correct predictions
error_indices = []
correct_indices = []
for i in range(len(y_pred)):
has_error = False
for j in range(sequence_lengths[i]):
if np.argmax(y_pred[i, j]) != np.argmax(y_true[i, j]):
has_error = True
break
if has_error:
error_indices.append(i)
else:
correct_indices.append(i)
# Show correct examples
print(f"\n✅ CORRECT EXAMPLES:")
for idx in correct_indices[:num_examples]:
tokens = dataset[idx]['tokens'][:sequence_lengths[idx]]
print("\nExample:")
for j, token in enumerate(tokens):
tag = tag_names[np.argmax(y_true[idx, j])]
print(f"{token} ({tag})", end=" ")
print()
# Show incorrect examples
print(f"\n❌ INCORRECT EXAMPLES:")
for idx in error_indices[:num_examples]:
tokens = dataset[idx]['tokens'][:sequence_lengths[idx]]
print("\nExample:")
for j, token in enumerate(tokens):
true_tag = tag_names[np.argmax(y_true[idx, j])]
pred_tag = tag_names[np.argmax(y_pred[idx, j])]
if true_tag != pred_tag:
print(f"{token} (True: {true_tag}, Pred: {pred_tag})", end=" ")
else:
print(f"{token} ({true_tag})", end=" ")
print()
def show_examples(dataset, y_true, y_pred, sequence_lengths, model_name, num_examples=3):
"""Show correct and incorrect examples"""
print(f"\n{'='*20} {model_name.upper()} EXAMPLES {'='*20}")
# Find examples with errors and correct predictions
error_indices = []
correct_indices = []
for i in range(len(y_pred)):
has_error = False
for j in range(sequence_lengths[i]):
if np.argmax(y_pred[i, j]) != np.argmax(y_true[i, j]):
has_error = True
break
if has_error:
error_indices.append(i)
else:
correct_indices.append(i)
# Show correct examples
print(f"\n✅ CORRECT EXAMPLES:")
for idx in correct_indices[:num_examples]:
tokens = dataset[idx]['tokens'][:sequence_lengths[idx]]
print("\nExample:")
for j, token in enumerate(tokens):
tag = tag_names[np.argmax(y_true[idx, j])]
print(f"{token} ({tag})", end=" ")
print()
# Show incorrect examples
print(f"\n❌ INCORRECT EXAMPLES:")
for idx in error_indices[:num_examples]:
tokens = dataset[idx]['tokens'][:sequence_lengths[idx]]
print("\nExample:")
for j, token in enumerate(tokens):
true_tag = tag_names[np.argmax(y_true[idx, j])]
pred_tag = tag_names[np.argmax(y_pred[idx, j])]
if true_tag != pred_tag:
print(f"{token} (True: {true_tag}, Pred: {pred_tag})", end=" ")
else:
print(f"{token} ({true_tag})", end=" ")
print()
7. 📈 Visualizing Model Performance by Entity Type¶
We’ll compare F1 scores across different entity types (PER, LOC, ORG, MISC) and display overall metrics (precision, recall, F1) for each model.
In [47]:
Copied!
def plot_metrics(metrics_dict):
"""Plot metrics comparison across models, handling entities properly"""
models = list(metrics_dict.keys())
# 1. First plot: Entity-type performance
plt.figure(figsize=(12, 6))
# Entity types
entity_types = ['PER', 'ORG', 'LOC', 'MISC']
x = np.arange(len(entity_types))
width = 0.2
# Check for entity types and calculate F1 scores
entity_scores = {}
for model in models:
entity_scores[model] = []
for entity in entity_types:
b_tag = f'B-{entity}'
i_tag = f'I-{entity}'
# Get scores (safely with defaults)
b_score = metrics_dict[model].get(b_tag, {}).get('f1-score', 0)
i_score = metrics_dict[model].get(i_tag, {}).get('f1-score', 0)
# Average them (or just use one if only one is available)
if b_tag in metrics_dict[model] and i_tag in metrics_dict[model]:
entity_scores[model].append((b_score + i_score) / 2)
elif b_tag in metrics_dict[model]:
entity_scores[model].append(b_score)
elif i_tag in metrics_dict[model]:
entity_scores[model].append(i_score)
else:
# No score for this entity type
entity_scores[model].append(0)
# Plot entity scores
for i, model in enumerate(models):
offset = width * (i - len(models)/2 + 0.5)
plt.bar(x + offset, entity_scores[model], width, label=model)
plt.xlabel('Entity Type')
plt.ylabel('F1 Score')
plt.title('Entity Recognition Performance by Type')
plt.xticks(x, entity_types)
plt.legend()
plt.grid(axis='y', linestyle='--', alpha=0.7)
plt.tight_layout()
plt.show()
# 2. Second plot: Overall macro metrics (precision, recall, f1)
plt.figure(figsize=(12, 6))
# Metrics to plot
metrics_to_plot = ['precision', 'recall', 'f1-score']
x = np.arange(len(metrics_to_plot))
width = 0.2
# Plot overall metrics
for i, model in enumerate(models):
# Get macro average values
values = []
for metric in metrics_to_plot:
try:
values.append(metrics_dict[model]['macro avg'][metric])
except KeyError:
# Handle missing metrics gracefully
values.append(0)
offset = width * (i - len(models)/2 + 0.5)
plt.bar(x + offset, values, width, label=model)
plt.title("Overall Macro-Average Performance")
plt.xlabel("Metric")
plt.ylabel("Score")
plt.xticks(x, metrics_to_plot)
plt.legend()
plt.grid(axis='y', linestyle='--', alpha=0.7)
plt.ylim(0, 1.0)
plt.tight_layout()
plt.show()
# 3. Third plot: Weighted F1 score comparison
plt.figure(figsize=(10, 6))
# Extract weighted F1 scores
weighted_f1 = []
for model in models:
try:
weighted_f1.append(metrics_dict[model]['weighted avg']['f1-score'])
except KeyError:
weighted_f1.append(0)
# Plot weighted F1
plt.bar(models, weighted_f1, color='green')
plt.title("Weighted F1 Score Comparison")
plt.ylabel("Weighted F1 Score")
plt.ylim(0, 1.0)
# Add value labels on top of bars
for i, v in enumerate(weighted_f1):
plt.text(i, v + 0.02, f"{v:.4f}", ha='center')
plt.grid(axis='y', linestyle='--', alpha=0.7)
plt.tight_layout()
plt.show()
def plot_metrics(metrics_dict):
"""Plot metrics comparison across models, handling entities properly"""
models = list(metrics_dict.keys())
# 1. First plot: Entity-type performance
plt.figure(figsize=(12, 6))
# Entity types
entity_types = ['PER', 'ORG', 'LOC', 'MISC']
x = np.arange(len(entity_types))
width = 0.2
# Check for entity types and calculate F1 scores
entity_scores = {}
for model in models:
entity_scores[model] = []
for entity in entity_types:
b_tag = f'B-{entity}'
i_tag = f'I-{entity}'
# Get scores (safely with defaults)
b_score = metrics_dict[model].get(b_tag, {}).get('f1-score', 0)
i_score = metrics_dict[model].get(i_tag, {}).get('f1-score', 0)
# Average them (or just use one if only one is available)
if b_tag in metrics_dict[model] and i_tag in metrics_dict[model]:
entity_scores[model].append((b_score + i_score) / 2)
elif b_tag in metrics_dict[model]:
entity_scores[model].append(b_score)
elif i_tag in metrics_dict[model]:
entity_scores[model].append(i_score)
else:
# No score for this entity type
entity_scores[model].append(0)
# Plot entity scores
for i, model in enumerate(models):
offset = width * (i - len(models)/2 + 0.5)
plt.bar(x + offset, entity_scores[model], width, label=model)
plt.xlabel('Entity Type')
plt.ylabel('F1 Score')
plt.title('Entity Recognition Performance by Type')
plt.xticks(x, entity_types)
plt.legend()
plt.grid(axis='y', linestyle='--', alpha=0.7)
plt.tight_layout()
plt.show()
# 2. Second plot: Overall macro metrics (precision, recall, f1)
plt.figure(figsize=(12, 6))
# Metrics to plot
metrics_to_plot = ['precision', 'recall', 'f1-score']
x = np.arange(len(metrics_to_plot))
width = 0.2
# Plot overall metrics
for i, model in enumerate(models):
# Get macro average values
values = []
for metric in metrics_to_plot:
try:
values.append(metrics_dict[model]['macro avg'][metric])
except KeyError:
# Handle missing metrics gracefully
values.append(0)
offset = width * (i - len(models)/2 + 0.5)
plt.bar(x + offset, values, width, label=model)
plt.title("Overall Macro-Average Performance")
plt.xlabel("Metric")
plt.ylabel("Score")
plt.xticks(x, metrics_to_plot)
plt.legend()
plt.grid(axis='y', linestyle='--', alpha=0.7)
plt.ylim(0, 1.0)
plt.tight_layout()
plt.show()
# 3. Third plot: Weighted F1 score comparison
plt.figure(figsize=(10, 6))
# Extract weighted F1 scores
weighted_f1 = []
for model in models:
try:
weighted_f1.append(metrics_dict[model]['weighted avg']['f1-score'])
except KeyError:
weighted_f1.append(0)
# Plot weighted F1
plt.bar(models, weighted_f1, color='green')
plt.title("Weighted F1 Score Comparison")
plt.ylabel("Weighted F1 Score")
plt.ylim(0, 1.0)
# Add value labels on top of bars
for i, v in enumerate(weighted_f1):
plt.text(i, v + 0.02, f"{v:.4f}", ha='center')
plt.grid(axis='y', linestyle='--', alpha=0.7)
plt.tight_layout()
plt.show()
8. ⚙️ Model Training and Evaluation Pipeline¶
This function:
- Prepares data
- Builds and trains the model
- Evaluates on test data
- Displays predictions and returns metrics
In [44]:
Copied!
from tensorflow.keras.callbacks import EarlyStopping
from tensorflow.keras.layers import Masking
def run_baseline_pipeline():
"""Run training and evaluation for baseline model"""
print("\n" + "="*80)
print("BASELINE MODEL (TRAINABLE EMBEDDINGS)")
print("="*80)
# Prepare data
X_train, y_train, train_lengths = prepare_baseline_data(train_dataset)
X_val, y_val, val_lengths = prepare_baseline_data(val_dataset)
X_test, y_test, test_lengths = prepare_baseline_data(test_dataset)
# Build and train model
model = build_baseline_model()
early_stopping = EarlyStopping(monitor='val_loss', patience=2, restore_best_weights=True)
print("Training baseline model...")
history = model.fit(
X_train, y_train,
validation_data=(X_val, y_val),
epochs=5,
batch_size=64,
callbacks=[early_stopping]
)
# Evaluate
print("Evaluating baseline model...")
y_pred = model.predict(X_test)
report, cm = evaluate_model(y_test, y_pred, test_lengths)
# Display examples
show_examples(test_dataset, y_test, y_pred, test_lengths, "baseline")
return report, history, y_pred, y_test
def run_pretrained_pipeline(embedding_type):
"""Run training and evaluation for pre-trained embedding model"""
print("\n" + "="*80)
print(f"{embedding_type.upper()} MODEL")
print("="*80)
# Prepare data
print(f"Preparing {embedding_type} data...")
X_train_emb, X_train_idx, y_train, train_lengths = prepare_pretrained_data(train_dataset, embedding_type)
X_val_emb, X_val_idx, y_val, val_lengths = prepare_pretrained_data(val_dataset, embedding_type)
X_test_emb, X_test_idx, y_test, test_lengths = prepare_pretrained_data(test_dataset, embedding_type)
# Build and train model
model = build_pretrained_model(embedding_type)
early_stopping = EarlyStopping(monitor='val_loss', patience=2, restore_best_weights=True)
print(f"Training {embedding_type} model...")
history = model.fit(
[X_train_emb, X_train_idx], y_train, # Pass both inputs as a list
validation_data=([X_val_emb, X_val_idx], y_val),
epochs=5,
batch_size=64,
callbacks=[early_stopping]
)
# Evaluate
print(f"Evaluating {embedding_type} model...")
y_pred = model.predict([X_test_emb, X_test_idx]) # Pass both inputs as a list
report, cm = evaluate_model(y_test, y_pred, test_lengths)
# Display examples
show_examples(test_dataset, y_test, y_pred, test_lengths, embedding_type)
return report, history, y_pred, y_test
from tensorflow.keras.callbacks import EarlyStopping
from tensorflow.keras.layers import Masking
def run_baseline_pipeline():
"""Run training and evaluation for baseline model"""
print("\n" + "="*80)
print("BASELINE MODEL (TRAINABLE EMBEDDINGS)")
print("="*80)
# Prepare data
X_train, y_train, train_lengths = prepare_baseline_data(train_dataset)
X_val, y_val, val_lengths = prepare_baseline_data(val_dataset)
X_test, y_test, test_lengths = prepare_baseline_data(test_dataset)
# Build and train model
model = build_baseline_model()
early_stopping = EarlyStopping(monitor='val_loss', patience=2, restore_best_weights=True)
print("Training baseline model...")
history = model.fit(
X_train, y_train,
validation_data=(X_val, y_val),
epochs=5,
batch_size=64,
callbacks=[early_stopping]
)
# Evaluate
print("Evaluating baseline model...")
y_pred = model.predict(X_test)
report, cm = evaluate_model(y_test, y_pred, test_lengths)
# Display examples
show_examples(test_dataset, y_test, y_pred, test_lengths, "baseline")
return report, history, y_pred, y_test
def run_pretrained_pipeline(embedding_type):
"""Run training and evaluation for pre-trained embedding model"""
print("\n" + "="*80)
print(f"{embedding_type.upper()} MODEL")
print("="*80)
# Prepare data
print(f"Preparing {embedding_type} data...")
X_train_emb, X_train_idx, y_train, train_lengths = prepare_pretrained_data(train_dataset, embedding_type)
X_val_emb, X_val_idx, y_val, val_lengths = prepare_pretrained_data(val_dataset, embedding_type)
X_test_emb, X_test_idx, y_test, test_lengths = prepare_pretrained_data(test_dataset, embedding_type)
# Build and train model
model = build_pretrained_model(embedding_type)
early_stopping = EarlyStopping(monitor='val_loss', patience=2, restore_best_weights=True)
print(f"Training {embedding_type} model...")
history = model.fit(
[X_train_emb, X_train_idx], y_train, # Pass both inputs as a list
validation_data=([X_val_emb, X_val_idx], y_val),
epochs=5,
batch_size=64,
callbacks=[early_stopping]
)
# Evaluate
print(f"Evaluating {embedding_type} model...")
y_pred = model.predict([X_test_emb, X_test_idx]) # Pass both inputs as a list
report, cm = evaluate_model(y_test, y_pred, test_lengths)
# Display examples
show_examples(test_dataset, y_test, y_pred, test_lengths, embedding_type)
return report, history, y_pred, y_test
9. 🧪 Run All Embedding Models and Compare¶
We'll now run:
- A baseline model with trainable embeddings
- Word2Vec-based NER model
- FastText-based NER model
- ELMo-based NER model
We'll collect their metrics for comparison.
In [45]:
Copied!
"""
metrics = {}
print("Running baseline pipeline...")
baseline_report, _, _, _ = run_baseline_pipeline()
metrics["baseline"] = baseline_report
print("Running Word2Vec pipeline...")
w2v_report, _, _, _ = run_pretrained_pipeline("word2vec")
metrics["word2vec"] = w2v_report
print("Running FastText pipeline...")
ft_report, _, _, _ = run_pretrained_pipeline("fasttext")
metrics["fasttext"] = ft_report
"""
print("Running ELMo pipeline...")
elmo_report, _, _, _ = run_pretrained_pipeline("elmo")
metrics["elmo"] = elmo_report
"""
metrics = {}
print("Running baseline pipeline...")
baseline_report, _, _, _ = run_baseline_pipeline()
metrics["baseline"] = baseline_report
print("Running Word2Vec pipeline...")
w2v_report, _, _, _ = run_pretrained_pipeline("word2vec")
metrics["word2vec"] = w2v_report
print("Running FastText pipeline...")
ft_report, _, _, _ = run_pretrained_pipeline("fasttext")
metrics["fasttext"] = ft_report
"""
print("Running ELMo pipeline...")
elmo_report, _, _, _ = run_pretrained_pipeline("elmo")
metrics["elmo"] = elmo_report
Running ELMo pipeline... ================================================================================ ELMO MODEL ================================================================================ Preparing elmo data... Extracting elmo embeddings...
100%|██████████| 157/157 [09:00<00:00, 3.44s/it]
Extracting elmo embeddings...
100%|██████████| 16/16 [00:46<00:00, 2.89s/it]
Extracting elmo embeddings...
100%|██████████| 32/32 [01:38<00:00, 3.06s/it]
Training elmo model... Epoch 1/5 79/79 ━━━━━━━━━━━━━━━━━━━━ 12s 114ms/step - accuracy: 0.9454 - loss: 0.2046 - precision: 0.8788 - recall: 0.5675 - val_accuracy: 0.9902 - val_loss: 0.0371 - val_precision: 0.9783 - val_recall: 0.9175 Epoch 2/5 79/79 ━━━━━━━━━━━━━━━━━━━━ 10s 123ms/step - accuracy: 0.9893 - loss: 0.0386 - precision: 0.9785 - recall: 0.9274 - val_accuracy: 0.9944 - val_loss: 0.0215 - val_precision: 0.9824 - val_recall: 0.9570 Epoch 3/5 79/79 ━━━━━━━━━━━━━━━━━━━━ 10s 129ms/step - accuracy: 0.9936 - loss: 0.0229 - precision: 0.9839 - recall: 0.9594 - val_accuracy: 0.9958 - val_loss: 0.0158 - val_precision: 0.9837 - val_recall: 0.9718 Epoch 4/5 79/79 ━━━━━━━━━━━━━━━━━━━━ 10s 130ms/step - accuracy: 0.9959 - loss: 0.0152 - precision: 0.9876 - recall: 0.9747 - val_accuracy: 0.9962 - val_loss: 0.0133 - val_precision: 0.9848 - val_recall: 0.9763 Epoch 5/5 79/79 ━━━━━━━━━━━━━━━━━━━━ 10s 132ms/step - accuracy: 0.9973 - loss: 0.0108 - precision: 0.9914 - recall: 0.9839 - val_accuracy: 0.9965 - val_loss: 0.0119 - val_precision: 0.9860 - val_recall: 0.9784 Evaluating elmo model... 32/32 ━━━━━━━━━━━━━━━━━━━━ 1s 31ms/step ==================== ELMO EXAMPLES ==================== ✅ CORRECT EXAMPLES: Example: Nadim (B-PER) Ladki (I-PER) Example: AL-AIN (B-LOC) , (O) United (B-LOC) Arab (I-LOC) Emirates (I-LOC) 1996-12-06 (O) Example: But (O) China (B-LOC) saw (O) their (O) luck (O) desert (O) them (O) in (O) the (O) second (O) match (O) of (O) the (O) group (O) , (O) crashing (O) to (O) a (O) surprise (O) 2-0 (O) defeat (O) to (O) newcomers (O) Uzbekistan (B-LOC) . (O) ❌ INCORRECT EXAMPLES: Example: SOCCER (O) - (O) JAPAN (B-LOC) GET (O) LUCKY (O) WIN (O) , (O) CHINA (True: B-PER, Pred: B-LOC) IN (O) SURPRISE (O) DEFEAT (O) . (O) Example: Japan (B-LOC) began (O) the (O) defence (O) of (O) their (O) Asian (B-MISC) Cup (I-MISC) title (O) with (O) a (O) lucky (O) 2-1 (O) win (O) against (O) Syria (B-LOC) in (O) a (O) Group (O) C (True: O, Pred: I-MISC) championship (O) match (O) on (O) Friday (O) . (O) Example: RUGBY (B-ORG) UNION (I-ORG) - (O) CUTTITTA (True: B-PER, Pred: B-ORG) BACK (O) FOR (O) ITALY (B-LOC) AFTER (O) A (O) YEAR (O) . (O)
10. 📋 Final Summary and Comparison¶
Here’s the overall comparison of F1 scores and per-entity performance across all models.
In [50]:
Copied!
print("\n" + "="*80)
print("📊 FINAL COMPARISON")
print("="*80)
for model in metrics:
print(f"\n{model.upper()}:")
print(f"F1 Score (macro): {metrics[model]['macro avg']['f1-score']:.4f}")
print(f"F1 Score (weighted): {metrics[model]['weighted avg']['f1-score']:.4f}")
print(f"Precision (macro): {metrics[model]['macro avg']['precision']:.4f}")
print(f"Recall (macro): {metrics[model]['macro avg']['recall']:.4f}")
plot_metrics(metrics)
print("\n" + "="*80)
print("📊 FINAL COMPARISON")
print("="*80)
for model in metrics:
print(f"\n{model.upper()}:")
print(f"F1 Score (macro): {metrics[model]['macro avg']['f1-score']:.4f}")
print(f"F1 Score (weighted): {metrics[model]['weighted avg']['f1-score']:.4f}")
print(f"Precision (macro): {metrics[model]['macro avg']['precision']:.4f}")
print(f"Recall (macro): {metrics[model]['macro avg']['recall']:.4f}")
plot_metrics(metrics)
================================================================================ 📊 FINAL COMPARISON ================================================================================ BASELINE: F1 Score (macro): 0.5967 F1 Score (weighted): 0.8763 Precision (macro): 0.6659 Recall (macro): 0.5619 WORD2VEC: F1 Score (macro): 0.4983 F1 Score (weighted): 0.8835 Precision (macro): 0.6128 Recall (macro): 0.4797 FASTTEXT: F1 Score (macro): 0.6885 F1 Score (weighted): 0.9238 Precision (macro): 0.8085 Recall (macro): 0.6392 ELMO: F1 Score (macro): 0.8768 F1 Score (weighted): 0.9713 Precision (macro): 0.8868 Recall (macro): 0.8691
📈 Final Analysis: Named Entity Recognition with Different Embeddings¶
After training and evaluating four models on the CoNLL-2003 dataset, we can clearly observe how the choice of word embedding impacts downstream performance in a Named Entity Recognition (NER) task.
⚖️ Macro F1-Score Comparison:¶
| Model | Macro F1 | Weighted F1 | Precision | Recall |
|---|---|---|---|---|
| Baseline (trainable embedding) | 0.5967 | 0.8763 | 0.6659 | 0.5619 |
| Word2Vec | 0.4983 | 0.8835 | 0.6128 | 0.4797 |
| FastText | 0.6885 | 0.9238 | 0.8085 | 0.6392 |
| ELMo | 0.8768 | 0.9713 | 0.8868 | 0.8691 |
🔍 Key Takeaways:¶
Word Embeddings Improve Performance
Even the simplest static embeddings like Word2Vec or FastText improve over the baseline model (weighted F1), proving the value of transfer learning through pre-trained vectors.FastText Outperforms Word2Vec
FastText captures subword information and is especially useful for rare or out-of-vocabulary words — resulting in better generalization and stronger metrics than Word2Vec.ELMo Dominates the Task
With its deep, contextualized representations, ELMo delivers the highest performance across the board. It excels at capturing the meaning of words in their specific context, making it ideal for NER, where ambiguity and polysemy are common (e.g., “Apple” as a fruit vs. a company).Macro vs. Weighted F1
While weighted F1 scores are high across models due to class imbalance, the macro F1 highlights how well the model handles all entity types equally. ELMo again shines in this setting, offering balanced performance across all classes.
📌 Final Thoughts¶
This extrinsic evaluation validates the findings from our intrinsic experiments:
Contextual embeddings like ELMo significantly outperform static embeddings in real NLP tasks.
📌 Extrinsic Evaluation — Sentiment Classification on SST-2¶
In this final extrinsic evaluation, we assess the utility of word embeddings for sentence-level sentiment classification using the SST-2 (Stanford Sentiment Treebank) dataset.
💡 Why This Task?¶
While NER evaluated how well embeddings captured sequence labeling capabilities, sentiment classification examines how well they support global sentence understanding — especially important for capturing compositional meaning and nuanced expressions.
🧪 Embedding Models Compared:¶
- Baseline: Trainable embedding from scratch
- Word2Vec: Static embedding trained on a large corpus
- FastText: Subword-aware static embedding
- ELMo: Deep, contextual embedding from biLMs
📥 Loading the SST-2 Dataset¶
We load a subset of the SST-2 dataset from Hugging Face, limiting each split for fast experimentation.
In [68]:
Copied!
# Load the SST-2 dataset
# Load the SST-2 dataset
print("Loading SST-2 dataset...")
sst2_dataset = load_dataset("stanfordnlp/sst2")
# Check the label distribution to understand the dataset
print("\nExamining label distribution...")
train_labels = sst2_dataset["train"]["label"]
label_counts = {}
for label in train_labels:
if label not in label_counts:
label_counts[label] = 0
label_counts[label] += 1
print(f"Label distribution in training set: {label_counts}")
# Get number of classes
num_classes = len(label_counts)
print(f"Number of classes: {num_classes}")
# Subsample for quicker evaluation
dataset= sst2_dataset["train"].shuffle(seed=42)
train_dataset = dataset.select(range(5000))
test_dataset = dataset.select(range(5000, 6000))
val_dataset = dataset.select(range(6000, 6500))
print(f"Train set: {len(train_dataset)} examples")
print(f"Validation set: {len(val_dataset)} examples")
print(f"Test set: {len(test_dataset)} examples")
# View a sample
sample = train_dataset[0]
print("\nSample data:")
print(f"Sentence: {sample['sentence']}")
print(f"Label: {sample['label']}")
# Load the SST-2 dataset
# Load the SST-2 dataset
print("Loading SST-2 dataset...")
sst2_dataset = load_dataset("stanfordnlp/sst2")
# Check the label distribution to understand the dataset
print("\nExamining label distribution...")
train_labels = sst2_dataset["train"]["label"]
label_counts = {}
for label in train_labels:
if label not in label_counts:
label_counts[label] = 0
label_counts[label] += 1
print(f"Label distribution in training set: {label_counts}")
# Get number of classes
num_classes = len(label_counts)
print(f"Number of classes: {num_classes}")
# Subsample for quicker evaluation
dataset= sst2_dataset["train"].shuffle(seed=42)
train_dataset = dataset.select(range(5000))
test_dataset = dataset.select(range(5000, 6000))
val_dataset = dataset.select(range(6000, 6500))
print(f"Train set: {len(train_dataset)} examples")
print(f"Validation set: {len(val_dataset)} examples")
print(f"Test set: {len(test_dataset)} examples")
# View a sample
sample = train_dataset[0]
print("\nSample data:")
print(f"Sentence: {sample['sentence']}")
print(f"Label: {sample['label']}")
Loading SST-2 dataset...
Examining label distribution...
Label distribution in training set: {0: 29780, 1: 37569}
Number of classes: 2
Train set: 5000 examples
Validation set: 500 examples
Test set: 1000 examples
Sample data:
Sentence: klein , charming in comedies like american pie and dead-on in election ,
Label: 1
🧾 Vocabulary Construction and Sequence Padding¶
We tokenize each sentence and build a word-to-index mapping to prepare sequences for the models. We also fix the maximum sequence length.
In [69]:
Copied!
# Create vocabulary from training data
print("\nCreating vocabulary...")
word_to_idx = {'PAD': 0, 'UNK': 1}
for example in train_dataset:
for token in example['sentence'].split():
if token not in word_to_idx:
word_to_idx[token] = len(word_to_idx)
print(f"Vocabulary size: {len(word_to_idx)}")
# Set maximum sequence length
max_len = max(len(example['sentence'].split()) for example in train_dataset)
print(f"Maximum sequence length: {max_len}")
max_len = min(max_len, 50) # Cap at 50 tokens
print(f"Using sequence length: {max_len}")
# Create vocabulary from training data
print("\nCreating vocabulary...")
word_to_idx = {'PAD': 0, 'UNK': 1}
for example in train_dataset:
for token in example['sentence'].split():
if token not in word_to_idx:
word_to_idx[token] = len(word_to_idx)
print(f"Vocabulary size: {len(word_to_idx)}")
# Set maximum sequence length
max_len = max(len(example['sentence'].split()) for example in train_dataset)
print(f"Maximum sequence length: {max_len}")
max_len = min(max_len, 50) # Cap at 50 tokens
print(f"Using sequence length: {max_len}")
Creating vocabulary... Vocabulary size: 7652 Maximum sequence length: 50 Using sequence length: 50
⚙️ Data Preparation¶
We define separate functions to prepare input data for the baseline (trainable embeddings) and pretrained models (Word2Vec, FastText, ELMo).
In [71]:
Copied!
# Function to prepare data for the baseline model
def prepare_baseline_data(dataset):
X = []
y = []
sequence_lengths = []
for example in dataset:
# Tokenize the sentence
tokens = example['sentence'].split()
seq_len = min(len(tokens), max_len)
sequence_lengths.append(seq_len)
# Convert tokens to indices
token_indices = [word_to_idx.get(token, word_to_idx['UNK']) for token in tokens]
# Pad or truncate to max_len
padded_indices = pad_sequences([token_indices], maxlen=max_len, padding='post', truncating='post')[0]
X.append(padded_indices)
# Get label
y.append(example['label'])
return np.array(X), np.array(y), sequence_lengths
def prepare_pretrained_data(dataset, embedding_type):
"""Prepare data with pre-trained embeddings"""
X_indices = []
y = []
sequence_lengths = []
# First pass: collect tokens and prepare indices and labels
for example in dataset:
# Tokenize the sentence
tokens = example['sentence'].split()
seq_len = min(len(tokens), max_len)
sequence_lengths.append(seq_len)
# Convert tokens to indices
token_indices = [word_to_idx.get(token, word_to_idx['UNK']) for token in tokens]
# Pad to max_len
padded_indices = pad_sequences([token_indices], maxlen=max_len, padding='post', truncating='post')[0]
X_indices.append(padded_indices)
# Get label
y.append(example['label'])
X_indices = np.array(X_indices)
y = np.array(y)
# Determine embedding dimension
embedding_dim = 300 # Default for Word2Vec and FastText
if embedding_type == "elmo":
embedding_dim = 512
# Create empty embedding matrix
X_embeddings = np.zeros((len(dataset), max_len, embedding_dim))
# Extract embeddings based on embedding type
print(f"Extracting {embedding_type} embeddings...")
if embedding_type == "elmo":
# Optimized batch processing for ELMo
batch_size = 64 # Process this many examples at once
for batch_start in tqdm(range(0, len(dataset), batch_size)):
batch_end = min(batch_start + batch_size, len(dataset))
# Process each example in the batch
for i in range(batch_end - batch_start):
# Get actual index in the full dataset
example_idx = batch_start + i
# Get current example from the dataset
if example_idx < len(dataset):
example = dataset[example_idx]
# Get tokens
tokens = example['sentence'].split()[:max_len]
# Create a sentence
full_sentence = example['sentence']
try:
# Get embeddings for the entire sentence at once
embeddings = elmo_model.signatures["default"](
tf.constant([full_sentence])
)
# Extract embeddings for each token position
word_embeddings = embeddings["word_emb"][0, :len(tokens), :].numpy()
# Store in our embedding matrix
X_embeddings[example_idx, :len(tokens), :] = word_embeddings
except Exception as e:
print(f"Error getting ELMo embeddings for example {example_idx}: {e}")
# Leave as zeros for this example
else:
# Word2Vec and FastText extraction
for i, example in enumerate(tqdm(dataset)):
tokens = example['sentence'].split()[:max_len]
for j, token in enumerate(tokens):
if embedding_type == "word2vec":
embedding = get_word2vec_embedding(token.lower())
elif embedding_type == "fasttext":
embedding = get_fasttext_embedding(token.lower())
# Use zeros for missing embeddings
if embedding is not None:
X_embeddings[i, j] = embedding
return X_embeddings, X_indices, y, sequence_lengths
# Function to prepare data for the baseline model
def prepare_baseline_data(dataset):
X = []
y = []
sequence_lengths = []
for example in dataset:
# Tokenize the sentence
tokens = example['sentence'].split()
seq_len = min(len(tokens), max_len)
sequence_lengths.append(seq_len)
# Convert tokens to indices
token_indices = [word_to_idx.get(token, word_to_idx['UNK']) for token in tokens]
# Pad or truncate to max_len
padded_indices = pad_sequences([token_indices], maxlen=max_len, padding='post', truncating='post')[0]
X.append(padded_indices)
# Get label
y.append(example['label'])
return np.array(X), np.array(y), sequence_lengths
def prepare_pretrained_data(dataset, embedding_type):
"""Prepare data with pre-trained embeddings"""
X_indices = []
y = []
sequence_lengths = []
# First pass: collect tokens and prepare indices and labels
for example in dataset:
# Tokenize the sentence
tokens = example['sentence'].split()
seq_len = min(len(tokens), max_len)
sequence_lengths.append(seq_len)
# Convert tokens to indices
token_indices = [word_to_idx.get(token, word_to_idx['UNK']) for token in tokens]
# Pad to max_len
padded_indices = pad_sequences([token_indices], maxlen=max_len, padding='post', truncating='post')[0]
X_indices.append(padded_indices)
# Get label
y.append(example['label'])
X_indices = np.array(X_indices)
y = np.array(y)
# Determine embedding dimension
embedding_dim = 300 # Default for Word2Vec and FastText
if embedding_type == "elmo":
embedding_dim = 512
# Create empty embedding matrix
X_embeddings = np.zeros((len(dataset), max_len, embedding_dim))
# Extract embeddings based on embedding type
print(f"Extracting {embedding_type} embeddings...")
if embedding_type == "elmo":
# Optimized batch processing for ELMo
batch_size = 64 # Process this many examples at once
for batch_start in tqdm(range(0, len(dataset), batch_size)):
batch_end = min(batch_start + batch_size, len(dataset))
# Process each example in the batch
for i in range(batch_end - batch_start):
# Get actual index in the full dataset
example_idx = batch_start + i
# Get current example from the dataset
if example_idx < len(dataset):
example = dataset[example_idx]
# Get tokens
tokens = example['sentence'].split()[:max_len]
# Create a sentence
full_sentence = example['sentence']
try:
# Get embeddings for the entire sentence at once
embeddings = elmo_model.signatures["default"](
tf.constant([full_sentence])
)
# Extract embeddings for each token position
word_embeddings = embeddings["word_emb"][0, :len(tokens), :].numpy()
# Store in our embedding matrix
X_embeddings[example_idx, :len(tokens), :] = word_embeddings
except Exception as e:
print(f"Error getting ELMo embeddings for example {example_idx}: {e}")
# Leave as zeros for this example
else:
# Word2Vec and FastText extraction
for i, example in enumerate(tqdm(dataset)):
tokens = example['sentence'].split()[:max_len]
for j, token in enumerate(tokens):
if embedding_type == "word2vec":
embedding = get_word2vec_embedding(token.lower())
elif embedding_type == "fasttext":
embedding = get_fasttext_embedding(token.lower())
# Use zeros for missing embeddings
if embedding is not None:
X_embeddings[i, j] = embedding
return X_embeddings, X_indices, y, sequence_lengths
🧠 Model Definitions¶
Two types of models are used:
- Baseline Model with a trainable embedding layer
- Pretrained Models using fixed embeddings (passed as input)
In [72]:
Copied!
def build_baseline_model():
"""Build a baseline BiLSTM model with trainable embeddings"""
input_layer = Input(shape=(max_len,))
# Trainable embedding layer
embedding_layer = Embedding(
input_dim=len(word_to_idx),
output_dim=100,
input_length=max_len,
mask_zero=True
)(input_layer)
# Bidirectional LSTM
bilstm = Bidirectional(LSTM(units=100))(embedding_layer)
bilstm = Dropout(0.3)(bilstm)
# Output layer
output = Dense(1, activation='sigmoid')(bilstm)
model = Model(inputs=input_layer, outputs=output)
model.compile(
optimizer='adam',
loss='binary_crossentropy',
metrics=['accuracy', 'recall', 'precision']
)
return model
def build_pretrained_model(embedding_type):
"""Build a model for pre-trained embeddings"""
embedding_dim = 300 # Default for Word2Vec and FastText
if embedding_type == "elmo":
embedding_dim = 512
# Input layer
embedding_input = Input(shape=(max_len, embedding_dim))
# Add masking layer
masked_embeddings = Masking(mask_value=0.0)(embedding_input)
# BiLSTM layer
bilstm = Bidirectional(LSTM(units=100))(masked_embeddings)
bilstm = Dropout(0.3)(bilstm)
# Output layer
output = Dense(1, activation='sigmoid')(bilstm)
model = Model(inputs=embedding_input, outputs=output)
model.compile(
optimizer='adam',
loss='binary_crossentropy',
metrics=['accuracy', 'recall', 'precision']
)
return model
def build_baseline_model():
"""Build a baseline BiLSTM model with trainable embeddings"""
input_layer = Input(shape=(max_len,))
# Trainable embedding layer
embedding_layer = Embedding(
input_dim=len(word_to_idx),
output_dim=100,
input_length=max_len,
mask_zero=True
)(input_layer)
# Bidirectional LSTM
bilstm = Bidirectional(LSTM(units=100))(embedding_layer)
bilstm = Dropout(0.3)(bilstm)
# Output layer
output = Dense(1, activation='sigmoid')(bilstm)
model = Model(inputs=input_layer, outputs=output)
model.compile(
optimizer='adam',
loss='binary_crossentropy',
metrics=['accuracy', 'recall', 'precision']
)
return model
def build_pretrained_model(embedding_type):
"""Build a model for pre-trained embeddings"""
embedding_dim = 300 # Default for Word2Vec and FastText
if embedding_type == "elmo":
embedding_dim = 512
# Input layer
embedding_input = Input(shape=(max_len, embedding_dim))
# Add masking layer
masked_embeddings = Masking(mask_value=0.0)(embedding_input)
# BiLSTM layer
bilstm = Bidirectional(LSTM(units=100))(masked_embeddings)
bilstm = Dropout(0.3)(bilstm)
# Output layer
output = Dense(1, activation='sigmoid')(bilstm)
model = Model(inputs=embedding_input, outputs=output)
model.compile(
optimizer='adam',
loss='binary_crossentropy',
metrics=['accuracy', 'recall', 'precision']
)
return model
🧪 Evaluation Utilities¶
We define utilities for:
- Computing metrics like Accuracy and F1 Score
- Plotting bar charts and confusion matrices
- Printing correct/incorrect predictions
In [73]:
Copied!
from sklearn.metrics import accuracy_score, f1_score, classification_report, confusion_matrix
# Evaluation helpers
def evaluate_model(y_true, y_pred):
"""Evaluate classification performance"""
# Convert probabilities to binary predictions
y_pred_binary = (y_pred > 0.5).astype(int)
# Compute metrics
accuracy = accuracy_score(y_true, y_pred_binary)
f1 = f1_score(y_true, y_pred_binary, average='macro')
report = classification_report(y_true, y_pred_binary, output_dict=True)
cm = confusion_matrix(y_true, y_pred_binary)
return accuracy, f1, report, cm
def plot_results(metrics):
"""Plot comparison between models"""
models = list(metrics.keys())
# Accuracy and F1 score
plt.figure(figsize=(10, 6))
accuracy_scores = [metrics[model]['accuracy'] for model in models]
f1_scores = [metrics[model]['f1'] for model in models]
x = np.arange(len(models))
width = 0.35
plt.bar(x - width/2, accuracy_scores, width, label='Accuracy', color='skyblue')
plt.bar(x + width/2, f1_scores, width, label='F1 Score', color='lightgreen')
plt.xlabel('Model')
plt.ylabel('Score')
plt.title('Model Performance Comparison')
plt.xticks(x, models)
plt.ylim(0, 1)
plt.legend()
plt.grid(axis='y', linestyle='--', alpha=0.7)
plt.tight_layout()
plt.show()
# Confusion matrices
plt.figure(figsize=(15, 5))
for i, model in enumerate(models):
plt.subplot(1, len(models), i+1)
cm = metrics[model]['confusion_matrix']
sns.heatmap(cm, annot=True, fmt='d', cmap='Blues', cbar=False)
plt.title(f'{model.capitalize()} Confusion Matrix')
plt.xlabel('Predicted')
plt.ylabel('True')
plt.tight_layout()
plt.show()
def show_examples(dataset, y_true, y_pred, model_name, num_examples=5):
"""Show correct and incorrect examples"""
print(f"\n{'='*20} {model_name.upper()} EXAMPLES {'='*20}")
# Convert probabilities to binary predictions
y_pred_binary = (y_pred > 0.5).astype(int)
# Find examples with errors and correct predictions
correct_indices = []
error_indices = []
for i in range(len(y_pred)):
if y_pred_binary[i] == y_true[i]:
correct_indices.append(i)
else:
error_indices.append(i)
# Show correct examples
print(f"\n✅ CORRECT EXAMPLES:")
for idx in correct_indices[:num_examples]:
text = dataset[idx]['sentence']
true_label = "Positive" if y_true[idx] == 1 else "Negative"
confidence = max(y_pred[idx], 1 - y_pred[idx])
print(f"Text: {text}")
print(f"True label: {true_label}, Model confidence: {confidence:.4f}")
print()
# Show incorrect examples
print(f"\n❌ INCORRECT EXAMPLES:")
for idx in error_indices[:num_examples]:
text = dataset[idx]['sentence']
true_label = "Positive" if y_true[idx] == 1 else "Negative"
pred_label = "Positive" if y_pred_binary[idx] == 1 else "Negative"
confidence = max(y_pred[idx], 1 - y_pred[idx])
print(f"Text: {text}")
print(f"True label: {true_label}, Predicted: {pred_label}, Confidence: {confidence:.4f}")
print()
from sklearn.metrics import accuracy_score, f1_score, classification_report, confusion_matrix
# Evaluation helpers
def evaluate_model(y_true, y_pred):
"""Evaluate classification performance"""
# Convert probabilities to binary predictions
y_pred_binary = (y_pred > 0.5).astype(int)
# Compute metrics
accuracy = accuracy_score(y_true, y_pred_binary)
f1 = f1_score(y_true, y_pred_binary, average='macro')
report = classification_report(y_true, y_pred_binary, output_dict=True)
cm = confusion_matrix(y_true, y_pred_binary)
return accuracy, f1, report, cm
def plot_results(metrics):
"""Plot comparison between models"""
models = list(metrics.keys())
# Accuracy and F1 score
plt.figure(figsize=(10, 6))
accuracy_scores = [metrics[model]['accuracy'] for model in models]
f1_scores = [metrics[model]['f1'] for model in models]
x = np.arange(len(models))
width = 0.35
plt.bar(x - width/2, accuracy_scores, width, label='Accuracy', color='skyblue')
plt.bar(x + width/2, f1_scores, width, label='F1 Score', color='lightgreen')
plt.xlabel('Model')
plt.ylabel('Score')
plt.title('Model Performance Comparison')
plt.xticks(x, models)
plt.ylim(0, 1)
plt.legend()
plt.grid(axis='y', linestyle='--', alpha=0.7)
plt.tight_layout()
plt.show()
# Confusion matrices
plt.figure(figsize=(15, 5))
for i, model in enumerate(models):
plt.subplot(1, len(models), i+1)
cm = metrics[model]['confusion_matrix']
sns.heatmap(cm, annot=True, fmt='d', cmap='Blues', cbar=False)
plt.title(f'{model.capitalize()} Confusion Matrix')
plt.xlabel('Predicted')
plt.ylabel('True')
plt.tight_layout()
plt.show()
def show_examples(dataset, y_true, y_pred, model_name, num_examples=5):
"""Show correct and incorrect examples"""
print(f"\n{'='*20} {model_name.upper()} EXAMPLES {'='*20}")
# Convert probabilities to binary predictions
y_pred_binary = (y_pred > 0.5).astype(int)
# Find examples with errors and correct predictions
correct_indices = []
error_indices = []
for i in range(len(y_pred)):
if y_pred_binary[i] == y_true[i]:
correct_indices.append(i)
else:
error_indices.append(i)
# Show correct examples
print(f"\n✅ CORRECT EXAMPLES:")
for idx in correct_indices[:num_examples]:
text = dataset[idx]['sentence']
true_label = "Positive" if y_true[idx] == 1 else "Negative"
confidence = max(y_pred[idx], 1 - y_pred[idx])
print(f"Text: {text}")
print(f"True label: {true_label}, Model confidence: {confidence:.4f}")
print()
# Show incorrect examples
print(f"\n❌ INCORRECT EXAMPLES:")
for idx in error_indices[:num_examples]:
text = dataset[idx]['sentence']
true_label = "Positive" if y_true[idx] == 1 else "Negative"
pred_label = "Positive" if y_pred_binary[idx] == 1 else "Negative"
confidence = max(y_pred[idx], 1 - y_pred[idx])
print(f"Text: {text}")
print(f"True label: {true_label}, Predicted: {pred_label}, Confidence: {confidence:.4f}")
print()
🚀 Training Pipelines¶
We define one function for each model type that handles training and evaluation:
In [74]:
Copied!
def run_baseline_pipeline():
"""Run training and evaluation for baseline model"""
print("\n" + "="*80)
print("BASELINE MODEL (TRAINABLE EMBEDDINGS)")
print("="*80)
# Prepare data
X_train, y_train, train_lengths = prepare_baseline_data(train_dataset)
X_val, y_val, val_lengths = prepare_baseline_data(val_dataset)
X_test, y_test, test_lengths = prepare_baseline_data(test_dataset)
# Build and train model
model = build_baseline_model()
early_stopping = EarlyStopping(monitor='val_loss', patience=2, restore_best_weights=True)
print("Training baseline model...")
history = model.fit(
X_train, y_train,
validation_data=(X_val, y_val),
epochs=10,
batch_size=64,
callbacks=[early_stopping]
)
# Evaluate
print("Evaluating baseline model...")
y_pred = model.predict(X_test).flatten()
accuracy, f1, report, cm = evaluate_model(y_test, y_pred)
print(f"Accuracy: {accuracy:.4f}")
print(f"F1 Score: {f1:.4f}")
# Display examples
show_examples(test_dataset, y_test, y_pred, "baseline")
return {
'accuracy': accuracy,
'f1': f1,
'report': report,
'confusion_matrix': cm,
'predictions': y_pred,
'true_labels': y_test
}
def run_pretrained_pipeline(embedding_type):
"""Run training and evaluation for pre-trained embedding model"""
print("\n" + "="*80)
print(f"{embedding_type.upper()} MODEL")
print("="*80)
# Prepare data
print(f"Preparing {embedding_type} data...")
X_train_emb, X_train_idx, y_train, train_lengths = prepare_pretrained_data(train_dataset, embedding_type)
X_val_emb, X_val_idx, y_val, val_lengths = prepare_pretrained_data(val_dataset, embedding_type)
X_test_emb, X_test_idx, y_test, test_lengths = prepare_pretrained_data(test_dataset, embedding_type)
# Build and train model
model = build_pretrained_model(embedding_type)
early_stopping = EarlyStopping(monitor='val_loss', patience=2, restore_best_weights=True)
print(f"Training {embedding_type} model...")
history = model.fit(
X_train_emb, y_train,
validation_data=(X_val_emb, y_val),
epochs=10,
batch_size=64,
callbacks=[early_stopping]
)
# Evaluate
print(f"Evaluating {embedding_type} model...")
y_pred = model.predict(X_test_emb).flatten()
accuracy, f1, report, cm = evaluate_model(y_test, y_pred)
print(f"Accuracy: {accuracy:.4f}")
print(f"F1 Score: {f1:.4f}")
# Display examples
show_examples(test_dataset, y_test, y_pred, embedding_type)
return {
'accuracy': accuracy,
'f1': f1,
'report': report,
'confusion_matrix': cm,
'predictions': y_pred,
'true_labels': y_test
}
def run_baseline_pipeline():
"""Run training and evaluation for baseline model"""
print("\n" + "="*80)
print("BASELINE MODEL (TRAINABLE EMBEDDINGS)")
print("="*80)
# Prepare data
X_train, y_train, train_lengths = prepare_baseline_data(train_dataset)
X_val, y_val, val_lengths = prepare_baseline_data(val_dataset)
X_test, y_test, test_lengths = prepare_baseline_data(test_dataset)
# Build and train model
model = build_baseline_model()
early_stopping = EarlyStopping(monitor='val_loss', patience=2, restore_best_weights=True)
print("Training baseline model...")
history = model.fit(
X_train, y_train,
validation_data=(X_val, y_val),
epochs=10,
batch_size=64,
callbacks=[early_stopping]
)
# Evaluate
print("Evaluating baseline model...")
y_pred = model.predict(X_test).flatten()
accuracy, f1, report, cm = evaluate_model(y_test, y_pred)
print(f"Accuracy: {accuracy:.4f}")
print(f"F1 Score: {f1:.4f}")
# Display examples
show_examples(test_dataset, y_test, y_pred, "baseline")
return {
'accuracy': accuracy,
'f1': f1,
'report': report,
'confusion_matrix': cm,
'predictions': y_pred,
'true_labels': y_test
}
def run_pretrained_pipeline(embedding_type):
"""Run training and evaluation for pre-trained embedding model"""
print("\n" + "="*80)
print(f"{embedding_type.upper()} MODEL")
print("="*80)
# Prepare data
print(f"Preparing {embedding_type} data...")
X_train_emb, X_train_idx, y_train, train_lengths = prepare_pretrained_data(train_dataset, embedding_type)
X_val_emb, X_val_idx, y_val, val_lengths = prepare_pretrained_data(val_dataset, embedding_type)
X_test_emb, X_test_idx, y_test, test_lengths = prepare_pretrained_data(test_dataset, embedding_type)
# Build and train model
model = build_pretrained_model(embedding_type)
early_stopping = EarlyStopping(monitor='val_loss', patience=2, restore_best_weights=True)
print(f"Training {embedding_type} model...")
history = model.fit(
X_train_emb, y_train,
validation_data=(X_val_emb, y_val),
epochs=10,
batch_size=64,
callbacks=[early_stopping]
)
# Evaluate
print(f"Evaluating {embedding_type} model...")
y_pred = model.predict(X_test_emb).flatten()
accuracy, f1, report, cm = evaluate_model(y_test, y_pred)
print(f"Accuracy: {accuracy:.4f}")
print(f"F1 Score: {f1:.4f}")
# Display examples
show_examples(test_dataset, y_test, y_pred, embedding_type)
return {
'accuracy': accuracy,
'f1': f1,
'report': report,
'confusion_matrix': cm,
'predictions': y_pred,
'true_labels': y_test
}
In [75]:
Copied!
# Run all pipelines
metrics = {}
print("Running baseline pipeline...")
baseline_results = run_baseline_pipeline()
metrics["baseline"] = baseline_results
print("Running Word2Vec pipeline...")
w2v_results = run_pretrained_pipeline("word2vec")
metrics["word2vec"] = w2v_results
print("Running FastText pipeline...")
ft_results = run_pretrained_pipeline("fasttext")
metrics["fasttext"] = ft_results
print("Running ELMo pipeline...")
elmo_results = run_pretrained_pipeline("elmo")
metrics["elmo"] = elmo_results
# Run all pipelines
metrics = {}
print("Running baseline pipeline...")
baseline_results = run_baseline_pipeline()
metrics["baseline"] = baseline_results
print("Running Word2Vec pipeline...")
w2v_results = run_pretrained_pipeline("word2vec")
metrics["word2vec"] = w2v_results
print("Running FastText pipeline...")
ft_results = run_pretrained_pipeline("fasttext")
metrics["fasttext"] = ft_results
print("Running ELMo pipeline...")
elmo_results = run_pretrained_pipeline("elmo")
metrics["elmo"] = elmo_results
Running baseline pipeline... ================================================================================ BASELINE MODEL (TRAINABLE EMBEDDINGS) ================================================================================ Training baseline model... Epoch 1/10
/Users/agomberto/Library/Caches/pypoetry/virtualenvs/bse-nlp-DetGwK6_-py3.11/lib/python3.11/site-packages/keras/src/layers/core/embedding.py:90: UserWarning: Argument `input_length` is deprecated. Just remove it. warnings.warn(
79/79 ━━━━━━━━━━━━━━━━━━━━ 5s 43ms/step - accuracy: 0.5420 - loss: 0.6800 - precision: 0.5437 - recall: 0.7726 - val_accuracy: 0.7220 - val_loss: 0.5864 - val_precision: 0.7305 - val_recall: 0.7658 Epoch 2/10 79/79 ━━━━━━━━━━━━━━━━━━━━ 6s 81ms/step - accuracy: 0.8438 - loss: 0.4340 - precision: 0.8409 - recall: 0.8732 - val_accuracy: 0.7640 - val_loss: 0.5127 - val_precision: 0.7745 - val_recall: 0.7918 Epoch 3/10 79/79 ━━━━━━━━━━━━━━━━━━━━ 7s 85ms/step - accuracy: 0.9352 - loss: 0.2012 - precision: 0.9431 - recall: 0.9353 - val_accuracy: 0.7880 - val_loss: 0.5525 - val_precision: 0.8099 - val_recall: 0.7918 Epoch 4/10 79/79 ━━━━━━━━━━━━━━━━━━━━ 7s 93ms/step - accuracy: 0.9691 - loss: 0.1035 - precision: 0.9716 - recall: 0.9703 - val_accuracy: 0.7700 - val_loss: 0.7560 - val_precision: 0.7984 - val_recall: 0.7658 Evaluating baseline model... 32/32 ━━━━━━━━━━━━━━━━━━━━ 1s 30ms/step Accuracy: 0.8070 F1 Score: 0.8036 ==================== BASELINE EXAMPLES ==================== ✅ CORRECT EXAMPLES: Text: retelling a historically significant , and personal , episode True label: Positive, Model confidence: 0.9959 Text: by turns fanciful , grisly and engagingly quixotic . True label: Positive, Model confidence: 0.6074 Text: the greatest date movies in years True label: Positive, Model confidence: 0.9636 Text: a sluggish pace and lack of genuine narrative True label: Negative, Model confidence: 0.9441 Text: you go into the theater expecting a scary , action-packed chiller True label: Positive, Model confidence: 0.9537 ❌ INCORRECT EXAMPLES: Text: putrid writing , direction and timing with a smile that says , ` if i stay positive , maybe i can channel one of my greatest pictures , drunken master . True label: Negative, Predicted: Positive, Confidence: 0.9523 Text: there is something that is so meditative and lyrical about babak payami 's boldly quirky iranian drama secret ballot ... a charming and evoking little ditty that manages to show the gentle and humane side of middle eastern world politics True label: Positive, Predicted: Negative, Confidence: 0.8747 Text: to find a place among the studio 's animated classics True label: Positive, Predicted: Negative, Confidence: 0.6753 Text: uptight , middle class bores like antonia True label: Negative, Predicted: Positive, Confidence: 0.7888 Text: an adult who 's apparently been forced by his kids to watch too many barney videos True label: Positive, Predicted: Negative, Confidence: 0.9987 Running Word2Vec pipeline... ================================================================================ WORD2VEC MODEL ================================================================================ Preparing word2vec data... Extracting word2vec embeddings...
100%|██████████| 5000/5000 [00:02<00:00, 1739.93it/s]
Extracting word2vec embeddings...
100%|██████████| 500/500 [00:00<00:00, 3375.89it/s]
Extracting word2vec embeddings...
100%|██████████| 1000/1000 [00:00<00:00, 4091.53it/s]
Training word2vec model... Epoch 1/10 79/79 ━━━━━━━━━━━━━━━━━━━━ 8s 75ms/step - accuracy: 0.6466 - loss: 0.6438 - precision: 0.6498 - recall: 0.7087 - val_accuracy: 0.8000 - val_loss: 0.4955 - val_precision: 0.8789 - val_recall: 0.7286 Epoch 2/10 79/79 ━━━━━━━━━━━━━━━━━━━━ 6s 71ms/step - accuracy: 0.8039 - loss: 0.4868 - precision: 0.8343 - recall: 0.7909 - val_accuracy: 0.8180 - val_loss: 0.4304 - val_precision: 0.8803 - val_recall: 0.7658 Epoch 3/10 79/79 ━━━━━━━━━━━━━━━━━━━━ 7s 88ms/step - accuracy: 0.8289 - loss: 0.4255 - precision: 0.8472 - recall: 0.8305 - val_accuracy: 0.8440 - val_loss: 0.3665 - val_precision: 0.8898 - val_recall: 0.8104 Epoch 4/10 79/79 ━━━━━━━━━━━━━━━━━━━━ 7s 83ms/step - accuracy: 0.8459 - loss: 0.3828 - precision: 0.8630 - recall: 0.8465 - val_accuracy: 0.8440 - val_loss: 0.3591 - val_precision: 0.8898 - val_recall: 0.8104 Epoch 5/10 79/79 ━━━━━━━━━━━━━━━━━━━━ 6s 80ms/step - accuracy: 0.8497 - loss: 0.3578 - precision: 0.8645 - recall: 0.8528 - val_accuracy: 0.8360 - val_loss: 0.3912 - val_precision: 0.8945 - val_recall: 0.7881 Epoch 6/10 79/79 ━━━━━━━━━━━━━━━━━━━━ 6s 79ms/step - accuracy: 0.8563 - loss: 0.3447 - precision: 0.8820 - recall: 0.8446 - val_accuracy: 0.8540 - val_loss: 0.3388 - val_precision: 0.8952 - val_recall: 0.8253 Epoch 7/10 79/79 ━━━━━━━━━━━━━━━━━━━━ 6s 72ms/step - accuracy: 0.8743 - loss: 0.3128 - precision: 0.8926 - recall: 0.8700 - val_accuracy: 0.8500 - val_loss: 0.3515 - val_precision: 0.9076 - val_recall: 0.8030 Epoch 8/10 79/79 ━━━━━━━━━━━━━━━━━━━━ 6s 71ms/step - accuracy: 0.8829 - loss: 0.2919 - precision: 0.9017 - recall: 0.8770 - val_accuracy: 0.8200 - val_loss: 0.4402 - val_precision: 0.9050 - val_recall: 0.7435 Evaluating word2vec model... 32/32 ━━━━━━━━━━━━━━━━━━━━ 1s 19ms/step Accuracy: 0.8380 F1 Score: 0.8367 ==================== WORD2VEC EXAMPLES ==================== ✅ CORRECT EXAMPLES: Text: retelling a historically significant , and personal , episode True label: Positive, Model confidence: 0.9800 Text: by turns fanciful , grisly and engagingly quixotic . True label: Positive, Model confidence: 0.9729 Text: the greatest date movies in years True label: Positive, Model confidence: 0.9632 Text: a sluggish pace and lack of genuine narrative True label: Negative, Model confidence: 0.9876 Text: imaginatively mixed True label: Positive, Model confidence: 0.9177 ❌ INCORRECT EXAMPLES: Text: you go into the theater expecting a scary , action-packed chiller True label: Positive, Predicted: Negative, Confidence: 0.8745 Text: an adult who 's apparently been forced by his kids to watch too many barney videos True label: Positive, Predicted: Negative, Confidence: 0.9902 Text: a so-called ` comedy ' and not laugh True label: Negative, Predicted: Positive, Confidence: 0.5616 Text: sinister happy ending True label: Positive, Predicted: Negative, Confidence: 0.5407 Text: , swimming gets the details right , from its promenade of barely clad bodies in myrtle beach , s.c. , to the adrenaline jolt of a sudden lunch rush at the diner . True label: Positive, Predicted: Negative, Confidence: 0.7784 Running FastText pipeline... ================================================================================ FASTTEXT MODEL ================================================================================ Preparing fasttext data... Extracting fasttext embeddings...
100%|██████████| 5000/5000 [00:01<00:00, 2959.40it/s]
Extracting fasttext embeddings...
100%|██████████| 500/500 [00:00<00:00, 5373.69it/s]
Extracting fasttext embeddings...
100%|██████████| 1000/1000 [00:00<00:00, 5233.77it/s]
Training fasttext model... Epoch 1/10 79/79 ━━━━━━━━━━━━━━━━━━━━ 6s 50ms/step - accuracy: 0.6283 - loss: 0.6571 - precision: 0.6269 - recall: 0.7540 - val_accuracy: 0.7940 - val_loss: 0.5059 - val_precision: 0.7730 - val_recall: 0.8736 Epoch 2/10 79/79 ━━━━━━━━━━━━━━━━━━━━ 5s 64ms/step - accuracy: 0.7938 - loss: 0.5205 - precision: 0.7955 - recall: 0.8280 - val_accuracy: 0.8160 - val_loss: 0.4841 - val_precision: 0.8865 - val_recall: 0.7546 Epoch 3/10 79/79 ━━━━━━━━━━━━━━━━━━━━ 5s 67ms/step - accuracy: 0.8106 - loss: 0.4586 - precision: 0.8252 - recall: 0.8207 - val_accuracy: 0.8300 - val_loss: 0.4142 - val_precision: 0.8898 - val_recall: 0.7807 Epoch 4/10 79/79 ━━━━━━━━━━━━━━━━━━━━ 6s 75ms/step - accuracy: 0.8348 - loss: 0.4101 - precision: 0.8505 - recall: 0.8391 - val_accuracy: 0.8220 - val_loss: 0.4095 - val_precision: 0.8879 - val_recall: 0.7658 Epoch 5/10 79/79 ━━━━━━━━━━━━━━━━━━━━ 5s 67ms/step - accuracy: 0.8383 - loss: 0.3821 - precision: 0.8579 - recall: 0.8370 - val_accuracy: 0.8320 - val_loss: 0.4208 - val_precision: 0.8970 - val_recall: 0.7770 Epoch 6/10 79/79 ━━━━━━━━━━━━━━━━━━━━ 6s 80ms/step - accuracy: 0.8530 - loss: 0.3656 - precision: 0.8706 - recall: 0.8521 - val_accuracy: 0.8360 - val_loss: 0.4371 - val_precision: 0.9083 - val_recall: 0.7732 Evaluating fasttext model... 32/32 ━━━━━━━━━━━━━━━━━━━━ 1s 26ms/step Accuracy: 0.8220 F1 Score: 0.8214 ==================== FASTTEXT EXAMPLES ==================== ✅ CORRECT EXAMPLES: Text: retelling a historically significant , and personal , episode True label: Positive, Model confidence: 0.9251 Text: by turns fanciful , grisly and engagingly quixotic . True label: Positive, Model confidence: 0.8855 Text: the greatest date movies in years True label: Positive, Model confidence: 0.8628 Text: a sluggish pace and lack of genuine narrative True label: Negative, Model confidence: 0.9833 Text: imaginatively mixed True label: Positive, Model confidence: 0.7151 ❌ INCORRECT EXAMPLES: Text: you go into the theater expecting a scary , action-packed chiller True label: Positive, Predicted: Negative, Confidence: 0.5326 Text: 's easy to like True label: Positive, Predicted: Negative, Confidence: 0.5219 Text: an adult who 's apparently been forced by his kids to watch too many barney videos True label: Positive, Predicted: Negative, Confidence: 0.9921 Text: , swimming gets the details right , from its promenade of barely clad bodies in myrtle beach , s.c. , to the adrenaline jolt of a sudden lunch rush at the diner . True label: Positive, Predicted: Negative, Confidence: 0.8425 Text: attributable to a movie like this True label: Positive, Predicted: Negative, Confidence: 0.7568 Running ELMo pipeline... ================================================================================ ELMO MODEL ================================================================================ Preparing elmo data... Extracting elmo embeddings...
100%|██████████| 79/79 [07:14<00:00, 5.50s/it]
Extracting elmo embeddings...
100%|██████████| 8/8 [00:46<00:00, 5.78s/it]
Extracting elmo embeddings...
100%|██████████| 16/16 [01:28<00:00, 5.52s/it]
Training elmo model... Epoch 1/10 79/79 ━━━━━━━━━━━━━━━━━━━━ 8s 79ms/step - accuracy: 0.7104 - loss: 0.5580 - precision: 0.7176 - recall: 0.7464 - val_accuracy: 0.8500 - val_loss: 0.3529 - val_precision: 0.8975 - val_recall: 0.8141 Epoch 2/10 79/79 ━━━━━━━━━━━━━━━━━━━━ 8s 98ms/step - accuracy: 0.8682 - loss: 0.3109 - precision: 0.8925 - recall: 0.8569 - val_accuracy: 0.8660 - val_loss: 0.3209 - val_precision: 0.9040 - val_recall: 0.8401 Epoch 3/10 79/79 ━━━━━━━━━━━━━━━━━━━━ 7s 90ms/step - accuracy: 0.9243 - loss: 0.2148 - precision: 0.9382 - recall: 0.9192 - val_accuracy: 0.8560 - val_loss: 0.3389 - val_precision: 0.8924 - val_recall: 0.8327 Epoch 4/10 79/79 ━━━━━━━━━━━━━━━━━━━━ 9s 114ms/step - accuracy: 0.9537 - loss: 0.1401 - precision: 0.9662 - recall: 0.9468 - val_accuracy: 0.8560 - val_loss: 0.3925 - val_precision: 0.8893 - val_recall: 0.8364 Evaluating elmo model... 32/32 ━━━━━━━━━━━━━━━━━━━━ 1s 27ms/step Accuracy: 0.8480 F1 Score: 0.8467 ==================== ELMO EXAMPLES ==================== ✅ CORRECT EXAMPLES: Text: retelling a historically significant , and personal , episode True label: Positive, Model confidence: 0.9807 Text: by turns fanciful , grisly and engagingly quixotic . True label: Positive, Model confidence: 0.9468 Text: the greatest date movies in years True label: Positive, Model confidence: 0.9395 Text: a sluggish pace and lack of genuine narrative True label: Negative, Model confidence: 0.9918 Text: you go into the theater expecting a scary , action-packed chiller True label: Positive, Model confidence: 0.9047 ❌ INCORRECT EXAMPLES: Text: putrid writing , direction and timing with a smile that says , ` if i stay positive , maybe i can channel one of my greatest pictures , drunken master . True label: Negative, Predicted: Positive, Confidence: 0.9003 Text: if the title is a jeopardy question , then the answer might be `` how does steven seagal come across these days ? '' True label: Negative, Predicted: Positive, Confidence: 0.5212 Text: there is something that is so meditative and lyrical about babak payami 's boldly quirky iranian drama secret ballot ... a charming and evoking little ditty that manages to show the gentle and humane side of middle eastern world politics True label: Positive, Predicted: Negative, Confidence: 0.7730 Text: an adult who 's apparently been forced by his kids to watch too many barney videos True label: Positive, Predicted: Negative, Confidence: 0.9953 Text: even accepting this in the right frame of mind can only provide it with so much leniency . True label: Negative, Predicted: Positive, Confidence: 0.5123
📊 Final Results & Comparison¶
Once all models have run, we display their performance metrics and plots.
In [76]:
Copied!
# Compare all models
print("\n" + "="*80)
print("FINAL COMPARISON")
print("="*80)
for model in metrics:
print(f"\n{model.upper()}:")
print(f"Accuracy: {metrics[model]['accuracy']:.4f}")
print(f"F1 Score: {metrics[model]['f1']:.4f}")
# Plot comparison
plot_results(metrics)
# Compare all models
print("\n" + "="*80)
print("FINAL COMPARISON")
print("="*80)
for model in metrics:
print(f"\n{model.upper()}:")
print(f"Accuracy: {metrics[model]['accuracy']:.4f}")
print(f"F1 Score: {metrics[model]['f1']:.4f}")
# Plot comparison
plot_results(metrics)
================================================================================ FINAL COMPARISON ================================================================================ BASELINE: Accuracy: 0.8070 F1 Score: 0.8036 WORD2VEC: Accuracy: 0.8380 F1 Score: 0.8367 FASTTEXT: Accuracy: 0.8220 F1 Score: 0.8214 ELMO: Accuracy: 0.8480 F1 Score: 0.8467
✅ Conclusion: Extrinsic Evaluation¶
Across two downstream NLP tasks — Named Entity Recognition (NER) and Sentiment Classification (SST-2) — we assessed the real-world effectiveness of different types of word embeddings.
🔬 Summary of Results¶
| Task | Model | F1 Score |
|---|---|---|
| NER (CoNLL-2003) | Baseline | 0.8763 |
| Word2Vec | 0.8835 | |
| FastText | 0.9238 | |
| ELMo | 0.9713 | |
| SST-2 Sentiment | Baseline | 0.8036 |
| Word2Vec | 0.8367 | |
| FastText | 0.8214 | |
| ELMo | 0.8467 |
🧠 Key Takeaways¶
Pretrained Embeddings Improve Generalization:
Both Word2Vec and FastText improve upon the baseline, confirming that pretrained embeddings offer strong inductive bias, even with small training sets.FastText Outperforms Word2Vec:
FastText's subword modeling gives it an edge, especially for rare or OOV words — visible in both NER and SST-2 tasks.ELMo Leads the Pack:
ELMo consistently achieves the best performance. Its contextual nature allows it to dynamically adapt word representations based on sentence-level context — especially helpful in cases where static embeddings fail to disambiguate meaning.NER vs. Classification:
- On NER, the performance gap between models is more pronounced due to the complexity of multi-class sequence tagging and entity boundaries.
- On SST-2, the difference is smaller but still significant, suggesting that even sentence-level tasks benefit from contextual encoding.
🧩 Final Thoughts¶
This extrinsic evaluation confirms that:
- Static embeddings (Word2Vec, FastText) are fast and effective.
- Contextual embeddings (ELMo) are more powerful, especially when polysemy and context sensitivity matter.
- For modern NLP pipelines, contextual models — even if more expensive — offer the best trade-off between performance and robustness.
In [ ]:
Copied!