GenerateSimilar.hs

Embedding-based semantic similarity system for generating "Similar Links" recommendations

Pathbuild/GenerateSimilar.hs

LanguageHaskell

Lines1610

Sourcebuild/GenerateSimilar.hs
at 406d3e423

Read this when Use this page when tracing the Haskell build pipeline, generators, metadata code, or backend utility behavior around GenerateSimilar.

Overview

GenerateSimilar implements gwern.net's "Similar Links" feature, which uses OpenAI embeddings to find semantically related pages. When you see the "Similar Links" tab on an annotation popup, this module generated those recommendations.

The system works in three phases: (1) document formatting, converting metadata items into embedding-friendly plaintext; (2) embedding generation through the OpenAI API wrapper; and (3) exact nearest-neighbor lookup over normalized vectors. The embeddings are stored in a binary cache (metadata/embeddings.bin) and rebuilt incrementally as new annotations are added.

A secondary capability is "sort by magic"—using embeddings to order lists by semantic similarity rather than date or alphabetically. This creates visually coherent clusters when displaying tag pages, making related items appear adjacent.

---

Public API

singleShotRecommendations :: String -> IO T.Text

Generate recommendations HTML for arbitrary HTML text (used during annotation editing).

Called by: preprocessMarkdown.hs Calls: readEmbeddings, embed, embeddings2Index, lookupEmbeddingK, seriateGreedy, generateMatches

readEmbeddings :: IO Embeddings

Load the embedding database from metadata/embeddings.bin.

type Embedding = (String,    -- URL/path
                  Integer,   -- ModifiedJulianDay / embedding age
                  String,    -- Formatted document text
                  String,    -- Model version ID
                  [Double])  -- Embedding vector
type Embeddings = [Embedding]

Called by: generateSimilarLinks.hs, singleShotRecommendations, sortTagByTopic Calls: DB.decodeFileOrFail

writeEmbeddings :: Embeddings -> IO ()

Persist embeddings to disk with corruption check.

Called by: generateSimilarLinks.hs Calls: DB.encodeFile, atomic rename via temp file

embed :: Embeddings -> Metadata -> Backlinks -> (String, MetadataItem) -> IO Embedding

Create an embedding for a single document. Checks for existing embedding with same basename (handles renames). Appends backlink metadata for richer context.

Called by: generateSimilarLinks.hs Calls: formatDoc, oaAPIEmbed

missingEmbeddings :: Metadata -> Embeddings -> [(String, MetadataItem)]

Find metadata entries that don't yet have embeddings.

Called by: generateSimilarLinks.hs

pruneEmbeddings :: Metadata -> Embeddings -> Embeddings

Remove stale embeddings that no longer have corresponding metadata entries.

Called by: generateSimilarLinks.hs

writeOutMatch :: Metadata -> Backlinks -> (String, [String]) -> IO ()

Write a similar-links HTML fragment to metadata/annotation/similar/.

Called by: generateSimilarLinks.hs Calls: generateMatches, writeUpdatedFile

sortTagByTopic :: Metadata -> String -> IO [FilePath]

Sort all pages with a given tag by semantic similarity (for tag index pages).

Called by: no current external callers (tag directory pages use sortSimilarsStartingWithNewestWithTagEdb from generateDirectory.hs) Calls: sortSimilars

sortSimilars :: Embeddings -> ListSortedMagic -> FilePath -> [FilePath] -> IO [FilePath]

Sort a list of paths by embedding similarity, starting from a seed. Caches results in metadata/listsortedmagic.hs.

Called by: sortTagByTopic, sortSimilarsStartingWithNewestEdb Calls: seriateGreedy (via embeddings2Index)

---

Internal Architecture

Data Flow

Metadata Item
    ↓
formatDoc (title, author, date, tags, abstract, backlinks → plaintext)
    ↓
oaAPIEmbed (shell out to embed.sh → OpenAI API)
    ↓
Embedding (URL, date, text, model, [Double])
    ↓
embeddings2Index (normalize vectors into an in-memory index)
    ↓
lookupPathK / lookupEmbeddingK (exact cosine-distance scan)
    ↓
generateMatches (filter, format as Pandoc → HTML)

Key Data Structures

Embedding tuple: (URL, ModifiedJulianDay, FormattedText, ModelID, Vector)

• URL: Path like /doc/ai/gpt.pdf or full URL
• CreationDate: Julian day integer for age-based expiry
• FormattedText: The plaintext sent to OpenAI (debugging)
• ModelID: e.g., text-embedding-3-large
• Vector: Double-valued embedding vector

EmbeddingIndex: Normalized in-memory search index

data EmbeddingIndex = EmbeddingIndex
  { eiRows   :: Vector EmbeddingRow
  , eiByPath :: Map FilePath Int
  }

Search Semantics

The current implementation performs exact cosine-distance search. Vectors are normalized once when converted into EmbeddingIndex, and lookup retains only the best k candidates during the scan rather than allocating and sorting every scored candidate.

Distances are sorted ascending: 0 means identical direction, 1 means orthogonal, and 2 means opposite direction. Embeddings from different model IDs are not compared.

---

Key Patterns

Document Formatting

formatDoc converts metadata to embedding-friendly text:

formatDoc (path, (title, author, date, dateModified, _, tags, abstract)) =
    -- "'Title' (URL), by Author (2024; updated 2024-06)."
    -- "Keywords: tag1, tag2."
    -- [abstract text]
    -- "References:\n1. /doc/foo.pdf Title..."

The format is designed to be comprehensible to language models without HTML parsing. URLs are extracted and listed as numbered references since simplifiedDoc strips links.

Backlink Enrichment

Embeddings include reverse citations to improve similarity matching:

let backlinksMetadata = "\n\nReverse citations:\n\n- " ++
    intercalate "\n- " (map formatBacklink backlinks)

Incremental Updates

New embeddings are generated only for items missing from the cache. When an item is embedded, related items are expired to force regeneration:

expireMatches :: [String] -> IO ()
expireMatches = mapM_ (removeFile . fst . getSimilarLink)

Clustering for Display

clusterIntoSublist works on an already-seriated list, computes adjacent embedding distances, splits at the largest adjacent gaps, and returns the original list unsplit when computed k == 1:

clusterIntoSublist es list =
    let k = max 1 (round(sqrt(fromIntegral $ length list)) - 1)
    in if k == 1
       then [list]
       else splitAtLargestAdjacentGaps list

---

Configuration

From Config/GenerateSimilar.hs:

Setting	Value	Purpose
bestNEmbeddings	20	Max similar links to show
maximumLength	32,700	Max chars for embedding (≈8k tokens)
minimumSuggestions	3	Skip writing if the pre-pruning match list has fewer matches
maxDistance	0.95	Cosine-distance threshold for relevance
embeddingsPath	metadata/embeddings.bin	Cache location
blackList	/index, /changelog, etc.	Exclude pathological matches

---

Integration Points

External Dependencies

embed.sh: Shell script calling OpenAI API

ENGINE="text-embedding-3-large"
ENGINE_DIMENSION="512"
curl "https://api.openai.com/v1/engines/$ENGINE/embeddings" \
    -d "{\"input\": \"$TEXT\", \"dimensions\": $ENGINE_DIMENSION}"

Vector index: normalized in-memory EmbeddingIndex

• Stores normalized embedding rows and a path lookup map
• Uses exact cosine-distance scans with bounded top-k collection

File Outputs

• metadata/embeddings.bin — Binary cache (Data.Binary format)
• metadata/annotation/similar/*.html — Pre-rendered HTML fragments
• metadata/listsortedmagic.hs — Cached sort-by-similarity results
• metadata/listname.hs — LLM-generated cluster names

Shared State

• Reads: LinkMetadata (annotation database), Backlinks (link graph)
• LLM integration: tagguesser.py generates cluster names from titles

---

See Also

• Config.GenerateSimilar - Configuration constants for embedding parameters and thresholds
• generateSimilarLinks.hs - CLI entry point that uses GenerateSimilar functions
• embed.sh - Shell script that calls OpenAI API for embedding generation
• LinkMetadata.hs - Annotation database that provides content for embeddings
• LinkBacklink.hs - Backlinks database used to enrich embedding context
• sync.sh - Build orchestrator that schedules similarity generation
• extracts.js - Frontend displaying similar links in annotation popups