preprocessMarkdown.hs

Standalone preprocessor that transforms Markdown abstracts into cleaned HTML with interwiki expansion and recommendations

Pathbuild/app/preprocessMarkdown.hs

LanguageHaskell

Lines47

Sourcebuild/app/preprocessMarkdown.hs
at 406d3e423

Read this when Use this page when tracing the Haskell build pipeline, generators, metadata code, or backend utility behavior around preprocessMarkdown.

Overview

preprocessMarkdown.hs is a small, focused command-line utility that reads Markdown from stdin, applies a series of Pandoc AST transformations, and outputs enriched HTML. It serves as a preprocessing step for annotations and abstracts before they enter the main Hakyll build pipeline.

The module performs three key operations: (1) interwiki link expansion, (2) Wikipedia link validation, and (3) generation of "See Also" recommendations using embedding-based similarity matching. The active source imports LinkMetadata, Interwiki, GenerateSimilar, Utils, Config.Misc, and Query.

This tool is designed for single-document processing (annotations, abstracts) rather than full pages. It is invoked during annotation writing in Emacs: build/markdown.el sets markdown-command to build/preprocess-annotation.sh, which pipes the annotation Markdown through the preprocessMarkdown binary. (Because the recommendation step calls the OpenAI embedding API, OPENAI_API_KEY must be set in the Emacs environment.)

---

Public API

main :: IO ()

Entry point that orchestrates the preprocessing pipeline.

Pipeline:

1. Read Markdown from stdin
2. Parse to Pandoc AST with full Pandoc extensions
3. Apply convertInterwikiLinks transformation
4. Validate Wikipedia links via checkWP
5. Render to HTML5
6. Clean abstracts via cleanAbstractsHTML
7. Remove class="link-live" from the standalone output
8. Generate embedding-based recommendations
9. Output HTML with optional "See Also" section

Called by: build/preprocess-annotation.sh (which Emacs runs as markdown-command, configured in build/markdown.el) Calls: convertInterwikiLinks, checkWP, cleanAbstractsHTML, singleShotRecommendations

---

Internal Architecture

Processing Pipeline

stdin (Markdown)
    │
    ▼
┌─────────────────────┐
│ readMarkdown        │  Parse with pandocExtensions
└─────────────────────┘
    │
    ▼
┌─────────────────────┐
│ convertInterwikiLinks│  !W, !G, etc. → full URLs
└─────────────────────┘
    │
    ▼
┌─────────────────────┐
│ checkWP             │  Validate Wikipedia links
└─────────────────────┘
    │
    ▼
┌─────────────────────┐
│ writeHtml5String    │  Render to HTML
└─────────────────────┘
    │
    ▼
┌─────────────────────┐
│ cleanAbstractsHTML  │  Sanitize output
└─────────────────────┘
    │
    ▼
┌─────────────────────┐
│ singleShotRecommendations │  Embedding similarity
└─────────────────────┘
    │
    ▼
stdout (HTML + See Also)

Wikipedia Validation

checkWP :: Pandoc -> IO ()
checkWP p = do
  let links = filter ("wikipedia.org"`T.isInfixOf`) $ extractURLs p
  mapM_ (isWPArticle True) links  -- Check existence
  mapM_ isWPDisambig links        -- Warn on disambiguation pages

This validation catches two common errors:

• Links to non-existent Wikipedia articles (typos, deleted pages)
• Links to disambiguation pages (should link to specific article)

---

Key Patterns

Standalone Document Processing

Unlike the main Hakyll build which processes whole sites, this tool handles single documents in isolation:

main = do
  originalMarkdown <- TIO.getContents  -- Single document from stdin
  -- ... process ...
  putStrLn html                         -- Single document to stdout

This design enables:

• Integration with Unix pipelines
• Use in annotation creation workflows
• Testing transformations on individual documents

Working Directory Management

The tool explicitly sets the working directory before loading databases:

C.cd  -- Ensure correct directory for metadata databases
matchList <- GS.singleShotRecommendations html

This is necessary because singleShotRecommendations reads the embeddings database and metadata from project-relative paths. In this single-shot mode, backlinks context is empty rather than loaded from the backlinks database.

See Also Formatting

Recommendations are wrapped in a collapsible div for consistent styling:

"<div class=\"aux-links-append see-also-append collapse\">\n\n" ++
"<p><strong>See Also</strong>:</p>\n\n" ++
matchList ++
"\n</div>"

The collapse class allows the recommendations to be hidden by default on pages where they might be distracting.

---

Configuration

Setting	Source	Purpose
pandocExtensions	Pandoc	Full Markdown extension set
safeHtmlWriterOptions	Utils	HTML output settings
Interwiki prefixes	Config.Interwiki	Shorthand → URL mappings
C.cd	Config.Misc	Working directory path

---

Integration Points

Dependencies

Module	Usage
LinkMetadata	cleanAbstractsHTML for output sanitization
Interwiki	convertInterwikiLinks, isWPArticle, isWPDisambig
GenerateSimilar	singleShotRecommendations for See Also generation
Query	extractURLs for Wikipedia link validation

Input/Output

• Input: Markdown text via stdin
• Output: HTML with optional See Also section via stdout
• Side effects: HTTP requests to Wikipedia API for link validation

Database Access

Reads (via singleShotRecommendations):

• Embeddings database (metadata/embeddings.bin)
• Metadata database (.gtx file, for rendering matched entries)
• No backlinks database; single-shot recommendations use empty backlinks context

---

See Also

• hakyll.hs - Main build system that uses similar transforms
• sync.sh - Build orchestrator (does not invoke this tool; it is run from Emacs via preprocess-annotation.sh)
• Typography.hs - Typography transforms shared with hakyll
• Interwiki.hs - Interwiki link expansion
• GenerateSimilar.hs - Embedding-based recommendations
• LinkMetadata.hs - Metadata and abstract handling