Skip to main content

preprocess-annotation.sh

preprocess-annotation.sh is a minimal wrapper script that reads raw annotation content from stdin and pipes it through preprocessMarkdown to prepare it for manual editing in.

Pathbuild/preprocess-annotation.sh
LanguageBash
Lines7

Read this when Use this page when tracing shell automation, compression, upload/download helpers, linting, or preprocessing around preprocess-annotation.

Overview​

preprocess-annotation.sh is a minimal wrapper script that reads raw annotation content from stdin and pipes it through preprocessMarkdown to prepare it for manual editing in Emacs. The preprocessed output is then stored in annotation database files (full.gtx or half.gtx).

This script is part of the manual annotation editing workflow on gwern.net. When adding or updating link annotations, the raw scraped or draft content is passed through this script to normalize formatting before being opened in an editor buffer. Historically, this also included Pandoc HTML conversion and HTML Tidy formatting (now commented out), but the current version only performs Markdown preprocessing.

The simplicity of this script reflects the design philosophy that most annotation processing should happen in dedicated tools (preprocessMarkdown) rather than being scattered across multiple scripts.

Key Commands/Variables​

Main pipeline:

cat - | preprocessMarkdown

Components:

  • cat - - Read from stdin (explicit pipe-through idiom)
  • preprocessMarkdown (from $PATH) - Core Markdown normalizer

Commented-out legacy pipeline:

# pandoc --mathjax --metadata title='Annotation preview' --to=html5 --from=html
# tidy -quiet --show-warnings no --show-body-only auto -indent -wrap 0 \
#      --clean yes --merge-divs no --break-before-br yes --logical-emphasis yes \
#      --quote-nbsp no || true
  • pandoc - Would re-render the HTML output as HTML5 (--from=html --to=html5) with MathJax support
  • tidy - Would format/clean HTML with specific options
  • || true - Suppress tidy's warning exit status (always exits 1)

Purpose of legacy code: These post-processing stages operated on preprocessMarkdown's HTML output (note the --from=html), re-rendering and pretty-printing it. They are now disabled; the raw preprocessMarkdown output is used directly. (Annotation abstracts are stored as HTML in the .gtx databases, so the HTML conversion happens here, inside preprocessMarkdown, not at build time.)

Usage​

Standard invocation (from stdin):

echo "Some *raw* annotation text" | ./preprocess-annotation.sh > processed.txt

In Emacs workflow (actual invocation): The script is invoked by gwern.net's Emacs configuration (build/markdown.el), which binds it as the markdown-command when compiling an annotation draft buffer:

;; from build/markdown.el:
(let ((markdown-command "preprocess-annotation.sh"))
  (markdown-kill-ring-save)
  ...)

Because preprocessMarkdown calls the OpenAI API (for the GenerateSimilar recommendations), $OPENAI_API_KEY must be defined in the Emacs environment.

Manual testing:

# Test preprocessing on a sample annotation
cat <<EOF | ./preprocess-annotation.sh
A *neural* network paper about [transformers](https://example.com).

Contains **bold** and _italic_ text.
EOF

What preprocessMarkdown actually does (source: build/app/preprocessMarkdown.hs):

  • Parses the stdin Markdown with Pandoc
  • Converts interwiki links (Interwiki.convertInterwikiLinks)
  • Validates any Wikipedia links (checks articles exist and are not disambiguation pages)
  • Renders to HTML5 (annotation abstracts are stored as HTML in the .gtx databases)
  • Cleans the HTML with LinkMetadata.cleanAbstractsHTML
  • Appends a GenerateSimilar "See Also" recommendations block (embedding-based; requires $OPENAI_API_KEY)

No arguments:

  • Script takes no command-line arguments
  • All input via stdin, output to stdout (Unix filter pattern)

Exit status:

  • Returns exit status of preprocessMarkdown (pipe propagates last command status)
See Also