Skip to main content

date-guesser.py

Many strings—page titles, abstracts, URLs, file names—contain dates in wildly varying formats.

Pathbuild/date-guesser.py
LanguagePython
Lines858
Sourcebuild/date-guesser.py
at 406d3e423

Read this when Use this page when tracing Python helper scripts for metadata cleanup, text processing, PDFs, dates, or generated content around date-guesser.

Overview​

Many strings—page titles, abstracts, URLs, file names—contain dates in wildly varying formats. Writing regular expressions to cover all possible formats would be labor-intensive and error-prone. This script uses a hybrid pipeline: an exact-match lookup table of manually verified dates, then deterministic regex extractors (ArXiv IDs, Unix timestamps, written-out month names, labeled numeric dates, URL paths, local PDF filenames), and only falls back to an LLM (GPT-5-mini by default; overridable via the OPENAI_MODEL environment variable) when deterministic extraction fails. A guard (should_skip_model) prevents opaque URL/ID families (PMC, Hacker News, Twitter snowflakes, etc.) from reaching the LLM at all, to avoid plausible hallucinated dates.

The script is conservative by design: when in doubt, it returns an empty string rather than risk guessing incorrectly. This makes it safe for automated processing pipelines where false positives would be costly. It understands complex edge cases like ArXiv IDs (YYMM.NNNNN format), leap years, month/day validity, Internet Archive timestamps vs. original publication dates, and more.

Date extraction returns the most precise format available: full dates when possible (YYYY-MM-DD), month precision when only year and month are clear (YYYY-MM), or just the year (YYYY) when that's all that's reliably available. The script validates all dates against calendar rules and filters out future dates and dates before 1000 AD.

Key Functions​

  • validate_date_format(date_str): Validates date string matches YYYY[-MM[-DD]] regex patterns
  • validate_date_not_future(date_str): Ensures date is not in the future and respects leap years and month lengths
  • exact_example_lookup(target): Checks the input against the EXAMPLES table for exact-match manually verified dates (rows flagged False are corrections hidden from the LLM prompt)
  • deterministic_extract(text): Regex-based extraction for ArXiv IDs, Unix timestamps, month-name dates, labeled numeric dates, URL path dates, and local PDF filenames (Wikipedia URLs are excluded outright)
  • should_skip_model(target): Guard that returns empty for opaque URL/ID families rather than letting the LLM guess
  • main(): Orchestrates the pipeline: exact lookup → deterministic extraction → skip-guard → LLM fallback (ask_model), then validates the result
  • EXAMPLES table: Hundreds of input/output examples, doubling as few-shot prompt material and exact-match overrides for URLs, metadata, ArXiv IDs, DOIs, timestamps, version numbers, and ambiguous formats

Command Line Usage​

# From stdin
echo 'https://erikbern.com/2016/04/04/nyc-subway-math' | OPENAI_API_KEY="sk-XXX" python date-guesser.py
# Output: 2016-04-04

# From command-line argument
OPENAI_API_KEY="sk-XXX" python date-guesser.py "Posted on April 4, 2016"
# Output: 2016-04-04

# Handle ArXiv IDs
echo "arXiv:2401.12345" | OPENAI_API_KEY="sk-XXX" python date-guesser.py
# Output: 2024-01

# Conservative on ambiguity
echo "Updated: 2 days ago" | OPENAI_API_KEY="sk-XXX" python date-guesser.py
# Output: (empty string)

Requirements:

  • Python 3
  • openai Python package
  • Valid OpenAI API key in $OPENAI_API_KEY environment variable

Edge cases handled:

  • ArXiv IDs (YYMM.NNNNN format where only first 4 digits are date)
  • Leap year validation (e.g., 2024-02-29 valid, 2023-02-29 invalid)
  • Month/day validity (e.g., rejects 2024-13-01, 2024-04-31)
  • Date ranges (returns earliest date)
  • Internet Archive vs. original publication dates
  • Wikipedia articles (returns empty—no meaningful date)
  • Product numbers, ISBNs, postal codes, room numbers (ignored)
See Also